Skip to main content

Cornelis Technical Documentation

6.5.4. MPI Bandwidth

MPI bandwidth is measured between two nodes using one or more ranks per node. Increasing the number of ranks per node helps scale aggregate bandwidth across smaller message sizes.

Cornelis recommends using the Uniband (unidirectional) and Biband (bidirectional) benchmarks from IMB, as they issue simultaneous non-blocking sends, which can saturate the link.

6.5.4.1. Example: Unidirectional Bandwidth with 1 Rank per Node

mpirun -np 2 -ppn 1 -host hostA,hostB \
  -genv FI_PROVIDER=opx \
  -genv I_MPI_FABRICS=shm:ofi \
  -genv I_MPI_PIN_PROCESSOR_LIST=<local_numa_cores> \
  IMB-MPI1 Uniband -npmin 2

Sample output:

#---------------------------------------------------
# Benchmarking Uniband
# #processes = 2
#---------------------------------------------------
#bytes #repetitions Mbytes/sec Msg/sec
...

  • Mbytes/sec indicates bandwidth.

  • Msg/sec represents the message rate (covered in the next section).

  • For bidirectional tests, replace Uniband with Biband.

The equivalent OSU benchmarks are osu_bw (unidirectional) and osu_bibw (bidirectional).

6.5.4.2. Example: Run Uniband and Biband Simultaneously (4 Ranks per Node)

mpirun -np 8 -ppn 4 -host hostA,hostB \
  -genv FI_PROVIDER=opx \
  -genv I_MPI_FABRICS=shm:ofi \
  -genv I_MPI_PIN_PROCESSOR_LIST=<local_numa_cores> \
  IMB-MPI1 Uniband Biband -npmin 8

Note

Use -npmin 8 to ensure that 8 ranks (4 per node) participate.

Note

The equivalent OSU benchmark for multi-pair bandwidth is osu_mbw_mr. There is no direct OSU benchmark for bidirectional multi-pair traffic.