Skip to main content

Cornelis Technical Documentation

6.9.5. Running SOS Benchmarks over CN5000

SOS benchmarks are launched with Hydra’s mpiexec.hydra as the PMI-simple launcher, with the OFI provider set to opx for CN5000. For starters two ranks (one per node) are used for the SOS shmem_*_perf benchmarks:

export PATH=<hydrainstdir>/bin:$PATH
export  LD_LIBRARY_PATH=<sosinstdir>/lib:$LD_LIBRARY_PATH

mpiexec.hydra -host <host1>,<host2> -n 2 -ppn 1 \
 -genv LD_LIBRARY_PATH ${LD_LIBRARY_PATH} \
 -genv FI_PROVIDER opx \
 -genv SHMEM_OFI_STX_MAX 0 \
 -genv SHMEM_OFI_PROVIDER opx \
 <sosinstdir>/bin/shmem_latency_put_perf

Other available SOS benchmarks to substitute for shmem_latency_put_perf:

shmem_bibw_atomics_perfshmem_bw_get_perf_nb
shmem_bibw_get_perfshmem_bw_nb_atomics_perf
shmem_bibw_get_perf_nbshmem_bw_put_perf
shmem_bibw_nb_atomics_perfshmem_bw_put_perf_nb
shmem_bibw_put_perfshmem_latency_get_perf
shmem_bibw_put_perf_nbshmem_latency_get_perf_nb
shmem_bw_atomics_perfshmem_latency_put_perf
shmem_bw_get_perfshmem_latency_put_perf_nb

If you want to increase the number of ranks per node, two additional parameters, FI_OPX_SDMA_MIN_PAYLOAD_B and FI_OPX_HFISVC_MIN_PAYLOAD_BYTES, should be taken into account. These environment parameters influence the OPX Software stack for when to switch between optimization for small or large packets. The default value is 32kb, but this might not optimal when running eight or more processes per interconnect adapter.

The following is an example for running with 64 processes per node:

mpiexec.hydra -host HOST1,HOST2 \
              -n 128 -ppn 64 \
              -genv FI_OPX_HFI_SELECT 0 \
              -genv SHMEM_OFI_STX_MAX 0 \
              -genv SHMEM_OFI_PROVIDER opx \
              -genv FI_OPX_HFISVC 1 \
              -genv FI_OPX_SDMA_MIN_PAYLOAD_BYTES 1048577 \
              -genv FI_OPX_HFISVC_MIN_PAYLOAD_BYTES 1048577 \
              <sosinstdir>/bin/shmem_bibw_get_perf_nb

                    Sandia OpenSHMEM Performance Suite
                    ==================================
  Total Number of PEs:         128       Window size:                   64
  Number of source PEs:         64       Maximum message size:     8388608
  Number of target PEs:         64       Number of threads:              1
  Iteration count:            1000       Thread safety:             SINGLE

Bandwidth test type:       Bi-dir

 Message Size              Bandwidth                Message Rate
     in bytes          in mbytes/sec                 in msgs/sec
...