6.5.9. Using Multiple CN5000 SuperNICs
For systems with multiple SuperNICs installed, MPI processes will, by default, use NUMA-local SuperNICs first before looking off-NUMA for a SuperNIC to use. For example, a dual-socket server with one SuperNIC installed on NUMA 0 (cores 0-15) and one SuperNIC installed on NUMA 1 (cores 16-31), a two-process per node benchmark will use both SuperNICs by default with the following Intel MPI pinning:
mpirun -np 4 -ppn 2 -host hostA,hostB -genv I_MPI_PIN_PROCESSOR_LIST=0,16 ...
Applications that are run with many cores per node, occupying all NUMAs, will use all installed SuperNICs by default.