Skip to main content

Cornelis Technical Documentation

6.10.5. NCCL Benchmarks

6.10.5.1. Running NCCL Benchmarks with Open MPI

NVIDIA Collective Communications Library (NCCL) provides highly optimized collective and point-to-point (P2P) communication primitives tailored for multi-GPU systems. It is designed to deliver low-latency and high-bandwidth performance over high-speed interconnects such as PCIe and NVLink.

For more information, visit the official NCCL page: NVIDIA NCCL.

Benchmarks are built from nccl-tests, which produces per-collective binaries (all_reduce_perf, all_gather_perf, broadcast_perf, reduce_scatter_perf, sendrecv_perf, etc.).

Two networking approaches are shown in the following sections: the aws-ofi-nccl plugin (NCCL_NET unset, defaulting to ofi) and NCCL’s built-in Verbs/IB plugin (NCCL_NET=IB). Both approaches should be evaluated on a given deployment, since results can shift with topology, SuperNIC count, and driver/plugin versions.

6.10.5.2. Approach 1: aws-ofi-nccl Plugin

6.10.5.2.1. Example: NCCL all_reduce_perf with Open MPI and OPX across two nodes, one process per GPU

Compile and install the AWS OFI libfabric plugin found on aws-ofi-nccl (GitHub). The following instructions assume the plugin is installed in <path-to-aws-ofi-nccl>.

The following is a two-node, four-GPU-per-node run against two hosts (node1, node2), using the aws-ofi-nccl plugin. Each of the four ranks per node uses its own nearest CN5000 SuperNIC (by PCIe/NUMA locality); all four SuperNICs on the node are active. On this particular server, there are four NUMA nodes, one GPU, and one NIC per NUMA.

mpirun -np 8 -host node1:4,node2:4 \
 --map-by ppr:1:numa:PE=4:ordered \
 -mca mtl ofi -mca btl self,sm -mca pml cm \
 -x FI_OPX_HFISVC=1 -x FI_HMEM_CUDA_USE_GDRCOPY=1 \
 -x CUDA_VISIBLE_DEVICES=0,1,2,3 \
 -x LD_LIBRARY_PATH=/usr/local/cuda/lib64:<path-to-aws-ofi-nccl>/lib:<path-to-openmpi>/lib: \
 -x NCCL_SOCKET_IFNAME=ib0 \
 -x NCCL_CROSS_NIC=1 \
 <path-to-nccl-tests-build>/all_reduce_perf -b 67108864 -e 1073741824 -f 2 -g 1

Replace the placeholders (<...>) with the paths appropriate for your system. Consider adding the additional tunables in the following section (NCCL_ALGO/NCCL_PROTO, and channel counts) to further optimize multi-SuperNIC scaling.

If there are GPUs on the platform not using NVLink, an additional flag may be required to avoid hangs with the aws‑ofi‑nccl plugin: NCCL_CUMEM_ENABLE=0. This workaround is under investigation.

6.10.5.2.2. Key Performance Variables

  • -x CUDA_VISIBLE_DEVICES=0,1,2,3: Orders the logical GPU-to-rank mapping so that each GPU’s nearest SuperNIC (by PCIe/NUMA locality) and its NVLink neighbors are used efficiently. The correct order is platform-specific and should be derived from the system’s PCIe/NVLink topology (see nvidia-smi topo -m or the vendor topology tool).

  • -x FI_OPX_HFISVC=1: Enables the OPX HFI1 Service layer required for the Libfabric OPX Provider to correctly initialize on CN5000.

  • -x FI_HMEM_CUDA_USE_DMABUF=1: Enables dma-buf on CN5000.

  • -x FI_HMEM_CUDA_USE_GDRCOPY=1: Enables GDRCopy support for low-latency CUDA memory registration on CN5000.

  • -x NCCL_SOCKET_IFNAME=ib0: Restricts the out-of-band NCCL socket bootstrap to the desired network interface (e.g., the IPoIB management interface) to avoid inadvertently bootstrapping over an unintended SuperNIC.

  • -x NCCL_CROSS_NIC=1: Allows a single collective operation ring/tree to cross between different SuperNICs.

  • --map-by ppr:1:numa:PE=4: Pins each of the four ranks per node to a unique NUMA node, allowing four cores to be used per rank. The resulting bindings with --report-bindings. A PE value of 2, 4, 8, and so on can be studied.

Additional tunables to evaluate, especially with multiple SuperNICs per node:

  • -x NCCL_ALGO=Tree -x NCCL_PROTO=Simple: Selects the Tree algorithm with the Simple protocol; on multi‑node GPU clusters this combination is often a good starting point, but Ring and LL/LL128 variants should also be evaluated for the target message-size range.

  • -x NCCL_MIN_NCHANNELS=<n> -x NCCL_MAX_NCHANNELS=<n>: Fixes the number of communication channels; tune upward to better utilize multiple SuperNICs per node, generally starting near the SuperNIC count.

  • -x NCCL_P2P_LEVEL=sys: Allows peer-to-peer transfers across the full system hierarchy (e.g., through NVLink) rather than restricting P2P to GPUs on the same PCIe switch.

  • -x NCCL_NET_GDR_LEVEL=sys: Enables GPUDirect RDMA across the full system hierarchy, consistent with NCCL_P2P_LEVEL=sys.

  • -x NCCL_NET_GDR_READ=0: Disables sending data to the SuperNIC directly for optimal performance. Users may wish to experiment with this, with mixed impact dependent on exact cluster topology.

6.10.5.2.3. GPU, SuperNIC, and Rank Binding

NCCL performance depends heavily on placing each MPI rank on the CPU core(s) local to its assigned GPU and SuperNIC. On systems with one GPU and one SuperNIC per NUMA, --map-by ppr:1:numa works well. Another option is to use --map-by pe-list=<core-list>:ordered (or an equivalent OpenMPI/MPI binding option) to pin ranks to the correct NUMA node, and verify the resulting bindings with --report-bindings. Combine this with a CUDA_VISIBLE_DEVICES ordering and, where necessary, a custom NCCL_TOPO_FILE, so that each rank’s GPU, SuperNIC, and CPU core all reside on the same NUMA node/socket.

Note

It is advised to configure GPU nodes such that each NUMA node has a GPU and a CN5000 SuperNIC.

With careful pinning as shown here, all communications from a GPU use the NUMA-local dedicated SuperNIC.

6.10.5.3. Approach 2: Built-in Verbs/IB Plugin (NCCL_NETIB)

Rather than the aws-ofi-nccl plugin, NCCL can use its built-in Verbs-based IB plugin directly by setting NCCL_NET=IB. In this mode, NCCL uses the CN5000 SuperNICs with standard verbs instead of Libfabric, and is pointed at all NICs on a node via NCCL_IB_HCA. NCCL_DMABUF_ENABLE=1 is required in this mode: without it, ibv_reg_mr of GPU memory over hfi1 fails with Bad address (NCCL_NET_GDR_LEVEL=LOC also works, but is slower). A NCCL_TOPO_FILE may also be required when the auto-detected topology does not accurately reflect the platform (for example, when GPUs report ambiguous or identical identifiers that confuse topology auto-detection).

Additional tunables that can meaningfully improve this path on other CN5000 platforms include NCCL_IB_QPS_PER_CONNECTION (queue pairs per connection) and NCCL_BUFFSIZE (internal communication buffer size). Both are platform-dependent and should be swept for the target system.

mpirun -np 8 -host node1:4,node2:4 \
 --map-by ppr:1:numa:PE=4:ordered \
 -mca mtl ofi -mca btl self,sm -mca pml cm \
 -x CUDA_VISIBLE_DEVICES=1,2,3,4 \
 -x  LD_LIBRARY_PATH=<path-to-openmpi>/lib:/usr/local/cuda/lib64:  \
 -x NCCL_SOCKET_IFNAME=ib0 \
 -x NCCL_CROSS_NIC=1 \
 -x NCCL_NET=IB \
 -x NCCL_IB_HCA=hfi1_0:2,hfi1_1:2,hfi1_2:2,hfi1_3:2 \
 -x NCCL_TOPO_FILE=<path-to-custom-topology-xml> \
 -x NCCL_DMABUF_ENABLE=1 \
 -x NCCL_IB_QPS_PER_CONNECTION=<n> \
 -x  NCCL_BUFFSIZE=<buffer-size-bytes>  \
 <path-to-nccl-tests-build>/all_reduce_perf -b 67108864 -e 1073741824 -f 2 -g 1
6.10.5.3.1. Key Performance Variables

  • -x NCCL_NET=IB: Forces NCCL to use its built-in Verbs-based IB plugin instead of an external plugin.

  • -x NCCL_IB_HCA=hfi1_0:2,...,hfi1_3:2: Lists all four CN5000 ports on the node so the built-in IB plugin can select and spread traffic across them, even though only one NIC was used for bootstrap (NCCL_SOCKET_IFNAME=ib0).

  • -x NCCL_TOPO_FILE=<path-to-custom-topology-xml>: Supplies a custom topology XML describing GPU, NIC, and PCIe/NVLink interconnect relationships; use this when the auto-detected topology does not accurately reflect the platform.

  • -x NCCL_DMABUF_ENABLE=1: Required for this plugin to register GPU memory via ibv_reg_mr over hfi1. Without it, registration fails with Bad address.

  • -x NCCL_IB_QPS_PER_CONNECTION=<n>: Sets the number of queue pairs per connection. Sweep this value (e.g., 1-4) to find the best-performing setting on the target platform.

  • -x NCCL_BUFFSIZE=<buffer-size-bytes>: Adjusts the internal NCCL communication buffer size. Sweep this value (e.g., 4 MiB-32 MiB) to find the best-performing setting on the target platform.

6.10.5.4. Verifying SuperNIC Utilization with opainfo Counters

When running NCCL across multiple SuperNICs per node, it is useful to confirm that traffic is actually distributed across all configured SuperNICs rather than concentrated on a subset. This can be done by capturing opainfo transmit/receive counters on each node immediately before and after the benchmark run, then computing the per-SuperNIC delta.

  1. Capture per-SuperNIC Xmit/Recv byte counters before the run.

    ssh <hostname> opainfo > before.out

  2. Run the NCCL benchmark.

  3. Capture the counters again after the run.

    ssh <hostname> opainfo > after.out

  4. Compare Xmit Data/Recv Data fields per hfi1_N port between before.out and after.out to compute the delta (MB) transferred by each SuperNIC.

A roughly even delta across all SuperNICs on a node indicates that NCCL (via NCCL_IB_HCA, NCCL_CROSS_NIC, and channel count tuning) is successfully load-balancing the collective traffic across the available SuperNICs. A skewed distribution suggests revisiting the HCA list, channel count, NCCL_ALGO/NCCL_PROTO choice, or the GPU/NIC affinity mapping.