6.10.6. RCCL Benchmarks
6.10.6.1. Running RCCL Benchmarks with Open MPI
AMD ROCm Communication Collectives Library (RCCL) is a port of NCCL designed for AMD GPUs, pro-viding optimized collective and point-to-point (P2P) communication primitives over interconnects such as PCIe, Infinity Fabric (XGMI), and RDMA-capable NICs. RCCL implements the same API as NCCL, so tools such as rccl-tests and most of the NCCL environment variables described in NCCL Benchmarks (GitHub) apply directly to RCCL.
For more information, visit the official RCCL page: https://github.com/ROCm/rocm-systems/tree/develop/projects/
Benchmarks are built from rccl-tests, which produces per-collective binaries (all_reduce_perf, all_gather_perf, broadcast_perf, reduce_scatter_perf, sendrecv_perf, etc.) analogous to the NCCL test suite.
Two networking approaches are shown in the following sections: the aws-ofi-rccl plugin (NCCL_NET unset, defaulting to ofi) and RCCL’s built-in Verbs/IB plugin (NCCL_NET=IB). Both approaches should be evaluated on a given deployment, since results can shift with topology, SuperNIC count, and driver/plugin versions.
6.10.6.2. Approach 1: aws-ofi-rccl Plugin
6.10.6.2.1. Example: RCCL all_reduce_perf with Open MPI and OPX across two nodes, one process per GPU
Compile and install the AWS OFI libfabric plugin found on aws-ofi-nccl (GitHub). The instructions below assume the plugin is installed in <path-to-aws-ofi-rccl>.
The following is a two-node, eight-GPU-per-node run against two MI300X (mi300x-001, mi300x-002) hosts, using the aws-ofi-rccl plugin. Each of the eight ranks per node uses its own nearest CN5000 SuperNIC (by PCIe/NUMA locality); all eight SuperNICs on the node are active.
mpirun -np 16 -host mi300x-001:8,mi300x-002:8 \ --map-by ppr:1:numa:PE=4:ordered \ -mca mtl ofi -mca btl self,sm -mca pml cm \ -x FI_OPX_HFISVC=1 -x FI_HMEM_ROCR_USE_DMABUF=1 \ -x HIP_VISIBLE_DEVICES=2,3,1,0,6,7,5,4 \ -x LD_LIBRARY_PATH=/opt/rocm/lib:<path-to-aws-ofi-rccl>/lib:<path-to-openmpi>/lib: \ -x NCCL_SOCKET_IFNAME=ib0 \ -x NCCL_CROSS_NIC=1 \ <path-to-rccl-tests-build>/all_reduce_perf -b 67108864 -e 1073741824 -f 2 -g 1
Replace the placeholders (<...>) with the paths appropriate for your system. Consider adding the additional tunables below (NCCL_ALGO/NCCL_PROTO, and channel counts to further optimize multi-SuperNIC scaling.
6.10.6.2.2. Key Performance Variables
-x HIP_VISIBLE_DEVICES=2,3,1,0,6,7,5,4: Reorders the logical GPU-to-rank mapping so that each GPU’s nearest SuperNIC (by PCIe/NUMA locality) and its XGMI (Infinity Fabric) neighbors are used efficiently. The correct order is platform-specific and should be derived from the system’s PCIe/XGMI topology (seerocm-smi --showtopoor the vendor topology tool).-x FI_OPX_HFISVC=1: Enables the OPX HFI Service layer required for the Libfabric OPX Provider to correctly initialize on CN5000.-x FI_HMEM_ROCR_USE_DMABUF=1: Enables dma-buf on CN5000.-x NCCL_SOCKET_IFNAME=ib0: Restricts the out-of-band RCCL/NCCL socket bootstrap to the desired network interface (e.g., the IPoIB management interface) to avoid inadvertently bootstrapping over an unintended SuperNIC.-x NCCL_CROSS_NIC=1: Allows a single collective operation ring/tree to cross between different SuperNICs.--map-by ppr:1:numa:PE=4: Pins each of the four ranks per node to a unique NUMA node, allowing four cores to be used per rank. The resulting bindings with--report-bindings. A PE value of 2, 4, 8, and so on can be studied.
Additional tunables to evaluate, especially with multiple SuperNICs per node:
-x NCCL_ALGO=Tree -x NCCL_PROTO=Simple: Selects the Tree algorithm with the Simple protocol; on multi-node AMD GPU clusters this combination is often a good starting point, but Ring and LL/LL128 variants should also be evaluated for the target message-size range.-x NCCL_MIN_NCHANNELS=<n> -x NCCL_MAX_NCHANNELS=<n>: Fixes the number of communication channels; tune upward to better utilize multiple SuperNICs per node, generally starting near the SuperNIC count.-x NCCL_P2P_LEVEL=sys: Allows peer-to-peer transfers across the full system hierarchy (e.g., through XGMI) rather than restricting P2P to GPUs on the same PCIe switch.-x NCCL_NET_GDR_LEVEL=sys: Enables GPUDirect RDMA across the full system hierarchy, consistent withNCCL_P2P_LEVEL=sys.
6.10.6.2.3. GPU, SuperNIC, and Rank Binding
As with NCCL, RCCL performance depends heavily on placing each MPI rank on the CPU core(s) lo-cal to its assigned GPU and SuperNIC. On systems with one GPU and one SuperNIC per NUMA, --map-by ppr:1:numa works well. Another option is to use --map-by pe-list=<core-list>:ordered (or an equivalent Open-MPI/MPI binding option) to pin ranks to the correct NUMA node, and verify the resulting bindings with --report-bindings. Combine this with a HIP_VISIBLE_DEVICES ordering and, where necessary, a custom NCCL_TOPO_FILE, so that each rank’s GPU, SuperNIC, and CPU core all reside on the same NUMA node/socket.
Note
It is advised to configure GPU nodes such that each NUMA node has a GPU and a SuperNIC. With careful pinning as shown here, all communications from a GPU use the NUMA-local dedicated SuperNIC.
6.10.6.2.4. Example Results
Running the command above (all_reduce_perf, 16 MI300X GPUs across two nodes, all eight CN5000 SuperNICs per node active) produced the following out-of-place bus bandwidth results, with a peak of approximately 260 GBps:
Size (B) | Count (Elements) | Time (μs) | AlgBW (GBps) | BusBW (GBps) |
|---|---|---|---|---|
67,108,864 | 16,777,216 | 1573.94 | 42.64 | 79.95 |
134,217,728 | 33,554,432 | 1977.28 | 67.88 | 127.27 |
268,435,456 | 67,108,864 | 2414.88 | 111.16 | 208.42 |
536,870,912 | 134,217,728 | 3747.67 | 143.25 | 268.60 |
1,073,741,824 | 268,435,456 | 7764.00 | 138.30 | 259.31 |
6.10.6.3. Approach 2: Built-in Verbs/IB Plugin (NCCL_NETIB)
Rather than the aws-ofi-rccl plugin, RCCL can use its built-in Verbs-based IB plugin directly by setting NCCL_NET=IB. In this mode, RCCL uses the CN5000 SuperNICs with standard verbs instead of Libfabric, and is pointed at all SuperNICs on a node via NCCL_IB_HCA. NCCL_DMABUF_ENABLE=1 is required in this mode. Without it, ibv_reg_mr of GPU memory over hfi1 fails with Bad address (NCCL_NET_GDR_LEVEL=LOC also works, but is slower). A NCCL_TOPO_FILE is also required to disambiguate the MI300X GPUs, which report a shared/unique GUID that confuses topology auto-detection (an upstream RCCL PR exists for this issue: ROCm Issue #11138).
Two additional tunables were found to meaningfully improve this path: NCCL_IB_QPS_PER_CONNECTION=2 (two queue pairs per connection; three or four QPs measured worse) and NCCL_BUFFSIZE=33554432 (32 MiB).
mpirun -np 16 -host mi300x-001:8,mi300x-002:8 \ --map-by ppr:1:numa:PE=4:ordered \ -mca mtl ofi -mca btl self,sm -mca pml cm \ -x HIP_VISIBLE_DEVICES=2,3,1,0,6,7,5,4 \ -x LD_LIBRARY_PATH=<path-to-openmpi>/lib:/opt/rocm/lib: \ -x NCCL_SOCKET_IFNAME=ib0 \ -x NCCL_CROSS_NIC=1 \ -x NCCL_NET=IB \ -x NCCL_IB_HCA=hfi1_0:2,hfi1_1:2,hfi1_2:2,hfi1_3:2,hfi1_4:2,hfi1_5:2,hfi1_6:2,hfi1_7:2 \ -x NCCL_TOPO_FILE=<path-to-custom-topology-xml> \ -x NCCL_DMABUF_ENABLE=1 \ -x NCCL_IB_QPS_PER_CONNECTION=2 \ -x NCCL_BUFFSIZE=33554432 \ <path-to-rccl-tests-build>/all_reduce_perf -b 67108864 -e 1073741824 -f 2 -g 1
6.10.6.3.1. Key Performance Variables
-x NCCL_NET=IB: Forces RCCL to use its built-in Verbs-based IB plugin instead of an external plugin.-x NCCL_IB_HCA=hfi1_0:2,...,hfi1_7:2: Lists all eight CN5000 ports on the node so the built-in IB plugin can select and spread traffic across them, even though only one SuperNIC was used for bootstrap (NCCL_SOCKET_IFNAME=ib0).-x NCCL_TOPO_FILE=<path-to-custom-topology-xml>: Supplies a custom topology XML describing GPU, SuperNIC, and XGMI/PCIe interconnect relationships; required here because the MI300X GPUs report a shared/unique GUID that the auto-detected topology does not disambiguate correctly.-x NCCL_DMABUF_ENABLE=1: Required for this plugin to register GPU memory viaibv_reg_mrover hfi1; without it, registration fails withBad address.-x NCCL_IB_QPS_PER_CONNECTION=2: Uses two queue pairs per connection, the best-measured value on this platform (measured ~1.5x over baseline at 64 MiB, scaling up through 1 GiB); three or four QPs measured worse.-x NCCL_BUFFSIZE=33554432: Increases the internal RCCL communication buffer to 32 MiB, improving bandwidth further on top ofNCCL_IB_QPS_PER_CONNECTION=2, especially at smaller message sizes.
6.10.6.3.2. Example Results
Running the command above (all_reduce_perf, 16 MI300X GPUs across two nodes, NCCL_NET=IB with NCCL_IB_QPS_PER_CONNECTION=2 and NCCL_BUFFSIZE=32MiB) produced the following out-of-place bus bandwidth results, comparable to the libfabric performance:
Size (B) | Count (elements) | Time (μs) | AlgBW (GBps) | BusBW (GBps) |
|---|---|---|---|---|
67,108,864 | 16,777,216 | 2343.89 | 28.63 | 53.68 |
134,217,728 | 33,554,432 | 2678.87 | 50.10 | 93.94 |
268,435,456 | 67,108,864 | 3888.50 | 69.03 | 129.44 |
536,870,912 | 134,217,728 | 5240.87 | 102.44 | 192.07 |
1,073,741,824 | 268,435,456 | 7978.74 | 134.58 | 252.33 |
6.10.6.4. Verifying SuperNIC Utilization with opainfo Counters
When running RCCL across multiple SuperNICs per node, it is useful to confirm that traffic is actually distributed across all configured SuperNICs rather than concentrated on a subset. This can be done by capturing opainfo transmit/receive counters on each node immediately before and after the benchmark run, then computing the per-SuperNIC delta.
Capture per-SuperNIC
Xmit/Recvbyte counters before the run.ssh <hostname> opainfo > before.out
Run the RCCL benchmark.
Capture the counters again after the run.
ssh <hostname> opainfo > after.out
Compare
Xmit Data/Recv Datafields perhfi1_Nport betweenbefore.outandafter.outto compute the delta (MB) transferred by each SuperNIC.
A roughly even delta across all SuperNICs on a node indicates that RCCL (via NCCL_IB_HCA, NCCL_CROSS_NIC, and channel count tuning) is successfully load-balancing the collective traffic across the available SuperNICs. A skewed distribution suggests revisiting the HCA list, channel count, NCCL_ALGO/NCCL_PROTO choice, or the GPU/NIC affinity mapping.