FastFabric and TCP/IP must resolve host names to the management network and IPoIB IP addresses. If the management network is not IPoIB, each host must have both a management network name and an IPoIB network name. To do this, use the actual host name as the management network name and HOSTNAME-opa as the IPoIB network name, where HOSTNAME is the management network name of the given host.

Name resolution is accomplished by configuring a DNS server on the management network, with both management network and IPoIB addresses for each host and each chassis.

Alternatively, an /etc/hosts file needs to be created on the Management Node; FastFabric can then propagate this /etc/hosts file to all the other hosts.

If using the /etc/hosts file approach and NOT using Domain Name System (DNS):

If using DNS:

Cornelis recommends preconfiguring servers and settings to tune fabric performance to meet the needs of the system. These tasks can be performed before or after the installation. Refer to the CN5000 Performance Tuning Guide, which describes BIOS settings and parameters that have been shown to improve performance or make performance more consistent on the CN5000 Omni-Path Fabric.

The ROCm Collective Communications Library (RCCL) provides collective and point-to-point communication primitives optimized for multi-GPU, multi-node applications.

Installation instructions and downloads for RCCL can be found on GitHub at ROCm/rccl.

RCCL uses the Amazon AWS OFI NCCL plugin for libfabric support. The source files for the plugin are found on GitHub at aws/aws-ofi-nccl. Details for building the plugin using GNU Autotools are included in the INSTALL.md text file.

Cornelis recommends that you use the ./configure command of the form:

./configure --with-libfabric=</path to your gpu-enabled build of libfabric> --with-rocm=/opt/rocm --with-mpi=</path to your gpu-enabled mpi build> --prefix=</location where amazon-plugin will be installed>

cd </location where amazon-plugin will be installed>/lib
ln -s librccl-net.so libnccl-net.so

Use the following instructions to download and build the NVIDIA driver:

For instructions on installing CUDA, refer to the NVIDIA CUDA Installation Guide for Linux.

The following instructions use CUDA 12.8 on Rocky Linux 9.5. Replace the versions in the steps with the version you are installing.

Whereas NVIDIA’s GPUDirect RDMA allows direct communication between GPUs and other PCIe devices, GDRCopy allows local CPUs to directly map and access the GPU memory.

Installation instructions and download locations for GDRCopy are found on GitHub at NVIDIA/gdrcopy. Note that the NVIDIA components described in NVIDIA GPU Requirements must be installed before installing GDRCopy.

Cornelis recommends installing and using GDRCopy to achieve maximum GPU performance with the OPX Provider.

Once installed, GDRCopy functionality is disabled or enabled through the environment variable: FI_HMEM_CUDA_USE_GDRCOPY=0 or 1, respectively.

When used with an NVIDIA GPU, the hfi1 module must be loaded after the NVIDIA module. In some cases, the hfi1 may load earlier in the boot process than the NVIDIA module. If this occurs, the module loads successfully and is functional, but without NVIDIA support.

Use dmesg to determine if hfi1 has been loaded with NVIDIA support. If dmesg returns that NVIDIA support is enabled (as shown in the following example), then no other actions are required.

[root@cornelis ~]# dmesg | grep hfi1 | grep -i nvidia
    hfi1 Nvidia p2p DMA support enabled
    hfi1 Nvidia p2p TID-DMA support enabled
    hfi1 Nvidia GDRCopy support enabled

If dmesg responds that NVIDIA support is disabled (as shown in the following example), then the hfi1 module must be removed and reloaded.

[root@cornelis ~]# dmesg | grep hfi1 | grep -i nvidia
    hfi1 Nvidia p2p DMA support disabled (missing symbol nvidia_p2p_free_page_table)
    hfi1 Nvidia p2p TID-DMA support disabled
    hfi1 Nvidia GDRCopy support disabled

To remove and reload the hfi1 module, use the following command:

modprobe -r hfi1 && modprobe hfi1

Once this has been completed, verify that the hfi1 module was reloaded with NVIDIA support using the dmesg command as shown previously.

To ensure that the GPUDirect-capable software is installed, verify as follows.

For driver, run:

$ modinfo hfi1 | grep nvidia

Sample output:

softdep: pre: nvidia

The NVIDIA Collective Communications Library (NCCL) provides collective and point-to-point communication primitives optimized for multi-GPU, multi-node applications.

Installation instructions and downloads for NCCL can be found on GitHub at NVIDIA/nccl. Note that an NVIDIA user account may be required to download the installation files.

As NCCL does not natively support libfabric, you must install the Amazon AWS OFI NCCL plugin. The source files for the plugin are found on GitHub at aws/aws-ofi-nccl. Details for building the plugin using GNU Autotools are included in the INSTALL.md text file.

Cornelis recommends that you use the ./configure command of the form:

./configure --with-libfabric=</path to your gpu-enabled build of libfabric> --with-cuda=/usr/local/cuda --with-mpi=</path to your gpu-enabled mpi build> --prefix=</location where amazon-plugin will be installed>

Once the plugin has been installed, you can enable logging using the environment variable: NCCL_DEBUG=info. Once the variable is set, look for the following log messages: