6.3.6. HFI1 Driver Parameters
Options for the hfi1 driver should be set in /etc/modprobe.d/hfi1.conf to ensure persistence on reboot. Populate the file named /etc/modprobe.d/hfi1.conf with the following recommended module parameters for optimal performance:
options hfi1 num_sdma=8 num_vls=4
To apply the HFI1 driver parameter changes:
Rebuild initramfs
sudo dracut -f
Reboot the node
sudo reboot
6.3.6.1. Bulk Transfer Service Performance Optimization
A new software technology referred to as Bulk Transfer Service (BTS) greatly improves the bandwidth of CN5000, currently supported by MPI (through the OPX Provider) and Verbs. To enable this feature, set the use_bulksvc=Y module parameter and apply using the guidance above. Note there are other module parameters available for bulk service tuning which can be seen with modinfo hfi1 | grep bulksvc. As of now, no recommendations other than enabling bulk service are recommended.
No additional flags are required to use bulk service with Verbs. To use bulk service with MPI/OPX, you must set FI_OPX_HFISVC=1 at runtime.
When BTS is enabled, all kernel verbs transfers greater than 2 KB message size will use BTS. User space verbs, such as perftest using the Cornelis-provided libibverbs, will use BTS for all message sizes. For the OPX libfabric provider, it is user-configurable using the eager-to-rendezvous threshold user environment variable FI_OPX_RZV_MIN_PAYLOAD_BYTES=16385 (default).
6.3.6.2. Modifications for High Core Count Systems
CN5000 currently supports a maximum of 208 MPI ranks per node. By default, the driver will load with only 90 user contexts. To use more than 90 MPI ranks per node, you have to adjust the num_user_contexts module parameter. Typically you should set this to the number of physical CPU cores on the system. Port 2 is the active port on CN5000 adapters.
To check the number of free contexts:
grep . /sys/class/infiniband/hfi1_0/ports/2/num_freectxts
Example output:
/sys/class/infiniband/hfi1_0/ports/2/num_freectxts:90
If the number is too low, reload the HFI1 driver with the appropriate context setting. You can edit /etc/modprobe.d/hfi1.conf as discussed previously, or you can load in real time:
modprobe -r hfi1 modprobe hfi1 num_user_contexts=0,128
Note the syntax is num_user_contexts=<numPort1>,<numPort2>, where numPort1 can be set to 0 for CN5000.
If there is more than one CN5000 SuperNIC installed in the system, the values need to be repeated. For example, a dual card system needs num_user_contexts=0,128,0,128.