A new software technology referred to as Bulk Transfer Service (BTS) greatly improves the bandwidth of CN5000, currently supported by MPI (through the OPX Provider) and Verbs. To enable this feature, set the use_bulksvc=Y module parameter and apply using the guidance above. Note there are other module parameters available for bulk service tuning which can be seen with modinfo hfi1 | grep bulksvc. As of now, no recommendations other than enabling bulk service are recommended.

No additional flags are required to use bulk service with Verbs. To use bulk service with MPI/OPX, you must set FI_OPX_HFISVC=1 at runtime.

When BTS is enabled, all kernel verbs transfers greater than 2 KB message size will use BTS. User space verbs, such as perftest using the Cornelis-provided libibverbs, will use BTS for all message sizes. For the OPX libfabric provider, it is user-configurable using the eager-to-rendezvous threshold user environment variable FI_OPX_RZV_MIN_PAYLOAD_BYTES=16385 (default).

CN5000 has approximately 208 hardware contexts available for use. This means when running MPI jobs with more than 208 MPI ranks per node, context sharing must be enabled using FI_OPX_CONTEXT_SHARING and FI_OPX_ENDPOINTS_PER_HFI_CONTEXT. Refer to MPI Libraries for details.

By default, the driver will load with only 90 user contexts. The num_user_contexts module parameter should be set to the number of physical CPU cores on the system, or the number of physical CPU cores on the local socket in the case of multiple CN5000 SuperNICs per node. Port 2 is the active port on CN5000 SuperNICs.

To check the number of free contexts:

grep . /sys/class/infiniband/hfi1_0/ports/2/num_freectxts

Example output:

/sys/class/infiniband/hfi1_0/ports/2/num_freectxts:90

If the number is too low, reload the HFI1 driver with the appropriate context setting. You can edit /etc/modprobe.d/hfi1.conf as discussed previously, or you can load in real time:

modprobe -r hfi1
modprobe hfi1 num_user_contexts=0,128

Note the syntax is num_user_contexts=<numPort1>,<numPort2>, where numPort1 can be set to 0 for CN5000.

If there is more than one CN5000 SuperNIC installed in the system, the values need to be repeated. For example, a dual card system needs num_user_contexts=0,128,0,128.

A typical use case for high core count systems (dual socket, 128 core per socket):

This is because each OPX process will only use the socket-local SuperNIC.