As of OPX Software 12.4, the hfi1 driver can be loaded with module parameter ipoib_tx_mode=hfisvc, which significantly improves bandwidth when using IPoIB on CN5000.

IMPORTANT: Additional steps must be taken by the system administrator to ensure the ib_ipoib module is loaded after hfi1. If ib_ipoib is loaded before hfi1 during boot, there may be undesirable messages such as:

>dmesg | grep 'DMS not available'
DMS not available

Or, only one transmit queue is available:

ls /sys/class/net/<dev>/queues | grep -c ^tx-

To prevent this, Cornelis recommends to blacklist the ib_ipoib driver:

echo 'blacklist ib_ipoib' > /etc/modprobe.d/blacklist-ipoib.conf
dracut -f
reboot

After the system boots, modprobe ib_ipoib, or incorporate this logic into the cluster job scheduler or other automation frameworks.