STL-79912

NCCL collectives may timeout or hang when running with non-NVLink GPU pairs on the same node.

Set NCCL_CUMEM_ENABLE=0.

STL-79864

When MPS is set to 256 before loading the hfi1 kernel module, a kernel halt condition may occur.

Set MPS=256 after hfi1 is loaded.

Refer to the CN5000 Performance Tuning Guide for more details.

STL-79845

When ipoib_tx_mode=hfisvc is set, transmit throughput remains low.

IPoIB HFI Service (HFISVC) transmit mode requires that hfi1 be loaded before ib_ipoib.

Refer to the CN5000 Performance Tuning Guide for the required load order and how to verify it.

STL-79799

Unexpected system fault and reboot may occur while running traffic with ipoib_tx_mode=hfisvc and use_bulksvc=N.

Operating with ipoib_tx_mode=hfisvc and use_bulksvc=N is an unsupported configuration.

Refer to the CN5000 Performance Tuning Guide or contact Cornelis Technical Support for help with configurations.

STL-79771

Installing the Ubuntu distribution perftest package attempts to uninstall OPX Software due to package conflicts.

Build and run perftest from source using the upstream repository: Perftest (GitHub).

STL-79587

In distro Truescale drivers (ib_qib.ko) are incompatible with modern rdmavt kernel modules.

Blacklist ib_qib or remove the .ko file to prevent error messages. Truescale is no longer compatible with the shipping OPX Software 12.x releases.

STL-79559

When running RCCL over its built-in verbs transport (NCCL_NET=IB), RCCL uses only two SuperNICs per node for inter-node communication, regardless of how many are installed. Additional SuperNICs are initialized but transmit no traffic, limiting collective communication bandwidth.

NOTE: This is a RCCL defect (ROCm/rocm-systems#11138).

Perform one of the following:

STL-79524

In certain Linux distributions, the default DHCP service does not support InfiniBand or IP over InfiniBand (IPoIB) interfaces.

Contact your Linux distribution vendor for approved alternative DHCP packages or configuration workarounds for IPoIB interfaces.

STL-79445

Five-minute timeouts may occur in BeeGFS when RDMA connections are enabled (connUseRDMA = true).

NOTE: A patch is pending per Beegfs (GitHub).

Set connUseRDMA = false.

STL-79193

On SLES distros, the hfi1 driver version may appear incorrect.

To ensure the correct driver is running, use the srcversion field.

STL-78398

On nodes running Ubuntu 24.04 with OPX Software installed, the srp_daemon service continuously generates ib_srp errors in dmesg. These errors include repeated SRP LOGIN rejections, failed path record queries (status −22), and failed SCSI host connections.

The errors persist indefinitely and can consume excessive CPU resources until the srp_daemon process is stopped.

Stop the srp_daemon service or disable it from starting at boot.

NOTE: Disabling srp_daemon will prevent automatic discovery of SCSI RDMA Protocol (SRP) targets. If SRP storage is not in use on the affected nodes, this workaround has no functional impact.

STL-78397

On Ubuntu 24.04, the opafm service may intermittently fail to start at boot, reporting:

ibwarn: get_abi_version: can't read ABI version from /sys/class/infiniband_mad/abi_version
Invalid config detected! Couldn't find device for instance 0!

Couldn't find device for instance 0! This is caused by a boot-time race condition.

After a reboot, perform the following:

STL-78250

A Verbs SEND operation using PIO may fail to detect that send credits have been returned. This causes the send path to get stuck, preventing further send operations even though the hardware is ready.

Set piothreshold=0 in the HFI1 driver module parameters.

STL-78089

ROCm HIP pointer queries may cause a process hang during teardown. This is a known ROCm issue related to OpenMP target allocations recorded in https://github.com/ROCm/rocm-systems/issues/7099.

Use HIP-managed allocations (e.g., hipMalloc()) instead of OpenMP target allocations for buffers used with OPX GPU-direct/HMEM operations. Avoid mixing OpenMP target allocations with HIP pointer introspection until the ROCm issue is resolved upstream.

STL-77891

When installing or uninstalling the opxs-kernel-updates-kmp-default package on SLES 16, warnings appear stating that the RPM's post-install or post-uninstall scriptlet failed.

These warnings can be ignored.

STL-77578

CableInfo errors encountered when using the opasmaquery -o cableinfo command with direct-routed packets.

Use LID-routed packets to get cable information.

STL-77479

Running opatop may report over line rate.

None.

STL-77171

On SLES, building the kernel driver RPMs may fail when multiple kernels are installed.

STL-76907

The following Lustre Error may occur:

Write request failed with -14

NOTE: This is a general defect that affects all Lustre 2.17.0 stacks (refer to Lustre-Release Issue #16

Avoid this issue by disabling hybrid IO functionality using the following command on the MGS server:

lctl set_param -P llite.*.hybrid_io=0

STL-76435

opapmaquery fails with the following error when querying performance counters for more than approximately 14 ports simultaneously on a single switch.

Invalid Attribute or Attribute Modifier

NOTE: The PMA response has a maximum payload size that limits the number of ports that can be returned in a single query.

Split the port select mask (-n option) into multiple smaller queries, each covering a subset of ports.

For example, on a 48-port switch, query ports in groups of 20 or fewer:

opapmaquery -o getdatacounters -l <LID> -n 0x0000000FFFFF # ports 0-19

opapmaquery -o getdatacounters -l <LID> -n 0x00FFFFF00000 # ports 20-39

opapmaquery -o getdatacounters -l <LID> -n 0xFF0000000000 # ports 40-47

STL-76348

Running opapaquery -o focusport reports same neighbor LID if the switch is connected to a dual-port SuperNIC.

None.

STL-76323

SuperNICs may intermittently fail to enumerate on the PCIe bus or initialize after a cold boot or power cycle. In affected systems, one or more SuperNICs may be absent from the OS, preventing the node from participating in the fabric until the device is successfully detected and initialized.

Power cycle the server.

STL-76313

Kernel messages such as the following may be seen:

CPORT request wait interrupted

Dropping incoming CPORT message

Set the hfi1 module parameter cport_mad_to=5.

STL-75568

Performance test failures may occur when running ib_send_bw and ib_read_bw with the -D (duration) option.

Do not use the -D option.

STL-75484

Running opaportconfig with the disable option immediately followed by running opaportconfig with the enable option will result in the link staying down.

To drop and reinitialize a link, use the bounce option to opaportconfig.

Alternately, if opaportconfig is run from a script, insert a one second delay (sleep 1) between the disable and enable commands.

STL-74968

Updating the Fabric Manager using DNF may not succeed even though an update is available.

STL-74477

Building opxs-kernel-updates from the src.rpm file fails with nvidia/nv-p2p.h not detected.

Use an environment variable to allow the header file to be detected and used in the build:

NVIDIA_DIR=$(ls -d /usr/src/nvidia-* | head -1) 
ln -s ${NVIDIA_DIR}/nvidia /usr/src/kernels/5.14.0-503.14.1.el9_5.x86_64/include/nvidia

Note that the exact directory required for the symbolic link will depend on your kernel version.

STL-74025

When using IPoIB, SuperNICs do not change MTU or join MC groups automatically.

Use the ip tool from the command line to manually take the interface down and back up.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73714

Performing an opapmaquery to a "B" port on a switch where subdivision is not enabled returns non-zero counter values.

Counter values in this scenario should be ignored.

STL-73713

In a fabric with no data traffic, the Xmit Time Congestion port counter will increment.

No workaround.

STL-73669

Executing opasmaquery -o scvlr for nodes that do not support SC-VLr mappings will result in a message indicating a failure.

The message may be ignored.

STL-73215

When running opasmaquery -o sminfo on CN5000 SuperNICs, SMA fields show all 0s and SM State Inactive.

Use opasaquery -o sminfo or opareport to show correct SA SMInfo data.

STL-73160

The comments in the Adaptive Routing section of the opafm.xml file do not explain which settings are required for MGAR or for FGAR.

Currently the only field related to FGAR is FineGrained. In order to enable FGAR, set the following:

<AdaptiveRouting>1</AdaptiveRouting> 

<FineGrained>1</FineGrained> 

All other fields are related to legacy MGAR and will be ignored if FineGrained is enabled.

STL-73140

Note the following:

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-72541

Running the opainfo command when ports are inactive results in the following errors: MAD failed and Failed to get PMA Port Status.

These errors can be ignored.

STL-72536

STL-72537

CableInfo errors encountered when using the opasmaquery -o cableinfo command.

Cable information reporting not currently supported.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

For details, contact support@cornelis.com for technical advisory TA0061.

STL-72420

On some versions of RHEL 9.x, there is a known failure (rdmavt driver deadlock.) recorded in https://issues.redhat.com/browse/RHEL-83851.

NOTE: Cornelis has not replicated this problem, but it is possible that customers may be affected.

No workaround.

STL-72160

After uninstalling the meta packages with a package manager, several Cornelis Networks packages remain.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Remove OPX Software from a Host” for removal of the remaining packages.

STL-72028

Hangs have been seen in the NCCL alltoall_perf test when using the OPX provider to libfabric.

No workaround.

STL-71364

STL-72423

Some legacy Omni-Path host utilities have been deprecated for CN5000 but are still supported for OPA100.   

For a list of commands supported for CN5000, refer to the Man Pages or CN5000 Commands Guide.

STL-70552

Rebuilding Open MPI or mvapich2 with FastFabric utilities (do_openmpi_build and do_mvapich2_build) will return an error if using the Intel icx compiler.

These utilities are compatible only with the Intel ICC compiler. Rebuild Open MPI or mvapich2 using the ICC compiler, or rebuild manually.

STL-67178

After boot, the OPX Software hfi1-NVIDIA support is not available.

Review boot configuration to ensure that nvidia-driver will be available when hfi1 is loaded.

STL-64974

Failures have been observed when running the IMB-EXT "Window" benchmark using the OPX Provider and ppn greater than (num_user_contexts/2).

To run the IMB-EXT benchmark with the OPX Provider and higher PPN, reload the hfi1 module with an increased num_user_contexts value. For example, if the server has 40 physical cores and you want to run one IMB-EXT process per core, increase the number of hfi1 contexts to 80. See the CN5000 Performance Tuning Guide, “HFI1 Driver Module Parameters” for instructions.

The benchmark can now be run with #contexts/2 = 40 PPN, one for each CPU core.

STL-79445

When the hfi1 driver is loaded with the Bulk Transfer Service (BTS) module parameter enabled (use_bulksvc=Y), BeeGFS file systems may report the following communication error:

Communication error in SENDDATA stage

Perform one of the following:

STL-79194

Transfers using GPU (ROCm) buffers over the OPX Provider may abort with SDMA Abort errno=22 (Invalid argument) and dmesg entry amdgpu: Error in Pinning BO to domain: 4.

Set the following environment variables to prevent the failure path:

STL-79193

On SLES distros, the hfi1 driver version may appear incorrect.

To ensure the correct driver is running, use the srcversion field.

STL-78983

Running certain verbs tests, such as nsdperf with verbs MR sizes > 4 GB, may result in errors.

Restrict memory buffer sizes to less than 4GB.

STL-78397

On CN5000 nodes running Ubuntu 24.04 with OPX Software installed, the srp_daemon service continuously generates ib_srp errors in dmesg. These errors include repeated SRP LOGIN rejections, failed path record queries (status −22), and failed SCSI host connections.

The errors persist indefinitely and can consume excessive CPU resources until the srp_daemon process is stopped.

Stop the srp_daemon service or disable it from starting at boot.

NOTE: Disabling srp_daemon will prevent automatic discovery of SCSI RDMA Protocol (SRP) targets. If SRP storage is not in use on the affected nodes, this workaround has no functional impact.

STL-78250

A Verbs SEND operation using PIO may fail to detect that send credits have been returned. This causes the send path to get stuck, preventing further send operations even though the hardware is ready.

Set piothreshold=0 in the HFI1 driver module parameters.

STL-78089

ROCm HIP pointer queries may cause a process hang during teardown. This is a known ROCm issue related to OpenMP target allocations recorded in https://github.com/ROCm/rocm-systems/issues/7099.

Use HIP-managed allocations (e.g., hipMalloc()) instead of OpenMP target allocations for buffers used with OPX GPU-direct/HMEM operations. Avoid mixing OpenMP target allocations with HIP pointer introspection until the ROCm issue is resolved upstream.

STL-77891

When installing or uninstalling the opxs-kernel-updates-kmp-default package on SLES 16, warnings appear stating that the RPM's post-install or post-uninstall scriptlet failed.

These warnings can be ignored.

STL-77873

Megatron-LM training (Llama3-8B) hangs on multi-node configurations when using NCCL_NET=OFI with the OPX Provider.

Use NCCL_NET=IB instead of NCCL_NET=OFI.

STL-77578

CableInfo errors encountered when using the opasmaquery -o cableinfo command with direct-routed packets.

Use LID-routed packets to get cable information.

STL-77479

Running opatop may report over line rate.

None.

STL-77171

On SLES, building the kernel driver RPMs may fail when multiple kernels are installed.

STL-76907

The following Lustre Error may occur:

Write request failed with -14

NOTE: This is a general defect that affects all Lustre 2.17.0 stacks (refer to Lustre-Release Issue #16

Avoid this issue by disabling hybrid IO functionality using the following command on the MGS server:

lctl set_param -P llite.*.hybrid_io=0

STL-76435

opapmaquery fails with the following error when querying performance counters for more than approximately 14 ports simultaneously on a single switch.

Invalid Attribute or Attribute Modifier

NOTE: The PMA response has a maximum payload size that limits the number of ports that can be returned in a single query.

Split the port select mask (-n option) into multiple smaller queries, each covering a subset of ports.

For example, on a 48-port switch, query ports in groups of 20 or fewer:

opapmaquery -o getdatacounters -l <LID> -n 0x0000000FFFFF # ports 0-19

opapmaquery -o getdatacounters -l <LID> -n 0x00FFFFF00000 # ports 20-39

opapmaquery -o getdatacounters -l <LID> -n 0xFF0000000000 # ports 40-47

STL-76348

Running opapaquery -o focusport reports same neighbor LID if the switch is connected to a dual-port SuperNIC.

None.

STL-76323

SuperNICs may intermittently fail to enumerate on the PCIe bus or initialize after a cold boot or power cycle. In affected systems, one or more SuperNICs may be absent from the OS, preventing the node from participating in the fabric until the device is successfully detected and initialized.

Power cycle the server.

STL-76313

Kernel messages such as the following may be seen:

CPORT request wait interrupted

Dropping incoming CPORT message

Set the hfi1 module parameter cport_mad_to=5.

STL-76144

Setting the link speed policy to 100G in the opafm.xml file results in the following Fabric Manager error:

Invalid Value for Link Speed Policy: 100G

Set the link speed using the Switch CLI command port modify speed.

STL-75568

Performance test failures may occur when running ib_send_bw and ib_read_bw with the -D (duration) option.

Do not use the -D option.

STL-75526

After installing the OPX Software package on Ubuntu, the opafm service is enabled by default.

Manually disable the opafm service by default.

STL-75484

Running opaportconfig with the disable option immediately followed by running opaportconfig with the enable option will result in the link staying down.

To drop and reinitialize a link, use the bounce option to opaportconfig.

Alternately, if opaportconfig is run from a script, insert a one second delay (sleep 1) between the disable and enable commands.

STL-75395

When AdaptiveRouting is enabled in the opafm.xml file, but FineGrained is not, the Fabric Manager will crash.

Disable AdaptiveRouting in the opafm.xml file and restart the Fabric Manager.

STL-75362

When using the TCP provider of libfabric, some MPI applications may perform poorly or hang when launched with 1 process per core. This has been observed in benchmarks such as OSU Congestion, OSU Collectives, and IMB P2P.

For best results, users should use the OPX provider whenever available. OPX can be explicitly selected by setting the environment variable, FI_PROVIDER=opx.

STL-75322

Upon reverting from 12.0.3 to a previous RHEL/Rocky release certain packages may not be removed.

Use the following command to remove the remaining packages:

dnf remove ibacm infiniband-diags 
libfabric libhfi1verbs libibumad 
libibverbs libpsm2 librdmacm 
librdmacm-utils mpi-selector 
opa-basic-tools opa-fastfabric 
opa-fm opa-libopamgt rdma-core 
rdma-core-devel srp_daemon

STL-75031

MPI RMA benchmarks may produce an SDMA error (bad address) when using the OPX libfabric provider and environment variable FI_OPX_MIXED_NETWORK=0.

FI_OPX_MIXED_NETWORK is deprecated and should no longer be set. Contact support@cornelis.com.

STL-74968

Updating the Fabric Manager using DNF may not succeed even though an update is available.

STL-74933

Entries of the form hfi1_0: lost_mctxt_intr forcing int for 2 and MCTXT interrupt, but no status bits set may appear in logs.

These messages can be ignored.

STL-74477

Building opxs-kernel-updates from the src.rpm file fails with nvidia/nv-p2p.h not detected.

Use an environment variable to allow the header file to be detected and used in the build:

NVIDIA_DIR=$(ls -d /usr/src/nvidia-* | head -1) 
ln -s ${NVIDIA_DIR}/nvidia /usr/src/kernels/5.14.0-503.14.1.el9_5.x86_64/include/nvidia

Note that the exact directory required for the symbolic link will depend on your kernel version.

STL-74025

When using IPoIB on CN5000, SuperNICs do not change MTU or join MC groups automatically.

Use the ip tool from the command line to manually take the interface down and back up.

STL-73928

HFI links bounce and the SM reports mapping warnings following the second FM restart.

No Workaround. Links that bounce recover and the fabric stabilizes.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73716

The opatop and opapaquery commands may report congestion incorrectly when the fabric is idle.

Ignore the congestion flag.

STL-73714

Performing an opapmaquery to a "B" port on a switch where subdivision is not enabled returns non-zero counter values.

Counter values in this scenario should be ignored.

STL-73713

In a fabric with no data traffic, the Xmit Time Congestion port counter will increment.

No workaround.

STL-73669

Executing opasmaquery -o scvlr for nodes that do not support SC-VLr mappings will result in a message indicating a failure.

The message may be ignored.

STL-73215

When running opasmaquery -o sminfo on CN5000 SuperNICs, SMA fields show all 0s and SM State Inactive.

Use opasaquery -o sminfo or opareport to show correct SA SMInfo data.

STL-73160

The comments in the Adaptive Routing section of the opafm.xml file do not explain which settings are required for MGAR or for FGAR.

Currently the only field related to FGAR is FineGrained. In order to enable FGAR, set the following:

<AdaptiveRouting>1</AdaptiveRouting> 

<FineGrained>1</FineGrained> 

All other fields are related to legacy MGAR and will be ignored if FineGrained is enabled.

STL-73140

Note the following:

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-72573

Hangs and errors have been seen in the OSU alltoall collective test when using the OPX Provider to libfabric with ROCm.

No workaround.

STL-72541

Running the opainfo command when ports are inactive results in the following errors: MAD failed and Failed to get PMA Port Status.

These errors can be ignored.

STL-72536

STL-72537

CableInfo errors encountered when using the opasmaquery -o cableinfo command.

Cable information reporting not currently supported.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

For details, contact support@cornelis.com for technical advisory TA0061.

STL-72420

On some versions of RHEL 9.x, there is a known failure (rdmavt driver deadlock.) recorded in https://issues.redhat.com/browse/RHEL-83851.

NOTE: Cornelis has not replicated this problem, but it is possible that customers may be affected.

No workaround.

STL-72160

After uninstalling the meta packages with a package manager, several Cornelis Networks packages remain.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Remove OPX Software from a Host” for removal of the remaining packages.

STL-71364

STL-72423

Some legacy Omni-Path host utilities have been deprecated for CN5000 but are still supported for OPA100.   

For a list of commands supported for CN5000, refer to the Man Pages or CN5000 Commands Guide.

STL-67178

After boot, the OPX Software hfi1-NVIDIA support is not available.

Review boot configuration to ensure that nvidia-driver will be available when hfi1 is loaded.

STL-78250

A Verbs SEND operation using PIO may fail to detect that send credits have been returned. This causes the send path to get stuck, preventing further send operations even though the hardware is ready.

Set piothreshold=0 in the HFI1 driver module parameters.

STL-78089

ROCm HIP pointer queries may cause a process hang during teardown. This is a known ROCm issue related to OpenMP target allocations recorded in https://github.com/ROCm/rocm-systems/issues/7099.

Use HIP-managed allocations (e.g., hipMalloc()) instead of OpenMP target allocations for buffers used with OPX GPU-direct/HMEM operations. Avoid mixing OpenMP target allocations with HIP pointer introspection until the ROCm issue is resolved upstream.

STL-77956

GPU applications hang with Open MPI 5.0.8 and earlier

Use Open MPI 5.0.9 or newer.

STL-77891

When installing or uninstalling the opxs-kernel-updates-kmp-default package on SLES 16, warnings appear stating that the RPM's post-install or post-uninstall scriptlet failed.

These warnings can be ignored.

STL-77873

Megatron-LM training (Llama3-8B) hangs on multi-node configurations when using NCCL_NET=OFI with the OPX Provider.

Use NCCL_NET=IB instead of NCCL_NET=OFI.

STL-77578

CableInfo errors encountered when using the opasmaquery -o cableinfo command with direct-routed packets.

Use LID-routed packets to get cable information.

STL-76942

Lustre errors occur with Bulk Transfer Service (BTS) enabled (hfi1 driver parameter use_bulksvc=Y).

When mounting Lustre, BTS should not be enabled (use_bulksvc=N, which is default).

STL-76907

The following Lustre Error may occur:

Write request failed with -14

NOTE: This is a general defect that affects all Lustre 2.17.0 stacks (refer to Lustre-Release Issue #16

Avoid this issue by disabling hybrid IO functionality using the following command on the MGS server:

lctl set_param -P llite.*.hybrid_io=0

STL-76435

opapmaquery fails with the following error when querying performance counters for more than approximately 14 ports simultaneously on a single switch.

Invalid Attribute or Attribute Modifier

NOTE: The PMA response has a maximum payload size that limits the number of ports that can be returned in a single query.

Split the port select mask (-n option) into multiple smaller queries, each covering a subset of ports.

For example, on a 48-port switch, query ports in groups of 20 or fewer:

opapmaquery -o getdatacounters -l <LID> -n 0x0000000FFFFF # ports 0-19

opapmaquery -o getdatacounters -l <LID> -n 0x00FFFFF00000 # ports 20-39

opapmaquery -o getdatacounters -l <LID> -n 0xFF0000000000 # ports 40-47

STL-76348

Running opapaquery -o focusport reports same neighbor LID if the switch is connected to a dual-port SuperNIC.

None.

STL-76323

SuperNICs may intermittently fail to enumerate on the PCIe bus or initialize after a cold boot or power cycle. In affected systems, one or more SuperNICs may be absent from the OS, preventing the node from participating in the fabric until the device is successfully detected and initialized.

Power cycle the server.

STL-76313

Kernel messages such as the following may be seen:

CPORT request wait interrupted

Dropping incoming CPORT message

Set the hfi1 module parameter cport_mad_to=5.

STL-76144

Setting the link speed policy to 100G in the opafm.xml file results in the following Fabric Manager error:

Invalid Value for Link Speed Policy: 100G

Set the link speed using the Switch CLI command port modify speed.

STL-76131

When using SR-IOV, the dmesg log in the VMs may occasionally show the following warning message:

memcpy: detected field-spanning write

Do not run Omni-Path management tools, such as opafabricinfo, on virtual machines. These tools should only be run on the host.

STL-75568

Performance test failures may occur when running ib_send_bw and ib_read_bw with the -D (duration) option.

Do not use the -D option.

STL-75535

GADGET4 MPI application hangs with OPX Provider.

Contact support@cornelis.com.

STL-75526

After installing the OPX Software package on Ubuntu, the opafm service is enabled by default.

Manually disable the opafm service by default.

STL-75484

Running opaportconfig with the disable option immediately followed by running opaportconfig with the enable option will result in the link staying down.

To drop and reinitialize a link, use the bounce option to opaportconfig.

Alternately, if opaportconfig is run from a script, insert a one second delay (sleep 1) between the disable and enable commands.

STL-75395

When AdaptiveRouting is enabled in the opafm.xml file, but FineGrained is not, the Fabric Manager will crash.

Disable AdaptiveRouting in the opafm.xml file and restart the Fabric Manager.

STL-75362

When using the TCP provider of libfabric, some MPI application may perform poorly or hang when launched with 1 process per core. This has been observed in benchmarks such as OSU Congestion, OSU Collectives, and IMB P2P.

For best results, users should use the OPX provider whenever available. OPX can be explicitly selected by setting the environment variable, FI_PROVIDER=opx.

STL-75322

Upon reverting from 12.0.3 to a previous RHEL/Rocky release certain packages may not be removed.

Use the following command to remove the remaining packages:

dnf remove ibacm infiniband-diags 
libfabric libhfi1verbs libibumad 
libibverbs libpsm2 librdmacm 
librdmacm-utils mpi-selector 
opa-basic-tools opa-fastfabric 
opa-fm opa-libopamgt rdma-core 
rdma-core-devel srp_daemon

STL-75031

STL-76644

MPI RMA benchmarks may produce an SDMA error (bad address) when using the OPX libfabric provider and environment variable FI_OPX_MIXED_NETWORK=0.

FI_OPX_MIXED_NETWORK is deprecated and should no longer be set. Contact support@cornelis.com.

STL-74968

Updating the Fabric Manager using DNF may not succeed even though an update is available.

STL-74933

Entries of the form hfi1_0: lost_mctxt_intr forcing int for 2 and MCTXT interrupt, but no status bits set may appear in logs.

These messages can be ignored.

STL-74679

While running IPoIB in Connected mode, application hangs have been observed.

Configure IPoIB to Datagram mode and rerun the application.

STL-74625

When using both ports of a dual-port SuperNIC, the ibstat rate field may show an incorrect value.

Use opainfo, which is the preferred method for querying port information.

STL-74477

Building opxs-kernel-updates from the src.rpm file fails with nvidia/nv-p2p.h not detected.

Use an environment variable to allow the header file to be detected and used in the build:

NVIDIA_DIR=$(ls -d /usr/src/nvidia-* | head -1) 
ln -s ${NVIDIA_DIR}/nvidia /usr/src/kernels/5.14.0-503.14.1.el9_5.x86_64/include/nvidia

Note that the exact directory required for the symbolic link will depend on your kernel version.

STL-74025

When using IPoIB on CN5000, SuperNICs do not change MTU or join MC groups automatically.

Use the ip tool from the command line to manually take the interface down and back up.

STL-73928

HFI links bounce and the SM reports mapping warnings following the second FM restart.

No Workaround. Links that bounce recover and the fabric stabilizes.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73716

The opatop and opapaquery commands may report congestion incorrectly when the fabric is idle.

Ignore the congestion flag.

STL-73714

Performing an opapmaquery to a "B" port on a switch where subdivision is not enabled returns non-zero counter values.

Counter values in this scenario should be ignored.

STL-73713

In a fabric with no data traffic, the Xmit Time Congestion port counter will increment.

No workaround.

STL-73669

Executing opasmaquery -o scvlr for nodes that do not support SC-VLr mappings will result in a message indicating a failure.

The message may be ignored.

STL-73531

Configuring more than two data VLs is not currently supported. Doing so may lead to incorrect mappings of SC to VL.

Limit the number of configured data VLs to two.

STL-73215

When running opasmaquery -o sminfo on CN5000 SuperNICs, SMA fields show all 0s and SM State Inactive.

Use opasaquery -o sminfo or opareport to show correct SA SMInfo data.

STL-73160

The comments in the Adaptive Routing section of the opafm.xml file do not explain which settings are required for MGAR or for FGAR.

Currently the only field related to FGAR is FineGrained. In order to enable FGAR, set the following:

<AdaptiveRouting>1</AdaptiveRouting> 

<FineGrained>1</FineGrained> 

All other fields are related to legacy MGAR and will be ignored if FineGrained is enabled.

STL-73140

Note the following:

STL-72673

OS-release file captured by opacapture is empty. 

Use cat /etc/os-release command.

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-72573

Hangs and errors have been seen in the OSU alltoall collective test when using the OPX Provider to libfabric with ROCm.

No workaround.

STL-72541

Running the opainfo command when ports are inactive results in the following errors: MAD failed and Failed to get PMA Port Status.

These errors can be ignored.

STL-72536

STL-72537

CableInfo errors encountered when using the opasmaquery -o cableinfo command.

Cable information reporting not currently supported.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

For details, contact support@cornelis.com for technical advisory TA0061.

STL-72420

On some versions of RHEL 9.x, there is a known failure (rdmavt driver deadlock.) recorded in https://issues.redhat.com/browse/RHEL-83851.

NOTE: Cornelis has not replicated this problem but it is possible that customers may be affected.

No workaround.

STL-72160

After uninstalling the meta packages with a package manager, several Cornelis Networks packages remain.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Remove OPX Software from a Host” for removal of the remaining packages.

STL-71364

STL-72423

Some legacy Omni-Path host utilities have been deprecated for CN5000 but are still supported for OPA100.   

For a list of commands supported for CN5000, refer to the Man Pages or CN5000 Commands Guide.

STL-67178

After boot, the OPX Software hfi1-NVIDIA support is not available.

Review boot configuration to ensure that nvidia-driver will be available when hfi1 is loaded.

STL-77956

GPU applications hang with Open MPI 5.0.8 and earlier

Use Open MPI 5.0.9 or newer.

STL-77873

Megatron-LM training (Llama3-8B) hangs on multi-node configurations when using NCCL_NET=OFI with the OPX Provider.

Use NCCL_NET=IB instead of NCCL_NET=OFI.

STL-77578

CableInfo errors encountered when using the opasmaquery -o cableinfo command with direct-routed packets.

Use LID-routed packets to get cable information.

STL-76942

Lustre errors occur with Bulk Transfer Service (BTS) enabled (hfi1 driver parameter use_bulksvc=Y).

When mounting Lustre, BTS should not be enabled (use_bulksvc=N, which is default).

STL-76907

The following Lustre Error may occur:

Write request failed with -14

NOTE: This is a general defect that affects all Lustre 2.17.0 stacks (refer to Lustre-Release Issue #16

Avoid this issue by disabling hybrid IO functionality using the following command on the MGS server:

lctl set_param -P llite.*.hybrid_io=0

STL-76435

opapmaquery fails with the following error when querying performance counters for more than approximately 14 ports simultaneously on a single switch.

Invalid Attribute or Attribute Modifier

NOTE: The PMA response has a maximum payload size that limits the number of ports that can be returned in a single query.

Split the port select mask (-n option) into multiple smaller queries, each covering a subset of ports.

For example, on a 48-port switch, query ports in groups of 20 or fewer:

opapmaquery -o getdatacounters -l <LID> -n 0x0000000FFFFF # ports 0-19

opapmaquery -o getdatacounters -l <LID> -n 0x00FFFFF00000 # ports 20-39

opapmaquery -o getdatacounters -l <LID> -n 0xFF0000000000 # ports 40-47

STL-76348

Running opapaquery -o focusport reports same neighbor LID if the switch is connected to a dual-port SuperNIC.

None.

STL-76323

SuperNICs may intermittently fail to enumerate on the PCIe bus or initialize after a cold boot or power cycle. In affected systems, one or more SuperNICs may be absent from the OS, preventing the node from participating in the fabric until the device is successfully detected and initialized.

Power cycle the server.

STL-76313

Kernel messages such as the following may be seen:

CPORT request wait interrupted

Dropping incoming CPORT message

Set the hfi1 module parameter cport_mad_to=5.

STL-76144

Setting the link speed policy to 100G in the opafm.xml file results in the following Fabric Manager error:

Invalid Value for Link Speed Policy: 100G

Set the link speed using the Switch CLI command port modify speed.

STL-76137

Open MPI traffic fails on SR-IOV Virtual Function devices.

Use the oversubscribe option.

STL-76131

When using SR-IOV, the dmesg log in the VMs may occasionally show the following warning message:

memcpy: detected field-spanning write

Do not run Omni-Path management tools, such as opafabricinfo, on virtual machines. These tools should only be run on the host.

STL-75568

Performance test failures may occur when running ib_send_bw and ib_read_bw with the -D (duration) option.

Do not use the -D option.

STL-75535

GADGET4 MPI application hangs with OPX Provider.

Contact support@cornelis.com.

STL-75526

After installing the OPX Software package on Ubuntu, the opafm service is enabled by default.

Manually disable the opafm service by default.

STL-75484

Running opaportconfig with the disable option immediately followed by running opaportconfig with the enable option will result in the link staying down.

To drop and reinitialize a link, use the bounce option to opaportconfig.

Alternately, if opaportconfig is run from a script, insert a one second delay (sleep 1) between the disable and enable commands.

STL-75395

When AdaptiveRouting is enabled in the opafm.xml file, but FineGrained is not, the Fabric Manager will crash.

Disable AdaptiveRouting in the opafm.xml file and restart the Fabric Manager.

STL-75362

When using the TCP provider of libfabric, some MPI application may perform poorly or hang when launched with 1 process per core. This has been observed in benchmarks such as OSU Congestion, OSU Collectives, and IMB P2P.

For best results, users should use the OPX provider whenever available. OPX can be explicitly selected by setting the environment variable, FI_PROVIDER=opx.

STL-75322

Upon reverting from 12.0.3 to a previous RHEL/Rocky release certain packages may not be removed.

Use the following command to remove the remaining packages:

dnf remove ibacm infiniband-diags 
libfabric libhfi1verbs libibumad 
libibverbs libpsm2 librdmacm 
librdmacm-utils mpi-selector 
opa-basic-tools opa-fastfabric 
opa-fm opa-libopamgt rdma-core 
rdma-core-devel srp_daemon

STL-75031

STL-76644

MPI RMA benchmarks may produce an SDMA error (bad address) when using the OPX libfabric provider and environment variable FI_OPX_MIXED_NETWORK=0.

FI_OPX_MIXED_NETWORK is deprecated and should no longer be set. Contact support@cornelis.com.

STL-74968

Updating the Fabric Manager using DNF may not succeed even though an update is available.

STL-74933

Entries of the form hfi1_0: lost_mctxt_intr forcing int for 2 and MCTXT interrupt, but no status bits set may appear in logs.

These messages can be ignored.

STL-74679

While running IPoIB in Connected mode, application hangs have been observed.

Configure IPoIB to Datagram mode and rerun the application.

STL-74625

When using both ports of a dual-port SuperNIC, the ibstat rate field may show an incorrect value.

Use opainfo, which is the preferred method for querying port information.

STL-74504

Occasional drops in bandwidth observed during microbenchmark testing.

Refer to the Performance Tuning Guide for platform-specific tuning.

STL-74477

Building opxs-kernel-updates from the src.rpm file fails with nvidia/nv-p2p.h not detected.

Use an environment variable to allow the header file to be detected and used in the build:

NVIDIA_DIR=$(ls -d /usr/src/nvidia-* | head -1) 
ln -s ${NVIDIA_DIR}/nvidia /usr/src/kernels/5.14.0-503.14.1.el9_5.x86_64/include/nvidia

Note that the exact directory required for the symbolic link will depend on your kernel version.

STL-74025

When using IPoIB on CN5000, SuperNICs do not change MTU or join MC groups automatically.

Use the ip tool from the command line to manually take the interface down and back up.

STL-73928

HFI links bounce and the SM reports mapping warnings following the second FM restart.

No Workaround. Links that bounce recover and the fabric stabilizes.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73716

The opatop and opapaquery commands may report congestion incorrectly when the fabric is idle.

Ignore the congestion flag.

STL-73714

Performing an opapmaquery to a "B" port on a switch where subdivision is not enabled returns non-zero counter values.

Counter values in this scenario should be ignored.

STL-73713

In a fabric with no data traffic, the Xmit Time Congestion port counter will increment.

No workaround.

STL-73669

Executing opasmaquery -o scvlr for nodes that do not support SC-VLr mappings will result in a message indicating a failure.

The message may be ignored.

STL-73531

Configuring more than two data VLs is not currently supported. Doing so may lead to incorrect mappings of SC to VL.

Limit the number of configured data VLs to two.

STL-73215

When running opasmaquery -o sminfo on CN5000 SuperNICs, SMA fields show all 0s and SM State Inactive.

Use opasaquery -o sminfo or opareport to show correct SA SMInfo data.

STL-73160

The comments in the Adaptive Routing section of the opafm.xml file do not explain which settings are required for MGAR or for FGAR.

Currently the only field related to FGAR is FineGrained. In order to enable FGAR, set the following:

<AdaptiveRouting>1</AdaptiveRouting> 

<FineGrained>1</FineGrained> 

All other fields are related to legacy MGAR and will be ignored if FineGrained is enabled.

STL-73140

Note the following:

STL-72673

OS-release file captured by opacapture is empty. 

Use cat /etc/os-release command.

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-72573

Hangs and errors have been seen in the OSU alltoall collective test when using the OPX Provider to libfabric with ROCm.

No workaround.

STL-72541

Running the opainfo command when ports are inactive results in the following errors: MAD failed and Failed to get PMA Port Status.

These errors can be ignored.

STL-72536

STL-72537

CableInfo errors encountered when using the opasmaquery -o cableinfo command.

Cable information reporting not currently supported.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

For details, contact support@cornelis.com for technical advisory TA0061.

STL-72523

Errors occur on RHEL 9.5 after installation of meta package.

Refer to Disable the Repository After OPX Software Installation.

STL-72420

On some versions of RHEL 9.x, there is a known failure (rdmavt driver deadlock.) recorded in https://issues.redhat.com/browse/RHEL-83851.

NOTE: Cornelis has not replicated this problem but it is possible that customers may be affected.

No workaround.

STL-72205

FI_OPX_HFI_SELECT=[] does not correctly map to the available adapter in the case in which other adapters in the system have been disabled.

Unload and reload the driver.

STL-72160

After uninstalling the meta packages with a package manager, several Cornelis Networks packages remain.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Remove OPX Software from a Host” for removal of the remaining packages.

STL-71364

STL-72423

Some legacy Omni-Path host utilities have been deprecated for CN5000 but are still supported for OPA100.   

For a list of commands supported for CN5000, refer to the Man Pages or CN5000 Commands Guide.

STL-67178

After boot, the OPX Software hfi1-NVIDIA support is not available.

Review boot configuration to ensure that nvidia-driver will be available when hfi1 is loaded.

STL-76942

Lustre errors occur with Bulk Transfer Service (BTS) enabled (hfi1 driver parameter use_bulksvc=Y).

When mounting Lustre, BTS should not be enabled (use_bulksvc=N, which is default).

STL-76907

The following Lustre Error may occur:

Write request failed with -14

NOTE: This is a general defect that affects all Lustre 2.17.0 stacks (refer to Lustre-Release Issue #16

Avoid this issue by disabling hybrid IO functionality using the following command on the MGS server:

lctl set_param -P llite.*.hybrid_io=0

STL-76435

opapmaquery fails with the following error when querying performance counters for more than approximately 14 ports simultaneously on a single switch.

Invalid Attribute or Attribute Modifier

NOTE: The PMA response has a maximum payload size that limits the number of ports that can be returned in a single query.

Split the port select mask (-n option) into multiple smaller queries, each covering a subset of ports.

For example, on a 48-port switch, query ports in groups of 20 or fewer:

opapmaquery -o getdatacounters -l <LID> -n 0x0000000FFFFF # ports 0-19

opapmaquery -o getdatacounters -l <LID> -n 0x00FFFFF00000 # ports 20-39

opapmaquery -o getdatacounters -l <LID> -n 0xFF0000000000 # ports 40-47

STL-76348

Running opapaquery -o focusport reports same neighbor LID if the switch is connected to a dual-port SuperNIC.

None.

STL-76323

SuperNICs may intermittently fail to enumerate on the PCIe bus or initialize after a cold boot or power cycle. In affected systems, one or more SuperNICs may be absent from the OS, preventing the node from participating in the fabric until the device is successfully detected and initialized.

Power cycle the server.

STL-76313

Kernel messages such as the following may be seen:

CPORT request wait interrupted

Dropping incoming CPORT message

Set the hfi1 module parameter cport_mad_to=5.

STL-76263

In some circumstances, rebooting a host may cause a panic.

Power cycle the system.

STL-76144

Setting the link speed policy to 100G in the opafm.xml file results in the following Fabric Manager error:

Invalid Value for Link Speed Policy: 100G

Set the link speed using the Switch CLI command port modify speed.

STL-76137

Open MPI traffic fails on SR-IOV Virtual Function devices.

Use the oversubscribe option.

STL-76131

When using SR-IOV, the dmesg log in the VMs may occasionally show the following warning message:

memcpy: detected field-spanning write

Do not run Omni-Path management tools, such as opafabricinfo, on virtual machines. These tools should only be run on the host.

STL-76108

SR-IOV Virtual Function devices on CN5000 cannot query the Subnet Administration.

Do not run tools such as opareport or opasmquery on Virtual Function interfaces. Run them only on Physical Function interfaces.

STL-75568

Performance test failures may occur when running ib_send_bw and ib_read_bw with the -D (duration) option.

Do not use the -D option.

STL-75535

GADGET4 MPI application hangs with OPX Provider.

Contact support@cornelis.com.

STL-75526

After installing the OPX Software package on Ubuntu, the opafm service is enabled by default.

Manually disable the opafm service by default.

STL-75484

Running opaportconfig with the disable option immediately followed by running opaportconfig with the enable option will result in the link staying down.

To drop and reinitialize a link, use the bounce option to opaportconfig.

Alternately, if opaportconfig is run from a script, insert a one second delay (sleep 1) between the disable and enable commands.

STL-75395

When AdaptiveRouting is enabled in the opafm.xml file, but FineGrained is not, the Fabric Manager will crash.

Disable AdaptiveRouting in the opafm.xml file and restart the Fabric Manager.

STL-75394

On some systems running Ubuntu 24.04, the hfi1 driver may fail to start when it is unloaded and reloaded. The system log will contain a message saying CPORT start failed.

If this message is seen, unload and reload the hfi1 driver again.

STL-75362

When using the TCP provider of libfabric, some MPI application may perform poorly or hang when launched with 1 process per core. This has been observed in benchmarks such as OSU Congestion, OSU Collectives, and IMB P2P.

For best results, users should use the OPX provider whenever available. OPX can be explicitly selected by setting the environment variable, FI_PROVIDER=opx.

STL-75322

Upon reverting from 12.0.3 to a previous RHEL/Rocky release certain packages may not be removed.

Use the following command to remove the remaining packages:

dnf remove ibacm infiniband-diags 
libfabric libhfi1verbs libibumad 
libibverbs libpsm2 librdmacm 
librdmacm-utils mpi-selector 
opa-basic-tools opa-fastfabric 
opa-fm opa-libopamgt rdma-core 
rdma-core-devel srp_daemon

STL-75031

STL-76644

MPI RMA benchmarks may produce an SDMA error (bad address) when using the OPX libfabric provider and environment variable FI_OPX_MIXED_NETWORK=0.

FI_OPX_MIXED_NETWORK is deprecated and should no longer be set. Contact support@cornelis.com.

STL-74968

Updating the Fabric Manager using DNF may not succeed even though an update is available.

STL-74933

Entries of the form hfi1_0: lost_mctxt_intr forcing int for 2 and MCTXT interrupt, but no status bits set may appear in logs.

These messages can be ignored.

STL-74679

While running IPoIB in Connected mode, application hangs have been observed.

Configure IPoIB to Datagram mode and rerun the application.

STL-74625

When using both ports of a dual-port SuperNIC, the ibstat rate field may show an incorrect value.

Use opainfo, which is the preferred method for querying port information.

STL-74593

The Quantum Espresso application with Intel MPI may abort with SIGABRT when executed with a higher number of MPI ranks than available user HFI contexts and multiple endpoints per HFI context (FI_OPX_ENDPOINTS_PER_HFI_CONTEXT=2)

Set FI_OPX_ENDPOINTS_PER_HFI_CONTEXT=1 and reduce the number of MPI ranks to be less than or equal to the number of available HFI user contexts.

STL-74504

Occasional drops in bandwidth observed during microbenchmark testing.

Refer to the Performance Tuning Guide for platform-specific tuning.

STL-74477

Building opxs-kernel-updates from the src.rpm file fails with nvidia/nv-p2p.h not detected.

Use an environment variable to allow the header file to be detected and used in the build:

NVIDIA_DIR=$(ls -d /usr/src/nvidia-* | head -1) 
ln -s ${NVIDIA_DIR}/nvidia /usr/src/kernels/5.14.0-503.14.1.el9_5.x86_64/include/nvidia

Note that the exact directory required for the symbolic link will depend on your kernel version.

STL-74230

IPoIB interfaces on CN5000 currently do not support failover with the Channel Bonding driver in the Linux kernel.

No workaround.

STL-74025

When using IPoIB on CN5000, SuperNICs do not change MTU or join MC groups automatically.

Use the ip tool from the command line to manually take the interface down and back up.

STL-73928

HFI links bounce and the SM reports mapping warnings following the second FM restart.

No Workaround. Links that bounce recover and the fabric stabilizes.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73716

The opatop and opapaquery commands may report congestion incorrectly when the fabric is idle.

Ignore the congestion flag.

STL-73714

Performing an opapmaquery to a "B" port on a switch where subdivision is not enabled returns non-zero counter values.

Counter values in this scenario should be ignored.

STL-73713

In a fabric with no data traffic, the Xmit Time Congestion port counter will increment.

No workaround.

STL-73709

The opasaquery -o trace -P (GUIDs) and opasaquery -o trace -G (GIDs) commands fail when targeting CN5000 SuperNIC cards.

No workaround.

STL-73669

Executing opasmaquery -o scvlr for nodes that do not support SC-VLr mappings will result in a message indicating a failure.

The message may be ignored.

STL-73531

Configuring more than two data VLs is not currently supported. Doing so may lead to incorrect mappings of SC to VL.

Limit the number of configured data VLs to two.

STL-73330

Some LS-DYNA R14.1.0. benchmarks hang with Open MPI when using the OPX Provider for libfabric.

Set the environment variable FI_OPX_RELIABILITY_SERVICE_USEC_MAX=5000.

STL-73215

When running opasmaquery -o sminfo on CN5000 SuperNICs, SMA fields show all 0s and SM State Inactive.

Use opasaquery -o sminfo or opareport to show correct SA SMInfo data.

STL-73160

The comments in the Adaptive Routing section of the opafm.xml file do not explain which settings are required for MGAR or for FGAR.

Currently the only field related to FGAR is FineGrained. In order to enable FGAR, set the following:

<AdaptiveRouting>1</AdaptiveRouting> 

<FineGrained>1</FineGrained> 

All other fields are related to legacy MGAR and will be ignored if FineGrained is enabled.

STL-73140

Note the following:

STL-72673

OS-release file captured by opacapture is empty. 

Use cat /etc/os-release command.

STL-72649

An Open MPI error may occur when using --mca btl self,vader and RMA-style functionality (for example, the OSU Put and Get test or IMB-RMA).

Contact support@cornelis.com.

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-72573

Hangs and errors have been seen in the OSU alltoall collective test when using the OPX Provider to libfabric with ROCm.

No workaround.

STL-72541

Running the opainfo command when ports are inactive results in the following errors: MAD failed and Failed to get PMA Port Status.

These errors can be ignored.

STL-72536

STL-72537

CableInfo errors encountered when using the opasmaquery -o cableinfo command.

Cable information reporting not currently supported.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

The port number used by the FM is equivalent to twice the switch CLI port name minus one. Contact Cornelis Customer Support for details.

STL-72523

Errors occur on RHEL 9.5 after installation of meta package.

Refer to Disable the Repository After OPX Software Installation.

STL-72431

The command opasaquery -o node -n [NodeGUID] results in output of No Records Returned. 

Use opsmaquery -o nodeinfo -l 4, where l corresponds to the LID of the node.

STL-72420

On some versions of RHEL 9.x, there is a known failure (rdmavt driver deadlock.) recorded in https://issues.redhat.com/browse/RHEL-83851.

NOTE: Cornelis has not replicated this problem but it is possible that customers may be affected.

No workaround.

STL-72205

FI_OPX_HFI_SELECT=[] does not correctly map to the available adapter in the case in which other adapters in the system have been disabled.

Unload and reload the driver.

STL-72160

After uninstalling the meta packages with a package manager, several Cornelis Networks packages remain.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Remove OPX Software from a Host” for removal of the remaining packages.

STL-71364

STL-72423

Some legacy Omni-Path host utilities have been deprecated for CN5000 but are still supported for OPA100.   

For a list of commands supported for CN5000, refer to the Man Pages or CN5000 Commands Guide.

STL-67178

After boot, the OPX Software hfi1-NVIDIA support is not available.

Review boot configuration to ensure that nvidia-driver will be available when hfi1 is loaded.

STL-76376

Kernel crash may occur when hfi1 module is loaded with use_bulksvc=Y.

12.1.0.1

STL-76375

MPI or middleware using the OPX Libfabric Provider may experience unreliable results.

12.1.0.1

STL-76373

Benign hfi1 messages relating to MCTXT interrupt in kernel logs can be ignored.

12.1.0.1

STL-75066

If IPv6 is enabled on any IPoIB interfaces, the FM sweeps constantly due to changes in multicast group membership.

12.1.0

STL-74951

STL-74672

On rare occasions, application hangs with an SDMA engine error in DMSG have been observed.

12.1.0

STL-74696

When running an NVIDIA GPU Cloud (NGC) container such as the "NVIDIA HPC-Benchmarks" package, a segmentation fault may occur.

12.1.0

STL-74672

On rare occasions, application hangs with an SDMA engine error in DMSG have been observed.

12.1.0

STL-74234

The mvapich2 packages provided with OPX Software should be ignored and not used.

12.1.0

STL-74218

IPoIB interfaces on CN5000 currently do not support failover with the Channel Bonding driver in the Linux kernel.

12.1.0

STL-73861

Hangs occur when using NVMe over Fabric.

12.1.0

STL-73759

When the FM restarts, different LIDs may get assigned to the nodes.

12.1.0

STL-73267

When running the opasmaquery -o portinfo - g command, the output does not display a value for VLSchedulingConfig.

12.1.0

STL-73245

CN5000 Switch-Switch links do not appear in either opareport -o extlinks or opareport -o extislinks command output.

12.1.0

STL-72864

High CPU utilization seen for hfi1 related interrupts, potentially impacting performance.

12.1.0

STL-72815

opapaquery -o vfConfig -V 'Default' may display duplicated port entries.

12.1.0

STL-72640

In some cases the dmesg log may contain a message stating:

hfi1_tid_rdma_flush_wait: no rcd for QP

12.1.0

STL-72575

TID updates under heavy load may result in an -ENOMEM error.

12.1.0

STL-72572

In opasmaquery -o swinfo, there is no indication whether the CN5000 fine-grained adaptive routing (FGAR) is supported.

12.1.0

STL-72547

Class Port Info queries are not reporting properly.

12.1.0

STL-76263

In some circumstances, rebooting a host may cause a panic.

Power cycle the system.

STL-76222

Memory overflow may occur when running MVAPICH on Ubuntu platforms.

Set MVP_ENABLE_AFFINITY=0.

STL-76218

Crashes may occur when SuperNIC ports have inconsistent bulksvc settings, with some ports enabled and some not.

Ensure that bulksvc is enabled on all ports of all SuperNICs in the fabric.

STL-76144

Setting the link speed policy to 100G in the opafm.xml file results in the following Fabric Manager error:

Invalid Value for Link Speed Policy: 100G

Set the link speed using the Switch CLI command port modify speed.

STL-76137

Open MPI traffic fails on SR-IOV Virtual Function devices.

Use the oversubscribe option.

STL-76135

When using SR-IOV, jobs running high processes per node (PPN) may fail on virtual machines with limited resources.

Ensure that the virtual machine is provisioned with at least twice the number of virtual CPUs as the desired number of processes to be run.

STL-76131

When using SR-IOV, the dmesg log in the VMs may occasionally show the following warning message:

memcpy: detected field-spanning write

Do not run Omni-Path management tools, such as opafabricinfo, on virtual machines. These tools should only be run on the host.

STL-76108

SR-IOV Virtual Function devices on CN5000 cannot query the Subnet Administration.

Do not run tools such as opareport or opasmquery on Virtual Function interfaces. Run them only on Physical Function interfaces.

STL-75568

Performance test failures may occur when running ib_send_bw and ib_read_bw with the -D (duration) option.

Do not use the -D option.

STL-75555

Errors can occur when installing Ubuntu meta-packages with apt or apt-get if the required dependencies are not the latest versions available on Ubuntu.

Use one of the following:

STL-75535

GADGET4 MPI application hangs with OPX Provider.

Set FI_OPX_MIXED_NETWORK=0 environment variable.

STL-75526

After installing the OPX Software package on Ubuntu, the opafm service is enabled by default.

Manually disable the opafm service by default.

STL-75484

Running opaportconfig with the disable option immediately followed by running opaportconfig with the enable option will result in the link staying down.

To drop and reinitialize a link, use the bounce option to opaportconfig.

Alternately, if opaportconfig is run from a script, insert a one second delay (sleep 1) between the disable and enable commands.

STL-75395

When AdaptiveRouting is enabled in the opafm.xml file, but FineGrained is not, the Fabric Manager will crash.

Disable AdaptiveRouting in the opafm.xml file and restart the Fabric Manager.

STL-75394

On some systems running Ubuntu 24.04, the hfi1 driver may fail to start when it is unloaded and reloaded. The system log will contain a message saying CPORT start failed.

If this message is seen, unload and reload the hfi1 driver again.

STL-75362

When using the TCP provider of libfabric, some MPI application may perform poorly or hang when launched with 1 process per core. This has been observed in benchmarks such as OSU Congestion, OSU Collectives, and IMB P2P.

For best results, users should use the OPX provider whenever available. OPX can be explicitly selected by setting the environment variable, FI_PROVIDER=opx.

STL-75331

Using RDMA Core persistent names with settings other than KERNEL will result in RCCL segmentation faults.

Perform the following:

STL-75322

Upon reverting from 12.0.3 to a previous RHEL/Rocky release certain packages may not be removed.

Use the following command to remove the remaining packages:

dnf remove ibacm infiniband-diags 
libfabric libhfi1verbs libibumad 
libibverbs libpsm2 librdmacm 
librdmacm-utils mpi-selector 
opa-basic-tools opa-fastfabric 
opa-fm opa-libopamgt rdma-core 
rdma-core-devel srp_daemon

STL-75031

MPI RMA benchmarks may produce an SDMA error (bad address) when using the OPX libfabric provider and environment variable FI_OPX_MIXED_NETWORK=0.

Set FI_OPX_MIXED_NETWORK=1.

STL-74968

Updating the Fabric Manager using DNF may not succeed even though an update is available.

STL-74933

Entries of the form hfi1_0: lost_mctxt_intr forcing int for 2 and MCTXT interrupt, but no status bits set may appear in logs.

These messages can be ignored.

STL-74679

While running IPoIB in Connected mode, application hangs have been observed.

Configure IPoIB to Datagram mode and rerun the application.

STL-74625

When using both ports of a dual-port SuperNIC, the ibstat rate field may show an incorrect value.

Use opainfo, which is the preferred method for querying port information.

STL-74593

The Quantum Espresso application with Intel MPI may abort with SIGABRT when executed with a higher number of MPI ranks than available user HFI contexts and multiple endpoints per HFI context (FI_OPX_ENDPOINTS_PER_HFI_CONTEXT=2)

Set FI_OPX_ENDPOINTS_PER_HFI_CONTEXT=1 and reduce the number of MPI ranks to be less than or equal to the number of available HFI user contexts.

STL-74504

Occasional drops in bandwidth observed during microbenchmark testing.

Refer to the Performance Tuning Guide for platform-specific tuning.

STL-74477

Building opxs-kernel-updates from the src.rpm file fails with nvidia/nv-p2p.h not detected.

Use an environment variable to allow the header file to be detected and used in the build:

NVIDIA_DIR=$(ls -d /usr/src/nvidia-* | head -1) 
ln -s ${NVIDIA_DIR}/nvidia /usr/src/kernels/5.14.0-503.14.1.el9_5.x86_64/include/nvidia

Note that the exact directory required for the symbolic link will depend on your kernel version.

STL-74230

IPoIB interfaces on CN5000 currently do not support failover with the Channel Bonding driver in the Linux kernel.

No workaround.

STL-74025

When using IPoIB on CN5000, SuperNICs do not change MTU or join MC groups automatically.

Use the ip tool from the command line to manually take the interface down and back up.

STL-73928

HFI links bounce and the SM reports mapping warnings following the second FM restart.

No Workaround. Links that bounce recover and the fabric stabilizes.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73716

The opatop and opapaquery commands may report congestion incorrectly when the fabric is idle.

Ignore the congestion flag.

STL-73714

Performing an opapmaquery to a "B" port on a switch where subdivision is not enabled returns non-zero counter values.

Counter values in this scenario should be ignored.

STL-73713

In a fabric with no data traffic, the Xmit Time Congestion port counter will increment.

No workaround.

STL-73709

The opasaquery -o trace -P (GUIDs) and opasaquery -o trace -G (GIDs) commands fail when targeting CN5000 SuperNIC cards.

No workaround.

STL-73669

Executing opasmaquery -o scvlr for nodes that do not support SC-VLr mappings will result in a message indicating a failure.

The message may be ignored.

STL-73531

Configuring more than two data VLs is not currently supported. Doing so may lead to incorrect mappings of SC to VL.

Limit the number of configured data VLs to two.

STL-73330

Some LS-DYNA R14.1.0. benchmarks hang with Open MPI when using the OPX Provider for libfabric.

Set the environment variable FI_OPX_RELIABILITY_SERVICE_USEC_MAX=5000.

STL-73215

When running opasmaquery -o sminfo on CN5000 SuperNICs, SMA fields show all 0s and SM State Inactive.

Use opasaquery -o sminfo or opareport to show correct SA SMInfo data.

STL-73160

The comments in the Adaptive Routing section of the opafm.xml file do not explain which settings are required for MGAR or for FGAR.

Currently the only field related to FGAR is FineGrained. In order to enable FGAR, set the following:

<AdaptiveRouting>1</AdaptiveRouting> 

<FineGrained>1</FineGrained> 

All other fields are related to legacy MGAR and will be ignored if FineGrained is enabled.

STL-73140

Note the following:

STL-72673

OS-release file captured by opacapture is empty. 

Use cat /etc/os-release command.

STL-72649

An Open MPI error may occur when using --mca btl self,vader and RMA-style functionality (for example, the OSU Put and Get test or IMB-RMA).

Contact support@cornelis.com.

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-72573

Hangs and errors have been seen in the OSU alltoall collective test when using the OPX Provider to libfabric with ROCm.

No workaround.

STL-72541

Running the opainfo command when ports are inactive results in the following errors: MAD failed and Failed to get PMA Port Status.

These errors can be ignored.

STL-72536

STL-72537

CableInfo errors encountered when using the opasmaquery -o cableinfo command.

Cable information reporting not currently supported.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

The port number used by the FM is equivalent to twice the switch CLI port name minus one. Contact Cornelis Customer Support for details.

STL-72523

Errors occur on RHEL 9.5 after installation of meta package.

Refer to Disable the Repository After OPX Software Installation.

STL-72431

The command opasaquery -o node -n [NodeGUID] results in output of No Records Returned. 

Use opsmaquery -o nodeinfo -l 4, where l corresponds to the LID of the node.

STL-72420

On some versions of RHEL 9.x, there is a known failure (rdmavt driver deadlock.) recorded in https://issues.redhat.com/browse/RHEL-83851.

NOTE: Cornelis has not replicated this problem but it is possible that customers may be affected.

No workaround.

STL-72205

FI_OPX_HFI_SELECT=[] does not correctly map to the available adapter in the case in which other adapters in the system have been disabled.

Unload and reload the driver.

STL-72160

After uninstalling the meta packages with a package manager, several Cornelis Networks packages remain.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Remove OPX Software from a Host” for removal of the remaining packages.

STL-71364

STL-72423

Some legacy Omni-Path host utilities have been deprecated for CN5000 but are still supported for OPA100.   

For a list of commands supported for CN5000, refer to the Man Pages or CN5000 Commands Guide.

STL-70422

RCCL hangs on AMD GPUs with opx and aws-ofi-rccl plugin.

Use Verbs over OPX Provider when using RCCL.

STL-67178

After boot, the OPX Software hfi1-NVIDIA support is not available.

Review boot configuration to ensure that nvidia-driver will be available when hfi1 is loaded.