STL-76376

Kernel crash may occur when hfi1 module is loaded with use_bulksvc=Y.

12.1.0.1

STL-76375

MPI or middleware using the OPX Libfabric Provider may experience unreliable results.

12.1.0.1

STL-76373

Benign hfi1 messages relating to MCTXT interrupt in kernel logs can be ignored.

12.1.0.1

STL-75066

If IPv6 is enabled on any IPoIB interfaces, the FM sweeps constantly due to changes in multicast group membership.

12.1.0

STL-74951

STL-74672

On rare occasions, application hangs with an SDMA engine error in DMSG have been observed.

12.1.0

STL-74696

When running an NVIDIA GPU Cloud (NGC) container such as the "NVIDIA HPC-Benchmarks" package, a segmentation fault may occur.

12.1.0

STL-74672

On rare occasions, application hangs with an SDMA engine error in DMSG have been observed.

12.1.0

STL-74234

The mvapich2 packages provided with OPX Software should be ignored and not used.

12.1.0

STL-74218

IPoIB interfaces on CN5000 currently do not support failover with the Channel Bonding driver in the Linux kernel.

12.1.0

STL-73861

Hangs occur when using NVMe over Fabric.

12.1.0

STL-73759

When the FM restarts, different LIDs may get assigned to the nodes.

12.1.0

STL-73267

When running the opasmaquery -o portinfo - g command, the output does not display a value for VLSchedulingConfig.

12.1.0

STL-73245

CN5000 Switch-Switch links do not appear in either opareport -o extlinks or opareport -o extislinks command output.

12.1.0

STL-72864

High CPU utilization seen for hfi1 related interrupts, potentially impacting performance.

12.1.0

STL-72815

opapaquery -o vfConfig -V 'Default' may display duplicated port entries.

12.1.0

STL-72640

In some cases the dmesg log may contain a message stating:

hfi1_tid_rdma_flush_wait: no rcd for QP

12.1.0

STL-72575

TID updates under heavy load may result in an -ENOMEM error.

12.1.0

STL-72572

In opasmaquery -o swinfo, there is no indication whether the CN5000 fine-grained adaptive routing (FGAR) is supported.

12.1.0

STL-72547

Class Port Info queries are not reporting properly.

12.1.0

STL-76263

In some circumstances, rebooting a host may cause a panic.

Power cycle the system.

STL-76222

Memory overflow may occur when running MVAPICH on Ubuntu platforms.

Set MVP_ENABLE_AFFINITY=0.

STL-76218

Crashes may occur when SuperNIC ports have inconsistent bulksvc settings, with some ports enabled and some not.

Ensure that bulksvc is enabled on all ports of all SuperNICs in the fabric.

STL-76144

Setting the link speed policy to 100G in the opafm.xml file results in the following Fabric Manager error:

Invalid Value for Link Speed Policy: 100G

Set the link speed using the Switch CLI command port modify speed.

STL-76137

Open MPI traffic fails on SR-IOV Virtual Function devices.

Use the oversubscribe option.

STL-76135

When using SR-IOV, jobs running high processes per node (PPN) may fail on virtual machines with limited resources.

Ensure that the virtual machine is provisioned with at least twice the number of virtual CPUs as the desired number of processes to be run.

STL-76131

When using SR-IOV, the dmesg log in the VMs may occasionally show the following warning message:

memcpy: detected field-spanning write

Do not run Omni-Path management tools, such as opafabricinfo, on virtual machines. These tools should only be run on the host.

STL-76108

SR-IOV Virtual Function devices on CN5000 cannot query the Subnet Administration.

Do not run tools such as opareport or opasmquery on Virtual Function interfaces. Run them only on Physical Function interfaces.

STL-75568

Performance test failures may occur when running ib_send_bw and ib_read_bw with the -D (duration) option.

Do not use the -D option.

STL-75555

Errors can occur when installing Ubuntu meta-packages with apt or apt-get if the required dependencies are not the latest versions available on Ubuntu.

Use one of the following:

STL-75535

GADGET4 MPI application hangs with OPX Provider.

Set FI_OPX_MIXED_NETWORK=0 environment variable.

STL-75526

After installing the OPX Software package on Ubuntu, the opafm service is enabled by default.

Manually disable the opafm service by default.

STL-75484

Running opaportconfig with the disable option immediately followed by running opaportconfig with the enable option will result in the link staying down.

To drop and reinitialize a link, use the bounce option to opaportconfig.

Alternately, if opaportconfig is run from a script, insert a one second delay (sleep 1) between the disable and enable commands.

STL-75395

When AdaptiveRouting is enabled in the opafm.xml file, but FineGrained is not, the Fabric Manager will crash.

Disable AdaptiveRouting in the opafm.xml file and restart the Fabric Manager.

STL-75394

On some systems running Ubuntu 24.04, the hfi1 driver may fail to start when it is unloaded and reloaded. The system log will contain a message saying CPORT start failed.

If this message is seen, unload and reload the hfi1 driver again.

STL-75362

When using the TCP provider of libfabric, some MPI application may perform poorly or hang when launched with 1 process per core. This has been observed in benchmarks such as OSU Congestion, OSU Collectives, and IMB P2P.

For best results, users should use the OPX provider whenever available. OPX can be explicitly selected by setting the environment variable, FI_PROVIDER=opx.

STL-75331

Using RDMA Core persistent names with settings other than KERNEL will result in RCCL segmentation faults.

Perform the following:

STL-75322

Upon reverting from 12.0.3 to a previous RHEL/Rocky release certain packages may not be removed.

Use the following command to remove the remaining packages:

dnf remove ibacm infiniband-diags 
libfabric libhfi1verbs libibumad 
libibverbs libpsm2 librdmacm 
librdmacm-utils mpi-selector 
opa-basic-tools opa-fastfabric 
opa-fm opa-libopamgt rdma-core 
rdma-core-devel srp_daemon

STL-75031

MPI RMA benchmarks may produce an SDMA error (bad address) when using the OPX libfabric provider and environment variable FI_OPX_MIXED_NETWORK=0.

Set FI_OPX_MIXED_NETWORK=1.

STL-74968

Updating the Fabric Manager using DNF may not succeed even though an update is available.

STL-74933

Entries of the form hfi1_0: lost_mctxt_intr forcing int for 2 and MCTXT interrupt, but no status bits set may appear in logs.

These messages can be ignored.

STL-74679

While running IPoIB in Connected mode, application hangs have been observed.

Configure IPoIB to Datagram mode and rerun the application.

STL-74625

When using both ports of a dual-port SuperNIC, the ibstat rate field may show an incorrect value.

Use opainfo, which is the preferred method for querying port information.

STL-74593

The Quantum Espresso application with Intel MPI may abort with SIGABRT when executed with a higher number of MPI ranks than available user HFI contexts and multiple endpoints per HFI context (FI_OPX_ENDPOINTS_PER_HFI_CONTEXT=2)

Set FI_OPX_ENDPOINTS_PER_HFI_CONTEXT=1 and reduce the number of MPI ranks to be less than or equal to the number of available HFI user contexts.

STL-74504

Occasional drops in bandwidth observed during microbenchmark testing.

Refer to the Performance Tuning Guide for platform-specific tuning.

STL-74477

Building opxs-kernel-updates from the src.rpm file fails with nvidia/nv-p2p.h not detected.

Use an environment variable to allow the header file to be detected and used in the build:

NVIDIA_DIR=$(ls -d /usr/src/nvidia-* | head -1) 
ln -s ${NVIDIA_DIR}/nvidia /usr/src/kernels/5.14.0-503.14.1.el9_5.x86_64/include/nvidia

Note that the exact directory required for the symbolic link will depend on your kernel version.

STL-74230

IPoIB interfaces on CN5000 currently do not support failover with the Channel Bonding driver in the Linux kernel.

No workaround.

STL-74025

When using IPoIB on CN5000, SuperNICs do not change MTU or join MC groups automatically.

Use the ip tool from the command line to manually take the interface down and back up.

STL-73928

HFI links bounce and the SM reports mapping warnings following the second FM restart.

No Workaround. Links that bounce recover and the fabric stabilizes.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73716

The opatop and opapaquery commands may report congestion incorrectly when the fabric is idle.

Ignore the congestion flag.

STL-73714

Performing an opapmaquery to a "B" port on a switch where subdivision is not enabled returns non-zero counter values.

Counter values in this scenario should be ignored.

STL-73713

In a fabric with no data traffic, the Xmit Time Congestion port counter will increment.

No workaround.

STL-73709

The opasaquery -o trace -P (GUIDs) and opasaquery -o trace -G (GIDs) commands fail when targeting CN5000 SuperNIC cards.

No workaround.

STL-73669

Executing opasmaquery -o scvlr for nodes that do not support SC-VLr mappings will result in a message indicating a failure.

The message may be ignored.

STL-73531

Configuring more than two data VLs is not currently supported. Doing so may lead to incorrect mappings of SC to VL.

Limit the number of configured data VLs to two.

STL-73330

Some LS-DYNA R14.1.0. benchmarks hang with Open MPI when using the OPX Provider for libfabric.

Set the environment variable FI_OPX_RELIABILITY_SERVICE_USEC_MAX=5000.

STL-73215

When running opasmaquery -o sminfo on CN5000 SuperNICs, SMA fields show all 0s and SM State Inactive.

Use opasaquery -o sminfo or opareport to show correct SA SMInfo data.

STL-73160

The comments in the Adaptive Routing section of the opafm.xml file do not explain which settings are required for MGAR or for FGAR.

Currently the only field related to FGAR is FineGrained. In order to enable FGAR, set the following:

<AdaptiveRouting>1</AdaptiveRouting> 

<FineGrained>1</FineGrained> 

All other fields are related to legacy MGAR and will be ignored if FineGrained is enabled.

STL-73140

Note the following:

STL-72673

OS-release file captured by opacapture is empty. 

Use cat /etc/os-release command.

STL-72649

An Open MPI error may occur when using --mca btl self,vader and RMA-style functionality (for example, the OSU Put and Get test or IMB-RMA).

Contact support@cornelis.com.

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-72573

Hangs and errors have been seen in the OSU alltoall collective test when using the OPX Provider to libfabric with ROCm.

No workaround.

STL-72541

Running the opainfo command when ports are inactive results in the following errors: MAD failed and Failed to get PMA Port Status.

These errors can be ignored.

STL-72536

STL-72537

CableInfo errors encountered when using the opasmaquery -o cableinfo command.

Cable information reporting not currently supported.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

The port number used by the FM is equivalent to twice the switch CLI port name minus one. Contact Cornelis Customer Support for details.

STL-72523

Errors occur on RHEL 9.5 after installation of meta package.

Refer to Disable the Repository After OPX Software Installation.

STL-72431

The command opasaquery -o node -n [NodeGUID] results in output of No Records Returned. 

Use opsmaquery -o nodeinfo -l 4, where l corresponds to the LID of the node.

STL-72420

On some versions of RHEL 9.x, there is a known failure (rdmavt driver deadlock.) recorded in https://issues.redhat.com/browse/RHEL-83851.

NOTE: Cornelis has not replicated this problem but it is possible that customers may be affected.

No workaround.

STL-72205

FI_OPX_HFI_SELECT=[] does not correctly map to the available adapter in the case in which other adapters in the system have been disabled.

Unload and reload the driver.

STL-72160

After uninstalling the meta packages with a package manager, several Cornelis Networks packages remain.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Remove OPX Software from a Host” for removal of the remaining packages.

STL-71364

STL-72423

Some legacy Omni-Path host utilities have been deprecated for CN5000 but are still supported for OPA100.   

For a list of commands supported for CN5000, refer to the Man Pages or CN5000 Commands Guide.

STL-70422

RCCL hangs on AMD GPUs with opx and aws-ofi-rccl plugin.

Use Verbs over OPX Provider when using RCCL.

STL-67178

After boot, the OPX Software hfi1-NVIDIA support is not available.

Review boot configuration to ensure that nvidia-driver will be available when hfi1 is loaded.