STL-76775

Ports that have been disabled may be re-enabled after a reboot of the switch.

12.4.0

STL-78122

After restarting the Fabric Manager, the switch does not rejoin IPoIB multicast groups.

Restart IPoIB on the switch by disabling and re-enabling it:

network ib --disable
network ib --enable

STL-77842

nmap port scan of IPoIB address causes switch instability.

Do not run nmap port scan against the switch over the IPoIB network.

STL-77720

The switch IPoIB interface (ib0) fails to establish carrier and remains down when the fabric's default multicast group MTU is configured above 4096 bytes (for example, 10K MTU for Accelerated IP/AIP support).

None.

STL-77407

Unable to delete IPoIB IPv6 address after adding it.

None.

STL-76677

When removing SSH keys, partial matches of the comment could remove more keys than intended.

When removing an SSH key, use the entire comment of the key.

STL-77756

After updating the SuperNIC firmware using updateAgent, one or more SuperNIC cards in a multi-card system may remain in a "Pending" firmware state following an AC power cycle. Additional AC power cycles do not resolve the issue.

Re-run the updateAgent firmware update on the affected SuperNIC card(s).

To help ensure all cards complete the firmware update on the first attempt, fully power off the system (not just a power cycle) and wait at least one minute before powering back on, allowing all card capacitors to fully discharge.

STL-75609

The updateAgent tool may become stuck when downgrading SuperNIC firmware.

Use updateAgent -f when downgrading firmware.

STL-73195

When updating the SuperNIC firmware using updateAgent -v, the following error may occur:

While waiting for a response to PLDM type 05 and command code 14, a message with type 05 and command 15 was received 

Retry the updateAgent -v command.

STL-79912

NCCL collectives may timeout or hang when running with non-NVLink GPU pairs on the same node.

Set NCCL_CUMEM_ENABLE=0.

STL-79864

When MPS is set to 256 before loading the hfi1 kernel module, a kernel halt condition may occur.

Set MPS=256 after hfi1 is loaded.

Refer to the CN5000 Performance Tuning Guide for more details.

STL-79845

When ipoib_tx_mode=hfisvc is set, transmit throughput remains low.

IPoIB HFI Service (HFISVC) transmit mode requires that hfi1 be loaded before ib_ipoib.

Refer to the CN5000 Performance Tuning Guide for the required load order and how to verify it.

STL-79799

Unexpected system fault and reboot may occur while running traffic with ipoib_tx_mode=hfisvc and use_bulksvc=N.

Operating with ipoib_tx_mode=hfisvc and use_bulksvc=N is an unsupported configuration.

Refer to the CN5000 Performance Tuning Guide or contact Cornelis Technical Support for help with configurations.

STL-79771

Installing the Ubuntu distribution perftest package attempts to uninstall OPX Software due to package conflicts.

Build and run perftest from source using the upstream repository: Perftest (GitHub).

STL-79587

In distro Truescale drivers (ib_qib.ko) are incompatible with modern rdmavt kernel modules.

Blacklist ib_qib or remove the .ko file to prevent error messages. Truescale is no longer compatible with the shipping OPX Software 12.x releases.

STL-79559

When running RCCL over its built-in verbs transport (NCCL_NET=IB), RCCL uses only two SuperNICs per node for inter-node communication, regardless of how many are installed. Additional SuperNICs are initialized but transmit no traffic, limiting collective communication bandwidth.

NOTE: This is a RCCL defect (ROCm/rocm-systems#11138).

Perform one of the following:

STL-79524

In certain Linux distributions, the default DHCP service does not support InfiniBand or IP over InfiniBand (IPoIB) interfaces.

Contact your Linux distribution vendor for approved alternative DHCP packages or configuration workarounds for IPoIB interfaces.

STL-79445

Five-minute timeouts may occur in BeeGFS when RDMA connections are enabled (connUseRDMA = true).

NOTE: A patch is pending per Beegfs (GitHub).

Set connUseRDMA = false.

STL-79193

On SLES distros, the hfi1 driver version may appear incorrect.

To ensure the correct driver is running, use the srcversion field.

STL-78398

On nodes running Ubuntu 24.04 with OPX Software installed, the srp_daemon service continuously generates ib_srp errors in dmesg. These errors include repeated SRP LOGIN rejections, failed path record queries (status −22), and failed SCSI host connections.

The errors persist indefinitely and can consume excessive CPU resources until the srp_daemon process is stopped.

Stop the srp_daemon service or disable it from starting at boot.

NOTE: Disabling srp_daemon will prevent automatic discovery of SCSI RDMA Protocol (SRP) targets. If SRP storage is not in use on the affected nodes, this workaround has no functional impact.

STL-78397

On Ubuntu 24.04, the opafm service may intermittently fail to start at boot, reporting:

ibwarn: get_abi_version: can't read ABI version from /sys/class/infiniband_mad/abi_version
Invalid config detected! Couldn't find device for instance 0!

Couldn't find device for instance 0! This is caused by a boot-time race condition.

After a reboot, perform the following:

STL-78250

A Verbs SEND operation using PIO may fail to detect that send credits have been returned. This causes the send path to get stuck, preventing further send operations even though the hardware is ready.

Set piothreshold=0 in the HFI1 driver module parameters.

STL-78089

ROCm HIP pointer queries may cause a process hang during teardown. This is a known ROCm issue related to OpenMP target allocations recorded in https://github.com/ROCm/rocm-systems/issues/7099.

Use HIP-managed allocations (e.g., hipMalloc()) instead of OpenMP target allocations for buffers used with OPX GPU-direct/HMEM operations. Avoid mixing OpenMP target allocations with HIP pointer introspection until the ROCm issue is resolved upstream.

STL-77891

When installing or uninstalling the opxs-kernel-updates-kmp-default package on SLES 16, warnings appear stating that the RPM's post-install or post-uninstall scriptlet failed.

These warnings can be ignored.

STL-77578

CableInfo errors encountered when using the opasmaquery -o cableinfo command with direct-routed packets.

Use LID-routed packets to get cable information.

STL-77479

Running opatop may report over line rate.

None.

STL-77171

On SLES, building the kernel driver RPMs may fail when multiple kernels are installed.

STL-76907

The following Lustre Error may occur:

Write request failed with -14

NOTE: This is a general defect that affects all Lustre 2.17.0 stacks (refer to Lustre-Release Issue #16

Avoid this issue by disabling hybrid IO functionality using the following command on the MGS server:

lctl set_param -P llite.*.hybrid_io=0

STL-76435

opapmaquery fails with the following error when querying performance counters for more than approximately 14 ports simultaneously on a single switch.

Invalid Attribute or Attribute Modifier

NOTE: The PMA response has a maximum payload size that limits the number of ports that can be returned in a single query.

Split the port select mask (-n option) into multiple smaller queries, each covering a subset of ports.

For example, on a 48-port switch, query ports in groups of 20 or fewer:

opapmaquery -o getdatacounters -l <LID> -n 0x0000000FFFFF # ports 0-19

opapmaquery -o getdatacounters -l <LID> -n 0x00FFFFF00000 # ports 20-39

opapmaquery -o getdatacounters -l <LID> -n 0xFF0000000000 # ports 40-47

STL-76348

Running opapaquery -o focusport reports same neighbor LID if the switch is connected to a dual-port SuperNIC.

None.

STL-76323

SuperNICs may intermittently fail to enumerate on the PCIe bus or initialize after a cold boot or power cycle. In affected systems, one or more SuperNICs may be absent from the OS, preventing the node from participating in the fabric until the device is successfully detected and initialized.

Power cycle the server.

STL-76313

Kernel messages such as the following may be seen:

CPORT request wait interrupted

Dropping incoming CPORT message

Set the hfi1 module parameter cport_mad_to=5.

STL-75568

Performance test failures may occur when running ib_send_bw and ib_read_bw with the -D (duration) option.

Do not use the -D option.

STL-75484

Running opaportconfig with the disable option immediately followed by running opaportconfig with the enable option will result in the link staying down.

To drop and reinitialize a link, use the bounce option to opaportconfig.

Alternately, if opaportconfig is run from a script, insert a one second delay (sleep 1) between the disable and enable commands.

STL-74968

Updating the Fabric Manager using DNF may not succeed even though an update is available.

STL-74477

Building opxs-kernel-updates from the src.rpm file fails with nvidia/nv-p2p.h not detected.

Use an environment variable to allow the header file to be detected and used in the build:

NVIDIA_DIR=$(ls -d /usr/src/nvidia-* | head -1) 
ln -s ${NVIDIA_DIR}/nvidia /usr/src/kernels/5.14.0-503.14.1.el9_5.x86_64/include/nvidia

Note that the exact directory required for the symbolic link will depend on your kernel version.

STL-74025

When using IPoIB, SuperNICs do not change MTU or join MC groups automatically.

Use the ip tool from the command line to manually take the interface down and back up.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73714

Performing an opapmaquery to a "B" port on a switch where subdivision is not enabled returns non-zero counter values.

Counter values in this scenario should be ignored.

STL-73713

In a fabric with no data traffic, the Xmit Time Congestion port counter will increment.

No workaround.

STL-73669

Executing opasmaquery -o scvlr for nodes that do not support SC-VLr mappings will result in a message indicating a failure.

The message may be ignored.

STL-73215

When running opasmaquery -o sminfo on CN5000 SuperNICs, SMA fields show all 0s and SM State Inactive.

Use opasaquery -o sminfo or opareport to show correct SA SMInfo data.

STL-73160

The comments in the Adaptive Routing section of the opafm.xml file do not explain which settings are required for MGAR or for FGAR.

Currently the only field related to FGAR is FineGrained. In order to enable FGAR, set the following:

<AdaptiveRouting>1</AdaptiveRouting> 

<FineGrained>1</FineGrained> 

All other fields are related to legacy MGAR and will be ignored if FineGrained is enabled.

STL-73140

Note the following:

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-72541

Running the opainfo command when ports are inactive results in the following errors: MAD failed and Failed to get PMA Port Status.

These errors can be ignored.

STL-72536

STL-72537

CableInfo errors encountered when using the opasmaquery -o cableinfo command.

Cable information reporting not currently supported.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

For details, contact support@cornelis.com for technical advisory TA0061.

STL-72420

On some versions of RHEL 9.x, there is a known failure (rdmavt driver deadlock.) recorded in https://issues.redhat.com/browse/RHEL-83851.

NOTE: Cornelis has not replicated this problem, but it is possible that customers may be affected.

No workaround.

STL-72160

After uninstalling the meta packages with a package manager, several Cornelis Networks packages remain.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Remove OPX Software from a Host” for removal of the remaining packages.

STL-72028

Hangs have been seen in the NCCL alltoall_perf test when using the OPX provider to libfabric.

No workaround.

STL-71364

STL-72423

Some legacy Omni-Path host utilities have been deprecated for CN5000 but are still supported for OPA100.   

For a list of commands supported for CN5000, refer to the Man Pages or CN5000 Commands Guide.

STL-70552

Rebuilding Open MPI or mvapich2 with FastFabric utilities (do_openmpi_build and do_mvapich2_build) will return an error if using the Intel icx compiler.

These utilities are compatible only with the Intel ICC compiler. Rebuild Open MPI or mvapich2 using the ICC compiler, or rebuild manually.

STL-67178

After boot, the OPX Software hfi1-NVIDIA support is not available.

Review boot configuration to ensure that nvidia-driver will be available when hfi1 is loaded.

STL-64974

Failures have been observed when running the IMB-EXT "Window" benchmark using the OPX Provider and ppn greater than (num_user_contexts/2).

To run the IMB-EXT benchmark with the OPX Provider and higher PPN, reload the hfi1 module with an increased num_user_contexts value. For example, if the server has 40 physical cores and you want to run one IMB-EXT process per core, increase the number of hfi1 contexts to 80. See the CN5000 Performance Tuning Guide, “HFI1 Driver Module Parameters” for instructions.

The benchmark can now be run with #contexts/2 = 40 PPN, one for each CPU core.

STL-74480

When a switch port operating in subdivided (2X) mode is disabled and re-enabled, it does not return to full-width 4X mode.

After replacing a bifurcated (Y-cable) connection with a straight QSFP cable, a switch power cycle is required to restore the port to 4X mode.

STL-74370

ASIC firmware update may fail with the following message:

Error during firmware update

Not that this message is seen when checking the status using the firmware update -s command.

Retry the ASIC firmware update.

STL-73361

On rare occasions, links connecting a CN5000 switch to an OPA100 adapter may unexpectedly bounce.

No action required, the link will self recover.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

For details, contact support@cornelis.com for technical advisory TA0061.