STL-76777

Using the network ip del command on a static IP address does not work.

12.2.0

STL-76486

Port configuration changes are not consistently applied when bouncing a link.

12.2.0

STL-75381

Prior to a reboot, the secondary Management Module (MM) may not reflect the current status of the image that should be booted after a BMC update.

12.2.0

STL-72221

Switch in-band management including configuring an IPoIB interface is not available. 

12.2.0

STL-78122

After restarting the Fabric Manager, the switch does not rejoin IPoIB multicast groups.

Restart IPoIB on the switch by disabling and re-enabling it:

network ib --disable
network ib --enable

STL-77842

nmap port scan of IPoIB address causes switch instability.

Do not run nmap port scan against the switch over the IPoIB network.

STL-77720

The switch IPoIB interface (ib0) fails to establish carrier and remains down when the fabric's default multicast group MTU is configured above 4096 bytes (for example, 10K MTU for Accelerated IP/AIP support).

None.

STL-77407

Unable to delete IPoIB IPv6 address after adding it.

None.

STL-76775

Ports that have been disabled may be re-enabled after a reboot of the switch.

Disable the ports again after a switch reboot.

STL-76677

When removing SSH keys, partial matches of the comment could remove more keys than intended.

When removing an SSH key, use the entire comment of the key.

STL-76490

Port configuration changes are not consistently applied when bouncing a link.

Explicitly disable the link and then enable the link for the changes to take effect.

STL-73311

If a server's SSH host identification (host key) is changed, such as by reinstalling the OS or routing a hostname to a different server, switch maintenance tasks run from the switch that use file transfers (such as firmware udpate and capture state copy) can fail with an error message about SSH usage.

Use scp for file transfers from the server into the switch.

STL-77756

After updating the SuperNIC firmware using updateAgent, one or more SuperNIC cards in a multi-card system may remain in a "Pending" firmware state following an AC power cycle. Additional AC power cycles do not resolve the issue.

Re-run the updateAgent firmware update on the affected SuperNIC card(s).

To help ensure all cards complete the firmware update on the first attempt, fully power off the system (not just a power cycle) and wait at least one minute before powering back on, allowing all card capacitors to fully discharge.

STL-75609

The updateAgent tool may become stuck when downgrading SuperNIC firmware.

Use updateAgent -f when downgrading firmware.

STL-73195

When updating the SuperNIC firmware using updateAgent -v, the following error may occur:

While waiting for a response to PLDM type 05 and command code 14, a message with type 05 and command 15 was received 

Retry the updateAgent -v command.

STL-72988

STL-72556

When updating firmware in a CN5000 SuperNIC in a management node, updateAgent times out when the FM is running at the same time.

Run the updateAgent with the -m option, which will stop the Fabric Manager if it is running, and then automatically start it when firmware update is finished if it had been running.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Update the SuperNIC Firmware” for the full procedure.

STL-78250

A Verbs SEND operation using PIO may fail to detect that send credits have been returned. This causes the send path to get stuck, preventing further send operations even though the hardware is ready.

Set piothreshold=0 in the HFI1 driver module parameters.

STL-78089

ROCm HIP pointer queries may cause a process hang during teardown. This is a known ROCm issue related to OpenMP target allocations recorded in https://github.com/ROCm/rocm-systems/issues/7099.

Use HIP-managed allocations (e.g., hipMalloc()) instead of OpenMP target allocations for buffers used with OPX GPU-direct/HMEM operations. Avoid mixing OpenMP target allocations with HIP pointer introspection until the ROCm issue is resolved upstream.

STL-77956

GPU applications hang with Open MPI 5.0.8 and earlier

Use Open MPI 5.0.9 or newer.

STL-77891

When installing or uninstalling the opxs-kernel-updates-kmp-default package on SLES 16, warnings appear stating that the RPM's post-install or post-uninstall scriptlet failed.

These warnings can be ignored.

STL-77873

Megatron-LM training (Llama3-8B) hangs on multi-node configurations when using NCCL_NET=OFI with the OPX Provider.

Use NCCL_NET=IB instead of NCCL_NET=OFI.

STL-77578

CableInfo errors encountered when using the opasmaquery -o cableinfo command with direct-routed packets.

Use LID-routed packets to get cable information.

STL-76942

Lustre errors occur with Bulk Transfer Service (BTS) enabled (hfi1 driver parameter use_bulksvc=Y).

When mounting Lustre, BTS should not be enabled (use_bulksvc=N, which is default).

STL-76907

The following Lustre Error may occur:

Write request failed with -14

NOTE: This is a general defect that affects all Lustre 2.17.0 stacks (refer to Lustre-Release Issue #16

Avoid this issue by disabling hybrid IO functionality using the following command on the MGS server:

lctl set_param -P llite.*.hybrid_io=0

STL-76435

opapmaquery fails with the following error when querying performance counters for more than approximately 14 ports simultaneously on a single switch.

Invalid Attribute or Attribute Modifier

NOTE: The PMA response has a maximum payload size that limits the number of ports that can be returned in a single query.

Split the port select mask (-n option) into multiple smaller queries, each covering a subset of ports.

For example, on a 48-port switch, query ports in groups of 20 or fewer:

opapmaquery -o getdatacounters -l <LID> -n 0x0000000FFFFF # ports 0-19

opapmaquery -o getdatacounters -l <LID> -n 0x00FFFFF00000 # ports 20-39

opapmaquery -o getdatacounters -l <LID> -n 0xFF0000000000 # ports 40-47

STL-76348

Running opapaquery -o focusport reports same neighbor LID if the switch is connected to a dual-port SuperNIC.

None.

STL-76323

SuperNICs may intermittently fail to enumerate on the PCIe bus or initialize after a cold boot or power cycle. In affected systems, one or more SuperNICs may be absent from the OS, preventing the node from participating in the fabric until the device is successfully detected and initialized.

Power cycle the server.

STL-76313

Kernel messages such as the following may be seen:

CPORT request wait interrupted

Dropping incoming CPORT message

Set the hfi1 module parameter cport_mad_to=5.

STL-76144

Setting the link speed policy to 100G in the opafm.xml file results in the following Fabric Manager error:

Invalid Value for Link Speed Policy: 100G

Set the link speed using the Switch CLI command port modify speed.

STL-76131

When using SR-IOV, the dmesg log in the VMs may occasionally show the following warning message:

memcpy: detected field-spanning write

Do not run Omni-Path management tools, such as opafabricinfo, on virtual machines. These tools should only be run on the host.

STL-75568

Performance test failures may occur when running ib_send_bw and ib_read_bw with the -D (duration) option.

Do not use the -D option.

STL-75535

GADGET4 MPI application hangs with OPX Provider.

Contact support@cornelis.com.

STL-75526

After installing the OPX Software package on Ubuntu, the opafm service is enabled by default.

Manually disable the opafm service by default.

STL-75484

Running opaportconfig with the disable option immediately followed by running opaportconfig with the enable option will result in the link staying down.

To drop and reinitialize a link, use the bounce option to opaportconfig.

Alternately, if opaportconfig is run from a script, insert a one second delay (sleep 1) between the disable and enable commands.

STL-75395

When AdaptiveRouting is enabled in the opafm.xml file, but FineGrained is not, the Fabric Manager will crash.

Disable AdaptiveRouting in the opafm.xml file and restart the Fabric Manager.

STL-75362

When using the TCP provider of libfabric, some MPI application may perform poorly or hang when launched with 1 process per core. This has been observed in benchmarks such as OSU Congestion, OSU Collectives, and IMB P2P.

For best results, users should use the OPX provider whenever available. OPX can be explicitly selected by setting the environment variable, FI_PROVIDER=opx.

STL-75322

Upon reverting from 12.0.3 to a previous RHEL/Rocky release certain packages may not be removed.

Use the following command to remove the remaining packages:

dnf remove ibacm infiniband-diags 
libfabric libhfi1verbs libibumad 
libibverbs libpsm2 librdmacm 
librdmacm-utils mpi-selector 
opa-basic-tools opa-fastfabric 
opa-fm opa-libopamgt rdma-core 
rdma-core-devel srp_daemon

STL-75031

STL-76644

MPI RMA benchmarks may produce an SDMA error (bad address) when using the OPX libfabric provider and environment variable FI_OPX_MIXED_NETWORK=0.

FI_OPX_MIXED_NETWORK is deprecated and should no longer be set. Contact support@cornelis.com.

STL-74968

Updating the Fabric Manager using DNF may not succeed even though an update is available.

STL-74933

Entries of the form hfi1_0: lost_mctxt_intr forcing int for 2 and MCTXT interrupt, but no status bits set may appear in logs.

These messages can be ignored.

STL-74679

While running IPoIB in Connected mode, application hangs have been observed.

Configure IPoIB to Datagram mode and rerun the application.

STL-74625

When using both ports of a dual-port SuperNIC, the ibstat rate field may show an incorrect value.

Use opainfo, which is the preferred method for querying port information.

STL-74477

Building opxs-kernel-updates from the src.rpm file fails with nvidia/nv-p2p.h not detected.

Use an environment variable to allow the header file to be detected and used in the build:

NVIDIA_DIR=$(ls -d /usr/src/nvidia-* | head -1) 
ln -s ${NVIDIA_DIR}/nvidia /usr/src/kernels/5.14.0-503.14.1.el9_5.x86_64/include/nvidia

Note that the exact directory required for the symbolic link will depend on your kernel version.

STL-74025

When using IPoIB on CN5000, SuperNICs do not change MTU or join MC groups automatically.

Use the ip tool from the command line to manually take the interface down and back up.

STL-73928

HFI links bounce and the SM reports mapping warnings following the second FM restart.

No Workaround. Links that bounce recover and the fabric stabilizes.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73716

The opatop and opapaquery commands may report congestion incorrectly when the fabric is idle.

Ignore the congestion flag.

STL-73714

Performing an opapmaquery to a "B" port on a switch where subdivision is not enabled returns non-zero counter values.

Counter values in this scenario should be ignored.

STL-73713

In a fabric with no data traffic, the Xmit Time Congestion port counter will increment.

No workaround.

STL-73669

Executing opasmaquery -o scvlr for nodes that do not support SC-VLr mappings will result in a message indicating a failure.

The message may be ignored.

STL-73531

Configuring more than two data VLs is not currently supported. Doing so may lead to incorrect mappings of SC to VL.

Limit the number of configured data VLs to two.

STL-73215

When running opasmaquery -o sminfo on CN5000 SuperNICs, SMA fields show all 0s and SM State Inactive.

Use opasaquery -o sminfo or opareport to show correct SA SMInfo data.

STL-73160

The comments in the Adaptive Routing section of the opafm.xml file do not explain which settings are required for MGAR or for FGAR.

Currently the only field related to FGAR is FineGrained. In order to enable FGAR, set the following:

<AdaptiveRouting>1</AdaptiveRouting> 

<FineGrained>1</FineGrained> 

All other fields are related to legacy MGAR and will be ignored if FineGrained is enabled.

STL-73140

Note the following:

STL-72673

OS-release file captured by opacapture is empty. 

Use cat /etc/os-release command.

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-72573

Hangs and errors have been seen in the OSU alltoall collective test when using the OPX Provider to libfabric with ROCm.

No workaround.

STL-72541

Running the opainfo command when ports are inactive results in the following errors: MAD failed and Failed to get PMA Port Status.

These errors can be ignored.

STL-72536

STL-72537

CableInfo errors encountered when using the opasmaquery -o cableinfo command.

Cable information reporting not currently supported.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

For details, contact support@cornelis.com for technical advisory TA0061.

STL-72420

On some versions of RHEL 9.x, there is a known failure (rdmavt driver deadlock.) recorded in https://issues.redhat.com/browse/RHEL-83851.

NOTE: Cornelis has not replicated this problem but it is possible that customers may be affected.

No workaround.

STL-72160

After uninstalling the meta packages with a package manager, several Cornelis Networks packages remain.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Remove OPX Software from a Host” for removal of the remaining packages.

STL-71364

STL-72423

Some legacy Omni-Path host utilities have been deprecated for CN5000 but are still supported for OPA100.   

For a list of commands supported for CN5000, refer to the Man Pages or CN5000 Commands Guide.

STL-67178

After boot, the OPX Software hfi1-NVIDIA support is not available.

Review boot configuration to ensure that nvidia-driver will be available when hfi1 is loaded.

STL-77756

After updating the SuperNIC firmware using updateAgent, one or more SuperNIC cards in a multi-card system may remain in a "Pending" firmware state following an AC power cycle. Additional AC power cycles do not resolve the issue.

Re-run the updateAgent firmware update on the affected SuperNIC card(s).

To help ensure all cards complete the firmware update on the first attempt, fully power off the system (not just a power cycle) and wait at least one minute before powering back on, allowing all card capacitors to fully discharge.

STL-76776

Network connectivity may fail after hot-swapping cables of different types on the SuperNIC.

Reboot the host to detect the new cables.

STL-76323

SuperNICs may intermittently fail to enumerate on the PCIe bus or initialize after a cold boot or power cycle. In affected systems, one or more SuperNICs may be absent from the OS, preventing the node from participating in the fabric until the device is successfully detected and initialized.

Power cycle the server.

STL-75609

The updateAgent tool may become stuck when downgrading SuperNIC firmware.

Use updateAgent -f when downgrading firmware.

STL-74502

When updating SuperNIC firmware with multiple SuperNIC adapters in a server, some cards may fail to properly update.

Disable the port on the failed SuperNIC cards using opaportconfig -p n disable (where “n” is the port number) and perform the update again.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73195

When updating the SuperNIC firmware using updateAgent -v, the following error may occur:

While waiting for a response to PLDM type 05 and command code 14, a message with type 05 and command 15 was received

Retry the updateAgent -v command.

STL-73088

On some dual-port devices, inbound IPoIB traffic may fail after a system reboot, or after restarting the FM and IPoIB interfaces on the FM node.

Ping out from the unreachable interface, and IP traffic will resume.

STL-72988

When updating firmware in a CN5000 SuperNIC in a management node, updateAgent times out when the FM is running at the same time.

Run the updateAgent with the -m option, which will stop the Fabric Manager if it is running, and then automatically start it when firmware update is finished if it had been running.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Update the SuperNIC Firmware” for the full procedure.

STL-72817

When reading hfi1 port counters from sysfs, you will receive an Invalid Argument return code.

This message can be ignored.

STL-72598

In some cases, when unloading hfi1 or ib_ipoib, the dmesg log may display an error stating:

hfi1_tid_rdma_flush_wait: no rcd for QP

This message can be safely ignored.

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-71647

On rare occasions, the link may come up at a width less than 4X.

Bounce the link.

STL-74370

ASIC firmware update may fail with the following message:

Error during firmware update

Not that this message is seen when checking the status using the firmware update -s command.

Retry the ASIC firmware update.

STL-73361

On rare occasions, links connecting a CN5000 switch to an OPA100 adapter may unexpectedly bounce.

No action required, the link will self recover.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

For details, contact support@cornelis.com for technical advisory TA0061.

STL-71647

On rare occasions, the link may come up at a width less than 4X.

Bounce the link.