STL-77092

Liquid-cooled switches running firmware BMC 12.1.1.0.9 and ASIC 12_1_1_0_6 display Fault detected in VR errors and Board hardware status: NOT_GOOD warnings during hardware checks.

12.1.1.1

STL-75426

When custom users are added to the DCS, they are not synchronized between the two Management Modules (MM).

12.1.0

STL-75420

When updating the BMC firmware on the DCS, if the selected firmware image number (1 or 2) of a board is not set to same firmware image number as the primary MM, then the firmware update will not propagate to the out of sync board/boards.

12.1.0

STL-75396

If the Real-Time Clock (RTC) has lost power, then the Network Time Protocol (NTP) cannot be set.

12.1.2

STL-75258

When running the switch CLI capture state take command, a message states to wait for 5 minutes for the statedump to complete. However, the statedump may take longer than 5 minutes.

12.1.0

STL-75248

When rebooting individual cards using the switch CLI reboot command, that CLI session can no longer connect to the chassis hardware.

12.1.1

STL-75223

The LED colors and statuses reported by switch CLI hardware led command do not match the physical LEDs.

12.1.0

STL-74875

When executing network chassis gateway <IP Address>, the command will set the chassis IP instead of a gateway. The chassis does not support setting a gateway.

12.1.0

STL-74852

The switch CLI watch command does not operate properly for DCS spine and leaf cards.

12.1.1

STL-74845

The switch CLI hardware led command does not show the state of the LEDs in a DCS.

12.1.0

STL-74450

Setting the NTP server on the primary Management Module (MM) does not propagate to the secondary MM.

12.1.0

STL-74386

When a new user is created within the admin group and you switch to the new user, they will not have access to the admin commands.

12.1.0

STL-74368

Aborting while creating a new user will create the user, but the user will not be accessible.

12.1.0

STL-74363

The beacon command does not cause the beacon LED to flash.

12.1.0

STL-73895

The capture state command fails with with Error running SCP command.

12.1.0

STL-73885

The port show command does not display subdivided child ports.

12.1.0

STL-73477

The capture port command fails with an Invalid port name error.

12.1.0

STL-73431

Pressing and holding Ctrl+C during command execution can cause the CLI to become non-responsive.

12.1.0

STL-73400

Rebooting the switch in a non-interactive SSH session may result in the following error:

EOFError: Ran out of input
Lost Communication with server

12.1.0

STL-73355

In the port show switch command, TX Packet Count and RX Packet Count display inaccurate values.

12.1.0

STL-73029

BEL DC power supplies VOUT_MODE command returns incorrect results.

12.1.0

STL-72795

Commands showing the QSFP max temperature will show an invalid number for the QSFP max temperature if no cables are installed.

12.1.0

STL-72706

CLI hang occurs if CTRL+C is used during password entry for SCP for firmware update.

12.1.0

STL-72570

IPv4 addresses display incorrectly when using the opasmaquery command.

12.1.0

STL-72273

Mistyped commands may cause the CLI to stop responding.

12.1.0

STL-72226

Switch CLI log command may take 6-8 minutes to complete.

12.1.0

STL-76777

Using the network ip del command on a static IP address does not work.

Reboot the BMC to clear the static IP.

STL-76775

Ports that have been disabled may be re-enabled after a reboot of the switch.

Disable the ports again after a switch reboot.

STL-76677

When removing SSH keys, partial matches of the comment could remove more keys than intended.

When removing an SSH key, use the entire comment of the key.

STL-76490

STL-76486

Port configuration changes are not consistently applied when bouncing a link.

Explicitly disable the link and then enable the link for the changes to take effect.

STL-75381

Prior to a reboot, the secondary Management Module (MM) may not reflect the current status of the image that should be booted after a BMC update.

Prior to a reboot of the secondary MM, select the image to be booted using firmware select bmc <1 or 2>.

STL-73311

If a server's SSH host identification (host key) is changed, such as by reinstalling the OS or routing a hostname to a different server, switch maintenance tasks run from the switch that use file transfers (such as firmware udpate and capture state copy) can fail with an error message about SSH usage.

Use scp for file transfers from the server into the switch.

STL-72221

Switch in-band management including configuring an IPoIB interface is not available. 

Use the switch's Ethernet or serial connections for switch management.

STL-77756

After updating the SuperNIC firmware using updateAgent, one or more SuperNIC cards in a multi-card system may remain in a "Pending" firmware state following an AC power cycle. Additional AC power cycles do not resolve the issue.

Re-run the updateAgent firmware update on the affected SuperNIC card(s).

To help ensure all cards complete the firmware update on the first attempt, fully power off the system (not just a power cycle) and wait at least one minute before powering back on, allowing all card capacitors to fully discharge.

STL-75609

The updateAgent tool may become stuck when downgrading SuperNIC firmware.

Use updateAgent -f when downgrading firmware.

STL-73195

When updating the SuperNIC firmware using updateAgent -v, the following error may occur:

While waiting for a response to PLDM type 05 and command code 14, a message with type 05 and command 15 was received 

Retry the updateAgent -v command.

STL-72988

STL-72556

When updating firmware in a CN5000 SuperNIC in a management node, updateAgent times out when the FM is running at the same time.

Run the updateAgent with the -m option, which will stop the Fabric Manager if it is running, and then automatically start it when firmware update is finished if it had been running.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Update the SuperNIC Firmware” for the full procedure.

STL-77956

GPU applications hang with Open MPI 5.0.8 and earlier

Use Open MPI 5.0.9 or newer.

STL-77873

Megatron-LM training (Llama3-8B) hangs on multi-node configurations when using NCCL_NET=OFI with the OPX Provider.

Use NCCL_NET=IB instead of NCCL_NET=OFI.

STL-77578

CableInfo errors encountered when using the opasmaquery -o cableinfo command with direct-routed packets.

Use LID-routed packets to get cable information.

STL-76942

Lustre errors occur with Bulk Transfer Service (BTS) enabled (hfi1 driver parameter use_bulksvc=Y).

When mounting Lustre, BTS should not be enabled (use_bulksvc=N, which is default).

STL-76907

The following Lustre Error may occur:

Write request failed with -14

NOTE: This is a general defect that affects all Lustre 2.17.0 stacks (refer to Lustre-Release Issue #16

Avoid this issue by disabling hybrid IO functionality using the following command on the MGS server:

lctl set_param -P llite.*.hybrid_io=0

STL-76435

opapmaquery fails with the following error when querying performance counters for more than approximately 14 ports simultaneously on a single switch.

Invalid Attribute or Attribute Modifier

NOTE: The PMA response has a maximum payload size that limits the number of ports that can be returned in a single query.

Split the port select mask (-n option) into multiple smaller queries, each covering a subset of ports.

For example, on a 48-port switch, query ports in groups of 20 or fewer:

opapmaquery -o getdatacounters -l <LID> -n 0x0000000FFFFF # ports 0-19

opapmaquery -o getdatacounters -l <LID> -n 0x00FFFFF00000 # ports 20-39

opapmaquery -o getdatacounters -l <LID> -n 0xFF0000000000 # ports 40-47

STL-76348

Running opapaquery -o focusport reports same neighbor LID if the switch is connected to a dual-port SuperNIC.

None.

STL-76323

SuperNICs may intermittently fail to enumerate on the PCIe bus or initialize after a cold boot or power cycle. In affected systems, one or more SuperNICs may be absent from the OS, preventing the node from participating in the fabric until the device is successfully detected and initialized.

Power cycle the server.

STL-76313

Kernel messages such as the following may be seen:

CPORT request wait interrupted

Dropping incoming CPORT message

Set the hfi1 module parameter cport_mad_to=5.

STL-76144

Setting the link speed policy to 100G in the opafm.xml file results in the following Fabric Manager error:

Invalid Value for Link Speed Policy: 100G

Set the link speed using the Switch CLI command port modify speed.

STL-76137

Open MPI traffic fails on SR-IOV Virtual Function devices.

Use the oversubscribe option.

STL-76131

When using SR-IOV, the dmesg log in the VMs may occasionally show the following warning message:

memcpy: detected field-spanning write

Do not run Omni-Path management tools, such as opafabricinfo, on virtual machines. These tools should only be run on the host.

STL-75568

Performance test failures may occur when running ib_send_bw and ib_read_bw with the -D (duration) option.

Do not use the -D option.

STL-75535

GADGET4 MPI application hangs with OPX Provider.

Contact support@cornelis.com.

STL-75526

After installing the OPX Software package on Ubuntu, the opafm service is enabled by default.

Manually disable the opafm service by default.

STL-75484

Running opaportconfig with the disable option immediately followed by running opaportconfig with the enable option will result in the link staying down.

To drop and reinitialize a link, use the bounce option to opaportconfig.

Alternately, if opaportconfig is run from a script, insert a one second delay (sleep 1) between the disable and enable commands.

STL-75395

When AdaptiveRouting is enabled in the opafm.xml file, but FineGrained is not, the Fabric Manager will crash.

Disable AdaptiveRouting in the opafm.xml file and restart the Fabric Manager.

STL-75362

When using the TCP provider of libfabric, some MPI application may perform poorly or hang when launched with 1 process per core. This has been observed in benchmarks such as OSU Congestion, OSU Collectives, and IMB P2P.

For best results, users should use the OPX provider whenever available. OPX can be explicitly selected by setting the environment variable, FI_PROVIDER=opx.

STL-75322

Upon reverting from 12.0.3 to a previous RHEL/Rocky release certain packages may not be removed.

Use the following command to remove the remaining packages:

dnf remove ibacm infiniband-diags 
libfabric libhfi1verbs libibumad 
libibverbs libpsm2 librdmacm 
librdmacm-utils mpi-selector 
opa-basic-tools opa-fastfabric 
opa-fm opa-libopamgt rdma-core 
rdma-core-devel srp_daemon

STL-75031

STL-76644

MPI RMA benchmarks may produce an SDMA error (bad address) when using the OPX libfabric provider and environment variable FI_OPX_MIXED_NETWORK=0.

FI_OPX_MIXED_NETWORK is deprecated and should no longer be set. Contact support@cornelis.com.

STL-74968

Updating the Fabric Manager using DNF may not succeed even though an update is available.

STL-74933

Entries of the form hfi1_0: lost_mctxt_intr forcing int for 2 and MCTXT interrupt, but no status bits set may appear in logs.

These messages can be ignored.

STL-74679

While running IPoIB in Connected mode, application hangs have been observed.

Configure IPoIB to Datagram mode and rerun the application.

STL-74625

When using both ports of a dual-port SuperNIC, the ibstat rate field may show an incorrect value.

Use opainfo, which is the preferred method for querying port information.

STL-74504

Occasional drops in bandwidth observed during microbenchmark testing.

Refer to the Performance Tuning Guide for platform-specific tuning.

STL-74477

Building opxs-kernel-updates from the src.rpm file fails with nvidia/nv-p2p.h not detected.

Use an environment variable to allow the header file to be detected and used in the build:

NVIDIA_DIR=$(ls -d /usr/src/nvidia-* | head -1) 
ln -s ${NVIDIA_DIR}/nvidia /usr/src/kernels/5.14.0-503.14.1.el9_5.x86_64/include/nvidia

Note that the exact directory required for the symbolic link will depend on your kernel version.

STL-74025

When using IPoIB on CN5000, SuperNICs do not change MTU or join MC groups automatically.

Use the ip tool from the command line to manually take the interface down and back up.

STL-73928

HFI links bounce and the SM reports mapping warnings following the second FM restart.

No Workaround. Links that bounce recover and the fabric stabilizes.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73716

The opatop and opapaquery commands may report congestion incorrectly when the fabric is idle.

Ignore the congestion flag.

STL-73714

Performing an opapmaquery to a "B" port on a switch where subdivision is not enabled returns non-zero counter values.

Counter values in this scenario should be ignored.

STL-73713

In a fabric with no data traffic, the Xmit Time Congestion port counter will increment.

No workaround.

STL-73669

Executing opasmaquery -o scvlr for nodes that do not support SC-VLr mappings will result in a message indicating a failure.

The message may be ignored.

STL-73531

Configuring more than two data VLs is not currently supported. Doing so may lead to incorrect mappings of SC to VL.

Limit the number of configured data VLs to two.

STL-73215

When running opasmaquery -o sminfo on CN5000 SuperNICs, SMA fields show all 0s and SM State Inactive.

Use opasaquery -o sminfo or opareport to show correct SA SMInfo data.

STL-73160

The comments in the Adaptive Routing section of the opafm.xml file do not explain which settings are required for MGAR or for FGAR.

Currently the only field related to FGAR is FineGrained. In order to enable FGAR, set the following:

<AdaptiveRouting>1</AdaptiveRouting> 

<FineGrained>1</FineGrained> 

All other fields are related to legacy MGAR and will be ignored if FineGrained is enabled.

STL-73140

Note the following:

STL-72673

OS-release file captured by opacapture is empty. 

Use cat /etc/os-release command.

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-72573

Hangs and errors have been seen in the OSU alltoall collective test when using the OPX Provider to libfabric with ROCm.

No workaround.

STL-72541

Running the opainfo command when ports are inactive results in the following errors: MAD failed and Failed to get PMA Port Status.

These errors can be ignored.

STL-72536

STL-72537

CableInfo errors encountered when using the opasmaquery -o cableinfo command.

Cable information reporting not currently supported.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

For details, contact support@cornelis.com for technical advisory TA0061.

STL-72523

Errors occur on RHEL 9.5 after installation of meta package.

Refer to Disable the Repository After OPX Software Installation.

STL-72420

On some versions of RHEL 9.x, there is a known failure (rdmavt driver deadlock.) recorded in https://issues.redhat.com/browse/RHEL-83851.

NOTE: Cornelis has not replicated this problem but it is possible that customers may be affected.

No workaround.

STL-72205

FI_OPX_HFI_SELECT=[] does not correctly map to the available adapter in the case in which other adapters in the system have been disabled.

Unload and reload the driver.

STL-72160

After uninstalling the meta packages with a package manager, several Cornelis Networks packages remain.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Remove OPX Software from a Host” for removal of the remaining packages.

STL-71364

STL-72423

Some legacy Omni-Path host utilities have been deprecated for CN5000 but are still supported for OPA100.   

For a list of commands supported for CN5000, refer to the Man Pages or CN5000 Commands Guide.

STL-67178

After boot, the OPX Software hfi1-NVIDIA support is not available.

Review boot configuration to ensure that nvidia-driver will be available when hfi1 is loaded.

STL-77756

After updating the SuperNIC firmware using updateAgent, one or more SuperNIC cards in a multi-card system may remain in a "Pending" firmware state following an AC power cycle. Additional AC power cycles do not resolve the issue.

Re-run the updateAgent firmware update on the affected SuperNIC card(s).

To help ensure all cards complete the firmware update on the first attempt, fully power off the system (not just a power cycle) and wait at least one minute before powering back on, allowing all card capacitors to fully discharge.

STL-76776

Network connectivity may fail after hot-swapping cables of different types on the SuperNIC.

Reboot the host to detect the new cables.

STL-76323

SuperNICs may intermittently fail to enumerate on the PCIe bus or initialize after a cold boot or power cycle. In affected systems, one or more SuperNICs may be absent from the OS, preventing the node from participating in the fabric until the device is successfully detected and initialized.

Power cycle the server.

STL-75609

The updateAgent tool may become stuck when downgrading SuperNIC firmware.

Use updateAgent -f when downgrading firmware.

STL-74502

When updating SuperNIC firmware with multiple SuperNIC adapters in a server, some cards may fail to properly update.

Disable the port on the failed SuperNIC cards using opaportconfig -p n disable (where “n” is the port number) and perform the update again.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73195

When updating the SuperNIC firmware using updateAgent -v, the following error may occur:

While waiting for a response to PLDM type 05 and command code 14, a message with type 05 and command 15 was received

Retry the updateAgent -v command.

STL-73088

On some dual-port devices, inbound IPoIB traffic may fail after a system reboot, or after restarting the FM and IPoIB interfaces on the FM node.

Ping out from the unreachable interface, and IP traffic will resume.

STL-72988

When updating firmware in a CN5000 SuperNIC in a management node, updateAgent times out when the FM is running at the same time.

Run the updateAgent with the -m option, which will stop the Fabric Manager if it is running, and then automatically start it when firmware update is finished if it had been running.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Update the SuperNIC Firmware” for the full procedure.

STL-72817

When reading hfi1 port counters from sysfs, you will receive an Invalid Argument return code.

This message can be ignored.

STL-72598

In some cases, when unloading hfi1 or ib_ipoib, the dmesg log may display an error stating:

hfi1_tid_rdma_flush_wait: no rcd for QP

This message can be safely ignored.

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-71647

On rare occasions, the link may come up at a width less than 4X.

Bounce the link.

STL-74370

ASIC firmware update may fail with the following message:

Error during firmware update

Not that this message is seen when checking the status using the firmware update -s command.

Retry the ASIC firmware update.

STL-73361

On rare occasions, links connecting a CN5000 switch to an OPA100 adapter may unexpectedly bounce.

No action required, the link will self recover.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

For details, contact support@cornelis.com for technical advisory TA0061.

STL-71647

On rare occasions, the link may come up at a width less than 4X.

Bounce the link.