STL-74368

Aborting while creating a new user will create the user, but the user will not be accessible.

Delete the user and recreate as needed.

STL-73895

The capture state command fails with with Error running SCP command.

None.

STL-73885

The port show command does not display subdivided child ports.

Use the port config show command.

STL-73431

Pressing and holding Ctrl+C during command execution can cause the CLI to become non-responsive.

Press Ctrl+\ to close the session and connect again.

STL-73400

Rebooting the switch in a non-interactive SSH session may result in the following error:

EOFError: Ran out of input
Lost Communication with server

This error may be ignored. The switch will reboot successfully.

STL-73311

If a server's SSH host identification (host key) is changed, such as by reinstalling the OS or routing a hostname to a different server, switch maintenance tasks run from the switch that use file transfers (such as firmware udpate and capture state copy) can fail with an error message about SSH usage.

Use scp for file transfers from the server into the switch.

STL-73121

The OpenBMC Web User Interface is not supported. 

No workaround.

STL-72795

Commands showing the QSFP max temperature will show an invalid number for the QSFP max temperature if no cables are installed.

No workaround.

STL-72706

CLI hang occurs if CTRL+C is used during password entry for SCP for firmware update.

To recover, perform one of the following:

STL-72570

IPv4 addresses display incorrectly when using the opasmaquery command.

Use the switch CLI network command to find the switch IPv4 and IPv6 addresses.

STL-72273

Mistyped commands may cause the CLI to stop responding.

Recover by starting a new ssh session.

STL-72226

Switch CLI log command may take 6-8 minutes to complete.

No workaround.

STL-72221

Switch in-band management including configuring an IPoIB interface is not available. 

Use the switch's Ethernet or serial connections for switch management.

STL-75031

MPI RMA benchmarks may produce an SDMA error (bad address) when using the OPX libfabric provider and environment variable FI_OPX_MIXED_NETWORK=0.

Set FI_OPX_MIXED_NETWORK=1.

STL-74968

Updating the Fabric Manager using DNF may not succeed even though an update is available.

STL-74951

STL-74672

On rare occasions, application hangs with an SDMA engine error in DMSG have been observed.

None. Rerun the job.

STL-74933

Entries of the form hfi1_0: lost_mctxt_intr forcing int for 2 and MCTXT interrupt, but no status bits set may appear in logs.

These messages can be ignored.

STL-74696

When running an NVIDIA GPU Cloud (NGC) container such as the "NVIDIA HPC-Benchmarks" package, a segmentation fault may occur.

Changes are required to the OPX configure file to include the appropriate Linux kernel user API ("uapi") directory. Contact support@cornelis.com.

STL-74679

While running IPoIB in Connected mode, application hangs have been observed.

Configure IPoIB to Datagram mode and rerun the application.

STL-74625

When using both ports of a dual-port SuperNIC, the ibstat rate field may show an incorrect value.

Use opainfo, which is the preferred method for querying port information.

STL-74593

The Quantum Espresso application with Intel MPI may abort with SIGABRT when executed with a higher number of MPI ranks than available user HFI contexts and multiple endpoints per HFI context (FI_OPX_ENDPOINTS_PER_HFI_CONTEXT=2).

Set FI_OPX_ENDPOINTS_PER_HFI_CONTEXT=1 and reduce the number of MPI ranks to be less than or equal to the number of available HFI user contexts.

STL-74504

Occasional drops in bandwidth observed during microbenchmark testing.

Refer to the Performance Tuning Guide for platform-specific tuning.

STL-74477

Building opxs-kernel-updates from the src.rpm file fails with nvidia/nv-p2p.h not detected.

Use an environment variable to allow the header file to be detected and used in the build:

NVIDIA_DIR=$(ls -d /usr/src/nvidia-* | head -1) 
ln -s ${NVIDIA_DIR}/nvidia /usr/src/kernels/5.14.0-503.14.1.el9_5.x86_64/include/nvidia

Note that the exact directory required for the symbolic link will depend on your kernel version.

STL-74234

The mvapich2 packages provided with OPX Software should be ignored and not used.

Compile and use mvapich3 and/or mvapich4 (preferred) with CN5000 and libfabric/OPX provider.

STL-74218

STL-74230

IPoIB interfaces on CN5000 currently do not support failover with the Channel Bonding driver in the Linux kernel.

No workaround.

STL-74025

When using IPoIB on CN5000, SuperNICs do not change MTU or join MC groups automatically.

Use the ip tool from the command line to manually take the interface down and back up.

STL-73928

HFI links bounce and the SM reports mapping warnings following the second FM restart.

No Workaround. Links that bounce recover and the fabric stabilizes.

STL-73759

When the FM restarts, different LIDs may get assigned to the nodes.

None. However a second restart may reassign the LIDs back to the original values.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73716

The opatop and opapaquery commands may report congestion incorrectly when the fabric is idle.

Ignore the congestion flag.

STL-73714

Performing an opapmaquery to a "B" port on a switch where subdivision is not enabled returns non-zero counter values.

Counter values in this scenario should be ignored.

STL-73713

In a fabric with no data traffic, the Xmit Time Congestion port counter will increment.

No workaround.

STL-73709

The opasaquery -o trace -P (GUIDs) and opasaquery -o trace -G (GIDs) commands fail when targeting CN5000 SuperNIC cards.

No workaround.

STL-73669

Executing opasmaquery -o scvlr for nodes that do not support SC-VLr mappings will result in a message indicating a failure.

The message may be ignored.

STL-73531

Configuring more than two data VLs is not currently supported. Doing so may lead to incorrect mappings of SC to VL.

Limit the number of configured data VLs to two.

STL-73401

During link bring up on a mixed OPA100/CN5000 fabric the FM log may show Mismatch/Unable to set SCVL_t Map warnings and link bounces may be observed as the fabric comes up.

These messages and link bounces are expected on mixed fabrics and do not indicate a fault.

STL-73330

Some LS-DYNA R14.1.0. benchmarks hang with Open MPI when using the OPX Provider for libfabric.

Set the environment variable FI_OPX_RELIABILITY_SERVICE_USEC_MAX=5000.

STL-73267

When running the opasmaquery -o portinfo - g command, the output does not display a value for VLSchedulingConfig.

No workaround.

STL-73245

CN5000 Switch-Switch links do not appear in either opareport -o extlinks or opareport -o extislinks command output.

No workaround.

STL-73215

When running opasmaquery -o sminfo on CN5000 SuperNICs, SMA fields show all 0s and SM State Inactive.

Use opasaquery -o sminfo or opareport to show correct SA SMInfo data.

STL-73160

The comments in the Adaptive Routing section of the opafm.xml file do not explain which settings are required for MGAR or for FGAR.

Currently the only field related to FGAR is FineGrained. In order to enable FGAR, set the following:

<AdaptiveRouting>1</AdaptiveRouting> 

<FineGrained>1</FineGrained> 

All other fields are related to legacy MGAR and will be ignored if FineGrained is enabled.

STL-73140

Note the following:

STL-72864

High CPU utilization seen for hfi1 related interrupts, potentially impacting performance.

Use process pinning to avoid cores servicing hfi1 interrupts.

STL-72815

opapaquery -o vfConfig -V 'Default' may display duplicated port entries.

The duplicated entries can be ignored.

STL-72805

Enabling vFabrics with QoS results in Neighbor Mismatch/Unable to set SCVL_nt Map and Failed to init SL Maps in the log, and port bounces.

No workaround.

Port bounces are part of the standard recovery mechanism. These messages should only occur once and only when there is a change in the SCVL maps.

STL-72758

Cableinfo errors may appear in the log file.

To suppress Cannot get CableInfo errors in the system log, perform the following:

STL-72673

OS-release file captured by opacapture is empty. 

Use cat /etc/os-release command.

STL-72649

An Open MPI error may occur when using --mca btl self,vader and RMA-style functionality (for example, the OSU Put and Get test or IMB-RMA).

Contact support@cornelis.com.

STL-72622

opainfo displays a single port adapter as a dual port adapter.  

No workaround. This is informational.

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-72575

TID updates under heavy load may result in an -ENOMEM error.

Reload the hfi1 driver with the module parameter cache_size=1024.

STL-72573

Hangs and errors have been seen in the OSU alltoall collective test when using the OPX Provider to libfabric with ROCm.

No workaround.

STL-72555

opapaquery is not reporting the proper ISL link speed. 

Use opareport.

STL-72547

Class Port Info queries are not reporting properly.

No workaround.

STL-72541

Running the opainfo command when ports are inactive results in the following errors: MAD failed and Failed to get PMA Port Status.

These errors can be ignored.

STL-72536

STL-72537

CableInfo errors encountered when using the opasmaquery -o cableinfo command.

Cable information reporting not currently supported.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

The port number used by the FM is equivalent to twice the switch CLI port name minus one. Contact Cornelis Customer Support for details.

STL-72523

Errors occur on RHEL 9.5 after installation of meta package.

Refer to Disable the Repository After OPX Software Installation.

STL-72431

The command opasaquery -o node -n [NodeGUID] results in output of No Records Returned. 

Use opsmaquery -o nodeinfo -l 4, where l corresponds to the LID of the node.

STL-72420

On some versions of RHEL 9.x, there is a known failure (rdmavt driver deadlock.) recorded in https://issues.redhat.com/browse/RHEL-83851.

NOTE: Cornelis has not replicated this problem but it is possible that customers may be affected.

No workaround.

STL-72205

FI_OPX_HFI_SELECT=[] does not correctly map to the available adapter in the case in which other adapters in the system have been disabled.

Unload and reload the driver.

STL-72160

After uninstalling the meta packages with a package manager, several Cornelis Networks packages remain.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Remove OPX Software from a Host” for removal of the remaining packages.

STL-71364

STL-72423

Some legacy Omni-Path host utilities have been deprecated for CN5000 but are still supported for OPA100.   

For a list of commands supported for CN5000, refer to the Man Pages or CN5000 Commands Guide.

STL-70422

RCCL hangs on AMD GPUs with opx and aws-ofi-rccl plugin.

Use Verbs over OPX Provider when using RCCL.

STL-67178

After boot, the OPX Software hfi1-NVIDIA support is not available.

Review boot configuration to ensure that nvidia-driver will be available when hfi1 is loaded.

STL-66199

On RHEL 9.x, the opaswitchadmin command used to update unmanaged switch firmware fails with the following error:

Invalid MAD: Invalid Session ID
Error: Failed to get ini descriptors - status 3
Error: Failed to acquire node description - status -1
Invalid MAD: Invalid Session ID
Error: Failed to get ini descriptors - status 3

STL-74502

When updating SuperNIC firmware with multiple SuperNIC adapters in a server, some cards may fail to properly update.

Disable the port on the failed SuperNIC cards using opaportconfig -p n disable (where “n” is the port number) and perform the update again.

STL-74377

In rare cases, nodes may drop from fabric with unreachable CPORT when attempting to delete or reset an IPoIB interface.

Power cycle the affected node.

STL-73920

System Image GUID is the same on all SuperNICs.

No workaround.

STL-73738

Reloading the ipoib kernel module without also reloading the hfi1 module may cause FM errors.

To avoid this, always reload the hfi1 module along with the ipoib module.

STL-73709

The opasaquery -o trace -P (GUIDs) and opasaquery -o trace -G (GIDs) commands fail when targeting CN5000 SuperNIC cards.

No workaround.

STL-73440

Running the updateAgent while the Fabric Manager is running may result in a segfault.

Disable the Fabric Manager prior to running the updateAgent.

STL-73195

When updating the SuperNIC firmware using updateAgent -v, the following error may occur:

While waiting for a response to PLDM type 05 and command code 14, a message with type 05 and command 15 was received

Retry the updateAgent -v command.

STL-73153

The SuperNIC firmware version is currently displayed differently in different places, such as the system log versus the updateAgent.

Use updateAgent -V when checking the SuperNIC firmware version.

STL-73088

On some dual-port devices, inbound IPoIB traffic may fail after a system reboot, or after restarting the FM and IPoIB interfaces on the FM node.

Ping out from the unreachable interface, and IP traffic will resume.

STL-73075

lspci does not enumerate the SuperNIC with ASPM enabled, causing the BIOS screen on Intel D50DNP platforms to lock up.

Disable ASPM in the BIOS.

STL-72988

STL-72556

When updating firmware in a CN5000 SuperNIC in a management node, updateAgent times out when the FM is running at the same time.

Run the updateAgent with the -m option, which will stop the Fabric Manager if it is running, and then automatically start it when firmware update is finished if it had been running.

Refer to the CN5000 Maintenance and Troubleshooting Guide, “Update the SuperNIC Firmware” for the full procedure.

STL-72817

When reading hfi1 port counters from sysfs, you will receive an Invalid Argument return code.

This message can be ignored.

STL-72796

iperf2 may occasionally hang with an SDMA status error.

Set CONNECTED_MODE=yes in ifcfg.

See the CN5000 Performance Tuning Guide for more details.

STL-72598

In some cases, when unloading hfi1 or ib_ipoib, the dmesg log may display an error stating:

hfi1_tid_rdma_flush_wait: no rcd for QP

This message can be safely ignored.

STL-72583

In some cases the dmesg log may contain a message stating:

invalid logical state transition PORT_DOWN > PORT_ACTIVE.

This is cosmetic and will not affect functionality.

STL-71647

On rare occasions, the link may come up at a width less than 4X.

Bounce the link.

STL-74370

ASIC firmware update may fail with the following message:

Error during firmware update

Not that this message is seen when checking the status using the firmware update -s command.

Retry the ASIC firmware update.

STL-73399

After rebooting the switch ASIC, port commands may show invalid information until the switch ASIC has completed initialization.

Allow time for the ASIC to complete initialization, then re-run the command.

STL-73361

On rare occasions, links connecting a CN5000 switch to an OPA100 adapter may unexpectedly bounce.

No action required, the link will self recover.

STL-72526

The Fabric Manager system log port numbering is different from Switch CLI command port numbering.

The port number used by the FM is equivalent to twice the switch CLI port name minus one. Contact Cornelis Customer Support for details.

STL-71647

On rare occasions, the link may come up at a width less than 4X.

Bounce the link.