Skip to main content

Cornelis Technical Documentation

3.6.1. Verify Servers

Perform the following steps to verify servers:

  1. Use lspci to verify the SuperNIC PCIe cards' operating speed and bus width. Possible sources for narrow PCIe width are:

    • CN5000 SuperNIC adapters use a PCIe Gen5 x16 host interface and must be installed in a host slot that provides all 16 PCIe lanes and is explicitly configured for x16 operation in the BIOS. The link may negotiate a reduced width—such as x8—either because the physical slot routes fewer than 16 lanes, or because the platform BIOS defaults to a reduced or auto link width setting. On any platform where reduced-width operation is observed, verify that the BIOS PCIe slot width is explicitly set to x16, not Auto. The CN5000 dual-port configuration refers to two fabric ports sharing one host PCIe interface; it does not provide two independent x8 PCIe links.

    • SuperNIC Card partial insertion into x16 slots. Initially, this appears to be a narrow width issue but re-inserting the card often resolves the issue. This may occur after a server is shipped. This step has resolved most width issues.

    • Server physical configuration. Many servers support different PCIe logical widths based on riser card configuration. The slot may be physically x16 but internally limited to x8. Check other servers of the same configuration in the fabric. Check the server configuration. This is also a common issue.

    • Swap the SuperNIC to another server to determine if the problem follows the card or the server.

  2. Use the Linux top command to identify the key CPU load processes.

    Note

    opatop may be useful for checking for loads that vary over time. Use the r (rev), f (forward), and L (live) options to look through PM snapshots of system activity. This is also helpful for monitoring application startup versus run time loads. The PM captures high resolution statistics, with very low system overhead, over periods up to two days.

  3. Check for high CPU percent processes. For example:

    • Screen savers - when a Linux GUI is enabled on hosts, the screen that runs when the interface is idle may have a high CPU load.

    • Test applications - look for MPI jobs or similar applications running in the background. This is a common issue particularly in a shared fabric bring-up environment. Use kill -p process to stop orphan applications or reboot the server to debug the issue.

  4. Review BIOS settings to isolate nodes with different or incorrect settings.

3.6.1.1. Verifying SuperNIC Speed and Bus Width Using lspci

After the OPX Software installation, verify the CN5000 Omni-Path SuperNIC is configured and visible to the host OS as x16 slot speed (output should be similar to the following with values in bold text):

lspci -vv -d 434e: |grep LnkSta
    LnkSta:  Speed 32GT/s (ok), Width x16 (ok)
    LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-

Performance Manager/Performance Management - This is the Fabric Management entity responsible for monitoring fabric information related to the port and virtual lane level counters and the picture that they convey.

Message Passing Interface - This is a standardized means of exchanging messages between multiple computers running a parallel program across distributed memory