Skip to main content

Cornelis Technical Documentation

7.2.3. Common Fabric Monitoring Tools

This section describes the common fabric monitoring tools and commands to check cluster and software status.

7.2.3.1. opafabricinfo

Use opafabricinfo to monitor fabric components, using the first active port on the given local host to perform its analysis.

From this command, you can determine information such as which host is running the primary FM and how many inter-switch links (ISLs) there are.

# opafabricinfo
Fabric 0:0 Information:
SM: opahsx151 hfi1_0 Guid: 0xd0066a0101671d2c
State: Master
Number of HFIs: 30
Number of Switches: 2
Number of Links: 33
Number of HFI Links: 30             (Internal: 0   External: 30)
Number of ISLs: 3                   (Internal: 0   External: 3)
Number of Degraded Links: 0         (HFI Links: 0   ISLs: 0)
Number of Omitted Links: 0          (HFI Links: 0   ISLs: 0)
-------------------------------------------------------------------------------

For additional details, refer to CN5000 Commands Guide, opafabricinfo.

7.2.3.2. opareport

Use opareport to generate reports about the current state of the fabric and output snapshot files for later use in debugging.

Common issues to analyze are:

  • Cable health, including link quality

  • Nodes that are Inactive, Priority, or Elevated Priority when controlling failover for SM, PM

The following provides examples for looking into cable health, through errors and slow links.

Note

Before running opareport to look for errors (-o), you should clear all the counters using opareport -o none --clearall or opareport -o none -Ca.

opahsx41:~ # opareport -o none --clearall
Getting All Node Records...
Done Getting All Node Records
Done Getting All Link Records
Done Getting All Cable Info Records
Done Getting All SM Info Records
Done Getting vFabric Records
Clearing Port Counters

Configured Counters to Clear:
XmitData
RcvData
XmitPkts
RcvPkts
MulticastXmitPkts
MulticastRcvPkts
UncorrectableErrors
LinkDowned
RcvErrors
ExcessiveBufferOverruns
FMConfigErrors
LinkErrorRecovery
LocalLinkIntegrityErrors
RcvRemotePhysicalErrors
XmitConstraintErrors
RcvConstraintErrors
RcvSwitchRelayErrors
XmitDiscards
CongDiscards
RcvFECN
RcvBECN
MarkFECN
XmitTimeCong
XmitWait
XmitWastedBW
XmitWaitData
RcvBubble
Clearing Port Counters...
Done Clearing Port Counters
Cleared 68 Ports on 32 Nodes

  • To check for cable errors and slow links, use opareport -o errors -o slowlinks.

    Cables with LinkQualityIndicator (LQI) "3" or less should be troubleshot or replaced. For more information on LQI, refer to Link Quality Indicator (LQI).

    opareport -o errors
Getting All Node Records...
Done Getting All Node Records
Done Getting All Link Records
Done Getting All Cable Info Records
Done Getting All SM Info Records
Done Getting vFabric Records
Getting All Port Counters...
Done Getting All Port Counters
Links with errors > threshold Summary

Configured Thresholds:
  LinkQualityIndicator           3
   UncorrectableErrors            1
   LinkDowned                     3
   RcvErrors                      1
   ExcessiveBufferOverruns        1
   FMConfigErrors                 1
   XmitConstraintErrors           10
   RcvConstraintErrors            10
   CongDiscards                   100
Rate NodeGUID          Port Type Name
400g 0xd0066a010170c572   1 FI   ime06 hfi1_0
   LinkQualityIndicator: 3 Below Threshold: 4
<->  0xd0066a01020c4bdb  10 SW  sw_ddn_r25u41
17688 of 17688 Links Checked, 1 Errors found

  • To check for slow links (links on which lanes, either RX or TX, are running at speed less than "4"), use opareport -o slowlinks.

    The following example shows two ends of a cable with “3” specifying one degraded lane TX/RX.

    opareport -o slowlinks
Getting All Node Records...
Done Getting All Node Records
Done Getting All Link Records
Done Getting All Cable Info Records
Done Getting All SM Info Records
Done Getting vFabric Records
Links running slower than expected Summary

Links running slower than expected:
Rate NodeGUID          Port Type Name
    Active                              Enabled
    Lanes, Used(Tx), Used(Rx), Rate,    Lanes,   DownTo,  Rates
-------------------------------------------------------------------------------
400g 0xd0066a010170b5a9   1 FI   ddn-mon02 hfi1_0
    4     3         4         100Gb     4        3,4      100Gb
<->  0xd0066a01020c4b3e  15 SW  sw_ddn_r25u42
    4     4         3         100Gb     1,2,3,4  3,4      100Gb

For additional details and the list of available report types, refer to CN5000 Commands Guide, opareport.

7.2.3.3. opainfo

Use opainfo to report on the status of the local SuperNICs.

The following provides an example output for a single port SuperNIC.

$ opainfo
hfi1_0:1                           PortGID:0xfe80000000000000:d0066a0101575fec
   PortState:     Active
   LinkSpeed      Act: 100Gb         En: 100Gb
   LinkWidth      Act: 4            En: 4
   LinkWidthDnGrd ActTx: 4  Rx: 4   En: 3,4
   LCRC           Act: 14-bit       En: 14-bit,16-bit,48-bit       Mgmt: True
   LID: 0x00000001-0x00000001       SM LID: 0x00000001 SL: 0
   QSFP: PassiveCu,   2m TE Connectivity   P/N 2821076-2         Rev B
   Xmit Data:                  5 MB Pkts:                28742
   Recv Data:                 17 MB Pkts:                28969
   Link Quality: 5 (Excellent)

For additional details, refer to CN5000 Commands Guide, opainfo.