7.4.2.5. Port Counters
Each port in a CN5000 Omni-Path Fabric maintains a set of port counters to indicate both traffic and error counts. These counters can be grouped into the categories described in this section. Each port stops incrementing when the max value is reached, irrespective of counter size. Most of the counters are 64-bits in size. Exceptions are noted.
7.4.2.5.1. Utilization
These counters reflect the normal utilization of the port and Virtual Lane (VL) when present.
Several of these counters are used during the calculation of Congestion, SMA Congestion, and the Bubble Categories. The Utilization metrics provide a way of giving some of the other counter's context by comparing them to the amount of data or packets that were transmitted or received.
PortXmitData (TxD) and PortVLXmitData[n]
These counters indicate the total number of fabric packet flits transmitted. This does not include idle nor other LF command flits.
PortRcvData (RxD) and PortVLRcvData[n]
These counters indicate the total number of fabric packet flits received.
PortMulticastXmitPkts (MTxP)
This counter indicates the number of multicast and collective packets transmitted.
PortMulticastRcvPkts (MRxP)
This counter indicates the number of multicast and collective packets received.
7.4.2.5.2. Link Integrity
These counters reflect errors in the Physical (PHY) and Link Layers, as well as errors in firmware. In some cases, these errors are benign and can be ignored. However, in other cases, excessive link integrity errors can indicate a hardware problem such as a poor connection, marginal cable, incorrect length/model cable for signal rate, or damaged/broken hardware, such as bad connectors.
When a bad packet is detected, one of these counters is incremented and the Link Layer may either discard or replay the packet.
During the link training sequence, assorted errors may be observed. This is a normal part of the link training and clock synchronization process. Hence, errors observed as part of rebooting nodes or moving cables should not be considered a problem.
The category is calculated as a weighted sum of the counters in the group, with the exception of ExcessiveBufferOverrunErrors. The counters report on the receive side of the link. However, the counter can indicate a problem on either side of the link.
Link Quality Indicator (LQI)
This is a status indicator, similar to the signal strength bar display on a mobile phone, that enumerates link quality as a range of 0-5, with 5 being very good. Values in the lower part of the range may indicate hardware problems with components such as ports and cables that surface as signal integrity issues, leading to performance and other problems. The LQI gives you an instantaneous view of a link's quality on every hardware port.
Link Quality Value | Description |
|---|---|
5 | Working at or above preferred link quality, no action needed. |
3 | Working at the low end of acceptable link quality, recommend corrective action on the next maintenance window. |
2 | Working below acceptable link quality, recommend timely corrective action. |
1 | Working far below acceptable link quality, recommend immediate corrective action. |
0 | Link down |
NoteCorrective action entails diagnosing the hardware (links/cables and ports/devices). For example: Are the cables bad or improperly placed? Is the SuperNIC/switch responsive? Does rebooting the device/server fix the issue? | |
LocalLinkIntegrityErrors (LLI) Counter
This counter indicates the number of retries initiated by a link transfer layer receiver.
The retry rate is represented by the Link Quality Indicator. A link that is meeting performance requirements has a Link Quality of 5, which corresponds to 1000 or fewer replays per second.
PortRcvErrors (RxE) Counter
This counter indicates the total number of packets containing an error that were received by the port, including Link Layer protocol violations and malformed packets. It indicates possible misconfiguration of a port, either by the Subnet Manager (SM) or by user intervention. It can also indicate hardware issues or extremely poor link signal integrity.
ExcessiveBufferOverrunErrors (EBO) Counter
This counter, associated with credit management, indicates an input buffer overrun. It indicates possible misconfiguration of a port, either by the SM or by user intervention. It can also indicate hardware issues or extremely poor link signal integrity.
LinkErrorRecovery (LER) Counter
This counter indicates the number of times the link has successfully completed the link error recovery process.
Link Quality Indicator is the primary indicator for link quality to use. This counter is factored into the value reported for Link Quality Indicator. This counter may be non-zero for a properly functioning link.
LinkDowned (LD) Counter
This counter indicates the total number of times the port has failed the link error recovery process and downed the link. These events can cause disruptions to fabric traffic.
UncorrectableErrors (Unc) Counter
This counter indicates the number of unrecoverable device errors. This may indicate a defect in the reporting device.
FMConfigErrors (FMC) Counter
This counter reports inconsistent configurations of the low-level SMA on either side of the link. It indicates possible misconfiguration of a port, either by the SM or by user intervention.
7.4.2.5.3. Congestion
These counters reflect possible errors that indicate traffic congestion in the fabric.
When congestion or a packet that has seen congestion is detected, one of these counters is incremented, and then depending on the issue reported, the packet must wait. In an extreme case, the packet may time out and be dropped.
The category is calculated as a weighted sum of the counters in the context of the utilization counters. With the exception of PortRcvFECN, the counters are all reported on the transmit side of the link. In addition, PortRcvBECN is only taken if the local node is a SuperNIC. However, the counter could indicate a problem on either side of the link.
CongDiscards (CD) Counter
Note
Formerly known as "SwPortCongestion".
This switch-only counter indicates the number of packets that were discarded as unable to transmit due to timeouts.
PortRcvFECN (RxF) Counter
When a device receives a packet with the Forward Explicit Congestion Notification (FECN) bit set to one, this counter is incremented.
PortRcvBECN (RxB) Counter
When a device receives a packet with the Backward Explicit Congestion Notification (BECN) bit set to one, this counter is incremented.
PortMarkFECN (MkF) Counter
This counter indicates the total number of packets that were marked Forward Explicit Congestion Notification (FECN) by the transmitter due to congestion.
PortXmitTimeCong (TxTC) Counter
This counter indicates the total number of flit times that the port was in a congested state for any data VL.
PortXmitWait (TxW) Counter
This counter indicates the amount of time (in flit times) any virtual lane had data but was unable to transmit due to no credits available.
7.4.2.5.4. SMA Congestion
These counters reflect congestion in the fabric specific to communication between the Subnet Manager and Subnet Manager Agents (SMA) using the management VL (VL 15).
The category is calculated exactly as the Congestion category using the same weights and the correct VL15 utilization counters.
PortVLXmitWait[15] (VLTxW[15]) Counter
This counter behaves the same as PortXmitWait, but it is restricted to VL 15, which carries only SM traffic.
VLCongDiscards[15] (VLCD[15]) Counter
Note
Formerly known as "SwPortVLCongestio.
This counter behaves the same as Cong Discards, but it is restricted to VL 15, which carries only SM traffic.
PortVLRcvFECN[15] (VLRxF[15]) Counter
This counter behaves the same as PortRcvFECN, but it is restricted to VL 15, which carries only SM traffic.
PortVLRcvBECN[15] (VLRxB[15]) Counter
This counter behaves the same as PortRcvBECN, but it is restricted to VL 15, which carries only SM traffic.
PortVLXmitTimeCong[15] (VLTxTC[15]) Counter
This counter behaves the same as PortXmitTimeCong, but it is restricted to VL 15, which carries only SM traffic.
PortVLMarkFECN[15] (VLMkF[15]) Counter
This counter behaves the same as PortMarkFECN, but it is restricted to VL 15, which carries only SM traffic.
7.4.2.5.5. Bubble
These counters occur when an unexpected idle flit is transmitted or received.
The transmit port sends idle flits until it can continue sending the rest of the packet. The category is calculated as follows:
The maximum value between the sum of the XmitWastedBW and XmitWaitData or the neighbor's PortRcvBubble.
Then divide the previous value by the port's utilization to provide context.
PortXmitWastedBW (WBW) Counter
This counter indicates the number of flit times where one or more packets have been started but the transmitters are forced to send idles due to bubbles in the ingress stream. Also, the VLs that have data to be sent are not permitted to preempt the currently transmitting VL.
PortXmitWaitData (TxWD) Counter
This counter indicates the number of flit times where one or more packets have been started but interrupted due to bubbles in the ingress stream.
PortRcvBubble (RxBb) Counter
This counter indicates the total number of flit times where one or more packets have started to be received, but the receiver received idle flits from the wire.
7.4.2.5.6. Security
These counters reflect possible security problems in the fabric.
Security problems can occur if a PKey or SLID violation occurs at the port during the ingress or egress of a packet.
The category is calculated as the sum of the neighbor's PortRcvConstraintErrors and the local port's PortXmitConstraintErrors.
PortRcvConstraintErrors (RxCE)
This counter is incremented when partition key or source LID violations are detected in a received packet, indicating a possible security issue or misconfiguration of device security settings.
PortXmitConstraintErrors (TxCE)
This counter is incremented when partition key violations are detected in a packet attempting to be transmitted, indicating a possible security issue or misconfiguration of device security settings.
7.4.2.5.7. Routing
These counters reflect possible routing issues. When a routing issue occurs, the offending packet is dropped.
A typical cause of this error is the routing to a wrong egress port or an improper Service Channel (SC) mapping. These errors can be a side effect of a port or device going down while traffic was still in flight to or through the given port or device.
PortRcvSwitchRelayErrors (RxSR)
This counter indicates the number of packets that were dropped due to internal routing errors. It indicates possible misconfiguration of a switch by the SM.
7.4.2.5.8. Other
These counters do not fit into any of the previous categories.
PortRcvRemotePhysicalErrors (RxRP)
This counter indicates the number of downstream effects of signal integrity (SI) problems. It indicates an SI issue in the upstream path.
This counter was not included as it does not directly indicate the link that had the issue, so it can be misleading.
PortXmitDiscards (TxDc)
This counter indicates the number of packets dropped due to several reasons including timeouts and improper packet lengths.
Note
This counter is a super set that includes Congestion Discards counter.