7.4.1.4. Mitigating Link Issues
For certain Fabric Manager configurations, you can allow for either more reliability or better performance during production runs.
The following sections describe the cost and benefits of changing certain Fabric Manager configuration options.
7.4.1.4.1. LinkPolicy
Several Link Policy options are available for both SuperNIC and ISL links. For each link type, the following options are included:
MaxDroppedLanes sets the number of lanes on a link that the port can downgrade before bouncing itself and bringing up the link again. This value alters the LinkWidthDowngradeEnabled field on the port's portinfo.
Changing the value of MaxDroppedLanes to 0 will prevent a link from ever operating in a downgraded mode. Instead, it will program an option on the port that will trigger the port to bounce itself instead of downgrading.
WidthPolicy prevents the link activation of ports when the minimum configured link width is not met. Do not change this value without significant knowledge of what it will do.
Important
When forcing a link policy change using the switch CLI, you will need to bounce the link from the switch side.
7.4.1.4.2. Timeouts
Several timeout mechanisms are available in the SM; some are replicated in the PM. This section describes the configuration options for per packet timeouts and cumulative sweep timeouts, including what effects will occur when you change them.
Per Packet Timeouts
Three main options are used for dealing with Per Packet Timeouts in both the SM and the PM:
MaxAttempts
RespTimeout
MinRespTimeout
The following two modes use the options above to manage timeouts:
The Increasing Timeouts mode is the default mode. When a packet is sent, it will wait up to the non-zero MinRespTimeout value (35 ms). If it times out, the packet will continue to retry at increasing multiples of MinRespTimeout until the timeout exceeds MaxAttrempts multiplied by RespTimeout. For example, the first timeout is 35 ms, the second will be 70 ms, and so on.
The Exact Timeout mode requires MinRespTimeout to be set to zero. When this is set, each packet will wait for RespTimeout and retry MaxAttempts.
Cumulative Sweep Timeouts
Cumulative Sweep Timeouts using the CumulativeTimeoutLimit option are available only in the SM. It is used to prevent too many nodes from timing out in one sweep, potentially extending a sweep significantly. The CumulativeTimeoutLimit value is in seconds and, once exceeded, will start to skip nodes instead of retrying. This value will allow for longer timeouts without also increasing the sweep time when multiple ports are experiencing errors.
7.4.1.4.3. Adaptive Routing LostRoutesOnly
When Adaptive Routing is enabled, rerouting for a port going down is always enabled. LostRouteOnly disables the congestion-based rerouting. When a port goes down, to prevent packets from being dropped, the switch will automatically reroute traffic when it finds a different route. For information about how to set up adaptive routing, refer to CN5000 Topologies and Routing Guide, Adaptive Routing.
When LostRouteOnly is disabled, the switch also reroutes traffic when a specific congestion threshold is reached.
Though setting LostRouteOnly may not ease congestion, it will allow failures to be more optimally handled.
7.4.1.4.4. PortErrorAction
PortErrorAction is an SM configuration option that is programmed on the ports, directing them to automatically bounce if certain errors occur. These errors are listed in the configuration file.
The benefit of this option is that some issues can only be fixed by bouncing the port; if not bounced, the port may hang which could block traffic until fixed. Conversely, bouncing the port will interrupt traffic; and, if the issue was not causing the port to stall, bouncing may have been unnecessary.
7.4.1.4.5. Ports Speed
Switch ports may appear offline or down when an incorrect port speed is set on the link. Ensure that the attached cable supports the speed by using the Switch CLI command hardware cable.
Example output:
root@CNEdge:~# hardware cable Port Cable Vendor Cable Part Rv Serial Spd sup Temp Name Type Name Length Number Number 25G 100G (C) ------- ----------- ---------------- ------ ---------------- -- ---------------- --- ---- ----- 1A QSFP ACC Amphenol 5.0m NJAAL6-CN05 A APF2442N052U17 Y Y 0.0 3A QSFP ACC Amphenol 3.0m NJAALR-CN03 A APF2445N034R1L Y Y 0.0 5A QSFP ACC Amphenol 3.0m NJYKLR-CN03 A APF2446N0374E7 Y Y 0.0 6A QSFP ACC Amphenol 3.0m NJYKLR-CN03 A APF2446N0374E7 Y Y 0.0 7A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H700TR Y N 38.0 8A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H7005V Y N 35.0 9A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H7006H Y N 35.0 11A QSFP ACC Amphenol 4.0m NJAAL1-CN04 A APF2520N04W98A Y Y 0.0 12A QSFP ACC Amphenol 5.0m NJAAL1-CN05 A APF2509N054G63 Y Y 0.0 13A QSFP ACC Amphenol 5.0m NJARL1-CN05 A APF2509N054G9R Y Y 0.0 14A QSFP DAC Amphenol 2.0m NJAAK3-CN02 A APF2509N024G4B Y Y 15A QSFP ACC Amphenol 5.0m NJAAL6-CN05 A APF2442N052U1J Y Y 0.0 16A QSFP AOC FINISAR CORP. 10.0m FCBR4X0QE1C10COR A0 YV936KD N Y 48.5 17A QSFP ACC Amphenol 4.0m NJAAL6-0004 A APF23430047K2Y Y Y 0.0 18A QSFP AOC FINISAR CORP. 10.0m FCBR4X0QE1C10 A0 YV8GRZ0 N Y 48.4 19A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H7003K Y N 38.0 21A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H7002M Y N 37.0 22A QSFP ACC Amphenol 5.0m NJAAL6-CN05 A APF2442N052U16 Y Y 0.0 24A QSFP ACC Amphenol 5.0m NJAAL1-CN05 A APF2509N054G66 Y Y 0.0 25A QSFP DAC Amphenol 2.0m NJAAK3-CN02 A APF2509N024G4C Y Y 26A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H70081 Y N 39.0 27A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H700AJ Y N 39.0 28A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H700P0 Y N 35.0 29A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H7004Q Y N 39.0 30A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H7002C Y N 39.0 31A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H7005X Y N 30.0 32A QSFP AOC Hisense 15.0m DQF8503-4C15 02 S5195AF000M Y N 35.0 33A QSFP DAC FCI Electronics 1.0m 10131941-2010LF 2 CN1515FA102L0225 Y N 34A QSFP DAC FCI Electronics 1.0m 10131941-2010LF 2 CN1515FA102L0167 Y N 35A QSFP DAC FCI Electronics 1.0m 10131941-2010LF 2 CN1515FA102L0101 Y N 36A QSFP DAC FCI Electronics 1.0m 10131941-2010LF 2 CN1515FA102L0010 Y N 37A QSFP DAC FCI Electronics 1.0m 10131941-2010LF 2 CN1449FA102L0029 Y N 38A QSFP DAC FCI Electronics 2.0m 10142057-2020LF E CN2504ZE202L58B0 Y N 39A QSFP DAC FCI Electronics 1.0m 10121178-2010LF G CN1539QV102L0022 Y N 40A QSFP DAC FCI Electronics 1.0m 10131941-2010LF 2 CN1515FA102L0165 Y N 41A QSFP DAC FCI Electronics 2.0m 10142057-2015LF E CN2440ZE152L4GW1 Y N 42A QSFP DAC FCI Electronics 1.0m 10131941-2010LF C CN1638FA102L0098 Y N 43A QSFP DAC FCI Electronics 3.0m 10142057-4030HLF E CN2434ZE304H40M1 Y N 44A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H7003E Y N 27.0 46A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H7003B Y N 34.0 47A QSFP DAC Amphenol 1.0m NJAAKR-CN01 A APF2446N0172C6 Y Y 48A QSFP AOC Hisense 7.0m DQF8503-4C07 02 S5189H7004F Y N 30.6
To correct, use port config modify speed to change the speed to a cable-supported value.