Skip to main content

Cornelis Technical Documentation

4.5.1. Fabric Unicast Routing

Unicast routing represents a critical aspect of fabric configuration. How packets traverse the network is determined by several factors. routing architecture is based on three main factors:

  • Addressing Mode

    • Linear Addressing

    • Hierarchical Addressing

  • Routing Mode

    • Deterministic/Static Routing

    • Dynamic/Adaptive Routing

  • Routing Algorithm

The SM must carefully balance several competing requirements when establishing routing paths throughout the network. This balancing act involves optimizing for:

  • Performance across diverse workloads and applications.

  • Resilience against network disruptions and component failures.

  • Prevention of credit loops.

  • Efficient use of limited network resources, including LIDs and routing table entries.

These requirements often present trade-offs, as optimizing for one factor may negatively impact another. For example, maximizing performance might require more routing table entries, while enhancing resilience could introduce additional hops that affect latency. To address these challenges, the SM provides administrators with configurable routing options. The routing algorithm selection and parameters can be tailored to specific network topologies and workload characteristics, allowing administrators to prioritize the factors most important for their environment. Understanding these routing considerations is essential for administrators seeking to optimize their deployment for specific application requirements and operational constraints.

4.5.1.1. Addressing Modes

Note

The current release only supports linear addressing.

Desired addressing mode is indicated to the SM by the system administrator through configuration file. SM will calculate and assign appropriate LIDs. Linear addressing essentially means a flat address space. In the implementation, device LIDs are assigned in the order in which they are discovered by SM’s discovery phase. Linear address space is determined by the size of the forwarding tables.

Hierarchical addressing has the potential to expand beyond the limits of linear addressing by identifying hierarchy in the topology and creating sub fields for each level of hierarchy – typically a group (or pod), switch in group (switch in pod), and terminal.

4.5.1.1. Routing Algorithms

Routing determines the path that traffic follows between endpoints in the fabric. For every pair of nodes, the SM selects the route used for communication. Route selection typically considers factors such as hop count and the bandwidth characteristics of each link, including link width and link speed, to identify efficient paths through the network.

Beyond selecting individual routes, the SM is also responsible for balancing traffic across the fabric. It statically distributes communication paths across the available links to help prevent congestion and ensure that network resources are utilized efficiently. By spreading traffic over multiple paths, the SM can improve overall fabric performance and avoid overloading specific links.

To accommodate different network sizes, topologies, and workload requirements, the fabric supports multiple routing strategies. These strategies are implemented as selectable routing algorithms, each designed to optimize path selection and load distribution for particular deployment scenarios. Choosing the appropriate routing algorithm can have a significant impact on bandwidth utilization, latency, and overall network performance.

The SM supports the following routing algorithms:

  • Shortest Path: The default option and works very well for most fabrics.

  • Fat Tree: An optimized, balanced routing specifically for fat tree topologies with credit loop avoidance.

  • Device Group Shortest Path: A variation of shortest path that can result in better balanced fabrics in some conditions.

The routing algorithm is selectable using the RoutingAlgorithm parameter in the opafm.xml file.

4.5.1.1.1. Shortest Path

The Shortest Path algorithm is the default routing method and provides excellent performance for most fabric topologies. It routes traffic along the lowest-cost path between endpoints. In many fabrics, multiple paths have the same cost. When equal-cost paths exist, the SM statically distributes routes across available inter-switch links (ISLs) to balance path utilization. SpineFirstRouting is an optional enhancement to the Shortest Path algorithm. When enabled, and multiple equal-length paths are available, the SM prefers routing traffic from a DCS leaf through a Product DCS spine rather than through a Switch located outside the DCS. This approach treats the spine as the "up" direction relative to the leaf, creating a clean and consistent up/down routing model throughout the fabric. By ensuring that traffic follows consistent routing paths, SpineFirstRouting helps prevent credit loops.

For fabrics where Switches are not located at the core of the topology, the Fat Tree routing algorithm can be used to achieve similar spine-first routing behavior. SpineFirstRouting is enabled by default and does not introduce any known adverse side effects. Unlike simpler routing implementations found in some other plane managers, the SM's Shortest Path algorithm includes advanced routing and traffic-balancing capabilities that deliver high performance across a wide range of fabric topologies.

4.5.1.1.2. Fat Tree

The Fat Tree routing algorithm typically provides better ISL traffic balancing in Fat-Tree topologies than the Shortest Path algorithm. It also implements an up/down routing model that helps prevent deadlocks. To optimize traffic distribution, the Fabric Manager analyzes the fabric topology, identifies the Fat-Tree structure, and determines the appropriate up/down routing paths based on the tier in which each switch resides.

To identify switch tiers, the Fabric Manager must know the number of switch-chip tiers in the fabric. When all SuperNICs and Target Fabric Interfaces (TFIs) are connected at the same tier, the Fabric Manager can automatically discover the Fat-Tree topology. In environments where endpoints are connected across multiple tiers, you can define a CoreSwitches device group to explicitly identify the root or core Switches of the topology.

When using a CoreSwitches device group, only devices that do not communicate directly with one another should be connected to the core Switches. If devices attached to the core Switches exchange traffic, they can potentially create a credit loop.

The Fat Tree algorithm also supports traffic balancing across groups of devices. By defining a RouteLast device group, you can ensure that routing is balanced for the devices within that group. For example, placing all compute nodes in a RouteLast device group can improve traffic distribution across the fabric. This feature can also be used to balance traffic between compute and I/O nodes. If all compute nodes are assigned to the RouteLast group, the Fabric Manager calculates routes for the compute nodes first and then for the I/O nodes, applying load balancing within each group to achieve more even traffic distribution across the fabric.

4.5.1.1.3. Device Group Shortest Path

Device Group Shortest Path (dgshortestpath) routing is a form of Min-Hop or Shortest Path (shortestpath) routing, except you can control the order in which routes to end nodes are assigned. This can be used to ensure diversity of routing within groups of devices as well as the entire fabric overall. End nodes that are not members of any listed groups will be routed last.

When the routing algorithm is set to dgshortestpath, the following section in the opafm.xml file is used to configure the algorithm.

<DGShortestPathTopology>
<!-- RoutingOrder lists the device groups in the order they should -->
<!-- be handled. Each device group must have been declared in the -->
<!-- DeviceGroups section. -->
<!-- <RoutingOrder> -->
<!-- <DeviceGroup>Compute</DeviceGroup> -->
<!-- <DeviceGroup>All</DeviceGroup> -->
<!-- <DeviceGroup>Storage</DeviceGroup> -->
<!-- </RoutingOrder> --></DGShortestPathTopology>
4.5.1.1.4. Credit Loops

Because Omni-Path Architecture uses credit-based link-layer flow control, credit loops can occur. Under heavy load, a credit loop may escalate into a fabric deadlock, forcing switch timeout mechanisms to discard packets. These deadlocks and packet drops can significantly degrade fabric performance.

Preventing credit loops is a key objective of many routing algorithms. Credit loops can be avoided in all commonly deployed fabric topologies, and the SM uses routing algorithms specifically designed to eliminate them. One method for preventing credit loops in tree-based topologies is up/down routing. When multiple equal-cost paths are available, this approach gives preference to links that move traffic "up" the tree rather than "down" it. In fabrics that use DCSs as the backbone, this behavior can be implemented with the shortest path routing algorithm by enabling SpineFirstRouting in the SM configuration.

When SpineFirstRouting is enabled, the SM prefers equal-length paths that route traffic from a DCS leaf through the DCS spine rather than through external edge connections. This effectively treats the spine as "up" relative to the leaf, creating a consistent up/down routing model across the fabric. By ensuring that traffic follows predictable, deterministic paths, the SM prevents the formation of credit loops. The Fat-Tree routing algorithm also avoids credit loops in fabrics whose backbone is not composed of DCSs. It applies the same spine-first routing principles to any switch type in the fabric core, providing loop-free routing while maintaining efficient path utilization.

If a credit loop is suspected, use the CLI command opareport -o validatecreditloop to check the fabric.

4.5.1.2. Adaptive Routing

A limitation of static routing is that it must be done before traffic begins to flow. Static routes are balanced using best guesses by the Fabric Manager and sysadmin of potential application traffic patterns. However, once applications start to run, those routes may not be ideal. Adaptive routing allows the switches to adjust their routes while the applications are running to balance the routes based on actual traffic patterns.

The Cornelis adaptive routing solution is highly scalable because it allows the Fabric Manager to provide the topology awareness and program the switches with the rules for adaptive routing. Thereafter, switches can dynamically and rapidly adjust the routes based on actual traffic patterns.

This approach ensures a scalable solution because as switches are added, each new switch works in parallel with others to dynamically adapt to traffic. Additionally, this approach removes the Fabric Manager as a bottleneck.

Adaptive routing provides a few important capabilities and options:

  • Adaptive routing can rapidly route around fabric disruptions and lost ISLs. When adaptive routing is enabled, this capability automatically occurs and limits the amount of lag time between an ISL going down and the traffic being redirected to alternate routes.

  • Adaptive routing can automatically balance and re-balance the fabric routes based on traffic patterns. Adaptive routing has the ability to handle changing traffic patterns that may occur due to different computational phases or the impacts of starting or completing multiple applications that are running on the same fabric.

When Fine Grained Adaptive Routing (FGAR) is enabled, the SM programs each Switch with a list of alternate, equal-cost routes for each destination in the fabric. When a Switch detects that a route is congested (due to congestion or a complete link or Switch failure), it selects an alternate from the list of alternates and updates its linear forwarding table (LFT) with the alternate route.

To enable FGAR, stop the Fabric Manager, edit opafm.xml file, reboot the Switches, and restart the Fabric Manager. The Switches must be rebooted if the Fabric Manager was previously in a non-FGAR, linear configuration, otherwise reboot is not required. Enabling FGAR also enables static dispersive routing (SDR) which uses the entropy bit in OPX Software to select paths for packets as they are being sent; thereby, dispersing the traffic across the network.

To use FGAR once it is enabled, a user must export FI_OPX_ROUTE_CONTROL=’4:4:4:4:4:4’ to the environment before launching jobs. To use SDR, export FI_OPX_ROUTE_CONTROL=’0:0:0:0:0:0’.

Note

Currently OPX Software does not support Token ID (TID) routing for FGAR packets using rendezvous transport and it will fall back to the SDR algorithm for those packets.

To enable full FGAR, the user should also disable TID transport by exporting to FI_OPX_TID_DISABLE=1. Alternatively, the user may keep TID enabled and avoid rendezvous transport by setting FI_OPX_RZV_MIN_PAYLOAD_BYTES to a very large value.

If the user does not disable TID and sets FI_OPX_ROUTE_CONTROL=’4:4:4:4:4:4’, it will effectively be reset to ’4:4:4:4:4:0’. To enable FGAR in opafm.xml , the RoutingAlgorithm must be set to fattree.

The following is the AdaptiveRouting section of the opafm.xml file configured for FGAR:

        <!-- Configures support for AdaptiveRouting in Cornelis Switches -->
    <!-- Adaptive Routing monitors the performance of the possible paths between -->     
<!-- fabric endpoints and periodically rebalances the routes to reduce -->
    <!-- congestion and achieve a more balanced packet load -->
    <AdaptiveRouting>       <!-- 1 = 
Enable, 0 = Disable -->
      <Enable>0</Enable>
      <!-- When set, Fine Grained routing algorithm will be used to route packets -->
      <!-- If set using fat tree, LMC will be replaced by SDR ignoring LMC tags -->
      <FineGrained>0</FineGrained>
      <!-- When set, only adjust routes when they are lost. -->       
<!-- If not set, adjust routes when they are lost and -->
      <!-- when congestion is indicated. -->
      <LostRouteOnly>0</LostRouteOnly>      
 <!-- Algorithm the switch should use when selecting an egress port -->
      <!-- Algorithms are currently 0 = Random, 1 = Greedy and -->
      <!-- 2 = GreedyRandom. Default is 2. -->
      <Algorithm>2</Algorithm>
      <!-- Update Frequency: Specifies the minimum time between -->       
<!-- AR adjustments. Values range from 0 to 7 and are read as 2^n -->
      <!-- times 64 ms. Default is 0. -->
      <ARFrequency>0</ARFrequency>
      <!-- Congestion threshold above which switch uses adaptive routing. -->       
<!-- Congestion threshold is per-VL and measured by tag consumption percentage. -->
      <!-- Values range from 0 to 7. -->
      <!-- 7, 6, 5, 4 correspond to 55%, 60%, 65%, and 70%, respectively. -->       
<!-- 3, 2, 1 correspond to 80%, 90%, and 100%, respectively. -->
      <!-- 0 means "Use firmware default". Default is 3. -->
      <!-- Higher Percentage means higher congestion is required before Adaptive -->
      <!-- routing takes control. Higher Percentage is less sensitive, less adaptive. -->
      <Threshold>3</Threshold>
    </AdaptiveRouting>$›$

The following describes the different settings for AdaptiveRouting:

  • The Algorithm setting the Switch uses for choosing the alternate route can be:

    • Random: Choose the alternate randomly.

    • Greedy: Choose the alternate that is least busy.

    • GreedyRandom: If there are multiple alternates that are not busy, randomly choose from them.

  • The Threshold setting determines how busy a route must be before it will be rerouted by Adaptive Routing.

  • The ARFrequency setting determines how frequently the switch checks for congestion.

  • If the LostRouteOnly setting is enabled, traffic is only rerouted if a route completely fails. Such rerouting is only done on the switch with the port that has gone down.

Note

To enable FGAR, both the AdaptiveRouting and FineGrained settings must be enabled.