4.5.3. Congestion Control Architecture
Even with well-balanced routing, transient traffic patterns can cause more data to converge on a link than that link can carry, creating congestion. Left unmanaged, congestion spreads backward through the fabric as switch buffers fill, forming congestion trees that degrade performance for flows that do not even share the original bottleneck. The Omni-Path Congestion Control Architecture (CCA) limits this behavior by detecting congestion where it occurs and signaling the traffic sources to slow down until the congestion clears.
CCA uses an explicit congestion notification mechanism. A Switch identifies a potentially congested link and forwards a notification toward the destination SuperNIC. The destination SuperNIC reflects that notification back to the sending SuperNIC, and the sending SuperNIC reduces its packet injection rate to the affected destination.
The amount by which a sender reduces its rate, and the timing of its recovery, are determined dynamically by the frequency and pattern of the notifications it receives. When notifications are frequent, the sender applies more aggressive rate reduction. As notifications subside, the sender gradually restores its transmission rate, optimizing fabric utilization while continuing to prevent congestion.
CCA operates as an end-to-end control loop with three cooperating roles: congestion detection and marking at the Switch, notification reflection at the destination SuperNIC, and injection-rate control at the source SuperNIC.
The Omni-Path CCA implements both the notification loop and the injection-rate control in hardware. Handling these functions in hardware—rather than in host software—provides three benefits that directly affect fabric behavior:
Faster reaction, smaller congestion trees: Removing host software from the notification loop shortens the time between the onset of congestion and the sender pacing its transmissions. Reacting sooner limits how far congestion trees can grow.
Selective, per-destination pacing: Hardware applies the pacing delay selectively per DLID, so packets destined for uncongested DLIDs continue to egress while only traffic to congested DLIDs is paced. A transmit queue stalls only when the packet at the head of the queue targets a DLID that is currently being paced.
Scalable congestion tracking: Each fabric port tracks congestion state for a bounded number of DLIDs. When more destinations must be tracked than the hardware can hold, the least recently used (LRU) entry is replaced, so tracking resources stay focused on the most active destinations.
4.5.3.1. Congestion Detection and Marking
Congestion detection and marking is performed by the Switch. A Switch detects when an egress port enters a congested state on a per–VL basis. When a VL becomes congested, the Switch sets the Forward Explicit Congestion Notification (FECN) bit in the Layer 2 header of a configurable subset of the packets leaving that port.
Marking only a subset of packets—rather than every packet—signals the presence of congestion without causing senders to over-react. This keeps the control loop responsive while avoiding the oscillation that would result from every source reacting to every packet at once.
4.5.3.2. Congestion Response
The congestion response completes the control loop in two stages: the destination SuperNIC reflects the notification back to the source, and the source SuperNIC paces its transmissions in response.
4.5.3.2.1. Notification Reflection
When a FECN-marked packet arrives at its destination, the destination SuperNIC completes the forward half of the loop by setting the Backward Explicit Congestion Notification (BECN) bit in a packet returned to the original sender. To avoid adding traffic to an already-congested fabric, the BECN response can piggyback on a normal response or acknowledgment packet rather than requiring a dedicated packet.
4.5.3.2.2. Injection Rate Control
On receiving BECN-marked packets, the source SuperNIC paces its transmission to the affected destination by inserting inter-packet delays. As BECN notifications subside, the pacing delay is gradually reduced until the full injection rate is restored.
Detection is per-VL, but pacing is per-destination. Switches detect and mark congestion on a per-VL basis, but the source SuperNIC applies pacing per DLID, not per VL. This is because the specific VL that experienced congestion is not carried back in the BECN signal—only the destination is identifiable to the source. Pacing therefore targets all traffic to a congested destination.