Skip to main content

Cornelis Technical Documentation

4.3.2. The Bandwidth Problem

Note

In this section, the use of the terms blocking and non-blocking is independent of the use of the same terms in the context of message passing in software.

If you were to construct a simple topology with two CN5000 Switches connected by a single ISL, they could divide the resulting fabric into two halves, each half managed by Switch A or Switch B. The ISL connecting the two switches also connects the two halves of the fabric. If the ISL were to be cut in half, the resulting cut would bisect the fabric. This allows for the definition of an important metric: the bisection bandwidth, which is the sum of the links that cross between the two halves of the fabric.

If we assume both Switches A and B are connected to 47 endpoints, each capable of sending and receiving at 400 Gbps, then up to 18.8 Tbps of traffic could originate from endpoints connected to each switch. Since only a single ISL contributes to the bisection bandwidth, dwarfed by the maximum origination bandwidth, traffic sent between the switches is heavily bottlenecked. This is considered a blocking configuration.

This issue can be mitigated by adding additional ISLs between Switches A and B. ISLs can be used concurrently, meaning that their bandwidths are additive. You can visualize these ISLs as a single composite link of a larger effective bandwidth than a single ISL. A larger or fatter pipe or cable can allow for more flow.

If we assume that 24 ISLs are established between Switches A and B, only 24 endpoints can be connected to each switch. This results in the bisectional bandwidth being identical to the origination bandwidth, eliminating the bottleneck. This is a non-blocking configuration. However, the elimination of the ISL bottleneck results in a drastic reduction in the number of total endpoints in the fabric. In practice, both the single ISL blocking configuration and the non-blocking configuration would likely result in a suboptimal amount of compute performance for such a small fabric, while a moderate number of ISLs would be more desirable. Refer to Oversubscription.