Skip to main content

Cornelis Technical Documentation

4.3.2. Bisection Bandwidth

Note

In this section, the use of the terms blocking and non-blocking is independent of the use of the same terms in the context of message passing in software.

A simple two-switch topology illustrates one of the most important concepts in fabric design: bisection bandwidth.

Consider two CN5000 Switches, Switch A and Switch B, connected by a single ISL. Endpoints connected to Switch A form one side of the fabric, while endpoints connected to Switch B form the other. Any traffic exchanged between endpoints on opposite sides of the fabric must traverse the ISL connecting the two Switches.

This example introduces the concept of bisection bandwidth, which is the total bandwidth available across a logical division that separates a network into two parts. In a two-switch topology, the bisection bandwidth is provided entirely by the ISL or ISLs connecting the Switches.

Suppose each Switch dedicates one port to the ISL and uses its remaining 47 ports for endpoint connections. Each side of the fabric can then support 47 endpoints. With each endpoint capable of transmitting at 400 Gbps, the endpoints attached to a Switch can collectively generate up to 47 × 400 Gbps = 18.8 Tbps. However, a single ISL provides only 1 × 400 Gbps = 400 Gbps of bandwidth between the two halves of the fabric.

As a result, while endpoints on one side may collectively generate up to 18.8 Tbps of traffic, only 400 Gbps can cross the bisection at any given time. When significant traffic must flow between the Switches, the ISL becomes a bottleneck. This is an example of a blocking configuration, in which the available network bandwidth is insufficient to support all possible traffic demands simultaneously. The bottleneck can be reduced by adding additional ISLs between the Switches. Because ISLs operate in parallel, their bandwidth is additive. Multiple ISLs can therefore be viewed as a single, wider connection between Switch A and Switch B:

  • 4 ISLs provide 1.6 Tbps of bisection bandwidth per direction.

  • 12 ISLs provide 4.8 Tbps of bisection bandwidth per direction.

  • 24 ISLs provide 9.6 Tbps of bisection bandwidth per direction.

A fully balanced configuration occurs when 24 ports on each switch are allocated to ISLs and the remaining 24 ports are used for endpoint connections. In this case, the endpoint bandwidth on each side of the fabric is 24 × 400 Gbps = 9.6 Tbps. The bisection bandwidth is also 24 × 400 Gbps = 9.6 Tbps. Because the bandwidth available across the bisection matches the maximum bandwidth that endpoints can generate on either side, the fabric can support any communication pattern between the two halves without creating a bottleneck. This is known as a non-blocking configuration across the bisection.

Achieving a non-blocking design; however, requires dedicating half of the switch ports to ISLs, reducing the number of ports available for endpoint connections. Consequently, neither extreme is typically ideal for a small two-switch fabric. A single-ISL design can create severe congestion during cross-switch communication, while a fully non-blocking design may sacrifice too many endpoint ports.

In practice, network designers usually choose an intermediate number of ISLs based on expected traffic patterns, application requirements, and acceptable levels of oversubscription. This approach balances fabric scalability, endpoint density, and communication performance while avoiding both excessive blocking and unnecessary consumption of Switch ports. For additional details on this tradeoff, see Oversubscription.