4.3.5. 3-Tier Fat Trees
While 2-tier Fat-Tree topologies are simple, efficient, and suitable for many deployments, their maximum size is ultimately constrained by Switch port counts. For larger environments, a 3-tier Fat Tree provides significantly greater scalability while preserving the high bandwidth, path redundancy, and fault tolerance that make Fat-Tree architectures attractive for HPC and AI infrastructures. A 3-tier Fat Tree introduces an additional Switch layer between the leaf and spine tiers. This intermediate layer, commonly called the aggregation tier, creates a deeper hierarchy that allows the fabric to scale to tens of thousands of endpoints while maintaining balanced bandwidth throughout the network.
In CN5000-based deployments, a 3-tier Fat Tree can be implemented using either: CN5000 Switch es at all tiers, or a combination of CN5000 Switches and Director Class Switch es (DCSs).
In a 3-tier Fat Tree built entirely from CN5000 Switches, the fabric is organized into pods. Each pod contains leaf Switches, which connect directly to endpoints, and aggregation Switches, which connect the edge layer to the core layer. The core tier interconnects the pods and provides communication paths between endpoints located in different pods. Consider a fully non-blocking (1:1 subscription) 3-tier Fat Tree constructed entirely with CN5000 Switches. This architecture can support up to 27,648 endpoints, far exceeding the scale of a 2-tier design. The fabric would consist of 48 compute pods, 24 edge Switches per pod, and 24 aggregation Switches per pod.
Each edge Switch supports 24 native 400 Gbps endpoint connections, resulting in 24 edge Switches × 24 endpoints = 576 endpoints per pod. Across all 48 compute pods, the total endpoint count becomes 48 pods × 576 endpoints = 27,648 endpoints. The pods are interconnected through 576 core Switches that form the third tier of the topology. Within the core tier, Switches can be logically grouped according to their connectivity patterns. Sets of 24 core Switches that connect to the same group of 24 aggregation Switches form a Core Group. Together, these Core Groups and their associated aggregation-switch connections can be viewed as Core Pods, although the aggregation switches themselves remain associated with their respective compute pods.
An alternative approach uses DCSs for the aggregation and core tiers. This design can reduce the total number of discrete Switches required while providing additional flexibility in fabric construction.
A DCS can be configured from 48 to 576 ports, in increments of 48 ports, enabling designers to tailor the upper tiers to the required scale. When DCSs are used in the aggregation and core layers, a non-blocking fabric can support up to 13,824 endpoints using native 400 Gbps connections. If the endpoint-facing connections on the edge Switches are subdivided into 200 Gbps links, the maximum endpoint count doubles to 27,648 endpoints. This increase in scale comes at the cost of reduced per-endpoint bandwidth.
As with 2-tier Fat Trees, oversubscription can be applied to a 3-tier architecture. For example, a 2:1 oversubscribed edge tier allocates more ports to endpoint connections and fewer ports to uplinks toward the aggregation tier. This increases the number of supported endpoints but introduces the possibility of congestion when traffic demand exceeds the available uplink bandwidth. Oversubscription can be implemented in all CN5000 deployments.
The performance impact depends on application communication patterns. Workloads with strong communication locality may experience little effect, while workloads that generate heavy cross-fabric traffic may encounter increased latency and reduced effective bandwidth. The choice between a 2-tier and 3-tier Fat Tree depends primarily on scale, performance requirements, and available hardware. A 2-tier Fat Tree is often the preferred solution for smaller deployments because it provides simpler management, lower cost, fewer Switches and cables, and predictable low-latency communication. However, as endpoint counts grow into the thousands, the port-count limitations of a 2-tier design become restrictive.
A 3-tier Fat Tree provides the additional scalability required for large HPC and AI clusters while maintaining the key benefits of Fat-Tree architectures:
High bisection bandwidth.
Predictable communication paths.
Redundant network paths.
Strong fault tolerance.
Large-scale endpoint support.
For deployments requiring only a few hundred or a few thousand endpoints, a 2-tier topology is often sufficient and more economical. For larger deployments, particularly those approaching or exceeding several thousand endpoints, a 3-tier Fat Tree becomes the practical choice for achieving the required scale without sacrificing network performance.
4.3.5.1. Advantages
Although a 3-tier Fat Tree introduces additional Switch hops compared to a 2-tier design, it retains the fundamental advantages of Fat-Tree architectures including high bisection bandwidth, path redundancy, and fault tolerance. Multiple parallel paths exist between any two endpoints, allowing traffic to be distributed across the fabric and reducing the likelihood of congestion on any single path.
For communication between endpoints located in different pods, the number of available paths is determined by the number of core Switches connected to both pods. In a fully populated 3-tier Fat Tree, this can provide a substantial number of parallel paths, increasing both aggregate bandwidth and network resiliency.
Another significant advantage of the 3-tier topology is its modularity. The fabric can be expanded incrementally as requirements grow. Additional compute pods can be added by connecting them to the existing core tier, provided sufficient capacity remains available within the core. This allows organizations to increase fabric size without redesigning the entire network. Similarly, the fabric's bisection bandwidth can be increased by adding core-switch capacity. As additional core switches are introduced, corresponding aggregation-tier resources may also be required to maintain the desired subscription ratio and preserve balanced bandwidth throughout the topology.
Designing a 3-tier Fat Tree typically involves balancing four key considerations:
Scale – The number of endpoints the fabric must support.
Bandwidth – The amount of communication capacity required between endpoints.
Latency – The performance impact of additional Switch tiers and longer communication paths.
Cost – The Switches, cabling, power, rack space, and operational resources needed to deploy and maintain the fabric.
For the largest and most demanding HPC and AI environments, a fully non-blocking 3-tier Fat Tree with a 1:1 subscription ratio provides the highest possible performance, ensuring full bandwidth availability throughout the fabric. For deployments with more moderate requirements, architectures that incorporate DCS or employ controlled oversubscription can offer a more cost-effective solution. These approaches reduce infrastructure requirements while still delivering sufficient bandwidth and scalability for the expected workload mix. As a result, the 3-tier Fat Tree remains one of the most widely used architectures for large-scale networks, combining exceptional scalability with the predictable performance and resiliency characteristics that are essential for modern HPC, AI, and data-center deployments.
4.3.5.2. Limitations
It is worth noting that as the number of tiers increases, so does the maximum hop count between any two endpoints. In a 3-tier Fat Tree, communication between endpoints in different pods requires traversing five hops:
From the source endpoint to its edge switch.
From the edge switch to an aggregation switch.
From the aggregation switch to a core switch.
From the core switch to the destination pod's aggregation switch.
From the aggregation switch to the destination edge switch and endpoint.
This increased hop count can lead to higher latency compared to a 2-tier topology, although the impact is often mitigated by the high-speed links and low-latency Switches.