4.3.2. 2-Tier Fat Trees
The 1-tier topology (a single-switch network) is simple and low latency, but its size is inherently limited by the number of ports available on the Switch. To support larger fabrics while maintaining high bandwidth between endpoints, networks commonly use a 2-tier Fat-Tree topology, often referred to in data center networking as a spine-leaf architecture a folded-Clos network.
In a 2-tier Fat Tree, the network is organized into two layers:
Leaf layer – Leaf Switches connect directly to endpoints, such as compute nodes, storage systems, or other devices.
Spine layer – Spine Switches form the network backbone and connect only to leaf Switches.
Each leaf Switch connects to every spine Switch, creating multiple paths through the fabric. This architecture provides predictable performance, balanced bandwidth, and efficient scaling as the number of endpoints grows.
Note
The terminology used here mirrors the spine and leaf module names used by the DCS. This is intentional, as the internal DCS network is implemented as a 2-tier Fat-Tree topology.
Consider a non-blocking 2-tier Fat Tree. In this configuration, each leaf Switch dedicates a portion of its ports to endpoint connections and the remaining ports to uplinks that connect to the spine layer. This balanced design ensures that the aggregate uplink bandwidth matches the aggregate endpoint bandwidth, eliminating bottlenecks between the leaf and spine tiers.
Using native 400 Gbps connections, each leaf Switch can support up to 24 endpoints while using its remaining ports for spine connectivity (see Figure 67, “2-Tier Fat Tree with 1:1 Subscription”).
Using subdivided 200 Gbps connections, each leaf Switch can support up to 48 endpoints while maintaining the same non-blocking characteristics (see Figure 68, “2-Tier Fat Tree with 1:1 Subscription and Subdivided SuperNIC Links”).
The spine Switches provide connectivity between leaf Switches but do not connect directly to endpoints. By using the spine layer to interconnect multiple leaves, the fabric can scale far beyond the capacity of a single Switch while preserving full bandwidth between endpoints.
A non-blocking 2-tier Fat Tree can support up to 1,152 endpoints using native 400 Gbps connections and 2,304 endpoints using subdivided 200 Gbps connections. This represents a substantial increase in scale compared to a single-switch topology while maintaining predictable performance and full bisection bandwidth.
As a result, the 2-tier Fat Tree is a common choice for HPC, AI, and large-scale data center environments that require both high endpoint density and high network throughput.
4.3.2.1. Advantages
The primary advantage of a 2-tier Fat-Tree topology is its ability to provide predictable, high-bandwidth, low-latency communication between endpoints throughout the fabric. In a non-blocking design, communication paths are uniform and highly scalable, making network performance easier to predict as the fabric grows.
For communication between endpoints connected to different leaf Switches, traffic follows a consistent path through the network:
The source endpoint sends data to its local leaf Switch.
The leaf Switch forwards the traffic to a spine Switch.
The spine Switch forwards the traffic to the destination leaf Switch, which then delivers it to the destination endpoint.
As a result, any communication between endpoints on different leaf Switches traverses the same number of Switch hops, helping to maintain consistent latency across the fabric. Unlike a simple multi-switch 1-tier topology, where communication between Switches may be constrained by a small number of ISLs, a 2-tier Fat Tree provides multiple parallel paths between leaf Switches. Traffic can be distributed across these paths, greatly increasing the bandwidth available between different parts of the network. This characteristic is the origin of the term "fat" tree—the aggregate bandwidth available in the upper levels of the hierarchy becomes progressively "fatter" as additional spine connectivity is added.
To illustrate this concept, consider a 2-tier Fat Tree consisting of four leaf Switches and four spine Switches. Each leaf Switch connects to every spine switch, resulting in four uplinks per leaf. If the network is bisected by conceptually separating the leaf layer from the spine layer, the cut passes through sixteen ISLs. The resulting bisection bandwidth is therefore sixteen times the bandwidth of a single ISL, greatly reducing the potential bottlenecks that can occur in a simple 1-tier topology.
Another important property of the 2-tier Fat Tree is that the number of available paths between any two leaf Switches is equal to the number of spine Switches. Adding spine Switches therefore increases both path diversity and aggregate bisection bandwidth, allowing the fabric to scale while maintaining balanced performance.
The topology also provides a high degree of modularity. Additional leaf Switches can be deployed to support more endpoints without altering the overall fabric structure, provided sufficient ports remain available on the spine layer. Likewise, additional spine Switches can be added to increase network bandwidth and improve scalability as workload requirements grow.
A further advantage is fault tolerance. In a 1-tier topology, the failure of a critical ISL can significantly restrict communication or potentially isolate portions of the fabric. In contrast, a 2-tier Fat Tree provides multiple redundant paths through the spine layer. The failure of a single ISL—and, in many cases, even the failure of an entire spine Switch—does not prevent communication between endpoints. Traffic can be automatically redirected through the remaining spine Switches, allowing the fabric to continue operating with reduced impact on application performance. These characteristics—predictable latency, high bisection bandwidth, scalability, modular growth, and path redundancy—make the 2-tier Fat Tree one of the most widely deployed network topologies for high-performance computing, AI, and large-scale data center environments.
4.3.2.2. Limitations
While the 2-tier Fat-Tree topology offers substantial advantages in performance, scalability, and resiliency, it also has practical limitations that must be considered during fabric design. The maximum size of a 2-tier Fat Tree is ultimately constrained by Switch port counts. The number of endpoints that can be supported depends on both the ports available on the leaf Switches for endpoint connections and the number of leaf Switches that the spine layer can accommodate. Once the available spine ports are exhausted, further expansion requires a different topology or an additional network tier.
Scalability also comes at the cost of increased infrastructure. As more leaf and spine Switches are added, the number of required ISLs grows rapidly. Each leaf Switch must maintain connections to every spine Switch, resulting in a significant increase in cabling, Switch ports dedicated to uplinks, and overall deployment complexity. As the fabric grows, these requirements can increase both capital and operational costs.
In addition, larger Fat-Tree deployments require more careful planning and management. Routing, cabling, power, cooling, and physical layout become increasingly important as the number of Switches and ISLs increases. Maintaining a fully non-blocking architecture may also require dedicating a substantial portion of Switch ports to uplink connections, reducing the number of ports available for endpoints.
Despite these challenges, the 2-tier Fat Tree remains a highly effective topology for medium-sized deployments. It provides significantly higher bisection bandwidth, better scalability, and greater fault tolerance than simpler 1-tier designs while maintaining predictable communication paths and low latency. For many HPC, AI, and data center environments, these benefits outweigh the additional complexity and infrastructure requirements, making the 2-tier Fat Tree a common and well-established fabric architecture.