Short answer
Large AI clusters connect thousands of GPUs with a back-end network that must move huge volumes of data with very low latency and no packet loss. InfiniBand, now mainly supplied by NVIDIA, has long been the default for AI training thanks to native RDMA, credit-based lossless flow control and in-network computing. Ethernet has caught up through RoCE (RDMA over Converged Ethernet), AI-tuned platforms like NVIDIA Spectrum-X and the open Ultra Ethernet standard. Both now run some of the world's largest clusters; Meta trained Llama 3 on two 24,000-GPU clusters, one using InfiniBand and one using RoCE Ethernet.
Why the network matters so much for AI
When you train a model across many GPUs, they constantly exchange data: averaging gradients, gathering sharded weights, passing activations between pipeline stages, and routing tokens to experts in mixture-of-experts models. See distributed training and parallelism.
These are collective operations (all-reduce, all-gather, reduce-scatter, all-to-all), and every GPU must wait for the slowest participant. A single congested link or dropped packet can stall thousands of GPUs worth millions of dollars. AI networking therefore cares about:
- Bandwidth: typically 400 to 800 Gb/s per GPU today.
- Tail latency: the slowest flows, not the average, decide step time.
- Losslessness: retransmissions are expensive at this scale.
- Scale: tens of thousands to over 100,000 GPUs.
Scale-up vs scale-out networks
| Scale-up | Scale-out (back-end) | Front-end | |
|---|---|---|---|
| Connects | GPUs inside a server or rack | Servers across the cluster | Cluster to storage, users, internet |
| Technology | NVLink and NVSwitch, AMD Infinity Fabric | InfiniBand or Ethernet with RDMA | Standard Ethernet |
| Bandwidth per GPU | ~900 GB/s to 1.8 TB/s (NVLink) | ~50–100 GB/s (400–800 Gb/s) | Lower |
| Used for | Tensor and expert parallelism | Data and pipeline parallelism | Data loading, checkpoints, serving |
Scale-up links are roughly 10 times faster than scale-out links, which is why bandwidth-hungry parallelism stays inside the NVLink domain; see NVLink and NVSwitch explained and GB200 NVL72. This article focuses on scale-out networking.
RDMA: the foundation
Remote Direct Memory Access (RDMA) lets one machine read or write another machine's memory directly, bypassing the operating system and CPU. With GPUDirect RDMA, the network card moves data straight to and from GPU memory. This cuts latency and frees CPUs. Both InfiniBand and modern AI Ethernet are built around RDMA; libraries such as NVIDIA's NCCL use it to run collectives across the cluster.
InfiniBand
InfiniBand was designed for high-performance computing and has native RDMA, credit-based flow control (a sender only transmits when the receiver has buffer space, so packets are not dropped), adaptive routing and centralised management through a subnet manager.
NVIDIA's InfiniBand platforms include Quantum-2 (400 Gb/s NDR) and Quantum-X800 (800 Gb/s XDR), with ConnectX network adapters. Its SHARP technology performs parts of reductions inside the switches, reducing traffic for all-reduce operations.
Strengths: mature, very low latency, predictable performance, excellent tooling for HPC and AI. Weaknesses: effectively one main vendor, specialised skills, often higher cost, separate from the Ethernet used elsewhere in the data center.
Ethernet for AI
Standard Ethernet is "best effort": switches drop packets when queues overflow. For AI, Ethernet adds:
- RoCEv2: RDMA carried over UDP/IP so it can be routed across large networks.
- Lossless or near-lossless behaviour: Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) with congestion control algorithms such as DCQCN.
- Advanced load balancing: packet spraying and adaptive routing spread traffic across all paths instead of pinning each flow to one.
NVIDIA Spectrum-X
An end-to-end Ethernet platform combining Spectrum switches with SuperNICs, adaptive routing, congestion control and telemetry tuned for AI collectives. NVIDIA positions it for very large AI factories that want Ethernet operations with InfiniBand-like performance.
Ultra Ethernet Consortium (UEC)
An industry group including AMD, Arista, Broadcom, Cisco, Intel, Meta, Microsoft and others, created to modernise Ethernet for AI and HPC. Its specification defines a new transport with multipath packet spraying, out-of-order delivery and modern congestion control, aiming to match InfiniBand's performance on open, multi-vendor hardware. UEC published its 1.0 specification in 2025.
Strengths of Ethernet: multi-vendor competition, huge ecosystem, familiar operations, scales to enormous sizes, one technology across the data center. Weaknesses: needs careful tuning (PFC storms, congestion), performance depends heavily on the switch and NIC features used.
InfiniBand vs Ethernet comparison
| InfiniBand | Ethernet (RoCE / Spectrum-X / UEC) | |
|---|---|---|
| RDMA | Native | RoCEv2, new UEC transport |
| Loss handling | Credit-based, lossless by design | PFC/ECN or loss-tolerant transports |
| Speeds | 400G NDR, 800G XDR | 400G and 800G, with 1.6T coming |
| Vendors | Mainly NVIDIA | NVIDIA, Broadcom, Arista, Cisco, others |
| In-network compute | SHARP | Emerging in some platforms |
| Operations | Specialist skills | Familiar to most network teams |
| Typical users | Many HPC and AI training clusters | Hyperscalers, very large AI clusters, inference |
Topology: how the switches connect
- Fat-tree / Clos: multiple tiers of switches give every server many equal paths with full bandwidth; the standard for AI.
- Rail-optimised design: GPU 0 of every server connects to the same "rail" of switches, GPU 1 to another, and so on. Because collectives often communicate between same-numbered GPUs, most traffic stays within one switch hop.
- Oversubscription: AI back-end networks are usually built non-blocking (1:1), unlike typical enterprise networks.
- Optics: at 800G, optical transceivers become a significant share of cost and power. Co-packaged optics, which put optics next to the switch chip, are arriving to improve efficiency.
Inference networking
Inference is less network-intensive than training, but large MoE models spread across many GPUs, and disaggregated prefill/decode serving, increasingly need fast RDMA networks too. See LLM inference engines compared.
How to choose
- Small to mid-size training clusters (up to a few thousand GPUs): InfiniBand is the simplest path to predictable performance.
- Hyperscale clusters or mixed workloads: AI-tuned Ethernet offers vendor choice and operational consistency.
- Cloud renters: check which interconnect a GPU cloud provides for multi-node jobs; it matters as much as the GPU model. Compare GPU prices in our GPU cloud price tracker and read What is a GPU cluster?.
Frequently asked questions
Is InfiniBand faster than Ethernet?
At the same link speed, raw bandwidth is similar. InfiniBand traditionally had lower and more predictable latency, while modern AI Ethernet platforms close much of the gap through better congestion control and load balancing.
What is RoCE?
RDMA over Converged Ethernet: a way to run RDMA on Ethernet networks. RoCEv2, which works over IP and can be routed, is the version used in AI clusters.
Does a single GPU server need InfiniBand?
No. Inside one server, GPUs talk over NVLink. Back-end networking only matters when a job spans multiple servers.
What does 800G mean?
800 gigabits per second per port, about 100 GB/s. AI clusters typically provide one such link per GPU.
Sources and further reading
- Meta Engineering, Building Meta's GenAI infrastructure (2024)
- Ultra Ethernet Consortium
- NVIDIA, Spectrum-X Ethernet networking platform
- NVIDIA, Quantum InfiniBand platform
- Meta, The Llama 3 Herd of Models (2024)




Leave a Reply