Skip to content
,

InfiniBand vs Ethernet for AI: RDMA, RoCE, Spectrum-X and Ultra Ethernet Explained

AI clusters live or die by their network. An in-depth comparison of InfiniBand and Ethernet for AI training and inference: RDMA and RoCE, scale-up vs scale-out, congestion control, NVIDIA Quantum and Spectrum-X, the Ultra Ethernet Consortium, topologies and how to choose.

5 min read

InfiniBand vs Ethernet for AI — cover image from TensorCore.si

Short answer

Large AI clusters connect thousands of GPUs with a back-end network that must move huge volumes of data with very low latency and no packet loss. InfiniBand, now mainly supplied by NVIDIA, has long been the default for AI training thanks to native RDMA, credit-based lossless flow control and in-network computing. Ethernet has caught up through RoCE (RDMA over Converged Ethernet), AI-tuned platforms like NVIDIA Spectrum-X and the open Ultra Ethernet standard. Both now run some of the world's largest clusters; Meta trained Llama 3 on two 24,000-GPU clusters, one using InfiniBand and one using RoCE Ethernet.

Why the network matters so much for AI

When you train a model across many GPUs, they constantly exchange data: averaging gradients, gathering sharded weights, passing activations between pipeline stages, and routing tokens to experts in mixture-of-experts models. See distributed training and parallelism.

These are collective operations (all-reduce, all-gather, reduce-scatter, all-to-all), and every GPU must wait for the slowest participant. A single congested link or dropped packet can stall thousands of GPUs worth millions of dollars. AI networking therefore cares about:

  • Bandwidth: typically 400 to 800 Gb/s per GPU today.
  • Tail latency: the slowest flows, not the average, decide step time.
  • Losslessness: retransmissions are expensive at this scale.
  • Scale: tens of thousands to over 100,000 GPUs.

Scale-up vs scale-out networks

Scale-upScale-out (back-end)Front-end
ConnectsGPUs inside a server or rackServers across the clusterCluster to storage, users, internet
TechnologyNVLink and NVSwitch, AMD Infinity FabricInfiniBand or Ethernet with RDMAStandard Ethernet
Bandwidth per GPU~900 GB/s to 1.8 TB/s (NVLink)~50–100 GB/s (400–800 Gb/s)Lower
Used forTensor and expert parallelismData and pipeline parallelismData loading, checkpoints, serving

Scale-up links are roughly 10 times faster than scale-out links, which is why bandwidth-hungry parallelism stays inside the NVLink domain; see NVLink and NVSwitch explained and GB200 NVL72. This article focuses on scale-out networking.

RDMA: the foundation

Remote Direct Memory Access (RDMA) lets one machine read or write another machine's memory directly, bypassing the operating system and CPU. With GPUDirect RDMA, the network card moves data straight to and from GPU memory. This cuts latency and frees CPUs. Both InfiniBand and modern AI Ethernet are built around RDMA; libraries such as NVIDIA's NCCL use it to run collectives across the cluster.

InfiniBand

InfiniBand was designed for high-performance computing and has native RDMA, credit-based flow control (a sender only transmits when the receiver has buffer space, so packets are not dropped), adaptive routing and centralised management through a subnet manager.

NVIDIA's InfiniBand platforms include Quantum-2 (400 Gb/s NDR) and Quantum-X800 (800 Gb/s XDR), with ConnectX network adapters. Its SHARP technology performs parts of reductions inside the switches, reducing traffic for all-reduce operations.

Strengths: mature, very low latency, predictable performance, excellent tooling for HPC and AI. Weaknesses: effectively one main vendor, specialised skills, often higher cost, separate from the Ethernet used elsewhere in the data center.

Ethernet for AI

Standard Ethernet is "best effort": switches drop packets when queues overflow. For AI, Ethernet adds:

  • RoCEv2: RDMA carried over UDP/IP so it can be routed across large networks.
  • Lossless or near-lossless behaviour: Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) with congestion control algorithms such as DCQCN.
  • Advanced load balancing: packet spraying and adaptive routing spread traffic across all paths instead of pinning each flow to one.

NVIDIA Spectrum-X

An end-to-end Ethernet platform combining Spectrum switches with SuperNICs, adaptive routing, congestion control and telemetry tuned for AI collectives. NVIDIA positions it for very large AI factories that want Ethernet operations with InfiniBand-like performance.

Ultra Ethernet Consortium (UEC)

An industry group including AMD, Arista, Broadcom, Cisco, Intel, Meta, Microsoft and others, created to modernise Ethernet for AI and HPC. Its specification defines a new transport with multipath packet spraying, out-of-order delivery and modern congestion control, aiming to match InfiniBand's performance on open, multi-vendor hardware. UEC published its 1.0 specification in 2025.

Strengths of Ethernet: multi-vendor competition, huge ecosystem, familiar operations, scales to enormous sizes, one technology across the data center. Weaknesses: needs careful tuning (PFC storms, congestion), performance depends heavily on the switch and NIC features used.

InfiniBand vs Ethernet comparison

InfiniBandEthernet (RoCE / Spectrum-X / UEC)
RDMANativeRoCEv2, new UEC transport
Loss handlingCredit-based, lossless by designPFC/ECN or loss-tolerant transports
Speeds400G NDR, 800G XDR400G and 800G, with 1.6T coming
VendorsMainly NVIDIANVIDIA, Broadcom, Arista, Cisco, others
In-network computeSHARPEmerging in some platforms
OperationsSpecialist skillsFamiliar to most network teams
Typical usersMany HPC and AI training clustersHyperscalers, very large AI clusters, inference

Topology: how the switches connect

  • Fat-tree / Clos: multiple tiers of switches give every server many equal paths with full bandwidth; the standard for AI.
  • Rail-optimised design: GPU 0 of every server connects to the same "rail" of switches, GPU 1 to another, and so on. Because collectives often communicate between same-numbered GPUs, most traffic stays within one switch hop.
  • Oversubscription: AI back-end networks are usually built non-blocking (1:1), unlike typical enterprise networks.
  • Optics: at 800G, optical transceivers become a significant share of cost and power. Co-packaged optics, which put optics next to the switch chip, are arriving to improve efficiency.

Inference networking

Inference is less network-intensive than training, but large MoE models spread across many GPUs, and disaggregated prefill/decode serving, increasingly need fast RDMA networks too. See LLM inference engines compared.

How to choose

  • Small to mid-size training clusters (up to a few thousand GPUs): InfiniBand is the simplest path to predictable performance.
  • Hyperscale clusters or mixed workloads: AI-tuned Ethernet offers vendor choice and operational consistency.
  • Cloud renters: check which interconnect a GPU cloud provides for multi-node jobs; it matters as much as the GPU model. Compare GPU prices in our GPU cloud price tracker and read What is a GPU cluster?.

Frequently asked questions

Is InfiniBand faster than Ethernet?

At the same link speed, raw bandwidth is similar. InfiniBand traditionally had lower and more predictable latency, while modern AI Ethernet platforms close much of the gap through better congestion control and load balancing.

What is RoCE?

RDMA over Converged Ethernet: a way to run RDMA on Ethernet networks. RoCEv2, which works over IP and can be routed, is the version used in AI clusters.

Does a single GPU server need InfiniBand?

No. Inside one server, GPUs talk over NVLink. Back-end networking only matters when a job spans multiple servers.

What does 800G mean?

800 gigabits per second per port, about 100 GB/s. AI clusters typically provide one such link per GPU.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *