Edited by 
Short answer
A GPU cluster is a group of computers, each packed with GPUs, connected by a very fast network so they can work together on one task. AI labs use clusters of thousands to hundreds of thousands of GPUs to train large models, because no single GPU has enough memory or speed. What makes a cluster work is the network: GPUs inside a server or rack talk over NVLink at up to 1.8 TB/s each, and servers talk to each other over InfiniBand or AI Ethernet at 400 to 800 Gb/s per GPU, so that the whole cluster behaves like one giant computer.
Why AI needs GPU clusters
A single top GPU is not enough for modern AI, for two reasons.
- Memory: a 405-billion-parameter model needs about 810 GB just to hold its weights in BF16, and training needs several times more for gradients and optimizer states. The largest GPUs have 141 to 288 GB.
- Time: training a frontier model takes on the order of 1025 to 1026 floating-point operations. On one GPU that would take centuries. Spread over 16,000 to 100,000 GPUs, it takes weeks to months.
So the model and the work are split across many GPUs, and they must constantly exchange data. The faster that exchange, the less time GPUs spend waiting. That is the whole design problem of a GPU cluster.
The building blocks of a GPU cluster
Nodes
A node is one server. The standard AI node has eight GPUs, two CPUs, lots of system memory, fast local NVMe storage and eight or more high-speed network cards, one per GPU. NVIDIA DGX and HGX B200 systems and AMD Instinct MI355X platforms all follow this pattern.
The scale-up network: NVLink
Inside a node, GPUs connect through NVLink and NVSwitch rather than PCIe. On Blackwell, each GPU has 1.8 TB/s of NVLink bandwidth, about 14 times PCIe 5.0 x16. The GB200 NVL72 extends this to a whole rack: 72 GPUs share one NVLink domain and can read each other's memory almost as if it were their own. Engineers call this the scale-up domain. Read NVLink and NVSwitch explained and GB200 NVL72 explained.
The scale-out network: InfiniBand or Ethernet
Between nodes, every GPU gets its own network port, usually 400 Gb/s (NDR InfiniBand or 400G Ethernet) or 800 Gb/s on the newest systems. These ports connect through layers of switches into a non-blocking fat-tree, often "rail-optimised" so that GPU number 1 in every server shares the same switch, GPU number 2 the next, and so on. NVIDIA Quantum InfiniBand and Spectrum-X Ethernet are the most common choices; the Ultra Ethernet Consortium is building an open alternative. The GPUs use RDMA (remote direct memory access) to move data straight between GPU memories without involving the CPU.
Storage and front-end network
A separate front-end network connects the cluster to a parallel file system for datasets and checkpoints, and to the outside world. Keeping this traffic off the GPU network stops it slowing down training.
Software
Clusters run job schedulers such as Slurm or Kubernetes, communication libraries such as NVIDIA NCCL or AMD RCCL, and training frameworks such as PyTorch with Megatron-LM or DeepSpeed that handle the splitting of work.
How training is split across a cluster
Large training runs combine several kinds of parallelism, each mapped onto the part of the network that suits it.
| Method | What is split | Communication | Where it usually runs |
|---|---|---|---|
| Data parallelism | Each GPU group trains on different data with a copy of the model; gradients are averaged | All-reduce of gradients every step | Across nodes (scale-out) |
| Tensor parallelism | Each layer's matrices are split across GPUs | Very frequent, inside every layer | Inside an NVLink domain |
| Pipeline parallelism | Groups of layers go to different GPUs, like an assembly line | Activations passed between stages | Across nodes |
| Expert parallelism | Experts in a mixture-of-experts model live on different GPUs | All-to-all token routing | Inside or across NVLink domains |
| Sharded data parallel (ZeRO/FSDP) | Weights, gradients and optimizer states are divided instead of copied | Gathers and reduce-scatters | Across nodes |
The heaviest pattern is the all-reduce: after each training step, every GPU must combine its gradients with every other GPU's. Libraries like NCCL do this with ring and tree algorithms that keep every link busy. If the network is slow, GPUs sit idle waiting, and expensive hardware is wasted.
How big are GPU clusters?
- Small clusters: 8 to 256 GPUs, used by universities, startups and companies fine-tuning models.
- Large clusters: thousands of GPUs, used to train mid-size models and serve popular ones.
- Frontier clusters: 16,000 to more than 100,000 GPUs. Meta trained Llama 3 405B on 16,384 H100 GPUs. xAI's Colossus in Memphis reached 100,000 H100s in 2024 and has since doubled. The Stargate campus in Abilene, Texas, is planned for around a million H100-equivalents.
The scale is now limited less by money than by power: 100,000 GPUs need well over 100 MW once you add CPUs, networking and cooling. Read what is an AI data center for the buildings that house these clusters.
What limits a GPU cluster
Failures
With tens of thousands of GPUs, something breaks every few hours. Meta reported 466 job interruptions during a 54-day period of Llama 3 405B training on 16,384 GPUs, 419 of them unexpected, with faulty GPUs and their HBM memory the most common causes. Clusters handle this by saving checkpoints often, detecting bad nodes automatically and restarting quickly; Meta still achieved more than 90% effective training time.
Network bottlenecks
A single congested link can slow an entire all-reduce. That is why AI networks use lossless fabrics, adaptive routing and congestion control, and why designers try to keep the most talkative traffic inside NVLink domains.
Power and cooling
Modern racks draw around 120 kW and need liquid cooling. Power swings when thousands of GPUs start or finish a step together are large enough that operators have to coordinate with the grid.
Stragglers
In synchronous training, every step waits for the slowest GPU. One overheating or misconfigured node can slow the whole cluster until it is found and removed.
Can you rent a GPU cluster?
Yes. AWS, Google Cloud, Microsoft Azure and Oracle, as well as specialist "neoclouds" such as CoreWeave, Lambda, Nebius and Crusoe, rent clusters by the hour, from a single eight-GPU node to thousands of GPUs on a reserved contract. Renting is the normal route for startups and researchers; building your own makes sense only at very large, steady scale. Our training cost calculator shows what a run might cost.
Frequently asked questions
What is the difference between a GPU cluster and a supercomputer?
Very little now. Traditional supercomputers were built for scientific simulation and FP64 maths; AI clusters are optimised for low-precision matrix maths and huge all-reduce traffic. The hardware and networks have converged, and many of the world's fastest supercomputers are GPU clusters.
How many GPUs do I need to train an LLM?
Fine-tuning a 7B to 8B model with LoRA fits on one GPU. Training a 7B model from scratch on trillions of tokens takes hundreds of GPUs for weeks. Frontier models use tens of thousands.
Can I build a GPU cluster at home?
You can link a few PCs with 10 to 100 GbE networking and use tools like PyTorch distributed or exo to split models across them, but home networks are thousands of times slower than NVLink, so it is mainly useful for running models too large for one machine, not for fast training.
What is the difference between InfiniBand and Ethernet for AI?
InfiniBand was designed for high-performance computing and has very low latency and built-in lossless transport. Ethernet is cheaper and more widely known, and new AI-tuned Ethernet such as Spectrum-X and Ultra Ethernet closes most of the gap. Both are used in the largest clusters.
Sources and further reading
- Meta, The Llama 3 Herd of Models (training infrastructure and reliability)
- NVIDIA, NVLink and NVLink Switch
- NVIDIA, NCCL user guide
- Epoch AI, OpenAI Stargate: where the US sites stand
Leave a Reply