Short answer
Training a large language model requires splitting the work across many GPUs in several ways at once. Data parallelism gives each GPU a different slice of the batch; ZeRO/FSDP shards optimizer states, gradients and weights so they fit in memory; tensor parallelism splits individual matrix multiplications across GPUs; pipeline parallelism places different layers on different GPUs; context parallelism splits long sequences; and expert parallelism spreads mixture-of-experts layers. Frontier runs combine these into 3D or 4D parallelism. Meta, for example, trained Llama 3 405B on up to 16,000 H100 GPUs using tensor, context, pipeline and data parallelism together.
Why one GPU is not enough
Two limits force parallelism:
1. Memory. Mixed-precision training with Adam needs about 16 bytes per parameter for weights, gradients and optimizer states, before activations. A 70B model needs over 1.1 TB; an H100 has 80 GB. See AI training vs inference. 2. Time. Training compute is about 6 × parameters × tokens. Llama 3.1 405B needed roughly 3.8 × 10²⁵ FLOPs; on one GPU that would take centuries. See LLM training cost.
The challenge is to split the work so that GPUs spend their time computing, not waiting for each other. Every strategy trades memory savings and speed against communication, which is why fast interconnects like NVLink and NVSwitch and InfiniBand or AI Ethernet matter so much.
1. Data parallelism (DP)
Every GPU holds a full copy of the model and processes a different mini-batch. After the backward pass, GPUs average their gradients with an all-reduce operation, so all copies stay identical.
- Pros: simple, scales well, communication once per step.
- Cons: each GPU must fit the whole model plus optimizer states, which is impossible for large models.
- Communication: all-reduce of all gradients every step.
2. ZeRO and FSDP: sharded data parallelism
ZeRO (from Microsoft's DeepSpeed) removes data parallelism's memory redundancy by sharding state across the data-parallel GPUs:
| Stage | What is sharded | Memory per GPU (approx.) |
|---|---|---|
| ZeRO-1 | Optimizer states | Large saving (optimizer is ~12 of 16 bytes/param) |
| ZeRO-2 | + Gradients | Further saving |
| ZeRO-3 / FSDP | + Weights | ~16 bytes × params ÷ number of GPUs |
With ZeRO-3, each GPU gathers the weights it needs for a layer just before using it, then discards them. PyTorch's Fully Sharded Data Parallel (FSDP) implements the same idea natively.
- Pros: trains models far larger than one GPU's memory using only data-parallel logic.
- Cons: more communication (all-gather weights, reduce-scatter gradients); works best with fast interconnects and large batches.
3. Tensor parallelism (TP)
Tensor parallelism splits individual layers. A large matrix multiplication is divided by columns or rows across GPUs, each computing part of the result. NVIDIA's Megatron-LM showed an efficient pattern for transformers: split the first MLP matrix by columns and the second by rows, so only one all-reduce is needed per MLP block; attention heads are split across GPUs the same way.
- Pros: reduces memory for weights and activations per GPU; lowers latency per step.
- Cons: communication inside every layer, multiple times per forward and backward pass.
- Rule of thumb: keep tensor parallelism within a single NVLink domain, typically the 8 GPUs of one server or the 72 GPUs of a GB200 NVL72 rack, because it needs the highest bandwidth.
4. Pipeline parallelism (PP)
Pipeline parallelism assigns different layers to different GPUs: GPU 1 runs layers 1–20, GPU 2 runs 21–40, and so on. Activations flow forward through the stages and gradients flow back.
The problem is the pipeline bubble: while GPU 1 works on a batch, later GPUs wait. The fix is splitting each batch into many micro-batches so stages work on different micro-batches at once. Schedules such as GPipe, 1F1B (one forward, one backward) and interleaved schedules shrink the bubble further.
- Pros: communication only between neighbouring stages, so it works across slower links between servers.
- Cons: bubbles waste compute; balancing layers across stages is fiddly; more micro-batches need more activation memory.
5. Sequence and context parallelism
Long sequences create huge activation memory, and attention cost grows with the square of sequence length. Sequence parallelism splits non-attention operations (such as layer norm and dropout) along the sequence dimension alongside tensor parallelism. Context parallelism goes further and splits the whole sequence across GPUs, using techniques like ring attention to pass keys and values between them. This is how models are trained with 128K-token and longer context windows.
6. Expert parallelism (EP)
In mixture-of-experts models, each MoE layer has many experts. Expert parallelism places different experts on different GPUs and routes each token to the GPUs holding its chosen experts using all-to-all communication. DeepSeek-V3, for example, spread its 256 routed experts across many GPUs.
- Pros: lets models grow to hundreds of billions of parameters with modest compute per token.
- Cons: all-to-all traffic is heavy and uneven; load balancing between experts is critical.
Putting it together: 3D and 4D parallelism
Real training runs combine strategies along different dimensions of the cluster:
| Dimension | Typical placement | Why |
|---|---|---|
| Tensor parallel | Inside a server or NVLink domain | Needs the fastest links |
| Context / sequence parallel | Inside or across nearby servers | Needed for long sequences |
| Pipeline parallel | Across servers | Only neighbour communication |
| Expert parallel | Across GPUs in the same or nearby nodes | All-to-all traffic |
| Data parallel (often with ZeRO/FSDP) | Across everything else | Scales to tens of thousands of GPUs |
Example: with 16,384 GPUs you might use tensor parallel 8 × pipeline parallel 16 × data parallel 128. Meta's Llama 3 paper describes exactly this kind of 4D parallelism (tensor, context, pipeline and data) to train the 405B model on up to 16,000 H100 GPUs.
Measuring efficiency
The key metric is model FLOPs utilization (MFU): the fraction of the GPUs' peak throughput spent on useful model maths. Large dense-model runs typically reach roughly 35–50% MFU; communication, pipeline bubbles, memory traffic and failures eat the rest. See MFU and GPU utilization explained.
Other techniques that save memory
- Activation checkpointing (recomputation): discard some activations in the forward pass and recompute them during the backward pass, trading compute for memory.
- Mixed precision and FP8 training: smaller numbers mean less memory and faster maths. See mixed precision training.
- Offloading: move optimizer states to CPU memory or NVMe (slower, but helpful on small setups).
- Gradient accumulation: simulate large batches on limited GPUs by summing gradients over several steps.
Frameworks
- Megatron-LM / Megatron Core (NVIDIA): tensor, pipeline, sequence, context and expert parallelism for large-scale training.
- DeepSpeed (Microsoft): ZeRO stages, pipeline parallelism and offloading.
- PyTorch FSDP and TorchTitan: native sharding and composable parallelism.
- JAX with XLA: used on TPUs and GPUs, with sharding annotations handled by the compiler.
Frequently asked questions
Which parallelism should I start with?
For fine-tuning on a few GPUs, start with FSDP or DeepSpeed ZeRO. Add tensor parallelism only when a single layer or the model's activations no longer fit, and keep it inside one server.
Why keep tensor parallelism within a server?
It communicates inside every layer, so it needs NVLink-class bandwidth (hundreds of GB/s to over 1 TB/s per GPU). Across a network it would leave GPUs waiting.
Is pipeline parallelism still used?
Yes, especially for very large models and clusters, because its point-to-point communication tolerates slower inter-node links. Better schedules have reduced the bubble problem.
Does inference use the same techniques?
Partly. Inference uses tensor, pipeline and expert parallelism to fit and speed up large models, but no gradients or optimizer states, so memory needs are much lower. See LLM inference engines compared.
Sources and further reading
- Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019)
- Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2019)
- Huang et al., GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (2018)
- Narayanan et al., Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021)
- Meta, The Llama 3 Herd of Models (2024)
- PyTorch, Fully Sharded Data Parallel




Leave a Reply