Skip to content
,

Data, Tensor, Pipeline and Expert Parallelism Explained: How LLMs Train on Thousands of GPUs

No single GPU can train a frontier model. An in-depth guide to data parallelism, ZeRO and FSDP, tensor parallelism, pipeline parallelism, sequence and context parallelism and expert parallelism: how each splits the work, what it costs in communication, and how they combine into 3D and 4D parallelism.

6 min read

How LLMs Train on Thousands of GPUs — cover image from TensorCore.si

Short answer

Training a large language model requires splitting the work across many GPUs in several ways at once. Data parallelism gives each GPU a different slice of the batch; ZeRO/FSDP shards optimizer states, gradients and weights so they fit in memory; tensor parallelism splits individual matrix multiplications across GPUs; pipeline parallelism places different layers on different GPUs; context parallelism splits long sequences; and expert parallelism spreads mixture-of-experts layers. Frontier runs combine these into 3D or 4D parallelism. Meta, for example, trained Llama 3 405B on up to 16,000 H100 GPUs using tensor, context, pipeline and data parallelism together.

Why one GPU is not enough

Two limits force parallelism:

1. Memory. Mixed-precision training with Adam needs about 16 bytes per parameter for weights, gradients and optimizer states, before activations. A 70B model needs over 1.1 TB; an H100 has 80 GB. See AI training vs inference. 2. Time. Training compute is about 6 × parameters × tokens. Llama 3.1 405B needed roughly 3.8 × 10²⁵ FLOPs; on one GPU that would take centuries. See LLM training cost.

The challenge is to split the work so that GPUs spend their time computing, not waiting for each other. Every strategy trades memory savings and speed against communication, which is why fast interconnects like NVLink and NVSwitch and InfiniBand or AI Ethernet matter so much.

1. Data parallelism (DP)

Every GPU holds a full copy of the model and processes a different mini-batch. After the backward pass, GPUs average their gradients with an all-reduce operation, so all copies stay identical.

  • Pros: simple, scales well, communication once per step.
  • Cons: each GPU must fit the whole model plus optimizer states, which is impossible for large models.
  • Communication: all-reduce of all gradients every step.

2. ZeRO and FSDP: sharded data parallelism

ZeRO (from Microsoft's DeepSpeed) removes data parallelism's memory redundancy by sharding state across the data-parallel GPUs:

StageWhat is shardedMemory per GPU (approx.)
ZeRO-1Optimizer statesLarge saving (optimizer is ~12 of 16 bytes/param)
ZeRO-2+ GradientsFurther saving
ZeRO-3 / FSDP+ Weights~16 bytes × params ÷ number of GPUs

With ZeRO-3, each GPU gathers the weights it needs for a layer just before using it, then discards them. PyTorch's Fully Sharded Data Parallel (FSDP) implements the same idea natively.

  • Pros: trains models far larger than one GPU's memory using only data-parallel logic.
  • Cons: more communication (all-gather weights, reduce-scatter gradients); works best with fast interconnects and large batches.

3. Tensor parallelism (TP)

Tensor parallelism splits individual layers. A large matrix multiplication is divided by columns or rows across GPUs, each computing part of the result. NVIDIA's Megatron-LM showed an efficient pattern for transformers: split the first MLP matrix by columns and the second by rows, so only one all-reduce is needed per MLP block; attention heads are split across GPUs the same way.

  • Pros: reduces memory for weights and activations per GPU; lowers latency per step.
  • Cons: communication inside every layer, multiple times per forward and backward pass.
  • Rule of thumb: keep tensor parallelism within a single NVLink domain, typically the 8 GPUs of one server or the 72 GPUs of a GB200 NVL72 rack, because it needs the highest bandwidth.

4. Pipeline parallelism (PP)

Pipeline parallelism assigns different layers to different GPUs: GPU 1 runs layers 1–20, GPU 2 runs 21–40, and so on. Activations flow forward through the stages and gradients flow back.

The problem is the pipeline bubble: while GPU 1 works on a batch, later GPUs wait. The fix is splitting each batch into many micro-batches so stages work on different micro-batches at once. Schedules such as GPipe, 1F1B (one forward, one backward) and interleaved schedules shrink the bubble further.

  • Pros: communication only between neighbouring stages, so it works across slower links between servers.
  • Cons: bubbles waste compute; balancing layers across stages is fiddly; more micro-batches need more activation memory.

5. Sequence and context parallelism

Long sequences create huge activation memory, and attention cost grows with the square of sequence length. Sequence parallelism splits non-attention operations (such as layer norm and dropout) along the sequence dimension alongside tensor parallelism. Context parallelism goes further and splits the whole sequence across GPUs, using techniques like ring attention to pass keys and values between them. This is how models are trained with 128K-token and longer context windows.

6. Expert parallelism (EP)

In mixture-of-experts models, each MoE layer has many experts. Expert parallelism places different experts on different GPUs and routes each token to the GPUs holding its chosen experts using all-to-all communication. DeepSeek-V3, for example, spread its 256 routed experts across many GPUs.

  • Pros: lets models grow to hundreds of billions of parameters with modest compute per token.
  • Cons: all-to-all traffic is heavy and uneven; load balancing between experts is critical.

Putting it together: 3D and 4D parallelism

Real training runs combine strategies along different dimensions of the cluster:

DimensionTypical placementWhy
Tensor parallelInside a server or NVLink domainNeeds the fastest links
Context / sequence parallelInside or across nearby serversNeeded for long sequences
Pipeline parallelAcross serversOnly neighbour communication
Expert parallelAcross GPUs in the same or nearby nodesAll-to-all traffic
Data parallel (often with ZeRO/FSDP)Across everything elseScales to tens of thousands of GPUs

Example: with 16,384 GPUs you might use tensor parallel 8 × pipeline parallel 16 × data parallel 128. Meta's Llama 3 paper describes exactly this kind of 4D parallelism (tensor, context, pipeline and data) to train the 405B model on up to 16,000 H100 GPUs.

Measuring efficiency

The key metric is model FLOPs utilization (MFU): the fraction of the GPUs' peak throughput spent on useful model maths. Large dense-model runs typically reach roughly 35–50% MFU; communication, pipeline bubbles, memory traffic and failures eat the rest. See MFU and GPU utilization explained.

Other techniques that save memory

  • Activation checkpointing (recomputation): discard some activations in the forward pass and recompute them during the backward pass, trading compute for memory.
  • Mixed precision and FP8 training: smaller numbers mean less memory and faster maths. See mixed precision training.
  • Offloading: move optimizer states to CPU memory or NVMe (slower, but helpful on small setups).
  • Gradient accumulation: simulate large batches on limited GPUs by summing gradients over several steps.

Frameworks

  • Megatron-LM / Megatron Core (NVIDIA): tensor, pipeline, sequence, context and expert parallelism for large-scale training.
  • DeepSpeed (Microsoft): ZeRO stages, pipeline parallelism and offloading.
  • PyTorch FSDP and TorchTitan: native sharding and composable parallelism.
  • JAX with XLA: used on TPUs and GPUs, with sharding annotations handled by the compiler.

Frequently asked questions

Which parallelism should I start with?

For fine-tuning on a few GPUs, start with FSDP or DeepSpeed ZeRO. Add tensor parallelism only when a single layer or the model's activations no longer fit, and keep it inside one server.

Why keep tensor parallelism within a server?

It communicates inside every layer, so it needs NVLink-class bandwidth (hundreds of GB/s to over 1 TB/s per GPU). Across a network it would leave GPUs waiting.

Is pipeline parallelism still used?

Yes, especially for very large models and clusters, because its point-to-point communication tolerates slower inter-node links. Better schedules have reduced the bubble problem.

Does inference use the same techniques?

Partly. Inference uses tensor, pipeline and expert parallelism to fit and speed up large models, but no gradients or optimizer states, so memory needs are much lower. See LLM inference engines compared.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *