Skip to content
,

MFU Explained: Model FLOPs Utilization and Why GPUs Never Hit Peak TFLOPS

Why does a GPU rated at 989 TFLOPS deliver far less in real AI training? An in-depth guide to model FLOPs utilization (MFU), hardware FLOPs utilization (HFU) and memory bandwidth utilization (MBU): how to calculate them, typical values, what wastes GPU time and how to improve it.

5 min read

MFU and GPU Utilization — cover image from TensorCore.si

Short answer

Model FLOPs utilization (MFU) measures how much of a GPU's peak compute is spent on the model's useful maths: achieved model FLOPs per second ÷ theoretical peak FLOPs per second. Large-scale LLM training typically reaches about 35–50% MFU; Meta reported 38–43% for Llama 3 405B on H100s, and Google reported 46.2% for PaLM 540B on TPUs. The rest is lost to communication, memory traffic, pipeline bubbles, inefficient kernels and failures. For inference, the more useful metric is often memory bandwidth utilization (MBU), because decoding is limited by memory, not compute.

Why "GPU utilization" in nvidia-smi is misleading

The utilization percentage shown by nvidia-smi means "a kernel was running during the sampling period", not "the tensor cores were busy". A GPU can show 100% utilization while doing only a small fraction of its possible maths, for example while running memory-bound kernels or waiting inside communication kernels. That is why AI engineers use MFU.

How to calculate MFU for training

For a dense transformer with N parameters, training costs about 6 FLOPs per parameter per token (2 for the forward pass, 4 for the backward pass), plus attention FLOPs that grow with sequence length:

FLOPs per token ≈ 6N + 12 × layers × d_model × seq_len   (attention term)
MFU = (tokens per second × FLOPs per token) / (number of GPUs × peak FLOPs per GPU)

Worked example

You train an 8B model on 64 H100 GPUs and process 1.2 million tokens per second across the cluster, ignoring the attention term for simplicity:

  • Model FLOPs per second = 1.2 × 10⁶ × 6 × 8 × 10⁹ = 5.76 × 10¹⁶
  • Peak = 64 × 989 × 10¹² (dense BF16) = 6.33 × 10¹⁶
  • MFU ≈ 91%, which is unrealistically high, so either the throughput number or the peak assumption needs checking.

A realistic run might achieve 500,000 tokens per second on those 64 GPUs: 2.4 × 10¹⁶ ÷ 6.33 × 10¹⁶ ≈ 38% MFU.

Always use dense peak figures at the precision you actually train in. Spec sheets often quote sparse numbers that are twice as high; see structured sparsity explained and how to read a GPU spec sheet.

MFU vs HFU

  • MFU counts only the FLOPs the model mathematically requires.
  • HFU (hardware FLOPs utilization) counts all FLOPs the hardware actually performs, including extra work such as recomputing activations for activation checkpointing.

HFU is always at least as high as MFU. MFU is preferred for comparing systems because it cannot be inflated by doing redundant work.

Typical MFU values

WorkloadTypical MFUNotes
Large dense LLM pre-training, well tuned~35–50%PaLM 540B: 46.2%; Llama 3 405B: 38–43%
Mixture-of-experts trainingOften lowerAll-to-all communication and load imbalance
Small-scale fine-tuning on a few GPUs~20–50%Depends on batch size and sequence length
Training with FP8Lower MFU vs FP8 peak, but higher absolute speedFP8 peak is 2× BF16 peak
Batch-1 LLM decodingOften under 5%Memory-bound; use MBU instead

Where the missing FLOPs go

1. Communication: waiting for gradient all-reduces, weight all-gathers in FSDP, tensor-parallel exchanges and MoE all-to-all traffic. See distributed training and parallelism and InfiniBand vs Ethernet for AI. 2. Memory-bound operations: layer norm, softmax, activation functions, optimizer steps and data movement do little maths per byte. See the roofline model simulator. 3. Pipeline bubbles: stages waiting for micro-batches. 4. Small matrix shapes: matrices that do not fill the GPU's tensor core tiles waste capacity. 5. Kernel launch and Python overhead: many small operations instead of fused kernels. 6. Data loading and checkpointing: GPUs waiting for input or saving state. 7. Failures and restarts: at tens of thousands of GPUs, hardware faults are frequent; time lost to restarts lowers effective utilization over a whole run. 8. Power and thermal limits: GPUs may run below boost clocks under sustained load.

How to improve MFU

  • Use fused, optimised kernels: FlashAttention, fused optimizers, torch.compile, and well-tuned GEMM libraries.
  • Overlap communication with computation: start sending gradients for finished layers while others are still computing.
  • Pick good parallelism layouts: keep tensor parallelism inside NVLink domains; use enough micro-batches to shrink pipeline bubbles.
  • Use larger batches and sequence lengths where the training recipe allows, so matrices fill the hardware.
  • Choose hardware-friendly dimensions: hidden sizes and head dimensions that are multiples of 64 or 128.
  • Lower precision: mixed precision and FP8 raise absolute throughput even if MFU against the higher FP8 peak looks lower.
  • Fast data pipelines and asynchronous checkpointing.
  • Monitor with profilers: Nsight Systems, PyTorch Profiler and per-step tokens-per-second logs.

MBU: the inference metric

When generating tokens one at a time, each step reads all model weights (and the KV cache) from memory. The limit is bandwidth, so the useful metric is:

MBU = (bytes read per token × tokens per second) / peak memory bandwidth

Example: a 7B model in FP16 (14 GB) generating 150 tokens per second on an H100 (3.35 TB/s): 14 GB × 150 ÷ 3,350 GB/s ≈ 63% MBU. Good inference engines reach 60–80% MBU at small batch sizes. Batching more users raises compute use and pushes serving towards the compute-bound regime, where MFU becomes relevant again. See prefill vs decode, LLM inference engines compared and our LLM inference simulator.

Why MFU matters for cost

Training cost scales with GPU-hours. Raising MFU from 30% to 45% cuts the GPU-hours for the same model by a third. At frontier scale that can be tens of millions of dollars. Use MFU together with rental prices from our GPU cloud price tracker and the formulas in LLM training cost to estimate real budgets:

GPU-hours ≈ 6 × N × tokens / (peak FLOPs per GPU × MFU × 3,600)

Frequently asked questions

What is a good MFU?

For large-scale LLM pre-training, 40% or more is considered strong; over 50% is excellent. Small or communication-heavy jobs often see less.

Why can't GPUs reach 100% MFU?

Real models include many operations that are not large matrix multiplications, data must move between memory and GPUs, and GPUs in a cluster must synchronise. Peak figures assume perfect conditions that never occur in full training runs.

Do newer GPUs have higher MFU?

Not necessarily. Compute has grown faster than memory bandwidth and networking, so keeping faster GPUs busy is harder. Software and system design often decide MFU more than the chip.

Is MFU used for inference?

For high-throughput batched serving, yes. For single-user generation, memory bandwidth utilization is more meaningful.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *