Mixed Precision Training Explained: FP16, BF16 and FP8 on Tensor Cores

Why not just train in FP32?

FP32 (32-bit floating point) was the default for deep learning for years. It is accurate, but it is slow and memory hungry compared with lower precisions:

• Speed: tensor cores run FP16 and BF16 matrix maths far faster than FP32. On an H100, dense BF16 tensor core throughput is roughly 15 times the plain FP32 vector rate.
• Memory: a 16-bit number uses half the memory of a 32-bit one, so activations take half the space and you can use bigger batches or longer sequences.
• Bandwidth: moving half as many bytes speeds up memory-bound operations too.

The catch is that 16-bit formats have less precision and, in FP16's case, a much smaller range. Train entirely in FP16 and small gradient values round down to zero, weight updates get lost and training can diverge. Mixed precision gets the speed without those problems.

The three ingredients of mixed precision

The method was set out in the 2017 paper "Mixed Precision Training" by researchers from Baidu and NVIDIA. It has three parts.

1. FP32 master weights

The optimizer keeps a master copy of every weight in FP32. Each step, the weights are cast to 16-bit for the forward and backward passes, and the resulting gradients are applied to the FP32 copy. This matters because weight updates are often tiny compared with the weights themselves; in FP16 those updates would round away to nothing.

2. Low-precision maths where it is safe

Matrix multiplications and convolutions, which make up most of the compute, run in 16-bit on tensor cores and accumulate their results in FP32 internally. Operations that need more precision, such as softmax, layer normalisation, loss calculation and large reductions, stay in FP32. Frameworks keep lists of which operations are safe to run in low precision.

3. Loss scaling (for FP16)

FP16 can only represent numbers down to about 6 x 10^-8 (with reduced precision near the bottom), and many gradients are smaller than that. Loss scaling multiplies the loss by a large factor, such as 65,536, before backpropagation, so all gradients are scaled up into FP16's range. The gradients are divided by the same factor before the weight update. Dynamic loss scaling lowers the factor automatically whenever an overflow (infinity or NaN) appears and raises it again after a run of good steps.

FP16 vs BF16 vs FP8

BF16 (brain floating point, created at Google) keeps FP32's 8 exponent bits, so it covers the same range and gradients rarely underflow. It has less precision than FP16, but neural network training tolerates that well. Today most large models, including nearly all LLMs, are trained in BF16 mixed precision.

FP8 goes further. Hopper GPUs introduced FP8 tensor cores and the Transformer Engine library, which uses E4M3 for forward-pass values and E5M2 (more range) for gradients, with a scaling factor per tensor that is updated from recent maximum values. FP8 roughly doubles throughput again over BF16 and has been used to train large production models, but it needs more care than BF16. Blackwell adds even smaller formats such as FP4 and FP6, mainly aimed at inference. Our guide to FP16, BF16, FP8 and FP4 explains the bit layouts in detail.

How to use mixed precision in PyTorch

PyTorch's automatic mixed precision (AMP) handles casting and loss scaling. With BF16 on an Ampere or newer GPU:

import torch

model = MyModel().cuda()
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)

for x, y in loader:
    x, y = x.cuda(), y.cuda()
    with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
        loss = torch.nn.functional.cross_entropy(model(x), y)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad(set_to_none=True)

With FP16 (for example on older Volta or Turing GPUs), add a gradient scaler:

scaler = torch.amp.GradScaler("cuda")

for x, y in loader:
    with torch.autocast(device_type="cuda", dtype=torch.float16):
        loss = loss_fn(model(x.cuda()), y.cuda())
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad(set_to_none=True)

The model weights and optimizer state stay in FP32 here; autocast only changes the precision of operations inside the block. Hugging Face Transformers and Accelerate expose the same feature with a single flag such as bf16=True.

How much memory does mixed precision save?

It saves memory on activations, which often dominate during training, but not on the optimizer. A common rule of thumb for training with Adam in mixed precision is about 16 bytes per parameter before activations: 2 bytes for 16-bit weights, 2 for 16-bit gradients, 4 for the FP32 master weights and 8 for Adam's two FP32 moment estimates. A 7-billion-parameter model therefore needs around 112 GB just for weights and optimizer state, which is why large training jobs shard this state across many GPUs. For inference-only memory planning, see how much VRAM you need for an LLM.

Common problems and fixes

• Loss becomes NaN early: with FP16, check the GradScaler is used; or switch to BF16.
• No speedup: make sure matrix dimensions are multiples of 8 (16 for some cases) so tensor cores are used, and that the GPU actually has tensor cores for that format.
• Accuracy slightly worse: keep sensitive layers such as the final softmax and normalisation in FP32 (autocast does this by default).
• TF32: on Ampere and newer GPUs, plain FP32 matrix maths can use TF32 tensor cores; enable it with torch.backends.cuda.matmul.allow_tf32 = True for a free speedup when you cannot use 16-bit.

Frequently asked questions

Is mixed precision training less accurate?

Done properly, no. Models trained with FP16 or BF16 mixed precision generally match FP32 accuracy, which is why it is the standard for modern training.

Should I use FP16 or BF16?

Use BF16 if your hardware supports it (NVIDIA Ampere and later, Google TPUs, AMD MI200 and later). It is simpler because it rarely needs loss scaling. Use FP16 with loss scaling on older GPUs.

Can I use mixed precision on a consumer GPU?

Yes. RTX 20 series and newer GeForce cards have tensor cores; RTX 30 and later also support BF16.

Sources and further reading

• Micikevicius et al., Mixed Precision Training (ICLR 2018)
• PyTorch, Automatic Mixed Precision documentation
• Micikevicius et al., FP8 Formats for Deep Learning (2022)
• NVIDIA, Transformer Engine documentation