2:4 Structured Sparsity Explained: How Sparse Tensor Cores Double AI Throughput

What is sparsity in AI models?

A neural network layer is mostly a large matrix of weights. Research has long shown that many of those weights are close to zero and can be removed with little effect on accuracy. Setting them to exactly zero is called pruning, and a matrix with many zeros is sparse.

The problem is that random zeros are hard for hardware to exploit. A GPU still has to load the matrix, check which values are zero and skip them, and that bookkeeping usually costs more than the multiplications it saves. Unstructured sparsity saves memory in theory but rarely makes a GPU faster unless more than about 90% of the weights are zero.

Structured sparsity fixes this by forcing the zeros into a fixed, predictable pattern that the tensor cores can handle directly.

How 2:4 sparsity works

The rule is simple: in every block of four consecutive weights along a row, at most two can be non-zero. So exactly 50% of the weights are zero, spread evenly through the matrix.

The hardware stores a sparse matrix in a compressed form:

• The values: only the two non-zero weights from each block of four, so the value array is half the original size.
• The metadata: a 2-bit index for each kept value that records its position (0 to 3) inside the block.

When a sparse tensor core multiplies this compressed matrix by the activations, it uses the metadata to pick the matching activation values and performs only half the multiply-adds. The work per instruction halves, so throughput can double.

The memory saving is therefore a little under half, which also helps on memory-bandwidth-bound workloads such as LLM decoding.

Why spec sheets show "with sparsity" numbers

Since Ampere, NVIDIA data center spec sheets list two figures for tensor core performance: dense and "with sparsity". The sparse number is exactly double. For example, the H100 SXM is rated at about 1,979 TFLOPS of dense FP8 and about 3,958 TFLOPS of FP8 with sparsity. AMD's Instinct MI300 series also lists structured-sparsity figures.

When you compare GPUs, always compare dense to dense. Most models people run are dense, so the sparse figure is an upper limit you only approach with a properly pruned model. Our guide on how to read a GPU spec sheet covers this and other marketing traps.

Which GPUs support structured sparsity?

Older generations such as Volta and Turing have tensor cores but no sparse mode. See tensor core generations for the full history.

How to make a model 2:4 sparse

You cannot just zero half the weights of a trained model and expect it to work. The usual recipe has three steps:

• Train (or download) a dense model. Start from normal, fully trained weights.
• Prune to the 2:4 pattern. In each group of four weights, keep the two with the largest magnitude and set the other two to zero.
• Fine-tune with the mask fixed. Continue training while keeping the pruned positions at zero, so the remaining weights adapt and recover the lost accuracy.

NVIDIA's research paper on this method reported that across a wide range of vision, language and speech networks, the pruned and retrained models matched the accuracy of the dense originals. NVIDIA's ASP (Automatic SParsity) tool in the Apex library automates the pruning and masking for PyTorch.

For large language models, retraining is expensive, so one-shot methods such as SparseGPT and Wanda prune to 2:4 without full retraining. They work, but at 50% sparsity LLMs usually lose some quality, so test carefully on your own tasks.

Running a 2:4 model in PyTorch

Recent PyTorch versions can convert a pruned weight into the semi-structured sparse format that uses sparse tensor cores:

import torch
from torch.sparse import to_sparse_semi_structured

linear = torch.nn.Linear(4096, 4096).half().cuda()
# prune: keep the 2 largest values in every group of 4
w = linear.weight.detach()
groups = w.abs().reshape(-1, 4)
mask = torch.zeros_like(groups, dtype=torch.bool)
mask.scatter_(1, groups.topk(2, dim=1).indices, True)
linear.weight = torch.nn.Parameter(to_sparse_semi_structured(w * mask.reshape(w.shape)))

For production inference, NVIDIA TensorRT and TensorRT-LLM can build engines that use sparse kernels automatically when the weights follow the 2:4 pattern.

How much faster is it really?

The tensor core math is up to twice as fast, but a full model includes attention, normalisation, activation functions and data movement that sparsity does not speed up. In practice:

• Large matrix multiplications see the biggest gains, often 1.3x to 1.8x.
• Whole-model inference speedups are usually lower, depending on how much time is spent in the sparse layers.
• Small batch sizes and small matrices may see little or no benefit.

The best results come from large, compute-bound layers, such as the feed-forward blocks of transformers at higher batch sizes.

Sparsity vs quantization

Sparsity and quantization both make models cheaper, in different ways. Quantization uses fewer bits per weight (FP8, INT4 or FP4); sparsity removes weights entirely. They can be combined: a 2:4 sparse FP8 model gets the benefits of both, which is how the highest "sparse FP8" and "sparse FP4" numbers on spec sheets are reached. Quantization is far more common today because it is easier to apply without hurting quality. For training-side precision, see mixed precision training.

Frequently asked questions

What does 2:4 sparsity mean?

Two of every four consecutive weights are zero, giving 50% sparsity in a regular pattern that sparse tensor cores can accelerate.

Do consumer RTX GPUs support sparsity?

Yes. RTX 30, 40 and 50 series tensor cores support 2:4 structured sparsity, although most consumer software does not use it yet.

Does sparsity reduce accuracy?

Pruning without retraining usually does. Pruning followed by fine-tuning typically recovers almost all of the original accuracy for vision and smaller language models; very large LLMs pruned in one shot tend to lose some quality.

Is the "with sparsity" TFLOPS figure realistic?

Only as a ceiling. It assumes every matrix multiplication uses a 2:4 sparse model, so compare dense figures when choosing hardware.

Sources and further reading

• NVIDIA, Accelerating Sparse Deep Neural Networks (Mishra et al., 2021)
• NVIDIA Developer Blog, Structured Sparsity in the NVIDIA Ampere Architecture
• PyTorch, Accelerating BERT with semi-structured (2:4) sparsity
• Frantar and Alistarh, SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023)