Edited by 
Short answer
GPU memory bandwidth is how many bytes per second the GPU can read from and write to its own memory (VRAM). It equals bus width in bits ÷ 8 × data rate per pin; an RTX 4090 has 384 ÷ 8 × 21 Gbps = about 1,008 GB/s. For running large language models one user at a time, bandwidth matters more than TFLOPS, because the GPU must read every weight for every token. A quick estimate is tokens per second ≈ bandwidth ÷ model size in bytes, then expect roughly 60 to 80% of that in practice.
What is memory bandwidth?
A GPU has two key speeds: how fast it can calculate (measured in TFLOPS) and how fast it can feed data to those calculations (memory bandwidth, measured in GB/s or TB/s). Think of a factory: TFLOPS is how fast the machines work, and bandwidth is how fast the conveyor belt delivers raw materials. If the belt is too slow, the machines sit idle.
Bandwidth is set by the memory technology and the width of the connection between the GPU and its VRAM.
How to calculate GPU memory bandwidth
The formula is:
Bandwidth (GB/s) = memory bus width (bits) ÷ 8 × effective data rate (Gbps per pin)
| GPU | Memory | Bus width | Data rate | Bandwidth |
|---|---|---|---|---|
| RTX 4060 Ti | GDDR6 | 128-bit | 18 Gbps | 288 GB/s |
| RTX 4090 | GDDR6X | 384-bit | 21 Gbps | 1,008 GB/s |
| RTX 5090 | GDDR7 | 512-bit | 28 Gbps | 1,792 GB/s |
| H100 SXM | HBM3 | 5,120-bit | about 5.2 Gbps | about 3,350 GB/s |
| H200 | HBM3E | 6,144-bit | about 6.3 Gbps | 4,800 GB/s |
| B200 | HBM3E | 8,192-bit | about 7.8 Gbps | 8,000 GB/s |
Data center GPUs reach their huge numbers with extremely wide HBM interfaces rather than faster pins. Apple silicon and other unified-memory systems sit in between: the M4 Max offers up to 546 GB/s and the M3 Ultra up to 819 GB/s.
Why bandwidth limits LLM speed
When an LLM generates text, each new token requires a full pass through the model. For a single user, that means reading every weight from memory once per token, while doing only about two floating point operations per weight. The GPU spends most of its time waiting for data, not calculating. This is called being memory-bound.
This is the decode phase. The prefill phase, when the model reads your prompt, processes many tokens at once and is usually compute-bound instead. Our article on prefill vs decode explains the difference, and the roofline model simulator lets you see where a workload sits.
Estimating tokens per second
Because decode reads all the weights once per token, you can estimate the maximum speed:
Max tokens/s ≈ memory bandwidth ÷ bytes read per token (≈ model size in memory)
Examples for a single request:
| Model and format | Approx. size | RTX 4090 (1,008 GB/s) | RTX 5090 (1,792 GB/s) | H100 (3,350 GB/s) |
|---|---|---|---|---|
| 8B at 4-bit | 5 GB | about 200 tok/s ceiling | about 360 | about 670 |
| 8B at FP16 | 16 GB | about 63 | about 112 | about 210 |
| 32B at 4-bit | 19 GB | about 53 | about 94 | about 176 |
| 70B at 4-bit | 40 GB | does not fit | does not fit | about 84 |
Real software typically reaches 60 to 80% of these ceilings, and the KV cache adds more reads as the conversation grows. The pattern is clear, though: halve the model size through quantization and you roughly double the speed; double the bandwidth and you double it again.
When compute matters more than bandwidth
Bandwidth is the bottleneck when the GPU does little maths per byte. Things change when you:
- Batch many users together: the weights are read once and reused for every request in the batch, so serving systems such as vLLM become compute-bound at large batch sizes.
- Train models: training processes large batches of tokens and is usually limited by tensor core throughput.
- Process long prompts: prefill handles thousands of tokens in parallel.
The ratio of operations to bytes is called arithmetic intensity. Each GPU has a balance point (peak FLOPS ÷ bandwidth). For an H100 it is roughly 300 FP16 operations per byte; workloads below that are memory-bound.
How GPUs work around limited bandwidth
- Caches: large L2 caches (50 MB on H100, 96 MB on RTX 5090) keep frequently used data on chip.
- Kernel fusion: techniques like FlashAttention keep intermediate results in fast on-chip memory instead of writing them back to VRAM.
- Lower precision: FP8 and FP4 formats move fewer bytes per weight.
- Speculative decoding: a small draft model proposes several tokens that the large model checks in one pass, reusing each weight read for several tokens.
Bandwidth vs capacity: which matters more?
For local AI, check capacity first, because a model that does not fit in VRAM either fails or spills to much slower system memory. Among GPUs that fit your model, more bandwidth means faster responses. That is why a used RTX 3090 (24 GB, 936 GB/s) remains popular for local LLMs, and why Macs with large unified memory can run big models but more slowly than high-end GPUs. See how much VRAM you need for an LLM to work out capacity.
Frequently asked questions
Is higher memory bandwidth always better?
For LLM inference and other memory-bound work, yes. For compute-bound workloads, extra bandwidth helps less once the compute units are kept busy.
Does system RAM bandwidth matter?
Only if the model does not fit in VRAM and layers are offloaded to the CPU. Dual-channel DDR5 offers under 100 GB/s, so offloaded layers run far slower.
Why does my GPU reach less than its rated bandwidth?
Rated bandwidth is a theoretical peak. Real kernels typically achieve 70 to 90% of it in well-optimised code and less in general workloads.
Sources and further reading
- NVIDIA, H100 Tensor Core GPU datasheet
- NVIDIA, GeForce RTX 5090 specifications
- Williams, Waterman and Patterson, Roofline: An Insightful Visual Performance Model (2009)
- Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2023)
Leave a Reply