Key findings
On the standard llama.cpp test (Llama 2 7B, 4-bit), the RTX 5090 generates about 300 tokens per second, the RTX PRO 6000 Blackwell and H100 about 281, the RTX 4090 about 189 and DGX Spark about 57. Generation speed follows memory bandwidth: the GPUs we could analyse used 60% to 82% of their peak bandwidth, with a median of 74%. That supports the 70% our VRAM calculator assumes. Prompt processing follows compute instead, which is why an RTX 4090 processes prompts 2.7 times faster than an RTX 3090 but generates only 17% faster.
Measured llama.cpp results
These results come from the public llama.cpp CUDA benchmark, where users run the same test on their own hardware and post the output. The test model is Llama 2 7B in the Q4_0 format (3.56 GiB), run fully on the GPU with flash attention on. pp512 is prompt processing of 512 tokens; tg128 is generating 128 tokens, the speed you see while a model types.
| GPU | Prompt processing (pp512, tokens/s) | Generation (tg128, tokens/s) | Memory bandwidth | Bandwidth used in generation |
|---|---|---|---|---|
| GeForce RTX 5090 | 14,970 | 300.4 | 1,792 GB/s | 64% |
| RTX PRO 6000 Blackwell | 16,619 | 281.1 | 1,792 GB/s | 60% |
| H100 80 GB | 11,263 | 280.7 | Variant not stated | – |
| A100 80 GB | 5,286 | 200.9 | Variant not stated | – |
| GeForce RTX 4090 | 14,771 | 189.0 | 1,008 GB/s | 72% |
| GeForce RTX 5080 | 9,488 | 184.7 | 960 GB/s | 74% |
| GeForce RTX 5070 Ti | 8,420 | 182.4 | 896 GB/s | 78% |
| GeForce RTX 3090 | 5,560 | 161.9 | 936 GB/s | 66% |
| NVIDIA L40 | 10,098 | 153.8 | 864 GB/s | 68% |
| GeForce RTX 4080 | 9,206 | 143.5 | 717 GB/s | 77% |
| GeForce RTX 3080 | 5,570 | 140.0 | Variant not stated | – |
| GeForce RTX 4070 Ti SUPER | 7,612 | 132.9 | 672 GB/s | 76% |
| GeForce RTX 3060 12 GB | 2,408 | 76.9 | 360 GB/s | 82% |
| NVIDIA DGX Spark | 3,661 | 56.7 | 273 GB/s | 79% |
Results are from different users and llama.cpp versions, so treat differences of a few percent as noise. "Bandwidth used" is our calculation: generation speed × 3.825 GB of weights ÷ the GPU's rated bandwidth. We left it out where the result does not say which memory variant was tested.
What the numbers show
Generation speed is set by memory bandwidth
Generating each token means reading all of the model's weights from memory, so tokens per second should track bandwidth, and they do. The RTX 5090 and RTX PRO 6000 have the same 1,792 GB/s and generate at almost the same speed, even though the PRO card has 3 times the memory. The RTX 3060 has a fifth of the RTX 5090's bandwidth and generates at about a quarter of its speed. Read prefill vs decode for why.
Faster cards use less of their bandwidth on small models
Mid-range cards reached 74% to 82% of their bandwidth, while the 1.8 TB/s cards reached 60% to 64%. At 300 tokens per second, each token takes about 3 milliseconds, and fixed overheads such as launching GPU kernels become a bigger share of that time. With larger models, which take longer per token, the fastest cards get closer to their peak.
Prompt processing is set by compute
Prompt processing is large matrix math, so it follows Tensor Core throughput. Newer cards process prompts much faster than older cards with similar bandwidth: an RTX 4090 manages 14,771 tokens per second against 5,560 for an RTX 3090, and an RTX 5070 Ti beats the RTX 3090 by about 50% with slightly less bandwidth. That matters most for long prompts and documents. Compare compute across chips in the GPU specs database.
DGX Spark trades speed for capacity
DGX Spark is the slowest GPU here at generation, because its 273 GB/s unified memory is far slower than GDDR7 or HBM. Its strength is 128 GB of memory, enough for models that do not fit on any consumer card at all. See DGX Spark explained.
Data-center results
| Measurement | Result | Source |
|---|---|---|
| AMD MI355X vs NVIDIA B200, MLPerf Inference 6.0, Llama 2 70B Interactive (8 GPUs) | MI355X at 119% of B200 | AMD's comparison of MLCommons-verified results |
| AMD MI355X vs NVIDIA B200, MLPerf Inference 6.0, gpt-oss-120b Offline (8 GPUs) | MI355X at 111% of B200 | AMD's comparison of MLCommons-verified results |
| FlashAttention-3 on H100, FP16 | Up to about 740 TFLOPS, 75% of peak | FlashAttention-3 authors |
| FlashAttention-4 on B200, BF16 | Up to about 1,613 TFLOPS, 71% of peak | FlashAttention-4 paper |
| Llama 3.1 405B training on H100s | About 35% of BF16 peak across the whole run (our calculation from Meta's 30.84 million GPU-hours) | Meta model card |
More detail is in MI355X vs B200, FlashAttention explained and how much it costs to train an LLM.
Run the benchmark yourself
You can reproduce the llama.cpp numbers on your own GPU in a few minutes. Build llama.cpp with CUDA, download the Llama 2 7B Q4_0 GGUF file, then run:
llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1This reports pp512 and tg128 with flash attention off and on. Close other GPU programs first, and note your driver version and llama.cpp build number.
Share your results
We add verified results from readers. Email hi@tensorcore.si with the full llama-bench output, your GPU model and memory size, driver version and operating system. We check each submission against similar hardware before adding it, and credit you by name if you wish. See our editorial policy for how we handle data.