LLM Inference Simulator: How GPUs Generate Tokens

This simulator models what happens inside a GPU when a large language model answers a prompt. Change any setting and every number updates. Press Play to watch the timeline: the purple block is prefill, the green ticks are tokens being generated.

How to use it

• Mouse: change any dropdown or slider, then press Play or Step.
• Keyboard: click the simulator once, then use Space to play or pause, ← and → to move one token at a time, ↑ and ↓ to change the number of users, [ and ] for prompt length, G for the next GPU and M for the next model.

What the simulator shows

Prefill: reading your prompt

The whole prompt is processed in one big parallel pass. That is heavy matrix maths, so it keeps the Tensor Cores busy and its time grows with prompt length and users. Very long prompts also make attention expensive, because every token is compared with every other token. The result is the time to first token.

Decode: writing the answer

After that, the model writes one token at a time. For every token it must read all of its weights from memory, plus the KV cache of everything written so far. The maths per token is small, so the GPU mostly waits for memory. That is why the "Where the time goes" chart is usually dominated by green and purple bars, and why tokens per second tracks memory bandwidth. See our measured benchmarks for real results.

Batching: why many users are cheap

When 8 or 64 people share a GPU, the weights are read once per step and used for everyone. Total throughput climbs almost in proportion, while each person's speed barely changes, until the KV cache or the maths catches up. That is how AI companies serve millions of users.

The KV cache: the hidden memory hog

Each token in the context stores a key and a value for every layer. For Llama 3.1 8B in FP16 that is 128 KB per token, so one user at 128K tokens needs about 16 GB just for the cache. Push the prompt slider up with several users and watch the purple part of the memory bar take over. Read KV cache explained for more.

Five experiments to try

1. Bandwidth wins: keep Llama 3.1 8B at 4-bit and switch between RTX 4090, RTX 5090 and H200. Tokens per second follows memory bandwidth, not TFLOPS.
2. Precision is speed: switch weights from FP16 to 4-bit. The model gets about 3.5 times smaller and decode gets about that much faster.
3. The batching free lunch: press ↑ repeatedly. Watch total throughput rise while per-user speed holds, then find the point where the bound changes to compute.
4. Long context hurts: set 32 users and raise the prompt to 32K. The KV cache, not the weights, now decides whether it fits.
5. Too big for one GPU: pick Llama 3.3 70B in FP16 on one H100. It does not fit. Change to 2 or 4 GPUs and see the memory split across them.

How the numbers are calculated

• Weights = parameters × bytes per weight (2 for FP16, 1 for FP8, about 0.56 for 4-bit GGUF).
• KV cache per token = 2 × layers × KV heads × head size × bytes.
• Decode step = the larger of (weights + KV cache) ÷ (bandwidth × 70%) and the step's maths ÷ (peak TFLOPS × 60%), plus a small sync cost per layer when using several GPUs.
• Prefill = 2 × parameters × prompt tokens, plus attention, ÷ (peak TFLOPS × 60%).
• Overhead adds 1 GB per GPU plus 6% of the weights for activations and the software runtime.

These are estimates. Real engines such as vLLM, TensorRT-LLM and llama.cpp differ by kernel quality, batching method and settings, typically within about 30%. GPU figures come from our GPU specs database; for GeForce cards the simulator uses the FP16 rate with FP16 accumulation, which inference engines typically use. To check which models fit your own card, use the VRAM calculator.