Edited by 
Short answer
LLM inference has two phases. Prefill processes the whole prompt in one pass, doing large matrix multiplications that keep the Tensor Cores busy, so it is compute-bound and sets the time to first token. Decode then generates the answer one token at a time; each step reads all the model's weights and its KV cache from memory but does little math per byte, so it is memory-bandwidth-bound and sets tokens per second. For Llama 3.1 8B at 16-bit on an H100, prefilling a 2,000-token prompt takes about 65 ms, while decode for one user tops out near 150 to 200 tokens per second.
Prefill vs decode at a glance
| Prefill | Decode | |
|---|---|---|
| What it does | Reads the whole prompt and builds the KV cache | Generates output tokens one by one |
| Tokens per step | All prompt tokens at once | One per user |
| Limited by | Compute (Tensor Core throughput) | Memory bandwidth |
| Main metric | Time to first token (TTFT) | Tokens per second, or time per output token |
| Gets slower with | Longer prompts | Bigger models, longer contexts |
| Hardware that helps most | More TFLOPS, FP8 and FP4 | More memory bandwidth, larger batches |
Prefill: compute-bound
During prefill, every prompt token passes through every layer at the same time. Each weight is read from memory once and used for all the prompt's tokens, so the GPU does thousands of operations for every byte it reads. That is what Tensor Cores are built for.
The work is about 2 × parameters × prompt tokens operations. For Llama 3.1 8B and a 2,000-token prompt, that is about 32 trillion operations. An H100 reaching half its 989 dense FP16 TFLOPS finishes in roughly 65 milliseconds. A 100,000-token document takes 50 times longer, plus extra attention work, which is why long prompts have a noticeable wait before the first word appears.
Decode: memory-bound
During decode, each new token needs one pass through the whole model, but for a single user it does only about two operations per weight. The GPU must still read every active weight from memory for every token. For Llama 3.1 8B at 16-bit, that is about 16 GB per token. At the H100's 3.35 TB/s, the ceiling is about 210 tokens per second, and real systems reach roughly 70% of that. The Tensor Cores sit mostly idle.
That is why memory bandwidth, not TFLOPS, predicts how fast a chatbot types, and why quantization to 8 or 4 bits speeds up generation: fewer bytes to read per token. The VRAM calculator uses this to estimate speed on any GPU.
The numbers behind it: arithmetic intensity
An H100 can perform about 295 FP16 operations for every byte it reads from memory (989 TFLOPS ÷ 3.35 TB/s). Work with fewer operations per byte is limited by memory; work with more is limited by compute. Single-user decode with 16-bit weights does about 1 operation per byte. Prefill of a long prompt does hundreds to thousands. Read how to read a GPU spec sheet for the roofline model behind this.
Why servers batch users together
Serving many users at once changes decode. If 64 users generate a token in the same step, the GPU reads the weights once and uses them 64 times, raising throughput almost 64 times for the same memory traffic. With 16-bit weights, decode only becomes compute-bound at batches of a few hundred, and each user's KV cache still has to be read separately, so long contexts keep decode memory-bound.
Batching trades speed per user for total throughput. Inference engines such as vLLM, SGLang and TensorRT-LLM use continuous batching, adding and removing requests every step to keep GPUs full.
How inference systems handle both phases
- Chunked prefill: splits long prompts into pieces mixed with decode steps, so one long prompt does not stall everyone else.
- Disaggregated serving: runs prefill and decode on separate GPUs, each tuned for its phase. NVIDIA's Dynamo software does this, and NVIDIA pairs Rubin GPUs for prefill with Groq LPUs for decode.
- Prefix caching: reuses the KV cache for repeated prompt beginnings, such as a long system prompt, skipping that prefill.
- Speculative decoding: a small draft model proposes several tokens that the big model checks in one pass, turning decode steps into a mini-prefill.
- KV cache quantization and paging: shrink and organize the KV cache so more users fit in memory.
- FlashAttention: speeds up attention in both phases by keeping data on chip. See FlashAttention explained.
Frequently asked questions
What is time to first token (TTFT)?
The delay between sending a prompt and seeing the first output token. It is mostly prefill time, plus any time waiting in a queue, so it grows with prompt length and server load.
Why is the first token slow and the rest fast?
The first token waits for the whole prompt to be processed. After that, each token needs only one decode step, which takes a few milliseconds to tens of milliseconds depending on model size and hardware.
Does a faster GPU speed up decode?
Only if it has more memory bandwidth. A GPU with twice the TFLOPS but the same bandwidth generates tokens at about the same speed for one user. Compare bandwidth in the GPU specs database.
Is prefill or decode more expensive?
Per token, prefill is far cheaper because it uses the hardware efficiently. Decode usually dominates total time and cost, especially for reasoning models that generate long answers.
