Skip to content
,

Speculative Decoding Explained: How LLMs Generate Text 2–3× Faster Without Losing Quality

Speculative decoding speeds up LLM inference by letting a small draft model guess several tokens ahead and having the big model verify them in one pass. How it works, why it is lossless, acceptance rates, draft models vs Medusa, EAGLE and multi-token prediction, and when it helps on real GPUs.

6 min read

Speculative Decoding Explained — cover image from TensorCore.si

Short answer

Speculative decoding makes large language models generate text faster by splitting the work in two: a cheap draft model quickly proposes the next several tokens, and the large target model checks all of them in a single forward pass, keeping every token it agrees with. Because the big model's verification step is mathematically exact, the output has the same quality and distribution as normal decoding, typically with 2–3× faster token generation for a single user. It works because decoding is limited by memory bandwidth, leaving GPU compute idle that verification can use for free.

The problem: decoding wastes the GPU

When a large language model generates text, it produces one token at a time. Each step requires reading essentially all of the model's weights from GPU memory. For a 70B-parameter model in FP16, that is about 140 GB of reads to produce a single token.

Modern GPUs can do far more arithmetic than their memory can feed. An H100 offers about 989 dense BF16 TFLOPS but "only" about 3.35 TB/s of memory bandwidth. In the decode phase, a single sequence uses roughly 2 FLOPs per byte of weights read, so the tensor cores sit mostly idle. You can see this on the roofline model simulator: batch-1 decoding sits far to the left, deep in the memory-bound region.

Key insight: checking k tokens costs almost the same as generating one, because the weights are read once either way. Speculative decoding exploits this.

How speculative decoding works

1. Draft: a small, fast model (or a cheap extra head) generates k candidate tokens, for example 4 to 8, one after another. 2. Verify: the large target model runs one forward pass over the prompt plus all k draft tokens, producing its own probability distribution at every position, just like prefill. 3. Accept or reject: going left to right, each draft token is accepted with a probability based on how much the target model agrees with it. At the first rejection, the target model's corrected token is used instead, and the rest of the draft is discarded. 4. Bonus token: if all k drafts are accepted, the target model's prediction for position k+1 is added for free. 5. Repeat from the new position.

Each verification step therefore produces between 1 and k+1 tokens for roughly the cost of one large-model step.

Why it is lossless

The acceptance rule from the original papers (Leviathan et al. at Google and Chen et al. at DeepMind, 2022–2023) is a form of rejection sampling:

accept draft token x with probability min(1, p_target(x) / p_draft(x))
if rejected: sample from normalize(max(0, p_target - p_draft))

This guarantees that the final tokens follow exactly the target model's distribution, at any temperature. With greedy decoding (temperature 0), the rule simplifies to "accept if the target model's top token matches". Unlike quantization or distillation, speculative decoding never changes what the model would say, only how fast it says it.

How much faster? The maths

Let α be the average probability that a draft token is accepted and k the draft length. The expected number of tokens produced per verification step is:

E[tokens per step] = (1 - α^(k+1)) / (1 - α)
Acceptance rate αk = 4k = 8
0.51.94 tokens2.00 tokens
0.72.77 tokens3.20 tokens
0.83.36 tokens4.33 tokens
0.94.10 tokens6.13 tokens

Real speed-up is lower than these numbers because drafting and verification have overhead. In practice, 2–3× end-to-end speed-ups are common for chat and coding, and more for highly predictable text such as code edits, structured output or summaries that copy from the input.

What drives acceptance:

  • Draft quality: a draft model from the same family, with the same tokenizer, agrees far more often.
  • Task predictability: boilerplate code and JSON are easier to guess than creative writing.
  • Temperature: higher temperatures reduce agreement. See LLM temperature explained.

Types of drafters

MethodHow it draftsProsCons
Separate draft modelA small model from the same family (e.g. 1B drafting for 70B)Simple, no training neededExtra memory, must share tokenizer
MedusaExtra decoding heads on the target model predict several future tokensNo second modelHeads need training
EAGLE / EAGLE-2 / EAGLE-3A lightweight head that drafts using the target model's hidden featuresVery high acceptance ratesNeeds training per model
Multi-token prediction (MTP)The model is trained to predict several tokens at once (used in DeepSeek-V3)Built in from pre-trainingMust be designed into the model
N-gram / prompt lookupCopies likely continuations from the prompt or earlier textZero cost, great for editing and RAGOnly helps when text repeats
Lookahead decodingGenerates and verifies n-grams in parallel with the target model aloneNo draft modelLower gains on varied text

When speculative decoding helps, and when it does not

Helps most:

  • Single-user or low-batch serving, where decode is heavily memory-bound
  • Local inference on consumer GPUs and Macs; see how to run an LLM locally
  • Long outputs: code generation, long answers and reasoning models that produce thousands of thinking tokens
  • Latency-sensitive applications such as coding assistants and voice agents

Helps less:

  • High-batch serving: with many concurrent users, the GPU is already busy computing, so spare FLOPs shrink. Gains fall and can turn negative at very large batch sizes.
  • Low acceptance: a poorly matched draft wastes compute.
  • Tight memory: a separate draft model needs extra VRAM and its own KV cache.

Hardware and memory considerations

  • Extra VRAM: a 1B draft model adds about 2 GB in FP16 plus its KV cache. Head-based methods (Medusa, EAGLE) add much less.
  • Compute headroom: GPUs with high FLOPs-to-bandwidth ratios, such as the H100, B200 and RTX 5090, have the most idle compute to spend on verification.
  • Mixture-of-experts models: verification of several tokens may activate more experts, reducing the free-compute advantage somewhat. See mixture of experts.
  • Attention cost: verifying k tokens adds k positions of attention, which grows with context length; FlashAttention keeps this efficient.

Using speculative decoding today

  • vLLM and SGLang: support draft models, EAGLE-style heads, MTP and n-gram speculation through configuration options. See LLM inference engines compared.
  • TensorRT-LLM: supports draft-target, Medusa, EAGLE and lookahead decoding on NVIDIA GPUs.
  • llama.cpp: supports a draft model with the same vocabulary, useful for local setups.
  • Hosted APIs: many providers already use speculative decoding behind the scenes; it is one reason some APIs are much faster than a naive deployment of the same model.

A typical local setup: run a 70B target with a 1B–3B draft from the same family, set draft length to 4–8, and measure tokens per second with and without it on your own prompts.

Frequently asked questions

Does speculative decoding reduce output quality?

No. With the standard acceptance rule, outputs follow exactly the same probability distribution as the target model alone. Only speed changes.

Why not just use the small model?

The small model is faster but less capable. Speculative decoding gives you the large model's quality at closer to the small model's speed.

What is a good draft length?

Usually 3 to 8 tokens. Longer drafts help when acceptance is high but waste compute when it is low. Many engines tune this automatically.

Does it reduce cost per token?

For low-batch, latency-focused serving, yes, because each GPU-second produces more tokens. For high-throughput batch serving, the benefit is smaller because the GPU is already compute-busy.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *