Short answer
A context window is the maximum number of tokens a language model can take into account at one time: your prompt, any documents or chat history, system instructions and the model's own reply together. It works like the model's short-term working memory. Context windows have grown from about 2,000 tokens in GPT-3 to 128,000 tokens to over 1 million in many current models, but long contexts need a lot of GPU memory and models do not use every part of a long context equally well.
What goes into the context window
Everything the model "sees" for a single response counts against the limit:
- System prompt: the hidden instructions that set the assistant's behaviour.
- Conversation history: every previous message in the chat.
- Documents and tool results: pasted text, uploaded files, search results or RAG snippets.
- Your current prompt.
- The model's output: the reply, plus any hidden reasoning tokens for reasoning models.
When a conversation grows beyond the window, the application must drop or summarise older messages. That is why a chatbot can "forget" what you said at the start of a very long chat. The model has no memory between requests other than what is placed in the context again.
How big are context windows?
| Model or era | Context window |
|---|---|
| GPT-3 (2020) | 2,048 tokens |
| Original GPT-4 (2023) | 8,192 and 32,768 tokens |
| Llama 3.1 (2024) | 128,000 tokens |
| Gemini 1.5 Pro (2024) | Up to 2 million tokens |
| GPT-4.1 (2025) | About 1 million tokens |
| Most frontier chat models today | Roughly 128,000 to 1,000,000+ tokens |
For scale, 128,000 tokens is about 96,000 English words, roughly a long novel. A million tokens can hold a medium-sized codebase or several books. Limits change often, so check the model provider's documentation for current figures, and remember that the maximum output length is usually much smaller than the total window.
Why long context is expensive: the KV cache
When a transformer processes text, each layer stores a key and a value vector for every token so it does not have to recompute them for the next token. This is the KV cache, and it grows linearly with context length.
The KV cache size per token is:
2 (K and V) × layers × KV heads × head dimension × bytes per valueFor Llama 3.1 8B (32 layers, 8 KV heads, head dimension 128, FP16):
- 2 × 32 × 8 × 128 × 2 bytes = 128 KB per token
- At 128,000 tokens: about 16 GB of KV cache for one conversation
That is on top of about 16 GB for the model weights in FP16. A larger model, or many users at once, multiplies the memory needed. This is a major reason data center GPUs pack so much HBM memory, and why how much VRAM you need for an LLM depends on context length as well as model size.
Why long context is slow: attention
Standard self-attention compares every token with every other token, so the compute for processing a prompt grows roughly with the square of its length. Doubling the prompt can make prefill take far more than twice as long. Techniques that make long context practical include:
- FlashAttention: computes attention in tiles that fit in fast on-chip memory, cutting memory traffic.
- Grouped-query attention (GQA): shares key and value heads, shrinking the KV cache.
- Sliding-window and sparse attention: some layers look only at nearby tokens.
- Position-encoding extensions: methods that stretch rotary position embeddings (RoPE) to longer sequences than the model saw most in training.
- KV cache quantization and paging: store the cache in 8-bit or smaller formats and allocate it in blocks.
You can watch how attention connects tokens in our attention mechanism simulator.
Bigger is not always better
A long context window is a maximum, not a guarantee of quality.
- Lost in the middle: research has shown models are often better at using information at the start and end of a long prompt than in the middle.
- Effective vs advertised context: a model may accept 1 million tokens but answer reliably only over a smaller range for complex tasks.
- Cost and latency: every token you send is billed and processed on every request.
- Distraction: irrelevant text can lower answer quality.
For many applications, retrieving only the most relevant passages with embeddings and RAG beats pasting everything into the prompt.
Practical tips
- Put the most important instructions at the start and repeat key questions at the end of a long prompt.
- Remove irrelevant text, boilerplate and duplicated content.
- Start a new chat when switching topics so old history does not waste tokens.
- Ask the model to quote the passage it relied on, which makes errors easier to spot.
- For large document sets, use RAG instead of a giant prompt.
- When running models locally, set the context length to what you actually need, since it directly controls VRAM use.
Context window vs memory features
Some chat apps advertise "memory" across conversations. That is not a bigger context window: the app saves notes about you and inserts the relevant ones into the context of future chats. The model itself still only sees what fits in its window for each request.
Frequently asked questions
What happens when I exceed the context window?
An API will return an error or truncate input. Chat apps usually drop the oldest messages or summarise them automatically, so the model loses details from earlier in the conversation.
Does the context window include the answer?
Yes. Input and output share the same window. If your prompt uses almost all of it, there is little room left for the reply.
Is a larger context window the same as a smarter model?
No. Context size is how much the model can read at once; intelligence depends on its training, size and architecture. A smaller model with a huge window can still give weaker answers than a stronger model with a shorter window.
How do I set context length when running a model locally?
Tools like Ollama, LM Studio and llama.cpp let you set the context length (for example the num_ctx option in Ollama or -c in llama.cpp). Larger values use more VRAM. See how to run an LLM locally.
Sources and further reading
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)
- Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (2021)
- Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models (2023)
- Meta, The Llama 3 Herd of Models (2024)




Leave a Reply