Short answer
Attention treats its input as an unordered set, so transformers need positional encodings to know that "dog bites man" differs from "man bites dog". Rotary Positional Embedding (RoPE), used by Llama, Mistral, Qwen, Gemma, DeepSeek and most modern LLMs, encodes position by rotating query and key vectors by an angle proportional to their position. The dot product between a query and a key then depends only on their relative distance. RoPE's structure also makes it possible to extend context length after training with methods like position interpolation, NTK-aware scaling and YaRN, which is how many models reach 128K tokens or more.
Why transformers need positional information
Self-attention compares every token with every other token using dot products between query and key vectors. Nothing in that calculation depends on where tokens sit in the sequence. Shuffle the words and, without positional information, each token would get the same attention scores. Positional encodings inject order. For a refresher on the architecture, see What is a transformer?.
Earlier approaches
| Method | How it works | Used by | Limitations |
|---|---|---|---|
| Sinusoidal (absolute) | Adds fixed sine/cosine waves of different frequencies to token embeddings | Original Transformer (2017) | Absolute positions; weak extrapolation |
| Learned absolute | Learns one vector per position, added to embeddings | BERT, GPT-2, GPT-3 | Hard limit at the trained maximum length |
| Relative position biases | Adds a learned bias based on distance to attention scores | T5 | Extra computation, bucketed distances |
| ALiBi | Subtracts a penalty proportional to distance from attention scores | BLOOM, MPT | Strong recency bias; less used in new frontier models |
| RoPE | Rotates queries and keys by position-dependent angles | Llama, Mistral, Qwen, Gemma, DeepSeek, gpt-oss and more | Needs scaling tricks for very long contexts |
How RoPE works
The intuition
Think of each pair of numbers in a query or key vector as a point on a 2D plane. RoPE rotates that point by an angle that grows with the token's position: position 1 rotates a little, position 100 rotates a lot.
When the model computes the dot product between a query at position m and a key at position n, the rotations combine so the result depends on the difference m − n. The model therefore sees relative distance naturally, without adding anything to the token embeddings.
Multiple frequencies
A head dimension of 128 gives 64 pairs. Each pair rotates at a different frequency:
θ_i = base^(-2i / d), for i = 0 … d/2 - 1
angle for pair i at position m = m × θ_i- High-frequency pairs (small i) rotate quickly and capture fine, local word order.
- Low-frequency pairs (large i) rotate slowly and capture long-range position.
The base (often written as rope_theta) controls the range of frequencies. The original RoPE used 10,000; Llama 3 raised it to 500,000 to better support long contexts. You can find it in any model's config.json.
Why RoPE became the standard
- Encodes relative position, which generalises better than absolute positions.
- Applied directly to Q and K inside attention, so it adds negligible compute and no parameters.
- Works with efficient kernels such as FlashAttention and with the KV cache, since keys are rotated once when cached.
- Supports context extension after training.
Extending context length with RoPE
A model trained on sequences of 8K tokens has never seen rotation angles for position 50,000. Feeding it longer sequences usually causes quality to collapse. Several techniques modify RoPE so longer sequences map onto angles the model understands:
Position interpolation (PI)
Proposed by Meta researchers in 2023: squeeze positions by dividing them by a scale factor. To go from 4K to 16K, positions are divided by 4 so every position falls inside the trained range. A short fine-tuning run adapts the model. Downside: high-frequency, local information gets compressed too.
NTK-aware scaling
Instead of scaling all frequencies equally, this approach increases the RoPE base, stretching low frequencies (long range) more than high frequencies (local detail). It works reasonably even without fine-tuning.
YaRN
YaRN (2023) combines frequency-dependent interpolation with an attention temperature adjustment, preserving local detail while extending range. It is widely used to extend open models to 128K tokens and is supported by inference engines such as vLLM and llama.cpp through rope_scaling settings.
Training with long context
The most reliable approach is to raise the RoPE base and continue pre-training on long documents. Llama 3.1, for example, was extended to a 128K-token context through a dedicated long-context training stage. Modern models may also mix layers with full attention and layers with sliding-window attention.
Long context is about more than positions
Even with good positional encoding, long context is limited by:
- KV cache memory, which grows linearly with length; see MHA vs MQA vs GQA vs MLA.
- Attention compute, which grows with the square of length during prefill.
- Training memory, solved with context parallelism; see distributed training and parallelism.
- Effective use of long inputs: models often use the middle of very long prompts less well. See What is a context window?.
A tiny RoPE implementation
import torch
def apply_rope(x, positions, base=10000.0):
# x: (seq, heads, dim) with even dim
d = x.shape[-1]
inv_freq = base ** (-torch.arange(0, d, 2, dtype=torch.float32) / d)
angles = positions[:, None].float() * inv_freq[None, :] # (seq, d/2)
cos, sin = angles.cos()[:, None, :], angles.sin()[:, None, :]
x1, x2 = x[..., 0::2], x[..., 1::2]
out = torch.empty_like(x)
out[..., 0::2] = x1 * cos - x2 * sin
out[..., 1::2] = x1 * sin + x2 * cos
return outApply it to queries and keys (not values) before computing attention scores. Production implementations often pair dimensions differently (first half with second half) for speed, but the maths is equivalent.
Frequently asked questions
Does RoPE add parameters to the model?
No. The rotation angles are fixed by the formula; nothing is learned.
Why is it applied only to queries and keys?
Position needs to influence which tokens attend to which, which is decided by the query-key dot product. Values carry content and do not need rotation.
Can I extend a model's context myself?
Inference engines let you set rope_scaling (for example YaRN with a factor) to run beyond the trained length, but quality usually drops without long-context fine-tuning. Check the model card for supported settings.
What does a larger RoPE base do?
It slows the lowest-frequency rotations, so positions remain distinguishable over longer distances, which helps long-context training.
Sources and further reading
- Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (2021)
- Press, Smith and Lewis, Train Short, Test Long: Attention with Linear Biases (ALiBi) (2021)
- Chen et al., Extending Context Window of Large Language Models via Positional Interpolation (2023)
- Peng et al., YaRN: Efficient Context Window Extension of Large Language Models (2023)
- Meta, The Llama 3 Herd of Models (2024)




Leave a Reply