Skip to content
,

RoPE Explained: Rotary Positional Embeddings and How LLMs Handle Long Context

Transformers have no built-in sense of word order, so they need positional encodings. An in-depth guide to sinusoidal, learned and ALiBi encodings and Rotary Positional Embeddings (RoPE), why RoPE won, and how techniques like position interpolation, NTK scaling and YaRN extend context windows to 128K tokens and beyond.

5 min read

RoPE Explained — cover image from TensorCore.si

Short answer

Attention treats its input as an unordered set, so transformers need positional encodings to know that "dog bites man" differs from "man bites dog". Rotary Positional Embedding (RoPE), used by Llama, Mistral, Qwen, Gemma, DeepSeek and most modern LLMs, encodes position by rotating query and key vectors by an angle proportional to their position. The dot product between a query and a key then depends only on their relative distance. RoPE's structure also makes it possible to extend context length after training with methods like position interpolation, NTK-aware scaling and YaRN, which is how many models reach 128K tokens or more.

Why transformers need positional information

Self-attention compares every token with every other token using dot products between query and key vectors. Nothing in that calculation depends on where tokens sit in the sequence. Shuffle the words and, without positional information, each token would get the same attention scores. Positional encodings inject order. For a refresher on the architecture, see What is a transformer?.

Earlier approaches

MethodHow it worksUsed byLimitations
Sinusoidal (absolute)Adds fixed sine/cosine waves of different frequencies to token embeddingsOriginal Transformer (2017)Absolute positions; weak extrapolation
Learned absoluteLearns one vector per position, added to embeddingsBERT, GPT-2, GPT-3Hard limit at the trained maximum length
Relative position biasesAdds a learned bias based on distance to attention scoresT5Extra computation, bucketed distances
ALiBiSubtracts a penalty proportional to distance from attention scoresBLOOM, MPTStrong recency bias; less used in new frontier models
RoPERotates queries and keys by position-dependent anglesLlama, Mistral, Qwen, Gemma, DeepSeek, gpt-oss and moreNeeds scaling tricks for very long contexts

How RoPE works

The intuition

Think of each pair of numbers in a query or key vector as a point on a 2D plane. RoPE rotates that point by an angle that grows with the token's position: position 1 rotates a little, position 100 rotates a lot.

When the model computes the dot product between a query at position m and a key at position n, the rotations combine so the result depends on the difference m − n. The model therefore sees relative distance naturally, without adding anything to the token embeddings.

Multiple frequencies

A head dimension of 128 gives 64 pairs. Each pair rotates at a different frequency:

θ_i = base^(-2i / d),  for i = 0 … d/2 - 1
angle for pair i at position m = m × θ_i
  • High-frequency pairs (small i) rotate quickly and capture fine, local word order.
  • Low-frequency pairs (large i) rotate slowly and capture long-range position.

The base (often written as rope_theta) controls the range of frequencies. The original RoPE used 10,000; Llama 3 raised it to 500,000 to better support long contexts. You can find it in any model's config.json.

Why RoPE became the standard

  • Encodes relative position, which generalises better than absolute positions.
  • Applied directly to Q and K inside attention, so it adds negligible compute and no parameters.
  • Works with efficient kernels such as FlashAttention and with the KV cache, since keys are rotated once when cached.
  • Supports context extension after training.

Extending context length with RoPE

A model trained on sequences of 8K tokens has never seen rotation angles for position 50,000. Feeding it longer sequences usually causes quality to collapse. Several techniques modify RoPE so longer sequences map onto angles the model understands:

Position interpolation (PI)

Proposed by Meta researchers in 2023: squeeze positions by dividing them by a scale factor. To go from 4K to 16K, positions are divided by 4 so every position falls inside the trained range. A short fine-tuning run adapts the model. Downside: high-frequency, local information gets compressed too.

NTK-aware scaling

Instead of scaling all frequencies equally, this approach increases the RoPE base, stretching low frequencies (long range) more than high frequencies (local detail). It works reasonably even without fine-tuning.

YaRN

YaRN (2023) combines frequency-dependent interpolation with an attention temperature adjustment, preserving local detail while extending range. It is widely used to extend open models to 128K tokens and is supported by inference engines such as vLLM and llama.cpp through rope_scaling settings.

Training with long context

The most reliable approach is to raise the RoPE base and continue pre-training on long documents. Llama 3.1, for example, was extended to a 128K-token context through a dedicated long-context training stage. Modern models may also mix layers with full attention and layers with sliding-window attention.

Long context is about more than positions

Even with good positional encoding, long context is limited by:

A tiny RoPE implementation

import torch

def apply_rope(x, positions, base=10000.0):
    # x: (seq, heads, dim) with even dim
    d = x.shape[-1]
    inv_freq = base ** (-torch.arange(0, d, 2, dtype=torch.float32) / d)
    angles = positions[:, None].float() * inv_freq[None, :]   # (seq, d/2)
    cos, sin = angles.cos()[:, None, :], angles.sin()[:, None, :]
    x1, x2 = x[..., 0::2], x[..., 1::2]
    out = torch.empty_like(x)
    out[..., 0::2] = x1 * cos - x2 * sin
    out[..., 1::2] = x1 * sin + x2 * cos
    return out

Apply it to queries and keys (not values) before computing attention scores. Production implementations often pair dimensions differently (first half with second half) for speed, but the maths is equivalent.

Frequently asked questions

Does RoPE add parameters to the model?

No. The rotation angles are fixed by the formula; nothing is learned.

Why is it applied only to queries and keys?

Position needs to influence which tokens attend to which, which is decided by the query-key dot product. Values carry content and do not need rotation.

Can I extend a model's context myself?

Inference engines let you set rope_scaling (for example YaRN with a factor) to run beyond the trained length, but quality usually drops without long-context fine-tuning. Check the model card for supported settings.

What does a larger RoPE base do?

It slows the lowest-frequency rotations, so positions remain distinguishable over longer distances, which helps long-context training.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *