Attention Mechanism Simulator: Self-Attention Step by Step

Attention is the step that lets every word in a transformer look at every other word. This simulator runs a real single attention head on your sentence, with 4 numbers per word instead of thousands, so every value fits on screen. Pick a word to see who it pays attention to, then step through the five stages.

How to use it

• Mouse: type a sentence, click a word in the arc diagram or a row in a matrix, and click the step buttons. Hover any matrix cell to see its calculation.
• Keyboard: click the simulator, then use ← and → to choose a word, 1 to 5 (or N and P) for the step, M for the causal mask, O for positions, H to switch head, W for new weights and ↑ and ↓ for temperature.

The five stages

1. Embed: each word becomes a vector. Turn Positions off and the two copies of "the" become identical: without position information, a transformer cannot tell word order.
2. Q, K and V: three learned matrices turn each vector into a query, a key and a value. Real models learn these during training; here they are random, so press New random to see how much the weights decide.
3. Scores: each query is compared with every key using a dot product and divided by √d (here √4 = 2), which stops scores growing too large as vectors get longer.
4. Softmax: each row of scores becomes weights that add up to 100%. Lower the temperature and attention snaps onto the best match; raise it and attention spreads evenly.
5. Output: each word's new vector is the weighted average of the value vectors. The word now carries context from the words it attended to.

Why the causal mask matters

Models like GPT, Llama and Claude generate text left to right, so during training a word must not see the words after it. The mask sets those scores to minus infinity, which softmax turns into exactly 0%. Turn it off to see a bidirectional encoder like BERT, where every word sees the whole sentence.

From one head to a real model

Real transformers run many heads side by side, each with its own weights and its own idea of what matters: one may track grammar, another names, another position. Switch between Head A and Head B to see two heads disagree on the same sentence. Llama 3.1 8B runs 32 heads in each of its 32 layers, with vectors of 128 numbers per head.

Why this matters for hardware

The score matrix grows with the square of the context length, which is why long prompts are expensive and why methods like FlashAttention exist. During generation, each new token must read the stored keys and values of every earlier token: that is the KV cache. See both effects in the LLM inference simulator, and the full model pipeline in how AI works.

Source

• Vaswani et al., Attention Is All You Need (2017)