Skip to content
,

What Is a Transformer in AI? The Architecture Behind ChatGPT Explained

The transformer is the neural network architecture behind ChatGPT, Claude, Gemini and most modern AI. How tokens, embeddings, self-attention and feed-forward layers work, encoder vs decoder models, and why transformers run so well on GPUs.

4 min read

What Is a Transformer in AI? — cover image from TensorCore.si

Short answer

A transformer is a type of neural network introduced by Google researchers in the 2017 paper "Attention Is All You Need". Its key idea is self-attention: every word (token) in the input can look at every other token and decide which ones matter most, all in parallel. Transformers power almost every modern AI model, including the GPT, Claude, Gemini and Llama large language models, and they also handle images, audio and video. Because attention and the rest of the model are built from large matrix multiplications, transformers run extremely well on GPU tensor cores.

Why transformers were a breakthrough

Before 2017, language models mostly used recurrent neural networks (RNNs), which read text one word at a time. That made them slow to train and bad at connecting words far apart. Transformers process all tokens at once and connect any two tokens directly through attention. This made it possible to train much larger models on much more data using thousands of GPUs, which led directly to today's large language models.

How a transformer works, step by step

1. Tokens and embeddings

Text is split into tokens, and each token becomes a vector of numbers called an embedding. A positional encoding adds information about where each token sits in the sequence, since attention on its own has no sense of order.

2. Self-attention

For each token, the model creates three vectors:

  • Query: what this token is looking for.
  • Key: what this token offers to others.
  • Value: the information it passes on if chosen.

Each token's query is compared with every other token's key to get attention scores. A softmax turns the scores into weights, and the token's new representation is a weighted mix of the values. In "The animal didn't cross the street because it was tired", attention lets "it" focus on "animal". Try it in our attention mechanism simulator.

3. Multi-head attention

The model runs many attention "heads" in parallel, each free to learn a different kind of relationship, such as grammar, meaning or long-distance references, then combines their results.

4. Feed-forward layers

After attention, each token passes through a small neural network (the feed-forward or MLP block). This is where much of the model's knowledge is thought to be stored. In many large models, a mixture-of-experts design picks only a few expert feed-forward blocks for each token to save compute.

5. Stack many layers

Attention plus feed-forward, with residual connections and normalisation, forms one transformer block. Models stack dozens to over a hundred blocks. The final layer predicts the next token.

Encoder, decoder and encoder-decoder models

TypeHow it worksExamplesUsed for
Encoder-onlyReads the whole input in both directionsBERTSearch, classification, embeddings
Decoder-onlyPredicts the next token, only looking backwardsGPT, Claude, Llama, GeminiChatbots, writing, coding
Encoder-decoderEncodes input, then decodes outputOriginal transformer, T5Translation, summarisation

Almost all modern chatbots are decoder-only transformers.

Transformers and hardware

Transformers are a perfect fit for GPUs because nearly every step is a matrix multiplication, exactly what tensor cores accelerate. Two hardware challenges stand out:

  • Attention cost grows with context length: comparing every token with every other token scales with the square of the sequence length. Techniques like FlashAttention make it much faster by keeping data in on-chip memory.
  • Memory for long conversations: during generation, models store keys and values for past tokens in the KV cache, which can need many gigabytes. See prefill vs decode.

Beyond text

Transformers now handle images (Vision Transformers), speech (Whisper), video, protein structures and robotics. Multimodal models such as GPT, Gemini and Claude process text and images with the same transformer backbone.

Frequently asked questions

What does GPT stand for?

Generative Pre-trained Transformer: a transformer pre-trained on large amounts of text to generate new text.

Who invented the transformer?

Eight researchers at Google, led by authors including Ashish Vaswani and Noam Shazeer, in the 2017 paper "Attention Is All You Need".

Are there alternatives to transformers?

Yes. State-space models such as Mamba and hybrid designs aim to handle very long sequences more efficiently, but transformers remain dominant in 2026.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *