Short answer
A token is the basic unit of text a language model reads and writes. It can be a whole word, part of a word, a punctuation mark or a space. In English, one token is on average about 4 characters or three quarters of a word, so 1,000 tokens ≈ 750 words. Tokens matter because AI APIs charge per token, context windows are measured in tokens, and generation speed is measured in tokens per second.
Why AI models use tokens
Neural networks only work with numbers. Before a large language model can process your prompt, a tokenizer splits the text into tokens and replaces each one with an integer ID from a fixed vocabulary. The model then works on those IDs, and its output IDs are turned back into text.
Why not just use whole words or single characters?
- Whole words would need a gigantic vocabulary and could not handle new words, typos or names.
- Single characters make sequences very long, which makes models slower and more expensive, because attention cost grows with sequence length.
- Subword tokens are the compromise: common words get one token, rare words are split into pieces.
A tokenization example
The sentence "Tensor cores accelerate unbelievably large matrix multiplications." might be split like this (exact splits vary by tokenizer):
Tensor | cores | accelerate | unbel | iev | ably | large | matrix | multip | lications | .Common words such as "large" and "matrix" are single tokens. The rare word "unbelievably" is broken into three pieces. A leading space is usually part of the token, so " cores" and "cores" can have different IDs.
How tokenizers are built
Most modern LLMs use a variant of byte pair encoding (BPE):
1. Start with individual bytes or characters. 2. Find the pair of symbols that appears most often in a large text corpus and merge it into a new symbol. 3. Repeat thousands of times until the vocabulary reaches the target size.
The result is a vocabulary where frequent strings ("the", "ing", " model") are single tokens. Other common methods are WordPiece (used by BERT) and SentencePiece/Unigram (used by many multilingual models).
Vocabulary sizes have grown over time: Llama 3 uses about 128,000 tokens and OpenAI's GPT-4o tokenizer (o200k_base) about 200,000. Larger vocabularies make text shorter in tokens, especially for non-English languages and code.
How many words is a token?
OpenAI's rule of thumb for English:
| Tokens | Approximate English text |
|---|---|
| 1 token | About 4 characters |
| 100 tokens | About 75 words |
| 1,000 tokens | About 750 words, one and a half pages |
| 128,000 tokens | About 96,000 words, a long novel |
| 1,000,000 tokens | About 750,000 words |
Other languages usually need more tokens for the same meaning. Languages with non-Latin scripts, such as Hindi, Urdu, Arabic, Thai or Burmese, can need two to several times more tokens than English, which makes them slower and more expensive on some models. Code also tokenizes less efficiently than prose because of spaces, symbols and indentation.
Why tokens matter
1. Price
AI APIs charge per million tokens, with separate prices for input tokens (your prompt, documents and chat history) and output tokens (the model's reply). Output tokens usually cost several times more because they are generated one at a time. Long chat histories get expensive because the whole conversation is sent again with every message. Reasoning models also bill for hidden "thinking" tokens.
2. Context window
A model's context window is its maximum number of tokens for the prompt plus the response. If a conversation or document exceeds it, older content must be cut or summarised.
3. Speed
Generation speed is measured in tokens per second. Each output token requires reading the model's weights from memory, so speed depends heavily on GPU memory bandwidth. Two numbers matter: time to first token (how long before the reply starts, driven by prefill) and tokens per second during generation.
4. Memory
Every token in the context adds to the KV cache stored in GPU memory. That is why very long prompts need much more VRAM.
Why tokens explain some odd AI mistakes
Because models see tokens, not letters, they can struggle with tasks that need character-level detail:
- Counting letters ("how many r's in strawberry?") is hard when the word is one or two tokens.
- Spelling backwards, rhymes and anagrams are harder than they look.
- Arithmetic on long numbers can suffer when digits are grouped into tokens in inconsistent ways.
Newer models handle many of these cases better, often by reasoning step by step or by using a code tool.
How to count tokens
- OpenAI Tokenizer (web tool) shows exactly how text is split for OpenAI models.
- tiktoken (Python) counts tokens for OpenAI models in code:
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
print(len(enc.encode("How many tokens is this sentence?")))- Hugging Face tokenizers load the exact tokenizer for open models such as Llama, Mistral or Qwen with
AutoTokenizer.from_pretrained(...). - Most API responses also report the input and output token counts used.
How sampling picks the next token
At each step the model outputs a probability for every token in its vocabulary. Settings such as temperature, top-k and top-p decide how the next token is chosen from those probabilities. You can experiment with this in our LLM sampling simulator.
Frequently asked questions
Is a token the same as a word?
No. Short common words are often one token, but longer or rarer words are split into several. On average an English word is about 1.3 tokens.
Do images and audio use tokens?
Yes. Multimodal models convert images into patches and audio into short segments, each represented as tokens. A single image can use hundreds or more than a thousand tokens, depending on its resolution and the model.
Are tokens the same across models?
No. Each model family has its own tokenizer, so the same text can be a different number of tokens on GPT, Claude, Gemini and Llama. Always count with the tokenizer of the model you use.
Why do output tokens cost more than input tokens?
Input tokens are processed in parallel in one pass, which uses the GPU efficiently. Output tokens are generated one at a time, and each step reads the full model from memory, so they use much more GPU time per token.
Sources and further reading
- Sennrich et al., Neural Machine Translation of Rare Words with Subword Units (BPE, 2015)
- OpenAI, Tokenizer tool
- OpenAI, tiktoken on GitHub
- Hugging Face, Summary of the tokenizers




Leave a Reply