Skip to content

What Are Tokens in AI? How LLMs Read Text, Count Words and Charge You

Tokens are the small chunks of text that language models read and write. What a token is, how tokenizers work, how many words are in 1,000 tokens, why tokens decide API prices, context limits and speed, and how to count them.

5 min read

What Are Tokens in AI? — cover image from TensorCore.si

Short answer

A token is the basic unit of text a language model reads and writes. It can be a whole word, part of a word, a punctuation mark or a space. In English, one token is on average about 4 characters or three quarters of a word, so 1,000 tokens ≈ 750 words. Tokens matter because AI APIs charge per token, context windows are measured in tokens, and generation speed is measured in tokens per second.

Why AI models use tokens

Neural networks only work with numbers. Before a large language model can process your prompt, a tokenizer splits the text into tokens and replaces each one with an integer ID from a fixed vocabulary. The model then works on those IDs, and its output IDs are turned back into text.

Why not just use whole words or single characters?

  • Whole words would need a gigantic vocabulary and could not handle new words, typos or names.
  • Single characters make sequences very long, which makes models slower and more expensive, because attention cost grows with sequence length.
  • Subword tokens are the compromise: common words get one token, rare words are split into pieces.

A tokenization example

The sentence "Tensor cores accelerate unbelievably large matrix multiplications." might be split like this (exact splits vary by tokenizer):

Tensor | cores | accelerate | unbel | iev | ably | large | matrix | multip | lications | .

Common words such as "large" and "matrix" are single tokens. The rare word "unbelievably" is broken into three pieces. A leading space is usually part of the token, so " cores" and "cores" can have different IDs.

How tokenizers are built

Most modern LLMs use a variant of byte pair encoding (BPE):

1. Start with individual bytes or characters. 2. Find the pair of symbols that appears most often in a large text corpus and merge it into a new symbol. 3. Repeat thousands of times until the vocabulary reaches the target size.

The result is a vocabulary where frequent strings ("the", "ing", " model") are single tokens. Other common methods are WordPiece (used by BERT) and SentencePiece/Unigram (used by many multilingual models).

Vocabulary sizes have grown over time: Llama 3 uses about 128,000 tokens and OpenAI's GPT-4o tokenizer (o200k_base) about 200,000. Larger vocabularies make text shorter in tokens, especially for non-English languages and code.

How many words is a token?

OpenAI's rule of thumb for English:

TokensApproximate English text
1 tokenAbout 4 characters
100 tokensAbout 75 words
1,000 tokensAbout 750 words, one and a half pages
128,000 tokensAbout 96,000 words, a long novel
1,000,000 tokensAbout 750,000 words

Other languages usually need more tokens for the same meaning. Languages with non-Latin scripts, such as Hindi, Urdu, Arabic, Thai or Burmese, can need two to several times more tokens than English, which makes them slower and more expensive on some models. Code also tokenizes less efficiently than prose because of spaces, symbols and indentation.

Why tokens matter

1. Price

AI APIs charge per million tokens, with separate prices for input tokens (your prompt, documents and chat history) and output tokens (the model's reply). Output tokens usually cost several times more because they are generated one at a time. Long chat histories get expensive because the whole conversation is sent again with every message. Reasoning models also bill for hidden "thinking" tokens.

2. Context window

A model's context window is its maximum number of tokens for the prompt plus the response. If a conversation or document exceeds it, older content must be cut or summarised.

3. Speed

Generation speed is measured in tokens per second. Each output token requires reading the model's weights from memory, so speed depends heavily on GPU memory bandwidth. Two numbers matter: time to first token (how long before the reply starts, driven by prefill) and tokens per second during generation.

4. Memory

Every token in the context adds to the KV cache stored in GPU memory. That is why very long prompts need much more VRAM.

Why tokens explain some odd AI mistakes

Because models see tokens, not letters, they can struggle with tasks that need character-level detail:

  • Counting letters ("how many r's in strawberry?") is hard when the word is one or two tokens.
  • Spelling backwards, rhymes and anagrams are harder than they look.
  • Arithmetic on long numbers can suffer when digits are grouped into tokens in inconsistent ways.

Newer models handle many of these cases better, often by reasoning step by step or by using a code tool.

How to count tokens

  • OpenAI Tokenizer (web tool) shows exactly how text is split for OpenAI models.
  • tiktoken (Python) counts tokens for OpenAI models in code:
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
print(len(enc.encode("How many tokens is this sentence?")))
  • Hugging Face tokenizers load the exact tokenizer for open models such as Llama, Mistral or Qwen with AutoTokenizer.from_pretrained(...).
  • Most API responses also report the input and output token counts used.

How sampling picks the next token

At each step the model outputs a probability for every token in its vocabulary. Settings such as temperature, top-k and top-p decide how the next token is chosen from those probabilities. You can experiment with this in our LLM sampling simulator.

Frequently asked questions

Is a token the same as a word?

No. Short common words are often one token, but longer or rarer words are split into several. On average an English word is about 1.3 tokens.

Do images and audio use tokens?

Yes. Multimodal models convert images into patches and audio into short segments, each represented as tokens. A single image can use hundreds or more than a thousand tokens, depending on its resolution and the model.

Are tokens the same across models?

No. Each model family has its own tokenizer, so the same text can be a different number of tokens on GPT, Claude, Gemini and Llama. Always count with the tokenizer of the model you use.

Why do output tokens cost more than input tokens?

Input tokens are processed in parallel in one pass, which uses the GPU efficiently. Output tokens are generated one at a time, and each step reads the full model from memory, so they use much more GPU time per token.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *