Skip to content

How Does ChatGPT Work? A Simple Step-by-Step Explanation

ChatGPT works by predicting the next token with a huge transformer model that was pre-trained on internet-scale text and then tuned with human feedback. A plain-English walkthrough of what happens from training to the moment you press Enter, including reasoning, tools and the GPUs behind it.

6 min read

How Does ChatGPT Work? — cover image from TensorCore.si

Short answer

ChatGPT works by repeatedly predicting the next token (a word or part of a word) using a very large transformer neural network. That model was first pre-trained on a vast amount of text to learn language and knowledge, then fine-tuned on example conversations and with reinforcement learning from human feedback to become a helpful assistant. When you send a message, your text is split into tokens, processed on GPUs in a data center, and the reply is generated one token at a time. Tools such as web search and code execution let it look things up and calculate instead of guessing.

The short version

1. Pre-training: a GPT model reads an enormous amount of text and learns to predict the next token. 2. Fine-tuning: it is trained on example conversations written or rated by people. 3. Reinforcement learning: it is rewarded for answers people prefer and for correct solutions. 4. Your message: your text plus the conversation so far is turned into tokens. 5. Generation: the model predicts the reply token by token, running on GPUs. 6. Extras: search, file reading, code execution, memory and image generation are tools around the model.

Step 1: Pre-training, learning from text

ChatGPT is powered by OpenAI's GPT family of large language models. "GPT" stands for Generative Pre-trained Transformer.

During pre-training, the model is shown huge amounts of text from public web pages, books, code and licensed data. Its task is simple: given the text so far, predict the next token. Every time it predicts wrongly, the training algorithm adjusts its billions of parameters slightly to make the correct token more likely next time.

Repeated over trillions of tokens, this teaches the model grammar, facts, writing styles, reasoning patterns and programming. GPT-3, released in 2020, had 175 billion parameters; OpenAI has not published the sizes of its later models. Pre-training a frontier model takes thousands of GPUs running for months. See AI training vs inference for what that involves.

After pre-training, the model is a powerful text completer, but not yet a good assistant. Ask it a question and it might continue with more questions.

Step 2: Supervised fine-tuning

Next, the model is fine-tuned on example conversations: prompts paired with high-quality answers. This teaches it the format of a helpful dialogue: answer the question, follow instructions, ask for clarification when needed.

Step 3: Reinforcement learning from human feedback (RLHF)

OpenAI described this approach in its 2022 InstructGPT paper, the forerunner of ChatGPT:

1. The model writes several answers to the same prompt. 2. Human reviewers rank the answers from best to worst. 3. A separate reward model learns to predict which answers people prefer. 4. The language model is then trained with reinforcement learning to produce answers the reward model scores highly.

This is a big part of why ChatGPT is polite, follows instructions and declines some harmful requests. It is also part of the AI alignment effort to make models behave as intended.

Step 4: What happens when you press Enter

1. Context building: the app combines a hidden system prompt, any memory or custom instructions, the conversation so far, uploaded files and your new message. 2. Tokenization: all of that text is split into tokens and converted to numbers. The total must fit inside the model's context window. 3. Prefill: the whole input is processed in one pass on GPUs. The model builds an internal representation of every token using attention, which lets each token draw information from all the others, and stores the results in a KV cache. 4. Decoding: the model outputs a probability for every possible next token. One is chosen (with some randomness controlled by settings like temperature), appended to the text, and the process repeats. That is why replies stream onto your screen word by word. Try it in our LLM sampling simulator. 5. Stopping: generation ends when the model produces a special end token or hits a length limit.

The model does not search a database of stored answers. Each reply is generated fresh, which is why asking the same question twice can give different wording.

Reasoning models: thinking before answering

Starting with OpenAI's o1 in September 2024, some models are trained with reinforcement learning to produce a long internal chain of reasoning before giving a final answer. They are rewarded for reaching correct solutions on maths, coding and logic problems. In ChatGPT, these models may show a summary of their "thinking". The reasoning tokens cost extra compute, so these answers are slower but usually more accurate on hard problems.

Tools: search, code and more

On its own, a language model only knows what was in its training data, up to a knowledge cut-off date, and it can make up facts (hallucinate). ChatGPT reduces this with tools the model can decide to call:

  • Web search: fetches current pages and cites them.
  • Code execution: runs Python to analyse data or do exact calculations.
  • File reading: extracts text from PDFs, spreadsheets and documents, similar to RAG.
  • Image generation and vision: creates images and understands uploaded photos.
  • Agent features: can browse and take multi-step actions, as described in What is an AI agent?.

The hardware behind ChatGPT

Every message runs on large clusters of NVIDIA GPUs in AI data centers operated by OpenAI and cloud partners such as Microsoft Azure. A large model is split across several GPUs connected by NVLink, and requests from many users are batched together so each GPU stays busy. The speed of each reply depends mostly on GPU memory bandwidth, because generating each token requires reading the model's weights. Serving hundreds of millions of weekly users is why OpenAI has signed some of the largest compute deals in history.

What ChatGPT is not

  • It is not a search engine, unless it is using its search tool.
  • It is not conscious and does not have goals of its own.
  • It does not learn from your chat in real time. Conversations can be used to improve future models depending on your settings, but the model's weights do not change during your chat.
  • It is not always right. Check important facts, figures and citations.

Frequently asked questions

Does ChatGPT copy answers from the internet?

No. It generates new text from patterns learned during training. It can repeat common phrases or well-known passages, and when it uses web search it summarises and cites the pages it found.

Why does ChatGPT make mistakes?

Because it predicts plausible text rather than checking facts, it can produce confident errors. Gaps in training data, ambiguous questions and its knowledge cut-off all contribute. See AI hallucinations.

How does ChatGPT remember earlier messages?

The whole conversation is sent to the model again with each new message, inside its context window. Very long chats may be trimmed or summarised.

Can I run ChatGPT on my own computer?

Not ChatGPT itself, but you can run open models with similar abilities, including OpenAI's open-weight gpt-oss models, on your own hardware. See how to run an LLM locally and the best GPU for local LLMs.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *