Skip to content

What Is a Reasoning Model? How "Thinking" AI Works and Why It Needs More Compute

Reasoning models such as OpenAI's o-series, DeepSeek-R1 and the thinking modes of Claude and Gemini work through a problem step by step before answering. How they are trained, why they are better at maths and code, and why test-time compute is reshaping AI hardware demand.

5 min read

What Is a Reasoning Model? — cover image from TensorCore.si

Short answer

A reasoning model is a large language model trained to think step by step before it answers, generating a hidden chain of reasoning tokens, checking its work and trying alternative approaches. This makes it much better at maths, coding, science and multi-step planning. The trade-off is test-time compute: a reasoning model may generate thousands of extra tokens per question, so answers are slower and cost more to serve. Well-known examples include OpenAI's o-series and GPT-5 thinking modes, DeepSeek-R1, Claude's extended thinking and Gemini's thinking models.

Regular LLMs vs reasoning models

A standard large language model starts writing its final answer immediately, one token after another. If its first step is wrong, it usually carries on regardless.

A reasoning model first produces a chain of thought: intermediate steps such as breaking the problem down, doing calculations, noticing a mistake ("wait, that doesn't match") and trying again. Only then does it write the final answer. Many apps hide the raw reasoning or show a short summary.

Standard LLMReasoning model
AnswersImmediatelyAfter a thinking phase
Output tokens per questionHundredsHundreds to tens of thousands
SpeedFastSeconds to minutes
Cost per questionLowerHigher
Strong atWriting, chat, summaries, simple factsMaths, code, logic, science, planning
Main training additionInstruction tuning and RLHFLarge-scale reinforcement learning on verifiable problems

How reasoning models are trained

Chain-of-thought prompting came first

In 2022, Google researchers showed that simply prompting a model with worked examples that include step-by-step reasoning improved its maths and logic results. Another paper found that adding "Let's think step by step" to a prompt helped too. But prompting only goes so far.

Reinforcement learning on problems with checkable answers

Reasoning models are trained with reinforcement learning (RL) on large sets of problems where the answer can be verified automatically: maths problems with known solutions, code that must pass tests, logic puzzles. The model generates reasoning and an answer; if the answer is correct it is rewarded. Over many rounds, it learns reasoning habits that lead to correct answers, such as checking its work, backtracking and splitting problems into parts.

DeepSeek's R1 paper (January 2025) showed that this process can produce strong reasoning from RL alone, and that useful behaviours such as self-verification and longer thinking on harder problems emerge on their own. DeepSeek also released the model's weights, which made the method widely studied and copied.

Distillation into smaller models

The reasoning traces of a large model can be used to fine-tune smaller models, giving them much of the ability at lower cost. DeepSeek released several distilled R1 models in sizes small enough to run locally.

Test-time compute: a new way to scale

Until 2024, AI progress was driven mainly by training compute: bigger models trained on more data. Reasoning models added a second lever: test-time (inference) compute. Letting a model think longer, or generate several solutions and pick the best, improves accuracy on hard problems.

When OpenAI introduced o1 in September 2024, it reported that o1 solved about 74% of problems from the AIME 2024 maths competition on a single attempt, compared with about 12% for GPT-4o, and that performance kept improving with more thinking time. Many models now offer a "reasoning effort" or thinking-budget setting so users can trade speed and cost for accuracy.

Why reasoning models matter for hardware

Reasoning shifts AI workloads heavily toward inference:

  • Many more output tokens: a single answer can involve thousands of hidden tokens, each generated one at a time in the memory-bound decode phase.
  • Longer contexts: long reasoning chains grow the KV cache, using more GPU memory per user.
  • Bandwidth matters most: generation speed depends on memory bandwidth, favouring GPUs with fast HBM such as the H200 and B200.
  • Low precision for serving: formats like FP8 and NVFP4 make each token cheaper.
  • Specialised inference chips: fast-decode hardware such as the Groq LPU and Cerebras' wafer-scale engine market their speed for reasoning workloads.

This is one reason demand for data center GPUs stayed high even as model training became more efficient, and why rental prices for high-memory GPUs remain firm (see our GPU cloud price tracker).

When to use a reasoning model

Use a reasoning model for:

  • Maths, statistics and financial calculations
  • Writing or debugging complex code
  • Science and engineering problems
  • Multi-step planning, analysis and agent tasks (see What is an AI agent?)
  • Tricky questions where a quick answer is likely to be wrong

Use a standard model for:

  • Quick questions, chat and casual writing
  • Summaries, translation and rewriting
  • High-volume, cost-sensitive tasks
  • Anything where speed matters more than depth

Limitations

  • Slower and costlier: thinking tokens are billed and take time.
  • Overthinking: reasoning models can spend many tokens on simple questions.
  • Still hallucinate: longer reasoning can introduce unsupported claims; check facts (see AI hallucinations).
  • Hidden reasoning may not reflect the real process: research suggests the written chain of thought does not always faithfully explain how the model reached its answer, an open question in AI alignment research.

Frequently asked questions

Do reasoning models actually reason?

They produce step-by-step working that substantially improves results on problems that need several steps. Whether this counts as reasoning in the human sense is debated. In practice, they are much more reliable than standard models on maths and code, but they can still fail on simple puzzles that differ from their training data.

Can I see the model's thinking?

It depends on the provider. Some show a summary of the reasoning, some show the full trace, and open models like DeepSeek-R1 output their full reasoning text, usually between special tags.

Are reasoning models more expensive?

Usually yes, because they generate many more tokens, and providers bill for reasoning tokens even when they are hidden. Adjusting the reasoning effort setting can control the cost.

Can I run a reasoning model at home?

Yes. Distilled reasoning models and open models with thinking modes, such as the Qwen3 family and OpenAI's gpt-oss, run on consumer GPUs. Long reasoning chains need extra VRAM for the KV cache; see how much VRAM you need for an LLM.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *