Skip to content
,

AI Scaling Laws Explained: Kaplan, Chinchilla, Over-Training and Test-Time Compute

Scaling laws describe how AI models improve predictably as you add compute, data and parameters. An in-depth guide to the Kaplan and Chinchilla laws, compute-optimal training, why modern models are over-trained, test-time scaling with reasoning models, data limits, and what scaling means for GPU demand.

6 min read

AI Scaling Laws Explained — cover image from TensorCore.si

Short answer

Scaling laws are empirical rules showing that a language model's loss falls smoothly and predictably as you increase training compute, model size and data. OpenAI's 2020 Kaplan laws favoured making models bigger; DeepMind's 2022 Chinchilla study showed models should be trained on about 20 tokens per parameter for compute-optimal results. Today, labs deliberately over-train smaller models on far more data to make them cheaper to run, and a new axis, test-time compute, lets reasoning models improve by thinking longer. Scaling is the main reason AI labs keep buying ever larger GPU clusters.

What a scaling law is

Train many models of different sizes with different amounts of data and compute, plot their test loss on log-log axes, and the points fall on remarkably straight lines. That means loss follows a power law:

Loss(C) ≈ L_min + (C_0 / C)^α

where C is training compute and α is a small exponent. The practical meaning: every time you multiply compute by a fixed factor, loss drops by a predictable amount, with diminishing returns. Labs use these curves to predict how a huge model will perform before spending months training it, by extrapolating from small experiments.

The training compute itself is approximately:

C ≈ 6 × N × D   (FLOPs; N = parameters, D = training tokens)

See AI training vs inference for where the 6 comes from.

The Kaplan scaling laws (OpenAI, 2020)

Kaplan et al. found that loss scales as a power law with model size, dataset size and compute, over many orders of magnitude, and that architecture details (depth vs width) matter much less than scale. Their analysis suggested that, for a fixed compute budget, it is best to grow model size faster than data, training very large models on relatively few tokens and stopping early.

That guided the era of GPT-3 (175B parameters trained on about 300 billion tokens, under 2 tokens per parameter) and other giant models.

The Chinchilla scaling laws (DeepMind, 2022)

Hoffmann et al. re-ran the analysis with better methodology, training over 400 models, and reached a different conclusion: parameters and data should grow roughly equally with compute. The compute-optimal ratio is about 20 training tokens per parameter.

They proved it with Chinchilla: a 70B-parameter model trained on 1.4 trillion tokens, using the same compute as their 280B-parameter Gopher. Chinchilla outperformed Gopher, GPT-3 and other larger models on most benchmarks, while being four times smaller and cheaper to run.

ModelParametersTraining tokensTokens per parameter
GPT-3 (2020)175B~300B~1.7
Gopher (2021)280B~300B~1.1
Chinchilla (2022)70B1.4T20
Llama 2 70B (2023)70B2T~29
Llama 3 8B (2024)8B~15T~1,900
Llama 3.1 405B (2024)405B~15.6T~38

Why modern models are "over-trained"

Chinchilla optimises training compute only. But a popular model is then used billions of times, and inference cost depends on model size. A smaller model trained far beyond the Chinchilla ratio costs more to train than strictly necessary but is much cheaper to serve, and it keeps improving with more data.

That is why Meta trained Llama 3 8B on about 15 trillion tokens, nearly 100 times the Chinchilla-optimal amount, and why small language models have become so capable. Inference-aware scaling laws formalise this trade-off by adding expected lifetime inference to the cost.

Test-time compute: the second scaling axis

Since 2024, reasoning models have shown a new kind of scaling: performance on hard problems improves as the model spends more compute at inference time, generating longer chains of thought or sampling multiple solutions and choosing the best. OpenAI reported that o1's accuracy rose steadily with both more reinforcement-learning training compute and more thinking time.

This changes hardware economics. Instead of compute being concentrated in one training run, a large share shifts to serving: every hard question can consume thousands of extra tokens. See how much energy AI uses and speculative decoding for how the industry is trying to make that cheaper.

Other scaling dimensions

  • Reinforcement learning compute: training reasoning models with large-scale RL has its own scaling behaviour. See What is reinforcement learning?.
  • Mixture of experts: MoE models shift the curves, giving lower loss for the same compute by adding parameters that are not all used per token. See mixture of experts.
  • Data quality: better-filtered and synthetic data can beat raw quantity, effectively moving the curve down.
  • Precision: recent research studies how low-precision training (FP8, FP4) interacts with scaling.

Limits and open questions

  • Data: high-quality public text is finite. Epoch AI estimates that the stock of human-generated public text could be fully used by frontier training runs around the late 2020s, pushing labs towards synthetic data, multimodal data and more efficient learning.
  • Loss vs abilities: smooth loss curves do not always translate into smooth gains on specific tasks. Some abilities seem to "emerge" suddenly, though researchers argue this is partly an artefact of how benchmarks are scored. See AI benchmarks explained.
  • Diminishing returns: power laws mean each improvement costs more compute than the last.
  • Energy and hardware: the largest training runs now need power on the scale of large industrial facilities and chip supply constrained by advanced packaging.
  • Does scaling lead to AGI? This is the central debate in AI. See What is AGI?, When will AGI arrive? and the intelligence explosion.

What scaling laws mean for GPU demand

Training compute for frontier models has grown by roughly four to five times per year over the past decade, according to Epoch AI. Combined with test-time scaling and billions of daily users, this drives demand for GPU clusters, AI data centers and new chips such as NVIDIA Rubin. You can estimate any model's training compute and cost with the formulas in LLM training cost, and follow rental prices in our GPU cloud price tracker.

Quick estimate: how many GPU-hours for a Chinchilla-optimal 70B model?

  • Tokens: 20 × 70B = 1.4T
  • Compute: 6 × 70 × 10⁹ × 1.4 × 10¹² ≈ 5.9 × 10²³ FLOPs
  • H100 at 989 dense BF16 TFLOPS and 40% MFU: about 396 TFLOPS effective
  • GPU-hours ≈ 5.9 × 10²³ ÷ (3.96 × 10¹⁴ × 3,600) ≈ 413,000 H100-hours

Frequently asked questions

What is the Chinchilla ratio?

About 20 training tokens per model parameter, the amount that minimises loss for a fixed training compute budget according to DeepMind's 2022 study.

Are scaling laws still valid?

The core finding that loss improves predictably with compute has held across many orders of magnitude. What has changed is how labs spend compute: more data per parameter, more reinforcement learning and more inference-time reasoning.

Why don't labs just train much bigger models?

Bigger models cost more to train and much more to serve, data is limited, and over-trained smaller models often give a better balance of quality and cost.

Do scaling laws apply to images and video?

Yes. Similar power laws have been found for vision, speech, video and multimodal models, as well as for diffusion models.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *