Skip to content

What Is Knowledge Distillation? How Big AI Models Teach Small Ones

Knowledge distillation trains a small "student" model to imitate a large "teacher" model, keeping much of its ability at a fraction of the size and cost. How distillation works, soft labels, examples like DistilBERT and DeepSeek-R1 distilled models, benefits, limits and legal questions.

4 min read

What Is Model Distillation? — cover image from TensorCore.si

Short answer

Knowledge distillation is a technique for compressing AI models: a large, capable teacher model generates outputs, and a smaller student model is trained to imitate them. The student ends up much faster and cheaper to run while keeping a large share of the teacher's ability. Distillation is behind many of today's efficient models, from DistilBERT, which kept about 97% of BERT's language understanding while being 40% smaller and 60% faster, to the small reasoning models DeepSeek distilled from R1.

The idea in plain English

A student can learn a subject from a textbook (the raw data), but learns faster from an expert teacher who explains how they think, including which answers are almost right. Distillation gives a small model the teacher's richer guidance instead of only the "correct answers".

How distillation works

Classic distillation with soft labels

Geoffrey Hinton and colleagues formalised the method in 2015:

1. Train (or take) a large teacher model. 2. Run the teacher on training examples and record its full probability distribution over possible answers, called soft labels. For an image of a husky, the teacher might say 80% husky, 15% wolf, 5% malamute. 3. Train the student to match those probabilities, usually combined with the true labels.

Soft labels carry "dark knowledge": the teacher's sense of which wrong answers are close. A temperature setting softens the probabilities so these relationships are easier to learn, the same idea as the sampling temperature explained in LLM temperature, top-k and top-p.

Distillation for large language models

For large language models, common approaches are:

  • Logit distillation: the student matches the teacher's probability for every possible next token. Needs access to the teacher's internals.
  • Sequence-level distillation: the teacher writes complete answers, explanations or step-by-step reasoning, and the student is fine-tuned on that synthetic data. This works even when you only have API access to the teacher.
  • On-policy distillation: the student generates answers and the teacher scores or corrects them, so the student learns from its own mistakes.

Real examples

StudentTeacherResult
DistilBERT (2019)BERT~40% smaller, ~60% faster, ~97% of language understanding
DeepSeek-R1 distilled models (2025)DeepSeek-R1Qwen- and Llama-based models from 1.5B to 70B with strong reasoning
Gemma, Llama and other small open modelsLarger in-house modelsSmaller sizes trained partly from larger models' outputs
On-device assistantsCloud modelsCompact models that run on phones and laptops

Many small language models owe much of their quality to distillation and synthetic training data generated by bigger models.

Why distillation matters

  • Lower cost: a smaller model needs fewer GPUs and less VRAM to serve, cutting inference costs dramatically.
  • Speed: fewer parameters means faster responses, because generation speed depends on how many bytes must be read from memory per token. See GPU memory bandwidth.
  • On-device AI: distilled models can run on phones, AI PCs and edge devices.
  • Energy: smaller models use less electricity per request. See how much energy AI uses.
  • Specialisation: a student can be distilled for one task, such as customer support or code completion, and match the teacher on that task.

Distillation vs quantization vs pruning

TechniqueWhat it doesChanges architecture?
DistillationTrains a new, smaller model to imitate a big oneYes, the student is a different model
QuantizationStores the same model's weights in fewer bitsNo
PruningRemoves less important weights, heads or layersPartly

They are often combined: distil a smaller model, then quantize it to 4 bits to run locally.

Limitations

  • Capability ceiling: the student rarely exceeds the teacher and usually loses some breadth, especially on rare knowledge and complex reasoning.
  • Inherited flaws: biases and errors in the teacher pass to the student. See AI hallucinations.
  • Compute still needed: generating teacher outputs and training the student takes significant GPU time, though far less than training a large model from scratch. See AI training vs inference.

Most commercial AI providers' terms of service prohibit using their model outputs to train competing models. In early 2025, distillation became a public controversy when OpenAI said it was reviewing indications that its models' outputs had been used to train a rival model. Distilling from open-weight models is generally allowed, subject to each model's licence, so check the licence and terms before training on another model's outputs.

How to try distillation yourself

1. Pick a capable teacher (a large open model, or an API you are allowed to use for training). 2. Generate high-quality answers for a few thousand prompts in your domain. 3. Fine-tune a small model on those prompt-answer pairs, for example with QLoRA on a single GPU. 4. Evaluate the student against the teacher on held-out examples.

Our fine-tuning guide covers the tools and GPU memory you need.

Frequently asked questions

Is distillation the same as fine-tuning?

Distillation usually uses fine-tuning as its training method, but the defining feature is that the training targets come from a teacher model rather than human-written labels.

Can a distilled model be better than the teacher?

Occasionally on a narrow task, especially when combined with extra data or reinforcement learning, but generally the student is slightly weaker overall.

Why is it called distillation?

Like distilling a liquid to concentrate its essence, the process extracts the most useful knowledge from a big model into a smaller one.

Do distilled models run on consumer GPUs?

Yes, that is a main purpose. Distilled models of 1.5B to 14B parameters run well on 8 to 16 GB GPUs or Apple Silicon. See how to run an LLM locally.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *