Short answer
Knowledge distillation is a technique for compressing AI models: a large, capable teacher model generates outputs, and a smaller student model is trained to imitate them. The student ends up much faster and cheaper to run while keeping a large share of the teacher's ability. Distillation is behind many of today's efficient models, from DistilBERT, which kept about 97% of BERT's language understanding while being 40% smaller and 60% faster, to the small reasoning models DeepSeek distilled from R1.
The idea in plain English
A student can learn a subject from a textbook (the raw data), but learns faster from an expert teacher who explains how they think, including which answers are almost right. Distillation gives a small model the teacher's richer guidance instead of only the "correct answers".
How distillation works
Classic distillation with soft labels
Geoffrey Hinton and colleagues formalised the method in 2015:
1. Train (or take) a large teacher model. 2. Run the teacher on training examples and record its full probability distribution over possible answers, called soft labels. For an image of a husky, the teacher might say 80% husky, 15% wolf, 5% malamute. 3. Train the student to match those probabilities, usually combined with the true labels.
Soft labels carry "dark knowledge": the teacher's sense of which wrong answers are close. A temperature setting softens the probabilities so these relationships are easier to learn, the same idea as the sampling temperature explained in LLM temperature, top-k and top-p.
Distillation for large language models
For large language models, common approaches are:
- Logit distillation: the student matches the teacher's probability for every possible next token. Needs access to the teacher's internals.
- Sequence-level distillation: the teacher writes complete answers, explanations or step-by-step reasoning, and the student is fine-tuned on that synthetic data. This works even when you only have API access to the teacher.
- On-policy distillation: the student generates answers and the teacher scores or corrects them, so the student learns from its own mistakes.
Real examples
| Student | Teacher | Result |
|---|---|---|
| DistilBERT (2019) | BERT | ~40% smaller, ~60% faster, ~97% of language understanding |
| DeepSeek-R1 distilled models (2025) | DeepSeek-R1 | Qwen- and Llama-based models from 1.5B to 70B with strong reasoning |
| Gemma, Llama and other small open models | Larger in-house models | Smaller sizes trained partly from larger models' outputs |
| On-device assistants | Cloud models | Compact models that run on phones and laptops |
Many small language models owe much of their quality to distillation and synthetic training data generated by bigger models.
Why distillation matters
- Lower cost: a smaller model needs fewer GPUs and less VRAM to serve, cutting inference costs dramatically.
- Speed: fewer parameters means faster responses, because generation speed depends on how many bytes must be read from memory per token. See GPU memory bandwidth.
- On-device AI: distilled models can run on phones, AI PCs and edge devices.
- Energy: smaller models use less electricity per request. See how much energy AI uses.
- Specialisation: a student can be distilled for one task, such as customer support or code completion, and match the teacher on that task.
Distillation vs quantization vs pruning
| Technique | What it does | Changes architecture? |
|---|---|---|
| Distillation | Trains a new, smaller model to imitate a big one | Yes, the student is a different model |
| Quantization | Stores the same model's weights in fewer bits | No |
| Pruning | Removes less important weights, heads or layers | Partly |
They are often combined: distil a smaller model, then quantize it to 4 bits to run locally.
Limitations
- Capability ceiling: the student rarely exceeds the teacher and usually loses some breadth, especially on rare knowledge and complex reasoning.
- Inherited flaws: biases and errors in the teacher pass to the student. See AI hallucinations.
- Compute still needed: generating teacher outputs and training the student takes significant GPU time, though far less than training a large model from scratch. See AI training vs inference.
Legal and ethical questions
Most commercial AI providers' terms of service prohibit using their model outputs to train competing models. In early 2025, distillation became a public controversy when OpenAI said it was reviewing indications that its models' outputs had been used to train a rival model. Distilling from open-weight models is generally allowed, subject to each model's licence, so check the licence and terms before training on another model's outputs.
How to try distillation yourself
1. Pick a capable teacher (a large open model, or an API you are allowed to use for training). 2. Generate high-quality answers for a few thousand prompts in your domain. 3. Fine-tune a small model on those prompt-answer pairs, for example with QLoRA on a single GPU. 4. Evaluate the student against the teacher on held-out examples.
Our fine-tuning guide covers the tools and GPU memory you need.
Frequently asked questions
Is distillation the same as fine-tuning?
Distillation usually uses fine-tuning as its training method, but the defining feature is that the training targets come from a teacher model rather than human-written labels.
Can a distilled model be better than the teacher?
Occasionally on a narrow task, especially when combined with extra data or reinforcement learning, but generally the student is slightly weaker overall.
Why is it called distillation?
Like distilling a liquid to concentrate its essence, the process extracts the most useful knowledge from a big model into a smaller one.
Do distilled models run on consumer GPUs?
Yes, that is a main purpose. Distilled models of 1.5B to 14B parameters run well on 8 to 16 GB GPUs or Apple Silicon. See how to run an LLM locally.
Sources and further reading
- Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network (2015)
- Sanh et al., DistilBERT, a distilled version of BERT (2019)
- DeepSeek-AI, DeepSeek-R1 (2025)
- Hugging Face, Knowledge distillation guide




Leave a Reply