Short answer
A small language model (SLM) is a language model with relatively few parameters, typically from under 1 billion to around 10 billion, compared with hundreds of billions or more for frontier large language models. SLMs are cheaper, faster and private: they can run on a laptop, phone, AI PC or a single GPU. They are less capable on broad, complex reasoning, but for focused tasks such as classification, extraction, summarising or powering an app feature, a well-trained or fine-tuned SLM can match much larger models at a fraction of the cost.
SLM vs LLM
There is no official cut-off, but this is how the terms are generally used:
| Small language model | Large language model | |
|---|---|---|
| Typical size | ~0.5B to ~10B parameters | Tens of billions to trillions |
| Runs on | Phones, laptops, AI PCs, one consumer GPU | Data center GPU clusters |
| Memory (4-bit) | ~0.5 to 6 GB | 40 GB to hundreds of GB |
| Speed | Very fast, low latency | Slower per token, higher latency |
| Cost per request | Very low or free locally | Higher |
| Strengths | Focused tasks, privacy, offline use | Broad knowledge, complex reasoning, long tasks |
| Weaknesses | Less world knowledge, weaker multi-step reasoning | Cost, latency, data leaves your device |
For background on how both work, see What is an LLM?.
Popular small language models
| Model family | Sizes | Made by | Notes |
|---|---|---|---|
| Phi | Phi-3-mini 3.8B, Phi-4 14B, Phi-4-mini | Microsoft | Trained heavily on high-quality and synthetic data |
| Gemma 3 | 1B, 4B, 12B, 27B | Open weights; 4B+ accept images | |
| Llama 3.2 | 1B, 3B | Meta | Designed for mobile and edge devices |
| Qwen | 0.5B to 8B range | Alibaba | Strong multilingual and coding ability |
| Mistral | Ministral 3B, 8B; Mistral 7B | Mistral AI | Efficient European models |
| SmolLM | 135M to 3B | Hugging Face | Fully open training data and recipes |
| gpt-oss-20b | 21B total, 3.6B active | OpenAI | A mixture-of-experts model that runs like a small one |
| On-device models | ~3B | Apple, Google (Gemini Nano) | Power built-in phone and laptop features |
How small models got so good
- Better data: Microsoft's Phi research showed that carefully filtered "textbook quality" and synthetic data lets small models punch far above their size.
- Knowledge distillation: a small "student" model learns to imitate a large "teacher" model's outputs, inheriting much of its ability.
- Over-training: small models are now trained on many trillions of tokens, far more than older scaling rules suggested, which improves quality per parameter.
- Pruning: removing less important weights and layers from a larger model, then retraining briefly.
- Quantization: storing weights in 4 or 8 bits shrinks memory with little quality loss. See LLM quantization.
- Fine-tuning: a small model fine-tuned on one task can beat a general large model on that task.
When to use a small language model
Good fits:
- On-device features: autocomplete, smart replies, summarising notifications, offline assistants.
- Privacy-sensitive data: medical, legal or company documents that should not leave your hardware.
- High-volume, simple tasks: classifying tickets, extracting fields, tagging content, routing requests.
- Low latency: real-time features where every millisecond counts.
- Cost control: millions of requests per day at near-zero marginal cost.
- Agents: many steps in an AI agent workflow are routine (formatting, tool calls) and can use an SLM, saving the large model for hard decisions.
- RAG systems: with good retrieval, a small model can answer accurately from your documents. See What is RAG?.
Use a large model instead for open-ended research, complex coding, long multi-step reasoning, broad world knowledge and high-stakes writing.
Hardware for small language models
SLMs are where local AI shines:
| Model size | 4-bit memory | Runs well on |
|---|---|---|
| 1B | ~1 GB | Phones, any laptop, Raspberry Pi-class boards |
| 3–4B | ~2.5–3 GB | Modern laptops, AI PCs, 6–8 GB GPUs |
| 7–8B | ~5–6 GB | 8–12 GB GPUs, Apple Silicon Macs |
| 12–14B | ~8–10 GB | 12–16 GB GPUs, 16 GB+ Macs |
Generation speed depends mainly on memory bandwidth, so even a mid-range GPU can produce more than 100 tokens per second with a 3B to 8B model. NPUs in phones and AI PCs run the smallest models with very low power. For exact numbers for your setup, use our VRAM guide, and to get started see how to run an LLM locally.
Frequently asked questions
How small is a "small" language model?
Most people mean models under roughly 10 billion parameters, but the line moves as hardware improves. Some include models up to about 14 billion parameters that fit on a single consumer GPU.
Are small language models as good as ChatGPT?
Not for everything. Frontier models are much stronger at broad knowledge and complex reasoning. For narrow, well-defined tasks, a good SLM, especially a fine-tuned one, can be close or equal.
Can I run an SLM without a GPU?
Yes. Models of 1B to 4B parameters run acceptably on a modern CPU or Apple Silicon, and on phones. A GPU makes them much faster.
Are SLMs safer for private data?
Running a model locally keeps data on your own device, which avoids sending it to a cloud provider. You still need normal security practices for the device and application.
Sources and further reading
- Abdin et al., Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024)
- Gunasekar et al., Textbooks Are All You Need (2023)
- Hinton et al., Distilling the Knowledge in a Neural Network (2015)
- Google, Gemma models documentation
- Hugging Face, SmolLM




Leave a Reply