Skip to content
,

What Is a Small Language Model (SLM)? SLM vs LLM, Examples and When to Use One

Small language models (SLMs) are compact AI models, usually from under 1 to around 10 billion parameters, that run fast and cheaply on laptops, phones and single GPUs. SLM vs LLM, popular examples like Phi, Gemma, Llama and Qwen, how they are made, and what hardware they need.

4 min read

Small Language Models (SLMs) — cover image from TensorCore.si

Short answer

A small language model (SLM) is a language model with relatively few parameters, typically from under 1 billion to around 10 billion, compared with hundreds of billions or more for frontier large language models. SLMs are cheaper, faster and private: they can run on a laptop, phone, AI PC or a single GPU. They are less capable on broad, complex reasoning, but for focused tasks such as classification, extraction, summarising or powering an app feature, a well-trained or fine-tuned SLM can match much larger models at a fraction of the cost.

SLM vs LLM

There is no official cut-off, but this is how the terms are generally used:

Small language modelLarge language model
Typical size~0.5B to ~10B parametersTens of billions to trillions
Runs onPhones, laptops, AI PCs, one consumer GPUData center GPU clusters
Memory (4-bit)~0.5 to 6 GB40 GB to hundreds of GB
SpeedVery fast, low latencySlower per token, higher latency
Cost per requestVery low or free locallyHigher
StrengthsFocused tasks, privacy, offline useBroad knowledge, complex reasoning, long tasks
WeaknessesLess world knowledge, weaker multi-step reasoningCost, latency, data leaves your device

For background on how both work, see What is an LLM?.

Model familySizesMade byNotes
PhiPhi-3-mini 3.8B, Phi-4 14B, Phi-4-miniMicrosoftTrained heavily on high-quality and synthetic data
Gemma 31B, 4B, 12B, 27BGoogleOpen weights; 4B+ accept images
Llama 3.21B, 3BMetaDesigned for mobile and edge devices
Qwen0.5B to 8B rangeAlibabaStrong multilingual and coding ability
MistralMinistral 3B, 8B; Mistral 7BMistral AIEfficient European models
SmolLM135M to 3BHugging FaceFully open training data and recipes
gpt-oss-20b21B total, 3.6B activeOpenAIA mixture-of-experts model that runs like a small one
On-device models~3BApple, Google (Gemini Nano)Power built-in phone and laptop features

How small models got so good

  • Better data: Microsoft's Phi research showed that carefully filtered "textbook quality" and synthetic data lets small models punch far above their size.
  • Knowledge distillation: a small "student" model learns to imitate a large "teacher" model's outputs, inheriting much of its ability.
  • Over-training: small models are now trained on many trillions of tokens, far more than older scaling rules suggested, which improves quality per parameter.
  • Pruning: removing less important weights and layers from a larger model, then retraining briefly.
  • Quantization: storing weights in 4 or 8 bits shrinks memory with little quality loss. See LLM quantization.
  • Fine-tuning: a small model fine-tuned on one task can beat a general large model on that task.

When to use a small language model

Good fits:

  • On-device features: autocomplete, smart replies, summarising notifications, offline assistants.
  • Privacy-sensitive data: medical, legal or company documents that should not leave your hardware.
  • High-volume, simple tasks: classifying tickets, extracting fields, tagging content, routing requests.
  • Low latency: real-time features where every millisecond counts.
  • Cost control: millions of requests per day at near-zero marginal cost.
  • Agents: many steps in an AI agent workflow are routine (formatting, tool calls) and can use an SLM, saving the large model for hard decisions.
  • RAG systems: with good retrieval, a small model can answer accurately from your documents. See What is RAG?.

Use a large model instead for open-ended research, complex coding, long multi-step reasoning, broad world knowledge and high-stakes writing.

Hardware for small language models

SLMs are where local AI shines:

Model size4-bit memoryRuns well on
1B~1 GBPhones, any laptop, Raspberry Pi-class boards
3–4B~2.5–3 GBModern laptops, AI PCs, 6–8 GB GPUs
7–8B~5–6 GB8–12 GB GPUs, Apple Silicon Macs
12–14B~8–10 GB12–16 GB GPUs, 16 GB+ Macs

Generation speed depends mainly on memory bandwidth, so even a mid-range GPU can produce more than 100 tokens per second with a 3B to 8B model. NPUs in phones and AI PCs run the smallest models with very low power. For exact numbers for your setup, use our VRAM guide, and to get started see how to run an LLM locally.

Frequently asked questions

How small is a "small" language model?

Most people mean models under roughly 10 billion parameters, but the line moves as hardware improves. Some include models up to about 14 billion parameters that fit on a single consumer GPU.

Are small language models as good as ChatGPT?

Not for everything. Frontier models are much stronger at broad knowledge and complex reasoning. For narrow, well-defined tasks, a good SLM, especially a fine-tuned one, can be close or equal.

Can I run an SLM without a GPU?

Yes. Models of 1B to 4B parameters run acceptably on a modern CPU or Apple Silicon, and on phones. A GPU makes them much faster.

Are SLMs safer for private data?

Running a model locally keeps data on your own device, which avoids sending it to a cloud provider. You still need normal security practices for the device and application.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *