Skip to content
,

What Is Mixture of Experts (MoE)? How Sparse AI Models Work and What They Mean for GPUs

Mixture of experts (MoE) models split their layers into many expert networks and use only a few per token. How routing works, total vs active parameters, real examples like Mixtral, DeepSeek-V3, Llama 4 and gpt-oss, and the memory and interconnect demands on hardware.

5 min read

Mixture of Experts (MoE) — cover image from TensorCore.si

Short answer

A mixture of experts (MoE) model replaces some of a neural network's layers with many smaller "expert" sub-networks plus a router that sends each token to only a few of them. The model can have a very large total parameter count while using only a small fraction, the active parameters, for each token. DeepSeek-V3, for example, has 671 billion parameters but activates about 37 billion per token. MoE makes large models faster and cheaper to run per token, but all experts must still fit in memory, which makes VRAM capacity and GPU interconnect the main hardware challenges.

The idea in plain English

Imagine a hospital. Instead of one doctor who must know everything, there are many specialists and a receptionist who sends each patient to the two most suitable ones. The hospital as a whole knows a lot, but each patient only takes up the time of two doctors.

An MoE model works the same way. The "receptionist" is a small router network, the "specialists" are expert networks, and each "patient" is a token.

How MoE works inside a transformer

A standard transformer layer has two main parts: an attention block and a feed-forward network (FFN). The FFN holds most of the parameters. In an MoE model:

1. The single FFN is replaced by many expert FFNs (for example 8, 64, 128 or 256). 2. For each token, a router computes a score for every expert. 3. The token is sent to the top-k experts, usually 1, 2 or 8. 4. The outputs of the chosen experts are combined, weighted by the router scores.

Attention layers are usually shared by all tokens. Some designs also include one or more shared experts that every token uses, alongside the routed ones.

Because only k experts run for each token, the compute per token depends on the active parameters, not the total.

Total vs active parameters: real examples

ModelTotal parametersActive per tokenExperts
Mixtral 8x7B (Mistral, 2023)46.7B12.9B8, top-2
DeepSeek-V3 (2024)671B37B256 routed + 1 shared, 8 routed active
Llama 4 Scout (Meta, 2025)109B17B16
Llama 4 Maverick (Meta, 2025)400B17B128
gpt-oss-120b (OpenAI, 2025)117B5.1B128, top-4
gpt-oss-20b (OpenAI, 2025)21B3.6B32, top-4

Many frontier models are also widely believed to use MoE, though most closed-model developers do not publish their architectures.

  • More knowledge for the same compute: a model with 400B total parameters can store far more than a 17B dense model, while costing roughly the same compute per token.
  • Cheaper training: for a fixed training compute budget, MoE models typically reach better quality than dense models.
  • Faster generation: token generation is limited by how many bytes of weights must be read from memory per token. With MoE, only active experts' weights are read, so decode can be much faster than a dense model of the same total size. See prefill vs decode.

The catch: memory and communication

All experts must live in memory

Even though only a few experts run per token, any expert could be chosen for the next token, so all of them must be loaded in GPU memory (or very fast memory nearby). A 671B-parameter model needs about 671 GB in FP8, or roughly 335 GB in 4-bit, before the KV cache. That is why MoE models favour GPUs with large HBM capacity, such as the H200 (141 GB) and B200 (180 GB class), and multi-GPU servers.

For local use, MoE is a mixed blessing: a large MoE model needs lots of VRAM or system RAM, but once loaded it runs fast because few parameters are read per token. This is why MoE models are popular on Apple Silicon Macs with large unified memory and why tools like llama.cpp can offload experts to system RAM. gpt-oss-120b was designed to run on a single 80 GB GPU using 4-bit MXFP4 weights.

Experts are spread across GPUs

Large MoE models use expert parallelism: different GPUs hold different experts. Every layer, tokens must be sent to whichever GPUs hold their chosen experts and the results sent back. This all-to-all communication is heavy, so MoE performance depends strongly on interconnect bandwidth. NVIDIA's NVLink and NVSwitch and rack-scale systems like the GB200 NVL72, which link 72 GPUs in one NVLink domain, are well suited to this workload.

Load balancing

If the router sends most tokens to a few popular experts, those GPUs become bottlenecks while others sit idle. Training uses extra techniques, such as auxiliary balancing losses or the bias-based balancing used in DeepSeek-V3, to spread tokens evenly.

MoE vs dense models

Dense modelMoE model
Parameters used per tokenAllA small fraction
Compute per tokenProportional to total sizeProportional to active size
Memory neededTotal sizeTotal size (same or more)
Generation speed at equal total sizeSlowerFaster
Training complexitySimplerRouting, balancing, communication
Fine-tuningStraightforwardTrickier, can be unstable
Best fitSmall and medium models, edge devicesVery large models served at scale

A short history

  • 1991: Jacobs, Jordan, Nowlan and Hinton propose "adaptive mixtures of local experts".
  • 2017: Google's Noam Shazeer and colleagues scale MoE layers to over 100 billion parameters in an LSTM model with a sparsely gated MoE layer.
  • 2021: Google's Switch Transformer routes each token to a single expert and scales to over a trillion parameters.
  • 2023 to 2025: Mixtral, DeepSeek-V2 and V3, Qwen MoE models, Llama 4 and gpt-oss bring MoE into the mainstream of open models.

Frequently asked questions

Are the experts specialised in topics like maths or history?

Not in a neat, human-readable way. Studies of models like Mixtral found experts tend to specialise more in token types and syntax than in subject areas. The specialisation is learned automatically and is hard to interpret.

Is an MoE model with 400B parameters as smart as a 400B dense model?

Usually not. An MoE model typically performs better than a dense model with the same number of active parameters, but somewhat below a dense model with the same total parameters. Its advantage is efficiency.

Can I run an MoE model on my PC?

Smaller ones, yes. gpt-oss-20b and Qwen's smaller MoE models run on GPUs with 16 GB or more, and quantized versions of larger MoE models can run on high-memory Macs or PCs with lots of system RAM. See how to run an LLM locally and how much VRAM you need.

Why does MoE matter for GPU buyers?

It shifts the bottleneck from compute towards memory capacity and interconnect. That is one reason newer data center GPUs keep adding HBM and NVLink bandwidth, and why prices for high-memory GPUs remain strong; see our GPU cloud price tracker.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *