Skip to content
,

MHA vs MQA vs GQA vs MLA: Attention Variants and How They Shrink the KV Cache

Multi-head, multi-query, grouped-query and multi-head latent attention compared. How each attention variant works, the KV cache memory it needs per token with worked examples, the quality trade-offs, and why GQA and MLA power models like Llama 3 and DeepSeek-V3.

5 min read

MHA vs MQA vs GQA vs MLA — cover image from TensorCore.si

Short answer

Modern LLMs use different flavours of attention mainly to shrink the KV cache, the memory that grows with every token and limits context length and the number of users per GPU. Multi-head attention (MHA) gives every query head its own keys and values. Multi-query attention (MQA) shares one key/value head across all query heads. Grouped-query attention (GQA), used by Llama 3, Mistral, Qwen and most open models, shares each key/value head among a group of query heads. Multi-head latent attention (MLA), introduced by DeepSeek, compresses keys and values into a small latent vector. GQA cuts KV memory by 4–8× with little quality loss; MLA cuts it further.

Why attention design matters for hardware

In a transformer, each attention head computes queries (Q), keys (K) and values (V) for every token. During generation, the keys and values of all previous tokens are stored in the KV cache so they are not recomputed. Its size is:

KV bytes per token = 2 × layers × KV heads × head_dim × bytes per value

The KV cache often uses more VRAM than the model weights at long context or high concurrency. It also has to be read from memory at every decoding step, so a smaller cache means faster, cheaper generation, because decode is limited by memory bandwidth. Reducing the number of KV heads is the most direct lever.

Multi-head attention (MHA)

The original 2017 design: if a layer has 32 query heads, it also has 32 key heads and 32 value heads. Each head can attend to different patterns, which is good for quality, but the KV cache is as large as it can be.

Used by: GPT-2, GPT-3, the original Llama 1 and many earlier models.

Multi-query attention (MQA)

Proposed by Noam Shazeer in 2019: keep many query heads but use a single shared key head and value head for all of them.

  • KV cache shrinks by a factor equal to the number of heads (for example 32×).
  • Decoding gets much faster.
  • Quality can drop, and training can be less stable, because all heads must use the same keys and values.

Used by: PaLM and Falcon, among others.

Grouped-query attention (GQA)

GQA (Ainslie et al., 2023) is the middle ground: divide query heads into groups, and give each group one shared key/value head. With 32 query heads and 8 KV heads, every 4 query heads share a KV pair.

  • KV cache shrinks by query heads ÷ KV heads (4× in this example).
  • Quality stays very close to MHA.
  • Existing MHA models can even be "uptrained" into GQA with a small amount of extra training.

Used by: Llama 2 70B, Llama 3 and 3.1, Mistral, Qwen, Gemma and most current open-weight models.

Multi-head latent attention (MLA)

Introduced in DeepSeek-V2 (2024) and used in DeepSeek-V3 and R1. Instead of caching full keys and values, MLA projects them into a compressed latent vector per token and caches only that, reconstructing per-head keys and values when needed. Matrix tricks let the up-projection be absorbed into other weights, so inference stays efficient. A small separate component carries rotary position information, because RoPE does not combine directly with the compression.

  • DeepSeek reported that DeepSeek-V2 reduced KV cache size by 93.3% compared with its earlier 67B MHA model, while matching or improving quality.
  • The trade-off is extra compute and implementation complexity.

Worked example: KV cache per token (FP16)

For a model with 32 layers, head dimension 128 and 32 query heads, similar to Llama 3 8B:

VariantKV headsKV cache per token128K-token context
MHA32512 KB64 GB
GQA (Llama 3 8B)8128 KB16 GB
GQA (aggressive)464 KB8 GB
MQA116 KB2 GB

MLA's cache depends on its latent size; in DeepSeek's design it is comparable to a GQA model with only a few KV heads, while keeping quality closer to full MHA.

That difference decides how many users or how much context fits on a GPU. On an 80 GB H100 with 16 GB of weights (8B in FP16), MHA leaves room for about one 128K-token conversation; GQA with 8 KV heads leaves room for about four.

Comparison table

MHAMQAGQAMLA
KV heads= query heads1Groups (e.g. 8)Compressed latent
KV cache sizeLargestSmallestMedium-smallVery small
QualityBest baselineCan degradeClose to MHAClose to or better than MHA (per DeepSeek)
Decode speedSlowestFastestFastFast
ComplexitySimpleSimpleSimpleMore complex
Typical modelsGPT-3, Llama 1PaLM, FalconLlama 3, Mistral, Qwen, GemmaDeepSeek-V2/V3/R1

Other ways to shrink attention memory

  • Sliding-window attention: some layers only look at the last few thousand tokens, used by Mistral 7B, Gemma and gpt-oss in some layers.
  • KV cache quantization: store K and V in FP8 or 4-bit, halving or quartering memory. See LLM quantization.
  • Paged KV cache and prefix sharing: avoid waste and reuse shared prompts. See LLM inference engines compared.
  • Cross-layer KV sharing: adjacent layers reuse the same KV cache in some research designs.
  • Linear and hybrid attention: models that mix attention with state-space or linear-attention layers to keep memory constant over long sequences.

Effect on compute and FlashAttention

GQA and MQA reduce memory traffic but not the number of query heads, so the attention FLOPs stay similar. Kernels like FlashAttention handle GQA natively by loading each KV head once and reusing it for its whole query group, which increases arithmetic intensity and moves decoding closer to the compute-bound region on the roofline. Explore the attention mechanism itself in our attention simulator.

Frequently asked questions

Why do most new models use GQA?

It delivers most of MQA's memory and speed benefit with almost no quality loss, and it is simple to implement and supported by every major inference engine.

Is MLA better than GQA?

DeepSeek's results suggest MLA can match or beat MHA quality with an even smaller cache, but it adds complexity and is less widely supported in tools. GQA remains the default for most labs.

How do I find a model's KV head count?

Check its config.json on Hugging Face for num_key_value_heads alongside num_attention_heads, num_hidden_layers and head_dim, then use the formula above or our VRAM guide.

Does GQA change how I run models locally?

It means you can use longer contexts with the same VRAM. Models with fewer KV heads let you raise the context setting in tools like Ollama further before running out of memory.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *