Edited by 
Short answer
To run an LLM, you need roughly VRAM = model weights + KV cache + 1 to 2 GB overhead. Weights take about parameters (in billions) × bits per weight ÷ 8 gigabytes: an 8B model needs about 16 GB at FP16, 8 GB at 8-bit and about 5 GB at 4-bit. The KV cache grows with context length; Llama 3.1 8B needs about 128 KB per token at FP16, so 1 GB for an 8,000-token context. In practice, a 12 to 16 GB GPU runs 8B to 14B models at 4-bit, 24 GB runs up to about 32B, and 70B models need about 40 to 48 GB at 4-bit.
The three parts of LLM memory use
When you load a model on a GPU, the VRAM holds three things:
- Model weights: the parameters themselves. This is usually the biggest part.
- KV cache: the stored keys and values for every token in the conversation so far, so the model does not recompute them. It grows with context length and with the number of simultaneous users.
- Overhead: the CUDA context, temporary activations and framework buffers, typically 0.5 to 2 GB.
Step 1: memory for the weights
Weights (GB) ≈ number of parameters (billions) × bits per weight ÷ 8
| Format | Bits per weight | Bytes per parameter | 8B model |
|---|---|---|---|
| FP32 | 32 | 4 | 32 GB |
| FP16 / BF16 | 16 | 2 | 16 GB |
| 8-bit (Q8_0, FP8, INT8) | about 8 to 8.5 | about 1 | 8 to 8.5 GB |
| 5-bit (Q5_K_M) | about 5.5 to 5.7 | about 0.7 | about 5.7 GB |
| 4-bit (Q4_K_M, AWQ, GPTQ) | about 4.5 to 4.9 | about 0.6 | about 4.9 GB |
GGUF quantization formats store some extra scaling data, so "4-bit" files usually average slightly more than 4 bits per weight. Read our guide to quantization formats to choose between them.
Step 2: memory for the KV cache
KV cache per token (bytes) = 2 × layers × KV heads × head dimension × bytes per value
The 2 is for keys and values. Modern models use grouped-query attention, with far fewer KV heads than attention heads, which keeps this small. Multiply by your context length (and number of parallel requests).
| Model | Layers | KV heads × head dim | KV cache per token (FP16) | At 8K tokens | At 32K tokens | At 128K tokens |
|---|---|---|---|---|---|---|
| Llama 3.1 8B | 32 | 8 × 128 | 128 KB | 1 GB | 4 GB | 16 GB |
| Llama 3.1 70B | 80 | 8 × 128 | 320 KB | 2.5 GB | 10 GB | 40 GB |
Long contexts can need as much memory as the model itself. Many runtimes let you quantize the KV cache to 8-bit, halving this. The KV cache explained article goes deeper.
VRAM needed by model size
Approximate totals for weights plus a modest 4K to 8K context and overhead. Use these as starting points.
| Model size | FP16 | 8-bit | 4-bit | Example GPUs that fit at 4-bit |
|---|---|---|---|---|
| 3 to 4B | 8 to 10 GB | 5 to 6 GB | 3 to 4 GB | Almost any 6 GB or larger GPU |
| 7 to 8B | 17 to 19 GB | 10 to 11 GB | 6 to 7 GB | RTX 3060 12 GB, RTX 4060 Ti |
| 12 to 14B | 28 to 32 GB | 15 to 17 GB | 9 to 11 GB | 12 to 16 GB cards |
| 27 to 32B | 58 to 68 GB | 30 to 36 GB | 18 to 22 GB | RTX 3090, RTX 4090 (24 GB) |
| 70B | 145 to 150 GB | 75 to 80 GB | 40 to 45 GB | 2 × 24 GB, RTX 5090 + offload, 48 GB cards, 64 GB+ Macs |
| 120B-class MoE | depends on format | depends on format | about 60 to 70 GB | 80 GB GPUs, 96 to 128 GB unified memory |
| 405B | about 810 GB | about 410 GB | about 230 GB | Multi-GPU servers |
Mixture-of-experts (MoE) models are a special case: every expert must sit in memory, so VRAM depends on the total parameter count, but speed depends on the smaller number of active parameters per token.
Worked example
You want to run Llama 3.1 8B at Q4_K_M with a 16,000-token context:
- Weights: 8 × 4.85 ÷ 8 ≈ 4.9 GB
- KV cache: 128 KB × 16,000 ≈ 2 GB at FP16
- Overhead: about 1 GB
Total: about 8 GB. An 8 GB card is tight; a 12 GB card is comfortable.
What about training and fine-tuning?
Training needs far more memory than inference, because it stores gradients and optimizer state:
- Full fine-tuning with Adam in mixed precision: about 16 bytes per parameter plus activations, so around 130 GB for an 8B model. That requires multiple data center GPUs or memory-saving techniques.
- LoRA: the base model is frozen, so memory is roughly inference memory plus activations and the small adapters. An 8B model at 16-bit with LoRA typically needs 20 to 24 GB.
- QLoRA: the frozen base is loaded in 4-bit, letting you fine-tune an 8B model in around 10 to 12 GB and a 70B model on a single 48 GB GPU.
How to fit a bigger model
- Quantize further: 4-bit is the usual sweet spot; below about 3.5 bits quality drops noticeably.
- Shorten the context or quantize the KV cache.
- Offload layers to CPU RAM: llama.cpp and Ollama can split a model between GPU and system memory. It works, but offloaded layers run much slower, limited by RAM bandwidth.
- Use more GPUs: inference engines can split a model across several cards.
- Use unified memory: Apple silicon Macs and systems like the DGX Spark can give the GPU 64 to 128 GB or more, at lower bandwidth than a high-end GPU.
Once a model fits, the speed you get depends mainly on memory bandwidth. Our best GPU for local LLMs guide combines both factors, and our guide to running an LLM locally shows how to get started.
Frequently asked questions
How much VRAM do I need for a 7B or 8B model?
About 6 to 7 GB at 4-bit, 10 GB at 8-bit and 17 GB or more at FP16, plus extra for long contexts.
Can I run a 70B model on a 24 GB GPU?
Not entirely in VRAM at usable quality. You can offload part of it to system RAM, but it will be slow. Two 24 GB GPUs, a 48 GB card or a Mac with 64 GB or more of unified memory are better options.
Does context length really use that much memory?
Yes. At 128K tokens, Llama 3.1 8B's KV cache alone takes about 16 GB at FP16, more than the 4-bit weights.
Is 8 GB of VRAM enough for AI?
It runs small and medium models (up to about 8B at 4-bit) with modest context. For larger models and longer contexts, 12 to 16 GB or more is better.
Sources and further reading
- Meta, Llama 3.1 model card and configuration
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)
- llama.cpp, GGUF quantization types
- Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models (2023)
Leave a Reply