Free tool · Local and data-center GPUs
LLM VRAM calculator
How much GPU memory do you need to run Llama, Qwen, DeepSeek, Gemma, Mistral or gpt-oss? Pick a model, a quantization (FP16 to GGUF Q4) and a context length, and see the exact memory for weights and KV cache, which GPUs it fits on, and roughly how fast it will run.
- Weights: parameters × bits per weight ÷ 8. A 70B model needs about 140 GB in FP16, 70 GB in 8-bit and about 42 GB in Q4_K_M.
- KV cache: 2 × layers × KV heads × head size × context × bytes. Llama 3.1 8B uses about 128 KB per token in FP16, so 32K tokens add about 4 GB.
- Speed: generating text for one user is limited by memory bandwidth, roughly bandwidth ÷ model size in tokens per second.
Custom model and overhead
Editing these switches the model to "Custom". Overhead covers the runtime, CUDA context and activations.
Total memory needed
Speed is a rough single-user estimate (about 70% of the memory-bandwidth limit). Mac figures assume about 75% of unified memory is available to the GPU. Real numbers vary by runtime (llama.cpp, vLLM, Ollama, LM Studio) and settings.
How the calculation works
1. Weights
Parameters × bits per weight ÷ 8. GGUF quantizations store a small scale with every block of weights, so Q4_K_M averages about 4.85 bits, not 4.
2. KV cache
Each token stores a key and a value vector in every layer: 2 × layers × KV heads × head size × bytes. Models with grouped-query attention (few KV heads) need far less.
3. Overhead and speed
Add about 10% for the runtime and activations. Each generated token reads every active weight once, so tokens per second ≈ bandwidth ÷ bytes read.
| Format | Bits per weight | 8B model | 70B model | Typical use |
|---|---|---|---|---|
| FP16 / BF16 | 16 | 16.1 GB | 141 GB | Full quality, training |
| FP8 / INT8 | 8 | 8.0 GB | 70.6 GB | Data-center serving on Hopper and Blackwell |
| GGUF Q8_0 | 8.5 | 8.5 GB | 75.0 GB | Near-lossless local inference |
| GGUF Q6_K | 6.56 | 6.6 GB | 57.9 GB | High-quality local inference |
| GGUF Q5_K_M | 5.69 | 5.7 GB | 50.2 GB | Balanced quality and size |
| GGUF Q4_K_M | 4.85 | 4.9 GB | 42.8 GB | The most popular local choice |
| GPTQ / AWQ 4-bit, MXFP4 | ≈ 4.25 | 4.3 GB | 37.5 GB | GPU-optimized 4-bit inference |
| GGUF Q3_K_M | 3.91 | 3.9 GB | 34.5 GB | Squeezing onto smaller GPUs |
Sizes are for weights only (8.03B and 70.6B parameters, decimal GB). Real files differ slightly because some layers, such as embeddings, are kept at higher precision.
Frequently asked questions
How much VRAM do I need to run a 70B model?
About 141 GB in FP16, 71 GB in 8-bit and about 43 GB in Q4_K_M for the weights alone, plus a few GB of KV cache. In Q4_K_M a 70B model fits on two 24 GB GPUs such as RTX 3090 or 4090 cards, a single 48 GB card, or a Mac with 64 GB or more of unified memory.
Can I run Llama 3.1 8B on a 12 GB or 8 GB GPU?
Yes. In Q4_K_M its weights are about 4.9 GB, so it fits on an 8 GB GPU with a moderate context; in Q8_0 (about 8.5 GB) it needs a 12 GB card. FP16 needs about 16 GB plus the KV cache.
Why does longer context use so much memory?
The KV cache grows linearly with context length and with the number of concurrent users. Llama 3.1 70B stores about 320 KB per token in FP16, so a full 128K-token context adds about 43 GB on top of the weights.
Do mixture-of-experts models need less VRAM?
No. All experts must be in memory, so VRAM depends on total parameters. MoE models are faster, though, because each token reads only the active experts; this calculator uses active parameters for its speed estimate.
Is the speed estimate accurate?
It is an upper-bound style estimate based on memory bandwidth, which is the main limit for single-user text generation. Prompt processing, batching, CPU offloading and the inference engine all change real results.
Choosing hardware? Read the best GPU for running LLMs locally. Confused by Q4_K_M or AWQ? See LLM quantization explained. Wondering why context uses so much memory? Read KV cache explained.
Go deeper with why memory bandwidth limits AI speed, compare GPUs in the GPU specs database, or see quantization formats bit by bit in the number format converter.
Cite this tool
TensorCore.si Editorial. (2026, November). LLM VRAM calculator. TensorCore.si. https://tensorcore.si/vram-calculator/