Edited by 
Short answer
For running large language models at home, VRAM decides which models you can run and memory bandwidth decides how fast they run. A 12–16 GB card handles 7B–14B models in 4-bit; 24–32 GB (a used RTX 3090, an RTX 4090 or an RTX 5090) runs models up to about 32B; 70B models need about 48 GB or more, such as two 24 GB cards, a 96 GB workstation card or a Mac with 64 GB or more of unified memory. Check any model and GPU in our LLM VRAM calculator.
What matters for local LLMs
Gaming benchmarks and TFLOPS figures are the wrong guide here. For running a chatbot on your own machine, four things matter, roughly in this order:
- VRAM capacity. The whole model, plus its KV cache, must fit in GPU memory. If it spills into system RAM, speed collapses.
- Memory bandwidth. Every generated token reads every weight once, so tokens per second scale with bandwidth, not with Tensor Core throughput.
- Software support. NVIDIA CUDA works with everything (llama.cpp, Ollama, LM Studio, vLLM). Apple Silicon is excellent with llama.cpp and MLX. AMD works well with llama.cpp through ROCm or Vulkan, with fewer options elsewhere.
- Power, noise and price. A 575 W RTX 5090 and a near-silent Mac Studio are very different things to live with.
How much VRAM each model needs
Most people run local models in 4-bit quantization (GGUF Q4_K_M, about 4.85 bits per weight), which keeps quality close to the original at about a third of the FP16 size. Here is what popular models need with an 8K-token context and about 10% runtime overhead:
| Model | Size in Q4_K_M + 8K context | Smallest comfortable GPU |
|---|---|---|
| Llama 3.1 8B, Qwen3 8B | about 6.5 GB | 8–12 GB card |
| Qwen2.5 14B, Phi-4 14B | about 11.6 GB | 16 GB card |
| Mistral Small 3 24B | about 17 GB | 24 GB card |
| Qwen3 32B, DeepSeek-R1-Distill 32B | about 24 GB | 24 GB card with a shorter context, or 32 GB |
| Llama 3.3 70B | about 50 GB | 2 × 32 GB (2 × 24 GB with a shorter context), a 96 GB card or a 64 GB+ Mac |
| gpt-oss-120b (native MXFP4) | about 69 GB | a 96 GB card or a 96 GB+ Mac |
GPUs compared
| GPU or system | Memory | Bandwidth | Best for |
|---|---|---|---|
| NVIDIA RTX 3060 | 12 GB | 360 GB/s | 7–8B models on a budget |
| NVIDIA RTX 4060 Ti / 5060 Ti 16GB | 16 GB | 288 / 448 GB/s | Up to 14B; slower generation |
| NVIDIA RTX 4070 Ti Super / 5070 Ti | 16 GB | 672 / 896 GB/s | Fast 7–14B models |
| NVIDIA RTX 3090 (used) | 24 GB | 936 GB/s | The value pick for 24–32B models |
| NVIDIA RTX 4090 | 24 GB | 1,008 GB/s | Fast 24–32B models |
| NVIDIA RTX 5090 | 32 GB | 1,792 GB/s | The fastest consumer card; 32B with long context |
| NVIDIA RTX PRO 6000 Blackwell | 96 GB | 1,792 GB/s | 70B and gpt-oss-120b on one card |
| AMD Radeon RX 7900 XTX | 24 GB | 960 GB/s | 24 GB for less, with llama.cpp |
| Apple Mac with M4 Max | up to 128 GB | 546 GB/s | Big models, quietly |
| Apple Mac Studio M3 Ultra | up to 512 GB | 819 GB/s | Even the largest open models |
| NVIDIA DGX Spark / AMD Ryzen AI Max+ 395 | 128 GB | 273 / 256 GB/s | Large models in a mini PC, slowly |
Which GPU should you buy?
- Just starting: a 12 GB RTX 3060 or a 16 GB RTX 5060 Ti runs every 7–8B model and most 14B models in 4-bit. Avoid 8 GB cards; they leave almost no room for context.
- Best value: a used RTX 3090. Its 24 GB and 936 GB/s still beat most new cards for local AI.
- Fastest single consumer GPU: the RTX 5090. 32 GB fits 32B models with long context, and 1.8 TB/s makes generation very fast.
- 70B models: two 32 GB cards, or two 24 GB cards with a shorter context (layer split works well over PCIe), a 96 GB RTX PRO 6000, or a Mac with 64 GB or more.
- The biggest open models: a Mac Studio with 256–512 GB of unified memory runs models that would otherwise need a data-center server, at modest speed.
How fast will it be?
A useful estimate for a single user is:
tokens per second ≈ 0.7 × memory bandwidth ÷ model size in bytes
For Llama 3.1 8B in Q4_K_M (about 4.9 GB), that gives roughly 145 tokens per second on an RTX 4090, 78 on an M4 Max and 41 on an RTX 4060 Ti. Anything above about 10 tokens per second reads comfortably. Mixture-of-experts models such as Qwen3 30B-A3B and gpt-oss read only their active experts per token, so they run much faster than their total size suggests. The reason bandwidth rules is explained in what is HBM and how a GPU works.
Using two or more GPUs
llama.cpp, Ollama and LM Studio can split a model's layers across several GPUs, so memory adds up: two 24 GB cards behave like 48 GB. Each token passes through the GPUs one after another, so speed is set by the slower card, and the data exchanged between them is small enough that PCIe is fine. Inference servers such as vLLM can use tensor parallelism for more speed, where a fast link helps; see NVLink and NVSwitch explained.
Common mistakes
- Buying for TFLOPS. A faster chip with less memory runs fewer models.
- Forgetting the KV cache. Long contexts can add many gigabytes; a 70B model at 128K tokens needs about 43 GB of cache in FP16.
- Offloading to system RAM. It works, but generation often drops to a few tokens per second.
Frequently asked questions
Is 8 GB of VRAM enough for a local LLM?
For 7–8B models in 4-bit with a short context, yes. Anything larger, or longer contexts, needs 12 GB or more.
Is a Mac good for running LLMs?
Yes, especially for large models. Unified memory lets a Mac load models no consumer GPU can hold. It generates tokens more slowly than a high-end NVIDIA card with the same model, and prompt processing is notably slower.
Can I use an AMD GPU?
Yes. llama.cpp, Ollama and LM Studio support AMD through ROCm or Vulkan, and a 24 GB RX 7900 XTX is a good value. NVIDIA still has the widest software support, especially for training and for tools such as vLLM.
Ready to check a specific setup? Use the LLM VRAM calculator, learn what the quantization labels mean in LLM quantization explained, or see why context costs memory in KV cache explained.
Sources and further reading
- NVIDIA GeForce RTX 50 series specifications
- Apple Mac Studio technical specifications
- llama.cpp (GGUF formats and multi-GPU support)
Leave a Reply