Edited by 
Short answer
The GeForce RTX 5090 is the fastest consumer GPU for local AI. Its 32 GB of GDDR7 at 1.79 TB/s runs models up to about 32B parameters at 4-bit, such as Qwen3 32B and Gemma 3 27B, at roughly 55 to 70 tokens per second, and smaller models much faster. It cannot hold a 70B model at 4-bit on its own; that needs two cards, an RTX PRO 6000 (96 GB) or a unified-memory machine. Against an RTX 4090 it has 33% more memory and 78% more bandwidth, so it can be up to about 1.8 times faster on models that fit both.
RTX 5090 specifications for AI
| Specification | RTX 5090 |
|---|---|
| Architecture | Blackwell (GB202) |
| CUDA cores | 21,760 |
| Tensor Cores | 680, fifth generation, with FP4 and FP6 |
| Memory | 32 GB GDDR7, 512-bit |
| Memory bandwidth | 1,792 GB/s |
| AI TOPS (NVIDIA rating) | 3,352 (FP4 with sparsity) |
| Dense FP4 / FP8 | 1,676 / 838 TFLOPS |
| Dense FP16 Tensor | 419 TFLOPS with FP16 accumulate, 209.5 with FP32 accumulate |
| Power | 575 W (1,000 W power supply recommended) |
| Connection | PCIe 5.0 x16, no NVLink |
| Launch price | $1,999 (January 30, 2025) |
The headline 3,352 AI TOPS assumes FP4 and 2:4 sparsity. For language models, the number that predicts speed is memory bandwidth, because generating each token means reading all the active weights. Learn why in TOPS vs TFLOPS.
Which models fit in 32 GB
Memory needed includes the weights, an 8K-token KV cache for one user and 10% overhead. Speeds are estimates from our VRAM calculator: 70% of the bandwidth divided by the bytes read per token.
| Model | Precision | Memory needed | Fits? | Estimated speed |
|---|---|---|---|---|
| Llama 3.1 8B | FP16 | 18.8 GB | Yes | ~73 tokens/s |
| Llama 3.1 8B | Q4_K_M | 6.5 GB | Yes | ~210 tokens/s |
| Qwen3 14B | Q8_0 | 18.8 GB | Yes | ~73 tokens/s |
| Mistral Small 3 24B | Q8_0 | 29.1 GB | Yes, tight | ~47 tokens/s |
| Gemma 3 27B | Q4_K_M | 19.5 GB | Yes | ~71 tokens/s |
| Qwen3 32B | Q4_K_M | 24.2 GB | Yes | ~57 tokens/s |
| Qwen3 30B-A3B (MoE) | Q4_K_M | 21.2 GB | Yes | very fast* |
| gpt-oss-20b (MoE) | MXFP4 | 12.4 GB | Yes | very fast* |
| Llama 3.3 70B | Q4_K_M | 50.0 GB | No, needs two cards | – |
| gpt-oss-120b (MoE) | MXFP4 | 68.6 GB | Only with experts offloaded to system RAM | – |
*Mixture-of-experts models read only their active parameters per token (3.3B for Qwen3 30B-A3B, 3.6B for gpt-oss-20b), so the simple estimate runs to several hundred tokens per second. Real results are lower because routing, attention and software overheads dominate at such small sizes, but these are among the fastest capable models you can run on one card.
Longer contexts need more memory. Qwen3 32B at 4-bit with a 32K context needs about 31 GB, right at the limit. Try your own model and context in the VRAM calculator.
Image and video generation
32 GB is comfortable for image models. The 12-billion-parameter FLUX.1 [dev] transformer fits at full 16-bit precision (about 24 GB), where a 24 GB card usually needs an 8-bit version; its large text encoder can be quantized or moved to system memory. Video models are larger and slower, and the extra memory over a 4090 is what lets many of them run without heavy offloading.
RTX 5090 vs other local AI hardware
| Hardware | Memory | Bandwidth | Power | Best for |
|---|---|---|---|---|
| RTX 4090 | 24 GB GDDR6X | 1,008 GB/s | 450 W | Models up to about 24B at 4-bit |
| RTX 5090 | 32 GB GDDR7 | 1,792 GB/s | 575 W | Fastest single consumer card, models up to about 32B at 4-bit |
| RTX PRO 6000 Blackwell | 96 GB GDDR7 | 1,792 GB/s | 600 W | 70B models at 8-bit on one card, same speed as a 5090 when both fit |
| DGX Spark | 128 GB unified | 273 GB/s | 240 W (system) | Large models and CUDA development, at much lower speed |
| Mac Studio M3 Ultra | Up to 512 GB unified | 819 GB/s | Low (whole system) | Very large models, including big mixture-of-experts models |
The pattern: bandwidth sets speed, capacity sets what fits. The 5090 wins on speed for models under 32 GB. Unified-memory machines win on capacity. Read our DGX Spark guide and best GPU for local LLMs for the full picture, or compare raw specs in the GPU specs database.
Fine-tuning on an RTX 5090
Full fine-tuning needs roughly 16 bytes per parameter for weights, gradients and optimizer state, so even an 8B model will not fit in 32 GB that way. Parameter-efficient methods change that. LoRA trains small adapter matrices on a frozen model, and QLoRA keeps the frozen model in 4-bit. With QLoRA, models up to roughly 30B parameters can be fine-tuned on one 5090 at modest sequence lengths. Estimate bigger jobs with the training cost calculator.
Using two RTX 5090s
Two cards give 64 GB, enough for a 70B model at 4-bit. The 5090 has no NVLink, so the cards talk over PCIe 5.0 at about 64 GB/s each way. Splitting layers across cards (pipeline style, as llama.cpp does) works well for one user. Tensor parallelism, as in vLLM, also works but loses some efficiency over PCIe. Plan for about 1,150 W of GPU power and a power supply and case to match.
Is the RTX 5090 worth it for AI?
- Yes if you mainly run models up to about 32B, want the fastest responses, also generate images or video, or game on the same machine.
- Consider an RTX 4090 or a 16 GB card if you only run 8B to 14B models, where they are fast enough and much cheaper.
- Consider an RTX PRO 6000, two 5090s, DGX Spark or a Mac if you need 70B-class or larger models on one machine.
Frequently asked questions
Can the RTX 5090 run a 70B model?
Not entirely in its own memory at usable quality. Llama 3.3 70B needs about 50 GB at 4-bit with an 8K context. You can split it between the GPU and system RAM, but speed drops sharply. Two RTX 5090s, an RTX PRO 6000 or a unified-memory system run it properly.
How much faster is the RTX 5090 than the RTX 4090 for LLMs?
For models that fit on both, token generation speed scales with memory bandwidth: 1,792 against 1,008 GB/s, about 1.78 times on paper. Real gains are usually somewhat lower because not every part of the work scales with bandwidth. The 5090 can also run models that do not fit in the 4090's 24 GB.
Does the RTX 5090 support FP4?
Yes. Its fifth-generation Tensor Cores support FP4 and FP6, including NVIDIA's NVFP4. Software such as TensorRT-LLM can use them; in llama.cpp, 4-bit formats such as Q4_K_M and MXFP4 mostly help by shrinking memory traffic.
What power supply does an RTX 5090 need?
NVIDIA recommends a 1,000 W power supply for the 575 W card, connected through a single 16-pin power connector or an adapter.
