Edited by 
Short answer
AI runs on GPUs because neural networks are mostly huge matrix multiplications: millions of independent multiply-adds that can all happen at once. A CPU has a few dozen powerful cores built to run varied, branching code one step after another. A GPU has thousands of simpler cores plus Tensor Cores that multiply whole matrices per instruction, and memory that is several times to tens of times faster. An H100 delivers 989 dense FP16 TFLOPS and 3.35 TB/s of memory bandwidth, far beyond any CPU. CPUs still handle data loading, orchestration and small models.
GPU vs CPU at a glance
| CPU | GPU | |
|---|---|---|
| Design goal | Finish each task as quickly as possible (low latency) | Finish as much work as possible per second (high throughput) |
| Cores | 8 to 24 in desktops, up to 192 in servers | Thousands: 21,760 CUDA cores on an RTX 5090 |
| Matrix units | Vector and matrix extensions on some chips | Tensor Cores in every streaming multiprocessor |
| Memory bandwidth | About 100 GB/s (desktop) to about 600 GB/s (12-channel server) | 1.8 TB/s (RTX 5090) to 8 TB/s (B200) and 19.2 TB/s (Rubin) |
| Memory size | Large: hundreds of GB to TBs of RAM | Smaller: 24 to 288 GB per GPU |
| Good at | Branching logic, operating systems, databases, small batches | The same operation on huge amounts of data |
Why neural networks suit GPUs
Each layer of a neural network multiplies an input matrix by a weight matrix. For a 70-billion-parameter model, producing one token takes about 140 billion floating-point operations, and every one of the multiply-adds in a layer can run independently. That is exactly the kind of work GPUs were built for in graphics, where millions of pixels each run the same small program.
Three things give GPUs their lead:
- Parallelism: tens of thousands of threads running at once, grouped into warps of 32 that execute the same instruction. See how a GPU works.
- Tensor Cores: dedicated units that multiply small matrices in one step, at FP16, FP8 and now FP4. They provide most of a modern GPU's AI throughput. See Tensor Cores vs CUDA Cores.
- Memory bandwidth: HBM or GDDR memory feeds the cores at terabytes per second. For generating text, bandwidth usually decides speed. See what is HBM.
What the gap looks like for an LLM
Token generation reads every active weight once per token, so a simple estimate is bandwidth divided by model size. For Llama 3.1 8B at 4-bit, about 4.9 GB of weights:
| Hardware | Memory bandwidth | Rough upper limit |
|---|---|---|
| Desktop CPU, dual-channel DDR5 | About 90 GB/s | About 13 tokens/s |
| Server CPU, 12-channel DDR5 | About 600 GB/s | About 85 tokens/s |
| RTX 4090 | 1,008 GB/s | About 145 tokens/s |
| RTX 5090 | 1,792 GB/s | About 255 tokens/s |
These assume 70% of the bandwidth is achieved. Real results also depend on software, and the CPU falls much further behind when processing long prompts, which is compute-bound. Read prefill vs decode for why. Try any model and GPU in the VRAM calculator.
Training: where the gap is largest
Training repeats the forward and backward pass over trillions of tokens, roughly 6 floating-point operations per parameter per token. That is pure compute, where Tensor Cores give GPUs a lead of one to two orders of magnitude over CPUs. Large models are trained on thousands of GPUs linked by NVLink and fast networks. Estimate a run with the training cost calculator or read how much it costs to train an LLM.
When a CPU is enough
- Small models: classic machine learning such as gradient-boosted trees, and small neural networks, run well on CPUs.
- Light local LLM use: a 3B to 8B model at 4-bit is usable on a modern CPU for occasional chat, just slower.
- Very large mixture-of-experts models that do not fit in GPU memory can run from system RAM on servers with many memory channels, with only the experts in use read per token.
- Everything around the model: data loading, tokenization and scheduling still run on the CPU in every AI system.
Laptops and phones also have NPUs, small AI accelerators for always-on features. See NPU vs GPU vs TPU.
Frequently asked questions
Can I run AI without a GPU?
Yes, for small models and patient use. Tools such as llama.cpp run language models on CPUs, but generation is several times slower than on a modern GPU and long prompts take much longer to process.
Why are GPUs better than CPUs for deep learning?
Deep learning is dominated by large matrix multiplications that can be split into millions of independent operations. GPUs have thousands of cores, Tensor Cores for matrix math and much faster memory, so they finish this work many times faster.
How many CPU cores equal a GPU?
There is no exact conversion, because they do different kinds of work. On matrix math, one data-center GPU matches many high-end server CPUs; on branching, serial code a single CPU core is faster than any GPU core.
Do I need a good CPU for an AI GPU?
A mid-range modern CPU is usually enough for one or two GPUs. It matters more for data-heavy training, where it must decode and feed data fast enough to keep the GPU busy.
