Edited by 
Short answer
The compute to train a dense LLM is about 6 × parameters × training tokens floating-point operations. Divide by what your GPUs actually deliver (typically 30% to 45% of their peak) to get GPU-hours, then multiply by the price per GPU-hour. A 7B model on 2 trillion tokens needs about 59,000 H100-hours, roughly $120,000 at $2 per hour. Meta reported 30.8 million H100-hours for Llama 3.1 405B; DeepSeek reported 2.8 million H800-hours, about $5.6 million, for DeepSeek-V3. Those figures cover only the final run, not research, failed runs, data or staff.
The 6ND rule
For a dense transformer with N parameters trained on D tokens:
Training compute ≈ 6 × N × D floating-point operations
The forward pass costs about 2 operations per parameter per token, one multiply and one add. The backward pass costs about twice that, giving 6 in total. For mixture-of-experts models, use the active parameters per token, not the total. Attention adds a little more for long sequences.
From FLOPs to GPU-hours and dollars
- Compute: FLOPs = 6 × N × D.
- Effective speed per GPU: peak dense throughput × utilization (MFU). An H100 has 989 dense BF16 TFLOPS; large runs typically reach 35% to 45%.
- GPU-hours: FLOPs ÷ effective speed ÷ 3,600.
- Cost: GPU-hours × price per GPU-hour.
The training cost calculator does these steps for any model, GPU, precision and price.
Worked examples on H100
Assuming BF16 at 40% utilization (about 396 TFLOPS per GPU) and $2 per GPU-hour:
| Model | Training tokens | Compute | H100-hours | Compute cost |
|---|---|---|---|---|
| 1B | 1 trillion | 6 × 1021 FLOPs | About 4,200 | About $8,400 |
| 7B | 2 trillion | 8.4 × 1022 FLOPs | About 59,000 | About $118,000 |
| 70B | 15 trillion | 6.3 × 1024 FLOPs | About 4.4 million | About $8.8 million |
| 405B | 15.6 trillion | 3.8 × 1025 FLOPs | About 26.6 million | About $53 million |
On 1,000 GPUs, the 7B run takes about 2.5 days and the 70B run about six months. Newer GPUs cut the hours: a B200 has about 2.3 times an H100's dense BF16 throughput, and FP8 training doubles throughput again where it can be used.
What real models used
| Model | Reported compute | Tokens | Notes |
|---|---|---|---|
| Llama 3.1 405B (Meta, 2024) | 30.84 million H100-hours | About 15 trillion | About 3.8 × 1025 FLOPs; implies roughly 35% of H100 BF16 peak across the whole run |
| DeepSeek-V3 (2024) | 2.788 million H800-hours | 14.8 trillion | Mixture-of-experts with 37B active parameters, trained mostly in FP8; DeepSeek put the cost at $5.576 million at $2 per GPU-hour |
DeepSeek-V3 used about 11 times fewer GPU-hours than Llama 3.1 405B for a model of similar quality. Most of the gap comes from activating only 37B of its 671B parameters per token, plus FP8 training and careful engineering. In 2023, OpenAI's chief executive Sam Altman said GPT-4 cost more than $100 million to train.
Costs the headline numbers leave out
- Experiments and failed runs: smaller test models, ablations and restarts can cost as much as the final run or more. DeepSeek's figure explicitly excludes earlier research.
- Owning hardware: buying GPUs, networking, power and cooling, instead of renting at an hourly rate.
- Data: collection, cleaning, licensing and human-written examples.
- Post-training: fine-tuning and reinforcement learning, which have become a growing share of compute for reasoning models.
- People: researchers and engineers, often the largest cost of all.
- Serving: a popular model can cost more to run for users than it cost to train.
What fine-tuning costs
Fine-tuning an existing model costs a tiny fraction of pretraining. Using the same formula, fine-tuning an 8B model on 100 million tokens is about 4.8 × 1018 FLOPs, a few H100-hours. With LoRA or QLoRA, models up to around 30B parameters can be fine-tuned on a single 32 GB card such as the RTX 5090.
Frequently asked questions
How much does it cost to train a 7B model from scratch?
For about 2 trillion tokens on H100s at 40% utilization, roughly 59,000 GPU-hours, or about $120,000 at $2 per hour. Training on more tokens, as modern small models do, raises the cost proportionally.
How many GPUs does it take to train an LLM?
It depends on the deadline. Total GPU-hours are fixed by the compute; more GPUs finish sooner. Meta trained Llama 3.1 405B on up to 16,000 H100s, while a 7B model can be trained on a few hundred GPUs in weeks.
What is MFU?
Model FLOPs utilization: the share of a GPU's peak throughput that a training run actually achieves on the model's math. It is reduced by communication, memory traffic, pipeline bubbles and failures. Large runs typically reach 30% to 50%.
Is it cheaper to train on TPUs or AMD GPUs?
Sometimes. The formula is the same; what changes is throughput per dollar and how well your software uses the hardware. Compare chips in the specs database and read TPU vs GPU and AMD vs NVIDIA for AI.
