Edited by 
Short answer
The H100 SXM delivers about 3.2 times the dense FP16/BF16 Tensor Core throughput of the A100 SXM (989 vs 312 TFLOPS), adds FP8 at 1,979 TFLOPS, and has 1.6 times the memory bandwidth (3.35 vs 2.04 TB/s). Both have 80 GB of memory. In practice, large-model training and inference typically run 2–3 times faster on H100, more when FP8 is used, while the A100 remains a cost-effective choice for smaller models.
H100 vs A100 specifications
| Specification | A100 SXM 80GB | H100 SXM | H100 advantage |
|---|---|---|---|
| Architecture | Ampere (2020) | Hopper (2022) | – |
| Process and transistors | TSMC 7 nm, 54.2 billion | TSMC 4N, 80 billion | – |
| Streaming multiprocessors | 108 | 132 | 1.2× |
| Tensor Cores | 432 (3rd gen) | 528 (4th gen) | 1.2× |
| FP16 / BF16 Tensor (dense) | 312 TFLOPS | 989 TFLOPS | 3.2× |
| TF32 Tensor (dense) | 156 TFLOPS | 495 TFLOPS | 3.2× |
| FP8 Tensor (dense) | not supported | 1,979 TFLOPS | new |
| FP64 Tensor | 19.5 TFLOPS | 67 TFLOPS | 3.4× |
| Memory | 80 GB HBM2e | 80 GB HBM3 | same |
| Memory bandwidth | 2.04 TB/s | 3.35 TB/s | 1.6× |
| NVLink | 600 GB/s (3rd gen) | 900 GB/s (4th gen) | 1.5× |
| Max power | 400 W | 700 W | 1.75× more |
What changed from Ampere to Hopper
- Twice the work per Tensor Core per clock. Each Hopper Tensor Core does 1,024 FP16 FLOPs per clock against 512 on Ampere, which, with more SMs and higher clocks, gives the 3.2× jump. See how peak FLOPS is calculated.
- FP8 and the Transformer Engine. Hopper added 8-bit floating point and software that chooses between FP8 and 16-bit per layer, doubling throughput again for transformer models. Learn more in FP16, BF16, FP8 and FP4 explained.
- Faster memory and links. HBM3 raises bandwidth to 3.35 TB/s, and fourth-generation NVLink reaches 900 GB/s per GPU.
- New data-movement hardware. The Tensor Memory Accelerator (TMA) and thread block clusters keep Tensor Cores fed more efficiently.
Real-world performance
NVIDIA's launch materials claimed up to 9 times faster training and up to 30 times faster inference on the largest language models versus A100, figures that depend on FP8, very large clusters and specific test setups. For typical work, expect roughly:
- 2–3× faster for BF16 training and compute-bound inference, tracking the 3.2× peak FLOPS gap.
- About 1.6× faster for single-user text generation, which is limited by memory bandwidth.
- Further gains when models run in FP8, which A100 cannot do.
Efficiency improved too: about 1.41 dense FP16 TFLOPS per watt on H100 against 0.78 on A100. Estimate your own job in the training time and cost calculator.
Which should you choose?
- H100 for training or serving large transformer models, anything that benefits from FP8, or when time to result matters more than hourly price.
- A100 for smaller models, fine-tuning, research and batch jobs where a lower hourly price outweighs the slower speed.
- Need more memory? Look at H100 vs H200; for the newest generation see B200 vs H100.
Frequently asked questions
How much faster is H100 than A100?
On paper, 3.2 times in dense FP16/BF16 Tensor Core throughput and 1.6 times in memory bandwidth. Real workloads usually see 1.6–3 times, and more with FP8.
Does A100 support FP8?
No. FP8 Tensor Cores arrived with Hopper. The A100's lowest floating-point Tensor Core formats are FP16 and BF16, plus INT8 and INT4 for integer inference.
Is the A100 still worth using in 2026?
Yes, for the right jobs. Its 80 GB of memory runs most open models up to about 70B in 8-bit or 4-bit, and it is widely available at lower rental prices.
Leave a Reply