Edited by 
Short answer
The H100 and H200 use the same Hopper GPU with the same Tensor Cores: 989 dense FP16 TFLOPS and 1,979 dense FP8 TFLOPS. The difference is memory. The H200 has 141 GB of HBM3e at 4.8 TB/s against the H100's 80 GB of HBM3 at 3.35 TB/s, which is 1.76 times the capacity and 1.43 times the bandwidth. That makes the H200 noticeably faster for large language model inference and long contexts, and identical for compute-bound work.
H100 vs H200 specifications
| Specification | H100 SXM | H200 SXM | Difference |
|---|---|---|---|
| GPU | Hopper GH100 | Hopper GH100 | same chip |
| FP16 / BF16 Tensor (dense) | 989 TFLOPS | 989 TFLOPS | same |
| FP8 Tensor (dense) | 1,979 TFLOPS | 1,979 TFLOPS | same |
| FP64 Tensor | 67 TFLOPS | 67 TFLOPS | same |
| Memory | 80 GB HBM3 | 141 GB HBM3e | 1.76× |
| Memory bandwidth | 3.35 TB/s | 4.8 TB/s | 1.43× |
| NVLink | 900 GB/s | 900 GB/s | same |
| Max power | up to 700 W | up to 700 W | same |
| Launched | 2022 | announced 2023, shipping 2024 | – |
Why memory alone makes a difference
When a model generates text, every new token reads all of the model's weights plus the KV cache from memory. The math per byte is small, so the GPU waits on memory, not on its Tensor Cores. Raising bandwidth from 3.35 to 4.8 TB/s therefore speeds up token generation by up to about 1.4 times, and the extra capacity lets each GPU hold a bigger model, a longer KV cache or more simultaneous users. NVIDIA reports up to 1.9 times faster Llama 2 70B inference on H200 than on H100, combining both effects. The background is in what is HBM.
What fits on each GPU
| Workload | H100 (80 GB) | H200 (141 GB) |
|---|---|---|
| 70B model in FP8 (about 71 GB of weights) | Fits, little room for KV cache | Fits with about 60 GB left for KV cache |
| 70B model in FP16 (about 141 GB) | Needs two GPUs | Weights only just fit; two GPUs in practice |
| Long contexts (128K tokens on a 70B model, about 43 GB of FP16 cache) | Needs extra GPUs | Much more headroom |
| Training throughput (compute-bound) | Baseline | About the same |
Which should you choose?
- H200 for serving large models, long contexts and high concurrency, where memory capacity and bandwidth decide cost per token.
- H100 for compute-bound training and smaller models, where the two perform the same and the H100 is usually cheaper and easier to get.
- Considering Blackwell? See B200 vs H100, or compare every chip in the GPU specs database.
Frequently asked questions
Is the H200 a new GPU?
It uses the same Hopper GPU as the H100, paired with newer, larger HBM3e memory. It is best thought of as an H100 with a memory upgrade.
Is H200 faster than H100 for training?
Only modestly. Training large models is mostly compute-bound, and both have the same Tensor Cores. Gains come from fitting larger batches or avoiding some model splitting thanks to the extra memory.
Can H100 and H200 run the same software?
Yes. They share the Hopper architecture, CUDA features and FP8 support, so code and containers run unchanged.
Leave a Reply