What does it take to run an AI model?
Pick a model to see how much GPU memory its weights need, how many Tensor Core operations each token costs, and roughly how fast it can run on today's data-center GPUs.
Parameter counts are as published by each developer. Closed models don't publish their sizes, so enter your own estimate.
Four formulas that size any model
Memory for weights
parameters × bytes per value. FP16 uses 2 bytes, FP8 uses 1 and FP4 uses half a byte, which is why lower precision lets a model fit on fewer GPUs.
Work per token
2 × active parameters FLOPs. Every active weight takes part in one multiply-add, and Tensor Cores do those multiply-adds in large matrix tiles.
Mixture-of-experts
MoE models store many experts but run only a few per token. Memory follows the total size, while compute follows the active size.
One user vs many
Generating text for one user is limited by memory bandwidth, since all active weights are read for every token. Large batches reuse those weights and become limited by Tensor Core compute.
Understand the numbers behind the calculator
FP16, BF16, FP8 and FP4 →
Why each halving of precision halves memory and doubles Tensor Core speed.
Hopper vs Blackwell →
H100, H200, B200 and B300 compared on memory, bandwidth and FP4.
Why bandwidth limits speed →
The roofline model explains why one-user text generation is memory-bound.
Test yourself →
The Architect level of the TensorCore.si Challenge has KV cache and MoE sizing problems.