Roofline Model Simulator: Memory-Bound or Compute-Bound?

The roofline model is how GPU engineers answer one question: is this operation limited by how fast the chip can do maths, or by how fast it can move data? The sloped line is memory bandwidth; the flat line is peak Tensor Core speed. A workload can never go above the roof.

How to use it

• Mouse: pick a GPU, a comparison GPU and a number format, choose a workload, then drag the orange dot left or right or use the size slider.
• Keyboard: click the chart, then use W or 1 to 6 to pick a workload, ← and → to change its size (hold Shift for bigger steps), G for the next GPU, C for the comparison GPU and P for the number format.

How to read the chart

Arithmetic intensity is the number of calculations done per byte moved from memory. Low intensity means the chip spends its time waiting for data. The best possible speed is the lower of two limits: peak TFLOPS, or intensity × memory bandwidth. The ridge point is where those meet. An H100 in FP16 has 989 TFLOPS and 3.35 TB/s, so its ridge is about 295 FLOPs per byte: anything below that is memory-bound.

The six workloads

1. LLM decode: each weight read is used once per user, so intensity ≈ 2 × users ÷ bytes per weight. With one user in FP16, it is about 1 FLOP per byte, far left on the chart. This is why chatbots are memory-bound.
2. LLM prefill: the whole prompt reuses each weight many times, so it climbs to the compute roof once the prompt is a few hundred tokens long.
3. Square matrix multiply: intensity grows with matrix size. Large ones are the ideal Tensor Core workload.
4. Attention in decode: reading the KV cache does little maths per byte. Grouped-query attention (GQA) shares each key/value head across several query heads, which multiplies intensity.
5. Element-wise add and 6. softmax / LayerNorm: almost no maths per byte, so they are always memory-bound. Real software fuses them into neighbouring operations instead of running them separately.

Experiments to try

1. Select LLM decode and drag from 1 user upward. Count how many users it takes to reach the ridge on an H100. Then switch to Rubin in FP4: its ridge is several times higher, so it needs far bigger batches to use all of its compute.
2. Switch from FP16 to FP8 and FP4. Each halving of bytes doubles intensity and raises the roof.
3. Compare an RTX 5090 with DGX Spark in FP4: Spark has a much lower memory slope, which is why it generates text slowly despite its 128 GB.
4. Select Attention in decode and raise the GQA group from 1 to 8, as Llama 3.3 70B does.

Peak figures are dense Tensor Core rates from our GPU specs database. Real kernels reach about 60% to 80% of the roof; our LLM benchmarks show measured bandwidth use of 60% to 82%. Related: prefill vs decode, TOPS vs TFLOPS and number formats.

Source

• Williams, Waterman and Patterson, Roofline: An Insightful Visual Performance Model (2009)