Edited by 
Short answer
A Tensor Core is a specialised unit inside NVIDIA GPUs that multiplies two small matrices and adds a third in a single operation: D = A × B + C. Neural networks spend most of their time doing exactly this, so Tensor Cores do the heavy lifting whenever an AI model is trained or answers a question.
Why AI is mostly matrix math
A neural network layer takes a list of input numbers, multiplies each one by a learned weight, and adds the results together. Do that for every output of the layer at once and you have a matrix multiplication. A large language model is billions of these weights arranged into matrices, and generating one word means pushing data through all of them.
Attention, the core of the transformer, is also matrix multiplication: queries times keys, then the result times values. Convolutions in image models can be rearranged into matrix multiplications too. Engineers call this general operation GEMM (general matrix multiply), and in most AI workloads it accounts for the large majority of the arithmetic.
That makes one question decisive for AI hardware: how many multiply-adds can the chip do per second? Tensor Cores exist to push that number as high as possible.
What a Tensor Core actually computes
A Tensor Core executes a matrix multiply-accumulate (MMA) on small blocks called tiles:
D = A × B + C
A and B are input tiles. C is the running total, called the accumulator. D is the result, which becomes the C for the next step. The first Tensor Cores, in the 2017 Volta V100, worked on 4×4 tiles. Multiplying two 4×4 matrices takes 4 × 4 × 4 = 64 fused multiply-adds, and each Volta Tensor Core finished all 64 in one clock cycle. A regular CUDA core would need 64 separate instructions for the same work.
There is a second trick in that formula. The inputs A and B use a compact, low-precision format such as FP16, while the accumulator C is kept in full 32-bit precision (FP32). Each product is small and cheap, but the sum that builds up over thousands of steps stays accurate. This is called mixed precision, and it is the reason Tensor Cores can be fast without ruining a model's results.
Worked example
For one cell of D, take a row of A, say [2, 1, 0, 3], and a column of B, say [1, 4, 2, 0]. The dot product is 2×1 + 1×4 + 0×2 + 3×0 = 6. Add the matching cell of C, say 5, and the result is 11. A Tensor Core does this for all 16 cells of the 4×4 tile at the same time.
Tensor Core vs CUDA core in one table
| CUDA core | Tensor Core | |
|---|---|---|
| Works on | Single numbers | Small matrix tiles |
| One instruction does | 1 fused multiply-add | A whole tile: 64 FMAs on Volta, far more on newer chips |
| Typical formats | FP32, FP64, INT32 | FP16, BF16, TF32, FP8, FP6, FP4, INT8 (by generation) |
| Best at | General code, control flow, element-wise math | Matrix multiplication in AI and HPC |
| H100 SXM peak | 67 TFLOPS (FP32) | 989 TFLOPS dense FP16/BF16, 1,979 dense FP8 |
The two work together. Tensor Cores handle the matrix multiplications; CUDA cores handle everything around them, such as activation functions, normalisation and softmax. Our full comparison, Tensor Cores vs CUDA Cores, goes deeper.
Where Tensor Cores live inside a GPU
An NVIDIA GPU is built from many copies of one building block, the streaming multiprocessor (SM). Each SM is split into four partitions. Since the Ampere generation, every partition has its own Tensor Core next to its CUDA cores, registers and warp scheduler. The H100 SXM has 132 SMs, which gives it 528 Tensor Cores working in parallel.
Feeding those units is the hard part. Data travels from high-bandwidth memory (HBM) through the L2 cache into each SM's shared memory and registers before a Tensor Core can use it. Newer generations added hardware just to move tiles faster, such as the Tensor Memory Accelerator in Hopper. See How a GPU works for the full picture, or hover over the exploded chip on our homepage to see the layers.
How big matrices become tiles
Real AI layers are huge. Multiplying two 4,096 × 4,096 matrices takes 4,096³ ≈ 68.7 billion fused multiply-adds. The GPU splits the job into tiles and spreads them across every Tensor Core. With 16 × 16 × 16 tiles, that one multiplication becomes 256 × 256 × 256 ≈ 16.8 million tile operations, executed in parallel across all the SMs.
Libraries such as cuBLAS, cuDNN and CUTLASS do this tiling automatically. When you train a model in PyTorch with mixed precision, these libraries are what put your matrices onto Tensor Cores.
Why smaller numbers mean more speed
Every Tensor Core generation has added smaller number formats. Fewer bits per number means more numbers move through memory and through the multipliers each second. On H100, FP8 math runs at twice the FP16 rate; on Blackwell, FP4 doubles it again.
| Format | Bits | Introduced on Tensor Cores | Typical use |
|---|---|---|---|
| FP16 | 16 | Volta (2017) | Training and inference |
| INT8 / INT4 | 8 / 4 | Turing (2018) | Quantized inference |
| TF32 and BF16 | 19 / 16 | Ampere (2020) | Training without code changes |
| FP8 | 8 | Hopper (2022) | Training and inference of transformers |
| FP6 and FP4 | 6 / 4 | Blackwell (2024) | Fast, low-memory inference |
Lower precision has a cost: each number can hold less detail. Scaling factors, careful rounding and FP32 accumulation keep the accuracy loss small. Our guide to FP16, BF16, FP8 and FP4 explains how each format works and includes a live rounding tool.
A short history
| Generation | Architecture | Year | Flagship | Dense FP16 Tensor peak |
|---|---|---|---|---|
| 1st | Volta | 2017 | V100 | 125 TFLOPS |
| 2nd | Turing | 2018 | T4 | 65 TFLOPS |
| 3rd | Ampere | 2020 | A100 | 312 TFLOPS |
| 4th | Hopper | 2022 | H100 SXM | 989 TFLOPS |
| 5th | Blackwell | 2024 | B200 | 2,250 TFLOPS |
In seven years, dense FP16 throughput grew about 18 times, and using FP4 on Blackwell (9 PFLOPS dense per B200) the gap is more than 70 times. The next platform, Rubin, ships in the second half of 2026 with HBM4 memory. Read the full timeline in Tensor Core generations: Volta to Blackwell.
What Tensor Cores are used for
- Training AI models. Large language and image models are trained on thousands of GPUs whose Tensor Cores run mixed-precision matrix math around the clock.
- Running AI models (inference). Every chatbot reply, image generation and recommendation runs matrix multiplications, increasingly in FP8 or FP4.
- Scientific computing. Since Ampere, data-center Tensor Cores also accelerate FP64 matrix math for simulations in physics, chemistry and climate science.
- Graphics. GeForce RTX cards use Tensor Cores for DLSS upscaling and frame generation, which are neural networks running in real time.
Try it yourself
Reading about D = A × B + C is one thing; computing it is another. Watch the live Tensor Core simulation, see how much hardware a model needs in the AI Model Explorer, then solve real problems in the TensorCore.si Challenge and earn a certificate. New terms are defined in our glossary.
Leave a Reply