Edited by 
Short answer
A CUDA core performs one fused multiply-add on single numbers per clock. A Tensor Core performs a whole small matrix multiply-accumulate at once. On an H100, Tensor Cores deliver 989 TFLOPS of dense FP16 math, about 15 times the 67 TFLOPS of its FP32 CUDA cores. AI uses Tensor Cores for the matrix math and CUDA cores for everything else.
What a CUDA core does
"CUDA core" is NVIDIA's name for one arithmetic lane inside a streaming multiprocessor (SM). In each clock cycle it can compute one fused multiply-add, a × b + c, on 32-bit floating-point numbers. An H100 SXM has 128 of these lanes in each of its 132 SMs, for 16,896 CUDA cores in total.
CUDA cores are general-purpose. They run the arithmetic, comparisons and memory addressing of ordinary GPU programs, from physics simulations to video encoding helpers to the small element-wise steps inside neural networks. They work in groups of 32 threads called warps, all running the same instruction on different data.
What a Tensor Core does
A Tensor Core is a dedicated matrix unit. Instead of one multiply-add, one instruction computes D = A × B + C on small tiles of numbers. The first-generation Tensor Core in Volta processed a 4×4×4 block, 64 fused multiply-adds, every clock. Later generations work on larger tiles in lower-precision formats and finish far more math per instruction. Each SM partition has one Tensor Core alongside its CUDA cores, so an H100 SXM has 528.
Because the operation is fixed, the hardware can be extremely efficient: the multipliers are wired in a grid, operands are reused across the tile, and the accumulator stays inside the unit. That efficiency is why Tensor Cores deliver so much more throughput per watt and per square millimetre of silicon. Our explainer What is a Tensor Core? walks through a worked example.
Side by side
| CUDA core | Tensor Core | |
|---|---|---|
| Operation | a × b + c on scalars | D = A × B + C on matrix tiles |
| Work per instruction | 1 FMA per thread | Hundreds to thousands of FMAs per warp-level instruction |
| Number formats | FP32, FP64, INT32, FP16 pairs | FP16, BF16, TF32, FP8, FP6, FP4, INT8, FP64 (data-center parts) |
| Programmability | Any code | Matrix multiply-accumulate only |
| Count on H100 SXM | 16,896 | 528 |
| Peak on H100 SXM | 67 TFLOPS FP32 | 989 TFLOPS dense FP16/BF16, 1,979 dense FP8 |
| Used for | Activations, softmax, normalisation, indexing, control flow | Linear layers, attention, convolutions |
How much faster is a Tensor Core?
Compare the same GPU. On an H100 SXM:
- FP32 on CUDA cores: 67 TFLOPS
- TF32 on Tensor Cores: 494 TFLOPS dense, about 7× faster for FP32 training code
- FP16 or BF16 on Tensor Cores: 989 TFLOPS dense, about 15×
- FP8 on Tensor Cores: 1,979 TFLOPS dense, about 30×
With 2:4 structured sparsity, where half of the weights are zero in a fixed pattern, each Tensor Core figure doubles again. On Blackwell the gap widens further: a B200 reaches 9 PFLOPS of dense FP4, 9,000 TFLOPS, on its Tensor Cores.
Peak is not the same as real speed
These are peak numbers. Real kernels reach a fraction of them because data has to arrive from memory fast enough. Large training runs commonly achieve 30% to 50% of Tensor Core peak. The AI Model Explorer shows how memory bandwidth limits text generation for real models.
Why AI still needs CUDA cores
A transformer layer is not only matrix multiplication. Between the big GEMMs come steps that are element-wise or need reductions: adding biases, GELU or SiLU activations, softmax in attention, layer normalisation, rotary position embeddings, sampling the next token. These run on CUDA cores. Fast libraries fuse them into the same kernel as the matrix multiplication, in a stage called the epilogue, so data does not make an extra trip to memory.
That balance shapes how GPUs are designed. If Tensor Cores get 2× faster but the CUDA-core work stays the same, the non-matrix steps start to dominate. This is why Hopper added hardware to speed up data movement and why Blackwell Ultra (B300) doubled the speed of the exponent calculations used in attention.
How software uses Tensor Cores
Most developers never program a Tensor Core directly. They use frameworks that call NVIDIA libraries:
- PyTorch and JAX send matrix multiplications to cuBLAS and cuDNN, which pick Tensor Core kernels automatically when inputs are FP16, BF16, TF32 or FP8. Mixed precision (for example
torch.autocast) is usually all it takes. - CUTLASS is NVIDIA's open-source C++ template library for writing custom Tensor Core kernels.
- WMMA is the CUDA C++ API (
nvcuda::wmma) that exposes warp-level matrix fragments, introduced with Volta. - PTX instructions give the lowest-level control:
mmasince Volta and Ampere, asynchronous warp-groupwgmmaon Hopper, andtcgen05on data-center Blackwell.
Do gaming GPUs have Tensor Cores?
Yes. Every GeForce RTX card since the RTX 20 series (Turing, 2018) has Tensor Cores. Games use them for DLSS, which upscales frames and generates new ones with a neural network. They also make RTX cards capable for running local AI models. RTX 40 (Ada Lovelace) cards have fourth-generation Tensor Cores with FP8, and RTX 50 (Blackwell) cards have fifth-generation Tensor Cores with FP4. Older GTX cards have only CUDA cores.
What AMD, Google and Intel call them
"Tensor Core" is NVIDIA's trademark. Other chips have their own matrix engines that do the same job: Matrix Cores in AMD Instinct GPUs, matrix multiply units built as systolic arrays in Google TPUs, and XMX engines in Intel GPUs. The details differ, but the principle is the same: a dedicated block that multiplies tiles of low-precision numbers and accumulates in higher precision.
Ready to test yourself? The TensorCore.si Challenge asks you to compute peak throughput and Tensor Core work for real GPUs.
Leave a Reply