Tensor Core & AI Hardware Glossary (A–Z)

Every term you will meet in Tensor Core and AI hardware articles, defined in plain language. Terms link together with our guides, so you can go deeper whenever you need to.

Accumulator: The running total that a multiply-add adds into. Tensor Cores usually keep the accumulator in higher precision (often FP32) than the inputs, so many small low-precision products can be summed without losing accuracy.
AI accelerator: A chip designed mainly to run neural networks fast and efficiently. GPUs with Tensor Cores, Google TPUs, AWS Trainium and Intel Gaudi are all AI accelerators.
AI TOPS: Trillions of operations per second, the headline AI figure NVIDIA gives for GeForce and RTX cards. It is usually measured at FP4 or INT8 with sparsity, so it is far higher than the dense FP16 rate.
Ampere: NVIDIA's 2020 GPU architecture behind A100 and the RTX 30 series. It added TF32, BF16 and 2:4 structured sparsity to Tensor Cores.
Arithmetic intensity: The number of floating-point operations a kernel performs for each byte it moves to or from memory. High intensity means the work is limited by compute; low intensity means it is limited by memory bandwidth.
ASIC: Application-specific integrated circuit: a chip built for one job. TPUs and Trainium are ASICs for neural networks, while GPUs stay programmable for many kinds of work.
Attention: The transformer operation in which each token weighs earlier tokens to decide what matters. Its cost grows with context length, and its stored keys and values form the KV cache.
Batch size: The number of inputs processed together. Larger batches reuse each weight read for more tokens, which raises arithmetic intensity and GPU utilization.
BF16 (bfloat16): A 16-bit format with 1 sign bit, 8 exponent bits and 7 mantissa bits. It keeps the same range as FP32 with less precision, which makes it popular for training. Tensor Cores support it from the Ampere generation.
Blackwell: NVIDIA's 2024 architecture behind B200, B300, GB200 and the RTX 50 series. Its fifth-generation Tensor Cores add FP4 and FP6, and the data-center chips join two dies into one GPU.
Chiplet: A smaller die combined with others in one package to act as one processor. AMD Instinct GPUs and NVIDIA Blackwell use multi-die designs to go beyond the size limit of a single die.
Context window: The maximum number of tokens a model can consider at once, including the prompt and its own output. Longer contexts need a larger KV cache.
CoWoS: Chip-on-Wafer-on-Substrate, the TSMC packaging that places GPU dies and HBM stacks side by side on a silicon interposer. Its capacity has been a key limit on AI GPU supply.
CUDA: NVIDIA's programming platform and toolkit for its GPUs. Libraries built on it, such as cuBLAS and cuDNN, call Tensor Cores for the heavy matrix math.
CUDA core: NVIDIA's name for a scalar arithmetic lane inside a streaming multiprocessor. Each one performs one fused multiply-add on single numbers per clock.
Data parallelism: Training on many GPUs by giving each a copy of the model and a different slice of the batch, then averaging the gradients. Sharded variants such as FSDP and ZeRO split the copies to save memory.
Decode: The token-by-token generation phase of LLM inference. Each new token reads all the active weights and the KV cache, so decode speed is usually limited by memory bandwidth.
DGX: NVIDIA's line of complete AI systems, from the eight-GPU DGX B200 server to the DGX Spark desktop.
Die: A single piece of silicon cut from a wafer. The largest AI GPU dies approach the reticle limit of roughly 850 square millimetres.
Expert parallelism: Splitting a mixture-of-experts model so that different GPUs hold different experts, with tokens routed between them.
FlashAttention: An attention algorithm that works on tiles in fast on-chip memory instead of writing the full attention matrix to HBM. It cuts memory traffic and makes long contexts practical.
FLOPS: Floating-point operations per second. TFLOPS is 10¹² and PFLOPS is 10¹⁵. One fused multiply-add counts as two floating-point operations.
FMA (fused multiply-add): One instruction that computes a × b + c with a single rounding step. It is the basic unit of work inside both CUDA cores and Tensor Cores.
FP16: The 16-bit IEEE half-precision format: 1 sign, 5 exponent and 10 mantissa bits, with a maximum value of 65,504. The original Tensor Core input format.
FP32: The 32-bit IEEE single-precision format: 1 sign, 8 exponent and 23 mantissa bits. The standard format for general GPU math and Tensor Core accumulation.
FP4 (E2M1): A 4-bit floating-point format with 1 sign, 2 exponent and 1 mantissa bit. Its positive values are 0, 0.5, 1, 1.5, 2, 3, 4 and 6. Blackwell Tensor Cores run it natively, combined with per-block scale factors.
FP6: A 6-bit floating-point format available as E3M2 or E2M3, supported by Blackwell Tensor Cores as part of the microscaling family.
FP8 (E4M3 and E5M2): Two 8-bit floating-point formats introduced with Hopper. E4M3 has more precision and a maximum of 448; E5M2 has more range, up to 57,344. FP8 Tensor Core math runs at twice the FP16 rate.
GDDR: Graphics DRAM soldered around the GPU on the circuit board. GDDR7 on the RTX 5090 reaches 1.79 TB/s, less than HBM but far cheaper per gigabyte.
GEMM: General matrix multiply, the operation C = αAB + βC. Fully connected layers, attention and, through transformations, convolutions all reduce to GEMMs, which is why they dominate AI compute.
GGUF: The model file format used by llama.cpp and tools built on it, such as Ollama and LM Studio. It stores quantized weights, for example Q4_K_M, with the metadata needed to run them.
GPU: Graphics processing unit: a processor with thousands of parallel lanes. Its speed on matrix math made it the main engine of deep learning.
Grace: NVIDIA's Arm-based data-center CPU. In GB200 and GB300 superchips it connects to Blackwell GPUs over NVLink-C2C at 900 GB/s.
HBM (High Bandwidth Memory): Stacked DRAM placed next to the GPU die on a silicon interposer. HBM3 on H100 delivers 3.35 TB/s; HBM3e on B200 delivers about 8 TB/s; HBM4 arrives with Rubin.
Hopper: NVIDIA's 2022 architecture behind H100 and H200. It introduced FP8 Tensor Cores, the Transformer Engine and the Tensor Memory Accelerator.
Inference: Running a trained model to produce outputs, such as answering a prompt. For language models it splits into prefill and decode.
InfiniBand: A high-speed, low-latency network used to connect GPU servers into training clusters. Ethernet with RDMA (RoCE) is the main alternative.
INT8: An 8-bit integer format used for quantized inference. Tensor Cores support it from the Turing generation.
Interposer: A silicon layer under the GPU and HBM stacks that carries thousands of fine wires between them, which makes the very wide HBM interface possible.
KV cache: The stored keys and values of every previous token in a transformer's attention layers. It lets a model generate new tokens without recomputing the past, and it often uses more GPU memory than the weights in long conversations.
L2 cache: The large on-chip cache shared by all SMs. It is 40 MB on A100 and 50 MB on H100, and it cuts trips to HBM for data that is reused.
Large language model (LLM): A transformer network trained on large amounts of text to predict the next token. Running one is dominated by matrix multiplies on Tensor Cores.
Matrix Core: AMD's equivalent of a Tensor Core, found in Instinct data-center GPUs.
Microscaling (MX formats): An Open Compute Project standard in which a block of 32 low-precision values shares one 8-bit power-of-two scale. MXFP8, MXFP6 and MXFP4 let very small formats cover a wide range.
MIG (Multi-Instance GPU): A feature from A100 onward that splits one GPU into up to seven isolated instances, each with its own memory and compute.
Mixed precision: Doing most math in a low-precision format while keeping sensitive values such as accumulators, master weights or loss scaling in higher precision. It is how Tensor Cores deliver speed without wrecking accuracy.
Mixture-of-experts (MoE): A model design with many expert sub-networks, of which only a few run for each token. Memory needs follow the total parameter count, while compute per token follows the active count.
MMA (matrix multiply-accumulate): The operation D = A × B + C on small matrix tiles. It is the single instruction that a Tensor Core is built to execute.
Model parameters: The learned weights of a model. Weight memory is the parameter count times the bytes per parameter, so a 70B model needs about 140 GB at 16 bits.
NPU: Neural processing unit: an AI accelerator built into phone, laptop and PC processors for efficient on-device inference. NPUs are rated in TOPS, usually at INT8.
NVFP4: NVIDIA's 4-bit format for Blackwell: FP4 E2M1 values in blocks of 16 that share an FP8 E4M3 scale, plus a per-tensor FP32 scale. The finer blocks preserve more accuracy than plain MXFP4.
NVL72: The NVIDIA rack design in which 72 Blackwell or Rubin GPUs share one NVLink domain, so they can work together like one very large GPU.
NVLink: NVIDIA's high-speed GPU-to-GPU interconnect. It lets several GPUs share work and memory traffic far faster than PCIe.
NVSwitch: The switch chip that connects many GPUs over NVLink so that every GPU can talk to every other at full speed.
PCIe: The standard expansion bus between CPU and GPU. PCIe 5.0 x16 moves about 64 GB/s each way, far less than NVLink.
Pipeline parallelism: Splitting a model by layers across GPUs, with micro-batches flowing through them like an assembly line.
Prefill: The first phase of LLM inference, in which the whole prompt is processed at once. It is large matrix math that keeps Tensor Cores busy, so it is usually compute-bound.
Quantization: Converting a model's weights or activations to fewer bits, for example from BF16 to FP8, INT8 or FP4, to cut memory use and raise throughput.
Register file: The fastest storage on the GPU, private to each thread. Each SM in recent data-center GPUs has 256 KB of registers, and matrix fragments for Tensor Core operations live there.
ROCm: AMD's open software platform for GPU computing, its counterpart to CUDA. PyTorch and vLLM run on Instinct GPUs through it.
Roofline model: A chart that shows the highest throughput a kernel can reach: the lower of peak compute and arithmetic intensity × memory bandwidth.
Rubin: NVIDIA's 2026 architecture that follows Blackwell. Each Rubin GPU has 288 GB of HBM4 and is paired with the Vera CPU in Vera Rubin NVL72 racks.
Scale-up and scale-out: Scale-up links a few dozen accelerators with a very fast fabric such as NVLink. Scale-out connects many such groups over a network such as InfiniBand or Ethernet.
Shared memory: Fast on-chip memory that threads in a block share. Kernels stage matrix tiles here before feeding them to Tensor Cores.
Sparsity (2:4 structured): A pattern in which two of every four consecutive values are zero. Sparse Tensor Cores, from Ampere onward, skip the zeros and double peak math throughput.
Speculative decoding: Speeding up generation by letting a small draft model propose several tokens that the large model then checks in a single pass.
Streaming multiprocessor (SM): The repeated building block of an NVIDIA GPU. Each SM holds CUDA cores, Tensor Cores, registers, shared memory and schedulers. H100 SXM has 132 SMs.
Systolic array: A grid of multiply-accumulate cells that pass data to their neighbours in rhythm. Google TPUs use systolic arrays as their matrix units.
TDP (thermal design power): The sustained power a chip is designed to draw and its cooling must remove. It ranges from 72 W for an NVIDIA L4 to about 1,400 W for a GB300 GPU.
Tensor Core: A matrix-math unit inside NVIDIA GPUs that executes D = A × B + C on small tiles in one operation. Introduced with Volta in 2017.
Tensor Memory Accelerator (TMA): A Hopper hardware unit that copies whole tiles between global and shared memory asynchronously, so threads spend less time moving data and more time feeding Tensor Cores.
Tensor parallelism: Splitting each weight matrix across several GPUs so they compute parts of every layer together. It needs fast links such as NVLink.
TF32: TensorFloat-32, a 19-bit Tensor Core format with FP32's 8-bit exponent and FP16's 10-bit mantissa. Introduced with Ampere, it speeds up FP32 code with no changes.
Tile: A small block of a large matrix that is processed at once. Big matrix multiplications are split into thousands of tiles that are spread across Tensor Cores.
Time to first token (TTFT): The delay between sending a prompt and seeing the first output token. It is set mainly by prefill speed.
Token: The unit of text a language model reads and writes, often a word or part of a word. Generation speed is measured in tokens per second.
Tokens per second: The rate at which a model generates output. For a single user it is roughly the memory bandwidth divided by the bytes read for each token.
TPU (Tensor Processing Unit): The AI accelerator Google designs for its own data centers and Google Cloud. TPUs use large systolic arrays as their matrix units.
Training: Teaching a model by adjusting its parameters on data. It costs roughly 6 × parameters × training tokens floating-point operations.
Transformer Engine: Hardware plus software in Hopper and Blackwell that picks FP8 or FP4 precision layer by layer and manages scaling factors, to speed up transformer models while keeping accuracy.
UALink: Ultra Accelerator Link, an open scale-up interconnect standard backed by AMD, Intel and others as an alternative to NVLink.
Unified memory: One pool of memory shared by CPU and GPU, as in Apple M-series chips and NVIDIA DGX Spark. It fits very large models on a desktop, with less bandwidth than HBM or GDDR.
Utilization (MFU): Model FLOPs utilization: the share of a GPU's peak that a real training or inference job achieves. Large training runs commonly reach 30% to 50%.
Volta: NVIDIA's 2017 architecture behind the V100, the first GPU with Tensor Cores.
VRAM: Video RAM: the GPU's own memory, either GDDR or HBM. A model must fit in VRAM, with room left for the KV cache, to run at full speed.
Wafer: The round silicon disc, 300 mm across, on which chips are made before being cut into dies.
Warp: A group of 32 threads that an NVIDIA GPU schedules together. Tensor Core instructions are issued per warp, with each thread holding part of the matrix tiles.

Learn more

Put the terms in context with What is a Tensor Core?, FP16, BF16, FP8 and FP4 explained and How a GPU works. Then test yourself in the TensorCore.si Challenge.