GPU Architecture
Inside the GPU: streaming multiprocessors, warps, CUDA cores, caches, HBM memory and the data paths that feed Tensor Cores.
How a GPU is built, and why memory matters as much as math
A GPU is thousands of small cores grouped into streaming multiprocessors and fed by very fast memory. This section covers how that hardware is organised, how software such as CUDA drives it, and why memory bandwidth often decides how fast AI runs.
Start with how a GPU works, then follow the path from memory and interconnects to how language models actually run on the hardware.
Start here
Tools for this topic
Edited by Sami Ullah Khan. Read our editorial policy.
All GPU Architecture articles
GPU Memory Bandwidth Explained: Why It Decides LLM Speed
What GPU memory bandwidth is, how to calculate it from bus width and memory speed, why it limits how fast large language models generate…
GDDR7 vs HBM3E vs HBM4: GPU Memory Types Explained
GDDR7 and HBM are the two kinds of memory used on GPUs. How each works, their bandwidth and capacity, why AI data center GPUs…
What Is VRAM? Video Memory Explained for Gaming and AI (2026)
VRAM is the memory soldered onto a graphics card. What it does, how it differs from system RAM, GDDR7 vs HBM, how to check…
What Is a GPU Cluster? How AI Training Clusters Work (2026)
A GPU cluster links many GPU servers with a fast network so they can work on one job. How clusters are built, how NVLink…
What Are CUDA Cores? How Many You Need and What They Do (2026)
CUDA cores are the small arithmetic units inside NVIDIA GPUs. What one actually is, how they are grouped into SMs and warps, how core…
FlashAttention Explained: How It Makes Transformers Faster on GPUs
FlashAttention computes exact attention without writing the huge attention matrix to GPU memory, using tiling and an online softmax. How it works, why it…
Prefill vs Decode: How LLM Inference Runs on a GPU
Every LLM response has two phases: prefill reads the whole prompt at once and is limited by compute; decode generates one token at a…
GPU vs CPU for AI: Why GPUs Run Machine Learning
Why AI runs on GPUs instead of CPUs: thousands of parallel cores, Tensor Cores and much faster memory. A side-by-side comparison, real numbers, and…
Tensor Cores vs RT Cores: What Each Does in an RTX GPU
Tensor Cores do matrix math for AI and DLSS; RT Cores trace rays for lighting and reflections; CUDA cores handle everything else. How the…
What Is CUDA? NVIDIA's GPU Computing Platform Explained
CUDA is NVIDIA's platform for running general programs on its GPUs. What it includes, how kernels, threads and warps work, why every AI framework…