What Is CUDA? NVIDIA's GPU Computing Platform Explained

What CUDA includes

How a CUDA program works

A CUDA program runs on the CPU and hands heavy work to the GPU as kernels. When a kernel launches, the GPU runs it on a grid of thread blocks. Each block runs on one streaming multiprocessor (SM), where its threads are scheduled in groups of 32 called warps. This simple kernel adds two vectors, with each thread handling one element:

__global__ void add(const float* a, const float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) c[i] = a[i] + b[i];
}
// Launch enough blocks of 256 threads to cover n elements
add<<<(n + 255) / 256, 256>>>(a, b, c, n);

An RTX 5090 has 170 SMs and a B200 has 148, so a single launch can keep tens of thousands of threads in flight at once. Matrix-heavy kernels use special warp-level instructions that feed the Tensor Cores, but most developers never write those by hand: they call cuBLAS, cuDNN or a framework that does.

CUDA the platform vs CUDA cores

NVIDIA uses the same name for two things. CUDA is the software platform. CUDA cores are the general-purpose arithmetic lanes inside each SM, which do one multiply-add per clock on single numbers. Tensor Cores do whole small matrix multiplies at once. CUDA software drives both. Read Tensor Cores vs CUDA Cores for the hardware side.

Why AI depends on CUDA

When you train or run a model in PyTorch on an NVIDIA GPU, almost every operation becomes a call to a CUDA library: matrix multiplies to cuBLAS or CUTLASS, attention to cuDNN or FlashAttention, gradient exchange to NCCL. NVIDIA tunes these libraries for each new GPU before it ships, so new hardware is fast from day one. Nearly two decades of code, tutorials, research and tools built on CUDA make it hard to replace. That ecosystem, more than any single chip, is often called NVIDIA's moat.

Compute capability by GPU

Each NVIDIA GPU generation has a compute capability version that tells CUDA which features it supports, such as FP8 or FP4 Tensor Cores.

Alternatives to CUDA

• AMD ROCm and HIP: AMD's open platform. HIP closely mirrors the CUDA API, and the HIPIFY tools convert much CUDA code automatically. See AMD vs NVIDIA for AI.
• Intel oneAPI and SYCL: an open, multi-vendor programming model for Intel GPUs and other hardware.
• Apple Metal and MLX: for the GPUs in Apple silicon Macs.
• Triton: a Python language for writing GPU kernels that compiles for both NVIDIA and AMD GPUs.
• Vulkan compute: used by tools such as llama.cpp to run models on almost any GPU, usually more slowly than with CUDA.

Do you need to learn CUDA?

To use AI tools, no. Ollama, LM Studio, ComfyUI and PyTorch install the CUDA pieces they need; you only need a recent NVIDIA driver. Learning CUDA pays off if you write custom kernels, optimize inference engines or work on GPU performance, where understanding warps, memory and Tensor Cores makes the difference between 20% and 70% of peak.

Frequently asked questions

Is CUDA free?

Yes. The CUDA Toolkit is a free download from NVIDIA for Windows and Linux. It is proprietary, though, and runs only on NVIDIA GPUs.

Does CUDA work on AMD or Intel GPUs?

Not officially. AMD GPUs use ROCm, where code written with HIP looks very similar to CUDA. Some third-party projects try to run CUDA programs on other GPUs, but they are not supported by NVIDIA.

Which CUDA version do I need?

Usually whichever your framework was built with. PyTorch packages include their own CUDA runtime, so you mainly need an NVIDIA driver new enough to support it. Newer GPUs such as the RTX 50 series need recent CUDA versions because older ones do not know their compute capability.

What does CUDA stand for?

Compute Unified Device Architecture, although NVIDIA now simply uses the name CUDA.

Sources and further reading

• NVIDIA CUDA Toolkit
• CUDA C++ Programming Guide
• NVIDIA: CUDA GPUs and compute capability
• TensorCore.si glossary