Skip to content
,

What Is a Diffusion Model? How AI Image and Video Generators Work

Diffusion models create images, video and audio by starting from random noise and removing it step by step. A plain-English explanation of how Stable Diffusion, FLUX, Midjourney and Sora-style models work, latent diffusion, guidance, samplers, transformers and the GPUs they need.

5 min read

What Is a Diffusion Model? — cover image from TensorCore.si

Short answer

A diffusion model is a type of generative AI that creates images, video or audio by starting from pure random noise and removing that noise step by step until a clear result appears, guided by your text prompt. It learns this by training on millions of images that were gradually corrupted with noise and learning to undo the damage. Stable Diffusion, FLUX, Midjourney, DALL-E 3 and AI video models such as Sora and Veo are all built on diffusion or its close relative, flow matching.

The idea in plain English

Imagine taking a photo and adding a little static, then a little more, hundreds of times, until it is nothing but grey noise. That is easy. Now imagine learning to run the process backwards: looking at a noisy image and guessing what a slightly cleaner version would look like. Do that enough times and you can start from pure noise and end up with a brand-new, sharp image.

A diffusion model is a neural network trained to do exactly that reverse step.

How diffusion models work

Training: learning to remove noise

1. Take a real image and its caption from the training set. 2. Add a random amount of Gaussian noise to the image. 3. Ask the network to predict the noise that was added (or, in newer methods, the direction back to the clean image), given the noisy image, the noise level and the caption. 4. Compare the prediction with the truth and adjust the network's weights.

Repeated over hundreds of millions of image-caption pairs, the model learns what images look like and how words relate to visual features.

Generation: from noise to picture

1. Start with an image made of pure random noise. 2. The model predicts the noise in it, using your prompt as guidance. 3. A sampler (scheduler) removes part of that noise. 4. Repeat for typically 20 to 50 steps (fewer with distilled "turbo" models). 5. Decode the result into the final image.

Because the starting noise is random, the same prompt gives a different image each time unless you fix the seed.

Key ideas that made diffusion practical

Latent diffusion

Running diffusion on full-resolution pixels is very expensive. Latent diffusion, introduced in 2021 and used by Stable Diffusion, first compresses images with an autoencoder (VAE) into a much smaller latent representation, runs diffusion there, then decodes back to pixels. This cut compute enough to run on consumer GPUs.

Text conditioning

A text encoder (such as CLIP or T5) turns your prompt into embeddings. The denoising network attends to those embeddings at every step, which is how the words steer the picture. The mechanism is the same attention used in language models.

Classifier-free guidance (CFG)

The model makes two predictions each step, one with your prompt and one without, and pushes the result further towards the prompted version. The CFG scale setting controls how strictly the image follows the prompt: higher values follow it more closely but can look oversaturated.

From U-Nets to transformers

Early diffusion models used a convolutional U-Net. Newer models such as Stable Diffusion 3, FLUX and most video models use diffusion transformers (DiT), the same transformer architecture behind LLMs, which scales better with more data and compute.

Flow matching

Many recent models are trained with flow matching or rectified flow, a closely related approach that learns straighter paths from noise to image. This allows good results in fewer steps. FLUX and Stable Diffusion 3 use it.

What diffusion models are used for

  • Image generation: Stable Diffusion, FLUX, Midjourney, DALL-E 3, Imagen.
  • Image editing: inpainting (replace part of an image), outpainting (extend it), style transfer, upscaling.
  • Video: text-to-video and image-to-video models such as Sora, Veo, Kling and the open Wan models.
  • Audio and music: speech, sound effects and music generation.
  • Science: protein design, molecule generation and weather forecasting.
  • Text (experimental): "diffusion language models" generate whole blocks of text in parallel instead of one token at a time.

Diffusion models vs LLMs vs GANs

Diffusion modelsLLMsGANs
GeneratesImages, video, audioText, codeMostly images
HowRemoves noise over many stepsPredicts the next tokenGenerator vs discriminator contest
Quality and diversityVery highVery high for textHigh, but less diverse
Training stabilityStableStableOften unstable
SpeedMany steps per outputOne step per tokenOne step (fast)

For the bigger picture, see What is generative AI?.

What hardware diffusion models need

Diffusion is mostly compute-bound: each step runs the whole network over the full latent image, so GPU TFLOPS and tensor cores matter more than they do for chatbots, where speed is limited mainly by memory bandwidth. VRAM still decides which models fit:

ModelTypical comfortable VRAM
Stable Diffusion 1.56 to 8 GB
SDXL10 to 12 GB
FLUX.1 (FP8 or GGUF)12 to 16 GB
FLUX.1 (FP16)24 GB+
Large video models24 to 80 GB

Our best GPU for Stable Diffusion guide covers which cards to buy, and the GPU cloud price tracker shows what it costs to rent data center GPUs for heavier jobs.

Common settings explained

  • Steps: more steps can add detail but take longer; 20 to 30 is typical.
  • CFG / guidance scale: how strongly to follow the prompt.
  • Sampler: the algorithm that removes noise (Euler, DPM++ and others).
  • Seed: the starting noise; reuse it to reproduce an image.
  • Negative prompt: what you do not want (supported by some models).
  • LoRA: a small add-on that teaches a style or subject, the same idea as LoRA fine-tuning for LLMs.

Frequently asked questions

Why is it called diffusion?

The name comes from physics, where diffusion describes particles spreading out, like ink in water. Adding noise to an image step by step is mathematically similar, and the model learns to reverse it.

Is Midjourney a diffusion model?

Midjourney has not published its architecture in detail, but it is widely understood to be diffusion-based, like most leading image generators.

Why do AI images sometimes have wrong hands or text?

Hands and lettering have precise structures that are easy to get slightly wrong when generating from noise. Larger, newer models with better text encoders have greatly improved both.

Can I run a diffusion model on my own computer?

Yes. Stable Diffusion and FLUX run locally through apps like ComfyUI or Forge on an NVIDIA GPU with 8 GB or more, ideally 16 GB.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *