Short answer
A diffusion model is a type of generative AI that creates images, video or audio by starting from pure random noise and removing that noise step by step until a clear result appears, guided by your text prompt. It learns this by training on millions of images that were gradually corrupted with noise and learning to undo the damage. Stable Diffusion, FLUX, Midjourney, DALL-E 3 and AI video models such as Sora and Veo are all built on diffusion or its close relative, flow matching.
The idea in plain English
Imagine taking a photo and adding a little static, then a little more, hundreds of times, until it is nothing but grey noise. That is easy. Now imagine learning to run the process backwards: looking at a noisy image and guessing what a slightly cleaner version would look like. Do that enough times and you can start from pure noise and end up with a brand-new, sharp image.
A diffusion model is a neural network trained to do exactly that reverse step.
How diffusion models work
Training: learning to remove noise
1. Take a real image and its caption from the training set. 2. Add a random amount of Gaussian noise to the image. 3. Ask the network to predict the noise that was added (or, in newer methods, the direction back to the clean image), given the noisy image, the noise level and the caption. 4. Compare the prediction with the truth and adjust the network's weights.
Repeated over hundreds of millions of image-caption pairs, the model learns what images look like and how words relate to visual features.
Generation: from noise to picture
1. Start with an image made of pure random noise. 2. The model predicts the noise in it, using your prompt as guidance. 3. A sampler (scheduler) removes part of that noise. 4. Repeat for typically 20 to 50 steps (fewer with distilled "turbo" models). 5. Decode the result into the final image.
Because the starting noise is random, the same prompt gives a different image each time unless you fix the seed.
Key ideas that made diffusion practical
Latent diffusion
Running diffusion on full-resolution pixels is very expensive. Latent diffusion, introduced in 2021 and used by Stable Diffusion, first compresses images with an autoencoder (VAE) into a much smaller latent representation, runs diffusion there, then decodes back to pixels. This cut compute enough to run on consumer GPUs.
Text conditioning
A text encoder (such as CLIP or T5) turns your prompt into embeddings. The denoising network attends to those embeddings at every step, which is how the words steer the picture. The mechanism is the same attention used in language models.
Classifier-free guidance (CFG)
The model makes two predictions each step, one with your prompt and one without, and pushes the result further towards the prompted version. The CFG scale setting controls how strictly the image follows the prompt: higher values follow it more closely but can look oversaturated.
From U-Nets to transformers
Early diffusion models used a convolutional U-Net. Newer models such as Stable Diffusion 3, FLUX and most video models use diffusion transformers (DiT), the same transformer architecture behind LLMs, which scales better with more data and compute.
Flow matching
Many recent models are trained with flow matching or rectified flow, a closely related approach that learns straighter paths from noise to image. This allows good results in fewer steps. FLUX and Stable Diffusion 3 use it.
What diffusion models are used for
- Image generation: Stable Diffusion, FLUX, Midjourney, DALL-E 3, Imagen.
- Image editing: inpainting (replace part of an image), outpainting (extend it), style transfer, upscaling.
- Video: text-to-video and image-to-video models such as Sora, Veo, Kling and the open Wan models.
- Audio and music: speech, sound effects and music generation.
- Science: protein design, molecule generation and weather forecasting.
- Text (experimental): "diffusion language models" generate whole blocks of text in parallel instead of one token at a time.
Diffusion models vs LLMs vs GANs
| Diffusion models | LLMs | GANs | |
|---|---|---|---|
| Generates | Images, video, audio | Text, code | Mostly images |
| How | Removes noise over many steps | Predicts the next token | Generator vs discriminator contest |
| Quality and diversity | Very high | Very high for text | High, but less diverse |
| Training stability | Stable | Stable | Often unstable |
| Speed | Many steps per output | One step per token | One step (fast) |
For the bigger picture, see What is generative AI?.
What hardware diffusion models need
Diffusion is mostly compute-bound: each step runs the whole network over the full latent image, so GPU TFLOPS and tensor cores matter more than they do for chatbots, where speed is limited mainly by memory bandwidth. VRAM still decides which models fit:
| Model | Typical comfortable VRAM |
|---|---|
| Stable Diffusion 1.5 | 6 to 8 GB |
| SDXL | 10 to 12 GB |
| FLUX.1 (FP8 or GGUF) | 12 to 16 GB |
| FLUX.1 (FP16) | 24 GB+ |
| Large video models | 24 to 80 GB |
Our best GPU for Stable Diffusion guide covers which cards to buy, and the GPU cloud price tracker shows what it costs to rent data center GPUs for heavier jobs.
Common settings explained
- Steps: more steps can add detail but take longer; 20 to 30 is typical.
- CFG / guidance scale: how strongly to follow the prompt.
- Sampler: the algorithm that removes noise (Euler, DPM++ and others).
- Seed: the starting noise; reuse it to reproduce an image.
- Negative prompt: what you do not want (supported by some models).
- LoRA: a small add-on that teaches a style or subject, the same idea as LoRA fine-tuning for LLMs.
Frequently asked questions
Why is it called diffusion?
The name comes from physics, where diffusion describes particles spreading out, like ink in water. Adding noise to an image step by step is mathematically similar, and the model learns to reverse it.
Is Midjourney a diffusion model?
Midjourney has not published its architecture in detail, but it is widely understood to be diffusion-based, like most leading image generators.
Why do AI images sometimes have wrong hands or text?
Hands and lettering have precise structures that are easy to get slightly wrong when generating from noise. Larger, newer models with better text encoders have greatly improved both.
Can I run a diffusion model on my own computer?
Yes. Stable Diffusion and FLUX run locally through apps like ComfyUI or Forge on an NVIDIA GPU with 8 GB or more, ideally 16 GB.
Sources and further reading
- Ho et al., Denoising Diffusion Probabilistic Models (2020)
- Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (2021)
- Ho and Salimans, Classifier-Free Diffusion Guidance (2022)
- Peebles and Xie, Scalable Diffusion Models with Transformers (2022)
- Lipman et al., Flow Matching for Generative Modeling (2022)




Leave a Reply