What is Diffusion model?

How it works

Training has two halves:

1. Forward process (adding noise): take a real image and add a little random noise, many times, until it is pure static. This step needs no learning.
2. Reverse process (removing noise): train a neural network to look at a noisy image and predict the noise that was added, so it can be subtracted. The network learns this at every noise level.

To generate, you start from pure random noise and run the trained network repeatedly, removing a bit of noise each step. Ho et al.'s 2020 DDPM paper showed this produces high-quality images.

Two refinements made diffusion practical:

• Latent diffusion (Rombach et al., the basis of Stable Diffusion) runs the process in a compressed "latent" space instead of on full-size pixels, which cuts compute dramatically. A decoder turns the final latent back into an image.
• Text conditioning: the prompt is turned into embeddings by a text encoder, and the denoiser is guided by them at each step, so the noise resolves into "a red fox in the snow" rather than any image.

Newer image and video models often replace the original U-Net denoiser with a transformer (diffusion transformers) and use related techniques such as flow matching, but the core idea of turning noise into data step by step is the same.

Why it matters

Diffusion models power most well-known image and video generators and many audio tools. They give fine control: you can edit part of an image (inpainting), extend it, or steer style. The trade-offs are speed (many denoising steps, though distilled models need few) and the same issues as other generative AI: training-data copyright questions, misuse for deepfakes, and errors such as distorted hands or garbled text in images.

Example

You type "a lighthouse at sunset, watercolour". The model starts with a grid of random noise in latent space, runs perhaps 20 to 50 denoising steps guided by the prompt's embeddings, and decodes the result into a watercolour-style lighthouse image. A different random starting noise gives a different picture from the same prompt.