Diffusion models

Diffusion models power today's image generators (Stable Diffusion, DALL·E, Imagen, Midjourney) and increasingly video and audio. The idea, from Sohl-Dickstein et al. (2015) and made practical by Ho, Jain and Abbeel's DDPM (2020), is beautifully simple: noise is easy to make, so learn to remove it.

The forward process: adding noise

Take a training image ${\mathbf{x}}_0$ and add a little Gaussian noise, then a little more, over $T$ steps (DDPM uses $T=1000$, with noise variances ${\beta }_t$ rising linearly from $10^{-4}$ to $0.02$). By the end, ${\mathbf{x}}_T$ is indistinguishable from pure noise. Conveniently, you can jump straight to any step $t$ in one shot:

The reverse process: learning to denoise

Train a network ${\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)$ to look at a noisy image and the step number, and predict the noise that was added. The DDPM training loss is plain squared error:

1. Pick a training image, a random step $t$, and random noise $\mathbf{ϵ}$.
2. Make ${\mathbf{x}}_t$ with the formula above.
3. Train the network (typically a U-Net, a CNN with skip connections, or a transformer) to predict $\mathbf{ϵ}$ from ${\mathbf{x}}_t$ and $t$.

To generate, start from pure noise ${\mathbf{x}}_T$ and repeatedly subtract a scaled version of the predicted noise (plus a little fresh noise), stepping $t$ down to 0. Each step is a small, learnable correction. Faster samplers (DDIM and others) cut the 1,000 steps to tens.

Text to image

• Conditioning: the denoiser also receives an embedding of a text prompt (from a text encoder such as CLIP or T5), usually through cross-attention, so it learns to denoise toward images matching the text.
• Classifier-free guidance (Ho & Salimans, 2022): run the denoiser with and without the prompt and push the prediction further in the direction the prompt suggests. Higher guidance gives images that follow the prompt more literally, with less variety.
• Latent diffusion (Rombach et al., 2022, the basis of Stable Diffusion): run the whole process in the compact latent space of an autoencoder (lesson 1), then decode, making high-resolution generation affordable.

Diffusion models learn to predict the noise in a noised image. Generation starts from pure noise and removes it step by step, steered by a text embedding and guidance.