A generative model learns the distribution of its training data well enough to produce new samples from it: faces that don't exist, molecules nobody has made. Text generation is solved by next-token prediction; images need other ideas. The first is the autoencoder.
Autoencoders
An autoencoder is two networks back to back. The encoder squeezes an input into a small latent vector ; the decoder reconstructs from it. Training minimises reconstruction error. The narrow bottleneck forces to capture the most important factors of variation: a non-linear cousin of PCA.
Can we generate new images by decoding random s? Usually not well. Nothing makes the latent space tidy: the codes of training images may sit in scattered islands, and a random point between them decodes to garbage.
Variational autoencoders
The variational autoencoder (VAE; Kingma & Welling, 2013) makes the latent space smooth by design. The encoder outputs a distribution for each input, a mean and a standard deviation , and the decoder must reconstruct from a sample of it. The loss has two terms:
The KL term pulls every input's distribution toward a standard Gaussian, so codes overlap and fill the space without gaps. Afterwards, sampling and decoding gives plausible new data, and walking in a straight line between two codes morphs one image into another. The loss is a bound on the log-likelihood of the data (the evidence lower bound, ELBO).
The reparameterisation trick
Sampling is random, and you can't backpropagate through "draw a random number". The fix is to move the randomness out of the way: sample noise separately and compute deterministically from it.
VAEs produce somewhat blurry images on their own, but the idea lives on: Stable Diffusion runs its diffusion process inside the latent space of an autoencoder, which is far cheaper than working on pixels.
A VAE encodes each input as a Gaussian, decodes a sample from it, and penalises distance from a standard Gaussian, which makes the latent space smooth enough to sample new data from.