How large language models and image generators work: tokens, next-token prediction, scaling, alignment and diffusion.

3 units / 9 lessons / about 90 min of reading / 1 project

6.1

Language models

Text as tokens, language modelling as next-token prediction, and how a model turns probabilities into text.

  1. lesson 1Tokens and byte-pair encoding10 min · Models read text as subword pieces, built by repeatedly merging the most frequent pair.
  2. lesson 2Next-token prediction10 min · A language model is a probability distribution over what comes next. That's the whole training objective.
  3. lesson 3Sampling: temperature, top-k, top-p9 min · Turning probabilities into text, and why the same prompt gives different answers.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalBPE and temperature4 hidden checks
6.2

Large language models

Pretraining at scale, turning a text predictor into an assistant, and what these models can and can't do.

  1. lesson 1Pretraining and scaling laws11 min · Train a transformer to predict text from a large slice of the internet, and loss falls predictably with scale.
  2. lesson 2From predictor to assistant11 min · Instruction tuning, reward models and RLHF: teaching a base model to be helpful and harmless.
  3. lesson 3Capabilities, limits and evaluation10 min · Context windows, prompting, retrieval and tools; hallucination; and why benchmarks need care.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalPreferences and scaling4 hidden checks
6.3

Generating images

Autoencoders and VAEs, adversarial networks, and the diffusion models behind modern image generators.

  1. lesson 1Autoencoders and VAEs11 min · Compress, reconstruct, and make the compressed space smooth enough to sample from.
  2. lesson 2Generative adversarial networks9 min · A forger and a detective train each other until the forgeries are convincing.
  3. lesson 3Diffusion models11 min · Destroy images with noise step by step, train a network to undo one step, then generate from pure noise.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalThe maths of diffusion and VAEs4 hidden checks
module project

A tiny language model

Build a character-level language model of first names: count it, train it as a neural network, sample new names, and give it more context.

Preview the project →
  1. Tokens and a held-out set
  2. Count the bigrams
  3. Probabilities and perplexity
  4. Sample new names
  5. The same model, as a neural network
  6. More context, chosen honestly