Convolutions for images, recurrence for sequences, and attention, the idea behind transformers.

3 units / 9 lessons / about 90 min of reading / 1 project

5.1

Convolutional networks

Exploit the structure of images: local patterns, shared weights, and a hierarchy of features.

  1. lesson 1Convolution10 min · Slide a small filter over an image; the same weights look for the same pattern everywhere.
  2. lesson 2Pooling and feature hierarchies9 min · Shrink the maps, grow the receptive field, and build edges into textures into objects.
  3. lesson 3From LeNet to ResNet10 min · The architectures that made deep learning, and the residual connection that made it deep.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalConvolution and pooling3 hidden checks
5.2

Sequence models

Text, audio and time series: embeddings, recurrent networks, and why they struggled with long contexts.

  1. lesson 1From one-hot to embeddings10 min · Represent each word as a learned vector, so that similar words get similar vectors.
  2. lesson 2Recurrent neural networks10 min · Carry a hidden state through time, and see why gradients fade over long sequences.
  3. lesson 3LSTMs, GRUs and the bottleneck10 min · Gates that decide what to remember, and the encoder-decoder bottleneck that led to attention.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalAn RNN forward pass4 hidden checks
5.3

Attention and transformers

Let every token look at every other: attention, self-attention, and the transformer block.

  1. lesson 1Attention10 min · A soft, differentiable lookup: queries, keys and values.
  2. lesson 2Self-attention and multi-head attention11 min · Every token queries every other token in the same sequence, in several ways at once.
  3. lesson 3The transformer block11 min · Attention plus an MLP, wrapped in residuals and layer norm, stacked many times.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalScaled dot-product attention4 hidden checks
module project

Train a convolutional network from scratch

Turn convolution into a matrix multiply, then train a small CNN on handwritten digits and look at the filters it learns.

Preview the project →
  1. Images and patches
  2. Convolution as one matrix multiply
  3. Forward and backward through the CNN
  4. Train it
  5. What did it learn?