From a single neuron to backpropagation, and the tricks that make deep networks train.

3 units / 9 lessons / about 90 min of reading / 1 project

4.1

Neurons and networks

The artificial neuron, why stacking needs non-linearity, and the multilayer perceptron.

  1. lesson 1The artificial neuron9 min · A weighted sum and a threshold, and the 1960s argument about what it can't do.
  2. lesson 2Why non-linearity matters9 min · Without an activation function, any depth collapses to one linear layer.
  3. lesson 3The multilayer perceptron10 min · Layers as matrix multiplies, the forward pass, and counting parameters.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalA forward pass5 hidden checks
4.2

Backpropagation

How a network computes the gradient of its loss with respect to every weight, efficiently.

  1. lesson 1Computational graphs9 min · Break any computation into simple nodes; each only needs its local derivative.
  2. lesson 2Backprop through a network11 min · The three gradients of a dense layer, the ReLU gate, and softmax with cross-entropy.
  3. lesson 3Automatic differentiation10 min · How PyTorch and JAX compute gradients for code you write, in about forty lines.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalBackward through a layer4 hidden checks
4.3

Training deep networks

Initialisation, optimisers and regularisers: the practical knowledge that makes deep networks learn.

  1. lesson 1Initialisation and vanishing gradients10 min · Start the weights at the right scale, or the signal dies (or explodes) layer by layer.
  2. lesson 2Optimisers: momentum and Adam10 min · Smarter steps than plain gradient descent: velocity, per-parameter scaling, and schedules.
  3. lesson 3Regularising networks10 min · Weight decay, dropout, normalisation and early stopping.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalAdam and He initialisation5 hidden checks
module project

A digit recogniser from scratch

Build and train a neural network on handwritten digits with nothing but numpy: forward pass, backpropagation, gradient check, mini-batch training.

Preview the project →
  1. Look at the data
  2. Initialise and run forward
  3. Backpropagation, checked
  4. Train with mini-batches
  5. Study the mistakes