Learning a mapping from labelled examples: regression, classification, neighbours, trees and ensembles.

3 units / 9 lessons / about 85 min of reading / 1 project

2.1

Linear regression

Predicting a number with a weighted sum: the model, its loss, two ways to fit it, and how features change what it can learn.

  1. lesson 1The linear model9 min · Predict a number as a weighted sum of features, and measure how wrong it is.
  2. lesson 2Fitting: gradients and the normal equation10 min · Two routes to the best weights: an exact formula, and gradient descent in batches.
  3. lesson 3Features: scaling, polynomials, categories10 min · What you feed a model shapes what it can learn. Scale numbers, expand curves, encode categories.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalLeast squares, two ways3 hidden checks
2.2

Classification

Predicting categories with probabilities: logistic regression, cross-entropy, decision boundaries and softmax.

  1. lesson 1Logistic regression9 min · Squash a weighted sum into a probability with the sigmoid.
  2. lesson 2Cross-entropy loss9 min · The right loss for probabilities, and why its gradient is so simple.
  3. lesson 3Decision boundaries and many classes10 min · Where a classifier switches its answer, and softmax for more than two classes.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalNumerically stable classification maths5 hidden checks
2.3

Neighbours, trees and ensembles

Non-linear models without gradients: nearest neighbours, decision trees, and combining many models into one.

  1. lesson 1k-nearest neighbours8 min · Predict like your neighbours do. No training, but distance is everything.
  2. lesson 2Decision trees10 min · Twenty questions with data: split on the feature that best separates the classes.
  3. lesson 3Ensembles: forests and boosting10 min · Many weak or unstable models combined are stronger than one. Bagging, random forests and gradient boosting.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalGrow the first split of a tree4 hidden checks
module project

A flower classifier from scratch

Classify all three iris species two ways, k-nearest neighbours and softmax regression, and compare them honestly on held-out flowers.

Preview the project →
  1. Shuffle and split
  2. Scale with training statistics only
  3. k-nearest neighbours
  4. Softmax regression by gradient descent
  5. Where do they go wrong?