Overfitting, regularisation, honest evaluation, and learning from data without labels.

3 units / 9 lessons / about 90 min of reading / 1 project

3.1

Overfitting and regularisation

Why a model can ace its training data and fail on new data, and the tools that prevent it.

  1. lesson 1Train, validation, test9 min · The goal is new data. Split what you have so you can measure it honestly.
  2. lesson 2Underfitting, overfitting, bias and variance10 min · Two ways to be wrong, and the trade-off between them.
  3. lesson 3Regularisation10 min · Penalise complexity: ridge, lasso, early stopping and more data.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalRidge regression from scratch5 hidden checks
3.2

Measuring models honestly

Cross-validation, metrics beyond accuracy, and the many ways an evaluation can fool you.

  1. lesson 1Cross-validation9 min · Use every example for both training and validation, and get an error bar for free.
  2. lesson 2Accuracy is not enough11 min · Confusion matrices, precision and recall, thresholds, and the ROC curve.
  3. lesson 3Leakage and other ways to fool yourself10 min · How evaluations go wrong without anyone cheating on purpose.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalMetrics and folds5 hidden checks
3.3

Learning without labels

Finding structure in unlabelled data: clusters, the directions that matter, and the odd ones out.

  1. lesson 1Clustering with k-means9 min · Group points around k centres by alternating two simple steps.
  2. lesson 2Principal component analysis11 min · Find the directions in which data varies most, and keep only those.
  3. lesson 3Densities, anomalies and maps of data9 min · Model what normal looks like to flag what isn't, and read 2-D maps of high-dimensional data with care.
  4. theoryTheory test6 questions, 80% to pass
  5. practicalk-means and PCA from scratch5 hidden checks
module project

An honest diagnosis model

Predict whether a breast tumour is malignant from biopsy measurements, choosing the model with cross-validation and opening the test set exactly once.

Preview the project →
  1. Lock away a test set
  2. A model that scales itself honestly
  3. Choose λ by cross-validation
  4. Open the test set, once
  5. Which measurements matter?