A model is only useful on data it hasn't seen: tomorrow's emails, next month's patients, the photo you take after it was trained. Performance on the training data is not that. A lookup table that memorised every training example would score 100% and be useless. What we care about is generalisation: performance on new examples from the same source.
We can't measure the future directly, so we simulate it. Before doing anything else, set aside part of the data and pretend it arrived later. The standard protocol uses three parts:
| split | typical share | used for |
|---|---|---|
| training | 60–80% | fitting the parameters (weights) |
| validation | 10–20% | choosing hyperparameters and models: the degree, , , when to stop |
| test | 10–20% | one final, unbiased estimate of performance, after every decision is made |
Why not choose hyperparameters on the test set? Because every choice you make by looking at a set leaks a little of it into the model. Try 50 settings and keep the best test score, and that score is optimistically biased: you have partly fitted the test set by selection. The validation set absorbs that bias so the test set doesn't have to. Once you've looked at the test score, any further tuning turns it into another validation set.
Training data fits the weights, validation data makes the choices, test data is opened once at the end to report the result.
The assumption underneath
All of this assumes the examples are i.i.d.: independent and identically distributed. Each example is drawn from the same distribution, independently of the others. A random split then gives a test set that looks like the future. When that assumption breaks, random splits lie:
- Time series (prices, sales, sensor readings): tomorrow depends on today. A random split puts future days in the training set, letting the model peek at the future. Split chronologically: train on the past, test on what came after.
- Groups: several X-rays from the same patient, several photos of the same person. If one patient's images land in both training and test, the model can recognise the patient instead of the disease. Split by group.
- Duplicates: near-identical examples on both sides inflate the test score. Deduplicate before splitting.
The generalisation gap is the difference between training and test performance. A small gap with poor performance means the model is too simple. A large gap means it learned things about the training set that aren't true in general. The next lesson names these two failures.