Go back to fitting polynomials to a dozen noisy points. A straight line misses the curve no matter how much data you give it: it underfits. A degree-11 polynomial threads through every point and swings wildly in between: it overfits. Somewhere in between, the model captures the real pattern and ignores the noise.
Bias and variance
Imagine retraining the same kind of model on many different training sets drawn from the same source, and look at its prediction at one input . Two things can go wrong:
- Bias: the average prediction is off. The model family can't represent the truth (a line for a curve), so every retraining makes the same systematic mistake.
- Variance: the predictions scatter from one training set to the next. The model is flexible enough to chase the particular noise in each sample.
For squared error, the expected error at a point splits exactly into three parts:
The noise is the floor no model can beat: the label itself is random. Making a model more flexible typically lowers bias and raises variance. The best test error sits where their sum is smallest.
Diagnosing which one you have
| symptom | diagnosis | what helps |
|---|---|---|
| high training error, test error about the same | high bias (underfitting) | a more flexible model, better features, train longer |
| low training error, much higher test error | high variance (overfitting) | more data, regularisation, a simpler model, averaging (ensembles) |
More data is the most reliable cure for variance: it's harder to fit noise when there's more signal. It doesn't help bias: a line stays a line with a million points. That's why plotting a learning curve (training and validation error against training-set size) is so informative: if the two curves have converged at a high error, collect better features, not more rows.
Modern deep networks complicate the classic U-shape. Very large models trained to zero training error can generalise better as they grow even further, a phenomenon called double descent (Belkin et al., 2019). The bias-variance intuition is still the right starting point, but don't assume more parameters always means more overfitting.
Error on new data is bias² + variance + noise. Simple models are biased, flexible models are variable; the art is landing between them.