Regularisation

Overfitting and regularisationabout 10 minutes

Overfitted models have a tell: enormous weights. To make a degree-11 polynomial pass through 12 noisy points, coefficients become huge and cancel each other out. Regularisation adds a penalty on weight size to the loss, so the model only uses big weights when the data truly demands them.

Ridge (L2)

L (w) = \frac{1}{n} ∥ X w - y ∥^{2} + λ ∥ w ∥_{2}^{2}

The strength $λ$ is a hyperparameter, chosen on validation data. Ridge regression still has a closed form, and it fixes the invertibility problem from module 2: adding $λ I$ makes the matrix invertible for any $λ > 0$ .

w^{*} = (X^{⊤} X + nλ I)^{- 1} X^{⊤} y

The same degree-11 polynomial with a ridge penalty. Turn λ up from nothing: the wild curve calms down, then flattens too much.

Larger $λ$ means smaller weights, smoother functions, more bias and less variance. Too large and the model underfits.
The bias (intercept) is normally not penalised: shifting all predictions up shouldn't cost anything.
Features must be standardised first, or the penalty hits features with small units harder than those with large units.
In neural networks the same idea is called weight decay: each gradient step also shrinks every weight slightly toward zero.

Lasso (L1) and sparsity

Penalise absolute values instead, $λ \sum_{j} ∣ w_{j} ∣$ , and something qualitatively different happens: many weights become exactly zero. The lasso (Tibshirani, 1996) performs feature selection as part of fitting. Geometrically, the L1 constraint region is a diamond with corners on the axes, and the loss contours tend to touch it first at a corner, where some coordinates are zero. The round L2 ball has no corners, so ridge shrinks weights but rarely zeroes them.

Use lasso when you believe only a few features matter, or want an interpretable model. Use ridge when many features each contribute a little. Elastic net mixes both.

Other regularisers

Early stopping: train by gradient descent and stop when validation error stops improving. Weights start small and grow during training, so stopping early keeps them small; it behaves much like L2.
Data augmentation: create plausible new training examples (flipped or cropped images, paraphrased text). It encodes knowledge about which changes shouldn't affect the label.
Dropout, batch norm, and noise for neural networks, covered in module 4.
More data remains the best regulariser of all.

try itedit the code and run it, nothing can break

python sleeps until you press run

Regularisation trades a little bias for a lot less variance by penalising large weights. L2 shrinks them, L1 zeroes some out, and λ is tuned on validation data.

← Underfitting, overfitting, bias and variance