Regularising networks

Neural networks often have more parameters than training examples, so module 3's lessons apply with force. Besides more data and data augmentation, four tools are standard.

Weight decay

The L2 penalty from module 3. With plain SGD it's equivalent to shrinking every weight by a factor slightly below 1 at each step, hence the name. With adaptive optimisers the two differ, which is why AdamW applies the decay directly to the weights.

Dropout

During training, dropout (Srivastava, Hinton et al., 2014) sets each hidden unit's output to zero with probability $p$ (often 0.1 to 0.5), with a fresh random mask on every step. No unit can rely on any particular other unit being present, so the network learns redundant, robust features. It can be seen as training a huge ensemble of thinned networks that share weights.

At test time nothing is dropped, so activations would be larger on average than during training. Inverted dropout, the standard implementation, fixes this during training instead: it divides the surviving activations by $1-p$, so the expected value is unchanged and the test-time network needs no adjustment.

Normalisation layers

Batch normalisation (Ioffe & Szegedy, 2015) standardises each feature across the mini-batch inside the network, then rescales it with two learned parameters. It allows higher learning rates, makes training much less sensitive to initialisation, and has a mild regularising effect from batch noise. At test time it uses running averages of the batch statistics. Layer normalisation (Ba et al., 2016) standardises across the features of each example instead, so it doesn't depend on the batch; it's the standard in transformers.

Early stopping

Track validation loss during training. It usually falls, flattens, and eventually rises as the network starts memorising. Keep the weights from the epoch with the best validation loss. It's free, effective, and nearly universal.

Weight decay keeps weights small, dropout prevents co-dependence, normalisation stabilises training, and early stopping picks the moment before memorisation.