Initialisation and vanishing gradients

Where gradient descent starts matters more for deep networks than for anything you've trained so far. Two things can go wrong at initialisation.

Symmetry

Set every weight to zero (or to the same constant) and every hidden unit in a layer computes the same thing, receives the same gradient, and updates identically. They stay clones forever, and a layer of 512 units behaves like one. Weights must start random to break the symmetry. Biases can start at zero.

Scale

Random, but how big? Each unit sums $n_{\text{in}}$ inputs times weights. If the weights have variance ${\sigma }^2$ and inputs have variance $v$, the sum has variance about $n_{\text{in}}{\sigma }^{2}v$. If $n_{\text{in}}{\sigma }^2>1$, activations grow layer after layer and explode; if it's $<1$, they shrink toward zero. The same multiplication happens to gradients flowing backward. Over 50 layers, a factor of 1.1 per layer becomes 117, and 0.9 becomes 0.005.

• Xavier / Glorot initialisation (2010): variance $2/(n_{\text{in}}+n_{\text{out}})$, derived for tanh and sigmoid-like activations.
• He (Kaiming) initialisation (He et al., 2015): variance $2/n_{\text{in}}$ for ReLU. The 2 compensates for ReLU zeroing half its inputs. This is PyTorch's default family for ReLU networks.

Vanishing and exploding gradients

Even with good initialisation, gradients through many layers are products of many factors: weight matrices and activation derivatives. Saturating activations like sigmoid shrink each factor, and this vanishing gradient problem (analysed by Hochreiter in 1991 and Bengio et al. in 1994) is why deep networks were considered nearly untrainable before about 2010. The cures, which you'll meet throughout module 5, are:

• ReLU-family activations, whose derivative is 1 where active.
• Careful initialisation, as above.
• Normalisation layers (batch norm, layer norm) that re-standardise activations inside the network.
• Residual connections $\mathbf{x}+F(\mathbf{x})$, which give gradients a direct path through every layer. They're what made 100+ layer networks practical.
• Gradient clipping for explosions: rescale the gradient when its norm exceeds a limit.

Initialise randomly to break symmetry, at a scale like He's $\sqrt{2/n_{\text{in}}}$ for ReLU, so signals neither vanish nor explode as they pass through depth.