Why non-linearity matters

Neurons and networksabout 9 minutes

Stack two linear layers: $h = W_{1} x$ , then $y = W_{2} h$ . Multiply it out and $y = (W_{2} W_{1}) x$ , which is a single linear layer with matrix $W_{2} W_{1}$ . A hundred linear layers are still one linear layer. Depth only buys anything if something non-linear happens between layers. That something is the activation function.

W_{2} (W_{1} x) = (W_{2} W_{1}) x but W_{2} ϕ (W_{1} x) \neq = W x for any W

The common activations

Each activation (white) and its derivative (blue). The derivative is what backpropagation multiplies by, so flat regions mean vanishing gradients.

activation	formula	notes
sigmoid	$1/ (1 + e^{- z})$	output in (0, 1); saturates at both ends; derivative at most 0.25. Now mostly used for output probabilities and gates.
tanh	$tanh z$	output in (−1, 1), zero-centred; still saturates.
ReLU	$max (0, z)$	the default since AlexNet (2012): cheap, gradient exactly 1 for $z > 0$ . Units stuck at $z < 0$ can "die".
leaky ReLU	$max (α z, z)$	small slope (e.g. 0.01) for $z < 0$ so units can't die.
GELU	$z Φ (z)$	a smooth ReLU, used in BERT, GPT and most transformers.

ReLU's success has a simple explanation: for positive inputs it passes gradients through unchanged, so deep stacks don't shrink them the way sigmoids do (each sigmoid multiplies the gradient by at most 0.25; ten of them, by at most $0.2 5^{10} \approx 1 0^{- 6}$ ). It also makes networks piecewise linear: a ReLU network carves its input space into many linear regions.

Universal approximation

How much can one hidden layer do? The universal approximation theorem (Cybenko 1989 for sigmoids, Hornik 1991 more generally) says a network with one hidden layer and enough units can approximate any continuous function on a bounded region to any accuracy. That is reassuring, and also much weaker than it sounds:

"Enough units" can mean exponentially many. Deeper networks can represent many functions with far fewer units than shallow ones.
It says such weights exist, not that gradient descent will find them.
It says nothing about generalising from finite data.

try itedit the code and run it, nothing can break

python sleeps until you press run

Activations are what make depth worthwhile. ReLU-like functions dominate because they keep gradients alive through many layers.

← The artificial neuron