Backprop through a network

Backpropagationabout 11 minutes

Scalar graphs are fine for understanding, but networks work with matrices. Happily, a whole dense layer has a compact backward rule. Let $Y = X W + b$ with $X$ of shape $(n, d)$ , $W$ of shape $(d, h)$ . Suppose the layer above has handed us $\overset{ˉ}{Y} = \partial L / \partial Y$ , shape $(n, h)$ . Then:

\overset{ˉ}{W} = X^{⊤} \overset{ˉ}{Y} (d, h), \overset{ˉ}{b} = i \sum \overset{ˉ}{Y}_{i, :} (h), \overset{ˉ}{X} = \overset{ˉ}{Y} W^{⊤} (n, d)

The shapes force the formulas: there's only one way to combine X and Ȳ into a (d, h) matrix.

A reliable way to remember these is shape matching. $\overset{ˉ}{W}$ must have $W$ 's shape $(d, h)$ ; the only things available are $X$ $(n, d)$ and $\overset{ˉ}{Y}$ $(n, h)$ , and the only product with the right shape is $X^{⊤} \overset{ˉ}{Y}$ . The bias was broadcast to every row in the forward pass, so its gradient sums over rows (the "used many times, so sum" rule again).

ReLU backward

For $A = ReLU (Z)$ , the gradient passes through where $Z > 0$ and is blocked elsewhere: $\overset{ˉ}{Z} = \overset{ˉ}{A} ⊙ 1 [Z > 0]$ , where $⊙$ is element-wise multiplication. Keep $Z$ (or the mask) from the forward pass.

Softmax and cross-entropy together

At the top of a classifier, softmax and cross-entropy combine into the cleanest gradient in deep learning. With one-hot labels $Y$ and the loss averaged over the batch:

\frac{\partial L}{\partial Z} = \frac{1}{n} (P - Y)

So a full backward pass through a one-hidden-layer classifier is five lines: start with $(P - Y) / n$ , push it through the output layer's rule to get $\overset{ˉ}{W}_{2}$ , $\overset{ˉ}{b}_{2}$ , $\overset{ˉ}{H}$ , gate by the ReLU mask, then apply the dense rule again for $\overset{ˉ}{W}_{1}$ , $\overset{ˉ}{b}_{1}$ .

Always check your gradients

Hand-written backward passes are bug magnets: a missing transpose, a forgotten sum, a mask from the wrong layer. Gradient checking compares analytic gradients with central differences on a few parameters, using the relative error $∣ g_{a} - g_{n} ∣/ max (∣ g_{a} ∣, ∣ g_{n} ∣)$ . With float64, a correct implementation typically gives around $1 0^{- 7}$ or smaller; around $1 0^{- 2}$ means a bug. Check on a tiny network and a handful of examples, then switch it off: it's far too slow for training.

try itedit the code and run it, nothing can break

python sleeps until you press run

For $Y = X W + b$ : $\overset{ˉ}{W} = X^{⊤} \overset{ˉ}{Y}$ , $\overset{ˉ}{b} =$ row-sum of $\overset{ˉ}{Y}$ , $\overset{ˉ}{X} = \overset{ˉ}{Y} W^{⊤}$ . Shapes tell you the formulas; gradient checks tell you they're right.

← Computational graphs