The chain rule

Derivatives and gradientsabout 9 minutes

Models are functions of functions. A prediction is computed from the weights, a loss is computed from the prediction, and in a neural network there are dozens of layers in between. To get the derivative of the loss with respect to a weight deep inside, we need the chain rule.

\frac{d y}{d x} = \frac{d y}{d u} \cdot \frac{d u}{d x} for y = f (u), u = g (x)

Read it as a chain of exchange rates. If $u$ changes 3 times as fast as $x$ , and $y$ changes 2 times as fast as $u$ , then $y$ changes $2 \times 3 = 6$ times as fast as $x$ . Example: $y = (2 x + 1)^{3}$ . Let $u = 2 x + 1$ , so $y = u^{3}$ . Then $d y / d u = 3 u^{2}$ and $d u / d x = 2$ , so $d y / d x = 6 (2 x + 1)^{2}$ .

A computation graph

Take the squared error of a one-weight model: prediction $p = w x + b$ , error $e = p - y$ , loss $L = e^{2}$ . Draw each operation as a node. The forward pass computes values left to right. The backward pass starts from $\partial L / \partial L = 1$ and moves right to left, multiplying by each node's local derivative.

Forward values (white) flow right. Gradients (blue) flow back, each one the gradient arriving from the right times a local derivative.

\frac{\partial L}{\partial w} = 2 e \frac{\partial L}{\partial e} \cdot 1 \frac{\partial e}{\partial p} \cdot x \frac{\partial p}{\partial w} = 2 (w x + b - y) x

Notice that $\partial L / \partial b = 2 e \cdot 1 \cdot 1$ reuses the first two factors. Backward passes share work: the gradient arriving at a node is computed once and passed to everything that feeds into it. That reuse is why computing the gradient for all parameters costs only about as much as one or two forward passes, no matter how many parameters there are.

When a value is used twice

If one variable feeds into several places, its gradient is the sum of the gradients from each path. This is the multivariable chain rule: for $f (x) = x \cdot x$ , each of the two uses contributes $x$ , giving $2 x$ . Forgetting to accumulate (sum) gradients is one of the most common bugs in hand-written backprop.

\frac{\partial L}{\partial x} = j \sum \frac{\partial L}{\partial u _{j}} \frac{\partial u _{j}}{\partial x}

try itedit the code and run it, nothing can break

python sleeps until you press run

Backpropagation is nothing but the chain rule applied to a computation graph, right to left, reusing work. Modern frameworks (PyTorch, JAX) record the graph as you compute and run this backward pass for you.

← Gradients and gradient descent