Module 1 introduced the chain rule on a computation graph. Backpropagation is that idea made systematic, so it works for any network without anyone deriving a formula by hand. The trick is that every node only needs to know two things: how to compute its output from its inputs (forward), and how to turn the gradient of the loss with respect to its output into gradients with respect to its inputs (backward).
Local rules for common nodes
Write for , the gradient of the final loss with respect to some intermediate value (often called its adjoint). For , the backward rule produces and from :
| node | forward | backward |
|---|---|---|
| add | , (copies the gradient) | |
| multiply | , (swaps the inputs) | |
| ReLU | if else 0 (a gate) | |
| exp | ||
| copy / reuse | used by several nodes | sum of the gradients from each use |
The two passes
- Forward: evaluate nodes in topological order (inputs first), storing every intermediate value. The backward rules need them: multiply needs and , ReLU needs to know whether .
- Backward: set and visit nodes in reverse order, applying each node's local rule and adding into its inputs' gradients.
The cost of the backward pass is a small constant multiple of the forward pass, regardless of how many parameters there are. That's the whole magic. Computing each of a million gradients with finite differences would take a million forward passes; backprop gets all of them in roughly two or three passes' worth of work. The price is memory: every intermediate activation is stored until the backward pass uses it, which is why training needs much more memory than inference.
Each node knows its local derivative. The backward pass multiplies them in reverse order and sums wherever a value fans out, giving every gradient for about the cost of one more forward pass.