Nobody writes backward passes by hand for real models. Frameworks use automatic differentiation (autodiff): as your code runs, each operation records itself and its inputs, building the computation graph on the fly. Calling loss.backward() walks that graph in reverse, applying each operation's local rule. It is exact (not numerical approximation) and not symbolic algebra either; it's the chain rule applied mechanically to the actual sequence of operations executed.
Forward mode and reverse mode
There are two ways to apply the chain rule through a graph. Forward mode pushes derivatives along with values from inputs to outputs; one pass gives the derivative of every output with respect to one input. Reverse mode (backprop) pulls gradients from outputs to inputs; one pass gives the derivative of one output with respect to every input.
Training has one scalar output (the loss) and millions or billions of inputs (the parameters), so reverse mode wins by a factor of the parameter count. Forward mode is useful when inputs are few and outputs many. JAX exposes both (jax.jvp and jax.vjp); PyTorch is built around reverse mode.
A tiny autograd engine
Below is a complete scalar autodiff engine in the style of Karpathy's micrograd. Each Value stores its number, its gradient, its parents, and a closure that applies its local backward rule. backward() sorts the graph topologically and runs the closures in reverse. Run it, then try adding another operation.
That's genuinely the core of PyTorch's autograd, minus tensors, a big library of operations, GPU kernels, and a great deal of engineering for memory and speed. Two practical consequences: gradients accumulate (hence optimizer.zero_grad() before each step), and the graph is rebuilt on every forward pass, so ordinary Python control flow (loops, ifs) just works.
Autodiff records operations as they run and replays them backward. Reverse mode gives the gradient of one loss with respect to all parameters in a single pass, which is exactly what training needs.