Training a model means adjusting numbers (its parameters) to make an error measure (its loss) smaller. To do that we need to know, for each parameter: if I nudge this up a little, does the loss go up or down, and how fast? That question has a precise answer, the derivative.
From average slope to instantaneous slope
The slope between two points on a curve is rise over run. Bring the two points closer and closer together and the slope settles to a single number: the slope of the tangent line at that point.
Read the figure like a loss curve. Where the derivative is positive, moving right increases , so to go down you move left. Where it is negative, you move right. Where it is zero, you are at a flat point: a minimum, a maximum, or a saddle. To decrease a function, step against the sign of its derivative. That sentence is gradient descent.
Rules you will actually use
| function | derivative | note |
|---|---|---|
| (a constant) | flat | |
| power rule: | ||
| its own slope | ||
| for | ||
| derivatives are linear | ||
| the chain rule, lesson 3 |
When you can't do the algebra: numerical derivatives
You can always estimate a derivative by evaluating the function twice. The central difference is much more accurate than the one-sided version in the definition: its error shrinks like rather than .
Numerical derivatives are too slow to train big models (one or two function evaluations per parameter, and modern models have billions), but they are the standard way to check that hand-written gradient code is correct. You will do exactly that when you write backpropagation in module 4.
The derivative is the local slope. Its sign says which way is uphill; its size says how steep. Learning is repeatedly stepping downhill.