Gradients and gradient descent

Derivatives and gradientsabout 10 minutes

Real models have many parameters, so the loss is a function of many variables. A partial derivative $\partial f / \partial x_{i}$ is the slope along one variable while all the others are held fixed. Collect every partial derivative into a vector and you get the gradient:

\nabla f (x) = [\frac{\partial f}{\partial x _{1}} \frac{\partial f}{\partial x _{2}} \dots \frac{\partial f}{\partial x _{n}}]^{⊤}

Example: for $f (x, y) = x^{2} + 3 y^{2}$ , treat $y$ as a constant to get $\partial f / \partial x = 2 x$ , and treat $x$ as a constant to get $\partial f / \partial y = 6 y$ . At the point $(1, 1)$ the gradient is $(2, 6)$ .

The gradient has a beautiful property: it points in the direction of steepest ascent, and its length is how steep that is. It is always perpendicular to the contour lines (lines of equal height). So the negative gradient is the direction of steepest descent.

Gradient descent

Start somewhere, compute the gradient, take a small step against it, repeat. The step size $η$ (eta) is the learning rate:

θ \leftarrow θ - η \nabla L (θ)

Gradient descent on an elongated bowl

L = x^{2} + 8 y^{2}

. Raise the learning rate until the path zig-zags, then until it flies off.

Too small a learning rate and progress is painfully slow.
Too large and each step overshoots the valley floor. The path zig-zags across the narrow direction, and past a threshold it diverges to infinity.
Elongated bowls are the hard case. The steep direction limits how big a step you can take, while the shallow direction needs many steps. This is why feature scaling (module 2) and adaptive optimisers like Adam (module 4) exist.

For a quadratic bowl you can compute the limit exactly: along a direction with curvature $λ$ (the second derivative), gradient descent is stable only if $η < 2/ λ$ . In the figure, the $y$ direction has curvature 16, so any learning rate above $2/16 = 0.125$ diverges.

Local minima and saddles

Gradient descent only sees the local slope, so on a bumpy surface it can settle in a local minimum. For the convex losses of linear and logistic regression there is only one minimum, so this can't happen. For neural networks it can, but in very high dimensions most flat points turn out to be saddle points (downhill in some directions), and the noise of stochastic gradient descent helps escape them. In practice, neural network training works far better than worst-case theory suggests.

try itedit the code and run it, nothing can break

python sleeps until you press run

The gradient is a vector of slopes that points uphill. Gradient descent steps the other way, scaled by the learning rate, and repeats.

← Slopes and derivatives