Cross-entropy loss

Classificationabout 9 minutes

To train logistic regression we need a loss. Squared error between a probability and a 0/1 label is a poor choice: combined with the sigmoid it creates a non-convex loss with flat regions where a confidently wrong model gets almost no gradient. Instead we go back to maximum likelihood.

Derived from likelihood

Each label is a Bernoulli outcome with probability $p_{i} = σ (z_{i})$ . The probability of the observed label is $p_{i}$ if $y_{i} = 1$ and $1 - p_{i}$ if $y_{i} = 0$ . Take the negative log and average, and you get binary cross-entropy (also called log loss):

BCE = - \frac{1}{n} i = 1 \sum n [y_{i} lo g p_{i} + (1 - y_{i}) lo g (1 - p_{i})]

The loss for one example. Confident and right costs almost nothing; confident and wrong costs without limit.

The shape is the point. Predicting 0.9 for a true 1 costs $- lo g 0.9 \approx 0.105$ . Predicting 0.01 for a true 1 costs $- lo g 0.01 \approx 4.6$ . Predicting exactly 0 costs infinity, which is why implementations clip probabilities to something like $[1 0^{- 12}, 1 - 1 0^{- 12}]$ or, better, compute the loss directly from the logit.

A beautifully simple gradient

Differentiate BCE through the sigmoid with the chain rule, and the $p (1 - p)$ from the sigmoid's derivative cancels against the $1/ p$ and $1/ (1 - p)$ from the logs. What's left is just the prediction error:

\frac{\partial BCE _{i}}{\partial z _{i}} = p_{i} - y_{i} ⟹ \nabla_{w} BCE = \frac{1}{n} X^{⊤} (p - y)

Compare with linear regression's $\frac{2}{n} X^{⊤} (X w - y)$ : the same form, residual times input. That's no accident: both are generalised linear models with their natural (canonical) loss. The BCE loss for logistic regression is also convex, so gradient descent finds the global minimum. Unlike linear regression there's no closed form, so we must iterate.

An information-theory view

The name comes from information theory. $- lo g p$ is the surprise of an event with probability $p$ , measured in nats (or bits with $lo g_{2}$ ). Cross-entropy is the average surprise when outcomes come from the true distribution but you predicted with the model's. It is minimised when the model's probabilities match reality. The same loss trains classifiers with a thousand classes and language models with a vocabulary of a hundred thousand tokens.

If the classes are perfectly separable, logistic regression's weights grow without bound: the loss can always be shrunk a little more by becoming more confident. Regularisation (module 3) or early stopping keeps them finite.

try itedit the code and run it, nothing can break

python sleeps until you press run

Binary cross-entropy is the negative log-likelihood of Bernoulli labels. Its gradient with respect to the logit is just $p - y$ .

← Logistic regression