Bayes' theorem and maximum likelihood

Probability and statisticsabout 11 minutes

Conditional probability $P (A ∣ B)$ is the probability of $A$ once you know $B$ happened. It is not symmetric: the probability that a person is a basketball player given they are 2 m tall is very different from the probability that a basketball player is 2 m tall. Bayes' theorem converts one into the other:

P (A ∣ B) = \frac{P ( B ∣ A ) P ( A )}{P ( B )}

The base-rate trap

A disease affects 1% of people. A test catches 90% of sick people (its sensitivity) but also flags 9% of healthy people (its false-positive rate). You test positive. What is the chance you are sick? Most people, including many doctors in studies, guess around 90%. Count it out for 1,000 people:

10 are sick; 9 of them test positive.
990 are healthy; 9% of them, about 89, also test positive.
So of roughly 98 positives, only 9 are sick: $9/98 \approx 9%$ .

1,000 people. Change the prevalence and the test's error rates and count who is actually sick among the positives.

The rare condition is swamped by false alarms from the huge healthy group. The same arithmetic explains why a fraud detector with "99% accuracy" can still produce mostly false alarms, a theme you'll meet again in module 3 when we discuss precision and recall.

P (sick ∣ +) = \frac{0.9 \times 0.01}{0.9 \times 0.01 + 0.09 \times 0.99} \approx 0.092

Likelihood: turning the question around

So far the probabilities were known and we reasoned about data. Learning goes the other way: the data is known and the parameters are not. The likelihood $L (θ) = P (data ∣ θ)$ treats the probability of the observed data as a function of the parameters $θ$ .

Maximum likelihood estimation (MLE) picks the $θ$ that makes the observed data most probable. Flip a coin 10 times and get 7 heads. If the probability of heads is $p$ , the likelihood of that exact sequence is $p^{7} (1 - p)^{3}$ , which peaks at $p = 0.7$ . The MLE is simply the observed fraction, which is reassuring.

The likelihood of 7 heads in 10 flips, as a function of p. Its peak is the maximum likelihood estimate.

In practice we maximise the log-likelihood instead: the log turns a product of many small probabilities (which underflows to 0 on a computer) into a sum, and it has the same maximum. Flip the sign and you get a loss to minimise, the negative log-likelihood:

NLL (θ) = - i = 1 \sum n lo g P (x_{i} ∣ θ)

Assume the labels are Bernoulli and the NLL becomes binary cross-entropy, the loss of logistic regression (module 2).
Assume Gaussian noise around a prediction and the NLL becomes, up to constants, the sum of squared errors, the loss of linear regression.
The MLE of a Gaussian's mean is the sample mean, and of its variance is the average squared deviation (dividing by $n$ , not $n - 1$ ).

Most loss functions are negative log-likelihoods in disguise. Training a model is maximum likelihood estimation: find the parameters under which the data you saw is least surprising.

← Expectation, variance and correlation