Random variables and distributions

Probability and statisticsabout 9 minutes

Data is noisy, labels are sometimes wrong, and the future is uncertain. Probability is the language for that uncertainty, and most ML models are, underneath, probability distributions with adjustable parameters. A classifier that says "90% cat" is outputting a distribution.

A random variable is a quantity whose value comes out of a random process: the result of a coin flip, the height of a randomly chosen person. Its distribution says how likely each value is.

Discrete: Bernoulli and categorical

A Bernoulli variable is a single yes/no event with probability $p$ of "yes" (1). It is the distribution behind every binary classifier. Its generalisation to $K$ outcomes is the categorical distribution, a list of $K$ probabilities that sum to 1, which is exactly what a softmax layer outputs. For discrete variables, the probabilities themselves are given by a probability mass function:

P (X = 1) = p, P (X = 0) = 1 - p ⟺ P (X = x) = p^{x} (1 - p)^{1 - x}

Continuous: the Gaussian

Heights, measurement errors and many other continuous quantities follow the normal (Gaussian) distribution, the bell curve, with mean $μ$ (the centre) and standard deviation $σ$ (the spread):

p (x) = \frac{1}{σ 2 π} exp (- \frac{( x - μ ) ^{2}}{2 σ ^{2}})

For continuous variables $p (x)$ is a density, not a probability: the probability of landing in an interval is the area under the curve over that interval, and the probability of any exact value is 0. A density can be greater than 1 (a very narrow Gaussian has a tall peak); only its total area must be 1. About 68% of the area lies within one $σ$ of the mean, and 95% within two.

Draw samples from a Gaussian. With more samples the histogram settles onto the true density curve: the law of large numbers in action.

Why is the Gaussian everywhere? The central limit theorem: the sum (or average) of many independent random effects is approximately normal, whatever their individual distributions, as long as their variance is finite. Measurement error is the sum of many small disturbances, so it tends to be Gaussian. That is also why the average of a sample has a predictable spread, as you'll see next lesson.

try itedit the code and run it, nothing can break

python sleeps until you press run

A distribution assigns probability to outcomes. Classifiers output categorical distributions; regression usually assumes Gaussian noise around the prediction.

← Numerical gradients and a descent loop