Expectation, variance and correlation

Probability and statisticsabout 9 minutes

The expected value (mean) of a random variable is its probability-weighted average: what you'd get on average over infinitely many draws. For a fair six-sided die, $E [X] = (1 + 2 + 3 + 4 + 5 + 6) /6 = 3.5$ , a value the die can never actually show.

E [X] = x \sum x P (X = x) or E [X] = \int x p (x) d x

Expectation is linear: $E [a X + bY] = a E [X] + b E [Y]$ , always, even when $X$ and $Y$ are dependent. The loss functions you'll minimise are expectations over the data distribution, approximated by averages over a training set.

Variance and standard deviation

The variance is the expected squared distance from the mean, and the standard deviation $σ$ is its square root, back in the original units. Scaling a variable scales its variance by the square: $Var (a X) = a^{2} Var (X)$ .

Var (X) = E [(X - E [X])^{2}] = E [X^{2}] - E [X]^{2}

How sure is an average?

Average $n$ independent samples and the variance of the average shrinks by a factor of $n$ , so its standard deviation, the standard error, shrinks by $n$ :

SE (\overset{x}{ˉ}) = \frac{σ}{n}

This matters every time you measure a model. A test accuracy of 90% measured on 100 examples has a standard error of about $0.9 \times 0.1/100 = 3$ percentage points, so a rival model scoring 92% might not be better at all. To halve the uncertainty you need four times as much test data.

Covariance and correlation

Covariance measures whether two variables move together: positive if one tends to be above its mean when the other is. The correlation coefficient $ρ$ rescales covariance to lie between −1 and 1:

ρ_{X Y} = \frac{Cov ( X , Y )}{σ _{X} σ _{Y}}, Cov (X, Y) = E [(X - μ_{X}) (Y - μ_{Y})]

Slide the correlation. At ρ = 0 the cloud is round; near ±1 it collapses onto a line.

Correlation only measures linear relationships. $Y = X^{2}$ with $X$ symmetric around 0 has correlation 0, yet $Y$ is completely determined by $X$ .
Correlation is not causation. Ice-cream sales and drowning deaths are correlated because both rise in summer. A model trained on correlations will happily use such confounded features, and it will break when the hidden cause changes.
Correlation is the cosine similarity of the two centred data vectors: the dot product again.

try itedit the code and run it, nothing can break

python sleeps until you press run

Averages get more reliable with $n$ , not $n$ . Before celebrating a 1% improvement, ask how big the test set was.

← Random variables and distributions