Principal component analysis

Learning without labelsabout 11 minutes

Real datasets have many features, but they rarely use all their dimensions. The 64 pixels of an 8×8 handwritten digit are strongly correlated: neighbouring pixels move together, and the corners are almost always blank. Principal component analysis (PCA, Pearson 1901 and Hotelling 1933) finds a new set of axes ordered by how much the data varies along them, so you can keep the first few and drop the rest.

The direction of maximum variance

Centre the data (subtract the mean). Project every point onto a unit vector $u$ : the projections are $X u$ . The first principal component is the $u$ that makes the variance of those projections as large as possible.

Rotate the projection direction. The spread of the shadows is largest along the first principal component and smallest perpendicular to it.

The answer comes from linear algebra: the variance along $u$ is $u^{⊤} C u$ , where $C = \frac{1}{n} X^{⊤} X$ is the covariance matrix, and this is maximised by the eigenvector of $C$ with the largest eigenvalue. That eigenvalue is the variance along it. The second component is the best direction perpendicular to the first, and so on.

C u_{j} = λ_{j} u_{j}, λ_{1} \geq λ_{2} \geq \dots \geq λ_{d}, explained variance ratio_{j} = \frac{λ _{j}}{\sum _{i} λ _{i}}

In practice PCA is computed with the singular value decomposition of the centred data, $X = U S V^{⊤}$ : the rows of $V^{⊤}$ are the principal directions and $S^{2} / n$ are the variances. It's more numerically stable than forming $C$ .

What it's used for

Visualisation: project to 2 components and plot. Digits that look alike land near each other.
Compression and denoising: keep the components that explain, say, 95% of the variance, and reconstruct from them. Small components often carry mostly noise.
Preprocessing: fewer, uncorrelated features can speed up and stabilise downstream models.

PCA is driven by variance, so standardise features with different units first, or the one with the biggest numbers wins. And high variance is not the same as useful: the direction that separates two classes may have small variance. PCA ignores labels entirely.

try itedit the code and run it, nothing can break

python sleeps until you press run

PCA rotates data onto the axes of greatest variance (the covariance matrix's eigenvectors) so you can keep the few that matter.

← Clustering with k-means