Densities, anomalies and maps of data

Learning without labelsabout 9 minutes

Clustering gives each point a hard label. Often it's more useful to model the density of the data: how likely is a point like this? Then rare points are, by definition, the ones in low-density regions.

Anomaly detection with a Gaussian

Collect examples of normal operation (engine sensor readings, transactions, server metrics). Fit a Gaussian to them, estimating the mean and variance by maximum likelihood as in module 1. For a new point, compute its density $p (x)$ . If $p (x) < ε$ , flag it.

p (x) = j = 1 \prod d \frac{1}{σ _{j} 2 π} exp (- \frac{( x _{j} - μ _{j} ) ^{2}}{2 σ _{j}^{2}}) flag if p (x) < ε

A Gaussian fitted to normal points. Raise ε and the flagged region grows: more anomalies caught, more false alarms.

This works when anomalies are rare and varied: you can't collect enough examples of every future failure to train a classifier, but you can describe normal.
Choose $ε$ with a small labelled validation set, using precision and recall from the previous unit. Accuracy is useless here.
Multiplying per-feature densities assumes independent features. A full covariance matrix (a multivariate Gaussian) captures correlations; Gaussian mixture models capture several modes of normal.

Gaussian mixtures: soft clustering

A Gaussian mixture model describes data as coming from $k$ Gaussians, each with its own mean, covariance and weight. It's fitted with the expectation-maximisation (EM) algorithm, which alternates exactly like k-means: compute each point's probability of belonging to each component (soft assignment), then refit each component to its weighted points. k-means is the limit of EM when all components are round and the assignments are hard.

Reading maps of high-dimensional data

PCA is linear: it can't unroll curved structure. t-SNE (van der Maaten & Hinton, 2008) and UMAP (McInnes et al., 2018) are non-linear methods that try to keep each point's nearest neighbours nearby in 2-D. They produce striking pictures of clusters in images, text or cells. Read them carefully:

Distances between clusters, and the sizes of clusters, are not meaningful. Only local neighbourhoods are.
Results depend on hyperparameters (perplexity, number of neighbours) and the random seed. Try several.
They can show clusters in pure noise if pushed. Use them to explore and form hypotheses, not as evidence.

try itedit the code and run it, nothing can break

python sleeps until you press run

Model normal, then flag the improbable. Density models detect anomalies you've never seen; non-linear maps like t-SNE are for exploring, not proving.

← Principal component analysis