Accuracy is not enough

Measuring models honestlyabout 11 minutes

A fraud detector reports 99.5% accuracy. Impressive, until you learn that 0.5% of transactions are fraudulent, so a model that says "not fraud" every time scores exactly 99.5% while catching nothing. On imbalanced problems (disease, fraud, defects, spam) accuracy hides what matters.

The confusion matrix

For a binary problem, call the rare, interesting class positive. Every prediction falls into one of four cells:

	predicted positive	predicted negative
actually positive	true positive (TP)	false negative (FN): a miss
actually negative	false positive (FP): a false alarm	true negative (TN)

Precision $= \frac{TP}{TP + FP}$ : of the alarms raised, how many were real?
Recall (sensitivity, true positive rate) $= \frac{TP}{TP + FN}$ : of the real cases, how many did we catch?
F1 $= \frac{2 \cdot precision \cdot recall}{precision + recall}$ : their harmonic mean, high only when both are.
False positive rate $= \frac{FP}{FP + TN}$ : of the negatives, how many did we wrongly flag?

Thresholds trade one for the other

A probabilistic classifier gives a score; the threshold turns it into a decision. Lower the threshold and you catch more positives (recall up) at the cost of more false alarms (precision usually down). There is no free lunch: the right point depends on the costs. A cancer screen wants high recall; a spam filter that deletes mail wants high precision.

Scores of positives (white) and negatives (blue). Slide the threshold and watch the confusion matrix, precision, recall, and the point on the ROC curve.

ROC curves and AUC

The ROC curve plots the true positive rate against the false positive rate for every threshold at once. A perfect classifier hugs the top-left corner; random guessing is the diagonal. The area under the curve (AUC) summarises it in one threshold-free number, with a lovely interpretation: AUC is the probability that a randomly chosen positive gets a higher score than a randomly chosen negative. AUC 0.5 is coin-flipping; 1.0 is perfect ranking.

With heavy imbalance, ROC can still look rosy because the false positive rate divides by the huge number of negatives. The precision-recall curve is then more telling, since precision feels every false alarm.

try itedit the code and run it, nothing can break

python sleeps until you press run

On imbalanced problems, report precision and recall (or AUC), not accuracy. The threshold is a business decision about which mistakes cost more.

← Cross-validation