The first principle, in Richard Feynman's words, is that you must not fool yourself, and you are the easiest person to fool. Machine learning evaluations are easy to fool because a model will exploit any signal that predicts the label, including signals that won't exist when it's deployed.
Data leakage
Leakage is when information that wouldn't be available at prediction time sneaks into training or evaluation. It produces spectacular test scores and models that fail in production. Common forms:
- Preprocessing on all the data. Scaling, imputing missing values, or selecting features using statistics of the full dataset, before splitting. The test set has influenced training. Fit every preprocessing step on the training fold only.
- Target leakage. A feature that is a consequence of the label. A model predicting pneumonia that uses "was prescribed antibiotics", or a churn model that uses "account closed date".
- Temporal leakage. Training on data from after the moment of prediction, e.g. random splits on time series, or features computed with future information.
- Duplicates and groups across splits. The same patient, user or near-identical image on both sides.
- Shortcut features. A famous case: a model that "detected" pneumonia from chest X-rays partly by recognising which hospital's scanner produced the image, because hospitals had different disease rates (Zech et al., 2018).
Overfitting the test set
Even without leakage, a test set loses its value with every look. Try many models and report the best test score, and you've selected for luck. The effect is easy to underestimate:
Machine learning competitions see this on public leaderboards: teams that tune against the leaderboard drop many places on the hidden final test set. Research benchmarks suffer the same slow erosion as thousands of papers tune to the same test sets. The defences are a test set used once, held-out data that nobody tunes on, and honest reporting of how many things were tried.
Distribution shift
Finally, even a flawless evaluation only describes the distribution the test data came from. If the world changes (new customer behaviour, a new camera, a pandemic) performance can collapse. Monitor deployed models, and when you can, evaluate on data from a later period or a different source to see how robust they are.
If a result looks too good, look for leakage first. Fit every preprocessing step on training data only, split by time and group when needed, and open the test set once.