A single validation split has two problems. It wastes data: the validation examples never help train the model. And it's noisy: with a few hundred examples, a different random split can change the score by several points, enough to pick the wrong hyperparameter.
k-fold cross-validation
Split the training data into equal folds (typically 5 or 10). Train times, each time holding out a different fold for validation and training on the other . Every example is used for validation exactly once. Report the mean of the scores, and their spread as a rough error bar.
- Cost: times the training. Fine for linear models and trees, often too expensive for big neural networks, which usually use a single validation split.
- Stratified k-fold keeps each fold's class proportions equal to the whole dataset's, important for imbalanced classes.
- Leave-one-out () is nearly unbiased but expensive and high-variance.
- The same i.i.d. caveats apply: use group k-fold for grouped data and time-series splits (always validate on later data) for temporal data.
Tuning with cross-validation
To choose a hyperparameter, run k-fold CV for each candidate value, pick the value with the best mean score, then retrain on all the training data with that value. Finally, evaluate once on the untouched test set.
Note what the best CV score is not: an unbiased estimate of the final model's performance. It's the maximum over candidates, so it's optimistic. That's why the test set is still needed. When data is too scarce for a test set, nested cross-validation runs the tuning loop inside each outer fold.
k-fold CV trains k models, validates each on a different fold, and averages. It uses the data efficiently and shows how much a score wobbles.