Cross validation
Cross validation is a technique for estimating model performance more reliably than a single train/test split, especially when data is limited. Instead of ...
Reliable Performance Estimates with Limited Data
Cross validation is a technique for estimating model performance more reliably than a single train/test split, especially when data is limited. Instead of holding out one fixed test portion, the data is divided into multiple folds and the model is trained and evaluated multiple times, each time using a different fold as the test set.
In k-fold cross validation the dataset is split into k equal parts. The model is trained on k minus one folds and tested on the remaining fold. This process repeats k times, each fold serving as the test set once. The k performance scores are then averaged to produce a single estimate. Common choices for k are 5 and 10, balancing computational cost against estimate stability.
Stratified k-fold cross validation preserves the class distribution in each fold. In a dataset where 10 percent of examples are positive, each fold is constructed so that approximately 10 percent of its examples are positive. This is important for imbalanced classification problems where a random split might leave some folds with very few positive examples.
Leave-one-out cross validation uses every example as its own test set in turn, training on all remaining examples. It provides an almost unbiased estimate but is computationally expensive for large datasets.
Time series data requires a special approach: walk-forward validation, where the model is always trained on past data and tested on future data. Shuffling time series into random folds would leak future information into the training set, producing misleadingly optimistic results.
A concrete example: a dataset of 500 medical records uses 5-fold cross validation. Five models are trained, each achieving AUC scores of 0.84, 0.81, 0.86, 0.83, and 0.85. The mean AUC of 0.838 is a more stable estimate than any single train/test split would provide.
Exam tip: know that cross validation is used for model selection and performance estimation, not as a replacement for a held-out test set.