Train validation test split
Splitting data into training, validation, and test sets is a foundational practice that keeps model evaluation honest. Each partition plays a specific role...
Three Partitions with Three Distinct Roles
Splitting data into training, validation, and test sets is a foundational practice that keeps model evaluation honest. Each partition plays a specific role, and confusing those roles leads to overly optimistic estimates of real-world performance.
The training set is used to fit the model's parameters. The model sees this data during the learning process and adjusts itself to minimize its error on these examples. Because the model is directly optimized on training data, performance here is not a reliable indicator of how it will perform on new data.
The validation set is used to make decisions about the model — selecting among candidate algorithms, tuning hyperparameters, choosing feature sets. Because the practitioner uses validation results to guide choices, the model indirectly adapts to this data over many iterations. This is why validation accuracy is a better estimate of generalization than training accuracy but still slightly optimistic.
The test set is reserved for a single, final evaluation after all modeling decisions are complete. It should never influence any modeling choices. Its sole purpose is to produce an honest estimate of how the deployed model will perform on completely new data. If you look at test results and then adjust the model, the test set becomes contaminated.
Common split ratios depend on dataset size. For moderately sized datasets a 70/15/15 or 60/20/20 split is typical. For very large datasets (millions of examples), the validation and test sets need far fewer than 20 percent — a few thousand examples suffice to estimate performance reliably.
A concrete example: a model is trained on 70 percent of data, tuned using the validation set across 50 hyperparameter combinations, then evaluated once on the test set. Reporting test accuracy as the final performance estimate is legitimate. Reporting the best validation accuracy would be optimistic.
Exam tip: be able to explain why each split exists and what goes wrong if you skip the validation set or peek at the test set during development.