Bias and variance
The bias-variance tradeoff is a theoretical framework for understanding prediction error in machine learning models. Every model's expected error on new da...
Two Sources of Prediction Error
The bias-variance tradeoff is a theoretical framework for understanding prediction error in machine learning models. Every model's expected error on new data can be decomposed into three components: bias, variance, and irreducible noise. Managing the first two is the central challenge of model selection.
Bias is the error introduced by the model's assumptions. A linear model applied to a non-linear relationship has high bias because its structural assumption — linearity — is wrong. It will consistently predict poorly regardless of how much training data it receives. High-bias models underfit: they are too simple to represent the true pattern.
Variance is the error introduced by the model's sensitivity to fluctuations in the training data. A complex model that fits its training data exactly (like a very deep tree without pruning) has high variance: small changes in the training set produce dramatically different models. When evaluated on new data, these models overfit: their learned patterns are too specific to the training examples.
Irreducible noise is the error that no model can eliminate — the inherent randomness in the real-world process being modeled. Even a perfect model cannot predict outcomes that are fundamentally random.
The tradeoff: increasing model complexity generally reduces bias but increases variance. Decreasing complexity does the opposite. Regularization, ensemble methods, and careful architecture choices aim to find models with low bias and low variance simultaneously, which is possible when sufficient data is available.
A concrete example: a polynomial regression model of degree 1 has high bias and low variance — its predictions barely change with different training sets but are consistently wrong. A degree-15 polynomial has low bias but high variance — it fits training data perfectly but produces wildly different predictions on different training samples.
Exam tip: connect bias-variance to underfitting and overfitting explicitly, and know that more data primarily helps reduce variance, not bias.