Back to Study Material Machine Learning

Overfitting and underfitting

Every machine learning model sits somewhere on a spectrum between two failure modes: overfitting and underfitting. Finding the right balance between them i...

Overfitting and underfitting

The Goldilocks Problem of Model Complexity

Every machine learning model sits somewhere on a spectrum between two failure modes: overfitting and underfitting. Finding the right balance between them is central to building models that work reliably in production.

Underfitting happens when a model is too simple to capture the true patterns in the data. A linear model trying to fit a curved relationship will consistently make large errors on both training and test data. Underfitting is also called high bias because the model is making strong, incorrect assumptions that prevent it from learning. Symptoms: poor performance on training data.

Overfitting happens when a model is too complex and learns the noise or peculiarities of the training data rather than the underlying pattern. A very deep decision tree might memorize every training example perfectly but fail on new data because it has learned patterns that do not generalize. Overfitting is also called high variance because small changes in the training data produce large changes in the model. Symptoms: excellent performance on training data but poor performance on test data.

Common remedies for overfitting include collecting more training data, reducing model complexity, applying regularization techniques (L1 or L2), using dropout in neural networks, and using ensemble methods. Common remedies for underfitting include using a more complex model, adding more informative features, and reducing regularization strength.

A concrete example: a polynomial regression model of degree 1 (a line) underfits a sinusoidal pattern. A polynomial of degree 20 overfits it, passing exactly through every training point but oscillating wildly between them. A degree 4 polynomial captures the general shape without chasing noise.

Exam tip: memorize this diagnostic: if training error is high, you are underfitting; if training error is low but test error is high, you are overfitting.