Ensemble learning
Ensemble learning is the practice of combining multiple models to produce a prediction that is more reliable than any single model alone. The intuition com...
The Wisdom of Many Models
Ensemble learning is the practice of combining multiple models to produce a prediction that is more reliable than any single model alone. The intuition comes from the idea of a committee: individual members may each make different mistakes, but when they vote together, errors tend to cancel out and correct judgments dominate. In practice, ensembles consistently outperform individual models on benchmark competitions and real-world tasks.
The oldest and simplest ensemble strategy is voting. For classification, each model predicts a class and the majority wins. For regression, predictions are averaged. This works well when the individual models are diverse — meaning they make different errors — because their mistakes do not reinforce each other.
Bagging, short for bootstrap aggregating, builds diversity by training each model on a different random subset of the training data drawn with replacement. Random forests are the most famous bagging method: they train many decision trees, each on a different data subset and with a random subset of features at each split, then average their predictions. The randomness ensures the trees are diverse despite coming from the same dataset.
Boosting takes a different approach. Models are trained sequentially, and each new model focuses on the examples the previous models got wrong. Gradient boosting frameworks such as XGBoost and LightGBM are among the most powerful general-purpose algorithms available, dominating tabular data competitions.
Stacking trains a meta-model on the outputs of base models, learning how to combine their predictions optimally rather than just averaging them.
A concrete example: in the Netflix Prize competition, the winning solution was a stacked ensemble of hundreds of models. No single model came close to the accuracy of the combined system.
Exam tip: know the difference between bagging, which reduces variance by training models in parallel on different data samples, and boosting, which reduces bias by training models sequentially to correct earlier errors.