Random forest
Random forest is an ensemble learning algorithm that builds many decision trees and combines their predictions. It addresses the fundamental weakness of in...
Strength in Numbers for Decision Trees
Random forest is an ensemble learning algorithm that builds many decision trees and combines their predictions. It addresses the fundamental weakness of individual decision trees — their tendency to overfit — by averaging the outputs of many diverse trees, each trained on a different random subset of data and features.
The algorithm builds each tree using bootstrap aggregating, or bagging. A bootstrap sample is drawn by randomly sampling the training data with replacement, creating a dataset of the same size as the original but with some examples repeated and some omitted. Each tree is trained on its own bootstrap sample, making the trees somewhat different from each other.
At each split in each tree, only a random subset of features is considered as candidates for splitting. This additional randomness further decorrelates the trees. If one feature is strongly predictive, a non-random forest would produce trees that all look similar because they all split on the same dominant feature first. Randomly restricting the feature candidates forces diversity.
Predictions from a random forest are made by majority vote (classification) or averaging (regression) across all trees. Because individual trees make errors in different places and the errors are uncorrelated, averaging smooths them out. The result is a model with lower variance than any single tree while retaining similarly low bias.
Random forests also provide feature importance scores: features used frequently in high-gain splits across trees receive high importance. This makes them useful for understanding which inputs most influence predictions.
A concrete example: a random forest of 500 trees applied to customer churn prediction uses 20 features. Feature importance reveals that contract type and tenure are the two most influential predictors, informing customer retention strategy.
Exam tip: be ready to explain why bagging reduces variance without significantly increasing bias, and contrast random forests with gradient boosting, where trees are built sequentially to correct each other's errors.