Back to Study Material Machine Learning

Information gain

Information gain measures how much a feature reduces uncertainty about the class labels when used to split a dataset. It is defined as the entropy of the p...

Information gain

Choosing the Feature That Tells You the Most

Information gain measures how much a feature reduces uncertainty about the class labels when used to split a dataset. It is defined as the entropy of the parent node minus the weighted average entropy of the child nodes produced by the split. Higher information gain means the feature does more to separate the classes, making it a better choice for a split.

The concept comes from Claude Shannon's information theory. Entropy measures uncertainty in bits. If knowing a feature's value reduces that uncertainty by a large amount, the feature is informative. If it barely changes the distribution of labels, it carries little information and should not be used for splitting.

The ID3 algorithm for building decision trees selects at each node the feature with the highest information gain among all remaining features. This greedy approach does not guarantee a globally optimal tree but produces useful results efficiently. C4.5, an improved version, divides information gain by the feature's own entropy — called the split information — yielding the gain ratio. This correction prevents the algorithm from unfairly favoring features with many distinct values, such as a unique identifier column.

One practical issue with raw information gain is that features with many categories tend to score high because they split the data into many pure groups, even if those splits are meaningless. A customer ID column perfectly separates the data but is useless for prediction. Gain ratio corrects this bias.

A concrete example: in a dataset predicting whether a person buys a product, the feature "age group" might split data into young, middle, and older categories. If the label distribution is very different across these groups, information gain is high, meaning age group is a strong predictor worth splitting on first.

Exam tip: understand why gain ratio is preferred over raw information gain in C4.5 — it penalizes features with many distinct values that create artificially pure but useless splits.