Gini index
The Gini index is an alternative impurity measure to entropy used in decision tree algorithms, most notably in CART, Classification and Regression Trees. I...
Measuring Impurity Without Logarithms
The Gini index is an alternative impurity measure to entropy used in decision tree algorithms, most notably in CART, Classification and Regression Trees. It measures the probability that a randomly chosen example from a node would be incorrectly classified if its label were assigned randomly according to the class distribution in that node. A perfectly pure node has a Gini index of zero. A node with equal representation of two classes has a Gini index of 0.5, the maximum for binary classification.
The formula for the Gini index is one minus the sum of the squared class probabilities over all classes. For a binary node with proportions p and one minus p, it simplifies to two times p times one minus p. This is computationally cheaper than entropy because it avoids logarithms, making CART faster to train on large datasets.
When building a tree, CART evaluates all possible splits and selects the one that minimizes the weighted average Gini index of the child nodes. The logic mirrors entropy-based splitting: find the split that produces the purest possible children. In practice, trees built with Gini index and trees built with entropy produce very similar structures on most datasets, so the choice between them rarely affects final model performance significantly.
Both Gini and entropy are symmetric and reach their maximum at equal class proportions, which means they agree on what constitutes maximum impurity. The main practical reason to prefer Gini is speed.
A concrete example: a node in a medical diagnosis tree contains 70 patients with a condition and 30 without. The Gini index is one minus 0.7 squared minus 0.3 squared, which equals one minus 0.49 minus 0.09, giving 0.42. A split that produces two children, each with 90 percent of one class, would have weighted Gini near 0.18 — a large reduction worth choosing.
Exam tip: when comparing Gini and entropy in an interview, say they produce similar trees but Gini is computationally faster because it avoids computing logarithms.