Back to Study Material Machine Learning

Feature selection

Feature selection is the process of choosing a subset of input features to include in a model, discarding the rest. Good feature selection improves model p...

Feature selection

Keeping Only What Matters

Feature selection is the process of choosing a subset of input features to include in a model, discarding the rest. Good feature selection improves model performance by removing noise, reduces training and prediction time by shrinking the input space, and makes models easier to interpret and maintain. More features is not always better: irrelevant or redundant features add variance without adding signal.

Filter methods evaluate each feature independently using a statistical criterion and rank features by their relationship to the target. Common criteria include correlation coefficients for regression targets and chi-squared statistics or mutual information for classification. Filter methods are fast and model-agnostic but may miss interactions between features.

Wrapper methods train the model repeatedly on different subsets of features and select the subset that produces the best validation performance. Recursive feature elimination is a popular wrapper approach that trains the model, discards the least important feature, and repeats until the desired number of features remains. Wrapper methods consider feature interactions but are computationally expensive.

Embedded methods perform feature selection as part of the training process. Lasso regression adds an L1 penalty to the loss function, which drives the weights of irrelevant features exactly to zero, effectively removing them. Tree-based models compute feature importances based on how much each feature reduces impurity across all splits, which can be used to prune unimportant features after training.

A concrete example: a customer churn model starts with 80 features including customer ID, join date, and redundant billing copies. After recursive feature elimination, 12 features remain, including contract length, number of support calls, and last login date. Model accuracy barely changes but training time drops by 70 percent.

Exam tip: know the three categories of feature selection — filter, wrapper, and embedded — and give one algorithm from each category. Also mention that feature selection should be done inside cross-validation folds to avoid data leakage.