Back to Study Material Machine Learning

Feature scaling

Feature scaling is the preprocessing step of transforming numerical features so they occupy comparable numerical ranges. Many algorithms are sensitive to t...

Feature scaling

Putting All Features on an Equal Footing

Feature scaling is the preprocessing step of transforming numerical features so they occupy comparable numerical ranges. Many algorithms are sensitive to the scale of inputs. Distance-based methods like KNN and SVM compute distances between points, so a feature measured in thousands will dominate a feature measured in fractions unless both are scaled. Gradient descent converges much faster when all features are on a similar scale because the loss surface becomes more symmetric and well-conditioned.

Standardization, also called Z-score normalization, subtracts the feature's mean and divides by its standard deviation, producing a distribution with mean zero and standard deviation one. It does not bound values to a fixed range, so features with genuine outliers will have values far from zero, but the transformation is stable and the resulting scale is consistent.

Min-max normalization, also called feature rescaling, maps values to the range zero to one by subtracting the minimum and dividing by the range. It preserves zero values and is appropriate when the algorithm requires bounded inputs, such as neural networks with sigmoid output layers. However, it is sensitive to outliers: a single extreme value compresses all other values near zero.

Robust scaling uses the median and the interquartile range instead of the mean and standard deviation, making it less sensitive to outliers than standardization.

A critical rule: fit the scaler only on the training set, then apply the same transformation to validation and test sets. Fitting on the full dataset before splitting leaks information from the test set into preprocessing and produces unrealistically optimistic estimates.

A concrete example: a model predicts house prices using square footage ranging from 500 to 5000 and number of bathrooms ranging from 1 to 5. Without scaling, gradient descent takes thousands of extra iterations because the loss surface is very elongated. After standardization, convergence is roughly ten times faster.

Exam tip: algorithms that rely on distances or gradients — KNN, SVM, logistic regression, neural networks — require scaling. Tree-based algorithms like decision trees and random forests do not, because they split by thresholds rather than measuring distances.