Data normalization
Data normalization, or feature scaling, transforms numerical features so they occupy comparable ranges. Many machine learning algorithms perform poorly or ...
Putting Features on Equal Footing
Data normalization, or feature scaling, transforms numerical features so they occupy comparable ranges. Many machine learning algorithms perform poorly or converge slowly when features have very different scales. A feature measured in thousands of dollars and a feature measured in single-digit ratings pull gradient-based optimization in unbalanced directions, causing slow training and suboptimal solutions.
Min-max scaling transforms each feature to a range between zero and one by subtracting the minimum value and dividing by the range. It preserves the shape of the distribution and works well when you need values in a bounded range, but it is sensitive to outliers — a single extreme value compresses all other values into a narrow band.
Standardization (Z-score normalization) subtracts the mean and divides by the standard deviation, resulting in a distribution with mean zero and standard deviation one. It does not bound values to a range, which suits algorithms that assume or work better with normally distributed inputs. It is less affected by outliers than min-max scaling.
Robust scaling uses the median and interquartile range instead of the mean and standard deviation, making it resistant to extreme values. It is appropriate when outliers cannot be removed and must remain in the training data.
Algorithms sensitive to scale include gradient descent-based methods (linear regression, neural networks, SVMs), distance-based methods (K-nearest neighbors, K-means), and regularized models (Lasso, Ridge). Tree-based methods like decision trees and random forests are scale-invariant — they split on threshold values, so relative scale does not matter.
A concrete example: a model predicting insurance premiums uses age (range 18 to 80) and annual income (range 20,000 to 200,000). Without scaling, income dominates gradient updates entirely.
Exam tip: always specify which scaler you would use and why. Mentioning that tree-based models do not need scaling shows deeper understanding.