Back to Study Material Machine Learning

Data drift

Data drift refers to changes in the statistical properties of the data a deployed model receives over time, compared to the data it was trained on. Because...

Data drift

When the World Changes and Your Model Does Not

Data drift refers to changes in the statistical properties of the data a deployed model receives over time, compared to the data it was trained on. Because machine learning models learn patterns from historical data, they implicitly assume the future will resemble the past. When this assumption breaks, model performance degrades — often silently, without obvious errors.

Two main types of drift affect deployed models. Feature drift (also called covariate shift) occurs when the distribution of input features changes. A fraud detection model trained before a new payment method was introduced will encounter transactions with unfamiliar patterns. Consumer behavior models trained before a major economic event may receive inputs from a very different population than the training set represented.

Concept drift is more fundamental: the underlying relationship between features and the target changes. In credit scoring, the economic factors that predicted default risk before a recession may have different weights after it. The features are the same, but the mapping from inputs to outcomes has shifted.

Detecting drift requires ongoing monitoring. Statistical tests like the Kolmogorov-Smirnov test or Population Stability Index compare feature distributions between a baseline (training data) and current production data. Monitoring prediction distributions — sudden shifts toward one class or unusual confidence scores — can also signal drift. Tracking downstream business metrics (conversion rates, claim rates) provides a real-world signal but with lag.

Addressing drift involves retraining the model on more recent data, collecting labels for new production examples through active learning or human annotation, or building drift-adaptive models that weight recent examples more heavily.

A concrete example: a model predicting restaurant demand was trained on pre-pandemic data. After lockdowns ended, the model systematically underestimated demand because new dining patterns differed dramatically from its training distribution. Retraining on post-pandemic data restored accuracy.

Exam tip: distinguish feature drift (input distribution changes) from concept drift (the input-output relationship changes) — the distinction shapes the appropriate monitoring and retraining strategy.