Model monitoring
Model monitoring is the practice of continuously tracking a deployed model's behavior and performance in production to detect when it starts degrading or b...
Keeping a Deployed Model Healthy Over Time
Model monitoring is the practice of continuously tracking a deployed model's behavior and performance in production to detect when it starts degrading or behaving unexpectedly. A model that performs excellently at launch can silently deteriorate over weeks or months as the real world changes. Without monitoring, that degradation goes unnoticed until it causes business impact.
Data drift is one of the most common causes of model degradation. It occurs when the statistical distribution of incoming features changes relative to the training data. A model trained on user behavior from one year may become less accurate when user behavior shifts due to seasonal changes, economic events, or platform changes. Monitoring compares the distributions of incoming features against training distributions using statistical tests or distance metrics.
Concept drift is more fundamental: the relationship between features and labels changes. A fraud detection model trained before a new type of scam emerges will assign low fraud probabilities to the new scam patterns because it has never seen them. Monitoring model predictions, error rates, and calibration helps detect when the learned relationship no longer holds.
Operational metrics must also be tracked: prediction latency, throughput, memory usage, and error rates from the serving infrastructure. A model that is logically correct but too slow in production causes a different kind of failure.
Alert thresholds trigger notifications when metrics fall outside acceptable ranges. Human review or automatic retraining pipelines are triggered in response. The cadence of review depends on how quickly the environment changes: a news recommendation model may need daily evaluation while a medical imaging model may need only monthly review.
A concrete example: an e-commerce recommendation model's click-through rate dropped by 18 percent over six weeks following a major product catalogue expansion. Monitoring detected the drift in item popularity distributions and triggered a retraining job, restoring click-through rate within a week.
Exam tip: distinguish between data drift, where feature distributions change, and concept drift, where the target relationship changes. Both require monitoring but may call for different responses.