Model evaluation
Evaluating a machine learning model correctly is just as important as training it. Poor evaluation leads to overconfidence, unexpected failures in producti...
Measuring What a Model Actually Learned
Evaluating a machine learning model correctly is just as important as training it. Poor evaluation leads to overconfidence, unexpected failures in production, and decisions based on misleading metrics. Good evaluation requires choosing the right metric, the right data split, and understanding what each number actually means.
For classification tasks, accuracy — the proportion of correct predictions — is the most intuitive metric. However it is misleading when classes are imbalanced. If 95 percent of loan applications are legitimate, a model that predicts "legitimate" for every application achieves 95 percent accuracy while being completely useless. Precision, recall, F1 score, and the ROC-AUC curve provide a richer view of classifier behavior.
For regression tasks, common metrics include Mean Absolute Error (the average size of prediction errors), Mean Squared Error (which penalizes large errors more heavily), and R-squared (which measures how much of the variance in the target the model explains).
Choosing the right metric requires understanding the business context. In medical screening, missing a true case (low recall) may be far more harmful than raising false alarms (low precision). In spam filtering, falsely blocking legitimate emails (low precision) may be more damaging than letting some spam through. The metric should reflect these priorities.
A concrete example: a fraud detection model flags 100 transactions as fraud per day. Of those, 80 are genuine fraud (precision of 80 percent). Meanwhile it misses 40 fraud cases in a day of 10,000 transactions (recall of 67 percent). The business must decide whether missing fewer fraud cases or flagging fewer legitimate transactions matters more.
Exam tip: always ask which metric matters most before training. Optimizing for the wrong metric is one of the most common and costly mistakes in applied machine learning.