Explainable machine learning
Explainable machine learning, often called XML or XAI, refers to techniques and practices that make a model's predictions understandable to human beings. A...
Opening the Black Box
Explainable machine learning, often called XML or XAI, refers to techniques and practices that make a model's predictions understandable to human beings. As models grew more powerful — from simple decision trees to deep neural networks — they also grew harder to interpret. A neural network might predict that a loan applicant should be rejected, but it offers no plain-language reason. Explainability addresses this gap.
The need for explanation is not merely academic. Regulations in many countries require that individuals be given a reason when an automated system makes a decision that affects them. Medical professionals will not act on a diagnosis model they cannot understand. Engineers need to know why a model fails before they can fix it. Explainability is therefore both an ethical requirement and a practical engineering tool.
Several techniques have emerged to add interpretability. LIME, Local Interpretable Model-agnostic Explanations, works by perturbing a single input slightly and observing how the prediction changes, then fitting a simple linear model around that neighborhood to approximate local behavior. SHAP, SHapley Additive exPlanations, draws on game theory to assign each feature a fair share of the credit or blame for a prediction, giving a consistent global and local picture.
Model choice matters too. Logistic regression and shallow decision trees are intrinsically interpretable because their logic can be read directly. Deep networks are not, so post-hoc tools like LIME and SHAP are applied after the fact to approximate what the model learned.
A concrete example: a credit-scoring model using SHAP values might reveal that a high debt-to-income ratio contributed plus 120 points toward rejection, while a long credit history contributed minus 80 points, yielding a net decision to reject. The applicant receives a meaningful explanation rather than a black-box answer.
Exam tip: know the difference between intrinsic interpretability, where the model itself is readable, and post-hoc interpretability, where a separate technique explains an already-trained opaque model.