A model declines a loan application. The borrower asks why. If the honest answer is that a gradient-boosted ensemble of four hundred trees produced a score of 0.34 against a cut-off of 0.30, the institution has a problem — not a technical one, a regulatory and ethical one.
What Explainable AI Means
Explainable AI (XAI) refers to methods that make a model’s decisions understandable to humans — being able to state which factors drove a particular outcome and by how much.
Two related but distinct concepts get conflated:
| Interpretability | Explainability | |
|---|---|---|
| Meaning | The model is inherently understandable | The model’s output can be explained after the fact |
| Achieved by | Choosing a simple model | Applying analysis techniques to a complex model |
| Example | Logistic regression with visible coefficients | SHAP values applied to a neural network |
| Reliability | Complete — you read the model | An approximation of the model’s reasoning |
An interpretable model needs no explanation technique. An explanation technique applied to an opaque model produces an approximation of why it decided as it did — useful, but not the same as knowing.
Why Lending Specifically Requires It
Three reasons, in increasing order of consequence.
Customer fairness. A declined borrower deserves an actionable reason. “Insufficient repayment history” tells them something they can work on. A score does not.
Regulatory expectation. Explainability is one of the seven principles in RBI’s FREE-AI framework. Institutions are expected to be able to account for decisions affecting customers.
Institutional self-protection. A model nobody can explain is a model nobody can defend — to a regulator, in a complaint, or in litigation. It is also a model whose failures are invisible until they show up as losses.
The Main Techniques
| Method | What it does | Practical note |
|---|---|---|
| Inherently interpretable models | Logistic regression, small decision trees — read the model directly | Often accurate enough on structured data |
| Reason codes | Maps the outcome to a ranked list of contributing factors | Standard in credit; what customers actually receive |
| SHAP | Attributes the prediction across input variables, showing each one’s contribution | Widely used; computationally heavy on large models |
| LIME | Builds a simple local model approximating behaviour near one case | Fast, but explanations can be unstable |
| Counterfactuals | States what would need to change for a different outcome | Most actionable form for a customer |
Counterfactual explanations deserve attention because they are the most useful to the person affected. “Your application would likely have been approved with six more months of repayment history” is genuinely actionable in a way that a factor ranking is not.
The Accuracy Trade-Off — and Its Limits
The conventional framing is that explainability costs accuracy: simple models explain well but predict worse; complex models predict better but explain poorly.
This is real but frequently overstated for lending. On structured application data — income, tenure, bureau variables — well-constructed simple models often perform close to complex ones. The dramatic accuracy gains from deep learning appear mainly on unstructured data such as images and text.
| Data type | Complexity premium | Sensible choice |
|---|---|---|
| Structured application data | Small | Interpretable model usually adequate |
| Transaction sequences | Moderate | Complex model plus explanation layer |
| Documents, images, voice | Large | Complex model necessary |
The practical implication: for many credit decisions, choosing an interpretable model costs little accuracy and removes the explainability problem entirely rather than patching it afterwards.
Where Explanations Mislead
Post-hoc techniques have genuine limitations that practitioners should know.
- They approximate. SHAP and LIME estimate how the model behaved. They do not read its reasoning, because complex models do not reason in a form that can be read.
- Correlated variables confuse attribution. When two inputs move together, attributing importance between them is somewhat arbitrary, and the explanation can shift with method choice.
- Local explanations do not generalise. Why the model declined this applicant may not describe how it behaves overall.
- An explanation is not a justification. Knowing that pin code drove the decision explains the mechanism. It does not make using pin code appropriate — and it may reveal Explainable AI in Lending.
That last point is the most important. Explainability is a diagnostic tool. Its value lies partly in surfacing decisions that should not have been made that way.
What Good Practice Looks Like
- Prefer interpretable models where accuracy is comparable
- Generate reason codes for every adverse decision, stored and retrievable
- Test explanations for stability — similar applicants should get similar reasons
- Review which variables drive decisions for proxy discrimination
- Give customers actionable language, not variable names
- Document the approach so it can be shown to an auditor or regulator
Key Takeaways
- Interpretability is built in; explainability is applied afterwards — and only approximates
- Explainability is a principle in RBI’s FREE-AI framework, not an optional extra
- Main techniques: interpretable models, reason codes, SHAP, LIME, counterfactuals
- Counterfactuals are the most actionable for the customer
- The accuracy trade-off is small on structured lending data
- Explanations can be unstable and misleading where variables are correlated
- An explanation is not a justification — it may reveal a decision that should not have been made
- A model nobody can explain is a model nobody can defend
Frequently Asked Questions (FAQ)
Q: What is explainable AI?
Explainable AI refers to methods that make a model’s decisions understandable — identifying which factors drove a particular outcome and by how much. In lending it allows an institution to state why an application was declined.
Q: What is the difference between interpretability and explainability?
An interpretable model is inherently understandable — you can read its logic directly, as with logistic regression. Explainability applies analysis techniques to an opaque model after the fact, producing an approximation of its reasoning rather than the reasoning itself.
Q: What is SHAP in machine learning?
SHAP attributes a model’s prediction across its input variables, showing how much each contributed to that specific outcome. It is widely used in credit modelling for generating reason codes, though it is computationally demanding on large models.
Q: Does explainable AI reduce model accuracy?
Less than commonly assumed for lending. On structured application data, well-built interpretable models often perform close to complex ones. The large accuracy gains from complex models appear mainly on unstructured data such as images, documents and voice.
Q: Is explainable AI required by RBI?
Explainability is one of the seven principles set out in the FREE-AI committee report, and institutions are expected to be able to account for AI-driven decisions affecting customers. Whether this becomes a binding requirement depends on RBI’s forthcoming guidance.
Q: What is a counterfactual explanation?
A counterfactual states what would need to change for a different outcome — for example, that the application would likely have been approved with a longer repayment history. It is the most actionable form of explanation from a customer’s perspective.
Q: Can explanations be wrong?
They can be misleading. Post-hoc methods approximate rather than read the model’s reasoning, attribution between correlated variables is somewhat arbitrary, and an explanation for one case may not describe overall behaviour. They remain valuable, but should be treated as evidence rather than proof.
Related Reading: