A credit model deployed in 2019 performed excellently. By 2021 it was making materially worse decisions, and nobody noticed for months. Nothing had broken. No error appeared in any log. The model kept returning confident scores. The world had simply moved, and the model had not.
What Model Drift Is
Model drift is the gradual degradation of a model’s accuracy as the relationships it learned during training stop holding in the real world.
The property that makes it dangerous: there is no failure signal. Conventional software fails visibly — an exception, a crash, a wrong output that looks wrong. A drifting model produces perfectly well-formed predictions that are progressively less correct.
The Two Types
| Data drift | Concept drift | |
|---|---|---|
| What changes | The inputs the model sees | The relationship between inputs and outcome |
| Example | Applicant mix shifts younger after a new product launch | Income stability stops predicting repayment the way it used to |
| Detectable without outcomes? | Yes — compare input distributions | No — needs actual outcomes |
| Severity | Often manageable | More serious — the learned logic is now wrong |
Data drift is easier to catch because you can monitor it immediately — compare the distribution of incoming applications against the training distribution and flag divergence.
Concept drift is harder because confirming it requires knowing what actually happened. Which brings us to the core difficulty in credit.
The Feedback Delay Problem
In fraud detection, you learn within days whether a flagged transaction was genuinely fraudulent. Feedback is fast, so drift surfaces quickly.
In credit, a loan approved today produces a default outcome in twelve to twenty-four months. So a model that started drifting in month one is not provably drifting until month eighteen — by which time eighteen months of decisions were made on degraded logic.
| Application | Feedback delay | Drift detection |
|---|---|---|
| Fraud detection | Days | Fast |
| Collections propensity | Weeks | Reasonably fast |
| Churn prediction | Months | Slow |
| Credit default | 12–24 months | Very slow |
This is why leading indicators matter in lending — early delinquency at 30 days, input distribution shifts, score distribution changes — rather than waiting for definitive default data.
What Causes Drift
- Economic change. The pandemic altered repayment, spending and employment patterns in ways no pre-2020 model anticipated.
- Portfolio change. Entering a new segment or geography changes who applies.
- Competitive change. A competitor tightening criteria sends you applicants you did not previously see.
- Product change. New terms attract a different customer profile.
- Regulatory change. New rules alter both behaviour and available data.
- Adversarial adaptation. In fraud, attackers deliberately change behaviour to defeat the model — drift here is not passive.
- Upstream data change. A vendor changes how a field is calculated and the model silently receives different inputs.
That last cause is underrated and common. Nobody tells the model owner that a data provider changed a definition.
The Self-Reinforcing Trap
Credit models suffer a particular version of this worth understanding.
A model approves applicants it scores well. Only approved applicants generate outcome data. Retraining therefore uses data from a population the model itself selected. If the model has developed a blind spot, retraining on its own filtered output reinforces rather than corrects it.
Partial mitigations exist — reject inference, or approving a small random sample below cut-off to generate unbiased outcomes — but the problem is structural rather than fully solvable.
How Drift Is Detected
| Monitor | What it catches | Speed |
|---|---|---|
| Input distribution comparison | Data drift | Immediate |
| Score distribution shift | Changing model behaviour | Immediate |
| Early delinquency rates | Leading indicator of credit drift | 1–3 months |
| Population stability index | Standard drift measure in credit | Immediate |
| Performance against outcomes | Definitive answer | Slow but conclusive |
| Champion-challenger testing | Whether a newer model would do better | Ongoing |
Good practice combines fast leading indicators with slower definitive ones. Relying only on the definitive measure means learning about the problem far too late.
The Regulatory Angle
RBI’s FREE-AI framework expects institutions to maintain model inventories and ongoing monitoring rather than validating once at deployment. Drift is the reason that distinction matters.
A model validated thoroughly in 2024 and never monitored since is not a validated model — it is a model that was validated once, making decisions today on relationships that may no longer hold.
What Good Practice Looks Like
- Define monitoring before deployment, not after something goes wrong
- Track input distributions continuously — the fastest available signal
- Use leading indicators where outcome feedback is slow
- Set retraining triggers in advance rather than deciding case by case
- Watch for upstream data changes from vendors and internal systems
- Keep a champion-challenger process running
- Document the monitoring so it can be evidenced to an auditor
Key Takeaways
- Model drift is silent degradation as learned relationships stop holding
- No error appears — the model keeps producing confident, well-formed output
- Data drift changes inputs; concept drift changes the input-outcome relationship
- Data drift is detectable immediately; concept drift needs outcomes
- Credit has a 12–24 month feedback delay, so leading indicators are essential
- Retraining on self-selected data can reinforce blind spots
- Fraud drift is adversarial — attackers actively cause it
- One-time validation is not validation — monitoring is the requirement
Frequently Asked Questions (FAQ)
Q: What is model drift in machine learning?
Model drift is the gradual loss of accuracy that occurs when the patterns a model learned during training no longer reflect the real world. The model continues producing confident predictions, so the degradation is invisible without deliberate monitoring.
Q: What is the difference between data drift and concept drift?
Data drift means the inputs the model receives have changed — a different applicant mix, for example. Concept drift means the relationship between inputs and outcomes has changed, so the model’s learned logic is now wrong. Concept drift is more serious and harder to detect.
Q: How do you detect model drift?
By monitoring input distributions against training data, tracking score distribution shifts, watching leading indicators such as early delinquency, and eventually comparing predictions against actual outcomes. Fast indicators matter most where outcome feedback is slow.
Q: Why is drift hard to detect in credit models?
Because default outcomes take twelve to twenty-four months to materialise. A model that began drifting in month one cannot be proven to have drifted until well over a year later, by which time many decisions were made on degraded logic. Leading indicators partially compensate.
Q: How often should models be retrained?
It depends on how fast the environment changes. Fraud models may need frequent retraining because attackers adapt deliberately. Credit models change more slowly. Rather than a fixed schedule, good practice sets monitoring thresholds that trigger retraining when drift is detected.
Q: Can retraining make a model worse?
It can. In credit, models retrain on data from applicants the model itself approved, so an existing blind spot gets reinforced rather than corrected. Reject inference and small random approvals below cut-off partially address this, but the issue is structural.
Q: Does RBI require model monitoring?
The FREE-AI framework expects institutions to maintain model inventories and monitor models on an ongoing basis rather than validating only at deployment. Drift is precisely why continuous monitoring rather than one-time validation is the expectation.
Related Reading: