Ask a language model for the tax audit threshold under Section 44AB and you may get a clean, confident, well-formatted answer with a specific figure and a section reference. It may also be entirely wrong. Nothing in the output distinguishes the two cases — and that is the whole problem.
What Hallucination Is
Hallucination is when an AI model generates output that is fluent and plausible but factually incorrect or fabricated.
The term is somewhat misleading. The model is not malfunctioning. It is doing exactly what it was built to do — producing the most likely continuation of text. It simply has no mechanism that distinguishes “likely because true” from “likely because it sounds like the sort of thing that would be true”.
Why It Happens
An LLM is optimised for plausibility, not accuracy. Four structural causes follow:
| Cause | What happens |
|---|---|
| No truth mechanism | The model has no internal fact-checker to consult |
| Trained to always answer | Producing text is the objective; declining is not rewarded |
| Fixed knowledge | Anything after training is unknown, but the model does not know what it does not know |
| Pattern completion | “Section ___ of the Income-tax Act” has a strong pattern; the model fills the blank plausibly |
That last row explains why fabricated citations are so common. Section numbers, case references, circular numbers and IFSC-style codes all have recognisable formats. The model reproduces the format convincingly while the specific value is invented.
Why Finance Is the Worst Place for It
| Property of finance | Consequence |
|---|---|
| Precision matters | A threshold that is wrong by one digit is wrong entirely |
| Answers look verifiable | A confident figure invites trust rather than checking |
| Errors compound | A wrong input flows through a model, a filing, a decision |
| Regulatory consequence | Advice based on a fabricated provision creates real liability |
| Non-experts are the users | The people most likely to ask are least able to spot the error |
That final row is the sharpest point. Someone who already knows the correct audit threshold will notice a wrong one immediately. Someone asking because they do not know has no defence.
Where It Shows Up Most
- Specific numbers — tax thresholds, deduction limits, interest rates, dates
- Legal and regulatory references — section numbers, circular references, rule citations
- Current data — NAV, share prices, index levels, which the model cannot know
- Arithmetic — models predict plausible digits rather than calculating
- Company specifics — financial figures, management names, event dates
- Attribution — quotes and studies confidently attributed to sources that never said them
What Reduces It
| Control | Effect |
|---|---|
| Supply the source document | Strongest single control — the model works from real text rather than memory |
| What is an LLM? | Retrieves relevant documents automatically; grounds answers and enables citation |
| Ask for sources | Helps, but the sources themselves can be fabricated — verify them |
| Ask the same question twice | Inconsistent answers signal the model is guessing |
| Explicit permission to decline | Instructing the model to say when it does not know reduces confident invention |
| Human review | The only reliable control for anything consequential |
Note the third row carefully. Asking “what is your source?” produces a source-shaped answer, which may be as fabricated as the original claim. Fabricated citations are among the most common failure modes precisely because people ask for them as a check.
What Does Not Work
- Asking the model if it is sure. It will express confidence either way. Self-reported certainty is not calibrated.
- Assuming newer models solved it. Rates have improved; the failure mode has not been eliminated.
- Using a second model to check the first. It may agree confidently and wrongly.
- Trusting formatting. Clean structure, headings and confident tone correlate with nothing.
A Working Rule
Sort tasks by what a wrong answer costs.
| Task | Hallucination risk | Approach |
|---|---|---|
| Summarising a document you supplied | Low | Skim to confirm nothing invented |
| Rewriting or reformatting your text | Low | Normal review |
| Explaining a well-known concept | Moderate | Verify specifics |
| Any specific number or threshold | High | Verify against the primary source, always |
| Legal or regulatory citation | Very high | Verify the provision exists and says that |
| Current market data | Very high | Do not use — the model cannot know |
The practical formulation: use AI to draft and structure, not to supply facts you cannot check.
The Governance Dimension
For regulated entities this is not just a quality issue. Under RBI’s FREE-AI framework, accountability for outcomes stays with the institution — including outcomes produced by a vendor’s model.
A customer given incorrect information by an AI assistant has been given incorrect information by the bank. That framing shapes what controls are needed: human review before AI output reaches a customer, logging of what was said, and clear escalation where the model is uncertain.
Key Takeaways
- Hallucination is fluent, confident, fabricated output — not a malfunction but a design consequence
- Models optimise for plausible, not true, and have no internal fact-checker
- Fabricated citations are common because reference formats are easy to imitate
- Finance is the worst domain for it — precision matters and non-experts are the users
- Supplying the source document is the strongest control
- Asking for sources does not work — the sources can be fabricated too
- Self-reported confidence is not calibrated and tells you nothing
- Use AI to draft and structure, not to supply facts you cannot verify
Frequently Asked Questions (FAQ)
Q: What is AI hallucination?
Hallucination is when an AI model produces output that reads as fluent and confident but is factually wrong or entirely invented. It happens because the model is built to generate plausible text, not to verify truth, and it has no mechanism for knowing when it lacks the information.
Q: Why do AI models make up section numbers and citations?
Because reference formats follow strong patterns. “Section ___ of the Income-tax Act” is a highly predictable structure, so the model fills the blank with something format-appropriate. The format is reproduced convincingly while the specific value is invented.
Q: Can I stop hallucination by asking the model for its sources?
Not reliably. The model can fabricate sources as readily as it fabricates facts, producing citations to documents that do not exist or that say something different. If you ask for sources, you must verify the sources themselves.
Q: Do newer AI models still hallucinate?
Yes. Rates have improved substantially with better training and grounding techniques, but the failure mode is structural rather than a bug to be fixed. Any model generating text from learned patterns can produce plausible-sounding fabrication.
Q: What is the best way to reduce hallucination?
Supply the source material directly rather than relying on the model’s memory. When the relevant text is in front of it, the model summarises rather than recalls, which dramatically reduces invention. Retrieval-augmented generation automates this.
Q: Can I ask the model whether it is confident?
You can, but the answer is not informative. Models express confidence in both correct and fabricated output, and self-reported certainty is not calibrated to actual accuracy. Confident phrasing tells you nothing about reliability.
Q: Is it safe to use AI for financial research?
For drafting, structuring and summarising documents you supply, yes. For obtaining specific numbers, thresholds, rates or regulatory citations, treat every output as unverified until checked against the primary source. Use it to organise information rather than to originate facts.
Related Reading: