An LLM asked about your company’s leave policy will produce a confident, well-structured answer describing a policy it invented. RAG solves this by doing something unglamorous: finding the actual policy document and putting it in front of the model before it answers.
What RAG Is
Retrieval Augmented Generation (RAG) is an architecture that retrieves relevant documents from a source you control and supplies them to a language model alongside the user’s question.
The model then answers from the supplied text rather than from its training memory. This changes the task from recall — which invites fabrication — to reading comprehension, which models do far more reliably.
How It Works
| Step | What happens |
|---|---|
| 1. Prepare | Documents are split into chunks and converted to embeddings — numerical representations of meaning |
| 2. Store | Embeddings are indexed in a vector database |
| 3. Query | The user’s question is converted to an embedding the same way |
| 4. Retrieve | The system finds chunks whose embeddings are closest in meaning |
| 5. Augment | Retrieved text is inserted into the prompt with the question |
| 6. Generate | The model answers using the supplied text, and can cite it |
The key idea is semantic search rather than keyword search. A question about “notice period for resignation” retrieves a passage about “employee separation timelines” because the embeddings are close in meaning, even though no words match.
RAG vs Fine-Tuning
These are constantly confused and solve different problems.
| RAG | Fine-tuning | |
|---|---|---|
| Changes | What the model can see | How the model writes |
| Handles new information | Immediately — add the document | No — requires retraining |
| Reduces hallucination | Substantially | Marginally |
| Can cite sources | Yes | No |
| Cost to update | Low | High |
| Best for | Answering from a body of documents | Adopting a house style, tone or format |
The rule of thumb: RAG changes what the model knows, fine-tuning changes how it sounds. If your problem is that the model gives wrong answers about your circulars, fine-tuning will not fix it.
Why This Matters in Finance
Financial institutions have exactly the problem RAG addresses — large bodies of authoritative text that change regularly:
- RBI master directions and circulars
- Internal credit and operations policy
- Product terms and conditions
- Compliance manuals
- Contracts and agreements
An assistant answering staff or customer questions from these documents is far safer than one answering from training memory. It can also cite the source, which makes the answer checkable — important given RBI’s expectations around explainability and accountability.
It also addresses a data governance point. With RAG, your policy documents are supplied at query time rather than absorbed into model weights, which keeps a clearer boundary around where your data sits.
Where RAG Fails
RAG reduces hallucination substantially. It does not eliminate it, and the failure modes are worth knowing.
| Failure | What goes wrong |
|---|---|
| Retrieval misses | The right passage is not found, so the model answers from memory anyway |
| Chunking splits meaning | A rule and its exception land in different chunks; one is retrieved |
| Stale documents | RAG faithfully retrieves an outdated circular and answers confidently from it |
| Contradictory sources | Two versions exist and the model reconciles them incorrectly |
| Answer beyond the text | The model extrapolates past what the document says |
| Table and structure loss | Threshold tables lose meaning when flattened to plain text |
The third row is the most dangerous in a regulated context. RAG makes the model faithful to your document set — which is only an improvement if that set is current. A superseded circular retrieved accurately produces a confident, well-cited, wrong answer.
The last row matters for finance specifically. Threshold tables — audit limits, slab rates, fee schedules — frequently lose their row-column relationships during chunking, which is precisely the content where precision is essential.
What Makes RAG Work Well
- Curate the document set. Retrieval quality is bounded by source quality. Superseded documents must be removed, not just deprioritised.
- Chunk thoughtfully. Respect document structure so a provision stays with its exception.
- Preserve tables. Handle structured content separately from prose.
- Always show sources. The citation is what makes the answer verifiable.
- Instruct the model to decline. If retrieval returns nothing relevant, saying so beats guessing.
- Test retrieval separately. Most RAG failures are retrieval failures, not generation failures.
That final point is worth emphasising. When a RAG system gives a bad answer, the instinct is to blame the model. Usually the model answered reasonably from the wrong passage.
Key Takeaways
- RAG retrieves real documents and supplies them to the model before it answers
- It converts the task from recall to reading comprehension
- Uses semantic search via embeddings, not keyword matching
- RAG changes what the model knows; fine-tuning changes how it sounds
- Well suited to circulars, policies, product terms and compliance manuals
- It reduces but does not eliminate hallucination
- A stale document set produces confident wrong answers — curation is the real work
- Most RAG failures are retrieval failures; test that layer separately
Frequently Asked Questions (FAQ)
Q: What is RAG in AI?
Retrieval Augmented Generation is an architecture where relevant documents are retrieved from a source you control and supplied to a language model along with the question. The model answers from that text rather than from its training memory, which substantially reduces fabrication.
Q: How does RAG reduce hallucination?
By changing the task. Instead of recalling facts from training — where the model fills gaps with plausible invention — it reads supplied text and summarises it. Models are considerably more reliable at reading comprehension than at recall.
Q: What is the difference between RAG and fine-tuning?
RAG changes what information the model can see and handles new documents immediately. Fine-tuning further trains the model, changing its style and vocabulary but not keeping its knowledge current. For answering from a document set, RAG is the appropriate approach.
Q: Does RAG completely stop AI from making things up?
No. If retrieval misses the relevant passage, the model may answer from memory anyway. It can also extrapolate beyond what the document says, or answer confidently from a superseded document. RAG reduces the problem substantially rather than removing it.
Q: What is a vector database and why does RAG need one?
A vector database stores embeddings — numerical representations of meaning — and finds the closest matches to a query embedding. This enables semantic search, so a question retrieves relevant passages even when the wording differs entirely.
Q: Is RAG suitable for regulatory documents?
It is well suited to them, provided the document set is actively curated. RAG makes the model faithful to whatever you give it, so superseded circulars must be removed rather than simply deprioritised, or you get accurate answers from outdated rules.
Q: Why do RAG systems give wrong answers?
Most often because retrieval returned the wrong passage rather than because the model reasoned badly. Poor chunking, stale documents and lost table structure are common causes. Testing the retrieval layer separately from generation is what diagnoses this.
Related Reading: