Retrieval-augmented generation (RAG)
RAG, or retrieval-augmented generation, is a technique that grounds an AI answer in your own content. The system searches a source for passages relevant to the question, passes them to the model alongside the question, and the model answers from what it was given.
RAG became the default because the alternative was worse. A model answering from training data alone has no access to your policies, cannot know anything recent, and invents freely when it has nothing relevant. RAG is not a knowledge base and it is not a substitute for system access. It is a way of putting the right text in front of the model at the right moment.
How does RAG work?
RAG runs in two stages, and almost all of the quality comes from the first. Retrieval searches your content for passages matching the question, usually by comparing meaning rather than keywords. Generation passes those passages to the model along with the question, with an instruction to answer from the supplied text. The pipeline underneath has more moving parts than that summary suggests:
Chunking: splitting documents into passages small enough to retrieve precisely
Embedding: converting passages into vectors that capture meaning
Search: finding the closest matches to the question
Re-ranking: reordering candidates so the most useful passage reaches the model
Generation: producing the answer from what was retrieved
Chunking decides more than teams expect. A policy split across two passages may be retrieved as half a rule, and the model will answer confidently from the half it received.
Where does RAG fail?
Retrieval failures are quieter than generation failures. When a model invents something, the answer often looks wrong. When retrieval returns the wrong passage, the model answers fluently and accurately from content that did not apply. Four failure modes recur:
Nothing relevant exists. The content was never written, so retrieval returns the closest thing it can find
The wrong passage wins. Two policies look similar, and the retrieved one is not the one that applies
The passage is incomplete. Chunking split a rule, and half of it arrives
The model drifts. The answer goes beyond what the retrieved text supports
Each needs a different fix, which is why measuring retrieval separately from generation matters. A system judged only on its final answers cannot tell you which half is broken.
Why is RAG not enough for financial services?
RAG retrieves text. Most financial services questions need data, and no amount of retrieval quality closes that gap.
A customer asking whether their payment cleared cannot be answered from any document. Answering needs the payment record, its current status, and the policy that applies to that status, combined. RAG supplies the third and has no route to the first two.
That is the difference between a chatbot and an agent. RAG is one component inside an agent, sitting alongside tool access, permissions, and guardrails, and on its own it caps you at questions a well-written help centre could already answer. Our blog on why your knowledge base is not enough goes further into this.
Frequently asked questions
What is the difference between RAG and fine-tuning?
RAG supplies knowledge at the moment of the question, while fine-tuning changes the model's behaviour through training. Use RAG for facts that change, because updating content is faster and cheaper than retraining. Use fine-tuning for consistent style, format, or task-specific behaviour.
Does RAG stop hallucination?
It reduces it rather than stopping it. A grounded model has relevant text to work from, so it invents less. It can still answer beyond what the retrieved passage supports, which is why a separate check comparing the reply against its source belongs in production systems.
How much content do you need for RAG to work?
Quality matters more than volume. A small, accurate, current content set outperforms a large one full of superseded policies, because retrieval has no way of telling which version is authoritative. The usual blocker is missing content rather than insufficient content.
Is RAG still relevant with larger context windows?
Yes, for most production uses. Longer context windows let you supply more text without making retrieval unnecessary, because putting an entire policy library into every request is slow, expensive, and dilutes the relevant passage. Retrieval is still doing the useful work.
Learn more