Hallucination
An AI hallucination is an output presented confidently that turns out not to be true. Hallucination is a property of how language models work rather than a bug someone can patch out. The model has no way to flag what it doesn't know, so a gap in its training produces an answer that sounds exactly like a fact. Grounding reduces the rate substantially and no technique removes it, which is why production systems check the output rather than trusting the model.
Why do AI hallucinations happen?
A language model predicts the next piece of text from patterns in its training data. Nothing in that process distinguishes a fact it absorbed from a plausible-sounding pattern it assembled, and the model has no mechanism for reporting the difference. Three conditions raise the rate:
Gaps in what the model was given. With nothing relevant to draw on, it produces the shape of an answer
Pressure to answer. A system treating every question as answerable will fill gaps rather than decline
Specificity. Names, numbers, dates, and references invite invention, because they are exactly what a plausible pattern can generate
The last one matters most in financial services, where the specific detail is usually the point. A model asked about a transfer limit will produce a number, and that number will look exactly like the correct one.
How do you reduce AI hallucinations?
The main lever is grounding: giving the model the relevant content and constraining it to answer from that. Retrieval-augmented generation is the common implementation, and it changes the failure mode rather than eliminating it. A grounded system fails by retrieving the wrong passage, or by drifting beyond the passage it retrieved. What production systems layer on top:
A grounding check comparing the reply against its retrieved source before sending
Permission to decline. A system allowed to say it does not know hallucinates less than one that must always answer
Identifier checks on account numbers, references, and amounts, which are the highest-risk inventions
Human review for cases where an error is expensive or irreversible
What do AI hallucinations mean for financial services?
A hallucination in a support conversation is not only a quality problem. Depending on what the model invented, it can create a liability: an invented fee, a promised refund, or a rate that does not exist may bind the firm or create a complaint with a clock attached.
Two cases are worth separating. The invented fact is the obvious one. The subtler one is the true statement that should never have been made, where nothing was hallucinated and the reply was still wrong to send. Grounding catches the first and does nothing about the second.
Firms using AI remain responsible for complying with FCA rules, so the question is not whether the model can hallucinate but what guardrails catch it when it does. Our guide on evaluating AI agents covers how to test for them.
How do you measure hallucination rates?
Published rates deserve scepticism, because the number depends almost entirely on the test set behind it. A rate measured on the easy cases your knowledge base already answers well won't tell you anything about the harder queries in your actual queue. A more useful approach:
Test on your own hard cases, pulled from real conversations rather than a scripted scenario
Separate grounded from ungrounded errors. Retrieving the wrong thing and inventing a thing need different fixes
Count what reached the customer, rather than what the model produced, since checks sit between the two
Track refusals as well. A system that never hallucinates because it answers nothing is not working either
Any vendor's published rate is a starting question rather than an answer.
Frequently asked questions
Can AI hallucinations be eliminated completely?
No. Grounding, checking, and constraining reduce the rate substantially, and none of them removes the possibility. Systems designed on the assumption that hallucination is solved tend to lack the checks that catch it, which makes the rare failure worse when it arrives.
Does retrieval-augmented generation stop hallucination?
It reduces it and changes its shape. A grounded model has relevant content to work from, so it invents less. It can still retrieve the wrong passage, or answer beyond what that passage supports, which is why a separate grounding check on the output earns its place.
Are some models more prone to hallucination than others?
Yes, and the difference matters less than the system around the model. A more accurate model with no grounding check will still send an invented answer to a customer, while a less accurate model behind proper checks will not.
What should an AI agent do when it does not know?
Say so, and hand over. Declining is the correct behaviour when the content source has no answer, and a system treating every question as answerable will fill the gap instead. Worth testing explicitly in any evaluation, because vendors rarely demonstrate it.