The situation
A district council put a resident-services assistant on its website four months ago. It answers questions about bin collections, parking permits, council tax bands and planning applications. It calls a foundation model on Amazon Bedrock, and it answers from the council’s own published pages, indexed into a knowledge base and searched on every question.
Three complaints have arrived in a fortnight, and they have been passed to the team that built it.
A resident asked when garden waste is collected and was told it goes out fortnightly, on the alternate week to recycling. The council collects garden waste weekly from March to October and monthly through the winter. There is no fortnightly anything. The resident left a full bin at the kerb for three weeks. A second resident asked what a bulky waste collection costs and was told £35. The fee went to £48 in April 2024, and the assistant stated the old number flatly, with nothing attached to it. A third asked what happens when a crew misses a street, and was told the collection is re-attempted within 24 hours, cited to the Household Waste Collection Policy, section 4. That document exists, that section exists, and what it says is that a missed collection is re-attempted by the end of the next working day.
The team has a meeting on Thursday to decide what to add. Four things have been proposed already: better prompts, a guardrail, a human reading every answer, and switching to a larger model.
What actually matters
Start with what the assistant is doing when it produces a wrong sentence, because three of those four proposals assume something about the mechanism that is not true.
A language model produces text by predicting a likely continuation of the text in front of it, one token at a time, against patterns learned from an enormous training corpus. Fluency is what it is optimising for; nothing in that process compares a finished sentence against a source and asks whether it holds. Fortnightly, on the alternate week to recycling is an extremely plausible thing for a council website to say. Plausible is the property the model can score. True is not, and there is no separate step inside the model that supplies it. If the mechanism is unfamiliar, how a model turns a prompt into the next word covers it properly.
That is a hallucination: a confident, fluent statement that is not so, produced in exactly the same voice as the true sentences around it. The model gives no signal separating the two. It cannot, because internally there is no difference between them.
Now put the three complaints next to each other, because they are not one problem.
The garden waste answer is invention about a fact the council publishes. The collection calendar is on the website. The model was not handed it, so it wrote something calendar-shaped. That fault closes by putting the source text in front of the model before it answers.
The £35 is different. It is a fact that used to be true. Either the model recited it from training data, or retrieval returned an archived fees page nobody removed from the index. Handing the model documents does not fix this, because the wrong number is in a document. Fees, dates, band thresholds and permit prices change on a schedule and live in a system that owns them, and a system that owns a number can be asked for it directly.
The missed-collection answer is the one that should worry the team most, because everything worked and the answer was still wrong. The right document was retrieved. The citation is genuine. The model read by the end of the next working day and wrote within 24 hours, which is close enough to sound like a paraphrase and different enough to matter on a Friday. Grounding put the passage there. Nothing checked that the answer followed from it.
So the properties that separate the options are: which of those three faults a technique actually catches, whether it acts before the answer exists or checks it afterwards, what it needs to be given, what it adds to every reply in latency and cost, and whether a person has to be in the path.
The AI Practitioner material files all of this under one heading, hallucination detection methods and grounding techniques to improve output accuracy, and gives three examples: Retrieval Augmented Generation [RAG] grounding, output validation, and confidence scoring. Those three plus citations and human review are the whole toolkit here, and they divide the work between them rather than competing for it.
What we’ll filter on
- Which fault does it catch? Invention about a published fact, a fact that has since changed, or a claim that does not follow from the source it cites.
- Before or after? Does it change what the model is given, or check what the model produced?
- What does it need? Retrieved passages, a system of record, a threshold somebody has to set, or a person.
- What does it add to every answer? Latency and cost per reply, which decides whether it can run on all of them or only some.
- Does a person have to see the reply first? That decides throughput and staffing, so it has to be reserved for the answers that warrant it.
The landscape
Retrieval Augmented Generation [RAG] grounding. Before the model answers, search the council’s own material for passages relevant to the question, and put those passages into the prompt with an instruction to answer from them. The model is then continuing text that already contains the answer, which is a very different task from continuing text that does not. This is the technique with the largest single effect on invention, and it is already how an assistant answers from the documents an organisation already has rather than from training data. Two limits define its reach. It reduces invention about facts the indexed corpus contains, and does nothing whatever about facts it does not contain, where the model is back to writing something plausible. And a passage in the prompt is an input, not a constraint. The model can still summarise it wrongly.
Citations. Render, next to each claim, which document and which passage it came from. A citation does not stop a hallucination; it turns an unverifiable answer into a checkable one, and moves the cost of checking from re-researching the question to reading one paragraph. It also changes the failure mode: the third complaint arrived because a resident clicked the citation and found it said something else, which is the system working. Making every claim traceable is a design decision taken at build time, and a retrieval system can be built to require one on every sentence.
The Amazon Bedrock Guardrails contextual grounding check. A managed check that takes the passages supplied as the grounding source, the resident’s question and the model’s response, and returns two numbers between 0 and 1. A grounding score says how well the response is supported by those passages. A relevance score says how well it answers what was actually asked. Set a threshold on each and the guardrail intervenes below it, returning a configured message instead of the answer. This is the machine check for the third complaint, and the closest thing available to automated hallucination detection at runtime. It sits alongside the other filters in a single guardrail configuration, and it needs retrieval context at invocation time, so it only works on answers that had a source in the first place.
Output validation. Checking the shape and the content of an answer in your own code, before it reaches a resident. Three kinds do most of the work. A schema check, where the model is asked for structured output and the application rejects a reply that does not parse or is missing a field. A range or format check, where a date must be a real date and a fee must be a positive number under a sane ceiling. And a lookup against a system of record, where any figure the reply contains is compared with the authoritative table and the reply is rejected if it disagrees. This is ordinary deterministic code, it costs milliseconds, and it is exactly as good as the rules somebody wrote. It is also the only technique in this list that catches the £35, because £35 is a well-formed, plausible, correctly-cited-if-you-kept-the-old-page fee that is simply no longer the fee.
Confidence scoring. Two different things share this name and only one of them is a number you can act on. Purpose-built AWS AI services return a genuine confidence value from a classifier: Amazon Comprehend gives a score for each entity or sentiment it detects, and Amazon Textract gives one for every field it lifts off a scanned form. Those are calibrated outputs, and thresholding them is sound engineering, which is how a form-processing step decides what to send for checking. A number a generative model states about its own answer is not that. I am 95% confident is text the model generated because it was a likely continuation, and a model hallucinating a bin rule will report high confidence in the bin rule. Threshold the classifier scores, and treat the model’s self-report as prose.
Human-in-the-loop evaluation. A person reading answers and judging them against what the source says. It comes in two shapes and they solve different problems. Sampling, where somebody reviews a fixed number of transcripts each week against the published pages, tells you the rate at which the assistant is wrong and whether last month’s change helped. Escalation, where a defined class of question never reaches the resident without a person seeing it first, protects the answers where being wrong costs somebody money or a legal right. Sampling is a measurement; escalation is a control. Neither scales to every reply, so escalation has to be defined by the class of question rather than by how confident anything looks.
Evaluation
Side by side
The three complaints as three columns, and each technique against them.
| Technique | Invented bin rule | Fee that has changed | Claim the source does not support | Before or after | Cost per answer | Needs a person |
|---|---|---|---|---|---|---|
| RAG grounding | ✓ | ✗ | ✗ | Before | A search plus a longer prompt | ✗ |
| Citations | ✗ | ✗ | ✓ once read | Before, checked after | Negligible | ✓ to be any use |
| Guardrails contextual grounding check | partly | ✗ | ✓ | After | One extra scored call | ✗ |
| Output validation | ✗ | ✓ | ✗ | After | A lookup, milliseconds | ✗ |
| Confidence scoring | ✗ | ✗ | ✗ | After | Negligible | ✗ |
| Human-in-the-loop evaluation | ✓ | ✓ | ✓ | After | Minutes to days | ✓ |
Read down the middle column first. One technique catches the stale fee without a person, and it is the boring one: a lookup in code against the fees table. No amount of grounding, guardrail configuration or model upgrade helps, because the wrong figure is a perfectly plausible fee and there is a document somewhere that agrees with it. Numbers that change belong to a system of record, and the assistant’s job is to fetch them rather than to say them.
Read the confidence scoring row and notice it catches nothing here. That row is on the table because the technique gets reached for in exactly this situation and does not apply: there is no classifier in this pipeline producing a score, and the model’s own stated confidence tracks fluency rather than truth.
Read the bottom row and the shape of the answer appears. A person catches all three and cannot read every reply, so the design problem is choosing which questions go to a person, and that choice is made by subject rather than by any score.
What to add for a given question
The chain sorts by the kind of fact rather than by the kind of complaint, which is what makes it usable on a question nobody has seen yet. Every gate is answerable by a person who knows the council’s material and nothing about machine learning.
The solution
The council assembles five things into one request path, and each of them is there for a fault the others miss.
Retrieval runs over the published pages, and the index gets an owner. Superseded pages are removed rather than left in place, because an archived fees page in a knowledge base is a hallucination the retrieval step performs on the model’s behalf. That change alone would have avoided one of the three complaints and cost nothing to make.
The reply carries citations, rendered as a link to the page and the section each claim came from. This is what a resident is given to check, and it is also what the review sampling reads.
The Guardrails contextual grounding check runs on every answer that had retrieved passages, with a threshold on both the grounding score and the relevance score. Below either one, the assistant returns its configured message and offers a phone number instead of an answer. The council starts the threshold low, watches the intervention rate for a fortnight, and tightens it once it can see how often a good answer is being blocked. Setting it high on day one produces an assistant that refuses to answer anything, which gets it switched off by the end of the week.
Output validation runs in the application code after the model and before the resident. Any figure in the reply is matched against the fees table, and a mismatch means the reply is discarded and the figure is rendered from the table. Any date is checked for being a real date in a sensible range. Anything that fails is not repaired by asking the model again; it is replaced with the value the system of record holds.
Escalation is defined by subject. Statutory deadlines, appeal rights, anything about non-collection of clinical waste, and anything a resident is charged for go to a person before the reply is sent. Everything else answers directly. On this council’s volumes that is around four per cent of questions, which one officer absorbs alongside existing work.
Behind all of it, human-in-the-loop evaluation as sampling: twenty transcripts a week, read against the pages they cite, scored right or wrong, with the wrong ones logged by which of the three faults caused them. That gives the team a rate to watch and, more usefully, tells them which of the five controls to spend the next fortnight on. Measured this way, grounding stops being an assumption; a professional-level treatment of how a team scores hallucination in a running system goes considerably further.
Worked example
The three complaints, run through the assembled path.
Garden waste. Retrieval returns the collection calendar page. The passage in the prompt says weekly March to October, monthly November to February. The model answers from it, the reply cites the calendar page, and the grounding score is high because every clause traces to the passage. No invention, because the model was not asked to supply a fact it did not have. Caught by grounding, before the answer existed.
Bulky waste fee. Retrieval returns the current fees page, but suppose it also returns the archived one and the model writes £35. Output validation reads £35 out of the draft reply, queries the fees table, gets £48, discards the reply and renders the figure from the table. The resident sees £48 with a link to the fees page. Caught by code, after the answer existed, and the model’s opinion about the fee never reaches anybody.
Missed collection. Retrieval returns the Household Waste Collection Policy. The model writes within 24 hours, cited to section 4. The contextual grounding check scores the response against that passage, the paraphrase does not follow from by the end of the next working day, the grounding score falls below the threshold, and the guardrail intervenes. The resident gets the configured message and a link to section 4 rather than a wrong deadline. Caught by the check, and the citation that exposed it originally is what the check reads.
Three faults, three different controls, none of which would have caught the other two.
What’s worth remembering
- A hallucination is a fluent, confident statement the model cannot distinguish from a true one, because the model scores plausibility and nothing in it compares a sentence against a source.
- Retrieval Augmented Generation [RAG] grounding reduces invention about facts your corpus contains and does nothing about facts it does not, and a stale document in the index is a wrong answer waiting to be retrieved.
- Citations do not prevent a hallucination; they make one checkable in seconds, which is a different and still useful job.
- The Amazon Bedrock Guardrails contextual grounding check scores a response against the supplied passages and against the question, and intervenes below the thresholds you set, so it catches an answer that does not follow from a source it correctly cites.
- Output validation in your own code is the only cheap catch for a figure that has changed, because the fix is a lookup against the system of record rather than anything the model can be persuaded to do.
- Confidence scoring means a calibrated score from a classifier such as Amazon Comprehend or Amazon Textract; a generative model’s stated confidence in itself is generated text, so route by subject to human-in-the-loop evaluation rather than by any number the model reports.