The situation
A city council put a resident-services assistant on its website four months ago. It answers questions about bin collections, parking permits, rates and planning applications. It calls a foundation model on Amazon Bedrock, and it answers from the council’s own published pages, indexed into a knowledge base and searched on every question.
Three complaints have arrived in a fortnight, and they have been passed to the team that built it.
A resident asked when garden waste is collected and was told it goes out fortnightly, on the alternate week to recycling. The council collects garden waste weekly from March to October and monthly through the winter. There is no fortnightly anything. The resident left a full bin at the kerb for three weeks. A second resident asked what a bulky waste collection costs and was told AUD$35. The fee went to AUD$48 in April 2024, and the assistant stated the old number flatly, with nothing attached to it. A third asked what happens when a crew misses a street, and was told the collection is re-attempted within 24 hours, cited to the Household Waste Collection Policy, section 4. That document exists, that section exists, and what it says is that a missed collection is re-attempted by the end of the next working day.
The team has a meeting on Thursday to decide what to add. Four things have been proposed already: better prompts, a guardrail, a human reading every answer, and switching to a larger model.
What actually matters
Start with what the assistant is doing when it produces a wrong sentence, because three of those four proposals assume something about the mechanism that is not true.
A language model produces text by predicting a likely continuation of the text in front of it, one token at a time, against patterns learned from an enormous training corpus. Its training objective is next-token prediction. Nothing in that process compares a finished sentence against a source. Fortnightly, on the alternate week to recycling is an extremely plausible thing for a council website to say. Plausible is the property the model scores against. True is not, and no separate step inside the model supplies it. If the mechanism is unfamiliar, how a model turns a prompt into the next word covers it properly.
That is a hallucination: a confident, fluent statement that is not so, produced in exactly the same voice as the true sentences around it. The model gives no signal separating the two. It cannot, because internally there is no difference between them.
Now put the three complaints next to each other, because they are not one problem.
The garden waste answer is invention about a fact the council publishes. The collection calendar is on the website. The model was not handed it, so it wrote something calendar-shaped. That fault closes by putting the source text in front of the model before it answers.
The AUD$35 is different. It is a fact that used to be true. Either the model recited it from training data, or retrieval returned an archived fees page nobody removed from the index. Handing the model documents does not fix this, because the wrong number is in a document. Fees, dates, band thresholds and permit prices change on a schedule and live in a system that owns them, and a system that owns a number can be asked for it directly.
The missed-collection answer is the one that should worry the team most, because everything worked and the answer was still wrong. The right document was retrieved. The citation is genuine. The model read by the end of the next working day and wrote within 24 hours, which is close enough to sound like a paraphrase and different enough to matter on a Friday. Grounding put the passage there. Nothing checked that the answer followed from it.
So the properties that separate the options are: which of those three faults a technique actually catches, whether it acts before the answer exists or checks it afterwards, what it needs to be given, what it adds to every reply in latency and cost, and whether a person has to be in the path.
The AI Practitioner material files all of this under one heading, hallucination detection methods and grounding techniques to improve output accuracy, and gives three examples: Retrieval Augmented Generation [RAG] grounding, output validation, and confidence scoring. Those three plus citations and human review are the whole toolkit here, and they divide the work between them rather than competing for it.
What we’ll filter on
- Which fault does it catch? Invention about a published fact, a fact that has since changed, or a claim that does not follow from the source it cites.
- Before or after? Does it change what the model is given, or check what the model produced?
- What does it need? Retrieved passages, a system of record, a threshold somebody has to set, or a person.
- What does it add to every answer? Latency and cost per reply, which decides whether it can run on all of them or only some.
- Does a person have to see the reply first? That decides throughput and staffing, so it has to be reserved for the answers that warrant it.
The landscape
Retrieval Augmented Generation [RAG] grounding. Before the model answers, search the council’s own material for passages relevant to the question, and put those passages into the prompt with an instruction to answer from them. The model is then continuing text that already contains the answer, which is a very different task from continuing text that does not. This is the technique with the largest single effect on invention, and it is already how an assistant answers from the documents an organisation already has rather than from training data. Two limits define its reach. It reduces invention about facts the indexed corpus contains, and does nothing whatever about facts it does not contain, where the model is back to writing something plausible. And a passage in the prompt is an input, not a constraint. The model can still summarise it wrongly.
Citations. Render, next to each claim, which document and which passage it came from. A citation does not stop a hallucination; it turns an unverifiable answer into a checkable one, and reduces checking from re-researching the question to reading one paragraph. It also changes the failure mode: the third complaint arrived because a resident clicked the citation and found it said something else, which is the system working. Making every claim traceable is a design decision taken at build time, and a retrieval system can be built to require one on every sentence.
The Amazon Bedrock Guardrails contextual grounding check. A managed check that takes three things: the passages supplied as the grounding source, the resident’s question, and the model’s response. It returns a confidence score for grounding, meaning how far the response is supported by those passages, and one for relevance, meaning how far it answers what was asked. Each threshold is set between 0 and 0.99, and responses scoring below it are blocked and replaced with the guardrail’s configured message. AWS documents the supported use cases as summarisation, paraphrasing and question answering, and says multi-turn conversational chatbots are not among them, so a single question answered from retrieved passages is in scope and a running conversation is not. It sits alongside the other filters in a single guardrail configuration. It runs on the response only, and needs the grounding source and the question passed in with it, so it applies to answers that had a source in the first place. Guardrails also has Automated Reasoning checks, which validate a response against rules extracted from a policy document you supply; those return findings rather than blocking, so they are a verification layer to act on rather than a filter.
Output validation. Checking the shape and the content of an answer in your own code, before it reaches a resident. Three kinds do most of the work. A schema check, where the model is asked for structured output and the application rejects a reply that does not parse or is missing a field. A range or format check, where a date must be a real date and a fee must be a positive number under a sane ceiling. And a lookup against a system of record, where any figure the reply contains is compared with the authoritative table and the reply is rejected if it disagrees. This is ordinary deterministic code, it runs in milliseconds, and it is exactly as good as the rules somebody wrote. It is also the only technique in this list that catches the AUD$35, because AUD$35 is a well-formed, plausible, correctly-cited-if-you-kept-the-old-page fee that is simply no longer the fee.
Confidence scoring. Two different things share this name and only one of them is a number you can act on. Purpose-built AWS AI services return a confidence value from a classifier: Amazon Comprehend attaches a score from 0 to 1 to each entity and each sentiment it detects, and Amazon Textract attaches one from 0 to 100 to every block it lifts off a scanned form, form fields included. Those scores come from the model that made the detection, and thresholding them is sound engineering, which is how a form-processing step routes low-scoring fields to a human reviewer. A number a generative model states about its own answer is not that. I am 95% confident is text the model generated because it was a likely continuation, and a model that has just produced a wrong bin rule will produce a high confidence figure next to it. Threshold the classifier scores, and treat the model’s self-report as prose.
Human-in-the-loop evaluation. A person reading answers and judging them against what the source says. It comes in two shapes and they solve different problems. Sampling, where somebody reviews a fixed number of transcripts each week against the published pages, tells you the rate at which the assistant is wrong and whether last month’s change helped. Escalation, where a defined class of question never reaches the resident without a person seeing it first, protects the answers where a wrong reply costs somebody money or loses them a legal right. Sampling is a measurement; escalation is a control. Neither scales to every reply, so escalation has to be defined by the class of question rather than by how confident anything looks.
Evaluation
Side by side
The three complaints as three columns, and each technique against them.
| Technique | Invented bin rule | Fee that has changed | Claim the source does not support | Before or after | Cost per answer | Needs a person |
|---|---|---|---|---|---|---|
| RAG grounding | ✓ | ✗ | ✗ | Before | A search plus a longer prompt | ✗ |
| Citations | ✗ | ✗ | ✓ once read | Before, checked after | Negligible | ✓ to be any use |
| Guardrails contextual grounding check | partly | ✗ | ✓ | After | One extra scored call | ✗ |
| Output validation | ✗ | ✓ | ✗ | After | A lookup, milliseconds | ✗ |
| Confidence scoring | ✗ | ✗ | ✗ | After | Negligible | ✗ |
| Human-in-the-loop evaluation | ✓ | ✓ | ✓ | After | Minutes to days | ✓ |
Read down the middle column first. One technique catches the stale fee without a person, and it is the boring one: a lookup in code against the fees table. No amount of grounding, guardrail configuration or model upgrade helps, because the wrong figure is a perfectly plausible fee and there is a document somewhere that still carries it. Numbers that change belong to a system of record, and the assistant’s job is to fetch them rather than to say them.
Read the confidence scoring row and notice it catches nothing here. That row is on the table because the technique gets reached for in exactly this situation and does not apply: there is no classifier in this pipeline producing a score, and the model’s own stated confidence tracks fluency rather than truth.
Read the bottom row and the shape of the answer appears. A person catches all three and cannot read every reply, so the design problem is choosing which questions go to a person, and that choice is made by subject rather than by any score.
What to add for a given question
The chain sorts by the kind of fact rather than by the kind of complaint, which is what makes it usable on a question nobody has seen yet. Every gate is answerable by a person who knows the council’s material and nothing about machine learning.
The solution
The council assembles five things into one request path, and each of them is there for a fault the others miss.
Retrieval runs over the published pages, and the index gets an owner. Superseded pages are removed rather than left in place, because an archived fees page in the index puts a wrong fact into the prompt and the model then repeats it. Removing that one page would have avoided one of the three complaints.
The reply carries citations, rendered as a link to the page and the section each claim came from. This is what a resident is given to check, and it is also what the review sampling reads.
The Guardrails contextual grounding check runs on every answer that had retrieved passages, with a threshold on both the grounding score and the relevance score. Below either one, the assistant returns its configured message and offers a phone number instead of an answer. The council starts the thresholds low, watches how often the guardrail fires for a fortnight, and raises them once it can see how often a good answer is being blocked. Set near the 0.99 ceiling on day one, the guardrail blocks almost everything, and the assistant gets switched off by the end of the week.
Output validation runs in the application code after the model and before the resident. Any figure in the reply is matched against the fees table, and a mismatch means the reply is discarded and the figure is rendered from the table. Any date is checked for being a real date in a sensible range. Anything that fails is not repaired by asking the model again; it is replaced with the value the system of record holds.
Escalation is defined by subject. Statutory deadlines, appeal rights, anything about non-collection of clinical waste, and anything a resident is charged for go to a person before the reply is sent. Everything else answers directly. On this council’s volumes that is around four per cent of questions, which one officer absorbs alongside existing work.
Behind all of it, human-in-the-loop evaluation as sampling: twenty transcripts a week, read against the pages they cite, scored right or wrong, with the wrong ones logged by which of the three faults caused them. That gives the team a rate to watch and, more usefully, tells them which of the five controls to spend the next fortnight on. Measured this way, grounding stops being an assumption; a professional-level treatment of how a team scores hallucination in a running system goes considerably further.
Worked example
The three complaints, run through the assembled path.
Garden waste. Retrieval returns the collection calendar page. The passage in the prompt says weekly March to October, monthly November to February. The model answers from it, the reply cites the calendar page, and the grounding score is high because every clause traces to the passage. No invention, because the model was not asked to supply a fact it did not have. Caught by grounding, before the answer existed.
Bulky waste fee. Retrieval returns the current fees page, but suppose it also returns the archived one and the model writes AUD$35. Output validation reads AUD$35 out of the draft reply, queries the fees table, gets AUD$48, discards the reply and renders the figure from the table. The resident sees AUD$48 with a link to the fees page. Caught by code, after the answer existed, and the model’s version of the fee never reaches anybody.
Missed collection. Retrieval returns the Household Waste Collection Policy. The model writes within 24 hours, cited to section 4. The contextual grounding check scores the response against that passage, the paraphrase does not follow from by the end of the next working day, the grounding score falls below the threshold, and the guardrail intervenes. The resident gets the configured message and a link to section 4 rather than a wrong deadline. Caught by the check, and the citation that exposed it originally is what the check reads.
Three faults, three different controls, none of which would have caught the other two.
What’s worth remembering
- A hallucination is a fluent, confident statement the model cannot distinguish from a true one, because the model scores plausibility and nothing in it compares a sentence against a source.
- Retrieval Augmented Generation [RAG] grounding reduces invention about facts your corpus contains and does nothing about facts it does not, and a stale document in the index is a wrong answer waiting to be retrieved.
- Citations do not prevent a hallucination; they make one checkable in seconds, which is a different and still useful job.
- The Amazon Bedrock Guardrails contextual grounding check scores a response against the supplied passages and against the question, and blocks it below the thresholds you set anywhere from 0 to 0.99, so it catches an answer that does not follow from a source it correctly cites.
- Output validation in your own code is the only cheap catch for a figure that has changed, because the fix is a lookup against the system of record rather than anything the model can be persuaded to do.
- Confidence scoring means a classifier’s own score, as Amazon Comprehend and Amazon Textract return; a generative model’s stated confidence in itself is generated text, so route to human-in-the-loop evaluation by subject rather than by any number the model produces.