The situation
An internal policy assistant runs on Amazon Bedrock over a Knowledge Base of HR and finance documents. Employees ask it things like “how many days of carer’s leave am I entitled to” and “what is the mileage reimbursement rate”, and it retrieves a few passages and generates an answer with citations. Most of the time it is genuinely useful. Perhaps one answer in fifteen is confidently, specifically wrong: a leave figure that appears nowhere in the policy, a reimbursement rate from a document that was superseded two years ago, a crisp paragraph about a benefit the company does not offer.
The team already measures the pipeline end to end and knows the overall answer quality is not where they want it. What they cannot currently do is say why any single bad answer went wrong. Sometimes the retrieved passages plainly did not contain the answer and the model made something up anyway. Sometimes the right passage was sitting in the context and the answer still went past what it said. Those are two different failures with two different fixes. Right now both land in the same “it hallucinated” bucket, so nobody can tell whether to spend the next week on ChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. and retrieval or on the generation prompt and its guardrails.
Underneath the complaint is a measurement problem. Before you can reduce hallucination you have to define it precisely enough to count it, and attribute each instance to the stage that caused it.
What actually matters
The first distinction to get right is between faithfulness and correctness, because they are not the same property and a system can pass one while failing the other. Faithfulness, sometimes called groundedness, asks a narrow question: is every claim in the answer supported by the retrieved context? Correctness asks a broader one: is the answer actually right? An answer can be perfectly faithful to a passage that is out of date, and be wrong in the world. An answer can be right by luck, pulled from the model’s parametric memory rather than the retrieved text, and be unfaithful even though it happens to be correct. Faithfulness is reference-free; you can judge it with only the answer and the passages in front of you. Correctness needs a known-good reference answer to compare against. Conflating them is how teams end up chasing the wrong metric.
The second thing that matters is that a hallucination has two possible origins, and you cannot fix what you cannot locate. If retrieval returned passages that never contained the answer, there was nothing in the context to answer from. The correct output at that point is an abstention, and a fabricated answer is a retrieval failure compounded by a failure to abstain. If retrieval returned a passage that did contain the answer and the model still asserted something the passage did not say, that is a generation failure, and better retrieval will not help. Attribution has to come before the fix. A grounding score of 0.4 tells you the answer is unsupported. It does not tell you whether the support was never retrieved or was retrieved and then not used.
Third, decide whether you are measuring at runtime or offline, because they serve different purposes. A runtime check scores each response as it is produced and can block or flag a low-scoring answer before it reaches the employee. It is a live safety gate, with a latency and cost budget attached to every call. An offline evaluation runs a fixed set of questions on a schedule or before a release, and gives you a trend line and a regression signal. It is where you catch a new embedding model or a reworded prompt making things worse before anyone reports it. You want both, and they use different tooling.
Fourth, the eval set has to include questions the system should not be able to answer. If every question in your set is answerable from the corpus, you never test the behaviour that matters most for hallucination: abstaining when the context is insufficient. A system that always produces a confident answer will score fine on an all-answerable set and fabricate freely in production the moment it meets a question outside the corpus. Known-unanswerable questions, where the correct output is “I do not have that information”, are the ones that expose a system that answers anyway instead of declining.
And finally, whatever automated judge you use is itself a model that can be wrong, so it needs validating against human labels before you trust its numbers. An LLM scoring groundedness is cheaper and faster than a human reviewer and drifts in its own ways; you calibrate it by having people label a sample and checking the judge agrees, then let it scale.
What we’ll filter on
- Faithfulness or correctness, is the method scoring support-by-the-context, or right-in-the-world? They need different inputs.
- Attribution, can it separate a retrieval-caused hallucination from a generation-caused one, or does it collapse both into one score?
- Runtime gate or offline signal, does it block a bad answer live, or track quality across a fixed eval set?
- Ground truth required, does it need reference answers and labelled unanswerable cases, or can it run reference-free?
- Cost and latency, what does each check add per answer, and can the budget carry it on every call?
The landscape
Amazon Bedrock Guardrails contextual grounding check. A guardrail policy that takes three inputs: a grounding source (the retrieved passages), the user query, and the content to guard, which is the model response. It returns a grounding confidence score, how far the response is supported by the source, and a relevance confidence score, how far the response addresses the query. You set a threshold on each, anywhere from 0 to 0.99; a threshold of 1 is rejected, because it would block everything. When a score falls below its threshold the guardrail intervenes and returns your configured blocked message instead of the ungrounded answer. It runs at runtime, either inline on a model call or through the ApplyGuardrail API, so it is a live gate rather than an offline report. The check needs a response to score, so it applies to the output only and never to the prompt. What it does not do on its own is attribute the failure; a low grounding score tells you the answer is unsupported, not whether the support was retrievable. Guardrails also carry an automated reasoning check, which validates a response against logical policy rules you write rather than against the retrieved passages, so it answers a different question and does not place a miss at a stage either.
LLM-as-a-judge groundedness scoring. A second model prompted to read the answer and the retrieved passages and score whether each claim in the answer is supported. This is reference-free like the grounding check, but you own the rubric, so you can ask it for a per-claim verdict, a short rationale, and a citation to the supporting sentence rather than a single number. That granularity is what lets you build attribution: pair it with a separate check on whether the passage even contained the answer, and you can place each miss. The cost is that you are now running and validating another model, and its scores need calibrating against human labels before they carry weight.
Amazon Bedrock evaluations, RAG evaluation job. A managed offline evaluation over a Knowledge Base or your own RAG source, run in retrieve-only or retrieve-and-generate mode. Retrieve-only offers two built-in metrics, context relevance and context coverage, and coverage needs ground-truth passages in the dataset. Retrieve-and-generate adds correctness, completeness, faithfulness, helpfulness, logical coherence, citation precision, citation coverage and a refusal metric that scores how evasive the answers are, plus harmfulness and stereotyping. Faithfulness there measures how far the response avoids hallucination with respect to the retrieved texts, which is the same property the grounding check scores. Because retrieval quality and generation faithfulness come back from the same job, it is the natural place to do attribution at the eval-set level. A question with poor context relevance and low faithfulness points at retrieval; good context relevance with low faithfulness points at generation. The dataset is JSON Lines in S3 and holds up to a thousand prompts per job, and the scoring is an evaluator model, so it is LLM-as-a-judge underneath. You can also define custom metrics with your own judging prompt.
Human review of a labelled sample. People reading answers against the criteria you define, which is the reference standard you calibrate the automated judges against rather than a thing you run on every release. Bedrock’s human-worker evaluation jobs cover model evaluation, not RAG evaluation jobs, so grading a RAG system by hand is a process you run yourself. Slower and more expensive per answer than any judge model.
A labelled eval set with answerable and unanswerable questions. Not a service but the input every other method leans on. A curated set where each question is tagged answerable or unanswerable, answerable ones carry a reference answer and the passage that supports it, and unanswerable ones expect an abstention. This is what turns any of the scoring methods above from a vibe into a number you can track, and it is the only way to measure whether the system abstains when it should.
Evaluation
Side by side
| Method | Measures faithfulness | Measures correctness | Attributes retrieval vs generation | Runtime or offline | Needs reference answers |
|---|---|---|---|---|---|
| Guardrails contextual grounding | ✓ | ✗ | ✗ | Runtime | ✗ |
| LLM-as-a-judge groundedness | ✓ | ✗ (unless given reference) | ✓ (paired with a retrieval check) | Either | ✗ |
| Bedrock RAG evaluation job | ✓ | ✓ | ✓ (retrieval + generation metrics together) | Offline | ✓ (for correctness and coverage) |
| Human review of a sample | ✓ | ✓ | ✓ (with the right rubric) | Offline | ✓ |
| Answerable / unanswerable eval set | ✗ (it is the input) | ✗ (it is the input) | Enables it | Offline | ✓ |
Reading the table against the assistant: the runtime gate needs the Guardrails contextual grounding check on every answer; the offline regression signal comes from a Bedrock RAG evaluation job on a fixed set; the attribution the team is actually missing comes from reading retrieval quality and faithfulness together, either in that evaluation job or through an LLM judge paired with a retrieval check. No single method does all of it, and the unanswerable questions have to be built by hand whichever way you go.
The solution
For the live safety gate, the contextual grounding check is the right tool because it scores faithfulness at the moment of answering with no reference data required. On a Knowledge Base pipeline you attach the guardrail to the RetrieveAndGenerate call through its generation configuration, or call ApplyGuardrail yourself with the retrieved passages as the grounding source and the employee’s question as the query. Set a grounding threshold that reflects how much unsupported content you will tolerate, and let the guardrail block anything below it. The grounding and relevance scores are separate levers. A supported answer that wanders off the question is caught by the relevance threshold, and an on-topic answer carrying invented detail is caught by the grounding threshold. Set both from data, by scoring a labelled sample and finding the cut-off that blocks the fabrications without blocking the good answers.
The gotchas are worth knowing before you wire it in. A strict grounding threshold turns some correct-but-loosely-worded answers into blocked responses, so tune against real answers rather than reaching for 0.9 because it sounds safe. The policy caps the grounding source at 100,000 characters, the query at 1,000 and the response at 5,000, which a wide retrieval set will exceed. With a streaming response the irrelevance verdict only arrives once the whole answer has streamed, so the employee has read it by then. And AWS scopes the check to summarisation, paraphrasing and question answering, not conversational chatbot use, so a multi-turn version of this assistant falls outside what it covers.
For the offline signal, the Bedrock RAG evaluation job is the pick because it reports retrieval and generation quality in one place over a fixed dataset, which is what a regression check needs. Run it before any change to the embedding model, chunking, RerankingA second pass that re-scores a wide set of retrieved candidates and keeps only the few most relevant, so the expensive model reads less., or the generation prompt, and watch faithfulness and context relevance as separate lines. A change that improves retrieval can still drop faithfulness, because a longer context gives the answer more material to wander through, and a single blended number hides that trade. Supply ground truth so the job can also score correctness and context coverage, because faithfulness alone will pass a well-grounded answer that cites a superseded document. The thousand-prompt ceiling per job keeps the eval set a curated sample rather than a replay of the query log.
The attribution the team is missing comes from a cross. For each question in the eval set, establish two facts independently. First, did retrieval surface a passage that actually contains the answer? That is a retrieval question, answered by context relevance or coverage against the passage you labelled as the source. Second, is the generated answer faithful to what was retrieved? That is the grounding or faithfulness score. Crossing the two places every failure:
The top half is the branch where retrieval missed. If the model abstained, the system did the right thing on a question it could not answer. If it produced a confident answer anyway, that is a retrieval-caused hallucination. The fix there is better retrieval or a firmer instruction to abstain, and a stricter faithfulness threshold will not supply it. The bottom half is the branch where retrieval succeeded: a faithful answer is grounded and correct, and an unfaithful one is a generation-caused hallucination that better retrieval will never touch. Two failures that looked identical in the complaints now point at two different pieces of work. The broader end-to-end view of the pipeline, retrieval Recall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks., answer relevance, and cost together, is worth reading alongside this narrower hallucination cut; see evaluating a RAG pipeline end to end.
Worked example
Take two questions from the labelled set. The first, “what is the current mileage reimbursement rate”, is answerable, and the labelled source is the 2026 expenses policy. The second, “what is the sabbatical policy for contractors”, is unanswerable, because the company has no such policy and no document describes one; the expected output is an abstention.
Run the first. Context relevance is high, the retrieved passages include the 2026 expenses policy, so retrieval surfaced the answer and we are in the bottom branch. The generated answer quotes a rate from a 2024 document that also got retrieved. The grounding check scores it faithful, because the number really does appear in a retrieved passage, but the correctness check against the reference answer fails, because it is the superseded rate. This is the case faithfulness alone would pass: grounded and wrong. It reads as a generation problem only if you stop at faithfulness; the correctness comparison and the retrieval detail together show the real fix is at retrieval and ranking, keeping the superseded document out of the top passages.
Run the second. Retrieval surfaces some loosely related benefits text but nothing about contractor sabbaticals, so context relevance against the expected source is low and we are in the top branch. The model, rather than abstaining, produces a fluent paragraph describing a three-month unpaid sabbatical available after two years. The contextual grounding check scores it low, because none of that is in the retrieved passages, and at runtime the guardrail would block it. In the offline attribution it lands squarely as retrieval-caused, compounded by a failure to abstain. The fix is to strengthen the instruction to decline, in the Knowledge Base generation prompt template where the retrieved passages get substituted in. Keep the unanswerable question in the eval set too, so the abstention rate is tracked rather than assumed.
Two questions, two branches, two different remedies, and neither remedy is the one you would have reached for from the single sentence “it hallucinated”.
What’s worth remembering
- Faithfulness and correctness are different properties: faithfulness asks whether the answer is supported by the retrieved context, correctness asks whether it is right in the world, and an answer can pass one while failing the other.
- A hallucination comes from one of two stages: retrieval, where the context never held the answer, or generation, where the model ran past what the context said; attribute every miss to its stage before you try to fix it.
- Amazon Bedrock Guardrails contextual grounding checks score a response against the grounding source and the query, return a grounding score and a relevance score, and block answers that fall below the thresholds you set anywhere from 0 to 0.99.
- Bedrock RAG evaluation jobs report retrieval metrics and generation faithfulness together over a fixed set, which makes them the place to do attribution offline and to catch regressions before a release.
- Include known-unanswerable questions in the eval set; they are the only way to measure whether the system abstains rather than fabricating when a question falls outside the corpus.
- Cross two independent facts per question, did retrieval surface the answer and is the answer faithful to what was retrieved, and every failure sorts cleanly into retrieval-caused or generation-caused.