Exam-style
A Bedrock RAG evaluation job scores 0.95 on Builtin.Faithfulness across the prompt dataset, yet a manual review of fifty answers finds eight of them factually wrong. What is the most likely explanation?
Reveal the answer
B. Retrieval is returning the wrong passages and the answers faithfully reflect them, so correctness against ground truth needs scoring too
Faithfulness measures only whether the answer follows from the passages that were retrieved. The Builtin.Faithfulness evaluator prompt says so outright: rate how much of the answer is present in the related passages, and ignore other mistakes such as untruthful answers. An answer grounded perfectly in the wrong passage therefore scores 0.95 and is still wrong. Builtin.Correctness grades the answer against the ground truth you supply in referenceResponses, so the two diverge exactly on retrieval misses. Supply that ground truth: with no referenceResponses, the correctness prompt falls back to the retrieved passages and scores the same eight answers as correct. The other options do not fit a 0.95. Detail added beyond the retrieved context pulls faithfulness down. Larger chunks give the evaluator more text to match the answer against, which raises the score. And a contextual grounding check scores the response against the same retrieved source, so it passes an answer faithful to the wrong passage at any threshold.
Q. A RAG answer scores high on faithfulness but is still wrong. How?
A. Builtin.Faithfulness scores how much of the answer appears in the retrieved passages, and its prompt says to ignore untruthful answers. Builtin.Correctness grades the answer against the ground truth in referenceResponses. An answer can be perfectly faithful to the wrong retrieved ChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense.. Score both.
Why? Faithfulness locates generation problems; correctness covers the whole pipeline. They diverge on retrieval misses. Leave referenceResponses out and correctness falls back to judging against the retrieved passages, which removes the second check.