Exam-style
A RAG assistant gets about one answer in five wrong, and the team cannot agree whether the next sprint goes on retrieval tuning or on prompt work. What should they run to settle it?
Reveal the answer
D. Run both Amazon Bedrock RAG evaluation job types, retrieve-only and retrieve-and-generate, and compare the retriever scores against the answer scores
Amazon Bedrock RAG evaluations come in two job types, and running both is what separates the halves. A retrieve-only job scores the retriever on context relevance and context coverage; a retrieve-and-generate job scores the answer on correctness, completeness and faithfulness. Compare the two sets of numbers and the failing half shows up. A single end-to-end correctness score cannot attribute the failures, so tuning the prompt against it is guesswork. Invocation logs and contextual grounding flags describe symptoms without saying which half produced them, and changing the retrieved-chunk count experiments on one half before anyone knows it is the broken one.
Q. You cannot tell if a wrong RAG answer is a retrieval or a generation problem. What evaluates each half?
A. Amazon Bedrock RAG evaluation comes in two job types. A retrieve-only job scores the retriever on context relevance and context coverage. A retrieve-and-generate job scores the answer on correctness, completeness and faithfulness. Context coverage needs a ground truth in the prompt dataset, which is where the Recall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks. signal comes from.
Why? Measure the two failure surfaces separately or you will fix the wrong one.