Evaluating a RAG Pipeline End to End

The situation

An internal assistant answers staff questions from a corpus of policy documents, runbooks, and past support threads. Every answer carries citations back to the source passages; that was a hard requirement from the start, covered when the team first built the citations-required retrieval layer. Most days it works. Roughly one answer in twenty is wrong, and “wrong” arrives as a Slack complaint with a screenshot, not a metric.

The team set out to fix the wrong answers. The trouble is they cannot see where the wrongness enters. A retrieval-augmentedRAGA pattern where you retrieve relevant documents at query time and stuff them into the prompt so the model can ground its answer on them. answer passes through two stages, and either one can sink it. The retriever might fetch the wrong passage, or no relevant passage at all, in which case the answer rests on nothing relevant. Or the retriever might fetch exactly the right passage and the answer contradict it, skip past it, or add a detail that was never in it.

Those two failures look identical from the outside. Same wrong answer, same annoyed user. But the retrieval failure lives in chunking, embeddingsEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together., hybrid search, or the value of k, and the generation failure lives in the prompt and the model. Fixing the prompt when the real problem is a retrieval miss changes nothing except your confidence. The team needs an evaluation that scores each half on its own, and one that re-runs on demand so a change to chunking or the reranker can be checked before it ships.

What actually matters

The decision that shapes everything else is to measure the two stages separately, because they have separate causes and separate fixes. An end-to-end score that says “82% correct” tells you the pipeline is imperfect and nothing about which half to touch.

Retrieval quality is measured against a labelled set: a list of queries, each mapped to the passage or passages that actually answer it. With those labels you get context recall (of the passages that should have been fetched, how many were), context precision (of the passages that were fetched, how many are relevant, and are they ranked near the top), plus the ranking metrics, hit-rate, MRR, and NDCG@k. Recall is usually the one that matters most for a wrong answer: if the right chunk never entered the context windowContext windowThe maximum number of tokens an LLM can attend to in a single call – prompt plus output combined., no prompt change can recover the answer.

Generation quality is measured given the retrieved context, and the central metric is faithfulness, sometimes called groundedness: does every claim in the answer follow from the passages that were actually retrieved, with nothing invented? Alongside it sit answer relevance (does the response address the question that was asked) and citation correctness (do the cited sources genuinely support the sentences that cite them). Faithfulness is not correctness. An answer can be perfectly faithful to a retrieved passage that happens to be the wrong passage. Grade faithfulness against the retrieved context and you learn whether the output stayed inside the passages it was handed; grade correctness against the ground truth and you learn whether the pipeline as a whole got it right. You want both, and you want to know which stage is responsible when they diverge.

All of this rests on a golden dataset: queries paired with ground-truth answers, and, for the retrieval half, the relevant-chunk labels. The labels are the expensive part. Writing a ground-truth answer is quick; deciding exactly which of fifty thousand passages are the relevant ones for a query is slow human work, and it is what makes retrieval measurable.

There is also a split between component evaluation and end-to-end evaluation, and both matter. Evaluating the retriever alone is cheap, fully repeatable, and isolates any change to chunking, the embedding model, or a reranker; you change one knob and watch recall@k move with nothing else in the way. Evaluating the whole pipeline measures the answer the user actually sees. Run the component eval constantly and the end-to-end eval to confirm the user-visible result.

What we’ll filter on

1. Does it separate retrieval failures from generation failures, or collapse them into one number?
2. Does it measure faithfulness or groundedness of the answer against the retrieved context?
3. Does it need labelled relevant-chunks, and can it produce retrieval metrics from them?
4. Managed service or custom code to build and maintain?
5. Scale and cost, how many queries can it grade for what outlay?
6. Repeatable as a regression harness gated on every pipeline change?

The landscape

Amazon Bedrock RAG evaluation. The managed, AWS-native option. It runs against an Amazon Bedrock Knowledge Base, or against inference responses you supply from a RAG source outside Bedrock. A job is one of two types. A retrieve-only job scores Context relevance, and Context coverage where the dataset carries ground-truth answers. A retrieve-and-generate job scores Correctness, Completeness, Helpfulness, Logical coherence, Faithfulness, Citation precision and Citation coverage, with Harmfulness, Stereotyping and Refusal alongside. An evaluator LLM computes all of them, and every score is an average between 0 and 1. Datasets are JSON Lines in S3, up to 1,000 prompts a job. Nothing in the built-in set is ranked: no recall@k, no MRR, no NDCG.

Bedrock model evaluation with a judge model on the generation step. A judge-based model-evaluation job grading the generation stage on its own. Built-in metrics include Faithfulness (Builtin.Faithfulness), which checks whether the response contains information absent from the prompt, plus Relevance, Correctness and Completeness; custom metrics let you supply your own judge prompt. Each record carries a prompt, and where the answers already exist a modelResponses entry, in which case Bedrock skips the invoke step and grades what you sent. The retrieved context has to travel inside the prompt text, a prompt is capped at 4KB, and a dataset holds 1,000 prompts.

RAGAS-style metrics through a custom pipeline. The open-source metric family, context precision, context recall, faithfulness, answer relevance, computed in your own code. Maximum flexibility over exactly what gets measured and how; more to write and maintain, and no managed reports or audit trail.

Retrieval-only metrics against a labelled relevance set. Recall@k, precision@k, MRR, and NDCG computed directly from the labels, driving the retriever through the Bedrock Knowledge Bases Retrieve API and matching what comes back against the known-relevant passages. Each result carries a documentId, a source location, a similarity score and its metadata, which is enough to identify it. This is the fast component eval: no generation, no judge, just the retriever measured against ground truth. It is the harness you re-run every time you touch chunking or embeddings, and it is the natural place to take retrieval latency measurements as well, timing the retrieval leg on the same queries it is already scoring.

Human review on a small stratified sample. The highest-fidelity signal, and too slow and costly to run on everything. Its real job is calibration: score a couple of hundred stratified examples by hand and correlate the human scores against the LLM judge, per metric, so you know which of the judge’s numbers to trust.

Evaluation

Side by side

No single row covers the whole pipeline and stays fast enough to run often. The managed RAG evaluation grades both halves in one report; the retrieval-only harness gives the fast, ranked retriever signal; the human sample calibrates the judge. The working answer stacks them.

The two stages, and where each is graded

Relevance and latency, read together

Retrieval quality testing usually runs on two axes: relevance scoring against the labelled set, and context matching verification, checking that the passages that came back genuinely contain what the query needed. There is a third, and it is the one that gets left out. Time the retrieval leg in three parts, because the three move for different reasons. Embedding the query is a model call and shifts when the embedding model or its provider changes. The vector search shifts with index size, filter complexity, and the value of k. Fetching the chunk text, plus any reranking pass over the candidates, shifts with how much came back. Record each as p50 and p99 across the same golden queries, and read the tail: a comfortable p50 over a p99 of four seconds means some slice of staff waits a long time for an answer.

Read the two axes together or you will optimise one into the other. The usual ways to lift recall@k are widening top-k and adding a reranker pass over the candidates, and both add time: more candidates to score, an extra model call in the path, more text pulled back. Recall@5 climbing from 0.55 to 0.88 reads as a clean win right up until you notice the p99 on the retrieval leg went from 180ms to 1.4 seconds, which delays the moment the answer starts streaming by more than a second. Put the retrieval latency measurements in the same report as the relevance numbers, one row per configuration, so the trade shows up when you make it rather than when someone says the assistant feels slow.

The solution

Start with Bedrock RAG evaluation for the managed end-to-end split. A retrieve-and-generate job over the golden query set returns Correctness, Faithfulness, Citation precision and Citation coverage on the answer; a retrieve-only job over the same queries returns Context relevance and Context coverage on what was fetched. Run both and the user-visible result is graded on both halves. There is no single job that scores two knowledge bases against each other, so run one job per configuration and compare the report cards; “we changed the chunk size, is it better?” becomes a pair of runs rather than an argument.

On its own, though, the managed job is heavier than you want after every small change, and its retrieval metrics are unranked. Context relevance says the fetched passages were on topic; it does not say the one passage you needed sat near the top. So add a retrieval-only recall@k harness against a labelled relevance set. Drive each golden query through Retrieve, match the returned documentId and source location against the known-relevant passages, and compute recall@k, precision@k, and MRR yourself. Retrieve returns five results unless you set numberOfResults, which is why recall@5 is the natural first number. No model call, no judge, a few seconds a query. This is the harness you gate on: change the chunking strategy, the embedding model, or add a reranker, and re-run it to see the retriever move in isolation with nothing downstream muddying the number.

Grade faithfulness with an LLM judge on the generation step, feeding it the question, the retrieved context, and the answer, and scoring against a rubric that asks whether every claim is supported by the context and whether the citations point at passages that back the sentences citing them. Then calibrate the judge: score a small stratified sample by hand, a couple of hundred queries spread across topics and across the easy and hard cases, and correlate the human scores against the judge per metric. A strong correlation means the judge’s number can stand in at scale; a weak one on, say, citation scoring means tightening the rubric or falling back on human scores there.

Wire the whole thing as a regression harness gated on every pipeline change. The retrieval-only run is fast enough for a pull request; the full RAG evaluation jobs and the judge run on a schedule or before a config ships. The rule to hold onto: any change to chunking, the embedding model, or the reranker invalidates every prior retrieval number, so re-evaluate rather than assume.

Then keep the same scored set running after it has passed. Drift monitoring is that retrieval-only harness on a schedule, nightly or weekly, over a golden set held still while the corpus underneath it grows, so a slow decline in recall@k arrives as a trend line instead of as a screenshot in Slack. The shape of the decline says where to look. A step change between two consecutive runs points at an event: an ingestion job that failed halfway, a re-index that ran with the wrong chunking config, an embedding model version that moved under you. A slow slide over weeks points at corpus growth, with more documents crowding the top-k and near-duplicate policy revisions competing for the same query, and that one is answered with chunking, metadata filters, or a larger k rather than a rollback. Run the retrieval latency measurements on the same schedule; index growth usually shows up in p99 before it shows up in recall.

A few gotchas are worth knowing. Faithfulness is not correctness, so an answer that is faithful to a wrong retrieved passage will score well at gate B and still be wrong; that is precisely the case gate A catches. Retrieval recall needs labels, and labels are costly, so bootstrap them by having a capable modelModelA trained set of weights plus the architecture that makes them useful – the thing you load up and run inference against. propose the relevant chunks for each query and a human verify the shortlist; verifying a shortlist is far faster than searching the corpus cold. Use the same calibration sample to check the judge for bias: split the human-versus-judge comparison by answer length and by which model produced the answer, and see whether either moves the judge’s score independently of quality. And the value of k sets the ceiling on recall: too small and relevant chunks fall off the list before generation runs, too large and precision drops while the tokenTokenThe unit of text an LLM actually sees – usually a short character sequence, not a whole word. bill on every query rises.

Worked example

A recurring complaint: staff asking about the parental-leave top-up policy get an answer that states the wrong eligibility window. The instinct is to blame the prompt, tighten the instruction to stick to the source, and ship. Run both gates first.

Policy-eligibility query set (40 queries), before any fix:

  Gate A · retrieval
    recall@5:      0.55      relevant chunk fetched in ~half of cases
    precision@5:   0.38
    MRR:           0.41

  Gate B · generation (graded on retrieved context)
    faithfulness:  0.93      answers stick closely to what was fetched
    answer relev.: 0.90

  End-to-end correctness (vs ground truth): 0.60

Read the two gates together. Faithfulness is high, so the answers stay inside the passages that were handed over, with almost nothing added. But recall@5 is 0.55, so nearly half the time the passage that actually contains the eligibility window never reached the model. The wrong answer is a retrieval miss, not a generation fault. A prompt change would move gate B, which is already fine, and leave the real problem untouched.

The fix belongs upstream. The eligibility details lived in a table. Bedrock’s default parser had flattened it to plain text, the fixed-size chunker then split what was left mid-row, and semantic search kept missing the exact policy term. Switch the data source to Amazon Bedrock Data Automation or a foundation model as the parser, either of which extracts tables and figures from a PDF rather than dropping them to text, move off fixed-size chunking to hierarchical or semantic chunking, and set overrideSearchType to HYBRID so the literal string “top-up” is searched alongside the embeddings. Hybrid needs an Amazon RDS, OpenSearch Serverless or MongoDB vector store with a filterable text field; on anything else the query runs as semantic search. Then re-run the retrieval-only harness.

Same 40 queries, after table-aware parsing + hybrid search:

  Gate A · retrieval
    recall@5:      0.88      (+0.33)
    precision@5:   0.61      (+0.23)
    MRR:           0.74      (+0.33)

  Gate B · generation
    faithfulness:  0.93      unchanged, as expected
    answer relev.: 0.91

  End-to-end correctness: 0.86   (+0.26)

Correctness jumped because the retriever now delivers the right passage; generation never needed touching. Had the team read only the end-to-end score, they would have seen 0.60, guessed at the prompt, and watched the number stay put. The two-gate split named the stage, and the fix landed where the failure actually was.

What’s worth remembering

1. A RAG answer can fail in retrieval or in generation, and the two have different fixes, so measure them separately or you will fix the wrong half.
2. Faithfulness is not correctness; an answer can be perfectly faithful to a passage that was the wrong passage to retrieve.
3. Bedrock RAG evaluation runs retrieve-only or retrieve-and-generate jobs of up to 1,000 prompts, and its retrieval metrics are Context relevance and Context coverage, neither of them ranked.
4. A fast retrieval-only recall@k harness against a labelled set isolates the retriever, so you can re-run it on every chunking, embedding, or reranker change.
5. Labels are the expensive input; bootstrap them by having a capable model propose relevant chunks and a human verify the shortlist.
6. Score retrieval latency beside the relevance numbers, because widening top-k and adding a reranker lift recall and add latency at the same time.