The situation
The retrieval-augmented assistant behind the subscriber help desk started as a textbook RAG pipeline. A question comes in, an embedding model turns it into a vector, the vector store returns the closest few ChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. of help-article text, and those chunks go into the prompt as context for the model to answer from. On Bedrock this is a Knowledge Base doing the embedding, storage, and retrieval, and a single RetrieveAndGenerate call stitching the fetched passages into a grounded reply. For “how do I pause my box” or “when is my next delivery cut-off”, it works and it is fast.
The questions have outgrown the single pass. A subscriber asks something like “why was I charged after I paused, and does the refund policy differ for the summer boxes”. That is two facts from possibly two places: the pause-and-billing rules and the seasonal-refund schedule. The phrasing is also nothing like the way the source documents are written, so the raw query embeds poorly and the top matches come back weak. Sometimes a good answer needs the model to see the first batch of results, notice what is missing, and go looking again with a sharper query.
The team can leave the fixed pipeline in place and accept that some questions get thin answers, or they can let the model drive the retrieval: decide whether to search, which source to search, how to word the search, and whether one round was enough. That second shape has a name now, agentic RAG, and it costs more than the pipeline it replaces. The call is when the extra machinery is worth it.
What actually matters
The first thing to name is who decides the retrieval. In plain RAG the pipeline decides: the query is always embedded, the store is always queried once, the Top-kHow many chunks a retrieval step returns per query – the dial that trades answer coverage against token cost. always goes into the prompt. Nothing about that sequence depends on the question, which is exactly why it is cheap to run and easy to reason about. In agentic RAG the foundation model decides. It can look at the question and choose not to retrieve at all when it already holds the answer, pick which of several sources to query, rewrite the question into something that embeds well, or fire the search, read the results, and decide a second search is needed. The control over retrieval moves from a fixed pipeline to the model at run time, and that single shift is what everything else trades against.
The second is how many hops the question needs. A single-hop question resolves from one retrieval: one fact, one place, one pass. A multi-hop question needs the answer to the first lookup before you can even phrase the second, the classic “find X, then use X to find Y”. A fixed pipeline cannot do the second one, because it only retrieves once and it retrieves before it has seen anything. The moment a question genuinely chains, one lookup feeding the next, you have left the territory a single pass can cover.
The third is how many sources are in play and whether the query needs reformulating before it will match anything. One well-indexed source and questions phrased like the documents, and plain top-k retrieval does fine. Several sources with different content, or questions worded nothing like the source text, and someone has to choose the source and rewrite the query. A model can do both: self-querying turns “refunds for summer boxes since June” into a metadata filter (season = summer, date after June) plus a semantic search over the refund text, which finds the right passages that a raw embedding of the whole sentence would miss. That decomposition is retrieval reasoning, and the fixed pipeline has no place to put it.
The fourth is the budget for the extra cost. Every retrieval decision the model makes is at least one more foundation-model call, and an iterative loop that retrieves, reads, reformulates, and retrieves again can be several. That multiplies latency and token spend, and it widens the failure surface: more calls, more tool invocations, more chances to loop without converging or to talk itself out of a retrieval it needed. Plain RAG is a handful of calls you can count in advance; agentic RAG is a loop you can only bound loosely.
Underneath all of it, the plain pipeline is the floor and most questions never leave it. Single-fact, single-source, well-phrased questions are the bulk of real traffic, and paying for a reasoning loop to answer them is spend with nothing to show. The agentic machinery is for the questions that provably cannot be answered in one pass, not a blanket upgrade.
What we’ll filter on
- Single-hop or multi-hop? Does answering need the result of one retrieval before the next can be phrased?
- One source or several? Does the model need to choose where to look, or is there only one place?
- Does the query need reformulating, decomposing, or turning into a metadata filter before it will match the source text?
- What is the latency and cost budget for extra model calls per question?
- How predictable does the retrieval path need to be, and how much of the loop is the team willing to own and observe?
The landscape
Plain RAG, a fixed pipeline. Embed the query, retrieve the top-k once, put the passages in the prompt, generate. On Bedrock this is a Knowledge Base with a single RetrieveAndGenerate call, or Retrieve plus your own generation step. There is no decision to make at run time; the sequence is the same for every question. Its strength is that it is cheap, fast, and predictable, and it is enough for the single-fact questions that make up most traffic. Its ceiling is that it retrieves exactly once, before it has seen anything, from wherever you pointed it, using the query exactly as asked.
Query reformulation on top of plain RAG. Still one retrieval pass, but the query is improved before it runs: a preprocessing step rewrites vague phrasing, or Bedrock Knowledge Bases’ own query-decomposition breaks a multi-part question into sub-queries whose results are combined. This gives better matches for awkwardly worded or compound questions without a full reasoning loop. It stops short of true iteration, because the rewrite happens before retrieval, not in response to what the first retrieval returned.
Agentic RAG, a model-driven loop. An agent on AgentCore with one or more Knowledge Bases attached as retrieval tools, alongside any other actions it needs. The foundation model runs a reason-act-observe loop: it decides whether to retrieve, which knowledge base to query, how to word or decompose the query, reads what comes back, and decides whether to retrieve again before answering. This is where multi-hop, multi-source, self-querying, and iterative retrieval live. The cost is more model calls, higher and less predictable latency, and a larger failure surface. It is worth it when a single pass genuinely cannot reach the answer.
A fixed pipeline with one reformulation step, the middle ground. Where questions are mostly single-hop but often badly phrased, a deterministic rewrite-then-retrieve keeps the predictability of the pipeline while fixing the match quality, without opening the door to an unbounded loop. It handles reformulation but not iteration or genuine multi-hop, which still need the model in the driving seat.
Evaluation
Side by side
| Option | Retrieval decided by | Multi-hop / iterative | Chooses among sources | Reformulates the query | Cost and latency | Predictable path |
|---|---|---|---|---|---|---|
| Plain RAG (fixed pipeline) | Pipeline, always once | ✗ | ✗ | ✗ | Low | ✓ |
| RAG + query reformulation | Pipeline, one rewrite | ✗ | ✗ | ✓ (before retrieval) | Low to moderate | ✓ |
| Agentic RAG (model loop) | Model, at run time | ✓ | ✓ | ✓ (and re-queries) | High, hard to bound | ✗ |
| Pipeline + one rewrite step | Pipeline, one rewrite | ✗ | ✗ | ✓ | Low to moderate | ✓ |
Reading it for this help desk, most questions are single-hop and stay on the fixed pipeline; a good fraction are badly phrased and need reformulation; a smaller set are genuinely multi-hop or multi-source and are the only ones that justify the agentic loop. The field narrows to two live shapes: keep the pipeline (with a reformulation step) for the common case, and route the questions that need to reason about their own retrieval to an agent.
The fixed pipeline against the model loop
The solution
Keep a deterministic rewrite-then-retrieve pipeline for the bulk of the traffic, and route only the questions that provably need it to an agent. Two shapes in production, with something cheap deciding between them. Neither shape alone fits the help desk: the fixed pipeline cannot answer a question that chains, and an agent on every question pays a reasoning loop for the single-fact questions that are most of the traffic.
The default path stays a fixed pipeline with one rewrite in front of it. A rewrite step, or Bedrock Knowledge Bases’ own query decomposition, turns a messy or compound question into sub-queries that each match well, then merges what comes back. That recovers most of the answer quality raw top-k loses on awkward phrasing, for the cost of one extra step rather than a loop. It is still deterministic: you can trace it, its cost is countable, and it never runs away. What it cannot do is react to what the first retrieval returned, because the rewrite happens before the retrieval, not in response to it.
The escalation path is an agent with the knowledge bases attached as retrieval tools. The model decides the retrieval: skip it when the answer is already in hand, pick the billing knowledge base over the delivery one, decompose “refunds for summer boxes since June” into a metadata filter plus a semantic search, read the passages, and go back for a second, sharper query when the first came up short. Multi-hop, multi-source, and iterative retrieval need exactly that, and none of it is expressible in a pipeline that retrieves once. The bill is real: each decision is at least one more model call, latency climbs, and the loop is a new thing that can fail to converge or reason itself out of a retrieval it needed.
A retrieval agent is one agent with knowledge bases as its actions, and the same reasoning that lets one agent drive its own tools is what lets one agent drive several, covered in orchestrating multiple agents.
The failure mode that makes the routing worth building is quiet. When a question needs a second hop and the pipeline answers anyway, it returns something thin rather than an error, so the signal to escalate is falling answer quality on compound or awkwardly worded questions, not an exception in a log.
What actually does the routing. Two shapes in production implies something choosing between them, and that chooser has to be cheaper than the thing it is protecting you from, or it eats the saving it exists to make. Sending every question to the capable model to ask “does this need an agent?” costs a capable-model call on every question, which is the bill you were avoiding.
Three ways to make the call, cheapest first. Heuristics get further than they sound: question length, a question mark count above one, conjunctions like “and” or “then”, a date range or a metadata-ish phrase (“since June”, “for summer boxes”), the presence of two nouns that live in different knowledge bases. These are free, they run in microseconds, and on a help desk with a narrow domain they catch a useful share of the multi-hop traffic.
A small model does the rest. This is a classification task with three labels and no need for reasoning, so it runs on the cheapest model available, with a tight prompt, a handful of examples, and a single-token answer:
ROUTE_PROMPT = """Classify the support question. Answer with one word.
SIMPLE one fact, answerable from a single lookup
REFORMULATE one fact, but phrased nothing like the documentation
AGENT needs two or more lookups, or spans billing and delivery
Question: {question}
Answer:"""
resp = bedrock.converse(
modelId="anthropic.claude-haiku-4-5",
messages=[{"role": "user",
"content": [{"text": ROUTE_PROMPT.format(question=q)}]}],
inferenceConfig={"maxTokens": 4, "temperature": 0},
)
route = resp["output"]["message"]["content"][0]["text"].strip().upper()
Four output tokens and a temperature of zero, because this is a label and not a sentence. Measure it the way you would any classifier: hold out a few hundred real questions, label them by hand, and read the confusion matrix rather than the accuracy, because the two mistakes cost very different amounts.
The errors are asymmetric, and that asymmetry is the whole design. Routing a simple question to the agent wastes money and adds a few seconds. Routing a multi-hop question to the pipeline produces a confident, thin, wrong answer that a subscriber acts on. The second is much worse, so the router should lean towards the agent whenever it is unsure, and the label set should make that easy: three coarse buckets with a bias, not a fine-grained taxonomy nobody can hold in their head.
The cheaper design is not to classify at all. Run the pipeline first and escalate when the retrieval comes back weak: top score under a threshold, or the generation step declining to answer from the passages it was given. That is a router built out of evidence rather than prediction, it costs nothing on the common path, and it never mistakes a question it has not seen before. The price is latency on the escalated tail, which pays for itself when the tail is small. Start here, and reach for the classifier only when the tail stops being small.
The same trade recurs whenever two model sizes sit behind one endpoint, and it gets its own treatment in routing between a cheap and a capable model.
Worked example
A subscriber writes: “why was I charged after I paused last month, and is the refund different for the summer boxes I had before that?”
As plain RAG. The pipeline embeds the whole sentence, queries the one knowledge base once, and gets back a mix of pause-policy and general-refund passages, none of them a clean match because the question braids two topics and mentions a season the base indexes under a metadata field, not in the prose. The model generates a partly-right answer about pausing and hedges on the summer refund, because the passage that would have settled it never made the top-k. One pass, low cost, thin answer. Nothing errored; the answer was just weaker than the subscriber needed, which is the plain pipeline’s characteristic failure on a two-hop, self-querying question.
As agentic RAG. The agent reads the question and decomposes it. First it retrieves the pause-and-billing rules and confirms a charge after a valid pause is an error. Then, needing the seasonal-refund rule, it self-queries: a metadata filter for the summer season plus a semantic search over the refund text, which lands on the exact clause. Seeing both facts, it decides it has enough and drafts a reply that resolves the charge and states the summer-box refund correctly. It cost the agent’s reasoning plus two retrievals rather than one, and the reply came back a couple of seconds slower, and in exchange the answer was complete where the single pass left a gap. That trade, more calls and more latency for an answer a fixed pipeline could not assemble, is the whole case for the loop, and it only pays on questions that actually need the second hop.
What’s worth remembering
- The deciding axis is who controls retrieval. A pipeline that always fetches once, or a model that chooses each step.
- Multi-hop is the clearest trigger. If the second lookup needs the result of the first, a single-pass pipeline cannot get there.
- The cost is real: more model calls, higher and less predictable latency, and a wider failure surface, including loops that do not converge.
- Query reformulation is the cheap middle ground. It fixes awkward phrasing in one rewrite before retrieval, without a full reasoning loop.
- Most questions never need the loop. Reserve agentic RAG for multi-hop, multi-source, or reformulation-heavy questions, and keep the pipeline for the rest.
The help desk keeps the fixed pipeline, with a reformulation step, for the bulk of its traffic, and routes the questions that have to reason about their own retrieval to an agent with the knowledge bases as tools. The choice is not one shape for everything: a single well-phrased fact stays on the pipeline, a question that chains or spans sources tips to the loop, and the line between them is always the same, whether one retrieval can answer it or the model has to decide how to look.