Agentic RAG: When Retrieval Needs to Reason

The situation

The retrieval-augmented assistant behind the subscriber help desk started as a textbook RAG pipeline. A question comes in, an embedding model turns it into a vector, the vector store returns the closest few chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. of help-article text, and those chunks go into the prompt as context for the model to answer from. On Bedrock this is a Knowledge Base doing the embedding, storage, and retrieval, and a single RetrieveAndGenerate call stitching the fetched passages into a grounded reply. For “how do I pause my box” or “when is my next delivery cut-off”, it works and it is fast.

The questions have outgrown the single pass. A subscriber asks something like “why was I charged after I paused, and does the refund policy differ for the summer boxes”. That is two facts from possibly two places: the pause-and-billing rules and the seasonal-refund schedule. The phrasing is also nothing like the way the source documents are written, so the raw query embeds poorly and the top matches come back weak. Sometimes a good answer needs the model to see the first batch of results, notice what is missing, and go looking again with a sharper query.

The team can leave the fixed pipeline in place and accept that some questions get thin answers, or they can let the model drive the retrieval: decide whether to search, which source to search, how to word the search, and whether one round was enough. That second shape has a name now, agentic RAG, and it costs more than the pipeline it replaces. The call is when the extra machinery earns its keep.

What actually matters

The first thing to name is who decides the retrieval. In plain RAG the pipeline decides: the query is always embedded, the store is always queried once, the top-kTop-kHow many chunks a retrieval step returns per query – the dial that trades answer coverage against token cost. always goes into the prompt. Nothing about that sequence depends on the question, which is exactly why it is cheap to run and easy to reason about. In agentic RAG the foundation model decides. It can look at the question and choose not to retrieve at all when it already holds the answer, pick which of several sources to query, rewrite the question into something that embeds well, or fire the search, read the results, and decide a second search is needed. The control over retrieval moves from a fixed pipeline to the model at run time, and that single shift is what everything else trades against.

The second is how many hops the question needs. A single-hop question resolves from one retrieval: one fact, one place, one pass. A multi-hop question needs the answer to the first lookup before you can even phrase the second, the classic “find X, then use X to find Y”. A fixed pipeline cannot do the second one, because it only retrieves once and it retrieves before it has seen anything. The moment a question genuinely chains, one lookup feeding the next, you have left the territory a single pass can cover.

The third is how many sources are in play and whether the query needs reformulating before it will match anything. One well-indexed source and questions phrased like the documents, and plain top-k retrieval does fine. Several sources with different content, or questions worded nothing like the source text, and someone has to choose the source and rewrite the query. A model can do both: self-querying turns “refunds for summer boxes since June” into a metadata filter (season = summer, date after June) plus a semantic search over the refund text, which finds the right passages that a raw embedding of the whole sentence would miss. That decomposition is retrieval reasoning, and the fixed pipeline has no place to put it.

The fourth is the budget for the extra cost. Every retrieval decision the model makes is at least one more foundation-model call, and an iterative loop that retrieves, reads, reformulates, and retrieves again can be several. That multiplies latency and token spend, and it widens the failure surface: more calls, more tool invocations, more chances to loop without converging or to talk itself out of a retrieval it needed. Plain RAG is a handful of calls you can count in advance; agentic RAG is a loop you can only bound loosely.

Underneath all of it, the plain pipeline is the floor and most questions never leave it. Single-fact, single-source, well-phrased questions are the bulk of real traffic, and paying for a reasoning loop to answer them is spend with nothing to show. The agentic machinery is for the questions that provably cannot be answered in one pass, not a blanket upgrade.

What we’ll filter on

1. Single-hop or multi-hop? Does answering need the result of one retrieval before the next can be phrased?
2. One source or several? Does the model need to choose where to look, or is there only one place?
3. Does the query need reformulating, decomposing, or turning into a metadata filter before it will match the source text?
4. What is the latency and cost budget for extra model calls per question?
5. How predictable does the retrieval path need to be, and how much of the loop is the team willing to own and observe?

The retrieval-control landscape

1. Plain RAG, a fixed pipeline. Embed the query, retrieve the top-k once, put the passages in the prompt, generate. On Bedrock this is a Knowledge Base with a single RetrieveAndGenerate call, or Retrieve plus your own generation step. There is no decision to make at run time; the sequence is the same for every question. Its strength is that it is cheap, fast, and predictable, and it is enough for the single-fact questions that make up most traffic. Its ceiling is that it retrieves exactly once, before it has seen anything, from wherever you pointed it, using the query exactly as asked.

2. Query reformulation on top of plain RAG. Still one retrieval pass, but the query is improved before it runs: a preprocessing step rewrites vague phrasing, or Bedrock Knowledge Bases’ own query-decomposition breaks a multi-part question into sub-queries whose results are combined. This buys better matches for awkwardly worded or compound questions without a full reasoning loop. It stops short of true iteration, because the rewrite happens before retrieval, not in response to what the first retrieval returned.

3. Agentic RAG, a model-driven loop. A Bedrock Agent with one or more Knowledge Bases attached as retrieval tools, alongside any other actions it needs. The foundation model runs a reason-act-observe loop: it decides whether to retrieve, which knowledge base to query, how to word or decompose the query, reads what comes back, and decides whether to retrieve again before answering. This is where multi-hop, multi-source, self-querying, and iterative retrieval live. The cost is more model calls, higher and less predictable latency, and a larger failure surface. It earns its keep when a single pass genuinely cannot reach the answer.

4. A fixed pipeline with one reformulation step, the middle ground. Where questions are mostly single-hop but often badly phrased, a deterministic rewrite-then-retrieve keeps the predictability of the pipeline while fixing the match quality, without opening the door to an unbounded loop. It handles reformulation but not iteration or genuine multi-hop, which still need the model in the driving seat.

Side by side

Option	Retrieval decided by	Multi-hop / iterative	Chooses among sources	Reformulates the query	Cost and latency	Predictable path
Plain RAG (fixed pipeline)	Pipeline, always once	✗	✗	✗	Low	✓
RAG + query reformulation	Pipeline, one rewrite	✗	✗	✓ (before retrieval)	Low to moderate	✓
Agentic RAG (model loop)	Model, at run time	✓	✓	✓ (and re-queries)	High, hard to bound	✗
Pipeline + one rewrite step	Pipeline, one rewrite	✗	✗	✓	Low to moderate	✓

Reading it for this help desk, most questions are single-hop and stay on the fixed pipeline; a good fraction are badly phrased and want reformulation; a smaller set are genuinely multi-hop or multi-source and are the only ones that justify the agentic loop. The field narrows to two live shapes: keep the pipeline (with a reformulation step) for the common case, and route the questions that need to reason about their own retrieval to an agent.

The fixed pipeline against the model loop

The picks in depth

Plain RAG, the fixed pipeline. The right shape for the common case: a single fact from a single source, phrased closely enough to the documents that a raw embedding lands on the right passages. On Bedrock a Knowledge Base plus one RetrieveAndGenerate call is the whole thing, and its predictability is a feature, not a limitation. Every question costs the same, the latency is a single retrieval and a single generation, and there is no loop to run away. Keep questions here as long as one pass reaches the answer. The failure mode is quiet: when a question needs a second hop or a rewritten query, the pipeline still returns something, just something thin, so the signal to move is falling answer quality on compound or awkwardly worded questions, not an error.

RAG with query reformulation. The right shape when the questions are single-hop but often phrased nothing like the source text, or bundle two askable things into one sentence. A rewrite step, or Bedrock Knowledge Bases’ query-decomposition, turns the messy question into sub-queries that each match well, then merges the retrieved passages. This recovers most of the answer quality that raw top-k loses on awkward phrasing, and it does so for the cost of one extra step rather than a full reasoning loop. It stops at reformulation: the rewrite happens before the single retrieval, so it cannot react to what came back or chase a fact it did not know it needed until it saw the first results.

Agentic RAG, the model-driven loop. The right shape when the question genuinely cannot be answered in one pass. A Bedrock Agent with one or more Knowledge Bases attached as retrieval tools lets the foundation model decide the retrieval: skip it when the answer is already in hand, pick the billing knowledge base over the delivery one, decompose “refunds for summer boxes since June” into a metadata filter plus a semantic search, read the passages, and go back for a second, sharper query when the first came up short. That is what multi-hop, multi-source, and iterative retrieval require, and none of it is expressible in a pipeline that retrieves once. The bill is real: each decision is at least one more model call, an iterative loop can be several, latency climbs, and the loop is a new thing that can fail to converge or reason itself out of a retrieval it needed. This connects to the wider agent story; a retrieval agent is one agent with knowledge bases as its actions, and the same reasoning that lets one agent drive its own tools is what lets a supervisor drive several, covered in orchestrating multiple Bedrock agents. Reach for the loop when a single pass provably falls short, not as a default upgrade.

The pipeline-with-one-rewrite middle ground. The right shape when most traffic is single-hop but poorly phrased, and you want the match quality of reformulation without opening the door to an unbounded loop. A deterministic rewrite-then-retrieve is still a fixed pipeline: you can trace it, its cost is countable, and it never loops. It handles the phrasing problem and leaves the genuine multi-hop questions to be routed to an agent, which keeps the expensive shape reserved for the questions that actually need it.

A worked example: one subscriber question, two ways

A subscriber writes: “why was I charged after I paused last month, and is the refund different for the summer boxes I had before that?”

As plain RAG. The pipeline embeds the whole sentence, queries the one knowledge base once, and gets back a mix of pause-policy and general-refund passages, none of them a clean match because the question braids two topics and mentions a season the base indexes under a metadata field, not in the prose. The model generates a partly-right answer about pausing and hedges on the summer refund, because the passage that would have settled it never made the top-k. One pass, low cost, thin answer. Nothing errored; the answer was just weaker than the subscriber needed, which is the plain pipeline’s characteristic failure on a two-hop, self-querying question.

As agentic RAG. The agent reads the question and decomposes it. First it retrieves the pause-and-billing rules and confirms a charge after a valid pause is an error. Then, needing the seasonal-refund rule, it self-queries: a metadata filter for the summer season plus a semantic search over the refund text, which lands on the exact clause. Seeing both facts, it decides it has enough and drafts a reply that resolves the charge and states the summer-box refund correctly. It cost the agent’s reasoning plus two retrievals rather than one, and the reply came back a couple of seconds slower, and in exchange the answer was complete where the single pass left a gap. That trade, more calls and more latency for an answer a fixed pipeline could not assemble, is the whole case for the loop, and it only pays on questions that actually need the second hop.

What’s worth remembering

1. Plain RAG is a fixed pipeline: embed, retrieve top-k once, generate. Cheap, fast, predictable, and enough for most single-fact questions.
2. Agentic RAG moves the retrieval decision to the model at run time: whether to retrieve, which source, how to word it, and whether to retrieve again.
3. The deciding axis is who controls retrieval. A pipeline that always fetches once, or a model that chooses each step.
4. Multi-hop is the clearest trigger. If the second lookup needs the result of the first, a single-pass pipeline cannot get there.
5. Self-querying is retrieval reasoning: turning a question into a metadata filter plus a semantic search finds passages a raw embedding misses.
6. On Bedrock, agentic RAG is an agent with one or more Knowledge Bases attached as retrieval tools, reasoning across steps.
7. The cost is real: more model calls, higher and less predictable latency, and a wider failure surface, including loops that do not converge.
8. Query reformulation is the cheap middle ground. It fixes awkward phrasing in one rewrite before retrieval, without a full reasoning loop.
9. Most questions never need the loop. Reserve agentic RAG for multi-hop, multi-source, or reformulation-heavy questions, and keep the pipeline for the rest.

The help desk keeps the fixed pipeline, with a reformulation step, for the bulk of its traffic, and routes the questions that have to reason about their own retrieval to an agent with the knowledge bases as tools. The choice is not one shape for everything: a single well-phrased fact stays on the pipeline, a question that chains or spans sources tips to the loop, and the line between them is always the same, whether one retrieval can answer it or the model has to decide how to look.