Agentic RAG: When Retrieval Needs to Reason

The situation

The retrieval-augmented assistant behind the subscriber help desk started as a textbook RAG pipeline. A question comes in, an embedding model turns it into a vector, the vector store returns the closest few chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. of help-article text, five by default, and those chunks go into the prompt as context for the model to answer from. On Bedrock this is a Knowledge Base doing the embedding, storage, and retrieval, and a single RetrieveAndGenerate call stitching the fetched passages into a grounded reply. For “how do I pause my box” or “when is my next delivery cut-off”, it works and it is fast.

The questions have outgrown the single pass. A subscriber asks something like “why was I charged after I paused, and does the refund policy differ for the summer boxes”. That is two facts from possibly two places: the pause-and-billing rules and the seasonal-refund schedule. The phrasing is also nothing like the way the source documents are written, so the raw query embeds poorly and the top matches come back weak. Sometimes a good answer needs the first batch of results back before a sharper second query can be written at all.

The team can leave the fixed pipeline in place and accept that some questions get thin answers, or they can let the model drive the retrieval: decide whether to search, which source to search, how to word the search, and whether one round was enough. That second shape is agentic RAG, and it costs more than the pipeline it sits behind. The call is when the extra machinery is worth it.

What actually matters

The first axis is who decides the retrieval. In plain RAG the pipeline decides: the query is always embedded, the store is always queried once, the top-kTop-kHow many chunks a retrieval step returns per query – the dial that trades answer coverage against token cost. always goes into the prompt. Nothing about that sequence depends on the question, which is why it is cheap to run and easy to reason about. In agentic RAG the foundation model decides. It can skip retrieval when nothing in the question needs looking up, pick which of several sources to query, or rewrite the question into something that embeds well. It can also run the search, take the passages back as input, and issue a second, sharper query. Control over retrieval moves from a fixed pipeline to the model at run time, and everything else trades against that shift.

The second is how many hops the question needs. A single-hop question resolves from one retrieval: one fact, one place, one pass. A multi-hop question needs the answer to the first lookup before you can even phrase the second, the classic “find X, then use X to find Y”. A fixed pipeline cannot do the second one, because it only retrieves once and it retrieves before it has seen anything. The moment a question genuinely chains, one lookup feeding the next, you have left the territory a single pass can cover.

The third is how many sources are in play and whether the query needs reformulating before it will match anything. One well-indexed source and questions phrased like the documents, and plain top-k retrieval does fine. Several sources with different content, or questions worded nothing like the source text, and someone has to choose the source and rewrite the query. A model can do both. Self-querying, which Bedrock calls implicit metadata filtering, turns “refunds for summer boxes since June” into a metadata filter (season = summer, date after June) plus a semantic search over the refund text. That lands on passages a raw embedding of the whole sentence would miss. The decomposition is retrieval reasoning, and the fixed pipeline has no place to put it.

The fourth is the budget for the extra cost. Every retrieval decision the model makes is at least one more foundation-model call, and an iterative loop that retrieves, reads, reformulates, and retrieves again can be several. That multiplies latency and token spend, and it widens the failure surface: more calls, more tool invocations, more chances to loop without converging, or to stop one hop short of the passage the answer needed. Plain RAG is a handful of calls you can count in advance; a loop has to be capped instead.

Underneath all of it, the plain pipeline is the floor and most questions never leave it. Single-fact, single-source, well-phrased questions are the bulk of real traffic, and running a reasoning loop over them adds calls and latency without improving the answer. The agentic machinery is for the questions that provably cannot be answered in one pass, not a blanket upgrade.

What we’ll filter on

1. Single-hop or multi-hop? Does answering need the result of one retrieval before the next can be phrased?
2. One source or several? Does the model need to choose where to look, or is there only one place?
3. Does the query need reformulating, decomposing, or turning into a metadata filter before it will match the source text?
4. What is the latency and cost budget for extra model calls per question?
5. How predictable does the retrieval path need to be, and how much of the loop is the team willing to own and observe?

The landscape

Plain RAG, a fixed pipeline. Embed the query, retrieve the top-k once, put the passages in the prompt, generate. On Bedrock this is a Knowledge Base with a single RetrieveAndGenerate call, or Retrieve plus your own generation step. There is no decision to make at run time; the sequence is the same for every question. Its strength is that it is cheap, fast, and predictable, and it is enough for the single-fact questions that make up most traffic. Its ceiling is that it retrieves exactly once, before it has seen anything, from wherever you pointed it, using the query exactly as asked.

Query reformulation on top of plain RAG. Still one retrieval pass, but the query is improved before it runs. Setting queryTransformationConfiguration.type to QUERY_DECOMPOSITION on a RetrieveAndGenerate call has Bedrock break a multi-part question into sub-queries and combine their results. A Retrieve call can also carry an implicitFilterConfiguration, a schema of metadata attributes from which a Claude model generates the metadata filter for that query. Both improve matching on awkwardly worded or compound questions, and both stop short of iteration, because they run before retrieval rather than in response to what retrieval returned.

Managed agentic retrieval. AgenticRetrieveStream takes a conversation and a list of retrievers, plans a retrieval strategy, runs several retrieval steps across those knowledge bases, and streams back the passages plus a synthesized answer with citations. maxAgentIteration caps the planning and retrieval rounds, and traceEvent messages report each step as it runs, so the loop is bounded and observable without being yours to operate. Two constraints: it works only with managed knowledge bases, which are exactly the ones RetrieveAndGenerate will not serve, and the loop only retrieves, so anything the answer needs besides a lookup sits outside it.

An agent loop you run. An agent on AgentCore with one or more knowledge bases attached as retrieval tools, alongside any other actions it needs. The foundation model runs the reason-act-observe loop: whether to retrieve, which knowledge base, how to word or decompose the query, then what the passages imply for the next step. This is the shape for questions where retrieval interleaves with actions that are not retrieval, such as reading a subscriber’s live billing record between two lookups. The cost is more model calls, higher and less predictable latency, and a failure surface you own.

Evaluation

Side by side

Reading it for this help desk, most questions are single-hop and stay on the fixed pipeline; a good fraction are badly phrased and need reformulation; a smaller set are genuinely multi-hop or multi-source. None of them need an action that is not a lookup, which rules out the last row and leaves the managed API as the escalation path.

The fixed pipeline against the model loop

The solution

Keep a deterministic rewrite-then-retrieve pipeline for the bulk of the traffic, and escalate only the questions that provably need it to AgenticRetrieveStream. Two shapes in production, with something cheap choosing between them. Neither shape alone fits the help desk: the fixed pipeline cannot answer a question that chains, and a reasoning loop on every question adds calls and seconds to the single-fact questions that are most of the traffic.

The default path stays a fixed pipeline with one rewrite in front of it. QUERY_DECOMPOSITION turns a messy or compound question into sub-queries that each match well, then merges what comes back. That recovers most of the answer quality raw top-k loses on awkward phrasing, in one extra step rather than a loop. It is still deterministic: you can trace it, its calls are countable, and it cannot run unbounded. What it cannot do is react to what the first retrieval returned, because the rewrite runs before the retrieval rather than in response to it.

The escalation path is the managed loop. Hand AgenticRetrieveStream the conversation and a retriever per knowledge base, and the service plans, queries the billing base and the delivery base as the plan needs them, expands to full documents where a chunk is not enough, and returns passages plus a cited answer in one call. Multi-hop and multi-source retrieval need exactly that, and neither is expressible in a pipeline that retrieves once. The cost is real: several model calls per question instead of one, latency measured in seconds rather than a second, and a plan that can terminate at maxAgentIteration with the answer still incomplete.

Build the agent loop yourself only when retrieval has to interleave with actions that are not retrieval. That is a different problem, and the reasoning that lets one agent drive its own tools is what lets one agent drive several, covered in orchestrating multiple agents.

Both paths read the same corpus, and once more than one thing retrieves from it, the useful unit is a single retrieval interface with a fixed contract rather than two implementations that drift. The contract is small: a query string, optional metadata filters, and a top-K, returning chunks with their source identifier, a citation, and a score. Every caller goes through it, whether that is a Flow node, a tool on the agent, the nightly batch job that pre-answers the common questions, or a second product built next year, so chunk shape, filter semantics, and citation format cannot come apart. Consistent access mechanisms keep the same corpus from answering three different ways depending on who asked.

That contract takes three shapes on AWS, and it is the same contract each time. As a Converse toolSpec whose inputSchema is the contract, retrieval becomes a function any tool-using model on Bedrock can call. As an AgentCore Gateway target, it reaches Model Context Protocol clients whatever framework the agent is built on, with no per-agent adapter; the managed-knowledge-base connector does this natively, publishing Retrieve and AgenticRetrieveStream as MCP tools an agent finds through tools/list. And where the corpus lives in one knowledge base and nothing bespoke is needed, Retrieve and RetrieveAndGenerate are the interface AWS ships rather than one you maintain.

The gotcha is identity. A shared retrieval tool has to carry the caller’s identity through to the filter it applies, or the permission-safe filtering the corpus already does is lost the moment retrieval becomes a shared service. Both Retrieve and AgenticRetrieveStream take a userContext and filter on it, and the Gateway passes it straight through, but it does not fill it in from the caller’s IAM identity. Your application has to supply it on every call. Leave it out and the tool runs under its own role, every caller sees everything, and the leak arrives looking like a correct answer.

The failure mode that makes the routing worth building is silent. When a question needs a second hop and the pipeline answers anyway, it returns something thin rather than an error, so the signal to escalate is falling answer quality on compound or awkwardly worded questions, not an exception in a log.

What actually does the routing. Two shapes in production implies something choosing between them, and that chooser has to cost less than what it saves. Sending every question to a capable model to ask “does this need the loop?” adds a capable-model call to every question, which is the spend you were avoiding.

Three ways to make the call, cheapest first. Heuristics get further than they sound: question length, a question mark count above one, conjunctions like “and” or “then”, a date range or a metadata-ish phrase (“since June”, “for summer boxes”), the presence of two nouns that live in different knowledge bases. These are free, they run in microseconds, and on a help desk with a narrow domain they catch a useful share of the multi-hop traffic.

A small model does the rest. This is a classification task with three labels and no need for reasoning, so it runs on the smallest and fastest model you have access to, with a tight prompt, a handful of examples, and a single-token answer:

ROUTE_PROMPT = """Classify the support question. Answer with one word.

SIMPLE      one fact, answerable from a single lookup
REFORMULATE one fact, but phrased nothing like the documentation
AGENTIC     needs two or more lookups, or spans billing and delivery

Question: {question}
Answer:"""

resp = bedrock.converse(
    modelId="au.anthropic.claude-haiku-4-5-20251001-v1:0",
    messages=[{"role": "user",
               "content": [{"text": ROUTE_PROMPT.format(question=q)}]}],
    inferenceConfig={"maxTokens": 4, "temperature": 0},
)
route = resp["output"]["message"]["content"][0]["text"].strip().upper()

Claude Haiku 4.5 has no in-Region on-demand option on bedrock-runtime, so the modelId is a geo inference profile, au. from Australia or New Zealand and us., eu. or jp. elsewhere. Four output tokens and a temperature of zero, because the output is a label and not a sentence. Measure it the way you would any classifier: hold out a few hundred real questions, label them by hand, and read the confusion matrix rather than the accuracy, because the two mistakes have very different consequences.

The errors are asymmetric, and the design follows from that. Routing a simple question to the loop wastes a handful of model calls and a few seconds. Routing a multi-hop question to the pipeline produces a confident, thin, wrong answer that a subscriber acts on. The second is much worse, so the router should lean towards the loop whenever it is unsure, and three coarse buckets with a bias make that easier than a fine-grained taxonomy nobody can hold in their head.

The cheaper design is not to classify at all. Run the pipeline first and escalate when the retrieval comes back weak: top score under a threshold, or a generation step that returns an “I don’t know” from the passages it was given. That is a router built out of evidence rather than prediction, it adds no model call on the common path, and it cannot misjudge a question shape it has never seen. It does add latency on the escalated tail, which stays acceptable while the tail is small. Start here, and reach for the classifier only when the tail stops being small.

The same trade recurs whenever two model sizes sit behind one endpoint, and it gets its own treatment in routing between a cheap and a capable model.

Worked example

A subscriber writes: “why was I charged after I paused last month, and is the refund different for the summer boxes I had before that?”

As plain RAG. The pipeline embeds the whole sentence, queries the one knowledge base once, and gets back a mix of pause-policy and general-refund passages, none of them a clean match. The question braids two topics, and it names a season the base stores in a metadata field rather than in the prose. The generated answer is partly right about pausing and vague about the summer refund, because the passage that would have settled it never made the top-k. One pass, low cost, thin answer. Nothing errored. The answer was weaker than the subscriber needed, which is how the plain pipeline fails on a two-hop question.

As agentic retrieval. The plan splits the question in two. The first retrieval returns the pause-and-billing rules, which state that a charge after a valid pause is an error. The second targets the seasonal-refund rule, combining a metadata filter for the summer season with a semantic search over the refund text, and lands on the exact clause. With both passages in hand the plan ends, and the synthesized reply resolves the charge and states the summer-box refund correctly, each half carrying a citation. Two retrievals and a plan rather than one query, a couple of seconds more on the clock, and an answer the single pass could not assemble. That is worth doing on the questions that need the second hop and nowhere else.

What’s worth remembering

1. The deciding axis is who controls retrieval. A pipeline that always fetches once, or a model that chooses each step.
2. Multi-hop is the clearest trigger. If the second lookup needs the result of the first, a single-pass pipeline cannot get there.
3. AgenticRetrieveStream is the managed form of the loop, capped by maxAgentIteration and traced step by step. Build your own agent only when retrieval has to interleave with actions that are not retrieval.
4. Query reformulation is the cheap middle ground. QUERY_DECOMPOSITION and implicitFilterConfiguration both run before retrieval, so neither reacts to what came back.
5. A shared retrieval interface has to carry userContext explicitly. Nothing downstream infers the caller from IAM, and an unfiltered result reads like a correct answer.
6. Most questions never need the loop. Reserve it for multi-hop, multi-source or reformulation-heavy questions, and keep the pipeline for the rest.