The situation
A retailer is building a customer assistant on Amazon Bedrock. It has to do three quite different jobs from one chat surface. It answers policy questions (“how long do I have to return an item?”), which live in a few hundred pages of help-centre articles and terms documents that change a handful of times a year. It answers order questions (“where is order 55130, and when will it arrive?”), which live in the orders database and change minute to minute. And it answers account questions (“what is my current store-credit balance?”), which are specific to the signed-in customer and have to be exactly right.
The team’s first build put everything through one Amazon Bedrock Knowledge Base. The policy answers are good. The order answers are a disaster: the knowledge base was last synced overnight, so it tells a customer their parcel is “preparing to ship” when it was delivered two hours ago. The balance answers are worse, because no document anywhere holds a live per-customer number, so the assistant either reports that it cannot find one or returns a figure that reads as plausible and is not the customer’s balance.
The instinct is to sync the knowledge base more often. That is the wrong lever. Some of these answers should never have come from a retrieval index at all.
What actually matters
The first thing to name is that a retrieval index is a cache of documents, and every cache has a staleness bound. A Bedrock Knowledge Base answers from whatever was present at the last ingestion job; between syncs, it holds whatever was true when that job ran. For a returns policy that changes twice a year, that bound is invisible and retrieval is close to perfect. For an order status that changes every few minutes, the same bound guarantees wrong answers, and no sync frequency short of “continuously, per request” closes it. Once you need per-request freshness, you are describing a tool call, not an index.
The second axis is the shape of the answer. Retrieval is built to return passages: spans of text that a document contains, ranked by relevance, handed to the model as grounding context. That is exactly right when the answer is explanatory (“here is what the policy says, in its own words”) and exactly wrong when the answer is a single precise value that no document contains as prose. A live order’s delivery estimate, an account balance, today’s price: these are computed or looked up, not written down in an article. A tool call, function calling against an API or a database, returns that value directly, and the model quotes it rather than paraphrasing a passage.
The third is who the data belongs to. Policy documents are shared: one corpus serves every customer, so indexing it once and retrieving many times is efficient and safe. A balance is per-user, and a shared retrieval index is the wrong home for it, both because it is volatile and because it moves an access-control problem inside a vector store. Retrieval does have a per-user story for documents: a Bedrock Managed Knowledge Base can crawl document ACLs at ingestion and filter results against a user context your application supplies. AWS is careful to call that filtering rather than authorisation, because the service does not authenticate the user you name. A balance is not a document, so none of that reaches it. A tool call carries the signed-in customer to a system that already enforces who can see what.
The fourth is latency and cost shape. Retrieval adds an embedding lookup and some context tokens; it is cheap and predictable, and it amortises the ingestion cost across many queries. A tool call adds a round-trip to a live system and, in an agentic flow, a second model turn to read the result, so it costs more per answer and its latency depends on the downstream service. Accept that overhead for a fact that has to be current and exact. A policy that a cached passage answers just as well does not need it.
None of this makes retrieval and tools rivals. The strong build uses both: retrieve the policy passage that explains the returns window, and in the same conversation call a tool for the live order status, then let the model compose one answer from the shared document and the per-user fact. The design question is which one to use for each piece of the answer.
What we’ll filter on
- Data volatility, does the underlying fact change by the year, or by the minute?
- Answer shape, is the answer a document passage the model paraphrases, or a precise current value it must quote exactly?
- Ownership, is the data shared across all users, or specific to one signed-in user?
- Source of truth, does the value live as prose in documents, or in an API or database that computes it on demand?
- Latency and cost tolerance, can the answer absorb a live round-trip, or does it need to come from a cheap cached lookup?
The landscape
Retrieval-augmented generation (RAG). An ingestion job chunks and embeds a document corpus into a vector store; at query time the question is embedded, the nearest ChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. are retrieved, and they are passed to the model as grounding context. On Bedrock this is a Knowledge Base. A customer-managed one sits on a vector store you provision, such as Amazon OpenSearch Serverless, Aurora PostgreSQL with pgvector, Amazon S3 Vectors, or a Neptune Analytics graph for GraphRAG; a Bedrock Managed Knowledge Base runs the datastore, the embedding model and the reranker for you. Its strength is a large, slowly changing body of unstructured text: policies, manuals, help articles, contracts. Its hard limit is freshness, because the answer can only be as current as the last successful sync, so it is the wrong tool for a fact that moves faster than you ingest.
Live tool calling (function calling). The model is given a set of tools with typed schemas; when a question needs live data, it emits a call with arguments, the runtime executes it against an API or database, and the returned value comes back into the context for the model to answer from. On Bedrock this is the Converse API tool-use flow. AgentCore adds orchestration around it, and AgentCore Gateway turns the Lambda functions that reach the live system into MCP tools behind one endpoint. Build new agents there; the older Amazon Bedrock Agents, renamed Agents Classic, is no longer open to new customers. The strength here is exactly retrieval’s weakness: a volatile, precise, per-user value fetched at request time. It adds a live round-trip and, usually, an extra model turn to read the result.
Text-to-SQL. A specific and useful tool pattern for structured data: instead of hitting a hand-written API, the model translates the natural-language question into a SQL query, the query runs against the database, and the rows come back as the grounding value. Bedrock Knowledge Bases support this natively as a knowledge base over a structured data store. Amazon Redshift, provisioned or Serverless, is the query engine, and the tables it reads sit either in Redshift itself or in the AWS Glue Data Catalog through Lake Formation. A GenerateQuery call does the translation alone if you want the SQL without the retrieval. It suits questions whose answer is a live aggregate or lookup over a relational source (“how many orders shipped today”, “this customer’s current balance”) where writing a bespoke API per question would be tedious. It is a tool call in spirit; the value is computed at request time, not read from a stale index.
Retrieval plus tools together. The two combine in one conversation. Retrieve the shared, slow-moving passage; call a tool for the volatile, per-user number; compose one answer. This is the normal shape for an assistant that spans reference material and live state. One agent can hold both a Knowledge Base and a set of gateway tools, so a single reasoning loop does both.
Evaluation
Side by side
| Property | RAG (Knowledge Base) | Live tool call | Text-to-SQL |
|---|---|---|---|
| Fast-moving facts | ✗ (bounded by last sync) | ✓ | ✓ |
| Slow-moving document corpus | ✓ | ✗ | ✗ |
| Answer is a text passage | ✓ | ✗ | ✗ |
| Answer is a precise current value | ✗ | ✓ | ✓ |
| Per-user, access-controlled data | ✗ | ✓ | ✓ |
| Structured/relational source | ✗ | ✓ (via API) | ✓ (native) |
| Per-answer latency and cost | Low, predictable | Higher, live round-trip | Higher, query round-trip |
| Freshness at answer time | Last ingestion | Request time | Request time |
Reading the table against the three jobs: the returns policy is a slow-moving shared document, so RAG; the order status is a fast-moving per-user value from a live system, so a tool call; the store-credit balance is a precise per-user number in the database, so a tool call or, if you would rather not maintain a bespoke API, text-to-SQL. None of the three is fixed by syncing the knowledge base more often.
The solution
The returns policy stays on RAG, and the earlier build already had this part right. A few hundred pages of help articles and terms is precisely what a Bedrock Knowledge Base is for: a large, mostly static, unstructured corpus where the answer is a passage the model paraphrases. The staleness bound is real but invisible here, because a nightly or even weekly sync is faster than the documents change. The only addition worth making is treating the sync as a first-class step. A policy edit should fire a StartIngestionJob, or push the changed article straight into the index with IngestKnowledgeBaseDocuments, which works for S3 and custom data sources. Either beats waiting for the next scheduled run.
The order status moves to a live tool call, and this is the fix the team kept avoiding. Order state changes every few minutes and is specific to the signed-in customer, so it fails both the volatility test and the ownership test for an index. Declare a tool such as get_order_status(order_id), back it with a Lambda that reads the orders service, and let the model call it mid-conversation through the Converse API tool-use flow, or, where the assistant runs as an agent, through a gateway target that fronts the same Lambda. The value comes back at request time, the model quotes it, and “delivered two hours ago” is now something the assistant can actually say. It adds a round-trip and an extra model turn, and nothing shorter produces a current answer.
The balance is the same shape as the order status, with one extra choice about how to reach the data. It is a precise per-user value that lives in a relational store, so it is a tool call; the question is whether you hand-write a get_balance(customer_id) API or let text-to-SQL generate the query. If you already expose a clean balance endpoint, call it. If the questions are open-ended over structured data (“how much did I spend last quarter”, “how many open orders do I have”), a knowledge base over a structured data store can translate the question to SQL and run it on Redshift, against your warehouse tables or your Glue Data Catalog tables. That saves writing an API per question. The two routes differ on identity. A hand-written endpoint takes the signed-in customer, and the balance service enforces what they may see. Structured-data retrieval connects as the knowledge base service role, which holds a plain GRANT SELECT, so nothing about the signed-in customer reaches Redshift and you have to scope the query yourself. AWS puts it plainly: generated SQL is a security risk, so keep the role restricted, the database read-only, and the schema you expose small. Either way the value is computed at request time rather than read from a stale index.
The composed answer is where the two patterns meet. “Can I still return order 55130, and how long do I have?” needs the shared policy passage (retrieved) and the live per-user order date (a tool call) in the same turn. An agent holding both a Knowledge Base and a gateway tool gathers both and lets the model write one grounded reply, quoting the current fact and paraphrasing the policy; without an agent, the application retrieves the passage and runs the tool call in the same Converse conversation, and the composition is identical. The failure to avoid is forcing everything through one mechanism: pushing live state into the index gives stale answers, and pushing the policy through a bespoke tool throws away the cheap, shared, well-understood retrieval path for no gain.
Worked example
Take three questions arriving at the same chat surface, and route each by volatility and answer shape.
“What is your returns window for electronics?” The fact changes maybe twice a year, the answer is a passage, and the corpus is shared. This is RAG: the Knowledge Base retrieves the relevant clause from the terms document, and the model paraphrases it. No live call, low latency, and the answer is as current as the last ingestion, which is plenty.
“Where is my order 55130?” The fact changes by the minute and belongs to one customer. Retrieval cannot help; there is no document that holds a live tracking state, and even if there were it would be stale by the time it was indexed. The model calls get_order_status(55130), the Lambda reads the orders service, and the reply quotes the returned status and estimate. Request-time freshness, per-user identity carried to the source.
“How much store credit do I have right now?” A precise per-user number in the database. The model either calls get_balance(customer_id) or, if the assistant leans on text-to-SQL, the knowledge base generates SELECT balance FROM store_credit WHERE customer_id = :id, runs it on Redshift, and returns the row. A passage is no use here; the answer is the exact current figure, quoted, and it must be right, which is why it never came from a document.
Now stack them. “Can I return 55130, and how long have I got?” pulls the policy passage from the Knowledge Base and the order’s delivery date from the tool in one turn, and the model composes: the window from the shared document, the clock started by the per-user fact. One assistant, three grounding routes, each chosen by how fast the data moves and what the answer actually is.
What’s worth remembering
- A model answers from a frozen snapshot, so any fact that has changed since training has to be grounded from outside; the choice is retrieval, a live tool, or both.
- RAG suits a large, slowly changing corpus of documents where the answer is a passage the model paraphrases, and its freshness is bounded by the last ingestion or sync.
- That staleness bound makes RAG the wrong tool for fast-moving facts; syncing more often narrows the gap but never closes it for data that changes by the minute.
- Route by volatility and answer shape: slow document passage to RAG, fast or precise current value to a tool, per-user data to a tool that carries the user’s identity.
- A volatile per-user value does not belong in a shared retrieval index; document ACL filtering exists on managed knowledge bases, but a live balance is not a document and the source system is what enforces access.