Cutting Cost per Query in a RAG System

The situation

A team runs a documentation assistant on Amazon Bedrock. A user asks a question, the app embeds it, queries a vector store for the most similar chunks, and stuffs the top matches into a prompt alongside the question and a block of standing instructions. That whole prompt goes to a Claude model for the answer. The corpus is a few hundred thousand chunks of product docs, support articles, and policy pages, refreshed nightly from the source systems.

It works well and it is getting expensive. Traffic has grown to tens of thousands of queries a day, and the Bedrock line on the bill has climbed faster than traffic did. The team assumes the generation model is the culprit and starts pricing a cheaper one. The token counts tell a different story. The prompt going in is enormous, because someone set retrieval to return the top twenty chunks at a generous chunk size. Every one of those chunks is input tokens on every call. The vector store is a second surprise, running on a minimum capacity setting that bills whether or not anyone is querying. And the nightly refresh re-embeds the entire corpus, most of which has not changed since yesterday.

Answer quality is not up for negotiation. So the useful question is where the money goes in a single query, and which levers cut cost without cutting the answers.

What actually matters

The instinct is to shop for a cheaper generation model. Sometimes that helps. But it treats a RAG query as if it were a plain chat call, and it is not. Retrieved context gets prepended to the prompt on every call, and it is billed as input tokens exactly like the question and the instructions. On a well-fed RAG prompt those passages dwarf everything else going in. So the largest cost lever is how much context gets retrieved and sent, and it is the one most teams never touch. Twenty fat chunks and five tight ones cost very different amounts. The five tight ones often produce the better answer, because there is less irrelevant text between the model and the useful sentence.

Once retrieved context is named as the big lever, the rest decomposes cleanly. Generation is priced per input and output token, with the model tier and the output length setting the rate. Output is dearer: Claude 3.5 Sonnet v2 lists output at five times its input rate. The vector store behaves differently, because its compute is a capacity setting you choose rather than a per-query charge. The index size behind that setting follows the embedding dimensionEmbedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching. and the number of vectors. Embedding is trivial per query and potentially large at ingestion. Then there is the prompt scaffolding, the standing instructions and formatting boilerplate that ride along on every call and rarely change between queries.

That last observation is what makes caching the second big lever. The standing instructions repeat verbatim, so a prompt cache can process them once and read them back at the cache-read rate. On Claude models that rate lists around 90 percent below the standard input rate. Whole questions repeat too, more than teams expect, so a response cache keyed on the question can skip retrieval and generation entirely on a hit. Trimming makes each query smaller. Caching removes the repeated work.

Cost per query is a sum of parts, and none of those parts is actionable until it is attributed. Bedrock records input, output, cache-read and cache-write token counts for every request. Split that by feature, and per feature where several share one store, and the scary aggregate becomes a list of sized levers.

What we’ll filter on

1. Share of the bill, does this lever touch the biggest slice of a query cost or a rounding error?
2. Per-query versus capacity cost, is it billed on every call or on units that run while idle?
3. Quality risk, does pulling the lever threaten answer accuracy, and how much?
4. Repetition, does the work repeat across queries in a way caching can exploit?
5. Implementation cost, is it a config change or a rebuild of the pipeline?

The landscape

Retrieved context, the top-k and chunk-size lever. Every retrieved chunk is input tokens on every call. This is the largest RAG-specific cost and the one most under a team’s direct control. Cutting top-kTop-kHow many chunks a retrieval step returns per query – the dial that trades answer coverage against token cost. from twenty to five roughly quarters the retrieved-token bill, and tightening chunk size trims it further. The catch is recall, since fewer chunks risks dropping the passage that held the answer. Reranking recovers it. Retrieve a wide candidate set cheaply from the vector store, score it with a reranker, and send only the few most relevant to the generation model. Bedrock offers Amazon Rerank 1.0 (amazon.rerank-v1:0) and Cohere Rerank 3.5 (cohere.rerank-v3-5:0), through the standalone Rerank operation or as a rerankingConfiguration on a Knowledge Bases retrieval. Amazon Rerank 1.0 is not offered in us-east-1, where Cohere Rerank 3.5 is the only choice. Reranking bills separately and sends far fewer, far more relevant tokens to the expensive step. Chunk size and overlap set the same tension at ingestion: smaller chunks mean tighter, cheaper context, and more of them to store and search.

The generation model tier. Bedrock spans a wide range of per-token rates, from the small and cheap (Claude Haiku, Amazon Nova Micro and Lite) to the large and capable (Claude Sonnet and Opus, Nova Pro and Premier). Routing every query to the biggest model overpays for the many questions a small model answers perfectly. Amazon Bedrock Intelligent Prompt Routing automates part of that choice. It puts one serverless endpoint in front of two models from the same family, predicts each model’s response quality for the incoming prompt, and forwards the request accordingly. It does not route across families, and it is tuned for English prompts. Output length matters here too, because output tokens cost more per token than input, so a prompt that asks for a tight answer is cheaper.

Prompt caching. Bedrock supports two forms. Implicit caching reuses eligible prompt prefixes automatically on supported models, best effort, with no change to the request. Explicit caching lets you place a cache checkpoint at the end of a stable prefix. Either way, tokens read from cache bill at the cache-read rate, well below the standard input rate, while tokens written to cache can bill above it on some models. The natural target in RAG is the standing instruction block and any fixed examples. Retrieved chunks change with every question and never cache. Two constraints bound the saving. The prefix has to clear the model’s minimum checkpoint size, which runs from 512 tokens on Claude Opus 5 to 4,096 on Claude Haiku 4.5. And the cache expires on a TTL, five minutes by default with a one-hour option on recent Claude models, reset by each hit. Caching covers on-demand inference only, not the batch inference API.

Response and semantic caching. A different cache sits in front of the whole pipeline. Keep a store of previously answered questions and their answers. When a new question arrives, check it against that store, exactly or by embedding similarity, and on a hit return the stored answer with no retrieval and no generation. That skips the two most expensive steps for repeated questions, and in a docs assistant the same handful of questions recur constantly. The risk is staleness. A cached answer can outlive the document it came from, so entries need a time-to-live and an invalidation path when the underlying content changes.

The vector store capacity setting. Unlike the per-query charges, store compute runs on units you configure and bills while idle. Amazon OpenSearch Serverless bills OpenSearch Compute Units against a minimum and maximum you set per collection group, separately for indexing and for search. The minimum can be zero, in which case no OCUs run when idle, which adds a cold start on the next request. Above zero, OCU counts step through 2, 4, 8, 16, and multiples of 16. Aurora PostgreSQL Serverless v2 with pgvector works the same way with Aurora Capacity Units, and a minimum of 0 ACUs pauses the cluster automatically when idle. Amazon S3 Vectors takes a third shape, charging for stored vectors and per request with no capacity setting at all. It suits an index that is large or queried in bursts, and it targets sub-second latency for infrequent queries, as low as 100 ms when queries are frequent, so a latency-sensitive assistant should measure before moving. Index size drives cost in every case, and index size follows the embedding dimension and the vector count. Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0) returns 1,024 dimensions by default and can be configured at 512 or 256. That shrinks storage and search work, with some loss of retrieval quality worth measuring rather than assuming.

Embedding at ingestion. Embedding one short question at query time is cheap. The expensive embedding happens at ingestion, and re-embedding the whole corpus on every refresh is the classic waste, since most documents have not changed. Amazon Bedrock Knowledge Bases already syncs incrementally, processing only the documents added, modified or deleted since the last sync and skipping the rest. Metadata-only changes can avoid the embedding model altogether, merging new metadata into the stored vectors, provided the content file is not a CSV and the data source has no custom transformation Lambda. A hand-rolled pipeline that re-embeds everything nightly is doing work the managed path already avoids.

Prompt scaffolding. The standing instructions, role framing, and formatting boilerplate are input tokens on every call. A 400-word instruction block that grew by accretion is per-query overhead and nothing else. Trimming it to the minimum that holds quality, and caching what remains, removes a small constant from every query. At scale that constant adds up.

Underneath all of these sits measurement. Model invocation logging writes inputTokenCount and outputTokenCount per request to CloudWatch Logs or Amazon S3. Application inference profilesInference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. carry cost allocation tags, so Bedrock spend splits by application or feature in Cost Explorer and the Cost and Usage Report. Per-request metadata, up to 16 key-value entries on a Converse or InvokeModel call, tags individual calls in those logs, though it never reaches the bill. Without attribution the bill is one number. With it, every lever above has a size.

Evaluation

Side by side

Lever	Cost slice it cuts	Per-query or capacity	Quality risk	Effort
Lower top-k	Retrieved context (largest)	Per-query	Recall drop if too aggressive	Config
Reranking	Retrieved context	Per-query	✗ (improves relevance)	Add a step
Smaller chunk size	Retrieved context and index	Both	Context fragmentation	Re-ingest
Model routing	Generation	Per-query	Wrong-tier misses	Config / router
Shorter output	Generation output	Per-query	Truncated answers	Prompt
Prompt caching	Scaffolding prefix	Per-query	✗	Config
Response / semantic cache	Retrieval + generation	Per-query	Staleness without TTL	Build a cache
Smaller embedding dimension	Vector store index	Capacity	Retrieval quality	Re-embed
Lower store minimum capacity	Vector store idle units	Capacity	Cold start on first call	Config
Incremental ingestion	Embedding at ingestion	Pipeline	✗	Config
Trim scaffolding	Every prompt	Per-query	If over-trimmed	Prompt

Read against the docs assistant: the top-20 retrieval is the first and biggest fix. Reranking recovers the recall a lower top-k would lose. The store’s minimum capacity and the full nightly re-embed are idle waste with clean fixes, and prompt plus response caching mop up the repetition. Swapping the generation model, the team’s first instinct, is real, and it is not the largest slice on this query.

The solution

Retrieval settings hold the largest saving, so they come first. Dropping top-k from twenty to five cuts the retrieved-token bill by roughly three quarters. Retrieved context is the dominant slice of the prompt, so that is the single biggest number on the whole query. The fear is that five chunks miss what twenty would have caught. Reranking answers that fear better than a high top-k does. Retrieve a wide net cheaply, say forty candidates, well inside the ceiling of 100 results on a Knowledge Bases retrieval. Pass them through a reranker and keep the five most relevant for the model. You pay a modest reranking charge and a cheap wide vector query. In exchange the generation step sees a short, dense, highly relevant context instead of twenty passages of which fifteen were noise. Answer quality usually improves while cost falls, because there is less irrelevant material in the prompt. Chunk size is the same lever at ingestion. Tighter chunks make the five you send smaller and sharper, at the risk of more chunks in the index and of splitting a coherent passage across a boundary, which chunk overlap softens.

Caching is the second pick, on two levels that do not overlap. The prompt cache targets the stable prefix: the standing instructions and any fixed examples, marked so Bedrock processes them once and reads them back at the cache-read rate. It cannot hold the retrieved chunks, because those change with the question, so its value is bounded by the size of the fixed scaffolding. Check the model’s minimum checkpoint size before counting on it. Keep traffic steady enough that the five-minute TTL does not lapse between calls, or use the one-hour TTL that recent Claude models accept. The response cache targets whole questions. Embed the incoming question, compare it against a store of previously answered questions, and on a close match return the stored answer with no retrieval and no generation. In a docs assistant the same questions recur relentlessly, so the hit rate is high, and each hit removes the two most expensive steps. It needs invalidation. Give cached answers a time-to-live and clear the relevant entries when the source document changes, or the cache will serve last month’s answer about this month’s pricing.

The vector store and the ingestion pipeline are the idle-cost picks, easy to forget because they never show up per query. Both OpenSearch Serverless and Aurora Serverless v2 run against a minimum capacity you set, and both accept a minimum of zero. Zero stops the idle charge and adds a cold start on the first request after a quiet spell, so pick that trade deliberately rather than leaving a default in place. If the index is large, or query rates are bursty, Amazon S3 Vectors charges for stored vectors and per request with no capacity to size at all, though it is designed for less frequent querying. Whatever the store, index size follows the embedding dimension, so Titan Text Embeddings V2 at 512 or 256 instead of its default 1,024 shrinks storage and speeds search. Measure that against retrieval quality on your own corpus rather than guessing. And the nightly refresh should embed only what changed. Knowledge Bases syncs incrementally out of the box, processing added, modified and deleted documents and skipping the rest, which turns a full re-embed into a small delta.

The model tier and the scaffolding are the smaller, faster picks. Not every question needs the largest model. Routing easy lookups to a cheaper tier, by hand or with Intelligent Prompt Routing choosing between two models of one family per prompt, stops you overpaying for questions a small model answers perfectly. Asking for a concise answer trims output tokens, which carry the higher rate. The scaffolding trim is the smallest number and the easiest change. Every needless word in the standing instruction block is input tokens on every call, so cutting a 400-word preamble to the 80 words that hold quality removes a constant from tens of thousands of daily queries. None of this is guesswork once spend is attributed. Turn on model invocation logging for the token counts and tag inference with application inference profiles, so cost-allocation tags split the Bedrock bill by feature. Each lever then stops being a hunch and becomes a sized line you can rank.

Worked example

Take one representative query. Before, retrieval returns the top twenty chunks at a generous size. The prompt carries a large block of context, a 400-word standing instruction preamble, and the short question, all sent to a large Claude model that writes a long answer. The vector store sits on a minimum capacity that never drops, indexed at 1,024 dimensions, and the nightly job re-embeds all few-hundred-thousand chunks. Retrieved context is the great majority of the input tokens. The preamble rides along uncached on every call. Common questions are answered from scratch each time, and the ingestion bill arrives in full nightly for a corpus that barely changed.

After, the same query looks different at every step. Retrieval pulls a wide cheap candidate set and a reranker keeps the five best, so the context block shrinks to roughly a quarter while answer quality holds or improves. The standing preamble is trimmed and marked as a cached prefix, so its tokens are charged once and read back cheaply. A response cache sits in front of the pipeline, so the large fraction of queries repeating a known question return with no retrieval and no generation. Easy questions route to a cheaper model tier and the prompt asks for a tighter answer. The store’s minimum capacity is sized to real traffic, or dropped to zero where a cold start is acceptable, and the embedding dimension comes down where quality allows. Ingestion embeds only the delta each night. No single change is the whole saving. The retrieved-context trim is the largest slice, caching removes the repeated work, and the capacity fixes stop the charge that was accruing whether or not anyone asked a question.

What’s worth remembering

1. In a RAG query the retrieved context is billed as input tokens on every call and is usually the largest slice, so top-k and chunk size are the biggest and most overlooked cost lever, not the generation model.
2. Lower top-k to cut retrieved tokens and recover the recall with reranking: retrieve a wide cheap candidate set, rerank it with Amazon Rerank 1.0 or Cohere Rerank 3.5, and send only the few best to the generation model.
3. A response or semantic cache in front of the pipeline skips retrieval and generation on repeated questions, while a Bedrock prompt cache covers only the stable prefix, subject to a per-model minimum checkpoint size and a TTL of five minutes or an hour.
4. Vector store compute is a capacity setting rather than a per-query charge: OpenSearch Serverless OCUs and Aurora Serverless v2 ACUs both accept a minimum of zero, which removes the idle charge and adds a cold start, while Amazon S3 Vectors bills stored vectors and requests with no capacity to set.
5. Do not re-embed an unchanged corpus; Bedrock Knowledge Bases syncs incrementally, processing only the documents added, modified or deleted since the last sync.