Cutting Cost per Query in a RAG System

The situation

A team runs a documentation assistant on Amazon Bedrock. A user asks a question, the app embeds it, queries a vector store for the most similar chunks, stuffs the top matches into a prompt alongside the question and a block of standing instructions, and sends the whole thing to a Claude model for the answer. The corpus is a few hundred thousand chunks of product docs, support articles, and policy pages, refreshed nightly from the source systems.

It works well and it is getting expensive. Traffic has grown to tens of thousands of queries a day, and the Bedrock line on the bill has climbed faster than traffic did. The team assumes the generation model is the culprit and starts pricing a cheaper one, but a look at the token counts tells a different story: the prompt going in is enormous, because someone set retrieval to return the top twenty chunks at a generous chunk size, and every one of those chunks is input tokens on every call. The vector store is a second surprise, billing a steady hourly rate whether or not anyone is querying it. And the nightly refresh re-embeds the entire corpus each run, most of which has not changed since yesterday.

Nobody wants to hurt answer quality to save money. The question is where the money actually goes in a single query, and which levers cut cost without cutting the answers.

What actually matters

The instinct is to shop for a cheaper generation model, and sometimes that helps, but it treats a RAG query as if it were a plain chat call. It is not. The defining feature of retrieval-augmented generation is that a chunk of retrieved context gets prepended to the prompt on every call, and that context is billed as input tokens exactly like the question and the instructions are. On a well-fed RAG prompt the retrieved passages dwarf everything else going in, so the single largest cost lever is one most teams never touch: how much context gets retrieved and sent. Twenty fat chunks and five tight ones cost very different amounts, and the five tight ones are frequently the better answer because the model is not wading through irrelevant passages to find the useful sentence.

Once retrieved context is named as the big lever, the rest of the cost decomposes cleanly. There is the generation itself, priced per input and output token, where the model tier and the output length set the rate. There is the vector store, which unlike the per-query charges runs as a standing cost: some stores bill compute by the hour whether idle or busy, and the index size that drives that cost is set by the embedding dimensionEmbedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching. and the corpus. There is the embedding step, small per query for the incoming question but potentially large at ingestion when the corpus is re-embedded. And there is the prompt scaffolding, the standing instructions and formatting boilerplate that ride along on every call and are usually identical from one query to the next.

That last observation, that a large part of every RAG prompt is identical call to call, is what makes caching the second big lever. The standing instructions repeat verbatim, so a prompt cache can charge for them once and read them cheaply thereafter. Whole questions repeat too, more than teams expect, so a response cache keyed on the question can skip retrieval and generation entirely on a hit. Caching attacks cost from a different direction to trimming: trimming makes each query smaller, caching makes repeated work free.

The honest framing is that cost per query is a sum of parts, and you cannot cut what you have not measured. Attributing spend to retrieval, generation, embedding, and the vector store per query, and per feature if several share one store, is what turns a scary aggregate bill into a list of specific, sized levers.

What we’ll filter on

1. Share of the bill, does this lever touch the biggest slice of a query cost or a rounding error?
2. Per-query versus standing cost, is it billed on every call or as an idle-time floor?
3. Quality risk, does pulling the lever threaten answer accuracy, and how much?
4. Repetition, does the work repeat across queries in a way caching can exploit?
5. Implementation cost, is it a config change or a rebuild of the pipeline?

The cost landscape

Retrieved context, the top-k and chunk-size lever. Every retrieved chunk is input tokens on every call, so this is the largest RAG-specific cost and the one most under a team direct control. Cutting top-kTop-kHow many chunks a retrieval step returns per query – the dial that trades answer coverage against token cost. from twenty to five roughly quarters the retrieved-token bill, and tightening chunk size trims it further. The catch is recall: fewer chunks risks dropping the passage that held the answer. The move that buys back that recall is reranking. Retrieve a wide candidate set cheaply from the vector store, then use a reranker such as the Amazon Bedrock Rerank API (with Cohere Rerank or Amazon Rerank models) to score them and keep only the few most relevant to send to the generation model. You pay a small reranking charge to send far fewer, far more relevant tokens to the expensive step. Chunk size and overlap set the same tension at ingestion: smaller chunks mean tighter, cheaper context but more of them to store and search.

The generation model tier. Bedrock offers a spread of models at very different per-token rates, from the small and cheap (Claude Haiku, Amazon Nova Micro and Lite) to the large and capable (Claude Sonnet and Opus, Nova Pro). Routing every query to the biggest model overpays for the many questions a small model would answer perfectly. Sending easy lookups to a cheap tier and reserving the expensive tier for genuinely hard synthesis is a large saving; Amazon Bedrock Intelligent Prompt Routing can do this automatically within a model family, predicting the complexity of each prompt and dispatching it to the cheapest model likely to answer it well. Output length matters here too, since output tokens usually cost more per token than input; a prompt that asks for a tight answer rather than an essay pays less.

Prompt caching. Bedrock supports prompt caching for supported models, letting you mark a stable prefix so its tokens are processed once and read from cache on later calls at a large discount, with cache reads billed far below the normal input rate. In RAG the natural cache target is the standing instruction block and any fixed few-shot examples, because they are identical every call. The retrieved chunks are not a good cache target, because they change with every question; only the invariant scaffolding caches cleanly. So caching pairs with trimming rather than replacing it.

Response and semantic caching. A different cache sits in front of the whole pipeline. Keep a store of previously answered questions and their answers; when a new question arrives, check it against that store, exactly or by embedding similarity, and on a hit return the stored answer without retrieving or generating at all. This skips the two most expensive steps entirely for repeated questions, and in a docs assistant the same handful of questions recur constantly. The risk is staleness: a cached answer can outlive the document it came from, so cache entries need a time-to-live and an invalidation path when the underlying content changes.

The vector store running cost. Unlike the per-query charges, the store bills whether or not anyone is querying. Amazon OpenSearch Serverless bills OpenSearch Compute Units by the hour with a minimum floor, so a small corpus still carries a standing cost. Aurora PostgreSQL Serverless v2 with pgvector bills Aurora Capacity Units with a configurable minimum. Amazon S3 Vectors takes a different shape, charging for stored vectors and per query with no compute floor, which suits a large or bursty corpus that cannot justify an always-on cluster. Index size drives cost across all of them, and index size is set by the embedding dimension and the number of vectors. Choosing an embedding model that supports a smaller dimension, or configuring one that does such as Amazon Titan Text Embeddings V2 at 512 or 256 dimensions instead of 1024, shrinks the index, the storage bill, and the search work, at some cost to retrieval quality that is worth measuring rather than assuming.

Embedding at ingestion. The query-time embedding of one short question is cheap. The expensive embedding happens at ingestion, and re-embedding the whole corpus on every refresh is the classic waste: most documents have not changed since the last run. An incremental sync that embeds only new and modified chunks, which is how Amazon Bedrock Knowledge Bases ingestion behaves when it detects unchanged source content, turns a full re-embed into a small delta and cuts the ingestion bill to a fraction.

Prompt scaffolding. The standing instructions, role framing, and formatting boilerplate are input tokens on every call. Bloated scaffolding, the 400-word instruction block that grew by accretion, is pure per-query overhead. Trimming it to the minimum that holds quality, and caching what remains, removes a small constant from every single query, which adds up at scale.

Underneath all of these sits measurement. Bedrock model invocation logging records the token counts per request, and application inference profilesInference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. let you tag inference calls so cost allocation tags attribute Bedrock spend per application or feature. Without that attribution the bill is one scary number; with it, each lever above has a size, and you pull the big ones first.

Side by side

Lever	Cost slice it cuts	Per-query or standing	Quality risk	Effort
Lower top-k	Retrieved context (largest)	Per-query	Recall drop if too aggressive	Config
Reranking	Retrieved context	Per-query	✗ (improves relevance)	Add a step
Smaller chunk size	Retrieved context and index	Both	Context fragmentation	Re-ingest
Model routing	Generation	Per-query	Wrong-tier misses	Config / router
Shorter output	Generation output	Per-query	Truncated answers	Prompt
Prompt caching	Scaffolding prefix	Per-query	✗	Config
Response / semantic cache	Retrieval + generation	Per-query	Staleness without TTL	Build a cache
Smaller embedding dimension	Vector store index	Standing	Retrieval quality	Re-embed
Right-sized / serverless store	Vector store floor	Standing	✗	Migrate
Incremental ingestion	Embedding at ingestion	Pipeline	✗	Config
Trim scaffolding	Every prompt	Per-query	If over-trimmed	Prompt

Read against the docs assistant: the top-20 retrieval is the first and biggest fix, reranking buys back the recall a lower top-k would cost, the vector store floor and the full nightly re-embed are standing waste with clean fixes, and prompt plus response caching mop up the repetition. Swapping the generation model, the team first instinct, is real but it is not the largest slice on this query.

The picks in depth

The retrieval settings are where the largest saving lives, so they come first. Dropping top-k from twenty to five cuts the retrieved-token bill by roughly three quarters, and because retrieved context is the dominant slice of the prompt, that is the single biggest number on the whole query. The fear is that five chunks miss the answer twenty chunks would have caught, and the answer to that fear is reranking rather than a high top-k. Retrieve a wide net cheaply from the vector store, say the top forty candidates, then pass them through the Bedrock Rerank API and keep the five most relevant to actually send to Claude. You pay a modest reranking charge and a cheap wide vector query, and in exchange the expensive generation step sees a short, dense, highly relevant context instead of twenty passages of which fifteen were noise. Answer quality usually goes up while cost goes down, because the model is not distracted by irrelevant chunks. Chunk size is the same lever at ingestion: tighter chunks mean the five you send are smaller and sharper, at the price of more chunks in the index and a risk of splitting a coherent passage across a boundary, which chunk overlap exists to soften.

Caching is the second pick, and it works on two levels that do not overlap. The prompt cache targets the stable prefix, the standing instructions and any fixed examples, marking them so Bedrock processes them once and reads them cheaply on every subsequent call. It cannot cache the retrieved chunks, because those change with the question, so its value is bounded by how large the fixed scaffolding is; trim the scaffolding and cache what remains. The response cache targets whole questions. Embed the incoming question, compare it against a store of previously answered questions, and on a close match return the stored answer with no retrieval and no generation at all. In a docs assistant the same questions recur relentlessly, so the hit rate is high and each hit removes the two most expensive steps from the query. The discipline this needs is invalidation: give cached answers a time-to-live and clear the relevant entries when the source document changes, or the cache will happily serve last month answer about this month pricing.

The vector store and the ingestion pipeline are the standing-cost picks, easy to forget because they do not show up per query. The store bills by the hour: OpenSearch Serverless by the OCU with a floor, Aurora Serverless v2 by the ACU with a configurable minimum, so a modest corpus on an always-on cluster can carry a meaningful idle cost. If the corpus is large, bursty, or the query rate does not justify a running cluster, Amazon S3 Vectors bills for stored vectors and per query with no compute floor, which changes the shape of the bill. Whatever the store, the index size is set by the embedding dimension, so choosing or configuring a smaller-dimension embedding, Titan Text Embeddings V2 supports 512 and 256 alongside its default 1024, shrinks storage and speeds search, a trade against retrieval quality worth measuring on your own corpus rather than guessing. And the nightly refresh should embed only what changed. Re-embedding an unchanged corpus every night is paying the ingestion bill in full to produce the same vectors, where an incremental sync that touches only new and modified chunks, which Bedrock Knowledge Bases does when it detects unchanged content, cuts that to a small delta.

The model tier and the scaffolding are the smaller, faster picks. Not every question needs the largest model; routing easy lookups to a cheaper tier, by hand or with Bedrock Intelligent Prompt Routing choosing within a model family per prompt, stops the assistant overpaying for questions a small model answers perfectly, and asking for a concise answer trims the output tokens, which are usually the priciest per token. The scaffolding trim is the smallest number but the easiest win: every needless word in the standing instruction block is input tokens on every single call, so cutting a 400-word preamble to the 80 words that actually hold quality removes a small constant from tens of thousands of daily queries. None of this is guesswork once the spend is attributed. Turn on model invocation logging for the token counts and tag inference with application inference profiles so cost-allocation tags split the Bedrock bill by feature, and each lever above stops being a hunch and becomes a sized line you can rank.

A worked example: the docs assistant, per-query before and after

Take one representative query. Before, retrieval returns the top twenty chunks at a generous size, so the prompt carries a large block of context, a 400-word standing instruction preamble, and the short question, all sent to a large Claude model that writes a long answer. The vector store runs on an always-on cluster indexed at 1024 dimensions, and the nightly job re-embeds all few-hundred-thousand chunks. Roughly speaking, the retrieved context is the great majority of the input tokens, the preamble rides along uncached on every call, common questions are re-answered from scratch each time, and the ingestion bill is paid in full nightly for a corpus that barely changed.

After, the same query looks different at every step. Retrieval pulls a wide cheap candidate set and a reranker keeps the five best, so the context block shrinks to roughly a quarter of its size while the answer quality holds or improves. The standing preamble is trimmed and marked as a cached prefix, so its tokens are charged once and read cheaply thereafter. A response cache sits in front of the pipeline, so the large fraction of queries that repeat a known question return instantly with no retrieval and no generation. Easy questions route to a cheaper model tier and the prompt asks for a tighter answer, cutting the generation cost on the calls that do run. The store moves to a serverless shape sized to real traffic and, where quality allows, a smaller embedding dimension, cutting the standing floor. And ingestion embeds only the delta each night. No single change is the whole saving; the retrieved-context trim is the largest slice, caching removes the repeated work, and the standing-cost fixes drain the bill that was accruing whether or not anyone asked a question.

What’s worth remembering

1. In a RAG query the retrieved context is billed as input tokens on every call and is usually the largest slice, so top-k and chunk size are the biggest and most overlooked cost lever, not the generation model.
2. Lower top-k to cut retrieved tokens, and buy back the recall with reranking: retrieve a wide cheap candidate set, rerank, and send only the few best to the generation model.
3. Prompt caching helps the stable prefix, the standing instructions and fixed examples, but not the retrieved chunks, which change every query; trim the scaffolding, then cache what is left.
4. A response or semantic cache in front of the whole pipeline skips retrieval and generation on repeated questions, which recur far more than teams expect; give cached answers a time-to-live and an invalidation path.
5. The vector store bills as a standing cost by the hour with a floor (OpenSearch Serverless OCUs, Aurora Serverless ACUs); Amazon S3 Vectors trades that floor for pay-per-storage-and-query.
6. Index size is set by the embedding dimension, so a smaller-dimension model or configuration (Titan Text Embeddings V2 supports 512 and 256) shrinks storage and search cost, at a retrieval-quality trade worth measuring.
7. Do not re-embed an unchanged corpus; incremental ingestion that embeds only new and modified chunks turns a full nightly re-embed into a small delta.
8. Route easy questions to a cheaper model tier (by hand or with Bedrock Intelligent Prompt Routing) and ask for concise answers, since output tokens usually cost the most per token.
9. Every needless word in the standing instruction block is input tokens on every call; trimming the preamble removes a small constant from every query at scale.
10. You cannot cut what you have not measured; use Bedrock model invocation logging and tag inference with application inference profiles so cost-allocation tags size each lever and you pull the biggest first.