The situation
A regional insurer runs home, motor and small-business cover through a call centre and a claims team. The knowledge those teams need sits in about four thousand documents: policy wordings and their endorsements, claims-handling procedures, and an internal product wiki that explains how the wordings are meant to be applied. Most of it is in an S3 bucket, some of it in the wiki export that lands in the same bucket every night.
Staff currently find answers by searching filenames and reading. A new claims handler needs to know whether a bicycle stolen from a locked garden shed is covered under home contents. That means knowing which of eleven wordings applies, finding the right clause, and checking that no endorsement has replaced it. That takes minutes when it should take seconds, and the answer is only trustworthy if the handler can see the wording it came from.
Somebody has already tried the obvious thing: typing the question into a foundation model in Amazon Bedrock. The reply was fluent, and it was about insurance in general rather than about this insurer’s wordings. Those wordings were never in the training data, so the output reads like a policy answer without being one. Pasting the documents into the prompt instead does not rescue it either, because four thousand documents is several million words and no context window holds that. Even one wording plus its endorsements is a long input to send with every question, which the context-window budget covers in its own right.
What actually matters
A model can only draw on its training data, as that data stood on the day it was trained. Anything private, internal, or newer than that is absent, and there are two ways to change it. You can change the model, through fine-tuning or continued pre-training, which is slow, costs real money each time, and goes out of date on the day a wording is amended. Or you can change what the model is given at question time, which takes effect immediately and leaves the model itself untouched. For a corpus that changes monthly, the second route is the one to reach for.
That route has a name and a fixed shape. Retrieval Augmented Generation (RAG) works like this. Each document is split into chunks of a few hundred words. An embedding model turns each chunk into a vector: a list of a few hundred or a thousand numbers that stands for what the chunk means. Two passages about stolen bicycles end up close together in that number space even when they share no words. Those vectors, the embeddings, are stored in a vector database, which is a store built to answer “which of my millions of vectors are nearest to this one” quickly. When a member of staff asks a question, the same embedding model turns the question into a vector. The store returns the handful of nearest chunks, and those chunks are pasted into the prompt above the question as context. The model then answers from the text it was handed rather than from its training data. Because you know exactly which chunks you passed in, you can show the staff member the source alongside the answer.
The business applications are all variations on the same situation: the facts a model needs are yours, and they change faster than any training run. Internal helpdesk and HR question answering, where staff ask about leave policy or expense rules. Customer support grounded in the current policy wording rather than last year’s. Product search, where the shopper describes what they want instead of guessing at keywords. Anything where an answer without a checkable source is worthless. This insurer is doing the first and the second at once.
What RAG does not fix is worth stating early, because teams expect too much of it. It changes the facts in front of the model. It does not change the writing, so answers that are too long, too formal, or in the wrong format stay that way until the prompt or the model changes. Retrieval also caps answer quality. If the clause that settles the question never comes back from the store, no model recovers it, and the answer gets written from the chunks that did come back. An index that has not been updated since a wording was amended returns a fluent, well-cited, wrong answer, which is more dangerous than no answer at all.
With the approach settled, two decisions are left. Whether to run the ingest-chunk-embed-retrieve pipeline yourself or hand it to a managed service, and where the vectors live: one of the AWS vector databases, or a vector index in Amazon S3.
What we’ll filter on
- Operational ownership. How much of the chunking, embedding, indexing and retrieval does the team write and run, and how much arrives managed?
- Something you already run. Is this a database the organisation already operates, backs up and secures, or a new thing to learn?
- Shape of the retrieval. Similarity on its own, similarity with metadata filters, similarity alongside keyword matching, or similarity plus relationships between documents.
- Where the data already lives. Files in a bucket, rows in a relational database, or a graph of connected records.
- Cost floor. What the store costs each month before anybody asks a question, which matters most for a small corpus.
- Citations. Whether the source of each retrieved chunk comes back with it, so the answer can be checked.
The landscape
Two ways to build it
Amazon Bedrock Knowledge Bases is the managed version of the whole pipeline. You point it at a data source (an S3 bucket, most commonly), choose an embedding model, and choose a vector store. It reads the documents, splits them into chunks, embeds each chunk, writes the vectors to the store, and keeps the store in step with the source when you run a sync. At query time there are two calls. Retrieve returns the nearest chunks with their source locations, and RetrieveAndGenerate returns a written answer with citations attached. Very little of it is code you own.
Building it yourself means the same steps, written out. Your own job splits the documents, calls an embedding model through the Bedrock API, writes the vectors to a store you chose, and your application code runs the similarity query and assembles the prompt. You get to decide exactly how documents are chunked, how retrieval is filtered and ranked, and how the prompt is built. You also own every part of it, including the job that keeps the store fresh.
Five places to put the vectors
Amazon OpenSearch Service is the general default and the one most teams land on. It does vector similarity search alongside ordinary keyword search and metadata filtering, so a query can ask for chunks that are similar in meaning and also carry document_type = wording and an effective date in range. It comes in two shapes. A provisioned domain is a cluster you size, tune and pay for by the hour. A serverless collection removes the sizing decision and bills by OpenSearch Compute Unit instead, at USD$0.24 per OCU-hour, with the minimum and maximum set per collection group. A NextGen collection can be set to a minimum of zero and scales to no capacity after ten minutes without traffic; an older classic collection bills a floor of two OCUs whether or not anybody queries it. Both shapes work as a Knowledge Bases vector store.
Amazon S3 Vectors is vector storage inside S3 itself. A vector bucket holds vector indexes, there is no cluster or instance to provision, and the bill is storage plus requests. AWS positions it for workloads queried infrequently, with sub-second responses and as low as 100 milliseconds once queries are frequent enough. Metadata attached to each vector is filterable by default, and it is a supported Knowledge Bases store, so it removes the standing capacity charge the other stores carry. What it does not do is hybrid search over vectors and raw text together.
Amazon Aurora and Amazon RDS for PostgreSQL both run PostgreSQL with the pgvector extension, which adds a vector column type and nearest-neighbour search to ordinary SQL. The attraction is that there is one database rather than two. If the documents, or the records they relate to, already live in Postgres, the vectors sit in a table next to them. They are backed up with them, secured by the same grants, and filtered in the same query with a normal WHERE clause. Aurora is the AWS-built engine with faster failover, and Aurora Serverless v2 can be set to a minimum of zero capacity units so an idle cluster pauses; RDS for PostgreSQL is the standard engine on managed instances. One difference decides between them here. Knowledge Bases connects to an Aurora cluster and not to an RDS for PostgreSQL instance, so picking RDS means writing the pipeline yourself.
Amazon Neptune is a graph database, and Neptune Analytics adds vector similarity to it. Choose it when finding similar text is only half of what retrieval has to do, and the other half is following relationships. This endorsement amends that wording; this procedure applies to those product lines; this claim type escalates to that team. A question that needs the chunk and everything connected to it is a graph question. A question that needs the five most relevant passages is not, and a graph database is a heavy way to answer it. The vector index on a Neptune Analytics graph can only be created when the graph is created, so the embedding dimension is fixed at that moment.
Evaluation
Side by side
| Option | Knowledge Bases store | Filters on metadata | Hybrid search | Relational data alongside | Graph relationships | Idles at no compute cost |
|---|---|---|---|---|---|---|
| Amazon S3 Vectors | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ |
| Amazon OpenSearch Service, serverless collection | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
| Amazon OpenSearch Service, provisioned domain | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| Amazon Aurora, pgvector | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ |
| Amazon RDS for PostgreSQL, pgvector | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ |
| Amazon Neptune Analytics | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ |
Every row does similarity search, which is why that column is not in the table: at four thousand documents any of them retrieves well enough, and ranking them on nearest-neighbour quality would be inventing a difference. The first column is the one that catches teams out. Knowledge Bases writes into five of these six, and an RDS for PostgreSQL instance is not one of them, so choosing pgvector on RDS rules out the managed pipeline as well. The remaining columns are questions about the estate and the query pattern rather than about retrieval.
Which store the estate picks
The solution
Run Amazon Bedrock Knowledge Bases over an Amazon OpenSearch Serverless collection. Nothing in this corpus needs graph traversal, nothing already lives in Postgres, and no team here has asked for a cluster to size. S3 Vectors costs less and is the better answer for a corpus queried a few times a day. This one is queried all day by a call centre, and the questions arrive full of clause names and policy numbers, so hybrid search across both the vectors and the raw text justifies a collection that stays warm. Managed ingestion removes the parts most likely to be built badly the first time: the chunker, the embedding job, and the code that keeps the store in step with the bucket. Point the knowledge base at the S3 prefix, pick an embedding model, run the first sync, and the retrieval half of the application exists.
Attach metadata to each document so retrieval can be narrowed. Product line, document type (wording, endorsement, procedure, wiki page), and effective date are the three that matter here. A filter on effective date stops a superseded wording being retrieved as though it were current, and it is far more reliable than leaving the date buried in the text for the model to weigh. Range filters compare numbers rather than strings, so store the date as a number: epoch seconds, or 20260827. Put it in the chunk text as well, so the answer can name the version it was written from.
Use RetrieveAndGenerate rather than Retrieve, at least to begin with, and render each citation as a link to the source document. A claims handler who can open the clause will trust the tool; one who has to take the answer on faith will not use it twice. The developer-level version of that requirement, where every sentence has to trace to a retrieved passage, is worked through in a build where citations are non-negotiable.
Four things go wrong from here, and they go wrong in roughly this order. Freshness first: the wiki export lands nightly and wordings are amended monthly, so the sync has to run on that rhythm rather than when somebody remembers. A stale index returns a fluent answer citing a clause that no longer applies, and nothing downstream will flag it; keeping an index in step with changing sources is a design problem of its own. Second, retrieval quality sets the ceiling on answer quality. Keep thirty real staff questions with the document that should answer each one, and check that the right chunk comes back before blaming the model for a poor answer. Third, tone and format complaints are prompt problems. The instruction that sits above the retrieved context sets length, register, and what happens when the chunks do not cover the question. Tell it to say so rather than fill the gap. Fourth, permissions: a knowledge base applies no per-user access control of its own at retrieval time, so any document that only some staff may read stays out of this corpus, goes into a second knowledge base with its own access path, or is fenced off by a metadata filter the application sets from the caller’s role.
The choice of store is reversible and the choice of embedding model is less so, because changing the embedding model means re-embedding every chunk. At four thousand documents that is a job of minutes, which is one more reason this decision is easier now than it will be at ten times the size. The trade-offs across the stores at larger scale are worked through in a developer-level comparison of the same options.
Worked example
Take the bicycle question and follow it through.
A claims handler types “is a bike stolen from a locked shed covered on home contents”. The application sends that sentence to the same embedding model that indexed the corpus, and gets back one vector.
That vector goes to the OpenSearch Serverless collection with a filter attached: product line is home, and the effective date range includes today. The store returns the five nearest chunks. Three come from the current home contents wording: the specified-items clause, the away-from-home clause, and the outbuildings definition. The other two are a claims-handling procedure about theft without forced entry and a wiki page explaining the outbuildings definition in plain English.
Those five chunks are assembled into a prompt above the handler’s question. The instruction above them says to answer only from the passages supplied, to name the document each fact came from, and to say plainly when the passages do not settle the question.
The model answers: cover applies up to the outbuildings limit where the shed was locked, the limit is lower than the main contents limit, and a bicycle above a stated value needs to have been specified. Three citations sit under the answer, and the handler opens the outbuildings definition to confirm it before quoting the limit to the customer.
Now amend the wording so that the outbuildings limit changes, and run the same question before the next sync. The retrieval returns the old chunk, the model answers from it, and the citation makes the answer look more trustworthy rather than less. That is the failure this design has to be defended against, and the defence is the sync schedule and the effective-date filter rather than anything about the model.
What’s worth remembering
- Retrieval Augmented Generation (RAG) chunks your documents, turns each chunk into an embedding with an embedding model, stores the embeddings in a vector database, then embeds the user’s question and pastes the nearest chunks into the prompt as context.
- The business applications are the ones where the facts are yours and change faster than a training run: internal helpdesk and HR question answering, customer support grounded in current policy, and product search.
- Amazon Bedrock Knowledge Bases runs the ingest, chunk, embed, index and retrieve pipeline for you, and
RetrieveAndGeneratereturns an answer with citations attached, so the build decision is usually managed pipeline versus writing all of it yourself. - The AWS vector databases are Amazon OpenSearch Service (the general default), Amazon Aurora and Amazon RDS for PostgreSQL (pgvector, when the data already lives in Postgres), and Amazon Neptune (when retrieval must walk relationships as well as similarity); Amazon S3 Vectors adds a vector index with no capacity to provision, and Knowledge Bases writes to all of them except an RDS for PostgreSQL instance.
- RAG changes the facts in front of the model and leaves the style of the writing alone, so tone and format complaints are prompt problems, and retrieval quality sets the ceiling on answer quality.
- An index that has fallen behind its sources returns a fluent, cited, wrong answer, which is why the sync schedule and a filter on effective date are part of the design rather than an afterthought.