The situation
A regional insurer runs home, motor and small-business cover through a call centre and a claims team. The knowledge those teams need sits in about four thousand documents: policy wordings and their endorsements, claims-handling procedures, and an internal product wiki that explains how the wordings are meant to be applied. Most of it is in an S3 bucket, some of it in the wiki export that lands in the same bucket every night.
Staff currently find answers by searching filenames and reading. A new claims handler wants to know whether a bicycle stolen from a locked garden shed is covered under home contents. That means knowing which of eleven wordings applies, finding the right clause, and checking that no endorsement has replaced it. That takes minutes when it should take seconds, and the answer is only trustworthy if the handler can see the wording it came from.
Somebody has already tried the obvious thing: typing the question into a foundation model in Amazon Bedrock. It answered fluently and it answered about insurance in general, not about this insurer’s wordings. The wordings were never in the model’s training data, so the model produced something that reads like a policy answer and is not one. Pasting the documents into the prompt instead does not rescue it either, because four thousand documents is several million words and no context window holds that. Even one wording plus its endorsements is a long input to send with every question, which the context-window budget covers in its own right.
What actually matters
A model knows what it was trained on, at the moment it was trained. Anything private, internal, or newer than that is missing, and there are only two ways to change that. You can change the model, through fine-tuning or continued pre-training, which is slow, costs real money each time, and goes out of date on the day a wording is amended. Or you can change what the model is given at question time, which is cheap, takes effect immediately, and leaves the model alone. For a corpus that changes monthly, the second route is the one to reach for.
That route has a name and a fixed shape. Retrieval Augmented Generation (RAG) works like this. Each document is split into chunks of a few hundred words. An embedding model reads each chunk and turns it into a vector: a list of a few hundred or a thousand numbers that stands for what the chunk means. Two passages about stolen bicycles end up close together in that number space even when they share no words. Those vectors, the embeddings, are stored in a vector database, which is a store built to answer “which of my millions of vectors are nearest to this one” quickly. When a member of staff asks a question, the same embedding model turns the question into a vector. The store returns the handful of nearest chunks, and those chunks are pasted into the prompt above the question as context. The model then answers from text it has been handed rather than from memory. Because you know exactly which chunks you passed in, you can show the staff member the source alongside the answer.
The business applications are all variations on the same situation: the facts a model needs are yours, and they change faster than any training run. Internal helpdesk and HR question answering, where staff ask about leave policy or expense rules. Customer support grounded in the current policy wording rather than last year’s. Product search, where the shopper describes what they want instead of guessing at keywords. Anything where an answer without a checkable source is worthless. This insurer is doing the first and the second at once.
What RAG does not fix is worth stating early, because teams expect too much of it. It changes what the model knows, not how the model writes. If the answers are too long, too formal, or in the wrong format, that is a prompting or a customisation problem and retrieval will not touch it. It also caps the answer at the quality of the retrieval. If the clause that settles the question never comes back from the store, no model recovers it, and the answer gets written from the chunks that did come back. And an index that has not been updated since a wording was amended produces a confident, well-cited, wrong answer, which is more dangerous than no answer at all.
With the approach settled, two decisions are left. Whether to run the ingest-chunk-embed-retrieve pipeline yourself or hand it to a managed service, and which of the AWS vector databases holds the vectors.
What we’ll filter on
- Operational ownership. How much of the chunking, embedding, indexing and retrieval does the team write and run, and how much arrives managed?
- Something you already run. Is this a database the organisation already operates, backs up and secures, or a new thing to learn?
- Shape of the retrieval. Similarity on its own, similarity with metadata filters, or similarity plus relationships between documents.
- Where the data already lives. Files in a bucket, rows in a relational database, or a graph of connected records.
- Cost floor. What the store costs each month before anybody asks a question, which matters most for a small corpus.
- Citations. Whether the source of each retrieved chunk comes back with it, so the answer can be checked.
The landscape
Two ways to build it
Amazon Bedrock Knowledge Bases is the managed version of the whole pipeline. You point it at a data source (an S3 bucket, most commonly), choose an embedding model, and choose a vector store. It reads the documents, splits them into chunks, embeds each chunk, writes the vectors to the store, and keeps the store in step with the source when you run a sync. At query time it gives you two calls: one returns the nearest chunks with their source locations, and one goes further and returns a generated answer with citations attached. Very little of it is code you own.
Building it yourself means the same steps, written out. Your own job splits the documents, calls an embedding model through the Bedrock API, writes the vectors to a store you chose, and your application code runs the similarity query and assembles the prompt. You get to decide exactly how documents are chunked, how retrieval is filtered and ranked, and how the prompt is built. You also own every part of it, including the job that keeps the store fresh.
Four places to put the vectors
Amazon OpenSearch Service is the general default and the one most teams land on. It does vector similarity search alongside ordinary keyword search and metadata filtering, so a query can ask for chunks that are similar in meaning and also carry document_type = wording and an effective date in range. It comes in two shapes. A provisioned domain is a cluster you size, tune and pay for by the hour. A serverless collection removes the sizing decision and bills by capacity units instead, with a floor per collection. Both work as a Knowledge Bases vector store.
Amazon Aurora and Amazon RDS for PostgreSQL both run PostgreSQL with the pgvector extension, which adds a vector column type and nearest-neighbour search to ordinary SQL. The attraction is that there is one database rather than two. If the documents, or the records they relate to, already live in Postgres, the vectors sit in a table next to them. They are backed up with them, secured by the same grants, and filtered in the same query with a normal WHERE clause. Aurora is the AWS-built engine with faster failover and the option of scaling capacity down when the load is small; RDS for PostgreSQL is the standard engine on managed instances. Choose between them the way you would for any other Postgres workload.
Amazon Neptune is a graph database, and Neptune Analytics adds vector similarity to it. Choose it when finding similar text is only half of what retrieval has to do, and the other half is following relationships. This endorsement amends that wording; this procedure applies to those product lines; this claim type escalates to that team. A question that needs the chunk and everything connected to it is a graph question. A question that needs the five most relevant passages is not, and a graph database is a heavy way to answer it.
Evaluation
Side by side
| Option | Pipeline is managed | Vector search | Filters on metadata | Relational data alongside | Graph relationships | Low cost when small |
|---|---|---|---|---|---|---|
| Bedrock Knowledge Bases (over any store below) | ✓ | ✓ | ✓ | ✗ | ✗ | depends on the store |
| Amazon OpenSearch Service, serverless collection | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ |
| Amazon OpenSearch Service, provisioned domain | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ |
| Amazon Aurora, pgvector | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ |
| Amazon RDS for PostgreSQL, pgvector | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ |
| Amazon Neptune Analytics | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ |
The first row sits apart from the others because it is not a store. Knowledge Bases is the pipeline, and it writes into one of the stores underneath it, so the first column and the rest of the table are separate choices rather than competing ones. Reading the store rows on their own, the three vector-search columns are ticked everywhere, which is the useful finding: for a corpus of this size, any of them will retrieve well enough. What separates them is the last three columns, and those are questions about the estate rather than about retrieval.
Which store the estate picks
The solution
Run Amazon Bedrock Knowledge Bases over an Amazon OpenSearch Serverless collection. Nothing in this corpus needs graph traversal, nothing already lives in Postgres, and there is no team that wants a cluster to size, so the two overriding answers are absent and the default stands. Managed ingestion removes the parts most likely to be built badly the first time: the chunker, the embedding job, and the code that keeps the store in step with the bucket. Point the knowledge base at the S3 prefix, pick an embedding model, run the first sync, and the retrieval half of the application exists.
Attach metadata to each document so retrieval can be narrowed. Product line, document type (wording, endorsement, procedure, wiki page), and effective date are the three that matter here. A filter on effective date is what stops a superseded wording being retrieved as though it were current, and it is far more reliable than hoping the model notices the date inside the text. Put the effective date in the chunk text as well, so the answer can name the version it read.
Use the call that returns an answer with citations rather than the one that returns chunks, at least to begin with, and render each citation as a link to the source document. A claims handler who can open the clause will trust the tool; one who has to take the answer on faith will not use it twice. The developer-level version of that requirement, where every sentence has to trace to a retrieved passage, is worked through in a build where citations are non-negotiable.
Four things go wrong from here, and they go wrong in roughly this order. Freshness first: the wiki export lands nightly and wordings are amended monthly, so the sync has to run on that rhythm rather than when somebody remembers. A stale index answers confidently and cites a clause that no longer applies, and nothing downstream will flag it; keeping an index in step with changing sources is a design problem of its own. Second, retrieval quality sets the ceiling on answer quality. Keep thirty real staff questions with the document that should answer each one, and check that the right chunk comes back before blaming the model for a poor answer. Third, tone and format complaints are prompt problems. The instruction that sits above the retrieved context decides length, register, and what happens when the chunks do not cover the question. Tell it to say so rather than fill the gap. Fourth, permissions: a knowledge base does not know who is asking, so any document that only some staff may read either stays out of this corpus or goes into a second knowledge base with its own access path.
The choice of store is reversible and the choice of embedding model is less so, because changing the embedding model means re-embedding every chunk. At four thousand documents that is a job of minutes, which is one more reason this decision is easier now than it will be at ten times the size. The trade-offs across the stores at larger scale are worked through in a developer-level comparison of the same options.
Worked example
Take the bicycle question and follow it through.
A claims handler types “is a bike stolen from a locked shed covered on home contents”. The application sends that sentence to the same embedding model that indexed the corpus, and gets back one vector.
That vector goes to the OpenSearch Serverless collection with a filter attached: product line is home, and the effective date range includes today. The store returns the five nearest chunks. Three come from the current home contents wording: the specified-items clause, the away-from-home clause, and the outbuildings definition. The other two are a claims-handling procedure about theft without forced entry and a wiki page explaining the outbuildings definition in plain English.
Those five chunks are assembled into a prompt above the handler’s question. The instruction above them says to answer only from the passages supplied, to name the document each fact came from, and to say plainly when the passages do not settle the question.
The model answers: cover applies up to the outbuildings limit where the shed was locked, the limit is lower than the main contents limit, and a bicycle above a stated value needs to have been specified. Three citations sit under the answer, and the handler opens the outbuildings definition to confirm it before quoting the limit to the customer.
Now amend the wording so that the outbuildings limit changes, and run the same question before the next sync. The retrieval returns the old chunk, the model answers from it, and the citation makes the answer look more trustworthy rather than less. That is the failure this design has to be defended against, and the defence is the sync schedule and the effective-date filter rather than anything about the model.
What’s worth remembering
- Retrieval Augmented Generation (RAG) chunks your documents, turns each chunk into an embedding with an embedding model, stores the embeddings in a vector database, then embeds the user’s question and pastes the nearest chunks into the prompt as context.
- The business applications are the ones where the facts are yours and change faster than a training run: internal helpdesk and HR question answering, customer support grounded in current policy, and product search.
- Amazon Bedrock Knowledge Bases runs the ingest, chunk, embed, index and retrieve pipeline for you and returns answers with citations, so the build decision is usually managed pipeline versus writing all of it yourself.
- The four AWS vector databases in scope are Amazon OpenSearch Service (the general default), Amazon Aurora and Amazon RDS for PostgreSQL (pgvector, when the data already lives in Postgres), and Amazon Neptune (when retrieval must walk relationships as well as similarity).
- RAG changes what a model knows, not how it writes, so tone and format problems are prompt problems and retrieval quality sets the ceiling on answer quality.
- An index that has fallen behind its sources produces a confident, cited, wrong answer, which is why the sync schedule and a filter on effective date are part of the design rather than an afterthought.