The situation
A product team wants a support assistant that answers customer questions from three bodies of knowledge. The product manual is a 400-page PDF that engineering edits weekly. The pricing sheet is a set of Markdown files in a Git repo that finance updates at month-end. The operations runbook is a Confluence space that on-call engineers amend throughout the day, sometimes hourly.
Today there’s no assistant at all, customers raise tickets and humans search the three sources by hand. Leadership wants a first version in front of customers in six weeks, accurate enough that wrong answers are rare and caught fast, and maintainable enough that two engineers can keep it running alongside their other work. No base Bedrock model has seen any of the three sources. The manual and the runbook were never public, and any model’s training data closed before the last pricing change.
Fine-tuning is off the table for a reason worth naming: the content changes faster than any training pipeline could keep up. Retraining weekly for the manual, monthly for pricing, hourly for the runbook is a job, not a project.
What actually matters
Before reaching for an architecture, it’s worth asking what we’re actually trading.
Retrieval-augmented generation puts the answer in the prompt rather than in the weights. We turn the three knowledge sources into a searchable corpus, retrieve the relevant passages when a question arrives, place them in the PromptThe input you hand to an LLM – system instructions, user message, examples, retrieved documents, tool descriptions, the lot. at InferenceRunning a trained model to produce output – as opposed to training it. time, and the model composes prose from them. The model supplies the comprehension and the writing. The retrieval system supplies the passages.
That framing exposes the decisions. The first is ingestion: how documents get from their home (S3, Git, Confluence, SharePoint) into a VectorAn ordered list of numbers – in AI usage, almost always an embedding – and by extension the databases that index them for nearest-neighbour search. index, and whether we write that pipeline or have AWS run it. The second is chunking. Documents are too big to embed whole, so they get split, and how we split affects what the retriever can find: a paragraph-level chunk answers “what is the warranty period?” cleanly, a page-level chunk buries the answer. The third is EmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together., since we need a model that turns text into vectors and the vectors’ quality caps the retriever’s quality. The fourth is storage: vectors live somewhere searchable, managed or self-run, cheap or fast or both. The fifth is retrieval strategy, whether pure vector, hybrid with keyword, re-ranked, or filtered on metadata. The sixth is the prompt, how the retrieved passages meet the user’s question and how the model is instructed to cite sources and to say when the passages contain no answer. The seventh, always, is observability: what the retriever returned, what the model did with it, and which step failed when the answer was wrong.
The sharp edges are not where teams expect them. Chunking, embedding choice and metadata filtering decide what the retriever can find at all, and a passage that never surfaces cannot be reasoned over no matter which model receives the prompt. That shapes which knobs matter.
The team’s planning horizon belongs in the picture too. With two engineers, a managed service that reaches a working assistant in two weeks and runs the plumbing is the right trade. With twenty engineers and retrieval as the product, a custom stack that exposes every knob is.
What we’ll filter on
Distilling that into filters:
- Time to first working system, weeks or months?
- Flexibility at each step, can we swap embedding model, chunking strategy, retriever?
- Operational surface, how much infrastructure do we run ourselves?
- Cost shape, per-token, per-call, per-hour, and how they compound?
- Source-of-truth fidelity, how quickly do changes in the underlying docs show up in answers?
The landscape
Bedrock Knowledge Bases. The managed option, and now two of them. A customer-managed knowledge base has us bring a vector store (OpenSearch Serverless, an OpenSearch managed cluster, S3 Vectors, Aurora PostgreSQL with pgvector, Neptune Analytics for GraphRAG, or a third-party store such as Pinecone, MongoDB Atlas or Redis Enterprise Cloud) and choose an embedding model (Titan Text Embeddings V2, Cohere Embed English v3, Cohere Embed Multilingual v3). Connectors cover S3, Confluence, SharePoint, Salesforce, a web crawler, and a custom source we push documents into. Bedrock runs parsing, chunking, embedding and retrieval, and RetrieveAndGenerate turns a query into a grounded answer with citations. A Bedrock Managed knowledge base goes further, supplying the datastore, the embedding model and a reranker, adding Google Drive and OneDrive to the connector list, and billing per retrieval call; AWS now recommends it as the starting point. Either way there is no built-in sync schedule. Ticks attributes 1, 3, and 5; falls short on 2.
LangChain (or LlamaIndex) on our own infrastructure. A framework-assembled stack. LangChain wraps the pieces, loaders per source type, splitters for chunking, embeddings wrappers around Bedrock or Cohere or OpenAI, vector stores (Chroma, Pinecone, pgvector, OpenSearch, FAISS), retrievers (vector, hybrid BM25+vector, multi-query, parent-document), and a chain that glues retrieval to generation. Runs wherever Python runs: Lambda, Fargate, EKS. More knobs exposed; more moving parts to own. Ticks attributes 2 and (partly) 4; falls short on 1 and 3.
Custom pipeline. We write the ingestion, chunking, embedding call, vector write, retriever, and prompt assembly by hand, using Bedrock’s InvokeModel for embeddings and generation and a vector store of our choice. No framework. Maximum control, custom chunking, custom metadata, custom retriever logic, custom prompt assembly, custom evaluation. Maximum code. Ticks 2 to the hilt; loses badly on 1 and 3.
An agent on Amazon Bedrock AgentCore with a knowledge base behind it. AgentCore is a set of separately usable services (Runtime, Memory, Gateway, Identity, Observability among them) composed around an agent loop, and a managed knowledge base reaches that loop through AgentCore Gateway. If the assistant has to do things beyond answering, look up a subscription, trigger a refund, that layer is what provides it. For pure question-answering it adds a tier that does no work here, and Knowledge Bases alone are enough. Amazon Bedrock Agents, the older feature, is now Amazon Bedrock Agents Classic, in maintenance mode and closed to new customers, so don’t start there.
Fine-tuning instead of retrieval. Mentioned only to rule it out. Fine-tuning embeds knowledge in model weights, which is slow to update and expensive to retrain. For content that changes weekly or hourly the model would be out of date before it shipped. Fine-tuning suits style, format, and domain vocabulary, not facts that mutate.
Evaluation
Side by side
| Option | Time to first system | Flexibility | Ops surface | Cost shape | Source fidelity |
|---|---|---|---|---|---|
| Bedrock Knowledge Bases | Days | Medium | Minimal | Per retrieval call or vector-store hours, plus tokens | Sync on demand, on our own schedule |
| LangChain stack | Weeks | High | Moderate | Compute + vector store + per-token | Whatever we build |
| Custom pipeline | Weeks to months | Total | Heavy | Compute + vector store + per-token | Whatever we build |
| AgentCore agent + KB | Days (for Q&A) | Medium | Minimal | AgentA system that wraps an LLM with tools, memory, and a loop, so it can take multi-step actions toward a goal rather than just answering one prompt. consumption + KB retrieval | Same as KB |
| Fine-tuning | Weeks per update | Wrong axis | Moderate | Training + hosting | Stale between trainings |
Reading the table against our situation: a six-week deadline with two engineers, content changing weekly to hourly, and accuracy that matters but doesn’t need state-of-the-art retrieval research. Knowledge Bases is the path of least resistance that also ticks the attributes this team cares about.
The three shapes, side by side
The solution
Knowledge Bases wins the shape test for this team. The interesting work is setting it up well, not building it from parts.
Data sources. Three sources map to three data sources, comfortably inside the limit of five per knowledge base. The 400-page manual goes into an S3 bucket, where each file has to stay under the 50 MB ingestion limit. The pricing sheet goes into the same bucket under a different prefix. The runbook uses the Confluence connector, pointed at one space, with either an Atlassian API token or OAuth 2.0 credentials in Secrets Manager. Two caveats on that connector: it is a preview release, and it works only with an OpenSearch Serverless vector store.
Syncing. Bedrock has no built-in schedule, so every refresh is a StartIngestionJob call. A GitHub Action fires one on each commit to main for the pricing sheet. EventBridge Scheduler fires one weekly for the manual and every fifteen minutes for the runbook, which is close enough to hourly edits that on-call engineers see their own changes. Syncing is incremental, so only added, changed and deleted documents are reprocessed, and one ingestion job runs per data source at a time, which sets the floor on how tight the cadence can go.
Chunking strategy. The default splits content into chunks of roughly 300 tokens while honouring sentence boundaries, and that suits the pricing sheet and the runbook, both short self-contained sections. Fixed-size chunking instead lets us set a token ceiling and an overlap percentage. The manual is better served by hierarchical chunking, where we set a parent size, a child size, and an overlap in tokens: retrieval matches the small child chunks, then substitutes their parents before generation, so the context around the match travels with it. Expect one consequence. numberOfResults counts child chunks, and children sharing a parent collapse into that one parent, so fewer results come back than were asked for. Chunking strategy is fixed when the data source is created and cannot be changed afterwards.
Embedding model. Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0) at 1024 dimensions, with 256 and 512 also available where index size matters more than accuracy, and an 8K-token input window. Cohere Embed English v3 and Embed Multilingual v3 are the alternatives, both fixed at 1024 dimensions; take the multilingual one if the corpus crosses languages. Embedding quality caps retrieval quality, so this is a choice to measure rather than argue about. Assemble real questions with known answers, run them against each candidate, and switch on the numbers.
Vector store. With the Confluence connector in the design, OpenSearch Serverless is the only option. It is also the quick-create path from the knowledge base console, and capacity there is a minimum and maximum OCU band on the collection group rather than a cluster to size, with the minimum able to sit at zero. Aurora PostgreSQL with pgvector and S3 Vectors are both worth weighing elsewhere, S3 Vectors in particular for corpora queried infrequently. Neither is open to us while the runbook rides the Confluence connector, so “migrate later if the bill justifies it” would mean moving the runbook off that connector first.
Retrieval. RetrieveAndGenerate runs query embedding, vector search and generation in one call, returning output.text plus a citations array that ties spans of the answer to the chunks behind them. RetrieveAndGenerateStream does the same and streams the response. Retrieve returns the chunks and their relevance scores with no generation, which makes it the backing for a debug endpoint that shows what the retriever found, the most useful observability surface in a RAG system. One boundary to know: RetrieveAndGenerate doesn’t work against a Bedrock Managed knowledge base, where retrieval goes through Retrieve or AgenticRetrieveStream and generation is a call we make ourselves.
Metadata filtering. Each data source carries metadata, auto-detected by the connector or supplied alongside S3 objects in a .metadata.json file. Tag manual chunks source: "manual", pricing chunks source: "pricing", runbook chunks source: "runbook". A query passes a filter inside retrievalConfiguration.vectorSearchConfiguration, combining up to five operators per group under andAll or orAll. A customer-facing assistant excludes the internal runbook with a notEquals, an internal assistant drops the filter, and neither needs the index rebuilt. The in and notIn operators are best supported on OpenSearch Serverless, which is where we already are.
The prompt template. The default system prompt is serviceable; a custom textPromptTemplate in generationConfiguration is where the last stretch of accuracy comes from. Keep the $output_format_instructions$ placeholder, because citations don’t render in the response without it. Past that, instruct the model to answer only from the retrieved passages and to state plainly when they hold no answer, to write in a defined tone, and to ask a clarifying question when the query is ambiguous instead of picking a reading.
Worked example
A customer asks: “What’s the difference between the Pro and Team plans, and when does the Team plan discount kick in?”
RetrieveAndGenerateembeds the query with Titan Text Embeddings V2.- OpenSearch Serverless runs a k-nearest-neighbour search over the indexed chunks, filtered to exclude
source: "runbook". Up to five results come back, which is the default: two pricing sections describing each plan, one manual section on volume tiers, two adjacent pricing sections on overage and billing. - Bedrock assembles the custom prompt from those chunks, the session history and the question, then calls the generation model. The model returns the answer text.
- The response carries
output.textand acitationsarray, each entry pairing a span of that answer with the chunks supporting it and their source locations. The web front-end renders those as clickable links to the source documents. - Bedrock also returns a
sessionId. The application can’t choose one, so it stores what came back and sends it on the next turn, which is how follow-up questions keep their context.
The bill for one question is the query embedding tokens, the generation tokens over the assembled prompt, the vector-store time, and on a managed knowledge base the retrieval charge (USD$1.00 per 1,000 Retrieve calls at list price). What the team had to build: three data-source configs, one custom prompt template, one GitHub Action, two EventBridge schedules, and a thin API Gateway and Lambda in front of RetrieveAndGenerate.
What’s worth remembering
- Retrieval puts the answer in the prompt instead of the weights, so the corpus can change daily with nothing retrained.
- Chunking decides what the retriever can find, and it’s fixed when the data source is created: roughly 300-token default chunks for short sections, hierarchical parent and child chunks for long structured documents.
- Titan Text Embeddings V2 offers 256, 512 or 1024 dimensions; Cohere Embed English v3 and Multilingual v3 are fixed at 1024. Pick one, measure it against real questions, switch only on evidence.
- The connector chosen constrains the rest of the design. Confluence is a preview connector that works only with OpenSearch Serverless, which rules out pgvector and S3 Vectors for the whole knowledge base.
- Nothing syncs on its own.
StartIngestionJob, from the console or from something we schedule, is what carries a document change into an answer. - Fine-tuning suits style, format and domain vocabulary, not facts that change weekly.