Getting Documents Into a Bedrock Knowledge Base

The situation

A knowledge team is standing up a Retrieval-Augmented Generation assistant on Amazon Bedrock. The retrieval model, the vector store, and the prompt are settled; what is not settled is how the source documents actually reach the index. The corpus is a mix: a few thousand policy PDFs sitting in Amazon S3, a live SharePoint site the operations team edits weekly, a public product documentation site that changes daily, and a Confluence space full of engineering runbooks. Some of the PDFs are plain text; a good number are layout-heavy, with pricing tables, scanned forms, and embedded diagrams that carry the actual answer.

The first attempt loaded everything from a single S3 bucket with the defaults, and retrieval was patchy. Answers grounded in the plain memos came back clean, but questions whose answer lived in a table came back wrong or empty, because the table had been flattened into a wall of numbers with no structure. Nobody could filter a query to a single department, because no metadata came along for the ride. And every content change triggered a full reprocess of the whole corpus, which was slow and cost more than it should.

The problem underneath all of it is the ingestion pipeline: which connector pulls the documents, how they get parsed, how they get chunked, what metadata rides along, and how updates flow in without reprocessing everything each time.

What actually matters

Ingestion is a pipeline, not a switch, and each stage decides something the next stage cannot recover. A document enters through a data source, gets parsed into text, gets split into chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. , gets embedded into vectors, and lands in the index. Get the parse wrong and no chunking strategy saves you; get the chunking wrong and the best embedding model still retrieves noise. The stages compound, so the choices are worth making deliberately rather than accepting the defaults across the board.

The first thing that matters is where the documents live and whether Bedrock has a managed way to reach them. S3 is the primary and simplest source, but a live SharePoint site or a Confluence space is not something you want to export to S3 by hand on a schedule; a managed connector that reads the source directly, honours its structure, and re-syncs on demand is worth a lot more than a bespoke export job you now own forever.

The second is document complexity, because it decides the parser. The default parser reads the text content of a document and does well on prose. It struggles the moment the meaning lives in layout: a pricing table, a two-column form, a figure with a caption that matters. For those, a foundation-model parser can read the document the way a person would, preserving table structure and describing images, at a higher per-document cost. Spending that cost on plain memos is waste; not spending it on the table-heavy PDFs is why the table answers came back empty.

The third is how often the data changes, because it decides the sync story. A corpus that is loaded once and rarely touched can afford a full ingestion each time. A source that changes daily needs incremental sync, where only the added, changed, and deleted documents are reprocessed, so the cost and time of an update track the size of the change rather than the size of the corpus.

The fourth is metadata and filtering. If you want a query to be scoped to one department, one product, or one date range, that metadata has to be attached at ingestion time and stored alongside each chunk. It cannot be reconstructed at query time from the vector alone. Deciding the filter dimensions up front, and attaching them as the documents come in, is what makes scoped retrieval possible at all.

Chunking and the embedding model both matter too, but both are covered in depth elsewhere and only need placing here. Chunking splits each parsed document into retrievable units, and Bedrock offers several strategies; the embedding model turns each chunk into the vector the index searches. Pick both with the retrieval-quality trade-offs in mind, then let the pipeline carry them.

What we’ll filter on

1. Source and connector, does Bedrock have a managed connector for where the documents live, or does the content have to reach S3 first?
2. Document complexity, is the meaning in the prose (default parser) or in tables, forms, and images (foundation-model parsing)?
3. Change frequency, is the corpus loaded once, or does it change often enough to need incremental sync?
4. Metadata and filtering, which dimensions must a query be able to scope to, and are they attached at ingestion?
5. Chunking and embedding fit, does the chosen chunking strategy and embedding model suit the document shape and the retrieval budget?

The ingestion landscape

Amazon S3 as the primary source. The default and simplest path: point the Knowledge Base at a bucket or prefix, and it ingests the supported document formats it finds. Everything else is, in effect, a way of getting content Bedrock can read; S3 is the one that needs no connector because the content is already sitting in object storage. If a source has no managed connector, exporting it to S3 is the fallback.

Managed connectors. Bedrock provides connectors that read directly from common content systems rather than requiring a manual export. The web crawler follows a set of seed URLs within a configured scope to pull public web content. The SharePoint, Confluence, and Salesforce connectors authenticate against those systems and ingest their documents and records in place, honouring the source structure. The value is not just convenience: a connector re-syncs against the live source, so an edit in SharePoint can flow into the index without anyone rebuilding an export.

Direct and custom ingestion. For content that does not fit a connector, documents can be ingested directly through the API, including pre-processed or application-generated content. This is the path when your pipeline already produces the text, or when the documents come from somewhere with no managed connector at all.

The default parser. Reads a document’s text content and passes it downstream. Fast, cheap, and entirely adequate for prose. Its limit is layout: it does not reliably reconstruct a table, a multi-column form, or the meaning carried by an image, so documents whose answer lives in structure come through degraded.

Foundation-model parsing. Instead of extracting raw text, an FM reads the document and produces a structured, layout-aware representation, keeping tables intact and describing images and figures. This is what rescues the table-heavy and scanned PDFs. It costs more per document because it runs a model over each one, so it earns its place on the complex documents and is waste on the plain ones. Amazon Bedrock Data Automation can serve as the parser for multimodal content, handling documents that mix text, tables, and images in one managed step.

Chunking strategies. Once parsed, a document is split into retrievable chunks. Bedrock offers default chunking, fixed-size chunking, hierarchical chunking, and semantic chunking, plus a no-chunking option for content that is already pre-chunked into one-chunk-per-file units. Which strategy suits which document shape, and the retrieval trade-offs between them, are a topic in their own right; the point at the ingestion stage is that the strategy is chosen per data source and applied as documents come in.

Metadata. Alongside each document, you can attach metadata, typically as key-value attributes, that is stored with the resulting chunks and used to filter retrieval at query time. This is what lets a query say “only the operations department” or “only documents from this year”. It has to be supplied at ingestion; the filter dimensions are a design decision, not something recoverable later.

Ingestion jobs and sync. Loading a data source runs an ingestion job that parses, chunks, embeds, and indexes its documents. On subsequent syncs, incremental sync reprocesses only what has changed, added, modified, and deleted documents, rather than the whole corpus, so an update to a fast-moving source costs and takes time in proportion to the change. A full sync reprocesses everything and is the right tool only for the first load or a deliberate rebuild.

Embedding model. Each chunk is embedded into a vector by the embedding model configured on the Knowledge Base. The choice affects retrieval quality, vector dimensionalityEmbedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching. , and cost, and it is worth a deliberate decision, but it is covered in depth elsewhere; at the ingestion stage it is one configured component the pipeline runs for you.

The pipeline reads left to right, with the two decisions that change the outcome most sitting on the parse and sync stages.

Side by side

Choice	Best for	Handles complex layout	Reprocesses only changes	Enables query filtering	Relative cost
S3 source	Content already in object storage	n/a	via incremental sync	with attached metadata	Lowest
Web crawler connector	Public sites within a scope	n/a	via incremental sync	limited	Low
SharePoint / Confluence / Salesforce connector	Live content systems	n/a	via incremental sync	with source metadata	Medium
Direct / custom ingestion	Pre-processed or connector-less content	depends on your pipeline	you control it	if you attach it	Varies
Default parser	Prose documents	✗	n/a	n/a	Lowest
Foundation-model parsing	Tables, forms, images, scans	✓	n/a	n/a	Higher (per document)
Metadata attributes	Scoped, filterable retrieval	n/a	n/a	✓	Negligible
Incremental sync	Fast-changing sources	n/a	✓	n/a	Tracks change size
Full sync	First load or rebuild	n/a	✗	n/a	Tracks corpus size

The picks in depth

The layout-heavy PDFs are the clearest case for changing the default. The default parser flattened the pricing tables into unstructured runs of numbers, which is why table questions came back empty: the retrieval was working, but the chunk it retrieved no longer held a table, just noise. Route those documents through foundation-model parsing, or through Bedrock Data Automation for the ones that also carry scanned forms and figures, so the table structure survives into the chunk and the embedding captures it. Leave the plain policy memos on the default parser; running an FM over prose that parses cleanly is spending money to gain nothing. The decision is per data source, so a bucket of complex PDFs can use FM parsing while a bucket of memos stays on the default.

The SharePoint and Confluence content is the case for connectors over exports. Hand-exporting a live site to S3 on a schedule is a job you then own, maintain, and eventually forget to run; the managed connectors read the source directly and re-sync against it, so a weekly edit in SharePoint reaches the index by running a sync rather than by rebuilding an export. Pair that with incremental sync and the cost of keeping a fast-moving source current tracks the handful of pages that changed, not the whole space. The daily-changing documentation site is the same story through the web crawler: set the scope, and each sync picks up what moved.

The filtering requirement has to be designed in at ingestion, not bolted on later. If retrieval needs to scope to a department or a product line, that dimension must be attached as metadata as the documents come in, because it is stored with the chunks and cannot be reverse-engineered from a vector afterwards. Decide the filter dimensions first, make sure the source or the ingestion path can supply them, and attach them from the start; retrofitting metadata means re-ingesting.

Chunking and the embedding model round out the pipeline and are worth choosing deliberately, but they are each a subject on their own. At this stage the useful move is to set a chunking strategy that suits each source’s document shape, confirm the embedding model fits the retrieval budget, and let the ingestion job carry both. The pipeline runs them in order; the quality comes from having made each choice on purpose.

A worked example: the four sources, configured

The team splits the corpus by source and stops treating it as one bucket with one setting.

The plain policy memos stay in their S3 prefix on the default parser, with department and effective-date metadata attached so retrieval can be scoped. They rarely change, so a full sync on first load and occasional incremental syncs after are enough.

The table-heavy pricing PDFs move to their own S3 prefix and switch to foundation-model parsing, with Bedrock Data Automation handling the ones that mix scanned forms and figures. The tables survive into the chunks, and the pricing questions that used to come back empty now retrieve the right rows.

The SharePoint operations site connects through the SharePoint connector rather than a hand-rolled export, carrying its own structure and metadata, and re-syncs incrementally each week so only the edited pages are reprocessed.

The public documentation site connects through the web crawler within a configured scope, syncing daily; incremental sync means each run reprocesses only the pages that changed overnight, not the entire site.

Same Knowledge Base, four data sources, each with the parser, sync mode, and metadata that fit its documents. The retrieval quality that the single-bucket first attempt could not reach came from the ingestion choices, not from changing the model.

What’s worth remembering

1. Ingestion is a pipeline of source, parse, chunk, embed, and index; each stage decides something the next cannot recover, so the choices compound.
2. S3 is the primary and simplest source; the web crawler, SharePoint, Confluence, and Salesforce connectors read live systems directly, and direct ingestion handles content with no connector.
3. Managed connectors beat hand-rolled exports for live sources because they re-sync against the source instead of leaving you to maintain an export job.
4. The default parser suits prose; foundation-model parsing preserves tables, forms, and images at a higher per-document cost, and Bedrock Data Automation can parse multimodal content.
5. Spend FM parsing on the layout-heavy documents and leave plain prose on the default parser; the decision is per data source.
6. Chunking offers default, fixed-size, hierarchical, and semantic strategies, plus no-chunking for pre-chunked content; choose per source with retrieval quality in mind.
7. Metadata must be attached at ingestion to enable query-time filtering; it cannot be reconstructed from the vector later, so design the filter dimensions up front.
8. Incremental sync reprocesses only added, changed, and deleted documents, so keeping a fast-moving source current costs in proportion to the change, not the corpus.
9. Use a full sync for the first load or a deliberate rebuild; use incremental sync for everything after.
10. The embedding model is a deliberate choice made once on the Knowledge Base; the ingestion job runs it for every chunk, so pick it for the retrieval budget and move on.