Getting Documents Into a Bedrock Knowledge Base

The situation

A knowledge team is standing up a retrieval-augmented generation assistant on Amazon Bedrock. The vector store, the retrieval settings, and the prompt are settled. What is not settled is how the source documents reach the index. The corpus is mixed: a few thousand policy PDFs in Amazon S3, a SharePoint site the operations team edits weekly, a public documentation site that changes daily, and a Confluence space of engineering runbooks. Some of the PDFs are plain text. A good number are layout-heavy, with pricing tables, scanned forms, and diagrams that carry the actual answer.

The first attempt loaded everything from one S3 bucket on the defaults, and retrieval was patchy. Answers grounded in the plain memos came back clean. Questions whose answer sat inside a table came back wrong or empty, because the table had been flattened into a run of numbers with no structure. Nothing could be scoped to a single department, because no metadata was attached to any document. A nightly export also rewrote every object in the bucket, so each sync saw the whole corpus as modified and re-ingested it.

Underneath all of that is the ingestion pipeline: which data source the documents arrive through, how they are parsed, how they are chunked, what metadata is attached, and how a change reaches the index.

What actually matters

Ingestion is a pipeline, and each stage limits what the next one can do. A document enters through a data source, is parsed into text, is split into chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense., is embedded, and lands in the index. A bad parse cannot be rescued by a chunking strategy, and bad chunks cannot be rescued by a better embedding model. The stages compound, so each default deserves a decision.

The first question is where the documents live and whether a managed connector reaches them. S3 needs no connector credentials, because the content already sits in object storage. A live SharePoint site or a Confluence space is another matter. Exporting either on a schedule leaves you owning the export job, while a connector authenticates against the source and re-crawls it on every sync. The trade is maturity. Outside S3 and the custom data source, every connector is still a preview release, and each one requires an OpenSearch Serverless vector store.

The second question is document complexity, which sets the parser choice. The default parser extracts text from .txt, .md, .html, Word, Excel and PDF files, and it does well on prose. It does not reconstruct a pricing table, a two-column form, or the meaning carried by a figure. A layout-aware parser keeps tables intact and describes images, and it charges per page or per token. That choice is made per data source. Once made, it applies to every PDF in that source, including the ones that are only text.

A related constraint sits under it. Images, audio and video are only processed from S3 and custom data sources; the SharePoint, Confluence, Salesforce and web crawler connectors skip those files. A runbook whose answer is a screenshot does not reach the index through the Confluence connector at all, whatever parser is configured.

The third question is how a change reaches the index. Syncing is incremental by design: Bedrock re-ingests the documents added, modified or deleted since the last sync, and skips the rest. What defeats it is a landing process that rewrites files it has not changed, which is what turned the team’s nightly export into a full re-ingest. Fix the export and the time and cost of an update track the size of the change.

The fourth is metadata and filtering. Scoping a query to a department, a product line or a date range needs that attribute attached at ingestion and stored with each chunk. It cannot be reconstructed from a vector afterwards. Decide the filter dimensions before the first load, because adding one later means re-ingesting the documents that need it.

Chunking and the embedding model matter too, and both are subjects in their own right. Chunking splits each parsed document into retrievable units, and the strategy is fixed once the data source is connected. The embedding model turns each chunk into the vector the index searches.

What we’ll filter on

1. Data source, is there a managed connector for where the documents live, or does the content have to reach S3 first?
2. Document complexity, is the meaning in the prose, or in tables, forms, and figures?
3. Multimodal reach, does the source pass images through at all, or skip them during ingestion?
4. Change flow, does the landing process touch only what actually changed?
5. Metadata, which dimensions must a query scope to, and can the source supply them?
6. Maturity, is the connector generally available, and does it work with the vector store and Region in play?

The landscape

Two shapes of knowledge base. A customer-managed knowledge base is the one this scenario describes: you choose the vector store, the parser, the chunking strategy and the embedding model. A Bedrock Managed Knowledge Base runs the datastore, a managed embedding model and a managed reranker for you, carries seven native connectors (S3, SharePoint, Confluence, web crawler, Google Drive, OneDrive and custom), and parses multimodal files with one built-in parser. It also removes most of the choices below: default or fixed-size chunking only, and no parser selection. AWS recommends it for accuracy and ease of setup. Everything that follows is the customer-managed path, where the pipeline is yours to configure.

Amazon S3 as the primary source. Point the data source at a general purpose bucket or a prefix, and it ingests the supported formats it finds. The bucket must be in the same Region as the knowledge base, and a bucket in another account works if you name the owner account. Each file is capped at 50 MB and an ingestion job at 100 GB. S3 and the custom data source are the two generally available sources, and the only ones that accept images, audio and video, so content with no connector is normally routed to S3 first.

Preview connectors. Bedrock provides four connectors that read a live source rather than an export, all of them preview releases that work only with an OpenSearch Serverless vector store. The web crawler follows seed URLs within a configured scope. The SharePoint, Confluence and Salesforce connectors authenticate with credentials held in Secrets Manager, detect the main document fields, and take inclusion and exclusion patterns. All four crawl added, modified and deleted content on each sync. None of the four ingest images, tables as images, or other multimodal files.

The web crawler in detail. It takes up to ten seed URLs and crawls static pages, with three scope settings: the same host and path, the same host, or any subdomain of the primary domain. Crawl rate is capped between 1 and 300 URLs per host per minute, and the page ceiling runs to 25,000. Exceeding that ceiling fails the sync outright rather than truncating it, so the scope and the exclusion patterns do real work. The crawler follows robots.txt, and looks for its own bedrockbot rules before generic ones.

Getting content into S3 when there is no connector. The connector list does not cover every source an organisation runs: document management systems, ticketing tools, line-of-business applications, and internal wikis nobody remembers commissioning. Where it stops, ordinary AWS transfer services land the content in S3 as documents the data source can ingest.

Amazon AppFlow. Moves records out of SaaS applications that have a supported connector, on a schedule or on an event in the source, and writes them to S3 with field mapping and filtering applied on the way. It suits a ticketing system or a CRM whose records become documents. Choose the fields that carry the answer, drop the ones that add noise, and what lands is close to ingestible rather than a raw dump.

AWS DataSync. Bulk and recurring transfer into S3 from on-premises NFS, SMB or HDFS, from other object storage, and from other clouds. It preserves the directory structure and moves only what changed after the first pass. The S3 connector documentation names it as the way to keep a bucket topped up from an on-premises file server.

AWS Transfer Family. Managed SFTP, FTPS, FTP and AS2 endpoints, plus browser-based transfers, for the case where a third party pushes documents to you rather than you pulling them. An endpoint backed by S3 turns a supplier’s or an auditor’s delivery into an ordinary object write.

None of the three carry the source system’s permission model or its metadata schema. AppFlow maps fields, DataSync preserves paths, Transfer Family preserves filenames. What reaches S3 is content with no record of who was allowed to see it or which department owned it. That work stays yours, so an ingestion Lambda writes the sidecar metadata alongside each object and sets the attributes retrieval will filter on.

Direct ingestion. IngestKnowledgeBaseDocuments indexes documents straight into the vector store, for S3 and custom data sources, up to 25 documents and 6 MB of payload per request. With a custom data source there is nothing to sync afterwards. With an S3 data source there is a catch: the change is not written back to the bucket, so the next sync reverses it unless you mirror it into S3 as well.

The default parser. Reads a document’s text and passes it downstream, at no parsing charge. Its limit is layout: it does not reliably reconstruct a table, a multi-column form, or the meaning carried by an image, so documents whose answer sits in structure arrive degraded.

Foundation-model parsing. A vision-capable model reads the document and returns a layout-aware representation, keeping tables intact and describing figures, with the extraction prompt open to customisation. The Claude, Nova and Llama 4 vision families are supported. Billing is per input and output token, and the total across all files in the data source is capped at 100 GB.

Bedrock Data Automation as the parser. A managed alternative that handles multimodal documents with no prompting, billed per page rather than per token. It is in preview and supported in US West (Oregon) only, so treat it as a Region-limited option rather than a default.

Chunking strategies. Default chunking splits content at sentence boundaries into roughly 300-token chunks. Fixed-size chunking takes a token count and an overlap percentage. Hierarchical chunking retrieves child chunks and returns their parents, and semantic chunking splits on meaning by running a foundation model over the content, which adds a charge that scales with how much data there is. No chunking treats each file as one chunk, which suits pre-split content. A custom transformation Lambda covers chunking logic Bedrock does not implement, or adds chunk-level metadata to a built-in strategy.

Metadata. For an S3 source, each document takes a sidecar named <filename>.<extension>.metadata.json in the same folder, capped at 10 KB, holding string, number or boolean attributes. An includeForEmbedding flag controls whether an attribute is concatenated to the chunk before embedding or only stored for filtering. Queries then filter with equality, numeric comparison, and list membership operators, up to five per group. On OpenSearch Serverless the index has to use the faiss engine for filtering to work.

Ingestion jobs and sync. StartIngestionJob parses, chunks, embeds and indexes a data source. The first run ingests everything because nothing is indexed yet; every run after that skips unchanged documents, re-ingests changed ones, and removes deleted ones. A metadata-only edit can be applied by merging the new attributes into the stored vectors, avoiding a call to the embedding model, unless the document is a CSV or a custom transformation Lambda is configured.

Embedding model. Each chunk is embedded by the model configured on the knowledge base. Titan Text Embeddings V2 supports 256, 512 or 1,024 dimensionsEmbedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching., the Cohere Embed models 1,024, and the older Titan G1 text model 1,536. The choice affects retrieval quality, index size and cost, and it is covered in depth elsewhere.

The pipeline reads left to right, with the parser choice and the sync behaviour doing the most to change the outcome.

Evaluation

Side by side

Choice	Best for	Multimodal content	Metadata for filtering	Status	Relative cost
S3 data source	Content already in object storage	✓	Sidecar .metadata.json	GA	Lowest
Custom data source	Pre-processed or connector-less content	✓	Whatever you attach	GA	Varies
Web crawler	Public pages you are authorised to crawl	✗	Crawled page fields	Preview, OpenSearch Serverless only	Low
SharePoint / Confluence / Salesforce	Live content systems	✗	Source fields, auto-detected	Preview, OpenSearch Serverless only	Medium
Amazon AppFlow to S3	SaaS records from a supported app	✓ once in S3	Fields the flow maps through	GA	Low
AWS DataSync to S3	On-premises NFS, SMB, HDFS, object storage	✓ once in S3	Written on landing	GA	Low
AWS Transfer Family to S3	Documents a third party pushes in	✓ once in S3	Written on landing	GA	Low
Default parser	Prose documents	✗	n/a	GA, no parsing charge	Lowest
Foundation-model parsing	Tables, forms, figures, scans	✓	n/a	GA	Per input and output token
Bedrock Data Automation parser	Mixed multimodal documents	✓	n/a	Preview, US West (Oregon)	Per page

Reaching a source with no connector

The three transfer services answer different source shapes, and the last column says how much ingestion work is left over.

Service	Source shape	Scheduling	Carries source permissions
Amazon AppFlow	SaaS application with a supported connector	Scheduled or event-triggered flows	✗
AWS DataSync	On-premises NFS, SMB, HDFS, or object storage	Scheduled tasks, incremental after the first run	✗
AWS Transfer Family	A third party pushing over SFTP, FTPS, FTP or AS2	Whenever they upload	✗

The solution

The layout-heavy PDFs are the clearest case for moving off the default. The default parser flattened the pricing tables into unstructured runs of numbers, which is why table questions came back empty. Retrieval was working; the chunk it retrieved no longer held a table. Route those documents through foundation-model parsing so the structure survives into the chunk. Bedrock Data Automation handles the scanned forms and figures well, but it is in preview in a single Region, so check the workload can sit in US West (Oregon) before planning around it. Leave the plain memos on the default parser, because the parser choice covers every PDF in a data source and running a model over clean prose adds cost without adding retrieval quality. Splitting the corpus across data sources by prefix is what makes both settings available at once.

The SharePoint and Confluence content is the case for connectors over exports, with a caveat. A hand-exported site is a job you own, maintain and eventually forget to run, while the connectors read the source directly and re-crawl on each sync. Both are preview releases that require an OpenSearch Serverless vector store, and neither passes images or diagrams through. Runbook screenshots have to reach S3 by another route if their content has to be searchable. The documentation site is the same trade through the web crawler: set the scope, the exclusion patterns and a page ceiling under 25,000, then let each sync pick up what moved.

The sync behaviour needs no configuration and a little discipline. Bedrock already re-ingests only added, modified and deleted documents, so the fix for the nightly full reprocess is upstream: stop the export rewriting objects it has not changed. Once modification times reflect real edits, an update costs in proportion to the edit. Where a file changes often and its attributes change with it, keep the metadata sidecar separate from the document so a metadata edit can be merged into the stored vectors instead of re-embedding.

Filtering has to be designed in at ingestion. If retrieval needs to scope to a department or a product line, that attribute must be attached as the documents come in, since it is stored with the chunks and cannot be recovered from a vector. Decide the dimensions first, confirm the source or the landing Lambda can supply them, and keep each sidecar under 10 KB. Retrofitting an attribute means re-ingesting the documents that carry it.

Worked example

The team splits the corpus by source and stops treating it as one bucket with one setting. A knowledge base takes five data sources, so four leaves room for one more.

The plain policy memos stay in their S3 prefix on the default parser, with department and effective-date attributes in a sidecar beside each file. They rarely change, and the export that writes them now preserves modification times, so a sync moves a handful of documents instead of thousands.

The table-heavy pricing PDFs move to their own S3 prefix and switch to foundation-model parsing. The tables survive into the chunks, and the pricing questions that used to come back empty now retrieve the right rows. The scanned forms go through the same parser, since the vision models read them too, rather than waiting on Bedrock Data Automation to leave preview.

The SharePoint operations site connects through the SharePoint connector instead of a hand-rolled export, with credentials in Secrets Manager and the auto-detected document fields carrying the filter attributes. The documentation site connects through the web crawler, scoped to the host and path and capped well under the page ceiling. Both are preview connectors on an OpenSearch Serverless index, which is a constraint the team accepts knowingly rather than discovers later.

Same knowledge base, four data sources, each with the parser and metadata that fit its documents. The retrieval quality the single-bucket first attempt could not reach came from the ingestion choices, not from changing the model.

What’s worth remembering

1. Ingestion is a pipeline of source, parse, chunk, embed and index, and each stage limits what the next one can do.
2. Syncing is incremental by design, so a full reprocess points at an upstream process rewriting files it has not changed.
3. The default parser handles prose at no parsing charge; foundation-model parsing bills per token and Bedrock Data Automation per page, and the choice applies to every PDF in the data source.
4. Only S3 and custom data sources ingest images and other multimodal files; the SharePoint, Confluence, Salesforce and web crawler connectors skip them and are preview releases tied to OpenSearch Serverless.
5. Filter attributes must be attached at ingestion, from a 10 KB sidecar for S3 or the source’s own fields, and cannot be recovered from a vector later.
6. Where no connector exists, Amazon AppFlow, AWS DataSync and AWS Transfer Family land content in S3, but none of them bring the source’s permissions or metadata, so ingestion has to attach both.