The situation
A knowledge team is standing up a retrieval-augmented generation assistant on Amazon Bedrock. The vector store, the retrieval settings, and the prompt are settled. What is not settled is how the source documents reach the index. The corpus is mixed: a few thousand policy PDFs in Amazon S3, a SharePoint site the operations team edits weekly, a public documentation site that changes daily, and a Confluence space of engineering runbooks. Some of the PDFs are plain text. A good number are layout-heavy, with pricing tables, scanned forms, and diagrams that carry the actual answer.
The first attempt loaded everything from one S3 bucket on the defaults, and retrieval was patchy. Answers grounded in the plain memos came back clean. Questions whose answer sat inside a table came back wrong or empty, because the table had been flattened into a run of numbers with no structure. Nothing could be scoped to a single department, because no metadata was attached to any document. A nightly export also rewrote every object in the bucket, so each sync saw the whole corpus as modified and re-ingested it.
Underneath all of that is the ingestion pipeline: which data source the documents arrive through, how they are parsed, how they are chunked, what metadata is attached, and how a change reaches the index.
What actually matters
Ingestion is a pipeline, and each stage limits what the next one can do. A document enters through a data source, is parsed into text, is split into ChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense., is embedded, and lands in the index. A bad parse cannot be rescued by a chunking strategy, and bad chunks cannot be rescued by a better embedding model. The stages compound, so each default deserves a decision.
The first question is where the documents live and whether a connector reaches them. S3 needs no connector credentials, because the content already sits in object storage. A live SharePoint site or a Confluence space is another matter. Exporting either on a schedule leaves you owning the export job, while a connector authenticates against the source and re-crawls it on every sync. The trade is which kind of knowledge base the connector sits on. A new customer-managed knowledge base takes S3 and a custom data source and nothing else: AWS stopped supporting new Confluence, SharePoint, Salesforce and web-crawler connectors there on 30 September 2026, though ones created earlier keep working. Those four sources now arrive through a Bedrock Managed Knowledge Base, which brings its own datastore and embedding model with them. A live source reaches the index that way, or it reaches S3 first.
The second question is document complexity, which sets the parser choice. The default parser extracts text from .txt, .md, .html, Word, Excel and PDF files, and it does well on prose. It does not reconstruct a pricing table, a two-column form, or the meaning carried by a figure. A layout-aware parser keeps tables intact and describes images, and it charges per page or per token. That choice is made per data source. Once made, it applies to every PDF in that source, including the ones that are only text.
A related constraint sits under it. On a customer-managed knowledge base, images, audio and video are only processed from S3 and custom data sources, and the Confluence, SharePoint, Salesforce and web crawler connectors skip those files during ingestion. A runbook whose answer is a screenshot does not reach that index through the Confluence connector at all, whatever parser is configured. A managed knowledge base closes the gap, because image, audio and video extraction is configurable on each of its connectors.
The third question is how a change reaches the index. Syncing is incremental by design: Bedrock re-ingests the documents added, modified or deleted since the last sync, and skips the rest. What defeats it is a landing process that rewrites files it has not changed, which is what turned the team’s nightly export into a full re-ingest. Fix the export and the time and cost of an update track the size of the change.
The fourth is metadata and filtering. Scoping a query to a department, a product line or a date range needs that attribute attached at ingestion and stored with each chunk. It cannot be reconstructed from a vector afterwards. Decide the filter dimensions before the first load, because adding one later means re-ingesting the documents that need it.
Chunking and the embedding model matter too, and both are subjects in their own right. Chunking splits each parsed document into retrievable units, and the strategy is fixed once the data source is connected. The embedding model turns each chunk into the vector the index searches.
What we’ll filter on
- Data source, is there a managed connector for where the documents live, or does the content have to reach S3 first?
- Document complexity, is the meaning in the prose, or in tables, forms, and figures?
- Multimodal reach, does the source pass images through at all, or skip them during ingestion?
- Change flow, does the landing process touch only what actually changed?
- Metadata, which dimensions must a query scope to, and can the source supply them?
- Buildability, can this data source still be created on the kind of knowledge base in play, and in the Region in play?
The landscape
Two shapes of knowledge base. A customer-managed knowledge base is the one this scenario describes: you choose the vector store, the parser, the chunking strategy and the embedding model. A Bedrock Managed Knowledge Base runs the datastore, a managed embedding model and a managed reranker for you, carries seven native connectors (S3, SharePoint, Confluence, web crawler, Google Drive, OneDrive and custom), and parses multimodal files with one built-in parser. It also removes most of the choices below: default or fixed-size chunking only, and no parser selection. AWS recommends it for accuracy and ease of setup, and it is now the only path to the live-source connectors. It runs in eight Regions: N. Virginia, Oregon, Ireland, London, Frankfurt, Tokyo, Sydney and GovCloud (US-West), where only the S3 connector is available. Everything that follows is the customer-managed path, where the pipeline is yours to configure.
Amazon S3 as the primary source. Point the data source at a general purpose bucket or a prefix, and it ingests the supported formats it finds. The bucket must be in the same Region as the knowledge base, and a bucket in another account works if you name the owner account. Each file is capped at 50 MB and an ingestion job at 100 GB. S3 and the custom data source are the only two a new customer-managed knowledge base can take, and the only ones that accept images, audio and video, so content with no connector is routed to S3 first.
The live-source connectors. Four connectors read a live source rather than an export: Confluence, SharePoint, Salesforce and a web crawler. None is selectable on a customer-managed knowledge base any more. AWS stopped supporting new ones there on 30 September 2026; they never left preview, they needed an OpenSearch Serverless vector store, and none ingested images or tables held as images. Connectors created earlier keep ingesting and retrieving. For new work the same sources, plus Google Drive and OneDrive, arrive through a Bedrock Managed Knowledge Base, which reads credentials from a Secrets Manager secret, detects the main document fields, takes inclusion and exclusion patterns, crawls added, modified and deleted content on each sync, and extracts images, audio and video where you enable it. It also takes a daily, weekly or monthly sync schedule, which the customer-managed path has never had.
The web crawler in detail. The managed knowledge base’s crawler takes up to ten seed URLs, or up to three sitemap URLs, and traverses HTML pages under one of three scope settings: the same host and path, the same host, or any subdomain of the primary domain. Depth runs from 0 to 10 and defaults to 2, links followed per URL from 1 to 1,000 and default to 100, and the rate is capped between 1 and 300 URLs per minute. Inclusion and exclusion regular expressions trim what is left. It respects robots.txt per RFC 9309 and identifies itself as amazon-bedrock-knowledgebase-on-behalf-of-<hash>, which is the user agent a site’s rules have to name. It signs in with basic, form or SAML authentication where a site needs it, and it is the one connector with no document-level access control, so anything it indexes is readable by anyone who can query the knowledge base.
Getting content into S3 when there is no connector. The connector list does not cover every source an organisation runs: document management systems, ticketing tools, line-of-business applications, and internal wikis nobody remembers commissioning. Where it stops, ordinary AWS transfer services land the content in S3 as documents the data source can ingest.
Amazon AppFlow. Moves records out of SaaS applications that have a supported connector, on a schedule or on an event in the source, and writes them to S3 with field mapping and filtering applied on the way. It suits a ticketing system or a CRM whose records become documents. Choose the fields that carry the answer, drop the ones that add noise, and what lands is close to ingestible rather than a raw dump.
AWS DataSync. Bulk and recurring transfer into S3 from on-premises NFS, SMB or HDFS, from other object storage, and from other clouds. It preserves the directory structure and moves only what changed after the first pass. The S3 connector documentation names it as the way to keep a bucket topped up from an on-premises file server.
AWS Transfer Family. Managed SFTP, FTPS, FTP and AS2 endpoints, plus browser-based transfers, for the case where a third party pushes documents to you rather than you pulling them. An endpoint backed by S3 turns a supplier’s or an auditor’s delivery into an ordinary object write.
None of the three carry the source system’s permission model or its metadata schema. AppFlow maps fields, DataSync preserves paths, Transfer Family preserves filenames. What reaches S3 is content with no record of who was allowed to see it or which department owned it. That work stays yours, so an ingestion Lambda writes the sidecar metadata alongside each object and sets the attributes retrieval will filter on.
Direct ingestion. IngestKnowledgeBaseDocuments indexes documents straight into the vector store, for S3 and custom data sources, up to 25 documents and 6 MB of payload per request. With a custom data source there is nothing to sync afterwards. With an S3 data source there is a catch: the change is not written back to the bucket, so the next sync reverses it unless you mirror it into S3 as well.
The default parser. Reads a document’s text and passes it downstream, at no parsing charge. Its limit is layout: it does not reliably reconstruct a table, a multi-column form, or the meaning carried by an image, so documents whose answer sits in structure arrive degraded.
Foundation-model parsing. A vision-capable model reads the document and returns a layout-aware representation, keeping tables intact and describing figures, with the extraction prompt open to customisation. The Claude, Nova and Llama 4 vision families are supported. Billing is per input and output token, and the total file size across all files must stay under 100 GB. The quota that bites first is the file count: at most 1,000 files go through a foundation-model parser, and Bedrock Data Automation carries the same ceiling.
Bedrock Data Automation as the parser. A managed alternative that handles multimodal documents with no prompting, billed per page rather than per token. It is in preview and supported in US West (Oregon) only, so treat it as a Region-limited option rather than a default.
Chunking strategies. Default chunking splits content at sentence boundaries into roughly 300-token chunks. Fixed-size chunking takes a token count and an overlap percentage. Hierarchical chunking retrieves child chunks and returns their parents, and semantic chunking splits on meaning by running a foundation model over the content, which adds a charge that scales with how much data there is. No chunking treats each file as one chunk, which suits pre-split content. A custom transformation Lambda covers chunking logic Bedrock does not implement, or adds chunk-level metadata to a built-in strategy.
Metadata. For an S3 source, each document takes a sidecar named <filename>.<extension>.metadata.json in the same folder, capped at 10 KB, holding string, number, boolean or string-list attributes. An includeForEmbedding flag controls whether an attribute is concatenated to the chunk before embedding or only stored for filtering. Queries then filter with equality, numeric comparison, and list membership operators, up to five per group. On OpenSearch Serverless the index has to use the faiss engine for filtering to work.
Ingestion jobs and sync. StartIngestionJob parses, chunks, embeds and indexes a data source. The first run ingests everything because nothing is indexed yet; every run after that skips unchanged documents, re-ingests changed ones, and removes deleted ones. A metadata-only edit can be applied by merging the new attributes into the stored vectors, avoiding a call to the embedding model, unless the document is a CSV or a custom transformation Lambda is configured.
Embedding model. Each chunk is embedded by the model configured on the knowledge base. Titan Text Embeddings V2 supports 256, 512 or 1,024 Embedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching., the Cohere Embed models 1,024, and the older Titan G1 text model 1,536. The choice affects retrieval quality, index size and cost, and it is covered in depth elsewhere.
The pipeline reads left to right, with the parser choice and the sync behaviour doing the most to change the outcome.
Evaluation
Side by side
| Choice | Best for | Multimodal content | Metadata for filtering | Status | Relative cost |
|---|---|---|---|---|---|
| S3 data source | Content already in object storage | ✓ | Sidecar .metadata.json |
GA | Lowest |
| Custom data source | Pre-processed or connector-less content | ✓ | Whatever you attach | GA | Varies |
| Managed knowledge base connectors | Live SharePoint, Confluence, Google Drive, OneDrive, or web pages | ✓ | Source fields, auto-detected | GA, managed knowledge base only | Medium |
| Amazon AppFlow to S3 | SaaS records from a supported app | ✓ once in S3 | Fields the flow maps through | GA | Low |
| AWS DataSync to S3 | On-premises NFS, SMB, HDFS, object storage | ✓ once in S3 | Written on landing | GA | Low |
| AWS Transfer Family to S3 | Documents a third party pushes in | ✓ once in S3 | Written on landing | GA | Low |
| Default parser | Prose documents | ✗ | n/a | GA, no parsing charge | Lowest |
| Foundation-model parsing | Tables, forms, figures, scans | ✓ | n/a | GA | Per input and output token |
| Bedrock Data Automation parser | Mixed multimodal documents | ✓ | n/a | Preview, US West (Oregon) | Per page |
Reaching a source with no connector
The three transfer services answer different source shapes, and the last column says how much ingestion work is left over.
| Service | Source shape | Scheduling | Carries source permissions |
|---|---|---|---|
| Amazon AppFlow | SaaS application with a supported connector | Scheduled or event-triggered flows | ✗ |
| AWS DataSync | On-premises NFS, SMB, HDFS, or object storage | Scheduled tasks, incremental after the first run | ✗ |
| AWS Transfer Family | A third party pushing over SFTP, FTPS, FTP or AS2 | Whenever they upload | ✗ |
The solution
The layout-heavy PDFs are the clearest case for moving off the default. The default parser flattened the pricing tables into unstructured runs of numbers, which is why table questions came back empty. Retrieval was working; the chunk it retrieved no longer held a table. Route those documents through foundation-model parsing so the structure survives into the chunk. Bedrock Data Automation handles the scanned forms and figures well, but it is in preview in a single Region, so check the workload can sit in US West (Oregon) before planning around it. Leave the plain memos on the default parser, because the parser choice covers every PDF in a data source and running a model over clean prose adds cost without adding retrieval quality. Splitting the corpus across data sources by prefix is what makes both settings available at once.
The SharePoint and Confluence content is the case for connectors over exports, and on a customer-managed knowledge base that case can no longer be made. A hand-exported site is a job you own, maintain and eventually forget to run, but new Confluence, SharePoint, Salesforce and web-crawler connectors stopped being supported on this kind of knowledge base on 30 September 2026. Two ways in remain. A Bedrock Managed Knowledge Base carries all three live sources natively, extracts images and diagrams per connector, and syncs daily, weekly or monthly, but it does not offer the parser and chunking control the pricing PDFs need. Or the content lands in S3, which keeps one knowledge base and one set of parser choices and hands the team the crawl job. Split it by source: keep the pricing and policy corpus on the customer-managed knowledge base, and put the live sources behind a managed one rather than a hand-rolled export.
The sync behaviour needs no configuration and a little discipline. Bedrock already re-ingests only added, modified and deleted documents, so the fix for the nightly full reprocess is upstream: stop the export rewriting objects it has not changed. Once modification times reflect real edits, an update costs in proportion to the edit. Where a file changes often and its attributes change with it, keep the metadata sidecar separate from the document so a metadata edit can be merged into the stored vectors instead of re-embedding.
Filtering has to be designed in at ingestion. If retrieval needs to scope to a department or a product line, that attribute must be attached as the documents come in, since it is stored with the chunks and cannot be recovered from a vector. Decide the dimensions first, confirm the source or the landing Lambda can supply them, and keep each sidecar under 10 KB. Retrofitting an attribute means re-ingesting the documents that carry it.
Worked example
The team splits the corpus by source and stops treating it as one bucket with one setting. The S3 content stays on the customer-managed knowledge base, where each data source carries its own parser; a knowledge base takes five, so two prefixes leave room to spare.
The plain policy memos stay in their S3 prefix on the default parser, with department and effective-date attributes in a sidecar beside each file. They rarely change, and the export that writes them now preserves modification times, so a sync moves a handful of documents instead of thousands.
The table-heavy pricing PDFs move to their own S3 prefix and switch to foundation-model parsing. The tables survive into the chunks, and the pricing questions that used to come back empty now retrieve the right rows. The scanned forms go through the same parser, since the vision models read them too, rather than waiting on Bedrock Data Automation to leave preview.
The SharePoint operations site, the Confluence runbooks and the documentation site go to a second, managed knowledge base, because that is where their connectors now are. Credentials sit in a Secrets Manager secret, the auto-detected document fields carry the filter attributes, image extraction is enabled so the runbook screenshots index, and the crawler is scoped to the host and path at a depth of two. Retrieval fans out to both knowledge bases, which is what the split costs.
Two knowledge bases, five data sources, each with the parser and metadata that fit its documents. The retrieval quality the single-bucket first attempt could not reach came from the ingestion choices, not from changing the model.
What’s worth remembering
- Ingestion is a pipeline. Source, parse, chunk, embed, index; each stage limits what the next one can do.
- Syncs are incremental. A full reprocess means an upstream export is rewriting files it has not changed.
- Parsers are per data source. The default is free; a foundation model bills per token, Bedrock Data Automation per page, both capped at 1,000 files.
- Live-source connectors moved. New Confluence, SharePoint, Salesforce and web-crawler connectors ended on customer-managed knowledge bases on 30 September 2026; a managed one has them.
- Attach filter metadata at ingestion. S3 takes a 10 KB sidecar; the attribute is stored with the chunk and cannot be recovered from a vector.
- Transfer services carry no permissions. AppFlow, DataSync and Transfer Family land content in S3 without permissions or metadata; ingestion must attach both.