Choosing a Chunking Strategy for Bedrock Knowledge Bases

The situation

The knowledge base backs an internal assistant that answers policy and product questions with citations. Three kinds of source material feed it. There are long PDFs, product manuals and onboarding guides running to a hundred pages, written as flowing prose. There are structured policy documents with numbered sections, sub-clauses, and the occasional table (notice periods, fee schedules, eligibility grids). And there are short FAQ entries, a question and a two-sentence answer, hundreds of them exported from the help desk.

All of it was ingested with Bedrock’s default chunking, which splits content into chunks of approximately 300 tokens. That split preserves complete sentences, and that is the only structural guarantee it makes: it has nothing to say about clauses, headings, sections or tables. Retrieval is mediocre. Some answers come back half-formed because the relevant clause was split across a chunk boundary and only one half scored highly enough to be retrieved. Others come back buried, because a 300-token chunk swept up three unrelated ideas and the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. averaged them into something that matches nothing well.

Chunking sits upstream of embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. and retrieval, so it sets the ceiling for everything downstream. A fact that never gets retrieved cannot be generated, and whether it gets retrieved is decided the moment the document is cut into pieces.

What actually matters

Chunk size is the trade nobody escapes. A small chunk embeds a single idea and matches a query precisely, but it may omit the surrounding context the model needs to answer, the clause without the section it sits in, the answer without the question it responds to. A large chunk carries that context and dilutes the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together.: the vector becomes the average of several ideas and matches queries about none of them cleanly. It also takes up more of the context windowContext windowThe maximum number of tokens an LLM can attend to in a single call – prompt plus output combined. when it lands in the promptPromptThe input you hand to an LLM – system instructions, user message, examples, retrieved documents, tool descriptions, the lot.. Precision pulls one way, context pulls the other, and the right point on that line depends on the document.

Boundaries matter as much as size. Splitting mid-sentence or mid-section destroys meaning: half a sentence embeds as noise, and a clause severed from its heading loses what it was about. Overlap is the guard against this, a run of shared tokens between adjacent chunks so a sentence that straddles a boundary survives whole in at least one of them. Bedrock expresses it two ways: fixed-size chunking takes an overlap percentage between 1 and 99, and hierarchical chunking takes an absolute token count.

Structure-aware chunking beats blind fixed-size on structured documents, because the document already marks where the seams are: headings, sections, clause numbers, table rows. Ignoring that markup and counting tokens instead throws away the one signal that reliably marks a good cut point.

Two practical constraints sit underneath all of this. A chunk must fit inside the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model’s maximum input, and that maximum varies by an order of magnitude. Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0) accepts 8,192 tokens, which matches the 8,192-token ceiling Bedrock puts on every chunking configuration. Cohere Embed v3 accepts 512 tokens per text and discards the end of anything longer, because the truncate parameter defaults to END. Pick Cohere and a 1,500-token chunk loses its tail without an error.

The second constraint is that chunking is fixed at creation. You cannot change chunkingConfiguration on an existing data source connector; the strategy set at creation is the one that data source uses for its lifetime, and moving to another means creating a new data source and ingesting the corpus again. That has a cost in embedding charges and time, so it is a decision rather than a setting you flip between.

Finally, chunking sets citation granularity. Cite a small child chunk and you point the reader at the exact clause; cite one whole file and the citation covers a 100-page manual. Choose no chunking and Bedrock cannot give you a page number in the citation at all, or let you filter on the page-number metadata attribute.

What we’ll filter on

1. Does it follow document structure (headings, sections, tables), or split blind?
2. Where does it sit on the precision-versus-context trade?
3. Does it handle long and structured documents, including tables?
4. How precise is the resulting citation granularity?
5. What’s the ingestion cost and complexity?
6. Is it available natively in Bedrock Knowledge Bases, or does it need custom code?

The landscape

No chunking. Set the strategy to NONE and each file becomes one chunk. This suits material that is already the right size: FAQ entries where the question-and-answer pair is the natural unit, or short documents that cutting would only damage. It fails on long documents. One 100-page PDF becomes one vector that matches everything vaguely and nothing sharply, and it exceeds the embedding model’s input by a wide margin, so the tail is either discarded or rejected depending on the model. Citations lose page numbers too.

Fixed-size chunking. Split by a target token count with an overlap percentage. Bedrock accepts a maxTokens between 1 and 8,192 and an overlapPercentage between 1 and 99. It is simple, uniform, and blind to structure: it cuts on token count whether that lands mid-clause, mid-section, or mid-table. It’s a reasonable floor for uniform prose of consistent density. On structured or mixed corpora it produces exactly the failure the team is seeing.

Hierarchical chunking. Parent and child chunks. The document is split into large parent chunks (a section) and each parent into small child chunks (a paragraph or clause). Retrieval matches on the small child, then swaps in the larger parent before the text reaches the model. Each level takes its own maxTokens, up to 8,192, plus an overlap expressed in tokens. Two behaviours to know: because children collapse into their parents, a query can return fewer results than the number requested, and AWS advises against hierarchical chunking on an S3 vector bucket, where combined chunk sizes above about 8,000 tokens run into metadata size limits.

Semantic chunking. Split at semantic boundaries by comparing each sentence with the next and cutting where the dissimilarity is largest, keeping one coherent idea per chunk. breakpointPercentileThreshold (50 to 99) sets how dissimilar a sentence pair must be before it becomes a break, bufferSize (0 or 1) decides whether a sentence is embedded alone or together with its neighbours, and maxTokens caps the result. This is strongest on flowing prose with no reliable structural markers, the long manual written as continuous narrative. It carries an extra ingest charge, because it calls a foundation model to work out where the breaks go.

Custom chunking via a Lambda transformation. Full control through a transformation Lambda. Set the chunking strategy to NONE, point the data source at your Lambda and at an S3 bucket for intermediate storage, and Bedrock writes files there for your function to chunk and write back. Split on markdown headings, keep a table intact as one chunk, apply a domain rule such as never splitting a clause. The same hook runs in a second mode: keep a managed strategy and use the Lambda only to attach chunk-level metadata after chunking, where stepToApply is POST_CHUNKING. Maximum fidelity to the document, maximum code and maintenance.

Advanced parsing. Before any of the above, parse complex documents, tables, images and multi-column layout into clean text. The default Bedrock parser reads text only and adds no charge. Bedrock Data Automation handles figures, charts and tables, priced per page, and at the time of writing it is in preview in US West (Oregon) only. A foundation model parser handles the same material with a prompt you can customise, drawn from the Claude, Nova and Llama 4 vision families, priced on input and output tokens. Parsing is not a chunking strategy; it is a pre-step you pair with one. Note that choosing either non-default parser applies it to every PDF in the data source, text-only ones included, and you are charged for all of them.

Evaluation

Side by side

No single row is the answer for the whole corpus. The three source types need different cuts, and chunking is configured per data source, so matching the strategy to the document type is a configuration Bedrock supports directly.

Three ways to cut one document

The solution

Match the strategy to the document type. Chunking is set per data source, so the mixed corpus becomes three data sources in one knowledge base, each with its own configuration. The default quota is five data sources per knowledge base, so three fits with room to spare.

Hierarchical for the long structured policy PDFs. This is the workhorse choice for the material causing the pain. Set a parent size that captures a whole section (say 1,500 tokens) and a child size that isolates a clause or paragraph (say 300 tokens), both well inside the 8,192-token ceiling each level allows. A query about a specific clause matches the child, which embeds that one idea cleanly, and Bedrock swaps in the parent, so the clause reaches the model with its section around it. Precision from the child, context from the parent, and a citation that points at the clause rather than the manual. This is the single change most likely to fix the team’s retrieval.

Semantic for the flowing-prose manuals. Where a document is continuous narrative without reliable headings, semantic chunking finds the topic shifts and keeps each idea whole, which fixed-size can’t do and hierarchical only approximates through size. It charges more at ingest, but for prose that exposes no structure it retrieves noticeably better.

Fixed-size for the uniform short content. The FAQ entries and other short, consistent material don’t need anything cleverer. Fixed-size with a modest overlap, or no chunking at all when each entry is already one chunk, is correct here; hierarchical or semantic would add complexity and change nothing.

Custom Lambda when structure must be preserved exactly. If a policy document has tables that must stay intact as one chunk, or a rule such as never splitting a numbered clause, the managed strategies cannot express it and a transformation Lambda can. Reach for it when a specific structural guarantee matters, not by default.

Advanced parsing first when documents carry tables or images. Run Bedrock Data Automation or a vision-capable foundation model over any source with tables or figures, so a fee schedule arrives as clean text rather than scrambled tokens. Parsing and chunking are separate decisions; parse to clean up the input, then chunk the result with whichever strategy fits.

On overlap, Bedrock accepts 1 to 99 percent for fixed-size chunking. Somewhere around 10 to 20 percent keeps boundary sentences from being orphaned without inflating the vector count; much more than that mostly duplicates content.

The gotchas are consistent. chunkingConfiguration is immutable once a data source exists, so a change of strategy means a new data source and a full re-embed of that corpus. Parent and child token sizes both need setting deliberately; defaults are a starting point, not an answer. No chunk may exceed the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model’s maximum input, which is 512 tokens on Cohere Embed v3 and 8,192 on Titan Text Embeddings V2. Chunks that are too small leave the model without the context it needs, even when retrieval is perfect. And always re-evaluate retrieval after changing the chunking, a citations-required RAG pipeline depends on the right chunk coming back, and the only way to know a chunking change helped is to measure retrieval before and after on a fixed set of questions.

Worked example

A subscriber-facing policy PDF has a section headed “Termination” with several numbered clauses. One reads: “4.3 Early termination. A party may terminate this agreement before the end of the term. The terminating party must give the other party no less than sixty (60) days’ written notice. Notice takes effect on receipt.” The section around it defines what “the term” means and how notice is served.

Under default chunking at roughly 300 tokens, the cut fell on a sentence boundary partway through clause 4.3. The chunk carrying the heading and the first sentence ended there, and the sentence with the number opened the next one. A query for “what is the notice period for early termination?” embeds cleanly, but neither chunk holds both the trigger and the number. The chunk that scores highest is the one with the phrase “early termination”, not the one with “sixty (60) days”, so the retrieved text carries the topic and not the answer.

Query: "what is the notice period for early termination?"

Default chunking, top retrieved chunk:
  "...4.3 Early termination. A party may terminate this agreement
   before the end of the term."
  -> topic matches, the number is in the NEXT chunk. Answer: incomplete.

Ingested again through a second data source configured for hierarchical chunking, the parent chunk is the whole “Termination” section and clause 4.3 is one child chunk. The child embeds the complete clause, trigger and number together, and matches the query precisely. Bedrock swaps in the parent, so the definition of “the term” and the rules on serving notice reach the model as well.

Query: "what is the notice period for early termination?"

Hierarchical, matched child chunk:
  "4.3 Early termination. A party may terminate this agreement before
   the end of the term. The terminating party must give the other
   party no less than sixty (60) days' written notice. Notice takes
   effect on receipt."
  -> clause complete. Parent (the Termination section) returned for context.
  Answer: "Sixty days' written notice, effective on receipt."

Same document, same embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model, same query. The only change was where the document got cut, and that was the difference between a wrong answer and a cited, correct one.

What’s worth remembering

1. Chunking sits upstream of everything and sets the ceiling for retrieval and generation, because a fact that never gets retrieved cannot be generated.
2. Chunk size is a trade: small chunks match precisely but drop context, large chunks carry context but dilute the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. and take up more of the window.
3. Hierarchical chunking resolves that trade instead of settling it one way: match the small child, then swap in the large parent for context. It is native in Bedrock and the strongest default for long structured documents.
4. A chunk must fit the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model’s maximum input, 512 tokens on Cohere Embed v3 against 8,192 on Titan Text Embeddings V2, and overlap keeps boundary sentences whole.
5. chunkingConfiguration cannot be changed on an existing data source, so switching strategy means a new data source and a full re-embed.
6. Chunking is set per data source, so a mixed corpus can run three strategies at once inside one knowledge base.