The situation
A RAG assistant answers questions over a corpus of long, structured documents: contracts, engineering runbooks, a compliance handbook. Retrieval quality is uneven in a specific way. When someone asks a precise question (“what is the notice period for termination on breach?”) the system either matches the exact clause and answers well, or it matches a large section that mentions termination six times and the model gives a vague, hedged answer that never lands on the number.
The team has already been round the chunk-size dial once. At 300 tokens per chunk, retrieval is sharp: the clause about breach embeds as one clean idea and matches the query, but the model receives just that clause with none of the definitions around it, so the answer conflates “the term” with the defined “Term” and comes back incomplete. At 1,500 tokens per chunk, the model gets the whole section and answers with full context when it retrieves the right chunk, but the embedding now averages a whole section of mixed ideas, so the breach question often matches the wrong section entirely and the good context is context about something else.
Every setting of the single chunk-size knob trades precision for context or context for precision. There is no value that gives both, because one number is being asked to do two jobs.
What actually matters
The retrieved chunk plays two roles that need opposite sizes. As a search unit it should be small and focused, because an embedding is an average of everything in the chunk, and a small chunk that contains exactly one idea produces a vector that sits near queries about that idea. Pile several ideas into one chunk and the vector drifts to the centroid of all of them, near no single query in particular; that is why large chunks retrieve less precisely. As a context unit the same chunk needs to be large, because the model answers better when it can see the definitions, the preceding clause, the caveat two paragraphs down. Fragment the context and the model fills the gap from its own weights, or returns an answer that stops short.
Nothing forces the search unit and the context unit to be the same span of text. You can embed and search on small child chunks for precision, then, once a child matches, return its larger parent (the enclosing section, or the whole document) to the model for context. The retrieval index is built at one granularity and the generation payload is assembled at another. The single knob becomes two knobs, each set for its own job.
Decoupling has effects on both axes worth naming up front. Token budget: returning parents instead of the matched children puts more tokens into the model’s context window per retrieved hit, so a Top-kHow many chunks a retrieval step returns per query – the dial that trades answer coverage against token cost. of five small children becomes up to five large parents. Where several matched children share a parent, that parent should reach the model once. Bedrock collapses them for you, which is why a hierarchical knowledge base often returns fewer results than the numberOfResults you asked for; a hand-rolled store leaves the deduplication to your code. Cost and latency: a bigger context payload is more input tokens per call, and it leaves less room for everything else in the window. Recall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks. shape: matching on children changes what surfaces, usually for the better on precise questions, but a query that is genuinely about a whole section (a summarisation ask) sometimes matched a large chunk better and now has to be reassembled from child hits.
There is also ingest-side work. The decoupled patterns need a link between each child and its parent, maintained at indexing time, so the retriever can walk from the matched child up to the span it returns. In a managed knowledge base that mapping is handled for you; in a hand-rolled store it is metadata you have to store and keep consistent.
What we’ll filter on
- Retrieval precision, does the matching happen on a small, single-idea unit so the embedding is clean?
- Answer context, does the model receive enough surrounding text to answer completely, not just the matched fragment?
- Token budget, how many tokens does each retrieved hit put into the context window, and is there deduplication when children share a parent?
- Ingest complexity, does the pattern need a maintained child-to-parent mapping, and is that managed or hand-rolled?
- Managed support, can Amazon Bedrock Knowledge Bases do this natively, or does it need application-side retrieval logic?
The landscape
Single-granularity chunking (the baseline that forces the trade). One chunk size, used for both search and context. Whatever size you pick is a compromise between a clean embedding and enough surrounding text. Fixed-size and semantic chunking both sit here when used plainly, one span embedded and the same span returned. It is the right choice only when the natural chunk already carries its own context, like a short FAQ entry where the question and answer are one self-contained unit.
Parent-document retrieval. Split each document into large parent chunks, then split each parent into small child chunks. Embed and index only the children. At query time, match against the child vectors for precision, then look up the parent each matched child belongs to and return the parent (not the child) to the model. Matching runs against the child; generation reads the parent. In the application-framework world this is the parent-document retriever pattern: the vector store holds child embeddings, a separate document store holds the parents, and a mapping links them. The parent can be the enclosing section or the whole source document depending on how much context the answers need.
Hierarchical chunking. The same idea as a managed feature. Amazon Bedrock Knowledge Bases offer it as a chunking strategy alongside fixed-size, semantic, and no chunking, at exactly two levels: a document becomes parent chunks, and each parent becomes child chunks. You set a maximum token size for each level, anywhere from 1 to 8,192 tokens, plus an overlap. The overlap here is an absolute number of tokens repeated between consecutive chunks in the same layer, where fixed-size chunking takes a percentage instead. The knowledge base then builds the structure, embeds the children, matches on them at query time, and replaces each matched child with its parent before the text reaches the model. The child-to-parent mapping and the return-the-parent behaviour are handled by the service instead of your application. Choosing among the Bedrock chunking options covers where hierarchical fits against fixed-size and semantic for a mixed corpus; the narrower claim here is that hierarchical is the built-in way to get small-match, big-return without writing the retrieval glue yourself.
Sentence-window retrieval. A close relative aimed at flowing prose rather than structured documents. Embed and match on a single sentence for maximum precision, then, instead of returning a predefined parent, return that sentence plus a fixed window of the sentences immediately around it (say three before and three after). The context unit is assembled dynamically as a neighbourhood of the match rather than a fixed structural parent. It fits where documents have no reliable section structure to serve as parents, and where the useful context is “the few sentences around this one” rather than “the whole enclosing section”. Bedrock has no sentence-window strategy; you store each sentence with its neighbours as metadata and swap the matched sentence for its window before sending to the model. Inside a knowledge base that means the no-chunking strategy plus a custom transformation Lambda to do the splitting at ingest.
No chunking (whole document as its own context). When documents are short enough that the whole thing fits comfortably in the context window, you can embed the document and return the document, and the trade never arises because the search unit and the context unit are both just “the document”. In Bedrock this is the no-chunking strategy, which treats each file as one chunk, so you pre-split the corpus into files yourself. It also gives up page-number citations, since the page-number metadata field is not populated. This only holds while documents stay small; it degrades exactly as they grow, which is the situation that creates the trade in the first place.
Evaluation
Side by side
| Pattern | Match precision | Answer context | Tokens per hit | Ingest complexity | Native in Bedrock KB |
|---|---|---|---|---|---|
| Single-granularity chunk | Trade-off | Trade-off | Chunk size | Low | ✓ (fixed / semantic) |
| Parent-document retrieval | ✓ (child) | ✓ (parent) | Parent size | Moderate (mapping) | Via hierarchical |
| Hierarchical chunking | ✓ (child) | ✓ (parent) | Parent size | Low (managed) | ✓ |
| Sentence-window | ✓ (sentence) | Window around match | Window size | Moderate (neighbours) | ✗ (app-side) |
| No chunking | ✗ (whole doc) | ✓ (whole doc) | Whole document | Lowest | ✓ (no-chunk option) |
The three middle rows are all the same move (search small, return large) expressed at different levels of managedness and for different document shapes. Hierarchical is the same idea as parent-document retrieval with Bedrock maintaining the mapping; sentence-window is the same idea with the parent replaced by a dynamic neighbourhood, suited to prose without structure.
The solution
For the structured-document corpus that started this, hierarchical chunking in Bedrock Knowledge Bases is the direct fix and the one that needs the least code. Set a parent max-token size that captures a whole section (something like 1,500 tokens) and a child max-token size that isolates a clause or paragraph (something like 300 tokens), with a modest overlap. Keep the two sizes under about 8,000 tokens combined, because the parent-child relationships are stored as metadata and larger settings can exceed the metadata size limit. For the same reason, hierarchical chunking is not recommended on an S3 vector bucket; pick OpenSearch Serverless, Aurora or another supported store for it. The breach question now matches the child that contains exactly that clause, so the embedding is clean and the match is precise, and the knowledge base returns the parent section, so the model reads the clause with its definitions and its notice provisions around it. Precision from the child, context from the parent, and the child-to-parent mapping maintained by the service rather than by you. This is the same small-match, big-return behaviour as the parent-document retriever pattern, with Bedrock doing the plumbing.
Parent-document retrieval as an application-side pattern is the pick when you are not on a managed knowledge base, or when you need the returned parent to be something other than a fixed structural chunk, for instance the entire source document rather than a section. You hold child embeddings in the vector store and full parents in a separate document store, keyed by a parent id carried on each child’s metadata. Retrieve children, collect their distinct parent ids, fetch those parents, deduplicate so a section that produced three child hits is sent once, and assemble the context from the parents. The deduplication step is the one people miss. Without it, top-k on children sends the same large parent several times, consuming the token budget and adding no information.
Sentence-window retrieval is the pick for flowing prose with no dependable section structure to act as parents. Documents like interview transcripts, narrative reports, or long-form articles do not divide into clean sections, so the parent to return is better defined as a neighbourhood than a structural unit. Embed each sentence, match on it, then replace the matched sentence with itself plus a fixed number of neighbours before generation. The window size is the context knob: too small and you are back to fragments, too large and you send prose the answer does not need. It is application-side work, storing each sentence’s neighbours as metadata, and it is the right shape precisely when hierarchical parents would be arbitrary.
The one case where none of this applies is short, self-contained content. An FAQ entry, a product blurb, a glossary definition already carries its own context in a small span, so single-granularity chunking (or no chunking at all) is correct and the decoupled patterns add complexity without improving the answer. Reach for parent-document retrieval when the match unit and the useful context unit genuinely differ in size, not by reflex.
Worked example
The corpus holds a services agreement. The relevant text is one clause inside a “Termination” section: “4.3 Either party may terminate on material breach by the other, on thirty days written notice served in accordance with clause 9.” Clause 9, in the same section, defines how notice is served. The word “Term” is defined at the top of the section.
Flat 1,500-token chunks. The whole “Termination” section is one chunk. Its embedding averages termination-for-convenience, termination-on-breach, notice mechanics, and the survival clause, so the vector for “notice period for termination on breach” sits near the section but not sharply, and on a large corpus it competes with, and sometimes loses to, a different agreement’s termination section. When it does win, the model has everything it needs and answers well. The retrieval is the weak link.
Flat 300-token chunks. Clause 4.3 is its own chunk and embeds as one idea, so the breach query matches it cleanly and reliably. But the model receives only “thirty days written notice served in accordance with clause 9” with no clause 9 and no definition of “Term”, so it answers “thirty days” and cannot say how notice is served or from when the thirty days run. The context is the weak link.
Hierarchical chunking. Clause 4.3 is a child; the “Termination” section is its parent. The child embeds the clause alone, so the breach query matches it precisely, beating the competing agreements because the vector is about exactly this clause. Bedrock returns the parent, so the model reads 4.3 together with clause 9’s service-of-notice mechanics and the definition of “Term” in the same section. The answer is now complete: thirty days written notice, served per clause 9, running from the date of service. Precise match and full context, from the same corpus, by setting two sizes instead of one.
What’s worth remembering
- One chunk size is asked to do two jobs that pull opposite ways: search needs small and focused for a clean embedding, context needs large and surrounding for a complete answer.
- The fix is to decouple the search unit from the context unit: embed and match on small children, then return their larger parents to the model.
- Hierarchical chunking is parent-document retrieval as a managed Bedrock Knowledge Bases feature, at exactly two levels, each capped at 8,192 tokens, with the service maintaining the mapping and swapping matched children for their parents.
- Returning parents puts more tokens into the context window per hit, so a parent shared by several matched children should reach the model once: Bedrock collapses them and returns fewer results than you requested, while a hand-rolled store needs that deduplication in your code.
- Short, self-contained content (FAQ entries, glossary terms) already carries its context in a small span, so single-granularity or no chunking is correct there and the decoupled patterns are needless complexity.