The situation
A RAG assistant answers questions over a corpus of long, structured documents: contracts, engineering runbooks, a compliance handbook. Retrieval quality is uneven in a specific way. When someone asks a precise question (“what is the notice period for termination on breach?”) the system either matches the exact clause and answers well, or it matches a large section that mentions termination six times and the model gives a vague, hedged answer that never lands on the number.
The team has already been round the chunk-size dial once. At 300 tokens per chunk, retrieval is sharp: the clause about breach embeds as one clean idea and matches the query, but the model receives just that clause with none of the definitions around it, so it can’t tell “the term” from “the Term” and answers incompletely. At 1,500 tokens per chunk, the model gets the whole section and answers with full context when it retrieves the right chunk, but the embedding now averages a whole section of mixed ideas, so the breach question often matches the wrong section entirely and the good context is context about something else.
Every setting of the single chunk-size knob trades precision for context or context for precision. There is no value that gives both, because one number is being asked to do two jobs.
What actually matters
The retrieved chunk plays two roles, and they want opposite sizes. As a search unit it wants to be small and focused, because an embedding is an average of everything in the chunk, and a small chunk that contains exactly one idea produces a vector that sits near queries about that idea. Pile several ideas into one chunk and the vector drifts to the centroid of all of them, near no single query in particular; that is why large chunks retrieve less precisely. As a context unit the same chunk wants to be large, because the model answers better when it can see the definitions, the preceding clause, the caveat two paragraphs down. Fragment the context and the model either guesses at the missing surroundings or refuses.
The insight that resolves this is that nothing forces the search unit and the context unit to be the same span of text. You can embed and search on small child chunks for precision, then, once a child matches, return its larger parent (the enclosing section, or the whole document) to the model for context. The retrieval index is built at one granularity and the generation payload is assembled at another. The single knob becomes two knobs, each set for its own job.
That decoupling costs something on both axes worth naming up front. Token budget: returning parents instead of the matched children puts more tokens into the model’s context window per retrieved hit, so a Top-kHow many chunks a retrieval step returns per query – the dial that trades answer coverage against token cost. of five small children can become five large parents, and if several children share a parent you want to deduplicate so you don’t send the same section three times. Cost and latency: a bigger context payload is more input tokens per call, and it competes with everything else in the window. Recall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks. shape: matching on children changes what surfaces, usually for the better on precise questions, but a query that is genuinely about a whole section (a summarisation ask) sometimes matched a large chunk better and now has to be reassembled from child hits.
There is also an ingest-side cost. The decoupled patterns need a link between each child and its parent, maintained at indexing time, so the retriever can walk from the matched child up to the span it returns. In a managed knowledge base that mapping is handled for you; in a hand-rolled store it is metadata you have to store and keep consistent.
What we’ll filter on
- Retrieval precision, does the matching happen on a small, single-idea unit so the embedding is clean?
- Answer context, does the model receive enough surrounding text to answer completely, not just the matched fragment?
- Token budget, how many tokens does each retrieved hit put into the context window, and is there deduplication when children share a parent?
- Ingest complexity, does the pattern need a maintained child-to-parent mapping, and is that managed or hand-rolled?
- Managed support, can Amazon Bedrock Knowledge Bases do this natively, or does it need application-side retrieval logic?
The retrieval-granularity landscape
Single-granularity chunking (the baseline that forces the trade). One chunk size, used for both search and context. Everything above is the story of why this can’t win: whatever size you pick is a compromise between a clean embedding and enough surrounding text. Fixed-size and semantic chunking both sit here when used plainly, one span embedded and the same span returned. It is the right choice only when the natural chunk already carries its own context, like a short FAQ entry where the question and answer are one self-contained unit.
Parent-document retrieval. Split each document into large parent chunks, then split each parent into small child chunks. Embed and index only the children. At query time, match against the child vectors for precision, then look up the parent each matched child belongs to and return the parent (not the child) to the model. The child does the finding; the parent does the answering. In the application-framework world this is the parent-document retriever pattern: the vector store holds child embeddings, a separate document store holds the parents, and a mapping links them. The parent can be the enclosing section or the whole source document depending on how much context the answers need.
Hierarchical chunking. The same idea expressed as a tree rather than two flat levels: a document becomes parent chunks, each parent becomes child chunks, and retrieval matches children then returns parents. Amazon Bedrock Knowledge Bases support this as a native chunking option, alongside fixed-size, semantic, and no chunking. You configure a parent max-token size, a child max-token size, and an overlap, and the knowledge base builds the parent-child structure, embeds the children, matches on them at query time, and returns the parent chunks to the model. It is parent-document retrieval as a managed feature: the child-to-parent mapping and the return-the-parent behaviour are handled by the service instead of your application. Choosing among the Bedrock chunking options covers where hierarchical fits against fixed-size and semantic for a mixed corpus; here the point is narrower, that hierarchical is the built-in way to get small-match, big-return without writing the retrieval glue yourself.
Sentence-window retrieval. A close relative aimed at flowing prose rather than structured documents. Embed and match on a single sentence for maximum precision, then, instead of returning a predefined parent, return that sentence plus a fixed window of the sentences immediately around it (say three before and three after). The context unit is assembled dynamically as a neighbourhood of the match rather than a fixed structural parent. It shines where documents have no reliable section structure to serve as parents, and where the useful context is “the few sentences around this one” rather than “the whole enclosing section”. It is an application-side pattern; you store each sentence with its neighbours as metadata and swap the matched sentence for its window before sending to the model.
No chunking (whole document as its own context). When documents are short enough that the whole thing fits comfortably in the context window, you can embed the document and return the document, and the trade never arises because the search unit and the context unit are both just “the document”. This only holds while documents stay small; it degrades exactly as they grow, which is the situation that creates the trade in the first place.
Side by side
| Pattern | Match precision | Answer context | Tokens per hit | Ingest complexity | Native in Bedrock KB |
|---|---|---|---|---|---|
| Single-granularity chunk | Trade-off | Trade-off | Chunk size | Low | ✓ (fixed / semantic) |
| Parent-document retrieval | ✓ (child) | ✓ (parent) | Parent size | Moderate (mapping) | Via hierarchical |
| Hierarchical chunking | ✓ (child) | ✓ (parent) | Parent size | Low (managed) | ✓ |
| Sentence-window | ✓ (sentence) | Window around match | Window size | Moderate (neighbours) | ✗ (app-side) |
| No chunking | ✗ (whole doc) | ✓ (whole doc) | Whole document | Lowest | ✓ (no-chunk option) |
The three middle rows are all the same move (search small, return large) expressed at different levels of managedness and for different document shapes. Hierarchical is the same idea as parent-document retrieval with Bedrock maintaining the mapping; sentence-window is the same idea with the parent replaced by a dynamic neighbourhood, suited to prose without structure.
The picks in depth
For the structured-document corpus that started this, hierarchical chunking in Bedrock Knowledge Bases is the direct fix and the one that needs the least code. Set a parent max-token size that captures a whole section (something like 1,500 tokens) and a child max-token size that isolates a clause or paragraph (something like 300 tokens), with a modest overlap. The breach question now matches the child that contains exactly that clause, so the embedding is clean and the match is precise, and the knowledge base returns the parent section, so the model reads the clause with its definitions and its notice provisions around it. Precision from the child, context from the parent, and the child-to-parent mapping maintained by the service rather than by you. This is the same small-match, big-return behaviour as the parent-document retriever pattern, with Bedrock doing the plumbing.
Parent-document retrieval as an application-side pattern is the pick when you are not on a managed knowledge base, or when you need the returned parent to be something other than a fixed structural chunk, for instance the entire source document rather than a section. You hold child embeddings in the vector store and full parents in a separate document store, keyed by a parent id carried on each child’s metadata. Retrieve children, collect their distinct parent ids, fetch those parents, deduplicate so a section that produced three child hits is sent once, and assemble the context from the parents. The deduplication step is the one people miss: without it, top-k on children can quietly send the same large parent several times and blow the token budget while adding no information.
Sentence-window retrieval is the pick for flowing prose with no dependable section structure to act as parents. Documents like interview transcripts, narrative reports, or long-form articles do not divide into clean sections, so the parent to return is better defined as a neighbourhood than a structural unit. Embed each sentence, match on it, then replace the matched sentence with itself plus a fixed number of neighbours before generation. The window size is the context knob: too small and you are back to fragments, too large and you are paying for prose the answer does not need. It is application-side work, storing each sentence’s neighbours as metadata, and it is worth it precisely when hierarchical parents would be arbitrary.
The one case where none of this applies is short, self-contained content. An FAQ entry, a product blurb, a glossary definition already carries its own context in a small span, so single-granularity chunking (or no chunking at all) is correct and the decoupled patterns are complexity with no payoff. Reach for parent-document retrieval when the match unit and the useful context unit genuinely differ in size, not by reflex.
A worked example: the termination clause
The corpus holds a services agreement. The relevant text is one clause inside a “Termination” section: “4.3 Either party may terminate on material breach by the other, on thirty days written notice served in accordance with clause 9.” Clause 9, in the same section, defines how notice is served. The word “Term” is defined at the top of the section.
Flat 1,500-token chunks. The whole “Termination” section is one chunk. Its embedding averages termination-for-convenience, termination-on-breach, notice mechanics, and the survival clause, so the vector for “notice period for termination on breach” sits near the section but not sharply, and on a large corpus it competes with, and sometimes loses to, a different agreement’s termination section. When it does win, the model has everything it needs and answers well. The retrieval is the weak link.
Flat 300-token chunks. Clause 4.3 is its own chunk and embeds as one idea, so the breach query matches it cleanly and reliably. But the model receives only “thirty days written notice served in accordance with clause 9” with no clause 9 and no definition of “Term”, so it answers “thirty days” and cannot say how notice is served or from when the thirty days run. The context is the weak link.
Hierarchical chunking. Clause 4.3 is a child; the “Termination” section is its parent. The child embeds the clause alone, so the breach query matches it precisely, beating the competing agreements because the vector is about exactly this clause. Bedrock returns the parent, so the model reads 4.3 together with clause 9’s service-of-notice mechanics and the definition of “Term” in the same section. The answer is now complete: thirty days written notice, served per clause 9, running from the date of service. Precise match and full context, from the same corpus, by setting two sizes instead of one.
What’s worth remembering
- One chunk size is asked to do two jobs, and they want opposite sizes: search wants small and focused for a clean embedding, context wants large and surrounding for a complete answer.
- An embedding is an average of everything in the chunk, so piling several ideas into one large chunk pulls its vector to the centroid and away from any single query; that is why large chunks retrieve less precisely.
- The fix is to decouple the search unit from the context unit: embed and match on small children, then return their larger parents to the model.
- Parent-document retrieval names the pattern directly: child embeddings in the vector store, full parents in a document store, a mapping linking them, and the parent returned once a child matches.
- Hierarchical chunking is parent-document retrieval as a managed Bedrock Knowledge Bases feature; you set a parent max-token size and a child max-token size, and the service maintains the mapping and returns parents.
- Sentence-window retrieval is the same idea for prose without structure: match on one sentence, return that sentence plus a fixed window of neighbours instead of a structural parent.
- Returning parents costs tokens, so deduplicate when several matched children share a parent, or you send the same section several times and blow the context budget.
- Recall shape shifts when you match on children; it usually improves for precise questions, and a genuine whole-section ask has to be reassembled from child hits.
- Short, self-contained content (FAQ entries, glossary terms) already carries its context in a small span, so single-granularity or no chunking is correct there and the decoupled patterns are needless complexity.
- Set two knobs deliberately: a child size small enough to isolate one idea for matching, and a parent size large enough to answer the question completely.