Chunking Code, Tables, and Mixed Content

The situation

An engineering-docs team is building a retrieval assistant over a large internal corpus on Amazon Bedrock. The source material is not clean prose. It is runbooks and API references full of code blocks, architecture docs with wide comparison tables, and onboarding pages that mix headings, bullet lists, sample payloads, and screenshots with captions. They loaded the lot into a Bedrock knowledge base with the default fixed-size chunking, embedded everything, and started asking questions.

The answers are subtly broken in ways that are easy to miss until you read the retrieved passages. A question about a deployment helper returns the second half of a Python function with no signature and no imports, so the model can see the loop but not what it operates on. A question about instance pricing returns a table body with the header row stranded in a different chunk, so the columns are unlabelled numbers. A question about a diagram returns the caption without the figure and the figure reference without the caption. The embeddings are fine; the chunks they were computed from are nonsense, because a 512-token window drawn across a code listing or a table cuts wherever the counter runs out, not where the content actually divides.

Nobody wants to re-embed the corpus twice. The question underneath all three failures is the same: where should the boundary between chunks fall when the content is not a stream of sentences?

What actually matters

Chunking is a retrieval decision before it is a storage decision. Each chunk is the unit that gets embedded, indexed, and returned whole, so a chunk that splits a meaningful thing in half produces an embedding for half a thing and hands the model half a thing at answer time. For prose this is forgiving, because a paragraph cut mid-sentence still carries most of its meaning and the surrounding chunks overlap. For code, tables, and mixed layouts it is not forgiving, because the meaning lives in structure that a fixed window is blind to.

The property that decides the most is whether a boundary respects the content’s own units. A function is a unit; a class is a unit; a table with its header is a unit; a figure with its caption is a unit; a markdown section under one heading is a unit. Cutting inside any of these destroys retrievability in both directions. The chunk that should have matched the query now embeds a fragment that matches it weakly, and the chunk that does come back is missing the context that makes it usable. A table body without its header is not a smaller answer, it is a wrong one, because the reader cannot tell which column is price and which is throughput.

The second property is self-sufficiency. A good chunk carries enough context to be understood alone, because at answer time it usually arrives alone. That means the surrounding heading, the table caption, and the section title often belong attached to the chunk rather than left in a neighbouring one, either as a prefix in the text or as metadata travelling alongside it. A code block is far more retrievable when the chunk also says which file and which class it came from; a table is far more usable when the caption above it rides along.

The third is that structure has to be known before you can cut on it, and raw text has already thrown it away. By the time a PDF or an HTML page is flattened to a character stream, the table is just tab-spaced numbers and the code block is just indented lines. Recovering the structure means parsing the document with something layout-aware first, so the chunker is dividing a document whose tables, headings, and code regions are labelled, rather than guessing at boundaries from whitespace.

The fourth is that keeping a unit intact sometimes means letting a chunk run large. A table or a function that exceeds the nominal chunk size is better kept whole and slightly oversized than cut to fit, because a complete oversized chunk still answers the question and a tidy half-chunk does not. Size targets are a guide for prose and a constraint to relax for indivisible structure.

And the operational one: this is a preprocessing pipeline, not a prompt tweak. The strategy lives in how documents are parsed and split before embedding, so getting it wrong means re-ingesting, and choosing well up front is cheaper than any amount of clever querying afterwards.

What we’ll filter on

1. Boundary fidelity, does the split fall on the content’s natural units (function, class, whole table, section) rather than a token count?
2. Header and caption integrity, does a table keep its header and a figure keep its caption in the same chunk?
3. Context carried, does the chunk bring its surrounding heading, source, or caption along as a prefix or metadata?
4. Structure awareness, is the document parsed into known regions before chunking, or split from raw flattened text?
5. Unit integrity over size, can an indivisible block stay whole even when it exceeds the target size?
6. Pipeline fit, does the approach run inside the ingestion path without hand-built infrastructure?

The content landscape

Fixed-size chunking. A sliding window of N tokens with some overlap, cutting wherever the counter lands. It is the cheapest option and the right default for uniform prose, where any given cut point is about as good as any other and the overlap patches the seams. On code, tables, or mixed layouts it is the source of every failure above, because it is structurally blind: it cannot see that it is halfway through a function or one row into a table.

Semantic chunking. Split where the topic shifts, by measuring the embedding distance between adjacent sentences or blocks and cutting at the large gaps. This keeps semantically coherent prose together and is a genuine improvement for narrative documents. It still reasons about text as sentences, so a code block or a table is not something it models well; the boundary lands better than fixed-size but the internals of a listing or a grid are not its concern.

Hierarchical chunking. Build parent and child chunks, embedding the small children for precise matching while returning the larger parent for context. This directly addresses self-sufficiency: a match on a child code snippet can return the whole parent section that frames it, so the model sees the function and the heading it sits under. It maps well onto documents with real section structure, and it is one of the built-in strategies in a Bedrock knowledge base.

Structure-aware splitting. Cut on the document’s own syntax: functions and classes for code, sections under headings for markdown, whole rows or whole tables for tabular data, with the header repeated on each table chunk and the surrounding heading prefixed. This is the boundary-fidelity option, and it depends entirely on knowing the structure first, which is why it pairs with a layout-aware parse rather than raw text.

Layout-aware parsing as the front half. Before any chunking, parse the document so its regions are labelled. Amazon Bedrock Data Automation extracts structured content (text, tables, figures, layout) from documents, images, and other media into a normalised form. A Bedrock knowledge base can also parse complex documents with a foundation model during ingestion, so tables and figures survive as structure. For table and form extraction specifically, Amazon Textract recovers cells, rows, and key-value pairs from scanned or image-based pages that a text extractor would flatten. The parse produces the labelled structure that structure-aware chunking then cuts on.

Custom chunking with a Lambda transform. A Bedrock knowledge base lets you supply your own chunking logic as a Lambda function in the ingestion pipeline, so you can apply per-type rules the built-in strategies do not: keep code on function boundaries, hold a table and its header together as one chunk, attach the section heading as a prefix, and pass structured blocks through untouched. This is the escape hatch for exactly the mixed-content case, at the cost of writing and maintaining the transform.

Side by side

Reading the table against the three failures: the half-a-function problem wants structure-aware splitting on code boundaries; the headerless-table problem wants a layout-aware parse plus keeping the table and header as one chunk; the caption-adrift problem wants the caption attached to the figure as metadata or a prefix. Hierarchical chunking helps all three by returning a framing parent, and the general-purpose way to encode the per-type rules is a custom Lambda transform over a parsed document. None of them is served by the fixed-size default they started on.

The picks in depth

The code case is a boundary-fidelity problem, and the fix is to cut where the language cuts. Split on function and class boundaries so each chunk is a whole callable with its signature, and prefix it with the source path and the enclosing class or module so the embedding and the retrieved text both carry that context. A helper method comes back as the whole method, framed by where it lives, rather than a loop with no signature. When a single function is larger than the target size, let the chunk run large rather than cutting it, because a complete oversized function answers the question and a tidy fragment does not. Structurally, this is either a custom Lambda transform that understands code, or hierarchical chunking so a match on an inner snippet returns the parent that contains the whole definition.

The table case is where parsing has to come before chunking. A text extractor flattens a grid into ambiguous whitespace, so the header is already lost before the chunker sees it; recover the structure first with Bedrock Data Automation or a foundation-model parse in the knowledge base, or with Amazon Textract for scanned and image-based tables. Once the table is known as a table, keep it and its header together in one chunk, and if the table is wide, repeat the header on each chunk so every piece stays self-labelling. The caption above the table rides along as a prefix or as metadata, so a retrieved slice of pricing data still says what it is a table of. The rule is one chunk per whole table where it fits, and header-repeated splits where it does not, never a blind cut through the rows.

The mixed-layout case is about self-sufficiency across types on one page. Parse the page into its regions, then chunk on the section structure so a heading and the content beneath it travel together, and attach the surrounding heading to each child chunk as context. A figure keeps its caption; a sample payload keeps the heading that says what it demonstrates; a bullet list stays under the section it belongs to. Hierarchical chunking is a natural fit here, embedding the specific child for a precise match and returning the parent section for the frame, and a custom transform is the tool when the built-in strategies do not encode a particular rule you need. Across all three, the connective tissue is the same as any retrieval build: the parse and the chunk boundary decide as much as the embedding model does, and this is the same discipline as choosing where the retrieval index lives, a preprocessing choice made once, deliberately, before anything is embedded.

A worked example: the pricing table and the deploy helper

Two documents ingest badly under the fixed-size default. The first is a markdown page whose relevant fragment is a comparison table under a heading:

## Instance pricing by workload

| Instance | vCPU | Memory | On-demand /hr | Spot /hr |
|----------|-----:|-------:|--------------:|---------:|
| m6i.large  | 2 |  8 GiB | 0.096 | 0.031 |
| m6i.xlarge | 4 | 16 GiB | 0.192 | 0.061 |
| c6i.xlarge | 4 |  8 GiB | 0.170 | 0.054 |

Under a 512-token window the header row and the first two data rows land in one chunk and c6i.xlarge lands in the next, headerless. A query about spot pricing for compute-optimised instances retrieves the second chunk, and the model sees c6i.xlarge 4 8 GiB 0.170 0.054 with no column labels, so it cannot tell which number is the spot price. Parse the page first so the table is known as a table, keep the whole table in one chunk with the ## Instance pricing by workload heading prefixed, and every row stays labelled. If the table were long enough to need splitting, the header row repeats on each piece so no chunk is ever a grid of unlabelled numbers.

The second is a Python helper in a runbook:

def deploy_stack(name, template, params):
    client = boto3.client("cloudformation")
    client.create_stack(
        StackName=name,
        TemplateBody=template,
        Parameters=[{"ParameterKey": k, "ParameterValue": v}
                    for k, v in params.items()],
    )
    waiter = client.get_waiter("stack_create_complete")
    waiter.wait(StackName=name)

A fixed cut through the middle returns the waiter lines with no def line, so a query about deploying a stack retrieves code that waits on a stack it never shows being created. Split on the function boundary instead, so the chunk is the whole deploy_stack definition with its signature, prefixed with the file and runbook it came from. The match improves because the embedded chunk now contains the signature the query is really about, and the returned text is runnable context rather than an orphaned tail. Two documents, two content types, one rule: the boundary follows the structure, not the token counter.

What’s worth remembering

1. Chunking is a retrieval decision; the chunk is the unit that gets embedded and returned whole, so a split through a meaningful thing embeds and returns half a thing.
2. Fixed-size windows are fine for prose and ruinous for code, tables, and mixed layouts, because they cut where the counter lands, not where the content divides.
3. Cut on the content’s own units: functions and classes for code, whole tables for tabular data, sections under headings for markdown.
4. A table must keep its header in the same chunk, and a wide table that has to split should repeat the header on every piece, so no chunk is a grid of unlabelled numbers.
5. Make chunks self-sufficient by attaching the surrounding heading, source path, or caption as a prefix or as metadata, because a chunk usually arrives at answer time alone.
6. Keep an indivisible block whole even when it exceeds the target size; a complete oversized function or table answers the question and a tidy fragment does not.
7. Structure has to be recovered before you can cut on it; parse with a layout-aware tool first, since flattened text has already thrown the tables and code regions away.
8. Amazon Bedrock Data Automation and foundation-model parsing in a knowledge base recover document structure; Amazon Textract recovers cells and key-value pairs from scanned or image-based tables.
9. A Bedrock knowledge base offers fixed-size, semantic, and hierarchical chunking built in, plus custom chunking via a Lambda transform for per-type rules the built-ins do not cover.
10. Chunking is a preprocessing choice made once before embedding; getting it wrong means re-ingesting the corpus, so match the boundary to the structure up front.