Chunking Code, Tables, and Mixed Content

The situation

An engineering-docs team is building a retrieval assistant over a large internal corpus on Amazon Bedrock. The source material is not clean prose. It is runbooks and API references full of code blocks, architecture docs with wide comparison tables, and onboarding pages that mix headings, bullet lists, sample payloads, and captioned screenshots. They loaded the lot into a Bedrock knowledge base, took the default chunking strategy, embedded everything, and started asking questions.

The answers are broken in ways that are easy to miss until you read the retrieved passages. A question about a deployment helper returns the second half of a Python function, with no signature and no imports. The loop comes back without the thing it loops over. A question about instance pricing returns a table body whose header row landed in a different chunk, so the columns are unlabelled numbers. A question about a diagram returns the caption without the figure, and the figure reference without the caption.

The embeddings are fine. The chunks they were computed from are not. Default chunking splits at roughly 300 tokens and honours sentence boundaries, which is careful treatment for prose and no help inside a code listing or a grid of rows. Re-embedding the corpus twice is not on the table. One question sits underneath all three failures: where should a boundary fall when the content is not a stream of sentences?

What actually matters

Chunking is a retrieval decision before it is a storage decision. Each chunk is the unit that gets embedded, indexed, and returned whole. A chunk that splits a meaningful thing in half produces an embedding for half a thing, and returns half a thing at answer time. Prose forgives that. A paragraph cut mid-sentence still carries most of its meaning, and the overlap between chunks patches the seam. Code, tables, and mixed layouts do not forgive it, because their meaning lives in structure that a token counter does not record.

The property that matters most is whether a boundary respects the content’s own units. A function is a unit. A class is a unit. A table with its header is a unit, a figure with its caption is a unit, and a markdown section under one heading is a unit. Cutting inside any of these damages retrieval in both directions. The chunk that should have matched the query now embeds a fragment that matches it weakly, and the chunk that does come back is missing the context that makes it usable. A table body without its header is wrong rather than merely incomplete, because nothing in it says which column is price and which is throughput.

The second property is self-sufficiency. A good chunk carries enough context to stand alone, because at answer time it usually arrives alone. The enclosing heading, the table caption, and the section title often belong attached to the chunk rather than left in a neighbouring one. Attach them as a prefix in the text, or as metadata travelling alongside. A code block is far more retrievable when the chunk also names the file and the class it came from.

The third is that structure has to be recovered before you can cut on it, and raw text has already discarded it. Once a PDF or an HTML page is flattened to a character stream, the table is tab-spaced numbers and the code block is indented lines. So the document needs a layout-aware parse first. That gives the chunker labelled tables, headings, and code regions to divide, rather than whitespace to infer from.

The fourth is that keeping a unit intact sometimes means letting a chunk run large, and the ceiling on that is real. Fixed-size chunking caps a chunk at 8,192 tokens, and the embedding model sets its own limit. Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0) accepts 8,192 tokens; Cohere Embed v3 accepts 512 tokens per text and by default discards the end of anything longer. So an oversized table embedded through Cohere is truncated without an error. All of this is a preprocessing decision, not a prompt tweak, and getting it wrong means re-ingesting the corpus.

What we’ll filter on

1. Boundary fidelity, does the split fall on the content’s natural units (function, class, whole table, section) rather than a token count?
2. Header and caption integrity, does a table keep its header and a figure keep its caption in the same chunk?
3. Context carried, does the chunk bring its enclosing heading, source path, or caption along as a prefix or metadata?
4. Structure awareness, is the document parsed into labelled regions before chunking, or split from flattened text?
5. Unit integrity over size, can an indivisible block stay whole when it exceeds the target size?
6. Pipeline fit, does the approach run inside the ingestion path without hand-built infrastructure?

The landscape

Default and fixed-size chunking. Default chunking splits content into chunks of roughly 300 tokens, honouring sentence boundaries. Fixed-size chunking makes you set both numbers: maxTokens from 1 to 8,192, and an overlap percentage from 1 to 99. Both are the right choice for uniform prose, where one cut point is about as good as another and the overlap patches the seams. Neither adds cost beyond the embedding calls. On code, tables, or mixed layouts they produce every failure above, because a token counter records nothing about being halfway through a function or one row into a table.

Semantic chunking. Split where the topic shifts. Bedrock’s version embeds each sentence together with a buffer of neighbouring sentences, then breaks where dissimilarity crosses a percentile threshold you set, up to a maximum token size. This keeps coherent prose together and is a real improvement for narrative documents. It works over sentences, so a listing or a grid is not a boundary it detects. It also invokes a foundation model during ingestion, which the standard strategies do not, so it costs more.

Hierarchical chunking. Build parent and child chunks, with a maximum token size for each and an overlap in tokens. Retrieval matches the small children, then returns the broader parent in place of the child. A match on an inner code snippet comes back framed by the section around it. Because parents replace children, the number of results returned can be lower than the number requested. It maps well onto documents with real section structure, and it is built into a Bedrock knowledge base.

Structure-aware splitting. Cut on the document’s own syntax: functions and classes for code, sections under headings for markdown, whole tables for tabular data. The header is repeated on each table chunk and the enclosing heading prefixed. This is the boundary-fidelity option, and it depends on knowing the structure first, which is why it pairs with a layout-aware parse.

Layout-aware parsing as the front half. The knowledge base default parser extracts text only, from .txt, .md, .html, .doc/.docx, .xls/.xlsx and .pdf files. Two parsers go further on figures, charts and tables in PDFs, and on .jpeg and .png images, and they can write those elements out as files in an S3 location you nominate. One is a vision foundation model used as a parser, from the Claude, Nova or Llama 4 vision families, billed on input and output tokens. The other is Amazon Bedrock Data Automation, billed per page; as a knowledge base parser it is in preview and offered only in US West (Oregon), so confirm that before designing around it. For scanned and image-based pages, Amazon Textract returns table cells, table titles and footers, form key-value pairs, and layout elements including figures and section headers.

Custom chunking with a Lambda transform. A knowledge base can run your own chunking logic instead of a built-in strategy. Set the chunking strategy to none, nominate an S3 bucket for the intermediate files, and point the data source at a Lambda function that reads them, chunks them, and writes them back. Per-type rules live there: function boundaries for code, a table and its header held together, the section heading attached as a prefix. The same hook attached to a built-in strategy instead adds chunk-level metadata to chunks the knowledge base has already made. You write and maintain the function.

Evaluation

Side by side

Read the table against the three failures. The half-a-function problem calls for structure-aware splitting on code boundaries. The headerless-table problem needs a layout-aware parse, plus keeping the table and header as one chunk. The caption-adrift problem is fixed by attaching the caption to the figure as metadata or a prefix. Hierarchical chunking helps all three by returning a framing parent, and a custom Lambda transform over a parsed document is the general way to encode per-type rules. None of the three is addressed by the default they started on.

The solution

The code case is a boundary-fidelity problem, and the fix is to cut where the language cuts. Split on function and class boundaries, so each chunk is a whole callable with its signature. Prefix it with the source path and the enclosing class or module, so the embedding and the retrieved text both carry that context. A helper method then comes back as the whole method, framed by where it lives, rather than a loop with no signature. When one function is larger than the target size, let the chunk run large rather than cutting it, keeping an eye on the embedding model’s input limit. Structurally this is either a custom Lambda transform that understands code, or hierarchical chunking, so a match on an inner snippet returns the parent holding the whole definition.

The table case is where parsing has to come before chunking. A text-only parser flattens a grid into ambiguous whitespace, so the header is gone before the chunker sees it. Recover the structure first with a vision model as parser, with Bedrock Data Automation where its preview Region suits you, or with Amazon Textract for scanned and image-based tables. Advanced parsing also changes the chunker’s behaviour: on parsed content it respects logical document boundaries such as pages and sections, and does not merge content across them. Once the table is labelled as a table, keep it and its header in one chunk. If the table is long, repeat the header on each piece so every chunk stays self-labelling. The caption travels as a prefix or as metadata, so a retrieved slice of pricing data still states what it is a table of.

The mixed-layout case is about self-sufficiency across types on one page. Parse the page into its regions, then chunk on the section structure, so a heading and the content beneath it travel together. A figure keeps its caption. A sample payload keeps the heading that says what it demonstrates, and a bullet list stays under the section it belongs to. Hierarchical chunking is a natural fit, embedding the specific child for a precise match and returning the parent section for the frame. A custom transform is the tool when no built-in strategy encodes the rule you need. Across all three, the parse and the chunk boundary matter as much as the embedding model, which makes this the same kind of decision as choosing where the retrieval index lives: a preprocessing choice made once, deliberately, before anything is embedded.

Worked example

Two documents ingest badly under the default. The first is a markdown page whose relevant fragment is a comparison table under a heading:

## Instance pricing by workload

| Instance | vCPU | Memory | On-demand USD$/hr | Spot USD$/hr |
|----------|-----:|-------:|------------------:|-------------:|
| m6i.large  | 2 |  8 GiB | 0.096 | 0.031 |
| m6i.xlarge | 4 | 16 GiB | 0.192 | 0.061 |
| c6i.xlarge | 4 |  8 GiB | 0.170 | 0.054 |

The page runs long, and a chunk boundary lands inside the table. The heading, the header row and the first rows go in one chunk; c6i.xlarge goes in the next, headerless. A query about spot pricing for compute-optimised instances retrieves the second chunk. What comes back is c6i.xlarge 4 8 GiB 0.170 0.054, with nothing in the chunk labelling which number is the spot price. Parse the page first so the table is labelled as a table, then keep the whole table in one chunk with ## Instance pricing by workload prefixed. Every row stays labelled. If the table were long enough to need splitting, the header row repeats on each piece.

The second is a Python helper in a runbook:

def deploy_stack(name, template, params):
    client = boto3.client("cloudformation")
    client.create_stack(
        StackName=name,
        TemplateBody=template,
        Parameters=[{"ParameterKey": k, "ParameterValue": v}
                    for k, v in params.items()],
    )
    waiter = client.get_waiter("stack_create_complete")
    waiter.wait(StackName=name)

A cut through the middle returns the waiter lines with no def line. A query about deploying a stack then retrieves code that waits on a stack it never shows being created. Split on the function boundary instead, so the chunk is the whole deploy_stack definition with its signature, prefixed with the file and runbook it came from. The match improves, because the embedded chunk now contains the signature the query is about. The returned text is runnable context rather than an orphaned tail.

What’s worth remembering

1. Chunking is a retrieval decision; the chunk is the unit that gets embedded and returned whole, so a split through a meaningful thing embeds and returns half a thing.
2. Cut on the content’s own units: functions and classes for code, whole tables for tabular data, sections under headings for markdown.
3. A table must keep its header in the same chunk, and a wide table that has to split should repeat the header on every piece.
4. Make chunks self-sufficient by attaching the enclosing heading, source path, or caption as a prefix or as metadata, because a chunk usually arrives at answer time alone.
5. Let an indivisible block exceed the target size, within the 8,192-token chunk cap and the embedding model’s input limit; Cohere Embed v3 stops at 512 tokens and drops the rest.
6. Structure has to be recovered before you can cut on it, so parse with a layout-aware tool first; flattened text has already lost the tables and code regions.