When a Document Won't Fit the Context Window

The situation

The legal team has a recurring task: given a pair of documents (typically a vendor contract and an internal policy), identify clauses that speak to a specific topic (refund disputes, data handling, liability limits) and surface both the clauses and any conflicts between them. They’ve been doing this by hand, which takes a day per pair; they’ve asked whether an LLMLLMA neural network trained to predict the next token in a sequence, large enough that it generalises to tasks it wasn’t explicitly trained for. can help.

Documents are long. A typical contract is 60,000 tokensTokenThe unit of text an LLM actually sees – usually a short character sequence, not a whole word. of dense legal prose; a policy manual is 100,000 tokens. That pair, plus a 2,000-token system prompt, sits well inside Claude Sonnet 5’s 1M-token context window on Bedrock, so nothing rejects the call. Fitting is the easy part. Quality falls off long before the hard limit, and the legal team has hundreds of contract-and-policy pairs, not one.

Concrete constraints: a 1M-token input window and a 128K maximum output on Claude Sonnet 5, Bedrock per-token pricing, a preference for five focused calls over one enormous one at similar total tokens, and a user-facing latency target of under 30 seconds per query.

What actually matters

Every token in the prompt competes with every other token for the model’s attentionAttentionThe mechanism inside a transformer that lets each token weigh how much every other token in the context matters to it.. Dumping both documents in and asking the question sends 160k tokens, almost none of which bear on any specific question, and answer quality drops as the relevant passage sits deeper inside irrelevant text. Tokens are also billed, so the wasted ones show up twice.

Two different failures get confused. A prompt assembled past the model’s input limit comes back with a model_context_window_exceeded stop reason and no usable answer. A prompt that fits but leaves too little room for the response comes back truncated, with a max_tokens stop reason and a sentence that ends mid-clause, because the output cap rather than the input was the ceiling. Both become arithmetic once you count tokens before the call and log the input and output counts after it: the first means trim the input, the second means raise maxTokens or narrow what was asked.

The first decision is chunking. A document split into passages of the right size can be searched by relevance before a prompt is built. The right size depends on the question: a question about a specific clause needs small, tight chunks; a question about broad themes does better with larger chunks that carry context. Chunking also interacts with embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. quality. Bedrock Knowledge Bases offers fixed-size, hierarchical, semantic and custom chunking, along with the option of none at all. Fixed-size defaults to 300 tokens with 20% overlap and accepts up to 8,192, which suits question-answering; summarisation does better further up that range. Fixing one size for the whole corpus is the easy thing to do and the thing that hurts later. Hierarchical and semantic strategies take their boundaries from the document instead, so a contract’s clauses survive and a policy manual’s sections aren’t sliced mid-argument.

The second is retrieval vs windowing. For specific questions, retrieve the top-k relevant chunks from each document, assemble a prompt with those, and answer. For exhaustive questions (“list every clause about X”), retrieval can miss relevant passages the embedding model didn’t rank high enough. A sliding window, process the document in overlapping segments, is the alternative. Trade-offs: retrieval is cheap and focused; windowing is exhaustive but expensive.

The third is map-reduce patterns. Run the same extraction prompt over every chunk (map), collect results, then combine them (reduce). Coverage is exhaustive, and it takes many calls to get there. Split at 500 tokens, the 60k-token contract is 120 map calls and the 100k-token manual another 200. That is a lot of calls, with the tokens and wall-clock to match. Worth it when exhaustive coverage matters.

The fourth is hierarchical summarisation. Summarise each chunk; summarise the summaries; produce a top-level summary. Useful for producing a structured understanding of a document before running targeted questions. The “parent-child” hierarchical chunking patterns from the RAG side of the house are a retrieval-time version of the same idea.

The fifth is context-window hygiene. Even when a document fits, the prompt shouldn’t just be “here’s the document, now the question.” Structure matters: headings preserved, chunk boundaries marked with [chunk N from document X], the question clearly separated, the expected output format spelled out. Prompts that treat the context window as a structured container outperform prompts that treat it as a bucket.

Underneath the mechanics sits the question itself. “What clauses govern refund disputes?”, “summarise the contract” and “are there conflicts between these two documents?” are three different shapes. The first needs retrieval; the second calls for hierarchical summarisation; the third needs a map-reduce pair comparison. One architecture doesn’t fit all, so the question shapes the approach.

What we’ll filter on

1. Coverage, exhaustive or best-effort?
2. Cost, tokens consumed per query?
3. Latency, seconds to answer?
4. Quality at length, does the approach avoid “lost in the middle”?
5. Complexity, how much orchestration code does this need?

The landscape

1. Single large prompt (“just use the million-token window”). Put both documents in one prompt with the question. Cheapest orchestration, heaviest token bill: ~165k input tokens on every query, whether the answer lives in ten of them or a thousand. Quality falls off as the relevant passage sinks into the surrounding text. Correct for short documents; wrong for a 160k-token pair asked six questions a day.

2. Retrieval-augmented per query. Chunk both documents into 500-token passages, embed, store. For each query, retrieve top-k from each document (say 10 each), assemble a prompt with 10k tokens of context. Fast, cheap, focused. Misses passages that matter but weren’t retrieved. Correct for specific questions; wrong for exhaustive coverage.

3. Map-reduce extraction. Split each document into chunks. Map: run the extraction prompt (e.g., “does this chunk discuss refund disputes? If so, quote the relevant sentences”) over every chunk. Reduce: feed all extractions into a combining prompt that organises, dedupes, and cross-references. Exhaustive; expensive (many small calls); slow (parallelisable, but even then ~30 seconds for a pair of documents at 320 map calls).

4. Hierarchical summarisation. Summarise each chunk; cluster summaries by topic; summarise each cluster; produce document-level summaries. Query-time then operates on summaries (cheap, focused, may miss detail). Useful for multi-query workloads where the summary hierarchy is reused.

5. Sliding window. Walk the document with an overlapping window (e.g., 20k-token windows with 2k overlap); run the query per window; merge. Exhaustive; simpler than map-reduce; still expensive.

6. Hybrid: hierarchical-summary-first retrieval. Build a hierarchy (section → chapter → whole document summaries). At query time, start at the top, find the relevant sections via the summaries, retrieve detailed chunks only from those sections. Near-exhaustive without the full map-reduce bill.

Evaluation

Side by side

The correct approach depends on the question type. For the legal team’s three query shapes, find clauses, summarise, find conflicts, no single approach dominates. The realistic system picks per query.

A decision tree for “which approach per question”

The solution

Top-k retrieval for narrow questions. “What does the contract say about data retention?” is a clause-hunting query. Chunk both documents at 500 tokens with 50-token overlap, embed with Amazon Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0), store in Amazon OpenSearch Serverless. Query time: Retrieve with an in filter over the two document ids and numberOfResults raised from its default of 5 to 10, assemble a prompt with the 20 chunks marked by source, ask the question. Total prompt: ~12k tokens. Response: focused, cited, ~4s. Risk: if an answer sits in a chunk that didn’t rank top-10, we miss it. Mitigation: attach a reranker model to the Retrieve call so candidates are reordered by relevance before they reach the prompt, and raise numberOfResults to 15 for legal documents where precision matters.

Map-reduce for exhaustive extraction. “List every clause governing refund disputes” is an exhaustive query. The map prompt, run over each chunk: “Does this text contain any clause relating to refund disputes? If so, quote the relevant sentences verbatim and give a one-line explanation of the clause’s effect.” The reduce prompt takes all the map outputs (a few kilobytes of quoted passages) and dedupes, groups by topic, and produces a structured list. The map step runs on Claude Haiku 4.5, which has a 200K window and a lower per-token rate than Sonnet; the reduce runs on Sonnet 5 for quality. That is 320 small map calls and one large reduce call. Parallelise the map via asyncio or a Step Functions Map state; total wall-clock ~30 seconds.

Paired map-reduce for cross-document analysis. “Find conflicts between the contract’s refund policy and the company’s refund policy” is the hardest shape. Extract clauses on “refund” from each document via map-reduce (reusing extractions if they were computed earlier). Then run a pair-comparison prompt: given extractions from Document A and Document B, identify pairs where they speak to overlapping topics and call out differences. The comparison step can be quadratic (every A-clause against every B-clause), but clustering by topic first reduces it to O(topics × clauses_per_topic). Total time ~40 seconds, and the token count is roughly two map-reduce runs plus the comparison. Cache extractions so a second cross-document query on the same pair skips the extraction entirely.

Hierarchical summarisation for summary-style questions. “Give me an executive summary of the contract” or “what’s the shape of this policy manual.” Built offline: summarise each section (chunk), summarise each chapter (group of sections), summarise the whole document. Store as a tree. Query time: traverse the tree to find the correct granularity for the question. Build cost once per document; each query afterwards reads a few thousand tokens of summary.

Routing. A thin classifier in front, either a Haiku 4.5 call or a regex-based heuristic, picks the approach per question. “List all”, “every”, “find all” triggers map-reduce. “Compare”, “conflict”, “differ” triggers paired map-reduce. “Summarise”, “overview” triggers hierarchical. Everything else defaults to top-k retrieval. The router is allowed to be crude; the cost of mis-routing is at most “use a slower/more-expensive approach for a simpler question,” not wrong answers.

Worked example

Queries over a typical day on one contract/policy pair:

Q1 "What's the payment schedule in the contract?"
  → narrow, retrieval
  → 4s, ~12k Sonnet input

Q2 "List every clause about liability limits in both documents."
  → exhaustive, map-reduce
  → 35s, ~224k Haiku input + ~20k Sonnet reduce

Q3 "Does the policy actually align with the contract on refunds?"
  → cross-document, paired map-reduce
  → 45s, ~224k Haiku input + ~30k Sonnet comparison

Q4 "List every refund-related clause in both documents."
  → exhaustive, map-reduce
  → 30s, extractions cached from Q3; ~20k Sonnet reduce only

Q5 "Summarise the policy manual for me."
  → summary, hierarchical (prebuilt)
  → 3s, ~5k Sonnet input

Q6 "What does the contract say about force majeure?"
  → narrow, retrieval
  → 4s, ~12k Sonnet input

Total: 6 queries, ~2 minutes of model time,
       ~99k tokens on Sonnet and ~448k on Haiku

Compared to putting both documents in one prompt every time:

6 queries × 165k input tokens = ~990k tokens, all on Sonnet

The routed system sends about a tenth of the Sonnet tokens and moves the bulk of the reading onto the cheaper model, which is where the bill difference comes from. It also answers better: the exhaustive queries see every chunk instead of hoping the relevant clause survives 160k tokens of context, and the narrow questions read twelve thousand tokens instead of a hundred and sixty-five.

What’s worth remembering

1. Structure the prompt. Preserved headings, labelled chunk boundaries and a clearly separated question raise answer quality at the same token count.
2. Fitting is not the same as working. A million-token window holds both documents, and answer quality still falls off as the relevant passage sinks into the surrounding text.
3. One approach doesn’t fit all questions. Narrow questions need retrieval; exhaustive questions call for map-reduce; summaries run off hierarchical pre-builds. Route.
4. Map-reduce is the tool for exhaustive coverage. Run the map step on Claude Haiku 4.5 and the reduce on Claude Sonnet 5, and parallelise the map.
5. Cost scales with approach, not question difficulty. A hard question can be cheap if routed correctly; an easy one can be expensive if routed wrong.

A 400-page contract and a 200-page policy manual, six questions in a day, answers that cite their sources and don’t lose clauses in the middle. The window is large enough to hold both documents. The answers are good because the system doesn’t fill it every time.