The situation
A team has a batch job on Amazon Bedrock that reads contracts. Each document runs from a few pages to a couple of hundred, and for every one the job asks a model to pull out key dates, parties, obligations, and any unusual clauses, then write a short risk summary. It worked fine on the ten-page samples. In production, on the real spread of documents, two things break.
The long contracts overflow the context window. The job pastes the whole document into a single prompt. When the input alone is too large for the model, Bedrock returns a ValidationException and nothing runs. The harder case returns HTTP 200. Input and answer reach the ceiling part-way through generation, and the response carries partial text with a stopReason of model_context_window_exceeded. A summary cut off that way still arrives as a successful call. A second ceiling sits under the first: Claude accepts at most 100 pages per PDF in a request. A two-hundred-page contract cannot be attached whole, whatever the window allows.
The bill is the other problem. Feeding entire documents means paying for every token of every page on every call, whether or not the answer needed page 90. Someone suggested moving to a model with a much larger context window so the biggest contracts fit. That removes the overflow, but the cost per document goes up rather than down, the calls get slower, and on the longest documents the summaries start missing clauses that are demonstrably in the text. Bigger window, worse answers. What the job needs is a token budget that is cheap and reliable, rather than one that is merely large enough.
What actually matters
A token is the unit the model and the bill both count in. It is roughly a sub-word piece, not a word and not a character; common short words are a single token, longer or rarer words split into several, and a rough English rule of thumb is about four characters or three-quarters of a word per token. Every property that follows is denominated in tokens: the window limit, the input price, the output price. Estimating token counts is the first real skill of budgeting a workload.
The context window is a single ceiling over input plus output combined. It is not “how much document fits”, it is how much document plus instructions plus the model’s own answer can coexist in one call. Push the input close to the ceiling and there is no room left for the model to write, so the answer gets cut off mid-sentence or the request fails. Any budget has to reserve space for the output. The window and the maximum output length are two limits that both bind. The window caps the total; a separate max-output-tokens setting caps generation within whatever room is left.
Cost has two prices, not one. Input tokens and output tokens are billed separately, and output is typically the dearer of the two per token. A long-document job is usually input-heavy, so the document you paste dominates the bill. Shrinking the input is therefore the largest single cost lever. A job that writes long summaries for many documents can still run up meaningful output charges. Watch both sides rather than assuming input is the whole story.
A bigger window is not free and not automatically better. It costs more, because you are paying for more input tokens; it is slower, because the model has more to read before it answers; and answer quality can degrade as the context grows, because a relevant fact buried in the middle of a very long context turns up less reliably in the output than the same fact in a short, focused prompt. Fitting the document is necessary and not sufficient. A summary built from ten well-chosen pages is often better and cheaper than one built from two hundred pages of which five mattered.
That reframes the task. The goal is not to make the document fit; it is to put only the tokens the answer needs in front of the model, and to leave enough of the window for the answer. Three families of technique do that: retrieve the relevant parts instead of pasting all of them, compress before you reason, or split the document into ChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. and combine the results.
What we’ll filter on
- Does the whole document fit in the window with room reserved for the output, or does it overflow?
- Does the task need the whole document at once, or only the parts relevant to a specific question?
- Input token cost per document, since input usually dominates a long-document bill.
- How much of the window is left for the output, and does the job set an explicit max-output-tokens?
- Risk of losing text to truncation, or of lost-in-the-middle quality loss as the input grows.
- Latency budget: how slow per document is acceptable across the batch?
The landscape
Stuff the whole document. Paste everything into one prompt and ask for the answer. Simplest to build, and fine when documents are reliably small relative to the window. It fails on exactly this workload: the long ones overflow, you pay for every page on every call whether it was relevant or not, and even when it fits, quality can sag on the longest inputs as key facts get buried. It is the baseline the other strategies exist to beat.
Truncate to fit. Cut the document to the first N tokens so it slides under the ceiling. Cheap and trivial, and occasionally right when the answer genuinely lives at the top of every document. On contracts it is dangerous, because the clause that matters might be on page 120, and truncation removes it without warning. The output looks complete and is simply wrong about the part that got cut. If you truncate at all, do it knowingly and never on documents where the tail carries meaning.
Retrieval (RAG). Split documents into chunks, index them in a vector store, and at query time retrieve only the chunks relevant to the question and put those in the prompt. The model sees a handful of pages instead of two hundred, so input tokens per call drop sharply, the window stops overflowing, and the relevant text sits in a short focused context rather than in the middle of a long one. It adds an indexing pipeline, and the risk moves to the retriever: the answer is only as good as the chunks it surfaced. Amazon Bedrock Knowledge Bases runs that pipeline for you, covering chunking, embedding, indexing, and retrieval at query time, configured rather than coded. This is the default for question-answering over a large or growing corpus, where each question needs a small slice rather than the whole library.
Summarise then reason. Compress the document into a shorter representation first, then do the real task on the summary. Two passes, and the second one bills few input tokens because its input is small. It fits tasks where a faithful condensation preserves what the answer needs, and it loses on tasks that turn on exact wording, because summarising throws away the precise clause you might need to quote. Good for “what is the overall risk”, weaker for “what is the exact indemnity cap”.
Map-reduce over chunks. Split the document, run the task on each chunk independently (the map step), then combine the per-chunk results into a final answer (the reduce step). Every chunk fits comfortably, so nothing overflows and the whole document genuinely gets read, unlike truncation. It costs more calls and therefore more input tokens overall than retrieval, and the reduce step has to reconcile chunk results that each saw only part of the picture. This is the strategy when the task must cover the entire document, like extracting every obligation, rather than answering one question that lives in a few pages.
Bigger context window. Move to a model or configuration with a larger window so more of the document fits at once. It genuinely removes overflow and it is the least code to change. It does not remove the input cost, it raises it; it adds latency; and it does not fix lost-in-the-middle quality loss. It is worth it when a task truly needs long-range context that spans the whole document in one pass and the smarter strategies cannot preserve it, not as the reflexive answer to “it does not fit”.
Evaluation
Side by side
| Strategy | Reads whole document | Input tokens per document | Overflow risk | Quality risk | Best for |
|---|---|---|---|---|---|
| Stuff everything | ✓ | Highest | ✗ (overflows) | Lost-in-the-middle on long inputs | Reliably small documents |
| Truncate to fit | ✗ | Capped | ✓ safe | Drops the tail without warning | Answer always near the top |
| Retrieval (RAG) | ✗ (relevant slice) | Low | ✓ safe | Missed chunk on retrieval | One question over a large corpus |
| Summarise then reason | ✓ (compressed) | Low on the second pass | ✓ safe | Loses exact wording | Gist and overall-risk tasks |
| Map-reduce | ✓ | High (many calls) | ✓ safe | Reduce step reconciles partial views | Cover-everything extraction |
| Bigger window | ✓ | Highest | ✓ safe | Cost, latency, lost-in-the-middle | Genuine whole-document context |
Reading it against the contract job: the per-question lookups (“what is the termination notice period”) need retrieval; the “summarise the overall risk” step suits summarise-then-reason; the “list every obligation” step calls for map-reduce; and none of the three is served by pasting the whole document into the biggest window available.
The solution
Start by reserving the output. Before choosing a strategy, decide how many tokens the answer needs and set the cap explicitly. On the Bedrock Converse API that is inferenceConfig.maxTokens, and a response that runs into it comes back with a stopReason of max_tokens rather than an error, so a summary cut off mid-sentence still looks like a successful call unless something checks. Treat the window, minus that reservation, minus your instructions, as the real budget for document text. A job that leaves this implicit is the one that returns half-written summaries, because the input grew until there was no room to answer. On the Claude messages format under InvokeModel, max_tokens is a required field, so the reservation is explicit whether or not anyone thought about it. Under Converse it is optional, which is where jobs forget it.
For the fits-in-one-call decision, estimate tokens rather than guess from page count. The CountTokens operation on the Bedrock runtime returns the inputTokens a request would consume for a given model, and that count matches what the model would bill. Use it, or a rule-of-thumb ratio where a per-document call is not worth making, to turn document size into a token estimate. Compare that against the budget left after the output reservation, and route each document accordingly: small ones can go straight in, large ones need retrieval, map-reduce, or summarisation. The estimate is what lets the batch job branch per document instead of applying one strategy to a spread of sizes it does not fit.
The shape of the job is a lever of its own, separate from the shape of the prompt. Nothing in a nightly contract run is waiting on a response, and on the models that support it Bedrock prices batch inference at half the on-demand per-token rate: write the prompts as JSONL to S3, submit a CreateModelInvocationJob naming the model and the input and output locations, and collect the results from the output prefix when the job finishes. The rate cut applies whichever strategy you land on, so it stacks with the input-shrinking work rather than competing with it. The catches are worth knowing. Batch runs asynchronously, so it suits the overnight sweep and not the interactive lookup. It supports neither tool calling nor structured output, so a strategy that leans on a declared schema stays on the synchronous path. Prompt caching is on-demand only and does not apply to batch jobs, so the two reductions cannot be combined.
Prompt caching is the other billing lever, and it applies wherever a long prefix repeats across calls. A document the job queries several times is that case: mark a cache checkpoint at the end of the static content, and later calls read those tokens at the model’s cache-read rate instead of the standard input rate. Checkpoints have a per-model minimum, from 512 to 4,096 tokens, and an entry expires after a five-minute TTL unless a hit resets it or you ask for the one-hour option. On some models a cache write bills above the standard input rate, so a prefix read once costs more than not caching it. A hit is never guaranteed either, so read cacheReadInputTokens in the response rather than assuming. Caching does little for retrieval, where every question assembles a different set of chunks and the shared prefix is only the instructions.
Retrieval is the workhorse for question-answering because it attacks the input cost directly: instead of paying for every page, you pay for the handful of chunks that answer the question, and the answer usually improves because the relevant text is in a short focused prompt rather than buried on page 90. The failure mode moves from the window to the retriever, so chunking, embedding quality, and how many chunks you pull now decide correctness. Pull too few and you miss a relevant clause; pull too many and you are drifting back toward stuffing the window. Choosing where that index lives is its own decision, covered in picking a vector store for Bedrock RAG.
Map-reduce is the pick when the answer has to account for the entire document, because retrieval’s whole premise is that a small slice suffices and “every obligation in the contract” breaks that premise. Each chunk is processed in a call that comfortably fits, so nothing overflows and no text is left out, which is the specific weakness of truncation. The trade is total token cost: you are reading the whole document across many calls, so the input bill is higher than retrieval’s, and the reduce step has to merge partial answers that each saw only their chunk, which is where duplicates and contradictions creep in and need reconciling.
Summarise-then-reason compresses the input for gist-level tasks, and its one real hazard is that summarising discards exact wording. It is the right call for the risk overview and the wrong call the moment the task needs to quote or reason about a precise figure, because the number you need may not have survived the first pass. Where both matter, the strategies compose: summarise for the overview, retrieve the exact clause when a precise value is required, map-reduce when coverage has to be total. The reason to reach for a bigger window is narrow, when a task genuinely needs long-range context across the whole document in a single pass that none of these preserve, and even then it is a cost-and-latency decision made with eyes open, not a reflex.
Worked example
The batch mixes ten-page and two-hundred-page contracts, the model call has a window of, say, a few hundred thousand tokens, and three questions are asked of every document: the termination notice period, the overall risk summary, and the full list of obligations. Rather than one prompt per document, the job reserves output first and routes each question to the strategy that fits it.
Reserve the output. The risk summary needs room to write, so the job sets max-output-tokens to a firm ceiling, say 1,500 tokens, and subtracts that plus the instruction overhead from the window before counting any document text. That reservation is what stops a long input from crowding out the answer.
Route the notice-period question through retrieval. It is a single fact that lives in one or two clauses, so the job retrieves the chunks about termination and puts only those in the prompt. A two-hundred-page contract contributes a few hundred tokens to the call instead of its full length, the window never comes close to overflowing, and the answer is drawn from focused text rather than fished out of the middle of a huge context. Input cost per document for this question drops by more than an order of magnitude against pasting the whole thing.
Route the risk summary through summarise-then-reason. The job compresses each document, by map-reducing a summary if the document itself is too big to summarise in one pass, then reasons over the compressed version to write the overview. The final reasoning call is cheap because its input is a short summary, and gist-level risk survives compression even though exact clause wording does not.
Route the obligations list through map-reduce. Because it must cover the whole document, the job splits the contract into window-sized chunks, extracts obligations from each, then reduces the per-chunk lists into one deduplicated list. Every page is read, nothing is truncated, and the higher token cost is accepted deliberately because completeness is the requirement here in a way it was not for the single-fact question. Three questions, three strategies, one reserved output budget, and not one of them pastes two hundred pages into the largest window on the menu.
What’s worth remembering
- The context window is a single ceiling over input plus output combined, not a measure of how much document fits; push the input to the ceiling and there is no room left to answer.
- Reserve space for the output before you budget the input, and set max-output-tokens explicitly; the window caps the total while max-output-tokens caps generation within whatever room remains.
- A bigger window is not free: it costs more, adds latency, and does not fix quality loss when a relevant fact is buried in the middle of a very long context.
- Retrieval puts only the relevant chunks in the prompt, which slashes input cost and often improves the answer; the risk shifts to whether retrieval surfaced the right chunks.
- Batch inference bills at half the on-demand rate, and prompt caching bills a repeated prefix at the cache-read rate, but the two are mutually exclusive and batch rules out tool calling.
- Match the strategy to the question: retrieval for a fact in a few pages, summarise-then-reason for the gist, map-reduce for total coverage, and a bigger window only when the task genuinely needs whole-document context in one pass.