Budgeting Tokens for a Long-Document Workload

The situation

A team has a batch job on Amazon Bedrock that reads contracts. Each document runs from a few pages to a couple of hundred, and for every one the job asks a model to pull out key dates, parties, obligations, and any unusual clauses, then write a short risk summary. It worked fine on the ten-page samples. In production, on the real spread of documents, two things break.

The long contracts overflow the context window. The job pastes the whole document into a single prompt, and when a document is big enough the request is rejected before the model sees it, or the paste is silently truncated and the summary confidently describes a contract whose last forty pages were never read. Either way the output is wrong or missing.

The bill is the other problem. Feeding entire documents means paying for every token of every page on every call, whether or not the answer needed page 90. Someone suggested moving to a model with a much larger context window so the biggest contracts fit. That removes the overflow, but the cost per document goes up rather than down, the calls get slower, and on the longest documents the summaries start missing clauses that are demonstrably in the text. Bigger window, worse answers. The question underneath is how to fit a long-document task into a token budget that is cheap and reliable, not just large enough.

What actually matters

A token is the unit the model and the bill both count in. It is roughly a sub-word piece, not a word and not a character; common short words are a single token, longer or rarer words split into several, and a rough English rule of thumb is about four characters or three-quarters of a word per token. The reason it matters here is that every property that follows, the window limit, the input price, the output price, is denominated in tokens, so estimating token counts is the first real skill of budgeting a workload.

The context window is a single ceiling over input plus output combined. It is not “how much document fits”, it is how much document plus instructions plus the model’s own answer can coexist in one call. Push the input close to the ceiling and there is no room left for the model to write, so the answer gets cut off mid-sentence or the request fails. Any budget has to reserve space for the output, which is why the window and the maximum output length are two different limits that both bind: the window caps the total, and a separate max-output-tokens setting caps how much the model is allowed to generate within whatever room is left.

Cost has two prices, not one. Input tokens and output tokens are billed separately, and output is typically the dearer of the two per token. A long-document job is usually input-heavy, so the document you paste dominates the bill, which is exactly why shrinking the input is the highest-leverage cost move. But a job that generates long summaries for many documents can have output costs that matter too, so both sides of the ledger are worth watching rather than assuming input is the whole story.

A bigger window is not free and not automatically better. It costs more, because you are paying for more input tokens; it is slower, because the model has more to read before it answers; and quality can degrade as the context grows, because a fact that matters can sit in the middle of a very long context and get less attention than the same fact in a short, focused prompt. The lesson is that fitting the document is necessary but not sufficient. A summary built from ten well-chosen pages is often better and cheaper than one built from two hundred pages the model half-read.

That reframes the whole task. The goal is not to make the document fit, it is to put in front of the model only the tokens the answer actually needs, and to leave enough of the window for the answer. Everything below is a way to do that: retrieve the relevant parts instead of pasting all of them, compress before you reason, or split the document into chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. and combine the results.

What we’ll filter on

1. Does the whole document fit in the window with room reserved for the output, or does it overflow?
2. Does the task need the whole document at once, or only the parts relevant to a specific question?
3. Input token cost per document, since input usually dominates a long-document bill.
4. How much of the window is left for the output, and does the job set an explicit max-output-tokens?
5. Risk of silent truncation or lost-in-the-middle quality loss as the input grows.
6. Latency budget: how slow per document is acceptable across the batch?

The token-budget landscape

Stuff the whole document. Paste everything into one prompt and ask for the answer. Simplest to build, and fine when documents are reliably small relative to the window. It fails on exactly this workload: the long ones overflow, you pay for every page on every call whether it was relevant or not, and even when it fits, quality can sag on the longest inputs as key facts get buried. It is the baseline the other strategies exist to beat.

Truncate to fit. Cut the document to the first N tokens so it slides under the ceiling. Cheap and trivial, and occasionally right when the answer genuinely lives at the top of every document. On contracts it is dangerous, because the clause that matters might be on page 120, and truncation removes it without warning. The output looks complete and is simply wrong about the part that got cut. If you truncate at all, do it knowingly and never on documents where the tail carries meaning.

Retrieval (RAG). Split documents into chunks, index them in a vector store, and at query time retrieve only the chunks relevant to the question and put those in the prompt. The model sees a handful of pages instead of two hundred, so input tokens per call drop sharply, the window stops overflowing, and the relevant text sits in a short focused context where it gets full attention. The cost is an indexing pipeline and the risk that retrieval misses a relevant chunk, so the answer is only as good as what retrieval surfaced. Amazon Bedrock Knowledge Bases packages the whole pipeline (chunking, embedding, indexing, and retrieval at query time) so the build cost is configuration rather than code. This is the default for question-answering over a large or growing corpus, where each question needs a small slice rather than the whole library.

Summarise then reason. Compress the document into a shorter representation first, then do the real task on the summary. Two passes, and the second pass is cheap because its input is small. It fits tasks where a faithful condensation preserves what the answer needs, and it loses on tasks that turn on exact wording, because summarising throws away the precise clause you might need to quote. Good for “what is the overall risk”, weaker for “what is the exact indemnity cap”.

Map-reduce over chunks. Split the document, run the task on each chunk independently (the map step), then combine the per-chunk results into a final answer (the reduce step). Every chunk fits comfortably, so nothing overflows and the whole document genuinely gets read, unlike truncation. It costs more calls and therefore more input tokens overall than retrieval, and the reduce step has to reconcile chunk results that each saw only part of the picture. This is the strategy when the task must cover the entire document, like extracting every obligation, rather than answering one question that lives in a few pages.

Bigger context window. Move to a model or configuration with a larger window so more of the document fits at once. It genuinely removes overflow and it is the least code to change. It does not remove the input cost, it raises it; it adds latency; and it does not fix lost-in-the-middle quality loss. It earns its place when a task truly needs long-range context that spans the whole document in one pass and the smarter strategies cannot preserve it, not as the reflexive answer to “it does not fit”.

Side by side

Strategy	Reads whole document	Input tokens per document	Overflow risk	Quality risk	Best for
Stuff everything	✓	Highest	✗ (overflows)	Lost-in-the-middle on long inputs	Reliably small documents
Truncate to fit	✗	Capped	✓ safe	Silently drops the tail	Answer always near the top
Retrieval (RAG)	✗ (relevant slice)	Low	✓ safe	Missed chunk on retrieval	One question over a large corpus
Summarise then reason	✓ (compressed)	Low on the second pass	✓ safe	Loses exact wording	Gist and overall-risk tasks
Map-reduce	✓	High (many calls)	✓ safe	Reduce step reconciles partial views	Cover-everything extraction
Bigger window	✓	Highest	✓ safe	Cost, latency, lost-in-the-middle	Genuine whole-document context

Reading it against the contract job: the per-question lookups (“what is the termination notice period”) want retrieval; the “summarise the overall risk” step wants summarise-then-reason; the “list every obligation” step wants map-reduce; and none of the three wants the whole document pasted into the biggest window available.

The picks in depth

Start by reserving the output. Before choosing a strategy, decide how many tokens the answer needs and set the cap explicitly. On the Bedrock Converse API that is inferenceConfig.maxTokens, and a response that runs into it comes back with a stopReason of max_tokens rather than an error, so a summary cut off mid-sentence still looks like a successful call unless something checks. Treat the window, minus that reservation, minus your instructions, as the real budget for document text. A job that leaves this implicit is the one that returns half-written summaries, because the input grew until there was no room to answer. The window is a shared ceiling, and the output has to be first in the queue for space, not last.

For the fits-in-one-call decision, estimate tokens rather than guess from page count. Use a token count from the model’s tokeniser or a rule-of-thumb ratio to turn document size into a token estimate, compare it against the budget after the output reservation, and route each document accordingly: small ones can go straight in, large ones need retrieval, map-reduce, or summarisation. The estimate is what lets the batch job branch per document instead of applying one strategy to a spread of sizes it does not fit.

The shape of the job is a lever of its own, separate from the shape of the prompt. Nothing in a nightly contract run is waiting on a response, and on the models that support it Bedrock prices batch inference at half the on-demand per-token rate: write the prompts as JSONL to S3, submit a CreateModelInvocationJob naming the model and the input and output locations, and collect the results from the output prefix when the job finishes. The rate cut applies whichever strategy you land on, so it stacks with the input-shrinking work rather than competing with it. The catch is that batch runs asynchronously, so it suits the overnight sweep and not the interactive lookup, and it does not support tool calling, so a strategy that leans on a declared schema has to stay on the synchronous path.

Retrieval is the workhorse for question-answering because it attacks the input cost directly: instead of paying for every page, you pay for the handful of chunks that answer the question, and the answer usually improves because the relevant text is in a short focused prompt rather than buried on page 90. The failure mode moves from the window to the retriever, so chunking, embedding quality, and how many chunks you pull now decide correctness. Pull too few and you miss a relevant clause; pull too many and you are drifting back toward stuffing the window. Choosing where that index lives is its own decision, covered in picking a vector store for Bedrock RAG.

Map-reduce is the pick when the answer has to account for the entire document, because retrieval’s whole premise is that a small slice suffices and “every obligation in the contract” breaks that premise. Each chunk is processed in a call that comfortably fits, so nothing overflows and nothing is silently dropped, which is the specific weakness of truncation. The trade is total token cost: you are reading the whole document across many calls, so the input bill is higher than retrieval’s, and the reduce step has to merge partial answers that each saw only their chunk, which is where duplicates and contradictions creep in and need reconciling.

Summarise-then-reason is the cheap compressor for gist-level tasks, and its one real hazard is that summarising discards exact wording. It is the right call for the risk overview and the wrong call the moment the task needs to quote or reason about a precise figure, because the number you need may not have survived the first pass. Where both matter, the strategies compose: summarise for the overview, retrieve the exact clause when a precise value is required, map-reduce when coverage has to be total. The reason to reach for a bigger window is narrow, when a task genuinely needs long-range context across the whole document in a single pass that none of these preserve, and even then it is a cost-and-latency decision made with eyes open, not a reflex.

A worked example: routing one contract batch

The batch mixes ten-page and two-hundred-page contracts, the model call has a window of, say, a few hundred thousand tokens, and three questions are asked of every document: the termination notice period, the overall risk summary, and the full list of obligations. Rather than one prompt per document, the job reserves output first and routes each question to the strategy that fits it.

Reserve the output. The risk summary needs room to write, so the job sets max-output-tokens to a firm ceiling, say 1,500 tokens, and subtracts that plus the instruction overhead from the window before counting any document text. That reservation is what stops a long input from crowding out the answer.

Route the notice-period question through retrieval. It is a single fact that lives in one or two clauses, so the job retrieves the chunks about termination and puts only those in the prompt. A two-hundred-page contract contributes a few hundred tokens to the call instead of its full length, the window never comes close to overflowing, and the answer is drawn from focused text rather than fished out of the middle of a huge context. Input cost per document for this question drops by more than an order of magnitude against pasting the whole thing.

Route the risk summary through summarise-then-reason. The job compresses each document, by map-reducing a summary if the document itself is too big to summarise in one pass, then reasons over the compressed version to write the overview. The final reasoning call is cheap because its input is a short summary, and gist-level risk survives compression even though exact clause wording does not.

Route the obligations list through map-reduce. Because it must cover the whole document, the job splits the contract into window-sized chunks, extracts obligations from each, then reduces the per-chunk lists into one deduplicated list. Every page is read, nothing is truncated, and the higher token cost is accepted deliberately because completeness is the requirement here in a way it was not for the single-fact question. Three questions, three strategies, one reserved output budget, and not one of them pastes two hundred pages into the largest window on the menu.

What’s worth remembering

1. A token is a sub-word unit, roughly three-quarters of a word or four characters of English, and it is the unit both the window limit and the bill are counted in, so estimate token counts rather than guess from words or pages.
2. The context window is a single ceiling over input plus output combined, not a measure of how much document fits; push the input to the ceiling and there is no room left to answer.
3. Reserve space for the output before you budget the input, and set max-output-tokens explicitly; the window caps the total while max-output-tokens caps generation within whatever room remains.
4. Input and output tokens are priced separately and output usually costs more per token, but a long-document job is input-heavy, so shrinking the input is the biggest cost lever.
5. A bigger window is not free: it costs more, adds latency, and does not fix quality loss when a relevant fact is buried in the middle of a very long context.
6. Truncating to fit silently drops whatever is past the cut, so it is only safe when the answer reliably lives near the top of the document.
7. Retrieval puts only the relevant chunks in the prompt, which slashes input cost and often improves the answer; the risk shifts to whether retrieval surfaced the right chunks.
8. Summarise-then-reason is cheap for gist and overall-risk tasks but discards exact wording, so it is the wrong tool when the answer turns on a precise figure or clause.
9. Map-reduce reads the whole document across many chunked calls, so it is the pick for cover-everything extraction at the price of higher total token cost and a reduce step that reconciles partial views.
10. Match the strategy to the question: retrieval for a fact in a few pages, summarise-then-reason for the gist, map-reduce for total coverage, and a bigger window only when the task genuinely needs whole-document context in one pass.