Exam-style
A team runs a summarisation service on a foundation model. Every call sends a 400-token system prompt plus a document averaging 6,000 tokens, and gets back a summary of about 200 tokens. Cost was estimated from the summaries, and the monthly bill instead moves with document length. Which explanation accounts for it?
Reveal the answer
C. Input tokens are metered alongside output tokens, and the full 6,400-token prompt is counted on every call, so the input side is more than thirty times the output side
Amazon Bedrock meters the tokens going in as well as the tokens coming out, and prices the two separately. Here 6,000 tokens of document plus a 400-token system prompt make a 6,400-token input against a 200-token output, thirty-two to one. Output usually lists at four to five times the input rate, which thirty-two to one swamps. Lengthen the documents and the bill follows; shorten the summaries and barely anything moves. Input is counted on every call, so a system prompt sent with every request is a standing charge. Prompt caching can bill a repeated prefix at a lower cache-read rate, but it does little here: the documents differ each time, and 400 tokens sits under the cache-checkpoint minimum for most models. The context window is a ceiling on what one call may carry, not a flat-rate allowance, so a larger window leaves the per-token rates alone. Streaming changes when the first token arrives, not how many are counted. The lever is the input side: trim the system prompt, and send only the passages the summary needs.
Q. A 400-token system prompt plus a 6,000-token document goes in; a 200-token summary comes out. Why does the bill track the documents instead of the summaries?
A. Because input tokens are metered too. Under the token-based pricing model you are charged for the tokens sent as well as the tokens generated, and thirty-two times as many go in as come out.
Why? Output tokens usually carry the higher rate, which is what makes a summary-only estimate look reasonable. Current text models list output at roughly four to five times input, and a ratio of thirty-two to one swamps that. Input is counted on every call, so a system prompt sent with every request is a standing charge. Prompt caching can bill a repeated prefix at a lower cache-read rate, but the documents differ each time and 400 tokens sits under the cache minimum for most models. A bigger context window raises the ceiling on what one call can carry and leaves the rates alone; streaming changes when the first token arrives, not how many are counted. Budgeting a long-document workload starts on the input side: shorter system prompt, fewer pages per call, nothing sent twice.