Exam-style
A team runs a summarisation service on a foundation model. Every call sends a 400-token system prompt plus a document that averages 6,000 tokens, and gets back a summary of about 200 tokens. Cost was estimated from the summaries, so the team is surprised to find the monthly bill moving with the length of the documents rather than the length of the summaries. Which explanation accounts for it?
Reveal the answer
C. Input tokens are metered alongside output tokens, and the full 6,400-token prompt is charged on every single call, so the input side is more than thirty times the output side and document length dominates
Two rates apply to every call, not one. A token is a chunk of text a little shorter than a word. Inference is charged on the tokens going in as well as the tokens coming out. Here 6,000 tokens of document and 400 tokens of system prompt make a 6,400-token input against a 200-token output, a ratio of thirty-two to one. The output rate is typically several times the input rate. Even at five times, a ratio of thirty-two to one leaves the input side carrying most of the bill. Lengthen the documents and the bill follows; shorten the summaries and barely anything moves. The whole prompt is charged every time, because the model holds nothing between calls. A system prompt sent on every request is a standing charge, and a document sent twice is paid for twice. The context window is a ceiling on how much text one call may carry, not an allowance costing the same whether or not it is used. A bigger window buys room, not cheaper tokens. A longer document in a longer window costs more. Streaming changes when the first token appears on screen. That improves perceived latency; the token count is identical either way. The lever that applies here is the input side. Trim the system prompt, send the passages the summary needs rather than the whole document, and stop resending text the model has already been paid to read.
Q. A 400-token system prompt plus a 6,000-token document goes in; a 200-token summary comes out. Why does the bill track the documents instead of the summaries?
A. Because input tokens are metered too. Under the token-based pricing model you pay for what the model reads as well as what it writes, and here it reads thirty-two times as much as it writes.
Why? Output tokens usually carry the higher rate per token, which is what makes a summary-only estimate feel reasonable. A ratio of thirty-two to one drowns a rate difference of five to one. The full prompt is charged on every call, since nothing carries over between them. The system prompt is a standing per-request charge, and a document sent again is bought again. A bigger context window raises the ceiling on how much one call can carry without making any of it cheaper, and streaming moves when the first token lands rather than how many there are. Budgeting a long-document workload starts on the input side: shorter system prompt, fewer pages per call, nothing sent twice.