Exam-style
A retailer generates product descriptions with a foundation model, billed per input and output token. Descriptions come back two or three paragraphs longer than the page has room for, so a copywriter trims every one by hand. And every call opens with the same 3,000-token block of brand guidelines, unchanged for months, resent in full. Which pair of changes addresses the bill?
Reveal the answer
C. Set a maximum output length to cap how many tokens one call may generate, and cache the unchanging guidelines block so it is billed at the cache-read rate on later calls
Two costs, two settings. Output tokens are billed, so the over-long descriptions point at the maximum output length: maxTokens in the Converse API, a cap on how many tokens one call may generate. It truncates rather than summarises, returning a stop reason of max_tokens part-way through a sentence, so ask for the length wanted in the prompt and keep the cap behind it. The repeated block is the other half. Prompt caching reuses the processed form of an identical prefix and bills those tokens at the model’s cache-read rate rather than the standard input rate, so the product name and SKU go after the cached block, not inside it. A cache checkpoint also has a token minimum, 512 to 4,096 depending on the model. Input and output are metered separately, so neither setting touches the other half. Temperature and top-k change the next-token distribution, not the length. A larger context window raises what one call may carry and leaves the per-token rates alone. Batch inference does list at half the on-demand rate, but it runs asynchronously while a copywriter waits, and prompt caching is not supported with the batch API.
Q. Product descriptions run two or three paragraphs past what the page has room for, and the same 3,000-token brand-guidelines block opens every call. Which two settings deal with the bill?
A. A maximum output length to bound the answer, and prompt caching over the guidelines block. Both sit in the selection criteria and inference parameters you set per call.
Why? Input and output tokens are metered separately, at different rates, so an over-long answer and a repeated prefix are two bills with two fixes. The maximum output length is an inference parameter, maxTokens in the Converse API, capping how many tokens one call may generate. It truncates rather than summarises, coming back with a stop reason of max_tokens part-way through a sentence. So ask for the length you want in the prompt and keep the cap behind it. Prompt caching handles the other half. It reuses the processed form of an identical opening block and bills those tokens at the cache-read rate. The variable text goes after the block, not inside it. Temperature and top-k change how varied the wording is, not how long it runs. A bigger context window raises what one call can carry and leaves the per-token rates alone, and a batch discount applies to work with nobody waiting on it, where caching is unavailable.