Exam-style
A retailer generates product descriptions with a foundation model, billed per input and output token. Two things are wrong. Descriptions come back two or three paragraphs longer than the product page has space for, so a copywriter trims every one by hand. And every call opens with the same 3,000-token block of brand guidelines, unchanged for months, sent again in full each time. Which pair of changes addresses the bill?
Reveal the answer
C. Set a maximum output length to bound how many tokens the model may generate, and enable prompt caching over the unchanging guidelines block so its input tokens are not charged in full on every call
Two costs, two settings. The descriptions are too long, and output tokens are billed, so the lever is the maximum output length: an inference parameter that caps how many tokens one call may generate. The guidelines block is unchanging and sent again on every call, so the lever is prompt caching, which keeps the processed form of a repeated prefix and charges a reduced rate for the cached part on later calls. It needs an identical prefix, so anything variable, the product name or the SKU, goes after the cached block rather than inside it. Two gotchas belong with the first setting. A maximum output length truncates; it does not summarise. Set it below what the description needs and the last sentence stops mid-word, so the prompt still has to ask for the length wanted and the cap sits behind it as a ceiling. And input and output tokens are priced separately, at different rates, so capping the answer does nothing about a three-thousand-token prefix and caching the prefix does nothing about an over-long answer. Temperature controls how much randomness there is in the choice of the next token. Turn it down and the descriptions get more predictable and no shorter, and a larger model costs more per token in both directions. Top-k widens or narrows the pool the model samples from, which is another randomness control rather than a length control. A larger context window raises the ceiling on how much text one call may carry without changing the rate on any of it, so the same block costs the same in a bigger window. Batch inference does cut the rate, and suits work nobody is waiting for, but a copywriter is waiting on this one and the guidelines are still paid for on every request in the batch. Shortening the guidelines by summarising them is a real option, worth weighing: it cuts the input on every call, at the price of the instruction fidelity the full text buys, and it can be done alongside caching rather than instead of it.
Q. Product descriptions run two or three paragraphs past what the page has room for, and the same 3,000-token brand-guidelines block opens every call. Which two settings deal with the bill?
A. A maximum output length to bound the answer, and prompt caching over the guidelines block. Both are named in the selection criteria and inference parameters you set per call.
Why? Input and output tokens are metered separately, at different rates, so an over-long answer and a repeated prefix are two bills with two fixes. Input/output length is a property you design for: the maximum output length is one of the inference parameters set on each request, and it caps how many tokens the model may generate. It truncates rather than summarises, so ask for the length you want in the prompt and keep the cap behind it as a ceiling. Prompt caching handles the other half, holding the processed form of an identical opening block so later calls pay a reduced rate for it, which needs the variable text to sit after the block rather than inside it. Temperature and top-k change how varied the wording is, not how long it runs. A bigger context window buys room, never a cheaper token, and a batch discount is for work with nobody waiting on it. Summarising the guidelines down is worth weighing too, and it trades instruction fidelity for input tokens.