Exam-style
An extraction job pulls fixed fields out of invoices into a JSON schema. Run the same invoice twice and the field wording differs, and occasionally the model appends a sentence of commentary after the JSON. Which inference settings should change, and in which direction?
Reveal the answer
C. Lower temperature toward 0, leave top-p alone, and set max tokens and a stop sequence to bound the response
Temperature shapes the probability distribution the next token is drawn from, and a value near 0 steepens it so higher-probability tokens are selected. That is what a fixed-schema extraction needs. Bedrock documents adjusting either temperature or top-p, not both at once, and Claude Sonnet 4.5 and Haiku 4.5 accept only one of the two, so moving both is the wrong habit even on older models that allow it. Max tokens bounds the response length, and a stop sequence is the documented way to halt generation at the close of the JSON rather than after trailing commentary. Claude on Bedrock defaults to temperature 1, its documented maximum, and top-p 0.999, so the sampling already sits at the wide end: raising top-p widens it a little further, top-k at 250 trims only the long tail of a distribution that is otherwise untouched, and a larger model at the defaults samples just as freely. Cutting max tokens to 50 truncates the JSON mid-object, and the response comes back with a stop reason of max_tokens.
Q. Outputs are too random for a structured extraction task. Which inference parameters, and which way?
A. Lower temperature toward 0 so higher-probability tokens are selected. Claude’s documented default on Bedrock is temperature 1, its maximum, so an extraction job left alone is sampling at the widest setting the parameter offers. Adjust temperature or top-p, not both: Bedrock documents changing one of the two, and Claude Sonnet 4.5 and Haiku 4.5 accept only one. Set max tokens to bound the response length, and a stop sequence to end generation at the close of the JSON.
Why? Temperature reshapes the probability distribution, steepening it toward the likely tokens as it falls and flattening it as it rises. Top-p truncates that distribution instead, to the candidates making up its top P per cent. Extraction wants the narrow end of either, brainstorming the wide one. For schema conformance itself, structured outputs constrains the response to a supplied JSON schema, which parameter tuning only approximates.