Tuning How a Model Samples: Temperature, Top-P, and Top-K

The situation

A team is running four LLM features on Amazon Bedrock behind one shared client wrapper: a ticket classifier that returns one of six labels, an extraction job that turns emails into structured records, a marketing-copy generator that writes three subject-line options, and a code assistant that drafts small functions. When the wrapper was first written, someone set a single default for the whole fleet, temperature 0.7, and every feature inherited it.

The results follow from what that number does. The classifier returns different labels for the same ticket, “billing” on one call and “account” on the next, and the evaluation harness records the variation as noise nobody can chase down. The extraction job sometimes returns a field value that appears nowhere in the email. The copy generator is fine. The code assistant is merely inconsistent. One default suits one of the four features, by accident.

Guessing replacement numbers is not a plan. The same question sits under all four: what does each sampling parameter change, and which setting does this task need?

What actually matters

A model does not emit an answer in one step. At each step it produces a probability distribution over the vocabulary and samples one token from it. Every sampling parameter reshapes or truncates that distribution before the draw, which is why AWS files them together under randomness and diversity. Understanding them means understanding where in the distribution each one acts.

Start with the axis. Determinism and variety are the two ends of one line, and most task pain comes from sitting at the wrong end. A classifier, an extractor, a factual lookup, a tool-calling agent selecting a function: these need the highest-probability token nearly every time, because there is a right answer and drift off it is error. Brainstorming, subject lines, alternative phrasings: these need sampling from further down the distribution, because the value is in the candidates the top token excludes. Two of the four features above are running a distribution-widening setting on work that needed the opposite.

Next, the parameters act in different places. Temperature rescales the whole distribution, flattening it so tokens sit closer to equally likely, or steepening it so the mass concentrates on the front-runners. Top-p and top-k truncate instead. They cut the candidate set before sampling, so the long tail is never drawn from at all. Rescaling and truncating interact, and the model documentation is blunt about it: Anthropic’s Bedrock parameter page says to modify either temperature or top_p and not both at the same time, and the Amazon Nova schema repeats the instruction. On Claude Sonnet 4.5 and Claude Haiku 4.5 it is a constraint rather than advice, because those two models accept either temperature or top_p and not both.

Third, lower is not automatically safer. Temperature near zero is greedy decoding, the top token every step, which is what a label needs and what tends to flatten generative text into repetition. Zero is also not portable: Amazon Nova accepts temperature between 0.00001 and 1 inclusive, so a literal 0.0 falls outside its documented range. Nor is a low temperature a reproducibility contract. Cohere Command exposes a seed parameter for exactly this, and documents the limit in the same breath: repeated requests with the same seed and parameters should return the same result, but determinism cannot be totally guaranteed. Where a feature needs byte-identical output for audit, cache the result.

Fourth, the cap that has nothing to do with randomness. maxTokens bounds the response and it is a hard cut. When generation reaches it, Converse returns stopReason: max_tokens with the text stopped wherever it stopped, mid-sentence if that is where the ceiling fell. Set it too low and structured output arrives unparseable. Omit it and it defaults to the maximum the model allows, which for the Amazon Nova understanding models is 5,000 tokens and differs by family elsewhere, so a runaway generation runs up tokens and latency against a ceiling nobody chose.

These are inputs to the call, so they belong with the call. The Converse API carries temperature, topP, maxTokens and stopSequences in one inferenceConfig block, and model-native parameters travel alongside in additionalModelRequestFields. Setting them per feature rather than once for the fleet is the fix here.

What we’ll filter on

1. Determinism need: does the task have one right answer that must not drift, or is variety wanted?
2. Where it acts: does the parameter rescale the whole distribution, or truncate it to a candidate set?
3. Interaction safety: does the change move one dial, or two that aim at the same behaviour?
4. Portability: is the parameter in the Converse base set, or model-native and spelled differently per family?
5. Output-length control: is the response bounded, so structured output cannot be cut off?
6. Reproducibility expectation: is “usually the same” enough, or is byte-identical output being assumed where no documentation promises it?

The landscape

Temperature. A single scalar that scales the sharpness of the whole distribution before sampling. Lower values concentrate probability on the highest-scoring tokens, so output is focused and close to deterministic. That is the setting for classification, extraction, and factual answers. Higher values flatten the distribution, narrow the gap between likely and unlikely tokens, and produce less obvious continuations, which is what brainstorming and creative copy need. Both ends have failure modes. Too low and generative text turns flat and repetitive; too high and it drifts into incoherence and invented detail. Ranges differ by family, and Converse is the tighter of the two constraints: inferenceConfig.temperature accepts 0 to 1, while a native call to AI21 Jamba accepts 0.0 to 2.0 and defaults to 1.0. Nova’s floor is 0.00001 with a default of 0.7.

Top-p (nucleus sampling). Top-p truncates rather than rescales. Sort tokens by probability, accumulate downward, keep the smallest set whose cumulative probability reaches p, and sample from that set. AWS gives a worked example: with horses at 0.7, zebras at 0.2 and unicorns at 0.1, p = 0.7 leaves horses alone, and p = 0.9 leaves horses and zebras. Set size therefore tracks the shape of the distribution at that step, three tokens where it is peaked and forty where it is flat. Lower p tightens the pool toward the front-runners; p = 1 disables the cut. It sits in the Converse base set beside temperature, with a range of 0 to 1.

Top-k. The blunter truncation: keep the k highest-probability tokens and sample from those, whatever share of the mass they cover. k = 1 is greedy decoding. Because k is a fixed count, the pool does not grow where the distribution is flat or shrink where it is peaked. Top-k is not in the Converse base set, and it is not universal. Meta Llama on Bedrock takes temperature, top_p and max_gen_len and nothing else, and neither the Mistral chat completion schema nor the AI21 Jamba parameter set has an equivalent. Where it does exist the spelling moves. Claude takes top_k directly in additionalModelRequestFields; Nova takes it nested there as {"inferenceConfig": {"topK": 20}} and caps it at 128; Cohere Command R calls it k. A feature built on it does not move between families unchanged.

Max tokens. Not a sampling parameter, but set in the same block and easy to get wrong. It caps generated tokens, as a hard stop. Check it first when a structured response comes back truncated: the shape was fine, the ceiling was too low. Size it to the longest legitimate output the feature produces, with headroom, rather than to the typical one.

Evaluation

Side by side

The first two columns carry the rule. Temperature and top-p aim at the same behaviour, how far below the front-runner sampling may go, which is why the model documentation says to move one and leave the other alone. Read the table against the four features and the assignments fall out: the classifier and extractor want low temperature and no top-p change, the copy generator runs better at a higher temperature, the code assistant sits low but not at zero, and all four need a max-tokens ceiling sized to their real output.

The solution

The classifier and the extraction job are the determinism cases, and the shared 0.7 default mis-set both. Take temperature down to 0, or to the smallest value the model accepts, and leave top-p unset. At that setting the model returns its highest-probability token nearly every step, so the same ticket yields the same label call after call and the evaluation noise clears. For extraction, sampling stops dipping into the tail for a plausible-looking field value that was never in the source text, which is where the invented fields came from. One dial, moved on the two features that needed it. Do not tighten top-p or top-k as well, because stacking truncation on top of a low temperature makes the behaviour harder to reason about and changes nothing useful.

The copy generator is the feature the default happened to suit, and it is worth understanding why before somebody “fixes” it. Three distinct subject lines require sampling past the single most likely continuation, which a temperature around 0.7 to 1.0 provides. If the options come back samey, raise temperature; if they drift into nonsense, pull it back. Tune the dial that is already moving and leave top-p alone.

The code assistant sits in the middle, and it shows why “lower is safer” is not a rule. Code needs to be mostly deterministic, because there is usually a correct structure, but temperature 0 pushes models toward looping and rigid output, so a low-but-nonzero setting in the rough vicinity of 0.2 gives stable drafts without the degeneracy. Every feature also needs a max-tokens ceiling sized to its real output. The extraction job in particular must not have its JSON cut off by a ceiling set for one-line labels, because a half-emitted object fails the downstream parser exactly as a malformed one would.

On Bedrock the mechanics are the same across all four. Converse takes temperature, topP and maxTokens in inferenceConfig, and anything model-native, top-k included, goes through additionalModelRequestFields. Set these per feature in the client wrapper instead of inheriting one fleet-wide default.

Worked example

Before, every feature inherits the fleet default. The classifier call looks like this:

response = client.converse(
    modelId=MODEL_ID,
    messages=messages,
    inferenceConfig={"temperature": 0.7, "maxTokens": 1024},
)

At temperature 0.7 the distribution stays flat enough that the second- and third-choice labels retain real probability, so an ambiguous ticket scoring billing 0.55 and account 0.40 is sampled as account a meaningful fraction of the time. Run the same ticket ten times and the answers spread. That is the noise in the evaluation harness, and it makes the label boundaries look fuzzier than they are.

After, the sampling matches the task. Classification needs the top token and a maxTokens sized to a short label rather than a paragraph:

response = client.converse(
    modelId=MODEL_ID,
    messages=messages,
    inferenceConfig={"temperature": 0.0, "maxTokens": 16},
)

Now the model returns its most probable label on nearly every call, the same ticket comes back the same, and the harness measures the classifier rather than sampling jitter. Two details in that config matter. topP is left out rather than pinned to 1.0, because Claude Sonnet 4.5 and Claude Haiku 4.5 accept either temperature or top_p and not both, so sending the pair is a portability trap as well as a redundant dial. And a literal 0.0 is below Amazon Nova’s documented floor of 0.00001, so a wrapper that fans out across families should clamp rather than hard-code the zero. Even then this is strongly deterministic, not a guarantee of identical bytes; where a feature needs audit-grade repeatability, cache the result.

What’s worth remembering

1. Every sampling parameter reshapes or truncates the next-token probability distribution before the draw; temperature rescales it, top-p and top-k cut it down.
2. Move temperature or top-p, not both: they aim at the same behaviour, and Claude Sonnet 4.5 and Haiku 4.5 accept only one of the two.
3. Use low temperature for classification, extraction, factual answers and tool use; use higher temperature for brainstorming and creative copy.
4. Lower is not automatically safer. Temperature 0 flattens generative and code output, and it falls outside Amazon Nova’s accepted range of 0.00001 to 1.
5. Only temperature, topP, maxTokens and stopSequences are in the Converse base set. Top-k is model-native, spelled differently per family, and absent from Meta Llama, Mistral and AI21 Jamba.
6. Max tokens is a hard cap that returns stopReason: max_tokens mid-output, so size it to the longest legitimate response or watch structured payloads arrive truncated.