These are set per request, not per model, so two calls to the same model an hour apart can behave nothing like each other. The selection walkthrough works them through against three real features; this card is the four of them on one page, because a scenario names the symptom and expects you to reach for the right slider.
Two of them get confused for each other often enough to be worth separating. Temperature and top-p both control randomness, and max tokens controls length, so an over-long answer is never a temperature problem. Input/output length is the pricing half of the same story: tokens in and tokens out are billed separately, and both share the room in the context window.
Flash card
The inference parameters: settings sent with each request that change the answer without changing the model or the prompt. Temperature, top-p, top-k and max tokens.
- Temperature scales how much randomness is allowed into the choice of the next token. Low values give focused, repeatable output, which suits extraction, classification and structured summarising. High values give variety, which suits drafting and brainstorming.
- Top-p, also called nucleus sampling, restricts the choice to the smallest set of tokens whose probabilities add up to p. It does a similar job to temperature by a different route, and lowering both at once is the usual answer for a task with a fixed shape to hit.
- Top-k caps the candidate list at the k most likely tokens regardless of how their probabilities are spread, which is the same narrowing counted a different way.
- Max tokens bounds the length of the answer, and so bounds latency and output cost with it. It truncates when the answer runs long rather than summarising, so ask for the length wanted in the prompt and keep the cap behind it as a ceiling.
- Input and output length are metered separately for pricing, at different rates, and both are constrained by the model’’s context window. A cap on the answer does nothing about a long prompt.
Pick it when
Reach for temperature and top-p when the complaint is that answers vary between runs or ramble off the brief, and for max tokens when the complaint is length, latency or cost.
It's the wrong answer when
No setting guarantees identical output on every run: a low temperature narrows the variation without removing it. And none of them fixes a factual gap, which is a retrieval problem or a model problem rather than a decoding one.