Sampling: temperature, top-k, top-p

Language modelsabout 9 minutes

Given the next-token distribution, which token do we pick? The answer, called the decoding strategy, changes the character of generated text as much as the model does.

Greedy decoding always takes the most likely token. It's deterministic and often good for short factual answers, but on long text it tends to fall into repetitive loops.
Sampling draws a token at random according to the probabilities. It's varied and natural, which is why asking a chat model the same question twice gives different answers. Pure sampling occasionally picks a very unlikely token and derails.

Temperature

Divide the logits by a temperature $T$ before the softmax. $T < 1$ sharpens the distribution toward the top token; $T > 1$ flattens it toward uniform. As $T \to 0$ sampling becomes greedy.

p_{i} = \frac{e ^{z_{i} / T}}{\sum _{j} e ^{z_{j} / T}}

Next-token candidates after "The cat sat on the". Change the temperature and the top-p cutoff and watch which tokens remain possible.

Truncation: top-k and top-p

The long tail of unlikely tokens is where nonsense comes from. Top-k sampling keeps only the $k$ most likely tokens and renormalises. Top-p or nucleus sampling (Holtzman et al., 2019) keeps the smallest set of top tokens whose probabilities add up to at least $p$ , say 0.9. Top-p adapts: when the model is confident, the nucleus is a handful of tokens; when many continuations are plausible, it widens.

Typical settings: temperature around 0.7–1.0 with top-p 0.9–0.95 for open-ended writing, low temperature for code and factual answers. None of this changes what the model knows; it changes which of its guesses you see.

try itedit the code and run it, nothing can break

python sleeps until you press run

Decoding is a choice. Temperature reshapes the distribution; top-k and top-p cut off its unreliable tail before sampling.

← Next-token prediction