From hidden state to logits
At each decoding step the transformer produces a final hidden vector h of dimension d for the current position. The language-model head — a linear projection with weight matrix W_U of shape [V, d], where V is the vocabulary size — maps it to a vector of scores:
logits = W_U · h # shape [V], one real score per token
logits_i ∈ ℜ # unbounded: can be negative or large positiveThese logits are unnormalized and have no probabilistic meaning on their own: token i having logit_i = 8.3 and token j having logit_j = 8.1 tells you only that i scored a little higher. There is no ceiling, no floor, and the values do not sum to anything in particular. To choose a token we need a probability distribution over the vocabulary — V non-negative numbers that sum to 1 — and then a rule for drawing from it. Every sampling strategy in this article is either a way to build that distribution (softmax, temperature) or a way to reshape it before drawing (top-k, top-p, min-p, argmax). Understanding the logit vector as the raw material is the whole starting point.
Softmax: logits to a distribution
The canonical map from logits to probabilities is the softmax. For a logit vector z of length V:
softmax(z)_i = exp(z_i) / Σ_j exp(z_j) for j = 1 .. VExponentiating makes every term positive; dividing by the sum Σ_j exp(z_j) forces the outputs to add to 1. Two properties matter for sampling. First, softmax is monotonic: a larger logit always yields a larger probability, so the ranking of tokens is preserved. Second, it is shift-invariant — adding a constant c to every logit leaves the result unchanged, because exp(z_i + c) = exp(c) · exp(z_i) and the exp(c) cancels top and bottom. That is why implementations subtract max(z) before exponentiating: it prevents exp overflow without changing the distribution. What softmax is not is scale-invariant: multiplying all logits by a constant absolutely changes the answer — and that single fact is the entire mechanism behind temperature, which we reach shortly. Softmax is the base layer every other strategy sits on top of.
Greedy decoding: argmax
The simplest decoder skips the distribution entirely and takes the highest-scoring token every step:
next_token = argmax_i (logits_i) # equivalently argmax of softmaxBecause softmax is monotonic, the argmax of the logits and of the probabilities are the same token, so greedy decoding does not even need the softmax. It is fully deterministic: the same prompt always produces the same continuation, which is exactly what you want for reproducible extraction, classification, or code where there is a single right answer. The cost is that greedy is myopic — it commits to the locally best token with no lookahead, so it can walk into dead ends that a slightly lower-probability first step would have avoided (this is what beam search tries to fix). Worse, on open-ended generation greedy tends to degenerate: it falls into repetition loops (‘the the the’, or a sentence that repeats verbatim) because the most probable token given a repetitive context is often the token that continues the repetition. Greedy is a special case of temperature sampling in the limit T → 0, the point where the distribution collapses onto its single peak.
Temperature scaling: the transform
Temperature T > 0 rescales the logits before the softmax by dividing each one by T:
p_i(T) = exp(z_i / T) / Σ_j exp(z_j / T)Because softmax is not scale-invariant, this genuinely reshapes the distribution rather than just relabelling it. Dividing by a T < 1 magnifies the gaps between logits (a difference of 1.0 becomes 2.0 at T = 0.5), so after softmax the leading token pulls further ahead — the distribution gets sharper / more peaked. Dividing by a T > 1 shrinks the gaps, so probabilities move toward each other and the distribution gets flatter / more uniform. The two limits are instructive: as T → 0 the largest logit dominates completely and sampling becomes deterministic argmax (greedy); as T → ∞ every scaled logit approaches 0 and the distribution approaches uniform (1/V each), i.e. pure random choice. So one scalar sweeps the whole range from ‘always the safe token’ to ‘anything goes.’ T = 1 is the identity — the model’s own calibrated distribution, untouched.
Temperature and entropy
The precise statement of ‘sharper’ and ‘flatter’ is in terms of entropy, the standard measure of a distribution’s uncertainty:
H(p) = − Σ_i p_i · log2(p_i) # bits; 0 = certain, log2(V) = uniformEntropy is a monotonically increasing function of temperature: raising T raises H, lowering T lowers it. The intuition is direct — flattening the distribution spreads probability mass over more tokens, and spread-out mass is exactly high entropy; sharpening concentrates mass, which is low entropy. At the extremes, T → 0 drives H → 0 (all mass on one token, zero uncertainty), while T → ∞ drives H → log2(V), the maximum for a V-way choice. In the worked example below, the same 5-token logit vector yields H = 0.88 bits at T = 0.5, H = 1.74 bits at T = 1, and H = 2.15 bits at T = 2 — climbing steadily toward the log2(5) ≈ 2.32-bit ceiling. Reading temperature as an ‘entropy dial’ is the cleanest mental model: you are literally setting how many bits of surprise you will tolerate per token, which is the same thing as setting how adventurous the model is allowed to be.
Pure sampling and the unreliable tail
Given the (temperature-adjusted) distribution, the honest thing to do is sample from it directly — draw token i with probability p_i. This is pure / ancestral sampling, and at T = 1 it faithfully reflects what the model believes. The problem is the tail. A vocabulary has tens of thousands of tokens, and even after softmax each individual unlikely token carries a tiny probability — but there are so many of them that their combined mass is far from negligible. Worse, the model’s probabilities out in that long tail are poorly calibrated: the difference between a token it means to assign 0.0001 and one it would assign 0.00001 is mostly noise. Sample often enough and you will eventually draw one of these genuinely-wrong tokens, and a single incoherent token can derail an entire generation because the model must then condition on its own mistake. This is the failure that motivates every truncation strategy that follows: keep the trustworthy head of the distribution, throw away the unreliable tail, renormalize what remains, and sample only from that. Top-k, top-p, and min-p are three different answers to ‘where does the trustworthy head end?’
Top-k truncation
Top-k keeps the k highest-probability tokens, zeros out everything else, and renormalizes over the survivors:
S = indices of the k largest p_i # the kept set
p’_i = p_i / Σ_{j ∈ S} p_j if i ∈ S, else 0The renormalization — dividing by the surviving mass Σ_{j∈S} p_j — is what makes p’ a valid distribution again (it sums to 1). Top-k is cheap: a partial sort or a top-k selection, done in O(V) to O(V log k). Its weakness is that k is a fixed count that ignores the shape of the distribution. When the model is very confident — one token at 0.95 — a k of 40 drags in 39 tokens the model considered nearly impossible, needlessly injecting noise. When the model is genuinely torn across many plausible tokens, that same k = 40 may chop off tokens that deserved to stay. Top-k applies the same width to a needle-sharp peak and a broad plateau, which is exactly the mismatch top-p was designed to remove. It remains a reasonable, predictable default, but it treats ‘how many good options are there’ as a constant when it is really a property of each step.
Top-p / nucleus sampling
Top-p (nucleus sampling) fixes top-k’s rigidity by keeping a dynamic number of tokens — however many it takes to cover a target probability mass p. Sort tokens by probability descending, walk down accumulating mass, and stop as soon as the cumulative sum reaches p:
sort so p_(1) ≥ p_(2) ≥ ...
find smallest n with Σ_{i=1..n} p_(i) ≥ p # e.g. p = 0.9
keep tokens 1..n (the ‘nucleus’), renormalize, sampleThe size of the kept set now adapts to confidence. When one token already holds 0.92 of the mass, the nucleus at p = 0.9 is just that single token — the method behaves like greedy exactly when the model is sure. When the model is spread thin across twenty plausible continuations, the nucleus grows to include all of them, preserving diversity precisely when diversity is warranted. That is the elegant core idea: instead of choosing ‘how many tokens,’ you choose ‘how much of the probability mass to trust,’ and let the distribution decide the count. p = 0.9 to 0.95 is the workhorse setting for open-ended text. Its one blind spot: the cumulative rule can still admit a long run of individually-tiny tokens if the head is not dominant — the gap min-p targets next.
Min-p sampling
Min-p thresholds by relative probability rather than by count or cumulative mass. It sets a floor as a fraction of the top token’s probability and keeps every token above it:
p_max = max_i p_i
threshold = p_min · p_max # e.g. p_min = 0.1
keep token i iff p_i ≥ threshold, then renormalizeBecause the threshold is scaled by the peak, min-p is naturally adaptive to the distribution’s shape. If the model is confident (p_max = 0.9), the bar is a stiff 0.09 and only the strongest few tokens survive — aggressive pruning exactly when the tail is untrustworthy. If the model is uncertain (p_max = 0.15), the bar drops to 0.015 and many tokens are admitted — wide exploration exactly when many options are genuinely viable. This is arguably the ‘right’ adaptivity: it responds to how peaked the distribution is, not merely to a cumulative total. In practice min-p holds up much better than top-p at high temperatures, where flattening the distribution would otherwise let top-p’s cumulative window scoop up a lot of junk; min-p’s peak-relative floor keeps pruning hard on the outright-implausible tokens. A p_min of 0.05 to 0.1 is a common, robust choice.
A worked example: the logit vector
To make the transforms concrete, take five tokens A, B, C, D, E with logits and their plain T = 1 softmax:
logits z = [ A: 2.0, B: 1.0, C: 0.5, D: 0.0, E: -1.0 ]
exp(z) = [ 7.389, 2.718, 1.649, 1.000, 0.368 ] sum = 13.124
softmax = [ A: 0.563, B: 0.207, C: 0.126, D: 0.076, E: 0.028 ]
(sums to 1.000)This is the untouched distribution the model actually produced: A is the clear favourite at 0.563, but it is far from certain — more than 40% of the mass lives on the other four tokens. Greedy decoding would end the story here and always emit A. Every other method below starts from this same five-number vector and reshapes it. Keep the numbers in view: the leader A at 0.563, the mid-pack B and C, and the weak tail D and especially E at 0.028. The entropy of this distribution is H = 1.74 bits — comfortably below the log2(5) = 2.32-bit maximum, reflecting that A carries real weight. Watching how E and D get amplified or deleted, and how A’s lead grows or shrinks, is the whole point of the exercise.
Worked: temperature reshaping the distribution
Apply temperature by dividing the logits by T and re-softmaxing. The same vector at three temperatures:
A B C D E entropy
T=0.5 0.829 0.112 0.041 0.015 0.002 0.88 bits (sharper)
T=1.0 0.563 0.207 0.126 0.076 0.028 1.74 bits (as-is)
T=2.0 0.375 0.227 0.177 0.138 0.084 2.15 bits (flatter)Read across the rows and the mechanism is plain. At T = 0.5 the leader A swells from 0.563 to 0.829 while the tail token E is crushed from 0.028 to 0.002 — the distribution concentrates, entropy falls to 0.88 bits, output gets safer and more repetitive. At T = 2.0 the opposite: A gives up its lead down to 0.375, the neglected tokens D and E roughly double and triple their mass, entropy rises to 2.15 bits, output gets more varied and riskier. Note the ranking never changes — A > B > C > D > E throughout, because temperature is a monotonic reshaping. It moves probability mass around; it never reorders who is ahead. That is why temperature alone cannot fix a bad top choice; it only adjusts how often the alternatives get a turn.
Worked: top-k, top-p, and min-p on the same vector
Now the truncations, all starting from the T = 1 row [0.563, 0.207, 0.126, 0.076, 0.028]:
top-k, k=3 keep A,B,C (drop D,E); renormalize over 0.896
→ A: 0.629, B: 0.231, C: 0.140
top-p, p=0.9 cumulative: A .563, +B .770, +C .896, +D .972 ≥ 0.9
keep A,B,C,D (drop E); renormalize over 0.972
→ A: 0.579, B: 0.213, C: 0.129, D: 0.078
min-p, 0.1 threshold = 0.1 × 0.563 = 0.0563
keep p_i ≥ 0.0563: A,B,C,D (E=0.028 dropped)
→ A: 0.579, B: 0.213, C: 0.129, D: 0.078Three methods, three ways of drawing the line. Top-k took a fixed count of 3, cutting both D and E. Top-p at 0.9 walked the cumulative sum until it crossed 0.9 — that happened only after adding D (the running total at C was 0.896, just short), so it kept four tokens. Min-p at 0.1 set an absolute floor of 0.0563 from the peak and also kept four, deleting only the genuinely-implausible E. On this gentle distribution top-p and min-p happen to agree, but their logic differs, and under temperature they diverge sharply — which is the next point.
How the transforms compose: temperature then top-p
Real decoders stack these knobs, and order matters because each operates on the distribution the previous one produced. The standard pipeline is: temperature first (reshape the full distribution), then truncation (top-k / top-p / min-p on the reshaped distribution), then renormalize and draw. Watch what temperature does to top-p’s reach. Take T = 0.7 on our vector, then top-p = 0.9:
T=0.7 softmax: A .700, B .168, C .082, D .040, E .010
top-p 0.9 cumulative: A .700, +B .868, +C .950 ≥ 0.9 → keep A,B,C
renormalize over 0.950 → A: 0.737, B: 0.177, C: 0.086At T = 1 the same p = 0.9 kept four tokens; after sharpening to T = 0.7 it keeps only three, because the peakier distribution reaches 0.9 cumulative mass sooner. Temperature and top-p are not independent dials — lowering temperature quietly narrows the nucleus too. Reverse the order (truncate then apply temperature) and you get different numbers and, at high temperature, a genuinely worse result, since flattening after truncation re-inflates the tokens you just kept without being able to reconsider the ones you cut. Knowing the composition order is what lets you reason about a sampler config instead of tuning it blind.