Sampling Strategies — Greedy, Top-K, Top-P, Temperature

From hidden state to logits

At each decoding step the transformer produces a final hidden vector h of dimension d for the current position. The language-model head — a linear projection with weight matrix W_U of shape [V, d], where V is the vocabulary size — maps it to a vector of scores:

logits = W_U · h        # shape [V], one real score per token
logits_i ∈ ℜ             # unbounded: can be negative or large positive

These logits are unnormalized and have no probabilistic meaning on their own: token i having logit_i = 8.3 and token j having logit_j = 8.1 tells you only that i scored a little higher. There is no ceiling, no floor, and the values do not sum to anything in particular. To choose a token we need a probability distribution over the vocabulary — V non-negative numbers that sum to 1 — and then a rule for drawing from it. Every sampling strategy in this article is either a way to build that distribution (softmax, temperature) or a way to reshape it before drawing (top-k, top-p, min-p, argmax). Understanding the logit vector as the raw material is the whole starting point.

Softmax: logits to a distribution

The canonical map from logits to probabilities is the softmax. For a logit vector z of length V:

softmax(z)_i = exp(z_i) / Σ_j exp(z_j)     for j = 1 .. V

Exponentiating makes every term positive; dividing by the sum Σ_j exp(z_j) forces the outputs to add to 1. Two properties matter for sampling. First, softmax is monotonic: a larger logit always yields a larger probability, so the ranking of tokens is preserved. Second, it is shift-invariant — adding a constant c to every logit leaves the result unchanged, because exp(z_i + c) = exp(c) · exp(z_i) and the exp(c) cancels top and bottom. That is why implementations subtract max(z) before exponentiating: it prevents exp overflow without changing the distribution. What softmax is not is scale-invariant: multiplying all logits by a constant absolutely changes the answer — and that single fact is the entire mechanism behind temperature, which we reach shortly. Softmax is the base layer every other strategy sits on top of.

Greedy decoding: argmax

The simplest decoder skips the distribution entirely and takes the highest-scoring token every step:

next_token = argmax_i (logits_i)       # equivalently argmax of softmax

Because softmax is monotonic, the argmax of the logits and of the probabilities are the same token, so greedy decoding does not even need the softmax. It is fully deterministic: the same prompt always produces the same continuation, which is exactly what you want for reproducible extraction, classification, or code where there is a single right answer. The cost is that greedy is myopic — it commits to the locally best token with no lookahead, so it can walk into dead ends that a slightly lower-probability first step would have avoided (this is what beam search tries to fix). Worse, on open-ended generation greedy tends to degenerate: it falls into repetition loops (‘the the the’, or a sentence that repeats verbatim) because the most probable token given a repetitive context is often the token that continues the repetition. Greedy is a special case of temperature sampling in the limit T → 0, the point where the distribution collapses onto its single peak.

Temperature scaling: the transform

Temperature T > 0 rescales the logits before the softmax by dividing each one by T:

p_i(T) = exp(z_i / T) / Σ_j exp(z_j / T)

Because softmax is not scale-invariant, this genuinely reshapes the distribution rather than just relabelling it. Dividing by a T < 1 magnifies the gaps between logits (a difference of 1.0 becomes 2.0 at T = 0.5), so after softmax the leading token pulls further ahead — the distribution gets sharper / more peaked. Dividing by a T > 1 shrinks the gaps, so probabilities move toward each other and the distribution gets flatter / more uniform. The two limits are instructive: as T → 0 the largest logit dominates completely and sampling becomes deterministic argmax (greedy); as T → ∞ every scaled logit approaches 0 and the distribution approaches uniform (1/V each), i.e. pure random choice. So one scalar sweeps the whole range from ‘always the safe token’ to ‘anything goes.’ T = 1 is the identity — the model’s own calibrated distribution, untouched.

Temperature and entropy

The precise statement of ‘sharper’ and ‘flatter’ is in terms of entropy, the standard measure of a distribution’s uncertainty:

H(p) = − Σ_i p_i · log2(p_i)     # bits; 0 = certain, log2(V) = uniform

Entropy is a monotonically increasing function of temperature: raising T raises H, lowering T lowers it. The intuition is direct — flattening the distribution spreads probability mass over more tokens, and spread-out mass is exactly high entropy; sharpening concentrates mass, which is low entropy. At the extremes, T → 0 drives H → 0 (all mass on one token, zero uncertainty), while T → ∞ drives H → log2(V), the maximum for a V-way choice. In the worked example below, the same 5-token logit vector yields H = 0.88 bits at T = 0.5, H = 1.74 bits at T = 1, and H = 2.15 bits at T = 2 — climbing steadily toward the log2(5) ≈ 2.32-bit ceiling. Reading temperature as an ‘entropy dial’ is the cleanest mental model: you are literally setting how many bits of surprise you will tolerate per token, which is the same thing as setting how adventurous the model is allowed to be.

Pure sampling and the unreliable tail

Given the (temperature-adjusted) distribution, the honest thing to do is sample from it directly — draw token i with probability p_i. This is pure / ancestral sampling, and at T = 1 it faithfully reflects what the model believes. The problem is the tail. A vocabulary has tens of thousands of tokens, and even after softmax each individual unlikely token carries a tiny probability — but there are so many of them that their combined mass is far from negligible. Worse, the model’s probabilities out in that long tail are poorly calibrated: the difference between a token it means to assign 0.0001 and one it would assign 0.00001 is mostly noise. Sample often enough and you will eventually draw one of these genuinely-wrong tokens, and a single incoherent token can derail an entire generation because the model must then condition on its own mistake. This is the failure that motivates every truncation strategy that follows: keep the trustworthy head of the distribution, throw away the unreliable tail, renormalize what remains, and sample only from that. Top-k, top-p, and min-p are three different answers to ‘where does the trustworthy head end?’

Top-k truncation

Top-k keeps the k highest-probability tokens, zeros out everything else, and renormalizes over the survivors:

S = indices of the k largest p_i        # the kept set
p’_i = p_i / Σ_{j ∈ S} p_j   if i ∈ S,   else 0

The renormalization — dividing by the surviving mass Σ_{j∈S} p_j — is what makes p’ a valid distribution again (it sums to 1). Top-k is cheap: a partial sort or a top-k selection, done in O(V) to O(V log k). Its weakness is that k is a fixed count that ignores the shape of the distribution. When the model is very confident — one token at 0.95 — a k of 40 drags in 39 tokens the model considered nearly impossible, needlessly injecting noise. When the model is genuinely torn across many plausible tokens, that same k = 40 may chop off tokens that deserved to stay. Top-k applies the same width to a needle-sharp peak and a broad plateau, which is exactly the mismatch top-p was designed to remove. It remains a reasonable, predictable default, but it treats ‘how many good options are there’ as a constant when it is really a property of each step.

Top-p / nucleus sampling

Top-p (nucleus sampling) fixes top-k’s rigidity by keeping a dynamic number of tokens — however many it takes to cover a target probability mass p. Sort tokens by probability descending, walk down accumulating mass, and stop as soon as the cumulative sum reaches p:

sort so p_(1) ≥ p_(2) ≥ ...
find smallest n with  Σ_{i=1..n} p_(i) ≥ p     # e.g. p = 0.9
keep tokens 1..n (the ‘nucleus’), renormalize, sample

The size of the kept set now adapts to confidence. When one token already holds 0.92 of the mass, the nucleus at p = 0.9 is just that single token — the method behaves like greedy exactly when the model is sure. When the model is spread thin across twenty plausible continuations, the nucleus grows to include all of them, preserving diversity precisely when diversity is warranted. That is the elegant core idea: instead of choosing ‘how many tokens,’ you choose ‘how much of the probability mass to trust,’ and let the distribution decide the count. p = 0.9 to 0.95 is the workhorse setting for open-ended text. Its one blind spot: the cumulative rule can still admit a long run of individually-tiny tokens if the head is not dominant — the gap min-p targets next.

Min-p sampling

Min-p thresholds by relative probability rather than by count or cumulative mass. It sets a floor as a fraction of the top token’s probability and keeps every token above it:

p_max = max_i p_i
threshold = p_min · p_max         # e.g. p_min = 0.1
keep token i  iff  p_i ≥ threshold, then renormalize

Because the threshold is scaled by the peak, min-p is naturally adaptive to the distribution’s shape. If the model is confident (p_max = 0.9), the bar is a stiff 0.09 and only the strongest few tokens survive — aggressive pruning exactly when the tail is untrustworthy. If the model is uncertain (p_max = 0.15), the bar drops to 0.015 and many tokens are admitted — wide exploration exactly when many options are genuinely viable. This is arguably the ‘right’ adaptivity: it responds to how peaked the distribution is, not merely to a cumulative total. In practice min-p holds up much better than top-p at high temperatures, where flattening the distribution would otherwise let top-p’s cumulative window scoop up a lot of junk; min-p’s peak-relative floor keeps pruning hard on the outright-implausible tokens. A p_min of 0.05 to 0.1 is a common, robust choice.

A worked example: the logit vector

To make the transforms concrete, take five tokens A, B, C, D, E with logits and their plain T = 1 softmax:

logits z = [ A: 2.0,  B: 1.0,  C: 0.5,  D: 0.0,  E: -1.0 ]

exp(z)   = [ 7.389, 2.718, 1.649, 1.000, 0.368 ]   sum = 13.124

softmax  = [ A: 0.563, B: 0.207, C: 0.126, D: 0.076, E: 0.028 ]
                                                    (sums to 1.000)

This is the untouched distribution the model actually produced: A is the clear favourite at 0.563, but it is far from certain — more than 40% of the mass lives on the other four tokens. Greedy decoding would end the story here and always emit A. Every other method below starts from this same five-number vector and reshapes it. Keep the numbers in view: the leader A at 0.563, the mid-pack B and C, and the weak tail D and especially E at 0.028. The entropy of this distribution is H = 1.74 bits — comfortably below the log2(5) = 2.32-bit maximum, reflecting that A carries real weight. Watching how E and D get amplified or deleted, and how A’s lead grows or shrinks, is the whole point of the exercise.

Worked: temperature reshaping the distribution

Apply temperature by dividing the logits by T and re-softmaxing. The same vector at three temperatures:

          A       B       C       D       E      entropy
T=0.5   0.829   0.112   0.041   0.015   0.002   0.88 bits   (sharper)
T=1.0   0.563   0.207   0.126   0.076   0.028   1.74 bits   (as-is)
T=2.0   0.375   0.227   0.177   0.138   0.084   2.15 bits   (flatter)

Read across the rows and the mechanism is plain. At T = 0.5 the leader A swells from 0.563 to 0.829 while the tail token E is crushed from 0.028 to 0.002 — the distribution concentrates, entropy falls to 0.88 bits, output gets safer and more repetitive. At T = 2.0 the opposite: A gives up its lead down to 0.375, the neglected tokens D and E roughly double and triple their mass, entropy rises to 2.15 bits, output gets more varied and riskier. Note the ranking never changes — A > B > C > D > E throughout, because temperature is a monotonic reshaping. It moves probability mass around; it never reorders who is ahead. That is why temperature alone cannot fix a bad top choice; it only adjusts how often the alternatives get a turn.

Worked: top-k, top-p, and min-p on the same vector

Now the truncations, all starting from the T = 1 row [0.563, 0.207, 0.126, 0.076, 0.028]:

top-k, k=3   keep A,B,C (drop D,E); renormalize over 0.896
             → A: 0.629,  B: 0.231,  C: 0.140

top-p, p=0.9 cumulative: A .563, +B .770, +C .896, +D .972 ≥ 0.9
             keep A,B,C,D (drop E); renormalize over 0.972
             → A: 0.579,  B: 0.213,  C: 0.129,  D: 0.078

min-p, 0.1   threshold = 0.1 × 0.563 = 0.0563
             keep p_i ≥ 0.0563: A,B,C,D (E=0.028 dropped)
             → A: 0.579,  B: 0.213,  C: 0.129,  D: 0.078

Three methods, three ways of drawing the line. Top-k took a fixed count of 3, cutting both D and E. Top-p at 0.9 walked the cumulative sum until it crossed 0.9 — that happened only after adding D (the running total at C was 0.896, just short), so it kept four tokens. Min-p at 0.1 set an absolute floor of 0.0563 from the peak and also kept four, deleting only the genuinely-implausible E. On this gentle distribution top-p and min-p happen to agree, but their logic differs, and under temperature they diverge sharply — which is the next point.

How the transforms compose: temperature then top-p

Real decoders stack these knobs, and order matters because each operates on the distribution the previous one produced. The standard pipeline is: temperature first (reshape the full distribution), then truncation (top-k / top-p / min-p on the reshaped distribution), then renormalize and draw. Watch what temperature does to top-p’s reach. Take T = 0.7 on our vector, then top-p = 0.9:

T=0.7 softmax: A .700, B .168, C .082, D .040, E .010
top-p 0.9 cumulative: A .700, +B .868, +C .950 ≥ 0.9  → keep A,B,C
renormalize over 0.950  → A: 0.737,  B: 0.177,  C: 0.086

At T = 1 the same p = 0.9 kept four tokens; after sharpening to T = 0.7 it keeps only three, because the peakier distribution reaches 0.9 cumulative mass sooner. Temperature and top-p are not independent dials — lowering temperature quietly narrows the nucleus too. Reverse the order (truncate then apply temperature) and you get different numbers and, at high temperature, a genuinely worse result, since flattening after truncation re-inflates the tokens you just kept without being able to reconsider the ones you cut. Knowing the composition order is what lets you reason about a sampler config instead of tuning it blind.

Diversity versus coherence: the core trade-off

Every strategy in this article is a point on one axis: how much of the distribution’s diversity you keep versus how much coherence (safety, on-topic-ness, determinism) you buy by throwing diversity away. Greedy sits at the coherent extreme — maximally safe, maximally repetitive. High-temperature pure sampling sits at the diverse extreme — maximally creative, maximally prone to nonsense. The knobs move you along the axis in different ways: temperature rescales how much mass the tail gets, while top-k / top-p / min-p decide which tail tokens are even eligible. The reason the two families are usually combined is that they cover different failure modes. Temperature alone at a high value keeps the whole unreliable tail in play; truncation alone gives no control over the relative odds of the tokens that remain. Together — a moderate temperature to set the adventurousness, plus a truncation to fence off the junk — you get varied output that stays inside the set of tokens the model actually endorses. There is no globally best point on the axis; the right setting is a property of the task, which is the final section.

Practical defaults and CPU-SLM implications

Match the knob to the job. Deterministic tasks — extraction, classification, structured output, math, code with one right answer — want greedy or a near-zero temperature: you are not after variety, you are after the single most probable, reproducible answer. Open-ended generation — chat, brainstorming, prose — wants T ≈ 0.7–1.0 plus top-p ≈ 0.9–0.95 (or min-p ≈ 0.05–0.1), the combination that keeps output lively but fenced. For small language models on CPU the cost story is friendly: sampling operates on the final [V] logit vector once per token, so its arithmetic is trivial next to the matmuls of the forward pass — softmax, a sort or partial-select for truncation, one draw, all O(V) to O(V log k). Even a large vocabulary (V ≈ 32k–128k) is a millisecond-scale operation, negligible against decode’s memory-bound weight loading. The practical lever a small model gives you is that sampling quality partly compensates for capacity: a well-tuned sampler keeps a small model coherent where high-temperature pure sampling would expose its weaker calibration. Tune the sampler deliberately — it is nearly free and it materially changes the output.

Common pitfalls

A handful of mistakes recur. Temperature after truncation: apply temperature before top-k/top-p, not after — reversing the order changes which tokens survive and, at high temperature, re-inflates the tail you meant to cut. Stacking two truncations: running top-k and top-p and min-p together compounds their cuts and often leaves a degenerate, near-greedy set; pick one truncation and tune it rather than layering three. Temperature at exactly 0: z / T divides by zero — runtimes special-case T = 0 to mean greedy, but do not feed a literal zero into the softmax. Forgetting to renormalize: after zeroing tokens you must divide by the surviving mass, or you are sampling from something that no longer sums to 1. Assuming temperature reorders: it never does — if the top token is wrong, no temperature fixes it; you need a different truncation or a better prompt. Over-trusting the tail: the reason truncation exists is that the model’s far-tail probabilities are noise; pure sampling at T = 1 with no truncation is the classic recipe for occasional incoherent tokens that derail an otherwise good generation.