Why architecture matters here
Sampling deserves architectural attention because it is the highest-leverage free parameter in deployment: the same weights produce a precise coding assistant at one setting and an unusable rambler at another, and the difference costs nothing but understanding. The math explains phenomena teams otherwise attribute to the model. Why does T=1.3 hallucinate? Because flattening the distribution moves real probability mass into the enormous tail of individually-unlikely tokens — there are thousands of them, and collectively they get drawn. Why does greedy decoding repeat phrases? Because argmax is a deterministic map from context to token; once a context recurs (and generated text feeds back into context), the trajectory is a loop, and nothing stochastic breaks it. Why does top-p behave adaptively? Because when the model is confident (one token at 0.95), the nucleus is one token — sampling is effectively greedy; when it's uncertain (flat over 200 plausible continuations), the nucleus widens to admit them — the cut tracks the distribution's shape, which fixed top-k cannot.
Entropy is the unifying lens: each step's distribution has an entropy H = −Σ p_i log p_i measuring the model's genuine uncertainty. Good sampling policy respects it — harvest diversity where entropy is legitimately high (creative continuations, synonym choices) and suppress noise where it is low (syntax, arithmetic, the closing brace). Min-p is precisely this intuition as arithmetic (cutoff = min_p × max_i p_i); temperature applied globally ignores it, which is why high-T creative settings still botch code blocks embedded in prose. The most useful mental model: temperature reshapes, truncation edits support, penalties edit history — and every 'weird generation' bug is one of the three misapplied.
Operationally, sampling is also where reproducibility and evaluation variance live. Sampled outputs differ run to run; eval suites that compare single generations at T=0.8 measure noise as much as signal (hence N-run sampling and variance bands); 'deterministic' T=0 still varies across batch sizes and kernels (floating-point non-associativity changes logits in the last bits, occasionally flipping near-ties). Teams that pin sampling configs per use case, version them with prompts, and know their eval variance are debugging reality; the rest are debugging dice.
The architecture: every piece explained
Top row: the core transform. Logits arrive as a vector over the vocabulary (~32k–256k entries). Temperature divides before softmax: because softmax exponentiates, dividing logits by T multiplies log-probability ratios — at T=0.5, a token 2× more likely than another becomes 4× more likely; at T=2, the ratio compresses to √2. Limits: T→0 concentrates all mass on the argmax; T→∞ approaches uniform over the vocabulary. Softmax converts to probabilities (computed stably by subtracting max(z) first — an implementation detail that matters when you inspect raw logits). Truncation then edits the support: top-k keeps the k highest (k=1 is greedy; k=40 was the classic default); top-p keeps the smallest prefix of the sorted distribution with cumulative mass ≥ p (p=0.9 typical) — one token when confident, hundreds when not; min-p keeps tokens with p_i ≥ min_p × p_max (min_p=0.05–0.1) — a confidence-relative floor that many practitioners now prefer at higher temperatures because it prunes the tail without capping legitimate diversity.
Middle row: history and determinism. Penalties modify logits of tokens by their generation history: repetition penalty divides (or multiplies, for negative logits) seen tokens' logits by a factor (1.1–1.3 typical — and its asymmetry on negative logits is a known wart); presence penalty subtracts a flat amount from any seen token; frequency penalty subtracts proportionally to count. All three fight degenerate loops; all three, overdone, forbid necessary recurrence (variable names in code, the word 'the'). Renormalize and sample: after truncation and penalties, surviving probabilities rescale to sum to 1 and a categorical draw picks the token (seeded RNG where reproducibility matters). Greedy/beam: greedy takes argmax each step; beam search keeps B partial sequences maximizing total log-probability — still standard in translation/ASR, largely abandoned for open-ended LLM generation (it amplifies blandness and repetition). Entropy view: logging per-step entropy (or the nucleus size) is the cheapest generation diagnostic — spikes flag where the model was guessing.
Bottom rows: composition. Order matters: temperature-then-truncate differs from truncate-then-temperature (heating first admits tail tokens into the nucleus; truncating first then heating redistributes within a clean support — several stacks do the latter for stability); penalties-before-vs-after-temperature likewise shifts effective strengths. Serving stacks (HF, vLLM, llama.cpp) genuinely differ in default ordering — the reason 'same parameters, different vibes' across backends is not superstition. Task regimes: code/extraction wants T=0–0.3 with tight or no truncation (correctness over diversity); chat wants T≈0.7, top-p 0.9 or min-p 0.05 (natural but grounded); creative work runs T 0.9–1.2 with min-p carrying tail suppression; structured output under constrained decoding needs sampling that cooperates with grammar masks (masks apply before renormalization). The ops strip: configs versioned per use case, seeds for reproducible tests, N-run eval protocols, and drift checks when upgrading serving stacks (defaults change).
End-to-end flow
Walk one decode step with numbers. Context: mid-sentence in a support reply; the model's top logits (post-max-subtraction): 'refund' 0.0, 'replacement' −0.7, 'credit' −1.2, then a long tail. At T=1: softmax over the top three (relative masses e^0=1.00, e^{-0.7}=0.50, e^{-1.2}=0.30, tail Σ≈0.45 across ~500 tokens) gives p(refund)=0.44, p(replacement)=0.22, p(credit)=0.13, tail 0.21 collectively. At T=0.5 the ratios square: 1.00/0.25/0.09, tail crushed — p(refund)≈0.72; the reply gets decisive. At T=1.5 the ratios soften toward 1.00/0.63/0.45 with the tail swelling past 0.35 — one draw in three lands on something individually improbable, and that is exactly where the off-topic word arrives. Now apply top-p=0.9 at T=1: sorted cumulative mass crosses 0.9 after ~30 tokens — the bottom 470 vanish; renormalized, the plausible candidates share the mass. Min-p=0.08 instead: cutoff = 0.08 × 0.44 ≈ 0.035 — every token below 3.5% drops; had the distribution been flat (uncertain model), the cutoff would have fallen proportionally, keeping more options: the confidence-adaptive behavior in action.
The degenerate-loop case shows penalties earning their keep. Greedy decoding on a summarization prompt falls into 'the customer stated that the customer stated that…' — argmax at each step, each recurrence raising the phrase's contextual likelihood: a fixed point. Switching to T=0.7 alone usually breaks it (stochasticity escapes the loop); a frequency penalty of 0.3 makes recurrence increasingly expensive; both together are the standard chat configuration. But the code-generation channel keeps penalties near zero after a memorable bug: a repetition penalty of 1.3 taxed the repeated variable name result so hard the model started synonymizing it mid-function — result, res, output — producing plausible, broken code. The postmortem line became team folklore: penalties don't know semantics; they tax recurrence, and code is recurrence.
The eval story closes the loop. The team's regression suite once 'caught' a quality drop that was pure sampling variance — single runs at T=0.8 compared across versions. The fix: seeds pinned for smoke tests (bitwise-comparable), and quality evals run N=5 with mean±band reported; a real regression now must clear the noise floor. And when they migrated serving stacks, outputs shifted despite identical parameters — the new stack applied temperature after top-p rather than before. A day of A/B calibration (nudging top-p 0.9→0.92 and min-p in) restored the product's voice — parameters are only meaningful relative to the pipeline that applies them, which is why the config file names the stack version too.