Why architecture matters here
Decoding matters because it is the cheapest large lever in the stack. The forward pass costs billions of FLOPs; the sampling pipeline costs a sort and a draw over a single vector. Yet the sampling parameters change output quality about as much as a meaningful jump in model size does — in either direction. A well-tuned 8B model at sensible settings beats a 70B model at temperature 1.5 with no truncation, which will produce fluent nonsense. Nobody would deploy the 70B without benchmarking it. Everybody deploys sampling parameters copied from a README.
The second reason is that the failure modes are specific, diagnosable, and mapped to individual knobs — which makes decoding one of the few parts of an LLM system that rewards mechanical reasoning rather than vibes. Degenerate repetition, where the model locks into a phrase and emits it until the token limit, is a known consequence of greedy or near-greedy decoding on open-ended text: the model's most likely continuation of a repeated phrase is more of it, and without stochasticity there is no escape. Conversely, incoherent drift — plausible sentences that wander away from the question — is the signature of a fat tail: temperature too high or truncation too loose, so the 3,000th-most-likely token gets picked once every few hundred steps and drags the context somewhere new. These are not mysteries. Each has a mechanism and a knob.
Third, and most practically: different tasks want genuinely different distributions, and there is no universal setting. Extracting a JSON field, translating a sentence, and writing a story are not the same problem. The extraction task wants the argmax and nothing else — every alternative token is a bug, so temperature 0. The story wants a live tail, because the tail is where anything surprising lives. Serving all three from one endpoint with one temperature guarantees you are wrong for at least two of them. Sampling parameters belong to the request, not the deployment — and treating them as a global config value is a design error that no amount of prompt engineering compensates for.
The architecture: every piece explained
Logits and masking. The final hidden state times the lm_head matrix yields a vector of raw, unnormalized scores over the vocabulary. Before anything else, hard constraints apply by setting entries to negative infinity: banned tokens, suppressed EOS if a minimum length is required, and — for guided decoding — every token a grammar or JSON schema forbids at this position. Masking is the strongest tool in the pipeline because it is absolute: a masked token has zero probability regardless of every knob that follows. This is why structured output is enforced by masking rather than requested by prompting.
Penalties. Three related mechanisms discourage reuse of tokens already present. Repetition penalty divides (or multiplies, for negatives) the logit of any token seen so far by a factor — typically 1.05 to 1.2. Frequency penalty subtracts an amount proportional to how many times a token has appeared; presence penalty subtracts a flat amount if it has appeared at all. The distinction matters: frequency penalty scales with repetition and so fights loops, while presence penalty pushes toward new topics after a single use. All three carry a sharp edge — they are typically applied over the prompt as well as the completion, so a document-grounded task can find the model penalized for reusing the very terms it is supposed to quote.
Temperature. Divide every logit by T before softmax. T < 1 sharpens the distribution toward the argmax; T > 1 flattens it toward uniform; T = 1 leaves the model's own distribution untouched. T = 0 is not a temperature at all — it is a special case implementations short-circuit to greedy argmax, since the limit is undefined by division. Temperature is a rescaling: it changes relative odds everywhere and removes nothing, which is exactly why it is insufficient on its own. Even at low temperature the tail retains some mass, and over thousands of steps 'some mass' becomes 'happens regularly'.
Truncation: top-k, top-p, min-p. These delete the tail rather than reweighting it. Top-k keeps the k highest-scoring tokens and discards the rest — simple, but blind to shape: k=50 is far too permissive where the model is confident and possibly too strict where it is genuinely uncertain. Top-p (nucleus) sorts by probability and keeps the smallest set whose cumulative mass reaches p, so the surviving set is small when the model is confident and large when it is not — adaptive, and the reason it largely displaced top-k. Min-p takes a third angle: keep tokens whose probability is at least a fraction of the top token's, which is cheaper than a full sort and behaves more predictably at high temperature. After truncation, softmax renormalizes over the survivors so the kept mass sums to one.
Batched sampling on the GPU. In production every request in a batch carries its own parameters, so the sampler is a batched kernel applying different temperatures, different k, different p per row. Top-p's sort over a 128k-wide vocabulary is the expensive step — a real cost at high batch sizes, and the practical reason min-p and fused sampling kernels get attention. Each request also needs its own RNG stream for per-request seeding and reproducibility. Two hooks share this vector: guided decoding masks it before sampling, and speculative decoding's rejection sampling needs the target model's true distribution — which is why speculation is only lossless when the verification step reconstructs exactly the distribution the sampling parameters describe.
End-to-end flow
Walk one decoding step concretely. The prompt is a support question, the model has emitted 'The refund will be processed within ', and the request specifies temperature 0.7, top-p 0.9, frequency penalty 0.3, seed 42. The forward pass completes and hands the sampler a vector of 128,256 logits.
Mask. No grammar is active and minimum length is satisfied, so nothing is set to negative infinity. Had this been a JSON-constrained request, every token that could not legally follow at this position would be eliminated here — and no later stage could resurrect it.
Penalize. Frequency penalty 0.3 walks the tokens already in the context and subtracts proportionally to their counts. 'refund' has appeared four times, so its logit drops meaningfully; 'the' has appeared eleven times and takes a large hit. Note what just happened: the penalty was applied over the prompt tokens too, so the model is now mildly discouraged from reusing the customer's own vocabulary. For this task that is a small tax on a fluent answer; for document-grounded extraction it would be an outright bug.
Scale. Every logit is divided by 0.7. This sharpens the distribution — the gap between '5' and 'three' widens in odds terms. Nothing is removed; the 90,000th token still has a positive, tiny probability. If temperature were the only mechanism, that tiny probability would fire eventually, and 'The refund will be processed within pomegranate' would appear once every few thousand completions. This is precisely the gap truncation exists to close.
Truncate. Softmax over the scaled logits gives probabilities; the sampler sorts descending and accumulates until the mass reaches 0.9. Here the model is confident — after 'processed within', the plausible continuations are a small cluster of numerals and number-words. '5' holds 0.41, '3' 0.22, 'the' 0.11, 'seven' 0.09, 'a' 0.05: five tokens reach 0.88, the sixth crosses 0.9, and the set closes at six. The remaining 128,250 tokens are discarded outright — not made unlikely, deleted. This is the adaptive property that makes top-p work: at a genuinely open-ended position, the same p=0.9 would have admitted hundreds of tokens instead of six, because the mass would be spread thin. One parameter, two behaviours, driven by the model's own confidence.
Renormalize and draw. The six survivors are renormalized to sum to one, and the request's seeded RNG — stream 42, at this step's offset — draws once from the multinomial. It returns '5'. The token is appended to the sequence, its key and value vectors are written into the KV cache, and the next forward pass begins. The whole pipeline cost a sort and a draw over one vector, against billions of FLOPs for the forward pass that produced it.
Two properties of this trace are worth making explicit. First, determinism is conditional. Same seed, same parameters, same prompt gives the same token — but only if the logits are bit-identical, and in a batched server they are not. Batch composition changes the shapes of the GEMM kernels, which changes floating-point reduction order, which perturbs the logits in the last bits. Usually irrelevant; occasionally it flips a near-tie between two survivors and the completion diverges. Reproducibility in production is therefore statistical, not exact, and a seed is not a guarantee. Second, every stage depended on the ones before it. The penalty reshaped the logits temperature then scaled; the scaling determined the probabilities top-p accumulated over; top-p's cutoff decided what could be drawn at all. Reorder any two and the same nominal settings produce a different distribution — which is why the order is a contract, and why comparing 'temperature 0.7, top-p 0.9' across two engines that order these stages differently is not comparing the same thing.