Sampling Settings That Fix Real Failures: Temperature, Top-p and Min-p

Most production LLM failures that get blamed on "the model" are at least partly decoding failures. A summariser that repeats the same sentence until it hits the token limit, a brainstorming feature that occasionally emits a fragment of another language, an extraction endpoint that produces invalid JSON once in a few hundred calls, or a product-copy generator whose ten variants are nearly identical: each of these is shaped by how the next token is chosen from the model's distribution, and each has a sampling setting that makes it better or worse.

This guide starts from the softmax and works through what temperature, top-k, top-p (nucleus), min-p and repetition penalties actually do to the distribution, with the arithmetic of why a long tail of individually unlikely tokens matters. It then maps specific failures to specific settings, shows where each fix backfires, and lists the knobs and defaults to check in vLLM, Hugging Face Transformers and llama.cpp. The lab samples from a synthetic next-token model with the same filters, so you can break loops and cut junk while watching what happens to variety.

From logits to a choice

At every step the model produces a logit z_i for each token in its vocabulary, often more than 100,000 of them. Sampling turns logits into probabilities with a softmax and draws one token. Temperature T divides the logits first:

p_i = exp(z_i / T) / sum_j exp(z_j / T)

T -> 0 : all mass on the argmax (greedy decoding)
T = 1  : the model's own distribution
T > 1  : flatter; unlikely tokens gain probability

The detail that matters in production is the tail. Most of the vocabulary is irrelevant at any given step: tokens from other languages, code fragments in the middle of prose, partial words. Each has a tiny probability, but there are tens of thousands of them. Their combined mass is what produces the occasional nonsense token, and temperature changes it exponentially.

A worked example with the lab's numbers: an open-ended step has six plausible tokens with logits near 2 and seventy junk tokens with logits near −6.4. At T = 1 the junk share is roughly 70 × e^(−6.4) / (6 × e^2 + 70 × e^(−6.4)) ≈ 0.26% per step. At T = 1.6 the logits become 1.25 and −4, and the junk share rises to about 5.8%. If half of a 50-token answer consists of open-ended steps, the chance an answer contains no junk at all is about 0.9974^25 ≈ 94% at T = 1 and 0.942^25 ≈ 22% at T = 1.6. A real vocabulary has far more than seventy irrelevant tokens, so the effect in real models can be larger. That is why "just raise the temperature for creativity" so often ends in a support ticket.

Truncation filters: top-k, top-p and min-p

Truncation removes the tail before sampling and renormalises what is left. The three common filters differ in how they decide where the tail starts.

Filter	Keeps	Behaviour on a peaked step	Behaviour on a flat step
Top-k	The k most likely tokens	Keeps k tokens even if one has 99% of the mass	Cuts valid options if more than k are plausible
Top-p (nucleus)	The smallest set whose cumulative probability reaches p	Often keeps just one or two tokens	Keeps many tokens; if the tail is heavy, some of it
Min-p	Tokens with probability at least min_p × p_max	High threshold: keeps few tokens	Low threshold: keeps all comparable options, still excludes tokens far below the top

Top-k came from story-generation work and is crude because the right number of candidates varies wildly from step to step. Nucleus sampling was introduced in the paper that named the problem of neural text degeneration, which showed that greedy and beam search produce bland, repetitive text while pure sampling from the full distribution produces incoherent text, and proposed truncating to the top-p mass as a middle path. Min-p, proposed more recently, sets the cut relative to the most likely token, so the filter tightens automatically when the model is confident and loosens when it is not; its authors report that it keeps outputs coherent at higher temperatures than top-p does.

In practice both top-p and min-p are reasonable tail filters. The difference shows when the temperature is high or the distribution is unusually flat: top-p must keep adding tokens until it reaches its cumulative target, and when the plausible tokens together hold less than p of the mass, the remainder comes from the tail. Min-p never admits a token that is, say, twenty times less likely than the best one, however flat the distribution. Mission two in the lab asks you to keep temperature at 1.5 or above for variety and still produce 95% clean samples; either filter can get there, and you can watch the kept set in the probability chart.

Repetition loops and why greedy decoding falls into them

Language models are good at copying. Once a phrase has appeared in the context, attention mechanisms make the token that followed it last time more likely to follow it again. Research on this self-reinforcement effect found that the probability of repeating a sentence tends to increase with the number of times it has already been repeated. Under greedy decoding the result is deterministic: once the copied token is the argmax, it stays the argmax, and the output loops until it hits max_tokens. Smaller and heavily quantized models, long outputs and repetitive inputs (logs, lists, tables) make it worse.

The lab models this explicitly. Whatever token followed the previous token the last time it appeared receives a boost that puts it slightly above the current best token, growing with each consecutive copy. Greedy decoding on long outputs loops in every sample. Sampling helps because each step has some chance of escaping, but the escape probability falls as the loop strengthens, so on long outputs moderate temperature alone still leaves a fraction of samples stuck.

Penalties attack the loop directly:

Frequency penalty subtracts a constant times the number of times a token has already appeared from its logit. It grows with repetition, which is what a loop needs.
Presence penalty subtracts a constant once for any token that has appeared at all. It encourages new topics more than it breaks loops.
Repetition penalty in the CTRL style divides positive logits (and multiplies negative ones) by a factor such as 1.1 to 1.3 for tokens already in the context, including the prompt in many implementations.

Penalties have a cost. They push probability away from tokens that legitimately repeat: variable names in code, keys and quotes in JSON, a product name in a description, the word "the". Push hard enough and that mass lands on less suitable tokens, sometimes on the junk tail, which is exactly what the lab shows when you combine a high frequency penalty with temperature above 1. Mission one starts from greedy decoding on long outputs; the clean solution is a little temperature, a tail filter and a modest penalty, not a large penalty on its own.

Matching settings to the failure

The practical way to choose sampling settings is to start from the failure you see and the task you have.

Symptom	Likely cause	First fix	Watch for
Same phrase repeated until the length limit	Greedy or very low temperature on long output	Temperature around 0.5 to 0.8 with a tail filter; small frequency penalty	Penalty damaging code or structured output
Occasional nonsense token, foreign script, broken word	Tail mass reached by high temperature or a weak model	Add top-p or min-p; reduce temperature	Variety dropping more than needed
All variants nearly identical	Temperature too low, or top-p too tight	Raise temperature and keep a tail filter	Junk returning; check clean rate
Invalid JSON or schema violations	Any randomness on a task with one right answer per step	Low temperature plus constrained decoding	Loops on long, repetitive structures
Behaviour changed after a model swap	Different tail shape, different defaults	Re-tune on the new model; re-check defaults	Settings copied from the old model

For structured output, sampling settings are the weaker tool. Constrained decoding, where a grammar or JSON schema masks invalid tokens at each step, removes format errors that no temperature can guarantee away. Use low temperature for content quality and constraints for format.

The knobs in real servers

The same concepts appear under similar names across serving stacks, but defaults and the order in which filters are applied differ. Check the documentation for the exact version you run.

vLLM SamplingParams: temperature, top_p, top_k, min_p, frequency_penalty, presence_penalty, repetition_penalty and seed. The value that disables top-k has changed between versions, so check what yours expects.
Hugging Face Transformers generate: do_sample must be true for temperature and the filters to apply; top_k has a non-zero default in the generation config (50 at the time of writing), which silently truncates if you only set top_p; min_p and repetition_penalty are available.
llama.cpp: --temp, --top-k, --top-p, --min-p, --repeat-penalty and --repeat-last-n, with a configurable sampler order.
Hosted APIs expose a subset, typically temperature and top-p, sometimes top-k and frequency and presence penalties. Some providers recommend adjusting temperature or top-p but not both.

A configuration for a generic chat endpoint in vLLM might look like this; treat the values as a starting point to evaluate, not a recommendation:

from vllm import SamplingParams

chat = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=800)
brainstorm = SamplingParams(temperature=1.2, min_p=0.05, max_tokens=300, n=8)
long_summary = SamplingParams(temperature=0.6, top_p=0.9,
                              frequency_penalty=0.3, max_tokens=1500)
extraction = SamplingParams(temperature=0.0, max_tokens=400)   # plus a JSON schema constraint

Two operational details matter. First, log the effective sampling parameters with every request, including server-side defaults; many incidents turn out to be a default that changed in an upgrade. Second, when you compare models, compare them at settings tuned for each, because a temperature that is safe for one model's tail may not be safe for another's.

A worked example: product descriptions at Belgavi's Book Store

Suppose a hypothetical Belgavi's Book Store uses a model to draft eight alternative blurbs for each new title, and an editor picks one. Three settings were tried, with outcomes described using the lab's synthetic model rather than any measurement.

Greedy decoding. All eight drafts are identical, so the feature is pointless, and long blurbs loop. In the lab, greedy decoding gives about 2% distinct 3-grams across samples and loops on every long output.

Temperature 1.6, no filter. The drafts are varied but roughly three in four contain a junk token. The editor's trust in the feature evaporates after a day. The arithmetic from the first section explains why: the per-step junk share is around 5% and an answer has dozens of open-ended steps.

Temperature 1.6 with min-p 0.05 (or top-p around 0.9). The tail is cut, nearly every draft is clean, and distinct 3-grams stay high. When the team later swaps to a smaller quantized model to save cost, junk returns at the old settings because the new model's tail is heavier; mission three reproduces this. The fix is to re-tune the filter on the new model, not to keep the settings that were validated on the old one.

The general procedure: decide what you need from the outputs (variety, faithfulness, format), measure clean rate and diversity on a fixed prompt set across a small grid of settings, and choose the most varied setting that keeps the clean rate above your bar.

Measuring and detecting sampling failures

Sampling problems are statistical, so detect them statistically.

Repetition: the share of responses with a repeated n-gram run (for example, any 4-gram appearing three or more times), and the share that stop because they hit max_tokens. A rise in either is the earliest sign of loops.
Junk: the share of responses containing characters from unexpected scripts, malformed words or tokens with very low log-probability under the model. Some servers can return log-probabilities for the sampled tokens; a sampled token with a very low log-probability is a direct signal that the tail was reached.
Variety: distinct n-grams or embedding similarity across multiple samples for the same prompt, for features that need alternatives.
Format: parse rate for structured outputs, tracked separately from content quality.

Common mistakes are mostly about defaults and copying. Teams copy a "creative" temperature from a blog post without a tail filter, leave a hidden top-k default in place, apply a strong repetition penalty globally and damage code generation, or tune settings on one model and carry them to another. Each is cheap to avoid with a small evaluation grid run whenever the model or server version changes.

The lab's missions follow the same path: break a greedy loop on long outputs, keep a hot brainstorming setting clean, recover from a model swap with a fatter tail, and finally handle long answers from that weaker model, where low temperature loops and high temperature produces junk, so you need temperature, a filter and a penalty working together.

Run the lab: Sampling settings: break loops and cut junk without killing variety

The lab samples dozens of answers from a synthetic next-token model with 50 plausible content tokens and a long tail of junk tokens, using the same temperature, top-k, top-p, min-p and frequency-penalty logic as real servers. The left chart shows all next-token probabilities at one open-ended step on a log scale: blue tokens are plausible, red are junk, and dark bars are cut by your filters. The right grid shows generated samples, one cell per token, with junk in red and loops labelled. Start with the default temperature of 1.0 and no filters, then load mission one.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

Log the effective sampling parameters for every request, including server defaults such as a hidden top-k.
Add dashboards for responses that hit max_tokens and for repeated 4-gram runs, split by route and model.
For every feature, write down whether it needs variety, faithfulness or strict format, and set temperature accordingly.
Pair any temperature above about 1 with a tail filter (top-p or min-p) and measure the clean rate on a fixed prompt set.
Use constrained decoding for JSON and schema outputs instead of relying on low temperature alone.
Re-run a small sampling grid whenever you change model, quantization or server version.

Sources and further reading

Related in this series:

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.