Why architecture matters here
The architecture matters because temperature is the single most direct control over the exploration-exploitation tradeoff in generation, and getting it wrong degrades output in ways that look like model failures but are really sampling failures. A model that loops, repeats phrases, or collapses into a bland safe answer is often not undertrained — it is being sampled too cold, so the sampler keeps picking the same high-probability continuation and never escapes a rut. A model that produces fluent nonsense, invented words, or wild topic drift is often being sampled too hot, so the flattened tail lets low-probability tokens through that the model never genuinely endorsed. The same weights, sampled at two different temperatures, behave like two different systems, which is why temperature deserves to be understood as an architectural parameter of the generation pipeline rather than a magic number.
The second forcing function is that different tasks want different points on that tradeoff. A code-completion or structured-extraction task wants near-deterministic output — one correct answer, no surprises — so it runs cold, often at T well below one or with greedy decoding entirely. A brainstorming, story-writing, or paraphrase task wants variety, so it runs hot enough that repeated calls yield genuinely different results. A factual question-answering task wants the middle: enough determinism to stay accurate, enough softness to phrase naturally. Because the objective differs, a temperature that is perfect for one use case is actively harmful for another, and a serving stack that hard-codes a single value for all traffic is leaving quality on the table across most of it.
The third reason is that temperature does not act alone — it composes with top-k and top-p (nucleus) truncation, and the interaction is subtle enough that stacking them naively produces randomness nobody intended. Top-k keeps only the k highest-probability tokens; top-p keeps the smallest set whose cumulative probability exceeds p; temperature rescales the distribution those truncations operate on. Whether you scale before or after truncating changes the result, and combining an aggressive temperature with an aggressive nucleus can either double down on determinism or fight each other. Understanding the order of operations is what keeps a sampling config predictable rather than a pile of interacting knobs.
A fourth, quieter reason is calibration. A well-trained model's raw distribution at T=1 encodes real uncertainty: when it is genuinely unsure between two continuations, the probabilities are close; when it is confident, one token dominates. Temperature distorts that calibration on purpose — cooling makes the model act more certain than it is, heating makes it act less certain — which is exactly what you want for controlling style, but it means the sampled probabilities are no longer honest estimates of correctness. Teams that use token probabilities as confidence signals must remember that any temperature other than one has already reshaped those numbers, so a downstream 'confidence threshold' has to be interpreted relative to the temperature it was measured at, not in absolute terms.
The architecture: every piece explained
Left to right, the pipeline is short but every stage matters. The raw logits are the model's unnormalized scores over the whole vocabulary — a vector z with one entry per token, produced by the final linear projection of the transformer. These numbers carry the model's preferences but are not yet probabilities; their absolute scale is arbitrary and only their differences are meaningful. The divide by T stage rescales every logit elementwise: z_i becomes z_i / T. This is the entire intervention — one scalar division — and because softmax cares only about differences between logits, dividing by T uniformly stretches (T>1) or compresses (T<1) all those differences by the same factor.
The softmax then exponentiates and normalizes: p_i = exp(z_i / T) divided by the sum over all j of exp(z_j / T). The exponential is what turns a linear rescaling of logits into a dramatic reshaping of probabilities. When T is small, the largest logit's exponential dwarfs the rest and its probability approaches one — the distribution becomes peaked, near-greedy. When T is large, the divisions shrink every logit toward zero, their exponentials toward one, and the distribution flattens toward uniform. At T = 1 the softmax runs on the untouched logits and yields the model's own distribution. The resulting probability distribution over the next token is what the sampler consumes.
The three middle boxes name the regimes. T < 1 (cold) concentrates mass on the top candidates, making output confident, consistent, and at the extreme identical to greedy decoding. T = 1 reproduces the model's calibrated beliefs. T > 1 (hot) lifts the tail, admitting diverse and surprising tokens at the cost of coherence. The sampler — multinomial draw, usually after top-k or top-p truncation — makes the final selection from whatever distribution temperature produced.
The bottom row is where correctness lives. Interaction with top-k / top-p is the ordering question: conventionally temperature is applied to the logits first, then truncation selects the candidate set from the rescaled distribution, then a final renormalization and sample — but some stacks truncate first, and the two orders give different token probabilities. Numerical stability is the implementation detail that keeps the math honest: softmax must subtract the max logit before exponentiating to avoid overflow, and T must be guarded against zero (which would divide by zero) by treating T=0 as a special greedy case. The ops strip is the empirical loop: sweep T and watch repetition, coherence, and diversity metrics on the actual task.
End-to-end flow
Trace a single next-token step at three temperatures to see the reshaping concretely. Suppose the model has just written 'The capital of France is' and produces logits where 'Paris' scores far above every other token, with 'Lyon', 'a', and a long tail well below. At T = 1, softmax gives 'Paris' a high probability — say most of the mass — with a little spread over the alternatives; the sampler almost always picks 'Paris' but occasionally something else. Now cool to T = 0.5: dividing the logits by 0.5 doubles every difference, so 'Paris' pulls even further ahead and its probability approaches one; the sampler is now effectively deterministic here. Heat to T = 1.5: the differences compress, 'Paris' still leads but the alternatives gain real mass, and there is now a non-trivial chance the model writes something wrong or unusual — undesirable for a factual completion, which is why this task wants a cold temperature.
Contrast that with a creative step. The prompt is 'Write an opening line for a fantasy novel:' and the logits are broad — no single token dominates because many continuations are plausible. At T = 0.7 the model is fairly constrained and repeated calls produce similar, safe openings. Raise to T = 1.1 and the flattened tail lets genuinely different word choices through, so each call yields a distinct line — exactly the diversity a brainstorming task wants. Push to T = 1.8 and the tail is so heavy that grammatically odd or incoherent tokens start slipping in; the outputs become varied but unreliable. The same model, same prompt, three temperatures, three qualitatively different behaviors.
Now watch temperature compose with nucleus sampling in a real generation config. The stack applies T = 1.2 to the logits, then top-p = 0.9 truncation keeps the smallest set of tokens whose rescaled probabilities sum to 0.9, then renormalizes and samples. The temperature has already lifted the tail, so the nucleus at p = 0.9 now includes more candidates than it would have at T = 1 — the two knobs compound toward diversity. Swap the order — truncate at the raw distribution, then apply temperature to the survivors — and the candidate set is chosen from the sharper original distribution, so it is smaller, and temperature only reshapes within that smaller set. The generated text differs measurably between the two orders even though both configs 'use T = 1.2 and top-p = 0.9', which is why the pipeline's order of operations must be pinned and documented, not left to a library default.
Finally the degenerate edges. Set T toward zero and, without a special case, the division explodes; implementations therefore treat T = 0 as pure greedy decoding — take the argmax, skip sampling — which is deterministic and often correct but prone to looping on repetitive text. Set T very high and the distribution approaches uniform, so the model samples nearly at random over the vocabulary and produces gibberish. Between those poles lies the usable band, and finding the right point in it for a given task is the whole practical art of temperature tuning — done not by intuition but by sweeping T and measuring the output the task actually cares about.