Why architecture matters here
Architecture matters here because the decoding loop is where model quality becomes user-visible output, and it is the single most misconfigured part of most LLM stacks. A team can pick an excellent model and still ship bad results purely because the default generation parameters did not match their task. Knowing how the loop is assembled turns vague complaints — 'the model rambles', 'it cuts off answers', 'it is too repetitive' — into specific, fixable knobs, because each of those symptoms maps to a specific stage of the pipeline.
The problem is genuinely non-trivial because the same method must serve wildly different decoding algorithms without exposing their differences as separate APIs. Greedy decoding, multinomial sampling, and beam search have different memory profiles, different determinism properties, and different notions of what 'the batch' even means (beam search tracks multiple candidate sequences per input). Folding all of them into one loop, driven by one config, is what makes the library usable — but it also means a single wrong flag can silently switch you from one algorithm to another with very different behavior.
Consider a concrete failure to feel the stakes. A team sets a high temperature and a large top-k for a task that needs factual, deterministic answers, and the model produces creative but wrong output; another team leaves sampling off entirely for a brainstorming task and gets flat, repetitive text. Neither changed the model. Both are living entirely inside the generation config, and both would be fixed by understanding that temperature, top-k, and top-p are logits-processor stages that reshape the distribution before selection, while the do-sample flag decides whether selection is stochastic at all.
The payoff of understanding the architecture is control that transfers across models. Because every model in the library goes through the same generate() loop, a mental model of that loop applies whether you are running a 1B on-device model or a 70B server model. The logits processors, stopping criteria, cache, and streamer are the same machinery; only the weights differ. Learning the loop once pays off across the entire ecosystem, and it makes debugging portable — the same diagnostic questions apply everywhere.
There is also a real efficiency dimension. The key-value cache is what makes autoregressive decoding affordable: without it, each new token would re-attend over the entire prefix from scratch, making generation quadratic in length. With it, each step attends over the cached keys and values and appends one position, so the per-token cost is roughly constant. Configuring generation well therefore means not just choosing a good sampling strategy but ensuring the cache is enabled and sized correctly, because that single choice is the difference between linear and quadratic decoding cost.
The architecture: every piece explained
Top row: the control and forward path. The GenerationConfig holds every decoding parameter and is the single source of truth for the run; the method merges the model's default config with any per-call overrides. Prepare inputs tokenizes and pads the prompt, builds the attention mask, and initializes the cache. The model forward runs one step and returns logits for the next token over the full vocabulary. The KV cache stores the keys and values computed so far so the next step does not recompute them; it grows by exactly one position per generated token.
Middle row: the per-step decision machinery. The logits processors are an ordered pipeline that reshapes the raw logits before selection. Temperature scaling flattens or sharpens the distribution; top-k keeps only the k highest-probability tokens; top-p (nucleus) keeps the smallest set whose cumulative probability exceeds p; a repetition or no-repeat-ngram penalty suppresses tokens that would repeat recent context; a bad-words or constraint processor masks forbidden tokens to negative infinity. The selection stage then turns the reshaped distribution into one token — argmax for greedy, a multinomial draw for sampling, or the beam-scoring update for beam search. The stopping criteria decide whether the loop halts: an EOS token was produced, the maximum length or time was reached, or a custom stop condition fired.
Bottom row: the output path. Once a token is selected, append token extends the running sequence and the cache advances. If a streamer is attached, the token is emitted immediately — this is what powers token-by-token UIs, where text appears as it is generated rather than after the whole sequence completes. Without a streamer, tokens accumulate silently until a stopping criterion fires and the full sequence is returned.
The composition is the important architectural idea. Logits processors and stopping criteria are both lists assembled from the config, and each is applied in order every step. This is why the same loop serves every strategy: switching from greedy to nucleus sampling does not change the loop, it changes which processors are in the list and whether selection samples or takes the argmax. New behaviors — a custom penalty, a JSON-grammar constraint, a stop-on-phrase rule — are added by inserting a processor or criterion, not by forking the loop.
The ops strip is the feedback loop. Tokens per second measures decoding throughput and is dominated by the model forward and cache efficiency. Cache memory tells you how much device memory the KV cache consumes, which scales with batch, sequence length, and beam count. Truncation rate — how often generation hits max length instead of EOS — reveals prompts that need a longer budget or a model that will not stop. The stop-reason distribution ties it together, showing at a glance whether generations end naturally or are being cut off.
End-to-end flow
Trace one call to generate(). The application passes a tokenized prompt and a GenerationConfig that requests sampling with temperature 0.7, top-p 0.9, a repetition penalty, a maximum of 256 new tokens, and the model's EOS token as the stop signal. The method merges this config, tokenizes and pads the input, builds the attention mask, initializes an empty KV cache, and assembles two lists: a logits-processor pipeline (temperature, then top-p, then repetition penalty) and a stopping-criteria list (max-length and EOS).
Step one runs. The model attends over the prompt, populates the KV cache with the prompt's keys and values, and returns logits for the first generated token. The logits-processor pipeline runs in order: temperature divides the logits, the repetition penalty scales down tokens already present, and top-p zeroes out everything outside the nucleus. Because do-sample is on, selection draws a token multinomially from the reshaped distribution. The stopping criteria check: not EOS, not at max length, so the loop continues. The token is appended and, if a streamer is attached, emitted to the UI.
Step two is where the cache earns its keep. Instead of re-running attention over the whole prompt plus the first token, the model processes only the single new token, attending over the cached keys and values from step one and appending its own key and value to the cache. It returns logits for the second token. The same processor pipeline reshapes them, selection samples, stopping criteria check, and the token is appended. Every subsequent step costs roughly the same — one token's worth of forward compute — because the expensive prefix work is cached, not repeated.
The loop continues token by token. Suppose at step 140 the model samples the EOS token. The EOS stopping criterion fires, the loop halts, and the method returns the generated sequence (or, with a streamer, the final token was already emitted). Had the model kept going without ever producing EOS, the max-length criterion would have fired at 256 tokens and truncated the output — which is exactly the event the truncation-rate metric is designed to catch, because a high truncation rate means either the budget is too small or the model is failing to stop.
It is worth pausing on what the streamer changed about the user's experience during that trace. Without one, the application would have waited the full 140 steps before showing anything, so the user would stare at a blank screen for the entire generation. With the streamer attached, the first token was emitted the instant it was selected at step one, and every subsequent token appeared as it landed, so the perceived latency collapsed from full-sequence time to first-token time. Nothing about the decoding math changed — the same logits, the same processors, the same selection produced the same tokens — but the ordering of when those tokens became visible transformed a slow-feeling wait into a live, responsive stream. That is a pure architecture win: the loop already produces tokens one at a time, and the streamer simply surfaces that fact to the client instead of hiding it behind a final return.
Step back and see how the pieces cooperated. The config selected the algorithm and its knobs; the processor pipeline shaped every distribution consistently; the selection strategy turned distributions into tokens; the cache made each step cheap; the stopping criteria ended the run cleanly; and the streamer delivered a live experience. Change the task and you change the config, not the loop: a deterministic extraction task flips do-sample off and drops temperature to make selection argmax; a constrained-JSON task inserts a grammar processor that masks any token that would break the schema. The architecture absorbs all of these as configuration, which is precisely why one method can serve the whole zoo of models and decoding strategies.