DECODE · INSPECT · EXPLAIN
Autoregressive generation: one decision at a time
Predict → select → append → repeat. Inspect how each output becomes the next input.
On narrow screens, scroll the diagram horizontally or focus it and use the arrow keys to inspect full-size labels. Exact values follow below.
Your experiment
Compare greedy decoding with seeded sampling. Restart without changing the seed: the exact token sequence must repeat. Change the prompt’s last token and inspect the different score row.
Press Play for a full minute at 1×, or use 0.5× for two minutes. Pause to read a stage. Next stage and the timeline reveal the same computed history; changing any model input resets the inspection without autoplay.
What the calculation means
With caching enabled, the first pass processes the entire prompt. Later passes process only the newest input token. The just-selected output is not yet in the cache.
The timeline is teaching time, not measured inference latency. A five-stage inspection is allocated to each generated token, with the complete trace spread over 60 seconds.
Visible arithmetic
| Token | Logit | Penalized | Scaled | Softmax | Filtered |
|---|
Context and cache inspector
Offline teaching fixture, not a trained transformer. An explicit eight-token score table uses only the last context token; unknown last tokens use the “the” row. Prompt tokens are whitespace-separated, not BPE tokens. Greedy or fixed-seed sampling operates on exact visible probabilities. EOS or the selected token limit ends decoding. Cache dimensions are accounting inputs only: no attention matrices or real model weights are executed. Changing cache settings must not change this fixture’s output tokens. Cache mechanics · Decoding reference · Official AICassindra articles.
Autoregressive Generation: Inspect Every Decision: experiment guide
This experiment executes a small autoregressive decoder over an explicit eight-token vocabulary. It is a deterministic teaching fixture, not a trained language model. Prompt fixture supplies whitespace-separated context tokens; its final token selects a row of fixed scores. Unknown final tokens use the score row for the word the. Earlier context affects repetition penalties but does not otherwise enter the score table. Entering natural prose therefore does not invoke semantic understanding. The table exposes original logits, penalized logits, temperature-scaled logits, softmax probabilities and the final filtered distribution, making every selection reproducible.
Worked details 1
Strategy selects greedy argmax or seeded sampling. Temperature divides the penalized logits before softmax; positive temperatures change probability spread but cannot change the greedy ranking by themselves. Top-k retains the highest-scoring candidates. Nucleus top-p then keeps the smallest ranked prefix reaching the requested probability mass after the top-k probabilities have been renormalized. The final retained set is renormalized again. Repetition penalty divides positive scores of previously seen tokens and multiplies negative scores by the penalty. A value of one has no effect. Sampling seed initializes a fixed integer pseudorandom sequence; it affects selection only under seeded sampling. Maximum new tokens bounds the loop, while selecting the explicit end-of-sequence token stops it earlier.
Worked details 2
Use the initial prompt the cat sat on, greedy selection, temperature one, top-k eight, top-p one, penalty one and token limit eight. The generated suffix is mat, period, then end-of-sequence. With Retain K/V enabled, the three prediction steps compute four, one and one new positions: six cumulatively. Disable that checkbox and the same tokens must be produced, but the steps recompute four, five and six positions: fifteen cumulatively. The final selected end-of-sequence token has not yet been forwarded through the model and is not in the cache. With the fixed accounting dimensions, each processed position costs 12,288 bytes; six cached positions therefore occupy 73,728 bytes. This accounts for K/V storage only and does not execute attention or measure inference speed.
Worked details 3
Switch to seeded sampling with seed seven while retaining the other initial inputs. The suffix becomes sat, period, end-of-sequence. Restart reproduces that sequence. Set top-k to one: sampling becomes deterministic and agrees with greedy selection. A very small top-p can similarly leave a single candidate. Set Maximum new tokens to one to see token-limit termination after mat in the greedy fixture. Empty prompts fail validation and disable Play and Next stage; prompts are limited to 32 whitespace-separated tokens and 256 characters. No padding, beam search, batching, tokenizer or context-window eviction is modeled.
Worked details 4
Play advances through five stages per generated token over sixty teaching seconds. Pause freezes inspection. Next stage advances one stage, and Inspection seeks directly while pausing playback. Restart preserves every input and returns to the beginning; it does not restore default parameters. Speed offers quarter, half, normal and double speed without altering results. Changing a model input recomputes the history and leaves it paused. The context inspector supplies complete token strings and cache counts as text, including strings abbreviated in the canvas. Teaching seconds express reading time only. Compare the numerical table before and after each control change, since unchanged greedy output can still accompany a meaningful change in probabilities.