DECODE · INSPECT · EXPLAIN
Compare greedy, top-k and top-p decoding
Hold the fixture fixed while changing candidate selection and sampling policy.
Your experiment
Capture a baseline with all eight tokens. Apply top-k, then top-p, and inspect which token probabilities become zero. Switch to greedy and explain why changing the seed no longer changes the chosen token.
Press Play for a full minute at 1×, or use 0.5× for two minutes. Pause to read a stage. Next stage and the timeline reveal the same computed history; changing any model input resets the inspection without autoplay.
What the calculation means
The fixture has eight tokens. These outcomes demonstrate the algorithms, not comparative quality on a trained language model.
The timeline is teaching time, not measured inference latency. A five-stage inspection is allocated to each generated token, with the complete trace spread over 60 seconds.
Visible arithmetic
| Token | Logit | Penalized | Scaled | Softmax | Filtered |
|---|
Context and cache inspector
Offline teaching fixture, not a trained transformer. An explicit eight-token score table uses only the last context token; unknown last tokens use the “the” row. Prompt tokens are whitespace-separated, not BPE tokens. Greedy or fixed-seed sampling operates on exact visible probabilities. EOS or the selected token limit ends decoding. Cache dimensions are accounting inputs only: no attention matrices or real model weights are executed. Changing cache settings must not change this fixture’s output tokens. Cache mechanics · Decoding reference · Official AICassindra articles.
Compare policies on the same supplied scores
This experiment compares next-token selection rules on one fixed eight-token distribution. It performs actual softmax normalization, ranked filtering and deterministic seeded selection, but does not run a trained language model. Prompt fixture is whitespace-separated text; its final token selects one of six known score rows. An unknown final token uses the the row. Previous tokens are only used to identify repetition-penalty membership. With prompt on, scores for mat, the and a are three, two and one respectively. Changing Strategy does not change these logits or the probability table. The page appends exactly one new token.
Understand the declared filtering order
Top-k candidates limits the ranked list to between one and eight entries. Nucleus top-p operates afterward, using probabilities renormalized within that top-k list. It keeps the smallest ranked prefix whose cumulative mass reaches the chosen threshold, including the token that crosses it. Thus top-k and top-p are sequential, not independent filters whose outputs are intersected. Retained mass reports the surviving probability mass from the original unfiltered distribution. With prompt on, temperature one and penalty one, choose Top-k one: only mat survives and its filtered probability is one. Set Top-k two and top-p one: only mat and the remain, in the ratio exp(1). Set top-p 0.5: mat alone exceeds half of the two-candidate mass.
Compare randomness without confusing its cause
Greedy argmax takes the largest filtered probability and uses vocabulary order for exact ties. Seeded sampling uses a reproducible draw and the filtered cumulative probabilities. Sampling seed changes the draw rather than the scores. For prompt cat, temperature two, penalty one, Top-k eight and top-p one, seed one selects sat and seed 9999 selects cat. Greedy selects sat for both seeds. Record these outcomes with the table held constant. A single sampled token is not evidence of a long-run quality or diversity improvement; this small fixture does not measure either. Top-k one removes sampling variation even when you change seed.
Temperature and repetition are upstream operations
Temperature divides the penalized logits before stable softmax. A higher temperature reduces score separation but preserves the ranking unless another transformation changes the scores. For prompt cat with penalty one, sat and on have scores three and one, so their unfiltered probability ratio is exp(2/T). Repetition penalty applies only to tokens present anywhere in the prompt, ignoring case. It divides a positive seen score and multiplies a negative seen score by the selected penalty; multiple occurrences do not increase the penalty again. Enter sat cat with penalty three to reduce the sat score from three to one. Unseen on remains one. Restore penalty one when comparing filters in isolation.
Replay a decision and inspect its limits
Play exposes five stages across a sixty-second teaching timeline: context, score transformation, normalization and filtering, selection, and append. Pause freezes the display. Next stage advances twelve teaching seconds; Inspection seeks to any time, and Restart returns to zero while preserving settings. Speed selects quarter, half, normal or double speed. At double speed the full run still takes thirty seconds; these are teaching durations, not measured token latency. Any model-parameter edit resets and pauses the timeline. Empty prompts and more than thirty-two whitespace tokens clear the numerical table and disable playback. An unknown token is a documented fallback rather than a validation error. The Context and cache inspector exposes which output has been revealed; cache byte fields are inherited fixture accounting only. Both probability columns and signed logits are available in a horizontally scrollable table for phone and keyboard readers.