DECODING · DRAFT AND VERIFY · THROUGHPUT
Speculative Decoding
Generate 48 tokens with a draft model and a target verifier, and see how draft length, acceptance rate and draft cost decide whether it is faster.
Your experiment
Plain decoding costs one target pass per token. Start with 4 drafts per round, acceptance 0.8 and a draft that costs a tenth of a target pass, then change the draft length, lower the acceptance, make the draft expensive and change the seed near break-even.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, Speed changes the pace and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Time is counted in target forward passes; verifying gamma + 1 positions is assumed to cost one pass, as when decoding is memory-bandwidth bound. Acceptances are independent with one fixed rate, the assumption behind the expected-speedup formula. KV-cache rollback, batching, tree drafts and self-drafting heads are omitted.
Draft, verify, keep the prefix
Speculative decoding asks a small draft model for gamma tokens, set by Draft tokens per round, then runs the large target model once over all of them. Drafts are kept up to the first rejection, where the target supplies its own token; if every draft is accepted the target adds one bonus token. The lab generates 48 tokens. Plain decoding costs 48 target passes; here each round costs gamma times the draft cost plus one target pass, and each draft is accepted with the chosen rate, independently.
The expected speedup
With acceptance 0.8 and gamma = 4 the expected tokens per target pass are (1 - 0.8^5) / (1 - 0.8) = 3.362. Dividing by the round cost 4 x 0.1 + 1 = 1.4 gives an expected 2.401x speedup. The seeded run takes 13 rounds and 18.2 time units and measures 2.637x. The best draft length for these settings is 6, at 2.47x: beyond it extra drafts are mostly rejected and only add cost. These are the formulas of Leviathan, Kalman and Matias (2023), under their assumption of independent acceptances.
When it is slower
Lower Per-token acceptance rate to 0.5 and raise Draft cost per token to 0.3: the expected speedup is 0.881x, the run measures 0.779x, SLOWER THAN PLAIN DECODING, and the best draft length drops to 1. A draft that is expensive relative to the target, or that often disagrees with it, costs more than it saves. With acceptance 0.95, a cost of 0.02 and gamma = 8 the expected speedup is 6.375x.
One run is not the expectation
Acceptance seed changes which drafts are accepted. At acceptance 0.3 the expected speedup is 1.018x but the seeded run measures 0.952x; at a draft cost of 0.6 the expected value is 0.989x and the run measures 1.086x. Near break-even a single run can land on either side, so measure over many requests before deciding. Speculative decoding with exact verification never changes the output distribution; the verdict here is only about time.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time step through the setup, the formula, the first ten rounds, the last round, the comparison and the verdict across sixty inspection seconds; the chart plots tokens against time for both methods and the table lists every round. Time units are target passes, verification of gamma + 1 positions is assumed to cost one pass, and acceptances are independent with one rate. Real systems also pay for KV-cache rollback, and batching changes the trade-off.