Why architecture matters here
To understand why the architecture works you must understand the hardware imbalance it exploits. During decode, generating one token requires reading every weight of the model from GPU memory into the compute units exactly once, and doing a relatively tiny amount of arithmetic with each — a matrix-vector product per layer. The arithmetic intensity (flops per byte) is minuscule, so the operation is bounded by memory bandwidth: the tensor cores finish their work and wait for the next weights to arrive. The result is that a decode step at batch size one uses only a small fraction of the GPU's arithmetic throughput. There is enormous spare compute going unused on every single token.
Speculative decoding matters because it converts that idle compute into speed. When the target model verifies K draft tokens in one pass, it processes K positions' worth of arithmetic while reading the weights only once — the same single expensive weight read that a normal decode step pays, now amortized over K tokens instead of one. Because the step was memory-bound, doing K times the arithmetic costs almost nothing extra: you were reading the weights anyway. So a successful speculation of K tokens takes roughly the wall-clock time of one ordinary decode step but advances the sequence by up to K+1 tokens. The spare compute the GPU was wasting becomes the engine of the speedup.
The second load-bearing property is that this speedup is free of quality loss, which is what makes it deployable for production inference where correctness matters. The accept-reject procedure — often called speculative sampling — is constructed so that the sequence of accepted tokens is drawn from exactly the target model's distribution. It is not 'the draft model's answer if it's close enough'; it is a mathematically exact sampling of the target model, with the draft serving only to propose candidates that the rigorous verification either accepts or corrects. This distinguishes speculative decoding from lossy accelerations like using a smaller model outright or aggressive quantization: you pay nothing in output quality, only in the complexity of running two models in concert.
The third reason the architecture is subtle is that the speedup is entirely governed by the acceptance rate — how often the target model agrees with the draft's proposals — and that rate depends on how well the small draft model mimics the large one. A draft that is too weak has its proposals rejected constantly, so you pay for draft passes and gain almost nothing; a draft that is too strong is itself slow, eating the savings. The whole design is a balancing act: the draft must be cheap enough that proposing is nearly free, yet aligned enough with the target that its guesses are usually accepted. Everything in the operational playbook — draft-model choice, speculation length, self-speculation variants — is really about maximizing accepted tokens per unit of drafting cost, which is the single number that determines whether speculative decoding helps or hurts.
The architecture: every piece explained
Top row: proposal and parallel verification. The draft model — a much smaller model, a distilled head, or even an n-gram lookup — runs autoregressively for K steps to produce draft tokens, a candidate continuation of the sequence. Because the draft is small, these K sequential steps are cheap. Then the target model runs one forward pass over the original context plus all K draft tokens at once; a transformer naturally computes, at every position, the probability distribution for the next token, so this single pass yields the target's own predicted distribution at each of the K+1 positions — exactly the information needed to check every draft token in parallel.
Middle row: the accept-reject rule. Verification compares, position by position, the draft's proposed token against the target's distribution there. Under speculative sampling, each draft token is accepted with a probability derived from the ratio of the target's and draft's probabilities for that token; the accepted prefix is the longest run of tokens accepted from the start. At the first rejection, the algorithm samples a correction token from an adjusted residual distribution (the target's distribution with the already-rejected mass removed), which guarantees the overall output matches the target's distribution exactly. So a round yields the accepted prefix plus one correction token — between 1 and K+1 new tokens — all from a single target pass.
Bottom rows: state management and variants. Because the target model processed K speculative tokens, its KV cache now contains entries for positions that may have been rejected; the algorithm rolls back the KV cache, trimming it to the accepted length so the next round starts from a clean, correct state. Then it loops. The drafting strategy is where the variants live: a separate small draft model is the classic approach, but self-speculation methods avoid a second model entirely — Medusa adds extra prediction heads to the target that propose multiple future tokens, EAGLE drafts in the target's feature space for higher acceptance, and simple n-gram / prompt-lookup drafting proposes tokens copied from the context, which works startlingly well for repetitive or retrieval-grounded generation. The ops strip names the three numbers that matter: the acceptance rate, the draft's cost relative to the target, and the speculation length K, which must be tuned because too-long speculations waste draft work on tokens that will be rejected anyway.
The correctness argument deserves to be made concrete, because it is what elevates speculative decoding from a heuristic to a guarantee. Consider a single position where the draft proposed token t with draft-probability q(t) and the target assigns it p(t). The rule accepts t with probability min(1, p(t)/q(t)). When the draft over-proposes a token the target likes less, the min clips acceptance to the target's lower rate; when it rejects, resampling from the normalized positive part of (p − q) exactly fills in the missing probability mass. Summed over all outcomes, the probability that the algorithm emits any given token equals p(t) precisely — the draft distribution cancels out of the final result. That algebraic cancellation is the whole reason the speedup costs no quality: the draft influences only how fast you sample, never what you sample.
End-to-end flow
Trace one speculation round with K=4. The sequence so far ends in '...the capital of France is'. The target model would, on its own, take four sequential passes to emit the next four tokens. Instead, the draft model — a small distilled version — runs four quick steps and proposes the continuation ' Paris , which is'.
Parallel verification. The target model runs a single forward pass over the context plus the four draft tokens ' Paris', ' ,', ' which', ' is'. In that one pass it produces its own next-token distribution at each position: after 'is' it strongly predicts ' Paris'; after ' Paris' it predicts ' ,'; after ' ,' it predicts ' the' (not ' which'); and so on. The single pass has, in effect, graded all four guesses at once, using the spare compute that a normal single-token decode would have left idle.
Accept-reject. Verification walks the positions. ' Paris' — the target agrees, accept. ' ,' — the target agrees, accept. ' which' — the target's distribution favors ' the'; the speculative-sampling test rejects ' which'. The accepted prefix is ' Paris ,' (two tokens). At the rejection point the algorithm samples a correction token from the target's residual distribution and gets ' the'. So this round, from one target forward pass, the sequence advanced by three tokens — ' Paris', ' ,', ' the' — instead of the one token a normal decode step would have produced. On an easier, more predictable stretch of text the draft might have hit all four, advancing five tokens in one pass.
Rollback and loop. The target's KV cache had computed entries for all four speculative positions, but only three tokens survived (two accepted plus the correction). The cache is trimmed to that accepted length so the state is exactly as if the model had generated ' Paris , the' normally, and the next round begins from there with a fresh draft. Over a full generation the average number of tokens accepted per target pass — the acceptance rate times K, plus the guaranteed correction token — is what determines the realized speedup: a well-matched draft on predictable text might average three or four tokens per target pass, a two-to-four-times wall-clock speedup, while on hard, high-entropy text where the draft and target disagree often the acceptance collapses toward one and the technique quietly degrades to near-ordinary decoding plus the wasted draft cost. That variability is why acceptance rate is the metric you watch, and why the speculation length K is tuned rather than fixed.