Why architecture matters here
Architecture matters because the obvious latency fixes all cost quality, and speculative decoding is the rare one that does not. You can quantize the model (some quality loss, and still memory-bound), prune it (quality loss), or serve a smaller model (quality loss). Speculative decoding is lossless: with the correct acceptance rule the emitted sequence is drawn from precisely the target's distribution, token for token, so evaluation metrics are unchanged. That property — faster with zero quality regression — is why it is now standard in high-throughput inference stacks. It exploits a structural fact about the hardware (decoding is bandwidth-bound, so the target has spare compute) rather than trading away model capability.
The key insight is the asymmetry between generating and verifying. Generating K tokens autoregressively is inherently sequential: token 2 depends on token 1. But if you already have a candidate sequence of K tokens, checking whether the target model 'agrees' with each position is a single parallel forward pass — the target computes the probability of every drafted token given its predecessors simultaneously, the same way it processes a prompt during the prefill phase. So one expensive target pass can confirm several tokens instead of producing just one. When the draft is good, you get multiple tokens per target pass; when it is bad, you fall back to roughly one, never worse than the target alone plus a small draft overhead.
Understanding the architecture is what lets you reason about when it pays. The speedup is bounded by the acceptance rate and the draft/target cost ratio, and both are workload-dependent. A team that treats speculative decoding as a magic 3× will be disappointed on adversarial or highly creative generation where the draft rarely agrees; a team that understands the levers can pick a draft model, a value of K, and a serving configuration that actually delivers.
The losslessness property is also what makes speculative decoding safe to deploy without re-running your evaluation suite. Quantization, pruning, or swapping to a smaller model all change the output distribution, so each demands a fresh quality assessment before it ships. Speculative decoding, correctly implemented, provably emits from the target's own distribution, so the model you evaluated is byte-for-byte the model you serve — the only thing that changed is latency. That means it can be turned on as a pure infrastructure optimization, orthogonal to model quality, which is a rare and valuable property in an inference stack where most speedups force a quality trade-off.
The architecture: every piece explained
The draft model is small and fast — a distilled version of the target, a smaller model from the same family, or even a lightweight head bolted onto the target itself (self-speculation). It runs autoregressively for K steps to produce candidate tokens x₁…x_K, along with the draft's probability for each. K (the speculation length or 'lookahead') is a tunable constant, commonly 4–8; larger K risks more of the draft being rejected while smaller K limits the maximum speedup.
The target model then does one forward pass over the prompt-plus-draft, producing its own probability distribution at each of the K+1 positions (K over the drafted tokens, plus one more for the token after the last accepted one). The acceptance rule is modified rejection sampling applied left to right: for drafted token x_i, accept it with probability min(1, p_target(x_i) / p_draft(x_i)). If accepted, move to the next position; if rejected at position i, stop and resample that position from a corrected distribution — the positive part of (p_target − p_draft), renormalized — so the emitted token still comes from the target's distribution. Everything after the first rejection is discarded. This rule is the mathematical heart of the method: it is provably equivalent to sampling directly from the target.
Two consequences fall out. First, whenever the entire draft is accepted, the target's forward pass also gives you a free bonus token — the next-token distribution at position K+1 — so accepting all K yields K+1 tokens from one target pass. Second, the expected number of accepted tokens per pass is governed by the acceptance rate α, the average agreement between draft and target; a common approximation for expected tokens per target step is (1−α^(K+1))/(1−α). The realized speedup then trades this against the draft's cost: variants like Medusa (multiple prediction heads on the target, no separate draft model) and EAGLE (drafting in feature space) and tree/n-gram drafts (verify several candidate continuations at once) all aim to raise α or cut draft cost.
It is worth being precise about why verification is a single pass. During normal generation the target is autoregressive: token i+1 cannot be computed until token i exists, so K tokens require K sequential passes. But once the K draft tokens are in hand, the target can process the whole prompt-plus-draft sequence the way it processes a prompt during prefill — computing the conditional distribution at every position in parallel, because all the inputs already exist. The self-attention over K+1 positions is one matrix computation, and the memory-bandwidth cost of streaming the weights is paid once instead of K times. That is the entire source of the speedup: the target's weight-loading cost is amortized across multiple candidate tokens, turning a bandwidth-bound sequential bottleneck into a compute-bound parallel one where the accelerator's idle cycles finally get used.
End-to-end flow
Take a prompt and K=4. The draft model runs four fast steps and proposes the continuation 'the cat sat on'. Along the way it records its own probability for each of those four tokens. Total cost so far: four cheap draft passes.
The target model now processes the prompt plus those four tokens in one forward pass, yielding its probability distribution at five positions. The acceptance loop walks left to right. 'the' — target's probability is high relative to the draft's, accept. 'cat' — accept. 'sat' — accept. 'on' — here the target actually preferred 'the mat', so p_target('on') is low relative to p_draft('on'); the token is rejected. Generation stops at the first three accepted tokens, and the fourth position is resampled from the corrected (p_target − p_draft)+ distribution, which yields, say, 'a'. Net result from a single target pass: four emitted tokens ('the cat sat a'), versus one token from a normal pass.
Had all four drafted tokens been accepted, the target's fifth-position distribution would have handed over a free bonus token, for five tokens in one pass. The engine then loops: the accepted tokens extend the sequence, the draft speculates the next K, and the target verifies again. Over a full generation the average is 'tokens accepted per target pass' ≈ the expression above — so with a well-aligned draft accepting ~70–80% of tokens, you emit two to three tokens per expensive target pass instead of one, cutting wall-clock latency by roughly that factor minus the draft overhead. The emitted text is drawn from exactly the target's distribution throughout, so quality is untouched; only the number of sequential big-model passes fell.
The economics become intuitive once you see the two extremes. If the draft is perfectly aligned and every token is accepted, one target pass emits K+1 tokens, and latency drops by nearly a factor of K+1 (minus the small draft cost) — the best case. If the draft is useless and its very first token is rejected every time, you emit exactly one token per target pass (the resampled one), identical to plain decoding, and you have merely wasted the draft's cheap effort — the worst case, which is bounded and mild precisely because the draft is cheap. Real workloads sit between: predictable, low-entropy text (code, structured output, quoted RAG context) accepts long runs, while high-entropy creative generation accepts short ones, and the measured acceptance rate on your actual traffic is what tells you which regime you are in.