Why it matters

Speculative decoding is one of the biggest recent inference optimizations. Understanding it explains why modern LLM inference feels fast.

Advertisement

The architecture

Draft model: small, fast (e.g., 1B). Generates N candidate tokens autoregressively.

Target model: large. Verifies all N candidates in one forward pass (parallel).

Speculative decodingDraft modelpropose N tokensTarget modelverify in parallelAccept matchesreject mismatches1-3 tokens accepted on average; each accept saves target-model latency
Speculative flow.
Advertisement

How it works end to end

Acceptance: target model's distribution at each position determines whether draft token is accepted. Statistically identical to target-only sampling.

Speedup: proportional to average accepted tokens per step (2-3 typical).

Requirements: draft and target must be same tokenizer + compatible distributions.