Why it matters

Speculative decode boosts throughput 2-3x with no quality loss. Standard in prod serving.

Advertisement

The architecture

Draft model proposes K tokens.

Target verifies in one forward.

Accept prefix.

Speculative decodingDraft modelsmall, fastTarget verifyone forwardAccept prefix2-3x tokens/stepLeviathan et al. 2023 proved correctness; vLLM and TRT-LLM support in production
Speculative decoding.
Advertisement

How it works end to end

Leviathan et al. 2023.

Draft in vLLM.

TRT-LLM speculative.