Why it matters

Speculative decoding is production speedup. Understanding shapes serving.

Advertisement

The architecture

Draft: small fast model.

Target: full model.

Verify: accept prefix that matches.

Speculative decodingDraft modelpropose K tokensTarget modelverify in parallelAccept prefixmatching tokensRejection sampling ensures target model output distribution preserved
Speculative flow.
Advertisement

How it works end to end

Draft: 1B or smaller.

Target: 70B.

K tokens per step: 4-8 typical.

Accept rate: 60-80% common.