Why it matters

Speculative decoding cuts latency. Understanding shapes serving.

Advertisement

The architecture

Draft: fast small model.

Target: large accurate model.

Verify candidates in single forward.

Speculative decodingDraft modelsmall + fastVerify in paralleltarget modelAccept + iterateor rejectMedusa: multi-head extension. EAGLE: features + draft together
Speculative variants.
Advertisement

How it works end to end

Acceptance rate: high when draft matches target.

Speedup: 2-3x typical.

Variants: Medusa (multi-head), EAGLE, Lookahead.

Trained draft: same tokenizer + close distribution.