Why it matters

Medusa avoids managing a draft model. Simpler ops for similar speedup.

Advertisement

The architecture

K heads predict future tokens.

Verify tree.

Accept best branch.

Medusa multi-headK headspredict futureTree verifyin parallelAccept best2-3x speedCai et al. 2024; single model, no separate draft; adopted in TGI and TRT-LLM
Medusa multi-head.
Advertisement

How it works end to end

Cai et al. 2024.

Single model with heads.

Tree verification.