Core concept
Draft model much smaller (7B vs 70B). Parallelizes autoregressive step.
Advertisement
How it works
Draft quality matters. Poor drafts = low acceptance = wasted compute.
Advertisement
Trade-offs + gotchas
Latency-sensitive LLM inference. Standard in vLLM + TRT-LLM.