Why architecture matters here
Serving cost dominates LLM economics. If you can process 3 tokens per forward pass instead of 1, your GPU serves 3× the traffic at the same quality. The architecture matters because getting speculation right requires careful tuning: draft model choice, k (lookahead depth), accept criteria, and integration with continuous batching.
Wrong choices give small or negative gains. A draft model that disagrees with the target loses more time to rejection than it saves. Excessive k thrashes the KV cache. Tree speculation without the right verifier changes semantics.
Getting it right unlocks 2-3× effective throughput. Understanding the pipeline is required.
The architecture: every piece explained
The top strip is the base loop. Prompt comes in. Draft model is a small, fast model — often the same family, smaller size (e.g. 1B alongside a 70B target). It generates Speculative tokens — the next k proposals with their probabilities. The Target model then processes the prompt + speculative tokens in a single forward pass, computing target-model logits at every position simultaneously.
The middle row is the correctness machinery. Accept-reject compares target probability to draft probability for each proposed token; accept if target agrees with a probabilistic rule that preserves the target's distribution. The longest accepted prefix is committed. Tree speculation proposes multiple candidate paths (a tree) and verifies all in one pass, increasing accepted length per pass. Medusa heads attach small prediction heads to the target model that predict the next 2-4 tokens in one pass. Self-spec uses n-gram lookup or lookahead within the target model to generate speculative candidates without a separate draft model.
The lower rows are integration and observability. Accept rate is the quality signal — high accept rate means draft matches target; low means the speculation is not paying off. Serving integration is how speculative decoding fits into vLLM, TGI, or TRT-LLM's continuous batching. Metrics track mean accepted length, effective tokens per second, and cost per token.
End-to-end flow
End-to-end: a user asks the 70B target model a question. The 1B draft model runs on the same prompt and proposes 5 tokens. The target model processes prompt + 5 candidates in one pass, computes logits at each position. Accept-reject sees the first 3 match; the 4th diverges; commit 3 tokens plus the target's own choice at position 4 = 4 tokens in one forward pass. Effective throughput 4× base. Metrics show mean accepted length 3.2 over the last minute; accept rate 0.71; effective TPS 3.4× baseline. Draft model correlation stays healthy. If accept rate drops (a distribution shift), the team retrains the draft or reduces k to save GPU time on rejects.