Speculative Decoding: Faster Generation Without Changing the Model

Why decoding leaves compute on the table

A decode step for one sequence multiplies a single token's activations by every weight matrix in the model. For a model with billions of parameters that means streaming gigabytes of weights through the memory system to perform a comparatively tiny number of multiply-accumulates. Arithmetic intensity, the ratio of FLOPs to bytes moved, is very low, so the step time is set by memory bandwidth. Batching raises arithmetic intensity because the same weights are reused for every sequence in the batch, which is why continuous batching is the first optimization every serving engine applies.

Speculative decoding attacks the same inefficiency from a different direction. Instead of reusing weights across more sequences, it reuses them across more positions of the same sequence. Scoring k+1 positions in one pass costs little more than scoring one position while the step is still bandwidth bound, because the weights are read once either way. The catch is that those extra positions are only useful if the tokens placed in them are the tokens the target model would have produced, and that is what the proposer and the acceptance rule are for.

This predicts where the technique works: interactive traffic at low concurrency, where the GPU is underutilized. It helps least, and can hurt, on large batches, where every speculative position consumes arithmetic that could have served another request.

The acceptance rule

Each round, the proposer produces k draft tokens x1..xk along with the probabilities q(x) it assigned to them. The target model runs once over the prompt plus all k drafts and returns its own distribution p at every one of the k+1 positions. Draft tokens are then examined left to right. Token xi is accepted with probability min(1, p(xi)/q(xi)). If it is accepted the next token is examined; at the first rejection the process stops, and a replacement token is sampled from the residual distribution proportional to max(0, p - q). If all k drafts are accepted, the target's distribution at position k+1 is already available, so one bonus token is sampled from it for free.

This scheme, introduced independently by Leviathan et al. and Chen et al. in 2023, guarantees that every committed token is distributed exactly as if it had been sampled from the target model alone. The draft only influences how many tokens are committed per round, never which distribution they come from. Under greedy decoding the rule collapses to something simpler: a draft token is accepted if it equals the target's argmax at that position, and the first mismatch is replaced by the target's argmax.

import numpy as np

def verify(draft_tokens, q_probs, p_probs, rng):
    """draft_tokens: k ints; q_probs: k x V draft dists; p_probs: (k+1) x V target dists."""
    out = []
    for i, tok in enumerate(draft_tokens):
        p, q = p_probs[i][tok], q_probs[i][tok]
        if rng.random() < min(1.0, p / q):
            out.append(tok)                      # accept and continue
            continue
        residual = np.maximum(p_probs[i] - q_probs[i], 0.0)
        residual /= residual.sum()
        out.append(rng.choice(len(residual), p=residual))
        return out                               # stop at first rejection
    out.append(rng.choice(p_probs[-1].shape[0], p=p_probs[-1]))  # bonus token
    return out

Real engines implement this on the GPU in batched form, but the control flow is the same, and it is worth keeping a reference implementation like this one in the test suite. A distributional test that samples many continuations with and without speculation and compares token histograms is the cheapest way to catch a bug that silently changes model behavior.

Predicting the speedup

Under the simplifying assumption that each draft token is accepted independently with probability a, the expected number of tokens committed per round with draft length k is (1 - a^(k+1)) / (1 - a). With a = 0.8 and k = 4 that is about 3.36 tokens per target pass; with a = 0.6 it falls to about 2.31. The formula makes two things obvious. Returns diminish quickly with k, because later drafts only count if every earlier draft survived. And acceptance rate matters far more than draft length: raising a from 0.6 to 0.8 is worth more than doubling k.

Tokens per round is not the speedup, because drafting costs time. If one proposer step costs a fraction c of a target step, a round costs roughly k*c + 1 target-step equivalents, and the wall-clock improvement is the expected tokens per round divided by that cost. A draft model one tenth the cost of the target (c = 0.1) with a = 0.8 and k = 4 gives about 3.36 / 1.4, or roughly 2.4 times. The same draft with a = 0.5 gives about 1.94 / 1.4, or roughly 1.4 times, and with k = 8 at that acceptance rate the extra drafting eats most of the gain. These numbers are illustrative; measured speedups also depend on kernel efficiency, sampling overhead and scheduler behavior, so the formula is a planning tool, not a promise.

def expected_tokens(a, k):
    return (1 - a ** (k + 1)) / (1 - a)

def speedup(a, k, c):
    return expected_tokens(a, k) / (k * c + 1)

for a in (0.5, 0.6, 0.7, 0.8, 0.9):
    best_k = max(range(1, 11), key=lambda k: speedup(a, k, c=0.1))
    print(a, best_k, round(speedup(a, best_k, 0.1), 2))

Run this sweep against acceptance rates measured on your own traffic to choose a starting k. It also shows why one global k is usually wrong: code completion supports a longer draft than high-temperature chat.

Choosing a proposer

The classic proposer is a smaller model from the same family. It needs no training, but it is a second model to load, version and schedule, with its own KV cache competing for memory. It must share the target's tokenizer, or its token ids are meaningless to the target.

Head-based proposers remove the separate model. Medusa adds several extra decoding heads to the target that each predict a token further ahead, and verifies a tree of candidate continuations with a special attention mask. EAGLE trains a lightweight autoregressive head that drafts at the level of the target's hidden features rather than tokens, which tends to give higher acceptance than an independent small model, and later versions build the draft tree dynamically based on confidence. Some recent models ship with multi-token prediction modules trained alongside the main model that can serve the same drafting role. All of these need training data and must be retrained or at least revalidated whenever the target weights change.

The cheapest proposer needs no model at all. Prompt lookup, or n-gram decoding, finds the most recent occurrence of the last few generated tokens in the context and proposes whatever followed it. For workloads that copy from the input, such as retrieval-augmented answers, code editing and extraction, acceptance can be high at almost zero draft cost. For free-form generation it proposes little, which is harmless: an empty draft degrades to normal decoding.

Integration with the serving engine

Inside a continuous-batching engine, speculation changes the unit of work from one token per sequence per step to a variable number. The scheduler must reserve KV blocks for up to k+1 new positions per sequence before the verification pass, then release the blocks for rejected positions afterwards. Engines with paged KV caches handle this reasonably well, because rollback is a matter of truncating a sequence's logical length and returning whole or partial blocks to the pool, but the reservation still reduces how many sequences fit in memory at once.

A draft model adds its own KV cache, which must stay synchronized with the committed sequence. After a rejection at position i, both caches must forget everything past i and the draft must resume from the corrected token. Getting this wrong does not crash anything; it just lowers acceptance, because the draft is conditioning on tokens that were never committed. That makes it a classic silent regression, and a good reason to alert on acceptance rate rather than only on latency.

Constraints must be applied consistently. If the target samples under a JSON grammar, the draft must be masked by the same grammar, or it proposes tokens the target can never accept. And when an accepted prefix contains an end-of-sequence token or stop string, everything after it must be discarded before streaming.

When speculation stops paying

The same bandwidth argument that makes speculation attractive at low load works against it at high load. As batch size grows, each target step does more useful arithmetic, and verifying k extra positions per sequence multiplies that arithmetic. Rejected positions are pure waste. At some batch size, which depends on the model, hardware, proposer and acceptance rate, total throughput with speculation drops below throughput without it, even though individual requests may still see lower latency. Because the crossover moves with traffic mix, a static configuration that was a win in a benchmark can be a loss during the daily peak.

Production systems therefore treat speculation as a load-dependent policy. A common approach is to shrink k as the number of running sequences rises and disable drafting entirely above a threshold, then re-enable it as load falls. The controller can also be per request: a sequence whose recent acceptance has been poor gets a shorter draft, while one copying from its context gets a longer one.

class DraftLengthController:
    def __init__(self, k_max=6, off_above_batch=48, alpha=0.1):
        self.k_max, self.off_above = k_max, off_above_batch
        self.alpha, self.acc = alpha, 0.7   # EWMA of per-token acceptance

    def observe(self, proposed, accepted):
        if proposed:
            self.acc = (1 - self.alpha) * self.acc + self.alpha * accepted / proposed

    def k(self, running_batch, draft_cost=0.1):
        if running_batch >= self.off_above:
            return 0                      # compute bound: plain decoding
        a = min(self.acc, 0.95)
        return max(range(0, self.k_max + 1),
                   key=lambda k: (1 - a ** (k + 1)) / (1 - a) / (k * draft_cost + 1))

The thresholds in a controller like this must come from load tests on the actual deployment, not from the formula alone. Record throughput and tail latency across a sweep of batch sizes with speculation on and off, find the crossover, and set the cut-off below it with some margin.

Failure modes

• Draft drift after a target update. Re-tuning or re-quantizing the target without refreshing the draft lowers acceptance. Output stays correct, so only the acceptance metric reveals it.
• Template mismatch. A draft that formats special tokens differently is rejected immediately, making every round pure overhead.
• Memory pressure. Draft weights, draft cache and reserved verification slots reduce batch capacity and can raise queueing delay past the per-token gain.
• Tail latency from long drafts. A long draft rejected at its first token costs a full round for one token; per-request adaptive k contains this.
• Non-bitwise reproducibility. The distribution is preserved, but changed batch shapes and kernels can flip floating-point ties, so baselines must not assume bit-identical text.
• Logit-processor gaps. Penalties or processors applied to the target but not the draft break the q distribution the acceptance rule relies on.