Speculative decoding earns its speedup by making the target model check several tokens in one forward pass instead of producing one token per pass. The usual way to produce those guesses is a second, smaller draft model — and that second model is where most of the operational pain lives. It must be trained or sourced, aligned to the same tokenizer, served, versioned, and given a slice of the same HBM your target and its KV cache already fight over. The draft-free family — Jacobi decoding, lookahead decoding, n-gram and prompt-lookup drafting — keeps the verification trick and throws the second model away. Here is how they generate candidates without any auxiliary weights, what that costs on the GPU, and why they are often the only speculation you can deploy.
The second model is an operations problem, not a math problem
On paper a draft model is cheap: a few hundred million parameters next to a target of tens of billions. In a running service it is a second first-class artifact. Its tokenizer must match the target exactly or the candidate token IDs are meaningless; it needs its own quantization decision, CUDA graphs and warmup. And when you fine-tune the target, the drafter's agreement with it silently decays, so you have two release trains that must move together.
Then there is memory. Decode-time serving is dominated by weights plus KV cache, and the drafter takes a bite of both, since it keeps its own cache over the same sequence. That bite comes out of your maximum batch size or context length — the two numbers throughput depends on. Across a fleet serving dozens of fine-tuned variants, the drafter stops being a small model and starts being a second inventory. Draft-free methods exist because that inventory is often the binding constraint, not the FLOPs.
Jacobi decoding: generation as a fixed-point iteration
Autoregressive decoding is usually described as a sequential loop, but it can be rewritten as a system of equations: each output token is a function of everything before it. Solve that system by Jacobi iteration and the loop becomes parallel. Seed a window of n future positions with arbitrary guesses, run one forward pass that predicts all of them at once, overwrite each guess with the model's prediction, and repeat. It converges to the same tokens greedy decoding would produce, in at most n iterations and usually far fewer.
The GPU appeal is the same as with any speculation: one pass over n positions reads the weights once instead of n times, turning a memory-bound matrix-vector product into a small matrix-matrix product the tensor cores can use. The catch with plain Jacobi is that a corrected guess is discarded the moment the window slides — you predict a position several times over and keep only the last answer. Lookahead decoding is what you get when you stop throwing that work away.
Lookahead decoding: keep the trajectory, mine it for n-grams
Lookahead decoding runs the Jacobi iteration over a two-dimensional window — W positions wide, N steps deep — and keeps the history. The successive guesses along a diagonal of that history form short token sequences, which are harvested into an n-gram pool: a table of “after this token, these few plausibly follow,” built entirely from the target's own intermediate predictions.
Decoding then runs two branches inside a single forward pass: the lookahead branch keeps iterating the window to generate fresh n-grams, while the verification branch looks up n-grams matching the current tail and appends them as candidates. The pool is the drafter — a hash table on the host, not weights on the device — and it improves as generation proceeds, because it is fed by the very model it drafts for.
Prompt lookup: drafting with no model and no training at all
The most stripped-down member of the family iterates nothing. Prompt-lookup decoding takes the last few generated tokens, searches the prompt and the output so far for a matching occurrence, and proposes whatever followed it there as the candidate continuation. That is the entire drafter: a substring search over a few thousand token IDs, microseconds on the host, zero device memory, zero parameters, nothing to train.
Because it has no weights, it has no failure mode beyond “no match found,” in which case you fall back to one ordinary decode step and lose almost nothing. That asymmetry — near-zero cost on a miss, a multi-token jump on a hit — is what makes it worth leaving on by default, and it is compatible with any model, tokenizer and quantization because it never touches the model.
Why input-echoing workloads make it work startlingly well
Prompt lookup looks like it should be useless, and on open-ended creative generation it largely is. It shines on the enormous class of tasks where the output is mostly a rearrangement of the input: summarization copies entities, numbers and whole clauses verbatim; retrieval-augmented answering quotes the retrieved passages; structured extraction re-emits field values character for character.
Code editing is the extreme case. Ask a model to change one line in a sixty-line function and the correct output is fifty-nine lines already in the prompt. Each is a long exact match, so the lookup proposes long candidate spans and the target accepts nearly all of them — the speedup lands exactly where users are most impatient. The flip side is that acceptance is entirely workload-dependent, so treat it as a per-endpoint flag, not a global setting.
The verification branch is what keeps the output identical
However the candidates were produced, the correctness argument belongs to the target model alone. The candidates are appended to the sequence and run through one forward pass with an attention mask that lets each see only its own ancestors. That pass yields the target's prediction at every candidate position; the longest matching prefix is committed, the first mismatch is replaced by the target's own token, and everything after it is discarded.
Two consequences follow. A bad guess is never an incorrect output, only wasted compute, which is why a zero-cost drafter like prompt lookup is safe to leave on. And you always commit at least one token per pass, so the scheme cannot lose in step count. Draft-free methods are usually deployed in their greedy, exact-match form; the sampling-preserving verification rule and the acceptance-rate arithmetic live in the speculative-decoding and transformer-math articles.
Widening the window trades memory for compute
The one tuning knob that matters is how many candidate positions you submit per pass. Widening it raises the expected tokens committed, but the costs are real and non-linear. Every candidate needs a KV entry written and, if rejected, rolled back, so a wide window inflates the transient KV footprint. Attention over a tree of candidates needs a custom mask whose size grows with the square of the candidate count, rebuilt every step.
The forward pass also stops being free. At small candidate counts it is still memory-bound and the extra rows ride along inside the same weight read — that is the entire source of the speedup. Push the window wide enough and the pass turns compute-bound, where each extra guess costs real FLOPs whether accepted or not. Lookahead's W and N sit on this curve, which is why their best values differ per model and per GPU.
What concurrency does to a drafter made of tables
Like all speculation, this pays only while the decode pass still has idle arithmetic capacity to borrow, and that window closes as the running batch grows — the crossover itself belongs to the speculative-decoding article. What is specific to draft-free methods is where the drafting cost lands. A draft model amortizes: one drafter pass covers every sequence in the batch at once. An n-gram pool does not, because it is per-sequence state, built from one conversation's trajectory and useless to any other request.
So the drafting work is host-side and scales linearly with concurrency: a pool to maintain and a lookup to run per sequence per step, on the same CPU thread that runs the scheduler. At high concurrency that bookkeeping, not the GPU, is what starts to hurt. The compensation is that switching the scheme off is instant — no resident second model, no warm graph, no memory to reclaim. You stop appending candidate rows, which makes speculation a per-step scheduling decision rather than a deployment-time commitment.
Where draft-free is the only speculation you can ship
Several common deployments make a draft model a non-starter rather than a trade-off. A single-GPU or edge deployment where the target barely fits has no room for a second set of weights and a second KV cache. A fleet serving many LoRA adapters over one base model would need a matching drafter per adapter, or degraded agreement on all of them. And a checkpoint with an unusual tokenizer may simply have no suitable small sibling.
In all of those, the draft-free family gives a real fraction of the speculative win with an operational footprint near zero: a host-side table or a substring search, a wider forward pass, one artifact to deploy. It also composes cleanly with the rest of the stack — paged KV cache, quantized weights, tensor parallelism and continuous batching keep working, because nothing changed for them except that some steps submit several rows instead of one.
Honest limits, and when to reach for a drafter anyway
Draft-free schemes do not match a well-matched drafter on open-ended generation. A trained drafter has learned the target's distribution and can propose continuations of text that appears nowhere in the context; an n-gram pool cannot invent what it has never seen, and prompt lookup falls back to ordinary decoding whenever the output stops echoing the input. If your traffic is chat or long-form writing, expect modest gains and measure first.
There are also implementation costs people underestimate: the tree attention mask, the rollback path in a paged KV allocator, and the data-dependent shapes that make static graph capture harder. Draft-model speculation pays those too, so they are not an argument against draft-free specifically — but they mean “no second model” is not “no work.” This family removes the inventory cost of speculation, not its engineering cost.