Why architecture matters here
The reason speculation is architectural rather than incidental is that it changes the agent's critical path. In a naive agent, the critical path is strictly serial: generate, wait for the tool, generate, wait for the tool. Every tool latency is added, in full, to the user-perceived response time. On a five-step task with an average 600-millisecond tool round-trip, that is three seconds of pure waiting stacked on top of generation. Speculation attacks this by overlapping the tool round-trip with the generation that precedes the tool call — work the model is doing regardless. When it works, the tool latency is hidden entirely, and the task completes in close to the sum of the generation times alone. That is a structural win no amount of prompt tuning can deliver.
The second reason it is load-bearing is that the value is entirely a function of two rates that the architecture must be built to observe and control: the hit rate of the predictor and the side-effect safety of the tools. A predictor that is right 70% of the time on read-only tools converts most tool latency into free time; a predictor that is right 20% of the time mostly wastes compute and may even slow the system down through contention. Because the payoff swings so sharply with hit rate, the system cannot treat speculation as fire-and-forget — it has to measure prediction accuracy per tool, gate speculation on tools where accuracy clears a threshold, and back off when accuracy drops. The architecture is, in large part, the machinery for measuring and acting on those rates.
The third reason is correctness under mispredictions, which is where a careless design turns a latency optimization into a bug factory. The guarantee that must hold is invariant: a speculative agent produces exactly the same observable actions and results as a non-speculative one. That means a mispredicted speculative call must leave no trace — no rows written, no emails sent, no counters incremented, no state the real execution can observe. Achieving that requires the speculative executor to run in a sandbox with no write access to the real world, and it requires a hard classification of which tools are even eligible. The correctness argument, not the speedup, is what dictates most of the boxes in the diagram.
Finally, speculation matters because it interacts with cost, and cost is a first-class production constraint for agents. Every speculative execution that gets discarded is wasted compute — wasted tokens if the speculator is an LLM, wasted API calls if the tool bills per invocation, wasted sandbox time. A design that speculates indiscriminately can easily double the operational cost of an agent while delivering a modest latency win. The architecture therefore has to include a cost budget and a policy that only speculates when the expected latency saving, weighted by hit rate, justifies the expected wasted cost. Speculation is a bet, and the system is the bookmaker that decides which bets are worth placing.
The architecture: every piece explained
Top row: prediction. As the LLM step streams tokens, it is often obvious well before the call is complete what tool is coming — the model has emitted the tool name and the first few arguments, or the conversation context makes one call overwhelmingly likely. The speculator is the component that turns that partial signal into a concrete guess: a predicted tool plus predicted arguments. It can be a cheap heuristic (if the last user message asked a factual question, predict a search with the extracted query), a small distilled model trained on the agent's own traces, or the streaming prefix of the main model's own tool call. Its output is dispatched to the sandbox executor, which runs the predicted tool in an isolated environment that can read but cannot mutate real state.
Middle row: verification. When the main model finalizes its tool call, that confirmed call — the tool and arguments the agent actually emitted — is compared against the speculation in the match step. Matching is not always exact-string: for a search tool you might match on a normalized query, for a retrieval tool on the same document id, tolerating cosmetic differences in whitespace or ordering that don't change the result. If the confirmed call matches a speculation whose result is already sitting in the result cache — keyed by a hash of the tool name and normalized arguments — the expensive work is already done. The cache is what carries a speculative result forward to the moment it is needed.
Bottom rows: the two outcomes. On a match, the commit path reuses the cached speculative result directly, so the agent's tool call returns with essentially zero added latency — the round-trip happened during generation. On a miss, the rollback path discards the speculative result (which, because it ran in the sandbox, had no external effect) and runs the real tool call normally, paying the ordinary latency. Crucially, a miss is never slower than not speculating at all, apart from the wasted background compute. The ops strip names the guarantees that make this safe: a read-only guard that only lets side-effect-free tools speculate, hit-rate tracking to tune the predictor, idempotency keys so that even borderline tools cannot double-execute, and a cost budget bounding the wasted work.
The read-only guard deserves emphasis because it is the load-bearing safety mechanism. Every tool the agent can call is classified up front into one of two buckets: speculatable (pure reads — search, retrieval, lookups, computations with no external effect) and non-speculatable (anything that writes, sends, charges, or otherwise changes the world). The speculator is physically prevented from dispatching a non-speculatable tool; the sandbox executor refuses them even if asked. This binary classification is what lets the correctness argument be simple: because no side-effecting tool ever runs on speculation, a misprediction can never produce an unwanted action — it can only waste compute, and wasted compute is a cost problem, not a correctness problem. Getting this classification right, and keeping it right as tools are added, is the single most important operational responsibility in the whole design.
End-to-end flow
Trace one speculated tool call from the moment generation begins. The agent is answering a research question and the main model starts a turn. Within the first dozen tokens it has streamed the beginning of a tool call — the name web_search and the start of a query argument. The speculator, watching the stream, completes the guess: it predicts web_search(query="2026 EV battery density benchmarks") based on the partial tokens and the user's last message.
Speculative dispatch. The speculator checks the tool classification: web_search is read-only, so it is eligible. It dispatches the predicted call to the sandbox executor, which fires the actual HTTP search in the background — reads are safe — and, roughly 500 milliseconds later, deposits the search results into the result cache under the key hash("web_search", normalized_query). All of this happens while the main model is still generating the rest of its tool call and any reasoning around it. The user is waiting on the model's generation, not on the search.
Verification and commit. The main model finalizes its call: web_search(query="2026 EV battery density benchmarks"). The match step normalizes both the predicted and actual queries, hashes them, and finds a cache hit. The confirmed call is served the cached result instantly — the 500-millisecond search round-trip has been completely absorbed into the generation that was going to happen anyway. The agent proceeds to reason over the results with zero perceptible wait for the tool. This is the good case, and on a well-tuned agent with a good predictor it is the common case.
The miss case, for contrast. Suppose instead the model finalized web_search(query="solid-state battery energy density 2026") — a semantically similar but textually different query that normalizes to a different hash. The match step reports a miss. The rollback path discards the cached speculative result (harmless — it was a read that touched nothing) and runs the real search now, paying the full 500 milliseconds. The user experiences exactly the latency they would have without speculation; the only loss is the wasted background search. The system records the miss so the hit-rate tracker can decide whether the predictor is still worth trusting for this tool. Notice what did not happen in either branch: no external state changed on a wrong guess, no action was taken that the agent didn't ask for, and the worst outcome was a bounded amount of wasted compute. That invariant — same actions, same results, only faster on a hit — is the property the entire architecture exists to preserve.