Why architecture matters here
The architecture matters because tail latency does not average out — it compounds. An agent step that calls a tool which calls three backends is only as fast as the slowest backend; if each has a 1% chance of a slow response, the step has nearly a 3% chance of being slow, and a multi-step agent chains those probabilities into a near-certainty of at least one straggler per session. Users experience the p99 of the whole chain, not the median of any single call. Shaving the tail of each underlying call is therefore the highest-leverage latency work available, and hedging is one of the few techniques that does it without requiring the backends themselves to get faster.
Hedging works because it exploits independence. The reasons a particular request is slow — a GC pause, a hot lock, a queue that briefly backed up on one node — are mostly uncorrelated across replicas. A backup sent to a different node samples a fresh draw from the latency distribution, and the minimum of two independent draws has a far shorter tail than either alone. This is the same reason two independent dice rarely both roll high: the odds that both the primary and the backup are slow at once are the product of two small numbers, which is tiny.
The reason it needs careful architecture rather than a one-line retry is that the naive version is actively harmful. Fire a backup for every request and you have doubled your fleet's load; under any contention that pushes queueing latency up, which makes more requests miss the hedge delay, which fires more backups — a positive feedback loop that turns a mild slowdown into an outage. The delay timer and the budget exist precisely to break that loop: the timer ensures backups only fire for genuine stragglers, and the budget ensures that even a storm of stragglers cannot spend more than a few percent extra capacity.
It helps to put numbers on the payoff, because the effect is larger than intuition suggests. Suppose a backend answers in 5 ms at the median but 200 ms at its p99 — a heavy tail from occasional GC and queueing. A single call's p99 is 200 ms by definition. Now hedge with a delay of 40 ms: the roughly one percent of calls that would have taken 200 ms instead fire a backup at 40 ms, and the backup — an independent draw from a healthy node — almost always finishes within a few more milliseconds, so the observed p99 collapses toward 50 ms. A single cheap knob has cut the tail by a factor of four, and it has done so for the price of a backup on only that one percent of requests. Stack that improvement across the several backends a single agent step touches and across the several steps of a session, and the compounding works in your favour for once: each hedged call trims its own tail, so the end-to-end p99 the user actually experiences improves far more than any one call's numbers suggest. This leverage — large tail reduction for a few percent of extra load — is why hedging earns a place in latency-critical paths despite the machinery it demands.
There is a subtler architectural payoff, too: hedging makes tail latency a tunable property rather than a fixed one. By moving the hedge delay you trade extra load for tail reduction along a smooth curve — a shorter delay fires more backups and cuts the tail harder at higher cost, a longer delay is cheaper but rescues fewer stragglers. That single knob, driven from live percentile measurements, lets an operator dial the latency/cost trade-off to the moment's needs instead of over-provisioning the whole fleet to make the worst node fast. Understanding the coordinator is what turns that knob from a guess into a control.
The architecture: every piece explained
The hedge coordinator is the orchestrating component. It receives the caller's request, issues the primary attempt to a replica immediately, and arms a timer. If the primary responds before the timer fires — the common case — the coordinator returns that answer and disarms everything. If the timer fires first, the coordinator consults the budget and, if allowed, issues a backup attempt to a different replica. It then races the two and returns whichever completes first, cancelling the loser.
The hedge delay is the timer's duration and the single most important tuning parameter. Set it too short and you hedge almost everything, blowing the budget and adding load; set it too long and stragglers finish on their own before the backup helps. The right value tracks the service's own latency distribution — commonly near the p95 — so that roughly the slowest five percent of requests trigger a hedge. Because latency drifts with load and deploys, mature implementations compute the delay from a rolling percentile of recent responses rather than hard-coding it.
The hedge budget is a rate limiter on backups, usually expressed as a maximum fraction of total requests — say five percent — that may spawn a hedge, often implemented as a token bucket. When the bucket is empty, the timer firing is ignored and the request simply waits on its primary. This is the safety valve that keeps hedging from amplifying an overload: no matter how many stragglers appear, extra load is capped. Cancellation is the coordinator's other duty — when one attempt wins, the other must be actively cancelled so the losing replica stops working and frees its resources, and that cancellation must propagate down through the call so a connection or query is genuinely aborted, not just abandoned.
The idempotency precondition is what makes the whole scheme legal. Because a hedged operation may execute on two replicas — and both may partly run before one is cancelled — the operation must be safe to perform more than once. Reads are naturally idempotent; writes require an idempotency key or a conditional operation so a duplicated request does not double a charge or insert a row twice. This precondition is not optional plumbing; it is the load-bearing assumption of hedging, and the coordinator should refuse to hedge any call not marked idempotent. Together these four pieces — delay, budget, cancellation, and the idempotency guarantee — are what separate a hedging layer that reliably trims the tail from one that silently corrupts data or melts the fleet under pressure.
End-to-end flow
Trace a hedged tool call inside an agent. The agent step invokes a downstream lookup that has been wrapped by the hedge coordinator. At t=0 the coordinator issues the primary request to replica A and arms a timer set to the current rolling p95, say 40 ms. The overwhelming majority of the time, replica A answers in 8 ms; the coordinator returns that response, cancels the timer, and no backup ever exists. Hedging has cost nothing on the fast path.
Occasionally replica A lands on a node mid-GC and goes quiet. At 40 ms the timer fires. The coordinator checks the budget's token bucket; a token is available, so it spends it and issues a backup to replica B. Now both attempts are in flight. Replica B, on a healthy node, answers in 9 ms — so at t=49 ms the coordinator has an answer, versus the hundreds of milliseconds replica A would have taken. It returns B's response to the agent and immediately cancels the still-pending A, which propagates as a cancelled RPC so A's node stops the query and reclaims the thread.
Because the operation was idempotent, running partly on both replicas was harmless — a read touched no state, or a write carried an idempotency key so B's completion and A's aborted attempt could not both take effect. The caller saw a fast answer; the fleet spent one extra request out of its small budget. Had the budget been exhausted — a burst of stragglers all firing at once — the timer firing would simply have been ignored, and the request would have waited on A, trading a slower tail for a hard cap on added load.
Watch the feedback the coordinator gathers along the way. Every completed attempt feeds the rolling percentile that sets the next delay, so as the service warms up or a deploy shifts latency, the hedge point tracks it automatically. Every budget consultation feeds a metric — hedge rate, win rate of backups, budget-exhaustion count — that tells an operator whether hedging is helping (backups winning a meaningful share of the time) or merely adding load (backups rarely winning, meaning the delay is too short). The end-to-end flow is thus a small control loop: measure the distribution, hedge the tail within budget, cancel the loser, and feed the outcome back into the next delay — which is what keeps hedging self-correcting instead of a static setting that rots as the system changes.