Agent and AI engineering · Practical technical guide · 2 November 2026
A 20-step agent run where each step works 95 percent of the time finishes cleanly about a third of the time. The product rule, what retries really buy, why detector recall and shared causes cap them, checkpoints, voting, cost, and pass@k versus pass^k.
Twenty steps at 95 percent
The order assistant at Belgavi's Book Store handles a return for Geetha in about twenty tool steps: find the order, read the return policy, check the photo of the damaged book, compute a refund of 1,299 rupees, issue it, note it on CASE-4417, and so on. Each step does the right thing about 95 times in 100.
The run as a whole does not. If every step must be right and the steps fail independently, the chance that all twenty are right is 0.95 multiplied by itself twenty times: 35.8 percent. Nearly two runs in three contain a wrong step. This article works through what actually moves that number, and the assumptions each fix quietly needs.
The product rule
For a serial chain whose steps fail independently, task success is the product of per-step success. With the same success p at each of N steps it is p to the power N. The curve falls faster than intuition expects.
| Per-step success | 10 steps | 20 steps | 50 steps | 100 steps |
|---|---|---|---|---|
| 99 percent | 90.4% | 81.8% | 60.5% | 36.6% |
| 95 percent | 59.9% | 35.8% | 7.7% | 0.6% |
| 90 percent | 34.9% | 12.2% | 0.5% | below 0.01% |
So 99 percent per step over 50 steps gives about 60 percent. Long tasks need per-step reliability far above what feels good in a demo, and shortening the chain, by moving work out of the model into code, is often the cheapest fix.
Retries only fix what a detector catches
The usual answer is to retry a failed step. If each attempt fails independently with probability 1 - p, and every failure is noticed, a step with r retries succeeds with probability 1 - (1 - p) to the power r + 1. One retry turns 95 percent into 99.75 percent per step, and the 20-step task from 35.8 to 95.1 percent.
Both conditions matter. A retry happens only after something decides the attempt failed: a schema check, a test, a validator, a second model, a human. Call the share of failures it catches its recall d. A failure it misses is not retried; it is passed on as a correct result, and the run ends with a silent wrong answer that looks like a success. Per step, the chance of passing becomes (1 - f) times the sum of (f d) to the power i for i from 0 to r, where f is the per-attempt failure rate, and with 80 percent recall the 20-step task tops out near 81 percent however many retries you add. Worse, each extra attempt is one more chance for an uncaught failure, so retries behind a weak detector raise the silent-wrong rate as well as the success rate.
Checkpoints, restarts and resume
When a step exhausts its retries, the run can start again. Restarting from step 1 means redoing work that already succeeded and giving it another chance to fail. Resuming from a checkpoint at the failed step does not. LangGraph's checkpointer documentation, read on 2 November 2026, describes this mechanism: a checkpoint is created at each super-step boundary, and writes from nodes that completed successfully in a failed super-step are stored so that resuming does not re-run them. Resume also needs the steps themselves to be safe to repeat, which is what idempotency keys on writes are for.
The calculator
Runnable with Python 3.10 or newer, standard library only; the comments show its output.
from math import comb
def step_round(p, retries, recall, shared=0.0):
"""One step with retries: (pass, silent wrong, exhausted) for a step that is not poisoned."""
s = 1 - p
f = s * (1 - shared) / (1 - s * shared) # independent failure rate per attempt
q = f * recall # a failure that is caught and retried
geo = sum(q ** i for i in range(retries + 1))
return (1 - f) * geo, f * (1 - recall) * geo, q ** (retries + 1)
def task_success(n, p, retries=0, recall=1.0, shared=0.0, restarts=0, checkpoint=False):
ok, _, ex = step_round(p, retries, recall, shared)
clean = (1 - (1 - p) * shared) ** n # no step hit by a shared cause
if checkpoint: # resume at the failed step
extra = sum(comb(n + m - 1, m) * ex ** m for m in range(restarts + 1))
else: # start again from step 1
again = 1 - (1 - ex) ** n
extra = sum(again ** j for j in range(restarts + 1))
return clean * ok ** n * extra
def pass_hat_k(n, c, k): # tau-bench: all k trials succeed
return comb(c, k) / comb(n, k)
def pass_at_k(n, c, k): # at least one of k trials succeeds
return 1 - comb(n - c, k) / comb(n, k)
print('99%% x 50 steps, no recovery %.1f%%' % (100 * 0.99 ** 50))
print('95%% x 20 steps, no recovery %.1f%%' % (100 * 0.95 ** 20))
print('one retry, perfect detector %.1f%%' % (100 * task_success(20, 0.95, retries=1)))
print('one retry, 80%% recall %.1f%%' % (100 * task_success(20, 0.95, 1, 0.8, restarts=2)))
print('one retry, 50%% shared cause %.1f%%' % (100 * task_success(20, 0.95, 1, 1.0, 0.5)))
print('no retry, 2 restarts %.1f%%' % (100 * task_success(20, 0.95, 0, 1.0, restarts=2)))
print('no retry, 2 resumes %.1f%%' % (100 * task_success(20, 0.95, 0, 1.0, restarts=2, checkpoint=True)))
print('8 trials, 6 pass: pass@1 %.2f pass@4 %.2f pass^4 %.2f' % (pass_at_k(8, 6, 1), pass_at_k(8, 6, 4), pass_hat_k(8, 6, 4)))
# 99% x 50 steps, no recovery 60.5%
# 95% x 20 steps, no recovery 35.8%
# one retry, perfect detector 95.1%
# one retry, 80% recall 81.1%
# one retry, 50% shared cause 59.5%
# no retry, 2 restarts 73.6%
# no retry, 2 resumes 90.5%
# 8 trials, 6 pass: pass@1 0.75 pass@4 1.00 pass^4 0.21The function is the exact closed form of the companion lab's model, so both give the same numbers for the same settings: 81.1 percent for 20 steps, 95 percent, one retry, 80 percent recall and two restarts, and 73.6 versus 90.5 percent for restart against resume with no retries.
Redundancy and voting
Running k independent attempts in parallel and keeping one a verifier accepts gives 1 - (1 - p) to the power k: the arithmetic of retries, paid in parallel, with the same need for a good verifier. Majority voting helps when errors scatter; it fails where correlated errors live, because k copies of one mistake outvote one right answer.
What recovery costs
Each retry is another model call and tool call: tokens, latency and load. Expected attempts per step are the sum of (f d) to the power i, small per step but multiplied across every step and run. The Amazon Builders Library article on timeouts and retries calls retries selfish, because a client spends more of the server's time to raise its own odds, and shows that retries nested across five layers with three retries each can multiply load on the bottom layer 243 times. An agent retrying a tool that itself retries is such a stack. Retry at one layer and give the run a budget.
Pass at k versus pass hat k
Evaluation numbers are often reported as pass@k, the chance that at least one of k attempts succeeds, from the code-generation literature (Chen and colleagues, 2021). That is the right metric when a cheap check can pick the winner. An agent that acts once on a customer's money needs the opposite: the tau-bench paper (arXiv:2406.12045, v1, 17 June 2024) defines pass^k as the chance that all k independent trials of a task succeed, estimated per task as C(c, k) / C(n, k) from c successes in n trials. Its abstract reports that even state-of-the-art function-calling agents succeed on under 50 percent of tasks and that pass^8 is below 25 percent in its retail domain. The snippet shows the gap on one task: six passes in eight trials is pass@4 of 1.00 but pass^4 of 0.21.
Running the lab
The companion lab, a model rather than a measurement, computes the exact formula and runs a seeded simulation of 2,000 tasks. With its defaults (20 steps, 95 percent, one retry, 80 percent recall, no shared cause, restarts from step 1) analytic success is 81.1 percent and simulated 81.3 percent, against 35.8 percent with no recovery, and 374 tasks end with a silent wrong result. Set retries to 0: silent wrong results fall to 314 while success falls to 67.0 percent. Set recall to 100 with no retries: 73.6 percent and 525 tasks given up; turn on the checkpoint: 90.5 percent, 199 given up and fewer step attempts per task (20.38 instead of 26.43).
Measuring it in your own runs
Log each step with its outcome, whether a check flagged it and whether it was a retry. Estimate recall by auditing a sample of passed steps, and report pass^k on a fixed task set before trusting an agent with writes.
Sources and scope
Checked against these sources on 2 November 2026. The calculator output was produced by running the snippet; the lab numbers come from running the companion model, a deterministic simulation with synthetic tasks. No agent or model was run.
- Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan: tau-bench, A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv:2406.12045, v1, 17 June 2024; defines pass^k)
- Mark Chen et al.: Evaluating Large Language Models Trained on Code (arXiv:2107.03374, 2021; pass@k and repeated sampling)
- LangGraph documentation: Checkpointers (a checkpoint at each super-step, pending writes)
- Amazon Builders Library: Timeouts, retries and backoff with jitter
Task success in a serial agent run is the product of per-step success, so 95 percent per step over 20 steps finishes cleanly about a third of the time. Retries help only with failures a detector catches and that do not share a cause, and behind a weak detector they also raise silent wrong results. Shorten chains, resume from checkpoints, and measure pass^k, not a single pass rate.