Reliability math for multi-step agents: retries, detectors and checkpoints

What this experiment demonstrates

An order-handling task at Belgavi's Book Store is a chain of steps that must all be right. Each step attempt fails with a fixed probability. A detector catches each failed attempt with a chosen recall; a caught failure is retried, a missed one is passed on and the task ends with a silent wrong result. A chosen share of failures has a shared cause and repeats on every attempt. When a step runs out of retries, the task gets two restarts, from step 1 or, with a checkpoint, from the failed step.

The analytic figure is the exact closed form of this model. The simulated figure runs 2,000 tasks with a fixed seed, so it differs from the formula only by sampling noise. It is a model with synthetic tasks; no agent or model is called.

Run controlled comparisons

Start with the defaults: 20 steps at 95 percent, one retry, 80 percent recall, no shared cause, no checkpoint. Analytic success is 81.1 percent and simulated 81.3 percent, against 35.8 percent with no recovery at all, and 374 of 2,000 tasks return a silent wrong result.

Now change one thing at a time. Set retries to 0: success falls to 67.0 percent, and silent wrong results fall to 314, because every extra attempt was another chance for a missed failure. Set retries to 3: success stays at 81.1 percent; past the first retry the detector, not the retry count, is the limit. Set recall to 50: 59.5 percent with one retry or three. Set recall to 100 and retries to 0: 73.6 percent with 525 tasks given up; turn on the checkpoint and it is 90.5 percent with 199 given up and 20.38 step attempts per task instead of 26.43. With recall 100, set shared-cause failures to 100 and retries to 3: success is 35.8 percent, exactly the no-recovery figure, after 29.35 attempts per task. Finally set 50 steps at 99 percent with recall 0: 60.5 percent, the familiar 99 percent over 50 steps.

Inspect the trace and keep evidence

The table shows analytic success by chain length for the current settings, with restarts from step 1 and with resume from a checkpoint side by side. The trace states the per-step split between shared-cause and independent failures, follows the first simulated task that needed recovery, and ends with the simulated counts. Start trace, Next step, Show all and the slider control how much of the trace is visible; the metrics always describe the whole run. Reset restores the defaults and Export JSON downloads the inputs, metrics and trace.

Limits and production translation

Real steps differ in reliability, real detectors also reject correct results, failures cluster by task, and restarts cost time and tokens that the model counts only as attempts. Treat the numbers as a way to reason, not as a forecast. What carries over: task success is a product of step success, retries only recover failures something detects, shared causes need a different action and not another attempt, and resuming from a checkpoint beats starting again. Measure per-step success, detector recall and pass^k on your own traces before relying on any of these.