Retry, replan or give up: error policies for a refund agent

What this experiment demonstrates

A refund agent on Dinky's help-desk queue at Belgavi's Book Store works 40 cases, CASE-5001 to CASE-5040, for customers Jenya, Tanishka, Geetha and Hansini. Each run calls lookup_order, check_return_policy, issue_refund and close_case. The error mix scripts at most one fault per run and a fixed seed decides which run gets which, so the same settings always give the same runs. A timeout on issue_refund happens after the refund was made, so retrying it without an idempotency key pays twice. Every run stops at 30 tool calls.

The four policies: retry everything with the same arguments, then let the model fix arguments where it can and otherwise call again; classify, which retries only transient errors, fixes arguments for 400, 404 and empty results, escalates 403 and reads the refund status before retrying a keyless write; classify plus a run retry budget of 4; and no retries at all. It is a model with scripted faults; nothing is called.

Run controlled comparisons

Start with the defaults: retry everything, mixed errors, three retries per call, no idempotency keys, replan limit 2. 34 of 40 refunds complete, 3 are paid twice, 173 calls repeat a request that could not succeed, and 6 runs loop until the step budget, 374 tool calls in all.

Now change one thing at a time. Turn on idempotency keys: duplicates go to 0 and nothing else changes. Set max retries to 0 or 6: wasted calls become 140 or 206. Switch to classify: 34 complete, 0 duplicates, 0 wasted calls, 5 escalated, 1 loop on the payment outage, 206 calls. Switch to classify plus a run budget: the loop becomes a sixth escalation, 183 calls. Switch to no retries: safe, but only 23 complete and 17 escalate. With classify, set the replan limit to 0, 1 and 2: 25, 32 and 34 complete. Choose the payment outage mix: retry everything and classify both loop on all 10 outage runs (433 calls for classify), while the run budget escalates them in 203 calls. Choose the deterministic mix: retry everything wastes 321 calls and loops 9 times, classify with a budget wastes none and escalates 9.

Inspect the trace and keep evidence

The table lists the runs that met a fault, with the refund amount, the scripted fault, the number of calls and what happened. Red marks a duplicate refund or a run that looped until the step budget; amber marks an escalation or a run that completed after repeating a request that could not succeed. Start trace, Next step, Show all and the slider control how much of the trace is visible. Reset restores the defaults and Export JSON downloads the inputs, metrics and trace.

Limits and production translation

Real tools fail in combinations, recover after unpredictable delays, and return error bodies that need parsing; the model has one fault per run, fixed recovery points and no time. It also treats each argument fix as always correct when it is possible. What carries over: retry transient errors in one layer only, with backoff and a key; show deterministic errors to the model with a replan limit; escalate permissions at once; check results that look successful; and give each run a retry budget so a long outage ends in an escalation and not a loop.