Why architecture matters here
The reason recovery deserves its own architecture is that the naive alternatives are actively harmful. 'Wrap the tool in a try/except and retry three times' seems reasonable until the dependency is down: every agent instance retries in lockstep, the retries pile onto an already-struggling service, and the outage deepens — a retry storm that turns a blip into an incident. Worse, if the tool performed a write, a retry after a timeout can execute the operation twice, because the timeout told you nothing about whether the server processed the request. And 'let the model figure it out' fails the other way: feeding a rate-limit error back to the model wastes an expensive completion re-deciding something a fixed backoff would have handled for free.
Three properties define good recovery. It is classified: every failure is mapped to a class — transient, invalid-input, permanent, or unknown — because the correct action is a function of the class, not the fact of failure. It is bounded: total attempts, total time, and total cost per tool call are capped, so no failure mode can run forever or spend without limit. It is safe under replay: any retry of a state-changing operation carries an idempotency key so the dependency can dedupe, making 'retry' a decision you can take without fearing double effects.
The trade-off is that recovery adds latency and complexity to the happy path's shadow. Every retry delays the answer; every repair round costs a model call; a circuit breaker that trips too eagerly turns a recoverable blip into a user-visible failure, and one that trips too late lets an agent hammer a corpse. Tuning these — backoff curves, breaker thresholds, repair limits — is the real operational work, and it depends on knowing which class of failure you are actually seeing, which is why classification sits at the center of the diagram.
The architecture: every piece explained
Top row: the classification pipeline. A tool call — arguments emitted by the model — is executed against an API, local function, or MCP server. Its outcome flows into the error classifier, the linchpin. The classifier maps the outcome to a class using status codes, exception types, and error bodies: transient (429, 503, connection reset, timeout on an idempotent call), invalid-input (400, 422, schema violation — the arguments are wrong), permanent (401/403 auth, 404 for a truly-missing resource, business rule rejection), or unknown (an unrecognized error, treated conservatively). Alongside sits the idempotency key: for any call that mutates state, the agent generates a stable key per logical operation so retries are deduplicated by the dependency.
Middle row: the four responses, one per class. Retry with backoff handles transient errors — exponential backoff with jitter, honoring Retry-After when present, capped by the attempt budget. Repair loop handles invalid-input: the error message is fed back to the model as the tool result ('date must be ISO-8601; you sent 03/14/2026'), and the model re-emits corrected arguments — a fundamentally different action from retrying the same call, because the inputs change. Fallback tool handles the case where the primary route is unavailable but an alternate exists (a secondary provider, a cached answer, a degraded computation). Surface to agent handles permanent failures and exhausted budgets: return a structured error the agent can reason about ('refund failed: account frozen') so it can explain, escalate, or choose another plan — giving up gracefully is a valid outcome.
Bottom rows: the guardrails. The circuit breaker tracks failures per dependency; after a threshold it opens, short-circuiting further calls to that tool with an immediate failure (protecting the dead service and the agent's latency) and periodically half-opens to test recovery. The attempt budget caps total tries and wall-clock time per logical tool call across all retry and repair rounds, so no single call can loop forever. The ops strip is the control panel: retry rate per tool, repair success rate, breaker state transitions, and cost-per-resolved-call — the number that tells you whether recovery is saving you or quietly bankrupting you.
End-to-end flow
Trace an agent booking travel through a flight-search tool and a non-idempotent payment tool. The model calls search_flights(origin='SFO', date='03/14/2026'). The API returns 422: date must be ISO-8601. The classifier tags this invalid-input — retrying is pointless. The recovery layer enters the repair loop: it returns the error text to the model as the tool result. The model re-emits search_flights(origin='SFO', date='2026-03-14'). This time the API is briefly overloaded and returns 503 with Retry-After: 2. The classifier tags it transient; the layer waits two seconds plus jitter and retries the identical call (safe — search is idempotent). It succeeds. Two failures, two different responses, because they were two different classes.
Now the payment. The model calls charge_card(amount=482.00, token=...). The recovery layer generates an idempotency key for this logical charge and passes it in the request header. The call times out after 10 seconds — the classic trap. The classifier tags it transient, but this is a write, so a naive retry risks double-charging. Because the idempotency key is present, the retry is safe: the layer re-sends with the same key; the payment provider recognizes the key, sees it already processed the first request, and returns the original success instead of charging again. The customer is charged once. Without the key, the correct behavior would have been to not retry and instead surface the ambiguous state for reconciliation.
Consider the dependency going fully down. The flight API starts returning 500s on every call. The first few trip the retry-with-backoff path, but the circuit breaker is counting failures; after five in a row it opens. The next call to search_flights fails instantly with 'circuit open' — no network round trip, no backoff wait — and the recovery layer routes to the fallback: a secondary flight-data provider with slightly staler prices. The agent completes the task degraded rather than hung. Thirty seconds later the breaker half-opens, lets one probe through; it succeeds, the breaker closes, and normal routing resumes. Throughout, the attempt budget guaranteed that even in the worst case, this one booking could not consume more than, say, eight tool attempts and forty seconds before surfacing a clear failure to the user — the agent never spins.