Agent Retry Policies: When to Retry a Tool, Replan the Task or Give Up

Agent and AI engineering · Practical technical guide · 2 November 2026

Transient, deterministic and semantic tool errors need different answers: retry with backoff and an idempotency key, change the arguments, or stop and hand over. Status code meaning, write safety, run-wide budgets and what the model should see.

One refund, four kinds of failure

A refund agent works through Dinky's help-desk queue at Belgavi's Book Store. For CASE-5016 it looks up Hansini's order, checks the return policy and calls issue_refund for 2,450 rupees. On a bad day that call can fail in very different ways. The payment service returns 503 for a moment. The call times out after the refund was already made. The service rejects the amount with 400 because it was sent as text. It answers 403 because refunds above 5,000 rupees need a person. Or the order lookup succeeds with an empty list.

An agent that answers all of these with "try again" will double some refunds, burn calls on requests that can never succeed and sometimes loop until its step budget stops it. An agent that never retries hands every passing blip to a human. The useful policy classifies the failure and picks the matching action.

Three classes of error

Class	Typical signal	Retrying the same request	Right action
Transient	timeout, connection reset, 408, 429, 500, 502, 503, 504	can succeed later	retry with backoff, honour Retry-After, same idempotency key
Deterministic	400, 404, 409, 422 validation and state errors	fails the same way	show the error to the model and change the arguments or plan
Permission	401, 403	fails the same way	stop and escalate
Semantic	200 with an empty, partial or wrong result	returns the same result	validate in code, replan, or escalate

The status codes carry this meaning by specification. RFC 9110 says a 4xx code means the client seems to have erred, while 503 means a temporary overload or maintenance likely to clear after a delay, optionally with a Retry-After header, and 408 allows the client to repeat the request. 409 is a conflict the user might be able to resolve and resubmit. 429 comes from RFC 6585: too many requests in a given time, which may also carry Retry-After. The Amazon Builders Library article on retries says the same in practice: client errors should not be retried with the same request because they are not going to succeed later.

Provider documentation adds the fine print. Claude's API errors page, read on 2 November 2026, says to retry 500 api_error with exponential backoff, describes 529 overloaded_error, and asks callers to resolve a 409 conflict before retrying. It also notes that a 429 caused by a monthly spend cap has no retry-after header and keeps failing until access resumes. That is a transient-looking code that is not transient, and it is the reason a policy needs a budget and not only a classifier.

Writes are where retries hurt

A timeout on a read is harmless to retry. A timeout on a write is ambiguous: the server may have applied it and the response was lost. RFC 9110 section 9.2.2 defines PUT, DELETE and safe methods as idempotent and says a client should not automatically retry a non-idempotent request unless it knows the semantics are idempotent or can detect that the original was never applied. That gives two safe options. Send an idempotency key, so a repeat returns the first result; Stripe's API reference describes this for v1 POST requests, saving the status code and body of the first request, including 500 errors, rejecting a reused key with different parameters, and allowing keys to be pruned once they are at least 24 hours old. Or, without a key, read the state first: look up refunds for the case and retry only if none exists. Generate the key once per intended action, from the case and the step, not per attempt; a new key on each attempt protects nothing.

Count every retry layer

Retries stack. The same Claude errors page says the official SDK retries connection errors, rate limits and 5xx errors twice by default with exponential backoff. If the tool wrapper retries three times and the agent loop calls the tool again after each failure, one model decision can become many requests. The Builders Library example of five layers with three retries each multiplying load 243 times is the extreme form. Pick one layer to own retries for each call, set the others to zero, and give the run as a whole a retry budget, so a long outage becomes an escalation instead of thirty identical requests. Backoff curves and jitter are covered in depth in the retry and backoff article linked below.

What the model sees and what code handles

Transient errors should be handled in code and never shown to the model: it cannot fix a 503, and showing it invites the model to improvise. Deterministic and semantic errors should be shown, briefly and precisely, because the model is the component that can change the arguments: "amount must be an integer number of paise", not a stack trace. Permission errors and spent budgets should end the run with a clear escalation note that a person can act on, including what was already done. And every replan needs its own limit; an agent allowed unlimited fixes on a 404 for an order that does not exist will search forever.

A policy function with tests

Runnable with Python 3.10 or newer, standard library only; the last comment shows its output. Add jitter to the wait in production.

from dataclasses import dataclass

TRANSIENT = {408, 429, 500, 502, 503, 504}

@dataclass
class Failure:
    status: int | None          # None: timeout or connection reset, no response read
    write: bool = False         # the call changes state (issue_refund)
    has_key: bool = False       # an idempotency key was sent
    retry_after: float | None = None
    empty: bool = False         # 200 OK, but nothing usable came back

@dataclass
class Budget:
    retries_left: int = 4       # for the whole run, not per call
    replans_left: int = 2

def decide(f: Failure, b: Budget, attempt: int, max_retries: int = 3):
    if f.empty or f.status in (400, 404, 409, 422):
        if b.replans_left == 0:
            return ('escalate', 'could not fix the request')
        b.replans_left -= 1
        return ('replan', 'show the error to the model and let it change the arguments')
    if f.status in (401, 403):
        return ('escalate', 'permission or credentials, retrying cannot help')
    if f.status is None or f.status in TRANSIENT:
        if f.write and not f.has_key:
            return ('check_status', 'unknown outcome on a write without a key: read before retrying')
        if attempt >= max_retries or b.retries_left == 0:
            return ('escalate', 'retry budget spent')
        b.retries_left -= 1
        return ('retry', 'same arguments, same key, wait %.1fs' % (f.retry_after or 2 ** attempt))
    return ('escalate', 'unclassified status %s' % f.status)

def test_policy():
    b = Budget()
    assert decide(Failure(503), b, 0)[0] == 'retry'
    assert decide(Failure(None, write=True), b, 0)[0] == 'check_status'
    assert decide(Failure(None, write=True, has_key=True), b, 0)[0] == 'retry'
    assert decide(Failure(429), b, 1)[1].endswith('2.0s')
    assert decide(Failure(429, retry_after=3), b, 0)[1].endswith('3.0s')
    assert decide(Failure(403), b, 0)[0] == 'escalate'
    assert [decide(Failure(400), b, 0)[0] for _ in range(3)] == ['replan', 'replan', 'escalate']
    assert decide(Failure(503), Budget(retries_left=0), 0)[0] == 'escalate'
    assert decide(Failure(503), Budget(), 3)[0] == 'escalate'
    print('all policy checks passed')

test_policy()
# all policy checks passed

The function returns an action and a reason, and the reason is what goes into the trace and, for replans, into the tool result the model sees. The run budget is shared across all calls, so a spend-cap 429 that never clears is retried a few times and then escalated.

Running the lab

The companion lab is a model of 40 refund runs with one scripted fault each at most; no tool or model is called. With the defaults (retry everything, mixed errors, three retries per call, no idempotency keys, replan limit 2) 34 of 40 complete, but 3 refunds are doubled, 173 calls repeat a request that could not succeed and 6 runs loop until the 30-call step budget, 374 tool calls in all. Turning on idempotency keys removes the duplicates and nothing else. The classify policy completes the same 34 runs with no duplicates and no wasted calls in 206 calls, escalates the 5 that a person must handle, and still loops on the payment outage; adding the run retry budget turns that loop into a sixth escalation in 183 calls. No retries at all is safe but completes only 23. With classify, a replan limit of 0 completes 25 runs, 1 completes 32 and 2 completes 34.

Failure modes

Retrying 4xx with the same arguments. Wastes calls, and a model shown the same error repeatedly tends to loop.
A new idempotency key per attempt. The key must identify the intended effect.
Treating 200 as success. Empty or wrong results need a check in code before the next step.
Per-call limits with no run budget. Persistent "transient" errors turn into loops.
Escalating without context. A hand-off should say which effects already happened.

Verifying it

Replay recorded failures through the policy in tests, one per class, and assert the action. In production, count retries by status code, duplicate effects found by reconciliation, identical consecutive calls, escalations by reason and runs ended by the step budget. A rising count of identical calls after 4xx errors is the clearest sign that classification is missing somewhere.

Sources and scope

Checked against these sources on 2 November 2026. Status code meanings are quoted from the RFCs and provider documentation listed below, read on 2 November 2026. The policy tests were run; the lab is a deterministic model with scripted faults, not a measurement of any agent or service.

Classify each tool failure before acting: retry transient errors with backoff in one layer, with an idempotency key or a state check on writes; show deterministic and empty-result errors to the model with a replan limit; escalate permission errors at once. Give each run a retry budget so a persistent error ends in a clear escalation instead of a loop.