Design idempotent agent actions and explicit compensation

Pattern 11 companion: Python example · Behavioral tests · Example license

An incident command can outlive its response

An incident-response assistant requests a temporary maintenance hold on a failing service. The storage operation commits, but the response disappears. The assistant now faces an ambiguity: did the hold happen? Repeating the operation with a fresh identifier treats uncertainty as a new instruction. This original design assigns a stable operation identifier to the authorized intent and retains its receipt. A retry asks about that intent rather than inventing another one.

The system must distinguish transport failure from business failure. A timeout is evidence about communication, not proof that the tool did nothing. Define success as a recorded hold associated with the intended tenant, service, and incident. The assistant's final sentence is merely a report of that state. The model adapter can propose the operation; a deterministic gateway checks authorization and manages execution.

Architecture and request sequence

Component diagram for Design idempotent agent actions and explicit compensation — Original reference architecture by Sandeep Belgavi. Arrows show the responsibilities and transitions described below.

Request sequence for Design idempotent agent actions and explicit compensation — Illustrative sequence for this design; failure and recovery variants are explained in the article.

Define the identity and conflict contract

Our key is the pair of tenant and operation identifier. The stored intent contains the service targeted by the hold. The same key with the same intent returns its existing state; the same key with a different service fails. Include every effect-bearing argument in a production intent fingerprint, including operation type and relevant policy version. Do not use mutable prose as the identity of an action. Nor should a model choose a new key each time a tool reports a timeout.

Receipt retention is part of the contract. If old identifiers expire while delayed retries remain possible, a previously committed action can be treated as new. Choose retention from the retry horizon and business reconciliation requirements. Separately bound incoming key length and avoid embedding customer secrets. These are original application decisions, not properties supplied by the word idempotent.

Place the atomic boundary around the actual effect

The teaching implementation puts a hold row and its action receipt in one SQLite transaction. That makes the local demonstration precise: either both rows commit, or neither does. It does not create an atomic transaction spanning a database and an external maintenance API. A real remote tool needs its own idempotency support or an explicit reconciliation state when an acknowledgment is lost.

For a proposed distributed version, persist intent before dispatch, record attempts, and reconcile uncertain outcomes using the provider's operation lookup. Keep unknown separate from failed. A worker must not compensate an operation that might never have happened without understanding the tool's compensation semantics. The architecture diagram is a conceptual reference; only the local store behavior is implemented in the accompanying file.

Compensation is a new business action

If later incident analysis concludes that the maintenance hold was unnecessary, release it and retain the original receipt as compensated. That is a new action with its own authorization and observable result. It does not erase the fact that the hold existed. A repeated compensation should remain safe; a delayed retry of the original operation should return compensated rather than silently recreating the hold.

Some effects cannot be reversed. A notification has already been read; a destructive command has already removed data. For those operations, compensation can mean a corrective notification or a restoration process, not an undo button. The contract must name the residual impact and the operator who owns it. Avoid automatically chaining reversals merely because a planner decides the previous step was inconvenient.

Size the receipt store and reconcile deliberately

Assume, for this example only, 30 incident operations per minute and a seven-day retry horizon. That creates 302,400 potential receipt records before indexes and audit events. Measure actual record size and contention rather than turning this arithmetic into a database sizing promise. The local SQLite writer is deliberately serialized; a horizontally scaled service needs a storage design and load tests appropriate to its contention profile.

Compensation work should have its own queue capacity and age alarm. Otherwise an outage that generates many holds can leave cleanup permanently behind new work. Reserve operator visibility for unknown and partially compensated operations. A reconciliation sweep should emit decisions without deleting the history that explains them. Exact retention and escalation intervals depend on the incident process and are not universal defaults.

Primary sources and scope

Stripe documents API-specific idempotency semantics, including parameter-conflict checks and differences between API versions. Read the actual provider contract before assuming a retry returns the same response forever. Source: Stripe idempotent requests. AWS's saga guidance describes coordinated local transactions, compensating actions, and the lack of saga transaction isolation. Source: saga orchestration.

The maintenance-hold scenario, storage layout, capacity estimate, and gateway design are original educational proposals. Neither source establishes that this example coordinates a real cloud incident. Continue to the build article for the narrower behavior that the executable code actually demonstrates.