Why architecture matters here
The architecture matters because in a Java ADK agent the retry is not an edge case — it is designed in. The runtime retries transient tool failures, message queues redeliver on consumer restart, upstream callers retry on timeout even when the original succeeded, and the agent's own planner may re-invoke a tool after a partial failure it could not distinguish from a total one. Every one of these paths delivers the same side-effecting call more than once. Without idempotency, reliability and correctness are in direct conflict: the more you retry to be reliable, the more you duplicate to be wrong. Idempotency dissolves that conflict.
It matters because the failures it prevents are the ones customers feel most acutely and trust least. A double charge, a duplicate order, two identical emails, a doubled inventory decrement — these are not silent corruption buried in a log; they are visible, expensive, and reputation-damaging. Worse, they surface precisely under load and failure, exactly when the system is already stressed and least able to absorb a support incident. Building idempotency in from the start is far cheaper than reconciling double-charges after the fact.
It matters because the naive alternatives do not hold up. 'Just make retries at-most-once' drops legitimate work when a response is lost in transit — you cannot tell a lost request from a lost response, so at-most-once must sometimes not-execute a call that should have run. 'De-duplicate by content hashing after the fact' is a reconciliation nightmare and cannot prevent the second effect, only detect it. The only clean answer is to make the operation itself idempotent so that duplicate delivery is safe, decoupling delivery guarantees from effect guarantees.
It matters because the guarantee must survive crashes, not just clean retries. The hard case is a process that performs the side effect and then dies before recording that it did — on restart the retry sees no record and executes again. This is why the architecture insists that the effect and its bookkeeping be committed atomically, and why 'record after effect' as two separate steps is a subtle double-execution bug. Understanding the architecture is understanding exactly where atomicity is required and why it cannot be approximated.
Finally it matters because idempotency is the enabling property for aggressive, confident retrying — the foundation the rest of your resilience stack is built on. Circuit breakers, hedged requests, dead-letter redrive, and outbox relays all assume the operations they retry are safe to repeat. Get idempotency right and every other resilience pattern becomes usable; get it wrong and each of them becomes a duplication engine. It is the quiet contract underneath the whole reliability story.
The architecture: every piece explained
The first component is the idempotency key, and everything hinges on deriving it correctly. The key must identify the intent so that every retry of the same logical operation reuses the same key, while two genuinely different operations never share one. The best keys are supplied by the originator — a client-generated UUID attached to the request, an order ID, a message ID from the queue — so that a retry carries the identical key. Deriving a key by hashing the request payload works only if the payload is truly stable across retries; a timestamp or a nonce buried in the body will make every retry look like a new intent and defeat the whole scheme.
The second component is the idempotency store: a durable, low-latency key-value store mapping each key to a record of the operation's status and result. It holds, per key, a state (in-progress, completed, or failed), the stored response to replay on a hit, and metadata like a creation timestamp for expiry. In a Java service this is commonly a relational table with the key as primary key, or a distributed cache with persistence; the essential requirements are durability across restarts, a uniqueness constraint that rejects duplicate inserts, and fast lookups on the hot path.
The third component is the execute-or-replay decision, the branch at the heart of the handler. On receiving a call it attempts to claim the key by inserting an in-progress record; if the insert succeeds, this delivery owns the execution and proceeds; if the insert fails because the key already exists, this is a duplicate and the handler switches to replay mode, returning the stored result rather than executing. Structuring the decision as an atomic claim-by-insert — rather than a read-then-write — is what makes it safe against concurrent duplicates, because the store's uniqueness constraint serializes the race.
The fourth and most critical component is the atomic commit of effect-and-record. For side effects on the same database as the idempotency store, the effect and the completed record are written in one transaction, so either both happen or neither does — a crash cannot leave the effect done but unrecorded. For effects on an external system that cannot share the transaction (a payment gateway, a partner API), the pattern leans on the external system's own idempotency support: forward your idempotency key to it so that it, too, deduplicates, turning two systems' guarantees into one end-to-end guarantee.
The final component is record lifecycle management: expiry and cleanup. Idempotency records cannot live forever or the store grows without bound, but they must live long enough to cover the realistic retry window — well beyond the longest retry backoff, the longest queue redelivery delay, and any manual replay. A time-to-live on each record, sized to comfortably exceed the maximum plausible duplicate-arrival gap, bounds storage while preserving the guarantee. Records for still-in-progress operations need care too: a stale in-progress record from a crashed handler must eventually be reclaimable so a legitimate retry is not blocked forever.
End-to-end flow
Walk a payment tool call through the machinery. The agent decides to charge a customer and issues the charge_card tool call carrying an idempotency key that was generated once when the checkout intent was created — say chk-9f3a.... The Java handler receives the call and, before touching the payment gateway, attempts to insert an in-progress record for chk-9f3a into the idempotency table.
The insert succeeds — no record existed — so this delivery owns the execution. The handler calls the payment gateway to charge the card, forwarding chk-9f3a as the gateway's own idempotency key for belt-and-suspenders protection. The gateway returns a success with a transaction ID. The handler now, in a single transaction, updates the idempotency record to completed and stores the transaction ID as the result to replay. Effect and record are committed together; the handler returns the result to the agent.
Now suppose the response never reached the agent — a timeout — and the ADK runtime retries the identical tool call with the same key chk-9f3a. The handler attempts to insert the in-progress record and the store's uniqueness constraint rejects it: a record already exists. The handler switches to replay: it reads the completed record, finds the stored transaction ID, and returns it — without calling the gateway again. The customer is charged exactly once; the agent receives the same successful result it would have the first time.
Consider the crash case that makes atomicity non-negotiable. Imagine the handler charged the card but died before committing the completed record. On retry it inserts... but wait — under the atomic design, the charge and the record commit together, so if the record is absent the charge never committed either, and re-executing is correct. Under a naive design where the charge happened as a separate prior step, the retry would find no record and charge again. This is exactly why the effect and the record must be one atomic unit, or why the external gateway must itself honor the forwarded key.
Finally consider two duplicates racing in at the same instant — the original and a redelivery arriving concurrently on two threads. Both try to insert the in-progress record for chk-9f3a; the store's uniqueness constraint lets exactly one succeed and the other fail. The winner executes and commits; the loser, seeing the conflict, waits for or reads the winner's result and replays it. Even under a dead heat, the side effect happens once. Days later the record's TTL lapses and it is garbage-collected, long after any retry could plausibly arrive, and the store stays bounded.