Why architecture matters here
Idempotency is the difference between a mature API and one that occasionally double-charges. The architecture matters because it must catch not just accidental duplicates but retries that happen at multiple layers — client, gateway, service mesh.
Getting the dedupe store right — cheap lookups, TTL matching the retry window, atomic "first write wins" — is non-trivial. Propagating idempotency across service calls prevents the pattern from breaking one hop in.
With the pieces mapped, retries become boring and safe.
The architecture: every piece explained
The top strip is the primary path. Client request arrives with an idempotency key (UUID from the client). API gateway extracts the key, validates format, and consults the Dedupe store. If the key already has a stored result, return it; if in-flight, wait or return 409; if new, atomically claim and forward to the Business handler.
The middle row is the safety detail. Retry policy defines backoff and max attempts. Conflict handling is critical: same idempotency key with a different body must be rejected (usually 422) — a common bug. Cross-service propagation passes derived keys downstream so retries at any layer are safe. TTL + eviction keeps the store from growing forever.
The lower rows are ops. Observability tracks dedupe hits, conflicts, and store growth. Audit + refund path handles the rare cases when dedupe fails — e.g., store lost the entry between attempts. Ops covers schema versioning, client cooperation to generate keys, and periodic drills to verify behavior.
End-to-end flow
End-to-end: a client submits a payment with idempotency key K1. Gateway checks dedupe store — new. Claims K1 with a status of "in progress". Forwards to payment handler. Payment succeeds; result stored under K1. Response returned. Client's network hiccuped; client retries with the same K1. Gateway finds K1 with stored result; returns the same response. No duplicate charge. A different client mistakenly reuses K1 with a different body; gateway returns 422 conflict. TTL removes K1 after 24 hours; future reuses are treated as new. Observability shows 3% dedupe hit rate — healthy.