Why architecture matters here
The defining property of payments is that the expensive failure is silent inconsistency, not visible error. If an API returns 500, the client retries and life goes on. But if the gateway sends an authorization to the acquirer, the acquirer charges the card, and then the network times out before the response comes back, the gateway is in the worst possible position: money moved, but the gateway does not know it. Retry blindly and you double-charge; give up and you have captured funds with no record. This unknown-outcome problem is the central challenge, and every serious design choice — idempotency keys, the ledger, reconciliation — exists to make unknown outcomes recoverable rather than catastrophic.
The second forcing function is auditability. Money movement is regulated; a gateway must be able to answer, for any transaction, months later, exactly what happened and when, and prove that its internal record matches what the bank actually settled. This is why the ledger is immutable and double-entry: you never update a balance, you append a transaction that debits one account and credits another, and the balance is derived. An immutable append-only ledger means the history can never be quietly rewritten, which is both a correctness property (you can always reconstruct state by replaying entries) and a compliance property (auditors can trust the record). Mutable balances, by contrast, lose the very history regulators require and make bugs unrecoverable because the wrong number overwrote the right one.
The third is PCI scope. The moment a raw card number (PAN) touches a system, that system falls under the full weight of PCI-DSS audit. The architecture's job is to shrink that blast radius: a dedicated tokenization vault holds the PANs behind strong encryption and hands the rest of the system opaque tokens, so the orchestrator, ledger, and risk engine never see a card number and stay out of the strictest audit scope. This is not just compliance theater — it is genuine security architecture, concentrating the highest-value secret in the smallest, most hardened component.
These three forces — unknown outcomes, auditability, and PCI scope — do not act independently; they compound, and that compounding is what makes payments genuinely hard rather than merely careful. Consider how the immutable ledger and the unknown-outcome problem interlock: because you can never resolve a timeout by guessing, you resolve it by querying the acquirer for the authoritative answer and then recording that answer in the append-only ledger, so the resolution is itself permanent and auditable. And because the ledger is the source of truth rather than a log downstream of some mutable 'balance', a reconciliation process can, months later, replay it and prove that every dollar the bank settled corresponds to a ledger entry and vice versa. Strip out any one of the three and the others weaken: mutable state makes unknown outcomes unrecoverable, no reconciliation makes the ledger unverifiable, and lax PCI scope turns the whole audited system into an untrusted one. A payment gateway is therefore less a feature and more a set of mutually reinforcing invariants, and the architecture is the machinery that holds all of them true simultaneously under partial failure.
The architecture: every piece explained
Top row: entry and secrets. The merchant API authenticates the caller (API keys or OAuth), validates the request, and — critically — requires an idempotency key on every mutating call; the gateway stores the key with the result so that a repeat of the same key returns the original outcome instead of re-executing. The tokenization vault is where a raw PAN is exchanged for a token the first time a card is seen; the PAN is encrypted at rest under keys in a KMS/HSM, and only the vault can detokenize when an acquirer call genuinely needs the number. The orchestrator is the brain: a durable state machine, one instance per payment, that drives the payment through its states and persists every transition so a crash resumes exactly where it left off. The risk/fraud engine scores each attempt and may trigger a 3-D Secure challenge, shifting liability and adding a customer-facing authentication step.
Middle row: rails and truth. The acquirer/PSP is the connection to the actual card networks; the gateway may hold connections to several and route by cost, geography, or health. The ledger is the immutable double-entry record — every authorization, capture, refund, and fee is a balanced set of entries, and the operational state machine is reconciled against it. The webhook engine handles the asynchronous reality that many payment outcomes (settlement, disputes, delayed captures) arrive minutes or days later as callbacks from the acquirer, and must be delivered reliably to the merchant with retries and signatures. Reconciliation is the daily process that compares the gateway's internal ledger against the settlement files the banks send, catching any drift.
Bottom rows: recovery and protection. Retry/saga encodes the truth that a payment is a multi-step distributed transaction that cannot be a single database commit — so it is a saga with explicit compensations: if capture succeeds but the fulfillment step fails, you issue a refund (a compensating action), never leave a partial write. Vault / KMS centralizes encryption and enforces key rotation. The ops strip is the business-critical telemetry: authorization approval rates (a drop can mean an acquirer problem or a fraud-rule misfire), settlement drift, chargeback and dispute flows, and continuous PCI audit posture.
End-to-end flow
Walk a $40 charge from tap to settlement. The merchant's server calls POST /payments with an idempotency key, the amount, and a card token (the card was tokenized on a prior interaction, so no PAN is in this request). The API validates and checks the idempotency store: no prior result for this key, so it proceeds and records the key as in-flight. The orchestrator creates a payment record in state CREATED and persists it. The risk engine scores the transaction; the score is moderate, so it requires 3-D Secure, and the customer completes the bank's challenge — liability now shifts to the issuer. State advances to AUTHORIZING.
The orchestrator asks the vault to detokenize just long enough to call the acquirer, sends the authorization, and here is the dangerous moment: the acquirer request times out at 8 seconds with no response. The payment is now in AUTH_UNKNOWN — money may or may not be held. The orchestrator does not retry blindly; instead it queries the acquirer's status endpoint (or waits for the reversal window) with the same acquirer-side idempotency reference. The status query reveals the authorization actually succeeded. The orchestrator records the auth in the ledger as a balanced pair of entries, advances to AUTHORIZED, and returns success to the merchant — who, if they retried their original call with the same idempotency key during all this, gets the single consistent result, never a second charge.
Later the merchant captures (ships the goods): CAPTURING → acquirer capture → ledger capture entries → CAPTURED. A day later the acquirer's settlement file arrives via webhook; the webhook engine verifies its signature, and reconciliation matches the settled amount against the ledger's captured amount — they agree, and the payment reaches SETTLED. Now the unhappy branch: two weeks on, the cardholder disputes the charge. A chargeback webhook arrives; the orchestrator transitions to DISPUTED, the ledger records a compensating debit reversing the merchant's credit, and the dispute-evidence workflow begins. At no point was a balance overwritten — every state of this payment, including the money that came and went, is reconstructable by replaying the immutable ledger entries, which is exactly what an auditor or an incident responder needs.
Notice the role the idempotency key played across this whole timeline, because it is easy to underrate. The merchant's server, sitting behind its own flaky network, may well have retried the original POST /payments two or three times during the eight-second acquirer stall — it had no way to know whether the gateway received the first attempt. Each retry carried the same idempotency key, and each time the gateway looked it up, found the payment already in flight, and returned the current state instead of starting a second authorization. Without that key, those retries would have opened three parallel payment state machines against the same order, and the customer could have been authorized three times over — a bug that is invisible in testing (where networks behave) and catastrophic in production (where they do not). The idempotency key is what makes the merchant's retry safe, which in turn is what lets the merchant retry at all rather than leaving the customer staring at a spinner. This is the quiet theme of payment architecture: nearly every component exists so that some other participant is allowed to fail and recover without money moving twice or vanishing.