Payment systems must never charge twice, never miscount money, and survive every failure mode of every dependency. The core technical pattern is: idempotency keys + double-entry ledger + async reconciliation with the bank/network.
Idempotency keys
Every payment request carries a client-generated UUID. The server checks: have I seen this key before? If yes, return the original response (whatever the outcome was). If no, process and store result keyed by UUID. Retries become safe — duplicate charges become impossible.
Double-entry ledger
Every transaction is two entries: debit one account, credit another. Sum across all accounts always equals zero. The ledger is append-only (never UPDATE), making audit trivial. Modern frameworks: Tigerbeetle (Zig), Tinkoff Ledger, or hand-rolled on Postgres.
State machine for payment lifecycle
INITIATED → AUTHORIZED → CAPTURED → SETTLED
↓ ↓
CANCELLED REFUNDED → REFUND_SETTLED
↓
FAILEDAsync reconciliation
Every night, fetch settlement file from card network. For each entry, match against your internal ledger. Discrepancies (network says $100, your ledger says $99): create a reconciliation_dispute row, alert ops. Don't auto-correct money — humans must sign off.
PCI scope minimization
Card number storage is regulated (PCI DSS). To stay out of full PCI scope: use a tokenization service (Stripe, Braintree, your bank). They store the card; you store only a token. Your servers never see raw PAN. Cuts compliance burden dramatically.