Why architecture matters here
The forcing function is the impossibility triangle of distributed writes: atomic-looking multi-service updates, high availability, and loose coupling — pick two. 2PC picks the first two on paper and loses both in practice: the coordinator is a single point of blocking failure, participants hold locks across network round-trips, and one slow service degrades every transaction that touches it. Most teams' first instinct — call services sequentially and hope — is a saga with no compensations, no log, and no name: when step three fails after steps one and two committed, the system is inconsistent and nothing knows it. The saga pattern is that instinct made rigorous enough to trust with money.
Architecture matters here because the pattern's correctness lives entirely in its bookkeeping. A saga is a state machine whose transitions are distributed across services and delivered over an at-least-once message layer. Without a durable log, a crashed orchestrator forgets which steps committed — recovery is impossible. Without idempotency, a redelivered 'charge card' command double-bills. Without semantic locking, a concurrent read sells inventory that a compensation is about to restore. Each omission is invisible in the demo and catastrophic at scale, because sagas fail partway by design — the partial states are not edge cases, they are the normal operating regime under load.
Compensation design is also where engineering meets business policy, which is why it cannot be bolted on. A refund is not an un-charge: it may take days, cost fees, and be observable by the customer. Some steps are not compensable at all — an email sent, a shipment dispatched — and must be ordered last (the pivot: after it, the saga can only roll forward). Deciding step order, pivot placement, and what each compensation means is an architectural act with legal and financial consequences, not an implementation detail.
Finally, sagas are a commitment made under uncertainty about scale. At ten orders a minute, a stuck saga is a support ticket; at ten thousand, the pattern's bookkeeping — log writes per transition, idempotency lookups per step, lock state per in-flight saga — is a real workload with its own capacity planning, and the difference between a saga layer that was architected and one that accreted is measured in exactly those incidents.
The architecture: every piece explained
Top row: the forward path — local transactions T1 through T4, each committing in its own service's database. Order creates the order in PENDING; payment charges; inventory reserves; shipping creates the shipment; a final step flips the order to CONFIRMED. Below them, compensations C1 to C3 in reverse order: release the reservation, refund the charge, cancel the order. Compensations are ordinary local transactions too — they can fail, must be retried, and must therefore be idempotent and, ideally, incapable of semantic failure (a refund can be retried until it succeeds; it should never be 'rejected').
Center: the saga log — the pattern's memory. Every transition (saga started, T2 dispatched, T2 completed, T3 failed, C2 dispatched…) is recorded durably before the action's effects are relied upon. Recovery after a crash reads the log and resumes: re-dispatch the in-flight step (idempotency absorbs the duplicate if it had already executed) and continue forward or backward. The log turns 'the coordinator died mid-saga' from data corruption into a delay.
Right: coordination style. Orchestration puts a central state machine in charge — it sends commands (ChargePayment) and consumes replies, and the saga's definition lives in one legible place; the cost is another service to run and a hub every message crosses. Choreography distributes the flow: each service emits domain events (OrderCreated) and the next reacts; no central runtime, natural decoupling — but the saga exists only as an emergent property of scattered subscriptions, cyclic dependencies creep in, and answering 'where is saga 8812 right now?' requires correlation infrastructure that orchestration gets for free. The working heuristic: choreography for short sagas (2-3 steps) with stable flows; orchestration once branching, timeouts, and compensation chains make the flow a real program. Both ride on a broker with at-least-once delivery, and the transactional-outbox pattern (commit the state change and the outgoing message in the same local transaction, relay to the broker asynchronously) plugs the dual-write gap on every hop.
Bottom: the isolation countermeasures. The idempotency layer gives every step a deterministic key (saga id + step id); participants record processed keys with their local transaction, so redelivery is a no-op. Semantic locks replace the isolation the database no longer provides: the order's PENDING state and inventory's reserved count are application-level markers that tell concurrent transactions 'this resource has uncommitted intent' — downstream logic treats reserved stock as unavailable and pending orders as invisible to fulfillment. They are the difference between an intermediate state that is merely visible and one that is actionable by mistake.
You rarely build the log and orchestrator from scratch anymore. Temporal (and Cadence before it) turns the saga into ordinary code: the workflow function is the state machine, and the platform records every step in an event history that replays deterministically after a crash — the saga log as a managed service, with retries, timers, and compensation blocks as library calls. AWS Step Functions expresses the same thing as a declarative state machine with native service integrations. Axon and Eventuate embed saga managers in the JVM event-sourcing world. The build-vs-adopt line: if your organization already runs a workflow platform, hand-rolling saga plumbing is undifferentiated heavy lifting; if a saga is your first workflow, weigh the platform's operational footprint against three tables (saga instances, step log, idempotency keys) and a poller. Either way the design work — step order, pivot placement, compensation semantics — transfers unchanged; platforms execute sagas, they do not design them.
End-to-end flow
Walk order 8812 through a failure. Forward: the orchestrator logs SAGA_STARTED and commands T1; order service creates the row in PENDING and replies via its outbox. T2: payment charges $214 with idempotency key 8812:T2, records the key and the charge in one local transaction, replies success. The orchestrator logs each completion before dispatching the next step. T3: inventory finds the SKU short — a legitimate business failure, not an infrastructure blip — and replies ReservationFailed.
The turn: the orchestrator logs T3_FAILED, flips the saga to compensating, and walks the completed steps in reverse. C2: RefundPayment with key 8812:C2. Suppose the reply is lost in a broker hiccup — the orchestrator's timeout fires, it re-sends, and payment's idempotency table swallows the duplicate: one refund, exactly. C1: order service marks the order CANCELLED with a reason code. The orchestrator logs SAGA_COMPENSATED; the customer sees a clean 'out of stock' with the charge reversed. Note what the customer could have seen mid-saga: a charged card and no confirmed order. That window is real and unavoidable — the design choices are its duration (seconds, via timeouts) and its presentation (the UI shows 'processing', not the raw pending state).
The crash variant: same saga, but the orchestrator dies between logging T2_DISPATCHED and receiving the reply. A standby (or the restarted instance) scans the log for in-flight sagas, finds 8812 with T2 unresolved past its timeout, and re-dispatches. If payment had processed the original, the idempotency key returns the recorded result and the saga proceeds as if nothing happened; if not, it processes now. Either way the log converges to truth. The pivot variant: had T3 succeeded and T4 (shipping — assume dispatch is irreversible once the label prints) failed after printing, backward recovery is off the table; the saga rolls forward — retrying T4's remaining work until it completes, escalating to a human queue if it cannot. Pivot placement decided this at design time: everything compensable goes before the point of no return.