Saga Pattern Architecture in Depth: running sagas on a durable workflow engine, versioning in-flight sagas, testing compensations and operating a saga fleet

The one thing a saga must never do: forget

A saga orchestrator has exactly one hard job, which is to never lose its place; retries, compensation and timeouts all follow from knowing which steps have committed. A hand-rolled orchestrator needs four pieces of infrastructure for this: a saga log written before each next command, an outbox so that logging and sending are not a dual write, a scheduler that resumes non-terminal sagas after a crash, and a retry loop with backoff.

A durable workflow engine packages exactly those four pieces. The engine records an event history for each workflow execution: step scheduled, step completed with its result, timer fired, cancel requested. That history is the saga log. When a worker process dies mid-saga, another worker picks the execution up and replays the history through the same code; completed steps return their recorded results instantly instead of re-running, and the code arrives back at the line where it left off. The price is a strict rule: workflow code must be deterministic, so it cannot read the clock, generate random numbers or call the network directly. Anything with a side effect goes into an activity, and the engine records the activity's result.

The architecture

Read the diagram from the top. An API call starts a workflow execution, using the business key, here the order identifier, as the workflow ID. Engines reject a second start with the same ID while the first is running, which gives you duplicate-request protection for free: a double-clicked Book button starts one saga, not two. The engine puts a task on a queue; a saga worker runs the deterministic saga code, which asks the engine to schedule activities; activity workers call the real participants, the flight, hotel and payment services.

Two parts of the diagram are easy to overlook. Participants still need idempotency keys: activities run at least once, not exactly once, so an activity whose worker died after the service committed is retried, and a key from the workflow ID plus step name makes that retry a no-op. And the monitor on the left exists because the engine will retry a failing compensation forever; something outside the saga must notice and tell a human.

Worked example: the trip saga

from datetime import timedelta
import asyncio
from temporalio import workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError, ApplicationError

with workflow.unsafe.imports_passed_through():
    from trip.activities import (book_flight, cancel_flight, book_hotel,
                                 cancel_hotel, charge_card, send_itinerary)

RETRY = RetryPolicy(initial_interval=timedelta(seconds=1), backoff_coefficient=2.0,
                    maximum_interval=timedelta(minutes=1),
                    non_retryable_error_types=["SoldOut", "CardDeclined"])
UNDO_RETRY = RetryPolicy(initial_interval=timedelta(seconds=5),
                         maximum_interval=timedelta(minutes=10))  # no attempt limit

@workflow.defn
class TripSaga:
    @workflow.run
    async def run(self, trip: dict) -> dict:
        key = workflow.info().workflow_id              # stable across retries and replays
        undo = []                                      # compensation stack
        try:
            flight = await workflow.execute_activity(
                book_flight, args=[trip, f"{key}:flight"],
                start_to_close_timeout=timedelta(seconds=30), retry_policy=RETRY)
            undo.append((cancel_flight, flight))
            hotel = await workflow.execute_activity(
                book_hotel, args=[trip, f"{key}:hotel"],
                start_to_close_timeout=timedelta(seconds=30), retry_policy=RETRY)
            undo.append((cancel_hotel, hotel))
            await workflow.execute_activity(                         # pivot step
                charge_card, args=[trip, f"{key}:charge"],
                start_to_close_timeout=timedelta(seconds=60), retry_policy=RETRY)
        except (ActivityError, asyncio.CancelledError) as err:
            await self.compensate(undo, key)
            if isinstance(err, asyncio.CancelledError):
                raise                                  # honour the cancel after cleanup
            raise ApplicationError(f"trip failed: {err.cause}", non_retryable=True)
        # past the pivot: retriable steps only, never compensate
        await workflow.execute_activity(
            send_itinerary, args=[trip, f"{key}:email"],
            start_to_close_timeout=timedelta(seconds=30), retry_policy=UNDO_RETRY)
        return {"status": "BOOKED", "flight": flight, "hotel": hotel}

    async def compensate(self, undo, key):
        for fn, booking in reversed(undo):
            await workflow.execute_activity(
                fn, args=[booking, f"{key}:{fn.__name__}"],
                start_to_close_timeout=timedelta(seconds=30), retry_policy=UNDO_RETRY)

The shape is a plain try block with a compensation stack: each successful forward step pushes its undo, with the booking it created. If any step before the pivot fails, the stack unwinds in reverse order. The card charge is the pivot: once money has moved, the saga only moves forward, so the itinerary email after it is retried until it succeeds rather than triggering a refund.

Note the two retry policies. Forward steps retry transient errors with capped exponential backoff but list SoldOut and CardDeclined as non-retryable, because retrying a definitive business answer only delays the compensation. Compensations have no attempt limit at all. A compensation that stops retrying leaves a customer holding a hotel room they did not pay for, so an undo must keep trying until it succeeds or a person intervenes. The idempotency key is built from workflow.info().workflow_id, which is stable across retries and replays; building it from a random UUID inside workflow code would break determinism and idempotency at once.

Retry, compensate or wait: classifying failures

The most expensive mistake in saga design is treating an unknown outcome as a failure. A timeout on the charge does not mean the card was not charged; compensating then cancels the trip while the payment may have gone through. Resolve the ambiguity first, by retrying with the same idempotency key or querying the participant, and compensate only on a definitive no. In an engine, that means a business rejection becomes a non-retryable error type and everything else stays retryable.

Cancellation, deadlines and human steps

Real sagas get cancelled. A customer abandons checkout, an operator kills a runaway execution, a parent workflow gives up. In the Temporal Python SDK, a cancellation request is delivered to the workflow as asyncio.CancelledError, which the code may catch to clean up before re-raising, and that is exactly what the example does: the same compensation path runs whether the saga failed or was cancelled. The SDK documentation notes that a cancellation request is a state, not an event, so it is delivered once; code that swallows it cannot be asked again.

Deadlines are the other half. A saga waiting for a human approval or a bank transfer needs an explicit timer: wait for the signal or 48 hours, whichever comes first, and treat expiry as a business failure that compensates. Without it, a saga waiting on a signal that never arrives holds its reserved seat or pending inventory forever.

Changing a saga while thousands are running

Sagas live for seconds, days or months, and you will deploy new saga code while old executions are in flight. With a replaying engine this is the sharpest edge in the whole design: if the new code schedules a different sequence of steps than the recorded history, replay fails with a non-determinism error and the execution is stuck. Adding a fraud check before the charge is exactly that kind of change.

# Version 2 adds a fraud check before the charge. Executions that started on
# version 1 replay their history without it; new ones take the new path.
if workflow.patched("fraud-check-before-charge"):
    await workflow.execute_activity(
        fraud_check, args=[trip, f"{key}:fraud"],
        start_to_close_timeout=timedelta(seconds=20), retry_policy=RETRY)
await workflow.execute_activity(charge_card, ...)

# Weeks later, once no version-1 execution is still open or will be queried:
#   replace the if-block with  workflow.deprecate_patch("fraud-check-before-charge")
#   followed by the unconditional fraud_check call; later still, remove both.

workflow.patched returns true for executions that are new or that already passed this point on the new code, and false for executions replaying history recorded before the patch existed, so old sagas finish on their old path. Patches are removed in two later deployments, first by switching to deprecate_patch once no old execution remains, then by deleting it. The alternatives are running old and new worker versions side by side and routing each execution to the version it started on, or, for short-lived sagas, draining: stop starting new executions on the old code and wait for the in-flight ones to finish. A hand-rolled orchestrator faces the same problem in a different form: a saga row written by version 1 must still be understood by version 2, so the step list needs a version column.

Testing sagas: the fault-injection matrix

A saga with N compensatable steps has at least N distinct failure paths plus the unknown-outcome and compensation-failure variants, and in most codebases only the happy path is tested. Treat the failure paths as a matrix and generate one test per cell: fail step k definitively and assert that exactly steps k-1 down to 1 were compensated, in reverse order. The engine's time-skipping test environment makes timer-driven paths, such as a 48-hour approval timeout, run in milliseconds.

import pytest, uuid
from temporalio import activity
from temporalio.exceptions import ApplicationError
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker

calls = []

def fake(name, fail_with=None):
    @activity.defn(name=name)
    async def _impl(arg: dict, key: str) -> dict:     # every activity takes (payload, key)
        calls.append(name)
        if fail_with:
            raise ApplicationError(fail_with, type=fail_with, non_retryable=True)
        return {"id": name}
    return _impl

@pytest.mark.asyncio                                     # requires pytest-asyncio
@pytest.mark.parametrize("failing_step, expected_undo", [
    ("book_flight", []),
    ("book_hotel",  ["cancel_flight"]),
    ("charge_card", ["cancel_hotel", "cancel_flight"]),
])
async def test_compensation_order(failing_step, expected_undo):
    calls.clear()
    names = ["book_flight", "book_hotel", "charge_card", "send_itinerary",
             "cancel_flight", "cancel_hotel"]
    acts = [fake(n, "CardDeclined" if n == failing_step else None) for n in names]
    async with await WorkflowEnvironment.start_time_skipping() as env:
        async with Worker(env.client, task_queue="t", workflows=[TripSaga], activities=acts):
            with pytest.raises(Exception):
                await env.client.execute_workflow(
                    TripSaga.run, {"to": "LIS"}, id=str(uuid.uuid4()), task_queue="t")
    undo_calls = [n for n in calls if n.startswith("cancel_")]
    assert undo_calls == expected_undo

Extend the matrix along three more axes. Duplicate delivery: invoke each participant twice with the same key and assert that one effect occurred. Unknown outcome: make the charge activity time out after committing and assert that the saga ends booked, not compensated. Crash and replay: kill the worker between steps and assert that the saga completes without repeating a completed step. The durable-execution mechanics behind that last test are covered in durable agent workflows: state, retries and replay.

Operating a saga fleet

• Count sagas by state and age. Running, compensating, completed, compensated and failed, bucketed by age. A growing tail of running sagas older than the longest legitimate wait is the first symptom of almost every saga incident.
• Alert on compensation, not just failure. A compensation rate that doubles overnight usually means a participant started rejecting requests, and it is visible before customers complain.
• Page on compensation retries. A compensation that has retried for more than a few minutes needs a person. It usually means the undo API changed, a booking was modified by hand or the participant is down.
• Make every saga findable by business key. Using the order ID as the workflow ID means support can look up order 9001 and see each step, its input, its result and its retries without joining logs.
• Give operators safe, logged levers. Retry now, skip a step with a recorded reason, or force compensation; each is a manual edit to a financial process.
• Plan capacity for the bookkeeping. Engines cap history size, so sagas that loop for months should periodically continue as a new execution.

Trade-offs

A short saga across two services that already publish domain events is often simpler as choreography, with each service publishing through a transactional outbox; the engine earns its keep with more steps, branches, timers or human waits, or when "where is order 9001 right now?" needs one answer. The comparison with two-phase commit is made in Saga pattern architecture: distributed transactions without 2PC. For payments specifically, the participant-side idempotency that every approach relies on is covered in idempotency keys in agent payments.

What to do next

1. List every saga in your system and mark each step as compensatable, pivot or retriable.
2. For each participant, confirm it accepts an idempotency key derived from the saga ID and step name, and that a replay returns the original result.
3. Classify every participant error as transient, definitive or unknown, and map definitive errors to non-retryable types.
4. Give every wait a timer and a defined outcome on expiry.
5. Remove attempt limits from compensations and add an alert when one has retried for more than a few minutes.
6. Write the fault-injection matrix: one test per failing step, plus duplicate delivery, unknown outcome and crash-replay.
7. Before your next change to saga code, decide how in-flight executions will be handled: patch, versioned workers or drain.
8. Build a dashboard of sagas by state and age, and make each saga searchable by its business key.