Durable Execution for Agent Workflows: Replay Determinism, Model Calls as Activities, Versioning Prompts and Resuming a Run That Waited a Week

Agent-era architecture · Practical technical guide · 19 October 2026

How durable-execution engines resume an agent workflow by replaying its history: why workflow code must be deterministic, why every model and tool call belongs in an activity, how to change prompts and code while runs are in flight, and when a plain checkpointed loop is enough.

A run that waits a week

An agent that drafts a purchase order, waits for a manager to approve it, and then places it may sit idle for days. The process that started it will not survive that long: deployments happen, nodes are replaced. Checkpointing the loop state at step boundaries, as the checkpointing guide describes, handles the storage side. Durable execution engines such as Temporal and Restate go further: they record what the workflow code did and replay that record on a fresh worker, so the code picks up where it stopped without explicit load-and-branch logic.

The price is a rule that catches most teams once: workflow code must be deterministic, and most of what an agent does is not. The design question is where to draw the line between the workflow and the activities it schedules.

What stays deterministic

Element	Where it runs	Replayed on resume?
Control flow: which step comes next, loops, waits, timers	Workflow code	Yes, from the recorded history
Model calls, tool calls, database reads, HTTP, random numbers, the clock	Activities	No; their recorded results are returned to the workflow
Retry policy, timeouts and heartbeats of each activity	Engine, configured from workflow code	Yes
Waiting for approval, a signal or a timer	Engine	Yes; the wait completes from history
The prompt text and the model name	Activity input, passed as data	Yes, as data; changing it is a code change to version
What the model decides given the context	Inside the activity	Never re-run; the recorded decision is replayed

Temporal states the rule directly: workflow code must make the same API calls in the same order given the same history, and non-deterministic operations such as API calls, LLM invocations and database queries belong in activities, which run outside the replay path and are retried automatically. Restate frames the same idea as a journal that records every operation and its result, replays it, and skips completed steps.

Replay: how the engine resumes without re-running your effects

When a worker picks up a workflow it has never seen, it runs the workflow function from the top. Each time the code schedules an activity, the engine compares that command with the next event in the recorded history. If the history already holds a completed activity of the same type at that position, the engine does not execute anything; it hands the recorded result back to the code and continues. Only when the code issues a command that history does not contain does real execution resume. A toy replayer, runnable with tests, shows it.

Illustrative python, checked for syntax; adapt names and storage to your system.

class NonDeterministic(Exception):
    pass

class Replayer:
    """history: list of (activity_name, result) recorded by a previous worker."""
    def __init__(self, history, execute):
        self.history, self.execute, self.pos, self.new = list(history), execute, 0, []

    def activity(self, name, *args):
        if self.pos < len(self.history):
            recorded_name, result = self.history[self.pos]
            if recorded_name != name:
                raise NonDeterministic(f"history has {recorded_name} at step {self.pos}, code issued {name}")
            self.pos += 1
            return result                       # replay: nothing runs again
        result = self.execute(name, *args)      # live fixture: execute, then record in memory
        self.history.append((name, result))
        self.new.append((name, result))
        self.pos += 1
        return result

The exception is the part teams meet in production. If a deploy changes the code so that it schedules a different activity at a position that history already covers, replay cannot continue; Temporal calls this a non-deterministic error and the run is stuck until the code is fixed or versioned. Changing the arguments of an activity is tolerated, because arguments are not compared; changing its type or order is not.

Model calls are activities, never workflow code

Everything the model does happens inside an activity: build the prompt from the inputs, call the provider, parse the answer into a structured decision, return it. The workflow receives a plain value, such as a list of proposed steps, and branches on it with ordinary deterministic code. Replay of a durably recorded completed activity reuses its value without another provider request. A retry or crash before durable recording can still repeat the request and charge; activity execution is not exactly-once.

Illustrative shape, SDK-neutral; map wf.* to your engine. Checked for syntax only.

from dataclasses import dataclass
from datetime import timedelta

@dataclass
class Proposal:
    supplier: str
    amount: int
    rationale: str

async def procurement_workflow(wf, request):
    """wf stands for the engine's workflow API. Only deterministic code lives here."""
    proposal: Proposal = await wf.execute_activity(
        "draft_order", request, retry_policy=wf.retry(max_attempts=3),
        start_to_close_timeout=timedelta(minutes=2))
    if proposal.amount > 5000:
        approved = await wf.wait_condition(lambda: wf.signals.get("approval") is not None,
                                           timeout=timedelta(days=7))
        if not approved or not wf.signals["approval"]:
            return "rejected"
    order_id = await wf.execute_activity("place_order", proposal,
                                         retry_policy=wf.retry(max_attempts=5))
    await wf.execute_activity("notify", order_id)
    return order_id

Three things hide in this shape. Activities are retried, so an effecting one such as place_order must be idempotent and should carry the workflow id as its key. The seven-day wait costs nothing; the worker holds no state and can be redeployed. And the prompt is activity input, so a prompt change is not a non-deterministic change to the workflow, although it is a behaviour change that needs its own evaluation.

Versioning prompts and code while runs are in flight

A run that sleeps for a week will be resumed by code deployed after it started. New activity arguments and changed timeouts are safe. Adding, removing or reordering activities, or changing which activity type runs at a position, is not, and for those the engines provide a patch marker. In the Temporal Python SDK it is workflow.patched, which records in history that the new path was taken; runs whose history predates the patch replay the old path, new runs take the new one, and once no old runs remain the marker is deprecated and removed.

Illustrative patching shape following the Temporal Python versioning documentation. Checked for syntax only.

async def procurement_workflow_v2(wf, request):
    proposal = await wf.execute_activity("draft_order", request)
    if wf.patched("compliance-check-2026-10"):       # new runs record the patch
        await wf.execute_activity("compliance_check", proposal)
    if proposal.amount > 5000:
        await wf.wait_condition(lambda: "approval" in wf.signals, timeout=timedelta(days=7))
    return await wf.execute_activity("place_order", proposal)

Prompts deserve the same discipline even though they live in activity inputs. Record the prompt version in the activity input so that the history shows which prompt produced which decision. When a prompt changes, a run that is halfway through will have earlier decisions from the old prompt and later ones from the new; that is acceptable only if the two prompts agree on the output schema. Changing the schema is a workflow code change, and needs the patch marker.

Worked example: a procurement agent that waits for approval

Run P-204 starts on Monday. The draft_order activity calls the model, which proposes supplier B for 7,200 with a rationale; the result is recorded. The amount is above 5,000 so the workflow waits for an approval signal. On Wednesday the service is redeployed with the compliance check added behind a patch marker. A new worker loads P-204 and replays: draft_order is in history, so its recorded proposal is returned without a model call; the patch marker finds no record of the patch in this history and takes the old path; the wait resumes. On Friday the manager approves. The workflow schedules place_order, which has no history entry, so it runs for the first time, carrying P-204 as its idempotency key. If the worker dies after the order is placed but before the result is recorded, the activity is retried; the supplier API recognises the key and returns the same order. The notification runs, and the workflow completes with the same code path it would have taken on Monday.

A run started on Thursday takes the new path and records the patch. Both are correct; the Monday code never needed to survive.

Failure modes worth testing

A model call in workflow code. On replay the provider is called again, returns something different, and the run diverges or fails with a non-deterministic error. Test by replaying a recorded history against the workflow with the provider stubbed to fail.
Clock or random in workflow code. Same problem, subtler symptoms. Use the engine clock and side-effect helpers.
Unversioned reorder. Reordering two activities breaks every in-flight run. Keep a replay test that runs the current code against histories captured from production.
Non-idempotent effecting activity. Activities are retried by design; an order placed twice is a correctness bug, not an infrastructure one. The outbox and journal guide covers the effect side.
History growth and large payloads. Hundreds of tool calls make a long history and slow replays; megabyte tool results are recorded on every event. Continue as a new run after a bounded number of steps and store large results behind a handle, as the tool result budgets guide recommends.

Migration steps from a loop with checkpoints

List every place the loop touches the outside world: model calls, tools, clock, random, environment. Each becomes an activity.
Rewrite the loop body as workflow code that only branches on activity results; keep the model call activity thin.
Give every effecting activity an idempotency key from the workflow id and step, and a retry policy with a timeout the tool owner agrees to. Replace sleeps and polling with engine waits and signals.
Capture histories from staging and add a replay test to CI that fails on non-determinism.
Use the patch marker on the first code change after launch, remove it once old runs drain, and continue as new after a fixed number of steps.

When durable execution is unnecessary

A run that finishes in seconds and can be restarted from the beginning does not need a durable engine; a retry at the caller is enough. A run whose only long wait is for a human can often be two short runs joined by a database row, which is the checkpointing design. Durable execution pays off when runs are long, interrupted often and have many effecting steps. Even then the engine does not remove the need for idempotent effects and versioned prompts; it moves them from a loop you wrote to a contract you must keep.

Sources and scope

Checked against these sources on 19 October 2026. Examples and figures are illustrative, not measurements.

Keep workflow code deterministic, put every model and tool call in an activity with an idempotency key, version code changes with a patch marker, and bound history length.