Why architecture matters here
The naive recovery story — keep the transcript in memory, serialize it to disk occasionally, reload on crash — fails on three counts. First, granularity: if the checkpoint interval is every N minutes, everything since the last checkpoint re-executes on resume, and re-executing an agent step is not idempotent by default — the email sends twice, the ticket opens twice, and the model (nondeterministic even at temperature zero across serving fleets) may choose a different branch entirely, invalidating the remembered future. Second, consistency: a snapshot taken mid-step can capture a tool call issued but no result — an unresolvable state. Third, pauses: long-running agents do not only crash; they wait, sometimes for days, on human approval or a timer, and holding a process plus its context in memory for days is not an architecture.
Framing the agent as a state machine whose transitions are steps fixes all three at once. A step — one model call, or one tool execution — is the atomic unit; checkpoints happen exactly at step boundaries, never inside them. Persist the intention before the action and the outcome after it, and the log always tells an unambiguous story: this call completed with this result, that one was issued and must be reconciled. Waiting stops being a live process and becomes a row whose status says paused. This is the same move that took workflow engines from cron scripts to durable execution, applied to a loop whose planner happens to be a language model.
The architecture: every piece explained
At the core sits the event log: an append-only sequence per run — run started, model call requested (messages hash, params), model responded (completion, usage), tool call requested (name, args, idempotency key), tool returned (result), state patched (key, value), interrupted (reason), resumed. The checkpointer appends the record and commits before the loop consumes the result — write-ahead semantics, so nothing influences the agent's future that is not already durable. In-memory agent state is a pure fold over this log, which makes it disposable by construction.
Because replaying ten thousand events on every resume is wasteful, the snapshotter materializes the fold every K steps: snapshot plus tail replay gives O(K) resume cost regardless of run length. The idempotency layer stamps every side-effecting tool call with run_id:step_n and demands the downstream system (or a wrapping dedupe table) honor it — this is what converts at-least-once execution into effectively-exactly-once effects. Interrupt points are first-class events: an approval gate appends interrupted(awaiting_approval) and the process exits; the eventual approval is just another event that flips the run back to runnable, whether that takes ten seconds or ten days. Finally, the version gate records the code revision and prompt hash the run started under; on resume it decides between resuming verbatim, migrating the state, or refusing — because folding old events through new reducer code is where silent corruption lives. Underneath, the store is pluggable: Postgres for the log and snapshots is the common choice, with an object store for oversized payloads and Redis only as a cache, never as the system of record.
End-to-end flow
Trace a research agent through a crash. Step 17: the loop wants a model call, appends model_call_requested, gets the completion, appends model_responded, and only then lets the planner read it. The completion asks for create_ticket. The loop appends tool_call_requested with idempotency key run-8c1:18, invokes the ticketing API with that key in the header, and appends tool_returned with the ticket ID. Step 19 hits an approval gate: the loop appends interrupted(awaiting_approval), the snapshotter writes snapshot v3 (fold of events 0–19), and the process exits. Nothing is running; the run is a database row.
Two days later the approver clicks yes. The resume manager loads snapshot v3, replays the empty tail, checks the version gate — code revision matches — appends resumed(approved_by=…), and the loop continues at step 20. Mid-step-24, the pod is OOM-killed after tool_call_requested was appended but before tool_returned. On restart, the resume manager sees a dangling request: reconciliation queries the downstream API by idempotency key run-8c1:24. The call had landed — the API returns the existing resource — so the manager synthesizes tool_returned from that response and no duplicate is created. Every earlier step resumes from the log verbatim: model calls are not re-issued on replay; their recorded completions are read back, which is what makes resume cost cents instead of dollars and keeps the agent's past decisions frozen.