Agent checkpointing, in depth: step boundaries, write ordering, idempotent resume, human interrupts and forking

Why checkpoint at step boundaries

Keeping the transcript in memory and dumping it to disk every few minutes fails three ways. Everything since the last dump re-executes after a crash, and agent steps are not safe to repeat: the email goes out twice, and the model, which is not deterministic across a serving fleet even at temperature zero, may pick a different branch. A dump taken mid-step can capture a tool call that was issued with no result, a state nothing can interpret. And agents wait for approvals, timers and other agents; holding a live process for days is not an architecture.

The fix is to treat the agent as a state machine whose transitions are steps, and to checkpoint exactly at step boundaries, never inside a step. A waiting run becomes a stored row whose status is interrupted, using no memory or threads until something resumes it. The agent state machine article covers modelling the steps themselves.

What goes into a checkpoint

A checkpoint must contain everything the next step reads, and nothing that can be recomputed cheaply:

Keep large payloads such as retrieved documents, screenshots and file contents in an object store and record only a content hash and URI in the checkpoint. That keeps writes small and lets checkpoints share unchanged blobs. Do not store live handles: database connections, open sockets and client objects must be rebuilt on resume from configuration.

The architecture

The components and how they connect:

The checkpointer writes a new checkpoint after each step and commits it before the next step starts. The chain of checkpoints, linked by parent id, is the run's history. Pending writes record results from parallel tasks that finished inside a step that, as a whole, failed, so resuming does not redo them. Idempotency keys derived from run id and step make external side effects safe to repeat. Interrupt points end a step with a request for input and let the process exit. The resume manager loads the latest checkpoint and continues; the version gate refuses or migrates checkpoints written by incompatible code. The store is usually Postgres, with an object store for large blobs; a cache such as Redis can sit in front but should not be the system of record.

Write ordering and the resume loop

The rule is that nothing influences the agent's future until it is durable. In practice: compute the step, write the checkpoint containing its result and the next step, commit, then continue. A minimal checkpointer:

def run(run_id, store, nodes):
    ckpt = store.latest(run_id)                     # None for a new run
    state, nxt, step = (ckpt.state, ckpt.next, ckpt.step) if ckpt else (initial_state(), "plan", 0)
    check_version(ckpt)                             # version gate
    while nxt != "END":
        ctx = StepContext(run_id=run_id, step=step) # carries the idempotency key prefix
        try:
            update, nxt_after = nodes[nxt](state, ctx)
        except Interrupt as i:
            store.put(run_id, step, state, nxt, status="interrupted", waiting_for=i.payload)
            return "paused"
        state = merge(state, update)
        step += 1
        store.put(run_id, step, state, nxt_after, status="running")   # commit BEFORE continuing
        nxt = nxt_after
    store.mark_done(run_id)

If the process dies anywhere inside nodes[nxt], the last committed checkpoint still points at that node, so resume re-runs it from its start. That is the central consequence of snapshot checkpointing: execution is at least once per step. Steps should therefore be small, and side effects inside them must be idempotent.

Storage schema and size

A relational schema that works well:

CREATE TABLE checkpoints (
  run_id        text        NOT NULL,
  checkpoint_id uuid        NOT NULL,
  parent_id     uuid,
  step          int         NOT NULL,
  status        text        NOT NULL,      -- running | interrupted | done | failed
  next_nodes    text[]      NOT NULL,
  state         jsonb       NOT NULL,      -- blobs referenced by hash, not inlined
  code_version  text        NOT NULL,
  prompt_hash   text        NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (run_id, checkpoint_id)
);
CREATE INDEX ON checkpoints (run_id, step DESC);
CREATE TABLE pending_writes (
  run_id text, checkpoint_id uuid, task_id text, channel text, value jsonb,
  PRIMARY KEY (run_id, checkpoint_id, task_id, channel)
);

Full snapshots are simplest and make any checkpoint loadable in one read. Their cost is size: a 40-step run with a 200 KB message history writes about 8 MB. Two remedies are storing the message list as content-addressed chunks shared between checkpoints, and pruning old checkpoints of finished runs while keeping the final one. Set a retention period, because checkpoints hold user data, tool outputs and sometimes secrets; encrypt the state column and keep credentials out of state entirely.

Side effects and idempotency

Because a step can run more than once, any external effect in it can happen more than once. Three rules make that safe:

• Derive an idempotency key from run_id:step:call_index and pass it to the downstream API, or record it in a dedupe table checked before acting. The key is stable across re-runs because the step number comes from the checkpoint. See tool-call reliability.
• Put one side effect per step where possible, at the end of the step, so a crash before it means it never happened and a crash after it is caught by the key.
• Never perform a side effect before an interrupt in the same step. When the run resumes after approval, the step starts again from the top and the effect repeats.

The model call is a side effect too, in cost if not in the world. Caching the model response in the pending writes, keyed by step, prevents a resumed step from paying for and possibly diverging on a second completion.

Interrupts, time travel and forks

An interrupt is a step that ends by saving the state with status interrupted and a payload describing what is needed, for example the drafted refund and the reason. The process then exits. When a reviewer approves, edits or rejects, their answer is written to the run and the resume manager continues from that checkpoint with the answer as input. Nothing is held in memory between the two, so the wait can be minutes or weeks. Approval gates covers designing the review itself.

The same chain enables time travel. Every checkpoint has a parent, so you can load checkpoint 12 of a failed run, change its state, for example correcting a bad retrieval, and resume. The new checkpoints get checkpoint 12 as parent, forming a branch while the original history stays intact. This is the most effective way to debug agents: reproduce from the exact state, not from the start.

How LangGraph exposes checkpointing

LangGraph implements this model directly; see LangGraph for orchestration. You pass a checkpointer when compiling the graph (in-memory, SQLite and Postgres implementations exist) and a thread_id in the run configuration. The graph saves a checkpoint at every super-step, the tick in which all scheduled nodes run. If one node fails inside a super-step, the writes of nodes that succeeded are kept as pending writes and are not re-run on resume.

graph = builder.compile(checkpointer=checkpointer)
config = {"configurable": {"thread_id": "invoice-7781"}}
graph.invoke({"invoice_id": "7781"}, config, durability="sync")
# later, after a human decision:
graph.invoke(Command(resume={"approved": True}), config)

The durability option has three modes. "exit" saves only when the run finishes, fails or is interrupted, which is fastest but loses progress on a crash. "async" writes the checkpoint while the next step runs, with a small window of loss. "sync" writes before the next step starts. A node that calls interrupt() re-executes from its beginning when resumed, exactly the side-effect rule above. Check the current documentation for your version, as these APIs have changed across releases.

Worked example: an invoice agent that crashes twice

An invoice agent has five steps: extract fields from the PDF, match the purchase order, request approval if the amount exceeds 5,000, pay, and notify. Walk through a bad day:

1. Steps 1 and 2 run; checkpoints 1 and 2 are committed. Checkpoint 2 holds the extracted fields and the matched order, and points at the approval step.
2. The approval step finds 12,400 and interrupts. Checkpoint 3 has status interrupted and the payload. The worker exits.
3. Two days later a reviewer approves. The resume manager loads checkpoint 3, checks the code version (the agent was redeployed, but the prompt hash and state schema match) and runs the approval step again, which now returns approved. Checkpoint 4 points at pay.
4. The pay step calls the payments API with key inv-7781:4:0. The API succeeds, then the worker is killed before checkpoint 5 commits.
5. A new worker resumes from checkpoint 4 and runs pay again with the same key. The payments API returns the original result instead of paying twice. Checkpoint 5 commits; notify runs; the run ends.

Without the key, step 5 pays twice. With a full transcript dump every five minutes instead of per-step checkpoints, the extraction and matching would also have re-run, at extra cost and with a chance of a different extraction.

Snapshots versus event-sourced replay

Snapshots are simpler and suit agents whose state is mostly the message history. Event sourcing gives a complete audit trail and finer-grained recovery but constrains how you write code. Many systems combine both.

Operations and failure modes

What to watch and the failure modes behind it:

• Resume success rate. Runs that fail on resume usually hit a version or schema mismatch. Record code version and prompt hash in every checkpoint, and decide explicitly whether to resume, migrate or fail older runs after each deploy.
• Checkpoint write p99 and bytes per run. Growing state slows every step; move blobs out and prune.
• Duplicate side effects. Count idempotency-key hits downstream. A non-zero rate is expected; a missing key is a bug.
• Orphaned runs. Status running with no recent checkpoint means a dead worker. A sweeper should lease runs and hand expired leases to a new worker; two workers resuming the same run is prevented by the lease, or by a unique constraint on (run_id, step).
• Stale interrupts. Approvals waiting for weeks need expiry and a default action.

What to do next

To add checkpointing to an agent:

1. List every step and make each one small: one model call or one side-effecting tool call.
2. Define the state schema and move large payloads to an object store referenced by hash.
3. Commit a checkpoint after every step, before the next one starts.
4. Give every external call an idempotency key built from run id, step and call index.
5. Move side effects after interrupts, never before them in the same step.
6. Store code version and prompt hash, and write the resume policy for incompatible checkpoints.
7. Add leases, a sweeper for orphaned runs, retention and encryption.
8. Test by killing the worker at every step boundary and inside every step, and assert that each effect happens exactly once.