Agent outbox relay: crash points, duplicates and resume

What this experiment demonstrates

A support agent runs a three-step plan for order 7731: look the order up, issue a refund of 40, and let the accounting ledger know. The refund step is the one with effects. Its handler writes the refund row, the step journal row and an outbox row describing the event; a relay then publishes the event to a broker, and a ledger consumer applies it. The page models the handler, the relay and the consumer, lets you choose where a process dies, and reports what the stores contain afterwards: how many refund rows exist, how many ledger entries were posted, and whether the run resumed cleanly.

Nothing here calls a language model. The agent is a fixed policy that always proposes the same three steps and retries a tool call that returned no answer, which is what a real model loop does. Every difference in the outcome comes from a control you changed.

Run controlled comparisons

Start with the defaults: one transaction for the three rows, the relay publishing before it marks, consumer deduplication on, a step key on the tool call and resume from the journal, with no crash. One refund, one ledger entry, one message. Now make the handler die before the commit: the transaction rolls back, resume finds the step pending and runs it again, still one refund. Switch the single transaction off and repeat: the refund row survives without a journal or outbox row, the agent retries, and a second refund is issued that nobody downstream hears about.

Move the crash to after the commit and turn on the agent retry: the agent never saw an answer, so it calls the tool again. With the step key the journal answers the retry and there is still one refund; without it there are two. Crash the relay between its two actions. With publish-then-mark the message is delivered twice and the consumer, if it deduplicates, applies it once; switch deduplication off and the ledger is posted twice. With mark-then-publish the message is lost. Finally switch resume off with a crash after the commit: the run restarts from scratch and only the step key stands between it and a second refund.

Notice what does not change. The agent retries in every run where it saw no answer. What decides the outcome is whether the runtime recorded the step with the effect, whether the key lets a retry find that record, and whether delivery is at least once with deduplication at the consumer.

Inspect the trace and keep evidence

The store table lists the refund rows, journal rows, outbox rows with their published state, the messages the broker delivered and the ledger entries, with the rows that violate an invariant marked. Metrics count refunds issued, messages delivered, ledger entries, and three invariant violations: duplicate refunds, lost events and duplicate ledger postings, plus the number of journal resumes. Start trace, Next step, Show all and the slider control how much of the trace is visible; the metrics always describe the whole run. Reset restores the defaults and Export JSON downloads the inputs, the store contents and the trace.

Limits and production translation

One run, one effecting step and one consumer are modelled; real systems have many concurrent runs, several effecting tools and consumers with their own failure modes. Crash points are chosen deterministically rather than by timing, the broker never reorders, and the relay delay is a single number. What carries over is structural: the effect, the journal row and the outbox row commit together; the step key is derived from the run and step, never from the arguments; the relay publishes before it marks; and the consumer deduplicates on the outbox id inside its own transaction. Measure your own crash points with fault injection in staging, after the business write, after the commit and between the relay actions.