Why architecture matters here
The architecture matters because the dual-write problem is not an edge case you can test away — it is a structural gap that manifests exactly when you can least afford it: during crashes, deploys, and network partitions. Any time an agent must change its own state and notify others, and those two facts live in different systems, there is a window between them. Most of the time the window is microseconds and nothing goes wrong. But over millions of operations, and especially during the very incidents that cause crashes, that window is hit, and each hit is a permanent inconsistency: an event owed but never sent, or an event sent for a change that never durably happened. You cannot retry your way out of it after the fact because you do not even know it happened — there is no record of the intent that was lost.
The naive fixes all fail in instructive ways. 'Publish inside the transaction' does not work, because the broker is not transactional with your database — the publish may succeed and the transaction may still roll back. 'Publish after commit' does not work, because the process can die in the gap after commit and before publish, and now the state is durable but the event is lost with no trace. 'Retry the publish until it succeeds' does not work on its own, because if the retry state lives only in memory, a crash loses it too. Every one of these fails because it tries to make two independent systems agree without a shared, durable record of intent. The outbox supplies exactly that record.
The outbox works because it reduces two-system consistency to one-system consistency plus reliable relay. The only atomic thing you need is 'the state changed and the event is recorded,' and that is a single-database transaction — something your relational store has guaranteed for decades. Once the event is durably recorded as a committed row, delivering it is no longer a consistency problem; it is a liveness problem, solvable by an at-least-once relay that keeps trying until the broker acknowledges. You have traded an unsolvable distributed-atomicity requirement for a solvable retry-until-delivered requirement.
There is an operational dividend, too: the outbox makes the publish path observable. Every event that should be sent exists as a row before it is sent, so 'how many events are pending' is a query, 'how far behind is publishing' is the age of the oldest unpublished row, and 'did we ever fail to notify' is answerable by inspection rather than by hope. A dual-write has no such artifact — a lost event leaves nothing behind. An outbox turns the reliability of your event stream from an article of faith into a measurable, alarmable property.
The architecture: every piece explained
Top row: the atomic write. An agent handler — the code that runs when a task completes, a session updates, or a tool call finishes — needs to do two things: change durable state and emit an event. It opens a single database transaction and, within it, writes both the state table row (the updated session or task) and a row into the outbox table describing the event to publish. The outbox row is self-contained: it carries the event type, the serialized payload, a partition/ordering key, a unique event id, and a status of 'pending.' When the transaction commits, both rows are durable together; if it rolls back, neither exists. There is no window, because there is only one commit. This is the entire correctness argument, and it rests on nothing more exotic than the database's own atomicity.
Middle row: the relay. A separate relay process — a poller or a change-data-capture consumer — reads outbox rows that have not yet been published. In the polling variant, it periodically selects pending rows ordered by insertion, publishes each to the message broker (Kafka, Pub/Sub, RabbitMQ), and on broker acknowledgment marks the row published (or deletes it, or advances a CDC offset). In the log-based variant, a CDC connector tails the database's write-ahead log and streams outbox inserts to the broker with no polling at all — lower latency, more infrastructure. Either way, the relay is the only component that talks to the broker, and it is decoupled from the request path: the agent's response returned the moment the transaction committed, long before the event was actually delivered.
The crucial semantic: the relay gives at-least-once delivery, never exactly-once. Consider the relay publishing an event, the broker accepting it, and the relay crashing before it records the row as published. On restart the relay sees the row still pending and publishes again — a duplicate. This is not a bug to be eliminated; it is the fundamental limit of the pattern, and the design accepts it. The consumers — downstream agents reacting to the event — are therefore built to be idempotent: each carries a dedup store keyed by the event's unique id, and a second delivery of the same id is recognized and dropped. At-least-once from the relay plus idempotency at the consumer yields effectively-once processing, which is the achievable goal.
Bottom rows: ordering and operations. Downstream logic often cares about order — a TaskUpdated must not be processed before the TaskCreated it refers to. The outbox supports per-key ordering: rows carry a partition key (the task id, say), the relay publishes a given key's events in outbox insertion order, and the broker preserves per-partition order, so a single entity's events arrive in sequence even though unrelated entities interleave freely. Global total order is neither offered nor usually needed. The ops strip names the properties that keep the outbox healthy: a relay-lag SLO (how stale is the oldest unpublished row), an alarm on outbox depth (a growing backlog means the relay is stuck or the broker is down), a quarantine lane for poison events the relay cannot publish, and a TTL on the consumer dedup store so it does not grow without bound.
End-to-end flow
Trace an event through an adk-java agent that finalizes an order and notifies a fulfillment agent.
The atomic commit: the order agent's handler runs. In one JDBC transaction it updates the task row to completed with the order result and inserts an outbox row: type OrderFinalized, payload the order summary, key the order id, a fresh UUID as the event id, status pending. It commits. The agent returns its response to the caller immediately; as far as the request is concerned, the work is done. Nothing has been published yet, and that is fine — the intent to publish is now durable.
The relay picks it up: a fraction of a second later the relay's poll cycle selects pending outbox rows. It finds the OrderFinalized row, publishes it to the orders topic partitioned by order id, waits for the broker's acknowledgment, and updates the row to published. The event is now in the broker's log, durably, and the outbox row's job is done.
A crash injects a duplicate: suppose instead the relay published the event, the broker accepted it, but the relay process crashed before writing 'published' back to the row. On restart the relay sees the row still pending and publishes it a second time. The broker now holds two copies of OrderFinalized with the same event id. This is expected — at-least-once in action.
The consumer dedupes: the fulfillment agent consumes the orders topic. On the first delivery it checks its dedup store for the event id, does not find it, records it, and processes the order — reserving stock, scheduling a shipment. On the duplicate delivery it checks the dedup store, finds the event id, and drops the message without side effects. Two deliveries, one effect. The consumer's idempotency, not the relay's perfection, is what makes the duplicate harmless.
Ordering holds: later the order agent emits OrderUpdated for the same order. Because it shares the order-id key, it is published to the same partition after OrderFinalized, and the fulfillment agent processes them in order. A completely unrelated order's events interleave on other partitions without any coordination. The end result: the state change and its notification are atomic, the event is delivered at least once, duplicates are absorbed, per-order order is preserved, and every step is a row or an offset an operator can inspect — the dual-write hazard is gone entirely.