Why architecture matters here
The outbox matters because it targets the exact seam where distributed systems lie to themselves: the boundary between a database and a message broker, two systems with no shared transaction. There is no two-phase commit you can rely on between an arbitrary database and Kafka in practice, and even where XA exists it is slow, fragile, and operationally toxic. So any design that says 'commit here, then publish there' has a window — however small — where a crash leaves the two systems disagreeing. At low volume you might never notice; at millions of events a day, a one-in-ten-thousand failure rate is hundreds of lost or phantom events daily, each one a silent data-integrity bug that surfaces days later as a support ticket nobody can explain.
The outbox reframes the problem from 'coordinate two systems' to 'use one system's atomicity, then move the result reliably.' The event is just another row in the same database, so writing it is covered by the same transaction and the same durability guarantees as the business change — if the order commits, the event is guaranteed to exist; if the transaction rolls back, the event never existed. This collapses the impossible distributed-atomicity problem into a trivially-solved local one. What remains is the much easier problem of getting rows from a durable table to a broker, which is a retry-until-success delivery task, not an atomicity task.
The architectural trade you accept is at-least-once delivery and eventual propagation. The relay might publish a row, crash before marking it sent, and publish it again on restart — so consumers can see duplicates and must be idempotent. And there is a small delay between the commit and the event reaching consumers (poll interval or CDC lag). In exchange you get the guarantee that matters most: no event is ever lost, and no event is ever published for a change that did not happen. For the vast majority of systems, 'every real change is eventually announced exactly the right number of logical times, with possible physical duplicates the consumer dedupes' is precisely the contract you want — and it is achievable, whereas true distributed exactly-once across a DB and a broker is not.
The architecture: every piece explained
Top row: the atomic write. The service handles an incoming command. Within a single database transaction it updates the business table (insert the order, debit the account) and, in the same transaction, inserts a row into the outbox table. That outbox row carries everything needed to publish later: a unique event id, the aggregate/entity id, an event type, the serialized payload, a creation timestamp, and a status or sequence field. Because both writes are in one transaction, the database's atomicity guarantees they commit together or not at all — this single fact is the whole safety property of the pattern.
Middle row: getting events out. There are two relay strategies. The polling relay is a background worker that periodically queries the outbox for unsent rows (WHERE published = false ORDER BY id), publishes each to the message broker, and marks it sent (or deletes it). It is simple and works on any database, at the cost of poll latency and load. The more scalable strategy is CDC log tailing: a change-data-capture connector (Debezium and friends) tails the database's write-ahead log, so every committed insert into the outbox table appears as a change event that is streamed to the broker with no polling and minimal lag — the WAL is already an ordered, durable record of commits, so you are just forwarding it. Either way, consumers receive events with at-least-once semantics and must deduplicate.
Bottom rows: the properties consumers depend on. Idempotency is mandatory: each event carries a stable event id, and every consumer records the ids it has processed (or writes with an upsert keyed by the id) so a re-delivered event is a no-op. Ordering, when needed, is preserved per aggregate: rows carry a monotonic sequence or the CDC tail preserves commit order, and the broker is partitioned by aggregate id so all events for one entity land on the same partition in order — you rarely need or want global ordering, only per-entity. The ops strip names what you actually watch: relay lag (how stale the event stream is), outbox table growth (unpublished rows piling up, or old published rows never cleaned), poison events (a row that fails to publish or deserialize forever), and the consumer-side exactly-once/dedupe mechanism.
End-to-end flow
Trace a 'place order' command through an outbox-based order service. The request arrives; the handler opens a database transaction. It inserts the order row into orders, decrements inventory in stock, and inserts an outbox row: {event_id: uuid, aggregate_id: order-8842, type: OrderPlaced, payload: {...}, seq: 1, published: false}. It commits. At this instant the guarantee is locked in: the order exists and the event exists, atomically. The HTTP response returns success to the client. Notice the request path never touched the broker — no dependency on Kafka being up for the write to succeed, which also makes the write faster and more available.
Asynchronously, the relay does its job. In the CDC variant, Debezium is tailing the WAL; the committed outbox insert appears in the log stream within milliseconds, and the connector publishes it to the orders Kafka topic, partitioned by aggregate_id so all events for order-8842 stay ordered on one partition. In the polling variant, a worker running every 200ms picks up the unsent row, publishes it, and sets published = true. A downstream shipping service consumes OrderPlaced, checks its processed-ids table, sees this event id is new, creates a shipment, and records the id — all in its own local transaction, so its consume is idempotent too.
Now the failure that makes the pattern earn its keep. Suppose the relay publishes the OrderPlaced event to Kafka successfully but the worker process crashes before it can mark the row published = true (or, in CDC, before it commits its offset). On restart, the relay sees the row still marked unsent and publishes it again. The shipping consumer receives the duplicate, looks up the event id in its processed-ids table, finds it already handled, and drops it — no second shipment. The duplicate was harmless precisely because the consumer is idempotent, which is the price the outbox asks you to pay for never losing an event. Contrast this with the dual-write design: there, a crash between commit and publish loses the event permanently, and no amount of consumer cleverness can recover a message that was never sent.
One more scenario: the broker is down for ten minutes during a deploy. With the outbox, order writes keep succeeding the entire time — they only touch the database — and the unpublished rows simply accumulate in the outbox table. When the broker returns, the relay drains the backlog in order, and consumers catch up. No orders were rejected, no events were lost; the outage became a bounded, self-healing delay instead of a data-integrity incident. That decoupling of the write path from broker availability is a second, underrated benefit of the pattern.