Why architecture matters here
Ordering matters because a startling number of application bugs are really ordering bugs in disguise. A chat UI shows a reply before the message it answers. A collaborative editor applies an insert before the delete that should precede it and the document corrupts. A trading feed processes a cancel before the order it cancels and rejects it as unknown. A state-machine replica diverges because two updates landed in different sequences on two nodes. In every case the individual messages were correct; only their order was wrong. An ordering layer exists to make 'the order the receiver sees' a well-defined, reliable function of 'the order the sender intended', so applications can reason about causality instead of defending against chaos.
Three properties define a sound ordering layer. Well-defined order: the receiver delivers messages in a specified sequence (total, or per-key) that the sender controls, not whatever order packets happened to arrive. Gap-freedom on delivery: the application never sees message N+1 before message N within an ordered scope — out-of-order arrivals are held until the gap fills, and genuine loss is repaired or surfaced, never silently skipped. And exactly-once effect: duplicates from retransmission or reconnection are detected and dropped so the application processes each message's effect once, even though the wire may carry it more than once.
The trade-off you operate around is ordering strength versus head-of-line blocking. The stronger the ordering scope, the more one missing message stalls everything behind it: under strict total order, a single lost packet freezes the entire stream until it is recovered. Weaken the scope to per-key and only the affected key stalls while every other key flows. Weaken it further to no ordering and nothing blocks but the application must tolerate arbitrary sequence. The engineering job is to pick the weakest ordering that the application's correctness actually requires — almost always per-key — and then implement that faithfully, rather than paying for total order the application never needed or shipping no order and pushing the problem upstream.
The architecture: every piece explained
Top row: the send path. The producer stamps each message with a sequence number — monotonic within its ordering scope (global for total order, per-key for per-key order). The send buffer holds every message that has not yet been acknowledged, because until the receiver confirms it, the message may need retransmission; this buffer is what makes the stream reliable on top of a transport that can drop. The transport — a reliable stream like a WebSocket or gRPC channel, or an unreliable datagram path — carries the bytes, and may reorder or lose them. The receive buffer is the heart of reordering: a window that collects arrivals and holds those that came early until the messages before them arrive.
Middle row: correctness on the receive side. The gap detector notices when an expected sequence number is missing — a hole in the window — and issues a NACK (negative acknowledgement) asking the sender to retransmit, rather than waiting for a slow timeout. Dedup drops replays: a message whose sequence number was already delivered is discarded, so a retransmission that crossed with its ack, or a reconnection that resent recent history, does not duplicate an effect. The deliverer releases messages to the application strictly in order — it advances only when the next contiguous sequence number is present, so the application by construction never sees a gap. The ack/cursor reports progress back: a cumulative ack ('I have everything up to N') plus optional selective acks ('and also these later ones') lets the sender free its buffer and retransmit precisely the missing pieces.
Bottom rows: the scaling lever and the consumer. Per-key ordering is what keeps this affordable: the stream is logically partitioned by a key (conversation id, entity id, session), each key carries its own sequence space, and ordering is guaranteed within a key while different keys interleave freely. That confines head-of-line blocking to a single key when a message is lost. The application then consumes a stream it can trust to be consistently ordered within each key. The ops strip lists the health signals: buffer depth (how much is held awaiting reorder), reorder latency (how long messages wait for gaps to fill), gap/NACK rate (how often loss forces recovery), and ack lag (how far behind acknowledgement runs) — a rising buffer or reorder latency is the early sign of a stall.
It is worth naming what this layer deliberately does not do, because scope discipline is half the design. It does not impose ordering across keys — that is the whole point of partitioning — so two independent conversations make progress irrespective of each other's gaps. It does not guarantee anything about messages the producer never sent or the application never acknowledged consuming; delivery-in-order is a promise about sequence, not about semantics. And it does not, by itself, decide what a key is — that choice belongs to the application, and it is the single most consequential knob, because too coarse a key (say, one per user across all their conversations) reintroduces head-of-line blocking, while too fine a key (one per message) makes ordering meaningless. The sweet spot is the smallest unit within which causality actually matters — a document, a session, an entity — and getting it right is more important than any amount of buffer tuning, because it sets the ceiling on both the parallelism you can keep and the blast radius of a single lost message.
End-to-end flow
Follow a collaborative-editing session over a bidi stream, ordered per document. A user types three edits in a row; the producer stamps them seq 41, 42, 43 for that document's key and hands them to the send buffer, which keeps copies pending acknowledgement. On the wire, 41 and 43 arrive at the receiver but 42 is delayed by a momentary network hiccup. The receive buffer accepts 41 and delivers it — it is the next contiguous number — but holds 43, because delivering it now would show the application an edit out of order. The window has a gap at 42.
The gap detector notices the hole immediately and fires a NACK for 42 rather than waiting for the sender's retransmit timer. The sender, still holding 42 in its send buffer, retransmits it. It arrives, the receive buffer now has a contiguous 41-42-43, and the deliverer releases 42 and then 43 to the application in order. The document applies the three edits in the sequence the user intended, and the receiver sends a cumulative ack for 43, letting the sender free all three from its buffer. From the application's perspective nothing went wrong — it saw 41, 42, 43 — even though the wire delivered them 41, 43, 42.
Now the reconnection path, where duplicates come from. Suppose the connection drops after 42 was delivered but before its ack reached the sender. On reconnect, the client tells the server its cursor is at 42; the server, unsure whether 43 was received, resends 43 — and possibly 42 again if its records are stale. The receiver's dedup drops the duplicate 42 (already delivered) and accepts 43 exactly once. This is the exactly-once effect in action: the wire carried 42 twice, but the application processed its effect once. Without dedup, the editor would apply the same insert twice and corrupt the document — the classic reconnection-duplicates bug.
Consider the head-of-line stress path, and why per-key ordering earns its keep. Imagine the same server multiplexing edits for a thousand documents over one connection. If ordering were total across the whole stream, the lost message 42 for one document would stall delivery for all thousand documents until it was recovered — a single slow packet freezing every user. Because ordering is scoped per document key, only that one document's edits wait at the gap; the other 999 keys keep delivering uninterrupted. The buffer depth for the stalled key rises briefly and its reorder latency spikes, both visible in the metrics, while the system as a whole stays responsive. That confinement of blocking is the entire reason to prefer per-key over total order whenever the application allows it.
Now push on the hardest case: the producer itself restarts mid-stream. When a sender process crashes and comes back, its in-memory sequence counter is gone, and if it naively restarts numbering from a low value, the new seq 41 aliases the old seq 41 that the receiver already delivered — dedup would silently drop a genuinely new message as a supposed replay, which is far worse than a duplicate. This is why the epoch matters: every producer incarnation stamps its messages with a generation id, so the receiver treats (epoch 7, seq 41) and (epoch 8, seq 41) as unambiguously different, resetting its expected-sequence tracking at each new epoch while still deduplicating within one. The reconnection handshake carries both the epoch and the last delivered sequence, so producer and receiver agree on exactly where to resume. Flow control interlocks with all of this: the receiver's advertised window and the ack cursor together throttle a producer that would otherwise outrun the reorder buffer, so backpressure and ordering are not separate concerns but two views of the same bounded window. Get the epoch and the window right and a stream survives crashes, reconnects, and slow consumers while still delivering every message exactly once, in order, per key — which is the entire promise the layer exists to keep.