Why architecture matters here
Without resumption, a dropped connection forces one of two bad outcomes. Either the client rebuilds state from scratch — a full channel list, a full document, a full order book — which is slow for the user and a self-inflicted load spike for the backend (every blip becomes a cold start), or the client pretends continuity it does not have, silently missing the messages that arrived during the gap. The first is expensive; the second is a correctness bug that surfaces as ghost state: chat messages that never appear, dashboards that drift from reality until the next hard refresh.
Delivery semantics are decided here too. The transport guarantees ordered bytes within one connection, but says nothing across connections. If the server fires-and-forgets, messages in flight at disconnect time are simply gone — at-most-once. If the server naively re-sends recent history without the client saying where it stopped, the client sees duplicates — at-least-once with no dedup. Exactly-once processing across reconnects requires the full loop: server-assigned sequence numbers, client-tracked position, gap replay on resume, and idempotent application keyed by sequence. Each piece is simple; the guarantee only exists when all of them are present.
Scale multiplies the stakes. A single user reconnecting is trivial; a gateway node restarting disconnects hundreds of thousands at once, and they all come back together. Whether that is a non-event or an outage is determined by the architecture: jittered backoff spreads the arrivals, resume-instead-of-rebuild cuts per-connection cost by orders of magnitude, and shedding rules keep the herd from taking down the session store. Reconnect handling is not an edge case at scale — it is the peak-load case.
The architecture: every piece explained
Top row: the path and the state. The client persists a session id and, per stream, the highest contiguous sequence it has processed — in memory for tab-lifetime recovery, in local storage if resume should survive a page reload. The edge/LB must route a resume attempt somewhere that can find the session: either sticky-hash the session id to a gateway, or keep gateways stateless and put the session in a shared store any node can load. The gateway node owns the live socket and the hot session; the session store (Redis or equivalent, with TTL) holds the session's durable core — identity, subscriptions, per-stream watermarks, and optionally the replay buffer itself — so resumes survive gateway death, not just client-side blips.
Middle row: the message-level machinery. Sequence numbers are per-session, per-direction, assigned by the sender at write time; they make loss and duplication detectable, which is the precondition for fixing either. The replay buffer is a ring holding every server-to-client message not yet acknowledged — bounded by count, bytes, and age (say 1,000 messages / 1 MB / 60 seconds), because an unbounded buffer is a memory leak with a business justification. The ack channel lets the client periodically report its contiguous watermark (cumulative acks, TCP-style, so one ack covers many messages), which is what allows the buffer to trim. The backoff scheduler governs re-dial: exponential with full jitter, capped, and reset only after a connection proves healthy for a few seconds — not on connect, or a connect-crash loop hammers at the floor interval.
Bottom row: the two recovery outcomes. The resume handshake is the fast path — client sends RESUME(session_id, last_seq), server validates the session, replays last_seq+1..head, and switches to live flow. The resync fallback is the honest path when the gap outran the buffer: the server answers RESUME_FAIL, and the client performs a full state fetch — the same code path as initial load, which conveniently keeps it tested. Client sequence numbers mirror the scheme upstream so the server can dedup retried client sends; a client message id doubles as an idempotency key.
End-to-end flow
Healthy flow. Client connects, authenticates, receives session_id=S, seq starts at 1. Server numbers every push and appends it to S's replay ring before writing to the socket. The client processes 812 messages, acking its watermark every second or every N messages; each ack trims the ring. Heartbeats run both ways — because TCP will happily report a dead NAT-timeout connection as healthy for minutes, the application-level ping is what actually detects silent death.
The blip. The user walks out of WiFi range mid-scroll. The socket dies; three pushes (813-815) were in flight or unacked, so they sit in the ring. The client's scheduler dials at 1s, 2s, 4s with jitter. Fourteen seconds later the radio is back; the client reconnects — possibly landing on a different gateway behind the LB — and sends RESUME(S, 812) instead of re-authenticating from zero.
Resume. The new gateway loads S from the session store, checks the TTL has not lapsed, and compares 812 against the ring: 813-815 are present. It replays them in order with their original sequence numbers, then emits RESUMED and splices into live traffic. The client applies 813-815, notices nothing else, and the user saw a 14-second silence, not a reload. Any message the client had sent during the gap was queued locally with its own client-seq; it flushes now, and the server dedups if a retry already landed.
Resync. Same story, but the laptop was asleep for an hour. The TTL or the ring has moved on; the server returns RESUME_FAIL. The client tears down local stream state and runs initial-load: snapshot fetch, new session, fresh sequences. The design goal is a hard, explicit boundary between the two outcomes — the deadly zone is a resume that silently skips a gap it could not fill.