Why architecture matters here
The reason chat is an architecture problem and not a CRUD app is that its correctness guarantees are unforgiving and its load profile is spiky. A dropped message is not a degraded experience; it is a broken product — users trust that what they send arrives, and a single lost message erodes that trust permanently. A duplicated message is nearly as bad, turning one 'I love you' into two and one '$500 transferred' into confusion. And a reordered conversation — where a reply appears above the message it answers — is incoherent. These are not tail concerns to optimize later; they are the product.
The first structural challenge is multi-device consistency. A modern user has a phone, a laptop, and a tablet, all of which must show the same conversation in the same state. A message is not delivered to a user; it is delivered to every device the user owns, and each device independently acknowledges receipt and tracks how far it has read. The system's unit of delivery is therefore the (conversation, device) pair, and the state it must maintain is 'which messages has each device seen and acknowledged.'
The second challenge is the real-time/durable seam. When a message is sent, it must be persisted to durable storage before the sender is told it succeeded — otherwise a server crash between 'accepted' and 'stored' silently loses it. Only after the write is durable is the message fanned out over live sockets to online devices. Offline devices get nothing in real time; instead, the durable log is the source of truth they replay on reconnect. This is why the message store, not the socket, is the true system of record — the socket is an optimization for the online case.
The third challenge is ordering under concurrency. When two members of a conversation send simultaneously, the system must pick one order and make every device agree on it. It does this by funneling every message for a conversation through a single serialization point that assigns a monotonically increasing sequence number. The sequence, not the timestamp, defines order; devices sort by sequence and converge. This turns the distributed problem 'what order did things happen in' into the local problem 'replay the log by sequence,' which every device can do independently and get the same answer.
The architecture: every piece explained
Start at the edge. The client holds a persistent WebSocket to a gateway at the edge whose sole job is connection and session management: it terminates the socket, authenticates the device, tracks which user and device this connection belongs to, and maps the abstract 'deliver to device D' request to 'push down this specific socket.' Gateways are stateful about connections but stateless about business logic, so they scale horizontally and a device can reconnect to any of them; a routing layer knows which gateway currently holds a given device's socket.
The message service is the write path. When a client sends a message, the service validates it (membership, size, rate limits), assigns it the next per-conversation sequence number, and appends it to the message store — a durable, ordered, append-only log partitioned by conversation. The sequence assignment and the append are the serialization point: whatever order messages hit this point is the order the whole conversation will see. Only once the append is durable does the service acknowledge the send back to the author's device, so the sender's 'sent' checkmark means 'persisted,' not merely 'received by a server that might crash.'
The fan-out / delivery component takes the persisted message and routes a copy to every recipient device. For a one-to-one chat this is two devices; for a group it is every member's every device. Fan-out consults the presence set to learn which devices are online (and thus reachable over a live socket via their gateway) and which are offline (and must be reached via mobile push and durable catch-up). For each recipient it also updates the inbox state — the unread count and the pointer to the newest message — so a device that has been offline knows how much it missed.
The ordering and dedup box is the correctness backbone. Every message carries its conversation sequence, so a device sorts incoming messages by sequence regardless of the order the network delivered them. Every message also carries a client-generated idempotency key, so if the author's device retries a send it did not see acknowledged, the service recognizes the duplicate and returns the original sequence instead of appending twice. Sequence gives ordering; the idempotency key gives exactly-once. The push and sync-on-reconnect box handles the offline case: a reconnecting device tells the service the highest sequence it has seen per conversation, and the service streams everything after that, so catch-up is a range read on the log — lossless and duplicate-free by construction.
The bottom ops strip names the pressures that decide whether the system survives real traffic: backpressure when a slow device cannot keep up, hot conversations (a viral group where one message fans out to millions), read receipts and their own fan-out cost, and retention policy for how long the durable log is kept.
One subtlety worth making explicit is how the gateway and the routing layer cooperate to keep delivery cheap. Because a device may reconnect to any gateway after a network blip, the system maintains a lightweight registry mapping each online device to the gateway currently holding its socket. Fan-out consults this registry to turn 'deliver to device D' into 'send to gateway G, which owns D's connection right now,' avoiding a broadcast to every gateway. When a device disconnects, its registry entry is torn down and delivery falls back to the durable path; when it reconnects, a fresh entry is written and the sync-on-reconnect protocol runs. This registry is soft state — it can be rebuilt from reconnections — so losing a gateway costs its devices a reconnect and a catch-up, not a lost message, because the message store, not the registry, remains the system of record.
End-to-end flow
Trace one message through a small group chat of three members — Ada, Ben, and Cara — where Ada and Ben are online and Cara's phone is offline. Ada sends 'meeting at 4?'
Send and persist. Ada's client attaches a client-generated idempotency key and pushes the message up her WebSocket to her gateway, which forwards it to the message service. The service validates Ada's membership, assigns the next sequence number for this conversation — say 1047 — and appends the message to the conversation's durable log. Once the append is confirmed durable, the service acknowledges sequence 1047 back to Ada's device, which flips her message from 'sending' to 'sent.' The message now exists as the system of record regardless of what happens to any socket.
Fan-out to the online devices. Delivery looks up presence: Ben is online, so his device is reachable via a live socket on his gateway; the service routes message 1047 to that gateway, which pushes it down Ben's socket. Ben's client sorts it into the conversation by sequence and renders it, then sends a delivery acknowledgment so the system knows Ben's device has it. Ada is also updated that Ben received it, driving the 'delivered' indicator.
Durable delivery to the offline device. Cara is offline, so there is no socket to push down. Delivery instead increments Cara's unread count for this conversation and enqueues a mobile push notification ('Ada: meeting at 4?') via the platform push service. The message itself is already safe in the durable log; the push is only a nudge. When Cara's phone comes back online hours later, her client reconnects to some gateway and announces the highest sequence it has for each conversation — say 1042 for this one. The service reads the log range 1043–1047 and streams those five missed messages in order. Cara's device replays them by sequence and converges to exactly the same conversation view Ada and Ben already have, with no gaps and no duplicates.
The idempotency safeguard. Suppose Ada's original send timed out before her device saw the acknowledgment, so her client retried with the same idempotency key. The service sees the key already maps to sequence 1047, returns that sequence again instead of appending a second copy, and Ada sees one message, not two. Ordering came from the sequence; exactly-once came from the key; durability came from persisting before acknowledging — the three guarantees composing into a conversation that every device agrees on.