Why architecture matters here
Real-time fanout matters because it's the core challenge of any large-scale real-time application -- delivering events to many live subscribers instantly -- and it's deceptively hard at scale (millions of connections, cross-server distribution). Real-time applications (chat, live dashboards, collaboration, feeds) fundamentally need fanout (pushing events to many connected clients instantly). At small scale, this is easy (a few connections on one server). But at large scale (millions of persistent connections across many servers), it's hard: the connections are stateful and numerous (needing sharding across many servers), events must be distributed across all the servers (a backplane), subscribers must be routed to (matching events to interested subscribers), and slow consumers and reconnections must be handled. Getting this architecture right (the connection layer, the backplane, the routing, the scaling) is what makes large-scale real-time possible. For building real-time applications at scale (increasingly common -- collaboration, live features), understanding fanout architecture is essential, and it's the core challenge of real-time systems.
The two-layer (connection-plus-backplane) insight is the architectural core, and it's how fanout scales across servers. The fundamental structure of scalable fanout is two layers. The connection layer: stateful servers each holding a subset of the clients' persistent connections (WebSockets). Since each server can hold only a limited number of connections (memory, file descriptors), millions of connections are sharded across many connection servers (each holding a subset). The pub/sub backplane: an internal distribution layer connecting the connection servers. The problem it solves: an event might originate on any server (or from a backend), but its subscribers are spread across all the connection servers (a topic's subscribers could be connected to any of the servers). So when an event occurs, it must reach all the connection servers that have subscribers for it -- which the backplane does (the event published to the backplane, which distributes it to the relevant connection servers, each then delivering it to its local subscribers). This two-layer design (connection servers holding the sharded connections, a backplane distributing events across them) is what makes fanout scale: the connections scale by sharding (more connection servers), and events reach all subscribers via the backplane (regardless of which server they're on). Understanding the two-layer (connection-plus-backplane) architecture (sharded stateful connections, a backplane distributing events across servers) is understanding how fanout scales across many servers.
And the subscription-routing-plus-backpressure reality is what makes fanout correct and robust, delivering the right events without being overwhelmed by slow consumers. Two concerns make fanout work correctly at scale. Subscription routing: not every subscriber wants every event -- so events must be routed to the right subscribers (matching events to the subscribers interested in them -- e.g., by topic: an event on a topic goes to that topic's subscribers; by channel; by presence). Efficient routing (delivering each event only to its interested subscribers -- not broadcasting everything to everyone) is essential (both for correctness -- subscribers getting the right events -- and efficiency -- not wasting bandwidth delivering irrelevant events). Backpressure: subscribers consume at different rates -- and a slow consumer (a client that can't keep up with the event rate -- a slow connection, an overwhelmed client) is a problem: if the system tries to send to it faster than it can consume, the messages back up (buffering -- consuming memory, and potentially backing up the whole system if unbounded). So backpressure handling is needed: bounding the per-connection buffer (and, when a consumer is too slow, dropping messages, disconnecting it, or applying flow control -- so one slow consumer doesn't consume unbounded resources or back up the system). This -- subscription routing (delivering the right events to the right subscribers, efficiently) and backpressure (handling slow consumers without being overwhelmed) -- is what makes fanout correct (right events to right subscribers) and robust (not overwhelmed by slow consumers) at scale. Understanding subscription routing (efficient, correct delivery) and backpressure (handling slow consumers) is understanding what makes fanout work robustly at scale.
The architecture: every piece explained
Top row: the problem and layers. The problem: broadcasting an event to many connected clients instantly (fanout -- one event to many live subscribers). Connection layer: stateful servers holding the clients' persistent WebSocket connections (each server a subset -- sharded, since each holds a limited number). Pub/sub backplane: the internal distribution layer (distributing events across the connection servers -- since subscribers are spread across all servers -- e.g., Redis pub/sub, Kafka, a dedicated bus). Subscription routing: determining who gets what (matching events to interested subscribers -- by topic, channel, etc. -- delivering the right events to the right subscribers).
Middle row: patterns and robustness. Fanout patterns: the subscription patterns -- topic-based (subscribers to a topic get its events), channel-based, presence-based (who's online) -- how subscribers express interest. Sharding connections: distributing millions of connections across many connection servers (each holding a subset -- since each server is limited) -- scaling the connection layer. Backpressure: handling slow consumers (a client that can't keep up -- bounding its buffer, dropping/disconnecting/flow-controlling -- so it doesn't consume unbounded resources or back up the system). Delivery guarantees: at-most-once (fire-and-forget -- may lose messages) vs at-least-once (retried -- may duplicate) -- the guarantee trade-off (real-time often at-most-once for simplicity/speed, with catch-up for missed messages).
Bottom rows: state and recovery. Presence and state: tracking who's online (presence -- itself a fanout problem, since presence changes must be fanned out to interested subscribers) and shared state -- the real-time state. Reconnect and catch-up: when a client reconnects (after a disconnect -- common with mobile/flaky networks), delivering the messages it missed (catch-up -- from a buffer or log) -- so the client doesn't miss messages across a reconnect. The ops strip: connection scaling (scaling the connection layer -- sharding the millions of connections across servers, and scaling the servers -- the stateful connection scaling), backplane (the pub/sub backplane -- its capacity and reliability, since it distributes all events -- a critical component), and monitoring (monitoring the fanout -- connection counts, event/delivery rates, backpressure/slow consumers, latency -- for health and performance).
End-to-end flow
Trace an event fanned out to subscribers across servers. A chat message is sent to a channel with thousands of subscribers (spread across many connection servers). The message is published to the pub/sub backplane (the internal distribution layer). The backplane distributes it to all the connection servers that have subscribers for that channel (the subscription routing -- the backplane/servers knowing which servers have subscribers for the channel). Each of those connection servers then delivers the message to its local subscribers (the connections it holds that are subscribed to the channel -- pushing the message down their WebSockets). So the message reached all the channel's subscribers (thousands, across many servers) instantly -- via the backplane distributing it to the relevant servers, each delivering to its local subscribers. The two-layer design (the backplane distributing across servers, the connection servers delivering locally) fanned out the message to all subscribers regardless of which server they were on. And the routing ensured only the channel's subscribers got it (not every connection) -- efficient, correct delivery.
The sharding and backpressure vignettes show the scaling and robustness. A sharding case: the system has millions of connections -- far more than one server can hold. So the connections are sharded across many connection servers (each holding a manageable subset -- e.g., 50k connections each) -- scaling the connection layer horizontally (more servers for more connections). The sharding scaled the connections to millions (across the servers). A backpressure case: one subscriber is on a slow connection (can't keep up with the message rate -- e.g., a high-volume channel and a slow mobile connection). The system applies backpressure (bounding that connection's buffer -- and, if it stays too slow, dropping messages or disconnecting it) -- so the slow consumer doesn't back up the system (its buffer bounded -- not consuming unbounded memory, not stalling the fanout for others). The backpressure handling kept the slow consumer from overwhelming the system.
The reconnect and presence vignettes complete it. A reconnect case: a mobile client disconnects (a network blip) and reconnects a few seconds later -- having missed some messages. The catch-up mechanism delivers the missed messages (from a recent-message buffer or log for the channel -- the client requesting messages since its last-seen, getting the ones it missed) -- so the client doesn't miss messages across the reconnect (a good experience despite the flaky network). The reconnect/catch-up handled the missed messages. A presence case: the system tracks presence (who's online in a channel) -- and when a user's presence changes (comes online/goes offline), that change is itself fanned out (to the other subscribers interested in the presence -- e.g., showing who's online) -- presence being a fanout problem too (handled by the same fanout machinery). The consolidated discipline the team documents: architect fanout as two layers (a stateful connection layer holding the sharded WebSocket connections, and a pub/sub backplane distributing events across the servers -- so events reach subscribers regardless of which server they're on), route subscriptions efficiently (delivering the right events to the right subscribers -- by topic/channel/presence), shard the connections (scaling to millions across many servers), handle backpressure (slow consumers -- bounded buffers, dropping/disconnecting -- so they don't overwhelm the system), choose delivery guarantees (at-most-once with catch-up is common for real-time), handle reconnect/catch-up (delivering missed messages), manage presence (itself fanned out), scale the connection layer and the backplane, and monitor the fanout -- because real-time fanout (delivering events to many live subscribers instantly) is the core challenge of large-scale real-time applications, solved by the two-layer connection-plus-backplane architecture with efficient routing, sharding, and backpressure.