Designing a Real-Time Chat System (1M concurrent users)

Connection layer

WebSocket terminators behind a load balancer with sticky sessions. Rule of thumb: ~10,000 concurrent connections per termination process (GC pauses become noticeable above this). So 1M concurrent = ~100 termination nodes. Use a connection registry (Redis) mapping user_id → node_id.

That rule of thumb deserves unpacking, because the number it produces is not a property of WebSocket - it is a property of your runtime. A termination process does almost no work per connection in steady state: it parses a frame header, looks up a destination, and writes bytes. What limits it is the memory it must keep resident per idle socket and, on a managed runtime, the cost of walking whatever object graph represents those sockets during collection. Ten thousand is a conservative figure for a JVM or CLR process with default buffer sizing; event-loop runtimes with small fixed buffers routinely hold considerably more per process, and a process doing TLS termination in userspace holds considerably fewer. Treat the node count as an output of the arithmetic below, not an input.

What an idle connection actually costs

The instinct is to price a connection at the size of a socket structure, which is small and uninteresting. The real bill is buffers, and buffers are a configuration choice rather than a constant - which is why quoting a bytes-per-connection figure without stating the tuning alongside it is meaningless.

Each accepted socket carries a kernel receive queue and a kernel send queue. Linux sizes these dynamically between a floor and a ceiling you set, and the floor is what matters at scale, because an idle chat connection sits at the floor essentially forever. Multiply the receive floor plus the send floor by a million and you have the kernel's resident cost before your application has allocated anything. The ceiling matters too, but differently: it is the exposure when a subset of connections becomes active simultaneously, and it is why autotuning a generous ceiling that is fine for a hundred file-transfer sockets can be alarming across a million chat sockets.

Above the kernel, the application holds its own per-connection state: a read buffer for partial frames, an outbound queue, the identity and subscription metadata that says who this socket belongs to, and - if TLS terminates in your process rather than at a proxy or in the kernel - a session context with its own record buffers, which is typically the single largest per-connection allocation you control. A design that allocates a fixed large outbound buffer per connection because it is convenient will discover the convenience is unaffordable six orders of magnitude later. The pattern that scales is a small default with the ability to grow, and an explicit bound on growth so one slow reader cannot consume the node.

The practical consequence: capacity planning for this tier is a memory calculation, not a CPU or bandwidth calculation. Steady-state chat traffic is trivial in bytes per second per user. You are paying to keep the doors open, not to move the messages.

Descriptors, backlogs, and the limits you hit before memory

Long before a node runs out of RAM it runs out of permission to open files. Every socket is a file descriptor, and the defaults on virtually every distribution are sized for a process handling hundreds of connections, not hundreds of thousands. There are two ceilings and both must be raised: the system-wide maximum and the per-process limit, the latter of which is where service managers quietly override whatever you set in a shell.

# system-wide descriptor ceiling
fs.file-max = 2097152
fs.nr_open  = 2097152

# accept queue depth - a short queue drops connections during a reconnect surge
net.core.somaxconn             = 65535
net.ipv4.tcp_max_syn_backlog   = 65535

# per-socket buffer floor / default / ceiling (bytes)
# the FIRST number is what a million idle sockets each hold
net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.tcp_wmem = 4096 16384 4194304

# outbound source ports - matters for the node's connections to its
# dependencies, NOT for inbound client sockets (see below)
net.ipv4.ip_local_port_range = 10240 65535

# if the node sits behind a stateful firewall or NAT, this table is often
# the first hard wall a connection-heavy service hits
net.netfilter.nf_conntrack_max = 1048576

Per-process, the limit that binds in production is the service manager's, not the login shell's - under systemd that is LimitNOFILE in the unit file, and a process that inherits the default will refuse connections at a few thousand while fs.file-max sits untouched at two million.

The ephemeral port confusion, resolved

It is widely repeated that a server is capped near 65,000 connections by the port range. That is the wrong way round. A TCP connection is identified by the four-tuple of source address, source port, destination address and destination port. Clients arriving at your gateway all share one destination port, but each brings its own source address and source port, so the tuple stays unique and the accepting server is bounded by descriptors and memory - not by 65,535.

The port range does bind, but on the other side of the node: when a gateway opens outbound connections to a dependency, every one of those shares a destination and must take a distinct local source port. A hundred gateways each maintaining a large connection pool to a single registry endpoint is a configuration that can exhaust the local range on a busy node. The fix is pooling and multiplexing toward dependencies, or spreading them across more destination addresses, rather than raising limits.

Finally, the connection model itself: a million sockets requires readiness-based I/O, where one thread watches many descriptors via epoll (or kqueue, or completion ports) and does work only for sockets that have some. Thread-per-connection is not a tuning problem at this scale, it is an architectural dead end - a million kernel threads means a million stacks and a scheduler run queue nobody wants to own.

Why this tier cannot be balanced like a stateless one

A stateless HTTP fleet has a comfortable property: any request may go to any instance, so the balancer can weight, drain and replace instances freely, and the blast radius of removing one is a handful of in-flight requests measured in milliseconds. The connection tier has none of that. Once a socket is established it is pinned to one process for its entire life, which may be hours. The balancer's decision is made once, at connect time, and cannot be revisited.

Three consequences follow, and each surprises teams arriving from stateless services. First, load is distributed by connection count, not by request rate, and those diverge - a node holding many idle sockets and a node holding a few busy ones look identical to a connection-counting balancer. Balancing on connections is the only cheap signal available and it is a proxy for the thing you care about, so expect drift.

Second, a newly added node starts empty and stays comparatively empty. Because existing connections never migrate, a scaled-out node fills only from the arrival rate of new connections. If your churn is low - which is the goal - a node added at peak may take a long time to reach parity, so scaling out is not a response to an immediate overload the way it is for a stateless tier. Capacity must be in place before it is needed.

Third, health checking is asymmetric. A balancer marking a node unhealthy stops sending it new connections, but the sockets already on it are unaffected unless the balancer forcibly resets them. A node that is degraded but still holding several thousand live conversations is a state the stateless model has no equivalent for, and you must decide deliberately whether to sever those connections or leave them on a sick host.

Layer choice interacts with this: an L4 balancer treats the upgraded connection as opaque TCP and is cheap per connection, while an L7 balancer understands the HTTP upgrade and can route on headers at the cost of holding its own per-connection state - and at a million connections the balancer's memory becomes a line item too. The general L4/L7 comparison, algorithms and health-check mechanics are developed in Load Balancing - L4 vs L7, Algorithms, and Health Checks; what is specific here is that stickiness is not an optimisation for cache locality but a structural fact of the protocol.

Deploys, draining, and the reconnect storm

This is the operational reality that most distinguishes the tier, and it is rarely budgeted for. Deploying the connection layer means disconnecting everyone on the node being replaced. There is no graceful equivalent of finishing in-flight requests, because the in-flight request is a conversation that has no natural end. Draining a connection node means asking clients to leave, and they have no incentive to comply.

So a rolling deploy across a hundred nodes is a hundred scheduled disconnection events. Each one hands every affected client back to the balancer at once, and if clients reconnect immediately the arrival rate spikes to the full population of the drained node compressed into whatever your client's retry delay is. That surge lands on the remaining nodes, on the accept queues sized by somaxconn, on the authentication path, and on the registry - all simultaneously.

The compounding failure is well known in shape if not always by name: if the surge is large enough to degrade the nodes receiving it, those nodes drop connections, which produces more reconnects, which enlarges the surge. The tier can fail to recover from an event that was merely a deploy. Mitigations are all about spreading arrivals in time:

drain  -> close with a reason code the client understands
client -> delay = min(cap, base * 2^attempt) * random(0.5, 1.5)
                                               ^^^^^^^^^^^^^^^^
                                  the jitter is the load-bearing part
server -> stagger node drains; never drain a whole zone at once
       -> admission-limit accepts per second during recovery

Exponential backoff alone does not solve this. A population disconnected at the same instant and backing off with identical parameters reconverges on the same retry moments - the herd stays a herd, just at wider intervals. Randomised jitter is what actually decorrelates arrivals, and it is the single line most often omitted from client reconnect logic.

The capacity consequence is the one to take away: this tier must be provisioned for the reconnect peak rather than the steady state. A fleet sized so that steady state fits comfortably can still be unable to absorb the arrival burst caused by losing a single availability zone, because accepting a connection - TLS handshake, authentication, registry write, catch-up read - is enormously more expensive than holding one. Steady state is nearly free; arrivals are not. Reconnection triggered by network changes rather than by deploys, including the case where a client moves between networks and wants to keep its session, is treated separately in connection migration, and the liveness detection that decides a socket is dead in the first place is covered in heartbeat and keepalive.

The registry under churn

The routing question - which node currently holds the socket for a given recipient - is answered by the registry named in the connection layer section above. Its steady-state behaviour is unremarkable: a key per online device, a value naming a node, read on every delivery. What is worth attention here is its behaviour under exactly the conditions described above.

The registry's write rate is proportional to connection churn, not to message volume, which means it peaks precisely when the system is least healthy. A zone loss rewrites an entry for every affected device within seconds of those devices reconnecting, while the delivery path is simultaneously reading it harder than usual. Sizing it for average churn is a mistake of the same shape as sizing the gateway fleet for steady state.

Two properties keep this tractable. Entries must carry a TTL, because a node that dies does not delete its own keys and a registry full of pointers to a dead node sends deliveries into a void; the TTL bounds the staleness window and the heartbeat refreshes it. And entries must be treated as soft state - reconstructible from reconnections rather than authoritative - so that losing the registry entirely costs a period of degraded routing rather than lost messages. Delivery must therefore tolerate a stale pointer: a send to a node that no longer holds the socket has to fail cleanly and fall back to the durable path, not disappear.

How the access pattern shapes the message store

Chat has one of the more lopsided access patterns in common system design, and the schema should be built around it rather than around a generic entity model. Writes are pure appends and never update in place. Reads are overwhelmingly for the newest few dozen entries in one conversation - what the client needs to render a screen - with a long, thin tail of scrollback that is real but rare. There is essentially no aggregation and no ad-hoc query.

That points at a partition key of the conversation and a clustering key of the sequence in descending order, so the rows a client asks for on open are physically adjacent at the head of the partition and retrievable in a single contiguous read with no sort. Descending is deliberate: it makes "the most recent N" and "the N before cursor X" the same cheap operation.

Two wrinkles are worth planning for. Unbounded partitions: a conversation that runs for years grows without limit, and most partitioned stores degrade on very large partitions, so the conversation identifier is usually combined with a time bucket to cap partition size - which trades a slightly more complex scrollback read, occasionally spanning a bucket boundary, for bounded partitions. And deletion is the genuinely awkward case, because append-optimised stores are poor at removing individual rows; retention is best expressed as whole buckets ageing out, and per-message deletion is usually implemented as a tombstone the read path honours rather than as a physical removal.

What the message layer handles, and where it is covered

Everything above concerns keeping sockets open and knowing where they are. Sitting on top is the message semantics layer, which this corpus already develops in depth - each of the following is a full treatment, and this article deliberately does not restate them:

• Message ordering - per-conversation sequence assignment, and why a global order across conversations is neither achievable at reasonable cost nor something any user can perceive.
• Fan-out - delivering one message to many connections, and where the write-time and read-time strategies cross over as group size grows.
• Delivery and read receipts - the sent/delivered/read progression as an explicit state machine per recipient device.
• Presence tracking - heartbeats, TTLs and multi-session reconciliation, plus the fan-out cost that makes presence far more expensive than the feature appears.
• Typing indicators - the deliberately lossy signal, and why durability here would be a bug.
• Offline queues and the handoff to platform push notifications for devices with no live socket.
• Backpressure - bounding the outbound queue so a client that cannot keep up degrades itself rather than the node hosting it.
• Durable ordered logs and offline sync - the end-to-end message path including catch-up by cursor on reconnect.

The cross-node distribution mechanism these depend on is general pub/sub, covered in pub/sub system design and message queues; send-retry deduplication is covered in idempotency.

Being honest about the number in the title

A million concurrent connections is a memorable target and a poor specification. It says nothing about how many of those connections are active in a given second, what the group size distribution looks like, how often clients reconnect, or how much history a client pulls on open - and every one of those drives cost far more than the connection count does.

The mechanisms above are what the number actually decomposes into, and they are what can be reasoned about honestly: per-connection memory dominated by buffer configuration you choose; descriptor and backlog limits that must be raised deliberately; a tier whose instances cannot be swapped without disconnecting users; an arrival rate at deploy and failure time that dwarfs the steady state; and a registry whose load inverts relative to system health. Any of those can be measured on one node and multiplied. Throughput figures quoted without the buffer sizing, the TLS termination point, the runtime, and the churn assumption behind them cannot be, and should be treated as marketing rather than as capacity planning.