Connection layer

WebSocket terminators behind a load balancer with sticky sessions. Rule of thumb: ~10,000 concurrent connections per termination process (GC pauses become noticeable above this). So 1M concurrent = ~100 termination nodes. Use a connection registry (Redis) mapping user_id → node_id.

That rule of thumb deserves unpacking, because the number it produces is not a property of WebSocket - it is a property of your runtime. A termination process does almost no work per connection in steady state: it parses a frame header, looks up a destination, and writes bytes. What limits it is the memory it must keep resident per idle socket and, on a managed runtime, the cost of walking whatever object graph represents those sockets during collection. Ten thousand is a conservative figure for a JVM or CLR process with default buffer sizing; event-loop runtimes with small fixed buffers routinely hold considerably more per process, and a process doing TLS termination in userspace holds considerably fewer. Treat the node count as an output of the arithmetic below, not an input.

Advertisement

What an idle connection actually costs

The instinct is to price a connection at the size of a socket structure, which is small and uninteresting. The real bill is buffers, and buffers are a configuration choice rather than a constant - which is why quoting a bytes-per-connection figure without stating the tuning alongside it is meaningless.

Each accepted socket carries a kernel receive queue and a kernel send queue. Linux sizes these dynamically between a floor and a ceiling you set, and the floor is what matters at scale, because an idle chat connection sits at the floor essentially forever. Multiply the receive floor plus the send floor by a million and you have the kernel's resident cost before your application has allocated anything. The ceiling matters too, but differently: it is the exposure when a subset of connections becomes active simultaneously, and it is why autotuning a generous ceiling that is fine for a hundred file-transfer sockets can be alarming across a million chat sockets.

Above the kernel, the application holds its own per-connection state: a read buffer for partial frames, an outbound queue, the identity and subscription metadata that says who this socket belongs to, and - if TLS terminates in your process rather than at a proxy or in the kernel - a session context with its own record buffers, which is typically the single largest per-connection allocation you control. A design that allocates a fixed large outbound buffer per connection because it is convenient will discover the convenience is unaffordable six orders of magnitude later. The pattern that scales is a small default with the ability to grow, and an explicit bound on growth so one slow reader cannot consume the node.

The practical consequence: capacity planning for this tier is a memory calculation, not a CPU or bandwidth calculation. Steady-state chat traffic is trivial in bytes per second per user. You are paying to keep the doors open, not to move the messages.

Advertisement

Descriptors, backlogs, and the limits you hit before memory

Long before a node runs out of RAM it runs out of permission to open files. Every socket is a file descriptor, and the defaults on virtually every distribution are sized for a process handling hundreds of connections, not hundreds of thousands. There are two ceilings and both must be raised: the system-wide maximum and the per-process limit, the latter of which is where service managers quietly override whatever you set in a shell.

# system-wide descriptor ceiling
fs.file-max = 2097152
fs.nr_open  = 2097152

# accept queue depth - a short queue drops connections during a reconnect surge
net.core.somaxconn             = 65535
net.ipv4.tcp_max_syn_backlog   = 65535

# per-socket buffer floor / default / ceiling (bytes)
# the FIRST number is what a million idle sockets each hold
net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.tcp_wmem = 4096 16384 4194304

# outbound source ports - matters for the node's connections to its
# dependencies, NOT for inbound client sockets (see below)
net.ipv4.ip_local_port_range = 10240 65535

# if the node sits behind a stateful firewall or NAT, this table is often
# the first hard wall a connection-heavy service hits
net.netfilter.nf_conntrack_max = 1048576

Per-process, the limit that binds in production is the service manager's, not the login shell's - under systemd that is LimitNOFILE in the unit file, and a process that inherits the default will refuse connections at a few thousand while fs.file-max sits untouched at two million.

The ephemeral port confusion, resolved

It is widely repeated that a server is capped near 65,000 connections by the port range. That is the wrong way round. A TCP connection is identified by the four-tuple of source address, source port, destination address and destination port. Clients arriving at your gateway all share one destination port, but each brings its own source address and source port, so the tuple stays unique and the accepting server is bounded by descriptors and memory - not by 65,535.

The port range does bind, but on the other side of the node: when a gateway opens outbound connections to a dependency, every one of those shares a destination and must take a distinct local source port. A hundred gateways each maintaining a large connection pool to a single registry endpoint is a configuration that can exhaust the local range on a busy node. The fix is pooling and multiplexing toward dependencies, or spreading them across more destination addresses, rather than raising limits.

Finally, the connection model itself: a million sockets requires readiness-based I/O, where one thread watches many descriptors via epoll (or kqueue, or completion ports) and does work only for sockets that have some. Thread-per-connection is not a tuning problem at this scale, it is an architectural dead end - a million kernel threads means a million stacks and a scheduler run queue nobody wants to own.

Why this tier cannot be balanced like a stateless one

A stateless HTTP fleet has a comfortable property: any request may go to any instance, so the balancer can weight, drain and replace instances freely, and the blast radius of removing one is a handful of in-flight requests measured in milliseconds. The connection tier has none of that. Once a socket is established it is pinned to one process for its entire life, which may be hours. The balancer's decision is made once, at connect time, and cannot be revisited.

Three consequences follow, and each surprises teams arriving from stateless services. First, load is distributed by connection count, not by request rate, and those diverge - a node holding many idle sockets and a node holding a few busy ones look identical to a connection-counting balancer. Balancing on connections is the only cheap signal available and it is a proxy for the thing you care about, so expect drift.

Second, a newly added node starts empty and stays comparatively empty. Because existing connections never migrate, a scaled-out node fills only from the arrival rate of new connections. If your churn is low - which is the goal - a node added at peak may take a long time to reach parity, so scaling out is not a response to an immediate overload the way it is for a stateless tier. Capacity must be in place before it is needed.

Third, health checking is asymmetric. A balancer marking a node unhealthy stops sending it new connections, but the sockets already on it are unaffected unless the balancer forcibly resets them. A node that is degraded but still holding several thousand live conversations is a state the stateless model has no equivalent for, and you must decide deliberately whether to sever those connections or leave them on a sick host.

Layer choice interacts with this: an L4 balancer treats the upgraded connection as opaque TCP and is cheap per connection, while an L7 balancer understands the HTTP upgrade and can route on headers at the cost of holding its own per-connection state - and at a million connections the balancer's memory becomes a line item too. The general L4/L7 comparison, algorithms and health-check mechanics are developed in Load Balancing - L4 vs L7, Algorithms, and Health Checks; what is specific here is that stickiness is not an optimisation for cache locality but a structural fact of the protocol.

Deploys, draining, and the reconnect storm

This is the operational reality that most distinguishes the tier, and it is rarely budgeted for. Deploying the connection layer means disconnecting everyone on the node being replaced. There is no graceful equivalent of finishing in-flight requests, because the in-flight request is a conversation that has no natural end. Draining a connection node means asking clients to leave, and they have no incentive to comply.

So a rolling deploy across a hundred nodes is a hundred scheduled disconnection events. Each one hands every affected client back to the balancer at once, and if clients reconnect immediately the arrival rate spikes to the full population of the drained node compressed into whatever your client's retry delay is. That surge lands on the remaining nodes, on the accept queues sized by somaxconn, on the authentication path, and on the registry - all simultaneously.

The compounding failure is well known in shape if not always by name: if the surge is large enough to degrade the nodes receiving it, those nodes drop connections, which produces more reconnects, which enlarges the surge. The tier can fail to recover from an event that was merely a deploy. Mitigations are all about spreading arrivals in time:

drain  -> close with a reason code the client understands
client -> delay = min(cap, base * 2^attempt) * random(0.5, 1.5)
                                               ^^^^^^^^^^^^^^^^
                                  the jitter is the load-bearing part
server -> stagger node drains; never drain a whole zone at once
       -> admission-limit accepts per second during recovery

Exponential backoff alone does not solve this. A population disconnected at the same instant and backing off with identical parameters reconverges on the same retry moments - the herd stays a herd, just at wider intervals. Randomised jitter is what actually decorrelates arrivals, and it is the single line most often omitted from client reconnect logic.

The capacity consequence is the one to take away: this tier must be provisioned for the reconnect peak rather than the steady state. A fleet sized so that steady state fits comfortably can still be unable to absorb the arrival burst caused by losing a single availability zone, because accepting a connection - TLS handshake, authentication, registry write, catch-up read - is enormously more expensive than holding one. Steady state is nearly free; arrivals are not. Reconnection triggered by network changes rather than by deploys, including the case where a client moves between networks and wants to keep its session, is treated separately in connection migration, and the liveness detection that decides a socket is dead in the first place is covered in heartbeat and keepalive.