Why architecture matters here
The load balancer exists to decouple the client's view of a service from the reality of the fleet behind it. Backends are cattle: they crash, they get patched, they scale in and out, a deploy replaces all of them. If clients addressed individual instances, every one of those routine events would be an outage. By interposing a stable virtual address and continuously routing only to healthy backends, the load balancer turns a fleet of unreliable machines into a reliable service — and, simultaneously, spreads load so no single backend is overwhelmed while others sit idle. Availability and capacity are two sides of the same mechanism.
Three properties define a good load balancer. Health-awareness: it routes only to backends that are actually serving correctly, detecting and ejecting failures fast — a load balancer that keeps sending traffic to a dead or sick instance is worse than none. Even distribution: it spreads load so utilization is balanced across the fleet and across zones, because a lopsided distribution wastes capacity and creates hot spots that fail first. And graceful membership change: instances can join and leave — for scaling, deploys, or repair — without dropping in-flight requests, because draining lets a departing backend finish what it started before it is removed.
The trade-off you operate around is aggressiveness versus stability in health decisions. Eject a backend too eagerly on a single failed probe and a transient blip removes healthy capacity, concentrating load on the survivors and potentially cascading. Eject too reluctantly and a genuinely broken backend keeps receiving and failing requests, hurting users. The same tension runs through balancing: react hard to load imbalance and you risk oscillation as everyone piles onto whichever backend just looked least loaded; react slowly and imbalance persists. The engineering job is to make health and load decisions responsive enough to protect users but damped enough that the load balancer never amplifies a small problem into a fleet-wide one.
The architecture: every piece explained
Top row: the request path across layers. A client resolves the service's anycast virtual IP — the same address advertised from many locations, so the network routes each client to a near edge, giving geographic distribution and DDoS absorption for free. The L4 edge distributes incoming connections across the L7 fleet without inspecting payloads — fast, high-throughput, protocol-agnostic flow distribution. The L7 proxy does the intelligent work: it terminates TLS, parses the request, routes on host/path/header, adds and strips headers, and applies per-request policy like retries and timeouts. Behind it sits the backend pool: the actual service instances, deliberately spread across multiple availability zones so the loss of one zone removes only a fraction of capacity.
Middle row: the machinery that keeps routing healthy. Active health checks probe each backend on an interval and eject any that fail — the primary defence against routing to a dead instance. The algorithm decides which healthy backend gets each request: round-robin (simple, even under uniform work), least-connections (favours backends with fewer in-flight requests, good under variable request cost), or latency-aware EWMA (routes by a smoothed estimate of each backend's recent response time). Outlier detection is the passive complement to health checks: it watches real traffic and ejects a backend that starts returning errors or slow responses even before an active probe fails, catching partial failures fast. The autoscaler grows and shrinks the pool so capacity tracks load rather than being provisioned for peak at all times.
Bottom rows: the membership and stickiness controls. Session affinity pins a client to a backend when the workload genuinely needs it — a legacy in-memory session, a stateful connection — using a cookie or a consistent hash; but it is a constraint on balancing, used sparingly, because it undermines even distribution. Draining handles graceful membership change: when a backend is being removed (scale-in, deploy, repair) it stops receiving new requests but is allowed to finish in-flight ones before it leaves, so no user request is cut off mid-flight. The ops strip lists what you watch: p99 latency (tail latency is where imbalance and sick backends show first), the healthy fraction of the pool, per-zone balance (traffic and capacity even across zones), and connection errors (the direct symptom of routing to bad backends).
One subtlety in this layout repays attention: where connection state and load information actually live. Each L7 proxy makes routing decisions from its own view of the backend pool, and if proxies do not share a consistent picture — which backends are healthy, how loaded each is — they can collectively make bad choices, all piling onto the same instance because each independently judged it least loaded a moment ago. Real systems damp this with local, slightly-stale load estimates and randomized tie-breaking rather than a single global scoreboard that would itself become a bottleneck and a failure point. The same locality principle governs health: outlier detection is per-proxy and reacts in seconds to what that proxy actually observes, while active health checks give a shared, slower baseline. This deliberate mix of fast-local and slow-global signals is what lets the balancing tier scale horizontally itself — no proxy is authoritative, none is a single point of failure, and the pool's aggregate behaviour emerges from many independent, damped decisions rather than one central brain.
End-to-end flow
Trace a request under normal conditions. A user's client resolves the anycast VIP and its connection lands at the nearest L4 edge, which hashes the flow and forwards it to one of many L7 proxies. That proxy terminates TLS, reads the request's host and path, and matches it to a backend pool. It consults its view of that pool — which backends are currently healthy, and how loaded each is — and applies the balancing algorithm: say least-connections, so it picks the healthy backend in the nearest zone with the fewest in-flight requests. It forwards the request, the backend responds, and the proxy returns the answer, recording the backend's latency to feed future load estimates. Thousands of these decisions per second, spread evenly, keep every backend at similar utilization.
Now a backend goes sick. One instance's disk starts failing and it begins returning 503s and slow responses — but it is still 'up' enough that a shallow health check passes. Outlier detection catches it first: the L7 proxy notices this backend's error rate spiking relative to its peers and passively ejects it from rotation, stopping the bleeding within seconds. Moments later the active health check (which probes a deeper endpoint) also fails and confirms the ejection. Traffic that would have gone to the sick instance is redistributed across the healthy remainder. Because the pool was sized with headroom and spread across zones, the survivors absorb the load without tipping over. When the instance is repaired and its health checks pass again for a sustained window, it is gradually returned to rotation — gradually, so it is not slammed with full load the instant it reappears.
Now a deploy, which exercises draining. The platform rolls out a new version by replacing instances a few at a time. For each instance being replaced, the load balancer marks it draining: it stops sending new requests to that instance but lets the requests already in flight complete. Once the instance is idle (or a drain timeout elapses), it is removed and a new instance takes its place, joins the pool, passes health checks, and is slowly ramped up. Throughout, users see no dropped connections — the whole fleet is replaced under live traffic without a blip, because membership changes are graceful on both ends: drain on the way out, slow-start on the way in.
Consider the stress path: a zone failure. An entire availability zone goes dark — power, network, whatever. A third of the backend pool vanishes at once. Health checks and outlier detection eject the unreachable instances quickly, and the load balancer redistributes all traffic across the two surviving zones. This is the moment the earlier design choices pay off or fail: if the pool was provisioned so that N-1 zones can carry full load, users see a brief latency bump and nothing more; if it was provisioned assuming all zones always available, the survivors are now overloaded and the autoscaler races to add capacity while p99 latency climbs. The load balancer did its job — it stopped routing to the dead zone — but its usefulness depended entirely on there being healthy capacity to route to. Availability is a property of the whole system, and the balancer is only its traffic cop.
Zoom out one level, because a single load balancer is rarely the whole story. Production traffic usually passes through a tier of balancers: a global layer that steers each client to the best region — by latency, by health, by capacity — and a regional layer that spreads requests across the backends within that region. The global tier is where cross-region failover lives: if an entire region degrades, its health signals go red and the global layer withdraws it, shifting users to the next closest healthy region without a client changing a thing, because they were only ever talking to an anycast name. This is also where you decide the hard questions of geography and data residency — a request from Europe may be pinned to European regions for compliance even when a US region is momentarily faster. The regional tier, meanwhile, is where the health checks, algorithms, and draining described above actually run. Keeping the two tiers distinct matters: the global layer optimizes for coarse, slow-moving decisions about which region, while the regional layer makes fast, fine-grained per-request choices about which instance. Collapsing them — trying to make one balancer both globally aware and instance-precise — produces a component that is slow where it should be fast and coarse where it should be sharp.