Health checks and fail-open: a shared outage across a fleet

What this experiment demonstrates

A fleet of identical instances sits behind a load balancer for 300 seconds of steady traffic, and a fixed share of requests needs a database. You choose what breaks, the shared database or only instance 1's connections, and for how long. Every few seconds each instance is checked: a deep check queries the database, a shallow check only asks whether the process answers. A number of consecutive failures marks an instance unhealthy and a number of passes marks it healthy again. A liveness probe, if deep, restarts the container after the same number of failures, and a restarted instance serves nothing until it is back and has passed its checks.

The load balancer then routes each second's traffic. It can behave like a Kubernetes Service, which sends nothing when no endpoint is ready; like an AWS Application Load Balancer, which routes to all targets when all are unhealthy; or like Envoy, which ignores health and routes to all hosts when the healthy share falls below a panic threshold. Each request either succeeds, fails because its instance could not reach the database, fails because there was no target, fails because its instance was restarting, or fails because an instance received more than its capacity.

Run controlled comparisons

Start with the defaults: six instances of 100 requests per second serving 450, 60 percent needing the database, the database down from 60 to 180 seconds, deep readiness, deep liveness and Service-style routing. Only 59.3 percent of requests are served. The unavoidable loss is 24 percent, the requests that needed the database while it was down, but 36.7 percent fail with no target at all for 110 seconds, and every container is restarted twice. Switch the liveness probe to none and the restarts disappear while the result stays the same, because nothing was routed to them anyway.

Now switch the load balancer to fail open. With deep liveness the fleet serves 68 percent, losing 20 percent to restarts; with liveness off it serves 76 percent, the maximum. Shallow readiness reaches 76 percent behind any balancer. Then change the fault to instance 1 only: deep readiness removes that instance after three checks and serves 99.3 percent, while shallow readiness keeps sending it traffic and serves 96 percent. Raise the load to 550 requests per second with the local fault to see the surviving instances overload.

Inspect the trace and keep evidence

The instance table shows how often each container was restarted and whether it ended healthy. Metrics show the share of requests served, the four failure causes as percentages, container restarts, the seconds with no healthy target and the fewest healthy targets at any moment. Start trace, Next step, Show all and the slider control how much of the trace is visible; it records every change of health, every restart and every change of routing mode, and the metrics always describe the whole run. Reset restores the defaults and Export JSON downloads the inputs, the table and the trace.

Limits and production translation

All instances are identical and their checks run at the same instant, so they fail together; real checks are staggered and fail over a few seconds. A restart cannot cure a local fault here, while a real restart sometimes would. Traffic is spread exactly evenly, capacity is a hard limit with no queueing, and the database recovers instantly. What carries over is the decision structure: keep dependencies out of liveness, report only instance-local problems in readiness, and know what each traffic layer does when it sees no healthy target.