Core engineering · Practical technical guide · 21 October 2026
How a shared dependency outage plus deep health checks removes every instance at once, why liveness probes that check dependencies restart healthy processes, how ALB fail-open and Envoy panic thresholds differ from a Kubernetes Service with no ready endpoints, and what to check where.
The outage that health checks made worse
A fleet of six instances serves an API. Sixty percent of requests read the database; the rest are served from memory and a cache. The database becomes unreachable for two minutes. The outage you would expect is the sixty percent that need it. The outage that often happens is total: every instance fails its health check at the same moment, the load balancer removes all of them, and requests that never needed the database fail too. If the same check drives a liveness probe, the orchestrator also restarts every container, so when the database returns the fleet is busy starting up.
Health checks answer two questions that are easy to merge and dangerous to merge: should this instance receive traffic, and is this process broken beyond recovery. The answers depend on different evidence, and the systems that act on them have documented behaviours when everything looks unhealthy.
What each check should know
| Check | Question | Should include dependencies? | Consequence of failing |
|---|---|---|---|
| Liveness probe | Is this process wedged? | No | Container restart |
| Readiness probe | Can this instance serve its traffic now? | Only instance-local ones | Removed from Service endpoints |
| Load balancer health check | Should the balancer send traffic here? | Only instance-local ones | Target removed, unless the balancer fails open |
| Synthetic and dependency monitoring | Is the system working end to end? | Yes | An alert to a person |
The Kubernetes documentation warns that incorrect liveness probes can cause cascading failures: containers restarted under high load, failed client requests and more load on the remaining pods. A liveness probe that queries the database converts a database outage into a restart storm. Instance-local means the dependency is broken for this instance and not for its peers: its own connection pool is exhausted, its local disk is full. A shared database outage is not instance-local, and no routing decision can fix it.
What the traffic layer does when all targets fail
Three widely used systems behave differently, and the difference decides whether the cache-served requests survive a shared outage.
- AWS Application Load Balancer. The target group documentation states that if a target group contains only unhealthy targets, the load balancer routes requests to all of them regardless of health: it fails open.
- Envoy. When the percentage of available hosts drops below the panic threshold, 50 percent by default, Envoy disregards health and balances across all hosts, or, if configured, across none. Panic mode exists precisely to stop host failures cascading as load concentrates on the survivors.
- A Kubernetes Service. A pod that fails readiness is removed from the EndpointSlices of every matching Service. If every pod fails, the Service has no ready endpoints and requests fail. There is no fail-open.
So the same deep readiness check is survivable behind an ALB and fatal behind a plain Service. Many systems have both layers, with the outermost balancer failing open towards nodes whose Service has no endpoints, which is still a total outage.
Worked example
The interactive lab linked below simulates the opening scenario: six instances of 100 requests per second serving 450, 60 percent of requests needing the database, the database down from 60 to 180 seconds of a 300-second run. Checks run every 10 seconds; three failures mark a target unhealthy, two passes mark it healthy again, and a restart takes 30 seconds.
| Readiness | Liveness | Traffic layer | Served | Failed: no target | Failed: restarting | Restarts |
|---|---|---|---|---|---|---|
| deep | deep | Kubernetes Service | 59.3% | 36.7% | 0% | 12 |
| deep | none | Kubernetes Service | 59.3% | 36.7% | 0% | 0 |
| deep | deep | ALB fail-open | 68.0% | 0% | 20.0% | 12 |
| deep | none | ALB fail-open or Envoy panic | 76.0% | 0% | 0% | 0 |
| shallow | none | any | 76.0% | 0% | 0% | 0 |
The ceiling is 76 percent, because 24 percent of all requests needed the database while it was down. Deep readiness behind a Service loses another 36.7 points: 110 seconds with nowhere to send traffic. Fail-open recovers them, but a deep liveness probe then restarts every container twice during the outage and costs 20 points of its own. Now flip the fault: only instance 1 loses its database connections. Deep readiness removes it after three checks and the fleet serves 99.3 percent; shallow readiness keeps sending it traffic and serves 96.0 percent. That is the case deep checks exist for, and it is why the answer is not simply shallow everywhere.
A design that holds both cases
- Liveness checks only that the process can make progress: the event loop answers, a worker thread is not stuck. Never a dependency.
- Readiness reports instance-local inability to serve, such as an exhausted local pool, a full disk or startup not finished. When a shared dependency fails, degrade instead: serve what you can and return a clear error for the rest.
- Detect "everyone fails at once" in the instance: if the database check fails but the same check from peers or a canary also fails, it is shared, so stay ready.
- At the traffic layer, know your fail-open behaviour. Behind Envoy, keep the panic threshold; behind a Kubernetes Service, there is none, which makes the instance-side rule above essential.
- Alert on the dependency itself through monitoring, not through instances disappearing from the balancer.
Tuning thresholds
Detection time is roughly the interval times the failure threshold; recovery time is the interval times the success threshold, plus a restart if one happened. With a 10-second interval, three failures and two passes, an instance-local fault is removed in up to 30 seconds and restored 20 seconds after it clears. Faster removal means more false removals under transient blips; both directions cost requests. The AWS documentation lists the ALB defaults for instance and IP targets as a 30-second interval, 5-second timeout, five passes to become healthy and two failures to become unhealthy, so a recovered target can take two and a half minutes to return. Measure the cost of a false removal against a slow one with your own traffic.
Verification
- Game day: block the shared database from all instances in staging and confirm the fleet stays in rotation and serves the requests that do not need it.
- Block it from one instance and confirm that instance alone is removed.
- Count container restarts during the drill; any restart caused by a dependency is a liveness bug.
- Read the panic and fail-open settings of every balancer in the path, outermost first.
Failure modes
- A liveness probe that calls the database, cache or another service.
- Deep readiness behind a Service with no fail-open, so a shared outage removes every endpoint.
- One health endpoint used for liveness, readiness and the external balancer.
- Health handlers that share the request thread pool, so overload looks like death and triggers restarts.
- Success thresholds so high that the fleet returns slowly after the dependency recovers.
Sources and scope
Checked against these sources on 21 October 2026. Examples and figures are illustrative, not measurements.
Keep dependencies out of liveness, report only instance-local faults in readiness, and know whether each traffic layer fails open, panics or sends nothing when no target is healthy.