Why architecture matters here
Architecture matters here because health checks are the only place where a monitoring signal is wired directly to a destructive actuator with no human in the loop. Everywhere else in observability, a wrong signal costs you a false page. Here, a wrong signal costs you the process. The asymmetry demands a different standard of care than a dashboard.
The correlation structure is what makes it dangerous. Probes are per-instance but dependencies are shared, so a deep check against a common database means every replica's verdict is driven by the same variable. You have not built N independent health signals; you have built one signal reported N times, and the orchestrator will act on all of them at once.
There is also a subtler cost in the other direction. A check so shallow it only proves the process can return 200 will happily report health while every request fails — a deadlocked thread pool, an exhausted connection pool, a full disk. Health checking sits between two failure modes, and the entire architecture is about picking the right depth for each question rather than one depth for all of them.
The clarifying question, and the one worth writing on the wall, is not 'is this instance healthy?' — that framing is what produces the shared endpoint in the first place, because health sounds like a single property a service either has or lacks. The right question is remedy-shaped: given this condition, does the action this probe triggers actually fix it? A database outage fails that test for liveness — a restart does nothing — and passes it for readiness, where de-routing is exactly right. Once you ask about remedies rather than health, the three probes separate themselves and most cascade designs become obviously wrong on inspection.
The architecture: every piece explained
Liveness — should I be restarted? This probe answers exactly one question: is this process wedged in a state only a restart can fix? Deadlock, a hung event loop, an unrecoverable internal corruption. The correct liveness check is shallow and self-contained: it must never consult a dependency, because 'the database is down' is not a condition a restart repairs. If a restart cannot fix it, it does not belong in liveness.
Readiness — should traffic route to me? This is a routing decision, and it is reversible, which changes the risk calculus entirely. A failing readiness check removes the instance from the load balancer and puts it back when it recovers, with no state lost. Readiness is where dependency awareness legitimately belongs — an instance that cannot reach its database genuinely should not receive requests.
Startup — am I finished booting? The startup probe exists to solve a specific conflict: slow-starting applications need a generous liveness window, but a generous window means a wedged process runs for minutes before anyone notices. The startup probe holds liveness off entirely until the app reports up, letting you set a tight liveness threshold for steady state and a loose one for boot.
Shallow versus deep. A shallow check proves the process is scheduled and its event loop turns. A deep check exercises real dependencies. The rule follows from the remedy: liveness gets shallow because its action is destructive and irreversible; readiness may go deep because its action is reversible and proportionate. The same code behind both endpoints collapses that distinction.
Probe timing as a contract. Every probe carries four numbers — period, timeout, failure threshold, success threshold — and they multiply into the real detection window in a way that is easy to get wrong. A 10-second period with a 3-failure threshold means a wedged process runs for thirty seconds before restart, which may be fine for liveness and far too slow for readiness during a deploy. The one hard rule is that timeout must be strictly less than period: violate it and probes overlap, pile up against the same thread pool, and the checking itself becomes the load that fails the check.
Degraded mode. The binary healthy/unhealthy verdict is the least expressive part of the model, and most real services have a middle state — the recommendation cache is cold, so results are worse but valid. Encoding that as unhealthy withdraws a service that was 80 percent useful. Mature designs separate hard dependencies (no dependency, no service) from soft ones (degrade and serve), and only hard ones affect readiness.
The last line of defence. Every rule above is probe logic, and probe logic can be wrong — which is why mature setups add a layer that assumes it will be. Restart budgets cap how many instances an orchestrator may recycle concurrently; disruption budgets do the same for evictions. Their value is precisely that they do not trust the verdict: when a correlated false signal says all forty replicas are unhealthy, a budget means you lose some capacity rather than all of it, and an engineer gets minutes to intervene instead of seconds. Correct probes plus a blast-radius cap is a system that survives being wrong.
End-to-end flow
Watch the cascade form. A fleet of forty replicas serves an API. Someone wired liveness and readiness to the same /health, which runs SELECT 1 against the primary database. It has worked for a year, survived several code reviews, and looks like diligence: the check verifies a real dependency rather than trivially returning 200.
The database fails over. For roughly eight seconds, connections are refused while the replica promotes — a routine, survivable event the connection pool would have retried through invisibly. Nothing about the application is broken; the pods are healthy, the code is fine, and left entirely alone the system would have absorbed this without a single user noticing.
Every probe fires during the window. All forty replicas run SELECT 1, all forty get a connection error, all forty return 503. The signal is perfectly correlated because it was never about the replicas — forty independent-looking probes measured one shared variable and reported it forty times, and the orchestrator has no way to know that.
Readiness reacts first: all forty leave the load balancer. There are now zero healthy backends and the API returns 503 to users. Note the database has already almost finished failing over — the outage is now entirely self-inflicted. This part is at least recoverable; readiness is reversible, and had the story stopped here the fleet would have returned to rotation seconds later with nothing lost.
Then liveness reacts, and this is where recoverable becomes catastrophic. Three consecutive failures cross the threshold and the orchestrator restarts all forty pods. They come up cold: empty caches, empty connection pools, JIT unwarmed. Forty processes simultaneously open fresh connection pools against a database that just finished promoting, and the thundering herd of pool initialisation overwhelms it. The database, having recovered, falls over again.
The loop closes. Restarted pods probe the now-failing database, fail again, restart again. An eight-second failover has become a rolling twenty-minute outage sustained entirely by the health-check system, and the fix is a manifest change: point liveness at a shallow endpoint that does not know the database exists.
Now re-run the same failover against a correct configuration, because the contrast is the whole argument. Liveness points at /livez, which checks only that the process's request loop turns and touches nothing external — it passes throughout, so not one pod restarts. Readiness points at /readyz, which does check the database, so all forty briefly go unready and the load balancer, finding no healthy backends, queues rather than resets. Eight seconds later the database is up, readiness recovers, the fleet returns to rotation with caches warm and pools intact. Users saw a few seconds of elevated latency. Same fleet, same failover, same probe code — the only difference is which endpoint was wired to the destructive actuator.