Pod termination timeline: preStop, SIGTERM and dropped requests

What this experiment demonstrates

When a Pod is deleted, the kubelet starts shutting it down while, in parallel, every proxy that routes to it learns that its endpoint is terminating. This page models one Pod in that situation. Three kube-proxy nodes stop sending new connections at staggered times up to the slowest update you choose, an optional external load balancer stops at its own deregistration delay, and a pooled keep-alive client keeps sending on a connection it opened earlier. Each source opens one connection per second while it still routes here, and request durations follow a fixed pattern that includes your longest request.

The kubelet behaviour follows the Kubernetes documentation: the grace period counts from deletion and includes the preStop hook, an overrunning hook gets a one-off two second extension, and SIGKILL ends whatever is left. Everything else is a deliberate simplification.

Run controlled comparisons

The defaults are the example from the article: routes disappear by 8 seconds, a 10 second preStop sleep, a 25 second drain and a 45 second grace period. Every request succeeds. Set preStop to 0 and the listener closes while proxies are still sending new connections, so they are refused. Raise preStop back to 5 and only the slower sources fail; at 8 the refusals stop because the slowest route is gone before SIGTERM.

Next shorten the grace period to 20 seconds. The drain is cut by SIGKILL and the longest request fails mid-flight. Choose exit immediately on SIGTERM to see in-flight work cut even with a generous grace period, and ignore SIGTERM to see the process run until SIGKILL. Turn off closing idle keep-alive connections: the pooled client keeps using its old connection and one request is reset when the process stops. Finally raise the load balancer delay to 30 seconds and find the preStop and grace period that bring failures back to zero.

Inspect the trace and keep evidence

The timeline shows the preStop hook, the period when the listener accepts connections, the drain, and how long each source keeps routing here, with SIGKILL marked as a red line. Rows that extend past the listener closing are the cause of refusals. The trace lists the ordering of events and the first failures with their cause. Start trace, Next step, Show all and the slider control how much of the trace is visible; metrics always describe the whole run. Reset restores the defaults and Export JSON saves the inputs, every request with its outcome, and the trace.

Limits and production translation

Propagation in a real cluster is a distribution, not three fixed times, and it grows with cluster size and API server load. Clients retry, readiness probes run, sidecars have their own ordering, and many servers handle keep-alive differently from this model. Use the page to reason about ordering and budgets, then measure your own propagation with a load test during a rollout and size preStop, drain and grace period from those measurements.