Kubernetes Pod Termination Without Dropped Requests: EndpointSlices, preStop and the SIGTERM Race

Two clocks start when a Pod is deleted

A Deployment rollout, a node drain or a scale-down all end the same way: the API server marks a Pod for deletion with a grace period, 30 seconds by default. From that moment two independent processes run in parallel, and most dropped requests during rollouts come from assuming they run in sequence.

On the node, the kubelet begins local shutdown. If a container has a preStop hook it runs that first, then the container runtime sends SIGTERM to process 1 in each container. In the control plane, the EndpointSlice controller marks the Pod's endpoint as terminating, and every consumer of EndpointSlices must then notice and update its own state: kube-proxy on every node, ingress controllers, service-mesh proxies and any cloud load balancer that targets Pod IPs. Those updates are asynchronous and take a variable amount of time. If the application exits as soon as SIGTERM arrives, there is a window in which some proxies still send new connections to a Pod that no longer listens. Clients see connection refused, resets or 502s.

What the documentation guarantees, and what it does not

Notice what is missing: there is no promise about how long propagation takes. It depends on cluster size, API server load, how each proxy watches EndpointSlices and how a cloud load balancer deregisters targets. You have to measure it in your environment. Also notice that failing the readiness probe after SIGTERM, a common recommendation, does not remove the endpoint any faster: the terminating endpoint is already reported as not ready from the moment of deletion. The delay you are fighting is propagation, not detection.

The shutdown sequence that works

1. preStop sleep. Delay SIGTERM long enough for the slowest consumer to stop sending new connections. The Pod keeps serving normally during this time.
2. On SIGTERM, stop accepting. Close the listener so no new connections are accepted.
3. Drain in-flight work with a deadline. Let active requests finish, up to a drain budget shorter than what remains of the grace period.
4. Close idle keep-alive connections. Pooled client connections bypass endpoint changes, because kube-proxy rules apply to new connections. The server must close them, for HTTP/1.1 with Connection: close on the next response or by closing idle connections, and for HTTP/2 or gRPC with GOAWAY.
5. Exit before SIGKILL. Flush logs and metrics and return.

Illustrative Kubernetes manifest; the 10 second sleep must come from your own propagation measurements.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
spec:
  replicas: 6
  selector:
    matchLabels: {app: orders}
  strategy:
    rollingUpdate: {maxUnavailable: 0, maxSurge: 2}
  template:
    metadata:
      labels: {app: orders}
    spec:
      terminationGracePeriodSeconds: 45   # preStop 10 + drain 25 + margin 10
      containers:
      - name: app
        image: registry.example.com/orders:2026.10.17
        ports: [{containerPort: 8080}]
        readinessProbe:
          httpGet: {path: /ready, port: 8080}
          periodSeconds: 5
        lifecycle:
          preStop:
            sleep: {seconds: 10}

The application side, in Go

Illustrative Go; routes() stands for your handler.

package main

import (
	"context"
	"log"
	"net/http"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	srv := &http.Server{Addr: ":8080", Handler: routes(), IdleTimeout: 60 * time.Second}
	go func() {
		if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			log.Fatal(err)
		}
	}()
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
	defer stop()
	<-ctx.Done() // SIGTERM arrives after the preStop sleep has finished

	drain, cancel := context.WithTimeout(context.Background(), 25*time.Second)
	defer cancel()
	// Shutdown closes listeners, closes idle connections, then waits for active ones.
	if err := srv.Shutdown(drain); err != nil {
		log.Printf("drain deadline hit, %v; remaining connections will be cut", err)
	}
}

Go's http.Server.Shutdown does steps 2 to 4 for HTTP/1.1 and HTTP/2 served by the standard library. Other runtimes need explicit work: in Java, Spring Boot's graceful shutdown setting; in Node, closing the server and tracking sockets yourself. Make sure the process that receives SIGTERM is your server. A shell entrypoint that does not exec the server swallows the signal, and the Pod sits idle until SIGKILL.

Worked timeline

Suppose measurement shows kube-proxy on all nodes picks up endpoint changes within about 2 seconds in normal conditions and 6 seconds during a large rollout, and the external load balancer stops sending new connections within 8 seconds of deregistration. These are illustrative figures for the example, not general values. Choose a preStop sleep of 10 seconds to cover the slowest path with margin. Requests at this service take up to 20 seconds at p99.9, so give the drain 25 seconds. The grace period must cover all of it, 10 plus 25 plus a margin for logging and exit, so 45 seconds.

Now replay the failure. Without the sleep, SIGTERM arrives at about time 0, the server closes its listener at once, and for the next 2 to 8 seconds some nodes and the load balancer still send new connections to it. With the sleep, those seconds pass while the Pod still accepts traffic; by the time the listener closes, nothing new is routed to it. The pod termination timeline lab lets you vary each of these numbers and see which requests fail.

Failure modes

• Grace period shorter than preStop plus drain. The kubelet sends SIGKILL in the middle of requests. The countdown starts before the hook, so the sum must fit.
• Exec hooks in minimal images. exec: [sh, -c, "sleep 10"] fails in an image without a shell, and a failed hook kills the container. Use the sleep action.
• Long-lived connections. WebSockets, gRPC streams and HTTP/2 connections keep flowing to a Pod after its endpoint is gone. The server must end them deliberately and clients must reconnect.
• Sidecar ordering. If a proxy sidecar exits before the application finishes draining, in-flight responses fail. Kubernetes applies special ordering to sidecar containers; for older injection styles, delay the proxy shutdown explicitly.
• Rollout settings that remove capacity. With maxUnavailable above zero and a busy service, the remaining Pods may be overloaded while new ones warm up; that looks like a shutdown problem but is capacity.
• Fixed sleeps that age badly. Propagation time grows with cluster size. Re-measure after major growth or proxy changes.

How to prove zero dropped requests

1. Run a constant-rate load test against the Service through the same path users take, ingress included, and count every non-success and connection error with timestamps.
2. Trigger a rollout and a node drain during the test.
3. Watch the EndpointSlice for the Service and record when each endpoint turns terminating; compare with the first errors.
4. Record per-Pod SIGTERM and exit times from logs, and kube-proxy sync latency from its metrics, such as kubeproxy_sync_proxy_rules_duration_seconds.
5. Accept the configuration only when repeated rollouts show zero errors; keep the test in CI for the service template.

Trade-offs

A longer preStop sleep makes rollouts and scale-downs slower, and it delays node drains for every Pod on the node. For a batch worker that pulls work from a queue, none of this applies: it has no inbound connections, and its graceful shutdown is about finishing or returning the current job. Apply the sequence to Pods that receive traffic through Services, and tune the numbers per service instead of copying one template everywhere. The ADK-specific case of draining long agent turns is covered in running ADK Java agents on Kubernetes.

Practice and related reading

Hands-on: Pod termination timeline: preStop, SIGTERM and dropped requests, an interactive lab on synthetic data.

Further reading:

• Running ADK Java agents on Kubernetes
• DNS in microservices
• Kubernetes operator pattern

Sources and scope

Checked against these sources on 17 October 2026. Examples and figures are illustrative, not measurements.

• Kubernetes documentation: termination of Pods
• Kubernetes documentation: container lifecycle hooks
• Kubernetes documentation: EndpointSlices and endpoint conditions
• Kubernetes feature gates reference (PodLifecycleSleepAction)
• Go net/http: Server.Shutdown