PodDisruptionBudget drain: eviction budget, 429s and stuck drains

What this experiment demonstrates

A Deployment's replicas are spread round-robin across three nodes, A, B and C. You drain node A, or nodes A and B in parallel, which means one eviction request per pod on those nodes. Each request is decided the way the Eviction API decides it: the page computes desiredHealthy from minAvailable or maxUnavailable, with percentages rounded up as the documentation describes, counts the healthy pods, derives disruptionsAllowed, and answers 200 or 429. A rejected request is retried at a fixed interval, and every admitted eviction starts a replacement on a node that is not being drained, which becomes Ready after a delay you choose, never becomes Ready, or never schedules.

Some pods can be Running but not Ready, as crash-looping pods are. For those the unhealthyPodEvictionPolicy decides: IfHealthyBudget allows the eviction only while the application is not already disrupted, AlwaysAllow allows it regardless. Time advances in whole seconds and termination itself takes no time, so the drain time is the time until the last eviction is admitted.

Run controlled comparisons

Start with the defaults: six replicas, maxUnavailable 1, no unready pods, one node drained and replacements Ready after 40 seconds. The first eviction is admitted at once, the second receives 429 every five seconds until the replacement is Ready, and the drain finishes at 40 seconds. Drain two nodes in parallel and the same single disruption is shared, so the drain takes 120 seconds. Set maxUnavailable to 0 and nothing is ever admitted: the drain never finishes. Switch to minAvailable with a percentage and seven replicas to check the rounding: 50 percent keeps four pods, and maxUnavailable 30 percent also keeps four.

Now make two pods on node A unready and use minAvailable 5. Only four pods are healthy, below the five required, so under IfHealthyBudget the two broken pods cannot be evicted and the drain is stuck, although they serve nothing. Switch the policy to AlwaysAllow and both are evicted immediately. Finally set the replacements to never become Ready, or to stay Pending for lack of capacity, and watch the second eviction wait for ever; give the drain a timeout to see it give up instead.

Inspect the trace and keep evidence

The eviction table lists each pod on a drained node with its readiness at the start, the time of its first eviction request, the number of 429 responses, the time it was admitted and the final outcome. Metrics show the drain outcome and time, desiredHealthy, disruptionsAllowed at the start, admitted evictions, 429 responses, the lowest healthy count during the drain and pods left on drained nodes. Start trace, Next step, Show all and the slider control how much of the trace is visible; repeated 429 responses for one pod are folded into a single step. Reset restores the defaults and Export JSON downloads the inputs, the table and the trace.

Limits and production translation

The model has one Deployment, one budget and three identical nodes, termination is instantaneous and retries happen at exact intervals. Real drains also wait for graceful termination, may hit API rate limits that also return 429, involve DaemonSets and pods with local storage, and run against budgets for many workloads at once. Overlapping budgets, which make the API answer 500, are not modelled. What carries over is the arithmetic and the policy: read desiredHealthy and disruptionsAllowed before maintenance, avoid budgets equal to the replica count, set AlwaysAllow where crash-looping pods should not block a drain, and give automated drains a timeout and a report.