Engineering research / distributed

PodDisruptionBudgets During Node Drains: Budget Arithmetic, Eviction 429s, Drains That Never Finish and unhealthyPodEvictionPolicy

How a PodDisruptionBudget decides whether kubectl drain may evict a pod: desiredHealthy and disruptionsAllowed, percentage rounding, 429 responses while replacements start, budgets that block forever, and what unhealthyPodEvictionPolicy changes for crash-looping pods.

By Sandeep Belgavi · Published 2026-10-25 · Prepared 2026-10-20

Core engineering · Practical technical guide · 20 October 2026

How a PodDisruptionBudget decides whether kubectl drain may evict a pod: desiredHealthy and disruptionsAllowed, percentage rounding, 429 responses while replacements start, budgets that block forever, and what unhealthyPodEvictionPolicy changes for crash-looping pods.

A drain is a sequence of eviction requests

Node maintenance in Kubernetes usually starts with kubectl drain. The command cordons the node so nothing new lands on it, then asks the API server to evict each pod through the Eviction API rather than deleting it. Direct deletion bypasses PDB admission, but authorization, API errors and finalizers still apply; eviction checks the disruption budget. If the pod belongs to a workload guarded by a PodDisruptionBudget, the API server answers 200 and starts graceful termination only if the budget allows one more disruption, and can answer 429 when the budget blocks it. API throttling can also produce 429; overlapping PDB misconfiguration can produce 500. kubectl keeps retrying, and its --timeout defaults to zero, which means it waits forever.

That makes the budget the real scheduler of every maintenance window. A budget that is too loose lets a drain take a quorum system below quorum. A budget that is too tight, or that interacts badly with pods that are not Ready, turns a five-minute node upgrade into a drain that never finishes. Both failures come from the same few lines of arithmetic.

The arithmetic

A budget names a selector and exactly one of minAvailable or maxUnavailable, each either an integer or a percentage. The disruption controller writes four status fields: expectedPods, the replica count of the owning workload; currentHealthy, the selected pods that are Ready; desiredHealthy, the minimum number of healthy pods the budget requires; and disruptionsAllowed, which is currentHealthy minus desiredHealthy, never below zero. An eviction of a healthy pod is admitted only while disruptionsAllowed is at least one, and each admitted eviction reduces it until a replacement becomes Ready.

SpecReplicasdesiredHealthydisruptionsAllowed when all are Ready
minAvailable: 5651
maxUnavailable: 1651
minAvailable: "50%"74 (3.5 rounds up)3
maxUnavailable: "30%"74 (2.1 unavailable rounds up to 3)3
maxUnavailable: 0660, so no voluntary eviction ever succeeds
minAvailable: 3330, the same trap with an integer

The documentation states the rounding rule: a percentage of minAvailable rounds up the number of pods that must stay, and a percentage of maxUnavailable rounds up the number that may be unavailable. Both round in the direction that the field name suggests is generous, which is why 30 percent of 7 permits 3 unavailable pods rather than 2. The task documentation also recommends maxUnavailable over minAvailable for most workloads, because it keeps working when the replica count changes.

Worked example: why the second eviction waits

A Deployment runs 6 replicas, two on each of three nodes, with maxUnavailable: 1. Draining node A requests two evictions. The first is admitted: currentHealthy drops from 6 to 5 and disruptionsAllowed from 1 to 0. The second gets 429. The Deployment controller creates a replacement on node B or C, which needs time to pull, start and pass its readiness probe; until it does, every retry of the second eviction gets 429. With a replacement that takes 40 seconds to become Ready, draining two pods from one node takes at least 40 seconds plus the termination time of the second pod, and draining all three nodes in sequence takes at least five such waits.

Now drain nodes A and B in parallel, which the documentation permits. The budget is shared, so both drains compete for the same single disruption; one of them gets each slot and the other retries. The budget does exactly its job, and the drains are no faster than running them one after another.

Budgets that block forever

Three configurations never finish without intervention. The first is any budget whose desiredHealthy equals the replica count: maxUnavailable 0, minAvailable 100 percent, or minAvailable equal to the replicas. The task documentation describes this case: a voluntary eviction can never succeed, and someone must remove or edit the budget for the node to drain. Sometimes that is deliberate for a singleton, but then it must be paired with a documented manual procedure.

The second is a workload that cannot make its replacement Ready, because of a bad image, a failing readiness probe or a missing dependency. The Eviction API documentation lists this under stuck evictions: the API returns only 429 or 500 until you intervene. A 500 also appears when more than one budget selects the same pod, which is a misconfiguration the API refuses to resolve for you.

The third involves pods that are Running but not Ready, and it is where unhealthyPodEvictionPolicy matters.

unhealthyPodEvictionPolicy

The default policy, IfHealthyBudget, lets a running but unhealthy pod be evicted only when the application is not already disrupted, meaning currentHealthy is at least desiredHealthy. The reasoning is that a pod that is still starting deserves the chance to become healthy. The documented consequence is that drains can be blocked by misbehaving applications, such as pods in CrashLoopBackOff or pods that never report Ready.

Take 3 replicas with minAvailable: 2 after a bad rollout leaves two pods crash-looping on node A. currentHealthy is 1, desiredHealthy is 2, so the application counts as disrupted. Under IfHealthyBudget the two broken pods cannot be evicted and the drain of node A never completes, even though evicting them would not reduce availability at all; they serve nothing. With AlwaysAllow, running pods that are not healthy are treated as already disrupted and can be evicted regardless of the budget, while the healthy pod keeps its protection. The disruptions page and the drain task both recommend AlwaysAllow so that drains can evict misbehaving applications. Pods in Pending, Succeeded or Failed phase are always evictable under either policy.

Illustrative budget for a stateless service; quorum systems may prefer minAvailable set to the quorum size.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app: checkout-api

Choosing a budget

  • Stateless service with several replicas: maxUnavailable 1 or a small percentage, AlwaysAllow, and enough spare capacity that losing one pod does not overload the rest.
  • Quorum system such as etcd or ZooKeeper: maxUnavailable 1 or minAvailable equal to the quorum size, following the task documentation; check what a crash-looping member should do before choosing AlwaysAllow.
  • Singleton that must not be disrupted: a budget that blocks eviction is acceptable only with a written manual procedure for maintenance.
  • Anything selected by two budgets: fix the selectors, because the API answers 500 rather than choosing.

Remember what a budget does not cover. Involuntary disruptions such as a node failure are not prevented, although they count against the budget. Rolling updates of a Deployment or StatefulSet are governed by the workload's own update strategy, not by the budget. And kubectl drain --disable-eviction deletes instead of evicting, which bypasses every budget; it belongs in an incident runbook, not in automation.

Verification before maintenance

Read the budget status before you drain, not after the drain hangs. A budget with disruptionsAllowed 0 while every pod is healthy will block; fix it first.

List every budget with its status, then drain with a bounded timeout.

kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,\
EXPECTED:.status.expectedPods,HEALTHY:.status.currentHealthy,\
DESIRED:.status.desiredHealthy,ALLOWED:.status.disruptionsAllowed
kubectl drain node-a --ignore-daemonsets --timeout=15m

Give drains in automation a timeout and a report: which pods were still waiting and which budgets they were waiting on. The interactive lab linked below lets you change the budget, the number of unready pods, the replacement delay and the policy, and shows each eviction decision with the recomputed disruptionsAllowed.

Failure modes

  • A budget equal to the replica count, which silently turns every drain into a manual task.
  • Default IfHealthyBudget on a workload that crash-loops after a bad rollout, blocking node upgrades until someone fixes the application.
  • Replacement pods that take minutes to become Ready, so a drain of many nodes takes hours; measure readiness time.
  • Too little spare capacity, so the replacement stays Pending and the second eviction waits forever.
  • Two overlapping budgets on the same pods, answered with 500.
  • Using --disable-eviction in scripts, which removes the protection the budget was written to give.

Sources and scope

Checked against these sources on 20 October 2026. Examples and figures are illustrative, not measurements.

Read desiredHealthy and disruptionsAllowed before draining, prefer maxUnavailable, never set a budget equal to the replica count, use AlwaysAllow for crash-looping pods and give drains a timeout.