Kubernetes PodDisruptionBudget Example: minAvailable vs maxUnavailable

You want node drains and cluster upgrades to stop taking your app down, or you've got a kubectl drain that's been hanging for ten minutes and you don't know why. This post walks through writing a PodDisruptionBudget (PDB), choosing between minAvailable and maxUnavailable, and fixing the two classic stuck-drain cases (one replica, and pods that never become Ready). Everything below was run on a local kind cluster, and the output is pasted as I got it. The manifests are all included so you can reproduce it. For more on keeping clusters healthy, see the NextGenBeing blog.

Kubernetes pod disruption budget example: kind cluster and demo manifests

On my machine: kind v0.33.0, kubectl v1.36.1, Docker 29.7.2. kind printed Ensuring node image (kindest/node:v1.37.0) when it created the cluster, and kubectl version reported Server Version: v1.37.0. The kind quick start says to "check the release notes for your given kind version" for the list of images, and to use kind create cluster --image=... to pick another Kubernetes version (kind quick start).

kind.yaml (1 control-plane, 2 workers):

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

$ kind create cluster --name pdb --config kind.yaml

web.yaml, the healthy app (3 replicas, readiness probe on /):

apiVersion: apps/v1
kind: Deployment
metadata: {name: web}
spec:
  replicas: 3
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        readinessProbe: {httpGet: {path: /, port: 80}, periodSeconds: 2}

stuck.yaml, the broken app (probe hits /nope, which returns 404, so it never becomes Ready; pinned to one worker):

apiVersion: apps/v1
kind: Deployment
metadata: {name: stuck}
spec:
  replicas: 1
  selector: {matchLabels: {app: stuck}}
  template:
    metadata: {labels: {app: stuck}}
    spec:
      nodeSelector: {kubernetes.io/hostname: pdb-worker}
      containers:
      - name: nginx
        image: nginx:alpine
        readinessProbe: {httpGet: {path: /nope, port: 80}, periodSeconds: 2}

What a PodDisruptionBudget does

A PDB limits how many pods of an app can be down at once during voluntary disruptions: node drains, cluster upgrades, autoscaler scale-downs. The Kubernetes disruptions docs are explicit about the limits: involuntary disruptions (hardware failure, kernel panic, node out of resources) can't be prevented by a PDB, though they do count against the budget. And the docs add that "A disruption budget does not truly guarantee that the specified number/percentage of pods will always be up" (configure-pdb).

The PDB for web:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: web}
spec:
  minAvailable: 2
  selector: {matchLabels: {app: web}}

$ kubectl get pdb web
NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web    2               N/A               1                     21s

ALLOWED DISRUPTIONS is the column that matters. It's currentHealthy - desiredHealthy, floored at zero. "Healthy" means the pod's Ready condition is True, so a pod that's still failing its readiness probe doesn't count. If that column is 0, every eviction of a covered pod gets refused.

Two syntax rules:

• You can set only one of the two fields. Setting both is rejected by the API server:

  The PodDisruptionBudget "bad" is invalid: spec: Invalid value: {"MinAvailable":1,...,"MaxUnavailable":1,...}: minAvailable and maxUnavailable cannot be both set
  

• selector is required. Per the docs, an empty selector ({}) matches every pod in the namespace, which is rarely what you want.

minAvailable vs maxUnavailable: which one to use

Rule of thumb: autoscaled app, use maxUnavailable: 1; hard floor such as a quorum, use minAvailable. The measurements below are why.

They express the same budget from opposite ends. The difference shows up when the replica count changes or when you use percentages. For each row I scaled web to the replica count, waited for the rollout, applied a PDB with the given setting, and read the row. Run it with the web PDB from earlier deleted (kubectl delete pdb web); the script only creates and deletes a PDB named t. The Deployment must not be selected by any other PDB: the eviction docs say several PDBs on one pod give a 500. The script (table.sh <replicas> <field> <value>):

kubectl scale deploy web --replicas=$1 >/dev/null
kubectl rollout status deploy web >/dev/null
cat <<YAML | kubectl apply -f - >/dev/null
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: t}
spec:
  $2: $3
  selector: {matchLabels: {app: web}}
YAML
sleep 2
echo "replicas=$1 $2=$3"; kubectl get pdb t --no-headers
kubectl delete pdb t >/dev/null

Raw output, with the setting shown first. Each row is name, min, max, allowed disruptions, age:

replicas=3 minAvailable=50%
t     50%   N/A   1     3s
replicas=3 minAvailable=90%
t     90%   N/A   0     3s
replicas=3 maxUnavailable=34%
t     N/A   34%   2     3s
replicas=3 maxUnavailable=10%
t     N/A   10%   1     3s
replicas=3 maxUnavailable=1
t     N/A   1     1     3s
replicas=3 maxUnavailable=0
t     N/A   0     0     3s
replicas=2 minAvailable=2
t     2     N/A   0     2s
replicas=1 minAvailable=1
t     1     N/A   0     2s
replicas=1 maxUnavailable=30%
t     N/A   30%   1     2s
replicas=1 maxUnavailable=1
t     N/A   1     1     3s
replicas=1 maxUnavailable=0
t     N/A   0     0     2s

The allowed disruptions value is the second-from-last column. As a table:

Rows worth staring at. maxUnavailable: "10%" of three pods allows one disruption, and "34%" allows two. The docs describe this: percentages round up the number of pods that may be disrupted, "Thereby a disruption can exceed your defined maxUnavailable percentage." And minAvailable: "90%" of three pods allows none, since 90% of 3 rounds up to 3 required. The same minAvailable: 2 that allowed one disruption at three replicas allows zero at two.

Applying that:

• Use maxUnavailable for anything that scales (Deployments with an HPA). maxUnavailable: 1 allows one disruption at both three replicas and one in the table, whereas minAvailable: 2 drops to zero at two replicas.
• Use minAvailable when there's a hard floor, like a quorum. The docs suggest maxUnavailable: 1 or minAvailable set to quorum size for Consul, ZooKeeper and etcd.
• Avoid percentages on tiny replica counts. With one replica, maxUnavailable: "30%" still allows the single pod to be evicted (the row above shows 1). The docs call out this exact case: it leads "to an effective unavailability of 100%".

There's one more constraint from the docs: maxUnavailable only works for pods that all share the same controller.

How kubectl drain and the Eviction API honor a PDB

kubectl drain doesn't delete pods. It POSTs an Eviction object for each one, and the API server checks PDBs before agreeing. The API-initiated eviction docs list the outcomes: success if allowed, 429 Too Many Requests if a PDB would be violated, and 500 for misconfiguration such as several PDBs selecting the same pod. I didn't trigger the 500 case.

I drained the worker holding two of the three web pods, with minAvailable: 2:

$ kubectl drain pdb-worker2 --ignore-daemonsets --timeout=90s
node/pdb-worker2 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/kindnet-cw4pl, kube-system/kube-proxy-78p2z
evicting pod default/web-8477b64b77-94tbb
evicting pod default/web-8477b64b77-42xxj
error when evicting pods/"web-8477b64b77-42xxj" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
pod/web-8477b64b77-94tbb evicted
evicting pod default/web-8477b64b77-42xxj
pod/web-8477b64b77-42xxj evicted
node/pdb-worker2 drained

The first eviction succeeded and the second was refused. The node was cordoned, so the Deployment's replacement pod landed on the other worker. Once it passed its readiness probe, the budget had room again, and drain's retry five seconds later went through. The drain just took a few seconds longer.

The retry loop is the docs' description too: kubectl drain "periodically retries all failed requests until all Pods on the target node are terminated, or until a configurable timeout is reached." The safely drain a node page also says: "Multiple drain commands running concurrently will still respect the PodDisruptionBudget you specify."

You can see the raw refusal by calling the Eviction endpoint yourself. Start the proxy, set minAvailable: 3 on three Ready pods, and POST an eviction for one of them:

$ kubectl proxy --port=8012 &
$ kubectl get pdb web
NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web    3               N/A               0                     4s

$ curl -s -i -X POST -H 'Content-Type: application/json' \
    -d '{"apiVersion":"policy/v1","kind":"Eviction","metadata":{"name":"web-8477b64b77-k279q","namespace":"default"}}' \
    http://127.0.0.1:8012/api/v1/namespaces/default/pods/web-8477b64b77-k279q/eviction
HTTP/1.1 429 Too Many Requests
Content-Type: application/json

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Cannot evict pod as it would violate the pod's disruption budget.",
  "reason": "TooManyRequests",
  "details": {
    "causes": [
      {
        "reason": "DisruptionBudget",
        "message": "The disruption budget web needs 3 healthy pods and has 3 currently"
      }
    ]
  },
  "code": 429
}

That details.causes message names the PDB and the numbers. After kubectl patch pdb web --type merge -p '{"spec":{"minAvailable":2}}' (allowed disruptions 1), the same request returned (status lines only):

HTTP/1.1 201 Created
  "status": "Success",
  "code": 201

The docs list 200 OK for an allowed eviction; on this v1.37.0 cluster I saw 201, so check for 429 specifically rather than a particular success code.

What a PDB does not stop

I deleted a pod while minAvailable: 3 was in force (allowed disruptions 0):

$ kubectl delete pod web-8477b64b77-79nr5 --wait=false
pod "web-8477b64b77-79nr5" deleted from default namespace

It went straight through. The disruptions docs say so: "deleting deployments or pods bypasses Pod Disruption Budgets." They also say that pods going down in a rolling upgrade "do count against the disruption budget, but workload resources (such as Deployment and StatefulSet) are not limited by PDBs when doing rolling upgrades." So a PDB protects you from well-behaved tooling (drain, the cluster autoscaler, upgrade controllers that use the Eviction API), not from kubectl delete or a bad rollout.

kubectl drain stuck on a PodDisruptionBudget: single replica

Setup, starting from the three-replica web Deployment with its PDB deleted:

$ kubectl delete pdb web
$ kubectl scale deploy web --replicas=1
$ cat <<YAML | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: web}
spec:
  minAvailable: 1
  selector: {matchLabels: {app: web}}
YAML
$ kubectl get pods -l app=web -o wide

Wait until one pod is left; the last column group shows its node (here pdb-worker). One replica plus minAvailable: 1 is a budget of zero, forever. The PDB's status:

$ kubectl get pdb web
NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web    1               N/A               0                     4s

$ kubectl describe pdb web
Name:           web
Namespace:      default
Min available:  1
Selector:       app=web
Status:
    Allowed disruptions:  0
    Current:              1
    Desired:              1
    Total:                1
Events:                   <none>

Current equals Desired, so nothing can go. Draining the node that holds the pod:

$ kubectl drain pdb-worker --ignore-daemonsets --timeout=25s
node/pdb-worker cordoned
evicting pod default/web-8477b64b77-4dhsn
error when evicting pods/"web-8477b64b77-4dhsn" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod default/web-8477b64b77-4dhsn
error when evicting pods/"web-8477b64b77-4dhsn" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
...
error: unable to drain node "pdb-worker" due to error: error when evicting pods/"web-8477b64b77-4dhsn" -n "default": global timeout reached: 25s, continuing command...

(I trimmed the middle of the repeated retry lines.) After the timeout the node stays Ready,SchedulingDisabled: cordoned, not drained, not uncordoned. If you run drain from a script, clean that up with kubectl uncordon.

Evicting the pod would take availability from 1 to 0. Your options:

1. Run at least two replicas and keep minAvailable: 1 (or maxUnavailable: 1). This is the fix if the app can run twice.
2. Accept downtime during maintenance with maxUnavailable: 1 or no PDB for that app. With maxUnavailable: 1, the same drain gave:

   evicting pod default/web-8477b64b77-4dhsn
   pod/web-8477b64b77-4dhsn evicted
   node/pdb-worker drained
   

3. Keep the strict PDB and handle upgrades manually: the docs' advice for a single-instance stateful app is maxUnavailable: 0 and "delete the PDB when the owner agrees to disruption". That makes the block deliberate rather than accidental.

For a stateless single-replica app, I'd go with option 1. A PDB on a one-replica Deployment either does nothing (maxUnavailable: 1) or blocks every drain (minAvailable: 1).

Drain blocked by pods that never become Ready: unhealthyPodEvictionPolicy

The other way to wedge a drain is a covered app whose pods are broken. Setup: apply stuck.yaml, run kubectl rollout restart deploy stuck (this is what gives the Deployment a second pod: the new pod never becomes Ready, so the old one stays), and apply a PDB with minAvailable: 1 selecting app: stuck. Wait until both pods are Running but 0/1 Ready, then look at the budget:

$ kubectl get pdb stuck
NAME    MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
stuck   1               N/A               0                     25s

$ kubectl describe pdb stuck
...
Status:
    Allowed disruptions:  0
    Current:              0
    Desired:              1
    Total:                2

Compare with the healthy web PDB earlier (minAvailable: 2, three Ready pods, ALLOWED DISRUPTIONS 1). Here the app is already down (Current: 0), yet it can't be evicted:

$ kubectl drain pdb-worker --ignore-daemonsets --timeout=12s
node/pdb-worker cordoned
evicting pod default/stuck-6685996db7-h86cn
evicting pod default/stuck-5b86d99d79-stwd2
error when evicting pods/"stuck-6685996db7-h86cn" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"stuck-5b86d99d79-stwd2" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
...
error: unable to drain node "pdb-worker" due to error: [... global timeout reached: 12s ...]

This is the default IfHealthyBudget policy at work. The Unhealthy Pod Eviction Policy section of the configure-pdb page says that under it, "Running pods (.status.phase="Running"), but not yet healthy can be evicted only if the guarded application is not disrupted (.status.currentHealthy is at least equal to .status.desiredHealthy)", and that this "has negative implications for draining nodes", naming pods in CrashLoopBackOff or "just failing to report the Ready condition". Here currentHealthy is 0 and desiredHealthy is 1, so the app is disrupted and the unready pods are protected.

The fix is spec.unhealthyPodEvictionPolicy: AlwaysAllow. The same section says that under it, running-but-unhealthy pods "can be evicted regardless of whether the criteria in a PDB is met", and that the feature "is a stable feature in Kubernetes, and has been since the 1.31 release". I patched it in place:

$ kubectl patch pdb stuck --type merge -p '{"spec":{"unhealthyPodEvictionPolicy":"AlwaysAllow"}}'
poddisruptionbudget.policy/stuck patched
$ kubectl drain pdb-worker --ignore-daemonsets --timeout=40s
node/pdb-worker already cordoned
evicting pod default/stuck-6685996db7-h86cn
evicting pod default/stuck-5b86d99d79-stwd2
pod/stuck-5b86d99d79-stwd2 evicted
pod/stuck-6685996db7-h86cn evicted
node/pdb-worker drained

ALLOWED DISRUPTIONS was still 0 in kubectl get pdb before and after the patch; the policy changed which pods may be evicted, not that number. The replacement pods sat Pending because the nodeSelector pins them to the drained node; that's my test setup.

The same page says Pending, Succeeded and Failed pods "are always considered for eviction", so AlwaysAllow only changes the Running-but-unready case; healthy pods stay protected. The disruptions docs (section "Pod disruption budgets") say: "It is recommended to set AlwaysAllow Unhealthy Pod Eviction Policy to your PodDisruptionBudgets to support eviction of misbehaving applications during a node drain." That fits stateless services. For a database where "Running but not Ready" might mean "replaying its log", think first: the docs note that with AlwaysAllow such pods "might not get a chance to become healthy".

Runbook: my kubectl drain is stuck, what do I check?

1. List every PDB with its budget: kubectl get pdb -A. Any row with ALLOWED DISRUPTIONS of 0 is a suspect.
2. Match the stuck pod to a PDB. The drain output names the pod; the PDB's selector must match its labels (kubectl describe pdb <name> shows the selector). More than one PDB on a pod gives a 500 per the eviction docs.
3. Read the numbers: kubectl describe pdb <name> and compare Current with Desired.
   • Current = Desired (as in the single-replica case): the budget is simply zero. Add replicas, loosen the PDB, or drain on purpose after deleting it.
   • Current < Desired (as in the stuck case): the app is already disrupted. Find out why pods aren't Ready (kubectl get pods, kubectl describe pod).
4. Get the exact refusal by POSTing an Eviction through kubectl proxy, as above. The 429 details.causes message says "needs N healthy pods and has M currently".
5. If the pods are Running but unready and you can't fix them now, patch unhealthyPodEvictionPolicy: AlwaysAllow onto the PDB and retry.
6. Clean up: a timed-out drain leaves the node cordoned. Run kubectl uncordon <node> if you're abandoning the drain.

What to do next

• Run kubectl get pdb -A and look at ALLOWED DISRUPTIONS. Any 0 on a steady-state app is a future stuck drain.
• Replace minAvailable: N on autoscaled Deployments with maxUnavailable: 1.
• Find single-replica Deployments with a minAvailable: 1 PDB and either raise replicas or drop the PDB.
• Add unhealthyPodEvictionPolicy: AlwaysAllow to PDBs for stateless apps (stable since 1.31 per the docs above).
• Always pass --timeout to kubectl drain in scripts, and uncordon on failure. Not tested here: overlapping PDBs (the 500 case) and autoscalers racing a drain.