← blog · November 12, 2026

StatefulSets and Persistent Volumes in Kubernetes: Zone-Pinned Disks, Node Failures, Volume Expansion and Forgotten PVCs

Most StatefulSet trouble comes from the storage layer: pods stuck in Pending, replicas that never return after a node dies, disks you cannot grow from the template and PVCs left behind on scale-down. How WaitForFirstConsumer, the out-of-service taint, PVC retention policy and partitioned rollouts fix them.

In a stateless Deployment, pods are interchangeable: one dies, another starts on a different node, and nobody notices. A stateful workload such as a database, a message broker or a search engine is different, because every pod owns a disk and that disk physically lives somewhere. StatefulSet exists to manage that bond, but most of the trouble comes not from the StatefulSet itself but from how the storage layer underneath it behaves. This post covers why pods get stuck in Pending, why a replacement pod never appears after a node dies, and why you cannot grow a disk by editing the template.

The disk is pinned to a zone, the pod is not

Block volumes from cloud providers usually live in a single availability zone. A dynamically provisioned PersistentVolume therefore carries node affinity on topology.kubernetes.io/zone. As long as a pod is bound to that volume, it can only run on a node in that zone.

The trap is in when the disk gets created. With volumeBindingMode: Immediate on the StorageClass, the disk is provisioned the moment the PVC is created, and the provider picks the zone on its own. The scheduler then tries to place the pod; if the pod's other constraints (node selectors, resource requests, anti-affinity rules) cannot be met in that zone, the pod waits forever with the event "node(s) had volume node affinity conflict".

The fix is to create the disk only after the pod's location is known:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ssd-zonal
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Retain
parameters:
  type: gp3

provisioner and parameters depend on your provider; this one is AWS. Three fields matter. WaitForFirstConsumer waits for the scheduler to pick a node for the pod and provisions the disk in that zone. allowVolumeExpansion lets you grow the volume later. reclaimPolicy: Retain keeps the disk when its PVC is deleted; the default for dynamic provisioning is Delete, which is a dangerous default for stateful data.

WaitForFirstConsumer solves the first placement, not what comes after. Once a disk is born in a zone, its pod is tied to that zone for good. If the nodes there fill up or the zone goes down, the pod cannot move elsewhere. That is why high availability for stateful workloads has to come from the application, not the disk: each member of a three-replica database cluster should sit in a different zone, and the application should copy data with its own replication. To get there, spread the pods across zones:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: database

If you run a cluster autoscaler, define one node group per zone. When a single node group spans several zones, the autoscaler cannot guarantee that the node it adds will land in the zone where the pending pod's disk lives.

Why the pod does not come back when a node dies

When a node becomes unreachable, the Deployment controller eventually starts a new pod elsewhere. StatefulSet deliberately does not. Each pod has a stable identity (database-0, database-1), and StatefulSet guarantees that two pods with the same identity never exist at the same time. When a node is unreachable, the API server cannot tell whether the pod there is really dead; maybe only the network dropped and the process is still writing to its disk. So the pod stays in Terminating and no replacement is created.

The first thing people reach for is kubectl delete pod --force --grace-period=0. That only removes the record from the API; it does not kill the process on the node. If the node is actually alive, two pods start writing the same data and you get corruption. Force delete only when you are sure the node is really off.

The better path is to tell Kubernetes explicitly that the node is down. After you confirm the machine is powered off (or stopped from the provider console), add this taint:

kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute

The taint triggers forced deletion of the pods and detaches their volumes from that node, and the StatefulSet then starts the new pod on another node. Remember to remove the taint when the node comes back. You can automate this step, but the automation must get its "node is off" decision from a trustworthy source such as the provider API or out-of-band power management. If it only relies on "not answering on the network", you have automated exactly the risk of force deletion.

Watch the access mode as well. ReadWriteOnce does not mean "one pod", it means "one node"; two pods on the same node can both mount the volume read-write. If you truly need a single writer, use ReadWriteOncePod, provided your CSI driver supports it.

You cannot grow the disk from the template

When a disk starts filling up, the first instinct is to raise the size in the StatefulSet's volumeClaimTemplates. That field is immutable, so the API rejects the change. The template is only used when new PVCs are created; it is never applied back to existing ones.

What works is this:

kubectl patch pvc data-database-0 -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'
kubectl patch pvc data-database-1 -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'
kubectl patch pvc data-database-2 -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

kubectl delete statefulset database --cascade=orphan
kubectl apply -f database-statefulset.yaml

Grow each PVC individually first. Then remove the StatefulSet without deleting its pods (--cascade=orphan) and recreate it from a manifest that has the new size; the new object recognises the existing pods and PVCs by their labels and adopts them. That way any replica you add later is born with the right size. If a PVC shows the FileSystemResizePending condition afterwards, the driver cannot grow the filesystem online and the pod has to be restarted.

Shrinking is not supported at all. If you need a smaller disk, the only way is a new PVC and a data migration at the application level. Starting small and growing is therefore always cheaper than starting big and trying to shrink.

Scaling down leaves the disks behind

Scale a StatefulSet from three replicas to two and database-2 is deleted, but its PVC stays. This is the default, and it is intentional: scale back to three and the same pod returns with the same data. The side effect nobody notices is the bill; disks from replicas created and removed during experiments keep getting paid for months.

You can change this with persistentVolumeClaimRetentionPolicy:

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Delete

whenScaled: Delete removes surplus PVCs on scale-down, and whenDeleted decides what happens when the whole StatefulSet is deleted. On a production database, never set whenDeleted to anything but Retain; one delete command in the wrong namespace would take all the data with it. whenScaled: Delete only makes sense when replicas can rebuild their data from each other. reclaimPolicy: Retain on the StorageClass is a second safety net here: even if the PVC goes away, the disk stays at the provider.

When a rollout gets stuck

By default a StatefulSet starts pods in order (podManagementPolicy: OrderedReady) and rolls out updates one pod at a time, starting from the highest ordinal. If a bug in the new template keeps a pod from becoming Ready, the rollout stops there. Even after you fix the template, the controller may keep waiting for the broken pod to become Ready; in that case you have to delete the stuck pod by hand after applying the fix. The documentation describes this as known behaviour.

To try a risky change on a single replica first, use partition:

updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    partition: 2

With this setting only pods with an ordinal of 2 or higher are updated. If you like the result, lower the value step by step to zero.

A frequently missed source of slowness on large volumes is fsGroup. When fsGroup is set in the pod security context, the kubelet may recursively change ownership of every file on the volume at each mount; on a disk with millions of files, that means the pod takes minutes to start. fsGroupChangePolicy: OnRootMismatch only does the change when the root directory's permissions do not match, which usually makes the problem go away.

When not to choose a StatefulSet

You do not have to put everything stateful into the cluster. If you can use a managed database service, handing backups, upgrades and failover to the provider is cheaper for most teams. If you do decide to run it in the cluster, prefer a mature operator for that database over managing a raw StatefulSet; operators already know most of the steps above, from ordered upgrades to rebuilding replicas and restoring from backup.

Using a StatefulSet directly makes sense when the application handles its own replication and you design with the storage layer's behaviour in mind. For a single-replica workload with no availability expectations, StatefulSet's ordering guarantees buy you nothing; a single-replica Deployment with the Recreate strategy does the same job, as long as you make sure two pods never try to mount the same disk at once.