Stateful Resources Under Helm: Renames, resource-policy and CRD Pitfalls

Helm's mental model is simple: render templates, apply the resulting manifests, store the outcome as a release record. For a stateless web application that model holds up. Trouble starts when the chart contains something that carries data: a PersistentVolumeClaim, a StatefulSet, a password generated on first install, a CRD shipped with the chart. For these resources Helm's reflex of "delete the old, create the new" is the wrong behaviour, and Helm will not make that distinction for you. This article covers the ways data gets lost when stateful resources are managed with Helm, and a working countermeasure for each.

What Helm tracks, and what it does not

A release stores the full rendered manifest in a Secret. On upgrade Helm looks at three things: the previous release manifest, the new manifest, and the live object in the cluster. This three-way merge tries to preserve changes you made by hand, but its decision logic is keyed on resource identity: API group, kind, name and namespace. If the identity changes, Helm cannot know it was an edit. It deletes the object with the old identity and creates one with the new.

Some resources Helm never tracks at all. PVCs born from a StatefulSet's volumeClaimTemplates are not in the release manifest; the StatefulSet controller creates them. Definitions installed from the crds/ directory are applied only on first install. Objects marked as hooks are not considered part of the release. Each of these three exceptions is the source of a separate trap.

A rename is a delete

The most common data-loss scenario is a resource changing its name. Often this is not even a deliberate rename: changing fullnameOverride, a fix to the naming template in a new chart version, or reinstalling the release under a different name all produce the same result.

For a PVC whose name changed, Helm deletes the old PVC. If the StorageClass reclaim policy is Delete, the underlying disk goes with it. For a StatefulSet the situation is more insidious: Helm deletes the StatefulSet and creates the new one, but the old PVCs stay in place because they were never Helm's property. The new StatefulSet derives fresh PVC names from its own name and starts with empty disks. Nothing has been deleted, yet the application boots from scratch; and if you then clean up without noticing the old PVCs, this time the data really is gone.

The countermeasure has two layers. The first is to put this annotation on every resource that carries data:

metadata:
  annotations:
    helm.sh/resource-policy: keep

It takes effect in two situations: when the release is uninstalled, and when the resource is removed from the templates or renamed. Helm does not delete the resource; it merely stops tracking it. The price is orphaned resources accumulating in the cluster, so label the resource with the release name and write a periodic audit that lists orphans.

The second layer is to never change the name. Once a chart's naming template has shipped, treat it as a contract. If a chart upgrade changes naming, the chart author should announce it in a migration note, and before upgrading you should run helm diff upgrade and check the diff for a PVC or StatefulSet being deleted. Upgrading without reading the diff leaves your data to chance.

The same rule applies to external resources the chart creates through an operator. If you enabled an object storage bucket or a database with a "let the chart create it" flag, renaming that resource can cause the operator to delete the old one. Read the operator's deletion policy first, then rename.

Passwords regenerated on every upgrade

Charts frequently use this pattern: if the user supplied no password, generate one with randAlphaNum 32. Because templates are re-rendered on every upgrade, the password changes on every upgrade. The Secret is updated, but the user created with the old password still sits on the database disk. The result is an authentication failure after the upgrade, and most people investigate it as a network problem first.

The correct fix is to read the existing value from the cluster:

{{- $existing := (lookup "v1" "Secret" .Release.Namespace "db-credentials") }}
{{- if $existing }}
password: {{ index $existing.data "password" }}
{{- else }}
password: {{ randAlphaNum 32 | b64enc }}
{{- end }}

This approach has a limit worth knowing: lookup only works when connected to a cluster. In helm template output and in a client-side dry run it returns empty, so every render looks as if a new password will be generated. A server-side dry run closes that gap. The cleaner alternative is to generate the password outside the chart and hand the chart only the name of an existing Secret. Separating the secret lifecycle from the application lifecycle makes every subsequent upgrade easier.

CRDs and hooks live outside the release

Definitions in the crds/ directory are applied on first install and never touched again. If a new chart version added a field to the CRD, Helm does not carry that into the cluster. When the application starts using the new field, the API server silently drops it or rejects it with a validation error. Your upgrade flow needs a separate step that applies the CRDs with kubectl apply --server-side. For the same reason Helm does not delete CRDs on uninstall; that is not a bug but a deliberate choice that prevents every custom resource depending on the CRD from vanishing.

For hooks the trap runs the other way. A Job marked helm.sh/hook: pre-upgrade is not part of the release; even when it succeeds it stays in the cluster. When the next upgrade tries to create it again under the same name, you get an "already exists" error. Write the deletion policy explicitly:

metadata:
  annotations:
    helm.sh/hook: pre-upgrade
    helm.sh/hook-weight: "0"
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded

Leaving a failed hook in the cluster is a deliberate choice; you can inspect its logs. But without before-hook-creation the next attempt fails with the same error.

Immutable fields

Some fields are locked by the API server once created: volumeClaimTemplates and serviceName on a StatefulSet, spec.selector on a Deployment, spec.template on a Job. If a new chart version changes one of these, the upgrade stops with a "field is immutable" error and the release lands in failed state. There is no Helm-level fix. Either delete the resource without tearing down what it controls (kubectl delete statefulset --cascade=orphan leaves Pods and PVCs in place), or stand up the new one under a different name alongside the old and migrate the data.

Disk size is a partial exception: a PVC can only grow, and only if the StorageClass allows expansion. Changing the size in the StatefulSet template still hits the immutable-field error; you have to expand the existing PVCs one by one and then recreate the StatefulSet with --cascade=orphan to bring the template in line.

Half-finished upgrades

A helm upgrade that times out or gets killed in CI leaves the release in pending-upgrade. Every subsequent attempt fails with "another operation is in progress". The fix is to find the last healthy revision with helm history and run helm rollback; if that does not work, delete the Secret of the stuck release revision. To prevent it, set --timeout in CI according to the real startup time and use --atomic. --atomic rolls back automatically on failure, but if the first install fails it removes the release entirely. If the chart creates a PVC without the keep annotation, data written during that first attempt goes with it. On first install it is safer to skip --atomic, look at the error and decide by hand.

There is also a values trap. --reuse-values takes the previous release's values verbatim and ignores new defaults in the new chart version's values.yaml. A security setting added by the new version thus never takes effect. Keep your values in a file in the repository and pass it explicitly with -f on every upgrade.

When not to lean on Helm

If a system such as a database, message queue or search engine has a lifetime independent of the application's deployment rhythm, installing it as a subchart dependency of the application chart is a bad idea. When you rename the application, move the release to another namespace or swap the chart, you do not want the data layer swept along. Install these systems as their own release, preferably through their own operator, and give the application only a Secret carrying connection details. Helm is a good tool for things that are short-lived and reproducible. For long-lived data, at minimum, take away Helm's authority to delete the resources that surround it.