Stateful Resources Under Helm: Renames, resource-policy and CRD Pitfalls
Helm deletes a renamed PVC, restarts a StatefulSet on empty disks, regenerates passwords on every upgrade and never updates a CRD. Why data gets lost when stateful resources are managed with Helm, and a working countermeasure for each.
Helm's mental model is simple: render templates, apply the resulting manifests, store the outcome as a release record. For a stateless web application that model holds up. Trouble starts when the chart contains something that carries data: a PersistentVolumeClaim, a StatefulSet, a password generated on first install, a CRD shipped with the chart. For these resources Helm's reflex of "delete the old, create the new" is the wrong behaviour, and Helm will not make that distinction for you. This article covers the ways data gets lost when stateful resources are managed with Helm, and a working countermeasure for each.
What Helm tracks, and what it does not
A release stores the full rendered manifest in a Secret. On upgrade Helm looks at three things: the previous release manifest, the new manifest, and the live object in the cluster. This three-way merge tries to preserve changes you made by hand, but its decision logic is keyed on resource identity: API group, kind, name and namespace. If the identity changes, Helm cannot know it was an edit. It deletes the object with the old identity and creates one with the new.
Some resources Helm never tracks at all. PVCs born from a StatefulSet's volumeClaimTemplates are not in the release manifest; the StatefulSet controller creates them. Definitions installed from the crds/ directory are applied only on first install. Objects marked as hooks are not considered part of the release. Each of these three exceptions is the source of a separate trap.
A rename is a delete
The most common data-loss scenario is a resource changing its name. Often this is not even a deliberate rename: changing fullnameOverride, a fix to the naming template in a new chart version, or reinstalling the release under a different name all produce the same result.
For a PVC whose name changed, Helm deletes the old PVC. If the StorageClass reclaim policy is Delete, the underlying disk goes with it. For a StatefulSet the situation is more insidious: Helm deletes the StatefulSet and creates the new one, but the old PVCs stay in place because they were never Helm's property. The new StatefulSet derives fresh PVC names from its own name and starts with empty disks. Nothing has been deleted, yet the application boots from scratch; and if you then clean up without noticing the old PVCs, this time the data really is gone.
The countermeasure has two layers. The first is to put this annotation on every resource that carries data:
It takes effect in two situations: when the release is uninstalled, and when the resource is removed from the templates or renamed. Helm does not delete the resource; it merely stops tracking it. The price is orphaned resources accumulating in the cluster, so label the resource with the release name and write a periodic audit that lists orphans.
The second layer is to never change the name. Once a chart's naming template has shipped, treat it as a contract. If a chart upgrade changes naming, the chart author should announce it in a migration note, and before upgrading you should run helm diff upgrade and check the diff for a PVC or StatefulSet being deleted. Upgrading without reading the diff leaves your data to chance.
The same rule applies to external resources the chart creates through an operator. If you enabled an object storage bucket or a database with a "let the chart create it" flag, renaming that resource can cause the operator to delete the old one. Read the operator's deletion policy first, then rename.
Passwords regenerated on every upgrade
Charts frequently use this pattern: if the user supplied no password, generate one with randAlphaNum 32. Because templates are re-rendered on every upgrade, the password changes on every upgrade. The Secret is updated, but the user created with the old password still sits on the database disk. The result is an authentication failure after the upgrade, and most people investigate it as a network problem first.
The correct fix is to read the existing value from the cluster:
This approach has a limit worth knowing: lookup only works when connected to a cluster. In helm template output and in a client-side dry run it returns empty, so every render looks as if a new password will be generated. A server-side dry run closes that gap. The cleaner alternative is to generate the password outside the chart and hand the chart only the name of an existing Secret. Separating the secret lifecycle from the application lifecycle makes every subsequent upgrade easier.
CRDs and hooks live outside the release
Definitions in the crds/ directory are applied on first install and never touched again. If a new chart version added a field to the CRD, Helm does not carry that into the cluster. When the application starts using the new field, the API server silently drops it or rejects it with a validation error. Your upgrade flow needs a separate step that applies the CRDs with kubectl apply --server-side. For the same reason Helm does not delete CRDs on uninstall; that is not a bug but a deliberate choice that prevents every custom resource depending on the CRD from vanishing.
For hooks the trap runs the other way. A Job marked helm.sh/hook: pre-upgrade is not part of the release; even when it succeeds it stays in the cluster. When the next upgrade tries to create it again under the same name, you get an "already exists" error. Write the deletion policy explicitly:
Leaving a failed hook in the cluster is a deliberate choice; you can inspect its logs. But without before-hook-creation the next attempt fails with the same error.
Immutable fields
Some fields are locked by the API server once created: volumeClaimTemplates and serviceName on a StatefulSet, spec.selector on a Deployment, spec.template on a Job. If a new chart version changes one of these, the upgrade stops with a "field is immutable" error and the release lands in failed state. There is no Helm-level fix. Either delete the resource without tearing down what it controls (kubectl delete statefulset --cascade=orphan leaves Pods and PVCs in place), or stand up the new one under a different name alongside the old and migrate the data.
Disk size is a partial exception: a PVC can only grow, and only if the StorageClass allows expansion. Changing the size in the StatefulSet template still hits the immutable-field error; you have to expand the existing PVCs one by one and then recreate the StatefulSet with --cascade=orphan to bring the template in line.
Half-finished upgrades
A helm upgrade that times out or gets killed in CI leaves the release in pending-upgrade. Every subsequent attempt fails with "another operation is in progress". The fix is to find the last healthy revision with helm history and run helm rollback; if that does not work, delete the Secret of the stuck release revision. To prevent it, set --timeout in CI according to the real startup time and use --atomic. --atomic rolls back automatically on failure, but if the first install fails it removes the release entirely. If the chart creates a PVC without the keep annotation, data written during that first attempt goes with it. On first install it is safer to skip --atomic, look at the error and decide by hand.
There is also a values trap. --reuse-values takes the previous release's values verbatim and ignores new defaults in the new chart version's values.yaml. A security setting added by the new version thus never takes effect. Keep your values in a file in the repository and pass it explicitly with -f on every upgrade.
When not to lean on Helm
If a system such as a database, message queue or search engine has a lifetime independent of the application's deployment rhythm, installing it as a subchart dependency of the application chart is a bad idea. When you rename the application, move the release to another namespace or swap the chart, you do not want the data layer swept along. Install these systems as their own release, preferably through their own operator, and give the application only a Secret carrying connection details. Helm is a good tool for things that are short-lived and reproducible. For long-lived data, at minimum, take away Helm's authority to delete the resources that surround it.
[OK] 30 records · 9 featured · 1 retainer · 20 standard
/02 services
services.tree
> note
· Prices are in USD. Volume discounts of 7% for 6-12 day projects and 14% for 13+ day projects apply. Programs of 100+ person-days are priced individually. Prices are negotiable based on scope, urgency and long-term collaboration.
> ls ~/products/ --open-source
[OK] 2 entries · 1 live · 1 pre-release
/03 products
products.list
liveopen source · mit/prod/01
filex
the self-hosted file manager that embeds anywhere
single go binary · vue/react/web-component embed · 5 storage drivers · realtime collab · rbac · native multi-tenancy · mcp server for ai agents
We build infrastructure, automate everything, and keep systems running.
BRF Tech is a Bursa-based DevSecOps and software consulting firm. We operate across 30 service areas, from Kubernetes cluster management to serverless platform setup, CI/CD pipeline design to AI agent development.
What we do is simple: we set up your systems, automate them, and make sure they won't wake you up at 3 AM. We codify your infrastructure with Terraform, move your deployments to GitOps with ArgoCD, and monitor everything with Prometheus. If something breaks — we intervene before it does.
We're against vendor lock-in. We work with open-source tools, self-hosted solutions, and industry-standard technologies. We build your own serverless platform on Knative, isolate with Kata Containers, manage your secrets with Vault. Every project is delivered with clear scope, clear timeline, clear pricing.
DevSecOps & CI/CD
Kubernetes & Serverless
Infrastructure as Code
AI & Agent (MCP/ACP)
Security & Zero Trust
Full-Stack Development
Self-Hosted Solutions
> grep -i question ~/faq.md
[OK] 3 entries · click to expand
[?]What services are included in BRF Tech's DevOps solutions?▾
We offer CI/CD pipeline setup, Kubernetes cluster management, container migration, infrastructure automation with Terraform, monitoring & alerting, in-house technology installations and DevOps consulting services.
[?]What are your software development services?▾
We offer MVP development, existing software performance optimization, backend API development (Node.js, Go, Python) and full-stack application development. Every project is delivered with minimal technical debt.
[?]Which DevOps tools do you work with?▾
From Kubernetes and Docker to Talos Linux, from Terraform and OpenTofu to Ansible, and across Jenkins, GitLab CI/CD, GitHub Actions, ArgoCD, Flux, Helm and Rancher, we work with industry-standard tooling. For observability and error tracking we use Prometheus, Grafana, Loki, Tempo, OpenTelemetry and GlitchTip; for networking, Cilium and Envoy; for test automation, Playwright, Cypress, k6, Locust, Testcontainers and Trivy. We determine the most suitable toolset together, based on your project’s needs.
Automated build, test and deployment pipelines with GitHub Actions, GitLab CI/CD.
Automated build, test and deploy pipeline setup for existing or new repositories. GitHub Actions, GitLab CI/CD or preferred tool is used. Separate workflows are defined for staging + production environments.
Container architecture migration, cluster setup and orchestration.
Production-ready Kubernetes cluster setup on bare-metal or cloud. Includes Helm charts, Ingress, TLS certificate management, namespace isolation and RBAC.
Cozystack is an open-source cloud platform built on Kubernetes and the VMware alternative we recommend most. Virtual machines run on KubeVirt, while managed PostgreSQL, MySQL, Redis and Kafka services and dedicated tenant Kubernetes clusters are provisioned from the same dashboard and API. Includes installation on your own bare-metal servers on Talos Linux, replicated storage with LINSTOR/DRBD, network isolation with Kube-OVN, tenant and quota structure, monitoring, backup and a disaster recovery plan. After deployment we also handle upgrades, operations and support; since the platform is open source, there are no license fees and no vendor lock-in.
Packages
Virtual machine support on an existing Kubernetes cluster with KubeVirt
$2,800
4–6 person-day
Pilot setup (3 nodes, virtual machines + managed database)
$3,500
5–8 person-day
Production setup (HA + replicated storage + monitoring + backup)
$6,500
10–16 person-day
Multi-tenant private cloud (tenants, quotas, managed service catalog)
Planned, low-downtime migration of vSphere/ESXi environments to Cozystack, OpenStack or VHI.
A planned move to a VMware alternative for environments whose costs rose after Broadcom moved VMware licensing to subscription bundles, or whose renewal is coming up. We start with an inventory of your vSphere/ESXi estate (virtual machines, disks, networks, dependencies), then compare today's license cost with the total cost of the target platform. In most environments the target is Cozystack; OpenStack fits large-scale IaaS needs and Virtuozzo Hybrid Infrastructure fits storage-heavy service providers. A pilot runs on non-critical virtual machines, disks are converted with tools such as virt-v2v and Forklift, and the migration proceeds in waves within maintenance windows, with a rollback plan for every step.
Packages
Inventory + cost comparison + migration plan
$2,400
3–5 person-day
Pilot migration (up to 10 virtual machines)
$3,500
5–8 person-day
Phased migration (up to 100 virtual machines)
$9,750
15–22 person-day
Full data center migration (network + storage + backup)
OpenStack deployment, Neutron networking, Ceph/Cinder storage, upgrades and operations.
OpenStack deployment and operations in your own data center. A containerized, highly available control plane with Kolla-Ansible; Nova (compute), Keystone (identity), Glance (images) and Horizon (dashboard). On the network side, Neutron with OVN, VLAN/VXLAN, provider and tenant networks, floating IPs and the Octavia load balancer; on the storage side, Cinder block storage on Ceph RBD and optional S3-compatible object storage. For existing deployments: health checks, planned upgrades between releases, monitoring and alerting setup, and a backup strategy.
Packages
Health check + improvement report (existing deployment)
$2,400
3–5 person-day
Lab / PoC setup (single region, 3 nodes)
$3,500
5–8 person-day
Release upgrade (existing deployment)
$3,900
6–10 person-day
Production setup (HA control plane + Ceph + Neutron/OVN)
Hyperconverged infrastructure: multi-tenant cloud plus block and S3-compatible storage for service providers.
Hyperconverged infrastructure with Virtuozzo Hybrid Infrastructure, where compute, storage and networking run on the same servers. Block, S3-compatible object and NFS storage on software-defined storage, plus backup storage (Backup Gateway) for backup products such as Acronis. For service providers: a multi-tenant cloud with domains and projects, quotas, a self-service panel and an OpenStack-compatible API. Includes hardware sizing, network design, cluster deployment and expansion, version upgrades and operations support.
Packages
Pre-deployment assessment + hardware sizing
$2,400
3–4 person-day
Version upgrade / cluster expansion
$2,800
4–6 person-day
Hyperconverged cluster setup (storage + compute)
$3,900
6–10 person-day
Service provider setup (multi-tenant cloud + S3 + backup storage)
Migration of existing applications to container architecture.
Migration of existing applications to container architecture. Docker image design, Compose configuration and conversion to Kubernetes manifests. Kata Containers / gVisor can be included for advanced isolation.
Health check of your current environment, capacity planning, platform selection and a roadmap.
An independent assessment of your current virtualization environment (VMware vSphere, OpenStack, VHI or other KVM-based platforms). The health check covers cluster configuration, resource overcommit, storage latency, forgotten snapshots, backup and patch status. Capacity planning measures CPU, memory and storage growth to work out the hardware you will need next. Platform selection decides which workload should run where: Cozystack for teams running containers and virtual machines side by side, OpenStack for large-scale IaaS, VHI for storage-heavy service providers; staying on the current platform is also an option for some workloads. The deliverable is a written report with a cost comparison and a prioritized roadmap.
In-house tools like GitLab, Nextcloud, VPN, mail server.
Installation of tools like GitLab, Mattermost, Nextcloud, mail server, VPN, internal monitoring on company servers. Deployed on Docker Compose or Kubernetes.
DB + cache + code bottleneck identification and resolution.
Profiling of existing application, identification and resolution of bottlenecks. DB query optimization, caching layer (Redis), service-level improvements.
MCP/ACP agents, RAG pipelines, multi-agent orchestration, model serving.
LLM integration (OpenAI, Claude, Gemini, local models), tool-augmented AI agents with MCP (Model Context Protocol) and ACP (Agent Communication Protocol). RAG pipeline, multi-agent orchestration, model serving (vLLM/Ollama). AI layer for existing business processes.
Event-driven data pipeline with Apache Kafka, RabbitMQ, NATS. ETL/ELT orchestration with Apache Airflow, real-time data streaming, CDC (Change Data Capture). Data lake/warehouse design, schema registry, dead letter queue management.
Packages
Kafka/RabbitMQ setup + basic producer/consumer
$3,500
5–7 person-day
ETL pipeline (Airflow + source → warehouse)
$5,200
8–12 person-day
Full event-driven architecture (CDC + streaming + DLQ)
Hands-on technical training for your teams. Docker & container fundamentals, Kubernetes operations, Terraform IaC, CI/CD best practices, AI/LLM integration workshops. Practical exercises in live lab environments, content customized by skill level.
E2E test automation with Playwright, Cypress, load testing with k6/Gatling. Pre-deploy quality gate integrated into CI/CD pipeline, test coverage reporting, visual regression testing. Reduce test writing time by up to 60% with AI-assisted test generation.
Your Collected Personal Data, Collection Method and Legal Basis
Any information that identifies or makes you identifiable is considered "personal data." When you visit our website or use our services, contact information such as your name, surname, email address, phone number, as well as your IP address and browser cookie data may be collected through automatic or semi-automatic means. This data is processed based on the legal grounds specified in Article 5 of the Personal Data Protection Law No. 6698: "being directly related to the establishment or performance of a contract" and "being mandatory for the legitimate interests of the data controller, provided that it does not harm the fundamental rights and freedoms of the data subject."
Purpose of Processing Your Personal Data
Your collected personal data is processed for the purposes of providing and improving our services, fulfilling customer requests, meeting legal obligations, conducting information security processes and managing communication activities. Your data is processed in a limited and proportionate manner for the stated purposes; when the purpose ceases to exist, data is deleted, destroyed or anonymized.
To Whom and For What Purposes Collected Personal Data May Be Transferred
Your personal data may be transferred to public institutions and organizations as required by legal regulations, to our business partners and technical infrastructure providers for the purpose of delivering services, and to lawyers and consultants in legal disputes. Transfers are carried out in accordance with Articles 8 and 9 of the Law, with necessary technical and administrative measures in place.
Your Rights as a Data Subject
In accordance with Article 11 of Law No. 6698, you have the right to: learn whether your personal data is being processed; request information if it has been processed; learn the purpose of processing and whether it is used in accordance with its purpose; know the third parties to whom it is transferred domestically or abroad; request correction if it has been processed incompletely or incorrectly; request deletion or destruction within the framework of conditions set out in Article 7 of the Law; request that the operations carried out be notified to third parties to whom data has been transferred; object to any adverse result arising from the analysis of data exclusively through automated systems; and claim compensation for damages in case of unlawful processing. You may contact us through the communication channels on our website to exercise these rights.
Terms & Conditions
Terms & Conditions
1. Service Scope and Changes
BRF Tech conducts its software development, DevOps consulting, infrastructure setup and technical support services within the framework of these terms and conditions. The scope of service is determined separately for each project and finalized through mutual agreement. BRF Tech reserves the right to update the scope, pricing and technical details of its services, provided that prior notice is given.
2. Privacy Policy
Personal data collected within the scope of our services is processed in accordance with our Privacy Policy. For detailed information, please review our Privacy Policy page.
3. Service Usage and Responsibilities
Clients agree to use the provided services solely within legal and ethical boundaries. BRF Tech reserves the right to suspend or terminate services in the event of misuse, violation of third-party rights, or use for illegal activities. The client is responsible for the accuracy and currency of the information provided within the project scope.
4. Payment Terms
Payment terms are determined on a project basis and finalized through mutual agreement at the start of the project. Unless otherwise specified, invoices are payable within 15 days of project delivery. A monthly late payment interest of 2% may apply to overdue payments.
5. Cancellation and Refund Policy
Cancellation requests must be submitted in writing at least 7 days before the service start date. For projects already in progress, billing is based on the proportion of completed work. No refunds are issued for software and infrastructure components specifically developed within the project scope.