Zero-Downtime Schema Migrations: Expand and Contract, Locks and PostgreSQL Pitfalls
Rolling deploys are not enough; a schema change breaks the moment old and new versions run side by side. How to migrate PostgreSQL without downtime using expand and contract, lock timeouts, concurrent indexes and batched backfills, and where it goes wrong.
We learned how to deploy an application without downtime: multiple replicas, health checks, rolling updates. Then the first schema change arrives and a single ALTER TABLE throws all of that away. During a rollout the old and the new version look at the same database at the same time. If the schema only fits the new code, the old replicas start failing. If it only fits the old code, the new replicas never come up. Zero-downtime schema migration has exactly one rule: at every moment the database must be able to serve both versions. This article shows how to apply that rule on PostgreSQL and where it will trip you up.
Three approaches, one that works
Maintenance window. Stop traffic, run the migration, release the new version. Honest and simple. Still defensible for a single-tenant internal tool. But a product serving several time zones has no window, and every "3 a.m. deploy" tires the team and multiplies mistakes.
Hold the lock, be fast. Run the migration in the same step as the deploy and hope it is quick. It works on small tables. Then the table grows and one day the ALTER TABLE takes minutes instead of seconds. During those minutes every query against the table waits in the lock queue. The connection pool fills up, the backend times out, users see errors.
Expand and contract. Split every change into two backward-compatible halves: expand the schema first, teach the code to understand both shapes, then remove the old shape. More steps and more discipline. In return it works regardless of table size and can be rolled back at every stage. For a product running in production this is the right choice; the rest of the article is its detail.
Worked example: renaming a column
RENAME COLUMN is atomic, but the old code cannot see the column one second later. Instead you spread the change over at least three deployments.
The first deployment expands the schema and switches the code to dual writes:
ALTER TABLE orders ADD COLUMN customer_ref text;
The application now writes both customer_id and customer_ref and still reads the old one. Old replicas do not know the new column, and that does not bother them because the column accepts nulls.
The second step is backfilling existing rows. A single UPDATE orders SET customer_ref = customer_id over millions of rows means a long lock and a WAL burst. Move in small batches by primary key range:
UPDATE orders
SET customer_ref = customer_id::text
WHERE id BETWEEN 1 AND 10000
AND customer_ref IS NULL;
Each batch commits in its own transaction; leave a short pause between batches so replication lag does not balloon.
The second deployment switches reads to the new column and keeps writes dual. Waiting a week here to confirm everything is healthy is not a sign of weakness.
The third deployment contracts: dual writes go away and the old column is dropped. Dropping a column is its own trap, covered below.
What locks really cost
Most forms of ALTER TABLE in PostgreSQL take an ACCESS EXCLUSIVE lock. Acquiring the lock is fast, but acquiring it requires every open transaction on the table to finish first. One long-running report query or a forgotten idle-in-transaction session holds your migration back. The real disaster is this: a migration waiting for the lock also blocks every ordinary SELECT queued behind it. One slow query brings down the whole application this way.
The defense has two layers. Set a short lock timeout on the connection that runs the migration:
SET lock_timeout = '3s';
ALTER TABLE orders ADD COLUMN customer_ref text;
If the lock cannot be acquired within three seconds the statement fails; your migration tool retries and the application never notices. The second layer is having the migration tool do that retry itself. Most tools do not do it out of the box; you wrap them.
Rather than memorizing which statement takes which lock and whether it rewrites the table, measure. Since version 11, adding a column with a constant default does not rewrite the table. Adding a NOT NULL constraint requires a full scan; the fix is to add CHECK (col IS NOT NULL) NOT VALID first and run VALIDATE CONSTRAINT in a separate transaction. Validation takes only a SHARE UPDATE EXCLUSIVE lock and does not block reads or writes.
Indexes
CREATE INDEX blocks writes to the table. In production always use the concurrent form:
CREATE INDEX CONCURRENTLY idx_orders_customer_ref ON orders (customer_ref);
This statement has two habits. It cannot run inside a transaction block; if your migration tool wraps every migration in BEGIN automatically, you must disable that for this migration. Second, if it is interrupted it leaves behind an index marked INVALID. That index costs maintenance but serves no queries. Before rerunning the migration, find rows in pg_index with indisvalid = false and drop them. Teams that skip this spend weeks later on the "index exists but the query is slow" puzzle.
Why dropping a column bites back
DROP COLUMN holds its lock briefly and does not rewrite data. The problem is not in the database but in the application. Many ORMs read a table with SELECT * or with the column list from the model definition. The moment the column disappears, still-running old replicas get "column does not exist". So the order is: first ship a version whose code never mentions the column, confirm every replica has rolled over, and only then drop it. Most ORMs let you mark a column as ignored; that is exactly the job of the intermediate release.
The application side: speaking both shapes
The hard half of a schema migration is not SQL. It is making the application behave correctly against both schemas. Patterns that work in practice:
Make the code path that writes the new column switchable by configuration. If you see an error you flip the switch instead of rolling back a deploy.
Run a consistency check during the dual-write period: a query that counts rows where the old and new columns disagree, with an alert threshold. Dual writes you do not measure are not trustworthy.
Keep migration files and application code in the same repository and the same review. If a separate "database team" runs migrations, responsibility for keeping the two in sync becomes nobody's.
Scale problems in multi-tenant schemas
If you use a schema or a database per tenant, every migration runs hundreds of times. Three extra rules apply. The migration must be idempotent: if it dies halfway in one tenant and is rerun, it must not fail the second time. Track which tenant is on which version in one place; "all done" is only true if it was measured. And a tenant that signs up while the migration round is in progress must still receive the correct version; otherwise the round finishes, the new tenant is left on the old schema, and its first request fails.
When not to choose this path
The expand and contract cycle means three deployments and a week of waiting per change. On a system with no production users yet that burden is pointless; drop the table and recreate it. The same goes for single-tenant internal tools with an accepted maintenance window. The discipline pays off where replicas come and go and downtime lands directly on users.
One last warning: "this table is small, it will be fine" is among the most expensive sentences in operations. The small table grows, but the migration habit stays the same. Make lock timeouts, concurrent indexes and batched backfills a habit on day one, so there is nothing to change when you grow.
[OK] 30 records · 9 featured · 1 retainer · 20 standard
/02 services
services.tree
> note
· Prices are in USD. Volume discounts of 7% for 6-12 day projects and 14% for 13+ day projects apply. Programs of 100+ person-days are priced individually. Prices are negotiable based on scope, urgency and long-term collaboration.
> ls ~/products/ --open-source
[OK] 2 entries · 1 live · 1 pre-release
/03 products
products.list
liveopen source · mit/prod/01
filex
the self-hosted file manager that embeds anywhere
single go binary · vue/react/web-component embed · 5 storage drivers · realtime collab · rbac · native multi-tenancy · mcp server for ai agents
We build infrastructure, automate everything, and keep systems running.
BRF Tech is a Bursa-based DevSecOps and software consulting firm. We operate across 30 service areas, from Kubernetes cluster management to serverless platform setup, CI/CD pipeline design to AI agent development.
What we do is simple: we set up your systems, automate them, and make sure they won't wake you up at 3 AM. We codify your infrastructure with Terraform, move your deployments to GitOps with ArgoCD, and monitor everything with Prometheus. If something breaks — we intervene before it does.
We're against vendor lock-in. We work with open-source tools, self-hosted solutions, and industry-standard technologies. We build your own serverless platform on Knative, isolate with Kata Containers, manage your secrets with Vault. Every project is delivered with clear scope, clear timeline, clear pricing.
DevSecOps & CI/CD
Kubernetes & Serverless
Infrastructure as Code
AI & Agent (MCP/ACP)
Security & Zero Trust
Full-Stack Development
Self-Hosted Solutions
> grep -i question ~/faq.md
[OK] 3 entries · click to expand
[?]What services are included in BRF Tech's DevOps solutions?▾
We offer CI/CD pipeline setup, Kubernetes cluster management, container migration, infrastructure automation with Terraform, monitoring & alerting, in-house technology installations and DevOps consulting services.
[?]What are your software development services?▾
We offer MVP development, existing software performance optimization, backend API development (Node.js, Go, Python) and full-stack application development. Every project is delivered with minimal technical debt.
[?]Which DevOps tools do you work with?▾
From Kubernetes and Docker to Talos Linux, from Terraform and OpenTofu to Ansible, and across Jenkins, GitLab CI/CD, GitHub Actions, ArgoCD, Flux, Helm and Rancher, we work with industry-standard tooling. For observability and error tracking we use Prometheus, Grafana, Loki, Tempo, OpenTelemetry and GlitchTip; for networking, Cilium and Envoy; for test automation, Playwright, Cypress, k6, Locust, Testcontainers and Trivy. We determine the most suitable toolset together, based on your project’s needs.
Automated build, test and deployment pipelines with GitHub Actions, GitLab CI/CD.
Automated build, test and deploy pipeline setup for existing or new repositories. GitHub Actions, GitLab CI/CD or preferred tool is used. Separate workflows are defined for staging + production environments.
Container architecture migration, cluster setup and orchestration.
Production-ready Kubernetes cluster setup on bare-metal or cloud. Includes Helm charts, Ingress, TLS certificate management, namespace isolation and RBAC.
Cozystack is an open-source cloud platform built on Kubernetes and the VMware alternative we recommend most. Virtual machines run on KubeVirt, while managed PostgreSQL, MySQL, Redis and Kafka services and dedicated tenant Kubernetes clusters are provisioned from the same dashboard and API. Includes installation on your own bare-metal servers on Talos Linux, replicated storage with LINSTOR/DRBD, network isolation with Kube-OVN, tenant and quota structure, monitoring, backup and a disaster recovery plan. After deployment we also handle upgrades, operations and support; since the platform is open source, there are no license fees and no vendor lock-in.
Packages
Virtual machine support on an existing Kubernetes cluster with KubeVirt
$2,800
4–6 person-day
Pilot setup (3 nodes, virtual machines + managed database)
$3,500
5–8 person-day
Production setup (HA + replicated storage + monitoring + backup)
$6,500
10–16 person-day
Multi-tenant private cloud (tenants, quotas, managed service catalog)
Planned, low-downtime migration of vSphere/ESXi environments to Cozystack, OpenStack or VHI.
A planned move to a VMware alternative for environments whose costs rose after Broadcom moved VMware licensing to subscription bundles, or whose renewal is coming up. We start with an inventory of your vSphere/ESXi estate (virtual machines, disks, networks, dependencies), then compare today's license cost with the total cost of the target platform. In most environments the target is Cozystack; OpenStack fits large-scale IaaS needs and Virtuozzo Hybrid Infrastructure fits storage-heavy service providers. A pilot runs on non-critical virtual machines, disks are converted with tools such as virt-v2v and Forklift, and the migration proceeds in waves within maintenance windows, with a rollback plan for every step.
Packages
Inventory + cost comparison + migration plan
$2,400
3–5 person-day
Pilot migration (up to 10 virtual machines)
$3,500
5–8 person-day
Phased migration (up to 100 virtual machines)
$9,750
15–22 person-day
Full data center migration (network + storage + backup)
OpenStack deployment, Neutron networking, Ceph/Cinder storage, upgrades and operations.
OpenStack deployment and operations in your own data center. A containerized, highly available control plane with Kolla-Ansible; Nova (compute), Keystone (identity), Glance (images) and Horizon (dashboard). On the network side, Neutron with OVN, VLAN/VXLAN, provider and tenant networks, floating IPs and the Octavia load balancer; on the storage side, Cinder block storage on Ceph RBD and optional S3-compatible object storage. For existing deployments: health checks, planned upgrades between releases, monitoring and alerting setup, and a backup strategy.
Packages
Health check + improvement report (existing deployment)
$2,400
3–5 person-day
Lab / PoC setup (single region, 3 nodes)
$3,500
5–8 person-day
Release upgrade (existing deployment)
$3,900
6–10 person-day
Production setup (HA control plane + Ceph + Neutron/OVN)
Hyperconverged infrastructure: multi-tenant cloud plus block and S3-compatible storage for service providers.
Hyperconverged infrastructure with Virtuozzo Hybrid Infrastructure, where compute, storage and networking run on the same servers. Block, S3-compatible object and NFS storage on software-defined storage, plus backup storage (Backup Gateway) for backup products such as Acronis. For service providers: a multi-tenant cloud with domains and projects, quotas, a self-service panel and an OpenStack-compatible API. Includes hardware sizing, network design, cluster deployment and expansion, version upgrades and operations support.
Packages
Pre-deployment assessment + hardware sizing
$2,400
3–4 person-day
Version upgrade / cluster expansion
$2,800
4–6 person-day
Hyperconverged cluster setup (storage + compute)
$3,900
6–10 person-day
Service provider setup (multi-tenant cloud + S3 + backup storage)
Migration of existing applications to container architecture.
Migration of existing applications to container architecture. Docker image design, Compose configuration and conversion to Kubernetes manifests. Kata Containers / gVisor can be included for advanced isolation.
Health check of your current environment, capacity planning, platform selection and a roadmap.
An independent assessment of your current virtualization environment (VMware vSphere, OpenStack, VHI or other KVM-based platforms). The health check covers cluster configuration, resource overcommit, storage latency, forgotten snapshots, backup and patch status. Capacity planning measures CPU, memory and storage growth to work out the hardware you will need next. Platform selection decides which workload should run where: Cozystack for teams running containers and virtual machines side by side, OpenStack for large-scale IaaS, VHI for storage-heavy service providers; staying on the current platform is also an option for some workloads. The deliverable is a written report with a cost comparison and a prioritized roadmap.
In-house tools like GitLab, Nextcloud, VPN, mail server.
Installation of tools like GitLab, Mattermost, Nextcloud, mail server, VPN, internal monitoring on company servers. Deployed on Docker Compose or Kubernetes.
DB + cache + code bottleneck identification and resolution.
Profiling of existing application, identification and resolution of bottlenecks. DB query optimization, caching layer (Redis), service-level improvements.
MCP/ACP agents, RAG pipelines, multi-agent orchestration, model serving.
LLM integration (OpenAI, Claude, Gemini, local models), tool-augmented AI agents with MCP (Model Context Protocol) and ACP (Agent Communication Protocol). RAG pipeline, multi-agent orchestration, model serving (vLLM/Ollama). AI layer for existing business processes.
Event-driven data pipeline with Apache Kafka, RabbitMQ, NATS. ETL/ELT orchestration with Apache Airflow, real-time data streaming, CDC (Change Data Capture). Data lake/warehouse design, schema registry, dead letter queue management.
Packages
Kafka/RabbitMQ setup + basic producer/consumer
$3,500
5–7 person-day
ETL pipeline (Airflow + source → warehouse)
$5,200
8–12 person-day
Full event-driven architecture (CDC + streaming + DLQ)
Hands-on technical training for your teams. Docker & container fundamentals, Kubernetes operations, Terraform IaC, CI/CD best practices, AI/LLM integration workshops. Practical exercises in live lab environments, content customized by skill level.
E2E test automation with Playwright, Cypress, load testing with k6/Gatling. Pre-deploy quality gate integrated into CI/CD pipeline, test coverage reporting, visual regression testing. Reduce test writing time by up to 60% with AI-assisted test generation.
Your Collected Personal Data, Collection Method and Legal Basis
Any information that identifies or makes you identifiable is considered "personal data." When you visit our website or use our services, contact information such as your name, surname, email address, phone number, as well as your IP address and browser cookie data may be collected through automatic or semi-automatic means. This data is processed based on the legal grounds specified in Article 5 of the Personal Data Protection Law No. 6698: "being directly related to the establishment or performance of a contract" and "being mandatory for the legitimate interests of the data controller, provided that it does not harm the fundamental rights and freedoms of the data subject."
Purpose of Processing Your Personal Data
Your collected personal data is processed for the purposes of providing and improving our services, fulfilling customer requests, meeting legal obligations, conducting information security processes and managing communication activities. Your data is processed in a limited and proportionate manner for the stated purposes; when the purpose ceases to exist, data is deleted, destroyed or anonymized.
To Whom and For What Purposes Collected Personal Data May Be Transferred
Your personal data may be transferred to public institutions and organizations as required by legal regulations, to our business partners and technical infrastructure providers for the purpose of delivering services, and to lawyers and consultants in legal disputes. Transfers are carried out in accordance with Articles 8 and 9 of the Law, with necessary technical and administrative measures in place.
Your Rights as a Data Subject
In accordance with Article 11 of Law No. 6698, you have the right to: learn whether your personal data is being processed; request information if it has been processed; learn the purpose of processing and whether it is used in accordance with its purpose; know the third parties to whom it is transferred domestically or abroad; request correction if it has been processed incompletely or incorrectly; request deletion or destruction within the framework of conditions set out in Article 7 of the Law; request that the operations carried out be notified to third parties to whom data has been transferred; object to any adverse result arising from the analysis of data exclusively through automated systems; and claim compensation for damages in case of unlawful processing. You may contact us through the communication channels on our website to exercise these rights.
Terms & Conditions
Terms & Conditions
1. Service Scope and Changes
BRF Tech conducts its software development, DevOps consulting, infrastructure setup and technical support services within the framework of these terms and conditions. The scope of service is determined separately for each project and finalized through mutual agreement. BRF Tech reserves the right to update the scope, pricing and technical details of its services, provided that prior notice is given.
2. Privacy Policy
Personal data collected within the scope of our services is processed in accordance with our Privacy Policy. For detailed information, please review our Privacy Policy page.
3. Service Usage and Responsibilities
Clients agree to use the provided services solely within legal and ethical boundaries. BRF Tech reserves the right to suspend or terminate services in the event of misuse, violation of third-party rights, or use for illegal activities. The client is responsible for the accuracy and currency of the information provided within the project scope.
4. Payment Terms
Payment terms are determined on a project basis and finalized through mutual agreement at the start of the project. Unless otherwise specified, invoices are payable within 15 days of project delivery. A monthly late payment interest of 2% may apply to overdue payments.
5. Cancellation and Refund Policy
Cancellation requests must be submitted in writing at least 7 days before the service start date. For projects already in progress, billing is based on the proportion of completed work. No refunds are issued for software and infrastructure components specifically developed within the project scope.