Why a Distributed Lock Is Not Enough: Leases, Fencing Tokens, and Choosing Between Redis, PostgreSQL and etcd

Putting a lock around work so that two processes never do it at the same time is one of the most common fixes in distributed systems, and one of the most commonly wrong ones. On a single machine a mutex rests on the operating system's guarantees: if the process dies the lock is released, and two threads cannot be inside at once. A lock taken over the network has none of those guarantees. The holder can pause, lose its connection, or drift its clock, and while that happens the lock can expire and be handed to someone else. When the first process wakes up, it still believes it holds the lock. This post explains why that is unavoidable, which lock backend fits which job, and the one mechanism that actually makes a lock safe: the fencing token.

Decide what the lock is for

There are two kinds of lock, and the difference drives the whole design.

An efficiency lock prevents duplicate work, but duplicate work is not a disaster. Generating the same report twice, warming the same cache twice, occasionally sending the same email twice. The goal is to avoid wasted effort, and having two holders once in a while is acceptable.

A correctness lock protects cases where two concurrent writers corrupt data: updating an account balance, rewriting a file, assigning a resource to exactly one tenant. If this lock ever has two holders, even for a moment, the data is damaged.

A simple lock with a timeout is enough for efficiency. For correctness, no lock is enough on its own; the protected resource itself must be able to reject a stale holder. Picking a backend before making this distinction means answering the wrong question.

Why the lock must expire, and why that hurts

A network lock cannot be held forever. If the holder crashes, the lock is never released and the system stalls. So every distributed lock has a lease: if the holder does not renew it in time, the lock drops on its own.

The problem is that the holder may not notice its lease has run out. This sequence can happen in any garbage collected runtime, on any virtual machine, on any busy host:

1. Process A takes the lock with a 30 second lease.
2. Halfway through the work, A hits a long GC pause, or its VM is suspended.
3. The lease expires, process B acquires the lock and starts writing.
4. A wakes up, still thinks it holds the lock, and keeps writing.

Having A check the lock before writing does not help, because the pause can land between the check and the write. A longer lease lowers the odds but never removes them. This is not an implementation bug, it is the nature of a leased lock.

Fencing tokens: tie the lock to the resource

The fix is to issue a monotonically increasing number every time the lock is granted. A gets token 33, B gets token 34. Every write carries the writer's token. The protected resource remembers the highest token it has seen and rejects any write that arrives with a lower one. When A wakes up and tries to write with 33, the resource has already seen 34 and refuses.

If the resource is a database row, the check is a single conditional update:

UPDATE jobs
SET status = 'done', last_token = 34
WHERE id = 1207 AND last_token < 34;

If zero rows were affected, you no longer own the lock and should abandon the work. Because the check and the write are one atomic statement, the gap between them disappears.

The hard part is on the resource side, not the lock side. If you are writing to something that cannot check a token (a third party API, a plain file system, a mail server), you cannot fence. The best you can do there is make the operation idempotent, so that doing it twice has the same effect as doing it once. If you cannot do that either, what you have is an efficiency lock, and you should design accordingly.

Choosing a backend

Single node Redis

The most common setup, and a good one for efficiency locks. The lock is taken atomically with a random value and an expiry:

SET lock:report <random-value> NX PX 30000

Releasing it with a plain delete is wrong: if your lease expired and someone else took the lock, you just deleted their lock. Release with a Lua script that checks the value first:

if redis.call("get", KEYS[1]) == ARGV[1] then
  return redis.call("del", KEYS[1])
else
  return 0
end

Redis is limited by asynchronous replication. If the primary grants a lock and fails before replicating it, the promoted replica has no record of it and a second process takes it again. The Redlock algorithm, which takes a majority across several independent Redis nodes, aims to reduce that risk, but it relies on timing and pause assumptions, its use as a correctness lock is disputed, and it does not produce fencing tokens. You can generate a token with an increment command, but that counter can also go backwards on failover. If you need correctness, do not pick Redis.

PostgreSQL advisory locks

If you already run PostgreSQL, this needs no extra infrastructure. pg_try_advisory_lock(key) takes a session level lock, while pg_advisory_xact_lock(key) ties the lock to the current transaction. Because the lock lives on the connection, a crashed holder closes its connection and the lock is released automatically; there is no separate lease to manage.

There are two traps. First, session level locks are unreliable behind a connection pooler running in transaction mode: the server connection that holds your lock may be handed to another client on your next query. Use the transaction level variant there. Second, a process only notices a dropped connection on its next query, and the lock may have moved on in the meantime. When the protected data lives in the same database, the strongest option is to do the work inside the same transaction as the lock. If the lock goes away, so does the transaction.

etcd, ZooKeeper and Kubernetes Lease

These are the backends built for correctness locks. They run a consensus protocol, so losing a node does not cause a lock to be granted twice. In etcd a lock is a key attached to a lease, and key revisions increase monotonically across the cluster, which gives you a natural fencing token. ZooKeeper's sequential ephemeral nodes serve the same purpose. For a controller running in Kubernetes, the Lease object in the coordination.k8s.io group is the standard way to elect a leader, and the client libraries ship helpers for it.

The cost is operational: a consensus cluster needs at least three members and its latency is higher than a Redis call. If you take thousands of short locks per second, these are the wrong tool; at that point consider partitioning the work instead of locking it.

Pitfalls

Sizing the lease to the typical job. If jobs usually take 10 seconds and you set a 15 second lease, a slow day stretches a job to 20 seconds and the lock drops mid flight. Either renew the lease in the background or split the work into pieces much shorter than the lease.

Ignoring renewal failures. Code that does not stop immediately when renewal fails keeps working without owning the lock. A failed renewal must be a signal that cancels the work.

Trusting the wall clock. Computing lease expiry from local time breaks when the clock is adjusted or a VM resumes from suspension. Measure durations with a monotonic clock and let the lock server have the final word.

Holding the lock across slow calls. Taking a lock and then waiting on an external API ties the lock's lifetime to latency you do not control. Keep only the short, protected step inside the lock.

Using a lock as a queue. If hundreds of workers keep retrying the same lock, you do not have a locking problem, you have a work distribution problem. One queue row per job and something like SELECT ... FOR UPDATE SKIP LOCKED scales far better than a single hot lock.

When not to use a lock

A distributed lock moves a concurrency problem somewhere else; it does not solve it. If the data model can solve it, do that first: unique constraints, conditional updates, optimistic concurrency with a version column, or partitioning so that each record belongs to exactly one worker are usually simpler and more correct than a lock. If you still need one, ask first whether it is an efficiency lock or a correctness lock. For efficiency, a simple Redis lock is enough and anything more is wasted complexity. For correctness, make sure the resource can reject a stale holder alongside the lock; if it cannot, the choice of lock backend will not change much.