Idempotency Keys: Handling Retried POST Requests Without Doing the Work Twice, Scope, Concurrency and Stuck Records
Retrying a timed out POST can create the same order twice. Choosing between natural idempotency, a key stored in the same transaction as the business data and a gateway cache, a working PostgreSQL flow, and the pitfalls that quietly defeat the mechanism.
The network does not always tell you what happened to a request. A client sends a payment or an order, the connection times out, and the client is stuck between two possibilities: the request never reached the server, or it was processed and the response was lost on the way back. Without a retry the user's action may be half done; with a retry the same order may be created twice. GET, PUT and DELETE are idempotent by definition, but a POST that creates a resource is not. An idempotency key closes that gap: the client generates one key per logical operation, and the server answers a second request carrying the same key with the original result instead of doing the work again.
The idea is simple. The real work lives in concurrency, crashes and scoping.
Three approaches, and where each fits
Natural idempotency. Making the operation repeatable on its own is always the first choice. The client generates the resource ID itself (a UUID, for example) and sends PUT /orders/{id}, or there is a unique constraint on a business key such as the customer's order reference. The second request hits the constraint and the server returns the existing record. No extra table, no middleware, and the database guarantees correctness. Most endpoints that create resources can be reshaped this way.
A key and response stored on the server. When the operation cannot be made naturally idempotent (it calls an external provider, creates several records, moves money), the client sends an Idempotency-Key header. The server stores the key, a hash of the request and the response it produced in its own database, inside the same transaction as the business data. This is the model payment APIs use, and there is an IETF draft standard for the header.
A cache at the gateway. Some teams keep the key at the API gateway or in a separate store like Redis and cache the response there. It is easy to set up, but because the record is not written in the same transaction as the business data, it fails exactly when it matters: the work is committed, the process dies before the cache write, and the retry does the work a second time. Do not use this for anything where a duplicate is expensive, such as money or stock.
If I have to pick: natural idempotency first, and when that is not possible, a key stored in the same database as the business data. A separate cache only makes sense where duplicates are harmless and you just want to shed load.
Table and flow
The plainest version on PostgreSQL:
CREATE TABLE idempotency_keys (
client_id bigint NOT NULL,
idem_key text NOT NULL,
request_hash bytea NOT NULL,
state text NOT NULL DEFAULT 'processing',
response_code integer,
response_body jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (client_id, idem_key)
);
When a request arrives, claim the key inside the same transaction that will write the business data:
BEGIN;
INSERT INTO idempotency_keys (client_id, idem_key, request_hash)
VALUES ($1, $2, $3)
ON CONFLICT (client_id, idem_key) DO NOTHING
RETURNING idem_key;
If a row comes back, the key is yours: write the business records, then store the response in the same transaction and commit.
UPDATE idempotency_keys
SET state = 'done', response_code = $4, response_body = $5
WHERE client_id = $1 AND idem_key = $2;
COMMIT;
If no row comes back, the key has been used before. Read the record; if the stored request_hash matches the incoming one, replay the stored response with the same status code, and if it does not, reject the request.
PostgreSQL quietly does something useful here: a concurrent second INSERT with the same key waits on the unique index until the first transaction finishes. If the first commits, the second sees the conflict and replays the stored response. If the first rolls back, the second gets the key and does the work from scratch. Both outcomes are correct.
Long operations and external calls
That flow is ideal when the work takes milliseconds. If there is a step that takes seconds, like calling a payment provider, the waiting second request holds a database connection the whole time and your pool runs dry quickly. In that case record the key as processing in a separate short transaction and commit right away. A concurrent request with the same key sees processing and fails fast instead of waiting. The IETF draft suggests 409 Conflict for that case and 422 for reusing a key with a different body. Adding Retry-After tells the client when to try again.
The external call is a duplicate risk of its own. If your process calls the provider and crashes before recording the result, the next attempt calls the provider again. If the downstream service accepts idempotency keys, pass it a stable key derived from yours, so the retry chain stops at the same point at every hop. If it does not, split the operation into steps, record each completed step, and on retry first ask the provider whether the operation already happened.
Pitfalls
The client generates a new key on every attempt. This is the most common mistake and it defeats the whole mechanism. The key is generated once per logical operation, outside the retry loop, and persisted. A mobile client should be able to reuse the same key after the app is killed and reopened.
import uuid
import requests
key = str(uuid.uuid4())
for attempt in range(5):
try:
r = requests.post(url, json=payload,
headers={"Idempotency-Key": key}, timeout=10)
break
except requests.exceptions.ConnectionError:
continue
The key is scoped globally. A key only means something together with the client identity. If the scope is the whole system, two customers who happen to generate the same key will see each other's stored responses. That is a data leak. Always include the authenticated client, and in a multi-tenant system the tenant, in the primary key.
The body is not compared. Same key with a different body means the client either has a bug or is reusing keys. Replaying the stored response silently swallows the second request. Store a hash of the request and reject on mismatch. Hash the parsed and normalized body so that field order and whitespace do not change the hash.
Transient errors get stored. Storing deterministic 4xx responses like validation errors is right; the same request should get the same error. But if you store a 5xx caused by a timeout or a failed dependency, the client gets that error forever. When the transaction rolls back, the key record rolls back with it and the problem solves itself; when you record the key in a separate transaction, delete the record explicitly on transient failure.
Stuck processing records. If the process crashes after claiming the key, the record stays in processing and the client gets 409 forever. Give the record a lock timestamp and hand it to a recovery job after a deadline. Recovery is not "delete and redo": the side effect may already have happened, so first find out how far the operation got.
Retention shorter than the retry window. You do not need to keep keys forever, but you must keep them longer than the longest retry window of your clients and of any queues in between. Payment APIs commonly use 24 hours. Clean up in small batches rather than one large DELETE, or you get a job that locks the table and leaves a pile of dead rows behind.
The stored response goes stale. A stored response is the result at the time of the operation. If the resource changed later, a retry sees the old state. That is not a bug, it is what idempotency means, but your client documentation should say so.
When you do not need it
If the operation is already naturally idempotent, an extra layer only adds maintenance. A PUT with a client-generated ID, or a unique constraint on a business key, is usually enough. Operations where a duplicate is harmless (recomputing a counter, refreshing a cache) do not need it either. Idempotency keys are for places where a duplicate has a real cost and the operation cannot be made repeatable some other way. Even there, unless the key is stored in the same transaction as the business data, scoped to the client and checked against the body, it gives the impression of a guarantee without giving the guarantee.