The Quiet Cause of Sporadic 502s: Idle Connection Timeouts and Keepalive Ordering

Everything looks fine at idle, and nothing breaks under load. But a few times a day, a random request returns 502, or a client sees "connection reset by peer". Retry it and it succeeds. The application log has no trace of the failed request, because it never reached the application. Most of the time the cause is the same: two components in the chain have their idle connection timeouts in the wrong order.

Where it comes from

A load balancer, a reverse proxy or an HTTP client library does not close its connection to the backend after every request. It keeps the connection in a pool and reuses it. That is the right default, since it avoids paying for a TCP and TLS handshake on every call.

A pooled connection has two ends, and each end has its own idle timeout. The server decides "this connection has been idle long enough, I'm closing it". The client decides "this connection has been idle long enough, I'm dropping it". If the server's timeout is shorter, a race is unavoidable: the server starts closing the connection, and while its FIN is in flight, the client picks the same connection from the pool and writes a request. The server no longer accepts requests on that socket and answers with a reset. On the client side this looks like a connection that died before any response header arrived, and a proxy turns that into a 502 for its own caller. nginx logs it as "upstream prematurely closed connection while reading response header from upstream".

Because it depends on timing, the failure is rare and looks random. Under heavy traffic connections never sit idle long enough, so the bug hides. It shows up mostly on the first requests after a quiet period.

The rule: whoever sends the request lets go first

The fix fits in one sentence. On every connection, the side that sends requests must have a shorter idle timeout than the side that serves them. That way the client always retires the connection, and the server never closes a socket the client is about to write to.

In a longer chain, the rule applies to each hop separately. A path of client, load balancer, reverse proxy and application server has three connections, and on each one the serving side must be the more patient one. In practice timeouts grow as you move inward: if the load balancer's idle timeout is 60 seconds, the proxy behind it waits at least 65; if the proxy's upstream pool drops connections after 60 seconds, the application server waits 65.

Leave a margin of a few seconds. Equal values do not remove the race, because the two clocks do not start at the same instant.

Why the defaults are a trap

The reason this is so common is that popular defaults are ordered exactly backwards. Application servers pick short idle timeouts to save resources, load balancers pick long ones.

The Node.js built-in HTTP server defaults keepAliveTimeout to 5 seconds. Gunicorn's --keep-alive defaults to 2 seconds. Apache's KeepAliveTimeout defaults to 5 seconds. Meanwhile cloud application load balancers typically ship with a 60 second idle timeout. Put these next to each other with defaults and you get precisely the wrong order.

A typical Node.js fix:

const server = http.createServer(app);
server.keepAliveTimeout = 65_000;
server.headersTimeout = 66_000;
server.listen(port);

Keep headersTimeout above keepAliveTimeout; otherwise the server can hit its own header timeout while waiting on a reused connection.

In Gunicorn the same change is --keep-alive 65. In Go, the IdleTimeout field on http.Server controls it; if it is zero, ReadTimeout is used, and if both are zero idle connections are never closed by the server.

The upstream pool in a reverse proxy

In nginx, reusing upstream connections is something you turn on explicitly. Once you do, the time nginx keeps a pooled connection becomes another link in the chain:

upstream app_backend {
    server app.example.com;
    keepalive 32;
    keepalive_timeout 60s;
}

server {
    location / {
        proxy_pass http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

keepalive 32 is the number of idle connections cached per worker process, not a cap on total connections. keepalive_timeout is how long a pooled connection may stay idle, and the application behind it must wait longer than that. Clearing the Connection header stops a client's close from being forwarded upstream and defeating the pool.

If you never enable upstream keepalive, this race cannot happen: each request uses a fresh connection that closes after the response. The cost is a handshake per request and a pile of sockets in TIME_WAIT on the proxy under heavy traffic.

Why retrying is not the fix

The obvious idea is to resend the request that failed. Some HTTP clients already do this silently: if a reused connection dies before any response arrives, they replay idempotent requests on a new connection.

The trouble is non-idempotent requests. The client cannot know exactly where the connection broke; the server may have read the request and started processing it. That is why nginx, when proxy_next_upstream passes a failed request to another server, does not do so for non-idempotent methods such as POST, LOCK and PATCH unless you add non_idempotent explicitly. Turning that on means accepting the risk of duplicate payments or orders.

Retries are a safety net for the rare cases left after the timeouts are ordered correctly. Used instead of the real fix, they do not remove the error, they just leave it visible on POST requests only.

Long idle connections and network devices

The same problem has a non-HTTP form in database and message broker connections. When a pooled connection sits idle for a long time, a NAT gateway or firewall in between expires its connection tracking entry on its own schedule, without telling either end. The next query is either dropped or reset; the former means the query fails only after a long TCP retransmission sequence.

Two tools help here. The first is TCP keepalive: the kernel sends empty probes on idle connections so devices in the path keep the entry alive. On Linux net.ipv4.tcp_keepalive_time defaults to 7200 seconds, far longer than most NAT timeouts, so it has to be shortened at the application or driver level. PostgreSQL has tcp_keepalives_idle on the server side and keepalives_idle in the client library for exactly this. The second is the pool's maximum connection lifetime: the pool retires connections itself before network devices can. With both in place, the chance of getting a dead connection from the pool is practically gone.

HTTP/2 and gRPC

HTTP/2 solves part of this. Before closing a connection the server sends a GOAWAY frame stating which streams it will still process, and the client moves new requests to a new connection. That makes shutdown orderly, but it does nothing about middleboxes.

gRPC has client-side keepalive pings for that. The trap is that if the client pings more often than the server allows, the server closes the connection with "too_many_pings". The client's ping interval must be longer than the minimum the server permits. It is the same rule again: neither end should pick its value without knowing what the other expects.

Confirming the diagnosis

The fastest check is the time distribution of the errors. If 502s are not proportional to load but cluster after quiet periods, suspect an idle timeout race. Look for proxy log lines saying the upstream closed the connection early, and for requests with no matching entry in the application log.

To reproduce it, send one request with a pooling client, wait a few milliseconds longer than the application server's idle timeout, and send a second request on the same connection. Sweep the wait in small steps around the server's timeout and the failure becomes repeatable within a narrow window. After the fix, the same test should pass at any wait.

When it is not worth the effort

For very low traffic internal services that are not latency sensitive, disabling upstream connection reuse altogether is a legitimate choice. Each request opens a new connection and pays for the handshake, but the race never occurs and there is no chain to configure. Below a handful of requests per second, that cost is too small to measure.

Everywhere else, writing down the idle timeout of every hop in a table and checking that it grows inward is half an hour of work. Keeping that table current is part of the job: when a new proxy or load balancer is added to the path, this ordering is the thing that breaks most often.