Ephemeral Port Exhaustion and a Full conntrack Table: TIME_WAIT, Connection Reuse and the Kernel Knobs That Matter

Under load, a service starts failing while CPU is idle, memory is fine and the database is relaxed. The logs show connect: cannot assign requested address, or clients report timeouts with no explanation. The kernel log has a line like nf_conntrack: table full, dropping packet. Both symptoms belong to the same family: a per-connection kernel resource has run out. One is the local port range used for outgoing connections, the other is the connection tracking (conntrack) table. Both fill up silently, and both are usually "fixed" with the wrong knob.

Why local ports run out

A TCP connection is identified by four values: source IP, source port, destination IP, destination port. If your service talks to a single database or a single upstream, three of the four are fixed and only the source port varies. On Linux that range is set by net.ipv4.ip_local_port_range, and the default is 32768 60999, roughly 28 thousand ports.

That sounds like plenty. The real cost is in closed connections. The side that closes first keeps the socket in TIME_WAIT, and on Linux that lasts 60 seconds, hardcoded in the kernel. A client opening and closing 500 connections per second to the same destination parks 30 thousand ports in TIME_WAIT within a minute. The range is exhausted and new connect() calls fail with EADDRNOTAVAIL.

The usual culprits: code that builds a new HTTP client per request, an HTTP library with keepalive disabled, scripts that connect to the database for every query without a pool, and health checks that run very often and close the connection each time.

To see what is going on:

ss -s
ss -tan state time-wait | wc -l
ss -tan state time-wait '( dport = :5432 )' | wc -l
sysctl net.ipv4.ip_local_port_range

The third command counts TIME_WAIT sockets towards one destination port. If the number approaches the size of your port range, you have found the problem.

Fix it in the right order: reuse connections first

There are three approaches, and their order matters.

Connection reuse. This is the actual fix. Keep one HTTP client instance for the lifetime of the process, leave keepalive on, and reach the database through a pool. When 500 requests per second flow over a few dozen persistent connections, TIME_WAIT stops being a problem, and you also stop paying a TCP and TLS handshake per request, so latency drops. The most common mistake in almost every language is creating the client object inside the request handler. That is the first thing to look for in review.

Wider range and tcp_tw_reuse. If reuse is not possible right now, or you need to buy time, widen the range:

sysctl -w net.ipv4.ip_local_port_range="10240 65535"
sysctl -w net.ipv4.tcp_tw_reuse=1

tcp_tw_reuse lets the kernel safely reuse a port in TIME_WAIT for a new outgoing connection, using TCP timestamps to tell old segments from new ones. It only matters on the side that initiates connections; it does nothing for incoming ones. When you widen the range, make sure none of your services listen on ports inside it. Otherwise the kernel may hand that port to an outgoing connection, and the service gets address already in use on its next restart. Use net.ipv4.ip_local_reserved_ports to carve out specific ports if needed.

More destinations. If the destination IP changes, the same source port can be used again. Reaching an upstream through several IPs, or giving the client several source IPs, multiplies capacity. This is mostly useful at the load balancer and NAT layer.

Knobs that do not do what people think

Lowering net.ipv4.tcp_fin_timeout is a common suggestion, but it does not change TIME_WAIT. That setting controls how long a socket stays in FIN_WAIT_2. TIME_WAIT is fixed on Linux and cannot be shortened with a sysctl.

The old tcp_tw_recycle setting broke clients behind NAT and was removed in Linux 4.12. If an old hardening guide tells you to set it, skip that step; a modern kernel does not have it anyway.

Closing connections with SO_LINGER set to zero sends a RST and skips TIME_WAIT, but the peer may lose in-flight data and see an error. Do not use it as a performance tweak.

One more trap: if the application calls bind() to a specific source IP before connecting, the kernel has to pick the port without knowing the destination, so the port must be unique on its own rather than as part of the full four-tuple. That burns through the range much faster. On Linux, the IP_BIND_ADDRESS_NO_PORT socket option defers port selection to connect(), and most proxies that support multiple source IPs can use it.

Why the conntrack table fills up

On any Linux machine with firewall rules, NAT or container networking, netfilter tracks every flow in a connection tracking table. The table has a ceiling, net.netfilter.nf_conntrack_max. Once it is reached, packets for new flows are dropped. From the application's point of view this is the worst kind of failure: no error message, just lost SYN packets, client retransmits and latency spikes measured in seconds.

To check:

sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
conntrack -S
dmesg | grep -i conntrack

If the drop, early_drop or insert_failed counters in conntrack -S are climbing, the table is either full or hitting insertion races. The conntrack tool ships as a separate package, usually named conntrack or conntrack-tools.

There are two typical reasons the table fills. The first is genuine flow volume: a NAT gateway or reverse proxy handling many short connections. The second is timeouts: nf_conntrack_tcp_timeout_established defaults to five days. Clients that vanish without closing properly hold entries for days.

Tuning conntrack properly

Raising the limit is legitimate. Each entry costs a few hundred bytes, and a few hundred thousand entries are nothing on a modern server:

sysctl -w net.netfilter.nf_conntrack_max=524288
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=3600

When you raise the limit, look at the hash table too. The bucket count is readable and writable at /sys/module/nf_conntrack/parameters/hashsize; if the limit grows but the bucket count stays the same, every lookup walks long chains and CPU cost rises. Persist the settings in a file under /etc/sysctl.d/, with one caveat: if the nf_conntrack module loads after sysctl settings are applied, the conntrack values are silently not applied. Load the module at boot, or reapply the settings after it loads.

Cutting the established timeout to an hour is safe in most environments. If you have long-lived connections that send nothing for more than an hour (some database and message broker clients do), enable TCP keepalive on the application side so the flow stays alive and does not get evicted.

For very high volume, stateless traffic you can skip tracking entirely. A NOTRACK (or CT --notrack) target in the raw table keeps traffic to a given port out of conntrack. The price: stateful firewall rules (ESTABLISHED,RELATED matching) and NAT stop working for that traffic, and you need explicit rules for return packets as well. Do this only where you know the traffic well, for example a cache server reachable only from the internal network.

What to monitor

Both resources are invisible until they are full, so watch them as ratios. With Prometheus node_exporter you get node_nf_conntrack_entries / node_nf_conntrack_entries_limit and node_sockstat_TCP_tw out of the box. Alerting when the conntrack ratio passes 70 percent leaves time to act before the table fills. A sudden jump in TIME_WAIT almost always points to a deploy that broke connection reuse.

When not to tune

Widening the port range and raising the conntrack limit postpone the symptom. If the cause is code that opens a new connection per request, sysctl buys you a few times more headroom; as traffic keeps growing you end up in the same place, now with a higher limit and the memory and CPU that come with it. Start with ss to see how many TIME_WAIT sockets point at which destination, find the code that talks to it, and make it reuse connections. Kernel tuning is the second step, for a measured need.