Load Shedding Under Overload: Concurrency Limits, Queue Time, Deadlines and Health Checks That Sink You

When a service receives more requests than it can handle, it rarely crashes. It does something worse. It keeps accepting work, internal queues grow, every request waits longer than the client's timeout, and eventually the server is busy doing work the client gave up on long ago. CPU sits at 100 percent, throughput looks higher than usual, and the rate of successful responses approaches zero. This is goodput collapse. The cure is not more capacity. It is rejecting excess requests early and cheaply: load shedding.

This is not the same thing as a retry budget or per-user rate limiting. A rate limiter says "this customer exceeded their share." A load shedder says "I cannot take any more right now." One is about fairness, the other about survival, and you need both.

An unbounded queue turns overload into latency

Most server software, by default, accepts the connection and holds it until a worker thread or event loop is free. That waiting room is effectively an unbounded queue. Little's law makes the consequence concrete: the average number of items in a system equals arrival rate times average time in the system (L = λW). If a service can handle 500 requests per second and receives 600, the excess adds 100 items to the queue every second. After one minute there are 6000 queued items and a new request waits 12 seconds. If the client times out at 5 seconds, every bit of work done from that point on is wasted.

So the first rule: every queue gets an upper bound, and the bound is small enough that waiting never exceeds the caller's patience.

Three approaches

A fixed concurrency limit. Cap the number of requests in flight; when the cap is reached, new requests get a 503 immediately. It is simple, easy to reason about, and what you should reach for first. The hard part is the number. Choose it based on the actual bottleneck: if it is a database connection pool, a little above the pool size is a good start; if it is CPU, a small multiple of the core count. Find it with a load test: raise concurrency until p99 latency bends sharply, then pick a value just below that knee.

Shedding by time spent queued. Instead of counting items, bound how long a request may wait. When a request is picked up, check how long it has been waiting; if it exceeds the threshold, reject it without doing the work. This is the application level equivalent of the CoDel algorithm used in network gear, and it is more robust than a fixed count because it behaves correctly when request cost varies.

Adaptive limits. Algorithms that continuously tune the limit based on observed latency, borrowing from TCP congestion control (AIMD or gradient based methods). Latency rises, the limit shrinks; latency falls, it grows. Some proxies and client libraries ship this ready to use. It is powerful, but its behavior is hard to explain, and that is a real cost at 3 a.m. when you are trying to understand why requests are being rejected.

My recommendation: start with a fixed concurrency limit and add a queue time check on top. Move to adaptive limits only when request cost is highly variable and you have evidence that no fixed number works.

Implementation: a limiter that does not queue

A Go middleware that uses nothing beyond the standard library:

package main

import (
	"net/http"
	"time"
)

type Limiter struct {
	slots   chan struct{}
	maxWait time.Duration
}

func NewLimiter(n int, maxWait time.Duration) *Limiter {
	return &Limiter{slots: make(chan struct{}, n), maxWait: maxWait}
}

func (l *Limiter) Wrap(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		t := time.NewTimer(l.maxWait)
		defer t.Stop()
		select {
		case l.slots <- struct{}{}:
			defer func() { <-l.slots }()
			next.ServeHTTP(w, r)
		case <-t.C:
			w.Header().Set("Retry-After", "2")
			http.Error(w, "overloaded", http.StatusServiceUnavailable)
		case <-r.Context().Done():
		}
	})
}

func main() {
	api := http.NewServeMux()
	api.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(100 * time.Millisecond)
		w.Write([]byte("ok\n"))
	})

	root := http.NewServeMux()
	root.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("ok\n"))
	})
	root.Handle("/", NewLimiter(50, 20*time.Millisecond).Wrap(api))

	http.ListenAndServe(":8080", root)
}

Three decisions are encoded here. When all slots are taken, a request waits at most 20 milliseconds; that small window absorbs momentary bursts without building a long queue. If the client disconnects (r.Context().Done()), the work never starts. The health check endpoint sits outside the limiter; the pitfalls section explains why that matters.

Respect deadlines

The cheapest moment to reject is before any work starts. If the caller attaches a deadline, the server should look at the remaining time before starting and skip the work entirely if it cannot finish. gRPC does this at the protocol level: the client's deadline travels to the server and can be passed through the context to the server's own downstream calls. HTTP has no standard field for it; you define your own header and turn it into a context deadline. In Go, reading ctx.Deadline() at the top of a handler and rejecting when the remaining time is below the typical processing time takes a few lines.

Do this on every service to service hop, not just at the edge. A service in the middle of a call chain that does not know its caller has given up will keep loading its own database for nothing.

Prioritize

Not all requests are equal. If a payment confirmation and a background report export share the same pool, both go down together under overload. Define at least two classes and give them separate limits: interactive requests a user is waiting on, and batch work that can tolerate delay. When the batch limit fills, interactive traffic is untouched. A priority queue inside a single pool is finer grained, but separate pools are simpler and much easier to observe.

Pitfalls

Tying readiness to load. If the readiness check fails under overload, the load balancer pulls that instance from rotation. The remaining instances get more load, they fail too, and within minutes nothing is left. A failing liveness check is even worse: the instance is restarted and does no work at all while it warms up. Health endpoints belong outside the limiter and should answer "is the process alive and able to reach its dependencies," not "is it idle right now."

Expensive rejection. If the reject path reads the request body, authenticates, or touches the database, shedding buys you nothing. The decision must happen at the earliest possible point and cost close to nothing.

Clients retrying immediately. If a client that receives a 503 retries without waiting, the load you shed comes straight back, doubled. Send Retry-After, and enforce exponential backoff and a retry budget on the client side. Test how your own client libraries react to 503; many retry immediately by default.

Mixing up status codes. Return 429 when a caller exceeds its quota and 503 when the server is out of capacity. Split alerts the same way: a rise in 429s is usually one customer's behavior, a rise in 503s is your capacity problem.

Relying on autoscaling. New instances take minutes to come up, a spike arrives in seconds. Autoscaling does not replace shedding; shedding is what keeps the service alive until scaling catches up. The rejection rate is also a useful scaling signal.

Forgetting the limit is per instance. A concurrency limit applies to each instance. When the instance count changes, total capacity changes, but the database underneath does not. Ten instances at 50 concurrent requests each can send 500 concurrent queries to a single database. Size the limit against where the bottleneck really is.

Measure it

If shedding is invisible, you will not notice when it is misconfigured. At minimum, export the number of requests in flight, the number rejected, and the time requests spend queued. The rejection rate should be near zero on a normal day; if you see rejections under ordinary traffic, the limit is too tight.

When you do not need it

For a low traffic internal tool with predictable load, this complexity is not worth it. Asynchronous workers consuming from a queue do not need it either: the queue already provides backpressure, the worker pulls only what it can handle, and the excess waits in the broker. That is the right design, because no user is waiting on that work. Load shedding is for synchronous paths where a user or caller expects an answer within a bounded time. On those paths, rejecting is always better than answering late: a fast failure gives the client a chance to try something else, while a slow one sinks everyone together.