← blog · November 19, 2026

Capacity Planning for LLM Inference Servers: KV Cache, Continuous Batching, Time to First Token and the Invisible Queue

Fitting the weights in memory does not mean the service survives load. The KV cache decides concurrency, and the metrics that matter are time to first token and inter-token latency, not average response time. Arithmetic, settings, load testing and pitfalls.

When you decide to serve a language model on your own GPU, the first question is usually "how many users can this card handle", and the answer is usually sought in the wrong place. The most common mistake is assuming that because the model weights fit in memory, the service will also hold up under load. What limits an inference server is not the weights but the KV cache held for every concurrent request, and how the server batches those requests together. This post covers how to estimate capacity, which latency metrics actually matter, and how to keep the service healthy when load grows.

What fills the memory

While generating each new token, a transformer attends to the key and value vectors of every previous token. To avoid recomputing them at every step it keeps them in the KV cache. That cache is per request and grows linearly with context length.

The KV cache size per token is roughly:

2 x num_layers x num_kv_heads x head_dim x bytes_per_element

The leading 2 accounts for keys and values. Take a model with 32 layers, 8 KV heads (using grouped query attention) and a head dimension of 128, served at 16 bit precision: 2 x 32 x 8 x 128 x 2 bytes comes to 128 KiB per token. A single 8k token context holds 1 GiB. On a 24 GB card where 16 GB goes to weights and part of the rest goes to activations and working memory, the few gigabytes left can carry only a handful of full length requests at once.

Two things follow immediately. First, the variable that really decides concurrency is context length. Second, copying "the model supports 128k tokens" straight into your server config can reduce your capacity to a single request.

Static batching versus continuous batching

The older approach takes a fixed group of requests and accepts nothing new until all of them finish. Because output lengths vary widely, the short requests finish early and the GPU idles while waiting for the longest one.

Continuous batching fixes this by rebuilding the batch at every generation step: finished requests leave, waiting requests join immediately. Current servers such as vLLM, TGI and SGLang do this by default and split the KV cache into pages (PagedAttention, in vLLM's terms) to reduce fragmentation. Unless you are writing your own server, this is not a choice but a requirement. Wrapping a model in a plain HTTP handler that runs one request at a time gets several times less work out of the same hardware.

So the useful comparison is not "which server" but "which settings". The example below uses vLLM's command line options; other servers have equivalents for the same concepts.

vllm serve <model-name> \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64 \
  --enable-prefix-caching

--max-model-len sets the longest context the server will accept and is the basis of KV cache planning. Set it to what your application actually needs. --gpu-memory-utilization tells the server how much of the card to claim; lower it if another process shares the GPU. --max-num-seqs caps the number of sequences processed at once. --enable-prefix-caching lets requests that start with the same system prompt share the KV cache for that prefix.

The latency metrics that matter

In an LLM service, "average response time" tells you almost nothing. Response time scales with output length, and averaging a 20 token answer with a 2000 token answer is meaningless. Track two separate metrics instead:

  • Time to first token (TTFT): from the moment the request arrives until the first token is sent. Queueing and input processing (prefill) both land here. In a chat interface this is what decides whether the user feels the app is stuck.
  • Inter-token latency (ITL, also TPOT): the time between tokens once streaming has started. It sets the reading speed and grows as the batch gets bigger.

Capacity planning is the balance between these two. As you raise concurrency, total throughput in tokens goes up, but each request's inter-token latency also goes up. Past a certain point the KV cache fills, new requests start queueing and TTFT jumps sharply. Do not set your target as "this many tokens per second". Set it as "the highest request rate at which p95 TTFT and p95 inter-token latency stay under these limits".

Measuring capacity

The arithmetic gives you a starting point; a load test gives you the answer. For a useful test:

  1. Use an input and output length distribution that resembles real traffic. A test that always sends 100 tokens in and asks for 100 tokens out does not represent the user summarising a long document.
  2. Increase the request rate step by step and record p50 and p95 for both TTFT and inter-token latency at each step.
  3. Watch the server's own metrics at the same time. vLLM and TGI expose Prometheus metrics; the number of waiting requests, the number of running requests and KV cache usage are the three that matter most.
  4. Note the first rate at which TTFT breaks your target, and define capacity below that to leave headroom.

The vLLM repository ships a serving benchmark script for exactly this. If you prefer your own tooling, a general load tool such as k6 is enough as long as you measure the first byte of the streamed response and the total duration separately.

Pitfalls

The queue grows silently. Servers that use continuous batching do not reject requests, they queue them. When load exceeds capacity the error rate stays at zero while TTFT climbs into minutes, clients time out and retry, and load grows further. Alert on the number of waiting requests and enforce a concurrency limit at the gateway in front; returning a fast 429 is better than making a user wait two minutes.

Long inputs slow everyone down. The prefill of a large document delays token generation for every other request in the same batch. Chunked prefill (--enable-chunked-prefill in vLLM, enabled by default in most cases in recent versions) splits long inputs into pieces and reduces the effect, but does not remove it. Splitting very different workloads, short chat versus long document processing, onto separate instances is often the cleaner fix.

Preemption and recomputation. When the KV cache runs out, the server can evict some requests and recompute them later from scratch. This shows up as a warning in the logs and means the same work is done twice. If you see it often, lower --max-num-seqs or shorten the context limit.

Quantisation is not free. Taking weights down to 4 bits frees room for the KV cache, and storing the KV cache in 8 bits nearly doubles concurrency. Both can affect quality, and the effect depends on the task. Do not make that call without an evaluation set of your own.

Cold starts. Loading and warming up a large model can take minutes. Autoscaling on CPU usage is useless here, and the GPU always looks busy anyway. Use the number of waiting requests as the scaling signal and account for the long time a new instance needs before it is ready.

When not to run your own server

If your traffic is low and irregular, the cost of a GPU sitting idle most of the day easily exceeds the per token price of a hosted API. Your own inference server makes sense when data cannot leave your environment, when you have steady and predictable load, or when you need to serve a fine tuned model. If none of those apply, start with a hosted service and compare against the arithmetic above once your traffic pattern is clear.