Capacity Planning for LLM Inference Servers: KV Cache, Continuous Batching, Time to First Token and the Invisible Queue
Fitting the weights in memory does not mean the service survives load. The KV cache decides concurrency, and the metrics that matter are time to first token and inter-token latency, not average response time. Arithmetic, settings, load testing and pitfalls.
When you decide to serve a language model on your own GPU, the first question is usually "how many users can this card handle", and the answer is usually sought in the wrong place. The most common mistake is assuming that because the model weights fit in memory, the service will also hold up under load. What limits an inference server is not the weights but the KV cache held for every concurrent request, and how the server batches those requests together. This post covers how to estimate capacity, which latency metrics actually matter, and how to keep the service healthy when load grows.
What fills the memory
While generating each new token, a transformer attends to the key and value vectors of every previous token. To avoid recomputing them at every step it keeps them in the KV cache. That cache is per request and grows linearly with context length.
The KV cache size per token is roughly:
2 x num_layers x num_kv_heads x head_dim x bytes_per_element
The leading 2 accounts for keys and values. Take a model with 32 layers, 8 KV heads (using grouped query attention) and a head dimension of 128, served at 16 bit precision: 2 x 32 x 8 x 128 x 2 bytes comes to 128 KiB per token. A single 8k token context holds 1 GiB. On a 24 GB card where 16 GB goes to weights and part of the rest goes to activations and working memory, the few gigabytes left can carry only a handful of full length requests at once.
Two things follow immediately. First, the variable that really decides concurrency is context length. Second, copying "the model supports 128k tokens" straight into your server config can reduce your capacity to a single request.
Static batching versus continuous batching
The older approach takes a fixed group of requests and accepts nothing new until all of them finish. Because output lengths vary widely, the short requests finish early and the GPU idles while waiting for the longest one.
Continuous batching fixes this by rebuilding the batch at every generation step: finished requests leave, waiting requests join immediately. Current servers such as vLLM, TGI and SGLang do this by default and split the KV cache into pages (PagedAttention, in vLLM's terms) to reduce fragmentation. Unless you are writing your own server, this is not a choice but a requirement. Wrapping a model in a plain HTTP handler that runs one request at a time gets several times less work out of the same hardware.
So the useful comparison is not "which server" but "which settings". The example below uses vLLM's command line options; other servers have equivalents for the same concepts.
--max-model-len sets the longest context the server will accept and is the basis of KV cache planning. Set it to what your application actually needs. --gpu-memory-utilization tells the server how much of the card to claim; lower it if another process shares the GPU. --max-num-seqs caps the number of sequences processed at once. --enable-prefix-caching lets requests that start with the same system prompt share the KV cache for that prefix.
The latency metrics that matter
In an LLM service, "average response time" tells you almost nothing. Response time scales with output length, and averaging a 20 token answer with a 2000 token answer is meaningless. Track two separate metrics instead:
Time to first token (TTFT): from the moment the request arrives until the first token is sent. Queueing and input processing (prefill) both land here. In a chat interface this is what decides whether the user feels the app is stuck.
Inter-token latency (ITL, also TPOT): the time between tokens once streaming has started. It sets the reading speed and grows as the batch gets bigger.
Capacity planning is the balance between these two. As you raise concurrency, total throughput in tokens goes up, but each request's inter-token latency also goes up. Past a certain point the KV cache fills, new requests start queueing and TTFT jumps sharply. Do not set your target as "this many tokens per second". Set it as "the highest request rate at which p95 TTFT and p95 inter-token latency stay under these limits".
Measuring capacity
The arithmetic gives you a starting point; a load test gives you the answer. For a useful test:
Use an input and output length distribution that resembles real traffic. A test that always sends 100 tokens in and asks for 100 tokens out does not represent the user summarising a long document.
Increase the request rate step by step and record p50 and p95 for both TTFT and inter-token latency at each step.
Watch the server's own metrics at the same time. vLLM and TGI expose Prometheus metrics; the number of waiting requests, the number of running requests and KV cache usage are the three that matter most.
Note the first rate at which TTFT breaks your target, and define capacity below that to leave headroom.
The vLLM repository ships a serving benchmark script for exactly this. If you prefer your own tooling, a general load tool such as k6 is enough as long as you measure the first byte of the streamed response and the total duration separately.
Pitfalls
The queue grows silently. Servers that use continuous batching do not reject requests, they queue them. When load exceeds capacity the error rate stays at zero while TTFT climbs into minutes, clients time out and retry, and load grows further. Alert on the number of waiting requests and enforce a concurrency limit at the gateway in front; returning a fast 429 is better than making a user wait two minutes.
Long inputs slow everyone down. The prefill of a large document delays token generation for every other request in the same batch. Chunked prefill (--enable-chunked-prefill in vLLM, enabled by default in most cases in recent versions) splits long inputs into pieces and reduces the effect, but does not remove it. Splitting very different workloads, short chat versus long document processing, onto separate instances is often the cleaner fix.
Preemption and recomputation. When the KV cache runs out, the server can evict some requests and recompute them later from scratch. This shows up as a warning in the logs and means the same work is done twice. If you see it often, lower --max-num-seqs or shorten the context limit.
Quantisation is not free. Taking weights down to 4 bits frees room for the KV cache, and storing the KV cache in 8 bits nearly doubles concurrency. Both can affect quality, and the effect depends on the task. Do not make that call without an evaluation set of your own.
Cold starts. Loading and warming up a large model can take minutes. Autoscaling on CPU usage is useless here, and the GPU always looks busy anyway. Use the number of waiting requests as the scaling signal and account for the long time a new instance needs before it is ready.
When not to run your own server
If your traffic is low and irregular, the cost of a GPU sitting idle most of the day easily exceeds the per token price of a hosted API. Your own inference server makes sense when data cannot leave your environment, when you have steady and predictable load, or when you need to serve a fine tuned model. If none of those apply, start with a hosted service and compare against the arithmetic above once your traffic pattern is clear.
[OK] 30 records · 9 featured · 1 retainer · 20 standard
/02 services
services.tree
> note
· Prices are in USD. Volume discounts of 7% for 6-12 day projects and 14% for 13+ day projects apply. Programs of 100+ person-days are priced individually. Prices are negotiable based on scope, urgency and long-term collaboration.
> ls ~/products/ --open-source
[OK] 2 entries · 1 live · 1 pre-release
/03 products
products.list
liveopen source · mit/prod/01
filex
the self-hosted file manager that embeds anywhere
single go binary · vue/react/web-component embed · 5 storage drivers · realtime collab · rbac · native multi-tenancy · mcp server for ai agents
We build infrastructure, automate everything, and keep systems running.
BRF Tech is a Bursa-based DevSecOps and software consulting firm. We operate across 30 service areas, from Kubernetes cluster management to serverless platform setup, CI/CD pipeline design to AI agent development.
What we do is simple: we set up your systems, automate them, and make sure they won't wake you up at 3 AM. We codify your infrastructure with Terraform, move your deployments to GitOps with ArgoCD, and monitor everything with Prometheus. If something breaks — we intervene before it does.
We're against vendor lock-in. We work with open-source tools, self-hosted solutions, and industry-standard technologies. We build your own serverless platform on Knative, isolate with Kata Containers, manage your secrets with Vault. Every project is delivered with clear scope, clear timeline, clear pricing.
DevSecOps & CI/CD
Kubernetes & Serverless
Infrastructure as Code
AI & Agent (MCP/ACP)
Security & Zero Trust
Full-Stack Development
Self-Hosted Solutions
> grep -i question ~/faq.md
[OK] 3 entries · click to expand
[?]What services are included in BRF Tech's DevOps solutions?▾
We offer CI/CD pipeline setup, Kubernetes cluster management, container migration, infrastructure automation with Terraform, monitoring & alerting, in-house technology installations and DevOps consulting services.
[?]What are your software development services?▾
We offer MVP development, existing software performance optimization, backend API development (Node.js, Go, Python) and full-stack application development. Every project is delivered with minimal technical debt.
[?]Which DevOps tools do you work with?▾
From Kubernetes and Docker to Talos Linux, from Terraform and OpenTofu to Ansible, and across Jenkins, GitLab CI/CD, GitHub Actions, ArgoCD, Flux, Helm and Rancher, we work with industry-standard tooling. For observability and error tracking we use Prometheus, Grafana, Loki, Tempo, OpenTelemetry and GlitchTip; for networking, Cilium and Envoy; for test automation, Playwright, Cypress, k6, Locust, Testcontainers and Trivy. We determine the most suitable toolset together, based on your project’s needs.
Automated build, test and deployment pipelines with GitHub Actions, GitLab CI/CD.
Automated build, test and deploy pipeline setup for existing or new repositories. GitHub Actions, GitLab CI/CD or preferred tool is used. Separate workflows are defined for staging + production environments.
Container architecture migration, cluster setup and orchestration.
Production-ready Kubernetes cluster setup on bare-metal or cloud. Includes Helm charts, Ingress, TLS certificate management, namespace isolation and RBAC.
Cozystack is an open-source cloud platform built on Kubernetes and the VMware alternative we recommend most. Virtual machines run on KubeVirt, while managed PostgreSQL, MySQL, Redis and Kafka services and dedicated tenant Kubernetes clusters are provisioned from the same dashboard and API. Includes installation on your own bare-metal servers on Talos Linux, replicated storage with LINSTOR/DRBD, network isolation with Kube-OVN, tenant and quota structure, monitoring, backup and a disaster recovery plan. After deployment we also handle upgrades, operations and support; since the platform is open source, there are no license fees and no vendor lock-in.
Packages
Virtual machine support on an existing Kubernetes cluster with KubeVirt
$2,800
4–6 person-day
Pilot setup (3 nodes, virtual machines + managed database)
$3,500
5–8 person-day
Production setup (HA + replicated storage + monitoring + backup)
$6,500
10–16 person-day
Multi-tenant private cloud (tenants, quotas, managed service catalog)
Planned, low-downtime migration of vSphere/ESXi environments to Cozystack, OpenStack or VHI.
A planned move to a VMware alternative for environments whose costs rose after Broadcom moved VMware licensing to subscription bundles, or whose renewal is coming up. We start with an inventory of your vSphere/ESXi estate (virtual machines, disks, networks, dependencies), then compare today's license cost with the total cost of the target platform. In most environments the target is Cozystack; OpenStack fits large-scale IaaS needs and Virtuozzo Hybrid Infrastructure fits storage-heavy service providers. A pilot runs on non-critical virtual machines, disks are converted with tools such as virt-v2v and Forklift, and the migration proceeds in waves within maintenance windows, with a rollback plan for every step.
Packages
Inventory + cost comparison + migration plan
$2,400
3–5 person-day
Pilot migration (up to 10 virtual machines)
$3,500
5–8 person-day
Phased migration (up to 100 virtual machines)
$9,750
15–22 person-day
Full data center migration (network + storage + backup)
OpenStack deployment, Neutron networking, Ceph/Cinder storage, upgrades and operations.
OpenStack deployment and operations in your own data center. A containerized, highly available control plane with Kolla-Ansible; Nova (compute), Keystone (identity), Glance (images) and Horizon (dashboard). On the network side, Neutron with OVN, VLAN/VXLAN, provider and tenant networks, floating IPs and the Octavia load balancer; on the storage side, Cinder block storage on Ceph RBD and optional S3-compatible object storage. For existing deployments: health checks, planned upgrades between releases, monitoring and alerting setup, and a backup strategy.
Packages
Health check + improvement report (existing deployment)
$2,400
3–5 person-day
Lab / PoC setup (single region, 3 nodes)
$3,500
5–8 person-day
Release upgrade (existing deployment)
$3,900
6–10 person-day
Production setup (HA control plane + Ceph + Neutron/OVN)
Hyperconverged infrastructure: multi-tenant cloud plus block and S3-compatible storage for service providers.
Hyperconverged infrastructure with Virtuozzo Hybrid Infrastructure, where compute, storage and networking run on the same servers. Block, S3-compatible object and NFS storage on software-defined storage, plus backup storage (Backup Gateway) for backup products such as Acronis. For service providers: a multi-tenant cloud with domains and projects, quotas, a self-service panel and an OpenStack-compatible API. Includes hardware sizing, network design, cluster deployment and expansion, version upgrades and operations support.
Packages
Pre-deployment assessment + hardware sizing
$2,400
3–4 person-day
Version upgrade / cluster expansion
$2,800
4–6 person-day
Hyperconverged cluster setup (storage + compute)
$3,900
6–10 person-day
Service provider setup (multi-tenant cloud + S3 + backup storage)
Migration of existing applications to container architecture.
Migration of existing applications to container architecture. Docker image design, Compose configuration and conversion to Kubernetes manifests. Kata Containers / gVisor can be included for advanced isolation.
Health check of your current environment, capacity planning, platform selection and a roadmap.
An independent assessment of your current virtualization environment (VMware vSphere, OpenStack, VHI or other KVM-based platforms). The health check covers cluster configuration, resource overcommit, storage latency, forgotten snapshots, backup and patch status. Capacity planning measures CPU, memory and storage growth to work out the hardware you will need next. Platform selection decides which workload should run where: Cozystack for teams running containers and virtual machines side by side, OpenStack for large-scale IaaS, VHI for storage-heavy service providers; staying on the current platform is also an option for some workloads. The deliverable is a written report with a cost comparison and a prioritized roadmap.
In-house tools like GitLab, Nextcloud, VPN, mail server.
Installation of tools like GitLab, Mattermost, Nextcloud, mail server, VPN, internal monitoring on company servers. Deployed on Docker Compose or Kubernetes.
DB + cache + code bottleneck identification and resolution.
Profiling of existing application, identification and resolution of bottlenecks. DB query optimization, caching layer (Redis), service-level improvements.
MCP/ACP agents, RAG pipelines, multi-agent orchestration, model serving.
LLM integration (OpenAI, Claude, Gemini, local models), tool-augmented AI agents with MCP (Model Context Protocol) and ACP (Agent Communication Protocol). RAG pipeline, multi-agent orchestration, model serving (vLLM/Ollama). AI layer for existing business processes.
Event-driven data pipeline with Apache Kafka, RabbitMQ, NATS. ETL/ELT orchestration with Apache Airflow, real-time data streaming, CDC (Change Data Capture). Data lake/warehouse design, schema registry, dead letter queue management.
Packages
Kafka/RabbitMQ setup + basic producer/consumer
$3,500
5–7 person-day
ETL pipeline (Airflow + source → warehouse)
$5,200
8–12 person-day
Full event-driven architecture (CDC + streaming + DLQ)
Hands-on technical training for your teams. Docker & container fundamentals, Kubernetes operations, Terraform IaC, CI/CD best practices, AI/LLM integration workshops. Practical exercises in live lab environments, content customized by skill level.
E2E test automation with Playwright, Cypress, load testing with k6/Gatling. Pre-deploy quality gate integrated into CI/CD pipeline, test coverage reporting, visual regression testing. Reduce test writing time by up to 60% with AI-assisted test generation.
Your Collected Personal Data, Collection Method and Legal Basis
Any information that identifies or makes you identifiable is considered "personal data." When you visit our website or use our services, contact information such as your name, surname, email address, phone number, as well as your IP address and browser cookie data may be collected through automatic or semi-automatic means. This data is processed based on the legal grounds specified in Article 5 of the Personal Data Protection Law No. 6698: "being directly related to the establishment or performance of a contract" and "being mandatory for the legitimate interests of the data controller, provided that it does not harm the fundamental rights and freedoms of the data subject."
Purpose of Processing Your Personal Data
Your collected personal data is processed for the purposes of providing and improving our services, fulfilling customer requests, meeting legal obligations, conducting information security processes and managing communication activities. Your data is processed in a limited and proportionate manner for the stated purposes; when the purpose ceases to exist, data is deleted, destroyed or anonymized.
To Whom and For What Purposes Collected Personal Data May Be Transferred
Your personal data may be transferred to public institutions and organizations as required by legal regulations, to our business partners and technical infrastructure providers for the purpose of delivering services, and to lawyers and consultants in legal disputes. Transfers are carried out in accordance with Articles 8 and 9 of the Law, with necessary technical and administrative measures in place.
Your Rights as a Data Subject
In accordance with Article 11 of Law No. 6698, you have the right to: learn whether your personal data is being processed; request information if it has been processed; learn the purpose of processing and whether it is used in accordance with its purpose; know the third parties to whom it is transferred domestically or abroad; request correction if it has been processed incompletely or incorrectly; request deletion or destruction within the framework of conditions set out in Article 7 of the Law; request that the operations carried out be notified to third parties to whom data has been transferred; object to any adverse result arising from the analysis of data exclusively through automated systems; and claim compensation for damages in case of unlawful processing. You may contact us through the communication channels on our website to exercise these rights.
Terms & Conditions
Terms & Conditions
1. Service Scope and Changes
BRF Tech conducts its software development, DevOps consulting, infrastructure setup and technical support services within the framework of these terms and conditions. The scope of service is determined separately for each project and finalized through mutual agreement. BRF Tech reserves the right to update the scope, pricing and technical details of its services, provided that prior notice is given.
2. Privacy Policy
Personal data collected within the scope of our services is processed in accordance with our Privacy Policy. For detailed information, please review our Privacy Policy page.
3. Service Usage and Responsibilities
Clients agree to use the provided services solely within legal and ethical boundaries. BRF Tech reserves the right to suspend or terminate services in the event of misuse, violation of third-party rights, or use for illegal activities. The client is responsible for the accuracy and currency of the information provided within the project scope.
4. Payment Terms
Payment terms are determined on a project basis and finalized through mutual agreement at the start of the project. Unless otherwise specified, invoices are payable within 15 days of project delivery. A monthly late payment interest of 2% may apply to overdue payments.
5. Cancellation and Refund Policy
Cancellation requests must be submitted in writing at least 7 days before the service start date. For projects already in progress, billing is based on the proportion of completed work. No refunds are issued for software and infrastructure components specifically developed within the project scope.