DNS Resolution in Kubernetes: ndots, Search Domains, CoreDNS Caching and the Five-Second Stalls
Why a pod turns one external lookup into eight DNS queries, what lowering ndots breaks, when autopath and NodeLocal DNSCache are worth it, where the exact five-second stalls come from, and which fix belongs at the application, pod or CoreDNS level.
Nobody thinks about DNS when an application inside Kubernetes calls an external API or another service, until tail latency jumps by an unexplained 5 seconds or the CoreDNS pods buckle under load as the cluster grows. The root cause is rarely the application. It is the resolver configuration written into the pod. This post explains how a pod resolves a name, why the defaults stop scaling, and which fix belongs where.
How a pod resolves a name
With the default dnsPolicy: ClusterFirst, the kubelet writes a resolver file into every pod containing the cluster DNS service address, three search domains (<namespace>.svc.cluster.local, svc.cluster.local, cluster.local) and options ndots:5. The search domains are what let you reach a service in the same namespace as just database and one in another namespace as database.other-ns.
The ndots:5 rule says: if the queried name has fewer than five dots, treat it as relative, try it with each search domain in order, and only then ask for the absolute name. api.example.com has two dots, so the resolver first asks for api.example.com.<namespace>.svc.cluster.local, then with svc.cluster.local, then with cluster.local, and only after three NXDOMAIN answers does the real name go out. With A and AAAA requested for every attempt, a single call to an external host becomes eight DNS queries. An HTTP request that never touches the cluster starts with four round trips to CoreDNS.
dig does not apply search domains by default; the +search flag shows the behaviour your application actually sees. The query count and timing in that output are the concrete justification for everything that follows.
Three fixes, three different places
In the application: write the name as absolute. A trailing dot (api.example.com.) tells the resolver "this name is complete, do not append search domains" and removes every extra query. If you own the code or the configuration, this is the cheapest fix. The trap: some HTTP clients and TLS libraries carry the trailing dot into the Host header or SNI, and the server fails to match a virtual host. Test it with your library rather than assuming.
In the pod: lower ndots. If you cannot touch the source, change resolver options from the pod spec:
With ndots:2, api.example.com is queried as absolute immediately, while database and database.other-ns still go through the search list. The only thing it breaks is the three-part shorthand: a library that uses service.ns.svc now has two dots, is queried as absolute without search, and fails to resolve. Grep for that pattern across your workloads before rolling this out.
In the server: let CoreDNS walk the search path. The CoreDNS autopath plugin performs the search walk on the server instead of the client: it answers the first query with a CNAME pointing at the correct result and the client never makes the remaining attempts. The price is that the kubernetes plugin must run in pods verified mode, meaning CoreDNS watches every pod to know which IP belongs to which namespace. In a large cluster that watch noticeably raises memory use and API server load. It makes sense for third-party workloads where you can change neither the source nor the pod, not as a general default.
Cache and timeouts
The CoreDNS cache plugin stores both positive and negative answers; the NXDOMAINs produced by the ndots walk come back from the negative cache, which sharply reduces load for workloads that keep asking for the same external name. The default cache 30 caps positive answers at a 30-second TTL. A short TTL reflects in-cluster service changes quickly but raises query volume; keeping a longer cache for external names in a separate server block is a reasonable balance.
The first block of this Corefile answers only names under example.com with a long cache; the second is the default that kubeadm installs. The reload plugin picks up ConfigMap changes without a restart. The loop plugin detects CoreDNS forwarding to itself and stops the pod, so when you see CrashLoopBackOff, look at the node's own resolver file first.
Do not tune without measuring. coredns_dns_requests_total, coredns_dns_request_duration_seconds and coredns_cache_hits_total are labelled by query type and zone; the effect of an ndots change shows up in those three within minutes.
The five-second stalls
The classic symptom of DNS latency is a broken tail, not a broken average: most requests take milliseconds, some take exactly 5 seconds. That number is the default timeout of the glibc resolver and almost always points at a lost UDP packet. The best-known cause is the A and AAAA queries leaving the same socket at the same moment and racing to insert two conntrack entries during the DNAT to the cluster DNS service; the losing packet is silently dropped and the client waits out the timeout before retrying.
There are three remedies. On glibc-based images the single-request-reopen option sends the two queries from separate sockets and removes the race; add it via dnsConfig.options. musl-based images such as Alpine do not recognise the option and silently ignore it, so the problem persists on Alpine and many teams simply accept it as "DNS is sometimes slow". The durable fix is NodeLocal DNSCache: a DaemonSet on every node serves pod queries from a local cache, talks to the upstream over TCP, and never touches conntrack. Because the local hit rate is high, load on CoreDNS drops as well. The cost is one more component and a change to the kubelet's cluster DNS address; forgetting that DaemonSet during upgrades takes down DNS for the whole cluster.
Common traps
A pod with hostNetwork: true and a ClusterFirst policy inherits the node's resolver and cannot resolve cluster services; the correct value is ClusterFirstWithHostNet. dnsPolicy: None only works when dnsConfig supplies a complete nameservers list, otherwise the pod never starts.
If the cluster domain is anything other than cluster.local, every component with cluster.local hard-coded in source or Helm templates produces unresolvable names, and the symptom is usually connection refused rather than a DNS error. Before installing, check that the template reads the cluster domain from a value and that the value is not empty; an empty value produces a half name ending in svc. and warns nowhere.
Pushing the TTL towards zero for frequently queried workloads sends every request to CoreDNS and creates exactly the load you were trying to avoid. Instead of lowering the TTL, reduce the source of churn, for example headless service pods that are recreated far too often.
When to leave it alone
A small cluster, a single namespace and few external dependencies are fine on the defaults; ndots:5 is a genuine convenience and two CoreDNS replicas handle everything. The trouble starts as the cluster grows, external API calls multiply and the latency budget tightens. At that point the order is always the same: first see what is happening with dig +search, then fix it with the least intervention at the application or pod level, and only if the numbers are still bad move on to CoreDNS and the node-level cache.
[OK] 30 records · 9 featured · 1 retainer · 20 standard
/02 services
services.tree
> note
· Prices are in USD. Volume discounts of 7% for 6-12 day projects and 14% for 13+ day projects apply. Programs of 100+ person-days are priced individually. Prices are negotiable based on scope, urgency and long-term collaboration.
> ls ~/products/ --open-source
[OK] 2 entries · 1 live · 1 pre-release
/03 products
products.list
liveopen source · mit/prod/01
filex
the self-hosted file manager that embeds anywhere
single go binary · vue/react/web-component embed · 5 storage drivers · realtime collab · rbac · native multi-tenancy · mcp server for ai agents
We build infrastructure, automate everything, and keep systems running.
BRF Tech is a Bursa-based DevSecOps and software consulting firm. We operate across 30 service areas, from Kubernetes cluster management to serverless platform setup, CI/CD pipeline design to AI agent development.
What we do is simple: we set up your systems, automate them, and make sure they won't wake you up at 3 AM. We codify your infrastructure with Terraform, move your deployments to GitOps with ArgoCD, and monitor everything with Prometheus. If something breaks — we intervene before it does.
We're against vendor lock-in. We work with open-source tools, self-hosted solutions, and industry-standard technologies. We build your own serverless platform on Knative, isolate with Kata Containers, manage your secrets with Vault. Every project is delivered with clear scope, clear timeline, clear pricing.
DevSecOps & CI/CD
Kubernetes & Serverless
Infrastructure as Code
AI & Agent (MCP/ACP)
Security & Zero Trust
Full-Stack Development
Self-Hosted Solutions
> grep -i question ~/faq.md
[OK] 3 entries · click to expand
[?]What services are included in BRF Tech's DevOps solutions?▾
We offer CI/CD pipeline setup, Kubernetes cluster management, container migration, infrastructure automation with Terraform, monitoring & alerting, in-house technology installations and DevOps consulting services.
[?]What are your software development services?▾
We offer MVP development, existing software performance optimization, backend API development (Node.js, Go, Python) and full-stack application development. Every project is delivered with minimal technical debt.
[?]Which DevOps tools do you work with?▾
From Kubernetes and Docker to Talos Linux, from Terraform and OpenTofu to Ansible, and across Jenkins, GitLab CI/CD, GitHub Actions, ArgoCD, Flux, Helm and Rancher, we work with industry-standard tooling. For observability and error tracking we use Prometheus, Grafana, Loki, Tempo, OpenTelemetry and GlitchTip; for networking, Cilium and Envoy; for test automation, Playwright, Cypress, k6, Locust, Testcontainers and Trivy. We determine the most suitable toolset together, based on your project’s needs.
Automated build, test and deployment pipelines with GitHub Actions, GitLab CI/CD.
Automated build, test and deploy pipeline setup for existing or new repositories. GitHub Actions, GitLab CI/CD or preferred tool is used. Separate workflows are defined for staging + production environments.
Container architecture migration, cluster setup and orchestration.
Production-ready Kubernetes cluster setup on bare-metal or cloud. Includes Helm charts, Ingress, TLS certificate management, namespace isolation and RBAC.
Cozystack is an open-source cloud platform built on Kubernetes and the VMware alternative we recommend most. Virtual machines run on KubeVirt, while managed PostgreSQL, MySQL, Redis and Kafka services and dedicated tenant Kubernetes clusters are provisioned from the same dashboard and API. Includes installation on your own bare-metal servers on Talos Linux, replicated storage with LINSTOR/DRBD, network isolation with Kube-OVN, tenant and quota structure, monitoring, backup and a disaster recovery plan. After deployment we also handle upgrades, operations and support; since the platform is open source, there are no license fees and no vendor lock-in.
Packages
Virtual machine support on an existing Kubernetes cluster with KubeVirt
$2,800
4–6 person-day
Pilot setup (3 nodes, virtual machines + managed database)
$3,500
5–8 person-day
Production setup (HA + replicated storage + monitoring + backup)
$6,500
10–16 person-day
Multi-tenant private cloud (tenants, quotas, managed service catalog)
Planned, low-downtime migration of vSphere/ESXi environments to Cozystack, OpenStack or VHI.
A planned move to a VMware alternative for environments whose costs rose after Broadcom moved VMware licensing to subscription bundles, or whose renewal is coming up. We start with an inventory of your vSphere/ESXi estate (virtual machines, disks, networks, dependencies), then compare today's license cost with the total cost of the target platform. In most environments the target is Cozystack; OpenStack fits large-scale IaaS needs and Virtuozzo Hybrid Infrastructure fits storage-heavy service providers. A pilot runs on non-critical virtual machines, disks are converted with tools such as virt-v2v and Forklift, and the migration proceeds in waves within maintenance windows, with a rollback plan for every step.
Packages
Inventory + cost comparison + migration plan
$2,400
3–5 person-day
Pilot migration (up to 10 virtual machines)
$3,500
5–8 person-day
Phased migration (up to 100 virtual machines)
$9,750
15–22 person-day
Full data center migration (network + storage + backup)
OpenStack deployment, Neutron networking, Ceph/Cinder storage, upgrades and operations.
OpenStack deployment and operations in your own data center. A containerized, highly available control plane with Kolla-Ansible; Nova (compute), Keystone (identity), Glance (images) and Horizon (dashboard). On the network side, Neutron with OVN, VLAN/VXLAN, provider and tenant networks, floating IPs and the Octavia load balancer; on the storage side, Cinder block storage on Ceph RBD and optional S3-compatible object storage. For existing deployments: health checks, planned upgrades between releases, monitoring and alerting setup, and a backup strategy.
Packages
Health check + improvement report (existing deployment)
$2,400
3–5 person-day
Lab / PoC setup (single region, 3 nodes)
$3,500
5–8 person-day
Release upgrade (existing deployment)
$3,900
6–10 person-day
Production setup (HA control plane + Ceph + Neutron/OVN)
Hyperconverged infrastructure: multi-tenant cloud plus block and S3-compatible storage for service providers.
Hyperconverged infrastructure with Virtuozzo Hybrid Infrastructure, where compute, storage and networking run on the same servers. Block, S3-compatible object and NFS storage on software-defined storage, plus backup storage (Backup Gateway) for backup products such as Acronis. For service providers: a multi-tenant cloud with domains and projects, quotas, a self-service panel and an OpenStack-compatible API. Includes hardware sizing, network design, cluster deployment and expansion, version upgrades and operations support.
Packages
Pre-deployment assessment + hardware sizing
$2,400
3–4 person-day
Version upgrade / cluster expansion
$2,800
4–6 person-day
Hyperconverged cluster setup (storage + compute)
$3,900
6–10 person-day
Service provider setup (multi-tenant cloud + S3 + backup storage)
Migration of existing applications to container architecture.
Migration of existing applications to container architecture. Docker image design, Compose configuration and conversion to Kubernetes manifests. Kata Containers / gVisor can be included for advanced isolation.
Health check of your current environment, capacity planning, platform selection and a roadmap.
An independent assessment of your current virtualization environment (VMware vSphere, OpenStack, VHI or other KVM-based platforms). The health check covers cluster configuration, resource overcommit, storage latency, forgotten snapshots, backup and patch status. Capacity planning measures CPU, memory and storage growth to work out the hardware you will need next. Platform selection decides which workload should run where: Cozystack for teams running containers and virtual machines side by side, OpenStack for large-scale IaaS, VHI for storage-heavy service providers; staying on the current platform is also an option for some workloads. The deliverable is a written report with a cost comparison and a prioritized roadmap.
In-house tools like GitLab, Nextcloud, VPN, mail server.
Installation of tools like GitLab, Mattermost, Nextcloud, mail server, VPN, internal monitoring on company servers. Deployed on Docker Compose or Kubernetes.
DB + cache + code bottleneck identification and resolution.
Profiling of existing application, identification and resolution of bottlenecks. DB query optimization, caching layer (Redis), service-level improvements.
MCP/ACP agents, RAG pipelines, multi-agent orchestration, model serving.
LLM integration (OpenAI, Claude, Gemini, local models), tool-augmented AI agents with MCP (Model Context Protocol) and ACP (Agent Communication Protocol). RAG pipeline, multi-agent orchestration, model serving (vLLM/Ollama). AI layer for existing business processes.
Event-driven data pipeline with Apache Kafka, RabbitMQ, NATS. ETL/ELT orchestration with Apache Airflow, real-time data streaming, CDC (Change Data Capture). Data lake/warehouse design, schema registry, dead letter queue management.
Packages
Kafka/RabbitMQ setup + basic producer/consumer
$3,500
5–7 person-day
ETL pipeline (Airflow + source → warehouse)
$5,200
8–12 person-day
Full event-driven architecture (CDC + streaming + DLQ)
Hands-on technical training for your teams. Docker & container fundamentals, Kubernetes operations, Terraform IaC, CI/CD best practices, AI/LLM integration workshops. Practical exercises in live lab environments, content customized by skill level.
E2E test automation with Playwright, Cypress, load testing with k6/Gatling. Pre-deploy quality gate integrated into CI/CD pipeline, test coverage reporting, visual regression testing. Reduce test writing time by up to 60% with AI-assisted test generation.
Your Collected Personal Data, Collection Method and Legal Basis
Any information that identifies or makes you identifiable is considered "personal data." When you visit our website or use our services, contact information such as your name, surname, email address, phone number, as well as your IP address and browser cookie data may be collected through automatic or semi-automatic means. This data is processed based on the legal grounds specified in Article 5 of the Personal Data Protection Law No. 6698: "being directly related to the establishment or performance of a contract" and "being mandatory for the legitimate interests of the data controller, provided that it does not harm the fundamental rights and freedoms of the data subject."
Purpose of Processing Your Personal Data
Your collected personal data is processed for the purposes of providing and improving our services, fulfilling customer requests, meeting legal obligations, conducting information security processes and managing communication activities. Your data is processed in a limited and proportionate manner for the stated purposes; when the purpose ceases to exist, data is deleted, destroyed or anonymized.
To Whom and For What Purposes Collected Personal Data May Be Transferred
Your personal data may be transferred to public institutions and organizations as required by legal regulations, to our business partners and technical infrastructure providers for the purpose of delivering services, and to lawyers and consultants in legal disputes. Transfers are carried out in accordance with Articles 8 and 9 of the Law, with necessary technical and administrative measures in place.
Your Rights as a Data Subject
In accordance with Article 11 of Law No. 6698, you have the right to: learn whether your personal data is being processed; request information if it has been processed; learn the purpose of processing and whether it is used in accordance with its purpose; know the third parties to whom it is transferred domestically or abroad; request correction if it has been processed incompletely or incorrectly; request deletion or destruction within the framework of conditions set out in Article 7 of the Law; request that the operations carried out be notified to third parties to whom data has been transferred; object to any adverse result arising from the analysis of data exclusively through automated systems; and claim compensation for damages in case of unlawful processing. You may contact us through the communication channels on our website to exercise these rights.
Terms & Conditions
Terms & Conditions
1. Service Scope and Changes
BRF Tech conducts its software development, DevOps consulting, infrastructure setup and technical support services within the framework of these terms and conditions. The scope of service is determined separately for each project and finalized through mutual agreement. BRF Tech reserves the right to update the scope, pricing and technical details of its services, provided that prior notice is given.
2. Privacy Policy
Personal data collected within the scope of our services is processed in accordance with our Privacy Policy. For detailed information, please review our Privacy Policy page.
3. Service Usage and Responsibilities
Clients agree to use the provided services solely within legal and ethical boundaries. BRF Tech reserves the right to suspend or terminate services in the event of misuse, violation of third-party rights, or use for illegal activities. The client is responsible for the accuracy and currency of the information provided within the project scope.
4. Payment Terms
Payment terms are determined on a project basis and finalized through mutual agreement at the start of the project. Unless otherwise specified, invoices are payable within 15 days of project delivery. A monthly late payment interest of 2% may apply to overdue payments.
5. Cancellation and Refund Policy
Cancellation requests must be submitted in writing at least 7 days before the service start date. For projects already in progress, billing is based on the proportion of completed work. No refunds are issued for software and infrastructure components specifically developed within the project scope.