The Empty Map Trap When Talking to the Kubernetes API
When a Kubernetes map field is empty, whether JSON encodes it as an array or an object depends on your language, and getting it wrong silently rejects the whole object. Why it happens and how to guard against it.
When you send an object to the Kubernetes API server, fields like metadata.labels and metadata.annotations are typed as map[string]string in the schema. The trouble starts when these fields are empty. JSON lets you write "empty" two different ways: an empty array [] or an empty object {}. Both mean "nothing in here" in everyday speech, but the API server only accepts one of them. Send the other and the whole object is rejected. Kubernetes has no partial acceptance, either the entire request is valid or none of it is.
This is a mistake most people who write an operator, a controller, or a CI tool run into sooner or later, and one that tends to get patched over without anyone quite understanding why. The error message does not help either: a line like cannot unmarshal array into Go struct field ObjectMeta.metadata.annotations of type map[string]string points at Go's type system rather than at JSON's own ambiguity. The person reading it suspects the API server before suspecting their own code, because the message does not point at the actual source of the problem.
The sneaky part is timing. The code works flawlessly on any object carrying at least one label or annotation, and most tests get written against non-empty examples. The bug only shows up when the code meets a genuinely empty map, which usually happens in production, as a first-time edge case nobody wrote a test for.
How different languages carry this ambiguity
How big a risk this is depends on the language the client is written in.
In Go the risk is low, because the language keeps maps and slices as distinct types: map[string]string{} always encodes as {}, and []string{} always encodes as []. The one real trap is a nil map; a declared but unassigned var m map[string]string encodes as null. Most fields accept that, but some CRDs with strict structural schemas will reject it too.
In PHP the risk is high, because the language has no separate map type; both lists and associative arrays are the same array type. json_encode([]) always produces [], regardless of what the array was meant to represent. Encoding an empty map as an object requires an explicit (object) cast, or custom behavior through the JsonSerializable interface. This is the most common version of the bug in any Kubernetes client written in PHP.
In Python, native dict and list are distinct, so json.dumps({}) correctly produces {}. The risk here tends to come from deriving a map from a list structure, say a list of key-value pairs, and forgetting the final dict() conversion.
In JavaScript and TypeScript the distinction is native too, but because the type system is weak, a helper function that defaults an "empty" value to [] (the first thing many utility libraries reach for when they mean "empty collection") can leak into a map field completely unnoticed.
The common thread is this: the risk lives not in the language's type system but in whether the developer consciously tied "empty" to "map" at the point the value was produced. Languages that work through typed structs and schemas are safe. Dynamically typed languages, and hand-built JSON in general, are where this bites.
Actionable steps
Three layers of defense work together; none of them is sufficient on its own.
First, in the source code: if a map field can be empty, pin the value representing emptiness explicitly to the map type. In PHP that means $labels === [] ? (object) [] : $labels. In TypeScript it means a helper function whose return type is annotated as Record<string, string>. In Python it means a type hint (dict[str, str]) checked by a static analyzer such as mypy or pyright. This moves the "I forgot" failure mode from runtime to compile time or lint time.
Second, validation before the request ever reaches the server: kubectl apply --dry-run=server -f manifest.yaml sends the request to the real API server without persisting anything. If there is a schema error, you get the same 400 response inside your CI pipeline, without ever touching production. If the step that produces manifests has no cluster access at all, a standalone schema validator such as kubeconform catches the same class of error without needing a cluster. The two are complementary rather than redundant: --dry-run=server tells you whether the real API server will accept the object, while a standalone validator gives fast feedback with no cluster dependency at all.
Third, tests: if you are writing a client library or a controller, protect the empty-map case with its own unit test, and write that test against the ENCODED OUTPUT rather than the in-memory value. In PHP, [] == (object) [] evaluates to true, because the language's loose comparison treats them as equal, so an assertion like expect($x)->toEqual([]) will never catch the underlying encoding bug. The correct check actually serializes the value and compares the resulting string: json_encode($x) === '{}'. The same principle applies in every other language too. What you test should be the bytes that go out on the wire, not the in-memory representation.
A typical failure looks like this: a deployment script creates fifty objects in sequence. The first forty-nine pass without incident because each one carries at least one label. The fiftieth arrives with an empty map due to a small configuration difference, and the whole rollout stops there. The bug does not surface the day the code is written; it surfaces weeks later, the first time an object genuinely has nothing in that field, which makes it both hard to find and hard to explain in terms of "why did this work until now."
One more layer if you are writing your own CRD
If you are defining your own Custom Resource Definition, keeping the schema strict with type: object and, where possible, additionalProperties, protects you not just from bugs in your own client but from bugs in every third-party client that talks to that CRD. A CRD with a relaxed structural schema, one using x-kubernetes-preserve-unknown-fields: true, pushes this class of error past the API server and much further down, into your own controller's code, where the message points at your code even though the root cause is the caller's payload. Keeping the schema strict is how you catch the error at the earliest and most accurate layer.
When this much caution is unnecessary
If you are using an official client library, client-go for Go, the official kubernetes client for Python, fabric8 for Java, these libraries work through type-safe structs and handle serialization themselves. As long as you are not hand-assembling JSON or YAML, you will not run into this trap, because the struct definition already keeps the map field at the correct type.
The problem really shows up when a controller is written quickly as a script, with JSON produced by hand, or in a language with weak type checking. If that is your situation and adding an official client as a dependency feels disproportionate to the size of the job, writing a single helper function that normalizes empty map fields, and using it everywhere in the codebase, is enough to reduce the risk to essentially zero. What you want to avoid is trying to repeat that same check by hand at every call site. Eventually one of them will be forgotten, and the bug will surface only when an empty object is involved, at the least convenient possible moment.
[OK] 30 records · 9 featured · 1 retainer · 20 standard
/02 services
services.tree
> note
· Prices are in USD. Volume discounts of 7% for 6-12 day projects and 14% for 13+ day projects apply. Programs of 100+ person-days are priced individually. Prices are negotiable based on scope, urgency and long-term collaboration.
> ls ~/products/ --open-source
[OK] 2 entries · 1 live · 1 pre-release
/03 products
products.list
liveopen source · mit/prod/01
filex
the self-hosted file manager that embeds anywhere
single go binary · vue/react/web-component embed · 5 storage drivers · realtime collab · rbac · native multi-tenancy · mcp server for ai agents
We build infrastructure, automate everything, and keep systems running.
BRF Tech is a Bursa-based DevSecOps and software consulting firm. We operate across 30 service areas, from Kubernetes cluster management to serverless platform setup, CI/CD pipeline design to AI agent development.
What we do is simple: we set up your systems, automate them, and make sure they won't wake you up at 3 AM. We codify your infrastructure with Terraform, move your deployments to GitOps with ArgoCD, and monitor everything with Prometheus. If something breaks — we intervene before it does.
We're against vendor lock-in. We work with open-source tools, self-hosted solutions, and industry-standard technologies. We build your own serverless platform on Knative, isolate with Kata Containers, manage your secrets with Vault. Every project is delivered with clear scope, clear timeline, clear pricing.
DevSecOps & CI/CD
Kubernetes & Serverless
Infrastructure as Code
AI & Agent (MCP/ACP)
Security & Zero Trust
Full-Stack Development
Self-Hosted Solutions
> grep -i question ~/faq.md
[OK] 3 entries · click to expand
[?]What services are included in BRF Tech's DevOps solutions?▾
We offer CI/CD pipeline setup, Kubernetes cluster management, container migration, infrastructure automation with Terraform, monitoring & alerting, in-house technology installations and DevOps consulting services.
[?]What are your software development services?▾
We offer MVP development, existing software performance optimization, backend API development (Node.js, Go, Python) and full-stack application development. Every project is delivered with minimal technical debt.
[?]Which DevOps tools do you work with?▾
From Kubernetes and Docker to Talos Linux, from Terraform and OpenTofu to Ansible, and across Jenkins, GitLab CI/CD, GitHub Actions, ArgoCD, Flux, Helm and Rancher, we work with industry-standard tooling. For observability and error tracking we use Prometheus, Grafana, Loki, Tempo, OpenTelemetry and GlitchTip; for networking, Cilium and Envoy; for test automation, Playwright, Cypress, k6, Locust, Testcontainers and Trivy. We determine the most suitable toolset together, based on your project’s needs.
Automated build, test and deployment pipelines with GitHub Actions, GitLab CI/CD.
Automated build, test and deploy pipeline setup for existing or new repositories. GitHub Actions, GitLab CI/CD or preferred tool is used. Separate workflows are defined for staging + production environments.
Container architecture migration, cluster setup and orchestration.
Production-ready Kubernetes cluster setup on bare-metal or cloud. Includes Helm charts, Ingress, TLS certificate management, namespace isolation and RBAC.
Cozystack is an open-source cloud platform built on Kubernetes and the VMware alternative we recommend most. Virtual machines run on KubeVirt, while managed PostgreSQL, MySQL, Redis and Kafka services and dedicated tenant Kubernetes clusters are provisioned from the same dashboard and API. Includes installation on your own bare-metal servers on Talos Linux, replicated storage with LINSTOR/DRBD, network isolation with Kube-OVN, tenant and quota structure, monitoring, backup and a disaster recovery plan. After deployment we also handle upgrades, operations and support; since the platform is open source, there are no license fees and no vendor lock-in.
Packages
Virtual machine support on an existing Kubernetes cluster with KubeVirt
$2,800
4–6 person-day
Pilot setup (3 nodes, virtual machines + managed database)
$3,500
5–8 person-day
Production setup (HA + replicated storage + monitoring + backup)
$6,500
10–16 person-day
Multi-tenant private cloud (tenants, quotas, managed service catalog)
Planned, low-downtime migration of vSphere/ESXi environments to Cozystack, OpenStack or VHI.
A planned move to a VMware alternative for environments whose costs rose after Broadcom moved VMware licensing to subscription bundles, or whose renewal is coming up. We start with an inventory of your vSphere/ESXi estate (virtual machines, disks, networks, dependencies), then compare today's license cost with the total cost of the target platform. In most environments the target is Cozystack; OpenStack fits large-scale IaaS needs and Virtuozzo Hybrid Infrastructure fits storage-heavy service providers. A pilot runs on non-critical virtual machines, disks are converted with tools such as virt-v2v and Forklift, and the migration proceeds in waves within maintenance windows, with a rollback plan for every step.
Packages
Inventory + cost comparison + migration plan
$2,400
3–5 person-day
Pilot migration (up to 10 virtual machines)
$3,500
5–8 person-day
Phased migration (up to 100 virtual machines)
$9,750
15–22 person-day
Full data center migration (network + storage + backup)
OpenStack deployment, Neutron networking, Ceph/Cinder storage, upgrades and operations.
OpenStack deployment and operations in your own data center. A containerized, highly available control plane with Kolla-Ansible; Nova (compute), Keystone (identity), Glance (images) and Horizon (dashboard). On the network side, Neutron with OVN, VLAN/VXLAN, provider and tenant networks, floating IPs and the Octavia load balancer; on the storage side, Cinder block storage on Ceph RBD and optional S3-compatible object storage. For existing deployments: health checks, planned upgrades between releases, monitoring and alerting setup, and a backup strategy.
Packages
Health check + improvement report (existing deployment)
$2,400
3–5 person-day
Lab / PoC setup (single region, 3 nodes)
$3,500
5–8 person-day
Release upgrade (existing deployment)
$3,900
6–10 person-day
Production setup (HA control plane + Ceph + Neutron/OVN)
Hyperconverged infrastructure: multi-tenant cloud plus block and S3-compatible storage for service providers.
Hyperconverged infrastructure with Virtuozzo Hybrid Infrastructure, where compute, storage and networking run on the same servers. Block, S3-compatible object and NFS storage on software-defined storage, plus backup storage (Backup Gateway) for backup products such as Acronis. For service providers: a multi-tenant cloud with domains and projects, quotas, a self-service panel and an OpenStack-compatible API. Includes hardware sizing, network design, cluster deployment and expansion, version upgrades and operations support.
Packages
Pre-deployment assessment + hardware sizing
$2,400
3–4 person-day
Version upgrade / cluster expansion
$2,800
4–6 person-day
Hyperconverged cluster setup (storage + compute)
$3,900
6–10 person-day
Service provider setup (multi-tenant cloud + S3 + backup storage)
Migration of existing applications to container architecture.
Migration of existing applications to container architecture. Docker image design, Compose configuration and conversion to Kubernetes manifests. Kata Containers / gVisor can be included for advanced isolation.
Health check of your current environment, capacity planning, platform selection and a roadmap.
An independent assessment of your current virtualization environment (VMware vSphere, OpenStack, VHI or other KVM-based platforms). The health check covers cluster configuration, resource overcommit, storage latency, forgotten snapshots, backup and patch status. Capacity planning measures CPU, memory and storage growth to work out the hardware you will need next. Platform selection decides which workload should run where: Cozystack for teams running containers and virtual machines side by side, OpenStack for large-scale IaaS, VHI for storage-heavy service providers; staying on the current platform is also an option for some workloads. The deliverable is a written report with a cost comparison and a prioritized roadmap.
In-house tools like GitLab, Nextcloud, VPN, mail server.
Installation of tools like GitLab, Mattermost, Nextcloud, mail server, VPN, internal monitoring on company servers. Deployed on Docker Compose or Kubernetes.
DB + cache + code bottleneck identification and resolution.
Profiling of existing application, identification and resolution of bottlenecks. DB query optimization, caching layer (Redis), service-level improvements.
MCP/ACP agents, RAG pipelines, multi-agent orchestration, model serving.
LLM integration (OpenAI, Claude, Gemini, local models), tool-augmented AI agents with MCP (Model Context Protocol) and ACP (Agent Communication Protocol). RAG pipeline, multi-agent orchestration, model serving (vLLM/Ollama). AI layer for existing business processes.
Event-driven data pipeline with Apache Kafka, RabbitMQ, NATS. ETL/ELT orchestration with Apache Airflow, real-time data streaming, CDC (Change Data Capture). Data lake/warehouse design, schema registry, dead letter queue management.
Packages
Kafka/RabbitMQ setup + basic producer/consumer
$3,500
5–7 person-day
ETL pipeline (Airflow + source → warehouse)
$5,200
8–12 person-day
Full event-driven architecture (CDC + streaming + DLQ)
Hands-on technical training for your teams. Docker & container fundamentals, Kubernetes operations, Terraform IaC, CI/CD best practices, AI/LLM integration workshops. Practical exercises in live lab environments, content customized by skill level.
E2E test automation with Playwright, Cypress, load testing with k6/Gatling. Pre-deploy quality gate integrated into CI/CD pipeline, test coverage reporting, visual regression testing. Reduce test writing time by up to 60% with AI-assisted test generation.
Your Collected Personal Data, Collection Method and Legal Basis
Any information that identifies or makes you identifiable is considered "personal data." When you visit our website or use our services, contact information such as your name, surname, email address, phone number, as well as your IP address and browser cookie data may be collected through automatic or semi-automatic means. This data is processed based on the legal grounds specified in Article 5 of the Personal Data Protection Law No. 6698: "being directly related to the establishment or performance of a contract" and "being mandatory for the legitimate interests of the data controller, provided that it does not harm the fundamental rights and freedoms of the data subject."
Purpose of Processing Your Personal Data
Your collected personal data is processed for the purposes of providing and improving our services, fulfilling customer requests, meeting legal obligations, conducting information security processes and managing communication activities. Your data is processed in a limited and proportionate manner for the stated purposes; when the purpose ceases to exist, data is deleted, destroyed or anonymized.
To Whom and For What Purposes Collected Personal Data May Be Transferred
Your personal data may be transferred to public institutions and organizations as required by legal regulations, to our business partners and technical infrastructure providers for the purpose of delivering services, and to lawyers and consultants in legal disputes. Transfers are carried out in accordance with Articles 8 and 9 of the Law, with necessary technical and administrative measures in place.
Your Rights as a Data Subject
In accordance with Article 11 of Law No. 6698, you have the right to: learn whether your personal data is being processed; request information if it has been processed; learn the purpose of processing and whether it is used in accordance with its purpose; know the third parties to whom it is transferred domestically or abroad; request correction if it has been processed incompletely or incorrectly; request deletion or destruction within the framework of conditions set out in Article 7 of the Law; request that the operations carried out be notified to third parties to whom data has been transferred; object to any adverse result arising from the analysis of data exclusively through automated systems; and claim compensation for damages in case of unlawful processing. You may contact us through the communication channels on our website to exercise these rights.
Terms & Conditions
Terms & Conditions
1. Service Scope and Changes
BRF Tech conducts its software development, DevOps consulting, infrastructure setup and technical support services within the framework of these terms and conditions. The scope of service is determined separately for each project and finalized through mutual agreement. BRF Tech reserves the right to update the scope, pricing and technical details of its services, provided that prior notice is given.
2. Privacy Policy
Personal data collected within the scope of our services is processed in accordance with our Privacy Policy. For detailed information, please review our Privacy Policy page.
3. Service Usage and Responsibilities
Clients agree to use the provided services solely within legal and ethical boundaries. BRF Tech reserves the right to suspend or terminate services in the event of misuse, violation of third-party rights, or use for illegal activities. The client is responsible for the accuracy and currency of the information provided within the project scope.
4. Payment Terms
Payment terms are determined on a project basis and finalized through mutual agreement at the start of the project. Unless otherwise specified, invoices are payable within 15 days of project delivery. A monthly late payment interest of 2% may apply to overdue payments.
5. Cancellation and Refund Policy
Cancellation requests must be submitted in writing at least 7 days before the service start date. For projects already in progress, billing is based on the proportion of completed work. No refunds are issued for software and infrastructure components specifically developed within the project scope.