The Empty Map Trap When Talking to the Kubernetes API

Where the problem comes from

When you send an object to the Kubernetes API server, fields like metadata.labels and metadata.annotations are typed as map[string]string in the schema. The trouble starts when these fields are empty. JSON lets you write "empty" two different ways: an empty array [] or an empty object {}. Both mean "nothing in here" in everyday speech, but the API server only accepts one of them. Send the other and the whole object is rejected. Kubernetes has no partial acceptance, either the entire request is valid or none of it is.

// rejected: empty array
{ "metadata": { "annotations": [] } }

// accepted: empty object
{ "metadata": { "annotations": {} } }

This is a mistake most people who write an operator, a controller, or a CI tool run into sooner or later, and one that tends to get patched over without anyone quite understanding why. The error message does not help either: a line like cannot unmarshal array into Go struct field ObjectMeta.metadata.annotations of type map[string]string points at Go's type system rather than at JSON's own ambiguity. The person reading it suspects the API server before suspecting their own code, because the message does not point at the actual source of the problem.

The sneaky part is timing. The code works flawlessly on any object carrying at least one label or annotation, and most tests get written against non-empty examples. The bug only shows up when the code meets a genuinely empty map, which usually happens in production, as a first-time edge case nobody wrote a test for.

How different languages carry this ambiguity

How big a risk this is depends on the language the client is written in.

In Go the risk is low, because the language keeps maps and slices as distinct types: map[string]string{} always encodes as {}, and []string{} always encodes as []. The one real trap is a nil map; a declared but unassigned var m map[string]string encodes as null. Most fields accept that, but some CRDs with strict structural schemas will reject it too.

In PHP the risk is high, because the language has no separate map type; both lists and associative arrays are the same array type. json_encode([]) always produces [], regardless of what the array was meant to represent. Encoding an empty map as an object requires an explicit (object) cast, or custom behavior through the JsonSerializable interface. This is the most common version of the bug in any Kubernetes client written in PHP.

In Python, native dict and list are distinct, so json.dumps({}) correctly produces {}. The risk here tends to come from deriving a map from a list structure, say a list of key-value pairs, and forgetting the final dict() conversion.

In JavaScript and TypeScript the distinction is native too, but because the type system is weak, a helper function that defaults an "empty" value to [] (the first thing many utility libraries reach for when they mean "empty collection") can leak into a map field completely unnoticed.

The common thread is this: the risk lives not in the language's type system but in whether the developer consciously tied "empty" to "map" at the point the value was produced. Languages that work through typed structs and schemas are safe. Dynamically typed languages, and hand-built JSON in general, are where this bites.

Actionable steps

Three layers of defense work together; none of them is sufficient on its own.

First, in the source code: if a map field can be empty, pin the value representing emptiness explicitly to the map type. In PHP that means $labels === [] ? (object) [] : $labels. In TypeScript it means a helper function whose return type is annotated as Record<string, string>. In Python it means a type hint (dict[str, str]) checked by a static analyzer such as mypy or pyright. This moves the "I forgot" failure mode from runtime to compile time or lint time.

Second, validation before the request ever reaches the server: kubectl apply --dry-run=server -f manifest.yaml sends the request to the real API server without persisting anything. If there is a schema error, you get the same 400 response inside your CI pipeline, without ever touching production. If the step that produces manifests has no cluster access at all, a standalone schema validator such as kubeconform catches the same class of error without needing a cluster. The two are complementary rather than redundant: --dry-run=server tells you whether the real API server will accept the object, while a standalone validator gives fast feedback with no cluster dependency at all.

Third, tests: if you are writing a client library or a controller, protect the empty-map case with its own unit test, and write that test against the ENCODED OUTPUT rather than the in-memory value. In PHP, [] == (object) [] evaluates to true, because the language's loose comparison treats them as equal, so an assertion like expect($x)->toEqual([]) will never catch the underlying encoding bug. The correct check actually serializes the value and compares the resulting string: json_encode($x) === '{}'. The same principle applies in every other language too. What you test should be the bytes that go out on the wire, not the in-memory representation.

A typical failure looks like this: a deployment script creates fifty objects in sequence. The first forty-nine pass without incident because each one carries at least one label. The fiftieth arrives with an empty map due to a small configuration difference, and the whole rollout stops there. The bug does not surface the day the code is written; it surfaces weeks later, the first time an object genuinely has nothing in that field, which makes it both hard to find and hard to explain in terms of "why did this work until now."

One more layer if you are writing your own CRD

If you are defining your own Custom Resource Definition, keeping the schema strict with type: object and, where possible, additionalProperties, protects you not just from bugs in your own client but from bugs in every third-party client that talks to that CRD. A CRD with a relaxed structural schema, one using x-kubernetes-preserve-unknown-fields: true, pushes this class of error past the API server and much further down, into your own controller's code, where the message points at your code even though the root cause is the caller's payload. Keeping the schema strict is how you catch the error at the earliest and most accurate layer.

When this much caution is unnecessary

If you are using an official client library, client-go for Go, the official kubernetes client for Python, fabric8 for Java, these libraries work through type-safe structs and handle serialization themselves. As long as you are not hand-assembling JSON or YAML, you will not run into this trap, because the struct definition already keeps the map field at the correct type.

The problem really shows up when a controller is written quickly as a script, with JSON produced by hand, or in a language with weak type checking. If that is your situation and adding an official client as a dependency feels disproportionate to the size of the job, writing a single helper function that normalizes empty map fields, and using it everywhere in the codebase, is enough to reduce the risk to essentially zero. What you want to avoid is trying to repeat that same check by hand at every call site. Eventually one of them will be forgotten, and the bug will surface only when an empty object is involved, at the least convenient possible moment.