Secret Management in Kubernetes: Which Approach Fits Where

Kubernetes' built-in Secret object is not, despite its name, an encryption mechanism. Values are base64-encoded, nothing more. Without encryption-at-rest enabled on etcd, the data sits close to plaintext on disk; even with it enabled, anyone who can run get secrets under the right RBAC permissions can read the value. Plenty of teams have assumed a Secret object was "encrypted" and committed its contents straight to Git.

That's where the real question starts: where do you put the secret, who encrypts it, who decrypts it, and how many places do you have to touch when it rotates? There are four common approaches, and each rests on a different assumption.

The baseline first: etcd encryption-at-rest

Whichever approach you pick, put a baseline layer underneath it: enable encryption-at-rest on etcd itself. You define a KMS provider (your cloud provider's own KMS, or a local key) in an EncryptionConfiguration and pass it to the API server via --encryption-provider-config; after that, Secret objects written to etcd sit encrypted on disk. This doesn't replace any of the four approaches below, but it closes a vulnerability all of them share: if an etcd disk backup or snapshot is stolen, plaintext doesn't leak with it. Most teams skip this because it doesn't show up on a "secrets management tools" list, even though it's the cheapest and most fundamental safeguard available.

Sealed Secrets: encrypted secrets you can commit to Git

Bitnami's Sealed Secrets rests on a simple idea: you encrypt the secret with the cluster's public key and commit it to Git like any other YAML file. A controller running in the cluster decrypts it with its private key and turns it into a real Secret object.

kubectl create secret generic db-creds \
  --from-literal=password=change-me \
  --dry-run=client -o yaml > secret.yaml

kubeseal --format=yaml < secret.yaml > sealed-secret.yaml
git add sealed-secret.yaml

The upside is obvious: it fits a GitOps workflow with zero friction, there's no external system to depend on, and the encrypted file sits safely in your repo. The downside comes from that same simplicity. The private key lives only in that cluster; if you don't back it up, losing the cluster means losing every encrypted secret too, because the key to decrypt them is gone with it. Rotation is manual as well: changing a secret means running kubeseal again and pushing a new commit, there's no automatic trigger.

External Secrets Operator: the secret's real home is outside the cluster

External Secrets Operator (ESO) never puts the secret in Git at all. It pulls from an external source, Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager, and syncs on a regular interval to produce a Secret object inside the cluster.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-creds
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-creds
  data:
    - secretKey: password
      remoteRef:
        key: secret/db
        property: password

Its strength is that the single source of truth lives outside the cluster, in a central system. If you run multiple clusters, they all pull from the same source, rotation happens centrally, and it propagates to every cluster within the refreshInterval window. But note: ESO still ends up producing a plain Kubernetes Secret object. The etcd exposure problem isn't solved, only the secret's source and rotation get centralized. Leave refreshInterval too wide and rotation propagates slower than you'd expect; secrets injected as environment variables also don't update without a pod restart, only volume-mounted ones refresh live.

SOPS: encryption that keeps the diff readable

Mozilla's SOPS encrypts values inside a YAML or JSON file on a per-key basis; the file's structure and key names stay plaintext, only the values are encrypted. It works with age or PGP, has native support built into Flux, and Argo CD can use it through a plugin like KSOPS.

The difference here is diff readability: in a PR you can see which key changed, not its value, but you can see the shape of the change. For small teams this is the most practical way to bring secrets into a code review process. The trap sits in the path regexes inside .sops.yaml: get a file pattern wrong and that file gets committed unencrypted, and the mistake only surfaces when someone scans the Git history. Key rotation also requires re-encrypting every file that was encrypted with that key; a forgotten file stays locked with the old one.

Vault Agent Injector: the path where the secret never becomes a Secret object

This is the most different of the four. Vault's injector adds a sidecar container to the pod through a mutating webhook; that sidecar pulls the secret from Vault and writes it to the pod's filesystem, under /vault/secrets/. The secret never turns into a Kubernetes Secret object, it never touches etcd at all.

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "app"
  vault.hashicorp.com/agent-inject-secret-config: "secret/data/app/config"

What you get in return is the best audit trail of the four: every read lands in Vault's audit log, so who accessed which secret and when is recorded. There's a cost too: every pod gets a sidecar, pod startup time becomes dependent on Vault, and if the mutating webhook's namespace selector isn't scoped tightly, injection can be attempted on unrelated pods, which then crash-loop without Vault access. Vault itself is also a system you now run: it needs an unseal process, an HA configuration, and a backup strategy. You've added one more dependency to simplify secret management.

If you don't want the startup latency a sidecar adds, the Secrets Store CSI Driver offers a middle ground. It applies the same idea through a CSI volume instead of a separate sidecar container: the secret gets mounted into a tmpfs volume when the pod starts, with Vault, AWS, Azure, or GCP pluggable as the provider. You can optionally sync the same value into a native Kubernetes Secret object via the secretObjects field, but skip that step and the secret still never touches etcd. Without a sidecar, resource usage is lighter than the Vault Agent Injector, at the cost of the volume being a filesystem shared across the pod's containers, so make sure you're using a CSI ephemeral volume rather than a hostPath, to keep other pods on the same node from reaching it.

Which one, where

For a single cluster and a small team running a GitOps workflow, Sealed Secrets or SOPS is enough; standing up and operating an external system costs more than it buys you. Don't forget to back up the private key, or the age key you use with SOPS, that's the most common trap in both approaches.

If you run multiple clusters and already operate a secret manager, Vault or your cloud provider's own service, External Secrets Operator is the natural fit. Keeping the secret in one place and distributing it to several clusters is exactly what it's built for.

If you have an audit requirement, or the secret genuinely can't touch etcd (regulated environments like finance or healthcare), evaluate the Vault Agent Injector or the Secrets Store CSI Driver. But before you set it up, answer who's going to run Vault and who owns the unseal process; the system you stand up to simplify secret management shouldn't turn into a separate system that itself needs managing.

None of the four is "best" on its own. The three questions worth asking are: who triggers rotation, do your secrets disappear if you lose the cluster, and do you need a record of every read? The answers determine which of the four is right for you.