Feature Flags: Separating Deploy from Release, Consistent Percentage Rollouts and Flag Debt

If the only way to turn a feature off is to ship a new release, every deploy is also a product decision. Merging unfinished work to the main branch becomes scary, long-lived branches pile up, merges get painful, and rolling back is exactly as slow as your pipeline. Feature flags separate the two: code goes to production switched off, and who sees it is decided at runtime. The idea is simple. The hard parts are where flags get evaluated, how users are split, and how flags get removed once their job is done.

Not every flag is the same thing

Treating all flags as one concept is the first mistake. There are four kinds, and their lifetimes and risks differ.

A release flag rolls out a new or unfinished feature gradually. It should live for days or a few weeks and be deleted together with the old code path once the feature is fully on.

An experiment flag compares two variants. It lives as long as the experiment, and a user's variant must not change while it runs.

An operational switch (kill switch) turns off an expensive or fragile path: disabling a recommendation engine, cutting calls to a flaky third party. These are long-lived and get used at the worst possible moment, when the system is already struggling.

A permission flag encodes rules like "this plan gets this feature". Our recommendation is to keep these out of the flag system entirely. Plan and role rules belong to your authorization model, must agree with billing, and need to be auditable. An access rule that anyone can flip with one click in a flag dashboard is a security hole nobody notices.

Record the kind in the flag's name or metadata. Cleanup rules depend on the kind, and when nobody knows the kind, nothing ever gets deleted.

Where the flag is evaluated

There are three common approaches.

Config file or environment variable. Zero setup, but changing a value needs a restart or a redeploy. That throws away the main promise of flags, toggling independently of deploys. It is a poor fit for operational switches: in an incident you do not want to wait for the pipeline.

Asking a central service on every request. Values are always fresh, but every request gains a network call and the flag service sits on your critical path. When it slows down, the whole product slows down.

Local evaluation. The application pulls or streams the rule set (which flags, which targeting, which percentages) in the background and decides in memory. If the flag service goes down, it keeps working with the last known rules. This is the right default, and it is how most server-side SDKs of open source and commercial flag products work. If you want a vendor-neutral interface, the CNCF project OpenFeature is designed to separate the evaluation API from the provider behind it.

Browsers and mobile apps are different. Do not ship the rule set to the client: targeting rules contain customer lists, unannounced feature names and campaign dates. Evaluate on the server and send only the results.

Consistent percentage rollouts

"Turn it on for five percent of users" tempts people into random() < 0.05, which is wrong. The same user gets a different answer on every request, and the screen flips between old and new on each reload. Derive the decision from a stable key instead:

import hashlib

def bucket(flag: str, key: str) -> int:
    digest = hashlib.sha256(f"{flag}:{key}".encode()).digest()
    return int.from_bytes(digest[:8], "big") % 10000

def is_enabled(flag: str, key: str, percent: float) -> bool:
    return bucket(flag, key) < int(percent * 100)

Two properties matter here. First, raising the percentage keeps existing users in: someone inside at five percent is still inside at twenty, because their bucket never changes. Second, mixing the flag name into the hash gives every flag a different ordering of users. Leave it out and the same small group becomes the guinea pigs for every new feature, which degrades their experience and contaminates your experiment results.

In a multi-tenant product, pick the targeting key carefully. If it is the user ID, half of one company sees the new screen and half sees the old one, and support gets tickets saying "our screens look different". For features that change a workflow, the key should be the tenant ID; user ID is fine only for purely visual or personal features.

The default value is a design decision

Every evaluation call takes a default, which is returned when the rule set has not loaded, the flag was deleted, or its name was misspelled. For a release flag the safe default is almost always the old behaviour. For an operational switch, think harder: a default of "recommendation engine on" can make the switch useless exactly when the flag service is unreachable.

Another common problem is an instance taking traffic before its rule set has loaded. For a few seconds a fresh instance evaluates every flag to its default, and a feature that is at one hundred percent looks off on that instance. Tie your readiness check to the first successful rule set load.

Pitfalls

A flag does not roll back data. If the new code path writes to a new column or in a new format, turning the flag off does not restore what was written. If the old path cannot read the new format, your rollback switch is really a bug generator. Make schema changes independently of the flag and compatible in both directions: first deploy code that reads both formats, then turn on the flag.

Values can change mid-request. If a long request or a multi-step workflow re-evaluates the flag at each step, a rule change halfway through sends part of the work down the old path and part down the new one. Evaluate once at the start. For queued jobs, put the result into the job payload instead of letting the worker decide again.

Caches mix variants. If a server-rendered page or response is stored in a shared cache, the version produced for a flagged-on user can be served to a flagged-off one. Either keep those responses out of shared caches or include the variant in the cache key.

Flag combinations go untested. Ten independent flags are, in theory, 1024 different products. You cannot test them all. Avoid flags that depend on each other; when you must, declare the dependency explicitly and test a few meaningful states such as all off, all on, and the combination actually live in production.

If you cannot see the variant, you cannot debug it. When an error report or trace does not say which flag values the request ran with, "it only happens to some users" problems take days. Attach evaluated flags to the trace or log context, and to your error tracker if it supports tags.

Flag debt

The real cost of flags shows up not when they are added but when they are forgotten. A release flag sitting at one hundred percent for months keeps a dead code branch and its tests alive. One day someone switches it off by accident and behaviour from two years ago comes back.

The rules that work: every flag has an owner and an expiry date, and the removal task is filed the same day a release flag is created. A weekly report lists flags past their expiry and flags that have returned the same value for thirty days. Reference flag names as plain constants that are easy to search, so git grep finds every usage in one command; flag names built dynamically from string fragments make cleanup impossible.

When not to use them

For a small single-tenant product that deploys several times a day and can roll back in minutes, a flag service adds more complexity than it removes. Small, short-lived changes plus fast rollback are usually enough there. Do not use flags as an authorization layer, a plan rule engine or a general config store; each of those has a better home. Think of a flag as a replacement for a short-lived branch that gets deleted when the work is done. Every flag that becomes permanent is an unnamed configuration setting.