What You Gain and Lose With an Immutable Operating System

Keep a server running for two years, install and remove packages with apt or yum, hand-edit config files, forget about a script you ran once "just this one time." Two years later, reproducing that machine's exact state on another box is close to impossible. This is called config drift, and it is built into how traditional, mutable Linux distributions work: the package manager accumulates changes on the running system one step at a time, and few of those steps are easy to undo.

An immutable operating system is one way to stop that accumulation. The root filesystem is kept read-only, updates arrive as a full image swap rather than a series of individual package upgrades, and anything that changes the running state of the machine lives either inside a container or in a small number of explicitly persistent directories, usually /etc and /var.

Three different approaches

"Immutable operating system" is not one technology. It is a label that covers at least three different mechanisms.

Image-based distributions with atomic updates are one. Fedora CoreOS and Fedora Silverblue use rpm-ostree: the root filesystem is built from a git-like object store called ostree, an update pulls a new commit, and the next boot switches to it. Because the previous state is still on disk, rpm-ostree rollback returns to the prior image instantly. openSUSE MicroOS applies the same idea with btrfs snapshots: transactional-update applies an update inside a separate snapshot, the system switches to it on reboot, and if something breaks you pick the previous snapshot from the GRUB menu.

Minimal distributions built only to run containers are a second approach. Bottlerocket and Flatcar Container Linux never let you install general-purpose packages on the host at all. Bottlerocket does not even offer a shell; troubleshooting happens through a separate admin container or control container reached over an API. The assumption is explicit: this machine exists to run a container runtime and a kubelet, nothing else.

Declarative, whole-system-as-configuration distributions are the third path. NixOS does not keep its root filesystem read-only, but it delivers a similar guarantee a different way: the entire system is derived from one configuration, every rebuild is added as a new generation, and the previous one stays around rather than being overwritten. The result is not immutable in the strict sense, but it is reproducible in the sense that matters.

All three solve the same problem: the current state of the machine should be predictable from the definition that produced it, not from its history.

What you gain

Rollback becomes close to free. On a traditional distribution, recovering from a bad update means restoring from backup or manually downgrading packages one by one. On an image-based system it is a single command, because the previous image is already sitting on disk.

Fleet management gets simpler. When a hundred servers boot from the same image, they are bit-for-bit identical. The question "why does this one server behave differently" mostly disappears, because there is no accumulated, unrecorded difference between machines to cause it.

The attack surface shrinks. No shell, no general-purpose package manager, and on some distributions no SSH at all. Leaving a persistent backdoor is much harder when the root filesystem is read-only and every reboot returns to a known-clean image.

Testing gets more honest. You can build an image in CI and ship that exact image to both staging and production; the bits you tested are the bits that run in production, not a close approximation of them.

What you lose

Ad-hoc debugging habits stop working. A team used to SSHing in, installing a tool, and poking around will have a rough first week. Since packages cannot be installed directly on the host, a debugging tool has to run inside a container (a toolbox session, rpm-ostree's temporary package layer, or Bottlerocket's admin container), and that is one more step every time.

Some agents and kernel modules simply do not fit. A security agent that loads a kernel module, a monitoring tool that assumes it can call the host's package manager, or a custom filesystem driver was written under the assumption that you can install it on any machine. Moving it to an immutable host means either containerizing it or replacing it entirely, and that is not always possible.

Package layering, if overused, reproduces the same drift problem the whole approach was meant to solve. Both rpm-ostree and transactional-update let you layer packages directly onto the host. That escape hatch is useful, but abused it leads right back to "every machine has accumulated its own layer, no two look alike." Layering should be the exception, not a habit; anything you need permanently belongs baked into the image.

The learning curve is a genuine cost. Setting up Ignition/Butane, understanding ostree, working with btrfs snapshots, and building an image pipeline all require more upfront investment than managing a traditional distribution by hand. A small team needs time before that investment pays off.

The special handling of /etc causes confusion. Even with a read-only root, /etc is usually still writable, because it holds local configuration. rpm-ostree resolves this with a three-way merge between the old default /etc, the new default /etc, and your local changes. If you never touched a file, an update silently pulls in the new default. If you did touch it, your version is kept, and you quietly miss whatever security fix shipped in the new default. Both outcomes look correct in isolation and both can produce a surprise nobody remembers deciding on, which is why any file you hand-edit in /etc deserves a note explaining why it had to be touched.

When not to choose it

An immutable base usually hurts on developer workstations and on machines meant for trial and error, where flexibility is directly worth more than reproducibility. It can also lose on a small number of servers that each do something quite different from the others; maintaining a separate image pipeline for five servers running three unrelated workloads can cost more than just managing them by hand. The benefit scales with how many times you reuse the same image across machines. It shrinks fast for one-off, special-purpose boxes.

A practical starting point

On Fedora CoreOS, the first setup step is generating an Ignition config (JSON) from a Butane file (YAML):

variant: fcos
version: 1.5.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... user@example
systemd:
  units:
    - name: podman-example.service
      enabled: true
      contents: |
        [Unit]
        Description=Example container
        After=network-online.target
        [Service]
        ExecStart=/usr/bin/podman run --rm --name example nginx:alpine
        [Install]
        WantedBy=multi-user.target

This file is compiled with the butane tool, and the resulting Ignition config is applied once, on first boot. Every change after that has to arrive either as a new image or as an explicitly managed unit file; hand-editing /etc means you have stepped outside the intended workflow.

The decision comes down to one question: could you reproduce your machine's state from scratch tomorrow, using nothing but the definition files you kept? If the answer is no and that bothers you, an immutable base fixes that discomfort structurally. If the answer is "it doesn't matter, it's just one machine," the added complexity is not worth taking on.