Finding Out What Actually Filled Up: inodes, Deleted but Open Files and Reserved Blocks

When a disk space alert fires, the first instinct is to find the largest directory and delete something. That instinct fails about half the time, because "full" maps to at least three different conditions: data blocks are exhausted, inodes are exhausted, or part of the capacity is reserved for root and the application stopped being able to write long before the filesystem reported 100 percent. The symptom is identical in all three cases. The remedy is not. This post is about identifying which kind of full you have before you delete anything.

Measure the kind of fullness first

Block usage shows up in df -h, inode usage in df -i. The second command is missing from most triage runbooks, which is exactly why inode exhaustion wastes the most time: df -h reports 40 percent free while writes fail with "No space left on device". Workloads that create many small files (session files, mail spools, cache shards, millions of tiny objects) exhaust inodes long before they exhaust blocks. On ext4 the inode count is fixed when the filesystem is created and cannot be grown later, so the options are deleting files or recreating the filesystem. XFS allocates inodes dynamically and largely removes this failure class; if you are provisioning a volume that will hold hundreds of thousands of small files, that alone is a sufficient reason to choose it.

The third condition is sneakier. ext2, ext3 and ext4 reserve five percent of blocks for root by default. The intent is good: it reduces fragmentation and leaves an administrator room to act on a full disk. The consequence is that a service running as a non-root user stops writing before df ever says 100 percent. On a 20 TB volume that reserve approaches a terabyte. On a dedicated data volume, tune2fs -m 1 /dev/sdX is a reasonable setting. On the root filesystem, dropping the reserve to zero removes your own escape hatch.

Why df and du disagree

The most common contradiction during triage is the gap between df and du. du walks the directory tree and adds up the files it can see; df reads the filesystem's own accounting. Three things cause them to diverge.

First, deleted but still open files. If a process holds a file open and the file is unlinked, the directory entry is gone, du no longer sees it, but the blocks are not released until the last descriptor closes. The classic case is a log file removed with rm instead of being rotated, while the service keeps writing to the same descriptor. lsof +L1 finds them: the rows with NLINK equal to zero are precisely these files. The fix is not deletion but signalling or restarting the process that holds it. In an emergency, truncating in place through : > /proc/<pid>/fd/<n> reclaims the space, but that is first aid; the root cause lives in the rotation configuration.

Second, shadowed mount points. If something is mounted over a directory, files written to that directory before the mount still consume space while being invisible. To measure them, bind mount the filesystem elsewhere and look there:

mkdir -p /mnt/root
mount --bind / /mnt/root
du -xh --max-depth=1 /mnt/root

Third, du crossing filesystem boundaries. Without -x it also counts directories that live on another device or a network mount, and the total becomes meaningless.

Where space goes on a container host

On a host that runs containers, the thing that grows is usually not the application's data directory but three separate accumulations: image layers, container logs and anonymous volumes. docker system df -v separates them, and that output is the only sound place to answer "which one should I clean".

Logs grow without bound by default. A service printing a few lines per second reaches gigabytes within weeks. The correct fix is a limit on the log driver; deleting log files by hand recreates the deleted-but-open case described above:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "5" }
}

Be careful on the volume side. Commands that prune unused volumes count the data of any service that is not currently running as unused. Prove that a volume is genuinely orphaned before removing it; making bulk pruning a habit eventually deletes the disk of a stopped database.

The systemd journal follows the same logic. Measure with journalctl --disk-usage, set SystemMaxUse for a durable cap, and use journalctl --vacuum-size=200M when you need space right now.

Choosing between thresholds, quotas and separate volumes

There are three defences against filling up, and they do not substitute for each other.

An alert threshold is the cheapest and insufficient on its own: a warning at 90 percent is too late for a log growing five percent per hour. A rule that looks at absolute free space and at the growth rate behaves better. "Will be full in two hours" is far more actionable than "90 percent used".

Filesystem quotas keep one account from consuming the whole device on multi-user or multi-tenant systems. The price is operational complexity, and the error an application reports when it exceeds a quota is usually cryptic.

Separate volumes are, in my view, the most effective of the three. When logs, the database, container layers and the root filesystem live on separate volumes, one component overflowing does not make the machine unbootable. The cost is planning and some idle capacity; the return is a smaller blast radius. Putting everything on one volume and trusting a threshold is putting all the eggs in one basket and then weighing the basket.

Traps

A few traps recur in disk space incidents. Deleting the largest file reclaims nothing if another process holds it open. On a completely full volume some databases do not merely stop writing, they also need space to recover, so restarting the service before freeing space prolongs the outage. Sparse files show a large gap between du and ls -l, and virtual machine images usually are sparse. On snapshotting filesystems, deleted data keeps consuming space for as long as a snapshot references it, so the space is released by dropping the snapshot, not by deleting files. Finally, if /tmp is a tmpfs, files there consume memory rather than disk, which means the symptoms must be hunted on the memory side.

When not to go down this road

What makes a single space incident worth investigating is the chance that it repeats. If someone left one large file behind and the source of the growth is obvious, fixing that source is enough; there is no need to design quotas and partition layouts. The investment belongs to the moment the same alert fires a second time. At that point the answer is not deleting things, it is building a structure where growth has a boundary.