Storage has two limits: bytes and inode counts. When a partition runs out of inodes, writes fail even with terabytes free — usually a mail spool or session directory with millions of tiny files.
df -i shows IUse% at 100%. Some workload creates millions of tiny files — mail queues, PHP sessions, cache dirs.
A process holds a deleted log open; the space frees only when the process restarts. lsof shows them.
df -i | awk 'NR==1 || $5+0 > 80'du --inodes -x /var | sort -rn | headfind /var/lib/php/sessions -type f -mtime +7 -deletelsof +L1 | grep -i deletedThis is why 'rm the big log' sometimes frees nothing: if a daemon still has it open, the kernel keeps the blocks until the fd closes. Restart the daemon, then re-check df.
Deleted-but-open files: a process holds a large file's descriptor, the space stays allocated until the process closes it. lsof +L1 lists the culprits; restarting the holding process frees the space immediately.
Mount points shadowing full directories (a mount over a directory hides its contents), sparse-looking-but-dense filesystems, and reserved blocks (tune2fs -l shows the 5% root reserve df counts as 'used').
Our most-documented failures, packaged as ready-to-ship starter kits: Docker, Kubernetes, and Terraform.
Browse the template store →One-time. Yours to modify. Instant download from the NinjaOps template store.