No Space Left on Device but df Shows Free Space

pajus 2026-09-24 9 min read

A write fails with No space left on device (ENOSPC). You run df -h and it shows gigabytes free. Or it's the reverse: df says the disk is full, but du can't find the files taking up the space. In both cases df is telling the truth. It just doesn't show the whole picture.

This article covers the three causes I see most often on Debian 13 (trixie, systemd 257, kernel 6.12): exhausted inodes, deleted files that a process still holds open, and a systemd journal that has grown too large. At the end there's a short section on ext4 reserved blocks, which explain another gap between what df reports and what users can actually write.

Start with the right questions

First, work out which situation you're in. These two commands give you most of the answer:

df -hT /var
df -i /var

Then compare df with du on the same filesystem:

df -h /
du -shx / 2>/dev/null

Symptom	Likely cause	First check
ENOSPC, `df -h` shows free space, `IUse%` at 100%	Inodes exhausted	`df -i`
`df` shows much more used space than `du`	Deleted files still open	`lsof +L1`
`/var` fills slowly, `du` points at `/var/log/journal`	Oversized journal	`journalctl --disk-usage`
Non-root user gets ENOSPC, root can still write	ext4 reserved blocks	`tune2fs -l`

One Debian 13 detail to keep in mind: according to the trixie release notes, /tmp is now a tmpfs by default, capped at 50% of RAM. If the error comes from something writing to /tmp, run findmnt /tmp and df -h /tmp before you go looking at the root filesystem.

Cause 1: exhausted inodes

Every file, directory and symlink on ext4 uses one inode. The inode ratio is fixed when the filesystem is created (growing it with resize2fs adds inodes at the same ratio, but nothing else raises the count). mke2fs creates one inode for every bytes-per-inode of space (the -i option), and its man page states plainly that this ratio can't be changed afterwards. Millions of tiny files can therefore use up every inode while most of the data blocks are still free. Once that happens, creating a new file fails with the same ENOSPC error as a full disk.

Filesystem      Inodes   IUsed IFree IUse% Mounted on
/dev/vda1      1310720 1310720     0  100% /

Finding directories with many small files

GNU du has an --inodes option that counts inodes instead of blocks. Add -x so it doesn't cross into other mounted filesystems:

du --inodes -x / 2>/dev/null | sort -n | tail -20

Directory sizes add up toward the top of the tree, so the largest entries will always be / and /var. Follow the numbers down until you reach the directory that actually holds the files. On a very full disk, a faster way is to count files per path prefix:

find /var -xdev -type f | cut -d/ -f2-4 | sort | uniq -c | sort -n | tail

In my experience the usual culprits are application cache directories, PHP session files that never get garbage-collected, a stuck mail queue under /var/spool, and build or CI leftovers. Anything that writes one small file per request or per job can do it.

Cleaning up and preventing it

Deleting a directory with millions of entries using rm -rf dir/* can fail because the shell expands the glob into an argument list that's too long. Let find do the deletion instead:

find /var/lib/php/sessions -xdev -type f -mmin +1440 -delete

Afterwards, fix whatever produced the files. That could mean the application's cleanup setting, a systemd-tmpfiles rule with an age field, or a timer. If a volume is meant to hold lots of small files (a maildir, or a cache for a package mirror), create the filesystem with a lower -i ratio or an explicit -N inode count. You can't fix the ratio later without recreating the filesystem. Filesystems such as XFS and Btrfs allocate inodes dynamically, so this failure mode mostly applies to ext4.

Cause 2: deleted files held open by processes

This one produces the "df says full, du says half empty" pattern. rm calls unlink(), which only removes the file's name. The unlink(2) man page says that if a process still has the file open, the file stays on disk until its last file descriptor is closed. du walks directory entries and can no longer see the file. df asks the filesystem and still counts its blocks.

The classic case: someone deletes a 30 GB log file while the daemon writing to it is still running. The name is gone, the daemon keeps appending, and the space never comes back.

Finding them with lsof +L1

lsof +L1 lists open files whose link count is below 1, meaning they have been unlinked. Pass a mount point to limit the output to one filesystem:

lsof -nP +L1
lsof -nP +aL1 /var

COMMAND   PID     USER FD   TYPE DEVICE    SIZE/OFF NLINK   NODE NAME
nginx    1234 www-data 5w   REG  254,1 32212254720     0 524301 /var/log/nginx/access.log.1 (deleted)

The NLINK column shows 0. The FD column (5w) tells you the descriptor number, and SIZE/OFF shows how much space is still held. If lsof isn't installed (apt install lsof), you can get the same information from /proc. Each entry in /proc/PID/fd/ is a symlink to the file the process has open, and the kernel appends (deleted) to the target once the file has been unlinked:

find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null
ls -l /proc/1234/fd/5

Run these as root. Reading another user's fd directory requires ptrace-level access.

Restart or truncate?

The clean fix is to make the process close the file. Usually that means a restart or reload, which reopens the log files:

systemctl restart nginx.service

If you can't restart right now (a database in the middle of a transaction, or a single production instance during business hours), you can free the space by truncating the file through the process's own descriptor:

: > /proc/1234/fd/5

This gets the blocks back immediately, but it has trade-offs. The data is gone for good. If the process didn't open the file with O_APPEND, its write offset stays where it was, so the next write leaves a sparse file with a large apparent size. That looks alarming in ls -l but doesn't use real blocks. And the file stays unlinked, so it keeps growing out of sight until the process restarts. Treat truncation as a stopgap and schedule the restart.

To keep this from happening again, don't rm active logs by hand. Let logrotate handle rotation with a postrotate reload, or use copytruncate for programs that can't reopen their logs. If you're tightening service units anyway, sandboxing systemd services on Debian 13 shows how to limit where a service can write at all. That also shrinks the set of directories that can fill up.

Cause 3: an oversized systemd journal

On a busy or noisy host, du often shows /var/log/journal as the largest directory under /var. Check it directly:

journalctl --disk-usage

Archived and active journals take up 3.9G in the file system.

According to journald.conf(5), the defaults for persistent storage are SystemMaxUse= at 10% of the filesystem and SystemKeepFree= at 15%, each capped at 4G. On a small VPS root disk, 10% can still be a lot. On a large shared /var, the journal can take up to 4G on its own.

Vacuuming the journal now

journalctl --rotate --vacuum-size=500M
journalctl --vacuum-time=2weeks

Keep in mind that the vacuum options only delete archived journal files. The man page warns that --disk-usage counts active files too, so vacuuming alone can leave the number much higher than you expected. Adding --rotate to the same command archives the active files first, which lets the vacuum remove them as well. There's also --vacuum-files=N, which keeps only the N most recent files.

Setting a permanent limit with a drop-in

Don't edit /etc/systemd/journald.conf directly. journald reads drop-ins from /etc/systemd/journald.conf.d/*.conf, and a drop-in keeps your change out of the way of package upgrades:

[Journal]
SystemMaxUse=500M
SystemKeepFree=1G
MaxRetentionSec=1month

systemctl restart systemd-journald.service
journalctl --disk-usage

If the journal grows fast, the real problem is usually a single service logging in a tight loop. journalctl -p warning --since today and a quick look at which unit shows up most often will normally identify it. A web server getting hammered by scanners is a common one. The web server security basics for self-hosters article covers how to cut that noise at the source.

Related: ext4 reserved blocks

By default, ext4 reserves 5% of its blocks for privileged processes. The tune2fs man page gives two reasons: it limits fragmentation, and it lets system daemons keep running after ordinary users can no longer write. As a result, df shows Avail as 0 while Size - Used is still well above zero. Non-root processes get ENOSPC, and root can still write.

tune2fs -l /dev/vda1 | grep -Ei 'reserved block count|block count|reserved blocks (uid|gid)'

On a large data-only volume, such as a backup target or a media disk, 5% can mean hundreds of gigabytes. You can lower it while the filesystem is mounted:

tune2fs -m 1 /dev/sdb1

Leave the reserve at its default on / and /var. That headroom is what lets root log in, read logs and clean up when a runaway process has filled the disk. tune2fs -u and -g can give a specific service account access to the reserve, but I'd use that sparingly.

Takeaways

Run df -h and df -i together. ENOSPC with free blocks almost always means you're out of inodes.
Find inode hogs with du --inodes -x. On ext4 the inode count is fixed at mkfs time, so size -i/-N for the workload.
If df and du disagree, run lsof +L1 or look for (deleted) in /proc/*/fd. Restart the process. Truncate through /proc/PID/fd/N only as a stopgap.
Check journalctl --disk-usage, vacuum with --rotate --vacuum-size=, and cap the journal with SystemMaxUse= in a journald.conf.d drop-in.
If root can write but users can't, check the ext4 reserved blocks with tune2fs -l. Lower them only on data volumes.
On Debian 13, confirm whether /tmp is a RAM-backed tmpfs before blaming the root disk.