No Space Left: Device Shows Free Space, But Nothing Works (Debian)

When a Debian system reports zero available disk space despite df -h showing free blocks, the issue is almost always exhausted inodes or full reserved root space. This state prevents all write operations, causing package managers to fail and services to crash. Restoring functionality requires identifying which directory is consuming inode resources or filling the reserved block area, then removing unnecessary files to reclaim capacity.

Distinguishing Block Usage from Inode Exhaustion

The first step in diagnosing this problem is verifying whether the filesystem has run out of data blocks or inodes. Many administrators assume that because df -h shows free megabytes, the system should be able to create new files. However, Linux file systems have a fixed number of inodes, which represent individual files and directories. If every slot is occupied, no new files can be created, regardless of how much raw storage remains. To check inode usage, run the following command:

df -i

Look at the IUse% column. If it shows 100% on your root partition (/) or home partition (/home), you have an inode exhaustion problem. This commonly occurs when a process creates thousands of tiny files, such as log fragments, session cookies, or temporary cache entries. If the percentage is low but df -h shows 100% block usage, the issue is simply full storage space, not inodes. In cases where both blocks and inodes appear available, check the reserved space. By default, ext4 reserves five percent of total space for the root user. If regular users fill the remaining ninety-five percent, the system effectively becomes read-only for non-root processes, even though df -h might still show some free space until the absolute limit is hit.

Identifying the Culprit Directory

Once you confirm inode exhaustion, you must locate the directory generating excessive files. Use the following script to find the top ten directories with the most files:

find / -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -nr | head -n 10

This command scans the local filesystem, extracts the top-level directory names, counts occurrences, and sorts them in descending order. A high count in /var/tmp, /tmp, or a specific application’s data directory usually indicates a runaway process. For example, a misconfigured cron job or a broken web server cache can generate millions of small files rapidly. If you suspect a specific service, check its logs for errors related to file creation failures. Common offenders include systemd-journald, mail servers creating bounce messages, or database temporary tables. Once identified, stop the offending service to prevent further accumulation while you clean up the existing files.

Resolving Reserved Space Issues

If inode usage is normal but writes still fail, the reserved block space may be the bottleneck. The root user retains access to this reserved area, but standard applications do not. When the usable space drops below the reserved threshold, many daemons will refuse to start or write logs, leading to the "nothing works" symptom described in initial reports. You can view the reserved space percentage using:

tune2fs -l /dev/sda1 | grep "Reserved block count"

Replace /dev/sda1 with your actual root partition identifier. To adjust this setting, use the tune2fs utility. Reducing the reserved percentage can free up space for user processes, though this is generally only recommended if you are certain no other user will require root privileges for emergency repairs.

tune2fs -m 2 /dev/sda1

This command sets the reserved block count to two percent. Be cautious when modifying this value, as lowering it too much reduces the safety margin for critical system updates and kernel operations. After adjusting the setting, verify the change by running the previous tune2fs query again.

Cleaning Up Excessive Files

Deleting files one by one is inefficient when dealing with thousands of small files. Instead, use targeted removal strategies based on the culprit directory identified earlier. For temporary files, clear the system tmp directories:

sudo rm -rf /tmp/*
sudo rm -rf /var/tmp/*

Be extremely careful when using recursive delete commands. Ensure you are targeting the correct directory structure. For application-specific caches, such as those generated by apt or dpkg, use the appropriate cleanup tools rather than manual deletion to avoid breaking package states.

sudo apt-get clean
sudo apt-get autoremove

If the issue stems from journal logs, rotate and truncate them safely:

sudo systemctl restart systemd-journald

This service restart forces the journal to flush old entries and reset file handles, often releasing significant amounts of inodes and block space immediately.

Preventing Future Exhaustion

To avoid recurrence, implement monitoring and automated cleanup policies. Install inotify-tools to monitor directory sizes and trigger alerts when file counts exceed thresholds. Configure logrotate for all major services to ensure logs are compressed and archived regularly, preventing unbounded growth. Additionally, review cron jobs and scheduled tasks for any that generate temporary files without proper cleanup routines. For deeper insights into system stability, consider reviewing how network configurations interact with local storage limits, as seen in related troubleshooting guides for connectivity issues. Ensuring that disk I/O does not become a bottleneck during peak loads helps maintain overall system responsiveness.

Frequently Asked Questions

Why does df show free space but I cannot create files?

This occurs when inodes are fully utilized, leaving no metadata slots for new files. Even with abundant data blocks, the filesystem cannot allocate a new entry without an available inode. Check df -i to confirm if inode usage is at 100%.

How do I safely delete thousands of small files?

Use the find command with the -delete flag or pipe results to xargs for efficient processing. Avoid simple rm -rf * in directories with millions of files, as the shell argument list may overflow. Target specific subdirectories known to accumulate junk data.

Can changing reserved block space fix the issue permanently?

Adjusting reserved space provides temporary relief by freeing up blocks for user processes, but it does not solve underlying causes like runaway processes. It is a mitigation strategy rather than a cure. Always identify and stop the source of excessive file creation first.

What happens if I run out of inodes completely?

The system becomes effectively read-only for non-root users. Package managers fail, logs cannot be written, and new sessions may not start. Root can still operate, but system stability is compromised because core services rely on writing logs and status files.

How can I monitor inode usage automatically?

Integrate df -i checks into your monitoring stack using tools like Prometheus or Nagios. Set alert thresholds at 80% and 95% utilization to provide early warnings before complete exhaustion occurs. Automated scripts can also trigger cleanup routines when thresholds are breached.