The "No space left on device" error on an AWS EC2 Ubuntu instance with a 30GB attached volume typically stems from Docker consuming all available disk space or inodes, rather than the underlying EBS volume being full. This occurs because Docker stores images, containers, and logs in specific directories that can grow rapidly during builds or runtime operations. Resolving this requires identifying whether the issue is related to data blocks or inode exhaustion, then cleaning up unused resources.
Understanding the Root Cause
When you encounter a
no space left on device error despite having free space reported by
df -h, the problem is often related to how Docker manages storage. Docker uses a storage driver (typically
overlay2) to manage image layers and container writable layers. If these layers accumulate without cleanup, they fill up the allocated space on your EBS volume. Additionally, if every file on the volume is small (such as log files or temporary build artifacts), you might run out of inodes even if megabytes of space remain. This distinction is critical for diagnosing the issue correctly.
Detecting Inode Exhaustion
Before assuming the volume is full of data, verify if you have exhausted the number of inodes. Inodes are metadata structures that store information about files. Each file, directory, or symbolic link consumes one inode. When the inode count reaches the maximum limit defined when the filesystem was created, no new files can be created, resulting in the same error message. To check inode usage, run the following command:
df -i
Look at the
IUse% column for your root partition or the mounted volume. If it shows
100% or close to it, you have an inode exhaustion issue. This is common in environments where many small files are generated, such as during frequent Docker builds or application logging.
Cleaning Up Docker Resources
Docker is frequently the culprit for rapid disk space consumption. It retains unused images, stopped containers, and dangling volumes that take up significant space. To reclaim this space safely, use the Docker system prune command. This command removes stopped containers, unused networks, dangling images, and build cache. Execute the following command to clean up resources:
docker system prune -a --volumes
The
-a flag ensures all unused images are removed, not just dangling ones, and
--volumes deletes anonymous volumes that are no longer referenced by any container. Be cautious, as this action is irreversible. Ensure you do not need any of the removed resources before executing. For more targeted cleanup, you can remove specific stopped containers using
docker rm or images using
docker rmi.
Managing Disk Space on Ubuntu
If Docker cleanup does not resolve the issue, investigate other processes consuming space on the Ubuntu instance. Check large directories using the
du command to identify which folders are taking up the most space. For example, to find the top 10 largest directories in the current path, run:
du -ah . | sort -rh | head -n 10
Common culprits include old kernel versions, log files in
/var/log, and cached package files in
/var/cache/apt. You can clear the APT cache by running
sudo apt-get clean. Additionally, rotate or compress old log files to prevent them from growing indefinitely. Regular maintenance of these areas helps maintain sufficient free space for system operations.
Expanding the EBS Volume
If cleaning up resources is insufficient and your applications legitimately require more space, consider expanding the EBS volume. AWS allows you to increase the size of an existing EBS volume without stopping the instance. First, modify the volume through the AWS Management Console or CLI to increase its capacity. After resizing the volume, you must extend the filesystem to utilize the additional space. On Ubuntu, this typically involves using
resize2fs for ext4 filesystems or
xfs_growfs for XFS. Verify the new space with
df -h after completing the resize operation.
Preventive Measures
To avoid recurring space issues, implement automated cleanup policies. Configure Docker to automatically prune unused resources periodically using cron jobs or systemd timers. Monitor disk usage with tools like
prometheus-node-exporter integrated with Grafana for real-time alerts. Set up log rotation for application logs to ensure they do not consume excessive space. By proactively managing resources, you can maintain a stable environment and prevent unexpected service disruptions due to disk space exhaustion.
Frequently Asked Questions
How do I check if my EBS volume is actually full?
Run
df -h to view the percentage of used space on each mounted filesystem. If the usage is below 100% but you still get the error, the issue is likely inode exhaustion or Docker-specific storage management rather than raw volume capacity.
Can I recover space from stopped Docker containers?
Yes, stopped containers occupy disk space. Use
docker ps -a to list all containers, including stopped ones. Remove unwanted containers with
docker rm <container_id> to free up the space they were using.
What causes inode exhaustion on an Ubuntu instance?
Inode exhaustion happens when too many small files are created, exceeding the maximum number of inodes allowed on the filesystem. This is common with applications generating many temporary files or logs, even if total data size is low.
How do I expand an EBS volume online?
You can increase the volume size via the AWS Console while the instance is running. After resizing, use
resize2fs or
xfs_growfs to extend the filesystem. No downtime is required for the expansion process itself.
Why does Docker consume so much space on EC2?
Docker stores image layers, container writable layers, and build caches locally. Without regular pruning, these components accumulate over time, especially during development or continuous integration workflows, leading to rapid space consumption.