/var is full and mounted on a separate partition

When /var is full on a separate partition in Debian, you must identify the largest files using du, clean up logs or caches safely, and if necessary, resize the partition. This process involves stopping services like Postfix to prevent further writes, removing old compressed logs, and potentially expanding the logical volume using LVM tools.

Diagnosing Disk Usage

The first step is to understand what is consuming space within the /var directory. Since /var is mounted separately, standard disk usage commands might not show the specific constraint unless you target that mount point. Use the following command to see the total size of /var:

df -h /var

If the "Use%" column shows 100%, you are indeed out of space. To find which subdirectories are the largest, use du with the -sh flags (summarize and human-readable). Start from the root of the partition:

sudo du -sh /var/* | sort -hr

This output will likely reveal that /var/log, /var/lib, or /var/cache are the primary culprits. For deeper inspection, drill down into these directories. For example, to find large log files:

sudo du -sh /var/log/* | sort -hr

It is crucial to stop writing to /var before performing cleanup operations to avoid data corruption or service crashes. If you are running a mail server, which heavily uses /var/mail and /var/spool, stop the relevant services immediately. In many Debian setups, this includes Postfix or Exim.

sudo systemctl stop postfix
sudo systemctl stop exim4

Safely Cleaning Logs and Caches

Once the services are stopped, you can proceed with cleaning. Never delete active log files directly using rm while a service holds the file descriptor open; instead, truncate them. This preserves the file handle so the service can continue logging when restarted. To clear a specific log file without deleting it:

sudo truncate -s 0 /var/log/syslog
sudo truncate -s 0 /var/log/mail.log

For older rotated logs (e.g., syslog.1.gz), you can safely remove them. However, be cautious with /var/cache/apt. Clearing the apt cache is safe but does not free up much space compared to logs. If you need to reclaim significant space from package managers, consider purging unused kernels or cleaning up Docker images if applicable.

sudo apt-get clean
sudo apt-get autoremove --purge

If your system uses journalctl for logging, you can limit its size automatically. Edit /etc/systemd/journald.conf and set SystemMaxUse=50M to restrict journal size, then restart the service:

sudo systemctl restart systemd-journald

Handling Large Data Directories

If /var/lib is full, check for database files or container storage. For PostgreSQL or MySQL, you may need to archive old tables. For Docker, ensure you are not pruning images accidentally. As noted in our guide on managing disk space issues, understanding the specific application's data retention policy is key. Do not blindly delete files in /var/lib/docker unless you have verified they are unused containers.

Resizing the Partition

If cleaning does not resolve the issue, you may need to expand the /var partition. This assumes you are using LVM (Logical Volume Manager), which is common in modern Debian installations. First, check your volume group status:

sudo vgdisplay

Look for "Free PE / Size" in the output. If there is free space in the volume group, you can extend the logical volume. Replace vgname and lvname with your actual names:

sudo lvextend -l +100%FREE /dev/vgname/lvname

After extending the logical volume, you must resize the filesystem to utilize the new space. For ext4, use:

sudo resize2fs /dev/vgname/lvname

For XFS, use:

sudo xfs_growfs /var

Finally, restart any stopped services to resume normal operation.

sudo systemctl start postfix

Preventing Future Issues

Implement monitoring to alert you before /var reaches capacity. Tools like nagios, zabbix, or simple cron jobs checking df output can send email alerts at 80% usage. Additionally, configure log rotation policies to ensure logs do not grow indefinitely. Review server boot issues related to partitions to ensure your fstab entries remain stable after resizing.

Frequently Asked Questions

Can I delete files in /var/log directly?

Yes, but only if the associated service is stopped or if you use truncation. Deleting an active log file can cause the service to fail to write new logs until it is restarted. Always prefer truncating over deleting for active files.

What if I don't have LVM?

If you are not using LVM, resizing partitions requires more complex steps involving live CDs or backup and restore procedures. You may need to shrink adjacent partitions to create space, which carries a higher risk of data loss. Consider migrating to LVM for easier management in the future.

How do I find large files across the entire /var tree?

Use the command find /var -type f -size +100M to list files larger than 100MB. This helps identify unexpected large files that are not part of standard log or cache structures. Combine this with ls -lh to inspect their details.

Is it safe to run apt-get clean?

Yes, apt-get clean removes all cached .deb packages from /var/cache/apt/archives. It does not affect installed software. This is a safe way to reclaim several hundred megabytes of space on systems with frequent updates.

Why did my service crash after clearing logs?

Services expect log files to exist. If you deleted the file instead of truncating it, the service might lose its file descriptor. Restarting the service usually resolves this by recreating the log file. Ensure you stop services before making changes to prevent race conditions.