BTRFS scrub errors involving deleted files typically stem from metadata inconsistencies, delayed inode updates, or background tasks like snapshots and balance operations interfering with the deletion process. Resolving these issues requires stopping all write operations, running a full scrub with repair mode enabled, and verifying that no active snapshots are holding references to the removed data.
Understanding Scrub Errors on Deleted Files
When you encounter scrub errors specifically related to files that have already been deleted, it indicates that the file system metadata is out of sync with the actual state of the disk blocks. The BTRFS scrub tool reads every block in the file system to check for checksum mismatches. If a file has been logically deleted but its metadata (such as directory entries or inode structures) still points to valid data chunks, or if those chunks contain stale data, the scrub will report an error.
This scenario often arises because BTRFS uses Copy-on-Write (CoW). When a file is deleted, the operation does not immediately wipe the underlying data; it simply removes the reference count. If a snapshot exists at the time of deletion, the old version of the file remains accessible through that snapshot, keeping the data alive. If the data was corrupted before deletion, or if the metadata update failed partially, the scrub detects the inconsistency. It is crucial to distinguish between a simple reporting glitch and actual data corruption, which can be determined by the severity and type of error reported during the scrub.
Preparation and Safety Measures
Before attempting any repairs, you must ensure that no other processes are writing to the file system. Background tasks such as snapper, btrfs-send, or automated backups can interfere with the scrub process and mask the true state of the file system. Additionally, running a scrub while the file system is mounted read-write can lead to race conditions where new writes occur simultaneously with the verification process.
- Unmount the File System: Ideally, unmount the affected partition. If this is not possible due to it being the root file system, mount it read-only using
mount -o remount,ro /mount/point. - Stop Services: Halt any services that might access the file system, including database servers, web servers, or backup agents.
- Check for Snapshots: List all existing subvolumes and snapshots using
btrfs subvolume list /mount/point. If there are snapshots that include the deleted files, you must decide whether to keep them. Keeping them preserves the old data but may prevent the scrub from fully cleaning up the metadata references.
If you suspect that the deleted files were part of a critical backup, consider creating a raw image of the partition using dd or clonezilla before proceeding. This ensures that you have a fallback if the repair process causes further instability.
Running the Scrub with Repair Mode
The standard scrub command only checks for errors without fixing them. To resolve inconsistencies caused by deleted files, you must use the repair flag. This tells the kernel to attempt to fix correctable errors, such as mismatched checksums or missing tree roots.
btrfs scrub start -r /mount/pointThe -r flag enables repair mode. Note that this operation can take a significant amount of time depending on the size of the file system. Monitor the progress using:
btrfs scrub status /mount/pointIf the scrub reports that it fixed errors, verify the integrity again by running a non-repairing scrub. If errors persist, it indicates more severe corruption that cannot be automatically repaired, potentially requiring manual intervention or recovery tools.
Handling Persistent Metadata Issues
If scrub errors continue after a repair run, the issue may lie deeper within the file system tree structure. In such cases, checking the health of specific subvolumes is essential. You can inspect the status of individual subvolumes to see if they are flagged as inconsistent.
Another common cause is incomplete transactions. BTRFS relies on transaction commits to finalize changes. If the system crashed during a deletion, the transaction might be left in a limbo state. Running a filesystem check with btrfs-check (formerly btrfsck) can help identify and fix structural issues. However, btrfs-check should always be run on an unmounted file system, and it is recommended to use the --repair flag only if btrfs scrub fails to resolve the issue, as repair modes in btrfs-check can sometimes cause data loss if misapplied.
btrfs-check --readonly /dev/sdXNUse the --readonly flag first to assess the damage without making changes. If the output suggests minor inconsistencies that align with the scrub errors, you may then proceed with a repair run. Always back up your data before executing any repair commands in btrfs-check.
Post-Repair Verification
After successfully running the scrub with repair, remount the file system normally and verify that the errors no longer appear. Check the system logs using dmesg | grep btrfs for any residual warnings. It is also advisable to run a second scrub without the repair flag to confirm that the file system is now clean. If the errors return, it may indicate hardware issues, such as failing sectors on the disk, rather than file system corruption.
Frequently Asked Questions
Can I run btrfs scrub while the system is running?
Yes, you can run a scrub on a mounted file system, but it is less accurate and slower. For deleting file issues, running it read-only or unmounted is strongly recommended to prevent new writes from masking or complicating the errors.
What does it mean if scrub says "no errors found" but I still see issues?
This usually means the file system metadata is consistent, but application-level issues or user errors are causing the perceived problems. It could also indicate that the error lies outside the scope of scrub checks, such as in the kernel drivers or hardware controllers.
Should I delete snapshots before running scrub?
If the deleted files are referenced by snapshots, the scrub cannot fully remove their metadata until the snapshots are deleted. You should delete unnecessary snapshots first to allow the scrub to clean up all orphaned data references.
Is btrfs-check safe to use on a production server?
No, btrfs-check is a diagnostic and repair tool that should only be used when necessary. Using it incorrectly can corrupt the file system further. Always have a recent backup before running any repair functions.
How do I know if my disk hardware is failing?
If scrub errors persist after repairing the file system and checking for metadata issues, check SMART data using smartctl on the device. Recurring checksum errors on different files over time are a strong indicator of physical disk failure.
Can I recover files that were deleted before the scrub error appeared?
Recovery is difficult because BTRFS CoW makes traditional file carving ineffective. If you have snapshots, restore from them. Otherwise, specialized forensic tools may attempt recovery, but success is not guaranteed, especially if the space has been overwritten.