ZFS Linux: Corrupted Directory with Missing Files, Scrub Shows No Errors?

When ZFS scrub reports no errors but files are missing, the issue likely stems from a stale kernel cache, an unmounted dataset, or metadata inconsistency rather than disk corruption. Verify the mount state with zpool status and mount | grep zfs. If the pool is healthy, force a remount or check for application-level caching issues before attempting advanced recovery.

Understanding the Discrepancy Between Scrub and Missing Files

ZFS is designed to be self-healing, meaning that during a scrub, it reads every block of data and verifies its checksum against the stored value. If a checksum mismatch occurs due to bit rot or silent corruption, ZFS will attempt to repair the data using redundant copies from other disks in the mirror or RAID-Z configuration. Consequently, if the scrub completes without reporting any errors, it strongly suggests that the underlying storage blocks are intact and consistent. This creates a confusing scenario where the physical data appears safe, yet the file listing command fails or omits specific files. This discrepancy usually points to problems at the software layer rather than the hardware layer. The most common culprits include the Linux VFS (Virtual File System) cache holding onto stale directory entries, incorrect mount options preventing proper access, or a mismatch between the ZFS pool version and the running kernel module. It is also possible that the files were deleted by another process or user after the last backup, and the ZFS snapshot system has not been configured to retain them. Before assuming data loss, you must rule out these logical inconsistencies. For more context on how ZFS datasets behave under different conditions, you might find Understanding the Differences in Legacy Mountpoints vs None helpful, as improper mount settings can sometimes obscure visibility of datasets.

Checking Pool Health and Mount Status

The first step in diagnosis is to ensure that the ZFS pool is online and healthy. Run the following command to inspect the overall status:

zpool status -v

Look for any states other than ONLINE. If the pool shows DEGRADED or FAULTED, there may be underlying I/O errors that the scrub did not catch because they occurred in metadata areas not covered by the current scrub scope. However, if the pool is ONLINE, proceed to check which datasets are currently mounted. Use the zfs list command to see all datasets and their mountpoints:

zfs list -o name,mountpoint,state

If a dataset shows a state of UNMOUNTED or has a mountpoint that does not exist in your filesystem hierarchy, the files within it will not be visible via standard tools like ls. In such cases, you may need to manually mount the dataset using zfs mount <dataset_name>.

Addressing Stale Kernel Caches and Metadata Issues

Linux kernels maintain caches for directory entries (dentries) and inode information to speed up file access. When a ZFS pool is detached, crashed, or remounted abruptly, these caches can become desynchronized with the actual state of the filesystem. This often manifests as files appearing missing or directories returning errors despite the data being present on disk. To resolve this, you should clear the dentry and inode caches. While simply rebooting the server is the most effective way to flush these caches, you can also try remounting the affected dataset. Unmount and then remount the specific dataset using the following commands:

zfs umount <dataset_name>
zfs mount <dataset_name>

After remounting, verify if the files reappear. If the issue persists, consider checking the kernel ring buffer for any ZFS-related errors using dmesg | grep ZFS. These logs can reveal if there were any transient I/O errors or memory allocation failures during the crash that might have corrupted in-memory structures.

Investigating Application-Level and Snapshot Causes

Sometimes, the "missing" files are not actually gone from the ZFS perspective but are hidden due to application behavior or snapshot management. If your application writes to a directory and crashes, it might leave temporary files in a staging area that are not committed to the final directory structure. Additionally, if you are using ZFS snapshots, ensure that the files were not accidentally removed in a newer snapshot while older snapshots still contain them. Check recent snapshots to see if the files existed previously:

zfs list -t snapshot -r <pool_name>

If you find the files in an older snapshot, you can restore them using zfs rollback or by copying them out to a safe location. Be cautious with rollbacks, as this will discard all changes made after the specified snapshot. For scenarios involving complex partition layouts that might affect ZFS availability, reviewing Reassembling Disk Partitions from Separate Images in Linux could provide insights into low-level storage integrity.

Verifying Permissions and Ownership

Ensure that the user account you are using to access the directory has the necessary permissions. A change in ownership or permission bits due to the crash or subsequent restarts can make files appear inaccessible or invisible depending on the shell configuration. Use ls -la to check detailed permissions. If you lack read access, use sudo ls to verify if the files exist but are restricted.

Advanced Recovery Options

If standard troubleshooting steps fail, you may need to resort to more advanced recovery techniques. One option is to export and re-import the ZFS pool. This forces the kernel to re-read the pool labels and metadata structures, potentially resolving any internal inconsistencies:

zpool export <pool_name>
zpool import <pool_name>

Another approach is to create a new dataset and copy data from a known good backup, verifying the integrity of each file using checksums. If the data is critical and no backups exist, consider creating a raw image of the ZFS pool device and performing forensic analysis on a separate machine to avoid further modifications to the source data.

Frequently Asked Questions

Can a ZFS scrub miss corrupted files?

Yes, if the corruption affects only the metadata pointers and not the data blocks themselves, or if the error occurs in a region not covered by the scrub's current scope. Additionally, if the corruption is in the in-memory structures that are flushed to disk correctly later, the scrub might not detect it immediately.

What should I do if my ZFS pool is marked as DEGRADED?

Replace the faulty drive as soon as possible and allow the pool to resilver. Do not ignore a degraded state, as losing another drive could result in data loss. Monitor the resilvering process with zpool status until it completes successfully.

How can I recover files from a ZFS snapshot?

You can access old versions of files by navigating to the .zfs/snapshot directory within the dataset. Alternatively, you can clone a snapshot to a new dataset or use zfs rollback to revert the entire dataset to a previous state, keeping in mind that this will erase subsequent changes.

Is it safe to run a ZFS scrub while the system is in use?

It is generally safe, but it can impact performance by consuming I/O bandwidth. For production systems, it is recommended to schedule scrubs during off-peak hours to minimize the impact on active workloads and ensure accurate health checks.

Why does ls report errors even when zpool status is healthy?

This is often caused by stale kernel caches, permission issues, or application-level glitches. The filesystem itself may be intact, but the interface between the kernel and the user space is misaligned. Clearing caches or remounting the dataset usually resolves this discrepancy.