Understanding the Failure Mode
XFS has a unique characteristic compared to other Linux filesystems like ext4: it cannot be shrunk, and its online growth capabilities are strictly tied to the physical size of the block device. When an administrator expands a Logical Volume (LV) using lvextend, the block device grows immediately. However, if the subsequent xfs_growfs command is interrupted, skipped, or fails due to permissions or lock issues, the filesystem superblock still reports the old size. The kernel may detect this discrepancy during mount attempts, leading to I/O errors, read-only remounts, or a complete failure to boot if the root filesystem is affected.
This scenario creates a "dead system" where standard administrative tools become inaccessible. The primary symptom is usually an inability to mount the volume, with dmesg logs showing errors related to invalid superblock magic numbers or transaction failures. It is crucial to understand that XFS does not automatically adjust to the new LV size; manual intervention via the growfs utility is mandatory after every successful LV extension.
Distinguishing Between Corruption and Size Mismatch
Before attempting recovery, you must determine if the filesystem is truly corrupted or simply out of sync. A common mistake is assuming disk corruption when the issue is merely a pending resize. Check the current filesystem size against the logical volume size using df -h and lvs. If the LV is larger than the reported filesystem size, the issue is likely a missed xfs_growfs step rather than deep-seated bit rot or metadata corruption. This distinction saves time and prevents unnecessary destructive recovery operations.
Immediate Diagnostic Steps
When the system is unresponsive or the filesystem refuses to mount, start by collecting evidence from the kernel ring buffer. Run the following command to view recent hardware and filesystem events:
dmesg | tail -n 50Look for messages containing XFS, corruption, or I/O error. If you see warnings about the log being dirty or the superblock size mismatch, note these down. Next, attempt to check the filesystem health in a read-only mode. Never run repair utilities on a mounted filesystem unless explicitly instructed by the vendor documentation, as this can worsen the state.
If the filesystem is unmounted but the volume group is active, you can inspect the superblock details with:
xfs_info /dev/mapper/vg_name-lv_nameThis command will display the current geometry. Compare the data size field with the actual size of the logical volume obtained via lvs. If they do not match, the recovery path is straightforward: perform the missing resize operation. If they match but the filesystem is still broken, proceed to metadata repair.
Recovery Procedure
The most common fix for a "broken" XFS filesystem after an LVM expansion is simply completing the resize process. Ensure the logical volume is fully extended first:
lvextend -l +100%FREE /dev/mapper/vg_name-lv_nameOnce the LV is expanded, mount the filesystem to a temporary directory if it is not already mounted, then run the growfs command:
xfs_growfs /mount/pointThis command updates the superblock to reflect the new boundaries. In many cases, this single step resolves the "dead system" state immediately. If the filesystem was previously mounted in read-only mode due to errors, unmount it cleanly before running this command.
Handling Metadata Corruption
If xfs_growfs fails with errors indicating corruption, you must use xfs_repair. First, ensure the filesystem is completely unmounted. Then, run the repair tool:
xfs_repair /dev/mapper/vg_name-lv_nameFor severe corruption where the log is damaged, you may need to clear the log first using the -L flag. Be aware that this discards the journal, potentially losing unwritten transactions, but it is often necessary to recover the filesystem structure. After repair, attempt to mount the filesystem again. If the repair succeeds, verify data integrity by checking critical files.
Preventing Future Incidents
To avoid repeating this cycle, automate the resize process or create strict operational playbooks. Always verify the success of lvextend before moving to xfs_growfs. Consider integrating monitoring alerts that compare the LV size against the mounted filesystem size. Regular backups are essential, especially before performing any structural changes to storage volumes. For more information on managing LVM partitions correctly, see our guide on Fixing Incorrect LVM Partition Allocations.
Frequently Asked Questions
Can I shrink an XFS filesystem?
No, XFS does not support shrinking. You must backup your data, destroy the filesystem, recreate it at the desired smaller size, and restore the data. This limitation exists because XFS uses dynamic allocation structures that cannot be safely compacted.
What causes XFS to remount as read-only?
XFS mounts as read-only when it detects serious metadata corruption or hardware I/O errors to prevent further damage. Common triggers include power failures during writes, faulty cables, or inconsistent resize operations that leave the superblock in an invalid state.
Is xfs_repair safe to run on a production system?
No, xfs_repair must only be run on unmounted filesystems. Running it on a mounted volume can cause catastrophic data loss. Always take a snapshot or backup before initiating repair procedures.
How do I check if my LVM volume is fully extended?
Use the lvs command to view the current size of the logical volume. Compare this value with the output of df -h for the mounted path. If the LVS size is larger, the resize is incomplete.
What should I do if xfs_growfs fails?
If xfs_growfs fails, check dmesg for specific error codes. If it cites corruption, switch to xfs_repair. If it cites permission issues, ensure you are running as root. If the device is busy, ensure no processes are holding the filesystem open.