LVM: Broken XFS Filesystem - Dead System Recovery

LVM: Broken XFS Filesystem and Dead System Recovery

In this article, we will discuss how to recover from a broken XFS filesystem and a dead system when the admin team expanded an LVM volume, causing the whole system to break. We will cover the key concepts and provide a detailed, step-by-step guide to help you get your system up and running again.

Understanding the Problem

LVM (Logical Volume Management) is a disk management system used in Linux and other Unix-like operating systems. It allows administrators to allocate physical disk space flexibly and efficiently. However, if an LVM volume is expanded incorrectly, it can lead to a broken XFS filesystem and a dead system.

XFS is a high-performance filesystem used in Linux and other Unix-like operating systems. It is known for its stability and reliability, but it can become broken if the underlying disk structure is altered unexpectedly.

Recovering from a Broken XFS Filesystem and a Dead System

To recover from a broken XFS filesystem and a dead system caused by an incorrect LVM expansion, follow these steps:

1. First, you will need to boot your system using a live CD or USB drive. This will allow you to access your system without loading the broken operating system.

2. Next, you will need to identify the broken LVM volume and the XFS filesystem. You can use the lsblk and xfs_admin commands to do this.

   # lsblk
   NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
   sda      8:0    0 931.5G  0 disk
   ├─sda1   8:1    0   512M  0 part /boot/efi
   ├─sda2   8:2    0    1G  0 part /boot
   └─sda3   8:3    0 929.9G  0 part
     ├─vg00-lv_root (dm-0)  253:0    0    50G  0 lvm  /
     ├─vg00-lv_swap (dm-1)  253:1    0  15.7G  0 lvm  [SWAP]
     └─vg00-lv_home (dm-2)  253:2    0 864.2G  0 lvm  /home
   
   # xfs_admin /dev/mapper/vg00-lv_root
   

3. Once you have identified the broken LVM volume and XFS filesystem, you will need to remount the disk group volume. You can use the vgchange and mount commands to do this.

   # vgchange -ay
   # mount /dev/mapper/vg00-lv_root /mnt
   

4. Next, you will need to check the XFS filesystem for errors. You can use the xfs_repair command to do this.

   # xfs_repair /dev/mapper/vg00-lv_root
   

5. If the XFS filesystem check is successful, you can unmount the disk group volume and remount it normally.

   # umount /mnt
   # mount /dev/mapper/vg00-lv_root /
   

6. Finally, you can reboot your system to ensure that everything is working correctly.

   # reboot
   

References

• Type: Books
  Title: "Linux Logical Volume Manager (LVM) Administration"
  Author: Jan-Piet Mens
  Publisher: Packt Publishing

• Type: Articles
  Title: "How to Repair an XFS Filesystem"
  Publication: Linux.com
  Date: October 20, 2021

• Type: Online Resources
  Title: "LVM Administration"
  URL: https://wiki.archlinux.org/title/LVM

We hope this article has helped you understand how to recover from a broken XFS filesystem and a dead system caused by an incorrect LVM expansion. Remember to always be careful when modifying your system's disk structure, and have a backup plan in place in case anything goes wrong.