Identifying ZFS Busy Pool After System Migration

To identify a busy ZFS pool after migrating from Ubuntu, use zpool status to check for import errors and iostat -x 1 to monitor active I/O on underlying devices. If the pool is locked by stale mounts or background services, stop all dependent daemons like Docker or Nextcloud before attempting a clean reimport. This ensures no processes hold open file descriptors that prevent safe manipulation of the dataset.

Understanding Why a Pool Appears Busy

When you migrate a system, especially moving from one Linux distribution like Ubuntu to another, the storage configuration often retains references to the old environment. A ZFS pool might appear "busy" not because it is actively writing terabytes of data, but because the operating system holds locks on specific datasets. These locks can originate from several sources: stale mount points left in the filesystem hierarchy, running services that have cached handles to files within the pool, or kernel threads waiting for I/O operations initiated during the previous boot sequence. It is crucial to distinguish between a pool that is genuinely under heavy load and one that is technically locked. The former requires performance tuning, while the latter requires process termination or unmounting. In many migration scenarios, the issue is the latter. For instance, if you migrated your root filesystem but left user data on a separate pool, services configured to auto-start may attempt to access those datasets immediately upon boot, creating a lock conflict before you can manually intervene. For more context on how mount points affect pool availability, you might find Understanding the Differences Between Legacy Mountpoints and None helpful, as incorrect legacy settings can sometimes cause unexpected locking behaviors during imports.

Distinguishing Load from Locks

A common misconception is that zpool status will tell you which process is using the pool. It does not. That command provides health metrics, scrub status, and corruption reports. To see actual I/O activity, you need external tools. However, seeing zero I/O in iostat does not guarantee the pool is free; it only means no new read/write requests are being generated at that second. The lock may be held by a process in a sleeping state or waiting for a network response, such as a NFS client or a database connection timeout. Therefore, the first step in diagnosis is not just looking at the pool health, but auditing the system state. You must verify whether any services are currently interacting with the paths inside the pool. If the pool was imported automatically via systemd units during the migration, those units might be stuck in a failed or restarting state, holding onto the device nodes.

Step-by-Step Diagnosis Procedure

Follow this sequence to isolate the source of the busyness. This methodical approach prevents accidental data loss by ensuring you understand the state before forcing an export.

1. Check Pool Health: Run sudo zpool status. Look for any states other than ONLINE. If the pool is DEGRADED or FAULTED, the issue is hardware or configuration-related, not necessarily a "busy" lock. Resolve these errors first.
2. Monitor I/O Activity: Use sudo iostat -x 1 to watch the underlying virtual or physical disks (e.g., sda, nvme0n1) for ten seconds. Note the %util and await values. High utilization indicates active workloads. Low utilization suggests idle locks.
3. Identify Open Files: Use sudo lsof +D /path/to/pool/mount. Replace the path with the actual mount point of your dataset. This command lists all processes with open files in that directory tree. If this returns results, those processes are the culprits.
4. Check for Stale Mounts: Run mount | grep zfs or findmnt -t zfs. Verify that the pool is mounted where you expect it to be. Sometimes, a partial mount from a previous failed import attempt can confuse the kernel.

If lsof reveals no open files but the pool still refuses to export cleanly, the issue may lie deeper in the kernel's block layer. In such cases, checking for zombie processes or hung tasks in /proc can provide clues.

Resolving the Busy State

Once you have identified the offending processes or conditions, you can proceed to free the pool. The goal is a graceful release of resources rather than a forceful kick, which risks data corruption. First, stop any known services that depend on the storage. On a migrated system, this often includes container runtimes like Docker or Podman, media servers like Plex, or web servers like Apache/Nginx if they serve content from the pool. Stop them using your service manager:

sudo systemctl stop docker
sudo systemctl stop nextcloud
sudo systemctl stop plexmediaserver

After stopping services, retry the lsof command. If processes remain, you may need to terminate them. Be cautious when killing processes; ensure you are terminating application-level processes and not critical system components.

sudo kill -9 [PID]

If no processes are found via lsof, but the pool remains busy, try unmounting the datasets explicitly. This can clear kernel caches associated with the mount points.

sudo zfs unmount -a

Once unmounted, attempt to export the pool.

sudo zpool export [poolname]

If the export fails with a "device busy" error, you may need to force it, though this is risky. A safer alternative is to reboot the system into a rescue mode or live USB environment where the target pool is not mounted, ensuring zero interaction occurs.

Post-Migration Verification

After successfully exporting and re-importing the pool on the new platform, verify the integrity of the data. Check that all datasets are mounted correctly and that permissions are preserved. ZFS preserves ownership and permissions, but if you changed the UID/GID mapping between the old and new systems, you may need to adjust file ownership using chown. Additionally, review your systemd unit files to ensure they do not attempt to start before the ZFS pool is fully imported. Using After=zfs-import.target in your service definitions can prevent race conditions that lead to future "busy" states. For users encountering complex storage issues post-migration, understanding how partition layouts affect detection can be useful. See Fixing Incorrect LVM Partition Allocations if your migration involved hybrid LVM/ZFS setups, as misconfigured logical volumes can sometimes mask ZFS pool visibility.

Frequently Asked Questions

Why does zpool status show ONLINE but I cannot export?

The zpool status command reflects the health of the vdevs and the overall configuration, not the current I/O locks held by user-space processes. A pool can be healthy yet locked by a dormant process holding a file descriptor. Always use lsof or fuser to check for active locks instead of relying on pool status output.

Can I force an export if the pool is busy?

You can use zpool export -f, but this is dangerous as it forces the kernel to drop locks without gracefully closing file handles. This can lead to metadata corruption or inconsistent filesystem states. Only use force export if you have confirmed that no important writes are pending and you are prepared to run a scrub afterward to verify data integrity.

How do I prevent pools from becoming busy after a reboot?

Ensure that no services are configured to start before the ZFS import target is ready. Modify your systemd service files to include After=zfs-import.target and Requires=zfs-import.target. Additionally, avoid having multiple init systems or conflicting mount scripts that might attempt to mount datasets simultaneously.

What if lsof shows no processes but the pool is still busy?

This can happen if a kernel thread or a driver module holds a reference to the device node. Check for any drivers loaded specifically for that disk type. Rebooting into a live environment is the most reliable way to clear these deep kernel locks, as it starts with a clean slate and no persistent state from the previous OS installation.

Does migrating from Ubuntu change ZFS permissions?

ZFS stores ownership and permissions internally, so they should persist across migrations. However, if the new system has different user IDs (UIDs) for the same usernames, the files will appear owned by numeric IDs rather than names. Use chown to map these back to the correct users if needed, but this does not cause the pool to be "busy."