Use the -a (archive) flag combined with --link-dest to create efficient, space-saving incremental backups of directories like /mnt/tosh. This approach preserves hard links by pointing unchanged files to the previous backup snapshot, ensuring that your storage usage grows only with modified or new data while maintaining a complete file history.
The Challenge of Hard Links in Backups
When backing up a filesystem such as/mnt/tosh, you often encounter files that share the same inode. These are known as hard links. A standard copy operation duplicates every byte of data for each instance of the file, leading to massive redundancy and wasted disk space. If you have hundreds of music tracks or documents linked across multiple directories, a naive backup strategy will consume terabytes of unnecessary storage.
The goal is to create a series of snapshots where identical files are not physically duplicated on the destination drive. Instead, they should point to the same physical data block. Rsync provides a mechanism to achieve this through the --link-dest option, which allows it to compare current files against a previous backup directory. If a file has not changed since the last backup, rsync creates a hard link to the existing file rather than copying it again.
This technique is particularly vital for users managing large media libraries, such as the /mnt/tosh/vorbis directory mentioned in our context. By leveraging hard links, you can maintain daily or weekly backups without the storage costs scaling linearly with the size of your dataset. For more details on how rsync handles idempotency during these operations, you might find Is rsync idempotent? helpful for understanding why subsequent runs do not alter unchanged data unnecessarily.
Setting Up the Directory Structure
Before running the backup command, you need a logical structure on your destination drive to store the snapshots. It is best practice to organize these backups into dated subdirectories. This makes it easy to identify which backup corresponds to which day and allows the--link-dest flag to easily locate the most recent successful backup.
Create a base directory for your backups, for example, /backup/mnt_tosh. Inside this, you will create timestamped folders. A common naming convention uses the date format YYYY-MM-DD. When you perform your first backup, there is no previous snapshot to link against, so all files must be copied. However, for every subsequent backup, you will specify the path to the most recently created folder.
Ensure that your destination drive has sufficient space. While hard links save significant space, you still need room for new files and modifications. To verify if your destination drive can accommodate the upcoming transfer before committing resources, consider performing a dry run first. You can check available space and estimated transfer sizes by reviewing Determining If Enough Space is Available in Destination Drive for rsync Dry Run.
Constructing the Rsync Command
The core of this backup strategy relies on combining several flags to ensure data integrity, speed, and space efficiency. The primary command structure involves specifying the source directory, the destination directory, and the reference to the previous backup. For the initial backup, you do not use--link-dest. You simply archive the entire contents of /mnt/tosh to your first dated folder. As you prepare for subsequent backups, you construct the command to include the link destination.
Here is the general syntax for an incremental backup:
rsync -a --link-dest=/path/to/previous/backup /mnt/tosh/ /path/to/current/backup/-a flag is crucial because it enables archive mode. This sets recursive mode, preserves symbolic links, permissions, ownership, timestamps, groups, and device files. Without this flag, you would lose critical metadata associated with your files in /mnt/tosh/vorbis and its subdirectories.
It is important to note the trailing slashes. In rsync, a trailing slash on the source (/mnt/tosh/) means "copy the *contents* of the directory." Without the slash, it copies the directory itself. Ensure your paths are correct to avoid nesting issues where your backup ends up inside a folder named after the source directory.
If you are dealing with sparse files—files that contain large blocks of zeros that are not actually stored on disk—you may want to optimize space further. While --link-dest handles hard links, sparse file handling can sometimes require additional attention. You can learn more about shrinking sparse files in Linux after copying them without the -S operator by reading How to Shrink Sparse Files in Linux After Copying Them Without the -S Operator.
Handling Permissions and Ownership
When backing up system directories or user-specific data, preserving ownership is essential. The-a flag includes the -o (owner) and -g (group) options. However, if you are running rsync as a regular user trying to back up files owned by root or another user, you may encounter permission denied errors.
In such cases, you should run the command with sudo or ensure you have the necessary privileges. Alternatively, if you are backing up your own home directories, standard user permissions usually suffice. Always test the command with a small subset of files first to verify that permissions are being transferred correctly. Incorrect permissions can render your backup useless if you cannot read the restored files.
Additionally, consider using the --delete flag with caution. This flag ensures that files deleted from the source are also removed from the destination. While this keeps the backup mirror clean, it can be dangerous if used incorrectly. For a robust backup strategy, many administrators prefer to keep deleted files in older snapshots and prune them manually or via a separate cleanup script, rather than deleting them immediately.
Scheduling and Automation
To make this backup process reliable, automate it using cron. Create a shell script that generates the current date, creates the new backup directory, and executes the rsync command. Schedule this script to run at a time when your system load is low. Monitor the logs of your cron jobs to ensure backups are completing successfully. Check the exit status of rsync; a zero indicates success, while non-zero values indicate errors. Regularly verify that your hard links are intact by comparing inode numbers between the current and previous backup directories. This verification step confirms that your space-saving measures are working as intended. By following these steps, you can maintain a comprehensive, space-efficient backup history of your/mnt/tosh filesystem. This method scales well even as your data grows, making it ideal for long-term archival needs.
Frequently Asked Questions
What happens if I change the --link-dest path to a non-existent directory?
If the directory specified in --link-dest does not exist, rsync will ignore the flag and perform a full copy of all files, just as if the flag were not present. This results in significantly higher storage usage and longer transfer times. Always verify that the previous backup directory exists before running the command.
Can I use --link-dest with remote destinations over SSH?
Yes, but both the source and destination servers must support the necessary filesystem features. Typically, you use rsync over SSH to connect to a remote server, and the --link-dest path must be accessible on the remote side. Ensure that the remote user has read access to the previous backup directory referenced in the link destination.
Does --link-dest work with symlinks?
No, --link-dest specifically handles hard links for regular files. Symlinks are handled separately by the -l flag within the archive mode. If a file is a symlink in the source, rsync will recreate the symlink in the destination, regardless of whether it is linked to a previous backup. It does not convert symlinks to hard links.
How do I delete old backups to free up space?
You can safely remove old backup directories using standard rm commands. Since unchanged files are hard-linked, deleting an old directory only removes the link count for those files. The actual data remains on disk as long as at least one other hard link (in a newer backup) points to it. Monitor inode usage to ensure you are reclaiming space effectively.
What is the difference between -a and -r in rsync?
The -r flag only enables recursive directory traversal, copying files and folders but ignoring metadata like permissions, ownership, and timestamps. The -a flag is shorthand for -rlptgoD, which includes recursion plus preservation of permissions, links, timestamps, group, owner, and devices. For backups, -a is almost always required to maintain file integrity.