--delete flag to manually sync and rewrite scratch data between two 4TB HDDs. This command ensures that the destination disk becomes an exact mirror of the source, removing files that no longer exist on the source while copying new or modified data efficiently across Linux, Windows, and MacOS environments.
Understanding the Sync Strategy
When backing up personal data between two large hard drives, simply copying files is often insufficient because it does not handle deletions or updates gracefully. The goal is to maintain a one-to-one correspondence between the source drive (disk1) and the destination drive (disk2). Manual syncing allows you to control exactly when and how data is transferred, which is crucial for preserving the integrity of your backup without relying on automated scripts that might run unexpectedly.
The most robust tool for this task is rsync, available natively on Linux and macOS, and via Git Bash or WSL on Windows. It calculates differences between files and only transfers the changes, making subsequent syncs much faster than initial full copies. This method is superior to simple drag-and-drop operations because it preserves file permissions, timestamps, and handles symbolic links correctly, ensuring that your backup is a true reflection of the primary storage.
Preparation and Safety Checks
Before initiating any synchronization process, you must verify the health and connection status of both drives. A failing drive can cause the sync process to hang or corrupt data during transfer. Ensure that both disk1 and disk2 are mounted and accessible at their respective mount points. On Linux and macOS, you can check this using the df -h command to see available space and mount paths. On Windows, verify that both drives appear in File Explorer with sufficient free space to accommodate the data from disk1.
It is also critical to ensure that no other processes are writing to either drive during the sync operation. If you are working on a multi-user system, lock the directories or schedule the sync during off-hours. Additionally, consider creating a test directory with a few sample files to verify that the sync command works as expected before applying it to your entire 4TB dataset. This precautionary step helps identify potential permission issues or path errors early.
Executing the Sync Command
The core of the manual sync strategy involves running the rsync command with specific flags to ensure a complete and accurate copy. The command structure typically follows the pattern of specifying the source path, the destination path, and the necessary options. For a full mirror image that includes deletions, use the following command syntax:
rsync -av --delete /path/to/disk1/ /path/to/disk2/The -a flag enables archive mode, which preserves permissions, ownership, timestamps, and recursively copies directories. The -v flag provides verbose output, allowing you to monitor the progress and see which files are being transferred. The --delete flag is essential for this strategy; it instructs rsync to delete files on the destination that do not exist on the source, effectively keeping the two drives in perfect sync. Note the trailing slashes on the directory paths; they ensure that the contents of disk1 are copied into the root of disk2, rather than creating a nested folder named after disk1.
If you are using Windows, ensure you have installed a compatible version of rsync, such as through Cygwin, MSYS2, or the Windows Subsystem for Linux (WSL). The command syntax remains identical once the environment is configured. For macOS users, rsync is pre-installed, but be aware that newer versions may require additional flags depending on the specific implementation details of your system's package manager.
Handling Large Data Sets and Interruptions
Transferring terabytes of data can take considerable time, and interruptions due to power loss or system crashes are a risk. To mitigate this, rsync is designed to be resumable. If the process is interrupted, you can simply re-run the same command. Rsync will skip files that have already been successfully copied and continue from where it left off, checking for any partial transfers. This feature makes it highly reliable for large-scale backups compared to traditional copy tools that would require starting over from scratch.
You can also add the --progress flag to display detailed progress information for each file, which is helpful for monitoring long-running jobs. However, for very large datasets, this can generate significant output. In such cases, logging the output to a file using redirection (>> sync.log) is recommended. This allows you to review the sync history later if any errors occur. Monitoring the disk usage on both drives during the transfer can also help detect anomalies early.
Post-Sync Verification
After the sync completes, it is good practice to verify the integrity of the backup. You can compare the file counts and total sizes of both drives to ensure they match. Using the du -sh command on Linux/macOS or checking properties in Windows File Explorer can provide quick size comparisons. For a more thorough verification, you can use the --checksum option in a dry-run mode to check if any files differ, although this is computationally expensive for large datasets.
Regularly testing your backup by restoring a few random files ensures that the data is not just present but also readable and uncorrupted. This proactive approach saves time and frustration in the event of a catastrophic failure on the primary drive. By adhering to this manual sync strategy, you maintain full control over your data lifecycle, ensuring that your 4TB HDDs remain consistent and reliable for long-term storage needs.
Frequently Asked Questions
What happens if I forget the trailing slash in the rsync command?
Forgetting the trailing slash causes rsync to create a new directory with the name of the source folder inside the destination, rather than copying its contents directly into the destination root. This results in a nested structure like /disk2/disk1/... instead of /disk2/file.... Always double-check your paths to avoid this common mistake.
Can I use this strategy for incremental backups only?
Yes, rsync naturally performs incremental transfers by comparing file modification times and sizes. However, the --delete flag makes it a mirroring tool rather than a pure incremental backup tool. If you want to keep previous versions of deleted files, you should omit the --delete flag or use tools like rdiff-backup.
Is rsync safe to use on Windows?
Rsync is not native to Windows but is widely used through compatibility layers like Git Bash, WSL, or Cygwin. These implementations provide the same functionality as the Linux version. Ensure you are using a recent version of the tool to avoid bugs related to large file handling or network timeouts.
How do I monitor the progress of a large sync job?
Add the --progress flag to your rsync command to see real-time statistics for each file being transferred. For a summary of overall progress, you can use the --info=progress2 flag in newer versions of rsync, which displays a continuous progress bar without cluttering the screen with individual file names.
What should I do if the sync fails halfway through?
Rsync is designed to resume interrupted transfers. Simply re-run the same command with the same arguments. Rsync will check existing files and skip those that are already complete, continuing only with the remaining data. Check the error logs to understand why the interruption occurred before resuming.