You can merge two source directories into a single destination using rsync by running the command twice in succession. First, sync the primary directory to the backup location, then run a second rsync command with the secondary directory pointing to the same destination. This approach ensures that files from both sources are combined without deleting existing data, provided you avoid destructive flags like --delete.
Understanding the Merge Strategy
The core challenge when merging directories is preventing the second synchronization operation from overwriting or removing files established by the first. By default, rsync is safe for this task because it only copies new or modified files and creates missing directories. However, if you inadvertently include flags such as --delete, the tool will remove files in the destination that do not exist in the current source. Therefore, the strategy relies on executing two separate commands rather than trying to pass multiple sources in a single invocation.
This method is particularly useful for creating comprehensive backups where different data sets reside in separate locations on your local machine but need to be consolidated on an external drive. It allows you to treat each source independently while maintaining a unified view of the backup target. For more context on how rsync handles repeated operations safely, you might find it helpful to read about whether is rsync idempotent.
Step-by-Step Execution Guide
To successfully merge your directories, follow these precise steps. Ensure you have write permissions for the destination directory before starting.
- Identify Your Directories: Determine the paths for your two source directories (e.g.,
/home/user/docsand/home/user/photos) and your single destination directory (e.g.,/mnt/backup/merged). - Run the First Sync: Execute the rsync command for the first source. Use the
-aflag for archive mode, which preserves permissions, timestamps, and symlinks. Add-vfor verbose output to monitor progress.
rsync -av /home/user/docs/ /mnt/backup/merged/Note the trailing slash on the source directory. This syntax tells rsync to copy the contents of the directory rather than the directory itself, which is crucial for proper merging.
- Run the Second Sync: Immediately execute the same command structure for the second source directory, pointing to the identical destination.
rsync -av /home/user/photos/ /mnt/backup/merged/Rsync will now scan the destination. If a file exists in the photos folder but not in the docs folder, it will be copied. If a file exists in both, rsync will compare them based on size and modification time (or checksums if specified) and update only what has changed.
Optimizing for Backup Efficiency
When performing merges for backup purposes, efficiency is key. You should consider adding the -u (update) flag to ensure rsync skips files that are newer on the receiving side. This prevents unnecessary transfers if you have manually edited files in the backup location, although this is rare in pure backup scenarios.
Additionally, if you are dealing with large datasets, the --progress flag can provide real-time feedback on transfer speeds and file percentages. This helps diagnose issues early if one of the sources contains corrupted files or permission errors. For deeper insights into excluding specific folders during these operations to save space, check out our guide on Can't Get Rsync to Exclude Specific Directories? Here's the Include Option to Try.
Common Pitfalls and Troubleshooting
A frequent mistake is using the --delete flag. If you run rsync --delete -av source1 dest followed by rsync --delete -av source2 dest, the second command will delete all files from source1 that are not present in source2. Always omit --delete unless you explicitly want to mirror a single source exactly.
Another issue arises with hidden files. Standard globbing patterns often exclude dotfiles. Ensure your source paths are explicit and do not rely on shell expansion that might skip hidden directories. If you encounter errors related to extended attributes, refer to resources discussing Rsync backup failing on xattr to resolve permission-related blocking.
Frequently Asked Questions
Can I merge three or more directories?
Yes, simply repeat the rsync command for each additional source directory, ensuring they all point to the same destination path. The order does not matter for the final result, but processing larger directories first may improve perceived performance.
Does rsync overwrite conflicting files?
Rsync compares files by size and modification time by default. If the destination file is newer, it remains untouched. If the source file is newer or the sizes differ, the source file overwrites the destination version.
What happens if I use a trailing slash incorrectly?
Omitting the trailing slash on the source creates a nested directory inside the destination. Including it ensures the contents are merged directly into the target folder, which is essential for a clean merge.
How do I handle permission errors during merge?
Ensure you are running the command with sufficient privileges, often using sudo if writing to protected system directories. Check the destination filesystem for write permissions using ls -ld on the target path.
Is it safe to interrupt the process?
If interrupted, the partially transferred files may be corrupted. Restarting the command will resume from the beginning of those specific files due to the archive mode, ensuring data integrity upon completion.