rsync: Not Iterating Sub-Directories - Modify Timestamp?

The issue where rsync appears to skip sub-directories is rarely caused by timestamp modifications themselves. Instead, it typically stems from incorrect path syntax, such as trailing slashes that change source interpretation, or restrictive filter rules in your command. Understanding how rsync calculates file hashes and compares directory structures is essential for resolving these iteration failures effectively.

Common Causes for Missing Sub-Directories

When rsync does not iterate through expected sub-directories, the most frequent culprit is the presence or absence of a trailing slash on the source path. In rsync, the syntax rsync -av /source/ /dest/ copies the *contents* of the source directory into the destination. Conversely, using rsync -av /source /dest/ copies the source directory itself, creating a nested structure like /dest/source/.... If you expect files to appear directly in the destination root but they are nested inside a folder named after the source, this syntax error is likely responsible. Another common cause involves the use of exclude or include filters. If you have previously used options like --exclude='*/' or similar patterns in a script, rsync might be skipping directories based on those rules. Additionally, permission issues can prevent rsync from reading sub-directories. If the user running the rsync command lacks read and execute permissions on a sub-directory, rsync will silently skip it unless verbose logging is enabled. Always check the output of ls -l on the source directory to verify access rights.

Verifying Path Syntax and Structure

To diagnose whether path syntax is the issue, compare the directory tree before and after the sync operation. Use the tree command or find . -type d to visualize the structure. If the sub-directories are missing in the destination, check if they exist under a parent folder matching the source name. For example, if syncing /home/user/docs to /backup, ensure you are not inadvertently creating /backup/docs/... when you wanted /backup/.... You can also test the behavior with a small, controlled directory structure. Create a temporary folder with nested sub-directories and run rsync with the -v (verbose) flag. This will print every file and directory being transferred. If the verbose output stops at a certain level, the issue is likely related to filters or permissions rather than timestamps. Refer to our guide on Using rsync to Merge Two Source Directories for more insights on managing complex directory structures during synchronization.

The Role of Timestamps in Syncing

Timestamps play a critical role in determining whether rsync needs to transfer a file. By default, rsync uses a quick-check algorithm that compares modification times and file sizes. If the timestamps match, rsync assumes the files are identical and skips them. However, modifying timestamps should not prevent rsync from iterating over sub-directories. If sub-directories are not being visited at all, the problem lies elsewhere. It is important to distinguish between skipping files due to unchanged timestamps and skipping directories entirely. If rsync iterates through the directory tree but does not update files, adjusting the --modify-window option might help. This option allows rsync to consider two files identical if their timestamps differ by less than the specified number of seconds. This is useful when dealing with filesystems that have different timestamp granularities, such as transferring files between Linux and Windows systems. For cases where you need to preserve specific ownership or group attributes without affecting iteration, understanding options like --no-o is helpful. Our article Understanding the '--no-o' Option in rsync: Preserving Timestamps explains how various flags impact attribute preservation, which can indirectly affect sync behavior if permissions are involved.

Debugging with Verbose and Dry-Run Modes

Effective debugging requires enabling detailed logging. Use the -vv flag for very verbose output, which shows exactly what rsync is doing at each step. This includes checking directory contents, comparing timestamps, and deciding whether to transfer files. Combine this with -n (dry-run) to simulate the sync without making any changes. This allows you to see which files would be transferred and identify any silent skips. If you suspect permission issues, run rsync with sudo if appropriate, or check the logs for "Permission denied" errors. These errors often indicate that rsync cannot enter a sub-directory to inspect its contents. Ensure that the target directory has sufficient disk space and that the filesystem is not read-only. Additionally, verify that network connections are stable if syncing over SSH, as interrupted connections can sometimes lead to incomplete directory iterations.

Best Practices for Reliable Synchronization

To avoid common pitfalls, always double-check your source and destination paths. Use absolute paths to prevent confusion regarding the current working directory. Regularly review your exclude lists to ensure they do not inadvertently block sub-directories. Consider using the --stats flag to get a summary of transferred data, which can help verify that the expected volume of files was processed. For large-scale synchronizations involving many sub-directories, consider breaking the process into smaller chunks or using parallel transfers with tools like parallel-rsync. This can improve performance and make it easier to isolate issues. Always test your rsync commands in a safe environment before applying them to production data. For advanced scenarios involving root directories, consult Syncing Root Directories: A Safe Idea with Rsync? to understand the risks and best practices associated with high-level synchronization tasks.

Frequently Asked Questions

Why does rsync skip some files even with the -av flag?

Rsync skips files if their modification time and size match the destination copy, assuming no changes were made. You can force re-transfer by using the -c (checksum) option, though this is slower.

How can I fix permission denied errors during rsync?

Ensure the user running rsync has read and execute permissions on the source directories and write permissions on the destination. Using sudo may be necessary if the source contains protected files.

Does modifying timestamps affect directory iteration?

No, changing timestamps does not prevent rsync from entering or listing sub-directories. It only affects whether individual files are transferred based on the quick-check algorithm.

What is the difference between trailing slashes in rsync?

A trailing slash on the source (/src/) copies the contents into the destination, while omitting it (/src) copies the directory itself, creating a nested folder in the destination.

Can I use rsync to sync only new sub-directories?

Yes, rsync naturally handles this by checking directory structures. If a sub-directory exists in the source but not the destination, rsync will create it and copy its contents.