Rsync utilizes a delta-transfer algorithm to optimize data synchronization by transmitting only the differences between source and destination files rather than copying identical content. This process involves comparing file blocks using checksums to identify changes, ensuring that bandwidth is conserved during remote transfers. Understanding this mechanism clarifies why rsync behaves differently when handling modified versus unchanged files.
How Delta Transfers Work
The core efficiency of rsync lies in its ability to detect changes at the block level. When you initiate a transfer, rsync does not simply compare whole-file hashes immediately. Instead, it generates weak checksums (rolling checksums) for blocks of the source file. These are then compared against similar blocks on the destination. If a match is found, rsync verifies the match with a stronger cryptographic hash, such as MD5 or SHA1, to ensure integrity. Only if the strong hashes differ does rsync proceed to send the actual data difference.
This two-step verification process prevents false positives where different data might accidentally produce the same weak checksum. It allows rsync to quickly skip large portions of files that have not changed, focusing computational resources only on the segments that require updating. The result is a significantly faster transfer time, especially over high-latency networks or when dealing with large files that have minor modifications.
The Role of Checksums
It is important to distinguish between the rolling checksums used for initial matching and the final integrity checks. The rolling checksums allow rsync to slide across the file in fixed-size chunks, identifying potential matches efficiently. Once a potential match is identified, the stronger hash confirms whether the blocks are truly identical. If they are, no data is sent. If they differ, rsync calculates the exact byte-level differences and transmits only those deltas. This method ensures that even if a file has been shifted slightly due to an insertion earlier in the document, rsync can still identify unchanged trailing sections.
Identical vs. Similar Files
A common misconception is that rsync transfers "identical" files in some optimized way. In reality, if files are truly identical, rsync performs no data transfer at all. It merely updates metadata, such as timestamps, if the
--update flag is used or if the destination lacks the file. The term "delta transfer" specifically refers to the scenario where files are similar but not identical. For example, if you edit a single line in a 10GB log file, rsync will not resend the entire 10GB. It will analyze the file, find the changed block, and send only the necessary bytes to reconstruct the updated version on the destination.
This distinction is crucial for understanding network usage. If you observe zero bytes transferred during an rsync operation, it means the files were deemed identical based on size, modification time, and ownership, or the checksums matched perfectly. If you see partial transfers, it indicates that rsync detected structural similarities but also recognized specific differences that needed correction.
Practical Implications for Data Sync
When planning your synchronization strategy, recognizing how rsync handles these comparisons helps in configuring appropriate flags. For instance, using the
-u or
--update option tells rsync to skip files that already exist on the destination, regardless of their modification time. This is useful when you want to preserve newer files on the target. Conversely, without this flag, rsync will overwrite destination files if the source is newer, triggering the delta-transfer process for any modified content.
For scenarios involving sparse files, rsync can further optimize space usage. However, standard delta transfers focus on content differences. If you need to merge directories or handle sub-directory timestamp issues, additional flags like
--ignore-existing or specific recursion controls become relevant. You can explore more about directory handling in our guide on
Using rsync to Merge Two Source Directories.
Another critical aspect is error handling during transfers. If a transfer fails midway, rsync does not resume by re-evaluating checksums mid-stream in the way some protocols do. Instead, it typically restarts the file transfer from the beginning, although the delta-transfer logic remains active for subsequent attempts. This means that for very large files, a failed transfer can be costly in terms of time, unless you use tools that support true streaming resumption. For troubleshooting connectivity issues that prevent successful transfers, refer to our article on
Unable to Submit Files via Rsync on Ubuntu VPS.
Optimizing Transfer Performance
To maximize the benefits of delta transfers, ensure that both source and destination systems have compatible versions of rsync. Older versions may lack optimizations present in newer releases, affecting how efficiently blocks are compared. Additionally, network latency plays a significant role. Since rsync sends small packets of delta data, high latency can slow down the handshake process for each block. Using compression with the
-z flag can mitigate bandwidth issues but adds CPU overhead.
For large-scale synchronizations, consider using the
--whole-file option temporarily if delta computation becomes a bottleneck. This forces rsync to copy entire files instead of analyzing them, which can be faster on local networks with low latency but high CPU load. However, for most remote operations, the default delta-transfer behavior remains the most efficient choice for preserving bandwidth and reducing transfer times.
Frequently Asked Questions
Does rsync delete files on the destination that are missing from the source?
No, rsync does not delete files by default. It only copies new or modified files from the source to the destination. To remove extraneous files from the destination, you must explicitly use the
--delete flag, which makes the destination an exact mirror of the source.
Why does rsync sometimes take longer than a simple copy command?
Rsync spends time computing checksums and comparing blocks before transferring data. On fast local networks with minimal latency, this overhead can make rsync slower than basic tools like
cp or
scp, which do not perform detailed analysis before copying.
Can rsync resume a partially completed transfer?
Standard rsync restarts the entire file transfer if interrupted, though it uses delta-transfer logic to avoid sending unchanged blocks again. It does not resume individual byte streams mid-file in the traditional sense; instead, it re-evaluates the file from the start to determine what needs to be sent.
What happens if the source and destination have different block sizes?
Rsync uses a default block size (typically 700 bytes) for its rolling checksums. If files are significantly different in structure, the alignment of blocks may cause inefficiencies. You can adjust the block size using the
-B option, but this is rarely necessary for typical use cases.
Is rsync suitable for real-time synchronization?
While rsync is excellent for periodic backups, it is not designed for real-time synchronization out of the box. It requires a trigger mechanism, such as inotifywait or cron jobs, to monitor changes and initiate transfers, unlike dedicated daemon-based sync tools.