Using rsync delete-excluded without implied delete for backup

The Misconception About Excluded Files

A common misunderstanding among users new to rsync is that the --delete option interacts with --exclude patterns to remove files that match those patterns on the destination. This is incorrect. Rsync processes exclusion filters before any deletion logic occurs. If a file matches an exclude pattern, rsync ignores it entirely for the duration of the transfer. It does not check whether that file exists on the destination side, nor does it consider deleting it.

Consequently, if you rely solely on --exclude to prevent deletion, you will find that files matching your exclusion criteria remain on the destination even after you run subsequent syncs. They are never touched, neither copied nor deleted. This behavior is by design to protect sensitive or temporary files from being managed by the backup tool. However, this creates a problem for many users who want a "mirror" style backup where old files are cleaned up, but they also want to preserve certain directories like .cache or node_modules.

Understanding Deletion Logic in Rsync

To effectively manage deletions while maintaining exclusions, you must understand how rsync determines what to delete. The --delete flag tells rsync to delete files on the destination that do not exist on the source. This check happens after rsync has determined which files to transfer based on include/exclude rules.

Here is the critical sequence: first, rsync evaluates the file list against your filter rules. If a file is excluded, it is removed from the list of files to be considered for transfer. Second, rsync compares the remaining list on the source against the destination. Third, any file present on the destination but absent from this filtered source list is marked for deletion.

This means that if you exclude a directory, say /tmp, rsync will never see /tmp files on the source. Therefore, it cannot know if they have been deleted from the source. As a result, /tmp files on the destination will persist indefinitely, regardless of the --delete flag. This is why the previous assumption that --delete-excluded exists or functions as described in older documentation is false. There is no such flag in standard rsync.

Why You Cannot Delete Excluded Items Directly

Since rsync ignores excluded files, it lacks the context to decide their fate on the destination. It treats them as external entities outside its scope. This is a safety feature. Imagine you are backing up a system and excluding /var/log. If rsync were to delete logs on the destination that were missing from the source, you would lose historical log data that might still be useful locally. By ignoring them, rsync ensures that excluded areas remain stable and independent of the source state.

Strategies for Controlled Deletion

If your goal is to have a clean backup where unused files are removed, but you want to keep specific directories intact, you need a different strategy. You cannot simply exclude them and expect --delete to work around them. Instead, you must structure your includes and excludes to explicitly define what constitutes a "valid" file for deletion purposes.

One effective method is to use --include rules to whitelist the content you want to manage, and then use --exclude to hide everything else. However, this can be complex because rsync stops processing filters once a match is found. A more robust approach involves using two separate rsync commands or carefully ordering your filters.

• Whitelist Approach: Include all files by default, then exclude specific patterns. Use --delete. This deletes everything not in the whitelist. But wait, this deletes the excluded files too! So this doesn't solve the problem of keeping excluded files.
• Two-Step Sync: First, sync the main data with --delete. Second, manually copy or link the excluded directories from the source to the destination without deletion flags. This preserves the excluded data while cleaning the rest.

Another technique is to use --filter options which offer more granular control than simple --include/--exclude pairs. For example, you can use -f 'P +/' to prune empty directories, but this still does not allow selective deletion of excluded content.

Practical Example: Preserving Specific Directories

Let us assume you want to back up /home/user to /backup/home, but you want to keep /home/user/.config intact on the destination, even if it changes on the source. You also want to delete other files that no longer exist on the source.

You cannot use a single command to achieve both goals perfectly with --delete. Here is a recommended workflow:

1. Run rsync with --delete but exclude the protected directory. This cleans up everything except the protected folder.
2. Run a second rsync command specifically for the protected directory, without --delete. This ensures the latest version is available but old versions are not wiped.

Command 1 (Clean main data):

rsync -av --delete --exclude='.config' /home/user/ /backup/home/

Command 2 (Sync protected data):

rsync -av /home/user/.config /backup/home/.config

This two-step process gives you full control. The first step acts as a mirror for most files, removing stale ones. The second step ensures your configuration data is updated but never deleted by the broader cleanup operation.

Common Pitfalls and Best Practices

When managing backups with rsync, always test your commands with --dry-run before executing them. This prints out exactly what would happen without making any changes. It is particularly important when using --delete, as mistakes can lead to data loss.

Additionally, consider the order of your include and exclude rules. Rsync processes these rules sequentially. The first rule that matches a file path determines its fate. If you exclude a parent directory, any subdirectories within it are also excluded, regardless of later include rules. This is known as the "first match wins" principle.

For more details on managing complex exclusion lists, refer to our guide on excluding directories in rsync with include option. Understanding the filter chain is essential for avoiding unexpected results.

It is also worth noting that rsync is idempotent in terms of data integrity, meaning running the same sync multiple times yields the same result, provided the source hasn't changed. However, the presence of --delete makes the operation dependent on the current state of the source. See is rsync idempotent? for a deeper dive into this behavior.

Frequently Asked Questions

Does rsync have a --delete-excluded flag?

No, rsync does not have a --delete-excluded flag. Excluded files are ignored completely by rsync and are never considered for deletion or transfer. This is a fundamental aspect of how rsync handles filter rules.

How can I delete files on the destination that are not on the source, but keep some folders?

You should use a two-step process. First, run rsync with --delete and exclude the folders you want to preserve. Second, run a separate rsync command for those excluded folders without the --delete flag to update them safely.

Why do my excluded files stay on the destination after syncing?

Excluded files stay on the destination because rsync ignores them. Since they are not part of the transfer list, rsync does not check if they exist on the source or if they should be removed. They are treated as static data outside the backup scope.

What is the difference between --exclude and --include in rsync?

--exclude prevents files matching a pattern from being transferred or considered for deletion. --include allows files matching a pattern to be processed. Rules are evaluated in order, and the first match determines the action. Use --include to whitelist specific items before applying broad --exclude rules.

Can I use rsync to sync Dockerfiles safely?

Yes, but be cautious with permissions and metadata. If you encounter errors related to extended attributes (xattr) when syncing sensitive files like Dockerfiles, ensure your filesystem supports them or use appropriate flags. See our troubleshooting guide on rsync backup failing on xattr for resolution steps.