Using rsync filters stored in shell variables often fails because word splitting and globbing interfere with the filter syntax. To fix this, you must disable pathname expansion using
set -f before assigning the variable, or construct the filter rules as a single unquoted string passed via standard input. This ensures that special characters like asterisks are treated as literal pattern components rather than shell wildcards during synchronization.
The Core Problem: Word Splitting and Globbing
When you attempt to store complex rsync filter rules in a bash variable, the shell processes the content before rsync ever sees it. Bash performs two critical operations on unquoted strings: word splitting and pathname expansion (globbing). Word splitting breaks the string into separate arguments based on whitespace, which destroys the structure of rsync filter lines. Pathname expansion attempts to match patterns like
*.txt against existing files in your current directory. If no matching files exist, the pattern remains unchanged; if they do, it expands to a list of filenames. Both behaviors corrupt the intended filter logic, leading to silent failures where files are either included when they should be excluded or vice versa.
Consider the following common mistake:
FILTER="- *.log"
rsync -av --filter="$FILTER" source/ dest/
In this example, the variable is double-quoted, so word splitting does not occur. However, if the current directory contains a file named
*.log, the shell might expand it incorrectly depending on the context. More critically, if you try to add multiple filters by appending to the variable without proper quoting, you risk breaking the command line structure. The most robust solution involves preventing the shell from interpreting any special characters within the variable content.
Solution 1: Disabling Globbing with set -f
The most reliable method to use filter strings in variables is to temporarily disable pathname expansion. Bash provides the
set -f option for this purpose. When
f (noglob) is active, the shell will not expand wildcards. You can wrap your rsync command in a block where globs are disabled.
(
set -f
FILTERS="- *.log
+ *"
rsync -av --filter="$FILTERS" /source/ /dest/
)
By enclosing the commands in parentheses, you create a subshell. This means the
set -f change only affects the commands inside the block and does not alter the behavior of the rest of your script. Inside the subshell, the variable
FILTERS retains its exact string content, including the asterisk, which rsync then interprets correctly as a wildcard pattern for filtering.
Solution 2: Using Standard Input for Filters
Another powerful approach is to pass filter rules via standard input using the
--filter='merge -' option. This tells rsync to read filter rules from stdin. You can pipe a heredoc or echo statement directly into rsync. This method completely bypasses variable assignment issues regarding quoting.
rsync -av \
--filter='merge -' \
/source/ \
/dest/ <
This technique is particularly useful when dealing with dynamic filter lists generated by other parts of a script. It keeps the filter syntax clean and distinct from the shell's parsing rules. Note that the
merge - directive reads from stdin until EOF. Ensure that the heredoc delimiter is quoted (e.g.,
'EOF') if you want to prevent variable expansion within the filter rules themselves, although typically filter rules are static text.
Solution 3: Constructing Arrays for Complex Rules
For very complex scenarios involving many filters, you can build an array of filter rules and pass them individually. While this requires more code, it offers granular control. Each element in the array represents one filter line.
declare -a FILTERS
FILTERS[0]="- *.tmp"
FILTERS[1]="- .git/"
FILTERS[2]="+ *"
rsync -av "${FILTERS[@]}" /source/ /dest/
However, note that rsync expects the
--filter flag for each rule. Passing an array directly like this may not work as expected unless you iterate over the array and prepend
--filter= to each element. A better way with arrays is to generate the command dynamically or use the
printf command to format the arguments correctly. This method is less elegant than the subshell approach but provides explicit handling of each rule.
Common Pitfalls and Debugging
Even with these solutions, errors can persist. Always verify your filter syntax by running rsync with the
--dry-run flag first. This allows you to see which files would be transferred or excluded without actually moving data. Additionally, check your shell's history and alias definitions. Sometimes, aliased versions of rsync or modified environments can introduce unexpected behavior. If you are using
zsh instead of
bash, the default globbing behavior is different, and you may need to use
noglob instead of
set -f. For further insights on securing your shell environment while managing such scripts, consider reviewing best practices for password handling and performance optimization in bash.
Frequently Asked Questions
Why does my rsync filter ignore the asterisk?
The asterisk is being expanded by the shell into a list of actual filenames in your current directory. To prevent this, you must disable globbing using
set -f or quote the pattern in a way that prevents expansion, though disabling globbing is safer for complex rules.
Can I use environment variables inside rsync filter rules?
No, rsync filter rules are processed statically by rsync itself. Shell variables defined in the environment are not expanded within the filter syntax. You must resolve all variables before passing the final string to rsync.
How do I debug why a file is not being synced?
Use the
--dry-run flag combined with
-v to see exactly what rsync decides to transfer. Check the output for "skipping" messages and compare them against your filter rules to identify conflicting patterns.
Does this issue affect other rsync options stored in variables?
Yes, any rsync option containing wildcards or spaces can suffer from word splitting or globbing. However, flags like
-av are generally safe. The danger lies primarily in
--filter,
--include, and
--exclude options where pattern syntax is used.
Is there a performance difference between these methods?
There is negligible performance difference between using
set -f and piping via stdin. The choice depends on readability and maintainability. The subshell method is concise, while the stdin method is cleaner for large blocks of rules.