You cannot directly expand multiple command arguments stored in a single variable later in a POSIX shell script because the shell performs word splitting and globbing on variables during parsing, not execution. To safely handle multiple arguments, you must use an array-like structure such as set -- or positional parameters, which allows the shell to correctly tokenize the arguments when the command is eventually executed.
The Problem with Single Variables
When you store command arguments in a standard scalar variable, such as args="-l -a", the shell treats the entire string as a single unit of text unless explicitly forced otherwise. If you attempt to run a command like ls $args, the shell first expands the variable to -l -a. However, it then performs word splitting on that result. While this might seem to work for simple cases, it breaks down completely when arguments contain spaces, quotes, or special characters. For example, if args="-f 'my file.txt'", using $args will likely cause the shell to interpret the quotes as literal characters rather than delimiters, leading to errors or incorrect file selections.
This behavior is fundamental to how POSIX shells parse commands. The shell splits words based on the characters in the $IFS (Internal Field Separator) variable, typically space, tab, and newline. It does not re-parse the expanded value for further quoting rules. Therefore, storing complex argument lists in a single string variable is inherently unsafe for later expansion. This issue is distinct from simply running a command with a string variable, as discussed in our guide on troubleshooting bash commands with string variables.
Using Positional Parameters with set --
The most robust and portable solution in POSIX sh is to use the set -- builtin command. This command replaces the positional parameters ($1, $2, etc.) with the arguments provided. By using set --, you can construct an array of arguments that respects individual element boundaries, preventing unwanted word splitting.
Consider the following approach:
# Define arguments as separate items
set -- -l -a "my file.txt"
# Execute the command using all positional parameters
ls "$@"In this example, set -- assigns -l to $1, -a to $2, and "my file.txt" to $3. When you invoke ls "$@", the shell expands $@ into each positional parameter individually, preserving the integrity of each argument. The double quotes around $@ are critical; they ensure that each parameter is treated as a separate word, even if it contains spaces. Without the quotes, ls $@ would undergo word splitting again, potentially breaking arguments with spaces.
Constructing Arguments Dynamically
Often, you need to build these arguments dynamically within a script. You can achieve this by appending to the positional parameters or by using a temporary function scope. However, modifying $@ globally can be risky if other parts of your script rely on the original arguments. A safer method is to use a local context or a subshell if you are processing input line by line.
If you are reading arguments from a file or user input, you can iterate through them and set the positional parameters accordingly. For instance:
while IFS= read -r arg; do
# Append to current positional parameters
set -- "$@" "$arg"
done < arguments.txt
# Now execute the command
command "$@"This loop reads each line from arguments.txt and appends it to the list of positional parameters. The set -- "$@" "$arg" syntax ensures that existing arguments are preserved while the new one is added. This method is particularly useful when dealing with configuration files where users might specify flags and values in various formats.
Alternative: Using Arrays in Bash
While the question specifies POSIX sh, it is worth noting that if you are using Bash, you have access to true arrays. Bash arrays allow you to store multiple elements in a single variable name without relying on positional parameters. You can define an array with args=(-l -a "my file.txt") and expand it with ${args[@]}. However, since Bash arrays are not strictly POSIX compliant, they will not work in sh environments like Dash or Ash, which are common in minimal Linux distributions and embedded systems. Always stick to set -- if portability across different POSIX shells is required.
Handling Special Characters and Quotes
One of the main advantages of using set -- over string concatenation is how it handles special characters. When you pass arguments through set --, the shell does not perform quote removal on the arguments themselves. This means that if an argument contains characters like *, ?, or $, they are passed literally to the command, provided they were quoted or escaped during the assignment.
For example:
set -- "*.txt"
ls "$@"Here, *.txt is passed as a literal string to ls, not expanded by the current shell. If you had used args="*.txt" and then ls $args, the shell might expand the glob before passing it to ls, depending on whether globbing is enabled. Using set -- gives you precise control over when and how expansion occurs.
Best Practices for Script Maintenance
To maintain clean and readable scripts, consider wrapping the argument construction and execution in a function. This isolates the logic and prevents side effects on the global state of the script. For example:
run_command() {
set -- "$@"
echo "Executing: $*"
# Replace echo with actual command
exec "$@"
}
run_command ls -l "/path/to/file"By using a function, you can pass arguments directly to it, and inside the function, you can manipulate $@ as needed. The exec command replaces the current process with the new command, which is efficient and ensures that the exit status of the command is correctly propagated back to the caller.
Remember that always quoting $@ is the golden rule. Whether you use set -- or direct invocation, failing to quote $@ will lead to unpredictable behavior, especially with arguments containing whitespace. This principle applies universally across all shell scripting scenarios, including those involving complex variable expansions.
Frequently Asked Questions
Can I use eval to expand variables later?
Yes, you can use eval to force the shell to parse the expanded variable again, but it is generally discouraged due to security risks. If the variable content comes from user input, eval can execute arbitrary commands, leading to code injection vulnerabilities. Use set -- instead for safe argument handling.
Why does my command fail when arguments contain spaces?
Your command fails because the shell performs word splitting on unquoted variables. When you use $var without quotes, the shell splits the value at every space, creating multiple separate arguments. Always use "$@" or "$var" to preserve the integrity of arguments containing spaces.
Is there a performance difference between set -- and arrays?
In POSIX sh, there is no performance difference because arrays are not supported. In Bash, arrays are slightly more memory-intensive but offer better readability. For most scripts, the performance impact is negligible, so choose the method that best fits your portability requirements.
How do I clear positional parameters?
You can clear positional parameters by running set -- with no arguments. This resets $1, $2, etc., to empty values. Be cautious when doing this inside a function, as it may affect the calling context if not handled properly.
What happens if I forget to quote $@?
If you forget to quote $@, the shell will perform word splitting and globbing on each argument. This means arguments with spaces will be split into multiple arguments, and filenames matching patterns like * will be expanded to a list of files. This often leads to unexpected errors or incorrect command behavior.