GNU Parallel does not have a native
--tee flag, but you can replicate its behavior by redirecting both standard output and standard error to files using the
--results directory or shell redirection operators. This approach ensures that every job’s output is captured individually while maintaining the integrity of your data streams without mixing them up during parallel execution.
Understanding Output Capture in GNU Parallel
When running multiple jobs simultaneously, the terminal output from different processes can interleave, making it difficult to debug specific tasks. The standard
tee command writes to stdout and a file simultaneously, which is useful for single commands. In GNU Parallel, we need a strategy that handles this for many concurrent jobs. The most robust method involves letting GNU Parallel manage the output directories automatically.
By default, GNU Parallel sends all output to the terminal. To capture this, you must specify where the results should go. The
--results option creates a unique directory for each job ID, storing
stdout and
stderr separately. This is often cleaner than manual redirection because it preserves the association between the job ID and its output logs.
Using the --results Option
The
--results flag is the primary tool for achieving tee-like behavior in GNU Parallel. It creates a directory structure based on the job ID. Inside this directory, you will find files named
stdout and
stderr.
To use this, simply add
--results /path/to/output_dir to your command. For example:
parallel --results ./my_results ::: cmd1 cmd2 cmd3
This command runs
cmd1,
cmd2, and
cmd3 in parallel. After completion, you will find a folder structure like
./my_results/1/stdout,
./my_results/1/stderr, etc. You can then view these files individually or combine them if needed. This method is superior to piping because it avoids buffer issues and ensures that partial outputs are saved even if a job crashes.
Manual Redirection with Shell Pipes
If you prefer explicit control over filenames or want to mimic the exact syntax of
tee | cat, you can use shell redirection within the command string passed to GNU Parallel. This method requires careful handling of quotes and escaping.
You can redirect output directly inside the job definition:
parallel 'echo {} > {}.txt' ::: input1 input2 input3
In this example, each job redirects its output to a file named after the input. However, this only captures stdout. To capture stderr as well, you might use:
parallel '{ echo {}; } 2>&1 | tee {}.log' ::: input1 input2
Here, the
{ ... } group ensures that both stdout and stderr are piped to
tee. The
tee command then writes to the file
{}.log and also passes the data through to stdout, which GNU Parallel captures into its own result directory (if
--results is active) or the terminal. Note that combining
--results with manual redirection can lead to duplicate logging if not configured carefully. Usually, relying on
--results is safer for debugging.
Handling Standard Error Separately
A common requirement when replicating
tee is to keep errors separate from normal output. GNU Parallel handles this naturally with
--results. If you do not use
--results, stderr still goes to the terminal and may mix with stdout.
To force stderr into a separate file manually, you can use file descriptors:
parallel 'your_command > {}.out 2> {}.err' ::: file1 file2
This command redirects stdout to
file1.out and stderr to
file1.err. This is equivalent to running
your_command | tee file1.out >&2 but split into two distinct files. This approach is highly effective for large-scale batch processing where you need to audit errors independently.
Best Practices for Large Jobs
When dealing with thousands of jobs, capturing every line of output can be disk-intensive. Consider using
--joblog to record metadata about each job, such as exit codes and runtime, instead of full output logs. This reduces I/O overhead significantly.
Additionally, always test your parallel command with a small subset of inputs first. Verify that the output files are created correctly and contain the expected data before scaling up. This prevents accidental data loss or excessive disk usage.
For more information on managing file operations in Linux, see
Linux Command Line: Listing Files Explained. Understanding how to list and verify your output directories is crucial for ensuring your parallel jobs completed successfully.
Another related topic is running commands recursively on directories. See
Running CMD Recursively on Directories in Ubuntu to learn how to apply similar parallelization techniques to entire directory trees.
Frequently Asked Questions
Does GNU Parallel have a built-in --tee flag?
No, GNU Parallel does not include a
--tee flag. You must achieve this behavior using
--results or manual shell redirection. The
--results option is generally recommended for its simplicity and reliability.
How do I capture both stdout and stderr in a single file?
You can use shell redirection within the parallel command:
parallel 'cmd > output.txt 2>&1'. This merges standard error into standard output before writing to the file. Alternatively, use
tee inside the command block as shown in previous examples.
Can I use --results with custom filenames?
No,
--results uses the job ID to create subdirectories. If you need custom filenames, you must use manual redirection (
> or
>>) within the command string passed to GNU Parallel.
What happens if a job fails? Is the output saved?
Yes, if you use
--results or manual redirection, the output files are created and updated throughout the job's execution. Even if the job exits with an error code, the captured output remains available for inspection in the specified location.
Is --results slower than direct terminal output?
Slightly, due to the additional disk I/O required to write to the result directories. However, for most use cases, the performance difference is negligible compared to the benefit of having persistent logs for debugging and auditing purposes.