Transferring Files to Remote Hosts using Parallel SSH (GNU Parallel)
In this article, we will discuss how to transfer files to multiple remote hosts using parallel SSH and GNU Parallel. This approach is useful when you need to transfer the same file to many servers, saving time and resources.
What is Parallel SSH and GNU Parallel?
Parallel SSH is a method of executing commands on multiple remote servers simultaneously, allowing for efficient parallel processing. GNU Parallel is a free, open-source tool that simplifies the use of parallel SSH and provides additional functionality.
Prerequisites
Before proceeding, ensure that you have the following:
- Access to the remote servers via SSH
- GNU Parallel installed on your local machine
Preparing the Script
To transfer files to multiple remote hosts using parallel SSH and GNU Parallel, you'll need to create a script. In this example, we'll assume that you want to transfer a file named file.txt to all remote hosts.
First, create a file named transfer.sh on your local machine and include the following content:
#!/bin/bash
# Variables
FILE_TO_TRANSFER="file.txt"
REMOTE_USER="your_username"
REMOTE_DIR="/path/to/remote/directory"
# Transfer file using scp
scp "${FILE_TO_TRANSFER}" "${REMOTE_USER}@%h:${REMOTE_DIR}"
Replace your_username with your actual username and /path/to/remote/directory with the desired remote directory path.
Running the Script with GNU Parallel
Now that the script is prepared, you can use GNU Parallel to execute the script on multiple remote hosts. To do this, create a file named hosts.txt containing the list of remote hosts, one per line.
Once you have the list of remote hosts, execute the following command:
parallel --slf hosts.txt --nonall --work 80 --delay 1 --timeout 10 --joblog transfer.log --line-buffer bash transfer.sh
Let's break down the command:
--slf hosts.txt: Specifies the file containing the list of remote hosts--nonall: Only execute the command on hosts specified inhosts.txt--work 80: Allows up to 80 parallel jobs--delay 1: Delays the start of each job by 1 second--timeout 10: Terminates a job if it takes longer than 10 seconds--joblog transfer.log: Logs job execution information totransfer.log--line-buffer: Ensures that output is line-bufferedbash transfer.sh: Executes thetransfer.shscript on each remote host
Transferring files to multiple remote hosts using parallel SSH and GNU Parallel is an efficient way to save time and resources. By preparing a script and executing it with GNU Parallel, you can easily transfer files to a large number of remote hosts.