You can enable concurrent downloads in lftp by setting the
parallel variable to a positive integer, such as
set parallel 4, before executing your transfer command. This feature allows lftp to open multiple simultaneous connections to the server, significantly increasing throughput for large batches of files or high-latency networks.
Configuring Parallel Transfers
The primary mechanism for enabling parallel downloads in lftp is the
set parallel directive. This setting tells lftp how many concurrent file transfers it should attempt at any given time. By default, lftp may only use one connection per session depending on the version and configuration, which can bottleneck performance when downloading many small files or when network latency is high.
To apply this setting, you have two main options: passing it directly via the command line or configuring it within your
.lftprc initialization file. The command-line approach is useful for one-off tasks, while the configuration file method ensures the behavior persists across all sessions.
When using the command line, you can prepend the set command to your mirror or get operation. For example:
lftp -e "set parallel 4; mirror /remote/path /local/path" ftp://[email protected]
In this snippet,
set parallel 4 instructs lftp to maintain four simultaneous download threads. You can adjust the number based on your network capacity and server limits. A value between 3 and 8 is often optimal for most broadband connections, but higher values may be beneficial for fiber connections with low latency.
If you prefer a persistent configuration, add the following line to your
~/.lftprc file:
set parallel 4
This ensures that every time you launch lftp, it automatically initializes with four parallel connections. This is particularly helpful if you frequently perform bulk downloads or synchronize directories.
Understanding Connection Limits and Server Restrictions
While increasing the number of parallel connections can boost speed, it is not without risks. Many FTP and SFTP servers impose limits on the number of simultaneous connections per user to prevent abuse or resource exhaustion. If you exceed these limits, the server may reject new connections, leading to errors or incomplete transfers.
Before setting a high parallel value, check your server's documentation or contact the administrator to determine the maximum allowed connections. Common defaults range from 3 to 10 concurrent connections. Exceeding these limits might result in timeout errors or being temporarily banned from the server.
Additionally, some servers may throttle bandwidth individually per connection rather than per user. In such cases, increasing parallelism might not yield significant speed improvements because each thread receives a smaller slice of the total bandwidth. Monitor your actual transfer rates to determine if higher parallelism provides diminishing returns.
For SFTP connections, note that the SSH protocol itself does not natively support multiplexed data streams in the same way HTTP/2 does. Lftp handles this by opening multiple distinct SSH tunnels, which increases CPU overhead on both client and server. Ensure your system resources are sufficient to handle the additional encryption/decryption load.
Optimizing Mirror Commands for Parallelism
The
mirror command is the most common use case for parallel downloads in lftp. It recursively copies directories from the remote site to the local machine. When combined with the
parallel setting, it efficiently downloads entire directory trees.
However, the order of operations matters. Always set the parallel variable before invoking
mirror. If you set it after, the initial batch of files may already be queued with single-threaded behavior.
lftp -e "set parallel 6; set net:timeout 30; mirror --continue /data /backup/data" sftp://[email protected]
In this example, we also set a network timeout to prevent hanging connections from stalling the entire process. The
--continue flag ensures that interrupted downloads resume rather than restarting, which is crucial when managing many concurrent transfers.
You can also combine parallel downloads with filtering options. For instance, to download only specific file types in parallel:
lftp -e "set parallel 4; mirror -R --include='*.jpg' /photos /local/photos" ftp://images.example.com
This command mirrors only JPEG images, leveraging parallelism to accelerate the transfer of media-heavy directories.
Troubleshooting Common Issues
Users sometimes encounter issues when enabling parallel downloads, such as connection refusals or increased error rates. If you receive messages like "Connection refused" or "Too many connections," reduce the
parallel value immediately.
Another common problem is the "pseudo-tty allocation failed" error, which can occur when running lftp scripts in automated environments like cron jobs. This issue is unrelated to parallelism but can compound problems if not addressed separately. Ensure your shell environment is correctly configured for non-interactive sessions. See our guide on
Troubleshooting 'pseudo-tty allocation failed' error when running lftp in shell script via cronjob for detailed resolution steps.
If downloads fail intermittently, check your network stability. High packet loss can exacerbate issues with multiple simultaneous connections. Consider lowering the parallel count or increasing the timeout settings to allow more recovery time for individual streams.
For users experiencing stuck connections during SFTP sessions, particularly those showing "ls0[Connecting...]", refer to our article on
SFTP Connection Stuck: Troubleshooting 'ls0[Connecting...]' Issues to diagnose potential handshake or firewall interference.
Frequently Asked Questions
Does parallel downloading work with SFTP?
Yes, lftp supports parallel downloads over SFTP by opening multiple independent SSH connections. However, be aware that this increases CPU usage due to repeated encryption overhead. Test with lower parallel counts first to ensure your server can handle the load.
How do I limit bandwidth while using parallel downloads?
Use the
set limit rate command to cap the total bandwidth usage. For example,
set limit rate 10M restricts all combined transfers to 10 MB/s. This prevents lftp from saturating your internet connection when multiple threads are active.
Why are my parallel downloads slower than single-threaded ones?
This usually happens when the server throttles per-connection bandwidth or when network latency is high. Each parallel connection incurs a setup cost and competes for shared resources. Reducing the parallel count or switching to a server with higher per-connection limits often resolves this.
Can I use parallel uploads with lftp?
Yes, the
parallel setting applies to both uploads and downloads. Use
mirror -R (reverse mirror) to upload directories in parallel. Ensure the server allows incoming connections and has sufficient write permissions for the target directory.
What is the maximum recommended parallel value?
There is no universal maximum, but values above 10 often provide diminishing returns and may trigger server-side restrictions. Start with 4-8 connections and increase gradually while monitoring transfer stability and server response times.