SFTP Rename (mv) with Wildcard

The SFTP protocol does not support wildcard expansion for the rename (or mv) command because the server processes commands literally without shell interpretation. To rename multiple files efficiently, you must use a client-side loop in your local shell or script to iterate through the matching files and execute individual rename commands for each one. This approach ensures that every file is processed correctly while adhering to the strict syntax requirements of the SFTP subsystem.

Understanding SFTP Command Limitations

When you connect to an SFTP server, you are interacting with a remote subsystem that mimics basic file operations but lacks the intelligence of a full Linux shell. The rename command in SFTP corresponds directly to the POSIX rename() system call, which expects exactly two arguments: the source path and the destination path. Unlike the local mv command in Bash or Zsh, the SFTP client does not pass wildcards like *.txt to the server for expansion. If you attempt to run rename *.txt backup_*.txt, the server will look for a literal file named *.txt, which almost certainly does not exist, resulting in a "No such file" error. This limitation exists for security and consistency reasons. Allowing server-side globbing could lead to unexpected behavior if the directory structure changes between the time the client sends the pattern and the time the server processes it. It also prevents potential security issues where a maliciously crafted filename could exploit shell injection vulnerabilities. Therefore, the responsibility for expanding patterns falls entirely on the client side.

Using Local Shell Loops for Batch Renaming

The most robust solution is to handle the wildcard expansion locally before sending commands to the SFTP session. You can achieve this by using a for loop in your terminal. This method allows you to preview the changes and control the flow precisely. For example, if you want to add a prefix to all .log files in the current directory, you can use the following structure in your bash shell:

for file in *.log; do
  echo "Renaming $file to prefix_$file"
  # In a real script, you would pipe this to sftp or use expect
done

However, since SFTP requires an interactive session or a specific batch mode, you need to integrate this loop with the SFTP command. One effective way is to generate a batch file dynamically. You can create a temporary script that contains the sequence of rename commands.

#!/bin/bash
SFTP_USER="user"
SFTP_HOST="example.com"
REMOTE_DIR="/remote/path"

# Generate batch commands
{
  echo "cd $REMOTE_DIR"
  for file in *.log; do
    echo "rename $file prefix_${file}"
  done
} > /tmp/sftp_rename_batch.txt

# Execute SFTP with batch mode
sftp -b /tmp/sftp_rename_batch.txt ${SFTP_USER}@${SFTP_HOST}

This script first constructs a list of commands. The cd command ensures you are in the correct remote directory. Then, for each file matched by the local wildcard, it appends a rename instruction to the batch file. Finally, it executes the SFTP connection using the -b flag, which reads commands from the specified file. This method is efficient, safe, and easy to debug because you can inspect /tmp/sftp_rename_batch.txt before running the actual transfer.

Alternative: Using SSH for Remote Shell Execution

If you have SSH access to the server and trust the environment, you can bypass SFTP limitations entirely by executing the mv command remotely via SSH. The SSH protocol provides a full shell environment, which supports wildcards natively. You can use the ssh command with a single-line command string:

ssh [email protected] 'cd /remote/path && for f in *.log; do mv "$f" "prefix_${f}"; done'

This approach is often simpler for sysadmins who already have SSH keys configured. However, it requires that the user has write permissions in the target directory and that the SSH daemon is enabled. Note that this method does not use the SFTP subsystem, so it may behave differently regarding file locking or atomic operations depending on the server's configuration. For users restricted to SFTP-only access, as discussed in our guide on SSH User chroot Jail: Restricting SFTP Access to Specific Directory, this SSH method might not be available due to permission constraints.

Handling Errors and Edge Cases

When renaming multiple files, errors can occur if a destination file already exists. The standard rename command in SFTP will fail if the target name is occupied. To avoid stopping the entire batch process, you should check for existing files before attempting the rename. In the local loop approach, you can add a conditional check:

for file in *.log; do
  dest="prefix_${file}"
  # Check if destination exists locally? No, we need remote check.
  # SFTP doesn't have a simple 'exists' command in batch mode easily.
  # A safer bet is to catch errors or use unique names.
  echo "rename $file $dest"
done

Since SFTP batch mode does not provide easy feedback on individual command success within the loop generation phase, consider using unique timestamps or UUIDs for new filenames if conflicts are likely. Alternatively, use the ls command in SFTP to verify the remote state before generating the batch file. Another edge case involves filenames with spaces or special characters. Always quote your variables in the local shell script to prevent word splitting. In the SSH method, ensure proper escaping of quotes within the remote command string.

Performance Considerations

For thousands of files, generating a massive batch file might consume significant memory. In such cases, streaming the commands directly into SFTP is more memory-efficient. You can use process substitution or pipes to feed commands directly:

(echo "cd /remote/path"; for file in *.log; do echo "rename $file prefix_${file}"; done) | sftp [email protected]

This technique avoids creating intermediate files on disk and works well for moderate numbers of files. However, for very large datasets, consider using tools like rsync or specialized migration scripts that handle concurrency and error recovery better than simple SFTP loops.

Frequently Asked Questions

Why does my SFTP client ignore the asterisk symbol?

The SFTP protocol treats the rename command as a literal operation between two specific paths. It does not invoke a shell on the server side to expand wildcards. Consequently, the server looks for a file literally named with an asterisk, which usually fails.

Can I use wildcards with the get or put commands in SFTP?

No, the get and put commands in standard SFTP also require explicit file names. They do not support globbing patterns. You must either specify each file individually or use a client-side tool like lftp or curl that supports recursive transfers with pattern matching.

What happens if the destination file already exists during a rename?

The SFTP rename command will fail with an error if the target path already exists. It does not overwrite existing files by default. You must manually remove the old file or choose a different destination name to proceed.

Is there a way to rename files recursively in SFTP?

Standard SFTP does not have a recursive rename command. You would need to navigate into subdirectories using cd and then apply the rename logic to each level, or write a script that traverses the directory tree and generates rename commands for all nested files.

How can I troubleshoot empty file uploads related to SFTP sessions?

If you are experiencing issues with file integrity after moving or renaming, ensure that the transfer completes fully before disconnecting. For upload-specific problems, refer to our guide on Troubleshooting: Curl SFTP Uploads Empty File without Error to diagnose connectivity or buffer issues.