How to Remove BOM Mark in SFTP Username when Using OpenSSH

The Byte Order Mark (BOM) in an SFTP username is caused by invisible UTF-8 encoding characters, typically the EF BB BF sequence, embedded in your configuration files. To resolve this, you must edit your ~/.ssh/config or password store to remove these non-printing bytes, ensuring OpenSSH parses the username correctly without authentication failures.

Understanding the BOM Issue in SSH

When OpenSSH attempts to connect via SFTP, it reads credentials from configuration files such as ~/.ssh/config, ~/.ssh/known_hosts, or external helper programs like sshpass. If a file containing a username has been saved with a UTF-8 BOM, the client receives a string that starts with invisible control characters rather than the expected alphanumeric name. This results in errors like "Permission denied" or "Authentication failed," even if the password is correct. The server rejects the connection because the username presented includes hidden bytes that do not match any user account on the remote system. This issue often arises when text editors on Windows or macOS save files with specific encoding defaults, which are then transferred to a Linux environment using OpenSSH.

Detecting the BOM Character

Before attempting a fix, you should confirm that a BOM is present in your configuration file. You can use command-line tools to inspect the raw bytes of the file. The most effective method is using hexdump or od to view the hexadecimal representation of the first few lines. Run the following command on your ~/.ssh/config file:

hexdump -C ~/.ssh/config | head -n 5

If a BOM exists, the output will show the bytes ef bb bf at the very beginning of the file. These three bytes correspond to the UTF-8 encoded Byte Order Mark. If you see these bytes preceding your Host or User definitions, this confirms the root cause of your SFTP connection issues. Without removing these bytes, no amount of restarting the SSH agent or checking permissions will resolve the authentication error.

Using Vim to Inspect

Alternatively, you can open the file in Vim and check for the presence of a BOM by running:

:set bomb?

If Vim reports that the 'bomb' option is set, the file contains a BOM. This visual confirmation helps verify that the file encoding is the culprit before proceeding with removal steps.

Removing the BOM Using Command-Line Tools

The most reliable way to strip the BOM is using standard Unix utilities available on almost all Linux distributions. The sed command is particularly effective for this task because it can perform in-place editing while filtering out specific byte sequences. To remove the BOM from your SSH config file, execute the following command:

sed -i '1s/^\xef\xbb\xbf//' ~/.ssh/config

This command targets the first line (1s) of the file and substitutes the specific hex sequence of the BOM with nothing effectively deleting it. The -i flag ensures the changes are saved directly to the original file. After running this, verify the fix by running the hexdump command again; the ef bb bf sequence should no longer appear at the start of the file.

Alternative Method with Iconv

If sed is unavailable or behaves unexpectedly, you can use iconv to convert the file encoding. However, this requires creating a temporary file to avoid data loss during conversion:

iconv -f utf-8 -t utf-8 -c ~/.ssh/config > ~/.ssh/config.tmp && mv ~/.ssh/config.tmp ~/.ssh/config

The -c flag allows iconv to skip invalid characters, which may include the BOM depending on the implementation. Always back up your ~/.ssh/config file before performing mass conversions to prevent accidental corruption of your SSH settings.

Preventing Future BOM Injections

To ensure this issue does not recur, you should configure your text editor to save files without a BOM. Many modern editors allow you to set the default encoding to "UTF-8 without BOM." For example, in Visual Studio Code, you can change the setting "files.encoding": "utf8" instead of "utf8bom". In Notepad++, go to Settings > Preferences > New Document and select "UTF-8 without BOM" from the Encoding dropdown. Additionally, be cautious when transferring files between operating systems. If you edit configuration files on a Windows machine using a default text editor, they may automatically inject a BOM. It is best practice to perform all SSH configuration edits directly within the Linux environment using terminal-based editors like nano or vim, which do not add BOMs by default. For more information on managing SSH configurations securely, consider reviewing how to properly manage keys and access controls. See Fixing "Permission denied» with OpenSSH: Manually adding authorized_keys for related troubleshooting on authentication failures. Another useful resource is SSH User chroot Jail: Restricting SFTP Access to Specific Directory, which details further hardening measures for SFTP users.

Frequently Asked Questions

Why does my SFTP username work in some clients but not OpenSSH?

Some graphical SFTP clients may automatically strip whitespace or BOM characters before sending credentials, whereas OpenSSH strictly adheres to the raw content of the configuration file. This discrepancy means a file that works in FileZilla might fail in the command line due to hidden encoding artifacts.

Can I use sshpass with a BOM-containing username?

No, sshpass passes the username exactly as provided in the argument or script. If the variable holding the username contains a BOM, sshpass will transmit those invisible bytes, resulting in immediate authentication failure. You must sanitize the input string before passing it to sshpass.

How do I check if my known_hosts file has a BOM?

You can check ~/.ssh/known_hosts using the same hexdump command. While less common, if you generated this file on a different OS, it might contain a BOM. Removing it prevents potential parsing errors when SSH verifies host keys, though it rarely affects username authentication directly.

What happens if I accidentally delete part of the Host definition?

If the sed command removes too much content, your SSH configuration may become invalid, leading to connection timeouts or syntax errors. Always create a backup copy of ~/.ssh/config before running bulk edit commands, allowing you to restore the file if the substitution goes wrong.

Is the BOM issue specific to Linux systems?

The issue is primarily observed when Linux-based OpenSSH clients read files created or edited on Windows or macOS systems with specific default encodings. Linux text editors typically do not add BOMs unless explicitly configured to do so, making cross-platform file transfers the most common vector for this problem.