Prerequisites for Key-Based Authentication
Before attempting to connect, you must ensure that the Windows Server is configured to accept SSH connections via OpenSSH. Microsoft provides an official OpenSSH server implementation for Windows 10 and Windows Server 2019 or later. You need administrative privileges on the server to install and configure this service. Additionally, you must have a private key file (typically with a .ppk extension for PuTTY-based tools like FileZilla) ready on your client machine. If you only have an OpenSSH-formatted key (usually id_rsa), you will need to convert it using PuTTYgen.
Ensure that the OpenSSH Server feature is installed via PowerShell using Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0. Verify that the SSH service is running and set to start automatically. The firewall must also allow inbound traffic on port 22. Without these foundational steps, FileZilla will fail to establish any connection, regardless of how the key is configured.
Configuring FileZilla Site Manager
Open FileZilla and navigate to the Site Manager from the File menu. Create a new site entry and enter the hostname or IP address of your Windows Server. In the Protocol dropdown, strictly select SFTP - SSH File Transfer Protocol. Do not choose FTP or FTPS, as these do not support SSH key authentication natively in the same way. Enter your username, which is typically your Windows account name or Administrator, depending on how the SSH user was configured on the server.
Locate the Logon Type field and change it from "Normal" to Key file. This is the critical step that enables passwordless login. Once selected, a path field appears below. Click the browse button and locate your private key file. Ensure the file type filter is set to show all files or specifically *.ppk if you are using a PuTTY format key. If you use a standard OpenSSH key, FileZilla may not recognize it directly unless converted. Save this site configuration to avoid re-entering details every time.
Handling Key Format Compatibility
FileZilla relies on the PuTTY suite for cryptographic operations, meaning it expects keys in the PuTTY Private Key (.ppk) format. If you generated your key using ssh-keygen on Linux or macOS, it will be in OpenSSH format. Attempting to load this directly into FileZilla will result in a "Invalid key file" error. To resolve this, download and install PuTTYgen on your local machine. Load your existing OpenSSH private key into PuTTYgen, then save it as a .ppk file. This conversion ensures compatibility with FileZilla’s backend.
If you are managing multiple servers, consider organizing your keys in a dedicated directory. This reduces the chance of selecting the wrong key during connection attempts. Always keep backups of your private keys securely offline. Losing access to the private key means losing access to the server unless you have alternative recovery methods, such as console access or password authentication enabled temporarily.
Troubleshooting Connection Failures
If the connection fails after correct configuration, check the Message Log in FileZilla for specific error codes. Common issues include incorrect username permissions or firewall blocks. Ensure that the Windows user associated with the SSH key has appropriate shell access. By default, OpenSSH on Windows maps to the Windows user profile. Verify that the public key corresponding to your private key is present in the %USERPROFILE%\.ssh\authorized_keys file on the server. Permissions on this file must be strict; incorrect ownership can cause SSH to reject the key silently.
Another frequent issue is network latency or intermediate firewalls dropping idle connections. If the connection drops shortly after logging in, adjust the timeout settings in FileZilla’s Global Settings under Connection. Increasing the keep-alive interval can help maintain stability over unreliable networks. For persistent failures, try connecting via command-line SSH first using ssh -i key.ppk user@host if using WSL or Git Bash, to isolate whether the issue lies with FileZilla or the server configuration itself.
Security Best Practices
Using key-based authentication significantly enhances security compared to password logins. However, proper key management is essential. Never share your private key file via email or unencrypted channels. Store it in a secure location with restricted file permissions. Consider setting a passphrase on your private key for an additional layer of protection. While this requires entering a password each time, it prevents unauthorized use if the key file is stolen.
Regularly audit the authorized_keys file on your Windows Server to remove unused keys. This minimizes the attack surface and ensures that only authorized personnel have access. Monitor server logs for failed authentication attempts, which may indicate brute-force attacks or misconfigured clients. Integrating these practices creates a robust and secure environment for managing your Windows infrastructure remotely.
Frequently Asked Questions
Can I use an .pem key file directly in FileZilla?
No, FileZilla does not natively support standard OpenSSH .pem or .key files. You must convert them to the PuTTY .ppk format using PuTTYgen before loading them into the Site Manager.
Why does FileZilla say "Invalid key file" even though the path is correct?
This error usually occurs when the key is in the wrong format or corrupted. Ensure you are loading a valid .ppk file generated by PuTTYgen. Standard OpenSSH keys require conversion to work with FileZilla’s underlying libraries.
How do I add my public key to the Windows Server?
You must place the content of your public key (ending in .pub) into the %USERPROFILE%\.ssh\authorized_keys file on the server. Ensure the file has no line breaks other than at the end of the key string and that the folder permissions are correct.
Is SFTP the same as FTP over TLS?
No. SFTP uses the SSH protocol for both control and data channels, while FTP over TLS (FTPS) uses explicit encryption over the traditional FTP protocol. FileZilla supports both, but key authentication is specific to SFTP/SSH.
What should I do if I lose access to my private key?
You will need to regain access through another method, such as the Windows Server console or RDP. Then, generate a new key pair, update the authorized_keys file, and delete the old public key to prevent future unauthorized access.