Connection to Globalscape EFT ssh server from RHEL8 fails with "Permission denied (publickey)"

The "Permission denied (publickey)" error on RHEL8 when connecting to Globalscape EFT indicates that the SSH client cannot authenticate using the expected key pair. This is typically resolved by verifying that the correct private key is specified in the ssh command or configuration file, ensuring the corresponding public key is properly installed in the server's authorized keys list, and confirming that the SSH daemon on the Windows host accepts RSA or ED25519 keys as defined by its security policy.

Understanding the Authentication Failure

When an SSH connection attempt fails with a publickey rejection, the server has received your identity but explicitly refused it. Unlike password authentication failures, this error means the cryptographic handshake did not find a matching credential. In the context of Globalscape Enterprise File Transfer (EFT), which runs on Windows Server 2016, the SSH service is often configured to allow multiple authentication methods, but the client must present the exact key type and content that the server expects. RHEL8 uses OpenSSH by default, which may have stricter defaults regarding key algorithms than older Linux distributions. If the Globalscape EFT server is configured to reject weaker keys (such as DSA) or specific key formats, the RHEL8 client will fail before even prompting for a passphrase. It is crucial to distinguish between a network connectivity issue and an authentication failure; since the error is explicit about "publickey," the network path is likely functional, but the credentials are mismatched or missing. For more details on managing SSH connections securely, you might find Closing SSH Connection with Keygen helpful for understanding how keys are generated and managed locally.

Verifying Client-Side Key Configuration

The most common cause of this error on RHEL8 is specifying the wrong key file or having permissions issues on the local key files. You must ensure that the private key you intend to use is accessible and correctly referenced. First, check the permissions of your private key file. SSH requires strict permissions; if the file is readable by others, the client will ignore it silently or throw a warning. Use the following command to set the correct permissions:

chmod 600 ~/.ssh/id_rsa

Next, verify that you are invoking the SSH client with the correct identity file. If you are using a non-default key name, you must specify it explicitly:

ssh -i /path/to/private_key username@eft_server_ip

If you do not specify the key, SSH will try ~/.ssh/id_rsa, ~/.ssh/id_ecdsa, etc., in order. If your Globalscape EFT account is linked to a different key, the default attempt will fail with "Permission denied." You can also configure this permanently in your ~/.ssh/config file to avoid typing the flag every time. Additionally, ensure that the key format is compatible. Modern versions of OpenSSH on RHEL8 may refuse to use keys that are encrypted with legacy formats or are too short (e.g., 1024-bit RSA). If you recently generated a new key, consider generating an ED25519 key, which is more secure and widely supported by modern servers like Globalscape EFT:

ssh-keygen -t ed25519

Checking Globalscape EFT Server Settings

On the Windows side, the Globalscape EFT Administrator console controls which users are allowed to connect via SSH and which keys they possess. The error persists if the public key associated with your RHEL8 private key is not registered in the user's profile within EFT. Navigate to the User Management section in the Globalscape EFT Administrator interface. Locate the specific user account you are trying to access. Within the user's properties, look for the SSH Keys or Authorized Keys tab. Here, you must paste the content of your RHEL8 public key (~/.ssh/id_ed25519.pub or id_rsa.pub). Ensure that the key is saved and that the user account is enabled for SSH access. Some configurations require you to explicitly select "Allow SSH" for the user. If the key was added recently, restart the Globalscape EFT Service to ensure the changes are loaded into memory. It is also worth checking the global server settings for allowed key types. If the server is configured to only accept ECDSA keys, but you are sending an RSA key, the connection will be denied. Review the server's security policies to align them with your client's capabilities. For insights on debugging other connection protocols, see Troubleshooting curl not working on an RHEL8 Server.

Debugging with Verbose Output

If the above steps do not resolve the issue, use the verbose mode of the SSH client to see exactly where the negotiation fails. Run the connection command with the -v flag, increasing verbosity up to three times (-vvv) for detailed logs:

ssh -vvv -i ~/.ssh/id_ed25519 username@eft_server_ip

Look for lines indicating "Offering public key" followed by "Server accepts key." If the server rejects the offered key, the log will show why (e.g., "wrong key length" or "algorithm mismatch"). If no key is offered, the client may not be finding the key file at all, pointing back to permission or path issues. You can also query your effective SSH configuration to ensure no global settings are overriding your user-specific config:

ssh -G username@eft_server_ip | grep identityfile

This command outputs the final list of keys the client will try, helping you confirm that your intended key is included in the search path.

Firewall and Network Considerations

While less likely to cause a "publickey" error specifically, firewall rules on the Windows Server 2016 host or intermediate network devices might interfere with the SSH handshake if deep packet inspection is active. Ensure that port 22 (or your custom SSH port) is open and allows TCP traffic from your RHEL8 IP address. If you are using a proxy or jump host, ensure that the SSH tunneling is configured correctly so that the key exchange happens directly between the RHEL8 client and the EFT server. For information on verifying network port allocations that might affect such connections, refer to How to Verify the Dynamic Ports Allocation Settings in Linux.

Frequently Asked Questions

Why does my RHEL8 client reject the key I just created?

This usually happens due to incorrect file permissions. SSH requires private keys to be readable only by the owner (mode 600). If the permissions are too open, OpenSSH will refuse to use the key entirely, leading to authentication failure.

Can I use password authentication instead of keys?

Yes, if the Globalscape EFT server is configured to allow it. However, many enterprise setups disable passwords for security. You can force password auth on the client side using ssh -o PreferredAuthentications=password, but the server must permit this method for the user.

What if the key is accepted but I still get disconnected?

This may indicate a session timeout or a mismatch in encryption algorithms. Check the server logs in Globalscape EFT for specific rejection reasons. Ensure both the client and server support strong ciphers like AES-GCM, as older algorithms may be disabled by default on RHEL8.

Does the user account need special privileges in Windows?

No, the Windows user account does not need administrator rights to receive SSH files. It only needs a valid home directory configured in Globalscape EFT and the appropriate permissions to read their own authorized keys file.

How do I reset my SSH key if I lost the private one?

You must generate a new key pair on RHEL8 and update the public key in the Globalscape EFT Administrator console. Delete the old public key entry from the user's profile to prevent confusion, then test the connection with the new private key.