Troubleshooting SSH with Private Key: Forced Password Entry during Ceph Cluster Setup

Forced password entry during Ceph cluster setup using ceph-ansible typically occurs when the SSH agent does not hold the private key, file permissions are too restrictive, or the target host is missing in the known_hosts file. Verifying that the key has correct ownership and permissions, ensuring it is loaded into the ssh-agent, and confirming network connectivity can resolve this authentication failure without disabling security settings.

Understanding the Authentication Flow

When you run ceph-ansible, it relies on standard SSH mechanisms to distribute configuration files and execute commands across your cluster nodes. The tool itself does not bypass these mechanisms; it expects a seamless, passwordless connection established via public-key cryptography. If the SSH daemon on the target node rejects the provided key, it falls back to password authentication if configured to do so, or fails entirely if passwords are disabled. This fallback behavior is often what users perceive as an "unexpected" prompt, especially when they believe their key should work automatically. The most common culprit is that the SSH client does not have access to the private key at the moment of connection. Even if the key file exists on disk, the SSH protocol requires the client to prove possession of the corresponding private key. This proof is generated locally by the SSH client software. If the key is encrypted with a passphrase, the client must be able to decrypt it before attempting the handshake. If the decryption fails or the key is not available in memory, the connection cannot proceed using public-key authentication.

Verifying Key Permissions and Ownership

Linux systems enforce strict security policies regarding private keys. If the permissions on your private key file are open to other users, SSH will refuse to use it because it considers the key insecure. You must ensure that only the user running the ceph-ansible command has read and write access to the key file. You can check the current permissions using the ls -l command. The output should show permissions similar to -rw------- (600) or -r-------- (400). If you see any group or other read/write bits set, such as -rw-r--r--, you need to correct them immediately. Use the following command to restrict permissions:

chmod 600 ~/.ssh/id_rsa

Additionally, verify that the file is owned by the correct user. If you switched users or copied the key from another account, the ownership might still belong to root or another user. Incorrect ownership can also cause SSH to reject the key silently or fall back to password prompts. Use chown to fix ownership if necessary:

chown $USER:$USER ~/.ssh/id_rsa

These checks are foundational. Without proper permissions, no amount of agent management will allow the key to function securely. For more details on managing multiple keys and agents, see SSH Key Chain: Linking SSH-B to C.

Managing the SSH Agent

Even with perfect permissions, the SSH client needs to know which key to offer. Modern Linux distributions use an SSH agent to store decrypted keys in memory. When you connect to a remote host, the SSH client asks the agent for a key. If the agent is empty or does not contain the specific key required for the Ceph nodes, authentication will fail. First, ensure the agent is running. You can start it manually or rely on your desktop environment's startup scripts. To add your key to the agent, use the ssh-add command:

ssh-add ~/.ssh/id_rsa

If the key is protected by a passphrase, you will be prompted to enter it here. Once added, verify that the key is present in the agent by listing its contents:

ssh-add -l


This list should display the fingerprint of your private key. If it is missing, the agent cannot authenticate on your behalf. Note that if you are using a different user context for ceph-ansible, you must add the key to that user's agent session.

Checking Host Key Verification

A less obvious cause for authentication issues is the absence of the remote host's public key in your local known_hosts file. While this usually results in a warning rather than a password prompt, some configurations or older SSH clients may behave differently. If you recently rebuilt your Ceph nodes or changed their hostnames, the old keys might be conflicting with new ones.

Check your ~/.ssh/known_hosts file for entries related to your Ceph cluster IPs. If you suspect corruption or conflicts, you can remove the specific entry using ssh-keygen -R hostname. This forces SSH to accept the new host key upon the next connection, establishing a fresh trust relationship.

For cases where name resolution issues interfere with SSH connections, consider reviewing SSH: Temporary Name Resolution Failure for 'desktop11900' to ensure DNS or /etc/hosts entries are correctly pointing to your cluster nodes.

Testing the Connection Manually

Before re-running the entire ceph-ansible playbook, test the SSH connection manually using verbose mode. This provides detailed output about exactly where the authentication process fails. Run the following command, replacing user and host with your actual credentials:

ssh -v user@host


Look for lines indicating "Offering public key" and "Server accepts key." If you see "Authentications that can continue: password," it means the server did not accept your key. Check the server-side logs in /var/log/auth.log or /var/log/secure for reasons why the key was rejected, such as incorrect username or disabled key authentication in /etc/ssh/sshd_config.

If you encounter generic connection errors during the initial handshake, refer to Troubleshooting kex_exchange_identification Error in Linux SSH Connection to rule out network-level disruptions.

Frequently Asked Questions

Why does SSH ask for a password even though I added the key?
This usually happens because the SSH agent does not contain the key, or the key file permissions are too open. Ensure you run ssh-add to load the key into memory and verify that the file permissions are set to 600. Additionally, confirm that the username you are connecting with matches the user who owns the authorized_keys file on the remote server.

Can I disable password authentication to prevent prompts?
While you can disable password authentication in /etc/ssh/sshd_config on the remote servers, this is risky if your key setup is not yet verified. It is safer to troubleshoot why the key is failing first. Disabling passwords prematurely can lock you out of your cluster nodes if the key-based method is not working correctly.

What if my private key has a passphrase?
If your key is passphrase-protected, you must enter the passphrase when adding it to the SSH agent using ssh-add. The agent will then cache the decrypted key for the duration of your login session. If you restart your computer, you will need to add the key again unless you configure persistent agent forwarding or use a keychain utility.

How do I check if the remote server accepts public keys?
Log into the remote server and check the /etc/ssh/sshd_config file. Ensure that PubkeyAuthentication yes is present and not commented out. Also, verify that the AuthorizedKeysFile directive points to the correct location, typically .ssh/authorized_keys in the user's home directory.

Is it safe to use the same key for all Ceph nodes?
It is common practice to use a single key pair for cluster automation tools like ceph-ansible for simplicity. However, for enhanced security, you can generate unique keys for each node and manage them using an SSH agent or a configuration management tool. The trade-off is between convenience and granular access control.