When an Ansible playbook restarts the SSH daemon on a managed node, the connection often drops before subsequent tasks can execute. This article explains how to use the
wait_for module to pause execution until the service is fully operational again, ensuring reliable automation without manual intervention or connection timeouts.
The Challenge of Service Restart in Automation
Restarting critical network services like SSH presents a unique challenge for configuration management tools. When the
service or
systemd module issues a restart command, the process terminates and then initializes a new instance. During this brief window, the listening port is closed, causing the active SSH session used by Ansible to fail immediately. If the next task depends on that same connection, the entire play fails with a "Broken pipe" or "Connection refused" error.
This issue is particularly acute when managing remote infrastructure where physical access is unavailable. Unlike local scripts that might sleep for a few seconds, Ansible requires a programmatic way to detect readiness. Relying on static delays like
sleep is unreliable because system load varies, and the service might take longer to start under heavy disk I/O or memory pressure. Therefore, a dynamic check is necessary to ensure the service is actually accepting connections before proceeding.
Using the Wait For Module
The most robust solution is the
wait_for module, which allows you to wait for a condition to be met before continuing. Specifically, you want to wait for the SSH port (default 22) to become available. This module polls the target host at regular intervals until the specified condition is satisfied or a timeout occurs.
Here is a standard pattern for handling this scenario:
- name: Restart SSH service
ansible.builtin.service:
name: sshd
state: restarted
- name: Wait for SSH to come back online
ansible.builtin.wait_for:
host: "{{ inventory_hostname }}"
port: 22
delay: 5
timeout: 300
state: started
In this example, the first task restarts the service. The second task instructs Ansible to connect to
{{ inventory_hostname }} on port 22. It waits up to 300 seconds (5 minutes), checking every few seconds (default interval). The
delay parameter ensures Ansible waits five seconds after the restart command before starting the poll, giving the OS time to begin the initialization process.
Understanding Key Parameters
The
host parameter should usually be set to
{{ inventory_hostname }} or
{{ ansible_host }}. Using
127.0.0.1 will cause the module to check the *control* node's SSH service, not the target node's, leading to false positives if your control node has SSH running. Always verify the target address.
The
timeout value must be generous enough to account for slow boot times but short enough to fail fast if the service crashes entirely. A timeout of 300 seconds is generally safe for most Linux distributions. If you are restarting multiple nodes in parallel, consider reducing this slightly to avoid long hangs across the whole fleet.
Handling Connection Recovery Failures
Even with
wait_for, there is a race condition between the drop of the current connection and the establishment of the new one. In some cases, Ansible may lose the connection so quickly that it cannot even initiate the
wait_for task properly. To mitigate this, you can add a small initial sleep or rely on the
delay parameter effectively.
If the service fails to start within the timeout, Ansible will raise a fatal error. You can handle this gracefully by wrapping the recovery steps in a block with a rescue section. This allows you to log the failure or attempt a fallback strategy, such as rebooting the machine via IPMI or cloud provider APIs if SSH remains unreachable.
For more complex networking scenarios, such as using jump hosts, refer to our guide on Automating a menu presented by a remote SSH jump server. This context highlights additional complexity when managing connectivity through intermediate hops.
Optimizing for Parallel Execution
When restarting SSH on hundreds of servers simultaneously, the collective load on the network and the Ansible controller can be significant. Ensure that your
forks setting in
ansible.cfg is appropriate. If you are using connection pooling, note that restarting SSH breaks existing persistent connections. You may need to clear the cache or expect Ansible to re-establish them during the next task.
Additionally, consider using the
async and
poll parameters for the restart task itself if you want to fire-and-forget the restart command while waiting for health checks elsewhere. However, for most single-node operations, the sequential approach shown above is clearer and easier to debug.
Alternative Approaches and Considerations
Some administrators prefer using the
uri module to hit an HTTP endpoint instead of waiting for a TCP port. This is only viable if you have a web server or health check API running alongside SSH. For pure SSH recovery, TCP port checking via
wait_for is the standard and most reliable method.
Another consideration is the security implications of restarting SSH. Ensure that your firewall rules allow the incoming connection from the Ansible controller during the restart window. If you are using strict iptables or firewalld rules, they must not block the port during the service transition.
For related automation challenges, see
Can Ansible Reuse SSH Connection for Rsync? to understand how connection persistence affects other file transfer operations after a service restart.
Frequently Asked Questions
Why does my Ansible playbook fail immediately after restarting SSH?
The failure occurs because the active SSH session is terminated when the service stops. Since Ansible relies on that specific socket for communication, any subsequent task sent over that channel results in a broken pipe error before the new service instance is ready.
Can I use the sleep module instead of wait_for?
While you can use a simple sleep command, it is not recommended. Sleep introduces a fixed delay regardless of system load, meaning your playbook runs slower than necessary on fast systems or fails on slow ones. The wait_for module dynamically checks for readiness, providing a more efficient and reliable recovery mechanism.
What happens if the SSH service fails to start within the timeout?
If the service does not respond within the specified timeout period, the wait_for module raises a fatal error, terminating the playbook run. This prevents silent failures where subsequent tasks execute against a non-functional system, allowing you to investigate the root cause via logs or alternative access methods.
Do I need to specify the host parameter in wait_for?
Yes, you must explicitly set the host parameter to the target node's IP or hostname. If omitted, Ansible defaults to localhost, which checks the control node's status rather than the managed node's, leading to incorrect health checks and potential false positives.
How does this affect connection caching in Ansible?
Restarting SSH invalidates any existing persistent connections stored in the control node's connection cache. Ansible will need to establish a new SSH handshake once the service is back online. This adds a slight overhead to the first task executed after recovery compared to tasks using cached connections.