Restart= directive in the unit file. Setting Restart=always ensures the daemon attempts to recover from any exit code, while Restart=on-failure targets non-zero exits and signals. This mechanism is essential for high-availability applications where automatic recovery prevents manual intervention during unexpected crashes.
Understanding Restart Policies
Systemd provides several policies to determine when a service should be restarted after it stops. The most common setting for indefinite recovery is Restart=always. When this is active, Systemd will restart the service regardless of how it exited, whether due to a successful termination (exit code 0) or an error. This is ideal for daemons that are expected to run continuously and should never stop voluntarily.
Another robust option is Restart=on-failure. This policy restarts the service only if it exits with a non-zero status code or is terminated by a signal other than SIGTERM, SIGKILL, or SIGRTMIN. Use this when you want to preserve logs of intentional stops but automatically recover from crashes. It is often safer than always because it avoids restarting services that have been deliberately stopped for maintenance.
For scenarios involving rapid failure loops, Restart=on-abnormal is useful. It triggers a restart only if the process exits due to an uncaught signal, timeout, or cgroup export failure. This helps distinguish between a clean shutdown and a catastrophic event requiring immediate attention.
Configuring the Service Unit File
To implement these policies, edit the service unit file located in /etc/systemd/system/ or /usr/lib/systemd/system/. Add or modify the [Service] section to include the desired restart behavior. Below is a minimal example configuration:
[Unit]
Description=My Critical Application
After=network.target
[Service]
ExecStart=/usr/bin/my-app --config /etc/my-app.conf
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.targetThe RestartSec parameter is critical for preventing resource exhaustion. It defines the delay in seconds before Systemd attempts to restart the service. Without this, a crashing application could enter a tight loop, consuming CPU cycles and potentially overwhelming the system. A value of 5 or 10 seconds is typically sufficient to allow temporary issues to resolve themselves.
If your application relies on specific ordering during boot or shutdown, ensure that dependencies are correctly defined in the [Unit] section. Proper ordering guarantees that required resources are available before the service starts, reducing the likelihood of startup failures that would trigger unnecessary restarts. For more details on managing service dependencies, see Preserve Order of Closing Services While Shutting Down Linux.
Handling Restart Loops and Backoff
In some cases, a service may fail repeatedly due to a persistent configuration error or missing dependency. Systemd includes mechanisms to handle such "restart storms." By default, Systemd uses exponential backoff logic when a service fails multiple times in quick succession. If a service fails frequently, Systemd increases the delay between restart attempts.
You can fine-tune this behavior using StartLimitBurst and StartLimitIntervalSec in the [Unit] section. These directives limit the number of restarts allowed within a specified time window. If the limit is exceeded, Systemd stops attempting to restart the service until the interval resets. This prevents the system from being locked into a perpetual failure state.
[Unit]
StartLimitBurst=5
StartLimitIntervalSec=60This configuration allows up to five restarts within one minute. After that, Systemd waits for the interval to expire before allowing new restart attempts. This approach balances availability with system stability, ensuring that transient errors are handled without masking deeper issues.
Diagnostics and Monitoring
When debugging restart issues, use systemctl status <service> to view recent logs and exit codes. Look for patterns in the journal logs using journalctl -u <service> -f to observe real-time behavior. Identifying the root cause of the failure is more important than simply configuring restarts, as infinite retries do not fix underlying bugs.
Consider implementing health checks within your application to detect internal states that might lead to crashes. Logging detailed error messages at startup can also help identify misconfigurations early. For services that require precise timing control during execution, refer to Controlling Timing of Execution in *.service File with systemd to optimize startup sequences.
Frequently Asked Questions
What happens if I set Restart=always but the service crashes immediately?
Systemd will attempt to restart the service immediately, but RestartSec introduces a mandatory delay. Without this delay, the service could consume all available CPU resources in a crash loop. Always define RestartSec to provide a buffer for the system to stabilize.
How can I prevent a service from restarting after a deliberate stop?
Use Restart=on-failure instead of always. This ensures that if you manually stop the service using systemctl stop, it will not automatically restart unless it exits with an error code or receives a terminating signal.
Can I combine different restart policies for different conditions?
No, a single unit file supports only one Restart= directive. You must choose the policy that best fits your application's behavior. If you need complex logic, consider wrapping the main executable in a script that handles its own restart logic.
Why is my service stuck in a failed state despite having Restart=always?
This usually occurs when StartLimitBurst is reached. Check the output of systemctl status for messages indicating that the start limit has been exceeded. Adjusting StartLimitIntervalSec or increasing StartLimitBurst can resolve this issue.
Does Restart=always work for services managed by Docker?
Yes, but it is often better to configure restart policies within the Docker container itself using --restart=always. Systemd will still manage the container lifecycle, but Docker handles the internal retry logic more efficiently for containerized workloads.