Restart=on-failure and Requires= dependencies enters a restart loop, you must break the cycle by stopping the failing unit and its dependents. Use systemctl stop to halt the service, then verify the state with systemctl status. This prevents the system from immediately retrying a broken configuration while you diagnose the root cause.
Understanding the Restart Loop
The behavior described in this guide occurs when a service is configured to automatically restart after a failure but depends on other units that are also unstable or misconfigured. The Restart=on-failure directive tells systemd to attempt a restart if the process exits with a non-zero exit code or is killed by a signal (except for specific signals like SIGTERM or SIGINT). Meanwhile, Requires=other.service creates a hard dependency, meaning if the required service fails, the dependent service should also fail.
This combination can lead to a "flapping" state where services continuously start, fail, and restart, consuming CPU resources and generating log spam. This is particularly problematic in production environments where stability is paramount. Recognizing this pattern is the first step toward resolving the issue without manually editing configuration files repeatedly.
Stopping the Failing Service and Dependencies
To stop the restart loop, you need to halt the affected service and ensure its dependencies are also stopped. This prevents systemd from attempting to restart them immediately. Use the following command to stop the primary service:
sudo systemctl stop my-service.serviceIf the service has dependencies defined via Requires= or Wants=, you may need to stop those as well. Systemd does not always stop dependent units automatically when you stop a parent unit, depending on the relationship type. To stop all related units, you can use:
sudo systemctl stop my-service.service dependency-one.service dependency-two.serviceAlternatively, you can check which units are pulling in your service using systemctl list-dependencies. This helps identify hidden dependencies that might be contributing to the loop.
Diagnosing the Root Cause
Once the services are stopped, investigate why they are failing. Check the journal logs for detailed error messages:
journalctl -u my-service.service -eLook for errors such as missing files, permission issues, or network timeouts. If the service requires another service that is also failing, check that service's logs as well. In some cases, the dependency might be timing out or crashing due to resource constraints. Reviewing the RestartSec= setting in the service file can also provide clues; if it is set too low, systemd may restart the service before the system has fully recovered from the previous failure.
For more advanced debugging, consider using systemd-analyze blame to see which services are taking the longest to start. This can help identify bottlenecks that contribute to failures. Additionally, checking the exit codes of the service can provide specific insights into the type of failure encountered.
Fixing the Configuration
After identifying the root cause, edit the service file to address the issue. Common fixes include correcting file paths, adjusting permissions, or modifying the Restart= policy. For example, changing Restart=on-failure to Restart=no disables automatic restarts, which can be useful during troubleshooting:
[Service]
Restart=no
ExecStart=/usr/bin/my-appOnce the configuration is updated, reload the systemd daemon to apply changes:
sudo systemctl daemon-reloadThen, start the service again to verify that it remains stable:
sudo systemctl start my-service.serviceIf the service starts successfully, monitor it for a few minutes to ensure it does not enter another restart loop. You can also adjust the RestartSec= value to introduce a delay between restart attempts, giving the system time to stabilize.
Preventing Future Loops
To prevent similar issues in the future, implement robust monitoring and alerting. Tools like Prometheus with Grafana can track service health and notify you of failures before they become critical. Additionally, consider using SuccessExitStatus= to specify exit codes that should not trigger a restart, reducing false positives.
Regularly review service configurations and dependencies to ensure they remain valid as your system evolves. Document any custom settings or workarounds to aid future troubleshooting. For more information on managing variables in service files, see Expanding the Use of Variables in a systemd Service File. This can help streamline configuration management and reduce errors.
Frequently Asked Questions
How do I check if a service is in a restart loop?
Use systemctl status my-service.service to view the current state. If the output shows repeated "Started" and "Failed" messages, it is likely in a loop. Checking the timestamp of the last few states can confirm rapid cycling.
What is the difference between Requires and Wants?
Requires= creates a hard dependency; if the required unit fails, the dependent unit also fails. Wants= is a soft dependency; the dependent unit will try to start the wanted unit but will continue even if it fails. Understanding this distinction is crucial for configuring stable service relationships.
Can I disable automatic restarts temporarily?
Yes, change Restart=on-failure to Restart=no in the service file, then run systemctl daemon-reload. This stops systemd from attempting to restart the service after a failure, allowing you to troubleshoot without interruption.
How do I find all dependencies of a service?
Run systemctl list-dependencies my-service.service to display all units that the service requires or wants. This hierarchical view helps identify potential sources of failure in the dependency chain.
What should I do if the service keeps failing after fixing the config?
Check the journal logs again for new errors. Ensure that any required external resources (like databases or network shares) are available. If the issue persists, consider increasing RestartSec= to allow more time for recovery between attempts.