Why isn't systemd restarting simple service that has Restart=always after main process killed?

Systemd does not restart a simple service with Restart=always when the main process is killed by signals like SIGKILL or SIGTERM if the service is in an active state and the kill signal is sent directly to the process group rather than through systemd’s management interface. The daemon expects the process to exit cleanly or via specific failure conditions defined in the unit file, so external termination bypasses the restart logic unless configured otherwise.

Understanding Systemd Restart Logic

When you define Restart=always in a .service file, you are instructing systemd to restart the service regardless of the exit status or the signal that terminated the main process. However, this behavior assumes that systemd itself detects the termination. If you use tools like kill or pkill to send signals directly to the PID, systemd might not immediately recognize the change or might interpret it differently depending on the state of the service. The core issue often lies in how systemd monitors the process. For simple services, systemd tracks the main PID. When that PID dies, systemd checks the exit code or signal. If the signal is one that systemd considers "clean" or if the service was already marked as failed due to other reasons, the restart policy might not trigger as expected. It is crucial to understand that Restart=always is powerful but relies on systemd being the source of truth for the process lifecycle.

Common Misconceptions About Signals

A frequent mistake is assuming that sending SIGKILL (-9) will always prevent restart. This is incorrect. According to systemd documentation, Restart=always should restart the service even after SIGKILL. If it does not, the problem likely lies elsewhere, such as the service being masked, disabled, or in a failed state that requires manual intervention. Another misconception involves Restart=on-failure, which only restarts if the exit status indicates a failure (non-zero exit code or specific signals). Confusing these two directives leads to incorrect troubleshooting paths.

Diagnosing Why Restart Fails

To determine why your service is not restarting, you must first check the current status of the service using systemctl status your-service.service. Look for any error messages in the output that indicate why the restart did not occur. Common indicators include:

• Masked Service: If the service is masked, it cannot be started or restarted automatically. Check with systemctl is-enabled your-service.service.
• Failed State: Sometimes systemd marks a service as failed and requires a manual reset before it will attempt another start. Use systemctl reset-failed your-service.service to clear this state.
• Resource Limits: If the service is hitting memory or CPU limits defined in the unit file, it might be prevented from starting again until resources are freed.

Additionally, check the journal logs for detailed information about the termination event. Run journalctl -u your-service.service -e to see the most recent entries. Look for lines indicating the signal received and whether systemd acknowledged the death of the process.

Verifying Unit File Configuration

Ensure that your .service file correctly specifies the restart policy. The directive should be under the [Service] section:

[Service]
ExecStart=/path/to/your/program
Restart=always
RestartSec=5

If you have set Restart=no or omitted the directive entirely, systemd will not restart the service upon failure. Also, verify that RestartSec is set to a reasonable value to avoid rapid restart loops that might exhaust system resources.

Properly Terminating Services for Restart

If you need to restart a service manually, it is best practice to use systemd commands rather than sending raw signals. Using systemctl restart your-service.service ensures that systemd handles the lifecycle correctly, stopping the old process and starting a new one according to the defined policies. This method avoids the pitfalls of direct process termination. For testing purposes, if you want to simulate a crash, you can use systemctl kill --signal=SIGTERM your-service.service. This sends the signal through systemd, ensuring that the daemon is aware of the termination and can apply the restart policy appropriately. This approach is safer and more predictable than using external kill commands.

Handling Complex Dependencies

In some cases, other services or dependencies might prevent your service from restarting. Check if there are any ordering dependencies or socket activations involved. If your service depends on another service that is currently stopped, the restart might fail silently. Refer to Controlling Timing of Execution in *.service File with systemd for details on managing execution order and dependencies.

Advanced Troubleshooting Steps

If standard diagnostics do not reveal the issue, consider the following advanced steps:

1. Check SELinux/AppArmor: Security modules might be blocking the service from executing after a restart. Review audit logs for denials.
2. Verify File Permissions: Ensure that the executable and its configuration files have the correct permissions for the user running the service.
3. Review System Resources: High load or low disk space can prevent new processes from starting. Monitor system resources during the restart attempt.

For more insights on managing services, especially in specific distributions like Linux Mint, you might find Starting and Managing User Login Services with systemd on Linux Mint 21.2 helpful, as it covers nuances of service management in different environments.

Frequently Asked Questions

Why does systemctl restart work but killing the process does not?

Using systemctl restart explicitly tells systemd to stop and start the service, triggering all restart logic. Directly killing the process might bypass systemd’s monitoring if not done through the daemon interface, leading to inconsistent states where systemd does not initiate a restart.

How do I reset a failed service in systemd?

You can reset a failed service by running systemctl reset-failed your-service.service. This clears the failed state flag, allowing systemd to attempt starting or restarting the service normally based on its configuration.

What is the difference between Restart=always and Restart=on-failure?

Restart=always restarts the service regardless of the exit status or signal, while Restart=on-failure only restarts if the process exits with a non-zero exit code or receives certain fatal signals. Choose always for critical daemons that must remain running.

Can I prevent a service from restarting automatically?

Yes, you can set Restart=no in the [Service] section of your unit file. This disables automatic restarts, meaning the service will stay stopped after it terminates unless manually started again.

How can I see detailed logs for a service restart failure?

Use the command journalctl -u your-service.service -e to view the end of the log file for your service. This will show you the exact reason systemd rejected the restart attempt or why the service failed to start.