Systemd Service Timing Out/Restarting Every Few Minutes

Understanding Systemd Timeout Mechanisms

Systemd monitors services using several mechanisms that can inadvertently trigger restarts. The most common culprit for periodic restarts is the interaction between the service type and its configured timeouts. When a service starts, systemd waits for it to initialize. If the initialization takes longer than the specified limit, systemd assumes the service has failed and kills it, potentially restarting it depending on the Restart= policy.

The key parameters involved are TimeoutStartSec, which defines how long systemd waits for the main process to start, and TimeoutStopSec, which governs the shutdown phase. Additionally, if you are using Type=notify, systemd expects a specific readiness signal from the application. If this signal is not sent within the timeout window, systemd considers the startup incomplete. It is crucial to distinguish between a service that crashes (exit code non-zero) and one that simply hangs during startup. In both cases, the result may look like a restart loop, but the underlying cause differs significantly.

Checking Current Service Status

Before modifying configuration files, verify the exact state of the service. Use the following command to see recent logs and exit codes:

journalctl -u your-service-name --since "10 min ago" -f

Look for messages indicating Failed with result 'timeout'. This specific error message confirms that systemd killed the process due to exceeding time limits rather than an application crash. If you see segmentation faults or other errors, the issue lies within the application itself, not the systemd configuration.

Adjusting Timeout Values

If diagnostics confirm that the service is healthy but systemd is killing it prematurely, the solution is to increase the timeout values. You should edit the service unit file directly or create a drop-in override. Editing the original file is discouraged as updates may overwrite your changes; instead, use the systemctl edit command.

1. Open the editor for the specific service: sudo systemctl edit your-service-name.service
2. Add or modify the timeout sections under the [Service] header.
3. Set TimeoutStartSec to a value sufficient for your application to initialize. For example, TimeoutStartSec=300 allows five minutes.
4. Similarly, adjust TimeoutStopSec if the service requires significant time to flush data or close connections before exiting.

After saving the changes, reload the systemd daemon and restart the service to apply the new settings:

sudo systemctl daemon-reload
sudo systemctl restart your-service-name.service

This approach ensures that your customizations persist across package updates and clearly separate your operational tweaks from the default system configuration.

Configuring Readiness Signals

For modern applications, relying solely on process existence is often insufficient. If your service uses Type=notify, it must send a readiness notification via the sd_notify library. If the application does not support this natively, consider switching to Type=simple or Type=forking, depending on how the application manages its processes.

When using Type=simple, systemd considers the service started as soon as the main process launches. This bypasses the need for explicit readiness signals but removes the guarantee that the application is actually ready to handle requests. Conversely, Type=forking is suitable for daemons that fork into the background, where systemd tracks the parent PID. Ensure that the chosen type matches the behavior of your application to prevent false-positive timeout errors. For more details on controlling execution timing, you might find Controlling Timing of Execution in *.service File with systemd helpful.

Investigating Watchdog Settings

Another potential cause for unexpected restarts is the watchdog mechanism. If WatchdogSec is set, systemd sends periodic ping signals to the service. If the service fails to respond within the specified interval, systemd kills and restarts it. Check if this directive is active in your unit file.

grep -i watchdog /etc/systemd/system/your-service-name.service

If you do not intend to use the watchdog, remove or comment out the WatchdogSec line. Alternatively, ensure your application implements the watchdog protocol correctly. Some frameworks have built-in support for sd-watchdog, while others require external wrappers. Misconfigured watchdogs are a frequent source of intermittent restart loops that are difficult to diagnose without checking the journal logs closely.

Handling Restart Policies

The Restart= directive determines how systemd behaves after a failure. Common values include always, on-failure, and on-abnormal. If your service exits cleanly but systemd still restarts it, check if the exit status is considered successful. Sometimes, applications exit with non-standard codes that systemd interprets as failures.

You can also use RestartSec to introduce a delay between restart attempts. This prevents rapid restart loops that can overwhelm system resources. Setting RestartSec=10 adds a ten-second pause before each retry, giving the system time to stabilize. This is particularly useful when debugging transient issues that resolve themselves after a brief cooldown period.

Common Pitfalls and Verification

Ensure that the user and group permissions in the service file match the requirements of the application. Incorrect permissions can cause the service to fail silently during startup, leading to timeouts. Additionally, verify that any required environment variables are correctly passed to the service context. Missing dependencies or incorrect paths can also cause initialization delays that exceed the default timeout.

After applying fixes, monitor the service for at least 24 hours to ensure stability. Use systemctl status your-service-name.service to verify that the uptime is increasing and no restart events are recorded. Consistent monitoring helps identify if the issue was fully resolved or if there are secondary factors contributing to the instability.

Frequently Asked Questions

Why does my service restart immediately after I change the timeout?

Changing the timeout value does not automatically stop a currently running instance. You must manually restart the service using systemctl restart after editing the configuration. Additionally, ensure you ran daemon-reload so systemd recognizes the updated unit file definitions.

Can I disable the restart loop entirely without fixing the root cause?

While you can set Restart=no to prevent automatic restarts, this is generally not recommended. Disabling restarts masks the underlying problem, such as a hang or crash, making future troubleshooting more difficult. It is better to identify why the service is failing and address that specific issue.

What is the difference between TimeoutStartSec and WatchdogSec?

TimeoutStartSec applies only during the initial startup phase, waiting for the service to become ready. WatchdogSec operates continuously after startup, pinging the service periodically to ensure it remains responsive. Confusing these two can lead to misdiagnosis of whether an issue occurs at launch or during runtime.

How do I check if my application supports sd_notify?

Consult the documentation of the application framework you are using. Many modern web servers and databases support sd_notify natively. If unsupported, you may need to write a wrapper script or switch to Type=simple to avoid dependency on readiness signals.

Is it safe to increase TimeoutStartSec to infinity?

Setting TimeoutStartSec=infinity disables the timeout, which can be risky if the service hangs indefinitely. It prevents systemd from intervening, potentially leaving the system in a broken state until manual intervention. A large but finite value, such as 300 seconds, is usually a safer balance between allowing slow startups and maintaining system responsiveness.