Preserve Order of Closing Services While Shutting Down Linux

Systemd manages shutdown by reversing the start-up dependency graph, ensuring services stop in the correct order. You can preserve this order by relying on standard unit directives like After= and Requires=. If specific timing is critical, use TimeoutStopSec= to prevent premature termination of dependent processes.

Understanding Systemd Shutdown Logic

When you issue a shutdown or reboot command, systemd does not simply kill all running processes at once. Instead, it initiates a structured transition to the shutdown.target. This target triggers a reverse execution of the service activation sequence. The system looks at the dependencies defined in each service unit file and stops them in an order that respects those relationships. For example, if Service A depends on Database B, systemd will ensure Database B is stopped only after Service A has acknowledged the shutdown signal or timed out. This mechanism relies heavily on the concept of "activation targets." During boot, services are started to reach multi-user.target. During shutdown, systemd works backward from the current state to poweroff.target. Understanding this flow is essential for troubleshooting why a service might appear to hang during shutdown or why another service fails because its dependency was killed too early. If you have encountered issues with user sessions terminating unexpectedly, reviewing how PAM sessions interact with systemd units can provide additional context on session cleanup.

Key Directives for Order Preservation

The primary way to enforce order is through the [Unit] section of your service files. The After= directive is the most critical for ordering. It tells systemd that this unit should be started (or stopped) after the specified unit. Note that After= only affects ordering, not whether the unit is actually started. To ensure a dependency exists, you must combine it with Requires= or Wants=. For instance, if your web server needs the database to remain available until it gracefully closes connections, you would define:

[Unit]
Description=My Web Server
After=postgresql.service
Requires=postgresql.service

During shutdown, systemd sees that the web server is listed as After the PostgreSQL service. Therefore, it will send the stop signal to the web server first. Once the web server confirms it has stopped (or times out), systemd proceeds to stop PostgreSQL. This ensures that no new database connections are accepted while existing ones are being drained by the web server.

Handling Graceful Shutdowns and Timeouts

Even with correct dependency ordering, some services may take longer than expected to shut down. If a service hangs, it can block the entire shutdown process. Systemd handles this by enforcing timeouts. By default, systemd waits 90 seconds for a service to stop before sending a SIGKILL signal. However, you can customize this behavior using the TimeoutStopSec= directive in the [Service] section. Setting an appropriate timeout allows your application time to perform cleanup tasks, such as writing logs or closing sockets. If your application is known to take up to 30 seconds to flush data, setting TimeoutStopSec=45 provides a safe buffer. If the service does not exit within this window, systemd escalates to forceful termination. This prevents the system from hanging indefinitely during reboot. It is also important to note that signals sent during shutdown differ slightly from those sent during runtime. Systemd sends SIGTERM by default, which allows for graceful handling. If your application does not handle SIGTERM correctly, it may ignore the signal and continue running, leading to the eventual SIGKILL. Ensuring your application code properly traps and handles these signals is crucial for maintaining order.

Diagnosing Shutdown Issues

If services are stopping in the wrong order, the first step is to verify the dependency graph. You can use the systemctl show command to inspect the dependencies of a specific unit. Look for the After= and Requires= fields to ensure they match your intended architecture. Another common issue arises when services are not properly integrated into systemd. If a service is started manually via a cron job or a script rather than through a systemd unit, it will not be managed by the shutdown process. Such processes may be orphaned or terminated abruptly. Always ensure that long-running daemons are managed by systemd units to benefit from the ordered shutdown mechanism. In cases where the shutdown process seems stuck, checking the journal logs can reveal which service is delaying the transition. Use journalctl -b -1 to view logs from the previous boot cycle, specifically looking for messages related to shutdown.target. This can help identify if a specific service is failing to acknowledge the stop signal. Additionally, understanding how user units and PAM sessions terminate can help diagnose issues where user-space applications interfere with the shutdown sequence.

Best Practices for Service Configuration

To maintain a robust shutdown process, follow these best practices:

• Use Explicit Dependencies: Never rely on implicit ordering. Always define After= and Requires= explicitly in your unit files.
• Set Reasonable Timeouts: Configure TimeoutStopSec= based on your application's actual shutdown duration to avoid unnecessary delays or premature kills.
• Handle Signals Correctly: Ensure your application code listens for SIGTERM and performs necessary cleanup before exiting.
• Avoid Manual Start/Stop: Run all persistent services through systemd units to ensure they are included in the shutdown graph.
• Test Changes: After modifying unit files, always test the shutdown process in a non-production environment to verify the order of operations.

By adhering to these guidelines, you can ensure that your Linux system shuts down smoothly and predictably, preserving the integrity of your services and data.

Frequently Asked Questions

What happens if a service ignores SIGTERM?

If a service ignores the initial SIGTERM signal, systemd will wait for the configured TimeoutStopSec duration. After this period expires, systemd sends a SIGKILL signal, which cannot be caught or ignored by the application. This forces the process to terminate immediately, potentially leading to data loss if cleanup was not performed.

Can I change the global shutdown timeout?

Yes, you can modify the global timeout for all services by editing the /etc/systemd/system.conf file. Change the value of DefaultTimeoutStopSec to your desired number of seconds. This applies to all services that do not have a specific TimeoutStopSec defined in their unit file.

How do I check the order in which services are stopping?

You can view the shutdown sequence by examining the journal logs during the shutdown process. Use journalctl -b -1 | grep shutdown to see entries related to the previous shutdown. Additionally, systemctl list-dependencies --reverse shows which units depend on a specific service, helping you understand the stop order.

Why did my custom script fail to run during shutdown?

Custom scripts are not automatically part of the systemd shutdown graph unless they are defined as a service unit. If you need to run a script during shutdown, create a systemd service unit that uses ExecStop=/path/to/script.sh and ensure it is enabled and added to the appropriate target, such as shutdown.target.wants.

Does disabling a service affect shutdown order?

Disabling a service prevents it from starting automatically at boot, but it does not remove its dependencies from other active services. If another service still requires the disabled service, systemd will attempt to start it temporarily during shutdown if needed, or the dependent service may fail due to missing resources. Always review dependencies before disabling critical services.