Scripts in /lib/systemd/system-shutdown execute during the final stages of system shutdown, after all services have stopped. To verify they are running, check the journal logs for specific exit codes using systemctl status, inspect the /run/systemd/shutdown directory for temporary files created by your scripts, and use journalctl -b -1 to review logs from the previous boot cycle where these shutdown events occurred.
Understanding the Shutdown Sequence
The systemd initialization system manages the lifecycle of processes on Linux systems. When you initiate a shutdown, systemd does not simply kill all processes; it follows a structured sequence designed to ensure data integrity and proper resource release. The directory/lib/systemd/system-shutdown contains scripts that are intended to run after the main system has entered the shutdown target but before the kernel actually halts or reboots. These scripts operate in a minimal environment where many standard utilities might be unavailable or behave differently because the root filesystem may be mounted read-only or unmounted entirely depending on the timing.
It is crucial to understand that these scripts are executed synchronously. If a script hangs or waits for user input, the system will hang indefinitely during shutdown. This behavior makes verification critical, as a failing script can appear as if the system itself is frozen rather than the script being the culprit. The scripts must be executable and typically should return an exit code of zero to indicate success. Any non-zero exit code is logged, but systemd generally continues with the shutdown process unless the script explicitly causes a fatal error in the shutdown manager itself.
For more context on how systemd handles service ordering during complex transitions, you might find it helpful to read about preserving order of closing services while shutting down Linux. This knowledge helps in understanding why your script might be placed where it is in the execution timeline relative to other system components.
Checking Journal Logs for Execution Evidence
The most reliable way to confirm that a script in/lib/systemd/system-shutdown ran is to examine the system journal. Because the system is shutting down, the current journal session ends abruptly. Therefore, you cannot always see the output in real-time on the screen that displays the shutdown messages. Instead, you must look at the logs from the previous boot cycle.
To do this, reboot the system and then open a terminal. Use the command journalctl -b -1 to view logs from the previous boot. The -b flag specifies the boot ID, and -1 refers to the boot immediately preceding the current one. You can filter these logs for your specific script name or for the general shutdown target. Look for entries related to systemd-shutdown or the specific name of your script.
If your script produces output via echo or logger, those messages will appear in the journal under the context of the shutdown process. For example, if you have a script named my-cleanup.sh, searching for "my-cleanup" in the previous boot's journal will show if it was invoked. Additionally, checking the exit status is vital. Systemd logs the exit status of each unit. If your script failed, you will see a line indicating a failure with a non-zero exit code.
Another useful diagnostic step is to use journalctl -t systemd-shutdown. This filters the log to show only messages generated by the shutdown manager itself. While this might not show every detail of your custom script, it confirms that the shutdown phase was reached and processed. If you suspect race conditions or issues with subprocesses, reviewing articles on binary vs shell systemd subprocess issues can provide deeper insight into how different execution contexts behave during these low-level operations.
Using Logger for Persistent Messages
Standard output from scripts in this directory might be lost if the logging daemon shuts down before your script runs. To ensure your messages persist, use thelogger command within your script. For instance, adding logger -t my-script "Starting cleanup" ensures that the message is sent to the syslog/journal regardless of whether stdout is still connected to a terminal. This technique is particularly effective for debugging because it creates a distinct tag in the journal, making it easy to grep for later.
Creating Temporary Files for Verification
Sometimes, journal access is limited, or you want a physical confirmation that the script executed. A simple and robust method is to create a temporary file in a location that persists across reboots, such as/tmp or a dedicated directory like /var/log/shutdown-checks. Inside your script, add a command like touch /tmp/script-executed-$(date +%s).
After the system shuts down and restarts, check if the file exists. The presence of the file proves that the script ran. Using a timestamp in the filename prevents overwriting and allows you to correlate the file creation time with the shutdown event. This method is foolproof because it relies on the filesystem state rather than volatile memory logs. However, be aware that if the root filesystem is remounted read-only before your script runs, you may need to write to a tmpfs mount that is guaranteed to be available, or use /dev/shm if accessible.
Debugging Common Failures
Scripts in/lib/systemd/system-shutdown often fail due to missing dependencies or incorrect permissions. Ensure the script has the executable bit set using chmod +x /lib/systemd/system-shutdown/your-script.sh. Also, verify that any commands called within the script are available in the minimal environment provided during shutdown. For example, some network tools might not be loaded yet, or paths might differ.
If your script depends on other services stopping first, ensure there are no circular dependencies or timeouts causing premature termination. Checking custom systemd timers or service failures can sometimes reveal broader configuration issues that affect shutdown reliability. If you encounter baffling differences between executing binaries versus shell scripts, refer to guides on subprocess issues to understand environment variable inheritance and path resolution differences.
Finally, remember that changes to scripts in /lib/systemd/system-shutdown take effect immediately on the next shutdown; no daemon reload is required. However, if you move scripts to /etc/systemd/system-shutdown, which overrides the default directory, you must ensure the override directory exists and is properly configured. Always test in a safe environment where data loss is acceptable, as improper shutdown scripts can lead to filesystem corruption.
Frequently Asked Questions
Why can't I see the output of my shutdown script on the screen?
The display console often switches to a lower priority or disconnects before your script executes. Systemd redirects standard output to the journal, so you must check journalctl instead of relying on visual feedback. This design prevents screen clutter during the critical final moments of shutdown.
What happens if my shutdown script exits with a non-zero code?
Systemd logs the failure but typically continues the shutdown process unless the script triggers a critical error condition. It is best practice to ensure your script exits with code 0 to indicate successful completion. Non-zero codes are recorded in the journal for troubleshooting purposes.
Can I use network commands in a system-shutdown script?
Generally, no. By the time system-shutdown scripts run, network interfaces are usually brought down and network services are stopped. Attempting to use network commands may result in errors because the necessary daemons are no longer active. Stick to local filesystem operations or pre-configured connections.
How do I prevent my script from hanging the system?
Avoid any commands that require user interaction or wait indefinitely for external resources. Use timeouts for any potentially blocking operations, such as waiting for a device to respond. Testing your script manually in a simulated shutdown environment can help identify potential hangs before deploying them to production.
Is it better to store scripts in /lib or /etc for shutdown tasks?
Scripts in /lib/systemd/system-shutdown are part of the system package and may be overwritten during updates. Storing custom scripts in /etc/systemd/system-shutdown is recommended as it persists across updates and takes precedence. This separation ensures your custom logic remains intact even when system packages are upgraded.