Experiencing Baffling Difference Executing Binary vs Shell Systemd/Subprocess: A Tech Support Guide

The core difference lies in how the process environment is constructed. When a binary executes directly via systemd, it inherits a clean, minimal environment defined by the service unit file. In contrast, executing through a shell subprocess (like sh -c) introduces an intermediate layer that expands variables, handles redirections, and merges with the user's login environment if not strictly isolated. This discrepancy often causes scripts to fail when moved from interactive terminal testing to systemd deployment because the shell interpreter adds context that the raw binary never sees.

Understanding Environment Isolation

Systemd services are designed to be reproducible and isolated. By default, they do not inherit the full environment of the user who started the daemon or the administrator who issued the systemctl start command. Instead, they use a sanitized environment provided by systemd itself. When you run a binary directly in the ExecStart directive, such as /usr/bin/myapp, the process receives only the variables explicitly listed in the service file or set via Environment= directives. However, many developers test their applications using a shell wrapper. For example, running bash /scripts/run.sh or sh -c "/usr/bin/myapp" introduces a new process: the shell. Shells are interactive by nature and often source configuration files like .bashrc or .profile depending on how they are invoked. Even if non-interactive, shells perform variable expansion and path resolution before handing control to the target binary. This means your binary might see environment variables like $PATH, $HOME, or custom variables that were exported in your current terminal session but are completely absent in the direct systemd execution context. This isolation is a feature, not a bug, intended to prevent security vulnerabilities and ensure consistent behavior across reboots and deployments. If your application relies on external libraries located in non-standard paths, those paths must be available in the environment passed to the binary. If you rely on shell-specific features like globbing (*) or pipe operators (|), these will also fail if you bypass the shell.

Common Symptoms of Mismatched Execution

Users frequently encounter specific errors that indicate this environment mismatch. One common symptom is "Command not found" errors for tools that are clearly installed and working in the terminal. This happens because the $PATH variable in the systemd environment is minimal and does not include directories like /usr/local/bin or user-specific script folders. Another symptom is permission denied errors when accessing files that the user can read interactively but the service cannot, due to different effective users or capabilities. Additionally, logging output may appear differently; stderr and stdout buffering behavior changes between direct execution and shell-wrapped execution, which can make debugging difficult.

Configuring Systemd Service Files Correctly

To resolve these discrepancies, you must align your systemd configuration with the actual requirements of your binary. The most robust approach is to avoid shell wrappers entirely unless absolutely necessary. If your binary requires specific environment variables, define them explicitly in the service file using the Environment= directive. This ensures that every restart uses the exact same context. For example, if your application needs a specific library path, add the following to your [Service] section:

[Service]
Environment=LD_LIBRARY_PATH=/opt/myapp/lib
Environment=MY_CUSTOM_VAR=value
ExecStart=/usr/bin/myapp --config /etc/myapp.conf

If you must use a shell for complex logic involving pipes or redirects, ensure you understand that the shell becomes the parent process. In this case, any environment variables needed by the final binary must be visible to the shell. You can pass variables to the shell by exporting them before the command or by using the EnvironmentFile= directive to load a dedicated file containing all necessary settings.

Debugging Environment Differences

When troubleshooting why a command works in the terminal but fails in systemd, inspecting the environment is the first step. You can create a simple diagnostic script that dumps the current environment to a log file. Create a service that runs env > /tmp/env_dump.txt and compare its output with the output of env in your active terminal. Another powerful tool is systemd-run, which allows you to execute commands in a temporary service context similar to how systemd manages long-running services. Running systemd-run --user /bin/bash -c 'echo $PATH' gives you insight into what the systemd user manager provides by default. This helps identify missing paths or variables without modifying persistent service files. For more advanced debugging, check the journal logs for the specific service. Use journalctl -u myservice.service -f to watch real-time output. If the failure involves startup issues, look for exit codes and standard error messages. Often, the shell will provide a clear error message like "variable not set," whereas a direct binary might crash silently or return a generic error code. Understanding these nuances helps pinpoint whether the issue is environmental or logical.

Leveraging Variables and Dependencies

Proper management of variables and dependencies is crucial for stable systemd operations. As discussed in related guides on expanding variables in service files, using EnvironmentFile is often cleaner than hardcoding values. It allows you to separate configuration from logic. Furthermore, understanding service dependencies ensures that your binary starts only after required resources, such as network interfaces or mounted drives, are ready. Ignoring these dependencies can lead to race conditions where the binary attempts to access unavailable resources, mimicking environment errors. Expanding the Use of Variables in a systemd Service File provides detailed strategies for managing complex configurations without cluttering the main unit file. Similarly, Restarting Systemd Services Dependencies explains how to structure your service so that it waits for critical subsystems, reducing the likelihood of startup failures that look like execution errors. By strictly defining the environment and avoiding implicit shell behaviors, you create a predictable execution model. This approach eliminates the "it works on my machine" problem, ensuring that your binaries behave identically regardless of how they are launched. Consistency in environment setup is the key to reliable system automation and service management in Linux environments.

Frequently Asked Questions

Why does my script work in bash but fail under systemd?

Bash sources profile files and sets up a rich environment including custom aliases and functions. Systemd provides a minimal environment by design. Your script likely relies on a variable or tool that exists in your user's bash profile but is not defined in the systemd service unit.

How can I see the exact environment a systemd service has?

You can add EnvironmentFile=-/dev/null and then use ExecStartPre=/usr/bin/env > /var/log/myapp/env.log in your service file. This writes the complete environment snapshot to a log file just before the main process starts, allowing you to inspect it later.

Is it safe to use sh -c in ExecStart?

It is generally discouraged unless necessary for redirections or piping. Using sh -c adds overhead and introduces shell-specific variable expansion rules that may differ from direct execution. If you need redirections, consider using systemd's built-in StandardOutput= and StandardError= directives instead.

What causes "Command not found" errors in systemd services?

This usually indicates that the $PATH variable in the systemd environment is too narrow. Systemd does not include all user paths by default. You should either specify the absolute path to the executable in the ExecStart line or add the missing directory to the Environment=PATH=... directive.

Can I use environment variables from .bashrc in systemd?

No, systemd does not source .bashrc or .profile. These files are meant for interactive login shells. To use custom variables, you must define them explicitly in the service file using Environment= or load them from a dedicated configuration file using EnvironmentFile=.