Troubleshooting SSH Nohup Command Not Working as Expected

When nohup fails over SSH, the background process often terminates immediately because the remote shell closes or standard input is not properly redirected. This article details how to configure SSH sessions and nohup flags to ensure long-running scripts persist after your local terminal disconnects.

Understanding the Execution Context

The core issue usually stems from a misunderstanding of how SSH handles session termination. When you run a command like ssh user@host nohup script.sh, the SSH client establishes a connection, executes the command, and then waits for the command to finish before closing the channel. If the command runs in the background via nohup, it may still be tied to the SSH session's file descriptors. If the session ends abruptly due to network issues or explicit closure, the kernel sends a SIGHUP signal to all processes in the session's process group, killing the background job despite the nohup wrapper. To prevent this, you must ensure that the remote shell does not treat the command as part of an interactive login session that gets terminated upon exit. The most robust method involves using the -f flag with SSH, which requests SSH to go to the background just before command execution. This detaches the SSH client from the current terminal and allows the remote process to continue independently.

Proper Syntax for Detached Execution

Use the following structure to guarantee detachment:

ssh -f user@host "nohup /path/to/script.sh &"

The -f flag tells SSH to fork into the background after authentication. The & at the end of the remote command string ensures the script itself runs in the background on the remote host. Without the &, nohup might start the process but wait for it to complete if the shell interprets the sequence differently depending on the remote OS configuration.

Redirecting Standard Input and Output

A common failure mode occurs when nohup attempts to write output but cannot find a valid file descriptor. By default, nohup redirects stdout to nohup.out in the current working directory. However, if the remote user lacks write permissions in that directory, or if the directory does not exist, the command will fail silently or terminate immediately. You should explicitly redirect both standard output and standard error to specific log files. This practice also aids in debugging by providing clear evidence of where the process is writing its data.

ssh -f user@host "nohup /path/to/script.sh > /var/log/my_script.log 2>&1 &"

In this example, 2>&1 merges stderr into stdout, ensuring that error messages are captured alongside normal output. Always verify that the target directory (/var/log/ in this case) is writable by the executing user. If you are running as a non-root user, consider creating a dedicated logs directory within your home folder to avoid permission denied errors.

Checking for Environment Variable Issues

Non-interactive SSH sessions do not load the same environment variables as interactive shells. If your script relies on paths defined in .bashrc or .profile, those variables may be undefined when executed via nohup. This leads to commands failing because binaries cannot be found or configuration files are missing. To mitigate this, source your profile explicitly within the command string or use absolute paths for all executables. For example:

ssh -f user@host ". ~/.bash_profile; nohup /absolute/path/to/script.sh &"

This approach forces the loading of necessary environment settings before launching the script. If you frequently encounter these issues, consider moving critical scripts to system directories like /usr/local/bin or using systemd services for more complex dependencies. For deeper insights into why scripts behave differently inside automated contexts versus manual execution, refer to our guide on Linux RHEL 7 Shell Script Execution Issues.

Verifying Process Status Remotely

After initiating the command, you need a way to confirm it is actually running. Since you disconnected from the SSH session, you cannot see the process directly. Use the ps command to check for the existence of your script.

ssh user@host "ps aux | grep script.sh"

If the process appears in the list with a state of 'S' (sleeping) or 'R' (running), it is active. If it shows 'Z' (zombie), it has terminated but hasn't been cleaned up by its parent. If it is absent, check the nohup.out or your specified log files for error messages. Often, the log will reveal syntax errors in the script or missing dependencies that caused the immediate exit.

Using Screen or Tmux as Alternatives

While nohup is effective for simple tasks, it lacks interactivity. If you need to monitor progress or interact with the running application, consider using terminal multiplexers like screen or tmux. These tools create persistent virtual terminals that survive SSH disconnections.

ssh user@host "tmux new-session -d -s mysession '/path/to/script.sh'"

This command creates a detached tmux session named mysession and runs the script inside it. You can later reattach to view output using tmux attach -t mysession. This method provides greater control and visibility compared to the opaque nature of nohup.

Frequently Asked Questions

Why does my nohup process disappear after I close the terminal?

The process likely received a SIGHUP signal because it was tied to the SSH session. Ensure you use the -f flag with SSH and append & to the remote command to fully detach it from the controlling terminal.

Where is the output of nohup stored by default?

By default, nohup writes output to a file named nohup.out in the current working directory of the remote user. If you do not specify redirection, always check this file for results or errors.

Can I use nohup with Python scripts?

Yes, you can run Python scripts with nohup over SSH. Just ensure you invoke the correct Python interpreter path and handle any buffering issues by setting PYTHONUNBUFFERED=1 in the environment.

What if I get a permission denied error on nohup.out?

This happens when the remote user lacks write permissions in the current directory. Change the directory to one you own, such as ~/tmp, or explicitly redirect output to a writable location using > /path/to/file.log.

Is there a difference between nohup and disown?

nohup prevents the SIGHUP signal and redirects output, while disown removes the job from the shell's job table so it won't receive signals when the shell exits. Combining both offers maximum protection against accidental termination during session closure.