The slurmd binary is typically located in /usr/sbin/slurmd or /usr/bin/slurmd on most Linux HPC clusters. If the which command returns "no slurmd," it usually means the directory containing the daemon is not in your current user's $PATH. You can verify its existence by checking standard system paths or using find to locate the file directly.
Understanding the Slurmd Binary
In High-Performance Computing (HPC) environments, SLURM (Simple Linux Utility for Resource Manager) is the dominant job scheduler and resource manager. Theslurmd daemon is a critical component that runs on every compute node within the cluster. Its primary responsibility is to accept jobs from the central slurmctld controller, allocate resources such as CPUs, memory, and GPUs, and manage the lifecycle of those jobs until completion.
Unlike client-side tools like sbatch or srun, which users invoke to submit work, slurmd operates silently in the background on worker nodes. It does not require direct interaction from end-users but must be present and executable on all nodes for the cluster to function correctly. When administrators encounter errors regarding missing binaries, it often points to a configuration mismatch, a failed installation, or simply an incomplete shell environment for the user attempting to debug the system.
Troubleshooting the 'Which: No Slurmd' Error
The error messagewhich: no slurmd indicates that the shell has searched through every directory listed in the $PATH environment variable and could not find an executable named slurmd. This is a common occurrence for several reasons. First, many Linux distributions place administrative daemons in /usr/sbin rather than /usr/bin. Standard user accounts often do not include /usr/sbin in their default search path for security reasons, meaning even if the binary exists, the which command will not see it.
Second, the binary might be installed in a non-standard location if the cluster was built from source or uses a custom partitioning scheme. In some containerized or virtualized environments, the filesystem layout may differ significantly from bare-metal installations. To resolve this, you must move beyond the which command and use more robust discovery methods that do not rely solely on the $PATH variable.
Locating the Binary Using System Commands
To definitively find the location of theslurmd binary, you should use commands that scan the entire filesystem or check specific known directories. The most reliable method is using the find utility with appropriate permissions. Since /usr/sbin is often restricted, you may need to run these commands as the root user or with sudo.
Run the following command to search the entire filesystem for the binary:
sudo find / -name slurmd -type f 2>/dev/nullslurmd executable if it exists anywhere on the system. If the output is empty, the binary is genuinely missing from the node, indicating a potential installation failure or corruption.
Alternatively, you can check the standard locations manually. On Red Hat-based systems (like CentOS or RHEL), the binary is almost always at:
ls -l /usr/sbin/slurmdls -l /usr/bin/slurmdwhich still fails, the issue is strictly a $PATH configuration problem. You can temporarily add the directory to your path by running export PATH=$PATH:/usr/sbin and then re-running which slurmd.
Verifying Service Status and Installation Integrity
Finding the binary is only half the battle; ensuring it is part of a healthy SLURM installation is the other. If you locate the binary but the service is not running, the path is correct, but the deployment is broken. You can check the status of the daemon using the systemd service manager, which is standard on modern Linux distributions. Use the following command to check if the slurm daemon is active:systemctl status slurmd/var/log/messages or via journald. A missing binary often correlates with broader service failures, as seen in similar cluster issues where master nodes fail to recognize workers due to inconsistent software versions across nodes. For deeper troubleshooting on service failures, refer to our guide on Slurm Service Status: Failure on CentOS Cluster.
Additionally, ensure that the binary has the correct execution permissions. The file should be owned by root and have execute permissions for the owner. Incorrect permissions can prevent the daemon from starting, even if the file is present.
Best Practices for HPC Path Management
To avoid confusion in the future, system administrators should ensure that all necessary SLURM binaries are accessible to the services that need them. While user shells may not need/usr/sbin in their path, the service files for systemd must know exactly where the binary resides. This is configured in the /etc/systemd/system/slurmd.service file (or equivalent), where the ExecStart directive explicitly calls the full path to slurmd.
Regularly verifying the presence and version of slurmd across all compute nodes helps maintain cluster stability. Inconsistent versions between the controller and compute nodes can lead to subtle communication errors that are difficult to diagnose. Automating checks for binary existence during node provisioning ensures that new nodes are integrated correctly into the HPC fabric.
Frequently Asked Questions
Why does 'which slurmd' fail but the service is running?
This happens because the user's shell environment lacks the directory containing the binary (usually /usr/sbin) in its $PATH. The systemd service uses the absolute path defined in its unit file, so it runs regardless of the user's shell configuration.
Can I install SLURM without root privileges?
No, installing and configuring slurmd requires root access because it involves modifying system directories, creating system users, and managing system services. Regular users can only interact with the scheduler using client tools like sbatch.
What should I do if find returns no results for slurmd?
If find returns nothing, the SLURM package is likely not installed on that specific node. You should reinstall the slurm-daemon or slurm-pam package depending on your distribution, or check if the node image was provisioned incorrectly.
How do I check if slurmd is compatible with my slurmctld?
You can compare the version strings by running slurmd --version on the compute node and slurmctld --version on the controller. They must match exactly to ensure proper communication protocol support.
Is it safe to symlink slurmd to /usr/bin?
While technically possible, it is not recommended practice. Keeping binaries in their designated directories (/usr/sbin for system daemons) maintains a clean separation between user applications and system administration tools, reducing the risk of accidental modification or confusion.