Slurm Service Status: Failure on CentOS Cluster

The Slurm daemon (slurmd) failing to start after a reinstallation on a CentOS cluster typically stems from configuration mismatches, incorrect file ownership, or network binding issues. This article outlines the diagnostic steps to identify the root cause and provides safe resolution methods for common service failures without risking system integrity.

Understanding the Service Failure Context

When you reinstall Slurm on a High-Performance Computing (HPC) cluster running CentOS, the transition from an old configuration to a new one often introduces subtle incompatibilities. The slurmd service is responsible for managing job execution on compute nodes. If it fails to start, jobs cannot be dispatched, halting all computational work. The failure usually occurs during the systemd initialization phase, where the service manager reports an exit code that indicates why the process terminated immediately upon launch. It is critical to distinguish between a configuration error and a binary path issue. In many cases, administrators assume the binary is missing or corrupted, but the actual problem lies in how the service reads its configuration files. The configuration directory, typically /etc/slurm/, must contain valid slurm.conf and partition-specific configurations. If these files are malformed, slurmd will refuse to start to prevent data corruption or security breaches. For clusters experiencing broader node recognition issues, such as the master node failing to see compute nodes, refer to CentOS Cluster: Master Node Not Recognizing Nodes. This related issue often shares underlying causes with service startup failures, particularly regarding networking and authentication mechanisms.

Diagnosing the Specific Error Code

The first step in troubleshooting is to retrieve the specific error message generated by the systemd journal. Generic status checks like systemctl status slurmd often provide only a high-level summary. To get detailed logs, use the following command:

journalctl -u slurmd -n 50 --no-pager

Look for lines containing "error," "fatal," or "failed." Common errors include: - Permission Denied: The service user (slurm) cannot read configuration files or write to log directories. - Configuration Parse Error: A syntax error in slurm.conf prevents the daemon from initializing. - Network Binding Failed: The node cannot bind to the required IP address or port defined in the configuration. If the error relates to binary paths, ensure that the slurmd executable is in the expected location. You can verify the path using standard package management tools rather than guessing. For guidance on locating binaries in HPC environments, see Finding the Path of slurmd Binary in HPC Systems.

Verifying Configuration File Integrity

A reinstallation often leaves behind old configuration files that conflict with the new software version. Validate the syntax of your slurm.conf file using the built-in validation tool provided by Slurm. Run the following command as the root user:

scontrol show config

This command attempts to parse the configuration file and report any syntax errors. If it returns a list of settings, the configuration is syntactically correct. If it throws an error, note the line number mentioned and edit /etc/slurm/slurm.conf to fix the issue. Common mistakes include missing commas, incorrect indentation, or undefined variables. Ensure that the ControlMachine parameter matches the hostname of the Slurm controller exactly. Mismatches here cause immediate failure on both the controller and compute nodes. Additionally, check that the SlurmUser is set to slurm and that this user exists on the system.

Checking File Ownership and Permissions

Incorrect file permissions are a frequent cause of service failure after reinstallation. The slurm user must own the configuration directory and have read access to all files within it. However, modifying permissions directly on /etc/slurm/ using chown can be risky if not done correctly, potentially breaking other system services or causing security vulnerabilities. Instead of manually changing ownership, rely on the package manager to restore default permissions if possible. If manual intervention is necessary, ensure that: 1. The directory /etc/slurm/ is owned by root:slurm. 2. All files inside /etc/slurm/ are readable by the slurm group. 3. Log directories, typically /var/log/slurm/, are writable by the slurm user. Avoid using commands like restorecon unless you are certain that SELinux contexts are the issue, as this can inadvertently reset permissions for other services. Focus on the basic Unix ownership model first.

Restarting and Validating the Service

Once you have addressed potential configuration and permission issues, restart the service to test the changes. Use the following sequence:

systemctl stop slurmd
systemctl start slurmd
systemctl status slurmd

If the service starts successfully, verify that it is communicating with the controller by checking the state of the node:

sinfo -N

This should list the node with a status of IDLE or MIXED. If it remains in a DRAIN or FAIL state, revisit the journal logs for new errors. Persistent failures may require a full cluster reboot to clear stale network connections or locked resources.

Frequently Asked Questions

Why does slurmd fail immediately after starting?

Immediate failure usually indicates a fatal configuration error or a permission issue. Check the journal logs for "permission denied" or "parse error" messages to pinpoint the exact cause.

Can I manually change permissions on /etc/slurm/?

While possible, it is safer to let the package manager handle permissions or use targeted chown commands. Manual bulk changes can break SELinux policies or affect other services relying on those files.

What if scontrol show config returns no output?

This typically means the configuration file is empty or unreadable. Ensure that /etc/slurm/slurm.conf exists and contains valid parameters. Copy a working configuration from another node if available.

How do I check if the Slurm user exists?

Run id slurm to verify the user exists. If it does not, create it using useradd -r -s /bin/false slurm before restarting the service.

Does SELinux affect Slurm service startup?

Yes, SELinux can block Slurm from accessing certain ports or files. If you suspect SELinux, check /var/log/audit/audit.log for denials and adjust contexts accordingly, but avoid broad restorecon commands without verification.