When a CentOS master node fails to recognize compute nodes in SLURM 18.08.8, the issue typically stems from mismatched slurm.conf settings, unreachable communication ports, or service synchronization delays rather than a single port blockage. Verifying that all nodes share identical configuration files and that the slurmd daemon is actively listening on the correct interface resolves most recognition failures.
Verifying Configuration Consistency
The most common cause for a head node not seeing compute nodes is a discrepancy in theslurm.conf file across the cluster. SLURM requires strict consistency in several critical fields to establish trust and communication between the controller and the daemons running on worker nodes. If even one character differs, the nodes may fail to register correctly.
First, ensure that the SlurmctldHost parameter on every node points to the correct IP address or hostname of the master node. Similarly, verify that the NodeName entries in the configuration match exactly what is defined in the partition definitions. A typo in the hostname can prevent the master from associating incoming heartbeats with a valid node definition.
You should also check the SlurmctldPort and SlurmdPort values. While default ports are often used, custom configurations require both the controller and the daemons to agree on the specific TCP/UDP ports they are using. If these ports do not align, the handshake process will fail silently, leaving the master node unaware of the compute resources.
For a deeper understanding of binary paths and service locations, you might find Finding the Path of slurmd Binary in HPC Systems helpful, as incorrect binary execution paths can sometimes mimic configuration errors by causing the daemon to start with default or missing parameters.
Checking Network Connectivity and Ports
Network connectivity is the second pillar of cluster health. The master node must be able to reach the compute nodes, and vice versa, on the specific ports designated for SLURM communication. Unlike some services, SLURM uses multiple ports for different types of traffic, including control messages, job state updates, and accounting data. To diagnose this, use tools liketelnet or nc (netcat) from the master node to test connectivity to each compute node on the relevant ports. For example, if your configuration specifies port 6817 for the controller, run a command such as telnet compute-node-01 6817. A successful connection indicates that the network path is open and no firewall is blocking the traffic.
It is crucial to remember that firewalls on CentOS systems, such as firewalld, may restrict access to these ports by default. However, assuming a specific port is the sole culprit without verifying the actual configuration is risky. Instead, review the slurm.conf to identify which ports are explicitly set and ensure those are allowed through any intermediate firewalls or security groups.
If you are experiencing broader service issues, such as the SLURM controller failing to start entirely, refer to Slurm Service Status: Failure on CentOS Cluster - Reinstallation Troubleshooting for steps on diagnosing daemon startup failures that could indirectly affect node recognition.
Reviewing System Logs for Errors
When configuration and network checks pass, the next step is to examine the system logs for specific error messages. SLURM generates detailed logs that can pinpoint why a node is being rejected or ignored. The primary log files are usually located in/var/log/messages or within the SLURM-specific log directory, depending on your distribution's setup.
On the master node, look for entries related to slurmctld that mention "node" or "state." You might see warnings about nodes entering an invalid state or failing to respond to pings. These messages often provide the exact reason for the failure, such as a timeout waiting for a response or a mismatch in the node's reported version.
Similarly, check the logs on the compute nodes for slurmd entries. If the daemon on a worker node cannot connect back to the controller, it will log errors about authentication failures or connection refused messages. This bidirectional logging approach helps isolate whether the problem lies with the master's ability to initiate contact or the worker's ability to respond.
Synchronizing Time Across Nodes
Time synchronization is a subtle but critical factor in cluster computing. SLURM relies on timestamps for job scheduling, accounting, and node health checks. If the time difference between the master node and a compute node exceeds a certain threshold, the nodes may reject each other's messages as invalid or stale. Ensure that all nodes in the cluster are synchronized using NTP (Network Time Protocol). On CentOS, this is typically managed by thechronyd service. Verify that the service is active and that the offset between nodes is minimal, ideally less than a few seconds.
You can check the status of the time synchronization service by running systemctl status chronyd and checking the output of chronyc tracking. Large offsets can lead to cryptic errors where nodes appear to be online but are effectively invisible to the scheduler due to timestamp mismatches.
Restarting Services Correctly
After making any changes to the configuration or network settings, you must restart the SLURM services for the changes to take effect. It is important to follow the correct order to avoid leaving the cluster in an inconsistent state. Typically, you should stop the services on all compute nodes first, then the master node, and then start them in reverse order. Use thesystemctl commands to manage the services. For example, stop the controller with systemctl stop slurmctld and the daemons with systemctl stop slurmd. Once stopped, wait a few seconds to ensure the processes have fully terminated before starting them again.
Starting the master node first ensures that when the compute nodes come back online, there is a controller ready to accept their registration requests. Monitor the logs immediately after restarting to confirm that the nodes transition to an "idle" or "alloc" state rather than remaining in "unknown" or "drain" states.
Frequently Asked Questions
What happens if I change the SlurmctldHost IP address?
Changing the IP address requires updating the configuration on every node in the cluster, not just the master. If you only update the master, the compute nodes will continue trying to connect to the old IP, resulting in connection timeouts and failed registrations. Always synchronize this change across all nodes simultaneously.Can DNS resolution issues cause node recognition failures?
Yes, if your configuration uses hostnames instead of IP addresses, DNS must resolve correctly on all nodes. Ensure that the forward and reverse DNS records are consistent. If DNS is unreliable, consider using IP addresses inslurm.conf to eliminate resolution delays or failures during the handshake process.
Why do nodes show as "down" instead of "unreachable"?
A "down" state often indicates that the node was previously known to the cluster but has stopped responding or has been manually marked down. An "unreachable" state suggests a network or service issue preventing contact. Check the last heartbeat timestamp in the logs to determine if the node crashed or if the network simply dropped the packets.Is it safe to disable firewalld for testing?
While disabling firewalld can help isolate whether firewall rules are causing the issue, it is not recommended for production environments. Instead, add specific rules to allow SLURM ports usingfirewall-cmd. This maintains security while allowing you to verify if the firewall is the bottleneck.
How do I verify that slurmd is actually running on a compute node?
Runps aux | grep slurmd on the compute node to check for the process. Additionally, check the exit code of the service using systemctl is-active slurmd. If the process is running but the node is still not recognized, the issue likely lies in the configuration or network connectivity rather than the daemon itself.