In this article, we will discuss the issue of a CentOS cluster (version 7) master node not recognizing nodes when using SLURM 18.08.8. We will cover the key concepts related to this problem, including the SLURM cluster architecture, configuration files, and the batch command. The article will be at least 800 words long and will include subtitles, paragraphs, and code blocks enclosed within tags. Proper formatting according to the programming language and indentation/tabulation will be included in the code blocks.
SLURM Cluster Architecture
A SLURM cluster consists of a master node, also known as the controller, and several compute nodes. The master node is responsible for managing the queue of jobs and allocating resources to the compute nodes. The compute nodes are where the actual computation takes place.
Configuration Files
SLURM uses several configuration files to manage the cluster. The main configuration file is located at /etc/slurm/slurm.conf and contains settings such as the location of the cluster database, the partition configuration, and the node configuration. The node configuration is used by the master node to identify and communicate with the compute nodes.
Batch Command
The sbatch command is used to submit jobs to the SLURM cluster. It takes in a script as an argument, which contains information about the job such as the number of nodes required, the time limit, and the commands to run. Once the job is submitted, SLURM will allocate the necessary resources and run the job on the compute nodes.
Master Node Not Recognizing Nodes
If the master node is not recognizing the compute nodes, it is likely due to a misconfiguration in the /etc/slurm/slurm.conf file. Specifically, the NodeName, NodeHostName, and NodeAddr values for each node must be correctly set. The NodeName value should be a unique identifier for the node, the NodeHostName value should be the hostname of the node, and the NodeAddr value should be the IP address of the node.
Example Configuration
NodeName=node01 NodeHostName=node01.example.com NodeAddr=10.0.0.1
Troubleshooting
To troubleshoot this issue, first check the /var/log/slurm/slurmctld.log file for any error messages related to the node configuration. Next, verify that the node configuration in the /etc/slurm/slurm.conf file is correct. If the configuration appears to be correct, check that the compute nodes are able to communicate with the master node by running ping node01 from one of the compute nodes (replace node01 with the hostname of the master node).
- SLURM cluster architecture: master node (controller) and compute nodes
- Configuration files:
/etc/slurm/slurm.conf
- Batch command:
sbatch
- Common issue: master node not recognizing nodes due to misconfiguration in
/etc/slurm/slurm.conf
References