Slurm 安装部署

Slurm 单机环境搭建

说明

机器：2C4G100G
主机名：hostnamectl set-hostname sms
slurm 版本：20.11.8

快速 demo

安装数据库

安装

bash

yum install mariadb mariadb-server -y

修改配置文件 /etc/my.cnf

    
    text
  
[mysqld]
...
innodb_buffer_pool_size=1024M
innodb_log_file_size=64M
innodb_lock_wait_timeout=900

启动服务

bash

systemctl start mariadb.service

数据库初始化

    
      
      
    
    sql
  
CREATE DATABASE `slurm_acct_db` /*!40100 DEFAULT CHARACTER SET UTF8 */;
GRANT ALL ON slurm_acct_db.* TO 'slurm'@'localhost' IDENTIFIED BY 'password';
GRANT ALL ON slurm_acct_db.* TO 'slurm'@'sms' IDENTIFIED BY 'password';
GRANT ALL ON slurm_acct_db.* TO 'slurm'@'%' IDENTIFIED BY 'password';

CREATE DATABASE `slurm_jobcomp_db` /*!40100 DEFAULT CHARACTER SET UTF8 */;
GRANT ALL ON slurm_jobcomp_db.* TO 'slurm'@'localhost' IDENTIFIED BY 'password';
GRANT ALL ON slurm_jobcomp_db.* TO 'slurm'@'sms' IDENTIFIED BY 'password';
GRANT ALL ON slurm_jobcomp_db.* TO 'slurm'@'%' IDENTIFIED BY 'password';

munge

该组件所有节点都需要安装，且配置相同。

    
    bash
  
yum install -y munge

dd if=/dev/urandom bs=1 count=1024 >/etc/munge/munge.key
chmod 0400 /etc/munge/munge.key
chown munge:munge /etc/munge/munge.key

systemctl enable munge
systemctl start munge

说明：

所有节点 /etc/munge/munge.key 需要配置一致
munge 作用参考 munge 使用介绍

slurmdbd

部署在控制节点

安装

bash

yum install -y slurm-slurmdbd

配置

/etc/slurm/slurmdbd.conf

slurmdbd-conf ↕

    
      
      
    
    text
  
# Authentication info
AuthType=auth/munge
AuthInfo=/var/run/munge/munge.socket.2

# slurmdbd info
DebugLevel=4
#DefaultQOS=normal,standby
DbdAddr=172.20.0.20
DbdHost=sms
DbdPort=6819

LogFile=/var/log/slurm/slurmdbd.log
#MessageTimeout=300
PidFile=/var/run/slurm/slurmdbd.pid
#PluginDir=

PrivateData=accounts,users,usage,jobs
PurgeEventAfter=1month
PurgeJobAfter=1month
PurgeResvAfter=1month
PurgeStepAfter=1month
PurgeSuspendAfter=1month
PurgeTXNAfter=1month
PurgeUsageAfter=1month
SlurmUser=root
#TrackWCKey=yes

# Database info
StorageType=accounting_storage/mysql
StorageHost=sms
# StorageBackupHost=sms2
StoragePort=3306
StoragePass=password
StorageUser=slurm
StorageLoc=slurm_acct_db

说明：slurm 20.11 之后的版本，可以通过 StorageHost/StorageBackupHost 参数配置主备

启动服务

    
    bash
  
systemctl enable slurmdbd
systemctl start slurmdbd
systemctl status slurmdbd

slurmctld

部署在控制节点

安装

bash

yum install -y slurm-slurmctld

配置

文件初始化

    
    bash
  
touch /etc/slurm/gres.conf
touch /etc/slurm/slurm_node.conf
touch /etc/slurm/slurm_partition.conf
touch /etc/slurm/topology.conf

主配置文件：/etc/slurm/slurm.conf

slurm-conf ↕

    
      
      
    
    text
  
#
# See the slurm.conf man page for more information.
#
ControlMachine=sms
ControlAddr=172.20.0.20
#BackupController=
#BackupAddr=
#
AuthType=auth/munge
#CheckpointType=checkpoint/none
CryptoType=crypto/munge
#DisableRootJobs=NO
#EnforcePartLimits=NO
#Epilog=
#EpilogSlurmctld=
#FirstJobId=1
#MaxJobId=999999
#GresTypes=
#GroupUpdateForce=0
#GroupUpdateTime=600
#JobCheckpointDir=/var/slurm/checkpoint
#JobCredentialPrivateKey=
#JobCredentialPublicCertificate=
#JobFileAppend=0
#JobRequeue=1
#JobSubmitPlugins=
#KillOnBadExit=0          # 异常作业清理
#LaunchType=launch/slurm
#Licenses=foo*4,bar
#MailProg=/bin/true
#MaxJobCount=5000        # 限制系统中允许运行的最大作业量
#MaxStepCount=40000
#MaxTasksPerNode=128
MpiDefault=pmix
#MpiParams=ports=#-#
#PluginDir=
#PlugStackConfig=
#PrivateData=accounts,events,jobs,reservations,usage,users
PrivateData=jobs
# ProctrackType=proctrack/cgroup | proctrack/pgid
ProctrackType=proctrack/linuxproc # 进程跟踪插件
#Prolog=
#PrologFlags=
#PrologSlurmctld=
#PropagatePrioProcess=0
#PropagateResourceLimits=
#PropagateResourceLimitsExcept=
#RebootProgram=
ReturnToService=1                 # down节点恢复模式
#SallocDefaultCommand=
SlurmctldPidFile=/var/run/slurm/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurm/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurm/d
SlurmUser=root
#SlurmdUser=root
#SrunEpilog=
#SrunProlog=
StateSaveLocation=/var/spool/slurm/ctld  # Slurmctld 工作时存储状态的临时文件路径，包含正在运行和刚运行完不久的作业，刚运行完作业的保留时间由 MinJobAge 控制，默认 300s
SwitchType=switch/none
#TaskEpilog=
#TaskPlugin=task/none
TaskPlugin=task/affinity,task/cgroup  # 任务启动cpuset
#TaskPluginParam=
#TaskProlog=
TopologyPlugin=topology/tree    # 拓扑调度
#TmpFS=/tmp
#TrackWCKey=no
#TreeWidth=
#UnkillableStepProgram=
#UsePAM=0
#
#
# TIMERS
#BatchStartTimeout=10
#CompleteWait=0
#EpilogMsgTime=2000
#GetEnvTimeout=2
#HealthCheckInterval=0
#HealthCheckProgram=
InactiveLimit=0
KillWait=30
#MessageTimeout=10
#ResvOverRun=0
MinJobAge=300
#OverTimeLimit=0
SlurmctldTimeout=120       # 主备切换时间
SlurmdTimeout=300          # 计算代理响应时间
#UnkillableStepTimeout=60
#VSizeFactor=0
Waittime=0
#
#
# SCHEDULING
#DefMemPerCPU=0                       # https://slurm.schedmd.com/slurm.conf.html#OPT_DefMemPerCPU
#FastSchedule=1
#MaxMemPerCPU=0
#SchedulerTimeSlice=30
SchedulerType=sched/backfill           # 回填策略
#SelectType=select/linear              # 资源选择算法：控制 CPU 资源是以整个节点为单位分配给作业和作业步骤，还是CPU/Thread/Core
SelectType=select/cons_res
SelectTypeParameters=CR_CPU            # CR_(CPU|CORE|SOCKET)*
#SelectTypeParameters=batch_sched_delay=3,defer,sched_min_interval=10,sched_interval=30,default_queue_depth=100,bf_max_job_test=100,bf_interval=30
#
#
# JOB PRIORITY
#PriorityFlags=
PriorityType=priority/multifactor      # 优先级策略
PriorityDecayHalfLife=06:00:00         # 半衰期时长
PriorityCalcPeriod=15
#PriorityFavorSmall=
#PriorityMaxAge=
#PriorityUsageResetPeriod=
#PriorityWeightAge=
PriorityWeightFairshare=1000
#PriorityWeightJobSize=
PriorityWeightPartition=1000
#PriorityWeightQOS=
#
#
# LOGGING AND ACCOUNTING
#AccountingStorageEnforce=0
AccountingStorageEnforce=associations,limits
AccountingStorageTRES=cpu,mem,gres/gpu
AccountingStorageHost=sms
# AccountingStorageBackupHost=sms2
#AccountingStorageLoc=
#AccountingStoragePass=
AccountingStoragePort=6819
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageUser=slurm
AccountingStoreJobComment=YES
ClusterName=cluster
#DebugFlags=
JobCompHost=172.20.0.20
JobCompLoc=slurm_jobcomp_db
JobCompPass=password
JobCompPort=3306
JobCompType=jobcomp/mysql
JobCompUser=slurm
#JobContainerType=job_container/none
JobAcctGatherFrequency=30
# JobAcctGatherType=jobacct_gather/none
JobAcctGatherType=jobacct_gather/linux
SlurmctldDebug=3    # 越大，debug信息越详细
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurm/slurmd.log
#SlurmSchedLogFile=
#SlurmSchedLogLevel=
#
#
# POWER SAVE SUPPORT FOR IDLE NODES (optional)
#SuspendProgram=
#ResumeProgram=
#SuspendTimeout=
#ResumeTimeout=
#ResumeRate=
#SuspendExcNodes=
#SuspendExcParts=
#SuspendRate=
#SuspendTime=
#
#
# COMPUTE NODES
#NodeName=localhost CPUs=1 State=UNKNOWN
#PartitionName=debug Nodes=localhost Default=YES MaxTime=INFINITE State=UP

include slurm_node.conf
include slurm_partition.conf
include topology.conf
include gres.conf

说明：

通过指定多个 SlurmctldHost 实现 HA？
AccountingStorageType 控制将作业信息收集存储到文件或者数据库：
- accounting_storage/filetxt，默认记录在 /var/log/slurm_jobacct.log
- accounting_storage/slurmdbd

节点配置 /etc/slurm/slurm_node.conf

text

NodeName=c[1-3] CPUs=64 CoresPerSocket=16 ThreadsPerCore=2 RealMemory=772433 MemSpecLimit=51200 State=UNKNOWN

指标说明：

Sockets=lscpu | grep "^Socket(s):" | awk '{print $2}'
CoresPerSocket=lscpu | grep "^Core(s) per socket:" | awk -F ":" '{print $2}'
ThreadsPerCore=lscpu | grep "^Thread(s) per core:" | awk -F ":" '{print $2}'
若果需要 show node 显示 ip 地址，可以添加 NodeAddr 字段
通过 CoreSpec 指定给系统预留的 CPU 数量，默认需要给计算节点预留一个 CPU，否则调度失败

队列配置 /etc/slurm/slurm_partition.conf

    
    text
  
PartitionName=normal Nodes=c[1-3] Default=YES MaxTime=24:00:00 State=UP

GresTypes=gpu,bandwidth
NodeName=c[4-6] Gres=gpu:tesla:2,bandwidth:lustre:no_consume:25G
PartitionName=gpu Nodes=c[4-6] Default=YES DisableRootJobs=YES MaxTime=3-00:00:00 MinNodes=1 LLN=YES OverSubscribe=FORCE:1 PriorityJobFactor=1 PriorityTier=1000 DefMemPerCPU=1024 MaxMemPerCPU=2500 QOS=gpu_normal State=UP

说明：

PartitionName 指定 Partition 名称
Nodes 指定节点范围
Default 是否为默认
MaxTime 最大作业运行时长
State 该分区状态
修改后需要重启 sms 节点 slurmctld 和计算节点 slurmd 进程
执行 squeue 可以在查看到对应的 Partition

网络拓扑配置 /etc/slurm/topology.conf
资源配置 /etc/slurm/gres.conf

启动服务

    
    bash
  
systemctl enable slurmctld
systemctl start slurmctld
systemctl status slurmctld

验证

执行 sinfo 正常显示结果：

    
    bash
  
[root@sms ~]# sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
normal*      up 1-00:00:00      1    unk sms
[root@sms ~]# squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

slurmd

在计算节点安装

安装

bash

yum install -y slurm-slurmd

配置

将 sms 节点的配置文件发送到计算节点：

bash

scp /etc/slurm/slurm.conf /etc/slurm/slurm_node.conf /etc/slurm/slurm_partition.conf /etc/slurm/topology.conf /etc/slurm/gres.conf c1:/etc/slurm

启动服务

    
    bash
  
systemctl enable slurmd
systemctl start slurmd

slurm-nss_slurm

在所有节点配置，防止没有分配作业的节点被用户登录。参考：https://slurm.schedmd.com/nss_slurm.html

安装

bash

yum install -y slurm-nss_slurm

配置

配置 nss，在 /etc/slurm/slurm.conf 中配置：

text

LaunchParameters=enable_nss_slurm

重启 slurmd 服务

修改 PAM，作用为仅给 slurm 分配作业的机器登录认证，更多参考：Linux PAM 可插拔认证模块介绍

    
    bash
  
echo "account required pam_slurm.so" >> /etc/pam.d/sshd

不需要重启服务

部署相同的 /etc/slurm/slurm.conf 和 /etc/munge/munge.key
启动 munge 认证服务
安装 Slurm 客户端命令工具（sbatch、srun、squeue 等）

注意：SlurmctldHost 仅用于指定运行控制进程的主备控制器（第一个为主控制节点，后续为备用控制节点），切勿将普通登录节点指定为 SlurmctldHost。Job ID 均由集群的 slurmctld 主控制进程统一生成与维护。

SlurmctldHost 介绍

text

The short, or long, hostname of the machine where Slurm control daemon is executed (i.e. the name returned by the command "hostname -s"). This hostname is optionally followed by the address, either the IP address or a name by which the address can be identified, enclosed in parentheses (e.g. SlurmctldHost=slurmctl-primary(12.34.56.78)). This value must be specified at least once. If specified more than once, the first hostname named will be where the daemon runs. If the first specified host fails, the daemon will execute on the second host. If both the first and second specified host fails, the daemon will execute on the third host.

FAQ

更正 Slurm 节点状态

采用 scontrol show node 但节点状态不对，出现如：

Low RealMemory，作业一直处于 CG (completing) 状态
Node unexpectedly rebooted

如果节点没有作业在运行，修复命令如下：

    
    text
  
# 方案一（推荐）
scontrol update NodeName=<node> State=DOWN Reason=abc
scontrol update nodename=<node> state=resume  # 处于 down 状态无法自动恢复时执行

# 方案二
scontrol update nodename=<node> state=idle    # 状态非正常时，强制重置

Low CPUs 问题

采用 scontrol show node，Reason 如下：

Low socketcorethread count, Low CPUs

sms:/var/log/slurmctld.log

    
    text
  
[2021-07-06T23:13:49.934] error: Node c1 has low socket*core*thread count (4 < 32)
[2021-07-06T23:13:49.934] error: Node c1 has low cpu count (4 < 32)
[2021-07-06T23:13:49.934] error: _slurm_rpc_node_registration node=c1: Invalid argument

解决方法：

修改 sms:/etc/slurm/slurm.conf

text

NodeName=c[1-4] Sockets=2 CoresPerSocket=1 ThreadsPerCore=2 State=UNKNOWN

指标说明：

Sockets=lscpu | grep "^Socket(s):" | awk '{print $2}'
CoresPerSocket=lscpu | grep "^Core(s) per socket:" | awk -F ":" '{print $2}'
ThreadsPerCore=lscpu | grep "^Thread(s) per core:" | awk -F ":" '{print $2}'

重启：

bash

systemctl restart slurmctld.service

修改 c1:/etc/slurm/slurm.conf，与 sms:/etc/slurm/slurm.conf 保持一致：

text

NodeName=c[1-4] Sockets=2 CoresPerSocket=1 ThreadsPerCore=2 State=UNKNOWN
PartitionName=normal Nodes=c[1-4] Default=YES MaxTime=24:00:00 State=UP

并重启：

bash

systemctl restart slurmd.service

Slurm job id has expired 问题

错误信息如下：

    
    bash
  
[test@sms ~]$ srun -N 1 -l hostname
srun: error: Slurm job 12 has expired
srun: Check SLURM_JOB_ID environment variable. Expired or invalid job 12

解决方法：

退出 shell，然后重新进入。

SSH 问题

    
    bash
  
[test@sms ~]$ ssh c11 pwd
Access denied: user test (uid=1000) has no active jobs on this node.
Authentication failed.

解决方法：

属于正常现象，通过在 /etc/pam.d/sshd 中添加 pam_slurm.so 实现拒绝非运行任务的 ssh 登录。通过 salloc/srun 分配节点 job 即可登录。

    
    bash
  
[test@sms ~]$ salloc -N 1 -w c11
salloc: Granted job allocation 16
[test1@sms ~]$ ssh c11
[test1@c11 ~]$

# 或

[test@sms ~]$ srun -N 2 -l sleep 100  # 然后在登录即可

原理

安装 slurm-pam_slurm-ohpc 时，实际执行：

    
    bash
  
# 类似 `cp /opt/ohpc/admin/images/centos7.7/usr/lib64/security/pam_slurm.so /usr/lib64/security/`，实际 slurm-pam_slurm-ohpc 提供 `/usr/lib64/security/pam_slurm.so` 文件，并在 `/etc/pam.d/sshd` 中添加配置
echo "account    required     pam_slurm.so" >> /etc/pam.d/sshd

/var/log/slurmctld.log 日志错误问题

error: Node c2 appears to have a different slurm.conf than the slurmctld. This could cause issues with communication and functionality. Please review both files and make sure they are the same. If this is expected ignore, and set DebugFlags=NO_CONF_HASH in your slurm.conf.

使用命令systemctl restart slurmd时，slurmd进程并没有杀掉，需要手动杀掉该进程：

text

pdsh -w c[1-3] pkill slurmd

建议通过 reload 重启 slurmd 进程

text

pdsh -w c[1-3] systemctl reload slurmd

MPI 依赖问题

    
      
      
    
    bash
  
$ module load lammps/2020.10
$ lmp
[c11:22047] pmix_mca_base_component_repository_open: unable to open mca_pnet_opa: libpsm2.so.2: cannot open shared object file: No such file or directory (ignored)
[c11:22047] mca_base_component_repository_open: unable to open mca_oob_ud: libosmcomp.so.3: cannot open shared object file: No such file or directory (ignored)
[c11:22046] pmix_mca_base_component_repository_open: unable to open mca_pnet_opa: libpsm2.so.2: cannot open shared object file: No such file or directory (ignored)
[c11:22046] mca_base_component_repository_open: unable to open mca_oob_ud: libosmcomp.so.3: cannot open shared object file: No such file or directory (ignored)
[c11:22046] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)
[c11:22046] mca_base_component_repository_open: unable to open mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored)
[c11:22046] mca_base_component_repository_open: unable to open mca_mtl_psm2: libpsm2.so.2: cannot open shared object file: No such file or directory (ignored)
LAMMPS (29 Oct 2020)

解决方式：

text

pdsh -w c[1-3] yum install -y infinipath-psm libpsm2 psmisc compat-opensm-libs librdmacm

测试：

    
    bash
  
[test@sms ~]$ salloc -N 2 -w c11,c12
salloc: Granted job allocation 31
[test@sms ~]$ mpirun --allow-run-as-root -host c11,c12 uptime

slurm_recv_timeout at 0 of 4, recv zero bytes

slurmctld 错误日志：

text

Aug 14 16:33:41 sms1 slurmctld[2874575]: slurmctld: debug:  slurm_recv_timeout at 0 of 4, recv zero bytes

原因：

机器时间不一致：date
机器时区不同：date -R
硬件时钟不同步：hwclock –systohc

slurm_recv_timeout

slurmctld 错误日志：

    
    text
  
[2021-08-14T16:56:08.985] debug:  slurm_recv_timeout at 0 of 4, recv zero bytes
[2021-08-14T16:56:08.985] error: slurm_receive_msg: Zero Bytes were transmitted or received
[2021-08-14T16:56:08.985] error: Invalid RPC received 65534 while in standby mode

原因：

tcping 端口检测导致，slurm REQUEST_PING message type is 1008，可通过 sdiag 命令查看。

error: slurm_auth_get_host: Lookup failed

    
    text
  
[2021-08-15T14:47:59.804] debug2: Error connecting slurm stream socket at 172.20.0.20:6817: Connection refused
...
[2021-08-15T14:48:12.122] error: get_name_info: getnameinfo() failed: Name or service not known
[2021-08-15T14:48:12.122] error: slurm_auth_get_host: Lookup failed for 172.20.1.20
[2021-08-15T14:48:12.122] debug3: _set_hostname: Using auth hostname for alloc_node: 172.20.1.20

见：munge 使用介绍 faq

ldap 主从同步导致的新用户执行作业失败问题

新增用户无法执行作业，错误日志：

text

Batch job submission failed: Invalid account or account/partition combination specified

slurmctl 错误日志如下：

text

user add couldn't get a uid for user xxx

解决方法

ldap主从 同步时间间隔问题，默认连接nslcd主，若主出现故障，将会出现一直链接被备情况，建议使用 ldap主主 方案，并通过vip提供服务。

说明

快速 demo

安装数据库

munge

slurmdbd

安装

配置

slurmdbd-conf ↕

启动服务

slurmctld

安装

配置

slurm-conf ↕

启动服务

验证

slurmd

安装

配置

启动服务

slurm-nss_slurm

安装

配置

login 节点

FAQ

更正 Slurm 节点状态

Low CPUs 问题

Slurm job id has expired 问题

SSH 问题

MPI 依赖问题

slurm_recv_timeout at 0 of 4, recv zero bytes

slurm_recv_timeout

error: slurm_auth_get_host: Lookup failed

ldap 主从同步导致的新用户执行作业失败问题