Slurm FaQ

草稿: 本文因某种原因尚未公开。

发布时间：2021-08-15 更新时间：2026-11-01 总字数：937 阅读时间：2m 作者：谢先斌IP：上海网址

专栏文章

Slurm FaQ

作业交替 SUSPEND 问题

相同的 2 个作业同时调度到一个节点，会出现 RUNNING <-> SUSPEND 状态经常转化的情况，原理参考：https://slurm.schedmd.com/gang_scheduling.html

轮训周期有 SchedulerTimeslice = 30 控制，可以使用如下命令更新每个 core、CPU 共享的情况：

text

scontrol update PartitionName=normal OverSubscribe=YES:10

注意，slurm.conf 配置不同会出现不同的情况：

    
    text
  
SelectType = select/linear
SelectType = select/cons_res
SelectTypeParameters = CR_CPU / SelectTypeParameters = CR_CORE_MEMORY

partition OverSubscribe 的配置：

No：一个 core 给一个任务用，和 force：1 一样
YES:2 和 Force：2 一样，可以允许一个 core 给 2 个任务跑，就是轮流出现 suspend

申请一个 core，分配了 2 个

使用了OverSubscribe，作业以Core为单位分CPU，若 CPU ThreadsPerCore=2，就会分配一个 Core，即 2 个 CPU，参考：https://slurm.schedmd.com/faq.html#cpu_count

    
    text
  
Slurm documentation refers to CPUs, cores and threads. What exactly is considered a CPU?
If your nodes are configured with hyperthreading, then a CPU is equivalent to a hyperthread. Otherwise a CPU is equivalent to a core. You can determine if your nodes have more than one thread per core using the command "scontrol show node" and looking at the values of "ThreadsPerCore".

Note that even on systems with hyperthreading enabled, the resources will generally be allocated to jobs at the level of a core (see NOTE below). Two different jobs will not share a core except through the use of a partition OverSubscribe configuration parameter. For example, a job requesting resources for three tasks on a node with ThreadsPerCore=2 will be allocated two full cores. Note that Slurm commands contain a multitude of options to control resource allocation with respect to base boards, sockets, cores and threads.

禁用 job requeue

    
    bash
  
$ scontrol show config | grep JobRequeue
JobRequeue              = 1

参考：https://slurm.schedmd.com/slurm.conf.html#OPT_JobRequeue

如何远程计算节点 desktop

参考：https://slurm.schedmd.com/faq.html#x11

slurm_load_jobs error: Invalid job id specified

问题：

    
    bash
  
$ scontrol show job 1
slurm_load_jobs error: Invalid job id specified

原因：

MinJobAge: Controls how soon the record of a completed job can be purged from the slurmctld memory and thus not visible using the squeue command. The record of jobs run will be preserved in accounting records and logs. The default value is 300 seconds. The value should be reduced to a few seconds if possible. Use of accounting records for older jobs can increase the job throughput rate compared with retaining old jobs in the memory of the slurmctld daemon.

Requested operation is presently disabled for job

问题：

    
    bash
  
$ scontrol requeue 920
Requested operation is presently disabled for job 920

原因：

Access/permission denied for job

问题

    
    bash
  
$ scontrol suspend 160
Access/permission denied for job 160

原因

只有 admin 和 Operator 用户可以 suspend 作业。一旦用户挂起一个作业，它的 cpu 将被释放给其他作业，并可能被分配给其他作业。如果恢复挂起的作业，就会突然有两个作业同时在同一个 cpu 上运行，这两个作业的性能会显著降低。这就是为什么不允许普通用户 suspend/release 他们自己的作业。

Warning: Note very large processing time from _slurm_rpc_dump_jobs

错误日志

text

[2021-08-15T11:20:00.344] Warning: Note very large processing time from _slurm_rpc_dump_jobs: usec=7083022 began=11:19:53.261

原因

存储响应慢导致