Scheduling jobs with Slurm

Slurm is the workload manager for the Duke Compute Cluster (DCC). Use it to request resources and run work on compute nodes rather than on the login nodes.

Most DCC partitions contain lab-owned nodes. Members of the owning group receive high-priority access through the corresponding lab partition. See Partitions for general-use partitions, access policies, preemption, and walltime limits.

The current Introduction to the DCC slides provide additional background.

Slurm commands

Command	Description
`sbatch`	Submit a batch script
`srun`	Run a command or start an interactive job
`squeue`	List queued and running jobs
`scancel`	Cancel one or more jobs
`sinfo`	Show partition and node information
`scontrol`	Show detailed job, node, partition, and cluster configuration
`sacct`	Show accounting data for current and completed jobs

#SBATCH is not a shell command. It introduces Slurm directives near the beginning of a batch script.

Submitting a batch job

A batch script contains the requested resources followed by the commands to execute. For example, save the following as test.sh,

test.sh

#!/bin/bash
#SBATCH --job-name=hostname
#SBATCH --partition=common
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=00:10:00
#SBATCH --output=hostname-%j.out

# Commands to run
hostname

Submit the script from a login node,

sbatch test.sh

Slurm returns a job ID,

Submitted batch job 186554

When the job finishes, its standard output is written to hostname-186554.out. %j in an output filename expands to the job ID.

Options can also be supplied to sbatch. Command-line options override matching directives in the script,

sbatch --partition=scavenger --time=01:00:00 test.sh

Specify an account when a partition requires one,

sbatch --partition=<partition_name> --account=<account_name> test.sh

In general, the partition and account names are the same for lab-owned resources. See the sbatch documentation for all submission options.

Monitoring and managing jobs

List your jobs,

squeue --me

Show detailed information about a job,

scontrol show job <job_id>

Cancel a job,

scancel <job_id>

Show accounting information after a job starts or finishes,

sacct -j <job_id> --format=JobID,State,Elapsed,AllocCPUS,MaxRSS,ExitCode

Running an interactive job

Use an interactive allocation for testing, debugging, or other work that requires a shell on a compute node. This example requests one CPU, 4 GB of memory, and one hour on common,

srun --partition=common --cpus-per-task=1 --mem=4G --time=01:00:00 --pty bash -i

The shell prompt changes after Slurm allocates a node,

tm103@dcc-login-02 ~ $ srun --partition=common --mem=4G --time=01:00:00 --pty bash -i
srun: job 186535 queued and waiting for resources
srun: job 186535 has been allocated resources
tm103@dcc-core-ferc-u-ab39-5-1 ~ $

Specify an account when using a restricted partition,

srun --partition=<partition_name> --account=<account_name> --mem=4G --time=01:00:00 --pty bash -i

Exit the shell to end the interactive job,

exit

Requesting resources

Request only the resources the job will use. Oversized requests can increase queue time and prevent other users from using idle capacity.

Time

Set a walltime with --time. Common formats are HH:MM:SS and D-HH:MM:SS,

#SBATCH --time=04:00:00

The job is terminated if it exceeds the requested walltime. Partition maxima are listed in Partitions.

Memory

The default memory allocation is 2 GB per allocated CPU. This is a hard limit, so request enough memory for the peak usage of the job.

Directive	Meaning
`--mem=<size>`	Total memory per node, such as `--mem=16G`
`--mem-per-cpu=<size>`	Memory per allocated CPU, such as `--mem-per-cpu=2G`

--mem and --mem-per-cpu are mutually exclusive.

CPUs, tasks, and nodes

Directive	Meaning
`-N`, `--nodes`	Number of nodes
`-n`, `--ntasks`	Total number of tasks, normally MPI ranks
`--ntasks-per-node`	Number of tasks on each node
`-c`, `--cpus-per-task`	CPUs allocated to each task, normally threads

Use --cpus-per-task for a multithreaded program,

#!/bin/bash
#SBATCH --job-name=threaded
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=500M
#SBATCH --time=01:00:00
#SBATCH --output=threaded-%j.out

myApplication -n "$SLURM_CPUS_PER_TASK"

OpenMP multicore example

For OpenMP, set the thread count from the Slurm allocation,

#!/bin/bash
#SBATCH --job-name=openmp
#SBATCH --cpus-per-task=4
#SBATCH --mem=4G
#SBATCH --time=01:00:00

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
myOpenMPApplication

R multicore example

The R-openblas module links R against a multi-threaded OpenBLAS, so dense linear algebra routines such as matrix multiplication (%*%, crossprod), Cholesky decomposition (chol), solving linear systems (solve), and eigen-decomposition (eigen), uses the CPUs allocated with --cpus-per-task without changes to the R code. Set the BLAS thread count from the Slurm allocation,

R_job.sh

#!/bin/bash
#SBATCH --job-name=r-analysis
#SBATCH --cpus-per-task=4
#SBATCH --mem=4G
#SBATCH --time=01:00:00
#SBATCH --output=r-%j.out
#SBATCH --error=r-%j.err

module load R-openblas/4.6.1
export OPENBLAS_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
Rscript Rcode.R

Submit the job with sbatch R_job.sh as usual. See Running R in Parallel on DCC for details.

MPI multi-node example

DCC provides MPICH and OpenMPI through environment modules. Compile and test on a compute node requested through an interactive job.

For Open MPI,

module load OpenMPI/4.1.6
mpicc -o openhello hello.c

For MPICH,

module load mpich/5.0.1
mpicc -o openhello hello.c

An example two-node batch script with 10 MPI ranks per node is,

#!/bin/bash
#SBATCH --job-name=openhello
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=10
#SBATCH --time=01:00:00
#SBATCH --output=openhello-%j.out
#SBATCH --error=openhello-%j.err

module load OpenMPI/4.1.6
mpirun -n "$SLURM_NTASKS" ./openhello > openhello.out

If openhello prints its rank and hostname, its output will show ranks distributed across the allocated nodes. Rank output can arrive in any order,

dcc-core-01, rank 0 out of 20 processors
dcc-core-01, rank 1 out of 20 processors
dcc-core-01, rank 2 out of 20 processors
dcc-core-01, rank 3 out of 20 processors
dcc-core-01, rank 4 out of 20 processors
dcc-core-02, rank 13 out of 20 processors
dcc-core-02, rank 14 out of 20 processors
dcc-core-02, rank 10 out of 20 processors
dcc-core-02, rank 11 out of 20 processors
dcc-core-02, rank 12 out of 20 processors
dcc-core-02, rank 8 out of 20 processors
dcc-core-02, rank 9 out of 20 processors
dcc-core-02, rank 5 out of 20 processors
...

The DCC is heterogeneous. Multi-node MPI jobs should use homogeneous nodes with compatible interconnects; see Node configurations and CPU architectures. For a worked multi-node example, see Running a Multi-Node MPI Job on DCC.

Slurm job arrays

Job arrays submit many independent tasks from one batch script. Each task receives a different SLURM_ARRAY_TASK_ID.

Add --array or -a to the batch script or sbatch command.
An array can contain at most 50,001 tasks, with indices from 0 through 50,000.
Array tasks must be independent.
%A in a filename expands to the array job ID and %a expands to the array index.

Common index specifications include,

# Continuous indices: 0 through 31
#SBATCH --array=0-31

# Selected indices: 3, 5, 7, 8, 9, and 12
#SBATCH --array=3,5,7-9,12

# At most 20 array tasks running simultaneously
#SBATCH --array=0-4999%20

# A single array index
#SBATCH --array=7

The array can instead be specified during submission,

sbatch --array=1-30 array-job.sh

A single failed task can be rerun by submitting only its index,

sbatch --array=7 array-job.sh

Use a discontinuous list to rerun several selected tasks,

sbatch --array=3,5,7-9,12 array-job.sh

For example, this three-task array writes each task ID to a separate output file,

#!/bin/bash
#SBATCH --job-name=array-id
#SBATCH --array=1-3
#SBATCH --output=slurm-%A_%a.out

echo "$SLURM_ARRAY_TASK_ID"

Submit it normally,

sbatch array-id.sh

After the array finishes, the three output files contain 1, 2, and 3, respectively,

slurm-24845830_1.out
slurm-24845830_2.out
slurm-24845830_3.out

Python job array example

#!/bin/bash
#SBATCH --job-name=python-array
#SBATCH --array=1-5000
#SBATCH --mem=2G
#SBATCH --time=01:00:00
#SBATCH --output=slurm-%A_%a.out
#SBATCH --error=slurm-%A_%a.err

python task.py

Read the array index in task.py,

task.py

import os
task_id = int(os.environ["SLURM_ARRAY_TASK_ID"])

Processing separate input files

For an array indexed from 1 through the number of lines in files.txt,

#!/bin/bash
#SBATCH --array=1-100

mapfile -t files < files.txt
input_file=${files[$((SLURM_ARRAY_TASK_ID - 1))]}
myApplication "$input_file"

To construct the input list from a shell glob instead of a file, use a Bash array.

#!/bin/bash
#SBATCH --array=1-100

shopt -s nullglob
files=(myfile*)
index=$((SLURM_ARRAY_TASK_ID - 1))

if ((index >= ${#files[@]})); then
    echo "No input file for array index $SLURM_ARRAY_TASK_ID" >&2
    exit 1
fi

myApplication "${files[$index]}"

Set the array range to the number of matching files. For a reproducible mapping, keep the filenames stable between submission and execution or generate files.txt before submitting the array.

Alternatively, encode the array index directly in filenames and output directories,

#!/bin/bash
#SBATCH --array=1-100
#SBATCH --output=slurm-%A_%a.out
#SBATCH --error=slurm-%A_%a.err

output_dir="out_${SLURM_ARRAY_TASK_ID}"
mkdir -p "$output_dir"
cd "$output_dir" || exit 1
myApplication "../input_${SLURM_ARRAY_TASK_ID}.txt"

This creates directories out_1, out_2, and so on for input files input_1.txt, input_2.txt, and so on.

Unrolling a loop into an array

Consider a serial Python loop,

fibonacci = [0, 1, 1, 2, 3, 5, 8, 13, 21]

for i in range(len(fibonacci)):
    print(i, fibonacci[i])

Move one loop iteration into fibonacci.py by reading the array index,

import os

fibonacci = [0, 1, 1, 2, 3, 5, 8, 13, 21]
i = int(os.environ["SLURM_ARRAY_TASK_ID"])
print(i, fibonacci[i])

Save the following as fibonacci-array.sh to request one array task for each index,

#!/bin/bash
#SBATCH --job-name=fibonacci
#SBATCH --array=0-8
#SBATCH --output=fibonacci-%A_%a.out
#SBATCH --error=fibonacci-%A_%a.err

python fibonacci.py

Submit the script as usual,

sbatch fibonacci-array.sh

Submitted batch job 24856052

After the tasks finish, list the per-index output files,

ls fibonacci-24856052_*.out

fibonacci-24856052_0.out
fibonacci-24856052_1.out
fibonacci-24856052_2.out
fibonacci-24856052_3.out
fibonacci-24856052_4.out
fibonacci-24856052_5.out
fibonacci-24856052_6.out
fibonacci-24856052_7.out
fibonacci-24856052_8.out

After all tasks finish, concatenate the output files in index order,

cat fibonacci-24856052_{0..8}.out

This gives the same result as the serial loop,

Each task performs one independent iteration of the original loop. The tasks can finish in any order; the array index and output filename preserve the mapping to the original iteration. This pattern is appropriate only when iterations do not depend on results from earlier iterations.

Reading Slurm variables

Python

import os

num_cpus = int(os.environ["SLURM_CPUS_PER_TASK"])
task_id = int(os.environ["SLURM_ARRAY_TASK_ID"])

R

num_cpus <- as.integer(Sys.getenv("SLURM_CPUS_PER_TASK"))
task_id <- as.integer(Sys.getenv("SLURM_ARRAY_TASK_ID"))

MATLAB

numCPUs = str2double(getenv('SLURM_CPUS_PER_TASK'));
taskID = str2double(getenv('SLURM_ARRAY_TASK_ID'));

See the Slurm job array documentation for additional options.

Job dependencies

Submit a job that starts only after another job completes successfully,

first_job=$(sbatch --parsable first.sh)
sbatch --dependency="afterok:${first_job}" second.sh

Common dependency types are,

Dependency	Meaning
`after:<job_id>`	Start after the specified job begins or is cancelled
`afterok:<job_id>`	Start after the specified job completes successfully
`afterany:<job_id>`	Start after the specified job completes in any state
`afternotok:<job_id>`	Start after the specified job fails

Wait for specific array tasks,

sbatch --dependency=afterok:123_4:123_8 next-job.sh

Wait for an entire array to complete successfully,

sbatch --dependency=afterok:123 next-job.sh

See the Slurm dependency documentation for the complete syntax.

GPU jobs

Select a GPU partition and request a random GPU as a generic resource,

#!/bin/bash
#SBATCH --job-name=gpu-job
#SBATCH --partition=gpu-common
#SBATCH --gres=gpu:1
#SBATCH --mem=16G
#SBATCH --time=01:00:00
#SBATCH --output=gpu-%j.out

nvidia-smi

To start an interactive session with one GPU,

srun --partition=gpu-common --gres=gpu:1 --mem=16G --time=01:00:00 --pty bash -i

Cluster GPU types

Request a specific GPU type and count with,

#SBATCH --gres=gpu:<gpu_type>:<num_gpus>

For example, request one RTX 5000 Ada GPU,

#SBATCH --gres=gpu:5000_ada:1

The GPUs by type table is the authoritative list of GPU models, VRAM, and Slurm GRES names on DCC.

Checking GPU availability

From a DCC login node, gpuavail summarizes GPU configurations and currently available GPUs.

gpuavail -p <partition_name>

E.g.,

$ gpuavail -p gpu-common

      CONFIGURATION
  NODE                           NODE
  TYPE                          COUNT
  ---------------------
  gpu:a5000:1                    10
  gpu:2080:4                      5
  gpu:5000_ada:8                  3
  gpu:2080:3                      1


                  AVAILABILITY
  NODE                            GPU       GPU     GPUs    CPUs    GB MEM
  NAME                            TYPE      COUNT   AVAIL   AVAIL   AVAIL
  ------------------------------------------------------------------
  dcc-core-gpu-ferc-s-p15-20      2080      3       2       86      690

The slurm-usage tool provides a current usage summary by GPU type. Limit it to one or more comma-separated partitions with,

slurm-usage -r <partition_name>