Skip to content

Using the Slurm REST API on DCC

This guide explains how to talk to the DCC Slurm scheduler over its REST API (slurmrestd). You will authenticate with a JSON Web Token (JWT), inspect partitions and nodes, submit and cancel jobs, and query accounting data, all with curl and jq instead of sbatch, squeue, and scancel.

The Slurm commands you normally use on DCC (sinfo, squeue, sbatch, scancel, sacct) are clients over the scheduler. DCC also exposes the same functionality through slurmrestd, Slurm's REST daemon. This is useful when you want to drive the cluster from something that is not a login shell, for example a web application, a workflow engine, a CI pipeline, a Jupyter notebook running elsewhere, or an AI agent.

The DCC Slurm REST endpoint is,

https://dcc-sched-01.rc.duke.edu:6820

Warning

The REST API acts on your behalf with your Slurm privileges. The JWT you generate below is a bearer credential, meaning anyone who holds it can submit and cancel jobs as you until it expires. Never commit a token, paste it into a shared log, or store it in a script.

Prerequisites

Most commands in this guide run from any terminal that can reach the REST endpoint, including your laptop. You need,

curl --version
jq --version
ssh -V

You also need SSH access to a DCC login node, because the JWT is minted with scontrol on the cluster.

Set your Duke NetID once so the rest of the commands can reuse it,

export SLURM_USER="your_netid"

Optionally, confirm from a login node that JWT authentication is enabled,

scontrol show config | grep -E 'AuthAltTypes|AuthAltParameters'

which should report,

AuthAltTypes            = auth/jwt
AuthAltParameters       = jwt_key=/var/spool/slurm/statesave/jwt_hs256.key

Generate a Slurm JWT

From your local terminal, ask a DCC login node to mint a token,

ssh "${SLURM_USER}@dcc-login.oit.duke.edu" 'scontrol token lifespan=3600'

This prints a single line,

SLURM_JWT=eyJhbGciOiJIUzI1NiIs...

lifespan is in seconds. Keep it short, request a new token when it expires rather than minting a long-lived one. Export the value in the terminal where you will call the API,

export SLURM_JWT='paste-token-here'

Set connection variables

export SLURM_REST_URL="https://dcc-sched-01.rc.duke.edu:6820"
export SLURM_REST_VERSION="v0.0.43"

Every authenticated request carries two headers,

X-SLURM-USER-NAME:  your NetID
X-SLURM-USER-TOKEN: your JWT

which in curl looks like,

-H "X-SLURM-USER-NAME: $SLURM_USER" \
-H "X-SLURM-USER-TOKEN: $SLURM_JWT"

DCC currently serves data parser versions v0.0.41 through v0.0.44. The examples below use v0.0.43. Field names and response shapes change between versions, so pin a version explicitly rather than assuming the newest one.

Test the connection

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/ping/" | jq '.pings'

A working connection returns,

[
  {
    "hostname": "dcc-sched-01",
    "pinged": "UP",
    "responding": true,
    "latency": 701,
    "mode": "primary",
    "primary": true
  }
]

-k skips TLS certificate verification. Drop it if your machine trusts the certificate chain for this host.

Explore the API

The endpoint publishes its own OpenAPI schema, which is the authoritative reference for every field used below.

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/openapi/v3" | jq '.info'

List the paths available for your chosen version,

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/openapi/v3" \
  | jq -r '.paths | keys[]' \
  | grep "/slurm/$SLURM_REST_VERSION"

Inspect partitions

The equivalent of sinfo,

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/partitions/" \
  | jq '.partitions[] | {name, state: .partition.state, nodes: .nodes.total, cpus: .cpus.total}'

For names only,

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/partitions/" \
  | jq -r '.partitions[].name'

See Partitions for which of these you can submit to.

Inspect nodes

The equivalent of sinfo -N,

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/nodes/" \
  | jq '.nodes[] | {name, partitions, state, cpus, real_memory}'

Filter for GPU nodes by matching on the generic resource string,

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/nodes/" \
  | jq '.nodes[] | select((.gres // "") | contains("gpu")) | {name, state, gres}'

Look at the queue

The equivalent of squeue,

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/" \
  | jq '.jobs[] | {job_id, name, user_name, partition, job_state, nodes}'

The /jobs/ endpoint has no user filter

Passing jobs/?user=$SLURM_USER returns every job plus the warning Ignoring unknown field "user" of type string. Fetch the full list and filter client-side with jq instead.

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/" \
  | jq --arg user "$SLURM_USER" \
       '.jobs[] | select(.user_name == $user) | {job_id, name, partition, job_state, nodes, reason: .state_reason}'

The queue is large, so returning only jobs updated recently is much cheaper. update_time takes a Unix timestamp,

Linux

UPDATE_TIME=$(date -d '1 hour ago' +%s)

macOS

UPDATE_TIME=$(date -v-1H +%s)

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/?update_time=$UPDATE_TIME" \
  | jq '.jobs[] | {job_id, name, user_name, job_state}'

Submit a job

Submission is a POST of a JSON body to /slurm/$SLURM_REST_VERSION/job/submit. The body has two parts: script, the batch script as a single string with escaped newlines, and job, the resource request that would otherwise live in #SBATCH directives.

Create a working directory first, since current_working_directory must exist on the cluster before the job starts,

ssh "${SLURM_USER}@dcc-login.oit.duke.edu" 'mkdir -p /work/$USER/slurm_api_test'

Then write the request body locally,

submit_job.json
{
  "script": "#!/bin/bash\nhostname\ndate\nsleep 30\necho done\n",
  "job": {
    "name": "rest-api-test",
    "account": "your_account",
    "partition": "common",
    "tasks": 1,
    "cpus_per_task": 1,
    "memory_per_node": 1024,
    "time_limit": 5,
    "current_working_directory": "/work/your_netid/slurm_api_test",
    "standard_output": "/work/your_netid/slurm_api_test/rest-api-test-%j.out",
    "standard_error": "/work/your_netid/slurm_api_test/rest-api-test-%j.err",
    "environment": [
      "PATH=/usr/local/bin:/usr/bin:/bin"
    ]
  }
}

Note the units, memory_per_node is in MB and time_limit is in minutes. environment is required. A job submitted with an empty environment will fail to launch.

Submit it and capture the job ID,

JOB_ID=$(curl -k -sS -X POST \
  -H "Content-Type: application/json" \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  --data @submit_job.json \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/job/submit" \
  | jq -r '.job_id')

echo "$JOB_ID"

A successful submission returns the job ID with empty errors and warnings arrays. Always check .errors when a submission does not behave as expected, a rejected job still returns HTTP 200 with the reason in that array.

Requesting GPUs

Generic resources map to the tres_per_node field using Slurm's TRES syntax, so --gres=gpu:1 becomes,

{
  "job": {
    "partition": "gpu-common",
    "tres_per_node": "gres/gpu:1",
    "cpus_per_task": 1,
    "memory_per_node": 4096,
    "time_limit": 5
  }
}

To request a specific GPU model, include the type as you would on the command line, for example "gres/gpu:5000_ada:1". The GPUs by type table lists the valid GRES names.

Check a job

The equivalent of scontrol show job <jobid>,

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/job/$JOB_ID" \
  | jq '.jobs[0] | {job_id, name, partition, job_state, state_reason, nodes, submit_time, start_time, end_time}'

For polling, the dedicated state endpoint is much lighter,

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/state/?job_id=$JOB_ID" \
  | jq '.jobs'


[
  {
    "job_id": "53640342",
    "state": [
      "RUNNING"
    ]
  }
]

state is an array because a job can hold several state flags at once, for example ["CANCELLED", "COMPLETING"].

Cancel a job

The equivalent of scancel <jobid>,

curl -k -sS -X DELETE \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/job/$JOB_ID" \
  | jq '{errors, warnings}'

Then confirm with the state endpoint above.

Query accounting data

Accounting lives under /slurmdb/ rather than /slurm/, and is the equivalent of sacct. Unlike the queue endpoint, it does support server-side filtering.

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurmdb/$SLURM_REST_VERSION/jobs/?users=$SLURM_USER" \
  | jq '[.jobs[] | {job_id, name, state: .state.current}] | .[0:10]'

Details for one finished job,

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurmdb/$SLURM_REST_VERSION/job/$JOB_ID" | jq

Confirm the accounting database is reachable with,

curl -k -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurmdb/$SLURM_REST_VERSION/ping/" | jq '.pings'

A helper function

The header boilerplate gets repetitive. After setting SLURM_USER, SLURM_JWT, and SLURM_REST_URL, define,

slurm_api() {
  local method="$1"
  local path="$2"
  shift 2

  curl -k -sS -X "$method" \
    -H "X-SLURM-USER-NAME: $SLURM_USER" \
    -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
    "$@" \
    "$SLURM_REST_URL$path"
}

which reduces the examples above to,

slurm_api GET "/slurm/$SLURM_REST_VERSION/ping/" | jq '.pings'
slurm_api GET "/slurm/$SLURM_REST_VERSION/partitions/" | jq -r '.partitions[].name'
slurm_api GET "/slurm/$SLURM_REST_VERSION/jobs/" | jq '.jobs[] | {job_id, user_name, job_state}'

slurm_api POST "/slurm/$SLURM_REST_VERSION/job/submit" \
  -H "Content-Type: application/json" \
  --data @submit_job.json | jq

slurm_api DELETE "/slurm/$SLURM_REST_VERSION/job/$JOB_ID" | jq

Connecting through an SSH tunnel

Direct access to port 6820 works from many Duke networks. If yours blocks it, forward the port through a login node,

ssh -L 6820:dcc-sched-01.rc.duke.edu:6820 "${SLURM_USER}@dcc-login.oit.duke.edu"

Leave that session open, and in another local terminal point the API at the forwarded port,

export SLURM_REST_URL="https://localhost:6820"

Everything else, including token generation, is unchanged. -k is required here, since the certificate is issued for dcc-sched-01, not localhost.

Troubleshooting

curl: Failed to connect to dcc-login.oit.duke.edu port 6820

The REST service does not run on the login hosts. Use https://dcc-sched-01.rc.duke.edu:6820.

400 The plain HTTP request was sent to HTTPS port

Use https://, not http://.

401 Authentication failure

The token is missing, expired, malformed, or does not match X-SLURM-USER-NAME. Mint a fresh one with scontrol token.

SSL certificate problem

Pass -k, or add the Duke certificate chain to your local trust store.

Ignoring unknown field "user"

/slurm/<version>/jobs/ takes no user query parameter. Filter with jq, or use /slurmdb/<version>/jobs/?users= for accounting records.

Submission returns HTTP 200 but no job_id

Read the .errors array. Account, QoS, partition, and memory validation failures all report there. Match the account, partition, qos, and resource fields to what your normal sbatch submission uses.

Job submits but fails immediately

Check that current_working_directory exists and that environment is populated. Both are common causes of a job that starts and exits at once.

Quick reference

# Connection
export SLURM_USER="your_netid"
export SLURM_REST_URL="https://dcc-sched-01.rc.duke.edu:6820"
export SLURM_REST_VERSION="v0.0.43"

# Token
ssh "${SLURM_USER}@dcc-login.oit.duke.edu" 'scontrol token lifespan=3600'
export SLURM_JWT='paste-token-here'

# Ping
curl -k -sS -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/ping/" | jq '.pings'

# Your jobs in the queue
curl -k -sS -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/" \
  | jq --arg user "$SLURM_USER" '.jobs[] | select(.user_name == $user) | {job_id, name, job_state}'

# Job state
curl -k -sS -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/jobs/state/?job_id=$JOB_ID" | jq '.jobs'

# Submit
curl -k -sS -X POST -H "Content-Type: application/json" \
  -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  --data @submit_job.json \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/job/submit" | jq

# Cancel
curl -k -sS -X DELETE -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurm/$SLURM_REST_VERSION/job/$JOB_ID" | jq

# Accounting
curl -k -sS -H "X-SLURM-USER-NAME: $SLURM_USER" -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  "$SLURM_REST_URL/slurmdb/$SLURM_REST_VERSION/jobs/?users=$SLURM_USER" \
  | jq '[.jobs[] | {job_id, name, state: .state.current}] | .[0:10]'

For the command-line equivalents of everything above, see Scheduling Jobs.

Comments