Measuring Efficiency
The DCC is a shared resource. Slurm reserves everything a job asks for, whether or not the job uses it, so CPU cores, memory, and GPUs that were requested but left idle are unavailable to everyone else for the life of the job. Right-sizing requests also helps you as smaller requests are easier for the scheduler to place and generally start sooner. On billed allocations such as h200-hp, an account's GPU-hour quota is charged for every GPU allocated, for as long as the job runs. A job that ends early is charged only for the time it actually ran.
This page covers the tools available on DCC for measuring what your jobs actually consumed and for turning those numbers into better resource requests.
| Tool | Use for |
|---|---|
seff |
CPU and memory efficiency of a single completed job |
| sacct | Raw accounting fields, including peak memory per job step |
| slurm-report | CPU, memory, and GPU efficiency across many jobs over a date range |
| slurm-stat | Live CPU, memory, and GPU statistics for currently running jobs |
Measuring CPU and RAM efficiency
To check how efficiently a job used its allocated CPU cores and memory, run seff from a login node,
E.g.,
$ seff 25623378
Job ID: 25623378
Cluster: dcc
User/Group: ukh/dukeusers
State: COMPLETED (exit code 0)
Nodes: 5
Cores per node: 84
CPU Utilized: 1-03:31:25
CPU Efficiency: 46.44% of 2-11:16:00 core-walltime
Job Wall-clock time: 00:08:28
Memory Utilized: 1.31 TB
Memory Efficiency: 64.02% of 2.05 TB (5.00 GB/core)
CPU Efficiency is the fraction of the allocated core-walltime (cores × wall-clock time) that was actually spent computing. Memory Efficiency is the peak memory used relative to the memory requested. Low values indicate the job requested far more CPU cores or RAM than it needed. Use these numbers to right-size your --cpus-per-task, --ntasks, and --mem requests in future submissions so that CPU and RAM allocations match your job's actual needs.
Warning
seff only reports meaningful values once a job has completed. For a running job, CPU and memory utilization show as 0.
Monitoring running CPU jobs
Use the slurm-stat tool to monitor CPU and memory efficiency of currently running jobs,
E.g.,
$ slurm-stat -u ${USER}
Note: Filtering to user 'ukh' but you can only see detailed statistics for your own jobs.
Monitoring 1 jobs...
Processing job 54406921 (1/1)...
┌──────┬──────────┬─────────┬──────────┬─────────┬────────┬────────┬────────┬─────────┬───────────┬────────┬─────────────┐
│ User ┆ JobID ┆ State ┆ Elapsed ┆ TimeEff ┆ CPUEff ┆ MemEff ┆ GPUEff ┆ GPUUtil ┆ GPUMemEff ┆ GPUMem ┆ Partition │
╞══════╪══════════╪═════════╪══════════╪═════════╪════════╪════════╪════════╪═════════╪═══════════╪════════╪═════════════╡
│ ukh ┆ 54406921 ┆ RUNNING ┆ 00:00:44 ┆ 0.3% ┆ 0.1% ┆ 0.1% ┆ --- ┆ --- ┆ --- ┆ --- ┆ interactive │
└──────┴──────────┴─────────┴──────────┴─────────┴────────┴────────┴────────┴─────────┴───────────┴────────┴─────────────┘
Important
slurm-stat uses sstat in the backend to sample running averages of CPU and memory usage for job steps. Metrics from completed jobs through seff are more accurate than the running averages reported by slurm-stat.
Querying CPU and memory accounting
For detailed accounting information query the Slurm database directly with sacct. MaxRSS reports peak resident memory and is recorded per job step. TotalCPU is the sum of the SystemCPU and UserCPU time used by the job or job step. The total CPU time of the job may exceed the job's elapsed time for jobs that include multiple job steps or multiple cores.
E.g.,
$ sacct -j 53179553 --units=G --format=JobID,JobName%16,State,Elapsed,AllocCPUS,ReqMem,MaxRSS,TotalCPU
JobID JobName State Elapsed AllocCPUS ReqMem MaxRSS TotalCPU
------------ ---------------- ---------- ---------- ---------- ---------- ---------- ----------
53179553 test_gpu_burn-1+ COMPLETED 00:10:56 8 32G 00:10.743
53179553.ba+ batch COMPLETED 00:10:56 8 1.42G 00:10.743
53179553.ex+ extern COMPLETED 00:10:56 8 00:00:00
Measuring GPU efficiency
GPUs are the most contended resource on the cluster, and a GPU job that spends most of its time waiting on data loading or on the CPU holds a device that another job could use. The slurm-report tool reports GPU utilization alongside CPU and memory efficiency for a set of completed jobs.
From a login node, run the following command to check the efficiency of your jobs over a time range,
E.g.,
$ slurm-report -r scavenger-h200 -S 2026-03-10 -E 2026-03-20 -u ${USER}
┌──────┬────────────┬──────────────┬──────────┬─────────┬────────┬────────┬────────┬─────────┬───────────┬────────┬────────────────┐
│ User ┆ JobID ┆ State ┆ Elapsed ┆ TimeEff ┆ CPUEff ┆ MemEff ┆ GPUEff ┆ GPUUtil ┆ GPUMemEff ┆ GPUMem ┆ Partition │
╞══════╪════════════╪══════════════╪══════════╪═════════╪════════╪════════╪════════╪═════════╪═══════════╪════════╪════════════════╡
│ ukh ┆ 44235552_0 ┆ COMPLETED ┆ 00:10:53 ┆ 90.7% ┆ 2.1% ┆ 60.0% ┆ 100.0% ┆ 100% ┆ 89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh ┆ 44235552_1 ┆ COMPLETED ┆ 00:10:53 ┆ 90.7% ┆ 2.1% ┆ 60.7% ┆ 100.0% ┆ 100% ┆ 89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh ┆ 44235552_2 ┆ COMPLETED ┆ 00:10:53 ┆ 90.7% ┆ 2.1% ┆ 59.8% ┆ 100.0% ┆ 100% ┆ 89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh ┆ 44235552_3 ┆ COMPLETED ┆ 00:10:53 ┆ 90.7% ┆ 2.1% ┆ 60.3% ┆ 100.0% ┆ 100% ┆ 89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh ┆ 44235552_4 ┆ COMPLETED ┆ 00:10:53 ┆ 90.7% ┆ 2.1% ┆ 60.0% ┆ 100.0% ┆ 100% ┆ 89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh ┆ 44235552_5 ┆ COMPLETED ┆ 00:10:47 ┆ 89.9% ┆ 2.0% ┆ 61.2% ┆ 100.0% ┆ 100% ┆ 89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh ┆ 44235552_6 ┆ COMPLETED ┆ 00:10:48 ┆ 90.0% ┆ 3.5% ┆ 61.8% ┆ 100.0% ┆ 100% ┆ 89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh ┆ 44235552_7 ┆ COMPLETED ┆ 00:10:42 ┆ 89.2% ┆ 2.0% ┆ 64.9% ┆ 100.0% ┆ 100% ┆ 89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh ┆ 44235552_8 ┆ COMPLETED ┆ 00:10:47 ┆ 89.9% ┆ 2.0% ┆ 63.2% ┆ 100.0% ┆ 100% ┆ 89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh ┆ 44235552_9 ┆ COMPLETED ┆ 00:10:41 ┆ 89.0% ┆ 1.9% ┆ 66.7% ┆ 100.0% ┆ 100% ┆ 89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ┆ ┆ WEIGHTED AVG ┆ 01:48:10 ┆ ┆ 2.2% ┆ 61.8% ┆ 100.0% ┆ 100.0% ┆ 89.7% ┆ --- ┆ │
└──────┴────────────┴──────────────┴──────────┴─────────┴────────┴────────┴────────┴─────────┴───────────┴────────┴────────────────┘
For GPU workflows, the quantities that matter the most are GPUEff, GPUMemEff, and TimeEff.
- The GPU efficiency (
GPUEff) is the normalized per-GPU average utilization which represents the percentage of time GPU compute resources were actively engaged as reported bynvidia-smi. - The GPU memory efficiency (
GPUMemEff) represents the percentage of GPU memory that was actively used during the job. TimeEffis the job's elapsed time as a fraction of the requested walltime.
The lower CPU and memory efficiency values are not necessarily indicative of a problem for GPU jobs. A --- entry means the metric was not sampled by Slurm or that the job did not request GPUs.
Run slurm-report --help for the complete list of options.
Time-weighted averages
The last row of the output table shows the time-weighted average across all jobs in the specified time range. This quantity is formulated as,
where \(i\) is the index for your jobs, \(GPUEff_i\) is the GPU efficiency of job \(i\), and \(\text{time}_i\) is the elapsed time of job \(i\). The same formula applies for the other efficiency columns. Weighting by elapsed time keeps a handful of very short jobs from dominating the summary.
Monitoring running GPU jobs
slurm-stat reports the same metrics for jobs that are currently running. Use it to catch a misconfigured job early rather than discovering the problem after it has held a GPU for a day,
E.g.,
$ slurm-stat -u ${USER}
Note: Filtering to user 'ukh' but you can only see detailed statistics for your own jobs.
Monitoring 1 jobs...
Processing job 54410392 (1/1)...
┌──────┬──────────┬─────────┬──────────┬─────────┬────────┬────────┬────────┬─────────┬───────────┬────────┬───────────┐
│ User ┆ JobID ┆ State ┆ Elapsed ┆ TimeEff ┆ CPUEff ┆ MemEff ┆ GPUEff ┆ GPUUtil ┆ GPUMemEff ┆ GPUMem ┆ Partition │
╞══════╪══════════╪═════════╪══════════╪═════════╪════════╪════════╪════════╪═════════╪═══════════╪════════╪═══════════╡
│ ukh ┆ 54410392 ┆ RUNNING ┆ 00:00:32 ┆ 4.8% ┆ 1.6% ┆ 3.8% ┆ 100.0% ┆ 100.0% ┆ 89.7% ┆ 125.6G ┆ h200-hp │
└──────┴──────────┴─────────┴──────────┴─────────┴────────┴────────┴────────┴─────────┴───────────┴────────┴───────────┘
For a direct look at a GPU while a job runs, attach to the allocation and run nvidia-smi or gpustat (installable via pip install gpustat),
E.g.,
$ srun --jobid=54410392 --overlap --pty nvidia-smi
Fri Sep 4 23:53:41 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.43.02 KMD Version: 610.43.02 CUDA UMD Version: 13.3 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA H200 On | 00000000:29:00.0 Off | 0 |
| N/A 52C P0 698W / 700W | 128595MiB / 143771MiB | 100% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 3108923 C gpu_burn 12858... |
+-----------------------------------------------------------------------------------------+
$ srun --jobid=54410392 --overlap --pty gpustat
dcc-h200-gpu-ferc-l-ac46-25 Fri Sep 4 23:54:16 2026 610.43.02
[0] NVIDIA H200 | 51°C, 100 % | 128594 / 143771 MB | ukh(128586M)
You may also ssh into the node you are running your job on and issue the nvidia-smi or gpustat commands directly.
Querying GPU accounting
Similar to CPU and memory efficiency, you can query the Slurm database for GPU efficiency metrics with sacct by passing in the TRESUsageInTot field,
E.g.,
$ sacct -j 54410392 --units=G --format=JobID,JobName%16,State,AllocTRES%15,TRESUsageInTot%80
JobID JobName State AllocTRES TRESUsageInTot
------------ ---------------- ---------- --------------- --------------------------------------------------------------------------------
54410392 test_gpu_burn-1+ COMPLETED billing=1,cpu=+
54410392.ba+ batch COMPLETED cpu=8,gres/gpu+ cpu=00:00:11,energy=0,fs/disk=0.02G,gres/gpumem=125.57G,gres/gpuutil=100,mem=1.+
54410392.ex+ extern COMPLETED billing=1,cpu=+ energy=0
54410392.0 nvidia-smi COMPLETED cpu=8,gres/gpu+ cpu=00:00:00,energy=0,fs/disk=0,gres/gpumem=0,gres/gpuutil=0,mem=0.00G,pages=0,+
54410392.1 gpustat COMPLETED cpu=8,gres/gpu+ cpu=00:00:00,energy=0,fs/disk=0,gres/gpumem=0,gres/gpuutil=0,mem=0.00G,pages=0,+
The quantities gres/gpuutil and gres/gpumem report the GPU utilization and GPU memory usage, respectively, for each job step. Both are summed across all GPUs allocated to the step.
Acknowledgements
The slurm-report and slurm-stat tools deployed on the DCC were developed by Joe Shamblin of Duke CS. slurm-stat and slurm-report are symlinks into the parent slurm-gpu tool.