Skip to content

Optimizing Storage Performance

When using cluster storage and running parallel workloads, it is important to optimize your work to take advantage of available storage to ensure the speed and efficiency of your work as well as to avoid stressing the shared filesystem and negatively impacting performance for all users.

Storage Tiers explained

The DCC uses a hierarchical organization of storage media (called storage tiers) that differ in performance, cost, and capacity. Each option on the DCC has different intended use, performance, capabilities, and cost implications. See the full list of options here.

Using /scratch

Every DCC compute node has a local /scratch directory backed by NVMe, the highest throughput and lowest latency storage on the cluster. Staging work there speeds up I/O-heavy jobs and takes pressure off the shared filesystems. Three properties shape how you use it,

  • Node-local. Each node has its own /scratch. A multi-node job sees a different one on every node.
  • Private to the job, and temporary. Slurm gives each job its own /scratch and removes the contents when the job ends. Nothing needs cleaning up, and nothing survives.
  • Shared capacity. Size varies by node generation, from roughly 400 GB to 42 TB, and the space is shared with other jobs running on the same node. Check what a node has with df -h /scratch.

The pattern is therefore copy in, compute, copy out.

scratch-job.sh
#!/bin/bash
#SBATCH -J scratch-example
#SBATCH -p common
#SBATCH -c 8
#SBATCH --mem=32G
#SBATCH -t 04:00:00

IN=/work/$USER/project/input
OUT=/work/$USER/project/results
mkdir -p "$OUT" /scratch/in /scratch/out

# Copy results back even if the analysis fails partway through.
trap 'rsync -a /scratch/out/ "$OUT"/ || true' EXIT

# Stage in.
rsync -a "$IN"/ /scratch/in/

# Compute against node-local NVMe.
./my_analysis --input /scratch/in --output /scratch/out

Submit it with sbatch scratch-job.sh.

A few things worth knowing about that script. The trap covers a failing analysis, but a job killed at its time limit may not get the chance to run it, so request enough walltime or write periodic checkpoints straight to $OUT. Use rsync -a rather than cp so a re-run only transfers what changed. And because /scratch is node-local, a multi-node job has to stage on each node separately, for example with srun --ntasks-per-node=1 rsync -a "$IN"/ /scratch/in/.

Copying in is worth it when the job reads the same data repeatedly, works with many small files, or writes intermediate results it will read back. A job that streams a large file once, start to finish, gains little from staging and may simply pay the copy twice.

Storing Files

  • Avoid storing many files in a single directory – hundreds of files are ok, thousands are not. Our filesystem manages file locks (allows multiple nodes to access the same data concurrently safely) at the subdirectory level. Plan your data structure to use multiple sub-directories by default to minimize contention and locking events.
  • Avoid operating on many small files – the performance may depend on how you handle the data on disk. Create a smaller number of larger files.

Be aware of I/O load

  • When running multiple instance of the same job consider reading in all of the data at the beginning of the workflow, perform the entire analysis, then write the results at the end.
  • Avoid opening/closing or creating/deleting files repeatedly in tight loops. Creating and deleting files requires locking certain filesystem structures, and these “synchronization points” are computationally expensive to enforce. Try to create files “as needed” through your workflow, and minimize the number created within a single directory. Similarly, delete files “as needed” (which is less “expensive” for directories that are “properly sized” in terms of number of files). Try not to re-use filenames in the same directory, when creating and deleting files in a loop. If possible, open files once at the beginning of your workflow/program, then close them at the end.

Watch your quotas

You are limited in capacity and exceeding your storage quotas will impact performance.

Choose the right command(s) for the jobs

Avoid using ls -l in scripts – consider find which is more appropriate for scripts, if you use ls use it by itself or use ls -l <filename> if you want the long listing of a specific file.