Skip to content

Files

Overview

The DCC has several shared file systems available for all users of the cluster. General partitions are on Isilon, 40Gbps or 10Gbps network attached storage arrays.

Sensitive data is not permitted on cluster storage.

Path Size Description Backups
/work 650 TB Unpartitioned, high speed volume, shared across all users. Files older than 75 days are purged automatically. None
/cwork/<netid> 830 TB high speed volume, availabe to all users. Files older than 90 days are purged automatically. None
/hpc/home/<netid> 25 GB Use for personal scripts, and other environment setup. When a users Duke ID is deactivated, their access and home directory is automatically removed from the cluster. None
/hpc/group/<groupname> 1 TB, Expandable for a fee Private to each lab group, working space for applications and projects that last longer than 75 days. New 9/12/22: Group owners can now self-service increase group storage up to 10TB in Research Toolkits. 7 day snapshot
/datacommons/<groupname> Fee-based Archival storage that can be mounted to the DCC to ease transfer of data to computational space and results out to long term storage. Because I/O will not be as performant as with cluster storage, job file access should not be configured that will cause excessive read/write to Data Commons storage. Optional 30 day backup

OIT storage rates are set annually. Group storage is "Standard" and Data Commons is "Archive".

How should I use DCC storage?

To optimize the performance of the cluster and make your utilization efficient, we recommend the following:

  • /home -> personal scripts and configuration files, environment setup information
  • /group -> software installations, lab specific scripts, moderately sized data sets or intermediate results that are needed for longer than 75 days.

  • /work -> large data sets under analysis, intermediate results. In the root folder, create your own folder for your use with: mkdir <netid> ***Remember: Files older than 75 days are automatically purged!

  • /cwork/ -> newer version of /work for large data sets under analysis, intermediate data, and preliminary results.

    ***Remember: Files older than 90 days are automatically purged!

  • /datacommons -> long term storage for source data and results data

Some DCC compute nodes also have a /scratch volume that is local to the compute node. This can be used when highly performant storage is needed during a job, but data should be deleted at the completion of the job. /scratch is not available on every node, and sizes vary, so use of this requires strong working knowledge of the nodes on the DCC.