File Transfers

Globus is the recommended method to transfer data to and from the DCC and can be used with any OIT provided research storage.

• New users: Globus Web User Guide.
• Advanced users: Globus CLI User Guide

Globus Quick Links

• Duke Compute Cluster (DCC) Data Transfer Node – This endpoint can support any OIT provided storage.
• Duke Globus-Box Connector – Allows for movement of files to and from Duke Box.
  WARNING: Duke Globus-Box Connector is intended for occasional collaboration-related file transfers only. The transfer of large numbers of individual files (e.g., 100 or more) can disrupt service. Minimize transfers and always combine (zip/tar) multiple files before moving data. Box should not be used for backups, instead use: Duke Data Attic
• HARDAC to DCC Instructions

File transfer considerations and other methods

Bundle your files before transferring

.tar your files before transferring. Transferring a large number of small files is very inefficient and will take much longer than transferring a single large file.

There are many ways to transfer files to and from the DCC. When choosing consider your network path and try to optimize transfers through directly connected (wired) systems.

For SCP and RSYNC the initiating system must be connected to the Duke network, or off campus use VPN. When executing SCP for files to or from the DCC, MFA is required, and will default to your first available option. If you are having trouble with MFA and SCP, get help with MFA, or bypass MFA all together by setting up and using SSH Keys to access the DCC.

File transfers within the DCC

SCP (Secure Copy) a single file

Command FormatPushPullSample Execution

scp source destination

$scp <localpath.file> <netid>@dcc-login.oit.duke.edu:<dccpath>

$scp <netid>@dcc-login.oit.duke.edu:<dccpath.filename> <localpath>

# Note use of ssh keys to simplfy login
# Sample command and output pushing a file from my workstation to my group directory:

kk338@CDSS-5630 ~ % scp jobs.txt kk338@dcc-login.oit.duke.edu:/hpc/group/rescomp/kk338
Enter passphrase for key '/Users/kk338/.ssh/id_rsa': 
jobs.txt                                      100% 3847KB   8.4MB/s   00:00   

# Sample command and output pulling a file from my group directory to my local workstation (note the use of `.` to denote current working directory):

kk338@CDSS-5630 ~ % scp kk338@dcc-login.oit.duke.edu:/hpc/group/rescomp/kk338/DailyUsage.xlsx .
Enter passphrase for key '/Users/kk338/.ssh/id_rsa': 
DailyUsage.xlsx                                               100%   40KB 361.1KB/s   00:00

While you can use scp -r to recursively copy all of the files in a directory, we recommend the use of rsync for a large number of files.

RSYNC for a larghe number of files

Command FormatPushPull

# -r syncs recursively 
# -P updates the modify time, useful for /work purge script analysis
rsync –rP source destination

$rsync –rP dir1/ netid@dcc-login-02.oit.duke.edu:.

$rsync –rP netid@dcc-login.oit.duke.edu:~/dir1 .