Skip to content

File Transfers

Globus is the recommended method to transfer data to and from the DCC and can be used with any OIT provided research storage.

File transfer considerations and other methods

Bundle your files before transferring

.tar your files before transferring. Transferring a large number of small files is very inefficient and will take much longer than transferring a single large file.

There are many ways to transfer files to and from the DCC. When choosing consider your network path and try to optimize transfers through directly connected (wired) systems.

For SCP and RSYNC the initiating system must be connected to the Duke network, or off campus use VPN. When executing SCP for files to or from the DCC, MFA is required, and will default to your first available option. If you are having trouble with MFA and SCP, get help with MFA, or bypass MFA all together by setting up and using SSH Keys to access the DCC.

File transfers within the DCC

SCP (Secure Copy) a single file 
scp source destination

$scp <localpath.file> <netid>@dcc-login.oit.duke.edu:<dccpath>

$scp <netid>@dcc-login.oit.duke.edu:<dccpath.filename> <localpath>

# Note use of ssh keys to simplfy login
# Sample command and output pushing a file from my workstation to my group directory:

kk338@CDSS-5630 ~ % scp jobs.txt kk338@dcc-login.oit.duke.edu:/hpc/group/rescomp/kk338
Enter passphrase for key '/Users/kk338/.ssh/id_rsa': 
jobs.txt                                      100% 3847KB   8.4MB/s   00:00   

# Sample command and output pulling a file from my group directory to my local workstation (note the use of `.` to denote current working directory):

kk338@CDSS-5630 ~ % scp kk338@dcc-login.oit.duke.edu:/hpc/group/rescomp/kk338/DailyUsage.xlsx .
Enter passphrase for key '/Users/kk338/.ssh/id_rsa': 
DailyUsage.xlsx                                               100%   40KB 361.1KB/s   00:00

While you can use scp -r to recursively copy all of the files in a directory, we recommend the use of rsync for a large number of files.

RSYNC for a larghe number of files 
# -r syncs recursively 
# -P updates the modify time, useful for /work purge script analysis
rsync –rP source destination

$rsync –rP dir1/ netid@dcc-login-02.oit.duke.edu:.

$rsync –rP netid@dcc-login.oit.duke.edu:~/dir1 .