You are visiting the archived wiki pages which are for previous system and NO LONGER SUPPORTED.
Please go to wiki.hpcc.msu.edu the latest HPCC wiki site.

Skip to end of metadata
Go to start of metadata

For CentOS 6 nodes and Moab/Torque scheduling system only

The HPCC is migrating to a new operating system and scheduler. For more information, see "Introduction to new 2018 HPC systems".

For the new 2018 system, please see File and Data Transfer documentation

This document highlights several simple methods to transfer files to the HPCC home and research directories.  There are two main systems for copying files.   First,  simply "hpcc.msu.edu"  which is our main log-in gateway.   It can be used for file transfer but may have high traffic, is not meant for demanding file transfers (very large files or many files), and cannot access the scratch disk (/mnt/ls15/scratch or /mnt/scratch ).      We offer  a second gateway designed for file transfer, and does have access to the scratch file system with the host name  "rsync.hpcc.msu.edu"     =  Therefore we highly recommend you use rsync.hpcc.msu.edu for the 'host' or 'server' in all examples below, as it's the only way to transfer files from scratch.    Note that while it's named for the popular unix 'rsync' command, it can be used for sftp or scp as well. 

Using a GUI (Windows/OS X/Linux)

  1. Download and install the appropriate (free) Filezilla client from http://filezillaproject.org/download.php.
  2. To use, launch the program. You should see a screen similar to this:
  3. In the top dialog boxes, enter:
    • (Host) rsync.hpcc.msu.edu
    • (Username) <your username>
    • (Password) <your password>
    • (Port) 22
      Then click connect or quickconnect. The first time you use this, you will have to accept the host certificate.
  4. Once connected, the left column displays files on your local computer, the right column displays files on hpcc.
  5. You can select the appropriate directories by double clicking through each tree. Files can be dragged and dropped from one column to the next. (By dragging files from the left column to the right, you are uploading files to HPCC from your local computer. By dragging files from the right column to the left, you can download files from HPCC to your local computer.

Mapping Your Home or Research Space to your local computer

You can copy files using Windows Explorer or OSX Finder if you are on campus by using the following instructions. Mapping HPC drives to a campus computer with SMB Legacy

Using Dropbox (Windows/OS X/Linux)

If you are a Dropbox user, you can setup HPCC to sync automatically with your Dropbox account.

  1. Download the following file to your home directory on HPCC, http://www.dropbox.com/download/?plat=lnx.x86_64
  2. Log into one of the development nodes

      ssh dev-intel10 
  3. untar the downloaded file using the following command

     tar -zxvf dropbox-lnx.x86_64-x.x.xx.tar.gz
  4. start screen

     screen 
  5. run dropboxd in a screen session

     ~/.dropbox-dist/dropboxd 
  6. You should see output like this:

    This client is not linked to any account... Please visit https://www.dropbox.com/cli_link?host_id=7d44a557aa58f285f2da0x67334d02c1 to link this machine.  
  7. Copy the link to a web browser to activate your installation.
  8. After the client is registered, detach the screen session by pressing ctrl-a, and then d.

Using Unix commands

A number of different command-line utilities are available to OS X and Linux users. Each of them has its own advantages.

Basic file copy (scp)

A simple command for transferring files between the cluster and another host is scp. To copy a file from a local directory to file space on the cluster, use a line like

scp example.txt username@rsync.hpcc.msu.edu:example_copy.txt

This will copy the file named example.txt in the local host's home directory to the user's home directory on the cluster, with the copy having the name example_copy.txt. Leaving the space after the colon blank gives the new file the same name as the original.  Note: To transfer a file name with spaces you must put a backslash before each space in your file name, i.e.
scp "My File Name" username@hpcc.msu.edu:"My\ File\ Name"

To copy a file from the cluster to your local directory,

scp username@rsync.hpcc.msu.edu:example.txt ./example_copy.txt

will copy the file named example.txt from the user's home directory on the cluster to the home directory of the local host, naming the new file example_copy.txt. Leaving the space after the slash blank gives the new file the same name as the original. The -r option can be used to copy entire directories recursively. 

Synchronize directories (rsync)

If you are an advanced LINUX/Mac user, there is a wonderful little utility that makes mirroring directories simple. The syntax looks very similar to scp.

  • To mirror <local_dir> on my local computer to <hpcc_dir> on hpcc, the following command can be issued.

    rsync -ave ssh <local_dir> username@rsync.hpcc.msu.edu:<hpcc_dir>
    

    In the above command, rsync will scan through both directories. If any files in the <local_dir> are newer, they will be uploaded to <hpcc_dir>. (It is also possible to get rsync to upload ALL different files, regardless of which is newer).

  • To mirror the HPCC directory to your local system, call

    rsync -ave ssh username@rsync.hpcc.msu.edu:<hpcc_dir> <local_dir>
    

    the first time you use rsync, you might want to add the -n flag to do a dry run before any files are copied.

Interactive file copy (sftp)

When preforming several data transfers between hosts, the sftp command may be preferable, as it allows the user to work interactively. Running

sftp username@rsync.hpcc.msu.edu

from a local host establishes a connection between that host and the cluster. Both hosts can be navigated. For the local file system, lcd changes to the specified directory, lpwd prints the working directory, and lls prints a list of files in the current directory. For the remote file system, the same three commands are available, minus the leading "l." Also available are commands to change permissions, rename files, and manipulate directories on the remote host. The two key commands are get example.txt, which copies the file in the remote working directory to the local working directory, and put example.txt, which copies the file in the local working directory to the remote working directory. The quit command closes the connection between hosts.

Copy file from Internet (Wget)

Wget is a simple command useful for copying files from the Internet to a user's file space on the cluster.  Submitting the line

wget http://www.examplesite.com/examplefile.txt

downloads examplefile.txt to the user's working directory. Other protocols, such as ftp, are also available.