You are visiting the archived wiki pages which are for previous system and NO LONGER SUPPORTED.
Please go to wiki.hpcc.msu.edu the latest HPCC wiki site.
For CentOS 6 nodes and Moab/Torque scheduling system only
This document highlights several simple methods to transfer files to the HPCC home and research directories. There are two main systems for copying files. First, simply "hpcc.msu.edu" which is our main log-in gateway. It can be used for file transfer but may have high traffic, is not meant for demanding file transfers (very large files or many files), and cannot access the scratch disk (/mnt/ls15/scratch or /mnt/scratch ). We offer a second gateway designed for file transfer, and does have access to the scratch file system with the host name "rsync.hpcc.msu.edu" = Therefore we highly recommend you use rsync.hpcc.msu.edu for the 'host' or 'server' in all examples below, as it's the only way to transfer files from scratch. Note that while it's named for the popular unix 'rsync' command, it can be used for sftp or scp as well.
You can copy files using Windows Explorer or OSX Finder if you are on campus by using the following instructions. Mapping HPC drives to a campus computer with SMB Legacy
If you are a Dropbox user, you can setup HPCC to sync automatically with your Dropbox account.
Log into one of the development nodes
untar the downloaded file using the following command
run dropboxd in a screen session
You should see output like this:
A number of different command-line utilities are available to OS X and Linux users. Each of them has its own advantages.
A simple command for transferring files between the cluster and another host is scp. To copy a file from a local directory to file space on the cluster, use a line like
This will copy the file named example.txt in the local host's home directory to the user's home directory on the cluster, with the copy having the name example_copy.txt. Leaving the space after the colon blank gives the new file the same name as the original. Note: To transfer a file name with spaces you must put a backslash before each space in your file name, i.e.
scp "My File Name" email@example.com:"My\ File\ Name"
To copy a file from the cluster to your local directory,
will copy the file named example.txt from the user's home directory on the cluster to the home directory of the local host, naming the new file example_copy.txt. Leaving the space after the slash blank gives the new file the same name as the original. The -r option can be used to copy entire directories recursively.
If you are an advanced LINUX/Mac user, there is a wonderful little utility that makes mirroring directories simple. The syntax looks very similar to scp.
To mirror <local_dir> on my local computer to <hpcc_dir> on hpcc, the following command can be issued.
In the above command, rsync will scan through both directories. If any files in the <local_dir> are newer, they will be uploaded to <hpcc_dir>. (It is also possible to get rsync to upload ALL different files, regardless of which is newer).
To mirror the HPCC directory to your local system, call
the first time you use rsync, you might want to add the -n flag to do a dry run before any files are copied.
When preforming several data transfers between hosts, the sftp command may be preferable, as it allows the user to work interactively. Running
from a local host establishes a connection between that host and the cluster. Both hosts can be navigated. For the local file system, lcd changes to the specified directory, lpwd prints the working directory, and lls prints a list of files in the current directory. For the remote file system, the same three commands are available, minus the leading "l." Also available are commands to change permissions, rename files, and manipulate directories on the remote host. The two key commands are get example.txt, which copies the file in the remote working directory to the local working directory, and put example.txt, which copies the file in the local working directory to the remote working directory. The quit command closes the connection between hosts.
Wget is a simple command useful for copying files from the Internet to a user's file space on the cluster. Submitting the line
downloads examplefile.txt to the user's working directory. Other protocols, such as ftp, are also available.