You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Next »

This page explains how to copy files for your analysis to and from Hydra.

  • File transfers can be done through (Linux/Mac) command line, using scp, rsync or sftp, or

  • using a GUI application like FileZilla or WinSCP (etc, i.e, any tool that implements the scp or sftp protocol)
    While Cyberduck works, surprisingly, it uses a lot of CPU cycles (on Hydra), so we recommend that you use FileZilla or WinSCP instead, or just scp, rsync or sftp.

  • Transfers to Hydra can only be initiated from "trusted" computers (i.e., you may need to enable VPN);
  • Transfers from Hydra to your local machine are not limited to trusted destinations.

(warning) Files should always be copied directly to the /pool, /data or /scratch directories, not to your home directory (/home) because of space limitations. See here for more about data storage on Hydra.

These quick start instructions show how to configure FileZilla for file transfers, and a short introduction to scp, rsync or sftp.

Using FileZilla

  1. "FileZilla Client" is available for Mac, Windows and Linux systems.
    Mac screenshots are shown here, but use on other systems is similar.
  2. Open the FileZilla application.
  3. In the Quickconnect toolbar at the top of the window enter:
    1. Host: hydra-login01.si.edu or hydra-login02.si.edu
    2. Username: your Hydra username
    3. Password: your Hydra password
    4. Port: 22 


  4. Press the "Quickconnect" button to start the connection.

    (warning) If you get a warning about Saving passwords, choose "Do not save Passwords" and then the OK button.


    (warning) If you get a warning about Unknown host key, click the "Always trust..." checkbox and then the OK button.


  5. The files listed on the left side of the window are on your local computer, those on the right are on Hydra.
    1. Use the file tree on the left to navigate to the directory with your files to upload
    2. For the destination on Hydra, enter the path of your destination in the "Remote site:" text box
    3. Drag files from the left to a directory on the right side to upload them

Using SCP

The command scp is available on any Linux or Mac machine. To access it from a Mac, start a terminal to get a Unix prompt.

Use the command cd to go to where the files you want to copy are.

You can copy a local file with:
scp myfile remuser@hydra-login01.si.edu:/path/to/dest/

where:

  • myfile is the file name you want to copy on your (local) machine,
  • remuser is your Hydra username,
  • /path/to/dest is the directory specification where you want the file copied (it must exists), like /pool/genomics/kweskinm/big/stuff

You can copy multiple files with:
scp myfile1 myfile2 remuser@hydra-login01.si.edu:/path/to/dest/ 
scp myfile* remuser@hydra-login01.si.edu:/path/to/dest/

The command scp always copy the file(s), the option '-p' (like in scp -p) will preserve the date information of the file(s) copied.

And

To copy from Hydra
scp remuser@hydra-login01.si.edu:/path/to/location/myfile .
You can use wildcards (remote location), as follows:
scp 'remuser@hydra-login01.si.edu:/path/to/location/myfile*' .

To learn more about scp, read the manual page (man scp).

Using RSYNC

The command rsync is available on any Linux or Mac machine. To access it from a Mac, start a terminal to get a Unix prompt. 

It synchronizes files between two hosts (machines), so it will not copy files that exists already and are up-to-date: it will copy a file if the one on the remote location is older.

Use the command cd to go to where the files you want top copy are.

Examples of copying to Hydra:
rsync myfile remuser@hydra-login01.si.edu:/path/to/dest/
rsync myfile1 myfile2 remuser@hydra-login01.si.edu:/path/to/dest/
rsync myfile* remuser@hydra-login01.si.edu:/path/to/dest/

And

From Hydra to your local machine:
rsync remuser@hydra-login01.si.edu:/path/to/location/myfile .
rsync 'remuser@hydra-login01.si.edu:/path/to/location/myfile*' .

The difference with scp, is that rsync will only copy what is new,

so you can do:
rsync * remuser@hydra-login01.si.edu:/path/to/dest/

to synchronize the content of the current working directory.

Use the option '-n' (like in rsync -n) to check what rsync will do (it will list what will be copied, but not do it).

Three more useful options are:

  • -a - archive mode (equals -rlptgoD). It is a quick way of saying you want  recursion  and  want  to  preserve almost  everything  (with  -H being a notable omission).
  • -z - compress file data during the transfer (speed up)
  • -v - verbose mode

that can be combined as rsync -azv

To learn more about rsync, read the manual page (man rsync).

Using SFTP

You can also use the command sftp to copy files, it is also available on any Linux or Mac machine. To access it from a Mac, start a terminal to get a Unix prompt.

Use the command cd to go to where the files you want top copy are.

For example:
sftp remuser@hydra-login01.si.edu
sftp> cd /path/to/dest/
sftp> put myfile
sftp> get myresults
sftp> exit

The main sftp commands are cd, lcd, put and get:

  • cd  - change the directory on the remote host,
  • lcd - change the directory on the local host,
  • put - copy a file from the local host to the remote one,
  • get - copy a file from the remote host to teh local one.

To learn more about sftp, read the manual page (man sftp).

Last updated  MK/PF/SGK.

 



  • No labels