Difference between revisions of "File Transfer"

From HPC Wiki
Jump to: navigation, search
m
Line 15: Line 15:
  
 
== rsync ==
 
== rsync ==
 +
A command that is a little more sophisticated is rsync. It is mainly used to copy multiple files or mirror whole directories across different locations. It's basic usage works like this
 +
$ rsync [options] [source] [destination]
 +
or in a concrete example
 +
$ rsync -azvh -e ssh /my/source/directory/ user@123.123.123.123:/my/destination/directory
 +
where rsync is used on top of [[ssh]] to copy the source directory to the remote server over the internet. The options -azvh specify a compressed transfer with human readable output, preserving time stamps.
 +
 +
Due to the fact that rsync compares the contents of the source and destination directory and transfers only the necessary differences, it's usage is quite efficient, when mirroring multiple files.
 +
  
 
== File Transfer Protocol (ftp) ==
 
== File Transfer Protocol (ftp) ==
Line 27: Line 35:
 
== References ==
 
== References ==
 
[http://www.hypexr.org/linux_scp_help.php Basic Examples of SCP usage]
 
[http://www.hypexr.org/linux_scp_help.php Basic Examples of SCP usage]
 +
 
[https://filezilla-project.org/ Filezilla a free FTP client]
 
[https://filezilla-project.org/ Filezilla a free FTP client]
 +
 
[https://winscp.net/ WinSCP a free Windows FTP client]
 
[https://winscp.net/ WinSCP a free Windows FTP client]

Revision as of 13:19, 15 March 2018

To get your data (files) onto the supercomputer or back to your local machine, there are usually different ways. Sometimes there are computers specifically reserved for this purpose called copy Nodes

Please use these nodes to copy your data to or off the supercomputer, if available, since you will get a better/faster connection and disturb other users less. Also the tools mentioned below might only work on those nodes. If there are no dedicated copy nodes, you can usually use the Log-in Nodes for this purpose.

Secure Copy (scp)

This is generally the easiest way to transfer single files. It builds on ssh usually works on every node (machine) that you can build ssh connection to and can be used like this

$ scp your_username@remotehost.edu:foobar.txt /some/local/directory  

this copies the file foobar.txt from the remote machine to your local directory. Turning this around also works:

$ scp foobar.txt your_username@remotehost.edu:/some/remote/directory 

and copies the local file to the remote machine. Here you can find some more Examples of SCP syntax.

rsync

A command that is a little more sophisticated is rsync. It is mainly used to copy multiple files or mirror whole directories across different locations. It's basic usage works like this

$ rsync [options] [source] [destination]

or in a concrete example

$ rsync -azvh -e ssh /my/source/directory/ user@123.123.123.123:/my/destination/directory

where rsync is used on top of ssh to copy the source directory to the remote server over the internet. The options -azvh specify a compressed transfer with human readable output, preserving time stamps.

Due to the fact that rsync compares the contents of the source and destination directory and transfers only the necessary differences, it's usage is quite efficient, when mirroring multiple files.


File Transfer Protocol (ftp)

The file transfer protocol (ftp) is an easy way to view directories and exchange one or multiple files with a server. Usually a seperate program like Filezilla or WinSCP (for Windows) is employed to utilize ftp. If available, one can commonly simply connect to the Copy Nodes using this method. Utilizing a seperate program to exchange files with an intuitive graphical user interface can be a lot more flexible and easy to use for beginners.

Utilizing Filezilla e.g. looks like this:

Filezilla.png

You paste the respecting Copy Nodes into the 'server' field, authenticate with your username and password and have the server on the right and your local machine on the left.

References

Basic Examples of SCP usage

Filezilla a free FTP client

WinSCP a free Windows FTP client