File Transfer - Revision history

Daniel-schurhoff-de23@rwth-aachen.de at 07:55, 7 September 2020

2020-09-07T07:55:10Z

Daniel-schurhoff-de23@rwth-aachen.de at 13:07, 3 September 2019

2019-09-03T13:07:01Z

Christian-griebel-aab2@tu-darmstadt.de: /* rsync */

2019-06-03T14:29:01Z

rsync

Christian-griebel-aab2@tu-darmstadt.de: /* Secure Copy (scp) */

2019-06-03T14:22:38Z

Secure Copy (scp)

Christian-griebel-aab2@tu-darmstadt.de: /* Secure Copy (scp) */

2019-06-03T13:55:07Z

Secure Copy (scp)

Christian-griebel-aab2@tu-darmstadt.de: /* Secure Copy (scp) */

2019-06-03T11:03:17Z

Secure Copy (scp)

Daniel-schurhoff-de23@rwth-aachen.de: Reverted edits by Linda-engelmann-1168@rwth-aachen.de (talk) to last revision by [[User:Nina-loseke-fd7a@rwth-aachen.de|Nina-loseke-fd7a@rwt...

2019-03-21T10:12:59Z

Reverted edits by Linda-engelmann-1168@rwth-aachen.de (talk) to last revision by [[User:Nina-loseke-fd7a@rwth-aachen.de|Nina-loseke-fd7a@rwt...

Linda-engelmann-1168@rwth-aachen.de at 12:50, 20 March 2019

2019-03-20T12:50:12Z

Nina-loseke-fd7a@rwth-aachen.de at 13:30, 20 April 2018

2018-04-20T13:30:44Z

Christian-wassermann-e30b@rwth-aachen.de at 10:36, 16 April 2018

2018-04-16T10:36:58Z

@@ Line 37: / Line 37: @@
 Due to the fact that rsync compares the contents of the source and destination directory beforehand, transferring only the necessary differences (delta), it is quite efficient in mirroring multiple files or whole directory trees.<br />
-Further documentation can be found in [http://www.createdbypete.com/articles/a-practical-guide-to-using-rsync/ this basic guide], or in even more depth [https://wiki.ubuntuusers.de/rsync/ here].
+Further documentation can be found in the basic Rsync Guide <ref> [http://www.createdbypete.com/articles/a-practical-guide-to-using-rsync Basic Rsync guide] </ref>, or in even more depth in the Ubuntu Users Guide <ref>[https://wiki.ubuntuusers.de/rsync/ Ubuntu Users Guide to Rsync (pretty indepth)]</ref>.
 == File Transfer Protocol (ftp) ==
@@ Line 48: / Line 48: @@
 You paste the respecting [[Nodes#Copy|Copy Nodes]] into the 'server' field, authenticate with your username and password and have the server on the right and your local machine on the left.
-== References ==
+== further Material ==
 [http://www.hypexr.org/linux_scp_help.php Basic Examples of SCP usage]
 [https://filezilla-project.org/ Filezilla a free FTP client]
 [https://winscp.net/ WinSCP a free Windows FTP client]

← Older revision		Revision as of 13:07, 3 September 2019
Line 1:		Line 1:
		+	[[Category:Basics]]
		+
	To get your data (files) onto the supercomputer or back to your local machine, there are usually different ways. Sometimes there are computers specifically reserved for this purpose called [[Nodes#Copy\|copy nodes]].		To get your data (files) onto the supercomputer or back to your local machine, there are usually different ways. Sometimes there are computers specifically reserved for this purpose called [[Nodes#Copy\|copy nodes]].

@@ Line 28: / Line 28: @@
 == rsync ==
-A command that is a little more sophisticated is rsync. It is mainly used to copy multiple files or mirror whole directories across different locations. It's basic usage works like this
+Quite a bit more sophisticated, rsync is another and more efficient file transfer mechanism. It is mainly used to copy multiple files or even whole directory trees across different locations. Its basic usage is like this
   $ rsync [options] [source] [destination]
 or in a concrete example
   $ rsync -azvh -e ssh /my/source/directory/ user@123.123.123.123:/my/destination/directory
-where rsync is used on top of [[ssh]] to copy the source directory to the remote server over the internet. The options -azvh specify a compressed transfer with human readable output, preserving time stamps and is a good starting point. Another common option is -r which recursively synchronizes all subdirectories.
+where rsync is used on top of [[ssh]] to copy the source directory to the remote server (via network). The options <code>-azvh</code> specify a compressed transfer with human readable output, preserving time stamps, and are thus a good starting point. Another common option is <code>-r</code> which recursively synchronizes all subdirectories (part of <code>-a</code>).
-Due to the fact that rsync compares the contents of the source and destination directory and transfers only the necessary differences, its usage is quite efficient, when mirroring multiple files. Further documentation can be found in [http://www.createdbypete.com/articles/a-practical-guide-to-using-rsync/ this basic guide], or this more [https://wiki.ubuntuusers.de/rsync/ indepth guide]
+Due to the fact that rsync compares the contents of the source and destination directory beforehand, transferring only the necessary differences (delta), it is quite efficient in mirroring multiple files or whole directory trees.<br />
 == File Transfer Protocol (ftp) ==

@@ Line 19: / Line 19: @@
 ;Beware - <code>scp</code> or <code>sftp</code> might be confused by STDOUT/STDERR generated during your normal login session!
-: Most frequent example: you want to have a certain set of ''[[modules]]'' loaded automatically at each login, and you have edited your <code>$HOME/.bashrc</code> to contain "module load ..." commands. This in turn creates output on STDOUT, like "loading module ..." appears on your terminal while logging in. This is all very fine for interactive logins, but <code>scp</code> or <code>sftp</code> or Clients like Filezilla might fail for no apparent reason.
+: Most frequent example: you want to have a certain set of ''[[modules]]'' loaded automatically at each login, and you have edited your <code>$HOME/.bashrc</code> to contain "module load ..." commands. This in turn creates output on STDOUT, like "loading module ..." appears on your terminal while logging in. This is all very fine for interactive logins, but <code>scp</code> or <code>sftp</code> or graphical clients like Filezilla might fail for no apparent reason.
 : <u>Solution</u>: put such commands only ''after'' a check whether a shell is in fact an interactive one, like:
    ########################

@@ Line 19: / Line 19: @@
 ;Beware - <code>scp</code> or <code>sftp</code> might be confused by STDOUT/STDERR generated during your normal login session!
-: Most frequent example: you want to have a certain set of ''modules'' loaded automatically at each login, and you have edited your <code>$HOME/.bashrc</code> to contain "module load ..." commands. This in turn creates output on STDOUT, like "loading module ..." appears on your terminal while logging in. This is all very fine for interactive logins, but <code>scp</code> or <code>sftp</code> or Clients like Filezilla might fail for no apparent reason.
+: Most frequent example: you want to have a certain set of ''[[modules]]'' loaded automatically at each login, and you have edited your <code>$HOME/.bashrc</code> to contain "module load ..." commands. This in turn creates output on STDOUT, like "loading module ..." appears on your terminal while logging in. This is all very fine for interactive logins, but <code>scp</code> or <code>sftp</code> or Clients like Filezilla might fail for no apparent reason.
 : <u>Solution</u>: put such commands only ''after'' a check whether a shell is in fact an interactive one, like:
    ########################

← Older revision		Revision as of 11:03, 3 June 2019
Line 17:		Line 17:

	and copies the local file to the remote machine. Here you can find some more [http://www.hypexr.org/linux_scp_help.php examples of scp] and its usage.		and copies the local file to the remote machine. Here you can find some more [http://www.hypexr.org/linux_scp_help.php examples of scp] and its usage.
		+
		+	;Beware - <code>scp</code> or <code>sftp</code> might be confused by STDOUT/STDERR generated during your normal login session!
		+	: Most frequent example: you want to have a certain set of ''modules'' loaded automatically at each login, and you have edited your <code>$HOME/.bashrc</code> to contain "module load ..." commands. This in turn creates output on STDOUT, like "loading module ..." appears on your terminal while logging in. This is all very fine for interactive logins, but <code>scp</code> or <code>sftp</code> or Clients like Filezilla might fail for no apparent reason.
		+	: <u>Solution</u>: put such commands only ''after'' a check whether a shell is in fact an interactive one, like:
		+	########################
		+	# do NOT generate output unless we're an interactive shell:
		+	########################
		+	[ -z "$PS1" ] && return
		+	:Only if the prompt string #1 (''PS1'') is set, the shell is interactive - otherwise, continue execution of the calling script (''return'').

	== rsync ==		== rsync ==

@@ Line 32: / Line 32: @@
 Utilizing Filezilla e.g. looks like this:
-[[File:Filezilla1.png]]
+[[File:Filezilla.png]]
 You paste the respecting [[Nodes#Copy|Copy Nodes]] into the 'server' field, authenticate with your username and password and have the server on the right and your local machine on the left.

← Older revision		Revision as of 13:30, 20 April 2018
Line 2:		Line 2:

	If available to you, it is recommened to use these copy nodes to move data to or from the supercomputer, since this will result in a better connection and disturb other users less. Additionally the tools mentioned below might only work on these nodes. If there are no dedicated copy nodes, you can usually use the [[Nodes#Login\|Login Nodes]] for this purpose.		If available to you, it is recommened to use these copy nodes to move data to or from the supercomputer, since this will result in a better connection and disturb other users less. Additionally the tools mentioned below might only work on these nodes. If there are no dedicated copy nodes, you can usually use the [[Nodes#Login\|Login Nodes]] for this purpose.
		+
		+
		+	__TOC__
		+

	== Secure Copy (scp) ==		== Secure Copy (scp) ==

@@ Line 1: / Line 1: @@
-To get your data (files) onto the supercomputer or back to your local machine, there are usually different ways. Sometimes there are computers specifically reserved for this purpose called [[Nodes#Copy_Nodes|copy nodes]].
+To get your data (files) onto the supercomputer or back to your local machine, there are usually different ways. Sometimes there are computers specifically reserved for this purpose called [[Nodes#Copy|copy nodes]].
-If available to you, it is recommened to use these copy nodes to move data to or from the supercomputer, since this will result in a better connection and disturb other users less. Additionally the tools mentioned below might only work on these nodes. If there are no dedicated copy nodes, you can usually use the [[Nodes#Log-In|Log-in Nodes]] for this purpose.
+If available to you, it is recommened to use these copy nodes to move data to or from the supercomputer, since this will result in a better connection and disturb other users less. Additionally the tools mentioned below might only work on these nodes. If there are no dedicated copy nodes, you can usually use the [[Nodes#Login|Login Nodes]] for this purpose.
 == Secure Copy (scp) ==
@@ Line 24: / Line 24: @@
 == File Transfer Protocol (ftp) ==
-The file transfer protocol (ftp) is a network protocol used to exchange files with a server. Usually a seperate program like [https://filezilla-project.org/ Filezilla] or [https://winscp.net/ WinSCP] (for Windows) is employed to utilize ftp. If available, one can simply connect to the [[Nodes#Copy_Nodes|copy nodes]] using this method. Utilizing a seperate program to exchange files with an intuitive graphical user interface can be a lot more flexible and easy to use for beginners.
+The file transfer protocol (ftp) is a network protocol used to exchange files with a server. Usually a seperate program like [https://filezilla-project.org/ Filezilla] or [https://winscp.net/ WinSCP] (for Windows) is employed to utilize ftp. If available, one can simply connect to the [[Nodes#Copy|copy nodes]] using this method. Utilizing a seperate program to exchange files with an intuitive graphical user interface can be a lot more flexible and easy to use for beginners.
 Utilizing Filezilla e.g. looks like this:
@@ Line 30: / Line 30: @@
 [[File:Filezilla.png]]
-You paste the respecting [[Nodes#Copy_Nodes|Copy Nodes]] into the 'server' field, authenticate with your username and password and have the server on the right and your local machine on the left.
+You paste the respecting [[Nodes#Copy|Copy Nodes]] into the 'server' field, authenticate with your username and password and have the server on the right and your local machine on the left.
 == References ==