View source for Admin Guide Data Transfer with Globus Online

[[Category:HPC-Admin]]
[[Category:HPC.NRW-Best-Practices]]

This article shows how to handle Globus as a tool to transfer large amounts of data between servers. It is described here with the example of the Bielefeld cluster system.

= GridFTP (Globus Toolkit) =

[https://www.globus.org/ Globus] is a tool to transfer large amounts of data from server to server without you being required to be constantly logged in and watching. To use Globus you need to create an account at (https://globus.org) and connect the Bielefeld clusters endpoint to it. To get the Bielefeld cluster search for

<source lang="sh">phyadmin#influx1</source>
and connect via your local Bielefeld username/password combination.

After having set up the Bielefeld cluster ''and another server'', you can start a transfer between them either via the web interface or the command line

<source lang="sh">globus transfer [OPTIONS] SOURCE_ENDPOINT_ID[:SOURCE_PATH] DEST_ENDPOINT_ID[:DEST_PATH]</source>
You can also connect your local machine to globus. To do so, download and extract <code>globusconnectpersonal</code>

<source lang="sh">wget https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz
tar -xzf globusconnectpersonal-latest.tgz</source>
and install the <code>Globus CLI</code> (command line interface, requires Python)

<source lang="sh">pip install --upgrade --user globus-cli</source>
The <code>CLI</code> needs to be linked to your account to make use of the most commands, so log in by

<source lang="sh">globus login</source>
and follow the given instructions. Using the CLI you can then create a local endpoint

<source lang="sh">$ globus endpoint create --personal my-linux-laptop
Message:     Endpoint created successfully
Endpoint ID: <endpoint-id>
Setup Key:   <setup-key></source>
and add it as such to your account in the web interface using the <code>&lt;endpoint-id&gt;</code>. Then start your new endpoint using

<source lang="sh">./globusconnectpersonal -setup <setup-key>
./globusconnectpersonal -start &</source>
== Globus example: transfer files between Bielefeld and Jülich ==

Although <code>judac</code> is already a GridFTP server, it is easiest connected to globus via a personal endpoint, if you do not have the required Grid certificates.

To do so, download <code>globusconnectpersonal</code> to <code>judac</code> just as you did above for your local machine. You can then create an endpoint via the globus web interface (recommended) or via the globus-CLI (discouraged), where you would need to install pip first by

<source lang="sh">wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user
GLOBUS_CLI_INSTALL_DIR="$(python -c 'import site; print(site.USER_BASE)')/bin"
echo "GLOBUS_CLI_INSTALL_DIR=$GLOBUS_CLI_INSTALL_DIR"
export PATH="$GLOBUS_CLI_INSTALL_DIR:$PATH"
echo 'export PATH="'"$GLOBUS_CLI_INSTALL_DIR"':$PATH"' >> "$HOME/.bashrc"
pip install --upgrade --user setuptools
pip install --upgrade --user globus-cli</source>
You will probably want to connect more directories than just your <code>/home</code>, which is available per default. To do so, edit <code>.globusonline/lta/config-paths</code> to list all the directories you want to connect. For example for Jülich:

<source lang="sh">~/,0,1
/p/project/chbi18,0,1
/p/scratch/chbi18,0,1</source>
You can then setup and start an endpoint just as above and submit a transfer via the web interface.

Further information can be found here: 
* [https://www.globus.org/usage-brief-library Globus Example Library] 
* [https://docs.globus.org/how-to/ Globus How To Documentation]