How to Use Jupyter with virtual environments - Revision history

Christian-schmidt-sonntag-5c48@uni-bielefeld.de at 14:05, 17 September 2024

2024-09-17T14:05:10Z

Christian-schmidt-sonntag-5c48@uni-bielefeld.de at 14:04, 17 September 2024

2024-09-17T14:04:29Z

Christian-schmidt-sonntag-5c48@uni-bielefeld.de at 14:03, 17 September 2024

2024-09-17T14:03:07Z

Christian-schmidt-sonntag-5c48@uni-bielefeld.de at 14:02, 17 September 2024

2024-09-17T14:02:26Z

Rasmus-larsen-d7b7@uni-bielefeld.de at 09:17, 30 August 2024

2024-08-30T09:17:07Z

Rasmus-larsen-d7b7@uni-bielefeld.de: Created page with "Category:HPC-User This is a setup for using Jupyter on the HPC clusters using virtual environments installed with pip, and running the file as: 2) an interactive job. 3)..."

2024-08-30T09:09:35Z

Created page with "Category:HPC-User This is a setup for using Jupyter on the HPC clusters using virtual environments installed with pip, and running the file as: 2) an interactive job. 3)..."

New page

[[Category:HPC-User]]

This is a setup for using Jupyter on the HPC clusters using virtual environments installed with pip, and running the file as: 2) an interactive job. 3) Using slurm scripts with Papermill.

Some sites already have jupyterLab instances available. If these are available, we recommend using these instances, instead of setting up your own interactive job. The user interface might be different from place to place, but besides choosing the desired kernel to run, it should not require any setup and should launch a jupyterLab web interface automatically.

This setup is setup for using python and uses TensorFlow as an example.

__TOC__

== Setup Virtual Environment ==

Make a folder on the cluster where you want to save the virtual environment and go into the folder.

Load python using the installed modules. Example shown with the current latest python module on Influx Cluster
- `module load python/3.11`

Available modules can typically be found using
- `module avail`

Create the virtual environment. If you dont want to inherit already installed packages inside python on the server, remove "--system-site-packages". myevn is the name of the virtual environment, feel free to change it to whatever fits you, just remember to replace the name in the other scripts.
- `virtualenv --system-site-packages myenv`

The virtual environment can then be activated using:
- `source myenv/bin/activate`

Jupyter can then be installed using the pip command:
- `pip install jupyterlab`

In case a needed package does not exist, these can be installed in the same way. For instance, Tensorflow is already installed on most clusters, but if "--system-site-packages" is not used, you would need to install these yourself. At the time of writing, the basic functionality needed is
- `pip install tensorrt`
- `pip install tensorflow[and-cuda]`

When finished, deactivate the virtual environment with the command.
- `deactivate`

== Run jupyter using interactive job ==

We will here setup a jupyter-notebook running in your own web browser connected to an interactive job on the cluster. We here assume that the cluster uses slurm to manage the jobs.

First start a job using the slurm command srun on
- `CPU: srun --account=AccountName --partition=PatitionName --time=00:30:00 --pty $SHELL -i`
- `GPU: srun --account=AccountName --ntasks=1 --partition=PatitionName --qos=QOS --nodes=1 --time=00:30:00 --gpus-per-task=1 --cpus-per-task=1 --pty $SHELL -i`
Some HPC clusters do not require an account, and partition and qos specify where to run the jobs. This will depend on the specific cluster.

Then load the needed modules. For instance, on Influx this is currently:
- `module load python/3.11`
- `GPU:module load cuda/12.1`
cuda/11.0 is too old for the newest version of tensorflow, and will get an error when running model.

Activate virtual environment
- `source /path_to_virtual_environment/myenv/bin/activate`

Start jupyter using:

- `jupyter-lab --no-browser --ip $(hostname -f)`
Press enter once more if nothing is happening after a couple of seconds.
ctrl + c to exit when you dont need it anymore.

In the output, while not having exited, look for an URL looking like
http://c01.clusternet:8888/lab?token=813d3bf71988d0a21284c77ffa310446a7b5b9a80c1a579c
You need to connect c01.clusternet:8888 to a local port on your own machine.

On your own machines terminal, set up a tunnel to the cluster using:

- `ssh -L 9999:c01.clusternet:8888 username@hostname`

9999 can be changed to another local port. c01.clusternet:8888 needs to match what the above URL showed.

go to the address in a browser
- `http://localhost:9999/lab?token=....`

token=... is the last part of the URL we found above, in this example token=813d3bf71988d0a21284c77ffa310446a7b5b9a80c1a579c

Often the token is not needed, and one can simply use
- `http://localhost:9999/lab`

== Run ipynb Files using slurm script with Papermill ==

There are different ways to run a jupyter script on the cluster. One can for instance export to a python file and then run that. We here show how to run the ipynb file directly using Papermill.

First, we need to install Papermill. Load the virtual environment and python module and install using pip

- `source /path_to_virtual_environment/myenv/bin/activate`
- `pip install papermill`

On the Cluster make a slurm script using an editor, for instance vim:

- `vim runCPU.sh`

Inside the slurm script, add the options needed to run. Below is an example of a 1 cpu setup for bash shell.

#!/bin/bash

#Job parameters
#SBATCH --job-name=TensorTest

#Resources
####SBATCH --account=If_needed
#SBATCH --time=00:30:00
#SBATCH --partition=PatitionName(for example cpu_compute, depends on cluster)
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --tasks-per-node=1
#SBATCH --cpus-per-task=1
####SBATCH --gpus-per-task=1

#load modules available on cluster, for example
module load python/3.11
#module load cuda/12.1

#load virtual environment
source /path_to_virtual_environment/myenv/bin/activate

#Job step(s)
srun papermill in.ipynb out.ipynb

This will take in in.ipynb run the file and output out.ipynb
The progress can be seen in the slurm-JOBID.out file

run the slurm script using
- `sbatch runCPU.sh`

@@ Line 13: / Line 13: @@
 == Setup Virtual Environment ==
-Make a folder on the cluster where you want to save the virtual environment and go into the folder. For a detailed descrition on virtual environments see also [].
+Make a folder on the cluster where you want to save the virtual environment and go into the folder. For a detailed descrition on virtual environments see also [[How_to_virtual_environments]].
 Load python using the installed modules. Example shown with the current latest python module on Influx Cluster

@@ Line 100: / Line 100: @@
   #!/bin/bash
-−
   #Job parameters
   #SBATCH --job-name=TensorTest

@@ Line 16: / Line 16: @@
 Load python using the installed modules. Example shown with the current latest python module on Influx Cluster
-  - `module load python/3.11`
+  - module load python/3.11
 Available modules can typically be found using
-  - `module avail`
+  - module avail
 Create the virtual environment. If you dont want to inherit already installed packages inside python on the server, remove "--system-site-packages". myevn is the name of the virtual environment, feel free to change it to whatever fits you, just remember to replace the name in the other scripts.
-  - `virtualenv --system-site-packages  myenv`
+  - virtualenv --system-site-packages  myenv
 The virtual environment can then be activated using:
-  - `source myenv/bin/activate`
+  - source myenv/bin/activate
 Jupyter can then be installed using the pip command:
-  - `pip install jupyterlab`
+  - pip install jupyterlab
 In case a needed package does not exist, these can be installed in the same way. For instance, Tensorflow is already installed on most clusters, but if "--system-site-packages" is not used, you would need to install these yourself. At the time of writing, the basic functionality needed is
-  - `pip install tensorrt`
+  - pip install tensorrt
-  - `pip install tensorflow[and-cuda]`
+  - pip install tensorflow[and-cuda]
 When finished, deactivate the virtual environment with the command.
-  - `deactivate`
+  - deactivate
@@ Line 47: / Line 47: @@
 First start a job using the slurm command srun on
-  - `CPU: srun --account=AccountName --partition=PatitionName --time=00:30:00 --pty $SHELL -i`
+  - CPU: srun --account=AccountName --partition=PatitionName --time=00:30:00 --pty $SHELL -i
-  - `GPU: srun --account=AccountName --ntasks=1 --partition=PatitionName --qos=QOS --nodes=1 --time=00:30:00  --gpus-per-task=1 --cpus-per-task=1 --pty $SHELL -i`
+  - GPU: srun --account=AccountName --ntasks=1 --partition=PatitionName --qos=QOS --nodes=1 --time=00:30:00  --gpus-per-task=1 --cpus-per-task=1 --pty $SHELL -i
 Some HPC clusters do not require an account, and partition and qos specify where to run the jobs. This will depend on the specific cluster.
 Then load the needed modules. For instance, on Influx this is currently:
-  - `module load python/3.11`
+  - module load python/3.11
-  - `GPU:module load cuda/12.1`
+  - GPU:module load cuda/12.1
 cuda/11.0 is too old for the newest version of tensorflow, and will get an error when running model.
 Activate virtual environment
-  - `source /path_to_virtual_environment/myenv/bin/activate`
+  - source /path_to_virtual_environment/myenv/bin/activate
 Start jupyter using:
-  - `jupyter-lab --no-browser --ip $(hostname -f)`
+  - jupyter-lab --no-browser --ip $(hostname -f)
 Press enter once more if nothing is happening after a couple of seconds.
 ctrl + c  to exit when you dont need it anymore.
@@ Line 72: / Line 72: @@
 On your own machines terminal, set up a tunnel to the cluster using:
-  - `ssh -L 9999:c01.clusternet:8888 username@hostname`
+  - ssh -L 9999:c01.clusternet:8888 username@hostname
 can be changed to another local port. c01.clusternet:8888 needs to match what the above URL showed.
 go to the address in a browser
-  - `http://localhost:9999/lab?token=....`
+  - http://localhost:9999/lab?token=....
 token=...  is the last part of the URL we found above, in this example token=813d3bf71988d0a21284c77ffa310446a7b5b9a80c1a579c
 Often the token is not needed, and one can simply use
-  - `http://localhost:9999/lab`
+  - http://localhost:9999/lab
 == Run ipynb Files using slurm script with Papermill ==
@@ Line 90: / Line 90: @@
 First, we need to install Papermill. Load the virtual environment and python module and install using pip
-  - `source /path_to_virtual_environment/myenv/bin/activate`
+  - source /path_to_virtual_environment/myenv/bin/activate
-  - `pip install papermill`
+  - pip install papermill
 On the Cluster make a slurm script using an editor, for instance vim:
-  - `vim runCPU.sh`
+  - vim runCPU.sh
 Inside the slurm script, add the options needed to run. Below is an example of a 1 cpu setup for bash shell.
@@ Line 129: / Line 129: @@
 run the slurm script using
-- `sbatch runCPU.sh`
+- sbatch runCPU.sh