How to virtual environments

One possiblity to get access to to python packages on your HPC system that are not installed sytem wide is the local installation into you own virtual environment.

1. Understanding Virtual Environments:
   • Definition: A virtual environment is an isolated Python environment that allows you to install and manage dependencies separately for each project.
   • Advantages: Virtual environments prevent conflicts between different projects by keeping dependencies isolated. They also make it easier to manage dependencies and ensure reproducibility across different environments.
2. Installing Virtual Environment Tools:
   • Virtualenv: One of the most popular tools for creating virtual environments. Install it using pip: `pip install virtualenv`.
   • venv (Python 3.3+): A built-in module in Python for creating virtual environments. No need to install separately, but ensure you're using Python 3.3 or later.
   • Conda: A package manager, environment manager, and distribution of Python and other software packages for scientific computing. Install Anaconda or Miniconda from the official website: Anaconda or Miniconda.
3. Creating a Virtual Environment:
   • Using virtualenv: virtualenv myenv
   • Using venv: python -m venv myenv
   • Using Conda: conda create --name myenv
   Replace `myenv` with the desired name for your virtual environment.
4. Activating the Virtual Environment:
   • virtualenv or venv: source myenv/bin/activate
   • Conda: conda activate myenv
   After activation, your command line prompt should indicate the active virtual environment. Remember to add this line also to your job-script.
5. Installing Packages: Use `pip` to install packages within the activated virtual environment.
   • Use `pip` to install packages within virtualenv or venv: pip install package_name
   • Use `pip` or `conda` to install packages within Conda: pip install package_name` or `conda install package_name
6. Freezing Dependencies:
   • After installing packages, freeze the dependencies into a `requirements.txt` file with virtualenv or venv: pip freeze > requirements.txt
   • Conda automatically manages dependencies in its environment and does not require a separate requirements.txt file.
7. Deactivating the Virtual Environment:
   • To deactivate the virtual environment and return to the global Python environment:
     • virtualenv or venv: deactivate
     • Conda: <conda>conda deactivate
8. Version Control:
   • Include the `requirements.txt` file (if using virtualenv or venv) or `environment.yml` file (if using Conda) in your version control system (e.g., Git) to ensure that all collaborators can recreate the same environment.
   • Ignore the virtual environment directory (e.g., `myenv/`) to avoid cluttering the repository.
9. Updating Packages:
   • Regularly update packages within the virtual environment:
     • With virtualenv or venv: pip install --upgrade package_name
     • With Conda: conda update package_name
10. Cleaning Up:
    • Periodically clean up unused packages and their dependencies:
      • With virtualenv or venv: pip autoremove
      • With Conda: conda clean --all

Following these best practices ensures a clean and organized workflow when working with Python packages via virtual environments. It promotes reproducibility, simplifies dependency management, and helps avoid compatibility issues across different projects.