Streaming scientific software with EESSI

From HPC Wiki
Jump to navigation Jump to search


The European Environment for Scientific Software Installations (EESSI, pronounced as "easy", also see [1]), provides a "streaming service" for scientific software. It is based on the Cern VM File System (CVMFS), which is a read-only (from a user-perspective) file system for software distribution. CVMFS was originally developed in the context of worldwide distributed GRID computing and has been widely used in production for many years.

The general idea is to install a CVMFS client on all user-facing machines in an HPC cluster. The client is easily configured to connect to the EESSI servers to download and cache files via HTTP.

Motivation

Researchers migrating from their laptops and workstations to HPC systems may need rethink their software environment before they can start their calculations.

The same is true when researchers migrate between HPC systems, as many systems have their own conventions and technological solutions for providing the required research software, e.g. in the form of LMod modules.

Another reason is the heterogeneous hardware environment which may require special software installations and configurations, e.g. optimized builds for x86 CPUs, ARM CPUs, FPGAs or GPUs, and libraries for fast interconnect technologies.

Loading Software Modules

On an HPC cluster, you can check if CVMFS is available, e.g. by running 

less /cvmfs/software.eessi.io/README.eessi

In this case, an init script has to be sourced: 

$ source /cvmfs/software.eessi.io/versions/2023.06/init/bash 
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/intel/haswell
Using x86_64/intel/haswell as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/.lmod/lmodrc.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/modules/all as the directory to be added to MODULEPATH.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/.lmod/SitePackage.lua
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

{EESSI 2023.06} $ module avail # run the usual module commands

On a laptop, you would have to install CVMFS or use an EESSI container.

Adding Missing Software

EESSI is a community driven project built on top of EasyBuild to manage reproducible build recipes. If an important software module is missing or if you would like to distribute scientific software under your are developing, you can contribute by opening a pull request.

After the pull request is reviewed and approved, an automatic process is building and ingesting the software into the EESSI stack.

Additional Resources

Retrieved from "https://hpc-wiki.info/hpc/index.php?title=Streaming_scientific_software_with_EESSI&oldid=5036"