Containers at NHR - Revision history

Benjamin-juhl-d3d3@tu-darmstadt.de at 15:05, 5 November 2024

2024-11-05T15:05:32Z

Benjamin-juhl-d3d3@tu-darmstadt.de at 14:08, 5 November 2024

2024-11-05T14:08:42Z

Benjamin-juhl-d3d3@tu-darmstadt.de at 14:48, 29 October 2024

2024-10-29T14:48:28Z

Benjamin-juhl-d3d3@tu-darmstadt.de: Created page with "Category:BasicsCategory:HPC-User == General == Many users (especially new users) want to run docker containers on the NHR infratructure. However the NHR centers prov..."

2024-10-29T14:41:37Z

Created page with "Category:Basics Category:HPC-User == General == Many users (especially new users) want to run docker containers on the NHR infratructure. However the NHR centers prov..."

New page

[[Category:Basics]][[Category:HPC-User]]

== General ==
Many users (especially new users) want to run docker containers on the NHR infratructure. However the NHR centers provide singularity/apptainer as a container framework, and these cannot run docker containers natively. Singularity and apptainer commands can be used mostly interchangeably.

__TOC__

== Converting Containers ==
While it is not possible to run docker containers directly with the singularity/apptainer framework, there are several methods to convert a docker container into another format (e.g. sif or oci), and then use these converted containers with apptainer.

=== Build a sif container from a container in a docker registry ===
One can use the '''singualrity build''' command to build a singularity container directly from a container available from a docker repository.

Example:
<syntaxhighlight lang="bash">
singularity build lolcow.sif docker://godlovedc/lolcow
singularity run lolcow.sif
</syntaxhighlight>

=== Build a sif container from a docker container archive file ===
If you have a localy developed docker container, or a container from a registry with local changes, you can save it as a docker archive. You can then use '''singualrity build''' to build a singularity container from the archive.

Example:
<syntaxhighlight lang="bash">
singularity build lolcow_from_archive.sif docker-archive://$(pwd)/lolcow_docker_archive.tar
singularity run lolcow_from_archive.sif
</syntaxhighlight>

=== Use spython to translate a docker recipe into a singularity recipe and build from that recipe ===
In same cases, there might only be a docker recipe (also called a docker file) available, which is usualy used to build a docker container localy. Apptainer and singularity can also use such recipes to build containers, but the format is different. The singularity project provides a tool ([https://singularityhub.github.io/singularity-cli/ spython] ) to translate a docker recipe into a singularity recipe.

@@ Line 56: / Line 56: @@
 === Calling MPI externally ===
 In this approach the containerized application is called by the external MPI from the host. The host MPI takes care of the initialization. The MPI library within the container that the application was built with has to be compatible (ideally same version, this could also be achieved by binding the external library into the container) and the connects back to the external call.
 === Usining the internal MPI ===
 Work in progress

← Older revision		Revision as of 14:08, 5 November 2024
Line 49:		Line 49:
	deactivate		deactivate
	</syntaxhighlight>		</syntaxhighlight>
		+
		+
		+	== Running MPI Containers ==
		+	There are two way how to run mpi applications via containers. One method is to call mpi externally, and basically use the container instead of an mpi application. Another method is to call mpi within the container.
		+
		+	=== Calling MPI externally ===
		+	In this approach the containerized application is called by the external MPI from the host. The host MPI takes care of the initialization. The MPI library within the container that the application was built with has to be compatible (ideally same version, this could also be achieved by binding the external library into the container) and the connects back to the external call.
		+
		+
		+	=== Usining the internal MPI ===
		+	Work in progress

← Older revision		Revision as of 14:48, 29 October 2024
Line 32:		Line 32:
	=== Use spython to translate a docker recipe into a singularity recipe and build from that recipe ===		=== Use spython to translate a docker recipe into a singularity recipe and build from that recipe ===
	In same cases, there might only be a docker recipe (also called a docker file) available, which is usualy used to build a docker container localy. Apptainer and singularity can also use such recipes to build containers, but the format is different. The singularity project provides a tool ([https://singularityhub.github.io/singularity-cli/ spython] ) to translate a docker recipe into a singularity recipe.		In same cases, there might only be a docker recipe (also called a docker file) available, which is usualy used to build a docker container localy. Apptainer and singularity can also use such recipes to build containers, but the format is different. The singularity project provides a tool ([https://singularityhub.github.io/singularity-cli/ spython] ) to translate a docker recipe into a singularity recipe.
		+
		+	Example:
		+	<syntaxhighlight lang="bash">
		+	git clone https://github.com/GodloveD/lolcow
		+	spython recipe lolcow/Dockerfile ./sing.lolcow
		+	singularity build --fakeroot lolcow_from_dockerfile.sif sing.lolcow
		+	singularity run lolcow_from_dockerfile.sif
		+	</syntaxhighlight>
		+
		+	If spython is not natively available on the HPC system, you often install it in a virtual environment:
		+	<syntaxhighlight lang="bash">
		+	python -m venv myenv
		+	source myenv/bin/activate
		+	pip install --upgrade pip
		+	pip install spython
		+	deactivate
		+	</syntaxhighlight>