Snapshotting Jobs - Revision history

Robert-schade-e757@uni-paderborn.de at 14:24, 29 November 2021

2021-11-29T14:24:35Z

Christian-griebel-aab2@tu-darmstadt.de at 01:18, 4 November 2021

2021-11-04T01:18:28Z

Sebastian-potthoff-3c73@uni-muenster.de at 09:37, 20 September 2021

2021-09-20T09:37:08Z

Sebastian-potthoff-3c73@uni-muenster.de at 09:35, 20 September 2021

2021-09-20T09:35:20Z

Sebastian-potthoff-3c73@uni-muenster.de at 09:25, 20 September 2021

2021-09-20T09:25:23Z

Sebastian-potthoff-3c73@uni-muenster.de at 09:25, 20 September 2021

2021-09-20T09:25:02Z

Sebastian-potthoff-3c73@uni-muenster.de at 09:24, 20 September 2021

2021-09-20T09:24:24Z

Sebastian-potthoff-3c73@uni-muenster.de at 08:23, 20 September 2021

2021-09-20T08:23:19Z

Sebastian-potthoff-3c73@uni-muenster.de at 10:45, 17 September 2021

2021-09-17T10:45:12Z

Sebastian-potthoff-3c73@uni-muenster.de at 08:17, 9 September 2021

2021-09-09T08:17:58Z

← Older revision		Revision as of 14:24, 29 November 2021
Line 1:		Line 1:
−	[[Category:HPC-User]]	+	[[Category:HPC-User]][[Category:HPC-Admin]]
		+

	Snapshotting jobs (also known as '''C'''heck'''p'''oint/'''R'''estart), is a way to save the ''current state'' of a running job and continue its execution at a later point in time. This can be useful to overcome often imposed time limits or periods of maintenance where a cluster is not available. It can also be used to create regular snapshots of very long jobs to not loose the current progress in case something goes wrong. Some programs do have this functionality built into their code. Please inform yourself about this possibility as this is usually a much safer way to create snapshots. If this is not the case, keep on reading.		Snapshotting jobs (also known as '''C'''heck'''p'''oint/'''R'''estart), is a way to save the ''current state'' of a running job and continue its execution at a later point in time. This can be useful to overcome often imposed time limits or periods of maintenance where a cluster is not available. It can also be used to create regular snapshots of very long jobs to not loose the current progress in case something goes wrong. Some programs do have this functionality built into their code. Please inform yourself about this possibility as this is usually a much safer way to create snapshots. If this is not the case, keep on reading.

@@ Line 1: / Line 1: @@
 [[Category:HPC-User]]
-Snapshotting jobs is a way to save the current state of a running job and continue its execution at a later point in time. This can be useful to overcome often imposed time limits or periods of maintenance where a cluster is not available. It can also be used to create regular snapshots of very long jobs to not loose the current progress in case something goes wrong. Some programs do have this functionality built into their code. Please inform yourself about this possibility as this is usually a much safer way to create snapshots. If this is not the case, keep on reading.
+Snapshotting jobs (also known as '''C'''heck'''p'''oint/'''R'''estart), is a way to save the ''current state'' of a running job and continue its execution at a later point in time. This can be useful to overcome often imposed time limits or periods of maintenance where a cluster is not available. It can also be used to create regular snapshots of very long jobs to not loose the current progress in case something goes wrong. Some programs do have this functionality built into their code. Please inform yourself about this possibility as this is usually a much safer way to create snapshots. If this is not the case, keep on reading.
 __TOC__

@@ Line 11: / Line 11: @@
 === Current limitations ===
-At this point DMTCP reliably only works for non-MPI codes, i.e. multi-threaded codes using OpenMP or pthreads etc., even though MPI codes are officially supported. There are ongoing efforts to re-implement the support for MPI codes. The project is called [https://github.com/mpickpt/mana/tree/master/contrib/mpi-proxy-split MANA] and is in active development. However, at this point the code is highly tailored for the usage at [https://docs.nersc.gov/development/checkpoint-restart/mana/ NESC]. At a future point MANA will be merged into the official DMTCP code base.
+At this point DMTCP reliably only works for non-MPI codes, i.e. multi-threaded codes using OpenMP or pthreads etc., even though MPI codes are officially supported. There are ongoing efforts to re-implement the support for MPI codes. The project is called [https://github.com/mpickpt/mana/tree/master/contrib/mpi-proxy-split MANA] and is in active development. However, at this point the code is highly tailored for the usage at [https://docs.nersc.gov/development/checkpoint-restart/mana/ NERSC]. At a future point MANA will be merged into the official DMTCP code base.
 === Installation ===

← Older revision		Revision as of 09:35, 20 September 2021
Line 78:		Line 78:

	sbatch slurm_rstr.job		sbatch slurm_rstr.job
		+
		+	=== Tips and known issues ===
		+
		+	* When compiling code using the intel compiler with OpenMP use the flag <code>-qopenmp-link=static</code>, otherwise parallel regions are not entered without error messages
		+	* For Java applications set the following environment variables:
		+	export DMTCP_DL_PLUGIN=0
		+	export DMTCP_SIGCKPT=10
		+	*

@@ Line 40: / Line 40: @@
 === Automatically creating checkpoints ===
-You can also instruct the DMTC coordinator to periodically create checkpoints. Use
+You can also instruct the DMTCP coordinator to periodically create checkpoints. Use
   dmtcp_coordinator -i 300 &

@@ Line 29: / Line 29: @@
 . To create a new checkpoint, return to the shell where you started the coordinator. Type <code>c</code> followed by pressing <code>[RETURN]</code>
 .1 Restart: Creating a checkpoint causes the dmtcp_coordinator to write a checkpoint file (file type: .dmtcp) for each client process. If all processes were on the same processor, and there were no .dmtcp files prior to this checkpoint you can run:
   dmtcp_restart ckpt_*.dmtcp
 .2 Next to a checkpoint file, DMTCP also writes out a starting script. However, this script is looking if a resource manager (i.e. SLURM, PBS etc.) was used and is setting the the environment variable <code>RES_MANAGER</code> if you are running e.g. in an interactive SLURM session. You can try to use this script to restart your program but you might have to adjust it for your needs:
   ./dmtcp_restart_script.sh
 === Automatically creating checkpoints ===

@@ Line 9: / Line 9: @@
 [https://github.com/dmtcp/dmtcp DMTCP] (Distributed MultiThreaded CheckPointing) is a tool to create snapshots of currently running jobs. This is achieved by saving the current ''memory state'' to a checkpoint file. This file can then be used to continue the job.
 === Installation ===
 If not available on your site, installation should be done by an administrator. A viable option is to use Easybuild as an [https://github.com/easybuilders/easybuild-easyconfigs/tree/main/easybuild/easyconfigs/d/DMTCP easyconfig-file] for DMTCP is available.
-=== Manual ===
+=== Manual usage ===
 Here we assume you have direct access to a compute node via SSH, using e.g. an interactive job session and all DMTCP commands are available in your environment.

@@ Line 25: / Line 25: @@
   dmtcp_launch ./a.out
-. To create a new checkpointm, return to the shell where you started the coordinator. Type <code>c</code> followed by pressing <code>[RETURN]</code>
+. To create a new checkpoint, return to the shell where you started the coordinator. Type <code>c</code> followed by pressing <code>[RETURN]</code>
@@ Line 39: / Line 39: @@
 === Automatically creating checkpoints ===
 === Integration into SLURM ===

@@ Line 21: / Line 21: @@
   dmtcp_coordinator
-. In a separate shell (on the same node) launch your application:
+. In a '''separate shell''' (on the same node) launch your application:
   dmtcp_launch ./a.out
-. To create a new checkpoint type <code>c</code> followed by pressing <code>[RETURN]</code>
+. To create a new checkpointm, return to the shell where you started the coordinator. Type <code>c</code> followed by pressing <code>[RETURN]</code>
 . Restart: Creating a checkpoint causes the dmtcp_coordinator to write a checkpoint file (file type: .dmtcp) for each client process. If all processes were on the same processor, and there were no .dmtcp files prior to this checkpoint:
   dmtcp_restart ckpt_*.dmtcp