Benchmarking & Scaling Tutorial/Manual Benchmarking - Revision history

Sebastian-potthoff-3c73@uni-muenster.de at 12:17, 12 February 2025

2025-02-12T12:17:46Z

Sebastian-potthoff-3c73@uni-muenster.de at 09:03, 3 June 2022

2022-06-03T09:03:01Z

Sebastian-potthoff-3c73@uni-muenster.de at 13:22, 24 September 2021

2021-09-24T13:22:08Z

Sebastian-potthoff-3c73@uni-muenster.de at 12:37, 23 September 2021

2021-09-23T12:37:43Z

Sebastian-potthoff-3c73@uni-muenster.de at 10:14, 23 September 2021

2021-09-23T10:14:51Z

Sebastian-potthoff-3c73@uni-muenster.de at 10:14, 23 September 2021

2021-09-23T10:14:20Z

Sebastian-potthoff-3c73@uni-muenster.de at 10:05, 23 September 2021

2021-09-23T10:05:47Z

Sebastian-potthoff-3c73@uni-muenster.de at 10:01, 23 September 2021

2021-09-23T10:01:59Z

Sebastian-potthoff-3c73@uni-muenster.de at 10:01, 23 September 2021

2021-09-23T10:01:26Z

Sebastian-potthoff-3c73@uni-muenster.de at 13:11, 22 September 2021

2021-09-22T13:11:21Z

@@ Line 10: / Line 10: @@
 For this tutorials purpose we are using the Molecular Dynamics Code [https://manual.gromacs.org/documentation/ GROMACS] as this is a common simulation program and readily available on most clusters. We prepared three test systems with increasing amounts of water molecules which you can download from here:
-* [https://uni-muenster.sciebo.de/s/vyz8VwwunSodcrP/download Download test input] (tgz archive)
+* [https://uni-muenster.sciebo.de/s/gLEtX6m6g7x2ox6/download Download test input] (tgz archive)
 * SHA256 checksum: <code> d1677e755bf5feac025db6f427a929cbb2b881ee4b6e2ed13bda2b3c9a5dc8b0 </code>

← Older revision		Revision as of 09:03, 3 June 2022
Line 54:		Line 54:
	srun -n 18 gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000		srun -n 18 gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000

−	<code>~~mpirun~~ -n 18</code> will spawn 18 processes calling the <code>gmx_mpi</code> executable. <code>mdrun -deffnm MD_5NM_WATER</code> tells GROMACS to run a molecular dynamics simulation and use the MD_5NM_WATER.tpr as its input. <code>-nsteps 10000</code> tells it to only run 10000 steps of the simulation instead of the default 500000. We also added the <code>-quiet</code> flag to suppress some general output as GROMACS is quiet ''chatty''.	+	<code>srun -n 18</code> will spawn 18 processes calling the <code>gmx_mpi</code> executable. <code>mdrun -deffnm MD_5NM_WATER</code> tells GROMACS to run a molecular dynamics simulation and use the MD_5NM_WATER.tpr as its input. <code>-nsteps 10000</code> tells it to only run 10000 steps of the simulation instead of the default 500000. We also added the <code>-quiet</code> flag to suppress some general output as GROMACS is quiet ''chatty''.

@@ Line 3: / Line 3: @@
 __TOC__
-== Prepare your input ==
+== Preparations ==
 It is always a good idea to create some test input for your simulation. If you have a specific system you want to benchmark for a production run make sure you limit the time the simulation is run for. If you also want to test some code given different input sizes of data, prepare those systems (or data-sets) beforehand.
@@ Line 33: / Line 34: @@
   cd GROMACS_BENCHMARK_INPUT/
-== Allocate an interactive session ==
+=== Allocate an Interactive Session ===
 As a first measure you should allocate a full single node on your cluster and start an interactive session, i.e. login on the node. Often there are dedicated partitions/queues named "express" or "testing" or the like, for exactly this purpose with a time limit of a few hours. For a cluster running SLURM this could for example look like:
@@ Line 41: / Line 42: @@
 This will allocate 1 node with 72 tasks on the ''express'' partition for 2 hours and log you in (i.e. start a new bash shell) on the allocated node. Adjust this according to your cluster's provided resources! On the nodes we are using for this tutorial, hyperthreading is enabled. This means we requested 72 ''logical'' cores but actually only run on 36 ''physical'' cores. Keep this in mind as we also want to see the effect this has on our simulation.
-== First test run ==
 Next, you want to navigate to your input data and load the environment module, which gives you access to GROMACS binaries. Since environment modules are organized differently on many cluster the steps to load the correct module might look different at your site.
@@ Line 53: / Line 56: @@
 <code>mpirun -n 18</code> will spawn 18 processes calling the <code>gmx_mpi</code> executable. <code>mdrun -deffnm MD_5NM_WATER</code> tells GROMACS to run a molecular dynamics simulation and use the MD_5NM_WATER.tpr as its input. <code>-nsteps 10000</code> tells it to only run 10000 steps of the simulation instead of the default 500000. We also added the <code>-quiet</code> flag to suppress some general output as GROMACS is quiet ''chatty''.
-=== Output ===
 The simulation will presumably take a couple of seconds and the output will look similar to this:
@@ Line 76: / Line 80: @@
-== Second test run and monitoring with htop ==
+=== Live Monitoring with htop ===
 The 18 MPI processes each using 4 OpenMP threads add up to 18x4 (72) cores being used - so all of the resources we allocated when we requested the interactive session. However, we do want a bit more control over how many cores are actually being used. Therefore we can specify an environment variable to limit the number of OpenMP threads, i.e.

@@ Line 78: / Line 78: @@
 == Second test run and monitoring with htop ==
-The 18 MPI processes each using 4 OpenMP threads add up to 18x4 (72) cores being used - so all of the resources we allocated when we requested the interactive session. However, we do want a bit more control over how many cores are actually being used. Therefore we can either specify an environment variable to limit the number of OpenMP threads, i.e.
+The 18 MPI processes each using 4 OpenMP threads add up to 18x4 (72) cores being used - so all of the resources we allocated when we requested the interactive session. However, we do want a bit more control over how many cores are actually being used. Therefore we can specify an environment variable to limit the number of OpenMP threads, i.e.
   export OMP_NUM_THREADS=1
-or we can directly tell GROMACS to only use 1 OpenMP thread per process by adding the flag
+Alternatively we could directly tell GROMACS to only use 1 OpenMP thread per process by adding the flag <code>-ntomp 1</code>.
 For the second run we want to test the adjusted parameters and also monitor the CPU cores in real-time. We can do this for example by opening a second shell on the same node. On most clusters it is allowed to SSH on to a node you currently have allocated resources on. To do this, open a new terminal window (or tab), ssh on to the cluster and then on to the node your interactive session is running on
@@ Line 104: / Line 103: @@
 [[File:Benchmark_tutorial_gromacs_3.png|600px]]
-We can see that 18 cores are running at 100% as we intended. However, we can also observe that these are not always the same cores and the processes seem to randomly ''jump'' from one core to another. We can instruct GROMACS to pin those processes to the cores to prevent them from switching around using the flag
+We can see that 18 cores are running at 100% as we intended. However, we can also observe that these are not always the same cores and the processes seem to randomly ''jump'' from one core to another. We can instruct <code>srun</code> to pin those processes to the cores to prevent them from switching around using the flag
-  -pin on
+  --cpu-bind=cores
-Runnig the simulation again with
+Running the simulation again with
-  srun -n 18 gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000 -ntomp 1 -pin on
+  srun --cpu-bind=cores -n 18 gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000 -ntomp 1
 leads to the behavior as seen in the next screenshot:

@@ Line 117: / Line 117: @@
-Now that we know how to control our simulation, let's start to write a bash script to automate the process.
+Now that we know how to control our simulation, let's start to write a bash script to automate the process for different numbers of cores.
 ----
 '''Next''': [[Benchmarking_%26_Scaling_Tutorial/Automated_Benchmarking | Automated Benchmarking using a Job Script ]]
 '''Previous''': [[Benchmarking_%26_Scaling_Tutorial/Introduction | Introduction and Theory]]

@@ Line 69: / Line 69: @@
 At the end of the output, we will be given some performance metrics:
-* Wall time (or elapsed real time): 7.145s
+* Wall time (or elapsed real time): 5.795s
-* Core time (accumulated time of each core): 514.146s
+* Core time (accumulated time of each core): 417.111s
-* ns/day (nanoseconds one could simulate in 24h): 241.867
+* ns/day (nanoseconds one could simulate in 24h): 298.200
-* hours/ns (how many hours to simulate 1ns): 0.099
+* hours/ns (how many hours to simulate 1ns): 0.080
-* The total percentage of CPU usage (max = cores*100%): 7195.7%
+* The total percentage of CPU usage (max = cores*100%): 7197.4%

@@ Line 97: / Line 97: @@
 You will see a list of all running processes on the node as well as CPU gauges for all logical cores showing the current CPU usage. We can now run again the GROMACS command with adjusted parameters in the first shell
-  mpirun -n 18 gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000 -ntomp 1
+  srun -n 18 gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000 -ntomp 1
 and monitor the CPUs in the second one:
@@ Line 110: / Line 110: @@
 Runnig the simulation again with
-  mpirun -n 18 gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000 -ntomp 1 -pin on
+  srun -n 18 gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000 -ntomp 1 -pin on
 leads to the behavior as seen in the next screenshot: