Benchmarking & Scaling Tutorial/Automated Benchmarking JUBE - Revision history

Marc-andre-hermanns-bc32@rwth-aachen.de at 07:55, 25 June 2025

2025-06-25T07:55:50Z

Marc-andre-hermanns-bc32@rwth-aachen.de: Further clarifications to updating patterns and generating tables.

2025-06-25T07:31:42Z

Further clarifications to updating patterns and generating tables.

Marc-andre-hermanns-bc32@rwth-aachen.de: Fixing errors, extending workflow description

2025-06-25T07:05:53Z

Fixing errors, extending workflow description

Marc-andre-hermanns-bc32@rwth-aachen.de: Adding output parsing and result table

2025-06-24T18:55:25Z

Adding output parsing and result table

Marc-andre-hermanns-bc32@rwth-aachen.de: Fix incomplete sentence.

2022-11-18T12:10:50Z

Fix incomplete sentence.

Marc-andre-hermanns-bc32@rwth-aachen.de: Initial minimal config

2022-11-18T11:48:55Z

Initial minimal config

Marc-andre-hermanns-bc32@rwth-aachen.de: Fixing title

2022-11-17T14:39:47Z

Fixing title

Marc-andre-hermanns-bc32@rwth-aachen.de: Initial content

2022-11-17T14:37:33Z

Initial content

New page

{{DISPLAYTITLE:Automated Benchmarking using a Job Script}}<nowiki />
{{Syllabus Benchmarking & Scaling}}<nowiki />
__TOC__

== Introduction ==

The Jülich Benchmarking Environment is an application that helps you automate your workflow for system and application benchmarking.

JUBE allows you to define different steps of your workflow with dependencies between them.

One key advantage of using JUBE, as opposed to manually running an application in different configurations in a job script is that individual run configurations are automatically separated into separate workpackages with individual run directories, while common files and directories (like input files, preprocessing, etc.) can easily be integrated into the workflow.

Furthermore, application output (such as the runtime of the application) can easily be parsed and output in CSV or human-readable table format.

== Further information ==

* [https://apps.fz-juelich.de/jsc/jube/jube2/docu/index.html JUBE Documentation ]
* [https://www.youtube.com/watch?v=CTexZWKhF0I Video: Automated Benchmarking with JUBE ]
* [https://www.youtube.com/watch?v=pC5XZTILt1o Video: Jube by Example ]
* [https://www.youtube.com/watch?v=0u6uKMT79e8 Video: Application Benchmarking with JUBE: Lessons Learned ]
----
'''Next''': [[Benchmarking_%26_Scaling_Tutorial/Results | Plotting and Interpreting Results ]]

'''Previous''': [[Benchmarking_%26_Scaling_Tutorial/Automated_Benchmarking | Automated Benchmarking using a Job Script ]]

@@ Line 161: / Line 161: @@
 <pre>
-$ jube result -a -u gromacs.xml jube_run --id <jube_run_id>
+$ jube result -a -u gromacs.xml jube_run --id 1
 gromacs_run:
 | wp | tasks | gromacs_core_time[s] | gromacs_wall_time[s] | gromacs_core_perf[ns/day] | gromacs_wall_perf[hours/ns] |

@@ Line 156: / Line 156: @@
 </syntaxhighlight>
-Resulting in the following output.
+As especially in the initial specification phase of a workflow, it may not be clear which patterns need to be used for which files, JUBE allows to update the patterns, rerun the analysis, and reprint the result tables even after the workflow steps have finished.
 <pre>
-$ jube result -a jube_run --id <jube_run_id>
+$ jube result -a -u gromacs.xml jube_run --id <jube_run_id>
 gromacs_run:
 | wp | tasks | gromacs_core_time[s] | gromacs_wall_time[s] | gromacs_core_perf[ns/day] | gromacs_wall_perf[hours/ns] |
@@ Line 180: / Line 182: @@
 </pre>
 == Further information ==

@@ Line 49: / Line 49: @@
 </syntaxhighlight>
 This configuration will already create separate directories for each of the measurements, which makes sure that temporary files written by the application do not interact across the different measurements.
 == Parsing output ==
 GROMACS outputs performance numbers to `stderr`.
 As identifying such output from executions is a core part of benchmarking, JUBE provides infrastructure to parse output and store specific information to output this information later in result tables.
@@ Line 62: / Line 125: @@
          <pattern name="gromacs_core_time" unit="s">Time:\s*${jube_pat_fp}</pattern>
          <pattern name="gromacs_wall_time" unit="s">Time:\s*${jube_pat_nfp}\s*${jube_pat_fp}</pattern>
-         <pattern name="gromacs_core_perf" unit="ns/day">Time:\s*${jube_pat_fp}</pattern>
+         <pattern name="gromacs_core_perf" unit="ns/day">Performance:\s*${jube_pat_fp}</pattern>
-         <pattern name="gromacs_wall_perf" unit="hours/ns">Time:\s*${jube_pat_nfp}\s*${jube_pat_fp}</pattern>
+         <pattern name="gromacs_wall_perf" unit="hours/ns">Performance:\s*${jube_pat_nfp}\s*${jube_pat_fp}</pattern>
      </patternset>
 </syntaxhighlight>
@@ Line 100: / Line 163: @@
 | wp | tasks | gromacs_core_time[s] | gromacs_wall_time[s] | gromacs_core_perf[ns/day] | gromacs_wall_perf[hours/ns] |
 |----|-------|----------------------|----------------------|---------------------------|-----------------------------|
-|  0 |     1 |               44.366 |               44.366 |                    44.366 |                      44.366 |
+|  0 |     1 |               44.366 |               44.366 |                    38.952 |                       0.616 |
-|  1 |     2 |               46.942 |               23.471 |                    46.942 |                      23.471 |
+|  1 |     2 |               46.942 |               23.471 |                    73.630 |                       0.326 |
-|  2 |     4 |               49.548 |               12.387 |                    49.548 |                      12.387 |
+|  2 |     4 |               49.548 |               12.387 |                   139.513 |                       0.172 |
-|  3 |     8 |               52.969 |                6.621 |                    52.969 |                       6.621 |
+|  3 |     8 |               52.969 |                6.621 |                   261.002 |                       0.092 |
-|  4 |    12 |               59.370 |                4.948 |                    59.370 |                       4.948 |
+|  4 |    12 |               59.370 |                4.948 |                   349.291 |                       0.069 |
-|  5 |    18 |               66.097 |                3.672 |                    66.097 |                       3.672 |
+|  5 |    18 |               66.097 |                3.672 |                   470.607 |                       0.051 |
-|  6 |    24 |               76.391 |                3.183 |                    76.391 |                       3.183 |
+|  6 |    24 |               76.391 |                3.183 |                   542.905 |                       0.044 |
-|  7 |    30 |               89.233 |                2.975 |                    89.233 |                       2.975 |
+|  7 |    30 |               89.233 |                2.975 |                   580.965 |                       0.041 |
-|  8 |    36 |               91.187 |                2.533 |                    91.187 |                       2.533 |
+|  8 |    36 |               91.187 |                2.533 |                   682.175 |                       0.035 |
-|  9 |    42 |               99.743 |                2.375 |                    99.743 |                       2.375 |
+|  9 |    42 |               99.743 |                2.375 |                   727.597 |                       0.033 |
-| 10 |    48 |              183.114 |                3.815 |                   183.114 |                       3.815 |
+| 10 |    48 |              183.114 |                3.815 |                   452.978 |                       0.053 |
-| 11 |    54 |              121.728 |                2.255 |                   121.728 |                       2.255 |
+| 11 |    54 |              121.728 |                2.255 |                   766.524 |                       0.031 |
-| 12 |    60 |              199.882 |                3.332 |                   199.882 |                       3.332 |
+| 12 |    60 |              199.882 |                3.332 |                   518.714 |                       0.046 |
 | 13 |    66 |                      |                      |                           |                             |
-| 14 |    72 |              116.555 |                1.619 |                   116.555 |                       1.619 |
+| 14 |    72 |              116.555 |                1.619 |                  1067.403 |                       0.022 |
 </pre>
@@ Line 121: / Line 184: @@
 * [https://mahermanns.github.io/jube-novice/index.html Reproducible HPC Workflows using JUBE ]
-* [https://apps.fz-juelich.de/jsc/jube/jube2/docu/index.html JUBE Documentation ]
+* [https://apps.fz-juelich.de/jsc/jube/jube2/docu/index.html JUBE Documentation (incl. Tutorial) ]
 * [https://www.youtube.com/watch?v=CTexZWKhF0I Video: Automated Benchmarking with JUBE ]
 * [https://www.youtube.com/watch?v=pC5XZTILt1o Video: Jube by Example ]

@@ Line 27: / Line 27: @@
      <parameterset name="execute_pset">
        <parameter name="tasks">1,2,4,8,12,18,24,30,36,42,48,54,60,66,72</parameter>
        <!-- you could also compute the list using Python
           <parameter name="tasks" mode="python">
             ",".join([str(x) for x in range(1,73) if (x % 6 == 0 and x > 10) or (x % 4 == 0 and x < 10) or x == 2 or x==1 ])
@@ Line 33: / Line 33: @@
        -->
      </parameterset>
-−
      <!-- Input files -->
      <fileset name="gromacs_files">
@@ Line 43: / Line 43: @@
        <use>execute_pset</use>  <!-- use parameterset -->
        <use>gromacs_files</use> <!-- use fileset -->
-       <do>srun -n $ntasks gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000 -ntomp 1 -pin on</do> <!-- start GROMACS -->
+       <do>srun -n $tasks gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000 -ntomp 1 -pin on</do> <!-- start GROMACS -->
      </step>
    </benchmark>
 </jube>
 </syntaxhighlight>
 == Further information ==
 * [https://apps.fz-juelich.de/jsc/jube/jube2/docu/index.html JUBE Documentation ]
 * [https://www.youtube.com/watch?v=CTexZWKhF0I Video: Automated Benchmarking with JUBE ]

@@ Line 15: / Line 15: @@
 == Writing a minimal configuration ==
-As JUBE executes each workpackage (step with concrete configuration) in its own sandbox, the benchmark configuration must specify
+As JUBE executes each workpackage (step with concrete configuration) in its own sandbox, the benchmark configuration must specify a fileset that either copies or links files into the run directory.
 <syntaxhighlight lang="xml" line>
 <?xml version="1.0" encoding="UTF-8"?>

← Older revision		Revision as of 11:48, 18 November 2022
Line 12:		Line 12:

	Furthermore, application output (such as the runtime of the application) can easily be parsed and output in CSV or human-readable table format.		Furthermore, application output (such as the runtime of the application) can easily be parsed and output in CSV or human-readable table format.
		+
		+	== Writing a minimal configuration ==
		+
		+	As JUBE executes each workpackage (step with concrete configuration) in its own sandbox, the benchmark configuration must specify
		+	<syntaxhighlight lang="xml" line>
		+	<?xml version="1.0" encoding="UTF-8"?>
		+	<jube>
		+	<benchmark name="GROMACS" outpath="bench_run">
		+	<comment>A minimal JUBE config to run our GROMACS example</comment>
		+
		+	<!-- Configuration -->
		+	<parameterset name="execute_pset">
		+	<parameter name="tasks">1,2,4,8,12,18,24,30,36,42,48,54,60,66,72</parameter>
		+	<!-- you could also compute the list using Python
		+	<parameter name="tasks" mode="python">
		+	",".join([str(x) for x in range(1,73) if (x % 6 == 0 and x > 10) or (x % 4 == 0 and x < 10) or x == 2 or x==1 ])
		+	</parameter>
		+	-->
		+	</parameterset>
		+
		+	<!-- Input files -->
		+	<fileset name="gromacs_files">
		+	<link>MD_5NM_WATER.deff</link> <!-- link input file -->
		+	</fileset>
		+
		+	<!-- Operation -->
		+	<step name="run">
		+	<use>execute_pset</use> <!-- use parameterset -->
		+	<use>gromacs_files</use> <!-- use fileset -->
		+	<do>srun -n $ntasks gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000 -ntomp 1 -pin on</do> <!-- start GROMACS -->
		+	</step>
		+	</benchmark>
		+	</jube>
		+	</syntaxhighlight>

	== Further information ==		== Further information ==

@@ Line 1: / Line 1: @@
-{{DISPLAYTITLE:Automated Benchmarking using a Job Script}}<nowiki />
+{{DISPLAYTITLE:Automated Benchmarking using JUBE}}<nowiki />
 {{Syllabus Benchmarking & Scaling}}<nowiki />
 __TOC__