Automated Benchmarking using JUBE

Introduction

The Jülich Benchmarking Environment is an application that helps you automate your workflow for system and application benchmarking.

JUBE allows you to define different steps of your workflow with dependencies between them.

One key advantage of using JUBE, as opposed to manually running an application in different configurations in a job script is that individual run configurations are automatically separated into separate workpackages with individual run directories, while common files and directories (like input files, preprocessing, etc.) can easily be integrated into the workflow.

Furthermore, application output (such as the runtime of the application) can easily be parsed and output in CSV or human-readable table format.

Writing a minimal configuration

As JUBE executes each workpackage (step with concrete configuration) in its own sandbox, the benchmark configuration must specify a fileset that either copies or links files into the run directory. Parameters have a separator defined (default is ',') that is used to tokenize the parameter string. Each token will be part of a separate configuration. In this example, the comma-separated list of tasks will result in the parameter tasks with one specific value in 15 different workpackages.

 1<?xml version="1.0" encoding="UTF-8"?>
 2<jube>
 3  <benchmark name="GROMACS" outpath="bench_run">
 4    <comment>A minimal JUBE config to run our GROMACS example</comment>
 5
 6    <!-- Configuration -->
 7    <parameterset name="execute_pset">
 8      <parameter name="tasks">1,2,4,8,12,18,24,30,36,42,48,54,60,66,72</parameter>
 9      <!-- you could also compute the list using Python 
10         <parameter name="tasks" mode="python">
11           ",".join([str(x) for x in range(1,73) if (x % 6 == 0 and x > 10) or (x % 4 == 0 and x < 10) or x == 2 or x==1 ])
12         </parameter>
13      -->
14    </parameterset>
15    
16    <!-- Input files -->
17    <fileset name="gromacs_files">
18      <link>MD_5NM_WATER.deff</link> <!-- link input file -->
19    </fileset>
20
21    <!-- Operation -->
22    <step name="run">
23      <use>execute_pset</use>  <!-- use parameterset -->
24      <use>gromacs_files</use> <!-- use fileset -->
25      <do>srun -n $ntasks gmx_mpi -quiet mdrun -deffnm MD_5NM_WATER -nsteps 10000 -ntomp 1 -pin on</do> <!-- start GROMACS -->
26    </step>    
27  </benchmark>
28</jube>

Further information

• JUBE Documentation
• Video: Automated Benchmarking with JUBE
• Video: Jube by Example
• Video: Application Benchmarking with JUBE: Lessons Learned

---

Next: Plotting and Interpreting Results

Previous: Automated Benchmarking using a Job Script