Application benchmarking - Revision history

Jan-eitzinger-77c0@uni-erlangen.de: /* Saturating pattern */

2019-05-28T12:20:11Z

Saturating pattern

Jan-eitzinger-77c0@uni-erlangen.de: /* Linear pattern */

2019-05-28T12:19:35Z

Linear pattern

Jan-eitzinger-77c0@uni-erlangen.de: /* SMT feature */

2019-05-28T12:14:57Z

SMT feature

Jan-eitzinger-77c0@uni-erlangen.de: /* Best practices */

2019-05-28T12:12:42Z

Best practices

Jan-eitzinger-77c0@uni-erlangen.de: /* Affinity control */

2019-05-28T11:44:35Z

Affinity control

Jan-eitzinger-77c0@uni-erlangen.de: /* Dynamic processor clock */

2019-05-28T11:31:18Z

Dynamic processor clock

Jan-eitzinger-77c0@uni-erlangen.de: /* Node topology */

2019-05-28T11:26:00Z

Node topology

Jan-eitzinger-77c0@uni-erlangen.de: /* Documentation */

2019-05-28T11:23:30Z

Documentation

Jan-eitzinger-77c0@uni-erlangen.de: /* Timing */

2019-05-28T11:22:33Z

Timing

Jan-eitzinger-77c0@uni-erlangen.de: /* Timing */

2019-05-28T10:19:27Z

Timing

@@ Line 474: / Line 474: @@
 [[File:Scaling-saturating.png]]
-This pattern occurs if the performance of the code is limited by a shared resource, e.g. memory bandwidth. This is not necessarily bad. If memory bandwidth is the limiting resource of a code a saturating pattern indicates that the resource is addressed and the code operates at the resource limit. This should be confirmed by HPM measurements.
+This pattern occurs if the performance of the code is limited by a shared resource, e.g. memory bandwidth. This is not necessarily bad. If memory bandwidth is the limiting resource of a code a saturating pattern indicates that the resource is addressed and the code operates at its resource limit. This should be confirmed by HPM measurements.
 A saturating pattern might also occur in certain cases of load imbalance.

@@ Line 468: / Line 468: @@
 This pattern occurs if no serious issues prevent a scalable performance. The slope may be anything smaller than 1 with 1 being optimal parallel scaling.
-This case applies if all used resources used by the code are also scalable (this means private to the worker: for example execution units or inner cache levels).
+This case applies if all resources used by the code are scalable (this means private to the worker: for example execution units or inner cache levels). Be aware that this behaviour may be caused by a very slow code that does not address any shared resource bottleneck.
 ==== Saturating pattern ====

@@ Line 455: / Line 455: @@
 === SMT feature ===
-Many processors today support Simultaneous multithreading (SMT) as a technology to increase the usage of instruction level parallel processing (ILP). The processor allows to run multiple threads (common is 2, 4 or 8) simultaneously on one core, which gives the instruction scheduler more independent instructions to feed the execution pipelines. SMT may increase the efficient use of ILP but comes at the cost of synchronization penalty. For application benchmarking good practice is to measure for each topological entity once with and once without SMT to quantify the effect.
+Many processors support simultaneous multithreading (SMT) as a technology to increase the usage of instruction level parallelism (ILP). The processor allows to run multiple threads (common is 2, 4 or 8) simultaneously on one core, which gives the instruction scheduler more independent instructions to feed the execution pipelines. SMT may increase the efficient use of ILP but comes at the cost of a thread synchronisation penalty. For application benchmarking good practice is to measure for each topological entity once with and once without SMT to quantify the effect of using SMT.
 == Interpretation of results ==

@@ Line 426: / Line 426: @@
 === Scaling core count ===
-Scaling the number of workers (and therefore processor cores) used tests the parallel scalability of an application and also reveals scaling bottlenecks in node architectures. To separate influences good practice is to first scale out inside a memory domain. Main memory bandwidth within one memory domain is currently the most important performance limiting shared resource on compute nodes. After scaling from 1 to n cores within one memory domain next is to scale across memory domains and finally across sockets with the previous case being the baseline for speedup. Finally one scales across nodes, again the baseline is now the single node result. This helps to separate different influences on scaling. One must be aware that there is no way to separate the pure parallel scalability influenced by e.g. serial fraction and load imbalance. For all scalability measurements the machine should be operated with fixed clock, that means Turbo mode has to be disabled. With Turbo mode turned on the result is influenced in addition by how sensitive the code is to frequency changes. For finding the optimal operating point for production it still might be important to also measure with Turbo mode enabled.
+Scaling the number of workers (and therefore processor cores) tests the parallel scalability of an application and also reveals scaling bottlenecks in node architectures. To separate influences good practice is to initially scale out within a memory domain. Main memory bandwidth within one memory domain is currently the most important performance limiting shared resource on compute nodes. After scaling from 1 to n cores within one memory domain next is to scale across memory domains, and finally across sockets, with the previous case being the baseline for speedup. Finally one scales across nodes, the baseline is now the single node result. This helps to separate different influences on scalability. One must be aware that there is no way to separate the parallel scalability influenced by e.g. serial fraction and load imbalance from architectural influences. For all scalability measurements the machine should be operated with fixed clock, that means Turbo mode has to be disabled. With Turbo mode turned on the result is influenced by how sensitive the code reacts to frequency changes. For finding the optimal operating point for production runs it also may be meaningful to also measure with Turbo mode enabled.
-For plotting performance results larger should be better. Use either useful work per time or just inverse runtime as performance metric.
+For plotting performance results larger should be better. Use either useful work per time or inverse runtime as performance metric.
-Besides the performance scaling you should also plot results as parallel speedup and efficiency. Parallel speedup is defined as
+Besides the performance scaling you should also plot results as parallel speedup and parallel efficiency. Parallel speedup is defined as
 :<math>S_\text{N} = \frac{T_\text{1}}{T_\text{N}}</math>
@@ Line 443: / Line 443: @@
 * Set a fixed frequency
-* Measure sequential baseline
+* Measure the sequential baseline
-* Scale within a memory domain with baseline sequential  result
+* Scale within a memory domain with baseline sequential result
 * Scale across memory domains with baseline one memory domain
 * (if applicable) Scale across sockets with baseline one socket
@@ Line 451: / Line 451: @@
 === Scaling data set size ===
-The target for this scaling variation is to ensure that all (most) data is loaded from a specific memory hierarchy level (e.g. L1 cache, last level cache or main memory). In some cases it is not possible to vary the data set size in fine steps, in such case the data set size should be varied such that data is located in every memory hierarchy location. This experiment should be initially performed with one worker and reveals if runtime contributions from data transfers add to the critical path. If performance is insensitive to where the data is loaded from it is likely that the code is not limited by data access costs. It is important to assure with hardware performance profiling that measured data volumes are in line with the desired target memory hierarchy level.
+The target for this scaling variation is to ensure that all (most) data is loaded from a specific memory hierarchy level (e.g. L1 cache, last level cache or main memory). In some cases it is not possible to vary the data set size in fine steps, in such cases the data set size should be varied such that data is located at least once in every memory hierarchy location. This experiment should be initially performed with one worker and reveals if runtime contributions from data transfers add to the critical path. If performance is insensitive to where the data is loaded from it is likely that the code is not limited by data access costs. It is important to assure with hardware performance profiling that measured data volumes are in line with the desired target memory hierarchy level.
 === SMT feature ===

@@ Line 411: / Line 411: @@
 == Affinity control ==
-Affinity control allows to specify on which execution resources (cores or SMT threads)
+Affinity control allows to specify on which execution resources (cores or SMT threads) threads are executed.
-Affinity control is crucial to
+Affinity control is crucial to:
 * eliminate performance variation
-* make use of architectural features
+* make deliberate use of architectural features
 * avoid resource contention
-Almost any runtime environment comes with some kind of affinity control. With OpenMP 4 a standardized [https://hpc-wiki.info/hpc/Binding/Pinning pinning interface] was introduced. Most solutions are environment variable based. A command line wrapper alternative is available in the [https://github.com/RRZE-HPC/likwid Likwid tools]: [https://github.com/RRZE-HPC/likwid/wiki/Likwid-Pin likwid-pin] and [https://github.com/RRZE-HPC/likwid/wiki/Likwid-Mpirun likwid-mpirun].
+Almost every parallel runtime environment comes with some kind of affinity control. With OpenMP 4 a standardised [https://hpc-wiki.info/hpc/Binding/Pinning pinning interface] was introduced. Most solutions are environment variable based. A command line wrapper alternative is available in the [https://github.com/RRZE-HPC/likwid Likwid tools]: [https://github.com/RRZE-HPC/likwid/wiki/Likwid-Pin likwid-pin] and [https://github.com/RRZE-HPC/likwid/wiki/Likwid-Mpirun likwid-mpirun].
 == Best practices ==

@@ Line 217: / Line 217: @@
 == Dynamic processor clock ==
-Older processors supported to dynamically decrease the clock to save power. Newer multicore processors in addition are capable of dynamically overclocking beyond the nominal frequency.
+Processors support to dynamically decrease the clock to save power. Newer multicore processors in addition are capable of dynamically overclocking beyond the nominal frequency. On Intel processors this feature is called "turbo mode". Turbo mode is a flavour of so called "dark silicon" optimisations. A chip has a thermal design power (TDP).
-On Intel processors this feature is called "turbo mode". Turbo mode is a flavor of so called "dark silicon" optimisations. A chip has a thermal design power (TDP).
+The TDP is the maximum heat the chip can dissipate. The power a chip consumes depends on the frequency and the square of the voltage it is operated with and of course the total number of transistors and wires active. There is a minimum voltage the chip can be operated with using a given frequency. If not all cores are used, parts of the chip can be turned off. This gives enough power headroom to increase the voltage and frequency and still stay within the TDP. The overclocking happens dynamically and is controlled by the hardware itself.
-Many technological advances in processor designs are targeted on managing the power to stay within the TDP and still reach a high performance. This is reflected in different clock domains for cores and uncore and special SIMD width specific turbo mode clocks. This means if a code uses wider SIMD units more transistors and more important wires are active and the chip can therefore only run at a lower peak frequency.
+Many technological advances in processor designs are targeted on managing to stay within the TDP and still reach high performance. This is reflected in different clock domains for cores and uncore and special SIMD width specific turbo mode clocks. If a code uses wider SIMD units more transistors and more important wires are active, and the chip can therefore only run at a lower peak frequency.
-Useful tools in this context are likwid-powermeter and likwid-setFrequencies. The former reports on Turbo mode steps, the latter on the current frequency settings.
+Useful tools in this context are likwid-powermeter and likwid-setFrequencies. The former reports on turbo mode steps, the latter on the current frequency settings.
-Note that likwid-powermeter reads out a special register to query the frequency steps for regular use but not for the special SIMD frequencies. Those frequencies are only available in the processor documentation from Intel. Also be aware that the documented peak frequencies are not guaranteed. Depending on the specific chip the actually reached frequency can be lower. A major complication for benchmarking is that on recent Intel chips even if turbo mode is disabled the chip may clock lower if using wide SIMD units. Again this is a dynamic process and depends on the specific chip. To be really sure one has to measure the actual average frequency with a a HPM tool as e.g. likwid-perfctr. Almost any likwid-perfctr performance group also shows the measured clock.
+Note that likwid-powermeter reads out a special register to query the frequency steps for regular use, but not for the special SIMD frequencies. Those frequencies are only available in the processor documentation from Intel. Also be aware that the documented peak frequencies are not guaranteed. Depending on the specific chip the actually reached frequency can be lower. A major complication for benchmarking is, that on recent Intel chips even if turbo mode is disabled the chip may clock lower if using wide SIMD units. Again this is a dynamic process and depends on the specific chip. To be really sure one has to measure the actual average frequency with a a HPM tool as e.g. likwid-perfctr. Almost any likwid-perfctr performance group shows the measured clock as derived metric.
-Below is an example for querying the Turbo mode steps of a Intel Skylake chip:
+Below is an example for querying the Turbo mode steps of an Intel Skylake chip:
 <syntaxhighlight>
 $likwid-powermeter -i

@@ Line 97: / Line 97: @@
 Knowledge about node topology and properties is essential to plan benchmarking and interpret results.
-Important questions to ask are (Also see the extended list below):
+Important questions to ask are (also see the extended list in the next section):
 * What is the topology and size of all memory hierarchy levels.
 * Which caches levels are private to cores and which are shared?
 * How many and which processors share memory hierarchy levels?
-* What is the NUMA topology? This means how many and which processors share a memory interface and how many memory interfaces are there?
+* What is the NUMA main memory topology? How many and which processors share a memory interface and how many memory interfaces are there?
 * Is SMT available? How many and which processors share a core?
-* On Intel processors: Is cluster on die mode enabled? With cluster on die the memory interfaces within one socket are cut in two parts.
+* On Intel processors: Is cluster-on-die mode enabled? With cluster-on-die memory interfaces within one socket are cut in two parts.
 The [https://github.com/RRZE-HPC/likwid Likwid tools] provide a single tool likwid-topology reporting all required topology and memory hierarchy info from a single source.

@@ Line 92: / Line 92: @@
 == Documentation ==
-Without a proper documentation of code generation, system state and runtime modalities, it can be difficult to reproduce performance results. Best practice is to automate the automatic logging of build settings, system state and runtime settings using automated benchmark scripts. Still, too much automation might also result in errors or hinder a fast workflow due to inflexibilities in benchmarking or intransparency of what actually happens. Therefore it is recommended to also execute steps by hand in addition to automated benchmark execution.
+Without a proper documentation of code generation, system state and runtime modalities, it can be difficult to reproduce performance results. Best practice is to automate the automatic logging of build settings, system state and runtime settings using automated benchmark scripts. However, too much automation might also result in errors or hinder a fast workflow due to inflexibilities in benchmarking or in-transparency of what actually happens. Therefore it is recommended to also execute steps by hand in addition to automated benchmark execution.
 == Node topology ==

@@ Line 23: / Line 23: @@
 == Timing ==
-For benchmarking, an accurate wallclock timer (end-to-end stop watch) is required. Every timer has a minimal time resolution that can be measured. Therefore, if the code region to be measured is too short, the measurement must be extended until it reaches a time duration that can be resolved by the timer. There are OS-specific routines (POSIX and Windows), as well as programming-model or programming-language-specific solutions available. The latter have the advantage to be portable across operating systems. In any case, one has to read the documentation to ensure the exact properties of the routine used.
+For benchmarking, an accurate wallclock timer (end-to-end stop watch) is required. Every timer has a minimal time duration that can be measured. Therefore, if the code region to be measured is too short, the measurement must be extended until it reaches a time duration that can be resolved by the timer. There are OS-specific routines (POSIX and Windows), as well as programming-model or programming-language-specific solutions available. The latter have the advantage to be portable across operating systems. In any case, one has to read the documentation to ensure the exact properties of the routine used.
 Recommended timing routines are
@@ Line 31: / Line 31: @@
 * Timing in instrumented Likwid regions based on cycle counters for very short measurements
-While there are also programming language specific solutions (e.g. in C++ and Fortran), it is recommended to use the OS solution. In case of Fortran this requires providing a wrapper function to the C call (see example below).
+While there are also programming language specific solutions (e.g. in C++ and Fortran), it is recommended to use the OS solution. In case of Fortran this requires to provide a wrapper function to the C function call (see example below).
 === Examples ===
@@ Line 64: / Line 64: @@
 ==== Fortran example ====
-In Fortran just add the following wrapper to above C module. You may have to adjust the name mangling to your Fortran compiler. Then you can just link with your Fortran application against the object file.
+In Fortran just add the following wrapper to above C module. You may have to adjust the name mangling to your Fortran compiler. You can just link with your Fortran application against the generated object file.
 <syntaxhighlight lang="c">
@@ Line 87: / Line 87: @@
 ==== Example code ====
-This example code contains a ready-to-use timing routine with C and F90 examples as well as a more advanced timer C module based on the RDTSC instruction.
+The example code contains a ready-to-use timing routine with C and F90 examples as well as a more advanced timer C module based on the RDTSC instruction. You can download an archive containing working timing routines with examples [https://github.com/RRZE-HPC/Code-teaching/releases/download/v1.0-demos/timing-demo-1.0.zip here].
 For a working real example project including above timing module you can have a look at ''The Bandwidth Benchmark'' [https://github.com/RRZE-HPC/TheBandwidthBenchmark teaching code].