Benchmarking & Scaling Tutorial/Introduction

Scalability

Often users who start running applications on an HPC system tend to assume the more resources (compute nodes / cores) they use, the faster their code will run (i.e. they expect a linear behaviour). Unfortunately this is not the case for the majority of applications. How fast a program runs with different amounts of resources is referred to as scalability. For parallel programs a limiting factor is defined by [Amdahl's law](https://en.wikipedia.org/wiki/Amdahl%27s_law). It takes into account the fact, that a certain amount of work of your code is done in parallel but the speedup is ultimately limited by the sequential part of the program.

Speedup and Efficiency

We assume that the total execution time $T$ of a program is comprised of

• $t_{s}$, a part of the code which can only run in serial
• $t_{p}$, a part of the code which can be parallelized
• $t_{o}$, parallel overheads due to, e.g. communication

The execution time of a serial code would then be

$T(1)=t_{s}+t_{p}$

The time for a parallel code, where the work would be perfectly divided by $P$ processors, would be given by

$T(P)=t_{s}+{\frac {t_{p}}{P}}+t_{o}(P)$

$t_{p}/P$ is the speed up amount of time due to the usage of multiple CPUs. The total **speedup** $S$ is defined as the ratio of the sequential to the parallel runtime:

$S={\frac {T(1)}{T(P)}}$

The **efficiency** is the speedup per processor, i.e.

$E={\frac {S}{P}}={\frac {T(1)}{P\cdot T(P)}}$

Amdahl's Law

Knowing that $t_{o}>0$, and writing $F={\frac {t_{s}}{t_{s}+t_{p}}}$ as the fraction of the serial code, we can rewrite this to

$S\leq {\frac {1}{F+(1-F)/P}}$

$E\leq {\frac {1}{PF+(1-F)}}$

This places an upper limit on the **strong scalability**, i.e. how quickly can we solve a problem of fixed size $N$ by increasing $P$. It is known as *Amdahl's Law*.