Benchmarks

General

Benchmarks are applications designed to characterize the performance of a system. The results can then be used to compare and rank different systems (e.g. TOP500 list). Furthermore, they can be used to identify problems and monitor the progress of fixing them. They are also used by vendors to check if the performance requested by the customer is actually achieved. An important aspect for benchmarks is portability, as the main reason for not using "real" appliactions is usually that it takes too long and too many changes to get them to run on a different system. System-related adjustments should be avoided or at least kept to a minimum.

Individual Benchmarks usually cannot fully characterize all performance aspects of a system, but measure one specific metric that is then comparable to other systems. As a result, the peak performance achieved by the benchmark is usually not achievable with real applications, where other contributing factors and holdbacks occur. Different Benchmarks can be grouped together in so-called Benchmark suites to cover more performance aspects.

Benchmark types

Individual benchmarks can usually be placed in one of the following categories:

Category	Example	Description
Microbenchmarks	Stream	"small" basic components, e.g. latency, bandwidth, communication overhead...
Algorithmic benchmarks	Linpack	simple algorithms with increasing problem sizes, close to the "kernels"
Application Suites	SPEC	set of real key applications with different limiting factors
Compact Apps/Kernels	MINI-Apps/NPB	more realistic applications that consider more aspects
User-specific applications	Custom benchmarks	run your own application and use hardware counters to manually compute metrics of your choice
Raw machine performance	Tflops	machine specifications, e.g. FP peak performance

Resources

SPEC (Standard Performance Evaluation Cooperation)

STREAM (Sustainable Memory Bandwidth in High Performance Computers)

Linpack

NPB