Benchmarks

General

Benchmarks are applications designed to characterize different performance aspects of a system. The results can then be used to compare and rank different systems (e.g. TOP500 list). Furthermore, they can be used to identify problems and monitor the progress of fixing them. They are also used by vendors to check if the performance requested by the customer is actually achieved. An important aspect for benchmarks is portability, as the main reason for not using "real" appliactions is usually that it takes too long and too many changes to get them to run on a different system. System-related adjustments should be avoided or at least kept to a minimum.

Benchmark types

Individual benchmarks can usually be placed in one of the following categories:

Category	Example	Description
Raw machine performance	Tflops	machine specifications, e.g. FP peak performance
Microbenchmarks	Stream	"small" basic components, e.g. latency, bandwidth, communication overhead...
Algorithmic benchmarks	Linpack	simple algorithms with increasing problem sizes, close to the "kernels"
Compact Apps/Kernels	MINI-Apps/NPB	more realistic applications that consider more aspects
Application Suites	SPEC	set of real key applications with different limiting factors
User-specific applications	Custom benchmarks	run your own application and use hardware counters to manually compute metrics of your choice

Resources

SPEC (Standard Performance Evaluation Cooperation)

STREAM (Sustainable Memory Bandwidth in High Performance Computers)