Difference between revisions of "Benchmarks"
m |
|||
| Line 1: | Line 1: | ||
== General == | == General == | ||
| − | Benchmarks are applications designed to characterize | + | Benchmarks are applications designed to characterize the performance of a system. The results can then be used to compare and rank different systems (e.g. [https://www.top500.org/ TOP500 list]). Furthermore, they can be used to identify problems and monitor the progress of fixing them. They are also used by vendors to check if the performance requested by the customer is actually achieved. An important aspect for benchmarks is portability, as the main reason for not using "real" appliactions is usually that it takes too long and too many changes to get them to run on a different system. System-related adjustments should be avoided or at least kept to a minimum. |
| + | |||
| + | Individual Benchmarks usually cannot fully characterize all performance aspects of a system, but measure one specific metric that is then comparable to other systems. As a result, the peak performance achieved by the benchmark is usually not achievable with real applications, where other contributing factors and holdbacks occur. Different Benchmarks can be grouped together in so-called Benchmark suites to cover more performance aspects. | ||
== Benchmark types == | == Benchmark types == | ||
| Line 7: | Line 9: | ||
Individual benchmarks can usually be placed in one of the following categories: | Individual benchmarks can usually be placed in one of the following categories: | ||
| − | {| class="wikitable" style="width: | + | {| class="wikitable" style="width: 60%;" |
| '''Category''' || '''Example''' || '''Description''' | | '''Category''' || '''Example''' || '''Description''' | ||
| − | |||
| − | |||
|- | |- | ||
| Microbenchmarks || Stream || "small" basic components, e.g. latency, bandwidth, communication overhead... | | Microbenchmarks || Stream || "small" basic components, e.g. latency, bandwidth, communication overhead... | ||
|- | |- | ||
| Algorithmic benchmarks || Linpack || simple algorithms with increasing problem sizes, close to the "kernels" | | Algorithmic benchmarks || Linpack || simple algorithms with increasing problem sizes, close to the "kernels" | ||
| + | |- | ||
| + | | Application Suites || SPEC || set of real key applications with different limiting factors | ||
|- | |- | ||
| Compact Apps/Kernels || MINI-Apps/NPB || more realistic applications that consider more aspects | | Compact Apps/Kernels || MINI-Apps/NPB || more realistic applications that consider more aspects | ||
|- | |- | ||
| − | | | + | | User-specific applications || Custom benchmarks || run your own application and use hardware counters to manually compute metrics of your choice |
|- | |- | ||
| − | | | + | | Raw machine performance || Tflops || machine specifications, e.g. FP peak performance |
|} | |} | ||
| Line 28: | Line 30: | ||
[https://www.cs.virginia.edu/stream/ STREAM] (Sustainable Memory Bandwidth in High Performance Computers) | [https://www.cs.virginia.edu/stream/ STREAM] (Sustainable Memory Bandwidth in High Performance Computers) | ||
| + | |||
| + | [http://www.netlib.org/benchmark/hpl/ Linpack] | ||
| + | |||
| + | [https://www.nas.nasa.gov/publications/npb.html NPB] | ||
Revision as of 12:48, 31 January 2019
General
Benchmarks are applications designed to characterize the performance of a system. The results can then be used to compare and rank different systems (e.g. TOP500 list). Furthermore, they can be used to identify problems and monitor the progress of fixing them. They are also used by vendors to check if the performance requested by the customer is actually achieved. An important aspect for benchmarks is portability, as the main reason for not using "real" appliactions is usually that it takes too long and too many changes to get them to run on a different system. System-related adjustments should be avoided or at least kept to a minimum.
Individual Benchmarks usually cannot fully characterize all performance aspects of a system, but measure one specific metric that is then comparable to other systems. As a result, the peak performance achieved by the benchmark is usually not achievable with real applications, where other contributing factors and holdbacks occur. Different Benchmarks can be grouped together in so-called Benchmark suites to cover more performance aspects.
Benchmark types
Individual benchmarks can usually be placed in one of the following categories:
| Category | Example | Description |
| Microbenchmarks | Stream | "small" basic components, e.g. latency, bandwidth, communication overhead... |
| Algorithmic benchmarks | Linpack | simple algorithms with increasing problem sizes, close to the "kernels" |
| Application Suites | SPEC | set of real key applications with different limiting factors |
| Compact Apps/Kernels | MINI-Apps/NPB | more realistic applications that consider more aspects |
| User-specific applications | Custom benchmarks | run your own application and use hardware counters to manually compute metrics of your choice |
| Raw machine performance | Tflops | machine specifications, e.g. FP peak performance |
Resources
SPEC (Standard Performance Evaluation Cooperation)
STREAM (Sustainable Memory Bandwidth in High Performance Computers)