Relevant test case

A relevant test case is a combination of data set, application and parameters which reflects production behaviour or at least allow assumptions (to be proven) about real operating point. The definition of a relevant test case is essential and vital for performance engeneering both for developing an application and for efficient use of (even blackboxed) application. Typically it needs (at least basic) knowledge about algorithms used in the application, prediction about needed size of computation jobs, and of course knowledge, which features of software will be used.

In the first approximation a [set of] real production test cases[s] is a relevant test case for itself, sic! However these test cases are typicall large, long-running and unhandy to analyse. A reduced relevant test case is needed. There are some typical ways to get a reduced test case:

• Reduce the size of data set (e.g. grid resolution)
• Crop the execution after a handful of iterations (prior reaching the convergence)
• Omitting some computation parts with known behaviour

Pay attention to the following:

• The same software path as in production must be used. If you use in production a compute kernel A, a data set for kernel B would be not relevant.
• The hotspots of production runs must also be hotspost in the reduced relevant test case (rule-of-thumb: about half of the overall excution time should be in hotspots). This in turn leads to the rule: Do not downsize too much.
• Is the computation numerically stable?
• Know the scalability pattern of your application depending on data set size: linear, logarithmical, polynomial (O², O³?), or exponentially? (In latter case don't care about scalability - you won't be able to compute interesting data sets until the end of the world.)
• if you accieved any results (e.g. scalability improvement) on a reduced test case, confirm this with at least selective tests on a full-size test case.