Job efficiency - Revision history

Robert-schade-e757@uni-paderborn.de at 11:30, 19 July 2024

2024-07-19T11:30:36Z

Alex-wiens-5f0c@uni-paderborn.de: Add load imbalance example

2024-03-01T16:21:27Z

Add load imbalance example

Alex-wiens-5f0c@uni-paderborn.de: Add filesystem access example

2024-02-29T12:25:44Z

Add filesystem access example

Alex-wiens-5f0c@uni-paderborn.de: Add performance expectation details

2024-02-28T17:37:13Z

Add performance expectation details

Alex-wiens-5f0c@uni-paderborn.de: Add details to scaling

2024-02-28T17:27:01Z

Add details to scaling

Alex-wiens-5f0c@uni-paderborn.de: Add details about Filesystem access

2024-02-28T16:58:56Z

Add details about Filesystem access

Alex-wiens-5f0c@uni-paderborn.de: Add details to Load imbalance

2024-02-28T16:45:49Z

Add details to Load imbalance

Alex-wiens-5f0c@uni-paderborn.de: Add details to Resource oversubscription and Resource underutilization

2024-02-28T15:41:41Z

Add details to Resource oversubscription and Resource underutilization

Alex-wiens-5f0c@uni-paderborn.de: Add oversubscription example

2024-02-27T16:14:13Z

Add oversubscription example

Alex-wiens-5f0c@uni-paderborn.de: Add basic measurement considerations

2024-02-27T12:31:29Z

Add basic measurement considerations

← Older revision		Revision as of 11:30, 19 July 2024
Line 1:		Line 1:
	[[Category:HPC-User]]		[[Category:HPC-User]]
		+	[[Category:Benchmarking]]<nowiki />
		+

	Job efficiency describes how well a job makes use of the available resources.		Job efficiency describes how well a job makes use of the available resources.

@@ Line 70: / Line 70: @@
 == Load imbalance ==
 Load imbalance occurs when the work is unevenly distributed to the resources.
@@ Line 80: / Line 82: @@
 Imbalances can occur on different levels, for example, between cores on the same node, between sockets of the same node, between multiple nodes, etc.
@@ Line 85: / Line 94: @@
 == Filesystem access ==
-[[File:Job_efficiency_filesystem_access.png|400px|thumb|right|Filesystem access example: The job show regular behavior pattern and high network and filesystem utilization.]]
+[[File:Job_efficiency_filesystem_access.png|400px|thumb|right|Filesystem access example: The job shows regular behavior pattern and high network and filesystem utilization.]]
 Calculations can be slowed down by operations with high latency, for example, main memory accesses or filesystem accesses.

@@ Line 53: / Line 53: @@
 For the first job, the CPU time measurement shows 100% utilization, although the two threads are competing for the CPU core.
 For the second job, the CPU time measurement depicts degraded utilization, because of the thread scheduling overhead.
@@ Line 81: / Line 84: @@
 == Filesystem access ==
 Calculations can be slowed down by operations with high latency, for example, main memory accesses or filesystem accesses.
@@ Line 91: / Line 96: @@
 * Copying necessary data a local filesystem before access, e.g. memory mapped filesystems.
 * Writing result data to local filesystems before copying to destination filesystem.
@@ Line 109: / Line 121: @@
 Especially, one particular node transmits more packets than the other nodes.
 From this example, the communication overhead of increased numbers of cores can be recognized.

@@ Line 95: / Line 95: @@
 == Scaling ==
 [[File:Job_efficiency_scaling_performance_degradation.png|400px|thumb|right|Scaling performance degradation example: Measurements show how with increased number of cores and nodes the performance degrades and the communication overhead increases.]]
@@ Line 110: / Line 109: @@
 Especially, one particular node transmits more packets than the other nodes.
 From this example, the communication overhead of increased numbers of cores can be recognized.
 = Performance expectation and reality =

← Older revision		Revision as of 17:27, 28 February 2024
Line 95:		Line 95:

	== Scaling ==		== Scaling ==
		+
		+
		+	[[File:Job_efficiency_scaling_performance_degradation.png\|400px\|thumb\|right\|Scaling performance degradation example: Measurements show how with increased number of cores and nodes the performance degrades and the communication overhead increases.]]
		+
		+	[[Scaling]] issues can be sometimes spotted by comparing utilization measurements of jobs with different scaling parameters.
		+	For example, [[Job efficiency#Load_imbalance\|load imbalances]] can occur, when a workload is parallelized to multiple nodes.
		+	It is also possible to recognize the effect of degraded performance and increased communication overhead.
		+	However, the exact reason for scaling deficiencies have to be investigated during detailed [[Performance profiling]].
		+
		+	The Scaling performance degradation example shows an application that is executed with four different configurations on 1 core, 128 cores (1 node), 256 cores (2 nodes) and 512 cores (4 nodes).
		+	The core-specific Flops metric decreases with increased core count.
		+	The memory bandwidth metric increases with the number of involved CPU sockets.
		+	The transmitted network packets metric increases once more than a single node is involved.
		+	Especially, one particular node transmits more packets than the other nodes.
		+	From this example, the communication overhead of increased numbers of cores can be recognized.
		+

	= Performance expectation and reality =		= Performance expectation and reality =

← Older revision		Revision as of 16:58, 28 February 2024
Line 82:		Line 82:
	== Filesystem access ==		== Filesystem access ==

		+	Calculations can be slowed down by operations with high latency, for example, main memory accesses or filesystem accesses.
		+	Regular accesses to the filesystem can limit the application performance.
		+	Accesses to network filesystems are especially slow.
		+	Additionally, network operations (and accesses to shared filesystems) are susceptible to the network (and filesystem storage) utilization of other jobs and can impact other job's performance.
		+	Impact of filesystem accesses can be mitigated by:
		+
		+	* Reading necessary data into memory before access. Reading of larger contiguous datasets is often faster than irregular accesses.
		+	* Copying necessary data a local filesystem before access, e.g. memory mapped filesystems.
		+	* Writing result data to local filesystems before copying to destination filesystem.

@@ Line 42: / Line 42: @@
 Either, the application accidentally spawns more worker threads than intended or the job allocation includes too few CPU cores.
-The oversubscription example shows two jobs, with more threads than cores.
+Check the following configurations:
 The first job with 2 threads per core and the second one with 16 threads per core.
 Three metrics are shown: node-level CPU load, core-level CPU load and core-level CPU time.
@@ Line 52: / Line 57: @@
 == Resource underutilization ==
 == Load imbalance ==
 == Filesystem access ==

@@ Line 34: / Line 34: @@
 == Resource oversubscription ==
 Oversubscription of resources happens when the application's assignment of work to resources is flawed.
@@ Line 39: / Line 41: @@
 Usually, this indicates a misconfiguration.
 Either, the application accidentally spawns more worker threads than intended or the job allocation includes too few CPU cores.

@@ Line 14: / Line 14: @@
 This can be done by performing [[Performance profiling]] or by accessing the cluster's [[Performance Monitoring]].
 The measured [[Performance metrics]] give insight into the utilization of the allocated resources and possible deficiencies.
 = Common job efficiency pitfalls =
 == Resource oversubscription ==
 == Resource underutilization ==
 == Load imbalance ==