Job Classification - Revision history

Jan-eitzinger-77c0@uni-erlangen.de at 08:00, 19 July 2024

2024-07-19T08:00:18Z

Jan-eitzinger-77c0@uni-erlangen.de: Add explanations

2024-07-19T05:27:13Z

Add explanations

Jan-eitzinger-77c0@uni-erlangen.de at 05:22, 19 July 2024

2024-07-19T05:22:46Z

Jan-eitzinger-77c0@uni-erlangen.de: Add resource contention classes

2024-07-19T05:20:46Z

Add resource contention classes

Jan-eitzinger-77c0@uni-erlangen.de at 05:01, 19 July 2024

2024-07-19T05:01:46Z

Jan-eitzinger-77c0@uni-erlangen.de at 04:57, 19 July 2024

2024-07-19T04:57:23Z

Jan-eitzinger-77c0@uni-erlangen.de: Pathological job classes

2024-07-19T04:54:02Z

Pathological job classes

New page

= Pathologic Job Classes =


= Categories =


== A - Resource Allocation ==


=== A.1 Low CPU utilisation ===

* Pattern description: Job uses less resources than requested
* Pattern detection: <code>mean(cpu_load) < normal(cpu_load)</code>
* Applicable situations: Node exclusive jobs only, because <code>cpu_load</code> is a node scope metric
* Message to user: ''Your job uses significantly less resources than requested.''
* Pattern mitigation: Check job script for allocation and placement errors

<pre>"parameter":["cpuload_threshold"],
"rule_terms":[
{"load_mean":"cpu_load.mean('all')"},
{"lowload":"load_mean < job.numHwthreads * cpuload_threshold"}
],
"output":"lowload"</pre>

=== A.2 Low GPU utilisation ===

* Pattern description: Job uses less GPUs than requested
* Pattern detection: <code>mean(acc_utilization) < alert(acc_utilization)</code> on any of the requested GPUs
* Applicable situations: *
* Message to user: ''Your job does not use some of the requested GPUs.''
* Pattern mitigation: Check job script for allocation and placement errors. Check application code.

<pre>"parameter":["gpuload_threshold"],
"rule_terms":[
{"load_mean":"acc_utilization.mean('all')"},
{"load_thres":"load_mean < gpuload_threshold"},
{"lowload":"load_thres.any('all')"}
],
"output":"lowload"</pre>

=== A.3 Resource oversubscription ===

* Pattern description: Job with too high cpu load
* Pattern detection: <code>avg(cpu_load) < peak(cpu_load)</code>
* Applicable situations: peak value needs to be scaled for shared jobs.
* Message to user: ''Job overloads available resources.''
* Pattern mitigation: Check job script and parallelisation for errors.

<pre>"parameter":["oversubscription_threshold"],
"rule_terms":[
{"load_mean":"cpu_load.mean('all')"},
{"load_thres":"(job.numHwthreadsjob.numNodes) * oversubscription_threshold"},
{"oversubscription":"load_mean > load_thres"}
],
"output":"oversubscription"</pre>

=== A.4 Short job ===

* Pattern description: Job is very short
* Pattern detection: <code>duration < threshold</code>
* Applicable situations: *
* Message to user: ''Your jobs duration is very short.''
* Pattern mitigation: Try to combine jobs. Implement job scheduling within one job.

<pre>"parameter":["duration_threshold"],
"rule_terms":[
{"short_job":"job.duration < duration_threshold"}
],
"output":"short_job"</pre>

=== A.5 Failing Chain-/Array-Jobs ===

* Pattern description: Jobs fail in rapid succession
* Pattern detection: <code>job_state == failed && duration < threshold</code> within some time. Needs state?
* Applicable situations: *
* Message to user: ''Many of your jobs fail.''
* Pattern mitigation: Fix error in job script and/or application setup.


== B - Resource Utilisation ==


=== B.1 Idle job ===

* Pattern description: Job runs but does not show any activity
* Pattern detection: <code>mean(flops_any) < alert(flops_any) && mean(mem_bw) < alert(mem_bw) && mean(net_bw) < alert(net_bw)</code>
* Applicable situations: Set of metrics to check has to be adapted for node exclusive or shared jobs. In case of shared jobs thresholds have to be scaled to the applicable value.
* Message to user: ''Your job does not use any resources.''
* Pattern mitigation: Check application code or log for potential errors or deadlocks.


=== B.2 Low CPU resource utilisation ===

* Pattern description: Job runs but has very low resource utilisation
* Pattern detection: <code>mean(flops_any) < caution(flops_any) && mean(mem_bw) < caution(mem_bw)</code>
* Applicable situations: Set of metrics to check has to be adapted for node exclusive or shared jobs. In case of shared jobs thresholds have to be scaled to the applicable value.
* Message to user: ''Your job has a very low resource utilisation.''
* Pattern mitigation: Check compile options and/or configuration for more optimal settings. Consider to use a more efficient application. Contact support for help to optimise code.


=== B.3 Low GPU resource utilisation ===

* Pattern description: Job runs but has very low GPU utilisation
* Pattern detection: <code>mean(acc_utilisation) < caution(acc_utilisation)</code>
* Applicable situations: *
* Message to user: ''Your job has a very low GPU utilisation.''
* Pattern mitigation: Check compile options and/or configuration for more optimal settings. Consider to use a more efficient application. Contact support for help to optimise code.


=== B.4 Memory leak ===

* Pattern description: Usage of memory capacity monotonically increases
* Pattern detection: <code>avgSlope(mem_used) >> 0</code>
* Applicable situations: *
* Message to user: ''Your job might have a memory leak.''
* Pattern mitigation: Check application code, use other MPI implementation.


=== B.5 Exceed available memory capacity ===

* Pattern description: Main memory usage exceeds the available or requested memory capacity
* Pattern detection: <code>max(mem_used) > alert(mem_used)</code>
* Applicable situations: Metrics have to be scaled for shared jobs. For shared jobs the (not yet available) mem_requested has to be used.
* Message to user: ''Your job almost exceeds the available memory capacity.''
* Pattern mitigation: Check application code to reduce memory allocation, use other system with more memory capacity.


=== B.6 Access temporary files on network filesystem ===


=== B.7 Small file IO on parallel filesystem ===


=== B.8 Excessisve CPU Load ===

* Pattern description: Job uses more threads / compute entities than allocated
* Pattern detection: <code>mean(cpu_load) > peak(cpu_load)*0.95 && threads > cores</code>
* Applicable situations: TODO
* Message to user: ''Your job can use more execution units.''
* Pattern mitigation: Request more cores/processes in your job script. ## C - Resource Contention ### C.1 Excessive File IO ### C.2 Excessive Network IO ### C.3 Multi-process GPU utilisation


== D - Load Balancing ==


=== D.1 Load imbalance ===

* Pattern description: unequal distribution of compute load
* Pattern detection: <code>max(coreavg(cpu_load)) - min(coreavg(cpu_load)) > threshold || max(nodeavg(cpu_load)) - min(nodeavg(cpu_load)) > threshold</code>
* Rule:

<pre>"parameter": ["balance_threshold"]
"rule_terms":[
{"max_core": "max(cpu_load, 'core')"}
{"min_core": "min(cpu_load, 'core')"}
{"core_balance": "(max_core - min_core) > balance_threshold"}]
"output": "core_balance"

"parameter": ["balance_threshold"]
"rule_terms":[
{"max_node": "max(cpu_load, 'node')"}
{"min_node": "min(cpu_load, 'node')"}
{"node_balance": "(max_node - min_node) > balance_threshold"}]
"output": "node_balance"</pre>

== E - Abuse ==


=== E.1 Cryptomining ===


=== E.2 Network Backdoors ===


=== E.3 Excess Batch System use ===


=== E.4 Short JOB (TuDA) ===

* Pattern description: The job is too short to qualify for using a batch system.
* Pattern detection: <code>Peak(Runtime) < T_Short</code>
* Applicable situations: *
* Message to user: TODO
* Pattern mitigation: TODO

@@ Line 161: / Line 161: @@
 === C.1 Excessive File IO ===
-<span id="c.2-excessive-network-io"></span>
+* Pattern description: Job is limited by file IO to a network file system
-=== C.2 Excessive Network IO ===
+* Pattern detection: <code>mean(FS[*].read_bw) &gt; critical(FS[*].read_bw)*0.95 &amp;&amp; mean(FS[*].write_bw) &gt; critical(FS[*].read_bw)</code>
-<span id="c.3-multu-process-gpu-utilisation"></span>
+<span id="c.1-excessive-file-metadata-operations"></span>
-=== C.3 Multi-process GPU utilisation ===
+=== C.1 Excessive File Metadata operations ===
 <span id="d---load-balancing"></span>

@@ Line 1: / Line 1: @@
-[[Category:HPC-Admin|Monitoring]]
+[[Category:HPC-Admin]]
 <span id="pathologic-job-classes"></span>
@@ Line 14: / Line 14: @@
 == A - Resource Allocation ==
-Job classes related to hardware usage utilization.
+Job classes related to faulty resource allocation and process placement.
 <span id="a.1-low-cpu-utilisation"></span>
@@ Line 92: / Line 92: @@
 <span id="b---resource-utilisation"></span>
 == B - Resource Utilisation ==
 <span id="b.1-idle-job"></span>
@@ Line 155: / Line 156: @@
 <span id="c---resource-contention"></span>
 == C - Resource Contention ==
 <span id="c.1-excessive-file-io"></span>
@@ Line 167: / Line 169: @@
 <span id="d---load-balancing"></span>
 == D - Load Balancing ==
 <span id="d.1-load-imbalance"></span>
@@ Line 191: / Line 194: @@
 <span id="e---abuse"></span>
 == E - Abuse ==
 <span id="e.1-cryptomining"></span>

← Older revision		Revision as of 05:22, 19 July 2024
Line 1:		Line 1:
−	[[Category:HPC-Admin]]	+	[[Category:HPC-Admin\|Monitoring]]

	<span id="pathologic-job-classes"></span>		<span id="pathologic-job-classes"></span>

@@ Line 6: / Line 6: @@
 This is a list of job classification rules that was created as part of the NHR PathoJobs project.
 The purpose of this list is to automatically detect jobs that require attention or provide an optimization opportunity.
 <span id="categories"></span>
@@ Line 30: / Line 31: @@
 ],
 &quot;output&quot;:&quot;lowload&quot;</pre>
 <span id="a.2-low-gpu-utilisation"></span>
 === A.2 Low GPU utilisation ===
@@ Line 46: / Line 48: @@
 ],
 &quot;output&quot;:&quot;lowload&quot;</pre>
 <span id="a.3-resource-oversubscription"></span>
 === A.3 Resource oversubscription ===
@@ Line 62: / Line 65: @@
 ],
 &quot;output&quot;:&quot;oversubscription&quot;</pre>
 <span id="a.4-short-job"></span>
 === A.4 Short job ===
@@ Line 76: / Line 80: @@
 ],
 &quot;output&quot;:&quot;short_job&quot;</pre>
 <span id="a.5-failing-chain-array-jobs"></span>
 === A.5 Failing Chain-/Array-Jobs ===
@@ Line 139: / Line 144: @@
 === B.7 Small file IO on parallel filesystem ===
-<span id="b.8-excessisve-cpu-load"></span>
+<span id="b.8-excessive-cpu-load"></span>
-=== B.8 Excessisve CPU Load ===
+=== B.8 Excessive CPU Load ===
 * Pattern description: Job uses more threads / compute entities than allocated
@@ Line 146: / Line 151: @@
 * Applicable situations: TODO
 * Message to user: ''Your job can use more execution units.''
-* Pattern mitigation: Request more cores/processes in your job script. ## C - Resource Contention ### C.1 Excessive File IO ### C.2 Excessive Network IO ### C.3 Multi-process GPU utilisation
+* Pattern mitigation: Request more cores/processes in your job script.
 <span id="d---load-balancing"></span>
@@ Line 171: / Line 188: @@
 {&quot;node_balance&quot;: &quot;(max_node - min_node) &gt; balance_threshold&quot;}]
 &quot;output&quot;: &quot;node_balance&quot;</pre>
 <span id="e---abuse"></span>
 == E - Abuse ==
@@ Line 184: / Line 202: @@
 <span id="e.4-short-job-tuda"></span>
-=== E.4 Short JOB (TuDA) ===
+=== E.4 Short Job ===
 * Pattern description: The job is too short to qualify for using a batch system.

@@ Line 4: / Line 4: @@
 = Pathologic Job Classes =
 <span id="categories"></span>
 = Categories =
@@ Line 9: / Line 12: @@
 <span id="a---resource-allocation"></span>
 == A - Resource Allocation ==
 <span id="a.1-low-cpu-utilisation"></span>

← Older revision		Revision as of 04:57, 19 July 2024
Line 1:		Line 1:
		+	[[Category:HPC-Admin]]
		+
	<span id="pathologic-job-classes"></span>		<span id="pathologic-job-classes"></span>
	= Pathologic Job Classes =		= Pathologic Job Classes =