https://hpc-wiki.info/hpc/index.php?action=history&feed=atom&title=Admin_Guide_Lustre_Tuning
Admin Guide Lustre Tuning - Revision history
2026-06-02T02:43:25Z
Revision history for this page on the wiki
MediaWiki 1.35.9
https://hpc-wiki.info/hpc/index.php?title=Admin_Guide_Lustre_Tuning&diff=4504&oldid=prev
Mukund-pondkule-6a11@uni-paderborn.de at 18:11, 9 December 2020
2020-12-09T18:11:38Z
<p></p>
<table class="diff diff-contentalign-left diff-editfont-monospace" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 18:11, 9 December 2020</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td>
<td colspan="2" class="diff-lineno">Line 1:</td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC-Admin|Lustre Tuning]]</div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC-Admin|Lustre Tuning]]<ins class="diffchange diffchange-inline"><nowiki /></ins></div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC.NRW-Best-Practices|Lustre Tuning]]</div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC.NRW-Best-Practices|Lustre Tuning]]<ins class="diffchange diffchange-inline"><nowiki /></ins></div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>[[Category:File_Systems]]</div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>[[Category:File_Systems]]<ins class="diffchange diffchange-inline"><nowiki /></ins></div></td></tr>
<tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">{{DISPLAYTITLE:Lustre Tuning (Admin Guide)}}<nowiki /></ins></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>This article describes tuning options for the Lustre file system with the example of the Noctua cluster of PC².</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>This article describes tuning options for the Lustre file system with the example of the Noctua cluster of PC².</div></td></tr>
<!-- diff cache key hpc_wiki:diff::1.12:old-3908:rev-4504 -->
</table>
Mukund-pondkule-6a11@uni-paderborn.de
https://hpc-wiki.info/hpc/index.php?title=Admin_Guide_Lustre_Tuning&diff=3908&oldid=prev
Robert-schade-e757@uni-paderborn.de at 15:44, 2 November 2020
2020-11-02T15:44:06Z
<p></p>
<table class="diff diff-contentalign-left diff-editfont-monospace" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 15:44, 2 November 2020</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td>
<td colspan="2" class="diff-lineno">Line 1:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC-Admin|Lustre Tuning]]</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC-Admin|Lustre Tuning]]</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC.NRW-Best-Practices|Lustre Tuning]]</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC.NRW-Best-Practices|Lustre Tuning]]</div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>[[Category:File_Systems]</div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>[[Category:File_Systems<ins class="diffchange diffchange-inline">]</ins>]</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>This article describes tuning options for the Lustre file system with the example of the Noctua cluster of PC².</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>This article describes tuning options for the Lustre file system with the example of the Noctua cluster of PC².</div></td></tr>
<!-- diff cache key hpc_wiki:diff::1.12:old-3907:rev-3908 -->
</table>
Robert-schade-e757@uni-paderborn.de
https://hpc-wiki.info/hpc/index.php?title=Admin_Guide_Lustre_Tuning&diff=3907&oldid=prev
Robert-schade-e757@uni-paderborn.de at 15:43, 2 November 2020
2020-11-02T15:43:56Z
<p></p>
<table class="diff diff-contentalign-left diff-editfont-monospace" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 15:43, 2 November 2020</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td>
<td colspan="2" class="diff-lineno">Line 1:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC-Admin|Lustre Tuning]]</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC-Admin|Lustre Tuning]]</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC.NRW-Best-Practices|Lustre Tuning]]</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC.NRW-Best-Practices|Lustre Tuning]]</div></td></tr>
<tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">[[Category:File_Systems]</ins></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>This article describes tuning options for the Lustre file system with the example of the Noctua cluster of PC².</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>This article describes tuning options for the Lustre file system with the example of the Noctua cluster of PC².</div></td></tr>
<!-- diff cache key hpc_wiki:diff::1.12:old-3860:rev-3907 -->
</table>
Robert-schade-e757@uni-paderborn.de
https://hpc-wiki.info/hpc/index.php?title=Admin_Guide_Lustre_Tuning&diff=3860&oldid=prev
Robert-schade-e757@uni-paderborn.de at 14:43, 2 November 2020
2020-11-02T14:43:51Z
<p></p>
<table class="diff diff-contentalign-left diff-editfont-monospace" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 14:43, 2 November 2020</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td>
<td colspan="2" class="diff-lineno">Line 1:</td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC-Admin]]</div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC-Admin<ins class="diffchange diffchange-inline">|Lustre Tuning</ins>]]</div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC.NRW-Best-Practices]]</div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>[[Category:HPC.NRW-Best-Practices<ins class="diffchange diffchange-inline">|Lustre Tuning</ins>]]</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>This article describes tuning options for the Lustre file system with the example of the Noctua cluster of PC².</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>This article describes tuning options for the Lustre file system with the example of the Noctua cluster of PC².</div></td></tr>
<!-- diff cache key hpc_wiki:diff::1.12:old-3787:rev-3860 -->
</table>
Robert-schade-e757@uni-paderborn.de
https://hpc-wiki.info/hpc/index.php?title=Admin_Guide_Lustre_Tuning&diff=3787&oldid=prev
Robert-schade-e757@uni-paderborn.de: Created page with "Category:HPC-Admin Category:HPC.NRW-Best-Practices This article describes tuning options for the Lustre file system with the example of the Noctua cluster of PC². =..."
2020-10-31T18:06:39Z
<p>Created page with "<a href="/hpc/Category:HPC-Admin" title="Category:HPC-Admin">Category:HPC-Admin</a> <a href="/hpc/Category:HPC.NRW-Best-Practices" title="Category:HPC.NRW-Best-Practices">Category:HPC.NRW-Best-Practices</a> This article describes tuning options for the Lustre file system with the example of the Noctua cluster of PC². =..."</p>
<p><b>New page</b></p><div>[[Category:HPC-Admin]]<br />
[[Category:HPC.NRW-Best-Practices]]<br />
<br />
This article describes tuning options for the Lustre file system with the example of the Noctua cluster of PC².<br />
<br />
=== Lustre File System ===<br />
<br />
The high-performance parallel file system of Noctua is a Lustre File System.<br />
<br />
This Lustre file system has three major functional units<br />
* one '''Metadata Server''' (MDS) with two '''metadata targets''' (MDTs)<br />
** stores namespace metadata, such as filenames, directories, Access permissions, and file layout<br />
** stores small files on SSDs to accelerate data access<br />
* two '''Object Storage Servers''' (OSS), each with two '''Object Storage Targests''' (OST)<br />
** each OST manages a single local disc filesystem<br />
* Clients (the Noctua nodes) that access the data (read/write)<br />
** Lustre presents all Clients with a unified Namespace for all the files and data in the file System<br />
** allows concurrent and coherent read and write access to the files in the filesystem<br />
<br />
Key points<br />
* Lustre achieves high Performance through parallelism<br />
** best Performance from multiple Clients writing to multiple OSTs<br />
* Lustre is designed to achieve high bandwidth to/from a small number of files<br />
** used as a scratch file System<br />
** good match for scientific datasets and/or checkpoint data<br />
* Lustre is not designed to handle large numbers of small files<br />
** potential bottle necks at the MDS when files are opened<br />
** data will not be spread over multiple OSTs<br />
** not a good choice for program compilation<br />
<br />
A powerful Lustre utility is ''lfs''. The tool has a built-in help system.<br />
<pre><br />
> lfs help<br />
Available commands are:<br />
setstripe<br />
getstripe<br />
setdirstripe<br />
getdirstripe<br />
mkdir<br />
rm_entry<br />
pool_list<br />
find<br />
...<br />
</pre><br />
<br />
Metadata operations are expensive<br />
* the stat operations return information on file ownerships, permissions, size, update times etc.<br />
* to obtain the file size requires a lookup on the MDS '''and''' an enquiry for file size on '''each''' OST owning a stripe<br />
therefore <br />
* '''avoid''' ''ls -l'' (like color ''ls'')<br />
* '''avoid''' file completion in shells<br />
* open and fail instead of stat/INQUIRE<br />
* '''don't''' stripe small files, Lustre check every OST that might own a part of the file<br />
* open a file read-only if that is what you will do<br />
* use tools optimzied for (aware of) Lustre<br />
** lfs find, lfs df, ...<br />
** stripe-aware tar (''star'')<br />
* avoid to read the same files on many processes, better to read on one process and use MPI communication to move data to other processes<br />
* avoid large directories, organize directory structure by processes/clients<br />
* ''open()'' and ''seek()'' if you know the size, otherwise try to organize applications to write from only one process<br />
* use the Lustre API in your application (see ''man lustreapi'')<br />
<br />
<br />
More Tuning tips for Lustre are in [[Noctua-Tuning#Lustre File System | Noctua Tuning]].</div>
Robert-schade-e757@uni-paderborn.de