-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
HermeticFetchContent v1.0.X : Controlling build parallelism level
CHANGELOG - Improved the way that build parallelism settings are forwarded through the build graph so that the correct outcome is achieved depending on setting CMAKE_BUILD_PARALLEL_LEVEL via environment Change-Id: I33ae3760b2a0ab00048f07ac2a7c14d3bbc460b7
- Loading branch information
Showing
12 changed files
with
292 additions
and
6 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| .. title:: HermeticFetchContent / Recipes / Controlling Build Parallelism | ||
|
|
||
| Controlling Build Parallelism | ||
| ############################# | ||
|
|
||
| Because of the complex nature of the build graph in Hermetic FetchContent you may want to adjust the build | ||
| parallelity in some situations like CI or shared build nodes. | ||
|
|
||
| Hermetic FetchContent does use a combination of the following parameters to set the build parallelism: | ||
|
|
||
| - the number of CPU cores available on your system | ||
| - the value of the environment variable ``CMAKE_BUILD_PARALLEL_LEVEL`` | ||
| - the value supplied to ``cmake --build`` via the ``-j <jobs>`` / ``--parallel <jobs>`` argument | ||
|
|
||
| Generally not that dependencies that are made available at **build time** will have their own, separate | ||
| allocation of CPU ressources as these dependent builds are generally unaware of the ressource allocations | ||
| of peer and parent projects. This means that you may end up with sever over-loading of the system depending | ||
| on the build graph of your project and available ressources. | ||
|
|
||
| When no setting is provided, Hermetic FetchContent will defer to the invoked build system (``ninja`` or ``make`` | ||
| for example) to select the build parallelism (which typically fall back to the number of CPU cores). | ||
|
|
||
| The value provided via ``CMAKE_BUILD_PARALLEL_LEVEL`` is taken into account by CMake throughout the build graph, | ||
| meaning that the set value is used for each of the builds that make up the graph of your project. | ||
|
|
||
| The value passed to CMake via the ``-j <jobs>`` / ``--parallel <jobs>`` argument are taken into account **only | ||
| for the top level build graph**. This detail enables you to control the build parallelism for the project | ||
| and the dependency builds independently: | ||
|
|
||
| .. code-block:: bash | ||
| # $PWD = project root | ||
| export CMAKE_BUILD_PARALLEL_LEVEL=16 | ||
| cmake -S . -B build/release/ -DCMAKE_BUILD_TYPE=Release ... | ||
| export CMAKE_BUILD_PARALLEL_LEVEL=2 | ||
| cmake --build . -j10 | ||
| The example above will have the following behavior | ||
|
|
||
| - all dependencies **made available at configure time** will use 16 CPU cores (which is not an issue as they are run in sequence) | ||
| - during the build | ||
| - dependencies **made available at build time** will use 2 CPU cores | ||
| - the top level build graph (your project) will use 10 CPU cores | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,173 @@ | ||
|
|
||
| <!DOCTYPE html> | ||
|
|
||
| <html lang="en"> | ||
| <head> | ||
| <meta charset="utf-8" /> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" /> | ||
|
|
||
| <title>HermeticFetchContent / Recipes / Controlling Build Parallelism — hermetic-fetchcontent Documentation</title> | ||
|
|
||
| <link rel="stylesheet" type="text/css" href="../_static/pygments.css" /> | ||
| <link rel="stylesheet" type="text/css" href="../_static/cmake.css" /> | ||
|
|
||
| <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script> | ||
| <script src="../_static/doctools.js"></script> | ||
| <script src="../_static/sphinx_highlight.js"></script> | ||
|
|
||
| <link rel="icon" href="../_static/favicon.ico"/> | ||
| <link rel="index" title="Index" href="../genindex.html" /> | ||
| <link rel="search" title="Search" href="../search.html" /> | ||
| <link rel="prev" title="How to define build Environments" href="HowToDefineBuildEnvironments.html" /> | ||
| <link rel="search" type="application/opensearchdescription+xml" | ||
| title="Search within CMake Documentation of Latest Version" | ||
| href="../../latest-opensearch.xml"/> | ||
| <script type="text/javascript" src="../../version_switch.js"></script> | ||
|
|
||
|
|
||
| </head><body> | ||
| <div class="related" role="navigation" aria-label="related navigation"> | ||
| <h3>Navigation</h3> | ||
| <ul> | ||
| <li class="right" style="margin-right: 10px"> | ||
| <a href="../genindex.html" title="General Index" | ||
| accesskey="I">index</a></li> | ||
| <li class="right" > | ||
| <a href="HowToDefineBuildEnvironments.html" title="How to define build Environments" | ||
| accesskey="P">previous</a> |</li> | ||
| <li> | ||
| <img src="../_static/tipi-logo.png" alt="" | ||
| style="vertical-align: middle; margin-top: -2px" /> | ||
| </li> | ||
| <li> | ||
| <a href="https://tipi.build/">tipi</a> » | ||
| </li> | ||
| <li> | ||
| <span class="version_switch"></span> | ||
| <a href="../index.html">Documentation</a> » | ||
| </li> | ||
|
|
||
| <li class="nav-item nav-item-this"><a href="">HermeticFetchContent / Recipes / Controlling Build Parallelism</a></li> | ||
| </ul> | ||
| </div> | ||
|
|
||
| <div class="document"> | ||
| <div class="documentwrapper"> | ||
| <div class="bodywrapper"> | ||
| <div class="body" role="main"> | ||
|
|
||
| <section id="controlling-build-parallelism"> | ||
| <h1>Controlling Build Parallelism<a class="headerlink" href="#controlling-build-parallelism" title="Permalink to this heading">¶</a></h1> | ||
| <p>Because of the complex nature of the build graph in Hermetic FetchContent you may want to adjust the build | ||
| parallelity in some situations like CI or shared build nodes.</p> | ||
| <p>Hermetic FetchContent does use a combination of the following parameters to set the build parallelism:</p> | ||
| <blockquote> | ||
| <div><ul class="simple"> | ||
| <li><p>the number of CPU cores available on your system</p></li> | ||
| <li><p>the value of the environment variable <code class="docutils literal notranslate"><span class="pre">CMAKE_BUILD_PARALLEL_LEVEL</span></code></p></li> | ||
| <li><p>the value supplied to <code class="docutils literal notranslate"><span class="pre">cmake</span> <span class="pre">--build</span></code> via the <code class="docutils literal notranslate"><span class="pre">-j</span> <span class="pre"><jobs></span></code> / <code class="docutils literal notranslate"><span class="pre">--parallel</span> <span class="pre"><jobs></span></code> argument</p></li> | ||
| </ul> | ||
| </div></blockquote> | ||
| <p>Generally not that dependencies that are made available at <strong>build time</strong> will have their own, separate | ||
| allocation of CPU ressources as these dependent builds are generally unaware of the ressource allocations | ||
| of peer and parent projects. This means that you may end up with sever over-loading of the system depending | ||
| on the build graph of your project and available ressources.</p> | ||
| <p>When no setting is provided, Hermetic FetchContent will defer to the invoked build system (<code class="docutils literal notranslate"><span class="pre">ninja</span></code> or <code class="docutils literal notranslate"><span class="pre">make</span></code> | ||
| for example) to select the build parallelism (which typically fall back to the number of CPU cores).</p> | ||
| <p>The value provided via <code class="docutils literal notranslate"><span class="pre">CMAKE_BUILD_PARALLEL_LEVEL</span></code> is taken into account by CMake throughout the build graph, | ||
| meaning that the set value is used for each of the builds that make up the graph of your project.</p> | ||
| <p>The value passed to CMake via the <code class="docutils literal notranslate"><span class="pre">-j</span> <span class="pre"><jobs></span></code> / <code class="docutils literal notranslate"><span class="pre">--parallel</span> <span class="pre"><jobs></span></code> argument are taken into account <strong>only | ||
| for the top level build graph</strong>. This detail enables you to control the build parallelism for the project | ||
| and the dependency builds independently:</p> | ||
| <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># $PWD = project root</span> | ||
| <span class="nb">export</span><span class="w"> </span><span class="nv">CMAKE_BUILD_PARALLEL_LEVEL</span><span class="o">=</span><span class="m">16</span> | ||
| cmake<span class="w"> </span>-S<span class="w"> </span>.<span class="w"> </span>-B<span class="w"> </span>build/release/<span class="w"> </span>-DCMAKE_BUILD_TYPE<span class="o">=</span>Release<span class="w"> </span>... | ||
| <span class="nb">export</span><span class="w"> </span><span class="nv">CMAKE_BUILD_PARALLEL_LEVEL</span><span class="o">=</span><span class="m">2</span> | ||
| cmake<span class="w"> </span>--build<span class="w"> </span>.<span class="w"> </span>-j10 | ||
| </pre></div> | ||
| </div> | ||
| <p>The example above will have the following behavior</p> | ||
| <blockquote> | ||
| <div><ul class="simple"> | ||
| <li><p>all dependencies <strong>made available at configure time</strong> will use 16 CPU cores (which is not an issue as they are run in sequence)</p></li> | ||
| <li><p>during the build | ||
| - dependencies <strong>made available at build time</strong> will use 2 CPU cores | ||
| - the top level build graph (your project) will use 10 CPU cores</p></li> | ||
| </ul> | ||
| </div></blockquote> | ||
| </section> | ||
|
|
||
|
|
||
| <div class="clearer"></div> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> | ||
| <div class="sphinxsidebarwrapper"> | ||
| <div> | ||
| <h4>Previous topic</h4> | ||
| <p class="topless"><a href="HowToDefineBuildEnvironments.html" | ||
| title="previous chapter">How to define build Environments</a></p> | ||
| </div> | ||
| <div role="note" aria-label="source link"> | ||
| <h3>This Page</h3> | ||
| <ul class="this-page-menu"> | ||
| <li><a href="../_sources/recipes/ControllingBuildParallelism.rst.txt" | ||
| rel="nofollow">Show Source</a></li> | ||
| </ul> | ||
| </div> | ||
| <div id="searchbox" style="display: none" role="search"> | ||
| <h3 id="searchlabel">Quick search</h3> | ||
| <div class="searchformwrapper"> | ||
| <form class="search" action="../search.html" method="get"> | ||
| <input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/> | ||
| <input type="submit" value="Go" /> | ||
| </form> | ||
| </div> | ||
| </div> | ||
| <script>document.getElementById('searchbox').style.display = "block"</script> | ||
| </div> | ||
| </div> | ||
| <div class="clearer"></div> | ||
| </div> | ||
| <div class="related" role="navigation" aria-label="related navigation"> | ||
| <h3>Navigation</h3> | ||
| <ul> | ||
| <li class="right" style="margin-right: 10px"> | ||
| <a href="../genindex.html" title="General Index" | ||
| >index</a></li> | ||
| <li class="right" > | ||
| <a href="HowToDefineBuildEnvironments.html" title="How to define build Environments" | ||
| >previous</a> |</li> | ||
| <li> | ||
| <img src="../_static/tipi-logo.png" alt="" | ||
| style="vertical-align: middle; margin-top: -2px" /> | ||
| </li> | ||
| <li> | ||
| <a href="https://tipi.build/">tipi</a> » | ||
| </li> | ||
| <li> | ||
| <span class="version_switch"></span> | ||
| <a href="../index.html">Documentation</a> » | ||
| </li> | ||
|
|
||
| <li class="nav-item nav-item-this"><a href="">HermeticFetchContent / Recipes / Controlling Build Parallelism</a></li> | ||
| </ul> | ||
| </div> | ||
|
|
||
| <div class="footer" role="contentinfo"> | ||
| © Copyright tipi technologies Ltd.. | ||
| Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 6.2.1. | ||
| </div> | ||
| <script type="text/javascript"> | ||
| var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); | ||
| document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); | ||
| </script> | ||
| <script type="text/javascript"> | ||
| try { | ||
| var pageTracker = _gat._getTracker("UA-6042509-4"); | ||
| pageTracker._trackPageview(); | ||
| } catch(err) {} | ||
| </script> | ||
| </body> | ||
| </html> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| .. title:: HermeticFetchContent / Recipes / Controlling Build Parallelism | ||
|
|
||
| Controlling Build Parallelism | ||
| ############################# | ||
|
|
||
| Because of the complex nature of the build graph in Hermetic FetchContent you may want to adjust the build | ||
| parallelity in some situations like CI or shared build nodes. | ||
|
|
||
| Hermetic FetchContent does use a combination of the following parameters to set the build parallelism: | ||
|
|
||
| - the number of CPU cores available on your system | ||
| - the value of the environment variable ``CMAKE_BUILD_PARALLEL_LEVEL`` | ||
| - the value supplied to ``cmake --build`` via the ``-j <jobs>`` / ``--parallel <jobs>`` argument | ||
|
|
||
| Generally not that dependencies that are made available at **build time** will have their own, separate | ||
| allocation of CPU ressources as these dependent builds are generally unaware of the ressource allocations | ||
| of peer and parent projects. This means that you may end up with sever over-loading of the system depending | ||
| on the build graph of your project and available ressources. | ||
|
|
||
| When no setting is provided, Hermetic FetchContent will defer to the invoked build system (``ninja`` or ``make`` | ||
| for example) to select the build parallelism (which typically fall back to the number of CPU cores). | ||
|
|
||
| The value provided via ``CMAKE_BUILD_PARALLEL_LEVEL`` is taken into account by CMake throughout the build graph, | ||
| meaning that the set value is used for each of the builds that make up the graph of your project. | ||
|
|
||
| The value passed to CMake via the ``-j <jobs>`` / ``--parallel <jobs>`` argument are taken into account **only | ||
| for the top level build graph**. This detail enables you to control the build parallelism for the project | ||
| and the dependency builds independently: | ||
|
|
||
| .. code-block:: bash | ||
| # $PWD = project root | ||
| export CMAKE_BUILD_PARALLEL_LEVEL=16 | ||
| cmake -S . -B build/release/ -DCMAKE_BUILD_TYPE=Release ... | ||
| export CMAKE_BUILD_PARALLEL_LEVEL=2 | ||
| cmake --build . -j10 | ||
| The example above will have the following behavior | ||
|
|
||
| - all dependencies **made available at configure time** will use 16 CPU cores (which is not an issue as they are run in sequence) | ||
| - during the build | ||
| - dependencies **made available at build time** will use 2 CPU cores | ||
| - the top level build graph (your project) will use 10 CPU cores | ||
|
|