-
Notifications
You must be signed in to change notification settings - Fork 344
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This adds docs on remote caching.
- Loading branch information
Showing
9 changed files
with
313 additions
and
38 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| The following is the [four kinds of documentation](https://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/) structure, which I'm not really sure would work, but worth giving it some try. | ||
|
|
||
| ### tutorial | ||
|
|
||
| > Tutorials are lessons given by a teacher that take the reader by the hand through series of steps to complete a task. | ||
| > Your project needs to show the user that they achieve something with it. The tutorials are oriented towards user's learning. You get them to do specific things, and allow them to learn later. In tutorials, you don't explain anything they need to know now. | ||
| "sbt by example" is our tutorial. | ||
|
|
||
| ### how-to guides | ||
|
|
||
| > How-to guides take the reader through the steps required to solve specific, real-world problem. They are, recipes, directions to achieve specific ends. They are goal-oriented. They are completely distinct from tutorials. A how-to guide is an answer to a question. In tutorial, you are the teacher: In how-to guide, the user knows what the problem is that they are trying to solve, and you've got the answer for them. How-to guide shouldn't explain everything, because the explanation get in the way of action. | ||
| ### concept guides | ||
|
|
||
| > Explanations are discussions that illuminate or clarify a particular topic. They are oriented towards understanding. They are chances to step back from the machinery, and take a wider view. It provides historical, sociological background discussion. | ||
| 'The Book of sbt' calls the Explanations "Concepts." | ||
|
|
||
| ### reference guides | ||
|
|
||
| > Reference guides are technical descriptions of machinery and its operation. They have only one job, which is to describe. They are information-oriented. It's not a place for explaining basic concept. | ||
| 'The Book of sbt' calls this Reference. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| sbt 2.0 changes (draft) | ||
| ======================= | ||
|
|
||
| ```admonish warning | ||
| This is a draft documentation of sbt 2.x that is yet to be released. | ||
| While the general concept translates to sbt 1.x, | ||
| details of both 2.x and this doc are subject to change. | ||
| ``` | ||
|
|
||
| - sbt 2.x uses Scala 3.x for build definitions and plugins (Both sbt 1.x and 2.x are capable of building Scala 2.x and 3.x) by [@eed3si9n][@eed3si9n], [@adpi2][@adpi2], and others. | ||
| - Bare settings are added to all subprojects, as opposed to just the root subproject, and thus replacing the role that `ThisBuild` has played. [#6746][6746] by [@eed3si9n][@eed3si9n] | ||
| - sbt 2.x adds `platform` setting so `ModuleID`'s `%%` operator can cross build on JVM as well as JS and Native, as opposed to `%%%` operator that was created in a plugin to workaround this issue. [#6746][6746] by [@eed3si9n][@eed3si9n] | ||
| - Local/remote hybrid cache system. sbt 2.x implements cached task, which can automatically cache the task results to local disk and Bazel-compatible remote cache. [#7464][7464] / [#7525][7525] by [@eed3si9n][@eed3si9n] | ||
|
|
||
| Previously on sbt | ||
| ----------------- | ||
|
|
||
| See also: | ||
|
|
||
| - [sbt 1.0 changes](https://www.scala-sbt.org/1.x/docs/sbt-1.0-Release-Notes.html) | ||
|
|
||
| [6746]: https://github.com/sbt/sbt/pull/6746 | ||
| [7464]: https://github.com/sbt/sbt/pull/7464 | ||
| [7525]: https://github.com/sbt/sbt/pull/7525 | ||
| [@eed3si9n]: https://github.com/eed3si9n | ||
| [@adpi2]: https://github.com/adpi2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| Caching | ||
| ======= | ||
|
|
||
| sbt 2.0 introduces hybrid local/remote cache system, which can cache the task results to local disk and Bazel-compatible remote cache. Throughout sbt releases it has implemented various caches, like `update` cache, incremental compilation, but sbt 2.x's cache is a significant step change for a few reasons: | ||
|
|
||
| 1. **Automatic**. sbt 2.x cache automates the caching by embedding itself into the task macro unlike sbt 1.x wherein the plugin author called the cache functions manually in the task implementation. | ||
| 2. **Machine-wide**. sbt 2.x disk cache is shared among all builds on a machine. | ||
| 3. **Remote-ready**. In sbt 2.x, the cache storage is configured separately such that all cacheable tasks are automatically remote-cache-ready. | ||
|
|
||
| Basics of caching | ||
| ----------------- | ||
|
|
||
| The basic idea is treat as if the build process is a pure function that takes input `(A1, A2, A3, ...)` and return some outputs `(R1, List(O1, O2, O3, ...))`. For example, we can take a list of source files, Scala version, and produce a `*.jar` file at the end. If the assumption holds, then for the same inputs, we can memorize the ouput JAR for everyone. We are interested in this technique because using the memorized output JAR would be faster than performing the actual task like Scala compilation etc. | ||
|
|
||
| ### Hermetic build | ||
|
|
||
| As a mental model of the _build as a pure function_, build engineers sometimes use the term _hermetic build_, which is a build that takes place in a shipping container in a dessert with no clocks or the Internet. If we can produce a JAR file from that state, then the JAR file should be safe to be shared by any machine. Why did I mention the clock? It's because a JAR file could capture the timestamp, and thus produce slightly different JARs each time. To avoid this, hermetic build tools overwrite the timestamp to a fixed date 2010-01-01 regardless of when the build took place. | ||
|
|
||
| A build that ends up capturing ephemeral inputs, are said to _break the hermeticity_ or _non-hermetic_. Another common way the hermeticity is broken is capturing absolute paths as either input or output. Sometimes the path gets embedded into the JAR via a macro, you might not know until you inspect the bytecode. | ||
|
|
||
| Automatic caching | ||
| ----------------- | ||
|
|
||
| Here's a demonstration of the automatic caching: | ||
|
|
||
| ```scala | ||
| import sbt.util.CacheImplicits.given | ||
|
|
||
| val someKey = taskKey[String]("something") | ||
|
|
||
| someKey := (Def.cachedTask { | ||
| name.value + version.value + "!" | ||
| }).value | ||
| ``` | ||
|
|
||
| The task will be automatically cached based on the two settings `name` and `version`. The first time we run the task it will be executed onsite, but the second time onward, it will use the disk cache: | ||
|
|
||
| ``` | ||
| sbt:demo> show someKey | ||
| [info] demo0.1.0-SNAPSHOT! | ||
| [success] elapsed time: 0 s, cache 0%, 1 onsite task | ||
| sbt:demo> show someKey | ||
| [info] demo0.1.0-SNAPSHOT! | ||
| [success] elapsed time: 0 s, cache 100%, 1 disk cache hit | ||
| ``` | ||
|
|
||
| Remote caching | ||
| -------------- | ||
|
|
||
| You can optionally extend the build to use remote cache in addition to the local disk cache. Remote caching could improve build performance by allowing multiple machines to share build artifacts and outputs. | ||
|
|
||
| Imagine you have a dozen people in your project or a company. Each morning, you will `git pull` the changes the dozen people made, and you need to build their code. If you have a successful project, the code size will only get bigger over time, and the % of the time you spend building someone else's in your day increases. This becomes the limiting factor of your team size and code size. Remote caching reverses this tide by CI systems hydrate the cache and you can download the artifacts and task outputs. | ||
|
|
||
| sbt 2.x implements Bazel-compatible gRPC interface, which works with number of backend both open source and commercial. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| html { | ||
| font-family: "Lora"; | ||
| } | ||
|
|
||
| :root { | ||
| font-size: 80%; | ||
| } | ||
|
|
||
| pre { | ||
| border: solid 1px lightgray; | ||
| line-height: 1.8rem; | ||
| font-weight: 500; | ||
| } | ||
|
|
||
| code { | ||
| font-family: 'Inconsolata', monospace !important; | ||
| } | ||
|
|
||
| p code { | ||
| font-weight: 600; | ||
| } | ||
|
|
||
| li code { | ||
| font-weight: 600; | ||
| } | ||
|
|
||
| h1 { | ||
| font-family: 'Linux Biolinum'; | ||
| } | ||
|
|
||
| h1.menu-title { | ||
| font-family: "Lora"; | ||
| font-size: 1.5rem; | ||
| } | ||
|
|
||
| h2 { | ||
| font-family: 'Linux Biolinum'; | ||
| } | ||
|
|
||
| h3 { | ||
| font-family: 'Linux Biolinum'; | ||
| } |
4 changes: 2 additions & 2 deletions
4
src/reference/guide/running.md → src/reference/guide/basic-tasks.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.