Iterative crawl

[davidmorgan]

The current build_runner can't actually run the benchmarks I want due to getting a stack overflow, this is a minimal fix: copy the crawlAsync algorithm and test out of package:graph (which we own), make it iterative. The algorithm is significantly changed, the tests are the same.

Per suggestion on another PR I benchmarked using List instead of Queue, there was no consistent difference; so this uses Queue to make the processing order closer to the previous version.

Here is the same benchmark pasted in #3841 but now without failures due to stack overflows :)

dart run _benchmark --generator=json_serializable --build-repo-path=$PWD/.. benchmark
json_serializable
shape,libraries,clean/ms,no changes/ms,incremental/ms
loop,1,21745,4099,5313
loop,100,23777,4064,7543
loop,250,29418,4356,13627
loop,500,43098,5677,32238
loop,750,66256,8050,58112
loop,1000,110803,9825,102674
forwards,1,21460,4230,5508
forwards,100,24546,4102,7274
forwards,250,27010,4248,11152
forwards,500,35351,4809,21441
forwards,750,47161,6263,38171
forwards,1000,66103,8188,63542
backwards,1,22323,3930,5465
backwards,100,23352,4199,7667
backwards,250,27895,4307,11313
backwards,500,38363,5085,21476
backwards,750,53312,6178,42027
backwards,1000,66276,7393,62432

[github-actions]

Package publishing

Package	Version	Status	Publish tag (post-merge)
package:build	2.4.2	already published at pub.dev
package:build_config	1.1.2	already published at pub.dev
package:build_daemon	4.0.4	already published at pub.dev
package:build_modules	5.0.11	already published at pub.dev
package:build_resolvers	2.4.4	(error) pubspec version (2.4.4) and changelog (2.4.5-wip) don't agree
package:build_runner	2.4.15	already published at pub.dev
package:build_runner_core	8.0.1-dev	ready to publish	build_runner_core-v8.0.1-dev
package:build_test	2.2.4-wip	WIP (no publish necessary)
package:build_web_compilers	4.1.1	already published at pub.dev
package:scratch_space	1.0.3-wip	WIP (no publish necessary)

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

[github-actions]

PR Health

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

[jakemac53]

Fwiw it would be worth upstreaming the change to an iterative algorithm to package:graphs

[davidmorgan]

That didn't quite work; the new algorithm crawled each dep one by one, but the test "A hidden generated file does not poison resolving has a situation where one read will just block until others are complete; it relies on the crawl happening "in parallel".

Since the goal here is to match the existing algorithm, updated to also crawl "in parallel", i.e. using Future.wait.

Benchmark results look similar

json_serializable
shape,libraries,clean/ms,no changes/ms,incremental/ms
loop,1,22048,4005,5217
loop,100,24202,4117,7740
loop,250,29326,4523,13480
loop,500,43841,5451,33941
loop,750,64229,8037,61589
loop,1000,103161,11580,124393
forwards,1,23019,4010,5536
forwards,100,24074,4285,7375
forwards,250,27019,4344,10638
forwards,500,36553,4890,22636
forwards,750,47664,5718,37230
forwards,1000,65236,8210,59194
backwards,1,21972,4059,5531
backwards,100,24060,4161,7555
backwards,250,27380,4100,10820
backwards,500,37545,4952,21842
backwards,750,49847,6421,37447
backwards,1000,74118,7340,62329