syz-cluster: initial code

[a-nogikh]

The basic code of a k8s-based cluster that:

• Aggregates new LKML patch series.
• Determines the kernel trees to apply them to.
• Builds the basic and the patched kernel.
• Displays the results on a web dashboard.

This is a very rudimentary version with a lot of TODOs that provides a skeleton for further work.

The project makes use of Argo workflows and Spanner DB.
Bootstrap is used for the web interface.

Overall structure:

• syz-cluster/dashboard: a web dashboard listing patch series and their test results.
• syz-cluster/series-tracker: polls Lore archives and submits the new patch series to the DB.
• syz-cluster/controller: schedules workflows and provides API for them.
• syz-cluster/kernel-disk: a cron job that keeps a kernel checkout up to date.
• syz-cluster/workflow/*: workflow steps.

For the DB structure see syz-cluster/pkg/db/migrations/*.

---

TODO (for this PR):

• Go through the installation section, hopefully get rid of minio use.
• Figure out why go.mod bumped to 1.23.1 That is go: github.com/argoproj/argo-workflows/v3@v3.6.2 requires go >= 1.23.1 (running go 1.22.7)
• GAE still does not support a Go 1.23 runtime. So should we use an older Argo workflow version? v3.5.13 still only requires Go 1.21.
• go: module k8s.io/api@v0.32.0 requires go >= 1.23.0; switching to go1.23.4. v0.31.4 requires Go 1.22.
• Figure out how to run syz-cluster tests on GitHub CI (we'll need at least some local Spanner emulator binary).
• Move from minio to https://github.com/fsouza/fake-gcs-server? We might need to use https://argo-workflows.readthedocs.io/en/latest/environment-variables/ to pass STORAGE_EMULATOR_HOST to the argo server.

Questions/thoughts out loud:
1. What to do with the vendor folder
Our vendor/ folder is quite big, and this PR adds even more modules on top of that. Do we still want to keep that code in our repostory? Is it possible to only keep the modules needed by other components but not syz-cluster (since I do go mod download in the Docker containers anyway).
Filed #5645

2. Some pkg/ packages are too eager to depend on prog/ and sys/
It has made some Dockerfiles more complicated that they could have been.
Should we strive to remove these dependencies?

[dvyukov]

The first patch is not questionable. Let's review/merge it separately.

[a-nogikh]

FTR:

• We could add a standalone Spanner emulator binary to our syz-env image like described here: https://github.com/GoogleCloudPlatform/cloud-spanner-emulator/blob/master/README.md#via-pre-built-linux-binaries
• parallelism: 1 and failFast: true could be configured per workflow template: https://argo-workflows.readthedocs.io/en/latest/fields/#workflowspec. Thus the iterateItems step could be made its own template, but there's still a question on how to distinguish between "try the next kernel revision" and "I've managed to build this one. Stop iterating.".
• Lifecycle hooks seem to be the recommended way of implementing onExit callbacks: https://argo-workflows.readthedocs.io/en/latest/lifecyclehook/
• MinIO seems be unavoidable for local argo workflow runs, there are no simpler artifact storage options.

[tarasmadan]

We could add a standalone Spanner emulator binary

Please give more context. Do you need it for the local development or testing?
It should be Ok to use it as a dev environment but I'm hesitating to add it as a CI dependency.
Staging environment may be more reliable way for check your system. And spanner mocks looks like a better (more stable, predictable, faster) alternative for CI testing.

[a-nogikh]

We could add a standalone Spanner emulator binary

Please give more context. Do you need it for the local development or testing? It should be Ok to use it as a dev environment but I'm hesitating to add it as a CI dependency. Staging environment may be more reliable way for check your system. And spanner mocks looks like a better (more stable, predictable, faster) alternative for CI testing.

For dev environment, it already exists - there's make run-go-tests in syz-cluster that runs the tests in the k8s environment. I meant running those tests on the GitHub CI as well (ideally by getting rid of make run-go-tests as well).

I've just added a syz-cluster/pkg/db: add support for running under syz-env commit that runs the Spanner emulator binary for our Go tests. Seems to be quite simple, actually.

For some other external dependencies (e.g. Argo workflows or the blob storage), there are already mocks in the code, but for the DB I fear that there will just be too much boilerplate code. If we can just run a db emulator for the unit tests, why not do it?

Doing so lets us test not just the higher level logic, but also all SQL commands that are used in the implementations.

[tarasmadan]

Unit testing allows to test some component in the isolated environment where you can simulate any kind of the external service behavior (inclulding failure). Mocks give you the required flexibility and are the faster way to emulate external dependencies.
Using the emulator for unit testing we make UT slower and test some interaction... It is not the unit testing by definition... I think it worth discussion...

Emulator may be used to test interaction. In our case we can just use real service instead of the emulator that will always differ from it.

[tarasmadan]

Let's add the testing and error processing sections to the design doc?