Integrate bb-browser and bb-scheduler web interface into bb-portal

[isakstenstrom]

This PR does the following

1. Replaces the favicon to the correct one for Buildbarn.
2. Removes the animation that counts up some numbers in the frontend.
3. Integrates the bb-scheduler web interface into bb-portal.
4. Integrates the bb-browser web interface into bb-portal.

Let us know if change 1 and 2 does not fit in this large PR, so that we can create separate PRs for them.

This PR integrates the bb-scheduler and bb-browser web interfaces into bb-portal. This consolidates information about the status of the Buildbarn cluster into a single web app.

The frontend now communicates with the Buildbarn cluster using gRPC-web, that is proxied via the backend, which both provides translation between gRPC <-> gRPC-web and also access control.

The new interface is similar to the corresponding interfaces that it is replacing. It is close to full feature parity with the old interfaces. Some of the features that are missing are:

• The tree-page in bb-browser.
• Some columns are missing for the platform queues table, including the drains columns.
• It only supports SHA256 for Previous Execution Stats, and File System Access Cache.

New pages:

• Remote execution scheduler: An overview of all platform queues in the cluster, that the user is allowed to see. This is configured using the listPlatformQueueAuthorizer in the portal config.
• Workers: An overview of all workers in a specific queue, both executing and idle.
• Operations
• Operation
• Action
• Directory
• Previous execution stats
• Historical execute response
• Command

[trey-ivy]

super exciting! let me pull this down and take a look. nice work guys!

[trey-ivy]

hey @isakstenstrom, this seems really promising and i'm excited to land it and thank you for your efforts on this.

I had to run npm i to install the new packages et voila. pretty painless. 🎉

...however I'm not able to access the scheduler. Not with my existing dev scheduler service nor with the docker-compose instance. both of them throw an error.

however the scheduler service is def up

and i can grpcurl it directly

I first tried connecting to our dev instance of buildbarn and thought maybe my config was wrong or something... so I spun up an instance using the docker-compose script as that's what's referenced in the config file, but I kept getting the same connection refused message while trying to connect

There was a problem communicating with the backend server:
ClientError: /buildbarn.buildqueuestate.BuildQueueState/ListOperations UNKNOWN: Transport error: Failed to fetch

any ideas what I might be missing? I can try and throw the debugger on it later too. I haven't dug in to the code yet but after a cursory look I have a couple initial questions.

1. From a quick perusal its not clear if I could run bbportal anymore w/out connecting it to a buildbarn cache/scheduler service. is that accurate? Its not necessarily blocking, but just curious if it is possible to still run the app if I don't run buildbarn. Are there any configuration flags to turn this functionality on/off?
2. Can we call the scheduler tab just 'Scheduler'?
3. Feel free to squash em up.
4. I would have preferred that you provide functionality to optionally turn the count-ups on or off or at least ask... i'm not married to them and if people don't like it we can indeed remove them.
5. ❤️ the tests

[mortenmj]

Really nice work, very happy to see this. I'll defer to @trey-ivy to review, as I'm still on leave. Some comments to @trey-ivy's comments, though:

From a quick perusal its not clear if I could run bbportal anymore w/out connecting it to a buildbarn cache/scheduler service. is that accurate? Its not necessarily blocking, but just curious if it is possible to still run the app if I don't run buildbarn. Are there any configuration flags to turn this functionality on/off?

I think it's pretty important we keep this ability. I can certainly see users running portal in a way where it won't be able to access scheduler.

Can we call the scheduler tab just 'Scheduler'?

+1

Feel free to squash em up.

Do you mean the commits? I think the current 4 commits make logical sense, and would prefer to keep them

I would have preferred that you provide functionality to optionally turn the count-ups on or off or at least ask... i'm not married to them and if people don't like it we can indeed remove them.

I think the count-ups are pretty annoying after using portal for a bit, so I'm happy to see them go

❤️ the tests

+1

[oscar-meroton]

Hey @trey-ivy! Co-author for the PR here, thanks for the feedback.

We were able to replicate the error when changing allowedOrigins in the proxy configurations in portal.jsonnet, meaning the error (in our case) is a CORS issue. Are you sending the requests from the correct origin?

1. Good point! You are not able to start bb-portal without connecting it to the proxies. We don't want to remove the ability to do so, so we are working on a fix.
2. Absolutely.
3. We'd prefer to keep the commits as they are, but we can squash them if you prefer.
4. Fair enough, we'll look into making the animations optional.

[trey-ivy]

sounds like the consensus is to remove the count-ups, so don't waste effort trying to make them optional. we can just rip them out! 👍🏻

[trey-ivy]

okay so after adjusting the CORS and auth policy in the config file it works 🎉

I initially didn't make any changes to the configuration in the portal.jsonnet file. Its a valid config so it didn't throw any errors but it also didn't work. I feel like I shouldn't need to adjust the config for a fresh clone. Could we streamline the portal.jsonnet file so that it works out of the box with the default docker-compose bb-portal? Stripping the cors and setting auth policy to allow : {} could do the trick. I'm fine with keeping this version you've included as a second example config file in there as well. maybe name it example2.jsonnet or something with this additional detail. Perhaps we can also provide a link to the documentation in the readme as well.

[oscar-meroton]

@trey-ivy We intended it to work out the box with bb-deployments. I can replicate the error if I'm using the frontend directly (localhost:3000) instead of the portal backend reverse proxy (localhost:8081), might this be the issue? In the bb-portal config we specify to allow only the origin localhost:8081 since we assumed that localhost:3000 was not being used.

[trey-ivy]

well, turns out some of our security tooling at work quarantined something in in one of the runner processes and that may have been what was causing some of my issues. I got it whitelisted and then when i spun it up with a fresh clone and deployed it I was able to get it working without adjusting the config. I did notice though that I'm only seeing the fuse instance, and not the hard-linking instance. is this by design? I see both in your screenshots above.

bb-portal

bb-scheduler

[oscar-meroton]

I got it whitelisted and then when i spun it up with a fresh clone and deployed it I was able to get it working without adjusting the config.

Awesome! 🎉

I did notice though that I'm only seeing the fuse instance, and not the hard-linking instance. is this by design? I see both in your screenshots above.

Yes, this is by design. The screenshots are not entirely consistent with the bb-portal config, where we combine an authorizer and an authentication policy to only forward requests with permitted instance names, which includes fuse but not hardlinking.

[isakstenstrom]

Now we have updated the PR, with these changes

1. Remote execution scheduler -> Scheduler in the URL and the menu
2. The go backend can start without any of the new config for the proxies
3. Bug fix that makes the backend not crash on some malformed requests.

Let us know if you want anything else changed!

[trey-ivy]

I have a couple minor nits but nothing blocking or maybe you could consider them feature requests rather than nits but either way I'm just going to land this and they can always be addressed in a follow on PR.

Usually when using bb-scheduler in the past I always liked (and disliked) the feature that you had to forcibly refresh the page to get the operations queue to update. Actions often execute so quickly that they're gone before i can catch them otherwise.

I don't know how feasible it would be to have a pause button that pauses the fetches for the build queue, but that would be cool. Not to actually pause the build queue, just to pause the updates to the UI so I can look at it in a given state.

A similar thing happens when an operation expires. I occasionally want to keep that information around at least temporarily. With the current setup, as soon as an operation expires, its gone and even if i kept the operations page open, that also expires

Is there anyway we could stop updating this view once we get an operation so that it hangs around at least until the tab is closed? just stop refreshing once its found? I don't expect them to be available if I don't catch it in the act, or anything, but if I click thru from the scheduler and get a view I'd prefer it not expire

[isakstenstrom]

Good point!

The automatic refetching was not intentional and was the default of the data fetching library used. Automatic refetching of some parts of the data would be nice, but not on the operations pages in my opinion. I have created PR #92 that disables the refetching for the operations pages.