control-plane proxies connector invocations through data-planes #1601
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jgraettinger
force-pushed
the
johnny/proxy-connectors
branch
from
c343a97 to
6240858
Compare
TestingLast-resort connector timeouts still work:
Formatted connector errors still work:
We get streaming logs at expected log levels:
|
jgraettinger
force-pushed
the
johnny/proxy-connectors
branch
from
6240858 to
ac9f6cb
Compare
psFried
approved these changes
Sep 5, 2024
The ConnectorProxy service starts an identified, ephemeral runtime which exists for the duration of the RPC. It can then be referenced by other RPCs dispatched to the designated address. Issue #1602
Where before it took an ops.Publisher, which is used to attach a ShardRef to the log, it now takes a `func(ops.Log)`. Instead, OpsPublisher is updated to attach the *ShardRef it already has to the Log prior to writting it out to an ops journal. Basically, we're just moving the place where this happens down the stack. This allows LogWriteAdapter to be used with log handlers which _don't_ want to have this behavior, or don't know the proper *ShardRef. Issue #1602
jgraettinger
force-pushed
the
johnny/proxy-connectors
branch
from
ac9f6cb to
19e43cc
Compare
jgraettinger
commented
Sep 5, 2024
`connectorProxy` implements ProxyConnectors by starting a TaskService with a lifetime bound to the RPC, and with a semaphore to constrain maximum concurrency. Logs are streamed back to the caller as they're produced, and a `proxy-id` is returned which is then expected to be attached to various connector RPCs. Issue #1602
gazette::Metadata is metadata attached to a gRPC request, which is inclusive of authorization and other headers. Add new routines for signing Gazette-centric authorization tokens. Also refactor out and expose low-level routines for dialing channels.
NoOpWrapper encapsulates a common pattern of dynamically disabling validations for various connector types. Remove timeouts from unary RPCs directly invoked against the runtime. These timeouts are moving into the `agent`, which is the only place we actually care to time out these requests. When running locally with `flowctl`, for example, we never want to time out.
ProxyConnectors are used for all Validate RPCs and Discover. It's not (yet) used for connector spec requests. Issue #1602
jgraettinger
force-pushed
the
johnny/proxy-connectors
branch
from
19e43cc to
9273008
Compare
|
Completed another round of self-review and local testing, and caught a couple of things (now fixed). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
See individual commits for much more detail.
Fixes #1602
Workflow steps:
No changes to UX. Note that RPCs are now started against reactors as-needed to satisfy control-plane connector invocations.
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
(anything that might help someone review this PR)
This change is