broker: List RPC is now server-streaming and supports long-lived watches #389
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In uses of Gazette, labels often encode a hierarchical namespace such as "foo/bar/baz", and for such labels it's desireable to be able to create a LabelSelector that includes or excludes an entire sub-hierarchy of potential label values, such as including or excluding any label that begins with "foo/bar/". Extend Label to have a Prefix field which may only be set in the context of a LabelSelector (and is a Validate() error otherwise). In a LabelSelector context, Prefix instructs selector matching to match any label value which is prefixed by the given selector label. Introduce a convention of "my-label:prefix" as a special suffix which indicates that prefix matching is desired, and update LabelSelector parsing to round-trip such ":prefix" suffixes. Update the implementation of the special meta-label "prefix" to be in terms of a "name:prefix" selector label, and back out the bespoke implementation that has until-now been used for journal name prefix matching.
jgraettinger
force-pushed
the
johnny/list-watch
branch
from
c08fbaa to
fbda5ce
Compare
The List RPC was originally a unary API, which has two issues: * In large deployments, unary responses bump against gRPC maximum messages sizes. * Unary responses can't represent a long-lived watch of changes. Often callers want to react to changes in journals, such as reading a newly created journal or spreading appended messages across the current set of journals. client.NewPolledList _somewhat_ enabled this functionality, but is inefficient and requires selection of a polling interval. To address both issues, List is updated to become a server-streaming API while retaining backward comatibility with MOST clients expecting a unary response, so long as the response size is under one thousand journals. The List RPC is also extended with a Watch mode. When enabled, brokers will continue to stream real-time updates upon every change to the set of journals which match the selector. This lets clients react instantly to changes in journal topology. client.NewPolledList becomes client.NewWatchedList, with a nearly identical API (the polling interval is removed). The shard List RPC is not modified by this change, but could follow a similar path in the future.
jgraettinger
force-pushed
the
johnny/list-watch
branch
from
fbda5ce to
73637fc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The List RPC was originally a unary API, which has two issues:
Often callers want to react to changes in journals, such as reading a
newly created journal or spreading appended messages across the current
set of journals.
client.NewPolledList somewhat enabled this functionality, but is
inefficient and requires selection of a polling interval.
To address both issues, List is updated to become a server-streaming API
while retaining backward comatibility with MOST clients expecting a
unary response, so long as the response size is under one thousand
journals.
The List RPC is also extended with a Watch mode. When enabled, brokers
will continue to stream real-time updates upon every change to the set
of journals which match the selector. This lets clients react instantly
to changes in journal topology.
client.NewPolledList becomes client.NewWatchedList, with a nearly
identical API (the polling interval is removed).
The shard List RPC is not modified by this change, but could follow a
similar path in the future.
This change is