Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initial TLSRoute support #10601

Open
wants to merge 47 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
d4ab029
rbac
puertomontt Feb 7, 2025
78f3e15
add tls route to controller
puertomontt Feb 7, 2025
21fbb71
add tls route to proxy syncer
puertomontt Feb 7, 2025
e1615a7
add tls route to query
puertomontt Feb 7, 2025
e608c4f
add tls route to reporter
puertomontt Feb 7, 2025
57a0f0b
translate tls listener
puertomontt Feb 7, 2025
3cdddfc
fix type
puertomontt Feb 7, 2025
0abd5bd
insert tls route crd
puertomontt Feb 7, 2025
0cfb588
properly add tlsroute
puertomontt Feb 7, 2025
f5f6772
fix route report: add tls route
puertomontt Feb 7, 2025
bc0e562
report map
puertomontt Feb 7, 2025
d41ae97
add tls route status
puertomontt Feb 7, 2025
bda81f3
test
puertomontt Feb 7, 2025
369fbb2
e2e test wip
puertomontt Feb 8, 2025
1901114
typo
puertomontt Feb 10, 2025
36aed28
handle passthrough by adding filter chain match
puertomontt Feb 10, 2025
0ea19aa
handle aggregate tcp listener in tls inspector plugin
puertomontt Feb 10, 2025
d933553
update test
puertomontt Feb 10, 2025
22523e2
remove debug
puertomontt Feb 10, 2025
5499e9b
add aggregate listener tcp test
puertomontt Feb 10, 2025
533308a
multi svc test
puertomontt Feb 10, 2025
10e7646
add tls route to allowed kinds for tls listener
puertomontt Feb 11, 2025
54b0a71
add ref grant tests
puertomontt Feb 11, 2025
49a7a7d
tls route conformance
puertomontt Feb 13, 2025
e945e54
lint
puertomontt Feb 13, 2025
da7a056
fix test
puertomontt Feb 13, 2025
4c44938
remove comments
puertomontt Feb 13, 2025
4594c2c
cl
puertomontt Feb 13, 2025
de4215a
typo
puertomontt Feb 17, 2025
096a328
clean up
puertomontt Feb 17, 2025
9853dfa
nits
puertomontt Feb 17, 2025
ccc4050
comments
puertomontt Feb 17, 2025
0ab49b3
todo
puertomontt Feb 17, 2025
0a8f543
new line
puertomontt Feb 17, 2025
08185f0
cert info comment
puertomontt Feb 17, 2025
692dd23
add tls route e2e tests to workflow
puertomontt Feb 17, 2025
58613ab
Merge branch 'main' into puertomontt/tls-route
puertomontt Feb 17, 2025
8a5e843
Update changelog/v1.19.0-beta6/tls-route.yaml
puertomontt Feb 17, 2025
5ea5dd2
mv changelog
puertomontt Feb 17, 2025
b0c49dd
Adding changelog file to new location
Feb 17, 2025
113089b
Deleting changelog file from old location
Feb 17, 2025
75be17d
add tests
puertomontt Feb 17, 2025
e04714c
typo
puertomontt Feb 17, 2025
853ca65
Merge branch 'main' of github.com:solo-io/gloo into puertomontt/tls-r…
puertomontt Feb 18, 2025
9c60208
Adding changelog file to new location
Feb 18, 2025
32dac22
Deleting changelog file from old location
Feb 18, 2025
fb523b5
address comment
puertomontt Feb 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -1243,7 +1243,7 @@ $(TEST_ASSET_DIR)/conformance/conformance_test.go:
cat $(shell go list -json -m sigs.k8s.io/gateway-api | jq -r '.Dir')/conformance/conformance_test.go >> $@
go fmt $@

CONFORMANCE_SUPPORTED_FEATURES ?= -supported-features=Gateway,ReferenceGrant,HTTPRoute,HTTPRouteQueryParamMatching,HTTPRouteMethodMatching,HTTPRouteResponseHeaderModification,HTTPRoutePortRedirect,HTTPRouteHostRewrite,HTTPRouteSchemeRedirect,HTTPRoutePathRedirect,HTTPRouteHostRewrite,HTTPRoutePathRewrite,HTTPRouteRequestMirror
CONFORMANCE_SUPPORTED_FEATURES ?= -supported-features=Gateway,ReferenceGrant,HTTPRoute,HTTPRouteQueryParamMatching,HTTPRouteMethodMatching,HTTPRouteResponseHeaderModification,HTTPRoutePortRedirect,HTTPRouteHostRewrite,HTTPRouteSchemeRedirect,HTTPRoutePathRedirect,HTTPRouteHostRewrite,HTTPRoutePathRewrite,HTTPRouteRequestMirror,TLSRoute
CONFORMANCE_SUPPORTED_PROFILES ?= -conformance-profiles=GATEWAY-HTTP
CONFORMANCE_REPORT_ARGS ?= -report-output=$(TEST_ASSET_DIR)/conformance/$(VERSION)-report.yaml -organization=solo.io -project=gloo-gateway -version=$(VERSION) -url=github.com/solo-io/gloo -contact=github.com/solo-io/gloo/issues/new/choose
CONFORMANCE_ARGS := -gateway-class=gloo-gateway $(CONFORMANCE_SUPPORTED_FEATURES) $(CONFORMANCE_SUPPORTED_PROFILES) $(CONFORMANCE_REPORT_ARGS)
Expand Down
6 changes: 6 additions & 0 deletions changelog/v1.19.0-beta6/tls-route.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
changelog:
- type: NEW_FEATURE
issueLink: https://github.com/kgateway-dev/kgateway/issues/10074
resolvesIssue: false
description: >-
"Add support for TLS Routes."
2 changes: 2 additions & 0 deletions install/helm/gloo/templates/44-rbac.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ rules:
- gatewayclasses
- gateways
- tcproutes
- tlsroutes
- httproutes
- referencegrants
verbs: ["get", "list", "watch"]
Expand Down Expand Up @@ -50,6 +51,7 @@ rules:
- gateways/status
- httproutes/status
- tcproutes/status
- tlsroutes/status
verbs: ["update", "patch"]
- apiGroups:
- apiextensions.k8s.io
Expand Down
31 changes: 31 additions & 0 deletions projects/gateway2/controller/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ func NewBaseGatewayController(ctx context.Context, cfg GatewayConfig) error {
controllerBuilder.watchGw,
controllerBuilder.watchHttpRoute,
controllerBuilder.watchTcpRoute,
controllerBuilder.watchTlsRoute,
controllerBuilder.watchReferenceGrant,
controllerBuilder.watchNamespaces,
controllerBuilder.watchHttpListenerOptions,
Expand Down Expand Up @@ -140,6 +141,12 @@ func (c *controllerBuilder) addIndexes(ctx context.Context) error {
}
}

if c.cfg.CRDs.Has(wellknown.TLSRouteCRDName) {
if err := c.cfg.Mgr.GetFieldIndexer().IndexField(ctx, &apiv1a2.TLSRoute{}, query.TlsRouteTargetField, query.IndexerByObjType); err != nil {
errs = append(errs, err)
}
}

return errors.Join(errs...)
}

Expand Down Expand Up @@ -361,6 +368,19 @@ func (c *controllerBuilder) watchTcpRoute(ctx context.Context) error {
Complete(reconcile.Func(c.reconciler.ReconcileTcpRoutes))
}

func (c *controllerBuilder) watchTlsRoute(ctx context.Context) error {
if !c.cfg.CRDs.Has(wellknown.TLSRouteCRDName) {
log.FromContext(ctx).Info("TLSRoute type not registered in scheme; skipping TLSRoute controller setup")
return nil
}

// Proceed to set up the controller for TLSRoute
return ctrl.NewControllerManagedBy(c.cfg.Mgr).
WithEventFilter(predicate.GenerationChangedPredicate{}).
For(&apiv1a2.TLSRoute{}).
Complete(reconcile.Func(c.reconciler.ReconcileTlsRoutes))
}

func (c *controllerBuilder) watchReferenceGrant(_ context.Context) error {
return ctrl.NewControllerManagedBy(c.cfg.Mgr).
WithEventFilter(predicate.GenerationChangedPredicate{}).
Expand Down Expand Up @@ -551,6 +571,17 @@ func (r *controllerReconciler) ReconcileTcpRoutes(ctx context.Context, req ctrl.
return ctrl.Result{}, nil
}

func (r *controllerReconciler) ReconcileTlsRoutes(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
// TODO: consider finding impacted gateways and queue them
// TODO: consider enabling this
// // reconcile this specific route:
// queries := query.NewData(r.cli, r.scheme)
// httproute.TranslateGatewayHTTPRouteRules(queries, hr, nil)

r.kick(ctx)
return ctrl.Result{}, nil
}

func (r *controllerReconciler) ReconcileReferenceGrants(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
// reconcile all things?! https://github.com/solo-io/gloo/issues/9997
r.kick(ctx)
Expand Down
9 changes: 9 additions & 0 deletions projects/gateway2/controller/start.go
Original file line number Diff line number Diff line change
Expand Up @@ -290,5 +290,14 @@ func getGatewayCRDs(restConfig *rest.Config) (sets.Set[string], error) {
crds.Insert(wellknown.TCPRouteCRDName)
}

tlsRouteExists, err := glooschemes.CRDExists(restConfig, gwv1a2.GroupVersion.Group, gwv1a2.GroupVersion.Version, wellknown.TLSRouteKind)
if err != nil {
return nil, err
}

if tlsRouteExists {
crds.Insert(wellknown.TLSRouteCRDName)
}

return crds, nil
}
35 changes: 35 additions & 0 deletions projects/gateway2/proxy_syncer/proxy_syncer.go
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,9 @@ func (p glooProxy) Equals(in glooProxy) bool {
if !maps.Equal(p.reportMap.TCPRoutes, in.reportMap.TCPRoutes) {
return false
}
if !maps.Equal(p.reportMap.TLSRoutes, in.reportMap.TLSRoutes) {
return false
}
return true
}

Expand All @@ -273,6 +276,9 @@ func (r report) Equals(in report) bool {
if !maps.Equal(r.ReportMap.TCPRoutes, in.ReportMap.TCPRoutes) {
return false
}
if !maps.Equal(r.ReportMap.TLSRoutes, in.ReportMap.TLSRoutes) {
return false
}
return true
}

Expand Down Expand Up @@ -491,6 +497,19 @@ func (s *ProxySyncer) Init(ctx context.Context, dbg *krt.DebugHandler) error {
// obsGen will stay as-is...
maps.Copy(p.reportMap.TCPRoutes[rnn].Parents, rr.Parents)
}

// 4. merge tlsroute parentRefs into RouteReports
for rnn, rr := range p.reportMap.TLSRoutes {
// if we haven't encountered this route, just copy it over completely
old := merged.TLSRoutes[rnn]
if old == nil {
merged.TLSRoutes[rnn] = rr
continue
}
// else, let's merge our parentRefs into the existing map
// obsGen will stay as-is...
maps.Copy(p.reportMap.TLSRoutes[rnn].Parents, rr.Parents)
}
}
return &report{merged}
})
Expand Down Expand Up @@ -902,6 +921,12 @@ func (s *ProxySyncer) syncRouteStatus(ctx context.Context, rm reports.ReportMap)
return nil
}
r.Status.RouteStatus = *status
case *gwv1a2.TLSRoute:
status = rm.BuildRouteStatus(ctx, r, s.controllerName)
if status == nil || isRouteStatusEqual(&r.Status.RouteStatus, status) {
return nil
}
r.Status.RouteStatus = *status
default:
logger.Warnw(fmt.Sprintf("unsupported route type for %s", routeType), "route", route)
return nil
Expand Down Expand Up @@ -930,6 +955,16 @@ func (s *ProxySyncer) syncRouteStatus(ctx context.Context, rm reports.ReportMap)
logger.Errorw("all attempts failed at updating TCPRoute status", "error", err, "route", rnn)
}
}

// Sync TLSRoute statuses
for rnn := range rm.TLSRoutes {
err := syncStatusWithRetry(wellknown.TLSRouteKind, rnn, func() client.Object { return new(gwv1a2.TLSRoute) }, func(route client.Object) error {
return buildAndUpdateStatus(route, wellknown.TLSRouteKind)
})
if err != nil {
logger.Errorw("all attempts failed at updating TLSRoute status", "error", err, "route", rnn)
}
}
}

// syncGatewayStatus will build and update status for all Gateways in a reportMap
Expand Down
46 changes: 43 additions & 3 deletions projects/gateway2/query/httproute.go
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,8 @@ func (r *gatewayQueries) GetRouteChain(
case *gwv1a2.TCPRoute:
backends = r.resolveRouteBackends(ctx, typedRoute)
// TODO (danehans): Should TCPRoute delegation support be added in the future?
case *gwv1a2.TLSRoute:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@puertomontt can you create an issue to rename projects/gateway2/query/httproute.go to projects/gateway2/query/route.go and link to this PR?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

backends = r.resolveRouteBackends(ctx, typedRoute)
default:
return nil
}
Expand All @@ -151,7 +153,7 @@ func (r *gatewayQueries) allowedRoutes(gw *gwv1.Gateway, l *gwv1.Listener) (func
case gwv1.HTTPProtocolType:
allowedKinds = []metav1.GroupKind{{Kind: wellknown.HTTPRouteKind, Group: gwv1.GroupName}}
case gwv1.TLSProtocolType:
fallthrough
allowedKinds = []metav1.GroupKind{{Kind: wellknown.TLSRouteKind, Group: gwv1a2.GroupName}}
case gwv1.TCPProtocolType:
allowedKinds = []metav1.GroupKind{{Kind: wellknown.TCPRouteKind, Group: gwv1a2.GroupName}}
case gwv1.UDPProtocolType:
Expand Down Expand Up @@ -228,6 +230,14 @@ func (r *gatewayQueries) resolveRouteBackends(ctx context.Context, obj client.Ob
}
processBackendRefs(refs)
}
case *gwv1a2.TLSRoute:
for _, rule := range rt.Spec.Rules {
var refs []gwv1.BackendObjectReference
for _, ref := range rule.BackendRefs {
refs = append(refs, ref.BackendObjectReference)
}
processBackendRefs(refs)
}
default:
return out
}
Expand Down Expand Up @@ -365,6 +375,16 @@ func (r *gatewayQueries) GetRoutesForGateway(ctx context.Context, gw *gwv1.Gatew
routeListTypes = append(routeListTypes, &gwv1a2.TCPRouteList{})
}

// Conditionally include TLSRouteList
tlsRouteGVK := schema.GroupVersionKind{
Group: gwv1a2.GroupVersion.Group,
Version: gwv1a2.GroupVersion.Version,
Kind: wellknown.TLSRouteKind,
}
if r.scheme.Recognizes(tlsRouteGVK) {
routeListTypes = append(routeListTypes, &gwv1a2.TLSRouteList{})
}

var routes []client.Object
for _, routeList := range routeListTypes {
if err := fetchRoutes(ctx, r, routeList, nns, &routes); err != nil {
Expand Down Expand Up @@ -406,6 +426,10 @@ func fetchRoutes(ctx context.Context, r *gatewayQueries, routeList client.Object
if err := listAndAppendRoutes(list, TcpRouteTargetField); err != nil {
return fmt.Errorf("failed to list TCPRoutes: %w", err)
}
case *gwv1a2.TLSRouteList:
if err := listAndAppendRoutes(list, TlsRouteTargetField); err != nil {
return fmt.Errorf("failed to list TLSRoutes: %w", err)
}
default:
return fmt.Errorf("unsupported route list type: %T", list)
}
Expand Down Expand Up @@ -452,12 +476,22 @@ func (r *gatewayQueries) processRoute(ctx context.Context, gw *gwv1.Gateway, rou
}
anyListenerMatched = true

// If the route is an HTTPRoute, check the hostname intersection
// If the route is an HTTPRoute or TLSRoute, check the hostname intersection
var hostnames []string
if routeKind == wellknown.HTTPRouteKind {
if hr, ok := route.(*gwv1.HTTPRoute); ok {
var ok bool
ok, hostnames = hostnameIntersect(&l, hr)
ok, hostnames = hostnameIntersect(&l, hr.Spec.Hostnames)
if !ok {
continue
}
anyHostsMatch = true
}
}
if routeKind == wellknown.TLSRouteKind {
if tr, ok := route.(*gwv1a2.TLSRoute); ok {
var ok bool
ok, hostnames = hostnameIntersect(&l, tr.Spec.Hostnames)
if !ok {
continue
}
Expand Down Expand Up @@ -532,6 +566,12 @@ func getRouteItems(list client.ObjectList) ([]client.Object, error) {
objs = append(objs, &routes.Items[i])
}
return objs, nil
case *gwv1a2.TLSRouteList:
var objs []client.Object
for i := range routes.Items {
objs = append(objs, &routes.Items[i])
}
return objs, nil
default:
return nil, fmt.Errorf("unsupported route type %T", list)
}
Expand Down
21 changes: 21 additions & 0 deletions projects/gateway2/query/indexers.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ const (
HttpRouteTargetField = "http-route-target"
HttpRouteDelegatedLabelSelector = "http-route-delegated-label-selector"
TcpRouteTargetField = "tcp-route-target"
TlsRouteTargetField = "tls-route-target"
ReferenceGrantFromField = "ref-grant-from"
)

Expand All @@ -25,6 +26,7 @@ func IterateIndices(f func(client.Object, string, client.IndexerFunc) error) err
f(&gwv1.HTTPRoute{}, HttpRouteTargetField, IndexerByObjType),
f(&gwv1.HTTPRoute{}, HttpRouteDelegatedLabelSelector, IndexByHTTPRouteDelegationLabelSelector),
f(&gwv1a2.TCPRoute{}, TcpRouteTargetField, IndexerByObjType),
f(&gwv1a2.TLSRoute{}, TlsRouteTargetField, IndexerByObjType),
f(&gwv1b1.ReferenceGrant{}, ReferenceGrantFromField, IndexerByObjType),
)
}
Expand All @@ -33,6 +35,7 @@ func IterateIndices(f func(client.Object, string, client.IndexerFunc) error) err
//
// - HTTPRoute
// - TCPRoute
// - TLSRoute
// - ReferenceGrant
func IndexerByObjType(obj client.Object) []string {
var results []string
Expand Down Expand Up @@ -74,6 +77,24 @@ func IndexerByObjType(obj client.Object) []string {
}
results = append(results, nns.String())
}
case *gwv1a2.TLSRoute:
for _, pRef := range resource.Spec.ParentRefs {
if pRef.Group != nil && *pRef.Group != gwv1a2.GroupName {
continue
}
if pRef.Kind != nil && *pRef.Kind != wellknown.GatewayKind {
continue
}
ns := resolveNs(pRef.Namespace)
if ns == "" {
ns = resource.Namespace
}
nns := types.NamespacedName{
Namespace: ns,
Name: string(pRef.Name),
}
results = append(results, nns.String())
}
case *gwv1b1.ReferenceGrant:
for _, from := range resource.Spec.From {
if from.Namespace != "" {
Expand Down
Loading
Loading