add BackendSecurityPolicy for traffic (authN/authZ) from gateway to provider #43

aabchoo · 2024-12-12T18:24:40Z

add backend security policy for traffic from gateway to backend provider
introduces api key, oidc, and static key auth
add reference to backendSecurityPolicy on llmbackend
add provider type on llmbackend

api/v1alpha1/api.go

Signed-off-by: Aaron Choo <achoo30@bloomberg.net> Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

fix yaml files Signed-off-by: Choo <achoo30@bloomberg.net>

This commit is a follow up on #20. Basically, this makes LLMRoute a pure "addition" to the existing standardized HTTPRoute. This makes it possible to configure something like ``` kind: LLMRoute metadata: name: llm-route spec: inputSchema: OpenAI httpRouteRef: name: my-llm-route --- kind: HTTPRoute metadata: name: my-llm-route spec: matches: - headers: key: x-envoy-ai-gateway-llm-model value: llama3-70b backendRefs: - kserve: weight: 20 - aws-bedrock: weight: 80 ``` where LLMRoute is purely referencing HTTPRoute and users can configure whatever routing condition in a standardized way via HTTPRoute while leveraging the LLM specific information, in this case x-envoy-ai-gateway-llm-model header. In the implementation, though it's not merged yet, we have to do the routing calculation in the extproc by actually analyzing the referenced HTTPRoute, and emulate the behavior in order to do the transformation. The reason is that the routing decision is made at the very end of filter chain in general, and by the time we invoke extproc, we don't have that info. Furthermore, `x-envoy-ai-gateway-llm-model` is not available before extproc. As a bonus of this, we no longer need TargetRef at LLMRoute level since that's within the HTTPRoute resources. This will really simplify the PoC implementation. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

This adds `backendRef` to LLMBackendSpec which specifies the "backend" either Service or Backend resource of Envoy Gateway. The choice of not embedding is intentional - A backend can be a target of routing in HTTPRoute and can be either of these two types. Hence this is not suitable for embedding. In addition, in the implementation, we won't directly use or reference them, but just simply attach the necessary logic by the names, so basically no benefit by doing so. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

api/v1alpha1/api.go

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

api/v1alpha1/api.go

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

api/v1alpha1/api.go

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

yuzisun · 2025-01-04T00:09:05Z

@mathetake @arkodg @zhaohuabing can you take another look?

api/v1alpha1/api.go

mathetake

LGTM per nits above!

api/v1alpha1/api.go

mathetake · 2025-01-04T01:58:03Z

api/v1alpha1/api.go

+	// CloudProviderCredentials is a mechanism to access a backend(s). Cloud provider specific logic will be applied.
+	//
+	// +optional
+	CloudProviderCredentials *AuthenticationCloudProviderCredentials `json:"cloudProviderCredentials,omitempty"`


so i wonder what kind of logic will make cloud providers special compared to others (currently only apiKey though). how do you envision this additional indirect layer will be used? in other words, what kind of logic will be shared among cloud providers

ApiKey is a simple authentication mechanism which distributes the token to user and user is responsible to rotate the keys. Providers like Anthropic, Mistral, Cohere only support API keys.

Cloud authentication is more sophisticated and support issuing temporary tokens like AWS STS, Google STS to access cloud resources. You do not have to distribute the token and major clouds also support OIDC federation to allow managing your user identity in external system.
see https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html#:~:text=You%20can%20use%20the%20AWS,has%20permissions%20to%20do%20so.
https://cloud.google.com/docs/authentication#credentials

yeah i get that it's complex, but my question is why this another layer is needed and what kind of logic will be shared among cloud providers? what becomes difficult to achieve when you have *AWSCredentials directly here without another layer of AuthenticationCloudProviderCredentials. How does having AuthenticationCloudProviderCredentials helps implement various cloud providers vs having them in here

This is mainly a yaml UX, the additional layer for grouping is there for readability, if we list the individual types then it is not intuitive for user to see the classification. The cloud provider security implementations are different but they share the same high level concepts.

api/v1alpha1/api.go

Co-authored-by: Takeshi Yoneda <t.y.mathetake@gmail.com> Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

Co-authored-by: Takeshi Yoneda <t.y.mathetake@gmail.com> Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

api/v1alpha1/api.go

mathetake · 2025-01-07T23:57:14Z

api/v1alpha1/api.go

+// Only one type of BackendSecurityPolicy can be defined.
+// +kubebuilder:validation:MaxProperties=2
+type BackendSecurityPolicySpec struct {
+	// Type specifies the auth mechanism used to access the provider. Currently, only "APIKey", AND "AWS_IAM" are supported.


well, shouldn't the type be "CloudProviders" instead of AWS_IAM? The "type" field is for union type assertion and this feels a weird because AWS_IAM is not for the union type assertion but for the higher layer. @yuzisun @arkodg

This is relevant to the thread to #43 (comment) I am still not convinced by the "Yaml UX" stuff there. I am fine with that but at least to be consistent could you make this CloudProviders and have a type inside it for a specific cloud provider? (which i think is a worse UX vs the flat one)

I've flattened it out per req

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

mathetake

LGTM. Thank you for your hard word and multiple iterations here! @aabchoo 💯

**Commit Message**: This adds a secret watcher controller that enables the hot reload of any secret referenced by backendTrafficPolicy. **Related Issues/PRs (if applicable)**: Follow up on #43 #106 #161 Supersede #185 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

aabchoo requested review from mathetake, missBerg and wengyao04 as code owners December 12, 2024 18:24

mathetake reviewed Dec 12, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

aabchoo force-pushed the aaron/authorization-api branch 2 times, most recently from 52e389b to 89ee946 Compare December 12, 2024 19:35

aabchoo and others added 7 commits December 12, 2024 14:50

Set up API for AUTH api

08be1f7

Signed-off-by: Aaron Choo <achoo30@bloomberg.net> Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

2ac7698

fix yaml files Signed-off-by: Choo <achoo30@bloomberg.net>

updating test-cel

e200702

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

add llm provider policy as part of crd

43f77fc

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

Merge remote-tracking branch 'origin/main' into aaron/authorization-api

ea69193

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

aabchoo force-pushed the aaron/authorization-api branch from 9fb8a19 to ea69193 Compare December 12, 2024 19:54

typo in test-cel

7e9315c

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

Krishanx92 reviewed Dec 13, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

Krishanx92 reviewed Dec 13, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

update llmproviderpolicy -> llmsecuritypolicy

57e0fa1

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

aabchoo force-pushed the aaron/authorization-api branch from a829fe8 to 57e0fa1 Compare December 13, 2024 15:38

ref to llmsecuritypolicy in the backend

bc0dadd

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

mathetake reviewed Dec 14, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

api/v1alpha1/api.go Outdated Show resolved Hide resolved

api/v1alpha1/api.go Outdated Show resolved Hide resolved

aabchoo added 2 commits December 16, 2024 10:55

update documentation and tests

b071160

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

update crd name

d2f72ed

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

aabchoo force-pushed the aaron/authorization-api branch from 19c1337 to d2f72ed Compare December 16, 2024 15:56

yuzisun reviewed Dec 16, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

yuzisun reviewed Dec 16, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

aabchoo added 4 commits December 16, 2024 17:54

update to generalize aws auth

52ef4f9

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

update llmproviderpolicy type

7b811b7

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

auto gen update

7dbb202

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

add oidc as part of security policy spec

c04ec5e

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

mathetake reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

zhaohuabing reviewed Jan 6, 2025

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

aabchoo and others added 9 commits January 6, 2025 14:54

Update api/v1alpha1/api.go

f7ee160

Co-authored-by: Takeshi Yoneda <t.y.mathetake@gmail.com> Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

Update api/v1alpha1/api.go

c4f7baf

Co-authored-by: Takeshi Yoneda <t.y.mathetake@gmail.com> Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

make secretRef required

6b8f698

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

Update api/v1alpha1/api.go

1d71e7a

Co-authored-by: Takeshi Yoneda <t.y.mathetake@gmail.com> Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

Update api/v1alpha1/api.go

70a4b29

Co-authored-by: Takeshi Yoneda <t.y.mathetake@gmail.com> Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

Add validation and comments to BackendSecurityPolicy

99e8ae3

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

credentialFile to ref secret

70aa411

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

Merge branch 'main' into aaron/authorization-api

27df777

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

update description of oidc audience

dbc37e2

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

mathetake approved these changes Jan 7, 2025

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

mathetake reviewed Jan 7, 2025

View reviewed changes

mathetake mentioned this pull request Jan 8, 2025

Adds initial controller implementation #71

Merged

aabchoo added 2 commits January 8, 2025 11:51

update backendSecurityPolicy type + add CloudProvider type

21db108

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

flatten cloudprovivder

480b26b

Signed-off-by: Aaron Choo <achoo30@bloomberg.net>

mathetake approved these changes Jan 8, 2025

View reviewed changes

mathetake merged commit 7ada5ca into main Jan 8, 2025
7 checks passed

mathetake deleted the aaron/authorization-api branch January 8, 2025 23:50

mathetake mentioned this pull request Jan 29, 2025

controller: adds secret watcher to trigger extproc config updates #219

Merged

aabchoo mentioned this pull request Jan 30, 2025

feat: implementation of AWS OIDC #232

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add BackendSecurityPolicy for traffic (authN/authZ) from gateway to provider #43

add BackendSecurityPolicy for traffic (authN/authZ) from gateway to provider #43

aabchoo commented Dec 12, 2024 •

edited

Loading

yuzisun commented Jan 4, 2025

mathetake left a comment

mathetake Jan 4, 2025

yuzisun Jan 4, 2025 •

edited

Loading

mathetake Jan 6, 2025

yuzisun Jan 7, 2025

mathetake Jan 7, 2025

mathetake Jan 7, 2025

aabchoo Jan 8, 2025

mathetake left a comment

add BackendSecurityPolicy for traffic (authN/authZ) from gateway to provider #43

add BackendSecurityPolicy for traffic (authN/authZ) from gateway to provider #43

Conversation

aabchoo commented Dec 12, 2024 • edited Loading

yuzisun commented Jan 4, 2025

mathetake left a comment

Choose a reason for hiding this comment

mathetake Jan 4, 2025

Choose a reason for hiding this comment

yuzisun Jan 4, 2025 • edited Loading

Choose a reason for hiding this comment

mathetake Jan 6, 2025

Choose a reason for hiding this comment

yuzisun Jan 7, 2025

Choose a reason for hiding this comment

mathetake Jan 7, 2025

Choose a reason for hiding this comment

mathetake Jan 7, 2025

Choose a reason for hiding this comment

aabchoo Jan 8, 2025

Choose a reason for hiding this comment

mathetake left a comment

Choose a reason for hiding this comment

aabchoo commented Dec 12, 2024 •

edited

Loading

yuzisun Jan 4, 2025 •

edited

Loading