Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api: do not embed HTTPRoute as-is #47

Merged
merged 5 commits into from
Dec 21, 2024
Merged

api: do not embed HTTPRoute as-is #47

merged 5 commits into from
Dec 21, 2024

Conversation

mathetake
Copy link
Member

@mathetake mathetake commented Dec 18, 2024

This is a follow up on #39, and stops embedding HTTPRoute as-is.
The rationale here is somewhat complicated; We have to know which
backend we route traffics to before Envoy decides the routing as
we need to do perform the transformation etc. Hence, we need to
implement the routing logics (e.g. header matching, weights, etc.)
by ourselves, and cannot rely on the native router code path in Envoy.
That results in us only being able to support only a small subset of
HTTPRoute functionality as well as we cannot simply use the embedded
HTTPRoute as a "base" in the translation. If we embed HTTPRoute here,
it would create an impression that all HTTPRoute functionality can be
supported here no matter how much we document otherwise.

We discussed about this issue, and reached the consensus that we should
have our own LLMRoute rule definition and only have the fields that
actually can be supported.

The example API would look like;

kind: LLMRoute
metadata:
  name: 
  namespace: default
spec:
  inputSchema:
    schema: OpenAI
  rules:
    - matches:
      - headers:
        - type: Exact
          name: x-envoy-ai-gateway-llm-model
          value: llama3-70b
      backendRefs:
        - name: kserve
          weight: 20
        - name: aws-bedrock
          weight: 80

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
@mathetake
Copy link
Member Author

after this i finally will work on implementing the initial controller/extproc code

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
@mathetake mathetake requested a review from yuzisun December 19, 2024 23:43
@mathetake mathetake merged commit 2679739 into main Dec 21, 2024
6 checks passed
@mathetake mathetake deleted the llmroutenothttproute branch December 21, 2024 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants