Skip to content

Commit

Permalink
docs: tweaks rate limit doc on model (envoyproxy#318)
Browse files Browse the repository at this point in the history
**Commit Message**

The model name is extracted by AI Gateway filter, not the one explicitly
added by downstream clients.


**Related Issues/PRs (if applicable)**

Follow up on envoyproxy#260

---------

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
Signed-off-by: Loong <long0dai@foxmail.com>
  • Loading branch information
mathetake authored and daixiang0 committed Feb 19, 2025
1 parent 7453394 commit bcee9f7
Showing 1 changed file with 2 additions and 3 deletions.
5 changes: 2 additions & 3 deletions site/docs/capabilities/usage-based-ratelimiting.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ This guide focuses on AI Gateway's specific capabilities for token-based rate li
AI Gateway leverages Envoy Gateway's Global Rate Limit API to provide token-based rate limiting for LLM requests. Key features include:
- Token usage tracking based on model and user identifiers
- Configuration for tracking input, output, and total token metadata from LLM responses
- Model-specific rate limiting using AI Gateway headers (`x-ai-eg-model`)
- Model-specific rate limiting using AI Gateway headers (`x-ai-eg-model`) which is inserted by the AI Gateway filter with the model name extracted from the request body.
- Support for custom token cost calculations using CEL expressions

## Token Usage Behavior
Expand Down Expand Up @@ -151,18 +151,17 @@ When configuring rate limits:

For proper cost control and rate limiting, requests must include:
- `x-user-id`: Identifies the user making the request
- `x-ai-eg-model`: Identifies the model being used

Example request:
```shell
curl --fail \
-H "Content-Type: application/json" \
-H "x-user-id: user123" \
-H "x-ai-eg-model: gpt-4" \ # Both user ID and model are required
-d '{
"messages": [
{
"role": "user",
"model": "gpt-4",
"content": "Hello!"
}
]
Expand Down

0 comments on commit bcee9f7

Please sign in to comment.