From f21c4ea6c2aad327ae16639cd12642560d555cdd Mon Sep 17 00:00:00 2001 From: Eric Gustin Date: Mon, 27 Jan 2025 17:41:49 -0800 Subject: [PATCH 1/8] Tool Definition docs --- pages/home/build-tools/tool-definition.mdx | 49 ++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 pages/home/build-tools/tool-definition.mdx diff --git a/pages/home/build-tools/tool-definition.mdx b/pages/home/build-tools/tool-definition.mdx new file mode 100644 index 0000000..22cd9fc --- /dev/null +++ b/pages/home/build-tools/tool-definition.mdx @@ -0,0 +1,49 @@ +--- +title: "Tool Definition" +description: "Documentation for the Tool Context in the Arcade Tool SDK" +--- + +# Tool Definitions + +When defining an Arcade tool, it's important to understand which parts of the tool's definition are sent to the language model and which are not. This understanding helps in designing tools that effectively communicate their purpose and usage to the model. + +```python +@tool( + requires_auth=Google( + scopes=["https://www.googleapis.com/auth/gmail.compose"], + ) # Not sent to the model +) +async def write_draft_email( # Tool name and toolkit name is sent to the model + context: ToolContext, # ToolContext is never sent to the model + subject: Annotated[str, "Subject of the email"], # Sent to the model + body: Annotated[str, "Body content of the email"], # Sent to the model + recipient: Annotated[str, "Email recipient"], # Sent to the model + cc: Annotated[Optional[list[str]], "CC recipients"], # Sent to the model + bcc: Annotated[Optional[list[str]], "BCC recipients"], # Sent to the model +) -> Annotated[dict, "Email draft details"]: # Return type not sent to the model + """ + Drafts an email with the given subject and body. # Sent to the model + """ + # The body of the function is not sent to the model + ... +``` + +## Parts Sent to the Language Model + +1. _**Toolkit Name and Tool Name:**_ The names of the toolkit and the specific tool are sent to the language model. +2. _**Function Docstring:**_ The docstring of the function is sent to the language model. This provides a high-level description of what the tool does. +3. _**Parameter Names:**_ Each parameter's name is sent to the language model. +4. _**Parameter Types:**_ The type of each parameter is sent to the language model. +5. _**Enum Values:**_ If a parameter is an enum, each possible value for the enum is sent to the language model. This provides clarity on the valid options for that parameter. +6. _**Parameter Annotations:**_ The annotation for each parameter is sent to the language model. Annotations often provide additional context or constraints on the parameter. +7. _**Parameter Requirement:**_ Whether a parameter is required or optional is sent to the language model. + +## Parts Not Sent to the Language Model + +1. _**Default Values for Optional Parameters:**_ The default value assigned to an optional parameter is not sent to the language model. This means the language model does not have information on what default behavior to expect if a parameter is not provided. +2. _**Function Body:**_ The actual implementation of the function, i.e., the function body, is not sent to the language model. This keeps the focus on the interface rather than the implementation details. +3. _**`@tool` Decorator Details:**_ Any specifics within the `@tool` decorator, such as authentication requirements or other configurations, are not sent to the language model. +4. _**Return Type:**_ The type of the return value is not sent to the language model. This means the language model does not have explicit information on what type of data to expect as output. +5. _**Return Annotation:**_ Similar to the return type, the return annotation is not sent to the language model. This excludes any additional context or constraints on the return value from the language model's understanding. + +Understanding these distinctions helps in crafting tools that are clear and informative for the language model. If you are ever unsure about what is sent to the language model, you can always retrieve a tool's definition formatted for a specific model. For more details, refer to the [get tool definitions](home/use-tools/get-tool-definitions) documentation. From 280986844fd9537e8d38e3b7296a514fa9fc3cc3 Mon Sep 17 00:00:00 2001 From: Eric Gustin Date: Mon, 27 Jan 2025 17:44:32 -0800 Subject: [PATCH 2/8] Update file metadata --- pages/home/build-tools/tool-definition.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/home/build-tools/tool-definition.mdx b/pages/home/build-tools/tool-definition.mdx index 22cd9fc..4c29b99 100644 --- a/pages/home/build-tools/tool-definition.mdx +++ b/pages/home/build-tools/tool-definition.mdx @@ -1,6 +1,6 @@ --- title: "Tool Definition" -description: "Documentation for the Tool Context in the Arcade Tool SDK" +description: "Documentation for what is sent to the language model when defining a tool" --- # Tool Definitions From d4257c107d1b03445b0438524415ca34d3088f20 Mon Sep 17 00:00:00 2001 From: Eric Gustin Date: Mon, 27 Jan 2025 18:40:26 -0800 Subject: [PATCH 3/8] Address Sam's comments --- pages/home/build-tools/tool-definition.mdx | 23 ++++++---------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/pages/home/build-tools/tool-definition.mdx b/pages/home/build-tools/tool-definition.mdx index 4c29b99..46c8a42 100644 --- a/pages/home/build-tools/tool-definition.mdx +++ b/pages/home/build-tools/tool-definition.mdx @@ -28,22 +28,11 @@ async def write_draft_email( # Tool name and toolkit name is sent to the model ... ``` -## Parts Sent to the Language Model +## Additional Information -1. _**Toolkit Name and Tool Name:**_ The names of the toolkit and the specific tool are sent to the language model. -2. _**Function Docstring:**_ The docstring of the function is sent to the language model. This provides a high-level description of what the tool does. -3. _**Parameter Names:**_ Each parameter's name is sent to the language model. -4. _**Parameter Types:**_ The type of each parameter is sent to the language model. -5. _**Enum Values:**_ If a parameter is an enum, each possible value for the enum is sent to the language model. This provides clarity on the valid options for that parameter. -6. _**Parameter Annotations:**_ The annotation for each parameter is sent to the language model. Annotations often provide additional context or constraints on the parameter. -7. _**Parameter Requirement:**_ Whether a parameter is required or optional is sent to the language model. +- _**Defaults:**_ The default value for optional parameters is not sent to the language model. +- _**Enum Values:**_ If a parameter is an enum, each possible value for the enum is sent to the language model. This provides clarity on the valid options for that parameter. +- _**Annotations:**_ The annotation for each parameter is sent to the language model, thus each parameter must be annotated. +- _**Tool Definition Schema:**_ Check out the [Tool Definition JSON schema](https://github.com/ArcadeAI/arcade-ai/blob/main/schemas/preview/tool_definition.schema.jsonc) for more details on how to define your tools. -## Parts Not Sent to the Language Model - -1. _**Default Values for Optional Parameters:**_ The default value assigned to an optional parameter is not sent to the language model. This means the language model does not have information on what default behavior to expect if a parameter is not provided. -2. _**Function Body:**_ The actual implementation of the function, i.e., the function body, is not sent to the language model. This keeps the focus on the interface rather than the implementation details. -3. _**`@tool` Decorator Details:**_ Any specifics within the `@tool` decorator, such as authentication requirements or other configurations, are not sent to the language model. -4. _**Return Type:**_ The type of the return value is not sent to the language model. This means the language model does not have explicit information on what type of data to expect as output. -5. _**Return Annotation:**_ Similar to the return type, the return annotation is not sent to the language model. This excludes any additional context or constraints on the return value from the language model's understanding. - -Understanding these distinctions helps in crafting tools that are clear and informative for the language model. If you are ever unsure about what is sent to the language model, you can always retrieve a tool's definition formatted for a specific model. For more details, refer to the [get tool definitions](home/use-tools/get-tool-definitions) documentation. +Understanding these distinctions helps in crafting tools that are clear and informative for the language model. If you are ever unsure about what is sent to the language model, you can always retrieve a tool's definition formatted for a specific model. For more details, refer to the [get formatted tool definitions](home/use-tools/get-tool-definitions) documentation. From ec3f998d4e8bd0427629ec66b8d63a8e9f338974 Mon Sep 17 00:00:00 2001 From: Eric Gustin Date: Tue, 28 Jan 2025 15:08:55 -0800 Subject: [PATCH 4/8] Improve --- pages/home/build-tools/tool-definition.mdx | 246 +++++++++++++++++++-- 1 file changed, 228 insertions(+), 18 deletions(-) diff --git a/pages/home/build-tools/tool-definition.mdx b/pages/home/build-tools/tool-definition.mdx index 46c8a42..bbea323 100644 --- a/pages/home/build-tools/tool-definition.mdx +++ b/pages/home/build-tools/tool-definition.mdx @@ -1,38 +1,248 @@ --- -title: "Tool Definition" +title: "What is sent to language models?" description: "Documentation for what is sent to the language model when defining a tool" --- -# Tool Definitions +import { Tabs } from "nextra/components"; -When defining an Arcade tool, it's important to understand which parts of the tool's definition are sent to the language model and which are not. This understanding helps in designing tools that effectively communicate their purpose and usage to the model. +# Understanding what is sent to language models + +Which parts of your tool definitions are sent to language models? By understanding this, you can ensure that your tools effectively communicate their purpose and usage to the models. + +The Arcade worker converts tools (functions with the `@tool` decorator) into `ToolDefinition` objects. The `ToolDefinition` object can be thought of as the intermediate representation of a tool before it is formatted for a specific model provider and sent to the model. + +## Defining a tool in code + +First, let's take a look at a tool written in Python before it is converted into a `ToolDefinition` object. ```python +from enum import Enum +from typing import Annotated, Optional + +from arcade.sdk import ToolContext, tool + +class ConversationType(str, Enum): + # Enum values are sent to the model + PUBLIC_CHANNEL = "public_channel" + PRIVATE_CHANNEL = "private_channel" + MULTI_PERSON_DIRECT_MESSAGE = "multi_person_direct_message" + DIRECT_MESSAGE = "direct_message" + @tool( - requires_auth=Google( - scopes=["https://www.googleapis.com/auth/gmail.compose"], - ) # Not sent to the model + requires_auth=Slack( + scopes=["channels:read", "groups:read", "im:read", "mpim:read"], + ) # Not sent to the model ) -async def write_draft_email( # Tool name and toolkit name is sent to the model - context: ToolContext, # ToolContext is never sent to the model - subject: Annotated[str, "Subject of the email"], # Sent to the model - body: Annotated[str, "Body content of the email"], # Sent to the model - recipient: Annotated[str, "Email recipient"], # Sent to the model - cc: Annotated[Optional[list[str]], "CC recipients"], # Sent to the model - bcc: Annotated[Optional[list[str]], "BCC recipients"], # Sent to the model -) -> Annotated[dict, "Email draft details"]: # Return type not sent to the model - """ - Drafts an email with the given subject and body. # Sent to the model +async def list_conversations_metadata( # Tool name and toolkit name is sent to the model + context: ToolContext, # ToolContext is never sent to the model + conversation_types: Annotated[Optional[list[ConversationType]], "The type(s) of conversations to list. Defaults to all types."] = None, # Sent to the model + limit: Annotated[Optional[int], "The maximum number of conversations to list."] = None, # Sent to the model + next_cursor: Annotated[Optional[str], "The cursor to use for pagination." = None, # Sent to the model +) -> Annotated[dict,"The conversations metadata list and a pagination 'next_cursor', if there are more conversations to retrieve."]: # Not sent to the model """ + List metadata for Slack conversations (channels and/or direct messages) that the user + is a member of. + """ # Docstring is sent to the model # The body of the function is not sent to the model ... ``` +## A tool as a `ToolDefinition` object + +As mentioned above, the Arcade worker converts tools into `ToolDefinition` objects. Let's take a look at the `list_conversations_metadata` tool as a `ToolDefinition` object (the intermediate representation of the tool before it is formatted for a specific model). + + + +```python +from arcade_slack.tools.chat import list_conversations_metadata + +from arcade.sdk import ToolCatalog + +tool_def = ToolCatalog.create_tool_definition(list_conversations_metadata, "Slack") +tool_def_json = tool_def.model_dump_json() + +print(tool_def_json) +``` + + + +```bash +curl -X 'GET' 'https://api.arcade.dev/v1/tools/Slack.ListConversationsMetadata' \ + -H 'Accept: application/json' \ + -H 'Authorization: Bearer {{arcade_api_key}}' +``` + + + +
+ + + +```json +{ + "name": "ListConversationsMetadata", + "fully_qualified_name": "Slack.ListConversationsMetadata", + "description": "List metadata for Slack conversations (channels and/or direct messages) that the user\nis a member of.", + "toolkit": { "name": "Slack", "description": null, "version": null }, + "input": { + "parameters": [ + { + "name": "conversation_types", + "required": false, + "description": "The type(s) of conversations to list. Defaults to all types.", + "value_schema": { + "val_type": "array", + "inner_val_type": "string", + "enum": [ + "public_channel", + "private_channel", + "multi_person_direct_message", + "direct_message" + ] + }, + "inferrable": true + }, + { + "name": "limit", + "required": false, + "description": "The maximum number of conversations to list.", + "value_schema": { + "val_type": "integer", + "inner_val_type": null, + "enum": null + }, + "inferrable": true + }, + { + "name": "next_cursor", + "required": false, + "description": "The cursor to use for pagination.", + "value_schema": { + "val_type": "string", + "inner_val_type": null, + "enum": null + }, + "inferrable": true + } + ] + }, + "output": { + "description": "The conversations metadata list and a pagination 'next_cursor', if there are more conversations to retrieve.", + "available_modes": ["value", "error"], + "value_schema": { "val_type": "json", "inner_val_type": null, "enum": null } + }, + "requirements": { + "authorization": { + "provider_id": "slack", + "provider_type": "oauth2", + "id": null, + "oauth2": { + "scopes": ["channels:read", "groups:read", "im:read", "mpim:read"] + } + } + } +} +``` + + + +## Formatting tools for model compatibility + +Before tools are sent to models, the `ToolDefinition` object is formatted such that it is compatible with the specific model provider's function calling format. For example, let's take a look at the `list_conversations_metadata` tool formatted for OpenAI. + + + + + +```python +from arcadepy import Arcade + +client = Arcade() +list_conversations_metadata = client.tools.formatted.get( + name="Slack.ListConversationsMetadata", format="openai" +) + +print(list_conversations_metadata) +``` + + + + + +```js +import Arcade from "@arcadeai/arcadejs"; + +const client = new Arcade(); +const listConversationsMetadata = await client.tools.formatted.get( + "Slack.ListConversationsMetadata", + { + format: "openai", + }, +); + +console.log(listConversationsMetadata); +``` + + + + + +```bash +curl -X 'GET' 'https://api.arcade.dev/v1/formatted_tools/Slack.ListConversationsMetadata?format=openai' \ + -H 'Accept: application/json' \ + -H 'Authorization: Bearer {{arcade_api_key}}' +``` + + + + + + + +```json +{ + "type": "function", + "function": { + "name": "Slack_ListConversationsMetadata", + "description": "List metadata for Slack conversations (channels and/or direct messages) that the user\nis a member of.", + "parameters": { + "type": "object", + "properties": { + "conversation_types": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "public_channel", + "private_channel", + "multi_person_direct_message", + "direct_message" + ] + }, + "description": "The type(s) of conversations to list. Defaults to all types." + }, + "limit": { + "type": "integer", + "description": "The maximum number of conversations to list." + }, + "next_cursor": { + "type": "string", + "description": "The cursor to use for pagination." + } + }, + "required": [] + } + } +} +``` + + + ## Additional Information -- _**Defaults:**_ The default value for optional parameters is not sent to the language model. +- _**Defaults:**_ The default value for parameters listed as optional is not sent to the language model. - _**Enum Values:**_ If a parameter is an enum, each possible value for the enum is sent to the language model. This provides clarity on the valid options for that parameter. - _**Annotations:**_ The annotation for each parameter is sent to the language model, thus each parameter must be annotated. - _**Tool Definition Schema:**_ Check out the [Tool Definition JSON schema](https://github.com/ArcadeAI/arcade-ai/blob/main/schemas/preview/tool_definition.schema.jsonc) for more details on how to define your tools. -Understanding these distinctions helps in crafting tools that are clear and informative for the language model. If you are ever unsure about what is sent to the language model, you can always retrieve a tool's definition formatted for a specific model. For more details, refer to the [get formatted tool definitions](home/use-tools/get-tool-definitions) documentation. +Understanding how a tool is converted into a `ToolDefinition` object and then formatted for a specific model is important for crafting tools that are clear and informative for the language model. From 437b852f8e8ffa0496b0da1ac1e75a1200c6951e Mon Sep 17 00:00:00 2001 From: Eric Gustin Date: Tue, 28 Jan 2025 15:26:44 -0800 Subject: [PATCH 5/8] Use code examples --- .../defining_a_tool_in_code.py | 32 +++ .../get_formatted_tool_oai.js | 5 + .../get_formatted_tool_oai.py | 8 + .../get_formatted_tool_oai.sh | 3 + .../get_formatted_tool_oai_output.json | 1 + .../get_tool_definition.py | 8 + .../get_tool_definition.sh | 3 + .../get_tool_definition_output.json | 1 + pages/home/build-tools/tool-definition.mdx | 248 ------------------ .../what-is-sent-to-language-models.mdx | 94 +++++++ 10 files changed, 155 insertions(+), 248 deletions(-) create mode 100644 examples/code/home/build-tools/what-is-sent-to-language-models/defining_a_tool_in_code.py create mode 100644 examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js create mode 100644 examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.py create mode 100644 examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.sh create mode 100644 examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai_output.json create mode 100644 examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition.py create mode 100644 examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition.sh create mode 100644 examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition_output.json delete mode 100644 pages/home/build-tools/tool-definition.mdx create mode 100644 pages/home/build-tools/what-is-sent-to-language-models.mdx diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/defining_a_tool_in_code.py b/examples/code/home/build-tools/what-is-sent-to-language-models/defining_a_tool_in_code.py new file mode 100644 index 0000000..d900f18 --- /dev/null +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/defining_a_tool_in_code.py @@ -0,0 +1,32 @@ +from enum import Enum +from typing import Annotated, Optional + +from arcade.sdk.auth import Slack +from arcade.sdk import ToolContext, tool + + +class ConversationType(str, Enum): + # Enum values are sent to the model + PUBLIC_CHANNEL = "public_channel" + PRIVATE_CHANNEL = "private_channel" + MULTI_PERSON_DIRECT_MESSAGE = "multi_person_direct_message" + DIRECT_MESSAGE = "direct_message" + + +@tool( + requires_auth=Slack( + scopes=["channels:read", "groups:read", "im:read", "mpim:read"], + ) # Not sent to the model +) +async def list_conversations_metadata( # Tool name and toolkit name is sent to the model + context: ToolContext, # ToolContext is never sent to the model + conversation_types: Annotated[Optional[list[ConversationType]], "The type(s) of conversations to list. Defaults to all types."] = None, # Sent to the model + limit: Annotated[Optional[int], "The maximum number of conversations to list."] = None, # Sent to the model + next_cursor: Annotated[Optional[str], "The cursor to use for pagination." = None, # Sent to the model +) -> Annotated[dict,"The conversations metadata list and a pagination 'next_cursor', if there are more conversations to retrieve."]: # Not sent to the model + """ + List metadata for Slack conversations (channels and/or direct messages) that the user + is a member of. + """ # Docstring is sent to the model + # The body of the function is not sent to the model + ... diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js new file mode 100644 index 0000000..79004ea --- /dev/null +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js @@ -0,0 +1,5 @@ +import Arcade from "@arcadeai/arcadejs"; + +const client = new Arcade(); const listConversationsMetadata = await client.tools.formatted.get( "Slack.ListConversationsMetadata", { format: "openai", }, ); + +console.log(listConversationsMetadata); diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.py b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.py new file mode 100644 index 0000000..c31474c --- /dev/null +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.py @@ -0,0 +1,8 @@ +from arcadepy import Arcade + +client = Arcade() +list_conversations_metadata = client.tools.formatted.get( + name="Slack.ListConversationsMetadata", format="openai" +) + +print(list_conversations_metadata) diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.sh b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.sh new file mode 100644 index 0000000..c3ab097 --- /dev/null +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.sh @@ -0,0 +1,3 @@ +curl -X 'GET' 'https://api.arcade.dev/v1/formatted_tools/Slack.ListConversationsMetadata?format=openai' \ + -H 'Accept: application/json' \ + -H 'Authorization: Bearer {{arcade_api_key}}' diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai_output.json b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai_output.json new file mode 100644 index 0000000..fc50eb5 --- /dev/null +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai_output.json @@ -0,0 +1 @@ +{ "type": "function", "function": { "name": "Slack_ListConversationsMetadata", "description": "List metadata for Slack conversations (channels and/or direct messages) that the user\nis a member of.", "parameters": { "type": "object", "properties": { "conversation_types": { "type": "array", "items": { "type": "string", "enum": [ "public_channel", "private_channel", "multi_person_direct_message", "direct_message" ] }, "description": "The type(s) of conversations to list. Defaults to all types." }, "limit": { "type": "integer", "description": "The maximum number of conversations to list." }, "next_cursor": { "type": "string", "description": "The cursor to use for pagination." } }, "required": [] } } } diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition.py b/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition.py new file mode 100644 index 0000000..cd6e361 --- /dev/null +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition.py @@ -0,0 +1,8 @@ +from arcade_slack.tools.chat import list_conversations_metadata + +from arcade.sdk import ToolCatalog + +tool_def = ToolCatalog.create_tool_definition(list_conversations_metadata, "Slack") +tool_def_json = tool_def.model_dump_json() + +print(tool_def_json) diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition.sh b/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition.sh new file mode 100644 index 0000000..5879d3f --- /dev/null +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition.sh @@ -0,0 +1,3 @@ +curl -X 'GET' 'https://api.arcade.dev/v1/tools/Slack.ListConversationsMetadata' \ + -H 'Accept: application/json' \ + -H 'Authorization: Bearer {{arcade_api_key}}' diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition_output.json b/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition_output.json new file mode 100644 index 0000000..e535132 --- /dev/null +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition_output.json @@ -0,0 +1 @@ +{ "name": "ListConversationsMetadata", "fully_qualified_name": "Slack.ListConversationsMetadata", "description": "List metadata for Slack conversations (channels and/or direct messages) that the user\nis a member of.", "toolkit": { "name": "Slack", "description": null, "version": null }, "input": { "parameters": [ { "name": "conversation_types", "required": false, "description": "The type(s) of conversations to list. Defaults to all types.", "value_schema": { "val_type": "array", "inner_val_type": "string", "enum": [ "public_channel", "private_channel", "multi_person_direct_message", "direct_message" ] }, "inferrable": true }, { "name": "limit", "required": false, "description": "The maximum number of conversations to list.", "value_schema": { "val_type": "integer", "inner_val_type": null, "enum": null }, "inferrable": true }, { "name": "next_cursor", "required": false, "description": "The cursor to use for pagination.", "value_schema": { "val_type": "string", "inner_val_type": null, "enum": null }, "inferrable": true } ] }, "output": { "description": "The conversations metadata list and a pagination 'next_cursor', if there are more conversations to retrieve.", "available_modes": ["value", "error"], "value_schema": { "val_type": "json", "inner_val_type": null, "enum": null } }, "requirements": { "authorization": { "provider_id": "slack", "provider_type": "oauth2", "id": null, "oauth2": { "scopes": ["channels:read", "groups:read", "im:read", "mpim:read"] } } } } diff --git a/pages/home/build-tools/tool-definition.mdx b/pages/home/build-tools/tool-definition.mdx deleted file mode 100644 index bbea323..0000000 --- a/pages/home/build-tools/tool-definition.mdx +++ /dev/null @@ -1,248 +0,0 @@ ---- -title: "What is sent to language models?" -description: "Documentation for what is sent to the language model when defining a tool" ---- - -import { Tabs } from "nextra/components"; - -# Understanding what is sent to language models - -Which parts of your tool definitions are sent to language models? By understanding this, you can ensure that your tools effectively communicate their purpose and usage to the models. - -The Arcade worker converts tools (functions with the `@tool` decorator) into `ToolDefinition` objects. The `ToolDefinition` object can be thought of as the intermediate representation of a tool before it is formatted for a specific model provider and sent to the model. - -## Defining a tool in code - -First, let's take a look at a tool written in Python before it is converted into a `ToolDefinition` object. - -```python -from enum import Enum -from typing import Annotated, Optional - -from arcade.sdk import ToolContext, tool - -class ConversationType(str, Enum): - # Enum values are sent to the model - PUBLIC_CHANNEL = "public_channel" - PRIVATE_CHANNEL = "private_channel" - MULTI_PERSON_DIRECT_MESSAGE = "multi_person_direct_message" - DIRECT_MESSAGE = "direct_message" - -@tool( - requires_auth=Slack( - scopes=["channels:read", "groups:read", "im:read", "mpim:read"], - ) # Not sent to the model -) -async def list_conversations_metadata( # Tool name and toolkit name is sent to the model - context: ToolContext, # ToolContext is never sent to the model - conversation_types: Annotated[Optional[list[ConversationType]], "The type(s) of conversations to list. Defaults to all types."] = None, # Sent to the model - limit: Annotated[Optional[int], "The maximum number of conversations to list."] = None, # Sent to the model - next_cursor: Annotated[Optional[str], "The cursor to use for pagination." = None, # Sent to the model -) -> Annotated[dict,"The conversations metadata list and a pagination 'next_cursor', if there are more conversations to retrieve."]: # Not sent to the model - """ - List metadata for Slack conversations (channels and/or direct messages) that the user - is a member of. - """ # Docstring is sent to the model - # The body of the function is not sent to the model - ... -``` - -## A tool as a `ToolDefinition` object - -As mentioned above, the Arcade worker converts tools into `ToolDefinition` objects. Let's take a look at the `list_conversations_metadata` tool as a `ToolDefinition` object (the intermediate representation of the tool before it is formatted for a specific model). - - - -```python -from arcade_slack.tools.chat import list_conversations_metadata - -from arcade.sdk import ToolCatalog - -tool_def = ToolCatalog.create_tool_definition(list_conversations_metadata, "Slack") -tool_def_json = tool_def.model_dump_json() - -print(tool_def_json) -``` - - - -```bash -curl -X 'GET' 'https://api.arcade.dev/v1/tools/Slack.ListConversationsMetadata' \ - -H 'Accept: application/json' \ - -H 'Authorization: Bearer {{arcade_api_key}}' -``` - - - -
- - - -```json -{ - "name": "ListConversationsMetadata", - "fully_qualified_name": "Slack.ListConversationsMetadata", - "description": "List metadata for Slack conversations (channels and/or direct messages) that the user\nis a member of.", - "toolkit": { "name": "Slack", "description": null, "version": null }, - "input": { - "parameters": [ - { - "name": "conversation_types", - "required": false, - "description": "The type(s) of conversations to list. Defaults to all types.", - "value_schema": { - "val_type": "array", - "inner_val_type": "string", - "enum": [ - "public_channel", - "private_channel", - "multi_person_direct_message", - "direct_message" - ] - }, - "inferrable": true - }, - { - "name": "limit", - "required": false, - "description": "The maximum number of conversations to list.", - "value_schema": { - "val_type": "integer", - "inner_val_type": null, - "enum": null - }, - "inferrable": true - }, - { - "name": "next_cursor", - "required": false, - "description": "The cursor to use for pagination.", - "value_schema": { - "val_type": "string", - "inner_val_type": null, - "enum": null - }, - "inferrable": true - } - ] - }, - "output": { - "description": "The conversations metadata list and a pagination 'next_cursor', if there are more conversations to retrieve.", - "available_modes": ["value", "error"], - "value_schema": { "val_type": "json", "inner_val_type": null, "enum": null } - }, - "requirements": { - "authorization": { - "provider_id": "slack", - "provider_type": "oauth2", - "id": null, - "oauth2": { - "scopes": ["channels:read", "groups:read", "im:read", "mpim:read"] - } - } - } -} -``` - - - -## Formatting tools for model compatibility - -Before tools are sent to models, the `ToolDefinition` object is formatted such that it is compatible with the specific model provider's function calling format. For example, let's take a look at the `list_conversations_metadata` tool formatted for OpenAI. - - - - - -```python -from arcadepy import Arcade - -client = Arcade() -list_conversations_metadata = client.tools.formatted.get( - name="Slack.ListConversationsMetadata", format="openai" -) - -print(list_conversations_metadata) -``` - - - - - -```js -import Arcade from "@arcadeai/arcadejs"; - -const client = new Arcade(); -const listConversationsMetadata = await client.tools.formatted.get( - "Slack.ListConversationsMetadata", - { - format: "openai", - }, -); - -console.log(listConversationsMetadata); -``` - - - - - -```bash -curl -X 'GET' 'https://api.arcade.dev/v1/formatted_tools/Slack.ListConversationsMetadata?format=openai' \ - -H 'Accept: application/json' \ - -H 'Authorization: Bearer {{arcade_api_key}}' -``` - - - - - - - -```json -{ - "type": "function", - "function": { - "name": "Slack_ListConversationsMetadata", - "description": "List metadata for Slack conversations (channels and/or direct messages) that the user\nis a member of.", - "parameters": { - "type": "object", - "properties": { - "conversation_types": { - "type": "array", - "items": { - "type": "string", - "enum": [ - "public_channel", - "private_channel", - "multi_person_direct_message", - "direct_message" - ] - }, - "description": "The type(s) of conversations to list. Defaults to all types." - }, - "limit": { - "type": "integer", - "description": "The maximum number of conversations to list." - }, - "next_cursor": { - "type": "string", - "description": "The cursor to use for pagination." - } - }, - "required": [] - } - } -} -``` - - - -## Additional Information - -- _**Defaults:**_ The default value for parameters listed as optional is not sent to the language model. -- _**Enum Values:**_ If a parameter is an enum, each possible value for the enum is sent to the language model. This provides clarity on the valid options for that parameter. -- _**Annotations:**_ The annotation for each parameter is sent to the language model, thus each parameter must be annotated. -- _**Tool Definition Schema:**_ Check out the [Tool Definition JSON schema](https://github.com/ArcadeAI/arcade-ai/blob/main/schemas/preview/tool_definition.schema.jsonc) for more details on how to define your tools. - -Understanding how a tool is converted into a `ToolDefinition` object and then formatted for a specific model is important for crafting tools that are clear and informative for the language model. diff --git a/pages/home/build-tools/what-is-sent-to-language-models.mdx b/pages/home/build-tools/what-is-sent-to-language-models.mdx new file mode 100644 index 0000000..7cdd30f --- /dev/null +++ b/pages/home/build-tools/what-is-sent-to-language-models.mdx @@ -0,0 +1,94 @@ +--- +title: "What is sent to language models?" +description: "Documentation for what is sent to the language model when defining a tool" +--- + +import { Tabs } from "nextra/components"; + +# Understanding what parts of a tool are sent to language models + +Which parts of your tool definitions are sent to language models? By understanding this, you can ensure that your tools effectively communicate their purpose and usage to the models. + +The Arcade worker converts tools (functions with the `@tool` decorator) into `ToolDefinition` objects. The `ToolDefinition` object can be thought of as the intermediate representation of a tool before it is formatted for a specific model provider and sent to the model. + +## Defining a tool in code + +First, let's take a look at a tool written in Python before it is converted into a `ToolDefinition` object. + +```python file=/examples/code/home/build-tools/what-is-sent-to-language-models/defining_a_tool_in_code.py + +``` + +## A tool as a `ToolDefinition` object + +As mentioned above, the Arcade worker converts tools into `ToolDefinition` objects. Let's take a look at the `list_conversations_metadata` tool as a `ToolDefinition` object (the intermediate representation of the tool before it is formatted for a specific model). + + + +```python file=/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition.py + +``` + + + +```bash file=/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition.sh + +``` + + + +
+ + + +```json file=/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition_output.json + +``` + + + +## Formatting tools for model compatibility + +Before tools are sent to models, the `ToolDefinition` object is formatted such that it is compatible with the specific model provider's function calling format. For example, let's take a look at the `list_conversations_metadata` tool formatted for OpenAI. + + + + + +```python file=/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.py +``` + + + + + +```js file=/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js +``` + + + + + +```bash file=/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.sh +``` + + + + + + + +```json file=/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai_output.json +``` + + + +## Additional Information + +- _**Defaults:**_ The default value for parameters listed as optional is not sent to the language model. +- _**Enum Values:**_ If a parameter is an enum, each possible value for the enum is sent to the language model. This provides clarity on the valid options for that parameter. +- _**Annotations:**_ The annotation for each parameter is sent to the language model, thus each parameter must be annotated. +- _**Tool Definition Schema:**_ Check out the [Tool Definition JSON schema](https://github.com/ArcadeAI/arcade-ai/blob/main/schemas/preview/tool_definition.schema.jsonc) for more details on how to define your tools. + + +In this example, we've learned that tools are converted into `ToolDefinition` objects and then formatted for specific models. Understanding that a tool is converted into a `ToolDefinition` object and then formatted for a specific model is important because it informs you, the tool developer, what parts of the tool are sent to the language model. From 66bc325392d5e621b1397b36668baaf42a7ea7d9 Mon Sep 17 00:00:00 2001 From: Eric Gustin Date: Tue, 28 Jan 2025 15:29:15 -0800 Subject: [PATCH 6/8] Undo really annoying pre-commit formatting that literally commits its formatting changes automatically without telling me --- .../get_formatted_tool_oai_output.json | 35 +++++++++- .../get_tool_definition_output.json | 64 ++++++++++++++++++- 2 files changed, 97 insertions(+), 2 deletions(-) diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai_output.json b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai_output.json index fc50eb5..be26e24 100644 --- a/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai_output.json +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai_output.json @@ -1 +1,34 @@ -{ "type": "function", "function": { "name": "Slack_ListConversationsMetadata", "description": "List metadata for Slack conversations (channels and/or direct messages) that the user\nis a member of.", "parameters": { "type": "object", "properties": { "conversation_types": { "type": "array", "items": { "type": "string", "enum": [ "public_channel", "private_channel", "multi_person_direct_message", "direct_message" ] }, "description": "The type(s) of conversations to list. Defaults to all types." }, "limit": { "type": "integer", "description": "The maximum number of conversations to list." }, "next_cursor": { "type": "string", "description": "The cursor to use for pagination." } }, "required": [] } } } +{ + "type": "function", + "function": { + "name": "Slack_ListConversationsMetadata", + "description": "List metadata for Slack conversations (channels and/or direct messages) that the user\nis a member of.", + "parameters": { + "type": "object", + "properties": { + "conversation_types": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "public_channel", + "private_channel", + "multi_person_direct_message", + "direct_message" + ] + }, + "description": "The type(s) of conversations to list. Defaults to all types." + }, + "limit": { + "type": "integer", + "description": "The maximum number of conversations to list." + }, + "next_cursor": { + "type": "string", + "description": "The cursor to use for pagination." + } + }, + "required": [] + } + } +} diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition_output.json b/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition_output.json index e535132..65a277e 100644 --- a/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition_output.json +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/get_tool_definition_output.json @@ -1 +1,63 @@ -{ "name": "ListConversationsMetadata", "fully_qualified_name": "Slack.ListConversationsMetadata", "description": "List metadata for Slack conversations (channels and/or direct messages) that the user\nis a member of.", "toolkit": { "name": "Slack", "description": null, "version": null }, "input": { "parameters": [ { "name": "conversation_types", "required": false, "description": "The type(s) of conversations to list. Defaults to all types.", "value_schema": { "val_type": "array", "inner_val_type": "string", "enum": [ "public_channel", "private_channel", "multi_person_direct_message", "direct_message" ] }, "inferrable": true }, { "name": "limit", "required": false, "description": "The maximum number of conversations to list.", "value_schema": { "val_type": "integer", "inner_val_type": null, "enum": null }, "inferrable": true }, { "name": "next_cursor", "required": false, "description": "The cursor to use for pagination.", "value_schema": { "val_type": "string", "inner_val_type": null, "enum": null }, "inferrable": true } ] }, "output": { "description": "The conversations metadata list and a pagination 'next_cursor', if there are more conversations to retrieve.", "available_modes": ["value", "error"], "value_schema": { "val_type": "json", "inner_val_type": null, "enum": null } }, "requirements": { "authorization": { "provider_id": "slack", "provider_type": "oauth2", "id": null, "oauth2": { "scopes": ["channels:read", "groups:read", "im:read", "mpim:read"] } } } } +{ + "name": "ListConversationsMetadata", + "fully_qualified_name": "Slack.ListConversationsMetadata", + "description": "List metadata for Slack conversations (channels and/or direct messages) that the user\nis a member of.", + "toolkit": { "name": "Slack", "description": null, "version": null }, + "input": { + "parameters": [ + { + "name": "conversation_types", + "required": false, + "description": "The type(s) of conversations to list. Defaults to all types.", + "value_schema": { + "val_type": "array", + "inner_val_type": "string", + "enum": [ + "public_channel", + "private_channel", + "multi_person_direct_message", + "direct_message" + ] + }, + "inferrable": true + }, + { + "name": "limit", + "required": false, + "description": "The maximum number of conversations to list.", + "value_schema": { + "val_type": "integer", + "inner_val_type": null, + "enum": null + }, + "inferrable": true + }, + { + "name": "next_cursor", + "required": false, + "description": "The cursor to use for pagination.", + "value_schema": { + "val_type": "string", + "inner_val_type": null, + "enum": null + }, + "inferrable": true + } + ] + }, + "output": { + "description": "The conversations metadata list and a pagination 'next_cursor', if there are more conversations to retrieve.", + "available_modes": ["value", "error"], + "value_schema": { "val_type": "json", "inner_val_type": null, "enum": null } + }, + "requirements": { + "authorization": { + "provider_id": "slack", + "provider_type": "oauth2", + "id": null, + "oauth2": { + "scopes": ["channels:read", "groups:read", "im:read", "mpim:read"] + } + } + } +} From fe8e51af16173b005f6e493df7d64ce7491f7443 Mon Sep 17 00:00:00 2001 From: Eric Gustin Date: Tue, 28 Jan 2025 15:33:00 -0800 Subject: [PATCH 7/8] Undo annoying pre-commit formatting --- .../get_formatted_tool_oai.js | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js index 79004ea..6a22e55 100644 --- a/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js @@ -1,5 +1,8 @@ import Arcade from "@arcadeai/arcadejs"; -const client = new Arcade(); const listConversationsMetadata = await client.tools.formatted.get( "Slack.ListConversationsMetadata", { format: "openai", }, ); +const client = new Arcade(); +const listConversationsMetadata = await client.tools.formatted.get("Slack.ListConversationsMetadata", + { format: "openai", } +); console.log(listConversationsMetadata); From 62a9c79eac6f6b430afb2ed9643f90453de6aa24 Mon Sep 17 00:00:00 2001 From: Eric Gustin Date: Tue, 28 Jan 2025 15:39:24 -0800 Subject: [PATCH 8/8] Omg i hate this formatter --- .../what-is-sent-to-language-models/get_formatted_tool_oai.js | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js index 6a22e55..b63bf12 100644 --- a/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js +++ b/examples/code/home/build-tools/what-is-sent-to-language-models/get_formatted_tool_oai.js @@ -1,8 +1,10 @@ import Arcade from "@arcadeai/arcadejs"; const client = new Arcade(); + +// Get what is sent to the language model for a specific tool const listConversationsMetadata = await client.tools.formatted.get("Slack.ListConversationsMetadata", - { format: "openai", } + { format: "openai" } ); console.log(listConversationsMetadata);