Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use genai.protos to access the raw protos. #446

Merged
merged 5 commits into from
Jun 7, 2024
+51 −81
Merged

Use genai.protos to access the raw protos. #446

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b9174a0
replace glm reference with genai.protos
MarkDaoust Jun 3, 2024
ac321cc
fix links
MarkDaoust Jun 3, 2024
3f89dcc
Fix more glm references
MarkDaoust Jun 7, 2024
bd3b159
genai.protos
MarkDaoust Jun 7, 2024
460cf81
Generative Language API -> Gemini API
MarkDaoust Jun 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion examples/gemini/python/vectordb_with_chroma/vectordb_with_chroma.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,6 @@
"import pandas as pd\n",
"\n",
"import google.generativeai as genai\n",
"import google.ai.generativelanguage as glm\n",
"\n",
"# Used to securely store your API key\n",
"from google.colab import userdata\n",
Expand Down
9 changes: 4 additions & 5 deletions site/en/docs/search_reranking_using_embeddings.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,6 @@
"import textwrap\n",
"\n",
"import google.generativeai as genai\n",
"import google.ai.generativelanguage as glm\n",
"\n",
"import wikipedia\n",
"from wikipedia.exceptions import DisambiguationError, PageError\n",
Expand Down Expand Up @@ -821,7 +820,7 @@
"In the chat history you can see all 4 steps:\n",
"\n",
"1. The user sent the query.\n",
"2. The model replied with a `glm.FunctionCall` calling the `wikipedia_search` with a number of relevant searches.\n",
"2. The model replied with a `genai.protos.FunctionCall` calling the `wikipedia_search` with a number of relevant searches.\n",
"3. Because you set `enable_automatic_function_calling=True` when creating the `genai.ChatSession`, it executed the search function and returned the list of article summaries to the model.\n",
"4. Folliwing the instructions in the prompt, the model generated a final answer based on those summaries.\n"
]
Expand Down Expand Up @@ -1044,9 +1043,9 @@
],
"source": [
"response = chat.send_message(\n",
" glm.Content(\n",
" parts=[glm.Part(\n",
" function_response = glm.FunctionResponse(\n",
" genai.protos.Content(\n",
" parts=[genai.protos.Part(\n",
" function_response = genai.protos.FunctionResponse(\n",
" name='wikipedia_search',\n",
" response={'result': summaries}\n",
" )\n",
Expand Down
56 changes: 22 additions & 34 deletions site/en/gemini-api/docs/function-calling/python.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,6 @@
"import time\n",
"\n",
"import google.generativeai as genai\n",
"import google.ai.generativelanguage as glm\n",
"\n",
"from IPython import display\n",
"from IPython.display import Markdown\n",
Expand Down Expand Up @@ -206,7 +205,7 @@
"\n",
"To use function calling, pass a list of functions to the `tools` parameter when creating a [`GenerativeModel`](https://ai.google.dev/api/python/google/generativeai/GenerativeModel). The model uses the function name, docstring, parameters, and parameter type annotations to decide if it needs the function to best answer a prompt.\n",
"\n",
"> Important: The SDK converts function parameter type annotations to a format the API understands (`glm.FunctionDeclaration`). The API only supports a limited selection of parameter types, and the Python SDK's automatic conversion only supports a subset of that: `AllowedTypes = int | float | bool | str | list['AllowedTypes'] | dict`"
"> Important: The SDK converts function parameter type annotations to a format the API understands (`genai.protos.FunctionDeclaration`). The API only supports a limited selection of parameter types, and the Python SDK's automatic conversion only supports a subset of that: `AllowedTypes = int | float | bool | str | list['AllowedTypes'] | dict`"
]
},
{
Expand Down Expand Up @@ -327,13 +326,13 @@
"source": [
"Examine the chat history to see the flow of the conversation and how function calls are integrated within it.\n",
"\n",
"The `ChatSession.history` property stores a chronological record of the conversation between the user and the Gemini model. Each turn in the conversation is represented by a [`glm.Content`](https://ai.google.dev/api/python/google/ai/generativelanguage/Content) object, which contains the following information:\n",
"The `ChatSession.history` property stores a chronological record of the conversation between the user and the Gemini model. Each turn in the conversation is represented by a [`genai.protos.Content`](https://ai.google.dev/api/python/google/generativeai/protos/Content) object, which contains the following information:\n",
"\n",
"* **Role**: Identifies whether the content originated from the \"user\" or the \"model\".\n",
"* **Parts**: A list of [`glm.Part`](https://ai.google.dev/api/python/google/ai/generativelanguage/Part) objects that represent individual components of the message. With a text-only model, these parts can be:\n",
"* **Parts**: A list of [`genai.protos.Part`](https://ai.google.dev/api/python/google/generativeai/protos/Part) objects that represent individual components of the message. With a text-only model, these parts can be:\n",
" * **Text**: Plain text messages.\n",
" * **Function Call** ([`glm.FunctionCall`](https://ai.google.dev/api/python/google/ai/generativelanguage/FunctionCall)): A request from the model to execute a specific function with provided arguments.\n",
" * **Function Response** ([`glm.FunctionResponse`](https://ai.google.dev/api/python/google/ai/generativelanguage/FunctionResponse)): The result returned by the user after executing the requested function.\n",
" * **Function Call** ([`genai.protos.FunctionCall`](https://ai.google.dev/api/python/google/generativeai/protos/FunctionCall)): A request from the model to execute a specific function with provided arguments.\n",
" * **Function Response** ([`genai.protos.FunctionResponse`](https://ai.google.dev/api/python/google/generativeai/protos/FunctionResponse)): The result returned by the user after executing the requested function.\n",
"\n",
" In the previous example with the mittens calculation, the history shows the following sequence:\n",
"\n",
Expand Down Expand Up @@ -400,7 +399,7 @@
"source": [
"While this was all handled automatically, if you need more control, you can:\n",
"\n",
"- Leave the default `enable_automatic_function_calling=False` and process the `glm.FunctionCall` responses yourself.\n",
"- Leave the default `enable_automatic_function_calling=False` and process the `genai.protos.FunctionCall` responses yourself.\n",
"- Or use `GenerativeModel.generate_content`, where you also need to manage the chat history."
]
},
Expand Down Expand Up @@ -541,7 +540,7 @@
"\n",
"# Build the response parts.\n",
"response_parts = [\n",
" glm.Part(function_response=glm.FunctionResponse(name=fn, response={\"result\": val}))\n",
" genai.protos.Part(function_response=genai.protos.FunctionResponse(name=fn, response={\"result\": val}))\n",
" for fn, val in responses.items()\n",
"]\n",
"\n",
Expand Down Expand Up @@ -570,18 +569,7 @@
"AllowedType = (int | float | bool | str | list['AllowedType'] | dict[str, AllowedType]\n",
"```\n",
"\n",
"The `google.ai.generativelanguage` client library provides access to the low level types giving you full control."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "S53E0EE8TBUF"
},
"outputs": [],
"source": [
"import google.ai.generativelanguage as glm"
"The `google.generativeai.protos` submodule provides access to the low level types giving you full control."
]
},
{
Expand Down Expand Up @@ -648,7 +636,7 @@
"id": "qFD4U7ym04F5"
},
"source": [
"This returns the list of `glm.Tool` objects that would be sent to the API. If the printed format is not familiar, it's because these are Google protobuf classes. Each `glm.Tool` (1 in this case) contains a list of `glm.FunctionDeclarations`, which describe a function and its arguments."
"This returns the list of `genai.protos.Tool` objects that would be sent to the API. If the printed format is not familiar, it's because these are Google protobuf classes. Each `genai.protos.Tool` (1 in this case) contains a list of `genai.protos.FunctionDeclarations`, which describe a function and its arguments."
]
},
{
Expand All @@ -657,7 +645,7 @@
"id": "eY6RmFQ76FVu"
},
"source": [
"Here is a declaration for the same multiply function written using the `glm` classes.\n",
"Here is a declaration for the same multiply function written using the `genai.protos` classes.\n",
"\n",
"Note that these classes just describe the function for the API, they don't include an implementation of it. So using this doesn't work with automatic function calling, but functions don't always need an implementation."
]
Expand All @@ -670,16 +658,16 @@
},
"outputs": [],
"source": [
"calculator = glm.Tool(\n",
"calculator = genai.protos.Tool(\n",
" function_declarations=[\n",
" glm.FunctionDeclaration(\n",
" genai.protos.FunctionDeclaration(\n",
" name='multiply',\n",
" description=\"Returns the product of two numbers.\",\n",
" parameters=glm.Schema(\n",
" type=glm.Type.OBJECT,\n",
" parameters=genai.protos.Schema(\n",
" type=genai.protos.Type.OBJECT,\n",
" properties={\n",
" 'a':glm.Schema(type=glm.Type.NUMBER),\n",
" 'b':glm.Schema(type=glm.Type.NUMBER)\n",
" 'a':genai.protos.Schema(type=genai.protos.Type.NUMBER),\n",
" 'b':genai.protos.Schema(type=genai.protos.Type.NUMBER)\n",
" },\n",
" required=['a','b']\n",
" )\n",
Expand Down Expand Up @@ -753,7 +741,7 @@
}
],
"source": [
"glm.Tool(calculator)"
"genai.protos.Tool(calculator)"
]
},
{
Expand All @@ -762,7 +750,7 @@
"id": "jS6ruiTp6VBf"
},
"source": [
"Either way, you pass a representation of a `glm.Tool` or list of tools to"
"Either way, you pass a representation of a `genai.protos.Tool` or list of tools to"
]
},
{
Expand All @@ -787,7 +775,7 @@
"id": "517ca06297bb"
},
"source": [
"Like before the model returns a `glm.FunctionCall` invoking the calculator's `multiply` function:"
"Like before the model returns a `genai.protos.FunctionCall` invoking the calculator's `multiply` function:"
]
},
{
Expand Down Expand Up @@ -889,9 +877,9 @@
"outputs": [],
"source": [
"response = chat.send_message(\n",
" glm.Content(\n",
" parts=[glm.Part(\n",
" function_response = glm.FunctionResponse(\n",
" genai.protos.Content(\n",
" parts=[genai.protos.Part(\n",
" function_response = genai.protos.FunctionResponse(\n",
" name='multiply',\n",
" response={'result': result}))]))"
]
Expand Down
41 changes: 15 additions & 26 deletions site/en/gemini-api/docs/get-started/python.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -419,7 +419,7 @@
"source": [
"Gemini can generate multiple possible responses for a single prompt. These possible responses are called `candidates`, and you can review them to select the most suitable one as the response.\n",
"\n",
"View the response candidates with <a href=\"https://ai.google.dev/api/python/google/ai/generativelanguage/GenerateContentResponse#candidates\"><code>GenerateContentResponse.candidates</code></a>:"
"View the response candidates with <a href=\"https://ai.google.dev/api/python/google/generativeai/protos/GenerateContentResponse#candidates\"><code>GenerateContentResponse.candidates</code></a>:"
]
},
{
Expand Down Expand Up @@ -957,7 +957,7 @@
"id": "AwCqtZ6D4kvk"
},
"source": [
"`glm.Content` objects contain a list of `glm.Part` objects that each contain either a text (string) or inline_data (`glm.Blob`), where a blob contains binary data and a `mime_type`. The chat history is available as a list of `glm.Content` objects in `ChatSession.history`:"
"`genai.protos.Content` objects contain a list of `genai.protos.Part` objects that each contain either a text (string) or inline_data (`genai.protos.Blob`), where a blob contains binary data and a `mime_type`. The chat history is available as a list of `genai.protos.Content` objects in `ChatSession.history`:"
]
},
{
Expand Down Expand Up @@ -1033,7 +1033,7 @@
"source": [
"## Count tokens\n",
"\n",
"Large language models have a context window, and the context length is often measured in terms of the **number of tokens**. With the Gemini API, you can determine the number of tokens per any `glm.Content` object. In the simplest case, you can pass a query string to the `GenerativeModel.count_tokens` method as follows:"
"Large language models have a context window, and the context length is often measured in terms of the **number of tokens**. With the Gemini API, you can determine the number of tokens per any `genai.protos.Content` object. In the simplest case, you can pass a query string to the `GenerativeModel.count_tokens` method as follows:"
]
},
{
Expand Down Expand Up @@ -1188,9 +1188,9 @@
"id": "zBg0eNeml3d4"
},
"source": [
"While the `genai.embed_content` function accepts simple strings or lists of strings, it is actually built around the `glm.Content` type (like <a href=\"https://ai.google.dev/api/python/google/generativeai/GenerativeModel#generate_content\"><code>GenerativeModel.generate_content</code></a>). `glm.Content` objects are the primary units of conversation in the API.\n",
"While the `genai.embed_content` function accepts simple strings or lists of strings, it is actually built around the `genai.protos.Content` type (like <a href=\"https://ai.google.dev/api/python/google/generativeai/GenerativeModel#generate_content\"><code>GenerativeModel.generate_content</code></a>). `genai.protos.Content` objects are the primary units of conversation in the API.\n",
"\n",
"While the `glm.Content` object is multimodal, the `embed_content` method only supports text embeddings. This design gives the API the *possibility* to expand to multimodal embeddings."
"While the `genai.protos.Content` object is multimodal, the `embed_content` method only supports text embeddings. This design gives the API the *possibility* to expand to multimodal embeddings."
]
},
{
Expand Down Expand Up @@ -1248,7 +1248,7 @@
"id": "jU8juHCxoUKG"
},
"source": [
"Similarly, the chat history contains a list of `glm.Content` objects, which you can pass directly to the `embed_content` function:"
"Similarly, the chat history contains a list of `genai.protos.Content` objects, which you can pass directly to the `embed_content` function:"
]
},
{
Expand Down Expand Up @@ -1488,18 +1488,7 @@
"id": "-fthdIItnqki"
},
"source": [
"Underlying the Python SDK is the <a href=\"https://ai.google.dev/api/python/google/ai/generativelanguage\"><code>google.ai.generativelanguage</code></a> client library:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "l6aafWECnpX6"
},
"outputs": [],
"source": [
"import google.ai.generativelanguage as glm"
"The [`google.generativeai.protos`](https://ai.google.dev/api/python/google/generativeai/protos) submodule provides access to the low level classes used by the API behind the scenes:"
]
},
{
Expand All @@ -1508,10 +1497,10 @@
"id": "gm1RWcB3n_n0"
},
"source": [
"The SDK attempts to convert your message to a `glm.Content` object, which contains a list of `glm.Part` objects that each contain either:\n",
"The SDK attempts to convert your message to a `genai.protos.Content` object, which contains a list of `genai.protos.Part` objects that each contain either:\n",
"\n",
"1. a <a href=\"https://www.tensorflow.org/text/api_docs/python/text\"><code>text</code></a> (string)\n",
"2. `inline_data` (`glm.Blob`), where a blob contains binary `data` and a `mime_type`.\n",
"2. `inline_data` (`genai.protos.Blob`), where a blob contains binary `data` and a `mime_type`.\n",
"\n",
"You can also pass any of these classes as an equivalent dictionary.\n",
"\n",
Expand All @@ -1530,11 +1519,11 @@
"source": [
"model = genai.GenerativeModel('gemini-1.5-flash')\n",
"response = model.generate_content(\n",
" glm.Content(\n",
" genai.protos.Content(\n",
" parts = [\n",
" glm.Part(text=\"Write a short, engaging blog post based on this picture.\"),\n",
" glm.Part(\n",
" inline_data=glm.Blob(\n",
" genai.protos.Part(text=\"Write a short, engaging blog post based on this picture.\"),\n",
" genai.protos.Part(\n",
" inline_data=genai.protos.Blob(\n",
" mime_type='image/jpeg',\n",
" data=pathlib.Path('image.jpg').read_bytes()\n",
" )\n",
Expand Down Expand Up @@ -1581,9 +1570,9 @@
"\n",
"While the `genai.ChatSession` class shown earlier can handle many use cases, it does make some assumptions. If your use case doesn't fit into this chat implementation it's good to remember that `genai.ChatSession` is just a wrapper around <a href=\"https://ai.google.dev/api/python/google/generativeai/GenerativeModel#generate_content\"><code>GenerativeModel.generate_content</code></a>. In addition to single requests, it can handle multi-turn conversations.\n",
"\n",
"The individual messages are `glm.Content` objects or compatible dictionaries, as seen in previous sections. As a dictionary, the message requires `role` and `parts` keys. The `role` in a conversation can either be the `user`, which provides the prompts, or `model`, which provides the responses.\n",
"The individual messages are `genai.protos.Content` objects or compatible dictionaries, as seen in previous sections. As a dictionary, the message requires `role` and `parts` keys. The `role` in a conversation can either be the `user`, which provides the prompts, or `model`, which provides the responses.\n",
"\n",
"Pass a list of `glm.Content` objects and it will be treated as multi-turn chat:"
"Pass a list of `genai.protos.Content` objects and it will be treated as multi-turn chat:"
]
},
{
Expand Down
Loading
Loading