Gemini_Sentiment_Analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Gemini Sentiment Analysis Agent Demo\n"
      ],
      "metadata": {
        "id": "DblbGB3twJGQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get API keys\n",
        "\n",
        "Before you can use the Gemini API, you must first obtain an API key. If you don't already have one, create a key with one click in [Google AI Studio](https://makersuite.google.com/app/apikey).  You'll also need an [OpenAI API key](https://openai.com/).\n",
        "\n",
        "Once you have your API keys create the following Google Colab \"secrets\"\n",
        "`GOOGLE_API_KEY` and `OPENAI_API_KEY` which will contain your respective keys."
      ],
      "metadata": {
        "id": "bLPpwWCcxgRG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Initial Setup"
      ],
      "metadata": {
        "id": "AWxOoB_tncv_"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 167,
      "metadata": {
        "id": "81Mlx-E8m93K"
      },
      "outputs": [],
      "source": [
        "!pip install llama-index llama-hub google-generativeai openai pypdf youtube_transcript_api -q"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from google.colab import userdata\n",
        "import openai\n",
        "\n",
        "openai.api_key = userdata.get('OPENAI_API_KEY')"
      ],
      "metadata": {
        "id": "y6dqUiOdniTf"
      },
      "execution_count": 168,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import logging\n",
        "import sys\n",
        "\n",
        "logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True)\n",
        "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
      ],
      "metadata": {
        "id": "-sthewGLnj6Q"
      },
      "execution_count": 169,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Vectorize Content For RAG"
      ],
      "metadata": {
        "id": "T2QlbEhiOHW-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os.path\n",
        "from llama_index import (\n",
        "  VectorStoreIndex,\n",
        "  SimpleDirectoryReader,\n",
        "  StorageContext,\n",
        "  load_index_from_storage,\n",
        "  )\n",
        "from llama_hub.youtube_transcript import YoutubeTranscriptReader\n",
        "\n",
        "# Attempt to load embeddings\n",
        "try:\n",
        "  storage_context = StorageContext.from_defaults(\n",
        "      persist_dir=\"./storage/challenging_child\"\n",
        "  )\n",
        "  challenging_child_index = load_index_from_storage(storage_context)\n",
        "\n",
        "  storage_context = StorageContext.from_defaults(\n",
        "      persist_dir=\"./storage/mindfulness_TB_50\"\n",
        "  )\n",
        "  mindfulness_TB_50_index = load_index_from_storage(storage_context)\n",
        "\n",
        "  storage_context = StorageContext.from_defaults(\n",
        "      persist_dir=\"./storage/mindfulness_TB_relationships\"\n",
        "  )\n",
        "  mindfulness_TB_relationships_index = load_index_from_storage(storage_context)\n",
        "\n",
        "except:\n",
        "  # Need to generate embeddings\n",
        "  challenging_child_docs = SimpleDirectoryReader(\n",
        "      input_files=[\"./data/The_Challenging_Child_Toolbox.pdf\"]\n",
        "  ).load_data(show_progress=True)\n",
        "  mindfulness_TB_50_docs = SimpleDirectoryReader(\n",
        "      input_files=[\"./data/The_Mindfulness_Toolbox__50_Practical_Tips_Tools.pdf\"]\n",
        "  ).load_data(show_progress=True)\n",
        "  mindfulness_TB_relationships_docs = SimpleDirectoryReader(\n",
        "      input_files=[\"./data/The_Mindfulness_Toolbox_for_Relationships.pdf\"]\n",
        "  ).load_data(show_progress=True)\n",
        "\n",
        "  # Build index\n",
        "  challenging_child_index = VectorStoreIndex.from_documents(challenging_child_docs)\n",
        "  mindfulness_TB_50_index = VectorStoreIndex.from_documents(mindfulness_TB_50_docs)\n",
        "  mindfulness_TB_relationships_index = VectorStoreIndex.from_documents(mindfulness_TB_relationships_docs)\n",
        "\n",
        "  # Persist\n",
        "  challenging_child_index.storage_context.persist(persist_dir=\"./storage/challenging_child\")\n",
        "  mindfulness_TB_50_index.storage_context.persist(\"./storage/mindfulness_TB_50\")\n",
        "  mindfulness_TB_relationships_index.storage_context.persist(persist_dir=\"./storage/mindfulness_TB_relationships\")"
      ],
      "metadata": {
        "id": "emt6mmU2WIhQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5afabcb3-13d2-49a8-f380-a8eaf620f73e"
      },
      "execution_count": 170,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:llama_index.indices.loading:Loading all indices.\n",
            "Loading all indices.\n",
            "INFO:llama_index.indices.loading:Loading all indices.\n",
            "Loading all indices.\n",
            "INFO:llama_index.indices.loading:Loading all indices.\n",
            "Loading all indices.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "challenging_child_engine = challenging_child_index.as_query_engine()\n",
        "mindfulness_TB_50_engine = mindfulness_TB_50_index.as_query_engine()\n",
        "mindfulness_TB_relationships_engine = mindfulness_TB_relationships_index.as_query_engine()"
      ],
      "metadata": {
        "id": "CQOjWMl2X0t-"
      },
      "execution_count": 171,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Test to make sure our data loaded"
      ],
      "metadata": {
        "id": "aZwcWP9wS8QZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from IPython.display import Markdown"
      ],
      "metadata": {
        "id": "PpNMYU4mhj4G"
      },
      "execution_count": 172,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# The_Challenging_Child_Toolbox\n",
        "query_engine = challenging_child_index.as_query_engine()\n",
        "response = query_engine.query(\"What is the definition of a challenging child?\")\n",
        "display(Markdown(f\"**Response:**\\n\\n{response}\"))"
      ],
      "metadata": {
        "id": "dtGSbX93f7NC",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 161
        },
        "outputId": "2b7c9003-0289-4e85-c2f9-ea6092432480"
      },
      "execution_count": 173,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Response:**\n\nA challenging child refers to a child who has emotional and behavioral difficulties. These difficulties may require special attention and support from caregivers or professionals working with them."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# The_Mindfulness_Toolbox__50_Practical_Tips_Tools\n",
        "query_engine = mindfulness_TB_50_index.as_query_engine()\n",
        "response = query_engine.query(\"What are the benefits of a mindfulness practice?\")\n",
        "display(Markdown(f\"**Response:**\\n\\n{response}\"))"
      ],
      "metadata": {
        "id": "MnRhG-npgf0h",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 213
        },
        "outputId": "19c117f6-677c-4c8c-c630-de687e0edcdc"
      },
      "execution_count": 174,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Response:**\n\nMindfulness practice has several benefits. It enhances flexibility and adaptability, allowing individuals to break free from old habits and retrain their brain. It cultivates curiosity and greater ease, encouraging exploration of the journey and process rather than being overly focused on the outcome. Mindfulness also changes one's relationship with self-critical and self-blaming thoughts, promoting greater patience, kindness, acceptance, and hospitality towards oneself and others. Additionally, mindfulness encourages greater fulfillment in daily life by focusing on the present moment, reducing rumination and negative thoughts, as well as anxiety about the future."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# The_Mindfulness_Toolbox_for_Relationships\n",
        "query_engine = mindfulness_TB_relationships_index.as_query_engine()\n",
        "response = query_engine.query(\"How does mindfulness relate to relationships?\")\n",
        "display(Markdown(f\"**Response:**\\n\\n{response}\"))"
      ],
      "metadata": {
        "id": "diBQOjNpg98W",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 213
        },
        "outputId": "24c4730e-841d-438b-e9b7-d0d0bcad96c7"
      },
      "execution_count": 175,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Response:**\n\nMindfulness relates to relationships by engaging a sense of curiosity and openness, which helps individuals escape the ego-centered perspective and view relationships in a new way. It allows individuals to relate at a profound level to both their inner and outer environments. By practicing mindfulness, individuals can bring awareness, compassion, and openness into their relationships, leading to a sense of liberation and freedom from old habits and toxic ways of reacting. Mindfulness also provides practical tools for addressing issues that impact daily living and the stresses faced in the real world. Additionally, mindfulness can counteract the challenges posed by technology and social media, which can hinder face-to-face human connection and empathy."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Define Agent Tools"
      ],
      "metadata": {
        "id": "csBNMULgnwGq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Analyze image tool"
      ],
      "metadata": {
        "id": "FVTpWj0gvbwJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.tools import BaseTool, FunctionTool\n",
        "from llama_index.multi_modal_llms.gemini import GeminiMultiModal\n",
        "from llama_index.multi_modal_llms.generic_utils import load_image_urls\n",
        "from typing import List"
      ],
      "metadata": {
        "id": "g4TD2aKDPycm"
      },
      "execution_count": 176,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def analyze_image(img_urls: List[str]) -> str:\n",
        "  \"\"\"Calls our Gemini vision API to analyze the image and return a description of the text contained in the image.\n",
        "      Returns: A string with a description of the image and the mood it conveys if any.\n",
        "\n",
        "  Args:\n",
        "      img_urls (List[str]): The URL of one or more images that convey the users mood\n",
        "  \"\"\"\n",
        "  image_documents = load_image_urls(img_urls)\n",
        "\n",
        "  gemini = GeminiMultiModal(model=\"models/gemini-pro-vision\", api_key=userdata.get('GOOGLE_API_KEY'))\n",
        "\n",
        "  complete_response = gemini.complete(\n",
        "      prompt=\"Identify what you see in the image and what mood it conveys if any\",\n",
        "      image_documents=image_documents,\n",
        "  )\n",
        "\n",
        "  return complete_response\n",
        "\n",
        "vision_tool = FunctionTool.from_defaults(fn=analyze_image)"
      ],
      "metadata": {
        "id": "H4Ei7yzdn3Th"
      },
      "execution_count": 177,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Save session tool"
      ],
      "metadata": {
        "id": "HushMkWP-8bA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def save_session(chat_summary: str) -> bool:\n",
        "  \"\"\"Persists a summary of the user's chat history. Use this tool when the user is happy with your recomendations and done with the session\n",
        "      Returns: A boolean saying if the chat history was persisted\n",
        "\n",
        "  Args:\n",
        "      chat_summary (str): A summary of the chat history, including the users name if it was provided in the chat session\n",
        "  \"\"\"\n",
        "\n",
        "  # Here is where we would persist the chat summary so we can retrive it when we start a new sessions\n",
        "\n",
        "  return True\n",
        "\n",
        "save_tool = FunctionTool.from_defaults(fn=save_session)"
      ],
      "metadata": {
        "id": "eeCU_MQw-_0L"
      },
      "execution_count": 178,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Mindfulness routine recomendation tool"
      ],
      "metadata": {
        "id": "UTwpvzQnP4wn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Use Gemini Pro to examine the transcript of our mindfulness videos"
      ],
      "metadata": {
        "id": "mI1plRvdzbyA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_hub.youtube_transcript import YoutubeTranscriptReader\n",
        "\n",
        "# Download video transcripts\n",
        "loader = YoutubeTranscriptReader()\n",
        "documents = loader.load_data(ytlinks=['https://youtu.be/nPvN1OI7h80?si=iB3TYXI-9Ztc0Qpi',\n",
        "                                      'https://youtu.be/8ZhjZD8rj3E?si=n30dJ0D55tRl9frP',\n",
        "                                      'https://youtu.be/mGoGYu7F2PA?si=qsvemuKLDdlnvC6z',\n",
        "                                      'https://youtu.be/d0VZk3Dd0nw?si=yb3Zr4XGErHSENFB',\n",
        "                                      'https://youtu.be/mme5NC0F7wQ?si=Dd_KECuQCjPJSFXu',\n",
        "                                      'https://youtu.be/iPjFd1eL40Y?si=3RI6N1mw6J3unntk'\n",
        "                                      'https://youtu.be/iPjFd1eL40Y?si=bC0hbYPR0jbVRim9',\n",
        "                                      'https://youtu.be/zVbRxs_QFBA?si=Bn7MC5yG2QcXtHZP',\n",
        "                                      'https://youtu.be/6xTx984jSFc?si=J3rNB2v9OUvAEBsx',\n",
        "                                      'https://youtu.be/1qBbuKWWTGY?si=GO7hyw9za62_V9cB',\n",
        "                                      'https://youtu.be/fj0dh_KxIg4?si=dxVvRm6AC3Ixls1b',\n",
        "                                      'https://youtu.be/4ksAYqRku-s?si=q3cJ7Spvbgi6H2gi',\n",
        "                                      'https://youtu.be/f2ZwrQF6VQM?si=ujQw2fq5Ww3_b9Mv',\n",
        "                                      'https://youtu.be/IL_1DRZDzWc?si=-RutJc161ZO5wgO4'])"
      ],
      "metadata": {
        "id": "3hwyPgxkvRAK"
      },
      "execution_count": 179,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.llms import Gemini\n",
        "\n",
        "videos = {}\n",
        "\n",
        "# Have Gemini analyze the transcript sentiment and who this video is for\n",
        "for document in documents:\n",
        "  response = Gemini(model='models/gemini-pro', api_key=userdata.get('GOOGLE_API_KEY')).complete(f\"Summarize the transcript from this mindfulness video and who should use it: {document.text}\")\n",
        "  videos[document.metadata['video_id']] = response.text\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 958
        },
        "id": "kgS0TeLhwOVX",
        "outputId": "bbca8fbc-a414-4748-ce9c-64d3f20b14fa"
      },
      "execution_count": 180,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 505.96ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 505.96ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4217.35ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4217.35ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 253.96ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 253.96ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4670.73ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4670.73ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 381.43ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 381.43ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 5723.33ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 5723.33ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 202.95ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 202.95ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4860.63ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4860.63ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 380.39ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 380.39ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 5251.13ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 5251.13ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 382.68ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 382.68ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 5428.35ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 5428.35ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 223.72ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 223.72ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4697.53ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4697.53ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 329.28ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 329.28ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4107.04ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4107.04ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 203.34ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 203.34ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 3658.30ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 3658.30ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 254.32ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 254.32ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4086.69ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 4086.69ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 254.28ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 254.28ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 3709.81ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 3709.81ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 229.35ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 229.35ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 5192.02ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 5192.02ms\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 233.26ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 233.26ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 6726.41ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create a tool which recomends a mindfulness routine based on how the user is feeling"
      ],
      "metadata": {
        "id": "cjbxArR-zjEe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "\n",
        "json_string = json.dumps(videos)\n",
        "json_string"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 110
        },
        "id": "G_H-Qs8G39v4",
        "outputId": "bd43644d-be1a-4344-e689-e87d83dc528e"
      },
      "execution_count": 207,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'{\"nPvN1OI7h80\": \"**Summary of the Mindfulness Video:**\\\\n\\\\nThis mindfulness video guides individuals through a visualization exercise to address feelings of loneliness and misunderstanding within families. It encourages self-compassion, self-acceptance, and the recognition of one\\'s unique values and beliefs. The video includes a guided meditation that helps participants connect with their younger selves, acknowledge their feelings of loneliness, and offer comfort and reassurance.\\\\n\\\\n**Who Should Use This Video:**\\\\n\\\\n- Individuals who experience feelings of loneliness and misunderstanding within their families.\\\\n- People who struggle with self-acceptance and self-compassion.\\\\n- Those seeking to connect with their inner selves and explore their emotions.\\\\n- Individuals interested in practicing mindfulness and self-reflection.\\\\n- Anyone looking for a guided meditation to promote emotional healing and self-understanding.\", \"8ZhjZD8rj3E\": \"**Summary of the Mindfulness Video:**\\\\n\\\\nThis mindfulness video guides viewers through a visualization exercise to connect with their younger selves and reflect on their achievements, failures, and the pursuit of success. It emphasizes the importance of finding balance between striving for success and appreciating the present moment.\\\\n\\\\n**Who Should Use This Video:**\\\\n\\\\n- Individuals seeking to reduce stress and anxiety related to achieving success.\\\\n- People who want to cultivate a more balanced and fulfilling approach to their goals.\\\\n- Those looking to connect with their inner selves and gain a deeper understanding of their motivations and values.\\\\n- Anyone interested in practicing mindfulness and self-compassion.\", \"mGoGYu7F2PA\": \"**Summary of the Mindfulness Video:**\\\\n\\\\nThis mindfulness video guides viewers through a journey of self-reflection and release from the need for control. It invites participants to imagine themselves in a vibrant garden, where they encounter a younger version of themselves who is meticulously planning and controlling every aspect of the garden. Through a series of guided prompts, viewers are encouraged to explore moments in their lives where they felt the need to predict outcomes, suppress spontaneity, hold back from delegating, and fear the unknown. They are then guided to release these fears and embrace trust, spontaneity, collaboration, and resilience. The video concludes with a visualization of planting a tree together with the younger self, symbolizing growth without the need for control.\\\\n\\\\n**Who Should Use This Video:**\\\\n\\\\n- Individuals who struggle with the need for control in various aspects of their lives.\\\\n- Those seeking to cultivate greater trust, spontaneity, and resilience.\\\\n- People who want to reduce stress and anxiety caused by the desire to control outcomes.\\\\n- Individuals looking to enhance their creativity and embrace new experiences.\\\\n- Those seeking a guided meditation to promote inner peace and self-acceptance.\", \"d0VZk3Dd0nw\": \"This mindfulness video is designed for individuals who want to heal from past patterns and habits that have been holding them back. It guides viewers through a visualization exercise where they connect with their younger selves and revisit moments where they hesitated, withdrew, or felt exposed. The video encourages viewers to reinterpret these moments, learn from them, and embrace vulnerability as a pathway to authentic connections.\\\\n\\\\nThe video is suitable for anyone who wants to:\\\\n\\\\n- Heal from past patterns and habits\\\\n- Overcome hesitation and fear of expressing themselves\\\\n- Embrace new experiences and step out of their comfort zone\\\\n- Share their true feelings and be vulnerable\\\\n- Seek help and share responsibilities\\\\n- Face life\\'s challenges with courage and grace\\\\n\\\\nThe video provides a guided meditation that helps viewers connect with their younger selves and release old patterns and habits. It also includes affirmations and messages of reassurance to help viewers embrace their strength and resilience.\", \"mme5NC0F7wQ\": \"**Summary of the Mindfulness Video:**\\\\n\\\\nThe video is a guided meditation that takes you on a journey through your past memories to understand the root causes of your procrastination. It helps you to see these memories not as setbacks but as opportunities for growth and self-understanding. By reflecting on these memories and embracing them, you can let go of the fears and anxieties that have been holding you back and move forward with renewed confidence and determination.\\\\n\\\\n**Who Should Use This Video:**\\\\n\\\\n- Individuals who struggle with procrastination and want to understand the underlying causes of their behavior.\\\\n- People who want to develop a deeper understanding of themselves and their past experiences.\\\\n- Those who are looking for a guided meditation to help them let go of negative emotions and limiting beliefs.\\\\n- Individuals who want to cultivate self-compassion and acceptance.\", \"iPjFd1eL40Y\": \"**Summary of the Mindfulness Video:**\\\\n\\\\nThe video guides viewers through a mindfulness meditation to help them cope with feelings of loss and saying goodbye. It begins by inviting viewers to find a comfortable position, close their eyes, and focus on the sensations around them. The narrator then leads viewers through a visualization exercise, where they imagine a younger version of themselves experiencing a difficult goodbye. Viewers are encouraged to offer comfort and support to their younger selves, and to let go of any lingering emotions associated with past farewells. The meditation ends with viewers affirming their strength and resilience in facing future goodbyes.\\\\n\\\\n**Who Should Use This Video:**\\\\n\\\\n- Individuals who are struggling with feelings of loss, grief, or sadness related to saying goodbye.\\\\n- People who are anticipating a difficult goodbye in the near future.\\\\n- Individuals who want to develop greater resilience and coping skills for dealing with life\\'s challenges.\\\\n- Anyone interested in practicing mindfulness and self-compassion.\", \"zVbRxs_QFBA\": \"**Summary of the Mindfulness Video:**\\\\n\\\\nThe video guides viewers through a mindfulness meditation to address emotional challenges in relationships. It encourages self-reflection, acceptance, and healing of past experiences that may have impacted relationship dynamics. The meditation involves visualizing younger and present versions of oneself, along with an ideal secure self, to foster understanding, empathy, and resilience.\\\\n\\\\n**Who Should Use This Video:**\\\\n\\\\n- Individuals seeking to improve their emotional well-being and resilience in relationships.\\\\n- People who have experienced challenges, such as loneliness, betrayal, or trust issues, in their relationships.\\\\n- Those looking to cultivate self-awareness, acceptance, and healing of past experiences that may be affecting their current relationships.\\\\n- Individuals interested in exploring mindfulness as a tool for personal growth and emotional regulation.\", \"6xTx984jSFc\": \"**Summary of the Mindfulness Video:**\\\\n\\\\nThis mindfulness video guides individuals through a journey of self-understanding and healing. It begins by inviting participants to find a comfortable position, close their eyes, and immerse themselves in a serene and tranquil environment. The narrator then guides them through a visualization exercise, where they encounter a younger version of themselves and an older, wiser version of themselves.\\\\n\\\\n**Who Should Use This Video:**\\\\n\\\\n- Individuals seeking to develop self-awareness and understanding.\\\\n- Those looking to explore their emotional experiences and patterns.\\\\n- People who want to cultivate resilience and inner strength.\\\\n- Individuals seeking to heal from past experiences and build healthier relationships.\\\\n- Those interested in practicing mindfulness and meditation for personal growth.\", \"1qBbuKWWTGY\": \"**Summary of the Mindfulness Video:**\\\\n\\\\nThe video guides viewers through a mindfulness exercise aimed at healing suppressed emotions and fostering self-compassion. It involves visualizing a younger version of oneself, acknowledging their suppressed emotions, and providing them with comfort and reassurance. The exercise is accompanied by calming music, gentle rain sounds, and imagery of a cozy room and a rainbow.\\\\n\\\\n**Who Should Use This Video:**\\\\n\\\\n- Individuals who have experienced emotional suppression or difficulty expressing their emotions.\\\\n- Those seeking self-compassion and healing from past emotional experiences.\\\\n- People interested in practicing mindfulness and self-care techniques.\\\\n- Individuals looking for a guided meditation to promote emotional release and healing.\", \"fj0dh_KxIg4\": \"**Summary of the Mindfulness Video:**\\\\n\\\\nThis mindfulness video guides individuals through a visualization exercise to connect with their younger selves and address feelings of isolation, loneliness, and exclusion. It encourages self-compassion, understanding, and acceptance of past experiences and promotes a sense of belonging and connection.\\\\n\\\\n**Who Should Use This Video:**\\\\n\\\\n- Individuals who have experienced feelings of isolation, loneliness, or exclusion in their childhood or teenage years.\\\\n- People who struggle with self-esteem, self-worth, or a sense of belonging.\\\\n- Those seeking to connect with their inner child and heal past emotional wounds.\\\\n- Individuals interested in practicing self-compassion and mindfulness to promote emotional well-being.\", \"4ksAYqRku-s\": \"**Summary of the Mindfulness Video:**\\\\n\\\\nThe video guides viewers through a mindfulness exercise that involves connecting with their younger selves, particularly those who experienced loneliness and isolation during their boarding school days. The exercise aims to provide comfort, understanding, and a sense of belonging to the younger self.\\\\n\\\\n**Who Should Use This Video:**\\\\n\\\\n- Individuals who attended boarding school and experienced loneliness, isolation, or homesickness.\\\\n- People who want to connect with their past selves and gain a deeper understanding of their emotions and experiences.\\\\n- Those seeking comfort, reassurance, and a sense of belonging.\\\\n- Individuals interested in practicing mindfulness and self-compassion.\", \"f2ZwrQF6VQM\": \"**Summary of the Mindfulness Video:**\\\\n\\\\nThe video guides viewers through a mindfulness practice that focuses on healing memories of abandonment and building self-compassion. It begins with relaxation techniques and then invites viewers to travel back in time to a childhood memory of abandonment. The viewer is encouraged to approach the memory with love and compassion, offering comfort and understanding to their younger self. As memories arise, the viewer is guided to transform past hurts and fears into lessons of strength and bridges to healing. The practice concludes with the viewer bringing their younger self into the present moment, knowing that they are always loved and cherished.\\\\n\\\\n**Who Should Use This Video:**\\\\n\\\\n- Individuals who have experienced abandonment in their childhood or past relationships.\\\\n- People who struggle with feelings of loneliness, isolation, or low self-worth.\\\\n- Those seeking to develop self-compassion and resilience.\\\\n- Individuals interested in exploring mindfulness as a tool for healing and personal growth.\", \"IL_1DRZDzWc\": \"**Summary:**\\\\n\\\\nThis mindfulness video guides viewers through a visualization exercise to release the burden of perfectionism and embrace self-acceptance. The narrator invites viewers to imagine their younger selves carrying a backpack filled with rocks, each representing a facet of the desire for perfection. By gently removing these rocks, viewers can release the narratives and expectations that hinder their true expression. The video encourages viewers to give their younger selves a warm hug and express love and pride. A radiant light envelops the child, representing love, acceptance, and joy, symbolizing the infinite possibilities now open to them. The narrator reminds viewers that they are worthy and loved and encourages them to embrace their authentic selves.\\\\n\\\\n**Who Should Use It:**\\\\n\\\\n- Individuals struggling with perfectionism and self-criticism\\\\n- Those seeking to cultivate self-acceptance and self-compassion\\\\n- People looking to reduce stress and anxiety related to perfectionism\\\\n- Individuals interested in mindfulness and meditation practices\\\\n- Anyone seeking to connect with their inner child and heal past experiences\"}'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 207
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "\n",
        "\n",
        "def recomend_mindfulness(feelings_summary: str) -> str:\n",
        "  \"\"\"Recomends a mindfullness routine based on how the user is feeling\n",
        "      Returns: A string with a description of the mindfulness routine it recomends and a link to the youtube video\n",
        "\n",
        "  Args:\n",
        "      feelings_summary (str): A summary of the user's feeings\n",
        "  \"\"\"\n",
        "  videos_string = json.dumps(videos)\n",
        "\n",
        "  response = Gemini(model='models/gemini-pro', api_key=userdata.get('GOOGLE_API_KEY')).complete(f\"\"\"\n",
        "    I'm sending you a json with a list of youtube mindfulness videos\n",
        "    The key is the youtube video id and the value is a summary of the video transcript\n",
        "    from this mindfulness video and who should use it.  The json is here: {videos_string}.\n",
        "    Recomend a video based on my feelings and include the youtube link in the format https://www.youtube.com/watch?v=video_id\n",
        "    Here is a summary of the user's feelings: {feelings_summary}\"\"\")\n",
        "\n",
        "  return response.text\n",
        "\n",
        "recommend_tool = FunctionTool.from_defaults(fn=recomend_mindfulness)"
      ],
      "metadata": {
        "id": "Br0ONXurzv3_"
      },
      "execution_count": 208,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Query engine tool"
      ],
      "metadata": {
        "id": "oUXDfck7vA-P"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.tools import QueryEngineTool, ToolMetadata\n",
        "\n",
        "query_engine_tools = [\n",
        "    QueryEngineTool(\n",
        "        query_engine=challenging_child_engine,\n",
        "        metadata=ToolMetadata(\n",
        "            name=\"challenging_child\",\n",
        "            description=(\n",
        "                \"The Challenging Child Toolbox 75 Mindfulness Based Practices Tools and Tips for Therapists\"\n",
        "                \"Use a detailed plain text question as input to the tool.\"\n",
        "            ),\n",
        "        ),\n",
        "    ),\n",
        "    QueryEngineTool(\n",
        "        query_engine=mindfulness_TB_50_engine,\n",
        "        metadata=ToolMetadata(\n",
        "            name=\"mindfulness_TB_50\",\n",
        "            description=(\n",
        "                \"The Mindfulness Toolbox 50 Practical Tips Tools Handouts for Anxiety Depression Stress and Pain\"\n",
        "                \"Use a detailed plain text question as input to the tool.\"\n",
        "            ),\n",
        "        ),\n",
        "    ),\n",
        "    QueryEngineTool(\n",
        "        query_engine=mindfulness_TB_relationships_engine,\n",
        "        metadata=ToolMetadata(\n",
        "            name=\"mindfulness_TB_relationships\",\n",
        "            description=(\n",
        "                \"The Mindfulness Toolbox for Relationships 50 Practical Tips Tools Handouts for Building Compassionate Connections\"\n",
        "                \"Use a detailed plain text question as input to the tool.\"\n",
        "            ),\n",
        "        ),\n",
        "    ),\n",
        "]"
      ],
      "metadata": {
        "id": "T64SZPMRX3mN"
      },
      "execution_count": 209,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tools = query_engine_tools + [vision_tool, save_tool, recommend_tool]"
      ],
      "metadata": {
        "id": "9a8fSQLRcECy"
      },
      "execution_count": 210,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "YhI-waLZcDy1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create LlamaIndex Agent"
      ],
      "metadata": {
        "id": "GhKWmAL4vou-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### System prompt to give our agent some personality"
      ],
      "metadata": {
        "id": "RIJj4kQ1vuhk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "SYSTEM_PROMPT = \"\"\"You are an emotional support assistant with the expertise of an experienced counselor. Your primary role is to assist the user by encouraging them to provide a drawing that conveys their mood and then recomending a mindfulness routine.\n",
        "You offer professional, friendly, and helpful guidance based on current counseling and mindfulness practices. Once you receive the image, interpret it using your tools to discern what the user might be feeling and confirm with them if your observation is correct.\n",
        "If your interpretation does not align with their feelings, engage in a dialogue until you accurately understand their mood. Your knowledge is exclusively focused on understanding the user's emotions and recommending mindfulness routines using your tools tools, tailored to their mood.\n",
        "Your responses should be limited to one sentence when possible so the user doesn't have to do a lot of reading.  An example response is \"For feels of {insert feelings here} I recomend this mindfulness routine {insert link here}.\n",
        "Thus, you will only provide responses related to these areas. If a question falls outside your area of expertise or if you lack the necessary information, you will inform the user by saying,\n",
        "'Sorry, I do not know the answer to your question.' and then prompt for more information related to their feelings.\"\"\""
      ],
      "metadata": {
        "id": "VfeiB-LV6Xte"
      },
      "execution_count": 211,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Init our agent"
      ],
      "metadata": {
        "id": "TwMF84Anv4YB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using OpenAIAgent"
      ],
      "metadata": {
        "id": "2nUxQRXEqHmE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.agent import OpenAIAgent\n",
        "from llama_index.llms import OpenAI\n",
        "\n",
        "llm = OpenAI(model=\"gpt-4-1106-preview\") # Using GPT-4 Turbo (Beta)\n",
        "\n",
        "agent = OpenAIAgent.from_tools(\n",
        "    tools,\n",
        "    llm=llm,\n",
        "    verbose=False,\n",
        "    system_prompt=SYSTEM_PROMPT,\n",
        ")"
      ],
      "metadata": {
        "id": "db-7B9CXv7Bc"
      },
      "execution_count": 212,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Alternative approach using ReActAgent so we can use Gemini as our LLM.  ReActAgent is designed to allow alternatives to OpenAI.\n",
        "\n",
        "NOTE: This never worked reliably with Gemini because it's not conversational.  If you say 'Hi' it trys to ue a tool rather than just chatting.  Would be interesting to try it with other LLMs.  I tried using GPT-4 Turbo and it seemed to work."
      ],
      "metadata": {
        "id": "64JZmbdqqI07"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# from llama_index.llms import Gemini, ChatMessage\n",
        "# from llama_index.agent import AgentRunner, ReActAgentWorker, ReActAgent\n",
        "\n",
        "# llm = Gemini(model=\"models/gemini-pro\", api_key=userdata.get('GOOGLE_API_KEY'), temperature=0.1)\n",
        "# agent = ReActAgent.from_tools(tools, llm=llm, verbose=True, chat_history=[ChatMessage(role=\"user\", content=SYSTEM_PROMPT)])"
      ],
      "metadata": {
        "id": "dUQk-hzTjxbq"
      },
      "execution_count": 187,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Test Our Agent"
      ],
      "metadata": {
        "id": "y_nLfwGjwDPV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "I'm picturing the frontend give the user the option to chat or attach one or more images that would get uploaded to cloud storage.  Then our back could prompt the agent with something like \"Here's a drawing of how I'm feeling {Image URL Here}.\"\n",
        "\n",
        "For the demo below you need to manually paste the image URL into the chat (i.e. https://ih1.redbubble.net/image.3636044620.1142/bg,f8f8f8-flat,750x,075,f-pad,750x1000,f8f8f8.u5.jpg)"
      ],
      "metadata": {
        "id": "99sOLlPn6no4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Single prompt example"
      ],
      "metadata": {
        "id": "YyuY5bt-8Ono"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "response = agent.chat(\"hi I'm Andrew\")\n",
        "display(Markdown(f\"**Response:**\\n\\n{response}\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        },
        "id": "hhaRx6hk72l0",
        "outputId": "503b5f3c-cae2-491b-a702-3c33dfe491a4"
      },
      "execution_count": 213,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Response:**\n\nHello Andrew! How can I assist you today? If you're feeling up to it, you can share a drawing that conveys your mood, and I can help you with a mindfulness routine based on that."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Multi-turn chat example"
      ],
      "metadata": {
        "id": "yRpcVdr68Sp3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "response = agent.chat_repl()"
      ],
      "metadata": {
        "id": "bkLhhPGfwFdx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 701
        },
        "outputId": "744e4d62-037c-4f3f-bd89-70e77b82b6d3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "===== Entering Chat REPL =====\n",
            "Type \"exit\" to exit.\n",
            "\n",
            "Human: https://ih1.redbubble.net/image.3636044620.1142/bg,f8f8f8-flat,750x,075,f-pad,750x1000,f8f8f8.u5.jpg\n",
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro-vision?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 481.80ms\n",
            "200 GET /v1beta/models/gemini-pro-vision?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 481.80ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro-vision:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 13925.80ms\n",
            "200 POST /v1beta/models/gemini-pro-vision:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 13925.80ms\n",
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "Assistant: The drawing suggests you might be feeling sad or lonely; is that correct?\n",
            "\n",
            "Human: yes\n",
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 328.61ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 328.61ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 5087.89ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 5087.89ms\n",
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "Assistant: For feelings of sadness or loneliness, I recommend this mindfulness routine: [Guided Meditation for Loneliness](https://www.youtube.com/watch?v=nPvN1OI7h80).\n",
            "\n",
            "Human: what about for stress?\n",
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "INFO:tornado.access:200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 329.66ms\n",
            "200 GET /v1beta/models/gemini-pro?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 329.66ms\n",
            "INFO:tornado.access:200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 6347.23ms\n",
            "200 POST /v1beta/models/gemini-pro:generateContent?%24alt=json%3Benum-encoding%3Dint (127.0.0.1) 6347.23ms\n",
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "Assistant: For feelings of stress, I recommend this mindfulness routine: [Release the Burden of Perfectionism and Embrace Self-Acceptance](https://www.youtube.com/watch?v=IL_1DRZDzWc).\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Reset Agent State"
      ],
      "metadata": {
        "id": "v5ThlCEJGVJr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The agent's knowledge is based on it's chat history which we can easily reset."
      ],
      "metadata": {
        "id": "SGlyGG7o2m4N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "agent.reset()\n",
        "agent.chat_history"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "u9igwvqB2tpq",
        "outputId": "10829a98-2bf9-44e7-b7d9-cfff04d1de30"
      },
      "execution_count": 190,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[]"
            ]
          },
          "metadata": {},
          "execution_count": 190
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If we ask it our name it shouldn't know"
      ],
      "metadata": {
        "id": "3RA_FNyQ2ydF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "response = agent.chat(\"whats my name?\")\n",
        "display(Markdown(f\"**Response:**\\n\\n{response}\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 107
        },
        "id": "f8-uGSnp1GiJ",
        "outputId": "534f16bf-25bc-4693-ac7c-b98a87340996"
      },
      "execution_count": 191,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Response:**\n\nSorry, I do not know the answer to your question. Could you share with me how you're feeling today or provide a drawing that conveys your mood?"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's tell it our name and ask again"
      ],
      "metadata": {
        "id": "CpStTxTe3IVZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "response = agent.chat(\"hi I'm Andrew\")\n",
        "display(Markdown(f\"**Response:**\\n\\n{response}\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 107
        },
        "id": "Cp7UQTcx3DGS",
        "outputId": "9bfe6b61-f7da-44fe-893e-2219ed59d9fa"
      },
      "execution_count": 192,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Response:**\n\nHello Andrew! How are you feeling today? If you'd like, you can express your mood by sharing a drawing with me."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "response = agent.chat(\"whats my name?\")\n",
        "display(Markdown(f\"**Response:**\\n\\n{response}\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 107
        },
        "id": "7ZlF9H7j3MEf",
        "outputId": "b4763d48-7b7f-480f-b8b0-9955c8d471c0"
      },
      "execution_count": 193,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Response:**\n\nYour name is Andrew. How can I assist you with your emotions today, Andrew? Would you like to share a drawing that reflects how you're feeling?"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "It knows our name and we can see the chat history"
      ],
      "metadata": {
        "id": "c_iCOq6D3Pck"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "agent.chat_history"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aJyMnPUu2f2E",
        "outputId": "36deea0b-5beb-4def-9b73-0128bc240b24"
      },
      "execution_count": 194,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[ChatMessage(role=<MessageRole.USER: 'user'>, content='whats my name?', additional_kwargs={}),\n",
              " ChatMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=\"Sorry, I do not know the answer to your question. Could you share with me how you're feeling today or provide a drawing that conveys your mood?\", additional_kwargs={}),\n",
              " ChatMessage(role=<MessageRole.USER: 'user'>, content=\"hi I'm Andrew\", additional_kwargs={}),\n",
              " ChatMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=\"Hello Andrew! How are you feeling today? If you'd like, you can express your mood by sharing a drawing with me.\", additional_kwargs={}),\n",
              " ChatMessage(role=<MessageRole.USER: 'user'>, content='whats my name?', additional_kwargs={}),\n",
              " ChatMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=\"Your name is Andrew. How can I assist you with your emotions today, Andrew? Would you like to share a drawing that reflects how you're feeling?\", additional_kwargs={})]"
            ]
          },
          "metadata": {},
          "execution_count": 194
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As a sanity check lets reset the chat history one more time and ask it our name again."
      ],
      "metadata": {
        "id": "FsBffhY-3XTU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "agent.reset()\n",
        "agent.chat_history"
      ],
      "metadata": {
        "id": "dlsyjDjIGX8V",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0757d0f6-3ea6-409f-e383-0d37a7fd7ef7"
      },
      "execution_count": 195,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[]"
            ]
          },
          "metadata": {},
          "execution_count": 195
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "response = agent.chat(\"whats my name?\")\n",
        "display(Markdown(f\"**Response:**\\n\\n{response}\"))"
      ],
      "metadata": {
        "id": "gCKcWi2lGb7g",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        },
        "outputId": "960a6f6e-73a9-4076-ed4a-3d46bc149a45"
      },
      "execution_count": 196,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Response:**\n\nSorry, I do not know the answer to your question. If you're feeling up to it, would you like to share a drawing that conveys your mood? This can help me understand how you're feeling and recommend a mindfulness routine tailored to you."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Handle Returning User"
      ],
      "metadata": {
        "id": "K5cQa1zFGiZT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "When returning to the conversation we can start with a chat history, even if the agent was reset, so the agent knows what we talked about last.\n",
        "\n",
        "As a sanity check lets first reset out chat history and make sure it doesn't remember us."
      ],
      "metadata": {
        "id": "oKZZKYYDGq3Y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "agent.reset()\n",
        "agent.chat_history"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b-Zs0mMR8uj5",
        "outputId": "2eb44ae5-5c93-4aa8-cf0d-4594a63bc7bd"
      },
      "execution_count": 197,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[]"
            ]
          },
          "metadata": {},
          "execution_count": 197
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "response = agent.chat(\"whats my name?\")\n",
        "display(Markdown(f\"**Response:**\\n\\n{response}\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        },
        "id": "uP0Ial_z1Q6b",
        "outputId": "92437685-068b-441f-f44a-7dc8ab59ab5a"
      },
      "execution_count": 198,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Response:**\n\nSorry, I do not know the answer to your question. Could you share with me how you're feeling today? If you're comfortable, you can express your mood through a drawing and share it with me."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "It doesn't know our name.  Now let's pass a chat history with our name and a summary of our last session that we persisted when the last session ended."
      ],
      "metadata": {
        "id": "PP4cJGk86NIz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.llms import ChatMessage\n",
        "\n",
        "persisted_chat_history = \"Andrew shared a drawing that conveyed feelings of sadness and loneliness. After confirming his mood, a mindfulness exercise was recommended to help him feel more grounded and connected.\"\n",
        "chat_history = [ChatMessage(role=\"user\", content=persisted_chat_history, additional_kwargs={})]"
      ],
      "metadata": {
        "id": "1FaROq694Bqg"
      },
      "execution_count": 199,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "response = agent.chat(\"I'm a returning user.  Address me by name and ask how I'm doing since we last spoke.  Also reiterate how I was last feeling so I know you understand.\", chat_history=chat_history)\n",
        "display(Markdown(f\"**Response:**\\n\\n{response}\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        },
        "id": "VVxq_YyB4YQV",
        "outputId": "defd2a86-4ff9-4a8a-8a91-c68b7df42121"
      },
      "execution_count": 200,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
            "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Response:**\n\nHello Andrew, it's good to see you back. How have you been doing since we last spoke? Last time, you were feeling sadness and loneliness. Have you noticed any changes in your mood since then?"
          },
          "metadata": {}
        }
      ]
    }
  ]
}