notebooks/rosa-qa-experiments/rosa-demo-open-ai-wfaq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "46992c86-b56c-446c-a86e-3b8ba9b9f383",
   "metadata": {},
   "source": [
    "# ROSA QA demo\n",
    "This notebook demos our experiments with Question answering for documentation search with ROSA. The training set here all the public documents available. The results here reflect how the models will perform in deployment when it has access to all the data. For validation, we use a FAQ document present in the documentation dataset for ROSA.  This exercise reveals how the models will perform if they have access to the exact answer.  You will need to provide the api key as the `OPENAI_API_KEY` environment variable to execute this notebook the open-ai api based llms. You can place the key in a `credentials.env` at the root of this repository and this notebook will load it in the following cell. For an example, look at the `credentials_example.env` file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd41c9a6",
   "metadata": {},
   "source": [
    "## Import dependencies and credentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c0cc37b2-3872-4357-90ff-432e665d2dc0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/fm/lib/python3.10/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "# Import package\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "import re\n",
    "from sklearn.metrics import f1_score\n",
    "import nltk\n",
    "\n",
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "from langchain.text_splitter import MarkdownTextSplitter\n",
    "from langchain.vectorstores import Chroma\n",
    "from langchain.docstore.document import Document\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.chains.question_answering import load_qa_chain\n",
    "from langchain.llms import OpenAI\n",
    "from langchain.document_loaders import TextLoader, DirectoryLoader\n",
    "from langchain.chains.qa_with_sources import load_qa_with_sources_chain\n",
    "from langchain.prompts.prompt import PromptTemplate\n",
    "\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "import pandas as pd\n",
    "import time\n",
    "pd.set_option('display.max_colwidth', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "756b5eb5-05dc-4343-a60a-815eccaa2826",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "load_dotenv(find_dotenv(\"credentials.env\"), override=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ed950b1-cfb5-46ed-98fa-984081344254",
   "metadata": {},
   "source": [
    "## Load all the documents in a vectorstore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "555178f7-0ebd-49ef-b7c4-cb8b1049dec4",
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = DirectoryLoader('../data/external', glob=\"**/*.md\", loader_cls=TextLoader)\n",
    "documents = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "686fea83-abdb-4571-bc6f-ba0c96cc3702",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Verification\n",
    "'../data/external/rosaworkshop/14-faq.md' in [i.metadata['source'] for i in documents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "01d4516c-b7e8-44d2-9bcf-c24e1b6fc102",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Split the documents into chunks. Is there a better way than hardcoding size as 1000?\n",
    "text_splitter = MarkdownTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "texts = text_splitter.split_documents(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "b9a44a23-0f1f-4c62-8ddf-6412ce423eb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = OpenAIEmbeddings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "019cac62-823a-49a5-82a7-b4273d0051b7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Chroma using direct local API.\n",
      "No existing DB found in ../data/interim, skipping load\n",
      "No existing DB found in ../data/interim, skipping load\n"
     ]
    }
   ],
   "source": [
    "docsearch = Chroma.from_documents(texts, embeddings, persist_directory='../data/interim')\n",
    "#docsearch.persist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cf86406",
   "metadata": {},
   "source": [
    "## Solution 1: Documentation Search\n",
    "In this next section, we demonstrate an embedding based approach to search within the documentation of the ROSA product. The algorithm takes user queries and looks for relevant content in the product documents. The _relevance_ is determined by vector similarilty in the embedding space. Then, it fetches the top k excerpts and the markdown document that has the probability of having the answer. The end user would see the top k relevant documents to their query and click on the page that they think answers their query. \n",
    "The benefits of these approach is that it is that it requires less resources to integrate with existing workflows. The reasons for that are: \n",
    "* Since it is not generative text, it is easier to evaluate the correctness of the approach\n",
    "* The CPU, GPU, memory, and disk requirements of this approach would be less than hosting a llm\n",
    "* This notebook uses OpenAI embedding API but there are comparable open source alternatives that can be adapted for this task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ecb58bfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetch the evaluation dataset \n",
    "eval_dataset = pd.read_csv('../data/processed/validation_data.csv')[['Question', 'Answer']]\n",
    "questions = eval_dataset['Question'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "a8baf53b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get relevant documents for the questions\n",
    "data = list()\n",
    "for q in questions:\n",
    "    docs = docsearch.similarity_search(q, 5)\n",
    "    data.append([q, docs[0].page_content, docs[0].metadata['source'],\n",
    "                 docs[1].page_content, docs[1].metadata['source'],\n",
    "                 docs[2].page_content, docs[2].metadata['source'],\n",
    "                 docs[3].page_content, docs[3].metadata['source'],\n",
    "                 docs[4].page_content, docs[4].metadata['source'],\n",
    "                 ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "ec49c2b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>A1-text</th>\n",
       "      <th>A1-source</th>\n",
       "      <th>A2-text</th>\n",
       "      <th>A2-source</th>\n",
       "      <th>A3-text</th>\n",
       "      <th>A3-source</th>\n",
       "      <th>A4-text</th>\n",
       "      <th>A4-source</th>\n",
       "      <th>A5-text</th>\n",
       "      <th>A5-source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is Red Hat OpenShift Service on AWS (ROSA)?</td>\n",
       "      <td># Understanding ROSA\\n\\nLearn about Red Hat OpenShift Service on AWS (ROSA), interacting with ROSA using Red Hat OpenShift Cluster Manager and command-line interface (CLI) tools, consumption experience, and integration with Amazon Web Services (AWS) services.</td>\n",
       "      <td>../data/external/rosa-docs/rosa_architecture.md</td>\n",
       "      <td>About ROSA\\n\\nROSA is a fully-managed, turnkey application platform that allows you to focus on delivering value to your customers by building and deploying applications. Red Hat and AWS Site reliability engineering (SRE) experts manage the underlying platform so you do not have to worry about the complexity of infrastructure management. ROSA provides seamless integration with a wide range of AWS compute, database, analytics, machine learning, networking, mobile, and other services to further accelerate the building and delivering of differentiating experiences to your customers.\\n\\nYou subscribe to the service directly from your AWS account. After the clusters are created, you can operate your clusters with the OpenShift web console or through Red Hat OpenShift Cluster Manager. The ROSA service also uses OpenShift APIs and command-line interface (CLI) tools. These tools provide a standardized OpenShift experience to use your existing skills and tools knowledge.</td>\n",
       "      <td>../data/external/rosa-docs/rosa_architecture.md</td>\n",
       "      <td>Red Hat OpenShift Service on AWS service definition\\n\\nThis documentation outlines the service definition for the Red Hat OpenShift Service on AWS (ROSA) managed service.</td>\n",
       "      <td>../data/external/rosa-docs/rosa_architecture.md</td>\n",
       "      <td>General\\n\\n### What is Red Hat OpenShift Service on AWS (ROSA)?\\nRed Hat Openshift Service on AWS (ROSA) is a fully-managed turnkey application platform that allows you to focus on what matters most, delivering value to your customers by building and deploying applications. Red Hat SRE experts manage the underlying platform so you don’t have to worry about the complexity of infrastructure management.\\n\\n### Where can I go to get more information/details?\\n- [ROSA Webpage](https://www.openshift.com/products/amazon-openshift)\\n- [ROSA Workshop](https://www.rosaworkshop.io)\\n- [ROSA Documentation](https://docs.openshift.com/rosa/welcome/index.html)</td>\n",
       "      <td>../data/external/rosaworkshop/14-faq.md</td>\n",
       "      <td>What are the differences between Red Hat OpenShift Service on AWS and Kubernetes?\\nEverything you need to deploy and manage containers is bundled with ROSA, including container management, automation (Operators), networking, load balancing, service mesh, CI/CD, firewall, monitoring, registry, authentication, and authorization capabilities. These components are tested together for unified operations as a complete platform. Automated cluster operations, including over-the-air platform upgrades, further enhance your Kubernetes experience.</td>\n",
       "      <td>../data/external/rosaworkshop/14-faq.md</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           question  \\\n",
       "0  What is Red Hat OpenShift Service on AWS (ROSA)?   \n",
       "\n",
       "                                                                                                                                                                                                                                                               A1-text  \\\n",
       "0  # Understanding ROSA\\n\\nLearn about Red Hat OpenShift Service on AWS (ROSA), interacting with ROSA using Red Hat OpenShift Cluster Manager and command-line interface (CLI) tools, consumption experience, and integration with Amazon Web Services (AWS) services.   \n",
       "\n",
       "                                         A1-source  \\\n",
       "0  ../data/external/rosa-docs/rosa_architecture.md   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            A2-text  \\\n",
       "0  About ROSA\\n\\nROSA is a fully-managed, turnkey application platform that allows you to focus on delivering value to your customers by building and deploying applications. Red Hat and AWS Site reliability engineering (SRE) experts manage the underlying platform so you do not have to worry about the complexity of infrastructure management. ROSA provides seamless integration with a wide range of AWS compute, database, analytics, machine learning, networking, mobile, and other services to further accelerate the building and delivering of differentiating experiences to your customers.\\n\\nYou subscribe to the service directly from your AWS account. After the clusters are created, you can operate your clusters with the OpenShift web console or through Red Hat OpenShift Cluster Manager. The ROSA service also uses OpenShift APIs and command-line interface (CLI) tools. These tools provide a standardized OpenShift experience to use your existing skills and tools knowledge.   \n",
       "\n",
       "                                         A2-source  \\\n",
       "0  ../data/external/rosa-docs/rosa_architecture.md   \n",
       "\n",
       "                                                                                                                                                                      A3-text  \\\n",
       "0  Red Hat OpenShift Service on AWS service definition\\n\\nThis documentation outlines the service definition for the Red Hat OpenShift Service on AWS (ROSA) managed service.   \n",
       "\n",
       "                                         A3-source  \\\n",
       "0  ../data/external/rosa-docs/rosa_architecture.md   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         A4-text  \\\n",
       "0  General\\n\\n### What is Red Hat OpenShift Service on AWS (ROSA)?\\nRed Hat Openshift Service on AWS (ROSA) is a fully-managed turnkey application platform that allows you to focus on what matters most, delivering value to your customers by building and deploying applications. Red Hat SRE experts manage the underlying platform so you don’t have to worry about the complexity of infrastructure management.\\n\\n### Where can I go to get more information/details?\\n- [ROSA Webpage](https://www.openshift.com/products/amazon-openshift)\\n- [ROSA Workshop](https://www.rosaworkshop.io)\\n- [ROSA Documentation](https://docs.openshift.com/rosa/welcome/index.html)   \n",
       "\n",
       "                                 A4-source  \\\n",
       "0  ../data/external/rosaworkshop/14-faq.md   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         A5-text  \\\n",
       "0  What are the differences between Red Hat OpenShift Service on AWS and Kubernetes?\\nEverything you need to deploy and manage containers is bundled with ROSA, including container management, automation (Operators), networking, load balancing, service mesh, CI/CD, firewall, monitoring, registry, authentication, and authorization capabilities. These components are tested together for unified operations as a complete platform. Automated cluster operations, including over-the-air platform upgrades, further enhance your Kubernetes experience.   \n",
       "\n",
       "                                 A5-source  \n",
       "0  ../data/external/rosaworkshop/14-faq.md  "
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a dataframe with the top 5 texts and sources\n",
    "s1_answers = pd.DataFrame(data=data, columns=['question', 'A1-text', 'A1-source',\n",
    "                                              'A2-text', 'A2-source',\n",
    "                                              'A3-text', 'A3-source',\n",
    "                                              'A4-text', 'A4-source',\n",
    "                                              'A5-text', 'A5-source',\n",
    "                                              ])\n",
    "s1_answers.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "f5920f90",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAHFCAYAAAD2eiPWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA+IUlEQVR4nO3deVhUdf//8RcMiyghLrjikpYohoqSaOLtWqlppml9NbFuU1MzNMmFLJe0vNOyunEpt0rt1tLUFrcyLbsrl1wpg1vAXUO0xBAEGeb3hz+nJiBhGJjh9HxcF5fOZz5zzvu8HceX55w5x81isVgEAABgYO7OLgAAAKCkEXgAAIDhEXgAAIDhEXgAAIDhEXgAAIDhEXgAAIDhEXgAAIDhEXgAAIDhEXgAoJRwnVfAeQg8AEpMZGSkgoKCbH7CwsI0ePBg7dmzp8TXn5ubqzVr1uiRRx5ReHi4WrZsqT59+mjFihXKzs62ztu9e7eCgoK0e/fuEqtlwYIFWrp0aYktH8Bf83B2AQCMLTg4WFOnTpUkmc1m/frrr1q1apUef/xxrVu3TrfffnuJrDczM1MjRozQoUOHNGDAAA0dOlSenp7atWuXZs+erZ07d2r+/Pny8vIqkfX/2RtvvKHRo0eXyroA5EXgAVCifH191aJFC5uxu+66S23bttW6des0ceLEElnvrFmztH//fq1YscJm/REREWrcuLGio6O1evVqDR48uETWD8C1EHgAlDofHx95e3vLzc3NOmY2m7V06VJ9/PHHOnnypNzd3dW4cWONHTtWbdq0kSTFxsbq448/Vu/evbV8+XJ5eXlp48aNqlixos3yf/nlF3344Yfq379/nrAlST179tSPP/6o6tWr24wnJydr0aJF+v777+Xr66s+ffpo7Nix8vDwsC43NjZWX375pVJTU1W+fHndeeediomJUWBgoKTrh/GqV6+u7Oxs7dy5U6Ghofr2228lSfPmzdO8efOUkJDgsF4CKBwCD4ASZbFYlJOTY/39pUuX9O677yo7O1sPPvigdd4rr7yiVatWKTo6WkFBQUpJSdH8+fM1ZswYffnll/Lx8ZEknT17Vl999ZVee+01Xbp0KU/YkaTvvvtOOTk56tSpU4F15bdnadasWRoxYoSGDh2qL774QosXL1aNGjU0aNAgWSwWPfHEE0pLS9MzzzyjqlWrKiEhQa+//rqmTp1qc37O5s2bdf/992vhwoXKzc3VmDFj9PDDD6tfv37q37+/3b0EYD8CD4AStXfvXjVt2jTP+Lhx49SwYUPr4/Pnz+vpp59WZGSkdczb21tPPfWUEhISrHtqcnJyNHHiRIWFhRW4znPnzkmSda9LYQ0ePFijRo2SJLVp00bbtm3Trl27NGjQIJ0/f14+Pj426w4PD9fJkyf1/vvv2yzH09NT06dPz3N+UI0aNfLd4wSg5BF4AJSopk2bavr06ZKu7+G5fPmydu7cqddee00ZGRl6+umnJUmvvvqqpOuHjZKTk3XixAnt2LFDkmy+USVJTZo0+ct13jgElZubW6Ra/xii3NzcVLt2bV2+fFmSVL16dS1fvlwWi0WnT5/WiRMnlJycrP379+epr0GDBqV2MjSAwiHwAChRFSpUUEhIiM1YRESEMjIytGTJEg0ePFhVqlRRXFycpk+frri4OPn4+Oi2225TrVq1JOW9fk2FChX+cp03Xnf27NkCvwV2/vx5Va5c2RqOJFkPm93g7u5us+6PP/5Yc+fO1blz5+Tv768mTZqoXLly+W4zANfCdXgAOMUdd9yhnJwcnT59Wunp6Ro6dKjKly+vjRs3av/+/Vq7dq3NOT5F0aZNG3l6euqrr74qcM6wYcOKtPzvv/9eEydO1D333KOdO3dq9+7deueddzhEBZQRBB4ATnH48GGZTCbVqVNHycnJunTpkgYPHqzbbrtN7u7XP5p27twpqeiHpvz8/NSvXz998MEH+uGHH/I8v2HDBsXHx+v+++8v9DIPHDig3NxcPfXUU9Zvd5nNZus3sG5W441tAuAcHNICUKLS09N18OBB6+Ps7Gxt375dH374oR5++GFVrlxZnp6e8vX11ZtvvikPDw95eHho69atWrt2raTrFxEsqnHjxikuLk6RkZEaNGiQWrdurZycHO3cuVMffPCBOnXqpEcffbTQy2vWrJkk6YUXXtCDDz6otLQ0vffee4qPj5ckZWRkyNfXt8DX+/n5af/+/dq7d6/CwsJsvpIPoOQReACUqCNHjujhhx+2Pvb29lbdunX19NNP6/HHH5ck3XLLLVqwYIFmz56tMWPGqEKFCmrSpIlWrlypYcOG6fvvv1fnzp2LtF4/Pz+tWLFCK1eu1KZNm7Rq1SpZLBbVr19fzz33nPr162dz/s7NhIeHa8qUKXr77be1ZcsWVa1aVeHh4Zo3b56efPJJ7du3Tx06dCjw9SNGjNCCBQs0bNgwbdq0yXqeEYDS4WbhbnYAAMDgOKgMAAAMj8ADAAAMj8ADAAAMj8ADAAAMj8ADAAAMj8ADAAAMj+vw6PoVUnNycuTu7s7FwAAAKCMsFotyc3Pl4eFx06uZE3gk5eTkKC4uztllAAAAO4SEhMjLy+sv5xB49Ps9bkJCQmQymZxcDQAAKAyz2ay4uLhC3auOwCNZD2OZTCYCDwAAZUxhTkfhpGUAAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AkGQxm51dgkugDzAqD2cXAACuwM1k0k8zZijjxAlnl+I05evVU5Pnn3d2GUCJIPAAwP+XceKE0v931NllACgBHNICAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACGR+ABAACG5+HsAiQpOztbffv21fPPP6/w8HBNmjRJ69evzzMvPDxcy5cvzzOelpam1q1b24z5+/tr9+7dJVYzAAAoO5weeLKyshQdHa2jR49axyZPnqzo6Gjr4zNnzigyMlKDBw/OdxmJiYny9/fXp59+ah1zd2fnFQAAuM6pgScxMVHR0dGyWCw247fccotuueUW6+NJkyapW7du6tq1a77LSU5O1q233qqAgIASrRcAAJRNTt0NsmfPHoWHh+v9998vcM53332nvXv3aty4cQXOSUxMVP369UugQgAAYARO3cMzcODAm85ZtGiR+vTpo5o1axY4JykpSTk5OerXr59SUlIUFhammJgYVatWzZHlAgCAMsrp5/D8lVOnTmnXrl2aPHnyX85LTk5W5cqVFRMTI4vFotdee00jRozQmjVrZDKZCr0+s9lc3JIBlFFF+awwOj4LUVYU5b3q0oFn69atatKkiW677ba/nLdx40a5ubmpXLlykqR///vfioiI0KFDh9SyZctCry8uLq5Y9QIom3x8fBQcHOzsMlxGQkKCMjMznV0G4FAuHXi+/vprdenS5abzfHx8bB5XqVJF/v7+SklJKdL6QkJC+F8egL+9oKAgZ5cAFIrZbC70zgqXDTwWi0VxcXEaMWLEX85LT09Xp06dFBsbqzZt2kiSUlJS9Ouvv6pBgwZFWqfJZCLwAPjb43MQRuSyF6s5c+aMrly5ku/hrKtXryo1NVWS5Ovrq1atWmnWrFk6fPiwfvzxRz399NNq3749/0sBAACSXDjwXLx4UZJUsWLFPM9t2rRJERER1scvv/yygoODNXz4cEVGRqp27dp65ZVXSq1WAADg2lzmkFZCQoLN4+bNm+cZu6Fv377q27ev9XHFihU1a9asEq0PAACUXS67hwcAAMBRCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwXCLwZGdnq2fPntq9e7d1bObMmQoKCrL5WblyZYHLeOedd9S+fXuFhobq2WefVWZmZmmUDgAAygAPZxeQlZWl6OhoHT161GY8KSlJ0dHR6tOnj3XM19c332Vs3bpV8+bN05w5c1SlShXFxMRozpw5mjJlSonWDgAAygan7uFJTEzUQw89pJMnT+Z5LikpScHBwQoICLD++Pj45Luc5cuX69FHH1WnTp3UrFkzTZ8+XR9++CF7eQAAgCQnB549e/YoPDxc77//vs14enq6UlJSVL9+/Zsuw2w2Ky4uTmFhYdaxFi1a6Nq1a4qPj3d0yQAAoAxy6iGtgQMH5juelJQkNzc3vfnmm9q5c6f8/f31z3/+0+bw1g2XL19WVlaWqlWrZh3z8PCQv7+/fv755yLVYzabi7YBAAzDZDI5uwSXwWchyoqivFedfg5PfpKTk+Xm5qYGDRpo0KBB2rt3r55//nn5+vrq7rvvtpl79epVSZKXl5fNuJeXl7Kzs4u03ri4uOIVDqBM8vHxUXBwsLPLcBkJCQmcEuBknp6eahocLJOHS/4zXarMOTn68cgRXbt2rVjLcclOPvDAA+rUqZP8/f0lSY0bN9bx48e1atWqPIHH29tbkvKEm+zs7ALP+SlISEgI/8sD8LcXFBTk7BKg63sdNyx4QxfPnnZ2KU5TpVagHhg1Rk2bNs33+RuntRSGSwYeNzc3a9i5oUGDBtq1a1eeuf7+/vL29taFCxfUsGFDSVJOTo4uXbqkgICAIq3XZDIReAD87fE56Dounj2tn48fc3YZTueI96RLXIfnz9544w099thjNmPx8fFq0KBBnrnu7u4KCQnRvn37rGMHDx6Uh4eHGjduXNKlAgCAMsAlA0+nTp20d+9eLV26VCdPntR//vMfbdiwQUOGDJF0/byd1NRU6/yBAwdq6dKl2rZtmw4fPqxp06bpoYceKvIhLQAAYEwueUirWbNmeuONN/Tvf/9bb7zxhmrXrq1XX31VoaGhkqRNmzYpJiZGCQkJkqT77rtPZ86c0ZQpU5Sdna177rlH48ePd+YmAAAAF+IygedGeLmha9eu6tq1a75z+/btq759+9qMDR8+XMOHDy+x+gAAQNnlkoe0AAAAHInAAwAADI/AAwAADI/AAwAADI/AAwAADI/AAwAADI/AAwAADI/AAwAADI/AAwAADI/AAwAADI/AAwAADI/AAwBwmNxci7NLcAn0wfW4zM1DAQBln7u7mz57b79+TUl3dilOU6m6r+55pKWzy8CfEHgAAA71a0q6Us+kObsMwAaHtAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOG5RODJzs5Wz549tXv3buvYwYMH9X//938KDQ3VvffeqzVr1vzlMsLCwhQUFGTzc+XKlZIuHQAAlAEezi4gKytL0dHROnr0qHUsNTVVw4YN04ABA/Svf/1LP/74o2JiYhQQEKCOHTvmWUZKSop+++03bdu2TeXKlbOOly9fvjQ2AQAAuDinBp7ExERFR0fLYrHYjG/btk1Vq1bVuHHjJEn169fX7t279cknn+QbeJKSkhQQEKA6deqURtkAAKCMcWrg2bNnj8LDw/X000+rRYsW1vH27durSZMmeeanp6fnu5zExETdeuutJVUmAAAo45waeAYOHJjveGBgoAIDA62PL168qI0bN+qpp57Kd35SUpIyMzMVGRmpY8eOqUmTJnr22WeLHILMZnOR5gMwDpPJ5OwSXEZxPgvp4++K+28KvfxdQb0sSo+dfg7PzVy9elVPPfWUqlatqocffjjfOcnJyUpLS9O4cePk6+urxYsX67HHHtPGjRvl6+tb6HXFxcU5qmwAZYiPj4+Cg4OdXYbLSEhIUGZmZpFfRx9t2dtHiV7+WXF6eYNLB54rV65o1KhROn78uP7zn//Ix8cn33lLly7VtWvXVKFCBUnSK6+8og4dOmjHjh3q1atXodcXEhJCogbwtxcUFOTsEgyBPjpOQb00m82F3lnhsoEnPT1dQ4cO1cmTJ/Xuu++qfv36Bc718vKSl5eX9bG3t7cCAwOVkpJSpHWaTCYCD4C/PT4HHYM+Oo4jeukS1+H5s9zcXI0ePVqnT5/WihUrdPvttxc412KxqGvXrlq3bp11LCMjQydOnFCDBg1Ko1wAAODiXHIPz9q1a7V7924tXLhQfn5+Sk1NlSR5enrK399f2dnZSktLU+XKlWUymdSxY0fFxsaqdu3aqly5st544w3VqFFDHTp0cPKWAAAAV+CSgWfr1q3Kzc3VE088YTPeunVrrVixQgcOHNDgwYP1xRdfKDAwUOPHj5eHh4eio6OVnp6uNm3aaNGiRexOBAAAklwo8CQkJFh/v3Tp0r+cGx4ebjPf29tbkyZN0qRJk0qsPgAAUHa55Dk8AAAAjkTgAQAAhkfgAQAAhkfgAQAAhkfgAQAAhkfgAQAAhkfgAQAAhkfgAQAAhkfgAQAAhkfgAQAAhmdX4Nm1a5csFoujawEAACgRdt1La8yYMfL09FS3bt3Us2dPtWjRwsFlAQAAOI5dgeebb77RN998oy1btmj48OHy9fVV9+7ddd999yk4ONjRNQIAABSLXYHHw8NDHTp0UIcOHZSTk6Nvv/1W27dv18CBA1W9enX16tVLffv2Va1atRxdLwAAQJEV66Tl7OxsffXVV9q4caM2b96sSpUqqXPnzjp+/Ljuu+8+rVy50lF1AgAA2M2uPTzbtm3Tli1b9OWXX8rT01P33nuv5s+fr7CwMOuc9957T3PnztWgQYMcViwAAIA97Ao8EydOVNeuXTV37ly1a9dOJpMpz5w77rhD//znP4tdIAAAQHHZFXi+/fZbpaen6/Lly9aws2nTJt15550KCAiQJDVv3lzNmzd3XKUAAAB2suscnv379+vuu+/WJ598Yh1bvny5evTooX379jmsOAAAAEewK/C8/PLLGjFihKKioqxjq1ev1tChQ/XSSy85rDgAAABHsCvwHD9+XN26dcsz3r17dyUmJha7KAAAAEeyK/A0aNBAmzdvzjO+fft21a1bt9hFAQAAOJJdJy2PHTtWo0aN0jfffKOmTZtKkhISEvT9998rNjbWoQUCAAAUl117eP7xj39o/fr1Cg4OVnJysk6ePKnGjRtr48aN6tChg6NrBAAAKBa79vBI0u23365JkyY5shYAAIASYVfguXz5spYtW6a4uDjl5OTIYrHYPL98+XKHFAcAAOAIdgWeCRMmKC4uTr169ZKvr6+jawIAAHAou6+0vHLlSjVr1szR9QAAADicXSctV69eXe7uxbrROgAAQKmx+5DWtGnTFBUVpXr16snT09Pm+Vq1ajmkOAAAAEewK/A89dRTkqThw4dLktzc3CRJFotFbm5u+umnnxxUHgAAQPHZFXi++OILR9cBAABQYuw6Ead27dqqXbu2MjIydOTIEVWqVEm5ubmqVauWateu7egaAQAAisWuPTxpaWkaM2aM9uzZI0naunWrXnzxRZ06dUqLFi0i9AAAAJdi1x6emTNnysfHR7t27ZK3t7ck6aWXXlKNGjU0c+bMIi8vOztbPXv21O7du61jp06d0mOPPaYWLVqoR48e+u9///uXy/j000/VtWtXNW/eXE8++aR++eWXItcBAACMya7A8/XXX2vcuHHy8/OzjlWuXFkxMTHau3dvkZaVlZWlcePG6ejRo9Yxi8WiJ598UlWrVtWHH36o3r17a/To0Tp79my+yzh8+LAmT56s0aNH6/3339fly5cVExNjz6YBAAADsvteWllZWXnGfvnlF3l4FH6RiYmJio6OznNril27dunUqVNavXq1ypcvr4YNG+q7777Thx9+aP2G2B+tXLlS3bt31wMPPCBJmj17tjp16qRTp06pTp06RdswAABgOHbt4enZs6defPFFHT16VG5ubsrIyNCuXbv0/PPPq0ePHoVezp49exQeHq7333/fZvzQoUMKDg5W+fLlrWOtWrXSwYMH813OoUOHFBYWZn1cs2ZN1apVS4cOHSrahgEAAEOy+8KDc+fOVd++fXXt2jX17t1bJpNJ/fv314QJEwq9nIEDB+Y7npqaqmrVqtmMValSRT///HO+88+fP1+k+QUxm81Fmg/AOEwmk7NLcBnF+Sykj78r7r8p9PJ3BfWyKD22K/B4eXlp0qRJGjt2rE6dOiWz2aw6deqoQoUK9iwuj8zMTHl5eeVZZ3Z2dr7zr169WqT5BYmLiytaoQAMwcfHR8HBwc4uw2UkJCQoMzOzyK+jj7bs7aNEL/+sOL28wa7Ak9+JyUeOHLH+/s4777S/Ikne3t66dOmSzVh2drbKlStX4Pw/h5vs7Gz5+PgUab0hISEkagB/e0FBQc4uwRDoo+MU1Euz2VzonRV2BZ7IyMh8x728vBQQEFDsKzFXr15diYmJNmMXLlzIc9jqj/MvXLiQZ35AQECR1msymQg8AP72+Bx0DProOI7opV2BJz4+3uax2WzWyZMnNWPGDPXq1avYRTVv3lyLFi3S1atXrXt19u3bp1atWhU4f9++ferbt68k6dy5czp37pyaN29e7FoAAEDZZ9e3tP7MZDLp1ltv1aRJk/TGG28Ue3mtW7dWzZo1FRMTo6NHj2rRokU6fPiw+vXrJ+n64arU1FTryUoDBgzQRx99pDVr1ig+Pl4TJkxQx44d+Uo6AACQ5KDAc8PFixd1+fLlYi/HZDJpwYIFSk1NVd++ffXxxx9r/vz5qlWrliTpwIEDioiI0Llz5yRJoaGheuGFFzR//nwNGDBAFStW1KxZs4pdBwAAMAa7DmnldxXjK1eu6Ntvv1W3bt3sKiQhIcHmcb169bRy5cp854aHh+eZ37dvX+shLQAAgD+y+0rLf+bv76+JEyeqd+/ejlokAACAQ9gVeDhcBAAAyhK7As+8efMKPXf06NH2rAIAAMBh7Ao8J06c0JYtW+Tv76877rhDXl5eio+P18mTJ9WiRQvrDUTd3NwcWiwAAIA97L61RK9evTR9+nR5enpax19++WWlpaXppZdecliBAAAAxWXX19I3bdqkoUOH2oQdSXrooYe0adMmhxQGAADgKHYFnurVq+vrr7/OM75161Yu9gcAAFyOXYe0oqOjNXbsWH355Zdq3LixpOt3Gj9y5IjefPNNhxYIAABQXHbt4bn77ru1bt06NWrUSElJSTpz5oxat26trVu3qnXr1o6uEQAAoFjsvvBgUFCQYmJilJaWJl9fX7m7u/OtLAAA4JLs2sNjsVi0cOFChYeHq23btjp79qzGjx+vKVOmKDs729E1AgAAFItdgWf+/Pn6+OOP9a9//UteXl6SpD59+uibb77R7NmzHVogAABAcdkVeNavX68XXnhBnTp1sh7GateunV5++WVt3rzZoQUCKJg51+zsElwCfQBwM3adw3Px4kVVq1Ytz7ifn58yMjKKXRSAwjG5mzR963Qd//W4s0txmvqV6mvqvVOdXQYAF2dX4GnTpo2WLl2qF154wTqWnp6uuXPnKjw83GHFAbi5478e1/9S/+fsMgDApdl1SGvatGk6cuSI2rVrp6ysLI0aNUodOnTQmTNn9Nxzzzm6RgAAgGKxaw+Pn5+f1q5dq++++07JycnKycnRrbfeqoiICLm725WhAAAASoxdgadnz56aN2+e2rZtq7Zt2zq6JgAAAIeya3eMu7u7rl275uhaAAAASoRde3g6duyof/7zn+rUqZNq165tvRbPDaNHj3ZIcQAAAI5gV+BJSEhQ06ZNdf78eZ0/f97mOW4vAQAAXE2hA88jjzyihQsXys/PTytWrJAkXb16VeXKlSux4gAAAByh0Ofw7Nu3L895O3fddZdOnTrl8KIAAAAcqVjfIbdYLI6qAwAAoMRw0RwAAGB4BB4AAGB4RfqW1ubNm+Xr62t9nJubq88//1yVK1e2mffAAw84pDgAAABHKHTgqVWrlpYtW2YzVqVKFa1cudJmzM3NjcADAABcSqEDz/bt20uyDgAAgBLDOTwAAMDwCDwAAMDwCDwAAMDwCDwAAMDwCDwAAMDw7LpbemlYt26dYmJi8oy7ubkpPj4+z/j999+vhIQEm7FPPvlEjRo1KrEaAQBA2eCygadHjx5q37699XFOTo4effRRdezYMc9cs9ms48ePa+XKlapfv751vFKlSqVQKQAAcHUuG3jKlSuncuXKWR+/9dZbslgseuaZZ/LMPX36tK5du6ZmzZrJ29u7NMsEAABlQJk4h+fSpUtavHixoqOj5eXllef5xMRE1axZk7ADAADyVSYCz6pVq1StWjV169Yt3+eTkpLk6empJ554Qu3atdOgQYN0+PDhUq4SAAC4Kpc9pHWDxWLRmjVrNHTo0ALnHDt2TGlpaerfv7+ioqL0wQcf6NFHH9WmTZtUs2bNQq/LbDY7omSg1JhMJmeX4DKK+/eXXv6uOL2kj7/jPek4BfWyKD12+cATFxenlJQU3XfffQXOmTFjhq5evWq9k/u0adO0f/9+ffTRRxoxYkSR1gWUFT4+PgoODnZ2GS4jISFBmZmZdr2WXtqyt5f00RbvSccpTi9vcPnA8/XXXyssLEwVK1YscI6Hh4c17EjXv7reoEEDpaSkFGldISEhJGqgjAoKCnJ2CYZBLx2DPjpOQb00m82F3lnh8oHn8OHDatmy5V/OiYyMVHh4uEaPHi1Jys3NVUJCgh555JEirctkMhF4gDKKv7uOQy8dgz46jiN66fInLR89elS33XabzZjZbFZqaqqys7MlSZ07d9Y777yjL774QsnJyXrhhRf022+/qU+fPs4oGQAAuBiX38Nz4cIF+fn52YydO3dOXbp00fLlyxUeHq7HHntMWVlZmjlzpi5cuKDmzZvr7bfftjnMBQAA/r5cPvDk9/XywMBAm9tIuLm5acSIEUU6QRkAAPx9uPwhLQAAgOIi8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8AAAAMMj8KDUWXLNzi7BJdAHACg9Hs4uAH8/bu4mXVg3SdcuJDu7FKfxrNpAVfv+y9llAMDfBoEHTnHtQrKu/fyTs8sAAPxNcEgLAAAYHoEHAAAYHoEHAAAYHoEHAAAYHoEHAAAYHoEHAAAYHoEHAAAYHoEHAAAYHoEHAAAYHoEHAAAYnksHns8//1xBQUE2P1FRUfnO/fbbb9WzZ081b95cgwcP1qlTp0q5WgAA4Kpc+l5aiYmJ6tSpk2bMmGEd8/b2zjPv7NmzevLJJ/XUU0+pffv2mj9/vkaNGqWPP/5Ybm5upVkyAABwQS4deJKSktSoUSMFBAT85bw1a9bojjvu0JAhQyRJs2bNUrt27bRnzx6Fh4eXRqkAAMCFufQhraSkJNWvX/+m8w4dOqSwsDDrYx8fHzVt2lQHDx4sueIAAECZ4bJ7eCwWi44dO6b//ve/euutt2Q2m9WtWzdFRUXJy8vLZm5qaqqqVatmM1alShX9/PPPRVqn2Wwudt24OZPJ5OwSXEZx33P08nf00nGK00v6+Dvek45TUC+L0mOXDTxnz55VZmamvLy89Prrr+v06dOaOXOmrl69queee85m7o15f+Tl5aXs7OwirTMuLq7YdeOv+fj4KDg42NlluIyEhARlZmba9Vp6aYteOo69vaSPtnhPOk5xenmDywae2rVra/fu3apYsaLc3NzUpEkT5ebmavz48YqJibFJvt7e3nnCTXZ2tvz8/Iq0zpCQEBI1SlVQUJCzSzAMeuk49NIx6KPjFNRLs9lc6J0VLht4JMnf39/mccOGDZWVlaW0tDRVrlzZOl69enVduHDBZu6FCxfUpEmTIq3PZDIReFCqeL85Dr10HHrpGPTRcRzRS5c9afnrr79WeHi4zS6sn376Sf7+/jZhR5KaN2+uffv2WR9nZmbqyJEjat68eanVCwAAXJfLBp7Q0FB5e3vrueeeU3Jysr766ivNnj1bQ4cOldlsVmpqqvUw1oMPPqj9+/dr0aJFOnr0qGJiYhQYGMhX0gEAgCQXDjy+vr5aunSpfvnlFz344IOaPHmyHn74YQ0dOlTnzp1TRESEDhw4IEkKDAxUbGysPvzwQ/Xr10+XLl3S/PnzueggAACQ5OLn8Nx+++16++2384wHBgYqISHBZqxDhw7q0KFDaZUGAADKEJfdwwMAAOAoBB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4BB4AAGB4Lh14UlJSFBUVpdatW6t9+/aaNWuWsrKy8p07cuRIBQUF2fzs2LGjlCsGAACuyMPZBRTEYrEoKipKfn5+eu+995SWlqZnn31W7u7umjhxYp75SUlJmjNnjtq2bWsdq1ixYmmWDAAAXJTLBp7k5GQdPHhQ33zzjapWrSpJioqK0ssvv5wn8GRnZ+v06dMKCQlRQECAM8oFAAAuzGUPaQUEBGjJkiXWsHNDenp6nrnJyclyc3NTnTp1Sqs8AABQhrjsHh4/Pz+1b9/e+jg3N1crV65UmzZt8sxNTk6Wr6+vJkyYoD179qhGjRp66qmn1KFDhyKt02w2F7tu3JzJZHJ2CS6juO85evk7euk4xeklffwd70nHKaiXRemxywaeP5szZ46OHDmitWvX5nkuOTlZV69eVUREhIYPH67PP/9cI0eO1Pvvv6+QkJBCryMuLs6RJSMfPj4+Cg4OdnYZLiMhIUGZmZl2vZZe2qKXjmNvL+mjLd6TjlOcXt5QJgLPnDlz9O677+q1115To0aN8jw/atQoRUZGWk9Sbty4sX788Ud98MEHRQo8ISEhJGqUqqCgIGeXYBj00nHopWPQR8cpqJdms7nQOytcPvDMmDFDq1at0pw5c3TvvffmO8fd3T3PN7IaNGigxMTEIq3LZDIReFCqeL85Dr10HHrpGPTRcRzRS5c9aVmS5s2bp9WrV2vu3Lm67777Cpw3adIkxcTE2IzFx8erQYMGJV0iAAAoA1w28CQlJWnBggUaNmyYWrVqpdTUVOuPJKWmpurq1auSpM6dO+uTTz7Rhg0bdOLECc2bN0/79u3ToEGDnLkJAADARbjsIa0vvvhCZrNZCxcu1MKFC22eS0hIUEREhGbNmqW+ffvqnnvu0dSpU7Vw4UKdPXtWt99+u5YsWaLAwEAnVQ8AAFyJywae4cOHa/jw4QU+n5CQYPO4f//+6t+/f0mXBQAAyiCXPaQFAADgKAQeAABgeAQeAABgeASeIjDn5jq7BJdAHwAAZY3LnrTsikzu7nruP1/r2Pk0Z5fiNLdWq6iZA9vffCIAAC6EwFNEx86nKf7ML84uAwAAFAGHtAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOEReAAAgOG5dODJysrSs88+q7CwMEVERGjZsmUFzj1y5Ij69++v5s2b68EHH9QPP/xQipUCAABX5tKBZ/bs2frhhx/07rvvaurUqZo3b562bNmSZ15GRoaGDx+usLAwrVu3TqGhoXriiSeUkZHhhKoBAICrcdnAk5GRoTVr1mjy5Mlq2rSp7r77bg0dOlTvvfdenrmbNm2St7e3JkyYoIYNG2ry5MmqUKFCvuEIAAD8/bhs4ImPj1dOTo5CQ0OtY61atdKhQ4eUm5trM/fQoUNq1aqV3NzcJElubm5q2bKlDh48WJolAwAAF+WygSc1NVWVKlWSl5eXdaxq1arKysrSpUuX8sytVq2azViVKlX0888/l0apAADAxXk4u4CCZGZm2oQdSdbH2dnZhZr753kFsVgs1uWaTKYC55lMJt1eo6K8TG6FWq4R1Qvwk9lsltlstnsZJpNJpoBGynX3uvlkgzJVqV/sPkrXe3lb5dvk6e7poMrKnnr+9RzWS5+GDSXPv28vferWdcjf78o1K8it4I9Sw6tUrYLD3pNV69STu4fL/lNd4irXrP2XvbwxfuPf8b/isl309vbOE1huPC5Xrlyh5v55XkFuHCI7cuTITef2ur28dHv5Qi3XqBxyqLBuH6lu8RdTlp1y0CHXblW6SVUcsqgyy2GHr++5xzHLKaMy5ZheVm4kVW5UodjLKcsc9Z4MDGurQIcsqewqTC//fKpLflw28FSvXl2//vqrcnJy5PH/021qaqrKlSsnPz+/PHMvXLhgM3bhwoU8h7kK4uHhoZCQELm7u1vPAwIAAK7NYrEoNzfXmhP+issGniZNmsjDw0MHDx5UWFiYJGnfvn3WYPJHzZs31+LFi2WxWOTm5iaLxaL9+/drxIgRhVqXu7t7nkNiAADAOFz2pGUfHx898MADmjZtmg4fPqxt27Zp2bJlGjx4sKTre3uuXr0qSerWrZsuX76sF198UYmJiXrxxReVmZmp7t27O3MTAACAi3CzFOZMHyfJzMzUtGnT9Nlnn8nX11ePP/64HnvsMUlSUFCQZs2apb59+0qSDh8+rKlTpyopKUlBQUGaPn26goODnVg9AABwFS4deAAAABzBZQ9pAQAAOAqBBwAAGB6BBwAAGB6BBwAAGB6BpwR17txZQUFB1p/GjRurdevWGjlypM6dO+eQdcTGxioyMrLIr1u4cKEmTZrkkBpKmqv10WKxaNGiRercubNatmypRx99VImJiQ6po6S5Wi/NZrNeeeUVtWvXTqGhoRozZkyei4i6Klfr5R9t3rxZQUFBDqmhpLliH8PCwmxqCgoK0pUrVxxSy80EBQUpOjo6z/i6devUuXPnQi9n8+bNunjxYr7PTZo0ye7P/xMnTqh3794KCQnR66+/rs6dO2vdunV2Leu7775TUlJSgc+PHDkyz5/Djh077FqX5MIXHjSKZ599Vj169JB0/dLXiYmJmjp1qiZOnKjly5c7paZPP/1UsbGxuv/++52yfnu4Uh9Xr16tZcuWadasWapfv76WLFmiYcOGadOmTfLx8SnVWuzhSr1ctGiRNm3apNdff12VKlXSzJkzNWHCBC1btqxU67CXK/XyhhvXJCtLXKmPKSkp+u2337Rt2zab2xOVL196txT69NNP1a9fP7Vt29au1585c0Zjx47VF198ke/zkydPtru2lStXSpI2btyoihUravDgwXb35rHHHtPy5cvVsGHDfJ9PSkrSnDlzbPpQsWJFu9YlEXhK3C233KKAgADr4+rVqysqKkrjx4/Xb7/9pltuuaXUasnJydGMGTO0fv161alTp9TW6wiu1Mf169dryJAh6tSpkyRp2rRpat26tfbv36927dqVWh32cqVems1mxcTE6M4775QkRUZGaty4caW2/uJypV7eMHv2bNWpU0epqamlvm57uVIfk5KSFBAQ4NTPyNq1a+uFF17QRx99ZNddAG52tZni9DM9PV2NGzdW3bolezPE7OxsnT59WiEhITbvjeLgkJYT3HgD37hFRmJioh5//HGFhoYqJCREAwcOtO7m2717tzp37qz//Oc/at++vVq0aKHx48fneyf4rKwsDRgwQEOGDMn3+YyMDCUkJOiDDz5QaGhoCW5h6XBWHydMmGCzd+zG7Ux+++23ktjMUuGsXo4ePVp33323JOnixYtas2aNWrduXVKbWSqc1UtJ2rNnj/bs2VPo2+q4Mmf1MTExUbfeemsJbtnNjR07VikpKVq6dGmBc37++WeNGTNGrVu3Vnh4uGbOnGndni5dulh/ze9w0x8PacXGxio6OlpTp05Vy5Yt1bZtWy1evDjfdU6aNEnr1q3Thg0bFBQUpNOnT9sc0oqMjNSMGTPUpUsXdezYUenp6Vq+fLk6deqkkJAQ9e3bV99//70kWQ/PDR48WLGxsXnWlZycLDc3N4cGTwJPKTt58qQWLVqk9u3bq0KFCsrNzdWIESNUu3ZtffTRR1q9erXMZrPmzJljfc358+e1detWLVmyRLGxsfrss8+0YcMGm+Xm5uZq3Lhxys3N1bx58/L9X4Gfn59Wr16txo0bl/Rmljhn9jEsLEw1atSwPl6zZo1ycnLUqlWrEtvekuTMXt7w73//W3fddZf2799fZs4ty48ze5mdna3nn39eU6ZMsTkUUxY5s49JSUnKzMxUZGSkIiIiNGzYMB07dqykN9nGjT1cb775pk6dOpXn+ezsbD366KPKzMzUihUr9Prrr+vLL7/U7NmzJV3/TLrx641DhX9l69at8vb21vr16/X444/rlVdeyXebJ0+erO7du6t79+7673//q5o1a+aZs27dOs2ZM0fz5s3TyZMnNXv2bE2dOlWbN29WWFiYxo4dq9zcXK1du1bS9cA1ZMiQPMtJTk6Wr6+vJkyYoIiICPXr109fffXVTbflrxB4StjUqVMVGhpq/V/JAw88oIYNG1r/ol69elX/93//p0mTJqlu3bpq2rSp+vTpY3MS7LVr1/Tcc88pKChI7du3V/v27RUXF2eznhkzZujEiRN66623SvVYc2lx1T4eOnRIL7/8sh5//HGH7XYtaa7Yy969e2vt2rVq27athgwZovT0dMdveAlwpV7Onz9fTZs2VURERMltcAlxpT4mJycrLS1NI0eO1IIFC1SuXDk99thjpf6ejIyMVL169fI9H+vrr79WSkqK5syZo6CgILVt21ZTpkzRqlWrdOXKFVWuXFmSVLly5UKFX39/f02cOFH16tXT0KFD5e/vrx9++CHPvFtuuUXlypVTuXLlFBAQIJPJlGdOx44d1bJlS91xxx06c+aM3NzcVKtWLQUGBmrs2LGaM2eOcnNzrTVWrFhRFSpUyLOc5ORkXb16VREREVqyZIk6dOigkSNH5vkzLQrO4SlhUVFRuueee3TlyhXFxsbqzJkzio6OVqVKlSRdPxFuwIAB2rBhg3744QclJyfryJEjqlq1qs1y6tWrZ/29r6+vcnJyrI8PHDigvXv3qnnz5sU6ocuVuWIfDxw4oGHDhukf//iHxowZ46AtLXmu2Msby5o9e7b+8Y9/6LPPPrPeJ8+VuUov//e//+mDDz7QJ598UgJbWfJcpY+StHTpUl27ds36j/Arr7yiDh06aMeOHerVq5cjN/svmUwmTZs2TQMHDtS2bdtsnktKSlL9+vVttqNly5bKycnRyZMni3yOTmBgoE14qVChgk3viqJ27drW30dERKhRo0bq1auXgoOD1aVLF/Xv318eHjePHqNGjVJkZKR1Gxs3bqwff/xRH3zwgUJCQuyqjT08JaxKlSqqV6+egoOD9cYbb0i6/gd57do1SdKVK1fUr18/ffrpp2rQoIGioqI0YcKEPMv5867XP56UVqFCBa1YsUJHjx617iY0Glfr4+7duzVkyBC1adNGr776qvU8g7LAlXq5Y8cOpaSkWB97e3urTp06+vXXX4u1jaXFVXr52WefKS0tTXfffbdCQ0M1bNgwSVJoaKg+/vhjh2xrSXKVPt5Yxh/3OHh7eyswMNDmfVpaWrZsqQcffFAvvviiMjMzbWr6M7PZbPNrUXh6euYZs/c2m3+szcfHR2vWrNG7776r1q1ba926derbt2+heunu7p4nmDZo0KBYfw5l51PaALy8vDRz5kz99NNPeueddyRdP8nw/PnzWr58uYYOHaq77rpLZ8+eLdKbrVGjRrrzzjs1cuRIvfrqq7p06VLJbICLcHYf//e//2nkyJFq3769Xn/99Xw/LMoKZ/fy5ZdftjnPIj09XcePHy/wa6quzJm9HDRokDZv3qwNGzZow4YNmjlzpiRpw4YNRbp2iytwZh8tFou6du1qc6JvRkaGTpw4oQYNGhR30+zyzDPPKCMjw+YE5ltvvVXHjx+32YaDBw/Kw8NDdevWlZubmxMqzevAgQN666231KZNG8XExGjLli3KysrSvn37bvraSZMmKSYmxmYsPj6+WH8OBJ5S1qxZM/Xr108LFixQSkqK/P39lZGRoW3btun06dNas2aN3nvvvQK/hfFXHn30UVWsWFFz584tgcpdizP7OGXKFNWsWVMxMTH69ddflZqaqtTUVF29erW4m+UUzuzlI488oqVLl+qrr77S0aNHNX78eNWtW1f/+Mc/irtZTuGsXvr7+6tevXrWn+rVq0u6fojH19e32NtV2pzVRzc3N3Xs2FGxsbHavXu3jh49qgkTJqhGjRrq0KGDIzatyCpVqqRnnnlGZ86csY61a9dOderU0YQJE5SQkKBdu3ZpxowZ6tmzp/z8/KzXA4uPjy+1Cybmp1y5cpo/f77WrFmj06dPa+PGjcrIyLBeFLN8+fI6evRovt9w7dy5sz755BNt2LBBJ06c0Lx587Rv3z4NGjTI7noIPE7w9NNPy9PTU3PmzFFoaKiefPJJTZ8+Xffff7/WrVunKVOm6OLFi0Xedefl5aWYmBitWbOmWCd2lRXO6GNqaqoOHDigxMREdezYUREREdafTZs2OXLzSpWz3pOPPPKIhg4dqmnTpqlfv35yc3PTwoULy9Qhwj/j77djOKuP48eP17333qvo6Gj1799fOTk5WrRoUb4n6JaWfv362VxKxGQyacGCBZKkhx56SOPGjVOXLl30wgsvSLp+svL999+vsWPHWr+x5QxNmjTRiy++qCVLlqh79+568803NWfOHOse3MjISM2ePTvfr6Xfc889mjp1qhYuXKiePXtq+/btWrJkiQIDA+2ux81i74E6AACAMqLs/jcKAACgkAg8AADA8Ag8AADA8Ag8AADA8Ag8AADA8Ag8AADA8Ag8AADA8Ag8AEpUUFCQgoKCdPbs2TzPrVq1SkFBQfleeCw/Fy9e1ObNm2867/Tp0woKCtLp06eLXK8jrFu3rszd0gEwOgIPgBLn6emp7du35xnftm1bke7788orr+irr75yZGkA/iYIPABKXFhYWJ7Ak56ergMHDig4OLjQy+HC8ADsReABUOK6dOmiPXv2KD093Tr25ZdfKiwsTBUqVLCOZWdna9asWWrfvr2aNm2qzp076/3335ckxcbGav369Vq/fr31cNHFixc1duxYtWzZUu3atdPcuXNtQtG2bdvUtWtXNW/eXCNGjFBaWlq+9cXGxmrUqFF65JFH1Lp1a+3Zs0cpKSmKiorSnXfeqTvuuEN9+vSx3uX5xiGzzz77TF27dlVISIieeOKJfO/AnZubq6ioKPXu3VuXL18udi8B2IfAA6DENWrUSNWrV9fOnTutY59//rm6du1qM2/RokX68ssvFRsbqy1btuiBBx7QjBkzdOHCBQ0ZMkTdu3dX9+7dtXbtWknSk08+qdTUVK1cuVKvv/661q1bp/fee8+6vPXr12vu3Llavny5fvzxRy1evLjAGr/44gv17NlT7777rpo1a6ZnnnlGZrNZq1ev1oYNG1S9enVNmzbN5jVvvvmm5s6dq5UrVyouLk5vv/12nuW+9NJLio+P19KlS+Xn52dP+wA4gIezCwDw99ClSxdt375dPXr0UHZ2tr755htNmTJFn3zyiXVO48aN1aZNG7Vo0UKSNGLECM2fP1/Hjx9XWFiYypUrJ+n63aDj4+N14MABbdu2TXXq1JEkTZs2TRkZGdbljR8/Xs2aNZMkde/eXfHx8QXWV7VqVQ0YMEDS9UNnXbt21b333qsaNWpIun5n9+HDh9u8Jioqyrr8Xr165bn79uLFi7VlyxatWrVKVatWLXLPADgOgQdAqejSpYuioqKUk5Oj7777To0aNVKVKlVs5nTt2lXffPON/vWvfyk5OVlHjhyRJJnN5jzLO3bsmPz9/a1h58brJVm/nVW3bl3rc7fccouysrIKrK927drW37u5uWnAgAHatGmT9u/fr2PHjumHH35Qbm6uzWvq1atn/b2vr6+uXbtmfXz+/Hm99tprqlGjhgICAgpuDIBSwSEtAKWiVatWkqR9+/Zp27Ztuvvuu/PMee211zR+/Hh5eHjogQcesJ6/kx9PT8+brtPdvfAfcd7e3tbf5+bmasiQIVq2bJlq1aqlxx9/XLNnzy5SDW5ublq6dKksFosWLlxY6DoAlAwCD4BS4eHhoQ4dOmj79u3asWNHnvN3JGn16tV6/vnn9cwzz6hHjx7KzMyU9Pu3s/74FfZ69erp0qVLOnfunHVs+fLlGjVqVLFrTUxM1N69e/XOO+9oxIgR6tixo86fP29Ty80EBASobdu2Gj9+vJYtW6YTJ04Uuy4A9iPwACg1Xbp00Zo1a1SlShWbQ1E3+Pv7a8eOHTp16pS+//57TZgwQdL1b29Jko+Pj86cOaOUlBTdfvvtatOmjSZPnqyEhATt3r1bixYtUrt27Ypdp5+fn9zd3bVx40adOXNGW7ZssV4c8UYthdWjRw+1aNFCM2bMKHZdAOxH4AFQaiIiIpSTk5Pv3h3p+jeafvrpJ913332KiYlRt27d1KxZM/3000+SpN69e+vYsWO6//77ZbFYNGfOHPn4+Ojhhx9WdHS0Hn74YQ0cOLDYddaoUUPTpk3T4sWL1bNnTy1atEjPPfecPDw8rOcVFcXkyZP17bff6rPPPit2bQDs42bhSl4AAMDg2MMDAAAMj8ADAAAMj8ADAAAMj8ADAAAMj8ADAAAMj8ADAAAMj8ADAAAMj8ADAAAMj8ADAAAMj8ADAAAMj8ADAAAMj8ADAAAM7/8BGaJxclm/YaUAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Analyze results of the documentation search\n",
    "ans_doc = \"../data/external/rosaworkshop/14-faq.md\"\n",
    "c1, c2, c3, c4, c5, c0 = 0, 0, 0, 0, 0, 0\n",
    "for i, row in s1_answers.iterrows():\n",
    "    if ans_doc in row['A1-source']:c1+=1\n",
    "    elif ans_doc in row['A2-source']:c2+=1\n",
    "    elif ans_doc in row['A3-source']:c3+=1\n",
    "    elif ans_doc in row['A4-source']:c4+=1\n",
    "    elif ans_doc in row['A5-source']:c5+=1\n",
    "    else: c0+=1\n",
    "# Define the data\n",
    "x = ['Rank 1', 'Rank 2', 'Rank 3', 'Rank 4', 'Rank 5', 'Not in first 5']\n",
    "y = [2, 6, 9, 19, 14, 15]\n",
    "\n",
    "# Set the Seaborn style\n",
    "sns.set_style('whitegrid')\n",
    "\n",
    "# Create the bar chart using Seaborn\n",
    "sns.barplot(x=x, y=y)\n",
    "\n",
    "# Add some labels and titles\n",
    "plt.xlabel('Match rank')\n",
    "plt.ylabel('Frequency')\n",
    "plt.title('Bar Chart')\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84e24a61",
   "metadata": {},
   "source": [
    "### Next steps for documents retrieval\n",
    "* Calculate Mean reciprocal rank and improve the ranks of the documents retrieved\n",
    "* Create a more comprehensive testing set including questions that are semantically similar but syntatically different with FAQ questions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dd03fc3-3e3c-474e-98a7-f38cacf7c220",
   "metadata": {},
   "source": [
    "## Solution 2: Answer Generation after documentation search\n",
    "\n",
    "In the next section, we will demonstrate a text generation approach to answer questions. The algorithm takes user queries and generates the answer word by word. Each word in the output is conditioned on the input and the previous output words. This approach builds on the embedding based search and uses its result to provide context as a part of the input to the generative model. The benefits of the model are:\n",
    "* Users don't have to search for the answer in a document\n",
    "* It has a chat like capability to iteratively refine informaton retrieval\n",
    "\n",
    "The challenges with this approach are:\n",
    "* Generation can be grammatically correct but factually incorrect\n",
    "* It requires intesive compute\n",
    "* It is an emerging space with rapid changes in technology"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "20104763-9d31-4b7d-93d1-47b5dfcf225d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type=\"stuff\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1c838cb2-58d9-4017-90b9-0822ac8e2847",
   "metadata": {},
   "outputs": [],
   "source": [
    "def answer_question(query, index, chain):\n",
    "    \"\"\"\n",
    "    Takes in query, index to search from, and llm chain to generate answer\n",
    "    \"\"\"\n",
    "    ## Retrieve docs\n",
    "    docs = index.similarity_search(query)\n",
    "    ## Generate answer\n",
    "    answer = chain({\"input_documents\": docs, \"question\": query}, return_only_outputs=True)\n",
    "    return answer['output_text']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc0d1607-c370-42ff-8741-890cd5886d69",
   "metadata": {},
   "source": [
    "### Generate answers for the FAQ dataset questions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0bd12934-3eda-47d8-b263-1640db43581d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/65 done\n",
      "2/65 done\n",
      "3/65 done\n",
      "4/65 done\n",
      "5/65 done\n",
      "6/65 done\n",
      "7/65 done\n",
      "8/65 done\n",
      "9/65 done\n",
      "10/65 done\n",
      "11/65 done\n",
      "12/65 done\n",
      "13/65 done\n",
      "14/65 done\n",
      "15/65 done\n",
      "16/65 done\n",
      "17/65 done\n",
      "18/65 done\n",
      "19/65 done\n",
      "20/65 done\n",
      "21/65 done\n",
      "22/65 done\n",
      "23/65 done\n",
      "24/65 done\n",
      "25/65 done\n",
      "26/65 done\n",
      "27/65 done\n",
      "28/65 done\n",
      "29/65 done\n",
      "30/65 done\n",
      "31/65 done\n",
      "32/65 done\n",
      "33/65 done\n",
      "34/65 done\n",
      "35/65 done\n",
      "36/65 done\n",
      "37/65 done\n",
      "38/65 done\n",
      "39/65 done\n",
      "40/65 done\n",
      "41/65 done\n",
      "42/65 done\n",
      "43/65 done\n",
      "44/65 done\n",
      "45/65 done\n",
      "46/65 done\n",
      "47/65 done\n",
      "48/65 done\n",
      "49/65 done\n",
      "50/65 done\n",
      "51/65 done\n",
      "52/65 done\n",
      "53/65 done\n",
      "54/65 done\n",
      "55/65 done\n",
      "56/65 done\n",
      "57/65 done\n",
      "58/65 done\n",
      "59/65 done\n",
      "60/65 done\n",
      "61/65 done\n",
      "62/65 done\n",
      "63/65 done\n",
      "64/65 done\n",
      "65/65 done\n"
     ]
    }
   ],
   "source": [
    "questions = eval_dataset['Question'].tolist()\n",
    "real_answers = eval_dataset['Answer'].tolist()\n",
    "generated_answers = list()\n",
    "i=1\n",
    "for query in questions:\n",
    "    answer = answer_question(query, docsearch, chain)\n",
    "    generated_answers.append(answer)\n",
    "    ## Add delay to avoid rate limit error\n",
    "    time.sleep(1)\n",
    "    print(f\"{i}/{len(eval_dataset)} done\")\n",
    "    i+=1\n",
    "    \n",
    "eval_dataset['generated_answers'] = generated_answers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa4857d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save data for analysis\n",
    "eval_dataset.to_csv(\"../data/results/openai-0/generated-answers.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b7c159d",
   "metadata": {},
   "source": [
    "### Qualitative Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "680cb607",
   "metadata": {},
   "source": [
    "Next, we tried qualitatively evaluating the outputs of the model: \n",
    "\n",
    "Example 1\n",
    "- Question: _What exactly am I responsible for and what is Red Hat / AWS responsible for?_ \n",
    "- Actual Answer: \"In short, anything that is related to deploying the cluster or keeping the cluster running will be Red Hat’s or AWS’s responsibility, and anything relating to the applications, users, or data is the customers responsibility.  Please see our [responsibility matrix](https://docs.openshift.com/rosa/rosa_architecture/rosa_policy_service_definition/rosa-policy-responsibility-matrix.html) for more details.\"\n",
    "- Generated Answer:   Anything related to deploying the cluster or keeping the cluster running is Red Hat's or AWS's responsibility, and anything relating to the applications, users, or data is the customer's responsibility. Please see the responsibility matrix for more details.\n",
    "- **Observations: The answer is correct here but the link to the document is missing.**\n",
    "\n",
    "Example 2\n",
    "- Question: _Do I need to sign/have a contract with Red Hat?_ \n",
    "- Actual Answer: \"No. You do not need to have a contract with Red Hat to use ROSA. You will need a Red Hat account for use on console.redhat.com which includes accepting our Enterprise Agreement and Online services terms.\"\n",
    "- Generated Answer:   No, you do not need to sign/have a contract with Red Hat.\n",
    "- **Observations: The answer is correct but a subset of the actual answer. Model outputs are brief and partially correct frequently. They are exact responses to the prompts, for example they wouldn’t include links and descriptions if the input prompt doesn’t ask for them.**\n",
    "\n",
    "Example 3\n",
    "- Question: _Where can I go to get more information/details?_ \n",
    "- Actual Answer: \"[ROSA Webpage](https://www.openshift.com/products/amazon-openshift)...\"\n",
    "- Generated Answer:  You can find more information and details on the Home page of the OCM UI, which includes links to additional resources.\n",
    "- **Observations: If the input questions are out of context and they don’t mention ROSA, it’ll be very hard to get sensible answers. In the original FAQ document, they had a context that they belonged to the ROSA workshop FAQ document but here the context is not available.**\n",
    "\n",
    "Example 4\n",
    "- Question: _Which compliance certifications does ROSA have so far?_\n",
    "- Actual Answer: Red Hat OpenShift Service on AWS is currently compliant with SOC-1, SOC-2 type 1 & type 2, ISO-27001, & PCI-DSS. We are also currently working towards FedRAMP High, HIPAA, ISO 27017 and ISO 27018 as well.\n",
    "- Generated Answer: ROSA has not yet obtained any compliance certifications.\n",
    "- **Observation: The answer is completely incorrect here.**\n",
    "\n",
    "\n",
    "Some other examples that we evaluated can be seen in here ([1](https://docs.google.com/spreadsheets/d/1XQUn9IgzVXxTQxRdVWG7xfQOuowGIFLorXECKUKi16w/edit#gid=8386486),[2](https://docs.google.com/spreadsheets/d/1Z0m4FqRsWfV4PDWDZSK3vhSyf8sth7IUBAaFW7x-6mw/edit#gid=1290796759))\n",
    "\n",
    "We also have the quantitaitve metrics in the metrics notebook.\n",
    "\n",
    "The answers can be improved by writing tailored prompts for the task. There are different ways of prompt engineering and some of them are listed here [1](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/prompt-engineering):\n",
    "* Primary content or the system prompt or the main instruction: Adding a detailed context around the problem and assigning a persona to the model\n",
    "* Few-shot learning: Adding to show how the output looks \n",
    "* Cue: Adding a \"jumpstart\" phrase or cue to help the model generate the answer\n",
    "* Fallback: Adding an instruction for what to do when the model doesn't know the answer\n",
    "* Best practices: Adding instructions that are specific, descriptive, chronological, and emphasizes important aspects"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fff15494",
   "metadata": {},
   "source": [
    "### Prompt Generation\n",
    "The prompt was written following these prompt engineering [guidelines](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/prompt-engineering) and chat GPT. The template for this will be added to the repository. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "bfe40c7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "template = \"\"\"\n",
    "You are a support engineer who is trying to generate answers for questions around the ROSA product. Your goal is to answer questions that ROSA customers and users would find relevant, informative, and useful. You should be descriptive and provide links to support your answer. \n",
    "\n",
    "Here is a description of the product:\n",
    "ROSA is a fully-managed, turnkey application platform that allows you to focus on delivering value to your customers by building and deploying applications. You can learn more about it in the documentaion here: https://docs.openshift.com/rosa/welcome/index.html\n",
    "\n",
    "Here are a few input and output pairs examples to guide the model:\n",
    "\n",
    "Input: \"What is Red Hat OpenShift Service on AWS (ROSA)?\"\n",
    "Output: \"Red Hat Openshift Service on AWS (ROSA) is a fully-managed turnkey application platform that allows you to focus on what matters most, delivering value to your customers by building and deploying applications. Red Hat SRE experts manage the underlying platform so you don’t have to worry about the complexity of infrastructure management.\"\n",
    "\n",
    "Input: \"Where can I go to get more information/details?\"\n",
    "Output: \"You can find more information at the following links [ROSA Webpage](https://www.openshift.com/products/amazon-openshift)\n",
    "- [ROSA Workshop](https://www.rosaworkshop.io)\n",
    "- [ROSA Documentation](https://docs.openshift.com/rosa/welcome/index.html)\"\n",
    "\n",
    "\n",
    "If you don't know the answer, please respond with: \"I'm sorry, I don't have enough information to generate that content.\n",
    "Please help this ROSA customer to find the answer to this {question} given the relevant content around the question {context}:\n",
    "\"\"\"\n",
    "\n",
    "prompt = PromptTemplate(\n",
    "    input_variables=[\"question\", \"context\"],\n",
    "    template=template,\n",
    ")\n",
    "chain = load_qa_chain(OpenAI(temperature=0), chain_type=\"stuff\", prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "4422584c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/65 done\n",
      "2/65 done\n",
      "3/65 done\n",
      "4/65 done\n",
      "5/65 done\n",
      "6/65 done\n",
      "7/65 done\n",
      "8/65 done\n",
      "9/65 done\n",
      "10/65 done\n",
      "11/65 done\n",
      "12/65 done\n",
      "13/65 done\n",
      "14/65 done\n",
      "15/65 done\n",
      "16/65 done\n",
      "17/65 done\n",
      "18/65 done\n",
      "19/65 done\n",
      "20/65 done\n",
      "21/65 done\n",
      "22/65 done\n",
      "23/65 done\n",
      "24/65 done\n",
      "25/65 done\n",
      "26/65 done\n",
      "27/65 done\n",
      "28/65 done\n",
      "29/65 done\n",
      "30/65 done\n",
      "31/65 done\n"
     ]
    }
   ],
   "source": [
    "eval_dataset = pd.read_csv('../data/processed/validation_data.csv')[['Question', 'Answer']]\n",
    "questions = eval_dataset['Question'].tolist()\n",
    "real_answers = eval_dataset['Answer'].tolist()\n",
    "generated_answers = list()\n",
    "i=1\n",
    "for query in eval_dataset['Question'].tolist():\n",
    "    answer = answer_question(query, docsearch, chain)\n",
    "    generated_answers.append(answer)\n",
    "    ## Add delay to avoid rate limit error\n",
    "    time.sleep(1)\n",
    "    print(f\"{i}/{len(eval_dataset)} done\")\n",
    "    i+=1\n",
    "    \n",
    "eval_dataset['generated_answers'] = generated_answers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b5ec96f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save data for analysis\n",
    "eval_dataset.to_csv(\"../data/results/openai-1/generated-answers.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3f61363",
   "metadata": {},
   "source": [
    "### Qualitative evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "d0471580",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\n- [Adding support contacts](https://docs.openshift.com/rosa/4.5/managing_clusters/adding_support_contacts.html)\\n\\n#### Configuration examples\\n\\nYou can find configuration examples for ROSA in the [documentation](https://docs.openshift.com/rosa/4.5/configuring_clusters/configuring_clusters.html).\\n\\n#### Pricing\\n\\nYou can find pricing information for ROSA on the [Red Hat OpenShift Service on AWS](https://www.openshift.com/products/amazon-openshift) page.',\n",
       " '\\nRed Hat OpenShift Service on AWS is managed by Red Hat SRE experts who will monitor and maintain the underlying platform so you don’t have to worry about the complexity of infrastructure management. Red Hat will also provide support for the OpenShift software component.',\n",
       " '\\nYes, all Red Hat OpenShift Service on AWS 4.8 clusters require an administrator acknowledgment before they can be upgraded to Red Hat OpenShift Service on AWS 4.9. This acknowledgment is required to ensure that the customer is aware of the changes and new features that are included in the upgrade. The acknowledgment can be found in the Red Hat OpenShift Service on AWS console.',\n",
       " '\\nFor more information, please refer to the [Red Hat OpenShift Service on AWS documentation](https://docs.openshift.com/rosa/welcome/index.html).']"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qs = [\"What exactly am I responsible for and what is Red Hat / AWS responsible for?\",\n",
    "             \"Do I need to sign/have a contract with Red Hat?\",\n",
    "             \"Where can I go to get more information/details?\",\n",
    "             \"Which compliance certifications does ROSA have so far?\"]\n",
    "\n",
    "eval_dataset[eval_dataset['Question'].isin(qs)]['generated_answers'].to_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "426c57e5",
   "metadata": {},
   "source": [
    "Example 1\n",
    "- Question: _What exactly am I responsible for and what is Red Hat / AWS responsible for?_ \n",
    "- Actual Answer: \"In short, anything that is related to deploying the cluster or keeping the cluster running will be Red Hat’s or AWS’s responsibility, and anything relating to the applications, users, or data is the customers responsibility.  Please see our [responsibility matrix](https://docs.openshift.com/rosa/rosa_architecture/rosa_policy_service_definition/rosa-policy-responsibility-matrix.html) for more details.\"\n",
    "- Generated Answer: Red Hat OpenShift Service on AWS is managed by Red Hat SRE experts who will monitor and maintain the underlying platform so you don’t have to worry about the complexity of infrastructure management. Red Hat will also provide support for the OpenShift software component.'\n",
    "- **Observations: The answer is partially correct, it doesn't go into user responsibilities.**\n",
    "\n",
    "Example 2\n",
    "- Question: _Do I need to sign/have a contract with Red Hat?_ \n",
    "- Actual Answer: \"No. You do not need to have a contract with Red Hat to use ROSA. You will need a Red Hat account for use on console.redhat.com which includes accepting our Enterprise Agreement and Online services terms.\"\n",
    "- Generated Answer:  Yes, all Red Hat OpenShift Service on AWS 4.8 clusters require an administrator acknowledgment before they can be upgraded to Red Hat OpenShift Service on AWS 4.9. This acknowledgment is required to ensure that the customer is aware of the changes and new features that are included in the upgrade. The acknowledgment can be found in the Red Hat OpenShift Service on AWS console.',\n",
    "- **Observations: It is incorrect as it understands it as a contract for upgrade.**\n",
    "\n",
    "Example 3\n",
    "- Question: _Where can I go to get more information/details?_ \n",
    "- Actual Answer: \"[ROSA Webpage](https://www.openshift.com/products/amazon-openshift)...\"\n",
    "- Generated Answer:  [Adding support contacts](https://docs.openshift.com/rosa/4.5/managing_clusters/adding_support_contacts.html)\\n\\n#### Configuration examples\\n\\nYou can find configuration examples for ROSA in the [documentation](https://docs.openshift.com/rosa/4.5/configuring_clusters/configuring_clusters.html).\\n\\n#### Pricing\\n\\nYou can find pricing information for ROSA on the [Red Hat OpenShift Service on AWS](https://www.openshift.com/products/amazon-openshift) page.\n",
    "- **Observations: The links are different and two of them don't work.**\n",
    "\n",
    "Example 4\n",
    "- Question: _Which compliance certifications does ROSA have so far?_\n",
    "- Actual Answer: Red Hat OpenShift Service on AWS is currently compliant with SOC-1, SOC-2 type 1 & type 2, ISO-27001, & PCI-DSS. We are also currently working towards FedRAMP High, HIPAA, ISO 27017 and ISO 27018 as well.\n",
    "- Generated Answer: For more information, please refer to the [Red Hat OpenShift Service on AWS documentation](https://docs.openshift.com/rosa/welcome/index.html)\n",
    "- **Observation: The answer is not incorrect here.**\n",
    "\n",
    "Overall, the questions that lacked context have better answers but there are still incorrect responses. Let's try tweaking the prompt."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0851a78a",
   "metadata": {},
   "source": [
    "### Another tweak with reduced noise in the prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c896409",
   "metadata": {},
   "outputs": [],
   "source": [
    "template = \"\"\"\n",
    "You are a support engineer generating answers for questions around the Red Hat Openshift Service on AWS (ROSA) product. \n",
    "Your goal is to answer questions that ROSA customers and users would find correct and useful. \n",
    "You should be descriptive and provide documentation web links to support your answer. \n",
    "\n",
    "Here are a few input and output pairs examples to guide the model:\n",
    "\n",
    "I am a ROSA user and I want to know the answer to this question: \"What is Red Hat OpenShift Service on AWS (ROSA)?\"\n",
    "Answer: \"Red Hat Openshift Service on AWS (ROSA) is a fully-managed turnkey application platform that allows you to focus on what matters most, delivering value to your customers by building and deploying applications. Red Hat SRE experts manage the underlying platform so you don’t have to worry about the complexity of infrastructure management.\"\n",
    "\n",
    "I am a ROSA user and I want to know the answer to this question:: \"Where can I go to get more information/details?\"\n",
    "Answer: \"You can find more information at the following links [ROSA Webpage](https://www.openshift.com/products/amazon-openshift)\n",
    "- [ROSA Workshop](https://www.rosaworkshop.io)\n",
    "- [ROSA Documentation](https://docs.openshift.com/rosa/welcome/index.html)\"\n",
    "\n",
    "\n",
    "If you don't know the answer, please respond with: \"I'm sorry, I don't have enough information to generate that content.\n",
    "Use the above guidelines to answer the following question:\n",
    "I am a ROSA user and I want to know the answer to this question {question}. You can use the {context} for help.\n",
    "Answer:\n",
    "\"\"\"\n",
    "\n",
    "prompt = PromptTemplate(\n",
    "    input_variables=[\"question\", \"context\"],\n",
    "    template=template,\n",
    ")\n",
    "chain = load_qa_chain(OpenAI(temperature=0), chain_type=\"stuff\", prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e24b1d1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/65 done\n",
      "2/65 done\n",
      "3/65 done\n",
      "4/65 done\n",
      "5/65 done\n",
      "6/65 done\n",
      "7/65 done\n",
      "8/65 done\n",
      "9/65 done\n",
      "10/65 done\n",
      "11/65 done\n",
      "12/65 done\n",
      "13/65 done\n",
      "14/65 done\n",
      "15/65 done\n",
      "16/65 done\n",
      "17/65 done\n",
      "18/65 done\n",
      "19/65 done\n",
      "20/65 done\n",
      "21/65 done\n",
      "22/65 done\n",
      "23/65 done\n",
      "24/65 done\n",
      "25/65 done\n",
      "26/65 done\n",
      "27/65 done\n",
      "28/65 done\n",
      "29/65 done\n",
      "30/65 done\n",
      "31/65 done\n",
      "32/65 done\n",
      "33/65 done\n",
      "34/65 done\n",
      "35/65 done\n",
      "36/65 done\n",
      "37/65 done\n",
      "38/65 done\n",
      "39/65 done\n",
      "40/65 done\n",
      "41/65 done\n",
      "42/65 done\n",
      "43/65 done\n",
      "44/65 done\n",
      "45/65 done\n",
      "46/65 done\n",
      "47/65 done\n",
      "48/65 done\n",
      "49/65 done\n",
      "50/65 done\n",
      "51/65 done\n",
      "52/65 done\n",
      "53/65 done\n",
      "54/65 done\n",
      "55/65 done\n",
      "56/65 done\n",
      "57/65 done\n",
      "58/65 done\n",
      "59/65 done\n",
      "60/65 done\n",
      "61/65 done\n",
      "62/65 done\n",
      "63/65 done\n",
      "64/65 done\n",
      "65/65 done\n"
     ]
    }
   ],
   "source": [
    "eval_dataset = pd.read_csv('../data/processed/validation_data.csv')[['Question', 'Answer']]\n",
    "questions = eval_dataset['Question'].tolist()\n",
    "real_answers = eval_dataset['Answer'].tolist()\n",
    "generated_answers = list()\n",
    "i=1\n",
    "for query in questions:\n",
    "    answer = answer_question(query, docsearch, chain)\n",
    "    generated_answers.append(answer)\n",
    "    ## Add delay to avoid rate limit error\n",
    "    time.sleep(1)\n",
    "    print(f\"{i}/{len(eval_dataset)} done\")\n",
    "    i+=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "660ba04e",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_dataset['generated_answers'] = generated_answers\n",
    "eval_dataset.to_csv(\"../data/results/openai-2/generated-answers.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "328a1c1b",
   "metadata": {},
   "source": [
    "Example 1\n",
    "- Question: _What exactly am I responsible for and what is Red Hat / AWS responsible for?_ \n",
    "- Actual Answer: \"In short, anything that is related to deploying the cluster or keeping the cluster running will be Red Hat’s or AWS’s responsibility, and anything relating to the applications, users, or data is the customers responsibility.  Please see our [responsibility matrix](https://docs.openshift.com/rosa/rosa_architecture/rosa_policy_service_definition/rosa-policy-responsibility-matrix.html) for more details.\"\n",
    "- Generated Answer: 'Red Hat OpenShift Service on AWS (ROSA) is a fully-managed turnkey application platform that allows you to focus on what matters most, delivering value to your customers by building and deploying applications. Red Hat SRE experts manage the underlying platform so you don’t have to worry about the complexity of infrastructure management. \\n\\nWhen you provision ROSA clusters, you will incur the infrastructure and OpenShift charges at the pay-as-you-go hourly rate. Red Hat and AWS are responsible for the monitoring and maintenance of the ROSA cluster, while the customer is responsible for the applications, workloads, and data that they deploy to the cluster. For more information, please refer to our [responsibility matrix](https://docs.openshift.com/rosa/rosa_architecture/rosa_policy_service_definition/rosa-policy-responsibility-matrix.html) and [pricing page](https://www.openshift.com/products/amazon-openshift/pricing).'\n",
    "- **Observations: The answer is correct and comprehensive**\n",
    "\n",
    "Example 2\n",
    "- Question: _Do I need to sign/have a contract with Red Hat?_ \n",
    "- Actual Answer: \"No, you do not need to sign/have a contract with Red Hat to use ROSA. You will need a Red Hat account for use on console.redhat.com which includes accepting our Enterprise Agreement and Online services terms. All Red Hat OpenShift Service on AWS 4.8 clusters require this administrator acknowledgment before they can be upgraded to Red Hat OpenShift Service on AWS 4.9. For more information, please refer to the [ROSA Documentation](https://docs.openshift.com/rosa/welcome/index.html)',\n",
    "- **Observations: The first part of the answer is completely correct. Although it adds irrelevant information.**\n",
    "\n",
    "Example 3\n",
    "- Question: _Where can I go to get more information/details?_ \n",
    "- Actual Answer: \"[ROSA Webpage](https://www.openshift.com/products/amazon-openshift)...\"\n",
    "- Generated Answer: You can find more information about obtaining support for Red Hat OpenShift Service on AWS (ROSA) at the following links: \\n- [ROSA Home Page](../../#resources)\\n- [Adding Support Contacts](https://docs.openshift.com/rosa/4.5/managing_clusters/managing_cluster_support_contacts.html)\\n- [ROSA Documentation](https://docs.openshift.com/rosa/welcome/index.html)'.\n",
    "- **Observations: The links are different and two of them don't work.**\n",
    "\n",
    "Example 4\n",
    "- Question: _Which compliance certifications does ROSA have so far?_\n",
    "- Actual Answer: Red Hat OpenShift Service on AWS is currently compliant with SOC-1, SOC-2 type 1 & type 2, ISO-27001, & PCI-DSS. We are also currently working towards FedRAMP High, HIPAA, ISO 27017 and ISO 27018 as well.\n",
    "- Generated Answer: No, all nodes in a Red Hat OpenShift Service on AWS cluster must be located in the same AWS region; this follows the same model as that of OCP. For clusters configured for multiple availability zones control plane nodes and worker nodes will be distributed across the availability zones. For more information, please refer to the [Red Hat OpenShift Service on AWS documentation](https://docs.openshift.com/rosa/welcome/index.html).\n",
    "- **Observation: The answer is incorrect here.**\n",
    "\n",
    "The prompt needs to be refined further for this domain."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dc6d0f6-526b-4e4c-96cd-106dfa6e87e8",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f16458d9-e07d-4f5c-9f11-a1008ad7c12a",
   "metadata": {},
   "source": [
    "In this work, we generated answers using the OpenAI language model and evaluated their quality against the ground truth answers for a set of questions in the FAQ document.  We discussed two approaches: embedding search and text generation. \n",
    "\n",
    "The next steps here are to improve the results we obtained with both the approaches. Parallely, we want to think about the tools and pipeling required to bring the first version in front of support engineers. Additionally, we would require more examples of question and answers around ROSA. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}