chapter_11_evaluating_outputs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 11: Evaluating LLM Outputs\n",
    "\n",
    "In previous chapters, we explored how to guide LLMs towards generating structured outputs. While this approach is effective, it quickly becomes cumbersome to manually evaluate the outputs every time we build an AI agent or a workflow. Running inputs through the model and verifying each result by hand is not only time-consuming but also prone to inconsistency. This chapter introduces a more systematic approach: automated evaluation using test cases. By creating a small test set, consisting of around 10 test cases, we can streamline the process of validating LLM outputs. Furthermore, if we’ve already defined the expected output type from the LLM, we can also establish clear criteria for what the final answer should look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from pydantic import BaseModel, EmailStr, Field\n",
    "from collections import Counter\n",
    "from datetime import datetime\n",
    "from language_models.agent import Agent, OutputType, PromptingStrategy, get_schema_from_args\n",
    "from language_models.models.llm import OpenAILanguageModel\n",
    "from language_models.models.embedding import SentenceTransformerEmbeddingModel\n",
    "from language_models.proxy_client import ProxyClient\n",
    "from language_models.settings import settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "proxy_client = ProxyClient(\n",
    "    client_id=settings.CLIENT_ID,\n",
    "    client_secret=settings.CLIENT_SECRET,\n",
    "    auth_url=settings.AUTH_URL,\n",
    "    api_base=settings.API_BASE,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = OpenAILanguageModel(\n",
    "    proxy_client=proxy_client,\n",
    "    model=\"gpt-4\",\n",
    "    max_tokens=500,\n",
    "    temperature=0.2,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To grasp the basics of evaluating LLM outputs, we'll walk through a few examples. Rather than building an evaluation framework from scratch, we'll focus on a more streamlined, budget-friendly approach. The key takeaway is understanding how to compare outputs, which should feel familiar if you have experience in software development, as the principles are quite similar.\n",
    "\n",
    "**String**\n",
    "\n",
    "When evaluating string outputs, one straightforward method is to perform a direct equality comparison, particularly when the output is a single string or a distinct category. This method works well for cases where the expected output is clear-cut, such as predefined labels, specific commands, or short, unambiguous responses.\n",
    "\n",
    "However, not all LLM outputs are this simple. In cases where the output is a sentence or a paragraph, direct comparison may not be sufficient. For these scenarios, we can compute the similarity between the expected answer and the LLM-generated output. In addition to cosine similarity, we can use specialized metrics like BLEU and ROUGE scores to evaluate how closely the generated text matches the expected output, accounting for variations in content, structure, and phrasing.\n",
    "\n",
    "Additionally, it is possible to use another LLM to evaluate the output (correct/incorrect answer or rate answer quality from 1 to 10). However, one should exercise caution, as the evaluating LLM might have a tendency to prefer longer outputs or those that resemble its own writing style. This could potentially introduce bias into the evaluation process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = \"You are an AI assistant designed to help users with a variety of tasks.\"\n",
    "\n",
    "agent = Agent.create(\n",
    "    llm=llm,\n",
    "    system_prompt=system_prompt,\n",
    "    prompt=\"{prompt}\",\n",
    "    prompt_variables=[\"prompt\"],\n",
    "    output_type=OutputType.STRING,\n",
    "    prompting_strategy=PromptingStrategy.SINGLE_COMPLETION,\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[38;2;50;164;103mFinal Answer\u001b[0m\u001b[1m\u001b[0m: Paris\n"
     ]
    }
   ],
   "source": [
    "output = agent.invoke({\"prompt\": \"What is the capital city of France?\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paris\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check for equality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer == \"Paris\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check for similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cosine similarity: 1.0000\n"
     ]
    }
   ],
   "source": [
    "embedding_model = SentenceTransformerEmbeddingModel(model=\"all-MiniLM-L6-v2\")\n",
    "embedding1 = embedding_model.embed_query(output.final_answer)\n",
    "embedding2 = embedding_model.embed_query(\"Paris\")\n",
    "cosine_similarity = np.dot(embedding1, embedding2) / (np.linalg.norm(embedding1) * np.linalg.norm(embedding2))\n",
    "print(f\"Cosine similarity: {cosine_similarity:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Integer/Float**\n",
    "\n",
    "When evaluating integer or float outputs, we can use a variety of comparison methods depending on the context. A direct `==` check can be used for exact matches, while comparisons such as `<`, `<=`, `>`, or `>=` are useful when the output needs to be within a certain threshold. Additionally, we can verify if the value falls within a specific range, which is particularly helpful for scenarios where the exact number isn't as important as ensuring it lies within acceptable bounds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = \"You are an AI assistant designed to help users with a variety of tasks.\"\n",
    "\n",
    "agent = Agent.create(\n",
    "    llm=llm,\n",
    "    system_prompt=system_prompt,\n",
    "    prompt=\"{prompt}\",\n",
    "    prompt_variables=[\"prompt\"],\n",
    "    output_type=OutputType.FLOAT,\n",
    "    prompting_strategy=PromptingStrategy.SINGLE_COMPLETION,\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[38;2;50;164;103mFinal Answer\u001b[0m\u001b[1m\u001b[0m: 3.14\n"
     ]
    }
   ],
   "source": [
    "output = agent.invoke({\"prompt\": \"What is the value of Pi up to two decimal places?\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.14\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check for equality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer == 3.14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if the final answer is within a specific range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(3.13 < output.final_answer < 3.15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Boolean**\n",
    "\n",
    "When evaluating boolean outputs, the process is straightforward. We simply check if the output matches the expected boolean value, whether it's True or False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = \"You are an AI assistant designed to help users with a variety of tasks.\"\n",
    "\n",
    "agent = Agent.create(\n",
    "    llm=llm,\n",
    "    system_prompt=system_prompt,\n",
    "    prompt=\"{prompt}\",\n",
    "    prompt_variables=[\"prompt\"],\n",
    "    output_type=OutputType.BOOLEAN,\n",
    "    prompting_strategy=PromptingStrategy.SINGLE_COMPLETION,\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[38;2;50;164;103mFinal Answer\u001b[0m\u001b[1m\u001b[0m: True\n"
     ]
    }
   ],
   "source": [
    "output = agent.invoke({\"prompt\": \"Is the number 5 greater than 3?\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer == True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Date/Timestamp**\n",
    "\n",
    "When evaluating dates or timestamps, we check if the output matches the expected value and adheres to the specified format. This involves verifying that the date or timestamp is correctly formatted and accurately represents the intended moment in time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = \"You are an AI assistant designed to help users with a variety of tasks.\"\n",
    "\n",
    "agent = Agent.create(\n",
    "    llm=llm,\n",
    "    system_prompt=system_prompt,\n",
    "    prompt=\"{prompt}\",\n",
    "    prompt_variables=[\"prompt\"],\n",
    "    output_type=OutputType.DATE,\n",
    "    output_schema=\"%Y-%m-%d\",\n",
    "    prompting_strategy=PromptingStrategy.SINGLE_COMPLETION,\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[38;2;50;164;103mFinal Answer\u001b[0m\u001b[1m\u001b[0m: 2024-04-15\n"
     ]
    }
   ],
   "source": [
    "output = agent.invoke({\"prompt\": \"We are excited to announce that our annual company retreat will be held on April 15, 2024. This event will be a great opportunity for team building and strategic planning.\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2024-04-15\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer == datetime.strptime(\"2024-04-15\", \"%Y-%m-%d\").date())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Array**\n",
    "\n",
    "When evaluating arrays, the approach depends on the expected format and content. If the output array is supposed to be sorted, we can check for equality against a pre-sorted reference array. If the sorting is not a requirement, we can instead verify that all elements meet specific criteria, such as ensuring all elements are the same or checking for other expected patterns or values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = \"\"\"You are an AI assistant designed to help users with a variety of tasks.\n",
    "\n",
    "Extract all numbers from the user's input text.\"\"\"\n",
    "\n",
    "agent = Agent.create(\n",
    "    llm=llm,\n",
    "    system_prompt=system_prompt,\n",
    "    prompt=\"{prompt}\",\n",
    "    prompt_variables=[\"prompt\"],\n",
    "    output_type=OutputType.ARRAY_INTEGER,\n",
    "    prompting_strategy=PromptingStrategy.SINGLE_COMPLETION,\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[38;2;50;164;103mFinal Answer\u001b[0m\u001b[1m\u001b[0m: [6, 15, 7, 10, 4401, 2, 12, 5, 6, 12, 5]\n"
     ]
    }
   ],
   "source": [
    "prompt = \"\"\"Last weekend, six of us went on a 15-kilometer hike, starting at 7 AM.\n",
    "\n",
    "By noon, we had covered 10 kilometers and reached Mount Elbert's 4,401-meter summit by 2 PM, with a temperature of 12°C.\n",
    "\n",
    "We camped 5 kilometers away by 6 PM with 12 others and returned home by 5 PM the next day.\"\"\"\n",
    "\n",
    "output = agent.invoke({\"prompt\": prompt})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6, 15, 7, 10, 4401, 2, 12, 5, 6, 12, 5]\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if all elements are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(Counter(output.final_answer) == Counter([6, 15, 7, 10, 4401, 2, 12, 5, 6, 12, 5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Struct/Object**\n",
    "\n",
    "When comparing structs or objects, we perform equality checks to ensure that all elements match. Since these elements can include various data types—such as arrays, booleans, dates, or other structures - we can also apply the evaluation methods that we have introduced for each type. For instance, arrays are checked for equality or pattern and dates for correct formatting and value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = \"Write an email.\"\n",
    "\n",
    "class Email(BaseModel):\n",
    "    to: list[EmailStr] = Field(description=\"List of people to send it to\")\n",
    "    subject: str = Field(description=\"Subject of the email\")\n",
    "    body: str = Field(description=\"Body of the email\")\n",
    "\n",
    "agent = Agent.create(\n",
    "    llm=llm,\n",
    "    system_prompt=system_prompt,\n",
    "    prompt=\"{prompt}\",\n",
    "    prompt_variables=[\"prompt\"],\n",
    "    output_type=OutputType.OBJECT,\n",
    "    output_schema=Email,\n",
    "    prompting_strategy=PromptingStrategy.SINGLE_COMPLETION,\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[38;2;50;164;103mFinal Answer\u001b[0m\u001b[1m\u001b[0m: to=['johndoe@example.com', 'janedoe@example.com', 'alice.smith@company.org'] subject='Weekend Hiking Trip Recap' body='Dear All,\\n\\nI hope this email finds you well. I am writing to recap our amazing hiking trip last weekend. Six of us embarked on a 15-kilometer hike, starting at 7 AM. By noon, we had covered 10 kilometers and reached the summit of Mount Elbert, standing tall at 4,401 meters, by 2 PM. The weather was quite pleasant with a temperature of 12°C.\\n\\nWe set up our camp 5 kilometers away by 6 PM, where we were joined by 12 other fellow hikers. After a night under the stars, we packed up and returned home by 5 PM the next day.\\n\\nLooking forward to more such adventures in the future.\\n\\nBest,\\n[Your Name]'\n"
     ]
    }
   ],
   "source": [
    "prompt = \"\"\"Send the email to:\n",
    "1. johndoe@example.com\n",
    "2. janedoe@example.com\n",
    "3. alice.smith@company.org\n",
    "\n",
    "Here is what we did: Weekend Hiking Trip Recap\n",
    "\n",
    "Here is some context:\n",
    "Last weekend, six of us went on a 15-kilometer hike, starting at 7 AM.\n",
    "By noon, we had covered 10 kilometers and reached Mount Elbert's 4,401-meter summit by 2 PM, with a temperature of 12°C.\n",
    "We camped 5 kilometers away by 6 PM with 12 others and returned home by 5 PM the next day.\"\"\"\n",
    "\n",
    "output = agent.invoke({\"prompt\": prompt})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if all elements are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(Counter(output.final_answer.to) == Counter([\"johndoe@example.com\", \"janedoe@example.com\", \"alice.smith@company.org\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check for similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cosine similarity: 1.0000\n"
     ]
    }
   ],
   "source": [
    "embedding1 = embedding_model.embed_query(output.final_answer.subject)\n",
    "embedding2 = embedding_model.embed_query(\"Weekend Hiking Trip Recap\")\n",
    "cosine_similarity = np.dot(embedding1, embedding2) / (np.linalg.norm(embedding1) * np.linalg.norm(embedding2))\n",
    "print(f\"Cosine similarity: {cosine_similarity:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check for similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cosine similarity: 0.9664\n"
     ]
    }
   ],
   "source": [
    "embedding1 = embedding_model.embed_query(output.final_answer.body)\n",
    "\n",
    "body = \"\"\"Hello all,\n",
    "\n",
    "I hope this email finds you well.\n",
    "I wanted to share a quick recap of our weekend hiking trip.\n",
    "Last weekend, six of us embarked on a 15-kilometer hike, starting at 7 AM.\n",
    "By noon, we had covered 10 kilometers and reached Mount Elbert's 4,401-meter summit by 2 PM, with a temperature of 12°C.\n",
    "\n",
    "We set up camp 5 kilometers away by 6 PM, where we were joined by 12 others.\n",
    "The next day, we packed up and returned home by 5 PM. It was a memorable experience and I look forward to our next adventure.\n",
    "\n",
    "Best regards,\n",
    "[Your Name]\"\"\"\n",
    "embedding2 = embedding_model.embed_query(body)\n",
    "cosine_similarity = np.dot(embedding1, embedding2) / (np.linalg.norm(embedding1) * np.linalg.norm(embedding2))\n",
    "print(f\"Cosine similarity: {cosine_similarity:.4f}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm-powered-ai-agents",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}