Flashcard_Generator_pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FLASHCARD GENERATOR RAG MODEL (Open Source)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\schla\\Desktop\\Gitlab_Repositories\\master-thesis-ai-meets-classroom\\venv\\Lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n"
     ]
    }
   ],
   "source": [
    "import tools.pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DATA PREPARATION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Loading Wikipedia dataset: language=simple, date=20220301\n"
     ]
    }
   ],
   "source": [
    "data = tools.pipeline.load_wikipedia_dataset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TextCharacter Splitting & Providing Chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Processing documents into chunks\n",
      "INFO:tools.pipeline:Total chunks: 216232, Valid chunks: 179456\n"
     ]
    }
   ],
   "source": [
    "texts, chunk_counts, total_chunks, total_valid_chunks = tools.pipeline.process_documents(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### DATA STORAGE (ChromaDB, Nomic Embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#vectorestore, embeddings = tools.pipeline.setup_vectorstore_nomic(texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Loading vector store\n",
      "INFO:chromadb.telemetry.product.posthog:Anonymized telemetry enabled. See                     https://docs.trychroma.com/telemetry for more information.\n",
      "INFO:chromadb.telemetry.product.posthog:Anonymized telemetry enabled. See                     https://docs.trychroma.com/telemetry for more information.\n",
      "INFO:chromadb.api.segment:Collection opensource_rag_wikipedia is not created.\n",
      "INFO:tools.pipeline:Vector store loaded successfully\n"
     ]
    }
   ],
   "source": [
    "vectorstore, embeddings = tools.pipeline.load_vectorstore_nomic()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RETRIEVER AND GENERATOR (Llama 3.1 Instruct Turbo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic = \"Correlation\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Generating QA pairs\n",
      "INFO:tools.pipeline:Retrieving documents for topic: Correlation with threshold: 0.8\n",
      "INFO:tools.pipeline:Formatting documents\n",
      "ERROR:tools.pipeline:Failed to parse JSON response\n",
      "ERROR:tools.pipeline:Failed to parse JSON response\n",
      "ERROR:tools.pipeline:Failed to parse JSON response\n",
      "ERROR:tools.pipeline:Failed to parse JSON response\n",
      "INFO:tools.pipeline:Generated a total of 39 QA pairs\n"
     ]
    }
   ],
   "source": [
    "flashcard_set = tools.pipeline.generate_question_answer_pairs_open_source_json(topic, vectorstore, threshold = 0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "    {\n",
      "        \"question\": \"What does correlation measure in statistics and probability theory?\",\n",
      "        \"answer\": \"How closely related two sets of data are.\",\n",
      "        \"context\": \"In statistics and probability theory, correlation is a way to indicate how closely related two sets of data are.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is an important consideration when interpreting the relationship between two correlated variables?\",\n",
      "        \"answer\": \"A third factor may be involved, and correlation does not necessarily imply causation.\",\n",
      "        \"context\": \"Correlation does not always mean that one causes the other. In fact, it is very possible that there is a third factor involved.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What happens to two sets of data when they have a positive correlation?\",\n",
      "        \"answer\": \"They go up together.\",\n",
      "        \"context\": \"Correlation usually has one of two directions. These are positive or negative. If it is positive, then the two sets go up together. If it is negative, then one goes up while the other goes down.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What visual tool is often used on a scatter graph to illustrate the direction of a correlation?\",\n",
      "        \"answer\": \"A line of best fit\",\n",
      "        \"context\": \"Lots of different measurements of correlation are used for different situations. For example, on a scatter graph, people draw a line of best fit to show the direction of the correlation.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is indicated by the proximity of points in a correlation graph?\",\n",
      "        \"answer\": \"The proximity of points in a correlation graph indicates a strong correlation, whereas spread-out points indicate a weak correlation.\",\n",
      "        \"context\": \"Strong and weak are words used to describe the strength of correlation. If there is strong correlation, then the points are all close together. If there is weak correlation, then the points are all spread apart.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is the range of values that a correlation coefficient can give, and what do the extreme values represent?\",\n",
      "        \"answer\": \"A correlation coefficient gives a number between -1 and 1. The extreme values, 1 and -1, represent strong correlation, while 0 represents no correlation.\",\n",
      "        \"context\": \"There are ways of making numbers show how strong the correlation is. These measurements are called correlation coefficients. The best known is the Pearson product-moment correlation coefficient, sometimes denoted by  or its Greek equivalent . You put in data into a formula, and it gives you a number between -1 and 1. If the number is 1 or \\u22121, then there is strong correlation. If the answer is 0, then there is no correlation. Another kind of correlation coefficient is Spearman's rank correlation coefficient.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is an important consideration when interpreting the relationship between two correlated events?\",\n",
      "        \"answer\": \"That one event might not be the direct cause of the other, as there could be an underlying factor at play.\",\n",
      "        \"context\": \"Correlation does not always mean that one thing causes the other (causation), because there might be something else that is at play.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is an example of two events that are correlated but not directly causally linked?\",\n",
      "        \"answer\": \"Ice cream sales and shark attacks, as both increase when the temperature rises, but one does not cause the other.\",\n",
      "        \"context\": \"For example, on hot days people buy ice cream, and people also go to the beach where some are eaten by sharks. There is a correlation between ice cream sales and shark attacks (they both go up as the temperature goes up in this case). But just because ice cream sales go up does not mean ice cream sales cause (causation) more shark attacks or vice versa.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"Why do scientists and economists create isolated environments when testing their theories related to correlation?\",\n",
      "        \"answer\": \"To change only one factor at a time, where possible, in order to accurately determine causation.\",\n",
      "        \"context\": \"Because correlation does not imply causation, scientists, economists, etc. will test their theories by creating isolated environments where only one factor is changed (where this is possible). However, politicians, salesmen, news outlets and others often suggest that a particular correlation implies causation. This may be due to ignorance or a wish to persuade. Thus, a news report may attract attention by saying that people who consume a particular product more often have a particular health problem, implying a causation that could be actually due to something else.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is the primary purpose of Pearson's Correlation Coefficient in data analysis?\",\n",
      "        \"answer\": \"The primary purpose of Pearson's Correlation Coefficient is to measure the linear relationship between two continuous variables, determining the strength and direction of their correlation.\",\n",
      "        \"context\": \"Pearson's Correlation Coefficient \\u2013 How to work it out it quickly Learning by Simulations \\u2013 The spread of the correlation coefficient\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is the primary function of the CorrMatr.c program?\",\n",
      "        \"answer\": \"The primary function of the CorrMatr.c program is to work out a correlation matrix.\",\n",
      "        \"context\": \"CorrMatr.c  simple program for working out a correlation matrix Understanding Correlation \\u2013 More beginner's information by a Hawaii professor\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is the primary function of Pearson's correlation in data analysis?\",\n",
      "        \"answer\": \"To calculate correlation coefficients between two datasets.\",\n",
      "        \"context\": \"Pearson's correlation is a mathematical formula used to calculate correlation coefficients between two datasets. Most computer programs have a command to calculate this such as CORREL(dataset A: dataset B). You would calculate this your self by...\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Pearson%20product-moment%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is the purpose of subtracting the mean from every x and y value in the second step of calculating the Pearson product-moment correlation coefficient?\",\n",
      "        \"answer\": \"To create new values, 'a' and 'b', which are used to calculate the correlation between the variables x and y.\",\n",
      "        \"context\": \"Step 2: Subtract the mean of x from every x value (call them \\\"a\\\"), and subtract the mean of y from every y value (call them \\\"b\\\")\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Pearson%20product-moment%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What calculations are performed for every value in Step 3 of the Pearson product-moment correlation coefficient process?\",\n",
      "        \"answer\": \"In Step 3, the calculations performed for every value are ab, a2, and b2.\",\n",
      "        \"context\": \"Step 3: Calculate: ab, a2 and b2 for every value Step 4: Sum up ab, sum up a2 and sum up b2\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Pearson%20product-moment%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"Who developed the formula for dividing the sum of ab by the square root of [(sum of a2) \\u00d7 (sum of b2)] in the late 19th century?\",\n",
      "        \"answer\": \"Karl Pearson\",\n",
      "        \"context\": \"Step 5: Divide the sum of ab by the square root of [(sum of a2) \\u00d7 (sum of b2)] Developed by Karl Pearson in the 1880's, Mathematics\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Pearson%20product-moment%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What does Spearman's rank correlation coefficient measure in mathematics and statistics?\",\n",
      "        \"answer\": \"It measures how closely two sets of data are linked, showing the correlation between them.\",\n",
      "        \"context\": \"In mathematics and statistics, Spearman's rank correlation coefficient is a measure of correlation, named after its maker, Charles Spearman. It is written in short as the Greek letter rho () or sometimes as . It is a number that shows how closely two sets of data are linked. It only can be used for data which can be put in order, such as highest to lowest.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What type of relationship can be analyzed between the price and speed of different computers using correlation?\",\n",
      "        \"answer\": \"The relationship between the price and speed of different computers can be analyzed to see if they are linked, and how closely they are linked.\",\n",
      "        \"context\": \"For example, if you have data for how expensive different computers are, and data for how fast the computers are, you could see if they are linked, and how closely they are linked, using .\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is the initial step in calculating the correlation between two sets of data?\",\n",
      "        \"answer\": \"Ranking each piece of data.\",\n",
      "        \"context\": \"To work out  you first have to rank each piece of data. We are going to use the example from the intro of computers and their speed.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is the ranking process for two sets of data in correlation analysis?\",\n",
      "        \"answer\": \"Ranking involves assigning a rank of 1 to the lowest value, 2 to the next higher value, and so on, until all data points are ranked, and this process is applied to both sets of data.\",\n",
      "        \"context\": \"So, the computer with the lowest price would be rank 1. The one higher than that would have 2. Then, it goes up until it is all ranked. You have to do this to both sets of data.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What mathematical operation is performed on the difference between two ranks in Spearman's rank correlation coefficient?\",\n",
      "        \"answer\": \"Squaring, which involves multiplying the difference by itself.\",\n",
      "        \"context\": \"Next, we have to find the difference between the two ranks. Then, you multiply the difference by itself, which is called squaring. The difference is called , and the number you get when you square  is called .\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is the term used to describe the total number of data points in a dataset, such as the one with ranks 1 to 5?\",\n",
      "        \"answer\": \"The number of data points is called 'n'.\",\n",
      "        \"context\": \"Step three Count how much data we have. This data has ranks 1 to 5, so we have 5 pieces of data. This number is called . Step four\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What does the symbol represent in the context of calculating a total?\",\n",
      "        \"answer\": \"The symbol represents the total of all the numbers in a column.\",\n",
      "        \"context\": \"means that we take the total of all the numbers that were in the column . This is because  means total.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is the result of multiplying 4 by 6 in the context of the given formula?\",\n",
      "        \"answer\": \"24\",\n",
      "        \"context\": \"So,  is  which is 4. The formula says multiply it by 6, which is 24. is  which is 120. So, to find out , we simply do .\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What does a Spearman's rank correlation coefficient of 0.8 signify in a set of data?\",\n",
      "        \"answer\": \"A coefficient of 0.8 indicates a strong positive correlation between the two variables being measured, meaning that as one variable increases, the other variable also tends to increase.\",\n",
      "        \"context\": \"Therefore, Spearman's rank correlation coefficient is 0.8 for this set of data. What the numbers mean\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What does a correlation value of -1 indicate, and how does the graph of data with this value appear?\",\n",
      "        \"answer\": \"A correlation value of -1 indicates a very strong negative link. The graph of data with this value would have a line and points going from top left to bottom right.\",\n",
      "        \"context\": \"always gives an answer between \\u22121 and 1. The numbers between are like a scale, where \\u22121 is a very strong link, 0 is no link, and 1 is also a very strong link. The difference between 1 and \\u22121 is that 1 is a positive correlation, and \\u22121 is a negative correlation.  A graph of data with a  value of \\u22121 would look like the graph shown except the line and points would be going from top left to bottom right.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What does a correlation coefficient of 0.8 indicate about the relationship between two sets of data?\",\n",
      "        \"answer\": \"A correlation coefficient of 0.8 indicates a strong positive correlation, meaning the two sets of data are linked and tend to increase together.\",\n",
      "        \"context\": \"For example, for the data that we did above,  was 0.8. So this means that there is a positive correlation. Because it is close to 1, it means that the link is strong between the two sets of data. So, we can say that those two sets of data are linked, and go up together. If it was \\u22120.8, we could say it was linked and as one goes up, the other goes down.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What happens to the ranks of tied numbers when ranking data, and how are they calculated?\",\n",
      "        \"answer\": \"The tied numbers are ranked as if they were not tied, then the ranks are added up and divided by the number of tied ranks, resulting in a mean or average rank.\",\n",
      "        \"context\": \"Sometimes, when ranking data, there are two or more numbers that are the same. When this happens in , we take the mean or average of the ranks that are the same. These are called tied ranks. To do this, we rank the tied numbers as if they were not tied. Then, we add up all the ranks that they would have, and divide it by how many there are.  For example, say we were ranking how well different people did in a spelling test.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"How are the numbers used in Spearman's rank correlation coefficient similar to other ranking systems?\",\n",
      "        \"answer\": \"They are used in exactly the same way as normal ranks.\",\n",
      "        \"context\": \"These numbers are used in exactly the same way as normal ranks. Related pages Correlation Notes and references Statistics Mathematics\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Spearman%27s%20rank%20correlation%20coefficient\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What does a rank correlation coefficient measure between two rankings?\",\n",
      "        \"answer\": \"The degree of similarity\",\n",
      "        \"context\": \"A rank correlation is any statistic that measures the relationship between rankings. A \\\"ranking\\\" is the assignment of \\\"first\\\", \\\"second\\\", \\\"third\\\", etc. to different observations of a variable. A rank correlation coefficient measures the degree of similarity between two rankings.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Rank%20correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is an example of a potential correlation that can be tested between two college sports programs?\",\n",
      "        \"answer\": \"The correlation between the ranking of a college's basketball program and the ranking of its football program.\",\n",
      "        \"context\": \"One might test for do colleges with a higher-ranked basketball program tend to have a higher-ranked football program? A quite important question is do people with higher-ranked education tend to get higher levels of income?\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Rank%20correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What are some commonly used statistics for measuring the relationship between ranked data?\",\n",
      "        \"answer\": \"Spearman's \\u03c1, Kendall's \\u03c4, Goodman and Kruskal's \\u03b3\",\n",
      "        \"context\": \"Some of the most used rank correlation statistics are Spearman's \\u03c1 Kendall's \\u03c4 Goodman and Kruskal's \\u03b3\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Rank%20correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What range of values can the rank correlation coefficient take, and what does it signify when it reaches its maximum value?\",\n",
      "        \"answer\": \"The rank correlation coefficient is inside the interval [\\u22121, 1] and assumes the value 1 when there is perfect agreement between rankings.\",\n",
      "        \"context\": \"An increasing rank correlation coefficient implies increasing agreement between rankings. The coefficient is inside the interval [\\u22121,\\u00a01] and assumes the value:\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Rank%20correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What does a correlation coefficient of 1 signify in the context of rank correlation?\",\n",
      "        \"answer\": \"A correlation coefficient of 1 indicates that the agreement between the two rankings is perfect, meaning the two rankings are the same.\",\n",
      "        \"context\": \"1 if the agreement between the two rankings is perfect; the two rankings are the same. 0 if the rankings are completely independent.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Rank%20correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What does a correlation coefficient of -1 signify in the context of rank correlation?\",\n",
      "        \"answer\": \"A correlation coefficient of -1 indicates that the disagreement between the two rankings is perfect, meaning one ranking is the reverse of the other.\",\n",
      "        \"context\": \"\\u22121 if the disagreement between the two rankings is perfect; one ranking is the reverse of the other. Kerby simple difference formula\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Rank%20correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"Why did Dave Kerby recommend the rank-biserial as a measure to introduce students to rank correlation?\",\n",
      "        \"answer\": \"Because the general logic of rank-biserial can be explained at an introductory level.\",\n",
      "        \"context\": \"Dave Kerby recommended the rank-biserial as the measure to introduce students to rank correlation, because the general logic can be explained at an introductory level.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Rank%20correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What two key figures does Kerby use to express rank correlation, and how are they related to the data?\",\n",
      "        \"answer\": \"Kerby uses the percent of data that support a stated hypothesis and the percent of data that do not support it. The rank correlation is expressed as the difference between the proportion of favorable evidence and the proportion of unfavorable evidence.\",\n",
      "        \"context\": \"Kerby showed that this rank correlation can be expressed in terms of two figures: the percent of data that support a stated hypothesis, and the percent of data that do not support it. The Kerby simple difference formula states that the rank correlation can be expressed as the difference between the proportion of favorable evidence (f) minus the proportion of unfavorable evidence (u).\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Rank%20correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What is the primary goal of the coach's experiment in comparing the performance of Group A and Group B long-distance runners?\",\n",
      "        \"answer\": \"To determine if method A produces faster runners than method B.\",\n",
      "        \"context\": \"Suppose a coach trains long-distance runners for one month using two methods.  Group A has 5 runners, and Group B has 4 runners.  The hypothesis is that method A produces faster runners.  The race to assess the results finds that the runners from Group A do indeed run faster, with the following ranks: 1, 2, 3, 4, and 6.  The slower runners from Group B thus have ranks of 5, 7, 8, and 9.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Rank%20correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"In the context of rank correlation, what happens when a pair does not support the hypothesis, and how does it affect the overall correlation?\",\n",
      "        \"answer\": \"When a pair does not support the hypothesis, it means that the expected relationship between the two groups is not observed in that pair. This affects the overall correlation by reducing the percentage of data that supports the hypothesis, which in turn reduces the rank correlation coefficient (r).\",\n",
      "        \"context\": \"The analysis is done on pairs, a member of one group compared to a member of the other group. For example, the fastest runner in the study is a member of four pairs: (1,5), (1,7), (1,8), and (1,9).  All four of these pairs support the hypothesis, because in each pair the runner from Group A is faster than the runner from Group B.  There are a total of 20 pairs, and 19 pairs support the hypothesis.  The only pair which does not support the hypothesis are the two runners with ranks 5 and 6, because in this pair, the runner from Group B had the faster time.  By the Kerby simple difference formula, 95% of the data support the hypothesis (19 of 20 pairs), and 5% do not support (1 of 20 pairs), so the rank correlation is r = .95 - .05 = .90.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Rank%20correlation\"\n",
      "    },\n",
      "    {\n",
      "        \"question\": \"What does a correlation value of r = 1 signify in terms of the relationship between pairs and the hypothesis?\",\n",
      "        \"answer\": \"A correlation value of r = 1 means that 100% of the pairs favor the hypothesis.\",\n",
      "        \"context\": \"The maximum value for the correlation is r = 1, which means that 100% of the pairs favor the hypothesis.  A correlation of r = 0 indicates that half the pairs favor the hypothesis and half do not.  An effect size of r = 0 describes no relationship between group membership and the members' ranks.\",\n",
      "        \"source\": \"https://simple.wikipedia.org/wiki/Rank%20correlation\"\n",
      "    }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(flashcard_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### EVALUATION: RETRIEVER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Retriever evaluation complete\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>URL</th>\n",
       "      <th>Context</th>\n",
       "      <th>Similarity Score</th>\n",
       "      <th>Distance Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>64761</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Correlation</td>\n",
       "      <td>In statistics and probability theory, correlat...</td>\n",
       "      <td>0.867254</td>\n",
       "      <td>0.132746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804693</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Pearson%20pr...</td>\n",
       "      <td>Pearson's correlation is a mathematical formul...</td>\n",
       "      <td>0.812699</td>\n",
       "      <td>0.187301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>187866</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Spearman%27s...</td>\n",
       "      <td>In mathematics and statistics, Spearman's rank...</td>\n",
       "      <td>0.801714</td>\n",
       "      <td>0.198286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>531764</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Rank%20corre...</td>\n",
       "      <td>A rank correlation is any statistic that measu...</td>\n",
       "      <td>0.800087</td>\n",
       "      <td>0.199913</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>342118</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Denying%20th...</td>\n",
       "      <td>Denying the correlative is a mistake in logic ...</td>\n",
       "      <td>0.686136</td>\n",
       "      <td>0.313864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>470230</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Statistical%...</td>\n",
       "      <td>Statistics uses variables to describe a measur...</td>\n",
       "      <td>0.660952</td>\n",
       "      <td>0.339048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>76245</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Conjunction%...</td>\n",
       "      <td>Conjunction is a word used in astronomy and as...</td>\n",
       "      <td>0.654611</td>\n",
       "      <td>0.345389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>109208</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Pattern%20ma...</td>\n",
       "      <td>Pattern matching is a family of tools for find...</td>\n",
       "      <td>0.650239</td>\n",
       "      <td>0.349761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>821533</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Serial%20pos...</td>\n",
       "      <td>The serial position effect is a psychological ...</td>\n",
       "      <td>0.644498</td>\n",
       "      <td>0.355502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>472068</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Regression%2...</td>\n",
       "      <td>Regression toward the mean simply means that, ...</td>\n",
       "      <td>0.643775</td>\n",
       "      <td>0.356225</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       ID                                                URL  \\\n",
       "0   64761      https://simple.wikipedia.org/wiki/Correlation   \n",
       "1  804693  https://simple.wikipedia.org/wiki/Pearson%20pr...   \n",
       "2  187866  https://simple.wikipedia.org/wiki/Spearman%27s...   \n",
       "3  531764  https://simple.wikipedia.org/wiki/Rank%20corre...   \n",
       "4  342118  https://simple.wikipedia.org/wiki/Denying%20th...   \n",
       "5  470230  https://simple.wikipedia.org/wiki/Statistical%...   \n",
       "6   76245  https://simple.wikipedia.org/wiki/Conjunction%...   \n",
       "7  109208  https://simple.wikipedia.org/wiki/Pattern%20ma...   \n",
       "8  821533  https://simple.wikipedia.org/wiki/Serial%20pos...   \n",
       "9  472068  https://simple.wikipedia.org/wiki/Regression%2...   \n",
       "\n",
       "                                             Context  Similarity Score  \\\n",
       "0  In statistics and probability theory, correlat...          0.867254   \n",
       "1  Pearson's correlation is a mathematical formul...          0.812699   \n",
       "2  In mathematics and statistics, Spearman's rank...          0.801714   \n",
       "3  A rank correlation is any statistic that measu...          0.800087   \n",
       "4  Denying the correlative is a mistake in logic ...          0.686136   \n",
       "5  Statistics uses variables to describe a measur...          0.660952   \n",
       "6  Conjunction is a word used in astronomy and as...          0.654611   \n",
       "7  Pattern matching is a family of tools for find...          0.650239   \n",
       "8  The serial position effect is a psychological ...          0.644498   \n",
       "9  Regression toward the mean simply means that, ...          0.643775   \n",
       "\n",
       "   Distance Score  \n",
       "0        0.132746  \n",
       "1        0.187301  \n",
       "2        0.198286  \n",
       "3        0.199913  \n",
       "4        0.313864  \n",
       "5        0.339048  \n",
       "6        0.345389  \n",
       "7        0.349761  \n",
       "8        0.355502  \n",
       "9        0.356225  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe_retriever_evaluation = tools.pipeline.evaluate_retriever(topic, vectorstore)\n",
    "dataframe_retriever_evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Preparing dataset from JSON string\n",
      "Casting the dataset: 100%|██████████| 39/39 [00:00<00:00, 5260.58 examples/s]\n",
      "INFO:tools.pipeline:Dataset preparation complete\n",
      "INFO:tools.pipeline:Saving QA pairs to CSV: data_flashcard_generator/generated_qas_Correlation_flashcard_generator.csv\n",
      "INFO:tools.pipeline:File saved as CSV.\n"
     ]
    }
   ],
   "source": [
    "dataset = tools.pipeline.prepare_dataset(flashcard_set)\n",
    "tools.pipeline.save_qa_pairs(dataset, folder_name=\"data_flashcard_generator\", topic=topic, pipeline_name=\"flashcard_generator\",content=\"generated_qas\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### EVALUATION: GENERATOR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "⚡MANUAL STEP is required to add handwritten ground truths, or the evaluation is performed without them. However, not all evaluation metrics can be calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Loading QA pairs from CSV: data_flashcard_generator/generated_qas_Correlation_flashcard_generator.csv\n",
      "INFO:tools.pipeline:QA pairs loaded successfully\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "      <th>contexts</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What does correlation measure in statistics an...</td>\n",
       "      <td>How closely related two sets of data are.</td>\n",
       "      <td>In statistics and probability theory, correlat...</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Correlation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is an important consideration when interp...</td>\n",
       "      <td>A third factor may be involved, and correlatio...</td>\n",
       "      <td>Correlation does not always mean that one caus...</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Correlation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What happens to two sets of data when they hav...</td>\n",
       "      <td>They go up together.</td>\n",
       "      <td>Correlation usually has one of two directions....</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Correlation</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  What does correlation measure in statistics an...   \n",
       "1  What is an important consideration when interp...   \n",
       "2  What happens to two sets of data when they hav...   \n",
       "\n",
       "                                              answer  \\\n",
       "0          How closely related two sets of data are.   \n",
       "1  A third factor may be involved, and correlatio...   \n",
       "2                               They go up together.   \n",
       "\n",
       "                                            contexts  \\\n",
       "0  In statistics and probability theory, correlat...   \n",
       "1  Correlation does not always mean that one caus...   \n",
       "2  Correlation usually has one of two directions....   \n",
       "\n",
       "                                          source  \n",
       "0  https://simple.wikipedia.org/wiki/Correlation  \n",
       "1  https://simple.wikipedia.org/wiki/Correlation  \n",
       "2  https://simple.wikipedia.org/wiki/Correlation  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset_gt = tools.pipeline.load_qa_pairs(folder_name=\"data_flashcard_generator\", topic=topic, pipeline_name=\"flashcard_generator\", content=\"generated_qas\")\n",
    "dataset_gt = dataset_gt[:3]\n",
    "dataset_gt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RAGAS Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Map: 100%|██████████| 3/3 [00:00<00:00, 505.81 examples/s]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Casting the dataset: 100%|██████████| 3/3 [00:00<00:00, 641.76 examples/s]\n",
      "INFO:tools.pipeline:Data prepared for Ragas evaluation\n"
     ]
    }
   ],
   "source": [
    "ragas_data = tools.pipeline.ragas_prepare_data(dataset = dataset_gt, ground_truth=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['question', 'answer', 'contexts', 'source'],\n",
       "    num_rows: 3\n",
       "})"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ragas_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluating:   0%|          | 0/6 [00:00<?, ?it/s]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "Evaluating:  17%|█▋        | 1/6 [00:03<00:18,  3.71s/it]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "Evaluating:  33%|███▎      | 2/6 [00:04<00:09,  2.26s/it]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "Evaluating:  67%|██████▋   | 4/6 [00:08<00:03,  1.80s/it]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "Evaluating:  83%|████████▎ | 5/6 [00:08<00:01,  1.28s/it]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "Evaluating: 100%|██████████| 6/6 [00:13<00:00,  2.19s/it]\n",
      "INFO:tools.pipeline:Ragas evaluation complete\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'faithfulness': 1.0000, 'answer_relevancy': 0.9353}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ragas_evaluation_data = tools.pipeline.ragas_evaluate_data(ragas_data, ground_truth=False)\n",
    "ragas_evaluation_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Saving QA pairs to CSV: data_flashcard_generator/ragas_metrics_Correlation_flashcard_generator.csv\n",
      "INFO:tools.pipeline:File saved as CSV.\n"
     ]
    }
   ],
   "source": [
    "tools.pipeline.save_qa_pairs(ragas_evaluation_data, folder_name = \"data_flashcard_generator\", topic = topic, pipeline_name = \"flashcard_generator\", content=\"ragas_metrics\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DeepEval Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Preparing data for DeepEval evaluation\n",
      "INFO:tools.pipeline:Data prepared for DeepEval evaluation\n"
     ]
    }
   ],
   "source": [
    "deepeval_data = tools.pipeline.deepeval_prepare_data(dataset_gt, ground_truth=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Starting DeepEval evaluation\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">✨ You're running DeepEval's latest <span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">Faithfulness Metric</span>! <span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">(</span><span style=\"color: #374151; text-decoration-color: #374151\">using gpt-</span><span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">4</span><span style=\"color: #374151; text-decoration-color: #374151\">, </span><span style=\"color: #374151; text-decoration-color: #374151\">strict</span><span style=\"color: #374151; text-decoration-color: #374151\">=</span><span style=\"color: #374151; text-decoration-color: #374151; font-style: italic\">False</span><span style=\"color: #374151; text-decoration-color: #374151\">, </span><span style=\"color: #374151; text-decoration-color: #374151\">async_mode</span><span style=\"color: #374151; text-decoration-color: #374151\">=</span><span style=\"color: #374151; text-decoration-color: #374151; font-style: italic\">True</span><span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">)</span><span style=\"color: #374151; text-decoration-color: #374151\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "✨ You're running DeepEval's latest \u001b[38;2;106;0;255mFaithfulness Metric\u001b[0m! \u001b[1;38;2;55;65;81m(\u001b[0m\u001b[38;2;55;65;81musing gpt-\u001b[0m\u001b[1;38;2;55;65;81m4\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81mstrict\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mFalse\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81masync_mode\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mTrue\u001b[0m\u001b[1;38;2;55;65;81m)\u001b[0m\u001b[38;2;55;65;81m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">✨ You're running DeepEval's latest <span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">Answer Relevancy Metric</span>! <span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">(</span><span style=\"color: #374151; text-decoration-color: #374151\">using gpt-</span><span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">4</span><span style=\"color: #374151; text-decoration-color: #374151\">, </span><span style=\"color: #374151; text-decoration-color: #374151\">strict</span><span style=\"color: #374151; text-decoration-color: #374151\">=</span><span style=\"color: #374151; text-decoration-color: #374151; font-style: italic\">False</span><span style=\"color: #374151; text-decoration-color: #374151\">, </span><span style=\"color: #374151; text-decoration-color: #374151\">async_mode</span><span style=\"color: #374151; text-decoration-color: #374151\">=</span><span style=\"color: #374151; text-decoration-color: #374151; font-style: italic\">True</span><span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">)</span><span style=\"color: #374151; text-decoration-color: #374151\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "✨ You're running DeepEval's latest \u001b[38;2;106;0;255mAnswer Relevancy Metric\u001b[0m! \u001b[1;38;2;55;65;81m(\u001b[0m\u001b[38;2;55;65;81musing gpt-\u001b[0m\u001b[1;38;2;55;65;81m4\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81mstrict\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mFalse\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81masync_mode\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mTrue\u001b[0m\u001b[1;38;2;55;65;81m)\u001b[0m\u001b[38;2;55;65;81m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">✨ You're running DeepEval's latest <span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">Contextual Relevancy Metric</span>! <span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">(</span><span style=\"color: #374151; text-decoration-color: #374151\">using gpt-</span><span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">4</span><span style=\"color: #374151; text-decoration-color: #374151\">, </span><span style=\"color: #374151; text-decoration-color: #374151\">strict</span><span style=\"color: #374151; text-decoration-color: #374151\">=</span><span style=\"color: #374151; text-decoration-color: #374151; font-style: italic\">False</span><span style=\"color: #374151; text-decoration-color: #374151\">, </span><span style=\"color: #374151; text-decoration-color: #374151\">async_mode</span><span style=\"color: #374151; text-decoration-color: #374151\">=</span><span style=\"color: #374151; text-decoration-color: #374151; font-style: italic\">True</span><span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">)</span><span style=\"color: #374151; text-decoration-color: #374151\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "✨ You're running DeepEval's latest \u001b[38;2;106;0;255mContextual Relevancy Metric\u001b[0m! \u001b[1;38;2;55;65;81m(\u001b[0m\u001b[38;2;55;65;81musing gpt-\u001b[0m\u001b[1;38;2;55;65;81m4\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81mstrict\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mFalse\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81masync_mode\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mTrue\u001b[0m\u001b[1;38;2;55;65;81m)\u001b[0m\u001b[38;2;55;65;81m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">✨ You're running DeepEval's latest <span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">Bias Metric</span>! <span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">(</span><span style=\"color: #374151; text-decoration-color: #374151\">using gpt-</span><span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">4</span><span style=\"color: #374151; text-decoration-color: #374151\">, </span><span style=\"color: #374151; text-decoration-color: #374151\">strict</span><span style=\"color: #374151; text-decoration-color: #374151\">=</span><span style=\"color: #374151; text-decoration-color: #374151; font-style: italic\">False</span><span style=\"color: #374151; text-decoration-color: #374151\">, </span><span style=\"color: #374151; text-decoration-color: #374151\">async_mode</span><span style=\"color: #374151; text-decoration-color: #374151\">=</span><span style=\"color: #374151; text-decoration-color: #374151; font-style: italic\">True</span><span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">)</span><span style=\"color: #374151; text-decoration-color: #374151\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "✨ You're running DeepEval's latest \u001b[38;2;106;0;255mBias Metric\u001b[0m! \u001b[1;38;2;55;65;81m(\u001b[0m\u001b[38;2;55;65;81musing gpt-\u001b[0m\u001b[1;38;2;55;65;81m4\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81mstrict\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mFalse\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81masync_mode\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mTrue\u001b[0m\u001b[1;38;2;55;65;81m)\u001b[0m\u001b[38;2;55;65;81m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">✨ You're running DeepEval's latest <span style=\"color: #6a00ff; text-decoration-color: #6a00ff\">Toxicity Metric</span>! <span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">(</span><span style=\"color: #374151; text-decoration-color: #374151\">using gpt-</span><span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">4</span><span style=\"color: #374151; text-decoration-color: #374151\">, </span><span style=\"color: #374151; text-decoration-color: #374151\">strict</span><span style=\"color: #374151; text-decoration-color: #374151\">=</span><span style=\"color: #374151; text-decoration-color: #374151; font-style: italic\">False</span><span style=\"color: #374151; text-decoration-color: #374151\">, </span><span style=\"color: #374151; text-decoration-color: #374151\">async_mode</span><span style=\"color: #374151; text-decoration-color: #374151\">=</span><span style=\"color: #374151; text-decoration-color: #374151; font-style: italic\">True</span><span style=\"color: #374151; text-decoration-color: #374151; font-weight: bold\">)</span><span style=\"color: #374151; text-decoration-color: #374151\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "✨ You're running DeepEval's latest \u001b[38;2;106;0;255mToxicity Metric\u001b[0m! \u001b[1;38;2;55;65;81m(\u001b[0m\u001b[38;2;55;65;81musing gpt-\u001b[0m\u001b[1;38;2;55;65;81m4\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81mstrict\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mFalse\u001b[0m\u001b[38;2;55;65;81m, \u001b[0m\u001b[38;2;55;65;81masync_mode\u001b[0m\u001b[38;2;55;65;81m=\u001b[0m\u001b[3;38;2;55;65;81mTrue\u001b[0m\u001b[1;38;2;55;65;81m)\u001b[0m\u001b[38;2;55;65;81m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluating 3 test case(s) in parallel: |          |  0% (0/3) [Time Taken: 00:00, ?test case/s]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "Evaluating 3 test case(s) in parallel: |███▎      | 33% (1/3) [Time Taken: 00:32, 32.23s/test case]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "Evaluating 3 test case(s) in parallel: |██████▋   | 67% (2/3) [Time Taken: 00:32, 13.68s/test case]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "Evaluating 3 test case(s) in parallel: |██████████|100% (3/3) [Time Taken: 00:34, 11.37s/test case]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "======================================================================\n",
      "\n",
      "Metrics Summary\n",
      "\n",
      "  - ✅ Faithfulness (score: 1.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 1.00 because there are no contradictions between the actual output and the retrieval context. Great job!, error: None)\n",
      "  - ✅ Answer Relevancy (score: 1.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 1.00 because the output precisely addresses the input query about positive correlation between two sets of data, without any irrelevant statements., error: None)\n",
      "  - ✅ Contextual Relevancy (score: 1.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 1.00 because the input was perfectly matched with no irrelevancy found., error: None)\n",
      "  - ✅ Bias (score: 0.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 0.00 because there were no instances of bias identified in the actual output., error: None)\n",
      "  - ✅ Toxicity (score: 0.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 0.00 because the actual output showed no signs of toxicity., error: None)\n",
      "\n",
      "For test case:\n",
      "\n",
      "  - input: What happens to two sets of data when they have a positive correlation?\n",
      "  - actual output: They go up together.\n",
      "  - expected output: None\n",
      "  - context: None\n",
      "  - retrieval context: ['Correlation usually has one of two directions. These are positive or negative. If it is positive, then the two sets go up together. If it is negative, then one goes up while the other goes down.']\n",
      "\n",
      "======================================================================\n",
      "\n",
      "Metrics Summary\n",
      "\n",
      "  - ✅ Faithfulness (score: 1.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 1.00 because there are no contradictions present in the actual output. It aligns perfectly with the information provided in the retrieval context., error: None)\n",
      "  - ✅ Answer Relevancy (score: 1.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 1.00 because the answer directly addresses the question asked, providing relevant information pertaining to important considerations when interpreting the relationship between two correlated variables. There are no irrelevant statements made in the actual output., error: None)\n",
      "  - ✅ Contextual Relevancy (score: 1.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 1.00 because the retrieval context perfectly matched the input, with no reasons for irrelevancy provided., error: None)\n",
      "  - ✅ Bias (score: 0.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 0.00 because the actual output shows no signs of bias., error: None)\n",
      "  - ✅ Toxicity (score: 0.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 0.00 because the output doesn't contain any toxic elements., error: None)\n",
      "\n",
      "For test case:\n",
      "\n",
      "  - input: What is an important consideration when interpreting the relationship between two correlated variables?\n",
      "  - actual output: A third factor may be involved, and correlation does not necessarily imply causation.\n",
      "  - expected output: None\n",
      "  - context: None\n",
      "  - retrieval context: ['Correlation does not always mean that one causes the other. In fact, it is very possible that there is a third factor involved.']\n",
      "\n",
      "======================================================================\n",
      "\n",
      "Metrics Summary\n",
      "\n",
      "  - ✅ Faithfulness (score: 1.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 1.00 because there are no contradictions present between the actual output and the retrieval context. Great job on maintaining consistency!, error: None)\n",
      "  - ✅ Answer Relevancy (score: 1.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 1.00 because the response accurately and completely addressed the question, providing relevant information about correlation in statistics and probability theory., error: None)\n",
      "  - ✅ Contextual Relevancy (score: 1.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 1.00 because the retrieval context perfectly aligns with the input, with no mentioned irrelevancies., error: None)\n",
      "  - ✅ Bias (score: 0.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 0.00 because the actual output is completely unbiased., error: None)\n",
      "  - ✅ Toxicity (score: 0.0, threshold: 0.5, strict: False, evaluation model: gpt-4, reason: The score is 0.00 because the actual output didn't contain any form of toxicity., error: None)\n",
      "\n",
      "For test case:\n",
      "\n",
      "  - input: What does correlation measure in statistics and probability theory?\n",
      "  - actual output: How closely related two sets of data are.\n",
      "  - expected output: None\n",
      "  - context: None\n",
      "  - retrieval context: ['In statistics and probability theory, correlation is a way to indicate how closely related two sets of data are.']\n",
      "\n",
      "======================================================================\n",
      "\n",
      "Overall Metric Pass Rates\n",
      "\n",
      "Faithfulness: 100.00% pass rate\n",
      "Answer Relevancy: 100.00% pass rate\n",
      "Contextual Relevancy: 100.00% pass rate\n",
      "Bias: 100.00% pass rate\n",
      "Toxicity: 100.00% pass rate\n",
      "\n",
      "======================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #05f58d; text-decoration-color: #05f58d\">✓</span> Tests finished 🎉! Run <span style=\"color: #008000; text-decoration-color: #008000\">'deepeval login'</span> to view evaluation results on Confident AI. \n",
       "‼️  NOTE: You can also run evaluations on ALL of deepeval's metrics directly on Confident AI instead.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[38;2;5;245;141m✓\u001b[0m Tests finished 🎉! Run \u001b[32m'deepeval login'\u001b[0m to view evaluation results on Confident AI. \n",
       "‼️  NOTE: You can also run evaluations on ALL of deepeval's metrics directly on Confident AI instead.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:DeepEval evaluation complete\n"
     ]
    }
   ],
   "source": [
    "tools.pipeline.deepeval_evaluate_data(deepeval_data, ground_truth=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Haystack Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Preparing data for Haystack evaluation\n",
      "INFO:tools.pipeline:Data prepared for Haystack evaluation\n"
     ]
    }
   ],
   "source": [
    "haystack_data = tools.pipeline.haystack_prepare_data(dataset_gt, ground_truth=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'questions': ['What does correlation measure in statistics and probability theory?'],\n",
       "  'predicted_answers': ['How closely related two sets of data are.'],\n",
       "  'contexts': ['In statistics and probability theory, correlation is a way to indicate how closely related two sets of data are.']},\n",
       " {'questions': ['What is an important consideration when interpreting the relationship between two correlated variables?'],\n",
       "  'predicted_answers': ['A third factor may be involved, and correlation does not necessarily imply causation.'],\n",
       "  'contexts': ['Correlation does not always mean that one causes the other. In fact, it is very possible that there is a third factor involved.']},\n",
       " {'questions': ['What happens to two sets of data when they have a positive correlation?'],\n",
       "  'predicted_answers': ['They go up together.'],\n",
       "  'contexts': ['Correlation usually has one of two directions. These are positive or negative. If it is positive, then the two sets go up together. If it is negative, then one goes up while the other goes down.']}]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "haystack_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Starting Haystack evaluation\n",
      "INFO:haystack.core.pipeline.pipeline:Running component context_relevance_evaluator\n",
      "  0%|          | 0/1 [00:00<?, ?it/s]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "100%|██████████| 1/1 [00:02<00:00,  2.33s/it]\n",
      "INFO:haystack.core.pipeline.pipeline:Running component faithfulness_evaluator\n",
      "  0%|          | 0/1 [00:00<?, ?it/s]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "100%|██████████| 1/1 [00:00<00:00,  1.27it/s]\n",
      "INFO:haystack.core.pipeline.pipeline:Running component context_relevance_evaluator\n",
      "  0%|          | 0/1 [00:00<?, ?it/s]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "100%|██████████| 1/1 [00:00<00:00,  1.46it/s]\n",
      "INFO:haystack.core.pipeline.pipeline:Running component faithfulness_evaluator\n",
      "  0%|          | 0/1 [00:00<?, ?it/s]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "100%|██████████| 1/1 [00:00<00:00,  1.15it/s]\n",
      "INFO:haystack.core.pipeline.pipeline:Running component context_relevance_evaluator\n",
      "  0%|          | 0/1 [00:00<?, ?it/s]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "100%|██████████| 1/1 [00:00<00:00,  1.60it/s]\n",
      "INFO:haystack.core.pipeline.pipeline:Running component faithfulness_evaluator\n",
      "  0%|          | 0/1 [00:00<?, ?it/s]INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "100%|██████████| 1/1 [00:00<00:00,  1.34it/s]\n",
      "INFO:tools.pipeline:Overall Scores: {'context_relevance_score': 1, 'faithfulness_score': 0.0}\n"
     ]
    }
   ],
   "source": [
    "df_haystack_evaluation_data = tools.pipeline.haystack_evaluate_data(haystack_data, ground_truth=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>predicted_answer</th>\n",
       "      <th>context</th>\n",
       "      <th>context_relevance_score</th>\n",
       "      <th>faithfulness_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What does correlation measure in statistics an...</td>\n",
       "      <td>How closely related two sets of data are.</td>\n",
       "      <td>In statistics and probability theory, correlat...</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is an important consideration when interp...</td>\n",
       "      <td>A third factor may be involved, and correlatio...</td>\n",
       "      <td>Correlation does not always mean that one caus...</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What happens to two sets of data when they hav...</td>\n",
       "      <td>They go up together.</td>\n",
       "      <td>Correlation usually has one of two directions....</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  What does correlation measure in statistics an...   \n",
       "1  What is an important consideration when interp...   \n",
       "2  What happens to two sets of data when they hav...   \n",
       "\n",
       "                                    predicted_answer  \\\n",
       "0          How closely related two sets of data are.   \n",
       "1  A third factor may be involved, and correlatio...   \n",
       "2                               They go up together.   \n",
       "\n",
       "                                             context  context_relevance_score  \\\n",
       "0  In statistics and probability theory, correlat...                        1   \n",
       "1  Correlation does not always mean that one caus...                        1   \n",
       "2  Correlation usually has one of two directions....                        1   \n",
       "\n",
       "   faithfulness_score  \n",
       "0                 1.0  \n",
       "1                 1.0  \n",
       "2                 0.0  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_haystack_evaluation_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tools.pipeline:Saving QA pairs to CSV: data_flashcard_generator/haystack_metrics_Correlation_flashcard_generator.csv\n",
      "INFO:tools.pipeline:File saved as CSV.\n"
     ]
    }
   ],
   "source": [
    "tools.pipeline.save_qa_pairs(df_haystack_evaluation_data, folder_name=\"data_flashcard_generator\", topic=topic, pipeline_name=\"flashcard_generator\", content=\"haystack_metrics\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}