Class 21 - Performance Metrics/class_21_performance.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cfd2c9f4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# We'll start at 2:10ish PM \n",
    "\n",
    "Waiting for others to join the call :)\n",
    "\n",
    "https://www.youtube.com/watch?v=Xsx_dIJOtgI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa3b1a93",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='img/cdy.png' style='width:500px; float: left; margin: 0px 30px 15px 0px'></center>\n",
    "\n",
    "# Performance metrics\n",
    "## Class 21 - Data Science Curriculum \n",
    "\n",
    "<br>\n",
    "\n",
    "#### Women Building Change scholarship program 2023 🇧🇮\n",
    "May 12, 2023\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10398bcd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ⏪ Recap last class\n",
    "\n",
    "- Vectorial representation of words \n",
    "    - One-Hot Encoding\n",
    "    - Bag of Words\n",
    "    - TF-IDF\n",
    "    - Word2Vec\n",
    "- Similarity of texts using\n",
    "    - Euclidean distance\n",
    "    - Cosine distance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86946427",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Let's do a review... what is machine learning?\n",
    "\n",
    "<br>\n",
    "<center>\n",
    "<img src='img/ml_everywhere.webp' style='height:300px;'>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6e143af",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Machine Learning involves two key tasks:\n",
    "<br>\n",
    "<center><img src='img/bot.png' style='height:300px;'>\n",
    "</center>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88a5ec0c",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "💡 **Representation**: Extracting structure from data through a model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08da142e",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "💡 **Generalization**: Making predictions from data through a model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa0a8580",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='img/ml_map2.png' style='height:700px;'>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3fc9c65",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Exmaple\n",
    "\n",
    "If we buy one coffee for `$2.10` we know that if we buy our friend one too, it will cost us `$4.20`. The cost depends linearly on the number of cups we purchase.\n",
    "\n",
    "$$total = 2.10 + 2.10 \\times (\\text{number of friends})$$\n",
    "\n",
    "$$y=\\beta_0 + \\beta_1X_1$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eeb88fb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Regression and classification: Two sides of the same coin\n",
    "<center><img src='img/reg_class.png' style='height:500px;'>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "463c8781",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Elements of a model in the context of Machine Learning\n",
    "\n",
    "<br>\n",
    "<center><img src='img/iris.png' style='height:300px;'>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6c135fbf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>species</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal_length  sepal_width  petal_length  petal_width species\n",
       "0           5.1          3.5           1.4          0.2  setosa\n",
       "1           4.9          3.0           1.4          0.2  setosa\n",
       "2           4.7          3.2           1.3          0.2  setosa\n",
       "3           4.6          3.1           1.5          0.2  setosa\n",
       "4           5.0          3.6           1.4          0.2  setosa"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "df = sns.load_dataset('iris')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbe6d65f",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- <b>Model Inputs:</b>\n",
    "Features, Attributes, Predictors, Inputs, Independent Variables, Dimensions, X, probably more..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "872fd061",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- <b>Model Outputs (what you're trying to predict):</b>\n",
    "Target, Response, Output, Dependent Variable, Labels, Y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4b4644d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "• **Row of Data (Inputs + Outputs):** Observation, Datapoint, Record, Row"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e0233db",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "• **Labels:** The values on the target variables (in Supervised Learning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe781589",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 🗺️📍 Guide map \n",
    "\n",
    "<center>\n",
    "    <img src='img/classification.png' style='height:550px; float: center; margin: 0px 0px 0px 0px'>\n",
    "</center>\n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d02402b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Classification models\n",
    "\n",
    "There are three types of classification models:\n",
    "\n",
    "- Binary\n",
    "- Multiclass\n",
    "- Multilabel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf9604ee",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Classification models\n",
    "\n",
    "**Binary:** Classify two classes\n",
    "\n",
    "<center><img src='img/classf1.png' style='height:600px; float: center; margin: 0px 0px 0px 0px'></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19109bea",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Classification models\n",
    "\n",
    "**Multiclass:** Classify more than two classes\n",
    "\n",
    "<center><img src='img/classf2.png' style='height:600px; float: center; margin: 0px 0px 0px 0px'></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a85c4935",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Classification models\n",
    "\n",
    "**Multilabel:** When a single observation has more than one label\n",
    "\n",
    "<br>\n",
    "<center><img src='img/catdog.png' style='height:600px; float: center; margin: 0px 0px 0px 0px'></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbe17457",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 👮‍♀️ Check-in\n",
    "\n",
    "What other problems are examples of multilabel classification?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef0bdd95",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ❇️ Performance metrics for classification models\n",
    "\n",
    "### Performance metrics determine how good our model is"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1213853e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Some performance metrics for classification models\n",
    "\n",
    "- Accuracy\n",
    "- Precision \n",
    "- Recall\n",
    "- F1-score "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "287f45d0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# When using a classification model for prediction, you can get two results \n",
    "\n",
    "<br>\n",
    "<center><img src='img/right_wrong.png' style='height:300px; float: center; margin: 0px 0px 0px 0px'></center>\n",
    "\n",
    "- Correct \n",
    "- Incorrect"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "687d721c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Accuracy\n",
    "\n",
    "- The number of correct predictions divided by the total number of predictions\n",
    "\n",
    "<br>\n",
    "<center><img src='img/acc.png' style='height:280px; float: center; margin: 0px 0px 0px 0px'></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2fb6417",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "- Accuracy $\\dfrac{7}{10} = 0.7$ or 70% accurate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "297ec29b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Issues with accuracy\n",
    "\n",
    "- It needs a balanced data set \n",
    "\n",
    "<br>\n",
    "<center><img src='img/acc3.png' style='height:280px; float: center; margin: 0px 0px 0px 0px'></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff4e5792",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "- Accuracy $\\dfrac{9}{10} = 0.9$ or 90% accurate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77e9c5c6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Recall\n",
    "\n",
    "- The ability of the model to retrieve **all** relevant cases within a data set \n",
    "\n",
    "<br>\n",
    "<center><img src='img/acc.png' style='height:280px; float: center; margin: 0px 0px 0px 0px'></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb8e9d96",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "- Let `dog` be true positive (TP) and `cat` be true negative (TN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4da553bd",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "- Recall $= \\dfrac{\\text{TP}}{\\text{TP}+\\text{FN}} = \\dfrac{\\text{TP}}{\\text{everything that is actually positive}} = \\dfrac{2}{3} = \\text{67% recall}$ "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74910248",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Precision\n",
    "\n",
    "- The ability of the model to find **only** the relevant cases within a data set\n",
    "\n",
    "<br>\n",
    "<center><img src='img/acc.png' style='height:280px; float: center; margin: 0px 0px 0px 0px'></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "769b4a48",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "- Precision $= \\dfrac{\\text{TP}}{\\text{TP}+\\text{FP}} = \\dfrac{\\text{TP}}{\\text{everything that got classified as positive}} = \\dfrac{2}{4} = \\text{50% precision}$ "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "559c80a0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# F1\n",
    "\n",
    "- It is used to find an optimal balance between precision and recall\n",
    "\n",
    "<br>\n",
    "<center><img src='img/acc.png' style='height:280px; float: center; margin: 0px 0px 0px 0px'></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "000e3faa",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "- $F_1 = 2 \\cdot \\dfrac{\\text{precision}\\cdot\\text{recall}}{\\text{precision}+\\text{recall}} = 2 \\cdot \\dfrac{(2/3)\\cdot(2/4)}{(2/3)+(2/4)} = \\dfrac{4}{7} = 57\\%$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d397b967",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# When using a classification model for prediction, you can get two results: \n",
    "\n",
    "- Correct \n",
    "- Incorrect"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ad8fc8f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "That means that you end up with four groups (in a binary classification):\n",
    "\n",
    "- Correct classification `class 1`: True Positive (TP)\n",
    "<img src='img/TP.png' style='height:120px; float: center; margin: 0px 15px 15px 0px'>\n",
    "- Correct classification `class 2`: True Negative (TN)\n",
    "<img src='img/TN.png' style='height:120px; float: center; margin: 0px 15px 15px 0px'>\n",
    "- Incorrect classification `class 2`: False Positive (FP)\n",
    "<img src='img/FP.png' style='height:120px; float: center; margin: 0px 15px 15px 0px'>\n",
    "- Incorrect classification `class 1`: False Negative (FN)\n",
    "<img src='img/FN.png' style='height:120px; float: center; margin: 0px 15px 15px 0px'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c87909c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Confusion matrix\n",
    "\n",
    "- Tool that helps visualize the performance of a classification model\n",
    "- The name stems from the fact that it makes it easy to see whether the system is confusing two classes\n",
    "\n",
    "<center><img src=\"img/conf_matrix.png\" style='height:600px; float: center; margin: 0px 0px 0px 0px'></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0509112a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Exercise\n",
    "\n",
    "<center><img src=\"img/conf_matrix2.png\" style='height:450px; float: center; margin: 0px 0px 0px 0px'></center>\n",
    "<br>\n",
    "<center><img src='img/acc.png' style='height:200px; float: center; margin: 0px 0px 0px 0px'></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bd97bbe",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src=\"img/pregnant.jpeg\" style='height:500px; float: center; margin: 0px 0px 0px 0px'></center>\n",
    "\n",
    "<br>\n",
    "\n",
    "- The main purpose of the confusion matrix is to obtain measures to compare the predicted values with the true values\n",
    "- What constitutes a \"good\" measure depends on the situation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c288bce",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# More classification performance metrics\n",
    "\n",
    "- Terminology and derivations from a confusion matrix: [here](https://en.wikipedia.org/wiki/Precision_and_recall)\n",
    "- ROC (Receiver Operator Characteristic) graphs and AUC (the area under the curve): [here](https://www.youtube.com/watch?v=4jRBRDbJemM&ab_channel=StatQuestwithJoshStarmer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "043b4e32",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ⏪ Today's recap\n",
    "\n",
    "- Performance metrics for classification models\n",
    "    - TP/TN/FP/FN\n",
    "    - Accuracy (careful of unbalanced datasets!)\n",
    "    - Precision and Recall\n",
    "    - F1: Optimal balance between precision and recall\n",
    "    - Confusion matrix: Tool to obtain measures to compare the predicted values with the true values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d8c332c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='img/bye.gif' style='height:250px;'></center> \n",
    "\n",
    "# Next class: Classification models\n",
    "# See you next Monday!"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}