Class 19 - TF-IDF/class_19_tfidf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "674ac2a7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# We'll start at 2:10ish PM \n",
    "\n",
    "Waiting for others to join the call :)\n",
    "\n",
    "https://www.youtube.com/watch?v=X7_fB2L4hvc&ab_channel=Esteman"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "843e46f7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='img/cdy.png' style='width:500px; float: left; margin: 0px 30px 15px 0px'></center>\n",
    "\n",
    "# TF-IDF\n",
    "## Class 19 - Data Science Curriculum \n",
    "\n",
    "<br>\n",
    "\n",
    "#### Women Building Change scholarship program 2023 🇧🇮\n",
    "May 5, 2023\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48500da2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ⏪ Recap last class\n",
    "\n",
    "- Text pre-processing\n",
    "- Word Cloud"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82476939",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 🚀 Today's agenda\n",
    "\n",
    "- TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaee16fd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='img/pipeline.png' style='width:1500px; margin: 0px 30px 15px 0px'></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d90e61c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Common NLP terminology\n",
    "<center><img src='../Class 17 - Language/img/clase01/terms2.png' style='height:800px;'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "203a4030",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Machines can also learn through experience\n",
    "\n",
    "<br>\n",
    "\n",
    "<div align='center'>\n",
    "    <img src='../Class 17 - Language/img/clase08/maquina1.png' style='height:500px; float: left; margin: 0px 15px 15px 0px'>\n",
    "    <img src='../Class 17 - Language/img/clase08/maquina2.png' style='height:500px; float: left; margin: 0px 15px 15px 0px'>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63a8759b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Representation of data in numerical form\n",
    "\n",
    "**Example: Images**\n",
    "\n",
    "<center><img src='../Class 17 - Language/img/clase08/komp2.jpeg'>\n",
    "<small>Image credit: VideoNet</small></center>\n",
    "\n",
    "- An image is represented on a computer in the form of a matrix where each $m[i,j]$ represents pixel $i$,$j$ of the image\n",
    "\n",
    "- Similarly, a video is a collection of frames, where each frame is an image. Therefore, any video can be represented as a collection of matrices\n",
    "\n",
    "- (Un)fortunately, representing text numerically is not so simple"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "480b0484",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 🤔 The issue at hand...\n",
    "\n",
    "<br>\n",
    "<center><img src='../Class 17 - Language/img/clase01/gigo.png' style='height:350px;'>\n",
    "<small>Image credit: R-Bloggers</small></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7c97442",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Google Colab: https://colab.research.google.com/\n",
    "\n",
    "<br>\n",
    "<center><img src='img/girl_coding.jpg' style='height:350px; float: center; margin: 0px 30px 15px 0px'></center>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3130c953",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Let's do some text pre-processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "27a951e4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>description</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>princess</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Blanche-Neige</th>\n",
       "      <td>Blanche-Neige est une princesse d'une très gra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cendrillon</th>\n",
       "      <td>Cendrillon est une jeune femme soumise à l'aut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Aurore</th>\n",
       "      <td>La Princesse Aurore est une princesse, fille u...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ariel</th>\n",
       "      <td>Dans l'illustre film de 1989, Ariel est la plu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Belle</th>\n",
       "      <td>Belle est une jeune fille habitant dans un pet...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jasmine</th>\n",
       "      <td>L'héroïne contemporaine-type des films Disney....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pocahontas</th>\n",
       "      <td>Le personnage de Pocahontas s'inspire libremen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mulan</th>\n",
       "      <td>Le caractère du personnage de Mulan est différ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tiana</th>\n",
       "      <td>Tiana est la première princesse Disney afro-am...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Raiponce</th>\n",
       "      <td>Raiponce est décrite comme une jeune fille aux...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mérida</th>\n",
       "      <td>Âgée de 16 ans, Mérida est la fille de roi Fer...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     description\n",
       "princess                                                        \n",
       "Blanche-Neige  Blanche-Neige est une princesse d'une très gra...\n",
       "Cendrillon     Cendrillon est une jeune femme soumise à l'aut...\n",
       "Aurore         La Princesse Aurore est une princesse, fille u...\n",
       "Ariel          Dans l'illustre film de 1989, Ariel est la plu...\n",
       "Belle          Belle est une jeune fille habitant dans un pet...\n",
       "Jasmine        L'héroïne contemporaine-type des films Disney....\n",
       "Pocahontas     Le personnage de Pocahontas s'inspire libremen...\n",
       "Mulan          Le caractère du personnage de Mulan est différ...\n",
       "Tiana          Tiana est la première princesse Disney afro-am...\n",
       "Raiponce       Raiponce est décrite comme une jeune fille aux...\n",
       "Mérida         Âgée de 16 ans, Mérida est la fille de roi Fer..."
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "with open('princesses.json', 'r') as f:\n",
    "    data = json.load(f)\n",
    "    \n",
    "df = pd.DataFrame(data.items(), columns=['princess', 'description']).set_index('princess')\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "196c8a12",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/vmarquez/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import pandas as pd \n",
    "\n",
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bff42def",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/vivianamarquez/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "      <th>date</th>\n",
       "      <th>city</th>\n",
       "      <th>user_location</th>\n",
       "      <th>id</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@BridJackline 😂😂😂now I say it with who?</td>\n",
       "      <td>2023-04-24 12:16:28+00:00</td>\n",
       "      <td>Burundi</td>\n",
       "      <td>Arua, Uganda</td>\n",
       "      <td>1650473779720585218</td>\n",
       "      <td>https://twitter.com/i/web/status/1650473779720...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>@IsaiahTwesigye They move on bro😂</td>\n",
       "      <td>2023-04-24 12:16:16+00:00</td>\n",
       "      <td>Burundi</td>\n",
       "      <td>Arua, Uganda</td>\n",
       "      <td>1650473729393057792</td>\n",
       "      <td>https://twitter.com/i/web/status/1650473729393...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Of which she told me she can't. They are not d...</td>\n",
       "      <td>2023-04-24 12:15:57+00:00</td>\n",
       "      <td>Burundi</td>\n",
       "      <td>Gulu, Uganda</td>\n",
       "      <td>1650473651526045697</td>\n",
       "      <td>https://twitter.com/i/web/status/1650473651526...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>@PatrickAkuku And you haven't said any about me</td>\n",
       "      <td>2023-04-24 12:15:57+00:00</td>\n",
       "      <td>Burundi</td>\n",
       "      <td>Arua, Uganda</td>\n",
       "      <td>1650473651416834053</td>\n",
       "      <td>https://twitter.com/i/web/status/1650473651416...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>This week happens to be the official delivery ...</td>\n",
       "      <td>2023-04-24 12:15:53+00:00</td>\n",
       "      <td>Burundi</td>\n",
       "      <td>Gulu, Uganda</td>\n",
       "      <td>1650473636564992001</td>\n",
       "      <td>https://twitter.com/i/web/status/1650473636564...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               tweet  \\\n",
       "0            @BridJackline 😂😂😂now I say it with who?   \n",
       "1                  @IsaiahTwesigye They move on bro😂   \n",
       "2  Of which she told me she can't. They are not d...   \n",
       "3    @PatrickAkuku And you haven't said any about me   \n",
       "4  This week happens to be the official delivery ...   \n",
       "\n",
       "                        date     city user_location                   id  \\\n",
       "0  2023-04-24 12:16:28+00:00  Burundi  Arua, Uganda  1650473779720585218   \n",
       "1  2023-04-24 12:16:16+00:00  Burundi  Arua, Uganda  1650473729393057792   \n",
       "2  2023-04-24 12:15:57+00:00  Burundi  Gulu, Uganda  1650473651526045697   \n",
       "3  2023-04-24 12:15:57+00:00  Burundi  Arua, Uganda  1650473651416834053   \n",
       "4  2023-04-24 12:15:53+00:00  Burundi  Gulu, Uganda  1650473636564992001   \n",
       "\n",
       "                                                link  \n",
       "0  https://twitter.com/i/web/status/1650473779720...  \n",
       "1  https://twitter.com/i/web/status/1650473729393...  \n",
       "2  https://twitter.com/i/web/status/1650473651526...  \n",
       "3  https://twitter.com/i/web/status/1650473651416...  \n",
       "4  https://twitter.com/i/web/status/1650473636564...  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "import pandas as pd \n",
    "\n",
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "df = pd.read_csv(\"../Class 18 - Word Cloud/tweets_april24.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "74f7d099",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "sw = stopwords.words('french')\n",
    "\n",
    "def text_pre_processing(text):\n",
    "    text = text.lower() # Make it lowercase\n",
    "    text = re.sub(r\"[\\W\\d]\", \" \", text) # Remove punctuation\n",
    "    text = text.split() # Tokenize\n",
    "    text = [word for word in text if word not in sw] # Remove stop words\n",
    "    return \" \".join(text)\n",
    "\n",
    "df['text_pp'] = df['description'].apply(lambda row: text_pre_processing(row))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "36b96e8e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>description</th>\n",
       "      <th>text_pp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>princess</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Blanche-Neige</th>\n",
       "      <td>Blanche-Neige est une princesse d'une très gra...</td>\n",
       "      <td>blanche neige princesse très grande beauté ren...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cendrillon</th>\n",
       "      <td>Cendrillon est une jeune femme soumise à l'aut...</td>\n",
       "      <td>cendrillon jeune femme soumise autorité marâtr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Aurore</th>\n",
       "      <td>La Princesse Aurore est une princesse, fille u...</td>\n",
       "      <td>princesse aurore princesse fille unique roi st...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ariel</th>\n",
       "      <td>Dans l'illustre film de 1989, Ariel est la plu...</td>\n",
       "      <td>illustre film ariel plus jeune sept filles roi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Belle</th>\n",
       "      <td>Belle est une jeune fille habitant dans un pet...</td>\n",
       "      <td>belle jeune fille habitant petit village franç...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jasmine</th>\n",
       "      <td>L'héroïne contemporaine-type des films Disney....</td>\n",
       "      <td>héroïne contemporaine type films disney indépe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pocahontas</th>\n",
       "      <td>Le personnage de Pocahontas s'inspire libremen...</td>\n",
       "      <td>personnage pocahontas inspire librement person...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mulan</th>\n",
       "      <td>Le caractère du personnage de Mulan est différ...</td>\n",
       "      <td>caractère personnage mulan différent classique...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tiana</th>\n",
       "      <td>Tiana est la première princesse Disney afro-am...</td>\n",
       "      <td>tiana première princesse disney afro américain...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Raiponce</th>\n",
       "      <td>Raiponce est décrite comme une jeune fille aux...</td>\n",
       "      <td>raiponce décrite comme jeune fille magnifiques...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mérida</th>\n",
       "      <td>Âgée de 16 ans, Mérida est la fille de roi Fer...</td>\n",
       "      <td>âgée ans mérida fille roi fergus reine elinor ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     description  \\\n",
       "princess                                                           \n",
       "Blanche-Neige  Blanche-Neige est une princesse d'une très gra...   \n",
       "Cendrillon     Cendrillon est une jeune femme soumise à l'aut...   \n",
       "Aurore         La Princesse Aurore est une princesse, fille u...   \n",
       "Ariel          Dans l'illustre film de 1989, Ariel est la plu...   \n",
       "Belle          Belle est une jeune fille habitant dans un pet...   \n",
       "Jasmine        L'héroïne contemporaine-type des films Disney....   \n",
       "Pocahontas     Le personnage de Pocahontas s'inspire libremen...   \n",
       "Mulan          Le caractère du personnage de Mulan est différ...   \n",
       "Tiana          Tiana est la première princesse Disney afro-am...   \n",
       "Raiponce       Raiponce est décrite comme une jeune fille aux...   \n",
       "Mérida         Âgée de 16 ans, Mérida est la fille de roi Fer...   \n",
       "\n",
       "                                                         text_pp  \n",
       "princess                                                          \n",
       "Blanche-Neige  blanche neige princesse très grande beauté ren...  \n",
       "Cendrillon     cendrillon jeune femme soumise autorité marâtr...  \n",
       "Aurore         princesse aurore princesse fille unique roi st...  \n",
       "Ariel          illustre film ariel plus jeune sept filles roi...  \n",
       "Belle          belle jeune fille habitant petit village franç...  \n",
       "Jasmine        héroïne contemporaine type films disney indépe...  \n",
       "Pocahontas     personnage pocahontas inspire librement person...  \n",
       "Mulan          caractère personnage mulan différent classique...  \n",
       "Tiana          tiana première princesse disney afro américain...  \n",
       "Raiponce       raiponce décrite comme jeune fille magnifiques...  \n",
       "Mérida         âgée ans mérida fille roi fergus reine elinor ...  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f34111b1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 🤔 Cool, we have cleaned and pre-processed our text... but it's still not numeric"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "100fb4fe",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src='img/this.png' style='width:300px; float: left; margin: 0px 30px 15px 0px'></center> <big>Vectorial representation of texts!</big>\n",
    "\n",
    "\n",
    "(A vector is just a list of numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3439235",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Vectorial representation of texts\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1efd5c48",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- There are several methods"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86441748",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- What differentiates one method from the other is how well it captures the linguistic properties of the text it represents and the amount of space it takes up in memory\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b52f95a8",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Most popular methods:\n",
    "    - One-Hot Encoding\n",
    "    - Bag of Words \n",
    "    - TF-IDF \n",
    "    - Word embeddings (word2vec) \n",
    "        - CBOW (Continuous Bag of Words)\n",
    "        - SkipGram"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "666ea80d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# One-Hot Encoding \n",
    "\n",
    "Mapping each word in the vocabulary of the text corpus to a unique identification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d191cf45",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Advantages\n",
    "- Intuitive and easy to understand\n",
    "- Implementation is straightforward"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1b646d4",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Disadvantages\n",
    "- Generates a sparse matrix\n",
    "- The vector of each phrase does not have a constant size\n",
    "- No notion of similarity between words\n",
    "- Out-of-vocabulary problem"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98ebf4f0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Bag of Words (BoW)\n",
    "\n",
    "- Represent the text as a bag of words (ignoring order and context).\n",
    "- If two pieces of text have almost the same words, then they belong to the same bag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "5e2d5f5c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary:  {'blanche': 73, 'neige': 442, 'princesse': 519, 'très': 687, 'grande': 302, 'beauté': 65, 'rend': 561, 'jalouse': 357, 'belle': 66, 'mère': 434, 'reine': 557, 'cette': 110, 'dernière': 177, 'demande': 173, 'quotidiennement': 547, 'miroir': 421, 'magique': 397, 'plus': 502, 'royaume': 587, 'attendant': 43, 'comme': 136, 'réponse': 596, 'dise': 189, 'jour': 367, 'affirme': 4, 'jolie': 365, 'femme': 259, 'furieuse': 290, 'décide': 208, 'alors': 15, 'tuer': 691, 'jeune': 362, 'fille': 263, 'cependant': 107, 'chasseur': 116, 'charge': 114, 'tâche': 694, 'trouve': 685, 'courage': 159, 'accomplir': 1, 'permet': 485, 'enfuir': 237, 'perdue': 482, 'forêt': 281, 'bout': 83, 'force': 276, 'celle': 103, 'ci': 131, 'échoue': 725, 'maison': 402, 'où': 468, 'habitent': 310, 'sept': 616, 'nains': 439, 'cendrillon': 105, 'soumise': 637, 'autorité': 49, 'marâtre': 413, 'lady': 372, 'trémaine': 688, 'deux': 182, 'demi': 174, 'sœurs': 651, 'anastasie': 24, 'javotte': 361, 'bien': 71, 'maltraitée': 404, 'humiliée': 324, 'reléguée': 559, 'rôle': 602, 'servante': 618, 'maintient': 401, 'espoir': 242, 'rêves': 601, 'fidèle': 262, 'idée': 330, 'souhaits': 635, 'bonheur': 80, 'deviendront': 184, 'vrais': 717, 'orpheline': 465, 'seule': 621, 'marraine': 412, 'fée': 291, 'compagnons': 140, 'animaux': 27, 'soutiennent': 639, 'encouragent': 235, 'quête': 548, 'amour': 19, 'sait': 604, 'aussi': 47, 'montrer': 429, 'forte': 280, 'déterminée': 224, 'quand': 544, 'invitation': 356, 'bal': 61, 'royal': 586, 'arrive': 37, 'fait': 253, 'tout': 672, 'peut': 498, 'persuader': 492, 'autant': 48, 'droit': 203, 'autres': 51, 'être': 745, 'présente': 534, 'va': 701, 'impitoyable': 333, 'cruelle': 165, 'jusqu': 371, 'empêcher': 231, 'sciemment': 611, 'venir': 702, 'vue': 718, 'plusieurs': 503, 'tenues': 659, 'films': 266, 'robe': 576, 'haillons': 311, 'domestique': 196, 'chemise': 119, 'nuit': 453, 'bleue': 76, 'rose': 583, 'confectionnée': 145, 'amis': 18, 'déchirée': 207, 'apparue': 31, 'sous': 638, 'baguette': 59, 'mariage': 410, 'presque': 517, 'toujours': 670, 'représentée': 566, 'produits': 525, 'dérivés': 217, 'publicités': 539, 'bleu': 75, 'clair': 132, 'chaussures': 117, 'verre': 703, 'longs': 386, 'gants': 296, 'aurore': 46, 'unique': 698, 'roi': 577, 'stéphane': 644, 'oriane': 461, 'présentée': 535, 'aimable': 10, 'douce': 202, 'timide': 664, 'assez': 40, 'naïve': 441, 'aime': 11, 'parmi': 470, 'traits': 677, 'personnalités': 490, 'mis': 422, 'avant': 52, 'passion': 475, 'romantisme': 579, 'prédomine': 530, 'lors': 389, 'fête': 293, 'organisée': 460, 'honneur': 319, 'trois': 682, 'bonnes': 81, 'fées': 292, 'penchent': 479, 'berceau': 68, 'font': 275, 'don': 197, 'grâce': 305, 'voix': 713, 'maléfique': 405, 'décrit': 213, 'film': 265, 'certes': 109, 'admets': 2, 'rare': 551, 'cheveux': 122, 'blonds': 78, 'blés': 79, 'lèvres': 393, 'éclatant': 726, 'roses': 584, 'hélas': 325, 'désormais': 222, 'repose': 563, 'prénom': 532, 'vient': 708, 'conte': 153, 'perrault': 487, 'nomme': 447, 'ainsi': 12, 'tchaïkovski': 654, 'transfère': 678, 'nommant': 446, 'ballet': 62, 'walt': 721, 'disney': 190, 'fera': 260, 'après': 33, 'originale': 462, 'complètement': 141, 'absente': 0, 'version': 704, 'animée': 28, 'pseudonyme': 538, 'donné': 198, 'cacher': 93, 'nom': 444, 'allemande': 14, 'histoire': 316, 'revanche': 572, 'première': 514, 'française': 287, 'églantine': 730, 'correspond': 156, 'traduction': 674, 'exacte': 244, 'anglais': 25, 'briar': 86, 'certaines': 108, 'traductions': 675, 'anglophones': 26, 'nommée': 449, 'rosamond': 581, 'rosamund': 582, 'illustre': 332, 'ariel': 36, 'filles': 264, 'triton': 681, 'ans': 29, 'aventureuse': 55, 'curieuse': 166, 'monde': 426, 'humains': 323, 'fascination': 257, 'choque': 128, 'père': 542, 'dernier': 176, 'ondins': 458, 'prendre': 515, 'contact': 152, 'terrestre': 662, 'montre': 428, 'méfiant': 437, 'envers': 241, 'êtres': 746, 'meilleur': 417, 'ami': 17, 'poisson': 506, 'nommé': 448, 'polochon': 508, 'développe': 225, 'relation': 558, 'étroite': 742, 'bras': 84, 'crabe': 163, 'sébastien': 649, 'récupère': 591, 'divers': 193, 'objets': 454, 'haut': 313, 'garde': 297, 'secret': 613, 'grotte': 304, 'étendue': 739, 'toute': 673, 'collection': 135, 'cours': 161, 'sauve': 607, 'vie': 707, 'prince': 518, 'éric': 737, 'tombe': 668, 'amoureuse': 20, 'violente': 711, 'dispute': 191, 'visite': 712, 'sorcière': 630, 'mers': 418, 'ursula': 700, 'commerce': 138, 'contre': 155, 'jambes': 359, 'humaines': 322, 'pendant': 480, 'jours': 369, 'afin': 5, 'obtenir': 455, 'chance': 113, 'gagner': 295, 'cœur': 170, 'réussit': 599, 'vrai': 715, 'baiser': 60, 'arrêtée': 39, 'met': 419, 'œuvre': 747, 'ignoble': 331, 'tactique': 652, 'lorsque': 390, 'arrivent': 38, 'terme': 660, 'retrouve': 571, 'forme': 277, 'sirène': 628, 'doit': 194, 'céder': 168, 'couronne': 160, 'trident': 680, 'sauver': 608, 'bataille': 63, 'suit': 645, 'tue': 690, 'attributs': 45, 'termine': 661, 'retransformée': 570, 'humaine': 321, 'façon': 258, 'permanente': 483, 'épouse': 735, 'habitant': 309, 'petit': 493, 'village': 709, 'français': 286, 'maurice': 415, 'inventeur': 352, 'villageois': 710, 'prennent': 516, 'fou': 282, 'passionnée': 476, 'littérature': 384, 'découverte': 210, 'romance': 578, 'régulièrement': 594, 'chez': 123, 'libraire': 381, 'recherche': 554, 'nouvelles': 452, 'histoires': 317, 'si': 622, 'unanime': 697, 'quant': 545, 'beaucoup': 64, 'trouvent': 686, 'fantasque': 256, 'bizarre': 72, 'courtisée': 162, 'gaston': 298, 'hésite': 329, 'faire': 252, 'savoir': 610, 'veut': 706, 'mari': 409, 'téléfilm': 695, 'descendants': 179, 'sorti': 632, 'censé': 106, 'place': 499, 'événements': 744, 'bête': 92, 'scène': 612, 'enfants': 236, 'héros': 326, 'méchants': 436, 'univers': 699, 'dont': 199, 'ben': 67, 'fils': 267, 'interprété': 349, 'mitchell': 423, 'hope': 320, 'héroïne': 327, 'contemporaine': 154, 'type': 693, 'indépendante': 341, 'intelligente': 346, 'cran': 164, 'inclinations': 338, 'jasmine': 360, 'souffert': 634, 'confinée': 147, 'palais': 469, 'étouffe': 740, 'tutelle': 692, 'sultan': 647, 'quelques': 546, 'seize': 614, 'loi': 385, 'choisir': 127, 'prétendant': 536, 'fuguer': 289, 'réfugie': 593, 'marché': 408, 'là': 392, 'suite': 646, 'altercation': 16, 'marchand': 407, 'aladdin': 13, 'tire': 666, 'mauvais': 416, 'moment': 425, 'commence': 137, 'magnifique': 399, 'but': 91, 'permission': 486, 'réunir': 598, 'pauvre': 478, 'voleur': 714, 'personnage': 488, 'pocahontas': 505, 'inspire': 343, 'librement': 382, 'historique': 318, 'matoaka': 414, 'vraie': 716, 'particulièrement': 471, 'tragique': 676, 'mieux': 420, 'connue': 150, 'surnom': 648, 'signifie': 624, 'langue': 373, 'powhatan': 512, 'petite': 494, 'dévergondée': 226, 'existence': 247, 'origine': 463, 'nombreuses': 445, 'légendes': 394, 'noble': 443, 'car': 97, 'chef': 118, 'tribu': 679, 'simple': 627, 'spirituelle': 640, 'exprime': 249, 'sagesse': 603, 'delà': 172, 'âge': 723, 'prête': 537, 'aider': 8, 'ceux': 111, 'entourent': 238, 'aventure': 53, 'nature': 440, 'semble': 615, 'avoir': 56, 'pouvoirs': 511, 'chamaniques': 112, 'puisqu': 540, 'communique': 139, 'éléments': 733, 'comprend': 142, 'langues': 374, 'inconnues': 339, 'distingue': 192, 'siens': 623, 'ouverture': 466, 'esprit': 243, 'refus': 555, 'peur': 497, 'tenter': 658, 'intermédiaire': 348, 'entre': 240, 'peuple': 496, 'étrangers': 741, 'caractère': 98, 'mulan': 431, 'différent': 187, 'classiques': 133, 'notamment': 450, 'celui': 104, 'princesses': 520, 'brave': 85, 'peu': 495, 'intérêt': 351, 'encontre': 233, 'archétypes': 35, 'jeunes': 363, 'chinoises': 125, 'époque': 734, 'doivent': 195, 'calmes': 95, 'réservées': 597, 'polies': 507, 'obéissantes': 456, 'silencieuses': 625, 'lieu': 383, 'cela': 102, 'maladroite': 403, 'dégourdie': 215, 'franche': 285, 'début': 205, 'premier': 513, 'encore': 234, 'sûre': 650, 'regrette': 556, 'déshonorer': 221, 'famille': 255, 'temps': 655, 'gagne': 294, 'confiance': 146, 'aide': 7, 'chine': 124, 'détermination': 223, 'ingéniosité': 342, 'aideront': 9, 'aventures': 54, 'respect': 567, 'tiana': 663, 'afro': 6, 'américaine': 22, 'deuxième': 183, 'causé': 101, 'troubles': 684, 'studios': 643, 'aperçus': 30, 'maddy': 395, 'choisi': 126, 'proche': 524, 'mammy': 406, 'considéré': 151, 'offensant': 457, 'américains': 23, 'depuis': 175, 'modifié': 424, 'serveuse': 619, 'talent': 653, 'rêve': 600, 'posséder': 509, 'propre': 527, 'restaurant': 568, 'diamant': 186, 'brut': 90, 'tirer': 667, 'leçons': 380, 'atteindre': 42, 'statut': 641, 'véritable': 719, 'réalisateur': 589, 'john': 364, 'musker': 432, 'déclaré': 209, 'héroïnes': 328, 'fort': 279, 'motivée': 430, 'choses': 130, 'sans': 605, 'personne': 491, 'apprendre': 32, 'équilibre': 736, 'important': 334, 'heureux': 314, 'besoin': 69, 'carrière': 99, 'professionnelle': 526, 'interview': 350, 'réalisateurs': 590, 'ron': 580, 'clements': 134, 'indiquent': 340, 'inspirés': 344, 'personnages': 489, 'rencontres': 560, 'faites': 254, 'nouvelle': 451, 'orléans': 464, 'préparation': 533, 'partie': 472, 'leah': 378, 'chase': 115, 'restauratrice': 569, 'débuté': 206, 'fini': 270, 'ouvrir': 467, 'dookie': 200, 'raiponce': 549, 'décrite': 214, 'magnifiques': 400, 'yeux': 722, 'verts': 705, 'ravissante': 552, 'prisonnière': 523, 'tour': 671, 'méchante': 435, 'gothel': 299, 'passe': 474, 'journées': 368, 'jouer': 366, 'pascal': 473, 'caméléon': 96, 'cachette': 94, 'brosser': 88, 'chose': 129, 'découvrir': 212, 'extérieur': 250, 'interdit': 347, 'formellement': 278, 'sortir': 633, 'grand': 301, 'désappointement': 218, 'tombera': 669, 'flynn': 272, 'rider': 573, 'finalement': 269, 'marier': 411, 'célèbre': 169, 'longue': 387, 'chevelure': 121, 'blonde': 77, 'magiques': 398, 'cherche': 120, 'emparer': 230, 'trésors': 689, 'atterrit': 44, 'hasard': 312, 'investis': 355, 'pouvoir': 510, 'guérison': 307, 'précise': 529, 'incantation': 337, 'déployer': 216, 'dorée': 201, 'fleur': 271, 'pétales': 543, 'or': 459, 'répands': 595, 'magie': 396, 'inverse': 353, 'rends': 562, 'pris': 522, 'guéris': 306, 'blessures': 74, 'éloigne': 732, 'pluie': 501, 'destin': 180, 'impur': 336, 'coupe': 157, 'mèche': 433, 'deviennent': 185, 'bruns': 89, 'perdent': 481, 'leurs': 379, 'effets': 228, 'repoussent': 564, 'fin': 268, 'coupée': 158, 'éteint': 738, 'longueur': 388, 'seul': 620, 'ultime': 696, 'fragment': 284, 'concentré': 144, 'larme': 375, 'frères': 288, 'grimm': 303, 'investie': 354, 'découvre': 211, 'effet': 227, 'guérit': 308, 'cécité': 167, 'biais': 70, 'larmes': 376, 'éclatante': 727, 'lumière': 391, 'sauvé': 609, 'différente': 188, 'jugée': 370, 'trop': 683, 'exil': 246, 'désert': 219, 'principalement': 521, 'destiné': 181, 'âgée': 724, 'mérida': 438, 'fergus': 261, 'elinor': 229, 'règnent': 588, 'dun': 204, 'broch': 87, 'situé': 629, 'highlands': 315, 'écosse': 728, 'adolescente': 3, 'roux': 585, 'bouclés': 82, 'vêtue': 720, 'foncée': 274, 'silhouette': 626, 'élancée': 731, 'gracieuse': 300, 'rebelle': 553, 'repris': 565, 'fougueuse': 283, 'cause': 100, 'conflits': 149, 'permanents': 484, 'puisque': 541, 'tente': 657, 'désespérément': 220, 'comprendre': 143, 'assurer': 41, 'descendance': 178, 'préfère': 531, 'explorer': 248, 'montagnes': 427, 'entraîner': 239, 'tir': 665, 'arc': 34, 'plutôt': 504, 'soumettre': 636, 'éducation': 729, 'rigoureuse': 574, 'stricte': 642, 'imposée': 335, 'rang': 550, 'pleins': 500, 'autre': 50, 'exigences': 245, 'sorte': 631, 'davantage': 171, 'enclin': 232, 'rire': 575, 'facéties': 251, 'aînée': 57, 'éviter': 743, 'conflit': 148, 'laxiste': 377, 'sermonne': 617, 'jamais': 358, 'sauf': 606, 'instances': 345, 'passionnément': 477, 'amoureux': 21, 'tendance': 656, 'provoquer': 528, 'bagarre': 58, 'foncer': 273, 'réfléchir': 592}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>description</th>\n",
       "      <th>text_pp</th>\n",
       "      <th>bow</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>princess</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Blanche-Neige</th>\n",
       "      <td>Blanche-Neige est une princesse d'une très gra...</td>\n",
       "      <td>blanche neige princesse très grande beauté ren...</td>\n",
       "      <td>[0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cendrillon</th>\n",
       "      <td>Cendrillon est une jeune femme soumise à l'aut...</td>\n",
       "      <td>cendrillon jeune femme soumise autorité marâtr...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Aurore</th>\n",
       "      <td>La Princesse Aurore est une princesse, fille u...</td>\n",
       "      <td>princesse aurore princesse fille unique roi st...</td>\n",
       "      <td>[1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ariel</th>\n",
       "      <td>Dans l'illustre film de 1989, Ariel est la plu...</td>\n",
       "      <td>illustre film ariel plus jeune sept filles roi...</td>\n",
       "      <td>[0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Belle</th>\n",
       "      <td>Belle est une jeune fille habitant dans un pet...</td>\n",
       "      <td>belle jeune fille habitant petit village franç...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     description  \\\n",
       "princess                                                           \n",
       "Blanche-Neige  Blanche-Neige est une princesse d'une très gra...   \n",
       "Cendrillon     Cendrillon est une jeune femme soumise à l'aut...   \n",
       "Aurore         La Princesse Aurore est une princesse, fille u...   \n",
       "Ariel          Dans l'illustre film de 1989, Ariel est la plu...   \n",
       "Belle          Belle est une jeune fille habitant dans un pet...   \n",
       "\n",
       "                                                         text_pp  \\\n",
       "princess                                                           \n",
       "Blanche-Neige  blanche neige princesse très grande beauté ren...   \n",
       "Cendrillon     cendrillon jeune femme soumise autorité marâtr...   \n",
       "Aurore         princesse aurore princesse fille unique roi st...   \n",
       "Ariel          illustre film ariel plus jeune sept filles roi...   \n",
       "Belle          belle jeune fille habitant petit village franç...   \n",
       "\n",
       "                                                             bow  \n",
       "princess                                                          \n",
       "Blanche-Neige  [0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \n",
       "Cendrillon     [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \n",
       "Aurore         [1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, ...  \n",
       "Ariel          [0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \n",
       "Belle          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count_vect = CountVectorizer(binary=True)\n",
    "bow_rep = count_vect.fit_transform(df['text_pp'].values)\n",
    "\n",
    "print(\"Vocabulary: \", count_vect.vocabulary_)\n",
    "\n",
    "df['bow'] = [row for row in bow_rep.toarray()]\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6712c020",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Advantages\n",
    "- Intuitive and easy to understand\n",
    "- The implementation is straightforward\n",
    "- The vector of each phrase has a constant size"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69544f0d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Disadvantages\n",
    "- Generates a sparse matrix (solution?)\n",
    "- No notion of similarity between words\n",
    "- Out-of-vocabulary problem\n",
    "- Information order is lost"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fc650f4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# TF-IDF (Term Frequency, Inverse Document Frequency) \n",
    "\n",
    "- It seeks to quantify the importance of a word relative to the other words in the document and in the corpus.\n",
    "- It is frequently used in information retrieval systems and clustering algorithms.\n",
    "- The more a word helps to distinguish a document from others, the higher its TF-IDF score will be."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "980130c2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### TF: Term Frequency\n",
    "\n",
    "- Term frequency: Counting the number of occurrences of a word in a document, divided by the number of words in that document.\n",
    "\n",
    "$$ tf(t,d) = \\dfrac{count(t)}{|d|}$$\n",
    "\n",
    "where\n",
    "- $t$ is term\n",
    "- $d$ is document"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44103207",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "*Note: The term frequency is higher for words frequently used in a document.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4073cb5e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### DF: Document Frequency\n",
    "\n",
    "- Frequency in documents: It is the number of documents that have that word over the total number of documents.\n",
    "\n",
    "$$ df(t,N) = \\dfrac{|\\{d_i: t\\in d_i, i=1, \\dots , N\\}|}{N}$$\n",
    "\n",
    "where\n",
    "- $t$ is term\n",
    "- $N$ is Number of documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef64c423",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "*Note: Frequency in documents is higher for words used in many documents*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a521b2f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## So far we have TF and DF... how do we get to TD-IDF?\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90520e04",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- A specific word to some document will have very low term frequency"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10bb90d7",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Since the goal is to distinguish one document from another, we want to highlight words used frequently in one document but penalize them if they are present in all documents. This is called TF-IDF."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90f342b8",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Then, we obtain:\n",
    "\n",
    "$$tfidf (t,d,N) = tf(df) \\times \\log\\big(\\frac{1}{df(t,N)}\\big)$$\n",
    "\n",
    "where\n",
    "- $t$ is term\n",
    "- $d$ is document\n",
    "- $N$ is Number of documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "087c5706",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- When $t$ is in all documents, $idf = log(1) = 0$\n",
    " \n",
    "- This makes sense since a word that is in all documents is very bad at distinguishing between documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "55cc81c0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>princess</th>\n",
       "      <th>Blanche-Neige</th>\n",
       "      <th>Cendrillon</th>\n",
       "      <th>Aurore</th>\n",
       "      <th>Ariel</th>\n",
       "      <th>Belle</th>\n",
       "      <th>Jasmine</th>\n",
       "      <th>Pocahontas</th>\n",
       "      <th>Mulan</th>\n",
       "      <th>Tiana</th>\n",
       "      <th>Raiponce</th>\n",
       "      <th>Mérida</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>absente</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.081</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>accomplir</th>\n",
       "      <td>0.123</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>admets</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.081</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>adolescente</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>affirme</th>\n",
       "      <td>0.105</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.059</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>éviter</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>événements</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.126</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>être</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.059</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.098</td>\n",
       "      <td>0.119</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>êtres</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.069</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>œuvre</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.069</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>748 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "princess     Blanche-Neige  Cendrillon  Aurore  Ariel  Belle  Jasmine  \\\n",
       "absente              0.000       0.000   0.081  0.000  0.000      0.0   \n",
       "accomplir            0.123       0.000   0.000  0.000  0.000      0.0   \n",
       "admets               0.000       0.000   0.081  0.000  0.000      0.0   \n",
       "adolescente          0.000       0.000   0.000  0.000  0.000      0.0   \n",
       "affirme              0.105       0.000   0.000  0.059  0.000      0.0   \n",
       "...                    ...         ...     ...    ...    ...      ...   \n",
       "éviter               0.000       0.000   0.000  0.000  0.000      0.0   \n",
       "événements           0.000       0.000   0.000  0.000  0.126      0.0   \n",
       "être                 0.000       0.059   0.000  0.000  0.000      0.0   \n",
       "êtres                0.000       0.000   0.000  0.069  0.000      0.0   \n",
       "œuvre                0.000       0.000   0.000  0.069  0.000      0.0   \n",
       "\n",
       "princess     Pocahontas  Mulan  Tiana  Raiponce  Mérida  \n",
       "absente             0.0  0.000  0.000       0.0   0.000  \n",
       "accomplir           0.0  0.000  0.000       0.0   0.000  \n",
       "admets              0.0  0.000  0.000       0.0   0.000  \n",
       "adolescente         0.0  0.000  0.000       0.0   0.092  \n",
       "affirme             0.0  0.000  0.000       0.0   0.000  \n",
       "...                 ...    ...    ...       ...     ...  \n",
       "éviter              0.0  0.000  0.000       0.0   0.092  \n",
       "événements          0.0  0.000  0.000       0.0   0.000  \n",
       "être                0.0  0.098  0.119       0.0   0.000  \n",
       "êtres               0.0  0.000  0.000       0.0   0.000  \n",
       "œuvre               0.0  0.000  0.000       0.0   0.000  \n",
       "\n",
       "[748 rows x 11 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "tfidf_vect = TfidfVectorizer()\n",
    "tfidf = tfidf_vect.fit_transform(df['text_pp'].values)\n",
    "\n",
    "tfidf_matrix = pd.DataFrame(tfidf.toarray(), columns=tfidf_vect.get_feature_names())\n",
    "tfidf_matrix.index = df.index\n",
    "\n",
    "tfidf_matrix.T.round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "389a3040",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 🔮 In the next class\n",
    "\n",
    "#### Going from TF-IDF to Word2Vec\n",
    "\n",
    "- So far, the vector representations of text that we have seen treat linguistic units as atomic units.\n",
    "- Vectors are sparse\n",
    "- They have problems with words outside the vocabulary\n",
    "\n",
    "With distributed representations, such as **word2vec**, we can create dense, low-dimensional representations that capture distributional similarities between words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef222119",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Similarity measures\n",
    "How similar are the documents?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b12fbe32",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Euclidean distance\n",
    "\n",
    "<br>\n",
    "<center><img src='img/dist_euc.png' style='height:350px;'>\n",
    "<small>Image credit: R-Bloggers</small></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "adfcff6f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_b8ed1_row0_col0, #T_b8ed1_row1_col1, #T_b8ed1_row2_col2, #T_b8ed1_row3_col3, #T_b8ed1_row4_col4, #T_b8ed1_row5_col5, #T_b8ed1_row6_col6, #T_b8ed1_row7_col7, #T_b8ed1_row8_col8, #T_b8ed1_row9_col9, #T_b8ed1_row10_col10 {\n",
       "  background-color: #f7fbff;\n",
       "  color: #000000;\n",
       "}\n",
       "#T_b8ed1_row0_col1, #T_b8ed1_row1_col0, #T_b8ed1_row7_col8, #T_b8ed1_row8_col5, #T_b8ed1_row8_col6, #T_b8ed1_row9_col2, #T_b8ed1_row9_col10 {\n",
       "  background-color: #083776;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b8ed1_row0_col2, #T_b8ed1_row1_col10, #T_b8ed1_row2_col6, #T_b8ed1_row3_col5, #T_b8ed1_row3_col10, #T_b8ed1_row5_col8, #T_b8ed1_row6_col8, #T_b8ed1_row6_col9 {\n",
       "  background-color: #083674;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b8ed1_row0_col3, #T_b8ed1_row0_col6, #T_b8ed1_row1_col9, #T_b8ed1_row2_col4, #T_b8ed1_row3_col1, #T_b8ed1_row4_col8, #T_b8ed1_row5_col2, #T_b8ed1_row5_col7, #T_b8ed1_row6_col1, #T_b8ed1_row6_col4, #T_b8ed1_row7_col0, #T_b8ed1_row7_col5, #T_b8ed1_row8_col9, #T_b8ed1_row9_col5 {\n",
       "  background-color: #08316d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b8ed1_row0_col4, #T_b8ed1_row0_col8, #T_b8ed1_row0_col9, #T_b8ed1_row1_col5, #T_b8ed1_row2_col3, #T_b8ed1_row3_col2, #T_b8ed1_row3_col4, #T_b8ed1_row3_col8, #T_b8ed1_row4_col3, #T_b8ed1_row4_col6, #T_b8ed1_row4_col10, #T_b8ed1_row5_col1, #T_b8ed1_row6_col2, #T_b8ed1_row7_col9, #T_b8ed1_row8_col0, #T_b8ed1_row8_col10, #T_b8ed1_row10_col2, #T_b8ed1_row10_col8 {\n",
       "  background-color: #083370;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b8ed1_row0_col5, #T_b8ed1_row1_col7, #T_b8ed1_row2_col10, #T_b8ed1_row3_col9, #T_b8ed1_row6_col7, #T_b8ed1_row7_col6, #T_b8ed1_row9_col7, #T_b8ed1_row10_col1, #T_b8ed1_row10_col9 {\n",
       "  background-color: #083573;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b8ed1_row0_col7, #T_b8ed1_row1_col6, #T_b8ed1_row2_col1, #T_b8ed1_row2_col7, #T_b8ed1_row3_col0, #T_b8ed1_row3_col6, #T_b8ed1_row4_col1, #T_b8ed1_row4_col5, #T_b8ed1_row4_col7, #T_b8ed1_row4_col9, #T_b8ed1_row6_col10, #T_b8ed1_row7_col3, #T_b8ed1_row8_col3, #T_b8ed1_row9_col1, #T_b8ed1_row9_col4, #T_b8ed1_row9_col8, #T_b8ed1_row10_col6 {\n",
       "  background-color: #08326e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b8ed1_row0_col10, #T_b8ed1_row8_col2 {\n",
       "  background-color: #083d7f;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b8ed1_row1_col2, #T_b8ed1_row1_col3, #T_b8ed1_row1_col4, #T_b8ed1_row1_col8, #T_b8ed1_row4_col2, #T_b8ed1_row5_col4, #T_b8ed1_row5_col6, #T_b8ed1_row5_col9, #T_b8ed1_row5_col10, #T_b8ed1_row6_col0, #T_b8ed1_row6_col3, #T_b8ed1_row6_col5, #T_b8ed1_row7_col2, #T_b8ed1_row7_col4, #T_b8ed1_row7_col10, #T_b8ed1_row8_col1, #T_b8ed1_row8_col4, #T_b8ed1_row10_col4, #T_b8ed1_row10_col5, #T_b8ed1_row10_col7 {\n",
       "  background-color: #08306b;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b8ed1_row2_col0, #T_b8ed1_row2_col9, #T_b8ed1_row8_col7, #T_b8ed1_row9_col6 {\n",
       "  background-color: #083877;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b8ed1_row2_col5, #T_b8ed1_row3_col7, #T_b8ed1_row4_col0, #T_b8ed1_row5_col0, #T_b8ed1_row5_col3, #T_b8ed1_row7_col1, #T_b8ed1_row9_col0, #T_b8ed1_row9_col3, #T_b8ed1_row10_col3 {\n",
       "  background-color: #083471;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b8ed1_row2_col8 {\n",
       "  background-color: #083e81;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b8ed1_row10_col0 {\n",
       "  background-color: #083c7d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_b8ed1_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >princess</th>\n",
       "      <th class=\"col_heading level0 col0\" >Blanche-Neige</th>\n",
       "      <th class=\"col_heading level0 col1\" >Cendrillon</th>\n",
       "      <th class=\"col_heading level0 col2\" >Aurore</th>\n",
       "      <th class=\"col_heading level0 col3\" >Ariel</th>\n",
       "      <th class=\"col_heading level0 col4\" >Belle</th>\n",
       "      <th class=\"col_heading level0 col5\" >Jasmine</th>\n",
       "      <th class=\"col_heading level0 col6\" >Pocahontas</th>\n",
       "      <th class=\"col_heading level0 col7\" >Mulan</th>\n",
       "      <th class=\"col_heading level0 col8\" >Tiana</th>\n",
       "      <th class=\"col_heading level0 col9\" >Raiponce</th>\n",
       "      <th class=\"col_heading level0 col10\" >Mérida</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >princess</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "      <th class=\"blank col3\" >&nbsp;</th>\n",
       "      <th class=\"blank col4\" >&nbsp;</th>\n",
       "      <th class=\"blank col5\" >&nbsp;</th>\n",
       "      <th class=\"blank col6\" >&nbsp;</th>\n",
       "      <th class=\"blank col7\" >&nbsp;</th>\n",
       "      <th class=\"blank col8\" >&nbsp;</th>\n",
       "      <th class=\"blank col9\" >&nbsp;</th>\n",
       "      <th class=\"blank col10\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_b8ed1_level0_row0\" class=\"row_heading level0 row0\" >Blanche-Neige</th>\n",
       "      <td id=\"T_b8ed1_row0_col0\" class=\"data row0 col0\" >0.000000</td>\n",
       "      <td id=\"T_b8ed1_row0_col1\" class=\"data row0 col1\" >1.359752</td>\n",
       "      <td id=\"T_b8ed1_row0_col2\" class=\"data row0 col2\" >1.355156</td>\n",
       "      <td id=\"T_b8ed1_row0_col3\" class=\"data row0 col3\" >1.384905</td>\n",
       "      <td id=\"T_b8ed1_row0_col4\" class=\"data row0 col4\" >1.377530</td>\n",
       "      <td id=\"T_b8ed1_row0_col5\" class=\"data row0 col5\" >1.377952</td>\n",
       "      <td id=\"T_b8ed1_row0_col6\" class=\"data row0 col6\" >1.401031</td>\n",
       "      <td id=\"T_b8ed1_row0_col7\" class=\"data row0 col7\" >1.395073</td>\n",
       "      <td id=\"T_b8ed1_row0_col8\" class=\"data row0 col8\" >1.382965</td>\n",
       "      <td id=\"T_b8ed1_row0_col9\" class=\"data row0 col9\" >1.378498</td>\n",
       "      <td id=\"T_b8ed1_row0_col10\" class=\"data row0 col10\" >1.331469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b8ed1_level0_row1\" class=\"row_heading level0 row1\" >Cendrillon</th>\n",
       "      <td id=\"T_b8ed1_row1_col0\" class=\"data row1 col0\" >1.359752</td>\n",
       "      <td id=\"T_b8ed1_row1_col1\" class=\"data row1 col1\" >0.000000</td>\n",
       "      <td id=\"T_b8ed1_row1_col2\" class=\"data row1 col2\" >1.388122</td>\n",
       "      <td id=\"T_b8ed1_row1_col3\" class=\"data row1 col3\" >1.394957</td>\n",
       "      <td id=\"T_b8ed1_row1_col4\" class=\"data row1 col4\" >1.389816</td>\n",
       "      <td id=\"T_b8ed1_row1_col5\" class=\"data row1 col5\" >1.385304</td>\n",
       "      <td id=\"T_b8ed1_row1_col6\" class=\"data row1 col6\" >1.395007</td>\n",
       "      <td id=\"T_b8ed1_row1_col7\" class=\"data row1 col7\" >1.378746</td>\n",
       "      <td id=\"T_b8ed1_row1_col8\" class=\"data row1 col8\" >1.402277</td>\n",
       "      <td id=\"T_b8ed1_row1_col9\" class=\"data row1 col9\" >1.391096</td>\n",
       "      <td id=\"T_b8ed1_row1_col10\" class=\"data row1 col10\" >1.372576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b8ed1_level0_row2\" class=\"row_heading level0 row2\" >Aurore</th>\n",
       "      <td id=\"T_b8ed1_row2_col0\" class=\"data row2 col0\" >1.355156</td>\n",
       "      <td id=\"T_b8ed1_row2_col1\" class=\"data row2 col1\" >1.388122</td>\n",
       "      <td id=\"T_b8ed1_row2_col2\" class=\"data row2 col2\" >0.000000</td>\n",
       "      <td id=\"T_b8ed1_row2_col3\" class=\"data row2 col3\" >1.374415</td>\n",
       "      <td id=\"T_b8ed1_row2_col4\" class=\"data row2 col4\" >1.389045</td>\n",
       "      <td id=\"T_b8ed1_row2_col5\" class=\"data row2 col5\" >1.384152</td>\n",
       "      <td id=\"T_b8ed1_row2_col6\" class=\"data row2 col6\" >1.373416</td>\n",
       "      <td id=\"T_b8ed1_row2_col7\" class=\"data row2 col7\" >1.392205</td>\n",
       "      <td id=\"T_b8ed1_row2_col8\" class=\"data row2 col8\" >1.321143</td>\n",
       "      <td id=\"T_b8ed1_row2_col9\" class=\"data row2 col9\" >1.348926</td>\n",
       "      <td id=\"T_b8ed1_row2_col10\" class=\"data row2 col10\" >1.375829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b8ed1_level0_row3\" class=\"row_heading level0 row3\" >Ariel</th>\n",
       "      <td id=\"T_b8ed1_row3_col0\" class=\"data row3 col0\" >1.384905</td>\n",
       "      <td id=\"T_b8ed1_row3_col1\" class=\"data row3 col1\" >1.394957</td>\n",
       "      <td id=\"T_b8ed1_row3_col2\" class=\"data row3 col2\" >1.374415</td>\n",
       "      <td id=\"T_b8ed1_row3_col3\" class=\"data row3 col3\" >0.000000</td>\n",
       "      <td id=\"T_b8ed1_row3_col4\" class=\"data row3 col4\" >1.373776</td>\n",
       "      <td id=\"T_b8ed1_row3_col5\" class=\"data row3 col5\" >1.371409</td>\n",
       "      <td id=\"T_b8ed1_row3_col6\" class=\"data row3 col6\" >1.394758</td>\n",
       "      <td id=\"T_b8ed1_row3_col7\" class=\"data row3 col7\" >1.382011</td>\n",
       "      <td id=\"T_b8ed1_row3_col8\" class=\"data row3 col8\" >1.381844</td>\n",
       "      <td id=\"T_b8ed1_row3_col9\" class=\"data row3 col9\" >1.368437</td>\n",
       "      <td id=\"T_b8ed1_row3_col10\" class=\"data row3 col10\" >1.370533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b8ed1_level0_row4\" class=\"row_heading level0 row4\" >Belle</th>\n",
       "      <td id=\"T_b8ed1_row4_col0\" class=\"data row4 col0\" >1.377530</td>\n",
       "      <td id=\"T_b8ed1_row4_col1\" class=\"data row4 col1\" >1.389816</td>\n",
       "      <td id=\"T_b8ed1_row4_col2\" class=\"data row4 col2\" >1.389045</td>\n",
       "      <td id=\"T_b8ed1_row4_col3\" class=\"data row4 col3\" >1.373776</td>\n",
       "      <td id=\"T_b8ed1_row4_col4\" class=\"data row4 col4\" >0.000000</td>\n",
       "      <td id=\"T_b8ed1_row4_col5\" class=\"data row4 col5\" >1.394846</td>\n",
       "      <td id=\"T_b8ed1_row4_col6\" class=\"data row4 col6\" >1.388032</td>\n",
       "      <td id=\"T_b8ed1_row4_col7\" class=\"data row4 col7\" >1.391806</td>\n",
       "      <td id=\"T_b8ed1_row4_col8\" class=\"data row4 col8\" >1.393957</td>\n",
       "      <td id=\"T_b8ed1_row4_col9\" class=\"data row4 col9\" >1.381135</td>\n",
       "      <td id=\"T_b8ed1_row4_col10\" class=\"data row4 col10\" >1.390087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b8ed1_level0_row5\" class=\"row_heading level0 row5\" >Jasmine</th>\n",
       "      <td id=\"T_b8ed1_row5_col0\" class=\"data row5 col0\" >1.377952</td>\n",
       "      <td id=\"T_b8ed1_row5_col1\" class=\"data row5 col1\" >1.385304</td>\n",
       "      <td id=\"T_b8ed1_row5_col2\" class=\"data row5 col2\" >1.384152</td>\n",
       "      <td id=\"T_b8ed1_row5_col3\" class=\"data row5 col3\" >1.371409</td>\n",
       "      <td id=\"T_b8ed1_row5_col4\" class=\"data row5 col4\" >1.394846</td>\n",
       "      <td id=\"T_b8ed1_row5_col5\" class=\"data row5 col5\" >0.000000</td>\n",
       "      <td id=\"T_b8ed1_row5_col6\" class=\"data row5 col6\" >1.407037</td>\n",
       "      <td id=\"T_b8ed1_row5_col7\" class=\"data row5 col7\" >1.399469</td>\n",
       "      <td id=\"T_b8ed1_row5_col8\" class=\"data row5 col8\" >1.366180</td>\n",
       "      <td id=\"T_b8ed1_row5_col9\" class=\"data row5 col9\" >1.397112</td>\n",
       "      <td id=\"T_b8ed1_row5_col10\" class=\"data row5 col10\" >1.402512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b8ed1_level0_row6\" class=\"row_heading level0 row6\" >Pocahontas</th>\n",
       "      <td id=\"T_b8ed1_row6_col0\" class=\"data row6 col0\" >1.401031</td>\n",
       "      <td id=\"T_b8ed1_row6_col1\" class=\"data row6 col1\" >1.395007</td>\n",
       "      <td id=\"T_b8ed1_row6_col2\" class=\"data row6 col2\" >1.373416</td>\n",
       "      <td id=\"T_b8ed1_row6_col3\" class=\"data row6 col3\" >1.394758</td>\n",
       "      <td id=\"T_b8ed1_row6_col4\" class=\"data row6 col4\" >1.388032</td>\n",
       "      <td id=\"T_b8ed1_row6_col5\" class=\"data row6 col5\" >1.407037</td>\n",
       "      <td id=\"T_b8ed1_row6_col6\" class=\"data row6 col6\" >0.000000</td>\n",
       "      <td id=\"T_b8ed1_row6_col7\" class=\"data row6 col7\" >1.376486</td>\n",
       "      <td id=\"T_b8ed1_row6_col8\" class=\"data row6 col8\" >1.368221</td>\n",
       "      <td id=\"T_b8ed1_row6_col9\" class=\"data row6 col9\" >1.360213</td>\n",
       "      <td id=\"T_b8ed1_row6_col10\" class=\"data row6 col10\" >1.396024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b8ed1_level0_row7\" class=\"row_heading level0 row7\" >Mulan</th>\n",
       "      <td id=\"T_b8ed1_row7_col0\" class=\"data row7 col0\" >1.395073</td>\n",
       "      <td id=\"T_b8ed1_row7_col1\" class=\"data row7 col1\" >1.378746</td>\n",
       "      <td id=\"T_b8ed1_row7_col2\" class=\"data row7 col2\" >1.392205</td>\n",
       "      <td id=\"T_b8ed1_row7_col3\" class=\"data row7 col3\" >1.382011</td>\n",
       "      <td id=\"T_b8ed1_row7_col4\" class=\"data row7 col4\" >1.391806</td>\n",
       "      <td id=\"T_b8ed1_row7_col5\" class=\"data row7 col5\" >1.399469</td>\n",
       "      <td id=\"T_b8ed1_row7_col6\" class=\"data row7 col6\" >1.376486</td>\n",
       "      <td id=\"T_b8ed1_row7_col7\" class=\"data row7 col7\" >0.000000</td>\n",
       "      <td id=\"T_b8ed1_row7_col8\" class=\"data row7 col8\" >1.360454</td>\n",
       "      <td id=\"T_b8ed1_row7_col9\" class=\"data row7 col9\" >1.379291</td>\n",
       "      <td id=\"T_b8ed1_row7_col10\" class=\"data row7 col10\" >1.407170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b8ed1_level0_row8\" class=\"row_heading level0 row8\" >Tiana</th>\n",
       "      <td id=\"T_b8ed1_row8_col0\" class=\"data row8 col0\" >1.382965</td>\n",
       "      <td id=\"T_b8ed1_row8_col1\" class=\"data row8 col1\" >1.402277</td>\n",
       "      <td id=\"T_b8ed1_row8_col2\" class=\"data row8 col2\" >1.321143</td>\n",
       "      <td id=\"T_b8ed1_row8_col3\" class=\"data row8 col3\" >1.381844</td>\n",
       "      <td id=\"T_b8ed1_row8_col4\" class=\"data row8 col4\" >1.393957</td>\n",
       "      <td id=\"T_b8ed1_row8_col5\" class=\"data row8 col5\" >1.366180</td>\n",
       "      <td id=\"T_b8ed1_row8_col6\" class=\"data row8 col6\" >1.368221</td>\n",
       "      <td id=\"T_b8ed1_row8_col7\" class=\"data row8 col7\" >1.360454</td>\n",
       "      <td id=\"T_b8ed1_row8_col8\" class=\"data row8 col8\" >0.000000</td>\n",
       "      <td id=\"T_b8ed1_row8_col9\" class=\"data row8 col9\" >1.388926</td>\n",
       "      <td id=\"T_b8ed1_row8_col10\" class=\"data row8 col10\" >1.385810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b8ed1_level0_row9\" class=\"row_heading level0 row9\" >Raiponce</th>\n",
       "      <td id=\"T_b8ed1_row9_col0\" class=\"data row9 col0\" >1.378498</td>\n",
       "      <td id=\"T_b8ed1_row9_col1\" class=\"data row9 col1\" >1.391096</td>\n",
       "      <td id=\"T_b8ed1_row9_col2\" class=\"data row9 col2\" >1.348926</td>\n",
       "      <td id=\"T_b8ed1_row9_col3\" class=\"data row9 col3\" >1.368437</td>\n",
       "      <td id=\"T_b8ed1_row9_col4\" class=\"data row9 col4\" >1.381135</td>\n",
       "      <td id=\"T_b8ed1_row9_col5\" class=\"data row9 col5\" >1.397112</td>\n",
       "      <td id=\"T_b8ed1_row9_col6\" class=\"data row9 col6\" >1.360213</td>\n",
       "      <td id=\"T_b8ed1_row9_col7\" class=\"data row9 col7\" >1.379291</td>\n",
       "      <td id=\"T_b8ed1_row9_col8\" class=\"data row9 col8\" >1.388926</td>\n",
       "      <td id=\"T_b8ed1_row9_col9\" class=\"data row9 col9\" >0.000000</td>\n",
       "      <td id=\"T_b8ed1_row9_col10\" class=\"data row9 col10\" >1.367995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b8ed1_level0_row10\" class=\"row_heading level0 row10\" >Mérida</th>\n",
       "      <td id=\"T_b8ed1_row10_col0\" class=\"data row10 col0\" >1.331469</td>\n",
       "      <td id=\"T_b8ed1_row10_col1\" class=\"data row10 col1\" >1.372576</td>\n",
       "      <td id=\"T_b8ed1_row10_col2\" class=\"data row10 col2\" >1.375829</td>\n",
       "      <td id=\"T_b8ed1_row10_col3\" class=\"data row10 col3\" >1.370533</td>\n",
       "      <td id=\"T_b8ed1_row10_col4\" class=\"data row10 col4\" >1.390087</td>\n",
       "      <td id=\"T_b8ed1_row10_col5\" class=\"data row10 col5\" >1.402512</td>\n",
       "      <td id=\"T_b8ed1_row10_col6\" class=\"data row10 col6\" >1.396024</td>\n",
       "      <td id=\"T_b8ed1_row10_col7\" class=\"data row10 col7\" >1.407170</td>\n",
       "      <td id=\"T_b8ed1_row10_col8\" class=\"data row10 col8\" >1.385810</td>\n",
       "      <td id=\"T_b8ed1_row10_col9\" class=\"data row10 col9\" >1.367995</td>\n",
       "      <td id=\"T_b8ed1_row10_col10\" class=\"data row10 col10\" >0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f91ecbbad60>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import euclidean_distances\n",
    "\n",
    "dist_euc = euclidean_distances(tfidf_matrix.values)\n",
    "dist_euc = pd.DataFrame(dist_euc, columns = df.index, index = df.index)\n",
    "\n",
    "dist_euc.style.background_gradient(cmap='Blues')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28018a97",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "sns.heatmap(dist_euc, annot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9325fc0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Cosine distance\n",
    "\n",
    "<br>\n",
    "<center><img src='img/dist_cos.png' style='height:350px;'>\n",
    "<small>Image credit: R-Bloggers</small></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "b7caf908",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_16bb8_row0_col0, #T_16bb8_row1_col1, #T_16bb8_row2_col2, #T_16bb8_row3_col3, #T_16bb8_row4_col4, #T_16bb8_row5_col5, #T_16bb8_row6_col6, #T_16bb8_row7_col7, #T_16bb8_row8_col8, #T_16bb8_row9_col9, #T_16bb8_row10_col10 {\n",
       "  background-color: #f7fbff;\n",
       "  color: #000000;\n",
       "}\n",
       "#T_16bb8_row0_col1, #T_16bb8_row7_col8, #T_16bb8_row9_col2 {\n",
       "  background-color: #084082;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row0_col2, #T_16bb8_row3_col10, #T_16bb8_row5_col8, #T_16bb8_row6_col9, #T_16bb8_row8_col6 {\n",
       "  background-color: #083d7f;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row0_col3, #T_16bb8_row4_col8, #T_16bb8_row9_col5, #T_16bb8_row10_col6 {\n",
       "  background-color: #083370;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row0_col4, #T_16bb8_row0_col9, #T_16bb8_row3_col2, #T_16bb8_row4_col6, #T_16bb8_row4_col10, #T_16bb8_row5_col1, #T_16bb8_row6_col2, #T_16bb8_row7_col9, #T_16bb8_row8_col0 {\n",
       "  background-color: #083674;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row0_col5, #T_16bb8_row1_col7, #T_16bb8_row3_col9, #T_16bb8_row7_col6, #T_16bb8_row9_col7, #T_16bb8_row10_col1, #T_16bb8_row10_col9 {\n",
       "  background-color: #083a7a;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row0_col6, #T_16bb8_row1_col9, #T_16bb8_row2_col4, #T_16bb8_row3_col1, #T_16bb8_row5_col2, #T_16bb8_row5_col7, #T_16bb8_row6_col1, #T_16bb8_row6_col4, #T_16bb8_row7_col0, #T_16bb8_row7_col5, #T_16bb8_row8_col9 {\n",
       "  background-color: #08326e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row0_col7, #T_16bb8_row1_col6, #T_16bb8_row3_col6, #T_16bb8_row4_col1, #T_16bb8_row4_col5, #T_16bb8_row6_col10, #T_16bb8_row7_col3, #T_16bb8_row8_col3, #T_16bb8_row9_col1, #T_16bb8_row9_col8 {\n",
       "  background-color: #083471;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row0_col8, #T_16bb8_row1_col5, #T_16bb8_row2_col3, #T_16bb8_row3_col4, #T_16bb8_row3_col8, #T_16bb8_row4_col3, #T_16bb8_row8_col10 {\n",
       "  background-color: #083776;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row0_col10 {\n",
       "  background-color: #084b93;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row1_col0, #T_16bb8_row8_col5, #T_16bb8_row9_col10 {\n",
       "  background-color: #083e81;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row1_col2, #T_16bb8_row1_col4, #T_16bb8_row4_col2, #T_16bb8_row5_col10, #T_16bb8_row7_col4, #T_16bb8_row10_col4, #T_16bb8_row10_col5 {\n",
       "  background-color: #08316d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row1_col3, #T_16bb8_row1_col8, #T_16bb8_row5_col4, #T_16bb8_row5_col6, #T_16bb8_row5_col9, #T_16bb8_row6_col0, #T_16bb8_row6_col3, #T_16bb8_row6_col5, #T_16bb8_row7_col2, #T_16bb8_row7_col10, #T_16bb8_row8_col1, #T_16bb8_row8_col4, #T_16bb8_row10_col7 {\n",
       "  background-color: #08306b;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row1_col10, #T_16bb8_row2_col6, #T_16bb8_row3_col5, #T_16bb8_row6_col8 {\n",
       "  background-color: #083c7d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row2_col0, #T_16bb8_row8_col7, #T_16bb8_row9_col6 {\n",
       "  background-color: #084184;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row2_col1, #T_16bb8_row2_col7, #T_16bb8_row3_col0, #T_16bb8_row4_col7, #T_16bb8_row4_col9, #T_16bb8_row9_col4, #T_16bb8_row10_col2, #T_16bb8_row10_col8 {\n",
       "  background-color: #083573;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row2_col5, #T_16bb8_row4_col0, #T_16bb8_row5_col0, #T_16bb8_row5_col3, #T_16bb8_row7_col1, #T_16bb8_row9_col0, #T_16bb8_row10_col3 {\n",
       "  background-color: #083877;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row2_col8 {\n",
       "  background-color: #084d96;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row2_col9 {\n",
       "  background-color: #084285;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row2_col10, #T_16bb8_row6_col7 {\n",
       "  background-color: #083b7c;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row3_col7, #T_16bb8_row9_col3 {\n",
       "  background-color: #083979;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row8_col2 {\n",
       "  background-color: #084a91;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_16bb8_row10_col0 {\n",
       "  background-color: #084990;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_16bb8_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >princess</th>\n",
       "      <th class=\"col_heading level0 col0\" >Blanche-Neige</th>\n",
       "      <th class=\"col_heading level0 col1\" >Cendrillon</th>\n",
       "      <th class=\"col_heading level0 col2\" >Aurore</th>\n",
       "      <th class=\"col_heading level0 col3\" >Ariel</th>\n",
       "      <th class=\"col_heading level0 col4\" >Belle</th>\n",
       "      <th class=\"col_heading level0 col5\" >Jasmine</th>\n",
       "      <th class=\"col_heading level0 col6\" >Pocahontas</th>\n",
       "      <th class=\"col_heading level0 col7\" >Mulan</th>\n",
       "      <th class=\"col_heading level0 col8\" >Tiana</th>\n",
       "      <th class=\"col_heading level0 col9\" >Raiponce</th>\n",
       "      <th class=\"col_heading level0 col10\" >Mérida</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >princess</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "      <th class=\"blank col3\" >&nbsp;</th>\n",
       "      <th class=\"blank col4\" >&nbsp;</th>\n",
       "      <th class=\"blank col5\" >&nbsp;</th>\n",
       "      <th class=\"blank col6\" >&nbsp;</th>\n",
       "      <th class=\"blank col7\" >&nbsp;</th>\n",
       "      <th class=\"blank col8\" >&nbsp;</th>\n",
       "      <th class=\"blank col9\" >&nbsp;</th>\n",
       "      <th class=\"blank col10\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_16bb8_level0_row0\" class=\"row_heading level0 row0\" >Blanche-Neige</th>\n",
       "      <td id=\"T_16bb8_row0_col0\" class=\"data row0 col0\" >0.000000</td>\n",
       "      <td id=\"T_16bb8_row0_col1\" class=\"data row0 col1\" >0.924462</td>\n",
       "      <td id=\"T_16bb8_row0_col2\" class=\"data row0 col2\" >0.918223</td>\n",
       "      <td id=\"T_16bb8_row0_col3\" class=\"data row0 col3\" >0.958980</td>\n",
       "      <td id=\"T_16bb8_row0_col4\" class=\"data row0 col4\" >0.948795</td>\n",
       "      <td id=\"T_16bb8_row0_col5\" class=\"data row0 col5\" >0.949376</td>\n",
       "      <td id=\"T_16bb8_row0_col6\" class=\"data row0 col6\" >0.981444</td>\n",
       "      <td id=\"T_16bb8_row0_col7\" class=\"data row0 col7\" >0.973115</td>\n",
       "      <td id=\"T_16bb8_row0_col8\" class=\"data row0 col8\" >0.956296</td>\n",
       "      <td id=\"T_16bb8_row0_col9\" class=\"data row0 col9\" >0.950128</td>\n",
       "      <td id=\"T_16bb8_row0_col10\" class=\"data row0 col10\" >0.886405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_16bb8_level0_row1\" class=\"row_heading level0 row1\" >Cendrillon</th>\n",
       "      <td id=\"T_16bb8_row1_col0\" class=\"data row1 col0\" >0.924462</td>\n",
       "      <td id=\"T_16bb8_row1_col1\" class=\"data row1 col1\" >0.000000</td>\n",
       "      <td id=\"T_16bb8_row1_col2\" class=\"data row1 col2\" >0.963441</td>\n",
       "      <td id=\"T_16bb8_row1_col3\" class=\"data row1 col3\" >0.972953</td>\n",
       "      <td id=\"T_16bb8_row1_col4\" class=\"data row1 col4\" >0.965795</td>\n",
       "      <td id=\"T_16bb8_row1_col5\" class=\"data row1 col5\" >0.959533</td>\n",
       "      <td id=\"T_16bb8_row1_col6\" class=\"data row1 col6\" >0.973023</td>\n",
       "      <td id=\"T_16bb8_row1_col7\" class=\"data row1 col7\" >0.950471</td>\n",
       "      <td id=\"T_16bb8_row1_col8\" class=\"data row1 col8\" >0.983190</td>\n",
       "      <td id=\"T_16bb8_row1_col9\" class=\"data row1 col9\" >0.967574</td>\n",
       "      <td id=\"T_16bb8_row1_col10\" class=\"data row1 col10\" >0.941983</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_16bb8_level0_row2\" class=\"row_heading level0 row2\" >Aurore</th>\n",
       "      <td id=\"T_16bb8_row2_col0\" class=\"data row2 col0\" >0.918223</td>\n",
       "      <td id=\"T_16bb8_row2_col1\" class=\"data row2 col1\" >0.963441</td>\n",
       "      <td id=\"T_16bb8_row2_col2\" class=\"data row2 col2\" >0.000000</td>\n",
       "      <td id=\"T_16bb8_row2_col3\" class=\"data row2 col3\" >0.944508</td>\n",
       "      <td id=\"T_16bb8_row2_col4\" class=\"data row2 col4\" >0.964723</td>\n",
       "      <td id=\"T_16bb8_row2_col5\" class=\"data row2 col5\" >0.957938</td>\n",
       "      <td id=\"T_16bb8_row2_col6\" class=\"data row2 col6\" >0.943136</td>\n",
       "      <td id=\"T_16bb8_row2_col7\" class=\"data row2 col7\" >0.969117</td>\n",
       "      <td id=\"T_16bb8_row2_col8\" class=\"data row2 col8\" >0.872710</td>\n",
       "      <td id=\"T_16bb8_row2_col9\" class=\"data row2 col9\" >0.909801</td>\n",
       "      <td id=\"T_16bb8_row2_col10\" class=\"data row2 col10\" >0.946452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_16bb8_level0_row3\" class=\"row_heading level0 row3\" >Ariel</th>\n",
       "      <td id=\"T_16bb8_row3_col0\" class=\"data row3 col0\" >0.958980</td>\n",
       "      <td id=\"T_16bb8_row3_col1\" class=\"data row3 col1\" >0.972953</td>\n",
       "      <td id=\"T_16bb8_row3_col2\" class=\"data row3 col2\" >0.944508</td>\n",
       "      <td id=\"T_16bb8_row3_col3\" class=\"data row3 col3\" >0.000000</td>\n",
       "      <td id=\"T_16bb8_row3_col4\" class=\"data row3 col4\" >0.943630</td>\n",
       "      <td id=\"T_16bb8_row3_col5\" class=\"data row3 col5\" >0.940381</td>\n",
       "      <td id=\"T_16bb8_row3_col6\" class=\"data row3 col6\" >0.972675</td>\n",
       "      <td id=\"T_16bb8_row3_col7\" class=\"data row3 col7\" >0.954977</td>\n",
       "      <td id=\"T_16bb8_row3_col8\" class=\"data row3 col8\" >0.954746</td>\n",
       "      <td id=\"T_16bb8_row3_col9\" class=\"data row3 col9\" >0.936310</td>\n",
       "      <td id=\"T_16bb8_row3_col10\" class=\"data row3 col10\" >0.939180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_16bb8_level0_row4\" class=\"row_heading level0 row4\" >Belle</th>\n",
       "      <td id=\"T_16bb8_row4_col0\" class=\"data row4 col0\" >0.948795</td>\n",
       "      <td id=\"T_16bb8_row4_col1\" class=\"data row4 col1\" >0.965795</td>\n",
       "      <td id=\"T_16bb8_row4_col2\" class=\"data row4 col2\" >0.964723</td>\n",
       "      <td id=\"T_16bb8_row4_col3\" class=\"data row4 col3\" >0.943630</td>\n",
       "      <td id=\"T_16bb8_row4_col4\" class=\"data row4 col4\" >0.000000</td>\n",
       "      <td id=\"T_16bb8_row4_col5\" class=\"data row4 col5\" >0.972798</td>\n",
       "      <td id=\"T_16bb8_row4_col6\" class=\"data row4 col6\" >0.963316</td>\n",
       "      <td id=\"T_16bb8_row4_col7\" class=\"data row4 col7\" >0.968562</td>\n",
       "      <td id=\"T_16bb8_row4_col8\" class=\"data row4 col8\" >0.971558</td>\n",
       "      <td id=\"T_16bb8_row4_col9\" class=\"data row4 col9\" >0.953767</td>\n",
       "      <td id=\"T_16bb8_row4_col10\" class=\"data row4 col10\" >0.966170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_16bb8_level0_row5\" class=\"row_heading level0 row5\" >Jasmine</th>\n",
       "      <td id=\"T_16bb8_row5_col0\" class=\"data row5 col0\" >0.949376</td>\n",
       "      <td id=\"T_16bb8_row5_col1\" class=\"data row5 col1\" >0.959533</td>\n",
       "      <td id=\"T_16bb8_row5_col2\" class=\"data row5 col2\" >0.957938</td>\n",
       "      <td id=\"T_16bb8_row5_col3\" class=\"data row5 col3\" >0.940381</td>\n",
       "      <td id=\"T_16bb8_row5_col4\" class=\"data row5 col4\" >0.972798</td>\n",
       "      <td id=\"T_16bb8_row5_col5\" class=\"data row5 col5\" >0.000000</td>\n",
       "      <td id=\"T_16bb8_row5_col6\" class=\"data row5 col6\" >0.989876</td>\n",
       "      <td id=\"T_16bb8_row5_col7\" class=\"data row5 col7\" >0.979257</td>\n",
       "      <td id=\"T_16bb8_row5_col8\" class=\"data row5 col8\" >0.933223</td>\n",
       "      <td id=\"T_16bb8_row5_col9\" class=\"data row5 col9\" >0.975960</td>\n",
       "      <td id=\"T_16bb8_row5_col10\" class=\"data row5 col10\" >0.983521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_16bb8_level0_row6\" class=\"row_heading level0 row6\" >Pocahontas</th>\n",
       "      <td id=\"T_16bb8_row6_col0\" class=\"data row6 col0\" >0.981444</td>\n",
       "      <td id=\"T_16bb8_row6_col1\" class=\"data row6 col1\" >0.973023</td>\n",
       "      <td id=\"T_16bb8_row6_col2\" class=\"data row6 col2\" >0.943136</td>\n",
       "      <td id=\"T_16bb8_row6_col3\" class=\"data row6 col3\" >0.972675</td>\n",
       "      <td id=\"T_16bb8_row6_col4\" class=\"data row6 col4\" >0.963316</td>\n",
       "      <td id=\"T_16bb8_row6_col5\" class=\"data row6 col5\" >0.989876</td>\n",
       "      <td id=\"T_16bb8_row6_col6\" class=\"data row6 col6\" >0.000000</td>\n",
       "      <td id=\"T_16bb8_row6_col7\" class=\"data row6 col7\" >0.947356</td>\n",
       "      <td id=\"T_16bb8_row6_col8\" class=\"data row6 col8\" >0.936014</td>\n",
       "      <td id=\"T_16bb8_row6_col9\" class=\"data row6 col9\" >0.925090</td>\n",
       "      <td id=\"T_16bb8_row6_col10\" class=\"data row6 col10\" >0.974441</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_16bb8_level0_row7\" class=\"row_heading level0 row7\" >Mulan</th>\n",
       "      <td id=\"T_16bb8_row7_col0\" class=\"data row7 col0\" >0.973115</td>\n",
       "      <td id=\"T_16bb8_row7_col1\" class=\"data row7 col1\" >0.950471</td>\n",
       "      <td id=\"T_16bb8_row7_col2\" class=\"data row7 col2\" >0.969117</td>\n",
       "      <td id=\"T_16bb8_row7_col3\" class=\"data row7 col3\" >0.954977</td>\n",
       "      <td id=\"T_16bb8_row7_col4\" class=\"data row7 col4\" >0.968562</td>\n",
       "      <td id=\"T_16bb8_row7_col5\" class=\"data row7 col5\" >0.979257</td>\n",
       "      <td id=\"T_16bb8_row7_col6\" class=\"data row7 col6\" >0.947356</td>\n",
       "      <td id=\"T_16bb8_row7_col7\" class=\"data row7 col7\" >0.000000</td>\n",
       "      <td id=\"T_16bb8_row7_col8\" class=\"data row7 col8\" >0.925417</td>\n",
       "      <td id=\"T_16bb8_row7_col9\" class=\"data row7 col9\" >0.951222</td>\n",
       "      <td id=\"T_16bb8_row7_col10\" class=\"data row7 col10\" >0.990064</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_16bb8_level0_row8\" class=\"row_heading level0 row8\" >Tiana</th>\n",
       "      <td id=\"T_16bb8_row8_col0\" class=\"data row8 col0\" >0.956296</td>\n",
       "      <td id=\"T_16bb8_row8_col1\" class=\"data row8 col1\" >0.983190</td>\n",
       "      <td id=\"T_16bb8_row8_col2\" class=\"data row8 col2\" >0.872710</td>\n",
       "      <td id=\"T_16bb8_row8_col3\" class=\"data row8 col3\" >0.954746</td>\n",
       "      <td id=\"T_16bb8_row8_col4\" class=\"data row8 col4\" >0.971558</td>\n",
       "      <td id=\"T_16bb8_row8_col5\" class=\"data row8 col5\" >0.933223</td>\n",
       "      <td id=\"T_16bb8_row8_col6\" class=\"data row8 col6\" >0.936014</td>\n",
       "      <td id=\"T_16bb8_row8_col7\" class=\"data row8 col7\" >0.925417</td>\n",
       "      <td id=\"T_16bb8_row8_col8\" class=\"data row8 col8\" >0.000000</td>\n",
       "      <td id=\"T_16bb8_row8_col9\" class=\"data row8 col9\" >0.964558</td>\n",
       "      <td id=\"T_16bb8_row8_col10\" class=\"data row8 col10\" >0.960235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_16bb8_level0_row9\" class=\"row_heading level0 row9\" >Raiponce</th>\n",
       "      <td id=\"T_16bb8_row9_col0\" class=\"data row9 col0\" >0.950128</td>\n",
       "      <td id=\"T_16bb8_row9_col1\" class=\"data row9 col1\" >0.967574</td>\n",
       "      <td id=\"T_16bb8_row9_col2\" class=\"data row9 col2\" >0.909801</td>\n",
       "      <td id=\"T_16bb8_row9_col3\" class=\"data row9 col3\" >0.936310</td>\n",
       "      <td id=\"T_16bb8_row9_col4\" class=\"data row9 col4\" >0.953767</td>\n",
       "      <td id=\"T_16bb8_row9_col5\" class=\"data row9 col5\" >0.975960</td>\n",
       "      <td id=\"T_16bb8_row9_col6\" class=\"data row9 col6\" >0.925090</td>\n",
       "      <td id=\"T_16bb8_row9_col7\" class=\"data row9 col7\" >0.951222</td>\n",
       "      <td id=\"T_16bb8_row9_col8\" class=\"data row9 col8\" >0.964558</td>\n",
       "      <td id=\"T_16bb8_row9_col9\" class=\"data row9 col9\" >0.000000</td>\n",
       "      <td id=\"T_16bb8_row9_col10\" class=\"data row9 col10\" >0.935705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_16bb8_level0_row10\" class=\"row_heading level0 row10\" >Mérida</th>\n",
       "      <td id=\"T_16bb8_row10_col0\" class=\"data row10 col0\" >0.886405</td>\n",
       "      <td id=\"T_16bb8_row10_col1\" class=\"data row10 col1\" >0.941983</td>\n",
       "      <td id=\"T_16bb8_row10_col2\" class=\"data row10 col2\" >0.946452</td>\n",
       "      <td id=\"T_16bb8_row10_col3\" class=\"data row10 col3\" >0.939180</td>\n",
       "      <td id=\"T_16bb8_row10_col4\" class=\"data row10 col4\" >0.966170</td>\n",
       "      <td id=\"T_16bb8_row10_col5\" class=\"data row10 col5\" >0.983521</td>\n",
       "      <td id=\"T_16bb8_row10_col6\" class=\"data row10 col6\" >0.974441</td>\n",
       "      <td id=\"T_16bb8_row10_col7\" class=\"data row10 col7\" >0.990064</td>\n",
       "      <td id=\"T_16bb8_row10_col8\" class=\"data row10 col8\" >0.960235</td>\n",
       "      <td id=\"T_16bb8_row10_col9\" class=\"data row10 col9\" >0.935705</td>\n",
       "      <td id=\"T_16bb8_row10_col10\" class=\"data row10 col10\" >0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f91e97bddc0>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import cosine_distances\n",
    "\n",
    "dist_cos = cosine_distances(tfidf_matrix.values)\n",
    "dist_cos = pd.DataFrame(dist_cos, columns = df.index, index = df.index)\n",
    "dist_cos.style.background_gradient(cmap='Blues')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54c2a4bb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ⏪ Today's recap\n",
    "\n",
    "- Vectorial representation of texts\n",
    "    - Bag of Words\n",
    "    - TF-IDF\n",
    "- Distance metrics\n",
    "    - Euclidean distance\n",
    "    - Cosine distance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3c47662",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='img/bye.gif' style='height:250px;'></center> \n",
    "\n",
    "# Next class: Word2Vec\n",
    "# See you next Monday!"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}