Class 20 - Word2Vec/class_20_word2vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cfd2c9f4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# We'll start at 2:10ish PM \n",
    "\n",
    "Waiting for others to join the call :)\n",
    "\n",
    "https://www.youtube.com/watch?v=VHoT4N43jK8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa3b1a93",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='img/cdy.png' style='width:500px; float: left; margin: 0px 30px 15px 0px'></center>\n",
    "\n",
    "# TF-IDF and Word2Vec\n",
    "## Class 20 - Data Science Curriculum \n",
    "\n",
    "<br>\n",
    "\n",
    "#### Women Building Change scholarship program 2023 🇧🇮\n",
    "May 8, 2023\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10398bcd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ⏪ Recap last class\n",
    "\n",
    "- Vectorial representation of words \n",
    "    - One-Hot Encoding\n",
    "    - Bag of Words\n",
    "- Similarity of texts using\n",
    "    - Euclidean distance\n",
    "    - Cosine distance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6e027bf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 🚀 Today's agenda\n",
    "\n",
    "- TF-IDF\n",
    "- Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6860e16b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='img/pipeline.png' style='width:1500px; margin: 0px 30px 15px 0px'></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3e28c3c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Google Colab: https://colab.research.google.com/\n",
    "\n",
    "<br>\n",
    "<center><img src='img/girl_coding.jpg' style='height:350px; float: center; margin: 0px 30px 15px 0px'></center>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b67b137f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Let's do some text pre-processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4c82e196",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting fr-core-news-md==3.3.0\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/fr_core_news_md-3.3.0/fr_core_news_md-3.3.0-py3-none-any.whl (45.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.8/45.8 MB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0mm\n",
      "\u001b[?25hRequirement already satisfied: spacy<3.4.0,>=3.3.0.dev0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from fr-core-news-md==3.3.0) (3.3.0)\n",
      "Requirement already satisfied: thinc<8.1.0,>=8.0.14 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (8.0.15)\n",
      "Requirement already satisfied: jinja2 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (2.11.3)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (1.0.2)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (2.0.6)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (0.7.7)\n",
      "Requirement already satisfied: packaging>=20.0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (21.3)\n",
      "Requirement already satisfied: pathy>=0.3.5 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (0.6.1)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (0.9.1)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (1.0.7)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (2.0.7)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (2.26.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (4.62.3)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (3.0.9)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (1.8.2)\n",
      "Requirement already satisfied: setuptools in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (62.2.0)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (2.4.3)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (1.20.3)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (3.0.6)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (3.3.0)\n",
      "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (0.4.1)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from packaging>=20.0->spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (3.0.4)\n",
      "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from pathy>=0.3.5->spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (5.2.1)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (4.4.0)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (2.0.4)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (3.2)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (1.26.14)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (2021.10.8)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (7.1.2)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /Users/vmarquez/opt/anaconda3/lib/python3.9/site-packages (from jinja2->spacy<3.4.0,>=3.3.0.dev0->fr-core-news-md==3.3.0) (1.1.1)\n",
      "Installing collected packages: fr-core-news-md\n",
      "Successfully installed fr-core-news-md-3.3.0\n",
      "\u001b[33mWARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.\n",
      "You should consider upgrading via the '/Users/vmarquez/opt/anaconda3/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('fr_core_news_md')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/vmarquez/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "! python3 -m spacy download fr_core_news_md\n",
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "\n",
    "import json\n",
    "import pandas as pd\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "import spacy\n",
    "\n",
    "with open('princesses.json', 'r') as f:\n",
    "    data = json.load(f)\n",
    "\n",
    "df = pd.DataFrame(data.items(), columns=['princess', 'description']).set_index('princess')\n",
    "\n",
    "sw = stopwords.words(\"french\")\n",
    "lemma_obj = spacy.load('fr_core_news_md')\n",
    "\n",
    "def text_pre_processing(text):\n",
    "    text = text.lower() # Make it lowercase\n",
    "    text = re.sub(r\"[\\W\\d]\", \" \", text) # Remove punctuation (regex)\n",
    "    text = text.split() # Tokenize\n",
    "    text = [word for word in text if word not in sw] # Remove stop words\n",
    "    text = \" \".join(text) # Make it a string\n",
    "\n",
    "    # Lemmatization\n",
    "    doc = lemma_obj(text)\n",
    "    text = [token.lemma_ for token in doc]\n",
    "\n",
    "    text = \" \".join(text) # Make it a string\n",
    "    return text\n",
    "\n",
    "df['text_pp'] = df['description'].apply(lambda row: text_pre_processing(row))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "682e2da9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# TF-IDF (Term Frequency, Inverse Document Frequency) \n",
    "\n",
    "- It seeks to quantify the importance of a word relative to the other words in the document and in the corpus.\n",
    "- It is frequently used in information retrieval systems and clustering algorithms.\n",
    "- The more a word helps to distinguish a document from others, the higher its TF-IDF score will be."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66328a46",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### TF: Term Frequency\n",
    "\n",
    "- Term frequency: Counting the number of occurrences of a word in a document, divided by the number of words in that document.\n",
    "\n",
    "$$ tf(t,d) = \\dfrac{count(t)}{|d|}$$\n",
    "\n",
    "where\n",
    "- $t$ is term\n",
    "- $d$ is document"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27276a7d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "*Note: The term frequency is higher for words frequently used in a document.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6e69250",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### DF: Document Frequency\n",
    "\n",
    "- Frequency in documents: It is the number of documents that have that word over the total number of documents.\n",
    "\n",
    "$$ df(t,N) = \\dfrac{|\\{d_i: t\\in d_i, i=1, \\dots , N\\}|}{N}$$\n",
    "\n",
    "where\n",
    "- $t$ is term\n",
    "- $N$ is Number of documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c0d5045",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "*Note: Frequency in documents is higher for words used in many documents*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52e08927",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## So far we have TF and DF... how do we get to TD-IDF?\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e5b2be8",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- A specific word to some document will have very low term frequency"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f278045f",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Since the goal is to distinguish one document from another, we want to highlight words used frequently in one document but penalize them if they are present in all documents. This is called TF-IDF."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f941482",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Then, we obtain:\n",
    "\n",
    "$$tfidf (t,d,N) = tf(df) \\times \\log\\big(\\frac{1}{df(t,N)}\\big)$$\n",
    "\n",
    "where\n",
    "- $t$ is term\n",
    "- $d$ is document\n",
    "- $N$ is Number of documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f912d71",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- When $t$ is in all documents, $idf = log(1) = 0$\n",
    " \n",
    "- This makes sense since a word that is in all documents is very bad at distinguishing between documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9c7accd1",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>princess</th>\n",
       "      <th>Blanche-Neige</th>\n",
       "      <th>Cendrillon</th>\n",
       "      <th>Aurore</th>\n",
       "      <th>Ariel</th>\n",
       "      <th>Belle</th>\n",
       "      <th>Jasmine</th>\n",
       "      <th>Pocahontas</th>\n",
       "      <th>Mulan</th>\n",
       "      <th>Tiana</th>\n",
       "      <th>Raiponce</th>\n",
       "      <th>Mérida</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>absent</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.081</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>accomplir</th>\n",
       "      <td>0.127</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>admettre</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.081</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>adolescent</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>affirme</th>\n",
       "      <td>0.108</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.059</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>étroite</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.069</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>éviter</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>événement</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>être</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.053</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.046</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.091</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>œuvre</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.069</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>690 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "princess    Blanche-Neige  Cendrillon  Aurore  Ariel  Belle  Jasmine  \\\n",
       "absent              0.000       0.000   0.081  0.000  0.000      0.0   \n",
       "accomplir           0.127       0.000   0.000  0.000  0.000      0.0   \n",
       "admettre            0.000       0.000   0.081  0.000  0.000      0.0   \n",
       "adolescent          0.000       0.000   0.000  0.000  0.000      0.0   \n",
       "affirme             0.108       0.000   0.000  0.059  0.000      0.0   \n",
       "...                   ...         ...     ...    ...    ...      ...   \n",
       "étroite             0.000       0.000   0.000  0.069  0.000      0.0   \n",
       "éviter              0.000       0.000   0.000  0.000  0.000      0.0   \n",
       "événement           0.000       0.000   0.000  0.000  0.128      0.0   \n",
       "être                0.000       0.053   0.000  0.046  0.000      0.0   \n",
       "œuvre               0.000       0.000   0.000  0.069  0.000      0.0   \n",
       "\n",
       "princess    Pocahontas  Mulan  Tiana  Raiponce  Mérida  \n",
       "absent             0.0  0.000  0.000       0.0   0.000  \n",
       "accomplir          0.0  0.000  0.000       0.0   0.000  \n",
       "admettre           0.0  0.000  0.000       0.0   0.000  \n",
       "adolescent         0.0  0.000  0.000       0.0   0.093  \n",
       "affirme            0.0  0.000  0.000       0.0   0.000  \n",
       "...                ...    ...    ...       ...     ...  \n",
       "étroite            0.0  0.000  0.000       0.0   0.000  \n",
       "éviter             0.0  0.000  0.000       0.0   0.093  \n",
       "événement          0.0  0.000  0.000       0.0   0.000  \n",
       "être               0.0  0.091  0.104       0.0   0.000  \n",
       "œuvre              0.0  0.000  0.000       0.0   0.000  \n",
       "\n",
       "[690 rows x 11 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "tfidf_vect = TfidfVectorizer()\n",
    "tfidf = tfidf_vect.fit_transform(df['text_pp'].values)\n",
    "\n",
    "tfidf_matrix = pd.DataFrame(tfidf.toarray(), columns=tfidf_vect.get_feature_names())\n",
    "tfidf_matrix.index = df.index\n",
    "\n",
    "tfidf_matrix.T.round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e90b144",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Similarity measures\n",
    "How similar are the documents?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42036557",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Euclidean distance\n",
    "\n",
    "<br>\n",
    "<center><img src='img/dist_euc.png' style='height:350px;'>\n",
    "<small>Image credit: R-Bloggers</small></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "54070b8d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_77fd0_row0_col0, #T_77fd0_row1_col1, #T_77fd0_row2_col2, #T_77fd0_row3_col3, #T_77fd0_row4_col4, #T_77fd0_row5_col5, #T_77fd0_row6_col6, #T_77fd0_row7_col7, #T_77fd0_row8_col8, #T_77fd0_row9_col9, #T_77fd0_row10_col10 {\n",
       "  background-color: #fff5f0;\n",
       "  color: #000000;\n",
       "}\n",
       "#T_77fd0_row0_col1, #T_77fd0_row0_col3, #T_77fd0_row1_col5, #T_77fd0_row1_col6, #T_77fd0_row2_col1, #T_77fd0_row3_col8, #T_77fd0_row4_col2, #T_77fd0_row4_col6, #T_77fd0_row5_col7, #T_77fd0_row6_col2, #T_77fd0_row6_col4, #T_77fd0_row7_col0, #T_77fd0_row7_col1, #T_77fd0_row7_col5, #T_77fd0_row7_col9, #T_77fd0_row8_col3, #T_77fd0_row8_col9, #T_77fd0_row10_col9 {\n",
       "  background-color: #6b010e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_77fd0_row0_col2, #T_77fd0_row2_col7, #T_77fd0_row2_col9, #T_77fd0_row3_col5, #T_77fd0_row4_col9, #T_77fd0_row9_col7, #T_77fd0_row9_col10, #T_77fd0_row10_col3 {\n",
       "  background-color: #71020e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_77fd0_row0_col4, #T_77fd0_row0_col5, #T_77fd0_row0_col9, #T_77fd0_row1_col9, #T_77fd0_row2_col3, #T_77fd0_row2_col6, #T_77fd0_row3_col2, #T_77fd0_row3_col4, #T_77fd0_row4_col0, #T_77fd0_row5_col3, #T_77fd0_row9_col3 {\n",
       "  background-color: #6f020e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_77fd0_row0_col6, #T_77fd0_row0_col8, #T_77fd0_row1_col3, #T_77fd0_row1_col8, #T_77fd0_row4_col1, #T_77fd0_row4_col7, #T_77fd0_row4_col8, #T_77fd0_row4_col10, #T_77fd0_row5_col1, #T_77fd0_row5_col2, #T_77fd0_row5_col4, #T_77fd0_row5_col6, #T_77fd0_row5_col9, #T_77fd0_row5_col10, #T_77fd0_row6_col0, #T_77fd0_row6_col3, #T_77fd0_row6_col5, #T_77fd0_row7_col4, #T_77fd0_row7_col10, #T_77fd0_row8_col1, #T_77fd0_row10_col4, #T_77fd0_row10_col5, #T_77fd0_row10_col7 {\n",
       "  background-color: #67000d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_77fd0_row0_col7, #T_77fd0_row1_col0, #T_77fd0_row1_col7, #T_77fd0_row2_col4, #T_77fd0_row2_col10, #T_77fd0_row3_col0, #T_77fd0_row3_col7, #T_77fd0_row3_col9, #T_77fd0_row4_col3, #T_77fd0_row5_col0, #T_77fd0_row7_col2, #T_77fd0_row8_col10, #T_77fd0_row9_col5, #T_77fd0_row9_col8 {\n",
       "  background-color: #6d010e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_77fd0_row0_col10, #T_77fd0_row6_col7, #T_77fd0_row7_col6, #T_77fd0_row7_col8, #T_77fd0_row9_col6 {\n",
       "  background-color: #79040f;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_77fd0_row1_col2, #T_77fd0_row1_col4, #T_77fd0_row2_col5, #T_77fd0_row3_col1, #T_77fd0_row3_col6, #T_77fd0_row4_col5, #T_77fd0_row6_col1, #T_77fd0_row6_col10, #T_77fd0_row7_col3, #T_77fd0_row8_col0, #T_77fd0_row8_col4, #T_77fd0_row10_col2, #T_77fd0_row10_col6, #T_77fd0_row10_col8 {\n",
       "  background-color: #69000d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_77fd0_row1_col10, #T_77fd0_row8_col6, #T_77fd0_row10_col0 {\n",
       "  background-color: #77040f;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_77fd0_row2_col0, #T_77fd0_row6_col9, #T_77fd0_row9_col0, #T_77fd0_row9_col1, #T_77fd0_row9_col2, #T_77fd0_row10_col1 {\n",
       "  background-color: #73030f;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_77fd0_row2_col8, #T_77fd0_row8_col5 {\n",
       "  background-color: #7c0510;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_77fd0_row3_col10, #T_77fd0_row6_col8, #T_77fd0_row9_col4 {\n",
       "  background-color: #75030f;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_77fd0_row5_col8, #T_77fd0_row8_col2, #T_77fd0_row8_col7 {\n",
       "  background-color: #7a0510;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_77fd0_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >princess</th>\n",
       "      <th class=\"col_heading level0 col0\" >Blanche-Neige</th>\n",
       "      <th class=\"col_heading level0 col1\" >Cendrillon</th>\n",
       "      <th class=\"col_heading level0 col2\" >Aurore</th>\n",
       "      <th class=\"col_heading level0 col3\" >Ariel</th>\n",
       "      <th class=\"col_heading level0 col4\" >Belle</th>\n",
       "      <th class=\"col_heading level0 col5\" >Jasmine</th>\n",
       "      <th class=\"col_heading level0 col6\" >Pocahontas</th>\n",
       "      <th class=\"col_heading level0 col7\" >Mulan</th>\n",
       "      <th class=\"col_heading level0 col8\" >Tiana</th>\n",
       "      <th class=\"col_heading level0 col9\" >Raiponce</th>\n",
       "      <th class=\"col_heading level0 col10\" >Mérida</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >princess</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "      <th class=\"blank col3\" >&nbsp;</th>\n",
       "      <th class=\"blank col4\" >&nbsp;</th>\n",
       "      <th class=\"blank col5\" >&nbsp;</th>\n",
       "      <th class=\"blank col6\" >&nbsp;</th>\n",
       "      <th class=\"blank col7\" >&nbsp;</th>\n",
       "      <th class=\"blank col8\" >&nbsp;</th>\n",
       "      <th class=\"blank col9\" >&nbsp;</th>\n",
       "      <th class=\"blank col10\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_77fd0_level0_row0\" class=\"row_heading level0 row0\" >Blanche-Neige</th>\n",
       "      <td id=\"T_77fd0_row0_col0\" class=\"data row0 col0\" >0.000000</td>\n",
       "      <td id=\"T_77fd0_row0_col1\" class=\"data row0 col1\" >1.373237</td>\n",
       "      <td id=\"T_77fd0_row0_col2\" class=\"data row0 col2\" >1.355777</td>\n",
       "      <td id=\"T_77fd0_row0_col3\" class=\"data row0 col3\" >1.374014</td>\n",
       "      <td id=\"T_77fd0_row0_col4\" class=\"data row0 col4\" >1.368327</td>\n",
       "      <td id=\"T_77fd0_row0_col5\" class=\"data row0 col5\" >1.373438</td>\n",
       "      <td id=\"T_77fd0_row0_col6\" class=\"data row0 col6\" >1.393376</td>\n",
       "      <td id=\"T_77fd0_row0_col7\" class=\"data row0 col7\" >1.377620</td>\n",
       "      <td id=\"T_77fd0_row0_col8\" class=\"data row0 col8\" >1.386181</td>\n",
       "      <td id=\"T_77fd0_row0_col9\" class=\"data row0 col9\" >1.356605</td>\n",
       "      <td id=\"T_77fd0_row0_col10\" class=\"data row0 col10\" >1.345197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_77fd0_level0_row1\" class=\"row_heading level0 row1\" >Cendrillon</th>\n",
       "      <td id=\"T_77fd0_row1_col0\" class=\"data row1 col0\" >1.373237</td>\n",
       "      <td id=\"T_77fd0_row1_col1\" class=\"data row1 col1\" >0.000000</td>\n",
       "      <td id=\"T_77fd0_row1_col2\" class=\"data row1 col2\" >1.375634</td>\n",
       "      <td id=\"T_77fd0_row1_col3\" class=\"data row1 col3\" >1.381346</td>\n",
       "      <td id=\"T_77fd0_row1_col4\" class=\"data row1 col4\" >1.383473</td>\n",
       "      <td id=\"T_77fd0_row1_col5\" class=\"data row1 col5\" >1.382734</td>\n",
       "      <td id=\"T_77fd0_row1_col6\" class=\"data row1 col6\" >1.379908</td>\n",
       "      <td id=\"T_77fd0_row1_col7\" class=\"data row1 col7\" >1.375117</td>\n",
       "      <td id=\"T_77fd0_row1_col8\" class=\"data row1 col8\" >1.387408</td>\n",
       "      <td id=\"T_77fd0_row1_col9\" class=\"data row1 col9\" >1.354227</td>\n",
       "      <td id=\"T_77fd0_row1_col10\" class=\"data row1 col10\" >1.350327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_77fd0_level0_row2\" class=\"row_heading level0 row2\" >Aurore</th>\n",
       "      <td id=\"T_77fd0_row2_col0\" class=\"data row2 col0\" >1.355777</td>\n",
       "      <td id=\"T_77fd0_row2_col1\" class=\"data row2 col1\" >1.375634</td>\n",
       "      <td id=\"T_77fd0_row2_col2\" class=\"data row2 col2\" >0.000000</td>\n",
       "      <td id=\"T_77fd0_row2_col3\" class=\"data row2 col3\" >1.361973</td>\n",
       "      <td id=\"T_77fd0_row2_col4\" class=\"data row2 col4\" >1.372278</td>\n",
       "      <td id=\"T_77fd0_row2_col5\" class=\"data row2 col5\" >1.384766</td>\n",
       "      <td id=\"T_77fd0_row2_col6\" class=\"data row2 col6\" >1.368598</td>\n",
       "      <td id=\"T_77fd0_row2_col7\" class=\"data row2 col7\" >1.365271</td>\n",
       "      <td id=\"T_77fd0_row2_col8\" class=\"data row2 col8\" >1.326518</td>\n",
       "      <td id=\"T_77fd0_row2_col9\" class=\"data row2 col9\" >1.350259</td>\n",
       "      <td id=\"T_77fd0_row2_col10\" class=\"data row2 col10\" >1.378867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_77fd0_level0_row3\" class=\"row_heading level0 row3\" >Ariel</th>\n",
       "      <td id=\"T_77fd0_row3_col0\" class=\"data row3 col0\" >1.374014</td>\n",
       "      <td id=\"T_77fd0_row3_col1\" class=\"data row3 col1\" >1.381346</td>\n",
       "      <td id=\"T_77fd0_row3_col2\" class=\"data row3 col2\" >1.361973</td>\n",
       "      <td id=\"T_77fd0_row3_col3\" class=\"data row3 col3\" >0.000000</td>\n",
       "      <td id=\"T_77fd0_row3_col4\" class=\"data row3 col4\" >1.365308</td>\n",
       "      <td id=\"T_77fd0_row3_col5\" class=\"data row3 col5\" >1.363705</td>\n",
       "      <td id=\"T_77fd0_row3_col6\" class=\"data row3 col6\" >1.386612</td>\n",
       "      <td id=\"T_77fd0_row3_col7\" class=\"data row3 col7\" >1.378191</td>\n",
       "      <td id=\"T_77fd0_row3_col8\" class=\"data row3 col8\" >1.375674</td>\n",
       "      <td id=\"T_77fd0_row3_col9\" class=\"data row3 col9\" >1.361189</td>\n",
       "      <td id=\"T_77fd0_row3_col10\" class=\"data row3 col10\" >1.354413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_77fd0_level0_row4\" class=\"row_heading level0 row4\" >Belle</th>\n",
       "      <td id=\"T_77fd0_row4_col0\" class=\"data row4 col0\" >1.368327</td>\n",
       "      <td id=\"T_77fd0_row4_col1\" class=\"data row4 col1\" >1.383473</td>\n",
       "      <td id=\"T_77fd0_row4_col2\" class=\"data row4 col2\" >1.372278</td>\n",
       "      <td id=\"T_77fd0_row4_col3\" class=\"data row4 col3\" >1.365308</td>\n",
       "      <td id=\"T_77fd0_row4_col4\" class=\"data row4 col4\" >0.000000</td>\n",
       "      <td id=\"T_77fd0_row4_col5\" class=\"data row4 col5\" >1.389688</td>\n",
       "      <td id=\"T_77fd0_row4_col6\" class=\"data row4 col6\" >1.379667</td>\n",
       "      <td id=\"T_77fd0_row4_col7\" class=\"data row4 col7\" >1.390559</td>\n",
       "      <td id=\"T_77fd0_row4_col8\" class=\"data row4 col8\" >1.383216</td>\n",
       "      <td id=\"T_77fd0_row4_col9\" class=\"data row4 col9\" >1.349807</td>\n",
       "      <td id=\"T_77fd0_row4_col10\" class=\"data row4 col10\" >1.391718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_77fd0_level0_row5\" class=\"row_heading level0 row5\" >Jasmine</th>\n",
       "      <td id=\"T_77fd0_row5_col0\" class=\"data row5 col0\" >1.373438</td>\n",
       "      <td id=\"T_77fd0_row5_col1\" class=\"data row5 col1\" >1.382734</td>\n",
       "      <td id=\"T_77fd0_row5_col2\" class=\"data row5 col2\" >1.384766</td>\n",
       "      <td id=\"T_77fd0_row5_col3\" class=\"data row5 col3\" >1.363705</td>\n",
       "      <td id=\"T_77fd0_row5_col4\" class=\"data row5 col4\" >1.389688</td>\n",
       "      <td id=\"T_77fd0_row5_col5\" class=\"data row5 col5\" >0.000000</td>\n",
       "      <td id=\"T_77fd0_row5_col6\" class=\"data row5 col6\" >1.391717</td>\n",
       "      <td id=\"T_77fd0_row5_col7\" class=\"data row5 col7\" >1.381168</td>\n",
       "      <td id=\"T_77fd0_row5_col8\" class=\"data row5 col8\" >1.331025</td>\n",
       "      <td id=\"T_77fd0_row5_col9\" class=\"data row5 col9\" >1.378530</td>\n",
       "      <td id=\"T_77fd0_row5_col10\" class=\"data row5 col10\" >1.395605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_77fd0_level0_row6\" class=\"row_heading level0 row6\" >Pocahontas</th>\n",
       "      <td id=\"T_77fd0_row6_col0\" class=\"data row6 col0\" >1.393376</td>\n",
       "      <td id=\"T_77fd0_row6_col1\" class=\"data row6 col1\" >1.379908</td>\n",
       "      <td id=\"T_77fd0_row6_col2\" class=\"data row6 col2\" >1.368598</td>\n",
       "      <td id=\"T_77fd0_row6_col3\" class=\"data row6 col3\" >1.386612</td>\n",
       "      <td id=\"T_77fd0_row6_col4\" class=\"data row6 col4\" >1.379667</td>\n",
       "      <td id=\"T_77fd0_row6_col5\" class=\"data row6 col5\" >1.391717</td>\n",
       "      <td id=\"T_77fd0_row6_col6\" class=\"data row6 col6\" >0.000000</td>\n",
       "      <td id=\"T_77fd0_row6_col7\" class=\"data row6 col7\" >1.343865</td>\n",
       "      <td id=\"T_77fd0_row6_col8\" class=\"data row6 col8\" >1.349423</td>\n",
       "      <td id=\"T_77fd0_row6_col9\" class=\"data row6 col9\" >1.343287</td>\n",
       "      <td id=\"T_77fd0_row6_col10\" class=\"data row6 col10\" >1.386570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_77fd0_level0_row7\" class=\"row_heading level0 row7\" >Mulan</th>\n",
       "      <td id=\"T_77fd0_row7_col0\" class=\"data row7 col0\" >1.377620</td>\n",
       "      <td id=\"T_77fd0_row7_col1\" class=\"data row7 col1\" >1.375117</td>\n",
       "      <td id=\"T_77fd0_row7_col2\" class=\"data row7 col2\" >1.365271</td>\n",
       "      <td id=\"T_77fd0_row7_col3\" class=\"data row7 col3\" >1.378191</td>\n",
       "      <td id=\"T_77fd0_row7_col4\" class=\"data row7 col4\" >1.390559</td>\n",
       "      <td id=\"T_77fd0_row7_col5\" class=\"data row7 col5\" >1.381168</td>\n",
       "      <td id=\"T_77fd0_row7_col6\" class=\"data row7 col6\" >1.343865</td>\n",
       "      <td id=\"T_77fd0_row7_col7\" class=\"data row7 col7\" >0.000000</td>\n",
       "      <td id=\"T_77fd0_row7_col8\" class=\"data row7 col8\" >1.335878</td>\n",
       "      <td id=\"T_77fd0_row7_col9\" class=\"data row7 col9\" >1.366105</td>\n",
       "      <td id=\"T_77fd0_row7_col10\" class=\"data row7 col10\" >1.395790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_77fd0_level0_row8\" class=\"row_heading level0 row8\" >Tiana</th>\n",
       "      <td id=\"T_77fd0_row8_col0\" class=\"data row8 col0\" >1.386181</td>\n",
       "      <td id=\"T_77fd0_row8_col1\" class=\"data row8 col1\" >1.387408</td>\n",
       "      <td id=\"T_77fd0_row8_col2\" class=\"data row8 col2\" >1.326518</td>\n",
       "      <td id=\"T_77fd0_row8_col3\" class=\"data row8 col3\" >1.375674</td>\n",
       "      <td id=\"T_77fd0_row8_col4\" class=\"data row8 col4\" >1.383216</td>\n",
       "      <td id=\"T_77fd0_row8_col5\" class=\"data row8 col5\" >1.331025</td>\n",
       "      <td id=\"T_77fd0_row8_col6\" class=\"data row8 col6\" >1.349423</td>\n",
       "      <td id=\"T_77fd0_row8_col7\" class=\"data row8 col7\" >1.335878</td>\n",
       "      <td id=\"T_77fd0_row8_col8\" class=\"data row8 col8\" >0.000000</td>\n",
       "      <td id=\"T_77fd0_row8_col9\" class=\"data row8 col9\" >1.365730</td>\n",
       "      <td id=\"T_77fd0_row8_col10\" class=\"data row8 col10\" >1.378886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_77fd0_level0_row9\" class=\"row_heading level0 row9\" >Raiponce</th>\n",
       "      <td id=\"T_77fd0_row9_col0\" class=\"data row9 col0\" >1.356605</td>\n",
       "      <td id=\"T_77fd0_row9_col1\" class=\"data row9 col1\" >1.354227</td>\n",
       "      <td id=\"T_77fd0_row9_col2\" class=\"data row9 col2\" >1.350259</td>\n",
       "      <td id=\"T_77fd0_row9_col3\" class=\"data row9 col3\" >1.361189</td>\n",
       "      <td id=\"T_77fd0_row9_col4\" class=\"data row9 col4\" >1.349807</td>\n",
       "      <td id=\"T_77fd0_row9_col5\" class=\"data row9 col5\" >1.378530</td>\n",
       "      <td id=\"T_77fd0_row9_col6\" class=\"data row9 col6\" >1.343287</td>\n",
       "      <td id=\"T_77fd0_row9_col7\" class=\"data row9 col7\" >1.366105</td>\n",
       "      <td id=\"T_77fd0_row9_col8\" class=\"data row9 col8\" >1.365730</td>\n",
       "      <td id=\"T_77fd0_row9_col9\" class=\"data row9 col9\" >0.000000</td>\n",
       "      <td id=\"T_77fd0_row9_col10\" class=\"data row9 col10\" >1.367093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_77fd0_level0_row10\" class=\"row_heading level0 row10\" >Mérida</th>\n",
       "      <td id=\"T_77fd0_row10_col0\" class=\"data row10 col0\" >1.345197</td>\n",
       "      <td id=\"T_77fd0_row10_col1\" class=\"data row10 col1\" >1.350327</td>\n",
       "      <td id=\"T_77fd0_row10_col2\" class=\"data row10 col2\" >1.378867</td>\n",
       "      <td id=\"T_77fd0_row10_col3\" class=\"data row10 col3\" >1.354413</td>\n",
       "      <td id=\"T_77fd0_row10_col4\" class=\"data row10 col4\" >1.391718</td>\n",
       "      <td id=\"T_77fd0_row10_col5\" class=\"data row10 col5\" >1.395605</td>\n",
       "      <td id=\"T_77fd0_row10_col6\" class=\"data row10 col6\" >1.386570</td>\n",
       "      <td id=\"T_77fd0_row10_col7\" class=\"data row10 col7\" >1.395790</td>\n",
       "      <td id=\"T_77fd0_row10_col8\" class=\"data row10 col8\" >1.378886</td>\n",
       "      <td id=\"T_77fd0_row10_col9\" class=\"data row10 col9\" >1.367093</td>\n",
       "      <td id=\"T_77fd0_row10_col10\" class=\"data row10 col10\" >0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fcce05893d0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import euclidean_distances\n",
    "\n",
    "dist_euc = euclidean_distances(tfidf_matrix.values)\n",
    "dist_euc = pd.DataFrame(dist_euc, columns = df.index, index = df.index)\n",
    "\n",
    "dist_euc.style.background_gradient(cmap='Reds')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bd04ef2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Cosine distance\n",
    "\n",
    "<br>\n",
    "<center><img src='img/dist_cos.png' style='height:350px;'>\n",
    "<small>Image credit: R-Bloggers</small></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "17751a6e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_61814_row0_col0, #T_61814_row1_col1, #T_61814_row2_col2, #T_61814_row3_col3, #T_61814_row4_col4, #T_61814_row5_col5, #T_61814_row6_col6, #T_61814_row7_col7, #T_61814_row8_col8, #T_61814_row9_col9, #T_61814_row10_col10 {\n",
       "  background-color: #fff5f0;\n",
       "  color: #000000;\n",
       "}\n",
       "#T_61814_row0_col1, #T_61814_row4_col6, #T_61814_row5_col7, #T_61814_row6_col2, #T_61814_row7_col0, #T_61814_row7_col5 {\n",
       "  background-color: #71020e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row0_col2, #T_61814_row2_col9, #T_61814_row4_col9, #T_61814_row9_col7, #T_61814_row9_col10 {\n",
       "  background-color: #7a0510;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row0_col3, #T_61814_row1_col5, #T_61814_row1_col6, #T_61814_row2_col1, #T_61814_row3_col8, #T_61814_row4_col2, #T_61814_row6_col4, #T_61814_row7_col1, #T_61814_row7_col9, #T_61814_row8_col3, #T_61814_row8_col9, #T_61814_row10_col9 {\n",
       "  background-color: #6f020e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row0_col4, #T_61814_row0_col5, #T_61814_row0_col9, #T_61814_row1_col9, #T_61814_row3_col2, #T_61814_row5_col3 {\n",
       "  background-color: #77040f;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row0_col6, #T_61814_row0_col8, #T_61814_row1_col8, #T_61814_row5_col2, #T_61814_row5_col4, #T_61814_row5_col6, #T_61814_row5_col9, #T_61814_row5_col10, #T_61814_row6_col0, #T_61814_row6_col3, #T_61814_row7_col4, #T_61814_row7_col10, #T_61814_row8_col1, #T_61814_row10_col4, #T_61814_row10_col5, #T_61814_row10_col7 {\n",
       "  background-color: #67000d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row0_col7, #T_61814_row2_col10, #T_61814_row3_col7, #T_61814_row3_col9, #T_61814_row8_col10, #T_61814_row9_col5 {\n",
       "  background-color: #73030f;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row0_col10, #T_61814_row6_col7, #T_61814_row7_col8, #T_61814_row9_col6 {\n",
       "  background-color: #8a0812;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row1_col0, #T_61814_row1_col7, #T_61814_row2_col4, #T_61814_row3_col0, #T_61814_row4_col3, #T_61814_row5_col0, #T_61814_row7_col2, #T_61814_row9_col8 {\n",
       "  background-color: #75030f;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row1_col2, #T_61814_row1_col4, #T_61814_row2_col5, #T_61814_row6_col10, #T_61814_row7_col3, #T_61814_row8_col4, #T_61814_row10_col8 {\n",
       "  background-color: #6d010e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row1_col3, #T_61814_row4_col1, #T_61814_row4_col7, #T_61814_row4_col8, #T_61814_row4_col10, #T_61814_row5_col1, #T_61814_row6_col5 {\n",
       "  background-color: #69000d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row1_col10 {\n",
       "  background-color: #860811;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row2_col0, #T_61814_row6_col8, #T_61814_row9_col0, #T_61814_row10_col1 {\n",
       "  background-color: #800610;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row2_col3, #T_61814_row2_col6, #T_61814_row3_col4, #T_61814_row4_col0, #T_61814_row9_col3 {\n",
       "  background-color: #79040f;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row2_col7, #T_61814_row3_col5, #T_61814_row10_col3 {\n",
       "  background-color: #7c0510;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row2_col8, #T_61814_row8_col2, #T_61814_row8_col7 {\n",
       "  background-color: #900a12;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row3_col1, #T_61814_row3_col6, #T_61814_row4_col5, #T_61814_row6_col1, #T_61814_row8_col0, #T_61814_row10_col2, #T_61814_row10_col6 {\n",
       "  background-color: #6b010e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row3_col10 {\n",
       "  background-color: #820711;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row5_col8 {\n",
       "  background-color: #8e0912;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row6_col9, #T_61814_row9_col1, #T_61814_row9_col2 {\n",
       "  background-color: #7e0610;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row7_col6, #T_61814_row10_col0 {\n",
       "  background-color: #880811;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row8_col5 {\n",
       "  background-color: #940b13;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_61814_row8_col6, #T_61814_row9_col4 {\n",
       "  background-color: #840711;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_61814_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >princess</th>\n",
       "      <th class=\"col_heading level0 col0\" >Blanche-Neige</th>\n",
       "      <th class=\"col_heading level0 col1\" >Cendrillon</th>\n",
       "      <th class=\"col_heading level0 col2\" >Aurore</th>\n",
       "      <th class=\"col_heading level0 col3\" >Ariel</th>\n",
       "      <th class=\"col_heading level0 col4\" >Belle</th>\n",
       "      <th class=\"col_heading level0 col5\" >Jasmine</th>\n",
       "      <th class=\"col_heading level0 col6\" >Pocahontas</th>\n",
       "      <th class=\"col_heading level0 col7\" >Mulan</th>\n",
       "      <th class=\"col_heading level0 col8\" >Tiana</th>\n",
       "      <th class=\"col_heading level0 col9\" >Raiponce</th>\n",
       "      <th class=\"col_heading level0 col10\" >Mérida</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >princess</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "      <th class=\"blank col3\" >&nbsp;</th>\n",
       "      <th class=\"blank col4\" >&nbsp;</th>\n",
       "      <th class=\"blank col5\" >&nbsp;</th>\n",
       "      <th class=\"blank col6\" >&nbsp;</th>\n",
       "      <th class=\"blank col7\" >&nbsp;</th>\n",
       "      <th class=\"blank col8\" >&nbsp;</th>\n",
       "      <th class=\"blank col9\" >&nbsp;</th>\n",
       "      <th class=\"blank col10\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_61814_level0_row0\" class=\"row_heading level0 row0\" >Blanche-Neige</th>\n",
       "      <td id=\"T_61814_row0_col0\" class=\"data row0 col0\" >0.000000</td>\n",
       "      <td id=\"T_61814_row0_col1\" class=\"data row0 col1\" >0.942889</td>\n",
       "      <td id=\"T_61814_row0_col2\" class=\"data row0 col2\" >0.919066</td>\n",
       "      <td id=\"T_61814_row0_col3\" class=\"data row0 col3\" >0.943958</td>\n",
       "      <td id=\"T_61814_row0_col4\" class=\"data row0 col4\" >0.936159</td>\n",
       "      <td id=\"T_61814_row0_col5\" class=\"data row0 col5\" >0.943166</td>\n",
       "      <td id=\"T_61814_row0_col6\" class=\"data row0 col6\" >0.970748</td>\n",
       "      <td id=\"T_61814_row0_col7\" class=\"data row0 col7\" >0.948919</td>\n",
       "      <td id=\"T_61814_row0_col8\" class=\"data row0 col8\" >0.960749</td>\n",
       "      <td id=\"T_61814_row0_col9\" class=\"data row0 col9\" >0.920188</td>\n",
       "      <td id=\"T_61814_row0_col10\" class=\"data row0 col10\" >0.904777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_61814_level0_row1\" class=\"row_heading level0 row1\" >Cendrillon</th>\n",
       "      <td id=\"T_61814_row1_col0\" class=\"data row1 col0\" >0.942889</td>\n",
       "      <td id=\"T_61814_row1_col1\" class=\"data row1 col1\" >0.000000</td>\n",
       "      <td id=\"T_61814_row1_col2\" class=\"data row1 col2\" >0.946185</td>\n",
       "      <td id=\"T_61814_row1_col3\" class=\"data row1 col3\" >0.954058</td>\n",
       "      <td id=\"T_61814_row1_col4\" class=\"data row1 col4\" >0.956999</td>\n",
       "      <td id=\"T_61814_row1_col5\" class=\"data row1 col5\" >0.955976</td>\n",
       "      <td id=\"T_61814_row1_col6\" class=\"data row1 col6\" >0.952073</td>\n",
       "      <td id=\"T_61814_row1_col7\" class=\"data row1 col7\" >0.945473</td>\n",
       "      <td id=\"T_61814_row1_col8\" class=\"data row1 col8\" >0.962450</td>\n",
       "      <td id=\"T_61814_row1_col9\" class=\"data row1 col9\" >0.916965</td>\n",
       "      <td id=\"T_61814_row1_col10\" class=\"data row1 col10\" >0.911691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_61814_level0_row2\" class=\"row_heading level0 row2\" >Aurore</th>\n",
       "      <td id=\"T_61814_row2_col0\" class=\"data row2 col0\" >0.919066</td>\n",
       "      <td id=\"T_61814_row2_col1\" class=\"data row2 col1\" >0.946185</td>\n",
       "      <td id=\"T_61814_row2_col2\" class=\"data row2 col2\" >0.000000</td>\n",
       "      <td id=\"T_61814_row2_col3\" class=\"data row2 col3\" >0.927486</td>\n",
       "      <td id=\"T_61814_row2_col4\" class=\"data row2 col4\" >0.941573</td>\n",
       "      <td id=\"T_61814_row2_col5\" class=\"data row2 col5\" >0.958789</td>\n",
       "      <td id=\"T_61814_row2_col6\" class=\"data row2 col6\" >0.936531</td>\n",
       "      <td id=\"T_61814_row2_col7\" class=\"data row2 col7\" >0.931983</td>\n",
       "      <td id=\"T_61814_row2_col8\" class=\"data row2 col8\" >0.879825</td>\n",
       "      <td id=\"T_61814_row2_col9\" class=\"data row2 col9\" >0.911600</td>\n",
       "      <td id=\"T_61814_row2_col10\" class=\"data row2 col10\" >0.950637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_61814_level0_row3\" class=\"row_heading level0 row3\" >Ariel</th>\n",
       "      <td id=\"T_61814_row3_col0\" class=\"data row3 col0\" >0.943958</td>\n",
       "      <td id=\"T_61814_row3_col1\" class=\"data row3 col1\" >0.954058</td>\n",
       "      <td id=\"T_61814_row3_col2\" class=\"data row3 col2\" >0.927486</td>\n",
       "      <td id=\"T_61814_row3_col3\" class=\"data row3 col3\" >0.000000</td>\n",
       "      <td id=\"T_61814_row3_col4\" class=\"data row3 col4\" >0.932033</td>\n",
       "      <td id=\"T_61814_row3_col5\" class=\"data row3 col5\" >0.929845</td>\n",
       "      <td id=\"T_61814_row3_col6\" class=\"data row3 col6\" >0.961347</td>\n",
       "      <td id=\"T_61814_row3_col7\" class=\"data row3 col7\" >0.949705</td>\n",
       "      <td id=\"T_61814_row3_col8\" class=\"data row3 col8\" >0.946239</td>\n",
       "      <td id=\"T_61814_row3_col9\" class=\"data row3 col9\" >0.926418</td>\n",
       "      <td id=\"T_61814_row3_col10\" class=\"data row3 col10\" >0.917217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_61814_level0_row4\" class=\"row_heading level0 row4\" >Belle</th>\n",
       "      <td id=\"T_61814_row4_col0\" class=\"data row4 col0\" >0.936159</td>\n",
       "      <td id=\"T_61814_row4_col1\" class=\"data row4 col1\" >0.956999</td>\n",
       "      <td id=\"T_61814_row4_col2\" class=\"data row4 col2\" >0.941573</td>\n",
       "      <td id=\"T_61814_row4_col3\" class=\"data row4 col3\" >0.932033</td>\n",
       "      <td id=\"T_61814_row4_col4\" class=\"data row4 col4\" >0.000000</td>\n",
       "      <td id=\"T_61814_row4_col5\" class=\"data row4 col5\" >0.965616</td>\n",
       "      <td id=\"T_61814_row4_col6\" class=\"data row4 col6\" >0.951740</td>\n",
       "      <td id=\"T_61814_row4_col7\" class=\"data row4 col7\" >0.966827</td>\n",
       "      <td id=\"T_61814_row4_col8\" class=\"data row4 col8\" >0.956643</td>\n",
       "      <td id=\"T_61814_row4_col9\" class=\"data row4 col9\" >0.910989</td>\n",
       "      <td id=\"T_61814_row4_col10\" class=\"data row4 col10\" >0.968439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_61814_level0_row5\" class=\"row_heading level0 row5\" >Jasmine</th>\n",
       "      <td id=\"T_61814_row5_col0\" class=\"data row5 col0\" >0.943166</td>\n",
       "      <td id=\"T_61814_row5_col1\" class=\"data row5 col1\" >0.955976</td>\n",
       "      <td id=\"T_61814_row5_col2\" class=\"data row5 col2\" >0.958789</td>\n",
       "      <td id=\"T_61814_row5_col3\" class=\"data row5 col3\" >0.929845</td>\n",
       "      <td id=\"T_61814_row5_col4\" class=\"data row5 col4\" >0.965616</td>\n",
       "      <td id=\"T_61814_row5_col5\" class=\"data row5 col5\" >0.000000</td>\n",
       "      <td id=\"T_61814_row5_col6\" class=\"data row5 col6\" >0.968438</td>\n",
       "      <td id=\"T_61814_row5_col7\" class=\"data row5 col7\" >0.953813</td>\n",
       "      <td id=\"T_61814_row5_col8\" class=\"data row5 col8\" >0.885813</td>\n",
       "      <td id=\"T_61814_row5_col9\" class=\"data row5 col9\" >0.950172</td>\n",
       "      <td id=\"T_61814_row5_col10\" class=\"data row5 col10\" >0.973857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_61814_level0_row6\" class=\"row_heading level0 row6\" >Pocahontas</th>\n",
       "      <td id=\"T_61814_row6_col0\" class=\"data row6 col0\" >0.970748</td>\n",
       "      <td id=\"T_61814_row6_col1\" class=\"data row6 col1\" >0.952073</td>\n",
       "      <td id=\"T_61814_row6_col2\" class=\"data row6 col2\" >0.936531</td>\n",
       "      <td id=\"T_61814_row6_col3\" class=\"data row6 col3\" >0.961347</td>\n",
       "      <td id=\"T_61814_row6_col4\" class=\"data row6 col4\" >0.951740</td>\n",
       "      <td id=\"T_61814_row6_col5\" class=\"data row6 col5\" >0.968438</td>\n",
       "      <td id=\"T_61814_row6_col6\" class=\"data row6 col6\" >0.000000</td>\n",
       "      <td id=\"T_61814_row6_col7\" class=\"data row6 col7\" >0.902987</td>\n",
       "      <td id=\"T_61814_row6_col8\" class=\"data row6 col8\" >0.910472</td>\n",
       "      <td id=\"T_61814_row6_col9\" class=\"data row6 col9\" >0.902210</td>\n",
       "      <td id=\"T_61814_row6_col10\" class=\"data row6 col10\" >0.961288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_61814_level0_row7\" class=\"row_heading level0 row7\" >Mulan</th>\n",
       "      <td id=\"T_61814_row7_col0\" class=\"data row7 col0\" >0.948919</td>\n",
       "      <td id=\"T_61814_row7_col1\" class=\"data row7 col1\" >0.945473</td>\n",
       "      <td id=\"T_61814_row7_col2\" class=\"data row7 col2\" >0.931983</td>\n",
       "      <td id=\"T_61814_row7_col3\" class=\"data row7 col3\" >0.949705</td>\n",
       "      <td id=\"T_61814_row7_col4\" class=\"data row7 col4\" >0.966827</td>\n",
       "      <td id=\"T_61814_row7_col5\" class=\"data row7 col5\" >0.953813</td>\n",
       "      <td id=\"T_61814_row7_col6\" class=\"data row7 col6\" >0.902987</td>\n",
       "      <td id=\"T_61814_row7_col7\" class=\"data row7 col7\" >0.000000</td>\n",
       "      <td id=\"T_61814_row7_col8\" class=\"data row7 col8\" >0.892285</td>\n",
       "      <td id=\"T_61814_row7_col9\" class=\"data row7 col9\" >0.933121</td>\n",
       "      <td id=\"T_61814_row7_col10\" class=\"data row7 col10\" >0.974115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_61814_level0_row8\" class=\"row_heading level0 row8\" >Tiana</th>\n",
       "      <td id=\"T_61814_row8_col0\" class=\"data row8 col0\" >0.960749</td>\n",
       "      <td id=\"T_61814_row8_col1\" class=\"data row8 col1\" >0.962450</td>\n",
       "      <td id=\"T_61814_row8_col2\" class=\"data row8 col2\" >0.879825</td>\n",
       "      <td id=\"T_61814_row8_col3\" class=\"data row8 col3\" >0.946239</td>\n",
       "      <td id=\"T_61814_row8_col4\" class=\"data row8 col4\" >0.956643</td>\n",
       "      <td id=\"T_61814_row8_col5\" class=\"data row8 col5\" >0.885813</td>\n",
       "      <td id=\"T_61814_row8_col6\" class=\"data row8 col6\" >0.910472</td>\n",
       "      <td id=\"T_61814_row8_col7\" class=\"data row8 col7\" >0.892285</td>\n",
       "      <td id=\"T_61814_row8_col8\" class=\"data row8 col8\" >0.000000</td>\n",
       "      <td id=\"T_61814_row8_col9\" class=\"data row8 col9\" >0.932609</td>\n",
       "      <td id=\"T_61814_row8_col10\" class=\"data row8 col10\" >0.950664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_61814_level0_row9\" class=\"row_heading level0 row9\" >Raiponce</th>\n",
       "      <td id=\"T_61814_row9_col0\" class=\"data row9 col0\" >0.920188</td>\n",
       "      <td id=\"T_61814_row9_col1\" class=\"data row9 col1\" >0.916965</td>\n",
       "      <td id=\"T_61814_row9_col2\" class=\"data row9 col2\" >0.911600</td>\n",
       "      <td id=\"T_61814_row9_col3\" class=\"data row9 col3\" >0.926418</td>\n",
       "      <td id=\"T_61814_row9_col4\" class=\"data row9 col4\" >0.910989</td>\n",
       "      <td id=\"T_61814_row9_col5\" class=\"data row9 col5\" >0.950172</td>\n",
       "      <td id=\"T_61814_row9_col6\" class=\"data row9 col6\" >0.902210</td>\n",
       "      <td id=\"T_61814_row9_col7\" class=\"data row9 col7\" >0.933121</td>\n",
       "      <td id=\"T_61814_row9_col8\" class=\"data row9 col8\" >0.932609</td>\n",
       "      <td id=\"T_61814_row9_col9\" class=\"data row9 col9\" >0.000000</td>\n",
       "      <td id=\"T_61814_row9_col10\" class=\"data row9 col10\" >0.934472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_61814_level0_row10\" class=\"row_heading level0 row10\" >Mérida</th>\n",
       "      <td id=\"T_61814_row10_col0\" class=\"data row10 col0\" >0.904777</td>\n",
       "      <td id=\"T_61814_row10_col1\" class=\"data row10 col1\" >0.911691</td>\n",
       "      <td id=\"T_61814_row10_col2\" class=\"data row10 col2\" >0.950637</td>\n",
       "      <td id=\"T_61814_row10_col3\" class=\"data row10 col3\" >0.917217</td>\n",
       "      <td id=\"T_61814_row10_col4\" class=\"data row10 col4\" >0.968439</td>\n",
       "      <td id=\"T_61814_row10_col5\" class=\"data row10 col5\" >0.973857</td>\n",
       "      <td id=\"T_61814_row10_col6\" class=\"data row10 col6\" >0.961288</td>\n",
       "      <td id=\"T_61814_row10_col7\" class=\"data row10 col7\" >0.974115</td>\n",
       "      <td id=\"T_61814_row10_col8\" class=\"data row10 col8\" >0.950664</td>\n",
       "      <td id=\"T_61814_row10_col9\" class=\"data row10 col9\" >0.934472</td>\n",
       "      <td id=\"T_61814_row10_col10\" class=\"data row10 col10\" >0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fcd1c1d2c70>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import cosine_distances\n",
    "\n",
    "dist_cos = cosine_distances(tfidf_matrix.values)\n",
    "dist_cos = pd.DataFrame(dist_cos, columns = df.index, index = df.index)\n",
    "dist_cos.style.background_gradient(cmap='Reds')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9862bd9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 🔮 Going from TF-IDF to Word2Vec\n",
    "\n",
    "- So far, the vector representations of text that we have seen treat linguistic units as atomic units.\n",
    "- Vectors are sparse\n",
    "- They have problems with words outside the vocabulary\n",
    "\n",
    "With distributed representations, such as **word2vec**, we can create dense, low-dimensional representations that capture distributional similarities between words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03b954d4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# How does Word2Vec work?\n",
    "\n",
    "<br>\n",
    "<center><img src='img/espacio.jpg' style='height:350px;'>\n",
    "\n",
    "- Word2Vec derives the meaning of a word from its context. That is, if two different words occur in similar contexts, then they are likely to mean the same thing\n",
    "\n",
    "- Recently, [Mikolov et al., 2013] showed that neural networks do a good job representing the semantic vector space\n",
    "\n",
    "- Text representations using neural networks are usually called **embeddings**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d2ac025",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 🛠️ CBOW (Continous bag of words) & SkipGram\n",
    "\n",
    "<br>\n",
    "<center><img src='img/espacio.jpg' style='height:350px;'>\n",
    "\n",
    "- These are variations of Word2Vec\n",
    "\n",
    "- They are statistical models that seek to predict the probability of word distribution using contextual windows.\n",
    "\n",
    "- Both models reach a similar conclusion, but take almost inverse paths to get there\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d618ded6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 🛠️ CBOW (Continous bag of words) \n",
    "<br>\n",
    "<center><img src='img/cbow_fox.png' style='height:200px;'>\n",
    "    <small>What is the probability that \"jumped\" occurs given that...? $$P(jumped|...)$$</small></center>\n",
    "    \n",
    "<br>\n",
    "    \n",
    "<center><img src='img/cbow_prep.png' style='height:400px;'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dc9cc41",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='img/cbow_nn.png' style='height:600px;'>\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4b123a0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 🛠️ SkipGram\n",
    "<br>\n",
    "<center><img src='img/sg_fox.png' style='height:200px;'>\n",
    "    \n",
    "<br>\n",
    "    \n",
    "<center><img src='img/sg_prep.png' style='height:400px;'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac2b18c6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='img/sg_nn.png' style='height:600px;'>\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2216afc7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 🧠 Neural network?\n",
    "\n",
    "- We initialize each word $w$ in the corpus with a vector $v_w$ with random values\n",
    "\n",
    "- Then the Word2Vec model refines the vector $v_w$ by predicting $v_w$ using the vectors of the words in context using a neural network\n",
    "\n",
    "- Word2Vec ensures that these representations are low-dimensional and dense"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fe9ed35",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 🛠️ Pre-trained embeddings \n",
    "\n",
    "- Training your own embedding is a very expensive process (in terms of computation and time)\n",
    "\n",
    "- Fortunately, there are pre-trained embeddings available\n",
    "\n",
    "    - 🇬🇧 English: [Standford GloVe](https://nlp.stanford.edu/projects/glove/). Download [here](http://nlp.stanford.edu/data/glove.6B.zip)\n",
    "    - 🌎 More languages: [FastText](https://fasttext.cc/docs/en/crawl-vectors.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0669869b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import ast\n",
    "import pandas as pd\n",
    "import gensim.downloader as api\n",
    "from gensim.models import Word2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "23b1c519",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def analogy(model, worda, wordb, wordc):\n",
    "    '''\n",
    "    wordA is to wordB as wordC is to ...\n",
    "    '''\n",
    "    result = model.most_similar(negative=[worda], \n",
    "                                positive=[wordb, wordc])\n",
    "    return result[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ea907cb7",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[==================================================] 100.0% 1662.8/1662.8MB downloaded\n"
     ]
    }
   ],
   "source": [
    "google_wv = api.load('word2vec-google-news-300')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5d68ea52",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Bujumbura'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(google_wv, 'Colombia','Bogota','Burundi')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5a900bdc",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('games', 0.7636998295783997),\n",
       " ('play', 0.6501181125640869),\n",
       " ('match', 0.6485748887062073),\n",
       " ('matchup', 0.6120450496673584),\n",
       " ('agame', 0.5863147974014282),\n",
       " ('ballgame', 0.5731310248374939),\n",
       " ('thegame', 0.5718172192573547),\n",
       " ('opener', 0.5680001378059387),\n",
       " ('matches', 0.5580832958221436),\n",
       " ('tournament', 0.5496207475662231)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_wv.most_similar(\"game\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f066f7a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"rap.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "64b12d5c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ALink</th>\n",
       "      <th>SName</th>\n",
       "      <th>SLink</th>\n",
       "      <th>Lyric</th>\n",
       "      <th>language</th>\n",
       "      <th>pp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14946</th>\n",
       "      <td>/mos-def/</td>\n",
       "      <td>The Embassy</td>\n",
       "      <td>/mos-def/the-embassy.html</td>\n",
       "      <td>Mentioned that he worked for the embassy\\nPeop...</td>\n",
       "      <td>en</td>\n",
       "      <td>['mentioned', 'worked', 'embassy', 'people', '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16679</th>\n",
       "      <td>/master-p/</td>\n",
       "      <td>Rock It</td>\n",
       "      <td>/master-p/rock-it.html</td>\n",
       "      <td>(feat. 5th Ward Weebie, Krazy)\\n\\n[Master P an...</td>\n",
       "      <td>en</td>\n",
       "      <td>['feat', 'th', 'ward', 'weebie', 'krazy', 'mas...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4035</th>\n",
       "      <td>/lil-wayne/</td>\n",
       "      <td>That's What They Call Me</td>\n",
       "      <td>/lil-wayne/thats-what-they-call-me.html</td>\n",
       "      <td>[Lil Wayne - Verse 1]\\nMan, I aint got nothing...</td>\n",
       "      <td>en</td>\n",
       "      <td>['lil', 'wayne', 'verse', 'man', 'aint', 'got'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1506</th>\n",
       "      <td>/chris-brown/</td>\n",
       "      <td>Got To War For Ya</td>\n",
       "      <td>/chris-brown/got-to-war-for-ya.html</td>\n",
       "      <td>(trecho)\\n\\nWhat's the point in me havin' a cr...</td>\n",
       "      <td>en</td>\n",
       "      <td>['trecho', 'point', 'havin', 'crown', 'got', '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12302</th>\n",
       "      <td>/fugees/</td>\n",
       "      <td>Blunted On Reality</td>\n",
       "      <td>/fugees/blunted-on-reality.html</td>\n",
       "      <td>Intro)\\n(*inhales, then coughs*)\\nAy, nigga pa...</td>\n",
       "      <td>en</td>\n",
       "      <td>['intro', 'inhales', 'coughs', 'ay', 'nigga', ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               ALink                     SName  \\\n",
       "14946      /mos-def/               The Embassy   \n",
       "16679     /master-p/                   Rock It   \n",
       "4035     /lil-wayne/  That's What They Call Me   \n",
       "1506   /chris-brown/         Got To War For Ya   \n",
       "12302       /fugees/        Blunted On Reality   \n",
       "\n",
       "                                         SLink  \\\n",
       "14946                /mos-def/the-embassy.html   \n",
       "16679                   /master-p/rock-it.html   \n",
       "4035   /lil-wayne/thats-what-they-call-me.html   \n",
       "1506       /chris-brown/got-to-war-for-ya.html   \n",
       "12302          /fugees/blunted-on-reality.html   \n",
       "\n",
       "                                                   Lyric language  \\\n",
       "14946  Mentioned that he worked for the embassy\\nPeop...       en   \n",
       "16679  (feat. 5th Ward Weebie, Krazy)\\n\\n[Master P an...       en   \n",
       "4035   [Lil Wayne - Verse 1]\\nMan, I aint got nothing...       en   \n",
       "1506   (trecho)\\n\\nWhat's the point in me havin' a cr...       en   \n",
       "12302  Intro)\\n(*inhales, then coughs*)\\nAy, nigga pa...       en   \n",
       "\n",
       "                                                      pp  \n",
       "14946  ['mentioned', 'worked', 'embassy', 'people', '...  \n",
       "16679  ['feat', 'th', 'ward', 'weebie', 'krazy', 'mas...  \n",
       "4035   ['lil', 'wayne', 'verse', 'man', 'aint', 'got'...  \n",
       "1506   ['trecho', 'point', 'havin', 'crown', 'got', '...  \n",
       "12302  ['intro', 'inhales', 'coughs', 'ay', 'nigga', ...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "824ab327",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['pp'] = df['pp'].apply(lambda row: ast.literal_eval(row))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9af4e54c",
   "metadata": {},
   "outputs": [],
   "source": [
    "rap_model = Word2Vec(df['pp'].values,\n",
    "                     sg=1, # 1 skip-gram, 0 CBOW\n",
    "                     seed=1, vector_size=256, min_count=50, window=12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "43f694d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('fame', 0.5601926445960999),\n",
       " ('shame', 0.5253305435180664),\n",
       " ('rap', 0.49619030952453613),\n",
       " ('niggaz', 0.4614643156528473),\n",
       " ('aim', 0.45914676785469055),\n",
       " ('gain', 0.4551091194152832),\n",
       " ('gimmicks', 0.45426827669143677),\n",
       " ('games', 0.4541856348514557),\n",
       " ('maintain', 0.4448930621147156),\n",
       " ('name', 0.44138872623443604)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rap_model.wv.most_similar(\"game\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "cd5bde3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('games', 0.7636998295783997),\n",
       " ('play', 0.6501181125640869),\n",
       " ('match', 0.6485748887062073),\n",
       " ('matchup', 0.6120450496673584),\n",
       " ('agame', 0.5863147974014282),\n",
       " ('ballgame', 0.5731310248374939),\n",
       " ('thegame', 0.5718172192573547),\n",
       " ('opener', 0.5680001378059387),\n",
       " ('matches', 0.5580832958221436),\n",
       " ('tournament', 0.5496207475662231)]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_wv.most_similar(\"game\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b41a2b97",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>290</th>\n",
       "      <th>291</th>\n",
       "      <th>292</th>\n",
       "      <th>293</th>\n",
       "      <th>294</th>\n",
       "      <th>295</th>\n",
       "      <th>296</th>\n",
       "      <th>297</th>\n",
       "      <th>298</th>\n",
       "      <th>299</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>&lt;/s&gt;</th>\n",
       "      <td>0.067320</td>\n",
       "      <td>-0.053447</td>\n",
       "      <td>0.018991</td>\n",
       "      <td>0.091428</td>\n",
       "      <td>0.065955</td>\n",
       "      <td>-0.083695</td>\n",
       "      <td>-0.001819</td>\n",
       "      <td>-0.025018</td>\n",
       "      <td>-0.034342</td>\n",
       "      <td>0.064136</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.092337</td>\n",
       "      <td>0.081876</td>\n",
       "      <td>-0.003625</td>\n",
       "      <td>-0.049125</td>\n",
       "      <td>0.079146</td>\n",
       "      <td>0.069139</td>\n",
       "      <td>0.033887</td>\n",
       "      <td>-0.093247</td>\n",
       "      <td>-0.007335</td>\n",
       "      <td>-0.005146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>in</th>\n",
       "      <td>0.052956</td>\n",
       "      <td>0.065460</td>\n",
       "      <td>0.066195</td>\n",
       "      <td>0.047072</td>\n",
       "      <td>0.052221</td>\n",
       "      <td>-0.082009</td>\n",
       "      <td>-0.061415</td>\n",
       "      <td>-0.116210</td>\n",
       "      <td>0.015629</td>\n",
       "      <td>0.099293</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.127242</td>\n",
       "      <td>-0.066931</td>\n",
       "      <td>-0.060679</td>\n",
       "      <td>0.048911</td>\n",
       "      <td>0.046153</td>\n",
       "      <td>-0.035672</td>\n",
       "      <td>-0.044314</td>\n",
       "      <td>-0.035856</td>\n",
       "      <td>0.010895</td>\n",
       "      <td>-0.047072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>for</th>\n",
       "      <td>-0.008512</td>\n",
       "      <td>-0.034224</td>\n",
       "      <td>0.032284</td>\n",
       "      <td>0.045868</td>\n",
       "      <td>-0.013143</td>\n",
       "      <td>-0.046221</td>\n",
       "      <td>-0.000948</td>\n",
       "      <td>-0.052219</td>\n",
       "      <td>0.046574</td>\n",
       "      <td>0.062451</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.016318</td>\n",
       "      <td>0.002690</td>\n",
       "      <td>-0.059628</td>\n",
       "      <td>0.058923</td>\n",
       "      <td>0.005733</td>\n",
       "      <td>0.000345</td>\n",
       "      <td>0.013319</td>\n",
       "      <td>0.051513</td>\n",
       "      <td>-0.025227</td>\n",
       "      <td>0.017465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>that</th>\n",
       "      <td>-0.012361</td>\n",
       "      <td>-0.022230</td>\n",
       "      <td>0.065540</td>\n",
       "      <td>0.039477</td>\n",
       "      <td>-0.086620</td>\n",
       "      <td>0.024913</td>\n",
       "      <td>-0.011163</td>\n",
       "      <td>-0.070522</td>\n",
       "      <td>0.092369</td>\n",
       "      <td>0.092752</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.008863</td>\n",
       "      <td>-0.012265</td>\n",
       "      <td>-0.026254</td>\n",
       "      <td>-0.016193</td>\n",
       "      <td>-0.015235</td>\n",
       "      <td>0.050209</td>\n",
       "      <td>0.015810</td>\n",
       "      <td>0.005390</td>\n",
       "      <td>0.047909</td>\n",
       "      <td>-0.116515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>is</th>\n",
       "      <td>0.003746</td>\n",
       "      <td>-0.038920</td>\n",
       "      <td>0.091332</td>\n",
       "      <td>0.012000</td>\n",
       "      <td>-0.070575</td>\n",
       "      <td>0.105343</td>\n",
       "      <td>0.059936</td>\n",
       "      <td>-0.057342</td>\n",
       "      <td>0.038141</td>\n",
       "      <td>0.011092</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.124024</td>\n",
       "      <td>-0.019330</td>\n",
       "      <td>-0.049817</td>\n",
       "      <td>0.097040</td>\n",
       "      <td>0.014400</td>\n",
       "      <td>0.067980</td>\n",
       "      <td>-0.013168</td>\n",
       "      <td>0.005968</td>\n",
       "      <td>0.087180</td>\n",
       "      <td>0.056823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RAFFAELE</th>\n",
       "      <td>0.013637</td>\n",
       "      <td>-0.074288</td>\n",
       "      <td>-0.027634</td>\n",
       "      <td>0.043783</td>\n",
       "      <td>0.054908</td>\n",
       "      <td>0.013099</td>\n",
       "      <td>-0.084695</td>\n",
       "      <td>0.052396</td>\n",
       "      <td>-0.077159</td>\n",
       "      <td>0.114841</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.041271</td>\n",
       "      <td>-0.004194</td>\n",
       "      <td>-0.087207</td>\n",
       "      <td>-0.010856</td>\n",
       "      <td>-0.012650</td>\n",
       "      <td>-0.034632</td>\n",
       "      <td>0.024942</td>\n",
       "      <td>-0.018572</td>\n",
       "      <td>0.015611</td>\n",
       "      <td>0.039656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bim_Skala_Bim</th>\n",
       "      <td>0.016599</td>\n",
       "      <td>0.059948</td>\n",
       "      <td>-0.057047</td>\n",
       "      <td>-0.001974</td>\n",
       "      <td>0.034970</td>\n",
       "      <td>0.065427</td>\n",
       "      <td>-0.078319</td>\n",
       "      <td>-0.110227</td>\n",
       "      <td>-0.044478</td>\n",
       "      <td>0.131499</td>\n",
       "      <td>...</td>\n",
       "      <td>0.114740</td>\n",
       "      <td>-0.054791</td>\n",
       "      <td>-0.048668</td>\n",
       "      <td>-0.030780</td>\n",
       "      <td>0.016437</td>\n",
       "      <td>-0.148259</td>\n",
       "      <td>-0.014020</td>\n",
       "      <td>-0.068006</td>\n",
       "      <td>-0.018855</td>\n",
       "      <td>0.068006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mezze_Cafe</th>\n",
       "      <td>-0.020033</td>\n",
       "      <td>-0.092573</td>\n",
       "      <td>-0.019784</td>\n",
       "      <td>0.020033</td>\n",
       "      <td>-0.000234</td>\n",
       "      <td>0.041061</td>\n",
       "      <td>-0.051264</td>\n",
       "      <td>-0.012940</td>\n",
       "      <td>0.024636</td>\n",
       "      <td>0.013376</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.047282</td>\n",
       "      <td>0.001097</td>\n",
       "      <td>-0.139358</td>\n",
       "      <td>-0.004635</td>\n",
       "      <td>0.023268</td>\n",
       "      <td>0.043798</td>\n",
       "      <td>-0.047780</td>\n",
       "      <td>-0.016673</td>\n",
       "      <td>-0.013687</td>\n",
       "      <td>0.047531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pulverizes_boulders</th>\n",
       "      <td>0.027951</td>\n",
       "      <td>-0.027534</td>\n",
       "      <td>0.030871</td>\n",
       "      <td>0.001004</td>\n",
       "      <td>-0.031288</td>\n",
       "      <td>-0.072589</td>\n",
       "      <td>0.031497</td>\n",
       "      <td>0.027742</td>\n",
       "      <td>0.054233</td>\n",
       "      <td>0.098037</td>\n",
       "      <td>...</td>\n",
       "      <td>0.017104</td>\n",
       "      <td>-0.060074</td>\n",
       "      <td>-0.143509</td>\n",
       "      <td>0.002633</td>\n",
       "      <td>-0.051522</td>\n",
       "      <td>-0.006232</td>\n",
       "      <td>-0.008396</td>\n",
       "      <td>-0.007561</td>\n",
       "      <td>0.023049</td>\n",
       "      <td>0.016270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>snowcapped_Caucasus</th>\n",
       "      <td>0.033805</td>\n",
       "      <td>-0.033805</td>\n",
       "      <td>-0.002947</td>\n",
       "      <td>0.036546</td>\n",
       "      <td>0.066148</td>\n",
       "      <td>-0.092461</td>\n",
       "      <td>-0.025217</td>\n",
       "      <td>-0.106714</td>\n",
       "      <td>-0.090634</td>\n",
       "      <td>0.150570</td>\n",
       "      <td>...</td>\n",
       "      <td>0.011969</td>\n",
       "      <td>0.075285</td>\n",
       "      <td>-0.047144</td>\n",
       "      <td>-0.085152</td>\n",
       "      <td>0.052992</td>\n",
       "      <td>-0.009685</td>\n",
       "      <td>-0.031064</td>\n",
       "      <td>0.059570</td>\n",
       "      <td>0.054088</td>\n",
       "      <td>0.009730</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3000000 rows × 300 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          0         1         2         3         4    \\\n",
       "</s>                 0.067320 -0.053447  0.018991  0.091428  0.065955   \n",
       "in                   0.052956  0.065460  0.066195  0.047072  0.052221   \n",
       "for                 -0.008512 -0.034224  0.032284  0.045868 -0.013143   \n",
       "that                -0.012361 -0.022230  0.065540  0.039477 -0.086620   \n",
       "is                   0.003746 -0.038920  0.091332  0.012000 -0.070575   \n",
       "...                       ...       ...       ...       ...       ...   \n",
       "RAFFAELE             0.013637 -0.074288 -0.027634  0.043783  0.054908   \n",
       "Bim_Skala_Bim        0.016599  0.059948 -0.057047 -0.001974  0.034970   \n",
       "Mezze_Cafe          -0.020033 -0.092573 -0.019784  0.020033 -0.000234   \n",
       "pulverizes_boulders  0.027951 -0.027534  0.030871  0.001004 -0.031288   \n",
       "snowcapped_Caucasus  0.033805 -0.033805 -0.002947  0.036546  0.066148   \n",
       "\n",
       "                          5         6         7         8         9    ...  \\\n",
       "</s>                -0.083695 -0.001819 -0.025018 -0.034342  0.064136  ...   \n",
       "in                  -0.082009 -0.061415 -0.116210  0.015629  0.099293  ...   \n",
       "for                 -0.046221 -0.000948 -0.052219  0.046574  0.062451  ...   \n",
       "that                 0.024913 -0.011163 -0.070522  0.092369  0.092752  ...   \n",
       "is                   0.105343  0.059936 -0.057342  0.038141  0.011092  ...   \n",
       "...                       ...       ...       ...       ...       ...  ...   \n",
       "RAFFAELE             0.013099 -0.084695  0.052396 -0.077159  0.114841  ...   \n",
       "Bim_Skala_Bim        0.065427 -0.078319 -0.110227 -0.044478  0.131499  ...   \n",
       "Mezze_Cafe           0.041061 -0.051264 -0.012940  0.024636  0.013376  ...   \n",
       "pulverizes_boulders -0.072589  0.031497  0.027742  0.054233  0.098037  ...   \n",
       "snowcapped_Caucasus -0.092461 -0.025217 -0.106714 -0.090634  0.150570  ...   \n",
       "\n",
       "                          290       291       292       293       294  \\\n",
       "</s>                -0.092337  0.081876 -0.003625 -0.049125  0.079146   \n",
       "in                  -0.127242 -0.066931 -0.060679  0.048911  0.046153   \n",
       "for                 -0.016318  0.002690 -0.059628  0.058923  0.005733   \n",
       "that                -0.008863 -0.012265 -0.026254 -0.016193 -0.015235   \n",
       "is                  -0.124024 -0.019330 -0.049817  0.097040  0.014400   \n",
       "...                       ...       ...       ...       ...       ...   \n",
       "RAFFAELE            -0.041271 -0.004194 -0.087207 -0.010856 -0.012650   \n",
       "Bim_Skala_Bim        0.114740 -0.054791 -0.048668 -0.030780  0.016437   \n",
       "Mezze_Cafe          -0.047282  0.001097 -0.139358 -0.004635  0.023268   \n",
       "pulverizes_boulders  0.017104 -0.060074 -0.143509  0.002633 -0.051522   \n",
       "snowcapped_Caucasus  0.011969  0.075285 -0.047144 -0.085152  0.052992   \n",
       "\n",
       "                          295       296       297       298       299  \n",
       "</s>                 0.069139  0.033887 -0.093247 -0.007335 -0.005146  \n",
       "in                  -0.035672 -0.044314 -0.035856  0.010895 -0.047072  \n",
       "for                  0.000345  0.013319  0.051513 -0.025227  0.017465  \n",
       "that                 0.050209  0.015810  0.005390  0.047909 -0.116515  \n",
       "is                   0.067980 -0.013168  0.005968  0.087180  0.056823  \n",
       "...                       ...       ...       ...       ...       ...  \n",
       "RAFFAELE            -0.034632  0.024942 -0.018572  0.015611  0.039656  \n",
       "Bim_Skala_Bim       -0.148259 -0.014020 -0.068006 -0.018855  0.068006  \n",
       "Mezze_Cafe           0.043798 -0.047780 -0.016673 -0.013687  0.047531  \n",
       "pulverizes_boulders -0.006232 -0.008396 -0.007561  0.023049  0.016270  \n",
       "snowcapped_Caucasus -0.009685 -0.031064  0.059570  0.054088  0.009730  \n",
       "\n",
       "[3000000 rows x 300 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix = pd.DataFrame(google_wv.get_normed_vectors(), index = google_wv.key_to_index)\n",
    "matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "e2c57f40",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>246</th>\n",
       "      <th>247</th>\n",
       "      <th>248</th>\n",
       "      <th>249</th>\n",
       "      <th>250</th>\n",
       "      <th>251</th>\n",
       "      <th>252</th>\n",
       "      <th>253</th>\n",
       "      <th>254</th>\n",
       "      <th>255</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>like</th>\n",
       "      <td>0.085370</td>\n",
       "      <td>-0.075596</td>\n",
       "      <td>0.097258</td>\n",
       "      <td>-0.142571</td>\n",
       "      <td>0.024031</td>\n",
       "      <td>-0.033467</td>\n",
       "      <td>0.064075</td>\n",
       "      <td>-0.025131</td>\n",
       "      <td>0.025017</td>\n",
       "      <td>0.076124</td>\n",
       "      <td>...</td>\n",
       "      <td>0.035348</td>\n",
       "      <td>-0.052638</td>\n",
       "      <td>0.012715</td>\n",
       "      <td>-0.012325</td>\n",
       "      <td>-0.021964</td>\n",
       "      <td>0.050976</td>\n",
       "      <td>0.002912</td>\n",
       "      <td>-0.024124</td>\n",
       "      <td>-0.138690</td>\n",
       "      <td>0.053977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>get</th>\n",
       "      <td>-0.018450</td>\n",
       "      <td>-0.022875</td>\n",
       "      <td>0.033707</td>\n",
       "      <td>-0.100513</td>\n",
       "      <td>0.103531</td>\n",
       "      <td>-0.060459</td>\n",
       "      <td>0.025217</td>\n",
       "      <td>-0.094869</td>\n",
       "      <td>-0.002985</td>\n",
       "      <td>0.050575</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.038272</td>\n",
       "      <td>0.036821</td>\n",
       "      <td>-0.035520</td>\n",
       "      <td>-0.028045</td>\n",
       "      <td>-0.065190</td>\n",
       "      <td>0.020895</td>\n",
       "      <td>-0.110848</td>\n",
       "      <td>-0.073369</td>\n",
       "      <td>-0.161038</td>\n",
       "      <td>0.040252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>got</th>\n",
       "      <td>0.062818</td>\n",
       "      <td>-0.004717</td>\n",
       "      <td>0.056002</td>\n",
       "      <td>-0.055788</td>\n",
       "      <td>0.161225</td>\n",
       "      <td>-0.067321</td>\n",
       "      <td>-0.018808</td>\n",
       "      <td>-0.051887</td>\n",
       "      <td>-0.011999</td>\n",
       "      <td>0.092855</td>\n",
       "      <td>...</td>\n",
       "      <td>0.032761</td>\n",
       "      <td>0.107832</td>\n",
       "      <td>0.019446</td>\n",
       "      <td>-0.071148</td>\n",
       "      <td>-0.059053</td>\n",
       "      <td>0.065336</td>\n",
       "      <td>0.030922</td>\n",
       "      <td>-0.023967</td>\n",
       "      <td>-0.198040</td>\n",
       "      <td>0.054639</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>know</th>\n",
       "      <td>0.092219</td>\n",
       "      <td>-0.039774</td>\n",
       "      <td>0.021838</td>\n",
       "      <td>-0.165178</td>\n",
       "      <td>0.017003</td>\n",
       "      <td>-0.025243</td>\n",
       "      <td>0.018295</td>\n",
       "      <td>-0.079760</td>\n",
       "      <td>-0.010934</td>\n",
       "      <td>0.066495</td>\n",
       "      <td>...</td>\n",
       "      <td>0.055943</td>\n",
       "      <td>-0.097308</td>\n",
       "      <td>0.070908</td>\n",
       "      <td>-0.015377</td>\n",
       "      <td>0.020762</td>\n",
       "      <td>0.110590</td>\n",
       "      <td>0.031665</td>\n",
       "      <td>-0.000793</td>\n",
       "      <td>-0.085172</td>\n",
       "      <td>0.132437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nigga</th>\n",
       "      <td>0.031362</td>\n",
       "      <td>-0.091054</td>\n",
       "      <td>0.076987</td>\n",
       "      <td>0.003783</td>\n",
       "      <td>0.145514</td>\n",
       "      <td>0.019750</td>\n",
       "      <td>0.003984</td>\n",
       "      <td>-0.073198</td>\n",
       "      <td>-0.094429</td>\n",
       "      <td>0.069638</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.015929</td>\n",
       "      <td>-0.130118</td>\n",
       "      <td>0.017100</td>\n",
       "      <td>-0.001576</td>\n",
       "      <td>0.055559</td>\n",
       "      <td>0.015639</td>\n",
       "      <td>-0.092133</td>\n",
       "      <td>-0.019137</td>\n",
       "      <td>-0.133689</td>\n",
       "      <td>0.031998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>laker</th>\n",
       "      <td>-0.091635</td>\n",
       "      <td>-0.059359</td>\n",
       "      <td>0.049432</td>\n",
       "      <td>-0.027052</td>\n",
       "      <td>0.051276</td>\n",
       "      <td>0.040647</td>\n",
       "      <td>-0.022010</td>\n",
       "      <td>0.000144</td>\n",
       "      <td>0.049179</td>\n",
       "      <td>0.082729</td>\n",
       "      <td>...</td>\n",
       "      <td>0.039716</td>\n",
       "      <td>0.066397</td>\n",
       "      <td>0.075029</td>\n",
       "      <td>-0.024649</td>\n",
       "      <td>-0.020025</td>\n",
       "      <td>0.009812</td>\n",
       "      <td>-0.052290</td>\n",
       "      <td>0.013983</td>\n",
       "      <td>-0.082141</td>\n",
       "      <td>-0.044984</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aahh</th>\n",
       "      <td>-0.007977</td>\n",
       "      <td>-0.092939</td>\n",
       "      <td>-0.075158</td>\n",
       "      <td>-0.046331</td>\n",
       "      <td>-0.074863</td>\n",
       "      <td>0.020637</td>\n",
       "      <td>0.048279</td>\n",
       "      <td>-0.084653</td>\n",
       "      <td>0.012969</td>\n",
       "      <td>0.039376</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.023783</td>\n",
       "      <td>0.044097</td>\n",
       "      <td>0.002948</td>\n",
       "      <td>-0.097088</td>\n",
       "      <td>-0.004610</td>\n",
       "      <td>0.097879</td>\n",
       "      <td>0.036071</td>\n",
       "      <td>0.053854</td>\n",
       "      <td>0.159144</td>\n",
       "      <td>0.066427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>worn</th>\n",
       "      <td>0.032061</td>\n",
       "      <td>-0.041073</td>\n",
       "      <td>-0.028772</td>\n",
       "      <td>0.023050</td>\n",
       "      <td>0.008815</td>\n",
       "      <td>-0.021604</td>\n",
       "      <td>0.134567</td>\n",
       "      <td>-0.114270</td>\n",
       "      <td>0.101783</td>\n",
       "      <td>0.094920</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.004112</td>\n",
       "      <td>-0.004811</td>\n",
       "      <td>0.002921</td>\n",
       "      <td>-0.060393</td>\n",
       "      <td>-0.040535</td>\n",
       "      <td>-0.004875</td>\n",
       "      <td>-0.027006</td>\n",
       "      <td>0.049152</td>\n",
       "      <td>-0.103261</td>\n",
       "      <td>0.055036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spilling</th>\n",
       "      <td>-0.033893</td>\n",
       "      <td>-0.066507</td>\n",
       "      <td>0.078387</td>\n",
       "      <td>0.076874</td>\n",
       "      <td>0.106788</td>\n",
       "      <td>0.038120</td>\n",
       "      <td>0.076079</td>\n",
       "      <td>-0.046286</td>\n",
       "      <td>0.075716</td>\n",
       "      <td>0.010986</td>\n",
       "      <td>...</td>\n",
       "      <td>0.006373</td>\n",
       "      <td>-0.040128</td>\n",
       "      <td>-0.050631</td>\n",
       "      <td>0.059539</td>\n",
       "      <td>0.013564</td>\n",
       "      <td>0.024847</td>\n",
       "      <td>-0.076808</td>\n",
       "      <td>0.047732</td>\n",
       "      <td>-0.044988</td>\n",
       "      <td>-0.056792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>taller</th>\n",
       "      <td>0.049544</td>\n",
       "      <td>-0.047975</td>\n",
       "      <td>0.061402</td>\n",
       "      <td>-0.039939</td>\n",
       "      <td>-0.118103</td>\n",
       "      <td>-0.096739</td>\n",
       "      <td>0.020220</td>\n",
       "      <td>-0.063293</td>\n",
       "      <td>0.009372</td>\n",
       "      <td>-0.071070</td>\n",
       "      <td>...</td>\n",
       "      <td>0.096585</td>\n",
       "      <td>0.042076</td>\n",
       "      <td>0.084479</td>\n",
       "      <td>-0.024012</td>\n",
       "      <td>-0.118932</td>\n",
       "      <td>-0.018980</td>\n",
       "      <td>-0.040065</td>\n",
       "      <td>0.120313</td>\n",
       "      <td>-0.103421</td>\n",
       "      <td>0.098710</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7189 rows × 256 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               0         1         2         3         4         5    \\\n",
       "like      0.085370 -0.075596  0.097258 -0.142571  0.024031 -0.033467   \n",
       "get      -0.018450 -0.022875  0.033707 -0.100513  0.103531 -0.060459   \n",
       "got       0.062818 -0.004717  0.056002 -0.055788  0.161225 -0.067321   \n",
       "know      0.092219 -0.039774  0.021838 -0.165178  0.017003 -0.025243   \n",
       "nigga     0.031362 -0.091054  0.076987  0.003783  0.145514  0.019750   \n",
       "...            ...       ...       ...       ...       ...       ...   \n",
       "laker    -0.091635 -0.059359  0.049432 -0.027052  0.051276  0.040647   \n",
       "aahh     -0.007977 -0.092939 -0.075158 -0.046331 -0.074863  0.020637   \n",
       "worn      0.032061 -0.041073 -0.028772  0.023050  0.008815 -0.021604   \n",
       "spilling -0.033893 -0.066507  0.078387  0.076874  0.106788  0.038120   \n",
       "taller    0.049544 -0.047975  0.061402 -0.039939 -0.118103 -0.096739   \n",
       "\n",
       "               6         7         8         9    ...       246       247  \\\n",
       "like      0.064075 -0.025131  0.025017  0.076124  ...  0.035348 -0.052638   \n",
       "get       0.025217 -0.094869 -0.002985  0.050575  ... -0.038272  0.036821   \n",
       "got      -0.018808 -0.051887 -0.011999  0.092855  ...  0.032761  0.107832   \n",
       "know      0.018295 -0.079760 -0.010934  0.066495  ...  0.055943 -0.097308   \n",
       "nigga     0.003984 -0.073198 -0.094429  0.069638  ... -0.015929 -0.130118   \n",
       "...            ...       ...       ...       ...  ...       ...       ...   \n",
       "laker    -0.022010  0.000144  0.049179  0.082729  ...  0.039716  0.066397   \n",
       "aahh      0.048279 -0.084653  0.012969  0.039376  ... -0.023783  0.044097   \n",
       "worn      0.134567 -0.114270  0.101783  0.094920  ... -0.004112 -0.004811   \n",
       "spilling  0.076079 -0.046286  0.075716  0.010986  ...  0.006373 -0.040128   \n",
       "taller    0.020220 -0.063293  0.009372 -0.071070  ...  0.096585  0.042076   \n",
       "\n",
       "               248       249       250       251       252       253  \\\n",
       "like      0.012715 -0.012325 -0.021964  0.050976  0.002912 -0.024124   \n",
       "get      -0.035520 -0.028045 -0.065190  0.020895 -0.110848 -0.073369   \n",
       "got       0.019446 -0.071148 -0.059053  0.065336  0.030922 -0.023967   \n",
       "know      0.070908 -0.015377  0.020762  0.110590  0.031665 -0.000793   \n",
       "nigga     0.017100 -0.001576  0.055559  0.015639 -0.092133 -0.019137   \n",
       "...            ...       ...       ...       ...       ...       ...   \n",
       "laker     0.075029 -0.024649 -0.020025  0.009812 -0.052290  0.013983   \n",
       "aahh      0.002948 -0.097088 -0.004610  0.097879  0.036071  0.053854   \n",
       "worn      0.002921 -0.060393 -0.040535 -0.004875 -0.027006  0.049152   \n",
       "spilling -0.050631  0.059539  0.013564  0.024847 -0.076808  0.047732   \n",
       "taller    0.084479 -0.024012 -0.118932 -0.018980 -0.040065  0.120313   \n",
       "\n",
       "               254       255  \n",
       "like     -0.138690  0.053977  \n",
       "get      -0.161038  0.040252  \n",
       "got      -0.198040  0.054639  \n",
       "know     -0.085172  0.132437  \n",
       "nigga    -0.133689  0.031998  \n",
       "...            ...       ...  \n",
       "laker    -0.082141 -0.044984  \n",
       "aahh      0.159144  0.066427  \n",
       "worn     -0.103261  0.055036  \n",
       "spilling -0.044988 -0.056792  \n",
       "taller   -0.103421  0.098710  \n",
       "\n",
       "[7189 rows x 256 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix = pd.DataFrame(rap_model.wv.get_normed_vectors(), index = rap_model.wv.key_to_index)\n",
    "matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "043b4e32",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ⏪ Today's recap\n",
    "\n",
    "- TF-IDF\n",
    "- Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d8c332c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='img/bye.gif' style='height:250px;'></center> \n",
    "\n",
    "# Next class: Performance metrics\n",
    "# See you next Friday!"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}