preprocess_data2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "colab": {
      "name": "preprocess_data.ipynb",
      "provenance": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v5D6TsQlvSZH"
      },
      "source": [
        "# [Data Preprocessing](http://colab.research.google.com/github/boringPpl/presidential_debates_comments_clustering/blob/main/preprocess_data.ipynb)\n",
        "\n",
        "## 1. Setup\n",
        "\n",
        "### 1.1 Hardware\n",
        "\n",
        "First, check that the runtime in Google Colab is set to GPU. If it is not, go to **Runtime > Change runtime type** and change the **Hardware Accelerator** to **GPU**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IsSzCZyxy9As",
        "outputId": "0bc07627-7052-4879-d7df-f5743e48d293"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mon Jun 14 13:33:42 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 465.27       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   76C    P8    12W /  70W |      0MiB / 15109MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7f99sNEh2bnr"
      },
      "source": [
        "The following shows the CPU details for the Google Colab instance"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "klKuW-xB2FKl",
        "outputId": "14ef1611-6d36-46f9-d38d-dc83d77749b4"
      },
      "source": [
        "!lscpu"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Architecture:        x86_64\n",
            "CPU op-mode(s):      32-bit, 64-bit\n",
            "Byte Order:          Little Endian\n",
            "CPU(s):              2\n",
            "On-line CPU(s) list: 0,1\n",
            "Thread(s) per core:  2\n",
            "Core(s) per socket:  1\n",
            "Socket(s):           1\n",
            "NUMA node(s):        1\n",
            "Vendor ID:           GenuineIntel\n",
            "CPU family:          6\n",
            "Model:               79\n",
            "Model name:          Intel(R) Xeon(R) CPU @ 2.20GHz\n",
            "Stepping:            0\n",
            "CPU MHz:             2199.998\n",
            "BogoMIPS:            4399.99\n",
            "Hypervisor vendor:   KVM\n",
            "Virtualization type: full\n",
            "L1d cache:           32K\n",
            "L1i cache:           32K\n",
            "L2 cache:            256K\n",
            "L3 cache:            56320K\n",
            "NUMA node0 CPU(s):   0,1\n",
            "Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iN3t5qwA2rbb"
      },
      "source": [
        "The memory available for the instance is listed as follows"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "46fl6M1z2IVk",
        "outputId": "132820bc-dd17-4588-e658-06bae751d53a"
      },
      "source": [
        "!free -h"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              total        used        free      shared  buff/cache   available\n",
            "Mem:            12G        609M        5.1G        1.2M        7.0G         11G\n",
            "Swap:            0B          0B          0B\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6FT94kk61QKd"
      },
      "source": [
        "### 1.2 Install Dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "49kLid8avSZJ",
        "outputId": "22674c9d-ebea-4bb1-aba9-1b34f76779fc"
      },
      "source": [
        "!git clone https://github.com/boringPpl/presidential_debates_comments_clustering.git\n",
        "%cd presidential_debates_comments_clustering\n",
        "!pip install -qr requirements.txt"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "fatal: destination path 'presidential_debates_comments_clustering' already exists and is not an empty directory.\n",
            "/content/presidential_debates_comments_clustering\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8ZevAggEvSZK"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import emoji\n",
        "\n",
        "from sentence_transformers import SentenceTransformer\n",
        "from sklearn.cluster import KMeans\n",
        "\n",
        "embedder = SentenceTransformer('bert-base-nli-mean-tokens')"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eGSugYCGvSZK"
      },
      "source": [
        "## 2. Exploratory Data Analysis\n",
        "\n",
        "### 2.1 Load data into Pandas"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BWRl3_GzvSZK"
      },
      "source": [
        "def load_comments(video_id):\n",
        "    filename = f'data/{video_id}_csv_final.feather'\n",
        "    df = pd.read_feather(filename)\n",
        "    return df"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OLffUHsTvSZL"
      },
      "source": [
        "# This is the dataset for the first presidential debate.\n",
        "DEBATE_IDS = {1: 'wW1lY5jFNcQ', 2: 'bPiofmZGb8o'}\n",
        "DEBATE_NUMBER = 2\n",
        "df1 = load_comments(DEBATE_IDS[DEBATE_NUMBER])\n",
        "comments1 = df1['Comments']\n",
        "df1['Updated At'] = pd.to_datetime(df1['Updated At'], format='%Y-%m-%dT%H:%M:%SZ', errors='coerce')"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eNpUIfUnvSZL"
      },
      "source": [
        "### 2.2 Exploratory data analysis"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        },
        "id": "bnHLsvy7vSZL",
        "outputId": "e7c9d136-352f-4f73-b173-6afd74027cda"
      },
      "source": [
        "print(f'The shape of the dataframe is: {df1.shape}')\n",
        "print(f\"The time range for the data is: {df1['Updated At'].min():%Y-%m-%d %H%Mh} to {df1['Updated At'].max():%Y-%m-%d %H%Mh}\")\n",
        "df1.head()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The shape of the dataframe is: (49557, 6)\n",
            "The time range for the data is: 2020-10-23 0243h to 2021-03-29 1018h\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Comments</th>\n",
              "      <th>Comment ID</th>\n",
              "      <th>Reply Count</th>\n",
              "      <th>Like Count</th>\n",
              "      <th>Updated At</th>\n",
              "      <th>Viewer Rating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Our New President Joe Biden will be there for ...</td>\n",
              "      <td>Ugz3lAAG5Y1we-ySQ5J4AaABAg</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>2021-03-29 10:18:17</td>\n",
              "      <td>none</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>like a couple of kids. arguing like they're in...</td>\n",
              "      <td>Ugyf_v4oaA03atLW-pB4AaABAg</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2021-03-29 08:09:51</td>\n",
              "      <td>none</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>\"there are no accidents\" - Master Oogway\\nWatc...</td>\n",
              "      <td>UgxiFEtTpnsnVvixCuh4AaABAg</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2021-03-26 16:09:13</td>\n",
              "      <td>none</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Skip to 24:25 before there's nothing interesting</td>\n",
              "      <td>UgzSqqMk6VbxBLgaukd4AaABAg</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2021-03-26 10:11:39</td>\n",
              "      <td>none</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>I love not being a merican citizen\\nPs: visit ...</td>\n",
              "      <td>Ugz-cJWYPc4OVUgTePx4AaABAg</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>2021-03-26 10:07:58</td>\n",
              "      <td>none</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                            Comments  ... Viewer Rating\n",
              "0  Our New President Joe Biden will be there for ...  ...          none\n",
              "1  like a couple of kids. arguing like they're in...  ...          none\n",
              "2  \"there are no accidents\" - Master Oogway\\nWatc...  ...          none\n",
              "3   Skip to 24:25 before there's nothing interesting  ...          none\n",
              "4  I love not being a merican citizen\\nPs: visit ...  ...          none\n",
              "\n",
              "[5 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 320
        },
        "id": "2lhHs12yvSZL",
        "outputId": "78f8c779-3d01-4ae5-e3db-0ad62a3e5cb3"
      },
      "source": [
        "ax = df1['Updated At'].hist(bins=25, figsize=(12, 5))\n",
        "ax.set_yscale('log')"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsYAAAEvCAYAAABGywdiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAARrUlEQVR4nO3df+xd933X8eeHhKISs682UlklCbjI1SA0oK1WKw2EHCGE0+BlgsIaRRVBba1OigARCSyB2L/ZL4Q6FU1Wm3WgqlbVoSlOshWYsDLRMZqMrklXdfMiTzSUZqXju7mqVAIf/vCd+NbY8df29/re7/XjIV3l3nPOPedzv+97pVeO3+dzxpwzAAC41f2hVQ8AAADWgWAMAAAJxgAAUAnGAABQCcYAAFAJxgAAUNXtqx5A1Z133jkPHTq06mGs1De+8Y3uuOOOVQ+DPaKem019N5v67i/qtdmWUd8XXnjha3PON11u3UqD8RjjeHX88OHDPf/886scysqdPXu2o0ePrnoY7BH13Gzqu9nUd39Rr822jPqOMX77SutW2kox5zwz5zyxtbW1ymEAAIAeYwAAKMEYAAAqwRgAACrBGAAAqhUH4zHG8THGqe3t7VUOAwAAzEoBAACllQIAACrBGAAAKsEYAACqFd8SetUOnXxm6cc4/8SDSz8GAAA3zhljAADIdG0AAFCZrg0AACqtFAAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAULnBBwAAVG7wAQAAlVYKAACoBGMAAKgEYwAAqARjAACoBGMAAKgEYwAAqARjAACoBGMAAKgEYwAAqARjAACoBGMAAKhWHIzHGMfHGKe2t7dXOQwAAFhtMJ5znplzntja2lrlMAAAQCsFAACUYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAADVkoLxGOOOMcbzY4y/voz9AwDAXttVMB5jPDnGeHWM8dIly4+NMb40xjg3xji5Y9U/rj65lwMFAIBl2u0Z449Vx3YuGGPcVn24eqC6t3p4jHHvGOOvVr9evbqH4wQAgKW6fTcbzTmfG2McumTxO6pzc86Xq8YYp6uHqgPVHV0My98cYzw75/w/ezZiAABYgjHn3N2GF4Px03POty1ev7s6Nud8/+L1e6t3zjkfW7x+tPranPPpK+zvRHWi6uDBg28/ffr0DX2Q6/HiK9tLP8Z9d23tarsLFy504MCBJY+Gm0U9N5v6bjb13V/Ua7Mto77333//C3POI5dbt6szxtdjzvmxq6w/VZ2qOnLkyDx69OiyhnJFj558ZunHOP/I0V1td/bs2VbxN2A51HOzqe9mU9/9Rb02282u743MSvFKdc+O13cvlgEAwL5zI8H4s9VbxxhvGWO8oXpP9dS17GCMcXyMcWp7e/ktDQAA8Hp2O13bJ6pfrr57jPHlMcb75pyvVY9Vn66+WH1yzvmFazn4nPPMnPPE1tbu+nABAGBZdjsrxcNXWP5s9eyejggAAFbALaEBAKAVB2M9xgAArIuVBmM9xgAArAutFAAAkGAMAACVHmMAAKj0GAMAQKWVAgAAKsEYAAAqwRgAACoX3wEAQOXiOwAAqLRSAABAJRgDAEAlGAMAQOXiOwAAqFx8BwAAlVYKAACoBGMAAKgEYwAAqARjAACoBGMAAKhM1wYAAJXp2gAAoNJKAQAAlWAMAACVYAwAAFXdvuoBbLpDJ5/Z1XaP3/daj+5y253OP/HgNb8HAID/nzPGAACQYAwAAJVgDAAAlRt8AABA5QYfAABQaaUAAIBKMAYAgEowBgCASjAGAIBKMAYAgEowBgCASjAGAIBKMAYAgEowBgCAqm5f9QC4MYdOPrP0Y5x/4sGlHwMAYNWcMQYAgFYcjMcYx8cYp7a3t1c5DAAAWG0wnnOemXOe2NraWuUwAABAKwUAAJRgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAANUSgvEY48+OMX5qjPGpMcYP7fX+AQBgGXYVjMcYT44xXh1jvHTJ8mNjjC+NMc6NMU5WzTm/OOf8YPW3q7+490MGAIC9t9szxh+rju1cMMa4rfpw9UB1b/XwGOPexbrvr56pnt2zkQIAwBLtKhjPOZ+rvn7J4ndU5+acL885v1Wdrh5abP/UnPOB6pG9HCwAACzLmHPubsMxDlVPzznftnj97urYnPP9i9fvrd5Zfar6G9UfqT4/5/zwFfZ3ojpRdfDgwbefPn36hj7I9Xjxle2bfswrOfjG+uo3Vz2Ky7vvrq1VD2HfuXDhQgcOHFj1MFgS9d1s6ru/qNdmW0Z977///hfmnEcut+72PT1SNec8W53dxXanqlNVR44cmUePHt3roVzVoyefuenHvJLH73utn3hxz8uxJ84/cnTVQ9h3zp492yq+09wc6rvZ1Hd/Ua/NdrPreyOzUrxS3bPj9d2LZQAAsO/cSDD+bPXWMcZbxhhvqN5TPXUtOxhjHB9jnNreXp+WBgAAbk27na7tE9UvV989xvjyGON9c87XqseqT1dfrD455/zCtRx8znlmznlia0sPKwAAq7WrptY558NXWP5spmQDAGADuCU0AAC04mCsxxgAgHWx0mCsxxgAgHWhlQIAABKMAQCg0mMMAACVHmMAAKi0UgAAQCUYAwBAJRgDAEDl4jsAAKhcfAcAAJVWCgAAqARjAACoBGMAAKgEYwAAqMxKAQAAlVkpAACg0koBAACVYAwAAJVgDAAAlWAMAABV3b7qAbD+Dp18Zqn7P//Eg0vdPwDAbpiuDQAAMl0bAABUeowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKBygw8AAKjc4AMAACqtFAAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUK04GI8xjo8xTm1vb69yGAAAsNpgPOc8M+c8sbW1tcphAACAVgoAAKi6fdUDgEMnn1nq/s8/8eBS9w8AbAZnjAEAIMEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKreE5haw7FtOl9tOA8AmcMYYAAASjAEAoBKMAQCgWlKP8RjjB6oHq++oPjrn/LfLOA4AAOyVXZ8xHmM8OcZ4dYzx0iXLj40xvjTGODfGOFk15/y5OecHqg9WP7i3QwYAgL13La0UH6uO7Vwwxrit+nD1QHVv9fAY494dm/zTxXoAAFhruw7Gc87nqq9fsvgd1bk558tzzm9Vp6uHxkU/Uv38nPNX9264AACwHGPOufuNxzhUPT3nfNvi9burY3PO9y9ev7d6Z/Ub1d+pPlt9bs75U5fZ14nqRNXBgwfffvr06Rv6INfjxVe2b/oxr+TgG+ur31z1KLhe99219W2vL1y40IEDB1Y0GpZNfTeb+u4v6rXZllHf+++//4U555HLrVvKxXdzzg9VH7rKNqeqU1VHjhyZR48eXcZQXtejN+HGD7v1+H2v9RMvut/KfnX+kaPf9vrs2bOt4jvNzaG+l7cpN9NR3/1FvTbbza7vjSaxV6p7dry+e7EM2EPLDhzu3AcANz6P8Wert44x3jLGeEP1nuqp3b55jHF8jHFqe3t9WhoAALg17fqM8RjjE9XR6s4xxperH55zfnSM8Vj16eq26sk55xd2u88555nqzJEjRz5wbcOG9XLpGd3H73ttrVp1AICr23UwnnM+fIXlz1bP7tmIAK6DdhMAbpRbQgMAQEualWK3xhjHq+OHDx9e5TAA1oKz3gCrtdIzxnPOM3POE1tbW1ffGAAAlsjEucDGzD/L67sZdQbYz/QYAwBAKw7G5jEGAGBdrLSVwjzGcOvwz/gArDutFAAAkIvvANhHTGkHLJNgDLALOwOZW34DbCYX3wEAQG7wAQAAlYvvAACgEowBAKASjAEAoBKMAQCgWvF0bWOM49Xxw4cPr3IYAHBT3Iw7QJqLGa6fWSkAACCtFAAAUAnGAABQCcYAAFAJxgAAUK14VgoAWCeHTj7T4/e91qM3YfYIYP2s9IzxGOP4GOPU9vb2KocBAACrPWM85zxTnTly5MgHVjkOANgUy54r2TzJbDI9xgAAkGAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlRt8AABA5QYfAMAaudYblFzPLbzdpIQr0UoBAAAJxgAAUAnGAABQrbjHGADYX661Bxj2E2eMAQAgwRgAACqtFAAAe+pmtJuYcm45nDEGAIAEYwAAqARjAACoVhyMxxjHxxintre3VzkMAABYbTCec56Zc57Y2tpa5TAAAEArBQAAlGAMAACVYAwAAJVgDAAAlTvfAQC3mJtxZzr2J8EYAGCf2e/hfl1vaa2VAgAAEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoKox51z1GBpj/E7126sex4rdWX1t1YNgz6jnZlPfzaa++4t6bbZl1PdPzTnfdLkVaxGMqTHG83POI6seB3tDPTeb+m429d1f1Guz3ez6aqUAAIAEYwAAqATjdXJq1QNgT6nnZlPfzaa++4t6bbabWl89xgAAkDPGAABQCcbXbYxxzxjjP4wxfn2M8YUxxt9fLP+uMca/G2P85uK/37lY/sgY4/NjjBfHGJ8ZY/yFHfs6Nsb40hjj3Bjj5Osc8xfGGP9zjPH0JcsfW7x3jjHuXNZn3mRrVs+PL97/0hjjyTHGH17W575V7HF9nxxjvDrGeOkqx7zs98Dvde+tWX39fq9izer10THGry32/6kxxoFlfe5bxTrVd8f6D40xLuzqA8w5Pa7jUb25+t7F8z9W/UZ1b/Wj1cnF8pPVjyyef1/1nYvnD1S/snh+W/Vb1Z+u3lD9WnXvFY75V6rj1dOXLP+e6lB1vrpz1X+b/fhYs3q+qxqLxyeqH1r132e/P/aqvovXf7n63uql1zneFb8Hfq8bX1+/3/1Vr+/Ysd0//4Pje2xGfRfrj1T/urqwm/E7Y3yd5pxfmXP+6uL571dfrO6qHqp+ZrHZz1Q/sNjmM3PO310s/0/V3Yvn76jOzTlfnnN+qzq92MfljvmL1e9fZvl/mXOe34vPdatas3o+Oxeq/7xj31ynPaxvc87nqq9f5ZBX/B74ve69Nauv3+9VrFm9fq9qjDGqN1YuvLpB61TfMcZt1Y9V/2i34xeM98AY41AXzwL9SnVwzvmVxar/Xh28zFveV/384vld1X/dse7Li2WsyLrUc/FPsO+tfuF63s/l3WB9d8vvekXWpb5+v7uzDvUaY/z04nh/pvrJa9w3r2MN6vtY9dSO417V7dd4cC6x6Ef62eofzDl/7+L/dF4055xjjHnJ9vd3sfB/6aYOlF1Zs3r+y+q5OecvLWHft6Q1qy97bM3q6/d7FetSrznn312cWfzJ6gern97L/d+qVl3fMcafqP5WdfRa3ueM8Q1YnBH42erjc85/s1j81THGmxfr31y9umP7P199pHpozvk/Fotfqe7Zsdu7q1fGGO8cY3xu8fj+ZX8W1queY4wfrt5U/cMb/VxctEf1vdK+79lR3w92he/B3n0aLrVO9fX7vbp1qlfVnPN/d/Gf4P/mjX0yam3q+z3V4ercGON89UfHGOeuOvjdNCJ7XLbZe1T/qvoXlyz/sb69ufxHF8//ZHWu+r5Ltr+9erl6S/+vafzPvc5xj3bJxVo71p3PxTz7vp7V+6vPVG9c9d9lUx57Vd8d7zvU618MctXvgd/rZtbX73f/1GsxjsM7xvTj1Y+v+u+z3x/rUt/LbLeri+9W/gfcr48unuqf1eerzy0e76r+ePWL1W9W/776rsX2H6l+d8e2z+/Y17u6eNXmb1X/5HWO+UvV71Tf7GIPzV9bLP97i9evVf+t+siq/z777bFm9Xxt8d4/2Pc/W/XfZ78/9ri+n6i+Uv2vRd3ed4VjXvZ74Pe68fX1+90n9eriv5r/x+rF6qXq4+2YpcJjf9f3MtvsKhi78x0AAKTHGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAACq+r89jl5Y6fKOCgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 864x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QZyIA-tfvSZM",
        "outputId": "cff751ab-dc44-4042-d79f-755b8a0ccd7b"
      },
      "source": [
        "import emoji\n",
        "print(emoji.demojize('trending 😉\t'))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "trending :winking_face:\t\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lej-nAeXvSZM"
      },
      "source": [
        "df_corpus = df1"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UWq6agYPvSZM"
      },
      "source": [
        "df_corpus.rename(columns={'Comments': 'comment_text'}, inplace=True)"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nvynIneBvSZN"
      },
      "source": [
        "# De-emojize\n",
        "df_corpus['comments_cleaned'] = df_corpus['comment_text'].apply(emoji.demojize)\n",
        "\n",
        "# Replace the colons, and \\n with a space\n",
        "df_corpus['comments_cleaned'] = df_corpus['comments_cleaned'].str.replace('[\\n:]', ' ', regex=True)\n",
        "df_corpus['comments_cleaned'] = df_corpus['comments_cleaned'].str.replace(r'\\\\n', ' ', regex=True)"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9iwHuvFPvSZN"
      },
      "source": [
        "df_corpus['comments_cleaned'] = df_corpus['comments_cleaned'].str.lower()"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9YxEPBoTvSZN",
        "outputId": "c0c29405-0438-4d2b-9b11-b6b2e510be59"
      },
      "source": [
        "df_corpus.drop_duplicates(subset=['comments_cleaned'], inplace = True)\n",
        "df_corpus.shape"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(47164, 7)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZEWnr931vSZO"
      },
      "source": [
        "# remove special characters\n",
        "df_corpus['comments_cleaned'] = df_corpus['comments_cleaned'].str.replace('[^a-zA-Z0-9]', ' ')\n",
        "\n",
        "# remove white spaces\n",
        "df_corpus['comments_cleaned'] = df_corpus['comments_cleaned'].str.replace('\\s+', ' ', regex=True)\n",
        "df_corpus['comments_cleaned'] = df_corpus['comments_cleaned'].str.strip()"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w1DNb4RIvSZO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "97a81125-5cce-4178-b66c-cb8467d4c8b3"
      },
      "source": [
        "# Save metadata in .tsv and .parquet formats\n",
        "df_corpus['comments_cleaned'].to_csv(f'meta{DEBATE_NUMBER}.tsv', columns=['comments_cleaned'], index= False, header= False)\n",
        "df_corpus[['comments_cleaned']].to_parquet(f'meta{DEBATE_NUMBER}.parquet', index=False)\n",
        "!ls -lh meta*"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "-rw-r--r-- 1 root root 3.7M Jun 14 13:34 meta2.parquet\n",
            "-rw-r--r-- 1 root root 6.0M Jun 14 13:34 meta2.tsv\n",
            "-rw-r--r-- 1 root root 644K Jun 14 13:11 meta_lab_5k.tsv\n",
            "-rw-r--r-- 1 root root 4.0M Jun 14 13:11 meta_lab.parquet\n",
            "-rw-r--r-- 1 root root 6.1M Jun 14 13:11 meta_lab.tsv\n",
            "-rw-r--r-- 1 root root 3.6M Jun 14 12:47 meta.parquet\n",
            "-rw-r--r-- 1 root root 5.9M Jun 14 12:47 meta.tsv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iX6NhMd5vSZO"
      },
      "source": [
        "The longest comment has 16120 characters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6XEek6ClvSZP",
        "outputId": "c0eb8605-9511-46bd-af05-71abb8967f4f"
      },
      "source": [
        "df_corpus['comments_cleaned'].apply(len).sort_values(ascending=False).head(20)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "13884    9649\n",
              "9009     9572\n",
              "14882    9357\n",
              "39345    8032\n",
              "18823    7502\n",
              "13459    7429\n",
              "5080     7033\n",
              "48847    6823\n",
              "17625    6714\n",
              "26467    6246\n",
              "30141    6076\n",
              "9424     5886\n",
              "46644    5758\n",
              "5188     5732\n",
              "15819    5653\n",
              "5571     5653\n",
              "4104     5540\n",
              "29389    5498\n",
              "13292    5313\n",
              "7998     5309\n",
              "Name: comments_cleaned, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mcv7vIhgvSZP"
      },
      "source": [
        "## Sentence Bert Embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ZboLRN5vSZP"
      },
      "source": [
        "corpus_embeddings = embedder.encode(df_corpus[\"comments_cleaned\"].values.tolist())"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3UcZAIVdvSZP",
        "outputId": "0d36040e-5642-4b38-ca24-bb195b71693c"
      },
      "source": [
        "corpus_embeddings = np.array(corpus_embeddings)\n",
        "corpus_embeddings.shape"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(47164, 768)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MfBQw8jWvSZP"
      },
      "source": [
        "def plot_similarity(labels, features, rotation):\n",
        "  corr = np.inner(features, features)\n",
        "  sns.set(font_scale=1.2)\n",
        "  g = sns.heatmap(\n",
        "      corr,\n",
        "      xticklabels=labels,\n",
        "      yticklabels=labels,\n",
        "      vmin=0,\n",
        "      vmax=1,\n",
        "      cmap=\"YlOrRd\")\n",
        "  g.set_xticklabels(labels, rotation=rotation)\n",
        "  g.set_title(\"Semantic Textual Similarity\")\n",
        "\n",
        "def run_and_plot(messages_):\n",
        "  message_embeddings_ = embed(messages_)\n",
        "  plot_similarity(messages_, corpus_embeddings, 90)"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dDMPsKE5vSZQ"
      },
      "source": [
        "## Google Universal Encoder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kd-JU_exvSZQ",
        "outputId": "0fc3e24f-2828-4331-fb9a-78bea023cf73"
      },
      "source": [
        "from absl import logging\n",
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "import tensorflow_hub as hub\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import os\n",
        "import pandas as pd\n",
        "import re\n",
        "import seaborn as sns\n",
        "\n",
        "module_url = \"https://tfhub.dev/google/universal-sentence-encoder-large/5\" #@param [\"https://tfhub.dev/google/universal-sentence-encoder/4\", \"https://tfhub.dev/google/universal-sentence-encoder-large/5\"]\n",
        "model = hub.load(module_url)\n",
        "print (\"module %s loaded\" % module_url)\n",
        "def embed(input):\n",
        "  return model(input)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:absl:Using /tmp/tfhub_modules to cache modules.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "module https://tfhub.dev/google/universal-sentence-encoder-large/5 loaded\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "img28FjgvSZQ",
        "outputId": "721eb1c3-e6f8-4b81-9628-5bc707072352"
      },
      "source": [
        "#@title Compute a representation for each message, showing various lengths supported.\n",
        "word = \"Elephant\"\n",
        "sentence = \"I am a sentence for which I would like to get its embedding.\"\n",
        "paragraph = (\n",
        "    \"Universal Sentence Encoder embeddings also support short paragraphs. \"\n",
        "    \"There is no hard limit on how long the paragraph is. Roughly, the longer \"\n",
        "    \"the more 'diluted' the emtbedding will be.\")\n",
        "messages = [word, sentence, paragraph]\n",
        "\n",
        "# Reduce logging output.\n",
        "logging.set_verbosity(logging.ERROR)\n",
        "\n",
        "message_embeddings = embed(messages)\n",
        "\n",
        "for i, message_embedding in enumerate(np.array(message_embeddings).tolist()):\n",
        "  print(\"Message: {}\".format(messages[i]))\n",
        "  print(\"Embedding size: {}\".format(len(message_embedding)))\n",
        "  message_embedding_snippet = \", \".join(\n",
        "      (str(x) for x in message_embedding[:3]))\n",
        "  print(\"Embedding: [{}, ...]\\n\".format(message_embedding_snippet))"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Message: Elephant\n",
            "Embedding size: 512\n",
            "Embedding: [-0.0345856137573719, -0.01799013838171959, 0.00198058458045125, ...]\n",
            "\n",
            "Message: I am a sentence for which I would like to get its embedding.\n",
            "Embedding size: 512\n",
            "Embedding: [0.058333929628133774, -0.08185010403394699, 0.06890937685966492, ...]\n",
            "\n",
            "Message: Universal Sentence Encoder embeddings also support short paragraphs. There is no hard limit on how long the paragraph is. Roughly, the longer the more 'diluted' the emtbedding will be.\n",
            "Embedding size: 512\n",
            "Embedding: [-0.018883444368839264, -0.0031109170522540808, -0.033673401921987534, ...]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xFNDvKCSvSZQ"
      },
      "source": [
        "## Semantic Textual Similarity Task Example\n",
        "\n",
        "The embeddings produced by the Universal Sentence Encoder are approximately normalized. The semantic similarity of two sentences can be trivially computed as the inner product of the encodings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iyhhupo_vSZR"
      },
      "source": [
        "def plot_similarity(labels, features, rotation):\n",
        "  corr = np.inner(features, features)\n",
        "  sns.set(font_scale=1.2)\n",
        "  g = sns.heatmap(\n",
        "      corr,\n",
        "      xticklabels=labels,\n",
        "      yticklabels=labels,\n",
        "      vmin=0,\n",
        "      vmax=1,\n",
        "      cmap=\"YlOrRd\")\n",
        "  g.set_xticklabels(labels, rotation=rotation)\n",
        "  g.set_title(\"Semantic Textual Similarity\")\n",
        "\n",
        "def run_and_plot(messages_):\n",
        "  message_embeddings_ = embed(messages_)\n",
        "  plot_similarity(messages_, message_embeddings_, 90)"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZKYBmWgwvSZR"
      },
      "source": [
        "messages0 = df_corpus[\"comments_cleaned\"].values.tolist()"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OBn-kGVuvSZR",
        "outputId": "cc04a988-ba2c-4b70-e8d0-7982d664e3c5"
      },
      "source": [
        "df_corpus[\"comments_cleaned\"].apply(len).max()"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "9649"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QF4xETc3vSZS"
      },
      "source": [
        "# embeddings_long = embed([df_corpus['comments_cleaned'].loc[617593]])"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YOe8pabYvSZS"
      },
      "source": [
        "# [USComments.csv] Take only the first 2200 characters of each comment. Crashes at 2300+\n",
        "# [Presidential Debate 1] < 2200. Trying 2000 [Nope]. Trying 1800 [Nope].\n",
        "# Trying 1000 [Nope]. Trying 500 [Nope]. Trying 200.\n",
        "max_chars = 200\n",
        "messages = df_corpus[\"comments_cleaned\"].apply(\n",
        "    lambda x: x[:max_chars]\n",
        ").values.tolist()"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pL93dJEI51OD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a940da03-c95c-48a7-c41b-c680fd352913"
      },
      "source": [
        "print('Embeddings can only be done with about 500 rows at a time')\n",
        "n_chunks = len(messages) // 500\n",
        "message_list = np.array_split(np.array(messages), n_chunks)\n",
        "print(f'The {len(messages)} messages will be divided into {n_chunks} chunks of approximately 500 rows per chunk')"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Embeddings can only be done with about 500 rows at a time\n",
            "The 47164 messages will be divided into 94 chunks of approximately 500 rows per chunk\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oEr9Lh407TV_"
      },
      "source": [
        "for i, message in enumerate(message_list):\n",
        "    if i == 0:\n",
        "        embeddings = embed(message).numpy()\n",
        "    else:\n",
        "        embedding = embed(message).numpy()\n",
        "        embeddings = np.vstack((embeddings, embedding))"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DoqTDzWd8D2n",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7a868a55-0940-4b4e-80e7-a687c625fb58"
      },
      "source": [
        "print(f'The embeddings numpy array has shape: {embeddings.shape}')"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The embeddings numpy array has shape: (47164, 512)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9X2OiSSj26FO"
      },
      "source": [
        "Saving 47k rows of data will take a long time (the final size is 634MB)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p90i2R30lHbU"
      },
      "source": [
        "df_embeddings = pd.DataFrame(embeddings)\n",
        "df_embeddings.columns = [ str(x) for x in df_embeddings.columns ]\n",
        "df_embeddings.to_parquet(f'vecs0_{DEBATE_NUMBER}.parquet', index=False)\n",
        "np.savetxt(f'vecs0_{DEBATE_NUMBER}.tsv', embeddings , delimiter=\"\\t\")"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "70st8gsRAh0Q",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "18c62a89-f1a1-413b-f7d0-9182ae2c58f0"
      },
      "source": [
        "# Check the final size of the file\n",
        "!ls -lh vecs0*"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "-rw-r--r-- 1 root root 138M Jun 14 13:37 vecs0_2.parquet\n",
            "-rw-r--r-- 1 root root 588M Jun 14 13:37 vecs0_2.tsv\n",
            "-rw-r--r-- 1 root root 149M Jun 14 12:56 vecs0.parquet\n",
            "-rw-r--r-- 1 root root 634M Jun 14 12:57 vecs0.tsv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GlBGndw4vSZV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "27f82238-14f2-43fd-9328-0ed3337a26e9"
      },
      "source": [
        "# Find the maximum length of the messages\n",
        "pd.Series(messages).apply(len).max()"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "200"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IH6VTjAVvSZV"
      },
      "source": [
        "### PCA reduction\n",
        "\n",
        "The size of `vecs.tsv` can be reduced further.\n",
        "Using PCA, the number of dimensions of each embedding is reduced from 512 to 50."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gso-q5SPvSZV"
      },
      "source": [
        "from sklearn.decomposition import PCA"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b8e9cd7JvSZV"
      },
      "source": [
        "#scale the data 0-1\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "scaler = MinMaxScaler()\n",
        "rescaled = scaler.fit_transform(embeddings)"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JLfDLcITvSZV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "2b4081c4-5776-47af-9a60-188a4b3113ab"
      },
      "source": [
        "pca = PCA().fit(rescaled)\n",
        "plt.plot(np.cumsum(pca.explained_variance_ratio_))\n",
        "plt.xlabel('number of components')\n",
        "plt.ylabel('cumulative explained variance');"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5xU9b3/8ddnl2ULCyywS+9VUBB1IxAbdpJYctXEEjXFBI0xPeZn7r2PmHJL9KaZatSgxuRq1OQaTIy9xYICSq8LUpa2S1u2sHU+vz/OWRwI5VBmz+7M+/l4zGPnlJn5fJfhfPZ8q7k7IiKSubLiDkBEROKlRCAikuGUCEREMpwSgYhIhlMiEBHJcJ3iDuBwFRcX+9ChQ+MOQ0SkQ5k7d+5Wdy/Z37EOlwiGDh3KnDlz4g5DRKRDMbO1BzqmqiERkQynRCAikuGUCEREMpwSgYhIhlMiEBHJcClLBGY2w8wqzGzRAY6bmf3MzMrMbIGZnZyqWERE5MBSeUfwADDtIMc/BIwKH9OBX6cwFhEROYCUjSNw91fNbOhBTrkU+J0H82DPMrMiM+vn7ptSFZOISFtpSTj1TS3BozlBfVMLDU0J6ptb9jxvaknQnPDg0ZKguSV8nmh9nqCpxWkJj587tg8nDio65rHGOaBsALA+abs83PdPicDMphPcNTB48OA2CU5EMoe7U93QTFVdE1W7m6htaKa2sZmahhZq6pupbWimpqF5n/1N1Da0UNvYHF7wEzQ0Bz/rm1poThz7tV56d8tLu0QQmbvfA9wDUFpaqpV0ROSAEglnR10jW2sa2VrTwNaaBnbUNrJzdxM7wwv9zrpgu6quKfi5u4mWCBfugs7ZFOZ2ojC3E11yO9ElN5u+3fLI65xNXqds8nKyyA1/5uX883Zup2xyc7LIC392zs6iU7bRKSuLTln2/vNsIycri+xsC/ZnGdlZhpml5HcWZyLYAAxK2h4Y7hMR+SdNLQm27KpnU1XwqNhVT2VNA1urgwt+ZXVw0d9W23jAi3rXvE4UFeRQlN+ZooIcBhTl77XdPT+Hbvk5ey70hbnZFObm0CU3m4LOncjOSs2FOG5xJoKZwC1m9ggwCahS+4BIZnJ3KmsaWL99N5ur6tlUtZuNO8OfVfVsrtpNRXUD+66sm5NtlBTmUtw1l77d8xg/oDvFXTtTXJhLcWEuJV1zKS7sTI+CznTLzyEnWz3m9ydlicDMHgamAsVmVg7cDuQAuPvdwFPAh4EyoA74dKpiEZH4NTYn2LBzN2u31bJuex1rt9Wxbnsd68Kfu5ta9jq/oHM2/brn0b8on9G9S+hXlE//7nn0K8qnX/c8+nTNo1t+p5RVl2SSVPYauvoQxx34Qqo+X0TanrtTWd1AWWUNqypqWFVZS1lFDe9trWVT1W6Sa2zycrIY3LOAwT27cPqoYgb3LGBQz3z6dc+nf/d8XeTbUIdoLBaR9iWRcNZtr2PFluo9F/tVlcGjur55z3ldOmczonchHxjag8G9BjK4ZwFDehUwpGcBJV1zdaFvJ5QIROSgqnY3sXxzNcs272Lppl0s3VTN8s3Ve1Xl9O6ay8jehXx04gBG9i5kREkhI3sX0qebLvYdgRKBiABBtU75jt0s3FAVXvCDi/6Gnbv3nNM9P4ex/bpy5QcGMa5fN0b1KWRE70K65eXEGLkcLSUCkQxVsaueBeVVLCjfyYINVSwor2J7bSMA2VnG8OIunDKkB5+YPJixfbtxXL+u9O2Wp7/w05ASgUgGqG1oZt76nby7bgfzy6tYWF7F5l31AGQZjO7TlfPG9mb8wCImDOjOmL5dycvJjjlqaStKBCJpxt3ZsHM3c9fu2PNYumnXnh47w4u7MHl4T8YPLOLEgd0Z178bBZ11Kchk+tcX6eCaWhIs2biLOWt38M7aHcxZu50tuxqAoC/+SYOLuOWcUZwypAcTBxXRPV/1+bI3JQKRDqa5JcHCDVXMWr2dN1dvY86a7dQ1Bj14BhTlM3l4L04Z0oNThvRgTJ+udNJoWjkEJQKRdq4l4SzeWMWbq7Yxa/U2Zq/ZQU1D0Fd/VO9CrjhlIJOGBRf/vt3zYo5WOiIlApF2xt1Zs62OV1dU8o+Vlby1ejvV4YV/REkXLp3YnykjejFpWC9KuubGHK2kAyUCkXagur6JN1Zt49UVlby6spL124O++0N6FXDRif2ZPLwnU4b3onc3/cUvx54SgUgMEgln0caq4MK/YivvrNtBc8Lp0jmbKSOKmX7GcM4cXcKQXl3iDlUygBKBSBupa2zmtZVbeX7pFl5cVsHWmmDw1vgB3Zl+ZnDhP3lwDzp3UuOutC0lApEU2lxVzwvLtvD8ki28vmobjc0JuuZ14uwxvTnnuN6cPqqY4kLV80u8lAhEjiF3Z8mmXTy/pILnl25h4YYqAAb3LODaSUM4b1xvPjC0pxZIkXZFiUDkKLk7C8qreGrRJv6+cDPrttdhBicNKuKb08Zw/tg+jOxdqDl6pN1SIhA5Au7Ou+t38veFm3hq4WY27NxNpyzjtJHF3Dx1BOeN66MqH+kwlAhEInJ35q3fyZPzN/H0ok1srKonJ9s4Y1QJXzlvFBeM60v3Ak3fIB1PpERgZkOAUe7+vJnlA53cvTq1oYm0D6sra3hi3kb+Mm8Da7fV0Tk7izNHl/CNC8dw7tg+mrtHOrxDJgIz+xwwHegJjAAGAncD56Y2NJH4VFY38OT84OI/v7wKM/jgiF584eyRTDuhrxZikbQS5Y7gC8CpwFsA7r7SzHqnNCqRGNQ2NPPM4s08MW8jr62sJOFwfP9u/NuHx3Lxif01j4+krSiJoMHdG1t7PJhZJ8BTGpVIG3F35q7dwWNzyvnrgo3UNrYwsEc+N08dyUdP6s/I3l3jDlEk5aIkglfM7F+BfDM7H7gZeDK1YYmkVsWuev70zgYem7Oe1VtrKeiczUUT+vGx0kGUDumhrp6SUaIkgtuAG4CFwI3AU8B9qQxKJBWaWhK8sLSCx+as5+UVlbQknFOH9uSmqSP4yPh+dMlVJzrJTFG++fnADHe/F8DMssN9dakMTORY2VxVz8Nvr+Pht9dRUd1An2653HjmcK44ZSDDSwrjDk8kdlESwQvAeUBNuJ0PPAt8MFVBiRwtd+eNVdt46M21PLd0Cwl3zhpdwn9NGsLUMSVatUskSZREkOfurUkAd68xs4IUxiRyxKrqmnj8nXL+8NZaVlfW0qMgh8+eMYxPnDqEwb30tRXZnyiJoNbMTnb3dwDM7BRgd2rDEjk8ZRXV/Pa1Nfzfu+XUNyU4eXARP/74iXx4fD/ycrLjDk+kXYuSCL4CPGZmGwED+gJXpjQqkQhaq3/u+8dqXlpeSedOWVx20gCunTyEEwZ0jzs8kQ7jkInA3Web2XHAmHDXcndvSm1YIgfW0NzCk/M3cd8/VrNsczXFhZ356nmjuXbyYHppojeRwxa1v9wHgKHh+SebGe7+u5RFJbIfVXVNPDRrDQ++uZbK6gZG9ynkzssncMnE/qr+ETkKUeYaeohgjqF5QEu42wElAmkTFdX1/Pa19/j9m2upbWzhzNEl/OhjwzhjVLEGfokcA1HuCEqBce6uaSWkTa3fXsc9r67mj3PW09yS4CMT+nPz1BGM7dct7tBE0kqURLCIoIF4U4pjEQGCHkC/enkVf5m3kSyDy08eyI1njWBYcZe4QxNJS1ESQTGwxMzeBhpad7r7JYd6oZlNA+4CsoH73P0H+xwfDDwIFIXn3ObuT0UPX9JJWUUNP31+BX9buIncTllcP2UInztjOP2L8uMOTSStRUkE3zmSNw6novglcD5QDsw2s5nuviTptH8HHnX3X5vZOIJ5jIYeyedJx7V2Wy13Pb+SJ+ZtIC8nm5vOGsFnTx+mHkAibSRK99FXjvC9TwXK3H01gJk9AlwKJCcCB1orfLsDG4/ws6QDKt9Rxy9eLOOxueV0yjJuOH0YN541Qmv9irSxKL2GJgM/B8YCnQmqcGrd/VAtdgOA9Unb5cCkfc75DvCsmX0R6EIwp9H+YphOsEoagwcPPlTI0s5V7Krn5y+W8cjsdRjGdZOHcPPUEfTupoVfROIQpWroF8BVwGMEPYiuB0Yfo8+/GnjA3X9kZlOAh8zsBHdPJJ/k7vcA9wCUlpaq91IHVdPQzD2vrOLef7xHU0uCj39gELecPVJtACIxizSgzN3LzCzb3VuA+83sXeBbh3jZBmBQ0vbAcF+yG4Bp4We8aWZ5BI3TFVHiko6hqSXBI7PXc9fzK9ha08hFE/px64VjGNJLvYBE2oMoiaDOzDoD88zsToJupFHm8J0NjDKzYQQJ4Crgmn3OWQecCzxgZmOBPKAyavDSvrk7zy7Zwh1PL2N1ZS2nDuvJfZ8cy8RBRXGHJiJJoiSC6wjaBW4BvkrwV/7lh3qRuzeb2S3AM+HrZ7j7YjP7HjDH3WcCXwfuNbOvEjQcf0oD19LDvPU7+c+/LWH2mh2MKOnCfdeXcu7Y3hoJLNIOWUe77paWlvqcOXPiDkMOoKK6njufXs7jc8spLszlq+eP4srSQVoIRiRmZjbX3Uv3d+yAdwRm9qi7f9zMFhL8tb4Xd59wDGOUDq6xOcGDb6zhrhdW0tDcwk1njeCWc0ZSqHWARdq9g/0v/XL486K2CEQ6rldXVPLdJxezqrKWs8eU8O2Lj9d0ECIdyAETgbtvCkcHP+DuZ7dhTNJBbKrazXdnLuHpxZsZ2quAGZ8q5Zzj+sQdlogcpoPet7t7i5klzKy7u1e1VVDSvrUknIfeXMMPn11BcyLBrReO4bNnDCO3k9YEEOmIolTg1gALzew5oLZ1p7t/KWVRSbu1eGMV//rnhcwvr+LM0SX8x6UnaFF4kQ4uSiL4c/iQDFbb0MxPn1/BjNfX0KMgh59dfRIXT+in7qAiaSDKpHMPtkUg0n69uWob3/zTfNZv383Vpw7itmlj6V6QE3dYInKMRJl0bhTw38A4gpG/ALj78BTGJe1AXWMzdz69nAfeWMOQXgX8cfpkJg3vFXdYInKMRakauh+4HfgJcDbwaaJNMSEd2NvvbefWx+ezdlsdn/rgUL45bQwFnTUmQCQdRfmfne/uL5iZufta4DtmNhf4dopjkxjsbmzhf55Zzv1vvMegHgU8Mn0yk3UXIJLWoiSCBjPLAlaGcwdtAApTG5bEYcnGXXzx4XdYVVnL9VOG8P+mHUcXjQwWSXtR/pd/GSgAvgR8n6B66JOpDEralrtz/+tr+MHfl1FUkMPvb5jE6aOK4w5LRNpIlETQ4u41BOMJPp3ieKSNbatp4NbHF/DisgrOPa43d14xQWsFi2SYKIngR2bWF3gc+KO7L0pxTNJGXlu5la8+Oo+q3U1895LjuX7KEI0LEMlAUcYRnB0mgo8DvzGzbgQJ4T9SHp2kRCLh/PzFMn76wgpGlhTyu8+cyth+h1qCWkTSVaRuoO6+2d1/BtwEzEM9hjqsnXWN3PDgbH7y/Ar+ZeIAZt5yupKASIaLMqBsLHAlwapk24A/EqwsJh3Mog1V3PT7uWzZVc/3P3oC104arKogEYnURjADeAS40N03pjgeSZFH56zn359YRK8unXn0ximcNLhH3CGJSDsRpY1gSlsEIqnR2Jzg9pmLefjtdZw2shc/u+ok9QoSkb1otFAa217byE2/n8vb723n81NH8I0LxpCdpaogEdmbEkGaWrGlmhsenM2WXQ3cddVELp04IO6QRKSdUiJIQy8u28KXHp5HfudsHr1xChMHFcUdkoi0YwdMBGb2JOAHOu7ul6QkIjkqM157j+//bQnH9+/GvdeX0q97ftwhiUg7d7A7gh+GPy8D+gK/D7evBrakMig5fImEc8fTy/jNq6uZdnxffnzliZo2WkQiOeCVwt1fATCzH7l7adKhJ81sTsojk8iaWhL8v8cX8Od3N3D9lCHcfvHxahQWkcii/MnYxcyGu/tqADMbBnRJbVgSVW1DMzf/4R1eWVHJNy4YzRfOHqlBYiJyWKIkgq8CL5vZasCAIcCNKY1KItlW08BnHpjNwg1V/OCy8Vx16uC4QxKRDijKgLKnw3WLjwt3LXP3htSGJYeyqWo3n7j3LTbs3M1vrivl/HF94g5JRDqoKHMNFQBfA4a4++fMbJSZjXH3v6Y+PNmf9dvruOa+WeysbeL3n53EB4b2jDskEenAosw+ej/QCLRONbEB0BTUMVm7rZar7plFVZ2SgIgcG1ESwQh3vxNoAnD3OoK2Amljqypr+Phv3qSusZmHp0/mRA0UE5FjIEpjcaOZ5RMOLjOzEYDaCNpYWUUNV90zC3AemT6FMX27xh2SiKSJKIngduBpYJCZ/QE4DfhUKoOSva3bVscn7psFwCPTJzOyt5KAiBw7UXoNPWdm7wCTCaqEvuzuW1MemQCwcedurrlvFg3NCf44fYqSgIgcc5GWqgTygB3ALmCcmZ0Z5UVmNs3MlptZmZnddoBzPm5mS8xssZn9b8R4MkJldQPX3vcWVXVNPPSZSaoOEpGUiNJ99A6CpSoXA4lwtwOvHuJ12cAvgfOBcmC2mc109yVJ54wCvgWc5u47zKz3EZUiDe2obeTa+95iU1U9D91wKuMHdo87JBFJU1HaCD4KjDmCQWSnAmVJU1M8AlwKLEk653PAL919B4C7VxzmZ6Sl3Y0t3PDgbN7bVsv9n/oApeoiKiIpFKVqaDWQcwTvPQBYn7RdHu5LNhoYbWavm9ksM5t2BJ+TVppbEnzx4Xd5d/1OfnbVRE4bWRx3SCKS5qLcEdQB88zsBZK6jbr7l47R548CpgIDgVfNbLy770w+ycymA9MBBg9O3/l03J1vz1zM80u38L1Lj2faCf3iDklEMkCURDAzfByuDcCgpO2B4b5k5cBb7t4EvGdmKwgSw+zkk9z9HuAegNLS0gMultPR/fzFMv73rXXcPHUE108ZGnc4IpIhonQfffAI33s2MCqctnoDcBVwzT7nPEGw0M39ZlZMUFW0+gg/r0N7bM56fvzcCi47eQC3Xjgm7nBEJIMcbKnKR93942a2kP0sWenuEw72xu7ebGa3AM8A2cAMd19sZt8D5rj7zPDYBWa2BGgBbnX3bUdRng5pzprt/Ov/LeT0kcXccfkErScgIm3K3Pdf02Jm/dx9k5kN2d9xd1+b0sgOoLS01OfMSZ8F0jbu3M0lv3iNwtxO/OULp9O94Eja5UVEDs7M5u6z2uQeB1uqclP4M5YLfibY3djC9IfmUN+U4JHppUoCIhKLQ3YfNbPJZjbbzGrMrNHMWsxsV1sEl87cnVsfn8/ijbv42dUTNXWEiMQmyjiCXxA06K4E8oHPEowYlqMw4/U1/HXBJr554XGcc5xWFxOR+ESaa8jdy4Bsd29x9/uBjB/4dTTeWbeD/35qKeeP68NNZw2POxwRyXCRBpSZWWeCQWV3ApuIPlmd7GNHbSO3/OEd+hXl8cMrTlQPIRGJXZQL+nUE3T9vAWoJBoldnsqg0lUi4Xz9sflsrWnkV9ecosZhEWkXogwoa+01tBv4bmrDSW8PzVrLi8sq+N6lx2s2URFpNw42oGy/A8laHWpAmeytrKKa/3pqKecc15vrJu93aIaISCwOdkdwUZtFkeYamxN85Y/z6JLbiR9cPl7tAiLSrhxsQNmegWRm1pdgfQEHZrv75jaILW38/MWVLNqwi7uvPYXeXfPiDkdEZC9RBpR9FngbuAy4AphlZp9JdWDpYu7aHfzypTKuOGUg007oG3c4IiL/JEr30VuBk1ongzOzXsAbwIxUBpYOahua+dqj8+jXPZ/bLx4XdzgiIvsVJRFsA6qTtqvDfXIIP3x2Oeu21/HI5ybTNU9dRUWkfYqSCMqAt8zsLwRtBJcCC8zsawDu/uMUxtdhvbtuBw+8sYbrJg9h0vBecYcjInJAURLBqvDR6i/hT82SdgCNzQlu+9NC+nbL0yIzItLuRUkEd7h7ffIOMyt2960piqnDu/uVVSzfUs1vP1mqKiERafeiTDHxtplNbt0ws8sJGotlP8oqqvnFi2VcfGJ/zh2rWUVFpP2LckfwCWCGmb0M9Ad6AeekMqiOyt351p8XUpCbrV5CItJhRJlraKGZ/SfwEEGPoTPdvTzlkXVAM+dvZPaaHfzgsvEUF+bGHY6ISCSHTARm9ltgBDABGA381cx+7u5anCZJXWMz//3UMsYP6M7HSwfFHY6ISGRR2ggWAme7+3vu/gwwCTg5tWF1PL9+eRWbd9Vz+8XjyMrSXEIi0nEcMhG4+0+BwWZ2XrirEfhKSqPqYNZvr+M3r67m0on9KR3aM+5wREQOS5S5hj4HPA78Jtw1EHgilUF1NP/11FKyzbjtQ8fFHYqIyGGLUjX0BeA0YBeAu68EeqcyqI7kzVXb+Puizdw8dQT9uufHHY6IyGGLkgga3L2xdcPMOnGQBWsyibvzg6eX0b97Hp87U4vQi0jHFCURvGJm/wrkm9n5wGPAk6kNq2N4bskW5q/fyVfOG01eTnbc4YiIHJEoieA2oJKg99CNwFPAv6cyqI6gJeH86NkVDC/uwmUnD4g7HBGRIxZlQFkCuDd8SGjm/A0s31LNL645iU7ZUfKpiEj7pCvYEWhsTvCT51Yyrl83PnxCv7jDERE5KkoER+DxueWs217HrdPGaPCYiHR4kROBmRWkMpCOorklwa9fKePEQUVMHV0SdzgiIkctyoCyD5rZEmBZuH2imf0q5ZG1U08u2Mj67bu55eyRmOluQEQ6vih3BD8BLiRcp9jd5wNnpjKo9iqRcH710irG9OnKucdpTJ2IpIdIVUPuvn6fXS0piKXde3bJZlZW1HDz2SPUNiAiaSPKwjTrzeyDgJtZDvBlYGlqw2p/3J1fvrSKob0KuGhC/7jDERE5ZqLcEdxEMN/QAGADMDHcPiQzm2Zmy82szMxuO8h5l5uZm1lplPeNwz9WbmXhhipuOmsE2bobEJE0EuWOwNz9E4f7xmaWDfwSOB8oB2ab2Ux3X7LPeV0J7jLeOtzPaEszXn+Pkq65/ItGEYtImolyR/C6mT1rZjeYWdFhvPepQJm7rw4nrXsEuHQ/530fuAOoP4z3blOrK2t4eXkl104aQm4nzSkkIuklysI0ownmFjoeeMfM/mpm10Z47wFAciNzebhvDzM7GRjk7n872BuZ2XQzm2NmcyorKyN89LH14Btr6JydxTWTBrf5Z4uIpFrUXkNvu/vXCP7K3w48eLQfbGZZwI+Br0f4/HvcvdTdS0tK2nYQ1676Jh6fW85FJ/ajpKsWpBeR9BNlQFk3M/ukmf0deAPYRJAQDmUDkLyK+8BwX6uuwAnAy2a2BpgMzGxvDcaPzymntrGFT39wWNyhiIikRJTG4vkES1N+z93fPIz3ng2MMrNhBAngKuCa1oPuXgUUt26b2cvAN9x9zmF8Rkq5O79/ay0nDy5i/MDucYcjIpISURLBcHc/7BXJ3L3ZzG4BngGygRnuvtjMvgfMcfeZh/uebW1+eRWrK2u54/LxcYciIpIyB0wEZvZTd/8KQXXNPyUCd7/kUG/u7k8RLGSTvO/bBzh36iGjbWN/mltObqcsPjReU02LSPo62B3BQ+HPH7ZFIO1NQ3MLTy7YyIXH96VbXk7c4YiIpMwBE4G7zw2fTnT3u5KPmdmXgVdSGVjcXlpWwc66Ji1DKSJpL0r30U/uZ9+njnEc7c7jczfQu2sup48sPvTJIiId2MHaCK4m6OUzzMySG3a7EowlSFvbaxt5eXkFnzl9mNYjFpG0d7A2gtYxA8XAj5L2VwMLUhlU3J5etJnmhHPpRM0yKiLp72BtBGuBtcCUtgunffjbwo0MK+7CuH7d4g5FRCTloowsnmxms82sxswazazFzHa1RXBx2FrTwJurtvGR8f20FKWIZIQoFeC/AK4GVgL5wGcJppdOS88s3kzC4SMTNHZARDJD1EnnyoBsd29x9/uBaakNKz5/W7CJ4SVdOK5v17hDERFpE1GmmKgzs87APDO7k6ABOS270uyobWTW6m3cPHWkqoVEJGNEuaBfRzBX0C1ALcGMopenMqi4vLKikoTDeeP6xB2KiEibOeQdQdh7CGA38N3UhhOv55duobgwlwkDNNOoiGSOgw0oWwgccNZRd5+Qkohi0tSS4JUVlXzohL5kaXF6EckgB7sjuKjNomgHZr+3ner6Zs4dq2ohEckshxpQljGeX1pB505ZnDFKcwuJSGY5ZBuBmVXzfhVRZyAHqHX3tBp2++rKSiYP70VB5ygdqURE0keUxuI9Heot6FN5KcH6wmljc1U9ZRU1XFk66NAni4ikmcMaD+CBJ4ALUxRPLF4v2wrAaZpyWkQyUJSqocuSNrOAUqA+ZRHF4PWyrfTq0lmjiUUkI0WpEL846XkzsIageigtuDuvlW3lgyOL1W1URDJSlDaCT7dFIHFZVVlDRXUDp4/sFXcoIiKxiFI1NAz4IjA0+Xx3vyR1YbWd11YG7QMfHKH2ARHJTFGqhp4Afgs8CSRSG07be61sG0N6FTCoZ0HcoYiIxCJKIqh395+lPJIYNLckmLV6G5doSUoRyWBREsFdZnY78CzQ0LrT3d9JWVRtZNnmamoampk8XO0DIpK5oiSC8QRTUZ/D+1VDHm53aO+u3wnASYOKYo5ERCQ+URLBx4Dh7t6Y6mDa2rvrdlBc2JmBPfLjDkVEJDZRRhYvAtLyT+Z563cycVCRViMTkYwW5Y6gCFhmZrPZu42gQ3cfraprYnVlLZedNCDuUEREYhUlEdye8ihiML88aB+YOKhHzJGIiMQrysjiV9oikLY2b/1OzGDCIC1LKSKZLWPXI5i3ficjSgrplpcTdygiIrHK2PUIFpRXcdbokrjDEBGJXUauR7C1poGtNQ2M7adpp0VEUroegZlNA+4CsoH73P0H+xz/GvBZgumtK4HPtMVaySu2VAMwRusPiIikbj0CM8sGfgmcD5QDs81sprsvSTrtXaDU3evM7PPAncCVEWM/Yss3h4mgjxKBiEgq1yM4FShz99UAZvYIQQLZkwjc/aWk82cB1x7hZx2WFVuq6VGQQ0nX3Lb4OBGRdu2QbQRm9qCZFSVt9zCzGRHeewCwPmm7PIqXs2EAAArZSURBVNx3IDcAfz9ADNPNbI6ZzamsrIzw0Qe3fHM1o/t01YhiERGiNRZPcPedrRvuvgM46VgGYWbXErQ9/M/+jrv7Pe5e6u6lJSVH19PH3VmxpUbtAyIioSiJIMvM9gy/NbOeRGtb2AAMStoeGO7bi5mdB/wbcIm7N+x7/FjbVFVPTUMzo3oXpvqjREQ6hCgX9B8Bb5rZY+H2x4D/jPC62cCocKnLDcBVwDXJJ5jZScBvgGnuXhE56qOwZlstAMOKlQhERCBaY/HvzGwO768/cNk+PX8O9LpmM7sFeIag++gMd19sZt8D5rj7TIKqoELgsbC+fl2qJ7Nbs7UOgKHFWppSRASi3REQXvgPefHfz+ueAp7aZ9+3k56fd7jvebTWbqulc3YW/bprDQIRETjMkcXp4L2ttQzuVUB2lnoMiYhABiaCtdvqGNpL1UIiIq0yKhEkEs6abbUM7dUl7lBERNqNjEoEW6rraWhOMKRYiUBEpFVGJYI9PYZUNSQiskdGJYItu4JJU9VjSETkfRmVCCqqg0TQu5smmxMRaZVZiWBXA3k5WXTNjTR8QkQkI2RWIqhuoHfXPM06KiKSJMMSQb3WIBAR2UdGJYLK6gZ6KxGIiOwloxLBjromenbpHHcYIiLtSsYkgkTC2VnXSI8CJQIRkWQZkwiq65tJOBQV5MQdiohIu5IxiWDn7kYA3RGIiOwjYxLBjromQHcEIiL7yqBEENwRFOmOQERkLxmTCKrCO4IeuiMQEdlLxiQC3RGIiOxfxiSCAUX5XDCuD93zdUcgIpIsY2Zfu+D4vlxwfN+4wxARaXcy5o5ARET2T4lARCTDKRGIiGQ4JQIRkQynRCAikuGUCEREMpwSgYhIhlMiEBHJcObuccdwWMysElh7hC8vBrYew3Dau0wqbyaVFVTedJaqsg5x95L9HehwieBomNkcdy+NO462kknlzaSygsqbzuIoq6qGREQynBKBiEiGy7REcE/cAbSxTCpvJpUVVN501uZlzag2AhER+WeZdkcgIiL7UCIQEclwGZMIzGyamS03szIzuy3ueI4FM5thZhVmtihpX08ze87MVoY/e4T7zcx+FpZ/gZmdHF/kh8/MBpnZS2a2xMwWm9mXw/1pV14zyzOzt81sfljW74b7h5nZW2GZ/mhmncP9ueF2WXh8aJzxHykzyzazd83sr+F2WpbXzNaY2UIzm2dmc8J9sX6PMyIRmFk28EvgQ8A44GozGxdvVMfEA8C0ffbdBrzg7qOAF8JtCMo+KnxMB37dRjEeK83A1919HDAZ+EL4b5iO5W0AznH3E4GJwDQzmwzcAfzE3UcCO4AbwvNvAHaE+38SntcRfRlYmrSdzuU9290nJo0XiPd77O5p/wCmAM8kbX8L+FbccR2jsg0FFiVtLwf6hc/7AcvD578Brt7feR3xAfwFOD/dywsUAO8AkwhGm3YK9+/5TgPPAFPC553C8yzu2A+znAMJLoDnAH8FLF3LC6wBivfZF+v3OCPuCIABwPqk7fJwXzrq4+6bwuebgT7h87T5HYRVAScBb5Gm5Q2rSeYBFcBzwCpgp7s3h6ckl2dPWcPjVUCvto34qP0U+CaQCLd7kb7ldeBZM5trZtPDfbF+jzNm8fpM5O5uZmnVP9jMCoE/AV9x911mtudYOpXX3VuAiWZWBPwfcFzMIaWMmV0EVLj7XDObGnc8beB0d99gZr2B58xsWfLBOL7HmXJHsAEYlLQ9MNyXjraYWT+A8GdFuL/D/w7MLIcgCfzB3f8c7k7b8gK4+07gJYKqkSIza/3jLbk8e8oaHu8ObGvjUI/GacAlZrYGeISgeugu0rS87r4h/FlBkORPJebvcaYkgtnAqLAXQmfgKmBmzDGlykzgk+HzTxLUpbfuvz7shTAZqEq6FW33LPjT/7fAUnf/cdKhtCuvmZWEdwKYWT5BW8hSgoRwRXjavmVt/R1cAbzoYYVyR+Du33L3ge4+lOD/5ovu/gnSsLxm1sXMurY+By4AFhH39zjuhpM2bKD5MLCCoK713+KO5xiV6WFgE9BEUHd4A0Fd6QvASuB5oGd4rhH0nFoFLARK447/MMt6OkHd6gJgXvj4cDqWF5gAvBuWdRHw7XD/cOBtoAx4DMgN9+eF22Xh8eFxl+Eoyj4V+Gu6ljcs0/zwsbj1WhT391hTTIiIZLhMqRoSEZEDUCIQEclwSgQiIhlOiUBEJMMpEYiIZDglAunQzOxlM0v5Qt9m9iUzW2pmf0j1Z8XJzIrM7Oa445C2pUQgGStp1GoUNwPnezDQKZ0VEZRVMogSgaScmQ0N/5q+N5xf/9lwxOxef9GbWXE4zQBm9ikzeyKcm32Nmd1iZl8L56ufZWY9kz7iunBu90Vmdmr4+i4WrNfwdviaS5Ped6aZvUgwgGffWL8Wvs8iM/tKuO9ugoFAfzezr+5zfraZ/TA8f4GZfTHcf274uQvDOHLD/WvM7L9b56I3s5PN7BkzW2VmN4XnTDWzV83sbxasoXG3mWWFx64O33ORmd2RFEeNmf2nBWsYzDKzPuH+EjP7k5nNDh+nhfu/E8b1spmtNrMvhW/1A2BEGN//mFm/MJbW3+8ZR/xFkPYr7pF2eqT/g2Cq7GZgYrj9KHBt+PxlwtGSQDGwJnz+KYKRo12BEoIZJm8Kj/2EYNK51tffGz4/k3BKbuC/kj6jiGBUeZfwfcsJR27uE+cpBKM3uwCFBCM/TwqPrWGfqYPD/Z8HHuf96ZJ7Eox8XQ+MDvf9LineNcDnk8qxIKmMW8L9U4F6guSTTTD76BVAf2BdeG4n4EXgo+FrHLg4fH4n8O/h8/8lmOQMYDDBFB0A3wHeAHLD3/s2IId/ntb867w/+jUb6Br390mPY//Q7KPSVt5z93nh87kEF5xDecndq4FqM6sCngz3LySYhqHVwwDu/qqZdQvn6bmAYCKzb4Tn5BFcCAGec/ft+/m804H/c/daADP7M3AGwXQPB3IecLeH0yW7+3YzOzEs74rwnAeBLxBMtQzvz3O1EChMKmND6xxDwNvuvjqM4+EwtibgZXevDPf/gSD5PQE0EszjD8Hv9/yk+MbZ+7O0drNgBleAv7l7A9BgZhW8P/VxstnADAsm/Hsi6d9Q0ogSgbSVhqTnLUB++LyZ96so8w7ymkTSdoK9v7v7zpPiBHO0XO7uy5MPmNkkoPawIj/2ksuxbxlby7W/Mh1Mk7u3ntOS9D5ZwGR3r08+OUwM+/6b/NP1IEyuZwIfAR4wsx+7++8OEYt0MGojkLitIaiSgfdnmjxcVwKY2ekEszNWEaxi9UULr3hmdlKE9/kH8FEzKwhnhvyXcN/BPAfc2NrwHLZdLAeGmtnI8JzrgFcOs0ynWjBbbhZB+V4jmGDtrLAtJRu4OsL7Pgt8sXXDzCYe4vxqgqqq1vOHEFRZ3QvcB3SYtZ8lOiUCidsPgc+b2bsEddVHoj58/d28v67t9wnqvBeY2eJw+6Dc/R2CdaDfJlj97D53P1i1EAQXx3Xh58wHrgn/+v408JiZLST4S//uwyzTbOAXBNNPv0dQZbWJYC3blwhmr5zr7n858FsA8CWgNGzIXgLcdLCT3X0b8HrYMPw/BO0V88Pf75UE6wRImtHsoyLtjAWrdH3D3S+KOxbJDLojEBHJcLojEBHJcLojEBHJcEoEIiIZTolARCTDKRGIiGQ4JQIRkQz3/wFOPVkBTRXNKwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n4qYCfpbvSZV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "72657ece-1ed4-436d-8f1b-7447c51312f2"
      },
      "source": [
        "# need around 200 components to describe 100% of variance\n",
        "pca = PCA(n_components = 50)\n",
        "reduced_embeds = pca.fit_transform(rescaled)\n",
        "print(\"Original shape:   \", rescaled.shape)\n",
        "print(\"Transformed shape:\", reduced_embeds.shape)"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original shape:    (47164, 512)\n",
            "Transformed shape: (47164, 50)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pMYogewKvSZW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "57fbbfb1-ce5e-4c83-89f8-22f22efc715b"
      },
      "source": [
        "pca = PCA(n_components = 50)\n",
        "reduced_embeds = pca.fit_transform(rescaled)\n",
        "reduced_embeds.shape"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(47164, 50)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kVexQw8e9UYT"
      },
      "source": [
        "# Save embeddings in .parquet and .tsv\n",
        "df_reduced = pd.DataFrame(reduced_embeds)\n",
        "df_reduced.columns = [ str(x) for x in df_reduced.columns ]\n",
        "df_reduced.to_parquet(f'vecs{DEBATE_NUMBER}.parquet', index=False)\n",
        "np.savetxt(f'vecs{DEBATE_NUMBER}.tsv', reduced_embeds , delimiter=\"\\t\")\n",
        "np.savetxt(f'vecs{DEBATE_NUMBER}_5k.tsv', reduced_embeds[:5000] , delimiter=\"\\t\")"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-H6f_9ZR-Z3w"
      },
      "source": [
        "After dimension reduction using PCA, the file size has been reduced from 634 MB to 62 MB (or 149 MB to 15 MB for the .parquet format)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ubaCiKEK9bNe",
        "outputId": "6b1feea9-75b4-47ff-bc96-20be74fdc685"
      },
      "source": [
        "!ls -lh vecs*"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "-rw-r--r-- 1 root root 138M Jun 14 13:37 vecs0_2.parquet\n",
            "-rw-r--r-- 1 root root 588M Jun 14 13:37 vecs0_2.tsv\n",
            "-rw-r--r-- 1 root root 149M Jun 14 12:56 vecs0.parquet\n",
            "-rw-r--r-- 1 root root 634M Jun 14 12:57 vecs0.tsv\n",
            "-rw-r--r-- 1 root root 6.1M Jun 14 13:37 vecs2_5k.tsv\n",
            "-rw-r--r-- 1 root root  14M Jun 14 13:37 vecs2.parquet\n",
            "-rw-r--r-- 1 root root  58M Jun 14 13:37 vecs2.tsv\n",
            "-rw-r--r-- 1 root root 6.1M Jun 14 13:10 vecs_5k.tsv\n",
            "-rw-r--r-- 1 root root  15M Jun 14 13:10 vecs.parquet\n",
            "-rw-r--r-- 1 root root  62M Jun 14 13:10 vecs.tsv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wwHbLUzpvSZW"
      },
      "source": [
        "## Run HDBScan to find the clusters\n",
        "\n",
        "Note: the next code block takes a long time"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hVxDNTW4vSZX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "61dc5b11-d904-4f3a-ed6e-b5b320843cf6"
      },
      "source": [
        "import hdbscan\n",
        "\n",
        "clusterer = hdbscan.HDBSCAN()\n",
        "clusterer.fit(reduced_embeds)\n",
        "clusterer.labels_"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ -1,  -1,  -1, ...,  -1, 145, 201])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MdzdxO4gvSZX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b45b7805-e78f-4c82-acf3-97206ff60842"
      },
      "source": [
        "clusterer.labels_.max()"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "335"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GuyjHVj7vSZX"
      },
      "source": [
        "df_corpus['hdb_labels'] = clusterer.labels_"
      ],
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qJObqzO-ocCR"
      },
      "source": [
        "# df_meta_lab = df_corpus[['comments_cleaned','hdb_labels']].copy()\n",
        "df_meta_lab = df_corpus[['comment_text','hdb_labels']].copy()"
      ],
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JSv_FuBsolLb"
      },
      "source": [
        "df_meta_lab['comment_text'] = df_meta_lab['comment_text'].str.replace('\\n', ' ')"
      ],
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ItoZ2c79vSZX"
      },
      "source": [
        "df_meta_lab.to_csv(f'meta_lab{DEBATE_NUMBER}.tsv', index=False, header=True, sep='\\t')\n",
        "df_meta_lab.head(5000).to_csv(f'meta_lab{DEBATE_NUMBER}_5k.tsv', index=False, header=True, sep='\\t')\n",
        "df_meta_lab.to_parquet(f'meta_lab{DEBATE_NUMBER}.parquet', index=False)"
      ],
      "execution_count": 47,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iGK0xEcJvSZY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f4d36734-0057-4cbc-97bb-0289be626445"
      },
      "source": [
        "!head -7 meta.tsv"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "trump s america 7m cases biden 200k people have already died under his rule biden s america 30m cases 500k deaths biden well sometimes we fall and rise again\n",
            "wow joes laugh makes me cringe grimacing face\n",
            "dang hate to say it trump seems sharp compared to biden recently\n",
            "hey biden i have a feeling you were in on the scandal in new york to make trump look bad make more deaths on his watch\n",
            "fake briden look what he is doing to america\n",
            "why is there not a panel of questionnaires america united states is to only listen view 1 persons questions really\n",
            "i just watched bidens first news press conference and then thought i d stop by here again since it s so hilarious\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LVw6ZMisvSZY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b827c3fa-2022-42a3-91ca-f06165f3d4bc"
      },
      "source": [
        "!head -8 meta_lab.tsv"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "comment_text\thdb_labels\n",
            "Trump's America: *7M cases* Biden: 200K people have already died under his rule!  Biden's America: *30M CASES+500K deaths* Biden: Well sometimes we fall and rise again\t-1\n",
            "Wow Joes laugh makes me cringe 😬\t104\n",
            "Dang hate to say it,  Trump seems sharp compared to Biden recently\t-1\n",
            "HEY BIDEN, I HAVE A FEELING YOU WERE IN ON THE SCANDAL IN NEW YORK  TO MAKE TRUMP LOOK BAD, MAKE MORE DEATHS ON HIS WATCH!!!\t-1\n",
            "FAKE BRIDEN!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! LOOK WHAT HE IS DOING TO AMERICA!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\t-1\n",
            "Why is there NOT a panel of questionnaires? America 🇺🇸 is to only listen & view 1 persons questions?!? Really???\t-1\n",
            "I just watched bidens first news press conference and then thought I'd stop by here again since it's so hilarious\t-1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hcb3V6wK5ahi",
        "outputId": "30b1e51d-73ce-477d-b3ea-ef7fdba3742b"
      },
      "source": [
        "!ls -lh *.tsv"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "-rw-r--r-- 1 root root 6.0M Jun 14 13:34 meta2.tsv\n",
            "-rw-r--r-- 1 root root 626K Jun 14 13:39 meta_lab2_5k.tsv\n",
            "-rw-r--r-- 1 root root 6.2M Jun 14 13:39 meta_lab2.tsv\n",
            "-rw-r--r-- 1 root root 644K Jun 14 13:11 meta_lab_5k.tsv\n",
            "-rw-r--r-- 1 root root 6.1M Jun 14 13:11 meta_lab.tsv\n",
            "-rw-r--r-- 1 root root 5.9M Jun 14 12:47 meta.tsv\n",
            "-rw-r--r-- 1 root root 588M Jun 14 13:37 vecs0_2.tsv\n",
            "-rw-r--r-- 1 root root 634M Jun 14 12:57 vecs0.tsv\n",
            "-rw-r--r-- 1 root root 6.1M Jun 14 13:37 vecs2_5k.tsv\n",
            "-rw-r--r-- 1 root root  58M Jun 14 13:37 vecs2.tsv\n",
            "-rw-r--r-- 1 root root 6.1M Jun 14 13:10 vecs_5k.tsv\n",
            "-rw-r--r-- 1 root root  62M Jun 14 13:10 vecs.tsv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KYNQmv0SpFjt",
        "outputId": "99148e04-f82c-419c-bb6c-545f445a1cba"
      },
      "source": [
        "!ls -lh *.parquet"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "-rw-r--r-- 1 root root 3.7M Jun 14 13:34 meta2.parquet\n",
            "-rw-r--r-- 1 root root 4.1M Jun 14 13:39 meta_lab2.parquet\n",
            "-rw-r--r-- 1 root root 4.0M Jun 14 13:11 meta_lab.parquet\n",
            "-rw-r--r-- 1 root root 3.6M Jun 14 12:47 meta.parquet\n",
            "-rw-r--r-- 1 root root 138M Jun 14 13:37 vecs0_2.parquet\n",
            "-rw-r--r-- 1 root root 149M Jun 14 12:56 vecs0.parquet\n",
            "-rw-r--r-- 1 root root  14M Jun 14 13:37 vecs2.parquet\n",
            "-rw-r--r-- 1 root root  15M Jun 14 13:10 vecs.parquet\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mh1NbTfbvSZY"
      },
      "source": [
        "Save `vecs_5k.tsv` and `meta_lab_5k.tsv`, containing the first 5000 embedding & label pairs.  Next, open http://projector.tensorflow.org/. Press \"Load\" and use `vecs_5k.tsv` for the first file, and `meta_lab_5k.tsv` for the second file. For the full original embedddings & labels, download `vecs.parquet`, `vecs0.parquet`, `meta.parquet`, and `meta_lab.parquet`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8HLvTekSvSZY",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "e958c2dc-5564-4abd-c974-4b9e816872bd"
      },
      "source": [
        "from google.colab import files\n",
        "\n",
        "files.download(f'vecs{DEBATE_NUMBER}_5k.tsv')\n",
        "files.download(f'meta_lab{DEBATE_NUMBER}_5k.tsv')\n",
        "files.download(f'vecs{DEBATE_NUMBER}.parquet')\n",
        "files.download(f'vecs0_{DEBATE_NUMBER}.parquet')\n",
        "files.download(f'meta{DEBATE_NUMBER}.parquet')\n",
        "files.download(f'meta_lab{DEBATE_NUMBER}.parquet')"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_ad4e65a8-14df-4431-a283-72651c275a48\", \"vecs2_5k.tsv\", 6376665)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_e39628e6-2252-4ace-9840-d46cd8fca11f\", \"meta_lab2_5k.tsv\", 640472)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_12abf242-ad77-4af5-8c88-0b9d7ca694d6\", \"vecs2.parquet\", 14123458)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_bd562bb7-3456-4b60-989a-042dca76ddfb\", \"vecs0_2.parquet\", 144215487)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_cd169576-20d1-4097-967f-13f3c78f8b22\", \"meta2.parquet\", 3813661)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_deb48822-c6ac-4662-8047-3b8a3dd8d31d\", \"meta_lab2.parquet\", 4276781)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    }
  ]
}