FinalCode.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mcggv4CZBTuz"
   },
   "source": [
    "# **Capstone Project: Music Recommendation System**\n",
    "--------------\n",
    "## **Context**\n",
    "--------------\n",
    "\n",
    "With the advent of technology, societies have become more efficient with their lives. At the same\n",
    "time, however, individual human lives have also become more fast-paced and distracted, leaving\n",
    "little time to explore artistic pursuits. Also, technology has made significant advancements in the\n",
    "ability to coexist with art and general entertainment. It has in fact made it easier for humans with a\n",
    "shortage of time to find and consume good content.\n",
    "Almost every internet-based company's revenue relies on the time consumers spend on its platform.\n",
    "These companies need to be able to figure out what kind of content is needed in order to increase\n",
    "customer time spent and make their experience better. Therefore, one of the key challenges for\n",
    "these companies is figuring out what kind of content their customers are most likely to consume.\n",
    "Spotify is one such audio content provider with a huge market base across the world. With the\n",
    "ever-increasing volume of songs becoming available on the Internet, searching for songs of interest\n",
    "has become a tedious task in itself. However, Spotify has grown significantly in the market because\n",
    "of its ability to recommend the ‘best’ next song to each and every customer based on a huge\n",
    "preference database gathered over time - millions of customers and billions of songs. This is done\n",
    "by using smart recommendation systems that can recommend songs based on users’ likes/dislikes.\n",
    "\n",
    "----------------\n",
    "## **Objective**\n",
    "----------------\n",
    "\n",
    "Build a recommendation system to propose the top 10 songs for a user based on the likelihood of\n",
    "listening to those songs.\n",
    "\n",
    "-----------------------------\n",
    "## **Dataset** \n",
    "-----------------------------\n",
    "\n",
    "The core dataset is the Taste Profile Subset released by The Echo Nest as part of the Million Song\n",
    "Dataset. There are two files in this dataset. One contains the details about the song id, titles, release,\n",
    "artist name, and the year of release. The second file contains the user id, song id, and the play count\n",
    "of users.\n",
    "\n",
    "**song_data**\n",
    "- **song_id:** A unique id given to every song\n",
    "- **title:** Title of the song\n",
    "- **Release:** Name of the released album\n",
    "- **Artist_name:** Name of the artist\n",
    "- **year:** Year of release\n",
    "\n",
    "**count_data**\n",
    "- **user _id:** A unique id given to the user\n",
    "- **song_id:** A unique id given to the song\n",
    "- **play_count:** Number of times the song was played"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TwYP-CgJRZ6e"
   },
   "source": [
    "### **Importing Libraries and the Dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1BaOWvoa_47V",
    "outputId": "716429ac-83a9-40a2-8252-f8d1ba625847"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "# Mounting the drive\n",
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "nEvndTawSaWW"
   },
   "outputs": [],
   "source": [
    "# Used to ignore the warning given as output of the code\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# Basic libraries of python for numeric and dataframe computations\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Basic library for data visualization\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Slightly advanced library for data visualization\n",
    "import seaborn as sns\n",
    "\n",
    "# To compute the cosine similarity between two vectors\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "# A dictionary output that does not raise a key error\n",
    "from collections import defaultdict\n",
    "\n",
    "# A performance metrics in sklearn\n",
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hPWUF7QIScm7"
   },
   "source": [
    "### **Load the dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "8xNsdbHqSelQ"
   },
   "outputs": [],
   "source": [
    "# Importing the datasets\n",
    "count_df = pd.read_csv('/content/drive/MyDrive/Great Learning/Capstone Project/Dataset/count_data.csv')\n",
    "song_df = pd.read_csv('/content/drive/MyDrive/Great Learning/Capstone Project/Dataset/song_data.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8dLXp1otSlQE"
   },
   "source": [
    "### **Understanding the data by viewing a few observations**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "id": "_ZBzyzE0SrpZ",
    "outputId": "af5b3a21-39fb-402b-a269-6218e389fe85"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-f489a14e-c250-4350-8418-195b893354dd\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>user_id</th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>b80344d063b5ccb3212f76538f3d9e43d87dca9e</td>\n",
       "      <td>SOAKIMP12A8C130995</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>b80344d063b5ccb3212f76538f3d9e43d87dca9e</td>\n",
       "      <td>SOBBMDR12A8C13253B</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>b80344d063b5ccb3212f76538f3d9e43d87dca9e</td>\n",
       "      <td>SOBXHDL12A81C204C0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>b80344d063b5ccb3212f76538f3d9e43d87dca9e</td>\n",
       "      <td>SOBYHAJ12A6701BF1D</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>b80344d063b5ccb3212f76538f3d9e43d87dca9e</td>\n",
       "      <td>SODACBL12A8C13C273</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>b80344d063b5ccb3212f76538f3d9e43d87dca9e</td>\n",
       "      <td>SODDNQT12A6D4F5F7E</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>b80344d063b5ccb3212f76538f3d9e43d87dca9e</td>\n",
       "      <td>SODXRTY12AB0180F3B</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>b80344d063b5ccb3212f76538f3d9e43d87dca9e</td>\n",
       "      <td>SOFGUAY12AB017B0A8</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>b80344d063b5ccb3212f76538f3d9e43d87dca9e</td>\n",
       "      <td>SOFRQTD12A81C233C0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9</td>\n",
       "      <td>b80344d063b5ccb3212f76538f3d9e43d87dca9e</td>\n",
       "      <td>SOHQWYZ12A6D4FA701</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f489a14e-c250-4350-8418-195b893354dd')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-f489a14e-c250-4350-8418-195b893354dd button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-f489a14e-c250-4350-8418-195b893354dd');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   Unnamed: 0                                   user_id             song_id  \\\n",
       "0           0  b80344d063b5ccb3212f76538f3d9e43d87dca9e  SOAKIMP12A8C130995   \n",
       "1           1  b80344d063b5ccb3212f76538f3d9e43d87dca9e  SOBBMDR12A8C13253B   \n",
       "2           2  b80344d063b5ccb3212f76538f3d9e43d87dca9e  SOBXHDL12A81C204C0   \n",
       "3           3  b80344d063b5ccb3212f76538f3d9e43d87dca9e  SOBYHAJ12A6701BF1D   \n",
       "4           4  b80344d063b5ccb3212f76538f3d9e43d87dca9e  SODACBL12A8C13C273   \n",
       "5           5  b80344d063b5ccb3212f76538f3d9e43d87dca9e  SODDNQT12A6D4F5F7E   \n",
       "6           6  b80344d063b5ccb3212f76538f3d9e43d87dca9e  SODXRTY12AB0180F3B   \n",
       "7           7  b80344d063b5ccb3212f76538f3d9e43d87dca9e  SOFGUAY12AB017B0A8   \n",
       "8           8  b80344d063b5ccb3212f76538f3d9e43d87dca9e  SOFRQTD12A81C233C0   \n",
       "9           9  b80344d063b5ccb3212f76538f3d9e43d87dca9e  SOHQWYZ12A6D4FA701   \n",
       "\n",
       "   play_count  \n",
       "0           1  \n",
       "1           2  \n",
       "2           1  \n",
       "3           1  \n",
       "4           1  \n",
       "5           5  \n",
       "6           1  \n",
       "7           1  \n",
       "8           1  \n",
       "9           1  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See top 10 records of count_df data\n",
    "count_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "id": "2lyYwKzLSuJa",
    "outputId": "67b03ebd-6906-47b0-e21e-2e68aaa712eb"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-bdf32ef7-5032-43bf-a063-3766a8e4ee82\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>song_id</th>\n",
       "      <th>title</th>\n",
       "      <th>release</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>SOQMMHC12AB0180CB8</td>\n",
       "      <td>Silent Night</td>\n",
       "      <td>Monster Ballads X-Mas</td>\n",
       "      <td>Faster Pussy cat</td>\n",
       "      <td>2003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>SOVFVAK12A8C1350D9</td>\n",
       "      <td>Tanssi vaan</td>\n",
       "      <td>Karkuteillä</td>\n",
       "      <td>Karkkiautomaatti</td>\n",
       "      <td>1995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>SOGTUKN12AB017F4F1</td>\n",
       "      <td>No One Could Ever</td>\n",
       "      <td>Butter</td>\n",
       "      <td>Hudson Mohawke</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>SOBNYVR12A8C13558C</td>\n",
       "      <td>Si Vos Querés</td>\n",
       "      <td>De Culo</td>\n",
       "      <td>Yerba Brava</td>\n",
       "      <td>2003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SOHSBXH12A8C13B0DF</td>\n",
       "      <td>Tangle Of Aspens</td>\n",
       "      <td>Rene Ablaze Presents Winter Sessions</td>\n",
       "      <td>Der Mystic</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>SOZVAPQ12A8C13B63C</td>\n",
       "      <td>Symphony No. 1 G minor \"Sinfonie Serieuse\"/All...</td>\n",
       "      <td>Berwald: Symphonies Nos. 1/2/3/4</td>\n",
       "      <td>David Montgomery</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>SOQVRHI12A6D4FB2D7</td>\n",
       "      <td>We Have Got Love</td>\n",
       "      <td>Strictly The Best Vol. 34</td>\n",
       "      <td>Sasha / Turbulence</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>SOEYRFT12AB018936C</td>\n",
       "      <td>2 Da Beat Ch'yall</td>\n",
       "      <td>Da Bomb</td>\n",
       "      <td>Kris Kross</td>\n",
       "      <td>1993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>SOPMIYT12A6D4F851E</td>\n",
       "      <td>Goodbye</td>\n",
       "      <td>Danny Boy</td>\n",
       "      <td>Joseph Locke</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>SOJCFMH12A8C13B0C2</td>\n",
       "      <td>Mama_ mama can't you see ?</td>\n",
       "      <td>March to cadence with the US marines</td>\n",
       "      <td>The Sun Harbor's Chorus-Documentary Recordings</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bdf32ef7-5032-43bf-a063-3766a8e4ee82')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-bdf32ef7-5032-43bf-a063-3766a8e4ee82 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-bdf32ef7-5032-43bf-a063-3766a8e4ee82');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "              song_id                                              title  \\\n",
       "0  SOQMMHC12AB0180CB8                                       Silent Night   \n",
       "1  SOVFVAK12A8C1350D9                                        Tanssi vaan   \n",
       "2  SOGTUKN12AB017F4F1                                  No One Could Ever   \n",
       "3  SOBNYVR12A8C13558C                                      Si Vos Querés   \n",
       "4  SOHSBXH12A8C13B0DF                                   Tangle Of Aspens   \n",
       "5  SOZVAPQ12A8C13B63C  Symphony No. 1 G minor \"Sinfonie Serieuse\"/All...   \n",
       "6  SOQVRHI12A6D4FB2D7                                   We Have Got Love   \n",
       "7  SOEYRFT12AB018936C                                  2 Da Beat Ch'yall   \n",
       "8  SOPMIYT12A6D4F851E                                            Goodbye   \n",
       "9  SOJCFMH12A8C13B0C2                         Mama_ mama can't you see ?   \n",
       "\n",
       "                                release  \\\n",
       "0                 Monster Ballads X-Mas   \n",
       "1                           Karkuteillä   \n",
       "2                                Butter   \n",
       "3                               De Culo   \n",
       "4  Rene Ablaze Presents Winter Sessions   \n",
       "5      Berwald: Symphonies Nos. 1/2/3/4   \n",
       "6             Strictly The Best Vol. 34   \n",
       "7                               Da Bomb   \n",
       "8                             Danny Boy   \n",
       "9  March to cadence with the US marines   \n",
       "\n",
       "                                      artist_name  year  \n",
       "0                                Faster Pussy cat  2003  \n",
       "1                                Karkkiautomaatti  1995  \n",
       "2                                  Hudson Mohawke  2006  \n",
       "3                                     Yerba Brava  2003  \n",
       "4                                      Der Mystic     0  \n",
       "5                                David Montgomery     0  \n",
       "6                              Sasha / Turbulence     0  \n",
       "7                                      Kris Kross  1993  \n",
       "8                                    Joseph Locke     0  \n",
       "9  The Sun Harbor's Chorus-Documentary Recordings     0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See top 10 records of song_df data\n",
    "song_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_hBZRA0hSxEq"
   },
   "source": [
    "### **Let us check the data types and and missing values of each column**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ixfuY7p4Sxk1",
    "outputId": "2a06dd50-64cb-47f8-e7fd-37880e3c88c1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2000000 entries, 0 to 1999999\n",
      "Data columns (total 4 columns):\n",
      " #   Column      Dtype \n",
      "---  ------      ----- \n",
      " 0   Unnamed: 0  int64 \n",
      " 1   user_id     object\n",
      " 2   song_id     object\n",
      " 3   play_count  int64 \n",
      "dtypes: int64(2), object(2)\n",
      "memory usage: 61.0+ MB\n"
     ]
    }
   ],
   "source": [
    "# See the info of the count_df data\n",
    "count_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aAPywRJySzjh",
    "outputId": "0914e968-8135-4f2a-c9fb-7d0fd41f69d5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1000000 entries, 0 to 999999\n",
      "Data columns (total 5 columns):\n",
      " #   Column       Non-Null Count    Dtype \n",
      "---  ------       --------------    ----- \n",
      " 0   song_id      1000000 non-null  object\n",
      " 1   title        999985 non-null   object\n",
      " 2   release      999995 non-null   object\n",
      " 3   artist_name  1000000 non-null  object\n",
      " 4   year         1000000 non-null  int64 \n",
      "dtypes: int64(1), object(4)\n",
      "memory usage: 38.1+ MB\n"
     ]
    }
   ],
   "source": [
    "# See the info of the song_df data\n",
    "song_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DxwmjGW0S2NQ",
    "outputId": "42dee348-0c4b-4158-9a58-a7fbd6047a13"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Unnamed: 0    0\n",
       "user_id       0\n",
       "song_id       0\n",
       "play_count    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See the missing values of the count_df data\n",
    "count_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xJHTW5cTS2hE",
    "outputId": "93399b39-4b02-4ea7-c610-5f4995171f38"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "song_id         0\n",
       "title          15\n",
       "release         5\n",
       "artist_name     0\n",
       "year            0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See the missing values of the song_df data\n",
    "song_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qi7PsKDIS3-7"
   },
   "source": [
    "#### **Observations and Insights:**\n",
    "\n",
    "* There are **2,000,000 observations and 4 columns** in the count_df data.\n",
    "* The 'user_id' and 'song_id' are of **object data type**. The 'play_count' column is of **numeric data type**. We also have a column named 'Unnamed' that we are dropping because it doesn't add value to our analysis.\n",
    "* There are no **missing values** in the count_df data.\n",
    "* There are **1,000,000 observations and 5 columns** in the song_df data.\n",
    "* All the columns are of **object data type** except the **year column**. The year column is of **numeric data type**.\n",
    "* In the song_df data we have **15 missing values** in the **title column** and **5 missing values** in the **release column**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BtInPSkkS_jX"
   },
   "source": [
    "### **Merge the datasets and drop the 'Unnamed:0' column**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "qtlvcLxRS6KR"
   },
   "outputs": [],
   "source": [
    "# Left merge the count_df and song_df data on \"song_id\". Drop duplicates from song_df data simultaneously\n",
    "df = pd.merge(count_df, song_df.drop_duplicates(['song_id']), on = 'song_id', how = 'left')\n",
    "\n",
    "# Drop the column 'Unnamed: 0'\n",
    "df = df.drop(['Unnamed: 0'], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aRAya3lGazuT"
   },
   "source": [
    "**Think About It:** As the user_id and song_id are encrypted. Can they be encoded to numeric features?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bvBB_ufPa5a8"
   },
   "source": [
    "### **Apply Label Encoding to the columns 'user_id' and 'song_id'**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "3OHJgHfMbC3u"
   },
   "outputs": [],
   "source": [
    "# Apply label encoding for \"user_id\" and \"song_id\"\n",
    "from sklearn.preprocessing import LabelEncoder    # To do label encoding\n",
    "\n",
    "#Label Encoding\n",
    "le = LabelEncoder()\n",
    "\n",
    "#Fit transform the user_id column\n",
    "df['user_id'] = le.fit_transform(df['user_id'])\n",
    "\n",
    "#Fit transform the song_id column\n",
    "df['song_id'] = le.fit_transform(df['song_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "kkCN_I-_bGX8",
    "outputId": "d88f99d8-b68f-4f6d-d8db-9cb34c9ab4d7"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-70625af7-841c-4c71-8265-067564b1f4ae\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_count</th>\n",
       "      <th>title</th>\n",
       "      <th>release</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>54961</td>\n",
       "      <td>153</td>\n",
       "      <td>1</td>\n",
       "      <td>The Cove</td>\n",
       "      <td>Thicker Than Water</td>\n",
       "      <td>Jack Johnson</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>54961</td>\n",
       "      <td>413</td>\n",
       "      <td>2</td>\n",
       "      <td>Entre Dos Aguas</td>\n",
       "      <td>Flamenco Para Niños</td>\n",
       "      <td>Paco De Lucia</td>\n",
       "      <td>1976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>54961</td>\n",
       "      <td>736</td>\n",
       "      <td>1</td>\n",
       "      <td>Stronger</td>\n",
       "      <td>Graduation</td>\n",
       "      <td>Kanye West</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>54961</td>\n",
       "      <td>750</td>\n",
       "      <td>1</td>\n",
       "      <td>Constellations</td>\n",
       "      <td>In Between Dreams</td>\n",
       "      <td>Jack Johnson</td>\n",
       "      <td>2005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>54961</td>\n",
       "      <td>1188</td>\n",
       "      <td>1</td>\n",
       "      <td>Learn To Fly</td>\n",
       "      <td>There Is Nothing Left To Lose</td>\n",
       "      <td>Foo Fighters</td>\n",
       "      <td>1999</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-70625af7-841c-4c71-8265-067564b1f4ae')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-70625af7-841c-4c71-8265-067564b1f4ae button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-70625af7-841c-4c71-8265-067564b1f4ae');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   user_id  song_id  play_count            title  \\\n",
       "0    54961      153           1         The Cove   \n",
       "1    54961      413           2  Entre Dos Aguas   \n",
       "2    54961      736           1         Stronger   \n",
       "3    54961      750           1   Constellations   \n",
       "4    54961     1188           1     Learn To Fly   \n",
       "\n",
       "                         release    artist_name  year  \n",
       "0             Thicker Than Water   Jack Johnson     0  \n",
       "1            Flamenco Para Niños  Paco De Lucia  1976  \n",
       "2                     Graduation     Kanye West  2007  \n",
       "3              In Between Dreams   Jack Johnson  2005  \n",
       "4  There Is Nothing Left To Lose   Foo Fighters  1999  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Checking the merge dataset\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Ks4Vd80XbI8b",
    "outputId": "0d754d72-9255-48a0-9614-f1dc5b86abde"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 2000000 entries, 0 to 1999999\n",
      "Data columns (total 7 columns):\n",
      " #   Column       Dtype \n",
      "---  ------       ----- \n",
      " 0   user_id      int64 \n",
      " 1   song_id      int64 \n",
      " 2   play_count   int64 \n",
      " 3   title        object\n",
      " 4   release      object\n",
      " 5   artist_name  object\n",
      " 6   year         int64 \n",
      "dtypes: int64(4), object(3)\n",
      "memory usage: 122.1+ MB\n"
     ]
    }
   ],
   "source": [
    "# Checking the info of the merge dataset\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "emlVgRYDbKgh"
   },
   "source": [
    "**Observations:**\n",
    "\n",
    "* In the merge dataset, there are **2,000,000 observations and 7 columns**.\n",
    "* The **title, release** and **artist_name** are of **object data type**. The rest are of **numeric data type**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4lKmo_kKbMhi"
   },
   "source": [
    "**Think About It:** As the data also contains users who have listened to very few songs and vice versa, is it required to filter the data so that it contains users who have listened to a good count of songs and vice versa?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SzNJ32JtbPsK"
   },
   "source": [
    "## **Data Preparation**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HQsjYY_wbVUe"
   },
   "source": [
    "We will be taking users who have listen at least 90 songs, as we prefer to have some number of listeners of a song and the song which has at least 120 listeners."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "_FWr64lsbTth"
   },
   "outputs": [],
   "source": [
    "# Get the column containing the users\n",
    "users = df.user_id\n",
    "\n",
    "# Create a dictionary from users to their number of songs\n",
    "ratings_count = dict()\n",
    "\n",
    "for user in users:\n",
    "    # If we already have the user, just add 1 to their rating count\n",
    "    if user in ratings_count:\n",
    "        ratings_count[user] += 1\n",
    "    \n",
    "    # Otherwise, set their rating count to 1\n",
    "    else:\n",
    "        ratings_count[user] = 1    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "8PCIvevzbY5U"
   },
   "outputs": [],
   "source": [
    "# We want our users to have listened at least 90 songs\n",
    "RATINGS_CUTOFF = 90\n",
    "\n",
    "# Create a list of users who need to be removed\n",
    "remove_users = []\n",
    "\n",
    "for user, num_ratings in ratings_count.items():\n",
    "    \n",
    "    if num_ratings < RATINGS_CUTOFF:\n",
    "        remove_users.append(user)\n",
    "\n",
    "df = df.loc[ ~ df.user_id.isin(remove_users)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "G-Qw6Ek6baqC"
   },
   "outputs": [],
   "source": [
    "# Get the column containing the songs\n",
    "songs = df.song_id\n",
    "\n",
    "# Create a dictionary from songs to their number of users\n",
    "ratings_count = dict()\n",
    "\n",
    "for song in songs:\n",
    "    # If we already have the song, just add 1 to their rating count\n",
    "    if song in ratings_count:\n",
    "        ratings_count[song] += 1\n",
    "    \n",
    "    # Otherwise, set their rating count to 1\n",
    "    else:\n",
    "        ratings_count[song] = 1    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "rcBr7VQxbeIO"
   },
   "outputs": [],
   "source": [
    "# We want our song to be listened by atleast 120 users to be considred\n",
    "RATINGS_CUTOFF = 120\n",
    "\n",
    "remove_songs = []\n",
    "\n",
    "for song, num_ratings in ratings_count.items():\n",
    "  \n",
    "    if num_ratings < RATINGS_CUTOFF:\n",
    "        remove_songs.append(song)\n",
    "\n",
    "df_final= df.loc[ ~ df.song_id.isin(remove_songs)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "bH3_QSmAbgAP"
   },
   "outputs": [],
   "source": [
    "# Drop records with play_count more than(>) 5\n",
    "df_final = df_final[df_final['play_count'] <= 5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IAizGoIPbh-u",
    "outputId": "034f99be-0d9e-4c72-b2a7-fb3c0f630422"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(117876, 7)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the shape of the data\n",
    "df_final.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5PmSkT_9bmFI"
   },
   "source": [
    "**Observations:**\n",
    "\n",
    "* The **df_final** dataset has **117,876 observations and 7 columns** with the modifications made to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H5_bS-T0bovI"
   },
   "source": [
    "## **Exploratory Data Analysis**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CD2ZLAjkbrrW"
   },
   "source": [
    "### **Let's check the total number of unique users, songs, artists in the data**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iwtU9pUBbtrh"
   },
   "source": [
    "**Total number of unique user id**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "w8wPE-ZEbv32",
    "outputId": "fb2f4607-a7eb-4b29-91af-2b016952136a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3155"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display total number of unique user_id\n",
    "df_final['user_id'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dmOEmz40bye6"
   },
   "source": [
    "**Total number of unique song id**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "AxvHGw9Jb0BK",
    "outputId": "30e6507c-3e77-476d-f954-509dc689658e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "563"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display total number of unique song_id\n",
    "df_final['song_id'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5TOH4qbob2EU"
   },
   "source": [
    "**Total number of unique artists**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5kRQ1VDXb4s_",
    "outputId": "48281361-b3d3-40fa-83d0-c663dd27396b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3374"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display total number of unique artists\n",
    "df['artist_name'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-hsY8Pp1b7AU"
   },
   "source": [
    "#### **Observations and Insights:**\n",
    "\n",
    "* There are **3155 unique users** in the dataset.\n",
    "* There are **563 unique songs** in the dataset.\n",
    "* There are **3374 unique artists** in the dataset.\n",
    "* As per the number of unique users and songs, there is a **possibility of 3155 * 563 = 1,776,265 interactions/listenings** in the dataset. But **we only have 117,876 interactions/listenings**, i.e., not every user has listened to every song in the dataset. So, we can build a recommendation system to recommend songs to users which they have not listened/interacted with.\n",
    "* It seems a little bit ilogic that the dataset has more **unique artists** than **unique songs**. Maybe some names were written differently, but are the same artist. Also, there could be some different songs that have the same name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EaumY12lb-o4"
   },
   "source": [
    "### **Let's find out about the most interacted songs and interacted users**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sDxHw1hfcCbW"
   },
   "source": [
    "**Most interacted songs**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BbI2Pvz9cEh7",
    "outputId": "452b33c5-f7d7-4af7-e33b-4946e8563a69"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8582    751\n",
       "352     748\n",
       "2220    713\n",
       "1118    662\n",
       "4152    652\n",
       "       ... \n",
       "9048    103\n",
       "6450    102\n",
       "990     101\n",
       "4831     97\n",
       "8324     96\n",
       "Name: song_id, Length: 563, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_final['song_id'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "e6VWUAcZcKCT",
    "outputId": "f4ba1fb1-d330-4c7d-c03b-fab50154842b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-a7f44d1f-c0ed-4320-a0af-baba39f8a965\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_count</th>\n",
       "      <th>title</th>\n",
       "      <th>release</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1000</th>\n",
       "      <td>56576</td>\n",
       "      <td>8582</td>\n",
       "      <td>3</td>\n",
       "      <td>Use Somebody</td>\n",
       "      <td>Use Somebody</td>\n",
       "      <td>Kings Of Leon</td>\n",
       "      <td>2008</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a7f44d1f-c0ed-4320-a0af-baba39f8a965')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-a7f44d1f-c0ed-4320-a0af-baba39f8a965 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-a7f44d1f-c0ed-4320-a0af-baba39f8a965');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "      user_id  song_id  play_count         title       release    artist_name  \\\n",
       "1000    56576     8582           3  Use Somebody  Use Somebody  Kings Of Leon   \n",
       "\n",
       "      year  \n",
       "1000  2008  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_final[df_final['song_id'] == 8582].head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ih8uLYlscMeQ"
   },
   "source": [
    "**Most interacted users**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EeXLE8rucOTB",
    "outputId": "bf3b6ca0-6b26-4733-b439-8506f68272d6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "61472    243\n",
       "15733    227\n",
       "37049    202\n",
       "9570     184\n",
       "23337    177\n",
       "        ... \n",
       "19776      1\n",
       "45476      1\n",
       "17961      1\n",
       "14439      1\n",
       "10412      1\n",
       "Name: user_id, Length: 3155, dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_final['user_id'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xLUFYWsXcRtG"
   },
   "source": [
    "#### **Observations and Insights:**\n",
    "\n",
    "* The song with **song_id = 8582** has the highest interactions counts with **751**. This song_id belongs to the 'Use Somebody' song from Kings Of Leon.\n",
    "* But still, there is a possibility of **3155-751 = 2404 more interactions** as we have **3155 unique users** in our datasets. For those 2404 remaining users, we can build a recommendation system to predict who is most likely to listen to this song.\n",
    "\n",
    "* The user with **user_id = 61472** has interacted/listened to the most number of different **songs (243)**.\n",
    "* But still, there is a possibility of **563-243 = 320** more interactions as we have **563 unique songs** in our datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a4vhnIYqcm6A"
   },
   "source": [
    "### **Checking if there is only one interaction between a song and a user**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 455
    },
    "id": "E73dEpvWcT8M",
    "outputId": "a4c1a66d-e71f-4935-cc44-d31ac5d6d2a8"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-71f7f124-4dbc-4590-9101-18cf7415c4d9\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>play_count</th>\n",
       "      <th>title</th>\n",
       "      <th>release</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th>song_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">11</th>\n",
       "      <th>5158</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8019</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8882</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">17</th>\n",
       "      <th>314</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>703</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">76342</th>\n",
       "      <th>5375</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7889</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8494</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9942</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76347</th>\n",
       "      <th>5879</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>117876 rows × 5 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-71f7f124-4dbc-4590-9101-18cf7415c4d9')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-71f7f124-4dbc-4590-9101-18cf7415c4d9 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-71f7f124-4dbc-4590-9101-18cf7415c4d9');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                 play_count  title  release  artist_name  year\n",
       "user_id song_id                                               \n",
       "11      5158              1      1        1            1     1\n",
       "        8019              1      1        1            1     1\n",
       "        8882              1      1        1            1     1\n",
       "17      314               1      1        1            1     1\n",
       "        703               1      1        1            1     1\n",
       "...                     ...    ...      ...          ...   ...\n",
       "76342   5375              1      1        1            1     1\n",
       "        7889              1      1        1            1     1\n",
       "        8494              1      1        1            1     1\n",
       "        9942              1      1        1            1     1\n",
       "76347   5879              1      1        1            1     1\n",
       "\n",
       "[117876 rows x 5 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_final.groupby(['user_id', 'song_id']).count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZjMd_TaCcs-A",
    "outputId": "2c06fb3c-37d6-49de-9927-824592657007"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "117876"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_final.groupby(['user_id', 'song_id']).count()['title'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YldowFRlcwSJ"
   },
   "source": [
    "**Observation:**\n",
    "\n",
    "* The **sum is equal to the total number of observations**, which implies that **there is only one interaction between a song and a user**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FFdrhu8hc1oR"
   },
   "source": [
    "## **Songs played in a year**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "id": "ohkwk_4Pcu9j",
    "outputId": "143536ea-3cef-414f-97e5-f57cc1178fc5"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-5ec24273-2a8f-4067-9493-680e94e64640\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>7592</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007</th>\n",
       "      <td>13750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008</th>\n",
       "      <td>14031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009</th>\n",
       "      <td>16351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010</th>\n",
       "      <td>4087</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5ec24273-2a8f-4067-9493-680e94e64640')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-5ec24273-2a8f-4067-9493-680e94e64640 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-5ec24273-2a8f-4067-9493-680e94e64640');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "      title\n",
       "year       \n",
       "2006   7592\n",
       "2007  13750\n",
       "2008  14031\n",
       "2009  16351\n",
       "2010   4087"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_songs = df_final.groupby('year').count()['title']\n",
    "\n",
    "count = pd.DataFrame(count_songs)\n",
    "\n",
    "count.drop(count.index[0], inplace = True)\n",
    "\n",
    "count.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 606
    },
    "id": "g_dGpWcgc4EO",
    "outputId": "49c7848b-b46b-4ae1-933c-221d3cb81baf"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABswAAAJNCAYAAACV5FDlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdf9RuZVkn8O8FJzQdEZAjGshwTLJBR2foiLSqmcxSaDKoscKa8WSMrFUmqa1Mp1Y4prO0XxpNWRQnoExlGA1mhRI5FlOJCv4EzTiByiERFARHSwSv+ePdzDwd3/O+j3Ce54X3/nzW2uvZ+9r3fva1//6u+76ruwMAAAAAAACj2m+jGwAAAAAAAICNJDADAAAAAABgaAIzAAAAAAAAhiYwAwAAAAAAYGgCMwAAAAAAAIYmMAMAAAAAAGBoWza6gWU79NBD+6ijjtroNgAAAAAAAFiyK6+88tPdvXXP+nCB2VFHHZUrrrhio9sAAAAAAABgyarq46vVLckIAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMbctGNwAAAAAAAEBy02/82Ua3sKqHP/87N7qFhTPDDAAAAAAAgKEJzAAAAAAAABiawAwAAAAAAIChCcwAAAAAAAAYmsAMAAAAAACAoQnMAAAAAAAAGNrCArOq2llVN1XVVXvUn19Vf1NVV1fVL83UX1pVu6rqo1X19Jn6CVNtV1W9ZKa+rareNdXfVFUHLOpbAAAAAAAA2LwWOcPsnCQnzBaq6ilJTkryxO5+XJJfmerHJDklyeOmZ36rqvavqv2T/GaSE5Mck+RZ09gkeXWS13T3Y5LcmuTUBX4LAAAAAAAAm9TCArPuvizJLXuUfzzJq7r7i9OYm6b6SUne2N1f7O7rkuxKctx07Orua7v7jiRvTHJSVVWS70hywfT8uUlOXtS3AAAAAAAAsHktew+zb0jybdNSin9RVU+a6ocnuX5m3O6ptrf6w5J8trvv3KMOAAAAAAAAX5UtG/C+Q5Icn+RJSc6vqkcv+qVVdVqS05LkyCOPXPTrAAAAAAAAuB9Z9gyz3Une3CveneTLSQ5NckOSR82MO2Kq7a3+mSQHVdWWPeqr6u6zunt7d2/funXrPvsYAAAAAAAA7v+WHZj9cZKnJElVfUOSA5J8OslFSU6pqgdU1bYkRyd5d5L3JDm6qrZV1QFJTklyUXd3knckeeb0vzuSXLjULwEAAAAAAGBTWNiSjFX1hiTfnuTQqtqd5IwkO5PsrKqrktyRZMcUfl1dVecn+XCSO5M8r7vvmv7nJ5NckmT/JDu7++rpFT+b5I1V9Yok70ty9qK+BQAAAAAAgM1rYYFZdz9rL7f+w17GvzLJK1epX5zk4lXq1yY57t70CAAAAAAAAMtekhEAAAAAAADuUwRmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0BYWmFXVzqq6qaquWuXeT1dVV9Wh03VV1ZlVtauqPlhVx86M3VFV10zHjpn6N1XVh6ZnzqyqWtS3AAAAAAAAsHktcobZOUlO2LNYVY9K8rQkn5gpn5jk6Ok4LcnrprGHJDkjyZOTHJfkjKo6eHrmdUmeO/PcV7wLAAAAAAAA1rOwwKy7L0tyyyq3XpPkxUl6pnZSkvN6xeVJDqqqRyZ5epJLu/uW7r41yaVJTpjuHdjdl3d3JzkvycmL+hYAAAAAAAA2ry3LfFlVnZTkhu7+wB4rKB6e5PqZ691Tba367lXqAAAAAADA4D716+/c6Ba+wmE/9c0b3QJrWFpgVlUPSvKfs7Ic41JV1WlZWeoxRx555LJfDwAAAAAAwH3YIvcw29PXJ9mW5ANV9bEkRyR5b1U9IskNSR41M/aIqbZW/YhV6qvq7rO6e3t3b9+6des++BQAAAAAAAA2i6UFZt39oe5+eHcf1d1HZWUZxWO7+8YkFyV5dq04Pslt3f3JJJckeVpVHVxVB2dldtol073bq+r4Wlnb8dlJLlzWtwAAAAAAALB5LCwwq6o3JHlnksdW1e6qOnWN4RcnuTbJriS/m+QnkqS7b0nyi0neMx0vn2qZxvze9MzfJXnrIr4DAAAAAACAzW1he5h197PWuX/UzHkned5exu1MsnOV+hVJHn/vugQAAAAAAGB0y9zDDAAAAAAAAO5zBGYAAAAAAAAMbWFLMgIAAAAAAPdfn3rNBze6hVUd9sInbHQLbEJmmAEAAAAAADA0gRkAAAAAAABDE5gBAAAAAAAwNIEZAAAAAAAAQxOYAQAAAAAAMDSBGQAAAAAAAEMTmAEAAAAAADA0gRkAAAAAAABDE5gBAAAAAAAwNIEZAAAAAAAAQxOYAQAAAAAAMDSBGQAAAAAAAEMTmAEAAAAAADA0gRkAAAAAAABDE5gBAAAAAAAwNIEZAAAAAAAAQxOYAQAAAAAAMDSBGQAAAAAAAEMTmAEAAAAAADA0gRkAAAAAAABDE5gBAAAAAAAwNIEZAAAAAAAAQxOYAQAAAAAAMDSBGQAAAAAAAEMTmAEAAAAAADA0gRkAAAAAAABDE5gBAAAAAAAwNIEZAAAAAAAAQxOYAQAAAAAAMDSBGQAAAAAAAEMTmAEAAAAAADA0gRkAAAAAAABDE5gBAAAAAAAwNIEZAAAAAAAAQxOYAQAAAAAAMDSBGQAAAAAAAEMTmAEAAAAAADA0gRkAAAAAAABDE5gBAAAAAAAwNIEZAAAAAAAAQxOYAQAAAAAAMDSBGQAAAAAAAEMTmAEAAAAAADA0gRkAAAAAAABDE5gBAAAAAAAwNIEZAAAAAAAAQxOYAQAAAAAAMDSBGQAAAAAAAEMTmAEAAAAAADA0gRkAAAAAAABDE5gBAAAAAAAwtIUFZlW1s6puqqqrZmq/XFV/U1UfrKq3VNVBM/deWlW7quqjVfX0mfoJU21XVb1kpr6tqt411d9UVQcs6lsAAAAAAADYvBY5w+ycJCfsUbs0yeO7+wlJ/jbJS5Okqo5JckqSx03P/FZV7V9V+yf5zSQnJjkmybOmsUny6iSv6e7HJLk1yakL/BYAAAAAAAA2qYUFZt19WZJb9qj9aXffOV1enuSI6fykJG/s7i9293VJdiU5bjp2dfe13X1HkjcmOamqKsl3JLlgev7cJCcv6lsAAAAAAADYvDZyD7MfS/LW6fzwJNfP3Ns91fZWf1iSz86Eb3fXAQAAAAAA4KuyIYFZVf1ckjuTvH5J7zutqq6oqituvvnmZbwSAAAAAACA+4mlB2ZV9aNJvifJj3R3T+UbkjxqZtgRU21v9c8kOaiqtuxRX1V3n9Xd27t7+9atW/fJdwAAAAAAALA5LDUwq6oTkrw4yfd29xdmbl2U5JSqekBVbUtydJJ3J3lPkqOraltVHZDklCQXTUHbO5I8c3p+R5ILl/UdAAAAAAAAbB4LC8yq6g1J3pnksVW1u6pOTfLfkjwkyaVV9f6q+u0k6e6rk5yf5MNJ3pbked1917RH2U8muSTJR5KcP41Nkp9N8qKq2pWVPc3OXtS3AAAAAAAAsHltWX/IPdPdz1qlvNdQq7tfmeSVq9QvTnLxKvVrkxx3b3oEAAAAAACApe9hBgAAAAAAAPclAjMAAAAAAACGJjADAAAAAABgaAIzAAAAAAAAhiYwAwAAAAAAYGgCMwAAAAAAAIYmMAMAAAAAAGBoAjMAAAAAAACGJjADAAAAAABgaAIzAAAAAAAAhiYwAwAAAAAAYGgCMwAAAAAAAIYmMAMAAAAAAGBoAjMAAAAAAACGJjADAAAAAABgaAIzAAAAAAAAhiYwAwAAAAAAYGgCMwAAAAAAAIYmMAMAAAAAAGBoAjMAAAAAAACGJjADAAAAAABgaAIzAAAAAAAAhiYwAwAAAAAAYGgCMwAAAAAAAIYmMAMAAAAAAGBoAjMAAAAAAACGJjADAAAAAABgaAIzAAAAAAAAhiYwAwAAAAAAYGgCMwAAAAAAAIYmMAMAAAAAAGBoAjMAAAAAAACGJjADAAAAAABgaAIzAAAAAAAAhiYwAwAAAAAAYGgCMwAAAAAAAIYmMAMAAAAAAGBoAjMAAAAAAACGJjADAAAAAABgaAIzAAAAAAAAhiYwAwAAAAAAYGgCMwAAAAAAAIYmMAMAAAAAAGBoAjMAAAAAAACGJjADAAAAAABgaAIzAAAAAAAAhrZlbzeq6kVrPdjdv7bv2wEAAAAAAIDl2mtgluQh0+9jkzwpyUXT9TOSvHuRTQEAAAAAAMCy7DUw6+7/kiRVdVmSY7v7c9P1y5L8yVK6AwAAAAAAgAWbZw+zw5LcMXN9x1QDAAAAAACA+721lmS823lJ3l1Vb5muT05y7uJaAgAAAAAAgOVZNzDr7ldW1VuTfNtUek53v2+xbQEAAAAAAMByzLMkY5I8KMnt3f3rSXZX1bYF9gQAAAAAAABLs25gVlVnJPnZJC+dSl+T5A8X2RQAAAAAAAAsyzwzzL4vyfcm+XySdPffJ3nIeg9V1c6quqmqrpqpHVJVl1bVNdPvwVO9qurMqtpVVR+sqmNnntkxjb+mqnbM1L+pqj40PXNmVdX8nw0AAAAAAAAr5gnM7ujuTtJJUlUPnvO/z0lywh61lyR5e3cfneTt03WSnJjk6Ok4LcnrpncdkuSMJE9OclySM+4O2aYxz515bs93AQAAAAAAwLrmCczOr6rfSXJQVT03yZ8l+d31Huruy5Lcskf5pCTnTufnJjl5pn5er7h8etcjkzw9yaXdfUt335rk0iQnTPcO7O7LpzDvvJn/AgAAAAAAgLltWW9Ad/9KVX1XktuTPDbJL3T3pffwfYd19yen8xuTHDadH57k+plxu6faWvXdq9QBAAAAAADgq7JuYFZVz0/yh/ciJFtVd3dV9b78z72pqtOystRjjjzyyGW8EgAAAAAAgPuJeZZkPCzJe6rq/Ko6oarqXrzvU9Nyipl+b5rqNyR51My4I6baWvUjVqmvqrvP6u7t3b1969at96J9AAAAAAAANpt1A7Pu/vkkRyc5O8mPJrmmqv5rVX39PXjfRUl2TOc7klw4U392rTg+yW3T0o2XJHlaVR1cVQcneVqSS6Z7t1fV8VOA9+yZ/wIAAAAAAIC5rbskY/L/lk+8MSv7jt2Z5OAkF1TVpd394tWeqao3JPn2JIdW1e4kZyR5VZLzq+rUJB9P8oPT8IuTfHeSXUm+kOQ503tvqapfTPKeadzLu/uW6fwnkpyT5GuTvHU6AAAAAAAA4Ksyzx5mP5WVGVyfTvJ7SX6mu79UVfsluSbJqoFZdz9rL3/51FXGdpLn7eV/dibZuUr9iiSPX69/AAAAAAAAWMs8M8wOSfL93f3x2WJ3f7mqvmcxbQEAAAAAAMByrBuYdfcZSVJVD0/ywJn6J7r7IwvsDQAAAAAAABZuv/UGVNUzquqaJNcl+YskH4v9wgAAAAAAANgk1g3MkrwiyfFJ/ra7t2VlD7LLF9oVAAAAAAAALMk8gdmXuvszSfarqv26+x1Jti+4LwAAAAAAAFiKdfcwS/LZqvpnSS5L8vqquinJ5xfbFgAAAAAAACzHPDPMTkryD0lemORtSf4uyTMW2RQAAAAAAAAsy7ozzLp7djbZuQvsBQAAAAAAAJZur4FZVX0uSc+WputK0t194IJ7AwAAAAAAgIXba2DW3Q9ZZiMAAAAAAACwEdZdkjFJqurYJN+alRlmf9nd71toVwAAAAAAALAk+603oKp+ISt7lz0syaFJzqmqn190YwAAAAAAALAM88ww+5EkT+zuf0ySqnpVkvcnecUiGwMAAAAAAIBlWHeGWZK/T/LAmesHJLlhMe0AAAAAAADAcs0zw+y2JFdX1aVZ2cPsu5K8u6rOTJLuPn2B/QEAAAAAAMBCzROYvWU67vbni2kFAAAAAAAAlm/dwKy7z11GIwAAAAAAALAR5tnDDAAAAAAAADYtgRkAAAAAAABD+6oCs6rar6oOXFQzAAAAAAAAsGzrBmZV9UdVdWBVPTjJVUk+XFU/s/jWAAAAAAAAYPHmmWF2THffnuTkJG9Nsi3Jf1xoVwAAAAAAALAk8wRmX1NVX5OVwOyi7v5Skl5sWwAAAAAAALAc8wRmv5PkY0kenOSyqvrnSW5fZFMAAAAAAACwLFvWG9DdZyY5c6b08ap6yuJaAgAAAAAAgOVZd4ZZVR1WVWdX1Vun62OS7Fh4ZwAAAAAAALAE8yzJeE6SS5J83XT9t0lesKiGAAAAAAAAYJnmCcwO7e7zk3w5Sbr7ziR3LbQrAAAAAAAAWJJ5ArPPV9XDknSSVNXxSW5baFcAAAAAAACwJFvmGPOiJBcl+fqq+qskW5M8c6FdAQAAAAAAwJKsG5h193ur6t8meWySSvLR7v7SwjsDAAAAAACAJdhrYFZV37+XW99QVenuNy+oJwAAAAAAAFiatWaYPWONe51EYAYAAAAAAMD93l4Ds+5+TpJU1bbuvm72XlVtW3RjAAAAAAAAsAz7zTHmf6xSu2BfNwIAAAAAAAAbYa09zL4xyeOSPHSP/cwOTPLARTcGAAAAAAAAy7DWHmaPTfI9SQ7KP93P7HNJnrvIpgAAAAAAAGBZ1trD7MIkF1bVN3f3O5fYEwAAAAAAACzNWksyvri7fynJD1fVs/a8392nL7QzAAAAAAAAWIK1lmT8yPR7xTIaAQAAAAAAgI2w1pKM/3M6/UJ3//fZe1X1AwvtCgAAAAAAAJZkvznGvHTOGgAAAAAAANzvrLWH2YlJvjvJ4VV15sytA5PcuejGAAAAAAAAYBnW2sPs77Oyf9n3Jrlypv65JC9cZFMAAAAAAACwLGvtYfaBJB+oqj/q7i8tsScAAAAAAABYmnX3MBOWAQAAAAAAsJmtG5gBAAAAAADAZrbXwKyq/mD6/anltQMAAAAAAADLtdYMs2+qqq9L8mNVdXBVHTJ7LKtBAAAAAAAAWKQta9z77SRvT/LoJFcmqZl7PdUBAAAAAADgfm2vM8y6+8zu/hdJdnb3o7t728whLAMAAAAAAGBTWGuGWZKku3+8qp6Y5Num0mXd/cHFtgUAAAAAAADLsdYeZkmSqjo9yeuTPHw6Xl9Vz190YwAAAAAAALAM684wS/Kfkjy5uz+fJFX16iTvTPIbi2wMAAAAAAAAlmHdGWZJKsldM9d3TbV7rKpeWFVXV9VVVfWGqnpgVW2rqndV1a6qelNVHTCNfcB0vWu6f9TM/7x0qn+0qp5+b3oCAAAAAABgTPMEZr+f5F1V9bKqelmSy5OcfU9fWFWHJzk9yfbufnyS/ZOckuTVSV7T3Y9JcmuSU6dHTk1y61R/zTQuVXXM9NzjkpyQ5Leqav972hcAAAAAAABjWjcw6+5fS/KcJLdMx3O6+7X38r1bknxtVW1J8qAkn0zyHUkumO6fm+Tk6fyk6TrT/adWVU31N3b3F7v7uiS7khx3L/sCAAAAAABgMPPsYZbufm+S9+6LF3b3DVX1K0k+keQfkvxpkiuTfLa775yG7U5y+HR+eJLrp2fvrKrbkjxsql8+89ezzwAAAAAAAMBc5lmScZ+qqoOzMjtsW5KvS/LgrCypuMh3nlZVV1TVFTfffPMiXwUAAAAAAMD9zNIDsyTfmeS67r65u7+U5M1JviXJQdMSjUlyRJIbpvMbkjwqSab7D03ymdn6Ks/8E919Vndv7+7tW7du3dffAwAAAAAAwP3YmoFZVe1fVe/Yx+/8RJLjq+pB015kT03y4STvSPLMacyOJBdO5xdN15nu/6/u7ql+SlU9oKq2JTk6ybv3ca8AAAAAAABscmvuYdbdd1XVl6vqod192754YXe/q6ouyMqeaHcmeV+Ss5L8SZI3VtUrptrZ0yNnJ/mDqtqV5JYkp0z/c3VVnZ+VsO3OJM/r7rv2RY8AAAAAAGv5q/Pum1u/fMuzrbAFcE+sGZhN/k+SD1XVpUk+f3exu0+/py/t7jOSnLFH+dokx60y9h+T/MBe/ueVSV55T/sAAAAAAACAeQKzN08HAAAAAAAAbDrrBmbdfW5VfW2SI7v7o0voCQAAAAAAAJZmv/UGVNUzkrw/ydum639VVRctujEAAAAAAABYhnUDsyQvy8reYp9Nku5+f5JHL7AnAAAAAAAAWJp5ArMvdfdte9S+vIhmAAAAAAAAYNnW3cMsydVV9cNJ9q+qo5OcnuSvF9sWAAAAAAAALMc8M8yen+RxSb6Y5A1Jbk/ygkU2BQAAAAAAAMuy7gyz7v5Ckp+rqlevXPbnFt8WAAAAAAAALMe6gVlVPSnJziQPma5vS/Jj3X3lgnsDAAAAAID7vRt/+bqNbuErPOJntm10C3CfMs8eZmcn+Ynu/t9JUlXfmuT3kzxhkY0BAAAAAADAMsyzh9ldd4dlSdLdf5nkzsW1BAAAAAAAAMuz1xlmVXXsdPoXVfU7Sd6QpJP8UJI/X3xrAAAAAAAAsHhrLcn4q3tcnzFz3gvoBQAAAAAAAJZur4FZdz9lmY0AAAAAAADARlhrhlmSpKoOSvLsJEfNju/u0xfXFgAAAAAAACzHuoFZkouTXJ7kQ0m+vNh2AAAAAAAAYLnmCcwe2N0vWngnAAAAAAAAsAH2m2PMH1TVc6vqkVV1yN3HwjsDAAAAAACAJZhnhtkdSX45yc8l6anWSR69qKYAAAAAAABgWeYJzH46yWO6+9OLbgYAAAAAAACWbZ4lGXcl+cKiGwEAAAAAAICNMM8Ms88neX9VvSPJF+8udvfpC+sKAAAAAAAAlmSewOyPpwMAAAAAAAA2nXUDs+4+dxmNAAAAAAAAwEZYNzCrquuS9J717n70QjoCAAAAAACAJZpnScbtM+cPTPIDSQ5ZTDsAAAAAAACwXPutN6C7PzNz3NDdr03y75bQGwAAAAAAACzcPEsyHjtzuV9WZpzNMzMNAAAAAAAA7vPmCb5+deb8ziQfS/KDC+kGAAAAAAAAlmzdwKy7n7KMRgAAAAAAAGAjzLMk4wOS/PskR82O7+6XL64tAAAAAAAAWI55lmS8MMltSa5M8sXFtgMAAAAAAADLNU9gdkR3n7DwTgAAAAAAAGAD7DfHmL+uqn+58E4AAAAAAABgA8wzw+xbk/xoVV2XlSUZK0l39xMW2hkAAAAAAAAswTyB2YkL7wIAAAAAAAA2yLqBWXd/fBmNAAAAAAAAwEaYZw8zAAAAAAAA2LQEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADE1gBgAAAAAAwNAEZgAAAAAAAAxNYAYAAAAAAMDQBGYAAAAAAAAMTWAGAAAAAADA0ARmAAAAAAAADG1DArOqOqiqLqiqv6mqj1TVN1fVIVV1aVVdM/0ePI2tqjqzqnZV1Qer6tiZ/9kxjb+mqnZsxLcAAAAAAABw/7ZRM8x+PcnbuvsbkzwxyUeSvCTJ27v76CRvn66T5MQkR0/HaUlelyRVdUiSM5I8OclxSc64O2QDAAAAAACAeS09MKuqhyb5N0nOTpLuvqO7P5vkpCTnTsPOTXLydH5SkvN6xeVJDqqqRyZ5epJLu/uW7r41yaVJTljipwAAAAAAALAJbMQMs21Jbk7y+1X1vqr6vap6cJLDuvuT05gbkxw2nR+e5PqZ53dPtb3VAQAAAAAAYG4bEZhtSXJsktd1979O8vn8/+UXkyTd3Ul6X72wqk6rqiuq6oqbb755X/0tAAAAAAAAm8BGBGa7k+zu7ndN1xdkJUD71LTUYqbfm6b7NyR51MzzR0y1vdW/Qnef1d3bu3v71q1b99mHAAAAAAAAcP+39MCsu29Mcn1VPXYqPTXJh5NclGTHVNuR5MLp/KIkz64Vxye5bVq68ZIkT6uqg6vq4CRPm2oAAAAAAAAwty0b9N7nJ3l9VR2Q5Nokz8lKeHd+VZ2a5ONJfnAae3GS706yK8kXprHp7luq6heTvGca9/LuvmV5nwAAAAAAAMBmsCGBWXe/P8n2VW49dZWxneR5e/mfnUl27tvuAAAAAAAAGMlG7GEGAAAAAAAA9xkCMwAAAAAAAIYmMAMAAAAAAGBoAjMAAAAAAACGJjADAAAAAABgaAIzAAAAAAAAhiYwAwAAAAAAYGgCMwAAAAAAAIYmMAMAAAAAAGBoAjMAAAAAAACGJjADACUFqekAACAASURBVAAAAABgaAIzAAAAAAAAhiYwAwAAAAAAYGgCMwAAAAAAAIYmMAMAAAAAAGBoWza6AQAAAAAAWM/HXnvjRrewqqNe8IiNbgHYB8wwAwAAAAAAYGgCMwAAAAAAAIYmMAMAAAAAAGBoAjMAAAAAAACGJjADAAAAAABgaAIzAAAAAAAAhrZloxsAAAAAAMb11jd9eqNb+Aon/tChG90CAEtmhhkAAAAAAABDE5gBAAAAAAAwNIEZAAAAAAAAQxOYAQAAAAAAMDSBGQAAAAAAAEMTmAEAAAAAADA0gRkAAAAAAABDE5gBAAAAAAAwNIEZAAAAAAAAQ9uy0Q0AAAAAwH3Fa99y40a38BVe8H2P2OgWAGDTM8MMAAAAAACAoQnMAAAAAAAAGJrADAAAAAAAgKEJzAAAAAAAABiawAwAAAAAAIChCcwAAAAAAAAYmsAMAAAAAACAoQnMAAAAAAAAGJrADAAAAAAAgKEJzAAAAAAAABiawAwAAAAAAIChCcwAAAAAAAAYmsAMAAAAAACAoQnMAAAAAAAAGJrADAAAAAAAgKEJzAAAAAAAABiawAwAAAAAAIChCcwAAAAAAAAYmsAMAAAAAACAoQnMAAAAAAAAGJrADAAAAAAA+L/t3Xu8dXVdJ/DPFx7xgqGoYAooVmrReCfCsQvJqHhJNDExL3hpGC+90iZrdKamTH2ldhl1LI1RTK0k5aKEKJHJyy5igApySUEzgTDwQmqOkfibP9bvGTbPc87zcM6zb+vZ7/frdV5n799ae+3PWWfttX57fff6bVhpCyuYVdWeVfWJqjqj379XVX2sqq6oqj+tqr16+637/Sv69IMnlvGy3v7pqnrUYv4SAAAAAAAAxmzLAp/7RUkuS7JPv/+aJP+rtXZSVb05yXOTvKn//mpr7fuq6tg+31Oq6pAkxyb5wSR3T/IXVXWf1tqN8/5DAAAAAAAAVtm1v/e+RUfYzv4vPPoWz7uQK8yq6sAkj03yln6/kjw8ycl9lrcneUK/fXS/nz79yD7/0UlOaq39W2vtH5JckeSw+fwFAAAAAAAA7C4WNSTj65L8cpLv9Pt3TnJ9a+3b/f5VSQ7otw9IcmWS9On/0uf//+1rPAYAAAAAAABukbkXzKrqcUmuba1dMMfnPL6qzq+q86+77rp5PS0AAAAAAAAjsIgrzB6W5PFV9fkkJ2UYivH1Se5YVVu/U+3AJFf321cnOShJ+vQ7JPnyZPsaj7mZ1toJrbVDW2uH7rffftP9awAAAAAAABi1uRfMWmsva60d2Fo7OMmxSf6ytfa0JB9Ockyf7bgkW78d7vR+P336X7bWWm8/tqpuXVX3SnLvJH83pz8DAAAAAACA3cSWnc8yN/8tyUlV9cokn0jy1t7+1iTvrKorknwlQ5EtrbVLqurdSS5N8u0kL2yt3Tj/2AAAAAAAAIzZQgtmrbVzkpzTb38uyWFrzPOtJE9e5/GvSvKq2SUEAAAAAABgd7eI7zADAAAAAACApaFgBgAAAAAAwEpTMAMAAAAAAGClKZgBAAAAAACw0hTMAAAAAAAAWGkKZgAAAAAAAKw0BTMAAAAAAABWmoIZAAAAAAAAK03BDAAAAAAAgJWmYAYAAAAAAMBKUzADAAAAAABgpSmYAQAAAAAAsNIUzAAAAAAAAFhpCmYAAAAAAACsNAUzAAAAAAAAVpqCGQAAAAAAACtNwQwAAAAAAICVpmAGAAAAAADASlMwAwAAAAAAYKUpmAEAAAAAALDSFMwAAAAAAABYaQpmAAAAAAAArDQFMwAAAAAAAFaaghkAAAAAAAArTcEMAAAAAACAlaZgBgAAAAAAwEpTMAMAAAAAAGClKZgBAAAAAACw0hTMAAAAAAAAWGkKZgAAAAAAAKw0BTMAAAAAAABWmoIZAAAAAAAAK03BDAAAAAAAgJW2ZdEBAAAAAIBd865Trlt0hDU99Un7LToCANwirjADAAAAAABgpSmYAQAAAAAAsNIUzAAAAAAAAFhpCmYAAAAAAACsNAUzAAAAAAAAVpqCGQAAAAAAACtNwQwAAAAAAICVpmAGAAAAAADASlMwAwAAAAAAYKUpmAEAAAAAALDSFMwAAAAAAABYaQpmAAAAAAAArDQFMwAAAAAAAFaaghkAAAAAAAArTcEMAAAAAACAlbZl0QEAAAAAAJifi064dtERtnP/4/dfdARgxbnCDAAAAAAAgJWmYAYAAAAAAMBKUzADAAAAAABgpSmYAQAAAAAAsNLmXjCrqoOq6sNVdWlVXVJVL+rtd6qqs6vq8v57395eVfWGqrqiqi6qqgdPLOu4Pv/lVXXcvP8WAAAAAAAAxm8RV5h9O8kvttYOSXJ4khdW1SFJXprkQ621eyf5UL+fJI9Ocu/+c3ySNyVDgS3JryX54SSHJfm1rUU2AAAAAAAAuKXmXjBrrV3TWvt4v/31JJclOSDJ0Une3md7e5In9NtHJ3lHG5yb5I5Vdbckj0pydmvtK621ryY5O8lRc/xTAAAAAAAA2A0s9DvMqurgJA9K8rEkd22tXdMnfTHJXfvtA5JcOfGwq3rbeu0AAAAAAABwiy2sYFZVt09ySpIXt9a+NjmttdaStCk+1/FVdX5VnX/ddddNa7EAAAAAAADsBhZSMKuqW2Uolv1xa+3U3vzPfajF9N/X9varkxw08fADe9t67dtprZ3QWju0tXbofvvtN70/BAAAAAAAgNGbe8GsqirJW5Nc1lr73YlJpyc5rt8+Lsn7JtqfWYPDk/xLH7rxrCSPrKp9q2rfJI/sbQAAAAAAAHCLbVnAcz4syTOSfKqqPtnb/nuSVyd5d1U9N8k/JvnpPu3MJI9JckWSbyZ5dpK01r5SVa9Icl6f7zdaa1+Zz58AAAAAAADA7mLuBbPW2l8nqXUmH7nG/C3JC9dZ1olJTpxeOgAAAAAAAFbNQr7DDAAAAAAAAJaFghkAAAAAAAArTcEMAAAAAACAlaZgBgAAAAAAwEpTMAMAAAAAAGClKZgBAAAAAACw0rYsOgAAAAAAu5efP+3KRUdY0xueeNCiIwAAS8oVZgAAAAAAAKw0BTMAAAAAAABWmoIZAAAAAAAAK03BDAAAAAAAgJWmYAYAAAAAAMBKUzADAAAAAABgpW1ZdAAAAAAA1vbkUy5ZdITtvOdJP7joCAAAU+cKMwAAAAAAAFaaghkAAAAAAAArzZCMAAAAU/C4U05cdITtnPGk5yw6AgAAwCi4wgwAAAAAAICVpmAGAAAAAADASlMwAwAAAAAAYKX5DjMAAABG6XEnn7zoCNs545hjFh0BAADYBFeYAQAAAAAAsNIUzAAAAAAAAFhpCmYAAAAAAACsNN9hBgDAzD36ff950RG284Gj/8+iIwAAAABLwhVmAAAAAAAArDQFMwAAAAAAAFaaghkAAAAAAAArTcEMAAAAAACAlaZgBgAAAAAAwEpTMAMAAAAAAGClKZgBAAAAAACw0rYsOgAAAACL87iT/3jREdZ0xjFPW3QEAABghbjCDAAAAAAAgJWmYAYAAAAAAMBKUzADAAAAAABgpSmYAQAAAAAAsNIUzAAAAAAAAFhpCmYAAAAAAACsNAUzAAAAAAAAVtqWRQcAZu+iNz1+0RHWdP/nn77oCAAAAAAAoGAGAAC7o8ee9luLjrCd9z/xlxYdAQAAANZkSEYAAAAAAABWmivMAAAAYI4ef/IZi46wptOPedyiI8zMT51y7qIjrOnUJx2+6AgAAHQKZgAzcNZbH7PoCGt61HPPXHQEAAAAAIClY0hGAAAAAAAAVpqCGQAAAAAAACvNkIwAAMDSeOypv7/oCGt6/0+9YNERAAAAmCEFMwAAWMdj3vuri46wpjOf8IpFRwAAAIDdiiEZAQAAAAAAWGmuMIMN+MIbjll0hO3c4+dPXnQEAAAAAAAYtZUvmF33pj9adIQ17ff8py86AgBz8gfvfNSiI2znvzzjrEVHAAAAAIC5WfmCGQC7h3f+4fIVnZLkGc9SeAIAdh9POPlDi46wnfcec+SiIwAAsBtQMAOW2t+c8LhFR9jOw44/Y9ERAAAAAACYIgUzAGBTfuddy3lV3y8+1VV9AAAAAGzM6AtmVXVUktcn2TPJW1prr15wpLm57s2/v+gI29nveS/Y6Txf/P1fm0OSjfvuF7x80RFgKZz2tqMWHWE7T3z2Bxcdgd3Mr7xn+bbzVz5559v5s09bvtxJ8rYneo0CAAAA47bHogPsiqraM8nvJXl0kkOSPLWqDllsKgAAAAAAAMZk1AWzJIcluaK19rnW2g1JTkpy9IIzAQAAAAAAMCJjL5gdkOTKiftX9TYAAAAAAAC4Raq1tugMm1ZVxyQ5qrX2s/3+M5L8cGvt57aZ7/gkx/e7903y6RlFukuSL81o2bM01tzJeLOPNXcy3uxjzZ2MN/tYcyfjzT7W3Ml4s481dzLe7GPNnYw3+1hzJ+PNPtbcyXizjzV3Mt7sY82djDf7WHMn480+1tzJeLOPNXcy3uxjzZ2MN/tYcyfjzT7W3Ml4s8869z1ba/tt27hlhk84D1cnOWji/oG97WZaayckOWHWYarq/NbaobN+nmkba+5kvNnHmjsZb/ax5k7Gm32suZPxZh9r7mS82ceaOxlv9rHmTsabfay5k/FmH2vuZLzZx5o7GW/2seZOxpt9rLmT8WYfa+5kvNnHmjsZb/ax5k7Gm32suZPxZh9r7mS82ReVe+xDMp6X5N5Vda+q2ivJsUlOX3AmAAAAAAAARmTUV5i11r5dVT+X5KwkeyY5sbV2yYJjAQAAAAAAMCKjLpglSWvtzCRnLjpHN/NhH2dkrLmT8WYfa+5kvNnHmjsZb/ax5k7Gm32suZPxZh9r7mS82ceaOxlv9rHmTsabfay5k/FmH2vuZLzZx5o7GW/2seZOxpt9rLmT8WYfa+5kvNnHmjsZb/ax5k7Gm32suZPxZl9I7mqtLeJ5AQAAAAAAYCmM/TvMAAAAAAAAYJcomO1AVZ1YVddW1cUTbQ+oqo9W1aeq6s+qap+Jaffv0y7p02/T259SVRf19tcsW/aqelpVfXLi5ztV9cA+7SF9/iuq6g1VVSPJ/aqqurKqvjHLvDPK/sGqurBvL2+uqj3HkHvisadPLmsM2avqnKr69MS0/UeSe6+qOqGqPlNVf19VT5pl7ilnf2qf/6K+zd9lJLmXfX9+q6p6e2+/rKpeNvGYo/p2fkVVvXREubdbzhiyV9VBVfXhqrq0by8vGknu21TV39VNx6GXzzr3tLL3aXesqpP7PvGyqnrosueuqvtus8/5WlW9eJa5p5W9T/uFvq1cXFXvqt7/XZLce1XV23r7hVV1xMRj5trHnXL2ufZzp5G7qm5XVe/vr81LqurVY8nepy1z/3zd3BOPXdb++Y7W+TL3z3eUe9n75+u9Rr+rbn4s+lJVvW7Zc/dpi+ifr9nPq6o7VdXZVXV5/71vb68ajjVX9KwPnljWcX3+y6vquBHl/mBVXV9VZ8wy87SzV9UD66bzdhdV1VNGkvueVfXx/vq8pKqeN8vc08w+sbx9quqqqnrjWHJX1Y11037x9FnmnkH2e1TVn9fQd7+0qg5e9txV9RN182PRt6rqCbPKPc3sfdpr+zIuqxm/v5hy7tfU8F7u4prxPnGT2b+/hv32v1XVS7ZZ1mzOc7XW/Kzzk+THkjw4ycUTbecl+fF++zlJXtFvb0lyUZIH9Pt3TrJn//2FJPv19rcnOXKZsm/zuPsl+ezE/b9LcniSSvKBJI8eSe7Dk9wtyTeWcXvZSfZ9+u9KckqSY8eQu7f9VJI/mVzWGLInOSfJoSPcVl6e5JX99h5J7jKG7Bn2l9duzZvktUl+fQS5l35/nuRnkpzUb98uyeeTHJzhePTZJN+TZK8kFyY5ZNlzr7eckazzuyV5cG//riSfGcM6z3DsuX1vv1WSjyU5fAzrvN9/e5Kf7bf3SnLHMeSeeOyeSb6Y5J5jWOdJDkjyD0lu26e9O8mzlij3C5O8rd/eP8kFSfbo9+fax51y9rn2c6eRu283P9Hb90ryVyNb50vbP99R7t62tP3znazzc7Kk/fOd5F7q/vnOtpeJx1+Q5MeWPXcW1z9fs5+X4X3NS3v7S5O8pt9+TIZjTWXYh3+st98pyef673377X2XPXefdmSSn0xyxqzX95TX+X2S3LvfvnuSazLD/uIUc++V5Nb99u0z9MXuPoZ1PrG812c4Hr1xLLkzx3OKM8h+TpJHTGwztxtD7oll3inJV2aZe5rZk/zHJH+T4f3cnkk+muSIEeR+bJKzM5yn2zvDcXifJVvn+yf5oSSvSvKSieXM7DyXK8x2oLX2kQwvzkn3SfKRfvvsJFs/MfbIJBe11i7sj/1ya+3GDP+0y1tr1/X5/mLiMTOzweyTnprkpCSpqrtleJGc24Yt8R1JZlrZn0buvpxzW2vXzCTkOqaY/Wv95pYML/g23aQ3N63cVXX7JP81yStnEHNN08o+b1PM/Zwkv9mX+Z3W2pemHHU7U8pe/Wfv/ombfZL80/TT3mRKucewP28Z1uuWJLdNckOSryU5LMkVrbXPtdZuyPA3HT2C3OstZ6amkb21dk1r7eN9eV9PclmG4sKy526tta1Xrdyq/8z0OJRMJ3tV3SHDybe39mXe0Fq7ftlzb/PYIzMU6f9xNolvMsXsW5Lctk+7XZZrf35Ikr/sj7s2yfVJDl1EH3da2fv9ufZzp5G7tfbN1tqHe/sNST6e5MAZR5/mOl/m/vm6uUfQP183+7xNMfey9893us6r6j4ZTk791Ywipz//NHIvqn++Xj/v6AxFu/TfW48tRyd5R+9nnZvkjv1Y9KgkZ7fWvtJa+2qGv/moEeROa+1DSb4+q6yzyt5a+0xr7fK+nH/K8EHO/UaQ+4bW2r/1eW6dOYwYNs3tpaoekuSuSf58TLnnbVrZq+qQJFtaa2f3ZX2jtfbNZc+9zWKPSfKBWeaecvaW5Dbpxe0M76P/eQS5D0nykdbat1tr/5rhYqCZHYc2k721dm1r7bwk/77NomZ2nkvBbOMuyU0r/8lJDuq375OkVdVZNVwm/cu9/Yok962qg/uJhCdMPGbe1ss+6SlJ3tVvH5DkqolpV2XGJ/vWsdHcy2RT2avqrAwdt68nOXmWAdexmdyvSPI7SWZ6MLsFNru9vK1f8v2rs7xsegc2lLuq7tjbXtH3Oe+pqrvOPuaaNpS9tfbvSZ6f5FMZTqwekn6Ce842uq2MYX9+cpJ/zfBJyS8k+e3W2lcy7LuvnHj8su3P18u9TDadvYbhLx6U4Wqtedtw7qras6o+meE4dHZrbRG5k41nv1eS6zLszz9RVW+pqr3nnDnZte382Cy2P7Oh7K21q5P8dm+7Jsm/tNZmfiJkDevlvjDJ46tqS1XdK8lD+rRl6eMmG8++LDadu/dhfjLJh+aUdVubyr7E/fMd5V72/vnOtpdl7Z+vmXsk/fNbsm85NsmfttZm/oGZNWw098L759v08+7abvpAwxczFAiS9fviC+uj72LuhZpW9qo6LMPJ7c/OMO7k8x2cXcjdhzG7qE9/TS/4zcWuZK+qPTIci242lNo8TGFbuU1VnV9V59aMhwbc1i5mv0+S66vq1P6+6LdqxsNJTyn3pLm/L9qV7K21jyb5cIb3RNckOau1dtkcYu/qOr8wyVE1DJ9+lyQ/kTkeR29h9vXM7BilYLZxz0nygqq6IMNlgzf09i1JfiTJ0/rvJ1bVkf1TQs9P8qcZPqH1+SQ3zjt0t172JElV/XCSb7bW5vY9MbfQWHMnm8zeWntUhktUb53k4XPKOmlDuWv4fqfvba2dNvek29vMOn9aa+1+SX60/zxjXmEnbDT3lgyfzP7b1tqDM1zu/dtzzDtpo9vLrTLsFx+UYRiMi5K8LPO3odwj2Z8f1jPdPUPx4Ber6nsWE3FNY82dbDJ7/3T/KUle3G66QmGeNpy7tXZja+2BGfYxh1XVf5h/7CQbz74lw9BOb2qtPShDgWfm39e3hs1uK3sleXyS98w37s1sKHsN48of3dvunuEqtKfPP/a6uU/M8Mbp/CSvS/K3Wdx+ez1jzb6p3P2E9ruSvKG19rm5Jr7JprIvcf98zdwj6Z/vaJ0vc/98vdxj6J/fkn3LIj+8saHci+6f76if1wuOiyg67tRYcyfTy17DlRXvTPLs1tp3ph50++fb5dyttStba/dP8n1JjptXQX4K2V+Q5MzW2lU7mW+qprSt3LO1dmiG4clfV1XfO/2k25tC9i0Zjp0vyTCU3fckedb0k97clF+f90ty1tRDrv+cu5S9qr4vyQ9k6AcckOThVfWjM4o7+by7lLt/2PHMDMfVd2Xou8zlOLrMx6Iti3risWqt/X2G4Re3DlXw2D7pqgyXMH6pTzszw0mbD7XW/izJn/X247OgN7s7yL7Vth3jq3PzoVIO7G1ztYncS2NXsrfWvlVV78twIursWeZc47k3mvuhGYY4+nyG/cr+VXVOa+2I2ae9uc2s8/4J+bTWvl5Vf5LhxOA7Zp/2Zhk2mvvLGT4tfGq//54kz51xzDVtIvsD++M+2x/z7izgpPYmt5Vl35//TJIP9qv4rq2qv8kwXM2VufmnhJZtf75e7kWdTN3OZrL34vApSf64tXbqGouduV1Z562166vqwxmGZJj7h1I2kf0jSa6auCLu5CzXvmVn6/zRST7eWpvZ0B07s4nsLck/tD4UVlWdmmH8/j9ahtyttW8n+YWt81XV32YYI/+rWYI+brKp7EthF3KfkGH4tNfNL+3N7co6X8b++Q5y/3iWvH++o3W+zP3zHeRe+v75zrbzqnpAhmG8Lphr4G6T28pC+ufr9PP+uYbh867pJ3yv7e1XZ+2++NVJjtim/ZwR5F6IaWWvqn2SvD/J/2jD0GSjyL1Va+2fquriDAWRmV7xPKXsD03yo1X1ggzfpbVXVX2jtTazfvq01vnEsehzVXVOhg/9zvSKxCll35Lkk1s/nFRV783wvVUzG9lnytv5Tyc5rb/3mLkpZX96knNb/3qDqvpAhm1/ZsMbT3E7f1WG7wdL73PN/D3HBrOvZ2bHKFeYbVBV7d9/75HkV5K8uU86K8n9ariEcUuGNyiXbvOYfTN8suIt8869TY5ts29t++nc/Lu0rsnwnSCHV1UleWaS9801dDaee5lsNHtV3b5uGuN5S4Y3CH8/z8z9uTe6rbyptXb31trBGa6w/Mwi3oz3fBtd51tquOx46w77cVnAieFNrPOW4Y3hEb3pyPR9zrxt4jV6dZJDqmrrWPGPyDBm8VxtZt8ygv35F9I/9V7DUHSHZ9iHnJfk3lV1rxquYjk2yekjyL00Npq9HzffmuSy1trvzj/xYBO596s+pFRV3TbD63Mh/4uNZm+tfTHJlVV13z7fQvaLu7CdPzUL/gDQJrJ/Icnhvf9bGdb50uzPe669++1HJPl2a+3SZenjbib7IjKuZTO5q+qVSe6Q5MULCd1tNPuy9893sJ0vff98B+t8qfvnO1jnS98/vwX7loUeiza5b5l7/3wH/bzTkxzXbx+Xm44tpyd5Zg0OzzCE8TUZziM9sqr27fkfmRleUTHF3HM3rez9vdBpGb7LZ+bD604x94G9b751W/+RJJ8eQ/bW2tNaa/fox6OXZFj3syyWTWud71tVt+7LvEuSh2XG+/QpvkbPy/AdVVvPuTx8ltlnsG+Z27Foitm/kOTHex/mVhnqAjN7XzTF7XzPqrpzX+b9k9w/M/6uwU1kX8/sznO11vys85PhxXlNhi+VuyrDp8NelKHS+pkkr05SE/M/PcOY2xcnee02y7m0/xy7pNmPyFAJ33Y5h/a/57NJ3jj5mCXP/dr++O/0378+hnWeYXzW8zIMUXdxkv+d4RN+S517m+UdnOTisWznSfZOckFf55ckeX2SPZc9d2+/Z4YrKi7K8D0g9xjDOu/tz8vQebgow4mFO48k91LvzzN8Yu89fVu+NMkvTSznMX3+z2b4NOVYcm+3nDFkz/AmtvVt/JP95zEjyH3/JJ/ITceh/zmy7fyBGYZruijJe5PsO5Lce2e4MuEO81jfU87+8gzFg4szDG106yXKfXCGk0mXJfmLDMPqbF3OXPu4U84+137uNHJn+MRn6+1b94k/O4Z1niXvn+9oW5lY3sFZwv75Dtb5UvfPd/L6XOr++c62lwxXPX//WLaVieXMu3++Zj8vyZ37//3ynvNOff5K8nsZjjmfSnLoxLKek+G72K7IMDzgWHL/VYbvjv2//f/3qDFkz3De7t8nlvHJJA8cQe5H9GVc2H8fP6btfGKZz0ryxjHkzjBqwqf6Ov9U5vM+dJqv0a3bzKeS/GGSvUaS++AMH7LeY9bre8rby55J/iDDcerSJL87kty3yU3H0HMzw/3hLmT/7gzHma8lub7f3qdPm8l5rq2dDwAAAAAAAFhJhmQEAAAAAABgpSmYAQAAAAAAsNIUzAAAAAAAAFhpCmYAAAAAAACsNAUzAAAAAAAAVpqCGQAAAAAAACtNwQwAAIDtVNWei84AAAAwLwpmAAAAI1dVv1FVL564/6qqelFV/VJVnVdVF1XVyyemv7eqLqiqS6rq+In2b1TV71TVhUkeOuc/AwAAYGEUzAAAAMbvxCTPTJKq2iPJsUm+mOTeSQ5L8sAkD6mqH+vzP6e19pAkhyb5+aq6c2/fO8nHWmsPaK399Tz/AAAAgEXasugAAAAA7JrW2uer6stV9aAkd03yiSQ/lOSR/XaS3D5DAe0jGYpkT+ztB/X2Lye5Mckp88wOAACwDBTMAAAAdg9vSfKsJN+d4YqzI5P8ZmvtDyZnqqojkvynJA9trX2zqs5Jcps++VuttRvnFRgAAGBZGJIRAABg93BakqMyXFl2Vv95TlXdPkmq6oCq2j/JHZJ8tRfLvj/J4YsKDAAAsCxcYQYAALAbaK3dUFUfuKjcfQAAAKRJREFUTnJ9v0rsz6vqB5J8tKqS5BtJnp7kg0meV1WXJfl0knMXlRkAAGBZVGtt0RkAAADYRVW1R5KPJ3lya+3yRecBAAAYE0MyAgAAjFxVHZLkiiQfUiwDAADYOFeYAQAAAAAAsNJcYQYAAAAAAMBKUzADAAAAAABgpSmYAQAAAAAAsNIUzAAAAAAAAFhpCmYAAAAAAACsNAUzAAAAAAAAVtr/A4WQyWV7pq0zAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 2160x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create the plot\n",
    "\n",
    "# Set the figure size\n",
    "plt.figure(figsize = (30, 10))\n",
    "\n",
    "sns.barplot(x = count.index,\n",
    "            y = 'title',\n",
    "            data = count,\n",
    "            estimator = np.median)\n",
    "\n",
    "# Set the y label of the plot\n",
    "plt.ylabel('number of titles played') \n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_O-6R_8Nc-fD"
   },
   "source": [
    "#### **Observations and Insights:** # \n",
    "\n",
    "* Our dataset has songs from **1969 to 2010**.\n",
    "* The **year 2009** has the highest count of the number of titles played **(~16,000)**. \n",
    "* Followed by the year **2008 and 2007**, which have around **14,000 titles played**.\n",
    "* The **year 2010** has few titles played **(~4000)**, it may be that the data was taken up to the first quarter of the year."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nOmVPd4UdBiY"
   },
   "source": [
    "## **Checking the distribution of play_count**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 294
    },
    "id": "bicFcFtOdEJ3",
    "outputId": "9edcbbf9-b559-419f-d379-88c9dc172593"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuEAAAEVCAYAAABOqzsAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de9RdVXnv8e+PmyICAYkpEmioZejAW4AItN6xYkAF6vFaK5Gi0YoWT20reqxY1BbboxbU6uFgJFErIt6iojRFwMsQIdxveogUSiiSSLiJAgWf88eekU18Lyske++XN9/PGHvstZ4151rP3uyhzzsz11ypKiRJkiQNz2ajTkCSJEna1FiES5IkSUNmES5JkiQNmUW4JEmSNGQW4ZIkSdKQWYRLkiRJQ2YRLkmSJA2ZRbgkTSLJ/UkuSXJlkkuTvD3JZu3YvCQnTtB3TpI/meD445Kc3rZfl+Rj65nb65I8rm//5CR7rs851vN6j0jy7+37eOV65Lhen2uUksxNcvCo85A0vW0x6gQk6WHgV1U1FyDJY4F/BbYDjq2q5cDyCfrOAf6k9XmQJFtU1X8BL9uA3F4HXAH8F0BVvX4DztXFXu06cwd8nVGaC8wDzhh1IpKmL0fCJWk9VNUqYCHwlvQ8N8k3AJI8p40QX5Lk4iTbAscDz2qx/9lGhZcm+Q5wVhspv6LvErsmOSfJNUmObed9UJskf5XkvUleRq9Y/Fw7/9at77zW7tVJLk9yRZIP9vX/RZIPtFH985LMWvdzJtkxyVeTXNbaPLX9AfJZ4Onteo9fp885SU5ox65Isu8Y531Jkh+17+ffk8xKsln7vDNbm82SrFi7P8Y5ZiX5Ssv/0iR/2OJ/2a57RZK3TfTd9eX7wSTnJ/l/SZ6VZCvgOOCV6zPaL0nryyJcktZTVV0LbA48dp1DfwUc1UaJnwX8CjgG+F5Vza2qj7R2ewMvq6rnjHH6fYH/ATwVePnagnqcPE6nNwr/mnb+X6091qaofBA4gN7I7tOTHNYObwOcV1VPA74LvGGM0/8dcHFVPRV4F7Ck/QHy+r7P89Mx+j2qff43A4vGOP59YP+q2gs4Ffibqvo1veL+Na3NHwGXVtXqcT76icC5Lf+9gSuT7AMcAewH7A+8Icle4/Tvt0VV7Qu8jd6/bNwLvAf4QvuMX+hwDklabxbhkrTx/AD4cJK/AGZU1X3jtFtWVWsmOHZLK6i/DDzzIebydOCcqlrd8vgc8Ox27F7gG237QnpTZtb1TOAzAFX1HeAxSbbrcN3Ptz7fBbZLMmOd47OBM5NcDvw18KQWXwQc3rb/DPj0BNc4APhEu879VXV7y/crVXVXVf2C3nf3rA75frm9j/c9SNJAWIRL0npK8nvA/cCq/nhVHU9vpHhr4AdJnjjOKe6a4PQ1xv59PPh/rx+5Xgn/tv+uqrXXuZ+Ne3/QWPn3+yjwsap6CvBG2mepqhuAm5McQO9fA761kfKZ7Lu7p71v7O9BkiZkES5J66HNU/4kvUKy1jn2+Kq6vKo+CFwAPBG4E9h2PS7xgjYfe2vgMHqj6zcDj03ymCSPAF7c1368858PPCfJTkk2B14NnLseeXyPNj0kyXOBn1fVHR36vbL1eSZwexul7rc9cGPbXrDOsZPpTUv5YlXdP8E1zgL+vF1n8yTbt3wPS/KoJNsAf9xiE31341nf/2aStN78q1+SJrd1kkuALemNrH4G+PAY7d6W5HnAr4Er6Y3m/hq4P8mlwCnArZNc63zgS/SmbXy2rb5CkuPasRuBH/e1PwX4ZJJfAX+wNlhVNyU5BjgbCPDNqvraenzm9wKLklwG/JLfLpjHc3eSi+l9V382znm/mORW4DvA7n3HltKbhjLRVBSAo4GTkhxJbwT7z6vqh0lOofcdAZxcVRfDhN/deM4Gjmn/zf/BeeGSBiHrDORIkvSQJDkH+Ku1fzg8hP7zgI9UVZe53JL0sOZIuCRp5Nqo/Z/zwAopkjStORIuSZqSkvwv4OXrhL9YVR8YRT6StDFZhEuSJElDtslNR9lpp51qzpw5o05DkiRJ09yFF17486oa8+m/m1wRPmfOHJYvf0j3DEmSJEmdJbl+vGOuEy5JkiQNmUW4JEmSNGQW4ZIkSdKQWYRLkiRJQ2YRLkmSJA2ZRbgkSZI0ZBbhkiRJ0pBZhEuSJElDZhEuSZIkDdkm98TMh2qfv14y6hTU0YX/dPioU5AkSZqQI+GSJEnSkFmES5IkSUNmES5JkiQN2UCL8CRPSHJJ3+uOJG9LsmOSZUmuae87tPZJcmKSFUkuS7J337kWtPbXJFnQF98nyeWtz4lJMsjPJEmSJG2ogRbhVfWTqppbVXOBfYBfAl8BjgHOqqo9gLPaPsBBwB7ttRD4BECSHYFjgf2AfYFj1xburc0b+vrNH+RnkiRJkjbUMKejPB/4aVVdDxwKLG7xxcBhbftQYEn1nAfMSLIz8EJgWVWtqapbgWXA/HZsu6o6r6oKWNJ3LkmSJGlKGmYR/irg8217VlXd1LZ/Bsxq27sAN/T1WdliE8VXjhF/kCQLkyxPsnz16tUb+jkkSZKkDTKUIjzJVsAhwBfXPdZGsGuQ16+qk6pqXlXNmzlz5iAvJUmSJE1qWCPhBwEXVdXNbf/mNpWE9r6qxW8Edu3rN7vFJorPHiMuSZIkTVnDKsJfzQNTUQCWAmtXOFkAfK0vfnhbJWV/4PY2beVM4MAkO7QbMg8EzmzH7kiyf1sV5fC+c0mSJElT0sAfW59kG+AFwBv7wscDpyU5ErgeeEWLnwEcDKygt5LKEQBVtSbJ+4ALWrvjqmpN234zcAqwNfCt9pIkSZKmrIEX4VV1F/CYdWK30FstZd22BRw1znkWAYvGiC8HnrxRkpUkSZKGwCdmSpIkSUNmES5JkiQNmUW4JEmSNGQW4ZIkSdKQWYRLkiRJQ2YRLkmSJA2ZRbgkSZI0ZBbhkiRJ0pBZhEuSJElDZhEuSZIkDZlFuCRJkjRkFuGSJEnSkFmES5IkSUNmES5JkiQNmUW4JEmSNGQW4ZIkSdKQWYRLkiRJQ2YRLkmSJA2ZRbgkSZI0ZAMvwpPMSHJ6kh8nuTrJHyTZMcmyJNe09x1a2yQ5McmKJJcl2bvvPAta+2uSLOiL75Pk8tbnxCQZ9GeSJEmSNsQwRsJPAL5dVU8EngZcDRwDnFVVewBntX2Ag4A92msh8AmAJDsCxwL7AfsCx64t3FubN/T1mz+EzyRJkiQ9ZAMtwpNsDzwb+BRAVd1bVbcBhwKLW7PFwGFt+1BgSfWcB8xIsjPwQmBZVa2pqluBZcD8dmy7qjqvqgpY0ncuSZIkaUoa9Ej47sBq4NNJLk5ycpJtgFlVdVNr8zNgVtveBbihr//KFpsovnKM+IMkWZhkeZLlq1ev3ggfS5IkSXroBl2EbwHsDXyiqvYC7uKBqScAtBHsGmQSVXVSVc2rqnkzZ84c5KUkSZKkSQ26CF8JrKyqH7X90+kV5Te3qSS091Xt+I3Arn39Z7fYRPHZY8QlSZKkKWugRXhV/Qy4IckTWuj5wFXAUmDtCicLgK+17aXA4W2VlP2B29u0lTOBA5Ps0G7IPBA4sx27I8n+bVWUw/vOJUmSJE1JWwzhGm8FPpdkK+Ba4Ah6xf9pSY4Ergde0dqeARwMrAB+2dpSVWuSvA+4oLU7rqrWtO03A6cAWwPfai9JkiRpyhp4EV5VlwDzxjj0/DHaFnDUOOdZBCwaI74cePIGpilJkiQNjU/MlCRJkobMIlySJEkaMotwSZIkacgswiVJkqQh61SEJzmrS0ySJEnS5CZcHSXJI4FHATu19bnTDm3HGI+HlyRJkjS5yZYofCPwNuBxwIU8UITfAXxsgHlJkiRJ09aERXhVnQCckOStVfXRIeUkSZIkTWudHtZTVR9N8ofAnP4+VbVkQHlJkiRJ01anIjzJZ4DHA5cA97dwARbhkiRJ0nrq+tj6ecCe7bHykiRJkjZA13XCrwB+Z5CJSJIkSZuKriPhOwFXJTkfuGdtsKoOGUhWkiRJ0jTWtQh/7yCTkCRJkjYlXVdHOXfQiUiSJEmbiq6ro9xJbzUUgK2ALYG7qmq7QSUmSZIkTVddR8K3XbudJMChwP6DSkqSJEmazrqujvIb1fNV4IUDyEeSJEma9rpOR3lp3+5m9NYNv3sgGUmSJEnTXNfVUV7St30fcB29KSmSJEmS1lPXOeFHPNQLJLkOuJPe4+7vq6p5SXYEvgDMoVfQv6Kqbm3zzU8ADgZ+Cbyuqi5q51kAvLud9v1VtbjF9wFOAbYGzgCO9smekiRJmso6zQlPMjvJV5Ksaq8vJZm9Htd5XlXNrap5bf8Y4Kyq2gM4q+0DHATs0V4LgU+06+8IHAvsB+wLHJtkh9bnE8Ab+vrNX4+8JEmSpKHremPmp4GlwOPa6+st9lAdCixu24uBw/riS9rNn+cBM5LsTO8m0GVVtaaqbgWWAfPbse2q6rw2+r2k71ySJEnSlNS1CJ9ZVZ+uqvva6xRgZse+BfxbkguTLGyxWVV1U9v+GTCrbe8C3NDXd2WLTRRfOUb8QZIsTLI8yfLVq1d3TFuSJEkajK5F+C1J/jTJ5u31p8AtHfs+s6r2pjfV5Kgkz+4/2EawBzqHu6pOqqp5VTVv5syufztIkiRJg9G1CP8z4BX0Rq1vAl4GdLpZs6pubO+rgK/Qm9N9c5tKQntf1ZrfCOza1312i00Unz1GXJIkSZqyOhXhVXV9VR1SVTOr6rFVdVhV/edk/ZJsk2TbtdvAgcAV9OaXL2jNFgBfa9tLgcPTsz9we5u2ciZwYJId2g2ZBwJntmN3JNm/raxyeN+5JEmSpCmp6+ooi5PM6NvfIcmiDl1nAd9PcilwPvDNqvo2cDzwgiTXAH/U9qG3xOC1wArg/wJvBqiqNcD7gAva67gWo7U5ufX5KfCtLp9JkiRJGpWuD+t5alXdtnanrem912Sdqupa4GljxG8Bnj9GvICjxjnXIuC3Cv+qWg48ebJcJEmSpKmi65zwzfrW5V67bnfXAl6SJElSn66F9IeAHyb5Ytt/OfCBwaQkSZIkTW9dH1u/JMly4IAWemlVXbX2eJId2kN0JEmSJE2i85SSVnRfNc7hs4C9N0pGkiRJ0jTXdU74ZLKRziNJkiRNexurCB/oEy8lSZKk6WRjFeGSJEmSOnI6iiRJkjRkXZ+Y+aEkT5qgyW89eEeSJEnS2LqOhF8NnJTkR0nelGT7/oN9j5CXJEmSNIlORXhVnVxVzwAOB+YAlyX51yTPG2RykiRJ0nTUeU54ks2BJ7bXz4FLgb9McuqAcpMkSZKmpU4P60nyEeDFwHeAv6+q89uhDyb5yaCSkyRJkqajrk/MvAx4d1XdNcaxfTdiPpIkSdK016kIr6pPJ9mhrZDyyL74d6vq9oFlJ0mSJE1DXaejvB44GpgNXALsD/wQOGBwqUmSJEnTU9cbM48Gng5cX1XPA/YCbhtYVpIkSdI01rUIv7uq7gZI8oiq+jHwhMGlJUmSJE1fXW/MXJlkBvBVYFmSW4HrB5eWJEmSNH11fVjPH1fVbVX1XuBvgU8Bh3W9SJLNk1yc5Bttf/f29M0VSb6QZKsWf0TbX9GOz+k7xztb/CdJXtgXn99iK5Ic0zUnSZIkaVQmLMKT7LjuC7gc+D7w6PW4ztHA1X37HwQ+UlW/D9wKHNniRwK3tvhHWjuS7Am8CngSMB/4l1bYbw58HDgI2BN4dWsrSZIkTVmTjYRfCCxv7+u+lne5QJLZwIuAk9t+6K2qcnprspgHRtUPbfu0489v7Q8FTq2qe6rqP4AV9NYn3xdYUVXXVtW9wKmtrSRJkjRlTTgnvKp23wjX+Gfgb4Bt2/5jgNuq6r62vxLYpW3vAtzQrn1fkttb+12A8/rO2d/nhnXi+22EnCVJkqSB6XpjJkleCjwTKOB7VfXVDn1eDKyqqguTPPchZ7mBkiwEFgLstttuo0pDkiRJAjremJnkX4A30ZsPfgXwpiQf79D1GcAhSa6jN1XkAOAEYEaStX8AzAZubNs3Aru2a24BbA/c0h9fp8948QepqpOqal5VzZs5c2aHtCVJkqTB6bpO+AHAC6vq01X1aeBgOjwts6reWVWzq2oOvRsrv1NVrwHOBl7Wmi0Avta2l7Z92vHvVFW1+Kva6im7A3sA5wMXAHu01Va2atdY2vEzSZIkSSPRdTrKCmA3HlgbfNcWe6jeAZya5P3AxfSWPKS9fybJCmANvaKaqroyyWnAVcB9wFFVdT9AkrcAZwKbA4uq6soNyEuSJEkauK5F+LbA1UnOpzcnfF9geZKlAFV1yGQnqKpzgHPa9rXtHOu2uRt4+Tj9PwB8YIz4GcAZHT+HJEmSNHJdi/D3DDQLSZIkaRPSqQivqnMHnYgkSZK0qeh6Y6YkSZKkjcQiXJIkSRqyCYvwJGe19w8OJx1JkiRp+ptsTvjOSf6Q3gN3TgXSf7CqLhpYZpIkSdI0NVkR/h7gb+k9ifLD6xwrOjywR5IkSdKDTViEV9XpwOlJ/raq3jeknCRJkqRpresShe9Lcgjw7BY6p6q+Mbi0JEmSpOmr0+ooSf4BOJreY+OvAo5O8veDTEySJEmarro+MfNFwNyq+jVAksXAxcC7BpWYJEmSNF2tzzrhM/q2t9/YiUiSJEmbiq4j4f8AXJzkbHrLFD4bOGZgWUmSJEnTWNcbMz+f5Bzg6S30jqr62cCykiRJkqaxriPhVNVNwNIB5iJJkiRtEtZnTrgkSZKkjcAiXJIkSRqySYvwJJsn+fEwkpEkSZI2BZMW4VV1P/CTJLsNIR9JkiRp2ut6Y+YOwJVJzgfuWhusqkMGkpUkSZI0jXUtwv/2oZw8ySOB7wKPaNc6vaqOTbI7cCrwGOBC4LVVdW+SRwBLgH2AW4BXVtV17VzvBI4E7gf+oqrObPH5wAnA5sDJVXX8Q8lVkiRJGpZON2ZW1bnAdcCWbfsC4KIOXe8BDqiqpwFzgflJ9gc+CHykqn4fuJVecU17v7XFP9LakWRP4FXAk4D5wL+0ueqbAx8HDgL2BF7d2kqSJElTVqciPMkbgNOB/9NCuwBfnaxf9fyi7W7ZXgUc0M4HsBg4rG0f2vZpx5+fJC1+alXdU1X/AawA9m2vFVV1bVXdS290/dAun0mSJEkala5LFB4FPAO4A6CqrgEe26VjG7G+BFgFLAN+CtxWVfe1JivpFfW09xvaNe4Dbqc3ZeU38XX6jBdfN4eFSZYnWb569eouaUuSJEkD07UIv6eNNAOQZAt6I9qTqqr7q2ouMJveyPUT1zvLDVRVJ1XVvKqaN3PmzGFfXpIkSXqQrkX4uUneBWyd5AXAF4Gvr8+Fquo24GzgD4AZrZCHXnF+Y9u+EdgVflPob0/vBs3fxNfpM15ckiRJmrK6FuHHAKuBy4E3AmcA756sU5KZSWa07a2BFwBX0yvGX9aaLQC+1raXtn3a8e9UVbX4q5I8oq2ssgdwPr0bRPdIsnuSrejdvLm042eSJEmSRqLTEoVV9eski4Ef0ZuG8pNWHE9mZ2BxW8VkM+C0qvpGkquAU5O8H7gY+FRr/yngM0lWAGvoFdVU1ZVJTgOuAu4DjmoPESLJW4Az6S1RuKiqruzymSRJkqRR6VSEJ3kR8El6N1UG2D3JG6vqWxP1q6rLgL3GiF9Lb374uvG7gZePc64PAB8YI34GvZF5SZIk6WGh68N6PgQ8r6pWACR5PPBNYMIiXJIkSdJv6zon/M61BXhzLXDnAPKRJEmSpr0JR8KTvLRtLk9yBnAavTnhL6d3U6QkSZKk9TTZdJSX9G3fDDynba8Gth5IRpIkSdI0N2ERXlVHDCsRSZIkaVPRdXWU3YG3AnP6+1TVIYNJS5IkSZq+uq6O8lV6a3h/Hfj14NKRJEmSpr+uRfjdVXXiQDORJEmSNhFdi/ATkhwL/Btwz9pgVV00kKwkSZKkaaxrEf4U4LXAATwwHaXaviRJkqT10LUIfznwe1V17yCTkSRJkjYFXZ+YeQUwY5CJSJIkSZuKriPhM4AfJ7mAB88Jd4lCbdL+87injDoFdbTbey4fdQqSJP1G1yL82IFmIUmSJG1COhXhVXXuoBORJEmSNhVdn5h5J73VUAC2ArYE7qqq7QaVmCRJkjRddR0J33btdpIAhwL7DyopSZIkaTrrujrKb1TPV4EXDiAfSZIkadrrOh3lpX27mwHzgLsHkpEkSZI0zXUdCX9J3+uFwJ30pqRMKMmuSc5OclWSK5Mc3eI7JlmW5Jr2vkOLJ8mJSVYkuSzJ3n3nWtDaX5NkQV98nySXtz4ntukykiRJ0pTVdU74EQ/x/PcBb6+qi5JsC1yYZBnwOuCsqjo+yTHAMcA7gIOAPdprP+ATwH5JdqS3TOI8ejeIXphkaVXd2tq8AfgRcAYwH/jWQ8xXkiRJGrgJi/Ak75ngcFXV+ybqX1U3ATe17TuTXA3sQm8U/bmt2WLgHHpF+KHAkqoq4LwkM5Ls3Nouq6o1La9lwPwk5wDbVdV5Lb4EOAyLcEmSJE1hk42E3zVGbBvgSOAxwIRFeL8kc4C96I1Yz2oFOsDPgFltexfghr5uK1tsovjKMeKSJEnSlDVhEV5VH1q73aaTHA0cAZwKfGi8futK8mjgS8DbquqO/mnbVVVJatzOG0GShcBCgN12222Ql5IkSZImNemNme0myvcDl9Er2veuqndU1aouF0iyJb0C/HNV9eUWvrlNM6G9rz3XjcCufd1nt9hE8dljxB+kqk6qqnlVNW/mzJld0pYkSZIGZsIiPMk/ARfQWw3lKVX13nYzZCdtpZJPAVdX1Yf7Di0F1q5wsgD4Wl/88LZKyv7A7W3aypnAgUl2aCupHAic2Y7dkWT/dq3D+84lSZIkTUmTzQl/O3AP8G7gf/VNIwm9mSSTPbb+GcBrgcuTXNJi7wKOB05LciRwPfCKduwM4GBgBfBLelNfqKo1Sd5H7w8CgOPW3qQJvBk4Bdia3g2Z3pQpSZKkKW2yOeHr/UTNdfp/n17BPpbnj9G+gKPGOdciYNEY8eXAkzcgTUmSJGmoNqjIliRJkrT+LMIlSZKkIbMIlyRJkobMIlySJEkaMotwSZIkacgswiVJkqQhswiXJEmShswiXJIkSRoyi3BJkiRpyCzCJUmSpCGzCJckSZKGzCJckiRJGjKLcEmSJGnILMIlSZKkIbMIlyRJkobMIlySJEkaMotwSZIkacgswiVJkqQhswiXJEmShswiXJIkSRqygRbhSRYlWZXkir7YjkmWJbmmve/Q4klyYpIVSS5LsndfnwWt/TVJFvTF90lyeetzYpIM8vNIkiRJG8MWAz7/KcDHgCV9sWOAs6rq+CTHtP13AAcBe7TXfsAngP2S7AgcC8wDCrgwydKqurW1eQPwI+AMYD7wrQF/Jkma0DM++oxRp6COfvDWH4w6BUmbqIGOhFfVd4E164QPBRa37cXAYX3xJdVzHjAjyc7AC4FlVbWmFd7LgPnt2HZVdV5VFb1C/zAkSZKkKW4Uc8JnVdVNbftnwKy2vQtwQ1+7lS02UXzlGPHfkmRhkuVJlq9evXrDP4EkSZK0AUZ6Y2Ybwa4hXOekqppXVfNmzpw56MtJkiRJExpFEX5zm0pCe1/V4jcCu/a1m91iE8VnjxGXJEmSprRRFOFLgbUrnCwAvtYXP7ytkrI/cHubtnImcGCSHdpKKgcCZ7ZjdyTZv62KcnjfuSRJkqQpa6CroyT5PPBcYKckK+mtcnI8cFqSI4HrgVe05mcABwMrgF8CRwBU1Zok7wMuaO2Oq6q1N3u+md4KLFvTWxXFlVEkSZI05Q20CK+qV49z6PljtC3gqHHOswhYNEZ8OfDkDclRkiRJGjafmClJkiQNmUW4JEmSNGQW4ZIkSdKQWYRLkiRJQ2YRLkmSJA2ZRbgkSZI0ZANdolCSJPWc++znjDoFdfSc75476hS0CXAkXJIkSRoyi3BJkiRpyJyOIkmSNCIfe/vXR52COnrLh16yUc/nSLgkSZI0ZBbhkiRJ0pBZhEuSJElDZhEuSZIkDZlFuCRJkjRkFuGSJEnSkFmES5IkSUNmES5JkiQNmUW4JEmSNGQW4ZIkSdKQTYsiPMn8JD9JsiLJMaPOR5IkSZrIw74IT7I58HHgIGBP4NVJ9hxtVpIkSdL4HvZFOLAvsKKqrq2qe4FTgUNHnJMkSZI0rlTVqHPYIEleBsyvqte3/dcC+1XVW/raLAQWtt0nAD8ZeqJT007Az0edhKYcfxcai78LjcXfhcbi7+IBv1tVM8c6sMWwMxmFqjoJOGnUeUw1SZZX1bxR56Gpxd+FxuLvQmPxd6Gx+LvoZjpMR7kR2LVvf3aLSZIkSVPSdCjCLwD2SLJ7kq2AVwFLR5yTJEmSNK6H/XSUqrovyVuAM4HNgUVVdeWI03q4cIqOxuLvQmPxd6Gx+LvQWPxddPCwvzFTkiRJeriZDtNRJEmSpIcVi3BJkiRpyCzCN0FJFiVZleSKUeeiqSPJrknOTnJVkiuTHD3qnDR6SR6Z5Pwkl7bfxd+NOidNHUk2T3Jxkm+MOhdNDUmuS3J5kkuSLB91PlOZc8I3QUmeDfwCWFJVTx51PpoakuwM7FxVFyXZFrgQOKyqrhpxahqhJAG2qapfJNkS+D5wdFWdN+LUNAUk+UtgHrBdVb141Plo9JJcB8yrKh/WMwlHwjdBVfVdYM2o89DUUlU3VdVFbftO4Gpgl9FmpVGrnl+03S3by9EbkWQ28CLg5FHnIj0cWYRL+i1J5gB7AT8abSaaCtqUg0uAVcCyqvJ3IYB/Bv4G+PWoE9GUUsC/JbkwycJRJzOVWYRLepAkjwa+BLytqu4YdT4avaq6v6rm0nsi8b5JnMa2iUvyYmBVVV046lw05TyzqvYGDgKOalNgNQaLcEm/0eb8fgn4XFV9edT5aGqpqtuAs4H5o85FI/cM4JA2//dU4IAknx1tSpoKqurG9r4K+Aqw72gzmroswiUBv7kB71PA1VX14VHno6khycwkM9r21sALgB+PNiuNWlW9s6pmV9Uc4FXAd6rqT0eclkYsyTbtxn6SbAMcCLgS2zgswjdBST4P/BB4QpQmdQgAAALnSURBVJKVSY4cdU6aEp4BvJbeiNYl7XXwqJPSyO0MnJ3kMuACenPCXY5O0lhmAd9PcilwPvDNqvr2iHOaslyiUJIkSRoyR8IlSZKkIbMIlyRJkobMIlySJEkaMotwSZIkacgswiVJkqQhswiXJEmShswiXJIexpLc39Z0vyLJF5M8qsV/Merc1keSd406B0kaJotwSXp4+1VVza2qJwP3Am8adUIPkUW4pE2KRbgkTR/fA36/P5Dk0UnOSnJRksuTHNrixyV5W1+7DyQ5erwTJ3lH639pkuNbbG6S85JcluQrSXZo8XOSzGvbOyW5rm2/LsmXk3w7yTVJ/rHFjwe2biP6n9uo34gkTVEW4ZI0DSTZAjgIuHydQ3cDf1xVewPPAz6UJMAi4PDWdzPgVcBnxzn3QcChwH5V9TTgH9uhJcA7quqp7brHdkh1LvBK4CnAK5PsWlXH8MCI/mu6fmZJejjbYtQJSJI2yNZJLmnb3wM+tc7xAH+f5NnAr4FdgFlVdV2SW5LsBcwCLq6qW8a5xh8Bn66qXwJU1Zok2wMzqurc1mYx8MUO+Z5VVbcDJLkK+F3ghk6fVJKmEYtwSXp4+1VVzZ3g+GuAmcA+VfXfbWrII9uxk4HXAb9Db2R8Y7mPB/6l9ZHrHLunb/t+/P8hSZsop6NI0vS2PbCqFeDPozfyvNZXgPnA04EzJzjHMuCIvpVXdmyj2bcmeVZr81pg7aj4dcA+bftlHfP87yRbdmwrSQ97jkBI0vT2OeDrSS4HlgM/Xnugqu5NcjZwW1XdP94JqurbSeYCy5PcC5xBbzWTBcAnW3F+LXBE6/K/gdOSLAS+2THPk4DLklzkvHBJm4JU1ahzkCSNQLsh8yLg5VV1zajzkaRNidNRJGkTlGRPYAW9GyUtwCVpyBwJlyQBkOQpwGfWCd9TVfuNIh9Jms4swiVJkqQhczqKJEmSNGQW4ZIkSdKQWYRLkiRJQ2YRLkmSJA3Z/wcjuleWbEi8awAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Distribution of ratings\n",
    "plt.figure(figsize = (12, 4))\n",
    "sns.countplot(df_final['play_count'])\n",
    "plt.tick_params(labelsize = 10)\n",
    "plt.title(\"Distribution of play_count \", fontsize = 10)\n",
    "plt.xlabel(\"Play_count\", fontsize = 10)\n",
    "plt.ylabel(\"Number of play_count\", fontsize = 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-Peay_cldGho"
   },
   "source": [
    "**Observations:**\n",
    "\n",
    "* The play_count 1 has the highest number of observations (~70,000). \n",
    "* Following this, play_count '2', '3', '4', '5' with ~25000, ~13000, ~8000, and ~6000, respecitvely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4CRouuLLdIz_"
   },
   "source": [
    "**Think About It:** What other insights can be drawn using exploratory data analysis?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Pr8Ghko9dUzz"
   },
   "source": [
    "**Now that we have explored and prepared the data, let's build the first recommendation system.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MLLOR65edVvj"
   },
   "source": [
    "### **Model 1: Popularity-Based Recommendation Systems**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "10HhXEqTdu80"
   },
   "source": [
    "Let's take the count and sum of play counts of the songs and build the popularity recommendation systems based on the sum of play counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "npacy0aEdnrp"
   },
   "outputs": [],
   "source": [
    "# Calculating average play_count\n",
    "average_count = df_final.groupby('song_id').mean()['play_count'] \n",
    "\n",
    "# Calculating the frequency a song is played\n",
    "play_freq = df_final.groupby('song_id').sum()['play_count']  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "id": "1IUYfklRd1oQ",
    "outputId": "92d5c569-3aef-46ef-e918-c9fc8cd18d77"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-e681762f-233a-48e4-bd6c-6c7dc4018907\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg_count</th>\n",
       "      <th>play_freq</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>song_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>1.622642</td>\n",
       "      <td>430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>1.492424</td>\n",
       "      <td>197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>1.729216</td>\n",
       "      <td>728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>1.728070</td>\n",
       "      <td>197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>1.452174</td>\n",
       "      <td>167</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e681762f-233a-48e4-bd6c-6c7dc4018907')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-e681762f-233a-48e4-bd6c-6c7dc4018907 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-e681762f-233a-48e4-bd6c-6c7dc4018907');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "         avg_count  play_freq\n",
       "song_id                      \n",
       "21        1.622642        430\n",
       "22        1.492424        197\n",
       "52        1.729216        728\n",
       "62        1.728070        197\n",
       "93        1.452174        167"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Making a dataframe with the average_count and play_freq\n",
    "final_play = pd.DataFrame({'avg_count':average_count, 'play_freq':play_freq})\n",
    "\n",
    "\n",
    "# Let us see the first five records of the final_play dataset\n",
    "final_play.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qhPdLbOmd5yq"
   },
   "source": [
    "Now, let's create a function to find the top n songs for a recommendation based on the average play count of song. We can also add a threshold for a minimum number of playcounts for a song to be considered for recommendation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "zTkmNPsbd-ng"
   },
   "outputs": [],
   "source": [
    "# Build the function to find top n songs\n",
    "\n",
    "def top_n_songs(final_play, n, min_interaction):\n",
    "    \n",
    "    # Finding products with minimum number of interactions\n",
    "    recommendations = final_play[final_play['play_freq'] > min_interaction]\n",
    "    \n",
    "    # Sorting values with respect to average rating \n",
    "    recommendations = recommendations.sort_values(by = 'avg_count', ascending = False)\n",
    "    \n",
    "    return recommendations.index[:n]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lOL81fhoeQo8"
   },
   "source": [
    "### **Recommending top 10 songs with 200 minimum interactions based on popularity**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "URSMd-qreAFc",
    "outputId": "37de90b8-8728-4d50-c8f0-34c3773e7068"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Victoria (LP Version)',\n",
       " 'The Big Gundown',\n",
       " 'Brave The Elements',\n",
       " 'Greece 2000',\n",
       " 'Secrets',\n",
       " 'Transparency',\n",
       " 'Video Killed The Radio Star',\n",
       " 'Sehr kosmisch',\n",
       " 'Luvstruck',\n",
       " \"You're The One\"]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Recommend top 10 songs using the function defined above\n",
    "\n",
    "res = list(top_n_songs(final_play, 10, 200))\n",
    "\n",
    "# Name of the songs\n",
    "\n",
    "list_of_songs = []\n",
    "for i in res:\n",
    "    list_of_songs.append(df_final[df_final['song_id']== i]['title'].unique()[0])\n",
    "list_of_songs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CLDbWh5geEVd"
   },
   "source": [
    "### **Model 2: Collaborative Filtering Recommendation System**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gPDPt3bNfQ_5"
   },
   "source": [
    "In this type of recommendation system, `we do not need any information` about the users or items. We only need user item interaction data to build a collaborative recommendation system. For example - \n",
    "<ol>\n",
    "    <li><b>Ratings</b> provided by users. For example, ratings of books on goodread, movie ratings on imdb, etc.</li>\n",
    "    <li><b>Likes</b> of users on different facebook posts, likes on youtube videos.</li>\n",
    "    <li><b>Use/buying</b> of a product by users. For example, buying different items on e-commerce sites.</li>\n",
    "    <li><b>Reading</b> of articles by readers on various blogs.</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ein3wWqsfUrS"
   },
   "source": [
    "- Similarity/Neighborhood based\n",
    "\n",
    " * User-User Similarity Based  \n",
    " * Item-Item similarity based\n",
    "\n",
    "- Model based"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y2y8WcmPfYo5"
   },
   "source": [
    "### **Building a baseline user-user similarity based recommendation system**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HWie5jmQfbCO"
   },
   "source": [
    "- Below, we are building **similarity-based recommendation systems** using `cosine` similarity and using **KNN to find similar users** which are the nearest neighbor to the given user.  \n",
    "- We will be using a new library, called `surprise`, to build the remaining models. Let's first import the necessary classes and functions from this library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5uiM4yKveE9o",
    "outputId": "52d9c972-45de-4afd-a62a-154b3ce739cd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
      "Collecting surprise\n",
      "  Downloading surprise-0.1-py2.py3-none-any.whl (1.8 kB)\n",
      "Collecting scikit-surprise\n",
      "  Downloading scikit-surprise-1.1.3.tar.gz (771 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m772.0/772.0 KB\u001b[0m \u001b[31m13.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "Requirement already satisfied: joblib>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-surprise->surprise) (1.2.0)\n",
      "Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.8/dist-packages (from scikit-surprise->surprise) (1.21.6)\n",
      "Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.8/dist-packages (from scikit-surprise->surprise) (1.7.3)\n",
      "Building wheels for collected packages: scikit-surprise\n",
      "  Building wheel for scikit-surprise (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for scikit-surprise: filename=scikit_surprise-1.1.3-cp38-cp38-linux_x86_64.whl size=3366461 sha256=57c64760aa080ec5ed22470ce720afaad90a75b9c07f889fb4f557d18a470f47\n",
      "  Stored in directory: /root/.cache/pip/wheels/af/db/86/2c18183a80ba05da35bf0fb7417aac5cddbd93bcb1b92fd3ea\n",
      "Successfully built scikit-surprise\n",
      "Installing collected packages: scikit-surprise, surprise\n",
      "Successfully installed scikit-surprise-1.1.3 surprise-0.1\n"
     ]
    }
   ],
   "source": [
    "# Install the surprise package using pip. Uncomment and run the below code to do the same\n",
    "!pip install surprise "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "id": "kWJ4nssGeoTn"
   },
   "outputs": [],
   "source": [
    "# Import necessary libraries\n",
    "\n",
    "# To compute the accuracy of models\n",
    "from surprise import accuracy\n",
    "\n",
    "# This class is used to parse a file containing play_counts, data should be in structure - user; item; play_count\n",
    "from surprise.reader import Reader\n",
    "\n",
    "# Class for loading datasets\n",
    "from surprise.dataset import Dataset\n",
    "\n",
    "# For tuning model hyperparameters\n",
    "from surprise.model_selection import GridSearchCV\n",
    "\n",
    "# For splitting the data in train and test dataset\n",
    "from surprise.model_selection import train_test_split\n",
    "\n",
    "# For implementing similarity-based recommendation system\n",
    "from surprise.prediction_algorithms.knns import KNNBasic\n",
    "\n",
    "# For implementing matrix factorization based recommendation system\n",
    "from surprise.prediction_algorithms.matrix_factorization import SVD\n",
    "\n",
    "# For implementing KFold cross-validation\n",
    "from surprise.model_selection import KFold\n",
    "\n",
    "# For implementing clustering-based recommendation system\n",
    "from surprise import CoClustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nt08Nmn5erIi"
   },
   "source": [
    "**Before building the recommendation systems, let's  go over some basic terminologies we are going to use:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "stUupRl_oJ0R"
   },
   "source": [
    "**Relevant item:** An item (song in this case) that has actually a **play count higher than the threshold play count (here 1.5)** is relevant, if the **actual play count is below the threshold then it is a non-relevant item**.  \n",
    "\n",
    "**Recommended item:** An item that's **predicted play count is higher than the threshold (here 1.5) is a recommended item**, if the **predicted play count is below the threshold then that song will not be recommended to the user**.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SEI6xcmno8or"
   },
   "source": [
    "**False Negative (FN):** It is the **frequency of relevant items that are not recommended to the user**. If the relevant items are not recommended to the user, then the user might not buy the product/item. This would result in the **loss of opportunity for the service provider**, which they would like to minimize.\n",
    "\n",
    "**False Positive (FP):** It is the **frequency of recommended items that are actually not relevant**. In this case, the recommendation system is not doing a good job of finding and recommending the relevant items to the user. This would result in **loss of resources for the service provider**, which they would also like to minimize."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6nNiIXJpC_7g"
   },
   "source": [
    "**Recall:** It is the **fraction of actually relevant items that are recommended to the user**, i.e., if out of 10 relevant products, 6 are recommended to the user then recall is 0.60. Higher the value of recall better is the model. It is one of the metrics to do the performance assessment of classification models.\n",
    "\n",
    "**Precision:** It is the **fraction of recommended items that are relevant actually**, i.e., if out of 10 recommended items, 6 are found relevant by the user then precision is 0.60. The higher the value of precision better is the model. It is one of the metrics to do the performance assessment of classification models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ErlAVMHUDeNC"
   },
   "source": [
    "**While making a recommendation system, it becomes customary to look at the performance of the model. In terms of how many recommendations are relevant and vice-versa, below are some most used performance metrics used in the assessment of recommendation systems.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mGQV7XT-Dsg-"
   },
   "source": [
    "### **Precision@k, Recall@ k, and F1-score@k**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8X8llmeHDyFy"
   },
   "source": [
    "**Precision@k** - It is the **fraction of recommended items that are relevant in `top k` predictions**. The value of k is the number of recommendations to be provided to the user. One can choose a variable number of recommendations to be given to a unique user.  \n",
    "\n",
    "\n",
    "**Recall@k** - It is the **fraction of relevant items that are recommended to the user in `top k` predictions**.\n",
    "\n",
    "**F1-score@k** - It is the **harmonic mean of Precision@k and Recall@k**. When **precision@k and recall@k both seem to be important** then it is useful to use this metric because it is representative of both of them. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jGE7ppMRGGrw"
   },
   "source": [
    "**Think About It:** Which metric should be used for this problem to compare different models?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1tiYPzFUD0wA"
   },
   "source": [
    "### **Some useful functions**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9UgTiSQVF1Zq"
   },
   "source": [
    "- Below function takes the **recommendation model** as input and gives the **precision@k, recall@k, and F1-score@k** for that model.  \n",
    "- To compute **precision and recall**, **top k** predictions are taken under consideration for each user.\n",
    "- We will use the precision and recall to compute the F1-score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "id": "634NAAoxDxcn"
   },
   "outputs": [],
   "source": [
    "# The function to calulate the RMSE, precision@k, recall@k, and F_1 score\n",
    "def precision_recall_at_k(model, k = 30, threshold = 1.5):\n",
    "    \"\"\"Return precision and recall at k metrics for each user\"\"\"\n",
    "\n",
    "    # First map the predictions to each user.\n",
    "    user_est_true = defaultdict(list)\n",
    "    \n",
    "    # Making predictions on the test data\n",
    "    predictions=model.test(testset)\n",
    "    \n",
    "    for uid, _, true_r, est, _ in predictions:\n",
    "        user_est_true[uid].append((est, true_r))\n",
    "\n",
    "    precisions = dict()\n",
    "    recalls = dict()\n",
    "    for uid, user_ratings in user_est_true.items():\n",
    "\n",
    "        # Sort user ratings by estimated value\n",
    "        user_ratings.sort(key = lambda x : x[0], reverse = True)\n",
    "\n",
    "        # Number of relevant items\n",
    "        n_rel = sum((true_r >= threshold) for (_, true_r) in user_ratings)\n",
    "\n",
    "        # Number of recommended items in top k\n",
    "        n_rec_k = sum((est >= threshold) for (est, _) in user_ratings[ : k])\n",
    "\n",
    "        # Number of relevant and recommended items in top k\n",
    "        n_rel_and_rec_k = sum(((true_r >= threshold) and (est >= threshold))\n",
    "                              for (est, true_r) in user_ratings[ : k])\n",
    "\n",
    "        # Precision@K: Proportion of recommended items that are relevant\n",
    "        # When n_rec_k is 0, Precision is undefined. We here set Precision to 0 when n_rec_k is 0\n",
    "\n",
    "        precisions[uid] = n_rel_and_rec_k / n_rec_k if n_rec_k != 0 else 0\n",
    "\n",
    "        # Recall@K: Proportion of relevant items that are recommended\n",
    "        # When n_rel is 0, Recall is undefined. We here set Recall to 0 when n_rel is 0\n",
    "\n",
    "        recalls[uid] = n_rel_and_rec_k / n_rel if n_rel != 0 else 0\n",
    "    \n",
    "    # Mean of all the predicted precisions are calculated\n",
    "    precision = round((sum(prec for prec in precisions.values()) / len(precisions)), 3)\n",
    "\n",
    "    # Mean of all the predicted recalls are calculated\n",
    "    recall = round((sum(rec for rec in recalls.values()) / len(recalls)), 3)\n",
    "    \n",
    "    accuracy.rmse(predictions)\n",
    "\n",
    "    # Command to print the overall precision\n",
    "    print('Precision: ', precision)\n",
    "\n",
    "    # Command to print the overall recall\n",
    "    print('Recall: ', recall)\n",
    "    \n",
    "    # Formula to compute the F-1 score\n",
    "    print('F_1 score: ', round((2 * precision * recall) / (precision + recall), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lnRhw2LmGVQG"
   },
   "source": [
    "**Think About It:** In the function precision_recall_at_k above the threshold value used is 1.5. How precision and recall are affected by changing the threshold? What is the intuition behind using the threshold value of 1.5? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zXeyle1OGcn7"
   },
   "source": [
    "- To compute **precision and recall**, a **threshold of 1.5 and k value of 30 is taken for the recommended and relevant play counts**. \n",
    "- In the **present case, precision and recall both need to be optimized as the service provider would like to minimize both the losses discussed** above. Hence, we are going to use the performance measure of **F_1 score**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Xtlwtep7Ht1X"
   },
   "source": [
    "Below we are loading the **`df_final` dataset**, which is a **pandas DataFrame**, into a **different format called `surprise.dataset.DatasetAutoFolds`**, which is required by this library. To do this, we will be **using the classes `Reader` and `Dataset`.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "ovshB8uuH3rO"
   },
   "outputs": [],
   "source": [
    "# Instantiating Reader scale with expected rating scale \n",
    "reader = Reader(rating_scale= (0, 5)) #use rating scale (0, 5)\n",
    "\n",
    "# Loading the dataset\n",
    "data = Dataset.load_from_df(df_final[['user_id', 'song_id', 'play_count']], reader) # Take only \"user_id\",\"song_id\", and \"play_count\"\n",
    "\n",
    "# Splitting the data into train and test dataset\n",
    "trainset, testset = train_test_split(data, test_size=0.4, random_state = 42) # Take test_size = 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "98TXVPYRH-zq"
   },
   "source": [
    "- Now, we are **ready to build the first baseline similarity-based recommendation system** using the cosine similarity.\n",
    "- **KNNBasic** is an algorithm that is also **associated with the surprise package**. It is used to find the **desired similar items among a given set of items**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6YiPHRk-JMh4"
   },
   "source": [
    "**Think About It:** How changing the test size would change the results and outputs?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y8-ppqcuJUzu"
   },
   "source": [
    "### **User-user similarity-based recommendation system**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "G-O8nkGmJUIG",
    "outputId": "8e7bb370-5035-4545-9046-119e3068a624"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "RMSE: 1.0878\n",
      "Precision:  0.396\n",
      "Recall:  0.692\n",
      "F_1 score:  0.504\n"
     ]
    }
   ],
   "source": [
    "# Build the default user-user-similarity model\n",
    "sim_options = {'name': 'cosine',\n",
    "               'user_based':True}\n",
    "\n",
    "# KNN algorithm is used to find desired similar items\n",
    "sim_user_user = KNNBasic(sim_options = sim_options, random_state = 1) # Use random_state = 1 \n",
    "\n",
    "# Train the algorithm on the trainset, and predict play_count for the testset\n",
    "sim_user_user.fit(trainset)\n",
    "\n",
    "# Let us compute precision@k, recall@k, and f_1 score with k = 30\n",
    "precision_recall_at_k(sim_user_user) # Use sim_user_user model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K9lg42TPJiqP"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* We can observe that the default user-user model has a **RMSE = 1.08** on the train set.\n",
    "* Intuition of Recall - We are getting a **recall of ~0.69**, which means out of **all the relevant songs, 69% are recommended**. \n",
    "* Intuition of Precision - We are getting a **precision of ~ 0.39**, which means **out of all the recommended songs, 39% are relevant**.\n",
    "* Here **F_1 score** of the **default user-user model is ~0.50**. It indicates that **nearly 50% of the recommended songs were relevant and relevant songs were recommended**. We can try to improve the performance by using **GridSearchCV to tune different hyperparameters** of the algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MwNKAkhMJpOa"
   },
   "source": [
    "Let's now **predict rating for a user with `user_id=6958` and `song_id=1671`** as shown below. Here the user has already interacted or watched the product with song_id '1671' and having a play_count of 2 which is denoted by the parameter r_ui."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zfF4UY4jJqTt",
    "outputId": "5ab1a00b-6d4c-4bc9-966c-b6de5736f8b2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 1671       r_ui = 2.00   est = 1.80   {'actual_k': 40, 'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=1671, r_ui=2, est=1.8009387435128914, details={'actual_k': 40, 'was_impossible': False})"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predicting play_count for a sample user with a listened song\n",
    "sim_user_user.predict(6958, 1671, r_ui = 2, verbose = True) # Use user id 6958 and song_id 1671"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wReHiQWWKP_n"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* The above output shows that **the actual play_count for this user-item pair is 2, and the predicted play_count is 1.80** by this **user-user-similarity-based baseline model**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IKokawFNKHeF"
   },
   "source": [
    "Below we are **predicting rating for the same `user_id=6958` but for a song which this user has not listen yet i.e. `prod_id=3232`** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "au1JHvS7KCdu",
    "outputId": "88cfa194-3ab0-4fd7-e2a7-76d20005dab5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 3232       r_ui = None   est = 1.64   {'actual_k': 40, 'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=3232, r_ui=None, est=1.6386860897998294, details={'actual_k': 40, 'was_impossible': False})"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predicting play_count for a sample user with a song not-listened by the user\n",
    "sim_user_user.predict(6958, 3232, verbose = True) # Use user_id 6958 and song_id 3232"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vXYqgJMtapeH"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* The above code output shows that we have predicted the play_count for this user-item pair, we got an **estimated value of 1.64** based on this **user-user similarity-based baseline model**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JU8hGr4FbBRk"
   },
   "source": [
    "Now, let's try to tune the model and see if we can improve the model performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0vVKGCvca6F7"
   },
   "source": [
    "### **Improving similarity-based recommendation system by tuning its hyperparameters**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5Bp3XiM8bHkk"
   },
   "source": [
    "Below, we will be tuning hyperparameters for the `KNNBasic` algorithms. Let's try to understand some of the hyperparameters of the KNNBasic algorithm:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xL1eMdGIbIKj"
   },
   "source": [
    "- **k** (int) – The (max) number of neighbors to take into account for aggregation. Default is 40.\n",
    "- **min_k** (int) – The minimum number of neighbors to take into account for aggregation. If there are not enough neighbors, the prediction is set to the global mean of all ratings. Default is 1.\n",
    "- **sim_options** (dict) – A dictionary of options for the similarity measure. And there are four similarity measures available in surprise: \n",
    "    - cosine\n",
    "    - msd (default)\n",
    "    - Pearson\n",
    "    - Pearson baseline\n",
    "    - **min_support** (int) The minimum number of common items (when 'user_based' is 'True') or minimum number of common users (when 'user_based' is 'False') for the similarity not to be zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "jDpOYuanbHIA",
    "outputId": "051b7f56-b683-47c6-ff66-9649b412affd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0450706892741806\n",
      "{'k': 30, 'min_k': 9, 'sim_options': {'name': 'pearson_baseline', 'user_based': True, 'min_support': 2}}\n"
     ]
    }
   ],
   "source": [
    "# Setting up parameter grid to tune the hyperparameters\n",
    "param_grid = {'k': [10, 20, 30], 'min_k': [3, 6, 9],\n",
    "              'sim_options': {'name': [\"cosine\", 'pearson', \"pearson_baseline\"],\n",
    "                              'user_based': [True], \"min_support\": [2, 4]}\n",
    "              }\n",
    "\n",
    "# Performing 3-fold cross-validation to tune the hyperparameters\n",
    "gs = GridSearchCV(KNNBasic, param_grid, measures = ['rmse', 'mae'], cv = 3, n_jobs = -1)\n",
    "\n",
    "# Fitting the data\n",
    "gs.fit(data) # Use entire data for GridSearch\n",
    "\n",
    "# Best RMSE score\n",
    "print(gs.best_score['rmse'])\n",
    "\n",
    "# Combination of parameters that gave the best RMSE score\n",
    "print(gs.best_params['rmse'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tdpIgecNbd2N"
   },
   "source": [
    "Once the grid search is **complete**, we can get the **optimal values for each of those hyperparameters** as shown above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xTq2jpncbiGA"
   },
   "source": [
    "Now, let's build the **final model by using tuned values of the hyperparameters**, which we received by using **grid search cross-validation**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4Y1XDgDsbXPW",
    "outputId": "a77a709c-4aed-48b2-b76b-f18d54734281"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "RMSE: 1.0521\n",
      "Precision:  0.413\n",
      "Recall:  0.721\n",
      "F_1 score:  0.525\n"
     ]
    }
   ],
   "source": [
    "# Train the best model found in above gridsearch\n",
    "sim_options = {'name':'pearson_baseline', 'user_based':True, 'min_support':2}\n",
    "\n",
    "# Creating an instance of KNNBasic with optimal hyperparameter values\n",
    "sim_user_user_optimized = KNNBasic(sim_options = sim_options, k = 30, min_k = 9)\n",
    "\n",
    "# Training the algorithm on the train set\n",
    "sim_user_user_optimized.fit(trainset)\n",
    "\n",
    "# Let us compute precision@k and recall@k also with k = 30\n",
    "precision_recall_at_k(sim_user_user_optimized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "60dtQeaSbYmm"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* We can observe that after tuning hyperparameters, **F_1 score of the tuned model is 0.525, which is slightly better than the baseline model which was 0.504**. \n",
    "* **Also, the RMSE of the model has slightly reduced from 1.08 to 1.05  as compared to the baseline model**. Hence, we can say that the model performance has improved slightly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fwCRdAZwbncC"
   },
   "source": [
    "Let's now **predict the play_count for a user with `user_id` = \"6958\" and song_id = 1671 with the optimized model as shown below**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QzPaNUIybrAO",
    "outputId": "a46f05da-bd6a-4894-bc25-ccee6a6b0769"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=1671, r_ui=2, est=1.962926073914969, details={'actual_k': 24, 'was_impossible': False})"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predict the play count for a user who has listened to the song. Take user_id 6958, song_id 1671 and r_ui = 2\n",
    "sim_user_user_optimized.predict(6958, 1671, r_ui = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GAq4SC8Mb3PI"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* There is a difference between the **baseline model and the optimized model** with the estimated value of **play_count**, although both models would classify the song as relevant. With this model we got an **estimated value of 1.96** which is very close to the real value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Lm0NKMmDb95s"
   },
   "source": [
    "Now let's **predict the play_count for the same `user_id` = \"6958\", but with a song the user have not listened to yet, that is song_id = 3232 with the optimized model as shown below**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xHV1XXh1cDls",
    "outputId": "c660f78a-ad26-4739-f10e-194ff6f39732"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 3232       r_ui = None   est = 1.45   {'actual_k': 10, 'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=3232, r_ui=None, est=1.4516261428486725, details={'actual_k': 10, 'was_impossible': False})"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predict the play count for a song that is not listened to by the user (with user_id 6958)\n",
    "sim_user_user_optimized.predict(6958, 3232, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oq8IVg_DcDsx"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* For the **predicted value** of the song with **song_id = 3232** that the user has not interacted before we got an **estimated value of 1.45**. In this case the optimized model would not classify this song as relevant, but with the previous model yes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-3cm9yyvcu2x"
   },
   "source": [
    "**Think About It:** Along with making predictions on listened and unknown songs can we get 5 nearest neighbors (most similar) to a certain song?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kE_WjsXMcvYI"
   },
   "source": [
    "### **Identifying similar Users to a given User (nearest neighbors)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E5S5d66bc0Ac"
   },
   "source": [
    "We can also find out **similar users to a given user** or its **nearest neighbors** based on this KNNBasic algorithm. Below, we are finding the 5 most similar users to the first user in the list with internal id 0, based on the `msd` distance metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8CKL0JS0c5Ue",
    "outputId": "091b137b-0591-4a20-c2d0-78f5aa1b06d3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[42, 1131, 17, 186, 249]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use inner id 0\n",
    "sim_user_user_optimized.get_neighbors(0, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jf2953r6dCrR"
   },
   "source": [
    "### **Implementing the recommendation algorithm based on optimized KNNBasic model**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LQVkZCa8dDVz"
   },
   "source": [
    "Below we will be implementing a function where the input parameters are:\n",
    "\n",
    "- data: A **song** dataset\n",
    "- user_id: A user-id **against which we want the recommendations**\n",
    "- top_n: The **number of songs we want to recommend**\n",
    "- algo: The algorithm we want to use **for predicting the play_count**\n",
    "- The output of the function is a **set of top_n items** recommended for the given user_id based on the given algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "id": "vZVM4863dGK7"
   },
   "outputs": [],
   "source": [
    "def get_recommendations(data, user_id, top_n, algo):\n",
    "    \n",
    "    # Creating an empty list to store the recommended song ids\n",
    "    recommendations = []\n",
    "    \n",
    "    # Creating an user item interactions matrix \n",
    "    user_item_interactions_matrix = data.pivot_table(index = 'user_id', columns = 'song_id', values = 'play_count')\n",
    "    \n",
    "    # Extracting those songs ids which the user_id has not listened yet\n",
    "    non_interacted_products = user_item_interactions_matrix.loc[user_id][user_item_interactions_matrix.loc[user_id].isnull()].index.tolist()\n",
    "    \n",
    "    # Looping through each of the songs ids which user_id has not interacted yet\n",
    "    for song_id in non_interacted_products:\n",
    "        \n",
    "        # Predicting the play_count for those non listened songs ids by this user\n",
    "        est = algo.predict(user_id, song_id).est\n",
    "        \n",
    "        # Appending the predicted play_count\n",
    "        recommendations.append((song_id, est))\n",
    "\n",
    "    # Sorting the predicted play_count in descending order\n",
    "    recommendations.sort(key = lambda x : x[1], reverse = True)\n",
    "\n",
    "    return recommendations[:top_n] # Returing top n highest predicted play_count song for this user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-fhh6sJBdPCN"
   },
   "source": [
    "#### **Predicting top 5 songs for user_id = \"6958\" with similarity based recommendation system**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "id": "qpGvcGrmdPoL"
   },
   "outputs": [],
   "source": [
    "# Make top 5 recommendations for user_id 6958 with a similarity-based recommendation engine\n",
    "recommendations = get_recommendations(df_final, 6958, 5, sim_user_user_optimized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "70S6YxPydacz",
    "outputId": "f389f97c-6761-4263-cf8b-454682b3fadc"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-feacd0c3-b20c-48c0-a480-f2aaf88bb6ec\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>song_id</th>\n",
       "      <th>predicted_play_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5531</td>\n",
       "      <td>2.553335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>317</td>\n",
       "      <td>2.518269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4954</td>\n",
       "      <td>2.406776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8635</td>\n",
       "      <td>2.396606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5943</td>\n",
       "      <td>2.390723</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-feacd0c3-b20c-48c0-a480-f2aaf88bb6ec')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-feacd0c3-b20c-48c0-a480-f2aaf88bb6ec button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-feacd0c3-b20c-48c0-a480-f2aaf88bb6ec');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   song_id  predicted_play_count\n",
       "0     5531              2.553335\n",
       "1      317              2.518269\n",
       "2     4954              2.406776\n",
       "3     8635              2.396606\n",
       "4     5943              2.390723"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Building the dataframe for above recommendations with columns \"song_id\" and \"predicted_ratings\"\n",
    "pd.DataFrame(recommendations, columns = ['song_id', 'predicted_play_count'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KH8RGAjjde-O"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* With the above code output we got a dataframe with, in this case, 5 recommended songs for the user with **user_id = 6958** that have not listened before. \n",
    "* Also, it contains a **predicted_play_count** column where we got an estimated value the user will play that song."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WfplfblIdfaS"
   },
   "source": [
    "### **Correcting the play_counts and Ranking the above songs**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wo9DvYY9dlpy"
   },
   "source": [
    "While comparing the play_count of two songs, it is not only the **play_count** that describe the **likelihood of the user to that song**. Along with the play_count, the **number of users who have listen that song** also becomes important to consider. Due to this, we have calculated the **\"corrected_play_count\"** for each song. Commonly higher the **\"play_freq\" of a song more it is liked by users**. To interpret the above concept, a **song play_count of 5 with play_freq 20 is less liked in comparison to a song with play_count 4 with a play_freq count of 100**. It has been **empirically found that the likelihood of the song is directly proportional to the inverse of the square root of the play_freq of the product**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "id": "HgeLUMX_diya"
   },
   "outputs": [],
   "source": [
    "def ranking_songs(recommendations, final_rating):\n",
    "  # Sort the songs based on play counts\n",
    "  ranked_songs = final_rating.loc[[items[0] for items in recommendations]].sort_values('play_freq', ascending = False)[['play_freq']].reset_index()\n",
    "\n",
    "  # Merge with the recommended songs to get predicted play_count\n",
    "  ranked_songs = ranked_songs.merge(pd.DataFrame(recommendations, columns = ['song_id', 'predicted_play_count']), on = 'song_id', how = 'inner')\n",
    "\n",
    "  # Rank the songs based on corrected play_counts\n",
    "  ranked_songs['corrected_play_count'] = ranked_songs['predicted_play_count'] + 1 / np.sqrt(ranked_songs['play_freq'])\n",
    "\n",
    "  # Sort the songs based on corrected play_counts\n",
    "  ranked_songs = ranked_songs.sort_values('corrected_play_count', ascending = False)\n",
    "  \n",
    "  return ranked_songs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QnODl9Upe2gI"
   },
   "source": [
    "**Think About It:** In the above function to correct the predicted play_count a quantity 1/np.sqrt(n) is subtracted. What is the intuition behind it? Is it also possible to add this quantity instead of subtracting?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TpiTtpbge21D"
   },
   "source": [
    "**Note:** In the **above-corrected rating formula**, we can add the **quantity `1/np.sqrt(n)` instead of subtracting it to get more optimistic predictions**. But here we are **subtracting this quantity**, as there are some products with ratings 5 and **we can't have a rating more than 5 for a product**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "kLHfDtUTe82X",
    "outputId": "c390a20b-1ab0-4209-b2d0-777d44b88e24"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-a847e05e-a80e-49f9-b2ab-024098f83d8f\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_freq</th>\n",
       "      <th>predicted_play_count</th>\n",
       "      <th>corrected_play_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5531</td>\n",
       "      <td>1427</td>\n",
       "      <td>2.553335</td>\n",
       "      <td>2.579807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>317</td>\n",
       "      <td>836</td>\n",
       "      <td>2.518269</td>\n",
       "      <td>2.552855</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4954</td>\n",
       "      <td>338</td>\n",
       "      <td>2.406776</td>\n",
       "      <td>2.461169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8635</td>\n",
       "      <td>259</td>\n",
       "      <td>2.396606</td>\n",
       "      <td>2.458743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5943</td>\n",
       "      <td>830</td>\n",
       "      <td>2.390723</td>\n",
       "      <td>2.425434</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a847e05e-a80e-49f9-b2ab-024098f83d8f')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-a847e05e-a80e-49f9-b2ab-024098f83d8f button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-a847e05e-a80e-49f9-b2ab-024098f83d8f');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   song_id  play_freq  predicted_play_count  corrected_play_count\n",
       "0     5531       1427              2.553335              2.579807\n",
       "1      317        836              2.518269              2.552855\n",
       "3     4954        338              2.406776              2.461169\n",
       "4     8635        259              2.396606              2.458743\n",
       "2     5943        830              2.390723              2.425434"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Applying the ranking_songs function on the final_play data\n",
    "ranking_songs(recommendations, final_play)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vjeecSxYe-wS"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* In the above code output we got the corrected_play_count based on the predicted_play_count column using the formula 1/np.sqrt(n)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ThfmSFaSe_WA"
   },
   "source": [
    "### **Item Item Similarity-based collaborative filtering recommendation systems** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SCW_UEzKfD48"
   },
   "source": [
    "* Above we have seen **similarity-based collaborative filtering** where similarity is calculated **between users**. Now let us look into similarity-based collaborative filtering where similarity is seen **between items**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4II32jvufDYs",
    "outputId": "1fc4fc2e-9cef-4e24-a51b-3fbe7b7ad58d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.0394\n",
      "Precision:  0.307\n",
      "Recall:  0.562\n",
      "F_1 score:  0.397\n"
     ]
    }
   ],
   "source": [
    "# Apply the item-item similarity collaborative filtering model with random_state = 1 and evaluate the model performance\n",
    "\n",
    "# Declaring the similarity options\n",
    "sim_options = {'name': 'cosine',\n",
    "               'user_based': False}\n",
    "\n",
    "# KNN algorithm is used to find desired similar items\n",
    "sim_item_item = KNNBasic(sim_options = sim_options, random_state = 1, verbose = False)\n",
    "\n",
    "# Train the algorithm on the train set, and predict ratings for the test set\n",
    "sim_item_item.fit(trainset)\n",
    "\n",
    "# Let us compute precision@k, recall@k, and f_1 score with k = 30\n",
    "precision_recall_at_k(sim_item_item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OEapNnYbfZlF"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* We can observe that the baseline model has `RMSE = 1.03` & `F_1 Score = 0.39`on the test set.\n",
    "* We got a low value on **F_1 score**, we can try to **improve the performance number by using `GridSearchCV`** to tune different hyperparameters of this algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TgUwaHuHffIi"
   },
   "source": [
    "Let's now **predict a play_count for a user with `user_id = 6958` and `song_id = 1671`** as shown below. Here the user has already listened the song with song_id \"1671\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "tXzeme7ZffrC",
    "outputId": "94f904a6-6ec1-4239-c1a3-66b22861ffd2"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-06ec3fb1-0a47-4a6b-bd6d-ed035331dbea\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_count</th>\n",
       "      <th>title</th>\n",
       "      <th>release</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>215</th>\n",
       "      <td>6958</td>\n",
       "      <td>1671</td>\n",
       "      <td>2</td>\n",
       "      <td>Sleeping In (Album)</td>\n",
       "      <td>Give Up</td>\n",
       "      <td>Postal Service</td>\n",
       "      <td>2003</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-06ec3fb1-0a47-4a6b-bd6d-ed035331dbea')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-06ec3fb1-0a47-4a6b-bd6d-ed035331dbea button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-06ec3fb1-0a47-4a6b-bd6d-ed035331dbea');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "     user_id  song_id  play_count                title  release  \\\n",
       "215     6958     1671           2  Sleeping In (Album)  Give Up   \n",
       "\n",
       "        artist_name  year  \n",
       "215  Postal Service  2003  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Checking the real value of play_count on user id 6958 with song id 1671\n",
    "df_final[(df_final['user_id'] == 6958) & (df_final['song_id'] == 1671)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mbZVpe11fh-1",
    "outputId": "3e887b51-86b4-45bc-9abb-779e32c1f1d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 1671       r_ui = 2.00   est = 1.36   {'actual_k': 20, 'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=1671, r_ui=2, est=1.3614157231762556, details={'actual_k': 20, 'was_impossible': False})"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predicting play count for a sample user_id 6958 and song (with song_id 1671) heard by the user\n",
    "sim_item_item.predict(6958, 1671, r_ui = 2, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Pq1dqCtzfj2K"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* In the above code output shows that **the actual rating for this user-item pair is 2 and the predicted rating is 1.36** by the **item-item-similarity-based baseline model**. Hence, our model would not classify this song as relevant for this user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HkL7jcEWf4NW"
   },
   "source": [
    "Below we are **predicting rating for other `user_id = 47786` but for a song with which this user has not listened yet, i.e., `song_id = 1671`**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "id": "9hn66tS7gAoS",
    "outputId": "7f8ab6e8-06b5-482c-fd41-4a1a1fac0f74"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-0311217c-8b34-4f51-8e5c-b2c26d8fa506\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_count</th>\n",
       "      <th>title</th>\n",
       "      <th>release</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>6958</td>\n",
       "      <td>447</td>\n",
       "      <td>1</td>\n",
       "      <td>Daisy And Prudence</td>\n",
       "      <td>Distillation</td>\n",
       "      <td>Erin McKeown</td>\n",
       "      <td>2000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>6958</td>\n",
       "      <td>512</td>\n",
       "      <td>1</td>\n",
       "      <td>The Ballad of Michael Valentine</td>\n",
       "      <td>Sawdust</td>\n",
       "      <td>The Killers</td>\n",
       "      <td>2004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>6958</td>\n",
       "      <td>549</td>\n",
       "      <td>1</td>\n",
       "      <td>I Stand Corrected (Album)</td>\n",
       "      <td>Vampire Weekend</td>\n",
       "      <td>Vampire Weekend</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>204</th>\n",
       "      <td>6958</td>\n",
       "      <td>703</td>\n",
       "      <td>1</td>\n",
       "      <td>They Might Follow You</td>\n",
       "      <td>Tiny Vipers</td>\n",
       "      <td>Tiny Vipers</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>205</th>\n",
       "      <td>6958</td>\n",
       "      <td>719</td>\n",
       "      <td>1</td>\n",
       "      <td>Monkey Man</td>\n",
       "      <td>You Know I'm No Good</td>\n",
       "      <td>Amy Winehouse</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999734</th>\n",
       "      <td>47786</td>\n",
       "      <td>9139</td>\n",
       "      <td>1</td>\n",
       "      <td>Half Of My Heart</td>\n",
       "      <td>Battle Studies</td>\n",
       "      <td>John Mayer</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999736</th>\n",
       "      <td>47786</td>\n",
       "      <td>9186</td>\n",
       "      <td>1</td>\n",
       "      <td>Bitter Sweet Symphony</td>\n",
       "      <td>Bitter Sweet Symphony</td>\n",
       "      <td>The Verve</td>\n",
       "      <td>1997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999745</th>\n",
       "      <td>47786</td>\n",
       "      <td>9351</td>\n",
       "      <td>2</td>\n",
       "      <td>The Police And The Private</td>\n",
       "      <td>Live It Out</td>\n",
       "      <td>Metric</td>\n",
       "      <td>2005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999755</th>\n",
       "      <td>47786</td>\n",
       "      <td>9543</td>\n",
       "      <td>1</td>\n",
       "      <td>Just Friends</td>\n",
       "      <td>Back To Black</td>\n",
       "      <td>Amy Winehouse</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999765</th>\n",
       "      <td>47786</td>\n",
       "      <td>9847</td>\n",
       "      <td>1</td>\n",
       "      <td>He Can Only Hold Her</td>\n",
       "      <td>Back To Black</td>\n",
       "      <td>Amy Winehouse</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>117726 rows × 7 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0311217c-8b34-4f51-8e5c-b2c26d8fa506')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-0311217c-8b34-4f51-8e5c-b2c26d8fa506 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-0311217c-8b34-4f51-8e5c-b2c26d8fa506');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "         user_id  song_id  play_count                            title  \\\n",
       "200         6958      447           1               Daisy And Prudence   \n",
       "202         6958      512           1  The Ballad of Michael Valentine   \n",
       "203         6958      549           1        I Stand Corrected (Album)   \n",
       "204         6958      703           1            They Might Follow You   \n",
       "205         6958      719           1                       Monkey Man   \n",
       "...          ...      ...         ...                              ...   \n",
       "1999734    47786     9139           1                 Half Of My Heart   \n",
       "1999736    47786     9186           1            Bitter Sweet Symphony   \n",
       "1999745    47786     9351           2       The Police And The Private   \n",
       "1999755    47786     9543           1                     Just Friends   \n",
       "1999765    47786     9847           1             He Can Only Hold Her   \n",
       "\n",
       "                       release      artist_name  year  \n",
       "200               Distillation     Erin McKeown  2000  \n",
       "202                    Sawdust      The Killers  2004  \n",
       "203            Vampire Weekend  Vampire Weekend  2007  \n",
       "204                Tiny Vipers      Tiny Vipers  2007  \n",
       "205       You Know I'm No Good    Amy Winehouse  2007  \n",
       "...                        ...              ...   ...  \n",
       "1999734         Battle Studies       John Mayer     0  \n",
       "1999736  Bitter Sweet Symphony        The Verve  1997  \n",
       "1999745            Live It Out           Metric  2005  \n",
       "1999755          Back To Black    Amy Winehouse  2006  \n",
       "1999765          Back To Black    Amy Winehouse  2006  \n",
       "\n",
       "[117726 rows x 7 columns]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Looking for the user id that has not listened to song id 1671 before\n",
    "df_final[df_final['song_id'] != 1671]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49
    },
    "id": "7MW3SuE0gDJs",
    "outputId": "8dac72e2-9bf3-4874-d31d-f910c376737a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-3a4dcd2b-cf48-4161-8539-77ac0bbfdf4d\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_count</th>\n",
       "      <th>title</th>\n",
       "      <th>release</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3a4dcd2b-cf48-4161-8539-77ac0bbfdf4d')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-3a4dcd2b-cf48-4161-8539-77ac0bbfdf4d button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-3a4dcd2b-cf48-4161-8539-77ac0bbfdf4d');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [user_id, song_id, play_count, title, release, artist_name, year]\n",
       "Index: []"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Making sure the user has not listened to the song id 1671\n",
    "df_final[(df_final['user_id'] == 47786) & (df_final['song_id'] == 1671)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-EZadg2agFCm",
    "outputId": "62ed6afe-5761-4ead-a231-8dcb98dd1041"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 47786      item: 1671       r_ui = None   est = 1.94   {'actual_k': 40, 'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=47786, iid=1671, r_ui=None, est=1.9365719326237714, details={'actual_k': 40, 'was_impossible': False})"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predict the play count for a user  that has not listened to the song (with song_id 1671)\n",
    "sim_item_item.predict(47786, 1671, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R3GRUdXDgG4w"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* In the above code output we can see the **predicted rating for this user-item pair is 1.94** based on this **item-item similarity-based baseline model**. Hence, our model would clasiffy this song as relevant for this user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UvIeVuYJgKER"
   },
   "source": [
    "#### **Improving similarity-based recommendation system by tuning its hyperparameters**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y4KgIw0IglYq"
   },
   "source": [
    "Below we will be **tuning hyperparameters for the `KNNBasic` algorithms**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2_U4tNd0grOc"
   },
   "source": [
    "### **Hyperparameter tuning the item-item similarity-based model.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nhUhnnxHg3ta",
    "outputId": "e7994c95-b4b1-4c5d-c9c9-02d5831a6b24"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the pearson similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "1.0233470782580805\n",
      "{'k': 30, 'min_k': 6, 'sim_options': {'name': 'pearson_baseline', 'user_based': False, 'min_support': 2}}\n"
     ]
    }
   ],
   "source": [
    "# Apply grid search for enhancing model performance\n",
    "\n",
    "# Setting up parameter grid to tune the hyperparameters\n",
    "param_grid = {'k': [10, 20, 30], 'min_k': [3, 6, 9],\n",
    "              'sim_options': {'name': [\"cosine\", 'pearson', \"pearson_baseline\"],\n",
    "                              'user_based': [False], \"min_support\": [2, 4]}\n",
    "              }\n",
    "\n",
    "# Performing 3-fold cross-validation to tune the hyperparameters\n",
    "gs = GridSearchCV(KNNBasic, param_grid, measures = ['rmse'], cv = 3)\n",
    "\n",
    "# Fitting the data\n",
    "gs.fit(data)\n",
    "\n",
    "# Find the best RMSE score\n",
    "print(gs.best_score['rmse'])\n",
    "\n",
    "# Extract the combination of parameters that gave the best RMSE score\n",
    "print(gs.best_params['rmse'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4Euctu8ehGS3"
   },
   "source": [
    "**Think About It:** How do the parameters affect the performance of the model? Can we improve the performance of the model further? Check the list of hyperparameters [here](https://surprise.readthedocs.io/en/stable/knn_inspired.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y88jkculg--l"
   },
   "source": [
    "Once the **grid search** is complete, we can get the **optimal values for each of those hyperparameters as shown above.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dmWhYszug_Td"
   },
   "source": [
    "Now let's build the **final model** by using **tuned values of the hyperparameters** which we received by using grid search cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "97Mw2cJqhOrt",
    "outputId": "c83573d5-704a-414a-827c-29556a678ded"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.0328\n",
      "Precision:  0.408\n",
      "Recall:  0.665\n",
      "F_1 score:  0.506\n"
     ]
    }
   ],
   "source": [
    "# Apply the best modle found in the grid search\n",
    "\n",
    "# Using the optimal similarity measure for item-item based collaborative filtering\n",
    "sim_options = {'name':'pearson_baseline', 'user_based':False, 'min_support':2}\n",
    "\n",
    "# Creating an instance of KNNBasic with optimal hyperparameter values\n",
    "sim_item_item_optimized = KNNBasic(sim_options = sim_options , k = 30 , min_k = 6 , random_state = 1, verbose = False)\n",
    "\n",
    "# Training the algorithm on the train set\n",
    "sim_item_item_optimized.fit(trainset)\n",
    "\n",
    "# Let us compute precision@k and recall@k, f1_score and RMSE\n",
    "precision_recall_at_k(sim_item_item_optimized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YXTZ2LDEhQ9y"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* We observe that after tuning hyperparameters, **F_1 score of the model is better than the baseline model increasing from ~0.39 to ~0.50**. \n",
    "* Along with this, **the RMSE of the model has slightly gone down in comparison to the baseline item-item model**. Hence, we can say that the model performance has improved after hyperparameter tuning.\n",
    "* At this point, the model that has the best performance with the **F_1 score** is the optimized user-user model. Let's see if we can get better performance with other models. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d-_yFYA2hT9D"
   },
   "source": [
    "Let's us now predict **play_count for an user with `user_id = 6958` and for `song_id = 1671`** with the **optimized model** as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EGXuvq6WhUq4",
    "outputId": "25469da9-335b-4c09-dbc4-dc5e09b6f219"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 1671       r_ui = 2.00   est = 1.96   {'actual_k': 10, 'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=1671, r_ui=2, est=1.9634957386781853, details={'actual_k': 10, 'was_impossible': False})"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predict the play_count by a user(user_id 6958) for the song (song_id 1671)\n",
    "sim_item_item_optimized.predict(6958, 1671, r_ui = 2, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nZKfauORhe3-"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* The optimized model is predicting an **estimated play_count of 1.96**, a close value to the real one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zzjRo_Qihqzc"
   },
   "source": [
    "Below we are **predicting play_count** for the same **`user_id = 6958`** but for a song with which this user **has not listened before**, i.e., `song_id == 3232`, by using the optimized model as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "m2PPa2WqheQG",
    "outputId": "6fba3cd6-c8f6-4a80-98a9-4ecb2065ea58"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 3232       r_ui = None   est = 1.28   {'actual_k': 10, 'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=3232, r_ui=None, est=1.2759946618244609, details={'actual_k': 10, 'was_impossible': False})"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predicting play count for a sample user_id 6958 with song_id 3232 which is not heard by the user\n",
    "sim_item_item_optimized.predict(6958, 3232, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Xxm6o2rLhje-"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* For an un heard song the model is **predicting a play_count of 1.28**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "215-2ZEhh6sW"
   },
   "source": [
    "#### **Identifying similar items to a given item (nearest neighbors)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DAGNE2hLh_DC"
   },
   "source": [
    "We can also find out **similar items** to a given item or its nearest neighbors based on this **KNNBasic algorithm**. Below we are finding the 5 most similar items to the item with internal id 0 based on the `msd` distance metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "E_-SjU7eh7I5",
    "outputId": "a1634240-f424-46a4-e4b2-fc1376dbb128"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[124, 523, 173, 205, 65]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find five most similar items to the item with inner id 0\n",
    "sim_item_item_optimized.get_neighbors(0, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WIA56KxMiH6r"
   },
   "source": [
    "#### **Predicted top 5 songs for user_id = \"6958\" with similarity based recommendation system.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "id": "M1CCljUuiIpn"
   },
   "outputs": [],
   "source": [
    "# Making top 5 recommendations for user_id 6958 with item_item_similarity-based recommendation engine\n",
    "recommendations = get_recommendations(df_final, 6958, 5, sim_item_item_optimized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "6mP-6BjGicHz",
    "outputId": "8340bec6-b254-45d6-c247-5ade3856144a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-d7230605-66d8-4960-b7e8-145dd254f252\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>song_id</th>\n",
       "      <th>predicted_play_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2342</td>\n",
       "      <td>2.653903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5101</td>\n",
       "      <td>2.386577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>139</td>\n",
       "      <td>2.313727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7519</td>\n",
       "      <td>2.270864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8099</td>\n",
       "      <td>2.212702</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d7230605-66d8-4960-b7e8-145dd254f252')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-d7230605-66d8-4960-b7e8-145dd254f252 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-d7230605-66d8-4960-b7e8-145dd254f252');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   song_id  predicted_play_count\n",
       "0     2342              2.653903\n",
       "1     5101              2.386577\n",
       "2      139              2.313727\n",
       "3     7519              2.270864\n",
       "4     8099              2.212702"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Building the dataframe for above recommendations with columns \"song_id\" and \"predicted_play_count\"\n",
    "pd.DataFrame(recommendations, columns = ['song_id', 'predicted_play_count'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "xWMoU4nCiccM",
    "outputId": "08893658-d872-4e2a-ff45-b565ee337336"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-32a84ab9-15d7-4fc8-8749-cba6a09b5b56\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_freq</th>\n",
       "      <th>predicted_play_count</th>\n",
       "      <th>corrected_play_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2342</td>\n",
       "      <td>146</td>\n",
       "      <td>2.653903</td>\n",
       "      <td>2.736664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5101</td>\n",
       "      <td>192</td>\n",
       "      <td>2.386577</td>\n",
       "      <td>2.458746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>139</td>\n",
       "      <td>207</td>\n",
       "      <td>2.313727</td>\n",
       "      <td>2.383232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7519</td>\n",
       "      <td>235</td>\n",
       "      <td>2.270864</td>\n",
       "      <td>2.336096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8099</td>\n",
       "      <td>465</td>\n",
       "      <td>2.212702</td>\n",
       "      <td>2.259076</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-32a84ab9-15d7-4fc8-8749-cba6a09b5b56')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-32a84ab9-15d7-4fc8-8749-cba6a09b5b56 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-32a84ab9-15d7-4fc8-8749-cba6a09b5b56');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   song_id  play_freq  predicted_play_count  corrected_play_count\n",
       "4     2342        146              2.653903              2.736664\n",
       "3     5101        192              2.386577              2.458746\n",
       "2      139        207              2.313727              2.383232\n",
       "1     7519        235              2.270864              2.336096\n",
       "0     8099        465              2.212702              2.259076"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Applying the ranking_songs function\n",
    "ranking_songs(recommendations, final_play)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b3dpjSTEid81"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* With the **optimized item-item model** we got **different 5 similar neigbors** unlike the **optimized user-user model**.\n",
    "* Also, we got the **corrected_play_count** based on the **predicted_play_count column** using the **ranking_songs function** on recommendations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UP67S946ilIB"
   },
   "source": [
    "### **Model 3: Collaborative Filtering - Matrix Factorization**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GrZb9G7xion0"
   },
   "source": [
    "Model-based Collaborative Filtering is a **personalized recommendation system**, the recommendations are based on the past behavior of the user and it is not dependent on any additional information. We use **latent features** to find recommendations for each user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2mDtbASGit9F"
   },
   "source": [
    "### **Singular Value Decomposition (SVD)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vlOA6p5aiuce"
   },
   "source": [
    "SVD is used to **compute the latent features** from the **user-item matrix**. But SVD does not work when we **miss values** in the **user-item matrix**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "jxhnc6byilwe",
    "outputId": "aa48bc73-080b-46f3-b68b-71fb055a01b9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.0252\n",
      "Precision:  0.41\n",
      "Recall:  0.633\n",
      "F_1 score:  0.498\n"
     ]
    }
   ],
   "source": [
    "# Build baseline model using svd\n",
    "svd = SVD(random_state = 1)\n",
    "\n",
    "# Training the algorithm on the train set\n",
    "svd.fit(trainset)\n",
    "\n",
    "# Use the function precision_recall_at_k to compute precision@k, recall@k, F1-Score, and RMSE\n",
    "precision_recall_at_k(svd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zqSkGQLsi8N0"
   },
   "source": [
    "- Let's now predict the play_count for a user with `user_id = \"6958\"` and `song_id = \"1671` as shown below.\n",
    "- Here, the user has already listen the song."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cejMkYKAjB9M",
    "outputId": "8755189b-c92d-4dfe-deeb-db5409489081"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 1671       r_ui = 2.00   est = 1.27   {'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=1671, r_ui=2, est=1.267473397214638, details={'was_impossible': False})"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Making prediction for user (with user_id 6958) to song (with song_id 1671), take r_ui = 2\n",
    "svd.predict(6958, 1671, r_ui = 2, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PJfwWvbIjkN_"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "- As we can observe, **the actual play_count** for this user-item pair is 2, and the predicted play_count is **1.27** by this matrix **factorization-based baseline model**. It seems like we have under-estimated the play_count. We will try to fix this later by **tuning the hyperparameters** of the model using GridSearchCV."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zgjMbmYxjMmp"
   },
   "source": [
    "- Below we are predicting rating for the same user with `user_id = 6958` but for a song with which this user has not interacted before, i.e., `song_id = 3232`, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Qdzu-W7rjSpO",
    "outputId": "0dc22739-0ac9-4a15-9510-c30a189ce25b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 3232       r_ui = None   est = 1.56   {'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=3232, r_ui=None, est=1.5561675084403663, details={'was_impossible': False})"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Making a prediction for the user who has not listened to the song (song_id 3232)\n",
    "svd.predict(6958, 3232, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qcmIZTKRjx1A"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "- We can see that the **estimated play_count** for this **user-item pair** is 1.56 based on this **matrix factorization based baseline model**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zcO5OlUkkCqo"
   },
   "source": [
    "#### **Improving matrix factorization based recommendation system by tuning its hyperparameters**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xqmWuP4akDHT"
   },
   "source": [
    "In SVD, rating is predicted as: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XqzfcW3lkFqa"
   },
   "source": [
    "$$\\hat{r}_{u i}=\\mu+b_{u}+b_{i}+q_{i}^{T} p_{u}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J6FcWz6DkIEr"
   },
   "source": [
    "If user $u$ is unknown, then the bias $b_{u}$ and the factors $p_{u}$ are assumed to be zero. The same applies for item $i$ with $b_{i}$ and $q_{i}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kE3LnogakPKY"
   },
   "source": [
    "To estimate all the unknown, we minimize the following regularized squared error:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4NybTZZnkRuL"
   },
   "source": [
    "$$\\sum_{r_{u i} \\in R_{\\text {train }}}\\left(r_{u i}-\\hat{r}_{u i}\\right)^{2}+\\lambda\\left(b_{i}^{2}+b_{u}^{2}+\\left\\|q_{i}\\right\\|^{2}+\\left\\|p_{u}\\right\\|^{2}\\right)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Clqzdm_ZkUmF"
   },
   "source": [
    "The minimization is performed by a very straightforward **stochastic gradient descent**:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BEhh9XSNkWwU"
   },
   "source": [
    "$$\\begin{aligned} b_{u} & \\leftarrow b_{u}+\\gamma\\left(e_{u i}-\\lambda b_{u}\\right) \\\\ b_{i} & \\leftarrow b_{i}+\\gamma\\left(e_{u i}-\\lambda b_{i}\\right) \\\\ p_{u} & \\leftarrow p_{u}+\\gamma\\left(e_{u i} \\cdot q_{i}-\\lambda p_{u}\\right) \\\\ q_{i} & \\leftarrow q_{i}+\\gamma\\left(e_{u i} \\cdot p_{u}-\\lambda q_{i}\\right) \\end{aligned}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y_YxOgwxkY_V"
   },
   "source": [
    "There are many hyperparameters to tune in this algorithm, find a full list of hyperparameters [here](https://surprise.readthedocs.io/en/stable/matrix_factorization.html#surprise.prediction_algorithms.matrix_factorization.SVD)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JKiqI79HkboZ"
   },
   "source": [
    "Below we will be tuning only three hyperparameters:\n",
    "- **n_epochs**: The number of iterations of the SGD algorithm.\n",
    "- **lr_all**: The learning rate for all parameters.\n",
    "- **reg_all**: The regularization term for all parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YKMabS6vkinR",
    "outputId": "e2b1ca53-b81b-4414-b203-553dd6632b87"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.012572420819148\n",
      "{'n_epochs': 30, 'lr_all': 0.01, 'reg_all': 0.2}\n"
     ]
    }
   ],
   "source": [
    "# Set the parameter space to tune\n",
    "param_grid = {'n_epochs': [10, 20, 30], 'lr_all': [0.001, 0.005, 0.01],\n",
    "              'reg_all': [0.2, 0.4, 0.6]}\n",
    "\n",
    "# Performe 3-fold grid-search cross-validation\n",
    "gs_ = GridSearchCV(SVD, param_grid, measures = ['rmse'], cv = 3, n_jobs = -1)\n",
    "\n",
    "# Fitting data\n",
    "gs_.fit(data) \n",
    "\n",
    "# Best RMSE score\n",
    "print(gs_.best_score['rmse'])\n",
    "\n",
    "# Combination of parameters that gave the best RMSE score\n",
    "print(gs_.best_params['rmse'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J_JC-AkbluUe"
   },
   "source": [
    "**Think About It**: How do the parameters affect the performance of the model? Can we improve the performance of the model further? Check the available hyperparameters [here](https://surprise.readthedocs.io/en/stable/matrix_factorization.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IJ4HNSvqlwxv"
   },
   "source": [
    "Once the **grid search** is complete, we can get the **optimal values** for each of those hyperparameters, as shown above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B73CsMNGl0oM"
   },
   "source": [
    "Now, we will **the build final model** by using **tuned values** of the hyperparameters, which we received using grid search cross-validation above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WxXo6uDxl68K",
    "outputId": "12c4f9eb-cc8c-41c9-e9b8-073a9618e94d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.0141\n",
      "Precision:  0.415\n",
      "Recall:  0.635\n",
      "F_1 score:  0.502\n"
     ]
    }
   ],
   "source": [
    "# Building the optimized SVD model using optimal hyperparameters\n",
    "svd_optimized = SVD(n_epochs = 30, lr_all = 0.01, reg_all = 0.2, random_state = 1)\n",
    "\n",
    "# Train the algorithm on the train set\n",
    "svd_optimized = svd_optimized.fit(trainset)\n",
    "\n",
    "# Use the function precision_recall_at_k to compute precision@k, recall@k, F1-Score, and RMSE\n",
    "precision_recall_at_k(svd_optimized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qWgdLjmpl31c"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "*  We observe that **after tuning hyperparameters, the model performance has not improved by much**. We got almost the same values as the baseline model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BmbMKPMBmDmG"
   },
   "source": [
    "Let's now predict a play_count for a user with `user_id = 6958` and `song_id = 1671` with the optimized model as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ki0nEvNvmFtU",
    "outputId": "c18f8476-f8e5-4d0a-dcbd-84640a414902"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 1671       r_ui = 2.00   est = 1.34   {'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=1671, r_ui=2, est=1.3432395286125098, details={'was_impossible': False})"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using svd_algo_optimized model to recommend for userId 6958 and song_id 1671\n",
    "svd_optimized.predict(6958, 1671, r_ui = 2, verbose = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7jwqOW9amUZq",
    "outputId": "37119730-fe0a-40fd-c369-e45c7367e9ee"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 3232       r_ui = None   est = 1.44   {'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=3232, r_ui=None, est=1.4425484461176483, details={'was_impossible': False})"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using svd_algo_optimized model to recommend for userId 6958 and song_id 3232 with unknown baseline rating\n",
    "svd_optimized.predict(6958, 3232, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oyoj515zmMbI"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* With the **optimized matrix factorization model** we got a higher estimated play_count value of '1.34' that is closer to the real play_count unlike the **matrix factorization-based model** where we got an estimated play_count value of '1.27'. Although the two values are far from the actual value.\n",
    "\n",
    "* For the unknown real value, our **optimized matrix factorization-based model** under-estimated the play_count with a value of '1.44', unlike the **matrix factorization-based model** that got an estimated play_count value of '1.56'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gotdoJxSmcMf"
   },
   "source": [
    "### **Get 5 recommendations for the user with user_id = '6958' using the svd_optimized model.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "id": "OatzZtjsmfo9"
   },
   "outputs": [],
   "source": [
    "# Getting top 5 recommendations for user_id 6958 using \"svd_optimized\" algorithm\n",
    "svd_recommendations = get_recommendations(df_final, 6958, 5, svd_optimized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "1X73_UdUmhXi",
    "outputId": "27b04fce-31b9-4ebe-c7d9-dfe1beeda560"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-ceea656c-f3c9-460c-a38c-c0d7f3fa8c4b\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_freq</th>\n",
       "      <th>predicted_play_count</th>\n",
       "      <th>corrected_play_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7224</td>\n",
       "      <td>361</td>\n",
       "      <td>2.601899</td>\n",
       "      <td>2.654530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5653</td>\n",
       "      <td>248</td>\n",
       "      <td>2.108728</td>\n",
       "      <td>2.172228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8324</td>\n",
       "      <td>252</td>\n",
       "      <td>2.014091</td>\n",
       "      <td>2.077085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6450</td>\n",
       "      <td>263</td>\n",
       "      <td>1.952493</td>\n",
       "      <td>2.014155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9942</td>\n",
       "      <td>373</td>\n",
       "      <td>1.940115</td>\n",
       "      <td>1.991893</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ceea656c-f3c9-460c-a38c-c0d7f3fa8c4b')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-ceea656c-f3c9-460c-a38c-c0d7f3fa8c4b button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-ceea656c-f3c9-460c-a38c-c0d7f3fa8c4b');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   song_id  play_freq  predicted_play_count  corrected_play_count\n",
       "1     7224        361              2.601899              2.654530\n",
       "4     5653        248              2.108728              2.172228\n",
       "3     8324        252              2.014091              2.077085\n",
       "2     6450        263              1.952493              2.014155\n",
       "0     9942        373              1.940115              1.991893"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Ranking songs based on above recommendations\n",
    "ranking_songs(svd_recommendations, final_play)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v9zQXLInmmuy"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* We got 5 recommendations for the user with **user_id = 6958** with the **svd_optimized model** and with the **corrected_play_count**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1cKEt4DKmta7"
   },
   "source": [
    "### **Model 4: Cluster Based Recommendation System**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EAugd8kym-3y"
   },
   "source": [
    "In **clustering-based recommendation systems**, we explore the **similarities and differences** in people's tastes in songs based on how they rate different songs. We cluster similar users together and recommend songs to a user based on play_counts from other users in the same cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5tUC2NnhnCGo"
   },
   "source": [
    "- **Co-clustering** is a set of techniques in **Cluster Analysis**. Given some **matrix A, we want to cluster rows of A and columns of A simultaneously**, this is a common task for user-item matrices. \n",
    "\n",
    "- As it clusters both the rows and the columns simultaneously, it is also called **bi-clustering**. To understand the working of the algorithm, let A be m x n matrix, the goal is to generate co-clusters: a subset of rows that exhibit similar behavior across a subset of columns, or vice versa.\n",
    "\n",
    "- Co-clustering is defined as two map functions:\n",
    "rows -> row cluster indexes\n",
    "columns -> column cluster indexes.\n",
    "\n",
    "These map functions are learned simultaneously. It is **different from other clustering techniques** where we cluster **first the rows and then the columns**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "toUi_1J8nJgs",
    "outputId": "c30054b3-ef46-4aaf-f66b-d0b1573df5cd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.0487\n",
      "Precision:  0.397\n",
      "Recall:  0.582\n",
      "F_1 score:  0.472\n"
     ]
    }
   ],
   "source": [
    "# Make baseline clustering model\n",
    "clust_baseline = CoClustering(random_state = 1)\n",
    "\n",
    "# Training the algorithm on the train set\n",
    "clust_baseline.fit(trainset)\n",
    "\n",
    "# Let us compute precision@k, recall@k, and F_1 score with k = 10\n",
    "precision_recall_at_k(clust_baseline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HkpvVErcnZvz"
   },
   "source": [
    "Now, let's **predict the play_count for the user with `user_id = 6958` and the song with `song_id = 1671`** as shown below. Here, the user has already listened the song with song_id 1671."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "R4xxGIlXnb-0",
    "outputId": "0ec53d77-8f3e-4771-e4ff-fff0e302b575"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 1671       r_ui = 2.00   est = 1.29   {'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=1671, r_ui=2, est=1.2941824757363074, details={'was_impossible': False})"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Making prediction for user_id 6958 and song_id 1671\n",
    "clust_baseline.predict(6958, 1671, r_ui = 2, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t-wSM5X9nqcK"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "- The actual play_count for this user-item pair is 2 and the **predicted play_count by this Co-clustering is 1.29 which is not close to the predicted play_count**. The model has under-estimated the play_count. We will try to fix this by tuning the hyperparameters of the model using GridSearchCV."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mMnHgiVNn3iC"
   },
   "source": [
    "Below, we are **predicting the play_count for the same `user_id = 6958`** but for a song with which **this user has not interacted before, i.e., `song_id = 3232`**, as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-xlR6bcon-Mx",
    "outputId": "e680cc35-1627-4e9c-de86-fb590765473a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 3232       r_ui = None   est = 1.48   {'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=3232, r_ui=None, est=1.4785259100797417, details={'was_impossible': False})"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Making prediction for user (userid 6958) for a song(song_id 3232) not heard by the user\n",
    "clust_baseline.predict(6958, 3232, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nSW52LTooA5s"
   },
   "source": [
    "#### **Improving clustering-based recommendation system by tuning its hyper-parameters**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pIUSLwINoalN"
   },
   "source": [
    "Below, we will be tuning hyperparameters for the `CoClustering` algorithm. Let's try to understand the different hyperparameters of this algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c42YqW12oc_8"
   },
   "source": [
    "- **n_cltr_u** (int) – Number of **user clusters**. The default value is 3.\n",
    "- **n_cltr_i** (int) – Number of **item clusters**. The default value is 3.\n",
    "- **n_epochs** (int) – Number of **iteration of the optimization loop**. The default value is 20.\n",
    "- **random_state** (int, RandomState instance from NumPy, or None) – Determines the RNG that will be used for initialization. If int, random_state will be used as a seed for a new RNG. This is useful to get the same initialization over multiple calls to fit(). If RandomState instance, this same instance is used as RNG. If None, the current RNG from NumPy is used. The default value is None.\n",
    "- **verbose** (bool) – If True, the current epoch will be printed. The default value is False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wL2gNO0gofZj",
    "outputId": "ecbc8c05-23e6-4743-d4d2-28efe65d083e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.059275268857786\n",
      "{'n_cltr_u': 5, 'n_cltr_i': 5, 'n_epochs': 10}\n"
     ]
    }
   ],
   "source": [
    "# Set the parameter space to tune\n",
    "param_grid = {'n_cltr_u': [5, 6, 7, 8], 'n_cltr_i': [5, 6, 7, 8], 'n_epochs': [10, 20, 30]}\n",
    "\n",
    "# Performing 3-fold grid search cross-validation\n",
    "gs = GridSearchCV(CoClustering, param_grid, measures = ['rmse'], cv = 3, n_jobs = -1)\n",
    "\n",
    "# Fitting data\n",
    "gs.fit(data)\n",
    "\n",
    "# Best RMSE score\n",
    "print(gs.best_score['rmse'])\n",
    "\n",
    "# Combination of parameters that gave the best RMSE score\n",
    "print(gs.best_params['rmse'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "87Q2-TtjomrW"
   },
   "source": [
    "Once the grid search is **complete**, we can get the **optimal values** for each of those hyperparameters as shown above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "InxH3BuConAH"
   },
   "source": [
    "We will build **final model** by using tuned values of the hyperparameters received after using the Grid search cross-validation above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DvkpoTuoops8"
   },
   "source": [
    "**Think About It**: How do the parameters affect the performance of the model? Can we improve the performance of the model further? Check the available hyperparameters [here](https://surprise.readthedocs.io/en/stable/co_clustering.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lSIZZx3mosg_",
    "outputId": "e52657fa-ea10-4dcd-e6e8-33898921a163"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.0654\n",
      "Precision:  0.394\n",
      "Recall:  0.566\n",
      "F_1 score:  0.465\n"
     ]
    }
   ],
   "source": [
    "# Train the tuned Coclustering algorithm\n",
    "clust_tuned = CoClustering(n_cltr_u = 5, n_cltr_i = 5, n_epochs = 10, random_state = 1)\n",
    "\n",
    "# Training the algorithm on the train set\n",
    "clust_tuned.fit(trainset)\n",
    "\n",
    "# Let us compute precision@k, recall@k, and F_1 score with k = 10\n",
    "precision_recall_at_k(clust_tuned)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HS1Kggioowxv"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* We can see that the **F_1 score** for **tuned co-clustering model** is **0.46** and is slightly lower than the **F_1 score** for the **baseline Co-clustering model** that is **0.47**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z1DG-d1Fo0Tr"
   },
   "source": [
    "Let's now **predict the play_count** for the user with `user_id = 6958` and for the song with `song_id = 1671` as shown below. Here, the user has already listened the song."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5uhdWjGDptNc",
    "outputId": "b9575ef1-0d50-4073-e24b-782ce8f54071"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 1671       r_ui = 2.00   est = 1.91   {'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=1671, r_ui=2, est=1.9108882530486497, details={'was_impossible': False})"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using co_clustering_optimized model to recommend for userId 6958 and song_id 1671\n",
    "clust_tuned.predict(6958, 1671, r_ui = 2, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q18LSkMmp-pP"
   },
   "source": [
    "Below, we are **predicting the play_count for the same `user_id = 6958`** but for a song with which this **user has not interacted before, i.e., `song_id = 3232`**, as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LUKKro8Jp_eT",
    "outputId": "607a4fb0-5415-4c0e-8608-4a32811975e4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user: 6958       item: 3232       r_ui = None   est = 1.24   {'was_impossible': False}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(uid=6958, iid=3232, r_ui=None, est=1.2366916027865822, details={'was_impossible': False})"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use Co_clustering based optimized model to recommend for userId 6958 and song_id 3232 with unknown baseline rating\n",
    "clust_tuned.predict(6958, 3232, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mOOU-_KgqOHn"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* With the **optimized clustering model** we got a higher estimated play_count value of '1.91' that is closer to the real play_count unlike the **clustering-based model** where we got an estimated play_count value of '1.29'.\n",
    "\n",
    "* For the unknown real value, our **optimized clustering model** under-estimated the play_count with a value of '1.24', unlike the **clustering-based model** that got an estimated play_count value of '1.48'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Xqkxd9vOqQEu"
   },
   "source": [
    "#### **Implementing the recommendation algorithm based on optimized CoClustering model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "id": "8RcI_HOgqfQ4"
   },
   "outputs": [],
   "source": [
    "# Getting top 5 recommendations for user_id 6958 using \"Co-clustering based optimized\" algorithm\n",
    "clustering_recommendations = get_recommendations(df_final, 6958, 5, clust_tuned)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mikit4rMqdMO"
   },
   "source": [
    "### **Correcting the play_count and Ranking the above songs**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "qZ8PUn3yqkiF",
    "outputId": "22005519-fbf1-4ed8-ef7b-d1cd6438a615"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-d3782bd7-80f2-46bc-91dc-2ce16e7d7619\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_freq</th>\n",
       "      <th>predicted_play_count</th>\n",
       "      <th>corrected_play_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7224</td>\n",
       "      <td>361</td>\n",
       "      <td>3.711503</td>\n",
       "      <td>3.764135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5653</td>\n",
       "      <td>248</td>\n",
       "      <td>2.903883</td>\n",
       "      <td>2.967383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6860</td>\n",
       "      <td>367</td>\n",
       "      <td>2.691043</td>\n",
       "      <td>2.743242</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>657</td>\n",
       "      <td>335</td>\n",
       "      <td>2.606354</td>\n",
       "      <td>2.660989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8483</td>\n",
       "      <td>275</td>\n",
       "      <td>2.582807</td>\n",
       "      <td>2.643110</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d3782bd7-80f2-46bc-91dc-2ce16e7d7619')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-d3782bd7-80f2-46bc-91dc-2ce16e7d7619 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-d3782bd7-80f2-46bc-91dc-2ce16e7d7619');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   song_id  play_freq  predicted_play_count  corrected_play_count\n",
       "1     7224        361              3.711503              3.764135\n",
       "4     5653        248              2.903883              2.967383\n",
       "0     6860        367              2.691043              2.743242\n",
       "2      657        335              2.606354              2.660989\n",
       "3     8483        275              2.582807              2.643110"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Ranking songs based on the above recommendations\n",
    "ranking_songs(clustering_recommendations, final_play)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6tfW6goHql9G"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* We got 5 recommendations for the user with **user_id = 6958** with the **clust_tuned model** and with the **corrected_play_count**.\n",
    "* The first two songs that are being recommended by this model are also being recommended by the svd optimized model, but with different predicted values. The remaining three songs are different."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6qYaJaI2qoCm"
   },
   "source": [
    "### **Model 5: Content Based Recommendation Systems**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r84g6qirq2sb"
   },
   "source": [
    "In a **content-based recommendation system**, we will use the feature - **text**, i.e., play_count to find similar songs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2VNiVlYpq80F"
   },
   "source": [
    "Text data generally contains pronunciation, stopwords, and non-ASCII characters, which makes it **very noisy**. So, we will first need to **pre-process the text** and then we will **generate features from the text to compute similarities** between the texts/reviews.\n",
    "\n",
    "Let's load the **tags dataset.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cuW-D3bwq_iT"
   },
   "source": [
    "**Think About It:** So far we have only used the play_count of songs to find recommendations but we have other information/features on songs as well. Can we take those song features into account?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "id": "oi1kL4MIrBPV"
   },
   "outputs": [],
   "source": [
    "df_small = df_final"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "GA1QFr_prCnS",
    "outputId": "1f3c9201-65e0-434f-b151-7e2e7e2bf72c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-44617cc8-33c4-447c-97b7-4770a9cf8acc\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_count</th>\n",
       "      <th>title</th>\n",
       "      <th>release</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>year</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>6958</td>\n",
       "      <td>447</td>\n",
       "      <td>1</td>\n",
       "      <td>Daisy And Prudence</td>\n",
       "      <td>Distillation</td>\n",
       "      <td>Erin McKeown</td>\n",
       "      <td>2000</td>\n",
       "      <td>Daisy And Prudence Distillation Erin McKeown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>6958</td>\n",
       "      <td>512</td>\n",
       "      <td>1</td>\n",
       "      <td>The Ballad of Michael Valentine</td>\n",
       "      <td>Sawdust</td>\n",
       "      <td>The Killers</td>\n",
       "      <td>2004</td>\n",
       "      <td>The Ballad of Michael Valentine Sawdust The Ki...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>6958</td>\n",
       "      <td>549</td>\n",
       "      <td>1</td>\n",
       "      <td>I Stand Corrected (Album)</td>\n",
       "      <td>Vampire Weekend</td>\n",
       "      <td>Vampire Weekend</td>\n",
       "      <td>2007</td>\n",
       "      <td>I Stand Corrected (Album) Vampire Weekend Vamp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>204</th>\n",
       "      <td>6958</td>\n",
       "      <td>703</td>\n",
       "      <td>1</td>\n",
       "      <td>They Might Follow You</td>\n",
       "      <td>Tiny Vipers</td>\n",
       "      <td>Tiny Vipers</td>\n",
       "      <td>2007</td>\n",
       "      <td>They Might Follow You Tiny Vipers Tiny Vipers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>205</th>\n",
       "      <td>6958</td>\n",
       "      <td>719</td>\n",
       "      <td>1</td>\n",
       "      <td>Monkey Man</td>\n",
       "      <td>You Know I'm No Good</td>\n",
       "      <td>Amy Winehouse</td>\n",
       "      <td>2007</td>\n",
       "      <td>Monkey Man You Know I'm No Good Amy Winehouse</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-44617cc8-33c4-447c-97b7-4770a9cf8acc')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-44617cc8-33c4-447c-97b7-4770a9cf8acc button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-44617cc8-33c4-447c-97b7-4770a9cf8acc');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "     user_id  song_id  play_count                            title  \\\n",
       "200     6958      447           1               Daisy And Prudence   \n",
       "202     6958      512           1  The Ballad of Michael Valentine   \n",
       "203     6958      549           1        I Stand Corrected (Album)   \n",
       "204     6958      703           1            They Might Follow You   \n",
       "205     6958      719           1                       Monkey Man   \n",
       "\n",
       "                  release      artist_name  year  \\\n",
       "200          Distillation     Erin McKeown  2000   \n",
       "202               Sawdust      The Killers  2004   \n",
       "203       Vampire Weekend  Vampire Weekend  2007   \n",
       "204           Tiny Vipers      Tiny Vipers  2007   \n",
       "205  You Know I'm No Good    Amy Winehouse  2007   \n",
       "\n",
       "                                                  text  \n",
       "200       Daisy And Prudence Distillation Erin McKeown  \n",
       "202  The Ballad of Michael Valentine Sawdust The Ki...  \n",
       "203  I Stand Corrected (Album) Vampire Weekend Vamp...  \n",
       "204      They Might Follow You Tiny Vipers Tiny Vipers  \n",
       "205      Monkey Man You Know I'm No Good Amy Winehouse  "
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Concatenate the \"title\", \"release\", \"artist_name\" columns to create a different column named \"text\"\n",
    "df_small['text'] = df_small['title'] + ' ' + df_small['release'] + ' ' + df_small['artist_name']\n",
    "df_small.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4cY0RDaZrbTl"
   },
   "source": [
    "Now, we will **keep only four columns** - user_id, song_id, play_count, and text. We will drop the duplicate titles from the data and make it the **title column as the index** of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "id": "1NJ7va5FrcXX",
    "outputId": "1cadef17-b208-49d5-ec58-fb77675f512e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-888a89c3-3583-468d-85d0-c4fa0a0f42e8\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>song_id</th>\n",
       "      <th>play_count</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Daisy And Prudence</th>\n",
       "      <td>6958</td>\n",
       "      <td>447</td>\n",
       "      <td>1</td>\n",
       "      <td>Daisy And Prudence Distillation Erin McKeown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>The Ballad of Michael Valentine</th>\n",
       "      <td>6958</td>\n",
       "      <td>512</td>\n",
       "      <td>1</td>\n",
       "      <td>The Ballad of Michael Valentine Sawdust The Ki...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>I Stand Corrected (Album)</th>\n",
       "      <td>6958</td>\n",
       "      <td>549</td>\n",
       "      <td>1</td>\n",
       "      <td>I Stand Corrected (Album) Vampire Weekend Vamp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>They Might Follow You</th>\n",
       "      <td>6958</td>\n",
       "      <td>703</td>\n",
       "      <td>1</td>\n",
       "      <td>They Might Follow You Tiny Vipers Tiny Vipers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Monkey Man</th>\n",
       "      <td>6958</td>\n",
       "      <td>719</td>\n",
       "      <td>1</td>\n",
       "      <td>Monkey Man You Know I'm No Good Amy Winehouse</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-888a89c3-3583-468d-85d0-c4fa0a0f42e8')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-888a89c3-3583-468d-85d0-c4fa0a0f42e8 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-888a89c3-3583-468d-85d0-c4fa0a0f42e8');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                                 user_id  song_id  play_count  \\\n",
       "title                                                           \n",
       "Daisy And Prudence                  6958      447           1   \n",
       "The Ballad of Michael Valentine     6958      512           1   \n",
       "I Stand Corrected (Album)           6958      549           1   \n",
       "They Might Follow You               6958      703           1   \n",
       "Monkey Man                          6958      719           1   \n",
       "\n",
       "                                                                              text  \n",
       "title                                                                               \n",
       "Daisy And Prudence                    Daisy And Prudence Distillation Erin McKeown  \n",
       "The Ballad of Michael Valentine  The Ballad of Michael Valentine Sawdust The Ki...  \n",
       "I Stand Corrected (Album)        I Stand Corrected (Album) Vampire Weekend Vamp...  \n",
       "They Might Follow You                They Might Follow You Tiny Vipers Tiny Vipers  \n",
       "Monkey Man                           Monkey Man You Know I'm No Good Amy Winehouse  "
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select the columns 'user_id', 'song_id', 'play_count', 'title', 'text' from df_small data\n",
    "df_small = df_small[['user_id', 'song_id', 'play_count', 'title', 'text']]\n",
    "\n",
    "# Drop the duplicates from the title column\n",
    "df_small = df_small.drop_duplicates(subset = ['title'])\n",
    "\n",
    "# Set the title column as the index\n",
    "df_small = df_small.set_index('title')\n",
    "\n",
    "# See the first 5 records of the df_small dataset\n",
    "df_small.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HITBqd4SruBq",
    "outputId": "b4167710-0121-47ec-f028-2718ad2b3e9e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                 Daisy And Prudence\n",
       "1    The Ballad of Michael Valentine\n",
       "2          I Stand Corrected (Album)\n",
       "3              They Might Follow You\n",
       "4                         Monkey Man\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create the series of indices from the data\n",
    "indices = pd.Series(df_small.index)\n",
    "\n",
    "indices[ : 5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PYH1i5TMrxOP"
   },
   "source": [
    "**Now, let's process the text data and create features to find the similarity between songs.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c7o1S2vwrzGG"
   },
   "source": [
    "### **Loading libraries to handle the text data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mpDYdPbnr1S-",
    "outputId": "f79b61fb-697f-4b25-83ab-605e3f17c688"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
      "[nltk_data] Downloading package wordnet to /root/nltk_data...\n"
     ]
    }
   ],
   "source": [
    "# Importing necessary packages to work with text data\n",
    "import nltk\n",
    "\n",
    "# Download punkt library\n",
    "nltk.download(\"punkt\")\n",
    "\n",
    "# Download stopwords library\n",
    "nltk.download(\"stopwords\")\n",
    "\n",
    "# Download wordnet \n",
    "nltk.download(\"wordnet\")\n",
    "\n",
    "# Import regular expression\n",
    "import re\n",
    "\n",
    "# Import word_tokenizer\n",
    "from nltk import word_tokenize\n",
    "\n",
    "# Import WordNetLemmatizer\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "# Import stopwords\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "# Import CountVectorizer and TfidfVectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yBuizA6Sr5Tn",
    "outputId": "07afc625-7267-4495-c5be-6c1b5147e904"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package omw-1.4 to /root/nltk_data...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('omw-1.4')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kNrJpUyTsAu2"
   },
   "source": [
    "We will create a **function to pre-process the text data**. Before that, let's see some **terminology**.\n",
    "- **stopwords:** A stop word is a commonly used word (such as “the”, “a”, “an”, or “in”) that does not contain information in the text and can be ignored.\n",
    "- **Lemmatization:** Lemmatization in linguistics is the process of grouping together the inflected forms of a word so they can be analyzed as a single item. For example, runs, running, and ran are all forms of the word run, therefore run is the lemma of all these words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "id": "XA6jvZLbsEOj"
   },
   "outputs": [],
   "source": [
    "# Function to tokenize the text\n",
    "def tokenize(text):\n",
    "    \n",
    "    text = re.sub(r\"[^a-zA-Z]\",\" \", text.lower())\n",
    "    \n",
    "    tokens = word_tokenize(text)\n",
    "    \n",
    "    words = [word for word in tokens if word not in stopwords.words('english')]  # Use stopwords of english\n",
    "    \n",
    "    text_lems = [WordNetLemmatizer().lemmatize(lem).strip() for lem in words]\n",
    "\n",
    "    return text_lems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hhbJdGF6sOKN"
   },
   "source": [
    "### **Feature Extraction**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "peRugVjLsOiu"
   },
   "source": [
    "Below are some of the ways to extract features from texts:\n",
    "- Bag of words\n",
    "- TF-IDF\n",
    "- One hot encoding\n",
    "- Word vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZKQqmW8HsVh7"
   },
   "source": [
    "Here, we will be using **TF-IDF** as a feature extraction technique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "id": "CQlG2eswsGai"
   },
   "outputs": [],
   "source": [
    "# Create tfidf vectorizer \n",
    "tfidf = TfidfVectorizer(tokenizer = tokenize)\n",
    "\n",
    "# Fit_transfrom the above vectorizer on the text column and then convert the output into an array\n",
    "song_tfidf = tfidf.fit_transform(df_small['text'].values).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "id": "r2dw7HE-sIkP",
    "outputId": "ee5916fd-ccc6-4627-d88f-b095404e1036"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-ddff34c2-1f68-4612-b803-0d726c7e71fc\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>1427</th>\n",
       "      <th>1428</th>\n",
       "      <th>1429</th>\n",
       "      <th>1430</th>\n",
       "      <th>1431</th>\n",
       "      <th>1432</th>\n",
       "      <th>1433</th>\n",
       "      <th>1434</th>\n",
       "      <th>1435</th>\n",
       "      <th>1436</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>556</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>557</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>558</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>559</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>560</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>561 rows × 1437 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ddff34c2-1f68-4612-b803-0d726c7e71fc')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-ddff34c2-1f68-4612-b803-0d726c7e71fc button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-ddff34c2-1f68-4612-b803-0d726c7e71fc');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "     0     1     2     3     4     5     6     7     8     9     ...  1427  \\\n",
       "0     0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   0.0   \n",
       "1     0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   0.0   \n",
       "2     0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   0.0   \n",
       "3     0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   0.0   \n",
       "4     0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   0.0   \n",
       "..    ...   ...   ...   ...   ...   ...   ...   ...   ...   ...  ...   ...   \n",
       "556   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   0.0   \n",
       "557   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   0.0   \n",
       "558   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   0.0   \n",
       "559   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   0.0   \n",
       "560   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   0.0   \n",
       "\n",
       "     1428  1429  1430  1431  1432  1433  1434  1435  1436  \n",
       "0     0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "1     0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "2     0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "3     0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "4     0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "..    ...   ...   ...   ...   ...   ...   ...   ...   ...  \n",
       "556   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "557   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "558   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "559   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "560   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "\n",
       "[561 rows x 1437 columns]"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Making the DataFrame of movie_tfidf data\n",
    "pd.DataFrame(song_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j26f-OoxsYq5"
   },
   "source": [
    "We have **extracted features from the text data**. Now, we can find similarities between songs using these features. We will use cosine similarity to calculate the similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "jzYQBzSbscIv",
    "outputId": "d2036b75-64d9-44bd-def7-7ce3249e7c1e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 1., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 1., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [0., 0., 0., ..., 1., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 1., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 1.]])"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the cosine similarity for the tfidf above output\n",
    "similar_songs = cosine_similarity(song_tfidf, song_tfidf)\n",
    "\n",
    "# Let us see the above array\n",
    "similar_songs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pVJy07ixsoQx"
   },
   "source": [
    "**Finally, let's create a function to find most similar songs to recommend for a given song.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "id": "xjJSQSwUsrqV"
   },
   "outputs": [],
   "source": [
    "# Function that takes in song title as input and returns the top 10 recommended songs\n",
    "def recommendations(title, similar_songs):\n",
    "    \n",
    "    recommended_songs = []\n",
    "    \n",
    "    # Getting the index of the song that matches the title\n",
    "    idx = indices[indices == title].index[0]\n",
    "\n",
    "    # Creating a Series with the similarity scores in descending order\n",
    "    score_series = pd.Series(similar_songs[idx]).sort_values(ascending = False)\n",
    "\n",
    "    # Getting the indexes of the 10 most similar songs\n",
    "    top_10_indexes = list(score_series.iloc[1 : 11].index)\n",
    "    print(top_10_indexes)\n",
    "    \n",
    "    # Populating the list with the titles of the best 10 matching songs\n",
    "    for i in top_10_indexes:\n",
    "        recommended_songs.append(list(df_small.index)[i])\n",
    "        \n",
    "    return recommended_songs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M6SJ09zLsxRx"
   },
   "source": [
    "### **Recommending 10 songs similar to Learn to Fly**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UCyUdD8Fsztj",
    "outputId": "8517e3bb-a028-4d45-a609-bf77eaa53654"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[509, 234, 423, 345, 394, 370, 371, 372, 373, 375]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Everlong',\n",
       " 'The Pretender',\n",
       " 'Nothing Better (Album)',\n",
       " 'From Left To Right',\n",
       " 'Lifespan Of A Fly',\n",
       " 'Under The Gun',\n",
       " 'I Need A Dollar',\n",
       " 'Feel The Love',\n",
       " 'All The Pretty Faces',\n",
       " 'Bones']"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make the recommendation for the song with title 'Learn To Fly'\n",
    "recommendations('Learn To Fly', similar_songs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mveoOnoxs3mO"
   },
   "source": [
    "**Observations and Insights:**\n",
    "\n",
    "* The **Learn To Fly song** is from **Foo Fighters** and belongs to **Alternative/Rock and Pop Rock** genres, and the **majority of the recommendations** lie in one or both of these genres. \n",
    "* Also, the band or the artists that are recommended make music of similar genres.\n",
    "* We can imply that the resulting recommendation system is working well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vm6WGkPIs7y9"
   },
   "source": [
    "## **Conclusion and Recommendations:**\n",
    "\n",
    "In this case study, we built recommendation systems using four different algorithms. They are as follows:\n",
    "  - Rank-based using averages\n",
    "  - User-User similarity-based collaborative filtering\n",
    "  - Item-Item similarity-based collaborative filtering\n",
    "  - Model-based collaborative filtering (matrix factorization)\n",
    "  - Clustering-based recommendation systems\n",
    "  - Content-based recommendation systems\n",
    "  \n",
    "- The **surprise** library was used to demonstrate**\"user-user similarity-based collaborative filtering,\" \"item-item similarity-based collaborative filtering,\" \"model-based collaborative filtering (matrix factorization) and clustering-based recommendation systems\"** algorithms. For these algorithms, grid search cross-validation is used to find the optimal hyperparameters for the data, and related predictions are created using the optimal hyperparameters.\n",
    "\n",
    "- **For performance evaluation** of these models **precision@k and recall@k** are used. Using these two metrics, the **F_1 score** is calculated for each working model. \n",
    "- Overall, the optimized user-user similarity-based recommendation system has given the best performance in terms of the F1-Score with a score of 0.525.\n",
    "\n",
    "- We can try to further improve the performance of these models using hyperparameter tuning. \n",
    "- We can also combine different recommendation techniques to build hybrid recommendation systems."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}