Movie-recommendation-system.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d7aa1b0ad6979877450f9cd89e1e37289b51cf6e"
   },
   "source": [
    "# **The Age of Recommender Systems**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
    "collapsed": true
   },
   "source": [
    "The rapid growth of data collection has led to a new era of information. Data is being used to create more efficient systems and this is where Recommendation Systems come into play.  Recommendation Systems are a type of **information filtering systems** as they improve the quality of search results and provides items that are more relevant to the search item or are realted to the search history of the user.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "65dbae55f1e6e06c5fa7251f8ddae887d3fbf480"
   },
   "source": [
    "They are used to predict the **rating** or **preference** that a user would give to an item. Almost every major tech company has applied them in some form or the other: Amazon uses it to suggest products to customers, YouTube uses it to decide which video to play next on autoplay, and Facebook uses it to recommend pages to like and people to follow. \n",
    "Moreover,  companies like Netflix and Spotify  depend highly on the effectiveness of their recommendation engines for their business and sucees."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "af1e8d514096cde4abeae2d65a44cfdb01228d77"
   },
   "source": [
    "![](https://i.kinja-img.com/gawker-media/image/upload/s--e3_2HgIC--/c_scale,f_auto,fl_progressive,q_80,w_800/1259003599478673704.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f34fc4dcfba717692c620e1fdfa502ee910c6365"
   },
   "source": [
    "In this kernel we'll be building a baseline Movie Recommendation System using [TMDB 5000 Movie Dataset](https://www.kaggle.com/tmdb/tmdb-movie-metadata). For novices like me this kernel will pretty much serve as a foundation in recommendation systems and will provide you with something to start with. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1fa8fc3cb8348853bb51472b248134a59c24bf60"
   },
   "source": [
    "**So let's go!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f9a9405b7e81c1da449bd2e96c2849fb86caa614"
   },
   "source": [
    "There are basically three types of recommender systems:-\n",
    "\n",
    "> *  **Demographic Filtering**- They offer generalized recommendations to every user, based on movie popularity and/or genre. The System recommends the same movies to users with similar demographic features. Since each user is different , this approach is considered to be too simple. The basic idea behind this system is that movies that are more popular and critically acclaimed will have a higher probability of being liked by the average audience.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "60a2df15abf82ba21918e3a42cb0ee46d22fa764"
   },
   "source": [
    "> *  **Content Based Filtering**- They suggest similar items based on a particular item. This system uses item metadata, such as genre, director, description, actors, etc. for movies, to make these recommendations. The general idea behind these recommender systems is that if a person liked a particular item, he or she will also like an item that is similar to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b7e719fcc502c59f098a51ae35e2ceb6d7cdfe4e"
   },
   "source": [
    "> *  **Collaborative Filtering**- This system matches persons with similar interests and provides recommendations based on this matching. Collaborative filters do not require item metadata like its content-based counterparts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6b418588e3f9139f74cb3a9546f5dca49729579b"
   },
   "source": [
    "Let's load the data now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_uuid": "c1fdd129c1cbab68ae3e6bf2062575f01f80b87c"
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np \n",
    "df1=pd.read_csv('../input/tmdb-movie-metadata/tmdb_5000_credits.csv')\n",
    "df2=pd.read_csv('../input/tmdb-movie-metadata/tmdb_5000_movies.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "402a28d17c13bba3f2060d72c2ff75f5377a9f01"
   },
   "source": [
    "The first dataset contains the following features:-\n",
    "\n",
    "* movie_id - A unique identifier for each movie.\n",
    "* cast - The name of lead and supporting actors.\n",
    "* crew - The name of Director, Editor, Composer, Writer etc.\n",
    "\n",
    "The second dataset has the following features:- \n",
    "\n",
    "* budget - The budget in which the movie was made.\n",
    "* genre - The genre of the movie, Action, Comedy ,Thriller etc.\n",
    "* homepage - A link to the homepage of the movie.\n",
    "* id - This is infact the movie_id as in the first dataset.\n",
    "* keywords - The keywords or tags related to the movie.\n",
    "* original_language - The language in which the movie was made.\n",
    "* original_title - The title of the movie before translation or adaptation.\n",
    "* overview - A brief description of the movie.\n",
    "* popularity - A numeric quantity specifying the movie popularity.\n",
    "* production_companies - The production house of the movie.\n",
    "* production_countries - The country in which it was produced.\n",
    "* release_date - The date on which it was released.\n",
    "* revenue - The worldwide revenue generated by the movie.\n",
    "* runtime - The running time of the movie in minutes.\n",
    "* status - \"Released\" or \"Rumored\".\n",
    "* tagline - Movie's tagline.\n",
    "* title - Title of the movie.\n",
    "* vote_average -  average ratings the movie recieved.\n",
    "* vote_count - the count of votes recieved.\n",
    "\n",
    "Let's join the two dataset on the 'id' column\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_uuid": "c87bda9d56a936be126d03eda0bc743ee35be461"
   },
   "outputs": [],
   "source": [
    "df1.columns = ['id','tittle','cast','crew']\n",
    "df2= df2.merge(df1,on='id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e88ed16f798ec3a094fce8aaf8a971f7b2aae83e"
   },
   "source": [
    "Just a peak at our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_uuid": "71d266ed92947c51acf07189d3b42379134ef6e7"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>keywords</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>production_countries</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>tittle</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>237000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.avatarmovie.com/</td>\n",
       "      <td>19995</td>\n",
       "      <td>[{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...</td>\n",
       "      <td>en</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>150.437577</td>\n",
       "      <td>[{\"name\": \"Ingenious Film Partners\", \"id\": 289...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2009-12-10</td>\n",
       "      <td>2787965087</td>\n",
       "      <td>162.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Enter the World of Pandora.</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>7.2</td>\n",
       "      <td>11800</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>300000000</td>\n",
       "      <td>[{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...</td>\n",
       "      <td>http://disney.go.com/disneypictures/pirates/</td>\n",
       "      <td>285</td>\n",
       "      <td>[{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...</td>\n",
       "      <td>en</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>139.082615</td>\n",
       "      <td>[{\"name\": \"Walt Disney Pictures\", \"id\": 2}, {\"...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2007-05-19</td>\n",
       "      <td>961000000</td>\n",
       "      <td>169.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>At the end of the world, the adventure begins.</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>6.9</td>\n",
       "      <td>4500</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>245000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.sonypictures.com/movies/spectre/</td>\n",
       "      <td>206647</td>\n",
       "      <td>[{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...</td>\n",
       "      <td>en</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>107.376788</td>\n",
       "      <td>[{\"name\": \"Columbia Pictures\", \"id\": 5}, {\"nam...</td>\n",
       "      <td>[{\"iso_3166_1\": \"GB\", \"name\": \"United Kingdom\"...</td>\n",
       "      <td>2015-10-26</td>\n",
       "      <td>880674609</td>\n",
       "      <td>148.0</td>\n",
       "      <td>[{\"iso_639_1\": \"fr\", \"name\": \"Fran\\u00e7ais\"},...</td>\n",
       "      <td>Released</td>\n",
       "      <td>A Plan No One Escapes</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>6.3</td>\n",
       "      <td>4466</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>250000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...</td>\n",
       "      <td>http://www.thedarkknightrises.com/</td>\n",
       "      <td>49026</td>\n",
       "      <td>[{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...</td>\n",
       "      <td>en</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>112.312950</td>\n",
       "      <td>[{\"name\": \"Legendary Pictures\", \"id\": 923}, {\"...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2012-07-16</td>\n",
       "      <td>1084939099</td>\n",
       "      <td>165.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>The Legend Ends</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>7.6</td>\n",
       "      <td>9106</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>260000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://movies.disney.com/john-carter</td>\n",
       "      <td>49529</td>\n",
       "      <td>[{\"id\": 818, \"name\": \"based on novel\"}, {\"id\":...</td>\n",
       "      <td>en</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>43.926995</td>\n",
       "      <td>[{\"name\": \"Walt Disney Pictures\", \"id\": 2}]</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2012-03-07</td>\n",
       "      <td>284139100</td>\n",
       "      <td>132.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Lost in our world, found in another.</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>6.1</td>\n",
       "      <td>2124</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[{\"cast_id\": 5, \"character\": \"John Carter\", \"c...</td>\n",
       "      <td>[{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      budget                        ...                                                                       crew\n",
       "0  237000000                        ...                          [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...\n",
       "1  300000000                        ...                          [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...\n",
       "2  245000000                        ...                          [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...\n",
       "3  250000000                        ...                          [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...\n",
       "4  260000000                        ...                          [{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...\n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ee603279675033fc397f0c94738e20b34f35312b"
   },
   "source": [
    "# **Demographic Filtering** -\n",
    "   Before getting started with this  -\n",
    "* we need a metric to score or rate movie \n",
    "* Calculate the score for every movie \n",
    "* Sort the scores and recommend the best rated movie to the users.\n",
    "\n",
    "We can use the average ratings of the movie as the score but using this won't be fair enough since a movie with 8.9 average rating and only 3 votes cannot be considered better than the movie with 7.8 as as average rating but 40 votes.\n",
    "So, I'll be using IMDB's weighted rating (wr) which is given as :-\n",
    "\n",
    "![](https://image.ibb.co/jYWZp9/wr.png)\n",
    "where,\n",
    "* v is the number of votes for the movie;\n",
    "* m is the minimum votes required to be listed in the chart;\n",
    "* R is the average rating of the movie; And\n",
    "* C is the mean vote across the whole report\n",
    "\n",
    "We already have v(**vote_count**) and R (**vote_average**) and C can be calculated as "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_uuid": "5799b99c5e5ed5b7723ae8b31e1fc9fb1e7b89ec"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.092171559442011"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "C= df2['vote_average'].mean()\n",
    "C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "02fa8642f75bcef6d4a79e029af6dfeebf19e3a5"
   },
   "source": [
    "So, the mean rating for all the movies is approx 6 on a scale of 10.The next step is to determine an appropriate value for m, the minimum votes required to be listed in the chart. We will use 90th percentile as our cutoff. In other words, for a movie to feature in the charts, it must have more votes than at least 90% of the movies in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_uuid": "f2f1eaff1e4349b5d2d11dd5ce79c19a85561148"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1838.4000000000015"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m= df2['vote_count'].quantile(0.9)\n",
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b77dea5a38ca2c399e3abeac1487e784fe146078"
   },
   "source": [
    "Now, we can filter out the movies that qualify for the chart "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_uuid": "a22008df6d81d3b716d39a56efd3d547345bfbce"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(481, 23)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_movies = df2.copy().loc[df2['vote_count'] >= m]\n",
    "q_movies.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "cf1fe5a3692caee41a6e7a74c3fde6aeb8a62947"
   },
   "source": [
    "We see that there are 481 movies which qualify to be in this list. Now, we need to calculate our metric for each qualified movie. To do this, we will define a function, **weighted_rating()** and define a new feature **score**, of which we'll calculate the value by applying this function to our DataFrame of qualified movies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "_uuid": "bb680ed0fb1c3020785d34152c57c6e2279d4424"
   },
   "outputs": [],
   "source": [
    "def weighted_rating(x, m=m, C=C):\n",
    "    v = x['vote_count']\n",
    "    R = x['vote_average']\n",
    "    # Calculation based on the IMDB formula\n",
    "    return (v/(v+m) * R) + (m/(m+v) * C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "_uuid": "d2d189929715237ab19a18fb8747239b86092968"
   },
   "outputs": [],
   "source": [
    "# Define a new feature 'score' and calculate its value with `weighted_rating()`\n",
    "q_movies['score'] = q_movies.apply(weighted_rating, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "4cdd60c146173606146ec4fc3a1c9d8c184cb81c"
   },
   "source": [
    "Finally, let's sort the DataFrame based on the score feature and output the title, vote count, vote average and weighted rating or score of the top 10 movies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "_uuid": "a9a9fc3810ea67c31908bbdf8bb930daa918102b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1881</th>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>8205</td>\n",
       "      <td>8.5</td>\n",
       "      <td>8.059258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>662</th>\n",
       "      <td>Fight Club</td>\n",
       "      <td>9413</td>\n",
       "      <td>8.3</td>\n",
       "      <td>7.939256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>12002</td>\n",
       "      <td>8.2</td>\n",
       "      <td>7.920020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3232</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>8428</td>\n",
       "      <td>8.3</td>\n",
       "      <td>7.904645</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>Inception</td>\n",
       "      <td>13752</td>\n",
       "      <td>8.1</td>\n",
       "      <td>7.863239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3337</th>\n",
       "      <td>The Godfather</td>\n",
       "      <td>5893</td>\n",
       "      <td>8.4</td>\n",
       "      <td>7.851236</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>Interstellar</td>\n",
       "      <td>10867</td>\n",
       "      <td>8.1</td>\n",
       "      <td>7.809479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>809</th>\n",
       "      <td>Forrest Gump</td>\n",
       "      <td>7927</td>\n",
       "      <td>8.2</td>\n",
       "      <td>7.803188</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>329</th>\n",
       "      <td>The Lord of the Rings: The Return of the King</td>\n",
       "      <td>8064</td>\n",
       "      <td>8.1</td>\n",
       "      <td>7.727243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1990</th>\n",
       "      <td>The Empire Strikes Back</td>\n",
       "      <td>5879</td>\n",
       "      <td>8.2</td>\n",
       "      <td>7.697884</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              title    ...        score\n",
       "1881                       The Shawshank Redemption    ...     8.059258\n",
       "662                                      Fight Club    ...     7.939256\n",
       "65                                  The Dark Knight    ...     7.920020\n",
       "3232                                   Pulp Fiction    ...     7.904645\n",
       "96                                        Inception    ...     7.863239\n",
       "3337                                  The Godfather    ...     7.851236\n",
       "95                                     Interstellar    ...     7.809479\n",
       "809                                    Forrest Gump    ...     7.803188\n",
       "329   The Lord of the Rings: The Return of the King    ...     7.727243\n",
       "1990                        The Empire Strikes Back    ...     7.697884\n",
       "\n",
       "[10 rows x 4 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Sort movies based on score calculated above\n",
    "q_movies = q_movies.sort_values('score', ascending=False)\n",
    "\n",
    "#Print the top 15 movies\n",
    "q_movies[['title', 'vote_count', 'vote_average', 'score']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7625c6d5bd647f4f75eafd7880b7ca2d07938cf7"
   },
   "source": [
    "Hurray! We have made our first(though very basic) recommender.\n",
    "Under the **Trending Now** tab of these systems we find movies that are very popular and they can just be obtained by sorting the dataset by the popularity column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_uuid": "207f7058f92698b5fd776f7771a3ac0cc2928bf1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,1,'Popular Movies')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAy8AAAEWCAYAAABvx6h8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xm4XVV9//H3hwSFMAUJghAkWq01DAaIiggW1A4OCFasIFXgZ6G0gjj9CtYJWkdqSwWslKKiEAZBREr9OYCgyCQJAYKg1SIqkwoCyiACfn9/nHXlcLlTkpucu8n79Tz3yd5rr73W2of9HPLJWnvfVBWSJEmSNNWtNugBSJIkSdJEGF4kSZIkdYLhRZIkSVInGF4kSZIkdYLhRZIkSVInGF4kSZIkdYLhRZKkYZLsnOSmQY9jaST5bpKdBz0OSVqRDC+SpCktyY1J7k9yT5KfJTkxydqDHtfSSHJ4kkpyyLDyQ1r54cvbR1VtUVUXLm87kjSVGV4kSV2wa1WtDWwLzAfeM+DxjCrJ9FEO/Q/wxmFl+7RySdIEGF4kSZ1RVTcD/w/YEiDJJknOSfLLJD9Msv9Q3TbbcWaS05P8OsmVSZ7Td7ySPKNv/8QkHxip3ySHJfnf1s51SV7dd2zfJBcnOSrJHcDhowz/CmBGki3aeVsAa7Ty/r72b9fyy3Ztm7TyTyb52LC6X0ry9rZ9Y5KXtu3V+sZ8R5LPJ3lSO7ZGkpNb+V1Jrkiy0VifuyRNFYYXSVJnJNkMeDmwuBWdBtwEbALsAXwoyYv7TtkNOAN4EnAKcHaS1Zeh6/8FdgLWA44ATk7ylL7jzwduADYCPjhGOyfxyOzLPm3/99rYPwz8JfAU4Mf0rhHgVOB1SdLqrg/8ad/xfgcDuwN/TO+zuRP4RF+/6wGbARsABwL3jzFmSZoyDC+SpC44O8ldwLeBb9ILKZsBLwQOrarfVNVVwAk8emnWoqo6s6oeBP6V3kzH9kvbeVWdUVW3VNXvqup04AfA8/qq3FJVx1TVQ1U1VhA4GdirBag9236/vYFPV9WVVfUA8C7gBUnmABcBRS9EQS+sXVpVt4zQz4HAu6vqptbO4cAebUnbg/RCyzOq6uGqWlRVv5roZyFJg2R4kSR1we5VNbOqNq+qv2sBYRPgl1X16756PwY27dv/6dBGVf2OR2ZplkqSNya5qi2zuovesrVZI/Uzlqr6CfBD4EPAD6pq+HmbtGsYqn8PcAewaVUVvVmWvdrh1wMLRulqc+CLfeO9HniY3szQScBXgdOS3JLkyGWcjZKklc7wIknqqluAJyVZp6/sqcDNffubDW0kWQ2Y3c4DuA+Y0Vd345E6SbI58J/AQcAGVTUTuBZIX7VainF/DnhH+3O4W+gFj6G+16I3SzJ0TafSm0HZnN5StS+M0sdPgZe1wDf0s0ZV3VxVD1bVEVU1F9gBeCWPfZGAJE1JhhdJUie1WYtLgA+3h9C3Bt7Eo5dibZfkL9pyqbcCDwCXtWNXAa9PMi3Jn9N7PmQka9ELJ78ASLIf7YUBy+h0es+qfH6EY6cC+yWZl+SJ9GZoLq+qGwGqajFwO73lcV+tqrtG6eM44IMt5JBkwyS7te1dkmyVZBrwK3rLyH63HNcjSSuN4UWS1GV7AXPozVh8EXh/VZ3Xd/xLwOvoPbD+BuAv2vMvAIcAuwJ30XvW5OyROqiq64B/AS4FfgZsBVy8rAOuqvur6ryRno1pY38vvRmVW4E/oPdsTL9TgJe2P0fzceAc4GtJfk0vsD2/HdsYOJNecLme3jNEJ43UiCRNNektoZUk6fGl/eLHZ1TVXw16LJKkyeHMiyRJkqROMLxIkiRJ6gSXjUmSJEnqBGdeJEmSJHXC9EEPQFPbrFmzas6cOYMehiRJkh7HFi1adHtVbThePcOLxjRnzhwWLlw46GFIkiTpcSzJjydSz2VjkiRJkjrB8CJJkiSpEwwvkiRJkjrB8CJJkiSpEwwvkiRJkjrB8CJJkiSpEwwvkiRJkjrB8CJJkiSpE/wllRrTbfc9xEcW3z7oYUiSJGkFOmybWYMewoQ48yJJkiSpEwwvkiRJkjrB8CJJkiSpEwwvkiRJkjrB8CJJkiSpEwwvkiRJkjrB8CJJkiSpEwwvkiRJkjrB8CJJkiSpEwwvkiRJkjrB8LISJakkJ/ftT0/yiyTntv1XJTlsnDY2SXLmih6rJEmSNNVMH/QAVjH3AlsmWbOq7gf+BLh56GBVnQOcM1YDVXULsMcKHaUkSZI0BTnzsvJ9GXhF294LOHXoQJJ9kxzbtk9McnSSS5LckGSPVj4nybVte40kn0myJMniJLv0tXNWkq8k+UGSI1v5tNbute2ct63E65YkSZKWizMvK99pwPvaUrGtgU8DO41S9ynAjsAf0ZuRGb5c7M1AVdVWSf4I+FqSP2zH5gHbAA8A309yDPBkYNOq2hIgycyROk1yAHAAwMyNZy/TRUqSJEmTzZmXlayqrgHm0Jt1+fI41c+uqt9V1XXARiMc3xE4ubX7PeDHwFB4Ob+q7q6q3wDXAZsDNwBPT3JMkj8HfjXKGI+vqvlVNX+t9TdYuguUJEmSVhDDy2CcA3yMviVjo3igbztL2Uf/uQ8D06vqTuA5wIXAgcAJS9mmJEmSNDCGl8H4NHBEVS1ZznYuAvYGaMvFngp8f7TKSWYBq1XVF4D3ANsuZ/+SJEnSSuMzLwNQVTcBR09CU/8OfDLJEuAhYN+qeiAZdZJmU+AzSYZC67smYQySJEnSSpGqGvQYNIXNnjuvDlpw3qCHIUmSpBXosG1mDbT/JIuqav549Vw2JkmSJKkTDC+SJEmSOsHwIkmSJKkTDC+SJEmSOsHwIkmSJKkTDC+SJEmSOsHwIkmSJKkTDC+SJEmSOmH6oAegqW3jGdMH/kuLJEmSJHDmRZIkSVJHGF4kSZIkdYLhRZIkSVInGF4kSZIkdYLhRZIkSVIn+LYxjem2+x7iI4tvH/QwJElSR/iWUq1IzrxIkiRJ6gTDiyRJkqROMLxIkiRJ6gTDiyRJkqROMLxIkiRJ6gTDiyRJkqROMLxIkiRJ6gTDiyRJkqROMLxIkiRJ6gTDiyRJkqROMLwshST3TKDOW5PMmKT+dk6ywwTq7Zvk2LZ9eJJ3Tkb/kiRJ0lRieJl8bwWWKrwkmTbKoZ2BccPL8kgyfUW2L0mSJE0Ww8syaDMiFyY5M8n3kixIz1uATYALklzQ6v5pkkuTXJnkjCRrt/Ibk3w0yZXAa5O8Jcl1Sa5JclqSOcCBwNuSXJVkpyQbJvlCkivazwvHGef+rd7V7bwZrfzEJMcluRw4csV9UpIkSdLk8V/dl902wBbALcDFwAur6ugkbwd2qarbk8wC3gO8tKruTXIo8HbgH1sbd1TVtgBJbgGeVlUPJJlZVXclOQ64p6o+1uqcAhxVVd9O8lTgq8CzxxjjWVX1n+3cDwBvAo5px2YDO1TVw8NPSnIAcADAzI1nL+PHI0mSJE0uw8uy+05V3QSQ5CpgDvDtYXW2B+YCFycBeAJwad/x0/u2rwEWJDkbOHuUPl8KzG1tAaw7NJMzii1baJkJrE0v7Aw5Y6TgAlBVxwPHA8yeO6/GaF+SJElaaQwvy+6Bvu2HGfmzDPD1qtprlDbu7dt+BfAiYFfg3Um2GqH+asD2VfWbR3XySJgZ7kRg96q6Osm+9J6hGalvSZIkacrzmZfJ92tgnbZ9GfDCJM8ASLJWkj8cfkKS1YDNquoC4FBgPXozJf1tAXwNOLjvvHnjjGUd4NYkqwN7L9vlSJIkSVOD4WXyHQ98JckFVfULYF/g1CTX0Fsy9kcjnDMNODnJEmAxcHRV3QX8F/DqoQf2gbcA89tD/dfRe6B/LO8FLqf3TM73JuHaJEmSpIFJlY80aHSz586rgxacN+hhSJKkjjhsm1mDHoI6KMmiqpo/Xj1nXiRJkiR1guFFkiRJUicYXiRJkiR1guFFkiRJUicYXiRJkiR1guFFkiRJUicYXiRJkiR1guFFkiRJUidMH/QANLVtPGO6v2xKkiRJU4IzL5IkSZI6wfAiSZIkqRMML5IkSZI6wfAiSZIkqRMML5IkSZI6wbeNaUy33fcQH1l8+6CHIUmaJL5BUlKXOfMiSZIkqRMML5IkSZI6wfAiSZIkqRMML5IkSZI6wfAiSZIkqRMML5IkSZI6wfAiSZIkqRMML5IkSZI6wfAiSZIkqRMML5IkSZI6wfAySZI8nOSqJN9NcnWSdySZlM83yeFJ3jkZbfW1uW+SYyezTUmSJGlFmj7oATyO3F9V8wCSPBk4BVgXeP9ARyVJkiQ9TjjzsgJU1c+BA4CD0jMtyT8nuSLJNUn+BiDJ2knOT3JlkiVJdhtqI8m7k/xPkm8Dz+orvzDJx9ssz7VJntfKn5Tk7Nb+ZUm2HqtckiRJ6hpnXlaQqrohyTTgycBuwN1V9dwkTwQuTvI14KfAq6vqV0lmAZclOQfYFtgTmEfvv9GVwKK+5mdU1bwkLwI+DWwJHAEsrqrdk7wY+Fw7f7TyUSU5gF74YubGsyfl85AkSZKWl+Fl5fhTYOske7T99YBnAjcBH2oh5HfApsBGwE7AF6vqPoAWaPqdClBV30qybpKZwI7Aa1r5N5JskGTdMcpHVVXHA8cDzJ47r5bv0iVJkqTJYXhZQZI8HXgY+DkQ4OCq+uqwOvsCGwLbVdWDSW4E1phA88MDhQFDkiRJj3s+87ICJNkQOA44tqoK+Crwt0lWb8f/MMla9GZgft6Cyy7A5q2JbwG7J1kzyTrArsO6eF1rZ0d6y9HuBi4C9m7lOwO3V9WvxiiXJEmSOsWZl8mzZpKrgNWBh4CTgH9tx04A5gBXJgnwC2B3YAHwX0mWAAuB7wFU1ZVJTgeupjdzc8Wwvn6TZHHr6/+0ssOBTye5BrgP2GecckmSJKlT0psYUFckuRB4Z1UtXBn9zZ47rw5acN7K6EqStBIcts2sQQ9Bkh4jyaKqmj9ePZeNSZIkSeoEl411TFXtPOgxSJIkSYPgzIskSZKkTjC8SJIkSeoEw4skSZKkTjC8SJIkSeoEw4skSZKkTjC8SJIkSeoEX5WsMW08Y7q/0EySJElTgjMvkiRJkjrB8CJJkiSpEwwvkiRJkjrB8CJJkiSpEwwvkiRJkjrBt41pTLfd9xAfWXz7oIchSasM3/AoSaNz5kWSJElSJxheJEmSJHWC4UWSJElSJxheJEmSJHWC4UWSJElSJxheJEmSJHWC4UWSJElSJxheJEmSJHWC4UWSJElSJxheJEmSJHXCuOElyUZJTklyQ5JFSS5N8uoVMZgkc5Jc27bnJzl6RfQzzhg2THJ5ksVJdhp27K1JZvTt37OcfT0vyYVJfpDkyiT/nWSrcc7ZN8mxy9OvJEmS1EXTxzqYJMDZwGer6vWtbHPgVZPReZJpVfXwSMeqaiGwcDL6WUovAZZU1V+PcOytwMnAfcvbSZKNgM8Dr6+qS1rZjsAfAEuWt31JkiTp8Wa8mZcXA7+tquOGCqrqx1V1DDx2FiDJuUl2btufTLIwyXeTHNFX58YkH01yJfDaJNsluTrJ1cCb++rtnOTctv28NuOzOMklSZ7V1/9ZSb7SZi+ObOXTkpyY5NokS5K8bfiFtVmebyS5Jsn5SZ6aZB5wJLBbkquSrNlX/y3AJsAFSS7oK/9gG/9lLZAMzd58IckV7eeFI3y2B9ELhZf0fbbfrqqzWxu79s0AnTfU9rBrGLFOko8neV/b/rMk30qyXpIfJVm9la/bvy9JkiRNdeOFly2AK5ex7XdX1Xxga+CPk2zdd+yOqtq2qk4DPgMcXFXPGaOt7wE7VdU2wPuAD/Udmwe8DtgKeF2SzVrZplW1ZVVt1foY7hh64WFrYAFwdFVd1do/varmVdX9Q5Wr6mjgFmCXqtqlFa8FXNbG/i1g/1b+ceCoqnou8BrghBH6H++z/Tawfbvm04C/X4o672qfxS7A0cB+VXU3cCHwilZnT+CsqnpweKNJDmjBc+G9d94xxhAlSZKklWfMZWPDJfkEsCO92ZjnjlP9L5Mc0Pp4CjAXuKYdO721NxOYWVXfauUnAS8boa31gM8meSZQQP9swfntL+YkuQ7YHPgu8PQkxwD/DXxthDZfAPxFX79HjnM9I/ktcG7bXgT8Sdt+KTC3t+oOgHWTrF1Voz4jk+RyYF3ga1V1CDAbOD3JU4AnAD8a4bQR61TVfUn2pxeo3lZV/9vqn0Av4JwN7McjYetRqup44HiA2XPn1ZifgCRJkrSSjDfz8l1g26GdqnozvWdCNmxFDw1rYw2AJE8D3gm8pM1s/PfQsebepRznPwEXVNWWwK7D2nqgb/thYHpV3Qk8h95Mw4GMPPMxGR6sqqG/3D/MI2FwNXozIvPaz6YjBJfhn+3zgffSC2rQmxk6ts0c/Q2PvmYmUGcr4A56S92G+rgYmNOW9k2rqmuX9oIlSZKkQRkvvHwDWCPJ3/aVzejbvhGYl2S1tlzrea18XXoB5e72HMZIsylU1V3AXe1BdYC9RxnHesDNbXvfccZMklnAalX1BeA99IWEPpfQWzo11O9F47UL/BpYZwL1vgYc3DeeeSPU+QSwb5Id+sr6P9v+a95nlH5GrNNeqvAOYBvgZUme33fO54BTGHkpnSRJkjRljRle2qzC7vSeWflRku8AnwUObVUuprdU6Tp6z1Zc2c67GlhM71mVU1q90ewHfCLJVUBGqXMk8OEki5nYUrdNgQtbmyfTewZkuIOB/ZJcA7wBOGQC7R4PfKX/gf1RvAWY314GcB292Z9Hqarb6D2r8+EkP0xyCbAHMPQChMOBM5IsAm4fpZ/H1GlviPsU8M6qugV4E3BCkqFZmQXA+sCpE7heSZIkacrII6uetCpIsgewW1W9YSL1Z8+dVwctOG8Fj0qSNOSwbWYNegiStNIlWdRe9jWmpXpgX93WXmDwMuDlgx6LJEmStLQML6uQqjp4/FqSJEnS1DTeA/uSJEmSNCUYXiRJkiR1guFFkiRJUicYXiRJkiR1guFFkiRJUicYXiRJkiR1gq9K1pg2njHdX5gmSZKkKcGZF0mSJEmdYHiRJEmS1AmGF0mSJEmdYHiRJEmS1AmGF0mSJEmdYHiRJEmS1Am+Klljuu2+h/jI4tsHPQxJWuF8LbwkTX3OvEiSJEnqBMOLJEmSpE4wvEiSJEnqBMOLJEmSpE4wvEiSJEnqBMOLJEmSpE4wvEiSJEnqBMOLJEmSpE4wvEiSJEnqhJUeXpJUkpP79qcn+UWSc5eynQuTzB+l/CdJ0ld2dpJ7lm/ko47h+0muaj97rIA+Dk9yc2v/uiR7TWLbk/6ZSJIkSSvKIGZe7gW2TLJm2/8T4OZJ7uMu4IUASWYCT5nk9vvtXVXz2s+ZEz0pyfSl6OOoqpoH7Ab8R5LVl3qUkiRJUscNatnYl4FXtO29gFOHDiR5XpJLkyxOckmSZ7XyNZOcluT6JF8E1nxss793GrBn2/4L4Ky+9tdOcn6SK5MsSbJbK39ukmuSrJFkrSTfTbLl0l5YkjlJru3bf2eSw9v2hUn+LclC4N1JfjQURJKs278/kqr6AXAfsH47Z16Sy9q4v5hkqHz/JFckuTrJF5LMaOVPa5/tkiQfWNprkyRJkgZpUOHlNGDPJGsAWwOX9x37HrBTVW0DvA/4UCv/W+C+qno28H5guzHaPx94UZJp9ELM6X3HfgO8uqq2BXYB/iVJquoK4BzgA8CRwMlVdS1AkqvG6GtB37KxDSZw7U+oqvlVdQRwIY+EuD2Bs6rqwdFOTLIt8IOq+nkr+hxwaFVtDSyh97nQ2nluVT0HuB54Uyv/OPDJqtoKuHWMfg5IsjDJwnvvvGMClyRJkiSteAMJL1V1DTCH3qzLl4cdXg84o81eHAVs0cpfBJzcd/41Y3TxMPBteoFgzaq6se9YgA8luQY4D9gU2Kgd+0d6y9jm0wswQ+OdN0Zf/cvGJvI3/f4gdQKwX9veD/jMKOe8Lcl36YW8DwIkWQ+YWVXfbHU+S+8zgt6yvIuSLAH25pHP8IU8Mst10mgDrKrjW8Cav9b6E8ljkiRJ0oo3yLeNnQN8jL4lY80/ARdU1ZbArsAay9j+acDRwOeHle8NbAhs10LJz/r62ABYG1hnOfp9iEd/rsPbuXdoo6ouBuYk2RmYNjTTM4KjqmoL4DXAp9qM1VhOBA5qMyxHDBtDjXsFkiRJ0hQ0yPDyaeCIqloyrHw9HnmAf9++8m8Brwdoz6JsPU77FwEf5rHhaD3g51X1YJJdgM37jv0H8F5gAfDRiV3GY/wMeHKSDZI8EXjlOPU/B5zC6LMuv1dV5wALgX2q6m7gziQ7tcNvAIZmYdYBbm3Pz+zd18TFPPIsUH+5JEmSNOUNLLxU1U1VdfQIh44EPpxkMdD/Rq5PAmsnuZ7e8q5F47RfVfWxqrp92KEFwPy2pOqN9J6xIckbgQer6hTgI8Bzk7y4HRvrmZfh/T7Yxvcd4OtD7Y9hAb0H8IeHrNH8I/D2JKsB+wD/3JbAzWvHoBfALqcXVvr7PwR4c7v2TSfYnyRJkjQlpMpVRIPUfjfMblX1hkGPZSSz586rgxacN+hhSNIKd9g2swY9BElaZSVZVFWP+R2Owy3N7xrRJEtyDPAy4OWDHoskSZI01RleBqiqDh70GCRJkqSuGOQD+5IkSZI0YYYXSZIkSZ1geJEkSZLUCYYXSZIkSZ1geJEkSZLUCYYXSZIkSZ3gq5I1po1nTPcXt0mSJGlKcOZFkiRJUicYXiRJkiR1guFFkiRJUicYXiRJkiR1guFFkiRJUicYXiRJkiR1gq9K1phuu+8hPrL49kEPQ5KWia96l6THF2deJEmSJHWC4UWSJElSJxheJEmSJHWC4UWSJElSJxheJEmSJHWC4UWSJElSJxheJEmSJHWC4UWSJElSJxheJEmSJHXCKhVektwzgD4vmWC9mUnuSJK2/4IklWR2218vyS+TTPi/WZLDk7xzhPI5Sa6daDuSJEnSVLBKhZelkWTaZLRTVTtMsN5dwK3As1vRDsDi9ifA9sB3qup3E2kvyfSlHKokSZI0pa1y4SXJzknO7ds/Nsm+bfvGJB9NciXw2iT7J7kiydVJvpBkRqv32iTXtvJvtbItknwnyVVJrknyzFZ+T19fhyZZ0s77yAjDu4RHwsoOwFHD9i9u7cxLclnr54tJ1m/lFyb5tyQLgUOGXfd2rd+rgTcv14coSZIkDcAqF14m4I6q2raqTgPOqqrnVtVzgOuBN7U67wP+rJW/qpUdCHy8quYB84Gb+htN8jJgN+D57bwjR+j7Yh4JK08Hzmht0cqHlqB9Dji0qrYGlgDv72vjCVU1v6r+ZVjbnwEObn2PKckBSRYmWXjvnXeMV12SJElaKQwvj3V63/aWSS5KsgTYG9iilV8MnJhkf2BoedmlwD8kORTYvKruH9buS4HPVNV9AFX1yxH6vgTYIcnTgBur6jdAkqwNbAdcnmQ9YGZVfbOd81ngRaOMH3oNzGznfKsVnTTWB1BVx7cANH+t9TcYq6okSZK00qyK4eUhHn3daww7fm/f9onAQVW1FXDEUN2qOhB4D7AZsCjJBlV1Cr1ZmPuBLyd58dIOrKp+AMwEdqUXhgAWAfvRCzMTeeHAveNXkSRJkrpnVQwvPwbmJnlim5F4yRh11wFuTbI6vZkXAJL8QVVdXlXvA34BbJbk6cANVXU08CVg62FtfR3Yr++5mSeN0udl9J5XGQovlwJvpT3vUlV3A3cm2akdfwPwzeGN9GsvA7gryY6taO+x6kuSJElT0SrzRqr29q0HquqnST4PXAv8iN4bvUbzXuByegHlcnphBuCf2wP5Ac4HrgYOBd6Q5EHgNuBD/Q1V1VeSzAMWJvkt8GXgH0bo82Lg5cDCtn8pvedf+l+5vA9wXAtCN9CbmRnPfsCnkxTwtQnUlyRJkqaUVNWgx7BSJHkO8J9V9bxBj6VLZs+dVwctOG/Qw5CkZXLYNrMGPQRJ0gQkWVRV88ert0osG0tyIHAqvedUJEmSJHXQKrFsrKqOA44b9DgkSZIkLbtVYuZFkiRJUvcZXiRJkiR1guFFkiRJUicYXiRJkiR1guFFkiRJUicYXiRJkiR1wirxqmQtu41nTPeXvEmSJGlKcOZFkiRJUicYXiRJkiR1guFFkiRJUicYXiRJkiR1guFFkiRJUicYXiRJkiR1guFFkiRJUicYXiRJkiR1guFFkiRJUiekqgY9Bk1hSX4NfH/Q41CnzQJuH/Qg1HneR5oM3kdaXt5DK87mVbXheJWmr4yRqNO+X1XzBz0IdVeShd5DWl7eR5oM3kdaXt5Dg+eyMUmSJEmdYHiRJEmS1AmGF43n+EEPQJ3nPaTJ4H2kyeB9pOXlPTRgPrAvSZIkqROceZEkSZLUCYYXSZIkSZ1geNGIkvx5ku8n+WGSwwY9Hk1dSTZLckGS65J8N8khrfxJSb6e5Aftz/VbeZIc3e6ta5JsO9gr0FSRZFqSxUnObftPS3J5u1dOT/KEVv7Etv/DdnzOIMetqSPJzCRnJvlekuuTvMDvIi2NJG9r/y+7NsmpSdbwu2hqMbzoMZJMAz4BvAyYC+yVZO5gR6Up7CHgHVU1F9geeHO7Xw4Dzq+qZwLnt33o3VfPbD8HAJ9c+UPWFHUIcH3f/keBo6rqGcCdwJta+ZuAO1v5Ua2eBPBx4CtV9UfAc+jdT34XaUKSbAq8BZhfVVsC04A98btoSjG8aCTPA35YVTdU1W+B04DdBjwmTVFVdWtVXdm2f03vLwub0rtnPtuqfRbYvW3vBnyuei4DZiZ5ykoetqaYJLOBVwAntP0ALwbObFWG30ND99aZwEtafa3CkqwHvAj4FEBV/baq7sLvIi2d6cCaSaYDM4Bb8btoSjG8aCSbAj/t27+plUljalPm2wCXAxtV1a3t0G3ARm3b+0sj+Tfg74Hftf0NgLuq6qG233+f/P4easfvbvW1ansa8AvgM2354QlJ1sLvIk1QVd0MfAz4Cb3QcjewCL+LphTDi6RJkWRt4AvAW6vqV/3HqvdOdt/LrhEleSXw86paNOixqNOmA9sCn6yqbYB7eWSJGOB3kcavAP2XAAAELUlEQVTWnofajV4Q3gRYC/jzgQ5Kj2F40UhuBjbr25/dyqQRJVmdXnBZUFVnteKfDS3BaH/+vJV7f2m4FwKvSnIjvWWqL6b37MLMtnQDHn2f/P4easfXA+5YmQPWlHQTcFNVXd72z6QXZvwu0kS9FPhRVf2iqh4EzqL3/eR30RRieNFIrgCe2d6u8QR6D6udM+AxaYpq63s/BVxfVf/ad+gcYJ+2vQ/wpb7yN7Y3/WwP3N23pEOroKp6V1XNrqo59L5vvlFVewMXAHu0asPvoaF7a49W339NX8VV1W3AT5M8qxW9BLgOv4s0cT8Btk8yo/2/bege8rtoComfsUaS5OX01qBPAz5dVR8c8JA0RSXZEbgIWMIjzyv8A73nXj4PPBX4MfCXVfXL9j+EY+lNxd8H7FdVC1f6wDUlJdkZeGdVvTLJ0+nNxDwJWAz8VVU9kGQN4CR6z1f9Etizqm4Y1Jg1dSSZR++lD08AbgD2o/cPtX4XaUKSHAG8jt6bNBcDf03v2Ra/i6YIw4skSZKkTnDZmCRJkqROMLxIkiRJ6gTDiyRJkqROMLxIkiRJ6gTDiyRJkqROMLxIklYpSR5OclWSa5OckWTGJLe/b5Jjl/Kc+UmObts7J9lhMsckSY8XhhdJ0qrm/qqaV1VbAr8FDhzkYJJMr6qFVfWWVrQzYHiRpBEYXiRJq7KLgGcAJHl7m425NslbW9mcJN9LsiDJ9UnOHJqpSXJjkllte36SC4c3nmTXJJcnWZzkvCQbtfLDk5yU5GLgpDbbcm6SOfTC1Nva7NBOSX6UZPV23rr9+5K0qjG8SJJWSUmmAy8DliTZjt5vY38+sD2wf5JtWtVnAf9eVc8GfgX83VJ0821g+6raht5v6P77vmNzgZdW1V5DBVV1I3AccFSbHboIuBB4RauyJ3BWVT24NNcqSY8XhhdJ0qpmzSRXAQuBnwCfAnYEvlhV91bVPcBZwE6t/k+r6uK2fXKrO1Gzga8mWQL8X2CLvmPnVNX9E2jjBHrBivbnZ5aif0l6XJk+6AFIkrSS3V9V8/oLkoxVv0bZf4hH/hFwjVHOPQb416o6J8nOwOF9x+6dyGCr6uK2fG1nYFpVXTuR8yTp8ciZF0mSes++7J5kRpK1gFe3MoCnJnlB2349vaVgADcC27Xt14zS7nrAzW17nwmO5dfAOsPKPgecgrMuklZxhhdJ0iqvqq4ETgS+A1wOnFBVi9vh7wNvTnI9sD7wyVZ+BPDxJAuBh0dp+nDgjCSLgNsnOJz/Al499MB+K1vQ+j51whclSY9DqRo+Gy5JkqD3tjHg3PZa5UGOYw9gt6p6wyDHIUmD5jMvkiRNYUmOofdWtJcPeiySNGjOvEiSJEnqBJ95kSRJktQJhhdJkiRJnWB4kSRJktQJhhdJkiRJnWB4kSRJktQJ/x+23Q4Wmg1ZgAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pop= df2.sort_values('popularity', ascending=False)\n",
    "import matplotlib.pyplot as plt\n",
    "plt.figure(figsize=(12,4))\n",
    "\n",
    "plt.barh(pop['title'].head(6),pop['popularity'].head(6), align='center',\n",
    "        color='skyblue')\n",
    "plt.gca().invert_yaxis()\n",
    "plt.xlabel(\"Popularity\")\n",
    "plt.title(\"Popular Movies\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d1cf28fc0c22afa80edc6d464177035357721d35"
   },
   "source": [
    "Now something to keep in mind is that these demographic recommender provide a general chart of recommended movies to all the users. They are not sensitive to the interests and tastes of a particular user.\n",
    "This is when we move on to a more refined system-  Content Basesd Filtering. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "fe716df6e5e5a354ac53d556087147c0a64df2cc"
   },
   "source": [
    "# **Content Based Filtering**\n",
    "In this recommender system the content of the movie (overview, cast, crew, keyword, tagline etc) is used to find its similarity with other movies. Then the movies that are most likely to be similar are recommended.\n",
    "\n",
    "![](https://image.ibb.co/f6mDXU/conten.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b0a813c803b0ba1f0204188ab2a63dc7f59ce2eb"
   },
   "source": [
    "## **Plot description based Recommender**\n",
    "\n",
    "We will compute pairwise similarity scores for all movies based on their plot descriptions and recommend movies based on that similarity score. The plot description is given in the **overview** feature of our dataset. \n",
    "Let's take a look at the data. .. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "_uuid": "5e676c38ace04a24205b76b16dac0fa3e058027f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    In the 22nd century, a paraplegic Marine is di...\n",
       "1    Captain Barbossa, long believed to be dead, ha...\n",
       "2    A cryptic message from Bond’s past sends him o...\n",
       "3    Following the death of District Attorney Harve...\n",
       "4    John Carter is a war-weary, former military ca...\n",
       "Name: overview, dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['overview'].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "277a9bb5b00a6bd2469c45777f9c659066f402b3"
   },
   "source": [
    "For any of you who has done even a  bit of text processing before knows we need to convert the word vector of each overview.\n",
    "Now we'll compute Term Frequency-Inverse Document Frequency (TF-IDF) vectors for each overview.\n",
    "\n",
    "Now if you are wondering what is term frequency , it is the relative frequency of a word in a document and is given as\n",
    "   **(term instances/total instances)**.\n",
    "Inverse Document Frequency is the relative count of documents containing the term is given as \n",
    "**log(number of documents/documents with term)**\n",
    "The overall importance of each word to the documents in which they appear is equal to **TF * IDF**\n",
    "\n",
    "This will give you a matrix where each column represents a word in the overview vocabulary (all the words that appear in at least one document) and each column represents a movie, as before.This is done to reduce the importance of words that occur frequently in plot overviews and therefore, their significance in computing the final similarity score.\n",
    "\n",
    "Fortunately, scikit-learn gives you a built-in TfIdfVectorizer class that produces the TF-IDF matrix in a couple of lines. That's great, isn't it? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_uuid": "a92da8cde39c61deef5a1b8efa31ed84cda7f5fe"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4803, 20978)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Import TfIdfVectorizer from scikit-learn\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "#Define a TF-IDF Vectorizer Object. Remove all english stop words such as 'the', 'a'\n",
    "tfidf = TfidfVectorizer(stop_words='english')\n",
    "\n",
    "#Replace NaN with an empty string\n",
    "df2['overview'] = df2['overview'].fillna('')\n",
    "\n",
    "#Construct the required TF-IDF matrix by fitting and transforming the data\n",
    "tfidf_matrix = tfidf.fit_transform(df2['overview'])\n",
    "\n",
    "#Output the shape of tfidf_matrix\n",
    "tfidf_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6bde57434bf9a0e8f8b229d36901d75b77ff962f"
   },
   "source": [
    "We see that over 20,000 different words were used to describe the 4800 movies in our dataset.\n",
    "\n",
    "With this matrix in hand, we can now compute a similarity score. There are several candidates for this; such as the euclidean, the Pearson and the [cosine similarity scores](https://en.wikipedia.org/wiki/Cosine_similarity). There is no right answer to which score is the best. Different scores work well in different scenarios and it is often a good idea to experiment with different metrics.\n",
    "\n",
    "We will be using the cosine similarity to calculate a numeric quantity that denotes the similarity between two movies. We use the cosine similarity score since it is independent of magnitude and is relatively easy and fast to calculate. Mathematically, it is defined as follows:\n",
    "![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAa0AAAB1CAMAAADKkk7zAAAAh1BMVEX///8AAAD5+fnFxcWtra3z8/P8/Pzw8PBycnL09PTo6Oj39/e9vb3g4ODBwcGXl5ednZ3Pz8/c3NyIiIiOjo5FRUXS0tJ+fn64uLimpqZPT0/d3d1tbW1jY2NXV1c9PT02NjZ3d3cvLy9TU1NdXV0YGBggICBISEg5OTkqKioLCwsTExMdHR2pbwthAAARwklEQVR4nO1d2YKiOhBNARJAIIGETVkVtbX7/7/vJiCILTPdLr3ckfPQ4xoYDlU5VamUCE2YMGHChAkTng2+YS1Z8tNnMeFzyMhBm716P30aEz4DnGQ1UjbxT5/HhM8BYmSBqf/0aUz4DDDMEQusyRX+L5CskF5HxPrp85jwGcRiyvIi/6dPY8KECRN+BKZ1DvzTJzThL/DyrGIsapFmteG8+wA+f0GZmeZMQPm+U5zQA2/BCI9wSfoC7Pz9GVmbg6d6SAWShGr+/FvPc0IDF2CgBdX1y/nb3ls+NC7FS2EfUR4FhT3F0j8ABuVgsnJ29vBNzGARnn1c3XPpBeNNNgn+n8AaqpOZ6PYwFa+7xSJQzz6tQfMBe7WYMh8/AecNtD+9VSXG6izZa5KtfG7yzZT5+BloZ1PXAHON42h7tu7l1zsXW0uaTWT9FAxYjMZZoeHM+IHKh/pRArr7BUlouqJy8tKVSWl8P3AAbOS6Y5IgJQEuH/taK+Q1qGJN04zME1/w4snAfgDeAS6vu26XNI6jNgLzeGN9x2kLxQdDfMEm72PpCd+BVXbpCp2UUEIjiMRj5ZiS8uu6EfT2Pg+RaZkXX5rw9bDT5cVrcyppQiFUwifGSRt1qRsmGZoTiDBWkynk+gGo+eVlN+M2LA4hnSMvSknzahtt4aTcqchO6j8p/wlfhzAfBLp64/IUjwcrTRdGxSEgqqVltrAoj2aQCo3Bc8NThPBI3Z865ecFzgfxr642TxQ10RJNQVj81aiKNAMraK6SNqer2Y0moZE/Kfhvhl7RwTO/HrMXp+L+hfyzKs17Vk2oO+NLRthp8WXrhHywRqK4xsuYufhpGl8c30mrp4234nR0ytZplhpGmmVplKhfsQRID5rqNoiThAdSAV5i5tmXVjR/4uB4B/uxu1pXkxyaqZ1B8AWV6j7sF7sW5dsGDiv14+9MUN8AwvG3GDSJVSeA8vGX0rGHcG11WhL+BDIAMEbfUY5szSp44997UhPGYW3TA8Bs7K2erQy2UzT6K0DAWQCMzkuCrTfu+CF9W9Ev0IVsuLKvhyp5/CH+OQQB0l5hNfaWYOsQpFm9h+wPE9tdSIcL9grl+y84xj8GF8QdvhrqDMw6ryfY2rDQ8+JildrjX78H1jAs0J1k9I6ZMEQGVIsLodT7V1TYHh918xayAbJHR1whfVc/GE9sfQRr9VoUhbCtk87AaZe869myAILLhY37UKkgAmSaJIlMAOpIn9j6EOTg+b7v7ABOc3zvoXq2fIDdg5MHikdeRMzVAT0HW8ogLzO7XrgF6+Yf7QBvI2lS3ip3R4Rko2mhuwAi4O4qq8NfwJbuf3k2C/NWEihujJFDriuLxKnQ7oWJkPEiXGERnCcTdL7YAOzz9bqA8vEKXjhCguIjpIaxv4MtnbPLF+cxWyJvV4yFEBZJHvZfn1GjKVBw0rRiWLev22SoLIUXlAtFS797dHaifo8vKIPQDiREsw5IITnwr19inMPm0oc4iw0VlL1GI1+I9ztpAvEjQphw0dBjZQZWdxqa07Fyr98JUzt3BNYXLs2coEfR5RUyNXmX++UYWw5v3Ap7wI2Eq0Z4m3QXIj/gsszh/7MCPv+RG2s+ljtuCkz9lzG20FxqrvGV0ivhrJpZy38Txwn3wiM7xpR9/QM0TjwPhZ46Q47n+cjVhKgINalvLPn8yJZDiI2b11TLTxzF9zyMMDtQz5Gfk39Dz/9r8BkTu7klQkob/6knPPGWSIkDKWNm5E2Q5slaST3JnnaZ7u8Ispi+ErTYlo7O9mW0plVAI0a2DGGyLZIjW95LGkdBLATBvlznK2Yb2zy0og2sXhhOiv2eYb46/O0iWy+LJFoJLx/tE20nhrSCSGNljHCUy/eXgdyWZjd1yHYwremNwQVZBhKJuQMcZHpQWIoDpaZbmQgkFRXokS0bIrQ0ah/p2iGNs8rCUeEhhUNsmiJaqgthLbWGWyfu5kGPstN3Jei6IbigYCm6syGIFIKxNEFWncn3Q9gEu2C/l17RLbskn704DZV/RVK2ga4Vp8NUv7ckM4ZFgrEvF1gdmZtJ5U7anfgTyesfNmwVgi3sCbOhhbjl7QNtJjki2BKXvjUCBsKZGl1aRzFnJ+jdgZiULQoCudVTWZSYQxZjyxI2JlcRdRuYYnpBU0eplt3ahzIYqW+jNPPUx6DnZfQw1qMOcyfck1oXAf6brOE+siWJ2a/P2WrmLb8qV2+B0BTxa3slectWqzLUIprx+G+zVtXsSNMRhoW8OtnGwzsRtTLlyJaZbJtpq1E0akn/MhRSA3gMRuVTj2T/oMPci0EGxqv2UPdssRG2hG3pMWS+SWRdQ7x5x5YjlXy1CaveUeHBTekqHVttlhVDINmqNyrS3fT1wJC1lgIeS1udNdYr6Fh1tmUN7jGvS9nqyoPQq2/LPR2m32z/sMPci/40E8lbejiy5YyyJeItf7HwRNRa8Bjbb+/Y4mpzH+aslxheZPRIj95GPW5jQiCX65Ria3ExNl7UIiiW85ai7Xxhxm1G5TRvqdVpqK/ro6Tb6ekwv3SXjrSFRKgMxMVUwg/W0bbMfS4T14UU73Lt3CmZYEvMJ05WVAS7m/ZKkiBs5i2dy2h+lsEH2b1SHokwMYGp7W0RiZtCMSoRBjSa0Cs9ROs2N2QXkyZ8ByZrhCgsqgyW6BVgQyLhI4tY/qFbgEMqn0eq+LNQbShTI90AI+JpaSN1If61kZ/DSmt8HQ8+uvNXUGVCBopDrnNhaPMKcmNRWDI6kHezla2yqPV1SlL/zvv7B1GBuNstjNTPXRlHxcgP3+dZdC9s3TzXPiykw3YrGvVQmqFuzRW1cZ5Oo9pF/NC5UisdSTG/gytmS3W0xskSduk2NqrJ1cGRT+hOiHA4+uVfDHjQOCp3UHV7MIRZev6Cuv7YEVaejrLTfcbd/j6KxWhZM0AgQsZS3lHuEV77oZlG0ZLf1K7C0a6DfZniteLrhkja/+WyuOV8R8BAJfcsHS2LMy+K+SfShNEZW+4r9OGDLRx82rAlA/xCytATUnmaki3nNrbqK8V3cHkU7VoB36q610flBvx0m96T2NPt3ZDr2PgE85E6ZEtcRK0zroatJgyUbDVtl/QSmkVKcaGk/ryDrd39hdPaTSWofvrxZ74JijYoiv5cx9MztiwRx64644ov2cIrKHD7Oalk72Arv3vxyqJfUCb323HGFhMStm8Hc8bWvJQPera2IGOD2+etsLpbm4Tky9KdvxdnbJXR+gBZl5FMh7ZVygeSLUvXFRUO0ohvZ4vSuz2hS56wPeiQLQ18ddvXEjaa8MK2Nqlh5PAayat9O1uGd/dScfzXFOg/iiFbeTlHAcDxMoyztY0Yi4rXppHq7Ww9oB5Re3K2nC3FOn+FfaszztjqPWEgLQ/vYHXXvNU339JnM3OGb6jTxrSThLppiiGeY3PegC12EF4ue+32LPxh3pLXWdGaUvmb2fLTjq04K6JqfAfSB0OQLvL30jytSv4Us9iAraLmhJP60MRSR03YqOSerdklWzcpeI13FYmVB0tk1NcPofZ9CW1eaygZaTn0D6JjS9G1Q3PdwzcALJeC7J6thWBLls4qltSEiqIsM9jwO9iK+s7FilxB2NwwBfUiQxdMhYg9l23ZSQDE11GoCVdoUP89W6W0KLKBLUkSksNeNpq7ma1B++mCIgz4+tBpIDIImy2z2Pq/ZZdvQcdWtMjznaagZJ3n+Vr2WLiwLXMtPpMvBPIqvitPuD5JQrCQ9ur9vVRgBErSz3W40lBcM+0ZivHOs7pDuEO25mObCMwb2cKnJoSOmLLUNL26sN4hfbnsknjIY8ZTZDbOs7pDXHjCC9xqWza722t5/PfWv30h/mxbH7N1q21x7YoAaz66e8EmV5ijbv0rXpIJti5+4aWBGskNpvJRjZGyG/mEGSfCJV3vgtqeADN8jvFcFLYJHelnF8vdVfr83RDjN4HiUa7+G71cXXHbXbYlk3DEzND2/xOmoI8tJim+15azXomF/IpJidxmnCSUCIhI78JG5R7aGUlD9nJxQ+ha0/oz5schmjEIH2835C487bRsN+E6zGrJFt7BtrnWhDEWGWVTzTrEUpZmW4sKOZuLNXCLyDoRK4It0/ohAsj7EHnmqW0xpodRtDJREEw//XQb1KixZXUPg9IcN33Xr2heHTTpbGPkby82lYa87bWbAzmZUxjt+zoUh+1ldZmRAcNqoqBgPbF1G0jSXmF6GP4Qhrk4tyBPmF7zYJ6sLzyhe0xduMV24OJwNFjzYk0JupJBItv7r+LJE96GtFvciiAbTP7Ls9ZtuF4fGvp01bis3IqP1qYkr/WAcZ+e9JIBzWMGhoks4ws6Kj4J+kwGXp9l34fKVNdSrdkMoYfURxexVZ93ElMXG2ik08KJuS6bw2TCxDC10f3Ln88Jpe5VpLr9U+dKZ+24Ms2vhwZRtfcqw+oXt1BYv47KPbWMTFm2WEQmZpGr1tO8dRO86mQMFNaj0YPCKQrllp0ZeVns6vfVTf5A7tvleiw+J6+VFsfZns6RvVvsCjLZ1k1IyEn8KSmwsZvez3VkSU0/l52R1feMqvwU5JkExqL7FDillO2oiUIxxEjz5AmfQaUObnOngJGyQCWt7TiBsfxJg3joGq0KLiuo8Hp1nLYmMXgXzn5bFREY6T3hFlFVGW0NyBi0swAsLulFXsletYXKEUT/Rs7pm+Ed3ZmeD1NVijGybmzlnuVY4eZFpiaw1itC6yhJMB1KyRnhl8kvDu1+YCEJxb+xOhnYVbC6zZ3+Wak/H5m25u1ui/mL7DMhvtiJCEyPJuXzQQSma2xk8SSFZfPeprk3qmRi6yrQbs+nNqygSNILSTijDCJLSHfyBoaYdPS+1Yu3P25H89jAmbrsQhLOYloCT5KE7atmp5z/FEUbj4NVwzHfFA1aFsTZoFJJaWMoXFWRYENXIwGqO3ZnR5h1v4ds85OpeGzg5WbttkNMxVdlopc2P5mB1SkBfx1o9la1HnCQKVLrYWxM2jo1xRIQV3cm/8WY0uz4fhisjhtzB5LQZ8N+FVrk9EMchxFI7O0zVNg8DpjTYNFe6pPICPNhAlAbk/JCY/Bj+hDHLGsbIZmnCpolG278dYtkbIHL1l4mXXgNTMepN41ROX0Pq2XNl85y6cv9y37C3mDUXS2D47LK3PFZ29fX4R2vFos8McTSRLrlxzzYjFcd5HzyhFcibZdH4ug445sZyGK4fF1FRr4GgGzsW7q9sLqATNHa5sxeJwIxgbKWY6SRUa+3YojRjIW1da4pBJmAZJzayAGeHL2SorquLbecN78JJDBaxGTSIO4NRm1/8thlR383D9tN63Y/xCgp6opOCY0rQYDKuT67rkRK95PTWocHhuTJvrJHJk7iad66Etq+CYvza9Ors5NZhKXsDDe/eufWbPKD18ILCtmv7NQo03HP3dPHPyfoZC9xV0HTwHpXcPXJBjMTPoRZS5nhRv31pWdJp5huP/SRmMkugSHrozSXD/cBeWT3hNv8vwiNKCT09AspZ3ZhWfBh4xw9lg3k1FNdtmkN4yvsLSY18ShEMi3eV9Ag7f1q7sdsCVEom932zas8er7FDmcTW48COXCM6u76JuEqRDZpweV881GrPyRFYYr1fnHL03iCQnocQtwGgq2vOvmnQ7yqLbOPXzVrYwlneIR0aJ+wLb9YhJh0FTRhHCXCGR5hoomtByLcrSyv6iYrhb6vnv0EW066ipes/1ycnpvj5AkfB6UGJzkV0xZugmLegn3SE2J+SMJeVeqEeH7YDRFOtvVQpOBEdmdQ1ptKFPO0lcf34OO9RUIUErUXGU4VxY5yGsKKC+ZNWYsHQYjC9SkZSM534Wtxwj/efSxE4SnvpNv8jF9V0zif4uMHgW5YcLqYyrkVzBVl/vGaofe2YKcKGuX8h4UUMYQ5TVwPgv2yz+/cUuovDvXUhvtb4CzOdhjcAquCu7qjTvg8Mvjrj358AjMKH7fhnvAQGHB32wQXph+A+yaw7d2KTd1P++a+Ccm9IkOIwkf8ZOOEz8C/P3a1LnYHTfjH8R9wYiZCyjFLIwAAAABJRU5ErkJggg==)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c1c138ae64648cb5a94127a06441d88dddc2bd9a"
   },
   "source": [
    "Since we have used the TF-IDF vectorizer, calculating the dot product will directly give us the cosine similarity score. Therefore, we will use sklearn's **linear_kernel()** instead of cosine_similarities() since it is faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "_uuid": "5eb17d12220eecab4faf01bbfd13e79d8e446537"
   },
   "outputs": [],
   "source": [
    "# Import linear_kernel\n",
    "from sklearn.metrics.pairwise import linear_kernel\n",
    "\n",
    "# Compute the cosine similarity matrix\n",
    "cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f5ca112fbbe25b11f0f3356a31d1604727242700"
   },
   "source": [
    "We are going to define a function that takes in a movie title as an input and outputs a list of the 10 most similar movies. Firstly, for this, we need a reverse mapping of movie titles and DataFrame indices. In other words, we need a mechanism to identify the index of a movie in our metadata DataFrame, given its title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "_uuid": "55df2df36be98e6dec5f617a5aa51b77c500faa4"
   },
   "outputs": [],
   "source": [
    "#Construct a reverse map of indices and movie titles\n",
    "indices = pd.Series(df2.index, index=df2['title']).drop_duplicates()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "da5896c6ccfd44c3347af3097275d0aa707c1001"
   },
   "source": [
    "We are now in a good position to define our recommendation function. These are the following steps we'll follow :-\n",
    "* Get the index of the movie given its title.\n",
    "* Get the list of cosine similarity scores for that particular movie with all movies. Convert it into a list of tuples where the first element is its position and the second is the similarity score.\n",
    "* Sort the aforementioned list of tuples based on the similarity scores; that is, the second element.\n",
    "* Get the top 10 elements of this list. Ignore the first element as it refers to self (the movie most similar to a particular movie is the movie itself).\n",
    "* Return the titles corresponding to the indices of the top elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "_uuid": "9c383fcbb916dce464b01adf980d26ad96aebe0e"
   },
   "outputs": [],
   "source": [
    "# Function that takes in movie title as input and outputs most similar movies\n",
    "def get_recommendations(title, cosine_sim=cosine_sim):\n",
    "    # Get the index of the movie that matches the title\n",
    "    idx = indices[title]\n",
    "\n",
    "    # Get the pairwsie similarity scores of all movies with that movie\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "\n",
    "    # Sort the movies based on the similarity scores\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "\n",
    "    # Get the scores of the 10 most similar movies\n",
    "    sim_scores = sim_scores[1:11]\n",
    "\n",
    "    # Get the movie indices\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "\n",
    "    # Return the top 10 most similar movies\n",
    "    return df2['title'].iloc[movie_indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "_uuid": "14d722124f82e69cb444adcc589e396c75cbb4ff"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65                              The Dark Knight\n",
       "299                              Batman Forever\n",
       "428                              Batman Returns\n",
       "1359                                     Batman\n",
       "3854    Batman: The Dark Knight Returns, Part 2\n",
       "119                               Batman Begins\n",
       "2507                                  Slow Burn\n",
       "9            Batman v Superman: Dawn of Justice\n",
       "1181                                        JFK\n",
       "210                              Batman & Robin\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight Rises')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "_uuid": "902b9f1ab91921889c85e9008818dcc0b4710ccd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7               Avengers: Age of Ultron\n",
       "3144                            Plastic\n",
       "1715                            Timecop\n",
       "4124                 This Thing of Ours\n",
       "3311              Thank You for Smoking\n",
       "3033                      The Corruptor\n",
       "588     Wall Street: Money Never Sleeps\n",
       "2136         Team America: World Police\n",
       "1468                       The Fountain\n",
       "1286                        Snowpiercer\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Avengers')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "146302c25776b2c0076e64663a5e2e41e977fd2c"
   },
   "source": [
    "While our system has done a decent job of finding movies with similar plot descriptions, the quality of recommendations is not that great. \"The Dark Knight Rises\" returns all Batman movies while it is more likely that the people who liked that movie are more inclined to enjoy other Christopher Nolan movies. This is something that cannot be captured by the present system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "fcfe9db9c2fdd9334538256d233c6acf33c1c049"
   },
   "source": [
    "## **Credits, Genres and Keywords Based Recommender**\n",
    "It goes without saying that the quality of our recommender would be increased with the usage of better metadata. That is exactly what we are going to do in this section. We are going to build a recommender based on the following metadata: the 3 top actors, the director, related genres and the movie plot keywords.\n",
    "\n",
    "From the cast, crew and keywords features, we need to extract the three most important actors, the director and the keywords associated with that movie. Right now, our data is present in the form of \"stringified\" lists , we need to convert it into a safe and usable structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "_uuid": "59a8d0991e3cae9a44a4b351e154fd1000724448"
   },
   "outputs": [],
   "source": [
    "# Parse the stringified features into their corresponding python objects\n",
    "from ast import literal_eval\n",
    "\n",
    "features = ['cast', 'crew', 'keywords', 'genres']\n",
    "for feature in features:\n",
    "    df2[feature] = df2[feature].apply(literal_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "47d6062d1622a163f2bcf80b79eb7b1454003739"
   },
   "source": [
    "Next, we'll write functions that will help us to extract the required information from each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "_uuid": "783b0e89f1c04a12ff51eb29cc68e93c818896cd"
   },
   "outputs": [],
   "source": [
    "# Get the director's name from the crew feature. If director is not listed, return NaN\n",
    "def get_director(x):\n",
    "    for i in x:\n",
    "        if i['job'] == 'Director':\n",
    "            return i['name']\n",
    "    return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "_uuid": "86c4e9f4e6ef1e5ff287f58f3a1119fbddbdae09"
   },
   "outputs": [],
   "source": [
    "# Returns the list top 3 elements or entire list; whichever is more.\n",
    "def get_list(x):\n",
    "    if isinstance(x, list):\n",
    "        names = [i['name'] for i in x]\n",
    "        #Check if more than 3 elements exist. If yes, return only first three. If no, return entire list.\n",
    "        if len(names) > 3:\n",
    "            names = names[:3]\n",
    "        return names\n",
    "\n",
    "    #Return empty list in case of missing/malformed data\n",
    "    return []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "_uuid": "dd060c3c1d724de71555218f30cccafd4a8ad6af"
   },
   "outputs": [],
   "source": [
    "# Define new director, cast, genres and keywords features that are in a suitable form.\n",
    "df2['director'] = df2['crew'].apply(get_director)\n",
    "\n",
    "features = ['cast', 'keywords', 'genres']\n",
    "for feature in features:\n",
    "    df2[feature] = df2[feature].apply(get_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "_uuid": "87a96f835470aa3df590b74322c2717ff529d6ae"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>cast</th>\n",
       "      <th>director</th>\n",
       "      <th>keywords</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Sam Worthington, Zoe Saldana, Sigourney Weaver]</td>\n",
       "      <td>James Cameron</td>\n",
       "      <td>[culture clash, future, space war]</td>\n",
       "      <td>[Action, Adventure, Fantasy]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Johnny Depp, Orlando Bloom, Keira Knightley]</td>\n",
       "      <td>Gore Verbinski</td>\n",
       "      <td>[ocean, drug abuse, exotic island]</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Daniel Craig, Christoph Waltz, Léa Seydoux]</td>\n",
       "      <td>Sam Mendes</td>\n",
       "      <td>[spy, based on novel, secret agent]</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      title              ...                                     genres\n",
       "0                                    Avatar              ...               [Action, Adventure, Fantasy]\n",
       "1  Pirates of the Caribbean: At World's End              ...               [Adventure, Fantasy, Action]\n",
       "2                                   Spectre              ...                 [Action, Adventure, Crime]\n",
       "\n",
       "[3 rows x 5 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print the new features of the first 3 films\n",
    "df2[['title', 'cast', 'director', 'keywords', 'genres']].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "0bcb2c2e99ffd9ce73205c9c6ef6687d16caa31f"
   },
   "source": [
    "The next step would be to convert the names and keyword instances into lowercase and strip all the spaces between them. This is done so that our vectorizer doesn't count the Johnny of \"Johnny Depp\" and \"Johnny Galecki\" as the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "_uuid": "86af764c406a8b6184b37b57cfe499d20ce45f9c"
   },
   "outputs": [],
   "source": [
    "# Function to convert all strings to lower case and strip names of spaces\n",
    "def clean_data(x):\n",
    "    if isinstance(x, list):\n",
    "        return [str.lower(i.replace(\" \", \"\")) for i in x]\n",
    "    else:\n",
    "        #Check if director exists. If not, return empty string\n",
    "        if isinstance(x, str):\n",
    "            return str.lower(x.replace(\" \", \"\"))\n",
    "        else:\n",
    "            return ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "_uuid": "5728cc017ff6ed1dcd79da05b1dd57a60557e853"
   },
   "outputs": [],
   "source": [
    "# Apply clean_data function to your features.\n",
    "features = ['cast', 'keywords', 'director', 'genres']\n",
    "\n",
    "for feature in features:\n",
    "    df2[feature] = df2[feature].apply(clean_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b6b3e1c480a7c280fbe81e63c5c4cf3ce308dc28"
   },
   "source": [
    "We are now in a position to create our \"metadata soup\", which is a string that contains all the metadata that we want to feed to our vectorizer (namely actors, director and keywords)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "_uuid": "20aef87703c408926f7617573ed043605207767f"
   },
   "outputs": [],
   "source": [
    "def create_soup(x):\n",
    "    return ' '.join(x['keywords']) + ' ' + ' '.join(x['cast']) + ' ' + x['director'] + ' ' + ' '.join(x['genres'])\n",
    "df2['soup'] = df2.apply(create_soup, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7b79886883806b8fb58098f9f803dabeaa0cadf6"
   },
   "source": [
    "The next steps are the same as what we did with our plot description based recommender. One important difference is that we use the **CountVectorizer()** instead of TF-IDF. This is because we do not want to down-weight the presence of an actor/director if he or she has acted or directed in relatively more movies. It doesn't make much intuitive sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "_uuid": "b66a1afc1083917d5ef136ccdcd9b50cca087e2b"
   },
   "outputs": [],
   "source": [
    "# Import CountVectorizer and create the count matrix\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count = CountVectorizer(stop_words='english')\n",
    "count_matrix = count.fit_transform(df2['soup'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "_uuid": "3fa5539ed1680ed5323f8351ac7e4840f629e958"
   },
   "outputs": [],
   "source": [
    "# Compute the Cosine Similarity matrix based on the count_matrix\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "cosine_sim2 = cosine_similarity(count_matrix, count_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "_uuid": "b2b8565a04f4bda92d3ba9d15c348af1cd8f8b4d"
   },
   "outputs": [],
   "source": [
    "# Reset index of our main DataFrame and construct reverse mapping as before\n",
    "df2 = df2.reset_index()\n",
    "indices = pd.Series(df2.index, index=df2['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "3c5a3a44893f63aa558030d03e228a365d10d91f"
   },
   "source": [
    "We can now reuse our **get_recommendations()** function by passing in the new **cosine_sim2** matrix as your second argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "_uuid": "d1e0e02be7a9e71422d3a492834cb4f8434d1464"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65               The Dark Knight\n",
       "119                Batman Begins\n",
       "4638    Amidst the Devil's Wings\n",
       "1196                The Prestige\n",
       "3073           Romeo Is Bleeding\n",
       "3326              Black November\n",
       "1503                      Takers\n",
       "1986                      Faster\n",
       "303                     Catwoman\n",
       "747               Gangster Squad\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight Rises', cosine_sim2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "_uuid": "d6c4df85a80d830b2905f69e0e59ebb3461db3b7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "867      The Godfather: Part III\n",
       "2731      The Godfather: Part II\n",
       "4638    Amidst the Devil's Wings\n",
       "2649           The Son of No One\n",
       "1525              Apocalypse Now\n",
       "1018             The Cotton Club\n",
       "1170     The Talented Mr. Ripley\n",
       "1209               The Rainmaker\n",
       "1394               Donnie Brasco\n",
       "1850                    Scarface\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Godfather', cosine_sim2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "4d963ff547ee4980c0f23840394046d805fda574"
   },
   "source": [
    "We see that our recommender has been successful in capturing more information due to more metadata and has given us (arguably) better recommendations. It is more likely that Marvels or DC comics fans will like the movies of the same production house. Therefore, to our features above we can add *production_company* . \n",
    "We can also increase the weight of the director , by adding the feature multiple times in the soup."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "71b15b5c090694303fa5e8d67b8bf394e07f45d6"
   },
   "source": [
    "# **Collaborative Filtering**\n",
    "\n",
    "Our content based engine suffers from some severe limitations. It is only capable of suggesting movies which are close to a certain movie. That is, it is not capable of capturing tastes and providing recommendations across genres.\n",
    "\n",
    "Also, the engine that we built is not really personal in that it doesn't capture the personal tastes and biases of a user. Anyone querying our engine for recommendations based on a movie will receive the same recommendations for that movie, regardless of who she/he is.\n",
    "\n",
    "Therefore, in this section, we will use a technique called Collaborative Filtering to make recommendations to Movie Watchers.\n",
    "It is basically of two types:-\n",
    "\n",
    "*  **User based filtering**-  These systems recommend products to a user that similar users have liked. For measuring the similarity between two users we can either use pearson correlation or cosine similarity.\n",
    "This filtering technique can be illustrated with an example. In the following matrixes, each row represents a user, while the columns correspond to different movies except the last one which records the similarity between that user and the target user. Each cell represents the rating that the user gives to that movie. Assume user E is the target.\n",
    "![](https://cdn-images-1.medium.com/max/1000/1*9NBFo4AUQABKfoUOpE3F8Q.png)\n",
    "\n",
    "Since user A and F do not share any movie ratings in common with user E, their similarities with user E are not defined in Pearson Correlation. Therefore, we only need to consider user B, C, and D. Based on Pearson Correlation, we can compute the following similarity.\n",
    "![](https://cdn-images-1.medium.com/max/1000/1*jZIMJzKM1hKTFftHfcSxRw.png)\n",
    "\n",
    "From the above table we can see that user D is very different from user E as the Pearson Correlation between them is negative. He rated Me Before You higher than his rating average, while user E did the opposite. Now, we can start to fill in the blank for the movies that user E has not rated based on other users.\n",
    "![](https://cdn-images-1.medium.com/max/1000/1*9TC6BrfxYttJwiATFAIFBg.png)\n",
    "\n",
    "Although computing user-based CF is very simple, it suffers from several problems. One main issue is that users’ preference can change over time. It indicates that precomputing the matrix based on their neighboring users may lead to bad performance. To tackle this problem, we can apply item-based CF.\n",
    "\n",
    "* **Item Based Collaborative Filtering** - Instead of measuring the similarity between users, the item-based CF recommends items based on their similarity with the items that the target user rated. Likewise, the similarity can be computed with Pearson Correlation or Cosine Similarity. The major difference is that, with item-based collaborative filtering, we fill in the blank vertically, as oppose to the horizontal manner that user-based CF does. The following table shows how to do so for the movie Me Before You.\n",
    "![](https://cdn-images-1.medium.com/max/1000/1*LqFnWb-cm92HoMYBL840Ew.png)\n",
    "\n",
    "It successfully avoids the problem posed by dynamic user preference as item-based CF is more static. However, several problems remain for this method. First, the main issue is ***scalability***. The computation grows with both the customer and the product. The worst case complexity is O(mn) with m users and n items. In addition, ***sparsity*** is another concern. Take a look at the above table again. Although there is only one user that rated both Matrix and Titanic rated, the similarity between them is 1. In extreme cases, we can have millions of users and the similarity between two fairly different movies could be very high simply because they have similar rank for the only user who ranked them both.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "4307f75107f9c5e5f911d52a6f1dc5530990c75e"
   },
   "source": [
    "### **Single Value Decomposition**\n",
    "One way to handle the scalability and sparsity issue created by CF is to leverage a **latent factor model** to capture the similarity between users and items. Essentially, we want to turn the recommendation problem into an optimization problem. We can view it as how good we are in predicting the rating for items given a user. One common metric is Root Mean Square Error (RMSE). **The lower the RMSE, the better the performance**.\n",
    "\n",
    "Now talking about latent factor you might be wondering what is it ?It is a broad idea which describes a property or concept that a user or an item have. For instance, for music, latent factor can refer to the genre that the music belongs to. SVD decreases the dimension of the utility matrix by extracting its latent factors. Essentially, we map each user and each item into a latent space with dimension r. Therefore, it helps us better understand the relationship between users and items as they become directly comparable. The below figure illustrates this idea.\n",
    "\n",
    "![](https://cdn-images-1.medium.com/max/800/1*GUw90kG2ltTd2k_iv3Vo0Q.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "defca8163cfc24a97bee620d6a3d501aa2ec95ae"
   },
   "source": [
    "Now enough said , let's see how to implement this.\n",
    "Since the dataset we used before did not have userId(which is necessary for collaborative filtering) let's load another dataset. We'll be using the [**Surprise** ](https://surprise.readthedocs.io/en/stable/index.html) library to implement SVD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "_uuid": "9a7faf48bf42293d18b29efac95e15010f6c900e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1029</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1061</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1129</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1172</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759205</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating   timestamp\n",
       "0       1       31     2.5  1260759144\n",
       "1       1     1029     3.0  1260759179\n",
       "2       1     1061     3.0  1260759182\n",
       "3       1     1129     2.0  1260759185\n",
       "4       1     1172     4.0  1260759205"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from surprise import Reader, Dataset, SVD, evaluate\n",
    "reader = Reader()\n",
    "ratings = pd.read_csv('../input/the-movies-dataset/ratings_small.csv')\n",
    "ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f8a5cd5580510c27846a564bfa6d13f1a6dfa6de"
   },
   "source": [
    "Note that in this dataset movies are rated on a scale of 5 unlike the earlier one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "_uuid": "75166cecd9821bab4299605b66ea2a7787a4c3b7"
   },
   "outputs": [],
   "source": [
    "data = Dataset.load_from_df(ratings[['userId', 'movieId', 'rating']], reader)\n",
    "data.split(n_folds=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "_uuid": "17880aee6e7750afed98002593251010dcf5fb20"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/surprise/evaluate.py:66: UserWarning: The evaluate() method is deprecated. Please use model_selection.cross_validate() instead.\n",
      "  'model_selection.cross_validate() instead.', UserWarning)\n",
      "/opt/conda/lib/python3.6/site-packages/surprise/dataset.py:193: UserWarning: Using data.split() or using load_from_folds() without using a CV iterator is now deprecated. \n",
      "  UserWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating RMSE, MAE of algorithm SVD.\n",
      "\n",
      "------------\n",
      "Fold 1\n",
      "RMSE: 0.8950\n",
      "MAE:  0.6911\n",
      "------------\n",
      "Fold 2\n",
      "RMSE: 0.8973\n",
      "MAE:  0.6915\n",
      "------------\n",
      "Fold 3\n",
      "RMSE: 0.8925\n",
      "MAE:  0.6880\n",
      "------------\n",
      "Fold 4\n",
      "RMSE: 0.8953\n",
      "MAE:  0.6893\n",
      "------------\n",
      "Fold 5\n",
      "RMSE: 0.8913\n",
      "MAE:  0.6864\n",
      "------------\n",
      "------------\n",
      "Mean RMSE: 0.8943\n",
      "Mean MAE : 0.6893\n",
      "------------\n",
      "------------\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "CaseInsensitiveDefaultDict(list,\n",
       "                           {'rmse': [0.8949504273136506,\n",
       "                             0.8973417810012421,\n",
       "                             0.892512885451081,\n",
       "                             0.8953178473275957,\n",
       "                             0.8912804359970683],\n",
       "                            'mae': [0.6910768535860065,\n",
       "                             0.6915109640888799,\n",
       "                             0.6880338754546221,\n",
       "                             0.689286066248924,\n",
       "                             0.6864460615792892]})"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd = SVD()\n",
    "evaluate(svd, data, measures=['RMSE', 'MAE'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "81f6d5f460d4cbaaa43b6bb86a0abd9bce1a3134"
   },
   "source": [
    "We get a mean Root Mean Sqaure Error of 0.89 approx which is more than good enough for our case. Let us now train on our dataset and arrive at predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "_uuid": "58007ee500ce1735d173c247d188a5cd603b803c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<surprise.prediction_algorithms.matrix_factorization.SVD at 0x7f5be72cf7b8>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainset = data.build_full_trainset()\n",
    "svd.fit(trainset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e130ef65d2c6a59823869d5b08c44e96e06f3b94"
   },
   "source": [
    "Let us pick user with user Id 1  and check the ratings she/he has given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "_uuid": "5a526fddacb2f7234e524e71224fdd1aecdd6ec0"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1029</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1061</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1129</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1172</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>1263</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>1287</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759187</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>1293</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>1339</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1260759125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>1343</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>1371</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1</td>\n",
       "      <td>1405</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1260759203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1</td>\n",
       "      <td>1953</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>2105</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1</td>\n",
       "      <td>2150</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1</td>\n",
       "      <td>2193</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1</td>\n",
       "      <td>2294</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1</td>\n",
       "      <td>2455</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1</td>\n",
       "      <td>2968</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1260759200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1</td>\n",
       "      <td>3671</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759117</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    userId  movieId  rating   timestamp\n",
       "0        1       31     2.5  1260759144\n",
       "1        1     1029     3.0  1260759179\n",
       "2        1     1061     3.0  1260759182\n",
       "3        1     1129     2.0  1260759185\n",
       "4        1     1172     4.0  1260759205\n",
       "5        1     1263     2.0  1260759151\n",
       "6        1     1287     2.0  1260759187\n",
       "7        1     1293     2.0  1260759148\n",
       "8        1     1339     3.5  1260759125\n",
       "9        1     1343     2.0  1260759131\n",
       "10       1     1371     2.5  1260759135\n",
       "11       1     1405     1.0  1260759203\n",
       "12       1     1953     4.0  1260759191\n",
       "13       1     2105     4.0  1260759139\n",
       "14       1     2150     3.0  1260759194\n",
       "15       1     2193     2.0  1260759198\n",
       "16       1     2294     2.0  1260759108\n",
       "17       1     2455     2.5  1260759113\n",
       "18       1     2968     1.0  1260759200\n",
       "19       1     3671     3.0  1260759117"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings[ratings['userId'] == 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "_uuid": "0cb981abe36b30f8f27b4c5dc1b1d0e090431651"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prediction(uid=1, iid=302, r_ui=3, est=2.8876857985684516, details={'was_impossible': False})"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd.predict(1, 302, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "50cf59d88c55c17de150e6a84a190e179bec2d33"
   },
   "source": [
    "For movie with ID 302, we get an estimated prediction of **2.618**. One startling feature of this recommender system is that it doesn't care what the movie is (or what it contains). It works purely on the basis of an assigned movie ID and tries to predict ratings based on how the other users have predicted the movie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "aece2ca6c5dcfcc8287562733f74a31fa115605c"
   },
   "source": [
    "## **Conclusion** \n",
    "We create recommenders using demographic , content- based and collaborative filtering. While demographic filtering is very elemantary and cannot be used practically, **Hybrid Systems** can take advantage of content-based and collaborative filtering as the two approaches are proved to be almost complimentary.\n",
    "This model was very baseline and only provides a fundamental framework to start with.\n",
    "\n",
    "I would like to mention some excellent refereces that I learned from\n",
    "1. [https://hackernoon.com/introduction-to-recommender-system-part-1-collaborative-filtering-singular-value-decomposition-44c9659c5e75](https://hackernoon.com/introduction-to-recommender-system-part-1-collaborative-filtering-singular-value-decomposition-44c9659c5e75)\n",
    "2. [https://www.kaggle.com/rounakbanik/movie-recommender-systems](https://www.kaggle.com/rounakbanik/movie-recommender-systems)\n",
    "3. [http://trouvus.com/wp-content/uploads/2016/03/A-hybrid-movie-recommender-system-based-on-neural-networks.pdf](http://trouvus.com/wp-content/uploads/2016/03/A-hybrid-movie-recommender-system-based-on-neural-networks.pdf)\n",
    "\n",
    "If you enjoyed reading the kernel , hit the upvote button !\n",
    "Please leave the feedback or suggestions below. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}