Assignment2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<h1 align=\"center\">CS340 - Assignment 2</h1>\n",
    "<h3 align=\"center\">Due Date: 18 April 2017</h3>\n",
    "<br>\n",
    "<p style=\"text-indent: 40px\">In this project we are going to work on movielens 20m dataset with <b>PySpark Dataframes</b>. \n",
    "The dataset is included in the assignment folder. You should unzip it and upload each file to the ibm's datascience tool.\n",
    "</p>\n",
    "\n",
    "In order to understand the data in each csv file, you should read README.txt file inside the folder.\n",
    "\n",
    "## Reliable Users\n",
    "\n",
    "<p style=\"text-indent: 40px\">For the first part of this assignment you should find out the top 1000 reliable users. In order to do that you would need tags' relevance score from genome-scores.csv file. Assuming the relevance scores of movies' tags in genome-scores.csv are correct, we can find out if a user's tag to a particular movie is relevant or not. If we get the total of those relevance scores for each user then we can rank the them according to this criteria.</p>\n",
    "\n",
    "<p style=\"text-indent: 40px\">One caveat is that relevance scores are between 0 and 1. We want to scale it between -1 and 1 and also we want to punish more if the issued tag is not relevant. First multiply relevance with 20 and then substract 12 so relevance score range will be between -12 and 8 and then apply hyperbolic tangent. That way relevance's range will be between -1 and 1.</p>\n",
    "\n",
    "<p style=\"text-indent: 40px\"> Note: You should use PySpark's <b>tanh</b> function. Also use caching appropriately in order to be able to get the results in a reasonable time.</p>\n",
    "<p style=\"text-indent: 40px\"> Caution: Do not use RDDs. Every operation should be done on the executors not in the driver machine.</p>\n",
    "<p style=\"text-indent: 40px\">Important: Do not discuss the solution with your friends. <b>Plagiarism</b> will not be tolerated and issue will be referred to the <b>disciplinary committee</b>.</p>\n",
    "\n",
    "<div>\n",
    "    <img src=\"http://image.prntscr.com/image/1f91352835964c31b04be03da7d53581.jpg\" width=500>\n",
    "\n",
    "    <center><strong>Figure 1: Tanh function.</strong></center>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Best Movies\n",
    "\n",
    "<br>\n",
    "<p style=\"text-indent: 40px\">For the second part of this assignment, we are going to get best 20 movies from those 1000 people's scores. For each movie in the ratings, you should multiply rating with the users score that we previously calculated and get the total of this score for each movie.\n",
    "\n",
    "In the end we should be able to see the titles of the first 20 movies ranked by this criteria.\n",
    "</p>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    ""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2 with Spark 1.6",
   "language": "python",
   "name": "python2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}