Machine Learning - Hands on Lab - Session #4 - Model Design.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prelude\n",
    "\n",
    "## A. Choosing an Algorithm\n",
    "\n",
    "The first step to building a model is to decide what type of algorithm to use. Below we look at some of the options.\n",
    "\n",
    "### A.1. Decision Tree\n",
    "\n",
    "Arguably the most well known algorithm, and one of the simplest conceptually. The decision tree works in a similar manner to the decision tree that you might create when trying to understand which decision to make based on a range of variables.\n",
    "\n",
    "![Decision Tree](images/reg-decision-tree-1.png)\n",
    "\n",
    "The goal of the decision tree algorithm used for classification problems (like the one we are looking at) is to create one of these decision trees to classify records into a set number of categories. \n",
    "\n",
    "### A.2. Random Forest\n",
    "\n",
    "Given the limitations of decisions trees and the risk of overfitting, it may be tempting to think “why bother?” Fortunately, methods have been found to reduce the risk of overfitting and increase predictive power of decisions trees and the two most popular methods both have the same basic premise – to train multiple trees.\n",
    "\n",
    "The algorithm constructs a large number of different trees (as defined by the user) by randomly selecting the features that can be used to build each tree (as opposed to using all the features for each tree). Typically, the trees in a random forest also have the parameters set to ensure each tree will also be relatively shallow, meaning that the algorithm creates a large number of shallow decision trees (decision bonsai?). Once the trees are constructed, each tree is used to predict the outcome for a new record, with these multiple predictions then serving as votes, with a majority rules approach applied.\n",
    "\n",
    "![Random Forest](images/random_forest.jpg)\n",
    "\n",
    "\n",
    "### A.3. K-Nearest Neighbors\n",
    "\n",
    "The K-nearest neighbor algorithms are arguably one of the simplest algorithms in concept. The algorithm classifies a given object by looking at the classification of the k most similar records.\n",
    "\n",
    "This type of algorithm is called a lazy learner because during the training phase, it essentially just stores the data provided. Only when a new object needs to be classified does the algorithm start looking through the data to try to find the closest matches. **This algorithm scales extremly badly and is extremely sensitive with the number of features**\n",
    "\n",
    "![KNN](images/KNN.jpg)\n",
    "\n",
    "### A.4. Neural Networks\n",
    "\n",
    "As the name suggests, these algorithms simulate biological networks by creating a series of nodes and connecting them together. A neural network typically consists of three layers; an input layer, a hidden layer (although there can be multiple hidden layers) and an output layer.\n",
    "\n",
    "A model is trained by passing records through the network and weights adjusted at each node continually adjusted to ensure that the record ends up at the right ‘output node’.\n",
    "\n",
    "![Artificial Neural Network](images/ANN.png)\n",
    "\n",
    "### A.4. SVM: Support Vector Machine\n",
    "\n",
    "This type of algorithm, commonly used for text classification problems, is arguably the most difficult to visualize. At the simplest level, the algorithm tries to draw straight lines (or planes for classifications with more than 2 features) that best separate the classes provided. Although this sounds like a fairly simplistic approach to classifying objects, it becomes far more powerful due to the transformations (sometimes called a ‘kernel trick’) the algorithm can apply to the data before drawing these lines/planes. \n",
    "\n",
    "![SVM](images/SVM.jpg)\n",
    "\n",
    "## B. Creating the Model\n",
    "\n",
    "Often broken down into sub steps of feature construction and feature selection, here we will focus on feature construction. Below are a couple of ways additional features can be constructed and added to your dataset.\n",
    "\n",
    "### B.1. Cross Validation\n",
    "\n",
    "As mentioned in regards to decision trees, one of the keys risks when creating models of any type is the risk of overfitting. One of the primary ways data scientists will guard against overfitting is to estimate the accuracy of their models on data that was not used to train the model. To do this they typically use a method called cross validation. There are different methods for doing cross validation, but the method we will employ is called k-fold cross validation.\n",
    "\n",
    "**k-fold cross validation** involves splitting the training data into k subsets (where k is greater than or equal to 2), training the model using k – 1 of those subsets, then running the model on the subset that was not used in the training process. Because all of the data used in the cross validation process is training data, the correct classification for each record is known and so the predicted category can be compared to the actual category. Once all folds have been completed, the average score across all folds is taken as an estimate of how the model will perform on other data. An example of a 3-fold cross validation is shown below:\n",
    "\n",
    "![Cross Validation](images/cross-validation.png)\n",
    "\n",
    "### B.2. Parameter Tuning\n",
    "\n",
    "As you may have realized from the earlier description of the XGBoost algorithm – there are quite a few options (parameters) that we need to define to build the model. \n",
    "\n",
    "* How many trees to build? \n",
    "* How deep should each tree be? \n",
    "* How much extra weight will be attached to each misclassified record? \n",
    "\n",
    "Tuning these parameters to get the best results from the model is often one of the most time consuming things that data scientists do. \n",
    "\n",
    "Fortunately, the process can be automated to a large degree so that we do not have to sit there rerunning the model repeatedly and noting down the results. Even better, using the Scikit-Learn package, we can merge the parameter tuning and cross validation steps into one, allowing us to search for the best combination of parameters while using k-fold cross validation to verify the results.\n",
    "\n",
    "## D. The Importance of Domain Knowledge\n",
    "\n",
    "One of the things that may have occurred to you as you read through the various ways to modify and expand a dataset is how are you supposed to know what will help or not?\n",
    "\n",
    "This is where knowledge about the data you are using and what it represents becomes so important. This knowledge – referred to as domain knowledge – helps guide this entire process, including cleaning the data.\n",
    "\n",
    "Understanding how the data was collected helps to provide insight into potential errors in the data that might need to be addressed or shortcomings in the way the data was sampled (sample selection bias/errors). Understanding the relevant industry or market can also provide a range of insights including:\n",
    "\n",
    "* what additional information is available to expand your dataset\n",
    "* what information may help to increase prediction accuracy and what is likely to be irrelevant\n",
    "* if the model makes intuitive sense (e.g. can you predict the likelihood of a waking up with a headache based on whether someone slept with their shoes on?)\n",
    "* if the industry or market is changing in such a way that it is likely to make the model redundant in the near future."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's train the XGBoost Model buddy !\n",
    "\n",
    "## 1. Loading the Python libraries\n",
    "\n",
    "**Install needed for xgboost:*\n",
    "\n",
    "1. \\$ source activate python2\n",
    "2. \\$ pip install xgboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.5/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n",
      "/opt/conda/lib/python3.5/site-packages/sklearn/grid_search.py:43: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.\n",
      "  DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from datetime import datetime\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import sklearn as sk\n",
    "from sklearn.externals import joblib\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "\n",
    "import xgboost as xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Re-Import the Check NaN Function from Part 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def check_NaN_Values_in_df(df):\n",
    "    # searching for NaN values is all the columns\n",
    "    for col in df:\n",
    "        nan_count = df[col].isnull().sum()\n",
    "\n",
    "        if nan_count != 0:\n",
    "            print (col + \" => \"+  str(nan_count) + \" NaN Values\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Loading in the Data from Part 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>gende_other</th>\n",
       "      <th>gende_female</th>\n",
       "      <th>gende_male</th>\n",
       "      <th>gende_unknown</th>\n",
       "      <th>signu_basic</th>\n",
       "      <th>signu_facebook</th>\n",
       "      <th>signu_google</th>\n",
       "      <th>signu_weibo</th>\n",
       "      <th>signu_24</th>\n",
       "      <th>...</th>\n",
       "      <th>action_detail_view_resolutions</th>\n",
       "      <th>action_detail_view_search_results</th>\n",
       "      <th>action_detail_view_security_checks</th>\n",
       "      <th>action_detail_view_user_real_names</th>\n",
       "      <th>action_detail_wishlist</th>\n",
       "      <th>action_detail_wishlist_content_update</th>\n",
       "      <th>action_detail_wishlist_note</th>\n",
       "      <th>action_detail_your_listings</th>\n",
       "      <th>action_detail_your_reservations</th>\n",
       "      <th>action_detail_your_trips</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>jw6ama99vu</th>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>be8jznfeue</th>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>me9mt8pe66</th>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tvcpppste4</th>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5ncu73qoqt</th>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>106.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 703 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            age  gende_other  gende_female  gende_male  gende_unknown  \\\n",
       "jw6ama99vu   25            0             0           0              1   \n",
       "be8jznfeue   42            0             0           0              1   \n",
       "me9mt8pe66   60            0             0           0              1   \n",
       "tvcpppste4   24            0             0           1              0   \n",
       "5ncu73qoqt   -1            0             0           0              1   \n",
       "\n",
       "            signu_basic  signu_facebook  signu_google  signu_weibo  signu_24  \\\n",
       "jw6ama99vu            1               0             0            0         0   \n",
       "be8jznfeue            1               0             0            0         0   \n",
       "me9mt8pe66            1               0             0            0         0   \n",
       "tvcpppste4            0               1             0            0         0   \n",
       "5ncu73qoqt            1               0             0            0         0   \n",
       "\n",
       "                      ...             action_detail_view_resolutions  \\\n",
       "jw6ama99vu            ...                                        0.0   \n",
       "be8jznfeue            ...                                        0.0   \n",
       "me9mt8pe66            ...                                        0.0   \n",
       "tvcpppste4            ...                                        0.0   \n",
       "5ncu73qoqt            ...                                        0.0   \n",
       "\n",
       "            action_detail_view_search_results  \\\n",
       "jw6ama99vu                               15.0   \n",
       "be8jznfeue                               10.0   \n",
       "me9mt8pe66                                0.0   \n",
       "tvcpppste4                               17.0   \n",
       "5ncu73qoqt                              106.0   \n",
       "\n",
       "            action_detail_view_security_checks  \\\n",
       "jw6ama99vu                                 0.0   \n",
       "be8jznfeue                                 0.0   \n",
       "me9mt8pe66                                 0.0   \n",
       "tvcpppste4                                 0.0   \n",
       "5ncu73qoqt                                 0.0   \n",
       "\n",
       "            action_detail_view_user_real_names  action_detail_wishlist  \\\n",
       "jw6ama99vu                                 0.0                     0.0   \n",
       "be8jznfeue                                 0.0                     0.0   \n",
       "me9mt8pe66                                 0.0                     0.0   \n",
       "tvcpppste4                                 0.0                     0.0   \n",
       "5ncu73qoqt                                 0.0                     0.0   \n",
       "\n",
       "            action_detail_wishlist_content_update  \\\n",
       "jw6ama99vu                                    0.0   \n",
       "be8jznfeue                                    5.0   \n",
       "me9mt8pe66                                    1.0   \n",
       "tvcpppste4                                    9.0   \n",
       "5ncu73qoqt                                   32.0   \n",
       "\n",
       "            action_detail_wishlist_note  action_detail_your_listings  \\\n",
       "jw6ama99vu                          0.0                          0.0   \n",
       "be8jznfeue                          0.0                          0.0   \n",
       "me9mt8pe66                          0.0                          0.0   \n",
       "tvcpppste4                          0.0                          0.0   \n",
       "5ncu73qoqt                          0.0                          0.0   \n",
       "\n",
       "            action_detail_your_reservations  action_detail_your_trips  \n",
       "jw6ama99vu                              0.0                       0.0  \n",
       "be8jznfeue                              0.0                       0.0  \n",
       "me9mt8pe66                              0.0                       0.0  \n",
       "tvcpppste4                              0.0                       0.0  \n",
       "5ncu73qoqt                              0.0                       0.0  \n",
       "\n",
       "[5 rows x 703 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all = pd.read_csv(\"output/enriched.csv\", index_col=0)\n",
    "\n",
    "# Check for NaN Values => We must find: country_destination => 62096 NaN Values\n",
    "check_NaN_Values_in_df(df_all) \n",
    "\n",
    "df_all.sample(n=5) # Only display a few lines and not the whole dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4. Getting a training dataset\n",
    "\n",
    "In order to train the model (using cross validation and parameter tuning as outlined above), we first need to define our training dataset – remembering that we previously combined the training and test data to simplify the cleaning and transforming process. \n",
    "\n",
    "To feed these into the model, we also need to split the training data into the three main components \n",
    "- the **user IDs** (we don’t want to use these for training as they are randomly generated)\n",
    "- the **features** to use for training (X)\n",
    "- the **labels** we are trying to predict (y)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>date_account_created</th>\n",
       "      <th>timestamp_first_active</th>\n",
       "      <th>date_first_booking</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>signup_method</th>\n",
       "      <th>signup_flow</th>\n",
       "      <th>language</th>\n",
       "      <th>affiliate_channel</th>\n",
       "      <th>affiliate_provider</th>\n",
       "      <th>first_affiliate_tracked</th>\n",
       "      <th>signup_app</th>\n",
       "      <th>first_device_type</th>\n",
       "      <th>first_browser</th>\n",
       "      <th>country_destination</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gxn3p5htnn</td>\n",
       "      <td>2010-06-28</td>\n",
       "      <td>20090319043255</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-unknown-</td>\n",
       "      <td>NaN</td>\n",
       "      <td>facebook</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Mac Desktop</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>NDF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>820tgsjxq7</td>\n",
       "      <td>2011-05-25</td>\n",
       "      <td>20090523174809</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MALE</td>\n",
       "      <td>38.0</td>\n",
       "      <td>facebook</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>seo</td>\n",
       "      <td>google</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Mac Desktop</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>NDF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4ft3gnwmtx</td>\n",
       "      <td>2010-09-28</td>\n",
       "      <td>20090609231247</td>\n",
       "      <td>2010-08-02</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>56.0</td>\n",
       "      <td>basic</td>\n",
       "      <td>3</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Windows Desktop</td>\n",
       "      <td>IE</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>bjjt8pjhuk</td>\n",
       "      <td>2011-12-05</td>\n",
       "      <td>20091031060129</td>\n",
       "      <td>2012-09-08</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>42.0</td>\n",
       "      <td>facebook</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Mac Desktop</td>\n",
       "      <td>Firefox</td>\n",
       "      <td>other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>87mebub9p4</td>\n",
       "      <td>2010-09-14</td>\n",
       "      <td>20091208061105</td>\n",
       "      <td>2010-02-18</td>\n",
       "      <td>-unknown-</td>\n",
       "      <td>41.0</td>\n",
       "      <td>basic</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Mac Desktop</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           id date_account_created  timestamp_first_active date_first_booking  \\\n",
       "0  gxn3p5htnn           2010-06-28          20090319043255                NaN   \n",
       "1  820tgsjxq7           2011-05-25          20090523174809                NaN   \n",
       "2  4ft3gnwmtx           2010-09-28          20090609231247         2010-08-02   \n",
       "3  bjjt8pjhuk           2011-12-05          20091031060129         2012-09-08   \n",
       "4  87mebub9p4           2010-09-14          20091208061105         2010-02-18   \n",
       "\n",
       "      gender   age signup_method  signup_flow language affiliate_channel  \\\n",
       "0  -unknown-   NaN      facebook            0       en            direct   \n",
       "1       MALE  38.0      facebook            0       en               seo   \n",
       "2     FEMALE  56.0         basic            3       en            direct   \n",
       "3     FEMALE  42.0      facebook            0       en            direct   \n",
       "4  -unknown-  41.0         basic            0       en            direct   \n",
       "\n",
       "  affiliate_provider first_affiliate_tracked signup_app first_device_type  \\\n",
       "0             direct               untracked        Web       Mac Desktop   \n",
       "1             google               untracked        Web       Mac Desktop   \n",
       "2             direct               untracked        Web   Windows Desktop   \n",
       "3             direct               untracked        Web       Mac Desktop   \n",
       "4             direct               untracked        Web       Mac Desktop   \n",
       "\n",
       "  first_browser country_destination  \n",
       "0        Chrome                 NDF  \n",
       "1        Chrome                 NDF  \n",
       "2            IE                  US  \n",
       "3       Firefox               other  \n",
       "4        Chrome                  US  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Loading training csv file\n",
    "df_train = pd.read_csv(\"data/train_users_2.csv\")\n",
    "df_train.head(n=5) # Only display a few lines and not the whole dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10  7  7 ...,  7  7  4]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>gende_other</th>\n",
       "      <th>gende_female</th>\n",
       "      <th>gende_male</th>\n",
       "      <th>gende_unknown</th>\n",
       "      <th>signu_basic</th>\n",
       "      <th>signu_facebook</th>\n",
       "      <th>signu_google</th>\n",
       "      <th>signu_weibo</th>\n",
       "      <th>signu_24</th>\n",
       "      <th>...</th>\n",
       "      <th>action_detail_view_resolutions</th>\n",
       "      <th>action_detail_view_search_results</th>\n",
       "      <th>action_detail_view_security_checks</th>\n",
       "      <th>action_detail_view_user_real_names</th>\n",
       "      <th>action_detail_wishlist</th>\n",
       "      <th>action_detail_wishlist_content_update</th>\n",
       "      <th>action_detail_wishlist_note</th>\n",
       "      <th>action_detail_your_listings</th>\n",
       "      <th>action_detail_your_reservations</th>\n",
       "      <th>action_detail_your_trips</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ewyk4jrscx</th>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ohcbqsu9y6</th>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jhyslwyvsg</th>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rovxhr6nog</th>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71wrxhkhcc</th>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 703 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            age  gende_other  gende_female  gende_male  gende_unknown  \\\n",
       "ewyk4jrscx   -1            0             0           0              1   \n",
       "ohcbqsu9y6   28            0             1           0              0   \n",
       "jhyslwyvsg   -1            0             0           0              1   \n",
       "rovxhr6nog   -1            0             0           0              1   \n",
       "71wrxhkhcc   -1            0             0           0              1   \n",
       "\n",
       "            signu_basic  signu_facebook  signu_google  signu_weibo  signu_24  \\\n",
       "ewyk4jrscx            1               0             0            0         0   \n",
       "ohcbqsu9y6            0               1             0            0         0   \n",
       "jhyslwyvsg            1               0             0            0         0   \n",
       "rovxhr6nog            1               0             0            0         0   \n",
       "71wrxhkhcc            1               0             0            0         1   \n",
       "\n",
       "                      ...             action_detail_view_resolutions  \\\n",
       "ewyk4jrscx            ...                                        0.0   \n",
       "ohcbqsu9y6            ...                                        0.0   \n",
       "jhyslwyvsg            ...                                        0.0   \n",
       "rovxhr6nog            ...                                        0.0   \n",
       "71wrxhkhcc            ...                                        0.0   \n",
       "\n",
       "            action_detail_view_search_results  \\\n",
       "ewyk4jrscx                                6.0   \n",
       "ohcbqsu9y6                                4.0   \n",
       "jhyslwyvsg                               20.0   \n",
       "rovxhr6nog                               40.0   \n",
       "71wrxhkhcc                                3.0   \n",
       "\n",
       "            action_detail_view_security_checks  \\\n",
       "ewyk4jrscx                                 0.0   \n",
       "ohcbqsu9y6                                 0.0   \n",
       "jhyslwyvsg                                 0.0   \n",
       "rovxhr6nog                                 0.0   \n",
       "71wrxhkhcc                                 0.0   \n",
       "\n",
       "            action_detail_view_user_real_names  action_detail_wishlist  \\\n",
       "ewyk4jrscx                                 0.0                     0.0   \n",
       "ohcbqsu9y6                                 0.0                     0.0   \n",
       "jhyslwyvsg                                 0.0                     0.0   \n",
       "rovxhr6nog                                 0.0                     0.0   \n",
       "71wrxhkhcc                                 0.0                     0.0   \n",
       "\n",
       "            action_detail_wishlist_content_update  \\\n",
       "ewyk4jrscx                                    0.0   \n",
       "ohcbqsu9y6                                    1.0   \n",
       "jhyslwyvsg                                   35.0   \n",
       "rovxhr6nog                                    0.0   \n",
       "71wrxhkhcc                                    2.0   \n",
       "\n",
       "            action_detail_wishlist_note  action_detail_your_listings  \\\n",
       "ewyk4jrscx                          0.0                          0.0   \n",
       "ohcbqsu9y6                          0.0                          0.0   \n",
       "jhyslwyvsg                          0.0                          0.0   \n",
       "rovxhr6nog                          0.0                          0.0   \n",
       "71wrxhkhcc                          0.0                          0.0   \n",
       "\n",
       "            action_detail_your_reservations  action_detail_your_trips  \n",
       "ewyk4jrscx                              0.0                       0.0  \n",
       "ohcbqsu9y6                              0.0                       0.0  \n",
       "jhyslwyvsg                              0.0                       0.0  \n",
       "rovxhr6nog                              0.0                       0.0  \n",
       "71wrxhkhcc                              0.0                       0.0  \n",
       "\n",
       "[5 rows x 703 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prepare training data for modelling\n",
    "df_train.set_index('id', inplace=True)\n",
    "df_train = pd.concat([df_train['country_destination'], df_all], axis=1, join='inner')\n",
    "\n",
    "id_train = df_train.index.values\n",
    "labels = df_train['country_destination']\n",
    "le = LabelEncoder()\n",
    "y = le.fit_transform(labels)\n",
    "X = df_train.drop('country_destination', axis=1, inplace=False)\n",
    "\n",
    "# Checking Up\n",
    "print (y)\n",
    "X.sample(n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Training an XGBoost Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 12 candidates, totalling 36 fits\n",
      "[CV] n_estimators=25, learning_rate=0.1, max_depth=3 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.1, max_depth=3, score=0.695258 -  20.9s\n",
      "[CV] n_estimators=25, learning_rate=0.1, max_depth=3 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:   20.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  n_estimators=25, learning_rate=0.1, max_depth=3, score=0.692623 -  20.5s\n",
      "[CV] n_estimators=25, learning_rate=0.1, max_depth=3 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:   41.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  n_estimators=25, learning_rate=0.1, max_depth=3, score=0.690719 -  20.8s\n",
      "[CV] n_estimators=50, learning_rate=0.1, max_depth=3 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:  1.0min remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  n_estimators=50, learning_rate=0.1, max_depth=3, score=0.700540 -  38.7s\n",
      "[CV] n_estimators=50, learning_rate=0.1, max_depth=3 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed:  1.7min remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  n_estimators=50, learning_rate=0.1, max_depth=3, score=0.701036 -  39.0s\n",
      "[CV] n_estimators=50, learning_rate=0.1, max_depth=3 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:  2.3min remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  n_estimators=50, learning_rate=0.1, max_depth=3, score=0.700313 -  38.9s\n",
      "[CV] n_estimators=25, learning_rate=0.1, max_depth=4 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   6 out of   6 | elapsed:  3.0min remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  n_estimators=25, learning_rate=0.1, max_depth=4, score=0.697168 -  26.1s\n",
      "[CV] n_estimators=25, learning_rate=0.1, max_depth=4 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   7 out of   7 | elapsed:  3.4min remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  n_estimators=25, learning_rate=0.1, max_depth=4, score=0.695875 -  25.8s\n",
      "[CV] n_estimators=25, learning_rate=0.1, max_depth=4 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   8 out of   8 | elapsed:  3.8min remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  n_estimators=25, learning_rate=0.1, max_depth=4, score=0.695394 -  26.0s\n",
      "[CV] n_estimators=50, learning_rate=0.1, max_depth=4 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:  4.3min remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  n_estimators=50, learning_rate=0.1, max_depth=4, score=0.701434 -  49.5s\n",
      "[CV] n_estimators=50, learning_rate=0.1, max_depth=4 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.1, max_depth=4, score=0.702499 -  49.7s\n",
      "[CV] n_estimators=50, learning_rate=0.1, max_depth=4 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.1, max_depth=4, score=0.701980 -  49.8s\n",
      "[CV] n_estimators=25, learning_rate=0.1, max_depth=5 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.1, max_depth=5, score=0.698631 -  31.4s\n",
      "[CV] n_estimators=25, learning_rate=0.1, max_depth=5 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.1, max_depth=5, score=0.699898 -  31.3s\n",
      "[CV] n_estimators=25, learning_rate=0.1, max_depth=5 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.1, max_depth=5, score=0.699378 -  31.4s\n",
      "[CV] n_estimators=50, learning_rate=0.1, max_depth=5 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.1, max_depth=5, score=0.701881 - 1.0min\n",
      "[CV] n_estimators=50, learning_rate=0.1, max_depth=5 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.1, max_depth=5, score=0.703963 - 1.0min\n",
      "[CV] n_estimators=50, learning_rate=0.1, max_depth=5 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.1, max_depth=5, score=0.704216 - 1.0min\n",
      "[CV] n_estimators=25, learning_rate=0.3, max_depth=3 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.3, max_depth=3, score=0.700744 -  20.5s\n",
      "[CV] n_estimators=25, learning_rate=0.3, max_depth=3 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.3, max_depth=3, score=0.701971 -  20.8s\n",
      "[CV] n_estimators=25, learning_rate=0.3, max_depth=3 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.3, max_depth=3, score=0.700191 -  20.7s\n",
      "[CV] n_estimators=50, learning_rate=0.3, max_depth=3 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.3, max_depth=3, score=0.702978 -  38.7s\n",
      "[CV] n_estimators=50, learning_rate=0.3, max_depth=3 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.3, max_depth=3, score=0.702052 -  38.7s\n",
      "[CV] n_estimators=50, learning_rate=0.3, max_depth=3 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.3, max_depth=3, score=0.703484 -  38.9s\n",
      "[CV] n_estimators=25, learning_rate=0.3, max_depth=4 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.3, max_depth=4, score=0.700865 -  25.9s\n",
      "[CV] n_estimators=25, learning_rate=0.3, max_depth=4 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.3, max_depth=4, score=0.701931 -  25.9s\n",
      "[CV] n_estimators=25, learning_rate=0.3, max_depth=4 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.3, max_depth=4, score=0.701492 -  26.1s\n",
      "[CV] n_estimators=50, learning_rate=0.3, max_depth=4 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.3, max_depth=4, score=0.704075 -  49.7s\n",
      "[CV] n_estimators=50, learning_rate=0.3, max_depth=4 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.3, max_depth=4, score=0.702743 -  49.7s\n",
      "[CV] n_estimators=50, learning_rate=0.3, max_depth=4 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.3, max_depth=4, score=0.704825 -  49.8s\n",
      "[CV] n_estimators=25, learning_rate=0.3, max_depth=5 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.3, max_depth=5, score=0.702572 -  31.3s\n",
      "[CV] n_estimators=25, learning_rate=0.3, max_depth=5 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.3, max_depth=5, score=0.702662 -  31.4s\n",
      "[CV] n_estimators=25, learning_rate=0.3, max_depth=5 .................\n",
      "[CV]  n_estimators=25, learning_rate=0.3, max_depth=5, score=0.702142 -  31.6s\n",
      "[CV] n_estimators=50, learning_rate=0.3, max_depth=5 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.3, max_depth=5, score=0.701841 - 1.0min\n",
      "[CV] n_estimators=50, learning_rate=0.3, max_depth=5 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.3, max_depth=5, score=0.702499 - 1.0min\n",
      "[CV] n_estimators=50, learning_rate=0.3, max_depth=5 .................\n",
      "[CV]  n_estimators=50, learning_rate=0.3, max_depth=5, score=0.702915 - 1.0min\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  36 out of  36 | elapsed: 22.7min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best score: 0.704\n",
      "Best parameters set:\n",
      "\tlearning_rate: 0.3\n",
      "\tmax_depth: 4\n",
      "\tn_estimators: 50\n"
     ]
    }
   ],
   "source": [
    "# Grid Search - Used to find best combination of parameters\n",
    "XGB_model = xgb.XGBClassifier(objective='multi:softprob', subsample=0.5, colsample_bytree=0.5, seed=0)\n",
    "param_grid = {'max_depth': [3, 4, 5], 'learning_rate': [0.1, 0.3], 'n_estimators': [25, 50]}\n",
    "\n",
    "model = GridSearchCV(\n",
    "    estimator=XGB_model, \n",
    "    param_grid=param_grid, \n",
    "    scoring='accuracy', \n",
    "    verbose=10, \n",
    "    n_jobs=1, \n",
    "    iid=True, \n",
    "    refit=True, cv=3\n",
    ")\n",
    "\n",
    "model.fit(X, y)\n",
    "print(\"Best score: %0.3f\" % model.best_score_)\n",
    "print(\"Best parameters set:\")\n",
    "best_parameters = model.best_estimator_.get_params()\n",
    "for param_name in sorted(param_grid.keys()):\n",
    "    print(\"\\t%s: %r\" % (param_name, best_parameters[param_name]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 6. Saving the model \n",
    "\n",
    "=> loaded_model = joblib.load(\"models/0001.model\") # To load back the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['models/0001.model']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We create the output directory if necessary\n",
    "if not os.path.exists(\"models\"):\n",
    "    os.makedirs(\"models\")\n",
    "\n",
    "# save model to file\n",
    "joblib.dump(model, 'models/0001.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Making the Predictions\n",
    "\n",
    "Now that we have trained a model based on the best parameters, the next step is to use the model to make predictions for the records in the testing dataset. Again we need to extract the testing data out of the combined dataset we created for the cleaning and transformation steps, and again we need to separate the main components for the model. After these steps, we use the model created in the previous step to make the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>date_account_created</th>\n",
       "      <th>timestamp_first_active</th>\n",
       "      <th>date_first_booking</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>signup_method</th>\n",
       "      <th>signup_flow</th>\n",
       "      <th>language</th>\n",
       "      <th>affiliate_channel</th>\n",
       "      <th>affiliate_provider</th>\n",
       "      <th>first_affiliate_tracked</th>\n",
       "      <th>signup_app</th>\n",
       "      <th>first_device_type</th>\n",
       "      <th>first_browser</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5uwns89zht</td>\n",
       "      <td>2014-07-01</td>\n",
       "      <td>20140701000006</td>\n",
       "      <td>NaN</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>35.0</td>\n",
       "      <td>facebook</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Moweb</td>\n",
       "      <td>iPhone</td>\n",
       "      <td>Mobile Safari</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>jtl0dijy2j</td>\n",
       "      <td>2014-07-01</td>\n",
       "      <td>20140701000051</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-unknown-</td>\n",
       "      <td>NaN</td>\n",
       "      <td>basic</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Moweb</td>\n",
       "      <td>iPhone</td>\n",
       "      <td>Mobile Safari</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>xx0ulgorjt</td>\n",
       "      <td>2014-07-01</td>\n",
       "      <td>20140701000148</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-unknown-</td>\n",
       "      <td>NaN</td>\n",
       "      <td>basic</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>linked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Windows Desktop</td>\n",
       "      <td>Chrome</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6c6puo6ix0</td>\n",
       "      <td>2014-07-01</td>\n",
       "      <td>20140701000215</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-unknown-</td>\n",
       "      <td>NaN</td>\n",
       "      <td>basic</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>linked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Windows Desktop</td>\n",
       "      <td>IE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>czqhjk3yfe</td>\n",
       "      <td>2014-07-01</td>\n",
       "      <td>20140701000305</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-unknown-</td>\n",
       "      <td>NaN</td>\n",
       "      <td>basic</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Mac Desktop</td>\n",
       "      <td>Safari</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           id date_account_created  timestamp_first_active  \\\n",
       "0  5uwns89zht           2014-07-01          20140701000006   \n",
       "1  jtl0dijy2j           2014-07-01          20140701000051   \n",
       "2  xx0ulgorjt           2014-07-01          20140701000148   \n",
       "3  6c6puo6ix0           2014-07-01          20140701000215   \n",
       "4  czqhjk3yfe           2014-07-01          20140701000305   \n",
       "\n",
       "   date_first_booking     gender   age signup_method  signup_flow language  \\\n",
       "0                 NaN     FEMALE  35.0      facebook            0       en   \n",
       "1                 NaN  -unknown-   NaN         basic            0       en   \n",
       "2                 NaN  -unknown-   NaN         basic            0       en   \n",
       "3                 NaN  -unknown-   NaN         basic            0       en   \n",
       "4                 NaN  -unknown-   NaN         basic            0       en   \n",
       "\n",
       "  affiliate_channel affiliate_provider first_affiliate_tracked signup_app  \\\n",
       "0            direct             direct               untracked      Moweb   \n",
       "1            direct             direct               untracked      Moweb   \n",
       "2            direct             direct                  linked        Web   \n",
       "3            direct             direct                  linked        Web   \n",
       "4            direct             direct               untracked        Web   \n",
       "\n",
       "  first_device_type  first_browser  \n",
       "0            iPhone  Mobile Safari  \n",
       "1            iPhone  Mobile Safari  \n",
       "2   Windows Desktop         Chrome  \n",
       "3   Windows Desktop             IE  \n",
       "4       Mac Desktop         Safari  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test = pd.read_csv(\"data/test_users.csv\")\n",
    "df_test.head(n=5) # Only display a few lines and not the whole dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Prepare test data for prediction\n",
    "df_test.set_index('id', inplace=True)\n",
    "df_test = pd.merge(df_test.loc[:,['date_first_booking']], df_all, how='left', left_index=True, right_index=True, sort=False)\n",
    "X_test = df_test.drop('date_first_booking', axis=1, inplace=False)\n",
    "X_test = X_test.fillna(-1)\n",
    "id_test = df_test.index.values\n",
    "\n",
    "# Make predictions\n",
    "y_pred = model.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Taking the 5 classes with highest probabilities\n",
    "ids = []  #list of ids\n",
    "cts = []  #list of countries\n",
    "for i in range(len(id_test)):\n",
    "    idx = id_test[i]\n",
    "    ids += [idx] * 5\n",
    "    cts += le.inverse_transform(np.argsort(y_pred[i])[::-1])[:5].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 8. Saving Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We create the output directory if necessary\n",
    "if not os.path.exists(\"output\"):\n",
    "    os.makedirs(\"output\")\n",
    "\n",
    "#Generate submission\n",
    "sub = pd.DataFrame(np.column_stack((ids, cts)), columns=['id', 'country'])\n",
    "sub.to_csv('output/submission.csv', index=False, sep=',')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}