Click through rate using logistic classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CTR using logistic classifier\n",
    "\n",
    "In this notebook , we are going to use logistic classifier to predict ad click taking our avazu dataset. Now, tree based algorithms works fine for the this task but as the data grows, they become very computationaly inefficient very fast. \n",
    "\n",
    "Logistic based algorithms are considered most scalable algorithms for classification purposes in machine learning and are used widely when dataset is huge and we have limited computation power. Although, they might not give us accuracy as good as tree based algorithms, they really help in reducing our time to train a model and for online learning applications(ofcourse AD click prediction is one of them.)\n",
    "\n",
    "So, lets implement it using scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>click</th>\n",
       "      <th>hour</th>\n",
       "      <th>C1</th>\n",
       "      <th>banner_pos</th>\n",
       "      <th>site_id</th>\n",
       "      <th>site_domain</th>\n",
       "      <th>site_category</th>\n",
       "      <th>app_id</th>\n",
       "      <th>app_domain</th>\n",
       "      <th>...</th>\n",
       "      <th>device_type</th>\n",
       "      <th>device_conn_type</th>\n",
       "      <th>C14</th>\n",
       "      <th>C15</th>\n",
       "      <th>C16</th>\n",
       "      <th>C17</th>\n",
       "      <th>C18</th>\n",
       "      <th>C19</th>\n",
       "      <th>C20</th>\n",
       "      <th>C21</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.000009e+18</td>\n",
       "      <td>0</td>\n",
       "      <td>14102100</td>\n",
       "      <td>1005</td>\n",
       "      <td>0</td>\n",
       "      <td>1fbe01fe</td>\n",
       "      <td>f3845767</td>\n",
       "      <td>28905ebd</td>\n",
       "      <td>ecad2386</td>\n",
       "      <td>7801e8d9</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>15706</td>\n",
       "      <td>320</td>\n",
       "      <td>50</td>\n",
       "      <td>1722</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>-1</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.000017e+19</td>\n",
       "      <td>0</td>\n",
       "      <td>14102100</td>\n",
       "      <td>1005</td>\n",
       "      <td>0</td>\n",
       "      <td>1fbe01fe</td>\n",
       "      <td>f3845767</td>\n",
       "      <td>28905ebd</td>\n",
       "      <td>ecad2386</td>\n",
       "      <td>7801e8d9</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>15704</td>\n",
       "      <td>320</td>\n",
       "      <td>50</td>\n",
       "      <td>1722</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>100084</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.000037e+19</td>\n",
       "      <td>0</td>\n",
       "      <td>14102100</td>\n",
       "      <td>1005</td>\n",
       "      <td>0</td>\n",
       "      <td>1fbe01fe</td>\n",
       "      <td>f3845767</td>\n",
       "      <td>28905ebd</td>\n",
       "      <td>ecad2386</td>\n",
       "      <td>7801e8d9</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>15704</td>\n",
       "      <td>320</td>\n",
       "      <td>50</td>\n",
       "      <td>1722</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>100084</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.000064e+19</td>\n",
       "      <td>0</td>\n",
       "      <td>14102100</td>\n",
       "      <td>1005</td>\n",
       "      <td>0</td>\n",
       "      <td>1fbe01fe</td>\n",
       "      <td>f3845767</td>\n",
       "      <td>28905ebd</td>\n",
       "      <td>ecad2386</td>\n",
       "      <td>7801e8d9</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>15706</td>\n",
       "      <td>320</td>\n",
       "      <td>50</td>\n",
       "      <td>1722</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>100084</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.000068e+19</td>\n",
       "      <td>0</td>\n",
       "      <td>14102100</td>\n",
       "      <td>1005</td>\n",
       "      <td>1</td>\n",
       "      <td>fe8cc448</td>\n",
       "      <td>9166c161</td>\n",
       "      <td>0569f928</td>\n",
       "      <td>ecad2386</td>\n",
       "      <td>7801e8d9</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>18993</td>\n",
       "      <td>320</td>\n",
       "      <td>50</td>\n",
       "      <td>2161</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>-1</td>\n",
       "      <td>157</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             id  click      hour    C1  banner_pos   site_id site_domain  \\\n",
       "0  1.000009e+18      0  14102100  1005           0  1fbe01fe    f3845767   \n",
       "1  1.000017e+19      0  14102100  1005           0  1fbe01fe    f3845767   \n",
       "2  1.000037e+19      0  14102100  1005           0  1fbe01fe    f3845767   \n",
       "3  1.000064e+19      0  14102100  1005           0  1fbe01fe    f3845767   \n",
       "4  1.000068e+19      0  14102100  1005           1  fe8cc448    9166c161   \n",
       "\n",
       "  site_category    app_id app_domain ...  device_type device_conn_type    C14  \\\n",
       "0      28905ebd  ecad2386   7801e8d9 ...            1                2  15706   \n",
       "1      28905ebd  ecad2386   7801e8d9 ...            1                0  15704   \n",
       "2      28905ebd  ecad2386   7801e8d9 ...            1                0  15704   \n",
       "3      28905ebd  ecad2386   7801e8d9 ...            1                0  15706   \n",
       "4      0569f928  ecad2386   7801e8d9 ...            1                0  18993   \n",
       "\n",
       "   C15  C16   C17  C18  C19     C20  C21  \n",
       "0  320   50  1722    0   35      -1   79  \n",
       "1  320   50  1722    0   35  100084   79  \n",
       "2  320   50  1722    0   35  100084   79  \n",
       "3  320   50  1722    0   35  100084   79  \n",
       "4  320   50  2161    0   35      -1  157  \n",
       "\n",
       "[5 rows x 24 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "n_rows = 300000\n",
    "\n",
    "df = pd.read_csv(\"train.gz\", nrows = n_rows)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dropping the columns which seems to be useless."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(['click', 'id', 'hour', 'device_id', 'device_ip'], axis=1).values\n",
    "Y = df['click'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300000, 19)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training and validation sets\n",
    "\n",
    " We are going to split the data taking first 90% of data as train set and last 10% as validation set as data is chronological."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_train = int(n_rows * 0.9)\n",
    "\n",
    "X_train = X[:n_train]\n",
    "Y_train = Y[:n_train]\n",
    "X_test = X[n_train:]\n",
    "Y_test = Y[n_train:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# One Hot Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "enc = OneHotEncoder(handle_unknown='ignore')\n",
    "X_train_enc = enc.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 2)\t1.0\n",
      "  (0, 6)\t1.0\n",
      "  (0, 188)\t1.0\n",
      "  (0, 2608)\t1.0\n",
      "  (0, 2679)\t1.0\n",
      "  (0, 3771)\t1.0\n",
      "  (0, 3885)\t1.0\n",
      "  (0, 3929)\t1.0\n",
      "  (0, 4879)\t1.0\n",
      "  (0, 7315)\t1.0\n",
      "  (0, 7319)\t1.0\n",
      "  (0, 7475)\t1.0\n",
      "  (0, 7824)\t1.0\n",
      "  (0, 7828)\t1.0\n",
      "  (0, 7869)\t1.0\n",
      "  (0, 7977)\t1.0\n",
      "  (0, 7982)\t1.0\n",
      "  (0, 8021)\t1.0\n",
      "  (0, 8189)\t1.0\n"
     ]
    }
   ],
   "source": [
    "X_train_enc[0]\n",
    "print(X_train_enc[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test_enc = enc.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a logistic regression model\n",
    "\n",
    "In this section, we are going to train a logistic classifier using SGDclassifier(Stoichastic Gradient Descent classifier) which means that model weights are updated after every datapoint that goes in our model. This is really fast compared to standard gradient descent and tends to be more accurate as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "sgd_lr = SGDClassifier(loss='log', penalty=None, fit_intercept=True, n_iter=10, learning_rate='constant', eta0=0.01)\n",
    "sgd_lr.fit(X_train_enc.toarray(), Y_train)\n",
    "\n",
    "pred = sgd_lr.predict_proba(X_test_enc.toarray())[:, 1]\n",
    "print('Training samples: {0}, AUC on testing set: {1:.3f}'.format(n_train, roc_auc_score(Y_test, pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature selection with L1 regularisation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sgd_lr_l1 = SGDClassifier(loss='log', penalty='l1', alpha=0.0001, fit_intercept=True, n_iter=10, learning_rate='constant', eta0=0.01)\n",
    "sgd_lr_l1.fit(X_train_enc.toarray(), Y_train)\n",
    "\n",
    "coef_abs = np.abs(sgd_lr_l1.coef_)\n",
    "coef_abs\n",
    "\n",
    "# bottom 10 weights and the corresponding 10 least important features\n",
    "print(np.sort(coef_abs)[0][:10])\n",
    "\n",
    "feature_names = enc.get_feature_names()\n",
    "bottom_10 = np.argsort(coef_abs)[0][:10]\n",
    "print('10 least important features are:\\n', feature_names[bottom_10])\n",
    "\n",
    "# top 10 weights and the corresponding 10 most important features\n",
    "print(np.sort(coef_abs)[0][-10:])\n",
    "top_10 = np.argsort(coef_abs)[0][-10:]\n",
    "print('10 most important features are:\\n', feature_names[top_10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Online learning\n",
    "\n",
    "Now, as we can train our model and can select features using L1 regularisation, we are done with our logistic classifier.\n",
    "\n",
    "Its time to talk about online learning. Online learning basically means that data keeps coming in stream and we use the newest data to refine our model each times it comes in.\n",
    "\n",
    "Obviously, this is helpful in many cases where data is collected all the time and that is also the case with AD click. So, it works like that. \n",
    "1. Data is being collected everyday from websites and stored in a database.\n",
    "2. This data is used to train our model(or refine) every 10 days or so.\n",
    "3. Newer model is used to show new ads to users and the process repeats.\n",
    "\n",
    "The good thing about this kind of learning is that trends are always changing and our model knows those trends and can work better than when trained with a lot of data at once(known as Batch learning).\n",
    "\n",
    "One more thing to note here is that sometimes data that we have right now(our batch) is too big to fit in the memory and so, online learning can be used to take data as stream instead of batch(like streams of 1000 datapoints everytime) and it can be trained easily.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sgd_lr_l1 = SGDClassifier(loss='log', penalty='l1', alpha=0.0001, fit_intercept=True, n_iter=10, learning_rate='constant', eta0=0.01)\n",
    "sgd_lr_l1.fit(X_train_enc.toarray(), Y_train)\n",
    "\n",
    "coef_abs = np.abs(sgd_lr_l1.coef_)\n",
    "print(coef_abs)\n",
    "\n",
    "# bottom 10 weights and the corresponding 10 least important features\n",
    "print(np.sort(coef_abs)[0][:10])\n",
    "\n",
    "feature_names = enc.get_feature_names()\n",
    "bottom_10 = np.argsort(coef_abs)[0][:10]\n",
    "print('10 least important features are:\\n', feature_names[bottom_10])\n",
    "\n",
    "# top 10 weights and the corresponding 10 most important features\n",
    "print(np.sort(coef_abs)[0][-10:])\n",
    "top_10 = np.argsort(coef_abs)[0][-10:]\n",
    "print('10 most important features are:\\n', feature_names[top_10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The number of iterations is set to 1 if using partial_fit.\n",
    "sgd_lr_online = SGDClassifier(loss='log', penalty=None, fit_intercept=True, n_iter=1, learning_rate='constant', eta0=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the first 1,000,000 samples for training, and the next 100,000 for testing\n",
    "for i in range(10):\n",
    "    x_train = X_train[i*100000:(i+1)*100000]\n",
    "    y_train = Y_train[i*100000:(i+1)*100000]\n",
    "    x_train_enc = enc.transform(x_train)\n",
    "    sgd_lr_online.partial_fit(x_train_enc.toarray(), y_train, classes=[0, 1])\n",
    "\n",
    "print(\"--- %0.3fs seconds ---\" % (timeit.default_timer() - start_time))\n",
    "\n",
    "x_test_enc = enc.transform(X_test)\n",
    "\n",
    "pred = sgd_lr_online.predict_proba(x_test_enc.toarray())[:, 1]\n",
    "print('Training samples: {0}, AUC on testing set: {1:.3f}'.format(n_train * 10, roc_auc_score(Y_test, pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, for online leanring we got to use partial_fit function and fix the number of iterations to 1.\n",
    "\n",
    "Now, this is the example when our data was on disc but too big. We can easily change it for data coming from any external server or database using python functionings. \n",
    "\n",
    "Some hyperparameter for SGDclassifier:\n",
    "1. loss = 'log' defines that we want to use logitic clasifier. Default for SGD is SVM classifier.\n",
    "2. penalty . ofcourse we can L1 or L2 regularisation.\n",
    "3. n_iter . Number of iterations.\n",
    "4. learning_rate.\n",
    "5. eta0 = 0.01. Initial Leaning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}