Unbalanced_Data_Handling.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bf9ab542",
   "metadata": {
    "id": "b8AiVuH_phgT"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "pd.set_option('max_columns', 50)\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cc3d71d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the LendingClub dataset\n",
    "\n",
    "df = pd.read_csv('LendingClub.csv')\n",
    "\n",
    "# Convert categorical variable \"purpose\" to dummies, and drop the most frequent dummy\n",
    "df = pd.get_dummies(df, columns=['purpose']).drop(columns=['purpose_debt_consolidation'])\n",
    "\n",
    "X = df.drop(columns=['not_fully_paid'])\n",
    "y = df['not_fully_paid']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,random_state=365)\n",
    "X_train = X_train.copy()\n",
    "X_test = X_test.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4de7e8b",
   "metadata": {
    "id": "pm9X6Yu2ovyz"
   },
   "source": [
    "## Dealing with severely unbalanced data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfe22de0",
   "metadata": {},
   "source": [
    "#### class_weight:put weight into the target: is 0 and 1 equal: class_weight={0:1,1:2}: which type of mistake is more severe  \n",
    "#### class_weight is associated with classification mistake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "078bb49c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy is: 84.39%\n",
      "The confusion matrix is:\n",
      "[[1616    0]\n",
      " [ 299    1]]\n"
     ]
    }
   ],
   "source": [
    "# Logistic regression, penalty 还有一个参数叫elastic net！！！！\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "clf = LogisticRegression(penalty='none', max_iter=1000,class_weight={0:1,1:1})\n",
    "clf.fit(X_train,y_train)\n",
    "\n",
    "y_predict = clf.predict(X_test)\n",
    "\n",
    "accuracy = accuracy_score(y_test, y_predict).round(4)\n",
    "print(f\"The accuracy is: {accuracy:.2%}\")\n",
    "print(\"The confusion matrix is:\")\n",
    "cm = confusion_matrix(y_test, y_predict)\n",
    "print(cm)\n",
    "\n",
    "# save the results for later comparison\n",
    "clf_lr = clf\n",
    "accuracy_lr = accuracy\n",
    "cm_lr = cm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9d7d66e",
   "metadata": {},
   "source": [
    "The accuracy above, while pretty high, is misleading because we actually got an extremely biased trained model: this trained model almost always predicts that borrowers will not default, as evident from the confusion matrix. \n",
    "\n",
    "This extremely biased trained model is triggered by the severely unbalanced dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb7683e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4774d51",
   "metadata": {
    "id": "pm9X6Yu2ovyz"
   },
   "source": [
    "### Options for dealing with severely unbalanced data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43cf2655",
   "metadata": {
    "id": "0T3JL5D9phgV"
   },
   "source": [
    "+ **Option 1. Re-sampling the data to make it balanced.** This can be done in two ways:\n",
    "  + **undersampling** the majority class\n",
    "    + this is the usual choice when we have large enough data\n",
    "  + **oversampling** the minority class\n",
    "    + it may cause the [data leakage](https://towardsdatascience.com/data-leakage-in-machine-learning-10bdd3eec742) problem, thus should be avoided unless the data size is too small\n",
    "+ **Option 2. Do not use \"accuracy\" as the performance metric.** Instead, \n",
    "  + use alternative metrics that can give different weight to different classes of the target, e.g., counts '1' more heavily than '0' in the target of the LendingClub dataset (to be discussed in the next lecture)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c44a9d5b",
   "metadata": {
    "id": "NSKsulLJRcs5"
   },
   "source": [
    "### Undersampling the majority class\n",
    "\n",
    "The function for this is `sklearn.utils.resample()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "aa43f2a4",
   "metadata": {
    "id": "wLIv9mpVphgW"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The majority class contains 8045 records. \n",
      "The minority class contains 1533 records. \n"
     ]
    }
   ],
   "source": [
    "# First, separate the classes, where we already know 'not_fully_paid==0' is the majority class\n",
    "df_0 = df[df.not_fully_paid==0]\n",
    "df_1 = df[df.not_fully_paid==1]\n",
    "\n",
    "# Remember the sizes of the two classes\n",
    "n_majority_class = df_0.shape[0]\n",
    "n_minority_class = df_1.shape[0]\n",
    "print(f\"The majority class contains {n_majority_class} records. \\nThe minority class contains {n_minority_class} records. \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0982670b",
   "metadata": {
    "id": "gMAmb7rMzqIl"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1533, 19)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.utils import resample\n",
    "\n",
    "# undersample the majority class\n",
    "# replace=True means put back and shuffle, then choose\n",
    "df_0_undersampled = resample(df_0, replace=False, \n",
    "                             n_samples=n_minority_class, \n",
    "                             random_state=1234)\n",
    "df_0_undersampled.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "aa715ba8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8045, 19)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.utils import resample\n",
    "\n",
    "# oversample the majority class\n",
    "# replace=True means put back and shuffle, then choose\n",
    "#This is called bootstrapping: repeatedly put out data from the same dataset over and over again\n",
    "df_1_oversampled = resample(df_1, replace=True, \n",
    "                             n_samples=n_majority_class, \n",
    "                             random_state=1234)\n",
    "df_1_oversampled.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1a515fd",
   "metadata": {
    "id": "nGRI_vGou5EH"
   },
   "source": [
    "### Combining the two classes into a single (resampled) dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85ddcc9d",
   "metadata": {
    "id": "IgdJz3QwzlC5"
   },
   "outputs": [],
   "source": [
    "df_balanced = pd.concat([df_0_undersampled, df_1])\n",
    "df_balanced.not_fully_paid.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df8cd2fd",
   "metadata": {
    "id": "kfQ9RZiIphgX"
   },
   "outputs": [],
   "source": [
    "# Save the balanced data for future use\n",
    "df_balanced.to_csv('LendingClub_balanced.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfc88140",
   "metadata": {
    "id": "JLddQYvlsb7C"
   },
   "source": [
    "### Comments on oversampling\n",
    "\n",
    "The reason it should be avoided when possible: the the [data leakage](https://towardsdatascience.com/data-leakage-in-machine-learning-10bdd3eec742) problem.\n",
    "\n",
    "However, if you have to use it because the size of the minority class is too small, here are a few hints:\n",
    "+ make sure you do `train_test_split()` *before* oversampling (why?)\n",
    "+ ways to oversample:\n",
    "  + Use [bootstrapping](https://en.wikipedia.org/wiki/Bootstrapping_(statistics)), a.k.a. `resample()` with the option `replace=True`.\n",
    "  + Use [`imblearn.over_sampling.SMOTE`](https://imbalanced-learn.org/stable/over_sampling.html) -- a k-NN inspired method to create synthetic records\n",
    "    + [A nice tutorial on SMOTE](https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c69223f4",
   "metadata": {
    "id": "w8VGfXVI0RT8"
   },
   "source": [
    "### Splitting this balanced data into train and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eaae2b3d",
   "metadata": {
    "id": "zIFrz8Y3d2O_"
   },
   "outputs": [],
   "source": [
    "X = df_balanced.drop(columns=['not_fully_paid'])\n",
    "y = df_balanced['not_fully_paid']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,random_state=365)\n",
    "X_train = X_train.copy()\n",
    "X_test = X_test.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dad1f072",
   "metadata": {
    "id": "w8VGfXVI0RT8"
   },
   "source": [
    "### Training the logistic regression model over this balanced data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6184f1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf.fit(X_train,y_train)\n",
    "\n",
    "y_predict = clf.predict(X_test)\n",
    "\n",
    "accuracy = accuracy_score(y_test, y_predict).round(4)\n",
    "print(f\"The accuracy is: {accuracy:.2%}\")\n",
    "print(\"The confusion matrix is:\")\n",
    "cm = confusion_matrix(y_test, y_predict)\n",
    "print(cm)\n",
    "\n",
    "# save the results for later comparison\n",
    "clf_lr = clf\n",
    "accuracy_lr = accuracy\n",
    "cm_lr = cm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0a90820",
   "metadata": {},
   "source": [
    "As shown above, the predictions are no longer extremely biased."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d2d5871",
   "metadata": {
    "id": "pm9X6Yu2ovyz"
   },
   "source": [
    "## Normalize/standardize the data\n",
    "\n",
    "Recall that: \"normalize\" --> [0,1], and \"standardize\" --> mean 0 and std 1.\n",
    "\n",
    "The LendingClub dataset consists of columns of varying scales. In addition, some columns are significantly skewed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1157bee7",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.agg(['mean','std','skew'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "961ce758",
   "metadata": {},
   "source": [
    "Variables of varying scales, and skewed variables, are commonly seen in business datasets. \n",
    "+ E.g., salary is in the tens of thousands, while age is usually in two digits\n",
    "+ E.g., monetary variables (salary, spending, ...) are often right skewed "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c88ff1b",
   "metadata": {},
   "source": [
    "### *Do we need to normalize/standardize the data?*\n",
    "\n",
    "Nowadays, almost always **yes** because:\n",
    "+ Many learning algorithms are sensitive to varying data scales (e.g., kNN, SVM) or varying data distribution shapes (e.g., regression)\n",
    "+ **Regularization** is heavily used in modern machine learning. And regularization does NOT work without data normalization/stanardization\n",
    "    + See these two brief posts on the concept of regularization: [Over-fitting and Regularization](https://towardsdatascience.com/over-fitting-and-regularization-64d16100f45c), [L1 and L2 Regularization Methods](https://towardsdatascience.com/l1-and-l2-regularization-methods-ce25e7fc831c)\n",
    "\n",
    "Tree-based classifiers are an exception because they don't compare column values when splitting.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2710bdb",
   "metadata": {},
   "source": [
    "### Manually performing data normalization/standardization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0306bb0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a copy, as later we'll also try another standardization method\n",
    "X_train_std_manual = X_train.copy()\n",
    "X_test_std_manual = X_test.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce4b800c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Below we normalize/standardize some input columns\n",
    "# Remember we need to work on both train and test datasets\n",
    "# In practice, remember to update your data description file afterwards!\n",
    "\n",
    "for x in [X_train_std_manual, X_test_std_manual]:\n",
    "    x['installment1000'] = x.installment / 1000\n",
    "    x.drop('installment', axis=1, inplace=True)\n",
    "\n",
    "    x['fico_ratio'] = x.fico / 850\n",
    "    x.drop('fico', axis=1, inplace=True)\n",
    "\n",
    "    x['decades_with_cr_line'] = x.days_with_cr_line / 3650\n",
    "    x.drop('days_with_cr_line', axis=1, inplace=True)\n",
    "\n",
    "    x['log_revol_bal'] = np.log(x.revol_bal + 1)\n",
    "    x.drop('revol_bal', axis=1, inplace=True)\n",
    "\n",
    "    x.revol_util = x.revol_util / 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d228eb36",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the summary statistics of the transformed data\n",
    "X_train_std_manual.agg(['mean','std','skew'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bec8e24e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now let's run the logistic regression again with this transformed data\n",
    "clf.fit(X_train_std_manual,y_train)\n",
    "\n",
    "y_predict = clf.predict(X_test_std_manual)\n",
    "\n",
    "accuracy = accuracy_score(y_test, y_predict).round(4)\n",
    "print(f\"The accuracy is: {accuracy:.2%}\")\n",
    "print(\"The confusion matrix is:\")\n",
    "cm = confusion_matrix(y_test, y_predict)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "293fed23",
   "metadata": {},
   "source": [
    "### Automatically performing data normalization/standardization\n",
    "\n",
    "We can automatically standardize data using `sklearn.preprocessing.StandardScaler`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c32ec397",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a copy, as later we'll also try another standardization method\n",
    "X_train_std_auto = X_train.copy()\n",
    "X_test_std_auto = X_test.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55216c95",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ca283c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Warning: we don't want to standardize any categorical columns!\n",
    "# Therefore, let's pick out only the numerical ones.\n",
    "num_columns = ['int_rate', 'installment', 'log_annual_inc', 'dti', \n",
    "               'fico', 'days_with_cr_line', 'revol_bal', 'revol_util',\n",
    "               'inq_last_6mths', 'delinq_2yrs', 'pub_rec']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4c77a41",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "scaler.fit(X_train_std_auto[num_columns])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbfc6e95",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(scaler.mean_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd5c2f5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_std_auto[num_columns] = scaler.transform(X_train_std_auto[num_columns])\n",
    "X_test_std_auto[num_columns] = scaler.transform(X_test_std_auto[num_columns])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09dea431",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify that standardization is done\n",
    "X_train_std_auto.agg(['mean','std','skew'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12bdab1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "## The above scaler.fit() and scaler.transform() steps can be combined into one:\n",
    "# X_train_std_auto[num_columns] = scaler.fit_transform(X_train_std_auto[num_columns])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7e4bcca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now let's run the logistic regression again with this standardized data\n",
    "clf.fit(X_train_std_auto,y_train)\n",
    "\n",
    "y_predict = clf.predict(X_test_std_auto)\n",
    "\n",
    "accuracy = accuracy_score(y_test, y_predict).round(4)\n",
    "print(f\"The accuracy is: {accuracy:.2%}\")\n",
    "print(\"The confusion matrix is:\")\n",
    "cm = confusion_matrix(y_test, y_predict)\n",
    "print(cm)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}