Project Learning Analytics (Part IV).ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "In Part III, we performed exploratory data analysis extensively to develop an intuition behind our data. \n",
    "\n",
    "In this section, we dummify our categorical variables and prepare our data before model training in the last section.\n",
    "\n",
    "Recommended readings:\n",
    "1. https://towardsdatascience.com/feature-engineering-for-machine-learning-3a5e293a5114\n",
    "2. https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/\n",
    "\n",
    "In this notebook, we will do the following:\n",
    "1. one-hot encode our categorical data\n",
    "2. extract specific columns containing numerical data\n",
    "3. merge the numerical data with the dummified categorical data\n",
    "4. export the final DataFrame as CSV\n",
    "\n",
    "For one-hot encoding/dummification of the categorical values, there are various ways to do it. \n",
    "\n",
    "However, no matter which method you do it, make sure you drop one column to avoid the <strong>dummy variable trap</strong>.\n",
    "\n",
    "For example, if your column contains four categorical values, you'd have to drop one of the four columns after one-hot encoding.\n",
    "\n",
    "![OnehotEncodingExample.png](https://uplevelsg.s3-ap-southeast-1.amazonaws.com/ProjectLearningAnalytics/OnehotEncodingExample.png)\n",
    "\n",
    "In this example, we drop is_Purple because is_Purple is redundant, i.e. you can infer that row's values based on is_Black, is_Blue, and is_Red. If all of those values are 0, it means the row contains purple instead.\n",
    "\n",
    "Suggested reading: https://machinelearningmastery.com/one-hot-encoding-for-categorical-data/\n",
    "\n",
    "### Step 1: Import pandas\n",
    "You just need pandas in this library - export it as pd."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 1: Import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Read the CSV from Part II\n",
    "We will work with the CSV from the merged DataFrame from Part II"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 2: Read your CSV from Part II"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Replace 55<= in age_band to 55+\n",
    "'<' and '=' do not play well with certain models so we might as well replace all of the values in the age_band column first.\n",
    "\n",
    "![Replace55.png](https://uplevelsg.s3-ap-southeast-1.amazonaws.com/ProjectLearningAnalytics/Replace55.png)\n",
    "\n",
    "<strong>Hint: Google \"replace values in column pandas\"</strong>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Step 3: Replace '55<='' in age_band to '55+'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Dummify core_module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 4: Declare a variable and store the dummified/one-hot encoded values from 'core_module'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Dummify code_presentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 5: Declare a variable and store the dummified/one-hot encoded values from 'core_presentation'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Dummy gender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 6: Declare a variable and store the dummified/one-hot encoded values from 'gender'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 7: Dummify region"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 7: Declare a variable and store the dummified/one-hot encoded values from 'region'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8: Dummify imd_band"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 8: Declare a variable and store the dummified/one-hot encoded values from 'imd_bank'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 9: Dummify age_band"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 9: Declare a variable and store the dummified/one-hot encoded values from 'age_band'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 10: Dummify disability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 10: Declare a variable and store the dummified/one-hot encoded values from 'disability'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 11: Concatenate all of the dummies into a single DataFrame\n",
    "Now that we're done dummifying the categorical columns, let's concatenate them into a single horizontally long DataFrame.\n",
    "\n",
    "![ConcatenatedDummies.png](https://uplevelsg.s3-ap-southeast-1.amazonaws.com/ProjectLearningAnalytics/ConcatenatedDummies.png)\n",
    "\n",
    "Sanity check - if you dummified properly, i.e. remove first column from the dummies, you should have:\n",
    "1. 28,785 rows\n",
    "2. 35 columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 11: Concatenate all of your dummy DataFrame into a single DataFrame"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 12: Replace values in final_result with binaries\n",
    "Let's replace the 'final_result' values with 1 and 0 because we just have two outcomes in a programme - fail or pass. \n",
    "\n",
    "As such, we shall bin our categorical results like this:\n",
    "1. Pass/Distinction (1)\n",
    "2. Withdrawn/Fail (0)\n",
    "\n",
    "![TransformDependentVariable.png](https://uplevelsg.s3-ap-southeast-1.amazonaws.com/ProjectLearningAnalytics/TransformDependentVariable.png)\n",
    "\n",
    "There are a few ways to do it, e.g., mapping, replacing the column with a new list, etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 12: Replace the categorical values in final_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 13: Extract numerical columns\n",
    "Before we do anything else, let's take only the necessary data from the original DataFrame.\n",
    "\n",
    "We only need these columns for now:\n",
    "1. num_of_prev_attempts\n",
    "2. studied_credits\n",
    "3. sum_click\n",
    "4. mean\n",
    "5. max\n",
    "6. min\n",
    "7. final_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 13: Subset the original DataFrame"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 14: Concatenate the DataFrame from Step 13 to the DataFrame from Step 11\n",
    "Now that we have all we need, let's combine our DataFrame with the rests of the dummies.\n",
    "\n",
    "![FinalDataFrame.png](https://uplevelsg.s3-ap-southeast-1.amazonaws.com/ProjectLearningAnalytics/FinalDataFrame.png)\n",
    "\n",
    "Sanity check:\n",
    "1. 28,785 rows \n",
    "2. 42 columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 14: Concatenate everything from Steps 13 and Steps 11"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 15: Export the final DataFrame as a CSV\n",
    "Now you're finally done and can proceed to modelling after getting the final DataFrame form. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 15: Export the final DataFrame as a CSV"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}