Project Learning Analytics (Part V).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "Last but not least, with all the pieces in place, we can finally implement machine learning to predict student dropouts!\n",
    "\n",
    "## Machine Learning\n",
    "\n",
    "![MachineLearningProcess.png](https://uplevelsg.s3-ap-southeast-1.amazonaws.com/ProjectLearningAnalytics/MachineLearningProcess.png)\n",
    "\n",
    "We put this section on all of the projects so bear with us if you've seen this before. \n",
    "\n",
    "Generally, the machine learning process has five parts:\n",
    "1. <strong>Split your data into train and test set</strong>\n",
    "2. <strong>Model creation</strong>\n",
    "<br>\n",
    "Import your models from sklearn and instantiate them (assign model object to a variable)\n",
    "3. <strong>Model fitting</strong>\n",
    "<br>\n",
    "Fit your training data into the model and train train train\n",
    "4. <strong>Model prediction</strong>\n",
    "<br>\n",
    "Make a set of predictions using your test data, and\n",
    "5. <strong>Model assessment</strong>\n",
    "<br>\n",
    "Compare your predictions with ground truth in test data\n",
    "\n",
    "Highly recommended readings:\n",
    "1. [Important] https://scipy-lectures.org/packages/scikit-learn/index.html\n",
    "2. https://machinelearningmastery.com/a-gentle-introduction-to-scikit-learn-a-python-machine-learning-library/\n",
    "3. https://scikit-learn.org/stable/tutorial/basic/tutorial.html\n",
    "\n",
    "### Step 1: Import your libraries\n",
    "We will be using models from sklearn - a popular machine learning library. However, we won't import everything from sklearn and take just what we need. \n",
    "\n",
    "We'll need to import plotting libraries to plot our predictions against the ground truth (test data). \n",
    "\n",
    "Import the following:\n",
    "1. pandas\n",
    "2. matplotlib.pyplot as plt\n",
    "3. seaborn\n",
    "4. numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 1: Import the libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Read the CSV from Part IV Step \n",
    "In this section, we will read the CSV that we prepared from Part IV. \n",
    "\n",
    "Sanity check:\n",
    "1. 28,875 rows\n",
    "2. 42 columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 2: Read your CSV "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Prepare your independent and dependent variables\n",
    "Before we jump into training, we will split our DataFrame into two parts - indepedent and dependent variables. \n",
    "\n",
    "We'll be preparing a DataFrame containing our indepedent variables, and a separate list containing the \"final_result\".\n",
    "\n",
    "1. Declare a variable, and assign your independent variables to it, i.e. drop \"final_result\" from the DataFrame\n",
    "2. Declare a variable, and assign only values from \"final_result\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 3: Prepare your indepedent and dependent variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Import machine learning libraries\n",
    "Time to import other libraries. We hope you've taken a look at the two articles at the start of this notebook because it'll be useful. \n",
    "\n",
    "Import the following libraries and methods:\n",
    "1. train_test_split - sklearn.model_selection\n",
    "2. DummyClassifier - sklearn.dummy\n",
    "3. LogisticRegression - sklearn.linear_model\n",
    "4. DecisionTreeClassifier - sklearn.tree\n",
    "5. RandomForestClassifier - sklearn.ensemble\n",
    "6. GradientBoostClassifier - sklearn.ensemble\n",
    "7. f1_score - sklearn.metrics\n",
    "8. confusion_matrix - sklearn.metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 4: Import the machine learning libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Split your indepedent and dependent variables into train and test sets\n",
    "We'll be using a 80/20 split for train and test set respectively, using the train_test_split function. We will also stratify by y so that the proportions for our dependent variables are even."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 5: Split your data into train and test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Train your machine learning model\n",
    "Once you've split your data, machine learning begins. \n",
    "\n",
    "This is what you'll need to do:\n",
    "1. Start with a model\n",
    "2. Declare a variable, and store your model in it (don't forget to use brackets)\n",
    "3. Fit your training data into the instantiated model\n",
    "4. Declare a variable that contains predictions from the model you just trained, using the train dataset (X_test)\n",
    "5. Compare the prediction with the actual result (y_test) with f1_score and confusion matrix\n",
    "\n",
    "We will start with DummyClassifier to establish a baseline for your predictions. \n",
    "\n",
    "Also, the recommended readings will be very helpful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 6a: Declare a variable to store the dummy model\n",
    "\n",
    "# Step 6b: Fit your train dataset\n",
    "\n",
    "# Step 6c: Declare a variable and store your predictions that you make with your model using X test data\n",
    "\n",
    "# Step 6d: Print the f1_score between y_test and your prediction\n",
    "\n",
    "# Step 6e: Print the confusion matrix between y_test and your prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 7: Repeat Step 6 with LogisticRegression\n",
    "The performance for DummyClassifier is bad, and expectedly so because it's what you'd see if you randomly guess. \n",
    "\n",
    "Now, let's use other models to perform the classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 7a: Declare a variable to store the model\n",
    "\n",
    "# Step 7b: Fit your train dataset\n",
    "\n",
    "# Step 7c: Declare a variable and store your predictions that you make with your model using X test data\n",
    "\n",
    "# Step 7d: Print the f1_score between y_test and your prediction\n",
    "\n",
    "# Step 7e: Print the confusion matrix between y_test and your prediction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8: Repeat Step 6 with DecisionTreeClassifier\n",
    "Using LogisticRegression is not bad at all - we see a f1_score jump from 0.4+ to 0.7-0.8.\n",
    "\n",
    "What happens when we use other models? Let's find out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 8a: Declare a variable to store the model\n",
    "\n",
    "# Step 8b: Fit your train dataset\n",
    "\n",
    "# Step 8c: Declare a variable and store your predictions that you make with your model using X test data\n",
    "\n",
    "# Step 8d: Print the f1_score between y_test and your prediction\n",
    "\n",
    "# Step 8e: Print the confusion matrix between y_test and your prediction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 9: Repeat Step 6 with RandomForestClassifier\n",
    "Not bad, not bad - we should expect either slight improvements or on par performance. \n",
    "\n",
    "Next up, we will use a RandomForestClassifier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 9a: Declare a variable to store the model\n",
    "\n",
    "# Step 9b: Fit your train dataset\n",
    "\n",
    "# Step 9c: Declare a variable and store your predictions that you make with your model using X test data\n",
    "\n",
    "# Step 9d: Print the f1_score between y_test and your prediction\n",
    "\n",
    "# Step 9e: Print the confusion matrix between y_test and your prediction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Important] Step 10: Repeat Step 6 with GradientBoostClassifier\n",
    "The performance will improve once again!\n",
    "\n",
    "There are many models out there, and different models work differently depending on the dataset. \n",
    "\n",
    "Last one - we'll give it a GradientBoostClassifier a try."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 10a: Declare a variable to store the model\n",
    "\n",
    "# Step 10b: Fit your train dataset\n",
    "\n",
    "# Step 10c: Declare a variable and store your predictions that you make with your model using X test data\n",
    "\n",
    "# Step 10d: Print the f1_score between y_test and your prediction\n",
    "\n",
    "# Step 10e: Print the confusion matrix between y_test and your prediction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 11: Get feature_importances of your Step 10 model\n",
    "Now we have a great mode, we can then take a closer look at the feature importance so that we can intuitively identify which features are important not just for the model but the business context as well.\n",
    "\n",
    "We can use .feature_importances_ attribute of our models to get a list containing the importances of each feature. \n",
    "\n",
    "However, this list contains only numbers so we can create a DataFrame out of the list of features.\n",
    "\n",
    "![FeatureImportances.png](https://uplevelsg.s3-ap-southeast-1.amazonaws.com/ProjectLearningAnalytics/FeatureImportances.png)\n",
    "\n",
    "Your results will differ from ours so don't be alarmed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 11: Create a DataFrame containing feature importances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter tuning\n",
    "The GradientBoostClassifier model seems to perform the best so far. \n",
    "\n",
    "Is that the end? More or less.\n",
    "\n",
    "But can we improve it even further? Quite possibly, yes - with hyperparameter tuning. \n",
    "\n",
    "In this section, we will perform hyperparameter tuning to get the best parameters and improve our GradientBoostClassifier. \n",
    "\n",
    "### Step 12: Import GridSearchCV\n",
    "We're tuning our model, so let's import:\n",
    "1. GridSearchCV from sklearn.model_selection\n",
    "\n",
    "Useful reading: https://scikit-learn.org/stable/modules/grid_search.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 12: Import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 13: Define a parameter grid\n",
    "What is a parameter grid? A parameter grid contains a set of parameters that you'd like to explore, along with the values. \n",
    "\n",
    "For our own GridSearchCV, let's explore the following:\n",
    "1. n_estimators - 100, 250, 500, 750, 1000, 1250, 1500, 1750\n",
    "2. max_depth - 2, 3, 4, 5, 6, 7\n",
    "3. learning_rate - 0.15, 0.1, 0.05, 0.01, 0.005, 0.001\n",
    "\n",
    "You can choose any combination of the above, though take note that if you use all three sets of parameters to explore, you will need to run this all night."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 13: Define your parameter grid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 14: Create a GridSearchCV object\n",
    "Declare a variable containing your GridSearchCV with the following parameters:\n",
    "1. estimator - GradientBoostingClassifier()\n",
    "2. param_grid - your Step 12 variable\n",
    "3. scoring - 'accuracy'\n",
    "4. n_jobs - 4*\n",
    "5. cv - 5\n",
    "\n",
    "*this will speed your run up slightly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 14: Declare a GridSearchCV with the parameters specified"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Caution] Step 15: Fit your GridSearchCV object with training data\n",
    "We put caution here because of the amount of time it will take for your grid search. \n",
    "\n",
    "Allocate a fair amount of time, i.e. a few hours, for the grid search to finish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 15: Fit your X_train and y_train into your GridSearchCV object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 16: Get the best parameters from the grid search\n",
    "Once you're done, get the best parameters from your GridSearchCV object using the .best_params_ attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 16: Get the best parameters from the grid search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 17: Use your model to make predictions\n",
    "Now that you've identified the best parameters for your model, go ahead and use the GridSearchCV object like a model.\n",
    "\n",
    "Repeat Step 6 and see how much your tuned model has improved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 17a: Declare a variable and store your predictions that you make with your GridSearchCV using X test data\n",
    "\n",
    "# Step 17b: Print the f1_score between y_test and your prediction\n",
    "\n",
    "# Step 17c: Print the confusion matrix between y_test and your prediction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With tuning, you can see decent improvement in your model performance which is great! \n",
    "\n",
    "### Step 18: Get feature_importances of your best performing tuned model\n",
    "Now that we're done with our hyperparameter tuning, we can take another look at the best performing tuned model and assess the feature importances.\n",
    "\n",
    "Unlike Step 11, you'll have to get the best performing model/estimator first before you can retrieve the feature importances. \n",
    "\n",
    "<strong>Hint: Google \"feature importance gridsearchcv\"</strong>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Step 18: Get the feature importance from the best estimator in your GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The end\n",
    "And that's the end! To recap, you've:\n",
    "1. Retrieve learning analytics data from Open University\n",
    "2. Combined disparate pieces of data into a more coherent and complete dataset\n",
    "3. Explored the data through visualization\n",
    "4. Engineered new features for machine learning modelling\n",
    "5. Trained a machine learning model to predict student pass/fail\n",
    "6. Performed hyperparameter tuning to improve the best performing model even more\n",
    "\n",
    "Go on, give yourself a pat on the back. We hope this project series has give you more confidence in coding and machine learning. \n",
    "\n",
    "You have successfully implemented machine learning in predicting student pass/fail outcome for their courses. There's always more room for improvement, such as getting more data from Open University and updating the model.  \n",
    "\n",
    "That is the fate of a data scientist, to pursue better models that can help model the world out there.  \n",
    "\n",
    "Whatever you learn here is but a tip of the iceberg, and launchpad for bigger and better things to come. Come join us in our Telegram community over at https://bit.ly/UpLevelSG and our Facebook page at https://fb.com/UpLevelSG\n",
    "\n",
    "Whatever you learn here is but a tip of the iceberg, and launchpad for bigger and better things to come."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}