GFE_Classification_Training_Testing_SVM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "metallic-iraqi",
   "metadata": {},
   "source": [
    "# GFE Classification: Training & Testing Using Support Vector Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "explicit-panama",
   "metadata": {},
   "source": [
    "#####  This notebook aims to use Support Vector Machines (SVM) to classify facial capture footage as a specified emotion. Data exploration, training, and modelling will all be discussed below and mitigations will be provided. \n",
    "###### (i) Train/test SVM on GFE data on a single emotion and evaluate performance measures\n",
    "###### (ii) Repeat test on a different facial expression\n",
    "###### (iii) Invert the roles of the Users\n",
    "###### (iiii) Use Dimensionality Reduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "blank-passenger",
   "metadata": {},
   "source": [
    "# Part A: Training \n",
    "## Train \"Negative\" Emotion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "limited-murray",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn import svm\n",
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import train_test_split,GridSearchCV,KFold, cross_val_score\n",
    "from sklearn.metrics import accuracy_score, precision_score,plot_roc_curve, recall_score\n",
    "import plotly.express as px\n",
    "import plotly.graph_objects as go\n",
    "from plotly.subplots import make_subplots\n",
    "from collections import Counter\n",
    "from sklearn.decomposition import PCA\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "phantom-bicycle",
   "metadata": {},
   "outputs": [],
   "source": [
    "# select emotion\n",
    "emotion = \"negative\" \n",
    "\n",
    "# read in data file\n",
    "df_neg = pd.read_csv(f\"grammatical_facial_expression/a_{emotion}_datapoints.txt\",delimiter = \" \",)\n",
    "df_neg_target = pd.read_csv(f\"grammatical_facial_expression/a_{emotion}_targets.txt\",delimiter = \" \",header=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "charged-outreach",
   "metadata": {},
   "source": [
    "##### Data Exploration "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aggressive-hebrew",
   "metadata": {},
   "outputs": [],
   "source": [
    "# combine both dataframes using the target dataset\n",
    "df_neg['target'] = df_neg_target\n",
    "df_neg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vocal-liquid",
   "metadata": {},
   "outputs": [],
   "source": [
    "# summary statistics (exclude first/last columns)\n",
    "df_neg.iloc[:,1:-1].stack().describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "consolidated-breakfast",
   "metadata": {},
   "source": [
    "Statistics indicate that the data has a range from 0 to 1585 with a standard deviation of 471, which suggests a large distribution of data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hourly-guyana",
   "metadata": {},
   "outputs": [],
   "source": [
    "# collect x,y, & z coordinates as separate dataframes\n",
    "xs = df_neg[df_neg.columns[1::3]]\n",
    "ys = df_neg[df_neg.columns[2::3]]\n",
    "zs = df_neg[df_neg.columns[3::3]]\n",
    "\n",
    "# remove target col\n",
    "xs = xs.drop([\"target\"],axis=1)\n",
    "\n",
    "# array of 3 coordinate axes\n",
    "df_neg_coord = np.array((xs,ys,zs))\n",
    "\n",
    "print(xs.stack().describe())\n",
    "print(ys.stack().describe())\n",
    "print(zs.stack().describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "necessary-arrival",
   "metadata": {},
   "source": [
    "A look at the coordinate axes indicates significant differences between X/Y and Z coordinates based on descriptive statistics. Normalization and Standardization techniques will be applied to determine if there are significant differences. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "noticed-verse",
   "metadata": {},
   "outputs": [],
   "source": [
    "# collect x,y, & z coordinates as separate dataframes\n",
    "xs = df_neg[df_neg.columns[1::3]]\n",
    "ys = df_neg[df_neg.columns[2::3]]\n",
    "zs = df_neg[df_neg.columns[3::3]]\n",
    "\n",
    "# remove target col\n",
    "xs = xs.drop(['target'],axis=1)\n",
    "\n",
    "# Look at \n",
    "fig = make_subplots(rows=1, cols=3,subplot_titles=(\"X-Axis Values Hist.\",\"Y-Axis Values Hist.\", \"Z-Axis Values Hist.\"))\n",
    "fig.add_trace(go.Histogram(x=xs.values.ravel(),name=\"x-axis\"),\n",
    "    row=1, col=1\n",
    ")\n",
    "fig.add_trace(go.Histogram(x=ys.values.ravel(),name=\"y-axis\"),\n",
    "    row=1, col=2\n",
    ")\n",
    "fig.add_trace(go.Histogram(x=zs.values.ravel(),name=\"z-axis\"),\n",
    "    row=1, col=3\n",
    ")\n",
    "fig.update_layout(title_text=\"GFE (Negative) Data Distribution\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "finished-climb",
   "metadata": {},
   "source": [
    "The data distributions above indicate varying ranges between all three axes. Because of the varying scales, standardization would be an optimal preprocessing technique to apply to the data to ensure more accurate results.\n",
    "\n",
    "This analysis will therefore test the modelling effects with and without scaling the data, with the former likely to produce superior results.\n",
    "\n",
    "###### Split Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "actual-trading",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split train/test and validation\n",
    "X_neg = df_neg.iloc[:,1:-1]\n",
    "y_neg = df_neg.iloc[:,-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wicked-flexibility",
   "metadata": {},
   "outputs": [],
   "source": [
    "# scale data\n",
    "scaler = preprocessing.StandardScaler()\n",
    "X_neg_scaled = scaler.fit_transform(X_neg)\n",
    "\n",
    "scaler_norm = preprocessing.MinMaxScaler()\n",
    "X_neg_norm = scaler_norm.fit_transform(X_neg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "robust-basket",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at Distributions before and after standardizing and normalizing\n",
    "fig = make_subplots(rows=1, cols=3,subplot_titles=(\"Raw Data\", \"After Normalized\",\"After Standardized\"))\n",
    "fig.add_trace(go.Violin(y=X_neg.unstack(),name=\"Raw Data\"),\n",
    "    row=1, col=1\n",
    ")\n",
    "fig.add_trace(go.Violin(y=pd.DataFrame(X_neg_norm).unstack(),name=\"After Normalized\"),\n",
    "    row=1, col=2\n",
    ")\n",
    "fig.add_trace(go.Violin(y=pd.DataFrame(X_neg_scaled).unstack(),name=\"After Standardized\"),\n",
    "    row=1, col=3\n",
    ")\n",
    "fig.update_layout(title_text=\"GFE (Negative) Data Distributions: User A\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exceptional-algebra",
   "metadata": {},
   "source": [
    "The above figures indicate that standardization removes the large varying distributions in the raw data that were observed before. Normalization removes the large contrasts between the distribution peaks, however the standardization of the data scaled the data best, removing large deviations whilst keeping the distribution normal.\n",
    "\n",
    "### Support Vector Machine Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "smooth-exposure",
   "metadata": {},
   "source": [
    "In order to optimize parameter estimates, 5-fold cross validation will be used to compare accuracies to optimize the kernel type, regularization parameter, and for the polynomial kernel the degree of the function.\n",
    "\n",
    "Kernel Types:  linear, polynomial, radial   \n",
    "Polynomial Degrees: 2-9    \n",
    "C-Value: 0.1, 1, 10   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "polyphonic-number",
   "metadata": {},
   "source": [
    "##### Un-Standardized Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "collect-movie",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear','poly','rbf']\n",
    "degree = [2,3,4,5,6,7,8,9]\n",
    "C = [0.1,1,10]\n",
    "params = dict(kernel=kernel,degree=degree, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params)\n",
    "best = grid.fit(X_neg,y_neg)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "south-board",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Best 5-Fold CrossValidation Estimates for Un-Standardized Data\")\n",
    "print(\"Best Kernel:\", best.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best Degree:\", best.best_estimator_.get_params()['degree'])\n",
    "print(\"Best C:\", best.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "apparent-movement",
   "metadata": {},
   "source": [
    "##### Standardized Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "mexican-drama",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear','poly','rbf']\n",
    "degree = [2,3,4,5,6,7,8,9]\n",
    "C = [0.1,1,10,100]\n",
    "\n",
    "params = dict(kernel=kernel,degree=degree, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params)\n",
    "best_scaled = grid.fit(X_neg_scaled,y_neg)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "radio-belief",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Best 5-Fold CrossValidation Estimates for Standardized Data\")\n",
    "print(\"Best Kernel:\", best_scaled.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best Degree:\", best_scaled.best_estimator_.get_params()['degree'])\n",
    "print(\"Best C:\", best_scaled.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best_scaled.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "small-wagner",
   "metadata": {},
   "source": [
    "###### Normalized Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dedicated-joining",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear','poly','rbf']\n",
    "degree = [2,3,4,5,6,7,8,9]\n",
    "C = [0.1,1,10,100]\n",
    "params = dict(kernel=kernel,degree=degree, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params)\n",
    "best_norm = grid.fit(X_neg_norm,y_neg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "nutritional-evanescence",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Best 5-Fold CrossValidation Estimates for Normalized Data\")\n",
    "print(\"Best Kernel:\", best_norm.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best Degree:\", best_norm.best_estimator_.get_params()['degree'])\n",
    "print(\"Best C:\", best_norm.best_estimator_.get_params()['C'])\n",
    "print(\"Best Gamma:\", best_norm.best_estimator_.get_params()['gamma'])\n",
    "print(\"K-Fold Accuracy:\", best_norm.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acceptable-numbers",
   "metadata": {},
   "outputs": [],
   "source": [
    "# select best model\n",
    "optimized_clf_neg_usera = best_scaled.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "seeing-lodge",
   "metadata": {},
   "source": [
    "Raw Data Average Accuracy: 88.7% \n",
    "\n",
    "Standardized Average Accuracy: 90.5% \n",
    "\n",
    "Normalized Average Accuracy: 90.6% \n",
    "\n",
    "The above findings indicate that both Normalized and Standardized data perform better than the raw data. This can be attributed to the varying scales between the coordinate axes, with the z-axis (mm) varying from the x and y-axes coordinate systems which was evident in the exploratory analysis above.\n",
    "\n",
    "An observation of concern would be for the normalized dataset, using the \"polynomial\" kernel with a degree = 4. A 4th degree polynomial may fit this training data best, however it may pose a higher risk of overfitting in comparison to the \"linear\" kernel used by the standardized data cross validation. \n",
    "\n",
    "Because standardization improved the distribution of the data prior to cross validation, and nearly identical K-fold cross validation accuracy between normalization and standardization, when performing testing of this dataset standardization will be the preferred technique. \n",
    "\n",
    "Best 5-Fold CrossValidation Estimates for Standardized Data    \n",
    "Best Kernel: linear   \n",
    "Best Degree: 3   \n",
    "Best C: 0.1      \n",
    "K-Fold Accuracy: 0.9047698412698413    \n",
    "\n",
    "## Repeat Training on \"Emphasis\" User A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hired-passenger",
   "metadata": {},
   "outputs": [],
   "source": [
    "# select emotion\n",
    "emotion2 = \"emphasis\" \n",
    "\n",
    "# read in data file\n",
    "df_emp = pd.read_csv(f\"grammatical_facial_expression/a_{emotion2}_datapoints.txt\",delimiter = \" \",)\n",
    "df_emp_target = pd.read_csv(f\"grammatical_facial_expression/a_{emotion2}_targets.txt\",delimiter = \" \",header=None)\n",
    "\n",
    "# combine both dataframes using the target dataset\n",
    "df_emp['target'] = df_emp_target\n",
    "df_emp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "spoken-huntington",
   "metadata": {},
   "outputs": [],
   "source": [
    "# summary statistics (exclude first/last columns)\n",
    "# collect x,y, & z coordinates as separate dataframes\n",
    "xs = df_emp[df_emp.columns[1::3]]\n",
    "ys = df_emp[df_emp.columns[2::3]]\n",
    "zs = df_emp[df_emp.columns[3::3]]\n",
    "\n",
    "# remove target col\n",
    "xs = xs.drop([\"target\"],axis=1)\n",
    "\n",
    "# array of 3 coordinate axes\n",
    "df_neg_coord = np.array((xs,ys,zs))\n",
    "\n",
    "print(xs.stack().describe())\n",
    "print(ys.stack().describe())\n",
    "print(zs.stack().describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "catholic-equation",
   "metadata": {},
   "source": [
    "The data distributions are similar to the \"Negative\" emotions from above. \n",
    "\n",
    "##### Split Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "stock-baltimore",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split train/test and validation\n",
    "X_emp = df_emp.iloc[:,1:-1]\n",
    "y_emp = df_emp.iloc[:,-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "australian-shareware",
   "metadata": {},
   "outputs": [],
   "source": [
    "# scale data\n",
    "scaler = preprocessing.StandardScaler()\n",
    "X_emp_scaled = scaler.fit_transform(X_emp)\n",
    "\n",
    "scaler_norm = preprocessing.MinMaxScaler()\n",
    "X_emp_norm = scaler_norm.fit_transform(X_emp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "under-tuesday",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at Distributions before and after standardizing and normalizing\n",
    "fig = make_subplots(rows=1, cols=3,subplot_titles=(\"Raw Data\",\"After Normalized\",\"After Standardized\"))\n",
    "fig.add_trace(go.Violin(y=X_emp.unstack(),name=\"Raw Data\"),\n",
    "    row=1, col=1\n",
    ")\n",
    "fig.add_trace(go.Violin(y=pd.DataFrame(X_emp_norm).unstack(),name=\"After Normalized\"),\n",
    "    row=1, col=2\n",
    ")\n",
    "fig.add_trace(go.Violin(y=pd.DataFrame(X_emp_scaled).unstack(),name=\"After Standardized\"),\n",
    "    row=1, col=3\n",
    ")\n",
    "fig.update_layout(title_text=\"GFE (Emphasis) Data Distributions: User A\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "metric-cameroon",
   "metadata": {},
   "source": [
    "The above plots indicate that standardizing the data removes the unbalanced distributions in the data the best, which is identical to the \"negative\" emotion results as well. \n",
    "\n",
    "### Support Vector Machine Classification\n",
    "\n",
    "##### Un-Standardized Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "martial-destination",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear','poly','rbf']\n",
    "degree = [2,3,4,5,6,7,8,9,10]\n",
    "C = [0.1,1,10,100]\n",
    "\n",
    "params = dict(kernel=kernel,degree=degree, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params,n_jobs=4)\n",
    "best_emp = grid.fit(X_emp,y_emp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "atmospheric-groove",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Best 5-Fold CrossValidation Estimates for Un-Standardized Data\")\n",
    "print(\"Best Kernel:\", best_emp.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best Degree:\", best_emp.best_estimator_.get_params()['degree'])\n",
    "print(\"Best C:\", best_emp.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best_emp.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "convinced-process",
   "metadata": {},
   "source": [
    "##### Standardized Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "convenient-regular",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear','poly','rbf']\n",
    "degree = [2,3,4,5,6,7,8,9]\n",
    "C = [0.1,1,10]\n",
    "params = dict(kernel=kernel,degree=degree, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params,n_jobs = 4)\n",
    "best_emp_scaled = grid.fit(X_emp_scaled,y_emp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wanted-marriage",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Best 5-Fold CrossValidation Estimates for Standardized Data\")\n",
    "print(\"Best Kernel:\", best_emp_scaled.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best Degree:\", best_emp_scaled.best_estimator_.get_params()['degree'])\n",
    "print(\"Best C:\", best_emp_scaled.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best_emp_scaled.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sought-custody",
   "metadata": {},
   "source": [
    "##### Normalized Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "separate-inventory",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear','poly','rbf']\n",
    "degree = [2,3,4,5,6,7,8,9]\n",
    "C = [0.1,1,10]\n",
    "params = dict(kernel=kernel,degree=degree, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params)\n",
    "best_emp_norm = grid.fit(X_emp_norm,y_emp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "secondary-advocacy",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Best 5-Fold CrossValidation Estimates for Normalized Data\")\n",
    "print(\"Best Kernel:\", best_emp_norm.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best Degree:\", best_emp_norm.best_estimator_.get_params()['degree'])\n",
    "print(\"Best C:\", best_emp_norm.best_estimator_.get_params()['C'])\n",
    "print(\"Best Gamma:\", best_emp_norm.best_estimator_.get_params()['gamma'])\n",
    "print(\"K-Fold Accuracy:\", best_emp_norm.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "potential-diploma",
   "metadata": {},
   "outputs": [],
   "source": [
    "# select best model\n",
    "optimized_clf_emp_usera = best_emp_scaled.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caroline-administrator",
   "metadata": {},
   "source": [
    "Raw Data Average Accuracy: 97.0% \n",
    "\n",
    "Standardized Average Accuracy: 97.8% \n",
    "\n",
    "Normalized Average Accuracy: 97.9% \n",
    "\n",
    "For \"Emphasis\" emotions, the average accuracy varies slightly between cross validated models using different scaling techniques. Normalization and Standardization once again are the top two results. Standardization still exhibits the best transformation of the distribution of the data, and also based on it's nearly identical accuracy to normalization it will be chosen as the preferred technique of scaling the data.\n",
    " \n",
    "Best 5-Fold CrossValidation Estimates for Standardized Data  \n",
    "Best Kernel: linear   \n",
    "Best Degree: 3   \n",
    "Best C: 1        \n",
    "Best Gamma: scale     \n",
    "K-Fold Accuracy: 0.9778927300457549   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "attended-covering",
   "metadata": {},
   "source": [
    "## Implementation of Classifier from Scratch: K-Nearest Neighbor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "floating-trinity",
   "metadata": {},
   "source": [
    "A K-Nearest Neighbor Classifier will be implemented below using a class of functions. In this implementation Euclidean, Hamming, and Manhattan Distances will be compared to determine the superior distance function. \n",
    "\n",
    "This classification will only be using standardized data, since the previous classification indicated that standardized data produced highly accurate results and transformed the distribution of the data best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "upper-gardening",
   "metadata": {},
   "outputs": [],
   "source": [
    "class knn:\n",
    "    '''This is the implemented classifier for K-Nearest Neighbor Classification. Default value for k is 3.\n",
    "    '''\n",
    "    def __init__(self, k=3, distance_func=\"euclidean\"):\n",
    "        self.k = k\n",
    "        self.distance_func = distance_func\n",
    "        \n",
    "    def euclidean_distance(self, row1, row2):\n",
    "        return np.sqrt(np.sum((row1 - row2)**2))\n",
    "    \n",
    "    def hamming_distance(self, row1, row2):\n",
    "        return sum(abs(e1 - e2) for e1, e2 in zip(row1, row2)) / len(row1)\n",
    "    \n",
    "    def manhattan_distance(self, row1, row2):\n",
    "        return sum(abs(e1-e2) for e1, e2 in zip(row1,row2))\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        self.X_train = X\n",
    "        self.y_train = y\n",
    "\n",
    "    def predict(self, X):\n",
    "        # loop thru each row of test data and calculate the nearest neighbor\n",
    "        y_pred = []\n",
    "        for x in X:\n",
    "            y_pred.append(self.nearest_neighbor(x))\n",
    "            \n",
    "        return np.array(y_pred)\n",
    "\n",
    "    def nearest_neighbor(self, x):\n",
    "        # use the euclidean distance function above to calculate distances between rows of data\n",
    "        distances = []\n",
    "        for x_train in self.X_train:\n",
    "            if self.distance_func == 'euclidean':\n",
    "                distances.append(self.euclidean_distance(x,x_train))\n",
    "            elif self.distance_func == 'hamming':\n",
    "                distances.append(self.hamming_distance(x,x_train))\n",
    "            else: \n",
    "                distances.append(self.manhattan_distance(x,x_train))\n",
    "                \n",
    "        # sort by minimum distance and return the index\n",
    "        index = np.argsort(distances)[:self.k]\n",
    "        \n",
    "        # np.take uses the index to return the actual label\n",
    "        k_neighbor_labels = np.take(self.y_train,index)   \n",
    "        \n",
    "        # the Counter function returns the most common label\n",
    "        label = Counter(k_neighbor_labels).most_common(1)\n",
    "        \n",
    "        return label[0][0]\n",
    "    \n",
    "    def get_params(self, deep=True):\n",
    "        # require this setting in order to be compatible with GridSearchCV\n",
    "        return {\"k\": self.k, \n",
    "                \"distance_func\": self.distance_func}\n",
    "    \n",
    "    def set_params(self, **parameters):\n",
    "        for parameter, value in parameters.items():\n",
    "            setattr(self, parameter, value)\n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "preceding-spencer",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = knn()\n",
    "\n",
    "# optimization paramters\n",
    "distance_func = ['euclidean','manning','manhattan']\n",
    "k = [5,7,9,11,13,15,17,19,21]\n",
    "\n",
    "params = dict(k=k,distance_func=distance_func)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params, scoring=\"accuracy\")\n",
    "best_knn_scaled = grid.fit(X_neg_scaled,y_neg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "metropolitan-point",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Best 5-Fold CrossValidation Estimates for Standardized Data\")\n",
    "print(\"Best Kernel:\", best_knn_scaled.best_estimator_.get_params()['k'])\n",
    "print(\"Best Degree:\", best_knn_scaled.best_estimator_.get_params()['distance_func'])\n",
    "print(\"K-Fold Accuracy:\", best_knn_scaled.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mexican-investigation",
   "metadata": {},
   "source": [
    "Best 5-Fold CrossValidation Estimates for Standardized Data   \n",
    "Best Kernel: 15  \n",
    "Best Degree: euclidean   \n",
    "K-Fold Accuracy: 0.8353531746031747   \n",
    "\n",
    "Using the KNN implementation from scratch on \"negative\" emotion data, the results produced a model that was 83.5% accurate after 5-Fold Cross Validation. Euclidean distance was the preferred distance function, and the optimized number for k was 15. This is similar to using the \"rule of thumb\" method for determining k which involves taking the square-root of the number of features and dividing by 2, which yielded a k value of 13.\n",
    "\n",
    "Overall this classification method is not as accurate as SVM, and required nearly 8 hours of computing time to optimize the parameters. Therefore, the \"implemented from scratch\" KNN classifier will not be abandoned in the continuation of this analysis due to its inefficiency and lack of predictive capability.\n",
    "\n",
    "# Part B: Testing on User B\n",
    "\n",
    "## Test on \"Negative\" Emotion User B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "comparative-bosnia",
   "metadata": {},
   "outputs": [],
   "source": [
    "# select emotion\n",
    "emotion = \"negative\" \n",
    "\n",
    "# read in data file\n",
    "df_neg_userb = pd.read_csv(f\"grammatical_facial_expression/b_{emotion}_datapoints.txt\",delimiter = \" \",)\n",
    "df_neg_target_userb = pd.read_csv(f\"grammatical_facial_expression/b_{emotion}_targets.txt\",delimiter = \" \",header=None)\n",
    "\n",
    "# combine both dataframes using the target dataset\n",
    "df_neg_userb['target'] = df_neg_target_userb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interstate-hartford",
   "metadata": {},
   "source": [
    "##### Split Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fifth-season",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split train/test and validation\n",
    "X_neg_userb = df_neg_userb.iloc[:,1:-1]\n",
    "y_neg_userb = df_neg_userb.iloc[:,-1]\n",
    "\n",
    "# scale data\n",
    "scaler = preprocessing.StandardScaler()\n",
    "X_neg_scaled_userb = scaler.fit_transform(X_neg_userb)\n",
    "\n",
    "scaler = preprocessing.MinMaxScaler()\n",
    "X_neg_norm_userb = scaler.fit_transform(X_neg_userb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "psychological-observation",
   "metadata": {},
   "source": [
    "##### Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "instrumental-google",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions on test data using optimized model\n",
    "y_pred = optimized_clf_neg_usera.predict(X_neg_scaled_userb)\n",
    "\n",
    "# calculate model accuracy\n",
    "acc = accuracy_score(y_neg_userb, y_pred)\n",
    "\n",
    "# calculate model precision\n",
    "prec = precision_score(y_neg_userb, y_pred)\n",
    "\n",
    "# calculate model recall\n",
    "recall = recall_score(y_neg_userb, y_pred)\n",
    "\n",
    "print(\"Model Accuracy:\", acc)\n",
    "print(\"Model Precision:\", prec)\n",
    "print(\"Model Recall:\", recall)\n",
    "\n",
    "# plot ROC Curve\n",
    "plot_roc_curve(optimized_clf_neg_usera,X_neg_scaled_userb,y_neg_userb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "massive-character",
   "metadata": {},
   "source": [
    "Model Accuracy: 0.5474083438685209  \n",
    "Model Precision: 0.4973753280839895  \n",
    "Model Recall: 0.5323033707865169  \n",
    "\n",
    "The above results for the testing on \"Negative\" emotion User B indicate that the model performed very poorly. Accuracy, precision, and model recall are all near 50% and the ROC curve indicates that the model is unable to correctly classify true positives. Overall the model is making predictions at random, similar to flipping a coin. \n",
    "\n",
    "Reasons for this could be explained by SVM using a value of 0.1 for C, which is a normal regularization parameter value. The smaller this parameter is, the more likely it is to missclassify points, and therefore make a more \"generalized\" model. Perhaps re-tuning the parameters using smaller values of C will allow the model to \"generalize\" more and perform better on the user B test set. \n",
    "\n",
    "Therefore, the SVM will be optimized again on user A using smaller values for C and re-tested on user B. Optimally, we would like to improve the testing accuracy (user B) without compromising the training accuracy (user A) in order to prevent the model from being biased.\n",
    "\n",
    "###### Re-Optimize parameters using smaller value of C and test again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "monetary-night",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear','poly','rbf']\n",
    "degree = [2,3,4,5,6,7,8,9]\n",
    "C = [0.1,1,10]\n",
    "params = dict(kernel=kernel,degree=degree, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params)\n",
    "best_scaled = grid.fit(X_neg_scaled,y_neg)\n",
    "\n",
    "print(\"Best 5-Fold CrossValidation Estimates for Smaller Values of C\")\n",
    "print(\"Best Kernel:\", best_scaled.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best Degree:\", best_scaled.best_estimator_.get_params()['degree'])\n",
    "print(\"Best C:\", best_scaled.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best_scaled.best_score_)\n",
    "\n",
    "# save model as optimized v2\n",
    "optimized_clf_neg_usera_v2 = best_scaled.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "clear-beverage",
   "metadata": {},
   "source": [
    "Best 5-Fold CrossValidation Estimates for Smaller Values of C   \n",
    "Best Kernel: linear   \n",
    "Best Degree: 3   \n",
    "Best C: 0.01   \n",
    "K-Fold Accuracy: 0.9012222222222223     \n",
    "\n",
    "After re-tuning using smaller values of C, accuracy was only compromised by a reduction of 0.3% (90.1%) and the optimal C value was 0.01. Smaller values of C were explored, however large reductions in accuracy were observed in excess of 5%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wired-alarm",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions on test data using optimized model\n",
    "y_pred = optimized_clf_neg_usera_v2.predict(X_neg_scaled_userb)\n",
    "\n",
    "# calculate model accuracy\n",
    "acc = accuracy_score(y_neg_userb, y_pred)\n",
    "\n",
    "# calculate model precision\n",
    "prec = precision_score(y_neg_userb, y_pred)\n",
    "\n",
    "# calculate model recall\n",
    "recall = recall_score(y_neg_userb, y_pred)\n",
    "\n",
    "print(\"Model Accuracy:\", acc)\n",
    "print(\"Model Precision:\", prec)\n",
    "print(\"Model Recall:\", recall)\n",
    "\n",
    "# plot ROC Curve\n",
    "plot_roc_curve(optimized_clf_neg_usera,X_neg_scaled_userb,y_neg_userb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "differential-spokesman",
   "metadata": {},
   "source": [
    "Model Accuracy: 0.5689001264222503   \n",
    "Model Precision: 0.5201612903225806   \n",
    "Model Recall: 0.5435393258426966   \n",
    "\n",
    "Overall little effect on the model was observed, with little improvements in accuracy and precision and no overall change in the ROC curve.\n",
    "\n",
    "## Test on \"Emphasis\" Emotion User B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "retained-general",
   "metadata": {},
   "outputs": [],
   "source": [
    "# select emotion\n",
    "emotion2 = \"emphasis\" \n",
    "\n",
    "# read in data file\n",
    "df_emp_userb = pd.read_csv(f\"grammatical_facial_expression/b_{emotion2}_datapoints.txt\",delimiter = \" \",)\n",
    "df_emp_target_userb = pd.read_csv(f\"grammatical_facial_expression/b_{emotion2}_targets.txt\",delimiter = \" \",header=None)\n",
    "\n",
    "# combine both dataframes using the target dataset\n",
    "df_emp_userb['target'] = df_emp_target_userb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceramic-jacket",
   "metadata": {},
   "source": [
    "###### Split Data/Scale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "located-berry",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split train/test and validation\n",
    "X_emp_userb = df_emp_userb.iloc[:,1:-1]\n",
    "y_emp_userb = df_emp_userb.iloc[:,-1]\n",
    "\n",
    "# scale data\n",
    "scaler = preprocessing.StandardScaler()\n",
    "X_emp_scaled_userb = scaler.fit_transform(X_emp_userb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "collect-depression",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions on test data using optimized model\n",
    "y_pred = optimized_clf_emp_usera.predict(X_emp_scaled_userb)\n",
    "\n",
    "# calculate model accuracy\n",
    "acc = accuracy_score(y_emp_userb, y_pred)\n",
    "\n",
    "# calculate model precision\n",
    "prec = precision_score(y_emp_userb, y_pred)\n",
    "\n",
    "# calculate model recall\n",
    "recall = recall_score(y_emp_userb, y_pred)\n",
    "\n",
    "print(\"Model Accuracy:\", acc)\n",
    "print(\"Model Precision:\", prec)\n",
    "print(\"Model Recall:\", recall)\n",
    "\n",
    "# plot ROC Curve\n",
    "plot_roc_curve(optimized_clf_emp_usera,X_emp_scaled_userb,y_emp_userb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rural-examination",
   "metadata": {},
   "source": [
    "Model Accuracy: 0.7715773809523809   \n",
    "Model Precision: 0.6744548286604362   \n",
    "Model Recall: 0.815442561205273    \n",
    "\n",
    "Model performance on User B for the emotion \"emphasis\" exhibits a model with decent accuracy (77.2%) and recall (81.5%) and poor precision (67.4%). The ROC curve curves to the top left corner and has an AUC value of 0.86 which is good, so overall this model is a good model. \n",
    "\n",
    "The metrics above state that this model correctly identifies the emotion \"emphasis\" 81.5% of the time, however when it does predict \"emphasis\" it is correct only 67% of the time. Therefore the model is slightly overpredicting. \n",
    "\n",
    "We saw an increase in model performance using smaller values of C with the \"negative\" emotion. We will explore this possibility with \"Emphasis\" as well below.\n",
    "\n",
    "###### Re-Optimize parameters using smaller value of C to see if improvement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "elder-clear",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear']\n",
    "C = [0.01,0.0001,0.00001]\n",
    "\n",
    "params = dict(kernel=kernel, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params,n_jobs = 4)\n",
    "best_emp_scaled = grid.fit(X_emp_scaled,y_emp)\n",
    "\n",
    "print(\"Best 5-Fold CrossValidation Estimates for Smaller Values of C\")\n",
    "print(\"Best Kernel:\", best_emp_scaled.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best C:\", best_emp_scaled.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best_emp_scaled.best_score_)\n",
    "\n",
    "# select best model and save as v2\n",
    "optimized_clf_emp_usera_v2 = best_emp_scaled.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "traditional-photography",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions on test data using optimized model\n",
    "y_pred = optimized_clf_emp_usera_v2.predict(X_emp_scaled_userb)\n",
    "\n",
    "# calculate model accuracy\n",
    "acc = accuracy_score(y_emp_userb, y_pred)\n",
    "\n",
    "# calculate model precision\n",
    "prec = precision_score(y_emp_userb, y_pred)\n",
    "\n",
    "# calculate model recall\n",
    "recall = recall_score(y_emp_userb, y_pred)\n",
    "\n",
    "print(\"Model Accuracy:\", acc)\n",
    "print(\"Model Precision:\", prec)\n",
    "print(\"Model Recall:\", recall)\n",
    "\n",
    "# plot ROC Curve\n",
    "plot_roc_curve(optimized_clf_emp_usera_v2,X_emp_scaled_userb,y_emp_userb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dying-worse",
   "metadata": {},
   "source": [
    "Model Accuracy: 0.8244047619047619   \n",
    "Model Precision: 0.7547495682210709   \n",
    "Model Recall: 0.8229755178907722    \n",
    "\n",
    "Results above were very good. A regularization parameter value of 0.01 exhibited only a 1% decrease in accuracy during training on User A whilst improving accuracy (+ 5.3%), precision (+ 8.0%), recall (+ 0.7%), and AUC (+ 3.0 %) when testing on User B. \n",
    "\n",
    "Again, using a small C value increases the margin size and allows for more misclassified points and produces a more \"generalized\" model. \n",
    "\n",
    "# Part C: Additional Experimentation\n",
    "Training and testing will be performed by swapping the users now, with training on User B and testing on User A.\n",
    "\n",
    "## Train on \"Negative\" emotion User B and test on User A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "alike-publisher",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear','poly','rbf']\n",
    "degree = [2,3,4,5,6,7,8,9]\n",
    "C = [0.1,1,10]\n",
    "params = dict(kernel=kernel,degree=degree, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params,n_jobs=5)\n",
    "best_scaled = grid.fit(X_neg_scaled_userb, y_neg_userb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "accessible-package",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Best 5-Fold CrossValidation Estimates for Standardized Data\")\n",
    "print(\"Best Kernel:\", best_scaled.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best Degree:\", best_scaled.best_estimator_.get_params()['degree'])\n",
    "print(\"Best C:\", best_scaled.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best_scaled.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "automated-lucas",
   "metadata": {},
   "outputs": [],
   "source": [
    "# select best model for User B\n",
    "optimized_clf_neg_userb = best_scaled.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "universal-gender",
   "metadata": {},
   "source": [
    "Best 5-Fold CrossValidation Estimates for Standardized Data  \n",
    "Best Kernel: rbf    \n",
    "Best Degree: 3  \n",
    "Best C: 0.1   \n",
    "K-Fold Accuracy: 0.7483787884838079   \n",
    "\n",
    "5-Fold Cross Validation on \"Negative\" emotion on User B yields an average accuracy of only 74.8%, with the best kernel being 'rbf'. This is less accurate than when User A was trained (90.4%). Reasons for this may lie in the distribution of the data, which will be explored below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "urban-virtue",
   "metadata": {},
   "outputs": [],
   "source": [
    "# summary statistics (exclude first/last columns)\n",
    "# collect x,y, & z coordinates as separate dataframes\n",
    "xs = df_neg_userb[df_neg_userb.columns[1::3]]\n",
    "ys = df_neg_userb[df_neg_userb.columns[2::3]]\n",
    "zs = df_neg_userb[df_neg_userb.columns[3::3]]\n",
    "\n",
    "# remove target col\n",
    "xs = xs.drop([\"target\"],axis=1)\n",
    "\n",
    "# array of 3 coordinate axes\n",
    "df_neg_coord = np.array((xs,ys,zs))\n",
    "\n",
    "print(xs.stack().describe())\n",
    "print(ys.stack().describe())\n",
    "print(zs.stack().describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "natural-ecology",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compare distributions between User A and User B\n",
    "fig = make_subplots(rows=1, cols=2,subplot_titles=(\"Negative: User A\",\"User B\"))\n",
    "fig.add_trace(go.Violin(y=X_neg.unstack(),name=\"Negative: User A\"),\n",
    "    row=1, col=1\n",
    ")\n",
    "fig.add_trace(go.Violin(y=X_neg_userb.unstack(),name=\"Negative: User B\"),\n",
    "    row=1, col=2\n",
    ")\n",
    "\n",
    "fig.update_layout(title_text=\"GFE Data (Negative) Distribution Before Standardization\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fresh-latex",
   "metadata": {},
   "source": [
    "Both descriptive statistics and distributions of the data are very similar between User A and User B, with the only significant difference being that User B is a slightly larger dataset. However, there is no outright difference between the two datasets that could attribute to the large difference in k-fold cross validation training accuracy between User A and User B.\n",
    "\n",
    "###### Test on User A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "partial-subsection",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions on test data using optimized model\n",
    "y_pred = optimized_clf_neg_userb.predict(X_neg_scaled)\n",
    "\n",
    "# calculate model accuracy\n",
    "acc = accuracy_score(y_neg, y_pred)\n",
    "\n",
    "# calculate model precision\n",
    "prec = precision_score(y_neg, y_pred)\n",
    "\n",
    "# calculate model recall\n",
    "recall = recall_score(y_neg, y_pred)\n",
    "\n",
    "print(\"Model Accuracy:\", acc)\n",
    "print(\"Model Precision:\", prec)\n",
    "print(\"Model Recall:\", recall)\n",
    "\n",
    "# plot ROC Curve\n",
    "plot_roc_curve(optimized_clf_neg_userb,X_neg_scaled,y_neg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unexpected-machine",
   "metadata": {},
   "source": [
    "Model Accuracy: 0.6067615658362989  \n",
    "Model Precision: 0.6162162162162163   \n",
    "Model Recall: 0.4318181818181818     \n",
    "\n",
    "Results of testing on User A produced a model that is 60.7% accurate, 61.6% precise, and has a recall of 43.2%.\n",
    "\n",
    "AUC = 0.68 also is very poor. In comparison to training on User A and testing on User B, this model has performed similarly in terms of accuracy and precision. \n",
    "\n",
    "Adjustment of the regularization parameter may induce better model results, try below.\n",
    "\n",
    "###### Adjustment of regularization parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "pending-target",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['rbf']\n",
    "degree = [2,3,4,5,6]\n",
    "C = [1,0.1,0.01]\n",
    "\n",
    "params = dict(kernel=kernel, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params,n_jobs=5)\n",
    "best_scaled = grid.fit(X_neg_scaled_userb, y_neg_userb)\n",
    "\n",
    "print(\"Best 5-Fold CrossValidation Estimates for Smaller Values of C\")\n",
    "print(\"Best Kernel:\", best_scaled.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best C:\", best_scaled.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best_scaled.best_score_)\n",
    "\n",
    "# save model as optimized v2\n",
    "optimized_clf_neg_userb_v2 = best_scaled.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "temporal-martin",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions on test data using optimized model\n",
    "y_pred = optimized_clf_neg_userb_v2.predict(X_neg_scaled)\n",
    "\n",
    "# calculate model accuracy\n",
    "acc = accuracy_score(y_neg, y_pred)\n",
    "\n",
    "# calculate model precision\n",
    "prec = precision_score(y_neg, y_pred)\n",
    "\n",
    "# calculate model recall\n",
    "recall = recall_score(y_neg, y_pred)\n",
    "\n",
    "print(\"Model Accuracy:\", acc)\n",
    "print(\"Model Precision:\", prec)\n",
    "print(\"Model Recall:\", recall)\n",
    "\n",
    "# plot ROC Curve\n",
    "plot_roc_curve(optimized_clf_neg_userb_v2,X_neg_scaled,y_neg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "revolutionary-maker",
   "metadata": {},
   "source": [
    "Model Accuracy: 0.7215302491103203    \n",
    "Model Precision: 0.7494199535962877     \n",
    "Model Recall: 0.6117424242424242   \n",
    "\n",
    "After adjustment of the regularization parameter to C=1, the cross-validated model suffered a mean accuracy decrease of 2.7%, however precision and accuracy increased bh 12.0% and 14.0% respectively when testing the model. In addition the ROC curve improved (AUC +0.13). \n",
    "\n",
    "Overall the adjustment saw a slight worsening of the model training performance, but the tradeoff was a significant improvement on the testing performance of the model.\n",
    "\n",
    "\n",
    "## Train on \"Emphasis\" User B and test on User A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "american-agent",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear']\n",
    "degree = [2,3,4,5,6]\n",
    "C = [0.001, 0.01,0.1, 1, 10]\n",
    "\n",
    "params = dict(kernel=kernel, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params,n_jobs=5)\n",
    "best_emp_scaled = grid.fit(X_emp_scaled_userb, y_emp_userb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "measured-shareware",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Best 5-Fold CrossValidation Estimates for Standardized Data\")\n",
    "print(\"Best Kernel:\", best_emp_scaled.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best Degree:\", best_emp_scaled.best_estimator_.get_params()['degree'])\n",
    "print(\"Best C:\", best_emp_scaled.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best_emp_scaled.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "loose-anger",
   "metadata": {},
   "outputs": [],
   "source": [
    "# select best model for User B\n",
    "optimized_clf_emp_userb = best_emp_scaled.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dynamic-consumer",
   "metadata": {},
   "source": [
    "##### Test on User A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "documented-arkansas",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions on test data using optimized model\n",
    "y_pred = optimized_clf_emp_userb.predict(X_emp_scaled)\n",
    "\n",
    "# calculate model accuracy\n",
    "acc = accuracy_score(y_emp, y_pred)\n",
    "\n",
    "# calculate model precision\n",
    "prec = precision_score(y_emp, y_pred)\n",
    "\n",
    "# calculate model recall\n",
    "recall = recall_score(y_emp, y_pred)\n",
    "\n",
    "print(\"Model Accuracy:\", acc)\n",
    "print(\"Model Precision:\", prec)\n",
    "print(\"Model Recall:\", recall)\n",
    "\n",
    "# plot ROC Curve\n",
    "plot_roc_curve(optimized_clf_emp_userb,X_emp_scaled,y_emp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "embedded-vaccine",
   "metadata": {},
   "source": [
    "Model Accuracy: 0.8460441910192444    \n",
    "Model Precision: 0.7375          \n",
    "Model Recall: 0.5363636363636364       \n",
    "\n",
    "For \"Emphasis\", testing on User A resulted in an accuracy of 84.6%, precision of 73.8%, and recall of 53.6%. In terms of accuracy this model is similar to previously when we trained on User A and tested on User B (82.4% accuracy). However precision and recall are significantly less. The ROC curve indicates that the model does not have a good balance between specificity and sensitivity, as the model is too sensitive. And overall, the AUC value is 0.66 which is very poor.\n",
    "\n",
    "Adjusting the regularization parameter may improve these metrics, which will be displayed below. Decreasing C will increase the SVM margin and generalize the model more.\n",
    "\n",
    "##### Optimize regularization parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "lovely-chair",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear']\n",
    "degree = [2,3,4,5,6]\n",
    "C = [0.0001]\n",
    "\n",
    "params = dict(kernel=kernel, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params,n_jobs=5)\n",
    "best_emp_scaled = grid.fit(X_emp_scaled_userb, y_emp_userb)\n",
    "\n",
    "print(\"Best 5-Fold CrossValidation Estimates for Standardized Data\")\n",
    "print(\"Best Kernel:\", best_emp_scaled.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best Degree:\", best_emp_scaled.best_estimator_.get_params()['degree'])\n",
    "print(\"Best C:\", best_emp_scaled.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best_emp_scaled.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "italic-tsunami",
   "metadata": {},
   "outputs": [],
   "source": [
    "# save optimized regularization model as v2 \n",
    "optimized_clf_emp_userb_v2 = best_emp_scaled.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "challenging-registrar",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions on test data using optimized model\n",
    "y_pred = optimized_clf_emp_userb_v2.predict(X_emp_scaled)\n",
    "\n",
    "# calculate model accuracy\n",
    "acc = accuracy_score(y_emp, y_pred)\n",
    "\n",
    "# calculate model precision\n",
    "prec = precision_score(y_emp, y_pred)\n",
    "\n",
    "# calculate model recall\n",
    "recall = recall_score(y_emp, y_pred)\n",
    "\n",
    "print(\"Model Accuracy:\", acc)\n",
    "print(\"Model Precision:\", prec)\n",
    "print(\"Model Recall:\", recall)\n",
    "\n",
    "# plot ROC Curve\n",
    "plot_roc_curve(optimized_clf_emp_userb_v2,X_emp_scaled,y_emp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "weird-geography",
   "metadata": {},
   "source": [
    "Model Accuracy: 0.8524590163934426   \n",
    "Model Precision: 0.7757847533632287   \n",
    "Model Recall: 0.5242424242424243    \n",
    "AUC: 0.85   \n",
    "\n",
    "After decreasing the C value to 0.0001, the cross validation mean accuracy of the model increased from 90.9% to 85%, however the test accuracy on User A increased by 0.6% and more importantly the ROC curve is significantly improved (AUC = 0.85). Overall this would suggest a robust model, and it carries a good tradeoff between specificity and sensitivity.\n",
    "\n",
    "## Dimensionality Reduction: Principal Component Analysis\n",
    "\n",
    "### PCA on User A \"Negative\"\n",
    "PCA will be performed on \"Negative\" User A and trained, then tested on User B to view the performance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "embedded-motion",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform PCA on User A \"Negative\" emotion\n",
    "pca_neg = PCA(.99).fit(X_neg_scaled)\n",
    "\n",
    "# plot to see total variance explained by components\n",
    "px.scatter(np.cumsum(pca_neg.explained_variance_ratio_), title=\"PCA: 99% Cumulative Variance for Negative Emotion\", labels={\n",
    "    \"value\":\"Cumulative Variance\", \"index\":\"Components\"\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "south-swing",
   "metadata": {},
   "source": [
    "The graph above suggests that 99% of the variance lies within the first 23 components of the 300 components. Therefore, PCA will only use the first 23 components.\n",
    "\n",
    "##### Select PCA components and train model on User A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "strategic-taylor",
   "metadata": {},
   "outputs": [],
   "source": [
    "# save 23 components of PCA\n",
    "n_comp = 23\n",
    "\n",
    "# create pca instance and fit it to data\n",
    "pca = PCA(n_comp)\n",
    "pca.fit(X_neg_scaled)\n",
    "\n",
    "# transform dataset\n",
    "X_neg_scaled_pca = pca.transform(X_neg_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "possible-cartoon",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear']\n",
    "degree = [2,3,4,5,6,7,8,9]\n",
    "C = [0.01,0.1,1,10]\n",
    "\n",
    "params = dict(kernel=kernel, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params)\n",
    "best_scaled = grid.fit(X_neg_scaled_pca,y_neg)\n",
    "\n",
    "print(\"Best 5-Fold CrossValidation Estimates for Standardized Data\")\n",
    "print(\"Best Kernel:\", best_scaled.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best C:\", best_scaled.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best_scaled.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "middle-return",
   "metadata": {},
   "outputs": [],
   "source": [
    "# select best model\n",
    "optimized_clf_neg_usera_pca = best_scaled.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "yellow-prototype",
   "metadata": {},
   "source": [
    "###### Test on User B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "induced-insulin",
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform PCA on User B\n",
    "\n",
    "X_neg_scaled_userb_pca = pca.transform(X_neg_scaled_userb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "immune-vinyl",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions on test data using optimized model\n",
    "y_pred = optimized_clf_neg_usera_pca.predict(X_neg_scaled_userb_pca)\n",
    "\n",
    "# calculate model accuracy\n",
    "acc = accuracy_score(y_neg_userb, y_pred)\n",
    "\n",
    "# calculate model precision\n",
    "prec = precision_score(y_neg_userb, y_pred)\n",
    "\n",
    "# calculate model recall\n",
    "recall = recall_score(y_neg_userb, y_pred)\n",
    "\n",
    "print(\"Model Accuracy:\", acc)\n",
    "print(\"Model Precision:\", prec)\n",
    "print(\"Model Recall:\", recall)\n",
    "\n",
    "# plot ROC Curve\n",
    "plot_roc_curve(optimized_clf_neg_usera_pca,X_neg_scaled_userb_pca,y_neg_userb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unauthorized-chicago",
   "metadata": {},
   "source": [
    "Model Accuracy: 0.5960809102402023    \n",
    "Model Precision: 0.548472775564409    \n",
    "Model Recall: 0.5800561797752809    \n",
    "\n",
    "\n",
    "The results from PCA are similar to non-PCA modelling, with an accuracy of 59.6% (slightly higher than before) and a precision and recall of 54.8% and 58.0% accordingly. The ROC curve does not suggest a good balance between specificity and sensitivity, and therefore the model performs similarly to the non-PCA model.\n",
    "\n",
    "Overall PCA was successful as it produced a similar model using only 23 components in comparison to 300.\n",
    "\n",
    "Compare the differences below between User A and B to see if the number of principle components varies between the two users.\n",
    "\n",
    "###### Compare PCA between User A and User B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "north-haven",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform PCA on User B \"Negatve\" emotion\n",
    "pca_neg_userb = PCA().fit(X_neg_scaled_userb)\n",
    "\n",
    "x = np.arange(1,301)\n",
    "\n",
    "# Look at Distributions before and after standardizing and normalizing\n",
    "fig = make_subplots(rows=1, cols=2,subplot_titles=(\"User A\",\"User B\"))\n",
    "fig.add_trace(go.Scatter(x=x,y=np.cumsum(pca_neg.explained_variance_ratio_),name=\"User A\"),\n",
    "    row=1, col=1\n",
    ")\n",
    "fig.add_trace(go.Scatter(x=x,y=np.cumsum(pca_neg_userb.explained_variance_ratio_),name=\"User B\"),\n",
    "    row=1, col=2\n",
    ")\n",
    "fig.update_layout(title_text=\"PCA Comparisons: User A v.s. User B\")\n",
    "fig.update_yaxes(\n",
    "        title_text = \"Cumulative Variance\")\n",
    "fig.update_xaxes(\n",
    "        title_text = \"# of Components\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chicken-packet",
   "metadata": {},
   "source": [
    "The visualization above suggests that for User A ~99% of the variance is within the first 23 parameters, whist for User B it is within the first 81 parameters. Using less parameters showed very little change in model performance, which would suggest that for \"negative\" emotions the only a handful of components are of signficance, and perhaps the more data used the worse the predictions.\n",
    "\n",
    "\n",
    "### PCA on User A \"Emphasis\"\n",
    "PCA will be performed on \"Emphasis\" User A and trained, then tested on User B to view the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "included-mainland",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform PCA on User A \"Emphasis\" emotion\n",
    "pca_emp = PCA(0.99).fit(X_emp_scaled)\n",
    "\n",
    "# plot to see total variance explained by components\n",
    "px.scatter(np.cumsum(pca_emp.explained_variance_ratio_), title=\"PCA: 99% Cumulative Variance for Emphasis Emotion\", labels={\n",
    "    \"value\":\"Cumulative Variance\", \"index\":\"Components\"\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "breathing-doctor",
   "metadata": {},
   "source": [
    "The graph above suggests that 99% of the variance lies within the first 22 components of the 300 components. Therefore, PCA will only use the first 22 components.\n",
    "\n",
    "##### Select PCA components and train model on User A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "scheduled-football",
   "metadata": {},
   "outputs": [],
   "source": [
    "# save 22 components of PCA\n",
    "n_comp = 22\n",
    "\n",
    "pca = PCA(n_comp)\n",
    "pca.fit(X_emp_scaled)\n",
    "\n",
    "X_emp_scaled_pca = pca.transform(X_emp_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "palestinian-publisher",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty model object\n",
    "clf = svm.SVC()\n",
    "\n",
    "# optimization paramters\n",
    "kernel = ['linear']\n",
    "degree = [2,3,4,5,6,7,8,9]\n",
    "C = [0.001,0.01,0.1,1,10]\n",
    "\n",
    "params = dict(kernel=kernel, C=C)\n",
    "\n",
    "# Use GridSearch to optimize parameters through 5-Fold Cross Validation\n",
    "grid = GridSearchCV(clf, params,n_jobs = 4)\n",
    "best_emp_scaled = grid.fit(X_emp_scaled_pca,y_emp)\n",
    "\n",
    "print(\"Best 5-Fold CrossValidation Estimates for Standardized Data\")\n",
    "print(\"Best Kernel:\", best_emp_scaled.best_estimator_.get_params()['kernel'])\n",
    "print(\"Best C:\", best_emp_scaled.best_estimator_.get_params()['C'])\n",
    "print(\"K-Fold Accuracy:\", best_emp_scaled.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "favorite-france",
   "metadata": {},
   "outputs": [],
   "source": [
    "# select best model\n",
    "optimized_clf_emp_usera_pca = best_emp_scaled.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "protected-kinase",
   "metadata": {},
   "source": [
    "###### Test on User B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adaptive-organic",
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform PCA on User B\n",
    "\n",
    "X_emp_scaled_userb_pca = pca.transform(X_emp_scaled_userb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "decent-newfoundland",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions on test data using optimized model\n",
    "y_pred = optimized_clf_emp_usera_pca.predict(X_emp_scaled_userb_pca)\n",
    "\n",
    "# calculate model accuracy\n",
    "acc = accuracy_score(y_emp_userb, y_pred)\n",
    "\n",
    "# calculate model precision\n",
    "prec = precision_score(y_emp_userb, y_pred)\n",
    "\n",
    "# calculate model recall\n",
    "recall = recall_score(y_emp_userb, y_pred)\n",
    "\n",
    "print(\"Model Accuracy:\", acc)\n",
    "print(\"Model Precision:\", prec)\n",
    "print(\"Model Recall:\", recall)\n",
    "\n",
    "# plot ROC Curve\n",
    "plot_roc_curve(optimized_clf_emp_usera_pca,X_emp_scaled_userb_pca,y_emp_userb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "korean-treasury",
   "metadata": {},
   "source": [
    "Model Accuracy: 0.8139880952380952   \n",
    "Model Precision: 0.7513416815742398    \n",
    "Model Recall: 0.7909604519774012     \n",
    "\n",
    "Using only 22 components for the emotion \"Emphasis\", the model performance decrease by 1% in terms of accuracy in comparison to the non-PCA results. The ROC curve is very similar also, with only a 1% decrease of AUC. \n",
    "\n",
    "Again PCA shows slight decreases in terms of performance, but large reductions in dimensionality. This would again suggest that only a handful of parameters are of importance when predicting \"emphasis\" emotions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}