Dimension Reduction and Clustering Techniques on Housing Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Regression on Housing Data\n",
    "\n",
    "#### Team:\n",
    "1. Hoa Nguyen\n",
    "2. Shailesh Krishna\n",
    "3. Sheetal Kalburgi\n",
    "\n",
    "#### Introduction\n",
    "For this project, we will be applying the dimension reduction and clustering techniques on the dataset containing both numerical as well as categorical features. For this purpose we are using the well known [housing dataset](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/overview) on Kaggle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Package imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "from sklearn.model_selection import train_test_split\n",
    "import prince\n",
    "import gower\n",
    "import itertools\n",
    "from sklearn.linear_model import Ridge\n",
    "from sklearn import metrics\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.max_rows', 200)\n",
    "import time\n",
    "import pyclustering\n",
    "from pyclustering.cluster.kmedoids import kmedoids\n",
    "from sklearn.metrics.cluster import normalized_mutual_info_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data Cleaning**  \n",
    "* Removing the columns that contains too many missing values.\n",
    "* Remove the missing values if any in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load data set\n",
    "data = pd.read_csv('train.csv')\n",
    "data = data.drop('Id', axis = 1)\n",
    "\n",
    "# Remove columns that have too many missing values\n",
    "data = data.drop(data.columns[data.isnull().sum() > 30], axis = 1)\n",
    "\n",
    "# Remove missing values\n",
    "data.dropna(inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cleaned dataset contains 1451 rows and 64 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The cleaned dataset contains {} rows and {} columns\".format(data.shape[0],data.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Are there null values in the cleaned dataset?: False\n"
     ]
    }
   ],
   "source": [
    "# Check if there are any missing values in the dataset\n",
    "print(\"Are there null values in the cleaned dataset?: {}\".format(data.isnull().values.any()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part I: Dimension Reduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. Define X and y**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = data.drop(columns='SalePrice',axis=1)\n",
    "y = data['SalePrice']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Determine the unique data types**\n",
    "* Determine the unique data types in the cleaned dataset to segregate the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cleaned dataset contains 3 different data types and they are: float64, int64, object\n"
     ]
    }
   ],
   "source": [
    "# Data types\n",
    "column_datatypes = set()\n",
    "for column in data.columns:\n",
    "    column_datatypes.add(str(data[column].dtype))\n",
    "print(\"The cleaned dataset contains {} different data types and they are: {}\".format(len(column_datatypes), \", \".join(column_datatypes)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "numerical_features = list()\n",
    "categorical_features = list()\n",
    "for column in X.columns:\n",
    "    # In the dataset we only have float and int64.\n",
    "    if (data[column].dtype == 'float64' or data[column].dtype == 'int64'):\n",
    "        numerical_features.append(column)\n",
    "    # Categorical\n",
    "    elif (data[column].dtype == 'object'):\n",
    "        categorical_features.append(column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are a total of 34 numerical features in the dataset.\n"
     ]
    }
   ],
   "source": [
    "print('There are a total of {} numerical features in the dataset.'.format(len(numerical_features)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numerical features in the dataset are:MSSubClass,  LotArea,  OverallQual,  OverallCond,  YearBuilt,  YearRemodAdd,  MasVnrArea,  BsmtFinSF1,  BsmtFinSF2,  BsmtUnfSF,  TotalBsmtSF,  1stFlrSF,  2ndFlrSF,  LowQualFinSF,  GrLivArea,  BsmtFullBath,  BsmtHalfBath,  FullBath,  HalfBath,  BedroomAbvGr,  KitchenAbvGr,  TotRmsAbvGrd,  Fireplaces,  GarageCars,  GarageArea,  WoodDeckSF,  OpenPorchSF,  EnclosedPorch,  3SsnPorch,  ScreenPorch,  PoolArea,  MiscVal,  MoSold,  YrSold\n"
     ]
    }
   ],
   "source": [
    "print('Numerical features in the dataset are:{}'.format(\",  \".join(numerical_features)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are a total of 29 categorical features in the dataset.\n"
     ]
    }
   ],
   "source": [
    "print('There are a total of {} categorical features in the dataset.'.format(len(categorical_features)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical features in the dataset are: MSZoning,  Street,  LotShape,  LandContour,  Utilities,  LotConfig,  LandSlope,  Neighborhood,  Condition1,  Condition2,  BldgType,  HouseStyle,  RoofStyle,  RoofMatl,  Exterior1st,  Exterior2nd,  MasVnrType,  ExterQual,  ExterCond,  Foundation,  Heating,  HeatingQC,  CentralAir,  Electrical,  KitchenQual,  Functional,  PavedDrive,  SaleType,  SaleCondition\n"
     ]
    }
   ],
   "source": [
    "print('Categorical features in the dataset are: {}'.format(\",  \".join(categorical_features)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Train/Validation/Test split**\n",
    "* Split the data into train and test set. (75-25 split ratio)\n",
    "* Split the training set into training and validation sets. For the entire dataset we did a 60-15-25 split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_valid, X_test, y_train_valid, y_test = train_test_split(X, y, test_size = 0.25, random_state = 18)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train_valid, y_train_valid, test_size = 0.2, random_state = 18)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4. Feature separation train/validation set (numerical,categorical)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_numerical = X_train[numerical_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_train_numerical dataset contains 870 rows and 34 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_train_numerical dataset contains {} rows and {} columns\".format(X_train_numerical.shape[0],X_train_numerical.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_categorical = X_train[categorical_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_train_categorical dataset contains 870 rows and 29 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_train_categorical dataset contains {} rows and {} columns\".format(X_train_categorical.shape[0],X_train_categorical.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_valid_numerical = X_valid[numerical_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_valid_numerical dataset contains 218 rows and 34 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_valid_numerical dataset contains {} rows and {} columns\".format(X_valid_numerical.shape[0],X_valid_numerical.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_valid_categorical = X_valid[categorical_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_valid_categorical dataset contains 218 rows and 29 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_valid_categorical dataset contains {} rows and {} columns\".format(X_valid_categorical.shape[0],X_valid_categorical.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5. Maintain the same set of levels across the training and validation set for categorical features.**\n",
    "* The number of categorical features at the same level get reduced from 29 features to only 12 features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "keep = X_train_categorical.nunique() == X_valid_categorical.nunique()\n",
    "X_train_categorical = X_train_categorical[X_train_categorical.columns[keep]]\n",
    "X_valid_categorical = X_valid_categorical[X_valid_categorical.columns[keep]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "keep = list()\n",
    "for i in range(X_train_categorical.shape[1]):\n",
    "    keep.append(all(np.sort(X_train_categorical.iloc[:,i].unique()) == np.sort(X_valid_categorical.iloc[:,i].unique())))\n",
    "X_train_categorical = X_train_categorical[X_train_categorical.columns[keep]]\n",
    "X_valid_categorical = X_valid_categorical[X_valid_categorical.columns[keep]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_train_categorical dataset after running the above code now contains 870 rows and 12 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_train_categorical dataset after running the above code now contains {} rows and {} columns\".format(X_train_categorical.shape[0],X_train_categorical.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_valid_categorical dataset after running the above code contains 218 rows and 12 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_valid_categorical dataset after running the above code contains {} rows and {} columns\".format(X_valid_categorical.shape[0],X_valid_categorical.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6. Hyperparameter tuning**  \n",
    "To perform PCA and MCA, we are using the methods provided by **[prince](https://github.com/MaxHalford/prince)** library and will tune the same set of parameters for both PCA and MCA. Our dataset has around 1500 observations and we chose only a few parameters as the number of resulting combinations to check were way more than the number of observations in the dataset. Still we ended up with 4000+ combinations. Below mentioned are the parameters that will be tuned:\n",
    "\n",
    "1. Number of components (n_components): It denotes the number of components used to perform PCA or MCA.\n",
    "2. Number of iterations (n_iter): The number of iterations used for computing the SVD.\n",
    "\n",
    "To standardize the numerical data we will be setting the **rescale_with_mean** and **rescale_with_std parameters** to True.  \n",
    "\n",
    "As we are tuning parameters for PCA and MCA, we are also going to tune the alpha value for the ridge regression to determine the best alpha value for the specific parameter pair.  \n",
    "If we get the lowest validation MSE for the pair, then we can use the best value for alpha that we determined while hyperparameter tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The total number of hyperparameter pairs for PCA and MCA are: 1815\n"
     ]
    }
   ],
   "source": [
    "n_components_pca = range(1,34) \n",
    "n_components_mca = range(1,12) # there are only 29 categorical features.\n",
    "n_iter = range(1,6)\n",
    "# hyperparameter pairs\n",
    "hyperparameter_pairs = list(itertools.product(n_components_pca, n_components_mca, n_iter))\n",
    "print(\"The total number of hyperparameter pairs for PCA and MCA are: {}\".format(len(hyperparameter_pairs)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time() # Capture the execution start time.\n",
    "Validation_Scores = list()\n",
    "Best_Alphas = list()\n",
    "for pair in hyperparameter_pairs:\n",
    "    # Perform PCA\n",
    "    PCA = prince.PCA(n_components = pair[0],\n",
    "                     n_iter = pair[2],\n",
    "                     rescale_with_mean=True, # Standardization\n",
    "                     rescale_with_std=True, # Standardization\n",
    "                     copy=True, # Prevent inplace substitution\n",
    "                     check_input=True,\n",
    "                     engine = 'sklearn', # sklearn engine\n",
    "                     random_state = 18)\n",
    "    # Fit only the training set.\n",
    "    PCA.fit(X_train_numerical)\n",
    "    # Transform both the training and validation set.\n",
    "    X_train_numerical_transformed = PCA.transform(X_train_numerical)\n",
    "    X_valid_numerical_transformed = PCA.transform(X_valid_numerical)\n",
    "    \n",
    "    # Perform MCA\n",
    "    MCA = prince.MCA(n_components = pair[1],\n",
    "                     n_iter = pair[2],\n",
    "                     copy=True,\n",
    "                     check_input=True,\n",
    "                     engine='sklearn',\n",
    "                     random_state=18)\n",
    "    # Fit only the training set\n",
    "    MCA.fit(X_train_categorical)\n",
    "    # Transform both the training and validation set\n",
    "    X_train_categorical_transformed = MCA.transform(X_train_categorical)\n",
    "    X_valid_categorical_transformed = MCA.transform(X_valid_categorical)\n",
    "    \n",
    "    # Create a combined training and validation set containing reduced features.\n",
    "    X_train_combined = pd.concat([X_train_numerical_transformed, X_train_categorical_transformed],axis = 1)\n",
    "    X_valid_combined = pd.concat([X_valid_numerical_transformed, X_valid_categorical_transformed],axis = 1)\n",
    "    \n",
    "    # Tuning the alphas for the ridge\n",
    "    alphas = np.logspace(-10, 10, 21) # We will use lambda on powers of 10 scale\n",
    "    alphas_index = np.linspace(-10, 10, 21) # The lambda values on log scale\n",
    "    # validation scores for alphas\n",
    "    alpha_validation_score = []\n",
    "    for a in alphas:\n",
    "        ridge = Ridge(alpha=a,random_state=18)\n",
    "        ridge.fit(X_train_combined, y_train)\n",
    "        alpha_validation_score.append(metrics.mean_squared_error(ridge.predict(X_valid_combined), y_valid))\n",
    "        \n",
    "    best_alpha = alphas[np.argmin(alpha_validation_score)]\n",
    "    Best_Alphas.append(best_alpha)\n",
    "    \n",
    "    # Fitting the model using the best alpha value to calculate the validation MSE.\n",
    "    RIDGE = Ridge(alpha=best_alpha,random_state=18)\n",
    "    RIDGE.fit(X_train_combined, y_train)\n",
    "    Validation_Scores.append(metrics.mean_squared_error(RIDGE.predict(X_valid_combined), y_valid))\n",
    "    \n",
    "end = time.time() # Capture the execution end time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total execution time for hyperparameter tuning is: 14.96 mins\n"
     ]
    }
   ],
   "source": [
    "print(\"Total execution time for hyperparameter tuning is: {:.4} mins\".format(float(end-start)/60))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lowest MSE on the validation set is: 773388047.0924511\n"
     ]
    }
   ],
   "source": [
    "print('The lowest MSE on the validation set is: {}'.format(min(Validation_Scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**7. Best parameters**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best hyperparameter pair for PCA and MCA are (8, 11, 1) and the best alpha value for Ridge Regression is: 1e-10\n"
     ]
    }
   ],
   "source": [
    "print('Best hyperparameter pair for PCA and MCA are {} and the best alpha value for Ridge Regression is: {}'.\\\n",
    "      format(hyperparameter_pairs[np.argmin(Validation_Scores)],Best_Alphas[np.argmin(Validation_Scores)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**8. Reapplying the code on the (train+validation) and test set.**  \n",
    "* Performing the same steps to separate the numerical and categorical features from the training+validation and test set.\n",
    "* Maintaining the same levels for categorical features in the training+validation and test set.\n",
    "* Using the best parameters to perform PCA on numerical features and MCA on categorical features.\n",
    "* Combining the reduced numerical and categorical features to run Ridge regression using the best alpha value.\n",
    "* Calculating the validation MSE on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_valid_numerical = X_train_valid[numerical_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_train_valid_numerical dataset contains 1088 rows and 34 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_train_valid_numerical dataset contains {} rows and {} columns\".format(X_train_valid_numerical.shape[0],X_train_valid_numerical.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_valid_categorical = X_train_valid[categorical_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_train_valid_categorical dataset contains 1088 rows and 29 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_train_valid_categorical dataset contains {} rows and {} columns\".format(X_train_valid_categorical.shape[0],X_train_valid_categorical.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test_numerical = X_test[numerical_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_test_numerical dataset contains 363 rows and 34 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_test_numerical dataset contains {} rows and {} columns\".format(X_test_numerical.shape[0],X_test_numerical.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test_categorical = X_test[categorical_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_test_categorical dataset contains 363 rows and 29 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_test_categorical dataset contains {} rows and {} columns\".format(X_test_categorical.shape[0],X_test_categorical.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Maintaing the same levels across training+validation and test categorical features.\n",
    "keep2 = X_train_valid_categorical.nunique() == X_test_categorical.nunique()\n",
    "X_train_valid_categorical = X_train_valid_categorical[X_train_valid_categorical.columns[keep2]]\n",
    "X_test_categorical = X_test_categorical[X_test_categorical.columns[keep2]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "keep2 = list()\n",
    "for i in range(X_train_valid_categorical.shape[1]):\n",
    "    keep2.append(all(np.sort(X_train_valid_categorical.iloc[:,i].unique()) == np.sort(X_test_categorical.iloc[:,i].unique())))\n",
    "X_train_valid_categorical = X_train_valid_categorical[X_train_valid_categorical.columns[keep2]]\n",
    "X_test_categorical = X_test_categorical[X_test_categorical.columns[keep2]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_train_valid_categorical dataset after running the above code now contains 1088 rows and 16 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_train_valid_categorical dataset after running the above code now contains {} rows and {} columns\".format(X_train_valid_categorical.shape[0],X_train_valid_categorical.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_test_categorical dataset after running the above code now contains 363 rows and 16 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_test_categorical dataset after running the above code now contains {} rows and {} columns\".format(X_test_categorical.shape[0],X_test_categorical.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform PCA\n",
    "PCA = prince.PCA(n_components = hyperparameter_pairs[np.argmin(Validation_Scores)][0],\n",
    "                     n_iter = hyperparameter_pairs[np.argmin(Validation_Scores)][2],\n",
    "                     rescale_with_mean=True, # Standardization\n",
    "                     rescale_with_std=True, # Standardization\n",
    "                     copy=True, # Prevent inplace substitution\n",
    "                     check_input=True,\n",
    "                     engine = 'sklearn', # sklearn engine\n",
    "                     random_state = 18)\n",
    "    \n",
    "# Fit only the training set.\n",
    "PCA.fit(X_train_valid_numerical)\n",
    "# Transform both the training and validation set.\n",
    "X_train_valid_numerical_transformed = PCA.transform(X_train_valid_numerical)\n",
    "X_test_numerical_transformed = PCA.transform(X_test_numerical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform MCA\n",
    "MCA = prince.MCA(n_components = hyperparameter_pairs[np.argmin(Validation_Scores)][1],\n",
    "                     n_iter = hyperparameter_pairs[np.argmin(Validation_Scores)][2],\n",
    "                     copy=True,\n",
    "                     check_input=True,\n",
    "                     engine='sklearn',\n",
    "                     random_state=18)    \n",
    "# Fit only the training set\n",
    "MCA.fit(X_train_valid_categorical)\n",
    "# Transform both the training and validation set\n",
    "X_train_valid_categorical_transformed = MCA.transform(X_train_valid_categorical)\n",
    "X_test_categorical_transformed = MCA.transform(X_test_categorical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a combined training and validation set containing reduced features.\n",
    "X_train_valid_combined = pd.concat([X_train_valid_numerical_transformed, X_train_valid_categorical_transformed],axis = 1)\n",
    "X_test_combined = pd.concat([X_test_numerical_transformed, X_test_categorical_transformed],axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run RIDGE regression\n",
    "RIDGE = Ridge(alpha=Best_Alphas[np.argmin(Validation_Scores)],random_state=18)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation MSE on the test set is: 738752163.8196424\n"
     ]
    }
   ],
   "source": [
    "RIDGE.fit(X_train_valid_combined, y_train_valid)\n",
    "print(\"Validation MSE on the test set is: {}\".format(metrics.mean_squared_error(RIDGE.predict(X_test_combined), y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**9. Baseline Ridge Regression**\n",
    "* Comparing the above results (regression with reduced features) with the regression using the original dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training set\n",
    "X_train_dummy = pd.get_dummies(X_train_categorical,drop_first = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_baseline_combined = pd.concat([X_train_numerical,X_train_dummy], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_train_baseline_combined dataset contains 870 rows and 66 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_train_baseline_combined dataset contains {} rows and {} columns\".format(X_train_baseline_combined.shape[0],X_train_baseline_combined.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Validation set\n",
    "X_valid_dummy = pd.get_dummies(X_valid_categorical,drop_first = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_valid_baseline_combined = pd.concat([X_valid_numerical,X_valid_dummy], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_valid_baseline_combined dataset contains 218 rows and 66 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_valid_baseline_combined dataset contains {} rows and {} columns\".format(X_valid_baseline_combined.shape[0],X_valid_baseline_combined.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tuning the alphas for Ridge Regression\n",
    "alphas = np.logspace(-10, 10, 21) # We will use lambda on powers of 10 scale\n",
    "baseline_validation_scores = []\n",
    "for a in alphas:\n",
    "    lm = Ridge(alpha=a)\n",
    "    scaler = StandardScaler() # standardize the data.\n",
    "    scaler.fit(X_train_baseline_combined)\n",
    "    X_train_baseline_combined_scaled = scaler.transform(X_train_baseline_combined)\n",
    "    X_valid_baseline_combined_scaled = scaler.transform(X_valid_baseline_combined)\n",
    "    lm.fit(X_train_baseline_combined_scaled, y_train) # Fit model on training set\n",
    "    baseline_validation_scores.append(metrics.mean_squared_error(lm.predict(X_valid_baseline_combined_scaled), y_valid)) # Evaluate model on validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lowest MSE on the validation set is: 896261228.2120639\n"
     ]
    }
   ],
   "source": [
    "print('The lowest MSE on the validation set is: {}'.format(min(baseline_validation_scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best value of alpha for the Ridge regression is: 100.0\n"
     ]
    }
   ],
   "source": [
    "print(\"The best value of alpha for the Ridge regression is: {}\".format(alphas[np.argmin(baseline_validation_scores)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**10. Evaluation on the test set**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_valid_dummy = pd.get_dummies(X_train_valid_categorical,drop_first = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_valid_baseline_combined = pd.concat([X_train_valid_numerical,X_train_valid_dummy], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_train_valid_baseline_combined dataset contains 1088 rows and 108 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_train_valid_baseline_combined dataset contains {} rows and {} columns\".format(X_train_valid_baseline_combined.shape[0],X_train_valid_baseline_combined.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test_dummy = pd.get_dummies(X_test_categorical,drop_first = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test_baseline_combined = pd.concat([X_test_numerical,X_test_dummy], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The X_test_baseline_combined dataset contains 363 rows and 108 columns\n"
     ]
    }
   ],
   "source": [
    "print(\"The X_test_baseline_combined dataset contains {} rows and {} columns\".format(X_test_baseline_combined.shape[0],X_test_baseline_combined.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation MSE on the test set is: 716644289.4678497\n"
     ]
    }
   ],
   "source": [
    "# Scale the data\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X_train_valid_baseline_combined)\n",
    "X_train_valid_baseline_combined_scaled = scaler.transform(X_train_valid_baseline_combined)\n",
    "X_test_baseline_combined_scaled = scaler.transform(X_test_baseline_combined)\n",
    "\n",
    "# Fit the data\n",
    "lm2 = Ridge(alpha=alphas[np.argmin(baseline_validation_scores)],random_state=18)\n",
    "lm2.fit(X_train_valid_baseline_combined_scaled,y_train_valid)\n",
    "\n",
    "# Validation MSE\n",
    "print(\"Validation MSE on the test set is: {}\".format(metrics.mean_squared_error(lm2.predict(X_test_baseline_combined_scaled), y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dimensionality Reduction Summary**:  \n",
    "We performed dimensionality reduction on the dataset containing both numerical as well as categorical features and tuned the parameters to perform regression. The below mentioned table shows the validation and test MSE for the Ridge regression using reduced features and original baseline dataset.\n",
    "\n",
    "|MSE |Ridge Baseline |Ridge Reduced Features|\n",
    "|:-:|:-:|:-:|\n",
    "|Validation Set |896261228.21 |765992669.03 |\n",
    "|Test Set | 716644289.46 | 738752163.81|\n",
    "\n",
    "\n",
    "From the above table we can see that the MSE on the validation set was much lower for Ridge regression model with reduced features as compared to baseline Ridge regression model. However the performance of both the models on the test set is comparable.  \n",
    "We got lower MSE on the test set for baseline Ridge regression model as compared to the Ridge regression model having reduced features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part II: Clustering Analysis\n",
    "* Performing clustering analysis with numerical and categorical data combined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. Initial data analysis**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making a copy of the dataset\n",
    "clustering_df = data.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the predictors\n",
    "X2 = data.drop(columns='SalePrice',axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>LandSlope</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>Condition1</th>\n",
       "      <th>Condition2</th>\n",
       "      <th>BldgType</th>\n",
       "      <th>HouseStyle</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>RoofStyle</th>\n",
       "      <th>RoofMatl</th>\n",
       "      <th>Exterior1st</th>\n",
       "      <th>Exterior2nd</th>\n",
       "      <th>MasVnrType</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>ExterQual</th>\n",
       "      <th>ExterCond</th>\n",
       "      <th>Foundation</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>Heating</th>\n",
       "      <th>HeatingQC</th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>1stFlrSF</th>\n",
       "      <th>2ndFlrSF</th>\n",
       "      <th>LowQualFinSF</th>\n",
       "      <th>GrLivArea</th>\n",
       "      <th>BsmtFullBath</th>\n",
       "      <th>BsmtHalfBath</th>\n",
       "      <th>FullBath</th>\n",
       "      <th>HalfBath</th>\n",
       "      <th>BedroomAbvGr</th>\n",
       "      <th>KitchenAbvGr</th>\n",
       "      <th>KitchenQual</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>Functional</th>\n",
       "      <th>Fireplaces</th>\n",
       "      <th>GarageCars</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>PavedDrive</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2003</td>\n",
       "      <td>2003</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>196.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>706</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>856</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>856</td>\n",
       "      <td>854</td>\n",
       "      <td>0</td>\n",
       "      <td>1710</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>8</td>\n",
       "      <td>Typ</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>548</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Veenker</td>\n",
       "      <td>Feedr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>1Story</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>1976</td>\n",
       "      <td>1976</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>MetalSd</td>\n",
       "      <td>MetalSd</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>CBlock</td>\n",
       "      <td>978</td>\n",
       "      <td>0</td>\n",
       "      <td>284</td>\n",
       "      <td>1262</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>1262</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1262</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>6</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>460</td>\n",
       "      <td>Y</td>\n",
       "      <td>298</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MSSubClass MSZoning  LotArea Street LotShape LandContour Utilities  \\\n",
       "0          60       RL     8450   Pave      Reg         Lvl    AllPub   \n",
       "1          20       RL     9600   Pave      Reg         Lvl    AllPub   \n",
       "\n",
       "  LotConfig LandSlope Neighborhood Condition1 Condition2 BldgType HouseStyle  \\\n",
       "0    Inside       Gtl      CollgCr       Norm       Norm     1Fam     2Story   \n",
       "1       FR2       Gtl      Veenker      Feedr       Norm     1Fam     1Story   \n",
       "\n",
       "   OverallQual  OverallCond  YearBuilt  YearRemodAdd RoofStyle RoofMatl  \\\n",
       "0            7            5       2003          2003     Gable  CompShg   \n",
       "1            6            8       1976          1976     Gable  CompShg   \n",
       "\n",
       "  Exterior1st Exterior2nd MasVnrType  MasVnrArea ExterQual ExterCond  \\\n",
       "0     VinylSd     VinylSd    BrkFace       196.0        Gd        TA   \n",
       "1     MetalSd     MetalSd       None         0.0        TA        TA   \n",
       "\n",
       "  Foundation  BsmtFinSF1  BsmtFinSF2  BsmtUnfSF  TotalBsmtSF Heating  \\\n",
       "0      PConc         706           0        150          856    GasA   \n",
       "1     CBlock         978           0        284         1262    GasA   \n",
       "\n",
       "  HeatingQC CentralAir Electrical  1stFlrSF  2ndFlrSF  LowQualFinSF  \\\n",
       "0        Ex          Y      SBrkr       856       854             0   \n",
       "1        Ex          Y      SBrkr      1262         0             0   \n",
       "\n",
       "   GrLivArea  BsmtFullBath  BsmtHalfBath  FullBath  HalfBath  BedroomAbvGr  \\\n",
       "0       1710             1             0         2         1             3   \n",
       "1       1262             0             1         2         0             3   \n",
       "\n",
       "   KitchenAbvGr KitchenQual  TotRmsAbvGrd Functional  Fireplaces  GarageCars  \\\n",
       "0             1          Gd             8        Typ           0           2   \n",
       "1             1          TA             6        Typ           1           2   \n",
       "\n",
       "   GarageArea PavedDrive  WoodDeckSF  OpenPorchSF  EnclosedPorch  3SsnPorch  \\\n",
       "0         548          Y           0           61              0          0   \n",
       "1         460          Y         298            0              0          0   \n",
       "\n",
       "   ScreenPorch  PoolArea  MiscVal  MoSold  YrSold SaleType SaleCondition  \n",
       "0            0         0        0       2    2008       WD        Normal  \n",
       "1            0         0        0       5    2007       WD        Normal  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X2.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Assign labels**\n",
    "* Dividing the dataframe into **five** clusters by assigning each row a label between 1 and 5.\n",
    "* Using pd.qcut to divide (bin the response variable) the df into clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "clustering_df['Clusters'] = pd.qcut(clustering_df.SalePrice, q = 5, labels = [1, 2, 3, 4,5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>LandSlope</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>Condition1</th>\n",
       "      <th>Condition2</th>\n",
       "      <th>BldgType</th>\n",
       "      <th>HouseStyle</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>RoofStyle</th>\n",
       "      <th>RoofMatl</th>\n",
       "      <th>Exterior1st</th>\n",
       "      <th>Exterior2nd</th>\n",
       "      <th>MasVnrType</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>ExterQual</th>\n",
       "      <th>ExterCond</th>\n",
       "      <th>Foundation</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>Heating</th>\n",
       "      <th>HeatingQC</th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>1stFlrSF</th>\n",
       "      <th>2ndFlrSF</th>\n",
       "      <th>LowQualFinSF</th>\n",
       "      <th>GrLivArea</th>\n",
       "      <th>BsmtFullBath</th>\n",
       "      <th>BsmtHalfBath</th>\n",
       "      <th>FullBath</th>\n",
       "      <th>HalfBath</th>\n",
       "      <th>BedroomAbvGr</th>\n",
       "      <th>KitchenAbvGr</th>\n",
       "      <th>KitchenQual</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>Functional</th>\n",
       "      <th>Fireplaces</th>\n",
       "      <th>GarageCars</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>PavedDrive</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>Clusters</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2003</td>\n",
       "      <td>2003</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>196.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>706</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>856</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>856</td>\n",
       "      <td>854</td>\n",
       "      <td>0</td>\n",
       "      <td>1710</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>8</td>\n",
       "      <td>Typ</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>548</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Veenker</td>\n",
       "      <td>Feedr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>1Story</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>1976</td>\n",
       "      <td>1976</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>MetalSd</td>\n",
       "      <td>MetalSd</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>CBlock</td>\n",
       "      <td>978</td>\n",
       "      <td>0</td>\n",
       "      <td>284</td>\n",
       "      <td>1262</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>1262</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1262</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>6</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>460</td>\n",
       "      <td>Y</td>\n",
       "      <td>298</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2001</td>\n",
       "      <td>2002</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>162.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>486</td>\n",
       "      <td>0</td>\n",
       "      <td>434</td>\n",
       "      <td>920</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>920</td>\n",
       "      <td>866</td>\n",
       "      <td>0</td>\n",
       "      <td>1786</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>6</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>608</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>9550</td>\n",
       "      <td>Pave</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Crawfor</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1915</td>\n",
       "      <td>1970</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Wd Sdng</td>\n",
       "      <td>Wd Shng</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>BrkTil</td>\n",
       "      <td>216</td>\n",
       "      <td>0</td>\n",
       "      <td>540</td>\n",
       "      <td>756</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>961</td>\n",
       "      <td>756</td>\n",
       "      <td>0</td>\n",
       "      <td>1717</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>7</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>642</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>272</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>140000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>NoRidge</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>2000</td>\n",
       "      <td>2000</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>350.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>655</td>\n",
       "      <td>0</td>\n",
       "      <td>490</td>\n",
       "      <td>1145</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>1145</td>\n",
       "      <td>1053</td>\n",
       "      <td>0</td>\n",
       "      <td>2198</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>9</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>836</td>\n",
       "      <td>Y</td>\n",
       "      <td>192</td>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>250000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MSSubClass MSZoning  LotArea Street LotShape LandContour Utilities  \\\n",
       "0          60       RL     8450   Pave      Reg         Lvl    AllPub   \n",
       "1          20       RL     9600   Pave      Reg         Lvl    AllPub   \n",
       "2          60       RL    11250   Pave      IR1         Lvl    AllPub   \n",
       "3          70       RL     9550   Pave      IR1         Lvl    AllPub   \n",
       "4          60       RL    14260   Pave      IR1         Lvl    AllPub   \n",
       "\n",
       "  LotConfig LandSlope Neighborhood Condition1 Condition2 BldgType HouseStyle  \\\n",
       "0    Inside       Gtl      CollgCr       Norm       Norm     1Fam     2Story   \n",
       "1       FR2       Gtl      Veenker      Feedr       Norm     1Fam     1Story   \n",
       "2    Inside       Gtl      CollgCr       Norm       Norm     1Fam     2Story   \n",
       "3    Corner       Gtl      Crawfor       Norm       Norm     1Fam     2Story   \n",
       "4       FR2       Gtl      NoRidge       Norm       Norm     1Fam     2Story   \n",
       "\n",
       "   OverallQual  OverallCond  YearBuilt  YearRemodAdd RoofStyle RoofMatl  \\\n",
       "0            7            5       2003          2003     Gable  CompShg   \n",
       "1            6            8       1976          1976     Gable  CompShg   \n",
       "2            7            5       2001          2002     Gable  CompShg   \n",
       "3            7            5       1915          1970     Gable  CompShg   \n",
       "4            8            5       2000          2000     Gable  CompShg   \n",
       "\n",
       "  Exterior1st Exterior2nd MasVnrType  MasVnrArea ExterQual ExterCond  \\\n",
       "0     VinylSd     VinylSd    BrkFace       196.0        Gd        TA   \n",
       "1     MetalSd     MetalSd       None         0.0        TA        TA   \n",
       "2     VinylSd     VinylSd    BrkFace       162.0        Gd        TA   \n",
       "3     Wd Sdng     Wd Shng       None         0.0        TA        TA   \n",
       "4     VinylSd     VinylSd    BrkFace       350.0        Gd        TA   \n",
       "\n",
       "  Foundation  BsmtFinSF1  BsmtFinSF2  BsmtUnfSF  TotalBsmtSF Heating  \\\n",
       "0      PConc         706           0        150          856    GasA   \n",
       "1     CBlock         978           0        284         1262    GasA   \n",
       "2      PConc         486           0        434          920    GasA   \n",
       "3     BrkTil         216           0        540          756    GasA   \n",
       "4      PConc         655           0        490         1145    GasA   \n",
       "\n",
       "  HeatingQC CentralAir Electrical  1stFlrSF  2ndFlrSF  LowQualFinSF  \\\n",
       "0        Ex          Y      SBrkr       856       854             0   \n",
       "1        Ex          Y      SBrkr      1262         0             0   \n",
       "2        Ex          Y      SBrkr       920       866             0   \n",
       "3        Gd          Y      SBrkr       961       756             0   \n",
       "4        Ex          Y      SBrkr      1145      1053             0   \n",
       "\n",
       "   GrLivArea  BsmtFullBath  BsmtHalfBath  FullBath  HalfBath  BedroomAbvGr  \\\n",
       "0       1710             1             0         2         1             3   \n",
       "1       1262             0             1         2         0             3   \n",
       "2       1786             1             0         2         1             3   \n",
       "3       1717             1             0         1         0             3   \n",
       "4       2198             1             0         2         1             4   \n",
       "\n",
       "   KitchenAbvGr KitchenQual  TotRmsAbvGrd Functional  Fireplaces  GarageCars  \\\n",
       "0             1          Gd             8        Typ           0           2   \n",
       "1             1          TA             6        Typ           1           2   \n",
       "2             1          Gd             6        Typ           1           2   \n",
       "3             1          Gd             7        Typ           1           3   \n",
       "4             1          Gd             9        Typ           1           3   \n",
       "\n",
       "   GarageArea PavedDrive  WoodDeckSF  OpenPorchSF  EnclosedPorch  3SsnPorch  \\\n",
       "0         548          Y           0           61              0          0   \n",
       "1         460          Y         298            0              0          0   \n",
       "2         608          Y           0           42              0          0   \n",
       "3         642          Y           0           35            272          0   \n",
       "4         836          Y         192           84              0          0   \n",
       "\n",
       "   ScreenPorch  PoolArea  MiscVal  MoSold  YrSold SaleType SaleCondition  \\\n",
       "0            0         0        0       2    2008       WD        Normal   \n",
       "1            0         0        0       5    2007       WD        Normal   \n",
       "2            0         0        0       9    2008       WD        Normal   \n",
       "3            0         0        0       2    2006       WD       Abnorml   \n",
       "4            0         0        0      12    2008       WD        Normal   \n",
       "\n",
       "   SalePrice Clusters  \n",
       "0     208500        4  \n",
       "1     181500        4  \n",
       "2     223500        4  \n",
       "3     140000        2  \n",
       "4     250000        5  "
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clustering_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>LandSlope</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>Condition1</th>\n",
       "      <th>Condition2</th>\n",
       "      <th>BldgType</th>\n",
       "      <th>HouseStyle</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>RoofStyle</th>\n",
       "      <th>RoofMatl</th>\n",
       "      <th>Exterior1st</th>\n",
       "      <th>Exterior2nd</th>\n",
       "      <th>MasVnrType</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>ExterQual</th>\n",
       "      <th>ExterCond</th>\n",
       "      <th>Foundation</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>Heating</th>\n",
       "      <th>HeatingQC</th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>1stFlrSF</th>\n",
       "      <th>2ndFlrSF</th>\n",
       "      <th>LowQualFinSF</th>\n",
       "      <th>GrLivArea</th>\n",
       "      <th>BsmtFullBath</th>\n",
       "      <th>BsmtHalfBath</th>\n",
       "      <th>FullBath</th>\n",
       "      <th>HalfBath</th>\n",
       "      <th>BedroomAbvGr</th>\n",
       "      <th>KitchenAbvGr</th>\n",
       "      <th>KitchenQual</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>Functional</th>\n",
       "      <th>Fireplaces</th>\n",
       "      <th>GarageCars</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>PavedDrive</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Clusters</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "      <td>294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "      <td>284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "      <td>291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "      <td>287</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          MSSubClass  MSZoning  LotArea  Street  LotShape  LandContour  \\\n",
       "Clusters                                                                 \n",
       "1                295       295      295     295       295          295   \n",
       "2                294       294      294     294       294          294   \n",
       "3                284       284      284     284       284          284   \n",
       "4                291       291      291     291       291          291   \n",
       "5                287       287      287     287       287          287   \n",
       "\n",
       "          Utilities  LotConfig  LandSlope  Neighborhood  Condition1  \\\n",
       "Clusters                                                              \n",
       "1               295        295        295           295         295   \n",
       "2               294        294        294           294         294   \n",
       "3               284        284        284           284         284   \n",
       "4               291        291        291           291         291   \n",
       "5               287        287        287           287         287   \n",
       "\n",
       "          Condition2  BldgType  HouseStyle  OverallQual  OverallCond  \\\n",
       "Clusters                                                               \n",
       "1                295       295         295          295          295   \n",
       "2                294       294         294          294          294   \n",
       "3                284       284         284          284          284   \n",
       "4                291       291         291          291          291   \n",
       "5                287       287         287          287          287   \n",
       "\n",
       "          YearBuilt  YearRemodAdd  RoofStyle  RoofMatl  Exterior1st  \\\n",
       "Clusters                                                              \n",
       "1               295           295        295       295          295   \n",
       "2               294           294        294       294          294   \n",
       "3               284           284        284       284          284   \n",
       "4               291           291        291       291          291   \n",
       "5               287           287        287       287          287   \n",
       "\n",
       "          Exterior2nd  MasVnrType  MasVnrArea  ExterQual  ExterCond  \\\n",
       "Clusters                                                              \n",
       "1                 295         295         295        295        295   \n",
       "2                 294         294         294        294        294   \n",
       "3                 284         284         284        284        284   \n",
       "4                 291         291         291        291        291   \n",
       "5                 287         287         287        287        287   \n",
       "\n",
       "          Foundation  BsmtFinSF1  BsmtFinSF2  BsmtUnfSF  TotalBsmtSF  Heating  \\\n",
       "Clusters                                                                        \n",
       "1                295         295         295        295          295      295   \n",
       "2                294         294         294        294          294      294   \n",
       "3                284         284         284        284          284      284   \n",
       "4                291         291         291        291          291      291   \n",
       "5                287         287         287        287          287      287   \n",
       "\n",
       "          HeatingQC  CentralAir  Electrical  1stFlrSF  2ndFlrSF  LowQualFinSF  \\\n",
       "Clusters                                                                        \n",
       "1               295         295         295       295       295           295   \n",
       "2               294         294         294       294       294           294   \n",
       "3               284         284         284       284       284           284   \n",
       "4               291         291         291       291       291           291   \n",
       "5               287         287         287       287       287           287   \n",
       "\n",
       "          GrLivArea  BsmtFullBath  BsmtHalfBath  FullBath  HalfBath  \\\n",
       "Clusters                                                              \n",
       "1               295           295           295       295       295   \n",
       "2               294           294           294       294       294   \n",
       "3               284           284           284       284       284   \n",
       "4               291           291           291       291       291   \n",
       "5               287           287           287       287       287   \n",
       "\n",
       "          BedroomAbvGr  KitchenAbvGr  KitchenQual  TotRmsAbvGrd  Functional  \\\n",
       "Clusters                                                                      \n",
       "1                  295           295          295           295         295   \n",
       "2                  294           294          294           294         294   \n",
       "3                  284           284          284           284         284   \n",
       "4                  291           291          291           291         291   \n",
       "5                  287           287          287           287         287   \n",
       "\n",
       "          Fireplaces  GarageCars  GarageArea  PavedDrive  WoodDeckSF  \\\n",
       "Clusters                                                               \n",
       "1                295         295         295         295         295   \n",
       "2                294         294         294         294         294   \n",
       "3                284         284         284         284         284   \n",
       "4                291         291         291         291         291   \n",
       "5                287         287         287         287         287   \n",
       "\n",
       "          OpenPorchSF  EnclosedPorch  3SsnPorch  ScreenPorch  PoolArea  \\\n",
       "Clusters                                                                 \n",
       "1                 295            295        295          295       295   \n",
       "2                 294            294        294          294       294   \n",
       "3                 284            284        284          284       284   \n",
       "4                 291            291        291          291       291   \n",
       "5                 287            287        287          287       287   \n",
       "\n",
       "          MiscVal  MoSold  YrSold  SaleType  SaleCondition  SalePrice  \n",
       "Clusters                                                               \n",
       "1             295     295     295       295            295        295  \n",
       "2             294     294     294       294            294        294  \n",
       "3             284     284     284       284            284        284  \n",
       "4             291     291     291       291            291        291  \n",
       "5             287     287     287       287            287        287  "
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking the counts\n",
    "clustering_df.groupby(clustering_df.Clusters).count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Compute Gower distance**\n",
    "* Create an input matrix for the K-medoids clustering algorithm.\n",
    "* Need to install [gower](https://pypi.org/project/gower/) package before running the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gower distance\n",
    "matrix = gower.gower_matrix(X2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4. K-medoids clustering**\n",
    "* Using the methods provided by pyclustering package. Need to install it before running the next cell.\n",
    "* Set random initial medoids.\n",
    "* Run k-medoids for processing distance matrix instead of points.\n",
    "* Perform cluster analysis to get results.\n",
    "* Create clusters and medoids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# random medoids\n",
    "initial_medoids = np.random.randint(1,1451, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# kmedoids\n",
    "kmedoids_instance = kmedoids(matrix, initial_medoids, data_type='distance_matrix')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyclustering.cluster.kmedoids.kmedoids at 0x7fcd9d984090>"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Cluster analysis\n",
    "kmedoids_instance.process()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get clusters\n",
    "clusters = kmedoids_instance.get_clusters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of observations in cluster 1 is 61\n",
      "Number of observations in cluster 2 is 82\n",
      "Number of observations in cluster 3 is 620\n",
      "Number of observations in cluster 4 is 249\n",
      "Number of observations in cluster 5 is 439\n"
     ]
    }
   ],
   "source": [
    "# Count the number of observations in the new clusters.\n",
    "for i in range(len(clusters)):\n",
    "    print('Number of observations in cluster {} is {}'.format(i+1,len(clusters[i])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get updated medoids\n",
    "medoids = kmedoids_instance.get_medoids()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5. Map the newly created labels to the existing data**\n",
    "* Get the corresponding labels from the clusters and store it in the list.\n",
    "* Add the list to the clustering_df to perform comparision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the corresponding labels and create a list of labels.\n",
    "labels = list()\n",
    "for i in range(len(clustering_df.index)):\n",
    "    if i in clusters[0]:\n",
    "        labels.append(1) # cluster 1\n",
    "    elif i in clusters[1]:\n",
    "        labels.append(2)\n",
    "    elif i in clusters[2]:\n",
    "        labels.append(3)\n",
    "    elif i in clusters[3]:\n",
    "        labels.append(4)\n",
    "    elif i in clusters[4]:\n",
    "        labels.append(5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding the list as new column to the dataframe.\n",
    "clustering_df['NewClusters'] = labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>LandSlope</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>Condition1</th>\n",
       "      <th>Condition2</th>\n",
       "      <th>BldgType</th>\n",
       "      <th>HouseStyle</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>RoofStyle</th>\n",
       "      <th>RoofMatl</th>\n",
       "      <th>Exterior1st</th>\n",
       "      <th>Exterior2nd</th>\n",
       "      <th>MasVnrType</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>ExterQual</th>\n",
       "      <th>ExterCond</th>\n",
       "      <th>Foundation</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>Heating</th>\n",
       "      <th>HeatingQC</th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>1stFlrSF</th>\n",
       "      <th>2ndFlrSF</th>\n",
       "      <th>LowQualFinSF</th>\n",
       "      <th>GrLivArea</th>\n",
       "      <th>BsmtFullBath</th>\n",
       "      <th>BsmtHalfBath</th>\n",
       "      <th>FullBath</th>\n",
       "      <th>HalfBath</th>\n",
       "      <th>BedroomAbvGr</th>\n",
       "      <th>KitchenAbvGr</th>\n",
       "      <th>KitchenQual</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>Functional</th>\n",
       "      <th>Fireplaces</th>\n",
       "      <th>GarageCars</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>PavedDrive</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>Clusters</th>\n",
       "      <th>NewClusters</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2003</td>\n",
       "      <td>2003</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>196.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>706</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>856</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>856</td>\n",
       "      <td>854</td>\n",
       "      <td>0</td>\n",
       "      <td>1710</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>8</td>\n",
       "      <td>Typ</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>548</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Veenker</td>\n",
       "      <td>Feedr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>1Story</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>1976</td>\n",
       "      <td>1976</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>MetalSd</td>\n",
       "      <td>MetalSd</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>CBlock</td>\n",
       "      <td>978</td>\n",
       "      <td>0</td>\n",
       "      <td>284</td>\n",
       "      <td>1262</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>1262</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1262</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>6</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>460</td>\n",
       "      <td>Y</td>\n",
       "      <td>298</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2001</td>\n",
       "      <td>2002</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>162.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>486</td>\n",
       "      <td>0</td>\n",
       "      <td>434</td>\n",
       "      <td>920</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>920</td>\n",
       "      <td>866</td>\n",
       "      <td>0</td>\n",
       "      <td>1786</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>6</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>608</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>9550</td>\n",
       "      <td>Pave</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Crawfor</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1915</td>\n",
       "      <td>1970</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Wd Sdng</td>\n",
       "      <td>Wd Shng</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>BrkTil</td>\n",
       "      <td>216</td>\n",
       "      <td>0</td>\n",
       "      <td>540</td>\n",
       "      <td>756</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>961</td>\n",
       "      <td>756</td>\n",
       "      <td>0</td>\n",
       "      <td>1717</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>7</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>642</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>272</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>140000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>NoRidge</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>2000</td>\n",
       "      <td>2000</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>350.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>655</td>\n",
       "      <td>0</td>\n",
       "      <td>490</td>\n",
       "      <td>1145</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>1145</td>\n",
       "      <td>1053</td>\n",
       "      <td>0</td>\n",
       "      <td>2198</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>9</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>836</td>\n",
       "      <td>Y</td>\n",
       "      <td>192</td>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>250000</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MSSubClass MSZoning  LotArea Street LotShape LandContour Utilities  \\\n",
       "0          60       RL     8450   Pave      Reg         Lvl    AllPub   \n",
       "1          20       RL     9600   Pave      Reg         Lvl    AllPub   \n",
       "2          60       RL    11250   Pave      IR1         Lvl    AllPub   \n",
       "3          70       RL     9550   Pave      IR1         Lvl    AllPub   \n",
       "4          60       RL    14260   Pave      IR1         Lvl    AllPub   \n",
       "\n",
       "  LotConfig LandSlope Neighborhood Condition1 Condition2 BldgType HouseStyle  \\\n",
       "0    Inside       Gtl      CollgCr       Norm       Norm     1Fam     2Story   \n",
       "1       FR2       Gtl      Veenker      Feedr       Norm     1Fam     1Story   \n",
       "2    Inside       Gtl      CollgCr       Norm       Norm     1Fam     2Story   \n",
       "3    Corner       Gtl      Crawfor       Norm       Norm     1Fam     2Story   \n",
       "4       FR2       Gtl      NoRidge       Norm       Norm     1Fam     2Story   \n",
       "\n",
       "   OverallQual  OverallCond  YearBuilt  YearRemodAdd RoofStyle RoofMatl  \\\n",
       "0            7            5       2003          2003     Gable  CompShg   \n",
       "1            6            8       1976          1976     Gable  CompShg   \n",
       "2            7            5       2001          2002     Gable  CompShg   \n",
       "3            7            5       1915          1970     Gable  CompShg   \n",
       "4            8            5       2000          2000     Gable  CompShg   \n",
       "\n",
       "  Exterior1st Exterior2nd MasVnrType  MasVnrArea ExterQual ExterCond  \\\n",
       "0     VinylSd     VinylSd    BrkFace       196.0        Gd        TA   \n",
       "1     MetalSd     MetalSd       None         0.0        TA        TA   \n",
       "2     VinylSd     VinylSd    BrkFace       162.0        Gd        TA   \n",
       "3     Wd Sdng     Wd Shng       None         0.0        TA        TA   \n",
       "4     VinylSd     VinylSd    BrkFace       350.0        Gd        TA   \n",
       "\n",
       "  Foundation  BsmtFinSF1  BsmtFinSF2  BsmtUnfSF  TotalBsmtSF Heating  \\\n",
       "0      PConc         706           0        150          856    GasA   \n",
       "1     CBlock         978           0        284         1262    GasA   \n",
       "2      PConc         486           0        434          920    GasA   \n",
       "3     BrkTil         216           0        540          756    GasA   \n",
       "4      PConc         655           0        490         1145    GasA   \n",
       "\n",
       "  HeatingQC CentralAir Electrical  1stFlrSF  2ndFlrSF  LowQualFinSF  \\\n",
       "0        Ex          Y      SBrkr       856       854             0   \n",
       "1        Ex          Y      SBrkr      1262         0             0   \n",
       "2        Ex          Y      SBrkr       920       866             0   \n",
       "3        Gd          Y      SBrkr       961       756             0   \n",
       "4        Ex          Y      SBrkr      1145      1053             0   \n",
       "\n",
       "   GrLivArea  BsmtFullBath  BsmtHalfBath  FullBath  HalfBath  BedroomAbvGr  \\\n",
       "0       1710             1             0         2         1             3   \n",
       "1       1262             0             1         2         0             3   \n",
       "2       1786             1             0         2         1             3   \n",
       "3       1717             1             0         1         0             3   \n",
       "4       2198             1             0         2         1             4   \n",
       "\n",
       "   KitchenAbvGr KitchenQual  TotRmsAbvGrd Functional  Fireplaces  GarageCars  \\\n",
       "0             1          Gd             8        Typ           0           2   \n",
       "1             1          TA             6        Typ           1           2   \n",
       "2             1          Gd             6        Typ           1           2   \n",
       "3             1          Gd             7        Typ           1           3   \n",
       "4             1          Gd             9        Typ           1           3   \n",
       "\n",
       "   GarageArea PavedDrive  WoodDeckSF  OpenPorchSF  EnclosedPorch  3SsnPorch  \\\n",
       "0         548          Y           0           61              0          0   \n",
       "1         460          Y         298            0              0          0   \n",
       "2         608          Y           0           42              0          0   \n",
       "3         642          Y           0           35            272          0   \n",
       "4         836          Y         192           84              0          0   \n",
       "\n",
       "   ScreenPorch  PoolArea  MiscVal  MoSold  YrSold SaleType SaleCondition  \\\n",
       "0            0         0        0       2    2008       WD        Normal   \n",
       "1            0         0        0       5    2007       WD        Normal   \n",
       "2            0         0        0       9    2008       WD        Normal   \n",
       "3            0         0        0       2    2006       WD       Abnorml   \n",
       "4            0         0        0      12    2008       WD        Normal   \n",
       "\n",
       "   SalePrice Clusters  NewClusters  \n",
       "0     208500        4            5  \n",
       "1     181500        4            3  \n",
       "2     223500        4            5  \n",
       "3     140000        2            2  \n",
       "4     250000        5            5  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clustering_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6. Calculating the normalized mutual information (NMI)**\n",
    "* Compute the normalized mutual information (NMI) between your clustering results and the binned categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The NMI score between the clustering results and the binned categories is: 0.2458\n"
     ]
    }
   ],
   "source": [
    "print(\"The NMI score between the clustering results and the binned categories is: {:.4}\".format(normalized_mutual_info_score(clustering_df['Clusters'],clustering_df['NewClusters'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Clustering Analysis Summary**:  \n",
    "We binned the response variable 'SalePrice' into 5 bins which contain almost same number of observations.  \n",
    "Our NMI score is very low i.e. 0.24 which clearly indicates that the cluster assignment was only partially correct.  \n",
    "Since, the observations got assigned to clusters disproportionately and cluster 3 (620) and cluster 5 (439) got the major chunk of the observations the low NMI score is understandable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}