K-Nearest-neighbors-Clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<h1><center>K-Nearest Neighbors</center></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In this Lab you will load a customer dataset, fit the data, and use K-Nearest Neighbors to predict a data point. But what is **K-Nearest Neighbors**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**K-Nearest Neighbors** is an algorithm for supervised learning. Where the data is 'trained' with data points corresponding to their classification. Once a point is to be predicted, it takes into account the 'K' nearest points to it to determine it's classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Here's an visualization of the K-Nearest Neighbors algorithm.\n",
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/mgkn92xck0z05v7yjq8pqziukxvc2461.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In this case, we have data points of Class A and B. We want to predict what the star (test data point) is. If we consider a k value of 3 (3 nearest data points) we will obtain a prediction of Class B. Yet if we consider a k value of 6, we will obtain a prediction of Class A."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In this sense, it is important to consider the value of k. But hopefully from this diagram, you should get a sense of what the K-Nearest Neighbors algorithm is. It considers the 'K' Nearest Neighbors (points) when it predicts the classification of the test point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Table of contents</h1>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "    <ol>\n",
    "        <li><a href=\"#about_dataset\">About the dataset</a></li>\n",
    "        <li><a href=\"#visualization_analysis\">Data Visualization and Analysis</a></li>\n",
    "        <li><a href=\"#classification\">Classification</a></li>\n",
    "    </ol>\n",
    "</div>\n",
    "<br>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Lets load required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import itertools\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.ticker import NullFormatter\n",
    "import pandas as pd\n",
    "import matplotlib.ticker as ticker\n",
    "from sklearn import preprocessing\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<div id=\"about_dataset\">\n",
    "    <h2>About the dataset</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Imagine a telecommunications provider has segmented its customer base by service usage patterns, categorizing the customers into four groups. If demographic data can be used to predict group membership, the company can customize offers for individual prospective customers. It is a classification problem. That is, given the dataset,  with predefined labels, we need to build a model to be used to predict class of a new or unknown case. \n",
    "\n",
    "The example focuses on using demographic data, such as region, age, and marital, to predict usage patterns. \n",
    "\n",
    "The target field, called __custcat__, has four possible values that correspond to the four customer groups, as follows:\n",
    "  1- Basic Service\n",
    "  2- E-Service\n",
    "  3- Plus Service\n",
    "  4- Total Service\n",
    "\n",
    "Our objective is to build a classifier, to predict the class of unknown cases. We will use a specific type of classification called K nearest neighbour.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Lets download the dataset. To download the data, we will use !wget to download it from IBM Object Storage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-09-28 23:06:27--  https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/teleCust1000t.csv\n",
      "Resolving s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)... 67.228.254.193\n",
      "Connecting to s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)|67.228.254.193|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 37048 (36K) [text/csv]\n",
      "Saving to: ‘teleCust1000t.csv’\n",
      "\n",
      "teleCust1000t.csv   100%[===================>]  36.18K   118KB/s    in 0.3s    \n",
      "\n",
      "2019-09-28 23:06:29 (118 KB/s) - ‘teleCust1000t.csv’ saved [37048/37048]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -O teleCust1000t.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/teleCust1000t.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Load Data From CSV File  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>region</th>\n",
       "      <th>tenure</th>\n",
       "      <th>age</th>\n",
       "      <th>marital</th>\n",
       "      <th>address</th>\n",
       "      <th>income</th>\n",
       "      <th>ed</th>\n",
       "      <th>employ</th>\n",
       "      <th>retire</th>\n",
       "      <th>gender</th>\n",
       "      <th>reside</th>\n",
       "      <th>custcat</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>44</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>64.0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>136.0</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>68</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>116.0</td>\n",
       "      <td>1</td>\n",
       "      <td>29</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>33</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>33.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>23</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>30.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   region  tenure  age  marital  address  income  ed  employ  retire  gender  \\\n",
       "0       2      13   44        1        9    64.0   4       5     0.0       0   \n",
       "1       3      11   33        1        7   136.0   5       5     0.0       0   \n",
       "2       3      68   52        1       24   116.0   1      29     0.0       1   \n",
       "3       2      33   33        0       12    33.0   2       0     0.0       1   \n",
       "4       2      23   30        1        9    30.0   1       2     0.0       0   \n",
       "\n",
       "   reside  custcat  \n",
       "0       2        1  \n",
       "1       6        4  \n",
       "2       2        3  \n",
       "3       1        1  \n",
       "4       4        3  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('teleCust1000t.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<div id=\"visualization_analysis\">\n",
    "    <h2>Data Visualization and Analysis</h2> \n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Let’s see how many of each class is in our data set "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    281\n",
       "1    266\n",
       "4    236\n",
       "2    217\n",
       "Name: custcat, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['custcat'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### 281 Plus Service, 266 Basic-service, 236 Total Service, and 217 E-Service customers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can easily explore your data using visualization techniques:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7f138810e4e0>]],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEICAYAAAC9E5gJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAASZUlEQVR4nO3de7CcdX3H8fdH4q1ECYjN0CR6sKaOVKaKp4rjpSdCLRcrtJUWh9HU4qROtaPVTo3aae1MpxPaoV5Gq5MWxuBYg/UyZLCOZdCtMi0qUUQwtQQIGkmhykUPVlvw2z/2d3SJJ8nZk73C+zWzs8/ze3777Pf5seznPL99dpOqQpL00PawcRcgSRo/w0CSZBhIkgwDSRKGgSQJw0CShGGgB5kkNySZG3cd0rSJ3zOQJHlmIEkyDPTgkmRPklOTvC3Jh5NckuR7bfpotqffuiQfS/LfSb6T5N2t/WFJ/jTJrUnuaI8/qm2bSVJJXpnkm0nuSvLqJL+c5Lokdy/sp+d5fi/Jrtb3U0meONoRkZbGMNCD2UuA7cAqYAew8IZ/BHA5cCswA6xp/QB+t902AE8CVi48rsezgfXA7wDvAN4KnAr8IvDbSX6lPc/ZwFuA3wQeD3wO+NCAj1EaCD8z0INKkj3Aq4DnAc+rqlNb+wnAzqp6dJLn0A2H46rqvv0efyXw0ar6u7b+FOB64NHAWuAWYG1Vfatt/w7wB1V1aVv/KPC5qnpHkk8CH6mqi9q2hwHzwFOr6tZhjoPUL88M9GD2Xz3L3wcelWQFsA64df8gaH6O7hnDgluBFcDqnrbbe5b/Z5H1lW35icA72/TR3cCdQOieiUgTxTDQQ9E3gSe0YNjfbXTfxBc8AbiPB77h9/M8v19Vq3puj66qf1vGvqShMgz0UPQFYB+wJcmRSR6V5Llt24eAP0pyfJKVwF8Blx7gLOJQ3ge8OckvAiQ5Ksk5gzgAadAMAz3kVNX9wK8DTwa+Aeyl+2EwwMXAB4DP0v184AfAHy7zeT4OXABsT/Jdup89nH5YxUtD4gfIkiTPDCRJhoEkCcNAkoRhIEmi+2WasTv22GNrZmam78fde++9HHnkkYMvaMimse5prBmse9Sse7R27tz57ap6/CD2NRFhMDMzwzXXXNP34zqdDnNzc4MvaMimse5prBmse9Sse7SSDOxnTZwmkiQZBpIkw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kSE/IN5MMxs/kTi7bv2XLmiCuRpOnlmYEkyTCQJBkGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJNFHGCQ5IsmXk1ze1o9P8vkkNya5NMkjWvsj2/rutn1mOKVLkgalnzOD1wG7etYvAN5eVeuBu4DzW/v5wF1V9WTg7a2fJGmCLSkMkqwFzgT+oa0HeCHwkdZlG3B2Wz6rrdO2n9L6S5Im1FLPDN4B/Anwo7b+OODuqrqvre8F1rTlNcA3Adr2e1p/SdKEWnGoDkleDNxRVTuTzC00L9K1lrCtd7+bgE0Aq1evptPpLKXeB5ifn+eNJ96/6Lbl7G9U5ufnJ7q+xUxjzWDdo2bd0+uQYQA8F3hJkjOARwGPpXumsCrJivbX/1rgttZ/L7AO2JtkBXAUcOf+O62qrcBWgNnZ2Zqbm+u7+E6nw4VX3bvotj3n9b+/Uel0OizneMdpGmsG6x41655eh5wmqqo3V9XaqpoBzgU+XVXnAZ8BXtq6bQQua8s72jpt+6er6qfODCRJk+NwvmfwJuANSXbT/UzgotZ+EfC41v4GYPPhlShJGralTBP9WFV1gE5bvhl41iJ9fgCcM4DaJEkj4jeQJUmGgSTJMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJJYQBkkeleQLSb6S5IYkf9Haj0/y+SQ3Jrk0ySNa+yPb+u62fWa4hyBJOlxLOTP4IfDCqvol4OnAaUlOBi4A3l5V64G7gPNb//OBu6rqycDbWz9J0gQ7ZBhU13xbfXi7FfBC4COtfRtwdls+q63Ttp+SJAOrWJI0cKmqQ3dKjgB2Ak8G3gP8DXB1++ufJOuAT1bV05JcD5xWVXvbtpuAZ1fVt/fb5yZgE8Dq1aufuX379r6Ln5+f55Z77l9024lrjup7f6MyPz/PypUrx11GX6axZrDuUbPu0dqwYcPOqpodxL5WLKVTVd0PPD3JKuDjwFMX69buFzsL+KnEqaqtwFaA2dnZmpubW0opD9DpdLjwqnsX3bbnvP73NyqdToflHO84TWPNYN2jZt3Tq6+riarqbqADnAysSrIQJmuB29ryXmAdQNt+FHDnIIqVJA3HUq4menw7IyDJo4FTgV3AZ4CXtm4bgcva8o62Ttv+6VrKXJQkaWyWMk10HLCtfW7wMODDVXV5kq8B25P8JfBl4KLW/yLgA0l20z0jOHcIdUuSBuiQYVBV1wHPWKT9ZuBZi7T/ADhnINVJkkbCbyBLkgwDSZJhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJLEEn/CehrNbP7Eou17tpw54kokafJ5ZiBJMgwkSYaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJIklhEGSdUk+k2RXkhuSvK61H5PkiiQ3tvujW3uSvCvJ7iTXJTlp2AchSTo8SzkzuA94Y1U9FTgZeE2SE4DNwJVVtR64sq0DnA6sb7dNwHsHXrUkaaAOGQZVta+qvtSWvwfsAtYAZwHbWrdtwNlt+Szgkuq6GliV5LiBVy5JGphU1dI7JzPAZ4GnAd+oqlU92+6qqqOTXA5sqaqrWvuVwJuq6pr99rWJ7pkDq1evfub27dv7Ln5+fp5b7rm/r8ecuOaovp9n0Obn51m5cuW4y+jLNNYM1j1q1j1aGzZs2FlVs4PY14qldkyyEvgo8Pqq+m6SA3ZdpO2nEqeqtgJbAWZnZ2tubm6ppfxYp9Phwqvu7esxe87r/3kGrdPpsJzjHadprBmse9Sse3ot6WqiJA+nGwQfrKqPtebbF6Z/2v0drX0vsK7n4WuB2wZTriRpGJZyNVGAi4BdVfW3PZt2ABvb8kbgsp72V7Srik4G7qmqfQOsWZI0YEuZJnou8HLgq0mubW1vAbYAH05yPvAN4Jy27Z+BM4DdwPeBVw60YknSwB0yDNoHwQf6gOCURfoX8JrDrEuSNEJ+A1mSZBhIkgwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkScCKcRcwajObP3HAbXu2nDnCSiRpcnhmIEkyDCRJhoEkiSWEQZKLk9yR5PqetmOSXJHkxnZ/dGtPkncl2Z3kuiQnDbN4SdJgLOXM4P3Aafu1bQaurKr1wJVtHeB0YH27bQLeO5gyJUnDdMgwqKrPAnfu13wWsK0tbwPO7mm/pLquBlYlOW5QxUqShiNVdehOyQxweVU9ra3fXVWrerbfVVVHJ7kc2FJVV7X2K4E3VdU1i+xzE92zB1avXv3M7du39138/Pw8t9xzf9+PO5AT1xw1sH0dzPz8PCtXrhzJcw3KNNYM1j1q1j1aGzZs2FlVs4PY16C/Z5BF2hZNm6raCmwFmJ2drbm5ub6frNPpcOFV9/b9uAPZc17/NSxHp9NhOcc7TtNYM1j3qFn39Fru1US3L0z/tPs7WvteYF1Pv7XAbcsvT5I0CssNgx3Axra8Ebisp/0V7aqik4F7qmrfYdYoSRqyQ04TJfkQMAccm2Qv8OfAFuDDSc4HvgGc07r/M3AGsBv4PvDKIdQsSRqwQ4ZBVb3sAJtOWaRvAa853KIkSaPlN5AlSYaBJMkwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJLEEv6ls4eSmc2fWLR9z5YzR1yJJI2WZwaSJMNAkmQYSJIwDCRJGAaSJLyaaEm8ykjSg51nBpIkw0CSZBhIkjAMJEkYBpIkDANJEoaBJAm/Z3BY/P6BpAcLzwwkSYaBJMlpopFamFZ644n38bs9U0xOK0kaN8NgCA70WYIkTSqniSRJhoEkaUjTRElOA94JHAH8Q1VtGcbzPFgMalrJzx4kLdfAwyDJEcB7gF8F9gJfTLKjqr426OfS0vh9CEmHMowzg2cBu6vqZoAk24GzAMNgyPo9wzhY/wMFRb/B0m9Nw97PKExarf4xMBkm/b9DqmqwO0xeCpxWVa9q6y8Hnl1Vr92v3yZgU1t9CvD1ZTzdscC3D6PccZnGuqexZrDuUbPu0XpKVT1mEDsaxplBFmn7qcSpqq3A1sN6ouSaqpo9nH2MwzTWPY01g3WPmnWPVpJrBrWvYVxNtBdY17O+FrhtCM8jSRqQYYTBF4H1SY5P8gjgXGDHEJ5HkjQgA58mqqr7krwW+BTdS0svrqobBv08zWFNM43RNNY9jTWDdY+adY/WwOoe+AfIkqTp4zeQJUmGgSRpSsMgyWlJvp5kd5LN466nV5J1ST6TZFeSG5K8rrW/Lcm3klzbbmf0PObN7Vi+nuTXxlj7niRfbfVd09qOSXJFkhvb/dGtPUne1eq+LslJY6r5KT1jem2S7yZ5/SSOd5KLk9yR5Pqetr7HN8nG1v/GJBvHUPPfJPmPVtfHk6xq7TNJ/qdnzN/X85hnttfW7nZci12CPuy6+35NjPq95gB1X9pT854k17b2wY53VU3Vje6H0jcBTwIeAXwFOGHcdfXUdxxwUlt+DPCfwAnA24A/XqT/Ce0YHgkc347tiDHVvgc4dr+2vwY2t+XNwAVt+Qzgk3S/V3Iy8PkJGPsjgP8CnjiJ4w28ADgJuH654wscA9zc7o9uy0ePuOYXASva8gU9Nc/09ttvP18AntOO55PA6WMY675eE+N4r1ms7v22Xwj82TDGexrPDH78cxdV9b/Aws9dTISq2ldVX2rL3wN2AWsO8pCzgO1V9cOqugXYTfcYJ8VZwLa2vA04u6f9kuq6GliV5LhxFNjjFOCmqrr1IH3GNt5V9VngzkXq6Wd8fw24oqrurKq7gCuA00ZZc1X9S1Xd11avpvtdogNqdT+2qv69uu9Ul/CT4xyKA4z1gRzoNTHy95qD1d3+uv9t4EMH28dyx3saw2AN8M2e9b0c/M12bJLMAM8APt+aXttOrS9emA5gso6ngH9JsjPdnwsBWF1V+6AbdMDPtvZJqnvBuTzwf5RJH2/of3wnrf7fo/uX54Ljk3w5yb8meX5rW0O3zgXjrLmf18SkjfXzgdur6saetoGN9zSGwZJ+7mLckqwEPgq8vqq+C7wX+Hng6cA+uqd7MFnH89yqOgk4HXhNkhccpO8k1U26X3B8CfBPrWkaxvtgDlTnxNSf5K3AfcAHW9M+4AlV9QzgDcA/Jnksk1Nzv6+JSal7wct44B87Ax3vaQyDif+5iyQPpxsEH6yqjwFU1e1VdX9V/Qj4e34yNTExx1NVt7X7O4CP063x9oXpn3Z/R+s+MXU3pwNfqqrbYTrGu+l3fCei/vbB9YuB89pUBG2a5TtteSfd+fZfoFtz71TSWGpexmtiIsYaIMkK4DeBSxfaBj3e0xgGE/1zF21e7yJgV1X9bU9773z6bwALVwvsAM5N8sgkxwPr6X74M1JJjkzymIVluh8SXt/qW7hiZSNwWVveAbyiXfVyMnDPwnTHmDzgr6ZJH+8e/Y7vp4AXJTm6TXO8qLWNTLr/eNWbgJdU1fd72h+f7r9nQpIn0R3bm1vd30tycvv/4xX85DhHWXe/r4lJeq85FfiPqvrx9M/Ax3uYn4wP60b3Sov/pJuEbx13PfvV9jy6p2TXAde22xnAB4CvtvYdwHE9j3lrO5avM+SrLA5S95PoXi3xFeCGhXEFHgdcCdzY7o9p7aH7jxjd1I5rdoxj/jPAd4CjetombrzphtU+4P/o/vV2/nLGl+48/e52e+UYat5Ndy594fX9vtb3t9pr5yvAl4Bf79nPLN0335uAd9N+/WDEdff9mhj1e81idbf29wOv3q/vQMfbn6OQJE3lNJEkacAMA0mSYSBJMgwkSRgGkiQMA0kShoEkCfh/xFrTluN5AQAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.hist(column='income', bins=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Feature set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Lets define feature sets, X:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['region', 'tenure', 'age', 'marital', 'address', 'income', 'ed',\n",
       "       'employ', 'retire', 'gender', 'reside', 'custcat'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use scikit-learn library, we have to convert the Pandas data frame to a Numpy array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  2.,  13.,  44.,   1.,   9.,  64.,   4.,   5.,   0.,   0.,   2.],\n",
       "       [  3.,  11.,  33.,   1.,   7., 136.,   5.,   5.,   0.,   0.,   6.],\n",
       "       [  3.,  68.,  52.,   1.,  24., 116.,   1.,  29.,   0.,   1.,   2.],\n",
       "       [  2.,  33.,  33.,   0.,  12.,  33.,   2.,   0.,   0.,   1.,   1.],\n",
       "       [  2.,  23.,  30.,   1.,   9.,  30.,   1.,   2.,   0.,   0.,   4.]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df[['region', 'tenure','age', 'marital', 'address', 'income', 'ed', 'employ','retire', 'gender', 'reside']] .values  #.astype(float)\n",
    "X[0:5]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 4, 3, 1, 3])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = df['custcat'].values\n",
    "y[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Normalize Data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Data Standardization give data zero mean and unit variance, it is good practice, especially for algorithms such as KNN which is based on distance of cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.02696767, -1.055125  ,  0.18450456,  1.0100505 , -0.25303431,\n",
       "        -0.12650641,  1.0877526 , -0.5941226 , -0.22207644, -1.03459817,\n",
       "        -0.23065004],\n",
       "       [ 1.19883553, -1.14880563, -0.69181243,  1.0100505 , -0.4514148 ,\n",
       "         0.54644972,  1.9062271 , -0.5941226 , -0.22207644, -1.03459817,\n",
       "         2.55666158],\n",
       "       [ 1.19883553,  1.52109247,  0.82182601,  1.0100505 ,  1.23481934,\n",
       "         0.35951747, -1.36767088,  1.78752803, -0.22207644,  0.96655883,\n",
       "        -0.23065004],\n",
       "       [-0.02696767, -0.11831864, -0.69181243, -0.9900495 ,  0.04453642,\n",
       "        -0.41625141, -0.54919639, -1.09029981, -0.22207644,  0.96655883,\n",
       "        -0.92747794],\n",
       "       [-0.02696767, -0.58672182, -0.93080797,  1.0100505 , -0.25303431,\n",
       "        -0.44429125, -1.36767088, -0.89182893, -0.22207644, -1.03459817,\n",
       "         1.16300577]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = preprocessing.StandardScaler().fit(X).transform(X.astype(float))\n",
    "X[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Train Test Split  \n",
    "Out of Sample Accuracy is the percentage of correct predictions that the model makes on data that that the model has NOT been trained on. Doing a train and test on the same dataset will most likely have low out-of-sample accuracy, due to the likelihood of being over-fit.\n",
    "\n",
    "It is important that our models have a high, out-of-sample accuracy, because the purpose of any model, of course, is to make correct predictions on unknown data. So how can we improve out-of-sample accuracy? One way is to use an evaluation approach called Train/Test Split.\n",
    "Train/Test Split involves splitting the dataset into training and testing sets respectively, which are mutually exclusive. After which, you train with the training set and test with the testing set. \n",
    "\n",
    "This will provide a more accurate evaluation on out-of-sample accuracy because the testing dataset is not part of the dataset that have been used to train the data. It is more realistic for real world problems.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set: (800, 11) (800,)\n",
      "Test set: (200, 11) (200,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)\n",
    "print ('Train set:', X_train.shape,  y_train.shape)\n",
    "print ('Test set:', X_test.shape,  y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<div id=\"classification\">\n",
    "    <h2>Classification</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<h3>K nearest neighbor (KNN)</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Import library "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Classifier implementing the k-nearest neighbors vote."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Training\n",
    "\n",
    "Lets start the algorithm with k=4 for now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "                     metric_params=None, n_jobs=None, n_neighbors=4, p=2,\n",
       "                     weights='uniform')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k = 4\n",
    "#Train Model and Predict  \n",
    "neigh = KNeighborsClassifier(n_neighbors = k).fit(X_train,y_train)\n",
    "neigh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Predicting\n",
    "we can use the model to predict the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 3, 2, 4])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yhat = neigh.predict(X_test)\n",
    "yhat[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Accuracy evaluation\n",
    "In multilabel classification, __accuracy classification score__ is a function that computes subset accuracy. This function is equal to the jaccard_similarity_score function. Essentially, it calculates how closely the actual labels and predicted labels are matched in the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set Accuracy:  0.5475\n",
      "Test set Accuracy:  0.32\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "print(\"Train set Accuracy: \", metrics.accuracy_score(y_train, neigh.predict(X_train)))\n",
    "print(\"Test set Accuracy: \", metrics.accuracy_score(y_test, yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### What about other K?\n",
    "K in KNN, is the number of nearest neighbors to examine. It is supposed to be specified by the User. So, how can we choose right value for K?\n",
    "The general solution is to reserve a part of your data for testing the accuracy of the model. Then chose k =1, use the training part for modeling, and calculate the accuracy of prediction using all samples in your test set. Repeat this process, increasing the k, and see which k is the best for your model.\n",
    "\n",
    "We can calculate the accuracy of KNN for different Ks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.3  , 0.29 , 0.315, 0.32 , 0.315, 0.31 , 0.335, 0.325, 0.34 ])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Ks = 10\n",
    "mean_acc = np.zeros((Ks-1))\n",
    "std_acc = np.zeros((Ks-1))\n",
    "ConfustionMx = [];\n",
    "for n in range(1,Ks):\n",
    "    \n",
    "    #Train Model and Predict  \n",
    "    neigh = KNeighborsClassifier(n_neighbors = n).fit(X_train,y_train)\n",
    "    yhat=neigh.predict(X_test)\n",
    "    mean_acc[n-1] = metrics.accuracy_score(y_test, yhat)\n",
    "\n",
    "    \n",
    "    std_acc[n-1]=np.std(yhat==y_test)/np.sqrt(yhat.shape[0])\n",
    "\n",
    "mean_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Plot  model accuracy  for Different number of Neighbors "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd5hTZfbA8e9JMplGLyoKAiooqJRhhF0LNlBcWYoNLGtFcFcF13Vd/enuqttc1q5YULAiCCqIFRRBxQIMA4IgSLGAFJEOU5Oc3x83M4ZhBjJMMrnJnM/z8JDc3Nx7Zhhy5m3nFVXFGGOMcRtPogMwxhhjKmMJyhhjjCtZgjLGGONKlqCMMca4kiUoY4wxruRLdACx0qxZM23Tpk2iwzDGGFNN8+fP/1lVm1c8njIJqk2bNuTl5SU6DGOMMdUkIt9Xdty6+IwxxriSJShjjDGuZAnKGGOMK6XMGJQxxsRaaWkpa9eupaioKNGhpISMjAxatmxJWlpaVOdbgjLGmCqsXbuW+vXr06ZNG0Qk0eEkNVVl8+bNrF27lrZt20b1HuviM8aYKhQVFdG0aVNLTjEgIjRt2rRarVFLUMYYsw+WnGKnut9LS1DGGGNcyRKUMca43OTJkxERli1bluhQapUlKGOMcbnx48dz8sknM2HChLjeJxgMRn2uqhIMheIYjSUoY4xxtV27dvHpp58yZsyYvRLUyJEjOf744+ncuTO33XYbACtXrqRXr1507tyZnJwcVq1axaxZs+jbt2/5+2644Qaee+45wCkTd88993DyySczadIknn76aU444QQ6d+7M+eefT0FBAQAbN25k4MCBdO7cmc6dO/PJ7E/561//xsMPP1x+3TvuuINHHnkkZl+7TTM3xpgo3PTeTSzcsDCm1+xySBce6vPQPs+ZMmUKffr0oX379jRp0oT8/HxycnJ49913mTJlCnPmzCErK4stW7YAcOmll3LbbbcxcOBAioqKCIVCrFmzZp/3yMjIYPbs2QBs3ryZa6+9FoA777yTMWPGcOONNzJ8+HB69uzJq6+9RmkgyK5du2hx6KEMuvACRowYQSgUYsKECcydOzcG3xmHJShjjHGx8ePHc9NNNwEwePBgxo8fT05ODh988AFXXXUVWVlZADRp0oSdO3fy448/MnDgQMBJPNEYNGhQ+eOvvvqKO++8k23btrFr1y7OPvtsAD788EPGPPscIQWv10vDhg1p2LAhTZs2ZcGCBWzcuJGuXbvStGnTmH3tlqCMMSYK+2vpxMPmzZv58MMP+eqrrxARgsEgIsLIkSNR1b2mbatqpdfx+XyEIsaLKq5Fys7OLn985ZVXMmXKFDp37sxzzz3HrFmz9jnWNGTIEJ577jk2bNjA1VdffSBfZpVsDMoYY1zq1Vdf5fLLL+f777/nu+++Y82aNbRt25bZs2dz1llnMXbs2PIxoi1bttCgQQNatmzJlClTACguLqagoIDWrVuzdOlSiouL2b59OzNmzKjynjt37qRFixaUlpby0rhxhFQJKZx+xhk89eSTgDOZYseOHQAMHDiQ9957j3nz5pW3tmLFEpQxxrjU+PHjy7vrypx//vm8/PLL9OnTh379+pGbm0uXLl247777AHjxxRd55JFH6NSpEyeeeCIbNmygVatWXHTRRXTq1IlLL72Url27VnnPf/zjH/To0YNevXtz9NFHlx9/4MGHmDVrJl27dKZH9xNYumQJAH6/n9NPP52LLroIr9cb069fqmoSJpvc3Fy1DQuNMbH09ddf06FDh0SHUauc6eNKtJnBK5CTk8OkSZNo167dfs+v7HsqIvNVNbfiudaCMsYYA0AwpASqkZyWLl3KUUcdxZlnnhlVcqoumyRhjKlTSgIh0rxiNfYiVLfVVKZjx46sXr06LjGBJShjTB1SVBpke2EpIpDl95GV5sXjqduJKhRSgi4d6rEEZYypE0oCIXYUlgKgCruLAxQUB8jwe8lK8+Lz1q0RjwNtNdUmS1DGmJQXCIbYVliy14exAoUlQQpLgqT7PGT5ffh9qZ+o3NxqimQJyhiT0kIhZWtBKfv7PC4OhCgOlJDm9ZDl95KRFtsp026QDK2mSKn/q4Ixps5SVbYWlBCqRmuhNBhie2EpP+8qRlX3qM6wcUdRTP/Eyueff15eP68qjz/+BMd36kS3bjmc1rMnS5curfZ9vvvuO8aPf7nK10877TRiudzHEpQxJmVtKyglEDqw9kIw5FRQCITCrY4Ed4nNmjWLK6+8stLX3nvvPfr06VPpa6pKIBRi0MUXs2Dhl+TNz+dPf76FW2/5U7Vj+P6773hl/Phqv+9AWYIyxqSk7YWllARjs19RSDWcqEIJT1SVmTFjBr169drreFncqtCgQYPy47t37y6fZv/Qgw9y7ZBrAFi8eDFdOneioKCAjz/6iNxuOeR2y+GE3G7s3LmTO/7v/5g9eza53XJ4+KGHKCwsZPDgwXTq1IlBgwZRWFgY06/LxqCMMSlnV3GAotLoN9+LVkidD31B8XgEjwvWUv3888+kpaXRsGHD8mOqWl5DL9ITjz/Oww89SElJCdPe/wCA4SNG0OuMM5gyZTL3/uc/PP74E2RlZfHAA/fzyCOPcuJJJ7Fr1y4yMjL417//zYMP3M+UqW8CTnLLyspi0aJFLFq0iJycnJh+bdaCMsaklKLSILuLA3G9hxKuuhAMETrALsRo9ejRgy5dujBkyBCmTp1Kly5d6NKlC9OmTQNg+vTpnHXWWeXnl7WaKgvr93/4A8u+WcG//vMf/vPvfwHg8Xh4ZuxYrrriCnqe0pMTTzoJgBNPPIk/33ILjz36KNu2bcPn27s988knH3PZZZcB0KlTJzp16hTTr90SlDEmZRQHguVrnWqDAkFVSsOJKh7df3PmzGHhwoU888wz9OvXj4ULF7Jw4cLyyuHvvvsuffr0QVW58sqr6Nq1K/36nrvPaw4aNJipb7xR/nzlihXUq1ePdevXlR+79S9/4cnRoyksLOSUk05k2bJllV4rnhU5LEEZY1JCaTDE9oLShE2hDpaPU9XehApVZdGiRXTq3JlgSHl6zBjy5ucz9a239zp3xYoV5Y/fefttjgrXztu+fTs33/xHZsycxZbNm3nttVcBWLVqFccffzx/vvVWcrp1Y/nyZdSvX5+dO3eVX+eUU3oybtw4wNnocNGiRTH9+mwMyhiT9IIhZVstJKeDG0S3Qy2AR8Aj8a35N3/+fLp06Vppd15FTzw+ihkzZpCWlkbjRo0ZM/ZZAG75081cd93vad++PU89/Qxn9TqTU07pyaOPPMysWbPwer106NCBPn3OwePx4PP56JbTlcsvv4Jh113H0CHX0KlTJ7p06UL37t1j+vXFdbsNEekDPAx4gWdU9d4Kr18HXA8EgV3AUFVdGn6tE/AU0AAIASeoapULB2y7DWPqJlVly+6SA55Ovi+b1qzm6GOOqdE1BOIyoUJV+cc//skRRx3JoEGDY3rt6kirZomo6my3EbcWlIh4gVFAb2AtME9EppYloLCXVfXJ8Pn9gAeAPiLiA14CfqeqX4pIU6D2OpaNMUlBVWu01qk2lE2oCKF4RGJSnNZZo6XcfscdNQ/QxeI5BtUdWKmqq1W1BJgA9I88QVV3RDzNhvIW+lnAIlX9MnzeZlWN/ZxRY0xS21EUiNlap3iLxYQK1fDMQReuxYqHeCaow4A1Ec/Xho/tQUSuF5FVwEhgePhwe0BFZJqI5IvIrZXdQESGikieiORt2rQpxuEbY9wsXmudKorHMMiBTKgIVXMzQTeq7vcyngmqsnbsXtGp6ihVPRL4C3Bn+LAPOBm4NPz3QBE5s5L3jlbVXFXNbd68eewiN8a4WmFJ/Nc6Afj86WzZvCVus/KiqVBR1mpKhurj+6KqbN68mYyM6CeaxHMW31qgVcTzlsC6Ks4FpwvwiYj3fqSqPwOIyDtADjAjDnEaY5JIcSDIjqLaGZJu0PRgtm3eyM8/104PjQAiv6wt0kqqQbiNtxpjahkZGbRs2TLq8+OZoOYB7USkLfAjMBi4JPIEEWmnqmWT888Fyh5PA24VkSygBDgVeDCOsRpjkkDZWqfa4vX5aHzwXiMTcef3evCIUBRw99C7CBxUP/oWUXXFLUGpakBEbsBJNl5grKouEZF7gDxVnQrcICK9cGbobQWuCL93q4g8gJPkFHhHVfdeeWaMqTNqa62TGyTLxI94i+s6qNpk66CMSV2hkLKloISg2/u76phYtaCqWgdlpY6MMa6mqmwvLLXkVAdZgjLGuNqOwuRZ62RiyxKUMca1dhaVun6igIkfS1DGGFcqKAlQUGLJqS6zBGWMcZ2i0iA7i+K/ENe4myUoY4yrlAZDtbrpoHEvS1DGGNcIhpStBSV1Yq2T2T/bsNDUKaGQUhoKURpUSgNOfbM0j4c0n5Dm9VR7bxsTO6Gy5GTZyYRZgjIpLRB0klFJMERpMFTpWppgKEjZcIfgbMCW5vOQ5hXSPJ6Y7N9j9k1V2WZrnUwFlqBMylBVp2UUTkYlwVC1fxtXnDIzketuvB6ndeX3OknLZ62smNtRGKDU1jqZCixBmaQVCv3SMioNOlsSxOP372BInVZWeO8hEcLJquyPlFefNtVna51MVSxBmaRR1jIqDTiJKVG7iqpCcSBEccD5jV8AXzhRlbW0rFswOrbWyeyLJSjjSqplrSNnMkNpnFpHsaCUJU8A58PWI+K0smzyRZVsrZPZH0tQxhWCIS0fNyoNhAgk+WB5SJWiQMTkCyE8W9Bpafm9njrdLVgSsLVOZv8sQZlaV3EyQ2lQE9ZdV1tU95584fMIaT5P+XhWdXYmTWaBYIhthbbWyeyfJSgTd7U1mSHZBEJKoCRIYUS3YNk4VqpOvgiFnOnkKf77iIkRS1AmLooDQYpKq157ZPYWUqU4oCk7+cLWOpnqsgRlYq4kEGJ7HdmaO54qm3yRzGuytheW2lonUy2WoExM2fhCfCXrmqwdRaXlLUNjomUJysSMU0vNxhdqU2Vrsrwum3yxuzhAoa11MgfAEpSJCVWn0Geqz8ZzO6XyyReJWpNVVBpkV7GtdTIHxhKUiYnthaVJv3YpVe21JovaKYhra51MTVmCMjVm4wvJpaqCuH6fx1lMHIPJFzYWaWLBEpSpERtfSA3BkFIY0S1Yk8kXNhZpYsUSlDlgNr6Qug60IK6NRZpYsgRlDoiNL9Qt0azJ8nrExiJNTFmCMtVm4wsGKlmTBfYzYWIqeZahG1ew8QVTFfuRMLEW1wQlIn1EZLmIrBSR2yp5/ToRWSwiC0Vktoh0rPD64SKyS0RuiWecJjo2vmCMqU1xS1Ai4gVGAecAHYGLKyYg4GVVPV5VuwAjgQcqvP4g8G68YjTVY+MLxpjaFM8WVHdgpaquVtUSYALQP/IEVd0R8TSbiF4CERkArAaWxDFGEyVb62SMiVQcKObeT+/mmfxn4naPeCaow4A1Ec/Xho/tQUSuF5FVOC2o4eFj2cBfgLv3dQMRGSoieSKSt2nTppgFbvZka52MMZG+3JjP2S+fxINz7mXB+gVxu088E1Rlq/r26h9S1VGqeiROQrozfPhu4EFV3bWvG6jqaFXNVdXc5s2b1zhgszdb62SMKeO0mu7iN+N7sq1oCy8OeI1R546K2/3iOc18LdAq4nlLYN0+zp8APBF+3AO4QERGAo2AkIgUqepjcYnUVMrWOhljyizcMJ+bpg9j2eYlXNTxMu45dSSNMxvH9Z7xTFDzgHYi0hb4ERgMXBJ5goi0U9UV4afnAisAVPWUiHPuAnZZcqpdttbJGANOq+mBOf/msXn30zzrIF7s/zq9jzinVu4dtwSlqgERuQGYBniBsaq6RETuAfJUdSpwg4j0AkqBrcAV8YrHRM/WOhljwGk1jZg+lOWblzKo4++4+9T/0igjvq2mSKIp8imUm5ureXl5iQ4j6akqW3aX2HRyY+qwiq2m+3o/Tq+2ffY6TwQOqp9R4/uJyHxVza143EodmT3YWidj6rbIVtPgYy/n7p7/pWFGo4TEYgnKlLO1TsbUXcWBYu7/4l+MynuAg7IP5qUBkyttNdUmS1AGsLVOpm4oCZYw67v3ObFVT+r56yc6HNdYsCGPEdOG8s2Wr7n42Cu4q+e9CWs1RbIEZWytk6kTQhpixLRrmbx8IvX9Dbjs+Ku5psvvadng8ESHljBFgaLyVtPB2YcwbsAUzmx7dqLDKmcJqo6ztU6mLlBV7vroL0xePpFhOcPZuHs9o/MfZXT+o/RtN5Bh3YaTc8gJiQ6zVuVvmMdN04aVt5ruPvW/NEhvmOiw9mAJqg6ztU6mrhiV9wCjFzzGtV2v566e9yIi3HnyPxmz8AnGffUsb3zzKie0+BXDug3nnCP74fV4Ex1y3BQFirjv83/y+PwHOSS7BS8PfIMz2pyV6LAqZdPM66hQSNm827bOMKlv4tJxDJ82hAFHX8jj5zyHR/as8LarZCcTlrzA0wtG8f32b2nVoDXXdr2ei4+9gvrpDRIUdXzkb5jHiGlDWbFlGZccdyV39by3Rq2meE8ztwRVB9laJ1NXfPjddC5/43x+fdgpvDRgMum+9CrPDYaCTFv9FqPzH+WLHz+lvr8Blxx3JUO6/oFWDVrXYtSxV7HVdH/vJzi9Te8aX9cSVJRikaB2FpWSkeYlzZvaGw1vKyix6eQm5eVvmMf5k/pwVJP2vH7BtGq1hhZsyGN0/qNM/eY1FOXcowZwXbfhdGvRI44Rx0f++rmMmD6UFVuWc+lxV/H3nv+J2ViTJagoxSJBbdpZTEgVv9dDVrqXdF/q9UPvKCq16eQm5a3auoLfvnIG9dLq8dbgmRyUfcgBXefHnWsYu/BJXlw8hh3F2+nWojvDcobzm6P64/O4ewi/KFDE/z7/B0/Mf4gW9Q7lvl6Px6TVFMkSVJRimaDK+DxCdrqPjLTUSFS7iwM2ndykvI271tP3ldMpKC3grcEzadvoyBpfc3fJLiYsfZGn8x/ju+2radngcK7tej2XHHulK8ep5q+fw03Th7Fiy3IuO+5q/tbz33GZoWcJKkrxSFBlPCJk+b1k+b2IVLbNlfsVlQbZbtPJTYrbUbydgZPO4tttq3j9gml0OaRbTK8fDAWZvvptnsp/lC9+nE09f30uPe5KrulyPYc3TPw4VVGgiJGf3cOT+Q/Tot6h3N/7CU5r3Stu97MEFaV4JqgyIpCZ5iXL78PrSZ5EVRIIsa3AppOb1FYcKObiyf2Yu+4zXhowOa4fzODsKjs6/1He+OZVQhriN0f157qc4eQe+qu43rcqka2m3x1/DX875d9xb91ZgopSbSSoMgKkp3nJ9nvxuXxCRSAYYktBiW2dYVJaMBTkuncu580VrzOqz1jO73Bxrd173c615eNU24u3kXPICQzLGc657QbUyjhVYaCQkZ/dw1P5j9Ci3qE80PtJTm19ZtzvC5agolabCSpSus9Dlt+H3+e+RGVrnUxdoKrcMetmxi58kr/3/A+/73ZTQuLYXbKLV5a+xNMLHuPbbas4rH4rhnT9A5ced1XcKjTkrfuCm6YPY+XWb2qt1RQp3gnKfZ+qSaY4EGJrQQlbdpdQVOqe2XGqytYCS04m9T067z7GLnyS67qNSFhyAsj21+PqLtcx+4oveb7fJA5v2Ia7P76drk8fxV9n3cL3276N2b0KA4Xc9fFt/PaVMygMFDLxvLf5X6/HXDlhoyasBRXhQFpQFXk9zoSKzLTETqiwtU6mLhi/5AX+OH0Y5x8zmEf7jNmrSkSiLdq4gNH5jzLlm0mENMQ5R/6WYTnDOeHQXx/w58O8dZ9z0/RhrNq6gsuPH8Lfev47YZXZrYsvSm5JUGVEIMvvIyvNi6eWJ1TYWidTF0xf/Q5XTb2IUw4/nRf6v4bf6090SFVav+tHnl34FC8seoZtxVvpekguw3Ju5NyjBpLmTYvqGoWBQv772d08Nf8RDmvQigd6PUHP1mfEOfJ9swQVJbclqDICZPi9ZNfSzD9b62Tqgrx1X3Dha7+hXZMOvH7he0mzt9Pu0t1MWjqO0fmPsnrbSg6r35Kru/yey467ep/7L+3Raup0LX875V+u+JoTnqBE5AZgnKpurXEUceTWBBUpw+clKz1+pZRsrZOpC1ZsWU6/V86gYXoj3hw8k+ZZByU6pGoLaYgPVr/LUwse5dM1H5GVls0lx17BkK7X06bREeXnFZQW8N/P7mZ0/qMc1qAVD/Z+klMOPz2Bke/JDQnqn8BgIB8YC0xTFza7kiFBlfF7PWT6vTGtUGFrnUxdsH7Xj/SdcDrFwWLeGjRzjw/zZLX4p4U8lf8obyyfRCAU4JyjfsvQrjfiEQ83TR/G6m0ruaLTUP56yj9d0WqKlPAEFX6zAGcBVwG5wERgjKquqnFkMZJMCaqM1yNk+31kpHlqNKHC1jqZumB70Tb6T+zFmh3fM/nC6XQ6uGuiQ4qpDbvW8eyXzjjV1qItALRscDgP9X6Kkw8/LbHBVSHeCSqqVWSqqiKyAdgABIDGwKsi8r6q3lrj6OqoYEjZUVTKruIDL6UUCilbC0otOZmUVhQo4oqpF7Jq6zeMGzAl5ZITwCH1DuX2k+5mRPe/MGnpOLYWbWFI1z+4rtVUm/aboERkOHAF8DPwDPBnVS0VEQ+wArAEVUMhVXYVB9hdHCDTH30pJVvrZOqCYCjI9e9eyRc/zubJ3zyf8Jlr8ZaVlsUVna9NdBiuEE0Lqhlwnqp+H3lQVUMi0jc+YdVNChSUBCksCUZVSml7YaltOmhSmqryfzNv5u2Vb3DPqSMZcPRFiQ7J1KJoppO9A2wpeyIi9UWkB4Cqfh2vwOoyxZmRt3l3CVt3l1Ac2HtN046iUluIa1Leg3Pu5flFo7k+92aG5tyY6HBMLYsmQT0B7Ip4vjt8zNSCkmCIbQWlbN5VXF5KaXdxwBbimpT30uKxjPz8Hi7scCl3nvzPRIdjEiCaLj6JnFYe7tpz91aSKSgQUrYXlrKzKGBjTiblTVv1FrfOuJHT25zFA72fSNp92EzNRNOCWi0iw0UkLfxnBLA6mouLSB8RWS4iK0Xktkpev05EFovIQhGZLSIdw8d7i8j88GvzRSS1R0WrwZKTSXVzf/yMYW//jk4HdeWZc8dFXQrIpJ5oEtR1wInAj8BaoAcwdH9vEhEvMAo4B+gIXFyWgCK8rKrHq2oXYCTwQPj4z8BvVfV4nBmEL0YRpzEmyS37eSmXv3EBh9ZvyUsDJpPtr5fokEwC7berTlV/wqkkUV3dgZWquhpARCYA/YGlEdfeEXF+Ns78AFR1QcTxJUCGiKSravEBxGHMHooDxSz+aQF56+ewftc6Oh3UhdwWv+Lwhm2sKymBfty5hksm98PvTWf8eW/SLKt5okMyCRbNOqgM4BrgWKB8ybCqXr2ftx4GrIl4Xtb6qnj964GbAT9QWVfe+cCCypKTiAwl3Jo7/PDD9xOOqat+3LmGvHVzmL9+DvM3zGXxTwspCZYA4Pf6yx83yzqIbod0J7dFd7od+is6H5xDdlp2IkOvM7YVbeXi1/uzs2QnUy56n9YN2yQ6JOMC0Ux2eBFYBpwN3ANcCkQzvbyyX0X3GkBR1VHAKBG5BLgTp0vPuYDIscB/ccos7X0x1dHAaHBKHUURk0lxRYGi8tbR/PVzmR9uJQFkeDPofHAOQ7pcT+6hPeh2SHeaZR3Ess1LyFs3h7z1c8jfMJdpq98CwCteOjY/nm4tepAb/tO6YVtrZcVYYaCQK964gO+2r+LlgVM5tnmnRIdkXCKaYrELVLWriCxS1U4ikoZTMHafExdE5NfAXap6dvj57QCq+p8qzvcAW1W1Yfh5S+BD4CpV/XR/X0gy1uIzNaOqTutofbh1tN5pHZWGnIrurRq0JrdFj/IE07H58VHtGbSlcHN5cpu/fi75G+axu9RZadE0szndWnQvv26XQ7pZK6sGAqEAQ966mGmr3uapc1+kX/vzEx2SqQY31OIr279hm4gch1OPr00U75sHtBORtjgTLAYDl1QIqp2qrgg/PRendBIi0gh4G7g9muRk6oaiQBGLNubv0TrasHs9AJm+TDofnMPQnBvDyaM7B2UfckD3aZLZlN5HnEPvI84BnFI7yzcvJW99uJW1fi7TV78NOK2sDs2O+6WVdWgP2jQ8wlpZUVBVbpsxgvdWvcW/Tn/AkpPZSzQtqCHAa8DxwHNAPeCvqvrUfi8u8hvgIcALjFXVf4nIPUCeqk4VkYeBXjhJcCtwg6ouEZE7gdsJJ6yws8ITNiplLajUoqqs3flDeSsmb/0cvvrpy/LW0eEN2jjddC2crrpjm3eq1enIWwo3k79hHvPXzyFv3RwWbMxjV8lOAJpkNnPGssLxdT24m81Gq8T/Pv8n93/xL0Z0v5XbT7o70eGYA5DQ7TbC3W4XqOrEGkcQZ5agklthoLC8dZQfTkgbd28AylpH3cpbRjVpHcVLMBTkmy1fl49lzV8/h5VbvwHAIx46NjuenHDXYG6LHrRtdGSdbmU9/+XT/OXD4Qw+9nIe7P1knf5eJLOE7wclIh+ras8aRxBnlqCSh6qyZscPzF//RXl33VebviQQcraqb92wbfkYT7cW3enY7PikXKy5tWgL+evnlSfd/A3z2FnirKxoktF0j4TV9ZDcOtPKemflGwx56xLOaHM2z/WbiM9jhWmSlRsS1F+BQuAVnDp8AKjqlirflACWoNyrMFDIlxvml8+Sy1s3l58KylpHWXQ5JKJ1dEh3mmcfnOCI48NpZS1zugXDiXnFlmWA08o6pumxv0zqOLQHRzQ6KuVaFp+vnc3g1/ty3EGdmXT+u2SlZSU6JFMDbkhQ31ZyWFXVVXstW4JyD1Xl3VVT+XTNR+Stn8OSTYvKW0dtGh5Bbose5a2Hjs2Pr9O/QW8r2kr+hnnl67QiW1mNM5rs1cpK5s3rvv75K/pP7MVBWQczddCHNMlsmuiQTA0lPEElC0tQ7vGfT//Ow3NHkunLoushueXTsnNadKd51kGJDs/VQhrim82/tLLy1s/Zq5XljMM5SevIxu2SopW1dscP9J1wGgBvDp5JqwatExuQiYmEJygRubyy46r6Qo2jig7IlkAAACAASURBVCFLUO7wzILHuXPWn7j0uKv475mP1OnWUaxsK9rKgg155ZMv8jfMY0fxduCXVpZTAcNpZdVPb5DgiPe0pXAz/SeeycbdG3jjog/o0Oy4RIdkYsQN66BOiHicAZwJ5AOuSlAm8aZ+8xp/nXULZx/R15JTDDXKaMzpbXpzepvegNPKWrFl+S+trHVzmPHtewAIwjHNjt2jG/XIxu3wSDR1oWOvoLSA371xHj9s/47x571pyclUS7W7+ESkIfCiqvaLT0gHxlpQiTV7zUdcMrkfXQ7uxivnv02mLzPRIdUp24u2sWBj3h41B8taWY3SG+81llUbraxAKMBVUwfxwbfv8nTfcfRtNzDu9zS1yw0tqIoKgHY1jsikjCWbFnHV1Ito0/BInu//qiWnBGiY0YjTWvfitNa9gMhW1tzyltbM76ajKIJwdNOOzozBQ3vEpZWlqvz5gxt4/9t3uPeMhy05mQMSTTXzN/mlyKsHZ28n1y/cNbXjh+3fc/Hr/anvr8/4896gcUaTRIdkcCZUHN20A0c37cAlxzn1l3cUbw9Xv3AWQr+54nVe+mos4LSyurY4obyVlXPICTVqZf33s7sZv+R5/tjjdq7svN/t44ypVDSTJE6NeBoAvlfVtXGN6gBYF1/t+7lgE/0nnsnPBZt446IZHNOs4n6Uxs1CGmLllm/2KLa7fPPSPVpZkTMGj2rSPqpW1piFT3DHzJu59LiruK/XqKSYZWgOjBtm8bUF1qtqUfh5JnCwqn5X46hiyBJU7dpdupsLXu3D15u+YuL5b9P9sBMTHZKJgR3F2yNmDM4lf/1cthVvBaBheqNfZgwe6rSyGqQ33OP9U795jWFv/46zjjiXMb8dbxNlUpwbElQecKKqloSf+4FPVfWEfb6xllmCqj2lwVKumHohs75/n2d/+wpnH9k30SGZOAlpiFVbVzgJa50z+WLZz0vKW1ntm3YorwKSlZbF8GnX0vmgHF45/22rElEHuGGShK8sOQGoakk4SZk6SFW5+f3f8+F307iv1yhLTinOIx7aNTmadk2O5uJjnSWRO4t3lLey8tbP4e0VUxj31bMAtG/SgRcGvGbJycRENAlqk4j0U9WpACLSH/g5vmEZt/rX7L8y6etx/PnXf+Wy469OdDgmAeqnN6Bn6zPo2drZszSkIVZvXcnXP3/Fia162kQZEzPRJKjrgHEi8lj4+Vqg0uoSJrU9nf8Yj+Xdz+WdruXmHrcnOhzjEh7xcFST9hzVpH2iQzEpZr8JSlVXAb8SkXo4Y1Y74x+WcZspyyfy14/+zLlH9ec/pz9oM7OMMXG33zmjIvJvEWmkqrtUdaeINBaRf9ZGcMYdPvlhJje+N4RfHXYSo855Dq/Hm+iQjDF1QDRLx89R1W1lT1R1K/Cb+IVk3GTxTwu56s1BHNm4Pc/3e5UMX81n7BhjTDSiSVBeEUkvexJeB5W+j/NNivh+27dcMnkADdMbMf68N2iY0SjRIRlj6pBoJkm8BMwQkWdxSh5djVUyT3mbCn5i8OTfEgiV8toF79Gi3mGJDskY4yIC1E9Pi+s9opkkMVJEFgG9wjH9Q1WnxTUqk1C7S3Zx6eSBbNi1jknnv0P7psckOiRjjIuk+zzUz0jD64nvZKmo6pCo6nvAewAicpKIjFLV6+MamUmIkmAJ17x1MUs2fcmz/SaSe+ivEh2SMcYlRKBBRhoZabUzUSqqBCUiXYCLgUHAt8Dr8QzKJEZIQ/xx+jBmff8BD571FGcdYXNhjDGODJ+X+hk+PHFuNUWqMkGJSHtgME5i2gy8grMO6vRais3Usns++T9eWzaB20+6u7ysjTGmbvOIUD/DV2utpkj7akEtAz4BfquqKwFE5I+1EpWpdU/Mf4gn5z/MVZ2HMfyEPyc6HGOMC2T6vdRP9yVsYf6+ppmfD2wAZorI0yJyJs4kCZNiXvt6PHd/fDt92w3kn6fdb1UijKnjvB6hcZafBhlpCf08qDJBqepkVR0EHAPMAv4IHCwiT4jIWbUUn4mzWd9/wIjpQzmxZU8e6zPWqkQYU8dl+b00zfbj90WzTDa+9huBqu5W1XGq2hdoCSwEbot7ZCbuFm6Yz9VvDqZ9kw4812+iVYkwpg7zeYQm2X7qJ7jVFKlaKVJVt6jqU6p6RjTni0gfEVkuIitFZK+kJiLXichiEVkoIrNFpGPEa7eH37dcRM6uTpxm/77dtopLpwykSWZTXh44Za+dUY1JRQI0zHSmSbvjIzjxBMhO99Ek20+aN/Gtpkhx249ZRLzAKKA3zhYd80RkqqoujTjtZVV9Mnx+P+ABoE84UQ0GjgUOBT4QkfaqGoxXvHXJT7s3MPj136IaYsLANzmk3qGJDsmYuBOgYVYa6T4vGWleQuk+CkuDFJQE6+xO2mleDw0yfPhclpjKxDOq7sBKVV0d3pF3AtA/8gRV3RHxNBunlBLh8yaoarGqfgusDF8vbiYtmcQDX/yHzYWpvRfjzuIdXDplID/t3shLA163PXxMndEg00lOZTweITvdR/P66TTMTHNd6yGeBKif4bSa3JqcIL4J6jBgTcTzteFjexCR60VkFTASGF7N9w4VkTwRydu0aVONgv3kh0+497O76fZ0O/78wQ18s3lZja7nRiXBEq5+czBLNy3mmb4vk9MirjnfGNfYX/WDjDQvTbL9NMn2k+FL7e4/v9dD03rpZPnj1oEWM/FMUJX9G+/VjlbVUap6JPAX4M5qvne0quaqam7z5s1rFOwj5zzCJ1cs4IIOlzBx6Uv0fKErl0wewMfff4imQPM/pCFGTLuWT9bM5IHeT3JmWxvWM3VDvXQfmf7oZqemeT00zEqjWb10stN9uGSuQEyUlSlqnO2Pew29WIlngloLtIp43hJYt4/zJwADDvC9MXF00w7c13sU84es4NZf/41FPy3gotfP5YyXujN+yQsUB4rjHUJcqCp3ffQXJi+fyB0n/4NBx16W6JCMqRVZfi/Z6dVvKXg8Qr10H83rpdMgIw1fknygVyXd56FZdnrUidot4pmg5gHtRKStiPhxJj1MjTxBRNpFPD0XWBF+PBUYLCLpItIWaAfMjWOse2iW1Zybf3U786/5hofOGg3AH6cPI3fM0dz/xb/5uaBm3Ym1bVTeA4xe8BjXdr2eG3L/lOhwjKkVGWle6mfUbDsIESHT76VpvXQaZzndf8lExJm12CjLX6s19GJF4tl9JSK/AR4CvMBYVf2XiNwD5KnqVBF5GGcbj1JgK3CDqi4Jv/cOnL2nAsBNqvruvu6Vm5ureXl5NYp3087iSmfzqCqf/DCTpxY8yoxv3yPdm84FHS5haM6NHN20Q43uGW8Tl45j+LQh9G9/AU/85nk84t4BUWNiJcPnpWFWfPYqCoaUgpIAhaVB3Nz7n5HmlClKhsQkIvNVNXev46kwvgLxTVCRvtm8jKcXPMakpeMoChZxeuveDOs2nFMPP9M1i9vKfPjddC5/43x+fdgpvDRgMuk+2wjZpD6/10OjrPgvNlVVikpD7C4JEAy553PUI0KDTN8eMxbdzhJUFKJJUGU2F/7MC4ueYezCJ9lUsJGjm3ZkWM6NnHfMYFdUZMjfMI/zJ/XhqCbtef2CadRPb5DokIyJuzSvh8a1kJwqKg4EKSwJUhwI1ep9K0p0cdcDZQkqCtVJUGWKA8VMWT6Jp/IfYenPi2ma2ZyrOg/jis7X0jzroBrFc6BWbV3Bb185g3pp9Xhr8EwOyj4kIXEYU5u8HqFJgsdaAsEQBaVBikqCe087jiOvR2iQkeaK+nkHwhJUFA4kQZVRVT5d8xFP5j/CB9++S7o3nfM7XMzQrjdyTLOO+79AjGzctZ6+r5xOQWkBbw76kCMaH1Vr9zYmUTzi1JFzy/RpVS2vUhHv7r8sv5d6SdhqimQJKgo1SVCRVmxZztMLRjFp6UsUBgo5rXUvhuUM57TWveL6Q7SjeDsDJ53Ft9tW8foF0+hySLe43csYtxCBJlnurYhQVOp0/5UEY9v95/MIDVKkAoYlqCjEKkGV2VK4mRcXj2HMgif4qWAD7Zt0cMapOgwm05cZs/uA09V48eR+zF33GS8NmMxprXvF9PrGuJEINM5yX5HTysSq+6+suOuBrO9yq6oSlPv/VZNYk8ymjOh+K/OuWcYjZz+D3+vnTx/8gW7PtGfkZ/9g0+6NMblPMBTkhveu5rO1H/PQWaMtOZk6QYBGmcmRnAB8Xg8NMpwqFfXSfXgOoDclzeuhSbY/pZLTvlgLKkKsW1AVqSqfrv2Yp+Y/wvvfvoPf6+e8YwYzLOdGOjQ77oCvecesmxm78En+dsq/+UPuH2MctTHuE1mZPJlF2/0nQL0MX1LUzzsQVbWgUvOrdSkR4eRWp3Jyq1NZueUbnl4wiolLX2TCkhfoefgZDMsZzulteldrMe2j8+5j7MInGZYz3JKTqTMqViZPVhlpztYfpcEQBSVBikv37v5L93mon5HmmgkgtclaUBHi3YKqzJbCzby0eCxjFz7Bht3radfkGIZ2vYELOl6y33Gq8Ute4I/Th3HeMYN4rM9YqxJh6oT6KdySCIV+mf2n6H6rsKcKmyQRhUQkqDIlwRKmfvMaT+U/wuKfFtIksxlXdBrCVZ2HVbqOafrqd7hq6kWc3Oo0XhzwOn6vPwFRG1O7stN91Ksj4y+hkCZFmaJYsAQVhUQmqDKqymdrP2F0/qNMX/02ad40Bhx9EcNybuTY5p0AmL9+Dhe8eg7tmnTg9Qvfo56/fkJjNqY2ZPq9NKhh8VfjTpagouCGBBVp9daVPL3gMSYseZHCQAGntDqd8zoM5p6Pb6dheiPeHDwzYdUqjKlNGWleGmZackpVlqCi4LYEVWZr0RZeWjSGMeFxqmZZB/HWoJm0aXREokMzJu7SfR4aZVkXdiqzBBUFtyaoMiXBEt5f/Q5HN+3IUU3aJzocY+IuUcVfTe2yaeYpwO/1c267Afs/0ZgU4POIJac6zuYlG2Ncx+sRGmf5LTnVcZagjDGu4hEnOdWVKdamapagjDGu4RR/rZtVE8zebAzK1Dk+j+DzevB7PXg8EAgqpcEQJcEQLp4jk/IEpzK5W7fNMLXPEpRJaR4R0rxCmteDzyv4vZ69xjUiCxMEQ06ycv4ogWCoVndGravKir8mS2VyUzssQZmUIThbGpQlpDSvp9pdRV6P4PV4y+ufqSqBsqQVUEpDobjvkFoXpUrxVxNblqBM0vJ6yhLRLwkp1iSiBUZ4rWgo5CSq0qBSGghRGrKuwZqoKwVRTfVZgjJJQQT8Xs8vLSSPJ2GzvDweId3jdboG051jgWCIQEgpCYYoDTitLMtZ+1cv3Uem35KTqZwlKOM6ZV11ZWNGB9JVV9uceNmja7A0qARCTtdgSTDk6ioliZDl99aZnWHNgbGfDpNwXo/TIkrzCT6P00JK9gWaIoLfJ/jZs2uwJNzSqutdg5l+L/WtMrnZD0tQplaJEE5GHnweCU/1Tu5kFC2PR8jwhLuzIroGS4PhMa2Ak7xSXYbPts0w0bEEZeLK5xHSfB4nKXnF1rhUUNY1mMmeXYORU91TqWvQ7/XQINM+dkx07CfFxIUzM2vvNUdm38q7Bn2/JPJUWZuV5vXQyIq/mmqI66+zItJHRJaLyEoRua2S128WkaUiskhEZohI64jXRorIEhH5WkQeEfupThoNMtLI9HvtgyhGvB4hI80Zs2mS7eegBhk0yfaXT892+wQScFrSjTItOZnqiVsLSkS8wCigN7AWmCciU1V1acRpC4BcVS0Qkd8DI4FBInIicBLQKXzebOBUYFa84jWxYdOGa4ez7mvPrsGSYMiVZZus+Ks5UPHs4usOrFTV1QAiMgHoD5QnKFWdGXH+F8BlZS8BGTjznwRIAzbGMVYTAzZtOHFEhHSf13VlmzwiNMm25GQOTDw/TQ4D1kQ8Xwv02Mf51wDvAqjq5yIyE1iPk6AeU9WvK75BRIYCQwEOP/zwGIVtDkRZF5Rxj0SXbRKBRlaZ3NRAPBNUZT+Vlf5vEJHLgFycbjxE5CigA9AyfMr7ItJTVT/e42Kqo4HR4Gz5HqO4TTVl+Lw0zLTk5Hb7K9sUiGHXoACNMv1W/NXUSDwT1FqgVcTzlsC6iieJSC/gDuBUVS0OHx4IfKGqu8LnvAv8Cvi44vtNYtm04eS2R9mmsMiyTYED6Bosq0weORPRmAMRz5+geUA7EWkrIn5gMDA18gQR6Qo8BfRT1Z8iXvoBOFVEfCKShtOy2quLzySWTRtOTT6vh4w0ZzFtk2w/zeun0zjLT/0MHxk+L579/HtbZXITK3H71VdVAyJyAzAN8AJjVXWJiNwD5KnqVOB/QD1gUvhD7gdV7Qe8CpwBLMbpFnxPVd+MV6ym+mzacN1RnbJN9TN8VpncxIyoW+ai1lBubq7m5eXV6Bqbdhan1Kr9ePF6hCY2bdhUEAypTYgwB0RE5qtqbsXj1klsqsXWtJiqWHIysWYJykRNBBrbtGFjTC2xBGWiUjZt2Iq9GmNqi33amP2yacPGmESwTxyzXzZt2BiTCJagzD6VVcw2xpjaZgnKVMkqkxtjEskSlKlUdrrPKpMbYxLKEpTZS6bfSz1LTsaYBLMEZfaQ4XNqsBljTKJZgjLl0n0eGmZZcjLGuIMlKAM4lcltTydjjJtYgjJWmdwY40qWoOo4r8eKvxpj3MkSVB1mlcmNMW5mCaqOssrkxhi3swRVBwnQOMsqkxtj3M0+oSLUz/CRluIf2mWVyVP96zTGJD8rFxAhI81LRpqX0mCIgpIgxaVBUm0DeKtMboxJFpagKuGsCfIQSvdRWBqkoCRISJM/VTXMtMrkxpjkYQlqHzweKS+aWhROVKXBUKLDOiD1M3yWnIwxScUSVJSSufsvO91Hlt/+qY0xycU+taopsvuvoDRIocu7/6wyuTEmWdkn1wHyeIR66T6y/V6KAyFXdv9ZZXJjTDKzBFVDIrJn919xkOJA4rv/rDK5MSbZWYKKoTSvh4ZZHkKhxHb/+a0yuTEmBViCioNEdv/5PEKjLKtMboxJfpag4iiy+68kEKKwJL7df2WVyS05GWNSQVzr3YhIHxFZLiIrReS2Sl6/WUSWisgiEZkhIq0jXjtcRKaLyNfhc9rEM9Z484fHhJrVSyc73Uesc4hVJjfGpJq4JSgR8QKjgHOAjsDFItKxwmkLgFxV7QS8CoyMeO0F4H+q2gHoDvwUr1hrU1n3X/N66TTMTMMXg4RilcmNMakoni2o7sBKVV2tqiXABKB/5AmqOlNVC8JPvwBaAoQTmU9V3w+ftyvivJRQ1v3XtF46jbP8ZPi8HEh6scrkxphUFc9PtcOANRHP14aPVeUa4N3w4/bANhF5XUQWiMj/wi2yPYjIUBHJE5G8TZs2xSzw2lbW/de0mt1/AjTK8ltlcmNMSornJ1tlH7OVzg8QkcuAXOB/4UM+4BTgFuAE4Ajgyr0upjpaVXNVNbd58+axiDmhvBHdfw0y9t/91yAzDb/PkpMxJjXF89NtLdAq4nlLYF3Fk0SkF3AH0E9ViyPeuyDcPRgApgA5cYzVVUSETP+e3X8VWWVyY0yqi2eCmge0E5G2IuIHBgNTI08Qka7AUzjJ6acK720sImXNojOApXGM1bUqm/1nlcmNMXVB3NZBqWpARG4ApgFeYKyqLhGRe4A8VZ2K06VXD5gUXrvzg6r2U9WgiNwCzBDnhfnA0/GKNRmUdf9Z4VdjTF0h6uJK3NWRm5ureXl5iQ7DGGNMNYnIfFXNrXjcRtiNMca4kiUoY4wxrmQJyhhjjCtZgjLGGONKlqCMMca4kiUoY4wxrmQJyhhjjCtZgjLGGONKlqCMMca4UspUkhCRTcD3NbxMM+DnGIQTbxZn7CRDjGBxxlIyxAh1K87WqrrXlhQpk6BiQUTyKiu34TYWZ+wkQ4xgccZSMsQIFidYF58xxhiXsgRljDHGlSxB7Wl0ogOIksUZO8kQI1icsZQMMYLFaWNQxhhj3MlaUMYYY1zJEpQxxhhXsgQFiMhYEflJRL5KdCxVEZFWIjJTRL4WkSUiMiLRMVVGRDJEZK6IfBmO8+5Ex7QvIuIVkQUi8laiY6mKiHwnIotFZKGIuHLbaBFpJCKvisiy8M/orxMdU0UicnT4e1j2Z4eI3JTouCojIn8M///5SkTGi0hGomOqSERGhONbEq/vo41BASLSE9gFvKCqxyU6nsqISAugharmi0h9YD4wQFWXJji0PYiIANmquktE0oDZwAhV/SLBoVVKRG4GcoEGqto30fFURkS+A3JV1bWLNkXkeeATVX1GRPxAlqpuS3RcVRERL/Aj0ENVa7rAP6ZE5DCc/zcdVbVQRCYC76jqc4mN7BcichwwAegOlADvAb9X1RWxvI+1oABV/RjYkug49kVV16tqfvjxTuBr4LDERrU3dewKP00L/3Hlb0Ei0hI4F3gm0bEkMxFpAPQExgCoaombk1PYmcAqtyWnCD4gU0R8QBawLsHxVNQB+EJVC1Q1AHwEDIz1TSxBJSERaQN0BeYkNpLKhbvNFgI/Ae+rqivjBB4CbgVCiQ5kPxSYLiLzRWRoooOpxBHAJuDZcHfpMyKSneig9mMwMD7RQVRGVX8E7gN+ANYD21V1emKj2stXQE8RaSoiWcBvgFaxvoklqCQjIvWA14CbVHVHouOpjKoGVbUL0BLoHu4OcBUR6Qv8pKrzEx1LFE5S1RzgHOD6cJe0m/iAHOAJVe0K7AZuS2xIVQt3QfYDJiU6lsqISGOgP9AWOBTIFpHLEhvVnlT1a+C/wPs43XtfAoFY38cSVBIJj+m8BoxT1dcTHc/+hLt5ZgF9EhxKZU4C+oXHdyYAZ4jIS4kNqXKqui7890/AZJx+fzdZC6yNaCm/ipOw3OocIF9VNyY6kCr0Ar5V1U2qWgq8DpyY4Jj2oqpjVDVHVXviDJHEdPwJLEEljfDkgzHA16r6QKLjqYqINBeRRuHHmTj/2ZYlNqq9qertqtpSVdvgdPd8qKqu+i0VQESyw5NiCHebnYXTveIaqroBWCMiR4cPnQm4avJOBRfj0u69sB+AX4lIVvj//Zk4Y86uIiIHhf8+HDiPOHxPfbG+YDISkfHAaUAzEVkL/F1VxyQ2qr2cBPwOWBwe3wH4P1V9J4ExVaYF8Hx4lpQHmKiqrp3CnQQOBiY7n1P4gJdV9b3EhlSpG4Fx4e6z1cBVCY6nUuHxkt7AsETHUhVVnSMirwL5ON1mC3Bn2aPXRKQpUApcr6pbY30Dm2ZujDHGlayLzxhjjCtZgjLGGONKlqCMMca4kiUoY4wxrmQJyhhjjCtZgjJJQURURO6PeH6LiNwVo2s/JyIXxOJa+7nPheFK3zMrHG8T/vpujDj2mIhcuZ/rXScil+/nnCtF5LEqXttV2fHaEq5+fkT48Xci0iz8uJuIfCsiXUWkr9sr4pv4sQRlkkUxcF7Zh5hbhNd7Resa4A+qenolr/0EjAivI4qKqj6pqi9U4/5xFy5uGs15xwJeVV1d4XgnnEoUg1R1AfA2TsWPrJgHa1zPEpRJFgGcxYp/rPhCxRZQWctARE4TkY9EZKKIfCMi94rIpeLsV7VYRI6MuEwvEfkkfF7f8Pu9IvI/EZknIotEZFjEdWeKyMvA4kriuTh8/a9E5L/hY38DTgaeFJH/VfL1bQJmAFdUcr0jReS9cLHYT0TkmPDxu0TklvDjE8Ixfh6OObLaxKHh968QkZEVrn2/iOSLyAwRaR4+1kVEvghfb3K4NhwiMktEcsOPm4XLRJW10iaJyJs4RW1biMjH4uy59JWInFLJ13sp8EaFYx2AKcDvVHUuONXxccpluXIrFBNflqBMMhkFXCoiDavxns7ACOB4nEoc7VW1O84WGzdGnNcGOBVn+40nxdkg7hqcStInACcA14pI2/D53YE7VLVj5M1E5FCcIppnAF2AE0RkgKreA+QBl6rqn6uI9V7gT5W0ykYDN6pqN+AW4PFK3vsscJ2q/hoIVnitCzAo/D0YJCJlVaezcWrS5eBsl/D38PEXgL+oaiecBPx39u/XwBWqegZwCTAtXDC4M7CwkvNPwtnTLNIbwA2qOrvC8TygsiRnUpwlKJM0wtXbXwCGV+Nt88J7aRUDq4CybQsW4ySlMhNVNRTecG01cAxO3bvLw6Wl5gBNgXbh8+eq6reV3O8EYFa40GcAGIezV1I0X9+3wFycD3igvHr9icCkcBxP4ZSTIuKcRkB9Vf0sfOjlCpeeoarbVbUIp0Ze6/DxEPBK+PFLwMnh5N9IVT8KH38+yvjfV9WyPdXmAVeFxwiPD+9fVlELnFZjpA+AIZUk6J9wqnqbOsYSlEk2D+G0bCL3GwoQ/lkWp2hd5DhOccTjUMTzEHvWoqxY80sBwWm5dAn/aRuxL8/uKuKTaL+QKvwb+Au//N/0ANsiYuiiqh2qec/I70GQqmtw7q/uWfn3Gai4BXn59yO8AWhPnB1rX6xiIkdhJde4Ifx3xRZiRvh8U8dYgjJJJfxb+kScJFXmO6Bb+HF/nF18q+tCEfGEx6WOAJYD04Dfi7PNCSLSXva/Ed8c4NTwGI0Xp3L2R/t5TzlVXYbTyukbfr4D+FZELgzHICLSucJ7tgI7ReRX4UODo7ydBygbu7sEmK2q24GtEeNGv4uI/zt++T5XOetRRFrj7LX1NE4F/sq23vgaOKrCsRDO9+toEbkn4nh7XFbB3dQOS1AmGd0PRM7mexonKcwFelB162ZfluN8EL+LM5ZThDNOtRTID086eIr97ACgquuB24GZOJu45atqxckA+/MvnM0ey1wKXCMiXwJLcJJwRdcAo0Xkc5wW1fYo7rMbOFZE5uOMmZUlhSuA/4nIIpzxq7Lj9+Ek7M/Y8/tf0WnAQhFZAJwPPFzJOW+Hz9tDuCu2P87MvevDh08Pn2/qGKtmbkwKEJF6qlo2ILYGHgAAAHxJREFUe/E2oIWqjkhwWFUSZ6+wmTi7BVec1BF53sE4W4ycWWvBGdewBGVMChCRQTgtNx/wPXClqlachOAqInI2zgacP+zjnBOAUlWtbCagSXGWoIwxxriSjUEZY4xxJUtQxhhjXMkSlDHGGFeyBGWMMcaVLEEZY4xxpf8Hsk99YnA51FIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(1,Ks),mean_acc,'g')\n",
    "plt.fill_between(range(1,Ks),mean_acc - 1 * std_acc,mean_acc + 1 * std_acc, alpha=0.10)\n",
    "plt.legend(('Accuracy ', '+/- 3xstd'))\n",
    "plt.ylabel('Accuracy ')\n",
    "plt.xlabel('Number of Neighbours (K)')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best accuracy was with 0.34 with k= 9\n"
     ]
    }
   ],
   "source": [
    "print( \"The best accuracy was with\", mean_acc.max(), \"with k=\", mean_acc.argmax()+1) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}