NSL-KDD.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intrusion detection on NSL-KDD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is my try with [NSL-KDD](http://www.unb.ca/research/iscx/dataset/iscx-NSL-KDD-dataset.html) dataset, which is an improved version of well-known [KDD'99](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html) dataset. I've used Python, Scikit-learn and PySpark via [ready-to-run Jupyter applications in Docker](https://github.com/jupyter/docker-stacks).\n",
    "\n",
    "I've tried a variety of approaches to deal with this dataset. Here are presented some of them.\n",
    "\n",
    "To be able to run this notebook, use `make nsl-kdd-pyspark` command. It'll download the latest jupyter/pyspark-notebook docker image and start a container with Jupyter available at `8889` port."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Task description summary](#1.-Task-description-summary)\n",
    "2. [Data loading](#2.-Data-loading)\n",
    "3. [Exploratory Data Analysis](#3.-Exploratory-Data-Analysis)\n",
    "4. [One Hot Encoding for categorical variables](#4.-One-Hot-Encoding-for-categorical-variables)\n",
    "5. [Feature Selection using Attribute Ratio](#5.-Feature-Selection-using-Attribute-Ratio)\n",
    "6. [Data preparation](#6.-Data-preparation)\n",
    "7. [Visualization via PCA](#7.-Visualization-via-PCA)\n",
    "8. [KMeans clustering with Random Forest Classifiers](#8.-KMeans-clustering-with-Random-Forest-Classifiers)\n",
    "9. [Gaussian Mixture clustering with Random Forest Classifiers](#9.-Gaussian-Mixture-clustering-with-Random-Forest-Classifiers)\n",
    "10. [Supervised approach for dettecting each type of attacks separately](#10.-Supervised-approach-for-dettecting-each-type-of-attacks-separately)\n",
    "11. [Ensembling experiments](#11.-Ensembling-experiments)\n",
    "12. [Results summary](#12.-Results-summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Task description summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Software to detect network intrusions protects a computer network from unauthorized users, including perhaps insiders. The intrusion detector learning task is to build a predictive model (i.e. a classifier) capable of distinguishing between bad connections, called intrusions or attacks, and good normal connections.\n",
    "\n",
    "A connection is a sequence of TCP packets starting and ending at some well defined times, between which data flows to and from a source IP address to a target IP address under some well defined protocol. Each connection is labeled as either normal, or as an attack, with exactly one specific attack type. Each connection record consists of about 100 bytes.\n",
    "\n",
    "Attacks fall into four main categories:\n",
    "\n",
    "- DOS: denial-of-service, e.g. syn flood;\n",
    "- R2L: unauthorized access from a remote machine, e.g. guessing password;\n",
    "- U2R: unauthorized access to local superuser (root) privileges, e.g., various ''buffer overflow'' attacks;\n",
    "- probing: surveillance and other probing, e.g., port scanning.\n",
    "\n",
    "It is important to note that the test data is not from the same probability distribution as the training data, and it includes specific attack types not in the training data. This makes the task more realistic. Some intrusion experts believe that most novel attacks are variants of known attacks and the \"signature\" of known attacks can be sufficient to catch novel variants.  The datasets contain a total of 24 training attack types, with an additional 14 types in the test data only.\n",
    "\n",
    "The complete task description could be found [here](http://kdd.ics.uci.edu/databases/kddcup99/task.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NSL-KDD dataset description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[NSL-KDD](http://www.unb.ca/research/iscx/dataset/iscx-NSL-KDD-dataset.html) is a data set suggested to solve some of the inherent problems of the [KDD'99](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html) data set.\n",
    "\n",
    "The NSL-KDD data set has the following advantages over the original KDD data set:\n",
    "- It does not include redundant records in the train set, so the classifiers will not be biased towards more frequent records.\n",
    "- There is no duplicate records in the proposed test sets; therefore, the performance of the learners are not biased by the methods which have better detection rates on the frequent records.\n",
    "- The number of selected records from each difficultylevel group is inversely proportional to the percentage of records in the original KDD data set. As a result, the classification rates of distinct machine learning methods vary in a wider range, which makes it more efficient to have an accurate evaluation of different learning techniques.\n",
    "- The number of records in the train and test sets are reasonable, which makes it affordable to run the experiments on the complete set without the need to randomly select a small portion. Consequently, evaluation results of different research works will be consistent and comparable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Data loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Here are some imports that are used along this notebook\n",
    "import os\n",
    "import math\n",
    "import itertools\n",
    "import multiprocessing\n",
    "import pandas\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from time import time\n",
    "from collections import OrderedDict\n",
    "%matplotlib inline\n",
    "gt0 = time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from pyspark import SparkConf, SparkContext\n",
    "from pyspark.sql import SQLContext, Row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "conf = SparkConf()\\\n",
    "    .setMaster(f\"local[{multiprocessing.cpu_count()}]\")\\\n",
    "    .setAppName(\"PySpark NSL-KDD\")\\\n",
    "    .setAll([(\"spark.driver.memory\", \"8g\"), (\"spark.default.parallelism\", f\"{multiprocessing.cpu_count()}\")])\n",
    "\n",
    "# Creating local SparkContext with specified SparkConf and creating SQLContext based on it\n",
    "sc = SparkContext.getOrCreate(conf=conf)\n",
    "sc.setLogLevel('INFO')\n",
    "sqlContext = SQLContext(sc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from pyspark.sql.types import *\n",
    "from pyspark.sql.functions import udf, split, col\n",
    "import pyspark.sql.functions as sql\n",
    "\n",
    "train20_nsl_kdd_dataset_path = os.path.join(\"NSL_KDD_Dataset\", \"KDDTrain+_20Percent.txt\")\n",
    "train_nsl_kdd_dataset_path = os.path.join(\"NSL_KDD_Dataset\", \"KDDTrain+.txt\")\n",
    "test_nsl_kdd_dataset_path = os.path.join(\"NSL_KDD_Dataset\", \"KDDTest+.txt\")\n",
    "\n",
    "col_names = np.array([\"duration\",\"protocol_type\",\"service\",\"flag\",\"src_bytes\",\n",
    "    \"dst_bytes\",\"land\",\"wrong_fragment\",\"urgent\",\"hot\",\"num_failed_logins\",\n",
    "    \"logged_in\",\"num_compromised\",\"root_shell\",\"su_attempted\",\"num_root\",\n",
    "    \"num_file_creations\",\"num_shells\",\"num_access_files\",\"num_outbound_cmds\",\n",
    "    \"is_host_login\",\"is_guest_login\",\"count\",\"srv_count\",\"serror_rate\",\n",
    "    \"srv_serror_rate\",\"rerror_rate\",\"srv_rerror_rate\",\"same_srv_rate\",\n",
    "    \"diff_srv_rate\",\"srv_diff_host_rate\",\"dst_host_count\",\"dst_host_srv_count\",\n",
    "    \"dst_host_same_srv_rate\",\"dst_host_diff_srv_rate\",\"dst_host_same_src_port_rate\",\n",
    "    \"dst_host_srv_diff_host_rate\",\"dst_host_serror_rate\",\"dst_host_srv_serror_rate\",\n",
    "    \"dst_host_rerror_rate\",\"dst_host_srv_rerror_rate\",\"labels\"])\n",
    "\n",
    "nominal_inx = [1, 2, 3]\n",
    "binary_inx = [6, 11, 13, 14, 20, 21]\n",
    "numeric_inx = list(set(range(41)).difference(nominal_inx).difference(binary_inx))\n",
    "\n",
    "nominal_cols = col_names[nominal_inx].tolist()\n",
    "binary_cols = col_names[binary_inx].tolist()\n",
    "numeric_cols = col_names[numeric_inx].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Function to load dataset and divide it into 8 partitions\n",
    "def load_dataset(path):\n",
    "    dataset_rdd = sc.textFile(path, 8).map(lambda line: line.split(','))\n",
    "    dataset_df = (dataset_rdd.toDF(col_names.tolist()).select(\n",
    "                    col('duration').cast(DoubleType()),\n",
    "                    col('protocol_type').cast(StringType()),\n",
    "                    col('service').cast(StringType()),\n",
    "                    col('flag').cast(StringType()),\n",
    "                    col('src_bytes').cast(DoubleType()),\n",
    "                    col('dst_bytes').cast(DoubleType()),\n",
    "                    col('land').cast(DoubleType()),\n",
    "                    col('wrong_fragment').cast(DoubleType()),\n",
    "                    col('urgent').cast(DoubleType()),\n",
    "                    col('hot').cast(DoubleType()),\n",
    "                    col('num_failed_logins').cast(DoubleType()),\n",
    "                    col('logged_in').cast(DoubleType()),\n",
    "                    col('num_compromised').cast(DoubleType()),\n",
    "                    col('root_shell').cast(DoubleType()),\n",
    "                    col('su_attempted').cast(DoubleType()),\n",
    "                    col('num_root').cast(DoubleType()),\n",
    "                    col('num_file_creations').cast(DoubleType()),\n",
    "                    col('num_shells').cast(DoubleType()),\n",
    "                    col('num_access_files').cast(DoubleType()),\n",
    "                    col('num_outbound_cmds').cast(DoubleType()),\n",
    "                    col('is_host_login').cast(DoubleType()),\n",
    "                    col('is_guest_login').cast(DoubleType()),\n",
    "                    col('count').cast(DoubleType()),\n",
    "                    col('srv_count').cast(DoubleType()),\n",
    "                    col('serror_rate').cast(DoubleType()),\n",
    "                    col('srv_serror_rate').cast(DoubleType()),\n",
    "                    col('rerror_rate').cast(DoubleType()),\n",
    "                    col('srv_rerror_rate').cast(DoubleType()),\n",
    "                    col('same_srv_rate').cast(DoubleType()),\n",
    "                    col('diff_srv_rate').cast(DoubleType()),\n",
    "                    col('srv_diff_host_rate').cast(DoubleType()),\n",
    "                    col('dst_host_count').cast(DoubleType()),\n",
    "                    col('dst_host_srv_count').cast(DoubleType()),\n",
    "                    col('dst_host_same_srv_rate').cast(DoubleType()),\n",
    "                    col('dst_host_diff_srv_rate').cast(DoubleType()),\n",
    "                    col('dst_host_same_src_port_rate').cast(DoubleType()),\n",
    "                    col('dst_host_srv_diff_host_rate').cast(DoubleType()),\n",
    "                    col('dst_host_serror_rate').cast(DoubleType()),\n",
    "                    col('dst_host_srv_serror_rate').cast(DoubleType()),\n",
    "                    col('dst_host_rerror_rate').cast(DoubleType()),\n",
    "                    col('dst_host_srv_rerror_rate').cast(DoubleType()),\n",
    "                    col('labels').cast(StringType())))\n",
    "\n",
    "    return dataset_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first part of data preparation is deviding connections into normal and attack classes based on 'labels' column. Then attacks are splitted to four main categories: DoS, Probe, R2L and U2R. After this, all of those categories are indexed. Also, ID column is added to simplify work with clustered data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline, Transformer\n",
    "from pyspark.ml.feature import StringIndexer\n",
    "from pyspark import keyword_only\n",
    "from pyspark.ml.param.shared import HasInputCol, HasOutputCol, Param\n",
    "\n",
    "# Dictionary that contains mapping of various attacks to the four main categories\n",
    "attack_dict = {\n",
    "    'normal': 'normal',\n",
    "    \n",
    "    'back': 'DoS',\n",
    "    'land': 'DoS',\n",
    "    'neptune': 'DoS',\n",
    "    'pod': 'DoS',\n",
    "    'smurf': 'DoS',\n",
    "    'teardrop': 'DoS',\n",
    "    'mailbomb': 'DoS',\n",
    "    'apache2': 'DoS',\n",
    "    'processtable': 'DoS',\n",
    "    'udpstorm': 'DoS',\n",
    "    \n",
    "    'ipsweep': 'Probe',\n",
    "    'nmap': 'Probe',\n",
    "    'portsweep': 'Probe',\n",
    "    'satan': 'Probe',\n",
    "    'mscan': 'Probe',\n",
    "    'saint': 'Probe',\n",
    "\n",
    "    'ftp_write': 'R2L',\n",
    "    'guess_passwd': 'R2L',\n",
    "    'imap': 'R2L',\n",
    "    'multihop': 'R2L',\n",
    "    'phf': 'R2L',\n",
    "    'spy': 'R2L',\n",
    "    'warezclient': 'R2L',\n",
    "    'warezmaster': 'R2L',\n",
    "    'sendmail': 'R2L',\n",
    "    'named': 'R2L',\n",
    "    'snmpgetattack': 'R2L',\n",
    "    'snmpguess': 'R2L',\n",
    "    'xlock': 'R2L',\n",
    "    'xsnoop': 'R2L',\n",
    "    'worm': 'R2L',\n",
    "    \n",
    "    'buffer_overflow': 'U2R',\n",
    "    'loadmodule': 'U2R',\n",
    "    'perl': 'U2R',\n",
    "    'rootkit': 'U2R',\n",
    "    'httptunnel': 'U2R',\n",
    "    'ps': 'U2R',    \n",
    "    'sqlattack': 'U2R',\n",
    "    'xterm': 'U2R'\n",
    "}\n",
    "\n",
    "attack_mapping_udf = udf(lambda v: attack_dict[v])\n",
    "\n",
    "class Labels2Converter(Transformer):\n",
    "\n",
    "    @keyword_only\n",
    "    def __init__(self):\n",
    "        super(Labels2Converter, self).__init__()\n",
    "\n",
    "    def _transform(self, dataset):\n",
    "        return dataset.withColumn('labels2', sql.regexp_replace(col('labels'), '^(?!normal).*$', 'attack'))\n",
    "     \n",
    "class Labels5Converter(Transformer):\n",
    "    \n",
    "    @keyword_only\n",
    "    def __init__(self):\n",
    "        super(Labels5Converter, self).__init__()\n",
    "\n",
    "    def _transform(self, dataset):\n",
    "        return dataset.withColumn('labels5', attack_mapping_udf(col('labels')))\n",
    "    \n",
    "labels2_indexer = StringIndexer(inputCol=\"labels2\", outputCol=\"labels2_index\")\n",
    "labels5_indexer = StringIndexer(inputCol=\"labels5\", outputCol=\"labels5_index\")\n",
    "\n",
    "labels_mapping_pipeline = Pipeline(stages=[Labels2Converter(), Labels5Converter(), labels2_indexer, labels5_indexer])\n",
    "\n",
    "labels2 = ['normal', 'attack']\n",
    "labels5 = ['normal', 'DoS', 'Probe', 'R2L', 'U2R']\n",
    "labels_col = 'labels2_index'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of examples in train set: 125973\n",
      "Time: 12.46s\n"
     ]
    }
   ],
   "source": [
    "# Loading train data\n",
    "t0 = time()\n",
    "train_df = load_dataset(train_nsl_kdd_dataset_path)\n",
    "\n",
    "# Fitting preparation pipeline\n",
    "labels_mapping_model = labels_mapping_pipeline.fit(train_df)\n",
    "\n",
    "# Transforming labels column and adding id column\n",
    "train_df = labels_mapping_model.transform(train_df).withColumn('id', sql.monotonically_increasing_id())\n",
    "\n",
    "train_df = train_df.cache()\n",
    "print(f\"Number of examples in train set: {train_df.count()}\")\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of examples in test set: 22544\n",
      "Time: 1.29s\n"
     ]
    }
   ],
   "source": [
    "# Loading test data\n",
    "t0 = time()\n",
    "test_df = load_dataset(test_nsl_kdd_dataset_path)\n",
    "\n",
    "# Transforming labels column and adding id column\n",
    "test_df = labels_mapping_model.transform(test_df).withColumn('id', sql.monotonically_increasing_id())\n",
    "\n",
    "test_df = test_df.cache()\n",
    "print(f\"Number of examples in test set: {test_df.count()}\")\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Exploratory Data Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some descriptive statistics of available features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-----+\n",
      "|labels2|count|\n",
      "+-------+-----+\n",
      "| normal|67343|\n",
      "| attack|58630|\n",
      "+-------+-----+\n",
      "\n",
      "+-------+-----+\n",
      "|labels5|count|\n",
      "+-------+-----+\n",
      "| normal|67343|\n",
      "|    DoS|45927|\n",
      "|  Probe|11656|\n",
      "|    R2L|  995|\n",
      "|    U2R|   52|\n",
      "+-------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Labels columns\n",
    "(train_df.groupby('labels2').count().show())\n",
    "(train_df.groupby('labels5').count().sort(sql.desc('count')).show())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-----+\n",
      "|labels2|count|\n",
      "+-------+-----+\n",
      "| normal| 9711|\n",
      "| attack|12833|\n",
      "+-------+-----+\n",
      "\n",
      "+-------+-----+\n",
      "|labels5|count|\n",
      "+-------+-----+\n",
      "| normal| 9711|\n",
      "|    DoS| 7458|\n",
      "|    R2L| 2754|\n",
      "|  Probe| 2421|\n",
      "|    U2R|  200|\n",
      "+-------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "(test_df.groupby('labels2').count().show())\n",
    "(test_df.groupby('labels5').count().sort(sql.desc('count')).show())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------------+------+------+\n",
      "|protocol_type_labels2|attack|normal|\n",
      "+---------------------+------+------+\n",
      "|                 icmp|  6982|  1309|\n",
      "|                  tcp| 49089| 53600|\n",
      "|                  udp|  2559| 12434|\n",
      "+---------------------+------+------+\n",
      "\n",
      "+---------------------+-----+-----+---+---+------+\n",
      "|protocol_type_labels5|  DoS|Probe|R2L|U2R|normal|\n",
      "+---------------------+-----+-----+---+---+------+\n",
      "|                 icmp| 2847| 4135|  0|  0|  1309|\n",
      "|                  tcp|42188| 5857|995| 49| 53600|\n",
      "|                  udp|  892| 1664|  0|  3| 12434|\n",
      "+---------------------+-----+-----+---+---+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 'protocol_type' nominal column\n",
    "(train_df.crosstab(nominal_cols[0], 'labels2').sort(sql.asc(nominal_cols[0] + '_labels2')).show())\n",
    "(train_df.crosstab(nominal_cols[0], 'labels5').sort(sql.asc(nominal_cols[0] + '_labels5')).show())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "70\n",
      "+---------------+------+------+\n",
      "|service_labels2|attack|normal|\n",
      "+---------------+------+------+\n",
      "|            IRC|     1|   186|\n",
      "|            X11|     6|    67|\n",
      "|         Z39_50|   862|     0|\n",
      "|            aol|     2|     0|\n",
      "|           auth|   719|   236|\n",
      "|            bgp|   710|     0|\n",
      "|        courier|   734|     0|\n",
      "|       csnet_ns|   545|     0|\n",
      "|            ctf|   563|     0|\n",
      "|        daytime|   521|     0|\n",
      "|        discard|   538|     0|\n",
      "|         domain|   531|    38|\n",
      "|       domain_u|     9|  9034|\n",
      "|           echo|   434|     0|\n",
      "|          eco_i|  4089|   497|\n",
      "|          ecr_i|  2887|   190|\n",
      "|            efs|   485|     0|\n",
      "|           exec|   474|     0|\n",
      "|         finger|  1222|   545|\n",
      "|            ftp|   836|   918|\n",
      "|       ftp_data|  1876|  4984|\n",
      "|         gopher|   518|     0|\n",
      "|        harvest|     2|     0|\n",
      "|      hostnames|   460|     0|\n",
      "|           http|  2289| 38049|\n",
      "|      http_2784|     1|     0|\n",
      "|       http_443|   530|     0|\n",
      "|      http_8001|     2|     0|\n",
      "|          imap4|   644|     3|\n",
      "|       iso_tsap|   687|     0|\n",
      "|         klogin|   433|     0|\n",
      "|         kshell|   299|     0|\n",
      "|           ldap|   410|     0|\n",
      "|           link|   475|     0|\n",
      "|          login|   429|     0|\n",
      "|            mtp|   439|     0|\n",
      "|           name|   451|     0|\n",
      "|    netbios_dgm|   405|     0|\n",
      "|     netbios_ns|   347|     0|\n",
      "|    netbios_ssn|   362|     0|\n",
      "|        netstat|   360|     0|\n",
      "|           nnsp|   630|     0|\n",
      "|           nntp|   296|     0|\n",
      "|          ntp_u|     0|   168|\n",
      "|          other|  1755|  2604|\n",
      "|        pm_dump|     5|     0|\n",
      "|          pop_2|    78|     0|\n",
      "|          pop_3|    78|   186|\n",
      "|        printer|    69|     0|\n",
      "|        private| 20871|   982|\n",
      "|          red_i|     0|     8|\n",
      "|     remote_job|    78|     0|\n",
      "|            rje|    86|     0|\n",
      "|          shell|    61|     4|\n",
      "|           smtp|   284|  7029|\n",
      "|        sql_net|   245|     0|\n",
      "|            ssh|   306|     5|\n",
      "|         sunrpc|   381|     0|\n",
      "|         supdup|   544|     0|\n",
      "|         systat|   477|     0|\n",
      "|         telnet|  1436|   917|\n",
      "|         tftp_u|     0|     3|\n",
      "|          tim_i|     3|     5|\n",
      "|           time|   578|    76|\n",
      "|          urh_i|     0|    10|\n",
      "|          urp_i|     3|   599|\n",
      "|           uucp|   780|     0|\n",
      "|      uucp_path|   689|     0|\n",
      "|          vmnet|   617|     0|\n",
      "|          whois|   693|     0|\n",
      "+---------------+------+------+\n",
      "\n",
      "+---------------+-----+-----+---+---+------+\n",
      "|service_labels5|  DoS|Probe|R2L|U2R|normal|\n",
      "+---------------+-----+-----+---+---+------+\n",
      "|            IRC|    0|    1|  0|  0|   186|\n",
      "|            X11|    0|    6|  0|  0|    67|\n",
      "|         Z39_50|  851|   11|  0|  0|     0|\n",
      "|            aol|    0|    2|  0|  0|     0|\n",
      "|           auth|  703|   16|  0|  0|   236|\n",
      "|            bgp|  699|   11|  0|  0|     0|\n",
      "|        courier|  726|    8|  0|  0|     0|\n",
      "|       csnet_ns|  533|   12|  0|  0|     0|\n",
      "|            ctf|  538|   25|  0|  0|     0|\n",
      "|        daytime|  503|   18|  0|  0|     0|\n",
      "|        discard|  520|   18|  0|  0|     0|\n",
      "|         domain|  508|   23|  0|  0|    38|\n",
      "|       domain_u|    0|    9|  0|  0|  9034|\n",
      "|           echo|  416|   18|  0|  0|     0|\n",
      "|          eco_i|    0| 4089|  0|  0|   497|\n",
      "|          ecr_i| 2844|   43|  0|  0|   190|\n",
      "|            efs|  478|    7|  0|  0|     0|\n",
      "|           exec|  465|    9|  0|  0|     0|\n",
      "|         finger| 1168|   54|  0|  0|   545|\n",
      "|            ftp|  489|   32|312|  3|   918|\n",
      "|       ftp_data| 1209|   51|604| 12|  4984|\n",
      "|         gopher|  485|   33|  0|  0|     0|\n",
      "|        harvest|    0|    2|  0|  0|     0|\n",
      "|      hostnames|  447|   13|  0|  0|     0|\n",
      "|           http| 2255|   30|  4|  0| 38049|\n",
      "|      http_2784|    0|    1|  0|  0|     0|\n",
      "|       http_443|  523|    7|  0|  0|     0|\n",
      "|      http_8001|    0|    2|  0|  0|     0|\n",
      "|          imap4|  622|   11| 11|  0|     3|\n",
      "|       iso_tsap|  675|   12|  0|  0|     0|\n",
      "|         klogin|  425|    8|  0|  0|     0|\n",
      "|         kshell|  292|    7|  0|  0|     0|\n",
      "|           ldap|  403|    7|  0|  0|     0|\n",
      "|           link|  454|   21|  0|  0|     0|\n",
      "|          login|  420|    7|  2|  0|     0|\n",
      "|            mtp|  416|   23|  0|  0|     0|\n",
      "|           name|  428|   23|  0|  0|     0|\n",
      "|    netbios_dgm|  392|   13|  0|  0|     0|\n",
      "|     netbios_ns|  336|   11|  0|  0|     0|\n",
      "|    netbios_ssn|  349|   13|  0|  0|     0|\n",
      "|        netstat|  344|   16|  0|  0|     0|\n",
      "|           nnsp|  622|    8|  0|  0|     0|\n",
      "|           nntp|  281|   15|  0|  0|     0|\n",
      "|          ntp_u|    0|    0|  0|  0|   168|\n",
      "|          other|   58| 1689|  5|  3|  2604|\n",
      "|        pm_dump|    0|    5|  0|  0|     0|\n",
      "|          pop_2|   70|    8|  0|  0|     0|\n",
      "|          pop_3|   67|   11|  0|  0|   186|\n",
      "|        printer|   62|    7|  0|  0|     0|\n",
      "|        private|15971| 4900|  0|  0|   982|\n",
      "|          red_i|    0|    0|  0|  0|     8|\n",
      "|     remote_job|   60|   18|  0|  0|     0|\n",
      "|            rje|   68|   18|  0|  0|     0|\n",
      "|          shell|   53|    8|  0|  0|     4|\n",
      "|           smtp|  241|   43|  0|  0|  7029|\n",
      "|        sql_net|  233|   12|  0|  0|     0|\n",
      "|            ssh|  281|   25|  0|  0|     5|\n",
      "|         sunrpc|  369|   12|  0|  0|     0|\n",
      "|         supdup|  528|   16|  0|  0|     0|\n",
      "|         systat|  460|   17|  0|  0|     0|\n",
      "|         telnet| 1312|   33| 57| 34|   917|\n",
      "|         tftp_u|    0|    0|  0|  0|     3|\n",
      "|          tim_i|    3|    0|  0|  0|     5|\n",
      "|           time|  551|   27|  0|  0|    76|\n",
      "|          urh_i|    0|    0|  0|  0|    10|\n",
      "|          urp_i|    0|    3|  0|  0|   599|\n",
      "|           uucp|  769|   11|  0|  0|     0|\n",
      "|      uucp_path|  676|   13|  0|  0|     0|\n",
      "|          vmnet|  606|   11|  0|  0|     0|\n",
      "|          whois|  670|   23|  0|  0|     0|\n",
      "+---------------+-----+-----+---+---+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 'service' nominal column\n",
    "print(train_df.select(nominal_cols[1]).distinct().count())\n",
    "(train_df.crosstab(nominal_cols[1], 'labels2').sort(sql.asc(nominal_cols[1] + '_labels2')).show(n=70))\n",
    "(train_df.crosstab(nominal_cols[1], 'labels5').sort(sql.asc(nominal_cols[1] + '_labels5')).show(n=70))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11\n",
      "+------------+------+------+\n",
      "|flag_labels2|attack|normal|\n",
      "+------------+------+------+\n",
      "|         OTH|    35|    11|\n",
      "|         REJ|  8540|  2693|\n",
      "|        RSTO|  1343|   219|\n",
      "|      RSTOS0|   103|     0|\n",
      "|        RSTR|  2275|   146|\n",
      "|          S0| 34497|   354|\n",
      "|          S1|     4|   361|\n",
      "|          S2|     8|   119|\n",
      "|          S3|     4|    45|\n",
      "|          SF| 11552| 63393|\n",
      "|          SH|   269|     2|\n",
      "+------------+------+------+\n",
      "\n",
      "+------------+-----+-----+---+---+------+\n",
      "|flag_labels5|  DoS|Probe|R2L|U2R|normal|\n",
      "+------------+-----+-----+---+---+------+\n",
      "|         OTH|    0|   35|  0|  0|    11|\n",
      "|         REJ| 5671| 2869|  0|  0|  2693|\n",
      "|        RSTO| 1216|   80| 46|  1|   219|\n",
      "|      RSTOS0|    0|  103|  0|  0|     0|\n",
      "|        RSTR|   90| 2180|  5|  0|   146|\n",
      "|          S0|34344|  153|  0|  0|   354|\n",
      "|          S1|    2|    1|  1|  0|   361|\n",
      "|          S2|    5|    2|  1|  0|   119|\n",
      "|          S3|    0|    1|  3|  0|    45|\n",
      "|          SF| 4599| 5967|935| 51| 63393|\n",
      "|          SH|    0|  265|  4|  0|     2|\n",
      "+------------+-----+-----+---+---+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 'flag' nominal column\n",
    "print(train_df.select(nominal_cols[2]).distinct().count())\n",
    "(train_df.crosstab(nominal_cols[2], 'labels2').sort(sql.asc(nominal_cols[2] + '_labels2')).show())\n",
    "(train_df.crosstab(nominal_cols[2], 'labels5').sort(sql.asc(nominal_cols[2] + '_labels5')).show())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>summary</th>\n",
       "      <td>count</td>\n",
       "      <td>mean</td>\n",
       "      <td>stddev</td>\n",
       "      <td>min</td>\n",
       "      <td>max</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>land</th>\n",
       "      <td>125973</td>\n",
       "      <td>1.9845522453224102E-4</td>\n",
       "      <td>0.014086071671513094</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logged_in</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.3957355941352512</td>\n",
       "      <td>0.48901005300524175</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>root_shell</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.0013415573178379495</td>\n",
       "      <td>0.03660284383979861</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>su_attempted</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.0011034110483992602</td>\n",
       "      <td>0.04515438381386557</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>is_host_login</th>\n",
       "      <td>125973</td>\n",
       "      <td>7.938208981289641E-6</td>\n",
       "      <td>0.0028174827384191085</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>is_guest_login</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.009422654060790804</td>\n",
       "      <td>0.09661232709143104</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     0                      1                      2    3    4\n",
       "summary          count                   mean                 stddev  min  max\n",
       "land            125973  1.9845522453224102E-4   0.014086071671513094  0.0  1.0\n",
       "logged_in       125973     0.3957355941352512    0.48901005300524175  0.0  1.0\n",
       "root_shell      125973  0.0013415573178379495    0.03660284383979861  0.0  1.0\n",
       "su_attempted    125973  0.0011034110483992602    0.04515438381386557  0.0  2.0\n",
       "is_host_login   125973   7.938208981289641E-6  0.0028174827384191085  0.0  1.0\n",
       "is_guest_login  125973   0.009422654060790804    0.09661232709143104  0.0  1.0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Binary columns\n",
    "(train_df.select(binary_cols).describe().toPandas().transpose())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+------+------+\n",
      "|su_attempted_labels2|attack|normal|\n",
      "+--------------------+------+------+\n",
      "|                 2.0|     0|    59|\n",
      "|                 1.0|     1|    20|\n",
      "|                 0.0| 58629| 67264|\n",
      "+--------------------+------+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 'su_attempted' should be a binary feature, but has 3 values\n",
    "(train_df.crosstab('su_attempted', 'labels2').show())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# '2.0' value is replaced to '0.0' for both train and test datasets\n",
    "train_df = train_df.replace(2.0, 0.0, 'su_attempted')\n",
    "test_df = test_df.replace(2.0, 0.0, 'su_attempted')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>summary</th>\n",
       "      <td>count</td>\n",
       "      <td>mean</td>\n",
       "      <td>stddev</td>\n",
       "      <td>min</td>\n",
       "      <td>max</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>duration</th>\n",
       "      <td>125973</td>\n",
       "      <td>287.1446500440571</td>\n",
       "      <td>2604.515309867592</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42908.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>src_bytes</th>\n",
       "      <td>125973</td>\n",
       "      <td>45566.74300048423</td>\n",
       "      <td>5870331.181893551</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.379963888E9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dst_bytes</th>\n",
       "      <td>125973</td>\n",
       "      <td>19779.114421344257</td>\n",
       "      <td>4021269.1514414474</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.309937401E9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wrong_fragment</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.022687401268525795</td>\n",
       "      <td>0.25352998595201254</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>urgent</th>\n",
       "      <td>125973</td>\n",
       "      <td>1.1113492573805498E-4</td>\n",
       "      <td>0.014366026620154243</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hot</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.20440888126820828</td>\n",
       "      <td>2.1499684337047587</td>\n",
       "      <td>0.0</td>\n",
       "      <td>77.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_failed_logins</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.0012224841831186047</td>\n",
       "      <td>0.045239138981329835</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_compromised</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.279250315543807</td>\n",
       "      <td>23.942042242795125</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7479.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_root</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.30219173949973405</td>\n",
       "      <td>24.3996180888374</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7468.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_file_creations</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.012669381534138267</td>\n",
       "      <td>0.48393506939604286</td>\n",
       "      <td>0.0</td>\n",
       "      <td>43.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_shells</th>\n",
       "      <td>125973</td>\n",
       "      <td>4.1278686702706137E-4</td>\n",
       "      <td>0.02218112867869418</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_access_files</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.004096115834345455</td>\n",
       "      <td>0.09936955575066156</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_outbound_cmds</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>125973</td>\n",
       "      <td>84.1075547934875</td>\n",
       "      <td>114.50860735418405</td>\n",
       "      <td>0.0</td>\n",
       "      <td>511.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>srv_count</th>\n",
       "      <td>125973</td>\n",
       "      <td>27.737888277646796</td>\n",
       "      <td>72.6358396472384</td>\n",
       "      <td>0.0</td>\n",
       "      <td>511.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>serror_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.2844845323997998</td>\n",
       "      <td>0.4464556243310233</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>srv_serror_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.2824853738499519</td>\n",
       "      <td>0.44702249836401703</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rerror_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.11995848316702805</td>\n",
       "      <td>0.3204355207495171</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>srv_rerror_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.12118326943075099</td>\n",
       "      <td>0.3236472280054629</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>same_srv_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.660927659101567</td>\n",
       "      <td>0.4396228624074799</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>diff_srv_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.06305263826375185</td>\n",
       "      <td>0.18031440750857483</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>srv_diff_host_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.0973216482897124</td>\n",
       "      <td>0.25983049812115877</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dst_host_count</th>\n",
       "      <td>125973</td>\n",
       "      <td>182.14894461511594</td>\n",
       "      <td>99.20621303459785</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <td>125973</td>\n",
       "      <td>115.65300500900987</td>\n",
       "      <td>110.7027407808648</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.521241694648872</td>\n",
       "      <td>0.4489493637176792</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.08295110857087822</td>\n",
       "      <td>0.18892179990461458</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.14837885896184153</td>\n",
       "      <td>0.30899713037298737</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.03254244957252654</td>\n",
       "      <td>0.11256380488118982</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.2844524620355186</td>\n",
       "      <td>0.44478405031648904</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.2784845165233854</td>\n",
       "      <td>0.4456691238860301</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.11883181316631297</td>\n",
       "      <td>0.3065574580251695</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <td>125973</td>\n",
       "      <td>0.12023989267541413</td>\n",
       "      <td>0.3194593904552316</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  0                      1  \\\n",
       "summary                       count                   mean   \n",
       "duration                     125973      287.1446500440571   \n",
       "src_bytes                    125973      45566.74300048423   \n",
       "dst_bytes                    125973     19779.114421344257   \n",
       "wrong_fragment               125973   0.022687401268525795   \n",
       "urgent                       125973  1.1113492573805498E-4   \n",
       "hot                          125973    0.20440888126820828   \n",
       "num_failed_logins            125973  0.0012224841831186047   \n",
       "num_compromised              125973      0.279250315543807   \n",
       "num_root                     125973    0.30219173949973405   \n",
       "num_file_creations           125973   0.012669381534138267   \n",
       "num_shells                   125973  4.1278686702706137E-4   \n",
       "num_access_files             125973   0.004096115834345455   \n",
       "num_outbound_cmds            125973                    0.0   \n",
       "count                        125973       84.1075547934875   \n",
       "srv_count                    125973     27.737888277646796   \n",
       "serror_rate                  125973     0.2844845323997998   \n",
       "srv_serror_rate              125973     0.2824853738499519   \n",
       "rerror_rate                  125973    0.11995848316702805   \n",
       "srv_rerror_rate              125973    0.12118326943075099   \n",
       "same_srv_rate                125973      0.660927659101567   \n",
       "diff_srv_rate                125973    0.06305263826375185   \n",
       "srv_diff_host_rate           125973     0.0973216482897124   \n",
       "dst_host_count               125973     182.14894461511594   \n",
       "dst_host_srv_count           125973     115.65300500900987   \n",
       "dst_host_same_srv_rate       125973      0.521241694648872   \n",
       "dst_host_diff_srv_rate       125973    0.08295110857087822   \n",
       "dst_host_same_src_port_rate  125973    0.14837885896184153   \n",
       "dst_host_srv_diff_host_rate  125973    0.03254244957252654   \n",
       "dst_host_serror_rate         125973     0.2844524620355186   \n",
       "dst_host_srv_serror_rate     125973     0.2784845165233854   \n",
       "dst_host_rerror_rate         125973    0.11883181316631297   \n",
       "dst_host_srv_rerror_rate     125973    0.12023989267541413   \n",
       "\n",
       "                                                2    3              4  \n",
       "summary                                    stddev  min            max  \n",
       "duration                        2604.515309867592  0.0        42908.0  \n",
       "src_bytes                       5870331.181893551  0.0  1.379963888E9  \n",
       "dst_bytes                      4021269.1514414474  0.0  1.309937401E9  \n",
       "wrong_fragment                0.25352998595201254  0.0            3.0  \n",
       "urgent                       0.014366026620154243  0.0            3.0  \n",
       "hot                            2.1499684337047587  0.0           77.0  \n",
       "num_failed_logins            0.045239138981329835  0.0            5.0  \n",
       "num_compromised                23.942042242795125  0.0         7479.0  \n",
       "num_root                         24.3996180888374  0.0         7468.0  \n",
       "num_file_creations            0.48393506939604286  0.0           43.0  \n",
       "num_shells                    0.02218112867869418  0.0            2.0  \n",
       "num_access_files              0.09936955575066156  0.0            9.0  \n",
       "num_outbound_cmds                             0.0  0.0            0.0  \n",
       "count                          114.50860735418405  0.0          511.0  \n",
       "srv_count                        72.6358396472384  0.0          511.0  \n",
       "serror_rate                    0.4464556243310233  0.0            1.0  \n",
       "srv_serror_rate               0.44702249836401703  0.0            1.0  \n",
       "rerror_rate                    0.3204355207495171  0.0            1.0  \n",
       "srv_rerror_rate                0.3236472280054629  0.0            1.0  \n",
       "same_srv_rate                  0.4396228624074799  0.0            1.0  \n",
       "diff_srv_rate                 0.18031440750857483  0.0            1.0  \n",
       "srv_diff_host_rate            0.25983049812115877  0.0            1.0  \n",
       "dst_host_count                  99.20621303459785  0.0          255.0  \n",
       "dst_host_srv_count              110.7027407808648  0.0          255.0  \n",
       "dst_host_same_srv_rate         0.4489493637176792  0.0            1.0  \n",
       "dst_host_diff_srv_rate        0.18892179990461458  0.0            1.0  \n",
       "dst_host_same_src_port_rate   0.30899713037298737  0.0            1.0  \n",
       "dst_host_srv_diff_host_rate   0.11256380488118982  0.0            1.0  \n",
       "dst_host_serror_rate          0.44478405031648904  0.0            1.0  \n",
       "dst_host_srv_serror_rate       0.4456691238860301  0.0            1.0  \n",
       "dst_host_rerror_rate           0.3065574580251695  0.0            1.0  \n",
       "dst_host_srv_rerror_rate       0.3194593904552316  0.0            1.0  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Numeric columns\n",
    "print(len(numeric_cols))\n",
    "(train_df.select(numeric_cols).describe().toPandas().transpose())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------------+------+------+\n",
      "|num_outbound_cmds_labels2|attack|normal|\n",
      "+-------------------------+------+------+\n",
      "|                      0.0| 58630| 67343|\n",
      "+-------------------------+------+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "(train_df.crosstab('num_outbound_cmds', 'labels2').show())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As 'num_outbound_cmds' feature takes only 0.0 values, so it is dropped as redundant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_df = train_df.drop('num_outbound_cmds')\n",
    "test_df = test_df.drop('num_outbound_cmds')\n",
    "numeric_cols.remove('num_outbound_cmds')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Commented code below is related to removing highly correlated features. However, it hasen't been tested a lot yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# from pyspark.mllib.stat import Statistics\n",
    "# from pyspark.mllib.linalg import Vectors\n",
    "# from pyspark.ml.feature import VectorAssembler\n",
    "\n",
    "# t0 = time()\n",
    "# stat_assembler = VectorAssembler(\n",
    "#                 inputCols=numeric_cols,\n",
    "#                 outputCol='features')\n",
    "\n",
    "# stat_rdd = stat_assembler.transform(train_df).rdd.map(lambda row: row['features'].toArray())\n",
    "\n",
    "# pearson_corr = Statistics.corr(stat_rdd, method='pearson')\n",
    "# spearman_corr = Statistics.corr(stat_rdd, method='spearman')\n",
    "\n",
    "# print(time() - t0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# f, (ax1, ax2) = plt.subplots(1, 2, figsize=(18, 6))\n",
    "\n",
    "# ax1.set_title(\"Pearson\")\n",
    "# ax2.set_title(\"Spearman\")\n",
    "# sns.heatmap(pearson_corr, ax=ax1)\n",
    "# sns.heatmap(spearman_corr, ax=ax2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# inx_correlated_to_delete = [8, 15, 28, 17, 29]\n",
    "\n",
    "# for inx in inx_correlated_to_delete:\n",
    "#     train_df = train_df.drop(numeric_cols[inx])\n",
    "#     test_df = test_df.drop(numeric_cols[inx])\n",
    "\n",
    "# numeric_cols = [col for inx, col in enumerate(numeric_cols) if inx not in inx_correlated_to_delete]\n",
    "\n",
    "# train_df = train_df.cache()\n",
    "# test_df = test_df.cache()\n",
    "# print(train_df.count())\n",
    "# print(test_df.count())\n",
    "# print(len(numeric_cols))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. One Hot Encoding for categorical variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One Hot Encoding (OHE) is used for treating categorical variables. Custom function is created for demonstration purposes. However, it could be easily replaced by PySpark OneHotEncoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def ohe_vec(cat_dict, row):\n",
    "    vec = np.zeros(len(cat_dict))\n",
    "    vec[cat_dict[row]] = float(1.0)\n",
    "    return vec.tolist()\n",
    "\n",
    "def ohe(df, nominal_col):\n",
    "    categories = (df.select(nominal_col)\n",
    "                    .distinct()\n",
    "                    .rdd.map(lambda row: row[0])\n",
    "                    .collect())\n",
    "    \n",
    "    cat_dict = dict(zip(categories, range(len(categories))))\n",
    "    \n",
    "    udf_ohe_vec = udf(lambda row: ohe_vec(cat_dict, row), \n",
    "                      StructType([StructField(cat, DoubleType(), False) for cat in categories]))\n",
    "    \n",
    "    df = df.withColumn(nominal_col + '_ohe', udf_ohe_vec(col(nominal_col))).cache()\n",
    "    \n",
    "    nested_cols = [nominal_col + '_ohe.' + cat for cat in categories]\n",
    "    ohe_cols = [nominal_col + '_' + cat for cat in categories]\n",
    "        \n",
    "    for new, old in zip(ohe_cols, nested_cols):\n",
    "        df = df.withColumn(new, col(old))\n",
    "\n",
    "    df = df.drop(nominal_col + '_ohe')\n",
    "                   \n",
    "    return df, ohe_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of examples in train set: 125973\n",
      "Time: 10.34s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "train_ohe_cols = []\n",
    "\n",
    "train_df, train_ohe_col0 = ohe(train_df, nominal_cols[0])\n",
    "train_ohe_cols += train_ohe_col0\n",
    "\n",
    "train_df, train_ohe_col1 = ohe(train_df, nominal_cols[1])\n",
    "train_ohe_cols += train_ohe_col1\n",
    "\n",
    "train_df, train_ohe_col2 = ohe(train_df, nominal_cols[2])\n",
    "train_ohe_cols += train_ohe_col2\n",
    "\n",
    "binary_cols += train_ohe_cols\n",
    "\n",
    "train_df = train_df.cache()\n",
    "print(f\"Number of examples in train set: {train_df.count()}\")\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Custom list of test binary cols is used as test dataset could contain additional categories for 'service' and 'flag' features. However, those additional categories aren't used below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of examples in test set: 22544\n",
      "Time: 6.61s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "test_ohe_cols = []\n",
    "\n",
    "test_df, test_ohe_col0_names = ohe(test_df, nominal_cols[0])\n",
    "test_ohe_cols += test_ohe_col0_names\n",
    "\n",
    "test_df, test_ohe_col1_names = ohe(test_df, nominal_cols[1])\n",
    "test_ohe_cols += test_ohe_col1_names\n",
    "\n",
    "test_df, test_ohe_col2_names = ohe(test_df, nominal_cols[2])\n",
    "test_ohe_cols += test_ohe_col2_names\n",
    "\n",
    "test_binary_cols = col_names[binary_inx].tolist() + test_ohe_cols\n",
    "\n",
    "test_df = test_df.cache()\n",
    "print(f\"Number of examples in test set: {test_df.count()}\")\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Feature Selection using Attribute Ratio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attribute Ratio approach is used for feature selection purposes. This approach was described by Hee-su Chae and Sang Hyun Choi in [Feature Selection for efficient Intrusion Detection using Attribute Ratio](http://www.naun.org/main/UPress/cc/2014/a102019-106.pdf) and [Feature Selection for Intrusion Detection using NSL-KDD](http://www.wseas.us/e-library/conferences/2013/Nanjing/ACCIS/ACCIS-30.pdf)\n",
    "\n",
    "This approach is also used for nominal variables as they were encoded as binary variables above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it is a possible to have 'null' values because binary features could have Frequency(0) = 0, those 'null' values are replaced with 1000.0 (magic number). For NSL KDD dataset it is related only for 'protocol_type_tcp' ohe variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def getAttributeRatio(df, numericCols, binaryCols, labelCol):\n",
    "    ratio_dict = {}\n",
    "    \n",
    "    if numericCols:\n",
    "        avg_dict = (df\n",
    "                .select(list(map(lambda c: sql.avg(c).alias(c), numericCols)))\n",
    "                .first()\n",
    "                .asDict())\n",
    "\n",
    "        ratio_dict.update(df\n",
    "                .groupBy(labelCol)\n",
    "                .avg(*numericCols)\n",
    "                .select(list(map(lambda c: sql.max(col('avg(' + c + ')')/avg_dict[c]).alias(c), numericCols)))\n",
    "                .fillna(0.0)\n",
    "                .first()\n",
    "                .asDict())\n",
    "    \n",
    "    if binaryCols:\n",
    "        ratio_dict.update((df\n",
    "                .groupBy(labelCol)\n",
    "                .agg(*list(map(lambda c: (sql.sum(col(c))/(sql.count(col(c)) - sql.sum(col(c)))).alias(c), binaryCols)))\n",
    "                .fillna(1000.0)\n",
    "                .select(*list(map(lambda c: sql.max(col(c)).alias(c), binaryCols)))\n",
    "                .first()\n",
    "                .asDict()))\n",
    "        \n",
    "    return OrderedDict(sorted(ratio_dict.items(), key=lambda v: -v[1]))\n",
    "\n",
    "def selectFeaturesByAR(ar_dict, min_ar):\n",
    "    return [f for f in ar_dict.keys() if ar_dict[f] >= min_ar]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of features in Attribute Ration dict: 121\n",
      "Time: 7.35s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "OrderedDict([('protocol_type_tcp', 1000.0),\n",
       "             ('num_shells', 326.11353550295854),\n",
       "             ('urgent', 173.03983516483518),\n",
       "             ('num_file_creations', 62.23362492770388),\n",
       "             ('flag_SF', 51.0),\n",
       "             ('num_failed_logins', 46.03855641845592),\n",
       "             ('hot', 40.77451681709518),\n",
       "             ('logged_in', 10.569767441860465),\n",
       "             ('dst_bytes', 9.154854355343401),\n",
       "             ('src_bytes', 8.464064204948945),\n",
       "             ('duration', 7.225829157212557),\n",
       "             ('dst_host_srv_diff_host_rate', 5.756880682756574),\n",
       "             ('dst_host_diff_srv_rate', 4.83734184897426),\n",
       "             ('num_access_files', 4.694879248658319),\n",
       "             ('dst_host_same_src_port_rate', 4.393080378884017),\n",
       "             ('num_compromised', 4.338539274983927),\n",
       "             ('diff_srv_rate', 4.069085485070395),\n",
       "             ('dst_host_srv_rerror_rate', 3.667920527965924),\n",
       "             ('srv_rerror_rate', 3.667741802325429),\n",
       "             ('rerror_rate', 3.645586087828447),\n",
       "             ('dst_host_rerror_rate', 3.2795669242444494),\n",
       "             ('srv_diff_host_rate', 3.0815657101103984),\n",
       "             ('flag_S0', 2.965034965034965),\n",
       "             ('wrong_fragment', 2.742896335488928),\n",
       "             ('dst_host_srv_serror_rate', 2.6731595957140732),\n",
       "             ('srv_serror_rate', 2.643246318490161),\n",
       "             ('serror_rate', 2.6310546426370265),\n",
       "             ('dst_host_serror_rate', 2.6293396511768043),\n",
       "             ('num_root', 2.6091432537726016),\n",
       "             ('count', 2.1174082949142403),\n",
       "             ('service_telnet', 1.8888888888888888),\n",
       "             ('dst_host_srv_count', 1.6453161847397422),\n",
       "             ('dst_host_same_srv_rate', 1.557578827974319),\n",
       "             ('service_ftp_data', 1.5447570332480818),\n",
       "             ('same_srv_rate', 1.5079612006047083),\n",
       "             ('dst_host_count', 1.3428596865228266),\n",
       "             ('service_http', 1.2988666621151088),\n",
       "             ('srv_count', 1.1773191099992069),\n",
       "             ('root_shell', 1.0),\n",
       "             ('service_private', 0.7252812314979278),\n",
       "             ('protocol_type_icmp', 0.5497939103842574),\n",
       "             ('service_eco_i', 0.5403726708074534),\n",
       "             ('is_guest_login', 0.45894428152492667),\n",
       "             ('service_ftp', 0.4568081991215227),\n",
       "             ('flag_REJ', 0.3265050642995334),\n",
       "             ('flag_RSTR', 0.23005487547488393),\n",
       "             ('protocol_type_udp', 0.22644739478045495),\n",
       "             ('service_other', 0.16945921541085582),\n",
       "             ('service_domain_u', 0.15493320070658045),\n",
       "             ('service_smtp', 0.11654010677454654),\n",
       "             ('service_ecr_i', 0.06601211614790056),\n",
       "             ('flag_RSTO', 0.04847207586933614),\n",
       "             ('service_finger', 0.026095310440358364),\n",
       "             ('flag_SH', 0.02326398033535247),\n",
       "             ('service_Z39_50', 0.018879226195758277),\n",
       "             ('service_uucp', 0.01702909783427078),\n",
       "             ('service_courier', 0.0160615915577089),\n",
       "             ('service_auth', 0.015544843445957898),\n",
       "             ('service_bgp', 0.0154550278588485),\n",
       "             ('service_uucp_path', 0.014938896377980597),\n",
       "             ('service_iso_tsap', 0.014916467780429593),\n",
       "             ('service_whois', 0.014804339660163068),\n",
       "             ('service_nnsp', 0.013729168965897804),\n",
       "             ('service_imap4', 0.013729168965897804),\n",
       "             ('service_vmnet', 0.013371284834844774),\n",
       "             ('service_time', 0.012142983074753174),\n",
       "             ('service_ctf', 0.011853092158893123),\n",
       "             ('service_csnet_ns', 0.011741639864299247),\n",
       "             ('service_supdup', 0.011630212119209674),\n",
       "             ('service_http_443', 0.011518808915514052),\n",
       "             ('service_discard', 0.011451978769793203),\n",
       "             ('service_domain', 0.011184746471740902),\n",
       "             ('service_daytime', 0.01107344135258894),\n",
       "             ('service_gopher', 0.010672945733022314),\n",
       "             ('service_efs', 0.010517283108539242),\n",
       "             ('service_exec', 0.010228322555100963),\n",
       "             ('service_systat', 0.010117227879561),\n",
       "             ('service_link', 0.009983946517713808),\n",
       "             ('service_hostnames', 0.00982849604221636),\n",
       "             ('service_name', 0.009406800149453835),\n",
       "             ('service_klogin', 0.009340248780273395),\n",
       "             ('service_login', 0.009229349330872173),\n",
       "             ('service_mtp', 0.009140647316033486),\n",
       "             ('service_echo', 0.009140647316033486),\n",
       "             ('service_urp_i', 0.0089745894762076),\n",
       "             ('flag_RSTOS0', 0.008915433220808448),\n",
       "             ('service_ldap', 0.008852473420613302),\n",
       "             ('service_netbios_dgm', 0.008608762490392005),\n",
       "             ('service_sunrpc', 0.00809956538917424),\n",
       "             ('service_netbios_ssn', 0.007657203036552723),\n",
       "             ('service_netstat', 0.0075466731018142726),\n",
       "             ('service_netbios_ns', 0.007369875633348687),\n",
       "             ('service_kshell', 0.006398597567656404),\n",
       "             ('service_nntp', 0.006156070630504316),\n",
       "             ('service_ssh', 0.006156070630504316),\n",
       "             ('flag_S1', 0.005389507628915231),\n",
       "             ('service_sql_net', 0.005099137742373178),\n",
       "             ('flag_S3', 0.0030241935483870967),\n",
       "             ('flag_OTH', 0.0030117890026675844),\n",
       "             ('service_pop_3', 0.0027696293759399615),\n",
       "             ('service_IRC', 0.0027696293759399615),\n",
       "             ('service_ntp_u', 0.0025009304056568663),\n",
       "             ('flag_S2', 0.0017702011186481019),\n",
       "             ('service_remote_job', 0.0015466575012888812),\n",
       "             ('service_rje', 0.0015466575012888812),\n",
       "             ('service_pop_2', 0.0015264845061822622),\n",
       "             ('service_printer', 0.0013517933064428214),\n",
       "             ('service_shell', 0.0011553385359898854),\n",
       "             ('su_attempted', 0.001006036217303823),\n",
       "             ('service_X11', 0.0009958974968785302),\n",
       "             ('service_pm_dump', 0.0004291477126426916),\n",
       "             ('land', 0.00039207998431680063),\n",
       "             ('service_harvest', 0.00017161489617298782),\n",
       "             ('service_aol', 0.00017161489617298782),\n",
       "             ('service_http_8001', 0.00017161489617298782),\n",
       "             ('service_urh_i', 0.0001485155867108253),\n",
       "             ('service_red_i', 0.00011880894037276305),\n",
       "             ('service_http_2784', 8.58000858000858e-05),\n",
       "             ('service_tim_i', 7.425227954498203e-05),\n",
       "             ('service_tftp_u', 4.455004455004455e-05),\n",
       "             ('is_host_login', 1.4849573817231445e-05)])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t0 = time()\n",
    "ar_dict = getAttributeRatio(train_df, numeric_cols, binary_cols, 'labels5')\n",
    "\n",
    "print(f\"Number of features in Attribute Ration dict: {len(ar_dict)}\")\n",
    "print(f\"Time: {time() - t0:.2f}s\")\n",
    "ar_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standartization is necessary as a lot of distance based algorithms are used below. Custom standartization is created for demonstration purposes, so it could be easily replaced by PySpark StandardScaler. Note that data is sparse, so it is reasonable to not substract mean for avoiding violating sparsity. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "125973\n",
      "22544\n",
      "Time: 3.48s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "avg_dict = (train_df.select(list(map(lambda c: sql.avg(c).alias(c), numeric_cols))).first().asDict())\n",
    "std_dict = (train_df.select(list(map(lambda c: sql.stddev(c).alias(c), numeric_cols))).first().asDict())\n",
    "\n",
    "def standardizer(column):\n",
    "    return ((col(column) - avg_dict[column])/std_dict[column]).alias(column)\n",
    "\n",
    "# Standardizer without mean\n",
    "# def standardizer(column):\n",
    "#     return (col(column)/std_dict[column]).alias(column)\n",
    "\n",
    "train_scaler = [*binary_cols, *list(map(standardizer, numeric_cols)), *['id', 'labels2_index', 'labels2', 'labels5_index', 'labels5']]\n",
    "test_scaler = [*test_binary_cols, *list(map(standardizer, numeric_cols)), *['id', 'labels2_index', 'labels2', 'labels5_index', 'labels5']]\n",
    "\n",
    "scaled_train_df = (train_df.select(train_scaler).cache())\n",
    "scaled_test_df = (test_df.select(test_scaler).cache())\n",
    "\n",
    "print(scaled_train_df.count())\n",
    "print(scaled_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "VectorAssembler is used for combining a given list of columns into a single vector column. Then VectorIndexer is used for indexing categorical (binary) features. Indexing categorical features allows algorithms to treat them appropriately, improving performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import VectorIndexer, VectorAssembler\n",
    "assembler = VectorAssembler(inputCols=selectFeaturesByAR(ar_dict, 0.01), outputCol='raw_features')\n",
    "indexer = VectorIndexer(inputCol='raw_features', outputCol='indexed_features', maxCategories=2)\n",
    "\n",
    "prep_pipeline = Pipeline(stages=[assembler, indexer])\n",
    "prep_model = prep_pipeline.fit(scaled_train_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "125973\n",
      "22544\n",
      "Time: 1.40s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "scaled_train_df = (prep_model\n",
    "        .transform(scaled_train_df)\n",
    "        .select('id', 'indexed_features', 'labels2_index', 'labels2', 'labels5_index', 'labels5')\n",
    "        .cache())\n",
    "\n",
    "scaled_test_df = (prep_model \n",
    "        .transform(scaled_test_df)\n",
    "        .select('id', 'indexed_features','labels2_index', 'labels2', 'labels5_index', 'labels5')\n",
    "        .cache())\n",
    "\n",
    "print(scaled_train_df.count())\n",
    "print(scaled_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4667979835606274383\n"
     ]
    }
   ],
   "source": [
    "# Setting seed for reproducibility\n",
    "seed = 4667979835606274383\n",
    "print(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The train dataset is splitted into 80% train and 20% cross-validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100840\n",
      "25133\n"
     ]
    }
   ],
   "source": [
    "split = (scaled_train_df.randomSplit([0.8, 0.2], seed=seed))\n",
    "\n",
    "scaled_train_df = split[0].cache()\n",
    "scaled_cv_df = split[1].cache()\n",
    "\n",
    "print(scaled_train_df.count())\n",
    "print(scaled_cv_df.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional \"result\" dataframes are used to collect probabilities and predictions from different approaches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "22544\n"
     ]
    }
   ],
   "source": [
    "res_cv_df = scaled_cv_df.select(col('id'), col('labels2_index'), col('labels2'), col('labels5')).cache()\n",
    "res_test_df = scaled_test_df.select(col('id'), col('labels2_index'), col('labels2'), col('labels5')).cache()\n",
    "prob_cols = []\n",
    "pred_cols = []\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(res_test_df.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Different metrics from sklearn are used for evaluating results. The most important from them for this task are False positive Rate, Detection Rate and F1 score. \n",
    "As evaluating via sklearn requires to collect predicted and label columns to the driver, it will be replaced with PySpark metrics later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sklearn.metrics as metrics\n",
    "\n",
    "def printCM(cm, labels):\n",
    "    \"\"\"pretty print for confusion matrixes\"\"\"\n",
    "    columnwidth = max([len(x) for x in labels])\n",
    "    # Print header\n",
    "    print(\" \" * columnwidth, end=\"\\t\")\n",
    "    for label in labels:\n",
    "        print(\"%{0}s\".format(columnwidth) % label, end=\"\\t\")\n",
    "    print()\n",
    "    # Print rows\n",
    "    for i, label1 in enumerate(labels):\n",
    "        print(\"%{0}s\".format(columnwidth) % label1, end=\"\\t\")\n",
    "        for j in range(len(labels)):\n",
    "            print(\"%{0}d\".format(columnwidth) % cm[i, j], end=\"\\t\")\n",
    "        print()\n",
    "\n",
    "def getPrediction(e):\n",
    "    return udf(lambda row: 1.0 if row >= e else 0.0, DoubleType())\n",
    "        \n",
    "def printReport(resDF, probCol, labelCol='labels2_index', e=None, labels=['normal', 'attack']):\n",
    "    if (e):\n",
    "        predictionAndLabels = list(zip(*resDF.rdd\n",
    "                                       .map(lambda row: (1.0 if row[probCol] >= e else 0.0, row[labelCol]))\n",
    "                                       .collect()))\n",
    "    else:\n",
    "        predictionAndLabels = list(zip(*resDF.rdd\n",
    "                                       .map(lambda row: (row[probCol], row[labelCol]))\n",
    "                                       .collect()))\n",
    "    \n",
    "    cm = metrics.confusion_matrix(predictionAndLabels[1], predictionAndLabels[0])\n",
    "    printCM(cm, labels)\n",
    "    print(\" \")\n",
    "    print(\"Accuracy = %g\" % (metrics.accuracy_score(predictionAndLabels[1], predictionAndLabels[0])))\n",
    "    print(\"AUC = %g\" % (metrics.roc_auc_score(predictionAndLabels[1], predictionAndLabels[0])))\n",
    "    print(\" \")\n",
    "    print(\"False Alarm Rate = %g\" % (cm[0][1]/(cm[0][0] + cm[0][1])))\n",
    "    print(\"Detection Rate = %g\" % (cm[1][1]/(cm[1][1] + cm[1][0])))\n",
    "    print(\"F1 score = %g\" % (metrics.f1_score(predictionAndLabels[1], predictionAndLabels[0], labels)))\n",
    "    print(\" \")\n",
    "    print(metrics.classification_report(predictionAndLabels[1], predictionAndLabels[0]))\n",
    "    print(\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Visualization via PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PCA algorithm is used for visualization purposes. It's also used later as preprocessing for Gaussian Mixture clustering.\n",
    "\n",
    "First graph shows 'attack' vs 'normal' labels, second graph shows 4 different types of attacks vs normal connections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time: 1.78s\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.feature import VectorSlicer\n",
    "from pyspark.ml.feature import PCA\n",
    "\n",
    "t0 = time()\n",
    "pca_slicer = VectorSlicer(inputCol=\"indexed_features\", outputCol=\"features\", names=selectFeaturesByAR(ar_dict, 0.05))\n",
    "\n",
    "pca = PCA(k=2, inputCol=\"features\", outputCol=\"pca_features\")\n",
    "pca_pipeline = Pipeline(stages=[pca_slicer, pca])\n",
    "\n",
    "pca_train_df = pca_pipeline.fit(scaled_train_df).transform(scaled_train_df)\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOy9eXgU15X3/6mlN20IbUhICCQWsYtFgNkCeMUYbIydYDseO/E2M8lMnJnMO2ved2be95lJZsskM0kmP2wnsYMdMMY2GAMGjLHZ9x2xCQmxaEG71Oqtqu7vj5YaCXVL3doF9XmePHFX3bp1W0inTp17zvdIQghMTExMTAYucl8vwMTExMSka5iG3MTExGSAYxpyExMTkwGOachNTExMBjimITcxMTEZ4Kh9cdOkpCQxYsSIvri1iYmJyYDl6NGjFUKI5DuP94khHzFiBEeOHOmLW5uYmJgMWCRJuhrsuBlaMTExMRngmIbcxMTEZIBjGnITExOTAY5pyE1MTEwGOKYhNzExMRng9EnWionJ3URhYSFffvklXq8XgOTkZJYsWYLNZuvjlZncK5geuYlJFygtLWX79u0BIw5w69Yt1q5di6ksatJbmIbc5J7EMIxuMbR79uwJetztdlNcXNzl+U1MwsEMrZjcUxQXF7Nv3z7q6uqwWq1MmjSJadOmIUlSp+arr68Pea68vJzhw4d3dqkmJmFjeuQm9wwlJSVs27aNuro6ALxeL8eOHWP//v2dnjMmJibkuaSkpE7Pa2ISCd3ikUuS9GfAK4AATgPfFkK4u2Nuk+A0NDRQVFSEw+Fg+PDhqKr5ctUR+/btwzCMVseEEJw9e5aZM2eG9TNcvXo1jY2Ngc92uz3oOKvViqknZNJbdPmvX5KkdOB7wHghhEuSpPeBZ4DfdnVuk9a4XC4OHjzIxYsX25xbsmQJGRkZfbCqgUN1dXXQ40IIqqurSU5uo0XUilWrVrU55na39Veio6N58sknOx2uMTGJlO4KraiAQ5IkFYgCbnbTvCZNFBUV8bvf/S6oEQfYvHkzmqb18qp6Hp/Px9WrV7l27Rq6rndprt4yrE6nk9WrV3Pq1KleuZ+JSZc9ciHEDUmS/h0oBlzANiHEtjvHSZL0GvAaQGZmZldve08hhGDbtjY/0jYUFhYyevRowJ+V0dDQgMPhwGKx9PQSe4TDhw9z/PjxoOcee+wxkpKSIsrVHjp0KNeuXWtzXJIkBg8e3O61u3fvDvs+zRw4cICYmBiys7MjvtbEJBK6I7QyGHgCyAJqgHWSJD0vhFjdcpwQYhWwCiAvL89MsI2A/Pz8sMaVlJQwevRo1qxZE9jQa8mrr746YF73z5w5E9KIA3z66acBA7xgwYIOwyIAc+fOZd26da08e0mSwoqP19bWhr/4FnzxxRemITfpcbojtPIgUCiEuCWE8AEfAnO6YV4T/N54qFzlO6mrq+Ozzz4LasQB3njjDc6dO9edy+sRvF4v+/bt63CcEIKqqio2bdrUagMyFHFxcTz99NOMHDmS6OhokpKSePDBB8nNze3w2gcffDCstd9JV8NBJibh0B2pDsXAfZIkReEPrTwAmF0juol333037LE3b3a8NbFnzx7Onj3LwoULw/Ji+4KDBw9GNF7XdfLz85k+fXqHYwcNGsQDDzwQ8ZpCZaeYmPQHuuyRCyEOAh8Ax/CnHso0hVBMOo9hGKxatSosTzNSqqur+eijj4JmYfQHwnkgtcQwDGpqanpoNbcZN25cxNeYeismvUG3ZK0IIf5eCDFWCDFRCPEHQghPd8x7L7N27dpeuc+qVatYv359rxjCcImKiopovKqqDBkypIdWc5vOVGkmJib2wEpMTFpjVnb2Q7xeb7ul391NZWUl69at6zfGPC8vL+yxkiRhtVoZM2ZMD67Iz9atWyO+Jjo6ugdWYmLSGtOQ90PWrFnT6/cUQnD48OFev++dGIaBrusoihLW+DFjxrBixQqsVmsPr8zEpP9i1nX3M3RdD1ot2BsUFRX1yX2baWhoYOPGjXg8nrCyPWJiYliwYEEvrKzzpKSk9PUSTO4BTI+8n3Hy5Mk+u7cQgqNHj1JeXt4n99+xYwdOpxOfzxfW+N7eSHzooYcivsY05Ca9gemR9zOOHOnbzM2jR49y9OhR7HY7zz//PLLcO8/6xsZGKisrI9II7+1wyvbt2yO+5m6UTTDpf5geuUlQ3G43b7/9dq/drzOFM2VlZf2+C4+ZtWLSG5iGvB/R3/K6fT4fJSUlvXKvmJgYHA5HRNfcKUnb32jOqDEx6WlMQ95PeP/997t0vc1mizj/Ohw++eSTTl2naVpEhlaSJO6//35UVY0onNOb2jGRhpnMxhImvYUZI+8HeL3esHO4n3zySQ4cONDKUx49ejSLFi0C/K3Hbty4QXR0NPv37++W3PA9e/Ywb968wGeXy0V9fT1CCGJiYlrlSpeWlrJ7925qamqQZZnRo0czZ86csJo2pKamsnLlSs6fP09dXR2XLl3q8Bqv19trXu/LL7/MG2+8EdZYRVGYMWNGD6/IxMSPacj7Ab/97W/DGpebm8vGjRsD8WRJklAUhYkTJwbGxMbGMnbsWACGDRsG+NP63nvvvU6v79y5c8ydOxen0xnRPLquc+nSJRobG1m8eHFY10RHRwc0U8Ix5HV1db3m+Ubi/U+YMCHiRh+6rmMYxoCVHTbpO0xD3seEahQRjKKiolabgkIINE1j//79PP744yGvi4mJwW63dyk//d133+2U7ouu6xQXF/Pee++Rl5eH3W7HYrEgSRIpKSldzorZv38/y5Yt69IckfD888+zevXqDsedPXuWSZMmdVjZWVFRwZ49e1qlfCYkJIQtzWtiAmaMvM/Zu3dvWOOeeeaZkJrYt27d6vD6F154oUvx5K6KdzU0NLBr1y62bt3KJ598wsaNG/ntb3/L9evXuzRvaWlpr2auhLsPYRgGBQUF7Y6pqalhw4YNbfL2m6V5GxoaOr1Ok3sL05D3IQ0NDWEVv8yePZv169eHPB9uYcyrr77KxIkT+01zCU3T2Lx5M06ns9NzCCF6PQUxHPmA5j6g7XHixImQaZe6rg8I7XiT/oFpyPuI8vLysBQObTYbJ06cCGnwVVVl8uTJYd93zpw5vPrqq2FrmfQGhw4d6vS1Vqu114qWmhk5cmRY4yorK9s9X1FREfJcb0nzmtwdmDHyPkDTNDZt2tRhEYzNZmPx4sV8+umnIceMHz+eSZMmRbyGF154gd/85jcRX9cTXLp0iVmzZnUqfTLS3PNIMAyDmzdvUl9fT21tLdeuXUNV1bBCWeA31BUVFSE3YwcPHkxVVVXQc4qiBMr76+rq2Lx5c6vOT1FRUWiahtfrDRyTJImoqCjS09OxWCykpKSQlZUVVsaQycDG/BfuA/bu3dtu6baiKNx///1kZWVx4sSJkGOHDBnCfffd16k1WCwWhgwZQllZWaeu724OHTrEwoULI74uPT29+xcDOJ1OPv744y6FfcDfWzTU/sTUqVO5evVq0H9fi8XC2LFjWbt2bdC9kWB7FkIInE5nYAP9/PnzHDlyhOXLl/foA8+k7zFDK31Ae5tgDoeDl19+maysLNxuN8eOHQs6TlGUQJphZ3niiSe6dH13UlhY2OZYOLH/adOmdftaDMPg/fff77IRB/B4PCEflgkJCTz66KMMHjw4cEyWZbKzs1mxYgUXLlzodNNn8MfZnU5nl0JXJgMD0yPvA9oLqbTMCW9PVjYqKorRo0d3eS25ubl9qrjYTDCvtGXYIBgzZ87skWrWbdu2ha3AGA5btmzh+eefD5ofnpaWxte//nUMw0CSpFaee3cYYMMwuHDhAoWFhSQmJjJr1qw2iowXL15k165dgc8jR47sVF9Tk77DNOR9QFpaWtC+lHa7PdDRvbGxsd2wSnZ2drds8s2aNYuKigpu3LjRqeuXLl3K0KFDASgpKeGLL77oVNqcEILy8vJWRqajbBSPx8OVK1fweDykpaURHx8f8X3vxOv1Ulxc3OV5WuLz+Thw4ADz588POSbYv2V3ZuN4vV5KSkrYtGkTS5cuJSUlhQsXLgQN8xUUFFBeXs6zzz7bbfc36VnM0Eov4vV6aWxsZO7cuVit1kDmiCzLWCwWli1bhizLCCHabG61RFXVTvWPDMVjjz0WKPGPlE2bNgX+Oy0tjeeee47XXnuN5557LuLO85Hqupw8eZJdu3axf/9+1q9fz+7du7ts/DrKNOksV65c6ZF5I0XTNA4fPsyePXv48ssvQzoK9fX1eDxm692BgumR9wJut5tdu3YFil+aO9tUVlYGshrGjx8fCBNUVFSEjI1KkkR2dna3NxsePXo0DoeDLVu2RGwM33zzTWbPns348eMDoYGYmBheeOGFwJj6+nrWrFnT7ty6rnPz5s2Ahx8OLQ3RpUuXSE9PJzs7O6L1t8ThcCBJUrfnpvcnpcZw38COHz/e6c10k97F9Mh7mGbv+vr16xiGgWEY1NXV8cUXXzBmzBgWL15MXl5ewIi73W6++OKLkHH0xMREFixY0CNFPRkZGbzyyis8/fTTEbVQMwyDAwcOtBvTjYmJITY2tsO5du7cGfZ970TTNPLz8zt9PUB8fHyPaIh3JgzWU3HqcEXGEhISeuT+Jt2Pach7mJKSEqqqqtp4ZIZhcPbs2Tbjt2/fHtIbVxSF7OzsHq3MlCSJhIQEcnJyiImJCfs6Xdc5c+ZMyA1KSZJ44IEHOsxpbmxs7NJGY2caVNxJuAJfkeDxeNi4cWPIn4/b7WbPnj1s2rSJgwcPomkaI0eO7PZ2dqqqMmHChLDGjhkzplvvbdJzmIa8B6mqqmLr1q1BX6uDVe45nU7Ky8tDvtbb7XbGjx/fI2sNxjPPPBPReF3XOX/+fMjzycnJPPfccx16vE6ns9Mt77ojXzoqKqpDsavOUFpa2qa4q7i4mHXr1vHOO+9w7tw5bt64waljx1j9y/+huqCAF198kalTp3b53pIkYbfbmTt3LpMnT+7QK585c2aX72nSe5gx8h5k586dITeTFEUhLS2t1TG3240sy0G9SqvVyooVK3q144wsy8iyHFF898CBA5w4cYL7778/qIyr3W5n+vTpbNu2LeQc169fD5k/3xFXr15tt5oyXHpKSvbWrVvU1dURFxfHJ5980rYDkyQhFAWfEOz/j/9g4aNLmLFsaUDbfMeOHRFvnMqyzLJly0hJSQm8zS1fvpwPP/ywze9nYmIiS5cu7fXG1iZdwzTkPYTL5WpXK8NqtTJu3LhWx0Klz8myzJgxY/qkOu8b3/gGa9asiegat9vN5s2bycnJCRprD5WN08y+ffsiul9LmvOmu2rIs7KyOH78eJfmCMWlS5ew2+3tttETqkrpuHFUf//PsH9tPvKgQQA8+OCD3Lhxo13ZhjtJSUnB5XJx9uxZUlJSSE5OJj4+npdeeoni4mJqamoYMWIEcXFxXf5uJn2DGVrpIdqLY6uqyooVK9p4PYqiMHfu3FZxZFmWsdlsTJkypcfW2h5d+eO+cOECb7zxRpuUvp7eROuOYp5Im0JEQlFRUXjFPkIgKQruz1tvAKenp/Pss8+GtXkM/pDOjh07OHjwIJs2bWLz5s2Bt77MzEwmT55sGvEBjmnIewi73U5SUlIbg64oCrm5uSFjsGPGjGHJkiWMGDGC5ORkcnNzefrpp3ukgjFcXn311U5fK4Rg/fr1rV7h09PTe0x9UVXVLqUfNtOT6pCVlZUdPmwkTSPtbJOMbZDQVkxMDLNmzQr7noZhoOs6mqZRWlrKqVOnIlqzSf/GNOQ9yP3334/D4cBisSDLMqqqkpKSEqjeDEVqaioPP/wwTz75JDNmzOhzwSNJkrq8yfr+++8HQk2SJLFkyZLuWForZFkmIyMj0OKuK0SSsdMZQmbvCIHs9RJVXc3YHZ8jNA3b/fe3GlJfX897773Hjh07OnXvjjalTQYe3RIjlyQpHngTmAgI4CUhxP7umHsgExcXx7PPPktxcTH19fWkpKQwZMiQftPYIRLmzZtHTU1NUGmBcGhoaOD9999HVVWeeuop0tLSSEtLazdOHAlWq5VFixaRmZnZLT/fnt7si46Opr6+vvVGshBEV1QwducXpBQWIiMR99d/heuTT/xZJ488jEhM5MMPP+xy1WV/KlAy6Trdtdn5M2CrEOJpSZKsQN/FAfoZiqKQlZXV18voFpYuXcqqVau6NIemaaxdu5a5c+eSkpLSZUMuyzI5OTnMmjWrWzN6errxRl1dHbNnz+bIkSN4vV4URWHChAlMi4nFHRWNZLcjvF7qfvwv0Pxg+od/xPOnf4IvtutvC90p8WDS93TZkEuSFAd8DfgWgBDCC7QvW2cyYJk/fz67d+/u8jzh9ioNh6lTp/ZqWmZ3IIRg8ODBvPjii+i6jqIofmkATQPNh15RQdVrfwR3eN7Wn/0X1u/+Me6mLJaOiIqKCqpdfuXKFebNm0dFRQUHDx6kvLwch8PB1KlTGTNmzIB8a7yX6Y4YeTZwC/iNJEnHJUl6U5KkNjt5kiS9JknSEUmSjoTbYcWk/9FVDfTuxjCMfiHDGymyLONyuZAkCVVVkSQJ91e7KZkyjYpvPEPVK6+1MeIASBJDmxpHhCI9PZ3777+fl19+OaRBdrvdXLp0iY0bN3Ljxg18Ph91dXXs3buXEydOdMdXNOlFusOQq8A04H+EEFMBJ/DXdw4SQqwSQuQJIfKSk5O74bYmfUFzO7H+RHfLzvYGkiS1Ej7TS0upeullRHU1wumEEFktEgJ7BzIHzW3eFEXB7XaHHHfy5Mk2BUGapnH8+PF2O1iZ9D+6w5BfB64LIQ42ff4Av2E3uUt58skn+3oJrWhoaKC8vLzVsYqKCk6cOMGZM2eor6/v9TWlpqa2ez4nJ6dVHrhz/YeIcHRiZJmpr3+fzMzMkEOuXr3Khx9+SENDA4PaCcG0Z+S7ozuSSe/R5Ri5EKJUkqRrkiTlCCEuAA8A57q+NJP+SnR0dMSl+z2JEIJjx46xePFifD4fmzdvbqVZ01wpmpWVxZw5c4iOjqahoYGKigpiY2N7RO1wwoQJlJaWBj3ncDjIy8trdcwoL4cOOiJhsRD3+utEj81h8dgcrl69yldffYXL5Wo1TAiBz+fjxIkTzJ8/nw0bNrSZKikpCavVGrL3Z1+nvJpEhtQdusuSJE3Bn35oBa4A3xZCVIcan5eXJzorimTSP2hsbGT16tV9vYxWOBwO3G532FriFosFXdexWCxMnDiRCRMmtGqG0ZUMHVVVURSl3TTB6Oho8vLyGDNmDJ7Pd1L1ne/6wyqhiIpi6NnTSC02dsvKyti8eXPQAqOEhASefvpprl+/zq5du2hsbAz0BF24cCGlpaVs2bKllbaPoiiMGTOm3W5GJn2HJElHhRB5dx7vlvRDIcQJoM3kJncv/S1ODrTxTDui2fh5PB6OHj3KsWPHGD58ONOmTeuyVoumaUiSRExMTMjWd06nk6+++ory8nLmLVqIZeJEfKdOIUJ8D0mS0C4XYBl/W6MnJiYm5JtRc9l9RkYGzz//fJvzQ4cOZdGiRezbty8g2DZ27FizmcQAxBTNMuk0qqreVZtiQgiKioooKioiOTmZuLi4DgW+2sPn8zFq1Kh2m10IIbhw4QJTpkwhac17ONesoe6ffoRoMv4CuDVyJNdzJ4OiMNHrYYQQgWwUh9fL0LhB3KytQW9h0FVVDUufJzs7m6ysLLxeb6AC2WTgYRpyk04zfvz4u1azo7tSZMPtWFReXk7syJHEvPACSmoq1d/5E4TLxZnHHuXG5MnoTeGUW8eOkXXrFrOtdpzvvI175xdMiIpCf/ABSsePQ1JVbDZboOAqHCRJwqoouHfsQLt4CXX0KOwPPIDUQXaMSf/B/Jcy6TRZWVl3rSHvbVqKqDkefhjtz77P9d+t5npuLkYLbXRN07hy4QIp775HfPE1ABSvl6kfrEeLi8PxN39F8je/GVFBj15Vxa3Hl2OUlyPcbiS7HTkpkeSNG1C6GGIy6R3M9yiTTpOSkjLgKir7HUJgGAa7du0KyBXotbV4L1zk1uhRiCChDl1VuTViRJvjal0dxi9/FXFVZu3/+Xv0a9f8G626jnA60W/cpOZvf9ipr2TS+5iG3KTTSJLEww8/3NfL6P8Ey6IRAoTA2pSlUldXx+bNm8nfu4/S6TNwr1+PWluLFCS3XNZ11BDZMEZVVcTLc23eAnfudWga7s2bKVv6OHU/+y+MEH1kTfoHpiE36RJpaWlmW7AwkJrTA3Udyedj2OEjzPvlr8g46Q9NxZWUMOOt3xDzjZXQlLWSdi6fUL51QKv8DqzTurEWTwi048ep/9l/Uf7QIxjtdLwy6VtMQ27SJSRJ4qGHHrq7sh2ECNrMoSskFBWReOUKGSdPMfet3zB5y1asLhfF06dhq6sj7701JF692uoP0upyMe39D1A8HhS3G9XtRvF6mfbBh9jvTGlUFKSoKOJ++HcRr83+0IPQ3samx4NeUUHDW78Oe06hadT//BeUTM/jZs44Kl9+Fa2oKOK1mYRHtxQERYpZEHT3UVtby+nTp6mqqiItLY2JEydy8OBBLnYg8NSvaAp3ZB4+Qm1aGrXDMm5LyHbD3LN++zaDb/j13K9NyeXskkdBlpGbvPXJGzaSHsTT1lWVyhEjEJJEUmEhSsswiCQhJyVhmzeP2O9/D8uoUREvTS8v59bSxzGqqkLmsANYJk0iZevm21/J58O19TM8h4/gO3sWvagQye7A9rX5aOfP4z11GpplAGQZKTaWIbt2ooSZTRMKw+XC9dHHePbsQcnMJPr5b6L2YGu+/kSPFgSZmAwaNIh58+a1OrZgwYKIDLkkSQwbNqzbRLAyMjIYOXIkHo+HqqoqdF1HVdWA2mBMTAzDhw8nNjaW6upqqioqqPn9GqILC7keH093RoUlw6Bo5kyOjRhOUmIiJS10TpqzUk498TgJ167juCN3XdE0Ui5fbjupLBP7Fz8g7vXvdWltrr37uJY+lBtfm4/i8zHsxAmSLhe0CevIyUkYNTU0frIJ986deHZ8HvTNpTGY520YiLo6Su9/EMdDDxL73e9iGTWy3XUJITBu3UKKiUFuKkAzamspX7LUn2HT2AhWK8433yLxnbexzZndyZ/AwMc05CY9hiRJEemXCyG4fv16t9w7JiaGBQsWhOyNeieJiYl+zZV/+Ht27NjBtatXQddvb1R20TMXskzuho3UZmbi/NE/UX7+fKvS+OZ7XJ88iWHHjmNr0kBp966G4Tdmka7F6wWLxS+de+gQOw7sp/KB+wO56uWjR5F5+Ajjd3x++yKLBe1KISUTJwffvA3rxgKqq3Gt+wD3pk9JWr8O6+TJ/q/S0EDDm2/RuGEjksOBfdYsnBs2IGpqQAgcSx4l/l//hfqf/wL95s3bujReLwKo+t73SD186J7VUTdDK3c5t27d4vDhw1RWVhIbG8vo0aMD3umYMWNIS0vr8V/+t99+u8utySJFkiSSk5NZvnx5RNc1NDSwdu1a1JoaJn66mdKcHG5Mye16iEUI0o+fYPK27ZT/+J85XlfXtrS+KTav+nzYa2rI+eJLUi5eBElCEqKNUReAmpXFkM+3I3Ww4azduIHzvTU0rl2LUVaGFBVF9KuvcO36dQ5ljSDt3DmGHTuBJAQ16UOxOBsZcvkyagdNoruEw4Eky8gpyRj19YiKytBjbTZs8+aiXS5Av3q1zWnJ4SBlxzbUIGmZdxOhQiumIb+LKS8vZ9OmTe2W0aemprJs2bIeNeaGYbBu3TpqezmFTVEUnnrqKeLj48O+pri4mO2bt7DgZ/+Fva6OC4sWUTB/bvfEyoXgkX/+MeLrX+fzcTltPXIh2twnuqycUXv3UpWZib2mhvQzZ4lu+jkK/B67OmUKKRs+alOJ6blxg6pnnsUoLAruRdtsnHnkIZIuXCTpSmHAaDePbF7JnZ/7DJsNJSMDvaAg6LnU/XtRWmi8342EMuR3UaqByZ0cOHCgQy2U0tJSLly40KPrkGWZlStXsnLlyoiu62one13XWb9+PSdPngxbEfHKlSuk5J/D6nQiAdaG7tUyvzV6FPF2GxMmTEBtNrxNm6zBHhaa3YZmszH+s22MOnAQVdOoa2rMIgG6LKOdOYN762etrruRPoyKmfdhXCkMHQrxeIi9WtzKiDfP23Ild37uM4TA/tCDSHdK7CoKlgnj73oj3h6mIb9LMQwjbL2QSN+OPB4P+fn5HDt2jJs3b4ZtJAcNGsTEiRPDGivLMqmpqV1ugqzrOocOHQo8rNpbq9vtpqCggOiKSuRmb1nu3ibMtsZGop55hvvuu49HH32UYcOG+Y1kMI9fCEbv3kPGiZOomoas69icTqKrqwNesmIY6IaB86OPApfdSB/WdqoQ68k8eQqlyYj7bDZKxo3F20LKt1/h9WJ/+CHsS5aAzYYUHY0UHY2SkU7Cr37V16vrU8zNzruQ/fv3c/r06bDHeztqaNCC8vJyPv30U4QQaJqGqqoMGTKERx99NKxc8pZdcdpj0KBBzJw5k6tXr7YNQUSIEII9e/Zw/Phx6uvrcTgcTJ8+nXHjxrUKKdXcuoWsaTSk3G5FGF3VTtw2/AVgdToZcfAQ7lGjsEwYD/iLqTweD6Wlpa30xNOPn2DsF7s48vWnyDh5qnW6If4sFs1iQfX50CwW3DExKPv2I4TAWXDF/11Skjn83HN4Y2OQdJ3hBw8zdudO5Dti7RJ+I++OjuaL1/8UWddZ8Itfdv079xDO375D3J98F8PZgFZwBfvChcT93d8gt9CjuRcxDfldhBCCjRs3UlZWFtF14fZQFUKwffv2VkZH0zTKysrIz89nwoQJHc4xfPhwDh061KFx9ng8rFu3jsTEROrr67vceswwjEDLN5fLxYEDBxBCBNYshMD7t3+HPvs+ysaMQbNaUb1eBpUE7/IT4c2Ztm49CcXFCJuN6u98l8G/+DlS01tHy03P9OMnmLhlq3/Ds64BESI2L2sahiSh2W3+twePB62ggNonnqR8+HAOv/gH/oGShJBliubOpmr4MOb9+rdt5pIAi8eDo7YWZ1ISjfHx2Bqc/SOccgfuzWYeZM4AACAASURBVJtxb9wY+Oy8dAnnm2/CoEEoqangdmNUVqJkZBD3/ddxLH2sD1fbe5ihlbuIffv2RWzEARYuXBjWuOrq6qDZJ5qmhR1nj4uLY/r06R16742Njfh8voC3unz5cqZPnx7WPcJB0zSOHDkSCLV4jxxBPXCQ5EuXkQyDfS99C11RsDU0oIZoDNEhQuCorOS+d35HYnExEiB7PLh3fI57y1YA7HY706dP98fLhSDni12BeHXq+fNBQy7NYZKS8eM4+8jDRNXVgSzjfPc99Joajjz3jH9Ay2slibqMDBrjg/fwNBSFqOoaMg8fYVBpWb804kBbTRjw7wHU1KCfP49eVISor0fLz6fqe6/T8MYbvb/GPsA05HcJhmFw9uzZiK9LT08PO9zRXUyZMoUVK1bc3uzrAJ/Px4ULF0hJSSE7O7vb1uH1egNvBr4TJxGaxtQPP2LombM0JiSy8/uvc+lr89A6uemaWFjE/DfeIrFJbrYZ0diI8/33A5+nTJnC4sWLGZ6cjKNFo+j0M2coGzUSrUXYQAC6xcKe116h8L5ZTP7kU/8Jnw/nG29ydUYewmLBXl/PsKPHSD9xErVFtWZJztiga5V1jYaEwYzd8XmbUM6AxeOh7t/+A9HLqa99gRlauUtoryN6KGRZDrv5AMDgwYOx2+1tWpepqkpOTk5E905ISCA+Pp6KiooOxwohyM/Pp6CgoMvx8pbYbLbAZqoydCiS1YLS4CR34ydM/HQzusWCK3UIlzo5f+OgOIwQrq0ktfahhg4dinvKFFxxcYHKTglIP5ePOyYGQ1GQhKA2NZWC2bPIOH6C7EOHb0+gaeiyTMm4sWTtP0DOF7v8YRkJJn26mWNPraA8ZwxRQWL+hiRROnYsSjf+bPsNuo527VqnpAsGEqZHfhdQVlbGgQMHIr5OlmXGjRvX8cAmmgWyLBZLwJtWVZXU1NSI5mlm0qRJYXvl0NqDjgSLxdIm+0VVVfLy8gKbnfaHHkSKioKmkI+i61jdbuLaK1LpAFdCAmeWLWtzXNjtRD3zjVbHDJeLGz/7L8rGjEa742dicbtxxcViKAoJ164xc837rY14E9dyJzPnndXk7PwCRdNQfT5Urw9F05i2/kMsTidpl26X+gfyw4Xw39dqvZ2tc5cgdB0lzD2ggYzpkQ9wTp8+zeHDhyPunelwOHj44YcjztVOTk7mm9/8JleuXMHlcpGamkpqamqnCopGjRpFVVUVZ86cQVEUNE0L2Uj4TiRJClzTHqqq8vWvf52KigoOHjxIXV0d0dHRTJ8+vdVbhGS1kvzRh1R950/wnTvnb7AgBDQ0+PVEOpMGKUmUjs2hLjmZqJoaZF3HUBRqpuSS/sgjgWGu7Tuoeu0PGeHzIRQZWdNbpQvWJyXiqKnF1sFbV8Ypf6ZSMGMsZJl5q94MmrECMPXDjykfPYr65GTiysuRu1n9sa9wLF2KPCj4vsDdhFnZOYBxu92sXr06bOPXjKIofPvb3+430rMul4vKykq8Xi+7du0K66FksVhYtmwZ586d4+LFiwgh2uSIx8bGsnTp0oj3AKpe/z7ODRsDqoSb/+5vEF3oX6l4PAw9cxaL201Fdhb23NyAdEDjp5upfu0P270+nMrK0pEjGVLQVuiq5Rz9dgOzh7DOn0/S27/pUL5gIGGqH96F5OfnR2zEo6Ojefzxx/uNEQf/20FGRgZCCA4dOhRW53rDMLhy5QqGYZCQkICqqsTGxlJXVxcIGY0cOTLiNwWjtpbGjZ8EjDgQCLd0Ft1m49r02w0f4prEqYzaWqr/+DsdXt/RN2hITMQTF9eVJd51JH2yEdu0qX29jF7DNOQDFCEEJ06cCHneYrHg8/lQFIWxY8cycuRIHA4Hg/rxa6YkSSxZsoTNmzfT2NiIYRghH1S6rrf5/mVlZQwbNoxHHnkkLAPuPXMWz86dSFFR/nxjRcF3uQChKK2Mp6WhAW83GsqysjKEplH3k5/6FRa7yLUpuWCEnude88aj//qv7ikjDqYhH7DU19e3Ksy5kxkzZjBhwoQBJ+sZFxfHypUruXHjBlu3bo3oWiEEN27coKioiKysrMBx/WYJ9W+9he/ESSwTJxD98ks4f7XKnwLo9YKiUPsP/+j3vGW5jXGVfVpILZTO4PP5KJ33NYxO5PwHQ7PZuDlpImO/+LLNuXvNiAM4f/Kf2MaOxfHQg329lF7jnjDkBQUFHDp0CJfLRVJSErNmzWLIHQI7hmHw0UcfUVnpz1KQJIkpU6YwY8aMvlhyh3SkQZKQkDDgjHgzkiThdDqRZTni0FGztkp+fj6xsbGMiYpCf+55fy6xz4f36FGcq9/1G+sWfTQD/6/rCFlGU1XUpli9bu3e8u/BdXUYN2922hu/M2aeeuECNyZP4uSypeR+sqnN2IH5W9AFvF7q//u/TUN+N7Fv3z7OnDkT+FxaWsqGDRsYNWoUaWlpZGdno+s6q1evbnWdEILjx4+jKArTurOhbTcRHR1NbGxsoOy8JYqikJqa2uNrMAyDa9euBbTOs7KyIkonbI/OpBkqHi9jd+xAt1q5uGghhqpyQdeZPDzzdgs1n++2AQ81j2GgqSrumBgsLhe2hgZ8XVRibMn4Dz7slBFvNuA+m40bkyaSdeQoAEkFV0gsLKJ0wngUzcf4rdsC+uXhRPd1WUY2jLvK4BvdIa0wgLirDXlZWVkrI96Sy5cvc/ny5Q671xw7dqxfGnKAJUuWsH79+lZZHpIksXjx4h7fzPR6vWzcuDEQ4lFVlQMHDvDEE08Q1w3x5MzMTPbv3x/RNUKWGHrmrL812sWLyLqO4tMoHzUKXZZRIvDuVa+XmG98Hed3v4OxfXvw0vDOIATxN2926lIJ8Fmt7PxfP2Dszl0Y3Pa289a+T/noUSQXXEGOMBNNNoz+ozneTVhn9s836Z7irjXkpaWlbGwhrtNZIn21700GDRrEt771LfLz8ykpKSEhIYGJEydibcqKiBQhBBcvXuTixYtIkkRaWhoXL16koaEBi8VCbm4uU6ZMQZIkjhw5Qk1NTeDno2kauq6za9cuHn/88S5/t5iYGPLy8jhy5EjwBgxNSIaBrOsISWLKhx9jbcq1jqmoDBiljFOnQopPtYcsy6R5fUxa/S4Hn4lMS709vFYLFpfeKaMp6zp2t5usf/kxR1atIu3wYYae9+vcDLkUpK9nmPQbzfHuQJKI+8EP+noVvUq3GXJJkhTgCHBDCLG0u+btDIZhdIsRHwjIssyECRPCUh5sD03T+OCDD1ql/t1s4Tl6vV4OHz5MXV0dCxYs4PLly20eckIIysvL8fl8WLpBVjQ3N5dhw4axd+9eSkpKbp9oMsqK203O51+gaj6GXLiItYWmSEuj1NLjDBtZxjZ/PrX/7/8Re+06X/vlr4ipqKAxPp59334Rb2xspzc/6xMSSbhxo1PXClkmb/NW7C++SMWY0eBxk3LlCqq3ay3Z7hojDkS/9ipqdlbHA+8iuvP9+3Ugvxvn6zQHDx7strnGjBnTbXP1Z44fPx5W/vaFCxciriLtCgkJCaF1ZKxW0q8WMezEyVZGPBgRGSpJQk5KwvHQg3h278HqchFbUUH1sAz2vvZKx0Y8WGhDCJIuXMRRXc3NyRM7ZTg1VaV42lRiysoCujdlOTk0Dh6M3oUGHHeTEQdofOcdGu8RR66ZbjHkkiRlAI8Bb3bHfF3B7XZH1FShI8KVeB3oRNLuraysjFGjRrWJw0uSREpKSrd44y0JmaFjsaC89SZER7fyuIN634oC4WzEWixYpk0lecNH/g1JjwcJqEtJ4dDz38TncETsiUu6Tvz16+S9v46v/WpVxB14BGAA16ZOoSpzGI6vzce9bRsPxg/GYrVy6NVXKJw9m8ZBg/wCW7KMZrGgq6r/+0zJxfbQg8T84M8juu9ARbjcVP/lXyN6snF0P6O7Qis/Bf4S6F091BYUFBSwf/9+Ghsb+2oJ9wzR0dHk5eVx8+bNwGZns5BWTzz4xo0bx/79+9u8CQgh2HHyJLHffpFxGz4h/vp1hKKgqyqqx4PU0jO2WjvOFBk0CNvs+7DNmYOkWnCuvS01e3neXL9hbI87PHFJ10EIUi5dJnfjJ/7NVsNg4mfbw/reLTn21JM0pKUxZ+06XLW1uHfsAAEP2200/uhHlP7pJL66cAFd11FdLlIvXETx+agZP46vv/564KHrPXgI7969rdZ6V6Yout34LlzEOrFrIceBQpcNuSRJS4FyIcRRSZIWtjPuNeA18GckdCdffvlljzQQ7mq/yIHEuHHjOHr0aIfjHA5HoCv9ihUruH79OhUVFcTFxTFixIhuSz9sSU5ODjdu3ODq1auAf1+gWWDLMAyqExPZ99K3UIFly5eTIElU/+nreA4cQACNcXGceOJxhl64yIi9e0MbrdpaPFs/w7NtO3X/9M8QFRU4VT8kJXSpfrNRvMNTn7p2HclXr6Le0UrP2glnY+yuL0levBhXfT14vYjmORsaiP6bv0X/2X8GNoU1h4PrU3IBf4XvzZs3ycjIACDx//sfKl96Ge/BQ/6lA0aEGT0DAp8PKbb7Ukb7O93xVzcXeFySpCWAHYiTJGm1EOL5loOEEKuAVeAXzeqG+wJQU1PTY13g586d2yPz9kdyc3MpLCykqqoq5Bir1coTTzwR+CzLMpmZmd3+YL4TWZZ58MEHqaiooKysDI/Hw4kTJ9p46LokcS4/nwULFpC05j3OHTrE4T178UT7DfKIA2HunRgGeDz+/zUxqKQUZ2Ii4s6HewgjDlA2dgxpl9qqmUfq/UpATHUNrnUf+CtR70C43bjbyZtuWQEsx8cT99d/RcUzzwXCRh0ZcQHUpSQTU1k1oDTLlaSkvl5Cr9FlQy6E+BvgbwCaPPK/uNOI9yRfftm2LDkUzfojd6KqahujMGHCBMaODd5N5W5EVVWeeuopiouLuXTpEoqikJubiyzLlJaWEhsby9ChQ/u0WjQpKYmkpCSKioqCrkMIQWOL3p4HzpxBi4kOfLY31LcyopGEFEbu2UvpuLFtNxWF8PfPDJLyWZmVFWiS3GUMI6gRbybTauFGkN9jwzBIS0u7/dnpxHv0WMQFSbHltwZU+EVOSfbry98jDPg88nAyLcDfDWbRokXs2LEDvUlrulnTetmyZcTFxXHt2jVUVWX48OEDtry9K0iSxPDhwxk+fHir482hlP7CkCFDWqU+yprG2B2fM+z4cRSfxq3p04n+v/+IpmlM/ngD6adOIwkR2DSU8YdbykePIvP4ibC0t2MrKrjv7d9x5tFHqE1Pv+2B63rwzU8hcMfGUjp2LEPPnkXqwcpJ4fORPX481avfpdrlomRkNsJqRVEUZs6cib1pc9W57gNq/vKv/BdFkHk00HLMJYeDQf/4j/fU3/CA1yPftGlTq3znULz00kuoqkp1dTUnTpygqqqKxMREpkyZ0u8MlUnHHD16lJMnT6JpGtN/v5bkK1da95qMiqIyMYGEa9fbeOEAmtXK4WdXMvO9NRF5zAL46o9eo2HIEGSvl4xTp/FER3Fr1CiM5mwdIVA9HuKLi8k8doLkS5dQesiQSw4H1jlz8OzZg6QoGPi98LK//F+MWPEkyU3dcTyHD1Px5FPBUyMHOMqoUShDUtDyz6MMzyTuL36A/S7NNrsr9cg1TQvLiOfk5AQ24QYPHsyiRYt6emkmPcz06dNJTk7m0q5dpBQWIt/pYXq9ASOuKQqKrrfyLFWvl8kbN3Hk2WfI/XgDtoYGpCbPvD2DaygK1kYX1vp6sg4cZOT+AxiyzMmnVlAy1t9xKKqiklmr3/V3t2+i282nqmKdfR9RTzxBzf/+P+DxBEJFCpD+Hz8h6YU/CAyvev3PIjbiDQkJlI8ZjaJppOafx9YibNWfkGNiSPjZT1FahJDuNQa0Id+2bVtY4+6GTUvDMCgsLOTy5cuoqsrYsWNJT0/v1FyVlZUUFBQghCA7OzvgtfUkQggaGxtRFCXwqt9VMjMzSckYRpXd3jZnWNOQgJtjc0i7cLGNcZaA6OpqGpIS2fn97+GoqWFwURGTP/kUpR2Dp6gqM2prYc1avycvSajR0Sz+zh+jDB1K1R/+EZ79+8Hwz9GhhklqKtTUQKTNsxWFwf/2r9T//BetNmWbEU4n1X/xv0j4xc8RPh9GU8ZPUKxWf3ioxTznFy2kOnMYY3Z9SWxZOa74QTQMjifxeucqUnsS36lTlC9ZypDdXyJ3o7jZQGLAGnKfz8f169c7HDdt2rQeSYnrTQzDYM2aNa261xcWFjJ58mRmzpwZ0VzHjh3j+PHjGIaBEIKzZ88yceLEiOeJhMLCQnbu3NlGMyUuLo6FCxd2SalRHTUyeOGHqqIDZx9bQtr50FlNsqaBJOEaPBhPTAzIChO3bkXSDRSfr7UBttmwL1pE4i9/gXbjJt5DB5ETErDNnYvU9DuWtPp3aNeuoV+/TuPnO2n8n1+FXrzNRsr7a2lYtYrGd98L32O2WHA88jDqsGEIl9u/ERoE16eb8Ty3Dzmlgwe1EKhZWWg3b0J9PTVDh1KTkc6MFmEna6kbTVXRJQm5SVmx32AYGDU11P3nTxn0w7+7p2LjzfSffl8R0l53nJbk5bUJJ/UYXq+XxsbGNr0ju8qdRhz8xv3UqVNBZWxDUVtby/HjxwObveAPT50+fbrdtMOuUF1dzfbt24PK0tbV1bFx48bWOioRomZm+uOhd3j5kt1OVc5oDEVBD1FpasgyrqZ+norXi72ujpuTJ7H9L37A/pe+xenlT6APGQIWC9hsRC1fTsLP/8t/3/ShRD35JPYFCwJGPLCmYcNQhg1r34gDeL3U/eQ/aVyzNrKwh8+HVl2Doev+zkah3nB8PipWPkPFymc7nE+/do2oP/Anm92YNJExu75ss3egahooCq7oaHRZ7v5wUVfwenG+8SblC+9H78Lv00BlQBrya9eucfLkyQ7HPfbYY72wGr8swJYtW3jnnXf4/e9/z5o1a8KK3YfD+fPn2xjxZgzD4EYE4ktXr14N+pAxDIOioqLOLrFdwpGi3bVrV5fukfDLnxP9rReRYmNBUbDeN4vkjz8k4d//HWSZcw8/1MboCMDncDD82HFSzl8gZ8fnxF+7BpKEUBTqUlO5OW0qdW//hrTjR0n++EPs9y9CKy4Oa00Nv/yfjgcJgfvjjzslkevbvZvyRx7F/uAD7VcvChFWJyLR2Ihx9SqD31iFsNmILSsPOk7WNaKczh7bvO0Suo5WWEjlq6/19Up6nQEXcygpKWHbtm0dyss+8MADnY4hR4IQgs2bN1NVVRVYU319PVu3buWpp57qco/MjvTSI5GslWU56GunJEk9pl9eU1PT4ZhI3iqCIdlsxP/vHxL/v3/Y6niGYaAcP861vOn4HHbGb9uOrcGJJzoKef584mpqmbRvP6SmcnTaFEqCCKRlpqZS+Yd/hO/YcVAUhKZhmzmDxF+/heRwhFyTEebGoAH+/HTVQsbp0xEZRz0/H+3yZRJW/47SSbkdNsxoF0lCHjyYqCWPMnnqFOr27sVSGeQtrbfdcEm6rZHTtCfhX0eIheg6vvx8tBs3UHvh77+/MOAM+dGjRzvsHjNt2jRGjhzZK+uprKxspcvdjK7rnD17ljlz5nR67oaGhg7DNJFUVWZlZQVVhpQkiezs7IjXFw5JSUkh3yiaae8hUl9fz5kzZ6iqqiI5OZmJEycSFWahhyzLLF68mM2bN1M5ZQpfTp4M+OUIZs+e3eqhVnrsGOVNeweSJCFJEjNnzkT/95/gPXK01Uag5+Ahan/0Y+L/7z+GvHfUym/g+mB9x2vEv+la3VRCHymer3YT88rLxP/4R9T+8IcIjzdkzLxdhECy2zCcTmJ37+ZqVhb22rpAuzsAoyk+3qsIgWXaNGK+/SKS3YFRUQE2G65PP8Wz9bOgl0iKiuiiczDQGHCGvCMPb/Dgwb3a0aehoSFklWFtbW2X5u5ILjYjIyOijdzo6Gjmz5/P7t27A2sWQjB79uxu6eoTjNmzZ3cYtjEMg6tXrwYKkerq6gIPsc8++yygqVJSUsK5c+dYvnx52Ln/Q4YM4fnnn+fq1at4vV7S09ODviVNmzaN7OxsCgsLAw+22NhYbn7wQdusEI8H52/fxvPVbqzTphH7vT9BHTGCxg0bqf/vn2PcKseSl4c8ZEiHYQ1DkqhJT2fw9RudClUoTU2mo59ZiXXyZJzvvovz92uCZ7LQfmql8/drcb79O/D5GNa0NkOW/UZeiN434k34Dh6krqqKITt3IDU99KNXPEntP/0zDW++1bbi1WpFHT26D1badwwYQ97cH7I9idRZs2YxYcKEHm9z1pLExMSgYR5FUVqVRneGQYMGYbfbg+pxS5LEkiVLIp5zzJgxDBs2LBAvHz58eNgebmeIjY1l2bJlbN26Nag8QjO7du3iG9/4Bjt27KC8vBxFUfDe8QdqGAZer5f9+/fz6KOPhr0Gi8XCqFGjOhwXHx/P1KlTA5+FEKHL4nUd7dIltMuXcW3ahGPlN2h8+51A6XvAW1RVRFMqJLQ2pgLQLRYK5s5h7I7PiSstjcyYWyzYFy0EQLtxg/o33sDz5VdI8fGI6ipo0WxC4M+BN5okBspyxhBXVk5sefnte94RDpKF6DcFRHpREZ49e7F/bX7gWOx3v4Prk03oFRXgcoGiIFksDP6Pf0O6hwTvYIAY8sbGRjZs2IDb7Q7qpaqqSm5uLrm5ub2+ttjYWEaOHMmVK1cCa5MkCavVyrhx47o0tyRJPPDAA2zZsqXVw0JVVZ59toNMhHZwOBy9qiOTlpbGt7/9bTRN48MPPwz6VmUYBtu3b6e8vBzDMNoNn3XXRnJHSJKEZUYevialwKAIgXA6afz1b4Kf13Uahgwhuty/edgsrasrCpXDh5P/yEO4Bg/mytw5pJ3ruC+LAKqHDaNszGhUVSWmro4Yr5dbjzyKUVd3W0NFVcHhQLhc/riyECi6TnXmMI4/tQJdVRGyjK3BSd6atcSVB9/c7DdoGtrly9DCkMvx8aTs2IZzzVo8u75EyUgn5tvfwpKT04cL7RsGhCHfvXt3yHhxVFQUU6ZM6XKrs67wta99jcTERM6ePYvP5yMzM5O8vDxsNluX505PT2flypXk5+fT0NBARkYG2dnZAzI3XlXVkMVAhmEEjHg489y8eRObzUZCQkKb0FZ9fT379u3j+vXryLLMqFGjuO+++zrV8MK+cGH7hrwjhECdOYOvcsYw/yc/DYQn6lKSqRiZzaCbJbgGDaJ26FAqh2eSeLU4pFcugBPLn6Bs3Fh0iwVJ1ylYt468mlqSnM7WQliaRtH0aZRnZzF1/UdYvF7cMTEceWYleosNclf8IA68+Ac88JOf9m9lQ0kKGi6RY2KIfeVlYl95uQ8W1X/o99ZACEFxcXFQI26xWHj++V4TWgyJLMtMmjSJSZMm9cj8sbGxPVqw05tMmDCBioqKthK0ERgRj8fDpk2bkCSJ6Oholi1bRmxTPrjX6+Wjjz7C4/EghEDXdS5cuEBlZSVPPPFExMUi7i1bIxofjLiSEqbm5WFYLCgeD4YsE3ergridOxGKwoQtWzn0/Dc5/vWnmbvqTWxOZ1CjemvUSMrGjg0YYqGq6EJwOCaaByWJOx9TnpgYrK7bYbnruZPbNqGWJAxFoTxnTFhvBH2FnJqKbW7nEwfudvp9Hnl7WRt9Ifhl0jWys7PJyclBUZSI3iqsVmsbIyyEoKGhgY8//jjwu3Dx4kU0TWv1u2EYBlVVVZRHGD7QKyvRrl2L6Jo2SBLWSZMYNSMPa1PcVhICRdNQdAPV68Pi9ZK39n28Dge3fvVLEn76E6K/88dIGa3T525OnIBua5tuKgMVI9tmHSUVF1OVOQzZaIrbx8TcFvZqgZBlPNHRbY5DN2cbdnLvShmeScq2rYGNTpO29PufjCzLpKent/kjliSJESNG9M2iTDqNJEnMnTuXlStXkpWV1e7GtCzL2Gw2li5dyqJFi4gOYWxcLhfl5eUUFhZy+PDhkNk+1dXVHa6voaGBU6dOcWLLFkrnL0CEKZOsK0pwo+dwEPPaq1gmT0Y0GVEpiAMi+3wklZczdtIkYlasIP7v/paKH/8Id2wsWnMoRBA0tVCyWJDv3NyzWklNTsFITeVqXh6axUJSYRFKkGwWBCQWXUUANSkpNA6Kw1BkdEXB206ufJBp2kVKSvL3To0AZeRIUvftRRk8OKLr7jX6fWgFYP78+Xz88cf4fD40TcNisWCz2bjvvvv6emndTn19PadOnaK8vJyEhARyc3PvSpndmJgYdF0PGhOXJAm73Y7FYsFut1NdXU1OTg4ulyvkfOfPn6egoKDdlM2Ofo6XLl3iq6++QgjBlLXvI+rqghrdOykdPQrV6yWmorKVQqB18iQG//u/o2Zm4juX364wliQMdI+Xbdu2MXr0aNLT0zly6hRJDz/EoNISFJ+GNypE42dFYfT3X6fhr/8W0VCP0A3sixYx+Kc/4ZtRUXyVkcGZDz9i0iebiCsroy41NRCeUbxeUvPPE1VdTX1KCvtefQWhyFjcbnSLhcTCQqa9/0GrfPL2yF+0gFF792MJku1jvW8W9vvvx3fsGJ59+9ELCtrPirHZiLtHGkZ3lQFhyGNjY3n22WcpKCigpqaGxMREsrKy7rqemlVVVWzYsCEQGqioqKCgoIAlS5a0EpYSQqBpGqqqdkogyO12B/YdMjMzcUTgdXUngwcPRpblNsZckiS8Xi8ul4u6ujqqqqq4cOECDocjZHFRcXFxSCMuAYPi4hgyZEibc0Z1NQ1vvkXj9u00eL3E3zeLyuxskgquhGXEdVUlqehqQJdEs1i4mjedS/cv4tHly7EMHQqA+/PP2y3FV3waE7ZsYX/qEE5WVGJs+pSHzpzBUBSEJGFYLP4+nEHeTBctWkTsiBHEPPwwekkJcmwsclNdgAzc/8ADqBD2CQAAIABJREFU1J09R/3Wz5j1zmqKp03lRu5kZE0n8+gx0k6fpjEpiRPLn0AoMkgSvqbfiVujR3Pm0cVM2rK1td57EIQkEVNV3TYO34Rn4yd4d+70u+4JCR2mNsZ+54+JeuLxdseY+BnwjSXuJj799NOg2ikJCQk8/fTTAJw9e5aDBw8GjFZqaipLliwJO97c7HW2LAiaN28eOX2QstXY2MjatWtb5ZdLkhR070NVVUaPHk1+ftsNubi4uNCdooQg/dRpJuz4nPgVTzLoH/4eqSmbyKiupuzhRzAqKgP54prFwoUHFjFy7z7s9e1XpKKqGIbRpsOQrqp8+Z0/IjM5mXnLl6MkJFD/y/+h7l//rd0yel1RuDl+PKkXL6I29dNsiScqis///Put+obKskxOTg7z58+nPSq/9RLu7dtDnm/Oaf/qj17DlZDQ+qRhMOaLXYzes7fdewB47XYsbneXdViUrCxS93zVxVnuPkI1luj3MfJ7idLS4A10q6qq0HWdixcvsnfv3laeZ2lpKb/+9a/ZsGFDh5olTqeTr776Cl3X0TQNTdPQdZ09e/Z0WEbfE0RFRbF06VISEhKQZRlZlklMTAz6UNI0DZfLRV5eXqCEHvwPueXLlxMV4q3CVlePblHZ/v3v8f6I4Xz8i18GCqwafv0bjMqqVkU/qs/H2B07uTplClo7D0cpMRHnzBlB28QJSSLp0mWi1n1A6bQ8Kr//59T950871EJRdJ30M2eCGnHwS+4mFhYxbstnPPzjf2XOG28SVVpKfn4+69at4+zZswF54jtRx431646H+j6A4vMxaveetuvSNJIKi9pde2AeXeu6mJbVStKHH3R1lnuKARFauVewWq1B48CKoiDLclCdlGbKyspYs2YNTz/9NINDbAwVFhYGPS6E4MqVK0xu0iLpTZKTk3n66afxer3Iskx5eTmffdZWQ0OSJBwOB9OmTWPSpElUVVURFRVFbGwsQghyjp/gVHZWqxxp2etFt6iU5uQENtluRUfx7urVzLrvPpJ3fB60lN1QFKqysoitqiLt7Dn//e8YI+rrqXS5cMhyG2MuGQajd+9B8fnA58O9bl3YPw+pA61vW309WYcOIWSZU088TmPTv3V1dTX79u3jwIED6LpOdHQ0eXl5gTetmBf+AOevf4Nop4GzBCRdKUTxelvF0JMvXSY+yJvinSX/msWCZrdj8XVN52TwL36OmpLSpTnuNUyPvI8oLi5m3bp1vPXWW6xZs4bLly8zYcKENt6ooijk5OQgSVK7m33gN8hbt4bOe26pQ37ndZHkcfcEVqsVVVVJTU0NWrijKAqjR4/m9OnTbNu2jXPnzgXeTKq376C6thbV5ULSNBACa0MDQ8+c8YchWu6lSBK6YXB0+3acBQVBMy1kXccb5eDGxIn+S4KMMXQdT0wMIkjWjaEoOBoaUINliLRDR0FOWdfJ3n8ACSgZPx7XoEGt0glb/js6nU727t3L+fPnAVDS0kha/wGWKe1XPzvq6pi8cRNJBVdIKrjCpE82Me2D9a1kBZrXacgyuqris9nQVZUrc2ZTMWJE11IW7XYcDz/UlRn867zHUpNNj7wPuHLlCp9//nngl62uro6vvvqKOXPmkJ2dTUFBAYqioOs6w4YNC2Tn2Gw2PB0Yh/r6eqqrq4N65cOHDyfY3oQsywHBqr5GlmUee+wxtmzZgtvtDsTMZ8yYwfbt21s9zC5dukRUVBSuhgbEjDx/nrIQyL7/n73zDo/iPNf+b2a2aNV7QxJFFAkwvdpgwICNsTFuGNuxE3enOYmd5vScc3JOzpfmOImdEye4YWzcwMYYU00xHYREkQQSCAkEqPeyu1Pe749dLVptUUHGJdzXxWVrZuedd6TZZ5553vu5b5XsDRtpSUxE99ddKwQD9+3H0trqm2njMmYetG8/Gbl5gbssJYnW2FgOL17EmDVrXQFdCAyTiWMLb2TiO+/2OkuSCBzMBSDJMpHV1QDUDhro/9o6QdM0Dh48SFZWFpqmcbi9jaannmTYAw9RO3gwZZMnoYaEkJKfT0ZuHopbEyY1P5/U/HzALZzl1miRDIOW+Hii3SVAxTBw2GyUXzWa82OuQlFVhpWV9b20EhJC9G/+y8eoozew7/iExl/+Cq24GCkqivDHHyXiiSe+9Bz0K4H8MqO9vd0riHdA0zQOHDjAfffdx5QpU2hoaCAyMpLwTh6EV199NVu3bu32HKtXr+aBBx7w4WhHR0czZswYjh496slmTSYTI0eOJLbrAtdniJiYGO655x5qampwOp0kJSWxZ88ev28kbW1tLiZHB1NCkjAsFvJvXMCw7duRVdVvE0xERaXLmNlioXDeXM6PuQohSSQWFTNyw0YG5gZ3oJJ1ncrsLDSrlarhw4k5cxbDbKI+LY2h2/u+SBcoCEqShKoo6DYbIU1NpOTno9pCqMzK8vtW0IG2tjbOnTvHhx9+6NnWvuhmKkdme8onjSnJlI8fz9XLXvTpKFWtVsrHjcXcbieurMwTxJFd7BZrezuZOYfI3Bu47NdTyPFxmLo0NnV8T3rCznLmHKLuoYdd+jKAaGyk5a/PIZqaieqiVf9lwxXWymXGzp07KSgoCLj/oYceCspAWb58ebclFnDpgN9+++1+91VXV3Py5EkAMjMzSfwC1CNfffVVvyqQASEEkeXnaEob4E3ZEwJTezvz//AnJCHY9cjDNCUlIjp+57qOtb2d8es3EFFyGkug37WikPPD71PROSt2l3Tm/vkvfhdB+wqnzUbu7bdRN2ig6xxtbYx5fw0x587jCAtjz0MPuPxG/SA0NBS73e5N8TQMny5Lxelk1LqPSDt8xPdh0mHu4F6slWw2QpfeRcRPf0L1wpvR3fdSf0Cy2UhYvw6EoOHpn+Lcvx/MZkLvuIOoX/8SOUBTGEDNV+7DsW27746QEFKOHkb+FFU+LxeusFY+JwimzW2xWLrlxi9evLhH56mpqQkoG5uQkMD06dOZPn36FyKIO53OoBK4fiFJNA1IdXGVu/xLyzuMkGXqMjJoiY+7GMSBkJYWko4fp8lm48DSuzi68EYvXrQAHCEhtIWFMWXVe1x77bVERkaiKAqph48w+2/P92sQt4eFsfer91M7aCCGyYRhNmOPiiLn7qXYw8OxNTRw1ZoPXBz1LudVgKysLN+mKz/ZrW6xUOleGPVK7UJCCP3J04Tcfx9KRgamUSOJ+u/fEPWb/6Jm8W19CuLmiRORk5P9tuwLp5PmPz9L9S2Lce7b57omh4O2d96h9v6vBR1XKyr2u11SFPQL/hlhXxZcKa1cZgRT4Bs7dmxAk4rm5mZkWebYsWM9Pte5c+cuWcZAKy+n5cWX0AoKMI8fT/gDX0Px01jzaaG8vJwNGzb0bTG2c6AQwlOCSTtyBCHLNCcmeJUlhm7fwdBPdiJkGckwGHgol5w7b2fH448xasMG4k+X0piawoXsbEquuZrQujoWV1YxYulSJElCv+026qprcB48GFjHvIdoTEggb8kdtEZHgyx7ccfBtaBaOmUyo9dvILH4JHOf/St1GemcmHsdbdHRxJ8uZVxxMeZ33mVEVBSnZlyD1kHR7HSPKQ4HSSeKMNvtmNxvHw6TCcliwR4dRdXNN1NsMUNGOjFPfY+ZtbU0/e73NHz/B33WKrfevBDFGkLjb/4b2tq8d+o6jt17EM0t3uM7nahHjqDmF2AeNdLvuKbsLHQ/EsdC11FSL80b4POOK6WVy4xjx46xf/9+ny7EqKgo7rrrLp9Afv78ebZu3ep5Pe7N3+v666+/pEDuPHKEmjvvclHWVBUsFqSQEBI+WIN5qLeVnhCCU6dOkZeXR0tLC2FhYYwaNcojkNUXqKrK8uXLu3VK6i1kVUXIMrGlZTSkDUC3Wok5c5Ypr63wcY5XrVY2f/9JJCGIuFBBU0oyhsnkCYbW5mZuPnmK2Oef8yyo6RUV2Ld8TMOPfuz3/B1/QX9VX4GroWjzD55Ct1pJPXKUyuHD0P3I/yYUn2TK6290e726onB66hROzJvrFcRjS0uZ/Mabrh8Mw7PY2RYRztannvQZx9LczPSXXyW8zo+XZ28gy0ghIYiuQRzAbHbJ8fqTbggLI+p3/w+cTlr+tQzR3EzIDTcQ8Z1vo8TG4jx8mJo7lnhq5OAq1YQ9/BBRP3n60ub8OcGV0srnBCNHjmTQoEEoioLZbMZkMhETE8OiRYv86mqvX7+e1tbWgNTBYNi4caPfTsieouHHTyNaWy82sjidiOZmGn/t8qqsq6tj3bp1vPjii7z88st8/PHH1NXV4XQ6qa+vZ+fOnSxbtox33nmH8+fP90hrvDPKy8v7JEHQHQyzGaEo1GWku2zMNI20vDxkf+UbIYg/XYpusdCQkY5hsXgW+pAknGFhVOTmesndKsnJhH3lXmx3L/W8FdjDwjh8803s/tr9FM+YgcNPvVbg6ozc8Y3H0S0WFIeDYdu2+2Tj4HoYxXZjoeeZj66TcPIkUqe3GlnTmLTyLUxOp+ufO4gLwB7A9k8NDeXgPUsvXRHRMPwHcUlyUUUD3CfCbsexcSMNP/0ZWn4++pkztL78MlVuUw3L2LHEvfYq5tGjQFGQ4+OJ+MH3iXza/wP1y4QrpZXLDFmWue6662hqaqK6uprw8HASExP9BqzCwsJeB7+u+OSTT4iNjfWrMxIMQlVRj+X72SFw7NlDc3Mz77//fo9q13V1daxduxZwlZYyMjKYMmWKR0M8ED5tbrswmzEkiYSTp1CczoBZjRzkjUAyDOpTU6n79hMkZmVh7sS6iP6PX6Pl51Pb1MSepXcxcv0GRq9fj6GYkFUV1WzCrHYyN5Ylcu6529MiH1pfj7WtlQFHjnL+qtEelomkaZjtdgYeyu3xtUZVVZOSX8D5sa6mr7jTpwOWRgJpzAhFwR4R4fIY9dMgdKmQ4uNdjkuBFrUtFtrXb/Bu4lJVRF0drW+sJOLxx7BOm0bihkvXkP+i4UpG/hkhMjKSzMxMkpKSAmadTU1NlxzIwaXh0usSmqK4XnP9QLbZOHz4cJ9KHqqqcurUKd544w1eeukl1q5dS0lJid+gnVhXj94bpkofIOs6w7dtJzm/wH9zkGFQM2Rw4AGEQNZ1cDqpe+wx72PDw0lY9yHHHnqQwfv2k1JYiKLpmB0OFMNANrzPKGQFSbv4YLRHRiLrOlet/ZDsjZsIq67G2tREem4eM/7xT8y9/N2MWveRp2FK1g0kf+ULIPr8BeLLy/2OIQmBI8I/Q+ZSIaqrETU1Afebhg9D8iMzIOx2HH6kBf6dcCWQf46RmpraL5ZumqYFpTz6gyTLhN5+O3RtOgkJIez++6iqqrrk7jlVVTl//jybN29m2bJlHh0YAMfuPbTcey8jP/zIVfIIUDe9VAhJIqyuzrPo2dG5aEgSuslE/o0L0EJCAmavhtlM/g3Xs/Xb36TUZKZtvbe8gCTLNAKD9h/ApHo/+BRd9zw8BC5Nk2nLVzD7z39BcThQQ0O5MHIkhsnEwJxDzH7+/5j3zLOMXvcRIV2MknsCi9PJdc/+ldjSMuoy0gNz1oHEgkKvUkwHdLOZqPI+ZuOSBOa+38/hjz3mbWfXAUXBlJHe53G/DLjkQC5JUrokSVslSSqUJClfkqTv9sfErgCGDRuGzWbzauyRZRlFUVAUhZSUFO64444ejVVUVNTr80f956+xTp0CISFIERFgtRJy3RwinvyeX6/MS8Xx48dZtmwZq1ev5syf/4xotzMwN5cZL/yLzF27STmW37+u7kKQeiwfk6picj8kJPe/5vh4dj/4NS6MzEbSdRRVZfCu3ciqiux0ukoxmobicCDMZtri4ji66CZ2v/WWzwPOarV2264vdfoX2tjI3D8+A7rOkUU3c2bcWJdxhSShu+vl7ZGRHJ8zm91f+yr7vnIvNQMzelS7DmlpYfIbK7HVN3Dy6ukBj0k/cBBrVxE2IRCSxOHbbsXow99eGTwY67XX9vo4AEwmhKqiZKT7mFNIFgthDz7Qt3G/JLhk1ookSSlAihDikCRJEUAOcKsQImAK+O/MWukt7HY7eXl5lJSUYDabGTlyJNnZ2V7BvWvnnj8kJydzyy1903ZWT55CO30a8/BhmNyt/PX19axevbrfGSUeuO/L8Moqxq1+j6iqKgTw8fe+gz0qqt/OMXLtOgYfOuR3t2Y2U5eRwdnxY4k9c5aMnEOooaFUDRsKQpBQVExTSgoHv3KP15j3jhxJeCdZ2aNHjyIe/zqxQYyVfaYGFF8znVOzZiG7M/cRmzYzMDePk9fOpHjmjIv0SkkCw2Dim2+RVFTc7TkMi4WzI7NJKirG2tYWkD3TFBfHzm9/0+8YYz5YS3ovavQAMc8/h23RzVTdeBNaQUHv37BCQoh77VWaf/9HnLm5SCYTUlgoMX/8IyFzr+vdWF9QBGKt9Dv9UJKk94G/CSECih9fCeT9jxUrVtAa5HU7JSWFRYsW9es5L1y4wM6dO2loaECWZYYNG0ZycjL79+93tc73B9xNPKF19Qzat4/oykoOPPIwaj+VWWRVZdZzfye0sdHvfkOCHd/4OjP+8U9Mfl7rDVlm/U+fvsgsEYKZh3KJOXIU/fx5TBkZRPz0JxS9tpz4ba7W/c6aKgG1XHCVffbf/xVqB7tq9IrTyYhNmzk+f56LPeNzkGDaiy8Tc+4csvt73aHRQmQkNDQghYYi3boY9c23kHXd7/nbIiPZ9ejDLpu3ANTRmDNnuPqlVwLM3g9kmdSy00iyjFBVGn71a9peebXnxwMoCmEPfI3o//wP9KoqREsryqCBX3odlc4IFMj7lbUiSdIgYDzgI7wgSdJjwGMAGRkZ/XnaKwDmz5/Pe++9F3B/IK3zS0FKSgpLlixB0zSPnjjA8OHDPfrpeXl53eqkB4Wb5tcWH0fBwhuRHQ6yPtpA6913UdYDD87uICSJ86NHMXTX7ovbuBhgJQGznv+/IAMIJCEuBmbDwLZxE03h4aiJiUSUlaF981skdJW67WZeEq6FxYkr3+KTxx5FD7HitNkomT7dxWP3e5DE/q/eR9bmLQzMOYTUEagNA1paUAYPImHjBqpvvsVHU6UD50dmc/jWxa5uVrcQmL9OUNnqy2sPCquVmiV3Efnkk1hnXIMlO5s2szmwRrsk+ZbRdB2jthYAJTERPv9NyZcN/RbIJUkKB94FvieE8LFrEUK8ALwAroy8v857BS4kJiYSHR1NQ0OD3/2fZuOXvwVZRVHIzs4mOzsbcDFwDh06xOnTp3vfbt8BScIICaHg+nnQ0ZRyiXV6IcteBhJt0dHUDczA0tpGfEkJsmH46G57jgUa0tIuBlbDYMChXPY+8FWa4+M9NL7R69aTdvhwn1QBJV1n1t//z6U9kjaAvMW3BHWjN8xmTl47k4xDh7wXwDQNvbSMtuWvoZ844fdY1WLh8OLFfkXGOsOkKGRPnBDw9+IX7e049+6j9oEHiP7LXzBlZSGZzYhA94Kf+1UKDSXkhht6esZ/K/RLIJckyYwriK8QQqzqjzGvoPdYvHgxr7zi/3X3s/A3VfMLcOzahRQdTfjCG5k9ezazZ8/G4XBw+PBhjhw50jd6ZUfw7oeH04C8w4zY8QkCyF9wA2cnjHfR8gQomsq0V5YTEYQSd3ryJK951Gdk0JoQ72qrd287sngRhiQxMC+4omJXCPAq58ScOcvU199g+ze+HtSNPuZsOYZiQtG6ZN1C0Lp8ecDjaoYMQTJ0oEsgFwIMw2V6YRjEnDlL5Muv0DhgAOFVlT5snKDX1G6n6Ve/JnHfHpTBg9EKC3tUK5dsNsyjRmG7cUGPzqOdO0fLshdRDx/BfNVowh95GFNaWo/n+UXDJQdyyUVdWAYUCiH+dOlTuoK+wmq1kp6eztmzZ332TZgw4bLNQxgG9U8+RfvaD10BwGym8Re/IG7FCqyTJmK1WpkyZQpTpkyhpqaGnTt3UlVV1fsT9TUbdwfeuFMljP1grcuoITub8vHjvLJR3TBz8J6lzP7rc/4zclmmLiOdrPUbKL5uDnGlZQigLT6OiPMXGLR/PyFNzVQNH0bhDfPJv/EGsrZ8zOD9B3qUyXb9jAzYGhqJKTtDfdoAl3Wbn9KHIywsYFOPUe//jQ2CuBMJQUz5OZKKiogrLSW8ppa6tDRKp0xmYG4e0eXlmHqx6K1fuEDt/V9Fy/fTcNYZISFYp0wBwLZ4EaG3347UzdsCgFpYSPWttyMcDlBVnDk5tL3+BvGr3sUyelSP5/lFQn+wVmYAnwBHgY5H60+FEOsCHXNlsfPTgxCCHTt2UFRUhBACWZYZN24cEydO7DVdsKmpiYqKCkJCQkhOTiYvL4+ioiIMw2DQoEFMmTKFED8aIG1rPqDh+z/wacOW4+JIzs1BCpBNVldX88EHH3x6TBg3LI2NxJWd4aq1H2JSVSRg7/33Ueun8UdxOrl62UtEdnnQdPDNjyy6GWSJ0R9+5HKgByT9YvYqC4FmMuGIiGDnY4+gWa0s+O/fBqxRdweBSzDrQtYIqoYPpzozEz3EiqTrroeQO7DP/utz2BoaPIuenuvJzMQ0YgSOdb5fT81sdmm8dFlIVZxOJr++kriyMs+26sGDqRiZzZnx45j62griSi/BUCIApPBwUo7kecyye4rqO5bg3LvXZ7tl0iQS3l/dX9P7TPCpLXYKIXbSi1LZFXy6kCSJWbNmce2116KqKmazOWAAP3/+PMePH0dVVTIzMxkyZAiyLCOEYOfOnRQVFXmMjjsadTpKIUVFRZw7d44lS5b41MjbVq70q6UhHA6cuXlYJ030O5+EhAQeeughGhoa2Lt3L2fOnAm42HYpcEZGknLsGIbFjOSu0eoW/5meZBjofppYOjjfYz5Yi1AUV2B2P380sxlHeDghTU0u/XNNQ2puZuD+A5y6diYV2VmkHsvvW80ct0lzfgFpbj/R9shIHKGhHLrzduJPl3Ju7BgOLl3ChHdWEVpf78VO0c+eJfyRhzHq61D3eAc7k6oy/p13ObTkTiQhMGQZSQgyDuYQ2ymIa2YzSBLph3IpHzMGkPo/iNtsRP7k6V4HcQDngQP+t+fkIIT4VPR7Pmtc0Vr5kkKSJCxBXNN3797tJYlbVlbGJ598wqhRozCbzRQVFfltm48tLSP90CFMTidVV11FybhxDHcvaHZAdK3NdobRfSYaHR3NggULEEJQUFDAgQMHcDqdF2vR/fBFrMrKwtbcjLndjmwYpB49RlNioi+tTwiigmhZS+DpgGyNjibvtsU0DhiAZLha4Idt207m3n0omkZK4XFOXTuTolnXklJQ6LdFvqfo/BuwNTUhhIE9IoLyCeMRQmBrbAQhfDN/p5PGX/8HcgAd+qTik8x95lkujMxGN5tJOHnKa41A4NJciaqvx1JfT9bmLVRnDiGutLRfgrkUFoZ55Eginvh2n7nhUmgowg9TSrLZvpRBHK4E8n9LHDt2zK+uuaqq5AVZkBu6fQeZu3ajuMsR8SWnUU+cQHy41qtcErbkTtRDh7zkRAFQFCzjx/d4npIkMWrUKEaNGkVLSwvbt2/nXHn5JQV0yTCY9PpKYs+ccbnc4wpOGYdyOTdmDC0J8egWi4u6p+uMfW9NUKOIjhmUj7mKw7cs8igjdvDKi+fMJqSlhQHH8nGGuvTAHRERtMTFefw3LxUCsDgcXPfX59h/3700JyXRlJRE9ZDBhNfW+gZYhwPDzzpKB8zt7WTkXGyS6uC0I8u0JScT/ewzRNQ30PCDHzKkoACHyYSQpIB1+d4g8ukfYzQ0oJWUoF812kUz7CVCv3IvrS+/Ap21aKxWQu+9J/BBX3Bc0SP/N0NbWxuvvfZar4+zNjUx5y9/88nwjJAQ4p/7K7YFF9kEQtOoffBhnHv3ukosViuSLBO77J+EzJp1yddQUFDAzp07gwf0QPvc5syKqhFTfpYRW7YSWV2NkCRq0tM5MX8ezUmJDNq7j/TcPMJ7wFUXwEc/fRoRYCEutLaW6PJzVGaNQLdasbS2Mu8Pf+r3coQAnKGh7H3gq1z94kvIqtbrWnxH7b8zdVFYLJgmTUQ/fAQ5LIyw++8j4tvfAklCKypGigjHvmkzjb/81aVdQGgoEgLRbndp38sSsf/4R68zc+FwUPetJ7B//DGSxYJQnYRcO4vYvz+H5GdN54uEy9bZ2RNcCeT9AyEEhmEgy3KPXxk3btwY1G4uEAYcOcKoDz/C7Mf5xrZkCbF/9iYsaRcu4Nx3ALWwADk2ltDbbu1TdhUItY88SnlBAfuXLkWEuOuonX4Hqbl5XBhzlV8tbw/cZgrTXllOWE0NB+5eStOAVCa8/S6JvbAwa46LY+ejD2MEqud2fqi4O1WHbd/OsB07+z2YqxYLLXFxRF240Gshpe66TT0IsRIyew5xy/7ptflcWkb/auEAUngYKYfzfAKwdvYsqBrK4EEB733t7Fm0kycxDRnikZb4ouOydHZeweWBEILc3Fxyc3PRdR1Jkhg4cCDz58/vNqD3heYnyzK6zYYk+xlbUZCjL2qfqKdKqPvGN9BOngLAlDaAmOefu+Qgrus6JSUl1NTUEB0dTWx4OPFnzrLwd7+nKTaWnKV30ZYQ7/qwJDH8k53YY2KoT08LHMxlGd1ioXD+XKaseIPkoiLGfLiOcHf3YE/RkJoaeGfXNwN3p2rx7Nk4baGM3rCxV+fqCSIrK/ushtejB4vdgX3jRpy5uZ5SWcsrrwYP4v46NXsA0dJKxfULiP3jH7BOnoRaXEzdo4+jnT2LJEnIcXHEPv8clom+9FpTejqm9H8PVcQrGfkXEDk5OeTk5PhsVxSFu+++m7AgTuPvvvsutT0MVLIsM3jwYCIiIhgxZAit185GdNUkCQkhcd1azCNGIOx2KqZMw6ir8/rSShERJO/djRwdHfBc2pkzNP7nb3B88glSqI2w++8n/BtfR80vwK6qrDtdgt3hQFVVTCYTihAJZZ96AAAgAElEQVRc/fd/EHrhgmeMmrQ0cpfcgTMiguueeRaA3Q9+DXtMTPAL1TQycvNIKTxO3OnTvc6SHTYbHz/5Xd+OyG5q+ZKqMvOFf/ksJjYmJxNVUdGnbF03KeiKCUs3aov9AdPQoSRu3kjbqtU0/PBH/iVmASQJKTYW0csHZFeEPvQQ9vffw6ir976/wsJI2rsbxW3I8WXGldLKlwSGYfDSSy8Fdc8ZNmwYs2fP9pudl5WVsWHDBj9HecNkMjF16lRGjbrYQOHMy6P2/q8inO5FQk0j+r9/Q9jdSwE3f/yHP0K0tHiNJdlsRP78Z4Q/4N8FXa+tpWrWHIzGxotdfh3NLiYTh+fPwxEWhqzr1A4a6DERTtA0pvzhT64Fxk4LqxVDM9GsVlKOn0DWdTb98PuofqzVPHB/BxRVJbGomPHvrup1ED16w/WcmTrFa5usqsiahhbg3JKmkb15C4P37fdsa0hNYe9X78fa0sLU114nNIDkgs8l4FqQPHntTJd70IGDQRdpXROUu+2qDNqGbzJhys5GO3YsaLYtpaYi/Jgi9wVSqA3R1mURPSSEyKd/TMSjj/TLOT7PuFJa+RxC0zTa29sJDQ3tcQu9qqrdWqAVFxcTGRnJxIm+fO3Q0FBMJpNX040sy1xzzTU0NTVRVVWFzWZj9OjRJCcnex1rGTeO5NxDOPfuw2hvxzptKnInuzajosJl1NwFor0dPYg1WOuryzHa2ryDSsc4qkr2pk1Ihqu2rKgqdelp5C+8kZrkZOK3b0Pbs4eWF15AKz4Jmkayu6xjSBK62UzW5o8pWHC9T6OLB+4Hnm6xUDV8GNWZmSSeOhVwvl1hAOUTJ/hm3pJE6rF8zkye5F94yjBQ3NdpSBKGycTRm29Ct1ppM5vZ/o2vozgdRNTWMnzrdq+GHJ9LwNWZOXz7jp5NOiSEmGf+RP13v3fxd90FHQufAQO5JHUbxIF+C+IAwuFnrnY7xvkLvtv/jXAlkH8GMAyDvXv3UlhY6Gm4mThxImPGjAl6nDAMjH37UXQDXQleBc3JyfEJ5Lqus27dOp/OSUmSSE5O9ghcBYNkMmGdcY3ffebx45BMJp9gLoWFYQnQBASuRg2ClAIs7d6WZnFnznLt/73A+dGjMT30ENa7lhAybx51Dz2E8+gxMJvQ2+0UXT0ds+okpKWVlCNH/QfbLtAtFk5cN5uEU6d6nJVL4JcTbigKJqeTsOpqWhMS/J47pfC4K1gKQc3gQSSeKMLa0kr10EwMixnDYqYuPJy993+FrC1byNzjIyzaJ0T+4ufYblxA/ZNB7iNJctEOA5khu2molxWy7FPCkcLCsEy7+DZk376d1uUrEG2t2BYvJvS2W/1axH2Z8O8j5Ps5woEDBzh+/Di6rqNpGqqqcvDgQYqLiwMeY7S0UH3TzdQ/8ihZGzf1aOGouLjY1UjjRnl5uV+RKsMwOH78eN8uphMskyZhnjAeOjMMrFZMgwcRMnduwONMI0YE9Af1h46uytRjx1B3urwaldgYEt5bTdKmjcS//BLS+nWcvnYmaUeOUZeWhmazYemhnG5TUhKlU6Z0/0E3hJs7Hmiys/7+DxILC12St04nJrsdxeFg4ptvYbbbPdeTVFTM8O07mPDOO1zzrxddFncdUBSOz59P2bixPZ5XMDT97Oe079hB+CMP+78m8HTV+rvTRJf/XhZIEpapU73tB0NCMGUO8dxfDb/9X2ofegT7Rx/h2L6Dhp//gpq770V8yrIPnzWuBPLLDMMwyM/P98mKNU3jUACnGoCm3/0e9fgJRGsrg/bvZ9i27R4qWyBs3bqVl19+mQPulmVnoFdoIXBcwuKYEIILFy6Qn59P+2//h/CnnkQZMhhl4EAivvkN4levQgriPRr+4ANIAVrkg0ECmv/6nNc205DBWKdNIz07m/k2G4qq4gwPp2rEcM/vytLaysA9e5EDfbkVhZMz/b91dIXAlY3HlvqWPRRVJT0nFwmY/Pa7jPrgQ8avfo/x765i/u//SELJaZ/rkQCTUyWiqorMnbu6XLBE4Y0L0PvJSKH+4UewdzIt7hycO+Yiu7Nff3dZx2cuF8J/+APUkyddbwjuh2fInDnEr3oX4XRS981v0/q357wbgdraUI8exd6FHSSEoKWl5ZLu+88TrpRW+hHV1dWUlpaiKAqZmZlE+bEkU1U1oHRrMFedtndXedUyh+/4BEtjE/mLu3f9yc3NxTAMoqOj/QpSmUwmBvaRZ6uqKmvXrqW+vt4j0mWLi2Xxxg3Y3IuS3cGUlkb8myup/9HTaMeP98oCzKd7tBPiDUG9rhNfUkLN0EyPcfC05a+BYZBcXMy+++/zm007w8IC1odbYmNpjY8n/tQpT8PNxLffYfNT33MZNQOm9nZm/GsZoZ3UBgfl5mLQswCoaBppR45QPGe213bJMGgcMIDYIJ2ZPYaqoR0+7LWp6zV3djOiy/bLhshIEj54n9olSxFd6LOObdvQTpXQ9Nvf4uj64HNDtLXRvnEjtpsWAq4F/x07duB0OhFCkJ6ezuzZs7H2Qdfl84IrGXk/Yffu3axZs4bc3FwOHjzI22+/Tb4fmU6LxeJXMRAgPj4+8An8BLdBhw8T0cOFpMOHD7N9+3YfgwmTyURiYmKfA/n+/fupra1F0zR0XUdVVZqbm9mxo4eLbh3zGDaM6N/+D/QyMw+9M7D5tBQaiqTrpB05irWlBd1kYtzq9zA5nZg0jfjTpYQG0BpXnE50P+UezWymbNJE8m5bzOYfPEWdm6csaxoT33zL1fYvSWR9vBVbU7NPwOv4ua8lCSHLmIM8vHw+H+RcQpIonHsdRxfeSENqasDg/JmrkzQ1UXvPVzD8WBkKh4Pm55/HsXcfBHnDkuPiAKipqWHLli20t7ej6zqGYXD27Fk2bux/Pv/lxJVA3g+oqKggPz/fi01iGAa7du2iqcnbLEmSJKZPn+6jGNhB9/MHIQQNS+4kZ+kSDty9lPOjRnq0L+YePeZlxNwbKIrCjBkzWLhwYZ/HKC4u9nnDEEJw9uzZHplGCE2j4Wc/58KYcdQsuQvsAV51/cxPTk8jrAulsa2tjYOrVnF48a3U/ejHrlKFqjLjH//kqg/XEdrhLOTG6A0bvWvRbuhmM/aICIxO5zUkCcNsonzCeLSQELSQEA7ceze6yYQExJWdYc6zfyG2tJT0Q7meskRnSPQsIxfAudGjvY/VdULr6oMaXQSCP9f79shISq65mjMTJ7Dna/dz8pqrAx7fm2D+adTNjfPn/dvCGQb2nbuCuiZJZjNh99wNwJEjR3zeSg3DoKqqisYAvq1fBFwJ5P2A4uLigFZq77zzjk9Ay8zM5PrrrycpKQmbzUZ6ejq33HILiQG6H/fs2cPegRlUZGVRNWI4RxYt4uA9dyNFRxP7xz9w66239knVTdd1kpKS+hzEIbCFnBCiR/ZyTX/8E21vvuVirQSo4aMoRPzg+0T9x69RBmagpA0g4gffJ/mTHV7X3dTUxNrnniPh+z8kNueQVyA1O52kHznqE5ASTpUw/aWXydzxCZHnL2BpbSXmzBmsra3seegBqoYNw5BlhCRROWIEW777HbROr+ACqM4c4ilJWNramfLKch8d8EAItJAoAea2NmRVdS2OOp2E1dYx+Y2VvQqUngeHEGhuiqtuMmFIkkvz5Xd/IHPnLgyLheJZ12KPiAicwfdg7j3Z12cEuj9qayFQWdJkIur3v8M8bBiAT2LVAUVRgpqXf95xpUbeDwh2A2iaxsGDB5nShQWRlpZGWg+spxoaGigsLPTK9nWrhbphQ9EffwzT4MHEAwsWLOCjjz7q9dwbGxv91vJ7ikGDBnHy5EmfoJ2UlNQtN14IQeuyF4PWuQEki4WQ2bOwjB0bkGUBLjbQ4I2bXI04/jwfAxwXdaGCqAsVjNi6DXBnr7JMRVYWh5bcgeJwEHn+PHVDh/ocq1ssKF0aVGQuNugEC+gtUVGE+ckCO+Y56FAuA/ILaExNxdzWRmRlJQA1AwcSX9Z7IwdZ16kePMglOSvApGmYNI0RW7eRevQYux95iJrMIaTlHfY51mmx4AwPJ6S52a10aFA3II0Et3yt02ajdNJEGtIGEFFRyfDtO5CE6PEDrd8hSUg2GwmbNmAeNMizOTU1lZqaGp/kStd1Yr/AnaFXAnk/IC0tzWWCEAB5eXmYTKZe2a0JIaisrOTo0aN+M1sNKK+uJt0dXNLT05k+fTp79uzp1dwjOjX09AVTp07l/PnzOBwONE1ztc8rCrN6onKoqt0H8dBQQm5cgGVs97S78vJyZpw52+vg0TUgKkKArpN84gT2LR9zYs5sV2eoEITV1jLqo/XElZahm82cHTfWJXvrbywh0BXF82bQ9Ty2lpaAwdiTlTscxJ++yG7RFIWGtDTi/TQHBWve6cjM406X+pR2JCCipoaY0jJM7Xa/Y8hCcHzedbTGxWG222lMSUHWdSa8sRIUEwfuvdujaVM9dCi62cLw7dtRNC3g9X+asM6ZQ9SvfukVxAFGjx5NYWGhZ6ETXGXNkSNHBly7+iLgSiDvB2RnZ7N79+6gnzl48CDh4eEMHz682/E0TeOjjz6iqqoqYBdnZ+OI6upqqquriYqK4p577mHfvn2cPn2629JGWFgY0UH0T3qC0NBQli5dSklJCdXV1URHRzNs2LCgphaea7BYUDIy0P2oMUpRUVinTiH0riWELOiZ4a7VYsERHkZIF4mAvkLRNAYezOH4dXNoTkrC2trKNctexGR3uKh5DgcDcw5hDw+sbUMgH0wIKDHblaPdlJRElDsbN+k6Q3f5Z2d0MEyCBcxg9fnszZsJr/Gvh2JSVVIKCsm74/aLYxkGbXHx1AwZhOi05jP2vfdJLjyOqZPe++UM4pLNRvzyV/zuCw0N5Y477uDgwYOUl5djtVoZO3Ysw9ylly8qrgTyfoCiKMyZM4etW7cG/dy2bdsYMmSIz0JnV+Tl5VFZWRl0sVAIwaFDh8jPz0d1f2FkWcZqtXLLLbcwbdo0du/eTVlZmd+AbrFYWLhwoVeN2eFwdOss5A8mk4nhw4f36CHVFdG/+U/qHn38YmYuSUghIcQvf9VL0U4YBm1vvkXrq8sRDge2228l/KGHkDvpmIxwqjSkpBBZUdlvgaNDf6V62FAyDhxE1nSvsRVNI7TRf90Veh/A/AW9iKoqSidMYJC7z6AnC6XBMvNA+yNqagPuVxWF0smdJD6EwFZXh8npYOz7a2hJTKQlMZGwmhpSCgpRNA3NYuHE7FmM3Lipmxn3L5QuWXhXhIeHM3v27Msyl8uFK4G8nzB06FB2797dbYNBSUlJtwHvxIkTPWN8CIG9U/NDR6fopk2baGxsRFVVTxCXZZmYmBjS09NJTEwkPT3dU8Oura1l27Zt1LtNFFJSUpgzZw6hwYSm+gkhc+YQt/INmp95Bu3kKcyjRxH51FOYR430+lzdE9/Bvn6Dp9mj+ZlnsX/wIQlr13ic1VNXrcaZd7hfs7+W+HiSioupyRxC1IULKP4obgHefCRcD6DewN/cJSGoHTKYtKNHPVlub8foCM6BgrhuMlExaiQtcbEM37YDyTAQQO3gQZSPGUNdRgbtneSKkSTa4uJQVBVF08navIWD995D7JmzLkYVkHv7bTSkpjBy0+Z+1ykPBq2wkMrr5hH1s5/22S7ui4Yrgbyf0Nra2iP39wsXLnQbyC9FkVIIQU1Njc8YhmHQ2NjI4sWLvd4I7HY7a9as8WT14DJlXrNmDUuXLr0sHofWSROxrgjsWtSy4nXs773vvdFuRztdQvtH6wm9xdUUJRob+21xTeDibIc0NlL44NdAUWhKTia+5LRPSaRz00znYNld8PR3zkCfC6+u9v8Q6YLuuOD+9rdFRLD70YfRLBZ0q5WGlFQmvv0OJ+bMpnzCeBef3s99YJjNXMjOJrmomITik4z6cB2ypiEAe3g4NZlDMEwmyseMIe2w/wfsp1V20U6coO7xrxPxox8S8dijn8IZPl+4Qj+8zCguLqamGx7wkCFDLukcgR4EkiT5cGX9Zf9CCNrb2zkXRLHwUqDX1NDy8is0P/931MLCoJ9t/NvzNP7ox373idY2HJ3qxVIPFSR7Agm3OqGuI0wmJE3jzITxGCZTt+3qXQN7f3Cwh32ys9ustq+PsPyFN+IIC0N30yprh2ay6anvcWbyJJdipJ8gHnX+PNf9+VmuWudiSknAoIM5DDh6DJOqoprNHgmEI7fcTF1amn+qZT/JDfiDaG+n+Xe/dylrfslxJZD3E8LDw3tE4zMMo9susokTJ3ZbRw+GQMd2iHN15tI2Njb6XVA1DIPmHopM9QbtmzZTOe1qGv/rNzT9v99RvegWGn72c5+Hj15XT+XNt9Dy298GHsxkQunkzmMamtnv822Ni/WwUkZ9tJ6C+fPQe/i3CRbAuwY1zWzm7PhxrkYjd6mos4ysbBjdjhesizPYcfXpaT4NNcJqDRhkFYeDqa++hq2p2VPq6ZiborvWEMIaGy/ORZY5cttin07ZDp2aTxUmE9rxE353qUVF1Nx9L+cGZ3Jh9Biafvd7RA9KV59HXAnk/Yi5c+ditVoxm81BSxItLS2cOOH/5gKwWq3cdNNNvW7UURQFs9nM1KlTAwbzsrIyVq1a5eG+JyUl+f2sJEnBJQP6AKOtjfpvfsu1sGm3g6Yh2u20vfW2l06GdvYsFVOmoOXmBh9QgrC7lnh+DH/0UeinrFzgCq5Hb77Js82kqoxd+yGSEN0GzGC0wo6xNZMJXVHQzGZKJ0/i6M03sfU733aVNEaPCjqOv/P1tf1/+suvAhBbWsq4d1Yx6fU3GHD4MFIAVk1y4fFuA7BmMmFyOj1vEW2xseTecTtaJwmGyyG6JVQncmKC7/zOnad60WIcO3eC04lRX0/zP15w6bN/AXElkPcjYmJi+MpXvsKMGTNIDebjiEujJBiSkpJYuHChV5NCR3BNT0/HZDJ52CIzZswgOzubSZMmcffddzNq1CiuvvpqzAGkYTVN4+jRo4Cry9Rms3k9NBRFISkpiYQE3y/ApcDxySd+W6lFWxtt777r+n+7naqFN0MXDXJ/iHjqSZSUFM/P1kkTCf/ed/tlrs7QUHY+9gj1GRkAJBSfJLa0DMHFrNMzfz/HBwummtXCzocfREgSB+9awsYfPEXRdXNAklBDQigfN5a4sjO9DnLdBUZ/GbsEhNfUMGbVe0x+fSWp+fkkFZ9k9IcfMXX5a74BWwisra1+5Qc6o27wYBzh4V5lmaoRw9n85JM0u3VPPnWYzVjGj8fkp/GuZdkyhMPhXa6y22lfvwHtXP8ZYVwuXFns7GeYTCYGDRrUrWhUe3s7lZWV1NfXe9r0u2bgqamp3HnnnQgher3omJWVRXh4OJs2bfJayARX2eSC2+vSZDJx2223ceDAAU6fPo2iKIwYMYLxblPdnkLTNCRJCt7NGazG6w4YLa8uR3TRQ/ELm42IJ54AXAu2jY2NREZGEvXUkxh1dbS99HIvZu+L9shIWuPiiLhwgdiyM2Rv3uKzkCrA1dIuy1hbWjyLoLqiYCgKkq5j8rMwKqsa495bg6xpTHljJR3GxK0xMTjCwogrL7+kuQdaVFQtFiwB2twHHD3qldWZVJWY8nMkFRRSNXyY6y1Elrnq/TWYnSqGogS1kqsZPMj/AqlJoXrYUCIu0b+zJ7BMmkjcP1/wu0/Ny/Or3SJZLGjFRZgGBE/EPm+4Esg/BdTX17uc57vJWtasWYOiKEiShMlkYtGiRX4bdPrKHImMjPRLY5QkyaueHxISwsyZM5k5c2avz9HY2Mj27duprKxEkiQGDBjArFmz/FIXrTNn+jXolUJDCbn9Nk6fOEHj8uXEmM0owdxnrFYSPvwAZ3Exu/bspcTpQDGZMAyDzMxMpsydS9sbK711qXsBAURUVrLgv/4byd3QE4iyt/3b30Q3m0nPzWXQvgOYHA4qhw+j+NqZpB0+QtaWj30eAIph0JyYQO7SJbRHRWFrbGT4lq2kHTtGuJsCGmxufS1HOCLCMdfWBVRk7ArZMBixdSspBQXIhiD+1ClMmoYuSbRFR2NrafFq+gFXycglbzDC/5i6jqX10198lMLCXH0GAYy3zdnZOHMO+SgmCqcT0+DBn/r8+htXAvmnAJvN1mMeeAdlUVVVNm3axJIlS7o5queIjIwkOTmZiooKr4eKoijd2sr1BE6nk/fee8/DnRdCUF5ezvvvv8/SpUt93jDksDCi//Is9U98x5WdqyqS1Yr51lvZVFrKVY89Tnxrm1ett2uQMU+fRvzyV6l/4rsccToomToFw2zGcAeUC9t3UP3Syz3iWweChLtNvxuYNM1DzTs7YQJn3RIMprY2hKJwYfQoRmzb7hMsykePIv+mmzDc9eL26GiOLXLV4tOOHfN7rs4smL7AER6OPTyCiFrft51ADwdDkgitqye8zvvhoghBeH095WPGEFZfh6Qb1GQOwWmz4QwPpyJrBIbZ7PcNTDIEyf3gRtUtJCmo61T4Y4/S9tbb3s5BVivWmTMx9VHS+bPElRr5p4CIiIg+qQo2NTUFVGfrC9ra2oiJifGUO2RZJiwsjHnz5hEfH+/hnJeXlwd0DwqGkydP+rx1dDQplZeUYN+2DfuWj73oX6ELbyR51ydE/eRpIr7/FPGr3qF0yR1k/fEZLJ2COPgGl8j//R8S33mb1hdfwr51K6UTxrsChhvxp05xzYsvuTTBLwMcoTZkXWfA4cOMXf0ew7d8TFxJCVNXvI6QZexRUdSnpfk4+pyYN9cTxDugWywUzZ3j9zwCaI+KDKpKaBC4Lm8AjUlJKJraO+XEIA8zCUjNz6chNRV7RDhlkydROn0a50ePAiFQHA6y12/A0tKC4nCgOByENDYy7dXll/SQ7fH8DYOQAN6yAKaBA4l/+03MY65yBf2QEMKW3kXc/z3f57l9luiXjFySpAXAs4AC/EsI8b/9Me4XGfPnz2fLli2cP3++x5KukiT1qKmoJ2hpaeHdd9/1ciTq0ELPyMigubmZdevW0draiiRJGIbBpEmTGNsDcaoONDY2+p1vVPFJpP/5X+o6vCw1jehn/0yo26FFSU4m/NFHPJ+v/N//JamxMWi2GfWrXxJ+//0AtL66HOx2wqtryNy1m7DaWgxFIaymGuUyiu3Vpw5gxj/+ia2xEZOqoisKQ3ftRghBXGkZtYMGknPXnUx4511iz5zBkBWQJOyRkX7Ha4+K8sqOhSShWiycmH0tTQMGcPWLL/sc0xwXR1XmEIbsPxD49ydLJJ06hSFJAT9j4JvVdZf9y7rO4H2uRfvkomJOTZ+GZBhEVlQQe7YcRdcZfOAgTUlJSIZBRFVVn94oOjdW9fgYu52qRYuJX/6K14J4Z1jGjSPxo3UITUMtKUErKEAtPO4yEb8MjXD9iUsO5JIkKcBzwHygHDggSdIaIUTBpY79RYbVamXhwoW0tbXR0tJCUVERJ06cCFo3N5vNxASo6QXCihUrPFTCG264weP0k5OT46XwBq4W/l27djFw4EDWr19PU1OT1/6cnBzi4+MZMGBAj84dHx+PyWTyCuYmu52Jr7+BrHpnfw3f+S6WceN8FpHU06VYT53ytHV3hcAlgtTZQEK4a99Tl7+G7GaQ9KR2rIaEcPy6OVwYPQrJMBi6bTuDDub0umGn4/MJblXCjkVOWdcRsowkBBPffIvjc6/j7ITxHLj3HhKPn8CQZeoGDfS7CAi4XIzMZuwR4ZROmcJZdwMSgK2+nsL58xi8dx+2Tvz+w7cu9lJH9AfZcMs0uGmT/s6uWq1YHI4e18999glB5u49LmaMyYQaaqMuPoH406eJqqi4pNp+sK7UgDAMtMJCah94kMQN6wN+TDid1D3+Dew7drgayoRAGTyI+JUrUWJ79138LNEfGfkU4KQQogRAkqSVwGLg3zqQdyA0NJTQ0FASExOx2+2UlJT4/Zwsy8ydO7fHmUBraysrVqzw2rZhw4Zuj3M6nZw/f57m5maftwRN0zh27JhXIDdaWmj60zO0r1rt4uTGxSGHhWOdOYOMBx/AZrPR2trqyfpTi4r8XoMwDNpWryby29/ybHMczKFm8a2EjR8f8DVeAmLfXInUScjLOn8e7W+s9GqV7+63Zsgyux5+kLboaI9S3/Hr52OYFIbs3d8rvrYADEXh3FWjaYuJIen4CRoHDKB49rU4Q0OxNreQtWULozZuYlQnwShNljl66y2cHzXKl4YpBLb6enY+/iitfuh57bGxnJ4+jdPTpzFk5y6yP96KkCQaB6SScOpkD2cfQMsFsPTChFg3mSiZNpXzV41G0g0yDh0iI+cQcofSo6YhtbQixcVx8M47mbBqFUofG38utYVfLSpGPXkSsx8teYDmvz2HfccOsNsvLtgWFdPwgx8Q9+KySzjz5UV/BPIBQGcn2HLAx7NMkqTHgMcAMtzc3H832Gw2JEnyW2YxDIM9e/Zw0003eXSRdV3nwIEDHD9+HFVVSUpKYubMmURFRfkE8Z5C13UuXLgQ8IHR3kkfXBgGNXfciVp80uXgA+gNjeiAevw4ba+/wS1r3ufg2TOcPn3axVqJjkHxN7bTiehU/xdCUHvPvQCkHTlCw4ABRJ875wnOnsU9RaHthRdQvv44FjclMvyhB2l/Y2WvrrtyxHAXVdAdxGVNY/IbK4ku95Uh6PzX8fdbqh04kP33f8XT+Xhq5gzXwp77Z0dkBMduWois6aQWXMxnTIZBzJmzVIwYgdFVYVKSaOjue+H+vZ6+ejpxZWeILykhvKKC2NNl2MPDsXWR7+1NEOzp53RJYveDD9CSEO9Znzg+fx61QwYz8a13PJ+T3eWliAsVLiqmEB4GUE/RlyDeFhVFwdzraI2PJ/nYMUYczMEIwgRqXbHCl92kqtg/3opob0fqoYH4Z43+COSBaKveG4R4AXgBYNKkSZ+RbTEihHEAACAASURBVMhni5SUFL+GzB2ora3l1VdfZfLkydTX11NZWUlLS4sn8FdUVPD2228z+BLpUXl5eX63K4rCoE4SoI5t29FOl3qCuBecTgxdx/n888z6w+89RhLqyZNU+QmyUmgoIdddXMzTz5xBuBdBFV0n9swZ2qMiMdvtyKp2sSVd17F/uA7Hlo+JfuaPhC5ahGnQIBcjoReLZk3JyR4tEYAhu/cQc7bcrxBVQ0oypZMmEdrczPBt271ucM1sZv9993pMFC5eoPfXQLdYOHHdbK9ADpBSUEjh9fN9J2gY2BobEYriVUM3tbeTvWkzKfkFSEJQkTWCwhuuJ+eO2zBMJiRZ5tDdd2GYTCSdKGLs6vdc+jBBfheXkuVWjRhBa1ys1yKzbrFQnZlJYyfddLiY6XecqzkhAXtEOFEXKrC0t9MaE0N7dBQRlVVYu+ihGB0Ux+Zmn79R55q5kGVP09KxBTdwZvIkbI2NZOQcIrShkaPz5nJu71703FwSExO59dZbvccKRFEVAqGq/1aBvBxI7/RzGvDFa426RDQ3N1NfX09UVJRfzRUhRI/dew4cOBB0/+luaqI9RsebgSQhORyM/ngrSS/8i6rEBCK++Q20ouLgDj66jmO7d+OTeehQwu69h7aVb3oCNTYbllnXYu5kd2c4vFkyEgTW9RYC0d5Ow09+hu3GG9GKiz0NRD1FaH09isPhCebph3IDqgma7Q4qrhqNYTYTfe6cl+Jh1fBhvkE8ANqjotA72b1JgLWtjQnvvEvuHbe7HITMZqLLzzF+1Wqsra2AoCU+nkNL7qQtOpqrX3qFsLo6Tydlan4BsWfLObR4MU3pAxCK4vHirBw+jMIbrmf0uo+6rW33NZjXDhro9UDsgJAk6jLSXQuanTLvjnPVpaez/6v3ISTJZYGnqhffSoRgwJGjjH1/jefzRddeS+m0KUx/8SUiqms8C7Ed3amGorgWb4VAcf98ITuL2LIzTH5jJZJb8CyxuJjM3XvY+chDVFVW8sI//uGemMTw4cMZM28e7avf8+lvkIYMxg58+kLO/YP+COQHgGGSJA0GzgF3A/f2w7ifa2iaRk5ODsePH/dQ90zuppSUlBSuv/56Lw2Turq6PlH8PlV0yiKFxcLhBTdw1G5n4so3SfrOdwmZPx/JZkME8SSVYnwbmKL+8z8ImXsdbW+9jXqqBL2sDMfGTVSMGUfkU98j7MEHMQ8bCiaTD8c6KBwOtLIzNPzsF34bi4IhJb+A4/PmunjfnbI4fwivryexqJiqYUPJveN2xq1+n4STJxGyjNYL0w1zu51jCxfQGhePHmLF1tBA5q49JBUVM+8Pf+L0lMmUTp3C1BWvu3RJOs5fWcXwj7dSNGc2VcOGkpZ32JOxyu7M3REThaKqWOvraY+KcnHp3dZzIz9aH1DOVzObOXTn7Yxf9R7mXtTFO2BrbHQF4S4cbVk3sDU3+y2fSEBMeTmGonjuOaOzqqIkcW7sGEJraxm+cxe62UxzciK61crOxx8joqqagQcPMuBYPorT6fJE1XVkOj2MJAk1PJxxy17yojeaVBW5uZmhO3dReMP1ro1CgBAUnThB0chsGJkN7e1Mf2U5MbW1GEIgFxVTlz2K8qgoip56kqmLbyEsLOyyaPT3BdKlaF97BpGkhcCfcdEPXxRC/Hewz0+aNEkcPHjwks/7WUEIwQcffEB1dbVfFookSaSnp3PDDTd4atHV1dWsXbvWp13+cwf3/RBSU8OMFW9gM5sxamv9N3fYbET/7v8RevttfodqffsdGn/yU6+sXrLZiPzFzwn/2ldpfvElmn7xS5/jlMGD0f29dVgsJB3YR+W4CQHb/ZX0dPSqKpfjepfPtMbEcHjxLTSkDSBr02YGHTgYsM3cAEquns6ZSZPQzSZSjx5j6Cc70S0Wtn7n2341Y7wHMFwPDF13lcZMJjAMFE1jzPtrSC0oJP/6+ci6zvBt2z0ZvyHL7L/vXhoGDEB3S8FKQjD59TeIK7voC9sSG0toQ4OL1SIEJ2fO4NSMa5AMg+v/3+/9crUNoD0mhm1PfIv4k6eY9NbbrvGDX4kXHGFhbH3iW95ZuWFgaWtj7jPPBv59ShLbv/UN2oLorJjsdq755zIKbphPzZAhrjcf9/dHdjpRVJVrlr1EWH09AigfcxVHbl0MgGK3Y7HbmfX8//l902qLimLr977j/8Sd7hNZ00g/cJBRmzZ7LWw3pKRgtdupyRqBce89XDVr1iXbJPYFkiTlCCEm+Wzvj0DeW3zRA3llZSUffvhht5zvYcOGMXv2bA9P+7XXXvNy9PlcQwgQYGltYdjWbZRNm0p7VBTh1TVkbdlCfGkZAPLgQUT96IeE3nKLzxAVU6ej+9ENkePjSTnsUja07/iExv/4T/SKC5hHjUI8/WPaTpWg/PJXyJ2bo8xmrNdcQ/yK5ZwfkY3w48spRUaSWpiPVl4Ouk7Dr36NY9Nmz37DXeZQzWbybllE1vbthNbV+2VU+BXCUhROT55E6aSJ2GNjfWmEnb9Lbv0Uz/93grm1jdl//Rs5dy0htaCAgTmHPPtKJ07g+PXzXTrgnWBpaWHeH58JzAOXJMomTaJs0gRm//0ffq9HAC0J8ZyePp3ycWOJqKxk8usrveiMPUFdehq5d9yOarMhJInQ+gayNm+mesgQTA4HKQUFRFZVe3XoCklCKAoVWSPIu+1W/w9Ct+k1nQJ4Z4RVVzP+3VVEVVahm0wcvmURF0aNdI0lBJbWVq7781/8eqE2x8ez41vf6NkFuhlEALrZTEhjI9FnzpL18VZkQLXZ2P3Et5hz662k+RHk+jRxJZD3EFVVVWzdutVjwCBJEjabjUmTJpGVlUVDQwNHjhyhqKio2zZ8WZZZsGCB54/9/9s778Corjvff86dPqqoIyEJkBAgiimmY2yMwTjYuCX22nFeErzx28RJnPaSfcm+fbvr7G524/V6Uzd24hIbl2AMxtgYY8A2phuDACEQRQ1JSEJdmtGUe8/+MaNRm1FDMBK6n39AM3Pv/c2R7vf+zjm/UlFRwbvvvntFHYCuOf5paOcbT3G7mffa68QXl/huVqORmF/8C5EPPdTl0PLMCSGXTlLLShCdztnS0sK2bdtobm72Pfjcbqa+v53MUwXg8WCadQMJz/8JJTaWxp//My0vvNg12sBqxbpqJebcXAwZ6dhuv52Wl/5M8y+fQjqdXRJeSm+YyYm71yI0jaXP/pHo6uoe9nksFgpWrsDY6iCyvp6mlBQqp0/DbbeTeuIkrWNiaRw3LmRMeG8oHo9vTIUg+fQZZm7ditHt86D3Pvp1GoKIg8Hp5Jbf/A5rL00SJL5Zx9lbbsbc2krG50eJ8jcx6bwmrgGtfo++swfdlJBAwe0rufEvb/Za60YCbRERuCMiMLnd1KelcmHhQua9/gYmp7PX2ulek5Ezy5dTvGhhkBPLwAPQ5HBgbWzEHRGBKyqK1JMnuWHzlsDSjSYEmtFIc1IiB//XVwIPvkUvvEhs2cUuS0tek4nTt62gZP68kGMX0pb2//sxtray5E8vUDFzBpV33cmDDz7I0aNHA71zk5KSWLx48ZCXgG5HF/J+UFtby0Z/OdWhQgjBLbfcEujSfejQIfLy8q6JmCckJBAZGUlx9y71ITzFgRBTXsH8V9ZjbhdTkwlDcjKyzYn19tuxrfkC9U98H62mpsexhrQ0Ug4d6GSOZMOGDTQ2NvYYl5iqaqTNhiNuDIsXL2bKlClIt5v6J76Hc/sHCLMZ6XYjrFZfUwCXC2G1Imw24te/TO3Dj6A1NNAYH09UdTUK8NHj36TVf6PNfHsLaXnHe6wpe81mDn75YRrGpfXwHq0NDWR8doTCFbcObgw7iYRQVZY++xwRtXUYVJV9X/9qoHRuF1QVRUqm7NjJhF5KIHtNJo7ddy9Vk7JRNI35f36Z+BAhlt0t7yvsEuDitFzy13wB1WzB4HaRu+19bI2NjLlYHngoXLxhJueXLsHS3Ezu9h2+ce80vq1jxvDpNx712ds9KqSTmPs+4GXJ8y8QWVvXZblIFYIjf/UgNVkTfb8f/3hamppY+NLLWFtakPj2FCqnTiHvnrv7Xg7rD1L6qla+vJ7D33iUjIwMysrKuibFGY188YtfJDpEBu+VEErI9aJZneirc89gkFLyySefEBkZSV1dHaqq9qsy4lDQ0NBAbW0tQggyMzNZunQplZWV7Ny5s8PThkGJUUtiAg1jx5LUvpbt8QSWURzrX8Xx6mu+deFuCJuN6L/7aZfXamtru4RZBtA0FKcTxeHAGx3Fvn37GDNmDMnJycT9/neoFZV4i4pwbN2K442/BMIkZWsr0umk8Wd/R+K292h88kmKa6qJrKsDr7dLh5/iefMYm5+P4um4ETUhcEVG0JCW2uPmtzQ3M/ZUAWNKSln+q99gaGvjwx9+n+Rz50k+cwaP1UrZ7Fm0JCUFHTfh9aJ4vaj+XAFpMLBv3dfJ2ruP1JP5JJ49T0NqaiDePYDBgAacXnErcWWlxFReCnp+o8dDSkEBVVMmI1SVmKqesw0InRjUGy6bjRNr70Izm1G8Xha+9DIRtbUY/X06BVC47CYuLF5E2vETTNv2ftDNT1tjIyufehqA+nFp5N17D8729eZOG6AA8WVlRF72XaMdCRy7715qsrN6/H5c0dF8/O1vEVdairWpiYbU1F7X5QeMfyZQPmM6UkpKSkp6zMxVVSUvL69LNdGCggKOHTuGx+MhIyODRYsWYQkS/TNYdCH3c/jw4avS2gx8v9h33nknkAykKEqgdC0woA3QuLg4mpqa8Hq9gWJYoR4Knb2EkpIS6urqeOCBB1i3bh1lZWWUl5dzqj3OufNUsh9Ym5oou2FGh5B3x1/dMICiYMzKIuJrX0WtvETLiy9iu+MODMnJuFyu4AlKioIAbnztdU7esZry2bM4ceIEycnJABhSx2JIHUvdd77bM9Zd03DnHUeJjCD+v3/P9D17aHz4EQBSj5/gwk1LQQiaUsdycs0apr+3DSkEQtNwxMby2cN/5bPJ60UajRhcLuIvFDHnrU0I/wZh+2bYwpfXE1NZidHjQROCzM+OkH/HasrmdKrpLiVoGnElpczatJm6zAyO3Xcv0mBAtVgovHU555YuRTMZQVURXm+Xzb7A1zIaKJ01ixmVwdPONSHw+gUivrh4UH08Q4Umes0m4ktKcUVGEFNeQURdXUBgBb7ZwPkli5nx7nuknjgZVMQlIDotvYwpu8ii519k9xPfCRrWmVR4tsfmZX16OjU5k0J72EJQdyUVDPu6F4Tg0tQpJCQk0NDQ0CMarb0YXTs7duzoEjJcWFjIhQsXeOSRRzAPIAqqN3QhxyekoZJkhpJ2j1PTNBRFITc3F7fbTUEfDYg7Y7VaMRqNVFdXo6oqcXFxIftudr+20+nk1KlTHDlyBE3TkFJiMBiYWl1DgdWC2h5a1YegC1UlZ/fHjD01gCoMioJ51Uoa/+EffevmQtD4D/9E9FO/xFxQgBoXByYj9ro60o8cxdbUxOUJ44msrsbo8TD9/e0kFBVhN5pwuj1YV9+OWlxC4z//C1qQNe7uuJ96OrCpmbVvP0UL5iP9IXDlN8ykMncq0ZWX8NqstMTHY/B6yTx4iKaUFHK3f0BUTU0XAQqMBTCmrCywdKBICV4v07a9T2XuVLxWK8LrxdrUxLxXXw80VEgpOM3CF17k+D1305qQgMHlZlxeHmaHg/jiYurSx3H25pt7euaKgsdmw6soGILYIw0GymbPAsDc6hhUpcFQv31TmwtHTDSuyEgsTU3s/MH3EJpG2vETTN61G0dcHPHFJaQUnA4Z/tj9/IqUGF0ukgrPUjV1CkJVmf7OuyReuIA7wk5rkNpDNdlZPfp/Dil9OTRSosXEsGzZMjZt2hTkcBHo7NXc3Bw078Pr9bJ3716WLw9e8XKg6EIOQddmrzaaplFeXk7tADulVFR0zbWq6083HT9er5fDhw/3mAEkvbKezIYGHFFRnLhzDZezsjA7WnF3a9UFgJSkf/55j4zFwNuESPVVVRy/+33Hco7fQ23y90jMWbiQhvRx3LB5M4qqoWgayadPd3hjUhJXVIy9pYW6bz2OYeJEtIoKX4x7iN+dMWcSij85y338eOB1s8vFyqef4ei99wSm5zGlZaSdPMnlrCxiyitIP3aM+JLSftUBDyZamqIQX1RE1eTJ5L6/nfS84108S0VKxpRXsPQPz3F87VoQ0rcUga/D0OXMzOCJR5rEERNN/p1r8Fgt3PD2O75Rl76iXQW3raDJX+0v8dz5oPb29Z1CvW90u4krLePirBuomdThERcvmE/1pGwW/+kFEs+d67WMcLBrKqqKvb4exeXipmefw17fgCIltuZmImsu97TD5UJR1UBBsWuK31tXVZU333yTrKwsiouLe9T7b68iWlhYGPJUpaWlId8bKPpmJ7663a+88kq4zbjqBF2KkZIv/NPPu9wsqr/c6anVt3Nx1g2BiADhdhN9qYrFL77UU7z8yT31qWOJuVTVJRqiP2IIPvELFYesGgycun0lM94LXcmuCwIwmbEsXcKY3/2WS7nTB5wNeiVIwBEbi6KqfYb3qUajz5FoF8Yb51J46/IOb9wvHgaXizFlF7k8cULgs4rXS8KFIoxOB9bGJor9pWQBlv/q11gcPTNzJeCy2wNJRqLbe+1hmkGXV4xGtv/tj3s0uTY6nYw/cJCYqiqSCs/26pEHs0cC9enjiK2oDBo+2BlHVJQvJnwoNi+HgJkzZ1JQUIDH4yEhIYGlS5eS5N8jOXXqFJ9++mnQ4yIjI3n44YHlTuqbnb3QXp2wuh9T9OGKzWbD7XYH1t+FEF3K2CqKgt1up62trYuQG10uXJERWFs6sjfbu+NM2/Y+8UXFlN44B9VoIvXECdKP5QW9SS1Ll+Dau4/8L9zB3A1vYnI4MXo8IZcAghFKxCVQPSmbqsmT+y/kEnC7ce35lIbv/+CKInQGgwAiGhr69dl2T73dj8387AhtsbG+h6jRiMnpxGO3o5rNKE4HE/buo2jpEhSPF6Fp1GWkM3PzFmIrKhj/2RHqMjMwO5yYg4g4+BJcnDHRQUMZBf5yt/7092C2xlZU0pDuC5GMuHyZWZs2E33JX2NF0/oc62CzNgHEl/WvV6mmiODXClMN8RkzZrBw4cKgvXUnT57M3r17g874h6JLVzu6R+7H7XazdevWLpsUIwWj0chNN91EcnIybrebuLg42tra2L9/PyUlJQghmDhxIvPnz+f111/H4/Fgq69n1qbNxJZXgN/76q0jTK+YTKQVX0Brbuat996jsaaG1JP5xF68SNKZQiwOR7+EPBRum5VdT3wXe109y559buAnMBrBbIZeYrCvhCsttdrXuTWDgepJkyiZO4e6cWmsevoZivxeu6JpGNvaQAhmvvkWYy5dwuR297ps4jWbAyUBeqt4F+q9U6tWUrxoIQaXi1v/69eYnM4e7fmGckzaz9X+mC+bOZOWlGSyPt1LY1oqqslERW4uVZNzOmYK11DUzWYzX/va10K+X1xc3CMiLiMjg9WrVw/4WnoceT9xOBzU1dXR0tLCJ5980vcBYUYIgd1u58EHH+xS20XTNIqLiykqKsJsNjNlyhQSExMpKSlh5/vvc/PTz2Bube3XFLivGzvqO98h5m9/DPjCrPbv3x+ImFn95D8PuhY1+BJITtx5JwA5uz/qt5fbHcu99+DatHnQdgxHNCFwRUVhcvpmP80JCYFuRaG4UoGVQP7q2ylZMJ+sjz4m5+NPrmq/yFB/nW6zGUuQ2kUeg4HCm5dR3L3NW3+EvT0kt1sIZF9ERETw5S9/udfPqKpKQUEBTqeTnJycoIX1+oO+tNJP2htBgC95Zzin1CuKQmxsLJmZmbS0tARqP6iqyubNm6mvrw/EuBYWFrJgwQLS0tKY4+9+3l3EQ93kmsFAU1IisZ1ilyXQnJhA1Z1rWO4XcYApU6ZQXV3NuXPnUBQFt93eo052X0h8YXZIqMnOZvyhw8RcuhSoADgYXG9v6ZpocpW5ml56O4qU2DqVMYjqx2zySm0SQFxxCZHV1WR+fvSqf8dQ57e43UHH2KiqTNu1m2m7dtNms1E2axbnFy1AjYzsduKuRyoeDwkXipi55R2c0dGU3DiXi3Nm90vM+9Mw3WAwMH369D4/N1h0j7wX8vLyOHjwYLjNCIrZbA6sdbevzc2cOZMZM2awcePGQPu37iiKwvg9n5Jx+DPOLbuJmuwsTM42Jhw4QPqxPKDnzdEaG8tH3/22rzhSayuq1RrYADUYDDz00EPY7XaklLgOHsJ9+DDl4zM52txMS3MzprY2xu8/wKRP9/Zv+cZfOwNF8bWAG2Bhp+HAtRDycDISv19LbCyXpk6hKmcSTSkpaBYLij8UNulMITe8vSUwm/GYzRz+8sPUZ6T3cVZ47LHHrrbpAXSPfBDMmDFj2Ap5sJK4J06coKamJqSIg2/JpX5cGmWzZuGxWsBgwBUdzak7VtOUlEj2vv0Y21wYvV40RUEzGDh+91qfZ2Iw4O6Wdiy8Xi5+6QGiLl1Ca2mFlhYaU1LY9/WvBkqVemw2LixdgsduJ764mKqcHExOJ+nH8oJ7kf5ZhMdopHzmDKxNzSSdPRtyM3Qg9CcNfSgIt8i1F6rq3KdzKMU33N9vMEQ2NJC9/wDZ+33lIVQhcMTHYW1pxdR95i0EQvUtD7YLdUVFBVu3bg18ZOnSpeTm5l4b4/tg1Ai5w+Hgvffe6xF3HR0dzcKFCxk3blyXNWaAkTBr6IzX66W8vGddje7UZ2Qg2qvM+VHNZkoWLKB81izSjx4jvriY1vh4iufPw9lbQ2iXC9vxE13SlM/efFOPGF/VbKZ4/jyqJmWTdKEIaVA48sAXyf7kU8adPBn01IqqUrDyNoSmYXR7WPSir8nClTASBWgwNI4dS0NqKmMuXkQKgTvCjq2+nsi6+lEzBn1hkJKoy6HyOCQN6ek8+uijgVdSU1Ovqfc9EEaFkAfbNW6nqanpqtRYCRf9WioTomfGoP91r81G0eJFFC1eFHhZ8XoZm59P4tlzuKKiKJ07x5eN6PEw7b1tPTzl5qSkLjG+toYGoqpriLx0iZw9HTG1U3bupvCmpSE9RdXfLKH9/0e+dD/L/jCIqJURypV40HWZGZxecWtHj1KPh+TTp0k/8jkJJaW6mPdB1O2r+OtvfSvcZvSbUSHk15NQ94XRaOyzTnpIgmzsGNxuFj//Ava6el8tEUUh88jnnL3rLpKO5xEXJHswsfAsJfPnIaRk1qbNJJ8pRFMUjEHC4nL2fIorIgJrt+Ug1WikZO6cjhcUhdb4eJxRUQOunz3SaLPbqZ6cA0DK6dMoXrXXsrLdkYCtoRGDV8VrNBJVVcXCl15GqCqKv5+nLuShif39b4kIUl9/OHPdC/lIjAu/EgYt4iHIOPwZ9tqO4kiKpoGmkfXuVvJXrkS2uYjv1jxi2gc7qJ0wnpTTZ0g+U4jB6yVUl0vF68Xsr54n/ZXlFFXFGR3F5ayJ3epC0+9+mSMVCZjcbtKPHgt0/ylYdRuWpmay9+7rlwAL/A8A1UtVdjbZ+/Z3ifXW6URaGmmdSiqPVIZHjutVpKkpRENfneD4a6CAr4HEuOPHu5QQ7fgctCQncegrX+b0ilt7nGPCgYNkHv4sZIPjdgQdf4RSCITXC5pGRF0989a/xsy33wmEDFpaW7D1M45cdvu3P58dLhj8UToGrxeDqjJl5+4Bn0MAyWfPMe2DHdgaG3URD4Lt0XXXhYjDKPDI09LSwm3CyMDv+U7fspXG1LG0xcSQeO4ckTXBZzRCSlSLBc1spmjBfNLyjgciUAQQW1nZpaFwXwjo0VnG6F+br5iWS31GBrM3buqfIBmNuKKjaUiIx+R09qjf0Vm42+t8tD9Mwi14QZOuBHjsdjSDoc86JN0x+JdSdPxMnEjy5rcwDmWN8mHAdS/kQ1m8/Xolx2DgnD+NPv34cTL7KOmrAW3R0bT4u+xIIajOmRQQck0IWhIScMTEklRY2GPaFypTNJiIGbxeUvPzmfHuu9gbg8yuhACTCdvatRhnz+J8fBzFpaXUC4HXX8977ImTTN79EbbGRtx2G5fT0ym9cS7NKSl47XbmvLGB0rmzydn9MdFVVb5NX7MZY1vbFWWlDh0CR2wsrogI7IOYYYb74RR2Jk4kZdu7GLonBV1HXPdCDrBu3Tqef/75cJsxLDEYDJz1eHxNL3rx9iS+EEIJeC0WDj/0YEe7Mn/7q3Y0o4HzSxbjtViILy1FejwBz7C9IcNAsFosGB58EPXFlzB09/KlBLeb1rffZlfGOFxuF3SrVV05YzqVM6Z3fB66bOxeWLyQxrFj2feNbCzNzSheL66ICMYfOszUnbsGaG1oBrvJKKTEETcGywAzZEcUioKIicG84lY8pwrQeql1H/fyS9hu7bqcp9bU4Dl5EsPYsZimTAFAut1gMCCu830VGCVCbjQaWbduHRs2bLhqXYBGKqqq+jIo/ZmaDWNTGFNe0UVwvIrC4Yf/CqvDgSsigtrx47uEF0pFIanQF5nijIkmf9WqQE3sjx//JpmHDhNbdpGYS5c6enzSf2GLuecednk9JN59F7nvf4C1ubnHcZqUjD1VQPGC+b2fLEhkTkOnJsquqKjA62WzZw2ZkA9UxNs3f6XRSOWyZSx66eVhMjsYemKe+U/sq1YGascD1H7nu7S91bNpg+2BL/UQcQBDYiKGbk0axBB13xkJjMoU/ebmZl577bWwXX84Y21o4KZnn0PxeDF4vRSsWknR/HldGtwGQ2gaisfj69wiZY961QAJ5y8w6eNPiK6rwy0EtpaWXsVNAsJq5eMn/5EWvzeatedTcnZ/hCIlHquVz++/j9oJ45FC+Mq9RkRc2QB0wuBysfoX/x74WVUUX/sy/z0zUGEOVXRsVC59GI1YVt5GaHHj/wAADB9JREFU7M+fxJiS0uNtqWk0/du/0/LcH31t/CwWIr/xKNE/+QlimNQhDwd69cNe8Hg8vPDCC+E2Y9hgdLkYe/wEl6ZO8QnjUJcElRJUlQWvvEpcSUnQ0Kn28q37vvttmmNiAolOUVVVLP7j8yiqys4ffh+33d4pPHFgfUf7sjG6ooIZ720jpqISr9lMyY1zOXfTUsblHWf8oUNE1NUDPiF2WywY3e4BNVTQhKAmayLJITr5DAiLJbDMNKwxGDDffDMJz/8RcTXbtV2n6EI+QIqKitixY0e4zQgfQymKIVA8HnJ2f0T60aOY2lyBZgZSUfBYrexd9zXGlFeQVHiWtugoSufMxpGQQO772zE62zh+z9qrY6OUmBwObnh7C0e/eL9vltH5Ou1jo2mkHz1Kzo6dnL95GTm7P8LUzx6ZGuC1Wtnz2F+z4le/GRKzLatW4vrgGv3NBqskaTZjXrEC94cfdjTetlpAgm3VKszzbsQ8dw7mWbOujY3XIbqQDyEtLS1s3bpVj1EfKqREqCrJhWeJvHyZ5sRELk+cwMI/v4ylqZnTK2+jMncqUlEwulx4rVbMLS3Be4oOEJPJxPLlyzl79ixVVVWYhCB55y7S9+7D0tLC8bV3UjF9ui85x995yWey774xGo2MP36CyoREbnruuT7j5tvvNrVTKOGQPIpsNlDVa+KRG8aPR6uqQjo7dSCyWLDetoL4Z/8A+JZG1JIS1OpqTFOnonQrtqYzOHQhv8qoqsrHH3/M+fPnkVISExNDY2NjuM0aMCaTCVVVuxTBCosdra0knz5D09gUmpOSutaGGcLZwsMPP0xkt7A06XRS9cYbnP90LzVxY6jJzsZmt5OWlsbUqVNpa2vjzJkzgK+VV/HTT3N+0iSmfLiTcXnHA6VQg61/q/4SvaF6Yg4KiwVhtSCbW65+X1IhiHvuDwiTmYaf/hS1ugYUBft99xL75D8hbLare/1Rji7kYcTtdvPqq68GLT073EhLS6OysjLsQg74PMxgm6xd0vZ7F/XU1FQqKipCvt9XNTuHw4HBYAiajyClpPFff0HlmxvZ++jX0YxG0k6cIPPgYUxOJ7bGBhSt6/3VFhmJtY9N3oEg4uKI/MojODZtRh1oV3aTCbzeATXbsKxaSfyf/ohQFKSUaPUNKBF2hJ6vcU3Q65GHkc49/YZzswqA8vLy4TObMBiCi0woYfdvohpUFaPbzcS8POY89RTv79lDbW3PcqX9yfpt7xYFoF6+jLesjLZdu3Ht+BDvpUpkzWWigZSC01yaMpnymTMpnzkToaqYHQ4mbX2PtHPnUKSkOTaWqCFMl7c++ABx//GUr9F2fj5qWVm/RVnYrNjW3o0xawLN//lfyLa23o+NjSX2509iv+fuwPKSEAJDXC8ljnWuGbpHHmaKiorYtWtXl8724cZsNg/v2YNfuA1uN5mHDlNy441IJEuf+xMtiYmY29qIKy1FmEyYpk0jZuMGNm7cGAhhBIiNjeX+++/H0I9kEc+FC1y+/0to1dWhTRKC4nk3UjxvHqrZTMzFi+R+sIOI7g9EixlcVzi2UZEkrH8Fy9y5gZfcnx+l5oEHwNmpQYLVivXW5biPHkU2NftmN5pG7NP/gf3ONV2Obfnzn9Hq6rGt+QIiIgLn1q0Yc3OJ+tY3UYKVPNYJC1dlaUUI8UvgLsANnAe+LqXss6qRLuQDQ9M08vPzOXDgQJd640uWLGHy5MlomsaBAwc4ffp0GK28inRePtE0EAJLs68aYLZXxfGrZ2j8/X8Tv+FNlG4PIGG3k/DG65jnzKa2tpbq6mrGjh0b6G/a56Xb2qiYNgOGqnerydQR0dEfLBYsq1ehFhVjmb+AqB/9AEOnpKXOtH30EQ1/9/eoJSUIq5WIr32V6J/8GBQFT14esq0N8+zZCKt1aL6LzjXnagn5KmCXlNIrhPg3ACnlT/o6Thfyq0dJSQnbt28f9PHR0dHDKxpHSszNzXjsdgSCpPPnmfrBDiJcLoTNRuLbmzCOH0/dd5/AufGtHocLu52Ynz9JxIMPDOryra+/QcMPfzSwg4xGYn7xrzT+6P8Efz8xEWpr+96YNJuJevxbRP/ohwO6vGxrA7N5VCfOXK+EEvIr+k1LKT+QUrbHWx0Axl3J+XSunMzMTB577DESExN7/ZzRaGTGjBmB9U6AuLg47r333iGzZUgqTwqBOzqaW1au5NG/+d+sWnsXKY99g9h//wUpB/djHD8eANO0aSE9TdOkSYO+vGcQs5yIrzyCddkyX5JOEAx2O6mlxSQfP0bykcOkniskYfNbCLs9cIyw2zFOnEDkN/9mwNcXVqsu4qOMIVsjF0K8A7whpXwlxPuPAY8BZGRkzC0pKRmS6+r0n4MHD9LY2MiCBQuI8de1kFLS3NyMxWIJRGZs2LCB+vr6QV9nwoQJrFy5EoC9e/eSn58/oONnz55NSkoKTqcTq9VKWlpan2vZWkMDVUuXoTU2dni6ZjOm6dNI3PJ2lwfWQHBs20b9X/ezT6MQ2O65mzG//hVCCKpuvgVv96xNi4Wobz9O9A++3+NwtaqK1r9sQC0vx7JoIbY77hhV9UJ0+mbQSytCiA+BnsUQ4GdSyrf9n/kZcCNwn+zHk0FfWhn+PPvss72+b7FYyMzM5OLFi9jtdpYvX86YEE2aCwoK2LNnT7+uu2bNmkF78t7iYhp+9v9w7dkDJhP2++4l5v//PcoVlC+VXi+V8xYgg210WiwYs7IY86tnEFJiyM5G6SS8nvxT1HzxS0iPB5xOREQExuwsEja+iaLHW+sMgqsWRy6E+CrwN8AKKaWjP8foQj4yqK2tZcuWLXi9XlJTU8nMzGTatGkD9m7dbjfr16/HE2KTTwjBokWLyMnJwTwMPVCtsZG6x7+N65M9oGko48YR9cR3sMybjyk7q/djm5pwbnkHb0UFljlzsCy/ZVSUVdW5Olytzc7VwNPAzVLKmv4epwv56KOuro7t27d3KSMcERHBqlWr+lzP19HR8XG1hPwcYAHasy0OSCn73J3RhXz00traCvhEXEdHZ2BclcxOKWX2lRyvM/rQBVxHZ+jRY5R0dHR0Rji6kOvo6OiMcHQh19HR0Rnh6EKuo6OjM8LRhVxHR0dnhBOWMrZCiBpguOfoJwCXw23EMEEfiw70sfChj0MH13IsMqWUPRIvwiLkIwEhxGfB4jVHI/pYdKCPhQ99HDoYDmOhL63o6OjojHB0IdfR0dEZ4ehCHprey/+NLvSx6EAfCx/6OHQQ9rHQ18h1dHR0Rji6R66jo6MzwtGFXEdHR2eEowt5PxBC/EgIIYUQCeG2JVwIIX4phDgthDguhNgkhOhfG/rrBCHEaiHEGSHEOSHE34bbnnAhhEgXQuwWQhQIIfKFEE+E26ZwIoQwCCGOCiG2htMOXcj7QAiRDqwESsNtS5jZAUyXUs4ECoH/G2Z7rhlCCAPwW+AOIBd4SAiRG16rwoYX+KGUciqwEHh8FI8FwBNAQbiN0IW8b/4T+DEwqneFpZQfSCm9/h8PAOPCac81Zj5wTkp5QUrpBl4H7g6zTWFBSlkppfzc//9mfCI2uCarIxwhxDhgDfDHcNuiC3kvCCHWAuVSyrxw2zLMWAdsC7cR15A0oKzTzxcZpeLVGSHEeGA2cDC8loSNZ/A5eVq4DbmiDkHXA0KID4GUIG/9DPgpsOraWhQ+ehsLKeXb/s/8DN/0ev21tC3MBOs2PapnaEKISGAj8D0pZVO47bnWCCHuBKqllEeEELeE255RL+RSytuCvS6EmAFMAPL8XePHAZ8LIeZLKS9dQxOvGaHGoh0hxFeBO4EVcnQlIFwE0jv9PA6oCJMtYUcIYcIn4uullG+F254wsQRYK4T4AmAFooUQr0gpHwmHMXpCUD8RQhQDN0opR2XFNyHEauBp4GYpZU247bmWCCGM+DZ4VwDlwGHgYSllflgNCwPC59W8BNRJKb8XbnuGA36P/EdSyjvDZYO+Rq7TX34DRAE7hBDHhBD/HW6DrhX+Td5vA9vxbe79ZTSKuJ8lwFeAW/1/B8f8XqlOGNE9ch0dHZ0Rju6R6+jo6IxwdCHX0dHRGeHoQq6jo6MzwtGFXEdHR2eEowu5jo6OzghHF3IdHR2dEY4u5Do6OjojnP8BhHwQiTbduqwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOydd5gUVdaH31vVeXImR1ERUUTErKhgjqufYc05h13jqrtr2GAOa1yzrLq6iooRBSOYEARBJec0TE6du+p+f9zJ0z3TPd2ToN7nmYeZrlu3Tjczp26de87vCCklFhYWFhZ9F62nDbCwsLCwSA7LkVtYWFj0cSxHbmFhYdHHsRy5hYWFRR/HcuQWFhYWfRxbT1w0Pz9fDhs2rCcubWFhYdFnmT9/fpmUsqD16z3iyIcNG8a8efN64tIWFhYWfRYhxLpor1uhFQsLC4s+juXILSwsLPo4liO3sLCw6ONYjtzCwsKij2M5cgsLC4s+To9krVhYbFMseQfevQCCVYCAgXvBWTPAndPTlllsJ1grcguLZFg3B944ud6JA0jYNBce2wksZVGLbsJy5BbbJ0YkNY72o6uAKPP4SmH5R8nPb2ERB5Yjt9i+WPEx/GsU3O2Ae3Lgy7vANDs/X+Xq2Mc2fNv5eS0sEsBy5BbbD+tmw+snQsVKQEKwGr66Cz69vvNzZg+NfWzgXp2f18IiAVKy2SmE+ANwEeoZczFwvpQykIq5LWJQsxl+mwbpRbDTcWB397RFvZ+PrwUj1PI1acAPj8GhfweHp+M5Ht4Bqlc1/ewZEH2cKwd2Or7ztlpYJEDSjlwIMRC4BthFSukXQvwPOB14Kdm5LVrhLYWZN8PCqYDR7ICAs2fCyMN6yrK+Qelv0V+Xhjo2cEL759+h0SYe7tvcdlzmELhkLmjWA69F95Cq3zQb4BZC2AAPEOW32yIplr4P9xfCwhdp6cQBJPxnMoS3wYegkBeWfQArP4VIqOPx7SHa+3UXcUwQ5+ZozXp4oB9881B84y0skiTpFbmUcpMQ4gFgPeAHPpVSftp6nBDiEuASgCFDhiR72e0L04TX43hM/+0t2P2spnOqN0N6IdgdXWtfV/HFHfDVndGPXbACCvISy9Uefhis+KDt65odCndp/9yPb4//Og3MvF7F0MecnPi5FhYJkPSKXAiRA5wADAcGAGlCiLNaj5NSPiOlnCClnFBQ0EZO16I9Fr4U37iGLInHdoG7dHh0MPzdCXcI9dWX8prnPhnbiQO8MAruK4Qnd4PN8+Ob86hHwd4qDq7Z4bB/drzHUPpLfNdozbvndu48C4sESEVoZTKwRkpZKqUMA28D+6VgXgtQzve9S+IbW7EGXjsJypdEP36nBvNfTp1tXUXICx9d2fE4GYGSxfDSIVC7pePxuSPgsp9h7O8hYyAMmACnvA77x5G1cup/Ox4TjbC3c+dZWCRAKrJW1gP7CCE8qNDKYYDVNSJVPDSCtjHxGKye0fGY98+DuY/CCc/CgD2Tsazr+PSmxMYbIZj/LEz6S8dj83aAk19N3CaXlRVk0XtJekUupfwBeAv4CZV6qAHPJDvvdo8Rhjt0qF2b+rm3LoBnJsAdrtTPnQrWzEpsvBGEsqVdY0tzJt6Y+DkeK4xo0fWkJGtFSvlXKeXOUspdpZRnSymDqZh3u+bxXYAkKg7jIqhi50/vA2XLuvhaCZARIzc7FnYPDO6GaN6OxyV+Tr/dU2+HhUUrrETX3kigBipXdt/1in+AJ8dBaTesauPhkLviH6vZVPHNuG7YVHzloMTPyRicejssLFphOfLeyGOju/+aZgC++HP3X7eNHQZEAmBLi2/87ufAJfPAmdG1dllY9GIsPfLehhEBbw/VUy35sGeu20DVenjxIPBXQCSObI/s4XDC811vVzIMnNjTFlhsB1gr8t7GnPt67trSD1/eDZt6KOnordOgZiOEauMb78zqWntac8aXiZ8zaO+Um2Fh0RrLkfc2vrinZ6//5V/g2b3gviIV5uguaothywKlexIvrm525P+dlPg5YX/KzbCwaI3lyHsdca5Guxpfiaqc7C4igQ60UKKw8fveX63ab2xPW2CxHWA58t7EHfEIN3UjgQpY+033XCt7KKQleOMwgr3bkQvd2oS16BYsR95beHLf5M53F0DmoNTY0pyXOpFyB0qJMZHQjBCq4tKeBroz/vO6UyrWnpfY+AFWYwmL7sFy5L2BUABKvo9v7CXzYOgkmv7rBIw7H24ugT9ugOvWwfEvwJmfQt6YFBhnwkd/aPmStxw2zodNP0FNcctj67+BJ8fCP9LV13uXQMgX36WG7A9XL4MDb1VphfEQ7MZQ1J9K4h9rc8Ohd3edLRYWzbDSD3sD/4hTx+OA2+CFgyDiB6SKKdvcMLGZwFT2EBh/vvp+VL1iX+1WeLBf5+2b+wgc+aASpnpkmBKrioeIAYv+A3Vb4Pfvx3dO5sAmzZSfp3Y8vmI19O+m6slEVv8Tr4aRkxObPxJU0gzO9MTOs9jusVbkPc3CV+Ifu+RNiPhobHAgTaWuN+OP7Z+XUQRpAzttIgAPD1Ff8TrxBiIBWP4BPDwCfn4FVsxQvTPXf6ucVrLMuC75ORLhprr4xs19DGo2dTxu80/w3L5KV+dvLrgnC57avedSQC36JJYj72k+viq+cdeuh/IV0Y9t/rHj82/cCFoCsefW1G4iKe2X6jXwztnw6lGq6OeFA+HePFiVoEBWa9bPVk00ugtPnBWnRgh+/V/7Y8qWqyesjd/T+NlKE7YugpcPgeqNSZlqsf1gOfKepHqj6uTeEUc9DU/tSsxWY+7c+K73lwDscx1ovaFjkKkKf/5zBNTEoSUeC2mqr+7EFYeolzSgNIYufAOz/1kfJouCEYJ5Tydum8V2ieXIe4qNc+GxHTse5+kHs++AUE3043YP7BdHY4QGjnwY/hJU2SG9AhNm3dz5052ZoHfzVs/uZ8Y3rvjn9o9vmU/MpxwjFLtZtIVFK6zNzp4g5IOXD429GmvAXQBnvANTp0Q/LjSYcDnsfW3iNtxYAf9IItSSShb9B6bcBxmd2JBN7596exowDVjzOVStVTK/K2eAIwM2xZlhtHmuqlbtv0f044VjoeQXoj5p2dwwaB/1feVamHoUVDZTp8wcrjopBUoAodI30VQK6vBDVP76gL1gl1PA3kt15y1ShuXIe4IZ17bfAszmhlP+CzufoLRXwjE22AbvB0c80DkbHA4YMgnWf9m581PNZ7fCiS+0fE3YOt5cHXFY19hTswme3bt+byAJpk6Bm0rrHW0rDrwFlr0L4SjpmY50GH8RPD4GyqKszGvWNPtB1hdGmVC9Fha+qF62p8EXf4GLf4A0q8HFtowVWukJFrfT/zG9P9zuU07cVxG7AbHNDXtcmJwdF3yR3PmpZMk7bV9z57R/jtDg4C6Q3jUNeGJM8k4cwF/e1BS7NUVj4cyPoaAh31+A7oAxp8Kl82Hhy9GdeLyEvUqEbNYtnZ/Dok9grch7gkgg9rGJzbJYlr4be1zGANgtzlhtexxwK8z5R/LzJEu0J5RgB1K2h/4d0otSb8vrJ8e3CR0vrxwNN2wGR5R9iWEHwZW/KPliTW+5cp+ZYO/SaJhhWPgS/DZNhXgm39NWkXHBVJh+Po3x+t3Ogd/1gSbdFo1YK/KeYNjBQJRHbU8B7F/fF7JuK8z5Z/THboDRJ4NuT96WyX+HEYd3/vwLf4A7pPq6cC7kjOrcPGa4be600c4ND1Qnpd+mwbxnUteqLlgLy6enZq4GQjXw6Q3tj9FtbcMviebsx0Ka6sa09ku1N7OpPl31pxfgH5kw/VxabLoumgqP7pCaa1t0C0L2gOjQhAkT5Lx522HBQ7BWOWZ/hSoCMYJqda7ZweZSscyC0Sre+fQ4KPk1uqyrPQ3O/hSGpLBP5c//hXd+37lz74jyO1SzRRW2+Evjn8fmVmGlxnnjEBGzeeq/kaqs/9inosej42XdbHhpUupTGl05cEtFYud0lYjaiMmQtyP8+GT74/4UAGcv2RC3AEAIMV9KOaH161ZopTvwlcO758GqTwGhMgtOfAmKF8KWn6D/eJhwWVPWxpafVPFPNCeu2VQMdXCSIlut2f0MSC+AV48FM8He2Xe5VQn/Xpc3OdHM/kr/pYHKtfDYzu3PHfHD2jkw7ID6FwQxc+cbz2nm+Be9AiOnwC4nJ2Z/c9IKVew91Y5cdsIpCwfIUGrtAJVJs/qzjsd9dScc3gvCbhYdYoVWuhopVdHLyk9UbrARhMpV8PZZagX5+/fgkDuanLi3TB2LlZpYNE61N0tm1RmLkZPhz3644hc4IYEYqRlQoYPPbo09JnsoZMWhzjjt9GY/JPi0GPbCvH8ndk5r8neCwt2SmyManVFpPOX11NsB9Q054vhsC1MhumbRHViOvKtZ+zWULFYx4OYYIZj7eNvxb/4fVKyMPpfNDbue2jVOvAEh1B/wHudA9oj4z4v44ftHVNw61rynvA72DgShajep/OjO0t5Gcryc9RFR9zCSwV+myvFjqTX6q+Cjq+GlQ2HmLUoGeMxJqiAsldg9StArHsalYDPdoluwHHlXsnUxvHa0ctqtMcNQ3mqDrmaT0t0wo21yCfDkwZ6XdImpUbl6acdjmhMJwML/xD4+cAL8YS30bxPia0ntFvjqb4ldu4H0FDi+9CLIiaPqNlHWz4aprRQRV3wMT4yFe3Ng7uPINV8gv74X844MjOVz4aYtMCkFcriaTW2mH/MU7Htdx/1OJ9+f/DUtug3LkXcl086MnXVic8PQg1u+5itXG5/RcGXDpT91b59K3Q62BCVVZ1wF9/WDVTOjH/fkdZz7veJjVcjSGZa9r2LAyWKPU1o4UTbNhYpV6vuXD4NXj4bSXxoPC6EaCwktQuCOQ/G9/wFMur0pM2hMJ1bJugvOnwM3FMO4ep33i75XRUfNETr020Ntch7QQZaNRa/CcuRdRV0JlC+PfdyZCXte3PK1/J2jh000O4w7r2eq865cnPg5vq3wn8PhvUujH4+l4tjAjD+QcHy8ATMEC17s3LnNGX188nPEYvHrMPdJVf4fA6GDe5iPyuv+gFndLKf9/16BcxIs5Bq4F3i3qiyVjXPVvk3BznBrrSpIOuJhuHYN/DUCl/1kZar0QSxH3lVoOjGdkT1Nra5bVy7aHHDUYyqO2RCj1Z1K3fCAJISlkiFnGIhO/mH/9Azc5YCtv7R8vaijhsQJtIhrjTQhFKdmeHsMj6FvkwqWTINZf4prqNB1Ap+1cvgjJsEf1se/h7F+Drx5Ksy8UeWRv3IkROrDfaOOVKGWnGHxWm/RC7EceVfhyVNpha07w9vcsP9NkBlDCnXcOXD2TBh9khI92u9GuGJx11Qwxsuf42zVFg0zDE+NVZt3DYycoj6HrsCeroSikiUVxVaxKF4QW82yHmmAb3V9jnw0vfXMQUpoLK5NWdlUsxD2qlz57x5M2GyL3ovlyLuSk1+DtCKlmKc71Ep84F7KkbfHkP3gtGlwyVw47O6eFzzSNJjYCYXF5jyxm2qkACp8dPYnydvVGt2lbhI7HJn8XDnDk5+jPRzR9zqkVPe+SI2NmrlZyEgE56GHthxUtQ4eHgpvnkKnQlARP/z0XOLnWfRaUlIQJITIBp4DGrofXCCl/C4Vc/dpcobDdWtVq7PqdWqFPWT/rk0f7CqOfkRVmq7tZEefqhXw+E7qZnbZQhh6IAw7BNamQLhLaCoL46SpMOrozuVst8aVnfwc7ZE5GCp8LdJSpYRwlU7tvBwCm9NB6GTecjP+999HCIHriMPRc7PgmQngK0vu+qlos2fRa0hVZeejwAwp5SlCCAfg6eiE7QabA3b5XU9bkRrOm6l6SybT8i3shcdGwdFPwMC9k3fkulNtBE+5N7UZPbYu7qJUsVRtMn5xBwQqQHcg9r4WBpyDfdhMHC4XMhSi5p57m278d9xJ9mXHkBYrFz0Rdjwu+Tkseg1JO3IhRCZwEHAegJQyBHRBXbFFr+C4Z+D9i5Kf56Mrk58D1ObmQbd3b1pmKpAmFOwCN5eq2LXNBUJgj0QgEsYoK6Pikssg2FLSoOrJ6ThPicSRFaqp8Hlaf6iLIsf725tw7BMqVXPmTSotMq0IDrpNVRz3xafG7ZhUxMhHAKXAi0KIBUKI54QQbfQ6hRCXCCHmCSHmlZYmIKRk0bsYf0FPW9ASacC3fbB4RbOrlEAhVM66EAS+ns2WceMpO/V0Ki66pI0TB9S49R3ctEZMgZNfgdv8scNMvlJY9Jpqgr16FgRroGIFfHgFzLkn+fdn0a2kwpHbgPHAU1LKPQAv0EbJXkr5jJRygpRyQkGB1a2kzyIEZAzpaSuakKYqAuqLDG5SrzSKi6m44EJkZSXS64VwjBi2FMgYG6WNONJg4EQVHmovlv7N/W0L1sI++PrvLbOMLHo9qXDkG4GNUsof6n9+C+XYLbZVLorR8SYOJPBzeh7/GrQrfxs6nicG7MwiTzYROl0CpDaSN85t8dKqqpVMW/4W7696j1JvDzwBDj2k/ePjL1ZCYvV4p72NNOLIn9c13Le9DTu1U7C07D34955QtR7y2tGH95UQ9VMXAmo2dGyLRa8h6Ri5lLJYCLFBCLGTlHIZcBhgtf/elskaCJozYbnbCHDXiAkszMinef7zJwXDQZocUbqOY8rX47U5+dvw8ejAmNoyTi1eyYiQN3bGtDTh67vh9+8TiAT4yze3s6xyKbLeST27WCki7t//AC7Z7TJy3DmU+kpYVb2KIk8Rw7MSEAeLCwETr4R1MTZy04rgkLtavGSWlECog60lu53Ma6/FNnocjJ6unkTevwTqiluOk6ZaWX9zLxzzNLwQRbd+wASVFlu7ue0xM5IazRqLbiMljSWEEONQ6YcOYDVwvpSyMtb47baxxLZEXQU8kBfXUAkYCG4atQ8rPdmxN9Li+F0c5K2mMBzAbRrkhgN8lTuAGpsTmzTJcvejOlxFJI7OOgKBU3cRNsN4bG6OG3kCx4w4lgxHRtOgZBo7ODJUUZE/VjMJAVlDYNKdMO4cAp99TsUVV6qwSiw8Hgb8uhjhaJZRs+F7JYcQipLJUjgWrlikdG/ePU+JkTXo2Z80FdZ9Da8e01LT3e6B3c6G457uzLu26GK6tLGElHIh0IGkncU2RXpu3EMFMCenP2vcWe1nQ7RpdSabXq//fmNaFhtpGyOOCJ3yUPy51RJJwFCa77XhWl5b+gqvL3uNiUV7c/rOZzAie2RyjR1CtcqZZw+DqrVRLaB6Hbx3EWz6HudRj2PfdVfCixYh/dG16IUQRFauwr7L6KYXswa3lUhWoyG3vl3byClwfZTMleGT4HdT4eNr1eanZlPqmlPuS+y9WvQ4Vocgi85jz4Rw+6XmDXyVPQAj0UKd5o49kXS4eqefH/IhhUaVzRnXtU1p8n3xd3xf/B2jsnfkroJdSSv5KTGbmxOqhd3Ph3lPRO/2BKov54IXEAfcQv7rr+F9/XVq/v5PZF2DXozEOSiIZ5QX9Eq0mnkgm8TVTMMDBfsgSr5DGM1CXXZ3fPo8u5wMo38HgSqlhtiV0gQWXYZVom/ReSZeFvdQZyxH1hUIAUJQ5vBQ7nArJ55gCHFF1XKuzM/q/AZsAz/+K7YTb87GHxAOB+nnnEPOY48i3EqLJmv/KnInl+PZwY9nuBf9s8uRb55F4KvZlF94EVvGT2Dro5vxr7AjsSFtbkjvD797BQbtHZ+NQiDtGfg/+4Lax5/A/8knyEiKGj9bdAvWityi8+x8EnwT32P4EeUbmJ9RQFDvxl+51it6KRNa2VfZXV1gVFtMQMtoElFzH344kT9ch+/Fe/Ds6EOzNbudhL3IRf+l5uMvCRerdZgZClH5WTZVc51k3XwNaedelZBMgVFRQenxJ2KWlCADAYTLhZafR8F709Hz81P1Ni26EGtFbtF5Bk5UErtxMK62jKPK1iGkmfDqOGUkWK1oCpH8irwDpIQSTeeVH+5VbQEBo7qa0LLluAYHEaKtBUKTuPq3leqVtUFq//16wloz1X/5K8aGDWqj1TCQXi/Gps1U3Xp7596URbdjOXKLzqNpcOpbcQ0VwAVblvHMkq/UC7GceU85+Rikwppob0lK9VUbdHDrqH141xZm85u/w5j1EMV77kVg2jTM2hDSbHvzkSaY4eg3JbMiVpZMbPwffQytQymRCIGPPmLrscdT8+i/Wja3sOh1WI7cIjmGTVK9IOOkKOTnxtX1qaetPVyDd+stSEmdbkvKmTfIizW0YZWm+t77m4cN0/rxcu1ulDncDNzgp+ZtN8XnPgT1WSuB1bE02wX+VdF16RzjU1iLJyWRBQuoffRflEw5ArOqKnVzW6QUy5FbJIcQalWut1ILbN1QoxkH1pYy7ecZnLtpCVmhQMs0Q2hy6J117K3PS3QeKcFU4xeldz5GLIGIENw1fE9qil0ENjrxLfdQNr2Q6m9zCYdsRHYNc/Ta9Vz3xCoy1kPzQikzqFMxKw8zJJq+woLKz3Ix/XrLi+k6wuMh8/bbErbTNWUy2NrZuwgGMcrKqHv+hbjnlJEItY8/wZY9J7B5p9GUX3gxkbVrE7bNIj5SUhCUKFZB0DZI+Ur47mEoWayaSu99NXx2Kyx4vt3TJPBm/jBeGdiUG10YqCPNiDA45GPfqmKeGDKWuuZpcXHEuu1GBB0IaHr8sXEpQcJBX5WzYYibNSM8nFK8grO3royrD080ggguH30wu9ZWsN9LdfRbHwYB4V3CDNm7hIgmcIQlujSp+ioH/+o2enOgS5wDAggBwU1OpNHsJikEWn4+zgMOIOO6a7DvsEPCNholJZQeezxmRUXMHHYA+9ixFM74qPFnGQ7jn/EJwR/nEf71V4y1axAuN86DDiSydCmhRYshUK/ZommIjAyKvvwcvbAwYRubY/r9+N95l+CcOehDhpB21pnYBg1Kas6+QpcWBFlYkLeDkkVtzvHPdujIBXBq2VpOLVurVvE7Hg/LPm4xZv9ftjI3s4Avs/uz0ZWBANa60pFRVv120yA/HGCLK52xhXty8MBJ1IZqWVu7hpAZwiXc2G12dKGR7ylgYr+9KUorYn3NejZUraHk9akMW7KVYf61XDpsJ/at2prUx+JAMspfzezsAQQPrmDxztnsmTaEqxa/gFOaOE2gfnGdfXAloa1ODG+rP0tDENwQJcyiaWTccD2Z116TlI3Bb+bgyN2Ee9dyZAR8y9IIbnTSuo2cVpCPWVWF7/0PCHz+OcFZn0VtQ+eLtvI2TWRNDcWHTsY9ZTIZV16JfYeR7dolpcQsLUWkp6N5VCjJrK6m5OhjVYaNzwcOB97nnidv6ss499u3sx9Bn8dy5BZdhxBw3HPx65dLE1Z+jIr4tXQQE2tKmVhT2hivjiBY7Urns9xBeHUHBRE/k8s3khPyYxMa/twRZB72HFpG/7guPTxrOMOzhsPNh8KI02DZEq7cECbLiF9PRtK2g6YAztm8jO92LuKU/25h4og6dr51Ag6bq43yoADcO/jwLvMgA3qzV2NgmsqZJYgMhcBuRwhBcO4PaB+eTfaefjS7+nRdgwN4f0ujZm6zLkl2O5HVa9iy626d38eQEior8b/5FoEPPiR/2ps4dttNvZW6Ouqeex7f9PcQbjeuvffGO306sqoKpMR99FFk33cvtY8/gbF5c5MuTSiEBCquuYZ+P85FbKc66lZoZVtn0zz4/HYoXgg5I2D3s1XLtlAdjDtXbVZ29S//vYXg72YFQs2mhKEuSrDjYNV6eHwnjJoQVXNycA7xkbaTP6mPyACu3vEA7ps+n7J5/Rhw2xQcm15WDZGbISUgQYYF4Tqdmh8zCbVYibc2QqIPH0HRZzMRTme7NkQ2bcL72uv43ngDc+tWhMdD2sUXwaoZpGd/hn+1B98yD0iBvSCI4dcIbvCA0YW/G243QtPQCgswa2uRZeWxxzqdOA/Yn8jKVRjr1rU5LNxuCmd9im3YsK6ztxcQK7RiOfJtmY0/wMuHttWcbs6QA+H8r7rWmRtheHI3KF+awEkNYZMk2srpTrjsZyjYKf5zVsxAvnoSW1/LxvDqZEyoIWNcbVIfjwTWONMYHvCy+aUBpJ10GNn5b6kmyM3HRalXClXo1P2ciaMoiFFnw7fSjem1N5tZYBs3jsLp7yBabVgGN22i4vQzMNesjb6KdjrJ3KeC0BqNwGYnRBo+84axIspPPYjTiT5oEMaqVVGP9fvuG/Siou63qxuJ5citrJVtmU9vbN+JA6yfDT/Fn43QKXQ7XL0Erl6RwEkmZA3teFh7GEH493j45oH4wwG/vol/lcDwa4DA9CXvvgQwIqhUDZ2DQ5A+VMnc2tOQiMakmmg3C80hEQ6Tmu+zqVucAaaGnhNqmlmTRH75hcCMT1qct2ngYMom7oO5ek3s9x4MEtosCbZw4g0Wixg/9SBS4poyuVG+oBFdxz5ml23eibeH5ci3VUwTNsf51PPlXxOb218F856Br+6GNV/G7yTzdoB9/xjfWM0BQ/ZXvSyTIeKDz/4EC15UP7dnq68CFr9GpNoGDYU4WmpdmBnQ8Zx+Ohx+P5z1MWLU0SC0qE5cSqhdkIlvWZrKVDEFpl/HqLHRuE42BRhhvO+803jepoGD284Vw57AijRkRF3c79L4aVwmXo8eY3QPEwrhOnwKrqOPBqcTkZaGSEtDHzSQ3Ke3b9lda7NzW+STG+C7B+MfH4xPwRBQnXimTlFCUGGf0q8evB+c+WF8ynmZg1Hruw6cf94omPxPWPZu/La1onHz0YzAB5er5hNV6yC9CCbdoSRbm3lQuWUxmGDPaZKFjVQn79SkBMOn4f01DcfwbOxjdlEHhh4I/grE+tkt/g+8yzzUzMskd3IFvhVpbePUhgY2EyICbCa62yD47XdIKfGuWg1AjUfw6xgXYYcGJgzYHGL4ujAarVfXAomkJkPnz3eNRjckd9y1LOn33FV4X5pK5lVXYnrriKxajWvSJDJv+xOafftWbbRW5NsSUsILByfmxAH67xn//P87GUI1EPYCUv274RuY/2x8c+x0PNja35gDAYFKeHIM9NtDNWDoxJqjxe3CDNXrgkvVUeeTP8KPTzWOlVJS8ecHIaFsoA8AACAASURBVBzENTSAsKudx1BZR7bGgYSKz3Op+zkT3zw/lVdciWxI2xtyAESaNj29yzxUf5uN6bNheLWoWiuAcu5CojlMVcYfDBJZtYrqE06kPEOwcA+PcuJCgC7YPMjBz2OjP90IBO6ASW5liIBbpyzP3uUaM50l8NFHlBx+BMEZn2CsWIH32WfZMnwkm8aMpfiwKRTvfyCbd96FrZMPx//Bhz1tbrdhOfJtiY+ugfVfJ37eiS/FN67kV/BHafwU9sGCOOPsuSPUalhvz0FK1YIsWAPr50CgGi7+FibdHd81mpsmNH7zZEc54FMhpfpQS2jePII/LiOw0Yk0oeC4EtAl0q9hxK6RaRcpIVytU/ZhPuFiFyCQIZPArM8IfDxDDfLkwSF3gt2DlFAzLxNZH6/2r41Voq9uMq7hPrL2qcL02kDT8L76GkZVFb+NqS/fb6X+WJepE3BGDxVFdEF+WYgDvy5n8MZA74iJRyOavK6UUFWFsXQpxtq1yNpaIkuWUHHNtdQ9G+cCo49jOfJtBSMMPz6Z+HnDD4OcODcVU5XZcsDNcOl81UGnQ2T9jeJFGDgBxpwW92Uk8GV2P6YVxSg88VdCRFUehhf+jIxEqPg8D/8qD7asCP1O30L6HjXosfxpBwQ3Oyl9p4hQccuVsPT58P7vf00vHHAz/P5DzCFHYvqanjwCqzw4B/tVGKXZuxI2ScGJpaTvWkfVnHr1yXAY77PPsaWfDamDIyTpVxymaGsYW6RpfV2aFz1UZI9IthY6OOndLTjCvXU9niDBIDX3P4gMJtZbti9ixci3FXzlJJyqpztgUALVcAW7qBVkdau+knYPjL8wsWsXjlFx8C1xdOAxwzDvKVj8GhjRW69FK8YxgV/T8yhxxPDE7tzGzVR9wACEw46sC1H1dS5Vc3IQdok9XwMRpR9mHNjSIxAjNCJaV6UOn4S+Tyl6+rUYdU3FQIE1aWjuCFJT79CeFyJtbC3e5W58v2Y2nR+JYAgoy9MZuCnEsHVKCkACO6wKsmRHJxV5Nlz+tr8jpoAF4zKxR+sY19cxDCIbNnRKuqAvYa3ItwU2fAczb0z8PN0BEy6Nf7wQcOo0cGaCPU2V1NvTYOhBMD7O6s3m7PtHdX68BKvrc6/bJsRFe1aICI0MI8KuYaNt9ovdA4fe3fiU4ZoyGeHxNGl5mwIZ1AhVJ2BfK/RMg+wD24aihMuB5/RTW7xm+v3UPvRnnIN9oLd0tmZIQ08zQJOEip1Ufprf0onXs7VQZ/dfgwxfF0aXahqbqf4dvTyIHpYUVDbN3ZgfLmHxmEwCDoHN2EZW4/VIw0AviF+ds69ircj7Ot89Ap/fBuEEA7lphXD6dMhKUGxo4AT4wwb47U2o26oyL4Yc0Lmwy9jfw9ZF8MO/VMw8EmhT7RgdCUIHmxvCbRssNGd59iBuPPZVWP8tzLwJKlZB5kAVp9/j/MZxwuGg4J23qbjiKsK//QaGoWKvdd76Fj6Jvz0hwD08QG1OCKPWhjQEQpe4RjtwHXFE4zj/zFlUXHIphIOgNWSpNDlUW3aYSK0OofYzaApLVEu5aA8BEthzga/FDU+g3poALnh5A7+MyWBzfyeDNgVa30v6LO5jj0XLatuse1vDquzsy/gq4MEBcTq/Ztg8cGsNaL0kX9hbCsU/qwbA755XnxHTAY50uGAOzH0cFk5V6ZDNemMGNBurB+xO3hkfUpSWWKFIxbXX4Z/+NtTHivufvxEtiSWPGQL/qjTMkMA5MIhj3AS46FsAfB9+ROUlHT0VdVxbWZYlyKuWMUdECz1t6zgOPJD8l1/sUL6gL2GpH26LzPt34k48c7BygL3FiQOkFcDIyWoF/NmtULlKCWi1h2nAr2+BEYF+u6tQSfZwqFgOuhPXnhezy5jTEm57ZlZX43/vvUYnDu1Kq8eF5oC00c1uTq6sxmtVXn5FHDO074J9bkHIqQNWw+QG8t9/D+f4PXrajG7DcuR9FSnhm3tjH3dmqvQ9mwv2uAjGnqHCKXm9eNNHCDhrBrxyhMr1NiONWSVtiPiRs//WdCrAhm9hhyPh9HfjcuChX34l+PnnCI8H97HHgK4TXrkKockWedSGX2BLS+GT6/pvkJEINQ89okI4SbK10E57T9bb22o87ZabtysnDpYj77tUrW2/IvPQvys9j74m65k7Aq5eDms+h1ePaXdom3dmhmH1LFg2HUaf1PiysXkLtc8/T3jhz9h3HUPahRfgffoZlQIYCoGuU33Hncr5a5pa5TdDhmNroXQGGapj6wEHYmwtScl8ERuU5NsZvr5t2sn25sQBvA89jHPnnXFPmdzTpnQb24cj/+VN+OwW8JaoSsEp98LgVml3pgn/3hO2LlQ/Cx0O+BMclngRSregO5RnibUSKxrb95x4A0JA9XolRZto6Cjih1m3KC2Y7GGEC4+i9JybVS5xOExo/ny8r7yqVsLhesfXsCo2DPWlmarZQ30nHpHi6u+IPxdj85YkVuMtY+Z55QZbC+0s28HBTitDbUb20d+CzhMKUfvYY5Yj36aYcT18/1DTz+tnw/P7wdizYNhBsMspEAnDg602xKQBs/8GdjccdGv32hwPmQNVTLgyiqSnzQ2D9+96G0wDVnyktM6zh8MuJ6vPKxUYIRLtYW+GofqHbDR7MZkTlqvEFvMZXIVZTS3UwuEmBx5zIg1sEuE2kCENw6ehe5IPgQBIBJWzHJ104vWfh93Es4MP3xJVUJVTZZBTZVBWYEczTEauiTQmaMYV3tfMepGwbcflm1uKe9qEbmXbzlrZOA+e2yv2cWGv31Rr549Kc8BfemllWMUqeGq3llK1QodzZsLwQ7r22oEaeOEAqFoDIS840tSG44XfqfBIslRvgMd2jB0jj4IZERS/1h8ZAVu6gTQFMiJwDfYr8SmZmKPynHkaORcdhvnWBYhgcUoecKSEzS8OAaOT+X02g37nllM7L5Pan10tXG9FjkZOlYmW8J908xO2DWfuPvEEcp94vKfNSDnbnx75+m/ad+Kggp/tOXFQuWO9ldyRcEs1HP0EjDkVDrkbbqnsvBM3TVUK/+IkeHkyfHEXPDIS7rTBP7Nh9j1NoZwv/gLly1WnIaT611em0gdTQdZgOPRv6umi1a9pg363lOrByQwLzIig8oscZFADQyNSbcOotWH6dXwr0zrnn3QHYW1Hyj9IbWxF2MIk+rTRiKlB2I68522WjsuiLE9rXH3nVXbGibewLJmTew9CkHn99T1tRbeSstCKEEIH5gGbpJTHpmreTmGE1WoxJfTye51ug4lXqK9kCPvhqd2holnzhzWfNX0frFa63pWrVFPlxa+1jV9LEzZ+D8E6cKYnZw/AftfDyMPho6th3RwabroNK2MzCNU/ZoGhEVjnwgw2T6ls5pTMOGRzW6NpOA88kOq77yZcHKHkrSIiVTb0jAj5x5Wge2SnV+i2rAjhEkenzhWapPyH4TguL6Iyz4EMOMipDLSUY+nczMlO0GtIu+RibCOG97QZ3UoqvdS1wJIUztd5Zv0pdXM1q/7bppn9z5ZOPBY/PQ+hTsoBdoaiseAvJ9qTk3DaCZQU4Fue1sqJRyMBRyUEWn4+7imTCc6egwxqRKrsOIpCFJzYsROPFq2UEvzrHERqdTwjvYnZ04Bm4tnZi1EmyBo8CmdGNuV5OgGXlmRrzW3HiQP4pk7F9957PW1Gt5ISRy6EGAQcAzyXivmSwleRuB53e5zQ82+pW/jp+TgHSrXqHvv7tlK0QoNB+6RmNd4cPfrqVdjtFD59ByKt9QZrFE+q62CL4wHUbsc+fg8Kpr+jNiSDQUBgywmTd1QZuivxlbg0IFRip2JWAaXTihDORJfPEjBJ28mLoyiE86CDCXz6KfsdchF2Twa/TMxl02AnhkeiuZUmCzYJugS7Dfu43XFOmUz69XF2Z+rjSH+AyptuQXa0qb0NkarQyiPATUA8uqRdw+I34NM/Kh1ri8RJxDtlDYJD7oK1X0Dlagj5wOFRAljxapsnwp6Xwid/aNV/VIAZQf/iUgqOs1P1pYvQVjvoar9Xtk56ccSRKZKVhXPffXDutx/CZsf7RpPUbMa4GoTefnim9UpcmsqGwAYXlV/lKiEuU1DzfU4877oF2YdU4CgwKP9sKEbtRwRmzQIJ+zmdhG64CGnbzOBVDyOMAGZQ4F/nVhu9O7iw3f1OY/em0A9zCX3zTStjt8EkxUCA8LLlOHYd09OWdAtJZ60IIY4FjpZSXiGEmATcEC1GLoS4BLgEYMiQIXuuW7cuqeu2YPpFsCDeFWUC2D1wWxy6H9sCX94NX/6l43Fp/eDGLep704CVM5rSD0f/DuxJ9tiMhmnAW2fAig+VA9JsSo9FqgYLDUjNDRfOxnQMofLqawl+/z1goqeFyTm4Av/6LLwL4ti41DSw28HjgUqlXlh4cjH23Ogl8A1/Qq3vhaWf5BHe4kSGWz/4Juo4JVqmgeu4s/FNewcCLTN5tPx8+t05HrHwRdo8jTgz4f/ehB0OB8CsqqL8ggsJ/TC3yRYhQfbyvaBOUPjtHOxDk2zg3cvoyqyV/YHjhRBrgdeBQ4UQr7QeJKV8Rko5QUo5oSCVspJly7vGiQMcte2lL8Vk/xuhcLf2xziz4MJvm37WdNjxGDjoNtjt913jxBuuc+r/4IJv4PAH4IBblPBXK6clZBAx/2n0ggLyX3+N/i9eS7/TtlB0ymYcBQGM6jhz001ThVQqmyRow+X2qPIvsZw4gHuoP4oTh8RXvwKzzoHvzWltnDiADAQwN68m5nsLNSlEatnZZN5yMzQKSYk4nLjElhNSoZo+hJ6f39MmdBtJh1aklH8C/gTQbEV+VrLzxs30BDYjnVkq+6I19iwIt3p97+tg/Hay0QnKCV++UBX4/DxVpf3tdz3oLtgwR/XNHHZIwiJUKaX/OPW19L3odkgTWbO50U1q394G7qb0UdNro6UTjX9lXLswE9ewAKJ1fl99CmS06k/XwCA1NrOxdVtSmKaSE4iBkbUveuWPbZUjjRAMm9Q0jddLaP5PCRckRSr7VnNjrbBA6ctvJ/T9ys7KNfGNc+fBSVPhzf9TzW6lUa9p7YLzZkHejsqJ2T2w47E967B6CiHUCnvHVhon+aN6xp5YDN63RacgGVFpiL6l6UhjEY43TyL7ztuxh71UfJmNf0Xz5hDKeWtpEVUotDy9Pj2xfSJVdso+LCBr30ocheGmFbgZfTUuJWgeA9cwP/5VnvrFctfEoWU4jBh/Ft65s9BqV+AcUI1mF+p3e/K94FHt4LxvvkXVTTfXv6FElBL7VvxcuN1k3Xknoq9KVHSCvl/Z+fLhsGZmx+Nu86ny8dIlMOceKF4E/fdQ/RLzd0qNLRbdxxd3wrf3Q9hL2Sd5BDe5aJ6DJzxu9PRKIiVO2q7CAbtJ3uHlVHySn+CKWVJ48lbsuRHMMPhXeNDcJs7BgUbNcilBhgTBEjveJekE17vqq0pT71iE241jv/0IzpmD0HXAACNC3lUTcZ5+MwzYE4Dgjz9SdtLJsbV5+jD6DjugFxUSWbIUfegQMm+4HtekST1tVpewbeqRh/3xOfE9LmzSACkYDSe93LV2WXQ9h/wVBk0k8tGDBDcva5NmLkPhJieum/VOvpkzDWtUfZ1D3hHlVH6Zg+HXm7U8bcfhahIzoPRX6hanU7c4A5DkHFaDe5iKRUeqdMo/zsfwNg9HpNiB2mw49t0HzwknUPXnv0Aw2OIK5U8vov/luzS+k4pr/5CwE99a6OCXMZnYIiZ7LKwhs7Z36p1r6enkPvoIev/+PW1Kj9G34wevnxzfuKO3gU1L04Bf3oD/nghvnQmrP+/8XMWLYNatMPMW2Dw/dTa2h5RQsxm8Zambc9RRRMbehnBHyXqNRACBc6ivmRNvjsCotWHLClN0RjFF/1dM9oEVMZslN2J34xUnUfy/AdQtzgKhIzKysd/+E/KmOspXnUXJB8Mwwm3DOVHp1w9cndgk1nVy7r+P0MKF9bnuLZFeL5U3qD6uMhzGbC9LzOFotvmpmH5sEa+eMYjBG/xMmF9NVZaNlcNTJIiWYsKLFlFy9LGYde23/duW6bsr8mAdrPq443EH39l12RTdhWnAoyOhutkf45K3YL8b4LC/JzbX13+Hr/+hyuulhLmPwd7XwuR/pNbm5ix5F6ad0UoAS0DOSDjxRRjaeTkF2w4joxd+2GwIGSL7gCq2rovtgKQhEAJsmQaaxw8aVH+XozJUIq1uAE4nrkMOIfffzxLZtJnQ3B/QcnNx7r8/or7YKP+V/xDZsAFj40Z8n32O76mnienEnU4K//cGdc88g+/V1+JfMdvtuI84HNvgwUh/QG2ERsH/4UcEf/8tWmEHWWJSYhs+nMjmzVBby9qhblYPT+PKJ9fgrO+UlOYzCNoFEQE22cvyzk0Ts6qKmocfIev227ar2HgDfXdF3l53nEZscEgcudGpIlADtcWpj0M+0sqJg9rs++Z+qEogH798JXz9N4j46vtbmqrI5vtHYOvilJrcSOlSeOOkKCqGEipXwosHwtqvOz29bcgQFQ9ttaoVLheO4RpClwhbjP8PTaKlqZiMGRYYXh3PKD/9z95MwfGlZB9Si16QqXLKnU48J55I7uP/UtcdOADPSSfhOvjgRifeaNPgweiDB9c78XYIhah56GF8r7+R2O9MOEyksgrTMFRno1gr+nCYstNOp+y0Mzqcz9iwAc/ZKtls7oRsjvuwuNGJN+AMS0xdgMuof3LpRfH2UAjvs89RMulQjC1betqabqdvOvIVM5QT64hzPu16WwB85aqbzf0F8MgwtXpe+1Vq5l7wMtTEcNZmGFZ/Fv1YNJa/H70XphGCpdM7Z19HfBJHWXiSiom5Tz5O2nnnIjIyQNdx7LM3Be++TfaDDyOEIHOfKto6HYnmkPiWpuFf66JmbiahYgdCKKUBR16YtJ1DFD11Cf0XzKfg3bdxHXoIkfXr47Kp7smnOh4kJYF3300wg0QRnj2bkiOOwjX5sParF6XE3Lq1Y1N8Psx168h59hmk08GgTdHlg22GhICtyzZvk8IwiKxZQ/nFl/S0Jd1O3wutrJsNb/yu484xp/wPRnSxJjeoldR/joCti5RjBaXR/erRcNnPyffIfL+DX0pnZvxzaXaVctkaocXUM0masjh01BJ5qoiCcDrJ/vPtZP/59pYHjFHweS7po8vQHCbVP2Rj+nQ0t4HzgIkYdS5qfv0VvSiPzBELcQ9uKwYmhx9J+aWXEf5pAeg6MhLBOXEv8l54HuGOHbIxvfFWBEtcw/0IzcS/Ko1EnKOxZAmRlSvJfeU/FI/dveOGGe0hBFpODp6jj+LIiUOomnU8/Ura5q13tIWQcoRo0sgJh5tyPWM9wRgG4SVLiGzahG3gwO6xsRfQ91bkX/xVtfNqj0l3wq7/1z32FC+EsqVNTrwBIww/PpHc3FXrO9ZDb53z3R6jf0fUx2HNBmO66PPqN77jMXo764nKtTDjj/DyFLVBW5vAY7NugzPeA2cmntF2+p9TzcDLyun/2OnkvvAuBW++wYDffqHoi69wn3uTKoLSbOqGV5+DXf3wS4TmzUf6/ci6OggECP4wl+p/3tPupT2nnRqnkQKjxoZwdM5DBr+ejZ6RQfY9/0S4XZ2vf5AS4XJier0M+eJX1ozKImhveVMxtB5Yg0uJffx4ch57lNyXXiT7/vvI/tejOI88IuYpQrcha2u70ciep+/lkT8wAOpi/TELKNwVLv2pfeeQSpa8C++eG70R8g5HwVkfdX7u0qXwxBia5cW1ZNTRcOaHic3583/UKl/oqLJEE458BCZc2nk726NyLTw6gvbjqTqc8Q7sdJz6sWIV1GxUTZDfOFEVcJlhpbZod8NFP0D+jvHbEPLCsvchUAUjJsd+SipdCkvfUU8ou5yCzBnB5h12jFoWj65jGzECx/jxZFxzFbZhw/BNf4/axx7HLC3BPmEC4QULOw5rCIlnJy/hUgfh8sSfinKnvoz7sEMBCP+2BO+rr+L97+tRM1k6JC1NVY/Wr+wb9noFqhtcTwZS9FGjKPp8FqLZjar67/+g7rnn21S8ipwc+v+8oD6vftsiVh5533HkRgRWfgyf3qA607RBgyn3qc7x3ZmlUrUOHt+57WaezQ2T7oADbur83KYJD/QDX2nbY8IOf+1k96K6Elj2nnLiOx0HGV2cf7tujgo1hdpZJbly4Orl8L9TYNNcFeoJ1tD2BiBg1FGJ38A6gZSSzUOGxcwKUeYIhMeD+7RT8b08tW3pu82mesI2ukHZ4nthlxT+bivVc7MIrHGTkLu02xmweiVC04hs2kTNAw8S/OprJCArKyDU/ClRgka9gqPEOcRPpMJJpLK1bEEvxW4nb+rLuA46sPEls6qKkiOPxigrA78fdB1ht5Pz5OO4j4i9Yu/L9O2CoLqtqmGytzR6WMXugf1ugv17oL1T9lAYcxr89maTzKpmU7HrPS9Obm5Ng1P+C68eC0azG4UjA66NU5ogGumFsOdFydmWCEMPgFtrIByAZyZA6a9tx5gRtfex6Qe1+RozfCZhzRddam4DQgjse00g3KgUGM0cifR68b3wYvTjhoEtTxKpaBhf/68ucfYLkLVPNbZMg4zda+sdeUdIHEUhXEP8mPY0qFiNYWZSesRRmDU1TTcSmw3hdiH9zT5HU+DoFyTn0AqVzaNJDL9O+ad5RCq6aI8kVUQiRFauhGaOXMvOpnDWp3hff4Pgl1+hDxpI+vnnYd9p+6vU7huO/IPLoHq9+mNvpH4VkdFfqeFNvKpHTAPghOeh/3iVkx2sU3HrQ+4Cd+K6020YcRhcvRTmPwtVa2HkFHXj6Iu58XZX7M/EjMDGH+LrkWr3qKwgV47qINQ6b7hqHXx8Daz8RK3sdztLPa11ouGFa9Kk9h15R0iJc689yC/8kuJXshsbQGtZYRyDgoTKHOjpBo6CMI7+QUJbWksKtJiMnEkVSrzLJsGsgyfHEJDHq83V5k8DkQjunWtwDvBS9UUeMqyheQxyDy9Hszc95ejpBvnHlFL82gCSbDPUtQiBbVRbzR8tPZ2Miy4k46ILe8Co3kPvd+SmCcs/aOXEAaRamV6/qUfMaoGmwz7XqK+uIHsoHPa3rpm7u5l4FWxZEEWlLxg9NbI1wgaBSnj5UEBA5iA4/yv1GQEEquGZiao9nDTUvAtegOIFSoI3wWKRwMczEhofjXCZjfAe5yJs05EhteNhVDmomWtH0yXV32aTd1QZuYdVUPJOIaZfjyrk5RwUwDUs0OSIdcAM4THfplr2Q7bKXdA9EhnUG58CPKO8bdJOGtItXUP8BNb0XrVArV8/nPvv19Nm9Fr6QNaKjJ1qFM8fvkXvYsypqg+qzQX2dBpXn+39XwpdSRBrdiV1KM36L0MVSj23b9PvyM//gXBdfcFTPUZQFTxt/CEhU43yciIbNiT2/trYLrCPHYt24CVgU2X7AiX/IqRARjRkWKPi0zwMOxQ9eCo5jzxC2hWXIwa1TJ9zj/S3WE03INFxDm77JBMsduLoF0TWPwXoHqNR2KsFmkR3x/r8U7iH1smMGn3oEAo/ndFio9OiJb3/k9F0lWnQOv9Zs8HOJ/SMTRadRwg4+jG4ahnscnL7+eu6Ezz5cM4s+N0rkBEjL7hui9ogXfIOfH5bq5ZwzYgWm29N1Xr49kEi026m5MADkTVRspGiodX3dWuN2036JRdj3203pF2JaEV7JgiFdT77ZQw1u1+J5+RTyL7tVor+fDRamoGwNTnZaPc74XA0tnJrxOGAQQdiL8ombXQdwmYS3OTCDEe7uiC4xQlIbLkh9PSI6vupy4T6i3bk8kV+vuqdmgD6yJH0+/Yb9JwUhCm3YXp/aAXguH/Ds3urTidhLzjSlb74EQ/1tGWpp3ItfPsgbP5RpVLudwMU7NzTVqWe7CEq0ydaYZdmA0+B+n9OK4DS32CPC8DbTirf/Ofhl1djO3EhIL+Dz/HnV1RqpjSo/iQDs9pFPBkdzkF+ZEQQrrIhA02Oyr7bbuQ88AC2IUMI/7YkehpjPZqU7Jmzjox3j4PdzoQRUxDf3kXW3nbCZQ6kIdCcRlRzhCbw3Pw0oT/dgayrRRomrkMOIeeRh8BlI2vkxdjfeZ+q2RmEy+3Y88KNK3szLAisdRGp0dFzwhSeWAIayKBA2CTBzS7KZ+WqJPI4WD3EzpCNYWxR/L9jn71xHXoo4Z9+IvjtdxirVrUvTeB0krmdNIxOlr6Tfhjywa//U5WCRbur1ZzN2fF5fYmtv6jsnEhA5U03NL44+xMYsn/TOClVbrTd07nHVW9ZU//LHY9RzrIn+PIumP2Pts5cdwBaU6aO3aOcsL9SVc1GI71/zPoCiQ5FYxGX/dQmRm5WVlL33PMEPv0Ere5n0netwTUwyOaXBsRo09YKXSKEbNQ0FzYTz+g6Mvfyol0wE4ar6uLaxx6n5v4H2unMI7Hlhik4rhR0G1Wzs/GvcKj2agI0myTvqDIc+a0Kz4QGp70DOx+PNE2MLVvQMjLQMltW/Nb86zFqH30UQgHSdq7DM8qHNATepWn4V7qxZUfIObQSe264zTZC3VIP1d9md+jMTWBroU5+mYE9xkJepKeppXtuLrKDsFXGH64j84YeyETrxfT9PPLtgalTYPWstq8XjoUrFqnvf3gCZt1Uv/IUMPQgOOtTsMeZPrboVXjvIrVp2NCn7JinYI/zUvQmEqB2Czy2U8v8cs2m1B5bP6jb02DcufDjU22P5Y6CipVtX0fdq/wr3VT9WIDn5DPJuuOviHrJVrOykq2HH4FZVt5YVCJsJpkTqqldlIHp6+CB1WZTN9zWG5O6pPCUYhh8GNq5/0XPzaX2yaeoue/+9svoNRP3CB+BdR5kuK2WieYy6HfmFkRzf6o5VUvCY9vXdik/7wICM9vT7lfiYgUnb8We2fJmI02onpeJ09osiQAAIABJREFU9+eO5SDCOtiiPzgkhD58OP3mdF5MbVulK5svW6SK9d9Ef730V1XduHAqfHx1s/CBhHVfwd+d8PxBHWuW1GxWTjwSUBuCYa/6/sPLVWy4u8noD+d9oW5Umh00h3raskfJngh7oa5YpXUKm3paERoU7QYXfY9MK4p6CcMrQIf+p60ny/FPwnftoETOgLoXXsQsr2hRGSgjGjU/ZuHZ0at2JGMg8vJw7dE/anaJEJLABie17y6gePwEyq/7IzUPP9KxFoqp4V+VFtWJg5LcDWxyUvVtJptfGsCmaUXMX5LDB2/M5KMbjmf5jFcxTSNquMI2emcVN4/9jpARQe2Cttru0hD1aZEdI1JRAepwkP/2W8nOsl1hOfLeRCwBLN2pHN3MG4m5pbRhtlJdLF0ae/4l06K/Lk1V0NQTDNhTPW3cVAq3VMLh99NyyVmP0CGtEA6+XY27YA5cswou/xnpzqFu1eA2G3lmWCBs4B7mV35fgN22EfngIJj7BIFZM6OXsmvgHBjEPTSA+ryjrPRratBqlkRVkZIm1C3IxL/OBuEwgTffBF+M2H2bk6E9V2j6NLy/ZmBENOYNzWRFOIvakJ3qDSson34robtz4E4dHhoMC15qPC/9nLPVpmi7iDYbomZYENjgIlzS9tzW79zQIJKCHs05TzyOrbAw+Ym2IyxH3lOs+Bie3A3udsG/dlTdf/a+uu1q1OZS6XqaBr4OuutIA15rR0QrEqwPW7TCNNSxnsSVBQ6PChU5M2jjzGxOGHsWfPeIKuP/8alGm4Mz3oLyZZj1PbWlBMOn4VvpRmgt7wtCgDACGG//EWPFoqimSBN0l4l7ZEOuexTHaoTR3IbK7miF0MH02yCeGHvLK7d/1BTULlKfTVm+jYBTQ+rKtsEZ1exVsAGXrFXz1GyEj66EBariVO/fn/xpb2Eft3u71zC9OlVf5xDY6CSw0Unl7BwqP8ul4TNofluTqBqiiK6c+IaBdqqy9OQSFl0u3IdPSWYGZds22Ju0PSxH3hP8Ng1eOx5KFquNvooVMP0CyBigqjZ1p8qbtrmU8NbhD6rzXHGkYFWuhrJl0Y/tdBxRE4l1B+x0fOffTyrRdJVumDVEZa04M9XNbfI9yoF/8kdY9QksmgpP7AwPDsb5/emkj67Blq70RGQEqr/PRAY1tCiqglJC3WIXpj/aylei2SV1v6RTMTOfmKtjAbYsg5yDKhE2E2FXX5rLIPugiqZBCdHBeE3DqFIr46osDdPWNH73wjJsrW8qYR98Xi/tGw7gKHmDwguyQKjiotzDy/6fvfMOk6o6//jn3Hunz2xvwLKAgFQVFEGNFTR2jV1jNPYYS2IsUaNRE40xJqZaEqP+bLH33hUExUKVvsDCLsv2Pv2W8/vjzpbZmdkCi1HD93n2gb3lnHNn5773vW/5fik4ph7f5A47qZpYQ2Sjl6Y3C2l6s5DwRi+6phBxKcQcgspSd9cqFcDQoKZIY+luHlpzNVxxue2hFbebnNtuTRHqGAyi8z6m7uDZbC0tY+vkqbT/7W/IvrhyviP4dpQffpcQrIfnzrCtTU/oYXj/BrtTdc7tNjVu3ljIHtl9zBF/hxfP7H+O+2fCtY2ptcUFE2C/q+DTP9u8J0ibTXDvS6F46nZf2pChcBJcUWHricbaoXQfeOcqCNWmHtuxBej2uoWw+cSy92ulY1E2lpH+2WU022RRwmGRNbMN77gwCIhWumlbmEN4bT/t/Ba4R0dQnRL3qKgtSqFJHEVxOhan0RAdMDI9OARCMVF8FmZIpbDBQFcFTQUaCIHPkSH+3rHV1nd99FA6fensAzx4d+nuEHUUxPFOCNPwclFKzD/sUfl0Vi6+iMn48iCjtiQqiRQFhMBlSkY2CkbW9EMtPQAoBfloY3dJ2tbpWQ9Evi2+aDHN553fxS8j29oI/uMeZHsH2b256r9j2Fm18nXj9Uvhi3sz7BRwQ7hvHpU+aXx7YNhe8JMMn/HWRfDVk4C03wBKZ/Y/3n8bfyiEyMCFm6WEeL2Ks8hMKqeTEqyYoPbxYSAFhT+oR8vTURLl39ICK6rQ/FkZRlUMK5ahgUVVKTrPiYPy5LEjCrVPpk+CbisUl0nuIc24httdmlZUoWVuLuFaN7pTsHR3D4dPrCDgSmPMAyNssrkeHDbSSk1DWLqgbUEOoXJf6qOkU9whkawVHg/e004l8KvraTjqGMz164fsWoXHQ+Fbb4CUtF73K+Kffw4OB96TTiL7lptQfL6M5zae+SNiH6VR5nK7GfbVMhTvN5eCYKDYWbXyTcGalzLvc2f3Xxt/3vyBzVOzyK41T4fhe8Hhf4LD7/p2GPFoe/9iIr0gBDgLzS6Gh84fgPA6LwiBsySOlmN0GXEAM6wS2eRGc7SQ9/1Gsr/X0iuhKcFporhjNM8theMesMsfNTfhdR5qnykZWiPuMcg/uhHX8BhCtevJNb9J/vebcAd0XFHJruUxltcVYPQivZLCCXtemEpElmZ5ikPiHmV/xkmBCLcb1/VX4zjrDNSyMrQpk8n+3W1k33YrjcefsE1G3LHXXiglJWl7IGQ8Tsdf/0bDcccT/+wzm2spFiP83HM0nfXjPsc11pWn3S5UFbMmzdvcdwg7QytfN5x9vLLv98v0pE5S2syHQoPP/o79ZxuAzmPFB91iDdsIY8sWgg/9H8aqVTimT8d/zo9Ri9OX+u0QbHgXnvyBLRg9SPT0OqXs/mjD63wIReLI1emZYGxfHKBjaZbd4CMhvNZP7uxGCk+oo31hDrGtbhwFOp4xYfx7BDHam4i2XYfr8rUIIXA3N+P8yU+Jf/llitjBYKHlxsib04LqN7oqLZOvTeKbHMT4NJe8FpPARw7ahueSPbMN1W8SrXETqtsb6+P3cDqyyZrWjuLqDFN0j2PFBdFKN1ZMwYrZk4ScgFOlNc/F0pOn8/aI92AEjDx4GldtGI915x9pverqbRYZdx1zFKrLTdttv0ut5jFNYp98iuwIJo8fj6MvX46+chWOKZPTf2aTJmJu3ZqyXZom6vAdzLn/X8bO0MrXjYX/gPevS20lz98VLluTasgrPoIXz7YrVsxY4ss9wL/Z6a/CxGO2eanx5ctpPPlUZKdqjNOJcLspfPUVHOPGJh0rpWRe9VyeX/csjZFG8t35HLXLMRxadhiO3rH6AS8gZAtrxIPbfA3pIA3seHiNE1eRjuKUxGqdNL1Z0NWh2QnhsCg5cysgiDc6cBbotjhD4s9khhTagheRd+89XaROZm0t0fc/oPWX12ZaAX1R1aJKhv2oBsUpCZV78IyKpk3aRitdNL09gK5cReKb2kH2zPakr1dsq4umt/PtWTtLUBA05mj8+nepxjLQpnPVXzZQ3LB9DykUBeF2I9OVZDocdvdrmgSl8PnIvvMPEI8TfOBBZEcH7sMPJ/Czy1Dz8ogvW0bjSackcbALjwff+eeRff1127fmbwh2dnZ+U2CZtmr8qucSbH4W5I6xKzX8vTzdlk1w79RUytfB4Nj7t1ngov7Io9CXf5W8UQhcBx9MweOPUtm+mQdXPMCqppUIBDEzhkzzkBmTNYaLdr+YCXkT0dLS72XAqhfgpXMhPkDiqkHCMgBLIFRJ6/wcwutSxY+FwyJvdjPusmiSV98JaUHDa8MI3PoAnqOPStrXfNXVRJ55FiwLxWMQ2KsdLdsgWu0ivNqHjPX+LCQ4LYpOqEMLWEhdUP9iEcUn1dmNuL3W3rEoi+DygYlva7lxik6o7+KekwbUPJ5KQyCBjaPd/OmaVCk9xbQobIhz863rdoymkBDgcmXmpFFVPMccTeSdd21FIACHA6WwkOL330XJyiK2cCFtN9+CvnoNSm4u/p9ejP8nFw0oWfptwLdbIei7BEWFEx+zZeC2LrLL7EpnpQ+pLLo/VdR5sHj1p3bn5Mh9BnWa1HX0FWnYAqUk9umn1IXquGbeVUSNaFrj3RMV7RVcP9/2Tn2qj71K9ubHU35Mobefpg8zzpDSqPaCooE0JNEqN1anQGUayD4EF6QFzoIwzZddTtHEiTh6VF3k/OYWjJUroWkV+YfW0fZpDm2f5iIUiTSE3ctu9AjQC0nB4U04smxv1OjQsCIq4XIvnnHdFLbSBBlTCK0ZuFCG0eIkvNGDb7xtAGNbMyXUpa3PKbs0MLpgqQqtOQ4qRnvZZdPgQ139QRQU2IpLmQy500nkrbeTm7h0HdncTOjJpwj85CJc++xD0dvbzyH/bcPOZOd/C3ljYeqptoHN5C20bEwYs+2BaXO4DPbNS1Xt19w0UDweXlz/PHEz3q8R742QGWJe9Uec/865nPHaqdw4/1d8Ur0A3Ux9YMVDxcjodryNDADSEnQsChDd6CFtB6clcA3PzFqIFLahj8dpvuiipF2K30/hG6+Tf6Ig+JXfnsMUthcsRWpSVEmuSlV8JtIStM7PpW1hNnqLhhFSCK3xUf9iMTI+uNu3bX5OV8NUJ6V7bwgEoyqjTKhI3yAmJLRn7xj/TzY0IBszVyZpu45P250qo1FiHw+wCOA7ip2G/JuMMYek5x1JwgBuKj2YIJsaOISi4D3xRPtVtyfcbnxn/Yh1LeswZSYmv4EhZIRY3riMO764nZNfPYG7F/8DM8EOGPvkUxp/dBGt87OwDNGtJbFdM6aBkBjtjh7OeCIHISSoFjn7taC4ZMbnoNAk2fu2UnxqDQ59MeG33k7eryioRg3hVX5kb/ZASyGpT9IUNL1ZSM2TJVh6ort0TBhUSXiNn/rnSqh7Yjhtn+TYKkKDhDRUap8cRqzGiaM4ljE8ogC7LW1FNVIvOu4QjNpWb9zmSNi2cwH/RRelZ49UVbSykanb/4ew3YZcCDFSCPGhEGK1EGKlEOLnQ7GwncDWmvSXJIsvKA5Q3Xb35+iD4eIvGdCfcenDg54++7e34Jo1E9xuRCAALhfu2YcQ+MUVjAqUoaTjRNlGSCTvVL7FCa8ex9Uf/YIVd/8GGYkSXuun4cUiOpYFCG9wD6kllxLCG7x2glN2XosdYtFydAqObcAzJmJ7sYagY5kfadg115YubLGiuB2i0bJNcg5oxXjw9NT2cHcuVr/t+qLrxwqq1Dw+HMuEnANaEgReiYeLYgEC1WcQ2KuN/KPryT+iAeewTl6YvmFFVJrfLsAKavh278h4zn4f15PV3ustSUqkInjk7JEDpSdPgjpmDK4DDxz8iQCahtR11LKRKeIUwunEd+452zbudwTbnewUQgwDhkkpFwshAsAi4AdSylWZzvmfTnYOFuFmmP97Oznq8MHMS2Gvi0gqft74ATw6p+9xyg6A87aNFlRfvwGjogLHruPRRtnamJXtlVw19wpi6YQhhgJSgoSSmijnPlJJWXUMkBSfUYvm3743gZ5TtM7PJrwmfSem0CycJTE8u4bQ61yEVvtQ3BJ3WQSkIFrpwlGgU3BEU9KYsTnv4z5wdvdAC/9Gw6W3Ea/tS1g5ZXX4dm8na68Ou5pESNo+yya8xk9gz3YC0zq6nt9C2EUeze/mEav09D+HQ8E9pp1YpQcZVdIebwFbixz87uZJaYc464lq9lvQlHZfJuTeew+eY4+h/sijMVatSluZ0ifcbvIff5SOP95FfMkShKYhfF5y77oL95zZ/Z//HcDXVrUihHgZuFtKmZH8eKch3wG4qww6+iDqH32wTRk7hFjRuIL7lt1DdXALmtCYPXI2k/In88jKh2mONfc/wECQMOgFDTHmfNDInPX1FJ/QgTCGJnYuDah7tgQzmOmVX1J4ci0NLxanF1YQkuHnVndXg0ho3TyH2NoI5tataGVlZF1/HZEHryOysNPwCbo94T7KEIWk4MgmXCPsh6WlC9o/yyJrVhtKmvSFlNDwSgF6g7PHGwZ2401WFrS2IrxePMfNIfzsy13lhr3RlOPgjmvHEfKqSC296z1ufYir/rIhw9rTQFEYvrkCoShIXaf15lsIP/LowM8HUFV85/yYnN/+BrO+HhkMoY4e9T+l5fm1GHIhxGhgHjBVStnea99FwEUAZWVle23e3A939k4MDtWL4d97Zd6vOOCm7U2cpkfMjKEpGmoPXVXd0vlg8/u8UP4cNeEBUAoMBFLiipic8Votc+aUklX1xnaXwUkT2hdlEUwSTehZ593z/kinsyYZfk51V3mgNGHro8NtoWOXRG922KExS4IxgCau3nDY5YiKU6K4LIygiuY30zL9QoIw7PNsQqv9iRbNxJo1DXVkKYXvvE3DMcdhrk1PrLZoejaPnD0SSwEzwayYLhm/6xaLX/x+xcCvw+PBucfuZP3iF7j2/x6hxx6n9dc3ZeZoFyJtgt7zg+PJu+fugc/7HcMOLz8UQviB54ErehtxACnl/cD9YHvkQzXvTiQwYk8onAoNGW6u7S1j7AMuNZVWwKE4OHzMERw+5ggAakO1PLXmCRbWLCS8rd60EMS8Gg+fMoKHkRT5DuKqTUuYFN2OOnOFHsx/oAYMXCUxzKhCbIs7UYOXqYFH4iyKdRtxCcG1XgqPbUDL0bueAa2f5BBJU6M+IJhQ/1wxSIGzOE7OQU0ZjTjYzb+B6R2J0sQe8xkG5qbNhB97PKMRj7gVHjl7JLqzbw/XpTg5fPKRSFYM/IoiEeILP6PpnHPI+fvf0SZORDgcyEyGPI0RF14v7sMPH+iM/1MYEkMuhHBgG/H/SClfGIoxd2IbcN48+ENe+n3a108YpK9cRWzBAkRODkVHHckVe9lCusF4kBfKn+PlDS+hb8sDJuEh1ru8XDvhe3hMg+sqvmRaqGXQpjK01ktwcRYgyd6vFd+EEFIKm6PFEDS+XojRmrkz1Tsp1GVzpAR3SQwtJ9ljzj2wFQRE+mNUTIFMVLbYiNc6aXqrkKKT6pJSJL0Rr3Pateq969+lJPTYYxnPWzPBj2pKUv4isrO2XKKaMG5DE+PfeoTNo70Mq47g0gful8lIlPabb6Hos09Rx4zBWL16QLFy4fHgmDIFz5FHDGgeo7qa4IMPoS9bjmO3qfgvOB+ttHTA6/y2YSiSnQJ4BGiWUl4xkHN2xsh3IP5zLJS/lrp9zh1wQKaW8aGFtCxafnElkddeB8tCOOzyvvz//AfXjOTwT0XrRu5bdg9rWvpQNhrQpBKXZXBJ1QoOaK3p00Pp/MrHttiGEQTuMWFyD2rparqxrwPMoErd0yVkCqsUn76VjuUBsmd0EKtxARJ3aQy92UFwpR8rpOIqi+LdNYRQJG1fZBNe6U8/3kAgLPKOaMRVHLcbg9N0m8bqEnQDaSplRE4OsrU17dBLd8/ikbNHEvX0ekpYkrEbQ+z+VQe7rgtSUhejcoyfeftks/8nzYypCA/KmCMEzoMOJJ6OqbAn3G5cM21SN8/xx+I98cQBqByBvno1DT84ERmL2aEbhwPhdFLwwvM4p04Z+Dq/gdhhMXIhxP7Ax8BXdBOn/UpK+Uamc3Ya8h0IKeGVC+1yQ2na8dn9fwUH35S58SgTmjdC5ce2xFrZ/jD/DltxxtJh0okw+3fgK0g5LfzKq7RedXUKl4aSn0/JkkUINb07uaGlnOvnX0/U3A5uaykpike4a90CstM0GQEYQYjVuGmdnwuGXbWRf1QD7hGpFTiWLmh4pRCjubcBsevNsw9oQQhoW5DbVUUiO2PTZuJf1UL1mXaLvEOy9aHhSZ72IC8QVJup0F0WwT0yhuKQNjVtQk8bAXVPF9sJ3F7tmerYsWgTJhB7I/X2jDkFv7xjMnFX8t/HGTO59L5N7FreHRJbOdHP0mnZfLJvLpffU8GEdaEhb9sXfj/Dli/tEsseKBpOOoX4woUp250zZlD48otDtbz/CnZYjFxKOZ9tdi92YsghBBz/gM2xEu8AZyAtXSgAm+bCon/bx005DaacCqpmPwxev8R+GCiaTb2nR+yxOztNl/yfzUx46UpbnKIHwk89lZYQScZixJcsTfHKOzE2dzzPHPsc1cFqHlx+P1/Wf5ne5ezn+pscbu4vncI1m5emPUT12fXjikaiNd9u7EkLiU2SlToRIGj7OM+We7OUhOG2yxYVj4kZUm3XxlQwQxBc6SdregeeMREiG7xs220jwBREN3qJbvQlrsdA8ZjkHtJErMaFb3yEvMOaaPkgH6NDTapOMauq8F9wPlZLM/qnycbOFZdc8GAl/75gFEJKTEWgSNh/fjPjexjxmFNBAPt/0sxns3K7Po2hhPB4yLr+ukEbcYD4F1+k375oEVLK7wzvSk/s5Fr5rkJRbH7zTHjz5/D5PbbXDrD2VVv0YuYl4PDD0kfBSG1Nj9U4Ca32IXUFz4RWPMufQux1btIx0uijzjudZmgvjPCP4Kb9foOUkjc3vcHjqx4lqPegNe3nRjQVhU+zS/rkGPSOjhAMaTZ1qxRENnhw5MVTy/ok6I39vM4nWu1Vv07uIc04i3TbK7cE7YuyCK0IgKkQrfCQNb2DwPR2Ihu929nc1H1lZkhDSlB9Fr4J9tuMGVTtz6t3yWQ8Ttstv0HJIG6828oObr9xNUumZxNzKkxZ2cGwuu43FQu7mmVkgyTQFOGEl2pYNSnArutCQ9ImLnw+HJMnE7j8sm2uDRdeL7KjI3W7x/OdNOKw05D/b+LzexK85j0hIdYKH99Ocp1zN9oXBwguC9iETwhiWy3CVX8n/52zk8IlvlNORl+8OIlOFABVxTl9+oCXKYTgqDFHc9SYo2kMN/CPxX9naf3ibn6XPm7K3oRPXdstaHo7n3itK3Ed9rWH1vjxjg/bQhMOaXdzWoKWj3IzD2YvAgDP+BC5B7aAsJdlJzslWTPasSIqkQ1eFLcdeVR9Flq2jtHaf7x3YJDIuELd08MoOLIRR76OI1/HOTyG0eYg5XEWi2FVZe458IVM9p/f3QdgYX8EUhE0lAbI/dtfGNFg0nr1Ncz+MkrQEe4u7tlOZF13LVZrK8bGjZi7TUXN8MDpC94zf0jo4UeSWRRdLrw/PGP7F/gNxU4a2/81BOtsju9Bwgwp1D49LPGa3g3hdpB7z714juiuJpCGQdO55xNfuNAOsbhcCEUh78F/4z7ooO2+hLcq3uTeZXdn9tClZGZbPTdULEq3yzbSuiBe76L986yEQZU4S2Jkz2pDy9MJrvATXuPD7BgIl3qiKSiDW1QZCfAfMZnNU9yMsIKcuKmCnPsH0IE5aEgUt0X+0Q00vlpkE3ANMhbfyXbf8yzpdKDO2BNr+QoUrw/fWT8icNmlIATGunJEwE/03fdou+nm7Vu+14tAIiNRm/teEeT961+D9sxlLEbzpZcT/eADhNOJ1OO4DzyIvPvuQbj7kFH8FmAnH/l3EVLaMWvVOfA48lMn9C03lwHhci+tC3LSVkJ4TjmFvL/+OWmbUVND/LMv0FevQsnLw3vCD7bJu8qEpgsuZFn5PO69eDRxd+JtQAhUyyLLjHPZW6uZMaKmz5praYE0O0sMNfK+34izUKf5/TxiVZ7MJ/aCmm1zfafrtqx0+7lm/H7EhIql2E0uioSjXq/jqLfqhz627LC9fTscNLjR++s17YLbhfvgQ8h/8N9Jm6tLywbPstkPhN/HsGVLUwywUVUFuoE6ZnTGcIlRVYWxfj3aLrt0UUt827GTj/y7BClh3u3w8W12HFuoMPEHcOqz/Rv0LZ8Nfj7FgXBr9jy9359VFSWnOxavb9hI809/irHebt/WSkeQe+8922/EjZjNN1OzBAomILwaEzZE+NvVq9ha5OBfF42mvsTN+VtXcVjTFqoXDie+vxNncTyjMRcKCEWSPbOVxrcKiG520zY/NxGOGDichZk7Zh8eNoGIonb/XYTAEvDaMcUEfSqnPT9EXa89oDcP3oh3YkBnRWNE33mH+JIlXaGy4COP9m3EM3Rq9gcZDFH7/SPIu+tPuPaegV5eTvOFP8GoqkIIgZKfT9699+Dca8+Uc7WRI9FG/m+wIu70yL+NmHsbfHgTKUZV88HPyiGrD33Cf+4JtUsGNo/qhEknQc5o5NSzqZl9MrKtLfkYt5uiN17DMWECMhqlduY+WM3NSTetCAQoWfgJSk5OxqmMykrafnsbsY8/Rng9+M46C/9PL0ZfuQpiLTi/uAgRbrBl3xw+JBr1z+ZgNHQnT8vHeFhzsoNz2tZQ/0QJDmlReEw9WlbmhpNKt5/3sktpKs9i+tJ2Jq4NDtoEKi6Dkh/WpoRWpIQf7nYYIS39g0GLW/zqjvKkZKIEKke6KauKbpspVqVdRdMv2+L2Qxs3jqL33iH8wou0XvPL9BSzAEIg8vKQTYMj2eoN73nnEX35JazmluTvl89H8cJPUPMyNMN9h7AztPJdgWXC7YG+VeX3OAeOfzB92eHa1zGfPI7VvhxiisrkUDOelEoSYZcUHnanzbaYQHzpUprOOhsZt+uzpWGQ87vb8J1+GpCoH7/ml8hgssam8HjIuvEG/Of8OO1yzaYm6g86BKutrbvLz+m0b1ZNI3tmLao7DqbANTxmiwgLBUOMpu5hYV9nj8RqeIyBVzUxKlxopqTkrK2o7tTv+Rv5ZTw0YhIGAguBU7eYuqKDCx6qHLQRzdqnGf/U5JJLwxScO3EO7f70SU1Vtzjh5VrmfNgtprBplIe//mwXstoNfnb3RgqaBtr5al9fYHo7VlxJcK30cxWK0m9XZV+VP2ga2qRJGCtW9Olti+HDkWlEkbcFwutBhnt9991usq67lsCFFwzJHN9k7AytfBMRD0O4AfzDQBtgBUOs3RZh7gvLHrYViA66MWVXedGu/Hb6McSNKEJKDEXhkppyZu97AzStherPwVcMMy+DUfsnneucNo2SJYuJL/wMKxLBtc8slEA3BaxVW2sLNfeCjEQwq6szLjf06GNY4XCyUekcR9dpX5gwStKOaTuKY+Ts14qzsJKSj9YSXbiI4P33Y5SvB8PAW6EBGkKA7hC0f55N9r5tSV2b7aqDB0dMQu/R6x53qaycEmDVZD9TVg1G8FninxRU0Z6AAAAgAElEQVROiWqpwKGf1vHCoaVpQ16qJXFH7YeoIcBwKDxx+ghibpVGp8Jvb5iAK2ZSUhfj2NfrkhpyUmGP37Gkj5LTnnC7yf3Ln2n5+RXdn3XKVfVjyIXo14gDQ2bEAWQszVqjUaytQx+i+jbhf4f/8ZsEy4S3fgF3FsA9U+x/P/lzv6dJyyL6xVdIawDqMB/elLJJN3Vu/uRG2qRBRNUIaw7iisq9I3enctyhcNgfbKrbU55KMeKdEJqGa//v4Tns0CQjDuCYPg2hpfoGwufDmaEJCOxGjSQdxl6QMRWpKwkBCIFe66LhhWKa3s9GHT4C36mnUPDMMzinT7NFMPw+UCXZ05rJmdiGNAShcneSvVkSKEBLo3UWc6ssP9TLYGvprDTer1AkJ1ZXULI1msHYCaYtbberRCSs2dXHbivambLSJgHTXQrBLAfrx/v56+W78O7s1C7abUXWr2+0eUv6oICVou+iF0vXhzy52S/SrFf4fDj3mdn1e3TuXJouuIjGH55J6Oln0joX3zXsNOT/Dbx/gy2sbERAD9mdlR/+GpY9nvEUKxik4ehjaL7gIlo/DQzg/pGw/AmIdTdGLKlfnFaezZAm725+O2X7YOGcMQPHntOhZ4WBy4U2ZjTuOZmFL7QJEzLqg6aHXcceXe8husDuTlTzcil86UWK332H/EceZtgrd5M1PUxkvR9HUQzVLTFD3V93RzrBSkCxLPw5UXxTBuGRi8w5ZgHcdHs5eyxpBSlxxEzcERN31OTi+zfhi5gI7BtxjxUdHPN6PRc8WMkv/7QeR7x7jVIVvHDiMObvmznPMBi033AjkXnz8F9wftr9EhASLEWkfaR1eupfqxkXAuesWcnyg2432thdur5frb+/g6bzLiD65pvE5s6j9cZf03j6D5HbQiH8LcJOQ/51wzTg87tB79XCrodh3q0ZT2u/84/oa9YiQyHCK720L7KNeZ8G/YUz4Y5c+MD2zkNGKFWGDLCkSTCe2gk3UEgpWdG4gtcrXqPyz1fgv/IK1F3GoI4aReCSn1Lw4gtpPfVO+M89B+EcXKVIJzr+cU/S79ouY3Dtsw/KtB8QHXkL0hBoPgtPWXdziBlR2HVBBCuNUIQDyZzWaluBZ0CwhS9iW1NDY5YBoTU+BHDxg1Wc/uQWznmkkvMfquTOa1cxaU2vXELixx2zGF4d5fvv1CcPKATPnjICffBynWnRcv4FRHuIFvdQD+1aiyOh25mZpODrg/+aq9HXr7dDcIoCQuA+5BAKXngeGY/TfMllhO6+J7kRKBxG/+orom+/kzSWtCxCjVuJh7aDAvkbhJ0x8qHE1kV2jbbqgqmnQ/641GPiwW6+kt7oyBznCz//QlIsM7gkGyOkkHdgW8ZzALv7Zd6tmJaFNf5g4lbq3G7Vzcxh+/Q9TgZEjAi/nv8rKjsqsaSFqqjk7JrDH957hRzXwLxHrbSUgqefouWX12GsWTMoCbCU7tEeMLUypKkQrXbjKo0hNItYvYOmNwqRFlzcvJl/XVpmN6EgMIXgtNpyxkfakB7IFCFWs3QcOQbRandXg1Tz+/mU/LDGTsQCVkzQ8FIRZkf3LXbQghZMbO+pPwPoNCT7fN7C68ckN28pFmwe7WXchm0UQO4J3cBYtixpU+8rzuR1f62N7llZFL76Mk2nnIasT364xT76CGPDRtp//3ti8xekPV2Gw0TeeQfP0UcBUL3oQz6//yb0cBBpWQzb43vsc+kdOH1Zac//NmCnRz5UePPn8OD3YF6iNPCeKemV693Z4CtMP8aw1FrYLqQxbtF1AeKNjn5fby3g2toP+NeXf8SSVpIb71bdjM/dlZnDZvUzSno8tuoRKtoriJpR4laciBGhPlzPPUt6UwD0DW38eHJ+fzsM0jP3nnxSxn3C6wULIuVerLACKrR8mGc3NZkKE9cGeWDBXC6rWsGF1au4f9VHnFy/EbA7P9MSaWkWvklBcg9pZtiZW3EWJ2L7pqDp3XykIRAC2r/Ixgxp9DZ5Cpk93IHAUgS+0MA1SzsTlpn2bRztpHysk3a/ktE4/9fZSdrbaTrjTKxQarJXxmJ03HsvsYWfZVZgUlWU/HwAmitWseCvVxJtbcSMR7GMODXLFvDxny7bkVeww7HTkA8FKj+xjXZXNYkEK24zCLZWJh8rBBz+Z3D0FHoQ9u+H3Zl+fCkJHDeRvMOayft+I55dwnYAU1ForzkGofbddmwKQcDUiapa9xqkxCU0frrHpfxmv1uTZNoGg4+qPkwRhzClyZd1X2JY/cclpWHQesON1Ow+jcZTToVohqRnmiSXMrIUX++Sxo4a9Ecvpvnw8bRec3ViDoW6F4tpW5CD2Z78EhpfEOB7DbUc1ryFfKN7bqFJFK9pf87dq0VRJb6J4YT0miT/iMaEwpCdhK19soTYVifhtb605X8CMaCQhAQ+n5H8RqOYFoWNMYbVDl7wOt07TtQl2DLCQU2xg+W7eagckfkhOhhjviPi5tbWrell4SyL6PwFfSZthcOB74zTAVjz2sOYevJbqWXEaVq/nI7ab6/85E5DPhRY9khmKbV7p9hx8Z6Yehqc/hKM/B74S2DcEXDux1A6M/0Yb12BP/AWntFhPKOi5BzYQt7hrSg52eTedRdc+Bl99aKbCLJ6h3OEQJpxpuRNQlO2PcKWLnkKIJFp4/G90X7Xnwk//YxdtZKpukBVCVx9Fdm/uQV1VBlq6QgCV19Fycfzktuzmzei3zaVhpteJbIykmy9dIVIuS9l6NgWNw2vFtK+2E+8wYEZUYjVOrGiCoXHNeAuiyaMucQ9OkzxGTUozmTj7i6N0hmUkDGFxtcL+iHa6nl2+m0C8AUNHHELd8TEGTMpqo9zyb2bBmUoRY+fTo4wU9gfjSMu2WdhiNItcSxVUFnmJOZMn9xMt9a+1rFDkqCZvh9NTZCGNhmw+xD+eCeO8eMBCNZVdhLGJ0HRHISb6oZqpV87dsbIhwIdfdTJxoMw97cw+7fJ28ceZv/0h8a1sOjfiB4NQIpD4i4zKL7ydpQxY+yNZ74Jjx9BultIAdZ5U+PVEonSuMauY99G7DNsX+ZtmZtk0AWCSXmTcah9h0mklIQefKjPODeAcDpxH3wQzj32yFhlAcD7N9C+QCD1TD5vBk6OJgcdTdl0LOqxUQHPmDB5c5qwdIg3OvCUpj6shQZmCttvIrIs6NOgN+RqFLSkvrV0nnHQghb2XtzG5jIv/pBB6RZ7ojXjvExcHx58B6qE5iyFnHYLAWgSMGFMpU5Rvc7S6T5ac1WK61LXFHc6ifv9uDs6kEIgpEXVqFxGbWhAkRD0qcw9IJ+K0V5GbIlw7Jt1IEEbeLpjaCEEwuOh8N23cYwe3bW5aMpMWipWYxnJDwVT18kpG/81L3LosNOQDwV2OQzWpZFX68S8W+129zQNOhkhJVR9Agv/apc/9IKQMcSWD2HKMfaGcd+HI/4Kb11Bb2Ne6/TQ0Ev8QUjJ8HiEAm/xwNeUBudMOY+vGr8iGO8gakZxqS6cipPLp/+s/5N1vX8j7vXiPvIInHvs0f94G98jXrctPCNpjrcgssmD4rXImtGG6pJICUabRtsnOcRqXAhV2hJujp5j9BhLJtrlu4xZ8jw57WbGlXZ65d6IxaS13dUtMQdU7GIb8kznZLpCAV1GvHdC0xeFrBYDVZdpx1CkZM2hswnl5+OIRmkbNgxhmhjP/RNdk9x7yWhM1T5z1WQ/cZfgmDfqkDpohkT03Vo05HAdcgjZN9+UZMQBJhx5Nhvff5Z4yEQmOppVl4fxh52BK5D7ta1vqLHTkA8FZlxsN/ikjUQm8OGvIWc07PGj/sfTo7Z3Xf155lZ84QBXootv6yKo/gLyxsHPK+C962D1813hnmGxMBPCLZR7c9CFglNaOKTFtU0NUDhpUJfaG7nuXO479F/Mr/6Y9a3rGekfycEjD8Hr6F/sWTidqGVlmJs2pe7LzsY1aybeU0/BfcTABHelOxvF04EVHqL6PFMhvNpH1ow2HPkGVkSh4eUiZNw2hdIShNb4UDx9/N0lZDJgDjN9AKJnGaAEqke4GFltx8VdOhz5TmPa8zorTPpjT8+0f5dNMbypWiIAaLrOsFWr+fLk44iLOG5LxWFZOLPHMnePJkytO7R39uNVTF/ahiveM7H79Rlx4fFQ8Ngjafd5cgo4/I4X+OrZf1C7bAFOfw4Tjz2XMQce/7Wtb0dgJ9fKUGH5k/DCD/s+RqhwfTs4+zFyH95i62P214qvaODJg1gQkN2/nzvP3v/WL2DdK2AZSGCVL5e13hzy9Sj7xHVc58yFosnd40Va7DFcgXSz7RBEP/yQ5gt/0u2ZC4Fwuyl4+qkkRjtpWYSffobQo48hYzE8J/4A/3nnoXi7P8vYvecRevYFIuu2Q9w4BZLcwxpxl8YILgvQsSyQqrojZIYQSv/hldQzZCId2g1Dgfn75XFwD7GHzOcnltTPMb339zwv3f6YJnjw0t1YOV5BSIGCYGZ5Poe8X03B2lXcee2ubB3uprg2yq/uKMepS6IuhVePLubkF2q+1soXbdIkit97p/8Dv4XIxLWyM9k5VNjtdPBmKCvshDRh1bP9j7X0//o34pDoOKkHI2x77vEOaN8Cz5wM/5wGa1/pCssIxcGU7HGcOOF0Djrin7iurO424rXL4L5p8Mdi+EM+PPr9PmvahxLuQw4h/6kncR18EGppKe7Dv0/hyy+l0JI2X/4zWm/8Nfry5Rhr19Lxl7/ReMJJyB6VDG1vNxMpH0ojDlqOQWyzByyINzhTjTj0kdkTg8769TbiAKoF6yb4iTkHdl0ZHild/6bbH3cIFs7K5aVjizHV7mPXTPDz8Fml3HrjrqwcC5awMBUTXTH4YnQNTWILTgOOf8nOE43dGO4q1nno3JF8vnfOYJ5jQwJj9WrqZh9K9P0Pvt6J/4vYGVoZKrRV2YnN/rBpLkxLzwLYhQzq7wOCNKFmEbYR6VFRYunQuBrOnZsslhxqhP870Cbj6kTFh/DQAXD5uj7LuoYKrhl74fpPZnqC4H+eIPrSy8kbo1GMio1E3nwL73HHAmC1tQ/K++0btjdtBBUKjm1FKODMjxOrdqUhIOnZNtPTp+3PfKaZM8NxJTVRHHr/T4X+asHT7W/OcXDnNeOIuhVibpXNZR4ueqCSl48t5tP98og7FFBSz9QdgkXTs9h9RTtTVwY57aktOHQLKaAtS2P1xACGQ2HhrFz2XdiS8QGzI+y8sXYtzT+5mMAvryFw0YU7YIZvFnZ65EMFoTAg9+ur/8DWxX0fM/W07VtLp55ZCgQ0rUvetPTh1AeHTHj6Fe9v3zoywGxsJPjwI3Tcex/66tV9Htt29720/fLatPtkKExsQXc3X0/d0O2HsB8KpoJQBZYJ3okhW1uj34b13oZ9+6uwj3qrPrmkfcBn9o+nThtOe0AjllBaWjM5i2tvn8T8A/KJu9S0Rrxsc5jf/XoNP3zaZrUUwMEfN7P3l224Y5KoS6Al2vsfP7OUDWM8adeX7gVnqCAjETru/KPNrPkdx05DPlTILoW88fR705pxePrEvo856GZbyX5b4UgvUxbXQ9y7/jlqQ7XdG5vWpU+oShNaKrZ9DRkQefc96vbZj7Zbb6P9D3fScOxxtN5wY0rNudncQt0xxxH8/e8zD6ZpqMOHd/86buyQr1fLtsWYhRC0fpJD1qzWRN3eQJD5u5AygmbhmRBCOCVCtbqO6vRYNau/b5ZE2DLJA1xb9zo27OJDqsmj6261qwqlN1xRk5//fSO5rXpXQrPzSKdhV70UNOmolsWRDZu4c/0nFB/YiNmrGlVih412KDQNY83atLv0detoPP2HVI8ZS83U3Wm/849JobpvE3Ya8qHEKU/byUanH4SKiWCDO4vLdj2Af42YTIPmwgRo2wxL0mfVAfDkwNnv2SWLA0LiNtI84AzAYX/s1TkKcaGwNFDA241LuPKjn9MUSai1jNwXHKmNMkDflAHbACscpuWSS+3EZjQKhoGMRAk/82wST4ZRVUXtzJkYS/pRMhLgO/WUrl/9F14IQ+aVS4RmkXNAS/cmQ6Ftfl6iOKk/g5neCMqEcY46IeYQGBqgWfgnB8ndv5WS02rw7d2Oe2woTdqzv/k6jx6cMb/6L7Ys3/h1Qc57aDOX3FvBrM9aUDJU1Uxf2obSjwGOOwR3rF7IuVvXMCHcxkRnC7lzmpMKy78O0i2px1GKUnNXRvVWGo49ntj8+RCPY7W00PGv+21+9m8hdsbIhxKFk+DKKlj9AivWv8azHetZklUIQlDp8fN6wSgEMCXYzKXzb2PE9D5i5SNnwVnvwBuXQ/1X9jah2sbVVwibPrLDOZNOgtJ9bfm2vPEw7Wx7v+pEvnMVUT2EKiWLAwX8ZdQ0JJKoEeXlDS9x3tTzbXKvubfaSdLO7lTNY485IiU5vl2Iffxx2pi7DIcJP/887gP2R0aj1B91DEQy1MH1QODKX6AO625mcs3YC/8VPyd4V//c7v1BuEwKj2vEkWMni6NVbmJbXdj0C72vIVMdSPqmJMMFNWeYDH9C5Z/nj2ZSTi2nNtYSVjVUVaJOjBFe7t8GI9e3IU9X0SKAktoYZz+ymT2XtuOISxRgfHmQ7y1o4q8/H4vV0zOXkkCHgWb2bcm9w6PkayFcPboos0aG8Z0ZscnEBqmLuk1wOHBOn45WWpqyK/jgg8hYLJk+NBol8tbbGNVb0UYMTznnm4ydhnyo4fAQmXwit2x+nnh2D8HhRCu5BFb6c7nGk8VNTaup7Kgk153L9KI9U1vlRx8Elyy3v2z9iSr3xl4XsKp0Tx6cdz11wqKjhwKRIQ1WNq7oWi8Xfg4f/NquPVedsOf5cMCvBjefHrEfLJor8zF9lbomSMGCjz6GbO6/zA6Ph8Dll9v/DzVC83rIG0v2lb/Aam4m/H8PD3ztaaD6LbRsg3iDRqzOSftnOaQz1orPBCRWRO028IpEqBJpphp9AWCA8r4P1YhyyT83gYBGOZxwrkDxGHi2bu9bRep3RQIRt4I3mt4Az/y8jZ6zuuOSMZvCTFvaxoqpARQLTFXwo8ercMcsTFWgZfDYARwjYjjS9FUIVeIpixL8ascbcueMvcj/9/1p9+lLl6blbhFOJ0b5up2GfCegqqMSTahk0iWRQiGsCq77+Jc4VAeKUHCrbm4/4A+M8I9IPWGwRjyBfP9wKl3eFOpaBYXh/h5fVF8BHHuf/TNYNJXDy+fDlk8BYdMOHPcABFLb/l0HHJBWoFd4vXhPOA5r2bNEH7gBoTmQRh8v3i4Xha+/ilG+Ft66HK11Hjg8CCOG3O0MPLPPJPzkU8m81IOCxGjS2PrA8B5rSLMWVVJ8Sh1Ck4TWegmt9CPjCq6yCIHpHUTWeWn/MjulksZhwtbhbv594Wia8xzkNesc92ots75ohZa+jfj2VHm0ZWl4ovGU8wXpY6yaCce9Wsuei1tRLZi4pgN3XKIr0JjvJK9NT2r6kWBztaiCT/cq4thIMMkjB8AUmJEdH9EVPp/dZ5CbvlvTMWkS8UWLUxgTZTyO1kl78S3CTkO+A5DjykXvh/nPEgKwiCXqxaNGlN9/dht3z9kGY5oBJb4SJudPYWXTiiSGQofq4IRx/SRcB4JoOzywL0Sa6XpxX/+OTed7+TpQk79eis9Hzt//RsvlP7O9c11HuFx4jj8SbdGV1N7XgIz2bLFPNVuOffeh4LFHabn852hNzxGY0mrTzSa0HPX3Hqf51bftLpptxgCjt6aC0CRCgH9iGP9EuzrCjIJQBJ5xEdoXZ0OvZ9fCGTk8dXoputNeY3O+kyfOsF//Z33RmmGyzsTntpnx9iyN1myNYfVpNFVJf7WmgILGOMUNyec4LChpiPPprByK6+MolmTVJD8hn0ZHlsay3bPxKDpHr0yTLJcQ3ZQ+GT+kEKJP1Sn/RRcSfubZZOUglwvXAQegjRq149c3xNiZ7NwBKPIWMTFvIprI/Jx0WiZHN1Qws60OpJ0Cqw3XUhMaukaclmgzIwNluFQXAoEqVPLdBVw381fskjPWNqY1S2DDexDtR6AiHVY8CUYn818C0oBwI3L1a0Q/+ojo+x8klX95jzqSkgUfk339dQSuupKCF54j50g/jQ/WIaO9JReSzUvWHbdT9NyzhB76P6Iffoh/QhtKjwqS6BYXDa8UJrz5HQ/FbYIF4XVemj/Mpe2LLKLVLhrfLAJFovlNm69cSQ5BvHJ8SZcR70TcpfDKsckiEt2QKP7MvCzd/nD6UIcFVI1w49LloNKgiiRjyaMA9v6ylYrRHtqyNOYdWMCHswv5cq8chJTEDI1Vi4djhhWsuMDSBUZQpfH1Qlt7dRsx4PVbFu79v5dxtzZqFAXPPo1j991so+924zvtVPL/ee82r+2/iSHxyIUQRwB/wxYOf0BKecdQjPttxvUzf8WdX/6BFQ1fYUqTnreQIi28ps5ZNetQgNcKRvHo8IkoKOiZ1IMGiYZwA1d8eDkRM4JhGV2G/MLdLmKv4hnQutnmc2mrAkW1yyJn3wb7XTXwSRrX2pqjvRCrlDSdcBUoTvsmMQxy/vZXvAmFFrWkBP+FF3QdH7/2eMygSl9ecPbNN+E/6ywAQo8+ZjcEtWi0LAtgtGkIRWK0anydvolWGKP+hWLMoGobJ0USXGrTG8S3unANj5F/aBPN7+cTq3WBIomqGi256T3F5jxHsncs7MqZwF5tOIsMGl9Jrb7QsnWcpRHCKwP09flNXR3EVDIfYZH6yfX3ONRMmPOhXf20+4rVvD0nH80SlFZFGFsRwmlA7aJhOPJ0m3Cs2TGAUdMhQRE8mDOiUeqPPZ6Cxx5JSoj3hHPaNIrefANpGOgbN2KsWoW+eo0tIr6N4cz/FrbbkAshVOAe4DBgC/CFEOIVKeWq7R372wy/M8Bv97uNlmgzDeF63q/6gLmb3sEwo8xoq+eCravxJtjXjmvYxGsFo5GBXEoDIwc30V/GQ9t6+/8nvw9TZwPw5JonCOkhrETCSSKJW3HuW34vs4bNQv3PUXYNec8Y5oc3Qck02CWzUHIShu1p17vr3R2tVlzQ9GY20ohDjyxB689+jnPatJQkkl6xiVhFC4gMJZBI8HiTBCRkIvbd+GZhImwxELooEE6bydAzzuYqb18cILxqsC393fPEqxOCHomEprQEpipQTUnTewVk7d2Gb0KI/CMaCW/28NCIiSwbmYU3bBD2pRrznFaduFPQEVAZs2sj/okhUCVhVeMLqxB1tsqIzwzUHiLSuQc325JzfaDzaNXK/CnpLifOWPr4eSYkvTtJOOK9Jvu9QIDhgOZshdw2C73JuZ0dnKLftaTAsjBWr6bpnHMpevutjIfJeJzmn/yU6Lx5dkOZlKhjRlPw1FOoed8eNsSh8MhnAuullBsBhBBPAccD/9OGvBO57jxy3XnsmjeRn679CFa9nHKMIRT2iASZc8g1KH0IRCShoxbu6uVpPDcHnrP/e4Avl1WlU9nqSSbAiuhhGqs+obh1cyrBvh6Gz/6RZMitYJD2P/+FyAsv2jW5+fkoPj+uA/bHf85ZqL5CaI91lS5GNgeQabwZaVmEX3yRrMsu7doW+3IRjcf/AO+EvjhJBHlPP4VwdlfduA47lMiTT3XpZXYe1yeEpPD4erSAQacYUtbMNoQCoRWDMeb2Q0PXFD6fkU9jgZPdl7WzebSH148uIeRTyWo3+MHLNeyzUNC+sJsH/hi1kZK9Y5SP9fDFzMIkxkDVsNhtWQu3X7crp0fXMLE1jJKo8vFbBt+jhtg4FWWspH2Zn9jnWSAkjkKdSFXfhrz36tNtc8YG/iYYdwjen13AF3vnopqS/ec3sf+CZtTOpiUJIg7CDSsnOJm8Nr4d70nb9wjQ15Wjr1+PY1wa/Vyg4+57iM6bB9Fo11fQWFdO69VXk//Qg9s879eNoTDkI4CqHr9vAVIEIIUQFwEXAZSVlQ3BtN9CBIYhhYro1T7vsgzO2vIV6iuXwhmvgjfhCRgx+OBGWPQALdLAKp1F/pH/gPxdU414L0wLtXDf2o8JCpUPCkayxptDTLGpR70V822Ww3QIdYvbSsui8aST0cvX2wo+gNnahgnoa9YQfuJJil5+BXXFX2DV86BoyJJ9QFkNvWt24nFkezefi5SSpjNstshwuQ9nUYx4vbNHuV7itlI1wvffj3rxT3BOnw6A/7xzbUM+CLhHRVC9ZpcRlyY0v1OAXp/adNXzmZLOhKwZ7+cfl+2ClRjrrSOK7JMSrextOQ6ePL0UzZDMWNyde3CaktGbwljCREhYvkceIZ+GL2Qw9atm6ktcRPIFB66swdnrISsAd+J7k7V7kPatTmLVTj6LlTCy1kR4LGQkueJlMCZwoMcZCvzpyrHUlri74vwvnDCMNRP8/OSBbllDBZv73B+K2yRaclsagAZvxFW/TmDvNrQcnfB6D+G1RVgtLRmPD/3nP6nVTbpO9IMPkZEIwvM1JGaHAENhyDNx4SRvkPJ+4H6waWyHYN5vHSp3PYLiz++hZ6W1xE4sFOgRZNUCjDvz+OykR/g0VkegYi57Vi/liVFTqHTbLfsj3ruYX/rGkdrikIzOP4pfmhzXsInjeu7cnKFjUvPAxBO6fo19NBejYlOXEU9CPI5lmrTf9wi5f3oQjre9F9f69fDSkanr8Xpxzz6k63ezshLZmQS1BPFaF4rfRMawk5WdPN6mSfT1N4i9/wE5f7kL77HHoo0ebVckDKKd2pGvJ0m0dSwL2CIUacg+Npd5+OiAPPJadI55oz7Jm4w5BXdfNgZL6/W17/VrZ/KypyEHKGgy2TgGRleGGF1p5xfCHpXq4R7aslyMqg9iJDjjJVBJAPMTL651KkKCe3SE7H3aaDpaR1Hj7B3rQBxq12dHNntomZsHZt/x5O3xcZfvlkV9kSspWRt3qayanEVVqZuRW7qNosCOo3fOFfIqxJ0Cf/clLJkAACAASURBVNDEYUDELYi6FHwhE2dKkZdEDRiYYTXlb9S9/k6aYHu2wL7NBCaHMYIq4TU+rKBG1t51aC/OgtcllB4CFyQzIspMJapSInX9f8qQbwF6BnZLgT60z76bqA/XU9VRyTDfMIanqQWXUnLL+qeYULYbl1etACnxyORKBIFt1Me9/BMeGj+LJqeH10ftjv2FVTiycRM/3roW1fp4wOvq64bt7M8Rws51tn+WR+zV51GKPiZwyU8x1pX3reBjmsTmzkva5Bg3Dt8PzyD81NPdhtrjwXnQgThmdmuSWimv8gIrmOHrKCUyEqH1+hvwHHkkRnl5VwPRQGF2aFi6QHHYFx1e68vI2OQJmyyekYvuVBhdGWHSmiCOBAHUV1OzMnKQ9EZzngND6eYTEYBTl0xaG2X1BLftle+WTcjv5txHqgh01CEk1OUW4ft+K7ftMZ2z/1xLYUMMzERTzwYv8ToXow5qQi0xURVJp2fgHhUhe1YrbZ/k9hvb3lZjXj7e10Wu1RNSSDbs4mPElmha+rC2gMJXu3kSx8KWEV5W7JZH1K1SWB9hzvv17L041GWg9ZkxGmdI3M868DckmW7bbisSISx7MClAsfCOCROrddL8dgHSEmAJolVuQssDFB5fj1L1IdwsutagTP8Z7kMPJfLiSyn9DcqY0WBFgKxt+JS+fgyFIf8CGC+EGANUA6cD/SgsfPsRM6I8ueYJ3tv8LiEjhJQSl+bCtEymFEzlVzNvwKV1xy43tVcQjAdZkDuchdnF7NNayy+qvkr7Cl1kRHlg9VwaHG50ofBFVhErvbmcW7226/V6KNAzlK04IGdmNVasmuZ3c2n52Rrchx2G8HiQodTKlK4xclO1QLN/+xvcc2YTfuZZ9A0bMTdvJvbOu9TuPo2sK6/Ad+65OMaPA01LacjoE7EYxuZKWm/4ddrGor4Q2egha2YbUpUIJTU90BPFjXGmrmhnxZQsHjq3jHMeqWLKqg4sBaJpjFgmeEMmT506groiJzGPRn5TnMPea2CXTWH2/SzEiik+No/K4pZb1+GOdS/IahREP/dzkreC0cVh9FYPVhdJuMDsUNECJoou0cMamt9AaHa0zDchRNunORnpfGNOwf3nl3H+w1V4I4NnrMpt1XHErZTySdWEnDY9bfhEANkdNr0tQrB6QhZf7Z7XlSOoKvPz+FlevOGNTF0TIeZUeHT3cVTs4mH6JY3sX1HPLp/HiG7wInXFfkOyQKL2qPABxQOtL+cllzcaCmZI0LE0QM6+9tuRTJRVysV/JzcHcs+16f8bX8vDbPeDNLDKy6mZsheq36DgghFoZ/4bskaAf/ukEXcUtrtWS0ppAJcBbwOrgWeklCu3d9xvMqSU3PLpzby68VXa9XZMaWJhETEixK04yxqW8ocv7khi9DOl2VXSZCoqX2QXo2WwJp03Q5EeZUQ8zPGNm7iucsmQGvGUOYX9o7ig4JgWCo5aT/SD1+1Xywyum/B4CFx8cZqxBO6DD8Z1yCGYGzYgOzrANJEt/9/eeYfHUV2N+70z29W7ZFluso17xYVqA8bYFGPIB4QkENqPL0CIQychoYfwhUASAiQEAgktlGCajYOpDuBu3LBxEbZkWb2tyvadub8/ZtWsVXNbC837PH4s7dyZOXs1c+bMuafU0fDQw3heeBEhBIn33B31uGonmXUyHEYkJRrp1Z2g5uWB3d4hG1aGFarezSRYYUPq4Bzm6zxIGrjm7/s45/1y4jwar1yay+LzswhaFUbtaOy2nCwAuqQxycqqE1LYkx9H8SAnmyYm8qefDWPD5CSjWcSIBKavO6A4lSLJmF/NwJOqOFWWkjrFTdb3y7Flt3VvCaqXplP24gCq3s6k7MUBNG6MN96wFMPNElUkAfVJVraPTeTZKwcRtPQupA9g5ho3in7AXrrEGtIZ93Vjp1a+BBw+nbAq2DreUOKpQT+TG6rIDnkJWRQ+OCsLr0Owc7id475xM+/tUnI/8eNqDJE0w03WJeVYEo0Hv45g9bRkrntiPNc9MZ7bHxrF/lACmj+KStNFuySk5mu9+R+Aaoes79Uy4PJ9JE5rdocJtCYLFY+XU3HOPCpmTMR9bg7aS5cZobfHEIcljlxK+T7w/uE4Vl9gZ91OCty7CenRV/p1qbO+Yh1/+uoPLJpyE0IIhiYNw6rY8GG4KoKqhTJ7HLmBzq3dZppdLp1x4M14KBGwzRe2NVmSc0kRureY+g2JJIxrQo3XCLst1K9NIljmQPp81D/2GFhUXAsWdDhW4+8f7eCakT4fjY/9gfgfX07CVVdiHT6c+vvuRysvwzp2LEm3Xo6+eyt1D/wdvbGN/9JqxX7iiVjS0xFxccimjk08RGIi2atXEt6/HzQN9z33Evjwo5btwUYL1UsywaLz3I/ymF9VTXpVAGuU56kA5n5UzdyPWvtjaqpg5YxkUqoD1GZ0fFi0qyWjCJDSKDjVPE4RBO2Cf30/l1E7GinJdTJ5Yz22cOt+ruM82DKDLS4gI6dMkjqnhvKXcmj+62oNxiKtjLTjadiQRNinEjfaEzXhptklEVYFJ6yqY/XMFH53Wz7X/6WQVHfP34oSG8Pc+ORenr1qEF6XihSC9Jog179WQOq0WmRI4N3jQqtrjRlv/nZTN/koy7JiC2pcvf9rZteXGH50JBqCzxJy2Hx8HCHZHPAuCEuVrVXpAIxQG1qqJ4Ytgm1jE4w5VxQa42z8etR0fsOO6JE5XZQebvdntED8eA+OwT6EEAiLJNyoEqyw0rAuifBmBd/Oj8nc+RbqFW9D/pwez92RxOzZeQC76nbx6PpHKI9kWCpCIdmWzA9G/5Azh5xFSVMJb+1ezKf7PiYku15sswgrvz7hbiZnGuVgt1Rt5u4vf9US2z2psYpf7tmATeooRE/KaEunqdRAGEPZd50b2XtkRAO0jYrUQ4KaD9IIltmNM1gsJD38EPGXXtpu35LBQzt1nQwoLkK0rYTo3gevnAN1e5CKBQJ+3KsS8RWmIUMhbJMmkfbc31GSk6l/8Dc0Pf+P9tEGDgeOuWdiGzMGdVAezrPOoumfL9D4yO+RPh8arQ/DlTNSePGygaia5M7fFTCwpOOCl9ehsHhhNnFNGpk1QYoHOvlqajKN8SrT1rmpSrNROMzVUZn3AEtIR+hGIbSJm+v54b9KWlwr6QsqsWe1GgjN5Ye9IQvZz1lIaOrKHSKpT1N585wBxHs0TvmylpzyQGRL67UQBiozbWRWByP+e6OGuJocJHFmPe6P0ruudYNEOHXKkp2oQRiUVEfc+CZqlqchA4ph+neyr6bA3sFWSnNtpDl8zBxQRqLduI90wBu00Bi0YglBXY2THd4UPBYLOdVBRuwKtixshgWErQplA+z86cZhLX772x7dzbC93vauJYtO0vR64sd2bzS1fMM2deraqkjdB5XvZRI33Efi6SlwYwH89wFY9yQEGmHgTJj/OORM6vG5ekNnPTtNRd6GwvpCFn36U3qXyNw1KiqLpt7E7DwjYuOFbf9g8e43W5T5cK+bi8sLGOxvQiDJDna+uNiZIvcqFn47dAqbE9IjAyUuLcTI5JHcW1sD299AaVOJru0i58ESrLJS/X46MhhRj1YralYW0u/DcdZZOM85m7pFN6FXVXXYV83NJXvt6jZfTMJT44zX1TbuI4kgVOtEcVmxJGow708w5SpkMEjdop/j+2A5wmZDBoMIh8NoChAIIBwOhNNJ2ssvUvODH6G73RRnWcgtC6BIuPfXI6nINtYvLnuxmJlr6g7MosdvV3j8hiEUDnUhDyi9m1IT4JTPa3n3/OyDm8Q2WkIN6/zi4d1kVAWxhSXp51VizzYU+TeuZO7Ln2ZY07ogjMLCt8s547PqTg8dsAmeu2IQX49LRNUkP3v8W4bv7XhNRb+Wugu8BOcwD8knuxFWiQxB3ZcpaE2qEcYpBRKj9+cHczNIrA/xP4vLGFjibze/Podg4yQXEolq11mQvwdrZEW4eWqar1Fdg5LFOdCoHtCEQid1bg2OvAAI2GuP48O0POr8Ni7/awVEQjGlDs4hPlJm19HTFI2uaG6+Vb0sncwLmmDkubBrafvmLLZ4+MkmSD38jU46U+Rm0aw2/GbNA4dViQNoaDyx6c9kOjMpbNiLJjVURUWPRF0UuJJ5aNjx2LUwD+9exdvpQ5hXsw+71DvcSs0We7RbbHtcmyw0IfBabOz0VXChPYCYfA7Tsqdz3cQbSClcC6/OA3loCt2SHMaaHiRYGvE9hkJo+/cD4H35Fbyv/MtYzDwA4XSS+KsDSuSWbzZKBhy4BiAl0h9G82tYHAFYdiNkjEbknUDqX55CKy0jvHcv3iVL8L72ekuYpPR4DLfPXb8iY9n71D/wAAVNa8iqrMYWlgTaLNStmJXG1A1u7G36YWrCKDJVOKSjEk9yh5iysZ7hBY08cI8bh0/jjt+OYdy2RiZuacDrUll1QiqlA6In6ahhHWtI4ncaikazKDxy63DmLq/k+K/qCZQnMSizGl0V3Jc/Da8ayQCNPC/fOT+b4d96GFwc/YFvD0omb6pny8QkrCG9XThgu79DLz5t2WrTSD7VjWKVyDBUL8kk1GCJFCgzHg3vz89k+Zx0pq+r55LXS1qThNrK6JfMXGNYx40JCvtcSeQPNGK9m6/F5v9DFXbUJkH7iriSlNPrcOQFWpTz0KCHa8sMt4p+MYTKbWgeFVtGCEtSLxbUu0FE/JyufK8RObXzvY6N0sN++PL37auJbngWvnjY6I078lyY90dwHL6IGNMij/DS9hd4fddrR+z4CgqKUAjLMBZhQUfHoTqQSHyRp7mq6wgkuf4m7izaSFbAhyXyYAmisDUhneyUfHLLtiBDXmOFRggeyBvL+ra1z6OgCpVMVyZPzXkaTddYX7EOX/FaTl52M3bav0r2hJDbgr/SSsOKtB5OgIIlP5+4K36MDAQQdhvO+fNRs7Jgz8dG+7u2DaAjBEpt1HyQTtKJbuKO88HYi4xOTG0om3I8ekVFx3NareRs3oiSlETVFx/jvfQKLDosPi+LD8/KbPnCM9bU8v3XSiNRbJLqNBtPXTcUd7IVoUs0q4Ldr3Hczkaufr4YNSRbHqgS2D3cxaB9fhxBHU0xfOmvXTSAlSe1mRspUTTJiN0ervzHPgpGxPPclYPaNW2wBDR0i+D2vRtRJPxh6AR8lvap/ELTOeWLWi59PXqEr6bA5yen8doluYzb2sBVzxXhDPb2Ho9urytxIVJOqUdxagQrbTSsSWrnj/fbFW7/7Wh+8GoJ09a5jaJbUY5Mm6PrgHRI8n5YEtVidq9MxHNAHRlbVoC0+dUt6wiHm57cC5pXoI6cAVXfQCBKwbkB0+DatcbPr18M299ov93qgltKwZHUK9lMi7wL/GE//971RvcDDwEdHT0SpRKWYWzCxrwhZ+ML+1hWuBQALWL9FbmSuG7ULHIDTVxXvJ14KXk9YzBrUnIYlzaR4TmjcRV+TpPFyr7BJ7HVXwWdLLw2o0kNd8DN0j1LeHXHK0akjdT5y5SFXL0rnrlVL6FEjMjuLmKpQ+OGBHx7XF0PbIuiYJt7JvX33mf4zYWg/t77Sfz9I+jbviIx1IiwQLhexbMjzrCmBgQI11qQYQX3ymT8JXZYuRFX3DIc885CKyyi/jcPoVdWdn/+R55o6TJ21odVfDY7nZBdASFYMyOVDZOTyCv24XNZKM+yYwvpzFpRTXGek4vfKGVAmR8linUpgPxvvTQHiqg6qLrkkjdK+WpKMn6nihrSSXaHuOEve8mpMP5OkzbVc8tjBbxweR4VWQ7sfo0Za2pJaNII7EqmcSDIwVGcH6qCx6USUMGmdZRHUwUrTzDezhIaQzh6rcSbv1VHmjQb906dSX2ijfw9HgoWxqPokulr61iwpILqdBsjd3uYvKmBTgJnOhxdAWQY/EUOnEP9RqvYL5IJlDhQHDqWhI7rUPY8f5eLl4dKt9e/BDUpDs57Bv4WpYuWUCF7ovFzXVFHJQ5GOYz3b4QLXzh0gTEtcgC+dX/LzZ8tOuxule7ITxpOYf1etAMLVvcCgYKM0oklGioqdosdb7h9V/EHf72DtNogSlyIpJPqcOYF0fwC1SmjBmZ4truoX5ka9Ryd+fGlIoywvU6ut7jxjdgyA7hXpLYkcwhVR2rGwpmmQFgFewjDHz9sGHppqRHj3klykGXsGLKWfwBAydB8CLY+7Lx2heevGMj2UYnoFsGIXY1MW1/PjtEJWEM6J66qY2SBB52DSS0Hn13hH5fnsWVCIt9/rYQT1tRhC7X/7hKjbskLPxqIQHD18/sQQEiFD+ZmsvScrI5aRZcM3eshuzKI16Fy+cvFkXkFiyZ584IcVsw21kqufraIqRvro1rFkUjFqHT2ncMCXr1kAKtnpqIpAprfJqQkvSrI7b8vYP3UZGb9t6Z3cc2KNAqMjW6icnEmWqO1dbFS0Tt0WYqf0EDi1Aa6qBJ9xOhgrY+9BHa+295Hbo2D//0K0kfCZ/fDZ/dEP5gzDe7ofL0jGuZiZxfU+mu58j+XH3VFfrhobjbQnfw21Q5Stu8YJCVP/XRr+5s2kjGXdKIb10hvyyusHoJQjdUI4Tsw4SSS3LN3kJO8/b62PXZbFAN0oxCF7DSRRRdQMNTGyD09LO4kAKsN+8knkfLUk5SPGdfrbNBDQQLVqVZUTZJa37WPNmgRRs5BRDF+dkoa752XhWZtTgIytIfNrzFsj4cdoxJa6rpYQjqjdjbh8mik1Ab5eE4GaqST/f337CDB09FI0IGmeJWEJmNb2xmX0JKNGk0ZBy2CRY+OBUv7rQ5vmNM/qSJvv5/xXzd2aZEfiIxEzVgzA4Sq7ZGol85R4sJkXVrePAWx58RbYf1fIeiBAVPh7CdgYKTc1Lq/wtLrou+XNAhuKurVqUzXShekOlIZmXIcO+t2xFqUg0IiSbYl4w17URUVgYIqFLxhL1pkAdEiLKTYU2gINrRzwzh9Og2JKkkNbW54adyk9SuTCZQ6iBvdhLBIvAUuI7U9irK1n3wSgS9X8toluVz7bBEuj4YjqBNUjdZmPbrnOlHiEqhNUalNs0BPFbkEgkECn3+B+6abDy1E5yAQQEZtz2rBNMeR+4XhVD718xrq0mysmplK0CqI84TxxFsI2hWc3jBzPqzko7mZWEMSVZMU5Mdx2Yv7GFroY9bnNRQMjyfOEyY+ihIH0CyC2lQriU1R2u4RKXfbHA97ANawZOg+H3uHGWWHs8r9XPHPYgaWGBapokX27UKRd3xrMxKTQpU9q+JYbnPwUt4Ebtm3pd2fNWYlxE/6Fcx9xDAUDmwuPvlqeP+G6KnEJ9522EQwLfII3pCXu774Bd821/buQzhUB9dP+imjUkfhCXkZnDiYxmAjf9/6DGvKV6MIhVNyT+WyMT/mfz+8Bm/YS1p1kCv/uY/BRT6ElChSIA72WrBayS3cg97YyM9X30ZpbRHHb3AzbI+H8VsbSGjqqrtN94RUWDstDodfZ+qmLmq/dIbFAjYbeL3djz0IDqUIVU+OHbLA12MTWXFyGnuHuvjdL7fz6SmpLFmQg6pJnF4NXYErnyticLEfV0B2Ko8O+B0KDr/eqcvowAXJA7f9+8IcPjkjA7tf48G7d+DyaC3We/NcHN45MY6mYxjrq6YnUZ7rZMF/ykhMM/zlzmFenIP9La8RR1Wpu3Lg9i7KS+14F167oL0yH3Eu/PC9Xp/KdK30ELffzd6GPVR7q/jzpsdjLU63KEIl1ZHCX+c8g01tLcmq6Rqry1axsvRL4qxxzB1yFsOTR7C2bA2Prvot99y1mfjGcI9egbu7sRNuvJGkO28HYHnhBzyz9emWXqSP/2wL1kOoLKApsGu4URVqSFEQZ+Dgrlf7BQsJvPX2wQtyDKIpRsp9nCeMPSgpybaRXtvaEDkah6pgJfDaRQNYMTud+UvLOOf9qi6zjg8d2eanVsl9DoHL3/F76hZJwmQ3SRPbJ//0RLEfqAp7/DBIHAE37+p6jBYy3C9NFTDpSkg7uBhz07XSQ5IdyUx2GJmYL2z/J/XBg+hleZSwKBYGxucxPXsGld6Klu5CIS3ErStuprhxH2FptHn7pPgTrhhzJRMyJnJZ2RDswU0dlHhnN3nYIijNtjOoTUyyBEpz7Gz83kR+ElHiAHMGn8nOuh18VvwpFsWKJ85KckPPy802H1sXhix1KRZyS0PEe/QOSTu9IfDOu+0zTY4wR9JKb0bVIbWudW5zy7t3Ox2qTAIYsbuJnBIfp66sO+Lfsa3Ebc/l8suocyzCAs+6FDzrUhD2MM6RTSRMaEI9oBpthzXkMARK7Lj/m4oSpxE/ugnXKG/PlPkNW7ofo1phxo09ONjBYVrkXfDW7jd5fttzsRYjKgnWBAJaAIlESokiFBYOv5AFw89n0Sc3UuPvuBouEKiKymkflHLK5zX8Z14W28YkEOcNc8bH1Zywus6IpT7gkqhKs3H3fcchdElCY5iAQ21JibYqVp6Z+xypjlSklATWrCW4bh3rRll4RV9NlbcKlzfM6Z9UM/+Dyp4pY0VpqaEBEql1TI461jkaijyW9MXvpyYEcQz148jzY0sPIqzQHDDm2+cwIqYicfHCqpM2r7oly7Zz7HBvJzXNjwCma+UgCOthLnz3/FiL0WPsqp0xqWPZVLWxywiW/AIP5dl2fE61JSHFFtA48cta5n1YhcOnYQ9JwqoRl/zk9UPZPSI+6rEcusqd/6gnp8SD3uSBpib25Tl59KZ8gvbWhR9bQOfElTWM2uVhy3ijb+WJq+paaoFEw+tQWDs9hWR3iHHbGrFoh36tdp+E/t2gbYjhkfFbfwcQOmpiGN1nQQbbL1IKq07qmTU4cgNwb+Sq2bcFnpvYOuiCV2HiJUdRYNO1gtvv5v4v76WgcXe7z4fah/LDiZcxKXMyNkv7tl+vfPPS0RTxkAlogW6VOMC3+S5UTbbLKgzaVT6bnc6aGSmcuKqWkbs9VGbYWTE7jep0e6fHEv4g6Rv3tIvsWzo/k6C1vcoI2hVWzErn67EJjNnRhKYInv5/g5m/rIIZ66O7r5rjolVNYg/q3PKHb8ms6nlvyajyHtLefYfGeEFjvEpio2aEFEayU13+/jMH3SIVtPqOrf6asWUH4Fdt3IKDJrQq9WOMfqHI15au4cG190fdtjewt9Nt31mEQDuwVRmAAF+chY/nZPJxm+qclpDO1K/qGbutgfokK1+cnEpFlgNbUOfiSD2NtpQOcNA2yDe1JsiAMj8D9ns55z+tRbQWvlvO0nkZnVqKQZtC2KYQBgJ2hWeuGcxdv90dZeR3k0OxoBsSLewdYkNG/g6KJkmrCZFdHia5oe+5qo429vkLUB74S/cDjxH6hSLvL4paQcGm2vBrB+mzi7KyYwvo3PZoAelVQRxBnbACp35Ry9Lvj2DCukryv3F32Gfc1gZWzE5HSLjin/uYuLkBTRU4Ah0VyDn/qaIhQSWpsX1oS9Aq+PykNtmjiqA8y05dspUUd+8WT/saASvUphpVdjJqwig6UcsDdIYEHAEdRTciW+I8GhO2+hAykuuFaZV3RfJfniQuSn39Y5nvvCLf494TaxGOGjr6wSvxTjj1v9VkVgZa0sstOhDUmff6bt64MBs9EGDEnvbx2RctLmPncfFM3lzPxC0NRsJLOPorqTUkUcPG67+uGBEylrCkLtnKN6MT2uVEC2k0RvguIzEaFmdXhFsSHPcMtWELSAaVhHqkgAWQVqMxSvdRk6ySVxLCEjaVd1Ryc8ltW1K5j3IYKvQe25Q27Y+1CH0LKRGRRUVrQGfGWneHGiHGOCgd4OSJnw7jrQXZ7TYJCWd8Us2p/62Nvm/bsYAqW0MO1ZDxS0ZVkBue2stlL+1vCRlMbAiRXtMzH7k84P+ejD1WaC79qkrj39DCIL1tyiaAtDqd/MIQjqCpxKPhvPqq74QSh35gkU/JjFKdzKQjEcv3hy/vZ98gJ7WpNsZsbySnLLqFr0hJwKEStCt8elo6M9fUkVNhRKAIYHCxr11D4e5QAHFAKr89JJmywc36KUnsyY/jqueLe6aQLBYakm0UZqs4fRpDi3xY27wRtFWJEiMCrflGiLXCi3p+ASGrghT0rF9oG5ofkiYRhg0j6+3FWNJ6WH65j/CdV+QuWy9KrfZDBArzQiP4WNsOwMy1dZy8qq7LfXQBdclWyrONaBZdga3jE8mpqGrZXpZtpybVyvivGzvEjneWKRpNidlDkuPXu/nhv0pIq4viGxcCrFacCxagTJnAp7mNrKpYRZG1AZ/VeDgdv66O85ZWkFoboilOpWCokxWz0inNdeKNt3DN3wr58uRUFiypILfEjxTG4qrTp2E5cv2ue4XPLghaxUGVpY31wynmDBtG9rKlqPHRQ2i/C3znFTnAv897i/9574JYi3FMYlEsLFd2GpEnmt6p+aYLjM46wmgg8NR1Q1t814oO1lCr9R2yCJafmUnArjCiYDfWkMQall12OOqKJHsStoWnob/4Dkr4ACtfSggGaXz3be6aso2GJivENW80zrR+Wgrrp6W0jod2C7sfnZlBcZ6T/7s9kcT6ELaQTn2ChdNW1HDBO+W9lLZzDnqRUYLfqWA7qNrifQRFQSQlYTvjdELbv0Hfvr3Toakv/hPn6ae3+0yrqiL09deoOTlYR40CQAaDoKoI9cgWETgW6BeK3KbaeOPcN7npo5+z318ca3GOKUJ6EFTQVBVQ2TfIyZBCX7vFk6AKT14/hKRGjcYECztHxreEtYFhkY/7ugFNgdpUG69fmMP+PCMn+r5fH8esFdUM2+tlYLGPeF+b3qH0TLGNOvEcVq97nZThNobtCWAPdiwKpUuNKRvr+SxSi7tTokTmFA5pbaLckNTakefLE1MPmyLvrRJvTuiRiqB6VCaTvq78zi5oJf3xD7jmnomS1Notp+bGn+Ff/FaHsc6LL+qgL60n7gAADahJREFUxAHUjAzU005r95mwdR4j/l2jX2Z21nhruHL55TE7/7FMSk2AXz5cgDWkYwtJFl+Qy8enpRiKu4vCE4omsYZ0gjYFdIm0dFQ7o75p5Oz3KxhYqeFTwyTXa10qJwkIh4MN8wfhqTKqy+UVBxlcFEQBPE6Vv1+Vx87jEpACXJ4wngRrF0fsHXa/xh9v2dbyu0b7hgu9VczRxje/pfQ7LBbsZ84h+cEHsGRnd9gsdZ2G//sdTc88a/RitduJ/39Xk3jHHYgDS8X2I8wU/S4IhUJ8b+nCWItxzODwaRy/ro5Nk5JoSrAc/pqgEkRYZ9GTexle4IlagVFidAR65O6JnPJxIVI3nNVxHo1Jm41Stnc+NJqm+DbyddFsMdEdZMx2N2k1xuLt+mnpVGR3sX4iJWO3NXD5i8UkNOloKpRmWykeaCWjMkxuWRBXZB1YAB6HgiOg96qhQliB7aPjmbCtqec7dYbd3uJmOqZRVWyzZpH+3LMI6+F76PYX+n2KfldYrVbeXbi03Wdr96/lwfX3xUii2OJ3qnxxSvqRWyUTIK0KT14/lPOWlHPiylpcPh1dMRJWdBW8TpVHbs5n6J5acooDWIMaW8Yn8N65mbiTqnD6tPZKHDpV4g5viLOWl6BosqX2yGmflrPyhAz2DUnouIOUxDdpLHi3jC3jnUansTbHLh9gpTzXBlKSVR5iSGGQ98/O4twl5T1udKwLoy74qxfnMuGenT2fu84IBLDPPZPA8g8P/Vg9IVolSZsN2xlnEPzoIwhFFqYddpDgnDsX27TjsU2dgm3SpKMjYz/CtMgPghpfDQ9/8hA7Q32zo1B3CMTRbXsX6TA/cWsjWRV+yrIdfDMqnpv+tIek+iCLLxjAxslJ6IrA4dfwOVUSGsI0JnbztiAl6JIL3ynGHugYfhK0CpZ9fyw/n3IznxZ/yq66HTikhclLd3Lq8hLiPRo7h9uozrCiK0ZdXaGoIARSM9q3WexOPClJfDlKcNfDBe3CHKOKhNEIKaSCLdIB7rA8L51O0LSjYpGrQ4agV1QgfW2afNjtOOacQdrfngYM14hWVIRWWYl19GiUxMQjLld/wHStHGE0TePJr/7MFyWfEyTIYNcQ9nuLCdG30smdqpOwHiYkYyt3XGOIiVsaKM5zUpbjIGxt4xftwoXSFktIZ/aKCjKqoncVksCZz68gw5XZ/nOfj8rnn2bfOy9SK7zUpSjYk9LJHn8Cw+dcQqChlj0r3gIJw2Yv5P03H+HDSYIL3i5j5pq6lsYO0fziIRXQwSIP4wuP3Y5w2JGNTUe+L6kQpD7zNMJqw/3LX6JVVoGi4LrwApIfuB/hdHZ/DJODxlTkMcQb9PK/H1xDvXbsNqloZlLGZLbVfE1Ij/0DSIQlUiVKFwDZWpSrM6UuJU6fxrV7cqncuqqTEwgufbXzMDcAn7sK1WrHFtfRopRSUv/bh9n99vM8fGs+Iatg+jo3sz+rxuUJk1obateEGsCdqJLccGit79p9hdRU4i/7Ed633kbbt693O1utEA73qtmGfe6ZpP39WYSiIKVEr3OjxLkQ9s4rZJocPkwfeQxx2Vy8eN4rAPxn51Ke+uapGEvUOZuqNjIwbiD7PbEvbSAtnXT0ObB9epsxSkhi1SX2gM7lb1ZzwlN/5b9P3ELd3m86HCZ73AndyuBMzmj5WauuJlxcjP+TTwl8+BHh8jJkVTUDgEmb69k8IYm101NYOz0FJawT36Sx8F9FHP+NFyGhItVKdm34sClxxyUXk/ro7xFCENy2Da24uMdKWTgdOBecjyV/KI1/+BPS7+963+Rkkh98ANfC8xHNtW+EQE1NORxfxeQQMS3yGLOueA2PbXgUD57uBx8lEm1JNBzDLe6arXBbQGP2iho+PzkdXejc/kgBFVkO4nwawws8KFYb1rFjSX7tZZbdvhBvdWuD3MTcfOb97i1US/eRE6E9e6j+3kXolZWdjtEFfDYrjRWnphGwqwwp9PC9N0vJqA23H2i3QeAQ/dgJ8aS//BL2qVNbPgp+tZGqiy8GX5uSCg4HjtNPI7hxI7Kh0ei4pOskP/YornPPabdv0wsvoNfW4TznbERcHL4lS7CMGUPC9dehWEx771jhiLhWhBCPAOcBQeBb4EopZce6pgdgKvLeoUmNZd++z3PbniUsDcWgoHDt+J8wZ8iZhPUwz2/9Ox/uW47OEfaRxoK27hPdcC4nuUPM+7CaOY15fPv0HdT95SnGv7wKNdDeJSRcLtJfexXblMnUFe2kZvdmMkcfT2LusJ6d2u+ndOx48B+mqpJWa2tER0+w27HPm4u2txD79Bkk3HozakKUSBvA/9lnuH91N1pREcLhIO6KH5N4x+2gKIQ2b0b6/dgmT0Y4HIfnu5gcdY6UIp8LfCKlDAsh/g9ASnlHd/uZivzI8VXpBu5de/dB758TN4AyT2n3A48WUpJYH8ITb0FBMG67h+8tLiXdqyKcTjLeeQvLkCHU/mwRvjcXd9hduFwkPfgAcZdcfFCn97z6Gu5bbu3dThYLSQ//lvpbb4u+PSMDamq6X5i02Ui44XoSb72lV6eXfj/YbP06cea7SmeK/JD+0lLK5VLK5nfH1cDAQzmeyaEzZcBU3l24lFFJo7oc51JdLBx+AaKNx3ZI4hAem/XHwydL+tTuB3WHEHiSnSyacTv/vnAJv5h1P0OvXkTy7x4me80qLEOGAGAdO7ZTS9M6YsRBnz60o/chpnGX/QjHqacaSTpRUF0uBuwrJGvLJrI2rGNAwS7S316McLla9hEuF5ZhQ4m/7ie9Pr9wOEwl3s84bD5yIcR7wGtSyqiNLoUQ1wLXAgwaNGhqUVHRYTmvSc95eduL7Gss4spx15Adb6RF61Kn0ltJgjWBOJtRbepnH91AYVPhQZ/npJxTuGPGnQD8bcvTLNnzbq/2v2jkxYxLm4A7UEeiPZEJ6ROxql37snW3m4qTT0Wvr2+1dG02rOPGkvHuOy0LdL3Fu2wZdddc27PBQuBceD4pf34cIQQVs2YTLvi2/Ri7nYSf3kDizTd12F2rqMDz+htoJSXYT5iJc/78flUvxKR7Dtq1IoT4COhYDAHuklK+ExlzF3A8cKHswZPBdK0c+yx8+7wu/e2J1iSmZ03nq6oNpDnTueX42xgQPyDq2A8Kl/Hkpid6dN4HT3yICZkTux8YhXBhIe67fk3g88/BasV14QUk3XM3yiGUL5XhMGXTZiCjLXTa7Vjy80l5/I8IKVGHD0dpo3hD27ZT9T8XIUMh8PkQcXFYhueT/ua/Ucx4a5OD4IjFkQshfgz8BDhDSuntbjyYiryvUOQu4s7PbyOkhRibPo4ZOTOYN+xsFNG713ZvyMuVH1yOLxw9MUdF5arx13D6oDOIs8ZFHRNL9Pp6am/4KYH/fg66jjJwIAmLbsQ+bTrW4fld79vQgO/d9wiXlmKfMgX7abP7RVlVkyPDkVrsnAc8BsySUlZ1N74ZU5H3P4oaCvnN6gco97aWhc1wZPCLGb9ieMrwGEpmYtJ3OFKKvACwAzWRj1ZLKbtdnTEVef+lxlcNQJqzm7rhJiYmHTgimZ1SStOUMukVpgI3MTn8mDFKJiYmJn0cU5GbmJiY9HFMRW5iYmLSxzEVuYmJiUkfx1TkJiYmJn2cmJSxFUJUAcd6jn46UB1rIY4RzLloxZwLA3MeWjmaczFYSplx4IcxUeR9ASHE+mjxmv0Rcy5aMefCwJyHVo6FuTBdKyYmJiZ9HFORm5iYmPRxTEXeOX+LtQDHEOZctGLOhYE5D63EfC5MH7mJiYlJH8e0yE1MTEz6OKYiNzExMenjmIq8BwghbhVCSCFEvy3dJ4R4RAixQwixRQjxlhAiOdYyHU2EEPOEEDuFEAVCiDtjLU+sEELkCSE+FUJ8I4TYJoRYFGuZYokQQhVCbBRCLImlHKYi7wYhRB5wJrAv1rLEmA+BcVLKCcAu4BcxlueoIYRQgSeB+cAY4FIhxJjYShUzwsAtUsrRwEzghn48FwCLgG9iLYSpyLvnD8DtQL9eFZZSLpdShiO/rgYGxlKeo8x0oEBKuUdKGQReBc6PsUwxQUpZJqX8KvJzI4YSy42tVLFBCDEQOAd4NtaymIq8C4QQC4ASKeXmWMtyjHEVsCzWQhxFcoHiNr/vp58qr7YIIYYAk4E1sZUkZvwRw8jrvEv5UeKQOgR9FxBCfARkR9l0F/BLYO7RlSh2dDUXUsp3ImPuwni9fvloyhZjRJTP+vUbmhAiHngT+LmUsiHW8hxthBDnApVSyg1CiNmxlqffK3Ip5ZxonwshxgNDgc1CCDBcCV8JIaZLKcuj7dPX6WwumhFC/Bg4FzhD9q8EhP1AXpvfBwKlMZIl5gghrBhK/GUp5eJYyxMjTgIWCCHOBhxAohDiJSnlj2IhjJkQ1EOEEIXA8VLKflnxTQgxD3gMmCWlrIq1PEcTIYQFY4H3DKAEWAf8QEq5LaaCxQBhWDX/BGqllD+PtTzHAhGL/FYp5bmxksH0kZv0lCeABOBDIcQmIcRfYy3Q0SKyyPtT4AOMxb3X+6MSj3AScBlweuQ62BSxSk1iiGmRm5iYmPRxTIvcxMTEpI9jKnITExOTPo6pyE1MTEz6OKYiNzExMenjmIrcxMTEpI9jKnITExOTPo6pyE1MTEz6OP8fmfnpw2tYIQIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11.937225103378296\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "viz_train_data = np.array(pca_train_df.rdd.map(lambda row: [*row['pca_features'], row['labels2_index'], row['labels5_index']]).collect())\n",
    "plt.figure()\n",
    "plt.scatter(x=viz_train_data[:,0], y=viz_train_data[:,1], c=viz_train_data[:,2], cmap=\"Set1\")\n",
    "plt.figure()\n",
    "plt.scatter(x=viz_train_data[:,0], y=viz_train_data[:,1], c=viz_train_data[:,3], cmap=\"Set1\")\n",
    "plt.show()\n",
    "print(time() - t0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. KMeans clustering with Random Forest Classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea of the first approach is to clusterize data into clusters and then train different Random Forest classifiers for each of the clusters. As Random Forest returns probabilities, it is possible to improve detection rate for a new types of attacks by adjusting threshold."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As KMeans cannot truly handle binary/categorical features only numeric features are used for clustarization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "kmeans_prob_col = 'kmeans_rf_prob'\n",
    "kmeans_pred_col = 'kmeans_rf_pred'\n",
    "\n",
    "prob_cols.append(kmeans_prob_col)\n",
    "pred_cols.append(kmeans_pred_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time: 6.53s\n"
     ]
    }
   ],
   "source": [
    "# KMeans clustrering\n",
    "from pyspark.ml.clustering import KMeans\n",
    "\n",
    "t0 = time()\n",
    "kmeans_slicer = VectorSlicer(inputCol=\"indexed_features\", outputCol=\"features\", \n",
    "                             names=list(set(selectFeaturesByAR(ar_dict, 0.1)).intersection(numeric_cols)))\n",
    "\n",
    "kmeans = KMeans(k=8, initSteps=25, maxIter=100, featuresCol=\"features\", predictionCol=\"cluster\", seed=seed)\n",
    "\n",
    "kmeans_pipeline = Pipeline(stages=[kmeans_slicer, kmeans])\n",
    "\n",
    "kmeans_model = kmeans_pipeline.fit(scaled_train_df)\n",
    "\n",
    "kmeans_train_df = kmeans_model.transform(scaled_train_df).cache()\n",
    "kmeans_cv_df = kmeans_model.transform(scaled_cv_df).cache()\n",
    "kmeans_test_df = kmeans_model.transform(scaled_test_df).cache()\n",
    "\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Function for describing the contents of the clusters \n",
    "def getClusterCrosstab(df, clusterCol='cluster'):\n",
    "    return (df.crosstab(clusterCol, 'labels2')\n",
    "              .withColumn('count', col('attack') + col('normal'))\n",
    "              .withColumn(clusterCol + '_labels2', col(clusterCol + '_labels2').cast('int'))\n",
    "              .sort(col(clusterCol +'_labels2').asc()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------+------+------+-----+\n",
      "|cluster_labels2|attack|normal|count|\n",
      "+---------------+------+------+-----+\n",
      "|              0|  3582|  4273| 7855|\n",
      "|              1|   224|   398|  622|\n",
      "|              2| 27727|    99|27826|\n",
      "|              3|  5489| 46749|52238|\n",
      "|              4|  2172|   178| 2350|\n",
      "|              5|     2|    11|   13|\n",
      "|              6|  7627|  2265| 9892|\n",
      "|              7|     2|    42|   44|\n",
      "+---------------+------+------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kmeans_crosstab = getClusterCrosstab(kmeans_train_df).cache()\n",
    "kmeans_crosstab.show(n=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clustres are splitted into two categories. Frist category contains clusters that have both 'attack' and 'normal' connections and have more than 25 connections. For the first category Random Forest classifiers are aplied. Second category contains all other clusters and maps cluster to 'attack' or 'normal' based on majority. All clusters that contains less or equal than 25 connections are treated as outliers and are mapped to 'attack' type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7 1\n",
      "{5: 1.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{0: [7855, 0.4560152768936983],\n",
       " 1: [622, 0.36012861736334406],\n",
       " 2: [27826, 0.9964421763818012],\n",
       " 3: [52238, 0.10507676404150236],\n",
       " 4: [2350, 0.9242553191489362],\n",
       " 6: [9892, 0.7710270926000808],\n",
       " 7: [44, 0.045454545454545456]}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Function for splitting clusters\n",
    "def splitClusters(crosstab):\n",
    "    exp = ((col('count') > 25) & (col('attack') > 0) & (col('normal') > 0))\n",
    "\n",
    "    cluster_rf = (crosstab\n",
    "        .filter(exp).rdd\n",
    "        .map(lambda row: (int(row['cluster_labels2']), [row['count'], row['attack']/row['count']]))\n",
    "        .collectAsMap())\n",
    "\n",
    "    cluster_mapping = (crosstab\n",
    "        .filter(~exp).rdd\n",
    "        .map(lambda row: (int(row['cluster_labels2']), 1.0 if (row['count'] <= 25) | (row['normal'] == 0) else 0.0))\n",
    "        .collectAsMap())\n",
    "    \n",
    "    return cluster_rf, cluster_mapping\n",
    "\n",
    "kmeans_cluster_rf, kmeans_cluster_mapping = splitClusters(kmeans_crosstab)\n",
    "\n",
    "print(len(kmeans_cluster_rf), len(kmeans_cluster_mapping))\n",
    "print(kmeans_cluster_mapping)\n",
    "kmeans_cluster_rf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.classification import RandomForestClassifier\n",
    "\n",
    "# This function returns Random Forest models for provided clusters\n",
    "def getClusterModels(df, cluster_rf):\n",
    "    cluster_models = {}\n",
    "\n",
    "    labels_col = 'labels2_cl_index'\n",
    "    labels2_indexer.setOutputCol(labels_col)\n",
    "\n",
    "    rf_slicer = VectorSlicer(inputCol=\"indexed_features\", outputCol=\"rf_features\", \n",
    "                             names=selectFeaturesByAR(ar_dict, 0.05))\n",
    "\n",
    "    for cluster in cluster_rf.keys():\n",
    "        t1 = time()\n",
    "        rf_classifier = RandomForestClassifier(labelCol=labels_col, featuresCol='rf_features', seed=seed,\n",
    "                                               numTrees=500, maxDepth=20, featureSubsetStrategy=\"sqrt\")\n",
    "        \n",
    "        rf_pipeline = Pipeline(stages=[labels2_indexer, rf_slicer, rf_classifier])\n",
    "        cluster_models[cluster] = rf_pipeline.fit(df.filter(col('cluster') == cluster))\n",
    "        print(\"Finished %g cluster in %g s\" % (cluster, time() - t1))\n",
    "        \n",
    "    return cluster_models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# This utility function helps to get predictions/probabilities for the new data and return them into one dataframe\n",
    "def getProbabilities(df, probCol, cluster_mapping, cluster_models):\n",
    "    pred_df = (sqlContext.createDataFrame([], StructType([\n",
    "                    StructField('id', LongType(), False),\n",
    "                    StructField(probCol, DoubleType(), False)])))\n",
    "    \n",
    "    udf_map = udf(lambda cluster: cluster_mapping[cluster], DoubleType())\n",
    "    pred_df = pred_df.union(df.filter(col('cluster').isin(list(cluster_mapping.keys())))\n",
    "                            .withColumn(probCol, udf_map(col('cluster')))\n",
    "                            .select('id', probCol))\n",
    "\n",
    "                                       \n",
    "    for k in cluster_models.keys():\n",
    "        maj_label = cluster_models[k].stages[0].labels[0]\n",
    "        udf_remap_prob = udf(lambda row: float(row[0]) if (maj_label == 'attack') else float(row[1]), DoubleType())\n",
    "\n",
    "        pred_df = pred_df.union(cluster_models[k]\n",
    "                         .transform(df.filter(col('cluster') == k))\n",
    "                         .withColumn(probCol, udf_remap_prob(col('probability')))\n",
    "                         .select('id', probCol))\n",
    "\n",
    "    return pred_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished 0 cluster in 17.8515 s\n",
      "Finished 1 cluster in 2.84766 s\n",
      "Finished 2 cluster in 26.4936 s\n",
      "Finished 3 cluster in 256.526 s\n",
      "Finished 4 cluster in 3.38144 s\n",
      "Finished 6 cluster in 22.1199 s\n",
      "Finished 7 cluster in 0.931688 s\n",
      "Time: 330.16s\n"
     ]
    }
   ],
   "source": [
    "# Training Random Forest classifiers for each of the clusters\n",
    "t0 = time()\n",
    "kmeans_cluster_models = getClusterModels(kmeans_train_df, kmeans_cluster_rf)\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "Time: 24.28s\n"
     ]
    }
   ],
   "source": [
    "# Getting probabilities for CV data\n",
    "t0 = time()\n",
    "res_cv_df = (res_cv_df.drop(kmeans_prob_col)\n",
    "             .join(getProbabilities(kmeans_cv_df, kmeans_prob_col, kmeans_cluster_mapping, kmeans_cluster_models), 'id')\n",
    "             .cache())\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22544\n",
      "Time: 20.52s\n"
     ]
    }
   ],
   "source": [
    "# Getting probabilities for Test data\n",
    "t0 = time()\n",
    "res_test_df = (res_test_df.drop(kmeans_prob_col)\n",
    "               .join(getProbabilities(kmeans_test_df, kmeans_prob_col, kmeans_cluster_mapping, kmeans_cluster_models), 'id')\n",
    "               .cache())\n",
    "\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As CV data is from the same distribution as the train data it isn't needed to adjust threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t 13314\t    14\t\n",
      "attack\t    37\t 11768\t\n",
      " \n",
      "Accuracy = 0.997971\n",
      "AUC = 0.997908\n",
      " \n",
      "False Alarm Rate = 0.00105042\n",
      "Detection Rate = 0.996866\n",
      "F1 score = 0.997838\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       1.00      1.00      1.00     13328\n",
      "         1.0       1.00      1.00      1.00     11805\n",
      "\n",
      "    accuracy                           1.00     25133\n",
      "   macro avg       1.00      1.00      1.00     25133\n",
      "weighted avg       1.00      1.00      1.00     25133\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_cv_df, kmeans_prob_col, e=0.5, labels=labels2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because test data is from the different distribution and it is expected to face unseen attack types, it makes sence to adjust a probability threshold to something like 0.01 for attack connections (0.99 for normal connections). For this approach it gives around ~98-99% Detection Rate with around ~14-15% of False Alarm Rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  8293\t  1418\t\n",
      "attack\t   183\t 12650\t\n",
      " \n",
      "Accuracy = 0.928983\n",
      "AUC = 0.91986\n",
      " \n",
      "False Alarm Rate = 0.14602\n",
      "Detection Rate = 0.98574\n",
      "F1 score = 0.940485\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.98      0.85      0.91      9711\n",
      "         1.0       0.90      0.99      0.94     12833\n",
      "\n",
      "    accuracy                           0.93     22544\n",
      "   macro avg       0.94      0.92      0.93     22544\n",
      "weighted avg       0.93      0.93      0.93     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df, kmeans_prob_col, e=0.01, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "22544\n",
      "Time: 34.49s\n"
     ]
    }
   ],
   "source": [
    "# Adding prediction columns based on chosen thresholds into result dataframes\n",
    "t0 = time()\n",
    "res_cv_df = res_cv_df.withColumn(kmeans_pred_col, getPrediction(0.5)(col(kmeans_prob_col))).cache()\n",
    "res_test_df = res_test_df.withColumn(kmeans_pred_col, getPrediction(0.01)(col(kmeans_prob_col))).cache()\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Gaussian Mixture clustering with Random Forest Classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea of this approach is to clusterize data into clusters via Gaussian Mixture and then train different Random Forest classifiers for each of the clusters. Gaussian Mixture produces a diffirent clustering than KMeans, so results from both approaches could be combine for improving performance. As Gaussian Mixture clustering doesn't work well on high-demensional data PCA algorithm is used for preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "gm_prob_col = 'gm_rf_prob'\n",
    "gm_pred_col = 'gm_rf_pred'\n",
    "\n",
    "prob_cols.append(gm_prob_col)\n",
    "pred_cols.append(gm_pred_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100840\n",
      "25133\n",
      "22544\n",
      "Time: 15.03s\n"
     ]
    }
   ],
   "source": [
    "# Gaussian Mixture clustering\n",
    "from pyspark.ml.clustering import GaussianMixture\n",
    "\n",
    "t0 = time()\n",
    "gm = GaussianMixture(k=8, maxIter=150, seed=seed, featuresCol=\"pca_features\", \n",
    "                     predictionCol=\"cluster\", probabilityCol=\"gm_prob\")\n",
    "\n",
    "gm_pipeline = Pipeline(stages=[pca_slicer, pca, gm])\n",
    "gm_model = gm_pipeline.fit(scaled_train_df)\n",
    "\n",
    "gm_train_df = gm_model.transform(scaled_train_df).cache()\n",
    "gm_cv_df = gm_model.transform(scaled_cv_df).cache()\n",
    "gm_test_df = gm_model.transform(scaled_test_df).cache()\n",
    "\n",
    "gm_params = (gm_model.stages[2].gaussiansDF.rdd\n",
    "                  .map(lambda row: [row['mean'].toArray(), row['cov'].toArray()])\n",
    "                  .collect())\n",
    "gm_weights = gm_model.stages[2].weights\n",
    "\n",
    "print(gm_train_df.count())\n",
    "print(gm_cv_df.count())\n",
    "print(gm_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------+------+------+-----+\n",
      "|cluster_labels2|attack|normal|count|\n",
      "+---------------+------+------+-----+\n",
      "|              0|  4091|     0| 4091|\n",
      "|              1|    37|  1848| 1885|\n",
      "|              2| 22917|     0|22917|\n",
      "|              3|  5481|     0| 5481|\n",
      "|              4|  7517| 19873|27390|\n",
      "|              5|  2540|  5153| 7693|\n",
      "|              6|  4213|   316| 4529|\n",
      "|              7|    29| 26825|26854|\n",
      "+---------------+------+------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Description of the contents of the clusters \n",
    "gm_crosstab = getClusterCrosstab(gm_train_df).cache()\n",
    "gm_crosstab.show(n=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 3\n",
      "{0: 1.0, 2: 1.0, 3: 1.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{1: [1885, 0.019628647214854113],\n",
       " 4: [27390, 0.27444322745527566],\n",
       " 5: [7693, 0.3301702846743793],\n",
       " 6: [4529, 0.9302274232722455],\n",
       " 7: [26854, 0.0010799136069114472]}"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Splitting clusters\n",
    "gm_cluster_rf, gm_cluster_mapping = splitClusters(gm_crosstab)\n",
    "\n",
    "print(len(gm_cluster_rf), len(gm_cluster_mapping))\n",
    "print(gm_cluster_mapping)\n",
    "gm_cluster_rf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished 1 cluster in 2.26199 s\n",
      "Finished 4 cluster in 157.741 s\n",
      "Finished 5 cluster in 49.1167 s\n",
      "Finished 6 cluster in 9.94206 s\n",
      "Finished 7 cluster in 6.31671 s\n",
      "Time: 225.39s\n"
     ]
    }
   ],
   "source": [
    "# Training Random Forest classifiers for each of the clusters\n",
    "t0 = time()\n",
    "gm_cluster_models = getClusterModels(gm_train_df, gm_cluster_rf)\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "Time: 35.36s\n"
     ]
    }
   ],
   "source": [
    "# Getting probabilities for CV data\n",
    "t0 = time()\n",
    "res_cv_df = (res_cv_df.drop(gm_prob_col)\n",
    "             .join(getProbabilities(gm_cv_df, gm_prob_col, gm_cluster_mapping, gm_cluster_models), 'id')\n",
    "             .cache())\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22544\n",
      "Time: 32.74s\n"
     ]
    }
   ],
   "source": [
    "# Getting probabilities for Test data\n",
    "t0 = time()\n",
    "res_test_df = (res_test_df.drop(gm_prob_col)\n",
    "               .join(getProbabilities(gm_test_df, gm_prob_col, gm_cluster_mapping, gm_cluster_models), 'id')\n",
    "               .cache())\n",
    "\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t 13321\t     7\t\n",
      "attack\t    34\t 11771\t\n",
      " \n",
      "Accuracy = 0.998369\n",
      "AUC = 0.998297\n",
      " \n",
      "False Alarm Rate = 0.00052521\n",
      "Detection Rate = 0.99712\n",
      "F1 score = 0.998261\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       1.00      1.00      1.00     13328\n",
      "         1.0       1.00      1.00      1.00     11805\n",
      "\n",
      "    accuracy                           1.00     25133\n",
      "   macro avg       1.00      1.00      1.00     25133\n",
      "weighted avg       1.00      1.00      1.00     25133\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_cv_df, gm_prob_col, e=0.5, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  8279\t  1432\t\n",
      "attack\t   424\t 12409\t\n",
      " \n",
      "Accuracy = 0.917672\n",
      "AUC = 0.909749\n",
      " \n",
      "False Alarm Rate = 0.147462\n",
      "Detection Rate = 0.96696\n",
      "F1 score = 0.930419\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.95      0.85      0.90      9711\n",
      "         1.0       0.90      0.97      0.93     12833\n",
      "\n",
      "    accuracy                           0.92     22544\n",
      "   macro avg       0.92      0.91      0.91     22544\n",
      "weighted avg       0.92      0.92      0.92     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df, gm_prob_col, e=0.01, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "22544\n",
      "Time: 31.65s\n"
     ]
    }
   ],
   "source": [
    "# Adding prediction columns based on chosen thresholds into result dataframes\n",
    "t0 = time()\n",
    "res_cv_df = res_cv_df.withColumn(gm_pred_col, getPrediction(0.5)(col(gm_prob_col))).cache()\n",
    "res_test_df = res_test_df.withColumn(gm_pred_col, getPrediction(0.01)(col(gm_prob_col))).cache()\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Supervised approach for dettecting each type of attacks separately"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea of the following approach is training Random Forest Classifiers for each of four major 'attack' categories separately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.1 DoS and normal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dos_prob_col = 'dos_prob'\n",
    "dos_pred_col = 'dos_pred'\n",
    "\n",
    "prob_cols.append(dos_prob_col)\n",
    "pred_cols.append(dos_pred_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "90750\n",
      "+-------+-----+\n",
      "|labels5|count|\n",
      "+-------+-----+\n",
      "| normal|54015|\n",
      "|    DoS|36735|\n",
      "+-------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dos_exp = (col('labels5') == 'DoS') | (col('labels5') == 'normal')\n",
    "dos_train_df = (scaled_train_df.filter(dos_exp).cache())\n",
    "\n",
    "print(dos_train_df.count())\n",
    "(dos_train_df\n",
    "     .groupby('labels5')\n",
    "     .count()\n",
    "     .sort(sql.desc('count'))\n",
    "     .show())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Commented code below is related to undersampling 'normal' connections. It could give better results. However, it hasen't been tested a lot yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# dos_train_df = dos_train_df.sampleBy('labels5', fractions={'normal': 45927./67343, 'DoS': 1.0}).cache()\n",
    "\n",
    "# print(dos_train_df.count())\n",
    "# (dos_train_df\n",
    "#      .groupby('labels5')\n",
    "#      .count()\n",
    "#      .sort(sql.desc('count'))\n",
    "#      .show())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Diffirent AR feature selection is used as only normal and DoS connections are treated. Note that train dataframe without standartization is used for getting Attribute Ratio dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time: 6.62s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "OrderedDict([('flag_SF', 16.04886075949367),\n",
       "             ('protocol_type_tcp', 11.283230810377106),\n",
       "             ('flag_S0', 2.965034965034965),\n",
       "             ('wrong_fragment', 2.4663052235068696),\n",
       "             ('logged_in', 2.4559683875603),\n",
       "             ('dst_host_srv_serror_rate', 2.4369460592636014),\n",
       "             ('srv_serror_rate', 2.4091388710886883),\n",
       "             ('serror_rate', 2.403031496724026),\n",
       "             ('dst_host_serror_rate', 2.400741230991577),\n",
       "             ('count', 2.0805650842389314),\n",
       "             ('rerror_rate', 1.729470874459902),\n",
       "             ('srv_rerror_rate', 1.7274542811973763),\n",
       "             ('dst_host_srv_rerror_rate', 1.7208022988464688),\n",
       "             ('dst_host_rerror_rate', 1.7204184332241286),\n",
       "             ('num_failed_logins', 1.6819862494988345),\n",
       "             ('num_root', 1.6819862494988345),\n",
       "             ('urgent', 1.6819862494988342),\n",
       "             ('num_file_creations', 1.6819862494988342),\n",
       "             ('num_shells', 1.6819862494988342),\n",
       "             ('num_access_files', 1.6819862494988342),\n",
       "             ('duration', 1.6819438793516928),\n",
       "             ('num_compromised', 1.639589690499449),\n",
       "             ('dst_bytes', 1.6383223072580753),\n",
       "             ('srv_diff_host_rate', 1.635027252294015),\n",
       "             ('dst_host_srv_diff_host_rate', 1.6123221582570733),\n",
       "             ('src_bytes', 1.585158247265785),\n",
       "             ('dst_host_srv_count', 1.5359730763294566),\n",
       "             ('dst_host_same_srv_rate', 1.523983808504667),\n",
       "             ('hot', 1.5045374266320413),\n",
       "             ('diff_srv_rate', 1.4988971605166357),\n",
       "             ('same_srv_rate', 1.4819253849686713),\n",
       "             ('dst_host_same_src_port_rate', 1.3168437158004334),\n",
       "             ('dst_host_count', 1.30921165492205),\n",
       "             ('dst_host_diff_srv_rate', 1.3068775153420957),\n",
       "             ('service_http', 1.2988666621151088),\n",
       "             ('srv_count', 1.0994997402557993),\n",
       "             ('service_private', 0.5331486179730272),\n",
       "             ('protocol_type_udp', 0.22644739478045495),\n",
       "             ('service_domain_u', 0.15493320070658045),\n",
       "             ('flag_REJ', 0.14087341017488075),\n",
       "             ('service_smtp', 0.11654010677454654),\n",
       "             ('service_ftp_data', 0.07992430924164916),\n",
       "             ('protocol_type_icmp', 0.06608635097493036),\n",
       "             ('service_ecr_i', 0.06601211614790056),\n",
       "             ('service_other', 0.04022304947558659),\n",
       "             ('service_telnet', 0.02940715006163846),\n",
       "             ('flag_RSTO', 0.02719688667218358),\n",
       "             ('service_finger', 0.026095310440358364),\n",
       "             ('service_Z39_50', 0.018879226195758277),\n",
       "             ('service_uucp', 0.01702909783427078),\n",
       "             ('service_courier', 0.0160615915577089),\n",
       "             ('service_auth', 0.015544843445957898),\n",
       "             ('service_bgp', 0.0154550278588485),\n",
       "             ('service_uucp_path', 0.014938896377980597),\n",
       "             ('service_iso_tsap', 0.014916467780429593),\n",
       "             ('service_whois', 0.014804339660163068),\n",
       "             ('service_ftp', 0.013820097854723372),\n",
       "             ('service_nnsp', 0.013729168965897804),\n",
       "             ('service_imap4', 0.013729168965897804),\n",
       "             ('service_vmnet', 0.013371284834844774),\n",
       "             ('is_guest_login', 0.013133744546411915),\n",
       "             ('service_time', 0.012142983074753174),\n",
       "             ('service_ctf', 0.011853092158893123),\n",
       "             ('service_csnet_ns', 0.011741639864299247),\n",
       "             ('service_supdup', 0.011630212119209674),\n",
       "             ('service_http_443', 0.011518808915514052),\n",
       "             ('service_discard', 0.011451978769793203),\n",
       "             ('service_domain', 0.011184746471740902),\n",
       "             ('service_daytime', 0.01107344135258894),\n",
       "             ('service_gopher', 0.010672945733022314),\n",
       "             ('service_efs', 0.010517283108539242),\n",
       "             ('service_exec', 0.010228322555100963),\n",
       "             ('service_systat', 0.010117227879561),\n",
       "             ('service_link', 0.009983946517713808),\n",
       "             ('service_hostnames', 0.00982849604221636),\n",
       "             ('service_name', 0.009406800149453835),\n",
       "             ('service_klogin', 0.009340248780273395),\n",
       "             ('service_login', 0.009229349330872173),\n",
       "             ('service_mtp', 0.009140647316033486),\n",
       "             ('service_echo', 0.009140647316033486),\n",
       "             ('service_urp_i', 0.0089745894762076),\n",
       "             ('service_ldap', 0.008852473420613302),\n",
       "             ('service_netbios_dgm', 0.008608762490392005),\n",
       "             ('service_sunrpc', 0.00809956538917424),\n",
       "             ('service_netbios_ssn', 0.007657203036552723),\n",
       "             ('service_netstat', 0.0075466731018142726),\n",
       "             ('service_eco_i', 0.007434999850402417),\n",
       "             ('service_netbios_ns', 0.007369875633348687),\n",
       "             ('service_kshell', 0.006398597567656404),\n",
       "             ('service_nntp', 0.006156070630504316),\n",
       "             ('service_ssh', 0.006156070630504316),\n",
       "             ('flag_S1', 0.005389507628915231),\n",
       "             ('service_sql_net', 0.005099137742373178),\n",
       "             ('service_pop_3', 0.0027696293759399615),\n",
       "             ('service_IRC', 0.0027696293759399615),\n",
       "             ('service_ntp_u', 0.0025009304056568663),\n",
       "             ('flag_RSTR', 0.002172716043871006),\n",
       "             ('root_shell', 0.0020385084665059667),\n",
       "             ('flag_S2', 0.0017702011186481019),\n",
       "             ('service_pop_2', 0.0015264845061822622),\n",
       "             ('service_rje', 0.0014828059922806865),\n",
       "             ('service_printer', 0.0013517933064428214),\n",
       "             ('service_remote_job', 0.0013081300281247957),\n",
       "             ('service_shell', 0.0011553385359898854),\n",
       "             ('service_X11', 0.0009958974968785302),\n",
       "             ('flag_S3', 0.0006686677167226366),\n",
       "             ('land', 0.00039207998431680063),\n",
       "             ('su_attempted', 0.00029707529373319667),\n",
       "             ('flag_OTH', 0.00016336957167468663),\n",
       "             ('service_urh_i', 0.0001485155867108253),\n",
       "             ('service_red_i', 0.00011880894037276305),\n",
       "             ('service_tim_i', 7.425227954498203e-05),\n",
       "             ('service_tftp_u', 4.455004455004455e-05),\n",
       "             ('flag_SH', 2.969958866069705e-05),\n",
       "             ('is_host_login', 1.4849573817231445e-05),\n",
       "             ('service_http_2784', 0.0),\n",
       "             ('service_harvest', 0.0),\n",
       "             ('service_pm_dump', 0.0),\n",
       "             ('service_aol', 0.0),\n",
       "             ('service_http_8001', 0.0),\n",
       "             ('flag_RSTOS0', 0.0)])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t0 = time()\n",
    "dos_ar_dict = getAttributeRatio(train_df.filter(dos_exp), numeric_cols, binary_cols, 'labels5')\n",
    "\n",
    "print(f\"Time: {time() - t0:.2f}s\")\n",
    "dos_ar_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "22544\n",
      "Time: 212.05s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "dos_slicer = VectorSlicer(inputCol=\"indexed_features\", outputCol=\"features\", \n",
    "                          names=selectFeaturesByAR(dos_ar_dict, 0.05))\n",
    "\n",
    "dos_rf = RandomForestClassifier(labelCol=labels_col, featuresCol='features', featureSubsetStrategy='sqrt',\n",
    "                                numTrees=500, maxDepth=20, seed=seed)\n",
    "\n",
    "dos_rf_pipeline = Pipeline(stages=[dos_slicer, dos_rf])\n",
    "dos_rf_model = dos_rf_pipeline.fit(dos_train_df)\n",
    "\n",
    "dos_cv_df = dos_rf_model.transform(scaled_cv_df).cache()\n",
    "dos_test_df = dos_rf_model.transform(scaled_test_df).cache()\n",
    "print(dos_cv_df.count())\n",
    "print(dos_test_df.count())\n",
    "\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "Time: 15.69s\n"
     ]
    }
   ],
   "source": [
    "# Getting probabilities for CV data\n",
    "t0 = time()\n",
    "res_cv_df = (res_cv_df.drop(dos_prob_col)\n",
    "             .join(dos_cv_df.rdd\n",
    "                    .map(lambda row: (row['id'], float(row['probability'][1])))\n",
    "                    .toDF(['id', dos_prob_col]),\n",
    "                    'id')\n",
    "                    .cache())\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22544\n",
      "Time: 13.09s\n"
     ]
    }
   ],
   "source": [
    "# Getting probabilities for Test data\n",
    "t0 = time()\n",
    "res_test_df = (res_test_df.drop(dos_prob_col)\n",
    "               .join(dos_test_df.rdd\n",
    "                    .map(lambda row: (row['id'], float(row['probability'][1])))\n",
    "                    .toDF(['id', dos_prob_col]),\n",
    "                    'id')\n",
    "                    .cache())\n",
    "\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first report shows performance of classification for 'normal' and 'DoS' labels, the second report shows performance for the whole data with adjusted threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\t   DoS\t\n",
      "normal\t 13327\t     1\t\n",
      "   DoS\t     3\t  9189\t\n",
      " \n",
      "Accuracy = 0.999822\n",
      "AUC = 0.999799\n",
      " \n",
      "False Alarm Rate = 7.503e-05\n",
      "Detection Rate = 0.999674\n",
      "F1 score = 0.999782\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       1.00      1.00      1.00     13328\n",
      "         1.0       1.00      1.00      1.00      9192\n",
      "\n",
      "    accuracy                           1.00     22520\n",
      "   macro avg       1.00      1.00      1.00     22520\n",
      "weighted avg       1.00      1.00      1.00     22520\n",
      "\n",
      " \n",
      "      \tnormal\tattack\t\n",
      "normal\t 13247\t    81\t\n",
      "attack\t   415\t 11390\t\n",
      " \n",
      "Accuracy = 0.980265\n",
      "AUC = 0.979384\n",
      " \n",
      "False Alarm Rate = 0.00607743\n",
      "Detection Rate = 0.964845\n",
      "F1 score = 0.97869\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.97      0.99      0.98     13328\n",
      "         1.0       0.99      0.96      0.98     11805\n",
      "\n",
      "    accuracy                           0.98     25133\n",
      "   macro avg       0.98      0.98      0.98     25133\n",
      "weighted avg       0.98      0.98      0.98     25133\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_cv_df.filter(dos_exp), probCol=dos_prob_col, e=0.5, labels=['normal', 'DoS'])\n",
    "printReport(res_cv_df, probCol=dos_prob_col, e=0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\t   DoS\t\n",
      "normal\t  9624\t    87\t\n",
      "   DoS\t  1473\t  5985\t\n",
      " \n",
      "Accuracy = 0.909139\n",
      "AUC = 0.896768\n",
      " \n",
      "False Alarm Rate = 0.00895891\n",
      "Detection Rate = 0.802494\n",
      "F1 score = 0.884701\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.87      0.99      0.93      9711\n",
      "         1.0       0.99      0.80      0.88      7458\n",
      "\n",
      "    accuracy                           0.91     17169\n",
      "   macro avg       0.93      0.90      0.90     17169\n",
      "weighted avg       0.92      0.91      0.91     17169\n",
      "\n",
      " \n",
      "      \tnormal\tattack\t\n",
      "normal\t  8759\t   952\t\n",
      "attack\t  2782\t 10051\t\n",
      " \n",
      "Accuracy = 0.834368\n",
      "AUC = 0.842591\n",
      " \n",
      "False Alarm Rate = 0.0980332\n",
      "Detection Rate = 0.783215\n",
      "F1 score = 0.843346\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.76      0.90      0.82      9711\n",
      "         1.0       0.91      0.78      0.84     12833\n",
      "\n",
      "    accuracy                           0.83     22544\n",
      "   macro avg       0.84      0.84      0.83     22544\n",
      "weighted avg       0.85      0.83      0.84     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df.filter(dos_exp), probCol=dos_prob_col, e=0.5, labels=['normal', 'DoS'])\n",
    "printReport(res_test_df, probCol=dos_prob_col, e=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "22544\n",
      "Time: 5.51s\n"
     ]
    }
   ],
   "source": [
    "# Adding prediction columns based on chosen thresholds into result dataframes\n",
    "t0 = time()\n",
    "res_cv_df = res_cv_df.withColumn(dos_pred_col, getPrediction(0.05)(col(dos_prob_col))).cache()\n",
    "res_test_df = res_test_df.withColumn(dos_pred_col, getPrediction(0.01)(col(dos_prob_col))).cache()\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.2 Probe and normal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "probe_prob_col = 'probe_prob'\n",
    "probe_pred_col = 'probe_pred'\n",
    "\n",
    "prob_cols.append(probe_prob_col)\n",
    "pred_cols.append(probe_pred_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "63286\n",
      "+-------+-----+\n",
      "|labels5|count|\n",
      "+-------+-----+\n",
      "| normal|54015|\n",
      "|  Probe| 9271|\n",
      "+-------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "probe_exp = (col('labels5') == 'Probe') | (col('labels5') == 'normal')\n",
    "probe_train_df = (scaled_train_df.filter(probe_exp).cache())\n",
    "\n",
    "print(probe_train_df.count())\n",
    "(probe_train_df\n",
    "     .groupby('labels5')\n",
    "     .count()\n",
    "     .sort(sql.desc('count'))\n",
    "     .show())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Commented code below is related to undersampling 'normal' connections. It could give better results. However, it hasen't been tested a lot yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# probe_train_df = probe_train_df.sampleBy('labels5', fractions={'normal': 9274./53789, 'Probe': 1.0}).cache()\n",
    "\n",
    "# print(probe_train_df.count())\n",
    "# (probe_train_df\n",
    "#      .groupby('labels5')\n",
    "#      .count()\n",
    "#      .sort(sql.desc('count'))\n",
    "#      .show())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Diffirent AR feature selection is used as only normal and Probe connections are treated. Note that train dataframe without standartization is used for getting Attribute Ratio dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time: 4.46s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "OrderedDict([('flag_SF', 16.04886075949367),\n",
       "             ('dst_bytes', 5.954890189917854),\n",
       "             ('src_bytes', 5.663342453521275),\n",
       "             ('duration', 4.612329795738909),\n",
       "             ('dst_host_diff_srv_rate', 4.295388446518038),\n",
       "             ('srv_rerror_rate', 4.289248359824753),\n",
       "             ('rerror_rate', 4.279060093009581),\n",
       "             ('dst_host_srv_rerror_rate', 4.274560348159813),\n",
       "             ('diff_srv_rate', 4.111925438331699),\n",
       "             ('dst_host_rerror_rate', 4.008755744761299),\n",
       "             ('protocol_type_tcp', 3.900167357927672),\n",
       "             ('dst_host_srv_diff_host_rate', 3.761764380647146),\n",
       "             ('dst_host_srv_serror_rate', 3.5899755168448144),\n",
       "             ('dst_host_same_src_port_rate', 3.260144247441684),\n",
       "             ('serror_rate', 2.5392653643867047),\n",
       "             ('count', 2.520990695784333),\n",
       "             ('srv_serror_rate', 2.475062409348203),\n",
       "             ('logged_in', 2.4559683875603),\n",
       "             ('dst_host_serror_rate', 2.422128377779547),\n",
       "             ('srv_diff_host_rate', 1.9745638565439239),\n",
       "             ('service_http', 1.2988666621151088),\n",
       "             ('urgent', 1.1730840621890917),\n",
       "             ('num_access_files', 1.1730840621890917),\n",
       "             ('num_shells', 1.1730840621890914),\n",
       "             ('num_root', 1.1728674890158846),\n",
       "             ('num_compromised', 1.1728436409203076),\n",
       "             ('hot', 1.171650896218053),\n",
       "             ('num_file_creations', 1.1576487455813405),\n",
       "             ('dst_host_srv_count', 1.129554174884712),\n",
       "             ('num_failed_logins', 1.1247094616864488),\n",
       "             ('srv_count', 1.0980131509657178),\n",
       "             ('dst_host_same_srv_rate', 1.082859898411067),\n",
       "             ('same_srv_rate', 1.0432163005405286),\n",
       "             ('dst_host_count', 1.002234530632061),\n",
       "             ('service_private', 0.7252812314979278),\n",
       "             ('protocol_type_icmp', 0.5497939103842574),\n",
       "             ('service_eco_i', 0.5403726708074534),\n",
       "             ('flag_REJ', 0.3265050642995334),\n",
       "             ('flag_RSTR', 0.23005487547488393),\n",
       "             ('protocol_type_udp', 0.22644739478045495),\n",
       "             ('service_other', 0.16945921541085582),\n",
       "             ('service_domain_u', 0.15493320070658045),\n",
       "             ('service_smtp', 0.11654010677454654),\n",
       "             ('service_ftp_data', 0.07992430924164916),\n",
       "             ('flag_SH', 0.02326398033535247),\n",
       "             ('service_ftp', 0.013820097854723372),\n",
       "             ('service_telnet', 0.013804835455996147),\n",
       "             ('flag_S0', 0.013300878031817787),\n",
       "             ('is_guest_login', 0.013133744546411915),\n",
       "             ('service_urp_i', 0.0089745894762076),\n",
       "             ('flag_RSTOS0', 0.008915433220808448),\n",
       "             ('service_finger', 0.00815892691397946),\n",
       "             ('flag_RSTO', 0.006910850034554251),\n",
       "             ('flag_S1', 0.005389507628915231),\n",
       "             ('service_ecr_i', 0.0037027469215534315),\n",
       "             ('service_auth', 0.003516771722771097),\n",
       "             ('flag_OTH', 0.0030117890026675844),\n",
       "             ('service_gopher', 0.002839198141615762),\n",
       "             ('service_pop_3', 0.0027696293759399615),\n",
       "             ('service_IRC', 0.0027696293759399615),\n",
       "             ('service_ntp_u', 0.0025009304056568663),\n",
       "             ('service_time', 0.0023217817525152634),\n",
       "             ('service_ctf', 0.0021494282520849455),\n",
       "             ('service_ssh', 0.0021494282520849455),\n",
       "             ('root_shell', 0.0020385084665059667),\n",
       "             ('service_name', 0.001977134015301298),\n",
       "             ('service_mtp', 0.001977134015301298),\n",
       "             ('service_domain', 0.001977134015301298),\n",
       "             ('service_whois', 0.001977134015301298),\n",
       "             ('service_link', 0.0018048990116029222),\n",
       "             ('flag_S2', 0.0017702011186481019),\n",
       "             ('service_discard', 0.0015466575012888812),\n",
       "             ('service_daytime', 0.0015466575012888812),\n",
       "             ('service_echo', 0.0015466575012888812),\n",
       "             ('service_remote_job', 0.0015466575012888812),\n",
       "             ('service_rje', 0.0015466575012888812),\n",
       "             ('service_systat', 0.0014606065813214193),\n",
       "             ('service_supdup', 0.0013745704467353953),\n",
       "             ('service_netstat', 0.0013745704467353953),\n",
       "             ('service_nntp', 0.0012885490937204708),\n",
       "             ('service_uucp_path', 0.0011165507171691145),\n",
       "             ('service_netbios_dgm', 0.0011165507171691145),\n",
       "             ('service_netbios_ssn', 0.0011165507171691145),\n",
       "             ('service_hostnames', 0.0011165507171691145),\n",
       "             ('service_iso_tsap', 0.0010305736860185502),\n",
       "             ('service_sql_net', 0.0010305736860185502),\n",
       "             ('service_csnet_ns', 0.0010305736860185502),\n",
       "             ('service_sunrpc', 0.0010305736860185502),\n",
       "             ('service_X11', 0.0009958974968785302),\n",
       "             ('service_Z39_50', 0.0009446114212108201),\n",
       "             ('service_netbios_ns', 0.0009446114212108201),\n",
       "             ('service_vmnet', 0.0009446114212108201),\n",
       "             ('service_imap4', 0.0009446114212108201),\n",
       "             ('service_uucp', 0.0009446114212108201),\n",
       "             ('service_bgp', 0.0009446114212108201),\n",
       "             ('service_exec', 0.0007727311754099768),\n",
       "             ('service_courier', 0.0006868131868131869),\n",
       "             ('service_pop_2', 0.0006868131868131869),\n",
       "             ('service_nnsp', 0.0006868131868131869),\n",
       "             ('service_klogin', 0.0006868131868131869),\n",
       "             ('service_shell', 0.0006868131868131869),\n",
       "             ('flag_S3', 0.0006686677167226366),\n",
       "             ('service_ldap', 0.0006009099493518757),\n",
       "             ('service_login', 0.0006009099493518757),\n",
       "             ('service_kshell', 0.0006009099493518757),\n",
       "             ('service_efs', 0.0006009099493518757),\n",
       "             ('service_printer', 0.0006009099493518757),\n",
       "             ('service_http_443', 0.0006009099493518757),\n",
       "             ('service_pm_dump', 0.0004291477126426916),\n",
       "             ('su_attempted', 0.00029707529373319667),\n",
       "             ('service_harvest', 0.00017161489617298782),\n",
       "             ('service_aol', 0.00017161489617298782),\n",
       "             ('service_http_8001', 0.00017161489617298782),\n",
       "             ('service_urh_i', 0.0001485155867108253),\n",
       "             ('service_red_i', 0.00011880894037276305),\n",
       "             ('land', 0.00010395627895924914),\n",
       "             ('service_http_2784', 8.58000858000858e-05),\n",
       "             ('service_tim_i', 7.425227954498203e-05),\n",
       "             ('service_tftp_u', 4.455004455004455e-05),\n",
       "             ('is_host_login', 1.4849573817231445e-05),\n",
       "             ('wrong_fragment', 0.0)])"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t0 = time()\n",
    "probe_ar_dict = getAttributeRatio(train_df.filter(probe_exp), numeric_cols, binary_cols, 'labels5')\n",
    "\n",
    "print(f\"Time: {time() - t0:.2f}s\")\n",
    "probe_ar_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "22544\n",
      "Time: 222.12s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "probe_slicer = VectorSlicer(inputCol=\"indexed_features\", outputCol=\"features\",\n",
    "                            names=selectFeaturesByAR(probe_ar_dict, 0.05))\n",
    "\n",
    "probe_rf = RandomForestClassifier(labelCol=labels_col, featuresCol='features', featureSubsetStrategy='sqrt',\n",
    "                                  numTrees=500, maxDepth=20, seed=seed)\n",
    "probe_rf_pipeline = Pipeline(stages=[probe_slicer, probe_rf])\n",
    "\n",
    "probe_rf_model = probe_rf_pipeline.fit(probe_train_df)\n",
    "\n",
    "probe_cv_df = probe_rf_model.transform(scaled_cv_df).cache()\n",
    "probe_test_df = probe_rf_model.transform(scaled_test_df).cache()\n",
    "\n",
    "print(probe_cv_df.count())\n",
    "print(probe_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "Time: 4.96s\n"
     ]
    }
   ],
   "source": [
    "# Getting probabilities for CV data\n",
    "t0 = time()\n",
    "res_cv_df = (res_cv_df.drop(probe_prob_col)\n",
    "             .join(probe_cv_df.rdd\n",
    "                    .map(lambda row: (row['id'], float(row['probability'][1])))\n",
    "                    .toDF(['id', probe_prob_col]), 'id')\n",
    "                    .cache())\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22544\n",
      "Time: 4.36s\n"
     ]
    }
   ],
   "source": [
    "# Getting probabilities for Test data\n",
    "t0 = time()\n",
    "res_test_df = (res_test_df.drop(probe_prob_col)\n",
    "               .join(probe_test_df.rdd\n",
    "                    .map(lambda row: (row['id'], float(row['probability'][1])))\n",
    "                    .toDF(['id', probe_prob_col]), 'id')\n",
    "                    .cache())\n",
    "\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first report shows performance of classification for 'normal' and 'Probe' labels, the second report shows performance for the whole data with adjusted threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\t Probe\t\n",
      "normal\t 13324\t     4\t\n",
      " Probe\t    13\t  2372\t\n",
      " \n",
      "Accuracy = 0.998918\n",
      "AUC = 0.997125\n",
      " \n",
      "False Alarm Rate = 0.00030012\n",
      "Detection Rate = 0.994549\n",
      "F1 score = 0.996429\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       1.00      1.00      1.00     13328\n",
      "         1.0       1.00      0.99      1.00      2385\n",
      "\n",
      "    accuracy                           1.00     15713\n",
      "   macro avg       1.00      1.00      1.00     15713\n",
      "weighted avg       1.00      1.00      1.00     15713\n",
      "\n",
      " \n",
      "      \tnormal\tattack\t\n",
      "normal\t 13133\t   195\t\n",
      "attack\t   417\t 11388\t\n",
      " \n",
      "Accuracy = 0.97565\n",
      "AUC = 0.975023\n",
      " \n",
      "False Alarm Rate = 0.0146309\n",
      "Detection Rate = 0.964676\n",
      "F1 score = 0.973833\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.97      0.99      0.98     13328\n",
      "         1.0       0.98      0.96      0.97     11805\n",
      "\n",
      "    accuracy                           0.98     25133\n",
      "   macro avg       0.98      0.98      0.98     25133\n",
      "weighted avg       0.98      0.98      0.98     25133\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_cv_df.filter(probe_exp), probCol=probe_prob_col, e=0.5, labels=['normal', 'Probe'])\n",
    "printReport(res_cv_df, probCol=probe_prob_col, e=0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\t Probe\t\n",
      "normal\t  9498\t   213\t\n",
      " Probe\t   969\t  1452\t\n",
      " \n",
      "Accuracy = 0.902572\n",
      "AUC = 0.788909\n",
      " \n",
      "False Alarm Rate = 0.0219339\n",
      "Detection Rate = 0.599752\n",
      "F1 score = 0.71072\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.91      0.98      0.94      9711\n",
      "         1.0       0.87      0.60      0.71      2421\n",
      "\n",
      "    accuracy                           0.90     12132\n",
      "   macro avg       0.89      0.79      0.83     12132\n",
      "weighted avg       0.90      0.90      0.90     12132\n",
      "\n",
      " \n",
      "      \tnormal\tattack\t\n",
      "normal\t  8436\t  1275\t\n",
      "attack\t  2047\t 10786\t\n",
      " \n",
      "Accuracy = 0.852644\n",
      "AUC = 0.854597\n",
      " \n",
      "False Alarm Rate = 0.131294\n",
      "Detection Rate = 0.840489\n",
      "F1 score = 0.866554\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.80      0.87      0.84      9711\n",
      "         1.0       0.89      0.84      0.87     12833\n",
      "\n",
      "    accuracy                           0.85     22544\n",
      "   macro avg       0.85      0.85      0.85     22544\n",
      "weighted avg       0.86      0.85      0.85     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df.filter(probe_exp), probCol=probe_prob_col, e=0.5, labels=['normal', 'Probe'])\n",
    "printReport(res_test_df, probCol=probe_prob_col, e=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "22544\n",
      "Time: 4.88s\n"
     ]
    }
   ],
   "source": [
    "# Adding prediction columns based on chosen thresholds into result dataframes\n",
    "t0 = time()\n",
    "res_cv_df = res_cv_df.withColumn(probe_pred_col, getPrediction(0.05)(col(probe_prob_col))).cache()\n",
    "res_test_df = res_test_df.withColumn(probe_pred_col, getPrediction(0.01)(col(probe_prob_col))).cache()\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.3 R2L, U2R and normal types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As there are a few examples of both R2L and U2R attack types and they have similar behaviour, they are combined into one group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "r2l_u2r_prob_col = 'r2l_u2r_prob'\n",
    "r2l_u2r_pred_col = 'r2l_u2r_pred'\n",
    "\n",
    "prob_cols.append(r2l_u2r_prob_col)\n",
    "pred_cols.append(r2l_u2r_pred_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "54834\n",
      "+-------+-----+\n",
      "|labels5|count|\n",
      "+-------+-----+\n",
      "| normal|54015|\n",
      "|    R2L|  782|\n",
      "|    U2R|   37|\n",
      "+-------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "r2l_u2r_exp = (col('labels5') == 'R2L') | (col('labels5') == 'U2R') | (col('labels5') == 'normal')\n",
    "r2l_u2r_train_df = (scaled_train_df.filter(r2l_u2r_exp).cache())\n",
    "\n",
    "print(r2l_u2r_train_df.count())\n",
    "(r2l_u2r_train_df\n",
    "     .groupby('labels5')\n",
    "     .count()\n",
    "     .sort(sql.desc('count'))\n",
    "     .show())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Diffirent AR feature selection is used as only normal, R2L and U2R connections are treated. Note that train dataframe without standartization is used for getting Attribute Ratio dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time: 5.22s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "OrderedDict([('protocol_type_tcp', 1000.0),\n",
       "             ('num_shells', 177.04511834319524),\n",
       "             ('urgent', 93.9423076923077),\n",
       "             ('flag_SF', 51.0),\n",
       "             ('num_file_creations', 34.215028309254194),\n",
       "             ('num_failed_logins', 25.660569514237856),\n",
       "             ('hot', 23.850718086856727),\n",
       "             ('src_bytes', 17.67531409051069),\n",
       "             ('dst_bytes', 14.991946554922052),\n",
       "             ('logged_in', 10.569767441860465),\n",
       "             ('dst_host_same_src_port_rate', 4.6280053915595785),\n",
       "             ('duration', 3.6136716211332516),\n",
       "             ('dst_host_srv_diff_host_rate', 3.186499677327565),\n",
       "             ('serror_rate', 2.8336950817010025),\n",
       "             ('dst_host_srv_serror_rate', 2.550033066061569),\n",
       "             ('num_access_files', 2.5488223017292784),\n",
       "             ('num_compromised', 2.416504765066944),\n",
       "             ('diff_srv_rate', 2.2340127157717946),\n",
       "             ('service_telnet', 1.8888888888888888),\n",
       "             ('dst_host_serror_rate', 1.6957954188400486),\n",
       "             ('service_ftp_data', 1.5447570332480818),\n",
       "             ('num_root', 1.4167490243394714),\n",
       "             ('service_http', 1.2988666621151088),\n",
       "             ('rerror_rate', 1.1689013372512598),\n",
       "             ('srv_rerror_rate', 1.1637508327448693),\n",
       "             ('dst_host_rerror_rate', 1.0957297788537126),\n",
       "             ('dst_host_srv_rerror_rate', 1.059242501878443),\n",
       "             ('same_srv_rate', 1.0277652952510852),\n",
       "             ('count', 1.0144815637217448),\n",
       "             ('srv_count', 1.0141813848818528),\n",
       "             ('srv_diff_host_rate', 1.013512607275963),\n",
       "             ('dst_host_srv_count', 1.0121712135809817),\n",
       "             ('dst_host_diff_srv_rate', 1.0068745777366659),\n",
       "             ('dst_host_count', 1.0063161527990898),\n",
       "             ('srv_serror_rate', 1.00201435928576),\n",
       "             ('dst_host_same_srv_rate', 1.0015453739903877),\n",
       "             ('root_shell', 1.0),\n",
       "             ('is_guest_login', 0.45894428152492667),\n",
       "             ('service_ftp', 0.4568081991215227),\n",
       "             ('protocol_type_udp', 0.22644739478045495),\n",
       "             ('service_domain_u', 0.15493320070658045),\n",
       "             ('service_smtp', 0.11654010677454654),\n",
       "             ('service_other', 0.061224489795918366),\n",
       "             ('flag_RSTO', 0.04847207586933614),\n",
       "             ('flag_REJ', 0.04165506573859242),\n",
       "             ('protocol_type_icmp', 0.019823121422297606),\n",
       "             ('service_private', 0.014797848133692983),\n",
       "             ('service_imap4', 0.011178861788617886),\n",
       "             ('service_urp_i', 0.0089745894762076),\n",
       "             ('service_finger', 0.00815892691397946),\n",
       "             ('service_eco_i', 0.007434999850402417),\n",
       "             ('flag_S1', 0.005389507628915231),\n",
       "             ('flag_S0', 0.005284449685769305),\n",
       "             ('flag_RSTR', 0.005050505050505051),\n",
       "             ('flag_SH', 0.004036326942482341),\n",
       "             ('service_auth', 0.003516771722771097),\n",
       "             ('flag_S3', 0.0030241935483870967),\n",
       "             ('service_ecr_i', 0.002829359820112281),\n",
       "             ('service_pop_3', 0.0027696293759399615),\n",
       "             ('service_IRC', 0.0027696293759399615),\n",
       "             ('service_ntp_u', 0.0025009304056568663),\n",
       "             ('service_login', 0.002014098690835851),\n",
       "             ('flag_S2', 0.0017702011186481019),\n",
       "             ('service_time', 0.0011298259176119047),\n",
       "             ('su_attempted', 0.001006036217303823),\n",
       "             ('service_X11', 0.0009958974968785302),\n",
       "             ('service_domain', 0.0005645940123319218),\n",
       "             ('flag_OTH', 0.00016336957167468663),\n",
       "             ('service_urh_i', 0.0001485155867108253),\n",
       "             ('service_red_i', 0.00011880894037276305),\n",
       "             ('land', 0.00010395627895924914),\n",
       "             ('service_tim_i', 7.425227954498203e-05),\n",
       "             ('service_ssh', 7.425227954498203e-05),\n",
       "             ('service_shell', 5.940094150492285e-05),\n",
       "             ('service_tftp_u', 4.455004455004455e-05),\n",
       "             ('is_host_login', 1.4849573817231445e-05),\n",
       "             ('wrong_fragment', 0.0),\n",
       "             ('service_iso_tsap', 0.0),\n",
       "             ('service_systat', 0.0),\n",
       "             ('service_name', 0.0),\n",
       "             ('service_sql_net', 0.0),\n",
       "             ('service_ldap', 0.0),\n",
       "             ('service_discard', 0.0),\n",
       "             ('service_Z39_50', 0.0),\n",
       "             ('service_daytime', 0.0),\n",
       "             ('service_http_2784', 0.0),\n",
       "             ('service_mtp', 0.0),\n",
       "             ('service_harvest', 0.0),\n",
       "             ('service_link', 0.0),\n",
       "             ('service_courier', 0.0),\n",
       "             ('service_kshell', 0.0),\n",
       "             ('service_pop_2', 0.0),\n",
       "             ('service_exec', 0.0),\n",
       "             ('service_efs', 0.0),\n",
       "             ('service_nnsp', 0.0),\n",
       "             ('service_pm_dump', 0.0),\n",
       "             ('service_whois', 0.0),\n",
       "             ('service_nntp', 0.0),\n",
       "             ('service_netbios_ns', 0.0),\n",
       "             ('service_aol', 0.0),\n",
       "             ('service_klogin', 0.0),\n",
       "             ('service_uucp_path', 0.0),\n",
       "             ('service_vmnet', 0.0),\n",
       "             ('service_ctf', 0.0),\n",
       "             ('service_supdup', 0.0),\n",
       "             ('service_http_8001', 0.0),\n",
       "             ('service_netbios_dgm', 0.0),\n",
       "             ('service_printer', 0.0),\n",
       "             ('service_netbios_ssn', 0.0),\n",
       "             ('service_csnet_ns', 0.0),\n",
       "             ('service_hostnames', 0.0),\n",
       "             ('service_sunrpc', 0.0),\n",
       "             ('service_http_443', 0.0),\n",
       "             ('service_echo', 0.0),\n",
       "             ('service_netstat', 0.0),\n",
       "             ('service_remote_job', 0.0),\n",
       "             ('service_gopher', 0.0),\n",
       "             ('service_uucp', 0.0),\n",
       "             ('service_rje', 0.0),\n",
       "             ('service_bgp', 0.0),\n",
       "             ('flag_RSTOS0', 0.0)])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t0 = time()\n",
    "r2l_u2r_ar_dict = getAttributeRatio(train_df.filter(r2l_u2r_exp), numeric_cols, binary_cols, 'labels5')\n",
    "\n",
    "print(f\"Time: {time() - t0:.2f}s\")\n",
    "r2l_u2r_ar_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "22544\n",
      "Time: 187.78s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "r2l_u2r_slicer = VectorSlicer(inputCol=\"indexed_features\", outputCol=\"features\",\n",
    "                              names=selectFeaturesByAR(r2l_u2r_ar_dict, 0.05))\n",
    "\n",
    "r2l_u2r_rf = RandomForestClassifier(labelCol=labels_col, featuresCol='features', featureSubsetStrategy='sqrt',\n",
    "                                    numTrees=500, maxDepth=20, seed=seed)\n",
    "r2l_u2r_rf_pipeline = Pipeline(stages=[r2l_u2r_slicer, r2l_u2r_rf])\n",
    "\n",
    "r2l_u2r_rf_model = r2l_u2r_rf_pipeline.fit(r2l_u2r_train_df)\n",
    "\n",
    "r2l_u2r_cv_df = r2l_u2r_rf_model.transform(scaled_cv_df).cache()\n",
    "r2l_u2r_test_df = r2l_u2r_rf_model.transform(scaled_test_df).cache()\n",
    "print(r2l_u2r_cv_df.count())\n",
    "print(r2l_u2r_test_df.count())\n",
    "\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "Time: 5.90s\n"
     ]
    }
   ],
   "source": [
    "# Getting probabilities for CV data\n",
    "t0 = time()\n",
    "res_cv_df = (res_cv_df.drop(r2l_u2r_prob_col)\n",
    "             .join(r2l_u2r_cv_df.rdd\n",
    "                    .map(lambda row: (row['id'], float(row['probability'][1])))\n",
    "                    .toDF(['id', r2l_u2r_prob_col]), 'id')\n",
    "                    .cache())\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22544\n",
      "Time: 4.37s\n"
     ]
    }
   ],
   "source": [
    "# Getting probabilities for Test data\n",
    "t0 = time()\n",
    "res_test_df = (res_test_df.drop(r2l_u2r_prob_col)\n",
    "               .join(r2l_u2r_test_df.rdd\n",
    "                    .map(lambda row: (row['id'], float(row['probability'][1])))\n",
    "                    .toDF(['id', r2l_u2r_prob_col]), 'id')\n",
    "                    .cache())\n",
    "\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first report shows performance of classification for 'normal' and 'R2L&U2R' labels, the second report shows performance for the whole data with adjusted threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       \t normal\tR2L&U2R\t\n",
      " normal\t  13324\t      4\t\n",
      "R2L&U2R\t     15\t    213\t\n",
      " \n",
      "Accuracy = 0.998598\n",
      "AUC = 0.966955\n",
      " \n",
      "False Alarm Rate = 0.00030012\n",
      "Detection Rate = 0.934211\n",
      "F1 score = 0.957303\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       1.00      1.00      1.00     13328\n",
      "         1.0       0.98      0.93      0.96       228\n",
      "\n",
      "    accuracy                           1.00     13556\n",
      "   macro avg       0.99      0.97      0.98     13556\n",
      "weighted avg       1.00      1.00      1.00     13556\n",
      "\n",
      " \n",
      "      \tnormal\tattack\t\n",
      "normal\t 13246\t    82\t\n",
      "attack\t  4758\t  7047\t\n",
      " \n",
      "Accuracy = 0.807425\n",
      "AUC = 0.795399\n",
      " \n",
      "False Alarm Rate = 0.00615246\n",
      "Detection Rate = 0.59695\n",
      "F1 score = 0.744375\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.74      0.99      0.85     13328\n",
      "         1.0       0.99      0.60      0.74     11805\n",
      "\n",
      "    accuracy                           0.81     25133\n",
      "   macro avg       0.86      0.80      0.79     25133\n",
      "weighted avg       0.85      0.81      0.80     25133\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_cv_df.filter(r2l_u2r_exp), probCol=r2l_u2r_prob_col, e=0.5, labels=['normal', 'R2L&U2R'])\n",
    "printReport(res_cv_df, probCol=r2l_u2r_prob_col, e=0.05, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       \t normal\tR2L&U2R\t\n",
      " normal\t   9709\t      2\t\n",
      "R2L&U2R\t   2653\t    301\t\n",
      " \n",
      "Accuracy = 0.790367\n",
      "AUC = 0.550845\n",
      " \n",
      "False Alarm Rate = 0.000205952\n",
      "Detection Rate = 0.101896\n",
      "F1 score = 0.184833\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.79      1.00      0.88      9711\n",
      "         1.0       0.99      0.10      0.18      2954\n",
      "\n",
      "    accuracy                           0.79     12665\n",
      "   macro avg       0.89      0.55      0.53     12665\n",
      "weighted avg       0.83      0.79      0.72     12665\n",
      "\n",
      " \n",
      "      \tnormal\tattack\t\n",
      "normal\t  9413\t   298\t\n",
      "attack\t  5934\t  6899\t\n",
      " \n",
      "Accuracy = 0.723563\n",
      "AUC = 0.753456\n",
      " \n",
      "False Alarm Rate = 0.0306868\n",
      "Detection Rate = 0.537598\n",
      "F1 score = 0.688867\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.61      0.97      0.75      9711\n",
      "         1.0       0.96      0.54      0.69     12833\n",
      "\n",
      "    accuracy                           0.72     22544\n",
      "   macro avg       0.79      0.75      0.72     22544\n",
      "weighted avg       0.81      0.72      0.72     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df.filter(r2l_u2r_exp), probCol=r2l_u2r_prob_col, e=0.5, labels=['normal', 'R2L&U2R'])\n",
    "printReport(res_test_df, probCol=r2l_u2r_prob_col, e=0.01, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "22544\n",
      "Time: 4.46s\n"
     ]
    }
   ],
   "source": [
    "# Adding prediction columns based on chosen thresholds into result dataframes\n",
    "t0 = time()\n",
    "res_cv_df = res_cv_df.withColumn(r2l_u2r_pred_col, getPrediction(0.05)(col(r2l_u2r_prob_col))).cache()\n",
    "res_test_df = res_test_df.withColumn(r2l_u2r_pred_col, getPrediction(0.01)(col(r2l_u2r_prob_col))).cache()\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.4 Combining results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sup_prob_col = 'sup_prob'\n",
    "sup_pred_col = 'sup_pred'\n",
    "\n",
    "prob_cols.append(sup_prob_col)\n",
    "pred_cols.append(sup_pred_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t 13198\t   130\t\n",
      "attack\t     3\t 11802\t\n",
      " \n",
      "Accuracy = 0.994708\n",
      "AUC = 0.994996\n",
      " \n",
      "False Alarm Rate = 0.0097539\n",
      "Detection Rate = 0.999746\n",
      "F1 score = 0.994397\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       1.00      0.99      0.99     13328\n",
      "         1.0       0.99      1.00      0.99     11805\n",
      "\n",
      "    accuracy                           0.99     25133\n",
      "   macro avg       0.99      0.99      0.99     25133\n",
      "weighted avg       0.99      0.99      0.99     25133\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "res_cv_df = res_cv_df.withColumn(sup_prob_col, \n",
    "                                 (col(dos_prob_col) + col(probe_prob_col) + col(r2l_u2r_prob_col))/3).cache()\n",
    "\n",
    "printReport(res_cv_df, sup_prob_col, e=0.05, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t 13020\t   308\t\n",
      "attack\t     0\t 11805\t\n",
      " \n",
      "Accuracy = 0.987745\n",
      "AUC = 0.988445\n",
      " \n",
      "False Alarm Rate = 0.0231092\n",
      "Detection Rate = 1\n",
      "F1 score = 0.987123\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       1.00      0.98      0.99     13328\n",
      "         1.0       0.97      1.00      0.99     11805\n",
      "\n",
      "    accuracy                           0.99     25133\n",
      "   macro avg       0.99      0.99      0.99     25133\n",
      "weighted avg       0.99      0.99      0.99     25133\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "res_cv_df = res_cv_df.withColumn(sup_pred_col, col(dos_pred_col).cast('int')\n",
    "                                        .bitwiseOR(col(probe_pred_col).cast('int'))\n",
    "                                        .bitwiseOR(col(r2l_u2r_pred_col).cast('int'))).cache()\n",
    "\n",
    "printReport(res_cv_df, sup_pred_col, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  8363\t  1348\t\n",
      "attack\t   830\t 12003\t\n",
      " \n",
      "Accuracy = 0.903389\n",
      "AUC = 0.898256\n",
      " \n",
      "False Alarm Rate = 0.138812\n",
      "Detection Rate = 0.935323\n",
      "F1 score = 0.916819\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.91      0.86      0.88      9711\n",
      "         1.0       0.90      0.94      0.92     12833\n",
      "\n",
      "    accuracy                           0.90     22544\n",
      "   macro avg       0.90      0.90      0.90     22544\n",
      "weighted avg       0.90      0.90      0.90     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "res_test_df = res_test_df.withColumn(sup_prob_col, \n",
    "                                 (col(dos_prob_col) + col(probe_prob_col) + col(r2l_u2r_prob_col))/3).cache()\n",
    "\n",
    "printReport(res_test_df, sup_prob_col, e=0.005, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  8330\t  1381\t\n",
      "attack\t   763\t 12070\t\n",
      " \n",
      "Accuracy = 0.904897\n",
      "AUC = 0.899167\n",
      " \n",
      "False Alarm Rate = 0.14221\n",
      "Detection Rate = 0.940544\n",
      "F1 score = 0.918429\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.92      0.86      0.89      9711\n",
      "         1.0       0.90      0.94      0.92     12833\n",
      "\n",
      "    accuracy                           0.90     22544\n",
      "   macro avg       0.91      0.90      0.90     22544\n",
      "weighted avg       0.91      0.90      0.90     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "res_test_df = res_test_df.withColumn(sup_pred_col, col(dos_pred_col).cast('int')\n",
    "                                            .bitwiseOR(col(probe_pred_col).cast('int'))\n",
    "                                            .bitwiseOR(col(r2l_u2r_pred_col).cast('int'))).cache()\n",
    "\n",
    "printReport(res_test_df, sup_pred_col, labels=labels2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. Ensembling experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some experiments with ensembling and stacking results from different approaches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.1 Linear combination of all models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  8293\t  1418\t\n",
      "attack\t   183\t 12650\t\n",
      " \n",
      "Accuracy = 0.928983\n",
      "AUC = 0.91986\n",
      " \n",
      "False Alarm Rate = 0.14602\n",
      "Detection Rate = 0.98574\n",
      "F1 score = 0.940485\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.98      0.85      0.91      9711\n",
      "         1.0       0.90      0.99      0.94     12833\n",
      "\n",
      "    accuracy                           0.93     22544\n",
      "   macro avg       0.94      0.92      0.93     22544\n",
      "weighted avg       0.93      0.93      0.93     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "# Printing report of the best single model for comparison\n",
    "printReport(res_test_df, kmeans_pred_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  8171\t  1540\t\n",
      "attack\t    81\t 12752\t\n",
      " \n",
      "Accuracy = 0.928096\n",
      "AUC = 0.917553\n",
      " \n",
      "False Alarm Rate = 0.158583\n",
      "Detection Rate = 0.993688\n",
      "F1 score = 0.94024\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.84      0.91      9711\n",
      "         1.0       0.89      0.99      0.94     12833\n",
      "\n",
      "    accuracy                           0.93     22544\n",
      "   macro avg       0.94      0.92      0.92     22544\n",
      "weighted avg       0.93      0.93      0.93     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "# Linear combination of all models \n",
    "printReport(res_test_df\n",
    "            .select('labels2_index', ((3 * col(kmeans_prob_col) \\\n",
    "                                        + col(gm_prob_col) \\\n",
    "                                        + col(dos_prob_col) \\\n",
    "                                        + col(probe_prob_col) \\\n",
    "                                        + col(r2l_u2r_prob_col))/7)\n",
    "                    .alias('voting')), \n",
    "            'voting', e=0.005, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  8136\t  1575\t\n",
      "attack\t    57\t 12776\t\n",
      " \n",
      "Accuracy = 0.927608\n",
      "AUC = 0.916686\n",
      " \n",
      "False Alarm Rate = 0.162187\n",
      "Detection Rate = 0.995558\n",
      "F1 score = 0.939965\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.84      0.91      9711\n",
      "         1.0       0.89      1.00      0.94     12833\n",
      "\n",
      "    accuracy                           0.93     22544\n",
      "   macro avg       0.94      0.92      0.92     22544\n",
      "weighted avg       0.93      0.93      0.93     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df\n",
    "            .select('labels2_index', ((2 * col(kmeans_prob_col) \\\n",
    "                                        + col(gm_prob_col) \\\n",
    "                                        + col(sup_prob_col))/4)\n",
    "                    .alias('voting')), \n",
    "            'voting', e=0.005, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  8147\t  1564\t\n",
      "attack\t    91\t 12742\t\n",
      " \n",
      "Accuracy = 0.926588\n",
      "AUC = 0.915927\n",
      " \n",
      "False Alarm Rate = 0.161054\n",
      "Detection Rate = 0.992909\n",
      "F1 score = 0.939018\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.84      0.91      9711\n",
      "         1.0       0.89      0.99      0.94     12833\n",
      "\n",
      "    accuracy                           0.93     22544\n",
      "   macro avg       0.94      0.92      0.92     22544\n",
      "weighted avg       0.93      0.93      0.93     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df\n",
    "            .select('labels2_index', (col(kmeans_pred_col).cast('int')\n",
    "                                      .bitwiseOR(col(gm_pred_col).cast('int'))\n",
    "                                      .bitwiseOR(col(sup_pred_col).cast('int')))\n",
    "                    .alias('voting')), \n",
    "                    'voting', labels=labels2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.2 Logistic Regression and Random Forest Classifier "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22544\n",
      "Time: 73.75s\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.classification import LogisticRegression\n",
    "\n",
    "t0 = time()\n",
    "lr_assembler = VectorAssembler(inputCols=[\n",
    "                            kmeans_prob_col, \n",
    "                            gm_prob_col, \n",
    "                            dos_prob_col, \n",
    "                            probe_prob_col, \n",
    "                            r2l_u2r_prob_col\n",
    "                            ], \n",
    "                            outputCol=\"features\")\n",
    "\n",
    "lr = LogisticRegression(maxIter=100, labelCol=\"labels2_index\", standardization=False, weightCol='weights')\n",
    "lr_pipeline = Pipeline(stages=[lr_assembler, lr])\n",
    "\n",
    "weights_dict = {\n",
    "    'normal': 1.0,\n",
    "    'DoS': 100.0,\n",
    "    'Probe': 100.0,\n",
    "    'R2L': 100.0,\n",
    "    'U2R': 100.0\n",
    "}\n",
    "\n",
    "udf_weight = udf(lambda row: weights_dict[row], DoubleType())\n",
    "lr_model = lr_pipeline.fit(res_cv_df.withColumn('weights', udf_weight('labels5')))\n",
    "lr_test_df = lr_model.transform(res_test_df).cache()\n",
    "\n",
    "res_test_df = (res_test_df.drop('lr_prob')\n",
    "                    .join(lr_test_df.rdd\n",
    "                    .map(lambda row: (row['id'], float(row['probability'][1])))\n",
    "                    .toDF(['id', 'lr_prob']), 'id')\n",
    "                    .cache())\n",
    "\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  6973\t  2738\t\n",
      "attack\t   541\t 12292\t\n",
      " \n",
      "Accuracy = 0.854551\n",
      "AUC = 0.837947\n",
      " \n",
      "False Alarm Rate = 0.281948\n",
      "Detection Rate = 0.957843\n",
      "F1 score = 0.882317\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.93      0.72      0.81      9711\n",
      "         1.0       0.82      0.96      0.88     12833\n",
      "\n",
      "    accuracy                           0.85     22544\n",
      "   macro avg       0.87      0.84      0.85     22544\n",
      "weighted avg       0.87      0.85      0.85     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df, 'lr_prob', e=0.01, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22544\n",
      "Time: 16.53s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "rf_assembler = VectorAssembler(inputCols=[\n",
    "                            kmeans_pred_col, \n",
    "                            gm_pred_col, \n",
    "                            dos_pred_col, \n",
    "                            probe_pred_col, \n",
    "                            r2l_u2r_pred_col\n",
    "                            ],\n",
    "                            outputCol='features')\n",
    "\n",
    "rf_indexer =  VectorIndexer(inputCol='features', outputCol='indexed_features', maxCategories=2)\n",
    "\n",
    "rf = RandomForestClassifier(labelCol='labels2_index', featuresCol='features', seed=seed,\n",
    "                            numTrees=250, maxDepth=5, featureSubsetStrategy='auto')\n",
    "rf_pipeline = Pipeline(stages=[rf_assembler, \n",
    "                               rf_indexer,\n",
    "                               rf])\n",
    "rf_model = rf_pipeline.fit(res_cv_df)\n",
    "rf_test_df = rf_model.transform(res_test_df).cache()\n",
    "\n",
    "res_test_df = (res_test_df.drop('rf_prob')\n",
    "                    .join(rf_test_df.rdd\n",
    "                    .map(lambda row: (row['id'], float(row['probability'][1])))\n",
    "                    .toDF(['id', 'rf_prob']), 'id')\n",
    "                    .cache())\n",
    "\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  8147\t  1564\t\n",
      "attack\t    91\t 12742\t\n",
      " \n",
      "Accuracy = 0.926588\n",
      "AUC = 0.915927\n",
      " \n",
      "False Alarm Rate = 0.161054\n",
      "Detection Rate = 0.992909\n",
      "F1 score = 0.939018\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.84      0.91      9711\n",
      "         1.0       0.89      0.99      0.94     12833\n",
      "\n",
      "    accuracy                           0.93     22544\n",
      "   macro avg       0.94      0.92      0.92     22544\n",
      "weighted avg       0.93      0.93      0.93     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df, 'rf_prob', e=0.01, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22544\n",
      "Time: 3.06s\n"
     ]
    }
   ],
   "source": [
    "# Adding prediction columns based on chosen thresholds into result dataframes\n",
    "t0 = time()\n",
    "res_test_df = res_test_df.withColumn('lr_pred', getPrediction(0.01)(col('lr_prob'))).cache()\n",
    "res_test_df = res_test_df.withColumn('rf_pred', getPrediction(0.01)(col('rf_prob'))).cache()\n",
    "\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  8147\t  1564\t\n",
      "attack\t    91\t 12742\t\n",
      " \n",
      "Accuracy = 0.926588\n",
      "AUC = 0.915927\n",
      " \n",
      "False Alarm Rate = 0.161054\n",
      "Detection Rate = 0.992909\n",
      "F1 score = 0.939018\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.84      0.91      9711\n",
      "         1.0       0.89      0.99      0.94     12833\n",
      "\n",
      "    accuracy                           0.93     22544\n",
      "   macro avg       0.94      0.92      0.92     22544\n",
      "weighted avg       0.93      0.93      0.93     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df\n",
    "            .select('labels2_index', ((col('lr_prob') + col('rf_prob'))/2)\n",
    "                    .alias('voting')), \n",
    "                    'voting', e=0.01, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  6883\t  2828\t\n",
      "attack\t    17\t 12816\t\n",
      " \n",
      "Accuracy = 0.873802\n",
      "AUC = 0.85373\n",
      " \n",
      "False Alarm Rate = 0.291216\n",
      "Detection Rate = 0.998675\n",
      "F1 score = 0.900095\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       1.00      0.71      0.83      9711\n",
      "         1.0       0.82      1.00      0.90     12833\n",
      "\n",
      "    accuracy                           0.87     22544\n",
      "   macro avg       0.91      0.85      0.86     22544\n",
      "weighted avg       0.90      0.87      0.87     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df\n",
    "            .select('labels2_index', (col('lr_pred').cast('int').bitwiseOR(col('rf_pred').cast('int')))\n",
    "                    .alias('voting')), \n",
    "                    'voting', labels=labels2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.3 Stacking with Random Forest Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "22544\n"
     ]
    }
   ],
   "source": [
    "stack_cv_df = scaled_cv_df.join(res_cv_df.select('id', *[\n",
    "                            kmeans_pred_col, \n",
    "                            gm_pred_col, \n",
    "                            dos_pred_col, \n",
    "                            probe_pred_col, \n",
    "                            r2l_u2r_pred_col,\n",
    "                            sup_pred_col\n",
    "                            ]), 'id').cache()\n",
    "\n",
    "stack_test_df = scaled_test_df.join(res_test_df.select('id', *[\n",
    "                            kmeans_pred_col, \n",
    "                            gm_pred_col, \n",
    "                            dos_pred_col, \n",
    "                            probe_pred_col, \n",
    "                            r2l_u2r_pred_col,\n",
    "                            sup_pred_col\n",
    "                            ]), 'id').cache()\n",
    "\n",
    "print(stack_cv_df.count())\n",
    "print(stack_test_df.count())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "22544\n",
      "Time: 61.79s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "pred_assembler = VectorAssembler(inputCols=[\n",
    "                            kmeans_pred_col, \n",
    "                            gm_pred_col, \n",
    "                            dos_pred_col, \n",
    "                            probe_pred_col, \n",
    "                            r2l_u2r_pred_col,\n",
    "                            sup_pred_col\n",
    "                            ], outputCol='pred_features')\n",
    "pred_indexer = VectorIndexer(inputCol='pred_features', outputCol='indexed_pred_features', maxCategories=2)\n",
    "\n",
    "rf_stack_slicer = VectorSlicer(inputCol='indexed_features', outputCol='selected_features', \n",
    "                               names=selectFeaturesByAR(ar_dict, 1.5))\n",
    "\n",
    "rf_stack_assembler = VectorAssembler(inputCols=['selected_features', 'indexed_pred_features'], outputCol='rf_features')\n",
    "\n",
    "rf_stack_classifier = RandomForestClassifier(labelCol=labels_col, featuresCol='rf_features', seed=seed,\n",
    "                                             numTrees=500, maxDepth=20, featureSubsetStrategy=\"auto\")\n",
    "\n",
    "stack_pipeline = Pipeline(stages=[pred_assembler, \n",
    "                                  pred_indexer, \n",
    "                                  rf_stack_slicer, \n",
    "                                  rf_stack_assembler,\n",
    "                                  rf_stack_classifier\n",
    "                                 ])\n",
    "stack_model = stack_pipeline.fit(stack_cv_df)\n",
    "\n",
    "pred_stack_cv_df = stack_model.transform(stack_cv_df).cache()\n",
    "pred_stack_test_df = stack_model.transform(stack_test_df).cache()\n",
    "print(pred_stack_cv_df.count())\n",
    "print(pred_stack_test_df.count())\n",
    "\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25133\n",
      "Time: 4.43s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "res_cv_df = res_cv_df.drop('prob_stack_rf')\n",
    "res_cv_df = (res_cv_df.join(pred_stack_cv_df.rdd\n",
    "                            .map(lambda row: (row['id'], float(row['probability'][1])))\n",
    "                            .toDF(['id', 'prob_stack_rf']),\n",
    "                            'id')\n",
    "                        .cache())\n",
    "\n",
    "print(res_cv_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22544\n",
      "Time: 3.84s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "res_test_df = res_test_df.drop('prob_stack_rf')\n",
    "res_test_df = (res_test_df.join(pred_stack_test_df.rdd\n",
    "                            .map(lambda row: (row['id'], float(row['probability'][1])))\n",
    "                            .toDF(['id', 'prob_stack_rf']),\n",
    "                            'id')\n",
    "                        .cache())\n",
    "\n",
    "print(res_test_df.count())\n",
    "print(f\"Time: {time() - t0:.2f}s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      \tnormal\tattack\t\n",
      "normal\t  8146\t  1565\t\n",
      "attack\t    91\t 12742\t\n",
      " \n",
      "Accuracy = 0.926544\n",
      "AUC = 0.915876\n",
      " \n",
      "False Alarm Rate = 0.161157\n",
      "Detection Rate = 0.992909\n",
      "F1 score = 0.938983\n",
      " \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.84      0.91      9711\n",
      "         1.0       0.89      0.99      0.94     12833\n",
      "\n",
      "    accuracy                           0.93     22544\n",
      "   macro avg       0.94      0.92      0.92     22544\n",
      "weighted avg       0.93      0.93      0.93     22544\n",
      "\n",
      " \n"
     ]
    }
   ],
   "source": [
    "printReport(res_test_df, 'prob_stack_rf', e=0.01, labels=labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time: 1830.40s\n"
     ]
    }
   ],
   "source": [
    "print(f\"Time: {time() - gt0:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. Results summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best result from a single approach was achieved by KMeans Clustering with Random Forest Classifiers. It gives \n",
    "around ~98-99% of detection rate with reasonable ~14-15% of false alarm rate. F1 score is 0.94, weighted F1 score is 0.93.\n",
    "\n",
    "For improving detection rate ensembling approaches are used. The best of them gives ~99.5-99.6% of detection rate with ~16.1-16.6% of false alarm rate. So there are only about 40-90 attack connections from 12833 (including unknown before) which haven't been recognized."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}