4-TransferLearningVGG16.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VGG16 Transfer Learning for Colonoscopy Polyps\n",
    "\n",
    "In the previous script ([3-Small_CNNs.ipynb](3-Small_CNNs.ipynb)), a classifier with over 95% of accuracy was obtained using only a small CNN (Conv-Conv-Conv-FC).\n",
    "\n",
    "This script, I will try to apply Transfer Learning using pre-trained VGG16:\n",
    "* Freeze the bottom part of the VGG16 - keeping the pre-trained weights in the convolutional layers.\n",
    "* Save outputs from bottom moodel - because the bottom model is constant, it will be calculated only one time the outputs for the use of training and validation subsets.\n",
    "* Train only the FC layer - The outputs of the bottom model will be the inputs for the trainable model.\n",
    "* The best model will be saved on disk.\n",
    "\n",
    "Let's import the needed libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\program files\\python35\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dropout, Flatten, Dense, Activation, BatchNormalization\n",
    "from keras import applications\n",
    "from keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "from keras import backend as K\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "import keras\n",
    "from keras import optimizers\n",
    "from matplotlib import pyplot as plt\n",
    "from IPython.display import clear_output\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from numpy.random import seed\n",
    "from tensorflow import set_random_seed\n",
    "import time, os\n",
    "from __future__ import with_statement\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set calculation parameters such as image dimensions, folders for dataset, folder for saved models, number of training/validation images, training epochs and batch size.\n",
    "\n",
    "**Important note**: `batch_size` is used in the next training / dataget lines and the intermediate bottom outputs will depend on this value!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dimensions of our images\n",
    "img_width, img_height = 150, 150\n",
    "\n",
    "# Train & validation folders\n",
    "train_data_dir      = 'data_polyps/train'\n",
    "validation_data_dir = 'data_polyps/validation'\n",
    "\n",
    "# Folder to save the models\n",
    "modelFolder = 'saved_models'\n",
    "\n",
    "# Sample parameters\n",
    "nb_train_samples      = 910 # number of training images (all classes)\n",
    "nb_validation_samples = 302 # number of validation images (all classes)\n",
    "\n",
    "# Initial training params from the previous small CNN script\n",
    "epochs     = 50 # training epochs\n",
    "batch_size = 64 # batch size for training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the paths & filenames for the outputs for training and validation using the bottom pre-trained VGG model. These outpus will be calculated only one time and use as inputs for the training of the FC layer (top model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training results for our dataset using pre-trained VGG16 (Convolutional layers only, without top FC)\n",
    "bottleneck_features_train_path = './saved_models/transferVGG16_bottleneck_features_train.npy'\n",
    "\n",
    "# Validation results for our dataset using pre-trained VGG16 (Convolutional layers only, without top FC)\n",
    "bottleneck_features_validation_path = './saved_models/transferVGG16_bottleneck_features_validation.npy'\n",
    "\n",
    "# Top top model weights to used for future calculations for layer initialization\n",
    "top_model_weights_path = './saved_models/transferVGG16_bottleneck_fc_model.h5'\n",
    "\n",
    "# Earlystoping saved model\n",
    "earlystoping_path = './saved_models/transferVGG16_earlystop_model.h5'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Correct the order of the channels if needed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "if K.image_data_format() == 'channels_first':\n",
    "    input_shape = (3, img_width, img_height)\n",
    "else:\n",
    "    input_shape = (img_width, img_height, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1 - Calculate the outputs of bottom VGG model for training and validation\n",
    "\n",
    "You could observe that in `ImageDataGenerator` we do not use the data augmentation as in the previous script. So, we use only the original dataset of colonoscopy polyps/non-polyps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_bottlebeck_features(batch_size):\n",
    "    # Calculate the outputs for training and validations sets using bottom pre-trained VGG16\n",
    "    # Input: batch_size - it will be used to save to file\n",
    "    # Output: number of the examples for the saved outputs (it could be different from the original dataset!)\n",
    "    \n",
    "    # set numpy and tensorflow seeds for reproductibility\n",
    "    #seed(1)            # numpy seed\n",
    "    #set_random_seed(2) # tensorflow seed\n",
    "    \n",
    "    # No data augmentation!\n",
    "    datagen = ImageDataGenerator(rescale=1. / 255)\n",
    "    \n",
    "    # Build the VGG16 network - get VGG16 trained for imagenet without the top layer\n",
    "    model = applications.VGG16(include_top=False, weights='imagenet')\n",
    "\n",
    "    # Generate training data from train folder\n",
    "    # generator = datagen.flow_from_directory(\n",
    "    generator = datagen.flow_from_directory(\n",
    "        train_data_dir,\n",
    "        target_size=(img_width, img_height),\n",
    "        batch_size=batch_size,\n",
    "        class_mode=None,\n",
    "        shuffle=False)\n",
    "  \n",
    "    # Calculate the output of bottom VGG16 using training data\n",
    "    bottleneck_features_train = model.predict_generator(\n",
    "            generator, nb_train_samples // batch_size)\n",
    "    # Save the traing outputs\n",
    "    np.save(bottleneck_features_train_path, bottleneck_features_train)\n",
    "\n",
    "    # Generate validation data from train folder\n",
    "    generator = datagen.flow_from_directory(\n",
    "        validation_data_dir,\n",
    "        target_size=(img_width, img_height),\n",
    "        batch_size=batch_size,\n",
    "        class_mode=None,\n",
    "        shuffle=False)\n",
    "            \n",
    "    # Calculate the output of bottom VGG16 using validation data\n",
    "    bottleneck_features_validation = model.predict_generator(\n",
    "            generator, nb_validation_samples // batch_size)\n",
    "    \n",
    "    # Save the validation outputs\n",
    "    np.save(bottleneck_features_validation_path, bottleneck_features_validation)\n",
    "    return (bottleneck_features_train.shape[0], bottleneck_features_validation.shape[0]) # return the new shapes if modified"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the training and validation outputs from the bottom VGG16 model. Get the new values for the number of examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 910 images belonging to 2 classes.\n",
      "Found 302 images belonging to 2 classes.\n"
     ]
    }
   ],
   "source": [
    "nb_train_samples, nb_validation_samples = save_bottlebeck_features(batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation**: The initial number of training / validation examples in the subsets (910 and 302) could be modified by the `nb_train_samples // batch_size` and `nb_train_samples // batch_size` when use ` model.predict_generator`! If this division will give a rest of 0, you will not have problems. \n",
    "\n",
    "For this reason, the saved output matrices from bottom VGG16 for training and validation will contain different number of examples. Thus, it is safer to use these new dimensions for the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New no of train samples      =  896\n",
      "New no of validation samples =  256\n"
     ]
    }
   ],
   "source": [
    "print('New no of train samples      = ', nb_train_samples)\n",
    "print('New no of validation samples = ', nb_validation_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2 - Train the top FC model by using the outputs from the pre-trained bottom VGG16\n",
    "\n",
    "The saved outputs for training and validation from the bottom VGG16 will be used as inputs for the FC layer (top model to train)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a function to plot the training and validation loss during the training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PlotLosses(keras.callbacks.Callback):\n",
    "    # Show the live training and validation loss during the training\n",
    "    def on_train_begin(self, logs={}):\n",
    "        self.i = 0\n",
    "        self.x = []\n",
    "        self.acc = []\n",
    "        self.val_acc = []\n",
    "        self.fig = plt.figure()\n",
    "        self.logs = []\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs={}):\n",
    "        self.logs.append(logs)\n",
    "        self.x.append(self.i)\n",
    "        self.acc.append(logs.get('acc'))\n",
    "        self.val_acc.append(logs.get('val_acc'))\n",
    "        self.i += 1\n",
    "        clear_output(wait=True)\n",
    "        plt.plot(self.x, self.acc, label=\"ACC_Train\")\n",
    "        plt.plot(self.x, self.val_acc, label=\"ACC_Val\")\n",
    "        plt.legend(bbox_to_anchor=(0., 1.02, 1., .102), loc=3,\n",
    "                   ncol=2, mode=\"expand\", borderaxespad=0.)\n",
    "        plt.grid()\n",
    "        plt.show()\n",
    "\n",
    "plot_losses = PlotLosses() # initialize the class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, I will define a function to train the top model using different parameters such as number of epochs, batch size, optimizer type, drop rate and callback set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_top_model(epochs, batch_size, optimizer, drop_rate, callbacks, summary):\n",
    "    # Training top model\n",
    "    # Input:\n",
    "    # - epochs = no of training epochs\n",
    "    # - batch_size = size of sample batch during training\n",
    "    # - optimizer = string to define the optimizer (ex: 'adam', 'optimizers.RMSprop(lr=2e-4)')\n",
    "    # - drop_rate = rate of dropped neurons\n",
    "    # - callbacks = list of callbacks options\n",
    "    # - summary = if True, show the model summary (True, False)\n",
    "    \n",
    "    # set numpy and tensorflow seeds for reproductibility\n",
    "    #seed(1)            # numpy seed\n",
    "    #set_random_seed(2) # tensorflow seed\n",
    "    \n",
    "    # Load training outputs from the pre-trained bottom VGG16\n",
    "    train_data   = np.load(bottleneck_features_train_path)\n",
    "    train_labels = np.array([0] * (nb_train_samples // 2) + [1] * (nb_train_samples // 2))\n",
    "\n",
    "    # Load training outputs from the pre-trained bottom VGG16\n",
    "    validation_data   = np.load(bottleneck_features_validation_path)\n",
    "    validation_labels = np.array([0] * (nb_validation_samples // 2) + [1] * (nb_validation_samples // 2))\n",
    "\n",
    "    # Construct the top model using two parts: outputs from the bottom VGG16 + a FC layer\n",
    "    model = Sequential()\n",
    "    model.add(Flatten(input_shape=train_data.shape[1:]))\n",
    "    model.add(Dense(256, activation='relu'))\n",
    "    model.add(Dropout(drop_rate))\n",
    "    model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "    # Compile the graph\n",
    "    model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "    # Fit the model using the live plot of the loss\n",
    "    model.fit(train_data, train_labels,\n",
    "              epochs=epochs,\n",
    "              batch_size=batch_size,\n",
    "              validation_data=(validation_data, validation_labels),\n",
    "              verbose = 0,\n",
    "              callbacks=callbacks)\n",
    "    # Save the final model\n",
    "    model.save_weights(top_model_weights_path)\n",
    "\n",
    "    # Print model summary\n",
    "    if summary == True:\n",
    "        model.summary()\n",
    "    \n",
    "    # Evaluate final test loss and accuracy scores\n",
    "    score_val = model.evaluate(validation_data, validation_labels, verbose=0)\n",
    "    score_tr  = model.evaluate(train_data, train_labels, verbose=0)\n",
    "    print('Train loss    :', score_tr[0])\n",
    "    print('Train accuracy:', score_tr[1])\n",
    "    print('Validation loss    :', score_val[0])\n",
    "    print('Validation accuracy:', score_val[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's traing the FC model using `epochs = 50`, `batch_size = 64`, `optimizer = 'adam'` (the same as the previous script for small CNNs), original FC VGG16 layer topology, saving the last model and live plotting of loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEQCAYAAABLMTQcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl81NW9//HXmWSybySBJCRACATCjhAQATW4gaCAWhXcqFrtvZVKa6vXXrvcYnttf1Vb22J7ta5URaQVUFG0SiqoKCCyCwQQCBCykmSyT+b8/jgTSMIkGcIkk5n5PB+PPJKZ+c53zpkk7++Z8z3nfJXWGiGEEP7F4u0CCCGE8DwJdyGE8EMS7kII4Yck3IUQwg9JuAshhB+ScBdCCD8U7K0X3rJlS5/g4OC/ASORg4wQQnTEAey02+3fGT9+fGFHG3st3IODg/+WnJw8rHfv3mUWi0UG2wshRDscDocqKioaXlBQ8Ddgdkfbe7PFPLJ3794VEuxCCNExi8Wie/fuXY7p7eh4+y4uT7uvLcEuhBDuc2amW7ktfd1CCOGHAj7cX3755Til1PitW7eGNd23ffv20EsvvXRw//79R2ZkZIyYOXNmxtGjR4MB1q1bF5GdnT00PT195MCBA0fcfPPNAyorK896H5966qmErKys4VlZWcOtVuu4IUOGDM/Kyhr+ve99L9XdsuXl5VlnzZqV4ZmaCiFc6aoM2Lt3b0hSUtLoxsbGFvdnZWUNX7duXURb5fnjH/+YcMcdd/Q/33p57YRqT7Fs2bL4cePG2ZYuXRp/wQUXHK+urlbXXntt5mOPPXb0lltuKQd46623ogsKCoIBbr311kEvv/zywSuuuKLK4XDw0ksv9Tp16pQlOjra0Xy/ixYtKlm0aFEJQGpq6qh///vf+1JSUuytX7+hoQGr1eqybIMHD2545513Dnq80kKI07oqA4YOHVqfkpJS/95770XNmjXLBrB169awqqoqy7Rp06q7ul4B3XIvLy+3bN68OeqFF1745s033+wF8Mwzz8SPGzfO1vRLBbj22msrJ0yYUPvEE0/0uemmm0quuOKKKgCLxcKdd95Z1q9fv7NCuz33339/31tuuWXA5MmTM2+88caBu3btCh0/fvzQYcOGDR8xYsSwjz76KBJg586doVlZWcMBnnzyycQZM2ZkTJ06NXPAgAEj77vvPrc/AQghXOvqDPjWt75V+uqrr8Y33V66dGn8ddddVwrw6quvxo4ePTpr2LBhwydPnjyk6ZOBp/SIlvuDK7b121dQ2ebHlM4Ykhxd/btvjTna3javvPJKXE5OTvno0aPr4uLiGjds2BCxc+fO8HHjxrk8qu7evTv8jjvuKPFE+Xbs2BHx+eeffx0REaErKyst69ev3xcREaG3bt0atmDBgvTt27d/3fo5e/bsifjqq692W61WPXjw4FEPPvhgYXp6eoMnyiOEV628rx+Fuz2aAfQZXs3cJV7NgDvuuKN0/PjxwxsaGo5YrVZWrlzZ6/XXXz8AcOWVV9rmzZv3tcVi4cknn0xcvHhx8rPPPpt/bpVsW48Id29Zvnx5/KJFiwoBbrjhhtKlS5fGd/QcT5k5c2ZZRESEBqitrVV33333gD179kQEBQXpo0ePhrp6ztSpUyt69erlAMjIyKg5cOBAiIS7EJ3X1RnQv39/e2ZmZu3q1atjUlJSGoKDg/WECRNqAQ4dOhQyd+7ctKKiImt9fb2lX79+dZ587R4R7h21sLtCQUFB0MaNG2P27dsXvnDhQhobG5VSSj/00EMnPv744yhXzxk2bFjN5s2bI2677bZT5/v6kZGRp/vnHn300aS0tLT6lStXHqqvr1fR0dEXuHpOaGjo6aGjQUFB2O12db7lEKJH6KCF3RW6KwNuvPHG0tdeey2+T58+DTfccENp0/0LFy7sv2jRooJbb721/O23345evHhxX0/Uq0nA9rkvXbq01/XXX19y/PjxHceOHdtRUFCwPS0trX7YsGG1W7ZsiVq2bFls07YrVqyI+eKLL8J//OMfFy5fvjyhqU8c4Omnn44/cuTIeR0ky8vLg1JSUhosFgtLlixJkKtjCdH1uisDbr/99rJ169bFrlq1Kn7BggWnw72ysjKof//+DQAvvvhigqfrF7Dh/sYbbyRcf/31Zc3vmzNnTtnSpUvjV61albdkyZI+AwYMGDlo0KARL774YmJKSkpDv3797C+//PLBBx98MC09PX1kRkbGiA0bNkQ3dZV01gMPPFC4dOnSxDFjxmQdPnw4JCQkRNJdiC7WXRmQmJjYOHbsWFtiYmJDVlZWfdP9jzzyyPH58+cPGj9+/NCEhIRzGpThDuWtVuK2bdu+GTNmTLFXXlwIIXzUtm3bEseMGZPe0XYB23IXQgh/1iNOqPq6p556KuEvf/lLUvP7JkyYYFu6dOkRb5VJCNF9emIGSLeMEEL4EF/olnE4HA4ZyieEEG5yZqZbAzi8Ge47i4qKYiXghRCiY86LdcQCO93Z3mt97na7/TsFBQV/KygokMvsCSFEx05fZs+djb3W5y6EEKLrSItZCCH8kIS7EEL4Ia/1uScmJur09PROPbeqqorIyMiON/QzgVpvCNy6S70Dizv13rJlS7HWundH+/JauKenp7N58+ZOPTc3N5ecnBzPFsgHBGq9IXDrLvUOLO7UWyl12J19SbeMEEL4IQl3IYTwQxLuQgjhhzoMd6XU80qpQqWUy1lRyvijUipPKbVdKTXO88UUQghxLtxpub8IzGjn8auBTOfXvcBfzr9YQgghzkeH4a61/hgobWeTOcDL2tgIxCmlUjxVQCGEEOfOE33uqUDzi9vmO+8TQgjhJZ4Y5+5qVUeXC9Yope7FdN2QlJREbm5up17QZrN1+rm+LFDrDYFbd6l3++oaNduLGjla6XoVXIuCKKsiJkQRHeL8HqqIDIbaRqio01Q2aPO93vxsb2NB3UirIjVKkRplIS5UoVTnFrTVWqMBi4vne/L37Ylwzwf6NbudBhx3taHW+hngGYDs7Gzd2UkKMsEh8PhC3Y+dqmHjgRIqahuIjwwhMSqU+MgQEqJCiI8IITjIvQ/KeYWVvPzZYd7efoKYoCCuHNOHiwYlkJ0eT0yY9ZzKVF7TwKZDpew+UUGJrY7iqnpKbfWUVtVTUlVHeU0Djh64dmBKhIXLRyUyKSOBCzMSiI8MOf1YbUMjuXsLeXv7CT76upDq+kYAXGVtZ9ZFdGc/MWHBZCZFMyQpirReES6DutHhoLSqgZKqOkqr6im21VPq/PlXc0dy84T+Zz3Hk3/nngj31cBCpdQy4EKgXGt9wgP7FaJHO1Few2cHSth4sISNB0s5Ulrd7vbJMWFMHBjPRYMSmJSRQHpCxOnWn73Rwb/2nOTlzw7z6YESQoIsXDk8ibz8k7z06WGeXX8Ii4JRqbFMykhgcJ8oc9CIDCXBeQCJCAmmoraBzd+UOstVyq7j5afDOyYsmIQos/2AhAjGDYgjLiKEoE62QLuK3aFZv/MQyzfn89JnZjJmVnI0kzISKK2q5197TlJd30h8ZAhzL0jlmlEpTBwY7/LgaW90cKqmgRKbOZiVOA9sZdX1RIUGtzgAJ0San0OCXR+ES2x17DtpY39hJftOVrLvpI33dhZQVt3QZl0iQ4JIcL5GalwYo1JjSIgKZWhyjGferHZ0GO5KqdeAHCBRKZUP/AKwAmit/wqsAWYCeUA1cGdXFVaI83GwyEZFrZ2x/eLc2r6h0cG7OwvIL6um1FZPSZXzy1ZHsa2OkxV1AMSGW7lwYDzfnpzOpIwEkmJCKatuaqmZ7Uuq6jlQVMVnB0tYvc18sE2OCWNSRjwpceGs3HqME+W1pMaF89CModyc3Y+EqFByc3OZNOVivjxSxsaDpWw8UMLznxyiofHsJmm4NYg6eyMODSFBFi7oH8f3L8tkUkYCF/SPI8wa5Lk3s4tNCi9g8tRL2HHslKn3wRKWbTpCuDWIOWNTuWZ0Che2EejNBQdZSIwKJTEqFIg+rzIlRIVyUVQoFw1KOH2f1pq6NvpxlILQYO+95x2Gu9Z6fgePa+A+j5VICA/LK7Txp4/2s3rbcbSGuWP78j+zRxAXEdLmc/YWVPKjN75i57EKwASn6WoJoU90KFnJMQzvG8OkjHiGJcdgsbRs/SZEhTK4z9n71VpzsLiKjQdL+OxACRvySii21XFxZiK/nD2Cy4clEdRqX2HWICYPSmTyoES40nRLFFWaA4w5eJw56ESEBjMpI55x/Xv5VJi7EhJsYfyAeMYPiOe+aYOxNzpQSp31/niTUqrHvs9eWzhMBKbSqnqOlFYzPCWmzY+/nnKgyMafPjShHmYN4ruXDCIk2MLT6/L45EAJj103iiuGt7hgPfZGB//38UH+8K99xIRZWXLLOKZl9SYixDP/KkopBvWOYlDvKG69cABaa2x1dqLPoS89zBpEv/gI+sVHeKRMvsLdcxbCkHAX3WbL4VLufXkLJVX1hFuDyE7vxaQM0/88Oi0W6zn8856qriev0EZtw9kfie0OB6u+Os6qr44RGhzEPRdncO8lGSREhQIwfUQSP1q+je+8vJnrx6Xyi2tGEBthZf/JSn70xja255cza3QKi2ePOP2crqKUOqdgF8JdEu6iW/zzy3we/scO+saF8cisYWzPL2fjwRJ+t3YvABEhQYxOi6VPdJg5wRUZcvpEVFRoMOuONJC7etfpE1nFtrp2Xy/MauE7zlBPbBXQI/rGsnrhVP68Lo8l6/L4JK+YWaP68veNh4kKC2bJLeOYNVrm4QnfJuEuupTDoXn8/b08nXuAizIS+Mtt44iLCOH6cWmAGYHwxSFzwmzHsXK25Z+i1FZPZZ39rH1FhhxlcFI0OUN7MyQpisw+0USFuf4THtQ7qsXwudZCgi08cOUQrhqexI/f2Mbznxxi5qhkFs8ZedbBQAhfJOEuukx1vZ0fvv4Va3edZP7EfiyeM/KsrpeEqFCuHpXC1aNatpTr7I2nTxZW1to5tm8b10+fdtaJy/M1MjWWVQunkFdoY0TfWI/uWwhvknAX50RrzcmKOmf3SCWHiqsIDQ5yjhMOcY4ZDiUkyMLD/9zOnhMV/Oya4dw1Jf2cZvSFBgeREhtOSmw4ALlHLR4P9uavJcEu/I2Eu2jX8VM1bDxYwqZvyk4HemXtmS6T2HAr9kYHVc5Zgs1FhQbz3IIJTMtyMSZQCNGlJNxFCyfKa06PwW4+6zImLJislBjmjO3LkKRoMvtEk5kUdbp/urahkRLn1Pam6dbjB/RiQELgXeRYiJ5Awl0AUGyr4+erdrJmRwFw9qzLrOTodrtFwqxBpMaFkxoX3l1FFkK0Q8Jd8M72E/xs1U5stXbuv2ww00cmk5Uc06NmAgohzo2EewArsdXx81W7eGfHCUanxfL4jWMYknR+628IIXoGCfcA9e6OE/x05U4qaht4cPpQvntJhkzvFsKPSLj7KK01u45X8M6OE+TuLcIapJwzO0NbDEusb3S0WNGwtKqOwoo69hfaGJUay6s3TmJosrTWhfA3Eu4+RGvNzmPlrNlxgnd2nOBwSTXBFsXEgfFYgyyU2OrZV1BJSVX9WcuQRocFn57Sn54YyY3Zadw5ZeA5recihPAdEu49nNaa3ScqWLPjBCs+r+Hk2g0EWRSTByXwvZxBXDU8mV6tptlrramqb6TUVk9IsIVekVavristhOh+Eu49kNaarwsqeWe7aaEfKq7CoiAr3sIPZoxk+ojkdtdNUUoRFRpMVKj8eoUIVPLf30Nordl78kygHywygX7RoATuuTiD6SOS2LH5M3Imnn3dRSGEaE3C3cv2nazk7e0neGf7cQ44A/3CgQncPXUg00ckywqFQohOkXD3kuWbjvLs+oPsL7SdDvQ7pwxkxkgJdCHE+ZNw94K/rT/Ir97Zw5h+cTw6dyQzRiTTO1oCXQjhORLu3awp2GeNSuGpeWNl4pAQoktIsnSj5zYc4lfv7GHmqGT+IMEuhOhCki7d5PkNh3j07d1cPTKZp+ZdIJOHhBBdShKmG7z4ySEWv72bGSOS+eN8CXYhRNeTPvcuUmdv5FBxFe/vOsmTH+xj+ogk/nSLBLsQontIuHdSo0Nzqtq5IJetniJbHXmFNvY7L0X3TUk1jQ4NYIJ9/jgJdiFEt5FwPwc7j5Xz0IrtFFTUUlZdj9YtH7coGJAQSWafKK4emUJmUhRDkqLJSo4+p4tDCyHE+ZJwPwe/fe9rTpTXMHNUyukVFs0yu+bnAQkRhFllgS4hhPdJuLtp38lK1u8v5sHpQ7lv2mBvF0cIIdolncBuen7DIcKsFm6RhbuEED5Awt0NxbY6/rn1GNePSztr7XQhhOiJJNzd8MrGI9TbHdw1ZaC3iyKEEG6RcO9Anb2RpRsPM21obwb3ifJ2cYQQwi0S7h1Y/dVxim113D01w9tFEUIIt0m4t0NrzXMbDpGVHM2UwQneLo4QQrhNwr0dnx4o4euCSu6aMlAmIQkhfIqEezue23CIxKgQZo/t6+2iCCHEOZFwb8OBIhsffV3IbZMGyKxTIYTPcSvclVIzlFJ7lVJ5SqmHXTw+QCn1oVJqu1IqVymV5vmidq8XPjlESLCF2yYN8HZRhBDinHUY7kqpIGAJcDUwHJivlBrearPHgZe11qOBxcBjni5odyqrqmfFlnzmju0rF6sWQvgkd1ruE4E8rfVBrXU9sAyY02qb4cCHzp/XuXjcpzy34RC1DQ7umiqTloQQvsmdcE8Fjja7ne+8r7ltwA3On68DopVSPjl28B9b8vnzujzmju1LVnKMt4sjhBCdonTrRclbb6DUjcB0rfV3nLdvByZqrb/fbJu+wJ+BgcDHmKAfobUub7Wve4F7AZKSksYvW7asU4W22WxERXl+tujWQjt/2lpHVryFH44Pw2rpWcMfu6reviBQ6y71Dizu1HvatGlbtNbZHe5Ma93uF3ARsLbZ7Z8AP2ln+yggv6P9jh8/XnfWunXrOv3ctnx+sEQPeWSNvvZP63VlbYPH9+8JXVFvXxGodZd6BxZ36g1s1h3kq9barW6ZTUCmUmqgUioEmAesbr6BUipRKdW0r58Az7ux3x5j9/EK7n5pE6m9wnnh2xOICpVl7oUQvq3DcNda24GFwFpgD7Bca71LKbVYKTXbuVkOsFcptQ9IAn7dReX1uMMlVSx44QuiQoNZeveFJMjoGCGEH3Criaq1XgOsaXXfz5v9vAJY4dmidb3Cilpuf+4LGhodvHbPRaTGhXu7SEII4REBO0O1pr6RO1/cRLGtjhe+PYHBfaK9XSQhhPCYgOxc1lrzk39uZ/eJCp5bkM0F/Xt5u0hCCOFRAdlyf+GTb1j51XEeuGIIl2Ulebs4QgjhcQEX7p8dKOHXa/Zw1fAk7ps22NvFEUKILhFQ4X78VA0LX/2S9IQInrhpDJYeNklJCCE8JWDCvbahkf/4+xbq7A7+7/ZsosOs3i6SEEJ0mYA4oaq15qcrd7I9v5xnbh8vF7oWQvi9gGi5/33jYVZsyef+ywZz1YhkbxdHCCG6nN+He3W9nUff3kPO0N784Ioh3i6OEEJ0C78P97xCG/WNDuZN6CcnUIUQAcPvw33fSRsAmUkyA1UIETj8Ptz3F1YSEmRhQHyEt4sihAgUx7bAqzfDqSNeK4Lfj5bZf9JGRu9IgoP8/jgmxLlr72I9SroxO6XssAn2qiKor4I7VoOl+/PH/8O9sJIxaXHeLobwJwU74fVbIXEoTP9fSOwBM51ry+GNO6HiGFz1a8i8ooPtK+Dfv4UvnoXGurMfV0Fw0X1wxS+7N5iOfQnLFzDCmgKjB0C8j13HuLbcBHtjPUz9IWz4PWx6Fi78brcXxa/DvbreztHSGm4c38/bRRH+4mAuvH47WMPh8Kfw9CS46HtwyYMQ2s55nfoq848f09fzZSrPh1duhOJ9EJMKr9wAQ2fC9F9DfEbLbR0O2PYa/Ot/TMty9E1nbwNQvB8+/SOUH4W5fwVrmOfL3dq+9+GNBRAWS3zpNlhyIUy534RkSGTbz6uzQV0lxKS49zpaw6nD5r0K8uBkxsYGeOPbULIfbvsnDLzENAQ++AUMurzbGwF+He4HCqsAyJRJS8ITti2DVfdB4hC4dQVYguHDxfDJU+axKxfDqJvOtHTLvjGBtX8tHFpvWsiJQyDzKhgyHfpfdP7hUrDTBHu9DW77B/SfDBufho9/Z8Jx8vdh6gMQGgX5W+DdB01/cNpEuOV1SB3ner9aQ8oY+OBnUHkS5r0CEfHnV9b2bHkR3n4AkkfCLW/w+acbmGx7z9Tjq1fhqkdhxPVnuopKDsD+92HfWjj8iWkp9x4GQ66CzOnQ70IIahZvdTZzYN6/FvZ/AJUnIDQWBl9mts+8EiITO19+rWHNg3DgI5j9Z8i41Nw/+0+mAfDmd+GutS3L1MX8Otz3nawEZKRMwDm5Gwp2nNtzrOEwYApEJpz9mNaw/nH46FemNXbz3yEs1jw2dwlk3wnvPmT+gTc9B/0mmgAp3mu2iR8EE+42LcW8f8EXz8Bnf4bQGBg0DTJywOqiZaoshFc3tF3mA+vMp4jQaLjrPUgaYe6f+gMYfbNpna9/woRj/0mw602ISoLr/q/lQcgVpUyrOTYV3vwPeH66OaD1GuDGm3kOtDbv6/rHYfCVcOOLEBpFfWgCTH8Wsu8y7+2Ku8x7mzLWBHRJnnl+4hCYeK+pV94H8NkSc7ANizWt5aQR8M2GMweAkGgT6AOmQMF283va9SagIHW8OehmXmUObOdyzmHj07DlBfMpY9ztZ+6PSYFZT8A/7oZPn4KLf+TJd69dfh3u+wttWIMUAxJkpEzA2P4GrPxPcLQTim1RFkjNNv/gQ6ZD0khwNMI7D8CXL5nAnP1nCA5p+by0bLj7X87ujl/A8a2QPgXGf9vsJ2HQmW0nL2zZitz3Puxe1WaRLgTIexyGzDCt0v6Tzet/9RqsXmj6/W99w4RwczEpcP3/nQnHPW/DlEUddx+1NvIGiEqGZfPhuSvhluXQd6z7z2+PvR5Wfx+2L4Nxd8Cs35/dsh1wEdybC1++bD4l5W+C9ItNoGde1bJPfsr95lzCwXXOT0zvw65/QkLmme37X9Ty9+dwQMG2M5+w1v0vrPu1qXPmleZ9z8gxn3za8vU7sPYRGDYbLvv52Y+PvAH2vAXrHjNlSB7V+ffsHCjd3tnyLpSdna03b97cqefm5uaSk5PT4XZ3v7iJ/LIa1v7wkk69Tk/jbr39UYd119qcvPrwl6ZVNuvJc+vyqC4xrep9a+HEV+a+mFSI7G1uX/wjuOxnHbfm7PXmwNJeH3Hrcp86Ag67i33Vsv+DF8jUB03rs7HOtDz7joVv1sPAS+HmpWc+RbTF4TBlCj6P6wMXfg2vfAuqS2H8AtMldb7yN8GRz2DaT+GSH7d4b13+vu115mAb4mZjzeGAmtJz626xFTr/Dt4zn4zqKiAoxPxNJY8EWv3+tQM2Pw99hsGCt9suW1WJ6Z6J6gP3fNTm78Kd/3Gl1BatdXZHVfH7lvuotA7+8IXva7Sb1unm50y/7HV/PfcgSxhkulOm/TdUFpiP6/vXwontcM0fTNeLO4JDgJAONztNqXa7Oo6lXUNmTo45IXvw36Y1eujf5lPB1b87+1OEKxYLWM7zwu99suA7/4LlC2DzC+e3rybWcJj7Fxh7i3vbn+vv1GI59370qD6mPGNvMSdIj3xmDvj7P4AjG10/J2EwzHut/YNOZALM/iO8Ns+MUrrcRQvfw/w23GvqGzlaVs0N49K8XRT/dXK36eOc9J/OVo0X1FfBirth37sw+X7PDN2LTjb9ps37Tr0tJBKyZpovb4lOhrvXeu/1u1uQ1ZxjGXiJGXl0voZeDWNvM58wh1wN/Sac/z7b4bfhfqDIhtaQmSQjZTyupgxyf2PGSOtGKD8CC97q/nLYiuDVm0y3yczHYeI93V8GIc7FjMfgxDaoKuzyl/LbcN9faEbKDJFwd091KeQ+ZsZup081J37Sp7b8KOxohK1LzYmtmjIYfydEJMDH/w+OfA79L+yesmpt+kTXPGTGat/8d8ia1T2vLcT5CIuB737cLRPD/Dbc951sGinj5omtQOVoNGOMP3rUTLLpN8nc/vyvZnhexqUm6GP7wUeLTauj/2S4+reQMtp0i2x+3gT8bf9w4/UcgAZLUOfKW7wf3nvYnPRKHALffgfSxnduX0J4QzfN+PXbcN9/0sbAxEissqZM2w5/alq/J3eY4WVX/9aMC26oMZNumobq7V1jto/uCzc8Z4Z2NY1sCIk009Q//KWZHJPaTtBqDSvuNEMFb1luTtK5KcheDe//FDb+BawRZtr/xHs9O8NQCD/iv+FeWMnIvn40UkZroiv2wkEPLOakG2HrK7BzBcSkmYkjw+eeCWxruBlTPeQqmKmh6Gs4ucuM+XU13nfiPWbiyMdPwPxX237dr16F3SshOByevwrmvWq6ftrjcMD215n4xU+gvgwuuA0u/4UZ1SCEaJNfhnttQyNHSqu57oLUjjf2FbmPMf7L38KXHtpfUChc+l8w5QftD+FSyozh7TOs7W1Co2HS9yD3f810eFcjZ04dNd0pA6aaWZ2v3ARLrzND4UZ9y/V+j22Bd/8L8jdRF51J6B3/kC4YIdzkl+GeV+gcKdOnByw7cOAjM0Mu7jwWLzu2BT5+nMLeU+kz6789U674DPcXWnLHhd81U+o//h3c9FLLxxwOWPU9M+Fj7hLolW6G1C271UzLrjhmhjE2fXKwFZlunq1/N5OI5izhy1N9yZFgF8Jtfhvu0AOGQeZ9CH+/HiISTR9zZ8Kpocas7RGdzL4h/0mf9CmeL6cnhMeZ7pn1T0LRXug99Mxjm/4Ghz6Ga58ywQ4Q3gtuf9PU7YOfm1maV/3anJzN/Q00VJm+/EsfMjMwc3O9USshfJZfnm3cd7KSYIsi3ZsjZWpOwaqFZvZaSCS8OAu+XnPu+/nwUbOU65wl2K09fFjnpPvMyc71T5y5rzjPhPfgK2HcgpbbB4eaE7ST7zcHgMeHwNqfmIPgf35mJo50NLVeCOGSX4b7/kIb6YmRhAR7sXrv/hfYTsL1z5hp233Qjx3iAAARFUlEQVSyzAUevnjW/X18s8GsNjfhO2b1wJ4uMgEm3AU73jBLsjbaYeV/mBCf/SfX67JYLGY515mPmyUA5r1m1sLuPaT7yy+EH/HbbplhKV7sb9/zllnp7pKHzgwN/PY75ko5a35sLq5w+S/aH+9aV2lWN+yVbtYJ9xUXfd8cwDY8afr18zeZ1nlH/fsT75EZpkJ4kN+Fe21DI4dLqrh2TBdc8cYdtiJ46weQPNosr9okJNIM/VvzY/jkDybg5z7d9mJIax8xI0zues/9FQZ7gugk0/2y+TlAwfA5Zly8EKJb+V24HyyqwqE9uOyA1lC427ky3PsmcKcsMutkt157Wmt4+wdmmdDr3jp7xb6gYLjm92bkzIeLIf8Lczm01lP9971v1g+fsshcZMHXTFlkLlwQFmfW6JYLLQvR7fwu3JvWlHFrGGRthVne1ZXSg2dmaFbkm/uSR5tgfvdBE15X/9asGNdk+3L4+m2zMmHScNf7VcqsDd47yyyd2mKqfw4Mvhz+/f/MJcOmPeJutXuW2FSY/5q54IGrKxsJIbqc/4X7SRtBFsXAxHa6MhobzKXOcn9jWtltsUaaE5mXPmRa1zEppnW+5y14/xF46Vozs/OqX5mr+Kx50Fy7cfL3Oy5o1izzVV9thgmenur/jrkQwq3Lz+/iCt42+Apvl0CIgOZ34b7vZCXpCRFtj5Q58BG8+7C5vuXgK2D0PNfdBpGJzktytQpYpWD4bHMJrk//ZMZ171trWvSOBjPj8lwWxQqJgKEzzFdTF1BjvbmGoxBCdJLfhXteoY2hyS66ZMq+MScpv34beg2E+a+b61t2tj/YGm5a9GPmmyvE73rTXNqt+fUyz5VSZy5yLIQQ58GtcFdKzQCeAoKAv2mtf9Pq8f7AS0Ccc5uHtdadmLFzfmobGvmmpIprRjuH3WltrnC+8x+w8a+mu+PyX5iZj57q8ojrZxbemvmE9C8LIXqMDsNdKRUELAGuBPKBTUqp1Vrr3c02+ymwXGv9F6XUcGANkN4F5W3XoeIqwnUNFzduhNV/Ntc9rDxhHhx1oxkvHtNFQyQl2IUQPYg7LfeJQJ7W+iCAUmoZMAdoHu4aiHH+HAsc92Qh3VJdSvzKO/gy9DNCP7dDaIw5GZo53fSPyxKxQogAorTW7W+g1LeAGVrr7zhv3w5cqLVe2GybFOB9oBcQCVyhtd7iYl/3AvcCJCUljV+2bFmnCm2z2YiKajmOPb5kM6N3PMryxhwyLricqrjhaIt/nVJwVe9AEah1l3oHFnfqPW3atC1a6+yO9uVO+rk649j6iDAfeFFr/YRS6iJgqVJqpNba0eJJWj8DPAOQnZ2tc3Jy3Hj5s+Xm5nLWc7cegx2wKmY+r1w/r1P77elc1jtABGrdpd6BxZP1dmdlrXyg+WLkaZzd7XI3sBxAa/0ZEAYkeqKAbqsuBiCxT1q3vqwQQvRE7oT7JiBTKTVQKRUCzANWt9rmCHA5gFJqGCbcizxZ0I7YKwup0SH0T+7eY4oQQvREHYa71toOLATWAnswo2J2KaUWK6VmOzf7EXCPUmob8Brwbd1RZ76HVZUWUEIMmckxHW8shBB+zq0zjs4x62ta3ffzZj/vBrx6iaDGykKKdQypcWHeLIYQQvQIfnOxDktNMaU6hthwq7eLIoQQXuc34W6tLaFExxAdJuEuhBD+Ee5aE1pXRgmxxEi4CyGEn4R7XSXBup4yFUOY1T+qJIQQ58M/krDKjLqsDo5HyVV/hBDCX8LdTGCqDYn3ckGEEKJn8JNwNy33hnAJdyGEAD8Ld0eYzE4VQgjwl3B3risja6oLIYThH+FeVUwlEYSHt3NRbCGECCB+Eu5FlOhYYsL9a/12IYToLL8Id4etiCIdLROYhBDCyW/CvVTHECPrygghBOAn4a6qiijRMdItI4QQTr4f7g4HltpSiokhOlRa7kIIAf4Q7jVlKO1wnlCVcBdCCPCHcHdOYJJuGSGEOMN/wp0YGS0jhBBO/hPuOoboMGm5CyEE+EW4m6UHyoghMkTCXQghwC/CvQiNwh7aC4tF1nIXQggA32/qVhdjC4olKjTU2yURQogewy9a7hWWOBnjLoQQzfhBuBdTpmTRMCGEaM4Pwr2IElk0TAghWvCLcC9yyKJhQgjRnG/3ZdjrobacAh0lY9yFEKIZ3265V5cAcNwu3TJCCNGcb4d7i3VlJNyFEKKJX4R7sY4hRrplhBDiNB8Pd7P0QCkxREu3jBBCnObj4S7L/QohhCs+H+4OZaWCCDmhKoQQzfh4uBdTGxoPKGLlhKoQQpzm2+FeXUy1tReAjHMXQohmfDvcq4qwBcUBEBUq4S6EEE18PtwrLHFEhQYTHOTbVRFCCE9yKxGVUjOUUnuVUnlKqYddPP57pdRXzq99SqlTni+qC1XFlCkZ4y6EEK11mIpKqSBgCXAlkA9sUkqt1lrvbtpGa/3DZtt/H7igC8raUn0VNFQ7r50qJ1OFEKI5d1ruE4E8rfVBrXU9sAyY087284HXPFG4djnHuBc6ZIy7EEK05k64pwJHm93Od953FqXUAGAg8NH5F60DVWbRsIJGWTRMCCFac6fJ6+qq07qNbecBK7TWjS53pNS9wL0ASUlJ5ObmulPGs9hsNnZs3MQo4FCllXprSaf35UtsNltA1NOVQK271DuweLLe7oR7PtCv2e004Hgb284D7mtrR1rrZ4BnALKzs3VOTo57pWwlNzeXUX2TYScUqV5kD0gjJ2dkp/blS3Jzc+nse+brArXuUu/A4sl6u9MtswnIVEoNVEqFYAJ8deuNlFJDgV7AZx4pWUecfe5H6mTpASGEaK3DcNda24GFwFpgD7Bca71LKbVYKTW72abzgWVa67a6bDyrqhhtjcTmCJUTqkII0Ypbqai1XgOsaXXfz1vd/h/PFcsNVUU0hidAJdJyF0KIVnx3Wmd1MQ1hCQAyzl0IIVrx3XCvKnKuCIl0ywghRCs+HO7F1ASbFSGlW0YIIVryzXDX2qwIGWxWhJSLYwshREs+Ge7B9ipw2Cm3mHCXtdyFEKIlnwx3a4NZdLKMWEDCXQghWvPJcA+pLweghBjCrBZCg4O8XCIhhOhZfDLcrQ0VgHNFSDmZKoQQZ/HJcA+pN90yJ+1R0iUjhBAu+GQyWhtMt8zxhghiwl0tWimEEIHNR1vu5RAWR1mdkm4ZIYRwwSfD3dpQDpG9qaxpkDHuQgjhgk+Ge0i9CfeK2gbpcxdCCBd8MtxNyz2Rihq7dMsIIYQLPhvu9vAE6hsdsmiYEEK44Hvh3mjH2lBJXYhzRUhpuQshxFl8L9xrSlFoqq0m3KXPXQghzuZ74e68dmqlrAgphBBt8tlwr3CuCCndMkIIcTYfDPdiAMqIASBWTqgKIcRZfDbcS5zhLtdPFUKIs/leuMcP5GSfiymyRwLSLSOEEK74Xp/GkOnsOR5KRW0j1iBFmNX3jk9CCNHVfDYZK2sbiAmzopSsCimEEK35bLhX1NhljLsQQrTBd8O9VlaEFEKItvhuuNc0yMlUIYRog8+Ge2WtXRYNE0KINvhsuFfUNhAdKi13IYRwxXfDvUZa7kII0RafDHe7Q1PT0Ch97kII0QafDPcau/kuo2WEEMI1nwz36gYNyFruQgjRFt8Md7sJd+mWEUII13wz3BvMd+mWEUII13wz3Jta7jJaRgghXPLpcJe13IUQwjXfDPembhk5oSqEEC65Fe5KqRlKqb1KqTyl1MNtbHOTUmq3UmqXUupVzxazpWq7xqIgMkTCXQghXOkwHZVSQcAS4EogH9iklFqttd7dbJtM4CfAFK11mVKqT1cVGKCmQRMdZsVikbXchRDCFXda7hOBPK31Qa11PbAMmNNqm3uAJVrrMgCtdaFni9lStV1OpgohRHvcCfdU4Giz2/nO+5obAgxRSn2ilNqolJrhqQK6Ut2gZdEwIYRohzvNX1d9H9rFfjKBHCANWK+UGqm1PtViR0rdC9wLkJSURG5u7rmWF4DKOjvWRlunn++rbLbAq3OTQK271DuweLLe7oR7PtCv2e004LiLbTZqrRuAQ0qpvZiw39R8I631M8AzANnZ2TonJ6dThf7ZJ+8yKLk3OTnZnXq+r8rNzaWz75mvC9S6S70Diyfr7U63zCYgUyk1UCkVAswDVrfaZiUwDUAplYjppjnokRK6UN2gZXaqEEK0o8Nw11rbgYXAWmAPsFxrvUsptVgpNdu52VqgRCm1G1gHPKi1LumqQlfbtSwaJoQQ7XArIbXWa4A1re77ebOfNfCA86tLNTo0NXZZNEwIIdrjczNUbXVmMXfplhFCiLb5XLhX1Ji1B2TpASGEaJvvhXutCXdZNEwIIdrme+Fe09QtIy13IYRoi++Fe21Tt4y03IUQoi0+F+6VtablHisnVIUQok0+F+5NJ1RlnLsQQrTN58I9rVc445OCiAqVcBdCiLb4XEJeNSKZkKIwgoN87rgkhBDdRhJSCCH8kIS7EEL4IQl3IYTwQxLuQgjhhyTchRDCD0m4CyGEH5JwF0IIPyThLoQQfkiZiyh54YWVKgIOd/LpiUCxB4vjKwK13hC4dZd6BxZ36j1Aa927ox15LdzPh1Jqs9Y629vl6G6BWm8I3LpLvQOLJ+st3TJCCOGHJNyFEMIP+Wq4P+PtAnhJoNYbArfuUu/A4rF6+2SfuxBCiPb5astdCCFEO3wu3JVSM5RSe5VSeUqph71dnq6ilHpeKVWolNrZ7L54pdQHSqn9zu+9vFnGrqCU6qeUWqeU2qOU2qWUWuS836/rrpQKU0p9oZTa5qz3L533D1RKfe6s9+tKqRBvl7UrKKWClFJblVJvO2/7fb2VUt8opXYopb5SSm123uexv3OfCnelVBCwBLgaGA7MV0oN926pusyLwIxW9z0MfKi1zgQ+dN72N3bgR1rrYcAk4D7n79jf614HXKa1HgOMBWYopSYBvwV+76x3GXC3F8vYlRYBe5rdDpR6T9Naj202/NFjf+c+Fe7ARCBPa31Qa10PLAPmeLlMXUJr/TFQ2uruOcBLzp9fAuZ2a6G6gdb6hNb6S+fPlZh/+FT8vO7asDlvWp1fGrgMWOG83+/qDaCUSgNmAX9z3lYEQL3b4LG/c18L91TgaLPb+c77AkWS1voEmBAE+ni5PF1KKZUOXAB8TgDU3dk18RVQCHwAHABOaa3tzk389e/9D8BDgMN5O4HAqLcG3ldKbVFK3eu8z2N/5752DVXl4j4Z7uOHlFJRwD+AH2itK0xjzr9prRuBsUqpOOBNYJirzbq3VF1LKXUNUKi13qKUymm628WmflVvpyla6+NKqT7AB0qprz25c19ruecD/ZrdTgOOe6ks3nBSKZUC4Pxe6OXydAmllBUT7K9orf/pvDsg6g6gtT4F5GLOOcQppZoaYf749z4FmK2U+gbTzXoZpiXv7/VGa33c+b0QczCfiAf/zn0t3DcBmc4z6SHAPGC1l8vUnVYDC5w/LwBWebEsXcLZ3/ocsEdr/WSzh/y67kqp3s4WO0qpcOAKzPmGdcC3nJv5Xb211j/RWqdprdMx/88faa1vxc/rrZSKVEpFN/0MXAXsxIN/5z43iUkpNRNzZA8Cntda/9rLReoSSqnXgBzMKnEngV8AK4HlQH/gCHCj1rr1SVefppSaCqwHdnCmD/a/Mf3uflt3pdRozAm0IEyja7nWerFSKgPToo0HtgK3aa3rvFfSruPslvmx1voaf6+3s35vOm8GA69qrX+tlErAQ3/nPhfuQgghOuZr3TJCCCHcIOEuhBB+SMJdCCH8kIS7EEL4IQl3IYTwQxLuQgjhhyTchRDCD0m4CyGEH/r/1ZrxmTSjawsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_1 (Flatten)          (None, 8192)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 256)               2097408   \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 257       \n",
      "=================================================================\n",
      "Total params: 2,097,665\n",
      "Trainable params: 2,097,665\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train loss    : 0.023065590844323327\n",
      "Train accuracy: 0.9944196428571429\n",
      "Validation loss    : 0.8642008661990985\n",
      "Validation accuracy: 0.84375\n"
     ]
    }
   ],
   "source": [
    "epochs     = 50\n",
    "batch_size = 64 # this value should be the same with the one used for save_bottlebeck_features()\n",
    "optimizer  = 'adam'\n",
    "drop_rate  = 0.5\n",
    "callbacks  = [plot_losses]\n",
    "summary    = True # for the first time show the summary of the trained top model\n",
    "\n",
    "train_top_model(epochs, batch_size, optimizer, drop_rate, callbacks, summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The overfitting is huge. So, I tried to apply more regularization by increasing the drop rate to `0.9`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEQCAYAAABcE6TVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XlclVX+wPHPuewg4IIsAu4LoriBlppFZYaWaaZmWdreVE5N21S/manGampmqqmZbLGyxUqzLLW0zI0yzdwVxQ03RERRlEVZ7nJ+fzwXveAFLpvC5ft+ve5L7vOc53nOucj3Ofec85yjtNYIIYRoGkwXOwNCCCEuHAn6QgjRhEjQF0KIJkSCvhBCNCES9IUQogmRoC+EEE2I58XOQHkbNmwI9fT0/ADoidyUhBCiKjZgm8ViuSc+Pv5YVYkbXND39PT8IDw8vHvr1q1PmkwmeYhACCEqYbPZVHZ2dmxWVtYHwA1VpW+INemerVu3zpOAL4QQVTOZTLp169a5GK0jVaev5/zUhEkCvhBCuM4eM12K5w0x6AshhKgnEvQr8emnnzZXSsVv2rTJt3Tb1q1bfa644orObdu27dmxY8ceI0aM6Hjo0CFPgBUrVvgnJCR0a9++fc8OHTr0uPnmm9vl5+ef9xm/+eabrWJiYmJjYmJivby8+nXt2jU2JiYm9sEHH4x0NW9paWle1113Xce6KakQwpn6igG7du3yDgsL62W1Wstsj4mJiV2xYoV/Rfn573//22rSpElta1OmBteR25DMnj27Zb9+/QpmzpzZsm/fvplnzpxRI0eO7PLyyy8fuvXWW3MBvvvuu8CsrCxPgIkTJ3b69NNP9w0dOvS0zWbjk08+aXHq1ClTYGCgzfG8jzzyyIlHHnnkBEBkZGTczz//vDsiIsJS/vpmsxkvLy+neevcubN54cKF++q80EKIs+orBnTr1q0kIiKi5Mcff2x23XXXFQBs2rTJ9/Tp06Yrr7zyTH2WSWr6FcjNzTWtX7++2UcffXTg22+/bQEwffr0lv369Sso/WUDjBw5Mr9///5Fr732Wuj48eNPDB069DSAyWTizjvvPBkdHX1eMK/Mww8/3ObWW29tN2jQoC7jxo3rsH37dp/4+Phu3bt3j+3Ro0f35cuXBwBs27bNJyYmJhbg9ddfD0lKSup42WWXdWnXrl3Phx56yOVvDEII5+o7BowdOzbniy++aFn6fubMmS1vvPHGHIAvvvgiuFevXjHdu3ePHTRoUNfSbxJ1oUHX9J/8ekv07qz8Cr/q1ETX8MAz/x7b+1BV6T7//PPmiYmJub169Spu3ry59ddff/Xftm2bX79+/ZzehVNTU/0mTZp0oi7ymJKS4v/777/v9Pf31/n5+aaVK1fu9vf315s2bfKdPHly+61bt+4sf8yOHTv8N2/enOrl5aU7d+4c9+STTx5r3769uS7yI8RFNe+haI6l1mkcIDT2DKOnVRoH6jsGTJo0KSc+Pj7WbDane3l5MW/evBZffvnlXoBrrrmmYMKECTtNJhOvv/56yNSpU8Pff//9jOoV0rkGHfQvpjlz5rR85JFHjgHcdNNNOTNnzmxZ1TF1ZcSIESf9/f01QFFRkbr77rvb7dixw9/Dw0MfOnTIx9kxl112WV6LFi1sAB07dizcu3evtwR9IWquvmNA27ZtLV26dClasGBBUEREhNnT01P379+/CGD//v3eo0ePjsrOzvYqKSkxRUdHF9fVdRt00HelRl4fsrKyPNasWRO0e/duvylTpmC1WpVSSv/5z38+8ssvvzRzdkz37t0L169f73/bbbedqu31AwICzrb/vfDCC2FRUVEl8+bN219SUqICAwP7OjvGx8fn7DBXDw8PLBaLqm0+hGgQqqiR14cLFQPGjRuXM2vWrJahoaHmm266Kad0+5QpU9o+8sgjWRMnTsz9/vvvA6dOndqmLsoF0qbv1MyZM1uMGTPmRGZmZsrhw4dTsrKytkZFRZV07969aMOGDc1mz54dXJr266+/Dlq7dq3fE088cWzOnDmtStvcAd5+++2W6enptbqx5ubmekRERJhNJhPTpk1rJSudCVH/LlQMuP3220+uWLEieP78+S0nT558Nujn5+d7tG3b1gzw8ccft6rLsknQd+Krr75qNWbMmJOO20aNGnVy5syZLefPn582bdq00Hbt2vXs1KlTj48//jgkIiLCHB0dbfn000/3Pfnkk1Ht27fv2bFjxx6//vprYGmTS0099thjx2bOnBnSu3fvmIMHD3p7e3tL1Beinl2oGBASEmLt06dPQUhIiDkmJqakdPtf/vKXzFtuuaVTfHx8t1atWlVrMEhVVEOrOW7ZsuVA7969j1/sfAghRGOyZcuWkN69e7evKp3U9IUQoglp0B257uDNN99s9c4774Q5buvfv3/BzJkz0y9WnoQQF05DiwHSvCOEEG6gMTfv2Gw2mww3FEIIF9ljpkuDRhpi0N+WnZ0dLIFfCCGqZl9EJRjY5kr6Btemb7FY7snKyvogKytLlksUQoiqnV0u0ZXEDa5NXwghRP2RmrQQQjQhEvSFEKIJaXBt+iEhIbp9+/Y1Pv706dMEBARUndDNSLmbFil30+JKuTds2HBca926qnM1uKDfvn171q9fX+Pjk5OTSUxMrLsMNRJS7qZFyt20uFJupdRBV84lzTtCCNGESNAXQogmRIK+EEI0IRL0hRCiCXEp6CulkpRSu5RSaUqpp53sb6eUWqaU2qqUSlZKRTnsm6yU2mN/Ta7LzAshhKieKoO+UsoDmAYMB2KBW5RSseWSvQp8qrXuBUwFXrYf2xJ4DrgEGAA8p5RqUXfZF0IIUR2u1PQHAGla631a6xJgNjCqXJpYYJn95xUO+68Flmitc7TWJ4ElQFLtsy2EEKImXAn6kYDjavQZ9m2OtgA32X++EQhUSrVy8VghhGgQ1uw7wboDOVUnbMRceTjL2RTH5WdpewJ4Syl1B/ALcBiwuHgsSqn7gPsAwsLCSE5OdiFbzhUUFNTq+MZKyt20SLnrntWmeTT5DGYbvHSZHy19G844l7ostytBPwOIdngfBWQ6JtBaZwJjAJRSzYCbtNa5SqkMILHcscnlL6C1ng5MB0hISNC1eeJOnthrWqTcTUt9lnvFzmPklawD4LsjzZhxR3+UahjLetRluV25la0DuiilOiilvIEJwALHBEqpEKVU6bmeAWbYf14MDFNKtbB34A6zbxNCiAZl7sYMWvh78czwGFbsyuabjYcvdpbqRZVBX2ttAaZgBOsdwByt9Xal1FSl1A32ZInALqXUbiAMeMl+bA7wAsaNYx0w1b5NCCEajNxCMz+lHuWG3m24d0hHEtq14O/fbedoXtHFzlqdc6nRSmu9SGvdVWvdSWtdGtCf1VovsP/8tda6iz3NPVrrYodjZ2itO9tfH9VPMYQQouYWpRyhxGJjTL8oTCbFv8b2othi4y/fpuBuC001nJ4KIYS4SL7ZmEGn1gH0igoGoGPrZjwxrBtLdxxj/ubMKo5uXCToCyGatIMnTrPuwEnG9Isq03F712Ud6Nu2Oc9/t51j+e7TzCNBXwhRa5+sPsBVryU3yjbwbzcdRim4sW/ZR4g8TIp/j+3NmRIrf5u3zW2aeSToCyFqZdvhXF5cmMq+7NM8PmcLNlvNg6PNpjleUEyJxVaHOayY1ppvNh5mUKdWtGnud97+zqHNeHRoVxZvP8r3W49ckDzVtwa3cpYQovEoLLHy8OxNtArwYfKg9vzzx53MWLWfe4Z0rPQ4m00za106O4/kczSviKP5xRzLKyI7vxiLTdPC34ux8VHcMqAtHVs3q/A8xRYrv+09wZHcIm5OiMZkqt64+g0HT5Kec4ZHru5SYZp7h3Tgx21HeGzOZhZvz2LiJe24tGPLBjOGv7ok6AshauzFhansP36az+++hIGdWrEx/ST/+nEXgzqFENsmyOkxNpvmL/NSmLX2EMF+XoQH+RIa5EOX0BDCgnwIaebDugM5fLTqAO+v3M/Ajq2YeGlbhsWGA3CioJjlO4+xbMcxVu7J5nSJFTBuQHdd1qFa+Z+7MQM/Lw+SeoZXmMbTw8T7kxJ45+e9zN2Qwfdbj9CxdQC3DmjLTf2iaBHgXa1rXmwS9IUQNbI09Sif/57O/Zd3ZFDnEABeGRNH0psr+dOXm1gw5TJ8vTzKHGOzaf46fxuz1h7iwcROPHltN6c15jsHd+BYfhFfrc9g1tp0pnyxiZBm3gR7WNi3eClaQ3iQL6P7RjK0exifrTnIKz/uZFDnVsSEO7/ZlFdktvL91iMM7xlOgE/loTA0yJfnRvbgqaQYvt96hC9+P8iLC3fwr8W7GNo9lI4hzQgL8qF1oC9hQT6EBfnSOtAHL4+G14IuQV8IUW3H8ov489ytxEYE8diwrme3t2rmw6vjejN5xlpe+WEnz9/Q4+w+rTXPLtjGF7+n80AlAb9UaKAvD13ZmQeu6MQve7KZtTadPRnZPHJ1F4Z2D6NHm6Czx8dFBZP0xkoembWZ+VMGn3ezcWbpjqPkF1m4KT6qyrSlfL08GBsfxdj4KHZm5THr93R+Sj3K4u1HsZbry/A0KSYMiObJYTEE+3u5fI36JkFfCFEtWmue/Gorp4st/PeWPvh4lg2wV3RtzZ2D2/PRqgNc0a01V3YLRWvNcwu289madO6/oiN/riLgOzKZFIndQknsFmqfg6breWlCmvnw6rhe3PHROv75406eG9nDyZnK+mbjYSKCfbm0YyvXCl5OTHgQfx/Vk7+P6onVpjlxuphjecUcyy/iaF4xWzNO8cXv6fyQksXTw2O4yf7gV0WKzFaO5hXRrlVAjfLjqob33UMI0aB9svoAP+/O5q/XdadzaKDTNE8lxdAtLJAnv9rK8YJi/v5dKp/+dpD7Lu/I00kx9dIJmtgtlDsGGTebn3dnV5o2O7+Yn3dnM7pvJB7V7Px1xsOkCA30pWdkMFfFhHHLgLa8PKYX3/9xCO1DAnjy662Mf+83UjPzyhx3LL+I2WvTueeT9fSZ+hOPfrm51nmpitT0hRAu2300n3/8sJOrYkK57dJ2Fabz9fLgzVv6cMNbqxjx5kqO5Rdzz2UdeGZ4/QT8Uk8Pj2H13uM88dUWfnxkCK2a+ThNt2BLJlabZkzf+l3eI7ZNEF/dP5C5GzN45YedjHzrV26/tB2tArxZuvMYWw6dAiCyuR83J0RzTWzFHcp1RYK+ENVUYrExe106SinGxUe51H7sDrTW/OXbFIJ8PfnX2F5VBu+Y8CCeToph6vep3DW4A3+5rnu9D3P09fLgzQl9GfXWKp6am8L7k+LLXLPYYuX3fTl8vuYgvaKC6RLm/JtKXTKZFOMSohkWG86/f9rJJ78dQGvoHd2cJ4Z15eruYcSEB16wIaAS9IWohl/3HOfZBdvYl30agP8t28ODiZ2YMKDtRQ/+H6zcx/+Wp9EtPJBekcHERQUTFxlM+1YB1R6/7szqvSdYd+AkL4zqQUgFNejy7hzcnitjQmnfyv+CBbXuEUH8OakbLy7cway1h0jqGc6KncdYtvMov+w+TkGxBV8vE28O73tB8lMq2N+LF0fHcf/lnfDxMhEa6HtBr19Kgr4QLsjKLeKFhaks3HqEdq38+ejO/vh6evDG0t08/10q7/68jwev7MTN/aPPdmxarDbSsgvYmpFLSkYuZ0qs/N+ImAqbHGrjWH4Rry/ZTduW/pitNmauOUix/anWQB9Pekc3587B7bkqJrRGwVdrzZtL9xAe5Mv4/tFVH2CnlKJDSP12TDpz1+AO/Lw7m+cWbOOv81KwaQgN9GFk7wiGdg9jUKcQ/Lwvzk06uqX/RbluKQn6QlTCbLXx8aoDvLF0Nxab5tGhXbn/io5na/UDOw1k9d7jvLFkD8/O3847yXtJ7BbKrqw8Uo/kUWQ2Am8zH09KLMZNYNa9l+DvXbd/ev9ZsocSi413bounQ0gAZquNPUcL2HY4l62HT5G8K5u7P1lP76hg/jS0K4ndWlcr+P+29wRrD+Tw9xt6nDdapyEymRSvjuvNs/O30S08iKHdQ+nZJrhOvvE0dhL0hahAscXK2Hd+I+VwLlfFhPL8yB60bXV+LW1QpxAGdmzF6r0neGPpbr7fkkn3iCAmXtKOXlHB9IwMpkOrAJbtPMb9M9fzwGcb+WByQp09uLPnaD5frktn0sD2Z2vVXh4mYtsEEdsmiPH9ozFbbXyzMYP/LU/jzo/X0Se6OX8a2oUruroW/N9YtoewIB9urkYt/2ILC/LlvdsTLnY2GhwJ+kJUYNbv6aQczuXVcb25qV9kpcFRKcXgziEMtj+Z6sw1sWH848Y4nv4mhae+3sqr43rXSc3zlR92EuDtycOVzB/j5WHi5v5tubFvFHM3ZvDW8jTu+Ggdfds259VxvelUyfw2v+09wdr9OTw3Mvai91uI2pNx+kI4cabEwlsr0ri0Y8sqA351TBjQlsev6co3mw7zzx931vp8q/ceZ9nOYzxwZSdaujAHjLeniVsGtGXFE4m8dGNPDp44w6QP15KVW/GUyG8u203rQB9uGdC21vkVF58EfSGc+GjVAY4XlFQ5VUBNTLmqM5MGtuO9X/bxwcp9NT6Pzab5x6IdtAn25a7B1ZtozNvTxMRL2vHpXQM4daaEyTPWknvGfF663/edYM2+HP5wRSep5bsJl4K+UipJKbVLKZWmlHrayf62SqkVSqlNSqmtSqkR9u3tlVKFSqnN9te7dV0AIepabqGZ937ey1UxocS3a1nn51dK8dzIHoyIC+fFhTuYv/lwjc6zYEsm2w7n8cS13WockHtGBjN9UgL7jhdw76frKTJby+x/c9keWgf6MPESqeW7iyrb9JVSHsA04BogA1inlFqgtU51SPZXYI7W+h2lVCywCGhv37dXa92nbrMtmrrMU4U8O387Ww+ewW/dCpePCwvy5b8T+hIeXPEY6fd/2UdekYXHh50/x0td8TApXh/fhxMFa3l8zhZW7jnO0O5hDOkSUuWMjwAlVs2/F++iR5sgRvep3VOlgzuH8Pr4Pjw8exOPzN7E2xPj8TAp1h3IYfXeE/z1uu5Sy3cjrnTkDgDStNb7AJRSs4FRgGPQ10DpfKbBgHutJCwalAVbMvnrtylYbJperUyEhzV36TiNMR3w5BlrmfOHgQT7nT/zYXZ+MTNW7ef6XhH0aBNcxzkvy9fLg/cnJ/D3Ban8tD2Lrzdk4O1hYmCnVgyNDePqmFCnqzkBLE03c/iUmX+N7VUnncEje7c5O0fO3+Zv46XRPXlz6R5Cmvkw8ZKKp1sQjY8rQT8SOOTwPgO4pFya54GflFJ/BAKAoQ77OiilNgF5wF+11itrnl3RlOUVmXlu/na+3XSYvm2b88bNfdifso7ERNefrPx1z3Hu/Hgt9366nk/vGnBeDfbt5DSKzFYevab+avmOgny9eG18b8zWONYfOMmyHUdZuuMof5u3jb8B7Vr5ExdpPFkbZx/+abVqvttrJrFb60pHC1WXMYd9Me8k7yW30Myvacf5y4juF+0hJlE/VFWL/SqlxgHXaq3vsb+/HRigtf6jQ5rH7Od6TSk1EPgQ6Al4Ac201ieUUvHAPKCH1jqv3DXuA+4DCAsLi589e3aNC1RQUECzZhUPP3NX7l7uXTlWpm8t5mSx5oZOXozs6IWHSdWo3GuOWHh3SzHxYR481McHk72j9kShjad+KWRgG0/ujqv7p2ZdpbXmyGnNlmwre09Z2Z9r40TRub/TZl5w2qx5YbA/UYF1OxZDa82H20r49bCFQG949XJ/fDwbzgNN7v7/vCKulPvKK6/coLWu8sEEV2r6GYDjExlRnN98czeQBKC1/k0p5QuEaK2PAcX27RuUUnuBrsB6x4O11tOB6QAJCQk6MTHRhWw5Z8y3XfPjG6vGXO6jeUWkZOSy51gBVtv5C2IfyS1i1rp0olv68/5dfejXtsXZfTUpdyIQGr2fqd+nsuxUCC+O7olSiqfnbsVkOszLtw0hqsXFfVQe4FaHn3NOl5ByOJeUjFOkHM4lyJzDbSOvqpfrXna5jZcX7WRAhxZc2zOiXq5RU435/3lt1GW5XQn664AuSqkOwGFgAmX/PwKkA1cDHyulugO+QLZSqjWQo7W2KqU6Al2Amo9RE42e2Wpj5Z7ss/PRpBzO5Vh+caXHKAXj46P528hYmrnQyemKuy4zmjLe/XkvoYG+jOwdwVcbMrj90nYNIuCX1zLAmyu6tuaKrq0BIwjUFy8PE8+OjK2384uLq8q/IK21RSk1BVgMeAAztNbblVJTgfVa6wXA48D7SqlHMfrL7tBaa6XU5cBUpZQFsAJ/0Frn1FtpRIOmteZPszezMOUISkHn1s24rHPI2dkgYyKC8PE8v7lCYSxOXdeeSupGdn4x/1m6m/mbD+PtYeKhKzvX+XWEaEhcqjZprRdhDMN03Pasw8+pwGAnx80F5tYyj8JNfLz6AAtTjvCnoV24d0hHl4Ym1ielFK/cFEfO6WJW7MrmwcROtA68eG35QlwIMveOuCA2pp/kpYU7GNo9jEeu7nLB5lavipeHiWkT+zF/cyaj+rS52NkRot5J0Bf1Lud0CVM+30hEc19eG9e7wQT8Uv7enjKvjGgyJOiLemWzaf705WaOF5Qw94FBBPuf/0CUEOLCkQnXRL2atiKNX3Zn89wNscRF1e8TrkKIqklN3w3sysonr6Tyh+wuuG1z2Z15ktdXRDC6TxtuleaT6rFZ4ZdXocPl0G7gxc6N+9MaslJg+zdg8oSeN0Fo94udq3ohQb+RKzJbGT1tFZ5Y8Y3MYliP8IudJdj0Gcx/iLZ40yfkY166Ma7BteM3aDYbLPgjbP4c1r4HD6yGwAbwe3VHOfsgZS6kfAXHdxkBX2v45d8Q1hPixho3gObuU2mRoN/IbTucS6HZSpA33DdzA7cMiOZv18fW+RqsLtv+LSz4Iwe8u9C+ZA/Te+8hwOe6qo8zF8HW2dB2ILTuVnX6olzY9QPN8s9gPGPrJrSGxc8YAb/fZNj6Jcx7ECZ+DaYqWmNPHjA+f33+U834t4I+E8HDxT6VtKUQ0Boiele7CA2GpRg2zTT+r5y3r8Qo42H75ADtBsOlD0DsKONb1vZvjRvB0ueNV9uB0PFK8HDyd+UbbHy2Xs4nx2toJOg3cpvSTwHw3EA/0lQb3v15L2v25fCfm/vQJ9q12ScrY7HamLc5kznrDvHnpG4ktK9kfvndP8Hce8kP6cvwQw+RHPo6YTu/gKsfMR6rrczv7xh/XADhvSBunFHDCnaYNthcBHsWG3+Mu38CazF9PPwhvi+0cX3StQZtxUvw+7tw6UNw7UsQ0QsWPg5rp8Olf6j4uJMHYEYS5B+pOE1uBlz116rzcPA3+HwcoCDxGbjsUefBriGzWuDru2Dn9xWnCYuDa6ba/59Fld13yX3GK2c/bLN/E0j+R8XnWvs+jHnf+H01cI3sNynK25h+kuiWfrTyM3FTYgxXdG3N43O2cNM7q/nT1V14ILFTjZ5mtVhtLNiSyf+Wp7H/+GlMCv7v2xQWPjzE+YLeB36FObejw2KZYn2GwEBFi8v/AN9PgYOrof15z+6dYy6E396GdpdB9+uNP7Alf4Mlz0L7yyDmeqO9dccCKM6DgFBIuBM6D8Xy9YN4zhwDd/4AoTGVF8pcCCfSIDzO9Q8iaxvkZznf16YPBNTdLJf8+obRrNBvkhHwlYKEu2HPEuOz6DAEwnqcf1xeJnxyA1iK4P6Vzr8pff8orHwNOl1deR9BUS58c5/RnBEZDytehLQlcON70LKS1blOn4AzJ6D1hZmdtFI2G8x/yAj4Sa9Awl1OEinwrHp5SVp2gMufMF6WCqYLObAS5j0E718FVz8LA6dU/K1Mazi6zfgWdZGa7CToN3Kb0k8xoENLwPgKe2nHVix6ZAjPzt/Ga0t2czDnDK+Oc/0rutWm+W5LJv9dtod9x0/TPSKI926PR2v4w2cb+GzNQe4svzRfxgb44mZo3o5Vl77Hz7P28cLonnj3GgRL/wLrZ1Qe9Dd/DqePwdgPjY7LSx+AE3sh5WtImQM/PgXegRB7g9HG2v7yszXPzX2mcun25+HTUXDXjxUHpiNbYO69Rrttn4kw/J/gE1hxnsyFRqBdO73iNIFt4K4foEX7itO4at2HsPQ5o9Z5/RvnvhkpBTe8Be8MNPJ/73LwclgA5vQJ+HS0EXAnL6i4pjn8n3BwlRHQH/jVaJJwZuETkHfY+CyjB0DXJOObxrtDYMS/oPct5/JWchp2/QBb58DeZWCzQPydxg3LO6D2n0lNaA2LnjCaCq/6q/F/qa54VvC0duehRr/Ldw8blZU9P8GN75b99pC9y6jMpHxlfCtDGTfxuPHQfST41f5buask6DdimacKycorIj46ECzn2i2D/bx4c0JfAnw8+XpDBs+NjCXQt+q23IyTZ5g8Yy17s08TEx7Iu7fFMyw2DJNJobVmSJcQ/rNkNzf0bkOrZvY/gKPb4bMx4N8K223f8vKn+4hu6cfNCdHgaTKCxPoZcPq481qx1QKr/guRCdB+yLntrTpB4lNwxZ+NzragNk7bTIv8ImDSPPhoOHx6A9y12EhbymaF1f+F5S8Z10+4GzZ8ZATAG6dD2/JLQ1D2BnHpg9BjzPlpCnOMAPrpKLjzRwiqxWyUW740AmvX4UaN2lRu/vpmrWH0O/D5WFj2d0h62V74XPjsRjh1EG6ba9TMK+ITCGM+gBnXGoH9pvfPT7P1K+Mmm/h/RsAH6DUe2l4K39wP8x6A3T8agSp1HuxcCOYzEBRpfE42K6x526j5jnkfIvvV7POwlEDhSae7TNZKJufT2rhxrv8QBj8CQ56o2fVrIqAV3PyZMYjhh6fgnUFw7T+Mm3HKV8Y3VWWCDlfAkMch97DxWS+YAgsfgy7DjCbNrtfWe9+ABP1GbFP6KUzYmPD7aM54BEOf2dD83CzYY/pG8sXv6SzfeYxRLiyp98nqA6TnnOHtif1I6hFeZkUmpRTPXh9L0psrefWn3bw8Js74I581wfhPOmk+P6Sb2J6Zx+vje+NdOnFa/J1GG/Xmz40/xPK2f2MEraSXnbf7K2XcACqweFK2AAAgAElEQVQT2h1u+8Zo4vh0lNHUExACpw7Bt3+Ag79C9xtg5Jvg39L44/r2PvgoCS5/0nh5eJ1/g7j9W+hUyfTFt31j3GhmjoY7Fhl/+K46k2MEzpSvjRtQh8th3McVd7R2uQYG3GcE1c5DMVnN8Pl4OJoKt8wymsGqEt0frnjKaJvuMgx6jTu371S6EXyiLzGCkqPmbeGO72HVm0afQ+p88GsBvW42Psu2A881Z3RLMj7zD6851x9Q/iZWmbRlRtNMBX0Tg03ekHO9cd3OQ8s20ax8zchjwl0w9O9V9yPVNaWg3+3Gt9pv7jPKAUaFJumf0ONGCAw7lz7xacjcaPwf2DbXaI4K7QEPrq7fbFa1iMqFlpCQoNevX191wgo0pfm2X/g+lfVrfma+p32tep9guP51owkE42nYQa8sp1eUsfh1Zaw2zaBXlhEX2ZwPJlecdup3qXy0ej/fTbmMnsWbjaA3dgaW7jdy7Ru/YFKKH/90OR6OS/h9NMJod/7jxrJtnTYbvDvYGG3ywG9Vj05xoszv++BqmDkGQjobAXLxX0FbYfi/oM+tZYNAUR788GfYMsv4oxz6PCS/cv4Noir7Vxo18NbdYPJ3FTebwLnmkJSvjJEjNguEdDOC7yUPgE8Vi4OYC2F6IhSe5KRnGC1yt8HYGUYwcZXVAh+PgGM74A+/Qot2xs3uY3u/yR9WVt52n70Lcg8ZTWwVtYkXnoTvHzNu6G0HGt9eWlSx5KK5yOjI//0d4zMZcK9RMy7n8KYlRJ5aa9SgfZtDj9HGDSBrm9EM2OtmGP1ujf4v1SmrxegLCeladaUFjN/BgZXGt7fYUeftdiWuKaXqbBEV0UBtSj/JDS0OQD5s7v0CfU58B3Pvht2LYcS/Mfk1J6lnOF+sTaeg2FLpXPS/7zvB0bxi/nZ95ZOOPTK0C/M3H+b5Bdv5KmIOyrsZdB3Ot5sOszf7NO/e1q9swAej5jX3btifXLbmvGcxHEu1N2nUwR9pu0Ew4TP4YoIxzj1qAIyZ7jyI+QYZ7a5dhhmdnJ9cD97NYNTb598gKtNhCIyfCbNvMWret39Ttj3baoa9K4xAv3MhmE+faw6JG2d0Krt6LS8/uOkDeP8qWliPwqhp1Qv4YPSFjJkO71xm1Mjv+B5+/Q+kr666sxaMm1tVQ2r9Whg3o65JRvv6O4OMspb/VlAqaxvMvQeyd8CA++Gav1fYxLHndCcih8w895lu/Qo2fGzsjLne+P1d7IAPxufcbbjr6U0e0DGxvnJThgT9RqrYYmXb4TxeDNsDQVGcatELbngQfn3dqLGm/wY3vsuIuFg+Xn2AZTuOVtrEM2/zYQK8Pbg6JqzCNGD0Fzx5bTee+2YDlhPz8OoxkmKTD28s3UNcZDDXOns4rPtIY5z4+hnngr7WsPJ1CG5rdF7Wlc5DYeIcOL7HaL+vaqhhzzFGk8b6D6HvbdCyY/Wv2XWYEYy/vgu+vA0mzIIjm42gtP3bc7XSuLH2NvJBNQ9M4XEwfibbt2ygR9/banaOFu3hulfhW3s7/ba5xu+g1801O58zSkHvm43+gOUvGs8bbPjIuOH1vMm4AYT1NJqrlv3duFFM/NpoxqqKh5fxmXcddu7b04m0xjm09CKQT6iRSs3Mo8RqpeOZrdAl0djo4Wl0fHa6Cr65Fz6+nv59b2doQDd+3BpaYdAvMltZt20HL0Wk4LdogTHMzTeowmuPS4hm7y+z8CoooCjmJr5ce4jDpwp5eUwFT956+hgjZn6bBnlHjE7Pg6sgYy2MeNX1B4Zc1emqytviywuONIba1UaPG6G4wOiY+3cnKCkATz+jttdrvDFU0pUhgq7olkT2Ed+q01Wm183GN8KtX0JQFFz3ev20gbdoZ3QaOzZtrXnb6Dvxb2XcELtdBzf8t2bDX70DzjZnCtdI0G+kNqafIlodw7co2/jKfNphZ1SCMV576XOoTZ/zgbWQjL0hlCyeiHefCRBmXwqvKBdSF1Cw5nOW6jV4HNVwFGjVGYY8VuG1PUyKB1ttJDs/iHf3hLEgJY1LOrRkSJdK/mjj7zD+0Dd9Blc8adTyA1obtWt30e92ow9h909Gu2zMiMqHhV5MShn9PyYPo0mlvocMlgbnuLHnOrHTlhmjVfrefuE7XZswCfrVUXLGCFr9JpUdK10T6WuM8dA1bNrYlH6SYQH7wIIR9HccK5vApxlc9xoMfZ60lXPI+PkT2qx5C3570xgh0KKd8UdnLcbm2YYPTWO56w9P4Ln4aaMmdukDFQ8dKzxFi4zl/NxyJB+uPgTAOxP7VT6/TqtOxmPsGz6Gzlcb47qvfrbRPLrusvg7jFdj4NfCaJa60PxbGv08Th+aEvWtAfR4NCI/Pg0/PGmMVa4pS4nx0M+MJKMN+LdpNTrNpvRTXOW/z2grbl3Jk6g+gXS46i6e9H2Wp9vNMZpTvAPgyFZIuIuC23/issLXyOz7KJ5hMcZwvdPZxs2tIjsWgLWEnkn32PsBQiufnqFUwl2QlwFfTQafIOh/T/ULLoSoFZeCvlIqSSm1SymVppR62sn+tkqpFUqpTUqprUqpEQ77nrEft0spdW1dZv6C2vEdbPzE+DkrpWbnyN4FH1xljCWOn2wMDVz8f+dGHwA/787m6teSOV5Q8UMoR/OKOHyqkB6W7UZHWRWdgh4mRVKPcBbsNXOmz51wzxJ4bDsMf4VFOW0osWpG97W397e/DKL6G00xVovzE6Z8BS070qrrQJY8dgXTJrr4EE634dAszBgT3v+eyoc3CiHqRZVBXynlAUwDhgOxwC1Kqdhyyf4KzNFa9wUmAG/bj421v+8BJAFv28/XuOQdgQUPGzMOhnSFrK3VO15rY0Km9y43xqtPmGWMA7/pQ+h8DXz3J+MBDeB/y/awN/s0n/52sMLTbUo/SUvyaH7mgNG044IRcREUmW2s2JldZvu8zYdp38qf3qULnCgFlz1mBOZtTta0z8s0xqbHjQelaNPcD18vF3+lHl7Q/15jSoW6fDxeCOEyV2r6A4A0rfU+rXUJMBso//SABkqHewQDmfafRwGztdbFWuv9QJr9fPXj8Ebn08rWhs1mDGszFxpBuk2/6tX0848aMxYuesKoRT/wm9HBB8ZojptnGtO6fns/B1Z9zfqDJ2nm48lnaw5SWGJ1esqN6acY6LnbeONi0B/QoSUhzXxYlHLuSces3CJ+23eCUX0iy7bHd02C1t2N8du2cp/ntrmANobc1cSQx+DRFGgWWrPjhRC14krQjwQOObzPsG9z9Dxwm1IqA1gE/LEax9aN43vgw2votfXvRs28rvz+LuxbAUn/gJAuxoRW+UegILvqYwHmTDKetBv+b2MccmC5cfBefsZj9OG9iFz6AFd57+DNCX3IOV3C3I0ZTk+5Kf0kwwL3g6evMdOjCzxMiqSeYSzfeezszeS7LZloDaP6lHsgy2Qyxjxn7zi//2LrHGMa45DOLl33PCYPowNRCHFRuDJ6x9mQjPJzN9wCfKy1fk0pNRCYqZTq6eKxKKXuA+4DCAsLIzk52YVslT+rJqLzvXRO+xDzf/uzq9tDHG9du2XmAgoOEL/hb+S0GsC2/A6QnEzzk1b6AFt++oyTLSufw91kLWLIobUcbDeWA4Vd4eefK0x7ps2f6JTxF972+BfbdzejQ3A7/vfTdtoU7sPkUAu32DSb08/wL/+tnAroxOZffwOgoKCgys+tjdVKodnKW9+soH+4J5+tLqRDkIn07etJL5dW2UIY4BtKyaLn2XTED5TC//QhBmRtJa3T3WTU5HdUD1wptzuScjctdVpurXWlL2AgsNjh/TPAM+XSbAeiHd7vA0LLpwUWAwMru158fLyujTULZ2r97uVaPxek9bwHtS7Kq9mJSs5o/dYlWv+7i9YF2ee2nz5hnHvlf6o+x/6VRtpdP1aZ9I0lu3XCUzN1yeu9tH45Wv+4Zotu99T3evG2I2XSbTl0Usc89bW2Pt9C66VTz25fsWJFldcwW6y639Sf9EOfb9B7jubpdk99rz9Yua/iA36fbuR/3y/G+2UvaP18c63zsqq81oXiSrndkZS7aXGl3MB6XUU811q71LyzDuiilOqglPLG6JhdUC5NOnA1gFKqO+ALZNvTTVBK+SilOgBdgLU1uz25ptA/Cu5eYgw93PS5MQ/4oXXVP9HS543mjdFvl31S0L+lMXWAK525GfbrRlY+B1KJxcZnvx+kR7eueN02B4ryuCbvGyKb+/HByv1l0m48eJK+pjRM2upye34pTw8T1/YMZ/nOY3y57hAmBSN7VzIlcN/bjAeofn3d6IxO+cqYGrZ8E5UQotGoMuhrrS3AFIxa+g6MUTrblVJTlVI32JM9DtyrlNoCzALusN98tgNzgFTgR+AhrbXz3sm65OltPPhz5yJj9roZ18KKlyseguhIa9jwidGWf8kDxlwu5YXHudaZm7HemMuliil3F6UcITu/2FicpHU3iB2Faf2H3H9pCGsP5LD50KmzaTcdOkWi3160Mp2b87warouL4EyJlRmrDjC4cwihgZU8ZOblBwMfgr3LYd0HxuIPvcZX+5pCiIbDpXH6WutFWuuuWutOWuuX7Nue1VovsP+cqrUerLXurbXuo7X+yeHYl+zHddNa/1A/xahAu0HGKkFxY+HnV4zgf2JvxenP5BgPDn33sLGgx9DnnaeL6GV0HJecdr4fjJtHxjpjzHsltNZ8tGo/HVsHMKSz/RvFkMegOI+b+YlAX0/eX7nvbPqN6ScZ4r0HFdaj0vlxKnJJh5a0DPDGatPc0LvyGTUBY9Iyn2BjYQhPX2MmQyFEo+X+T+T6BhtTyd70IZzYYzT3bJxpBGVHe1cYU8DuXGQE+0nzK55qITwO0MYCFhXJPQQFR6sM+psOnWJLRi53Dmp/btGSiN7Q6Wp81r3LpP6h/JByhEM5Z8jOL+ZITj6dS3YYMzXWgKeHiRFx4fh7e5DU04U1On2DYMA9xpwyXZNqdKMRQjQc7h/0S8WNNdaxjOxnzIQ453ajZm8ugsV/MVY/8gmEe5ZWvdpP6cLalbXrl7bnR1Xenv/RqgME+noypl9U2R1DHoPT2dwXuAaTUsxYtZ9N6SeJVQfxshZWvrh1FZ4e3p2FDw9xaQlFwJj7PbwXXHJ/ja8phGgYmtaEa8FRMGkB/PYWLJsKhwYaHbPHUo1pAa55Abz9XThPtDHnTaVBf73RHBLWs8IkWblF/JByhDsGtSeg/AIn7QZD1ACCN7zN6F4f8eW6QxRbbFzqscvYX81OXEfNfDwrXVDlPAEhxopKQohGr+nU9EuZTDD4Ybh3uTGd7OlsuPUrY0ZKVwI+GFMVVNWZm7HeeIipkrniP1tzEKvWTB7U3vk1hjwGuek8Gp7CmRIrs9amc6X/XmjRAQJdaJoRQohyml7QLxXRy2ju+dM2YwWeah/fG45udz4iyFIMR7ZU2rRTZLbyxdp0hnYPI7plBTebLtdCaCyR299lSKeWaK3pZdtRq1q+EKJpa7pBH4x2+5rOix8eB5YiY5m28rK2gbXYaSdubqGZ1WnHeXFhKjmnS7hzcPtK8lc6HcJOnu60n47qCAGWU7VqzxdCNG1Nq02/LoX3Mv7N2gqh5eazd3goa//x0yxJzWJrRi7bDudy4MSZs8muigllYMfKx/DTYwwsf5Eeez9k5tBxsBKp6QshakyCfk2FdAEPHyPol39gKWMdBLbhCC0ZPW0luYVmIpv7ERcZzLiEaOIig+kZGUzLABfWTPXwNPogFj5OZHEe+IcYyxkKIUQNSNCvKQ8vCO3uvDM3Yx06KoHH52zBbLXx06OX0zWsFmul9rkNkv8Jx3cbD0fJeqJCiBpq2m36tRXRy1h20PFBr4JsOHWQ380dWb33BM+NjK1dwAej32Hgg8bP7Wr2UJYQQoAE/doJ7wWFOcZqUqUOrwfgjZ3BJPUIZ3xCdN1cq/+9MOB+6Dm2bs4nhGiSpHmnNhw7c4ONtWHMB9ei8CDTrxvvjIkruyJVbfg0gxH/qptzCSGaLKnp10ZYLKDKtOunb/2ZVFtb/jH+Elq40lErhBAXkAT92vAJNKZOtk/HsGx7JmH52ykJ78dlXUKqOFgIIS48Cfq1Ze/Mzc4vZvrcRTRTRfQeeM3FzpUQQjglQb+2wuPg1EGem7OKLuadAHi1rf7iJkIIcSFI0K+t8N4AnEjbyO1Rx8CvpdHkI4QQDZAE/Vqy2adOHtwsky7mXcYka/LwlBCigZKgX0s/HNBk62DGhhzEdHxXlStlCSHExeRS0FdKJSmldiml0pRSTzvZ/x+l1Gb7a7dS6pTDPqvDvgV1mfn6suFgDvfPXE9ekbnSdBarjdeW7GK/ZycislYAusqVsoQQ4mKq8uEspZQHMA24BsgA1imlFmitzy4Qq7V+1CH9H4G+Dqco1Fr3qbss17/vthxh8fajmNRW3p7Yr8IHrL7ddJh92acJ6Z2A2rURUBAZf2EzK4QQ1eBKTX8AkKa13qe1LgFmA6MqSX8LMKsuMnexpB7Jw9vTxA/bspix6oDTNMUWK28s3UOvqGA6xNmnOm7dzViIXQghGihXgn4kcMjhfYZ923mUUu2ADsByh82+Sqn1Sqk1SqnRNc7pBaK1ZkdmHuPioxgWG8bLi3aw4eDJ89LNXnuIw6cKefLabqjS6RikaUcI0cC5MveOs7YN7WQbwATga6211WFbW611plKqI7BcKZWitd5b5gJK3QfcBxAWFkZycrIL2XKuoKCgVsdnn7GRX2zBqyCLURGebNoP9370G88P8iPI2/goii2a134ppFsLE5aMbSRnaLqHXkGmjiW3FteujdqWu7GScjctUu46oLWu9AUMBBY7vH8GeKaCtJuAQZWc62NgbGXXi4+P17WxYsWKWh3/Q8oR3e6p7/Wm9JNaa61TMk7pLn9ZpG/7YI22WG1aa62nrdij2z31vV63/0StrlWXalvuxkrK3bRIuSsGrNdVxHOttUvNO+uALkqpDkopb4za/HmjcJRS3YAWwG8O21oopXzsP4cAg4HU8sc2JKlH8jAp6GafA79nZDDPj+zByj3HeWt5GrmFZt5N3stVMaEktG95kXMrhBDVU2XzjtbaopSaAiwGPIAZWuvtSqmpGHeW0hvALcBs+x2nVHfgPaWUDaP/4BXtMOqnIUrNzKNj62b4eXuc3XbLgGjWH8jhjWW72ZB+krwiC48P63oRcymEEDXj0nz6WutFwKJy254t9/55J8etBuJqkb8LbseRPOLbtSizTSnFizf2ZFtmLr/szub6XhH0aCOjdIQQjY88kevg1JkSDp8qJLZN0Hn7/L09eee2eK6JDePP18ZchNwJIUTtycpZDlKP5AEQG3F+0Afo1LoZ70+SYZlCiMZLavoOUjONoN+9gqAvhBCNnQR9B6lH8ggN9KF1oM/FzooQQtQLCfoOUjPznLbnCyGEu2gSQX/LoVNknDxTaZpii5W0YwUVtucLIYQ7aBJB/4+zNvHMNymVpkk7VoDFpqWmL4Rwa01i9E7OaWMo5snTJbQI8HaaprQTV2r6Qgh35vY1fatNU1BswWrT/JSaVWG61CN5+Ht70K5VwAXMnRBCXFhuH/QLiixnf16YUknQz8wjJjwQD5OsbyuEcF9uH/RLlzwMC/JhddpxTp0pOS+N1prUIzJyRwjh/ppM0B8XH43Fpvlp+9Hz0mScLCS/yEJshMynI4Rwb24f9PPtzTsDO7UiuqUfi7YdOS/N2ekXpKYvhHBzbh/08wqNmn6QrxcjekawKu04uWfMZdKkZpadQ18IIdyV2wf90pp+kJ8nI+IiMFvPH8WTeuT8OfSFEMIduX3QL23TD/T1oldUMJHN/ViUUraJJzUzT8bnCyGaBLcP+qU1/UBfT5RSjIgL59e04+Tam31yz5grnENfCCHcjdsH/bxCM35eHnh5GEUtbeJZmmqM4qlqDn0hhHAnbh/084ssBPmdm22iT3TzMk08pUFf5tAXQjQFLgV9pVSSUmqXUipNKfW0k/3/UUpttr92K6VOOeybrJTaY39NrsvMuyKvyEygr5djXhneM5yVe46TV2QmNVPm0BdCNB1VBn2llAcwDRgOxAK3KKViHdNorR/VWvfRWvcB/gd8Yz+2JfAccAkwAHhOKVV21fF6ll9kIci37Lxyw+MiKLHaWLbjqDyJK4RoUlyp6Q8A0rTW+7TWJcBsYFQl6W8BZtl/vhZYorXO0VqfBJYASbXJcHXlFZkJ8vMqs61vdHMign2ZtymTtGP50p4vhGgyXAn6kcAhh/cZ9m3nUUq1AzoAy6t7bH3JL7KUad4BMJkUw3tG8PPubMxWLe35Qogmw5X59J1NO6krSDsB+Fprba3OsUqp+4D7AMLCwkhOTnYhW84VFBSUOf543mkK/IrPO2eExXrumEM7ST65u8bXbAjKl7upkHI3LVLu2nMl6GcA0Q7vo4DMCtJOAB4qd2xiuWOTyx+ktZ4OTAdISEjQiYmJ5ZO4LDk5mdLjtdYUL/mRbh3bkZgYUybd5TbNhzuWk1toZvyIKxv9lMqO5W5KpNxNi5S79lwJ+uuALkqpDsBhjMB+a/lESqluQAvgN4fNi4F/OHTeDgOeqVWOq6HYYqPEaiszZLOUyaR46KrOZOScafQBXwghXFVl0NdaW5RSUzACuAcwQ2u9XSk1FVivtV5gT3oLMFtrrR2OzVFKvYBx4wCYqrXOqdsiVMxxCgZnbr+03YXKihBCNAgurZGrtV4ELCq37dly75+v4NgZwIwa5q9W8grtk635NomlgIUQokpu/URuftG5aZWFEEK4edDPc5hWWQghhJsH/fwq2vSFEKKpceugf65NX4K+EEKAmwf9czV9ad4RQghw86CfV2TGw6Twl2UQhRACcPOgb8y7Y6yYJYQQws2Dfl6hWdrzhRDCgVsH/dKavhBCCINbB/28IqnpCyGEI7cO+lLTF0KIstw66OcVnr9qlhBCNGVuHfSlpi+EEGW5bdC32jT5xRZp0xdCCAduG/QLio0pGKSmL4QQ57ht0M8rtE+rLG36QghxltsG/fwiWUBFCCHKc9ugnycLqAghxHncNuiX1vRlLn0hhDjHpaCvlEpSSu1SSqUppZ6uIM14pVSqUmq7UuoLh+1WpdRm+2uBs2Prw7k2fWneEUKIUlVGRKWUBzANuAbIANYppRZorVMd0nQBngEGa61PKqVCHU5RqLXuU8f5rpKsmiWEEOdzpaY/AEjTWu/TWpcAs4FR5dLcC0zTWp8E0Fofq9tsVl9ekQzZFEKI8lwJ+pHAIYf3GfZtjroCXZVSq5RSa5RSSQ77fJVS6+3bR9cyvy7LLzLj5+WBl4fbdlsIIUS1uVINdrYCiXZyni5AIhAFrFRK9dRanwLaaq0zlVIdgeVKqRSt9d4yF1DqPuA+gLCwMJKTk6tXCgcFBQUkJyeza38xPiZbrc7VmJSWu6mRcjctUu7acyXoZwDRDu+jgEwnadZorc3AfqXULoybwDqtdSaA1nqfUioZ6AuUCfpa6+nAdICEhASdmJhY/ZLYJScnk5iYyJzDGwgpKSAx8Yoan6sxKS13UyPlblqk3LXnStvHOqCLUqqDUsobmACUH4UzD7gSQCkVgtHcs08p1UIp5eOwfTCQygWQV2iRB7OEEKKcKqOi1tqilJoCLAY8gBla6+1KqanAeq31Avu+YUqpVMAKPKm1PqGUGgS8p5SyYdxgXnEc9VOf8ovMNPf3vhCXEkKIRsOlqrDWehGwqNy2Zx1+1sBj9pdjmtVAXO2zWX15RRbatgq4GJcWQogGy22HtuQXmWW4phBClOO2Qd9o05cHs4QQwpFbBv0is5USq01q+kIIUY5bBv2zM2zKXPpCCFGGWwZ9mUtfCCGcc8ugf3aGTWnTF0KIMtwy6OfLZGtCCOGUWwZ9adMXQgjn3DLoS01fCCGcc8ugL236QgjhnFsG/fwiCx4mhb+3x8XOihBCNChuGfTz7FMwKOVsKQAhhGi63DLo5xdZpD1fCCGccMugn1dolvZ8IYRwwi2DvtT0hRDCObcM+nlFUtMXQghn3DLoGzV9CfpCCFGeWwb9vEIzQX7SvCOEEOW5XdC3aU1BidT0hRDCGZeCvlIqSSm1SymVppR6uoI045VSqUqp7UqpLxy2T1ZK7bG/JtdVxitSaAGtZVplIYRwpsrIqJTyAKYB1wAZwDql1AKtdapDmi7AM8BgrfVJpVSofXtL4DkgAdDABvuxJ+u+KIZCiwZkCgYhhHDGlZr+ACBNa71Pa10CzAZGlUtzLzCtNJhrrY/Zt18LLNFa59j3LQGS6ibrzp0x24O+tOkLIcR5XAn6kcAhh/cZ9m2OugJdlVKrlFJrlFJJ1Ti2ThUaE2xKm74QQjjhSnXY2QQ22sl5ugCJQBSwUinV08VjUUrdB9wHEBYWRnJysgvZcu5EfiGg2LN9C+aMpjPhWkFBQa0+t8ZKyt20SLlrz5WgnwFEO7yPAjKdpFmjtTYD+5VSuzBuAhkYNwLHY5PLX0BrPR2YDpCQkKATExPLJ3HZqs+XACUkDr6U9iEBNT5PY5OcnExtPrfGSsrdtEi5a8+V5p11QBelVAellDcwAVhQLs084EoApVQIRnPPPmAxMEwp1UIp1QIYZt9Wb87Ym3dk1SwhhDhflTV9rbVFKTUFI1h7ADO01tuVUlOB9VrrBZwL7qmAFXhSa30CQCn1AsaNA2Cq1jqnPgpSqnT0jsy9I4QQ53MpMmqtFwGLym171uFnDTxmf5U/dgYwo3bZdN0Zs8bPywMvD7d77kwIIWrN7SLjGYvU8oUQoiLuF/TNWtrzhRCiAm4X9AstWmr6QghRAbcL+mcsMgWDEEJUxO2CfqFZavpCCFERtwv6ZyzSpi+EEBVxw6Avo3eEEKIibhX0i8xWLDZp0xdCiIq4VdDPLzLmYJAFVIQQwjm3Cvp5RWZA5t0RQsouzFYAAAZiSURBVIiKuFXQL63pS5u+EEI451ZBP6/QXtOXNn0hhHDKrYL+uZq+BH0hhHDGrYL+uTZ9ad4RQghn3Cro59uDvtT0hRDCObcK+nmFFhQQ4N101sYVQojqcKugn19kxt8LlHK2HrsQQgi3Cvp5RRb8PSXgCyFERdwq6OcXmfGToC+EEBVyKegrpZKUUruUUmlKqaed7L9DKZWtlNpsf93jsM/qsH1BXWa+vLxCC/7ShyuEEBWqcmyjUsoDmAZcA2QA65RSC7TWqeWSfqm1nuLkFIVa6z61z2rV8qSmL4QQlXKlpj8ASNNa79NalwCzgVH1m62ayZc2fSGEqJQrQT8SOOTwPsO+rbyblFJblVJfK6WiHbb7KqXWK6XWKKVG1yazVcmzj94RQgjhnCuPrjqrOuty778DZmmti5VSfwA+Aa6y72urtc5USnUEliulUrTWe8tcQKn7gPsAwsLCSE5Ork4ZALBpTUGRBU+brtHxjV1BQYGUuwmRcjctdVluV4J+BuBYc48CMh0TaK1POLx9H/inw75M+7/7lFLJQF9gb7njpwPTARISEnRiYqLLBSiVV2RGL/6JYH8fanJ8Y5ecnCzlbkKk3E1LXZbbleaddUAXpVQHpZQ3MAEoMwpHKRXh8PYGYId9ewullI/95xBgMFC+A7hO2Gya63tFENlM2vSFEKIiVdb0tdYWpdQUYDHgAczQWm9XSk0F1mutFwAPK6VuACxADnCH/fDuwHtKKRvGDeYVJ6N+6kRzf2/eurVfk/zqJ4QQrnJpOkqt9SJgUbltzzr8/AzwjJPjVgNxtcyjEEKIOuJWT+QKIYSonAR9IYRoQiToCyFEEyJBXwghmhAJ+kII0YRI0BdCiCZEgr4QQjQhSuvy0+hcXEqpbOBgLU4RAhyvo+w0JlLupkXK3bS4Uu52WuvWVZ2owQX92lJKrddaJ1zsfFxoUu6mRcrdtNRluf+/vfsJ0aoK4zj+/TEZRQWmaIRjWOAiFzVtRLCFDRFTSbYoKApcBG1aGBRhbqLAbbVpEyVt+ieVJa0azKiVlWloWGQhJSO+m6TaGNavxXmGXkzLxb1z8ZznA8O958zl5XmYM897OOedObm8k1JKDcmin1JKDamx6L88dAADybzbknm3pbO8q1vTTymldH41zvRTSimdRzVFX9KMpO8kHZW0deh4+iRph6SRpMNjfUskzUr6Pq5XDxlj1yStlLRX0hFJ30jaEv21532ZpM8lfR15Pxv910vaF3m/HQccVUfShKQDkj6Mdit5H5N0SNJBSV9GXydjvYqiL2kCeAm4E1gDPChpzbBR9eo1YOasvq3AHturgT3RrskZ4AnbNwLrgMfiZ1x73qeBads3A1PAjKR1lCNJX4i8fwEeGTDGPm0hTuILreQNcJvtqbGPanYy1qso+sBa4KjtH23/AbwFbBo4pt7Y/pRyQtm4TZQD6YnrvQsaVM9sn7D9Vdz/RikEK6g/b9v+PZqL4svANPBO9FeXN4CkSeBu4JVoiwby/g+djPVaiv4K4Oex9vHoa8k1tk9AKZDA8oHj6Y2kVcAtwD4ayDuWOA4CI2AW+AE4ZftMPFLreH8ReAr4K9pLaSNvKG/sH0naL+nR6OtkrF/QcYkXgXOdhp4fS6qQpCuBd4HHbf9aJn91s/0nMCVpMbCLcvb0vx5b2Kj6JWkjMLK9X9KG+e5zPFpV3mPW256TtByYlfRtVy9cy0z/OLByrD0JzA0Uy1BOSroWIK6jgePpnKRFlIL/uu33orv6vOfZPgV8QtnTWCxpftJW43hfD9wj6RhluXaaMvOvPW8AbM/FdUR5o19LR2O9lqL/BbA6dvYvBR4Adg8c00LbDWyO+83ABwPG0rlYz30VOGL7+bFv1Z73spjhI+ly4HbKfsZe4L54rLq8bT9te9L2Ksrv88e2H6LyvAEkXSHpqvl74A7gMB2N9Wr+OEvSXZSZwASww/b2gUPqjaQ3gQ2U/7x3EngGeB/YCVwH/ATcb/vszd6LlqRbgc+AQ/yzxruNsq5fc943UTbtJiiTtJ22n5N0A2UGvAQ4ADxs+/RwkfYnlneetL2xhbwjx13RvAR4w/Z2SUvpYKxXU/RTSin9v1qWd1JKKV2ALPoppdSQLPoppdSQLPoppdSQLPoppdSQLPoppdSQLPoppdSQLPoppdSQvwF1G7/bB637JQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train loss    : 0.11518474561827523\n",
      "Train accuracy: 0.9575892857142857\n",
      "Validation loss    : 0.46496586687862873\n",
      "Validation accuracy: 0.8203125\n"
     ]
    }
   ],
   "source": [
    "epochs     = 50\n",
    "batch_size = 64 # this value should be the same with the one used for save_bottlebeck_features()\n",
    "optimizer  = 'adam'\n",
    "drop_rate  = 0.9\n",
    "callbacks  = callbacks=[plot_losses]\n",
    "summary    = False # for the first time show the summary of the trained top model\n",
    "\n",
    "nb_train_samples, nb_validation_samples = save_bottlebeck_features(batch_size)\n",
    "train_top_model(epochs, batch_size, optimizer, drop_rate, callbacks, summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With increased drop rate, the overfitting is decreasing but results are not better than the previous small CNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "If we try to use VGG16 transfer learning for our current dataset, no better results will be obtained than a small CNN (over 90% test accuracy). This could be explained by the training of VGG16 with the Imagenet dataset that is very different with the polyps. In addition, we used the original dataset, without data augmentation because of the transfer learning advantage. \n",
    "\n",
    "Additional fine tunnings ([5-FineTuningVGG16.ipynb](5-FineTuningVGG16.ipynb)) should be used to search for better classifiers for the current task (prediction of a colonoscopy polyp image).\n",
    "\n",
    "Have fun with DL! @muntisa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Acknowledgements\n",
    "\n",
    "I gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research ([https://developer.nvidia.com/academic_gpu_seeding](https://developer.nvidia.com/academic_gpu_seeding))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}