2b_GRU.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    },
    "colab": {
      "name": "2b-GRU.ipynb",
      "provenance": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kkKORoJSWGEl"
      },
      "source": [
        "# GRU\n",
        "\n",
        "Dans ce notebook, nous utilisons un modèle de réseau de neurones récurrents particulier, le **GRU**.\n",
        "Son architecture est plus simple que le LSTM utilisé dans le notebook précédent. Voir [cet article](https://penseeartificielle.fr/comprendre-lstm-gru-fonctionnement-schema/) pour comprendre l'architecture du GRU et du LSTM.\n",
        "## Importer et préparer des données\n",
        "\n",
        "La première étape est la même que dans le notebook précédent.\n",
        "\n",
        "La seule différence est que nous définissons `include_lengths = True` pour notre champ `TEXT`. Cela fera que `batch.text` sera maintenant un tuple avec le premier élément étant notre phrase et le deuxième élément étant les longueurs réelles de nos phrases."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xGoVTz1cWGEm",
        "outputId": "be24cda8-1905-451f-b974-5073313cb162",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import pandas as pd\n",
        "import torch\n",
        "from torchtext import data\n",
        "import random\n",
        "from torchtext import datasets\n",
        "\n",
        "SEED = 1234\n",
        "torch.manual_seed(SEED)\n",
        "torch.backends.cudnn.deterministic = True\n",
        "\n",
        "TEXT = data.Field(sequential=True,lower=True, tokenize = 'spacy', include_lengths=True)\n",
        "LABEL = data.LabelField(dtype = torch.float)\n",
        "\n",
        "train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)\n",
        "test_data, valid_data = test_data.split(random_state = random.seed(1234))\n",
        "\n",
        "print(f'Taille des données train: {len(train_data)}')\n",
        "print(f'Taille des données de validation: {len(valid_data)}')\n",
        "print(f'Taille des données test: {len(test_data)}')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Taille des données train: 25000\n",
            "Taille des données de validation: 7500\n",
            "Taille des données test: 17500\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "94m-weCzWGEu"
      },
      "source": [
        "## Vocabulaire\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-D6U5b2MWGEv"
      },
      "source": [
        "Vient ensuite l'utilisation de word embeding pré-entraînées. Maintenant, au lieu d'avoir nos embeddings de mots initialisés au hasard, ils sont initialisés avec ces vecteurs pré-entraînés. Nous obtenons ces vecteurs simplement en spécifiant quels vecteurs nous voulons et en les passant comme argument à `build_vocab`. `TorchText` gère le téléchargement des vecteurs et les associe aux mots corrects de notre vocabulaire.\n",
        "\n",
        "Ici, nous allons utiliser les \"vecteurs\" glove.6B.100d. **GloVe** est l'algorithme utilisé pour calculer les vecteurs. 6B indique que ces vecteurs ont été entraînés sur 6 milliards de tokens et 100d indique que ces vecteurs sont 100 -dimensionnelle. Pour plus d'informations sur la représentation GloVe, lire [cet article](https://nlp.stanford.edu/projects/glove/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_GgNd5LpWGEw"
      },
      "source": [
        "MAX_VOCAB_SIZE = 50_000\n",
        "\n",
        "TEXT.build_vocab(train_data, \n",
        "                 max_size = MAX_VOCAB_SIZE, \n",
        "                 vectors = \"glove.6B.100d\", \n",
        "                 unk_init = torch.Tensor.normal_)\n",
        "LABEL.build_vocab(train_data)"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dL-CjPAOWGE2"
      },
      "source": [
        "## Itérateur\n",
        "\n",
        "Ici tous les tenseurs d'un batch doivent être triés par leurs longueurs. Ceci est géré dans l'itérateur en définissant `sort_within_batch = True`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Fc5GuU5WGE3"
      },
      "source": [
        "BATCH_SIZE = 32\n",
        "\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "#device = 'cpu'\n",
        "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
        "    (train_data, valid_data, test_data), \n",
        "    batch_size = BATCH_SIZE,\n",
        "    device = device,sort_key=lambda x: len(x.text),\n",
        "    sort_within_batch = True)"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j5hlZlTXWGE8"
      },
      "source": [
        "## Modèle\n",
        "\n",
        "Le modèle utilisé ici est le même que celui du notebook précédent, à la seule différence que l'on utilise ici un **GRU** au lieu d'un **LSTM**.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZVhOOkncWGE9"
      },
      "source": [
        "import torch.nn as nn\n",
        "\n",
        "class RNN(nn.Module):\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers, \n",
        "                 bidirectional, dropout, pad_idx):\n",
        "        \n",
        "        super().__init__()\n",
        "        \n",
        "        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx = pad_idx)\n",
        "        \n",
        "        self.rnn = nn.GRU(embedding_dim, \n",
        "                           hidden_dim, \n",
        "                           num_layers=n_layers, \n",
        "                           bidirectional=bidirectional, \n",
        "                           dropout=dropout)\n",
        "        \n",
        "        self.fc = nn.Linear(hidden_dim * 2, output_dim)\n",
        "        \n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        \n",
        "    def forward(self, text, text_lengths):\n",
        "        \n",
        "        #text = [sent len, batch size]\n",
        "        embedded = self.dropout(self.embedding(text))    \n",
        "        #embedded = [sent len, batch size, emb dim]\n",
        "        \n",
        "        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths)\n",
        "        packed_output, hidden = self.rnn(packed_embedded)\n",
        "        output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)\n",
        "\n",
        "        #output = [sent len, batch size, hid dim * num directions]   \n",
        "        #hidden = [num layers * num directions, batch size, hid dim]\n",
        "        #cell = [num layers * num directions, batch size, hid dim]\n",
        "        \n",
        "        hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1))           \n",
        "        #hidden = [batch size, hid dim * num directions]\n",
        "            \n",
        "        return self.fc(hidden)"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HrG9ZXpsWGFC"
      },
      "source": [
        "Comme auparavant, nous allons créer une instance de notre classe RNN, avec les nouveaux paramètres et arguments pour le nombre de couches, la bidirectionnalité et la probabilité de dropout.\n",
        "Pour garantir que les vecteurs pré-entraînés peuvent être chargés dans le modèle, `EMBEDDING_DIM` doit être égal à celui des vecteurs GloVe pré-entraînés chargés précédemment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cRFx6I8lWGFD",
        "outputId": "b72e067e-e3b0-479d-8f22-017f93f5e9b0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "INPUT_DIM = len(TEXT.vocab)\n",
        "EMBEDDING_DIM = 100\n",
        "HIDDEN_DIM = 100\n",
        "OUTPUT_DIM = 1\n",
        "N_LAYERS = 2\n",
        "BIDIRECTIONAL = True\n",
        "DROPOUT = 0.6\n",
        "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
        "\n",
        "model = RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM, N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
        "\n",
        "def count_parameters(model):\n",
        "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "\n",
        "print(f'Le modèle a {count_parameters(model):,} paramètres à entraîner')"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Le modèle a 5,302,801 paramètres à entraîner\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bpVUcsyOWGFH"
      },
      "source": [
        "Le dernier ajout consiste à copier les embeddings de mots pré-entraînées que nous avons chargées précédemment dans la couche d'embedding de notre modèle.\n",
        "Nous récupérons les embeddings à partir du vocabulaire de `Field` et vérifions qu'elles sont de la bonne taille, **[taille du vocabulaire,dim embedding]**\n",
        "\n",
        "Comme nos tokens `<unk>` et `<pad>` ne sont pas dans le vocabulaire pré-entraîné, ils ont été initialisés en utilisant `unk_init` (une distribution $ \\mathcal {N} (0,1) $) lors de la construction de notre vocabulaire. Il est préférable de les initialiser tous les deux à zéros pour indiquer explicitement à notre modèle que, au départ, ils ne sont pas pertinents pour déterminer le sentiment.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gygvwMD7WGFI"
      },
      "source": [
        "pretrained_embeddings = TEXT.vocab.vectors\n",
        "model.embedding.weight.data.copy_(pretrained_embeddings)\n",
        "\n",
        "UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]\n",
        "\n",
        "model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g1NifTvZWGFM"
      },
      "source": [
        "## Métriques\n",
        "\n",
        "On utilise l'**accuracy** comme métrique."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QVv7dlOyWGFN"
      },
      "source": [
        "def binary_accuracy(preds, y):\n",
        "    rounded_preds = torch.round(torch.sigmoid(preds))\n",
        "    correct = (rounded_preds == y).float()\n",
        "    acc = correct.sum() / len(correct)\n",
        "    return acc"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YV7BiND6WGFQ"
      },
      "source": [
        "## Optimiseur et fonction de perte"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "miVoFFFpWGFR"
      },
      "source": [
        "import torch.optim as optim\n",
        "\n",
        "def optimiseur(model, lr=1e-4, eps=1e-6, weight_decay_rate=0.001, second_weight_decay_rate=0.0):\n",
        "    param_optimizer = list(model.named_parameters())\n",
        "    no_decay = ['bias', 'gamma', 'beta']\n",
        "    optimizer_grouped_parameters = [\n",
        "        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
        "            'weight_decay_rate': weight_decay_rate},\n",
        "        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
        "            'weight_decay_rate': second_weight_decay_rate}]\n",
        "    return optim.Adam(\n",
        "        optimizer_grouped_parameters,\n",
        "        lr=lr,\n",
        "        eps=eps\n",
        "    )\n",
        "\n",
        "optimizer = optimiseur(model, lr=1e-4, eps=1e-6, weight_decay_rate=0.001, second_weight_decay_rate=0.0)\n",
        "criterion = nn.BCEWithLogitsLoss()\n",
        "model = model.to(device)\n",
        "criterion = criterion.to(device)"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zl6Gt4UDWGFV"
      },
      "source": [
        "## Entraîner le modèle"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A4TL6kDfWGFV",
        "outputId": "67b96277-35e6-4ae0-b604-0e6f65e5ee0a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "def train(model, iterator, optimizer, criterion):\n",
        "    \n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "    \n",
        "    model.train()\n",
        "    \n",
        "    for batch in iterator:\n",
        "        \n",
        "        optimizer.zero_grad()\n",
        "        text, text_lengths = batch.text       \n",
        "        predictions = model(text, text_lengths.to('cpu')).squeeze(1)\n",
        "        loss = criterion(predictions, batch.label)\n",
        "        \n",
        "        acc = binary_accuracy(predictions, batch.label)\n",
        "        \n",
        "        loss.backward()\n",
        "        \n",
        "        optimizer.step()\n",
        "        \n",
        "        epoch_loss += loss.item()\n",
        "        epoch_acc += acc.item()     \n",
        "        \n",
        "    return epoch_loss / len(iterator), epoch_acc / len(iterator)\n",
        "\n",
        "def evaluate(model, iterator, criterion):\n",
        "    \n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "    \n",
        "    model.eval()\n",
        "    \n",
        "    with torch.no_grad():\n",
        "    \n",
        "        for batch in iterator:\n",
        "            \n",
        "            text, text_lengths = batch.text       \n",
        "            predictions = model(text, text_lengths.to('cpu')).squeeze(1) \n",
        "            loss = criterion(predictions, batch.label)    \n",
        "\n",
        "            acc = binary_accuracy(predictions, batch.label)\n",
        "            \n",
        "            epoch_loss += loss.item()\n",
        "            epoch_acc += acc.item()\n",
        "        \n",
        "    return epoch_loss / len(iterator), epoch_acc / len(iterator)\n",
        "\n",
        "import time\n",
        "\n",
        "def epoch_time(start_time, end_time):\n",
        "    elapsed_time = end_time - start_time\n",
        "    elapsed_mins = int(elapsed_time / 60)\n",
        "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "    return elapsed_mins, elapsed_secs\n",
        "\n",
        "\n",
        "N_EPOCHS = 50\n",
        "\n",
        "best_valid_loss = float('inf')\n",
        "tloss = []\n",
        "tacc = []\n",
        "vloss = []\n",
        "vacc = []\n",
        "for epoch in range(N_EPOCHS):\n",
        "\n",
        "    start_time = time.time()\n",
        "    \n",
        "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
        "    tloss.append(train_loss)\n",
        "    tacc.append(train_acc)  \n",
        "    \n",
        "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)\n",
        "    vloss.append(valid_loss)\n",
        "    vacc.append(valid_acc)     \n",
        "    end_time = time.time()\n",
        "\n",
        "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "    \n",
        "    if valid_loss < best_valid_loss:\n",
        "        best_valid_loss = valid_loss\n",
        "        torch.save(model.state_dict(), 'tut2-model.pt')\n",
        "    \n",
        "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
        "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.690 | Train Acc: 52.95%\n",
            "\t Val. Loss: 0.669 |  Val. Acc: 61.12%\n",
            "Epoch: 02 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.635 | Train Acc: 63.60%\n",
            "\t Val. Loss: 0.633 |  Val. Acc: 66.64%\n",
            "Epoch: 03 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.547 | Train Acc: 72.73%\n",
            "\t Val. Loss: 0.554 |  Val. Acc: 75.62%\n",
            "Epoch: 04 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.492 | Train Acc: 76.65%\n",
            "\t Val. Loss: 0.479 |  Val. Acc: 80.12%\n",
            "Epoch: 05 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.446 | Train Acc: 79.70%\n",
            "\t Val. Loss: 0.609 |  Val. Acc: 76.04%\n",
            "Epoch: 06 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.418 | Train Acc: 81.00%\n",
            "\t Val. Loss: 0.422 |  Val. Acc: 81.77%\n",
            "Epoch: 07 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.387 | Train Acc: 82.77%\n",
            "\t Val. Loss: 0.325 |  Val. Acc: 86.14%\n",
            "Epoch: 08 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.360 | Train Acc: 84.31%\n",
            "\t Val. Loss: 0.353 |  Val. Acc: 86.09%\n",
            "Epoch: 09 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.340 | Train Acc: 85.47%\n",
            "\t Val. Loss: 0.347 |  Val. Acc: 86.17%\n",
            "Epoch: 10 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.320 | Train Acc: 86.71%\n",
            "\t Val. Loss: 0.328 |  Val. Acc: 86.83%\n",
            "Epoch: 11 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.312 | Train Acc: 86.75%\n",
            "\t Val. Loss: 0.299 |  Val. Acc: 87.36%\n",
            "Epoch: 12 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.296 | Train Acc: 87.85%\n",
            "\t Val. Loss: 0.301 |  Val. Acc: 87.13%\n",
            "Epoch: 13 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.288 | Train Acc: 88.00%\n",
            "\t Val. Loss: 0.274 |  Val. Acc: 88.62%\n",
            "Epoch: 14 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.276 | Train Acc: 88.68%\n",
            "\t Val. Loss: 0.299 |  Val. Acc: 88.13%\n",
            "Epoch: 15 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.268 | Train Acc: 89.15%\n",
            "\t Val. Loss: 0.255 |  Val. Acc: 89.13%\n",
            "Epoch: 16 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.258 | Train Acc: 89.65%\n",
            "\t Val. Loss: 0.253 |  Val. Acc: 89.47%\n",
            "Epoch: 17 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.250 | Train Acc: 89.73%\n",
            "\t Val. Loss: 0.261 |  Val. Acc: 89.73%\n",
            "Epoch: 18 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.240 | Train Acc: 90.26%\n",
            "\t Val. Loss: 0.278 |  Val. Acc: 89.07%\n",
            "Epoch: 19 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.238 | Train Acc: 90.42%\n",
            "\t Val. Loss: 0.260 |  Val. Acc: 89.17%\n",
            "Epoch: 20 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.234 | Train Acc: 90.85%\n",
            "\t Val. Loss: 0.267 |  Val. Acc: 89.33%\n",
            "Epoch: 21 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.224 | Train Acc: 91.24%\n",
            "\t Val. Loss: 0.286 |  Val. Acc: 88.89%\n",
            "Epoch: 22 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.220 | Train Acc: 91.30%\n",
            "\t Val. Loss: 0.248 |  Val. Acc: 90.40%\n",
            "Epoch: 23 | Epoch Time: 0m 52s\n",
            "\tTrain Loss: 0.211 | Train Acc: 91.82%\n",
            "\t Val. Loss: 0.299 |  Val. Acc: 89.14%\n",
            "Epoch: 24 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.202 | Train Acc: 92.22%\n",
            "\t Val. Loss: 0.263 |  Val. Acc: 90.43%\n",
            "Epoch: 25 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.201 | Train Acc: 92.27%\n",
            "\t Val. Loss: 0.255 |  Val. Acc: 90.56%\n",
            "Epoch: 26 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.194 | Train Acc: 92.39%\n",
            "\t Val. Loss: 0.251 |  Val. Acc: 90.63%\n",
            "Epoch: 27 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.184 | Train Acc: 92.79%\n",
            "\t Val. Loss: 0.265 |  Val. Acc: 90.42%\n",
            "Epoch: 28 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.185 | Train Acc: 92.90%\n",
            "\t Val. Loss: 0.258 |  Val. Acc: 90.48%\n",
            "Epoch: 29 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.182 | Train Acc: 93.00%\n",
            "\t Val. Loss: 0.305 |  Val. Acc: 89.18%\n",
            "Epoch: 30 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.178 | Train Acc: 93.11%\n",
            "\t Val. Loss: 0.244 |  Val. Acc: 90.58%\n",
            "Epoch: 31 | Epoch Time: 0m 52s\n",
            "\tTrain Loss: 0.174 | Train Acc: 93.11%\n",
            "\t Val. Loss: 0.266 |  Val. Acc: 90.19%\n",
            "Epoch: 32 | Epoch Time: 0m 52s\n",
            "\tTrain Loss: 0.168 | Train Acc: 93.81%\n",
            "\t Val. Loss: 0.239 |  Val. Acc: 90.50%\n",
            "Epoch: 33 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.162 | Train Acc: 93.80%\n",
            "\t Val. Loss: 0.276 |  Val. Acc: 89.66%\n",
            "Epoch: 34 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.160 | Train Acc: 93.87%\n",
            "\t Val. Loss: 0.250 |  Val. Acc: 90.58%\n",
            "Epoch: 35 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.155 | Train Acc: 94.21%\n",
            "\t Val. Loss: 0.249 |  Val. Acc: 90.41%\n",
            "Epoch: 36 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.149 | Train Acc: 94.40%\n",
            "\t Val. Loss: 0.258 |  Val. Acc: 90.39%\n",
            "Epoch: 37 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.145 | Train Acc: 94.52%\n",
            "\t Val. Loss: 0.274 |  Val. Acc: 90.30%\n",
            "Epoch: 38 | Epoch Time: 0m 49s\n",
            "\tTrain Loss: 0.139 | Train Acc: 94.67%\n",
            "\t Val. Loss: 0.267 |  Val. Acc: 90.47%\n",
            "Epoch: 39 | Epoch Time: 0m 50s\n",
            "\tTrain Loss: 0.133 | Train Acc: 95.00%\n",
            "\t Val. Loss: 0.284 |  Val. Acc: 90.42%\n",
            "Epoch: 40 | Epoch Time: 0m 49s\n",
            "\tTrain Loss: 0.136 | Train Acc: 94.84%\n",
            "\t Val. Loss: 0.296 |  Val. Acc: 90.03%\n",
            "Epoch: 41 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.134 | Train Acc: 94.91%\n",
            "\t Val. Loss: 0.290 |  Val. Acc: 90.20%\n",
            "Epoch: 42 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.129 | Train Acc: 95.25%\n",
            "\t Val. Loss: 0.294 |  Val. Acc: 90.27%\n",
            "Epoch: 43 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.124 | Train Acc: 95.29%\n",
            "\t Val. Loss: 0.309 |  Val. Acc: 90.19%\n",
            "Epoch: 44 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.117 | Train Acc: 95.59%\n",
            "\t Val. Loss: 0.293 |  Val. Acc: 90.31%\n",
            "Epoch: 45 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.122 | Train Acc: 95.54%\n",
            "\t Val. Loss: 0.275 |  Val. Acc: 90.62%\n",
            "Epoch: 46 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.116 | Train Acc: 95.64%\n",
            "\t Val. Loss: 0.297 |  Val. Acc: 90.40%\n",
            "Epoch: 47 | Epoch Time: 0m 52s\n",
            "\tTrain Loss: 0.112 | Train Acc: 95.84%\n",
            "\t Val. Loss: 0.302 |  Val. Acc: 90.33%\n",
            "Epoch: 48 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.110 | Train Acc: 95.94%\n",
            "\t Val. Loss: 0.313 |  Val. Acc: 90.32%\n",
            "Epoch: 49 | Epoch Time: 0m 50s\n",
            "\tTrain Loss: 0.107 | Train Acc: 95.92%\n",
            "\t Val. Loss: 0.307 |  Val. Acc: 90.29%\n",
            "Epoch: 50 | Epoch Time: 0m 51s\n",
            "\tTrain Loss: 0.106 | Train Acc: 96.00%\n",
            "\t Val. Loss: 0.303 |  Val. Acc: 90.11%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jfNVPGGbWGFZ",
        "outputId": "6b09f064-4d27-4e2a-8d97-65828e46c07c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        }
      },
      "source": [
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt \n",
        "import numpy as np \n",
        "sns.set()\n",
        "\n",
        "x = np.linspace(0, N_EPOCHS,N_EPOCHS)\n",
        "\n",
        "plt.plot(x,tloss)\n",
        "plt.plot(x,vloss)\n",
        "plt.title(\"Loss\")\n",
        "plt.legend([\"Train loss\", \"Valid loss\"])"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f98864e6630>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAELCAYAAADTK53JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xW9d3/8dc555rZyZVNdoAQIAwZsh2ooASxLlqqtrd3sa32trW970qX46a2N7a29bZqK7+6bmxr0SoaKYqKKMiegbBCEpKQkL2Ta5/fH4FgTEIG2fk8Hw8fwpVzzvX5JuSdk+/5DkXXdR0hhBDDjjrQBQghhOgbEvBCCDFMScALIcQwJQEvhBDDlAS8EEIMUxLwQggxTEnACyHEMCUBL0aca6+9ls8//3ygyxCiz0nACyHEMCUBLwTgdDp54oknmDdvHvPmzeOJJ57A6XQCUFlZybe//W2mT5/OzJkzWbFiBV6vF4AXXniB+fPnM3XqVBYtWsSOHTsGshlCtGIY6AKEGAyef/55Dh06xIYNG1AUhfvvv5/nnnuOH/zgB7z00ktERES0hPehQ4dQFIWcnBxee+013njjDSIiIigsLGwJfiEGA7mDFwJ49913eeCBB7DZbISEhPDAAw/wzjvvAGAwGCgrK6OoqAij0cj06dNRFAVN03A6nZw+fRqXy0VMTAxxcXED3BIhLpKAFwIoLS0lOjq65e/R0dGUlpYC8O///u/Ex8dz7733snDhQl544QUA4uPj+elPf8ozzzzDnDlzeOihhygpKRmQ+oVojwS8EEB4eDhFRUUtfy8uLiY8PBwAPz8/Vq1axUcffcTzzz/PSy+91NJds3TpUv72t7+xZcsWFEXht7/97YDUL0R7JODFiORyuXA4HC3/LVmyhOeff57KykoqKyt59tlnWbp0KQBbtmzhzJkz6LqOv78/mqa19MHv2LEDp9OJyWTCbDajqvItJQYPecgqRqT77ruv1d/vvfdeJk6cyM033wzA4sWLuf/++wE4c+YMq1evprKykoCAAL72ta8xa9Ysjh8/zlNPPcXp06cxGo1MnTqV//7v/+73tgjREUU2/BBCiOFJfp8UQohhSgJeCCGGKQl4IYQYpiTghRBimJKAF0KIYUoCXgghhqlBNQ6+qqoBr7f7ozZtNj8qKur7oKLBS9o8MkibR4aetllVFYKDfTv8+KAKeK9X71HAXzh3pJE2jwzS5pGhL9rcpYDPzc1l1apVVFdXExQUxJo1a0hISGh1zI9//GNOnDjR8vcTJ07w7LPPsnDhwl4tWAghRNd0KeAfffRRVqxYwbJly9iwYQOPPPIIr776aqtjnnzyyZY/Hz9+nG984xvMnz+/d6sVQgjRZZ0GfEVFBVlZWbz00ksApKent6zLERIS0u45b7zxBkuXLsVkMvVutUKIQUXXdaqqynA67UDvdDGUlqojbuOUS7dZwWSyEBwchqIo3bpupwFfXFxMREQEmqYBoGka4eHhFBcXtxvwTqeTd999l5dffrlbhUDzg4aeCgvz7/G5Q5W0eWQYzG0uLS3FYFAJC4tDUWRQXl/QdS+VleWAnbCw8G6d2+sPWT/88EOio6NJTU3t9rkVFfU9etAQFuZPWVldt88byqTNI8Ngb3NZWQUhIRF4PAC9c9dtMKi43SPrDr6zNvv6BlJWVoKiWFu9rqrKJW+MO/2RGxUVRUlJCZ7mryAej4fS0lKioqLaPf7NN9/ktttu6+yyQohhwOv1oGmDajDesKRpBrxeT7fP6zTgbTYbqampZGRkAJCRkUFqamq73TPnzp1j3759LRsl9IcjuRV87zcf43B2v/FCiMvX3X5h0X09/Rx36UfvY489xqpVq3juuecICAhgzZo1AKxcuZIHH3yQtLQ0AN566y2uueYaAgMDe1RMTxhUlTPn6jicU8GMcd3rnxJCDB8rV34Dl8uF2+2ioCCfxMRkAMaOTeGnP320S9d4++03cDgcLF/+9W699+23L+XJJ39PUtLobtfdl7oU8MnJyaxfv77N62vXrm319+9+97u9U1U3jI0NIsjfzJ5jJRLwQoxga9e+AkBxcRHf+tbdvPzyX9sc43a7MRg6jr1bbrm9z+obCEO+80xVFeakRfHh7nwcTg9mkzbQJQkhBpHbb1/KwoU3sH//HpKSRnPffffz2GM/o6GhAafTyZw5c7n//u8D8Je//Jmmpia+970fsHHju2zevAl//wByck7j7+/HL3/5JDZb6CXfr7CwgN/85ldUV1ehaRr33fcAs2bNwW6388tfPkpeXg6aZiAuLp7Vq/+H/Pw8fvWrx2lqsuP1erjxxqWsWHF3r7R9yAc8wLzJo9j4eZ500wgxwLZnFrPtcPFlXUNRoL2NROdNimJuWvuDOzrT0NDA2rXNkzMdDgdr1vweHx8f3G43P/zh99i583NmzZrT5rxjx7J45ZW/ERERyZo1v+SNN17n299+4JLv9fjjP2fZsq+Qnn4Lubk5fO97K1m37g0OHz5IY2MD69Y194bU1tYC8M9/vsH8+Vfx9a9/s9XrvWFYBPz4JBsBvib2HC+VgBdCtLF48ZKWP3u9Xp577mkyMw8DOhUVFZw6dbLdgJ80aTIREZEATJgwkT17dl3yfRobG8jOPslNNzVv3p6YmMTo0SkcPZrJ6NFjyMvL5amn1jB16jTmzJkHwJQpU3n++WdobGziiiumc8UV03up1cMk4DVVYdrYMLYfKcbh8mA2SjeNEANhblrP77Iv6Itx8D4+F8ePv/76a9TV1fLCCy9jNptZs+YJnE5Hu+d9cTa+qmotw8V7YtSoGNat+wd79+5h587tvPDCs7zyyt+5+uqFTJ48hR07Pmfdupd57713eOSR1T1+ny8a8lPPPFVFVHz0KtPH2nC6vGSerhjokoQQg1hdXR02Wyhms5myslK2bdvaa9f28fFl9Oix/OtfzcPK8/JyOX36JBMmpFFaWoKqaixYcDUPPvgjqqurqKurpbCwAJvNxk03LeXf/m0lWVlHe62eIX8HrzdUUrNzA0kLbAT4GNlzvJTp0k0jhOjAHXd8lV/84mHuvvtOwsIimDZtRq9e/9FHf8lvfvMr/vGPv6JpGj//+X8THBzMjh3b+dOf/gg0TxC7665vEhoaxquvvsjmzZswGAwoisL3v/+jXqtF0fX2HmcMjJ4sVaDrOq6MX+Ksq+btkHvZdrSMpx+cP+y7aQb7FPa+IG0efM6dO0NkZHyvXlOWKmhfe5/ry16qYLBTFIXgq1eg11ewwDdHummEEOK8IR/wANbEyWhRKYTkfYjNR2HP8dKBLkkIIQbcsAh4RVEwzbgNvamGWyPzOXS6HIdL1qYRQoxswyLgAQyRY9FiJ5HasAvVbZduGiHEiDdsAh7APONWNFcji/xPsPeEdNMIIUa2YRXwWmgChsTpzDce5dTpszilm0YIMYINq4AHME3/CgbdxXztEJk50k0jhBi5hl3Aa8GjMIyZzQLLCY4czRnocoQQ/eRHP3qQt99+o9Vruq5zxx3LOHBgX4fnPfHEY7z55utA83rwr7/+WrvHbdz4Lj//+Y/b/djtty8lJye7h5X3nWEX8ACWabegKTqRRZ9IN40QI8SSJTezcWNGq9cOHNiHqipMmXJFl65xyy23d3uzj8FsyC9V0B41IJyGmFlcWbCDUydzmTBhcO2yIsRw5Tq5HdeJTy/rGoqi0N4Ee2PKAoxj53Z43vz5V/HUU78mLy+XhIREAN577x1uumkpOTmneeqp/8Fub8LpdHLzzV/hzjtXtLnGF9eDd7lc/P73T7J//14CA4MYMyalS/XLevD9IGj6TXgKP6cx5yBIwAsx7BmNRq6//kY2bnyH++//Po2NDXz22VbWrfsHfn5+/OEPz2EymWhsbOS++77BzJmzW34QtGfDhjcpLi5i3br1uN1uHnhgJVFRna+UKevB9wNr2ChKsKBVnB7oUoQYMYxj517yLrsrLmctmiVLbuY///M/+Pa3v8dHH20mLW0y4eERVFZW8Mc//g/Z2SdRFJXy8jKys09eMuD379/HjTemYzAYMBgMLFp0I4cPH7zk+w+29eCHZR88NP+aV2WJI8Re2O6ve0KI4WfMmLHYbGHs3Pk5Gze+w5IlzUH75z8/S0iIjRdffI1XXvkbqakTcDqd/VrbhfXgZ8y4kr17d/HNb34Nh8PB1Vcv5E9/+sv5j7/M6tWP9Np7DtuAB9BDk7CpdVScKxnoUoQQ/WTJkpt58cUXKCjIZ/78qwCor68jPDwCg8FATk42hw5d+k4cYNq06WzatBG3243DYWfz5k2dniPrwfejgIRUKNxEefZRQqMiB7ocIUQ/uP76xTz77NPcfPNXMBqNAHzjG//O6tWP8N57G4iNjWPKlKmdXufmm28lOzubu+66g8DAIMaNm0BVVedza2Q9+A70ZD146HjNbJfTSd1L36UwaDqTln+3N0ocNAb7OuF9Qdo8+Mh68L1D1oPvAaPJRIkagU9d3kCXIoQQ/W5YBzxAY0ACoZ5SPE57j6/hqTyLO7/zPjshhBhMuhTwubm5LF++nEWLFrF8+XLy8vLaPW7jxo0sXbqU9PR0li5dSnl5eW/W2iOmqDFoik5pdlaPr+HcvwH7J3/pxaqEGD4GUS/vsNXTz3GXAv7RRx9lxYoVvP/++6xYsYJHHmk7jCczM5M//vGPvPjii2RkZPDXv/4Vf3//HhXVm0LHTASg7syxHl/DW1uCbq9D97h7qywhhgVV1fDI90Wf83jcqGr395nuNOArKirIysoiPT0dgPT0dLKysqisrGx13Msvv8y9995LWFgYAP7+/pjN5m4X1NsiIkM55w1CKevZhCdd1/HWNK8trzfV9GZpQgx5VqsfdXXV6PrIeijan3TdS11dFVZrxw9TO9LpMMni4mIiIiLQtOafHpqmER4eTnFxMSEhIS3HnT59mpiYGL7+9a/T2NjI9ddfz3e/+10URelyMZd6GtyZsLCOf1vYZ40l0X6C0FBfFKV7jx08DTXUu5oACDQ5sVziffrbpdo8XEmbBxebzZeCggLKy88iPTV9Q1HA19eX2NhRqGr38qvXxsF7PB5OnDjBSy+9hNPp5Fvf+hbR0dHccsstXb5Gbw+TvMAVnIj5XCZns7Iwh3dvSJen5OKSw5VFRRhNna9F0R8G+/C5viBtHpx8fELw8Qnp/MAuGgpt7m1daXNFRUOb1y57mGRUVBQlJSV4PM3L7no8HkpLS9ssuhMdHc3ixYsxmUz4+fmxcOFCDh8+3Nnl+4Vf3HgAyrOPdPtcb83FWbB6Q3Wv1SSEEH2t04C32WykpqaSkdE89TYjI4PU1NRW3TPQ3De/bds2dF3H5XKxc+dOxo0b1zdVd1NsUgI1XivOopPdPtdbW9L8O5KioDdKwAshho4udeg89thjrFu3jkWLFrFu3Toef/xxAFauXElmZiYAS5YsOb+ewk3ccsstjB49mttvv73vKu+G4AALhURgqcnr9rnemlIUv1AUiz96ozxkFUIMHV3qg09OTmb9+vVtXl+7dm3Ln1VV5Sc/+Qk/+clPeq+6XtTgF49fYx7ehipU3+Aun+etLUENjEBvqsMrd/BCiCFk2M9kvUCLHAtAU8Hxbp3nrS1FDQhH8QmUO3ghxJAyYgI+LHEsDt1AdV7XZ7Tq9npwNKAGRKD6BEkfvBBiSBkxAZ8wKoh8tw2ltOs7n3trmyc4qYHn7+DttehemdAhhBgaRkzA+1qMlBhG4Ws/h+7q2sJj3trmIZJKQASKTxDoOrq99/ZLFEKIvjRiAh7AHZKEio6nNKfzg+H8EgUKqn9oc8CDdNMIIYaMERXwvrEpeHWoz+/ag1ZvbQmKXwiKwYTqEwjIZCchxNAxogI+Ljacc54g7IVdDPiaEtSAcICWO3gZKimEGCpGVsCH+5HricBck9elh6V6bSlqQAQAyoU7eBkqKYQYIkZUwJuMGtU+sRi8TrxVhZc8Vnc0oNvrUAPP38FrRhSzn/TBCyGGjBEV8ACGiNEAuIsvvS6Nt7YMaB5Bc4EiY+GFEEPIiAv48JhYqjw+NORfeoenC0Mk1cAvBnwgXumiEUIMESMu4JOiAzntjoCSk5fc5/DCMsFqQFjLa3IHL4QYSkZcwEeH+lKoRGN01aHXlnR4nLe2FMU3GMVwcdtB1ScQvalGNhkWQgwJIy7gVVXBGpcKgKOw424a/fwiY1+k+ASB14PuqO/TGoUQojeMuIAHGDdxHHVeC5XZmR0e46051zJE8oKW2awy2UkIMQSMzICPDyFPj0Qra3/hMd3ZhN5UixL4pTt4X1muQAgxdIzIgDdoKh7baHy9tTiq2vbDt6wi+aU7eFXWoxFCDCEjMuABwsdNBiA/c3+bj10M+C/3wTfPZpWhkkKIoWDEBnzS+FQadRMNZ9puANIyBv7LAW8wg9Eqd/BCiCFhxAa80WCg2hpHYEMeTpen1cf0mlIUawCKydrmPNUnUAJeCDEkjNiAB7DEjSdUrePY8dxWr3trS9r0v1/QPNlJumiEEIPfiA74qPP98MVZB1u97q0tbTOC5gLFJ0iWDBZCDAkjOuCN4Qm4FBNKeXZLN43udqA3VF3iDj4QvVFmswohBr8RHfCKquEOSSJRKeZIbiXwxY222w941TcIPE5wNfVbnUII0RNdCvjc3FyWL1/OokWLWL58OXl5eW2OeeaZZ5g9ezbLli1j2bJlPP74471da5/wT5xAlKGGzKw84MI+rG3HwF/QsrOTzGYVQgxyhq4c9Oijj7JixQqWLVvGhg0beOSRR3j11VfbHHfLLbfw8MMP93qRfckUPQ43UH8mC5f7ipYFyL64iuQXtdp8Ozi6v8oUQohu6/QOvqKigqysLNLT0wFIT08nKyuLysrKPi+uP6hhiXhVI3HKOY7kVOKtKUWx+KOYfds9/uLWfXIHL4QY3DoN+OLiYiIiItA0DQBN0wgPD6e4uLjNse+99x5Lly7l3nvv5cCBA71fbR9QNAOGiNGMNZWw53gp3toSlID2R9DAF5crkKGSQojBrUtdNF3x1a9+le985zsYjUa2b9/O/fffz8aNGwkODu7yNWw2vx6/f1iYf4/PNYyehKf4dU6cLoKoMnzixnd4PV33o8FgwkIjtst4z95wOW0eqqTNI4O0uXd0GvBRUVGUlJTg8XjQNA2Px0NpaSlRUVFfKu5in/XcuXOJiori1KlTzJw5s8vFVFTU4/V2f/hhWJg/ZWV13T7vAndgAgoQ5y3EXVuB0xxy6etZA2koL8N7Ge95uS63zUORtHlkkDZ3naoql7wx7rSLxmazkZqaSkZGBgAZGRmkpqYSEhLS6riSkourMh47doyzZ8+SmJjY7YIHghaeDKqB2T45KOht1qD5MlW27hNCDAFd6qJ57LHHWLVqFc899xwBAQGsWbMGgJUrV/Lggw+SlpbG7373O44ePYqqqhiNRp588slWd/WDmWIwoYUnMe7cKQDsZhvGSx3vG4S3oqB/ihNCiB7qUsAnJyezfv36Nq+vXbu25c8XQn+o0iLH4jl3EoC9Z+GauI6PVXyC8BYc6afKhBCiZ0b0TNYv0qLHAWDHzMdHqy+5FIHiEwiuJnSXo7/KE0KIbpOAP08LTwZFxeMbxtnyBk6fre3wWNnZSQgxFEjAn6eYrBgSp+GfPBmzSWProbMdH3thuQIJeCHEICYB/wXW6x7Ad9YdXJkawZ5jpTTa3e0ed3E2q0x2EkIMXhLw7bhqSjROt5ddWefa/bgiXTRCiCFAAr4dCZH+xIb7sfVQUbsfV8x+oGoS8EKIQU0Cvh2KorBgcjT5JfXknWv7sFVRlPM7O0kXjRBi8JKA78DsCRGYDCqfHuzgLl423xZCDHIS8B3wsRiZMS6cHVkl2J1tH7aqsvm2EGKQk4C/hAVTonE4Pew+VtrmY81dNFUDUJUQQnSNBPwljB4VSJTNh0/bediq+ASCowHd4xqAyoQQonMS8JegKApXTY4mp6iWwtL61h+TjT+EEIOcBHwnZk+MxKApbYZMqrJ1nxBikJOA74S/j4npKeFsO1xMeU1Ty+sXlyuQO3ghxOAkAd8Fty5IAuDV90+0rDIps1mFEIOdBHwXhAZZue2qJI7kVLLjaPPyBYolABRFAl4IMWhJwHfRtVfEMHpUIH/78BQ1DU4UVUWxBspDViHEoCUB30WqqvDNG8fhcHn46+bmnZ8Un0BZMlgIMWhJwHdDdKgvS+cmsud4KftPlqHIbFYhxCAmAd9NN14ZR0yYH//3wQm8pgB0mc0qhBikJOC7yaCp3LtkHLUNTrJKPehNdehez0CXJYQQbUjA90BCZACLZ8ZxuNgL6DKSRggxKEnA99CyeYnYfcIBcJSeGeBqhBCiLQn4HjIZNa6/bjYAJw4dHuBqhBCiLQn4y5CSHEWdFkRDUS6lVY0DXY4QQrTSpYDPzc1l+fLlLFq0iOXLl5OXl9fhsTk5OUyePJk1a9b0Vo2Dmm90MqO0Sv7+UfZAlyKEEK10KeAfffRRVqxYwfvvv8+KFSt45JFH2j3O4/Hw6KOPct111/VqkYOZJTIBm1rH8dNFHMouH+hyhBCiRacBX1FRQVZWFunp6QCkp6eTlZVFZWVlm2NfeOEFrr76ahISEnq90MFKs8UDMCm4kb99eAqXW4ZMCiEGh04Dvri4mIiICDRNA0DTNMLDwykuLm513PHjx9m2bRvf/OY3+6TQwUoNjQPgphSV0uomNu3KH+CKhBCimaE3LuJyufjFL37Br3/965YfBD1hs/n1+NywMP8en3t5/DnjG0SsuYY5k1J4b2c+6QtGEx7i0+fvPHBtHjjS5pFB2tw7Og34qKgoSkpK8Hg8aJqGx+OhtLSUqKiolmPKysrIz8/nvvvuA6C2thZd16mvr2f16tVdLqaioh6vV+92I8LC/Ckrq+v2eb0mOIbGs9l85brl7M0q4bn1B3ng1rQ+fcsBb/MAkDaPDNLmrlNV5ZI3xp0GvM1mIzU1lYyMDJYtW0ZGRgapqamEhIS0HBMdHc2uXbta/v7MM8/Q2NjIww8/3O2ChyItNB7n4U2E+BlYMieBtz7N4UhuBRMTbQNdmhBiBOvSKJrHHnuMdevWsWjRItatW8fjjz8OwMqVK8nMzOzTAocC1RYHXg/eqrMsnhlHeLCVv24+hdvjHejShBAjWJf64JOTk1m/fn2b19euXdvu8f/xH/9xeVUNMRdG0ngr8jGGxrPiujH8Yf1hMj7P45b5SQNcnRBipJKZrL1ACQwHgxlPefOaNJOSQ5k9IZKMz8+QW1w7wNUJIUYqCfheoCgqqi0Wb8XFIZJfv34MgX4m1r6bhdMlY+OFEP1PAr6XaLY4PBX56Hpzv7uPxci9N6VyrrKRN7aeHuDqhBAjkQR8L1FD48FlR6+7uFzBhMQQFl4Rw4d7CzmW13bmrxBC9CUJ+F6i2ZpntF7oh7/g9muSiQjx4S8bj9Fodw9EaUKIEUoCvpeowaNAUVv1wwOYjRrfSk+lqs7B3z48OUDVCSFGIgn4XqIYTKhB0W3u4AGSowNZMjuB7UfOsf9k2QBUJ4QYiSTge5EaGtfmDv6Cm+cmEBfhxyubjlPb4OznyoQQI5EEfC/SbHHojdV4m9qOfTdoKivTx9Pk8PDcW5myrLAQos9JwPciNfTijNb2jArz49+XpHKysIYX3snq0cJqQgjRVRLwvUgLiQXAU95+wOtuJ2m5r/JI4h70vN38/f1MdF1CXgjRN3plPXjRTLH4ofjZ8Fa0fdAK4Nj7Fp6zRwm1BvANv2M48z8n9+9jiJ1xNYa4ySgmaz9XLIQYziTge5lma/9Bq6c0B1fmJozjrsY8/x48xac4ueV9ImqOYf/4T6AZME64DvOVy1EUZQAqF0IMNxLwvUwNjcd95iC6y4FiNAOge1zYt/4/FJ9gzLPuRFFUDNEpTP7qGJ554zANBcf4tzFlBBzeBLqOedZXJeSFEJdN+uB7mWqLA3S8lQUtrzn3v4O3qgjL/G+gmC5u5WfQVO7/Shp62Bj++1QaDfELcGW+j3Pf2wNQuRBiuJGA72Xa+ZE0nvPdNJ7yMzgPvodhzFwMcZPbHG82afzgjkmEBFj5ZVYyjrjZOPdvwHFwY7/WLYQYfiTge5niGwJmX7zl+eheN/atf0Gx+GOZ/bUOz/H3MfGfy6dgMRv45clxuGKm4dz9D5xHP+rHyoUQw40EfC9TFOX80sFncB7ciLciH/P8e1AsHW+MC2ALtPBfX50Kisavc6fgiZ6EY/v/4Tq5rZ8qF0IMNxLwfUC1xeGtKMC5/x0MSTMxJkzr0nkRIT78aPkU7C6d3xTOQI9Mxb71L7hy9vRxxUKI4UgCvg9otjjwulFMVsxz7+rWubHhfvzgzslUNnj5Q8lcCE3GvuXPeGvO9VG1QojhSgK+D2iRY8FgwjzvblRrQLfPT44O5MHbJ1FY7eaF6nmgGrF/9orMehVCdIsEfB9QA8Lw++afMCbN7PE1UuODuf8rEzlepvORdyaeomPSHy+E6BYJ+D6iqJf/qZ0yOpTvLJvIp42jyXGFUb31NfYdOo3H6+2FCoUQw50E/CA3LSWMNd+dg+OKFZhwUv3pa/zkzzt5b1sODpcsOSzEYOKtr8B95gC6s2mgSwFkqYIhwWjQmDl7KnbDEmYeeJezxhL+9JadAB8j938ljbGxQQNdohAjmu524jy8CeeBDPA4QTOgjZqIMWkGhvgpKGbfAamrSwGfm5vLqlWrqK6uJigoiDVr1pCQkNDqmDfffJOXX34ZVVXxer3ccccd3HPPPX1R84hlnroUd85ubtV3sPjbj/D0G0f47d8P8K308cxMjRjo8oQYcLruxXnwPbzlZzDP/waqxb/P39Odfxj756+h15ZgSJyOMWUB7sJM3Ln7sOcfBEVDixmPIWEahlETUPxD+22tKUXvwtCMe+65h9tuu41ly5axYcMG3nzzTV599dVWx9TX1+Pr64uiKNTX17N06VKef/55xo0b1+ViKirqe7QJRliYP2Vldd0+byhyFx2jKWMNQXNupSb5Rp558zCnCmu485rRLJoZO6wXKRtOX2dvXRmeivxO50gMpzZ3VU/brDubsHbrZH0AACAASURBVG95AfeZA6AoKP7h+Cx+CDUosg+qbP4aOj7/K+4zB1ADIzHPvQtDzMSL9ehevGW5uHL24s7di17XvB+z4mdDi0rBEDUOLXocin8Y4eEBPWqzqirYbB1Pouz0Dr6iooKsrCxeeuklANLT01m9ejWVlZWEhIS0HOfnd/FN7HY7LpdrWIfNQDFEp2IYO5/qnRvwiZ7Kf351CmszjvGPLdlUVDeyfJYNxVGLGprYKw96Rd9oDoaDaHc/3aOhtKI1b/U5mj54Gm9NCeY5X0cLTaDpg/+lYcNqrDc8iCEqpVfeR3c04C4+gafwCK4Tn4GiYpp5B6a0RSha6zhVFBUtPBktPBn9yjvxVhXhKT6Gp+g4noJM3Kc+bz7Oz4Zzxc+B4F6p8Ys6Dfji4mIiIiLQNA0ATdMIDw+nuLi4VcADfPTRR/zud78jPz+fH/3oR6Sk9M4nVbRmmbWcxsJD2D/+M1p4Mt8wlnFreDGWvGqazjT/BmSafBPmK+/s0vXc506B24khZkJflj1gXKd3oRgt7S72NhC8DVW48w8COp78Q6gp8we6pCHNnX+Qpo/+jKIZsC75LwzRqQD43PILmjb9nqb3nsSy4F6MY+d2+9q624Hn3Ck8Z7NwFx3DW54Hug6aCUPiNMwz70T1C+n0OoqioIWMQgsZBROuQ9d1vNVFeIqO460sRDFZwNnt8jrVqw9ZFy5cyMKFCykqKuKBBx5gwYIFJCUldfn8S/2q0ZmwsL7vaxs8/Klf9C1K3/o92GsxBkUQPnY8eXUmPjrexPTACkYf/hehU+djGTX2kldylhdy9l+/BVUj/sG1qCZLP7WhZ7r7dfa6HJz57CUUzUDEA39CNQ/8rllVxzfRoOuoVj/U4kzC5t10yeNHwr9tj72BhmOfYwqLQ9fHdKnNuq5Tvf1N6rb+HVNEAhF3/BhjYPjFA8L88dz7P5T887fYP1mLxV1N8IKub6jjri3n7N9+gqeuElQDllFjsMy7A2vCRCzRY1EMxp42t1l4AIy92IUddnlXa1enAR8VFUVJSQkejwdN0/B4PJSWlhIVFdXhOdHR0aSlpfHJJ590K+ClD77rwsbPpdE2AUXVWl5LAqqSynj53QP80LeIqlfWcHbmQ0wbH4PJqLW5hu520vj2b9F1wNFI8c7NmFKv7rc2dFdPvs6u7J3oTjs6ULT1LcxTl/ZNcV2ke7007N+MNmoCamAEjSe3UVpcgWIwtXv8SPi37crbj2Pbq+iN1QBovkGosZMwxF+BNmp8y+dG13X02hI8ZXl4ynLxnDuJtywXw+jZmBZ8k2qnGdr5XBkWfh/jtleo3rae+nMFWK76VpvulC/T3Q4a3/kVXnsj1hu+31yH0YwHqAfqq+yAvdc+Bz39Ol92H7zNZiM1NZWMjAyWLVtGRkYGqampbbpnTp8+TXJyMgCVlZXs2rWLG264odsFi677YrhfMC0ljDGxV3P0cyMTc17h0Na/86Mts5kzMYqrpkQTHXpxuJZj1+t4KwuwLn4Ix543cB39COO4q4bVsxNX9g4U3xDUkBichzdhmrCw1aYr/c1z9gh6fQXGWctRjFZcWR/jKcrCEDdlwGoaKN7GGhyfv4Y7ZzdqSCyWa7+N3liNVpxJQ/ZuXMc/BYMJw6gJzV0lZXngbGw+WTOi2uIwz70b4/hrL/lvVtEMmBfcixIQgXPPGzQ5GrFe/70Of6jquo79k7/gLc/Huvj7Q/pr06Uumscee4xVq1bx3HPPERAQwJo1awBYuXIlDz74IGlpabz++uts374dg8GAruvcddddzJs3r0+LF+0L8DEx+7prsG8r4KqsLbiCJ/Gv/YVs3lvA9HHh3HXDWKylmc2BnrYIQ9xkvA1VOD57GU/JKQyRl+7WGWi67kVROn+A7G2qxVOQiWnSYgxJM2l86zGcRzZjvmJZP1TZPtexrSgWfwzxVwA6GC248w4O6RDpLl3XcZ/6HPuOv4LLgWn6rZim3ISiNsdR2OwbKD1XiafoOO4zB3AXZKKYfTEmz0QNS0QLS0QNjm45visURcE8NR3F4ofjs1doev8PWG/4fsu2ml/kPPAO7pzdmGbeOeS/Ll36DCUnJ7N+/fo2r69du7blzz/96U97ryrRK8xX3om7IJMb3B9z3X2/YMuRCt7bkce5/AJ+6PcOxtAEzDNvB8A4ejaOXa/jOvrRoA54V/YO7NvXdWlkhPv0btC9GMbMQQuJwRA/9fxd/HUDMvHE21iN+8wBTJMWt3QRGGIm4s4/2OUfWkOdt7Ea+yf/D0/hEdSI0VgW3IsWHN3mOEUzYohNwxCb1qvvb0q9GkUzYt/6/2j611NYFz+EYrr4XMaVuxfn3rcwjJmDafKNvfreA2H4/4sawRSjBcvV30KvK8d05G2WzUvkkXuu4GvWT3G5XGzgOhqcF441Yxw7H3fuXrzn+0IHm/qs7di3vACOBpx7/9np8a7sz1FtsWghMQCYpt0CziacmR/0dant13PiM9C9GMctaHnNED8VvbEab1negNTUn7y1pTRueALPuVOY59yFz80/bTfc+5px7FwsC7+Lp+Q0je89iW6vB5q32bRveQE1PAnL/G8Oi65KCfhhzhCVgjHtBlxZH+MuPEpYwcfE6MXkjFrKlmwXv/jLLg5llwNgmnAteD24jm/t9To85Wfw1lf0+HxX7l5K3/4DWsQYTDNuw1N8AnfxiQ6P99acw1uag3H0nJbXtNB4DAnTcGZ+gO5o6HEtPaHrXlzHP0WLTkUNvDjxxhA3GRS1eXLOMOapLKRxwxPozkZ80h/GNPG6Af2NxZg0E+sN38NbUUBjxho8lYU0vf80itkX6w0Pdtg/P9RIwI8A5hm3oQZGYv/4TzgPvINh7FyuTF/Gz++Zjr/VyNNvHOb5t49wptEHLWYirmOfoHt7byEz16nPaXzrcRr/+RieioJun+/OO4D9w+cxR4/BuvghTGk3oFgDcO5/5xLvuQNQMIye1ep107RbwNWE8/CmbtdxOTxns9DryjCOu6rV64rFDy1yDO4zB3v9PR2719O0+Y/ofbj6qLf6HN7zMzQ74ik9TeO7vwZFwWfpT9HCuz6yri8Z4qdiXfwDvDUlNL75C/SmOqw3fB/VZ/is7SQBPwIoBhOWa1aiO+pRAiKwzL0bgPhIf37xjRksnZNAZk4Fq1/Zyzul8egNVbhy9/fKezuPfIh9ywtokWNAM9CY8T94yvO6fL47/xBNH/4RNTSeqK/+DMVkRTGYMU26Ec/Zo3hKT7c5R9d1XNk70Ealovq2nh2o2WIxJM3AeWRzy6/m/cF17BMUsx+GxLZLExjip+KtLOg0KLvDW1uK89C/cOfuxXloY69d94tceftoePPnNPztxzRu+gPuwiNtNqVxn82iMeNJFJMPPjf/rHmizyBiiJmI9aYfoQSEY7lmJVpYwkCX1Ksk4EcILTwZa/oqfJb8GMV4cTKT0aDylQVJPPXAXL66cAwHmqKo9Phy/MO3eH93Po12d4/eT9d1HPs34Ph8HYaEK7De+CN8lv4ExWhp/pW4nWD+MnfhEZo2P4MaEoPPTT9CtVx8MGocfw2K2Q9HO3fx3tLT6LWlGEfPbve6pituAZej3+7ivY01uPMOYEiZh6K1nRxjiG8eqdGbd/HOgxmgqmixaTj3voWnLLfXrg3gPPYJ9s1/RA2JwzRlCd7S0zRt/C2N63+K8+iH6M4mXHn7aPrX71D9w/BZ9jPUgL6YynP5DFEp+C1fgzFpxkCX0usk4EcQQ1RKh9OqrWYDN8yI5dffnotn9AISlSI++WQv//X8djZsy6XJ0fWg13Uvjp1/Pz8aYS6W6x5AMZhQA8LxufmnKBZ/Gt/7De5zJ9s939tYjeNgBk3vP40aFIXPTf/VZtSLYrRgnLQIT/6hNr8RuE7tAM2IIXF6u9fXQkZhSJ6J88hmvE21XW5XT7lObgPdg+lL3TMXqIGRqEHRlwx4XdfxlGSje1ydvp+3vgLXye0YU67Ceu13UHwCafr4z+guR4/b8MU6HPs24PjsZbSYNHzSH8Y883Z8VzyF5eqVYLTg2L6O+td+iH3zs6ihcfgsXTWsuj2GEu2xxx57bKCLuKCpyUlPth319TXT2NgHCzkMYn3VZkVRCIyOw3VkM9PHhVFmHc2WA2fZevAsAHHh/hi0ju8LdK8H+6cv4T62BePE67HMv6fVhCzF5IMhcTqe3H24sj5CixiN6h+G7nHjPnMAx67XcWz7Pzxnj6JFj8O66ActS75+uc2aLQ5n1hb0+kqMyVeef383jk/+giF2EsYxc+iIGjwK15HN4PWgxUy8rBETnnOnaNr4W1zHP0N3NqH6BrX8QNJ1L/atf0ELibnksDu9vgJ39i5Made3usu/0GbXkQ+wf/gcemMNhoSpl6zHsftNvOV5WK+7H9UnEDU0DlfmB+iOupbfFnpC93pxfL4O1+F/YRg7F+vC77Q8jFRUDc0Wh3HcVRhi05o/D0FRzROKzN2bWCbfz12nKAo+Ph0/EJaAH6L6ss2K0Yy3uhitYB9zb1/BlJRISqua2HLgLJ8dKkJTVeIi/NC+tFqlt7oY+9YX8eTsxjTtFswz72h3pIRismJImoHnzEFcRz9Er6/A8dnLuE9+1jzxZcJCLAvuxTRpEYrh4kSUL7dZ0YzgduI6tgVD4jRUayCe/MO4TnyKeeYdl1wmVrX6460rw33sE9xn9qMYzKhB0d1agVPXdVxHNmP/+M9gNKMYLbhPbcN1ZDPugkxwOdDrynAf/xTzjNvQbLEdf84NJlwntqLa4lqGdV5oc03WThxbX0SxBuI5d7J5ok9g+21rHme+FuPYuRjHNC+upfqHobscuI5+iGaLRw3qeJmRDtvqdmL/+E+4T21vXshu7t3tzqRWFAXVLwRj4nSMSTM6XRKgPfL93HUS8MNUX7dZ9QnClfUxip8NW+I4Zk2IZEJCCGfLG9hy4CzbM8/h8epEBlvRijKxb1+HY+ff0OvKMM9egXnKkktPHzdaMCTPxFNwGM/ZoxhiJmKetRzz3LsxxExEsbRdX6O9NjffxX+M3liDMWkGjr3/RG+swTz/nk6H4RliJ6H4h+ItOYXr+FZcJz5F97jQgkd1OkxOdzZh/2QtrsxNGOKn4HPjDzFNvB7jmDkoPoF4K/Jxn9yOO3cfmH2xLLi33UBs+Xz4BOPK+hh0D8YvdC0Zagqo2vAbVFs8vrf8Anf+Ydw5uzGOnddujY69b+EtOY31uvtbdWtpUSm48w/hzt6BYcycVs9hOuMpP4P9oz/hKTyCefbXMF+xrE/HiMv3c9d1FvCyZZ9olxqejGqLx3kgA29tGaqfjXi/EH54vY3sqlDe351P5c4NVB48QYjagNcShGX6rRjHXYXqE9i197D443PLL8Dt7PHMUsXsi2nCdTgPvoenNAf3mQMYUxZ0aRq7ohkxjbsKY8oCPIVHcGa+j3PPmzj3v4txzGy0UePRwpNQ/FrvwOOpPIt98zN4a0swzbwT0+QbWz6uBoRjnpKOeUo6nuoi3Dl7ULvwA0NRVbS4Kbjz9qF73SiqAW9dOefe+TWKxR/rou+jmH2xXPMtGt9ajf3zdViv/U6ra3ibapt/mxk9CzUgvPX1NSOWa79D4z8fxb71xeYZnJ2EtKc8D+e+Dc1j9E1WLNd+B+OXhp2KwU0CXrRLURTMs5Zj/+yV833VFx+yRgP/BuADpeY4XqmawcHKGMaZbFwX4GJSso7axTs8RTNCOyNLusOYdgPOIx/Q9P7T4HFhHNP+6JkOa1CUlmnxnspCXJnv48re0TLhS7EGoIUno0YkoxjMOHavRzFasC75ccva4+3RgqLRurHujSFhCu6Tn+EpPokWlkDTpj+A24n15p+1/NDUQhMwXXEzzn1v4UqY1mrkhyvzA3C7ME1Nb7+e4GjMs5bj2L4OV+YmjOOuBqOlTdB7yvJw7HsbT/5BMPlgmvaV5olJA7SvqOi5Lm3Z119kueCu688267oXvakOvaESb30Fen0lurMJQ+I0tJAY6hqdbD1YxMf7C6mud5IY5c/XFo5ldEzX7uS76lJttu/8O67Dm1ACwvFdvuayuxB0rxtv5Vk8pafxlJzGU3oaveYcAFrkWCwLv9tmjP3l0l0O6l/9HsaU+XhrSvAUHSfyaz+jwa/1xCDd66bx7V+i11fgc8cTqNYAdHs99X/7Twyxk7Bed3/H76HrNG36PZ6Cw80vaEYUawCKNRDFGgAeJ56zWWD2xZS2qDnY+3n1Tfl+7rrOlguWgB+iBmOb3R4vO4+W8Oanp6mpd3Ll+AjuuDqZkIDe2UTkUm32NlbT8PoqTFPSMXdwB3u5dHs93toS1ND4bq1k2B2Nm36PJ/8QAOYF/8ao+entttlTeZbGfz6KIW4yluu/h3P/Bpz73sbnttWXfJgLzT9I3Gf2ozdU422qQW+qbf6vsQbd7cQ4di6mide3WoSrPw3Gf9t9bcDWgxeiqwyayrxJUUwfF8bGnfm8vzufAyfLWHxlHDfOisfczqYjvUX1CcJvxVNg7LtQUix+aO08/O1NhoQr8OQfwjQlvcNx89A8lt8841Ycu/6B69gWnJkfYIif2mm4w/mF5TqYBCaGFwl40essJgO3LkhiweQo3vjkNO9sz+Ozw8XMmRhJaKCF0EAroYEWQgIsGA29N9duOPQRG1PmowZGNi/t0NmxaYtbdkMCMA3gOvdicJKAF30mNNDKd5ZN5Norqln/STb/2pmP90s9goF+JqJCfEiMCiAxKoCk6ACC/c3DYqnWnlAUtdN17luOVVWsV3+LhjcfQYsaN+zWURGXTwJe9LmxsUH87O7peLxequuclNc0UV5jp6LGTnmNnbPlDWzeW4Db0xz+gb4mEqMCGBsbxFVTorGa5Z9pR9TASHxvfwLFOvw35hbdJ985ot9oqoot0IIt0MKX71Fdbi+FZfXkFNWSU1RLbnEtB7PL2bTrDMvmJTJ/cv9vDDFUDNZFvMTAk4AXg4LRoLZ00yw8v6Lu6aIa1n+czf99cJIP9hby7zdPIDnCb8R23wjRXbKapBi0kqMDefjrV/DgbZNQFfjVy3v49Wv7OVVY3WbdcSFEW3IHLwY1RVGYMiaUtOQQDuVW8X8bj/HrdfvxMRuIi/AjNtz//P/9iA71veRKl0KMNBLwYkjQVJVFsxKYEBvErmMl5J2rI7+kjq0Hz+J0N29JZ9AUpqeEs3RuAlG2oT9kUojLJQEvhhSzSWPB5GgWTG7+u9erU1LVSH5JPdmFNWzLLGZXVgkzx0ewdE4C0aES9GLkkoAXQ5qqKkTZfImy+XLl+AiWzkvg/d35fLzvLLuzSpiRGs7SuYmMkqAXI5AEvBhWAnxM3HH1aBbPjOP93QV8tL+QPcdKmZEazrJ5idJ1I0aULgV8bm4uq1atorq6mqCgINasWUNCQkKrY5599lk2btyIqqoYjUYeeugh5s+f3xc1C9Epfx8Tt1+dzKKZsXywp4AP9xay53gpcyZGcvPcRMKCBmYhLSH6U5dWk7znnnu47bbbWLZsGRs2bODNN9/k1VdfbXXMZ599xvTp07FarRw/fpy77rqLbdu2YbF0fSVBWU2y66TN3VPb4GTjzjN8vP8suq4zf3I06bPje22ly74iX+eRYcBWk6yoqCArK4uXXnoJgPT0dFavXk1lZSUhISEtx33xbj0lJQVd16muriYysuN9MYXoLwG+Jr66cAyLZsaRsSOPTw8Wse1wMVNG2zCbNDRVRVMVNFVBVRV8LQaunBBJuNzpiyGs04AvLi4mIiICTWte6lXTNMLDwykuLm4V8F/09ttvExcXJ+EuBp1gfzN335DCjTPjeOfzPE7kV+Hx6ng8evP/vToerxeXy8vbn+WSlmzj2itimJgU0uVdqoQYLHr9Ievu3bt5+umnefHFF7t97qV+1ehMWNjIW2xJ2nx510kdE97hx8urm9i0M4/3d57hD+sPERXqy01zErhuRhx+l9jkuC/I13lk6Is2d9oHX1FRwaJFi9i1axeapuHxeLjyyiv54IMP2tzBHzhwgB/84Ac899xzTJgwodvFSB9810mb+4fb42XviVI+3neW7LM1mAwq11wxiptmxePfD0EvX+eRYcD64G02G6mpqWRkZLBs2TIyMjJITU1tE+6HDx/moYce4n//9397FO5CDEYGTWXW+EhmjY8kv6SO93fn88HuAj45WMQN02NZNDMOH4uMNhaDU5dG0Zw+fZpVq1ZRW1tLQEAAa9asISkpiZUrV/Lggw+SlpbGbbfdxtmzZ4mIiGg578knnyQlpWubF4DcwXeHtHngnC1vYMNnOew9UYavxcDiK+O4blosZlPvb0k4WNrcn6TNXSebbg9T0uaBd+ZcHW99lsPh0xUE+BgZFeaHw+Vp/s/pafmz1WQgOtSXaJsv0aE+zX8O9e1SF89ga3N/kDZ3nWy6LUQfiY/05wd3TCa7sIaNO89Qb3dhNRsI8jNjNqqYTQbMRpWGJjdFFQ1sP1KM3elpOT/Q18ToUYGMjmn+Lz7CX1bDFL1KAl6IyzQ6JpAHb5/U6XG6rlNV56CovIGz5Q3kl9RxqrCGfSfLgPObnkT6kxwTSGyYHzFhfgQFy9IKouck4IXoJ4qiEBJgISTAwsQkW8vr1fUOsgtryD7b/N8HuwvwnO+q1FSFyBAfRoX5EhPmx5QxocSE9Xw4sRhZJOCFGGBBfmamjwtn+rjmcfluj5dzlY0UltVT1eDi1Jkqcopq2X2slH9+mkNCpD9z06K4cnwEflbjAFcvBjMJeCEGGYOmEnO+i+aLD99qG53sOlrCtsxiXtt8ktc/PsXUMWHMTYtiQmIwmir996I1CXghhogAHxPXz4jl+hmx5JfUse1wMTuzSthzvBRfi4HxCSGkJdmYmBRCkJ95oMsVg4AEvBBDUFyEPyuu9+eOa0Zz+HQ5h7IryMytYM/x0uaPh/uRlmxj9KhAbIEWbAEWrGb5dh9p5CsuxBBmNKhMSwlnWko4uq5TUFpPZk4FmTmV/GtnPt4vTHOxmjVCAprDPjzYysREG+PigjAZe3+ClhgcJOCFGCYURSEuwp+4CH+WzE6g0d48/r6y1k5FrZ3KGgeVdc1/Pn6mig/3FmIyqIyLD2Zyso1JyaHYAgf3+viieyTghRimfCwGRo8KhFGBbT7mcns4nl/N4ewKDp0u5/DpCuAko8J8mZ4Szoxx4bJh+TAgAS/ECGQ0aKQl2UhLsrFCH8O5ykYOZVdw8FQZ72zLZcO2XGLCfJmRGsHM1HAign0GumTRAxLwQoxwiqIQZfMlyubL4ivjqKpzsPdEKXuOlfLWpzm89WkO8RH+pMQFERvuR1yEP1E2H1lWYQiQgBdCtBLsb+b66bFcPz2Wylo7e46Xsu9kGVsOnMXl9gLNM2xHhfoSG+HHuLhgpo4Jk2WTByH5igghOhQSYGHRzDgWzYzD4/VSUtlEfmkdBaX1FJTUc/h0Bdszz2HQjjMx0cbM1HAmjw6VIZmDhHwVhBBdoqlqy1LHs8Y3v6brOjnFtew5Vsqe46UczC7HaFCZlGxjxrhw0pJsEvYDSD7zQogeUxSF5OhAkqMDufPa0WQX1jSH/YlS9p0ow6CpTEwMYVpKGJNHh7a7dk5do5PiikYqauz4+xpJQQG3F6NB+vgvlwS8EKJXqIrC2NggxsYG8bXrxnCqsJp9J8vYf7KMg9nlqIrCuPggRo8KpLLWwbnKRoorGmiwu9tcSwGC/M3YAi2EBVqIjwwgJbb5Ia+qKv3fuCFKAl4I0etUVSElLpiUuGC+tnAMeefq2HeijH0nSsnKqyLA10RUiA8zxoUTafMlyuZDaKCF2gYnTh1yCqopr26ivMbO8fxqdhwtAcDHbGj5IZISF0R8hL8E/iVIwAsh+pSiKCRGBZAYFcBtVyXhdHsxd7A8QpTNl7Awf9Lig1u9Xllr50R+NScKqjiRX83B7HIArGYD4+KCSI0PJjUhhGibD4oigX+BBLwQot8oitJhuF9KSICF2RMjmT0xEoCqOgcnC6o5dqaKrLxKDpxqDvxAPxPj44OZmGTjijFhfbIR+lAiAS+EGHKC/c1cOT6CK8dHAFBW3dQS9kdyK9lxtASzSWPGuHDmpUUxJiawzZ29ruuUVTdxoqCa/HP1xEX6MXl0KAFd2Ax9qJCAF0IMeWFBVsKCrCyYHI1X18kurGFbZjF7jpey7XAxYUEW5k6MYkJiCAWl9ZwoqOZkQTVVdQ6geZMV934vigJjYoK4YkwoU8aGER5kHeCWXR5F17+wnugAq6iox+vtfjlf3PVmpJA2jwzS5svjcHrYd7KU7ZnnOHamquX1QD8TKbFBpMQFMzY2iCibDwUl9Rw4Vcb+k+UUltUDEBPmx8TEEMbEBjImJqjPtkjsaZtVVcFm63iPXgn4IUraPDJIm3tPeU0TOUW1xEf6Ex5kveTD2NLqJg6eLGP/qXJyimpwe5pzKTrUlzExgYyNCSIu0p/QAEuv9PP3VcBLF40QYkQIDbQSGti1LpfwICs3zIzjhplxuNwecovrOFlQzcnCanYfK2HrwaKWY30thpZds2yBFkIDLIQGWQkNtBAWZB3Qmbxdeufc3FxWrVpFdXU1QUFBrFmzhoSEhFbHbNu2jd/97necPHmSu+++m4cffrgv6hVCiH5lNGgtY+8BvF6dwrJ6iioaqKixU1HroKLGTmlVE1lnqnA4Pa3O97UYCAuyYgu04GM2YDEZsJo1LCYDFrOGv9XE9SF9s/Z+lwL+0UcfZcWKFSxbtowNGzbwyCOP8Oqrr7Y6JjY2lieeeIJNmzbhdDr7pFghhBhoqnpx56wv03WdBrubsvOTtMqrmyirbqKsxk5ReQNNDjd2pwf7l34IRIT5ERPS+w90Ow34iooKsrKyeOmllwBIT09n9erVVFZWEhIS0nJcfHw8AB9++KEEvBBiRFIUBT+rWDCm2wAABUhJREFUET+rkcSogA6P8+o6jvNB7/XqjBsd1ifPHToN+OLiYiIiItC05gcJmqYRHh5OcXFxq4AXQgjRNer/b+9+QqJawziOf50u2g3Kyf7glIsoKIZaBAnthE7BuLBxdsaQm8wWUosiSCnIrKApiIIGWhRtjNwEEhZRMAQVFCMWNBQVUhKMFaZh/4OZt0V47r10y+5tTuO85/dZOWcQnx/P8eH4zvG8ZWX8WfGH5+vzU+pD1h99GjyZefO+/XPJdsrsD8rsD15knnTAh0IhXrx4QS6XY9q0aeRyOV6+fEkoFCp4MbpN8ucpsz8osz94dZvkpA9cnjNnDuFwmL6+PgD6+voIh8NanhERmeJ+6on6nZ2ddHd3E4lE6O7uZt++fQC0trZy7949APr7+6mrq+PMmTP09PRQV1fH9evXvatcRER+SP/JWqKU2R+U2R+KtkQjIiKlaUrdRfMrO7P4cVcXZfYHZfaH/5N5su+ZUks0IiJSOFqiERGxlAa8iIilNOBFRCylAS8iYikNeBERS2nAi4hYSgNeRMRSGvAiIpbSgBcRsVTJD/gnT57Q1NREJBKhqamJp0+fFrukgkokEjiOw7Jly3j06JF73ObcY2NjtLa2EolEWL9+PVu3bmV0dBSAu3fvEo1GiUQibNq0iVevXhW52sJpa2sjGo0Si8WIx+M8ePAAsLvXACdOnPjH+W1zjwEcx6G+vp7GxkYaGxvdp+56ktuUuObmZtPb22uMMaa3t9c0NzcXuaLCSqfTJpvNmjVr1piHDx+6x23OPTY2Zm7duuW+PnTokOno6DC5XM6sW7fOpNNpY4wxyWTStLe3F6vMghsfH3e/vnr1qonFYsYYu3udyWRMS0uLe37b3mNjzDe/y8YYz3KX9BX8xIbgDQ0NwNcNwe/fv+9e7dmgtrb2m92zbM8dDAZZvXq1+3rlypVks1kymQwVFRXU1tYCsGHDBi5fvlysMgtu5sy/tmx7+/YtZWVlVvf68+fPdHV10dnZ6R6zvcff41XuKfU0yf/KrxuC+yl3Pp/n3LlzOI7D8PAwCxYscN+rqqoin8/z+vVrgsFgEassnN27d3Pz5k2MMZw6dcrqXh8/fpxoNEpNTY17zA89Bti5cyfGGFatWsWOHTs8y13SV/Biv/379zNjxgw2btxY7FJ+i4MHD3Lt2jW2b9/O4cOHi12OZ+7cuUMmkyEejxe7lN/u7NmzXLhwgfPnz2OMoaury7OfVdID/u8bggOebgg+lfgldyKRYGhoiGPHjhEIBAiFQmSzWff90dFRAoGAVVd2E2KxGLdv36a6utrKXqfTaQYHB1m7di2O4/D8+XNaWloYGhqyvscTvSsvLycejzMwMODZuV3SA96vG4L7IffRo0fJZDIkk0nKy8sBWLFiBR8/fqS/vx+Anp4e6uvri1lmwbx7947h4WH3dSqVorKy0tpeb9myhRs3bpBKpUilUlRXV3P69Gk2b95sbY8B3r9/z5s3X7fmM8Zw6dIlwuGwZ+d2yW/4MTg4SHt7O+Pj48yaNYtEIsHixYuLXVbBHDhwgCtXrjAyMsLs2bMJBoNcvHjR6tyPHz+moaGBRYsWMX36dABqampIJpMMDAywd+9ePn36xMKFCzly5Ahz584tcsW/bmRkhLa2Nj58+EAgEKCyspJdu3axfPlyq3s9wXEcTp48ydKlS63tMcCzZ8/Ytm0buVyOfD7PkiVL2LNnD/Pnz/ckd8kPeBER+XclvUQjIiLfpwEvImIpDXgREUtpwIuIWEoDXkTEUhrwIiKW0oAXEbGUBryIiKW+AMC4hoNM7mZoAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6YFEDipFWGFc",
        "outputId": "64c4619f-bf9d-4b1f-f635-42714bf66732",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        }
      },
      "source": [
        "x = np.linspace(0, N_EPOCHS,N_EPOCHS)\n",
        "\n",
        "plt.plot(x,tacc)\n",
        "plt.plot(x,vacc)\n",
        "plt.title(\"Accuracy\")\n",
        "plt.legend([\"Train acc\", \"Valid acc\"])"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f9886021c88>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAELCAYAAADTK53JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9b3/8dc5Z7ZMMtl3FkHWyL6jiK2oBTQsWiv9oV77s1Kt2vbW9udF7c+t2hZbW9uK19bWqg/6u70uV8CAa2kVXBBkN7JDwjLZ99nP8vtjYDBNgAAJSWY+z8djHsnMnJn5fGeS93zne77zPYplWRZCCCHijtrdBQghhOgaEvBCCBGnJOCFECJOScALIUSckoAXQog4JQEvhBBxSgJeCCHilAS86PVuvvlmJk2aRDgc7u5ShOhRJOBFr3b48GE2btyIoij8/e9/P2+Pq+v6eXssIc6WBLzo1ZYvX86YMWO49tprWb58eexyr9fL3XffzdSpU5kyZQqPPvpo7LqXX36Z2bNnM27cOK6++mo+//xzAIYNG0ZZWVlsu8WLF/Ob3/wGgPXr13PZZZfxxz/+kWnTpnHffffR2NjI7bffztSpU5k0aRK33347FRUVsds3NDRw3333cemllzJp0iTuvPNOAIqLi1mzZk1su0gkwpQpUygtLe2aJ0kkLFt3FyDEuVixYgXf+ta3GDNmDAsWLKCmpoaMjIxY8K5ZswZN09i+fTsAb775Jr///e9ZunQpo0aNory8HJutY/8GNTU1NDY28o9//APTNAkEAlx33XU89dRTGIbB/fffz6OPPsozzzwDwL333ovb7WbVqlW43W42b94MwLx581i5ciUzZswA4P333yc3N5eLLrqoC54hkcgk4EWvtXHjRo4ePcrs2bPJzMykX79+lJSUMGbMGKqqqrj33ntj4T1x4kQAXn31VW677TZGjx4NwAUXXNDhx1NVle9///s4HA4AXC4XM2fOjF3/3e9+l3/7t38DoKqqig8++ID169eTlpYGwOTJkwGYO3cuzzzzDC0tLaSkpLBy5Urmzp17js+GEG1JwItea/ny5UybNo3MzEwgOvTx+uuvk5ubS2FhYbs9c6/XS//+/c/q8TIyMnA6nbHzgUCAn//856xdu5bGxkYAfD4fhmFQUVFBWlpaLNy/LC8vj/Hjx/P2229z1VVX8cEHH/DAAw+cVU1CnIoEvOiVgsEgb775JqZpMm3aNADC4TBNTU1kZ2fj9XrRdb1NyBcUFFBeXt7ufSYlJREIBGLnq6urycvLi51XFKXV9s8//zwHDhzg5ZdfJicnhy+++IL58+djWRb5+fk0NjbS1NREampqm8e69tpreeWVVzAMg7Fjx7Z6HCE6i+xkFb3Se++9h6ZprFq1iuXLl7N8+XJWr17NxIkTee+998jJyeHJJ5/E7/cTCoX47LPPALj++ut5/vnn2bFjB5ZlUVZWxpEjRwAYPnw4JSUlGIbBBx98wIYNG05Zg8/nw+l0kpqaSkNDA08//XTsutzcXC677DIeeeQRGhsbiUQire7vyiuvpLS0lJdeeon58+d3wTMkhAS86KVef/11rrvuOgoLC8nJyYmdbrzxRlatWsWzzz5LWVkZl19+OZdddhlvvvkmALNnz+aOO+7gRz/6EePHj+euu+6KDa888MAD/OMf/2DixIm88cYbXHnllaes4ZZbbiEUCjF16lQWLFjA9OnTW13/xBNPYLPZmD17Npdccgkvvvhi7DqXy8XXvvY1Dh8+zFVXXdXJz44QUYoc8EOI7vH0009z8OBBfvWrX3V3KSJOSQ9eiG7Q0NDAa6+9xoIFC7q7FBHHJOCFOM9efvllvvrVrzJ9+nQmTZrU3eWIOCZDNEIIEaekBy+EEHFKAl4IIeKUBLwQQsSpHvVN1vp6H6Z55rsEsrJSqK1t6YKKei5pc2KQNieGs22zqipkZCSf9PoeFfCmaZ1VwB+/baKRNicGaXNi6Io2yxCNEELEKQl4IYSIUz1qiKY9lmVRX19NOBwE2v8IU1WlYprm+S2sm517mxUcDhcZGTltVkkUQsSHHh/wLS2NKIpCXl5fFKX9Dxw2m4quJ1bAn2ubLcukoaGGlpZGPJ70TqxMCNFT9PghmkCgBY8n/aThLs6Ooqh4PBkEAok1W0GIRNLjU9M0DTStx3/Q6JU0zYZpGt1dhhAJrStXi+kVySljxF1DnlchzlxEN6iqD1DfHKLZH6E5EKHZH6YlEKHZHyEU1lFUBU1RUNXoSVOj/2vBsIE/pBP40skwLR6/YxpZyfZOr7VXBHxPsWjRLUQiEXQ9wqFD5QwcOAiAoUOHcf/9D3XoPpYvf5VQKMSCBTd2ZalCiH9hWhaNLWE0TcFhU3HYNFS1dScnohu0BHT8wQi+oI4vEKGmMUhlvZ/KOj8VdQHqmoJtpnuoikJKkg2P24HLoWFa0XnthmlhWSd+upw23E4bqRlukpwaSU4bqW4HffM8BFqCnd5mCfgz8Nxz0SPyeL1Hue22m3nhhf/XZpv2jgP6ZfPnX99l9QkhTvAHI+w/2sTeI43sP9rE/qNN+EN6q200VcFhV7FpKsGwQeQkExeSnBp5GW6G9E0jL7OAvMwkslJdeNwOUpLsuF021HP4RJySZJeA76muv34OV1zxNTZt2sCFFw7mO9+5k4cffgCfz0c4HOaSS6Zx550/AODPf/4DgUCAu+/+d1avfoN3330LjyeV/fv34fGk8NhjT5CVld3mMZ5++im2bNlEJBIhPT2dn/zkIXJy8gH48MO1PP/8H9F1HVVVeOCBRxg8eAg7dmxj6dLf4vf7Abjrrh8wefLU8/fECNEFdMOkrLKZXeUN1DUF0Q2TiG4SMSx03SRimNQ1BfHWRv/uFaBPTjKTinLpl5uCZUFYN4hETMK6SVg30HUTl8OG22Uj2WUj+VhoJ7vsx4Lc3iuHNHtVwH+43cu6bd42lysKnOt+iktHFzBtVMFZ397n8/Hccy8BEAqFWLLkN7jdbnRd55577uaTTz5i6tRL2tzuiy9KefHF/yIvL58lSx7j1Vf/m9tvv6vNdjfd9C3uvvvfAXjjjeUsXfo7Hn74Z5SXl7FkyWMsXfoc/fr1JxwOo+sRmpoauf/+/8Pjjz/BqFFjMAwDn8931u0T4lwZpkl5ZQsVdX7Skx1kprnI9Dix27TT3q6sooVd5fV8UV7PnsONhMLRyQHJLhs2m4pdU7F/6WduehJTL8pjUJ80BhakkuTsVVHXaRKz1V1g1qxrYr+bpskzz/yW7du3ARa1tbXs2bO73YAfPXoMeXnRnviIESPZsGF9u/f/yScf8j//8wqBgB/DMGK9iQ0b1jN16iX069cfAIfDgcPh4KOP1jFgwEBGjRoDgKZppKamdmaThTiliG5ywNvErkMN7D7UwN4jJ4L5y1LddjJTXaSnONENE920aPaFCYZ1AiGDQFiPdeAKstxcMiKf4RdkMLRfOmnJjvPcqt6lVwX8tFHt97J7whed3O6k2O///d9/pbm5iT/+8QWcTidLljxOOBxq93YOx4k/UFXVMIy2/wAVFV5+//tf89xzL1FY2Ift27fyyCM/6fxGCNFBlmXhD+k0+yPUN4eoawpS1xw68XtTiIo6P7oR/b/sk5PMJSPzGdYvnT7ZyTT5wtQ1h6g9tm1dU5CaxiAOu0pqipNkpw2XUyPJEf3ZNyeFYf0zJNDPUK8K+N6iubmZrKxsnE4n1dVVrFv3PvPnf/2s78/n82Gz2cnKysI0TZYvfy123eTJU3nxxT9z6FB5qyGakSNHcfDgAXbs2MbIkaNjQzTSixcdEQjpx2aOBKis91NVH6DRF6bZF6bJH6bZH8FoZ/XDlCQ7mR4n2WkuLhqQwbB+6Qzpl05KUuspgH1yTv7YOTkeqqubO7tJCUkCvgt84xvf5P/+3//g5ptvICcnjwkTzu3AyoMGDebyy6/kpptuIC0tnYsvnsbWrZsB6NevP/fe+wAPPXQfhmGiaSoPPPAIgwYN5vHHn+D3v/8NwWAARVG5664fMGnSlM5ooogjwbDO3sON7CxvYO/hBirq/DT5I622yfA4SU9xkuFx0j/fQ6rbQarbjsftID3FQWaqiwyPE4f91OPp4vzqUQfdrq1tabMmckVFGfn5F5zydj1hiOZ866w2d+T57SkSsWfXWW3WDZNwxCAYjp7qmoLsOtTAzrJ6DlY0Y5gWmqowIN9DQXYyeRlJ5GW4yc90k5ORhPM8Bre8zh2nqgpZWSknvV568EL0UqZlsXFnFSUflVHXFERRot9OVpTo1EAUBdO0CIaN2Fj4l2mqwsCCVGZN6c/w/hkM7pOG0yE98HgiAS9EL2NZFlv31vL62v0cqmqhMDuZi0fmgwUWFpYVXVjbsixUVcHl0HDZNZwOGy6HhtOu4XHbubAwFZdDIiCeyasrRC9hWRalZfW8/sF+9h9tIjc9iUVzLmJKUV6br9wLARLwQpxXumHiC+q0BCL4AhGafGGa/WGa/JHo7BRfGF9Qx7IsFEXB4dAIhw0UBXwBnbLKZjJTnXxr9nAuGZmPTevxC8KKbiQBL8Q5siyLIzU+ahqCx6YQhmnyRY4Fd5gWfwRfMEJLUG/3iz7HJbtspCY7SHbZo9/OxoKIQkQ3sbBw2FVuvGool40pxG6TYBenJwEvxFkIhnW+OFjP1n21bN9fS31z6y+yOR0aqW47qW4HGR4n/XJTSE6yx9Y5SXbZSUmy43HbSU2OLljVXm88EWeUiM4jAS/ESViWRSCk0+gLHxtKiS4d+/nBOnaV16MbFi6HxoiBmYy+MIu+uSmkuh143HaZDy56BAn4M/CjH32f6dMva7Xkr2VZ3HDDfO6//0HGjZvQ7u0ef/xhhg8v4utfX3DK9eBXr36Djz5ay2OPPdFlbRDHhlSqfWzdV8Phah/hiBE96SbhSHR1wWDYoNkfRjfafk0kP9PNjPF9GTMoiyH90mUcXPRYEvBn4Jpr5vK3vy1rFfCbN3+GqiqMHTu+Q/ch68F3j4husqu8ni17a9i6t5bapuja29lpLlwOG067isOu4XbasNuj0wo9ydEhltTkYye3g7RjvwvRG/SqgI/s/pDIrg/aXK4oyjkf19A+7DLsQ6edcpvp07/Ck0/+nIMHDzBgwEAAVq1aydVXz2H//n08+eQvCAYDhMNh5s69lhtuWNjmPr68HnwkEuE3v3mCTZs2kpaWzpAhw0762LIe/JkzTYvPD9axbpuXbftqCUUMHDaViwZkUnzJBYwelE2Gx9ndZQrRZXpVwHc3u93OVVfNZvXqldx55w/w+32sXfs+y5a9TEpKCk899QwOhwO/3893vnMLkydfHHsjaM+KFa/h9R5l2bJX0HWdu+5aREFB+2vSy3rwHVdZ52fddi8f7aigvjlESpKdi0fkMXZINsP7Z8j4uEgYvSrg7UOntdvLPp9r0VxzzVx+/OPvcfvtd/P3v7/LqFFjyM3No66ulqef/gV79+5GUVRqaqrZu3f3KQN+06bPmD27GJvNhs1mY+bM2WzbtqXdbWU9+BPqm0P4gxGCEYNQOHoKRgxa/BE27qpiz+FGFAVGXZjF/7piCGMGZ8u0QpGQelXA9wRDhgwlKyuHTz75iNWrV/KNb0SHYf7wh6VkZmbx/PN/xWaz8cMf3kU4HO6Ux0z09eBDEYNd5fXsXXuATz+voKohcNJt8zLdXP/VQVw8Il+GX0TCk4A/C9dcM5fnn/8jlZVepk//CgAtLc0MGjQEm83G/v172bp1C1ddNeuU9zNhwkTeems1M2ZchWHovPvuW7GjO31ZIq4HX9MQYPPeGrbvr2VXeQMR3cRh1xjeP50rJvQl3ePEaddia6u4HBpOh0ZasqNXHjtTiK4gAX8WrrpqFkuX/pa5c6/Fbo8eyOCWW77NT3/6IKtWraBfv/6MHTvutPczd+517N27l5tu+gZpaekMHz6C+vraNtslwnrwx6cubtpdzabd1ZRXtQDRHvlXxhYyelAW08b1o7HBf37q0cNYkSBEgljHToSDWEYENSULNT0fxe7qmsc2DRS1a/cTWJEQ2OTNEMAMNmPWlKG401DTClC0+InFDq0Hf+DAARYvXkxDQwPp6eksWbKEAQMGtNqmurqaBx98kMOHD6PrOnfccQfz5s07o2JkPfiO683rwVuWRUsgQl1T9JBtew83sml3NVUNARRgUJ80xg/NYdzQbPIy3LHbne5bnUb9EVRPNoqtY0MzZlM1+oGNmP4GrGMn09+I5W+ASPC0t1eSM1HTC2InNFv0TeD4G8Kxn1rBMBxFX+1QTaEtqwlvfA01vRBbv1FkjZpKs6sQRW0dOpahY9aWY1Tuwag+gJY9EPvIK9ps16bNLXUEP3ge4/AOsLtQU7JRPFmoKVkoKdmoKRmg2kBVURQNFBVUFRQVxZGE4kxGcSaDIwlFObFfw4qEMJtrsFqqMZtqMJurIRIAmxPF5oi+mdgc0fOuFLTCIlSXp90au/Lbu5ZlYTVXY1TswajYjVGxG7PBe2IDRUVNy0PN6HPsVIjiTkexO6Nv6HZX9KfN0ar956pb14N/6KGHWLhwIfPmzWPFihU8+OCDvPTSS622+cUvfsHIkSP5z//8T+rq6rjuuuuYPHnySWeFiMRxtMbHh9u9lFe1UNcUpLYpSDhy4s1JUxWKLshg1pT+jBuSTVrKmY+dR/ZvIPjeM6jZ/Uma/SPUpFMPRRk1ZQRW/wor2Bz9Z3VnoLrT0LL6o/QbhZKUhuJwtfqnVuwu0GyYzTWY9UcxG7yYjRVEdq9r+4ag2aPbqxr63o+xWmpxTLzulD3m8Pa3CX/6MlqfEWDqhLe9jXfrarAnYetzEVqfIqyWOozKvRjVB8CIHnVJcXnQ935CZPc6XNNvQcsb3Oa+LctC3/MRwY+WgWngGHM1lhHBaq7BbKklUrkXQmcwy0pRwOGOhn0kiBVoatt+ZzKWHgY9DKb+r3eAmnshtv5jsPUfg5rVv9VzY1kWVqgFq6kas6UGK+Rv/WkqEox+CgEUpxvF4UZxJEVrciSBomAFmqInfxNWoBEz2IzVVI0VaIw+iMONlj8Ex9BL0XIGYgWaMOuPYNYfwag7jH7wMzhp/1cB7dgboKJGnw9FjYa+ooKqxU7K8d9tDtSUbNTUHNTUXJS0vOjPpLSOP+9n6LQ9+NraWmbOnMn69evRtOhBoadMmcI777xDZmZmbLtrrrmGn//854wePRqAO+64g8mTJ3Prrbd2uBjpwXdcT+/BhyIGG3dW8cHWo+w53IimKvTLTSErzUVWqovMVBdZqU6y0lzkZbhJcp6+r3GyXo7u3UVg9S9R0/IxG6tQPVkkXf1j1JSsdu/HqNyL/80nUexJJF3z4+jH8nMYqrAsK9rrtywUuzP6hnBsiMUyTULrXiCy8wPso2fhnLKg3ccKl64htO4lbAMn4rriuyiqhhX2k9x8gLrP16Mf2o7lqwdVQ80egJY3OHZS3OnoZZsIffhXLF8d9uFfxTnlG9HwBUx/I6G1L6CXbUbLH4rrK99GTctr245wANNfD6YBpglW9GSZZvSySAAr5MMK+rDCx36GfCh2B4onB9WTE/0E5cmOvkF+ObBNA/RwdOirpQb90Hb08q2Y1QcAUNzpaIXDscJB1EAdkYaqk3+K0hwoDhfYnICFFQ5A2H+SMFZQkjworlQUdyqKOwMtbxBa/tBo7/wUvXBLD2M2VmIFm1t9Iov9NHQsy4w+rnX8+bLANI49Z/qx59LAMnXQw8c+5dS0rtWRRJ+bHqHJlnvSWk7mnHvwXq+XvLw8NC36B6tpGrm5uXi93lYBP2LECFavXs2oUaM4fPgwmzdvpm/fvmdUbHuFVlWpaJpy2n9AWwJOgzvXNkcPCKGSk9P+R+WOMgwTf0inxRek3uvlg91+/rn5KL6gTmF2Mt+65iJmTOpHhufcx6z/tdZwdTlH3/0d9vRcCv/tMcI1h6h4+eeESn5OwY0PYc8sbLV94OB2Klb/CltKOoU3Powt7RRHfz4jJ//EYF33PWrfSaZp45sk2SFr5rdbBUvTlr/TvO4l3IMnkHf9j1G04weo9gB5JA+fGu2BN1ahpWSg2tr5Jm3uVzFHT6F+7X/T+OkqzEObybziFhTNRs1bz2GFg2ReeQtpk645xfi+BzjzkDlzfWHEWOBm9JYGAvs349/7GcFDu9DcKdjS80gaMAp7ei62tFxs6bloSR5Uhyv6qaqd+i3LwooEMYN+zJAfLBMtOR01KeXc9mcUtN9JOBeWoaM3VhOpryBSX4Hhb0LzZJGTcm7/h+3ptL0Jixcv5mc/+xnz5s2jsLCQiy++OPam0FHt9eBBIRQKY7PZ270NSA/+bOl6BFDOaOyvos7PRzsq+GLPUZJDlWQZ1eRatfSx1VGgNeBSDMabHtILLqHvlCsZekE2iqKgByNUByOnf4BT+NcevNlSh3/FY6DYcHzth9T5gKR+JF1zL4HVT3L4hQdIuvrHaFnR7wjo5VsJvPs0amouzmv+D/VhF5ynlRqtcTdgD0PTZ28SaPHhnP6/UVSVyN6PCa75I1rfkaiX3U5NXRA40XNt3WY31IeAUHsPETXm67j7TCS49kWqV/4OADVnIElfXUQko5Ca2vOzk7rjNCiYiFowkeN7W463OQzEJhqHgJAJnK5+R/SkHNvU39Pae1wKpA6OngBbSjeNwRcUFFBZWYlhGLEhmqqqqjZj65mZmfzqV7+KnV+0aBGDB7cdCzxTSUkpNDc3kJ6e1ak7NRKdZZk0N9eTlHTyP47jfMEIG3cc4tCOTXga9zPcXsGVtjoUG2CDiJaEP6mARk8RpGSRU7eF7Nq3UT7cQKTlauzDL4vuYOtIXXoYy1eH2VIHRgStYHh02ONftwv5CLz5a6ywH/ec+1A9J3riWvYA3HPvx7/ql/jf+DlJs+7B8jcQXPMsambf6PDNSXbwdRVFUXBOuQHF5iC8aQWWEcHWfyzBfzyHVjCMpK99r8PP0elo2RfgnvcTIrvXQjiIfeSVXT4rR/RMpw34rKwsioqKKCkpYd68eZSUlFBUVNRqeAagvr4ej8eDzWbj448/Zvfu3fzud7875wJTUtKor6+msvIw0SNNtqWqKqaZWD34c2+zgsPhIiWl/R08Ed1kZ+lemre+h6f5AGO0GsYrFqZbQ8m+EGe/6Wg5A6I7x5IzyWy1g2w+xuHthDeXEPpoGeHNK7GPmoWWPxhajd+2YIX80R1gLbVYLbXt7KxzYOs3CtvACZieS6P3r4cJvPM7zEYvSbN/hJbddh+Cml6Ae94D+Ff9ksCqX4IZQc0dhHv2PSgOd5vtzwdFUXBOvBZsDsKfvoK+9xPUvMEkzfr3Ds/86fBjqSqO4V/p1PsUvU+Hpknu27ePxYsX09TURGpqKkuWLOHCCy9k0aJFfP/732fUqFG8//77PP7446iqSkZGBg8++CBFRUVnVEz7QzSnl4gHReiKNhumyRdl9awvrSS8bwPXOdbhVHQaXYUk9R9JxpDR2PIHn1EY6d5dhDetxDjyeTvXKuB0o7o8KJ5s1JRMlJSs6NS9lEywLPSDn6Ef+Cy6E1O1ofW5CCwT4/AOXDPuwD741IunmYEmAm8/heJMIenKu9r9NNAdwqX/wDjyOa6v3HrKNxz5204MXTVNskMBf75IwHfc2bTZ0kOg2tp8XC+raOaDbUfZuLOKoD/A9Z7PmGrfRSi1P6mz7sKR3nbGxZkyasuxAk0ozpTotLZ25lKftG7LxKzaj71iG02lH2E11+CcugDH6NkdeuzjxzftjeRvOzF06zx40fuZjRX4V/4MANvgi7EPuYTDkQxWfHiQLXtrsNtUpg9QmBX6B06fF/voWaRMur7TvtV3fEfn2VAUFS1vMFkjx2GMvhYr0IjqTj+D2/fOcBfiXEnAJwDTV49/9a/AstDyhxDe8R6R7W8T0tPJNwfzzanTuSS7AevTv6LYnLhm3YOt/+juLrtdiqKgnEG4C5HIJODjnBXy4X/zScxAM1UT7uTNvQq76oYw2X2IGemHmB3YCLs3Yu0GrbAI1+XfQU3O6O6yhRCdQAI+zgRCOm+tL8db66OxyUdxYDl9lSr+2DyDXW/WkuS0MXPacK6ceCVulx2zqYrI3o9RHMnYL5qBospUVCHihQR8HKmq9/P717ZztNZHfoaLBba/c4FawY7C65jadwKzPS4G90nD7TrxsqupuTjHn9micEKI3kECvpcygz70wztQPTkonhxKyxp4dsUOAO65YQyDDq0ksnMfzktu5JKRV3VztUKI7iAB34tYloVRuZfIzn9Stn9DdKU+wFBsqBEPNyZnM3REEe4jq4jsfB/H2GIcEu5CJCwJ+F7ADDaj7/6QyM4PMBuOgt2FZ9RXCGaPYN2nO/FVlDMs1c8IRy3sWEkEsA+7DMekr3d36UKIbiQB34OZDV5aNr6BdWA9qmXQktwPb9/5VHiK0NRkPnnfS1lFLvOnT2HoJQNQFQUrHMDyN6Ck5cv8byESnAR8D2TUlhPeXEJk/wZ0S+Xj0GA+Cg3FW5cBhwCOAJDqtvO9r49i3JATC20pjqToAQ+EEAlPAr4HMar2Edr0Bkb5Fkybk7WRUXxsjOTmeZO4OCMJm03FrqnYbSr5eanU1LR0d8lCiB5MAr4LWOEAwbUvgKF/6ViUjugiXTY76JHokXFCPqywP7qiYqgFq6kKnMk0DprJr7dmYUtK4Uf/ayx5mW0Xo5LhFyHE6UjAdwH94Cb0fetR0wtjh+qy9NCxY1Ma0WM3OpPB6UZxJKM43aiebLSLLme7OoI/vLmPvEw399wwlgxPz1j9UAjR+0jAdwH94CaU5Azc33i8TU/bOh7w7fTA/7H5CMtW72JQ3zR+cP1okl0nP4qVEEKcjgR8J7P0MPrh7diHXtpuiJ/seJJvfHiQ5esOMHpQFt+dPxKnXY7AI4Q4NxLwncw4Ugp6GNuA8R3aXjdMXnxzJx/uqGDayHxumT0cmybrwQghzp0EfCfTD24CexJawfDTbusLRlj6P9vZWd7A/EsHMmfaANl5KoToNBLwncgyTfTyLdj6jTrtgTKqGgI89fJWahoDLCq+iItH5p+nKoUQiUICvhOZVfuwAiEurWgAABfCSURBVE2nHZ7Ze7iR3722Dcuy+NGCsQzrL+uvCyE6nwR8J9LLNoOqnfJoSBt2VvHcG6Vkpjr592+MIb+dOe5CCNEZJOA7UeTgJrSC4SiO9kN7855qnl2xg8F90rj7ulF43I7zXKEQIpHIdI1OYjQcxWqswDZgXLvX7z/axB9WfM6AfA/33DBWwl0I0eUk4DuJfnAzALYL2gZ8VUOA3766ldRkBz+4fgxOh8xxF0J0PQn4TqIf3ISaPQA1JavV5S2BCL95eSumafHDG8aQmiw9dyHE+SEB3wlMfwNm1f42wzPhiMHvXt1GbWOQ718/moKs5G6qUAiRiCTgO4FetgWwsF1wYnqkaVn8qaSUfUca+c6cixjSN737ChRCJCQJ+E6gl21G8eSgZvaNXfbymr1s3FXNghmDmTg8txurE0IkKgn4c2RFghhHPsd2wbjYMgNrtx3lnQ2HuHJCX66a1K+bKxRCJCoJ+HOkH9oOhh4bfy+vbGbZO7spuiCDb14xRNaWEUJ0Gwn4c6SXbQZnMlr+UPxBnWeW7yDZZeP2uSNQVQl3IUT3kYA/B5apo5dvxdZ/LCgqf1n9BTUNQb47f6RMhxRCdDsJ+HNgVOyBkA/bgHG8u+EQn+2u5huXD5IZM0KIHkEC/hwYhz8HRaVM6ccr/9zH+KE5fE12qgoheggJ+HNgVO7ByuzPf67aS1aqi1uvLpKdqkKIHkMC/ixZRgSjah/bmjJoCUS489qRuF2yOKcQoueQRDpLetUBMHQ2NqRx01VD6Z/n6e6ShBCiFenBnwXDNPnkn+sAGDRuPNPHFHZzRUII0VaHevAHDhxg8eLFNDQ0kJ6ezpIlSxgwYECrbWpra7nvvvvwer3ous6UKVP4yU9+gs0WXx8SdMPkDys/Z3ztfvwpWcyZcfKjNwkhRHfqUA/+oYceYuHChbz99tssXLiQBx98sM02zz77LIMGDeKNN95g5cqVfP7557zzzjudXnB3iugGS/9nO5t2VTE8qZa0gSO6uyQhhDip0wZ8bW0tpaWlFBcXA1BcXExpaSl1dXWttlMUBZ/Ph2mahMNhIpEIeXl5XVN1F/GveoLg2hfavS4UNvjtq9vYuq+W26anYzMCaAVDz2+BQghxBk47fuL1esnLy0PTokch0jSN3NxcvF4vmZmZse3uvPNOvve973HppZcSCAS48cYbmTBhwhkVk5WVcobln5CTc247OfWmWpqPlGJW7SdrzndQ7c7Ydf5ghCf/vJ6dZfX8YME4JmtfUPM55Fw0DntG9+1cPdc290bS5sQgbe4cnTZA/tZbbzFs2DBefPFFfD4fixYt4q233mLWrFkdvo/a2hZM0zrjx87J8VBd3XzGt/uycOmHQHR1yIrNH2MfeOLN6U8lpXxxoI5Fc0YwZmAGDWu2objTqY+4Uc7xcc9WZ7S5t5E2JwZpc8epqnLKjvFph2gKCgqorKzEMAwADMOgqqqKgoKCVtstW7aMuXPnoqoqHo+HGTNmsH79+jMuuLvo5VtQPNkozhT0/Rtil9c3h1hfWsmMCX2YclF0yMmo2IOWLytFCiF6ttMGfFZWFkVFRZSUlABQUlJCUVFRq+EZgL59+/LBBx8AEA6H+fjjjxkyZEgXlNz5LD2EcaQU2wXjsA0cj16+BUsPA/D3zw5jWhZXTYwuQWC21GK11KLly/i7EKJn69Asmocffphly5Yxc+ZMli1bxiOPPALAokWL2L59OwD3338/n332GXPmzGH+/PkMGDCAG264oesq70TG4VIwItGAv3AyRILoh7cTChu8v+UI44fmkJOeFN22YjeABLwQosfr0Bj8oEGDeOWVV9pc/txzz8V+79+/P3/5y186r7LzSC/fDPakaGgrgDMZff8GPq7LwxfUmTmpf2xbo2IP2F2ombKomBCiZ4uvbyGdBcsy0cu2Yus3CkWLPh32AROI7P+UNXtGMLAglUF9UmPbG97daHmDUVT5ErAQomdL+JQyqw9iBRqxXTA2dpntwkkQCZLZso+Zk/vFdqZaIR9m/WEZnhFC9AoJH/B6+RZQFGz9Tiw5oPUpIoiLKcmHmTAsJ3a5UbEner0EvBCiF5CAL9uCljcExXViLml5VYAtob5cZDuEauqxy42K3aBqaLkXdkepQghxRhI64M2WWszacrT+Y1td/s6GQ+wwB2IzQ9GjNh2jV+xGzRmIYpPjrQoher6EDni9fCtAq/H3+uYQn35RSX7ReHAmEzkQ/dKTpYcxqw+g5fWOuf1CCJHYAV+2BSU1FzX9xLdy12w6jGlaXDF5ALYLxqMf3Bw9elP1ATANbDL+LoToJRI24K1ICONoKbb+Y2OzZEIRg39uPsK4oTnkpidhv3AiRAIYhz//0hecpAcvhOgdEnYevH5kBxh6q+GZj3ZU4AvqfG1S9EtMWp8R4HAT2b8BK9iMmlHYamesEEL0ZAkb8EbZFnAkxdZ0tyyLNZ8dZkC+hyF90wBQNBu2AePQD24CC+yDp3RnyUIIcUYScojGskz08q3Y+o5CUaPvcUeqfRyp8TF9TGGrVSLtF06CcAAiAZn/LoToVRIy4M3qA1iBplbDM+u/qERVlFZfbALQ+owER3ShMQl4IURvkpABr5dtAUWNfXvVsizWl1Zy0YAMUt2t57grmg37oKmoafkoKVndUa4QQpyVhByD18u2RA/YcWyH6QFvMzWNQeZOG9ju9s5LbgQjIgf4EEL0KgnXg7eCLZh1h9D6jopdtr60EpumMH5odru3UTQbyrFhGiGE6C0SLuDNxgoAtMw+0fOmxac7Kxl1YRZul707SxNCiE6VsAGvpkW/vbrncAONLeHY8VaFECJeJF7AN1SAoqGkRodj1n9RhcOuMmZQ+8MzQgjRWyVewDdWoKTmoKg2dMNk484qxg7OxunQurs0IYToVIkX8A0VqGn5AHxRVk9LICLDM0KIuJRQAW+ZJmZTBWp6NOA/La0kyWlj5ECZ3y6EiD+JFfC+WjB01LR8IrrBpj3VTBiag92WUE+DECJBJFSymQ3HZtCkF7BtXx2BkMHki3K7uSohhOgaiRXwsSmSeXz6RSUet52iCzK6uSohhOgaiRXwDRVgTyKkJbN1bw0Th+eiqQn1FAghEkhCpZvZGN3BunVfLWHdZEqRzJ4RQsSvxAv4tHw+La0iw+Nk8LEDewghRDxKmIC39BBWSy2WJ4/t+2uZNDwXVVaHFELEsYQJeLOxCoBmLQPDtBhYkNrNFQkhRNdKoID3AlBlRodl8jPd3VmOEEJ0ucQJ+GNz4A+HosGemyHruwsh4lviBHxjBUpyJkcbdNJSHCQ5E/JgVkKIBJJQAa+m51NZFyA/Q4ZnhBDxLyEC3rKs2CqSFXV+8mT8XQiRABIj4IPNEPYTcefQEojIDlYhREJIiIA/vgZNg5oOQF6m7GAVQsS/Du1pPHDgAIsXL6ahoYH09HSWLFnCgAEDWm1z7733smvXrtj5Xbt2sXTpUq644opOLfhsmA3RKZIVeioQkB68ECIhdCjgH3roIRYuXMi8efNYsWIFDz74IC+99FKrbZ544onY7zt37uSWW25h+vTpnVvtWbIaK0G1ccjnRFEgJ1168EKI+HfaIZra2lpKS0spLi4GoLi4mNLSUurq6k56m1dffZU5c+bgcDg6r9JzYDZ4UdNyqawPkpOWhE1LiJEpIUSCO23Seb1e8vLy0LToQak1TSM3Nxev19vu9uFwmDfeeIOvf/3rnVvpOYguMlZAZb3MoBFCJI5O/7bPe++9R2FhIUVFRWd826yslLN+3JwcT7uXW6ZBc1M1nqIpVO0IMHZo7km37W3ipR1nQtqcGKTNneO0AV9QUEBlZSWGYaBpGoZhUFVVRUFBQbvbv/baa2fde6+tbcE0rTO+XU6Oh+rq5navMxsrwdSpM1MJhg1Sk2wn3bY3OVWb45W0OTFImztOVZVTdoxPO0STlZVFUVERJSUlAJSUlFBUVERmZmabbSsqKvjss8+YM2fOGRfaVY5Pkayzjk+RlCEaIURi6NDexocffphly5Yxc+ZMli1bxiOPPALAokWL2L59e2y7119/ncsvv5y0tJ5zII3ji4wdiUTf5WSZAiFEoujQGPygQYN45ZVX2lz+3HPPtTr/3e9+t3Oq6kRmYwU4kznarGC3qWSkOru7JCGEOC/ifr7g8cP0VdYFyMtIkqM4CSESRvwHfIMXNV0WGRNCJJ64DngrEsTyN6Ck5lPdIEsUCCESS1wH/PEZNC32TAzTIk92sAohEkh8B/yxGTTVZvQA29KDF0IkkvgO+MYKQOFIMBmAXFkmWAiRQOI74BsqUDxZeBsiuJ02PEn27i5JCCHOm/gO+MbWh+lTZIqkECKBxG3AW5Z1Yg58vZ98GZ4RQiSY+A34QCNEgpiePOqaQjIHXgiRcOI24M26IwA0qhmAzKARQiSeuA14o2I3KApecgFkDrwQIuHEb8B7d6JmXYC32QQgT8bghRAJJi4D3tLDGFX70AqGUVHnJz3FgcvR6QevEkKIHi0uA96oPgCGjlYwjMo6WYNGCJGY4jPgvbsABVv+UFlFUgiRsOI24NXMvvgsJy2BiOxgFUIkpLgLeMvUMSr3RIdn6v2ATJEUQiSmuAt4s/og6OFj4+/RgJcZNEKIRBR3Aa97dwEcm0ETQFUUctIl4IUQiSfuAt7w7kJNL0RNSqWyzk92ugubFnfNFEKI04qr5LNME6NiN1rBUAAq6/wy/i6ESFhxFfBmbTlEgmgFw7Esi4p6v8ygEUIkrLgKeMO7E4iOvze0hAlHTFkmWAiRsOIs4HehpOaiJmfEZtDkyhCNECJBxU3AW5aJXrEbW8EwAKobAwAyg0YIkbDiJuDNuiMQ8qEVDAegtjGIokCmx9nNlQkhRPeIm4A3vjT/HaC6IUimxylTJIUQCStu0s/w7kRJyUL1ZANQ2xggK02GZ4QQiSsuAt6yrGPz34fFLqtuDJKT5urGqoQQonvFRcBHao9gBZpiAR/RTRqaQ2RJwAshElhcBHywvBQgNoOmrjmIhcygEUIktrgI+ED55yjudJTUPABqGoIAZEsPXgiRwHp9wFuWRbCsFK1gGIqiAFBzbA58tuxkFUIksN4f8M3VGC11rXaw1jQG0VSFDJkDL4RIYL0+4I2KPQBtAj4z1YmqKt1VlhBCdLteH/Ba/hAyZ9yMml4Yu6ymISDDM0KIhNehgD9w4AALFixg5syZLFiwgIMHD7a73erVq5kzZw7FxcXMmTOHmpqazqy1XWpqLukXz4+Nv0O0By87WIUQic7WkY0eeughFi5cyLx581ixYgUPPvggL730Uqtttm/fztNPP82LL75ITk4Ozc3NOByOLin6VMIRg0ZfWAJeCJHwTtuDr62tpbS0lOLiYgCKi4spLS2lrq6u1XYvvPACt956Kzk5OQB4PB6czvO/k7O26dgUSZkDL4RIcKcNeK/XS15eHpqmAaBpGrm5uXi93lbb7du3j0OHDnHjjTdy7bXX8swzz2BZVtdUfQrVMgdeCCGADg7RdIRhGOzatYu//OUvhMNhbrvtNgoLC5k/f36H7yMrK+WsHz8nxwNAaE903H/Yhdlxv9jY8TYnEmlzYpA2d47TBnxBQQGVlZUYhoGmaRiGQVVVFQUFBa22KywsZNasWTgcDhwOB1dccQXbtm07o4CvrW3BNM+815+T46G6uhmAA4cbsGkqeihCdbV+xvfVW3y5zYlC2pwYpM0dp6rKKTvGpx2iycrKoqioiJKSEgBKSkooKioiMzOz1XbFxcWsW7cOy7KIRCJ88sknDB8+/IwLPlc1jUGy0lyoisyBF0Iktg5Nk3z44YdZtmwZM2fOZNmyZTzyyCMALFq0iO3btwNwzTXXkJWVxdVXX838+fMZPHgw119/fddVfhLROfAy/i6EEIrVHXtCT6Izhmi+/9u1TByWw7/NOv+fHs4n+RibGKTNiaHbhmh6k2BYpyUQkXXghRCCOAv4msboFElZB14IIeIt4I/NgZcevBBCxFvAH1sHPifO578LIURHxFnAB3HYVTxue3eXIoQQ3S7uAj47LanVypJCCJGo4ivgZQ68EELExFfAyzrwQggREzcB7w9G8Id0OZKTEEIcEzcBf3wOvPTghRAiKm4CPrYOfLoEvBBCQBwFfO2xOfAyRCOEEFFxE/DVjUFcDo1kV6cdw0QIIXq1uAn4WpkDL4QQrcRNwFc3yhx4IYT4srgIeMuyonPgZQerEELExEXAN/nChMKG7GAVQogviYuAr6r3A5AjQzRCCBETFwFfWRcNeFkHXgghToiLgK86FvAyRCOEECfERcBX1PlJdtlwyxx4IYSIiYuAr6zzS+9dCCH+RVwEfFWdX+bACyHEv+j1AW9ZVjTgZQ68EEK00usDvskXJqybMkQjhBD/otcHfLWsAy+EEO3q9QFfE1smWAJeCCG+rNcHPEBKkp3sdBmiEUKIL+v1E8enFOVx5dSBtDQFursUIYToUXp9D15RFJKcvf59SgghOl2vD3ghhBDtk4AXQog4JQEvhBBxSgJeCCHilAS8EELEKQl4IYSIUz1qfqGqKt1y295K2pwYpM2J4WzafLrbKJZlWWdbkBBCiJ5LhmiEECJOScALIUSckoAXQog4JQEvhBBxSgJeCCHilAS8EELEKQl4IYSIUxLwQggRpyTghRAiTvX6gD9w4AALFixg5syZLFiwgIMHD3Z3SZ1qyZIlzJgxg2HDhrF79+7Y5fHc7vr6ehYtWsTMmTOZM2cOd999N3V1dQBs2bKFuXPnMnPmTG699VZqa2u7udrOc+eddzJ37lzmz5/PwoUL+eKLL4D4fq0Bnn766VZ/3/H8GgPMmDGDWbNmMW/ePObNm8fatWuBLmq31cvdfPPN1vLlyy3Lsqzly5dbN998czdX1Lk2bNhgHT161Lr88sutXbt2xS6P53bX19dbn3zySez8L37xC+u+++6zDMOwrrzySmvDhg2WZVnW0qVLrcWLF3dXmZ2uqakp9vu7775rzZ8/37Ks+H6td+zYYX3729+O/X3H+2tsWVab/2XLsrqs3b26B19bW0tpaSnFxcUAFBcXU1paGuvtxYOJEydSUFDQ6rJ4b3d6ejpTpkyJnR87dixHjx5lx44dOJ1OJk6cCMA3v/lN3nrrre4qs9N5PJ7Y7y0tLSiKEtevdTgc5tFHH+Xhhx+OXRbvr/HJdFW7e9RqkmfK6/WSl5eHpmkAaJpGbm4uXq+XzMzMbq6u6yRSu03T5L/+67+YMWMGXq+XwsLC2HWZmZmYpklDQwPp6endWGXneeCBB/jwww+xLIs//elPcf1a//a3v2Xu3Ln07ds3dlkivMYAP/7xj7EsiwkTJnDPPfd0Wbt7dQ9exL+f/vSnuN1ubrrppu4u5bx4/PHH+ec//8kPf/hDnnjiie4up8ts3ryZHTt2sHDhwu4u5bz761//ysqVK3nttdewLItHH320yx6rVwd8QUEBlZWVGIYBgGEYVFVVtRnSiDeJ0u4lS5ZQVlbGU089haqqFBQUcPTo0dj1dXV1qKoaVz274+bPn8/69evJz8+Py9d6w4YN7Nu3jyuuuIIZM2ZQUVHBt7/9bcrKyuL+NT7+2jkcDhYuXMimTZu67G+7Vwd8VlYWRUVFlJSUAFBSUkJRUVGv/+h6OonQ7l//+tfs2LGDpUuX4nA4ABg5ciTBYJCNGzcC8Le//Y1Zs2Z1Z5mdxufz4fV6Y+fXrFlDWlpa3L7W3/nOd1i3bh1r1qxhzZo15Ofn8+c//5nbbrstbl9jAL/fT3NzMwCWZbF69WqKioq67G+71x/wY9++fSxevJimpiZSU1NZsmQJF154YXeX1Wkee+wx3nnnHWpqasjIyCA9PZ1Vq1bFdbv37NlDcXExAwYMwOVyAdC3b1+WLl3Kpk2beOihhwiFQvTp04df/vKXZGdnd3PF566mpoY777yTQCCAqqqkpaXxH//xH4wYMSKuX+vjZsyYwbPPPsvQoUPj9jUGOHToEN/73vcwDAPTNBk0aBA/+clPyM3N7ZJ29/qAF0II0b5ePUQjhBDi5CTghRAiTknACyFEnJKAF0KIOCUBL4QQcUoCXggh4pQEvBBCxCkJeCGEiFP/H5nPAi4N/vPIAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5eqRQlywWGFg"
      },
      "source": [
        "Les hyperparamètres choisis sont : \n",
        "\n",
        "\n",
        "|Embedding dim|Hidden dim|Learning rate|Number of layers|Dropout|\n",
        "|-------------|----------|-------------|----------------|-------|\n",
        "|50           | 256      | 1e-4        |   2            |  0.5  |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZFQHmKROWGFg"
      },
      "source": [
        "## Tester le modèle"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6r7vlOr5WGFh",
        "outputId": "d5d5aa3b-88da-4175-ec21-d27f534da697",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "model.load_state_dict(torch.load('tut2-model.pt'))\n",
        "test_loss, test_acc = evaluate(model, test_iterator, criterion)\n",
        "\n",
        "print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test Loss: 0.244 | Test Acc: 90.54%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lCWEAdcOWGFl"
      },
      "source": [
        "### Inférence"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VyiLWKJDWGFl"
      },
      "source": [
        "import spacy\n",
        "nlp = spacy.load('en')\n",
        "\n",
        "def predict_sentiment(model, sentence):\n",
        "    model.eval()\n",
        "    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]\n",
        "    indexed = [TEXT.vocab.stoi[t] for t in tokenized]\n",
        "    length = [len(indexed)]\n",
        "    tensor = torch.LongTensor(indexed).to(device)\n",
        "    tensor = tensor.unsqueeze(1)\n",
        "    length_tensor = torch.LongTensor(length)\n",
        "    prediction = torch.sigmoid(model(tensor, length_tensor))\n",
        "    if (prediction.item() > 0.5):\n",
        "      print (\"Sentiment: Positive\")\n",
        "    else:\n",
        "      print (\"Sentiment: Negative\")\n"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jIsH7rPSWGFo",
        "outputId": "2214fb83-ed19-4c1d-9942-809eb5591307",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "predict_sentiment(model, \"This film is great\")"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentiment: Positive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NSH4wQFrWGFr",
        "outputId": "fed2a963-23f8-4e7a-f315-29da24b3dae7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "predict_sentiment(model, \"This film is terrible\")"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentiment: Negative\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2jBo8L92WGFu",
        "outputId": "6a721a48-e6d2-4b04-85e0-afd1491932b2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "predict_sentiment(model, \"This film is not good at all\")"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentiment: Positive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ej-EptQ34fW8"
      },
      "source": [
        "Comme pour le modèle BiLSTM, on remarque que le modèle GRU nous donne de bons résultats d'acuracy mais curieusement, il ne prédit pas correctement la négation.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gVDHBD5TWGFy"
      },
      "source": [
        "## Référence : \n",
        "\n",
        " - https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/2%20-%20Upgraded%20Sentiment%20Analysis.ipynb"
      ]
    }
  ]
}