Edureka_Sequence_Learning_Assignment_2_NER_Task.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Edureka Sequence Learning - Assignment 2 - NER Task.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "1tmEozH8K0a772Qll8pn62qnHKBPsIpxJ",
      "authorship_tag": "ABX9TyNlcke5uLolwJ/Fau+Dezts",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nandinib1999/edureka-assignments/blob/main/Edureka_Sequence_Learning_Assignment_2_NER_Task.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9idmXq_t0mMz"
      },
      "source": [
        "### Problem Statement\n",
        "\n",
        "In Natural Language Processing (NLP), Entity Recognition is one of \n",
        "the common problems. NER is a method of extracting the relevant \n",
        "information from a large corpus and classifying those entities into \n",
        "predefined categories such as location, organization, name, and so \n",
        "on. \n",
        "In this problem you are given tagged data for entities, you have to \n",
        "predict those entities using LSTM models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pFYEF6puwha1"
      },
      "source": [
        "### Installing packages and loading data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "11msyHYSPYvm",
        "outputId": "00386fc2-e034-42c3-f55e-3249740f12be"
      },
      "source": [
        "!pip install sklearn_crfsuite"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting sklearn_crfsuite\n",
            "  Downloading https://files.pythonhosted.org/packages/25/74/5b7befa513482e6dee1f3dd68171a6c9dfc14c0eaa00f885ffeba54fe9b0/sklearn_crfsuite-0.3.6-py2.py3-none-any.whl\n",
            "Collecting python-crfsuite>=0.8.3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/79/47/58f16c46506139f17de4630dbcfb877ce41a6355a1bbf3c443edb9708429/python_crfsuite-0.9.7-cp37-cp37m-manylinux1_x86_64.whl (743kB)\n",
            "\u001b[K     |████████████████████████████████| 747kB 10.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sklearn_crfsuite) (1.15.0)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from sklearn_crfsuite) (0.8.9)\n",
            "Requirement already satisfied: tqdm>=2.0 in /usr/local/lib/python3.7/dist-packages (from sklearn_crfsuite) (4.41.1)\n",
            "Installing collected packages: python-crfsuite, sklearn-crfsuite\n",
            "Successfully installed python-crfsuite-0.9.7 sklearn-crfsuite-0.3.6\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z1sQlY1Tsxjq",
        "outputId": "e862e722-e2fb-41ff-c1d3-7867fc0817aa"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "data = pd.read_csv(\"/content/drive/MyDrive/ner.csv\", encoding = \"ISO-8859-1\", error_bad_lines=False)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "b'Skipping line 281837: expected 25 fields, saw 34\\n'\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 430
        },
        "id": "2TyYpFkQ9SRo",
        "outputId": "5ac18a67-b688-4fdf-9055-3fc9d3d828a8"
      },
      "source": [
        "data.head(10)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>lemma</th>\n",
              "      <th>next-lemma</th>\n",
              "      <th>next-next-lemma</th>\n",
              "      <th>next-next-pos</th>\n",
              "      <th>next-next-shape</th>\n",
              "      <th>next-next-word</th>\n",
              "      <th>next-pos</th>\n",
              "      <th>next-shape</th>\n",
              "      <th>next-word</th>\n",
              "      <th>pos</th>\n",
              "      <th>prev-iob</th>\n",
              "      <th>prev-lemma</th>\n",
              "      <th>prev-pos</th>\n",
              "      <th>prev-prev-iob</th>\n",
              "      <th>prev-prev-lemma</th>\n",
              "      <th>prev-prev-pos</th>\n",
              "      <th>prev-prev-shape</th>\n",
              "      <th>prev-prev-word</th>\n",
              "      <th>prev-shape</th>\n",
              "      <th>prev-word</th>\n",
              "      <th>sentence_idx</th>\n",
              "      <th>shape</th>\n",
              "      <th>word</th>\n",
              "      <th>tag</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>thousand</td>\n",
              "      <td>of</td>\n",
              "      <td>demonstr</td>\n",
              "      <td>NNS</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>IN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>of</td>\n",
              "      <td>NNS</td>\n",
              "      <td>__START1__</td>\n",
              "      <td>__start1__</td>\n",
              "      <td>__START1__</td>\n",
              "      <td>__START2__</td>\n",
              "      <td>__start2__</td>\n",
              "      <td>__START2__</td>\n",
              "      <td>wildcard</td>\n",
              "      <td>__START2__</td>\n",
              "      <td>wildcard</td>\n",
              "      <td>__START1__</td>\n",
              "      <td>1.0</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>Thousands</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>of</td>\n",
              "      <td>demonstr</td>\n",
              "      <td>have</td>\n",
              "      <td>VBP</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>have</td>\n",
              "      <td>NNS</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>IN</td>\n",
              "      <td>O</td>\n",
              "      <td>thousand</td>\n",
              "      <td>NNS</td>\n",
              "      <td>__START1__</td>\n",
              "      <td>__start1__</td>\n",
              "      <td>__START1__</td>\n",
              "      <td>wildcard</td>\n",
              "      <td>__START1__</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>Thousands</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>of</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>demonstr</td>\n",
              "      <td>have</td>\n",
              "      <td>march</td>\n",
              "      <td>VBN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>marched</td>\n",
              "      <td>VBP</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>have</td>\n",
              "      <td>NNS</td>\n",
              "      <td>O</td>\n",
              "      <td>of</td>\n",
              "      <td>IN</td>\n",
              "      <td>O</td>\n",
              "      <td>thousand</td>\n",
              "      <td>NNS</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>Thousands</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>of</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>have</td>\n",
              "      <td>march</td>\n",
              "      <td>through</td>\n",
              "      <td>IN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>through</td>\n",
              "      <td>VBN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>marched</td>\n",
              "      <td>VBP</td>\n",
              "      <td>O</td>\n",
              "      <td>demonstr</td>\n",
              "      <td>NNS</td>\n",
              "      <td>O</td>\n",
              "      <td>of</td>\n",
              "      <td>IN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>of</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>have</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>march</td>\n",
              "      <td>through</td>\n",
              "      <td>london</td>\n",
              "      <td>NNP</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>London</td>\n",
              "      <td>IN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>through</td>\n",
              "      <td>VBN</td>\n",
              "      <td>O</td>\n",
              "      <td>have</td>\n",
              "      <td>VBP</td>\n",
              "      <td>O</td>\n",
              "      <td>demonstr</td>\n",
              "      <td>NNS</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>have</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>marched</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5</td>\n",
              "      <td>through</td>\n",
              "      <td>london</td>\n",
              "      <td>to</td>\n",
              "      <td>TO</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>to</td>\n",
              "      <td>NNP</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>London</td>\n",
              "      <td>IN</td>\n",
              "      <td>O</td>\n",
              "      <td>march</td>\n",
              "      <td>VBN</td>\n",
              "      <td>O</td>\n",
              "      <td>have</td>\n",
              "      <td>VBP</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>have</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>marched</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>through</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>6</td>\n",
              "      <td>london</td>\n",
              "      <td>to</td>\n",
              "      <td>protest</td>\n",
              "      <td>VB</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>protest</td>\n",
              "      <td>TO</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>to</td>\n",
              "      <td>NNP</td>\n",
              "      <td>O</td>\n",
              "      <td>through</td>\n",
              "      <td>IN</td>\n",
              "      <td>O</td>\n",
              "      <td>march</td>\n",
              "      <td>VBN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>marched</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>through</td>\n",
              "      <td>1.0</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>London</td>\n",
              "      <td>B-geo</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>7</td>\n",
              "      <td>to</td>\n",
              "      <td>protest</td>\n",
              "      <td>the</td>\n",
              "      <td>DT</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>the</td>\n",
              "      <td>VB</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>protest</td>\n",
              "      <td>TO</td>\n",
              "      <td>B-geo</td>\n",
              "      <td>london</td>\n",
              "      <td>NNP</td>\n",
              "      <td>O</td>\n",
              "      <td>through</td>\n",
              "      <td>IN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>through</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>London</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>to</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>8</td>\n",
              "      <td>protest</td>\n",
              "      <td>the</td>\n",
              "      <td>war</td>\n",
              "      <td>NN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>war</td>\n",
              "      <td>DT</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>the</td>\n",
              "      <td>VB</td>\n",
              "      <td>O</td>\n",
              "      <td>to</td>\n",
              "      <td>TO</td>\n",
              "      <td>B-geo</td>\n",
              "      <td>london</td>\n",
              "      <td>NNP</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>London</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>to</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>protest</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>9</td>\n",
              "      <td>the</td>\n",
              "      <td>war</td>\n",
              "      <td>in</td>\n",
              "      <td>IN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>in</td>\n",
              "      <td>NN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>war</td>\n",
              "      <td>DT</td>\n",
              "      <td>O</td>\n",
              "      <td>protest</td>\n",
              "      <td>VB</td>\n",
              "      <td>O</td>\n",
              "      <td>to</td>\n",
              "      <td>TO</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>to</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>protest</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>the</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0     lemma next-lemma  ...        shape           word    tag\n",
              "0           0  thousand         of  ...  capitalized      Thousands      O\n",
              "1           1        of   demonstr  ...    lowercase             of      O\n",
              "2           2  demonstr       have  ...    lowercase  demonstrators      O\n",
              "3           3      have      march  ...    lowercase           have      O\n",
              "4           4     march    through  ...    lowercase        marched      O\n",
              "5           5   through     london  ...    lowercase        through      O\n",
              "6           6    london         to  ...  capitalized         London  B-geo\n",
              "7           7        to    protest  ...    lowercase             to      O\n",
              "8           8   protest        the  ...    lowercase        protest      O\n",
              "9           9       the        war  ...    lowercase            the      O\n",
              "\n",
              "[10 rows x 25 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qoYk4r8wI4EJ"
      },
      "source": [
        "df = data.drop(['Unnamed: 0', 'lemma', 'next-lemma', 'next-next-lemma', 'next-next-pos',\n",
        "                'next-next-shape', 'next-next-word', 'next-pos', 'next-shape',\n",
        "                'next-word', 'prev-iob', 'prev-lemma', 'prev-pos',\n",
        "                'prev-prev-iob', 'prev-prev-lemma', 'prev-prev-pos', 'prev-prev-shape',\n",
        "                'prev-prev-word', 'prev-shape', 'prev-word', \"pos\", \"shape\"], axis=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nQkj-alNJEdM",
        "outputId": "27c8ec2b-add0-4fd9-dd0f-a1e154dae244"
      },
      "source": [
        "df.columns"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['sentence_idx', 'word', 'tag'], dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "id": "0j6qzQQvJG6S",
        "outputId": "5f78bf40-df9b-4f6b-85bc-fa3551c659fe"
      },
      "source": [
        "df.head(10)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence_idx</th>\n",
              "      <th>word</th>\n",
              "      <th>tag</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>Thousands</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>of</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.0</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.0</td>\n",
              "      <td>have</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.0</td>\n",
              "      <td>marched</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.0</td>\n",
              "      <td>through</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1.0</td>\n",
              "      <td>London</td>\n",
              "      <td>B-geo</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.0</td>\n",
              "      <td>to</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.0</td>\n",
              "      <td>protest</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.0</td>\n",
              "      <td>the</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   sentence_idx           word    tag\n",
              "0           1.0      Thousands      O\n",
              "1           1.0             of      O\n",
              "2           1.0  demonstrators      O\n",
              "3           1.0           have      O\n",
              "4           1.0        marched      O\n",
              "5           1.0        through      O\n",
              "6           1.0         London  B-geo\n",
              "7           1.0             to      O\n",
              "8           1.0        protest      O\n",
              "9           1.0            the      O"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l_Cg1NhGKPHR",
        "outputId": "88ad761c-8e8e-4886-a393-39fe8175971e"
      },
      "source": [
        "df.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1050795, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AAB5LLe8KQ2-",
        "outputId": "57501a84-d94f-4194-819c-80b320d1cefe"
      },
      "source": [
        "df.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 1050795 entries, 0 to 1050794\n",
            "Data columns (total 3 columns):\n",
            " #   Column        Non-Null Count    Dtype  \n",
            "---  ------        --------------    -----  \n",
            " 0   sentence_idx  1050794 non-null  float64\n",
            " 1   word          1050794 non-null  object \n",
            " 2   tag           1050794 non-null  object \n",
            "dtypes: float64(1), object(2)\n",
            "memory usage: 24.1+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o6XkFdbvKWrp",
        "outputId": "98a553ae-79e1-44ed-a90b-5f2b74dec18a"
      },
      "source": [
        "df.isna().sum()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "sentence_idx    1\n",
              "word            1\n",
              "tag             1\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9dSlXaQ4KgLf",
        "outputId": "87e4dddc-a604-449f-f348-d2e65b5ecc95"
      },
      "source": [
        "df = df[~df.isna().any(axis=1)]\n",
        "df.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 1050794 entries, 0 to 1050794\n",
            "Data columns (total 3 columns):\n",
            " #   Column        Non-Null Count    Dtype  \n",
            "---  ------        --------------    -----  \n",
            " 0   sentence_idx  1050794 non-null  float64\n",
            " 1   word          1050794 non-null  object \n",
            " 2   tag           1050794 non-null  object \n",
            "dtypes: float64(1), object(2)\n",
            "memory usage: 32.1+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uYoFoLJoLCYw",
        "outputId": "be551ed3-7e58-4db9-f091-40f2db7ef748"
      },
      "source": [
        "df.isna().sum()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "sentence_idx    0\n",
              "word            0\n",
              "tag             0\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QOnx1301w0Mo"
      },
      "source": [
        "Upon checking the data manually, I saw that the sentences were repeated in the dataset. That's why I am deleting the duplicate rows from the dataframes for every repeated pair of \"sentence index\" and \"word\"."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sR7k3zvC-e0R",
        "outputId": "bc0efdac-5d2a-46aa-f953-bc83a0476206"
      },
      "source": [
        "df.drop_duplicates(subset =['sentence_idx','word'], keep = 'first', inplace = True)\n",
        "df.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(715447, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O7gIb8NsQFyy",
        "outputId": "4e8322ba-509d-4948-ad5c-1fb623716b84"
      },
      "source": [
        "df[\"tag\"].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "O        600897\n",
              "B-geo     26718\n",
              "B-tim     14735\n",
              "B-org     14504\n",
              "I-per     12467\n",
              "B-per     12140\n",
              "I-org     11949\n",
              "B-gpe     11221\n",
              "I-geo      5176\n",
              "I-tim      4403\n",
              "B-art       287\n",
              "B-eve       228\n",
              "I-eve       196\n",
              "I-art       185\n",
              "B-nat       156\n",
              "I-gpe       142\n",
              "I-nat        43\n",
              "Name: tag, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 124
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9QzuSucHxUgx"
      },
      "source": [
        "We can see that there is a class imbalance in the dataset for the tags. The number of words tagged as \"O\" surpasses the number of words tagged as the other classes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zamT4kIBxGWM"
      },
      "source": [
        "### Loading all the sentences and the tags in a list"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CYPPTKaaJzN6"
      },
      "source": [
        "grp_df = df.groupby('sentence_idx')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mupBBFXaMR9r",
        "outputId": "570a59f1-5d02-4943-8060-bcb90ea425cf"
      },
      "source": [
        "import tqdm\n",
        "\n",
        "all_sentences = []\n",
        "max_len = 0\n",
        "for sent_indx, sent_word_tags in tqdm.tqdm(grp_df, total=len(grp_df)):\n",
        "  word_lst = sent_word_tags['word'].tolist()\n",
        "  tag_lst = sent_word_tags['tag'].tolist()\n",
        "  sent = [(word.lower(), tag) for word, tag in zip(word_lst, tag_lst)]\n",
        "  if max_len < len(sent):\n",
        "    max_len = len(sent)\n",
        "  all_sentences.append(sent)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 35177/35177 [00:04<00:00, 7368.83it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p5PIlYARNpqU",
        "outputId": "6797aaf7-1961-4c3c-875c-075765f8ece5"
      },
      "source": [
        "max_len"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "58"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dXk0Yhu0NR8S",
        "outputId": "6775aeaf-e212-4b6c-9d88-9372b8876d22"
      },
      "source": [
        "all_sentences[1]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('families', 'O'),\n",
              " ('of', 'O'),\n",
              " ('soldiers', 'O'),\n",
              " ('killed', 'O'),\n",
              " ('in', 'O'),\n",
              " ('the', 'O'),\n",
              " ('conflict', 'O'),\n",
              " ('joined', 'O'),\n",
              " ('protesters', 'O'),\n",
              " ('who', 'O'),\n",
              " ('carried', 'O'),\n",
              " ('banners', 'O'),\n",
              " ('with', 'O'),\n",
              " ('such', 'O'),\n",
              " ('slogans', 'O'),\n",
              " ('as', 'O'),\n",
              " ('\"', 'O'),\n",
              " ('bush', 'B-per'),\n",
              " ('number', 'O'),\n",
              " ('one', 'O'),\n",
              " ('terrorist', 'O'),\n",
              " ('and', 'O'),\n",
              " ('stop', 'O'),\n",
              " ('bombings', 'O'),\n",
              " ('.', 'O')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TQ0zfP5SxrRE"
      },
      "source": [
        "#### Counting the unique words and tags in the dataset for mapping every word/tag to a unique index value"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6CBKzqKcaKTQ"
      },
      "source": [
        "words = list(set(df[\"word\"].values))\n",
        "words = [x.lower() for x in words]\n",
        "words.append(\"EOS\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ma1T_l8_a-ca",
        "outputId": "2417effc-c1f0-4cf3-ee6e-7687baf8eaab"
      },
      "source": [
        "n_words = len(words) \n",
        "n_words"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "30173"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aJ24pT5vbBWU"
      },
      "source": [
        "tags = list(set(df[\"tag\"].values))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dPvLMQA4bDt4",
        "outputId": "4d7eeab7-8f13-409b-95a4-4a7ede8af055"
      },
      "source": [
        "n_tags = len(tags) \n",
        "n_tags"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "17"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pAiUldIabIkk"
      },
      "source": [
        "word2idx = {w: i for i, w in enumerate(words)}\n",
        "tag2idx = {t: i for i, t in enumerate(tags)}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7fIeoCpVPQCE"
      },
      "source": [
        "idx2word = {i: w for w, i in word2idx.items()}\n",
        "idx2tag = {i: w for w, i in tag2idx.items()}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I8glAW5sx8Nt"
      },
      "source": [
        "#### Padding all the sentences and sequence of tags to the length of the longest sentence in the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6x0v61IhbMch"
      },
      "source": [
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "X = [[word2idx[w[0]] for w in s] for s in all_sentences]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sd3NsjVc_Bg6"
      },
      "source": [
        "X = pad_sequences(maxlen=max_len, sequences=X, padding=\"post\", value=n_words - 1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aNOuLzCj_Gw1"
      },
      "source": [
        "y = [[tag2idx[w[1]] for w in s] for s in all_sentences]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ji_FgyER_MsF"
      },
      "source": [
        "y = pad_sequences(maxlen=max_len, sequences=y, padding=\"post\", value=tag2idx[\"O\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WPWbSTi6yWHC"
      },
      "source": [
        "#### Converting the tags to categorical data to feed into the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PiopJsLO_PuO"
      },
      "source": [
        "from tensorflow.keras.utils import to_categorical\n",
        "y = [to_categorical(i, num_classes=n_tags) for i in y]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7UtSH25gyeYx"
      },
      "source": [
        "#### Splitting the data into train and test set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xqGtpmk6_ZiF"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RMwutdnc_c5r"
      },
      "source": [
        "from keras.models import Model, Input\n",
        "from keras.layers import LSTM, Embedding, Dense, TimeDistributed, Dropout, Bidirectional"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8x5n8MVGyi8a"
      },
      "source": [
        "#### Defining the four-layered model with BiLSTM"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TIgqjspB_gFj",
        "outputId": "5e74efa1-d1d7-435a-93af-d6017d9f02e1"
      },
      "source": [
        "input = Input(shape=(max_len,))\n",
        "model = Embedding(input_dim=n_words, output_dim=max_len, input_length=max_len)(input)\n",
        "model = Dropout(0.2)(model)\n",
        "model = Bidirectional(LSTM(units=100, return_sequences=True, recurrent_dropout=0.1))(model)\n",
        "out = TimeDistributed(Dense(n_tags, activation=\"softmax\"))(model)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Layer lstm will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
            "WARNING:tensorflow:Layer lstm will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
            "WARNING:tensorflow:Layer lstm will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XM-X9XI-_suE"
      },
      "source": [
        "model = Model(input, out)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sZHbW3aC_v8E"
      },
      "source": [
        "model.compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6pcPTBJO_xHS",
        "outputId": "7c4d3d09-a4f7-4e73-8b52-68a3b303453a"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         [(None, 58)]              0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 58, 58)            1750034   \n",
            "_________________________________________________________________\n",
            "dropout (Dropout)            (None, 58, 58)            0         \n",
            "_________________________________________________________________\n",
            "bidirectional (Bidirectional (None, 58, 200)           127200    \n",
            "_________________________________________________________________\n",
            "time_distributed (TimeDistri (None, 58, 17)            3417      \n",
            "=================================================================\n",
            "Total params: 1,880,651\n",
            "Trainable params: 1,880,651\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0HC3uH8hBOzU"
      },
      "source": [
        "!mkdir \"saved_model\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j7bATl4myrsa"
      },
      "source": [
        "#### Initializing some callbacks for model training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j_rTLEarAmz6"
      },
      "source": [
        "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
        "import numpy as np\n",
        "\n",
        "es = EarlyStopping(monitor=\"val_loss\", mode=\"min\", patience=3)\n",
        "checkpt = ModelCheckpoint(\"/content/saved_model\", monitor='val_loss', mode='min', save_best_only=True, verbose=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BdGetzR0_2Pc",
        "outputId": "683c7268-3229-4808-c3e0-400d6291de8b"
      },
      "source": [
        "history = model.fit(X_train, np.array(y_train), batch_size=128, epochs=200, validation_split=0.2, verbose=1, callbacks=[es, checkpt])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/200\n",
            "176/176 [==============================] - 93s 375ms/step - loss: 0.8422 - accuracy: 0.9125 - val_loss: 0.2055 - val_accuracy: 0.9446\n",
            "\n",
            "Epoch 00001: val_loss improved from inf to 0.20551, saving model to /content/saved_model\n",
            "INFO:tensorflow:Assets written to: /content/saved_model/assets\n",
            "Epoch 2/200\n",
            "176/176 [==============================] - 65s 367ms/step - loss: 0.1780 - accuracy: 0.9492 - val_loss: 0.1313 - val_accuracy: 0.9605\n",
            "\n",
            "Epoch 00002: val_loss improved from 0.20551 to 0.13130, saving model to /content/saved_model\n",
            "INFO:tensorflow:Assets written to: /content/saved_model/assets\n",
            "Epoch 3/200\n",
            "176/176 [==============================] - 65s 369ms/step - loss: 0.1173 - accuracy: 0.9652 - val_loss: 0.0902 - val_accuracy: 0.9758\n",
            "\n",
            "Epoch 00003: val_loss improved from 0.13130 to 0.09023, saving model to /content/saved_model\n",
            "INFO:tensorflow:Assets written to: /content/saved_model/assets\n",
            "Epoch 4/200\n",
            "176/176 [==============================] - 64s 366ms/step - loss: 0.0782 - accuracy: 0.9782 - val_loss: 0.0717 - val_accuracy: 0.9796\n",
            "\n",
            "Epoch 00004: val_loss improved from 0.09023 to 0.07168, saving model to /content/saved_model\n",
            "INFO:tensorflow:Assets written to: /content/saved_model/assets\n",
            "Epoch 5/200\n",
            "176/176 [==============================] - 64s 363ms/step - loss: 0.0595 - accuracy: 0.9826 - val_loss: 0.0658 - val_accuracy: 0.9813\n",
            "\n",
            "Epoch 00005: val_loss improved from 0.07168 to 0.06584, saving model to /content/saved_model\n",
            "INFO:tensorflow:Assets written to: /content/saved_model/assets\n",
            "Epoch 6/200\n",
            "176/176 [==============================] - 65s 369ms/step - loss: 0.0506 - accuracy: 0.9851 - val_loss: 0.0648 - val_accuracy: 0.9820\n",
            "\n",
            "Epoch 00006: val_loss improved from 0.06584 to 0.06477, saving model to /content/saved_model\n",
            "INFO:tensorflow:Assets written to: /content/saved_model/assets\n",
            "Epoch 7/200\n",
            "176/176 [==============================] - 65s 367ms/step - loss: 0.0443 - accuracy: 0.9867 - val_loss: 0.0637 - val_accuracy: 0.9824\n",
            "\n",
            "Epoch 00007: val_loss improved from 0.06477 to 0.06374, saving model to /content/saved_model\n",
            "INFO:tensorflow:Assets written to: /content/saved_model/assets\n",
            "Epoch 8/200\n",
            "176/176 [==============================] - 65s 372ms/step - loss: 0.0399 - accuracy: 0.9879 - val_loss: 0.0630 - val_accuracy: 0.9827\n",
            "\n",
            "Epoch 00008: val_loss improved from 0.06374 to 0.06304, saving model to /content/saved_model\n",
            "INFO:tensorflow:Assets written to: /content/saved_model/assets\n",
            "Epoch 9/200\n",
            "176/176 [==============================] - 65s 372ms/step - loss: 0.0369 - accuracy: 0.9888 - val_loss: 0.0635 - val_accuracy: 0.9827\n",
            "\n",
            "Epoch 00009: val_loss did not improve from 0.06304\n",
            "Epoch 10/200\n",
            "176/176 [==============================] - 65s 369ms/step - loss: 0.0333 - accuracy: 0.9897 - val_loss: 0.0636 - val_accuracy: 0.9827\n",
            "\n",
            "Epoch 00010: val_loss did not improve from 0.06304\n",
            "Epoch 11/200\n",
            "176/176 [==============================] - 68s 385ms/step - loss: 0.0312 - accuracy: 0.9903 - val_loss: 0.0644 - val_accuracy: 0.9826\n",
            "\n",
            "Epoch 00011: val_loss did not improve from 0.06304\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nuqhgaJKyzrH"
      },
      "source": [
        "#### Visualizing the accuracy and loss from model training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "nIlMQH7kFN3A",
        "outputId": "b26aa732-01f6-48bd-dd94-66080f06bae4"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.plot(history.history['accuracy'])\n",
        "plt.plot(history.history['val_accuracy'])\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'test'], loc='upper left')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3xW5f34/9c7m5CEkSmEESEsEVmiMmQ5cFRF26oUP3b8Sq3aqlWrflrbb+3Xr7Z170nVOqql2tqWKkOmgDJVZhIiSAJkAQnZ6/3749yBmxDgBHJyJ7nfz8cjj/us6+R9GOd9X9d1znWJqmKMMcY0FhLoAIwxxrRNliCMMcY0yRKEMcaYJlmCMMYY0yRLEMYYY5oUFugAWkpCQoL27ds30GEYY0y7snbt2kJVTWxqX4dJEH379mXNmjWBDsMYY9oVEdl5rH3WxGSMMaZJliCMMcY0yRKEMcaYJnWYPoim1NTUkJOTQ2VlZaBD8VxUVBSpqamEh4cHOhRjTAfRoRNETk4OsbGx9O3bFxEJdDieUVWKiorIyckhLS0t0OEYYzqIDt3EVFlZSXx8fIdODgAiQnx8fFDUlIwxrcfTBCEi00Rkm4hkici9TezvIyILReRLEVksIql++/4gIht9P9eeQgwnW7RdCZbrNMa0Hs+amEQkFHgWuBDIAVaLyIequtnvsEeAN1T1dRGZAjwE3CAilwEjgeFAJLBYRP6rqiVexWuMMW1ddW09RWVVFB6sprCsisKDVRSWVtOlUzgzzund4r/Pyz6IMUCWqmYDiMhfgSsB/wQxBPiFb3kR8A+/7UtVtRaoFZEvgWnAex7G64kDBw7w9ttvc/PNNzer3KWXXsrbb79N165dPYrMGNMWVNbUUXCwisJS52ZfWFpFkW+5oLQhCTjrxRU1TZ5jZO+u7S5B9AR2+a3nAOc0OuYL4GrgSWA6ECsi8b7tvxWRR4FoYDJHJhYARGQWMAugd++W/8NpCQcOHOC55547KkHU1tYSFnbsP/65c+d6HZoxxgOqSll1nd+NvYqC0upD60W+JNBw0y+tqm3yPLFRYSTGRBIfE8GA5FjG9oskISaShNgI5zMmkoQYZ7lzpDe38kA/xXQX8IyIfB9YCuQCdao6T0TOBlYABcBKoK5xYVV9CXgJYPTo0W1yarx7772X7du3M3z4cMLDw4mKiqJbt25s3bqVjIwMrrrqKnbt2kVlZSW33XYbs2bNAg4PHVJaWsoll1zC+PHjWbFiBT179uSf//wnnTp1CvCVGRNcVJXiihrySqrIK6kkr6SS/IOHl/NKDieEypr6Js/RLTr80M39zNSuh27whz8jSYiNJL5zBFHhoa18hUfzMkHkAr381lN92w5R1d04NQhEJAa4RlUP+PY9CDzo2/c2kHEqwfzuX5vYvLtluzCG9Ijjt98647jHPPzww2zcuJENGzawePFiLrvsMjZu3HjocdTZs2fTvXt3KioqOPvss7nmmmuIj48/4hyZmZm88847vPzyy3z3u9/l73//OzNnzmzRazEmWKkqpVW15JVUkV9SSd7BykNJIL8hGfi2VdcefePv0imc5LhIkmKjSEvo3OTNPjE2ku6dIwgPbV8PjnqZIFYD6SKShpMYrgNm+B8gIgnAPlWtB+4DZvu2hwJdVbVIRIYBw4B5HsbaasaMGXPEuwpPPfUUH3zwAQC7du0iMzPzqASRlpbG8OHDARg1ahQ7duxotXiNac8qquvI993c95ZUOgmgxC8B+GoA5dVHNVAQExlGUlwkybFRjOrdjeS4KJLiokiOiyQ5Lork2CiS4iLbxDd9r3iWIFS1VkRuBT4GQoHZqrpJRB4A1qjqh8Ak4CERUZwmplt8xcOBZb5HN0uAmb4O65N2om/6raVz586HlhcvXsyCBQtYuXIl0dHRTJo0qcl3GSIjIw8th4aGUlFR0SqxGtNW1dUrRaXOTX9vsXPT31Nc6UsCh5t9SiqPvm1EhoWQ0sW5wZ/RI44pg5IO3fSTYp0EkBQXRYxH7frtiad/Aqo6F5jbaNtv/JbnAHOaKFeJ8yRTuxcbG8vBgweb3FdcXEy3bt2Ijo5m69atrFq1qpWjM6btqaypI89349/r99mQBPKKnW/+tfVHdjuGhQhJsc7NvV9iDGP7xfu+8ft964+LIi4qzN4bcslSpMfi4+MZN24cQ4cOpVOnTiQnJx/aN23aNF544QUGDx7MwIEDOffccwMYqTHeaujkbfimn9coATR8Hig/+lHOzhGhJHeJ4rQuUZzbL56UOGc5OS6KlC7OT3znSEJD7MbfkkS1TT7802yjR4/WxhMGbdmyhcGDBwcootYXbNdr2hZVJf9gFVn5pWTll7L7QMVRCaCqiU7ehJgI5ybv+4bvf+NvWI6NskEovSIia1V1dFP7rAZhjGmW+nol90DFoUSQmX+QTN/yQb82/4jQEJK7RJISF8WZPbtw0ZDko278SbFRRIS1ryd7goklCGNMk2rr6tm5r/xwIsg7SFZBKdvzy6ioOfzUT0JMBP2TYrhyeA/Sk2LpnxRD/6QYkmIjra2/nbMEYUyQq6yp4+vCMl9toJTtvlrB14Vl1NQdboLu0SWK/smxjBkTT3qykwT6J8bQrXNEAKM3XrIEYUyQKKuqZXtBKZl5pWT5PrcXlLKzqIyGB4JCBHp3j6Z/UgxTBiWT7qsN9EuKscc+g5D9jRvTwZRX15KRV0rG3oNsyzt4qIko98Dh92fCQ4W0hM4MPi2Wb53Vg/5JMaQnxZCW0LlDv/hlmscShDHtVG1dPTuKyti69yDb9h5k696DZOQd5Jt95TQ8nBgVHkL/pBjO7tuNGcm96ZcYQ3pyDL27R7e7YR9M67ME4bGTHe4b4IknnmDWrFlER0d7EJlpL1SVPcWVRySBrXsPsj2/lOo657HR0BCnRjC0RxeuGZnKwJRYBibH0rt7NCH2boA5SZYgPHas4b7deOKJJ5g5c6YliCBSXF7D1r0lh5LANl8zkf/jo6d1iWJgSiznD0hgYHIsA1Ni6ZcYY01DpsVZgvCY/3DfF154IUlJSbz33ntUVVUxffp0fve731FWVsZ3v/tdcnJyqKur4/777ycvL4/du3czefJkEhISWLRoUaAvxbSgypo6svJLDyWAbb5ksLfk8FhcsVFhDEqJ5crhPRiYEseglFgGJMXSJdpeGjOtI3gSxH/vhb1ftew5U86ESx4+7iH+w33PmzePOXPm8Pnnn6OqXHHFFSxdupSCggJ69OjBf/7zH8AZo6lLly489thjLFq0iISEhJaN27SqnP3lbMwtZtveUrbllbB170F2FB5+cigi1OknGNsvngEpTo1gUEosKXFR9h6BCajgSRBtwLx585g3bx4jRowAoLS0lMzMTCZMmMCdd97JPffcw+WXX86ECRMCHKk5FSWVNazcXsTyzEKWZRawo6gcAPE9QjowOZbLzzyNgSlxDEyJoW98Z8Ksw9i0QcGTIE7wTb81qCr33XcfP/nJT47at27dOubOncuvf/1rpk6dym9+85smzmDaotq6er7IOcCyzEKWZRayYdcB6uqV6IhQzj09nhvH9mVk726kJ8cQHRE8/+VM+2f/Wj3mP9z3xRdfzP3338/3vvc9YmJiyM3NJTw8nNraWrp3787MmTPp2rUrr7zyyhFlrYmpbVFVdhaVsyyrkGUZBazcXsTBqlpEYFjPLvx0Yj/Gpycwsnc3G2fItGuWIDzmP9z3JZdcwowZMzjvvPMAiImJ4c033yQrK4u7776bkJAQwsPDef755wGYNWsW06ZNo0ePHtZJHWDF5TWs2F7I0sxClmcVsGuf89JZz66duPys05iQnsjYfvF0jbZhJ0zHYcN9dyDBdr1eqqmrZ/03B1iWWcCyzEK+zDlAvTrTUJ7XL57z0xMYn55I3/ho60g27ZoN923MCagq2YVlLMsoYHlWISu3F1FWXUeIwPBeXfnZlHQmpCdwVq+u9gayCRqWIEzQ2ldWzadZhYeeNtpd7LyD0Cc+mukjezK+fyLn9YunSyd778AEpw6fIFQ1KJoAOkpToZeqautYu3O/LyEUsnF3MaoQFxXGuP4J3DIlgQn9E+kdb2+uGwMdPEFERUVRVFREfHx8h04SqkpRURFRUVGBDqVN2llUxsvLsnl/XS7l1XWEhQgje3fjjgsGMCE9gWGpXW0u42DS8GVKFXC5fOgL2HGWASQUQsJ8P6HOyy/tWIdOEKmpqeTk5FBQUBDoUDwXFRVFampqoMNoU77KKeaFpdv571d7CAsJ4crhPbj4jBTO7RcfXHMb1NVCbSXUVvk+G5Yr/LY1+qypbHr7Mcv5lv3Lqf/803413CNqu41qviez76jac1M38QDVsI9IGL6k0eRn4/0u1sXvHN3TYPwdLR5+h/5fEh4eTlpaWqDDMK1IVVmWWciLS7fzaVYRsZFhzDq/Hz8c15ekuDZew6qvd26+VaVQXQrVZX4/pS6Wj7GudSf+3ccTEg5hURAWefRneCeIjIXOiUfvl0aDBx7r2/RR2+Uk9jU6TuTwtoblQ8cfb5lG2+XY52u8DE5SrK+F+jrfp/9P4211zt/N8fbX1UBNxfGPqa+FlGGWIIw5ltq6ev7z1R5eXJLN5j0lJMdF8r+XDuL6Mb2JjWqlTmZVqCyGsgIozYPSfN9yPlSVuLi5l+H6m66EQEQMRHT2/cQ4PzHJR24L7+T8HLp5Rx3/Zn/EcZEQGgmhdpsIVvY3b9q18upa/rYmh5eXZZOzv4J+iZ3547eHceXwHkSGtcDw16pQsf/wjb4sH0oLfJ95fsu+z7rqo88hIRDVpdENvTN06dXoBt/5GOtNLIdFtfv2bdP2WYIw7dK+smpeX7GDN1buYH95DaP6dOO33zqDqYOSTjxBTsNN/9ANP7+Jm7/ft//6mqPPIaFOs0pMkvOTOBhiEqGzb71zovNtPiYJOnWHEHt3wrQ/liBMu7JrXzmvLMvm3TW7qKyp54LBSdw0sR+j+3Y/+uDaKsjbCLnrYPd6yNvkfOsvK3DabRsLCfPd4H03+uQzjrzRNySEzknQqZvd9E2HZwnCtAubdhfz4pJs/vPVHkIErhrek1nnn056cqxzQH0dFGb4ksE6yF3rJISGJp/oBDhtGKQMbfQtP+nweqdu1mxjjB9LEKbNUlVWbi/i+SXbWZZZSExkGD8an8YPxvbhtPo82D0PNvhqB7s3QE2ZUzAiFnoMh3N/Cj1GQs+RTnu/3fyNaRZPE4SITAOeBEKBV1T14Ub7+wCzgURgHzBTVXN8+/4IXAaEAPOB29ReFw4KdfXKfzc6TyR9lVvMwM7lPDe6hCmxuUTlvwgvroeKfc7BoZFOzWDETCcR9BgJ8f2t+ceYFuBZghCRUOBZ4EIgB1gtIh+q6ma/wx4B3lDV10VkCvAQcIOIjAXGAcN8xy0HJgKLvYrXBF5lTR3/XLWZz5YvIKV0M7+M2smorl8TXZkPG3E6hpMGw6DLDieD5DMg1MZKMsYLXtYgxgBZqpoNICJ/Ba4E/BPEEOAXvuVFwD98ywpEARE4b6GEA3kexmoCoboM9nxJxY7VfLNxOZ0KvuBa9nItQDhol35Iz4mHm4lShkGEjZNkTGvxMkH0BHb5recA5zQ65gvgapxmqOlArIjEq+pKEVkE7MFJEM+o6pbGv0BEZgGzAHr37t3yV2BaVn0dbPkXZM2H3PVowRZE6+kExGp39kQPRgbNJHXoeKTHcKRTt0BHbExQC3Qn9V3AMyLyfWApkAvUiUh/YDDQMLjQfBGZoKrL/Aur6kvAS+BMGNRqUZvmqauBr/4Gyx6Foixqo7qRGZrOgtqr+Kr+dE4bMpbrp57NqJS4QEdqjPHjZYLIBXr5raf6th2iqrtxahCISAxwjaoeEJEfA6tUtdS377/AecARCcK0cbVVsOFtWP44HNiJJg/l7d6/59cZaXSKCOf6c3rz2/Fp9OzaKdCRGmOa4GWCWA2ki0gaTmK4Dpjhf4CIJAD7VLUeuA/niSaAb4Afi8hDOE1ME4EnPIzVtKSaClj3Bnz6JJTkOn0Il/yBV/YO4MH/buX7Y/ty+wXpNn+zMW2cZwlCVWtF5FbgY5zHXGer6iYReQBYo6ofApOAh0REcZqYbvEVnwNMAb7C6bD+SFX/5VWspoVUlcKa2bDiaWe4it7nwRVPQ78prMzex8Mff8YlQ1P47beGdOj5OYzpKKSjvFowevRoXbNmTaDDCE6VxfDZS7DqWWeMo9Mnwfl3Q9/xAOwpruDyp5bTNTqcf946PrjmYjCmjRORtao6uql99j/VnLzyfbDqOSc5VBVD+sVw/l3Qa8yhQ6pq6/jpm+uorKnjxRvOteRgTDti/1tN85XmO81Iq191hrcY/C2nxnDaWUcd+vt/b2bDrgM8972R9E+KDUCwxpiTZQnCuFecCyuegrWvOYPgDb0GJtzpvN3chDlrc3hz1Tf85PzTufTM01o3VmPMKbMEYU5s/w5Y/gRseMuZUnHYdc70hgn9j1lkY24xv/rgK847PZ67Lx7YerEaY1qMJQhzbIVZzsttX77rTIw+YiaMux269Tlusf1l1dz05lq6d47g6RkjCAu1gfOMaY8sQZij5W2GZY/Apg+c0VLP+QmM/RnE9Thh0bp65bZ3N5BfUsV7N51HQkxkKwRsjPGCJQhz2O4NsPRPsPXfztzHY38O593iTKbj0pMLMliaUcCD04cyvFdXD4M1xnjNEoSBXZ87iSFzHkR2gYn3wDk3QXQT03gex4LNeTz1SRbfGZXKjDE2eKIx7Z0liGClCjuWO4nh6yXQqTtMuR/G/BiiujT7dDsKy7jjvQ0M7RnH768aam9KG9MBWIIIRuX74N0bYOdyZz7mi/4vjPoBRMac3Omqa/nJX9YSGiI8/71RRIWHtnDAxphAsAQRjJY/Bjs/hWl/gFE3QvjJj6aqqtz3/ldk5B/k9R+MoVd3m9DHmI7CEkSwKdkNn78MZ10H5950yqd7bcUO/rlhN3ddNIDzByS2QIDGmLbCHlAPNkv/5MzsNuneUz7V6h37ePA/W7hgcDI3Tzr2S3PGmPbJEkQw2ZftzNMw6kbo1veUTpVfUsnNb62jV/doHrv2LEJCrFPamI7GmpiCyeKHISTcGVjvFNTU1XPzW+sorazlzR+dQ1xUeAsFaIxpS6wGESzyNsOX7zlvRcemnNKpHvzPFtbs3M8fvj2MgSk2QqsxHZUliGCx6EGIjIVxt53Saf65IZfXVuzgh+PSuOKsEw+9YYxpvyxBBIOctc7wGWN/3uy3o/1t2VPCPX//kjFp3bnv0kEtGKAxpi2yBBEMFv4OohNO6bHW4ooabnpzLXFR4TwzYwThNkKrMR2e/S/v6LIXO0NpTLjTaWI6CfX1yi/e3UDu/gqenzmSpNiolo3RGNMmWYLoyFRh4e8hrieM/uFJn+aZRVks3JrP/ZcPYVSfk2+iMsa0L5YgOrJt/4XcNc7orOEn961/0bZ8Hl+QwfQRPfmf844/UZAxpmOxBNFR1dfDJ7+H7v1g+IyTOsU3ReXc/tcNDEqJ4/9NP9NGaDUmyNiLch3Vxr9D/ma45lUIbf6LbBXVddz05lpUlRdmjqRThI3QakywsQTREdXVOO89JA+FM65udnFV5Vf/+IrNe0qY/f3R9Inv7EGQxpi2zhJER7T+Tdj/NVz/LoQ0vxXxzc++4f11udw2NZ0pg5I9CNAY0x5YH0RHU1MBS/4Ivc6BARc3u/janft54F+bmDwwkdumpnsQoDGmvbAaREez+lU4uBuueRma2alccLCKm99ay2ldOvHEtSNshFZjgpwliI6ksgSWPQr9pkDf8c0qWltXz61vr+NAeQ0f3DyGLtE2Qqsxwc7TJiYRmSYi20QkS0SOmqFGRPqIyEIR+VJEFotIqm/7ZBHZ4PdTKSJXeRlrh7DqeajYB1Pub3bRP3y0lc++3sdDV5/JkB5xHgRnjGlvPEsQIhIKPAtcAgwBrheRIY0OewR4Q1WHAQ8ADwGo6iJVHa6qw4EpQDkwz6tYO4SyIljxNAz+FvQc2ayi//5yNy8v+5r/Oa8PV49M9ShAY0x742UNYgyQparZqloN/BW4stExQ4BPfMuLmtgP8G3gv6pa7lmkHcGnj0N1KUz+dbOKZeQd5JdzvmRUn278+rLG+dsYE8y8TBA9gV1+6zm+bf6+ABoe1J8OxIpIfKNjrgPeaeoXiMgsEVkjImsKCgpaIOR2qmQ3fP4ynHUdJLkfhruksoab/rKW6IgwnvveSCLC7KE2Y8xhgb4j3AVMFJH1wEQgF6hr2CkipwFnAh83VVhVX1LV0ao6OjExsTXibZuW/gnq62DSUd08x6Sq3PXeF+zcV84zM0aQHGcjtBpjjuQqQYjI+yJymYg0J6HkAr381lN92w5R1d2qerWqjgB+5dt2wO+Q7wIfqGpNM35vcNmXDevegFE3Qre+ros9v2Q78zbncd8lgzj39MaVNmOMcV+DeA6YAWSKyMMiMtBFmdVAuoikiUgETlPRh/4HiEiCX9K5D5jd6BzXc4zmJeOz+GEICYfz73ZdZGNuMY98vI3Lh53Gj8aneRicMaY9c5UgVHWBqn4PGAnsABaIyAoR+YGINPnAvKrWArfiNA9tAd5T1U0i8oCIXOE7bBKwTUQygGTgwYbyItIXpway5CSuKzjkbYYv34NzZkFsiutij8zbRlyncB662kZoNcYcm+sX5XydxzOBG4D1wFvAeOBGnBv9UVR1LjC30bbf+C3PAeYco+wOju7UNv4WPejMEjfudtdF1u7cz+JtBdwzbRCxUfYynDHm2FwlCBH5ABgI/AX4lqru8e16V0TWeBWcOY6ctbD13zD5VxDtfpa3x+ZvIyEmghvH2uQ/xpjjc1uDeEpVFzW1Q1VHt2A8xq1PHoDoBDj3p66LrMou4tOsIn592WCiI2yUFWPM8bntpB4iIl0bVkSkm4jc7FFM5kSyl0D2Yphwp9PE5IKq8ti8DJLjIpl5rtUejDEn5jZB/Nj/8VNV3Q/82JuQzHGpOlOJxvWE0T90XWx5ViGf79jHrZP7ExVus8MZY07MbYIIFb/HXXzjLEV4E5I5rm3/hZzVMPEeCHf3cpuq8si8DHp27cR3z+514gLGGIP7BPERTof0VBGZivNuwkfehWWaVF/v1B6694PhM1wX+2RrPl/sOsDPpvQnMsxqD8YYd9z2VN4D/ARo6BGdD7ziSUTm2Db+HfI3wzWvQqi7R1RVlcfmZ9C7ezTXjLKRWo0x7rlKEKpaDzzv+zGBUFfjvPeQPBTOuPrEx/t8vGkvm3aX8Oh3ziI8NNBDbxlj2hO370Gk48zVMAQ41PCtqqd7FJdpbP2bsP9ruP5dCHF3o6+vVx6fn8npiZ25aoS9c2iMaR63Xyn/jFN7qAUmA28Ab3oVlGmkpgKW/BFSx8CAi10X+/dXe9iWd5A7LhhAqM0vbYxpJrcJopOqLgREVXeq6v8BLvMuLHOE1a/Cwd0w9Tfgcuyk2rp6npifwaCUWC478zSPAzTGdERuO6mrfKOuZorIrTjDdsd4F5Y5pLIElj0Kp0+GtAmui/1jw26yC8t4YeYoQqz2YIw5CW5rELcB0cDPgVE4g/bd6FVQxs+q56FiH0y933WRmrp6nlqYydCecVx8RrKHwRljOrIT1iB8L8Vdq6p3AaXADzyPyjjK98GKp2Hwt6DnKNfF5qzN4Zt95cz+/mgbztsYc9JOWINQ1TqcYb1Na1v+OFSXwuRfuy5SVVvH0wszGd6rK5MHJnkYnDGmo3PbB7FeRD4E/gaUNWxU1fc9icpAyR74/CU46zpIGuS62Lurd7G7uJI/fvssqz0YY06J2wQRBRQBU/y2KWAJwitL/wj1dTDpXtdFKmvqeOaTLMakdWdcf5tn2hhzaty+SW39Dq1pXzasewNGfR+69XVd7M1VO8k/WMXT14+w2oMx5pS5fZP6zzg1hiOoqvvxpo17ix+GkHA4/27XRcqqanl+8XbG90/gnNOt9mCMOXVum5j+7bccBUwHdrd8OIa8zfDlezDu5xCb4rrY6yt3UFRWzS8uGuBdbMaYoOK2ienv/usi8g6w3JOIgt2iB51Z4sbd7rrIwcoaXlqazeSBiYzs3c3D4IwxweRkh/dMB+wZypaWsxa2/hvG/gyiu7suNnv5Dg6U1/CLCwd6GJwxJti47YM4yJF9EHtx5ogwLemTByA6Hs796YmP9TlQXs0ry7K5+Ixkzkzt4mFwxphg47aJKdbrQIJe9hLIXgwX/z+nicmll5dlU1pdyx0XWt+DMaZluWpiEpHpItLFb72riFzlXVhBRtWZSjSuJ4z+ketiRaVV/PnTHVx25mkMSonzMEBjTDBy2wfxW1UtblhR1QPAb70JKQhlfAQ5q2HiPRAedeLjfV5cmk1lTR23X2C1B2NMy3ObIJo6zu0jsuZ46uth4e+hez8YPsN1sfySSt5YuYOrhvekf5KNvG6MaXluE8QaEXlMRPr5fh4D1noZWNDY9D7kb4LJ/wuh4a6LPbd4OzV1ys+npnsYnDEmmLlNED8DqoF3gb8ClcAtXgUVNOpq4JP/C8lD4YyrXRfbfaCCtz/7hu+MSqVvQmcPAzTGBDO3TzGVAe5HjfMRkWnAk0Ao8IqqPtxofx9gNpAI7ANmqmqOb19v4BWgF84jtpeq6o7mxtCmrX8T9n8N178LIe5fSXlmURaKcuuU/h4GZ4wJdm6fYpovIl391ruJyMcnKBMKPAtcAgwBrheRIY0OewR4Q1WHAQ8AD/ntewP4k6oOBsYA+W5ibTdqq2DpI5B6Ngy42HWxXfvKeW/1Lq47uzep3aI9DNAYE+zcfm1N8D25BICq7ufEb1KPAbJUNVtVq3Gapq5sdMwQ4BPf8qKG/b5EEqaq832/r1RVy13G2j6s/wuU5MCk+6AZI68+tTCTkBDhlslWezDGeMttgqj3NfkAICJ9aWJ010Z6Arv81nN82/x9ATQ0vk8HYkUkHhgAHBCR90VkvYj8yVcjOYKIzBKRNSKypqCgwOWltAG1VbDsMeh1DvSbcuLjfbILSnl/fS4zz+lDShf3j8MaY8zJcJsgfgUsF5G/iMibwBLgvhb4/fF9ou0AABJhSURBVHcBE0VkPTARyAXqcPpGJvj2nw2cDny/cWFVfUlVR6vq6MTExBYIp5WsewNKcptde3hyYSYRoSH8dFI/D4MzxhiHqwShqh8Bo4FtwDvAnUDFCYrl4nQwN0j1bfM/725VvVpVR+AkoYaX8HKADb7mqVrgH8BIN7G2eTWVvtrDuXD6JNfFMvIO8uEXu7lxbF8SYyM9C88YYxq4Hazv/wNuw7nJbwDOBVZy5BSkja0G0kUkDScxXAcc8SaYiCQA+1S1HqdGMtuvbFcRSVTVAt/vWeP2otq09X+Bg7th+vPNqj08sSCDzhFh/OT80z0MzhhjDnPbxHQbTlPPTlWdDIwADhyvgO+b/63Ax8AW4D1V3SQiD4jIFb7DJgHbRCQDSAYe9JWtw2leWigiXwECvNycC2uTaiph2aPQ+zxIm+i62Kbdxcz9ai8/HNeXbp0jPAzQGGMOcztcRqWqVooIIhKpqltF5ISTD6jqXGBuo22/8VueA8w5Rtn5wDCX8bUP616Hg3tg+ovNqj08Pj+TuKgwfjTBag/GmNbjNkHk+N6D+AcwX0T2Azu9C6sDauh76DMO0s53XeyLXQdYsCWPOy8cQJdO7ofiMMaYU+X2TerpvsX/IyKLgC7AR55F1RGtfQ1K98I1Lzer9vDo/Ay6RYfzg/Fp3sVmjDFNaPaIrKq6xItAOrSaClj+GPQZ36zaw5od+1iaUcC9lwwiJtIGzzXGtC6767SGta9BaR5c82qzij06L4OEmEj+57w+3sRljDHH4X6EOHNyaipg+ePQdwKkTXBdbMX2QlZmF3HzpH5ER1geN8a0PrvzeG3Nn53aw7f/7LqIqvLYvAxS4qKYcU7vExcwxhgPWA3CS9XlTu0h7XzoO851saWZhazZuZ9bpvQnKvyoIaiMMaZVWA3CS2tmQ1k+THrddRFV5dF52+jZtRPXju514gLGGOMRq0F4pbocPn3CeWO6z1jXxRZsyefLnGJ+PrU/EWH212OMCRy7A3llzatQVuCM2OpSfb3y2PwM+sZHc/XIVA+DM8aYE7ME4YXqMlj+hDNaa5/zXBf7aNNetuwp4bYL0gkPtb8aY0xg2V3IC6tfhfLCZtUe6uqVx+dn0D8phivOajyvkjHGtD5LEC2tugw+fRJOnwy9z3Vd7F9f7CYzv5TbL0gnNMT9UBzGGOMVSxAtbfUrTu1h8v+6LlJbV8+TCzMZlBLLpUNP8zA4Y4xxzxJES6oqdWoP/aZCrzGui72/PpevC8u448IBhFjtwRjTRliCaEmrX4byomb1PVTX1vPUwkzO7NmFi4YkexicMcY0jyWIllJVCp8+Bf0vgF5nuy72t7W7yNlfwS8uGoA0YxhwY4zxmiWIlvL5S1Cxr1m1h8qaOp75JIuRvbsyaUCih8EZY0zzWYJoCVUHYcVT0P9CSB3tutg7n3/DnuJK7rxooNUejDFtjiWIlvD5S1Cxv1m1h/LqWp5dlMU5ad0Z2y/ew+CMMebkWII4VZUlsOJpSL8YUke5LvbnT3dQWFrNL6cNstqDMaZNsgRxqg7VHu5xXeRAeTUvLNnOBYOTGNWnm4fBGWPMybMEcSoaag8DpkFP97WHF5ZkU1pVy10XD/QwOGOMOTWWIE7FZy9C5QGY6L72kF9SyWsrvubKs3owKCXOw+CMMebUWII4WZXFsPIZGHAJ9BzputjTn2RRW6fcceEAD4MzxphTZwniZDXUHibd67rIN0XlvPP5N1x7di/6xHf2MDhjjDl1liBORsUBp/Yw8FLoMdx1sccXZBAWKvx8arqHwRljTMuwBHEyPnvRaWJqRu1h694S/rEhlxvH9iU5LsrD4IwxpmVYgmiuigOw8lkYdDmcdpbrYo/OyyAmIoyfTuznYXDGGNNyPE0QIjJNRLaJSJaIHPV1W0T6iMhCEflSRBaLSKrfvjoR2eD7+dDLOJvlsxegqrhZTy6t+2Y/8zfnMev80+kaHeFhcMYY03LCvDqxiIQCzwIXAjnAahH5UFU3+x32CPCGqr4uIlOAh4AbfPsqVNV9A39rqDgAK5/z1R6GuSqiqvzpo20kxETww/FpHgdojDEtx8saxBggS1WzVbUa+CtwZaNjhgCf+JYXNbG/bVn1nFN7aEbfw/KsQlZmF3HL5P50jvQsHxtjTIvzMkH0BHb5ref4tvn7ArjatzwdiBWRhpHrokRkjYisEpGrmvoFIjLLd8yagoKCloz9aBX7YdXzMPhbkHKmqyKqyp8+3kbPrp2YcU5vb+MzxpgWFuhO6ruAiSKyHpgI5AJ1vn19VHU0MAN4QkSO6t1V1ZdUdbSqjk5M9Hg+hZXPQVUJTHRfe/h4016+zCnm9gvSiQwL9TA4Y4xpeV62eeQCvfzWU33bDlHV3fhqECISA1yjqgd8+3J9n9kishgYAWz3MN5jK9/nqz1cASlDXRWpq1cemZdBv8TOTB/RuOJkjDFtn5c1iNVAuoikiUgEcB1wxNNIIpIgIg0x3AfM9m3vJiKRDccA4wD/zu3Wteo5qD7YrL6H99flkJVfyl0XDSQsNNAVNWOMaT7P7lyqWgvcCnwMbAHeU9VNIvKAiFzhO2wSsE1EMoBk4EHf9sHAGhH5Aqfz+uFGTz+1nvJ9sOoFGHIVJJ/hqkhVbR1PLMhkWGoXpg1N8ThAY4zxhqeP1ajqXGBuo22/8VueA8xpotwKwF1PsNdWPgPVpc167+Htz74h90AFD19zpk0GZIxpt6zt43jK9znDapxxFSQPcVWkrMqZSvS80+MZ3z/B4wCNMcY7liCOZ8XTUF3WrNrDnz/9msLSau6eNtBqD8aYds0SxLGUFTnTiZ4xHZIGuypyoLyaF5dmc8HgZEb2tqlEjTHtmyWIY1nZ/NrD80u2U1pVy902lagxpgOwBNGUskL47CUYeg0kDXJVJK+kktc+3cFVw3syMCXW4wCNMcZ7liCasuJpqCmHib90XeTpTzKpq1fuuMCmEjXGdAyWIBorK4TPX4Yzvw2J7pqKdhaV8dfPd3H9mN70jo/2OEBjjGkdliAa+/RJqK2A893XHh6f70wl+rMp/T0MzBhjWpclCH+lBbD6FRj6bUh011S0dW8J//xiN98fm0aSTSVqjOlALEH4W/Ek1FY2q+/hkY8ziIm0qUSNMR2PJYgGpQXw+Stw5ncgId1VkbU797NgSx43TexHl+hwjwM0xpjWZQmiwadPQF2V674HZzKgrSTERPCDcX29jc0YYwLAEgTAwTxY/SoMuxYS3HU0L8ssZFX2Pm6d3J/oCJtK1BjT8ViCAFjxFNRVw/l3uzrcfyrR620qUWNMB2UJwr/2EO+uo/mjjXv5KreYOy4cYFOJGmM6LGsbCe8E4293OqddqK2r55F520hPirGpRI0xHZoliKi45k0luj6X7QVlvDBzFKEhNpy3MabjsiamZqiqrePJBZmcldqFi89IDnQ4xhjjKUsQzfDWKmcq0bsvHmSTARljOjxLEC41TCU6tl8849NtKlFjTMdnCcKl2cu/pqis2iYDMsYEDUsQLuwvq+alpdlcNCSZETaVqDEmSFiCcOGFJdspra7lzous9mCMCR6WIE5gb3Elr63YwXSbStQYE2QsQZzA059kUq/KHRfaVKLGmOBiCeI4dhaV8e5qZyrRXt1tKlFjTHCxBHEcj/mmEr3VphI1xgQhSxDHsGVPCR9+sZsfjEsjKdamEjXGBB9LEMfwyMfbiI0M46bzbSpRY0xw8jRBiMg0EdkmIlkictSIeCLSR0QWisiXIrJYRFIb7Y8TkRwRecbLOBtbu3MfC7fm8xObStQYE8Q8SxAiEgo8C1wCDAGuF5EhjQ57BHhDVYcBDwAPNdr/e2CpVzE2RVX540fbSIiJtKlEjTFBzcsaxBggS1WzVbUa+CtwZaNjhgCf+JYX+e8XkVFAMjDPwxiPsjSzkM++3sfPp9pUosaY4OZlgugJ7PJbz/Ft8/cFcLVveToQKyLxIhICPArcdbxfICKzRGSNiKwpKCg45YDr65U/fbyV1G6duO5sm0rUGBPcAt1JfRcwUUTWAxOBXKAOuBmYq6o5xyusqi+p6mhVHZ2YmHjKwXy0aS8bc0u444IBRIQF+o/GGGMCy8s2lFygl996qm/bIaq6G18NQkRigGtU9YCInAdMEJGbgRggQkRKVdX91G/N1DCV6IDkGK6yqUSNMcbTBLEaSBeRNJzEcB0ww/8AEUkA9qlqPXAfMBtAVb/nd8z3gdFeJgeA99flkl1Qxos32FSixhgDHjYxqWotcCvwMbAFeE9VN4nIAyJyhe+wScA2EcnA6ZB+0Kt4jqeypo4nFmQwvFdXLhpiU4kaYwx4W4NAVecCcxtt+43f8hxgzgnO8RrwmgfhHfLWZ9+wu7iSR75zlk0laowxPkHfE1taVctzi7IY1z+esf1tKlFjjGkQ9A/6l1fVcnbf7tw0yYbUMMYYf0GfIJLionjhhlGBDsMYY9qcoG9iMsYY0zRLEMYYY5pkCcIYY0yTLEEYY4xpkiUIY4wxTbIEYYwxpkmWIIwxxjTJEoQxxpgmiaoGOoYWISIFwM5TOEUCUNhC4bQXwXbNwXa9YNccLE7lmvuoapMT6nSYBHGqRGSNqo4OdBytKdiuOdiuF+yag4VX12xNTMYYY5pkCcIYY0yTLEEc9lKgAwiAYLvmYLtesGsOFp5cs/VBGGOMaZLVIIwxxjTJEoQxxpgmBX2CEJFpIrJNRLJE5N5Ax+M1EeklIotEZLOIbBKR2wIdU2sRkVARWS8i/w50LK1BRLqKyBwR2SoiW0TkvEDH5DURucP373qjiLwjIlGBjqmlichsEckXkY1+27qLyHwRyfR9dmuJ3xXUCUJEQoFngUuAIcD1IjIksFF5rha4U1WHAOcCtwTBNTe4DdgS6CBa0ZPAR6o6CDiLDn7tItIT+DkwWlWHAqHAdYGNyhOvAdMabbsXWKiq6cBC3/opC+oEAYwBslQ1W1Wrgb8CVwY4Jk+p6h5VXedbPohz0+gZ2Ki8JyKpwGXAK4GOpTWISBfgfOBVAFWtVtUDgY2qVYQBnUQkDIgGdgc4nhanqkuBfY02Xwm87lt+HbiqJX5XsCeInsAuv/UcguBm2UBE+gIjgM8CG0mreAL4JVAf6EBaSRpQAPzZ16z2ioh0DnRQXlLVXOAR4BtgD1CsqvMCG1WrSVbVPb7lvUByS5w02BNE0BKRGODvwO2qWhLoeLwkIpcD+aq6NtCxtKIwYCTwvKqOAMpooWaHtsrX7n4lTnLsAXQWkZmBjar1qfPuQou8vxDsCSIX6OW3nurb1qGJSDhOcnhLVd8PdDytYBxwhYjswGlGnCIibwY2JM/lADmq2lA7nIOTMDqyC4CvVbVAVWuA94GxAY6pteSJyGkAvs/8ljhpsCeI1UC6iKSJSAROh9aHAY7JUyIiOO3SW1T1sUDH0xpU9T5VTVXVvjh/x5+oaof+Zqmqe4FdIjLQt2kqsDmAIbWGb4BzRSTa9+98Kh28Y97Ph8CNvuUbgX+2xEnDWuIk7ZWq1orIrcDHOE88zFbVTQEOy2vjgBuAr0Rkg2/b/6rq3ADGZLzxM+At35efbOAHAY7HU6r6mYjMAdbhPK23ng447IaIvANMAhJEJAf4LfAw8J6I/Ahn2oPvtsjvsqE2jDHGNCXYm5iMMcYcgyUIY4wxTbIEYYwxpkmWIIwxxjTJEoQxxpgmWYIwpg0QkUnBMsqsaT8sQRhjjGmSJQhjmkFEZorI5yKyQURe9M0xUSoij/vmIVgoIom+Y4eLyCoR+VJEPmgYo19E+ovIAhH5QkTWiUg/3+lj/OZveMv3NrAxAWMJwhiXRGQwcC0wTlWHA3XA94DOwBpVPQNYgvNmK8AbwD2qOgz4ym/7W8CzqnoWzlhBDaNwjgBux5mb5HSct96NCZigHmrDmGaaCowCVvu+3HfCGRStHnjXd8ybwPu++Ri6quoS3/bXgb+JSCzQU1U/AFDVSgDf+T5X1Rzf+gagL7Dc+8sypmmWIIxxT4DXVfW+IzaK3N/ouJMdv6bKb7kO+/9pAsyamIxxbyHwbRFJgkPzAPfB+X/0bd8xM4DlqloM7BeRCb7tNwBLfLP45YjIVb5zRIpIdKtehTEu2TcUY1xS1c0i8mtgnoiEADXALTiT8Yzx7cvH6acAZ9jlF3wJwH801RuAF0XkAd85vtOKl2GMazaaqzGnSERKVTUm0HEY09KsickYY0yTrAZhjDGmSVaDMMYY0yRLEMYYY5pkCcIYY0yTLEEYY4xpkiUIY4wxTfr/ASxKfC/F6f9MAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "An9tgvMyFW5F",
        "outputId": "7c14d2e6-321a-4e0d-bedc-d0de0efbd8cc"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.plot(history.history['loss'])\n",
        "plt.plot(history.history['val_loss'])\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'test'], loc='upper left')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxU9Znv8c9T1ftC0w0NyNYNiApqBLrFNcZEjaAZ1NEoOuaabdCM3jhjJledaCZxrhMnmeuYxTWJ2TQalzgykbhGoxGJNogLmyyiNAi0zdr03v3cP6q6KZqi1zpd3dXf9+tVrzrb79RTifS3zvmd8zvm7oiIiHQUSnYBIiIyMCkgREQkLgWEiIjEpYAQEZG4FBAiIhJXWrILSJSRI0d6aWlpsssQERlUli5d+rG7F8dblzIBUVpaSkVFRbLLEBEZVMzsg0Ot0ykmERGJSwEhIiJxKSBERCSulOmDiKepqYnKykrq6+uTXUrgsrKyGD9+POnp6ckuRURSREoHRGVlJfn5+ZSWlmJmyS4nMO5OdXU1lZWVTJo0KdnliEiKSOlTTPX19YwYMSKlwwHAzBgxYsSQOFISkf6T0gEBpHw4tBkq31NE+k/KB0RXmlta2bannrrG5mSXIiIyoAz5gDCDbXvq2VMfTEDs2rWLu+66q8ftzjnnHHbt2hVARSIi3TPkAyIcCpGVHmZfQ/8GRHNz55+3aNEihg8fHkhNIiLdkdJXMXVXbkaYXbVNuHvCz+XfcMMNrF+/nhkzZpCenk5WVhaFhYWsXr2a9957j/PPP59NmzZRX1/Ptddey4IFC4D9Q4fU1NQwd+5cTj31VBYvXsy4ceN48sknyc7OTmidIiIdDZmA+O7/rGDllj1x1zW3Og1NLWRnhAn1ICCmjx3Gv/7N0Z1uc9ttt/Huu++yfPlyXnrpJc4991zefffd9stR77//foqKiqirq+P444/nwgsvZMSIEQfsY+3atTz00EP89Kc/5eKLL+bxxx/n8ssv73adIiK9MWQCojPhaCa0tjqhcLBXA82ePfuAexV+9KMf8cQTTwCwadMm1q5de1BATJo0iRkzZgBQVlbGxo0bA61RRASGUEB09kvf3Vm9dS+5mWlMLMoJtI7c3Nz26Zdeeonnn3+e1157jZycHE4//fS49zJkZma2T4fDYerq6gKtUUQE1EkNRO4hyMkIUxtAR3V+fj579+6Nu2737t0UFhaSk5PD6tWrWbJkScI/X0Skt4bMEURXcjLS2F3XRFNLK+nhxOXmiBEjOOWUUzjmmGPIzs5m9OjR7evmzJnDPffcw7Rp0zjyyCM58cQTE/a5IiJ9Ze4e3M7N5gA/BMLAz9z9tkNsdyHwGHC8u1dEl90IfAVoAb7u7s909lnl5eXe8YFBq1atYtq0ad2qtbaxmXXba5hYlMPwnIxutRloevJ9RUQAzGypu5fHWxfYKSYzCwN3AnOB6cClZjY9znb5wLXAX2OWTQfmA0cDc4C7ovsLTFZ65Aqm2saWID9GRGTQCLIPYjawzt03uHsj8DBwXpzt/g34DyC2d/Y84GF3b3D394F10f0FJmRGdkaYWg25ISICBBsQ44BNMfOV0WXtzGwWMMHdn+pp22j7BWZWYWYVVVVVfS44NyNMXWMrra3BnXYTERksknYVk5mFgNuBb/R2H+5+n7uXu3t5cXFxn2vKyUjDcZ1mEhEh2KuYNgMTYubHR5e1yQeOAV6KDm8xBlhoZvO60TYQORmRbo7axmbysnSBl4gMbUEeQbwBTDWzSWaWQaTTeWHbSnff7e4j3b3U3UuBJcC86FVMC4H5ZpZpZpOAqcDrAdYKQFo4RFZamH06ghARCS4g3L0ZuAZ4BlgFPOLuK8zsluhRQmdtVwCPACuBp4Gr3b1f/mrnZEY6qhN1+W9vh/sGuOOOO6itrU1IHSIiPRVoH4S7L3L3I9x9irvfGl32bXdfGGfb09vugYjO3xptd6S7/zHIOmPlZKTR0uo0NLcmZH8KCBEZrHSivYPcaD/EvsZmstL7futF7HDfZ511FqNGjeKRRx6hoaGBCy64gO9+97vs27ePiy++mMrKSlpaWrj55pvZtm0bW7Zs4dOf/jQjR47kxRdf7HMtIiI9MXQC4o83wNZ3utwsA2dKYwvhkEFaFwEx5liYG/fm8Haxw30/++yzPPbYY7z++uu4O/PmzePll1+mqqqKsWPH8tRTkat9d+/eTUFBAbfffjsvvvgiI0eO7PbXFBFJFA3W14FhhM0CuRfi2Wef5dlnn2XmzJnMmjWL1atXs3btWo499liee+45rr/+el555RUKCgoS/tkiIj01dI4guvilH2vv3no+2l3PtMOGJXTgPnfnxhtv5Morrzxo3bJly1i0aBE33XQTZ5xxBt/+9rcT9rkiIr2hI4g4cjIiuZmIG+Zih/s+++yzuf/++6mpqQFg8+bNbN++nS1btpCTk8Pll1/ON7/5TZYtW3ZQWxGR/jZ0jiB6IDs9jJlR29hMQXZ6n/YVO9z33LlzueyyyzjppJMAyMvL44EHHmDdunV885vfJBQKkZ6ezt133w3AggULmDNnDmPHjlUntYj0u0CH++5PfR3uu6N12yO/8g8fldfn2vqLhvsWkZ5KynDfg11uZpi6phYN3CciQ5YC4hByMtJwd+qaNOyGiAxNKR8QvT2FlhNzw9xgkCqnCkVk4EjpgMjKyqK6urpXfzzTwyEy08LUNgz8Iwh3p7q6mqysrGSXIiIpJKWvYho/fjyVlZX09mFCO/c1Ut/UQl1VdoIrS7ysrCzGjx+f7DJEJIWkdECkp6czadKkXrd/6PUPufHJd/jTNz7F5OLBczWTiEgipPQppr4qLykEoOKDnUmuRESk/ykgOjGlOI+C7HSWKSBEZAhSQHQiFDLKSgp1BCEiQ1KgAWFmc8xsjZmtM7Mb4qy/yszeMbPlZvYXM5seXV5qZnXR5cvN7J4g6+xMWUkh67bXsKu2MVkliIgkRWABYWZh4E5gLjAduLQtAGL81t2PdfcZwPeB22PWrXf3GdHXVUHV2ZWyaD/EUh1FiMgQE+QRxGxgnbtvcPdG4GHgvNgN3H1PzGwuMODu9jpu/HDSQqaAEJEhJ8iAGAdsipmvjC47gJldbWbriRxBfD1m1SQze9PM/mxmn4z3AWa2wMwqzKyit/c6dCU7I8zR4wrUDyEiQ07SO6nd/U53nwJcD9wUXfwRMNHdZwLXAb81s2Fx2t7n7uXuXl5cXBxYjWUTC3lr0y4am1sD+wwRkYEmyIDYDEyImR8fXXYoDwPnA7h7g7tXR6eXAuuBIwKqs0vlpYU0NLeyYsvuZJUgItLvggyIN4CpZjbJzDKA+cDC2A3MbGrM7LnA2ujy4mgnN2Y2GZgKbAiw1k6Vq6NaRIagwALC3ZuBa4BngFXAI+6+wsxuMbN50c2uMbMVZracyKmkK6LLTwPeji5/DLjK3XcEVWtXRg3LYkJRtgJCRIaUQMdicvdFwKIOy74dM33tIdo9DjweZG09VTaxkFfXR0aGNbNklyMiErikd1IPFmWlRVTtbaByZ12ySxER6RcKiG7aP3Bf0s50iYj0KwVENx0xOp/8zDQqNqofQkSGBgVEN4VDxoyJw9VRLSJDhgKiB8pLilizbS976puSXYqISOAUED1QXlqIO7z54a5klyIiEjgFRA8cN2E4IYOlG9VRLSKpTwHRA3mZaUw7bJgG7hORIUEB0UPlJYUs37SL5hYN3CciqU0B0UNlpUXUNraweuveZJciIhIoBUQPtT1hrkL9ECKS4hQQPTRueDaHFWSxVFcyiUiKU0D0QllJoa5kEpGUp4DohbKSQrbsrmfLLg3cJyKpSwHRC+UlRQC63FVEUpoCohemHZZPTkaYZQoIEUlhCoheSAuHmDFhuIb+FpGUFmhAmNkcM1tjZuvM7IY4668ys3fMbLmZ/cXMpsesuzHabo2ZnR1knb1RVlLIqo/2sq+hOdmliIgEIrCAMLMwcCcwF5gOXBobAFG/dfdj3X0G8H3g9mjb6cB84GhgDnBXdH8DRllJIS2tzvJNutxVRFJTkEcQs4F17r7B3RuBh4HzYjdw9z0xs7mAR6fPAx529wZ3fx9YF93fgDGrpBAz9HwIEUlZaQHuexywKWa+Ejih40ZmdjVwHZABfCam7ZIObcfFabsAWAAwceLEhBTdXcOy0jlydL6uZBKRlJX0Tmp3v9PdpwDXAzf1sO197l7u7uXFxcXBFNiJWSWFvPnBTlpaveuNRUQGmSADYjMwIWZ+fHTZoTwMnN/LtklRXlLI3oZm3tumgftEJPUEGRBvAFPNbJKZZRDpdF4Yu4GZTY2ZPRdYG51eCMw3s0wzmwRMBV4PsNZeabthTv0QIpKKAgsId28GrgGeAVYBj7j7CjO7xczmRTe7xsxWmNlyIv0QV0TbrgAeAVYCTwNXu3tLULX21oSibIrzMxUQIpKSguykxt0XAYs6LPt2zPS1nbS9Fbg1uOr6zswom1ioG+ZEJCUlvZN6sCsvLWTTjjq276lPdikiIgmlgOijtgcI6TSTiKQaBUQfHT22gMy0kO6HEJGUo4Doo4y0EMeNH66AEJGUo4BIgLLSQlZs3k1904C70EpEpNcUEAlQXlJIc6vzlgbuE5EUooBIgFkTIx3VOs0kIqlEAZEAhbkZTCnO1ZVMIpJSFBAJUl5SxLIPd9KqgftEJEUoIBKkrLSQXbVNbPi4JtmliIgkhAIiQdpumKvYqNNMIpIaFBAJMnlkLkW5GeqHEJGUoYBIEDNj1sRCBYSIpAwFRAKVlRSy4eN9VNc0JLsUEZE+U0AkUHmpBu4TkdShgEigY8cVkBEOsfRDBYSIDH6BBoSZzTGzNWa2zsxuiLP+OjNbaWZvm9kLZlYSs67FzJZHXws7th2IstLDHDNuGEt1JZOIpIDAAsLMwsCdwFxgOnCpmU3vsNmbQLm7fwJ4DPh+zLo6d58Rfc1jkCgrKeTtzbtpaNbAfSIyuAV5BDEbWOfuG9y9EXgYOC92A3d/0d1ro7NLgPEB1tMvykqKaGxu5d3Nu5NdiohInwQZEOOATTHzldFlh/IV4I8x81lmVmFmS8zs/HgNzGxBdJuKqqqqvlecAHrCnIikigHRSW1mlwPlwA9iFpe4ezlwGXCHmU3p2M7d73P3cncvLy4u7qdqO1ecn0npiBzdUS0ig16QAbEZmBAzPz667ABmdibwLWCeu7ffQODum6PvG4CXgJkB1ppQs0oiN8y5a+A+ERm8ggyIN4CpZjbJzDKA+cABVyOZ2UzgXiLhsD1meaGZZUanRwKnACsDrDWhykuKqN7XyMbq2q43FhEZoAILCHdvBq4BngFWAY+4+wozu8XM2q5K+gGQBzza4XLWaUCFmb0FvAjc5u6DJyB0w5yIpIC0IHfu7ouARR2WfTtm+sxDtFsMHBtkbUE6vDiPYVlpLP1gBxeVDfoLs0RkiOrWEYSZXWtmwyzi52a2zMw+G3Rxg1UoZMwqKVRHtYgMat09xfRld98DfBYoBL4A3BZYVSmgvKSQtdtr2F3blOxSRER6pbsBYdH3c4DfuPuKmGUSR1lJEQDLNC6TiAxS3Q2IpWb2LJGAeMbM8oHW4Moa/I6bUEA4ZFR8sCPZpYiI9Ep3O6m/AswANrh7rZkVAV8KrqzBLycjjaPHDlM/hIgMWt09gjgJWOPuu6J3Pd8EaLChLpSVFPJW5S6aWnSwJSKDT3cD4m6g1syOA74BrAd+HVhVKaK8pIj6plZWbtmT7FJERHqsuwHR7JFxI84DfuLudwL5wZWVGtoG7qvQDXMiMgh1NyD2mtmNRC5vfcrMQkB6cGWlhjEFWYwbns1SdVSLyCDU3YC4BGggcj/EViID7/2g8yYCkWE3NHCfiAxG3QqIaCg8CBSY2eeAendXH0Q3lJcUsm1PA5U765JdiohIj3R3qI2LgdeBzwMXA381s4uCLCxVzNIDhERkkOrufRDfAo5vG5LbzIqB54k8R1o6cdSYYeRlprH0g52cP7OzB+qJiAws3e2DCMU+rwGo7kHbIS0cMmZOHK4rmURk0OnuH/mnzewZM/uimX0ReIoOw3jLoc2aWMiarXvYW6+B+0Rk8OhuJ/U3gfuAT0Rf97n79UEW1q+q1kBrcHc7l5cW0urw5oe7AvsMEZFE6/ZpInd/3N2vi76e6E4bM5tjZmvMbJ2Z3RBn/XVmttLM3jazF8ysJGbdFWa2Nvq6ort19ljVe3DPJ+Gl7wX2ETMnFhIydVSLyODSaUCY2V4z2xPntdfMOh0/wszCwJ3AXGA6cKmZTe+w2ZtAubt/gkiH9/ejbYuAfwVOAGYD/2pmhb35gl0aORWOvQhe/j6sXNj19r2Ql5nGUWOGKSBEZFDpNCDcPd/dh8V55bv7sC72PRtY5+4b3L0ReJjIUB2x+3/R3Wujs0uI3IAHcDbwnLvvcPedwHPAnJ5+uW4xg3Nvh3Fl8MRVsC2YR1+XlRTy5oc7adbAfSIySAR5JdI4YFPMfGV02aF8BfhjT9qa2QIzqzCziqqqqt5Xmp4FlzwAmXnw8KVQm/ihMcpLC9nX2MLqrXsTvm8RkSAMiEtVo0OIl9PD4Tvc/T53L3f38uLi4r4VMWwsXPwb2L0ZHvsytDT3bX8dtA3cpyfMichgEWRAbAYmxMyPjy47gJmdSeRGvHnu3tCTtgk38QQ49//Bhhfhhe8kdNfjhmczZliWHiAkIoNGkAHxBjDVzCaZWQYwHzigF9jMZgL3EgmH2BvxngE+a2aF0c7pz0aXBa/sCjj+q7D4x/D2IwnbrZlRVlKojmoRGTQCCwh3bwauIfKHfRXwiLuvMLNbzGxedLMfAHnAo2a23MwWRtvuAP6NSMi8AdwSXdY/zv4eTDwZFv5v2LI8YbstKylk8646PtqtgftEZOCzVBmGury83CsqKhK3w5oquO/0yPSClyCvj30cwNuVu5j3k1f5yWUz+dwnxvZ5fyIifWVmS929PN66AdFJPSDlFcP8B6D2Y3j0Cmjp+zAZ0w4bRnZ6WP0QIjIoKCA6M3YmzPsxfPAqPH1jn3eXHg5x3IQC9UOIyKCggOjKJy6Gk66BN34Ky/r+jKTykiJWfrSH2sbEXkYrIpJoCojuOPO7MPnT8IfrYNPrfdpVWWkhLa3O8k0auE9EBjYFRHeE0+Ci+6FgHPzuctjzUa93NWtC9Alz6ocQkQFOAdFdOUUw/yFoqImERFN9r3ZTkJPOEaPz9AAhERnwFBA9MXo6XHAPbK6Ap74BvbxEuKykiGUf7qS1NTUuMRaR1KSA6Knp8+C0/wPLH4DXf9qrXZSXFLK3vpm122sSXJyISOIoIHrj9BvhiLnw9A3w/is9bt42cF/FB/13c7iISE8pIHojFIK/vQ9GTIncRLfrwx41LxmRw8i8DHVUi8iApoDoraxhkU7rliZ4+DJorO26TVT7wH0a+ltEBjAFRF+MPBwu/DlsfRcWXtOjTuvykiI+qK6lam9D1xuLiCSBAqKvjvgsnHEzvPs4vPrDbjebFe2HWKp+CBEZoBQQiXDqdXD0BfD8d2Dt891qcsy4YWSkhTQuk4gMWAqIRDCD8+6E0UfD41+G6vVdNslMC3Pc+AL+tHo7e+v7PlKsiEiiKSASJSMX5j8IFo50Wjfs7bLJl0+ZxMbqWi6+d4keIiQiA44CIpEKS+Hzv4SP18ITV0Fra6ebzz32MO7/4vFs2lHLBXcuZuWWPf1SpohIdwQaEGY2x8zWmNk6M7shzvrTzGyZmTWb2UUd1rVEH0Pa/ijSQWHyp+DsW2H1H+Dl73e5+aeOKOaRK08C4PP3LObP71UFXaGISLcEFhBmFgbuBOYC04FLzWx6h80+BL4I/DbOLurcfUb0NS/O+oHrhKvguMvgpe/Bqj90ufn0scP476tPYeKIXL78yzd46PWe3XgnIhKEII8gZgPr3H2DuzcCDwPnxW7g7hvd/W2g83Mxg40ZfO6/YOwseOJK2L66yyZjCrJ49KqTOPXwkdz4+3f4/tOrNZifiCRVkAExDtgUM18ZXdZdWWZWYWZLzOz8eBuY2YLoNhVVVQPs1Ex6FlzyAKTnwMOXQl3Xl7PmZabx8yvKuXT2RO56aT3/+LvlNDS39EOxIiIHG8id1CXuXg5cBtxhZlM6buDu97l7ubuXFxcX93+FXSkYB5f8BnZtgse/Cq1d/7FPC4f49wuO4fo5R7HwrS184Wevs3NfYz8UKyJyoCADYjMwIWZ+fHRZt7j75uj7BuAlYGYii+s3E0+Ec34A656HF27pVhMz42unT+HHl85k+aZdXHj3Yj6o3hdwoSIiBwoyIN4ApprZJDPLAOYD3boaycwKzSwzOj0SOAVYGVilQSv/EpR/GV69A955rNvN/ua4sTz49yewo7aRv71rMcs0uJ+I9KPAAsLdm4FrgGeAVcAj7r7CzG4xs3kAZna8mVUCnwfuNbMV0ebTgAozewt4EbjN3QdvQADM+Q+YeBI8eQ189Fa3mx1fWsTvv3YyuZlpXHrfEp5+t/fPwxYR6QnzXj42c6ApLy/3ioqKZJfRuZrtcN/pYCFY8BLkjux20+qaBr766wqWb9rFt86ZxldOnYSZBVWpiAwRZrY02t97kIHcSZ168kZFrmzaVwWPfjHyLIluGpGXyUN/fyJzjh7D/31qFd9ZuIIWXQYrIgFSQPS3cbPgb34IG1+BZ77Vo6ZZ6WHuvGwWC06bzK9e+4Arf1NBbWNzQIWKyFCngEiG4+bDiVfD6/fC0l/2qGkoZPzLOdP4t/OO5k+rt3PJvUvYvrc+mDpFZEhTQCTLWbfA5E/D/1wLC7/erdFfY33hpFJ++r/KWbe9hgvuXMx723rWXkSkKwqIZAmnwWW/g1OuhWW/hrtPgQ8W92gXZ0wbzSNXnkRjSysX3r2Yxes+DqhYERmKFBDJlJYZOZL40h8j4zf94hx49iZo6v4po2PHF/DEP5zMYQVZXPGL13l8aWWABYvIUKKAGAhKToKrXo3cULf4x5FLYbcs73bz8YU5PHrVyRxfWsQ3Hn2LO55/j1S5fFlEkkcBMVBk5kVGgP27x6F+F/zsDPjz96Gle1cpFWSn88svzebCWeO54/m1/POjb9PYnFqD5IpI/1JADDRTz4SvLYbp58OLt8LPz4Kq97rVNCMtxH9+/hP805lH8PiySr74i9fZXafnXYtI7yggBqKcIrjo53DRL2Dn+3DvJ2HJPV0+whQiA/1de+ZU/t/nj+ONjTu46O7FVO6s7YeiRSTVKCAGsmP+Fv5hCUz6FDx9Pfx6Huzq3tPmLiwbz6++NJute+q54K7FvF25K+BiRSTVKCAGuvwxkcth/+ZHsOVNuOtkePNB6EYn9MmHj+T3XzuZjHCIS+5dwgurtvVDwSKSKhQQg4EZlF0BX3sVDvsEPPkP8PBlkcH/ujB1dD5PXH0yh4/K4+9/XcFvXtsYeLkikhoUEINJYSlc8Qf47K2w7gW460RY2fUjNkblZ/G7K0/kM0eN4uYnV3DrUyv1vGsR6ZICYrAJheDka+DKl6FgPDzyBfj9lVDXeR9DTkYa936hnCtOKuGnr7zP1b9dxp56XeEkIoemgBisRh0FX30BPnU9vPMo3H0yrP9Tp03CIeM7847m5s9N5+kVW/n0D17igSUf0Nyi+yVE5GCBBoSZzTGzNWa2zsxuiLP+NDNbZmbNZnZRh3VXmNna6OuKIOsctMLp8Ol/ga8+Bxm58JsL4Kl/hsZDP7/azPjKqZNYePWpTBmVx03//S7n/OgV/vxeVT8WLiKDQWBPlDOzMPAecBZQSeQZ1ZfGPjrUzEqBYcA/Awvd/bHo8iKgAigHHFgKlLn7IR/KPCieKBekpjp44RZYchcUTYYL7oUJsztt4u48s2Ir/75oNR/uqOX0I4v51jnTmDo6v5+KFpFkS9YT5WYD69x9g7s3Ag8D58Vu4O4b3f1toOM5jrOB59x9RzQUngPmBFjr4JeeDXO+B1f8T+RJdfefDc9/F5obD9nEzJhzzGE8d91pfOucaSz9YCdzfvgKN//3u1TXNPRj8SIyEAUZEOOATTHzldFlCWtrZgvMrMLMKqqqdIoEgEmnRYbqOO4y+Mvt8NPPwNZ3O22SmRbm70+bzEv/fDp/d8JEfvv6h5z+ny9x38vraWhu6afCRWSgGdSd1O5+n7uXu3t5cXFxsssZOLKGwfl3wvyHoGZrZHTYv/wXtHb+x35EXia3nHcMT1/7ScpLCvn3Ras56/aX+eM7H2l0WJEhKMiA2AxMiJkfH10WdFtpc9Q5kaE6jpwDz38HfjEXqtd32Wzq6Hx+8aXZ/PrLs8lKD/G1B5dxyb1LNFyHyBATZEC8AUw1s0lmlgHMB7q+qyviGeCzZlZoZoXAZ6PLpKdyR8LFv4EL7oPtq+GeU+GNn3VrqI7Tjihm0dc/ya0XHMP6qhrm/eRVrntkOR/truuHwkUk2QK7ignAzM4B7gDCwP3ufquZ3QJUuPtCMzseeAIoBOqBre5+dLTtl4F/ie7qVnf/RWefNeSvYuqO3Zvhyathw4sw5TNw0tVQehqkZXTZdE99E3e9uJ77//I+oRBcedoUrvzUZHIy0vqhcBEJSmdXMQUaEP1JAdFN7pEjiOe/A401kJEPU8+Co86Fw8+E7OGdNt+0o5bbnl7NU29/xOhhmXzz7KP425njCIWsf+oXkYRSQMjBmupgw0uw+il472nYVwWhNCg9FY76HBw5NzKUxyFUbNzBv/1hJW9V7uaYccO4+dzpnDB5RP/VLyIJoYCQzrW2QGUFrP4DrFkE1esiyw87Do48N3J0MfroyKiysc1anYVvbeE/nl7NR7vrmXP0GG485yhKRuQm4UuISG8oIKRnqt6DNU9Fji4qKwCH4RP3h8XEkyC8v++hrrGFn72ygbv/vJ6mlla+eHIp13xmKgXZ6cn7DiLSLQoI6b292+C9P8LqRZFTUi0NkF0IU8+OXEY75QzIzANg2556/vOZNTy2rJLh2en801lHcNnsiaSFB/XtNiIpTQEhidFQA+tfiITFe09D/S4IZ8Lk0yNhccRcyB/Nu5t383+fWsmSDTs4fFQe3zp3GqcfUYyZOrJFBuV+10UAAA44SURBVBoFhCReSzN8uDgSFmueij4r22D88XDUOfiR5/Ds9gK+t2gVG6tr+eTUkdx07nSOHKOBAEUGEgWEBMsdtq2I9FmseQo+eiuyfMThtBxxDouaZnFzRRZ7GlqZP3si13z6cMYOz05uzSICKCCkv+2uhDV/jFwVtfEv0NpMa04xy7NP5K6tR/FGy1RGjhzFyYcXc/KUEZw4eQSFuV3frCciiaeAkOSp2wXrno8cXax9Dhr3AtBEOtt9ONu8gO1eSHPOKPJGjmPM2FJKSieTXTQW8sZEhgoJhZP8JURSlwJCBobmhsgRxfZVULOV1j1bqaneTNPurWTWbSfPaw5q4haG3GIsf3QkMA56HwN5oyOvbgwZIiIH6iwgNJCO9J+0TDj8jMiLyEiRw2JW19ft4901a1mzbi2VH75P7Y4tjGQnY/bsYmrTPsbteZ/hrUtJq6vGiPPDJrvwECESfc8tjoRIKD1y13g4PXJ0EjuvK61E2ikgZMDIys6lfMYMymfMACIDBL7x/g4Wr6/m5+s+ZvXWyOmpgkzjMxNCnHZYC7NGNDAhbS+hfdtg71aoib5XvxqZbjn0E/XisvChwyOU1s11afHn26c7to1+ZvuymPah6PqO++3J/kKh6PcKd3jX/Sk90na2xR04xLS3REYmaHs/YLo5Ot0aM52gbXNHwvTz4lXdJwoIGbCGZaVzxrTRnDFtNADVNQ0s2bCDxes/5rX11Tyxbh+QSWFOHidNOYaTpozk5BNGMHlkbuSeC3eo27k/NGqrI4HR0hT5R9X2ammC1qbIP7bYde3Tseuiy1qaD1zX3ACtNQeva2ne36a1ucNnNCX3f2A4dHBYx1DpJGQOWh7a32/krdE/nK1xplsPXM6h1rW96Hw9HrP/tiNMP3AaOv8Df6g2A924skACQn0QMmht2VXHa+urWby+msXrP+aj3fUAjBmWxclTRnDSlBGcfPhIxg3kS2rbfgG2h0bLgYHSEhNk8ULsoKCL077jr8+2X6YHLW+Ns11Pl8fs1ywaNNEXHDh/wMs6vHd4YXG2jde+bduYU4VmkWXt03TYpsN03DZdtW+bDO8/gjsgMNNiQrTDUV37tqGY9TGBe8D0IbYNZ0BOUa/+E1QntaQ8d+eD6lpeXf8xi9dX89r6anbsi5xeKhmRw8lTRnDCpBFMLs5lYlEOBdnpurNbBAWEDEGtrc572/eyeF3k6OKvG3awt6G5fX1+VhoTi3LaXxNipscOzyYjTefnZWhIWkCY2Rzgh0SeKPczd7+tw/pM4NdAGVANXOLuG82sFFgFrIluusTdr+rssxQQ0pnmllbWbq/hwx21bNpRy4fR16YdtWzaWUdjc2v7tiGDwwqymVCUHTdAinIzdPQhKSMpl7maWRi4EzgLqATeMLOF7r4yZrOvADvd/XAzmw/8B3BJdN16d58RVH0ytKSFQ0w7bBjTDht20LrWVmf73ob20IgNkRfXVFG1t+GA7XMzwgcExsQROUwojITI+MJsstJ1Y5+khiCvYpoNrHP3DQBm9jBwHhAbEOcB34lOPwb8xPTTTPpZKGSMKchiTEEWsycd3NFX29hM5c46PqzeHyCVO2vZWL2Pl9dWUd/UesD2Y4ZlHXjUMSKbMcOyKc7PpDg/k2FZaToCkUEhyIAYB2yKma8ETjjUNu7ebGa7gbbnVk4yszeBPcBN7v5KgLWKHFJORhpHjM7niNEHj0Tr7lTVNOw/bVVd134E8uq6j3l8T/1BbTLSQhTnZbYHRnF+5iHndTQiyTRQ74P4CJjo7tVmVgb8t5kd7e57YjcyswXAAoCJEycmoUwZ6syMUflZjMrPoqzk4KOP+qYWKnfWsX1PPVU1DVTtjXlFg2XZBzvZUdtIvO7A/Ky0AwJjZJwgGZWfSVFuhh7MJAkXZEBsBibEzI+PLou3TaWZpQEFQLVHes4bANx9qZmtB44ADuiFdvf7gPsg0kkdxJcQ6Yus9DCHj8rj8FF5nW7X1NLKjn2N7cHRMUiq9jawcsseqvY2HHA1VhszGJGbETdAhudkUJSbHnnPyaAwJ4P8rDRCIZ3mks4FGRBvAFPNbBKRIJgPXNZhm4XAFcBrwEXAn9zdzawY2OHuLWY2GZgKbAiwVpGkSg+HGD0si9HDsrrctq6xhY9rGtjeIUA+jgmWDVX7qKppOODqrFjhkDE8O53hOekU5WYwPCeDwpx0CnMjAVKUk8HwmPnCnEjAhBUqQ0pgARHtU7gGeIbIZa73u/sKM7sFqHD3hcDPgd+Y2TpgB5EQATgNuMXMmoBW4Cp33xFUrSKDSXb0KqoJRTmdbufu7KlvZldtIztrm9hZ28jOfdHpfY3srG1kV20TO/Y1smlHLW9XRtYdKlTMIsOfFB4QHBkd5vdP52elkZeVRm5GmoJlkNKNciLSzt2pa2phx7794REbLLtqG9nR9h7dZmdtI7WNLZ3uNzs9TF5WGnmZkVduZpi8zHTyMsPkZkaCJC8jGiiZaeRnpu1f3jYdfSlsEkvDfYtIt5gZORlp5GSkMb6w++3qm1raA6XtiGVvfRM1Dc3UNDSzL/pe09ASma5vZsuuuvZ1exuaD3nk0lF2eiRU8rPagubgEMltnw/HTEeOZtoCKjczjcy0kC457oQCQkT6LCs9zJiCMGMKuu5DOZTG5tb2INnXGAmR2IDZW9/MvoYW9jW2Te9fv2VXPfsa92/X0M2wSQvZ/vBoO5qJBkm8gMnJCHcIoP0hlZuZRnqKXUmmgBCRASEjLURGWkZCnk/e1NJKbUMLNY37g2Rf+5FMyyGX7WuMLN+2p559DS3t2zS3du9UfHrYyEoPk50eJjsj8p6ZHiY7PdS+rH19zPz+NpHtsjqsb18W3Wd/nWZTQIhIykkPhyjICVGQk97nfbk7DdGjm/bQaIwfMHVNLdQ1tlDfFHnVNbVQ19RKfWMLH9c0HrS+tqkl7v0vXckIh8hKD7UHxrHjh/PjS2f2+bt2pIAQEemEmbX/yh/R+e0sPebuNLa0Ut/YGg2TSIDUNbXQ0B4w+0MlMh3ZNjaEgnrmiQJCRCRJzIzMtDCZaWEK6PvRTqKlVo+KiIgkjAJCRETiUkCIiEhcCggREYlLASEiInEpIEREJC4FhIiIxKWAEBGRuFJmuG8zqwI+6MMuRgIfJ6icwWKofeeh9n1B33mo6Mt3LnH34ngrUiYg+srMKg41JnqqGmrfeah9X9B3HiqC+s46xSQiInEpIEREJC4FxH73JbuAJBhq33mofV/Qdx4qAvnO6oMQEZG4dAQhIiJxKSBERCSuIR8QZjbHzNaY2TozuyHZ9QTNzCaY2YtmttLMVpjZtcmuqb+YWdjM3jSzPyS7lv5gZsPN7DEzW21mq8zspGTXFDQz+6fof9fvmtlDZpaV7JoSzczuN7PtZvZuzLIiM3vOzNZG3wsT8VlDOiDMLAzcCcwFpgOXmtn05FYVuGbgG+4+HTgRuHoIfOc21wKrkl1EP/oh8LS7HwUcR4p/dzMbB3wdKHf3Y4AwMD+5VQXil8CcDstuAF5w96nAC9H5PhvSAQHMBta5+wZ3bwQeBs5Lck2BcveP3H1ZdHovkT8a45JbVfDMbDxwLvCzZNfSH8ysADgN+DmAuze6+67kVtUv0oBsM0sDcoAtSa4n4dz9ZWBHh8XnAb+KTv8KOD8RnzXUA2IcsClmvpIh8MeyjZmVAjOBvya3kn5xB/B/gNZkF9JPJgFVwC+ip9V+Zma5yS4qSO6+GfhP4EPgI2C3uz+b3Kr6zWh3/yg6vRUYnYidDvWAGLLMLA94HPhHd9+T7HqCZGafA7a7+9Jk19KP0oBZwN3uPhPYR4JOOwxU0fPu5xEJx7FArpldntyq+p9H7l1IyP0LQz0gNgMTYubHR5elNDNLJxIOD7r775NdTz84BZhnZhuJnEb8jJk9kNySAlcJVLp729HhY0QCI5WdCbzv7lXu3gT8Hjg5yTX1l21mdhhA9H17InY61APiDWCqmU0yswwiHVoLk1xToMzMiJyXXuXutye7nv7g7je6+3h3LyXy//Gf3D2lf1m6+1Zgk5kdGV10BrAyiSX1hw+BE80sJ/rf+RmkeMd8jIXAFdHpK4AnE7HTtETsZLBy92YzuwZ4hsgVD/e7+4oklxW0U4AvAO+Y2fLosn9x90VJrEmC8b+BB6M/fjYAX0pyPYFy97+a2WPAMiJX671JCg67YWYPAacDI82sEvhX4DbgETP7CpHHHlyckM/SUBsiIhLPUD/FJCIih6CAEBGRuBQQIiISlwJCRETiUkCIiEhcCgiRAcDMTh8qo8zK4KGAEBGRuBQQIj1gZpeb2etmttzM7o0+Y6LGzP4r+hyCF8ysOLrtDDNbYmZvm9kTbWP0m9nhZva8mb1lZsvMbEp093kxz294MHo3sEjSKCBEusnMpgGXAKe4+wygBfg7IBeocPejgT8TubMV4NfA9e7+CeCdmOUPAne6+3FExgpqG4VzJvCPRJ5NMpnIXe8iSTOkh9oQ6aEzgDLgjeiP+2wig6K1Ar+LbvMA8Pvo8xiGu/ufo8t/BTxqZvnAOHd/AsDd6wGi+3vd3Suj88uBUuAvwX8tkfgUECLdZ8Cv3P3GAxaa3dxhu96OX9MQM92C/n1KkukUk0j3vQBcZGajoP05wCVE/h1dFN3mMuAv7r4b2Glmn4wu/wLw5+hT/CrN7PzoPjLNLKdfv4VIN+kXikg3uftKM7sJeNbMQkATcDWRh/HMjq7bTqSfAiLDLt8TDYDY0VS/ANxrZrdE9/H5fvwaIt2m0VxF+sjMatw9L9l1iCSaTjGJiEhcOoIQEZG4dAQhIiJxKSBERCQuBYSIiMSlgBARkbgUECIiEtf/B4Vq8DFl3HtYAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zUJrwasOzHDt"
      },
      "source": [
        "#### Evaluating the model performance based on test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "icVm50SwFiZk"
      },
      "source": [
        "raw_test_predictions = model.predict(X_test)\n",
        "test_pred = np.argmax(raw_test_predictions, axis=-1)\n",
        "\n",
        "y_test_labels = np.argmax(y_test, axis=-1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lr3ZweCUGOur",
        "outputId": "2229242f-0103-4274-9c32-77b4eb35f9af"
      },
      "source": [
        "test_pred[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 1, 8, 8, 8, 8, 8, 8, 8, 8, 8,\n",
              "       8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,\n",
              "       8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "okRI2n0BGRTZ",
        "outputId": "ebf17e6b-fb38-40a3-9a04-3f567f94b43a"
      },
      "source": [
        "y_test_labels[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 1,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  9, 15,  8,  8,  8,\n",
              "        8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,\n",
              "        8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,\n",
              "        8,  8,  8,  8,  8,  8,  8])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CDRUZMqTGt9R",
        "outputId": "235a9b7d-46c8-4008-edd3-2c67b745adbc"
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "test_pred_fl = tf.reshape(test_pred, (test_pred.shape[0]*test_pred.shape[1]))\n",
        "test_pred_fl.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([408088])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lmitrUTfHLyw",
        "outputId": "8cf88b9a-aa1b-4c20-bb46-8c773945c518"
      },
      "source": [
        "y_test_labels_fl = tf.reshape(y_test_labels, (y_test_labels.shape[0]*y_test_labels.shape[1]))\n",
        "y_test_labels_fl.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([408088])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jUkZkfpOFqzI",
        "outputId": "a4b033bf-07e0-44f4-82da-96825477cc0f"
      },
      "source": [
        "from sklearn.metrics import f1_score, accuracy_score\n",
        "\n",
        "f1_score(y_test_labels_fl, test_pred_fl, average=\"weighted\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9820577018096401"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 71
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YebCxnZEKJew",
        "outputId": "44970011-fb33-4aa5-d80a-8637e0e565e4"
      },
      "source": [
        "i = np.random.randint(0, X_test.shape[0])\n",
        "p = model.predict(np.array([X_test[i]]))\n",
        "p = np.argmax(p, axis=-1)\n",
        "print(\"{:14} ({:5}): {}\".format(\"Word\", \"True\", \"Pred\"))\n",
        "for w, true_tag, pred in zip(X_test[i], y_test_labels[i], p[0]):\n",
        "    print(\"{:14}: {}\\t {}\".format(words[w], tags[true_tag], tags[pred]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Word           (True ): Pred\n",
            "pyongyang     : B-org\t B-org\n",
            "is            : O\t O\n",
            "demanding     : O\t O\n",
            "seoul         : B-org\t B-org\n",
            "pay           : O\t O\n",
            "at            : O\t O\n",
            "least         : O\t O\n",
            "$             : O\t O\n",
            "1             : O\t O\n",
            "billion       : O\t O\n",
            "in            : O\t O\n",
            "compensation  : O\t O\n",
            "for           : O\t O\n",
            "allegedly     : O\t O\n",
            "torturing     : O\t O\n",
            "north         : O\t B-geo\n",
            "korean        : B-gpe\t B-gpe\n",
            "spies         : O\t O\n",
            "and           : O\t O\n",
            "prisoners     : O\t O\n",
            "of            : O\t O\n",
            "war           : O\t O\n",
            "formerly      : O\t O\n",
            "held          : O\t O\n",
            "south         : B-geo\t B-geo\n",
            "korea         : I-geo\t I-geo\n",
            ".             : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pNv2qIMMzNtd"
      },
      "source": [
        "#### Model 2"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DV1Ol8mQLXyX",
        "outputId": "11c8cbe0-ae95-42bb-ac3d-10cb73660ffb"
      },
      "source": [
        "input = Input(shape=(max_len,))\n",
        "model_2 = Embedding(input_dim=n_words, output_dim=max_len, input_length=max_len)(input)\n",
        "model_2 = Dropout(0.2)(model_2)\n",
        "model_2 = Bidirectional(LSTM(units=100, return_sequences=True, recurrent_dropout=0.1))(model_2)\n",
        "output = TimeDistributed(Dense(n_tags, activation=\"softmax\"))(model_2)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Layer lstm_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
            "WARNING:tensorflow:Layer lstm_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
            "WARNING:tensorflow:Layer lstm_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xE3yWcfDLnBl"
      },
      "source": [
        "model_2 = Model(input, output)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KqHRTRFYzSm6"
      },
      "source": [
        "\"focal loss\" is most popularly used for training the model with imbalanced dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MTlCkeVILwfN"
      },
      "source": [
        "def focal_loss(gamma=2., alpha=4.):\n",
        "\n",
        "    gamma = float(gamma)\n",
        "    alpha = float(alpha)\n",
        "\n",
        "    def focal_loss_fixed(y_true, y_pred):\n",
        "        \"\"\"Focal loss for multi-classification\n",
        "        FL(p_t)=-alpha(1-p_t)^{gamma}ln(p_t)\n",
        "        Notice: y_pred is probability after softmax\n",
        "        gradient is d(Fl)/d(p_t) not d(Fl)/d(x) as described in paper\n",
        "        d(Fl)/d(p_t) * [p_t(1-p_t)] = d(Fl)/d(x)\n",
        "        Focal Loss for Dense Object Detection\n",
        "        https://arxiv.org/abs/1708.02002\n",
        "\n",
        "        Arguments:\n",
        "            y_true {tensor} -- ground truth labels, shape of [batch_size, num_cls]\n",
        "            y_pred {tensor} -- model's output, shape of [batch_size, num_cls]\n",
        "\n",
        "        Keyword Arguments:\n",
        "            gamma {float} -- (default: {2.0})\n",
        "            alpha {float} -- (default: {4.0})\n",
        "\n",
        "        Returns:\n",
        "            [tensor] -- loss.\n",
        "        \"\"\"\n",
        "        epsilon = 1.e-9\n",
        "        y_true = tf.convert_to_tensor(y_true, tf.float32)\n",
        "        y_pred = tf.convert_to_tensor(y_pred, tf.float32)\n",
        "\n",
        "        model_out = tf.add(y_pred, epsilon)\n",
        "        ce = tf.multiply(y_true, -tf.math.log(model_out))\n",
        "        weight = tf.multiply(y_true, tf.pow(tf.subtract(1., model_out), gamma))\n",
        "        fl = tf.multiply(alpha, tf.multiply(weight, ce))\n",
        "        reduced_fl = tf.reduce_max(fl, axis=1)\n",
        "        return tf.reduce_mean(reduced_fl)\n",
        "    return focal_loss_fixed"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Upe3UORaLnBl"
      },
      "source": [
        "model_2.compile(optimizer=\"nadam\", loss=focal_loss(alpha=1), metrics=[\"accuracy\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "H0LvSWQyLnBm",
        "outputId": "089ed50d-31da-4c73-8c0f-522236d62d41"
      },
      "source": [
        "model_2.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_1\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_2 (InputLayer)         [(None, 58)]              0         \n",
            "_________________________________________________________________\n",
            "embedding_1 (Embedding)      (None, 58, 58)            1750034   \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 58, 58)            0         \n",
            "_________________________________________________________________\n",
            "bidirectional_1 (Bidirection (None, 58, 200)           127200    \n",
            "_________________________________________________________________\n",
            "time_distributed_1 (TimeDist (None, 58, 17)            3417      \n",
            "=================================================================\n",
            "Total params: 1,880,651\n",
            "Trainable params: 1,880,651\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ruP-iUw3LnBm"
      },
      "source": [
        "!mkdir \"saved_model_2\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iJ7Vxpa1LnBm"
      },
      "source": [
        "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
        "import numpy as np\n",
        "\n",
        "es = EarlyStopping(monitor=\"val_loss\", mode=\"min\", patience=3)\n",
        "checkpt = ModelCheckpoint(\"/content/saved_model_2\", monitor='val_loss', mode='min', save_best_only=True, verbose=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BExyrGS2LnBn",
        "outputId": "0accff5c-b97f-4218-fd43-258d5067f4f2"
      },
      "source": [
        "history = model_2.fit(X_train, np.array(y_train), batch_size=128, epochs=200, validation_split=0.2, verbose=1, callbacks=[es, checkpt])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/200\n",
            "176/176 [==============================] - 72s 372ms/step - loss: 0.3417 - accuracy: 0.9069 - val_loss: 0.1502 - val_accuracy: 0.9694\n",
            "\n",
            "Epoch 00001: val_loss improved from inf to 0.15021, saving model to /content/saved_model_2\n",
            "INFO:tensorflow:Assets written to: /content/saved_model_2/assets\n",
            "Epoch 2/200\n",
            "176/176 [==============================] - 64s 362ms/step - loss: 0.1341 - accuracy: 0.9714 - val_loss: 0.0993 - val_accuracy: 0.9794\n",
            "\n",
            "Epoch 00002: val_loss improved from 0.15021 to 0.09925, saving model to /content/saved_model_2\n",
            "INFO:tensorflow:Assets written to: /content/saved_model_2/assets\n",
            "Epoch 3/200\n",
            "176/176 [==============================] - 64s 365ms/step - loss: 0.0848 - accuracy: 0.9811 - val_loss: 0.0854 - val_accuracy: 0.9817\n",
            "\n",
            "Epoch 00003: val_loss improved from 0.09925 to 0.08538, saving model to /content/saved_model_2\n",
            "INFO:tensorflow:Assets written to: /content/saved_model_2/assets\n",
            "Epoch 4/200\n",
            "176/176 [==============================] - 65s 367ms/step - loss: 0.0661 - accuracy: 0.9846 - val_loss: 0.0792 - val_accuracy: 0.9824\n",
            "\n",
            "Epoch 00004: val_loss improved from 0.08538 to 0.07917, saving model to /content/saved_model_2\n",
            "INFO:tensorflow:Assets written to: /content/saved_model_2/assets\n",
            "Epoch 5/200\n",
            "176/176 [==============================] - 65s 369ms/step - loss: 0.0546 - accuracy: 0.9864 - val_loss: 0.0801 - val_accuracy: 0.9831\n",
            "\n",
            "Epoch 00005: val_loss did not improve from 0.07917\n",
            "Epoch 6/200\n",
            "176/176 [==============================] - 65s 367ms/step - loss: 0.0478 - accuracy: 0.9875 - val_loss: 0.0770 - val_accuracy: 0.9835\n",
            "\n",
            "Epoch 00006: val_loss improved from 0.07917 to 0.07698, saving model to /content/saved_model_2\n",
            "INFO:tensorflow:Assets written to: /content/saved_model_2/assets\n",
            "Epoch 7/200\n",
            "176/176 [==============================] - 64s 364ms/step - loss: 0.0427 - accuracy: 0.9884 - val_loss: 0.0777 - val_accuracy: 0.9831\n",
            "\n",
            "Epoch 00007: val_loss did not improve from 0.07698\n",
            "Epoch 8/200\n",
            "176/176 [==============================] - 64s 366ms/step - loss: 0.0370 - accuracy: 0.9897 - val_loss: 0.0788 - val_accuracy: 0.9830\n",
            "\n",
            "Epoch 00008: val_loss did not improve from 0.07698\n",
            "Epoch 9/200\n",
            "176/176 [==============================] - 66s 377ms/step - loss: 0.0339 - accuracy: 0.9903 - val_loss: 0.0804 - val_accuracy: 0.9835\n",
            "\n",
            "Epoch 00009: val_loss did not improve from 0.07698\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "T7sZY-JwLnBn",
        "outputId": "bcbcaec7-d7ec-47d5-ae1c-bf805e5ca480"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.plot(history.history['accuracy'])\n",
        "plt.plot(history.history['val_accuracy'])\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'test'], loc='upper left')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxU9Z3v/9en970butmkhUYFBBVREKO4QBwnGI1bMsYFb7YRJzEZM9G5ibmJTvL45ZrJeHMzmWxjMuZqjBqDmhjDuKZbkqiJ4IIoNCCCNCC9AE3v6+f3xzndVDeFVEMV1V39fj4e9ag6S536FMv3Xed7zvkec3dEREQGS0t2ASIiMjwpIEREJCoFhIiIRKWAEBGRqBQQIiISVUayC4iXsrIyr6ioSHYZIiIjyurVq+vdfVy0ZSkTEBUVFaxatSrZZYiIjChmtvVgy9TFJCIiUSkgREQkKgWEiIhElTLHIKLp6uqipqaG9vb2ZJeScDk5OZSXl5OZmZnsUkQkRaR0QNTU1FBYWEhFRQVmluxyEsbdaWhooKamhmnTpiW7HBFJEQntYjKzJWZWbWabzOwrUZZPNbPnzGyNmVWZWXnEsn81s7Xh4+OH8/nt7e2UlpamdDgAmBmlpaWjYk9JRI6ehAWEmaUDPwQuAmYD15jZ7EGr3QXc5+5zgG8Cd4bvvRg4HZgLnAncamZFh1nH4X2BEWa0fE8ROXoS2cW0ANjk7psBzOwh4DLgrYh1ZgNfCl9XAr+JmL/S3buBbjNbAywBHk5gvSIiw467097Vy772Lva1dYXP3RHT3YzJy+LaM6fE/bMTGRCTgW0R0zUEewORXgeuBP4duAIoNLPScP4dZvZ/gDxgMQODBQAzWwYsA5gyJf5/OPGwd+9eHnjgAT73uc8N6X0f/vCHeeCBBygpKUlQZSJyNLg7rZ09/Q17U3v0Rv5gjf++ti66e9//vj2nTSkZcQERi1uBH5jZJ4GVwHagx92fNrMzgBeAOuBFoGfwm939buBugPnz5w/LOx/t3buXH/3oRwcERHd3NxkZB//jX7FiRaJLE5EY9fY69S0d7G7pDBrwsDFvitawR2nkew7RwOdkplGUk0lRbiZFORmMzc+iojSfotyMiPmZA6YLc4LXhTkZ5GSmJ+R7JzIgtgPHRkyXh/P6ufsOgj0IzKwA+Ki77w2XfQv4VrjsAWBDAmtNmK985Su8/fbbzJ07l8zMTHJychgzZgzr169nw4YNXH755Wzbto329nZuvvlmli1bBuwfOqS5uZmLLrqIc845hxdeeIHJkyfz29/+ltzc3CR/M5HU0NPr1DV1sLOxjfca29nZ2M57+8LnxjZ27G2ntqmdrp6DN/K5mekDGu+ygiyOG5d/QKMeOV2Yk9Hf0GdnJKaBP1KJDIiXgelmNo0gGK4Gro1cwczKgN3u3gvcBtwTzk8HSty9wczmAHOAp4+kmG/87k3e2rHvSDZxgNnHFHHHR05633W+/e1vs3btWl577TWqqqq4+OKLWbt2bf/pqPfccw9jx46lra2NM844g49+9KOUlpYO2MbGjRt58MEH+elPf8pVV13FI488wtKlS+P6XURSUVdPL7VNHbzX2BY2+O0Rz0Eg7GrqOOAXfnZGGpOKc5hYnMOCaWOZWJzDpOIcSvOzKc4NGvnCnODXfmFOJlkZqXnNccICwt27zezzwFNAOnCPu79pZt8EVrn748Ai4E4zc4IuppvCt2cCfwzPzNkHLA0PWI94CxYsGHCtwve//30ee+wxALZt28bGjRsPCIhp06Yxd+5cAObNm8eWLVuOWr0iw1VHdw+1+zrYGdHY9zf++4Jf/3VNHQzu3cnNTGdSSdDgn3V8WX8Q7H/OZUxeps4MJMHHINx9BbBi0LzbI14vB5ZHeV87wZlMcXOoX/pHS35+fv/rqqoqnn32WV588UXy8vJYtGhR1GsZsrOz+1+np6fT1tZ2VGoVSZb2rp6I7p5ov/7bqW/uOOB9hdkZTAwb+pkTxjGxOHdAAEwqyqUoN0ONf4ySfZA65RUWFtLU1BR1WWNjI2PGjCEvL4/169fz0ksvHeXqRI6ezu5eGlo6qGvqoL65g/qmTuqag9f985o7qW/uYG9r1wHvL87N7G/sT55cxMSi3AN+/RfmaKiZeFJAJFhpaSkLFy7k5JNPJjc3lwkTJvQvW7JkCT/5yU+YNWsWM2fO5AMf+EASKxUZus7u3rBhj2zoOyMa/P3zGtsObPQBCrIzGFeYTVlBFtPHF3DWcaWML8xmUsnAAMjLUnN1tJn7sDw7dMjmz5/vg28YtG7dOmbNmpWkio6+0fZ9JTE6untoGNTI9zX6dc0d1Ef82j9Yo1+YnUFZ2OgHjX/kY/+8cYXZCTtFU2JjZqvdfX60ZYpkkVGktbObrQ2tbG1oYUtDK+81tvc3+n3P+9qjnw9SmJPBuLCRnzmxkIUF2cF04YENvxr91KCAEEkx+9q7eLehlS0NLWxtaGVLffjc0EJt08ADu/2NfmE2syYWUXZCVv8v+7L+xj9Ljf4opYAQGYH2tnaypW9PoL5vjyAIgoaWzgHrTijKZmppPotmjmNqaT4VpflMLc1jammeDurK+1JAiAxD7k59c2d/V9CA5/qWAd1AZnBMcS5TS/P425MmUlGaFwRBWR5Txubp4K4cNv3LEUmS3l6ntqkj/OXfcsAeQUvn/uHH0gzKxwS/+i+bO5mppXlUhCFQPiZP3T+SEAoIkQRyd3bt62BzXTNbwuMAfccEtu5uob2rt3/djDRjytggBBZMGxvsCZQFXUKTS3JTdjgHGb4UEAl2uMN9A3zve99j2bJl5OXlJaAyiTd3Z2djO29sb2Tt9sb+5/rm/ccEstLTmFKaR0VpHudML6OiLJ+KcG9gUnEOGekKARk+FBAJdrDhvmPxve99j6VLlyoghqG+MFhTMzAM+g4QpxlMH1/I+TPGc8rkImZMKGRqWT4Ti3JIT9MwDzIyKCASLHK47wsvvJDx48fz8MMP09HRwRVXXME3vvENWlpauOqqq6ipqaGnp4evf/3r7Nq1ix07drB48WLKysqorKxM9lcZtdydHY3tvHGQMEhPM6aPL2DxieM5ZXIxJ08uZvakInKzdFxARrbRExD//RV47434bnPiKXDRt993lcjhvp9++mmWL1/OX//6V9ydSy+9lJUrV1JXV8cxxxzD73//eyAYo6m4uJjvfve7VFZWUlZWFt+65aDcne172/qD4I3t+1i7vZHdg8LggyeO55TyIAxmTVQYSGoaPQExDDz99NM8/fTTnHbaaQA0NzezceNGzj33XG655Ra+/OUvc8kll3DuuecmudLRwd2p2RMZBsGewZ5woLj0NGPGhEL+Ztb+PYNZk4p0xpCMGqMnIA7xS/9ocHduu+02brzxxgOWvfLKK6xYsYKvfe1rXHDBBdx+++1RtiCHqy8MIoPgje2N/aOGZoRh8LezJ3JyeTGnTC7mxImFCgMZ1UZPQCRJ5HDfH/rQh/j617/OddddR0FBAdu3byczM5Pu7m7Gjh3L0qVLKSkp4Wc/+9mA96qLaWjcnW27B4bB2h0Dw2DmxEKWnDSRkycHYTBTYSByAAVEgkUO933RRRdx7bXXctZZZwFQUFDA/fffz6ZNm/jnf/5n0tLSyMzM5Mc//jEAy5YtY8mSJRxzzDE6SP0+3J11O5t4cu1OVr+7h7Xb9/WPMpqZHoTBRScPDIPheg9gkeFEw32nkNH2fTfuauJ3a3byxJodbK5rIc3gpGOK+4PglMnFzJhYoDAQeR8a7ltSxjv1LTzx+g6eWLOT6l1NmMGZ08by6YXTuOjkiZQWZB96IyISEwWEDHvbdrfyRLin8OaOfQDMnzqGf/nIbD58yiTGF+UkuUKR1JTyAeHuo+IG5anSVdhnZ2Mbv1+zk9+t2cnr2/YCcOqxJXzt4ll8+JRJHFOSm+QKR6juTuhogo59wbOlQVb+/kdGLqRpuI9hr7cH2huDv8f2RrB0mHhy3D8mpQMiJyeHhoYGSktLUzok3J2GhgZyckb2L+napnZWrNnJE2t2smrrHgBOOqaILy85kUvmTOLYsaN0yJHeXuhqCRv2poENfEfzgfM6o8zrW7en49Cfl9kXGHmQVRC8zswbGCRZ+RHrRc6LeE/f+zPzgoeCJ+AOXW0DG/j2fdC+d9D04OUR053NA7c5eT7c8FzcS03pgCgvL6empoa6urpkl5JwOTk5lJeXJ7uMIWto7uC/177HE2t28Jd3duMOMycUcsuFM7h4ziSOG1eQ7BIPn3vwH7ptT/AffHADP6Ah72vMB89rgs6m2D4vIweyC/c/sgqhqHzgvOwCyC4KlxcADp2tQS2dLdDVGjxHPrpaguXNuyLmtQaPocjsC438/cFxsIDJyIK0jIhHevArefC8tPQhrhNtvfRBzxnBntXBflT2dO9vqA/ZoDdGX94b/bau/dIyIKc4+LvKKYacIig7AbKL909HLi+cNLS/ixildEBkZmYybdq0ZJchg+xt7eSpN9/jiTU7eeHtBnp6nePG5fOFD07nI3MmMX1CYbJLPFBXO7TthtbdQYPf/zqcbt1z4PK2PYduCCwtaMgjG/GcYiguH9iY9zf6Ueb1PdKP8t3henvCQBkcMM3hvDBYooZO33rN0Fw78P1DDZ5EiRY2PZ0H/nqPJqtgYANeMB7Kpg9s8PuXlxzY4GfmHjygjqKUDggZPprau3jmrV08sWYnf9xYR1ePM2VsHjeedxyXzDmGWZMKj043YE93sCvftmd/Ax9Lo9/ddvBtZuRC7hjIGxs8jz8xeM4du39eTsmBv+CzC4NfzMOgITgsaen7vwcT4rfd3h7o6QrC1XuC6d7u8NEz6Llvne4o68WyTsRzLOukZ4YNfPGBjX3fvOwiSE+NpjU1voUMSy0d3Ty7bhe/X7OTqg11dHb3Mrkkl08tnMYlcyZxyuTiIwuF7k5obYDWemipD19HNvBRAqC98eDbs/SIhn5s8Ct+0pywsY+Y39fo973O1AHzuOrrFpKkU0BIXLV39fCH9bU8sWYHf1hfS3tXLxOKsrnuzClcMucYTp9ScvBQ6GwJG/p6aGkY1Pj3zeub3h307x5MdjHkluxv1EuPH/SrPmzk8yLmZReN3F/zIgmggJAj1tHdw8oN9fzu9R08u24XrZ09lOVncv2pxVx8QhZzxnST1rYF6lfBuxENfWvD/l/+LfUH78ZJy4T8MsgrDR6TT4e8sv3z8suC6b7luSVHvz9eJAUpIGRoerpg5+t073mXt7du5Z2t77K7bgeFPXu5Lr2Z2/LaKM3bR2bnHmxtN6yNso3MvLCBL4X8cTB+1v7Gva+xj2z89cteJCkUEPL+entg5+uw5Y/wzkp864tYVwsZwMzw0ZpWAAWl5JSMJy1/SpSGPgyDvNLgddYovZ5BZIRRQMhAvb1Q+xa8szIIhS1/7u/r7xpzAk/a+TzZNZ0p0+dw9pyZnHnSDPKyNf6RSCpSQIx27lC/Ed55PgyEPwXHBADGVMBJl0HFefy5dxaf/c120tKM73/iNM6bMS6pZYtI4ikgRht32LNl/x7CO3+E5veCZUWTYfrfQsW5MO1cKJlCb6/zw8pNfPfZDZw4sYi7r583eoe8EBllFBCjQeP2iEBYCY3bgvn544MgqDgXpp0HY48bcDC4qb2LWx5+naff2sXlc4/hzivnkJul89NFRouEBoSZLQH+HUgHfubu3x60fCpwDzAO2A0sdfeacNl3gIuBNOAZ4GZPtSFLE6W5dmAg7N4czM8dAxXnwNn/GATCuJkHPTtoU20zN/5iFVsaWrn9ktl8amFFSg94KCIHSlhAmFk68EPgQqAGeNnMHnf3tyJWuwu4z93vNbMPAncC15vZ2cBCYE643p+A84GqRNU7orXuDo4d9AVC3fpgflYhVCyE+Z8JAmHCyTGNqPnUm+9xy8Ovk52Rxv2fOZOzji9N8BcQkeEokXsQC4BN7r4ZwMweAi4DIgNiNvCl8HUl8JvwtQM5QBZgQCawK4G1jizt++DdF4MweOd5eG8t4MH1BVM+AKdeDRXnwaRThzQmTE+v83+f2cAPKjdxankxP146T/ddEBnFEhkQk4FtEdM1wJmD1nkduJKgG+oKoNDMSt39RTOrBHYSBMQP3H3d4A8ws2XAMoApU6bE/xsMF50t8O5L+/cQdrwWDCyWng3HLoDFXw2OI0yeFwyTfBgaW7v4x4de5fkNdXx8/rF847KTyMnU8QaR0SzZB6lvBX5gZp8EVgLbgR4zOwGYBfTd4OAZMzvX3f8Y+WZ3vxu4G2D+/PmpdXyitxde+iGsXwE1L0NvVzDk8OR5cO6XgkA4dkFcBopbt3MfN/5iNTsb2/jWFSdz7YIpOt4gIgkNiO3AsRHT5eG8fu6+g2APAjMrAD7q7nvN7AbgJXdvDpf9N3AWMCAgUtozX4cXfxB0E531uaDLaMoHguGi4+jx13fw5eVrKMzJ4KFlZzFv6pi4bl9ERq5EBsTLwHQzm0YQDFcD10auYGZlwG537wVuIzijCeBd4AYzu5Ogi+l84HsJrHV4efGHQTgsWAYXfSch4xB19/Tyr0+u56d/fIf5U8fwo+tOZ3zRyL5lqYjEV8ICwt27zezzwFMEp7ne4+5vmtk3gVXu/jiwCLjTzJygi+mm8O3LgQ8CbxAcsH7S3X+XqFqHlTeWw1NfhdmXwZJvJyQcGpo7+MKDr/LC2w38j7Om8rWLZ5OVofsFi8hAliqXFsyfP99XrVqV7DKOzObn4f6PBscWlj4KmfH/Rf9GTSP/cP9q6po7+N9XnMLH5o28+1iLSPyY2Wp3nx9tWbIPUkuf996Ah66D0hPg6l8mJByWr67hq4+9wbiCbB75h7M5pbw47p8hIqlDATEc7NkK938suLft0keCK57jqLO7l//v929x34tbOfv4Uv7jmtMoLdAIrCLy/hQQyda6O+hW6m6DTz8FxZPjuvnapnY+d/8rrNq6hxvOncaXl5xIRrqON4jIoSkgkqmzFR74OOx9F/7Hb4I7q8XR6q17+Oz9q2lq7+b715zGpaceE9fti0hqU0AkS083PPKZ4CK4q+6DqWfHbdPuzgN/fZd/efxNJhXncu+nFzBrUlHcti8io4MCIhncYcUtUL0CPnwXzL40bptu7+rhjt++ya9WbeP8GeP496vnUpJ3eMNviMjopoBIhue/A6v/H5zzJVhwQ9w2u2NvG5+9fzWv1zTy+cUn8E8XziA9TUNmiMjhUUAcbavvhar/DadeCxfcHrfNvrS5gZt++Qod3b38ZOk8lpw8MW7bFpHRSQFxNFU/CU/8E5zwN3Dp9+NylbS78/M/b+FbK9YxtTSPu6+fxwnjC+NQrIiMdgqIo2Xby/DrT8KkOfB390J65hFvsq2zh9seXcNvXtvBhbMn8N2rTqUw58i3KyICCoijo34jPHAVFE6Ea38dlxFZt+1u5cZfrGbde/u45cIZ3LT4BNJ0vEFE4kgBkWhN78H9V4KlwfWPQsG4I97kyg11/ONDr9Lb69zziTNYfOL4OBQqIjKQAiKR2vfBLz8GLQ3wySdg7HFHtDl358fPv81dT1UzfXwh/3n9PCrK8uNUrIjIQAqIROnuhF8thdp1cM2vYPLpR7S55o5u/ufy11nxxntcPGcS3/noHPKz9dcnIomjFiYRenvht5+Dd56Hy38M0//miDa3ua6ZG3+xmrfrmvnqh0/khnOP0y1BRSThFBCJ8Ozt8Mavg+sc5l576PXfx3PrdvHFh14jI934xWfOZOEJZXEqUkTk/Skg4u3FH8EL/wFn3BBcKX0EfvCHjdz19AZOnlzET5bOo3xMXpyKFBE5NAVEPK19BJ66DWZdChf96xFdCLdtdyt3Pb2Bi0+ZxP+56lRyMtPjWKiIyKHpxgDx8s5KeOwfYMrZcOVPIe3IGvTK6loAbv3QTIWDiCSFAiIe+m4XOvZ4uOaBuNwutHJ9LRWleUzTaawikiQKiCO1993gdqFZBbB0eVxuF9re1cMLbzewaKYugBOR5NExiCMRebvQTz0JxeVx2eyLmxvo6O7VFdIiklQKiMPV1QYPXg17tsL1j8GE2XHbdNX6WnIy0zhz2ti4bVNEZKgUEIejpxuWfwa2/RWuuhcqFsZt0+5OZXUdC48v08FpEUkqHYMYKndYcStU/x4u+g7Mviyum99c38K7u1tZpO4lEUkyBcRQrfw3WP1zOOef4Mxlcd985frg9NZFM4581FcRkSOhgBiKV+6Dym/BqdfABXck5COqquuYPr6AY8fqqmkRSS4FRKw2PAW/+yIcfwFc+h9xuV3oYC0d3fzlnQadvSQiw4ICIhY1q+DhT8DEU+Cq++Jyu9Bo/rypnq4eZ9FMdS+JSPIpIA6lfhP88u+C24VeF5/bhR5MZXUdBdkZzJ+q01tFJPliCggze9TMLjaz0RUoTbvg/iuC24UufQQKEtf14+5UVddyzgllZGWMrj9mERmeYm2JfgRcC2w0s2+b2cwE1jQ89N8utB6uexhKj0/ox1XvamJnYzuLT1T3kogMDzEFhLs/6+7XAacDW4BnzewFM/uUmSWmQz6Zujvh4eth15vBMYfJ8xL+kZXr6wA0/pKIDBsx92WYWSnwSeDvgVeBfycIjGfe5z1LzKzazDaZ2VeiLJ9qZs+Z2RozqzKz8nD+YjN7LeLRbmaXD/G7HZ7eXvjtTbC5KjhbafqFR+VjK6trmT2piAlFRz4SrIhIPMR6DOIx4I9AHvARd7/U3X/l7l8Aoh61NbN04IfARcBs4BozGzxg0V3Afe4+B/gmcCeAu1e6+1x3nwt8EGgFnh7ytzscz94BbzwMH/w6nHbdUfnIxrYuVm/do+4lERlWYh2L6fvuXhltgbvPP8h7FgCb3H0zgJk9BFwGvBWxzmyg776clcBvomznY8B/u3trjLUevpd+DC98H874ezj3loR/XJ8/baynp9dZrO4lERlGYu1imm1mJX0TZjbGzD53iPdMBrZFTNeE8yK9DlwZvr4CKAy7siJdDTwY7QPMbJmZrTKzVXV1dYf6Du9v7aPw5G0w6yPBGEsJuBDuYCqraynOzWTusSWHXllE5CiJNSBucPe9fRPuvge4IQ6ffytwvpm9CpwPbAd6+haa2STgFOCpaG9297vdfb67zx837gi6Z95ZCY/dCFM+EJfbhQ5Fb69TVV3HeTPGkZGu01tFZPiItYsp3czM3R36jy9kHeI924FjI6bLw3n93H0H4R6EmRUAH40MIuAq4DF374qxzqFreDu8XehxcM2DkJmbsI+K5s0d+6hv7mCxrp4WkWEm1p+sTwK/MrMLzOwCgi6fJw/xnpeB6WY2zcyyCLqKHo9cwczKIi6+uw24Z9A2ruEg3UtxUzIV5n86uBAuDrcLHarK6lrM4DyN3ioiw0ysexBfBm4EPhtOPwP87P3e4O7dZvZ5gu6hdOAed3/TzL4JrHL3x4FFwJ1m5sBK4Ka+95tZBcEeyPOxfpnDkp4BF34joR/xfiqra5lTXkJZQXbSahARiSamgHD3XuDH4SNm7r4CWDFo3u0Rr5cDyw/y3i0ceFA7pexu6eS1bXu5+YLpyS5FROQAMQWEmU0nuEZhNtB/JZe7H5egukaFlRvqcEent4rIsBTrMYifE+w9dAOLgfuA+xNV1GhRWV1LaX4Wp0wuTnYpIiIHiDUgct39OcDcfau7/wtwceLKSn09vc7zG+o4f+Y40tKO3jUXIiKxivUgdUd4ttHG8MDzdg4yxIbE5rVte9nb2qXuJREZtmLdg7iZYBymfwTmAUuBTySqqNGgqrqWNIPzpuv0VhEZng65BxFeFPdxd78VaAY+lfCqRoHK6lrmTR1DcV7qjZYuIqnhkHsQ7t4DnHMUahk1ave1s3b7Pt37QUSGtViPQbxqZo8DvwZa+ma6+6MJqSrFVW0IBhbU8QcRGc5iDYgcoIHg3gx9HFBAHIaq6lomFGUza1JhsksRETmoWK+k1nGHOOnq6eWPG+q5eM4k7CgOKS4iMlSxXkn9c4I9hgHc/dNxryjFrd66h6aObh1/EJFhL9YupiciXucQ3NxnR/zLSX2V1bVkphsLTxh8XyQRkeEl1i6mRyKnzexB4E8JqSjFVa2v44yKsRTm6PRWERneDvcWZtMB9ZEM0fa9bVTvatLZSyIyIsR6DKKJgccg3iO4R4QMQVV1LQCLT9TV0yIy/MXaxaTzMeOgcn0d5WNyOX6chrESkeEvpi4mM7vCzIojpkvM7PLElZV6Orp7+POmehbPHK/TW0VkRIj1GMQd7t7YN+Hue4E7ElNSavrrO7tp6+pR95KIjBixBkS09WI9RVYIupeyMtI467iyZJciIhKTWANilZl918yODx/fBVYnsrBUU1Vdy1nHlZKblZ7sUkREYhJrQHwB6AR+BTwEtAM3JaqoVLOlvoXN9S0snqnuJREZOWI9i6kF+EqCa0lZfae3angNERlJYj2L6RkzK4mYHmNmTyWurNRSWV3HcWX5VJTlJ7sUEZGYxdrFVBaeuQSAu+9BV1LHpK2zhxc3N2jvQURGnFgDotfMpvRNmFkFUUZ3lQO9uLmezu5end4qIiNOrKeq/i/gT2b2PGDAucCyhFWVQirX15Gbmc6CaWOTXYqIyJDEepD6STObTxAKrwK/AdoSWVgqcHcqq2tZeEIZ2Rk6vVVERpZYB+v7e+BmoBx4DfgA8CIDb0Eqg7xd10zNnjY+u+j4ZJciIjJksR6DuBk4A9jq7ouB04C97/8WqVxfB+j0VhEZmWINiHZ3bwcws2x3Xw/MTFxZqaGyupaZEwqZXJKb7FJERIYs1oCoCa+D+A3wjJn9FtiauLJGvqb2Ll7esptFOntJREaoWA9SXxG+/BczqwSKgScTVlUK+POmBrp6XHePE5ERa8gjsrr784koJNVUVddSmJ3BvKljkl2KiMhhOdx7Usv76Du99dwZZWSm649YREamhLZeZrbEzKrNbJOZHTDYn5lNNbPnzGyNmVWZWXnEsilm9rSZrTOzt8Krt0eEdTub2LWvQ2cviciIlrCAMLN04IfARcBs4Bozmz1otbuA+9x9DvBN4M6IZfcB/+bus61+V6MAAA1ySURBVIAFQG2iao23yr7RW2foALWIjFyJ3INYAGxy983u3klwH4nLBq0zG/hD+Lqyb3kYJBnu/gyAuze7e2sCa42rqupaTp5cxPiinGSXIiJy2BIZEJOBbRHTNeG8SK8DV4avrwAKzawUmAHsNbNHzexVM/u3cI9kADNbZmarzGxVXV1dAr7C0DW2drF66x6dvSQiI16yj6DeCpxvZq8C5wPbgR6Cs6vODZefARwHfHLwm939bnef7+7zx40bHt05KzfW0eu6elpERr5EBsR24NiI6fJwXj933+HuV7r7aQQjxhLed6IGeC3snuomuEDv9ATWGjeV1bWU5GUy99iSQ68sIjKMJTIgXgamm9k0M8sCrgYej1zBzMrMrK+G24B7It5bYmZ9uwUfBN5KYK1x0dvrPF9dx/kzxpGeZskuR0TkiCQsIMJf/p8HngLWAQ+7+5tm9k0zuzRcbRFQbWYbgAnAt8L39hB0Lz1nZm8Q3IPip4mqNV7e2N5IQ0unjj+ISEoY8pXUQ+HuK4AVg+bdHvF6ObD8IO99BpiTyPrirbK6FjM4T6e3ikgKSPZB6pRSWV3H3GNLGJuflexSRESOmAIiTuqbO1hTs1fdSyKSMhQQcbJyQx3uKCBEJGUoIOKksrqOsoJsTjqmKNmliIjEhQIiDrp7elm5oY5FM8eRptNbRSRFKCDi4LVte2ls61L3koikFAVEHFRW15KeZpwzvSzZpYiIxI0CIg4q19cxb+oYinMzk12KiEjcKCCO0HuN7by1c5+6l0Qk5SggjtDzG4KbAy0+UVdPi0hqUUAcocr1dUwqzmHmhMJklyIiElcKiCPQ2d3LnzbVs2jmeMx0equIpBYFxBFYtXU3zR3dLJ6p7iURST0KiCNQVV1HZrqx8ASd3ioiqUcBcQQq19dy5rRS8rMTOmq6iEhSKCAO07bdrWysbWaRupdEJEUpIA5T1YY6ABafqOsfRCQ1KSAOU9X6WqaMzeO4svxklyIikhAKiMPQ3tXDn9+uZ/HMcTq9VURSlgLiMPzlnd20d/WySN1LIpLCFBCHoXJ9LdkZaZx1XGmySxERSRgFxGGoqq7l7ONLyclMT3YpIiIJo4AYonfqW9jS0Kqzl0Qk5SkghqhyfTB666IZCggRSW0KiCGqrK7l+HH5TCnNS3YpIiIJpYAYgtbObv6yebduDiQio4ICYghe2NRAZ0+vjj+IyKiggBiCyupa8rPSmV8xJtmliIgknAIiRu5OVXUdC08oIztDp7eKSOpTQMRoY20z2/e2qXtJREYNBUSM+k9v1fDeIjJKKCBiVFldy4kTC5lUnJvsUkREjgoFRAz2tXexassedS+JyKiigIjBnzfW093ruv5BREaVhAaEmS0xs2oz22RmX4myfKqZPWdma8ysyszKI5b1mNlr4ePxRNZ5KJXVtRTmZHD6lJJkliEiclRlJGrDZpYO/BC4EKgBXjazx939rYjV7gLuc/d7zeyDwJ3A9eGyNnefm6j6YuXuVFbXcd6McWSka4dLREaPRLZ4C4BN7r7Z3TuBh4DLBq0zG/hD+LoyyvKke3PHPuqaOtS9JCKjTiIDYjKwLWK6JpwX6XXgyvD1FUChmfXdhSfHzFaZ2Utmdnm0DzCzZeE6q+rq6uJZe7+q6uD01vNn6PRWERldkt1ncitwvpm9CpwPbAd6wmVT3X0+cC3wPTM7fvCb3f1ud5/v7vPHjUtMA15ZXcec8mLGFWYnZPsiIsNVIgNiO3BsxHR5OK+fu+9w9yvd/TTgf4Xz9obP28PnzUAVcFoCa41qT0snr767h0XqXhKRUSiRAfEyMN3MpplZFnA1MOBsJDMrM7O+Gm4D7gnnjzGz7L51gIVA5MHto2Llxjp6HRbr6mkRGYUSFhDu3g18HngKWAc87O5vmtk3zezScLVFQLWZbQAmAN8K588CVpnZ6wQHr7896Oyno6Kquo6x+VnMKdfprSIy+iTsNFcAd18BrBg07/aI18uB5VHe9wJwSiJrO5SeXuf5DXWcP2Mc6WmWzFJERJIi2Qeph601NXvZ3dKpwflEZNRSQBxEZXUdaQbnTVdAiMjopIA4iKrqWk6bMoYx+VnJLkVEJCkUEFHUNXWwpqZRZy+JyKimgIji+Q3BVdm6/kFERjMFRBSV1bWML8zmpGOKkl2KiEjSKCAG6e7pZeWGOhbNHIeZTm8VkdFLATHIK+/upam9W6O3isiop4AYpLK6low0Y+H0smSXIiKSVAqIQSrX1zK/YgxFOZnJLkVEJKkUEBF2Nrax/r0mdS+JiKCAGKCqOji9dfGJCggREQVEhMr1tUwuyWX6+IJklyIiknQKiFBHdw9/3lSv01tFREIKiNCqLXto6ezR8QcRkZACIlS5vpas9DTOPqE02aWIiAwLCohQZXUtZx43lryshN5DSURkxFBAAO82tPJ2XYu6l0REIigggKoNtYBObxURiaSAIDj+UFGax7Sy/GSXIiIybIz6gGjv6uGFtxt07wcRkUFGfUDsa+viQydN5EMnTUx2KSIiw8qoP2VnfFEO37/mtGSXISIy7Iz6PQgREYlOASEiIlEpIEREJCoFhIiIRKWAEBGRqBQQIiISlQJCRESiUkCIiEhU5u7JriEuzKwO2HoEmygD6uNUTjyprqFRXUOjuoYmFeua6u7joi1ImYA4Uma2yt3nJ7uOwVTX0KiuoVFdQzPa6lIXk4iIRKWAEBGRqBQQ+92d7AIOQnUNjeoaGtU1NKOqLh2DEBGRqLQHISIiUSkgREQkqlEfEGa2xMyqzWyTmX0l2fX0MbN7zKzWzNYmu5Y+ZnasmVWa2Vtm9qaZ3ZzsmgDMLMfM/mpmr4d1fSPZNUUys3Qze9XMnkh2LZHMbIuZvWFmr5nZqmTX08fMSsxsuZmtN7N1ZnbWMKhpZvjn1PfYZ2ZfTHZdAGb2T+G/+7Vm9qCZ5cRt26P5GISZpQMbgAuBGuBl4Bp3fyuphQFmdh7QDNzn7icnux4AM5sETHL3V8ysEFgNXJ7sPy8zMyDf3ZvNLBP4E3Czu7+UzLr6mNmXgPlAkbtfkux6+pjZFmC+uw+rC7/M7F7gj+7+MzPLAvLcfW+y6+oTthvbgTPd/Uguzo1HLZMJ/r3Pdvc2M3sYWOHu/y8e2x/texALgE3uvtndO4GHgMuSXBMA7r4S2J3sOiK5+053fyV83QSsAyYntyrwQHM4mRk+hsUvHzMrBy4GfpbsWkYCMysGzgP+C8DdO4dTOIQuAN5OdjhEyAByzSwDyAN2xGvDoz0gJgPbIqZrGAYN3khgZhXAacBfkltJIOzGeQ2oBZ5x92FRF/A94H8CvckuJAoHnjaz1Wa2LNnFhKYBdcDPw265n5lZfrKLGuRq4MFkFwHg7tuBu4B3gZ1Ao7s/Ha/tj/aAkMNgZgXAI8AX3X1fsusBcPced58LlAMLzCzp3XJmdglQ6+6rk13LQZzj7qcDFwE3hd2ayZYBnA782N1PA1qA4XRsMAu4FPh1smsBMLMxBL0e04BjgHwzWxqv7Y/2gNgOHBsxXR7Ok4MI+/gfAX7p7o8mu57Bwu6ISmBJsmsBFgKXhn39DwEfNLP7k1vSfuGvT9y9FniMoMs12WqAmog9wOUEgTFcXAS84u67kl1I6G+Ad9y9zt27gEeBs+O18dEeEC8D081sWvjL4Grg8STXNGyFB4P/C1jn7t9Ndj19zGycmZWEr3MJTjpYn9yqwN1vc/dyd68g+Lf1B3eP26+7I2Fm+eGJBoRdOH8LJP2MOXd/D9hmZjPDWRcAST9pJMI1DJPupdC7wAfMLC/8/3kBwbHBuMiI14ZGInfvNrPPA08B6cA97v5mkssCwMweBBYBZWZWA9zh7v+V3KpYCFwPvBH29wN81d1XJLEmgEnAveHZJWnAw+4+rE4pHYYmAI8FbQoZwAPu/mRyS+r3BeCX4Y+2zcCnklwP0B+kFwI3JruWPu7+FzNbDrwCdAOvEsdhN0b1aa4iInJwo72LSUREDkIBISIiUSkgREQkKgWEiIhEpYAQEZGoFBAiw4CZLRpuo72KKCBERCQqBYTIEJjZ0vDeE6+Z2X+GgwQ2m9n/Dcfkf87MxoXrzjWzl8xsjZk9Fo6bg5mdYGbPhveveMXMjg83XxBxH4RfhlfGiiSNAkIkRmY2C/g4sDAcGLAHuA7IB1a5+0nA88Ad4VvuA77s7nOANyLm/xL4obufSjBuzs5w/mnAF4HZwHEEV66LJM2oHmpDZIguAOYBL4c/7nMJhhfvBX4VrnM/8Gh4X4MSd38+nH8v8Otw/KPJ7v4YgLu3A4Tb+6u714TTrwEVBDeDEUkKBYRI7Ay4191vGzDT7OuD1jvc8Ws6Il73oP+fkmTqYhKJ3XPAx8xsPICZjTWzqQT/jz4WrnMt8Cd3bwT2mNm54fzrgefDO/HVmNnl4TayzSzvqH4LkRjpF4pIjNz9LTP7GsFd2NKALuAmgpvaLAiX1RIcpwD4BPCTMAAiRyW9HvhPM/tmuI2/O4pfQyRmGs1V5AiZWbO7FyS7DpF4UxeTiIhEpT0IERGJSnsQIiISlQJCRESiUkCIiEhUCggREYlKASEiIlH9/5hok1py1vSDAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "PcX-5gbGLnBo",
        "outputId": "f467364d-0aa8-45a9-f1ba-9b279cbf22f6"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.plot(history.history['loss'])\n",
        "plt.plot(history.history['val_loss'])\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'test'], loc='upper left')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxc9X3v/9dnFmm0L9biRcYWeEEOi41lA2HJYpvYQFkCJEBI0zatyW1ymza5SaAhyS90ubS5zc2vDSRQwm3SpCw2ELjBBBtiICEslo0hxgteMEjeJNtaLY2kkb73jzOyx/LY1jKjGUnv5+Mxj5k5mz4iznnrfL/n+z3mnENERKQ/X6oLEBGR9KSAEBGRuBQQIiISlwJCRETiUkCIiEhcgVQXkCglJSVu+vTpqS5DRGRUWb9+/UHnXGm8dWMmIKZPn05NTU2qyxARGVXM7P2TrVMTk4iIxKWAEBGRuBQQIiIS15jpg4inu7uburo6wuFwqktJulAoREVFBcFgMNWliMgYMaYDoq6ujry8PKZPn46ZpbqcpHHOcejQIerq6qisrEx1OSIyRozpJqZwOMyECRPGdDgAmBkTJkwYF1dKIjJyxnRAAGM+HPqMl99TREbOmA+I04n09HKgJUxHV0+qSxERSSvjPiDMoL6lk6aOrqQcv6mpifvuu2/Q+1155ZU0NTUloSIRkYEZ9wHh9/nIyfTT2hFJyvFPFhCRyKl/3qpVqygsLExKTSIiAzGm72IaqPxQkL3NHXRGesgM+BN67DvuuIOdO3cyd+5cgsEgoVCIoqIitm7dyrvvvst1111HbW0t4XCYL3/5yyxfvhw4NnVIW1sby5Yt49JLL+X3v/89U6ZM4amnniIrKyuhdYqI9DduAuK7//cdNu9tibvOOUd7Vw8ZAR9B/8AvquZMzuc7f/ShU25zzz33sGnTJjZu3MiLL77IVVddxaZNm47ejvrQQw9RXFxMR0cHCxYs4IYbbmDChAnHHWP79u08/PDD/Pu//zuf+tSnePzxx7ntttsGXKeIyFCMm4A4FTPD5zN6eh3BxF5AnGDhwoXHjVX413/9V5588kkAamtr2b59+wkBUVlZydy5cwGYP38+u3fvTm6RIiKMo4A43V/6+5s7aGjtompSHoFBXEUMVk5OztHPL774Is8//zyvvvoq2dnZfPSjH407liEzM/PoZ7/fT0dHR9LqExHpM+47qfvkhYI4HG2die2szsvLo7W1Ne665uZmioqKyM7OZuvWrbz22msJ/dkiIsMxbq4gTic7w0/A56OlI0JhdkbCjjthwgQuueQSzjnnHLKysigvLz+6bunSpfz4xz+mqqqK2bNnc9FFFyXs54qIDJc551JdQ0JUV1e7/g8M2rJlC1VVVQM+Rt3hdprD3VRNysc3CkcmD/b3FRExs/XOuep469TEFCMvK0hPr6M9wc1MIiKjkQIiRm5mADOjJayAEBFJakCY2VIz22ZmO8zsjjjrv2Jmm83sbTN7wcymxazrMbON0dfTyayzj99n5GUGaAl3M1aa3kREhippAWFmfuBeYBkwB7jFzOb02+xNoNo5dx6wEvjnmHUdzrm50dc1yaqzv7xQgK5IL52R3pH6kSIiaSmZVxALgR3OuV3OuS7gEeDa2A2cc2udc+3Rr68BFUmsZ0Dys7wnsrWEu1NciYhIaiUzIKYAtTHf66LLTubzwLMx30NmVmNmr5nZdfF2MLPl0W1qGhoahl8xEPT7yMrw05KkyftEREaLtOikNrPbgGrgezGLp0VvvboV+IGZndV/P+fcA865audcdWlpacLqyQ8Fae+K0N0z/GamoU73DfCDH/yA9vb2028oIpIEyQyIPcDUmO8V0WXHMbPFwDeBa5xznX3LnXN7ou+7gBeBeUms9Tj5IW/8YGsC7mZSQIjIaJXMkdTrgJlmVokXDDfjXQ0cZWbzgPuBpc65+pjlRUC7c67TzEqASzi+AzupQkE/Qb+Plo5uinOGN6o6drrvJUuWUFZWxmOPPUZnZyfXX3893/3udzly5Aif+tSnqKuro6enh29961scOHCAvXv38rGPfYySkhLWrl2boN9ORGRgkhYQzrmImX0JeA7wAw85594xs7uBGufc03hNSrnAiugzlT+I3rFUBdxvZr14Vzn3OOc2D6ugZ++A/X8Y0KYGnBnpIdLrcBl+jJOMqp54Liy755THip3ue/Xq1axcuZI33ngD5xzXXHMNL7/8Mg0NDUyePJlnnnkG8OZoKigo4Pvf/z5r166lpKRkML+piEhCJHUuJufcKmBVv2Xfjvm8+CT7/R44N5m1nY7fZ3T3OHp6HQFfYqbdWL16NatXr2bePK+1rK2tje3bt3PZZZfx1a9+lW984xtcffXVXHbZZQn5eSIiwzF+Jus7zV/6/fmcY/feFgqzg1QUZSekBOccd955J7fffvsJ6zZs2MCqVau46667WLRoEd/+9rfjHEFEZOSkxV1M6chnRl4oQGs4MqxR1bHTfX/iE5/goYceoq2tDYA9e/ZQX1/P3r17yc7O5rbbbuNrX/saGzZsOGFfEZGRNn6uIIYgPxSkuaObju4esjOG9p8qdrrvZcuWceutt3LxxRcDkJuby89//nN27NjB1772NXw+H8FgkB/96EcALF++nKVLlzJ58mR1UovIiNN036cQ6elly74WSvNDTMwPDbfEpNN03yIyWJrue4gCfh/ZGQFaOzTthoiMPwqI08jPCtDR3UOXJu8TkXFmzAfEcJvQ8kLe5H2taT5531hpKhSR9DGmAyIUCnHo0KFhnTwzAz4yA/60foiQc45Dhw4RCqV/P4mIjB5j+i6miooK6urqGO5Mr80d3bR1RmivD6Xts6pDoRAVFSmfLV1ExpAxHRDBYJDKysphH+e1XYf48wde48e3XcDScyYloDIRkfQ3ppuYEqV6WhEFWUHWbK4//cYiImOEAmIAAn4fH5tdytpt9fT0qjNYRMYHBcQALZ5TzuEjXbz5QWOqSxERGREKiAG6fFYpAZ+xZsuBVJciIjIiFBADlB8KctGZE3hhi/ohRGR8UEAMwuKqMnbUt/HewSOpLkVEJOkUEIOwqKocgBfUzCQi44ACYhCmFmdz9sQ8nldAiMg4oIAYpMVV5azb3Uhze3rPzSQiMlwKiEFaVFVGT6/jxXfVWS0iY5sCYpDOryikJDeTNZvVzCQiY5sCYpB8PmPR2WW8tK1Bz4gQkTFNATEEi+eU09oZYd3uw6kuRUQkaRQQQ3DpjBIyAz41M4nImKaAGIKsDD+Xzijh+S0H9CQ3ERmzFBBDtHhOOXWNHbx7oC3VpYiIJIUCYogWnV0GoEFzIjJmKSCGqCw/xPkVBQoIERmzFBDDsLiqnI21TdS3hlNdiohIwikghmFRVTnOwdqtGlUtImOPAmIYqiblMaUwi+f1jAgRGYMUEMNgZiyuKuO32xsId/ekuhwRkYRSQAzToqpywt29vLLjYKpLERFJKAXEMF14ZjG5mQE1M4nImJPUgDCzpWa2zcx2mNkdcdZ/xcw2m9nbZvaCmU2LWfc5M9sefX0umXUOR2bAz+WzSnhhywF6ezWqWkTGjqQFhJn5gXuBZcAc4BYzm9NvszeBaufcecBK4J+j+xYD3wEuBBYC3zGzomTVOlyLq8qpb+1k097mVJciIpIwybyCWAjscM7tcs51AY8A18Zu4Jxb65xrj359DaiIfv4EsMY5d9g51wisAZYmsdZh+djsMnwGz2vyPhEZQ5IZEFOA2pjvddFlJ/N54NnB7Gtmy82sxsxqGhoahlnu0BXlZFA9rZg16ocQkTEkLTqpzew2oBr43mD2c8494Jyrds5Vl5aWJqe4AVo8p4wt+1rY09SR0jpERBIlmQGxB5ga870iuuw4ZrYY+CZwjXOuczD7ppNFVeUAvKC5mURkjEhmQKwDZppZpZllADcDT8duYGbzgPvxwiG2feY54AozK4p2Tl8RXZa2zirN5cySHD1ESETGjKQFhHMuAnwJ78S+BXjMOfeOmd1tZtdEN/sekAusMLONZvZ0dN/DwN/hhcw64O7osrS2eE45r+06RGu4O9WliIgMWyCZB3fOrQJW9Vv27ZjPi0+x70PAQ8mrLvEWnV3GAy/v4rfbD3LluZNSXY6IyLCkRSf1WDF/WhGF2UE9I0JExgQFRAIF/D4+PruMtVvrifT0procEZFhUUAk2KKqchrbu9nwQVOqSxERGRYFRIJdPquEoN90u6uIjHoKiATLCwW56MwJrFFAiMgop4BIgsVV5exqOMKuhrZUlyIiMmQKiCRYVFUGwAuam0lERjEFRBJUFGVz9sQ8NTOJyKimgEiSJXPKWf9+I41HulJdiojIkCggkmRRVTk9vY4X31Uzk4iMTgqIJDlvSgGleZk8v1kBISKjkwIiSXw+Y3FVGS+920BXRKOqRWT0UUAk0aKzy2nrjPD6e4dSXYqIyKApIJLokhklhII+PataREYlBUQSZWX4uXRGKc9vqcc5l+pyREQGRQGRZIurytjT1MHW/a2pLkVEZFAUEEn28aOjqtXMJCKjiwIiycryQpw/tZA1mnZDREYZBcQIWFJVxlu1TdS3hFNdiojIgCkgRsDiOeUA/GarriJEZPRQQIyA2eV5TCnM0rOqRWRUUUCMADNjyZxyfrv9IB1dPakuR0RkQBQQI2RRVRmdkV5e2XEw1aWIiAyIAmKEXFg5gdzMgJqZRGTUUECMkIyAj4/MLuWFrfX09mpUtYikPwXECFpcVUZDaydv72lOdSkiIqelgBhBH5tdht9nmrxPREYFBcQIKszOoHpakfohRGRUUECMsMVV5Wzd30pdY3uqSxEROSUFxAjrG1X9guZmEpE0N6CAMLMvm1m+eX5iZhvM7IpkFzcWVZbkcGZpjpqZRCTtDfQK4s+ccy3AFUAR8FngnqRVNcYtqSrntV2HaA13p7oUEZGTGmhAWPT9SuA/nXPvxCyTQVo8p5zuHsfL72pUtYikr4EGxHozW40XEM+ZWR7Qe7qdzGypmW0zsx1mdkec9ZdHm6siZnZjv3U9ZrYx+np6gHWOChecUURRdlDNTCKS1gID3O7zwFxgl3Ou3cyKgT891Q5m5gfuBZYAdcA6M3vaObc5ZrMPgD8B/kecQ3Q45+YOsL5Rxe8zPnZ2GS9sqSfS00vAr3sFRCT9DPTMdDGwzTnXZGa3AXcBpxsOvBDY4Zzb5ZzrAh4Bro3dwDm32zn3NgO4Gkmqd56EriMj+iMXV5XT3NHN+vcbR/TniogM1EAD4kdAu5mdD3wV2An87DT7TAFqY77XRZcNVMjMaszsNTO7bhD7DU7Du7Dyz2Dl56F35KbivnxWKRl+n5qZRCRtDTQgIs45h3cF8EPn3L1AXvLKAmCac64auBX4gZmd1X8DM1seDZGahoaGof2U0lmw7J/h3Wfh2a+DG5mJ9HIzA1x01gSNhxCRtDXQgGg1szvxbm99xsx8QPA0++wBpsZ8r4guGxDn3J7o+y7gRWBenG0ecM5VO+eqS0tLB3roEy38C/jwf4d1D8Lv/23oxxmkxVVl7Dp4hJ0NbSP2M0VEBmqgAfFpoBNvPMR+vJP9906zzzpgpplVmlkGcDMwoLuRzKzIzDKjn0uAS4DNp95rmBbfDXOugzXfgk1PJPVH9VlU5Y2q1uR9IpKOBhQQ0VD4BVBgZlcDYefcKfsgnHMR4EvAc8AW4DHn3DtmdreZXQNgZgvMrA64CbjfzN6J7l4F1JjZW8Ba4J5+dz8lns8H198PUy+CJ2+H919N6o8DmFKYxZxJ+WpmEpG0ZG4Abe5m9im8K4YX8QbIXQZ8zTm3MqnVDUJ1dbWrqakZ/oHaD8NPlkD7Ifj8GiiZOfxjnsL3V2/jh2t3UHPXEopzMpL6s0RE+jOz9dH+3hMMtInpm8AC59znnHN/jHcL67cSVWBayS6Gz6wE88PPb4C25P51v3hOOb0O1m7VVYSIpJeBBoTPORd7Bjs0iH1Hn+JKuPUxLxz+69NJHSNxzuQCyvIyeWGr+iFEJL0M9CT/azN7zsz+xMz+BHgGWJW8stJAxXy48Sew9014/M+TNkbC5zMWVZXz0rYGOiMjNw5DROR0BtpJ/TXgAeC86OsB59w3kllYWjj7Km+MxLZV8Ow3kjZGYsmcMo509fD6rsNJOb6IyFAMdC4mnHOPA48nsZb0dOFyaHofXv0hFE3zxksk2IfPKiEU9EZVXz5rGOM5REQS6JRXEGbWamYtcV6tZtYyUkWm3JK/gznXwuq7vHmbEiwU9HPZzFKe33yAgdxVJiIyEk4ZEM65POdcfpxXnnMuf6SKTDmfD65/wBsj8cTt8MFrCf8Ri6vK2NscZsu+1oQfW0RkKMbunUiJFgzBLQ9DQQU8fDMc3JHQw3/87HLM0OR9IpI2FBCDkV0Mt0XHSPziBmgb4gSBcZTmZTJ3aqECQkTShgJisIrPhFsfhdYD8PCnoas9YYdeXFXO23XNHGgJJ+yYIiJDpYAYiopquOFB2LMhoWMkFkcn79PcTCKSDhQQQ1V1NSz7J9j2DPz6joSMkZhVnsvU4ixeUDOTiKQBBcRwXHg7XPwleOMBePXeYR/OzFh0djm/23GQ9q5IAgoUERk6BcRwLfk7qLoGVn8T3vnl8A83p5zOSC+/234wAcWJiAydAmK4fD745AMw9UJ4Yjl88PqwDrdgejF5mQH1Q4hIyikgEiGYBTcnZoxERsDHR2aX8sLWA/T2alS1iKSOAiJRciZEx0jYsMdILJlTzsG2LjbWNSWwQBGRwVFAJFLxmXDLo9C637uSGOIYiY/OKsPvM93NJCIppYBItKkLomMk1sMTfzGkMRIF2UEWTC/i+c3qhxCR1FFAJEPVH8HSe2Drr+DXdw5pjMTiqnK2HWil9nDiRmqLiAyGAiJZLvoCXPRFeON+eO2+Qe/eN6r6X1ZvoyvSm+jqREROSwGRTFf8vTdG4rlvwuanBrXr9JIc/mrRTH65cS+f/cnrNB7pSlKRIiLxKSCSqW+MRMWCIY2R+MqSWfzg03N5s7aJa+99he0H9KwIERk5CohkC2Z5z5HIn+zd2XRo56B2v27eFB5dfhEd3T1cf9/vWbtVHdciMjIUECMhpwQ+Ex0j8fMb4MjgptGYd0YRT33xEs4ozubzP13Hg7/dpUeTikjSKSBGyoSz4JZHoHUf/NfgnyMxuTCLlf/tYq6YM5G/f2YLdzz+B3Vei0hSKSBG0tSF8Ml/H/IYieyMAPd95gL+6uMzeLSmltsefJ1DbZ1JKlZExjsFxEibcw0s/Z/eGInnvjno3X0+4ytXzOZfb5nHW3Ve5/W2/eq8FpHEU0CkwkX/DS76S3j9R/Dq4MdIAFxz/mQevf1iuiK9fPK+VzQth4gknAIiVa74ezj7anjub2Hz00M6xNyphTz1pUuoLM3hz39WwwMv71TntYgkjAIiVXx+rz+iotrrj6h9Y0iHmVSQxYrbP8yV50ziH1dt5esr36YzkphnZIvI+KaASKWMbO/OprxJ3p1Ngxwj0Scrw8+/3TKPLy+ayYr1ddz24OscVOe1iAyTAiLVckrgtse9z7+4cdBjJPr4fMbfLJnFD2+dx9t1zVz7w1fYur8lgYWKyHijgEgHE86CWx+Flr3eaOvujiEf6urzJrPiCxcT6e3lhvt+z5rN6rwWkaFJakCY2VIz22ZmO8zsjjjrLzezDWYWMbMb+637nJltj74+l8w608LUhd68TXU18PifD+k5En3OqyjkqS9eyllluSz/zxp+/JI6r0Vk8JIWEGbmB+4FlgFzgFvMbE6/zT4A/gT4r377FgPfAS4EFgLfMbOiZNWaNuZcC5/4B2+MxOq7hnWoiQUhHl1+MVedO4l7nt3KV1e8pc5rERmUQBKPvRDY4ZzbBWBmjwDXApv7NnDO7Y6u6z9nxCeANc65w9H1a4ClwMNJrDc9XPSX0PSB9wyJwjO8MRND1Nd5Pas8j++veZfdB49w/2erKc3LTGDBIjJWJbOJaQpQG/O9LrosYfua2XIzqzGzmoaGhiEXmlbM4BP/6I2R+PWd8Ku/gZ2/gcjQngdhZvzVopnc95kL2LyvhevufYXNe9V5LSKnN6o7qZ1zDzjnqp1z1aWlpakuJ3H6xkic9ynY+DD85/XwvRmw8vOw6QkID/4Ef+W5k1j5hQ/T0+u48ce/57l39iehcBEZS5IZEHuAqTHfK6LLkr3v2JCR7XVaf30X3Pyw95zrXWth5Z/C987ypg2veQhaB36iP2dKAU9/6RJmluXyhZ+v5961O9R5LSInZck6QZhZAHgXWIR3cl8H3OqceyfOtv8B/Mo5tzL6vRhYD1wQ3WQDML+vTyKe6upqV1NTk9DfIe309kDt67D1Ga8ju3G3t7xiAcy+0muWKp112sOEu3v4+sq3efqtvVw/bwr/85PnEgr6k1u7iKQlM1vvnKuOuy6Zf0Ga2ZXADwA/8JBz7h/M7G6gxjn3tJktAJ4EioAwsN8596Hovn8G/G30UP/gnPs/p/pZ4yIgYjkH9VuOhcW+jd7yCTPh7Ku815Rq77GncXd33Lt2B/9r9bvMO6OQ+z87n7K80Aj+AiKSDlIWECNp3AVEf811sO1ZLyx2/w56I5BbDrOXweyroPJyCJ4YAL/etI+/efQtirKDPPDH1ZwzpSAFxYtIqiggxpuOJti+xguLHc9DVxtk5MKMxV4z1MwlkFV4dPNNe5r5i5/V0NTezf/+9PksPWdSCosXkZGkgBjPusOw+7deWGxdBUfqwReA6Zd6YTF7GRRUUN8aZvnP1rOxton/ccUsvvixGZhZqqsXkSRTQIintxf21Bzrtzi0w1s+aS6cfTWdM5byjZe7+eVb+7jm/Mn8843nqfNaZIxTQEh8De96QbFtFdStA8AVTeetnEv4x11n0T15Aff/8ULK8tV5LTJWKSDk9Fr3e0GxdRW89xL0dHHY5fGKfwHnLb6VaQuuhmBWqqsUkQRTQMjghFtgx/M0v/lLfDvXkEc7EX8WgZmLvNtnZy2F7OJUVykiCXCqgEjmZH0yWoXy4ZxPUnDOJ2loauX7//EfVB58iet3vUHe1l+B+aD4TO9VVHnsc3ElFE6DQEaqfwMRSQBdQchpdUZ6uPOJP/DEhjr+clYrXzljJ4FD26DxPTj8nncbbR/zQUFFTHDEBEjRdMjISdnvISIn0hWEDEtmwM+/3HQ+s8rz+Kdfb+V37R/lX276MjPL87wR3UcavKA4vMt7NUY/b/4ldDQef7DciTHBUXn8VUjM2AwRST1dQcigrNl8gL9+5E2OdPUwd2ohN86v4I/On0xBVjD+Dh2Nx8Kj74qj73tbv4kGs4r7XXHEXIXklHpToYtIQqmTWhLqYFsnv3xzDytq6th2oJXMgI9PfGgiN86v4JIZJfh9AzyRdx3xJhzsu/I4ehXyHjTXAjH/NjNyvaDo3+dRfCbkTT7pnFND4hz0dENvd/Q9cvz3k66LQE9X/HXBbMgqglCh955V5F0x+U8SrDK29UQg0uENZO1uh0jYexZ9JPq9O3xsfaTDWxe7rP8+RZWw7J4hlaKAkKRwzrFpTwsr1tfy1Ma9NHd0M6kgxA0XVHDD/AoqS4bR3xDp9J6sFxscfU1Xje97J90+/kyvf6O40rsV93Qn8d5u70R+9HO/bXojw/5vM2AZucfCIl6AHP3cb11Gjq6ohsq56P/eXd4r0nXsc0839HRG32OWRTqj23ae4sQ+gJN9d4f3eaj/xszv/RsPhLw/OoIh7/Ok8+Dae4d2SAWEJFu4u4cXttSzYn0tL7/bQK+DBdOLuHF+BVedN5nczAR2d/X2eJMTHtfnEX31dHl/lfsC0feg9x77+bh1AfBnHPvsC55kv0C//TPirzvZz4l0eM1tHY3eXFnHvTdCOOZz36vnFE8R9AVPHSAnWxcq8OodjqNXWNGA7e2JCdjI8a+evvUx63pit4mu77/vcSfo2JN39CTd/wR+0pN6vO2G9nTG+CzmRJ0V8551kmV9J/eY9xOWnex4WUm54lRAyIg60BLmiQ17WLG+ll0NR8gK+ll2rtcEdVHlBHwDbYIaz5zz/to8aYD0+x6OCZ3O0zxxMLMAsgq8wPAFB3CSjz2ZR8D1f4R8shkEMo+Fsj+j3ysYZ33Qu7KMXRbI7Lc+I7pNzDED/Y57wjEyvBN43wndnzHqr+QUEJISzjnerG1iRU0dv3prL62dEaYWZ3lNUBdUMLU4O9Uljk09EQg3nyRA+gVMbyTmSsfvBYYvEF0Wfff1revbLnD8tv6Y9UeXBY4dJ+7+Ma+4PzvmpO3zj/qTcDpTQEjKhbt7eO6d/ayoqeOVnQdxDi4+cwI3VVew9JyJZGfojmuRVFBASFrZ09TBE+vrWLmhjvcPtZObGeCqcydxU3UF86cVaZpxkRGkgJC05Jxj3e5GVtTU8swf9tHe1UNlSQ43zq/gkxdMYVKBJgcUSTYFhKS9I50Rnt20nxU1tbz+3mHM4NIZJdxUPZUr5pTruRQiSaKAkFHlg0PtrNxQx+Pr69jT1EFeKMA150/mpuqpnF9RoCYokQRSQMio1NvreG3XIVasr+PZTfsId/cysyyXG+dXcP0FUyjL04OMRIZLASGjXku4m1Vv72PF+jrWv9+I32d8ZFYpN82vYFFVORmBBE61ITKOKCBkTNnZ0Mbj6+t4YsMe9reEKcoOcu3cKdw4v4JzphSkujyRUUUBIWNST6/jdzsOsqKmltWbD9AV6WVyQYgFlcUsmF7MwspiZpTmauS2yCnoeRAyJvU1M31kVinN7d0884d9vLLzIK/uPMRTG/cCUJgdpHpaEQumF7OgsphzJheoOUpkgHQFIWOOc47awx28sfsw6947zLrdh9l18AgAoaCPuVMLWRgNjAvOKCInkRMJiowyamKSca+htZOa3YdZt7uRdbsP887eZnqddxXyocn53hXG9CKqpxdTkpuZ6nJFRowCQqSfts4IG973wuKN9w6zsbaJzog3S+mZpTneFUb0NbU4S2MvZMxSQIicRmekh017WlgX0yzVEvYe6lKen3m003vB9GJml+ep41vGDAWEyCD19jrerW/1mqSigbGvOQxAXoH6owMAAAx9SURBVCjgdXxXFrNwejHnVhSQGdBUIDI66S4mkUHy+YyzJ+Zz9sR8PnvRNJxz1DV2eFcY0WaptdsaAMgIeB3fC6Z7d0vNn1ZEXkjPmpbRT1cQIkN0qK2TmvePXWFs2ttCT6/DZ1A1Kf9oH8aCyiJNCyJpS01MIiPgSGeEjbVNvBENjDc/aKKjuwfw+jFmlOUyozSXGWW5nFXmvZfmZqoDXFJKTUwiIyAnM8AlM0q4ZEYJAN09vWza00zN7ka27G9hZ30bK9fXcaSr5+g++aGAFxyxr9I8Koqy1BEuKZfUgDCzpcD/D/iBB51z9/Rbnwn8DJgPHAI+7ZzbbWbTgS3AtuimrznnvpDMWkUSLej3Me+MIuadUXR0mXOO/S1hdtS3Hff6zdZ6HqupO7pdZsDHmaW5x111zCjLZXpJtjrEZcQkLSDMzA/cCywB6oB1Zva0c25zzGafBxqdczPM7Gbgn4BPR9ftdM7NTVZ9IqlgZkwqyGJSQRaXzSw9bl1Te9fxwdHQxpsfNPJ/39p7dBu/zzijOJuzSnNPuPLI1YhwSbBk/otaCOxwzu0CMLNHgGuB2IC4Fvj/op9XAj80NcjKOFWYnUH19GKqpxcft7yjq4edDW3sbDgWHtvr23hxWz2R3mN9iBPzQyc2V5XlMiEnQ/0cMiTJDIgpQG3M9zrgwpNt45yLmFkzMCG6rtLM3gRagLucc7/t/wPMbDmwHOCMM85IbPUiaSIrw885UwpOmMq8u6eX9w+1s6P++PB4rKaW9ph+jsLs4HHNVGdFm62mFKqfQ04tXa9J9wFnOOcOmdl84Jdm9iHnXEvsRs65B4AHwLuLKQV1iqRM0O87etKP1dvr2Nevn2NnfRurNx/gkXXH/mbLCvqZWpzFxIIsJheEvKavwhCTo++TCkJkZ6TrKUJGQjL/198DTI35XhFdFm+bOjMLAAXAIefde9sJ4Jxbb2Y7gVmA7mMVOQ2fz5hSmMWUwiw+Muv4fo7DR47v56hrbGdfc5jNe5s52NZ1wrEKsoJMKggxuTDr6PvE/NDRIJlYECIUVKf5WJXMgFgHzDSzSrwguBm4td82TwOfA14FbgR+45xzZlYKHHbO9ZjZmcBMYFcSaxUZF4pzMlhY6c0r1V+4u4cDLWH2NYfZ19zB3ibvfV9TmL3NYd78oJHG9u4T9puQk8GkwhAT87OYXOhdifS9TyoIUZ4f0jM4RqmkBUS0T+FLwHN4t7k+5Jx7x8zuBmqcc08DPwH+08x2AIfxQgTgcuBuM+sGeoEvOOcOJ6tWEYFQ0M+0CTlMm5Bz0m06unq80GgOs7epg/3NXnjsa+6g9nA7r793iNboJId9zKAkN/O4ZqxJBccHSVleJgG/QiTdaCS1iCRUW2eE/TFXIEevRJqjVydNHccNFgTwGZTlHd90VZ6fSXm+dwUyMfqelaHmrETTSGoRGTG5mQFmlOUxoywv7nrnHC3hyNHmq/5NWpv3tfCbrfVHpymJlR8KHA0N75XJxIIQZXmho6FSkptJUFcjCaGAEJERZWYUZAUpyApy9sT8uNs452jtjFDfEmZ/cycHWsLsbwlT3xLmQEsn+1vC7Np5kPrWzuPGgnjH95q0yvMzmZgfoiw/RHleiIkFmZTFXI0UZQc1PuQ0FBAiknbMjPxQkPxQ8KRXIuDd0nvoSBcHWsLRV+fRINnfEmZPU5g3P2ji0JET79DK8PsoizZjeUGSeTQ8+q5OyvND4/qZ5eP3NxeRUc/nM0rzMinNyzxhIGGszkgPDa2dR0Pk2BVJJ/ubw2zZ38KL28In9I0A5GUGvPAoCFGSm0lRdgaF2cHj3ouyMyjK8T5nZ/jHzJWJAkJExrzMgJ+KomwqirJPuV1bZ8QLkeYwB1qjVyTNYepbw+xvDrOxtonGI11HH0cbT4bfd2KA5BwLkthlhdFlBVlB/Gk4ql0BISISlZsZILc0l7NKc0+5XaSnl6aObprau2hs76bxSBdN7d00xnxvbPeW7Wxoo/F9b9v+/SV9zLxBifGvToIU5WSccMVSmB1M+iBFBYSIyCAF/D5Kcr07pgbKOUdbZ4TGI31BcnyoNLV3cTgaNAdawmzb30pje9dx82r1lxX0U5yTwQXTivi3W+Yl4lc7jgJCRGQEmBl5oSB5oSBnTDh1U1escHdPTJAcC5Wm6JXK4fYuJhUk55G2CggRkTQWCvqZWOBnYpJC4FQ0mkREROJSQIiISFwKCBERiUsBISIicSkgREQkLgWEiIjEpYAQEZG4FBAiIhLXmHminJk1AO8P4xAlwMEElZNIqmtwVNfgqK7BGYt1TXPOlcZbMWYCYrjMrOZkj91LJdU1OKprcFTX4Iy3utTEJCIicSkgREQkLgXEMQ+kuoCTUF2Do7oGR3UNzriqS30QIiISl64gREQkLgWEiIjENe4DwsyWmtk2M9thZnekup4+ZvaQmdWb2aZU19LHzKaa2Voz22xm75jZl1NdE4CZhczsDTN7K1rXd1NdUywz85vZm2b2q1TXEsvMdpvZH8xso5nVpLqePmZWaGYrzWyrmW0xs4vToKbZ0f9Ofa8WM/vrVNcFYGZ/E/13v8nMHjazhD1ZaFz3QZiZH3gXWALUAeuAW5xzm1NaGGBmlwNtwM+cc+ekuh4AM5sETHLObTCzPGA9cF2q/3uZmQE5zrk2MwsCvwO+7Jx7LZV19TGzrwDVQL5z7upU19PHzHYD1c65tBr4ZWY/BX7rnHvQzDKAbOdcU6rr6hM9b+wBLnTODWdwbiJqmYL3732Oc67DzB4DVjnn/iMRxx/vVxALgR3OuV3OuS7gEeDaFNcEgHPuZeBwquuI5Zzb55zbEP3cCmwBpqS2KnCetujXYPSVFn/5mFkFcBXwYKprGQ3MrAC4HPgJgHOuK53CIWoRsDPV4RAjAGSZWQDIBvYm6sDjPSCmALUx3+tIgxPeaGBm04F5wOuprcQTbcbZCNQDa5xzaVEX8APg60BvqguJwwGrzWy9mS1PdTFRlUAD8H+izXIPmllOqovq52bg4VQXAeCc2wP8L+ADYB/Q7Jxbnajjj/eAkCEws1zgceCvnXMtqa4HwDnX45ybC1QAC80s5c1yZnY1UO+cW5/qWk7iUufcBcAy4IvRZs1UCwAXAD9yzs0DjgDp1DeYAVwDrEh1LQBmVoTX6lEJTAZyzOy2RB1/vAfEHmBqzPeK6DI5iWgb/+PAL5xzT6S6nv6izRFrgaWprgW4BLgm2tb/CPBxM/t5aks6JvrXJ865euBJvCbXVKsD6mKuAFfiBUa6WAZscM4dSHUhUYuB95xzDc65buAJ4MOJOvh4D4h1wEwzq4z+ZXAz8HSKa0pb0c7gnwBbnHPfT3U9fcys1MwKo5+z8G462JraqsA5d6dzrsI5Nx3v39ZvnHMJ++tuOMwsJ3qjAdEmnCuAlN8x55zbD9Sa2ezookVAym8aiXELadK8FPUBcJGZZUf//7kIr28wIQKJOtBo5JyLmNmXgOcAP/CQc+6dFJcFgJk9DHwUKDGzOuA7zrmfpLYqLgE+C/wh2t4P8LfOuVUprAlgEvDT6N0lPuAx51xa3VKahsqBJ71zCgHgv5xzv05tSUf9d+AX0T/adgF/muJ6gKNBugS4PdW19HHOvW5mK4ENQAR4kwROuzGub3MVEZGTG+9NTCIichIKCBERiUsBISIicSkgREQkLgWEiIjEpYAQSQNm9tF0m+1VRAEhIiJxKSBEBsHMbos+e2Kjmd0fnSSwzcz+d3RO/hfMrDS67Vwze83M3jazJ6Pz5mBmM8zs+ejzKzaY2VnRw+fGPAfhF9GRsSIpo4AQGSAzqwI+DVwSnRiwB/gMkAPUOOc+BLwEfCe6y8+AbzjnzgP+ELP8F8C9zrnz8ebN2RddPg/4a2AOcCbeyHWRlBnXU22IDNIiYD6wLvrHfRbe9OK9wKPRbX4OPBF9rkGhc+6l6PKfAiui8x9Ncc49CeCcCwNEj/eGc64u+n0jMB3vYTAiKaGAEBk4A37qnLvzuIVm3+q33VDnr+mM+dyD/v8pKaYmJpGBewG40czKAMys2Mym4f3/6MboNrcCv3PONQONZnZZdPlngZeiT+KrM7ProsfINLPsEf0tRAZIf6GIDJBzbrOZ3YX3FDYf0A18Ee+hNguj6+rx+ikAPgf8OBoAsbOSfha438zujh7jphH8NUQGTLO5igyTmbU553JTXYdIoqmJSURE4tIVhIiIxKUrCBERiUsBISIicSkgREQkLgWEiIjEpYAQEZG4/h+5CbjrSFlemAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XfsL1aQGziS3"
      },
      "source": [
        "#### Evaluating the model performance"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kixoMHHOLnBo"
      },
      "source": [
        "raw_test_predictions = model_2.predict(X_test)\n",
        "test_pred = np.argmax(raw_test_predictions, axis=-1)   #taking index  of maximum probab\n",
        "\n",
        "y_test_labels = np.argmax(y_test, axis=-1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lGfv3TfYLnBo",
        "outputId": "89ebb587-bd84-45e7-ce86-c86f3064378f"
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "test_pred_fl = tf.reshape(test_pred, (test_pred.shape[0]*test_pred.shape[1]))\n",
        "test_pred_fl.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([408088])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 107
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nkqE5aHjLnBp",
        "outputId": "b0861a53-5d50-4ae7-d5df-7b68b11b4276"
      },
      "source": [
        "y_test_labels_fl = tf.reshape(y_test_labels, (y_test_labels.shape[0]*y_test_labels.shape[1]))\n",
        "y_test_labels_fl.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([408088])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 108
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MadBvyBhLnBp",
        "outputId": "0c3c0c2d-2d1d-4943-a909-aaf563e840ec"
      },
      "source": [
        "from sklearn.metrics import f1_score, accuracy_score\n",
        "\n",
        "f1_score(y_test_labels_fl, test_pred_fl, average=\"weighted\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9831531934280401"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "je2Ffn7bLnBq",
        "outputId": "eb0c0aae-76c6-4b6b-d35c-1a4a4b3e2840"
      },
      "source": [
        "i = np.random.randint(0, X_test.shape[0])\n",
        "p = model.predict(np.array([X_test[i]]))\n",
        "p = np.argmax(p, axis=-1)\n",
        "print(\"{:14} ({:5}): {}\".format(\"Word\", \"True\", \"Pred\"))\n",
        "for w, true_tag, pred in zip(X_test[i], y_test_labels[i], p[0]):\n",
        "    print(\"{:14}: {}\\t {}\".format(words[w], tags[true_tag], tags[pred]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Word           (True ): Pred\n",
            "the           : O\t O\n",
            "miami         : B-geo\t B-geo\n",
            "herald        : I-geo\t O\n",
            "newspaper     : O\t O\n",
            "reports       : O\t O\n",
            "the           : O\t O\n",
            "prison        : O\t O\n",
            "upheaval      : O\t O\n",
            "was           : O\t O\n",
            "quashed       : O\t O\n",
            "by            : O\t O\n",
            "mid-afternoon : B-tim\t B-tim\n",
            "when          : O\t O\n",
            "u.n.          : B-org\t B-org\n",
            "troops        : O\t O\n",
            "blocked       : O\t O\n",
            "off           : O\t O\n",
            "streets       : O\t O\n",
            "surrounding   : O\t O\n",
            ".             : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n",
            "EOS           : O\t O\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dNQaktsizpWX"
      },
      "source": [
        "#### Classification report for class-wise performance of the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tidT_qghOkmD"
      },
      "source": [
        "y_pred = [[idx2tag[i] for i in row] for row in test_pred]\n",
        "y_test_true = [[idx2tag[i] for i in row] for row in y_test_labels]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fEKTcg28OpJX",
        "outputId": "9c136864-b2b4-4708-fdb4-4922014caf67"
      },
      "source": [
        "from sklearn_crfsuite.metrics import flat_classification_report\n",
        "\n",
        "report = flat_classification_report(y_pred=y_pred, y_true=y_test_true)\n",
        "print(report)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "       B-art       0.67      0.06      0.11        66\n",
            "       B-eve       0.56      0.25      0.34        40\n",
            "       B-geo       0.83      0.85      0.84      5464\n",
            "       B-gpe       0.95      0.93      0.94      2239\n",
            "       B-nat       0.43      0.23      0.30        26\n",
            "       B-org       0.73      0.63      0.68      2931\n",
            "       B-per       0.76      0.81      0.78      2409\n",
            "       B-tim       0.88      0.87      0.87      2927\n",
            "       I-art       0.00      0.00      0.00        40\n",
            "       I-eve       0.47      0.24      0.32        33\n",
            "       I-geo       0.79      0.68      0.73      1098\n",
            "       I-gpe       0.80      0.48      0.60        25\n",
            "       I-nat       0.17      0.14      0.15         7\n",
            "       I-org       0.69      0.57      0.63      2361\n",
            "       I-per       0.84      0.80      0.82      2487\n",
            "       I-tim       0.80      0.65      0.71       878\n",
            "           O       0.99      1.00      0.99    385057\n",
            "\n",
            "    accuracy                           0.98    408088\n",
            "   macro avg       0.67      0.54      0.58    408088\n",
            "weighted avg       0.98      0.98      0.98    408088\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QdvVIe-0zz3u"
      },
      "source": [
        "### **Final Remarks**\n",
        "\n",
        "I think the model performance in both the cases is acceptable as we are using the conservative approach of BiLSTM modelling for NER task. To improve the performance furthermore, we can make use of CRF layers along with BiLSTMs in the model. It is the SOTA in the field of NLP-NER tasks."
      ]
    }
  ]
}