Edureka_Sequence_Learning_Assignment_1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Edureka Sequence Learning - Assignment 1.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "1ZsK2QorQ4Qd1aUghvGDQ8_6BTyL2l-kX",
      "authorship_tag": "ABX9TyPvuvoeSchil8N/ItzQDIGu",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nandinib1999/edureka-assignments/blob/main/Edureka_Sequence_Learning_Assignment_1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XhU0SLi40RyD"
      },
      "source": [
        "## Assignment 1 - Sequence Learning\n",
        "\n",
        "### **Problem Statement**\n",
        "You have been given a small dataset of sentences that are from a sports newspaper (HMM_Train_Sentences.txt), and you are also provided with the NER tagging of these sentences in a separate file (HMM_Train_NER.txt). The objective is to build a Named-entity recognition model using the Hidden Markov Model.\n",
        "\n",
        "#### **Question 1.1** \n",
        "\n",
        "- Read both the input data files and create two lists each for Words and NER tags. \n",
        "- Also, create the NER tags list for start and end token. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tIxHGBvgYbaa"
      },
      "source": [
        "with open(\"/content/drive/MyDrive/HMM_Train_Sentences.txt\") as fsent:\n",
        "  sentences = fsent.readlines()\n",
        "\n",
        "with open(\"/content/drive/MyDrive/HMM_Train_NER.txt\") as fner:\n",
        "  ner_tags = fner.readlines()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yUWBCKD7v7yz",
        "outputId": "fc05bf84-9f3d-44e9-f1e8-26bd9e60b6b7"
      },
      "source": [
        "sentences = [x.strip() for x in sentences]\n",
        "sentences"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Bailey named Australia captain',\n",
              " 'Starc player of 2015 World Cup',\n",
              " 'Australia won 2003 2007 2015 Cups',\n",
              " 'Melbourne Starc Warner knocks etched in memory',\n",
              " '2003 SA 2007 WI 2015 Australia were venues',\n",
              " 'Starc Warner Melbourne go as great combination']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zItsdvD6v9gj",
        "outputId": "9dbab8fd-6709-4fca-f367-0182e127f6b9"
      },
      "source": [
        "ner_tags = [x.strip() for x in ner_tags]\n",
        "ner_tags"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['PER O GEO O',\n",
              " 'PER O O TIM O O',\n",
              " 'GEO O TIM TIM TIM O',\n",
              " 'GEO PER PER O O O O',\n",
              " 'TIM GEO TIM GEO TIM GEO O O',\n",
              " 'PER PER GEO O O O O']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Lp3pGBBrwdnU",
        "outputId": "4afa3e45-64f5-4b92-9c59-2cc05bb2397e"
      },
      "source": [
        "start_tags = [x.split(\" \")[0] for x in ner_tags]\n",
        "start_tags"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['PER', 'PER', 'GEO', 'GEO', 'TIM', 'PER']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GNminmK_4Vr2",
        "outputId": "cc16fa42-e526-497d-a6e1-52c99de5a3f6"
      },
      "source": [
        "end_tags = [x.split(\" \")[-1] for x in ner_tags]\n",
        "end_tags"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['O', 'O', 'O', 'O', 'O', 'O']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oV0OOade0dqA"
      },
      "source": [
        "#### **Question 1.2** \n",
        "\n",
        "- Write functions for creating unigram, bigram tokens only for NER tags. \n",
        "- Also, calculate the NER Word count."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7jHbBTm-wpNR",
        "outputId": "ef3ae6c2-9492-4d01-9b80-f6c1cb9d95fa"
      },
      "source": [
        "ner_words = []\n",
        "for sentence, tag in zip(sentences, ner_tags):\n",
        "  sent = sentence.split(\" \")\n",
        "  tags = tag.split(\" \")\n",
        "  tmp = []\n",
        "  for t, word in zip(tags, sent):\n",
        "    if t == \"O\":\n",
        "      if len(tmp) > 0:\n",
        "        ner_words.append(\" \".join(tmp))\n",
        "        tmp = []\n",
        "    else:\n",
        "      tmp.append(t)\n",
        "\n",
        "ner_words"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['PER',\n",
              " 'GEO',\n",
              " 'PER',\n",
              " 'TIM',\n",
              " 'GEO',\n",
              " 'TIM TIM TIM',\n",
              " 'GEO PER PER',\n",
              " 'TIM GEO TIM GEO TIM GEO',\n",
              " 'PER PER GEO']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "61zsa88tyJBv"
      },
      "source": [
        "def unigrams(phrase):\n",
        "  lst = phrase.split(\" \")\n",
        "  return lst\n",
        "\n",
        "def bigram(phrase):\n",
        "  bigrams = []\n",
        "  lst = phrase.split()\n",
        "  for i, word in enumerate(lst):\n",
        "    if i + 1 < len(lst):\n",
        "      bigrams.append((word, lst[i+1]))\n",
        "  return bigrams"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xWA-7d32ytYe",
        "outputId": "3e1e8354-6c02-42ef-a186-99fe105320a4"
      },
      "source": [
        "for ner_ph in ner_words:\n",
        "  print(\"NER words:\", ner_ph)\n",
        "  print(\"Unigrams:\", unigrams(ner_ph))\n",
        "  print(\"Bigrams:\", bigram(ner_ph) if len(bigram(ner_ph)) else \"NA\")\n",
        "  print(\"------------------\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "NER words: PER\n",
            "Unigrams: ['PER']\n",
            "Bigrams: NA\n",
            "------------------\n",
            "NER words: GEO\n",
            "Unigrams: ['GEO']\n",
            "Bigrams: NA\n",
            "------------------\n",
            "NER words: PER\n",
            "Unigrams: ['PER']\n",
            "Bigrams: NA\n",
            "------------------\n",
            "NER words: TIM\n",
            "Unigrams: ['TIM']\n",
            "Bigrams: NA\n",
            "------------------\n",
            "NER words: GEO\n",
            "Unigrams: ['GEO']\n",
            "Bigrams: NA\n",
            "------------------\n",
            "NER words: TIM TIM TIM\n",
            "Unigrams: ['TIM', 'TIM', 'TIM']\n",
            "Bigrams: [('TIM', 'TIM'), ('TIM', 'TIM')]\n",
            "------------------\n",
            "NER words: GEO PER PER\n",
            "Unigrams: ['GEO', 'PER', 'PER']\n",
            "Bigrams: [('GEO', 'PER'), ('PER', 'PER')]\n",
            "------------------\n",
            "NER words: TIM GEO TIM GEO TIM GEO\n",
            "Unigrams: ['TIM', 'GEO', 'TIM', 'GEO', 'TIM', 'GEO']\n",
            "Bigrams: [('TIM', 'GEO'), ('GEO', 'TIM'), ('TIM', 'GEO'), ('GEO', 'TIM'), ('TIM', 'GEO')]\n",
            "------------------\n",
            "NER words: PER PER GEO\n",
            "Unigrams: ['PER', 'PER', 'GEO']\n",
            "Bigrams: [('PER', 'PER'), ('PER', 'GEO')]\n",
            "------------------\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vp_U6Rea1c5O",
        "outputId": "dd18832e-04f9-46e9-e488-922f77a90197"
      },
      "source": [
        "from collections import defaultdict\n",
        "\n",
        "ner_word_count = {}\n",
        "\n",
        "for sentence, tag_ls in zip(sentences, ner_tags):\n",
        "  sent = sentence.split(\" \")\n",
        "  tags = tag_ls.split(\" \")\n",
        "  for tag, word in zip(tags, sent):\n",
        "    if tag not in ner_word_count:\n",
        "      ner_word_count[tag] = defaultdict(int)\n",
        "      ner_word_count[tag][word] = 1\n",
        "    else:\n",
        "      if word not in ner_word_count[tag]:\n",
        "        ner_word_count[tag][word] = 1\n",
        "      else:\n",
        "        ner_word_count[tag][word] += 1\n",
        "\n",
        "ner_word_count"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'GEO': defaultdict(int, {'Australia': 3, 'Melbourne': 2, 'SA': 1, 'WI': 1}),\n",
              " 'O': defaultdict(int,\n",
              "             {'Cup': 1,\n",
              "              'Cups': 1,\n",
              "              'World': 1,\n",
              "              'as': 1,\n",
              "              'captain': 1,\n",
              "              'combination': 1,\n",
              "              'etched': 1,\n",
              "              'go': 1,\n",
              "              'great': 1,\n",
              "              'in': 1,\n",
              "              'knocks': 1,\n",
              "              'memory': 1,\n",
              "              'named': 1,\n",
              "              'of': 1,\n",
              "              'player': 1,\n",
              "              'venues': 1,\n",
              "              'were': 1,\n",
              "              'won': 1}),\n",
              " 'PER': defaultdict(int, {'Bailey': 1, 'Starc': 3, 'Warner': 2}),\n",
              " 'TIM': defaultdict(int, {'2003': 2, '2007': 2, '2015': 3})}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "StWAXjqe5Wbw",
        "outputId": "a76dcbb4-392a-4f1d-e624-087e244b7489"
      },
      "source": [
        "from collections import Counter\n",
        "\n",
        "start_ner_count = Counter(start_tags)\n",
        "print(\"Unique start NER tags:\", start_ner_count)\n",
        "\n",
        "end_ner_count = Counter(end_tags)\n",
        "print(\"Unique end NER tags:\", start_ner_count)\n",
        "\n",
        "ner_lst_unigrams = []\n",
        "for x in ner_tags:\n",
        "  ner_lst_unigrams.extend(unigrams(x))\n",
        "unigram_count = Counter(ner_lst_unigrams)\n",
        "print(\"Unique unigrams NER tags:\", unigram_count)\n",
        "\n",
        "ner_lst_bigrams = []\n",
        "for x in ner_tags:\n",
        "  lst = bigram(x)\n",
        "  if len(lst) > 0:\n",
        "    ner_lst_bigrams.extend(lst)\n",
        "bigram_count = Counter(ner_lst_bigrams)\n",
        "print(\"Unique bigrams NER tags:\", bigram_count)"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Unique start NER tags: Counter({'PER': 3, 'GEO': 2, 'TIM': 1})\n",
            "Unique end NER tags: Counter({'PER': 3, 'GEO': 2, 'TIM': 1})\n",
            "Unique unigrams NER tags: Counter({'O': 18, 'GEO': 7, 'TIM': 7, 'PER': 6})\n",
            "Unique bigrams NER tags: Counter({('O', 'O'): 9, ('GEO', 'O'): 4, ('PER', 'O'): 3, ('TIM', 'GEO'): 3, ('O', 'TIM'): 2, ('TIM', 'O'): 2, ('TIM', 'TIM'): 2, ('PER', 'PER'): 2, ('GEO', 'TIM'): 2, ('O', 'GEO'): 1, ('GEO', 'PER'): 1, ('PER', 'GEO'): 1})\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_mtmk7pH6qYD",
        "outputId": "70cde875-7093-4b9b-aae1-9fe3f6753cab"
      },
      "source": [
        "ner_list = []\n",
        "for ner in ner_tags:\n",
        "  ner_list.extend(ner.split(\" \"))\n",
        "ner_list = [x for x in ner_list if len(x) > 0]\n",
        "ner_list"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['PER',\n",
              " 'O',\n",
              " 'GEO',\n",
              " 'O',\n",
              " 'PER',\n",
              " 'O',\n",
              " 'O',\n",
              " 'TIM',\n",
              " 'O',\n",
              " 'O',\n",
              " 'GEO',\n",
              " 'O',\n",
              " 'TIM',\n",
              " 'TIM',\n",
              " 'TIM',\n",
              " 'O',\n",
              " 'GEO',\n",
              " 'PER',\n",
              " 'PER',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'TIM',\n",
              " 'GEO',\n",
              " 'TIM',\n",
              " 'GEO',\n",
              " 'TIM',\n",
              " 'GEO',\n",
              " 'O',\n",
              " 'O',\n",
              " 'PER',\n",
              " 'PER',\n",
              " 'GEO',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z1Mz84YY0wqj"
      },
      "source": [
        "#### Question 1.3 \n",
        "\n",
        "- Create HiddenMarkovModel (from pomegranate library) to identify entities for words. \n",
        "- Calculate the start, end, and emission probabilities"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 445
        },
        "id": "zUpmWCZg28ar",
        "outputId": "25b7d173-848d-4a49-e0c6-adf8b9af22f8"
      },
      "source": [
        "!pip install pomegranate"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting pomegranate\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bd/b9/b64c8722891d9089959b403b8b6a5413c325ed2f6b07f2b2d5246a151551/pomegranate-0.14.5-cp37-cp37m-manylinux2010_x86_64.whl (17.9MB)\n",
            "\u001b[K     |████████████████████████████████| 17.9MB 207kB/s \n",
            "\u001b[?25hRequirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.7/dist-packages (from pomegranate) (1.4.1)\n",
            "Requirement already satisfied: networkx>=2.4 in /usr/local/lib/python3.7/dist-packages (from pomegranate) (2.5.1)\n",
            "Requirement already satisfied: joblib>=0.9.0b4 in /usr/local/lib/python3.7/dist-packages (from pomegranate) (1.0.1)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from pomegranate) (3.13)\n",
            "Collecting numpy>=1.20.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a5/42/560d269f604d3e186a57c21a363e77e199358d054884e61b73e405dd217c/numpy-1.20.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.3MB)\n",
            "\u001b[K     |████████████████████████████████| 15.3MB 279kB/s \n",
            "\u001b[?25hRequirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.4->pomegranate) (4.4.2)\n",
            "\u001b[31mERROR: tensorflow 2.5.0 has requirement numpy~=1.19.2, but you'll have numpy 1.20.3 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.\u001b[0m\n",
            "Installing collected packages: numpy, pomegranate\n",
            "  Found existing installation: numpy 1.19.5\n",
            "    Uninstalling numpy-1.19.5:\n",
            "      Successfully uninstalled numpy-1.19.5\n",
            "Successfully installed numpy-1.20.3 pomegranate-0.14.5\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "numpy"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j348DLB40pvr"
      },
      "source": [
        "from pomegranate import State, HiddenMarkovModel, DiscreteDistribution\n",
        "\n",
        "model = HiddenMarkovModel(name=\"base-hmm-tagger\")"
      ],
      "execution_count": 65,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Plnzxdd13_tc",
        "outputId": "38d86d78-b979-44fd-f76b-20a579dbad3e"
      },
      "source": [
        "to_states = []\n",
        "for POS, words in ner_word_count.items():\n",
        "    total = float(sum(words.values()))\n",
        "    print('Total:',total)\n",
        "    print('**********')\n",
        "    distribution = {word: count/total for word, count in words.items()}\n",
        "    print('distribution:', distribution)\n",
        "    print('**********')\n",
        "    POS_emissions = DiscreteDistribution(distribution)\n",
        "    print('POS_emissions:', POS_emissions)\n",
        "    print('**********')\n",
        "    POS_state = State(POS_emissions, name=POS)\n",
        "    print('POS_state:', POS_state)\n",
        "    print('**********')\n",
        "    to_states.append(POS_state)"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total: 6.0\n",
            "**********\n",
            "distribution: {'Bailey': 0.16666666666666666, 'Starc': 0.5, 'Warner': 0.3333333333333333}\n",
            "**********\n",
            "POS_emissions: {\n",
            "    \"class\" : \"Distribution\",\n",
            "    \"dtype\" : \"str\",\n",
            "    \"name\" : \"DiscreteDistribution\",\n",
            "    \"parameters\" : [\n",
            "        {\n",
            "            \"Bailey\" : 0.16666666666666666,\n",
            "            \"Starc\" : 0.5,\n",
            "            \"Warner\" : 0.3333333333333333\n",
            "        }\n",
            "    ],\n",
            "    \"frozen\" : false\n",
            "}\n",
            "**********\n",
            "POS_state: {\n",
            "    \"class\" : \"State\",\n",
            "    \"distribution\" : {\n",
            "        \"class\" : \"Distribution\",\n",
            "        \"dtype\" : \"str\",\n",
            "        \"name\" : \"DiscreteDistribution\",\n",
            "        \"parameters\" : [\n",
            "            {\n",
            "                \"Bailey\" : 0.16666666666666666,\n",
            "                \"Starc\" : 0.5,\n",
            "                \"Warner\" : 0.3333333333333333\n",
            "            }\n",
            "        ],\n",
            "        \"frozen\" : false\n",
            "    },\n",
            "    \"name\" : \"PER\",\n",
            "    \"weight\" : 1.0\n",
            "}\n",
            "**********\n",
            "Total: 18.0\n",
            "**********\n",
            "distribution: {'named': 0.05555555555555555, 'captain': 0.05555555555555555, 'player': 0.05555555555555555, 'of': 0.05555555555555555, 'World': 0.05555555555555555, 'Cup': 0.05555555555555555, 'won': 0.05555555555555555, 'Cups': 0.05555555555555555, 'knocks': 0.05555555555555555, 'etched': 0.05555555555555555, 'in': 0.05555555555555555, 'memory': 0.05555555555555555, 'were': 0.05555555555555555, 'venues': 0.05555555555555555, 'go': 0.05555555555555555, 'as': 0.05555555555555555, 'great': 0.05555555555555555, 'combination': 0.05555555555555555}\n",
            "**********\n",
            "POS_emissions: {\n",
            "    \"class\" : \"Distribution\",\n",
            "    \"dtype\" : \"str\",\n",
            "    \"name\" : \"DiscreteDistribution\",\n",
            "    \"parameters\" : [\n",
            "        {\n",
            "            \"named\" : 0.05555555555555555,\n",
            "            \"captain\" : 0.05555555555555555,\n",
            "            \"player\" : 0.05555555555555555,\n",
            "            \"of\" : 0.05555555555555555,\n",
            "            \"World\" : 0.05555555555555555,\n",
            "            \"Cup\" : 0.05555555555555555,\n",
            "            \"won\" : 0.05555555555555555,\n",
            "            \"Cups\" : 0.05555555555555555,\n",
            "            \"knocks\" : 0.05555555555555555,\n",
            "            \"etched\" : 0.05555555555555555,\n",
            "            \"in\" : 0.05555555555555555,\n",
            "            \"memory\" : 0.05555555555555555,\n",
            "            \"were\" : 0.05555555555555555,\n",
            "            \"venues\" : 0.05555555555555555,\n",
            "            \"go\" : 0.05555555555555555,\n",
            "            \"as\" : 0.05555555555555555,\n",
            "            \"great\" : 0.05555555555555555,\n",
            "            \"combination\" : 0.05555555555555555\n",
            "        }\n",
            "    ],\n",
            "    \"frozen\" : false\n",
            "}\n",
            "**********\n",
            "POS_state: {\n",
            "    \"class\" : \"State\",\n",
            "    \"distribution\" : {\n",
            "        \"class\" : \"Distribution\",\n",
            "        \"dtype\" : \"str\",\n",
            "        \"name\" : \"DiscreteDistribution\",\n",
            "        \"parameters\" : [\n",
            "            {\n",
            "                \"named\" : 0.05555555555555555,\n",
            "                \"captain\" : 0.05555555555555555,\n",
            "                \"player\" : 0.05555555555555555,\n",
            "                \"of\" : 0.05555555555555555,\n",
            "                \"World\" : 0.05555555555555555,\n",
            "                \"Cup\" : 0.05555555555555555,\n",
            "                \"won\" : 0.05555555555555555,\n",
            "                \"Cups\" : 0.05555555555555555,\n",
            "                \"knocks\" : 0.05555555555555555,\n",
            "                \"etched\" : 0.05555555555555555,\n",
            "                \"in\" : 0.05555555555555555,\n",
            "                \"memory\" : 0.05555555555555555,\n",
            "                \"were\" : 0.05555555555555555,\n",
            "                \"venues\" : 0.05555555555555555,\n",
            "                \"go\" : 0.05555555555555555,\n",
            "                \"as\" : 0.05555555555555555,\n",
            "                \"great\" : 0.05555555555555555,\n",
            "                \"combination\" : 0.05555555555555555\n",
            "            }\n",
            "        ],\n",
            "        \"frozen\" : false\n",
            "    },\n",
            "    \"name\" : \"O\",\n",
            "    \"weight\" : 1.0\n",
            "}\n",
            "**********\n",
            "Total: 7.0\n",
            "**********\n",
            "distribution: {'Australia': 0.42857142857142855, 'Melbourne': 0.2857142857142857, 'SA': 0.14285714285714285, 'WI': 0.14285714285714285}\n",
            "**********\n",
            "POS_emissions: {\n",
            "    \"class\" : \"Distribution\",\n",
            "    \"dtype\" : \"str\",\n",
            "    \"name\" : \"DiscreteDistribution\",\n",
            "    \"parameters\" : [\n",
            "        {\n",
            "            \"Australia\" : 0.42857142857142855,\n",
            "            \"Melbourne\" : 0.2857142857142857,\n",
            "            \"SA\" : 0.14285714285714285,\n",
            "            \"WI\" : 0.14285714285714285\n",
            "        }\n",
            "    ],\n",
            "    \"frozen\" : false\n",
            "}\n",
            "**********\n",
            "POS_state: {\n",
            "    \"class\" : \"State\",\n",
            "    \"distribution\" : {\n",
            "        \"class\" : \"Distribution\",\n",
            "        \"dtype\" : \"str\",\n",
            "        \"name\" : \"DiscreteDistribution\",\n",
            "        \"parameters\" : [\n",
            "            {\n",
            "                \"Australia\" : 0.42857142857142855,\n",
            "                \"Melbourne\" : 0.2857142857142857,\n",
            "                \"SA\" : 0.14285714285714285,\n",
            "                \"WI\" : 0.14285714285714285\n",
            "            }\n",
            "        ],\n",
            "        \"frozen\" : false\n",
            "    },\n",
            "    \"name\" : \"GEO\",\n",
            "    \"weight\" : 1.0\n",
            "}\n",
            "**********\n",
            "Total: 7.0\n",
            "**********\n",
            "distribution: {'2015': 0.42857142857142855, '2003': 0.2857142857142857, '2007': 0.2857142857142857}\n",
            "**********\n",
            "POS_emissions: {\n",
            "    \"class\" : \"Distribution\",\n",
            "    \"dtype\" : \"str\",\n",
            "    \"name\" : \"DiscreteDistribution\",\n",
            "    \"parameters\" : [\n",
            "        {\n",
            "            \"2015\" : 0.42857142857142855,\n",
            "            \"2003\" : 0.2857142857142857,\n",
            "            \"2007\" : 0.2857142857142857\n",
            "        }\n",
            "    ],\n",
            "    \"frozen\" : false\n",
            "}\n",
            "**********\n",
            "POS_state: {\n",
            "    \"class\" : \"State\",\n",
            "    \"distribution\" : {\n",
            "        \"class\" : \"Distribution\",\n",
            "        \"dtype\" : \"str\",\n",
            "        \"name\" : \"DiscreteDistribution\",\n",
            "        \"parameters\" : [\n",
            "            {\n",
            "                \"2015\" : 0.42857142857142855,\n",
            "                \"2003\" : 0.2857142857142857,\n",
            "                \"2007\" : 0.2857142857142857\n",
            "            }\n",
            "        ],\n",
            "        \"frozen\" : false\n",
            "    },\n",
            "    \"name\" : \"TIM\",\n",
            "    \"weight\" : 1.0\n",
            "}\n",
            "**********\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cRlGVD3w5IwS",
        "outputId": "721bff09-6042-4013-e67a-0899996a9051"
      },
      "source": [
        "# Calculate start probability\n",
        "model.add_states()\n",
        "start_prob = {}\n",
        "for ner in ner_list:\n",
        "  start_prob[ner] = start_ner_count[ner]/unigram_count[ner]\n",
        "\n",
        "start_prob"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'GEO': 0.2857142857142857, 'O': 0.0, 'PER': 0.5, 'TIM': 0.14285714285714285}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sHDS8TXE7T7m",
        "outputId": "79260a49-ea67-46e3-fc3c-d990fbb66c98"
      },
      "source": [
        "for ner_state in to_states:\n",
        "  model.add_transition(model.start, ner_state, start_prob[ner_state.name])\n",
        "\n",
        "end_prob = {}\n",
        "for ner in ner_list:\n",
        "  end_prob[ner] = end_ner_count[ner]/unigram_count[ner]\n",
        "\n",
        "end_prob"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'GEO': 0.0, 'O': 0.3333333333333333, 'PER': 0.0, 'TIM': 0.0}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vo8L78pB87p_"
      },
      "source": [
        "# Add transition states to the HMM model\n",
        "for ner_state in to_states:\n",
        "  model.add_transition(ner_state, model.end, end_prob[ner_state.name])"
      ],
      "execution_count": 69,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xN80mXN-9Gqi",
        "outputId": "edec6e45-79b8-483c-c9c0-f513b97743d2"
      },
      "source": [
        "transition_prob_POS_word={}\n",
        "for key in bigram_count.keys():\n",
        "  transition_prob_POS_word[key] = bigram_count.get(key)/unigram_count[key[0]]\n",
        "\n",
        "transition_prob_POS_word"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{('GEO', 'O'): 0.5714285714285714,\n",
              " ('GEO', 'PER'): 0.14285714285714285,\n",
              " ('GEO', 'TIM'): 0.2857142857142857,\n",
              " ('O', 'GEO'): 0.05555555555555555,\n",
              " ('O', 'O'): 0.5,\n",
              " ('O', 'TIM'): 0.1111111111111111,\n",
              " ('PER', 'GEO'): 0.16666666666666666,\n",
              " ('PER', 'O'): 0.5,\n",
              " ('PER', 'PER'): 0.3333333333333333,\n",
              " ('TIM', 'GEO'): 0.42857142857142855,\n",
              " ('TIM', 'O'): 0.2857142857142857,\n",
              " ('TIM', 'TIM'): 0.2857142857142857}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w5gz7ZH3-Si4"
      },
      "source": [
        "transition_prob_POS_word[('O', 'PER')] = 0\n",
        "transition_prob_POS_word[('GEO', 'GEO')] = 0\n",
        "transition_prob_POS_word[('PER', 'TIM')] = 0\n",
        "transition_prob_POS_word[('TIM', 'PER')] = 0"
      ],
      "execution_count": 71,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-UVMJTm1A9pp",
        "outputId": "54b5fcd7-3248-47a2-ae67-db2f20e27d9c"
      },
      "source": [
        "transition_prob_POS_word"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{('GEO', 'GEO'): 0,\n",
              " ('GEO', 'O'): 0.5714285714285714,\n",
              " ('GEO', 'PER'): 0.14285714285714285,\n",
              " ('GEO', 'TIM'): 0.2857142857142857,\n",
              " ('O', 'GEO'): 0.05555555555555555,\n",
              " ('O', 'O'): 0.5,\n",
              " ('O', 'PER'): 0,\n",
              " ('O', 'TIM'): 0.1111111111111111,\n",
              " ('PER', 'GEO'): 0.16666666666666666,\n",
              " ('PER', 'O'): 0.5,\n",
              " ('PER', 'PER'): 0.3333333333333333,\n",
              " ('PER', 'TIM'): 0,\n",
              " ('TIM', 'GEO'): 0.42857142857142855,\n",
              " ('TIM', 'O'): 0.2857142857142857,\n",
              " ('TIM', 'PER'): 0,\n",
              " ('TIM', 'TIM'): 0.2857142857142857}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HmLEdrf9-GXV"
      },
      "source": [
        "for ner_state in to_states:\n",
        "  for next_ner_state in to_states:\n",
        "    model.add_transition(ner_state, next_ner_state, transition_prob_POS_word[(ner_state.name, next_ner_state.name)])\n",
        "\n",
        "model.bake()"
      ],
      "execution_count": 73,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uyaC7Inu_etY"
      },
      "source": [
        "def NER_decoding(sentence, model):\n",
        "    _, state_path = model.viterbi(sentence)\n",
        "    print(state_path)\n",
        "    return [state[1].name for state in state_path[1:-1]]"
      ],
      "execution_count": 74,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bXI4u03v_oqH",
        "outputId": "595323e3-b667-4a3a-dbbb-20aab14e88e3"
      },
      "source": [
        "test_sent = \"Starc named 2015 Australia player\"\n",
        "test_tuple = tuple(test_sent.split(\" \"))\n",
        "print(test_tuple)\n",
        "ner_tags = NER_decoding(('Starc', 'named', '2015', 'Australia', 'player'), model)\n",
        "\n",
        "print(\"Test Sentence:\", test_sent)\n",
        "print(\"Predicted Entities:\", ner_tags)"
      ],
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "('Starc', 'named', '2015', 'Australia', 'player')\n",
            "[(4, {\n",
            "    \"class\" : \"State\",\n",
            "    \"distribution\" : null,\n",
            "    \"name\" : \"base-hmm-tagger-start\",\n",
            "    \"weight\" : 1.0\n",
            "}), (2, {\n",
            "    \"class\" : \"State\",\n",
            "    \"distribution\" : {\n",
            "        \"class\" : \"Distribution\",\n",
            "        \"dtype\" : \"str\",\n",
            "        \"name\" : \"DiscreteDistribution\",\n",
            "        \"parameters\" : [\n",
            "            {\n",
            "                \"Bailey\" : 0.16666666666666666,\n",
            "                \"Starc\" : 0.5,\n",
            "                \"Warner\" : 0.3333333333333333\n",
            "            }\n",
            "        ],\n",
            "        \"frozen\" : false\n",
            "    },\n",
            "    \"name\" : \"PER\",\n",
            "    \"weight\" : 1.0\n",
            "}), (1, {\n",
            "    \"class\" : \"State\",\n",
            "    \"distribution\" : {\n",
            "        \"class\" : \"Distribution\",\n",
            "        \"dtype\" : \"str\",\n",
            "        \"name\" : \"DiscreteDistribution\",\n",
            "        \"parameters\" : [\n",
            "            {\n",
            "                \"named\" : 0.05555555555555555,\n",
            "                \"captain\" : 0.05555555555555555,\n",
            "                \"player\" : 0.05555555555555555,\n",
            "                \"of\" : 0.05555555555555555,\n",
            "                \"World\" : 0.05555555555555555,\n",
            "                \"Cup\" : 0.05555555555555555,\n",
            "                \"won\" : 0.05555555555555555,\n",
            "                \"Cups\" : 0.05555555555555555,\n",
            "                \"knocks\" : 0.05555555555555555,\n",
            "                \"etched\" : 0.05555555555555555,\n",
            "                \"in\" : 0.05555555555555555,\n",
            "                \"memory\" : 0.05555555555555555,\n",
            "                \"were\" : 0.05555555555555555,\n",
            "                \"venues\" : 0.05555555555555555,\n",
            "                \"go\" : 0.05555555555555555,\n",
            "                \"as\" : 0.05555555555555555,\n",
            "                \"great\" : 0.05555555555555555,\n",
            "                \"combination\" : 0.05555555555555555\n",
            "            }\n",
            "        ],\n",
            "        \"frozen\" : false\n",
            "    },\n",
            "    \"name\" : \"O\",\n",
            "    \"weight\" : 1.0\n",
            "}), (3, {\n",
            "    \"class\" : \"State\",\n",
            "    \"distribution\" : {\n",
            "        \"class\" : \"Distribution\",\n",
            "        \"dtype\" : \"str\",\n",
            "        \"name\" : \"DiscreteDistribution\",\n",
            "        \"parameters\" : [\n",
            "            {\n",
            "                \"2015\" : 0.42857142857142855,\n",
            "                \"2003\" : 0.2857142857142857,\n",
            "                \"2007\" : 0.2857142857142857\n",
            "            }\n",
            "        ],\n",
            "        \"frozen\" : false\n",
            "    },\n",
            "    \"name\" : \"TIM\",\n",
            "    \"weight\" : 1.0\n",
            "}), (0, {\n",
            "    \"class\" : \"State\",\n",
            "    \"distribution\" : {\n",
            "        \"class\" : \"Distribution\",\n",
            "        \"dtype\" : \"str\",\n",
            "        \"name\" : \"DiscreteDistribution\",\n",
            "        \"parameters\" : [\n",
            "            {\n",
            "                \"Australia\" : 0.42857142857142855,\n",
            "                \"Melbourne\" : 0.2857142857142857,\n",
            "                \"SA\" : 0.14285714285714285,\n",
            "                \"WI\" : 0.14285714285714285\n",
            "            }\n",
            "        ],\n",
            "        \"frozen\" : false\n",
            "    },\n",
            "    \"name\" : \"GEO\",\n",
            "    \"weight\" : 1.0\n",
            "}), (1, {\n",
            "    \"class\" : \"State\",\n",
            "    \"distribution\" : {\n",
            "        \"class\" : \"Distribution\",\n",
            "        \"dtype\" : \"str\",\n",
            "        \"name\" : \"DiscreteDistribution\",\n",
            "        \"parameters\" : [\n",
            "            {\n",
            "                \"named\" : 0.05555555555555555,\n",
            "                \"captain\" : 0.05555555555555555,\n",
            "                \"player\" : 0.05555555555555555,\n",
            "                \"of\" : 0.05555555555555555,\n",
            "                \"World\" : 0.05555555555555555,\n",
            "                \"Cup\" : 0.05555555555555555,\n",
            "                \"won\" : 0.05555555555555555,\n",
            "                \"Cups\" : 0.05555555555555555,\n",
            "                \"knocks\" : 0.05555555555555555,\n",
            "                \"etched\" : 0.05555555555555555,\n",
            "                \"in\" : 0.05555555555555555,\n",
            "                \"memory\" : 0.05555555555555555,\n",
            "                \"were\" : 0.05555555555555555,\n",
            "                \"venues\" : 0.05555555555555555,\n",
            "                \"go\" : 0.05555555555555555,\n",
            "                \"as\" : 0.05555555555555555,\n",
            "                \"great\" : 0.05555555555555555,\n",
            "                \"combination\" : 0.05555555555555555\n",
            "            }\n",
            "        ],\n",
            "        \"frozen\" : false\n",
            "    },\n",
            "    \"name\" : \"O\",\n",
            "    \"weight\" : 1.0\n",
            "}), (5, {\n",
            "    \"class\" : \"State\",\n",
            "    \"distribution\" : null,\n",
            "    \"name\" : \"base-hmm-tagger-end\",\n",
            "    \"weight\" : 1.0\n",
            "})]\n",
            "Test Sentence: Starc named 2015 Australia player\n",
            "Predicted Entities: ['PER', 'O', 'TIM', 'GEO', 'O']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ikhX2eUDxKT"
      },
      "source": [
        "### **Problem Statement**\n",
        "You are provided with a corpus that has a lot of sentences with POS \n",
        "tagging completed (CRF_POS_dataset.csv). The objective is to use \n",
        "this dataset and build a Condition Random Field (CRF model) with \n",
        "sequence modeling.\n",
        "\n",
        "#### **Question 2.1**\n",
        " \n",
        "- Perform appropriate data cleaning & preprocessing steps\n",
        "- Then, tokenize the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "I8fdkWaUD9ag",
        "outputId": "4c0b9f88-21c2-4cd8-f2a8-e9d96d7e5bee"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.read_csv(\"/content/drive/MyDrive/CRF_POS_dataset.csv\", encoding = \"ISO-8859-1\")\n",
        "df.head()"
      ],
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Num</th>\n",
              "      <th>Word</th>\n",
              "      <th>Tag_POS</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>A</td>\n",
              "      <td>DT</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "      <td>37-year-old</td>\n",
              "      <td>JJ</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>woman</td>\n",
              "      <td>NN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NaN</td>\n",
              "      <td>has</td>\n",
              "      <td>VBZ</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NaN</td>\n",
              "      <td>become</td>\n",
              "      <td>VBN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Num         Word Tag_POS\n",
              "0  1.0            A      DT\n",
              "1  NaN  37-year-old      JJ\n",
              "2  NaN        woman      NN\n",
              "3  NaN          has     VBZ\n",
              "4  NaN       become     VBN"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j3sSPzGOE4h9",
        "outputId": "0c0330be-413a-4a16-cb1b-ea1db9ddbbca"
      },
      "source": [
        "df.isnull().sum()"
      ],
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Num        541078\n",
              "Word            0\n",
              "Tag_POS         0\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "id": "SPa3pZd8E_99",
        "outputId": "003306c2-aeb1-4eaf-c0e5-a04c4170a306"
      },
      "source": [
        "df = df.fillna(method='ffill')\n",
        "df.head(10)"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Num</th>\n",
              "      <th>Word</th>\n",
              "      <th>Tag_POS</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>A</td>\n",
              "      <td>DT</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>37-year-old</td>\n",
              "      <td>JJ</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.0</td>\n",
              "      <td>woman</td>\n",
              "      <td>NN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.0</td>\n",
              "      <td>has</td>\n",
              "      <td>VBZ</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.0</td>\n",
              "      <td>become</td>\n",
              "      <td>VBN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.0</td>\n",
              "      <td>the</td>\n",
              "      <td>DT</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1.0</td>\n",
              "      <td>13th</td>\n",
              "      <td>JJ</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.0</td>\n",
              "      <td>person</td>\n",
              "      <td>NN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.0</td>\n",
              "      <td>in</td>\n",
              "      <td>IN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.0</td>\n",
              "      <td>Egypt</td>\n",
              "      <td>NNP</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Num         Word Tag_POS\n",
              "0  1.0            A      DT\n",
              "1  1.0  37-year-old      JJ\n",
              "2  1.0        woman      NN\n",
              "3  1.0          has     VBZ\n",
              "4  1.0       become     VBN\n",
              "5  1.0          the      DT\n",
              "6  1.0         13th      JJ\n",
              "7  1.0       person      NN\n",
              "8  1.0           in      IN\n",
              "9  1.0        Egypt     NNP"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EehLY7qNFL-z",
        "outputId": "69b2ed39-d62e-4fbf-cd9c-e71245137a5e"
      },
      "source": [
        "df['Tag_POS'].value_counts()"
      ],
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NN      78624\n",
              "NNP     71316\n",
              "IN      65465\n",
              "DT      52987\n",
              "JJ      42521\n",
              "NNS     40985\n",
              ".       25852\n",
              "VBD     21334\n",
              ",       17704\n",
              "VBN     17492\n",
              "CD      13538\n",
              "VBZ     13512\n",
              "VB      12939\n",
              "CC      12737\n",
              "TO      12393\n",
              "RB      10889\n",
              "VBG     10313\n",
              "VBP      8683\n",
              "PRP      7236\n",
              "POS      6094\n",
              "PRP$     4741\n",
              "MD       3759\n",
              "``       2123\n",
              "WDT      2018\n",
              "JJS      1683\n",
              "JJR      1640\n",
              "WP       1384\n",
              "NNPS     1362\n",
              "RP       1324\n",
              "WRB      1182\n",
              "$         625\n",
              "RBR       577\n",
              ":         426\n",
              "RRB       394\n",
              "LRB       393\n",
              "EX        343\n",
              "RBS       160\n",
              ";         104\n",
              "PDT        84\n",
              "WP$        57\n",
              "UH         13\n",
              "FW          1\n",
              "Name: Tag_POS, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "swRnqg1cGz62",
        "outputId": "606accbc-580f-491e-e3d4-d0dc9269bad1"
      },
      "source": [
        "df['Tag_POS_pr'] = df['Tag_POS'].mask(df['Tag_POS'] == \"PRP$\", \"PRP\")\n",
        "df['Tag_POS_pr'] = df[\"Tag_POS_pr\"].mask(df['Tag_POS_pr'] == \"WP$\", \"WP\")\n",
        "df['check'] = df['Tag_POS_pr'].apply(lambda x: 1 if not x.isalnum() else 0)\n",
        "df = df[df['check'] == 0]\n",
        "df['Tag_POS_pr'].value_counts()"
      ],
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NN      78624\n",
              "NNP     71316\n",
              "IN      65465\n",
              "DT      52987\n",
              "JJ      42521\n",
              "NNS     40985\n",
              "VBD     21334\n",
              "VBN     17492\n",
              "CD      13538\n",
              "VBZ     13512\n",
              "VB      12939\n",
              "CC      12737\n",
              "TO      12393\n",
              "PRP     11977\n",
              "RB      10889\n",
              "VBG     10313\n",
              "VBP      8683\n",
              "POS      6094\n",
              "MD       3759\n",
              "WDT      2018\n",
              "JJS      1683\n",
              "JJR      1640\n",
              "WP       1441\n",
              "NNPS     1362\n",
              "RP       1324\n",
              "WRB      1182\n",
              "RBR       577\n",
              "RRB       394\n",
              "LRB       393\n",
              "EX        343\n",
              "RBS       160\n",
              "PDT        84\n",
              "UH         13\n",
              "FW          1\n",
              "Name: Tag_POS_pr, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 96
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "id": "VeOvwdemJCbc",
        "outputId": "f795b17f-dba2-4a45-a884-802640f9c8f5"
      },
      "source": [
        "df = df[['Num', 'Word', 'Tag_POS_pr']]\n",
        "df.head(10)"
      ],
      "execution_count": 97,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Num</th>\n",
              "      <th>Word</th>\n",
              "      <th>Tag_POS_pr</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>A</td>\n",
              "      <td>DT</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>37-year-old</td>\n",
              "      <td>JJ</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.0</td>\n",
              "      <td>woman</td>\n",
              "      <td>NN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.0</td>\n",
              "      <td>has</td>\n",
              "      <td>VBZ</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.0</td>\n",
              "      <td>become</td>\n",
              "      <td>VBN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.0</td>\n",
              "      <td>the</td>\n",
              "      <td>DT</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1.0</td>\n",
              "      <td>13th</td>\n",
              "      <td>JJ</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.0</td>\n",
              "      <td>person</td>\n",
              "      <td>NN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.0</td>\n",
              "      <td>in</td>\n",
              "      <td>IN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.0</td>\n",
              "      <td>Egypt</td>\n",
              "      <td>NNP</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Num         Word Tag_POS_pr\n",
              "0  1.0            A         DT\n",
              "1  1.0  37-year-old         JJ\n",
              "2  1.0        woman         NN\n",
              "3  1.0          has        VBZ\n",
              "4  1.0       become        VBN\n",
              "5  1.0          the         DT\n",
              "6  1.0         13th         JJ\n",
              "7  1.0       person         NN\n",
              "8  1.0           in         IN\n",
              "9  1.0        Egypt        NNP"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T4cw1M-qFHPu"
      },
      "source": [
        "sent_groups = df.groupby('Num')"
      ],
      "execution_count": 98,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3EcG0Tf_KMAK"
      },
      "source": [
        "tokenized_sentences = []\n",
        "for sent_no, group in sent_groups:\n",
        "  sentence_data = []\n",
        "  words = group['Word'].tolist()\n",
        "  tags = group['Tag_POS_pr'].tolist()\n",
        "  for word, tag in zip(words, tags):\n",
        "    sentence_data.append((word, tag))\n",
        "  tokenized_sentences.append(sentence_data)"
      ],
      "execution_count": 99,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MN-vnuzdKsuk",
        "outputId": "c82d476b-8f94-4db0-967b-6e3252e63c94"
      },
      "source": [
        "tokenized_sentences[0]"
      ],
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('A', 'DT'),\n",
              " ('37-year-old', 'JJ'),\n",
              " ('woman', 'NN'),\n",
              " ('has', 'VBZ'),\n",
              " ('become', 'VBN'),\n",
              " ('the', 'DT'),\n",
              " ('13th', 'JJ'),\n",
              " ('person', 'NN'),\n",
              " ('in', 'IN'),\n",
              " ('Egypt', 'NNP'),\n",
              " ('to', 'TO'),\n",
              " ('die', 'VB'),\n",
              " ('of', 'IN'),\n",
              " ('the', 'DT'),\n",
              " ('H5N1', 'NNP'),\n",
              " ('strain', 'NN'),\n",
              " ('of', 'IN'),\n",
              " ('bird', 'NN'),\n",
              " ('flu', 'NN')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 100
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bBjg8WkmNr6s"
      },
      "source": [
        "#### **Question 2.2** \n",
        "- Extract features from the text. Add features like whether the word is in lower case, is it a title or is it \n",
        "a digit and what is its POS tag, whether it is at the beginning of the sentence or at the end of the \n",
        "sentence. These features should be part of the X variable. And Y should be the target variable, i.e., \n",
        "POS of the particular word.\n",
        "- Use sklearn_crfsuite and build a CRF model. \n",
        "- Also, run predictions on the same dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3XuSHykzLO_x"
      },
      "source": [
        "# Code Reference from https://sklearn-crfsuite.readthedocs.io/en/latest/tutorial.html\n",
        "\n",
        "def word2features(sent, i):\n",
        "    word = sent[i][0]\n",
        "    postag = sent[i][1]\n",
        "\n",
        "    features = {\n",
        "        'bias': 1.0,\n",
        "        'word.lower()': word.lower(),\n",
        "        'word[-3:]': word[-3:],\n",
        "        'word[-2:]': word[-2:],\n",
        "        'word.isupper()': word.isupper(),\n",
        "        'word.istitle()': word.istitle(),\n",
        "        'word.isdigit()': word.isdigit(),\n",
        "        'postag': postag,\n",
        "        'postag[:2]': postag[:2],\n",
        "    }\n",
        "    if i > 0:\n",
        "        word1 = sent[i-1][0]\n",
        "        postag1 = sent[i-1][1]\n",
        "        features.update({\n",
        "            '-1:word.lower()': word1.lower(),\n",
        "            '-1:word.istitle()': word1.istitle(),\n",
        "            '-1:word.isupper()': word1.isupper(),\n",
        "            '-1:postag': postag1,\n",
        "            '-1:postag[:2]': postag1[:2],\n",
        "        })\n",
        "    else:\n",
        "        features['BOS'] = True\n",
        "\n",
        "    if i < len(sent)-1:\n",
        "        word1 = sent[i+1][0]\n",
        "        postag1 = sent[i+1][1]\n",
        "        features.update({\n",
        "            '+1:word.lower()': word1.lower(),\n",
        "            '+1:word.istitle()': word1.istitle(),\n",
        "            '+1:word.isupper()': word1.isupper(),\n",
        "            '+1:postag': postag1,\n",
        "            '+1:postag[:2]': postag1[:2],\n",
        "        })\n",
        "    else:\n",
        "        features['EOS'] = True\n",
        "\n",
        "    return features\n",
        "\n",
        "\n",
        "def sent2features(sent):\n",
        "    return [word2features(sent, i) for i in range(len(sent))]\n",
        "\n",
        "def sent2postags(sent):\n",
        "    return [postag for token, postag in sent]\n",
        "\n",
        "def sent2tokens(sent):\n",
        "    return [token for token, postag in sent]"
      ],
      "execution_count": 106,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hzjWUiU_LUXQ",
        "outputId": "1f7ed2c5-6a47-4c61-de6a-f68f0c0ea940"
      },
      "source": [
        "import tqdm\n",
        "\n",
        "X = [sent2features(s) for s in tqdm.tqdm(tokenized_sentences, total=len(tokenized_sentences))]\n",
        "y = [sent2postags(s) for s in tqdm.tqdm(tokenized_sentences, total=len(tokenized_sentences))]"
      ],
      "execution_count": 107,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "  0%|          | 0/25928 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "\n",
            "  5%|▍         | 1277/25928 [00:00<00:01, 12767.09it/s]\u001b[A\u001b[A\n",
            "\n",
            "  9%|▉         | 2441/25928 [00:00<00:01, 12406.53it/s]\u001b[A\u001b[A\n",
            "\n",
            " 14%|█▍        | 3594/25928 [00:00<00:01, 12129.22it/s]\u001b[A\u001b[A\n",
            "\n",
            " 19%|█▉        | 4881/25928 [00:00<00:01, 12340.45it/s]\u001b[A\u001b[A\n",
            "\n",
            " 24%|██▍       | 6164/25928 [00:00<00:01, 12480.49it/s]\u001b[A\u001b[A\n",
            "\n",
            " 28%|██▊       | 7359/25928 [00:00<00:01, 12315.13it/s]\u001b[A\u001b[A\n",
            "\n",
            " 33%|███▎      | 8443/25928 [00:00<00:01, 11423.28it/s]\u001b[A\u001b[A\n",
            "\n",
            " 37%|███▋      | 9590/25928 [00:00<00:01, 11432.81it/s]\u001b[A\u001b[A\n",
            "\n",
            " 41%|████▏     | 10738/25928 [00:00<00:01, 11445.69it/s]\u001b[A\u001b[A\n",
            "\n",
            " 47%|████▋     | 12063/25928 [00:01<00:01, 11931.33it/s]\u001b[A\u001b[A\n",
            "\n",
            " 51%|█████     | 13266/25928 [00:01<00:01, 11957.61it/s]\u001b[A\u001b[A\n",
            "\n",
            " 56%|█████▌    | 14490/25928 [00:01<00:00, 12039.44it/s]\u001b[A\u001b[A\n",
            "\n",
            " 60%|██████    | 15682/25928 [00:01<00:00, 11966.69it/s]\u001b[A\u001b[A\n",
            "\n",
            " 65%|██████▌   | 16871/25928 [00:01<00:00, 11882.30it/s]\u001b[A\u001b[A\n",
            "\n",
            " 70%|██████▉   | 18054/25928 [00:01<00:00, 11412.82it/s]\u001b[A\u001b[A\n",
            "\n",
            " 74%|███████▍  | 19289/25928 [00:01<00:00, 11677.50it/s]\u001b[A\u001b[A\n",
            "\n",
            " 79%|███████▉  | 20459/25928 [00:01<00:00, 11669.88it/s]\u001b[A\u001b[A\n",
            "\n",
            " 83%|████████▎ | 21636/25928 [00:01<00:00, 11699.27it/s]\u001b[A\u001b[A\n",
            "\n",
            " 88%|████████▊ | 22932/25928 [00:01<00:00, 12050.30it/s]\u001b[A\u001b[A\n",
            "\n",
            " 93%|█████████▎| 24194/25928 [00:02<00:00, 12214.77it/s]\u001b[A\u001b[A\n",
            "\n",
            "100%|██████████| 25928/25928 [00:02<00:00, 11893.15it/s]\n",
            "\n",
            "\n",
            "  0%|          | 0/25928 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "\n",
            "100%|██████████| 25928/25928 [00:00<00:00, 111405.49it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JcbFcBVFL8CW",
        "outputId": "77f7c737-8792-4f8f-fc53-dbd364afb4e4"
      },
      "source": [
        "X[0], y[0]"
      ],
      "execution_count": 109,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "([{'+1:postag': 'JJ',\n",
              "   '+1:postag[:2]': 'JJ',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': '37-year-old',\n",
              "   'BOS': True,\n",
              "   'bias': 1.0,\n",
              "   'postag': 'DT',\n",
              "   'postag[:2]': 'DT',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': True,\n",
              "   'word.isupper()': True,\n",
              "   'word.lower()': 'a',\n",
              "   'word[-2:]': 'A',\n",
              "   'word[-3:]': 'A'},\n",
              "  {'+1:postag': 'NN',\n",
              "   '+1:postag[:2]': 'NN',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'woman',\n",
              "   '-1:postag': 'DT',\n",
              "   '-1:postag[:2]': 'DT',\n",
              "   '-1:word.istitle()': True,\n",
              "   '-1:word.isupper()': True,\n",
              "   '-1:word.lower()': 'a',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'JJ',\n",
              "   'postag[:2]': 'JJ',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': '37-year-old',\n",
              "   'word[-2:]': 'ld',\n",
              "   'word[-3:]': 'old'},\n",
              "  {'+1:postag': 'VBZ',\n",
              "   '+1:postag[:2]': 'VB',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'has',\n",
              "   '-1:postag': 'JJ',\n",
              "   '-1:postag[:2]': 'JJ',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': '37-year-old',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'NN',\n",
              "   'postag[:2]': 'NN',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'woman',\n",
              "   'word[-2:]': 'an',\n",
              "   'word[-3:]': 'man'},\n",
              "  {'+1:postag': 'VBN',\n",
              "   '+1:postag[:2]': 'VB',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'become',\n",
              "   '-1:postag': 'NN',\n",
              "   '-1:postag[:2]': 'NN',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'woman',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'VBZ',\n",
              "   'postag[:2]': 'VB',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'has',\n",
              "   'word[-2:]': 'as',\n",
              "   'word[-3:]': 'has'},\n",
              "  {'+1:postag': 'DT',\n",
              "   '+1:postag[:2]': 'DT',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'the',\n",
              "   '-1:postag': 'VBZ',\n",
              "   '-1:postag[:2]': 'VB',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'has',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'VBN',\n",
              "   'postag[:2]': 'VB',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'become',\n",
              "   'word[-2:]': 'me',\n",
              "   'word[-3:]': 'ome'},\n",
              "  {'+1:postag': 'JJ',\n",
              "   '+1:postag[:2]': 'JJ',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': '13th',\n",
              "   '-1:postag': 'VBN',\n",
              "   '-1:postag[:2]': 'VB',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'become',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'DT',\n",
              "   'postag[:2]': 'DT',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'the',\n",
              "   'word[-2:]': 'he',\n",
              "   'word[-3:]': 'the'},\n",
              "  {'+1:postag': 'NN',\n",
              "   '+1:postag[:2]': 'NN',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'person',\n",
              "   '-1:postag': 'DT',\n",
              "   '-1:postag[:2]': 'DT',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'the',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'JJ',\n",
              "   'postag[:2]': 'JJ',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': '13th',\n",
              "   'word[-2:]': 'th',\n",
              "   'word[-3:]': '3th'},\n",
              "  {'+1:postag': 'IN',\n",
              "   '+1:postag[:2]': 'IN',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'in',\n",
              "   '-1:postag': 'JJ',\n",
              "   '-1:postag[:2]': 'JJ',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': '13th',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'NN',\n",
              "   'postag[:2]': 'NN',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'person',\n",
              "   'word[-2:]': 'on',\n",
              "   'word[-3:]': 'son'},\n",
              "  {'+1:postag': 'NNP',\n",
              "   '+1:postag[:2]': 'NN',\n",
              "   '+1:word.istitle()': True,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'egypt',\n",
              "   '-1:postag': 'NN',\n",
              "   '-1:postag[:2]': 'NN',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'person',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'IN',\n",
              "   'postag[:2]': 'IN',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'in',\n",
              "   'word[-2:]': 'in',\n",
              "   'word[-3:]': 'in'},\n",
              "  {'+1:postag': 'TO',\n",
              "   '+1:postag[:2]': 'TO',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'to',\n",
              "   '-1:postag': 'IN',\n",
              "   '-1:postag[:2]': 'IN',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'in',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'NNP',\n",
              "   'postag[:2]': 'NN',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': True,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'egypt',\n",
              "   'word[-2:]': 'pt',\n",
              "   'word[-3:]': 'ypt'},\n",
              "  {'+1:postag': 'VB',\n",
              "   '+1:postag[:2]': 'VB',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'die',\n",
              "   '-1:postag': 'NNP',\n",
              "   '-1:postag[:2]': 'NN',\n",
              "   '-1:word.istitle()': True,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'egypt',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'TO',\n",
              "   'postag[:2]': 'TO',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'to',\n",
              "   'word[-2:]': 'to',\n",
              "   'word[-3:]': 'to'},\n",
              "  {'+1:postag': 'IN',\n",
              "   '+1:postag[:2]': 'IN',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'of',\n",
              "   '-1:postag': 'TO',\n",
              "   '-1:postag[:2]': 'TO',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'to',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'VB',\n",
              "   'postag[:2]': 'VB',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'die',\n",
              "   'word[-2:]': 'ie',\n",
              "   'word[-3:]': 'die'},\n",
              "  {'+1:postag': 'DT',\n",
              "   '+1:postag[:2]': 'DT',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'the',\n",
              "   '-1:postag': 'VB',\n",
              "   '-1:postag[:2]': 'VB',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'die',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'IN',\n",
              "   'postag[:2]': 'IN',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'of',\n",
              "   'word[-2:]': 'of',\n",
              "   'word[-3:]': 'of'},\n",
              "  {'+1:postag': 'NNP',\n",
              "   '+1:postag[:2]': 'NN',\n",
              "   '+1:word.istitle()': True,\n",
              "   '+1:word.isupper()': True,\n",
              "   '+1:word.lower()': 'h5n1',\n",
              "   '-1:postag': 'IN',\n",
              "   '-1:postag[:2]': 'IN',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'of',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'DT',\n",
              "   'postag[:2]': 'DT',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'the',\n",
              "   'word[-2:]': 'he',\n",
              "   'word[-3:]': 'the'},\n",
              "  {'+1:postag': 'NN',\n",
              "   '+1:postag[:2]': 'NN',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'strain',\n",
              "   '-1:postag': 'DT',\n",
              "   '-1:postag[:2]': 'DT',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'the',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'NNP',\n",
              "   'postag[:2]': 'NN',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': True,\n",
              "   'word.isupper()': True,\n",
              "   'word.lower()': 'h5n1',\n",
              "   'word[-2:]': 'N1',\n",
              "   'word[-3:]': '5N1'},\n",
              "  {'+1:postag': 'IN',\n",
              "   '+1:postag[:2]': 'IN',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'of',\n",
              "   '-1:postag': 'NNP',\n",
              "   '-1:postag[:2]': 'NN',\n",
              "   '-1:word.istitle()': True,\n",
              "   '-1:word.isupper()': True,\n",
              "   '-1:word.lower()': 'h5n1',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'NN',\n",
              "   'postag[:2]': 'NN',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'strain',\n",
              "   'word[-2:]': 'in',\n",
              "   'word[-3:]': 'ain'},\n",
              "  {'+1:postag': 'NN',\n",
              "   '+1:postag[:2]': 'NN',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'bird',\n",
              "   '-1:postag': 'NN',\n",
              "   '-1:postag[:2]': 'NN',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'strain',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'IN',\n",
              "   'postag[:2]': 'IN',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'of',\n",
              "   'word[-2:]': 'of',\n",
              "   'word[-3:]': 'of'},\n",
              "  {'+1:postag': 'NN',\n",
              "   '+1:postag[:2]': 'NN',\n",
              "   '+1:word.istitle()': False,\n",
              "   '+1:word.isupper()': False,\n",
              "   '+1:word.lower()': 'flu',\n",
              "   '-1:postag': 'IN',\n",
              "   '-1:postag[:2]': 'IN',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'of',\n",
              "   'bias': 1.0,\n",
              "   'postag': 'NN',\n",
              "   'postag[:2]': 'NN',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'bird',\n",
              "   'word[-2:]': 'rd',\n",
              "   'word[-3:]': 'ird'},\n",
              "  {'-1:postag': 'NN',\n",
              "   '-1:postag[:2]': 'NN',\n",
              "   '-1:word.istitle()': False,\n",
              "   '-1:word.isupper()': False,\n",
              "   '-1:word.lower()': 'bird',\n",
              "   'EOS': True,\n",
              "   'bias': 1.0,\n",
              "   'postag': 'NN',\n",
              "   'postag[:2]': 'NN',\n",
              "   'word.isdigit()': False,\n",
              "   'word.istitle()': False,\n",
              "   'word.isupper()': False,\n",
              "   'word.lower()': 'flu',\n",
              "   'word[-2:]': 'lu',\n",
              "   'word[-3:]': 'flu'}],\n",
              " ['DT',\n",
              "  'JJ',\n",
              "  'NN',\n",
              "  'VBZ',\n",
              "  'VBN',\n",
              "  'DT',\n",
              "  'JJ',\n",
              "  'NN',\n",
              "  'IN',\n",
              "  'NNP',\n",
              "  'TO',\n",
              "  'VB',\n",
              "  'IN',\n",
              "  'DT',\n",
              "  'NNP',\n",
              "  'NN',\n",
              "  'IN',\n",
              "  'NN',\n",
              "  'NN'])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YMv0f4roMIo8"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=0)"
      ],
      "execution_count": 110,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "21P5678nMPj5",
        "outputId": "2753873a-718a-4aaa-9aa4-863ed930255f"
      },
      "source": [
        "!pip install sklearn_crfsuite"
      ],
      "execution_count": 111,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting sklearn_crfsuite\n",
            "  Downloading https://files.pythonhosted.org/packages/25/74/5b7befa513482e6dee1f3dd68171a6c9dfc14c0eaa00f885ffeba54fe9b0/sklearn_crfsuite-0.3.6-py2.py3-none-any.whl\n",
            "Requirement already satisfied: tqdm>=2.0 in /usr/local/lib/python3.7/dist-packages (from sklearn_crfsuite) (4.41.1)\n",
            "Collecting python-crfsuite>=0.8.3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/79/47/58f16c46506139f17de4630dbcfb877ce41a6355a1bbf3c443edb9708429/python_crfsuite-0.9.7-cp37-cp37m-manylinux1_x86_64.whl (743kB)\n",
            "\u001b[K     |████████████████████████████████| 747kB 5.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sklearn_crfsuite) (1.15.0)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from sklearn_crfsuite) (0.8.9)\n",
            "Installing collected packages: python-crfsuite, sklearn-crfsuite\n",
            "Successfully installed python-crfsuite-0.9.7 sklearn-crfsuite-0.3.6\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QgFw5K_kMSf8",
        "outputId": "6d91e357-69ec-422b-b3cb-2576f26e7aa7"
      },
      "source": [
        "import sklearn_crfsuite\n",
        "\n",
        "crf = sklearn_crfsuite.CRF(\n",
        "    algorithm='lbfgs',\n",
        "    c1=0.1,\n",
        "    c2=0.1,\n",
        "    max_iterations=100,\n",
        "    all_possible_transitions=True\n",
        ")\n",
        "crf.fit(X_train, y_train)"
      ],
      "execution_count": 113,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/base.py:197: FutureWarning: From version 0.24, get_params will raise an AttributeError if a parameter cannot be retrieved as an instance attribute. Previously it would return None.\n",
            "  FutureWarning)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "CRF(algorithm='lbfgs', all_possible_states=None, all_possible_transitions=True,\n",
              "    averaging=None, c=None, c1=0.1, c2=0.1, calibration_candidates=None,\n",
              "    calibration_eta=None, calibration_max_trials=None, calibration_rate=None,\n",
              "    calibration_samples=None, delta=None, epsilon=None, error_sensitive=None,\n",
              "    gamma=None, keep_tempfiles=None, linesearch=None, max_iterations=100,\n",
              "    max_linesearch=None, min_freq=None, model_filename=None, num_memories=None,\n",
              "    pa_type=None, period=None, trainer_cls=None, variance=None, verbose=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 113
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VE5PmD5mN9GD"
      },
      "source": [
        "#### **Question 2.3**\n",
        "- Build a flat classification report for each POS that is present in the corpus and calculate the accuracy, \n",
        "F1 score"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fnieBCp7NAeO",
        "outputId": "849c3a6c-d105-47c4-8cd3-a68414de5f11"
      },
      "source": [
        "from sklearn_crfsuite import metrics\n",
        "\n",
        "y_pred = crf.predict(X_test)\n",
        "\n",
        "# Flat Classification Report\n",
        "print(metrics.flat_classification_report(y_test, y_pred))"
      ],
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "          CC       1.00      1.00      1.00      4110\n",
            "          CD       1.00      1.00      1.00      4433\n",
            "          DT       1.00      1.00      1.00     17528\n",
            "          EX       1.00      1.00      1.00       119\n",
            "          FW       0.00      0.00      0.00         1\n",
            "          IN       1.00      1.00      1.00     21541\n",
            "          JJ       1.00      1.00      1.00     14041\n",
            "         JJR       1.00      1.00      1.00       532\n",
            "         JJS       1.00      1.00      1.00       538\n",
            "         LRB       1.00      1.00      1.00       132\n",
            "          MD       1.00      1.00      1.00      1268\n",
            "          NN       1.00      1.00      1.00     25991\n",
            "         NNP       1.00      1.00      1.00     23491\n",
            "        NNPS       1.00      1.00      1.00       428\n",
            "         NNS       1.00      1.00      1.00     13373\n",
            "         PDT       1.00      1.00      1.00        34\n",
            "         POS       1.00      1.00      1.00      2021\n",
            "         PRP       1.00      1.00      1.00      3916\n",
            "          RB       1.00      1.00      1.00      3486\n",
            "         RBR       1.00      1.00      1.00       217\n",
            "         RBS       1.00      1.00      1.00        39\n",
            "          RP       1.00      1.00      1.00       399\n",
            "         RRB       1.00      1.00      1.00       132\n",
            "          TO       1.00      1.00      1.00      4041\n",
            "          UH       1.00      1.00      1.00         5\n",
            "          VB       1.00      1.00      1.00      4233\n",
            "         VBD       1.00      1.00      1.00      7009\n",
            "         VBG       1.00      1.00      1.00      3423\n",
            "         VBN       1.00      1.00      1.00      5762\n",
            "         VBP       1.00      1.00      1.00      2793\n",
            "         VBZ       1.00      1.00      1.00      4459\n",
            "         WDT       1.00      1.00      1.00       644\n",
            "          WP       1.00      1.00      1.00       494\n",
            "         WRB       1.00      1.00      1.00       373\n",
            "\n",
            "    accuracy                           1.00    171006\n",
            "   macro avg       0.97      0.97      0.97    171006\n",
            "weighted avg       1.00      1.00      1.00    171006\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zUm-D26iOEpH",
        "outputId": "b7d7e17f-533b-4ad1-ff82-f593ab670221"
      },
      "source": [
        "# F1 Score\n",
        "metrics.flat_f1_score(y_test, y_pred, average='weighted')"
      ],
      "execution_count": 119,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9999912287076994"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 119
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eCbQrTqjOdQc",
        "outputId": "8c8efce8-2108-4a45-c891-97f8c37e6e48"
      },
      "source": [
        "# Accuracy Score\n",
        "metrics.flat_accuracy_score(y_test, y_pred)"
      ],
      "execution_count": 122,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9999941522519677"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 122
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AHvFj7ZbOYrL"
      },
      "source": [
        "#### Miscellaneous"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9dT_aiXtOUcm",
        "outputId": "59442738-4ec7-413a-d261-1547d9c6da4b"
      },
      "source": [
        "from collections import Counter\n",
        "\n",
        "def print_transitions(trans_features):\n",
        "    for (label_from, label_to), weight in trans_features:\n",
        "        print(\"%-6s -> %-7s %0.6f\" % (label_from, label_to, weight))\n",
        "\n",
        "print(\"Top likely transitions:\")\n",
        "print_transitions(Counter(crf.transition_features_).most_common(20))\n",
        "\n",
        "print(\"\\nTop unlikely transitions:\")\n",
        "print_transitions(Counter(crf.transition_features_).most_common()[-20:])"
      ],
      "execution_count": 120,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Top likely transitions:\n",
            "MD     -> VB      1.865974\n",
            "TO     -> VB      1.398479\n",
            "NNS    -> VBP     0.722801\n",
            "PDT    -> DT      0.715155\n",
            "MD     -> RB      0.643649\n",
            "DT     -> NN      0.616562\n",
            "IN     -> DT      0.609975\n",
            "CD     -> NNS     0.541022\n",
            "PRP    -> VBD     0.532698\n",
            "NNP    -> NNP     0.489169\n",
            "VBN    -> IN      0.436435\n",
            "DT     -> JJ      0.436083\n",
            "JJ     -> NN      0.424841\n",
            "NN     -> IN      0.418557\n",
            "PRP    -> VBZ     0.415554\n",
            "NNP    -> POS     0.406314\n",
            "JJ     -> NNS     0.400868\n",
            "VBP    -> VBN     0.379748\n",
            "VBZ    -> VBN     0.369834\n",
            "NNS    -> IN      0.368807\n",
            "\n",
            "Top unlikely transitions:\n",
            "NN     -> CD      -0.019863\n",
            "VBN    -> DT      -0.020518\n",
            "NNS    -> DT      -0.024247\n",
            "NN     -> JJ      -0.025881\n",
            "IN     -> VB      -0.027619\n",
            "NNP    -> PRP     -0.042101\n",
            "DT     -> RB      -0.055126\n",
            "RBS    -> IN      -0.056912\n",
            "IN     -> VBZ     -0.056988\n",
            "NN     -> VBP     -0.066587\n",
            "PRP    -> DT      -0.071444\n",
            "NNS    -> NN      -0.078229\n",
            "IN     -> IN      -0.088220\n",
            "NNP    -> NNS     -0.089631\n",
            "CD     -> TO      -0.105771\n",
            "PRP    -> VBN     -0.107548\n",
            "IN     -> VBD     -0.118718\n",
            "DT     -> DT      -0.148214\n",
            "NNP    -> VBN     -0.190701\n",
            "NNS    -> VBZ     -0.291021\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}