Badge 6 - Part-of Speech Tagging, Named Entity Recognition v2.1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook:\n",
    "\n",
    "1. [**Part-of-Speech Tagging Text**](#1)\n",
    "2. [**Named Entity Recognition**](#2)\n",
    "3. [Extra: Python **Functions**](#3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"1\"></a>\n",
    "# 1. Part-of-Speech Tagging Text\n",
    "\n",
    "#### Questions & Objectives\n",
    "\n",
    "- How can I extract words that have a particular part of speech (POS) such as a noun or a verb?\n",
    "- How can I visualise those extracted words?\n",
    "- Understand what a POS tag is\n",
    "- Use a POS tagger to label a corpus\n",
    "- Extract words with a specific POS\n",
    "- Visualise the extracted words using a frequency distribution graph and a word cloud\n",
    "\n",
    "#### Key Points\n",
    "\n",
    "- We will use NLTK’s part-of-speech tagger, `averaged_perceptron_tagger`, to label each word with a part of speech, providing information on tense, number (plural or singular) and case.\n",
    "- We will use the text from the US Presidential Inaugural speeches, in particular the text from the last speech (Trump's).\n",
    "- We will then extract all nouns, both plural (which have the tag `NNS`) and singular (which have the tag `NN`).\n",
    "- We will  visualise the nouns extracted using a frequency distribution graph and a word cloud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell now. It's the usual imports of text mining libraries.\n",
    "import nltk\n",
    "import numpy\n",
    "import string\n",
    "import matplotlib.pyplot as plt\n",
    "from nltk.tokenize import word_tokenize\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In text mining it can be useful to extract words that have a particular part of speech (POS), such as a noun or a verb. For example, extracting all proper nouns can give us names and locations. This extraction is done using a POS tagger. \n",
    "\n",
    "**The POS tag of a word is a label of the word indicating its part of speech as well as grammatical categories such as tense, number (plural or singular) and case. POS tagging is the process of automatically determining the POS tags of the tokens in a corpus.**\n",
    "\n",
    "In this lesson, we will use NLTK’s `averaged_perceptron_tagger` as the POS tagger. It uses the perceptron algorithm to predict which POS tag is most likely given the word. We need to download the tagger in order to use it.\n",
    "\n",
    "The POS-tagger outputs tokens tagged with their POS tag. It uses the [Penn Treebank POS tagset](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html), which was created with a corpus of news articles from the *Wall Street Journal*, an American newspaper, and is widely used for POS tagging text.\n",
    "\n",
    "POS tagging text is very useful when analysing a corpus or document and will allow us to do more in-depth analysis and visualisations. In order to POS tag using NLTK, you also have to `import pos_tag` from the `tag` package.\n",
    "\n",
    "We are going to use the text from the US Presidential Inaugural Addresses. This is a dataset that we can download from NLTK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's import the needed libraries.\n",
    "import pprint as pp\n",
    "import nltk\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "from nltk.tag import pos_tag\n",
    "nltk.download('inaugural')\n",
    "from nltk.corpus import inaugural\n",
    "!pip install wordcloud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This corpus comes in raw format but also is pre-tokenised. Therefore we can call the `words()` method to retrieve the tokenised text of all speeches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get list of all presidential files:\n",
    "inaugural_tokens=inaugural.words()\n",
    "print(inaugural.fileids())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load a particular file\n",
    "inaugural_tokens_one_speech = inaugural.words('1789-Washington.txt')\n",
    "print(inaugural_tokens_one_speech)\n",
    "print(len(inaugural_tokens_one_speech))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load a particular file\n",
    "inaugural_tokens_all_speeches = inaugural.words()\n",
    "print(\" \".join(inaugural_tokens_all_speeches))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at the tokens from the last speech in this corpus, the one made by President Trump (note: Biden's speech has yet to be added by the NLTK folks), by looking at the last element of the list of speeches using the `fileids()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inaugural_tokens_trump = inaugural.words(inaugural.fileids()[0:-1])\n",
    "print(inaugural_tokens_trump)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can assign POS tags to all speeches using NLTK’s `pos_tag()` method and view the first 20:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagged_inaugural_tokens = nltk.pos_tag(inaugural_tokens)\n",
    "tagged_inaugural_tokens[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"list\"></a>\n",
    "What do the abbreviations in these tags mean? The complete list is available here: https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html\n",
    "\n",
    "But here's a cheat sheet with English examples for all of us who do not have a degree in linguistics (yet): \n",
    "\n",
    "```\n",
    "CC coordinating conjunction\n",
    "CD cardinal digit\n",
    "DT determiner\n",
    "EX existential there (like: “there is” … think of it like “there exists”)\n",
    "FW foreign word\n",
    "IN preposition/subordinating conjunction\n",
    "JJ adjective ‘big’\n",
    "JJR adjective, comparative ‘bigger’\n",
    "JJS adjective, superlative ‘biggest’\n",
    "LS list marker 1)\n",
    "MD modal could, will\n",
    "NN noun, singular ‘desk’\n",
    "NNS noun plural ‘desks’\n",
    "NNP proper noun, singular ‘Harrison’\n",
    "NNPS proper noun, plural ‘Americans’\n",
    "PDT predeterminer ‘all the kids’\n",
    "POS possessive ending parent’s\n",
    "PRP personal pronoun I, he, she\n",
    "PRP$ possessive pronoun my, his, hers\n",
    "RB adverb very, silently,\n",
    "RBR adverb, comparative better\n",
    "RBS adverb, superlative best\n",
    "RP particle give up\n",
    "TO, to go ‘to’ the store.\n",
    "UH interjection, errrrrrrrm\n",
    "VB verb, base form take\n",
    "VBD verb, past tense took\n",
    "VBG verb, gerund/present participle taking\n",
    "VBN verb, past participle taken\n",
    "VBP verb, sing. present, non-3d take\n",
    "VBZ verb, 3rd person sing. present takes\n",
    "WDT wh-determiner which\n",
    "WP wh-pronoun who, what\n",
    "WP$ possessive wh-pronoun whose\n",
    "WRB wh-abverb where, when\n",
    "```\n",
    "\n",
    "This cheat sheet is from: https://medium.com/@gianpaul.r/tokenization-and-parts-of-speech-pos-tagging-in-pythons-nltk-library-2d30f70af13b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then set up lists to hold specific parts of speech, such as nouns. First we set up an empty list and then we search for the noun tags, `NN` for singular and `NNS` for plural. We can then print the first 20:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nouns = [] \n",
    "nouns = [word \n",
    "         for (word, pos) in tagged_inaugural_tokens \n",
    "         if (pos == 'NN' or pos == 'NNS')] \n",
    "\n",
    "nouns[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have created this list of nouns, we can plot their counts in the corpus (see the lesson on frequency counts)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.probability import FreqDist\n",
    "fdist = FreqDist(nouns)\n",
    "fdist.plot(20,title='Frequency Distribution for the 20 Most Common Nouns in the Inaugural Corpus')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐛Minitask 1.1\n",
    "\n",
    "Plot the frequency distribution for the *30* most frequent nouns.  Take a look at the graph and spot any errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy-paste the code from the previous cell and change it to visualise the 30 most common nouns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary style='color:blue'>CLICK HERE TO SEE THE ANSWER. BUT REALLY TRY TO DO IT YOURSELF FIRST!</summary>\n",
    "    \n",
    "    ### BEGIN SOLUTION\n",
    "    fdist.plot(30,title='Frequency Distribution for the 30 Most Common Nouns in the Inaugural Corpus')\n",
    "    ### END SOLUTION\n",
    "    \n",
    "Notice one of the frequent nouns listed is apparently \"s\".  What could this be?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot the nouns as a word cloud like we did yesterday (see the lesson on counting tokens in text)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "cloud = WordCloud(width=800, height=400, max_font_size=160,colormap=\"hsv\").generate(' '.join(nouns))\n",
    "\n",
    "plt.rcParams[\"figure.figsize\"] = (16,12)\n",
    "plt.imshow(cloud, interpolation='bilinear')\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐛Minitask 1.2\n",
    "\n",
    "Change the code above to create a frequency list for the 10 most common adjectives in the inaugural corpus. The POS tag for adjective is `JJ`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your solution here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary style='color:blue'>CLICK HERE TO SEE THE ANSWER. BUT REALLY TRY TO DO IT YOURSELF FIRST!</summary>\n",
    "    \n",
    "    ### BEGIN SOLUTION\n",
    "    adjectives = []\n",
    "    adjectives = [word for (word, pos) in tagged_inaugural_tokens if (pos == 'JJ')]\n",
    "    fdist = FreqDist(adjectives)\n",
    "    fdist.plot(10,title='Frequency Distribution for the 10 Most Common Adjectives in the Inaugural Corpus')\n",
    "    ### END SOLUTION\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐛Minitask 1.3\n",
    "\n",
    "Plot a word cloud of the adjectives in the inaugural corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your solution here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary style='color:blue'>CLICK HERE TO SEE THE ANSWER. BUT REALLY TRY TO DO IT YOURSELF FIRST!</summary>\n",
    "\n",
    "    ### BEGIN SOLUTION\n",
    "    \n",
    "    # ORIGINAL CODE: cloud = WordCloud(max_font_size=60,colormap=\"hsv\").generate(' '.join(adjectives))\n",
    "    \n",
    "    # LUCY'S VERSION: cloud = WordCloud(width=800, height=400, max_font_size=160,colormap=\"hsv\").generate(' '.join(adjectives))\n",
    "    \n",
    "    plt.rcParams[\"figure.figsize\"] = (16,12)\n",
    "    plt.imshow(cloud, interpolation='bilinear')\n",
    "    plt.axis('off')\n",
    "    plt.show()\n",
    "    ### END SOLUTION\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐛Minitask 1.4\n",
    "\n",
    "You can do the same for another POS tag. For the cheat sheet and full list of Penn Treebank POS tags, see [here](#list) in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your solution here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary style='color:blue'>CLICK HERE TO SEE THE ANSWER. BUT REALLY TRY TO DO IT YOURSELF FIRST!</summary>\n",
    "\n",
    "    ### BEGIN SOLUTION\n",
    "    \n",
    "    # ORIGINAL CODE: cloud = WordCloud(max_font_size=60,colormap=\"hsv\").generate(' '.join(nouns))\n",
    "    \n",
    "    # LUCY'S VERSION: cloud = WordCloud(width=800, height=400, max_font_size=160,colormap=\"hsv\").generate(' '.join(nouns))\n",
    "    \n",
    "    plt.rcParams[\"figure.figsize\"] = (16,12)\n",
    "    plt.imshow(cloud, interpolation='bilinear')\n",
    "    plt.axis('off')\n",
    "    plt.show()\n",
    "    ### END SOLUTION\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"2\"></a>\n",
    "# 2. Named Entity Recognition\n",
    "\n",
    "\n",
    "#### Questions & Objectives\n",
    "\n",
    "- How can I identify all the names in a corpus?\n",
    "- How can I visualise those names?\n",
    "- Understand what named entity recognition is\n",
    "- Understand how to use a named entity recogniser\n",
    "- Extract named entities from text\n",
    "- Visualise the extracted entities\n",
    "\n",
    "#### Key Points\n",
    "\n",
    "- We will use spaCy's named entity recogniser to extract named entities from text.\n",
    "- We will continue to use the text from the US Presidential Inaugural Addresses.\n",
    "- We will then extract certain entity types (e.g., all person names) and will see some errors in the output.\n",
    "- We will visualise the entities extracted using a word cloud.\n",
    "\n",
    "Named entity recognition (NER) or named entity tagging is a way of locating and labelling named entities in text (e.g., names of people, locations and organisations). NER can be used to identify networks of people mentioned in data or as the first step towards geo-parsing text, i.e., extracting and disambiguating locations in text.\n",
    "There are several off-the-shelf NER taggers that we can use. Here we will use the [spaCy](https://spacy.io) tagger.\n",
    "\n",
    "To use the spaCy tagger, you first need to install spaCy and load the model required to run it. You also need to import spaCy and its models, and load it. After running the code cell below, you should get a bunch of messages and if it worked, it will say that the download and installation were successful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U spacy\n",
    "!python3 -m spacy download en_core_web_sm\n",
    "import spacy\n",
    "import en_core_web_sm\n",
    "nlp = en_core_web_sm.load()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have already loaded the inaugural corpus from NLTK as an example, so we will use its raw text (`inaugural.raw()`) as input to tag it with named entities (in this case names of people, organisations and locations). To do that we just call `nlp()` on that list of word tokens. This can take a few seconds as it has to process the entire inaugural corpus. The results are stored in the variable `doc`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text=inaugural.raw()\n",
    "len(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's just grab 1ast 100000 items (to save time)\n",
    "documents=nlp(text[-100000:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then print the first 10 entities found in the text and their entity tags using a for loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for index in range(10):\n",
    "    print(documents.ents[index].text + \", \" + documents.ents[index].label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can save the entities separately in a list of tuples by extracting them from the output `doc` and then print the first 20 entries of that list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ents=documents.ents\n",
    "named_entities = [(ent.text, ent.label_) \n",
    "                  for ent in ents] \n",
    "named_entities[0:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check out all the different entity types that have been found here (`ORG`, `DATE`, `ORDINAL`, `GPE` (geopolitical entity), etc.). The complete named entity tagset can be found in the spaCy [documentation](https://spacy.io/models/en#en_core_web_sm) but we list them here for convenience:\n",
    "\n",
    "```\n",
    "CARDINAL    ... Numerals that do not fall under another type\n",
    "DATE        ... Absolute or relative dates or periods\n",
    "EVENT       ... Named hurricanes, battles, wars, sports events, etc.\n",
    "FAC         ... Facility (Buildings, airports, highways, bridges, etc.)\n",
    "GPE         ... Geo-political entity (countries, cities, states)\n",
    "LANGUAGE    ... Any named language\n",
    "LAW         ... Named documents made into laws\n",
    "LOC         ... Non-GPE locations, mountain ranges, bodies of water\n",
    "MONEY       ... Monetary values, including unit\n",
    "NORP        ... Nationalities or religious or political groups\n",
    "ORDINAL     ... “first”, “second”\n",
    "ORG         ... Organisation (companies, agencies, institutions, etc.)\n",
    "PERCENT     ... Percentage (including “%”)\n",
    "PERSON      ... People, including fictional\n",
    "PRODUCT     ... Vehicles, weapons, foods, etc. (Not services)\n",
    "QUANTITY    ... Measurements, as of weight or distance\n",
    "TIME        ... Times smaller than a day\n",
    "WORK_OF_ART ... Titles of books, songs, etc.\n",
    "```\n",
    "\n",
    "The spaCy model (```en_core_web_sm```), which we use in this course, was trained using a large corpus called [OntoNotes](https://catalog.ldc.upenn.edu/docs/LDC2013T19/OntoNotes-Release-5.0.pdf).  Essentially the model was derived by means of a large text collection tagged with the information needed for each of the different tasks (e.g., POS tagging and named entity recognition).\n",
    "\n",
    "While we don't go into details on machine learning and how models are trained in this course, it is worth knowing that spaCy provides the functionaliy to train and thereby create your own models for sequence tagging, as is needed to process running text.  So for example, if you'd like to identify names of medicines in text, then this would be possible given a sufficiently sizeable training dataset marked up with medicine names. You would then be able to train your own model which can in turn be run on new text to perform the task it is trained to do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐛Minitask 2.1\n",
    "\n",
    "Print the last 20 entities in the inaugural corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your solution here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary style='color:blue'>CLICK HERE TO SEE THE ANSWER. BUT REALLY TRY TO DO IT YOURSELF FIRST!</summary>\n",
    "\n",
    "    ### BEGIN SOLUTION\n",
    "    # You need to specify the start counter to be 20 from the end (end minus 20)\n",
    "    # You can determine the end by using the len() method on the list\n",
    "    # The end counter can be left unspecified\n",
    "    # The following statement returns the last 20 elements of the list\n",
    "    named_entities[len(named_entities)-20:]\n",
    "    ### END SOLUTION\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point we will introduce the concept of a **function** to make our work easily repeatable for different types of entities.\n",
    "<a id=\"3\"></a>\n",
    "# 3. Python Functions: The Most Powerful Tool in a Programmer's Hands\n",
    "\n",
    "__Function:__ A function is a block of code that can be called to complete a task.\n",
    "\n",
    "We already used a lot of functions that Python creators have created for us, like `print()` or `len()`. Notice what `len()` does:\n",
    "\n",
    "`length_of_list = len( [4,5,6] )`\n",
    "\n",
    "It's like a magic spell that Harry Potter has learned, and now he can use it in different contexts and on different things.\n",
    " \n",
    "For example, once Harry learned how to freeze things with the `glacius` spell, he could then use it on various objects: \n",
    "\n",
    "`ice = glacius( water )`\n",
    "\n",
    "`frozen_malfoy = glacius( malfoy )`\n",
    "\n",
    "Notice that this spell **takes** some **ARGUMENTS** which are sort of like inputs, and **returns** something new or changed.\n",
    "\n",
    "The syntax of using a function is:\n",
    "\n",
    "`result = function_name( argument )` or even \n",
    "\n",
    "`result = function_name( argument1, argument2, argument3, etc. )`\n",
    "\n",
    "\n",
    "#### Creating our own functions\n",
    "\n",
    "The most amazing thing is that we can create our own functions:\n",
    "\n",
    "In Python, a function is defined using the ```def``` keyword. The name of the function is specified after ```def``` and the parameters passed to the function are specified in parentheses `()` after the name.  The indented bit of code inside a function specifies everything that the function computes. A function can return data as a result of the computation it has performed using the ```return``` keyword.\n",
    "\n",
    "`def my_own_function( argument1, argument2, argument3):\n",
    "    do something with arguments\n",
    "    return some_value`\n",
    "\n",
    "The important detail is: when you define a function, it is like LEARNING a spell. You did not yet CALL, TRIGGER, or CAST that spell, just as it is possible to learn a freezing spell without actually freezing things around you.\n",
    "\n",
    "#### Using our own functions\n",
    "\n",
    "- Define a function: teach the computer what to do, as in `def something():`\n",
    "- Call a function: use the skill you learned earlier, as in `something()`\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here I define the function - notice nothing happens yet when you run this cell!\n",
    "def add_numbers( number1, number2):\n",
    "    print(\"Adding numbers now!\")\n",
    "    return number1+ number2\n",
    "\n",
    "# \"Adding numbers now!\" is NOT printed yet, because the function is defined but not used (or called) yet!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here I call the function and it will return a value:\n",
    "print(\"I'm about to do addition.\")\n",
    "my_result = add_numbers(3,5)\n",
    "print(\"I'm done adding.\")\n",
    "print(my_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's another example of a function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_first_number_larger( number1, number2):\n",
    "    if number1 > number2:\n",
    "        return True\n",
    "    else:\n",
    "        return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_result = is_first_number_larger(3,5)\n",
    "print(my_result)\n",
    "my_result = is_first_number_larger(12,5)\n",
    "print(my_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You could even put the return value of a function right into a print statement:\n",
    "\n",
    "print(is_first_number_larger(12,5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐛Minitask 3.1\n",
    "\n",
    "Write a function that tells you if a word is longer (in terms of its length of characters) than a number, ```is_word_longer_than(word, number)```, that you could call like in the print statements below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your function here, so that below code (which calls your function many times)\n",
    "# works and doesn't result in an error.\n",
    "\n",
    "\n",
    "\n",
    "# Tests:\n",
    "print(is_word_longer_than(\"banana\", 4)) # should be true\n",
    "print(is_word_longer_than(\"banana\", 12)) # should be false\n",
    "print(is_word_longer_than(\"plum\", 3)) # should be true\n",
    "print(is_word_longer_than(\"plum\", 4)) # should be false\n",
    "print(is_word_longer_than(\"plum\", 5)) # should be false"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary style='color:blue'>CLICK HERE TO SEE THE ANSWER. BUT REALLY TRY TO DO IT YOURSELF FIRST!</summary>\n",
    "\n",
    "    ### BEGIN SOLUTION\n",
    "    def is_word_longer_than(word, number):\n",
    "        if len(word) > number:\n",
    "            return True\n",
    "        else:\n",
    "           return False\n",
    "    ### END SOLUTION\n",
    "    \n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Back into text mining, now using functions to help us with our work\n",
    "\n",
    "Below is the ```get_entities_of_type()``` function.  \n",
    "\n",
    "It contains code that extracts named entities of a specific type from NER-tagged output.  This function takes as input the chosen ```type``` of entity and  ```named_entities``` list. It will use them to filter entities and return only the entities of the specified type marked up in the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_entities_of_type(type_we_seek, entities):\n",
    "    ents = [string for (string, tag) in entities if (tag == type_we_seek)] \n",
    "    # You can add new lines to the above line if you like\n",
    "    print(\"Number of strings tagged as \" + type_we_seek + \" \" + str(len(ents)))\n",
    "    return ents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run the function, for example to select all entities tagged as ```PERSON```, you need to specify the type and the list of named entities to select from.  The output list can be stored in a new variable (```people```). You can then inspect the first few person entities found in the inaugural speeches and you will see that the tagger is not always correct (e.g., recourse, Hitherto)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type=\"PERSON\"\n",
    "people = get_entities_of_type(type, named_entities)\n",
    "people[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the difference between using and not using functions:\n",
    "\n",
    "- Without a function:\n",
    "\n",
    "`type=\"PERSON\"\n",
    "persons = [string for (string, tag) in named_entities if (tag == type)]\n",
    "type=\"ORG\"\n",
    "organisations = [string for (string, tag) in named_entities if (tag == type)]`\n",
    "\n",
    "- With a function:\n",
    "\n",
    "`types=\"PERSON\"\n",
    "persons = get_entities_of_type(type, named_entities)\n",
    "type=\"ORG\"\n",
    "organisations = get_entities_of_type(type, named_entities)`\n",
    "\n",
    "The code is cleaner and prettier with a function. Also, with functions, our calculations require less code. Imagine we needed 10 lines of calculations in the top bit...that's a lot more coding than the examples with a function!\n",
    "\n",
    "Using a function, you only need to define the code once and can execute it multiple times with different arguments.\n",
    "\n",
    "## Rule of Thumb: if you do something a lot, and catch yourself copy-pasting a lot of code, create a function for it, passing all the 'variables' (things that vary) into the function as arguments.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot the entities extracted by their frequency or create word clouds for them (as we did in the previous lesson).  For example, we can create a word cloud just for the person names mentioned in the corpus.\n",
    "\n",
    "To do that, we need to create a `Counter` dictionary containing the counts for each repetition of an entity.  A `Counter` is a subclass of a `Dictionary` in which elements are stored as keys and their counts are stored as values.\n",
    "\n",
    "As you can see, some person names in the inaugural speeches occur more than others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "people_dict=Counter(people)\n",
    "print(\"Number of unique person names: \" + str(len(people_dict)))\n",
    "print(people_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cloud = WordCloud(width=800, height=400, max_font_size=160,colormap=\"hsv\").generate_from_frequencies(people_dict)\n",
    "\n",
    "plt.figure(figsize=(16,12))\n",
    "plt.imshow(cloud, interpolation='bilinear')\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐛Minitask 3.2\n",
    "\n",
    "Create a similar word cloud but for the geopolitical entities (`GPE`), i.e., countries, cities or states, in the corpus. You can reuse the function we created earlier to select `GPE` entities.  If you reuse code, remember to adjust the variable names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your solution here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary style='color:blue'>CLICK HERE TO SEE THE ANSWER. BUT REALLY TRY TO DO IT YOURSELF FIRST!</summary>\n",
    "\n",
    "    ### BEGIN SOLUTION\n",
    "    gpes = get_entities_of_type(\"GPE\", named_entities)\n",
    "    gpe_dict=Counter(gpes)\n",
    "\n",
    "    cloud = WordCloud(width=800, height=400, max_font_size=160,colormap=\"hsv\").generate_from_frequencies(gpe_dict)\n",
    "    \n",
    "    plt.figure(figsize=(16,12))\n",
    "    plt.imshow(cloud, interpolation='bilinear')\n",
    "    plt.axis('off')\n",
    "    plt.show()\n",
    "    ### END SOLUTION\n",
    "    \n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🦋 Extra task (optional):\n",
    "\n",
    "Look at the most common parts of code that we reused in the notebooks and turn them into functions.\n",
    "\n",
    "These functions can be your tools to use later, like a personal cheat sheet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your solution here\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}