Badge 7 - Visualising a network of entities v2.1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook:\n",
    "\n",
    "1. [Visualising entities highlighted in text](#1)\n",
    "2. [Visualising a network of entities](#2)\n",
    "\n",
    "But before we start ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell now. It's the usual imports of libraries.\n",
    "\n",
    "import nltk\n",
    "import numpy\n",
    "import string\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll keep working with the inaugural corpus\n",
    "nltk.download('inaugural')\n",
    "from nltk.corpus import inaugural"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We're also going to import some more: this is to install spaCy and its model.  \n",
    "# You may have that installed already but we'll do that here just in case.\n",
    "!pip install -U spacy # or pip3\n",
    "!python -m spacy download en_core_web_sm # or python3\n",
    "import spacy\n",
    "import en_core_web_sm\n",
    "nlp = en_core_web_sm.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"1\"></a>\n",
    "# 1. Entities highlighted in text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the named entity recognition and highlighting it in text.  To do that we can use a tool that accompanies spaCy called ```displacy```.\n",
    "\n",
    "`displacy.render(doc, style=\"ent\",jupyter=True)`  would visualise the text with the entities highlighted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a list of entity types:\n",
    "\n",
    "```\n",
    "NER: \n",
    "CARDINAL, DATE, EVENT, FAC, GPE, LANGUAGE, LAW, LOC, MONEY, NORP, ORDINAL, ORG, PERCENT, PERSON, PRODUCT, QUANTITY, TIME, WORK_OF_ART\n",
    "```\n",
    "\n",
    "For more details browse the [spaCy documentation](https://spacy.io/models/en#en_core_web_sm) or go back to the notebook on named entity recognition to see the label scheme and what each of the tags mean.\n",
    "\n",
    "The following code imports spaCy, loads an English model, defines a bit of text, runs the model on the text and displays the entities recognised in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy import displacy\n",
    "\n",
    "# We reload the model (just in case)\n",
    "nlp = en_core_web_sm.load() \n",
    "\n",
    "# We define a example bit of text by concatenating a couple of sentences togeter.\n",
    "text = \"When Sebastian Thrun started working on self-driving cars at Google in 2007, \"\n",
    "text += \"few people outside of the company took him seriously. Now it changed the Unided States and the world\"\n",
    "\n",
    "# This runs spacy on the text\n",
    "doc = nlp(text)\n",
    "\n",
    "# This displays the text with the entities highlighted in different colours\n",
    "displacy.render(doc, style=\"ent\",jupyter=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐛Minitask 1.1\n",
    "\n",
    "Do this highlighting for Obama's speech. Tip: to get the Obama speech from NLTK you can call `inaugural.raw('2009-Obama.txt')`.  You need the raw text as input into spaCy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary style='color:blue'>CLICK HERE TO SEE THE ANSWER. BUT REALLY TRY TO DO IT YOURSELF FIRST!</summary>\n",
    "\n",
    "    ### BEGIN SOLUTION\n",
    "    text=inaugural.raw('2009-Obama.txt')\n",
    "    doc = nlp(text)\n",
    "    displacy.render(doc, style=\"ent\",jupyter=True)\n",
    "    ### END SOLUTION\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the output.  You'll notice that it's easier to spot mistakes but in general named entity recognition is fairly accurate for this speech.\n",
    "\n",
    "Here is a bunch of speeches from Obama which we provide for along with the notebooks.  Remember how you can use the PlaintextCorpusReader that comes with NLTK to load a corpus of text files.\n",
    "\n",
    "Have a look at what named entities spaCy recognises in these speeches. Here we are only displaying the start of the speech."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import PlaintextCorpusReader\n",
    "corpus_root = \"./data/barack_obama_speeches\"\n",
    "corpus_data = PlaintextCorpusReader(corpus_root, '.*', encoding='latin1') \n",
    "plain_text = corpus_data.raw()[:2000] # look at the first 2000 characters only\n",
    "doc = nlp(plain_text)\n",
    "displacy.render(doc, style=\"ent\",jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"2\"></a>\n",
    "# 2. Visualising a network of entities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "You can use network visualisations to show how things are connected in data.  Networks are often used for showing networks of people in social media.  Today we'll use a network visualisation to display how all US presidents are connected in terms of the people they mention in their speeches.\n",
    "\n",
    "To do that you'll first learn how to save data to a file. The content of the file serves as the input to the network. Let's first create the content (```file_content```) which we will write to a file.  We'll be creating a TSV (tab-separated values) file, meaning that each line contains different fields separated by a tab.  The first line is the header which lists the names for each column.  The first column is the names of presidents and the second column is the people mentioned in their speeches.\n",
    "\n",
    "The president/person pairs are created by looping through each speech, extracting the name of each president from the filename of the speech, NER tagging and extracting people names for each speech.  For all people extracted from each speech, we then concatenate the ```file_content``` variable with the president's name (```president```) and each person name (```person```) mentioned in his speech followed by a newline character (`\"\\n\"`) so that each pair is on one line. The final ```file_content``` contains the header and numerous lines listing all the president/person pairs in this data set.\n",
    "\n",
    "The following code was adapted from Jonathan Soma's tutorial on [NetworkX Graphs](http://jonathansoma.com/lede/algorithms-2017/classes/networks/networkx-graphs-from-source-target-dataframe/). __Note: this first bit of code takes some time to run because each speech has to be tagged individually.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this will be a file with all people spoken about, and name of president who spoke about them!\n",
    "header=\"president\\tperson\\n\"\n",
    "file_content=header\n",
    "\n",
    "for fileid in inaugural.fileids():\n",
    "    president=fileid[5:-4] # eg in file '1973-Nixon.txt' name of president is 'Nixon'\n",
    "    #nertagged_words = st.tag(inaugural.words(fileid))\n",
    "    doc = nlp(inaugural.raw(fileid))\n",
    "    named_entities = [(ent.text, ent.label_) \n",
    "                      for ent in doc.ents] \n",
    "    #named_entities=get_entities(nertagged_words)\n",
    "    people = [word \n",
    "              for (word, tag) in named_entities \n",
    "              if (tag == 'PERSON') and (not word.startswith('\\n'))]\n",
    "    if len(people) > 0:\n",
    "        for person in people:\n",
    "            file_content=file_content + president + \"\\t\" + person + \"\\n\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's see what we created:\n",
    "print(file_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To write to a file you need to use the ```open()``` method and specify the file's name and mode (in this case \"w\" for write to file).  You can then use the ```write()``` method and specify the content of the file in brackets.  Don't forget to close the file using ```close()``` once you're done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open(\"presidents.tsv\", \"w\")\n",
    "f.write(file_content)\n",
    "f.close()\n",
    "# look at your list of files! Do you see a new file called presidents.tsv? open it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check that file was created by looking at the Jupyter Notebook homepage in the browser.  The file (`presidents.tsv`) should now be listed in the same directory as you notebook.\n",
    "\n",
    "We need to import two further packages to visualise a network.  ```pandas``` is used to represent data frames. and ```networkx``` is needed to create a network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import networkx as nx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__DataFrame:__ A DataFrame is a 2-dimensional labeled data structure with columns of potentially different types.\n",
    "\n",
    "First we create a DataFrame by reading in the tsv file which we just created.  The top of the DataFrame can be displayed by calling the ```head()``` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"presidents.tsv\", sep='\\t')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first column in the DataFrame will serve as the source and the second column as the target in the network.  We can create a simple network specifying source and target using the headers of the columns in the data frame.\n",
    "\n",
    "__Node:__ A network node is a connection point.\n",
    "\n",
    "__Edge:__ A network edge is a connection between two nodes. \n",
    "\n",
    "This initial network isn't very informative yet as you can't see who each node (blue circles) refers to.  So next we need to make it more readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g = nx.from_pandas_edgelist(df, source='president', target='person') \n",
    "nx.draw(g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to create a list of all presidents by extracting their names from the DataFrame and removing repetitions. Note, these are only presidents whose speeches contain person names tagged by the NER tagger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "presidents = list(df.president.unique())\n",
    "presidents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll do the same for the names of people recognised in the speeches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "people = list(df.person.unique())\n",
    "people"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The degree of a node in a network is the number of connections it has to other nodes.  For example, the degree for \"Reagan\" can be calculated using the ```degree()``` method on a network.  We will use degree as a way of varying the node size of person nodes, people with a larger degree will be displayed bigger in the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g.degree(\"Reagan\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following bit of code draws the network. The first two lines set the size of the graph and the layout of the network.  \n",
    "\n",
    "After that the different nodes are drawn using different parameters for `node_color` and `node_size`.  Then the edges and node labels are drawn and the title of the network is specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting the size of the network\n",
    "plt.figure(figsize=(30, 30))\n",
    "\n",
    "# you can choose different ways of laying out the network\n",
    "layout = nx.spring_layout(g, k = 0.4)\n",
    "\n",
    "# go through every person name, ask the graph how many\n",
    "# connections it has. Multiply that by 100 to get the circle size\n",
    "person_size = [g.degree(person) * 100 for person in people]\n",
    "\n",
    "# draw the people nodes\n",
    "nx.draw_networkx_nodes(g, \n",
    "                       layout, \n",
    "                       nodelist=people,\n",
    "                       node_color='lightblue',\n",
    "                       node_size=person_size # a list of sizes, based on g.degree\n",
    "                       )\n",
    "\n",
    "# draw president nodes\n",
    "nx.draw_networkx_nodes(g, layout, nodelist=presidents, node_color='wheat', node_size=100)\n",
    "\n",
    "# popular presidents (with a degree that's greater than 1)\n",
    "popular_president = [president for president in presidents if g.degree(president) > 1 ]\n",
    "\n",
    "# draw popular president nodes\n",
    "nx.draw_networkx_nodes(g, layout, nodelist=popular_president, node_color='orange', node_size=100)\n",
    "\n",
    "# draw network edges\n",
    "nx.draw_networkx_edges(g, layout, width=1, edge_color=\"#cccccc\")\n",
    "\n",
    "# draw node labels\n",
    "nx.draw_networkx_labels(g, layout, font_size=12)\n",
    "\n",
    "plt.axis('off')\n",
    "plt.title(\"Presidents and people\", fontsize=20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When looking at this network, just as a reminder, orange nodes are popular presidents, wheat (beige) nodes are other presidents and blue nodes are people mentioned in the speeches.  In some cases presidents and people nodes overlap.  In some cases the same person is represented by different nodes (e.g., Roosevelt, Franklin Roosevelt and Franklin Delano Roosevelt).  So this network could be improved if we were able to link different entity mentions to the same person in real life."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐛Minitask 2.1\n",
    "\n",
    "Copy-paste the code of the above visualisation into the cell below and try to customise it.  For example, what does the network look like for presidents and geopolitical entities?  Consider writing functions to structure your code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here.  There's not one solution for this so we don't provide an answer in this case.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}