Day4_spacy_ner.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NER with spaCy\n",
    "**\"Named Entity Recognition\"** is a subtask of NLP where we extract specific named entities from the text. The definition of a \"named entity\" changes depending on the domain we're working on. We'll look at clinical NER later, but first we'll look at some examples in more general domains.\n",
    "\n",
    "NER is often performed using news articles as source texts. In this case, named entities are typically proper nouns, such as:\n",
    "- People\n",
    "- Geopolitical entities, like countries\n",
    "- Organizations\n",
    "\n",
    "We won't go into the details of how NER is implemented in spaCy. If you want to learn more about NER and various way it's implemented, a great resource is [Chapter 17.1 of Jurafsky and Martin's textbook \"Speech and Language Processing.\"](https://web.stanford.edu/~jurafsky/slp3/17.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy import displacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an excerpt from an article in the Guardian. We'll process this document with our nlp object and then look at what entities are extracted. One way to do this is using spaCy's `displacy` package, which visualizes the results of a spaCy pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"Germany will fight to the last hour to prevent the UK crashing out of the EU without a deal and is willing \n",
    "to hear any fresh ideas for the Irish border backstop, the country’s ambassador to the UK has said.\n",
    "Speaking at a car manufacturers’ summit in London, Peter Wittig said Germany cherished its relationship \n",
    "with the UK and was ready to talk about solutions the new prime minister might have for the Irish border problem.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use spaCy's `explain` function to see definitions of what an entity type is. Look up any entity types that you're not familiar with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spacy.explain(\"GPE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last example comes from a political news article, which is pretty typical for what NER is often trained on and used for. Let's look at another news article, this one with a business focus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example 2\n",
    "text = \"\"\"Taco Bell’s latest marketing venture, a pop-up hotel, opened at 10 a.m. Pacific Time Thursday. \n",
    "The rooms sold out within two minutes.\n",
    "The resort has been called “The Bell: A Taco Bell Hotel and Resort.” It’s located in Palm Springs, California.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "Compare how the NER performs on each of these texts. Can you see any errors? Why do you think it might make those errors?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Coding Exercise\n",
    "Write a function to that takes a doc as an argument and returns a dictionary mapping each entity type label to a list of that entity in the doc. Try creating a few different doc instances and testing this function out.\n",
    "\n",
    "**Note**: A doc's entities can be accessed in the attribute `doc.ents`. An entity's label can be accessed in the attribute `ent.label_`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "def collect_entities(doc):\n",
    "    \"\"\"\n",
    "    \"\"\"\n",
    "    d = defaultdict(list)\n",
    "    # Your code here\n",
    "    return d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "collect_entities(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clinical Text\n",
    "Let's now try using spaCy's built-in NER model on clinical text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clin_text = \"76 year old man with hypotension, CKD Stage 3, status post RIJ line placement and Swan.  \""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(clin_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion**\n",
    "- How did spaCy do with this sentence?\n",
    "- What do you think caused it to make errors in the classifications?\n",
    "\n",
    "General purpose NER models are typically made for extracting entities out of news articles. As we saw before, this includes mainly people, organizations, and geopolitical entities. \n",
    "\n",
    "**Discussion**\n",
    "- What are some entity types we are interested in in clinical domain?\n",
    "- Does spaCy's out-of-the-box NER handle any of these types?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner = nlp.pipeline[-1][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner.labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pattern Matching\n",
    "It's clear that spaCy's out-of-the-box NER is not going to fit our needs. In that case, we need to take matters into our own hands. SpaCy has several methods which enable us to do rule-based matching, while still having access to the many linguistic attributes which are classified by spaCy's statistical models. \n",
    "\n",
    "One such method is called the `Matcher`. This is a class which allows us to write rules which will match tokens based on various attributes in order to extract information according to our own needs. The simplext form of this is going to be matching the exact string: for example, match the strings \"hypotension\" and \"CKD Stage 3\". However, there may be lots of different variations, and using spaCy's Matcher allows us to be flexible. For example, we may want to match not only \"CKD Stage 3\", but also stages 1-6. We may also want to handle different abbreviations or capitalizations.\n",
    "\n",
    "Here is a good demonstration of the Matcher: https://explosion.ai/demos/matcher\n",
    "\n",
    "Let's demonstrate this by writing a few rules with a Matcher. We'll first write a list of **patterns**. Each pattern is a list of dicts, and each dict represents a single token. The dict maps a certain attribute to a value. The simplest form of this is to just look at the \"TEXT\" attribute, which matches an exact string:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pattern1 = [{'TEXT': 'hypotension'}]\n",
    "pattern2 = [{'TEXT': 'CKD'}, {'TEXT': 'Stage'}, {'TEXT': '3'}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then add each pattern to the Matcher object and run it on a doc:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.matcher import Matcher"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matcher = Matcher(nlp.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(clin_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matcher.add('CLINICAL_PATTERN1', None, pattern1)\n",
    "matcher.add('CLINICAL_PATTERN2', None, pattern2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matches = matcher(doc)\n",
    "for match_id, start, end in matches:\n",
    "    matched_span = doc[start:end]\n",
    "    print(matched_span.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, we can match on a lot more than just the text, and this is where those linguistic attributes we looked at yesterday come in handy. Open up spaCy's documentation to see more about this:\n",
    "https://spacy.io/usage/rule-based-matching#adding-patterns-attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you'll write a slightly more comlex pattern. Try writing a single pattern which matches both 'stage 4 ckd' and 'Stage 3 CKD'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clin_text2 = \"the pt presents for stage 4 ckd. He previously had Stage 3 CKD.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pattern = [\n",
    "    {'': 'stage'},\n",
    "    {'': ''},\n",
    "    {'LOWER': ''},\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matcher.add('CLINICAL_PATTERN', None, pattern)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(clin_text2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matches = matcher(doc)\n",
    "for match_id, start, end in matches:\n",
    "    matched_span = doc[start:end]\n",
    "    print(matched_span.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment: Write your own rule-based matcher\n",
    "Use the `Matcher` class to extract the following concepts from these texts:\n",
    "- \"Procedure\"\n",
    "- \"Condition\"\n",
    "\n",
    "You'll first have to identify all of the instances of these concepts in the text below. Then add to the `patterns` list to match all of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "long_text = (\n",
    "    \"There is continued mild-to-moderate congestive heart failure. \"\n",
    "    \n",
    "    \"87-year-old man with htn and end-stage renal disease. \"\n",
    "    \n",
    "    \"His wife recently died from end stage renal disease. \"\n",
    "    \n",
    "    \"The patient is s/p median sternotomy and right thoracotomy \"\n",
    "    \n",
    "    \"The pt presents for stage 4 ckd \" \n",
    "    \n",
    "    \"He previously had stage 3 CKD.\"\n",
    "    \n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patterns = [\n",
    "     [{'TEXT': 'htn'}],\n",
    "    # Add the ckd pattern which you wrote earlier\n",
    "    [{'': 'stage'}, {'': ''}, {'ower': 'ckd'}],\n",
    "    \n",
    "    # Add any other patterns\n",
    "\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matcher = Matcher(nlp.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for pattern in patterns:\n",
    "    matcher.add('CLINICAL_PATTERN', None, pattern)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(long_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matches = matcher(doc)\n",
    "for match_id, start, end in matches:\n",
    "    matched_span = doc[start:end]\n",
    "    print(matched_span.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A quick note about negation\n",
    "With pyConText, we looked at how we can detect modifiers in texts such as negation, certainty, and experiencer. These are very important concepts in clinical text, but aren't necessarily as much of a focus in other domains. There is (currently) no specific ConText module in spaCy, but we can do some basic negation detection by using dependency parsing.\n",
    "\n",
    "Let's see how spaCy parses the following sentence. We can then look for negated terms and children of the negated word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(\"There is not cancer.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "displacy.render(doc, style=\"dep\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  https://stackoverflow.com/questions/54849111/negation-and-dependency-parsing-with-spacy\n",
    "negation_tokens = [tok for tok in doc if tok.dep_ == 'neg']\n",
    "negation_head_tokens = [token.head for token in negation_tokens]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here are the negated terms\n",
    "for token in negation_head_tokens:\n",
    "    print(f\"Negated: {token.text}\")\n",
    "    print(\"Children:\")\n",
    "    # And here are all of its children\n",
    "    for child in token.children:\n",
    "        print('\\t', child, child.pos_)\n",
    "#     print([child for child in token.children])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a very simple negation rule: We'll say that if a term is negated (such as \"**is** not\"), then any of its children which are nouns should be considered a negated concept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_negated_concepts(doc):\n",
    "    negation_concepts = []\n",
    "    negation_tokens = [tok for tok in doc if tok.dep_ == 'neg']\n",
    "    negation_head_tokens = [token.head for token in negation_tokens]\n",
    "    \n",
    "    for token in negation_head_tokens:\n",
    "        # And here are all of its children\n",
    "        for child in token.children:\n",
    "            if child.pos_ == 'NOUN':\n",
    "                negation_concepts.append(child)\n",
    "    return negation_concepts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_negated_concepts(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This worked! Now lets' try it on another sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(\"The patient does not have pneumonia, hypertension, or heart disease.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "displacy.render(doc, style=\"dep\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_negated_concepts(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion**: How did our negation function do? Were there any concepts which should have been negated, but weren't? Were they any concepts which were negated and shouldn't have been? What does that tell you about this method of negation, and how does it compare to pyConText?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}