Badge 2 - Tokenising Text and Pre-processing Data Collections v2.0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook:\n",
    "\n",
    "1. [Loading and Tokenising Text](#1)\n",
    "2. [Loading a Text File: From Your Notebook or From a Website](#2)\n",
    "3. [Tokenising Strings: Splitting Them into Tokens (Words, etc.)](#3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"1\"></a>\n",
    "# 1. Loading and Tokenising Text\n",
    "\n",
    "### Recap of syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a list, get its first item and get its length.\n",
    "my_letters = [\"A\",\"B\",\"C\",\"D\",\"E\"] \n",
    "print(my_letters[0])\n",
    "print(len(my_letters))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Slice of items: from the first item up to (not including) the second-to-last item\n",
    "print(my_letters[:-2]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'Comprehend' a list as its lowercase values:\n",
    "my_letters_lowercased = [letter.lower() for letter in my_letters]\n",
    "print(my_letters_lowercased) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Also: run this cell now. It's the usual imports of text mining libraries.\n",
    "\n",
    "import nltk\n",
    "import numpy\n",
    "import string\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"2\"></a>\n",
    "# 2. Loading a Text File: From Your Notebook or From a Website"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions & Objectives\n",
    "\n",
    "- How can I load a text file from my hard drive or a website?\n",
    "\n",
    "#### Key Points\n",
    "\n",
    "- To open and read a file on your computer, we use `open()`, `read()` and `close()`\n",
    "- To open and read a file from the internet, we use `urllib.request.urlopen()` and `.read().decode('utf-8')`\n",
    "- Once the file is opened, you can store its contents in a variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Broadly speaking there are two contexts in which we load a text file for analysis:\n",
    "\n",
    "- Local file:  you have your file on your [virtualized] computer or hard drive because you created or downloaded it earlier\n",
    "- Remote file: you access the file directly from some website, 'on the fly', processing it with your code but never really saving it as your own (e.g., for copyright or convenience reasons)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading a local file:\n",
    "\n",
    "First let's load some file from your 'hard drive' - because we are working inside of Noteable, it acts as your hard drive. There's a file you downloaded called `file_inaugural_speech_obama.txt` and it is in the same folder as this notebook, so we reference it with the path `./file_inaugural_speech_obama.txt` (the `./` means 'same folder as this notebook')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = \"./data/barack_obama_speeches/inaugural_speech.txt\"\n",
    "my_file = open(file_name) # open the file\n",
    "speech = my_file.read()   # read contents of the file and put them in a variable\n",
    "my_file.close()           # close the file\n",
    "\n",
    "# After that you have access to the file as text in the speech variable you created.\n",
    "print(\"number of characters:\", len(speech)) \n",
    "print(speech[:50])       # first 50 characters\n",
    "print(speech[-50:])      # last 50 characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading a remote (online) file:\n",
    "To read the same file from an online source (like from the White House website) we need to import the url-handling library `urllib`, but otherwise the process is very similar to reading a local file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib                           # you only have to do this once per notebook\n",
    "link = \"https://raw.githubusercontent.com/drpawelo/efi_text_mining_bootcamp/master/data/inaugural_speech_obama.txt\"\n",
    "my_file = urllib.request.urlopen(link)  # download the file (no need to open-close)\n",
    "speech = my_file.read().decode('utf-8') # read and decode content and save it\n",
    "\n",
    "# After that you have access to the file as text.\n",
    "print(len(speech))  # how long is it?\n",
    "print(speech[:50])  # first 50 characters\n",
    "print(speech[-50:]) # last 50 characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What's similar and different:\n",
    "\n",
    "Notice similarities and differences in both methods:\n",
    "\n",
    "**GET LIBRARIES (TOOLS):** On top of what Python already gives you, you can use other libraries of code for special tasks such as text analysis and data visualisation. You do this only once per notebook.\n",
    "\n",
    "- `import ...`\n",
    "\n",
    "**OPEN**: Both methods need a name and address of the file (file path or website link).\n",
    "\n",
    "- local:  `open(file_name)`\n",
    "- remote: `urllib.request.urlopen(link)`\n",
    "\n",
    "**READ**: With both methods, once you have access to the file, you need to READ the contents of it and put them in a string variable. Notice that remote files can come in various 'encodings' (ways to understand special characters and punctuation), so we usually specify the `UTF-8` (Unicode Transformation Format) encoding for plain English. Another common one is `latin1`.\n",
    "\n",
    "- local: `my_file.read()`\n",
    "- remote: `my_file.read().decode('utf-8')`\n",
    "\n",
    "**CLOSE**: Only with the local file do we need to close it once we've read it. This is so that another script or user can open it later. This works like files on a computer: they can each be opened only one instance at a time.\n",
    "\n",
    "- local: `my_file.close()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐛Minitask\n",
    "\n",
    "Write some Python code to open the following online file and display the characters between indices 42380 and 42869 in that file (don't peek at what's in the file). Do you recognise what play this text is from?\n",
    "\n",
    "http://www.gutenberg.org/files/1513/1513-0.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your answer here:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<details><summary style='color:blue'>CLICK HERE TO SEE THE THE ANSWER. BUT REALLY TRY TO DO IT YOURSELF FIRST!</summary>\n",
    "\n",
    "    ### BEGIN SOLUTION\n",
    "    link = \"http://www.gutenberg.org/files/1513/1513-0.txt\"\n",
    "    my_file = urllib.request.urlopen(link)\n",
    "    text = my_file.read().decode('utf-8')\n",
    "\n",
    "    print(text[42380:42869])\n",
    "    ### END SOLUTION\n",
    "    \n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### So now we have a long string...what's next?\n",
    "\n",
    "But as we can see it is not particularly useful to operate on **characters** as the main measure of length and to access parts of text. It would be more meaningful to ask for the first 10 words or last 10 words. Indeed, we might want to consider punctuation and symbols too.\n",
    "\n",
    "This is where tokens come in:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"3\"></a>\n",
    "# 3. Tokenising Strings: Splitting Them into Tokens (Words, etc.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions & Objectives\n",
    "\n",
    "- What is tokenisation?\n",
    "- How can a string of raw text be tokenised?\n",
    "\n",
    "#### Key Points\n",
    "\n",
    "- Tokenisation means to split a string into separate words and punctuation marks, to be able to, for example, count them.\n",
    "- Text can be tokenised using a tokeniser, e.g., the `punkt` tokeniser in NLTK."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to process text we need to break it down into tokens. As we explained at the start, a token is a letter, word, number, or punctuation mark which is contained in a string.\n",
    "\n",
    "To tokenise we first need to import the `word_tokenize` method from the `tokenize` package of NLTK, which allows us to tokenise text without writing the code ourselves.\n",
    "\n",
    "We will also download a specific tokeniser that NLTK uses as default. There are different ways of tokenising text and today we will use NLTK’s built-in `punkt` tokeniser by calling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell now (it's fine if you see some pink messages underneath it).\n",
    "\n",
    "from nltk.tokenize import word_tokenize\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's tokenise (split into tokens) the nursery rhyme \"Humpty Dumpty\".\n",
    "\n",
    "We will save the tokenised output in a list using the `humpty_tokens` variable so we can inspect it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "humpty_string = \"Humpty Dumpty sat on a wall, Humpty Dumpty had a great fall; All the king's horses and all the king's men couldn't put Humpty together again.\"\n",
    "humpty_tokens = word_tokenize(humpty_string)\n",
    "print(humpty_tokens) # print all tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's print just a few of them to have a closer look:\n",
    "print(humpty_tokens[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unifying and cleaning up the text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To further analyse the data, we'll first learn how to perform some clean-up tasks. \n",
    "\n",
    "As you can see in the above example, some of the words are uppercase and some are lowercase. But Python is case-sensitive, which means that 'hope' and 'Hope' are considered two completely different strings.\n",
    "\n",
    "For example, when searching for a word or counting the occurrences of a word, we most likely will want to consider both the lowercase and uppercase versions of the word (e.g., `company` and `Company` ). That's why, to simplify the analysis, we often normalise the data by making it all lowercase. This way, both of the above words would simply become `company`, making the text easier to comprehend.\n",
    "\n",
    "Since our list of tokens is a list of strings (words and punctuation) we can apply the `list comprehension loop` we learned about before to transform our list of mixed-case words into a list of lowercase words. \n",
    "\n",
    "As you might remember, a syntax for such loop is `[output_format for item in items ]` where:\n",
    "\n",
    "- `output_format` is some operation we perform on item, like `item.lower()` or `len(item)`\n",
    "- `items` is the list with all the elements we want to transform\n",
    "- `item` is a temporary name we give to each element of `items`, for the purposes of using that name inside of `output_format`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's modify above example, so that we only work with lowercased tokens of the nursery rhyme:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "humpty_string = \"Humpty Dumpty sat on a wall, Humpty Dumpty had a great fall; All the king's horses and all the king's men couldn't put Humpty together again.\"\n",
    "humpty_tokens = word_tokenize(humpty_string)\n",
    "lowercase_tokens = [token.lower() for token in humpty_tokens]\n",
    "print(lowercase_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🖇💬Buddy Discussion: What would be the coolest text dataset to analyse?\n",
    "\n",
    "#### Ask your buddy now if they reached the **BUDDY TASK**. Once you both did, complete this task:\n",
    "\n",
    "Come up with ONE EXAMPLE EACH of a text source that you would LOVE to have access to and analyse. Don't worry if it would be very hard (or impossible) to acquire, just imagine you have a magic wand. For example:\n",
    "\n",
    "- All the chats in Edinburgh taxis\n",
    "- 1000 most popular recipies for apple pie\n",
    "- Transcripts of all job interviews for academic jobs in UK this year\n",
    "\n",
    "Don't spend too much time on this (2 minutes maximum) but take note of your favourite idea."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐛Minitask\n",
    "\n",
    "Let's do our first, very simple piece of analysis. Do you think there were more mentions of 'we' or 'they' in the innaugural speech we looked at before?\n",
    "\n",
    "Let's try to re-use some pieces of code we wrote before and do our first very simple analysis:\n",
    "\n",
    "First without the lowercasing:\n",
    "\n",
    "- Copy-paste your code from before to load the speech of President Obama.\n",
    "- Use `word_tokenize()` on that variable to turn it into a list of tokens.\n",
    "- Count all occurances of a word 'we'. You can use the `a_list.count( a_word )` method like this:  `how_many_we = speech_tokens.count('we')`.\n",
    "- Print how many there were.\n",
    "- Do the same to count occurences of 'they'.\n",
    "\n",
    "How many times are these words used?\n",
    "\n",
    "- Now add the list comprehension after you tokenised the text into a list, changing list items into their lowercased equivalents. Do this after you tokenise the string, but before you do the counting.\n",
    "\n",
    "Now which word is more frequent?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<details><summary style='color:blue'>CLICK HERE TO SEE THE THE ANSWER. BUT REALLY TRY TO DO IT YOURSELF FIRST!</summary>\n",
    "    \n",
    "    ### BEGIN SOLUTION\n",
    "    file_name = \"./data/barack_obama_speeches/inaugural_speech.txt\"\n",
    "    my_file = open(file_name) # open the file\n",
    "    speech_text = my_file.read() # read content of it and put them in a variable\n",
    "    my_file.close() # close the file\n",
    "\n",
    "    speech_tokens = word_tokenize(speech_text)\n",
    "    speech_tokens = [word.lower() for word in speech_tokens]\n",
    "    print(speech_tokens.count('we'))\n",
    "    print(speech_tokens.count('they'))\n",
    "    ### END SOLUTION\n",
    "    \n",
    "</details>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🦋 Extra Task (optional): if you have finished everything else already:\n",
    "\n",
    "What other words could you look for? Do you think you could create a list of words, like `['hope', 'fear' ,'can', 'cannot']` and use a for loop to print counts of all of these words in the speech?\n",
    "\n",
    "You can try to illustrate a particular point using data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<details><summary style='color:blue'>CLICK HERE TO SEE THE THE ANSWER. BUT REALLY TRY TO DO IT YOURSELF FIRST!</summary>\n",
    "    \n",
    "    ### BEGIN SOLUTION\n",
    "    file_name = \"./data/barack_obama_speeches/inaugural_speech.txt\"\n",
    "    my_file = open(file_name) # open the file\n",
    "    speech_text = my_file.read() # read content of it and put them in a variable\n",
    "    my_file.close() # close the file\n",
    "\n",
    "    speech_tokens = word_tokenize(speech_text)\n",
    "    speech_tokens = [word.lower() for word in speech_tokens]\n",
    "\n",
    "    words_of_interest = ['hope', 'fear' ,'can', 'cannot']  \n",
    "    for word in words_of_interest:\n",
    "        print(word, speech_tokens.count(word))\n",
    "    ### END SOLUTION\n",
    "    \n",
    "</details>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}