Basic Python in 2h.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Fundamentals\n",
    "\n",
    "**Time: 2 hours**\n",
    "\n",
    "**You will learn to:**\n",
    "\n",
    "- Use variables and print their values\n",
    "- Create a list and dictionary and iterate/loop through them\n",
    "- Use 'if' conditions and select what code to run\n",
    "\n",
    "**Objectives:**\n",
    "\n",
    "- Learn Python fundamentals and syntax\n",
    "- Run and write Python code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to learn using a notebook:\n",
    "\n",
    "#### - Read through the cells with explanations\n",
    "#### - Run all the cells that contain code \n",
    "\n",
    "#### (To run a selected cell use the keyboard shortcut Shift+Enter or click the >Run button at the top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction to Python and Jupyter Notebooks\n",
    "\n",
    "Python is a programming language. We will use it to interact with text and analyse text documents.\n",
    "\n",
    "This introduction to Python is important for understanding all the steps that follow in the text mining lesson. If you are new to programming it may not be entirely clear why you need to learn Python first. Bear with us; you will soon understand why it is needed and useful, and it will help to speed things up later on.\n",
    "\n",
    "We will use Jupyter Notebooks (this file you are looking at) for learning. Notebooks are really useful because you can have your notes cells and your code cells in one place:\n",
    "\n",
    "- Notes (or Markdown) Cells: like what you are reading now, are cells of written text that explain concepts\n",
    "- Code Cells: like the `2+3` below, are parts of Python programming code that you can RUN (we'll show you how). This means that the code (like `2+3`) will be interpreted by Python (`oh, I think they are asking me to add two numbers, 2 and 3`) and executed (`that's 5, here you go`). When the code has run, Python will kindly return you the result, and we will learn later how to show it (print it) in nicer ways."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "2 + 3\n",
    "\n",
    "# Above is your first piece of Python code and this line is a comment. \n",
    "# Comments are notes you can leave inside of your code. \n",
    "# They start with a # and are ignored when Python code runs.\n",
    "\n",
    "# To 'RUN' this cell, select this cell and click '> Run' above \n",
    "# (or if you're into keyboard shortcuts, use Shift+Enter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the code above is **interpreted** (or **run**) you should see the result below the cell you have ran.\n",
    "\n",
    "Do you see it? Spoiler alert, it should be 5, because if you interpret the `2+3` operation, it will result in the number `5`. \n",
    "\n",
    "Notice that when you run a cell, the next cell is automatically highlighted.\n",
    "\n",
    "Run the cells below and see what happens (basically click \"Run\" a few times, or press Shift+Enter a few times).\n",
    "\n",
    "What do you expect to see?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numbers need no quotes around them. Python will use maths to add them.\n",
    "20+30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "100 + 123"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# But words have quotes, and Python will combine them to add them.\n",
    "\"hello\" + \"my\" + \"friend\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Oh no, the result above is too close together. We could try to add spaces, but it starts looking complicated.\n",
    "\"hello\" + \" \" + \"my\"  + \" \" + \"friend\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Printing things (showing them on the screen in a readable way)\n",
    "\n",
    "So far we were sort of cheating by \"returning\" the result of our code. Returning happens when Python reaches the end of a cell and isn't told what to do, so it just panics and returns the most recent thing it thought about. Returning is not the kindest way to show something on the screen. A nicer way is to kindly ask Python to \"print\" things.\n",
    "\n",
    "`print()` is Python's **function** to print (or show or display) something on the screen. \n",
    "\n",
    "Functions are sort of like skills that Python creators created for you. You can use them to achieve what you're after. We will learn a few of them in a minute.\n",
    "\n",
    "Like every function, print() has a:\n",
    "\n",
    "- **name** usually describing the action it'll do, e.g., `print`\n",
    "- **arguments** in parentheses, the things we want use for that action, e.g., `(\"Hello\")` or `(\"Hello\",\"my\",\"friend\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Hello!\")\n",
    "print(\"I can print!\")\n",
    "print(\"All the things\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You know how you can teach a dog to `fetch`? It would not make much sense to tell a dog to fetch but not tell it what object it needs to fetch. That's why you would normally show the dog a ball or a stick, and then shout `fetch` as you throw it. In progamming terms you would do something like `dog.fetch(\"stick\")` or `dog.fetch(\"ball\")`, but this will all start looking familiar soon (Note: the `.fetch()` syntax is a **method**, which is similar to a function except it is called on an object, which in this example is `dog`).\n",
    "\n",
    "Did you notice that the results print underneath the cell without the ugly **Out[123]:** on the left, like it did in the previous examples?  That's because we printed the text, not just 'returned the result'. Printing is a nicer and often more readable way to get Python to show something.\n",
    "\n",
    "Also notice that you can give many arguments to the `print()` function. What will happen is they will get separated by spaces, so they look better. It will help us to improve that complicated code from above (with loads of plus signs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Hello\",\"my\",\"friend\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask: Printing text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add some code in this cell (below this comment) and then RUN this cell:\n",
    "# write a line that will print your full name, \n",
    "# and another line that will print your favourite pizza toppings.\n",
    "# Do you see your words displayed on the screen? Yay! You are a Python programmer now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables and using = to assign them\n",
    "\n",
    "Each line of code is a world of its own. Once it's 'Run', it forgets everything. It might have printed something or returned something but nothing is **stored for later**, unless we use a digital way to store things, called a VARIABLE. \n",
    "\n",
    "A Variable is like a locker in which you can store things, and just as a locker has its number and its contents, each variable has: \n",
    "\n",
    "- a **name**: like a label on the locker, so we can find it later\n",
    "- **content**: something we put inside the locker to use later\n",
    "\n",
    "For example, we can put the string “text mining” into the box named “superpower”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "superpower = \"Text Mining\"\n",
    "# Run this cell now, but do not worry if you do not see anything returned or printed!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This part will be a bit confusing to some of you: the symbol `=` means something else in programming than what it means in maths (where it means 'equals').\n",
    "\n",
    "The equals symbol `=` in Python is called an 'assignment operator' because it assigns what's on the right of it to what's on the left. It works a bit like a left-pointing arrow **<---**\n",
    "\n",
    "You can imagine \n",
    "\n",
    "`superpower = \"Text Mining\"`\n",
    "\n",
    "as\n",
    "\n",
    "`superpower <--- \"Text Mining\"`.\n",
    "\n",
    "This code will take the text value on the right hand side (`\"Text Mining\"`) and store it for later (assign it) to a variable on the left hand side (`superpower`). Notice variable names have no quotes around them.\n",
    "\n",
    "So the code\n",
    "\n",
    "`contents_of_my_locker = \"shoes\"`\n",
    "\n",
    "is a bit like \n",
    "\n",
    "`contents_of_my_locker <--- \"shoes\"`,\n",
    "\n",
    "putting the word `\"shoes\"` into the variable `contents_of_my_locker`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# When you ran the previous code cell it did NOT return or print anything,\n",
    "# but it did something even more useful: \n",
    "# it remembered the words \"Text Mining\" forever in a variable called superpower.\n",
    "# Now in other cells you can use that variable:\n",
    "\n",
    "print(superpower)\n",
    "print(\"My new superpower is \"+superpower)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can create as many variables as you want, as long as their names are unique \n",
    "# and do not contain spaces (Python likes to use _ underscores for multi-word variable names)\n",
    "\n",
    "student_name = \"Nicola Minestrone\"\n",
    "course_title = \"Text Mining\"\n",
    "print(student_name, \"is taking the\", course_title, \"course\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the difference between using \"superpower\" as a word, and superpower as a variable name. Python does not like guessing what we mean, so we need to explicitly specify if we mean something as a name of a variable or text. In Python, the colour highlighting often can help you identify what is interpreted as what."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask: Using variables\n",
    "\n",
    "Try to explain in your own words what's happening in the code below, then change the name of the variable into `surname` and put your own surname there. You might see some errors, but do not fret."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "friend = \"Natalie\"\n",
    "\n",
    "print(\"hello \" + \"friend\")\n",
    "print(\"hello \" + friend)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Types of variables: String, Integer, Float"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lesson we will be concentrating mainly on strings, but this is only one of the types of things variables can hold onto:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "superpower = \"Text Mining\"  # String: holds text, has quotes \"\" or ''\n",
    "number = 42                 # Integer: holds whole numbers, no quotes\n",
    "pi_value = 3.1415           # Float: numbers with decimal places, in some context you might also see 'double'\n",
    "\n",
    "print(superpower, number, pi_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Text variables are called **strings** because they are chains (or lists or strings) of characters. You can imagine them as a necklace with a string of beads that have letters on them, like the ones that spell out your name.\n",
    "\n",
    "For example, the string `\"Edinburgh\"` is made of 9 letters: `\"E\", \"d\", \"i\", \"n\", \"b\", \"u\", \"r\", \"g\" and \"h\"`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why are types important? Mainly because Python tries to be 'smart' and guess what we want it to do, but it's not always very good at it.\n",
    "\n",
    "Consider the two lines of code below. Try to guess what will be printed BEFORE you run them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Can you guess what will be printed before you run this cell?\n",
    "\n",
    "print(123 + 123)\n",
    "print(\"123\" + \"123\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This happened because when we add two words, we sort of glue them together in order, but when we add two numbers we use maths.\n",
    "\n",
    "That was quite straightforward. Python knows how to add two numbers or two strings together. What about when the below scenario happens?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The meaning of life is \" + 42)\n",
    "\n",
    "# BRACE BRACE! This code is not correct and Python will freak out,\n",
    "# but don't worry. Run this cell and see what happens!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Errors are your best friends and they really do their best to explain:\n",
    "\n",
    "- Where something went wrong (green arrow): `---->`\n",
    "- What is wrong and how you can fix it (last line): `TypeError: can only concatenate str (not \"int\") to str`\n",
    "\n",
    "By the way: concatenate is a fancy word for adding two things together.\n",
    "\n",
    "The above line \"Errored out\" - instead of returning the expected output you got a nice explanation of what can and can't be done."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask: Combining two items \n",
    "\n",
    "Can you fix the above line of code so that it prints \"The meaning of life is 42\" and run it again? Go on, I believe in you! There are a few ways to achieve that. Try to use the fact it knows how to add two strings - how do you make something a string?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lists are what we use to store a collection of things:  \n",
    "# days = [\"Monday\", \"Tuesday\", \"Wednesday\"] \n",
    "\n",
    "(They're used for storing a collection of items in one variable.)\n",
    "\n",
    "Data can be grouped together in an ordered way using a list. Lists are very common data structures used in Python, for example to represent text when it is split into smaller units.\n",
    "\n",
    "We can represent a sentence as one long string:\n",
    "\n",
    "`sentence = \"Just think happy thoughts and you’ll smile.\"`\n",
    "\n",
    "Or we can store the same content in a List named `sentence` that contains all the words and punctuation of the sentence \"Just think happy thoughts and you’ll smile.\"\n",
    "\n",
    "`sentence = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']`\n",
    "\n",
    "Lists are created by typing comma-separated values inside square brackets, e.g., `[\"one\",\"two\",\"three\"]` or `[4,5,6]`\n",
    "\n",
    "You can print out all elements in the list at once using `print(your_list_variable)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "\n",
    "print(tokens) # Print the list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice some really creative (but correct) use of puctuation above. It is very important to get your quotes correct when working with strings (text) and lists (collections).\n",
    "\n",
    "Colours in your code editor will help you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask: Errors in a List\n",
    "\n",
    "The code below is not correct: there are at least three things wrong in it.  Can you fix it?\n",
    "\n",
    "Try running the cell below and reading the error. Errors are your friends.\n",
    "\n",
    "I have a game for you: run the cell, read the error and ONLY FIX WHAT the ERROR TELLS YOU TO FIX, one step at a time. Then run the cell again, see the new error and fix only that. It's very good practice and you will learn a lot!\n",
    "\n",
    "Hint: single quotes and double quotes are interchangable, so you can use 'hello' or \"hello\" (but you can't use 'hello\" or \"hello')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell, and only fix what error tells you. Resist the urge to fix other mistakes.\n",
    "# Notice a little mini arrow pointing up  ^  to the exact point something is wrong.\n",
    "\n",
    "reaction = [\"It\"s\", \"my\" \"absolutely\", favourite, \"song\", !]\n",
    "print(reaction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing items of a List"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The list holds an **ordered** sequence of elements (in this case words or punctuation) and each element can be accessed using its index (position in the list).\n",
    "\n",
    "It means that if you want to get the n'th item in a list (where n is the item's index), you need to say something like `list_name[n]`. \n",
    "\n",
    "For example: `tokens[3]` - using the name of the list, e.g., `tokens`, and the desired index in square brackets, e.g., `[3]` will return the item in tokens list stored at index (place) 3. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For example, to print first item in the below list...  \n",
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "\n",
    "# ...you would use:\n",
    "print(tokens[1]) # print the first element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WAIT WHAT? That should have printed `'Just'`\n",
    "\n",
    "Note: Python indices start with 0 instead of 1. So the first element in the list has index 0, and is accessed with `tokens[0]`.  `tokens[1]` actually returns the second item in the list (which is `'think'`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask: accessing items in a list using indices\n",
    "\n",
    "Fix the code below to actually print the first (not the second) item in the list.\n",
    "\n",
    "How would you access the third item, or the last item? Add lines of code that do that.\n",
    "\n",
    "How about accessing item at an index that does not exist, like 100 or -1? Try it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tokens[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing sub-sections of a List (some people call them slices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also get a slice (a section) of the list by specifying the start and end of your slice (note that the end is not inclusive).\n",
    "\n",
    "The syntax is `some_list_variable[ beginning_index : end_index ]`, so to get the second half of our `tokens` list we could use: `tokens[4:9]`. It will be a slice which starts with the item at index 4 (`'and'`) and ends with the item at index 9 (`'.'`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "\n",
    "print(tokens[4:9])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The first two elements of the list:\n",
    "print(tokens[0:2])\n",
    "\n",
    "# Fom third (at index 2) item up to the sixth (at index 5) item:\n",
    "print(tokens[2:6])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It takes a bit of time to get used to counting from 0 and remembering the end index of a slice is not included, e.g. `tokens[0:2]` is not ` ['Just', 'think', 'happy']` but ` ['Just', 'think']` because the item at end_index (`2`) is not included."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask:\n",
    "\n",
    "Write code below that will:\n",
    "\n",
    "- Print the first 5 items of the tokens list\n",
    "- Print the last 5 items of the tokens list\n",
    "- Print all items apart of the first one and the last one in the tokens list\n",
    "- Print just `['happy','thoughts']`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "# Write your answers here:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional ways to use indices: default value and counting from the end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you skip the number before or after the : it will assume 'all the way from the beginning' \n",
    "# or 'all the way to the end',\n",
    "# so these two lines do the same thing, but the second one uses the 'default value' of 'from \n",
    "# the begining':\n",
    "print(tokens[0:3])\n",
    "print(tokens[:3]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here the second one uses the 'default value' of 'to the end' :\n",
    "print(tokens[3:9])\n",
    "print(tokens[3:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can use the minus sign to count backwards from the end using negative numbers,\n",
    "# e.g., the second line will get the last 3 items starting with \"the third-to-last\" and going to the end:\n",
    "print(tokens[6:])\n",
    "print(tokens[-3:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To get all the words apart from the last two, we can use negative values as the end index:\n",
    "print(tokens[:7])\n",
    "print(tokens[:-2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask:\n",
    "\n",
    "Re-do the previous minitask using the new ways to use indices:\n",
    "\n",
    "- Print the first 5 items of the tokens list\n",
    "- Print the last 5 items of the tokens list\n",
    "- Print all items apart of the first one and the last one in the tokens list\n",
    "- Print just `['happy','thoughts']`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "# Write your answers here:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing things from a List"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can **delete** an item in the list by using the `pop(n)` function specifying the index of the element, like `some_list.pop( index_to_remove )`.\n",
    "\n",
    "In our example above `tokens.pop(2)` would remove the word `'happy'` from the list (beacuse it's at index `2`).\n",
    "\n",
    "If you do not specify the index, the last one gets removed and `tokens.pop()` removes the last item in the `tokens`, which would be a  `'.'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "print(tokens)\n",
    "\n",
    "tokens.pop(2)\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "print(tokens)\n",
    "\n",
    "tokens.pop()\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Notice what happens if you run pop() again\n",
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "print(tokens)\n",
    "\n",
    "tokens.pop()\n",
    "print(tokens)\n",
    "\n",
    "tokens.pop()\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you do not re-establish the tokens again in a cell, it will just take the value from the previous cell,\n",
    "# so here the value of tokens is the same as it was at the end of the previous cell.\n",
    "\n",
    "# What will happen when you run this cell again? And again?  Do it a few times. Notice the output.\n",
    "# (You'll need to re-select the cell with your mouse and then press Shift+Enter again.)\n",
    "tokens.pop()\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you notice that when we change the list it gets changed in all the cells? On some level your notebook is like a computer: If you delete a file, go for a cup of tea and come back... the file is still deleted.\n",
    "\n",
    "Values stored in variables persist across the times you run a cell and they are shared between all cells. That's why you can create a variable in one cell and then use it in another cell.\n",
    "\n",
    "To reset your list, you need to re-run the cell where you have assigned the list with `tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding Things to a List"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Items can be added to a list either by inserting them at a specific position (index) or by appending them to the end of the list. \n",
    "\n",
    "The syntax you would use is: `collection.insert( where, what )`\n",
    "\n",
    "E.g., to insert \"Hello\" at index 3 you'll use: `tokens.insert( 3, \"Hello\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: this would replace the ending punctuation '.' with the word 'widely', but that's not what we want to do\n",
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "print(tokens)\n",
    "\n",
    "tokens[8] = 'widely'\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We want to insert something at a place in a list and push along everything else \n",
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "print(tokens)\n",
    "\n",
    "tokens.insert(8,'widely')\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that as you add things to a list, you push everything else along, rather than replacing list items."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simply add something to the end of a list, you can use: `collection.append( what )`\n",
    "\n",
    "E.g., to insert \"Bye\" at the end you'll use: `tokens.append(\"Bye\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "print(tokens)\n",
    "\n",
    "tokens.append('[J.M. Barrie]')\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask:\n",
    "\n",
    "Can you adjust the `tokens` list so that you replace `'smile'` with `'fly'` ?\n",
    "First remove the item you don't want using `your_list.pop(n)`, and then add the new item using `your_list.append(item, n)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking the Length of a List"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you might want to check how long a list is. For this you can pass the list into Python's length function `len( something )` where `something` is the collection you want to measure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "print(len(sentence))\n",
    "\n",
    "sentence.pop()\n",
    "print(len(sentence))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FUN FACT - Strings are basically lists of characters\n",
    "\n",
    "Because each string is sort of like a list of characters, you can use the list functions on the strings as well:\n",
    "\n",
    "- To get a length of the word you can use `len(a_word)` like `len(\"thoughts\")`\n",
    "- To get the nth character of the word  you can use `a_word(n)` like `\"thoughts\"[3]`\n",
    "- To get a slice of the word you can use `a_word.slice(start, end)` like `\"thoughts\"[3:-2]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(\"thoughts\"))\n",
    "print(\"thoughts\"[3])\n",
    "print(\"thoughts\"[3:-2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Or even\n",
    "word = \"thoughts\"\n",
    "print(len(word))\n",
    "print(word[3])\n",
    "print(word[3:-2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 'For' Loop: used to iterate through the elements in a list, one at a time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's best if we start by seeing a loop with our own eyes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This will loop through each word in the list and print \"I say:\" before printing each word\n",
    "\n",
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "for word in tokens:\n",
    "    print(\"I say:\", word) # This will happen 9 times, once for each item in our list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The syntax of a for loop starts with `for a_thing in many_things:` where:\n",
    "\n",
    "- `many_things` is the collection you want to go through (in our case, a list)\n",
    "- `a_thing` is a temporary variable name that will represent the exact element of `many_things` that we are dealing with\n",
    "\n",
    "The code above has run 9 times: the first time `word` was equal to `\"Just\"`, the second time it was equal to `\"think\"`, and so on.\n",
    "\n",
    "When we're looping through (going through) the list, the `a_thing` variable keeps changing its value until we've looped through every value in the list.\n",
    "\n",
    "The indented (moved to the side) pieces of code after the `:` specify code to be performed with `a_thing`. This needs to be indented using a tab (or four spaces) so that Python knows the instructions that follow need to be executed for each element.\n",
    "\n",
    "When the indentation ends, Python will know to stop looping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "print(\"about to loop ->\")\n",
    "\n",
    "for word in tokens:\n",
    "    print(\"- about to say it\")\n",
    "    print(\"the word is\", word)   \n",
    "    print(\"- just said it\")\n",
    "\n",
    "print(\"-> done looping\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify the collection to loop through 'by hand' rather than using a variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for day in ['Monday', 'Tuesday', 'Wednesday']:\n",
    "    print(\"The day is\", day)   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask:\n",
    "\n",
    "- Can you print each item in `tokens` but surrounded by `*`, as in `*Just*`?\n",
    "- *Tricky:* Can you print only the first 5 elements of the sentence? (Remember the slice syntax above: `your_list[ start : end]`.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "# Write your answer below\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare two values with == and you'll get True or False\n",
    "\n",
    "Note: because equals sign `=` already has a meaning in programming (it's used for assigning right to left), Python has to use another symbol for **EQUALS**.\n",
    "\n",
    "In Python, we use two equals signs `==` to check if two things are equal to each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Hello\" == \"Bye\")\n",
    "print(\"Hello\" == \"Hello\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Or even\n",
    "word = \"Hello\"\n",
    "print(word == \"Bye\")\n",
    "print(word == \"Hello\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also combine conditions with logic operators: and, or, not"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "username = \"Jackie\"\n",
    "password = \"secret\"\n",
    "print(username == \"Nicola\" and password == \"secret\") # False, because 'and' needs both sides to be true\n",
    "print(username == \"Nicola\" or password == \"secret\")  # True, because 'or' is fine with just one side being true\n",
    "print(not username == \"Nicola\")                      # True, because 'username == \"Nicola\"' is false, and 'not false' is true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conditionals  if / elif / else : used to pick which lines of code to skip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the time, all of your lines of code are run from top to bottom. Sometimes, though, you might want to choose to run some lines and not others.  For example, you might want to only print words if they are longer than 5 letters.\n",
    "\n",
    "In Python we can create conditions (simple rules) that will direct the 'flow' of our code, a little bit like how semaphores direct train tracks, or blinking roadwork signs direct car traffic, or that person with shining lollipops directs airplanes at the landing strip.\n",
    "\n",
    "Conditionals either say: *come this way (true)* or *don't come this way (false)*.\n",
    "\n",
    "If the condition (a sort of test) passes, the indented code underneath it is executed. If the condition doesn't pass, then the indented code underneath it is skipped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "password = \"secret\"\n",
    "if password == \"secret\":\n",
    "    print(\"your password is too easy\") # This happens if password is \"secret\"\n",
    "else:\n",
    "    print(\"what a safe password!\")     # This happens otherwise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change the value of the variable password in the cell above to be something else and run it again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can go further, providing more tests (conditions) to pass, using `elif`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "password = \"secret\"\n",
    "if password == \"\":\n",
    "    print(\"your password cannot be empty\")                    # this would happen if password was \"\" (which it is not)\n",
    "elif password == \"secret\":\n",
    "    print(\"that's way too easy\")                              # this would happen if password is equal to \"secret\"\n",
    "elif len(password) < 8:\n",
    "    print(\"your password need to have at least 8 characters\") # this happens if len(password) < 8\n",
    "else:\n",
    "    print(\"what a safe password!\")                            # this happens otherwise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the conditions will be checked from top to bottom, so even though the length of 'secret' is shorter than 8, the condition above it will evaluate to true first, so it will 'steal the show'.\n",
    " \n",
    "Only one among the `if`, `elif`, and `else` conditionals in an if-else tree is ever executed in (Note: `elif` stands for 'else if')."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask: Combining everything we learned so far\n",
    "\n",
    "Read the code below and deduce what it does. Then, change the values (edit, add, or remove) of the variable `words` to reach every branch (condition) of this if-else tree.  What's printed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = [\"apples\", \"pears\", \"bananas\", \"kiwi\"] \n",
    "\n",
    "\n",
    "if words[1] == \"plum\":\n",
    "    print(\"there's a plum where I expected it\") \n",
    "elif len(words) > 4:\n",
    "    print(\"there are more than 4 fruits\") \n",
    "elif len(words[0]) > 7:\n",
    "    print(\"first fruit has a long name\")\n",
    "elif words[0] == words[1] or words[-2] == words[-1]:\n",
    "    print(\"first two or last two fruits are the same\") \n",
    "else:\n",
    "    print(\"I don't know what to say!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final two collection types: Tuples (unchangable lists), Dictionaries (key -> value pairs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tuples - like lists, but can't be changed \n",
    "# my_tuple = ('blue', 'green', 'red')\n",
    "\n",
    "A tuple is similar to a list as it is an ordered sequence of elements. The difference is that tuples can’t be changed once created. You recognise them by their `( )` parentheses. As with lists, you separate items in a tuple with a comma, as in `('blue', 'green', 'red')`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tuples are basically lists that can't be changed, so there are no pop() or insert() methods.\n",
    "colour_tuple = ('blue', 'green', 'red')\n",
    "print(colour_tuple[0]) \n",
    "print(colour_tuple[2]) \n",
    "print(len(colour_tuple)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dictionaries - like a list, but instead of indices pointing to values, we can create our own 'keys' that point to values\n",
    "# student = {\"firstname\":\"Nicola\", \"surname\":\"Paczkova\", \"course\":\"Politics\"}\n",
    "\n",
    "\n",
    "#### Back when all we knew were lists\n",
    "\n",
    "Remember, in a list we used **NUMBER KEYS (indices)**, so numbers 0, 1, 2, 3 and so on were pointing to values, e.g., `tokens[2]` points to the value at the third place in the list. The `2` is the INDEX, which is sort of like an address pointing to a particular place in a list.\n",
    "\n",
    "Lists are like a room full of lockers, and each locker has a number by which we can find them. In the example below, the first locker has number 0 and to find the owner I need to use that number (index) to find out who owns it. E.g., `lockers_list[0]` will return the value `\"Nicola's\"`.\n",
    "\n",
    "`lockers_list = [\"Nicola's\", \"Kat's\", \"Jill's\", \"Pat's\"]`\n",
    "\n",
    "As long as these lockers do not move too much (we can trust that Nicola's will always be first) this works fine.\n",
    "\n",
    "\n",
    "But things get more interesting if I want to store information about something more real-life, for example, about a student. If I want to know their firstname, surname and course, I could just store them in a list like `[\"Nicola\", \"Pachkova\", \"Politics\"]` but then we need to remember that the first item is firstname, the second, surname, and the third, course, which can lead to all sorts of trouble.\n",
    "\n",
    "#### Dictionaries: when order does not matter or is likely to change\n",
    "\n",
    "In a dictionary we have **WORD KEYS** pointing to values, e.g., `\"firstname\"` will point to `\"Nicola\"`, `\"surname\"` will point to `\"Paczkova\"` and `\"course\"` will point to a value `\"Politics\"`.  `student[word_key]` points to the value at the place in the dictionary specified by whatever you put in as the `word_key`.\n",
    "\n",
    "`student = {\"firstname\":\"Nicola\", \"surname\":\"Paczkova\", \"course\":\"Politics\"}`\n",
    "\n",
    "We use curly braces `{ }` to define a Dictionary, a colon `:` to separate each key from its value, and a comma `,` to separate key-value pairs from each other.\n",
    "\n",
    "If we want to get that student's firstname we would use the **KEY** (word) the same way as we would have used the **INDEX** (number) in lists:\n",
    "\n",
    "`student['firstname']`\n",
    "\n",
    "Keys are nicer than indices, because we can make them more meaningful than just 0,1,2,3, etc. Our code becomes more readable and easier to maintain.\n",
    "\n",
    "#### Dictionaries are sort of like a bunch of variables in a wrapper\n",
    "\n",
    "Each key acts as a reference (or a name) for a single value in the dictionary, almost like a bunch of variables bundled up into a mega-variable.\n",
    "\n",
    "Instead of:\n",
    "\n",
    "`firstname = \"Nicola\"\n",
    "surname = \"Paczkova\"\n",
    "course = \"Politics\"`\n",
    "\n",
    "we combine these values and variables into one meaningful unit and give it a name:\n",
    "\n",
    "`student = {\"firstname\":\"Nicola\", \"surname\":\"Paczkova\", \"course\":\"Politics\"}`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "student = {\"firstname\":\"Nicola\", \"surname\":\"Paczkova\", \"course\":\"Politics\"}\n",
    "\n",
    "print(student)           # Print the whole dictionary\n",
    "print(student['course']) # Print just the value we stored at the 'course' key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will very often see dictionaries when analysing text, usually with keys as individual words and values as integers (e.g., counts of how many times the words appear in a text). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_frequencies = {'Mary':5,  'lamb':5 , 'the': 5, 'a': 2, 'had':1,  'little':1}\n",
    "\n",
    "print(word_frequencies['Mary']) # Prints the value of the key 'Mary'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask\n",
    "\n",
    "- Given the dictionary below, which is a result of analysing a text document, print the number of verbs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "types_of_speech = {'verbs':45, 'nouns':24, 'adjectives':33}\n",
    "# type your answer here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Looping through a dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also request all the keys, all the values, or all the key-value pairs as tuples from a dictionary. This will come in handy later: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "student = {\"firstname\":\"Nicola\", \"surname\":\"Paczkova\", \"course\":\"Politics\"}\n",
    "print(student.keys())   # Method for getting a list of keys\n",
    "print(student.values()) # Method for getting a list of values\n",
    "print(student.items())  # Method for getting a list of (key, value) tuples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is useful when we want to loop through items in a dictionary. Unlike in a list, instead of having only one `value` for each item in a list, we will have a `key` and a `value` for each item in a dictionary.\n",
    "\n",
    "To make sure that our dictionary is separated into keys and values when looping, we need to use `my_dictionary.items()`.\n",
    "\n",
    "```\n",
    "for key, value in my_dictionary.items():\n",
    "    # you can do something here with key and value, like\n",
    "    print(key, value) \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "types_of_speech = {'verbs':45, 'nouns':24, 'adjectives':33, 'pronouns': 12}\n",
    "\n",
    "for type_of_speach, count in types_of_speech.items(): # .items() splits dictionary into (key, value) pairs\n",
    "    print(count, \"is the number of\", type_of_speach)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask\n",
    "\n",
    "How would you change the code from the above example to only print information for types of speech where their count was larger than 30? You are given a partial answer and you need to fill in the gaps marked with `?????`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "types_of_speech = {'verbs':45, 'nouns':24, 'adjectives':33, 'pronouns': 12}\n",
    "\n",
    "for type_of_speach, count in types_of_speech.items(): # .items() splits our dictionary into (key, value) pairs\n",
    "    if ????? > 30 :\n",
    "        print(count, \"is the number of\", type_of_speach)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that I am giving the key and value variables meaningful names, `type_of_speach` and `count`, rather than just writing `key` and `value`, so that my code is easier to read."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# List Comprehension: another very useful loop  (that we will use a lot)\n",
    "\n",
    "# lowercase_tokens = [ word.lower() for word in tokens ]\n",
    "\n",
    "### Modify a list on the fly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use this simplified loop syntax when what you want is to take a list of items and change each item in that list into something else. \n",
    "\n",
    "Think about it like a conveyor belt: things go in on one side, and slightly changed things come out on the other side.\n",
    "\n",
    "The syntax for this is:\n",
    "\n",
    "\n",
    "`result = [ my_output for one_item in all_items]`\n",
    "\n",
    "For readability you can add an extra new line (Python basically ignores new lines) and write it like this:\n",
    "\n",
    "\n",
    "`result = [  my_output\n",
    "            for one_item in all_items]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: For each word in `tokens`, represent it as a lowercased version itself using `word.lower()`, e.g., change `Just` into `just`, `Think` into `think`, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: some_word.lower() turns that word into its lowercase equivalent\n",
    "\n",
    "tokens = ['Just', 'think', 'HAPPY','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "\n",
    "lower_case_words = [ word.lower() \n",
    "                     for word in tokens ]\n",
    "\n",
    "print(lower_case_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: For each word in `tokens`, represent it with its length, e.g., change `Just` into `4`, `think` into `5`, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "\n",
    "lengths_of_words = [ len(word) \n",
    "                     for word in tokens ]\n",
    "\n",
    "print(lengths_of_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we are taking a list of some things and returning a list of some other things, but both lists have the same length."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further filtering a list to keep only some of its elements:\n",
    "\n",
    "Optionally, you can add a third line with a condition that needs to be true for the item to be kept in the final result.\n",
    "\n",
    "The syntax becomes: \n",
    "\n",
    "\n",
    "```\n",
    "result = [  output\n",
    "            for one_item in all_items\n",
    "            if condition\n",
    "         ]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "\n",
    "lengths_of_long_words = [ len(word)          # Turn a word into its length\n",
    "                          for word in tokens # For each word in tokens list\n",
    "                          if len(word) > 4   # But only keep those where the word's length was over 4\n",
    "                        ]\n",
    "\n",
    "print(lengths_of_long_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even use the words as they came for output, not changing them at all, and still use the filtering part of our code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "\n",
    "long_words = [word # If you just want to filter words but the output word should be the same as the input word\n",
    "              for word in sentence\n",
    "              if len(word) > 4]\n",
    "\n",
    "print(long_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: Only keep words starting with the letter 't'. Since strings are, deep inside, just lists of characters, we can get the first letter of our words by asking for the first item in that string as we do with lists: `word[0]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = ['Just', 'think', 'happy','thoughts', 'and', 'you', '’ll', 'smile', '.']\n",
    "\n",
    "words_starting_with_t = [ word \n",
    "              for word in sentence\n",
    "              if word[0] == \"t\" ]\n",
    "\n",
    "print(words_starting_with_t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minitask\n",
    "\n",
    "- Given a list of names, create another list with just these names but in lowercase (you can use `your_word.lower()`)\n",
    "- Now filter this list to only include those names that are longer than 5 letters (you can use `len(word)` and a `<`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = ['Matilda', 'Jess', 'Pat', 'Yu', 'Prianka', 'Hermenegilda', 'Makiko']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# For the Curious: combining it all together\n",
    "\n",
    "Let's write a piece of code that will take the counts of words in the \"Mary had a little lamb\" song and:\n",
    "- Print the number of times each word occurs\n",
    "- Print the number of times each word occurs only for words that occur more often than 4 times\n",
    "\n",
    "Then, we'll change the code to:\n",
    "- Print words that occur more often than 4 times and do NOT start with 't'\n",
    "\n",
    "Then, we'll change the code again to:\n",
    "- Store these long, popular words in a separate list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_counts = {'mary': 5, 'had': 1, 'a': 2, 'little': 1, 'lamb': 5, \n",
    "               'whose': 1, 'fleece': 1, 'was': 3, 'white': 1, 'as': 1, \n",
    "               'snow': 1, 'and': 4, 'everywhere': 1, 'that': 1, 'went': 1,\n",
    "               'the': 8, 'sure': 1, 'to': 3, 'go': 1, 'it': 4, 'followed': 1,\n",
    "               'her': 1, 'school': 2, 'one': 1, 'day': 1, 'which': 1,\n",
    "               'against': 1, 'rules': 1, 'made': 1, 'children': 2, 'laugh': 1,\n",
    "               'play': 1, 'see': 1, 'at': 1, 'so': 2, 'teacher': 2, 'turned': 1,\n",
    "               'out': 1, 'but': 1, 'still': 1, 'lingered': 1, 'near': 1, \n",
    "               'waited': 1, 'patiently': 1, 'about': 1, 'till': 1, 'did': 2,\n",
    "               'appear': 1, 'why': 2, 'does': 1, 'love': 1, 'eager': 1, 'cry': 1,\n",
    "               'loves': 1, 'you': 1, 'know': 1, 'reply': 1}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To only print words that occur more often than 4 times\n",
    "for word, count in word_counts.items():\n",
    "    if count > 4:\n",
    "        print(word, \"appears\", count, \"times\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only print words that occur more often than 4 times and do not start with 't'\n",
    "# Note: you can combine conditions with logic operators: and, or, not\n",
    "for word, count in word_counts.items():\n",
    "    if count > 4 and not word[0] == 't':\n",
    "        print(word, \"appears\", count, \"times\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Store these long, popular words in a separate list\n",
    "long_popular_words = [] # Notice: I am creating a list but putting nothing in it, yet\n",
    "for word, count in word_counts.items():\n",
    "    if count > 4 and not word[0] == 't':\n",
    "        long_popular_words.append(word)\n",
    "\n",
    "print(long_popular_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Final Note\n",
    "\n",
    "We will download some files and libraries on the first day. To speed up this process, run the below cell with some imports and downloads, so that you do not have to wait for it to happen on the first day. Thanks!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell now. It's the usual imports of text mining libraries.\n",
    "\n",
    "import nltk\n",
    "import numpy\n",
    "import string\n",
    "import matplotlib.pyplot as plt\n",
    "from nltk.tokenize import word_tokenize\n",
    "nltk.download('punkt')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}