scraping_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.7"
    },
    "colab": {
      "name": "scraping_2.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/drive/1p2qotYuC1VE6sffZWRKoQqdlrgbgLk8v?usp=sharing\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KEC19MCLrBKB",
        "colab_type": "text"
      },
      "source": [
        "# Extracting Web Data via Scraping and Automation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xCnr0YjArBKG",
        "colab_type": "text"
      },
      "source": [
        "### Part 2: But how do I extract the data I want?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UItnRVsprBKH",
        "colab_type": "text"
      },
      "source": [
        "James Bowden, 07/2020. This tutorial is the second of a two part series on how to scrape web data. The first can be found here (https://colab.research.google.com/drive/1KH18f5y9rbvEga6aDdUV907E-rxw453D?usp=sharing). This series was written as a part of Caltech's CS42 course (Computer Science Education)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Y-jMjkarBKI",
        "colab_type": "text"
      },
      "source": [
        "#### What next?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "luQmpIHrrBKJ",
        "colab_type": "text"
      },
      "source": [
        "In the first part, we learned how to find web data and download it! Now that we've gotten our data, we need to figure out how to extract just the information we want. We do this by **parsing** the data. Note that we will want to be able to parse HTML data as well as other kinds of data. Data comes in all shapes and sizes and is often *not* in the format you'd like it to be in. In order to be able to use all sorts of data, we need to understand how to parse anything. Once we know how to get the data we want, we can create functions to do it automatically and package it into a nice command tool for ease of use."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "szm6m2ylrBKL",
        "colab_type": "text"
      },
      "source": [
        "#### What is parsing, and how do I do it?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HeLw6ioGrBKL",
        "colab_type": "text"
      },
      "source": [
        "**Parsing** is a technique by which we separate out data we want based on patterns. The basic idea is that if we can identify the data we want with a **unique pattern or identifier**, then we can separate it out from everything else by searching for this particular pattern and only taking things that come after it (or before it, or are it, etc.). \n",
        "\n",
        "Parsing is usually done on strings of characters, which HTML code happens to be. Most data is stored as a sequence of characters, and then separated out via parsing. One common example is text files, which consist of characters with newlines (denoted by '\\n') throughout them that signal to the program when to start a newline. Another example is .csv files, which consist of values separated by commas (and newline characters) so that a program like Excel can find the commas and put each separate value into its own cell. Let's try writing our own basic function to parse a string."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ofDEtBdrBKN",
        "colab_type": "text"
      },
      "source": [
        "As an example, let's take a string consisting of a few sentences. I'd like to identify all of the things that the writer did. Now obviously, the English language is very complicated and there are many ways to express actions, but for this example we'll just count verbs that come immediately after 'I', such as 'today I biked to school', in which case 'biked' would be the desired verb. Note that when used in this context, 'I' is always followed by a space, so we should parse for 'I '. \n",
        "\n",
        "The first thing we need to do is just find all of the instances of 'I ', and denote their location. We'll write a simple function that does just that: takes in a string to search and a string to search for, and returns a list of their indices. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5fPX7s6_rBKO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def find(subject, search):\n",
        "    '''Given a subject string to search and another string to search for, returns a list of their starting indices.'''\n",
        "    indices = []\n",
        "    # we'll need the length of the search string to know how many characters to compare against and to skip once we find it\n",
        "    length = len(search)\n",
        "    ind = 0\n",
        "    # want to search from the start of the subject until the last possible match, which is found \n",
        "    # length characters before the end (has to be space for the search string to fit!)\n",
        "    for x in range(0, len(subject) - length):\n",
        "        # 'slice' the string to get only the first length characters.\n",
        "        s_compare = subject[ind:(ind + length)]\n",
        "        # if s_compare equals search, record the index and skip to the end of the search string!\n",
        "        if s_compare == search:\n",
        "            indices.append(ind)\n",
        "            ind += length\n",
        "        # if not, move forward one character in the subject string\n",
        "        else:\n",
        "            ind += 1\n",
        "    return indices"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "78edd_MBrBKW",
        "colab_type": "code",
        "colab": {},
        "outputId": "907c61af-5ab7-4e31-9bdb-4bcd300058f8"
      },
      "source": [
        "# test case\n",
        "paragraph = 'My name is James. Today I ate pizza for lunch. After that, I worked on some homework. I called my friends to see how they were doing. Then I napped for a few hours.'\n",
        "search = 'I '\n",
        "# call our function:\n",
        "indices = find(paragraph, search)\n",
        "print(indices)\n",
        "# make sure there is an 'I' at these indices\n",
        "for index in indices:\n",
        "    print(paragraph[index:index + len(search)])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[24, 59, 86, 139]\n",
            "I \n",
            "I \n",
            "I \n",
            "I \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ACiq7Q44rBKc",
        "colab_type": "text"
      },
      "source": [
        "We see that our function is indeed finding all of the occurrences of 'I '. Great! Now let's write a function that can take the next word after 'I'. We'll define a word as ending in a space or a period, though in reality there are a bunch of other punctuations symbols to check for. It's a bit more convenient to just call the find() method we wrote from within extract() since we always need to find the indices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NEwekDY7rBKd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def extract(subject, search):\n",
        "    '''Returns list of words in subject following the search string.'''\n",
        "    # get indices\n",
        "    indices = find(subject, search)\n",
        "    words = []\n",
        "    for index in indices:\n",
        "        # skip over the search string since we don't actually want that\n",
        "        start = index + len(search)\n",
        "        # set end index at end of subject string by default\n",
        "        end = len(subject)\n",
        "        # from start to end, check each character\n",
        "        for x in range(start, end):\n",
        "            # if we find a space or period, we know that our word ends here! exit the for loop.\n",
        "            if subject[x] == ' ' or subject[x] == '.':\n",
        "                end = x\n",
        "                break\n",
        "        # get next word by slicing from start to end\n",
        "        next_word = subject[start:end]\n",
        "        words.append(next_word)\n",
        "    return words"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ykfqWnsQrBKi",
        "colab_type": "code",
        "colab": {},
        "outputId": "7530ee01-4a7f-4a54-fdd5-48fe8ef1d9ea"
      },
      "source": [
        "# test on same case again:\n",
        "verbs = extract(paragraph, search)\n",
        "print(verbs)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['ate', 'worked', 'called', 'napped']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1jSNAvQjrBKm",
        "colab_type": "text"
      },
      "source": [
        "And now we have the verbs that follow 'I'. I could use this to collect all of the things somebody has done from their diary or lab notebook automatically, which is...very useful!? Though this example is basic, it's good to understand what exactly is going on when we parse strings. If we wanted to parse based on more rules (for example, accounting for newlines, or taking the verbs after pronouns), it wouldn't be difficult to add in a few more conditions or parse multiple times.\n",
        "\n",
        "There are functions built into Python that you can use to do these things, but they can't cover all cases. We'll see this in just a moment when we try to parse some of our data files. A few of these standard functions are shown below to illustrate their use, but you can Google them as needed. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "uARLmZmWrBKn",
        "colab_type": "code",
        "colab": {},
        "outputId": "17ff3856-79e2-4fb6-e9bf-8de7517137ad"
      },
      "source": [
        "paragraph = '''My name is James. Today I ate pizza for lunch. After that, I worked on some homework. I called my friends to see how they were doing. Then I napped for a few hours.'''\n",
        "search = 'I '\n",
        "# builtin find method gets 1st occurrence\n",
        "first_index = paragraph.find(search)\n",
        "print(first_index)\n",
        "print(paragraph[first_index:first_index + len(search)])\n",
        "# builtin split method--very useful for extracting certain parts of a string\n",
        "# here we split the paragraph by spaces:\n",
        "words = paragraph.split(' ')\n",
        "print('Split by \\' \\':')\n",
        "print(words)\n",
        "# or by periods:\n",
        "words = paragraph.split('.')\n",
        "print('Split by \\'.\\':')\n",
        "print(words)\n",
        "# or by 'I 's, which we could use by just taking the first word of each list entry except the first.\n",
        "words = paragraph.split('I ')\n",
        "print('Split by \\'I \\':')\n",
        "print(words)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "24\n",
            "I \n",
            "Split by ' ':\n",
            "['My', 'name', 'is', 'James.', 'Today', 'I', 'ate', 'pizza', 'for', 'lunch.', 'After', 'that,', 'I', 'worked', 'on', 'some', 'homework.', 'I', 'called', 'my', 'friends', 'to', 'see', 'how', 'they', 'were', 'doing.', 'Then', 'I', 'napped', 'for', 'a', 'few', 'hours.']\n",
            "Split by '.':\n",
            "['My name is James', ' Today I ate pizza for lunch', ' After that, I worked on some homework', ' I called my friends to see how they were doing', ' Then I napped for a few hours', '']\n",
            "Split by 'I ':\n",
            "['My name is James. Today ', 'ate pizza for lunch. After that, ', 'worked on some homework. ', 'called my friends to see how they were doing. Then ', 'napped for a few hours.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WIX2M9ZjrBKs",
        "colab_type": "text"
      },
      "source": [
        "#### Parsing HTML with BeautifulSoup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XGYIak8OrBKt",
        "colab_type": "text"
      },
      "source": [
        "Back to our example from last time: we have a URL and we've downloaded the HTML code from the server. It's now stored in our variable, and we need to parse it. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pxgylltgrBKu",
        "colab_type": "code",
        "colab": {},
        "outputId": "c8e45129-0d0f-4164-a5af-209e86e2884c"
      },
      "source": [
        "import requests\n",
        "\n",
        "# get HTML from url\n",
        "url = 'https://www.ncbi.nlm.nih.gov/assembly/GCF_000005845.2'\n",
        "response = requests.get(url)\n",
        "print('Preview:')\n",
        "print(response.text[:500])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Preview:\n",
            "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
            "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n",
            "<html xmlns=\"http://www.w3.org/1999/xhtml\" lang=\"en\" xml:lang=\"en\">\n",
            "    <head xmlns:xi=\"http://www.w3.org/2001/XInclude\"><meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n",
            "    <!-- meta -->\n",
            "    <meta name=\"robots\" content=\"index,nofollow,noarchive\" />\n",
            "<meta name=\"ncbi_app\" content=\"entrez\" /><meta name=\"ncbi_db\" conten\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CqHT7r8QrBKz",
        "colab_type": "text"
      },
      "source": [
        "In the first tutorial, we inspected the NCBI webpage of an *E. coli* strain (https://www.ncbi.nlm.nih.gov/assembly/GCF_000008865.2) to figure out where the data we wanted was stored in the HTML code. We want to extract the 'Total sequence length', and then extract the url for the 'FTP directory for RefSeq assembly', which contains several links to downloads. We can request this url the same way we did the first, parse its HTML code, and then download the data for coding genes to use locally.\n",
        "\n",
        "The first step is parsing the HTML code. From inspecting, we know that the total sequence length is stored here, in a table:\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1Nx-uR7wSxAxFFqP7aUGYFy8ectC-Ts8B)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OeuV2LiPrBK0",
        "colab_type": "text"
      },
      "source": [
        "While we could parse this ourselves, Python has a package that will do a lot of it for us! The package is called BeautifulSoup and you should install it now if you don't already have it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fgjVeJm3rBK1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# installation command for miniconda/anaconda environment\n",
        "conda install -c anaconda beautifulsoup4"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jTMW8RNHrBK5",
        "colab_type": "text"
      },
      "source": [
        "BeautifulSoup simplifies the parsing of HTML code by dealing with a lot of the formatting internally and picking out the tags and attributes that we want. We first import the package and initialize BeautifulSoup with an HTML parser on the content of the HTML code we've downloaded from the web server."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ak8f0ffArBK6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from bs4 import BeautifulSoup\n",
        "\n",
        "# parse HTML and store in variable\n",
        "soup = BeautifulSoup(response.content, 'html.parser')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ACK_ob-grBK_",
        "colab_type": "text"
      },
      "source": [
        "We now have a variable named soup that contains all of the HTML data in a format that can be easily parsed by BeautifulSoup. We can use the find() method to search for tables. Our unique identifier here is going to be the table's summary attribute, which equals 'Global statistics'."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "seG-hhDqrBLA",
        "colab_type": "code",
        "colab": {},
        "outputId": "e2392bb3-1487-4443-f459-b9ed00e249ae"
      },
      "source": [
        "# parse HTML for 'td' tag\n",
        "table = soup.find('table', {'summary':'Global statistics'})\n",
        "print(table)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<table class=\"margin_t0 jig-ncbigrid\" summary=\"Global statistics\"><tbody><tr><td>Total sequence length</td><td class=\"align_r\">4,641,652</td></tr><tr><td>Total ungapped length</td><td class=\"align_r\">4,641,652</td></tr><tr><td>Total number of chromosomes and plasmids</td><td class=\"align_r\">1</td></tr></tbody></table>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "In92GPb7rBLH",
        "colab_type": "text"
      },
      "source": [
        "And now that we have the table, we can pull out the total sequence length pretty easily by getting all of the table entries (denoted by 'td') and then taking the one that is after 'Total sequence length'."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iuq1tmQYrBLJ",
        "colab_type": "code",
        "colab": {},
        "outputId": "7f45bfb1-276f-48f6-83c3-a08b9e657db0"
      },
      "source": [
        "# get all entries in the table\n",
        "tds = table.find_all('td')\n",
        "print(tds)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[<td>Total sequence length</td>, <td class=\"align_r\">4,641,652</td>, <td>Total ungapped length</td>, <td class=\"align_r\">4,641,652</td>, <td>Total number of chromosomes and plasmids</td>, <td class=\"align_r\">1</td>]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DsdiAUWBrBLQ",
        "colab_type": "code",
        "colab": {},
        "outputId": "ccb1d5c7-f4ec-4a22-af13-7325bf5a21a0"
      },
      "source": [
        "# extract the length after total sequence length string has been found\n",
        "ind = 0\n",
        "for entry in tds:\n",
        "    if 'Total sequence length' == entry.text:\n",
        "        # get the next entry!\n",
        "        length = tds[ind + 1].text\n",
        "        break\n",
        "print(length)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "4,641,652\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b4hQuMqNrBLV",
        "colab_type": "text"
      },
      "source": [
        "Just like that, we've got the length! Now we can apply a similar process to get the hyperlink for the FTP directory by parsing for a distinct part of the displayed link name ('FTP directory for RefSeq assembly') in all of the hyperlinks and then extracting the url once we've found it with link['href']."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "tMWiYeDVrBLW",
        "colab_type": "code",
        "colab": {},
        "outputId": "966ae110-3d9f-4d19-ac97-ad5c4643c9c3"
      },
      "source": [
        "links = soup.find_all('a')\n",
        "for link in links:\n",
        "    if 'RefSeq assembly' in link.text:\n",
        "        # extract url\n",
        "        ftp_url = link['href']\n",
        "print(ftp_url)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/005/845/GCF_000005845.2_ASM584v2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "01Vyd9zzrBLd",
        "colab_type": "text"
      },
      "source": [
        "#### Getting FTP via urllib"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8bYug8BkrBLe",
        "colab_type": "text"
      },
      "source": [
        "We'll do this process one more time to get the link to the coding genes file, now requesting the FTP url. This is a good real world example of the fact that data doesn't always come in the format that we want it in. As a result, we'll have to try some other packages and parsing in order to extract what we want.\n",
        "\n",
        "The requests package we've been using is good for almost every site you'll want to go to, but unfortunately doesn't support FTP. We're going to use the urllib package in this case, which is a bit more complicated but does essentially the same thing as requests. Situations like this come up often when web scraping--just search Google or Stack Overflow for the problem you're having, try different solutions, and make sure to read the documentation for packages you're trying to use if you keep getting errors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2nMfQFferBLg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# installation command for miniconda/anaconda environment\n",
        "conda install -c ulmo urllib3"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tglc7PUorBLl",
        "colab_type": "code",
        "colab": {},
        "outputId": "f97f8716-b890-402f-89ab-6566dbe909db"
      },
      "source": [
        "import urllib.request\n",
        "\n",
        "# make a request object\n",
        "ftp = urllib.request.Request(ftp_url)\n",
        "# open, read, and decode the object\n",
        "with urllib.request.urlopen(ftp) as f:\n",
        "    ftp_html = f.read().decode()\n",
        "print(ftp_html)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "-r--r--r--   1 ftp      anonymous     1208 Feb 11 20:00 GCF_000005845.2_ASM584v2_assembly_report.txt\r\n",
            "-r--r--r--   1 ftp      anonymous     3746 Feb 11 20:00 GCF_000005845.2_ASM584v2_assembly_stats.txt\r\n",
            "-r--r--r--   1 ftp      anonymous  1504909 Oct 14  2018 GCF_000005845.2_ASM584v2_cds_from_genomic.fna.gz\r\n",
            "-r--r--r--   1 ftp      anonymous      244 Oct 14  2018 GCF_000005845.2_ASM584v2_feature_count.txt.gz\r\n",
            "-r--r--r--   1 ftp      anonymous   230172 Oct 14  2018 GCF_000005845.2_ASM584v2_feature_table.txt.gz\r\n",
            "-r--r--r--   1 ftp      anonymous  1379902 Oct 31  2014 GCF_000005845.2_ASM584v2_genomic.fna.gz\r\n",
            "-r--r--r--   1 ftp      anonymous  3462090 Mar  2  2019 GCF_000005845.2_ASM584v2_genomic.gbff.gz\r\n",
            "-r--r--r--   1 ftp      anonymous   465467 Feb 11 20:00 GCF_000005845.2_ASM584v2_genomic.gff.gz\r\n",
            "-r--r--r--   1 ftp      anonymous   483693 Feb 11 20:00 GCF_000005845.2_ASM584v2_genomic.gtf.gz\r\n",
            "-r--r--r--   1 ftp      anonymous   898290 Oct 14  2018 GCF_000005845.2_ASM584v2_protein.faa.gz\r\n",
            "-r--r--r--   1 ftp      anonymous  3674701 Mar  2  2019 GCF_000005845.2_ASM584v2_protein.gpff.gz\r\n",
            "-r--r--r--   1 ftp      anonymous    15100 Oct 14  2018 GCF_000005845.2_ASM584v2_rna_from_genomic.fna.gz\r\n",
            "-r--r--r--   1 ftp      anonymous  1082603 Oct 14  2018 GCF_000005845.2_ASM584v2_translated_cds.faa.gz\r\n",
            "lr--r--r--   1 ftp      anonymous       25 Sep 20  2016 README.txt -> ../../../../../README.txt\r\n",
            "-r--r--r--   1 ftp      anonymous      410 Feb 11 20:00 annotation_hashes.txt\r\n",
            "-r--r--r--   1 ftp      anonymous       14 Jul 10 04:51 assembly_status.txt\r\n",
            "-r--r--r--   1 ftp      anonymous     1094 Feb 11 20:00 md5checksums.txt\r\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vtumFRDjrBLq",
        "colab_type": "text"
      },
      "source": [
        "BeautifulSoup is great for parsing HTML data! But as you can see, this isn't in HTML format--instead, it looks like directory information was just printed out. Because this isn't standard HTML data, we'll have to parse it a bit differently, which is why it's important to understand how to parse strings in general.\n",
        "\n",
        "It appears that this is all one long string, so we can just separate it by spaces. However, we can see that here there are multiple spaces to filter out in between different items. We could do this ourselves with a while loop, but we'll use Python's built in re (regular expressions) package because it's easier. Regular expressions are very useful, but also quite complicated and could be a whole separate tutorial. I'd recommend learning how to use them whenever you have a chance though, as they make parsing *much* simpler!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t3d4fBelrBLr",
        "colab_type": "code",
        "colab": {},
        "outputId": "3448680c-a22d-4711-b129-ed9416007cf4"
      },
      "source": [
        "import re\n",
        "\n",
        "# split by spaces; the + denotes that spaces next to each other will also be removed\n",
        "items = re.split(' +', ftp_html)\n",
        "print(items)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['-r--r--r--', '1', 'ftp', 'anonymous', '1208', 'Feb', '11', '20:00', 'GCF_000005845.2_ASM584v2_assembly_report.txt\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '3746', 'Feb', '11', '20:00', 'GCF_000005845.2_ASM584v2_assembly_stats.txt\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '1504909', 'Oct', '14', '2018', 'GCF_000005845.2_ASM584v2_cds_from_genomic.fna.gz\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '244', 'Oct', '14', '2018', 'GCF_000005845.2_ASM584v2_feature_count.txt.gz\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '230172', 'Oct', '14', '2018', 'GCF_000005845.2_ASM584v2_feature_table.txt.gz\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '1379902', 'Oct', '31', '2014', 'GCF_000005845.2_ASM584v2_genomic.fna.gz\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '3462090', 'Mar', '2', '2019', 'GCF_000005845.2_ASM584v2_genomic.gbff.gz\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '465467', 'Feb', '11', '20:00', 'GCF_000005845.2_ASM584v2_genomic.gff.gz\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '483693', 'Feb', '11', '20:00', 'GCF_000005845.2_ASM584v2_genomic.gtf.gz\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '898290', 'Oct', '14', '2018', 'GCF_000005845.2_ASM584v2_protein.faa.gz\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '3674701', 'Mar', '2', '2019', 'GCF_000005845.2_ASM584v2_protein.gpff.gz\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '15100', 'Oct', '14', '2018', 'GCF_000005845.2_ASM584v2_rna_from_genomic.fna.gz\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '1082603', 'Oct', '14', '2018', 'GCF_000005845.2_ASM584v2_translated_cds.faa.gz\\r\\nlr--r--r--', '1', 'ftp', 'anonymous', '25', 'Sep', '20', '2016', 'README.txt', '->', '../../../../../README.txt\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '410', 'Feb', '11', '20:00', 'annotation_hashes.txt\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '14', 'Jul', '10', '04:51', 'assembly_status.txt\\r\\n-r--r--r--', '1', 'ftp', 'anonymous', '1094', 'Feb', '11', '20:00', 'md5checksums.txt\\r\\n']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3lAjWlJ9rBLv",
        "colab_type": "text"
      },
      "source": [
        "And now that we have this as a list, we can simply parse for the filename that we want. The unique substring in this case is going to be 'cds_from_genomic'."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ilxW56iRrBLw",
        "colab_type": "code",
        "colab": {},
        "outputId": "9adb8746-f93a-4326-cbd2-1731082b9351"
      },
      "source": [
        "for item in items:\n",
        "    if 'cds_from_genomic' in item:\n",
        "        file = item\n",
        "        break\n",
        "print(file)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "GCF_000005845.2_ASM584v2_cds_from_genomic.fna.gz\r\n",
            "-r--r--r--\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UR8gaAkYrBL2",
        "colab_type": "text"
      },
      "source": [
        "Whoops, apparently there are newlines too, which wouldn't have separated the last element of a line from the first of the next. We can easily get rid of that though:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j8xbcs6PrBL2",
        "colab_type": "code",
        "colab": {},
        "outputId": "6a8943f5-0719-4a8b-d0fd-9197781ff2ed"
      },
      "source": [
        "# split string by newline and take the first part\n",
        "file = file.split('\\n')[0]\n",
        "print(file)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "GCF_000005845.2_ASM584v2_cds_from_genomic.fna.gz\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DNqJOocdrBL9",
        "colab_type": "text"
      },
      "source": [
        "And finally, we'll concatenate this filename with the url of the FTP directory, since we need the whole path to download it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VJGjPN0DrBMA",
        "colab_type": "code",
        "colab": {},
        "outputId": "e1564b00-3c48-40a4-9af4-42b08fb7bd06"
      },
      "source": [
        "file = ftp_url + '/' + file\n",
        "print(file)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/005/845/GCF_000005845.2_ASM584v2/GCF_000005845.2_ASM584v2_cds_from_genomic.fna.gz\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bm_A05c3rBMT",
        "colab_type": "text"
      },
      "source": [
        "#### Downloading files with wget"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3O2-UuLArBMT",
        "colab_type": "text"
      },
      "source": [
        "So we have our sequence length and we have the path to the file we'd like to download. We can do this using wget, a tool that comes with most terminals. There's also a Python package that you can install as follows."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gFeTnKG9rBMU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# installation command for miniconda/anaconda environment\n",
        "conda install -c anaconda wget"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y5hGDZ7CrBMX",
        "colab_type": "text"
      },
      "source": [
        "Now literally all we have to do is call wget(url) to download the file into our current directory. I'm going to use the normal wget with os.system, since this is the more common practice. os is a default Python package and os.system allows you to run command line commands through Python."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SPn5NQK6rBMY",
        "colab_type": "code",
        "colab": {},
        "outputId": "6477c791-e746-49a5-b1b1-e4e9ba851ebe"
      },
      "source": [
        "import os\n",
        "\n",
        "os.system('wget ' + file)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2048"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OR_tVdngrBMf",
        "colab_type": "text"
      },
      "source": [
        "There seems to be an issue when I try to wget, and the out of 2048 tells me this (in general, anything nonzero is probably not right). The log on my command line looks like this:\n",
        "![](https://drive.google.com/uc?id=1e38dOeM6vn1lN6Y1d446ZA8BI9wZr5Ph)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NREYPG5WrBMg",
        "colab_type": "text"
      },
      "source": [
        "It seems that there was an extra '\\r' on the end of our string, which is the return character. This was probably just included in the data format that urllib returned (which we can confirm by looking back), and we can easily get rid of this by slicing off the last character."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "8LBtES39rBMh",
        "colab_type": "code",
        "colab": {},
        "outputId": "06414b09-9f35-419b-a7e3-8fddfd7b8661"
      },
      "source": [
        "file = file[:-1]\n",
        "print(file)\n",
        "os.system('wget ' + file)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/005/845/GCF_000005845.2_ASM584v2/GCF_000005845.2_ASM584v2_cds_from_genomic.fna.gz\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9U6h61UqrBMj",
        "colab_type": "text"
      },
      "source": [
        "And we get an output of 0, so we're good! If you check in your current directory you should also see the file there. Note that the file we've downloaded is a gzip file (compressed format for sharing), so we'll have to unzip it. We can do this with the 'gunzip' command (terminal command) and os.system."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LPyqHHE0rBMk",
        "colab_type": "code",
        "colab": {},
        "outputId": "bf82b28a-23d2-4270-c04f-4e11c1c98d5b"
      },
      "source": [
        "# get filename back from whole path, and take last part of the path (actual filename)\n",
        "print(file)\n",
        "zip_name = file.split('/')[-1]\n",
        "print(zip_name)\n",
        "# unzip\n",
        "os.system('gunzip ' + zip_name)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/005/845/GCF_000005845.2_ASM584v2/GCF_000005845.2_ASM584v2_cds_from_genomic.fna.gz\n",
            "GCF_000005845.2_ASM584v2_cds_from_genomic.fna.gz\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wnptj2GsrBMn",
        "colab_type": "text"
      },
      "source": [
        "Another 0 output, looks good. We'll have to slice zip_name to get rid of the .gz (gunzip automatically names the new file without .gz), and then we can read the file in and show some of it to get an idea of what the data looks like. Once we've seen the data, we can identify some unique patterns and use that to filter out what we want."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZSri_emErBMn",
        "colab_type": "code",
        "colab": {},
        "outputId": "116a3859-b38b-45a3-c11e-77eb26250405"
      },
      "source": [
        "# chop .gz off of our filename\n",
        "data_name = zip_name[:-3]\n",
        "print(data_name)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "GCF_000005845.2_ASM584v2_cds_from_genomic.fna\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ChxF3YIirBMq",
        "colab_type": "code",
        "colab": {},
        "outputId": "86e1b2b6-b974-4d2c-e487-3f269cb4d340"
      },
      "source": [
        "lines = []\n",
        "with open(data_name, 'r') as f:\n",
        "    lines = f.readlines()\n",
        "# preview\n",
        "for line in lines[:10]:\n",
        "    print(line)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            ">lcl|NC_000913.3_cds_NP_414542.1_1 [gene=thrL] [locus_tag=b0001] [db_xref=UniProtKB/Swiss-Prot:P0AD86] [protein=thr operon leader peptide] [protein_id=NP_414542.1] [location=190..255] [gbkey=CDS]\n",
            "\n",
            "ATGAAACGCATTAGCACCACCATTACCACCACCATCACCATTACCACAGGTAACGGTGCGGGCTGA\n",
            "\n",
            ">lcl|NC_000913.3_cds_NP_414543.1_2 [gene=thrA] [locus_tag=b0002] [db_xref=UniProtKB/Swiss-Prot:P00561] [protein=fused aspartate kinase/homoserine dehydrogenase 1] [protein_id=NP_414543.1] [location=337..2799] [gbkey=CDS]\n",
            "\n",
            "ATGCGAGTGTTGAAGTTCGGCGGTACATCAGTGGCAAATGCAGAACGTTTTCTGCGTGTTGCCGATATTCTGGAAAGCAA\n",
            "\n",
            "TGCCAGGCAGGGGCAGGTGGCCACCGTCCTCTCTGCCCCCGCCAAAATCACCAACCACCTGGTGGCGATGATTGAAAAAA\n",
            "\n",
            "CCATTAGCGGCCAGGATGCTTTACCCAATATCAGCGATGCCGAACGTATTTTTGCCGAACTTTTGACGGGACTCGCCGCC\n",
            "\n",
            "GCCCAGCCGGGGTTCCCGCTGGCGCAATTGAAAACTTTCGTCGATCAGGAATTTGCCCAAATAAAACATGTCCTGCATGG\n",
            "\n",
            "CATTAGTTTGTTGGGGCAGTGCCCGGATAGCATCAACGCTGCGCTGATTTGCCGTGGCGAGAAAATGTCGATCGCCATTA\n",
            "\n",
            "TGGCCGGCGTATTAGAAGCGCGCGGTCACAACGTTACTGTTATCGATCCGGTCGAAAAACTGCTGGCAGTGGGGCATTAC\n",
            "\n",
            "CTCGAATCTACCGTCGATATTGCTGAGTCCACCCGCCGTATTGCGGCAAGCCGCATTCCGGCTGATCACATGGTGCTGAT\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cRiuP6O_rBMw",
        "colab_type": "text"
      },
      "source": [
        "This data is in what is called fasta format (.fna). It has a pretty simple structure: for each entry, there is a header (denoted by a '>' at the start), and then lines of sequence follow starting on the next line. We can separate out headers and their respective sequences by parsing. If we were interested in the actual sequences we could log them, but since we only want lengths in this case, we'll make a dictionary that goes from header : sequence length. The unique identifier in this case is the '>', which tells us if a line is a header or a sequence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z80gni_TrBMx",
        "colab_type": "code",
        "colab": {},
        "outputId": "a5664924-a86e-47c7-9a0d-7217f76ec44b"
      },
      "source": [
        "genes = {}\n",
        "\n",
        "curr_header = None\n",
        "curr_length = 0\n",
        "# this file is really long! Let's only do the first 100 lines.\n",
        "for line in lines[:100]:\n",
        "    # check if a header. if yes, add last header to the dictionary and reset with new header and 0 length\n",
        "    if line[0] == '>':\n",
        "        # add the last entry if this is not the first iteration\n",
        "        if curr_header != None and curr_length != 0:\n",
        "            genes[curr_header] = curr_length\n",
        "        # don't want to keep the '>' or the newline \n",
        "        curr_header = line[1:-1]\n",
        "        curr_length = 0\n",
        "    # we have a sequence, so get length (chopping off newline at end) and add it\n",
        "    else:\n",
        "        curr_length += len(line[:-1])\n",
        "        \n",
        "# show\n",
        "for header in genes.keys():\n",
        "    print(header)\n",
        "    print(genes[header])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "lcl|NC_000913.3_cds_NP_414542.1_1 [gene=thrL] [locus_tag=b0001] [db_xref=UniProtKB/Swiss-Prot:P0AD86] [protein=thr operon leader peptide] [protein_id=NP_414542.1] [location=190..255] [gbkey=CDS]\n",
            "66\n",
            "lcl|NC_000913.3_cds_NP_414543.1_2 [gene=thrA] [locus_tag=b0002] [db_xref=UniProtKB/Swiss-Prot:P00561] [protein=fused aspartate kinase/homoserine dehydrogenase 1] [protein_id=NP_414543.1] [location=337..2799] [gbkey=CDS]\n",
            "2463\n",
            "lcl|NC_000913.3_cds_NP_414544.1_3 [gene=thrB] [locus_tag=b0003] [db_xref=UniProtKB/Swiss-Prot:P00547] [protein=homoserine kinase] [protein_id=NP_414544.1] [location=2801..3733] [gbkey=CDS]\n",
            "933\n",
            "lcl|NC_000913.3_cds_NP_414545.1_4 [gene=thrC] [locus_tag=b0004] [db_xref=UniProtKB/Swiss-Prot:P00934] [protein=threonine synthase] [protein_id=NP_414545.1] [location=3734..5020] [gbkey=CDS]\n",
            "1287\n",
            "lcl|NC_000913.3_cds_NP_414546.1_5 [gene=yaaX] [locus_tag=b0005] [db_xref=UniProtKB/Swiss-Prot:P75616] [protein=DUF2502 domain-containing protein YaaX] [protein_id=NP_414546.1] [location=5234..5530] [gbkey=CDS]\n",
            "297\n",
            "lcl|NC_000913.3_cds_NP_414547.1_6 [gene=yaaA] [locus_tag=b0006] [db_xref=UniProtKB/Swiss-Prot:P0A8I3] [protein=peroxide stress resistance protein YaaA] [protein_id=NP_414547.1] [location=complement(5683..6459)] [gbkey=CDS]\n",
            "777\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KtFegycorBM3",
        "colab_type": "text"
      },
      "source": [
        "Perfect! We now have a dictionary containing the headers mapped to the length of their respective sequences. \n",
        "\n",
        "Looking at the headers, we see that different pieces of information are separated by square brackets and have some sort of attribute to describe them, such as 'gene' or 'location'. You know what that means...we can parse it! When we want to find the length for a particular gene, we can just parse the headers for the gene name and then get the corresponding sequence length. I encourage you to try using what you've learned and parsing out the gene names. Now that we've extracted all of the data we needed, we can write a simple Python script to calculate relative standard deviations and compare them. We won't cover this since it's relatively straightforward with some coding background, but it would be a great exercise if you'd like to get some practice writing a script to scrape and compare data *en masse*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E23oZowIrBM4",
        "colab_type": "text"
      },
      "source": [
        "#### Wrapping up"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BBjSSkmgrBM5",
        "colab_type": "text"
      },
      "source": [
        "In this tutorial and the last, we learned what web scraping was, how to locate data of interest on a webpage, and how to parse both HTML and other data formats to extract the important parts! I hope you've learned a lot and encourage you to try it yourself. If you don't have any webpage/data in mind, feel free to use the examples I suggested at the beginning of the first tutorial. I find that contriving some motivating purpose (i.e. some sort of data analysis after you've extracted it) makes learning to scrape a lot more fun too!\n",
        "\n",
        "You are now free of the confines of nice, organized data--you can find and organize it yourself--and the massive amounts of data on the web are now yours to extract and analyze. Good luck, and happy scraping!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oaETRSLirBM6",
        "colab_type": "code",
        "colab": {},
        "outputId": "c33c11e8-4b76-4e33-bf70-c8d51b5ae0fb"
      },
      "source": [
        "%load_ext watermark\n",
        "%watermark -v -p bs4,requests,urllib3,re"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPython 3.7.7\n",
            "IPython 7.15.0\n",
            "\n",
            "bs4 4.9.1\n",
            "requests 2.23.0\n",
            "urllib3 1.25.9\n",
            "re 2.2.1\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}