intro-to-hadoop-01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center> Introduction to Hadoop MapReduce </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python Jupyter notebook supports execution of Linux command inside the notebook cells. This is done by adding the **!** to the beginning of the command line. It should be noted that each command begins with a **!** will create a new bash shell and close this cell once the execution is done:\n",
    "- Full path is required\n",
    "- Temporary results and environmental variables will be lost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Currently Loaded Modulefiles:\r\n",
      "  1) anaconda3/4.2.0   3) zeromq/4.1.5\r\n",
      "  2) matlab/2015a      4) hdp/0.1\r\n"
     ]
    }
   ],
   "source": [
    "!module list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to initialize Kerberos authentication mechanism"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!cypress-kinit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ticket cache: FILE:/home/lngo/.krb5cc\r\n",
      "Default principal: lngo@PALMETTO.CLEMSON.EDU\r\n",
      "\r\n",
      "Valid starting       Expires              Service principal\r\n",
      "10/04/2017 12:44:29  10/11/2017 12:44:29  krbtgt/PALMETTO.CLEMSON.EDU@PALMETTO.CLEMSON.EDU\r\n"
     ]
    }
   ],
   "source": [
    "!klist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interaction with Hadoop Distributed File System is done through `hdfs` and its sub-commands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!hdfs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!hdfs dfs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "Create a directory named **intro-to-hadoop** inside your user directory on HDFS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!hdfs dfs -ls /"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!ls /"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!hdfs dfs -ls /user/lngo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!hdfs dfs -mkdir intro-to-hadoop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "Upload the **text** directory into the newly created **intro-to-hadoop** directory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!hdfs dfs -put"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!hdfs dfs -put text intro-to-hadoop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge \n",
    "\n",
    "Check the health status of the directories above in HDFS using fsck:\n",
    "```\n",
    "hdfs fsck <path-to-directory> -files -blocks -locations\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connecting to namenode via http://dscim002.palmetto.clemson.edu:50070/fsck?ugi=lngo&files=1&blocks=1&locations=1&path=%2Fuser%2Flngo%2Fintro-to-hadoop%2Fgutenberg-shakespeare.txt\n",
      "FSCK started by lngo (auth:KERBEROS_SSL) from /10.125.3.168 for path /user/lngo/intro-to-hadoop/gutenberg-shakespeare.txt at Wed Oct 04 12:53:58 EDT 2017\n",
      "/user/lngo/intro-to-hadoop/gutenberg-shakespeare.txt 5447744 bytes, 1 block(s):  OK\n",
      "0. BP-1143747467-10.125.40.142-1413584797204:blk_1108454815_34728420 len=5447744 repl=2 [DatanodeInfoWithStorage[10.125.8.217:1019,DS-5960dbe8-cb5f-40a6-834f-f2edbd236732,DISK], DatanodeInfoWithStorage[10.125.8.227:1019,DS-91f408e1-e851-4308-bb72-0be27c3b689c,DISK]]\n",
      "\n",
      "Status: HEALTHY\n",
      " Total size:\t5447744 B\n",
      " Total dirs:\t0\n",
      " Total files:\t1\n",
      " Total symlinks:\t\t0\n",
      " Total blocks (validated):\t1 (avg. block size 5447744 B)\n",
      " Minimally replicated blocks:\t1 (100.0 %)\n",
      " Over-replicated blocks:\t0 (0.0 %)\n",
      " Under-replicated blocks:\t0 (0.0 %)\n",
      " Mis-replicated blocks:\t\t0 (0.0 %)\n",
      " Default replication factor:\t2\n",
      " Average block replication:\t2.0\n",
      " Corrupt blocks:\t\t0\n",
      " Missing replicas:\t\t0 (0.0 %)\n",
      " Number of data-nodes:\t\t40\n",
      " Number of racks:\t\t1\n",
      "FSCK ended at Wed Oct 04 12:53:58 EDT 2017 in 1 milliseconds\n",
      "\n",
      "\n",
      "The filesystem under path '/user/lngo/intro-to-hadoop/gutenberg-shakespeare.txt' is HEALTHY\n"
     ]
    }
   ],
   "source": [
    "!hdfs fsck intro-to-hadoop/text/gutenberg-shakespeare.txt -files -blocks -locations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MapReduce Programming Paradigm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is “map”?**\n",
    "– A function/procedure that is applied to every individual\n",
    "elements of a collection/list/array/…\n",
    "\n",
    "```\n",
    "int square(x) { return x*x;}\n",
    "map square [1,2,3,4] -> [1,4,9,16]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is “reduce”?**\n",
    "– A function/procedure that performs an operation on a list.\n",
    "This operation will “fold/reduce” this list into a single value\n",
    "(or a smaller subset)\n",
    "\n",
    "```\n",
    "reduce ([1,2,3,4]) using sum -> 10\n",
    "reduce ([1,2,3,4]) using multiply -> 24\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MapReduce is an old concept in functional programming. It is naturally applicable in HDFS: \n",
    "- `map` tasks are performed on top of individual data blocks (mainly to filter and decrease raw data contents while increase data value\n",
    "- `reduce` tasks are performed on intermediate results from `map` tasks (should now be significantly decreased in size) to calculate the final results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. The Hello World of Hadoop: Word Count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!mkdir codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1609\r\n",
      "\r\n",
      "THE SONNETS\r\n",
      "\r\n",
      "by William Shakespeare\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "                     1\r\n",
      "  From fairest creatures we desire increase,\r\n",
      "  That thereby beauty's rose might never die,\r\n",
      "  But as the riper should by time decease,\r\n",
      "  His tender heir might bear his memory:\r\n",
      "  But thou contracted to thine own bright eyes,\r\n",
      "  Feed'st thy light's flame with self-substantial fuel,\r\n",
      "  Making a famine where abundance lies,\r\n",
      "  Thy self thy foe, to thy sweet self too cruel:\r\n",
      "  Thou that art now the world's fresh ornament,\r\n",
      "  And only herald to the gaudy spring,\r\n",
      "  Within thine own bud buriest thy content,\r\n",
      "  And tender churl mak'st waste in niggarding:\r\n",
      "    Pity the world, or else this glutton be,\r\n",
      "    To eat the world's due, by the grave and thee.\r\n",
      "\r\n",
      "\r\n",
      "                     2\r\n",
      "  When forty winters shall besiege thy brow,\r\n",
      "  And dig deep trenches in thy beauty's field,\r\n",
      "  Thy youth's proud livery so gazed on now,\r\n",
      "  Will be a tattered weed of small worth held:  \r\n",
      "  Then being asked, where all thy beauty lies,\r\n",
      "  Where all the treasure of thy lusty days;\r\n",
      "  To say within thine own deep sunken eyes,\r\n",
      "  Were an all-eating shame, and thriftless praise.\r\n",
      "  How much more praise deserved thy beauty's use,\r\n",
      "  If thou couldst answer 'This fair child of mine\r\n",
      "  Shall sum my count, and make my old excuse'\r\n",
      "  Proving his beauty by succession thine.\r\n",
      "    This were to be new made when thou art old,\r\n",
      "    And see thy blood warm when thou feel'st it cold.\r\n",
      "\r\n",
      "\r\n",
      "                     3\r\n",
      "  Look in thy glass and tell the face thou viewest,\r\n",
      "  Now is the time that face should form another,\r\n",
      "  Whose fresh repair if now thou not renewest,\r\n",
      "  Thou dost beguile the world, unbless some mother.\r\n",
      "  For where is she so fair whose uneared womb\r\n",
      "  Disdains the tillage of thy husbandry?\r\n",
      "  Or who is he so fond will be the tomb,\r\n",
      "  Of his self-love to stop posterity?  \r\n",
      "  Thou art thy mother's glass and she in thee\r\n",
      "  Calls back the lovely April of her prime,\r\n",
      "  So thou through windows of thine age shalt see,\r\n",
      "  Despite of wrinkles this thy golden time.\r\n",
      "    But if thou live remembered not to be,\r\n",
      "    Die single and thine image dies with thee.\r\n",
      "\r\n",
      "\r\n",
      "                     4\r\n",
      "  Unthrifty loveliness why dost thou spend,\r\n",
      "  Upon thy self thy beauty's legacy?\r\n",
      "  Nature's bequest gives nothing but doth lend,\r\n",
      "  And being frank she lends to those are free:\r\n",
      "  Then beauteous niggard why dost thou abuse,\r\n",
      "  The bounteous largess given thee to give?\r\n",
      "  Profitless usurer why dost thou use\r\n",
      "  So great a sum of sums yet canst not live?\r\n",
      "  For having traffic with thy self alone,\r\n",
      "  Thou of thy self thy sweet self dost deceive,\r\n",
      "  Then how when nature calls thee to be gone,\r\n",
      "  What acceptable audit canst thou leave?  \r\n",
      "    Thy unused beauty must be tombed with thee,\r\n",
      "    Which used lives th' executor to be.\r\n",
      "\r\n",
      "\r\n",
      "                     5\r\n",
      "  Those hours that with gentle work did frame\r\n",
      "  The lovely gaze where every eye doth dwell\r\n",
      "  Will play the tyrants to the very same,\r\n",
      "  And that unfair which fairly doth excel:\r\n",
      "  For never-resting time leads summer on\r\n",
      "  To hideous winter and confounds him there,\r\n",
      "  Sap checked with frost and lusty leaves quite gone,\r\n",
      "  Beauty o'er-snowed and bareness every where:\r\n",
      "  Then were not summer's distillation left\r\n",
      "  A liquid prisoner pent in walls of glass,\r\n",
      "  Beauty's effect with beauty were bereft,\r\n",
      "  Nor it nor no remembrance what it was.\r\n",
      "    But flowers distilled though they with winter meet,\r\n",
      "    Leese but their show, their substance still lives sweet.\r\n",
      "\r\n",
      "\r\n",
      "                     6  \r\n",
      "  Then let not winter's ragged hand deface,\r\n",
      "  In thee thy summer ere thou be distilled:\r\n",
      "  Make sweet some vial; treasure thou some place,\r\n",
      "  With beauty's treasure ere it be self-killed:\r\n",
      "  That use is not forbidden usury,\r\n",
      "  Which happies those that pay the willing loan;\r\n"
     ]
    }
   ],
   "source": [
    "!hdfs dfs -cat intro-to-hadoop/text/gutenberg-shakespeare.txt \\\n",
    "    2>/dev/null | head -n 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing codes/wordcountMapper.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile codes/wordcountMapper.py\n",
    "#!/usr/bin/env python                                          \n",
    "import sys                                                                                                \n",
    "for oneLine in sys.stdin:\n",
    "    oneLine = oneLine.strip()\n",
    "    for word in oneLine.split(\" \"):\n",
    "        if word != \"\":\n",
    "            print ('%s\\t%s' % (word, 1)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1609\t1\r\n",
      "THE\t1\r\n",
      "SONNETS\t1\r\n",
      "by\t1\r\n",
      "William\t1\r\n",
      "Shakespeare\t1\r\n",
      "1\t1\r\n",
      "From\t1\r\n",
      "fairest\t1\r\n",
      "creatures\t1\r\n",
      "we\t1\r\n",
      "desire\t1\r\n",
      "increase,\t1\r\n",
      "That\t1\r\n",
      "thereby\t1\r\n",
      "beauty's\t1\r\n",
      "rose\t1\r\n",
      "might\t1\r\n",
      "never\t1\r\n",
      "die,\t1\r\n",
      "But\t1\r\n",
      "as\t1\r\n",
      "the\t1\r\n",
      "riper\t1\r\n",
      "should\t1\r\n",
      "by\t1\r\n",
      "time\t1\r\n",
      "decease,\t1\r\n",
      "His\t1\r\n",
      "tender\t1\r\n",
      "heir\t1\r\n",
      "might\t1\r\n",
      "bear\t1\r\n",
      "his\t1\r\n",
      "memory:\t1\r\n",
      "But\t1\r\n",
      "thou\t1\r\n",
      "contracted\t1\r\n",
      "to\t1\r\n",
      "thine\t1\r\n",
      "own\t1\r\n",
      "bright\t1\r\n",
      "eyes,\t1\r\n",
      "Feed'st\t1\r\n",
      "thy\t1\r\n",
      "light's\t1\r\n",
      "flame\t1\r\n",
      "with\t1\r\n",
      "self-substantial\t1\r\n",
      "fuel,\t1\r\n",
      "Making\t1\r\n",
      "a\t1\r\n",
      "famine\t1\r\n",
      "where\t1\r\n",
      "abundance\t1\r\n",
      "lies,\t1\r\n",
      "Thy\t1\r\n",
      "self\t1\r\n",
      "thy\t1\r\n",
      "foe,\t1\r\n",
      "to\t1\r\n",
      "thy\t1\r\n",
      "sweet\t1\r\n",
      "self\t1\r\n",
      "too\t1\r\n",
      "cruel:\t1\r\n",
      "Thou\t1\r\n",
      "that\t1\r\n",
      "art\t1\r\n",
      "now\t1\r\n",
      "the\t1\r\n",
      "world's\t1\r\n",
      "fresh\t1\r\n",
      "ornament,\t1\r\n",
      "And\t1\r\n",
      "only\t1\r\n",
      "herald\t1\r\n",
      "to\t1\r\n",
      "the\t1\r\n",
      "gaudy\t1\r\n",
      "spring,\t1\r\n",
      "Within\t1\r\n",
      "thine\t1\r\n",
      "own\t1\r\n",
      "bud\t1\r\n",
      "buriest\t1\r\n",
      "thy\t1\r\n",
      "content,\t1\r\n"
     ]
    }
   ],
   "source": [
    "!hdfs dfs -cat intro-to-hadoop/text/gutenberg-shakespeare.txt \\\n",
    "    2>/dev/null \\\n",
    "    | head -n 20 \\\n",
    "    | python ./codes/wordcountMapper.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\t1\r\n",
      "1609\t1\r\n",
      "a\t1\r\n",
      "abundance\t1\r\n",
      "And\t1\r\n",
      "art\t1\r\n",
      "as\t1\r\n",
      "bear\t1\r\n",
      "beauty's\t1\r\n",
      "bright\t1\r\n",
      "bud\t1\r\n",
      "buriest\t1\r\n",
      "But\t1\r\n",
      "But\t1\r\n",
      "by\t1\r\n",
      "by\t1\r\n",
      "content,\t1\r\n",
      "contracted\t1\r\n",
      "creatures\t1\r\n",
      "cruel:\t1\r\n",
      "decease,\t1\r\n",
      "desire\t1\r\n",
      "die,\t1\r\n",
      "eyes,\t1\r\n",
      "fairest\t1\r\n",
      "famine\t1\r\n",
      "Feed'st\t1\r\n",
      "flame\t1\r\n",
      "foe,\t1\r\n",
      "fresh\t1\r\n",
      "From\t1\r\n",
      "fuel,\t1\r\n",
      "gaudy\t1\r\n",
      "heir\t1\r\n",
      "herald\t1\r\n",
      "his\t1\r\n",
      "His\t1\r\n",
      "increase,\t1\r\n",
      "lies,\t1\r\n",
      "light's\t1\r\n",
      "Making\t1\r\n",
      "memory:\t1\r\n",
      "might\t1\r\n",
      "might\t1\r\n",
      "never\t1\r\n",
      "now\t1\r\n",
      "only\t1\r\n",
      "ornament,\t1\r\n",
      "own\t1\r\n",
      "own\t1\r\n",
      "riper\t1\r\n",
      "rose\t1\r\n",
      "self\t1\r\n",
      "self\t1\r\n",
      "self-substantial\t1\r\n",
      "Shakespeare\t1\r\n",
      "should\t1\r\n",
      "SONNETS\t1\r\n",
      "spring,\t1\r\n",
      "sweet\t1\r\n",
      "tender\t1\r\n",
      "that\t1\r\n",
      "That\t1\r\n",
      "the\t1\r\n",
      "the\t1\r\n",
      "the\t1\r\n",
      "THE\t1\r\n",
      "thereby\t1\r\n",
      "thine\t1\r\n",
      "thine\t1\r\n",
      "thou\t1\r\n",
      "Thou\t1\r\n",
      "thy\t1\r\n",
      "thy\t1\r\n",
      "thy\t1\r\n",
      "thy\t1\r\n",
      "Thy\t1\r\n",
      "time\t1\r\n",
      "to\t1\r\n",
      "to\t1\r\n",
      "to\t1\r\n",
      "too\t1\r\n",
      "we\t1\r\n",
      "where\t1\r\n",
      "William\t1\r\n",
      "with\t1\r\n",
      "Within\t1\r\n",
      "world's\t1\r\n"
     ]
    }
   ],
   "source": [
    "!hdfs dfs -cat intro-to-hadoop/text/gutenberg-shakespeare.txt \\\n",
    "    2>/dev/null \\\n",
    "    | head -n 20 \\\n",
    "    | python ./codes/wordcountMapper.py \\\n",
    "    | sort"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing codes/wordcountReducer.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile codes/wordcountReducer.py\n",
    "#!/usr/bin/env python\n",
    "import sys\n",
    "\n",
    "current_word = None\n",
    "total_word_count = 0\n",
    "\n",
    "for line in sys.stdin:\n",
    "    line = line.strip()\n",
    "    word, count = line.split(\"\\t\", 1)\n",
    "    try:\n",
    "        count = int(count)\n",
    "    except ValueError:\n",
    "        continue\n",
    "    \n",
    "    if current_word == word:\n",
    "        total_word_count += count\n",
    "    else:\n",
    "        if current_word:\n",
    "            print (\"%s\\t%s\" % (current_word, total_word_count))\n",
    "        current_word = word\n",
    "        total_word_count = 1\n",
    "        \n",
    "if current_word == word:\n",
    "    print (\"%s\\t%s\" % (current_word, total_word_count))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\t1\r\n",
      "1609\t1\r\n",
      "a\t1\r\n",
      "abundance\t1\r\n",
      "And\t1\r\n",
      "art\t1\r\n",
      "as\t1\r\n",
      "bear\t1\r\n",
      "beauty's\t1\r\n",
      "bright\t1\r\n",
      "bud\t1\r\n",
      "buriest\t1\r\n",
      "But\t2\r\n",
      "by\t2\r\n",
      "content,\t1\r\n",
      "contracted\t1\r\n",
      "creatures\t1\r\n",
      "cruel:\t1\r\n",
      "decease,\t1\r\n",
      "desire\t1\r\n",
      "die,\t1\r\n",
      "eyes,\t1\r\n",
      "fairest\t1\r\n",
      "famine\t1\r\n",
      "Feed'st\t1\r\n",
      "flame\t1\r\n",
      "foe,\t1\r\n",
      "fresh\t1\r\n",
      "From\t1\r\n",
      "fuel,\t1\r\n",
      "gaudy\t1\r\n",
      "heir\t1\r\n",
      "herald\t1\r\n",
      "his\t1\r\n",
      "His\t1\r\n",
      "increase,\t1\r\n",
      "lies,\t1\r\n",
      "light's\t1\r\n",
      "Making\t1\r\n",
      "memory:\t1\r\n",
      "might\t2\r\n",
      "never\t1\r\n",
      "now\t1\r\n",
      "only\t1\r\n",
      "ornament,\t1\r\n",
      "own\t2\r\n",
      "riper\t1\r\n",
      "rose\t1\r\n",
      "self\t2\r\n",
      "self-substantial\t1\r\n",
      "Shakespeare\t1\r\n",
      "should\t1\r\n",
      "SONNETS\t1\r\n",
      "spring,\t1\r\n",
      "sweet\t1\r\n",
      "tender\t1\r\n",
      "that\t1\r\n",
      "That\t1\r\n",
      "the\t3\r\n",
      "THE\t1\r\n",
      "thereby\t1\r\n",
      "thine\t2\r\n",
      "thou\t1\r\n",
      "Thou\t1\r\n",
      "thy\t4\r\n",
      "Thy\t1\r\n",
      "time\t1\r\n",
      "to\t3\r\n",
      "too\t1\r\n",
      "we\t1\r\n",
      "where\t1\r\n",
      "William\t1\r\n",
      "with\t1\r\n",
      "Within\t1\r\n",
      "world's\t1\r\n"
     ]
    }
   ],
   "source": [
    "!hdfs dfs -cat intro-to-hadoop/text/gutenberg-shakespeare.txt \\\n",
    "    2>/dev/null \\\n",
    "    | head -n 20 \\\n",
    "    | python ./codes/wordcountMapper.py \\\n",
    "    | sort \\\n",
    "    | python ./codes/wordcountReducer.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: `intro-to-hadoop/output-wordcount': No such file or directory\n",
      "17/10/04 13:02:39 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.\n",
      "packageJobJar: [./codes/wordcountMapper.py, ./codes/wordcountReducer.py] [/usr/hdp/2.6.0.3-8/hadoop-mapreduce/hadoop-streaming-2.7.3.2.6.0.3-8.jar] /hadoop_java_io_tmpdir/streamjob2536420095245052849.jar tmpDir=null\n",
      "17/10/04 13:02:41 INFO client.AHSProxy: Connecting to Application History server at dscim003.palmetto.clemson.edu/10.125.8.215:10200\n",
      "17/10/04 13:02:41 INFO client.AHSProxy: Connecting to Application History server at dscim003.palmetto.clemson.edu/10.125.8.215:10200\n",
      "17/10/04 13:02:41 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 14303 for lngo on ha-hdfs:dsci\n",
      "17/10/04 13:02:41 INFO security.TokenCache: Got dt for hdfs://dsci; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:dsci, Ident: (HDFS_DELEGATION_TOKEN token 14303 for lngo)\n",
      "17/10/04 13:02:42 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library\n",
      "17/10/04 13:02:42 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 8787857212dae53ffae3b3113abc894e6743b4ab]\n",
      "17/10/04 13:02:42 INFO mapred.FileInputFormat: Total input paths to process : 1\n",
      "17/10/04 13:02:42 INFO mapreduce.JobSubmitter: number of splits:2\n",
      "17/10/04 13:02:43 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505269880969_0039\n",
      "17/10/04 13:02:43 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:dsci, Ident: (HDFS_DELEGATION_TOKEN token 14303 for lngo)\n",
      "17/10/04 13:02:43 INFO impl.TimelineClientImpl: Timeline service address: http://dscim003.palmetto.clemson.edu:8188/ws/v1/timeline/\n",
      "17/10/04 13:02:44 INFO impl.YarnClientImpl: Submitted application application_1505269880969_0039\n",
      "17/10/04 13:02:44 INFO mapreduce.Job: The url to track the job: http://dscim001.palmetto.clemson.edu:8088/proxy/application_1505269880969_0039/\n",
      "17/10/04 13:02:44 INFO mapreduce.Job: Running job: job_1505269880969_0039\n",
      "17/10/04 13:02:54 INFO mapreduce.Job: Job job_1505269880969_0039 running in uber mode : false\n",
      "17/10/04 13:02:54 INFO mapreduce.Job:  map 0% reduce 0%\n",
      "17/10/04 13:03:01 INFO mapreduce.Job:  map 50% reduce 0%\n",
      "17/10/04 13:03:08 INFO mapreduce.Job:  map 100% reduce 0%\n",
      "17/10/04 13:03:10 INFO mapreduce.Job:  map 100% reduce 100%\n",
      "17/10/04 13:03:10 INFO mapreduce.Job: Job job_1505269880969_0039 completed successfully\n",
      "17/10/04 13:03:11 INFO mapreduce.Job: Counters: 50\n",
      "\tFile System Counters\n",
      "\t\tFILE: Number of bytes read=8529635\n",
      "\t\tFILE: Number of bytes written=17548535\n",
      "\t\tFILE: Number of read operations=0\n",
      "\t\tFILE: Number of large read operations=0\n",
      "\t\tFILE: Number of write operations=0\n",
      "\t\tHDFS: Number of bytes read=5476624\n",
      "\t\tHDFS: Number of bytes written=713504\n",
      "\t\tHDFS: Number of read operations=9\n",
      "\t\tHDFS: Number of large read operations=0\n",
      "\t\tHDFS: Number of write operations=2\n",
      "\tJob Counters \n",
      "\t\tKilled map tasks=1\n",
      "\t\tLaunched map tasks=2\n",
      "\t\tLaunched reduce tasks=1\n",
      "\t\tRack-local map tasks=2\n",
      "\t\tTotal time spent by all maps in occupied slots (ms)=51279\n",
      "\t\tTotal time spent by all reduces in occupied slots (ms)=17241\n",
      "\t\tTotal time spent by all map tasks (ms)=17093\n",
      "\t\tTotal time spent by all reduce tasks (ms)=5747\n",
      "\t\tTotal vcore-milliseconds taken by all map tasks=17093\n",
      "\t\tTotal vcore-milliseconds taken by all reduce tasks=5747\n",
      "\t\tTotal megabyte-milliseconds taken by all map tasks=220362956\n",
      "\t\tTotal megabyte-milliseconds taken by all reduce tasks=74090324\n",
      "\tMap-Reduce Framework\n",
      "\t\tMap input records=124213\n",
      "\t\tMap output records=899681\n",
      "\t\tMap output bytes=6730267\n",
      "\t\tMap output materialized bytes=8529641\n",
      "\t\tInput split bytes=240\n",
      "\t\tCombine input records=0\n",
      "\t\tCombine output records=0\n",
      "\t\tReduce input groups=67109\n",
      "\t\tReduce shuffle bytes=8529641\n",
      "\t\tReduce input records=899681\n",
      "\t\tReduce output records=67109\n",
      "\t\tSpilled Records=1799362\n",
      "\t\tShuffled Maps =2\n",
      "\t\tFailed Shuffles=0\n",
      "\t\tMerged Map outputs=2\n",
      "\t\tGC time elapsed (ms)=248\n",
      "\t\tCPU time spent (ms)=9660\n",
      "\t\tPhysical memory (bytes) snapshot=3859386368\n",
      "\t\tVirtual memory (bytes) snapshot=39889514496\n",
      "\t\tTotal committed heap usage (bytes)=3936878592\n",
      "\tShuffle Errors\n",
      "\t\tBAD_ID=0\n",
      "\t\tCONNECTION=0\n",
      "\t\tIO_ERROR=0\n",
      "\t\tWRONG_LENGTH=0\n",
      "\t\tWRONG_MAP=0\n",
      "\t\tWRONG_REDUCE=0\n",
      "\tFile Input Format Counters \n",
      "\t\tBytes Read=5476384\n",
      "\tFile Output Format Counters \n",
      "\t\tBytes Written=713504\n",
      "17/10/04 13:03:11 INFO streaming.StreamJob: Output directory: intro-to-hadoop/output-wordcount\n"
     ]
    }
   ],
   "source": [
    "!hdfs dfs -rm -R intro-to-hadoop/output-wordcount\n",
    "!yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar \\\n",
    "    -input intro-to-hadoop/text/gutenberg-shakespeare.txt \\\n",
    "    -output intro-to-hadoop/output-wordcount \\\n",
    "    -file ./codes/wordcountMapper.py \\\n",
    "    -mapper wordcountMapper.py \\\n",
    "    -file ./codes/wordcountReducer.py \\\n",
    "    -reducer wordcountReducer.py \\"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2 items\r\n",
      "-rw-r--r--   2 lngo hdfs          0 2017-10-04 13:03 intro-to-hadoop/output-wordcount/_SUCCESS\r\n",
      "-rw-r--r--   2 lngo hdfs     713504 2017-10-04 13:03 intro-to-hadoop/output-wordcount/part-00000\r\n"
     ]
    }
   ],
   "source": [
    "!hdfs dfs -ls intro-to-hadoop/output-wordcount"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"\t241\r\n",
      "\"'Tis\t1\r\n",
      "\"A\t4\r\n",
      "\"Air,\"\t1\r\n",
      "\"Alas,\t1\r\n",
      "\"Amen\"\t2\r\n",
      "\"Amen\"?\t1\r\n",
      "\"Amen,\"\t1\r\n",
      "\"And\t1\r\n",
      "\"Aroint\t1\r\n",
      "\"B\t1\r\n",
      "\"Black\t1\r\n",
      "\"Break\t1\r\n",
      "\"Brutus\"\t1\r\n",
      "\"Brutus,\t2\r\n",
      "\"C\t1\r\n",
      "\"Caesar\"?\t1\r\n",
      "\"Caesar,\t1\r\n",
      "\"Caesar.\"\t2\r\n",
      "\"Certes,\"\t1\r\n",
      "\"Come\t1\r\n",
      "\"Cursed\t1\r\n",
      "\"D\t1\r\n",
      "\"Darest\t1\r\n",
      "\"Do\t1\r\n",
      "\"E\t1\r\n",
      "\"Fear\t2\r\n",
      "\"Fly,\t1\r\n",
      "\"Gentle\t1\r\n",
      "\"Give\t2\r\n",
      "\"Glamis\t1\r\n",
      "\"God\t2\r\n",
      "\"Good\t1\r\n",
      "\"Havoc!\"\t1\r\n",
      "\"He\t1\r\n",
      "\"Help\t1\r\n",
      "\"Help,\t2\r\n",
      "\"Here\t1\r\n",
      "\"Hold,\t2\r\n",
      "\"I\t4\r\n",
      "\"Indeed!\"\t1\r\n",
      "\"King\t1\r\n",
      "\"Liberty,\t1\r\n",
      "\"Lo,\t1\r\n",
      "\"Long\t1\r\n",
      "\"Murther!\"\t2\r\n",
      "\"Neither\t1\r\n",
      "\"Now\t1\r\n",
      "\"O\t2\r\n",
      "\"Peace,\t1\r\n",
      "\"Shall\t1\r\n",
      "\"Sing\t2\r\n",
      "\"Sir,\t1\r\n",
      "\"Sleep\t2\r\n",
      "\"Speak,\t1\r\n",
      "\"Sweet\t1\r\n",
      "\"That\t1\r\n",
      "\"The\t1\r\n",
      "\"These\t1\r\n",
      "\"They\t2\r\n",
      "\"This\t2\r\n",
      "\"Thus\t2\r\n",
      "\"Tis\t2\r\n",
      "\"Where\t1\r\n",
      "\"Willow,\t1\r\n",
      "\"You'll\t1\r\n",
      "\"better\"?\t1\r\n",
      "\"hem,\"\t1\r\n",
      "\"never.\"\t1\r\n",
      "\"not\"\t1\r\n",
      "\"then\"\t1\r\n",
      "\"thrusting\"\t1\r\n",
      "\"thy\t1\r\n",
      "\"twas\t1\r\n",
      "\"whore\"\t1\r\n",
      "\"whore.\"\t1\r\n",
      "\"willow\";\t1\r\n",
      "&\t3\r\n",
      "&C.\t2\r\n",
      "&c.\t12\r\n",
      "&c.'\t2\r\n",
      "&c.,\t2\r\n",
      "'\"All\t1\r\n",
      "'\"Among\t1\r\n",
      "'\"And,\t1\r\n",
      "'\"But,\t1\r\n",
      "'\"Gamut\"\t1\r\n",
      "'\"How\t1\r\n",
      "'\"Lo,\t2\r\n",
      "'\"Look\t1\r\n",
      "'\"My\t1\r\n",
      "'\"Now\t1\r\n",
      "'\"O\t2\r\n",
      "'\"The\t1\r\n",
      "'\"When\t1\r\n",
      "''Tis\t3\r\n",
      "'-on\t1\r\n",
      "'A\t53\r\n",
      "'A-down\t1\r\n",
      "'Above\t1\r\n"
     ]
    }
   ],
   "source": [
    "!hdfs dfs -cat intro-to-hadoop/output-wordcount/part-00000 \\\n",
    "    2>/dev/null | head -n 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "Modify *wordcountMapper.py* so that punctuations and capitalization are no longer factors in determining unique words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%writefile codes/wordcountEnhancedMapper.py\n",
    "#!/usr/bin/env python                                          \n",
    "import sys                     \n",
    "import string\n",
    "\n",
    "translator = str.maketrans('', '', string.punctuation)\n",
    "\n",
    "for oneLine in sys.stdin:\n",
    "    oneLine = oneLine.strip()\n",
    "    for word in oneLine.split(\" \"):\n",
    "        if word != \"\":\n",
    "            newWord = word.translate(translator).lower()\n",
    "            print ('%s\\t%s' % (_______, 1)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!hdfs dfs -rm -R intro-to-hadoop/output-wordcount-enhanced\n",
    "!ssh dsciutil yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar \\\n",
    "    -input intro-to-hadoop/text/gutenberg-shakespeare.txt \\\n",
    "    -output intro-to-hadoop/output-wordcount \\\n",
    "    -file ____________________________________________________ \\\n",
    "    -mapper _____________________ \\\n",
    "    -file ____________________________________________________ \\\n",
    "    -reducer _____________________ \\"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (Anaconda 5.1.0)",
   "language": "python",
   "name": "anaconda3-5.1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}