UDS-workshop-working-with-textual-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--<h1 style=\"font-size:40px; font-family:Verdana\" align=\"center\"> UDS-Club Workshop </h1> -->\n",
    "<h2 style=\"font-size:34px; font-family:Verdana\" align=\"center\"> Working With Textual Data: Features </h2>\n",
    "<!-- <img src='http://www.cmu.edu/africa/files/images/AppliedMachineLearningLogo.png'/> -->\n",
    "<img src='https://files.slack.com/files-pri/T41777KHA-F4TTMNKNK/ua-parrots.jpg'/>\n",
    "<h4 style=\"font-size:18px; font-family:Verdana\" align=\"right\"> by Fred Navruzov <br> <pre>    2017-04-23</pre> </h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='0'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height: 1px; background-color: #808080\">\n",
    "<h2 style=\"font-size:25px; font-family:Verdana\" align=\"left\"> Table of Contents </h2>\n",
    "<ol>\n",
    "    <li style=\"font-size:20px; font-family:Verdana\">[Prerequisites](#1)</li>\n",
    "    <li style=\"font-size:20px; font-family:Verdana\">[Basics of Textual Features](#2)</li>\n",
    "    <li style=\"font-size:20px; font-family:Verdana\">[Bag-of-Words Approach](#3)\n",
    "        <ul> \n",
    "            <li style=\"font-size:16px; font-family:Verdana\">[Intuition Behind the Model. Word Counters](#3_1)</li>\n",
    "            <li style=\"font-size:16px; font-family:Verdana\">[Capturing Dependencies. N-grams Recall](#3_2)</li>\n",
    "            <li style=\"font-size:16px; font-family:Verdana\">[CountVectorizer](#3_3)</li>\n",
    "            <li style=\"font-size:16px; font-family:Verdana\">[Tf-idf Augmentation. TfIdfVectorizer](#3_4)</li>\n",
    "            <li style=\"font-size:16px; font-family:Verdana\">[Hashes. HashingVectorizer](#3_5)</li>\n",
    "            \n",
    "        </ul>\n",
    "    </li>\n",
    "    <br><li style=\"font-size:20px; font-family:Verdana\">[Going Beyond: Custom Features](#4)\n",
    "        <ul>\n",
    "            <li style=\"font-size:16px; font-family:Verdana\">[Token-based Level](#4_1)</li>\n",
    "            <li style=\"font-size:16px; font-family:Verdana\">[Sentence-based / text-based Level](#4_2)</li>\n",
    "            <li style=\"font-size:16px; font-family:Verdana\">[Combining Features: FeatureUnion / Pipeline](#4_3)</li>\n",
    "        </ul>\n",
    "    </li>\n",
    "    \n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='1'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height: 1px; background-color: #808080\">\n",
    "## 1. Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Installation-based\n",
    "<ol>\n",
    "    <li>Python 3.x (or Anaconda3 for Python 3.5, https://www.continuum.io/downloads)</li>\n",
    "    <li>Scikit-learn 0.18.x (pip install scikit-learn==0.18.1, http://scikit-learn.org/)</li>\n",
    "    <li>TextBlob (pip install textblob, https://textblob.readthedocs.io/en/dev/)</li>\n",
    "    <li>GIT (https://git-scm.com/downloads)</li>\n",
    "</ol>\n",
    "\n",
    "#### Knowledge-based\n",
    "<ol>\n",
    "    <li>Token definition</li>\n",
    "    <li>Text normalization (stemming, lemmatization)</li>\n",
    "    <li>Stop-words definition</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 style=\"font-size:16px; font-family:Verdana\">[To the table of contents](#0)</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='2'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height: 1px; background-color: #808080\">\n",
    "## 2. Basics of Textual Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In supervised learning domain, for example, to perform classification tasks, usually our goal is to find a parametrized model, best in its class: <br><br> $A(X, \\hat{w}): A(X, \\hat{w}) \\simeq f(X) \\Leftrightarrow A(X, \\hat{w}) = \\operatorname*{arg\\,min}_w \\left\\|A(X, w) - f(X)\\right\\|$\n",
    "\n",
    "Where $X \\in R^{ n\\times m}$ - feature matrix ($n$ observations with $m$ features), $w \\in R^{m}$ - vector of model parameters, $\\hat{w}$ - \"best\" model parameters\n",
    "\n",
    "However, as a candidate for X - all that we have <strong>is raw text input, algorithms can't use it as is :(  </strong>\n",
    "\n",
    "In order to apply machine learning on textual data, we first need to transform such content into some numerical format (to form feature vectors). \n",
    "\n",
    "In Natural Language Processing automated feature extraction may be achieved in many ways <strong>(bag-of-words, word embeddings, tree-based representation etc.)</strong>\n",
    "\n",
    "Today, we will dive into details of <strong>bag-of-words</strong> approach and methods, built atop of it in Scikit-Learn library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 style=\"font-size:16px; font-family:Verdana\">[To the table of contents](#0)</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='3'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height: 1px; background-color: #808080 border: dashed 1px\">\n",
    "## 3. Bag-of-Words Approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='3_1'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<hr style=\"border: 1px dashed #ffffff;\" />\n",
    "### 3.1 Intuition Behind the Model. Word Counters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In bag-of-words approach we work under the following assumptions:\n",
    "<ul>\n",
    "    <li> The text can be analyzed without taking into account the word/token order </li>\n",
    "    <li> We only need to know what words/tokens the text consists of and how many times we met them </li>\n",
    "    <li> **The more often a word/token appears in a document, the more important it is** </li>\n",
    "</ul>\n",
    "\n",
    "More formal, given the collection of texts $T_1, T_2, ... , T_n$, we extract unique tokens $w_1, w_2, ..., w_m$ to form a dictionary.\n",
    "\n",
    "Thus, each text $T_i$ is represented by feature vector $F_j = \\{x_{ij},\\ j \\in [1,m]\\}$, where $x_{ij}$ corresponds to number of occurences of word $w_j$ in text $T_i$\n",
    "\n",
    "Say, out corpus only consists of **2 texts**:\n",
    "\n",
    "[\"I love data science\", \n",
    "\"A data scientist is often smarter than a data analyst\"]\n",
    "\n",
    "\\* **As a preprocessing step, all letters are usually made lowercase, sometimes stemming/lemmatization are performed, as well as stop-words/punctuation removals, but that's not obligatory.**\n",
    "\n",
    "Suppose our tokens are simple unigrams (words), therefore there are **11 unique words**: {i, love, data, science, a, scientist, is, often, smarter, than, analyst}\n",
    "\n",
    "Then, our corpus is mapped to feature vectors $T_1=(1,1,1,1,0,0,0,0,0,0,0)$, $T_2=(0,0,2,0,2,1,1,1,1,1,1)$\n",
    "\n",
    "|Text #|i|love|data|science|a|scientist|is  |often|smarter|than|analyst|\n",
    "|------|------|------|------|------|------|------|------|------|------|------|------|\n",
    "|$T_1$|1|1|1|1|0|0|0|0|0|0|0|\n",
    "|$T_2$|0|0|2|0|2|1|1|1|1|1|1|\n",
    "\n",
    "If n == 20000, storing X as an array of type float32 would require 20000 x 100000 x 4 bytes = **8GB in RAM** which is barely manageable on today’s computers.\n",
    "\n",
    "Fortunately, **most values in X will be zeros** since for a given document less than a couple thousands (or even hundreds) of distinct words will be used. For this reason we say that bags of words are **typically high-dimensional sparse datasets**. We can save a lot of memory by only storing the non-zero parts of the feature vectors in memory.\n",
    "Sparse matrices are data structures that do exactly this, and scikit-learn has built-in support for these structures.\n",
    "\n",
    "<h2 style=\"font-size:20px; font-family:Verdana; color: #003300\" align=\"left\"> PROS: </h2>\n",
    "* Very intuitive approach, easy to use, understand and apply - you can code it yourself\n",
    "* Built-in support in many scientific/NLP libraries\n",
    "* Memory-efficient sparse format, acceptable by most algorithms \n",
    "* Despite its simplicity, works well, good results could be reached\n",
    "* Fast preprocessing, even on 1 core\n",
    "\n",
    "<h2 style=\"font-size:20px; font-family:Verdana; color: #680000\" align=\"left\"> CONS: </h2>\n",
    "* Huge corpus usually leads to huge vocabulary size (millions of words), even sparse format wouldn't help you (only hashing tricks)\n",
    "* There are other approaches, manageable to catch more details (semantics, relations, structure) - word embeddings etc.\n",
    "* A bag of words is an orderless representation: throwing out spatial relationships between features leads to the fact that simplified model cannot let us to distinguish between sentences, built from the same words while having opposite meanings:\n",
    "<br>\"New episodes **don't** feel like the first - watch it!\" (positive)\n",
    "<br>\"New episodes feel like the first - **don't** watch it!\" (negative)\n",
    "<br>**However, it is somehow treated by increasing the \"length\" of the token (unigrams $\\rightarrow$ bigrams, n-grams etc.), gluing negative particles with next word (not like $\\rightarrow$ not_like), using character n-grams, skip-grams etc.** (see [this section for n-grams details](#3_5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 style=\"font-size:16px; font-family:Verdana\">[To the table of contents](#0)</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='3_2'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<hr style=\"border: 1px dashed #ffffff;\" />\n",
    "### 3.2 Capturing Dependencies. N-grams Recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Simple Bag-of-Words(BoW) model, built on simple tokens (unigrams), is too simplified and catch no spatial dependencies.\n",
    "<br>To deal with it and to expand our knowledge, let's briefly recall what a **N-gram** is:\n",
    "<br>N-gram is a sequence of $N$ basic tokens. \n",
    "<br>N-grams can be defined in different ways, based on token definition. ('word', 'character', 'character_wb' etc.)\n",
    "\n",
    "1) **Word n-grams: (to catch more semantics)** \n",
    "* unigrams: 'I love data science' $\\rightarrow$ [i, love, data, science]\n",
    "* bigrams (2-grams): 'I love data science' $\\rightarrow$ [i love, love data, data science]\n",
    "* 3-grams: 'I love data science' $\\rightarrow$ [i love data, love data science]\n",
    "* ...\n",
    "\n",
    "2) **Character n-grams: (allows to catch features like \":)\", deal somehow with misspeled words like \"looong\" etc.)**\n",
    "* 5-grams: 'I love data science' $\\rightarrow$ [\"i lov\", \" love\", ... , \"cienc\", \"ience\"]\n",
    "* ...\n",
    "\n",
    "3) **Character-wb n-grams (n-grams, only in word boundaries):**\n",
    "* 5-grams: 'I love data science' $\\rightarrow$ {\" i \", \" love\", \"love \", ... , \"cienc\", \"ience\"]\n",
    "* ...\n",
    "\n",
    "4) **Skip-n-grams or k-skip-n-grams (both character- and word-based, extends spatial dependencies)**\n",
    "* A sequence of $N$ basic tokens, having distance of $\\leq K$ tokens between them\n",
    "* 1-skip-2-grams: 'I love data science' $\\rightarrow$ [i data, love science]\n",
    "* ...\n",
    " \n",
    "\n",
    "\n",
    "\n",
    "<h2 style=\"font-size:20px; font-family:Verdana; color: #003300\" align=\"left\"> PROS: </h2>\n",
    "\n",
    "The same as in Bag-of-Words + more context can be captured\n",
    "\n",
    "<h2 style=\"font-size:20px; font-family:Verdana; color: #680000\" align=\"left\"> CONS: </h2>\n",
    "\n",
    "Don't forget that with the increase of n-gram range the vocabulary **rapidly grows up**!\n",
    "<br>**|(1,1)-grams| << |(1,2)-grams| << |(1,3)-grams| << ...**\n",
    "<br>where (1,1)-grams = unigrams, (1,2)-grams = unigrams AND bigrams, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 style=\"font-size:16px; font-family:Verdana\">[To the table of contents](#0)</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='3_3'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border: 1px dashed #ffffff;\" />\n",
    "### 3.3  CountVectorizer\n",
    "\n",
    "CountVectorizer in Sklearn implements aforementioned Bag-of-Words approach:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Commonly used parameters:**\n",
    "* **analyzer**={‘word’, ‘char’, ‘char_wb’} - what token to use (word, char-n-grams etc.)\n",
    "* **ngram_range**=(min_n, max_n) - what N to use: say, ngram_range=(1,2) $\\rightarrow$  use both unigrams and bigrams\n",
    "* **stop_words**={‘english’, list_of_words, or None} - whether to filter stop-words or not\n",
    "* **vocabulary**={None, your_own_dictionary} - whether to use given vocabulary or to build it from extracted tokens\n",
    "* **max_features**={N, None} - to build a vocabulary that consider **top-N** terms ordered by term frequency (TF) across the corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Obtained feature matrix X:\n",
      "[[0 1 1 0 0 0]\n",
      " [0 0 0 0 1 1]\n",
      " [1 0 0 0 0 1]\n",
      " [2 0 0 1 0 1]] \n",
      "\n",
      "Dictionary:\n",
      "column index:0, token: like\n",
      "column index:1, token: me\n",
      "column index:2, token: nobody\n",
      "column index:3, token: not\n",
      "column index:4, token: of\n",
      "column index:5, token: text\n",
      "\n",
      "New sentence (transformed):\n",
      "[[1 0 0 0 0 0]] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# usage example\n",
    "\n",
    "# import CountVectorizer from sklearn library\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# create CountVectorizer object\n",
    "cv = CountVectorizer(\n",
    "                    analyzer='word', # token = word\n",
    "                    ngram_range=(1,1), # only unigrams are used, (1,2) - unigrams/bigrams, ..., etc.\n",
    "                    stop_words=['my', 'stop', 'word', 'list'], # or stop_words='english'\n",
    "                    vocabulary=None, # or vocabulary=your_own_dictionary\n",
    "                    max_df=1.0, # don't filter words by their frequency\n",
    "                    max_features=6 # only top-6 words will be used as columns\n",
    "                    )\n",
    "\n",
    "# We'll be using it as an example for the other feature extraction methods\n",
    "# You can use iterables, numpy arrays, pandas DataFrames as an input.\n",
    "texts = [\n",
    "    'nobody can stop me', # \"stop\" will be filtered by stop_words list\n",
    "    'word is a building blocks of a text', # \"word\" will be filtered by stop_words list\n",
    "    'I like doing feature extraction on text',\n",
    "    'I do not like digits in text like 12345'\n",
    "    ]\n",
    "\n",
    "# apply CountVectorizer to text corpus\n",
    "transformed_texts_cv = cv.fit_transform(texts)\n",
    "# convert sparse representation of transformed texts to dense format and explore it\n",
    "print('Obtained feature matrix X:')\n",
    "print(transformed_texts_cv.todense(), '\\n')\n",
    "\n",
    "# print dictionary (sorted by column index) to see mapping between indices/columns and words \n",
    "print('Dictionary:')\n",
    "for k,v in sorted(cv.vocabulary_.items(), reverse=False):\n",
    "    print('column index:{}, token: {}'.format(v,k))\n",
    "    \n",
    "# transform new sentences (having CountVectorizer trained)\n",
    "new_text = ['i like feature extraction very much'] \n",
    "new_transformed = cv.transform(new_text)\n",
    "# some words, like \"very\" and \"much\", were not used to build the dictionary, thus, they will be skipped\n",
    "print('\\nNew sentence (transformed):')\n",
    "print(new_transformed.todense(), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More details about CountVectorizer in Sklearn: <br> http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test yourself with an exercise:\n",
    "\n",
    "Given a vocabulary, use **CountVectorizer** object to make Bag-of-Words representation of the following sentences:\n",
    "* **\"Sometimes you eat the bear ... and sometimes the bear eats you\"**\n",
    "* **\"This is wrong with your brain: On the left side, there is nothing right, and on the right side, there is nothing left\"**\n",
    "\n",
    "What is **the sum** of obtained feature matrix $F$?\n",
    "<br>Hint! use **np.sum(F)** to calculate it, F is your feature matrix, or directly call **F.sum()**, as its type is np.array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# text to process\n",
    "texts = [\"Sometimes you eat the bear ... and sometimes the bear eats you\",\n",
    "\"This is wrong with your brain: On the left side, there is nothing right, \\\n",
    " and on the right side, there is nothing left\"\n",
    "]\n",
    "\n",
    "# vocabulary to use\n",
    "vocab = {'sometimes':0, 'bear':1, 'left':2, 'right':3, 'and':4, 'side':5, 'eat':6}\n",
    "\n",
    "# your code is here\n",
    "\n",
    "# 1. create CountVectorizer object with given vocabulary\n",
    "# cv = ...\n",
    "\n",
    "# 2. obtain feature matrix F\n",
    "# F = ...\n",
    "\n",
    "# 3. print out the sum of the elements of F\n",
    "# print(...)\n",
    "\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "  <summary>Click to see answer</summary>\n",
    "  <img alt=\"Smiley face\" align=\"left\" src=\"http://1.1m.yt/_FT_6m0.png\">\n",
    "  <p align='left'>Are you sure you tried to solve it on your own?</p>\n",
    "      <pre>\n",
    "          <code>\n",
    "              1. cv = CountVectorizer(vocabulary=vocab)\n",
    "              2. F = cv.fit_transform(texts)\n",
    "              3. print(F.sum()) or print(np.sum(F))\n",
    "          </code>\n",
    "      </pre>\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 style=\"font-size:16px; font-family:Verdana\">[To the table of contents](#0)</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='3_4'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border: 1px dashed #ffffff;\" />\n",
    "### 3.4 TF-IDF Augmentation. TfIdfVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In TF-IDF approach (term frequency - inverse document frequency), in addition to usual BoW-model, the following augmentation is made:\n",
    "<ul>\n",
    "    <li> The text can be analyzed without taking into account the word/token order </li>\n",
    "    <li> We only need to know what words/tokens the text consists of and how many times we met them </li>\n",
    "    <li> The more often a word/token appears in a document, the more important it is </li>\n",
    "    <li> **If a word/token appears in a document, but rarely appears in other documents - it is important and vice versa: <br>if its commonly across most documents - then we cannot rely on this word to help us distinquish between texts** </li>\n",
    "</ul>\n",
    "\n",
    "Thus, we are looking on the whole corpus, usual word counters (term frequencies, TF) are weighted by IDF multiplier:\n",
    "\n",
    "$$  \n",
    "    \\begin{cases} TF(w,T)=n_{Tw} \\\\ IDF(w, T)= log{\\frac{N}{n_{w}}}\\end{cases} \\implies \n",
    "    TF\\text{-}IDF(w, T) = n_{Tw}\\ log{\\frac{N}{n_{w}}} \\ \\ \\ \\ \\forall w \\in W\n",
    "$$\n",
    "\n",
    "<br> where $T$ corresponds to current document (text), \n",
    "<br>$w$ - selected word in document T, \n",
    "<br>$n_{Tw}$ - number of occurences of $w$ in text $T$, \n",
    "<br>$n_{w}$ - number of documents, containing word $w$, \n",
    "<br> $N$ - total number of documents in a corpus.\n",
    "\n",
    "$$\\lim_{n_{w} \\to N} {TF\\text{-}IDF(w, T)} = 0 $$\n",
    "\n",
    "\n",
    "**Commonly used parameters:**\n",
    "* **analyzer**={‘word’, ‘char’, ‘char_wb’} - what token to use (word, char-n-grams etc.)\n",
    "* **ngram_range**=(min_n, max_n) - what N to use: say, ngram_range=(1,2) $\\rightarrow$  use both unigrams and bigrams\n",
    "* **stop_words**={‘english’, list_of_words, or None} (default) - whether to filter stop-words or not\n",
    "* **vocabulary**={None, your_own_dictionary} - whether to use given vocabulary or to build it from exracted tokens\n",
    "* **max_features**={N, None} - to build a vocabulary that only consider the top N ordered by term frequency across the corpus\n",
    "* **norm**={‘l1’, ‘l2’ or None, optional} - norm feature vector to unit norm ($L_2-$, $L_1-$ norms)\n",
    "* **smooth_idf**={True, False} Smooth idf weights by adding one to document frequencies, as if an extra document was seen containing every term in the collection exactly once. Prevents zero divisions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Obtained feature matrix X (see, L2-norm is used):\n",
      "[[ 0.          0.70710678  0.70710678  0.          0.          0.        ]\n",
      " [ 0.          0.          0.          0.          0.84292635  0.53802897]\n",
      " [ 0.77722116  0.          0.          0.          0.          0.62922751]\n",
      " [ 0.79909272  0.          0.          0.5067739   0.          0.32346721]] \n",
      "\n",
      "Dictionary:\n",
      "column index:0, token: like\n",
      "column index:1, token: me\n",
      "column index:2, token: nobody\n",
      "column index:3, token: not\n",
      "column index:4, token: of\n",
      "column index:5, token: text\n",
      "\n",
      "New sentence (transformed):\n",
      "[[ 1.  0.  0.  0.  0.  0.]] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# usage example\n",
    "\n",
    "# import TfidfVectorizer from sklearn library\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# create TfidfVectorizer object\n",
    "tv = TfidfVectorizer(\n",
    "                    analyzer='word', # token = word\n",
    "                    ngram_range=(1,1), # only unigrams are used, (1,2) - unigrams/bigrams, ..., etc.\n",
    "                    stop_words=['my', 'stop', 'word', 'list'], # or stop_words='english'\n",
    "                    vocabulary=None, # or vocabulary=your_own_dictionary\n",
    "                    max_df=1.0, # don't filter words by their frequency\n",
    "                    max_features=6, # only top-6 words will be used as columns,\n",
    "                    smooth_idf=True,\n",
    "                    norm='l2' # euclidean norm is used by default\n",
    "                    )\n",
    "\n",
    "# We'll be using it as an example for the other feature extraction methods\n",
    "# You can use iterables, numpy arrays, pandas DataFrames as an input.\n",
    "texts = [\n",
    "    'nobody can stop me', # \"stop\" will be filtered by stop_words list\n",
    "    'A word is a building blocks of a text', # \"word\" will be filtered by stop_words list\n",
    "    'I like doing feature extraction on text',\n",
    "    'I do not like digits in text like 12345'\n",
    "    ]\n",
    "\n",
    "# apply TfidfVectorizer to text corpus\n",
    "transformed_texts_tv = tv.fit_transform(texts)\n",
    "# convert sparse representation of transformed texts to dense format and explore it\n",
    "print('Obtained feature matrix X (see, L2-norm is used):')\n",
    "print(transformed_texts_tv.todense(), '\\n')\n",
    "\n",
    "# print dictionary (sorted by column index) to see mapping between indices/columns and words \n",
    "print('Dictionary:')\n",
    "for k,v in sorted(tv.vocabulary_.items(), reverse=False):\n",
    "    print('column index:{}, token: {}'.format(v,k))\n",
    "    \n",
    "# transform new sentences (having TfidfVectorizer trained)\n",
    "new_text = ['i like extraction very much'] \n",
    "new_transformed = tv.transform(new_text)\n",
    "# \"very\", \"much\" etc. were not used to build the dictionary, thus, they will be skipped\n",
    "print('\\nNew sentence (transformed):')\n",
    "print(new_transformed.todense(), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More details about TfIdfVectorizer in Sklearn: <br> http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 style=\"font-size:16px; font-family:Verdana\">[To the table of contents](#0)</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='3_5'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border: 1px dashed #ffffff;\" />\n",
    "\n",
    "### 3.5 Hashes. HashingVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A hash function is any function that **can be used to map data of arbitrary size to data of fixed size**. \n",
    "<br>The values returned by a hash function are called hash values, hash codes, or simply hashes.\n",
    "<br>$f(X) \\rightarrow \\{0,N-1\\}:\\ f(X) = X\\  mod\\ N$, function, that maps input into a set of $N$ \"buckets\", is an example of a hash function:\n",
    "\n",
    "Say, $N = 2^k = 2^3 = 8$, then $\\ f(15)=15\\ mod \\ 8 = 7,\\ f(9)=9\\ mod \\ 8 = 1,\\ ...$\n",
    "\n",
    "This vectorizer implementation uses the hashing trick to find the mapping of **token string name** to **feature integer index**.\n",
    "\n",
    "<h2 style=\"font-size:20px; font-family:Verdana; color: #003300\" align=\"left\"> PROS: </h2>\n",
    "\n",
    "* **Very memory-scalable to large datasets** as there is no need to store a vocabulary dictionary in memory\n",
    "* Fast to serialize/deserialize as it holds no state besides the constructor parameters\n",
    "* Can be used in a streaming (partial fit) and/or be parallelized as there is no state computed during fit\n",
    "* Can be used as a \"silly\" dimensionality reduction\n",
    "\n",
    "<h2 style=\"font-size:20px; font-family:Verdana; color: #680000\" align=\"left\"> CONS (vs Vectorizers with in-memory vocabulary): </h2>\n",
    "\n",
    "* There is no way to compute the inverse transform (to get from feature indices to string feature names) <br> which **can be a problem when trying to introspect which features are most important to a model**.\n",
    "* There can be **collisions**: distinct tokens can be mapped to the same \"bucket\" (feature index). \n",
    "<br>However, in practice this is rarely an issue if number of bins is large enough (e.g. $2^{18}$ for text classification problems)\n",
    "\n",
    "\n",
    "\\* The hash function used is the signed 32-bit version of Murmurhash3 (for those, who are really interested :)  )\n",
    "\n",
    "**Commonly used parameters:**\n",
    "* **analyzer**={‘word’, ‘char’, ‘char_wb’} - what token to use (word, char-n-grams etc.)\n",
    "* **ngram_range**=(min_n, max_n) - what N to use: say, ngram_range=(1,2) $\\rightarrow$  use both unigrams and bigrams\n",
    "* **stop_words**={‘english’, list_of_words, or None} (default) - whether to filter stop-words or not\n",
    "* **n_features**={N} - how many \"buckets\" to use\n",
    "* **norm**={‘l1’, ‘l2’ or None, optional} - norm feature vector to unit norm ($L_2-$, $L_1-$ norms)\n",
    "* **non_negative**={True,False} whether to use non-negative values only (othervise, they will be centered around 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Obtained feature matrix X (see, no norm is used):\n",
      "[[ 0.  0.  0.  0.  0.  1.]\n",
      " [ 1.  1.  1.  1.  1.  0.]\n",
      " [ 1.  1.  0.  1.  0.  1.]\n",
      " [ 1.  0.  0.  0.  1.  0.]] \n",
      "\n",
      "Dictionary:\n",
      "Oops, Hashing trick assumes no vocabulary will be used at all, online learning :)\n",
      "However, we won't be able to do reverse transform and to get exact words :( \n",
      "\n",
      "New sentence (transformed):\n",
      "[[ 0.  1.  1.  0.  1.  1.]] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# usage example\n",
    "\n",
    "# import HashingVectorizer from sklearn library\n",
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "\n",
    "# create HashingVectorizer object\n",
    "hv = HashingVectorizer(\n",
    "                    analyzer='word', # token = word\n",
    "                    ngram_range=(1,1), # only unigrams are used, (1,2) - unigrams/bigrams, ..., etc.\n",
    "                    stop_words=['my', 'stop', 'word', 'list'], # or stop_words='english'\n",
    "                    n_features=6, # only 6 bins will be used as columns, high probability of collisions!\n",
    "                    non_negative=True, # only non-negative values (othervise, it will be centered around 0)\n",
    "                    norm=None\n",
    "                    )\n",
    "\n",
    "# We'll be using it as an example for the other feature extraction methods\n",
    "# You can use iterables, numpy arrays, pandas DataFrames as an input.\n",
    "texts = [\n",
    "    'nobody can stop me', # \"stop\" will be filtered by stop_words list\n",
    "    'word is a building blocks of a text', # \"word\" will be filtered by stop_words list\n",
    "    'I like doing feature extraction on text',\n",
    "    'I do not like digits in text like 12345'\n",
    "    ]\n",
    "\n",
    "# apply HashingVectorizer to text corpus\n",
    "transformed_texts_hv = hv.fit_transform(texts)\n",
    "# convert sparse representation of transformed texts to dense format and explore it\n",
    "print('Obtained feature matrix X (see, no norm is used):')\n",
    "print(transformed_texts_hv.todense(), '\\n')\n",
    "\n",
    "# no dictionary ...\n",
    "print('Dictionary:')\n",
    "print('Oops, Hashing trick assumes no vocabulary will be used at all, online learning :)')\n",
    "print(\"However, we won't be able to do reverse transform and to get exact words :( \")\n",
    "    \n",
    "# transform new sentences (having HashingVectorizer trained)\n",
    "new_text = ['i like extraction very much'] \n",
    "new_transformed = hv.transform(new_text)\n",
    "# \"very\", \"much\" etc. were not used to build the dictionary, thus, they will be skipped\n",
    "print('\\nNew sentence (transformed):')\n",
    "print(new_transformed.todense(), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More details about HashingVectorizer in Sklearn: <br> http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 style=\"font-size:16px; font-family:Verdana\">[To the table of contents](#0)</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='4'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height: 1px; background-color: #808080 border: dashed 1px\">\n",
    "## 4. Going Beyond: Custom Features\n",
    "\n",
    "Usually, specific domain leads to specific information, hidden inside of your data. \n",
    "<br>You need to extract it, as much as possible. \n",
    "\n",
    "This is often called **Feature Engineering**.\n",
    "<br>For example, if we want to run sentiment analysis (classification task) on the IMDB dataset (movie reviews) and it seems to us that **many reviews may contain explicit marks (say, in a form of x/xx)**, than we should check this out and extract useful custom feature:\n",
    "\n",
    "[\"Average film, however, starring Matt Damon, 8/10\", 1] $\\rightarrow$ {\"8/10\"} $\\rightarrow$ 8/10=0.8 ~ 1 $\\rightarrow$ review is positive\n",
    "<br>[\"2/10, there is nothing to add\", 0] $\\rightarrow$ {\"2/10\"} $\\rightarrow$ 2/10=0.2 ~ 0 $\\rightarrow$ review is negative.\n",
    "\n",
    "<br>However, be aware of dates and outliers (in relation to this particular feature) or whatever else - always check your code / regular expressions:\n",
    "<br>Say, incorrect parsing of **'01/10/1999'** would lead to **{1/10, 10/1999} or {1/10}  ~ 0 (negative review?!)** errors.\n",
    "\n",
    "### Hereinafter, we'll discuss domain specific features, they are no panacea in general."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='4_1'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border: 1px dashed #ffffff;\" />\n",
    "### 4.1 Token-based Level\n",
    "\n",
    "We need to look on tokens (words, entities like smiles etc.) and try to extract meaningful features\n",
    "\n",
    "* positive smiles\n",
    "* negative smiles\n",
    "* explicit rating (marks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# to separate import from actual code\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from textblob import Word, TextBlob\n",
    "import re # for regular expressions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# this implemenation does not deal with aforementioned cases, \n",
    "# to extract rating \"candidates\" in a text s\n",
    "def get_rate(s):\n",
    "    candidates = re.findall(r'(\\d{1,3}[\\\\|/]{1}\\d{1,2})', s)\n",
    "    rates = []\n",
    "    for c in candidates:\n",
    "        try:\n",
    "            rates.append(eval(c)) # by the way, \"eval\" is a prime evil, it may lead you to the dark side :)\n",
    "            # instead, say, install sympy\n",
    "            # from sympy import sympify\n",
    "            # sympify(\"1*5/6*(7+8)\").evalf()\n",
    "        except SyntaxError:\n",
    "            pass\n",
    "        except ZeroDivisionError:\n",
    "            return 0\n",
    "    return np.mean(rates)\n",
    "\n",
    "# lists of positive/negative smiles\n",
    "positive_smiles = set([\n",
    "\":‑)\",\":)\",\":-]\",\":]\",\":-3\",\":3\",\":->\",\":>\",\"8-)\",\"8)\",\":-}\",\":}\",\":o)\",\":c)\",\":^)\",\"=]\",\"=)\",\":‑D\",\":D\",\"8‑D\",\"8D\",\n",
    "\"x‑D\",\"xD\",\"X‑D\",\"XD\",\"=D\",\"=3\",\"B^D\",\":-))\",\";‑)\",\";)\",\"*-)\",\"*)\",\";‑]\",\";]\",\";^)\",\":‑,\",\";D\",\":‑P\",\":P\",\"X‑P\",\"XP\",\n",
    "\"x‑p\",\"xp\",\":‑p\",\":p\",\":‑Þ\",\":Þ\",\":‑þ\",\":þ\",\":‑b\",\":b\",\"d:\",\"=p\",\">:P\", \":'‑)\", \":')\",  \":-*\", \":*\", \":×\"\n",
    "])\n",
    "negative_smiles = set([\n",
    "\":‑(\",\":(\",\":‑c\",\":c\",\":‑<\",\":<\",\":‑[\",\":[\",\":-||\",\">:[\",\":{\",\":@\",\">:(\",\"D‑':\",\"D:<\",\"D:\",\"D8\",\"D;\",\"D=\",\"DX\",\":‑/\",\n",
    "\":/\",\":‑.\",'>:\\\\', \">:/\", \":\\\\\", \"=/\" ,\"=\\\\\", \":L\", \"=L\",\":S\",\":‑|\",\":|\",\"|‑O\",\"<:‑|\"\n",
    "])\n",
    "\n",
    "def get_token_level_features(text):\n",
    "    # assume text = pd.Series with review text\n",
    "    print('extracting token-level features...')\n",
    "    tdf = pd.DataFrame()\n",
    "    tdf['text'] = text # this is our review\n",
    "    \n",
    "    # try to extract rating :) like \"great film. 9/10\" will yield 0.9\n",
    "    tdf['rating'] = tdf['text'].apply(get_rate).fillna(-1) # rating (if found in review)\n",
    "\n",
    "    # try to extract smiles and count positive/negative smiles per review\n",
    "    tdf['positive_smiles'] = tdf.text.apply(lambda s: len([x for x in s.split() if x in positive_smiles]))\n",
    "    tdf['negative_smiles'] = tdf.text.apply(lambda s: len([x for x in s.split() if x in negative_smiles]))\n",
    "    \n",
    "    return tdf\n",
    "    # from scipy.sparse import csr_matrix # to get sparse representation of feature matrix\n",
    "    # return csr_matrix(tdf[tdf.columns[2:]].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 style=\"font-size:16px; font-family:Verdana\">[To the table of contents](#0)</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='4_2'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border: 1px dashed #ffffff;\" />\n",
    "### 4.2 Sentence-based / Text-based Level\n",
    "\n",
    "We moved up to sentence/text level.\n",
    "<br><i>Someone can argue about level of these features, but let us just put them here</i>\n",
    "<br>Let's see what features we can search for:\n",
    "* **Sentence count** (text must be split into sentences, then extract length of obtained list) \n",
    "* **Exclamation marks count** (integer) or presence (boolean) - catching stress, expecially if we use probabilistic output instead of binary classification\n",
    "* **Question marks count** (integer) or presence (boolean) - can sometimes help in catching sarcasm\n",
    "* **Uppercase word count** (of length > 1, to omit \"A\"s) - stress of a text, expecially if we use probabilistic output instead of binary classification\n",
    "* **Contrast conjugations**, like {'instead','nevertheless','on the contrary','on the other hand'} - to catch possible changes of a sentiment\n",
    "\n",
    "Some information regarding text \"edges\" - first/last sentences in a review:\n",
    "* **\"polarity\" of first/last sentence[s]**\n",
    "* **\"subjectivity\" of first/last sentence[s]**\n",
    "* **\"purity\" of first/last sentence[s] or the whole set of sentences** - to catch a change of a sentiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# let's continue...\n",
    "\n",
    "# contrast conjugations\n",
    "contrast_conj = set([\n",
    "'alternatively','anyway','but','by contrast','differ from','elsewhere','even so','however','in contrast','in fact',\n",
    "'in other respects','in spite of','in that respect','instead','nevertheless','on the contrary','on the other hand',\n",
    "'rather','though','whereas','yet'])\n",
    "\n",
    "# to get review \"purity\" ~ same sentiment over review (~1) or changing (~0)\n",
    "def purity(sentences):\n",
    "    # obtain polarities across the sentences\n",
    "    polarities = np.array([TextBlob(x).sentiment.polarity for x in sentences])\n",
    "    return polarities.sum() / np.abs(polarities).sum()\n",
    "\n",
    "# uppercase pattern, quick-n-dirty implementation\n",
    "uppercase_pattern = re.compile(r'(\\b[0-9]*[A-Z]+[0-9]*[A-Z]+[0-9]*\\b)')\n",
    "\n",
    "# regular expression to split review on sentences\n",
    "sentence_splitter = re.compile('(?<!\\w\\.\\w.)(?<![A-Z][a-z]\\.)(?<![A-Z]\\.)(?<=\\!|\\?|\\.)\\s')\n",
    "# you can https://regex101.com/ for regex creation/checking, very convenient\n",
    "\n",
    "# feature engineering\n",
    "def get_text_level_features(text):\n",
    "    # assume text = pd.Series with review text\n",
    "    print('extracting text-level features...')\n",
    "    tdf = pd.DataFrame()\n",
    "    tdf['text'] = text # this is our review\n",
    "    tdf['sentences'] = tdf.text.apply(lambda s: re.split(sentence_splitter, s)) # split it into sentences\n",
    "    \n",
    "    tdf['sentence_cnt'] = tdf['sentences'].apply(len) # sentence count\n",
    "    tdf['exclamation_cnt'] = tdf.text.str.count('\\!') # exclamation mark count\n",
    "    tdf['question_cnt'] = tdf.text.str.count('\\?') # question mark count\n",
    "    \n",
    "    # uppercase words cnt (like HOLY JESUS!)\n",
    "    tdf['upper_word_cnt'] = tdf.text.apply(lambda s: len(re.findall(uppercase_pattern, s)))\n",
    "    \n",
    "    # extract smiles and count positive/negative smiles per review\n",
    "    tdf['positive_smiles'] = tdf.text.apply(lambda s: len([x for x in s.split() if x in positive_smiles]))\n",
    "    tdf['negative_smiles'] = tdf.text.apply(lambda s: len([x for x in s.split() if x in negative_smiles]))\n",
    "    \n",
    "    # not so informative, but still - contrast conjugations\n",
    "    tdf['contrast_conj_cnt'] = tdf.text.apply(lambda s: len([c for c in contrast_conj if c in s]))\n",
    "    \n",
    "    # polarity of 1st sentence\n",
    "    tdf['polarity_1st_sent'] = tdf.sentences.apply(lambda s: TextBlob(s[0]).sentiment.polarity)\n",
    "    # subjectivity of 1st sentence\n",
    "    tdf['subjectivity_1st_sent'] = tdf.sentences.apply(lambda s: TextBlob(s[0]).sentiment.subjectivity)\n",
    "    # polarity of last sentence\n",
    "    tdf['polarity_last_sent'] = tdf.sentences.apply(lambda s: TextBlob(s[-1]).sentiment.polarity)\n",
    "    # subjectivity of last sentence\n",
    "    tdf['subjectivity_last_sent'] = tdf.sentences.apply(lambda s: TextBlob(s[-1]).sentiment.subjectivity)\n",
    "    # subjectivity of review itself\n",
    "    tdf['polarity'] = tdf.text.apply(lambda s: TextBlob(s[-1]).sentiment.polarity)\n",
    "    # \"purity\" of review, |sum(sentence polarity) / sum(|sentence polarity|))|, ~ 1 is better, ~ 0 -> mixed\n",
    "    tdf['purity'] = tdf.sentences.apply(purity)\n",
    "    tdf['purity'].fillna(0, inplace=True)\n",
    "    \n",
    "    return tdf\n",
    "    # from scipy.sparse import csr_matrix\n",
    "    # return csr_matrix(tdf[tdf.columns[2:]].values) # to get sparse representation of feature matrix\n",
    "\n",
    "### BE CAREFUL, if you use LINEAR MODELS and have MOSTLY SHORT REVIEWS (1 sentence), then\n",
    "### tdf['subjectivity_1st_sent'] ~ tdf['subjectivity_last_sent'], two same columns, leads to multicollinearity!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "extracting token-level features...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\lib\\site-packages\\numpy\\core\\fromnumeric.py:2889: RuntimeWarning: Mean of empty slice.\n",
      "  out=out, **kwargs)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>rating</th>\n",
       "      <th>positive_smiles</th>\n",
       "      <th>negative_smiles</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Waste of time :( 2/10 for the plot and 4/10 fo...</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Awful film! Nobody can like it</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Wow! Am I impressed?? TOTALLY :D</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7/10</td>\n",
       "      <td>0.7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  rating  positive_smiles  \\\n",
       "0  Waste of time :( 2/10 for the plot and 4/10 fo...     0.3                0   \n",
       "1                     Awful film! Nobody can like it    -1.0                0   \n",
       "2                   Wow! Am I impressed?? TOTALLY :D    -1.0                1   \n",
       "3                                               7/10     0.7                0   \n",
       "\n",
       "   negative_smiles  \n",
       "0                1  \n",
       "1                0  \n",
       "2                0  \n",
       "3                0  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's test custom features:\n",
    "\n",
    "reviews = [\n",
    "    \"Waste of time :( 2/10 for the plot and 4/10 for acting!\",\n",
    "    'Awful film! Nobody can like it',\n",
    "    'Wow! Am I impressed?? TOTALLY :D',\n",
    "    '7/10'\n",
    "]\n",
    "\n",
    "# token-based\n",
    "token_lf = get_token_level_features(reviews)\n",
    "token_lf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "extracting text-level features...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentences</th>\n",
       "      <th>sentence_cnt</th>\n",
       "      <th>exclamation_cnt</th>\n",
       "      <th>question_cnt</th>\n",
       "      <th>upper_word_cnt</th>\n",
       "      <th>positive_smiles</th>\n",
       "      <th>negative_smiles</th>\n",
       "      <th>contrast_conj_cnt</th>\n",
       "      <th>polarity_1st_sent</th>\n",
       "      <th>subjectivity_1st_sent</th>\n",
       "      <th>polarity_last_sent</th>\n",
       "      <th>subjectivity_last_sent</th>\n",
       "      <th>polarity</th>\n",
       "      <th>purity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Waste of time :( 2/10 for the plot and 4/10 fo...</td>\n",
       "      <td>[Waste of time :( 2/10 for the plot and 4/10 f...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.316667</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>-0.316667</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Awful film! Nobody can like it</td>\n",
       "      <td>[Awful film!, Nobody can like it]</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Wow! Am I impressed?? TOTALLY :D</td>\n",
       "      <td>[Wow!, Am I impressed??, TOTALLY :D]</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.125000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.875000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7/10</td>\n",
       "      <td>[7/10]</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  \\\n",
       "0  Waste of time :( 2/10 for the plot and 4/10 fo...   \n",
       "1                     Awful film! Nobody can like it   \n",
       "2                   Wow! Am I impressed?? TOTALLY :D   \n",
       "3                                               7/10   \n",
       "\n",
       "                                           sentences  sentence_cnt  \\\n",
       "0  [Waste of time :( 2/10 for the plot and 4/10 f...             1   \n",
       "1                  [Awful film!, Nobody can like it]             2   \n",
       "2               [Wow!, Am I impressed??, TOTALLY :D]             3   \n",
       "3                                             [7/10]             1   \n",
       "\n",
       "   exclamation_cnt  question_cnt  upper_word_cnt  positive_smiles  \\\n",
       "0                1             0               0                0   \n",
       "1                1             0               0                0   \n",
       "2                1             2               1                1   \n",
       "3                0             0               0                0   \n",
       "\n",
       "   negative_smiles  contrast_conj_cnt  polarity_1st_sent  \\\n",
       "0                1                  0          -0.316667   \n",
       "1                0                  0          -1.000000   \n",
       "2                0                  0           0.125000   \n",
       "3                0                  0           0.000000   \n",
       "\n",
       "   subjectivity_1st_sent  polarity_last_sent  subjectivity_last_sent  \\\n",
       "0               0.333333           -0.316667                0.333333   \n",
       "1               1.000000            0.000000                0.000000   \n",
       "2               1.000000            0.500000                0.875000   \n",
       "3               0.000000            0.000000                0.000000   \n",
       "\n",
       "   polarity  purity  \n",
       "0       0.0    -1.0  \n",
       "1       0.0    -1.0  \n",
       "2       0.0     1.0  \n",
       "3       0.0     0.0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# token-based\n",
    "token_lf = get_text_level_features(reviews)\n",
    "token_lf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 style=\"font-size:16px; font-family:Verdana\">[To the table of contents](#0)</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='4_3'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border: 1px dashed #ffffff;\" />\n",
    "### 4.3 Combining Features: FeatureUnion / Pipeline\n",
    "\n",
    "Sometimes, we need to \"glue\" different feature \"bricks\" into one feature matrix $X$\n",
    "<br>Say, you want to use both auto-generated features (like bag-of-words) and your custom features\n",
    "<br> Sklearn has a built-in support - **FeatureUnion**\n",
    "\n",
    "However, there are several requirements: all this feature blocks must be wrapped with a class that implements **.fit() and .transform()** methods. \n",
    "<br>You can achieve this by deriving a class from **BaseTransformer**, \n",
    "<br>or by writing a custom function and passing it to **FunctionTransformer**:\n",
    "(we' ll use second approach later)\n",
    "\n",
    "**Commonly used parameters:**\n",
    "* **transformer_list** - list of features to combine, in a form of  [('feature_name1', features1), ('feature_name2', features2), ...]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X:\n",
      "[[1 0 0]\n",
      " [0 1 0]\n",
      " [0 0 1]]\n",
      "FeatureUnion(n_jobs=1,\n",
      "       transformer_list=[('f1', FunctionTransformer(accept_sparse=True,\n",
      "          func=<function features at 0x00000217A78C6158>, inv_kw_args=None,\n",
      "          inverse_func=None, kw_args=None, pass_y=False, validate=False)), ('f2', FunctionTransformer(accept_sparse=True,\n",
      "          func=<function features2 at 0x00000217A78B5A60>,\n",
      "          inv_kw_args=None, inverse_func=None, kw_args=None, pass_y=False,\n",
      "          validate=False))],\n",
      "       transformer_weights=None)\n",
      "\n",
      "United Features:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 0, 3, 1, 1],\n",
       "       [0, 1, 0, 1, 3, 1],\n",
       "       [0, 0, 1, 1, 1, 3]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# toy example \n",
    "\n",
    "from sklearn.pipeline import FeatureUnion, Pipeline\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "\n",
    "# the good news - CountVectorizer / TfIdfVectorizer / etc. are transformers itself\n",
    "# you won't have to perform manipulations, listed below\n",
    "\n",
    "# wrapper functions, you can use any function that return desired values in (numpy.array, list or pandas.DataFrame)\n",
    "def features(X):\n",
    "    return X\n",
    "\n",
    "def features2(X):\n",
    "    # let's make small changes to our X\n",
    "    return 1+ X*2 # be careful, \"+\", \"*\" are element-wise operations in numpy\n",
    "\n",
    "features1 = FunctionTransformer(func=features,\n",
    "                                validate=False, # to silence many warnings\n",
    "                                accept_sparse=True # to use convenient sparse representations\n",
    "                                )\n",
    "\n",
    "features2 = FunctionTransformer(func=features2,\n",
    "                                validate=False, # to silence many warnings\n",
    "                                accept_sparse=True # to use convenient sparse representations\n",
    "                                )\n",
    "\n",
    "# toy data\n",
    "X = np.array([[1,0,0], [0,1,0], [0,0,1]])\n",
    "print('X:\\n{}'.format(X))\n",
    "\n",
    "# create FeatureUnion object\n",
    "combined_features = FeatureUnion(\n",
    "    [\n",
    "        ('f1', features1),\n",
    "        ('f2', features2)\n",
    "    ]\n",
    ")\n",
    "# let's check what it's look like\n",
    "print(combined_features)\n",
    "# let's see if a transformation is correct - yes, it is!\n",
    "print('\\nUnited Features:')\n",
    "combined_features.transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 style=\"font-size:16px; font-family:Verdana\">[To the table of contents](#0)</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://i.ytimg.com/vi/0FHEeG_uq5Y/maxresdefault.jpg\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}