Retrieval-based.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building intelligent bots. Retrieval-based chatbots\n",
    "\n",
    "In this section we build a retrieval-based chatbot with Rasa. Before we go to this point, we go through a few NLP methods and word vectorization.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NLP methods for NLU\n",
    "\n",
    "Let's take one of President Trump's speech and divide into words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">  Thank you, everybody.\n",
      ">  Thank you.\n",
      ">  Thank you very much.\n",
      ">  Thank you, Matt, for that great introduction.\n",
      ">  And thank you for this big crowd.\n",
      ">  This is incredible.\n",
      ">  Really incredible.\n",
      ">  We have all come a long way together.\n",
      ">  We have come a long way together.\n",
      "\n",
      ">  I’m thrilled to be back at CPAC, with so many of my wonderful friends and amazing supporters and proud conservatives.\n",
      ">  Remember when I first started running?\n",
      ">  Because I wasn’t a politician, fortunately, but do you remember I started running and people said, are you sure he’s a conservative?\n",
      ">  I think I proved I’m a conservative.\n",
      "\n",
      ">  For more than four decades, this event has served as a forum for our nation’s top leaders, activists, writers, and thinkers.\n",
      "\n",
      ">  Year after year, leaders have stood on this stage to discuss what we can do together to protect our heritage, to promote our culture, and to defend our freedom.\n",
      ">  CPAC has always been about big ideas, and it has also been about putting those ideas into action — and CPAC really has put a lot of ideas into action.\n",
      ">  We’ll talk about some of them this morning.\n",
      "\n",
      ">  For the last year with your help, we have put more great conservative ideas into use than perhaps ever before in American history.\n",
      ">  What a nice picture that is.\n",
      ">  Look at that.\n",
      ">  I would love to watch that guy speak.\n",
      ">  Oh, boy.\n",
      ">  Oh, I try like hell to hide that bald spot, folks.\n",
      ">  I work hard at it.\n",
      ">  Doesn’t look bad.\n",
      ">  Hey, we’re hanging in.\n",
      ">  We’re hanging in.\n",
      ">  We’re hanging in there, right?\n",
      ">  Together we’re hanging in.\n",
      ">  We have confirmed a record number, so important, of circuit court judges and we’re going to be putting in a lot more.\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "file = open(\"trump.txt\", \"r\",encoding='utf-8') \n",
    "trump = file.read() \n",
    "\n",
    "nlp = spacy.load(\"en\")\n",
    "doc = nlp(trump)\n",
    "\n",
    "for span in doc.sents:\n",
    "    print(\"> \", span)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have are able to divide it using SpaCy and get the part of speech of each word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Thank VERB\n",
      "1 you PRON\n",
      "2 , PUNCT\n",
      "3 everybody PRON\n",
      "4 . PUNCT\n",
      "5 Thank VERB\n",
      "6 you PRON\n",
      "7 . PUNCT\n",
      "8 Thank VERB\n",
      "9 you PRON\n",
      "10 very ADV\n",
      "11 much ADV\n",
      "12 . PUNCT\n",
      "13 Thank VERB\n",
      "14 you PRON\n",
      "15 , PUNCT\n",
      "16 Matt PROPN\n",
      "17 , PUNCT\n",
      "18 for ADP\n",
      "19 that DET\n",
      "20 great ADJ\n",
      "21 introduction NOUN\n",
      "22 . PUNCT\n",
      "23 And CCONJ\n",
      "24 thank VERB\n",
      "25 you PRON\n",
      "26 for ADP\n",
      "27 this DET\n",
      "28 big ADJ\n",
      "29 crowd NOUN\n",
      "30 . PUNCT\n",
      "31 This DET\n",
      "32 is AUX\n",
      "33 incredible ADJ\n",
      "34 . PUNCT\n",
      "35 Really ADV\n",
      "36 incredible ADJ\n",
      "37 . PUNCT\n",
      "38 We PRON\n",
      "39 have AUX\n",
      "40 all ADV\n",
      "41 come VERB\n",
      "42 a DET\n",
      "43 long ADJ\n",
      "44 way NOUN\n",
      "45 together ADV\n",
      "46 . PUNCT\n",
      "47 We PRON\n",
      "48 have AUX\n",
      "49 come VERB\n",
      "50 a DET\n",
      "51 long ADJ\n",
      "52 way NOUN\n",
      "53 together ADV\n",
      "54 . PUNCT\n",
      "55 \n",
      " SPACE\n",
      "56 I PRON\n",
      "57 ’m VERB\n",
      "58 thrilled ADJ\n",
      "59 to PART\n",
      "60 be AUX\n",
      "61 back ADV\n",
      "62 at ADP\n",
      "63 CPAC PROPN\n",
      "64 , PUNCT\n",
      "65 with ADP\n",
      "66 so ADV\n",
      "67 many ADJ\n",
      "68 of ADP\n",
      "69 my PRON\n",
      "70 wonderful ADJ\n",
      "71 friends NOUN\n",
      "72 and CCONJ\n",
      "73 amazing ADJ\n",
      "74 supporters NOUN\n",
      "75 and CCONJ\n",
      "76 proud ADJ\n",
      "77 conservatives NOUN\n",
      "78 . PUNCT\n",
      "79 Remember VERB\n",
      "80 when ADV\n",
      "81 I PRON\n",
      "82 first ADV\n",
      "83 started VERB\n",
      "84 running VERB\n",
      "85 ? PUNCT\n",
      "86 Because SCONJ\n",
      "87 I PRON\n",
      "88 was AUX\n",
      "89 n’t PART\n",
      "90 a DET\n",
      "91 politician NOUN\n",
      "92 , PUNCT\n",
      "93 fortunately ADV\n",
      "94 , PUNCT\n",
      "95 but CCONJ\n",
      "96 do AUX\n",
      "97 you PRON\n",
      "98 remember VERB\n",
      "99 I PRON\n",
      "100 started VERB\n",
      "101 running VERB\n",
      "102 and CCONJ\n",
      "103 people NOUN\n",
      "104 said VERB\n",
      "105 , PUNCT\n",
      "106 are AUX\n",
      "107 you PRON\n",
      "108 sure ADJ\n",
      "109 he PRON\n",
      "110 ’s VERB\n",
      "111 a DET\n",
      "112 conservative ADJ\n",
      "113 ? PUNCT\n",
      "114 I PRON\n",
      "115 think VERB\n",
      "116 I PRON\n",
      "117 proved VERB\n",
      "118 I PRON\n",
      "119 ’m VERB\n",
      "120 a DET\n",
      "121 conservative NOUN\n",
      "122 . PUNCT\n",
      "123 \n",
      " SPACE\n",
      "124 For ADP\n",
      "125 more ADJ\n",
      "126 than SCONJ\n",
      "127 four NUM\n",
      "128 decades NOUN\n",
      "129 , PUNCT\n",
      "130 this DET\n",
      "131 event NOUN\n",
      "132 has AUX\n",
      "133 served VERB\n",
      "134 as SCONJ\n",
      "135 a DET\n",
      "136 forum NOUN\n",
      "137 for ADP\n",
      "138 our PRON\n",
      "139 nation NOUN\n",
      "140 ’s PART\n",
      "141 top ADJ\n",
      "142 leaders NOUN\n",
      "143 , PUNCT\n",
      "144 activists NOUN\n",
      "145 , PUNCT\n",
      "146 writers NOUN\n",
      "147 , PUNCT\n",
      "148 and CCONJ\n",
      "149 thinkers NOUN\n",
      "150 . PUNCT\n",
      "151 \n",
      " SPACE\n",
      "152 Year PROPN\n",
      "153 after ADP\n",
      "154 year NOUN\n",
      "155 , PUNCT\n",
      "156 leaders NOUN\n",
      "157 have AUX\n",
      "158 stood VERB\n",
      "159 on ADP\n",
      "160 this DET\n",
      "161 stage NOUN\n",
      "162 to PART\n",
      "163 discuss VERB\n",
      "164 what PRON\n",
      "165 we PRON\n",
      "166 can AUX\n",
      "167 do AUX\n",
      "168 together ADV\n",
      "169 to PART\n",
      "170 protect VERB\n",
      "171 our PRON\n",
      "172 heritage NOUN\n",
      "173 , PUNCT\n",
      "174 to PART\n",
      "175 promote VERB\n",
      "176 our PRON\n",
      "177 culture NOUN\n",
      "178 , PUNCT\n",
      "179 and CCONJ\n",
      "180 to PART\n",
      "181 defend VERB\n",
      "182 our PRON\n",
      "183 freedom NOUN\n",
      "184 . PUNCT\n",
      "185 CPAC PROPN\n",
      "186 has AUX\n",
      "187 always ADV\n",
      "188 been AUX\n",
      "189 about ADP\n",
      "190 big ADJ\n",
      "191 ideas NOUN\n",
      "192 , PUNCT\n",
      "193 and CCONJ\n",
      "194 it PRON\n",
      "195 has AUX\n",
      "196 also ADV\n",
      "197 been AUX\n",
      "198 about ADP\n",
      "199 putting VERB\n",
      "200 those DET\n",
      "201 ideas NOUN\n",
      "202 into ADP\n",
      "203 action NOUN\n",
      "204 — PUNCT\n",
      "205 and CCONJ\n",
      "206 CPAC PROPN\n",
      "207 really ADV\n",
      "208 has AUX\n",
      "209 put VERB\n",
      "210 a DET\n",
      "211 lot NOUN\n",
      "212 of ADP\n",
      "213 ideas NOUN\n",
      "214 into ADP\n",
      "215 action NOUN\n",
      "216 . PUNCT\n",
      "217 We PRON\n",
      "218 ’ll VERB\n",
      "219 talk VERB\n",
      "220 about ADP\n",
      "221 some DET\n",
      "222 of ADP\n",
      "223 them PRON\n",
      "224 this DET\n",
      "225 morning NOUN\n",
      "226 . PUNCT\n",
      "227 \n",
      " SPACE\n",
      "228 For ADP\n",
      "229 the DET\n",
      "230 last ADJ\n",
      "231 year NOUN\n",
      "232 with ADP\n",
      "233 your PRON\n",
      "234 help NOUN\n",
      "235 , PUNCT\n",
      "236 we PRON\n",
      "237 have AUX\n",
      "238 put VERB\n",
      "239 more ADV\n",
      "240 great ADJ\n",
      "241 conservative ADJ\n",
      "242 ideas NOUN\n",
      "243 into ADP\n",
      "244 use NOUN\n",
      "245 than SCONJ\n",
      "246 perhaps ADV\n",
      "247 ever ADV\n",
      "248 before ADV\n",
      "249 in ADP\n",
      "250 American ADJ\n",
      "251 history NOUN\n",
      "252 . PUNCT\n",
      "253 What PRON\n",
      "254 a DET\n",
      "255 nice ADJ\n",
      "256 picture NOUN\n",
      "257 that PRON\n",
      "258 is AUX\n",
      "259 . PUNCT\n",
      "260 Look VERB\n",
      "261 at ADP\n",
      "262 that DET\n",
      "263 . PUNCT\n",
      "264 I PRON\n",
      "265 would AUX\n",
      "266 love VERB\n",
      "267 to PART\n",
      "268 watch VERB\n",
      "269 that DET\n",
      "270 guy NOUN\n",
      "271 speak VERB\n",
      "272 . PUNCT\n",
      "273 Oh INTJ\n",
      "274 , PUNCT\n",
      "275 boy INTJ\n",
      "276 . PUNCT\n",
      "277 Oh INTJ\n",
      "278 , PUNCT\n",
      "279 I PRON\n",
      "280 try VERB\n",
      "281 like INTJ\n",
      "282 hell NOUN\n",
      "283 to PART\n",
      "284 hide VERB\n",
      "285 that SCONJ\n",
      "286 bald ADJ\n",
      "287 spot NOUN\n",
      "288 , PUNCT\n",
      "289 folks NOUN\n",
      "290 . PUNCT\n",
      "291 I PRON\n",
      "292 work VERB\n",
      "293 hard ADV\n",
      "294 at ADP\n",
      "295 it PRON\n",
      "296 . PUNCT\n",
      "297 Does AUX\n",
      "298 n’t PART\n",
      "299 look VERB\n",
      "300 bad ADJ\n",
      "301 . PUNCT\n",
      "302 Hey INTJ\n",
      "303 , PUNCT\n",
      "304 we PRON\n",
      "305 ’re VERB\n",
      "306 hanging VERB\n",
      "307 in ADP\n",
      "308 . PUNCT\n",
      "309 We PRON\n",
      "310 ’re VERB\n",
      "311 hanging VERB\n",
      "312 in ADP\n",
      "313 . PUNCT\n",
      "314 We PRON\n",
      "315 ’re VERB\n",
      "316 hanging VERB\n",
      "317 in ADV\n",
      "318 there ADV\n",
      "319 , PUNCT\n",
      "320 right ADJ\n",
      "321 ? PUNCT\n",
      "322 Together ADV\n",
      "323 we PRON\n",
      "324 ’re VERB\n",
      "325 hanging VERB\n",
      "326 in ADP\n",
      "327 . PUNCT\n",
      "328 We PRON\n",
      "329 have AUX\n",
      "330 confirmed VERB\n",
      "331 a DET\n",
      "332 record NOUN\n",
      "333 number NOUN\n",
      "334 , PUNCT\n",
      "335 so ADV\n",
      "336 important ADJ\n",
      "337 , PUNCT\n",
      "338 of ADP\n",
      "339 circuit NOUN\n",
      "340 court NOUN\n",
      "341 judges NOUN\n",
      "342 and CCONJ\n",
      "343 we PRON\n",
      "344 ’re VERB\n",
      "345 going VERB\n",
      "346 to PART\n",
      "347 be AUX\n",
      "348 putting VERB\n",
      "349 in ADP\n",
      "350 a DET\n",
      "351 lot NOUN\n",
      "352 more ADJ\n",
      "353 . PUNCT\n"
     ]
    }
   ],
   "source": [
    "for span in doc.sents:\n",
    "    for i in range(span.start, span.end):\n",
    "        token = doc[i]\n",
    "        print(i, token.text, token.pos_)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A smaller example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Broadcasting PROPN\n",
      "today NOUN\n",
      ", PUNCT\n",
      "live ADV\n",
      "from ADP\n",
      "Kraków PROPN\n",
      ", PUNCT\n",
      "on ADP\n",
      "chatbots NOUN\n",
      ". PUNCT\n"
     ]
    }
   ],
   "source": [
    "sample = \"Broadcasting today, live from Kraków, on chatbots.\"\n",
    "\n",
    "doc = nlp(sample)\n",
    "for token in doc:\n",
    "    print(token.text, token.pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Noun chunks\n",
    "\n",
    "This NLP method is used to get the nouns from any sentene. It's important to understand what is the sentence about."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Kraków\n",
      "chatbots\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(sample)\n",
    "for nc in doc.noun_chunks:\n",
    "    print(nc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named Entity Recognition\n",
    "\n",
    "NER is a NLP method where we get not the nouns or part of speech, but meanings of the words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DATE today\n",
      "GPE Kraków\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(sample)\n",
    "for entity in doc.ents:\n",
    "    print(entity.label_, entity.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word vectorization\n",
    "\n",
    "Word vectorization is a process of preparing a vector representing each word. Gensim has an implementation of Word2Vec. We use a dimension of 100 and distance between two words in a sentence to 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.test.utils import common_texts, get_tmpfile\n",
    "from gensim.models import Word2Vec\n",
    "\n",
    "model = Word2Vec(common_texts, size=100, window=5, min_count=1, workers=4)\n",
    "model.save(\"word2vec.model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the vocabulary as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "human\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "vocab = list(model.wv.vocab)\n",
    "X = model[vocab]\n",
    "print(vocab[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train we just use the TSNE to reduce the dimensionality:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>human</th>\n",
       "      <td>234.548721</td>\n",
       "      <td>-163.857681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>interface</th>\n",
       "      <td>11.533824</td>\n",
       "      <td>-125.698242</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>computer</th>\n",
       "      <td>-259.990326</td>\n",
       "      <td>177.726364</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>survey</th>\n",
       "      <td>319.183319</td>\n",
       "      <td>67.650078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user</th>\n",
       "      <td>-145.300079</td>\n",
       "      <td>-17.329945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>system</th>\n",
       "      <td>48.006931</td>\n",
       "      <td>-325.026062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>response</th>\n",
       "      <td>119.843498</td>\n",
       "      <td>31.176579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <td>-73.440628</td>\n",
       "      <td>338.882019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eps</th>\n",
       "      <td>-36.990463</td>\n",
       "      <td>139.544861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trees</th>\n",
       "      <td>158.066574</td>\n",
       "      <td>254.191193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>graph</th>\n",
       "      <td>-183.524872</td>\n",
       "      <td>-240.342575</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>minors</th>\n",
       "      <td>-344.641174</td>\n",
       "      <td>-53.773724</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    x           y\n",
       "human      234.548721 -163.857681\n",
       "interface   11.533824 -125.698242\n",
       "computer  -259.990326  177.726364\n",
       "survey     319.183319   67.650078\n",
       "user      -145.300079  -17.329945\n",
       "system      48.006931 -325.026062\n",
       "response   119.843498   31.176579\n",
       "time       -73.440628  338.882019\n",
       "eps        -36.990463  139.544861\n",
       "trees      158.066574  254.191193\n",
       "graph     -183.524872 -240.342575\n",
       "minors    -344.641174  -53.773724"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.manifold import TSNE\n",
    "import pandas as pd\n",
    "\n",
    "tsne = TSNE(n_components=2)\n",
    "X_tsne = tsne.fit_transform(X)\n",
    "\n",
    "df = pd.DataFrame(X_tsne, index=vocab, columns=['x', 'y'])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can draw the words in a two-dimensional space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAD4CAYAAAAtrdtxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXgVZZb48e9JwBAQCSjaJKhAdwgkJCQkpEGkZRFCS4Q0QiM/XEC6aRVx7BmwYRTFkW6xYdRxaRh61LC1oIBAoyMuyNggCIEECMgSIIiBkQRIJJiQ7fz+uJU7l8h6s9ws5/M89VB16r1Vp2LMufVW1VuiqhhjjDHe8PN1AsYYY+ouKyLGGGO8ZkXEGGOM16yIGGOM8ZoVEWOMMV5r5OsErsQNN9yg7dq183UaxhhTp2zbti1HVVtX5z7qRBFp164dKSkpvk7DGGPqFBE5Ut37sO4sY4wxXrMiYuqd3Nxc/vKXvwBw7Ngxhg8f7uOMjKm/rIiYeseziAQHB7Ns2TIfZ2RM/VUnrokYczWmTJnCwYMHiY6OJjQ0lK+//pr09HSSk5NZuXIlZ8+e5cCBA0yaNImioiIWLlxIQEAAH374Ia1ateLgwYNMmDCB7OxsmjZtyl//+lc6derk68MyplayMxFT78ycOZOf/vSnpKWlMWvWrPPWpaens2LFCrZu3cpTTz1F06ZNSU1NpWfPnixYsACA8ePH89prr7Ft2zZmz57No48+6ovDMKZOsDMRU2+sTM1i1tp9HDmSyamcs6xMzSK65flt+vbtS/PmzWnevDktWrTg7rvvBiAyMpKdO3eSn5/Pl19+yYgRI9yfOXfuXE0ehjF1ihURUy+sTM1i6opdFBSXAlBSWsbUFbv4fY+g89oFBAS45/38/NzLfn5+lJSUUFZWRlBQEGlpaTWXvDF1mHVnmXph1tp97gIi1wRSVlRAQXEp//nFoavaznXXXUf79u157733AFBVduzYUeX5GlNfWBEx9cKx3AL3vH/gdQSEhHPszUfZu+ovV72txYsX8+abb9K1a1ciIiJYtWpVVaZqTL0ideGlVHFxcWpPrJtL6TVzHVkehaRcSFAgG6f080FGxvieiGxT1bjq3IediZh6YXJCGIGN/c+LBTb2Z3JCmI8yMqZhsAvrpl5IigkBXNdGjuUWEBwUyOSEMHfcGFM9rIiYeiMpJsSKhjE1zLqzjDHGeM2KiDHGGK9ZETHGGOM1KyLGGGO8VukiIiJNRGSLiOwQkd0i8pwTby8iX4lIhogsFZFrnHiAs5zhrG9X2RyMMcb4RlWciZwD+qlqVyAaGCQiPYAXgZdV9WfAaWCc034ccNqJv+y0M8YYUwdVuoioS76z2NiZFOgHlL8NaD6Q5MwPdZZx1vcXEalsHsYYY2pelVwTERF/EUkDTgCfAAeBXFUtcZp8C5TfwB8CHAVw1ucB119gm+NFJEVEUrKzs6siTWOMMVWsSoqIqpaqajTQFogHKv0aOFWdp6pxqhrXunXrSudojDGm6lXp3Vmqmgt8DvQEgkSk/In4tkCWM58F3AzgrG8BnKzKPIwxxtSMqrg7q7WIBDnzgcAA4GtcxWS40+xBoHw87dXOMs76dVoXhhI2xhjzI1UxdlYbYL6I+OMqSu+q6hoR2QMsEZEZQCrwptP+TWChiGQAp4B7qyAHY4wxPlDpIqKqO4GYC8QP4bo+UjFeCIyoGDfGGFP32BPrxhifyM3N5S9/ufo3T5raxYqIMcYnLlZESkpKLtDa1FZWRIwxPjFlyhQOHjxIdHQ03bt3p3fv3gwZMoTw8HAAFi1aRHx8PNHR0fzud7+jtLQUgI8//piePXvSrVs3RowYQX5+vnt74eHhREVFMWnSJJ8dV4OjqrV+io2NVWNM/XL48GGNiIhQVdXPP/9cmzZtqocOHVJV1T179mhiYqIWFRWpquojjzyi8+fP1+zsbO3du7fm5+erqurMmTP1ueee05ycHO3YsaOWlZWpqurp06d9cES1D5Ci1fz32d5saIypUStTs5i1dh9HjmRyKucsK1OzCALi4+Np3749AJ999hnbtm2je/fuABQUFHDjjTeyefNm9uzZQ69evQAoKiqiZ8+etGjRgiZNmjBu3DgSExNJTEz01eE1OFZEjDE1ZmVqFlNX7KKg2NU1VVJaxtQVuxh9yxmaNWvmbqeqPPjgg7zwwgvnff7vf/87AwYM4J133vnRtrds2cJnn33GsmXLeP3111m3bl31HowB7JqIMaYGzVq7z11A5JpAyooKKCguZcnWo+e169+/P8uWLePEiRMAnDp1iiNHjtCjRw82btxIRkYGAGfPnmX//v3k5+eTl5fHXXfdxcsvv8yOHTtq9sAaMDsTMcbUmGO5Be55/8DrCAgJ59ibjyKNAmgX29G9Ljw8nBkzZjBw4EDKyspo3Lgxb7zxBj169CA5OZlRo0Zx7tw5AGbMmEHz5s0ZOnQohYWFqCovvfRSjR9bQyVaB0YciYuL05SUFF+nYYyppF4z15HlUUjKhQQFsnFKPx9kVL+JyDZVjavOfVh3ljGmxkxOCCOwsf95scDG/kxOCPNRRqayrDvLGFNjkmJcrxWatXYfx3ILCA4KZHJCmDtu6h4rIsaYGpUUE2JFox6x7ixjjDFesyJijDHGa1ZEjDHGeM2KiDHGGK9ZETHGGOM1KyLGGGO8ZkXEGGOM16yIGGOM8Vqli4iI3Cwin4vIHhHZLSL/5MRbicgnInLA+belExcReVVEMkRkp4h0q2wOxhhjfKMqzkRKgH9R1XCgBzBBRMKBKcBnqhoKfOYsA/wSCHWm8cCcKsjBGGOMD1S6iKjqcVXd7syfAb4GQoChwHyn2XwgyZkfCixw3t64GQgSkTaVzcMYY0zNq9JrIiLSDogBvgJuUtXjzqr/BW5y5kMAzzfQfOvEjI+kpaXx4Ycf+joNY0wdVGVFRESuBZYDT6jq957rnBfGX9WLS0RkvIikiEhKdnZ2VaVpLsCbIlJSUlJN2Rhj6pIqKSIi0hhXAVmsqiuc8Hfl3VTOvyeceBZws8fH2zqx86jqPFWNU9W41q1bV0WatdaCBQuIioqia9eu3H///WRmZtKvXz+ioqLo378/33zzDQBjxozhkUceoUePHnTo0IH169fz0EMP0blzZ8aMGePe3rXXXsvvf/97IiIi6N+/P+VFuE+fPpS/3CsnJ4d27dpRVFTEM888w9KlS4mOjmbp0qWcPXuWhx56iPj4eGJiYli1ahUAycnJDBkyhH79+tG/f/+a/SEZY2onVa3UBAiwAHilQnwWMMWZnwL82ZkfDPy387kewJbL7SM2Nlbrq/T0dA0NDdXs7GxVVT158qQmJiZqcnKyqqq++eabOnToUFVVffDBB3XkyJFaVlamK1eu1ObNm+vOnTu1tLRUu3XrpqmpqarqOutbtGiRqqo+99xzOmHCBFVVveOOO3Tr1q2qqpqdna233nqrqqq+/fbb7jaqqlOnTtWFCxeqqurp06c1NDRU8/Pz9e2339aQkBA9efJkNf9UjDFVAUjRSv6Nv9xUFe8T6QXcD+wSkTQn9q/ATOBdERkHHAF+7az7ELgLyAB+AMZWQQ51ysrULPdLeWTPR3TrPYgbbrgBgFatWrFp0yZWrHCd0N1///08+eST7s/efffdiAiRkZHcdNNNREZGAhAREUFmZibR0dH4+fkxcuRIAO677z6GDRt2Vfl9/PHHrF69mtmzZwNQWFjoPhsaMGAArVq1qtwPwBhTb1S6iKjqBlxnFRfyoz4PpzpOqOx+66qVqVlMXbGLguJSAPIKilm/L5eVqVlX9KKegIAAAPz8/Nzz5csXu04h4vrP06hRI8rKygBXYbgYVWX58uWEhZ3/ytKvvvqKZs2aXTZHY0zDYU+s17BZa/e5CwhAk1uiyNvzD/60YgsAp06d4rbbbmPJkiUALF68mN69e1/VPsrKyli2bBkAf/vb37j99tsBaNeuHdu2bQNwrwdo3rw5Z86ccS8nJCTw2muvlXdLkpqaerWHaYxpIKyI1LBjuQXnLV/T+lZa9BxJ2twn6Nq1K//8z//Ma6+9xttvv01UVBQLFy7kP/7jP65qH82aNWPLli106dKFdevW8cwzzwAwadIk5syZQ0xMDDk5Oe72ffv2Zc+ePe4L69OmTaO4uJioqCgiIiKYNm1a5Q/cGFMvSfm3zdosLi5Oy+8qqut6zVxHVoVCAhASFMjGKf2qZB/XXnst+fn5VbItY0zdJSLbVDWuOvdhZyI1bHJCGIGN/c+LBTb2Z3JC2EU+YYwxtVdV3J1lrkL5xfPyu7OCgwKZnBB2RRfVr5SdhRhjaooVER9Iigmp0qJhjDG+Yt1ZxhhjvGZFxJiLWLRoEfHx8URHR/O73/2O0tLSiw4p8+qrrxIeHk5UVBT33nuvjzM3puZYETHmAr7++muWLl3Kxo0bSUtLw9/fn8WLF3P27Fni4uLYvXs3d9xxB8899xwAM2fOJDU1lZ07dzJ37lwfZ29MzbFrIsZcwGeffca2bdvo3r07AAUFBdx4440XHVImKiqK0aNHk5SURFJS0kW3a0x9Y2cixnhYmZpFr5nreHZVOn4d+zD97Q9IS0tj3759TJ8+/Ufty4eU+eCDD5gwYQLbt2+ne/fuNlS+aTCsiBjjKB/XLCu3gIBbu/LdzvVMXvgPVqZmcerUKY4cOXLBIWXKyso4evQoffv25cUXXyQvL89uszYNhnVnGePwHNfsmhtuIaj3/RxZ/K+Mfgc6BbfkjTfecA8pM2PGDG688UaWLl1KaWkp9913H3l5eagqjz/+OEFBQT4+GmNqhg17Yoyj/ZQPLvj6TQEOzxwM2JAypm6xYU+MqUHBQYFXFTfGWBExxu1KxjWzsxBjzmfXRIxx1MS4ZsbUN1ZEjPFg45oZc3WsO8sYY4zXrIgYY4zxmhURY4wxXquSIiIib4nICRFJ94i1EpFPROSA829LJy4i8qqIZIjIThHpVhU5GGOMqXlVdSaSDAyqEJsCfKaqocBnzjLAL4FQZxoPzKmiHIwxxtSwKikiqvoFcKpCeCgw35mfDyR5xBeoy2YgSETaVEUexhhjalZ1XhO5SVWPO/P/C9zkzIcARz3afevEziMi40UkRURSyl/8Y4wxpnapkQvr6hqg66oG6VLVeaoap6pxrVu3rqbMjDHGVEZ1FpHvyrupnH9POPEs4GaPdm2dmDHGmDqmOovIauBBZ/5BYJVH/AHnLq0eQJ5Ht5cxxpg6pEqGPRGRd4A+wA0i8i3wLDATeFdExgFHgF87zT8E7gIygB+AsVWRgzHGmJpXVXdnjVLVNqraWFXbquqbqnpSVfuraqiq3qmqp5y2qqoTVPWnqhqpqvaiEGOMuYSzZ88yePBgunbtSpcuXVi6dCnt2rUjJycHgJSUFPr06QPA9OnTuf/+++nVqxdAexHZLCIR5dsSkfUiEicizZxn/LaISKqIDHXWfyEi0R7tN4hI14vlZk+sG2NMLffRRx8RHBzMjh07SE9PZ9Cgio/lnW/Pnj18+umnAIeBpTg9Qc716TbOl/engHWqGg/0BWaJSDPgTWCM074j0ERVd1xsXzaKrzHG1EIrU7PcryVoWZzPtx98RKs//IHExER69+59yc8OGTKEwED3y9TeBT7GdZnh18AyJz4QGCIik5zlJsAtwHvANBGZDDyE62Hyi7IiYowxtczK1CymrthFQXEpAKca30DQ/3uJc82P8/TTT9O/f38aNWpEWVkZAIWFhed9vlmzZu55Vc0SkZMiEgWMBB52Vglwj6ruq7h/EfkE14PhvwZiL5WrdWcZY0wtM2vtPncBASg5c5JzNGJroy5MnjyZ7du3065dO7Zt2wbA8uXLL7fJpcCTQAtV3enE1gITRUQARCTGo/1/Aa8CW1X19KU2bGcixhhTyxzLLThvuTg7kxPr3+a4CM/dcj1z5syhoKCAcePGMW3aNPdF9UtYBvwH8LxH7HngFWCniPjhun6SCKCq20Tke+Dty21YXA+T125xcXGakmI3cRljGoZeM9eRVaGQAIQEBbJxSr8r3o6IbFPVuKvdv4gEA+uBTqpadqm21p1ljDG1zOSEMAIb+58XC2zsz+SEsGrft4g8AHwFPHW5AgLWnWWMMbVOUoxrTNryu7OCgwKZnBDmjlcnVV0ALLjS9lZEjDGmFkqKCamRolFZ1p1ljKk0VXXfbmoaFisixhivZGZmEhYWxgMPPECXLl1YuHAhPXv2pFu3bowYMYL8/HwApkyZQnh4OFFRUUya5HqubcyYMTz88MPExcXRsWNH1qxZA7iedxg7diyRkZHExMTw+eefA5CcnMywYcMYNGgQoaGhPPnkkwCUlpYyZswYunTpQmRkJC+//DIABw8eZNCgQcTGxtK7d2/27t1b0z+eBsO6s4wxXjtw4ADz58/nZz/7GcOGDePTTz+lWbNmvPjii7z00ktMmDCB999/n7179yIi5Obmuj+bmZnJli1bOHjwIH379iUjI4M33ngDEWHXrl3s3buXgQMHsn//fgDS0tJITU0lICCAsLAwJk6cyIkTJ8jKyiI9PR3Avf3x48czd+5cQkND+eqrr3j00UdZt25dzf+AGgArIsaYK+Y5FEcrzaN1m7b06NGDNWvWsGfPnvJB/ygqKqJnz560aNGCJk2aMG7cOBITE0lMTHRv69e//jV+fn6EhobSoUMH9u7dy4YNG5g4cSIAnTp14tZbb3UXkf79+9OiRQsAwsPDOXLkCBERERw6dIiJEycyePBgBg4cSH5+Pl9++SUjRoxw7+vcuXM19SNqcKyIGGOuSMWhOL77vpDcYj9Wpmbhr8qAAQN45513fvS5LVu28Nlnn7Fs2TJef/119xmB86C0W8XligICAtzz/v7+lJSU0LJlS3bs2MHatWuZO3cu7777Lq+88gpBQUGkpaVV9pDNFbBrIsaYK1JxKA5wXVCftXYfPXr0YOPGjWRkZACuocv3799Pfn4+eXl53HXXXbz88svs2PF/g8G+9957lJWVcfDgQQ4dOkRYWBi9e/dm8eLFAOzfv59vvvmGsLCLPxuRk5NDWVkZ99xzDzNmzGD79u1cd911tG/fnvfee8+do+d+TdWyMxFjzBWpOBSHZ7x169YkJyczatQod9fRjBkzaN68OUOHDqWwsBBV5aWXXnJ/7pZbbiE+Pp7vv/+euXPn0qRJEx599FEeeeQRIiMjadSoEcnJyeedgVSUlZXF2LFj3XeGvfDCCwAsXryYRx55hBkzZlBcXMy9995L164XfSWGqQQb9sQYc0WqaigOcN2dlZiYyPDhw6sqPXMB3g57cjWsO8sYc0V8ORSHqb2sO8sYc0WqciiO5OTkKs7O+IoVEWPMFasrQ3GYmmPdWcYYY7zmsyIiIoNEZJ+IZIjIFF/lYYwxxns+KSIi4g+8AfwSCAdGiUi4L3IxxhjjPV+dicQDGap6SFWLgCW4XgpvzI+UlJT4OgVjzEX46sJ6CHDUY/lb4OeeDURkPDAeXA8lmbojMzOTxMRE96B4s2fPJj8/n1atWjF37lwaNWpEeHg4S5Ys4ezZs0ycOJH09HSKi4uZPn06Q4cOJTk5mRUrVpCfn09paSn/8z//4+OjMsZcSK29O0tV5wHzwPWwoY/TMVVg5syZHD58mICAAPdoq3/84x/p168fb731Frm5ucTHx3PnnXcCsH37dnbu3EmrVq18mbYx5hJ8VUSygJs9lts6MVNHVRzd9fvCH3dBRUVFMXr0aJKSkkhKSgLg448/ZvXq1cyePRtwvU/im2++AWDAgAFWQIyp5Xx1TWQrECoi7UXkGuBeYLWPcjGVVD66a1ZuAQp8d6aY7/J+YGWq63tBYWEhAB988AETJkxg+/btdO/enZKSElSV5cuXk5aWRlpaGt988w2dO3cGoFmzZr46JGPMFfJJEVHVEuAxYC3wNfCuqu72RS6m8iqO7urfLIiSs3n8acUWzp07x5o1aygrK+Po0aP07duXF198kby8PPLz80lISOC1116jfAy31NRUXx2GMcYLPrsmoqofAh/6av+m6lQc3VX8G9HitntJff1RBvxPKJ06daK0tJT77ruPvLw8VJXHH3+coKAgpk2bxhNPPEFUVBRlZWW0b9/e/apUY0ztZ6P4mkqrytFdjTFVx0bxNXWCje5qTMNVa2/xNXVHVY7uaoypW6yIeFi9ejV79uxhyhQbyutq2eiuxjRMdk2kGpSUlNCokdVnY4xv2TWRKpSZmUmnTp0YM2YMHTt2ZPTo0Xz66af06tWL0NBQtmzZQnJyMo899hjgen3n448/zm233UaHDh1YtmwZAKrK5MmT6dKlC5GRkSxduhSA9evX07t3b4YMGUJ4eDhnz55l8ODBdO3alS5durjbGWNMfdKgvi5nZGTw3nvv8dZbb9G9e3f+9re/sWHDBlavXs2f/vQn91PU5Y4fP86GDRvYu3cvQ4YMYfjw4axYsYK0tDR27NhBTk4O3bt35xe/+AXgGqYjPT2d9u3bs3z5coKDg/nggw8AyMvLq/HjNcaY6lavi0jFoThuDL6ZyMhIACIiIujfvz8iQmRkJJmZmT/6fFJSEn5+foSHh/Pdd98BsGHDBkaNGoW/vz833XQTd9xxB1u3buW6664jPj6e9u3bAxAZGcm//Mu/8Ic//IHExER69+5dY8dtjDE1pd52Z/1oKI7vCzlZqO6hOPz8/AgICHDPX2i48fL1AFdy7chzmI6OHTuyfft2IiMjefrpp/m3f/u3Sh6RMcbUPvW2iFQcigNchWDW2n2V2m7v3r1ZunQppaWlZGdn88UXXxAfH/+jdseOHaNp06bcd999TJ48me3bt1dqv8YYUxvV2+6sikNxXC5+pX71q1+xadMmunbtiojw5z//mZ/85Cfs3bv3vHa7du1i8uTJ+Pn50bhxY+bMmVOp/RpjTG1Ub2/xtaE4jDENnd3iWwk2FIcxxlS/etudZUNxGGNM9au3RQRsKA5jjKlu9bY7yxhjTPWzImKMMcZrVkSMMcZ4zYqIMcYYr1kRMcYY4zUrIsYYY7xWqSIiIiNEZLeIlIlIXIV1U0UkQ0T2iUiCR3yQE8sQEXuFoDHG1GGVPRNJB4YBX3gGRSQcuBeIAAYBfxERfxHxB94AfgmEA6OctsYYY+qgSj1sqKpfA4hIxVVDgSWqeg44LCIZQPlQtxmqesj53BKn7Z7K5GGMMcY3quuaSAhw1GP5Wyd2sfiPiMh4EUkRkZTs7OxqStOYmnfbbbddts0rr7zCDz/8cNXb3rt3L9HR0cTExHDw4EFv0jPmqly2iIjIpyKSfoFpaHUmpqrzVDVOVeNat25dnbsypkZ9+eWXl23jTREpLS1l5cqVDB8+nNTUVH760596m6IxV+yyRURV71TVLheYVl3iY1nAzR7LbZ3YxeLGNBjXXnstAOvXr6dPnz4MHz6cTp06MXr0aFSVV199lWPHjtG3b1/69u0LwMcff0zPnj3p1q0bI0aMID8/H4B27drxhz/8gW7durF06VJeeeUV5syZ4/5cUlISsbGxREREMG/ePHcOH330Ed26daNr1670798fgLNnz/LQQw8RHx9PTEwMq1Zd6n9xYxyqWukJWA/EeSxHADuAAKA9cAjwx3UN5pATu8ZpE3G57cfGxqox9UWzZs1UVfXzzz/X6667To8ePaqlpaXao0cP/cc//qGqqrfeeqtmZ2erqmp2drb27t1b8/PzVVV15syZ+txzz7nbvfjii+5tP/vsszpr1iz38smTJ1VV9YcfftCIiAjNycnREydOaNu2bfXQoUPntZk6daouXLhQVVVPnz6toaGh7n2auglI0Sr4G3+pqVIX1kXkV8BrQGvgAxFJU9UEVd0tIu/iumBeAkxQ1VLnM48Ba52i8paq7q5MDsbUZfHx8bRt2xaA6OhoMjMzuf32289rs3nzZvbs2UOvXr0AKCoqomfPnu71I0eOvOj2X331Vd5//30Ajh49yoEDB8jOzuYXv/gF7du3B6BVq1aA62xn9erVzJ49G4DCwkK++eYbOnfuXEVHa+qjyt6d9T7w/kXW/RH44wXiHwIfVma/xtQ1K1Oz3O+2KSguZWVqFkFAQECAu42/vz8lJSU/+qyqMmDAAN55550LbrtZs2YXjK9fv55PP/2UTZs20bRpU/r06UNhYeFFc1RVli9fTliYvbjNXDl7Yt2YarYyNYupK3aRlVuAAqowdcUuNhy4+F2HzZs358yZMwD06NGDjRs3kpGRAbiuXezfv/+y+83Ly6Nly5Y0bdqUvXv3snnzZvf2vvjiCw4fPgzAqVOnAEhISOC1114r75ImNTXV62M2DYcVEWOq2ay1+ygoLj0vVlBcypKtRy/yCRg/fjyDBg2ib9++tG7dmuTkZEaNGkVUVBQ9e/Zk7969l93voEGDKCkpoXPnzkyZMoUePXoA0Lp1a+bNm8ewYcPo2rWruzts2rRpFBcXExUVRUREBNOmTavEUZuGQsq/ddRmcXFxmpKS4us0jPFK+ykfcKH/ywQ4PHNwTadjGhAR2aaqcZdv6T07EzGmmgUHBV5V3Ji6xIqIMdVsckIYgY39z4sFNvZncoJdwDZ1X6XuzjLGXF5SjGtkn/K7s4KDApmcEOaOG1OXWRExpgYkxYRY0ajlMjMzSUxMJD093dep1CnWnWWMMcZrVkSMMcZRWlrKb3/7WyIiIhg4cCAFBQX06dOH8rtDc3JyaNeuHQDJyckkJSUxYMAA2rVrx+uvv85LL71ETEwMPXr0cD9/89e//pXu3bvTtWtX7rnnHvfAmmPGjOHxxx/ntttuo0OHDixbtswnx1xZVkSMMcZx4MABJkyYwO7duwkKCmL58uWXbJ+ens6KFSvYunUrTz31FE2bNiU1NZWePXuyYMECAIYNG8bWrVvZsWMHnTt35s0333R//vjx42zYsIE1a9YwZUrdfNGrXRMxxjRYnsPRtNI8bgy+mejoaABiY2PJzMy85Of79u1L8+bNad68OS1atODuu+8GIDIykp07dwKuQvP000+Tm5tLfn4+CQnut4WTlJSEn58f4eHhfPfdd9VzkNXMzkSMMQ1SxeFovg09QJgAAA6wSURBVPu+kJOFyspU19spyscya9SoEWVlZQA/GnvMc+wzPz8/97Kfn597HLQxY8bw+uuvs2vXLp599tnztuH5+brw4PeFWBExxjRIFxqORlWZtXbfebF27dqxbds2AK+uW5w5c4Y2bdpQXFzM4sWLvU+4lrIiYoxpkI7lFlxRfNKkScyZM4eYmBhycnKuej/PP/88P//5z+nVqxedOnXyKtfazMbOMsY0SL1mriPrAoUkJCiQjVP6+SCjqmdjZxljTDWx4Wiqht2dZYxpkGw4mqphRcQY02DZcDSVZ91ZxhhjvGZFxBhjjNesiBhjjPFapYqIiMwSkb0islNE3heRII91U0UkQ0T2iUiCR3yQE8sQkbo5WIwxxhig8mcinwBdVDUK2A9MBRCRcOBeIAIYBPxFRPxFxB94A/glEA6MctoaY4ypgypVRFT1Y1UtcRY3A22d+aHAElU9p6qHgQwg3pkyVPWQqhYBS5y2xhhj6qCqvCbyEPDfznwIcNRj3bdO7GLxHxGR8SKSIiIp2dnZVZimMcaYqnLZ50RE5FPgJxdY9ZSqrnLaPAWUAFU2upiqzgPmgWvYk6rarjHGmKpz2SKiqndear2IjAESgf76fwNxZQE3ezRr68S4RNwYY0wdU9m7swYBTwJDVPUHj1WrgXtFJEBE2gOhwBZgKxAqIu1F5BpcF99XVyYHY4wxvlPZYU9eBwKAT0QEYLOqPqyqu0XkXWAPrm6uCapaCiAijwFrAX/gLVXdXckcjDHG+IgNBW+q3fr165k9ezZr1qzxdSrGNCg2FLypdcpf+WmMMWCj+JoKnn/+eRYtWkTr1q25+eabiY2NZc2aNURHR7NhwwZGjRpFx44dmTFjBkVFRVx//fUsXryYm266ienTp3Pw4EEyMjLIycnhySef5Le//S0A+fn5DB8+nPT0dGJjY1m0aBFOF6gxpg6zImLctm7dyvLly9mxYwfFxcV069aN2NhYAIqKiijvUjx9+jSbN29GRPiv//ov/vznP/Pv//7vAOzcuZPNmzdz9uxZYmJiGDx4MACpqans3r2b4OBgevXqxcaNG7n99tt9c6DGmCpjRcSwMjWLWWv38fUnS2h2fRQffX2SpJgQ7r77bnebkSNHuue//fZbRo4cyfHjxykqKqJ9+/budUOHDiUwMJDAwED69u3Lli1bCAoKIj4+nrZtXQMaREdHk5mZaUXEmHrArok0cCtTs5i6Ypf7XdNnCkuYumIXK1PPf3ynWbNm7vmJEyfy2GOPsWvXLv7zP/+TwsJC97qKXVTlywEBAe6Yv7+/XVsxpp6wItLAzVq7j4LiUgAC2nam4OAWfigoYObqtIveTZWXl0dIiGu0mvnz55+3btWqVRQWFnLy5EnWr19P9+7dq/cAjDE+Zd1ZDdwx5wwEIKBNRwJ/Fs+xtx7ju2ZBDI6PpEWLFj/6zPTp0xkxYgQtW7akX79+HD582L0uKiqKvn37kpOTw7Rp0wgODmb//v01cizGmJpnz4k0cL1mrnN3ZQGUFRXgd00gP2kqlKx+hnnz5tGtW7cr2tb06dO59tprmTRpUnWla4y5CjXxnIidiTRwkxPCmLpil7tL6+RHr1N66iglgcKE8eOuuIAYYxomOxMx7ruzjuUWEBwUyOSEMJJiLjhCvzGmDrEzEVMjkmJCrGgYY7xid2cZY4zxmhURY4wxXrMiYowxxmtWRIwxxnjNiogxxhivWRExxhjjNSsixhhjvGZFxBhjjNesiBhjjPFapYqIiDwvIjtFJE1EPhaRYCcuIvKqiGQ467t5fOZBETngTA9W9gCMMcb4TmXPRGapapSqRgNrgGec+C+BUGcaD8wBEJFWwLPAz4F44FkRaVnJHIwxxvhIpYqIqn7vsdgMKB/NcSiwQF02A0Ei0gZIAD5R1VOqehr4BBhUmRyMMcb4TqUHYBSRPwIPAHlAXyccAhz1aPatE7tY/ELbHY/rLIZbbrmlsmkaY4ypBpc9ExGRT0Uk/QLTUABVfUpVbwYWA49VVWKqOk9V41Q1rnXr1lW1WWMavOTkZI4dO+brNEw9cdkioqp3qmqXC0yrKjRdDNzjzGcBN3usa+vELhY3xtQQKyKmKlX27qxQj8WhwF5nfjXwgHOXVg8gT1WPA2uBgSLS0rmgPtCJGWOu0NmzZxk8eDBdu3alS5cuLF26lKSkJPf6Tz75hF/96leUlpYyZswYunTpQmRkJC+//DLLli0jJSWF0aNHEx0dTUFBAdu2beOOO+4gNjaWhIQEjh8/DkCfPn34/e9/T1xcHJ07d2br1q0MGzaM0NBQnn76aV8dvqltVNXrCVgOpAM7gb8DIU5cgDeAg8AuIM7jMw8BGc409kr2Exsbq8YYl2XLlulvfvMb93Jubq6GhYXpiRMnVFV11KhRunr1ak1JSdE777zT3e706dOqqnrHHXfo1q1bVVW1qKhIe/bs6f7skiVLdOzYse52Tz75pKqqvvLKK9qmTRs9duyYFhYWakhIiObk5FT/wZpKAVK0En/jr2Sq1IV1Vb3nInEFJlxk3VvAW5XZrzENUflrjI8cOknOsr9zsvhRfj9uFL179+b+++9n0aJFjB07lk2bNrFgwQLOnDnDoUOHmDhxIoMHD2bgwIE/2ua+fftIT09nwIABAJSWltKmTRv3+iFDhgAQGRlJRESEe12HDh04evQo119/fQ0cuanN7PW4xtQBK1OzmLpiFwXFpTRqFULrB15h85HtPPzEZEYOvYvf/OY33H333TRp0oQRI0bQqFEjWrZsyY4dO1i7di1z587l3Xff5a23zv/+pqpERESwadOmC+43ICAAAD8/P/d8+XJJSUn1HbCpM2zYE2PqgFlr91FQXApAyZmT+DUO4JpOd1DW5W62b99OcHAwwcHBzJgxg7FjxwKQk5NDWVkZ99xzDzNmzGD79u0ANG/enDNnzgAQFhZGdna2u4gUFxeze/duHxyhqavsTMSYOuBYboF7vjg7kxPr3wYRxK8RC//+NwBGjx5NdnY2nTt3BiArK4uxY8dSVlYGwAsvvADAmDFjePjhhwkMDGTTpk0sW7aMxx9/nLy8PEpKSnjiiSeIiIio4SM0dZW4Ll/UbnFxcZqSkuLrNIzxmV4z15HlUUjKhQQFsnFKPwAee+wxYmJiGDduXE2nZ2opEdmmqnHVuQ/rzjKmDpicEEZgY//zYoGN/ZmcEAZAbGwsO3fu5L777vNFeqYBs+4sY+qApBjX6ECz1u7jWG4BwUGBTE4Ic8e3bdvmy/RMA2ZFxJg6IikmxF00jKktrDvLGGOM16yIGGOM8ZoVEWOMMV6zImKMMcZrVkSMMcZ4rU48bCgi2cARX+fh4QYgx9dJeMHyrlmWd82pizlD9ed9q6pW61v96kQRqW1EJKW6nwKtDpZ3zbK8a05dzBnqbt6erDvLGGOM16yIGGOM8ZoVEe/M83UCXrK8a5blXXPqYs5Qd/N2s2sixhhjvGZnIsYYY7xmRcQYY4zXrIhchog8LyI7RSRNRD4WkWAnLiLyqohkOOu7eXzmQRE54EwP+iDnWSKy18nrfREJ8lg31cl5n4gkeMQHObEMEZlS0zk7OYwQkd0iUiYicRXW1dq8K6qNOZUTkbdE5ISIpHvEWonIJ87v6yci0tKJX/R33Ad53ywin4vIHud35J/qQu4i0kREtojIDifv55x4exH5yslvqYhc48QDnOUMZ307X+R9VVTVpktMwHUe848Dc535u4D/BgToAXzlxFsBh5x/WzrzLWs454FAI2f+ReBFZz4c2AEEAO2Bg4C/Mx0EOgDXOG3CffCz7gyEAeuBOI94rc67wjHUupwq5PcLoBuQ7hH7MzDFmZ/i8ftywd9xH+XdBujmzDcH9ju/F7U6d2f/1zrzjYGvnHzeBe514nOBR5z5Rz3+xtwLLPX178zlJjsTuQxV/d5jsRlQfifCUGCBumwGgkSkDZAAfKKqp1T1NPAJMKiGc/5YVUucxc1AW4+cl6jqOVU9DGQA8c6UoaqHVLUIWOK0rVGq+rWq7rvAqlqddwW1MSc3Vf0COFUhPBSY78zPB5I84hf6Ha9xqnpcVbc782eAr4EQannuzv7zncXGzqRAP2CZE6+Yd/nxLAP6i4jUULpesSJyBUTkjyJyFBgNPOOEQ4CjHs2+dWIXi/vKQ7i+kUHdybmiupR3bczpcm5S1ePO/P8CNznztfJYnC6eGFzf6mt97iLiLyJpwAlcXyoPArkeX/Q8c3Pn7azPA66v2YyvjhURQEQ+FZH0C0xDAVT1KVW9GVgMPObbbF0ul7PT5imgBFfetcKV5G18R139KLX2vn8RuRZYDjxRoZeg1uauqqWqGo2rRyAe6OTjlKqUvR4XUNU7r7DpYuBD4FkgC7jZY11bJ5YF9KkQX1/pJCu4XM4iMgZIBPo7/3PBxXPmEvEqdRU/a08+z/sqXCrX2uo7EWmjqsedLp8TTrxWHYuINMZVQBar6gonXCdyB1DVXBH5HOiJq3utkXO24Zlbed7fikgjoAVw0icJXyE7E7kMEQn1WBwK7HXmVwMPOHeB9ADynNPqtcBAEWnp3Cky0InVZM6DgCeBIar6g8eq1cC9zh0g7YFQYAuwFQh17hi5BtcFvdU1mfNl1KW8a2NOl7MaKL+L8EFglUf8Qr/jNc65LvAm8LWqvuSxqlbnLiKtxbk7UkQCgQG4rud8Dgx3mlXMu/x4hgPrPL4E1k6+vrJf2ydc33zSgZ3A34EQ/b+7Lt7A1b+5i/PvJnoI18XfDGCsD3LOwNWvmuZMcz3WPeXkvA/4pUf8Llx3vBwEnvLRz/pXuPqHzwHfAWvrQt4XOI5al5NHbu8Ax4Fi52c9Dlef+2fAAeBToJXT9qK/4z7I+3ZcXVU7PX6v76rtuQNRQKqTdzrwjBPvgOuLUAbwHhDgxJs4yxnO+g6+/p253GTDnhhjjPGadWcZY4zxmhURY4wxXrMiYowxxmtWRIwxxnjNiogxxhivWRExxhjjNSsixhhjvPb/AVsRnDe6HIxvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "\n",
    "ax.scatter(df['x'], df['y'])\n",
    "\n",
    "for word, pos in df.iterrows():\n",
    "    ax.annotate(word, pos)\n",
    "plt.show()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Negative sampling\n",
    "\n",
    "It is a simpler implementation of word2vec. It is faster as it takes only a few terms in each iteration for training insted of the whole dataset as in previous example. This is why it's called negative sampling.\n",
    "\n",
    "First of all, we define helper methods that are used later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def zeros(*dims):\n",
    "    return np.zeros(shape=tuple(dims), dtype=np.float32)\n",
    "\n",
    "def ones(*dims):\n",
    "    return np.ones(shape=tuple(dims), dtype=np.float32)\n",
    "\n",
    "def rand(*dims):\n",
    "    return np.random.rand(*dims).astype(np.float32)\n",
    "\n",
    "def randn(*dims):\n",
    "    return np.random.randn(*dims).astype(np.float32)\n",
    "\n",
    "def sigmoid(batch, stochastic=False):\n",
    "    return  1.0 / (1.0 + np.exp(-batch))\n",
    "\n",
    "def as_matrix(vector):\n",
    "    return np.reshape(vector, (-1, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import gzip\n",
    "\n",
    "with gzip.open(\"datasets/text8.dat.gz\", \"rb\") as f:\n",
    "    train_dict, train_set, train_tokens = pickle.load(f)\n",
    "\n",
    "train_set = np.random.permutation(train_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "Config = namedtuple(\"Config\", [\"dict_size\", \"vect_size\", \"neg_samples\", \"updates\", \"learning_rate\", \n",
    "                               \"learning_rate_decay\", \"decay_period\", \"log_period\"])\n",
    "conf = Config(\n",
    "    dict_size=len(train_dict),\n",
    "    vect_size=100,\n",
    "    neg_samples=10,\n",
    "    updates=5000000,\n",
    "    learning_rate=0.1,\n",
    "    learning_rate_decay=0.995,\n",
    "    decay_period=10000,\n",
    "    log_period=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neg_sample(conf, train_set, train_tokens): # implemented\n",
    "    Vp = randn(conf.dict_size, conf.vect_size)\n",
    "    Vo = randn(conf.dict_size, conf.vect_size)\n",
    "\n",
    "    J = 0.0\n",
    "    learning_rate = conf.learning_rate\n",
    "    for i in range(conf.updates):\n",
    "        idx = i % len(train_set)\n",
    "\n",
    "        word    = train_set[idx, 0]\n",
    "        context = train_set[idx, 1]\n",
    "        \n",
    "        neg_context = np.random.randint(0, len(train_tokens), conf.neg_samples)\n",
    "        neg_context = train_tokens[neg_context]\n",
    "\n",
    "        word_vect = Vp[word, :]              # word vector\n",
    "        context_vect = Vo[context, :];       # context wector\n",
    "        negative_vects = Vo[neg_context, :]  # sampled negative vectors\n",
    "\n",
    "        # Cost and gradient calculation starts here\n",
    "        score_pos = word_vect @ context_vect.T\n",
    "        score_neg = word_vect @ negative_vects.T\n",
    "\n",
    "        J -= np.log(sigmoid(score_pos)) + np.sum(np.log(sigmoid(-score_neg)))\n",
    "        if (i + 1) % conf.log_period == 0:\n",
    "            print('Update {0}\\tcost: {1:>2.2f}'.format(i + 1, J / conf.log_period))\n",
    "            final_cost = J / conf.log_period\n",
    "            J = 0.0\n",
    "\n",
    "        pos_g = 1.0 - sigmoid(score_pos)\n",
    "        neg_g = sigmoid(score_neg)\n",
    "\n",
    "        word_grad = -pos_g * context_vect + np.sum(as_matrix(neg_g) * negative_vects, axis=0)\n",
    "        context_grad = -pos_g * word_vect\n",
    "        neg_context_grad = as_matrix(neg_g) * as_matrix(word_vect).T\n",
    "\n",
    "        Vp[word, :] -= learning_rate * word_grad\n",
    "        Vo[context, :] -= learning_rate * context_grad\n",
    "        Vo[neg_context, :] -= learning_rate * neg_context_grad\n",
    "\n",
    "        if i % conf.decay_period == 0:\n",
    "            learning_rate = learning_rate * conf.learning_rate_decay\n",
    "\n",
    "    return Vp, Vo, final_cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Update 10000\tcost: 36.24\n",
      "Update 20000\tcost: 28.56\n",
      "Update 30000\tcost: 23.27\n",
      "Update 40000\tcost: 19.40\n",
      "Update 50000\tcost: 17.25\n",
      "Update 60000\tcost: 15.22\n",
      "Update 70000\tcost: 14.06\n",
      "Update 80000\tcost: 13.08\n",
      "Update 90000\tcost: 12.25\n",
      "Update 100000\tcost: 11.50\n",
      "Update 110000\tcost: 10.96\n",
      "Update 120000\tcost: 10.44\n",
      "Update 130000\tcost: 10.13\n",
      "Update 140000\tcost: 9.77\n",
      "Update 150000\tcost: 9.29\n",
      "Update 160000\tcost: 9.24\n",
      "Update 170000\tcost: 8.77\n",
      "Update 180000\tcost: 8.56\n",
      "Update 190000\tcost: 8.35\n",
      "Update 200000\tcost: 8.25\n",
      "Update 210000\tcost: 8.03\n",
      "Update 220000\tcost: 7.79\n",
      "Update 230000\tcost: 7.60\n",
      "Update 240000\tcost: 7.55\n",
      "Update 250000\tcost: 7.46\n",
      "Update 260000\tcost: 7.40\n",
      "Update 270000\tcost: 7.19\n",
      "Update 280000\tcost: 7.05\n",
      "Update 290000\tcost: 7.05\n",
      "Update 300000\tcost: 6.89\n",
      "Update 310000\tcost: 6.87\n",
      "Update 320000\tcost: 6.83\n",
      "Update 330000\tcost: 6.63\n",
      "Update 340000\tcost: 6.57\n",
      "Update 350000\tcost: 6.52\n",
      "Update 360000\tcost: 6.44\n",
      "Update 370000\tcost: 6.38\n",
      "Update 380000\tcost: 6.30\n",
      "Update 390000\tcost: 6.26\n",
      "Update 400000\tcost: 6.28\n",
      "Update 410000\tcost: 6.10\n",
      "Update 420000\tcost: 6.10\n",
      "Update 430000\tcost: 6.01\n",
      "Update 440000\tcost: 5.97\n",
      "Update 450000\tcost: 5.94\n",
      "Update 460000\tcost: 5.85\n",
      "Update 470000\tcost: 5.87\n",
      "Update 480000\tcost: 5.78\n",
      "Update 490000\tcost: 5.78\n",
      "Update 500000\tcost: 5.70\n",
      "Update 510000\tcost: 5.72\n",
      "Update 520000\tcost: 5.65\n",
      "Update 530000\tcost: 5.73\n",
      "Update 540000\tcost: 5.55\n",
      "Update 550000\tcost: 5.52\n",
      "Update 560000\tcost: 5.53\n",
      "Update 570000\tcost: 5.50\n",
      "Update 580000\tcost: 5.52\n",
      "Update 590000\tcost: 5.42\n",
      "Update 600000\tcost: 5.34\n",
      "Update 610000\tcost: 5.40\n",
      "Update 620000\tcost: 5.30\n",
      "Update 630000\tcost: 5.30\n",
      "Update 640000\tcost: 5.35\n",
      "Update 650000\tcost: 5.22\n",
      "Update 660000\tcost: 5.25\n",
      "Update 670000\tcost: 5.16\n",
      "Update 680000\tcost: 5.18\n",
      "Update 690000\tcost: 5.13\n",
      "Update 700000\tcost: 5.16\n",
      "Update 710000\tcost: 5.10\n",
      "Update 720000\tcost: 5.09\n",
      "Update 730000\tcost: 5.07\n",
      "Update 740000\tcost: 5.08\n",
      "Update 750000\tcost: 5.08\n",
      "Update 760000\tcost: 5.06\n",
      "Update 770000\tcost: 4.98\n",
      "Update 780000\tcost: 4.95\n",
      "Update 790000\tcost: 4.91\n",
      "Update 800000\tcost: 4.90\n",
      "Update 810000\tcost: 4.89\n",
      "Update 820000\tcost: 4.91\n",
      "Update 830000\tcost: 4.92\n",
      "Update 840000\tcost: 4.90\n",
      "Update 850000\tcost: 4.86\n",
      "Update 860000\tcost: 4.84\n",
      "Update 870000\tcost: 4.79\n",
      "Update 880000\tcost: 4.77\n",
      "Update 890000\tcost: 4.86\n",
      "Update 900000\tcost: 4.80\n",
      "Update 910000\tcost: 4.74\n",
      "Update 920000\tcost: 4.72\n",
      "Update 930000\tcost: 4.73\n",
      "Update 940000\tcost: 4.70\n",
      "Update 950000\tcost: 4.65\n",
      "Update 960000\tcost: 4.63\n",
      "Update 970000\tcost: 4.63\n",
      "Update 980000\tcost: 4.69\n",
      "Update 990000\tcost: 4.67\n",
      "Update 1000000\tcost: 4.65\n",
      "Update 1010000\tcost: 4.61\n",
      "Update 1020000\tcost: 4.63\n",
      "Update 1030000\tcost: 4.58\n",
      "Update 1040000\tcost: 4.52\n",
      "Update 1050000\tcost: 4.59\n",
      "Update 1060000\tcost: 4.53\n",
      "Update 1070000\tcost: 4.53\n",
      "Update 1080000\tcost: 4.58\n",
      "Update 1090000\tcost: 4.54\n",
      "Update 1100000\tcost: 4.54\n",
      "Update 1110000\tcost: 4.46\n",
      "Update 1120000\tcost: 4.48\n",
      "Update 1130000\tcost: 4.45\n",
      "Update 1140000\tcost: 4.45\n",
      "Update 1150000\tcost: 4.41\n",
      "Update 1160000\tcost: 4.45\n",
      "Update 1170000\tcost: 4.43\n",
      "Update 1180000\tcost: 4.43\n",
      "Update 1190000\tcost: 4.45\n",
      "Update 1200000\tcost: 4.36\n",
      "Update 1210000\tcost: 4.38\n",
      "Update 1220000\tcost: 4.37\n",
      "Update 1230000\tcost: 4.33\n",
      "Update 1240000\tcost: 4.38\n",
      "Update 1250000\tcost: 4.40\n",
      "Update 1260000\tcost: 4.33\n",
      "Update 1270000\tcost: 4.32\n",
      "Update 1280000\tcost: 4.35\n",
      "Update 1290000\tcost: 4.32\n",
      "Update 1300000\tcost: 4.33\n",
      "Update 1310000\tcost: 4.30\n",
      "Update 1320000\tcost: 4.30\n",
      "Update 1330000\tcost: 4.27\n",
      "Update 1340000\tcost: 4.24\n",
      "Update 1350000\tcost: 4.29\n",
      "Update 1360000\tcost: 4.25\n",
      "Update 1370000\tcost: 4.25\n",
      "Update 1380000\tcost: 4.24\n",
      "Update 1390000\tcost: 4.23\n",
      "Update 1400000\tcost: 4.20\n",
      "Update 1410000\tcost: 4.26\n",
      "Update 1420000\tcost: 4.18\n",
      "Update 1430000\tcost: 4.18\n",
      "Update 1440000\tcost: 4.19\n",
      "Update 1450000\tcost: 4.21\n",
      "Update 1460000\tcost: 4.20\n",
      "Update 1470000\tcost: 4.18\n",
      "Update 1480000\tcost: 4.16\n",
      "Update 1490000\tcost: 4.17\n",
      "Update 1500000\tcost: 4.16\n",
      "Update 1510000\tcost: 4.19\n",
      "Update 1520000\tcost: 4.16\n",
      "Update 1530000\tcost: 4.09\n",
      "Update 1540000\tcost: 4.16\n",
      "Update 1550000\tcost: 4.06\n",
      "Update 1560000\tcost: 4.14\n",
      "Update 1570000\tcost: 4.12\n",
      "Update 1580000\tcost: 4.10\n",
      "Update 1590000\tcost: 4.14\n",
      "Update 1600000\tcost: 4.09\n",
      "Update 1610000\tcost: 4.10\n",
      "Update 1620000\tcost: 4.07\n",
      "Update 1630000\tcost: 4.08\n",
      "Update 1640000\tcost: 4.10\n",
      "Update 1650000\tcost: 4.07\n",
      "Update 1660000\tcost: 4.02\n",
      "Update 1670000\tcost: 4.04\n",
      "Update 1680000\tcost: 4.09\n",
      "Update 1690000\tcost: 4.07\n",
      "Update 1700000\tcost: 4.04\n",
      "Update 1710000\tcost: 4.05\n",
      "Update 1720000\tcost: 4.02\n",
      "Update 1730000\tcost: 4.01\n",
      "Update 1740000\tcost: 4.05\n",
      "Update 1750000\tcost: 4.07\n",
      "Update 1760000\tcost: 4.00\n",
      "Update 1770000\tcost: 3.98\n",
      "Update 1780000\tcost: 3.97\n",
      "Update 1790000\tcost: 3.96\n",
      "Update 1800000\tcost: 4.01\n",
      "Update 1810000\tcost: 4.02\n",
      "Update 1820000\tcost: 3.97\n",
      "Update 1830000\tcost: 3.98\n",
      "Update 1840000\tcost: 3.98\n",
      "Update 1850000\tcost: 3.94\n",
      "Update 1860000\tcost: 3.95\n",
      "Update 1870000\tcost: 3.96\n",
      "Update 1880000\tcost: 3.98\n",
      "Update 1890000\tcost: 3.96\n",
      "Update 1900000\tcost: 3.94\n",
      "Update 1910000\tcost: 3.94\n",
      "Update 1920000\tcost: 3.94\n",
      "Update 1930000\tcost: 3.96\n",
      "Update 1940000\tcost: 3.92\n",
      "Update 1950000\tcost: 3.88\n",
      "Update 1960000\tcost: 3.92\n",
      "Update 1970000\tcost: 3.87\n",
      "Update 1980000\tcost: 3.90\n",
      "Update 1990000\tcost: 3.89\n",
      "Update 2000000\tcost: 3.92\n",
      "Update 2010000\tcost: 3.90\n",
      "Update 2020000\tcost: 3.89\n",
      "Update 2030000\tcost: 3.87\n",
      "Update 2040000\tcost: 3.94\n",
      "Update 2050000\tcost: 3.88\n",
      "Update 2060000\tcost: 3.90\n",
      "Update 2070000\tcost: 3.89\n",
      "Update 2080000\tcost: 3.88\n",
      "Update 2090000\tcost: 3.88\n",
      "Update 2100000\tcost: 3.87\n",
      "Update 2110000\tcost: 3.86\n",
      "Update 2120000\tcost: 3.87\n",
      "Update 2130000\tcost: 3.85\n",
      "Update 2140000\tcost: 3.85\n",
      "Update 2150000\tcost: 3.82\n",
      "Update 2160000\tcost: 3.84\n",
      "Update 2170000\tcost: 3.84\n",
      "Update 2180000\tcost: 3.83\n",
      "Update 2190000\tcost: 3.82\n",
      "Update 2200000\tcost: 3.86\n",
      "Update 2210000\tcost: 3.85\n",
      "Update 2220000\tcost: 3.84\n",
      "Update 2230000\tcost: 3.79\n",
      "Update 2240000\tcost: 3.85\n",
      "Update 2250000\tcost: 3.85\n",
      "Update 2260000\tcost: 3.82\n",
      "Update 2270000\tcost: 3.80\n",
      "Update 2280000\tcost: 3.79\n",
      "Update 2290000\tcost: 3.80\n",
      "Update 2300000\tcost: 3.81\n",
      "Update 2310000\tcost: 3.81\n",
      "Update 2320000\tcost: 3.75\n",
      "Update 2330000\tcost: 3.81\n",
      "Update 2340000\tcost: 3.80\n",
      "Update 2350000\tcost: 3.79\n",
      "Update 2360000\tcost: 3.79\n",
      "Update 2370000\tcost: 3.77\n",
      "Update 2380000\tcost: 3.80\n",
      "Update 2390000\tcost: 3.79\n",
      "Update 2400000\tcost: 3.78\n",
      "Update 2410000\tcost: 3.75\n",
      "Update 2420000\tcost: 3.78\n",
      "Update 2430000\tcost: 3.74\n",
      "Update 2440000\tcost: 3.80\n",
      "Update 2450000\tcost: 3.77\n",
      "Update 2460000\tcost: 3.75\n",
      "Update 2470000\tcost: 3.74\n",
      "Update 2480000\tcost: 3.77\n",
      "Update 2490000\tcost: 3.72\n",
      "Update 2500000\tcost: 3.72\n",
      "Update 2510000\tcost: 3.76\n",
      "Update 2520000\tcost: 3.75\n",
      "Update 2530000\tcost: 3.75\n",
      "Update 2540000\tcost: 3.73\n",
      "Update 2550000\tcost: 3.72\n",
      "Update 2560000\tcost: 3.73\n",
      "Update 2570000\tcost: 3.74\n",
      "Update 2580000\tcost: 3.74\n",
      "Update 2590000\tcost: 3.75\n",
      "Update 2600000\tcost: 3.73\n",
      "Update 2610000\tcost: 3.72\n",
      "Update 2620000\tcost: 3.69\n",
      "Update 2630000\tcost: 3.70\n",
      "Update 2640000\tcost: 3.72\n",
      "Update 2650000\tcost: 3.71\n",
      "Update 2660000\tcost: 3.70\n",
      "Update 2670000\tcost: 3.74\n",
      "Update 2680000\tcost: 3.68\n",
      "Update 2690000\tcost: 3.69\n",
      "Update 2700000\tcost: 3.70\n",
      "Update 2710000\tcost: 3.72\n",
      "Update 2720000\tcost: 3.69\n",
      "Update 2730000\tcost: 3.70\n",
      "Update 2740000\tcost: 3.71\n",
      "Update 2750000\tcost: 3.67\n",
      "Update 2760000\tcost: 3.67\n",
      "Update 2770000\tcost: 3.69\n",
      "Update 2780000\tcost: 3.73\n",
      "Update 2790000\tcost: 3.68\n",
      "Update 2800000\tcost: 3.69\n",
      "Update 2810000\tcost: 3.71\n",
      "Update 2820000\tcost: 3.67\n",
      "Update 2830000\tcost: 3.64\n",
      "Update 2840000\tcost: 3.69\n",
      "Update 2850000\tcost: 3.63\n",
      "Update 2860000\tcost: 3.68\n",
      "Update 2870000\tcost: 3.70\n",
      "Update 2880000\tcost: 3.68\n",
      "Update 2890000\tcost: 3.67\n",
      "Update 2900000\tcost: 3.68\n",
      "Update 2910000\tcost: 3.67\n",
      "Update 2920000\tcost: 3.69\n",
      "Update 2930000\tcost: 3.65\n",
      "Update 2940000\tcost: 3.66\n",
      "Update 2950000\tcost: 3.65\n",
      "Update 2960000\tcost: 3.68\n",
      "Update 2970000\tcost: 3.64\n",
      "Update 2980000\tcost: 3.63\n",
      "Update 2990000\tcost: 3.67\n",
      "Update 3000000\tcost: 3.62\n",
      "Update 3010000\tcost: 3.61\n",
      "Update 3020000\tcost: 3.60\n",
      "Update 3030000\tcost: 3.63\n",
      "Update 3040000\tcost: 3.64\n",
      "Update 3050000\tcost: 3.63\n",
      "Update 3060000\tcost: 3.61\n",
      "Update 3070000\tcost: 3.63\n",
      "Update 3080000\tcost: 3.60\n",
      "Update 3090000\tcost: 3.63\n",
      "Update 3100000\tcost: 3.63\n",
      "Update 3110000\tcost: 3.62\n",
      "Update 3120000\tcost: 3.63\n",
      "Update 3130000\tcost: 3.61\n",
      "Update 3140000\tcost: 3.61\n",
      "Update 3150000\tcost: 3.60\n",
      "Update 3160000\tcost: 3.61\n",
      "Update 3170000\tcost: 3.59\n",
      "Update 3180000\tcost: 3.60\n",
      "Update 3190000\tcost: 3.61\n",
      "Update 3200000\tcost: 3.58\n",
      "Update 3210000\tcost: 3.62\n",
      "Update 3220000\tcost: 3.59\n",
      "Update 3230000\tcost: 3.61\n",
      "Update 3240000\tcost: 3.61\n",
      "Update 3250000\tcost: 3.57\n",
      "Update 3260000\tcost: 3.60\n",
      "Update 3270000\tcost: 3.61\n",
      "Update 3280000\tcost: 3.58\n",
      "Update 3290000\tcost: 3.63\n",
      "Update 3300000\tcost: 3.59\n",
      "Update 3310000\tcost: 3.57\n",
      "Update 3320000\tcost: 3.59\n",
      "Update 3330000\tcost: 3.58\n",
      "Update 3340000\tcost: 3.57\n",
      "Update 3350000\tcost: 3.59\n",
      "Update 3360000\tcost: 3.59\n",
      "Update 3370000\tcost: 3.61\n",
      "Update 3380000\tcost: 3.59\n",
      "Update 3390000\tcost: 3.56\n",
      "Update 3400000\tcost: 3.58\n",
      "Update 3410000\tcost: 3.59\n",
      "Update 3420000\tcost: 3.57\n",
      "Update 3430000\tcost: 3.58\n",
      "Update 3440000\tcost: 3.58\n",
      "Update 3450000\tcost: 3.58\n",
      "Update 3460000\tcost: 3.58\n",
      "Update 3470000\tcost: 3.57\n",
      "Update 3480000\tcost: 3.54\n",
      "Update 3490000\tcost: 3.59\n",
      "Update 3500000\tcost: 3.55\n",
      "Update 3510000\tcost: 3.58\n",
      "Update 3520000\tcost: 3.54\n",
      "Update 3530000\tcost: 3.60\n",
      "Update 3540000\tcost: 3.61\n",
      "Update 3550000\tcost: 3.56\n",
      "Update 3560000\tcost: 3.56\n",
      "Update 3570000\tcost: 3.57\n",
      "Update 3580000\tcost: 3.52\n",
      "Update 3590000\tcost: 3.55\n",
      "Update 3600000\tcost: 3.55\n",
      "Update 3610000\tcost: 3.59\n",
      "Update 3620000\tcost: 3.55\n",
      "Update 3630000\tcost: 3.53\n",
      "Update 3640000\tcost: 3.56\n",
      "Update 3650000\tcost: 3.56\n",
      "Update 3660000\tcost: 3.53\n",
      "Update 3670000\tcost: 3.57\n",
      "Update 3680000\tcost: 3.52\n",
      "Update 3690000\tcost: 3.56\n",
      "Update 3700000\tcost: 3.52\n",
      "Update 3710000\tcost: 3.52\n",
      "Update 3720000\tcost: 3.58\n",
      "Update 3730000\tcost: 3.53\n",
      "Update 3740000\tcost: 3.55\n",
      "Update 3750000\tcost: 3.53\n",
      "Update 3760000\tcost: 3.55\n",
      "Update 3770000\tcost: 3.52\n",
      "Update 3780000\tcost: 3.56\n",
      "Update 3790000\tcost: 3.53\n",
      "Update 3800000\tcost: 3.53\n",
      "Update 3810000\tcost: 3.51\n",
      "Update 3820000\tcost: 3.52\n",
      "Update 3830000\tcost: 3.51\n",
      "Update 3840000\tcost: 3.54\n",
      "Update 3850000\tcost: 3.54\n",
      "Update 3860000\tcost: 3.52\n",
      "Update 3870000\tcost: 3.55\n",
      "Update 3880000\tcost: 3.55\n",
      "Update 3890000\tcost: 3.55\n",
      "Update 3900000\tcost: 3.53\n",
      "Update 3910000\tcost: 3.55\n",
      "Update 3920000\tcost: 3.52\n",
      "Update 3930000\tcost: 3.52\n",
      "Update 3940000\tcost: 3.49\n",
      "Update 3950000\tcost: 3.50\n",
      "Update 3960000\tcost: 3.52\n",
      "Update 3970000\tcost: 3.52\n",
      "Update 3980000\tcost: 3.51\n",
      "Update 3990000\tcost: 3.53\n",
      "Update 4000000\tcost: 3.52\n",
      "Update 4010000\tcost: 3.51\n",
      "Update 4020000\tcost: 3.51\n",
      "Update 4030000\tcost: 3.56\n",
      "Update 4040000\tcost: 3.51\n",
      "Update 4050000\tcost: 3.54\n",
      "Update 4060000\tcost: 3.53\n",
      "Update 4070000\tcost: 3.53\n",
      "Update 4080000\tcost: 3.53\n",
      "Update 4090000\tcost: 3.50\n",
      "Update 4100000\tcost: 3.52\n",
      "Update 4110000\tcost: 3.51\n",
      "Update 4120000\tcost: 3.50\n",
      "Update 4130000\tcost: 3.51\n",
      "Update 4140000\tcost: 3.49\n",
      "Update 4150000\tcost: 3.49\n",
      "Update 4160000\tcost: 3.50\n",
      "Update 4170000\tcost: 3.52\n",
      "Update 4180000\tcost: 3.50\n",
      "Update 4190000\tcost: 3.53\n",
      "Update 4200000\tcost: 3.47\n",
      "Update 4210000\tcost: 3.50\n",
      "Update 4220000\tcost: 3.51\n",
      "Update 4230000\tcost: 3.50\n",
      "Update 4240000\tcost: 3.48\n",
      "Update 4250000\tcost: 3.54\n",
      "Update 4260000\tcost: 3.50\n",
      "Update 4270000\tcost: 3.50\n",
      "Update 4280000\tcost: 3.47\n",
      "Update 4290000\tcost: 3.48\n",
      "Update 4300000\tcost: 3.49\n",
      "Update 4310000\tcost: 3.48\n",
      "Update 4320000\tcost: 3.54\n",
      "Update 4330000\tcost: 3.50\n",
      "Update 4340000\tcost: 3.47\n",
      "Update 4350000\tcost: 3.49\n",
      "Update 4360000\tcost: 3.46\n",
      "Update 4370000\tcost: 3.49\n",
      "Update 4380000\tcost: 3.50\n",
      "Update 4390000\tcost: 3.47\n",
      "Update 4400000\tcost: 3.50\n",
      "Update 4410000\tcost: 3.49\n",
      "Update 4420000\tcost: 3.48\n",
      "Update 4430000\tcost: 3.48\n",
      "Update 4440000\tcost: 3.48\n",
      "Update 4450000\tcost: 3.46\n",
      "Update 4460000\tcost: 3.50\n",
      "Update 4470000\tcost: 3.48\n",
      "Update 4480000\tcost: 3.50\n",
      "Update 4490000\tcost: 3.49\n",
      "Update 4500000\tcost: 3.50\n",
      "Update 4510000\tcost: 3.47\n",
      "Update 4520000\tcost: 3.51\n",
      "Update 4530000\tcost: 3.48\n",
      "Update 4540000\tcost: 3.49\n",
      "Update 4550000\tcost: 3.51\n",
      "Update 4560000\tcost: 3.50\n",
      "Update 4570000\tcost: 3.48\n",
      "Update 4580000\tcost: 3.46\n",
      "Update 4590000\tcost: 3.47\n",
      "Update 4600000\tcost: 3.47\n",
      "Update 4610000\tcost: 3.50\n",
      "Update 4620000\tcost: 3.46\n",
      "Update 4630000\tcost: 3.47\n",
      "Update 4640000\tcost: 3.45\n",
      "Update 4650000\tcost: 3.46\n",
      "Update 4660000\tcost: 3.48\n",
      "Update 4670000\tcost: 3.46\n",
      "Update 4680000\tcost: 3.48\n",
      "Update 4690000\tcost: 3.48\n",
      "Update 4700000\tcost: 3.47\n",
      "Update 4710000\tcost: 3.46\n",
      "Update 4720000\tcost: 3.46\n",
      "Update 4730000\tcost: 3.48\n",
      "Update 4740000\tcost: 3.47\n",
      "Update 4750000\tcost: 3.48\n",
      "Update 4760000\tcost: 3.48\n",
      "Update 4770000\tcost: 3.43\n",
      "Update 4780000\tcost: 3.46\n",
      "Update 4790000\tcost: 3.48\n",
      "Update 4800000\tcost: 3.49\n",
      "Update 4810000\tcost: 3.46\n",
      "Update 4820000\tcost: 3.45\n",
      "Update 4830000\tcost: 3.47\n",
      "Update 4840000\tcost: 3.45\n",
      "Update 4850000\tcost: 3.47\n",
      "Update 4860000\tcost: 3.44\n",
      "Update 4870000\tcost: 3.46\n",
      "Update 4880000\tcost: 3.44\n",
      "Update 4890000\tcost: 3.46\n",
      "Update 4900000\tcost: 3.45\n",
      "Update 4910000\tcost: 3.44\n",
      "Update 4920000\tcost: 3.48\n",
      "Update 4930000\tcost: 3.43\n",
      "Update 4940000\tcost: 3.45\n",
      "Update 4950000\tcost: 3.45\n",
      "Update 4960000\tcost: 3.46\n",
      "Update 4970000\tcost: 3.46\n",
      "Update 4980000\tcost: 3.43\n",
      "Update 4990000\tcost: 3.44\n",
      "Update 5000000\tcost: 3.45\n"
     ]
    }
   ],
   "source": [
    "Vp, Vo, J = neg_sample(conf, train_set, train_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lookup_word_idx(word, word_dict):\n",
    "    try:\n",
    "        return np.argwhere(np.array(word_dict) == word)[0][0]\n",
    "    except:\n",
    "        raise Exception(\"No such word in dict: {}\".format(word))\n",
    "\n",
    "def similar_words(embeddings, word, word_dict, hits):\n",
    "    word_idx = lookup_word_idx(word, word_dict)\n",
    "    similarity_scores = embeddings @ embeddings[word_idx]\n",
    "    similar_word_idxs = np.argsort(-similarity_scores)    \n",
    "    return [word_dict[i] for i in similar_word_idxs[:hits]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training cost: 3.45\n",
      "\n",
      "\n",
      "Words similar to zero: zero, four, five, three, eight\n",
      "Words similar to computer: computer, computers, systems, software, digital\n",
      "Words similar to cars: cars, performance, even, amount, rare\n",
      "Words similar to home: home, first, well, company, team\n",
      "Words similar to album: album, song, released, band, music\n"
     ]
    }
   ],
   "source": [
    "print('\\n\\nTraining cost: {0:>2.2f}\\n\\n'.format(J))\n",
    "\n",
    "sample_words = ['zero', 'computer', 'cars', 'home', 'album']\n",
    "\n",
    "Vp_norm = Vp / as_matrix(np.linalg.norm(Vp , axis=1))\n",
    "for w in sample_words:\n",
    "    similar = similar_words(Vp_norm, w, train_dict, 5)\n",
    "    print('Words similar to {}: {}'.format(w, \", \".join(similar)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Similarity measure through vectors\n",
    "\n",
    "SpaCy already has words vectorized and we can simply check the similarity between two sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "0.7141499079552157\n",
      "0.6306364163302318\n",
      "0.7141499079552157\n",
      "1.0\n",
      "0.5037601918826777\n",
      "0.6306364163302318\n",
      "0.5037601918826777\n",
      "1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/lib/python3.6/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/usr/lib/python3.6/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/usr/lib/python3.6/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/usr/lib/python3.6/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/usr/lib/python3.6/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/usr/lib/python3.6/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load('en')\n",
    "\n",
    "doc1 = nlp(u\"Warsaw is the largest city in Poland.\")\n",
    "doc2 = nlp(u\"Crossaint is baked in France.\")\n",
    "doc3 = nlp(u\"An emu is a large bird.\")\n",
    "\n",
    "for doc in [doc1, doc2, doc3]:\n",
    "    for other_doc in [doc1, doc2, doc3]:\n",
    "        print(doc.similarity(other_doc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A nice example of word vectorization done by some researchers at Warsaw University: [Word2Vec](https://lamyiowce.github.io/word2viz/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieval-based chatbot\n",
    "\n",
    "In this section we use Rasa to build a very simple HR assistant bot. We can use Rasa as a server or use it directly from Python level. Before we start the bot, let's create a directory where the new project will be placed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir assistant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a new project with the ``init`` command. The command create the project structure and train the model with the default intents and stories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92mWelcome to Rasa! ����\n",
      "\u001b[0m\n",
      "To get started quickly, an initial project will be created.\n",
      "If you need some help, check out the documentation at https://rasa.com/docs/rasa.\n",
      "\n",
      "Created project directory at '/home/codete/workshop/assistant'.\n",
      "\u001b[92mFinished creating project structure.\u001b[0m\n",
      "\u001b[92mTraining an initial model...\u001b[0m\n",
      "\u001b[94mTraining Core model...\u001b[0m\n",
      "2019-11-18 12:49:45 \u001b[1;30mINFO    \u001b[0m \u001b[34mabsl\u001b[0m  - Entry Point [tensor2tensor.envs.tic_tac_toe_env:TicTacToeEnv] registered with id [T2TEnv-TicTacToeEnv-v0]\n",
      "Processed Story Blocks: 100%|������������������| 5/5 [00:00<00:00, 722.63it/s, # trackers=1]\n",
      "Processed Story Blocks: 100%|������������������| 5/5 [00:00<00:00, 387.74it/s, # trackers=5]\n",
      "Processed Story Blocks: 100%|���������������| 5/5 [00:00<00:00, 105.34it/s, # trackers=20]\n",
      "Processed Story Blocks: 100%|���������������| 5/5 [00:00<00:00, 129.27it/s, # trackers=24]\n",
      "Processed trackers: 100%|������������������������������| 5/5 [00:00<00:00, 277.12it/s, # actions=16]\n",
      "Processed actions: 16it [00:00, 607.27it/s, # examples=16]\n",
      "Processed trackers: 100%|���������������| 231/231 [00:02<00:00, 104.40it/s, # actions=126]\n",
      "2019-11-18 12:49:48.561009: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "masking (Masking)            (None, 5, 21)             0         \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 32)                6912      \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 14)                462       \n",
      "_________________________________________________________________\n",
      "activation (Activation)      (None, 14)                0         \n",
      "=================================================================\n",
      "Total params: 7,374\n",
      "Trainable params: 7,374\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "2019-11-18 12:49:50 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.core.policies.keras_policy\u001b[0m  - Fitting model with 126 total samples and a validation split of 0.1\n",
      "Train on 126 samples\n",
      "Epoch 1/100\n",
      "126/126 [==============================] - 1s 9ms/sample - loss: 2.6296 - acc: 0.1349\n",
      "Epoch 2/100\n",
      "126/126 [==============================] - 0s 247us/sample - loss: 2.5672 - acc: 0.2540\n",
      "Epoch 3/100\n",
      "126/126 [==============================] - 0s 242us/sample - loss: 2.5075 - acc: 0.4048\n",
      "Epoch 4/100\n",
      "126/126 [==============================] - 0s 283us/sample - loss: 2.4435 - acc: 0.5000\n",
      "Epoch 5/100\n",
      "126/126 [==============================] - 0s 412us/sample - loss: 2.3827 - acc: 0.5000\n",
      "Epoch 6/100\n",
      "126/126 [==============================] - 0s 388us/sample - loss: 2.3253 - acc: 0.4841\n",
      "Epoch 7/100\n",
      "126/126 [==============================] - 0s 364us/sample - loss: 2.2461 - acc: 0.4683\n",
      "Epoch 8/100\n",
      "126/126 [==============================] - 0s 291us/sample - loss: 2.1399 - acc: 0.4683\n",
      "Epoch 9/100\n",
      "126/126 [==============================] - 0s 309us/sample - loss: 2.0486 - acc: 0.4683\n",
      "Epoch 10/100\n",
      "126/126 [==============================] - 0s 322us/sample - loss: 1.9558 - acc: 0.4603\n",
      "Epoch 11/100\n",
      "126/126 [==============================] - 0s 350us/sample - loss: 1.8631 - acc: 0.4683\n",
      "Epoch 12/100\n",
      "126/126 [==============================] - 0s 281us/sample - loss: 1.8179 - acc: 0.4603\n",
      "Epoch 13/100\n",
      "126/126 [==============================] - 0s 283us/sample - loss: 1.7442 - acc: 0.4603\n",
      "Epoch 14/100\n",
      "126/126 [==============================] - 0s 365us/sample - loss: 1.6971 - acc: 0.4603\n",
      "Epoch 15/100\n",
      "126/126 [==============================] - 0s 403us/sample - loss: 1.6693 - acc: 0.4603\n",
      "Epoch 16/100\n",
      "126/126 [==============================] - 0s 340us/sample - loss: 1.6729 - acc: 0.4603\n",
      "Epoch 17/100\n",
      "126/126 [==============================] - 0s 309us/sample - loss: 1.6424 - acc: 0.4603\n",
      "Epoch 18/100\n",
      "126/126 [==============================] - 0s 222us/sample - loss: 1.6197 - acc: 0.4603\n",
      "Epoch 19/100\n",
      "126/126 [==============================] - 0s 358us/sample - loss: 1.5916 - acc: 0.4603\n",
      "Epoch 20/100\n",
      "126/126 [==============================] - 0s 411us/sample - loss: 1.5777 - acc: 0.4603\n",
      "Epoch 21/100\n",
      "126/126 [==============================] - 0s 1ms/sample - loss: 1.5753 - acc: 0.4683\n",
      "Epoch 22/100\n",
      "126/126 [==============================] - 0s 407us/sample - loss: 1.5521 - acc: 0.4603\n",
      "Epoch 23/100\n",
      "126/126 [==============================] - 0s 808us/sample - loss: 1.5354 - acc: 0.4683\n",
      "Epoch 24/100\n",
      "126/126 [==============================] - 0s 599us/sample - loss: 1.4939 - acc: 0.4683\n",
      "Epoch 25/100\n",
      "126/126 [==============================] - 0s 795us/sample - loss: 1.4967 - acc: 0.4683\n",
      "Epoch 26/100\n",
      "126/126 [==============================] - 0s 416us/sample - loss: 1.4874 - acc: 0.4603\n",
      "Epoch 27/100\n",
      "126/126 [==============================] - 0s 759us/sample - loss: 1.4678 - acc: 0.4603\n",
      "Epoch 28/100\n",
      "126/126 [==============================] - 0s 802us/sample - loss: 1.4341 - acc: 0.4683\n",
      "Epoch 29/100\n",
      "126/126 [==============================] - 0s 346us/sample - loss: 1.4381 - acc: 0.4683\n",
      "Epoch 30/100\n",
      "126/126 [==============================] - 0s 622us/sample - loss: 1.4031 - acc: 0.4683\n",
      "Epoch 31/100\n",
      "126/126 [==============================] - 0s 327us/sample - loss: 1.3868 - acc: 0.4683\n",
      "Epoch 32/100\n",
      "126/126 [==============================] - 0s 309us/sample - loss: 1.3728 - acc: 0.4762\n",
      "Epoch 33/100\n",
      "126/126 [==============================] - 0s 293us/sample - loss: 1.3316 - acc: 0.4762\n",
      "Epoch 34/100\n",
      "126/126 [==============================] - 0s 264us/sample - loss: 1.3342 - acc: 0.4762\n",
      "Epoch 35/100\n",
      "126/126 [==============================] - 0s 287us/sample - loss: 1.3013 - acc: 0.4683\n",
      "Epoch 36/100\n",
      "126/126 [==============================] - 0s 280us/sample - loss: 1.3142 - acc: 0.4683\n",
      "Epoch 37/100\n",
      "126/126 [==============================] - 0s 332us/sample - loss: 1.2711 - acc: 0.4762\n",
      "Epoch 38/100\n",
      "126/126 [==============================] - 0s 262us/sample - loss: 1.2337 - acc: 0.4762\n",
      "Epoch 39/100\n",
      "126/126 [==============================] - 0s 223us/sample - loss: 1.2247 - acc: 0.4683\n",
      "Epoch 40/100\n",
      "126/126 [==============================] - 0s 271us/sample - loss: 1.2336 - acc: 0.4841\n",
      "Epoch 41/100\n",
      "126/126 [==============================] - 0s 254us/sample - loss: 1.2119 - acc: 0.4841\n",
      "Epoch 42/100\n",
      "126/126 [==============================] - 0s 192us/sample - loss: 1.1768 - acc: 0.4841\n",
      "Epoch 43/100\n",
      "126/126 [==============================] - 0s 313us/sample - loss: 1.1687 - acc: 0.4921\n",
      "Epoch 44/100\n",
      "126/126 [==============================] - 0s 790us/sample - loss: 1.1460 - acc: 0.5079\n",
      "Epoch 45/100\n",
      "126/126 [==============================] - 0s 476us/sample - loss: 1.1308 - acc: 0.5397\n",
      "Epoch 46/100\n",
      "126/126 [==============================] - 0s 1ms/sample - loss: 1.1140 - acc: 0.5476\n",
      "Epoch 47/100\n",
      "126/126 [==============================] - 0s 589us/sample - loss: 1.0832 - acc: 0.5714\n",
      "Epoch 48/100\n",
      "126/126 [==============================] - 0s 484us/sample - loss: 1.0726 - acc: 0.5714\n",
      "Epoch 49/100\n",
      "126/126 [==============================] - 0s 480us/sample - loss: 1.0536 - acc: 0.5873\n",
      "Epoch 50/100\n",
      "126/126 [==============================] - 0s 662us/sample - loss: 1.0353 - acc: 0.5556\n",
      "Epoch 51/100\n",
      "126/126 [==============================] - 0s 855us/sample - loss: 1.0285 - acc: 0.6111\n",
      "Epoch 52/100\n",
      "126/126 [==============================] - 0s 788us/sample - loss: 1.0130 - acc: 0.6190\n",
      "Epoch 53/100\n",
      "126/126 [==============================] - 0s 1ms/sample - loss: 0.9740 - acc: 0.6746\n",
      "Epoch 54/100\n",
      "126/126 [==============================] - 0s 1ms/sample - loss: 0.9796 - acc: 0.6667\n",
      "Epoch 55/100\n",
      "126/126 [==============================] - 0s 3ms/sample - loss: 0.9623 - acc: 0.6984\n",
      "Epoch 56/100\n",
      "126/126 [==============================] - 0s 1ms/sample - loss: 0.9203 - acc: 0.7143\n",
      "Epoch 57/100\n",
      "126/126 [==============================] - 0s 920us/sample - loss: 0.9176 - acc: 0.7222\n",
      "Epoch 58/100\n",
      "126/126 [==============================] - 0s 2ms/sample - loss: 0.8512 - acc: 0.8175\n",
      "Epoch 59/100\n",
      "126/126 [==============================] - 0s 1ms/sample - loss: 0.9149 - acc: 0.7778\n",
      "Epoch 60/100\n",
      "126/126 [==============================] - 0s 938us/sample - loss: 0.8095 - acc: 0.8413\n",
      "Epoch 61/100\n",
      "126/126 [==============================] - 0s 943us/sample - loss: 0.8099 - acc: 0.8651\n",
      "Epoch 62/100\n",
      "126/126 [==============================] - 0s 1ms/sample - loss: 0.8099 - acc: 0.8492\n",
      "Epoch 63/100\n",
      "126/126 [==============================] - 0s 684us/sample - loss: 0.8172 - acc: 0.8413\n",
      "Epoch 64/100\n",
      "126/126 [==============================] - 0s 393us/sample - loss: 0.8112 - acc: 0.8175\n",
      "Epoch 65/100\n",
      "126/126 [==============================] - 0s 790us/sample - loss: 0.7597 - acc: 0.8492\n",
      "Epoch 66/100\n",
      "126/126 [==============================] - 0s 361us/sample - loss: 0.7512 - acc: 0.8889\n",
      "Epoch 67/100\n",
      "126/126 [==============================] - 0s 189us/sample - loss: 0.7457 - acc: 0.8571\n",
      "Epoch 68/100\n",
      "126/126 [==============================] - 0s 529us/sample - loss: 0.7125 - acc: 0.9127\n",
      "Epoch 69/100\n",
      "126/126 [==============================] - 0s 193us/sample - loss: 0.6725 - acc: 0.9365\n",
      "Epoch 70/100\n",
      "126/126 [==============================] - 0s 212us/sample - loss: 0.6620 - acc: 0.9206\n",
      "Epoch 71/100\n",
      "126/126 [==============================] - 0s 204us/sample - loss: 0.6621 - acc: 0.8730\n",
      "Epoch 72/100\n",
      "126/126 [==============================] - 0s 278us/sample - loss: 0.6348 - acc: 0.8968\n",
      "Epoch 73/100\n",
      "126/126 [==============================] - 0s 359us/sample - loss: 0.6213 - acc: 0.9286\n",
      "Epoch 74/100\n",
      "126/126 [==============================] - 0s 1ms/sample - loss: 0.6296 - acc: 0.8968\n",
      "Epoch 75/100\n",
      "126/126 [==============================] - 0s 947us/sample - loss: 0.5994 - acc: 0.9206\n",
      "Epoch 76/100\n",
      "126/126 [==============================] - 0s 244us/sample - loss: 0.6000 - acc: 0.8810\n",
      "Epoch 77/100\n",
      "126/126 [==============================] - 0s 797us/sample - loss: 0.5803 - acc: 0.9444\n",
      "Epoch 78/100\n",
      "126/126 [==============================] - 0s 774us/sample - loss: 0.5645 - acc: 0.9048\n",
      "Epoch 79/100\n",
      "126/126 [==============================] - 0s 969us/sample - loss: 0.5125 - acc: 0.9524\n",
      "Epoch 80/100\n",
      "126/126 [==============================] - 0s 756us/sample - loss: 0.5205 - acc: 0.9444\n",
      "Epoch 81/100\n",
      "126/126 [==============================] - 0s 567us/sample - loss: 0.4864 - acc: 0.9603\n",
      "Epoch 82/100\n",
      "126/126 [==============================] - 0s 649us/sample - loss: 0.4982 - acc: 0.9683\n",
      "Epoch 83/100\n",
      "126/126 [==============================] - 0s 575us/sample - loss: 0.5145 - acc: 0.9127\n",
      "Epoch 84/100\n",
      "126/126 [==============================] - 0s 960us/sample - loss: 0.4665 - acc: 0.9603\n",
      "Epoch 85/100\n",
      "126/126 [==============================] - 0s 875us/sample - loss: 0.4385 - acc: 0.9683\n",
      "Epoch 86/100\n",
      "126/126 [==============================] - 0s 484us/sample - loss: 0.4506 - acc: 0.9524\n",
      "Epoch 87/100\n",
      "126/126 [==============================] - 0s 927us/sample - loss: 0.4325 - acc: 0.9683\n",
      "Epoch 88/100\n",
      "126/126 [==============================] - 0s 731us/sample - loss: 0.4345 - acc: 0.9206\n",
      "Epoch 89/100\n",
      "126/126 [==============================] - 0s 557us/sample - loss: 0.4053 - acc: 0.9683\n",
      "Epoch 90/100\n",
      "126/126 [==============================] - 0s 687us/sample - loss: 0.4453 - acc: 0.9524\n",
      "Epoch 91/100\n",
      "126/126 [==============================] - 0s 318us/sample - loss: 0.4074 - acc: 0.9444\n",
      "Epoch 92/100\n",
      "126/126 [==============================] - 0s 274us/sample - loss: 0.4276 - acc: 0.9206\n",
      "Epoch 93/100\n",
      "126/126 [==============================] - 0s 298us/sample - loss: 0.3845 - acc: 0.9603\n",
      "Epoch 94/100\n",
      "126/126 [==============================] - 0s 538us/sample - loss: 0.3963 - acc: 0.9444\n",
      "Epoch 95/100\n",
      "126/126 [==============================] - 0s 998us/sample - loss: 0.3837 - acc: 0.9444\n",
      "Epoch 96/100\n",
      "126/126 [==============================] - 0s 417us/sample - loss: 0.3636 - acc: 0.9683\n",
      "Epoch 97/100\n",
      "126/126 [==============================] - 0s 273us/sample - loss: 0.3847 - acc: 0.9524\n",
      "Epoch 98/100\n",
      "126/126 [==============================] - 0s 409us/sample - loss: 0.3603 - acc: 0.9524\n",
      "Epoch 99/100\n",
      "126/126 [==============================] - 0s 304us/sample - loss: 0.3391 - acc: 0.9683\n",
      "Epoch 100/100\n",
      "126/126 [==============================] - 0s 275us/sample - loss: 0.3392 - acc: 0.9603\n",
      "2019-11-18 12:50:02 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.core.policies.keras_policy\u001b[0m  - Done fitting keras policy model\n",
      "2019-11-18 12:50:02 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.core.agent\u001b[0m  - Persisted model to '/tmp/tmpzuivfmvs/core'\n",
      "\u001b[94mCore model training completed.\u001b[0m\n",
      "\u001b[94mTraining NLU model...\u001b[0m\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.training_data.training_data\u001b[0m  - Training data stats: \n",
      "\t- intent examples: 43 (7 distinct intents)\n",
      "\t- Found intents: 'goodbye', 'bot_challenge', 'greet', 'deny', 'mood_unhappy', 'mood_great', 'affirm'\n",
      "\t- Number of response examples: 0 (0 distinct response)\n",
      "\t- entity examples: 0 (0 distinct entities)\n",
      "\t- found entities: \n",
      "\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component WhitespaceTokenizer\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component RegexFeaturizer\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component CRFEntityExtractor\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component EntitySynonymMapper\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component CountVectorsFeaturizer\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 12:50:03 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component CountVectorsFeaturizer\n",
      "2019-11-18 12:50:04 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 12:50:04 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component EmbeddingIntentClassifier\n",
      "Epochs:   0%|                                           | 0/300 [00:00<?, ?it/s]^C\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py\", line 1365, in _do_call\n",
      "    return fn(*args)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py\", line 1350, in _run_fn\n",
      "    target_list, run_metadata)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py\", line 1443, in _call_tf_sessionrun\n",
      "    run_metadata)\n",
      "tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence\n",
      "\t [[{{node IteratorGetNext}}]]\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/local/bin/rasa\", line 8, in <module>\n",
      "    sys.exit(main())\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/__main__.py\", line 76, in main\n",
      "    cmdline_arguments.func(cmdline_arguments)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/cli/scaffold.py\", line 195, in run\n",
      "    init_project(args, path)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/cli/scaffold.py\", line 113, in init_project\n",
      "    print_train_or_instructions(args, path)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/cli/scaffold.py\", line 50, in print_train_or_instructions\n",
      "    args.model = rasa.train(domain, config, training_files, output)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/train.py\", line 45, in train\n",
      "    kwargs=kwargs,\n",
      "  File \"uvloop/loop.pyx\", line 1450, in uvloop.loop.Loop.run_until_complete\n",
      "  File \"uvloop/loop.pyx\", line 1443, in uvloop.loop.Loop.run_until_complete\n",
      "  File \"uvloop/loop.pyx\", line 1351, in uvloop.loop.Loop.run_forever\n",
      "  File \"uvloop/loop.pyx\", line 519, in uvloop.loop.Loop._run\n",
      "  File \"uvloop/loop.pyx\", line 436, in uvloop.loop.Loop._on_idle\n",
      "  File \"uvloop/cbhandles.pyx\", line 90, in uvloop.loop.Handle._run\n",
      "  File \"uvloop/cbhandles.pyx\", line 70, in uvloop.loop.Handle._run\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/train.py\", line 96, in train_async\n",
      "    kwargs,\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/train.py\", line 182, in _train_async_internal\n",
      "    kwargs=kwargs,\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/train.py\", line 231, in _do_training\n",
      "    persist_nlu_training_data=persist_nlu_training_data,\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/train.py\", line 459, in _train_nlu_with_validated_data\n",
      "    persist_nlu_training_data=persist_nlu_training_data,\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/nlu/train.py\", line 80, in train\n",
      "    interpreter = trainer.train(training_data, **kwargs)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/nlu/model.py\", line 195, in train\n",
      "    updates = component.train(working_data, self.config, **context)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/nlu/classifiers/embedding_intent_classifier.py\", line 574, in train\n",
      "    self.evaluate_every_num_epochs,\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/rasa/utils/train_utils.py\", line 891, in train_tf_dataset\n",
      "    [train_op, loss, acc], feed_dict={is_training: True}\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py\", line 956, in run\n",
      "    run_metadata_ptr)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py\", line 1180, in _run\n",
      "    feed_dict_tensor, options, run_metadata)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py\", line 1359, in _do_run\n",
      "    run_metadata)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py\", line 1378, in _do_call\n",
      "    message = error_interpolation.interpolate(message, self._graph)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/error_interpolation.py\", line 454, in interpolate\n",
      "    common_prefix = traceback_files_common_prefix(tagged_ops)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/error_interpolation.py\", line 358, in traceback_files_common_prefix\n",
      "    for frame in op.traceback:\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py\", line 2309, in traceback\n",
      "    return tf_stack.convert_stack(self._traceback)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/tf_stack.py\", line 199, in convert_stack\n",
      "    return tuple(_tuple_generator())\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/tf_stack.py\", line 189, in _tuple_generator\n",
      "    line = linecache.getline(filename, lineno, frame.globals)\n",
      "  File \"/usr/lib/python3.6/linecache.py\", line 16, in getline\n",
      "    lines = getlines(filename, module_globals)\n",
      "  File \"/usr/lib/python3.6/linecache.py\", line 47, in getlines\n",
      "    return updatecache(filename, module_globals)\n",
      "  File \"/usr/lib/python3.6/linecache.py\", line 136, in updatecache\n",
      "    with tokenize.open(fullname) as fp:\n",
      "  File \"/usr/lib/python3.6/tokenize.py\", line 454, in open\n",
      "    encoding, lines = detect_encoding(buffer.readline)\n",
      "  File \"/usr/lib/python3.6/tokenize.py\", line 423, in detect_encoding\n",
      "    first = read_or_stop()\n",
      "  File \"/usr/lib/python3.6/tokenize.py\", line 381, in read_or_stop\n",
      "    return readline()\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/socketio/client.py\", line 25, in signal_handler\n",
      "    return original_signal_handler(sig, frame)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/engineio/client.py\", line 43, in signal_handler\n",
      "    return original_signal_handler(sig, frame)\n",
      "KeyboardInterrupt\n"
     ]
    }
   ],
   "source": [
    "!cd assistant && PYTHONIOENCODING='utf8' rasa init --no-prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The structure of the created project looks like following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__init__.py  actions.py  credentials.yml  domain.yml\r\n",
      "__pycache__  config.yml  data\t\t  endpoints.yml\r\n"
     ]
    }
   ],
   "source": [
    "!ls assistant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two directories: \n",
    "\n",
    "- ``data`` where all intents and stories are placed,\n",
    "- ``models`` where trained models are placed.\n",
    "\n",
    "The intents are stored in ``nlu.md`` and stories in ``stories.md``. Models are stored as ``.tar.gz`` files with the data in the name. \n",
    "\n",
    "Other files that are in the main directory are:\n",
    "\n",
    "- actions.py - contains custom actions,\n",
    "- config.yml - the pipeline, language and other configuration details,\n",
    "- credentials.yml - credentials to platforms like messenger, slack, and other,\n",
    "- domain.yml - utters set,\n",
    "- endpoints.yml - endpoints that the bot can use to get data from outside sources.\n",
    "\n",
    "We can start a server on default port ``5005``. You can test it using the request package. We should get the intent of the phrase `hi`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-11-18 12:50:15 \u001b[1;30mINFO    \u001b[0m \u001b[34mroot\u001b[0m  - Starting Rasa server on http://localhost:5005\n",
      "2019-11-18 12:50:15 \u001b[1;30mWARNING \u001b[0m \u001b[34mrasa.core.agent\u001b[0m  - \u001b[33mCould not load local model in 'models'\u001b[0m\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!cd assistant && rasa run --enable-api"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can invoke the ``model/parse`` API to find the intent with the code below. We can find all other API methods that can be called in the documentation: [https://rasa.com/docs/rasa/api/http-api/](https://rasa.com/docs/rasa/api/http-api/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': '/hi',\n",
       " 'intent': {'name': 'hi', 'confidence': 1.0},\n",
       " 'intent_ranking': [{'name': 'hi', 'confidence': 1.0}],\n",
       " 'entities': []}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "def get_intent(sentence):\n",
    "    url = \"http://localhost:5005/model/parse\"\n",
    "    payload = {'text':sentence}\n",
    "    response = requests.post(url,json=payload)\n",
    "    return response.json()\n",
    "\n",
    "get_intent(\"hi\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The intents can be defined in ``nlu.md`` as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## intent:greet\r\n",
      "- hey\r\n",
      "- hello\r\n",
      "- hi\r\n",
      "- good morning\r\n",
      "- good evening\r\n",
      "- hey there\r\n",
      "\r\n",
      "## intent:goodbye\r\n",
      "- bye\r\n",
      "- goodbye\r\n",
      "- see you around\r\n",
      "- see you later\r\n",
      "\r\n",
      "## intent:affirm\r\n",
      "- yes\r\n",
      "- indeed\r\n",
      "- of course\r\n",
      "- that sounds good\r\n",
      "- correct\r\n",
      "\r\n",
      "## intent:deny\r\n",
      "- no\r\n",
      "- never\r\n",
      "- I don't think so\r\n",
      "- don't like that\r\n",
      "- no way\r\n",
      "- not really\r\n",
      "\r\n",
      "## intent:mood_great\r\n",
      "- perfect\r\n",
      "- very good\r\n",
      "- great\r\n",
      "- amazing\r\n",
      "- wonderful\r\n",
      "- I am feeling very good\r\n",
      "- I am great\r\n",
      "- I'm good\r\n",
      "\r\n",
      "## intent:mood_unhappy\r\n",
      "- sad\r\n",
      "- very sad\r\n",
      "- unhappy\r\n",
      "- bad\r\n",
      "- very bad\r\n",
      "- awful\r\n",
      "- terrible\r\n",
      "- not very good\r\n",
      "- extremely sad\r\n",
      "- so sad\r\n",
      "\r\n",
      "## intent:bot_challenge\r\n",
      "- are you a bot?\r\n",
      "- are you a human?\r\n",
      "- am I talking to a bot?\r\n",
      "- am I talking to a human?\r\n"
     ]
    }
   ],
   "source": [
    "! cat assistant/data/nlu.md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We can add more examples by appending other intents as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Appending to /home/codete/workshop/assistant/data/nlu.md\n"
     ]
    }
   ],
   "source": [
    "%%writefile -a /home/codete/workshop/assistant/data/nlu.md\n",
    "\n",
    "## synonym:candidate\n",
    "- candidate\n",
    "- developer\n",
    "- data scientist\n",
    "\n",
    "## intent:add_candidate\n",
    "- add [candidate](candidate)\n",
    "- adding [candidate](candidate)\n",
    "- please add [candidate](candidate)\n",
    "- please add new [candidate](candidate)\n",
    "- we have new prescreening upcoming\n",
    "- we have a new candidate [candidate](candidate) prescreening"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training is straight forward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[94mTraining Core model...\u001b[0m\n",
      "2019-11-18 13:15:00 \u001b[1;30mINFO    \u001b[0m \u001b[34mabsl\u001b[0m  - Entry Point [tensor2tensor.envs.tic_tac_toe_env:TicTacToeEnv] registered with id [T2TEnv-TicTacToeEnv-v0]\n",
      "Processed Story Blocks: 100%|���������������| 5/5 [00:00<00:00, 1192.58it/s, # trackers=1]\n",
      "Processed Story Blocks: 100%|������������������| 5/5 [00:00<00:00, 302.79it/s, # trackers=5]\n",
      "Processed Story Blocks: 100%|���������������| 5/5 [00:00<00:00, 237.36it/s, # trackers=20]\n",
      "Processed Story Blocks: 100%|���������������| 5/5 [00:00<00:00, 127.44it/s, # trackers=24]\n",
      "Processed trackers: 100%|������������������������������| 5/5 [00:00<00:00, 359.52it/s, # actions=16]\n",
      "Processed actions: 16it [00:00, 5054.52it/s, # examples=16]\n",
      "Processed trackers: 100%|���������������| 231/231 [00:00<00:00, 310.22it/s, # actions=126]18]60.94it/s, # actions=126]\n",
      "2019-11-18 13:15:01.144663: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "masking (Masking)            (None, 5, 21)             0         \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 32)                6912      \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 14)                462       \n",
      "_________________________________________________________________\n",
      "activation (Activation)      (None, 14)                0         \n",
      "=================================================================\n",
      "Total params: 7,374\n",
      "Trainable params: 7,374\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "2019-11-18 13:15:02 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.core.policies.keras_policy\u001b[0m  - Fitting model with 126 total samples and a validation split of 0.1\n",
      "Train on 126 samples\n",
      "Epoch 1/100\n",
      "126/126 [==============================] - 1s 10ms/sample - loss: 2.6185 - acc: 0.1190\n",
      "Epoch 2/100\n",
      "126/126 [==============================] - 0s 355us/sample - loss: 2.5480 - acc: 0.3571\n",
      "Epoch 3/100\n",
      "126/126 [==============================] - 0s 587us/sample - loss: 2.5063 - acc: 0.4127\n",
      "Epoch 4/100\n",
      "126/126 [==============================] - 0s 559us/sample - loss: 2.4512 - acc: 0.4603\n",
      "Epoch 5/100\n",
      "126/126 [==============================] - 0s 486us/sample - loss: 2.3951 - acc: 0.4524\n",
      "Epoch 6/100\n",
      "126/126 [==============================] - 0s 707us/sample - loss: 2.3423 - acc: 0.4524\n",
      "Epoch 7/100\n",
      "126/126 [==============================] - 0s 483us/sample - loss: 2.2816 - acc: 0.4603\n",
      "Epoch 8/100\n",
      "126/126 [==============================] - 0s 434us/sample - loss: 2.1899 - acc: 0.4524\n",
      "Epoch 9/100\n",
      "126/126 [==============================] - 0s 634us/sample - loss: 2.1391 - acc: 0.4603\n",
      "Epoch 10/100\n",
      "126/126 [==============================] - 0s 433us/sample - loss: 2.0201 - acc: 0.4603\n",
      "Epoch 11/100\n",
      "126/126 [==============================] - 0s 458us/sample - loss: 1.9640 - acc: 0.4603\n",
      "Epoch 12/100\n",
      "126/126 [==============================] - 0s 683us/sample - loss: 1.9286 - acc: 0.4603\n",
      "Epoch 13/100\n",
      "126/126 [==============================] - 0s 814us/sample - loss: 1.8267 - acc: 0.4603\n",
      "Epoch 14/100\n",
      "126/126 [==============================] - 0s 494us/sample - loss: 1.8102 - acc: 0.4603\n",
      "Epoch 15/100\n",
      "126/126 [==============================] - 0s 506us/sample - loss: 1.7369 - acc: 0.4603\n",
      "Epoch 16/100\n",
      "126/126 [==============================] - 0s 726us/sample - loss: 1.7380 - acc: 0.4603\n",
      "Epoch 17/100\n",
      "126/126 [==============================] - 0s 784us/sample - loss: 1.6981 - acc: 0.4603\n",
      "Epoch 18/100\n",
      "126/126 [==============================] - 0s 708us/sample - loss: 1.6732 - acc: 0.4603\n",
      "Epoch 19/100\n",
      "126/126 [==============================] - 0s 510us/sample - loss: 1.6716 - acc: 0.4603\n",
      "Epoch 20/100\n",
      "126/126 [==============================] - 0s 496us/sample - loss: 1.6309 - acc: 0.4603\n",
      "Epoch 21/100\n",
      "126/126 [==============================] - 0s 554us/sample - loss: 1.6121 - acc: 0.4603\n",
      "Epoch 22/100\n",
      "126/126 [==============================] - 0s 447us/sample - loss: 1.5936 - acc: 0.4603\n",
      "Epoch 23/100\n",
      "126/126 [==============================] - 0s 454us/sample - loss: 1.6100 - acc: 0.4603\n",
      "Epoch 24/100\n",
      "126/126 [==============================] - 0s 444us/sample - loss: 1.5782 - acc: 0.4603\n",
      "Epoch 25/100\n",
      "126/126 [==============================] - 0s 651us/sample - loss: 1.5563 - acc: 0.4603\n",
      "Epoch 26/100\n",
      "126/126 [==============================] - 0s 475us/sample - loss: 1.5411 - acc: 0.4603\n",
      "Epoch 27/100\n",
      "126/126 [==============================] - 0s 483us/sample - loss: 1.5465 - acc: 0.4603\n",
      "Epoch 28/100\n",
      "126/126 [==============================] - 0s 289us/sample - loss: 1.5245 - acc: 0.4603\n",
      "Epoch 29/100\n",
      "126/126 [==============================] - 0s 445us/sample - loss: 1.5063 - acc: 0.4603\n",
      "Epoch 30/100\n",
      "126/126 [==============================] - 0s 441us/sample - loss: 1.5012 - acc: 0.4683\n",
      "Epoch 31/100\n",
      "126/126 [==============================] - 0s 403us/sample - loss: 1.4742 - acc: 0.4683\n",
      "Epoch 32/100\n",
      "126/126 [==============================] - 0s 651us/sample - loss: 1.4752 - acc: 0.4603\n",
      "Epoch 33/100\n",
      "126/126 [==============================] - 0s 493us/sample - loss: 1.4434 - acc: 0.4683\n",
      "Epoch 34/100\n",
      "126/126 [==============================] - 0s 927us/sample - loss: 1.4399 - acc: 0.4603\n",
      "Epoch 35/100\n",
      "126/126 [==============================] - 0s 483us/sample - loss: 1.4395 - acc: 0.4603\n",
      "Epoch 36/100\n",
      "126/126 [==============================] - 0s 400us/sample - loss: 1.4252 - acc: 0.4683\n",
      "Epoch 37/100\n",
      "126/126 [==============================] - 0s 516us/sample - loss: 1.4296 - acc: 0.4683\n",
      "Epoch 38/100\n",
      "126/126 [==============================] - 0s 449us/sample - loss: 1.3988 - acc: 0.4683\n",
      "Epoch 39/100\n",
      "126/126 [==============================] - 0s 462us/sample - loss: 1.3871 - acc: 0.4603\n",
      "Epoch 40/100\n",
      "126/126 [==============================] - 0s 515us/sample - loss: 1.3756 - acc: 0.4683\n",
      "Epoch 41/100\n",
      "126/126 [==============================] - 0s 412us/sample - loss: 1.3534 - acc: 0.4683\n",
      "Epoch 42/100\n",
      "126/126 [==============================] - 0s 424us/sample - loss: 1.3252 - acc: 0.4683\n",
      "Epoch 43/100\n",
      "126/126 [==============================] - 0s 395us/sample - loss: 1.3270 - acc: 0.4603\n",
      "Epoch 44/100\n",
      "126/126 [==============================] - 0s 387us/sample - loss: 1.3044 - acc: 0.4841\n",
      "Epoch 45/100\n",
      "126/126 [==============================] - 0s 601us/sample - loss: 1.3083 - acc: 0.4762\n",
      "Epoch 46/100\n",
      "126/126 [==============================] - 0s 394us/sample - loss: 1.3084 - acc: 0.4603\n",
      "Epoch 47/100\n",
      "126/126 [==============================] - 0s 413us/sample - loss: 1.2916 - acc: 0.4603\n",
      "Epoch 48/100\n",
      "126/126 [==============================] - 0s 446us/sample - loss: 1.2710 - acc: 0.4841\n",
      "Epoch 49/100\n",
      "126/126 [==============================] - 0s 484us/sample - loss: 1.2449 - acc: 0.5000\n",
      "Epoch 50/100\n",
      "126/126 [==============================] - 0s 518us/sample - loss: 1.2537 - acc: 0.4841\n",
      "Epoch 51/100\n",
      "126/126 [==============================] - 0s 499us/sample - loss: 1.2301 - acc: 0.4921\n",
      "Epoch 52/100\n",
      "126/126 [==============================] - 0s 404us/sample - loss: 1.2088 - acc: 0.5079\n",
      "Epoch 53/100\n",
      "126/126 [==============================] - 0s 442us/sample - loss: 1.1954 - acc: 0.5159\n",
      "Epoch 54/100\n",
      "126/126 [==============================] - 0s 415us/sample - loss: 1.1925 - acc: 0.5000\n",
      "Epoch 55/100\n",
      "126/126 [==============================] - 0s 359us/sample - loss: 1.1589 - acc: 0.5079\n",
      "Epoch 56/100\n",
      "126/126 [==============================] - 0s 389us/sample - loss: 1.1510 - acc: 0.5079\n",
      "Epoch 57/100\n",
      "126/126 [==============================] - 0s 371us/sample - loss: 1.1282 - acc: 0.5476\n",
      "Epoch 58/100\n",
      "126/126 [==============================] - 0s 213us/sample - loss: 1.1258 - acc: 0.5476\n",
      "Epoch 59/100\n",
      "126/126 [==============================] - 0s 299us/sample - loss: 1.0793 - acc: 0.5714\n",
      "Epoch 60/100\n",
      "126/126 [==============================] - 0s 245us/sample - loss: 1.0787 - acc: 0.5714\n",
      "Epoch 61/100\n",
      "126/126 [==============================] - 0s 232us/sample - loss: 1.0722 - acc: 0.5952\n",
      "Epoch 62/100\n",
      "126/126 [==============================] - 0s 280us/sample - loss: 1.0433 - acc: 0.5873\n",
      "Epoch 63/100\n",
      "126/126 [==============================] - 0s 298us/sample - loss: 1.0089 - acc: 0.5714\n",
      "Epoch 64/100\n",
      "126/126 [==============================] - 0s 217us/sample - loss: 1.0061 - acc: 0.5794\n",
      "Epoch 65/100\n",
      "126/126 [==============================] - 0s 285us/sample - loss: 0.9820 - acc: 0.5873\n",
      "Epoch 66/100\n",
      "126/126 [==============================] - 0s 320us/sample - loss: 0.9795 - acc: 0.6111\n",
      "Epoch 67/100\n",
      "126/126 [==============================] - 0s 334us/sample - loss: 0.9579 - acc: 0.6032\n",
      "Epoch 68/100\n",
      "126/126 [==============================] - 0s 280us/sample - loss: 0.9390 - acc: 0.6667\n",
      "Epoch 69/100\n",
      "126/126 [==============================] - 0s 405us/sample - loss: 0.8885 - acc: 0.6905\n",
      "Epoch 70/100\n",
      "126/126 [==============================] - 0s 298us/sample - loss: 0.8894 - acc: 0.7698\n",
      "Epoch 71/100\n",
      "126/126 [==============================] - 0s 362us/sample - loss: 0.8777 - acc: 0.7381\n",
      "Epoch 72/100\n",
      "126/126 [==============================] - 0s 295us/sample - loss: 0.8498 - acc: 0.7540\n",
      "Epoch 73/100\n",
      "126/126 [==============================] - 0s 314us/sample - loss: 0.8585 - acc: 0.7698\n",
      "Epoch 74/100\n",
      "126/126 [==============================] - 0s 227us/sample - loss: 0.8353 - acc: 0.8095\n",
      "Epoch 75/100\n",
      "126/126 [==============================] - 0s 251us/sample - loss: 0.7816 - acc: 0.8571\n",
      "Epoch 76/100\n",
      "126/126 [==============================] - 0s 301us/sample - loss: 0.7902 - acc: 0.8651\n",
      "Epoch 77/100\n",
      "126/126 [==============================] - 0s 209us/sample - loss: 0.7514 - acc: 0.8651\n",
      "Epoch 78/100\n",
      "126/126 [==============================] - 0s 240us/sample - loss: 0.7506 - acc: 0.8968\n",
      "Epoch 79/100\n",
      "126/126 [==============================] - 0s 247us/sample - loss: 0.7187 - acc: 0.8571\n",
      "Epoch 80/100\n",
      "126/126 [==============================] - 0s 209us/sample - loss: 0.7351 - acc: 0.8651\n",
      "Epoch 81/100\n",
      "126/126 [==============================] - 0s 286us/sample - loss: 0.6974 - acc: 0.8968\n",
      "Epoch 82/100\n",
      "126/126 [==============================] - 0s 262us/sample - loss: 0.7121 - acc: 0.9048\n",
      "Epoch 83/100\n",
      "126/126 [==============================] - 0s 251us/sample - loss: 0.6668 - acc: 0.9127\n",
      "Epoch 84/100\n",
      "126/126 [==============================] - 0s 361us/sample - loss: 0.6282 - acc: 0.8968\n",
      "Epoch 85/100\n",
      "126/126 [==============================] - 0s 292us/sample - loss: 0.6702 - acc: 0.8651\n",
      "Epoch 86/100\n",
      "126/126 [==============================] - 0s 339us/sample - loss: 0.6246 - acc: 0.9048\n",
      "Epoch 87/100\n",
      "126/126 [==============================] - 0s 367us/sample - loss: 0.6398 - acc: 0.9127\n",
      "Epoch 88/100\n",
      "126/126 [==============================] - 0s 280us/sample - loss: 0.5908 - acc: 0.9286\n",
      "Epoch 89/100\n",
      "126/126 [==============================] - 0s 205us/sample - loss: 0.5719 - acc: 0.9206\n",
      "Epoch 90/100\n",
      "126/126 [==============================] - 0s 299us/sample - loss: 0.5648 - acc: 0.9524\n",
      "Epoch 91/100\n",
      "126/126 [==============================] - 0s 259us/sample - loss: 0.5614 - acc: 0.9524\n",
      "Epoch 92/100\n",
      "126/126 [==============================] - 0s 218us/sample - loss: 0.5567 - acc: 0.9365\n",
      "Epoch 93/100\n",
      "126/126 [==============================] - 0s 281us/sample - loss: 0.5030 - acc: 0.9444\n",
      "Epoch 94/100\n",
      "126/126 [==============================] - 0s 254us/sample - loss: 0.5554 - acc: 0.9365\n",
      "Epoch 95/100\n",
      "126/126 [==============================] - 0s 260us/sample - loss: 0.5185 - acc: 0.9206\n",
      "Epoch 96/100\n",
      "126/126 [==============================] - 0s 286us/sample - loss: 0.5103 - acc: 0.9286\n",
      "Epoch 97/100\n",
      "126/126 [==============================] - 0s 238us/sample - loss: 0.4884 - acc: 0.9444\n",
      "Epoch 98/100\n",
      "126/126 [==============================] - 0s 251us/sample - loss: 0.4810 - acc: 0.9524\n",
      "Epoch 99/100\n",
      "126/126 [==============================] - 0s 283us/sample - loss: 0.5261 - acc: 0.9365\n",
      "Epoch 100/100\n",
      "126/126 [==============================] - 0s 416us/sample - loss: 0.4608 - acc: 0.9444\n",
      "2019-11-18 13:15:10 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.core.policies.keras_policy\u001b[0m  - Done fitting keras policy model\n",
      "2019-11-18 13:15:10 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.core.agent\u001b[0m  - Persisted model to '/tmp/tmpc5rpfpun/core'\n",
      "\u001b[94mCore model training completed.\u001b[0m\n",
      "\u001b[94mTraining NLU model...\u001b[0m\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.training_data.training_data\u001b[0m  - Training data stats: \n",
      "\t- intent examples: 49 (8 distinct intents)\n",
      "\t- Found intents: 'goodbye', 'bot_challenge', 'mood_unhappy', 'deny', 'add_candidate', 'affirm', 'mood_great', 'greet'\n",
      "\t- Number of response examples: 0 (0 distinct response)\n",
      "\t- entity examples: 5 (1 distinct entities)\n",
      "\t- found entities: 'candidate'\n",
      "\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component WhitespaceTokenizer\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component RegexFeaturizer\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component CRFEntityExtractor\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component EntitySynonymMapper\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component CountVectorsFeaturizer\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component CountVectorsFeaturizer\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 13:15:11 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component EmbeddingIntentClassifier\n",
      "Epochs: 100%|������������������������������| 300/300 [00:08<00:00, 35.89it/s, loss=0.475, acc=1.000]\n",
      "2019-11-18 13:15:21 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.utils.train_utils\u001b[0m  - Finished training embedding policy, train loss=0.475, train accuracy=1.000\n",
      "2019-11-18 13:15:21 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 13:15:21 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Successfully saved model into '/tmp/tmpc5rpfpun/nlu'\n",
      "\u001b[94mNLU model training completed.\u001b[0m\n",
      "\u001b[92mYour Rasa model is trained and saved at '/home/codete/workshop/assistant/models/20191118-131521.tar.gz'.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!cd assistant && PYTHONIOENCODING='utf8' rasa train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the intent we use the parse method as in the previous example. Please remember to start the server first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-11-18 13:11:09 \u001b[1;30mINFO    \u001b[0m \u001b[34mroot\u001b[0m  - Starting Rasa server on http://localhost:5005\n",
      "2019-11-18 13:11:09 \u001b[1;30mWARNING \u001b[0m \u001b[34mrasa.core.agent\u001b[0m  - \u001b[33mCould not load local model in 'models'\u001b[0m\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!cd assistant && rasa run --enable-api"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can check the intents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'intent': {'name': 'add_candidate', 'confidence': 0.9999175072},\n",
       " 'entities': [{'start': 15,\n",
       "   'end': 24,\n",
       "   'value': 'candidate',\n",
       "   'entity': 'candidate',\n",
       "   'confidence': 0.9031959996,\n",
       "   'extractor': 'CRFEntityExtractor'}],\n",
       " 'intent_ranking': [{'name': 'add_candidate', 'confidence': 0.9999175072},\n",
       "  {'name': 'mood_unhappy', 'confidence': 3.02982e-05},\n",
       "  {'name': 'deny', 'confidence': 1.47811e-05},\n",
       "  {'name': 'bot_challenge', 'confidence': 1.15172e-05},\n",
       "  {'name': 'greet', 'confidence': 1.0192e-05},\n",
       "  {'name': 'goodbye', 'confidence': 9.412e-06},\n",
       "  {'name': 'affirm', 'confidence': 4.8702e-06},\n",
       "  {'name': 'mood_great', 'confidence': 1.3678e-06}],\n",
       " 'text': 'can we add the candidate?'}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_intent(sentence):\n",
    "    url = \"http://localhost:5005/model/parse\"\n",
    "    payload = {'text':sentence}\n",
    "    response = requests.post(url,json=payload)\n",
    "    return response.json()\n",
    "\n",
    "get_intent(\"can we add the candidate?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EXERCISE 2\n",
    "\n",
    "Extend the training examples and add an intent `change_status` with entities: `passed` and `failed`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /home/codete/workshop/assistant/data/nlu.md\n"
     ]
    }
   ],
   "source": [
    "%%writefile /home/codete/workshop/assistant/data/nlu.md\n",
    "\n",
    "## intent:change_status\n",
    "- "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please add the other intents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Appending to /home/codete/workshop/assistant/data/nlu.md\n"
     ]
    }
   ],
   "source": [
    "%%writefile -a /home/codete/workshop/assistant/data/nlu.md\n",
    "## intent:greet\n",
    "- hey\n",
    "- hello\n",
    "- hi\n",
    "- good morning\n",
    "- good evening\n",
    "- hey there\n",
    "\n",
    "## intent:goodbye\n",
    "- bye\n",
    "- goodbye\n",
    "- see you around\n",
    "- see you later\n",
    "\n",
    "## intent:affirm\n",
    "- yes\n",
    "- indeed\n",
    "- of course\n",
    "- that sounds good\n",
    "- correct\n",
    "\n",
    "## intent:deny\n",
    "- no\n",
    "- never\n",
    "- I don't think so\n",
    "- don't like that\n",
    "- no way\n",
    "- not really\n",
    "\n",
    "## intent:mood_great\n",
    "- perfect\n",
    "- very good\n",
    "- great\n",
    "- amazing\n",
    "- wonderful\n",
    "- I am feeling very good\n",
    "- I am great\n",
    "- I'm good\n",
    "\n",
    "## intent:mood_unhappy\n",
    "- sad\n",
    "- very sad\n",
    "- unhappy\n",
    "- bad\n",
    "- very bad\n",
    "- awful\n",
    "- terrible\n",
    "- not very good\n",
    "- extremely sad\n",
    "- so sad\n",
    "\n",
    "## intent:bot_challenge\n",
    "- are you a bot?\n",
    "- are you a human?\n",
    "- am I talking to a bot?\n",
    "- am I talking to a human?\n",
    "\n",
    "## synonym:candidate\n",
    "- candidate\n",
    "- developer\n",
    "- data scientist\n",
    "\n",
    "## intent:add_candidate\n",
    "- add [candidate](candidate)\n",
    "- adding [candidate](candidate)\n",
    "- please add [candidate](candidate)\n",
    "- please add new [candidate](candidate)\n",
    "- we have new prescreening upcoming\n",
    "- we have a new candidate [candidate](candidate) prescreening"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train it as in the cell above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-11-18 14:04:43 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.model\u001b[0m  - Data (messages) for NLU model changed.\n",
      "\u001b[94mCore stories/configuration did not change. No need to retrain Core model.\u001b[0m\n",
      "\u001b[94mTraining NLU model...\u001b[0m\n",
      "2019-11-18 14:04:48 \u001b[1;30mINFO    \u001b[0m \u001b[34mabsl\u001b[0m  - Entry Point [tensor2tensor.envs.tic_tac_toe_env:TicTacToeEnv] registered with id [T2TEnv-TicTacToeEnv-v0]\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.training_data.training_data\u001b[0m  - Training data stats: \n",
      "\t- intent examples: 55 (9 distinct intents)\n",
      "\t- Found intents: 'greet', 'bot_challenge', 'affirm', 'mood_unhappy', 'mood_great', 'deny', 'goodbye', 'change_status', 'add_candidate'\n",
      "\t- Number of response examples: 0 (0 distinct response)\n",
      "\t- entity examples: 11 (2 distinct entities)\n",
      "\t- found entities: 'candidate', 'status'\n",
      "\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component WhitespaceTokenizer\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component RegexFeaturizer\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component CRFEntityExtractor\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component EntitySynonymMapper\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component CountVectorsFeaturizer\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component CountVectorsFeaturizer\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 14:04:49 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Starting to train component EmbeddingIntentClassifier\n",
      "2019-11-18 14:04:51.551268: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "Epochs: 100%|������������������������������| 300/300 [00:10<00:00, 29.47it/s, loss=0.489, acc=1.000]\n",
      "2019-11-18 14:05:01 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.utils.train_utils\u001b[0m  - Finished training embedding policy, train loss=0.489, train accuracy=1.000\n",
      "2019-11-18 14:05:02 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Finished training component.\n",
      "2019-11-18 14:05:02 \u001b[1;30mINFO    \u001b[0m \u001b[34mrasa.nlu.model\u001b[0m  - Successfully saved model into '/tmp/tmpdyc0ce30/nlu'\n",
      "\u001b[94mNLU model training completed.\u001b[0m\n",
      "\u001b[92mYour Rasa model is trained and saved at '/home/codete/workshop/assistant/models/20191118-140502.tar.gz'.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!cd assistant && PYTHONIOENCODING='utf8' rasa train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To test the intent we use the parse method as in the previous example. Please remember to start the server first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd assistant && rasa run --enable-api"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can check the intents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'intent': {'name': 'change_status', 'confidence': 0.9999986887},\n",
       " 'entities': [{'start': 14,\n",
       "   'end': 20,\n",
       "   'value': 'failed',\n",
       "   'entity': 'status',\n",
       "   'confidence': 0.7578336823,\n",
       "   'extractor': 'CRFEntityExtractor'}],\n",
       " 'intent_ranking': [{'name': 'change_status', 'confidence': 0.9999986887},\n",
       "  {'name': 'affirm', 'confidence': 9.753e-07},\n",
       "  {'name': 'add_candidate', 'confidence': 1.689e-07},\n",
       "  {'name': 'mood_great', 'confidence': 8.56e-08},\n",
       "  {'name': 'mood_unhappy', 'confidence': 2.96e-08},\n",
       "  {'name': 'greet', 'confidence': 1.04e-08},\n",
       "  {'name': 'goodbye', 'confidence': 4e-09},\n",
       "  {'name': 'deny', 'confidence': 1e-09},\n",
       "  {'name': 'bot_challenge', 'confidence': 1e-10}],\n",
       " 'text': 'the candidate failed the interview'}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_intent(sentence):\n",
    "    url = \"http://localhost:5005/model/parse\"\n",
    "    payload = {'text':sentence}\n",
    "    response = requests.post(url,json=payload)\n",
    "    return response.json()\n",
    "\n",
    "get_intent(\"the candidate failed the interview\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building stories with Rasa\n",
    "\n",
    "To build a chatbot with Rasa that has a focus on the context management, we need to build the dataset of intents as before, but also stories. \n",
    "\n",
    "Rasa core for building stories need a bit more configuration than in the previous example. We need to setup the following:\n",
    "- the configuration of language and machine learning backend,\n",
    "- setup the domain with sample chatbot responses,\n",
    "- define the stories.\n",
    "After this step we need to train Rasa, but we still need feed it with intents after it.\n",
    "\n",
    "A basic configuration for Rasa is needed like to language and pipeline. The pipeline defines the way how we want to train our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Configuration for Rasa NLU.\r\n",
      "# https://rasa.com/docs/rasa/nlu/components/\r\n",
      "language: en\r\n",
      "pipeline: supervised_embeddings\r\n",
      "\r\n",
      "# Configuration for Rasa Core.\r\n",
      "# https://rasa.com/docs/rasa/core/policies/\r\n",
      "policies:\r\n",
      "  - name: MemoizationPolicy\r\n",
      "  - name: KerasPolicy\r\n",
      "  - name: MappingPolicy\r\n"
     ]
    }
   ],
   "source": [
    "!cat /home/codete/workshop/assistant/config.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To test the chatbot stories that are configured to train are set in ``stories.md`` like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## happy path\r\n",
      "* greet\r\n",
      "  - utter_greet\r\n",
      "* mood_great\r\n",
      "  - utter_happy\r\n",
      "\r\n",
      "## sad path 1\r\n",
      "* greet\r\n",
      "  - utter_greet\r\n",
      "* mood_unhappy\r\n",
      "  - utter_cheer_up\r\n",
      "  - utter_did_that_help\r\n",
      "* affirm\r\n",
      "  - utter_happy\r\n",
      "\r\n",
      "## sad path 2\r\n",
      "* greet\r\n",
      "  - utter_greet\r\n",
      "* mood_unhappy\r\n",
      "  - utter_cheer_up\r\n",
      "  - utter_did_that_help\r\n",
      "* deny\r\n",
      "  - utter_goodbye\r\n",
      "\r\n",
      "## say goodbye\r\n",
      "* goodbye\r\n",
      "  - utter_goodbye\r\n",
      "\r\n",
      "## bot challenge\r\n",
      "* bot_challenge\r\n",
      "  - utter_iamabot\r\n"
     ]
    }
   ],
   "source": [
    "!cat /home/codete/workshop/assistant/data/stories.md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to set the domain:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "intents:\r\n",
      "  - greet\r\n",
      "  - goodbye\r\n",
      "  - affirm\r\n",
      "  - deny\r\n",
      "  - mood_great\r\n",
      "  - mood_unhappy\r\n",
      "  - bot_challenge\r\n",
      "\r\n",
      "actions:\r\n",
      "- utter_greet\r\n",
      "- utter_cheer_up\r\n",
      "- utter_did_that_help\r\n",
      "- utter_happy\r\n",
      "- utter_goodbye\r\n",
      "- utter_iamabot\r\n",
      "\r\n",
      "templates:\r\n",
      "  utter_greet:\r\n",
      "  - text: \"Hey! How are you?\"\r\n",
      "\r\n",
      "  utter_cheer_up:\r\n",
      "  - text: \"Here is something to cheer you up:\"\r\n",
      "    image: \"https://i.imgur.com/nGF1K8f.jpg\"\r\n",
      "\r\n",
      "  utter_did_that_help:\r\n",
      "  - text: \"Did that help you?\"\r\n",
      "\r\n",
      "  utter_happy:\r\n",
      "  - text: \"Great, carry on!\"\r\n",
      "\r\n",
      "  utter_goodbye:\r\n",
      "  - text: \"Bye\"\r\n",
      "\r\n",
      "  utter_iamabot:\r\n",
      "  - text: \"I am a bot, powered by Rasa.\"\r\n"
     ]
    }
   ],
   "source": [
    "!cat /home/codete/workshop/assistant/domain.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test the bot in Jupyter with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:\n",
      "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "  * https://github.com/tensorflow/io (for I/O related ops)\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/adafactor.py:27: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/multistep_optimizer.py:32: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
      "\n",
      "WARNING:tensorflow:Failed to load tensor2tensor\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/glow_init_hook.py:25: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/neural_stack.py:51: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:111: The name tf.OptimizerOptions is deprecated. Please use tf.compat.v1.OptimizerOptions instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:111: The name tf.OptimizerOptions is deprecated. Please use tf.compat.v1.OptimizerOptions instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_gan/python/estimator/tpu_gan_estimator.py:42: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_gan/python/estimator/tpu_gan_estimator.py:42: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from /tmp/tmpgl4n03n0/nlu/component_6_EmbeddingIntentClassifier.ckpt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from /tmp/tmpgl4n03n0/nlu/component_6_EmbeddingIntentClassifier.ckpt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/rasa/utils/train_utils.py:964: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/rasa/utils/train_utils.py:964: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops.py:97: calling Orthogonal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops.py:97: calling Orthogonal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/backend.py:3994: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/backend.py:3994: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your bot is ready to talk! Type your messages here or send '/stop'.\n",
      "Hello\n",
      "\u001b[92mHey! How are you?\u001b[0m\n",
      "Good, thanks\n",
      "\u001b[92mGreat, carry on!\u001b[0m\n",
      "Bye\n",
      "\u001b[92mBye\u001b[0m\n",
      "/stop\n"
     ]
    }
   ],
   "source": [
    "from rasa.jupyter import chat\n",
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()\n",
    "\n",
    "model_path = '/home/codete/workshop/assistant/models/20191118-131521.tar.gz'\n",
    "chat(model_path)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}