QA with Tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question answering with TensorFlow\n",
    "## Using advanced neural networks to tackle challenging natural language tasks.\n",
    "\n",
    "##### By Steven Hewitt\n",
    "\n",
    "A question answering (QA) system is a system designed to answer questions posed in natural language. Some QA systems draw information from a source such as text or an image in order to answer a specific question. These \"sourced\" systems can be partitioned into two major subcategories: open domain, in which the questions can be virtually anything, but aren't focused on specific material, and closed domain, in which the questions have concrete limitations, in that they relate to some predefined source (e.g., a provided context or a specific field, like medicine).\n",
    "\n",
    "This  article will guide you through the task of creating and coding a question answering system using [TensorFlow](https://www.tensorflow.org/). We'll create a QA system that is based on a neural network, and sourced using a closed domain. In order to do this, we'll use a simplified version of a model known as a dynamic memory network (DMN), introduced by Kumar, et al, in their paper [\"Ask Me Anything: Dynamic Memory Networks for Natural Language Processing.\"](https://arxiv.org/abs/1506.07285)\n",
    "\n",
    "\n",
    "### Before we get started\n",
    "\n",
    "In addition to installing [TensorFlow](https://www.tensorflow.org/) version 1.2+ in Python 3, make sure you’ve installed each of the following:\n",
    "- [Jupyter](http://jupyter.org)\n",
    "- [Numpy](http://www.numpy.org)\n",
    "- [Matplotlib](http://matplotlib.org)\n",
    "\n",
    "Optionally, you can install TQDM to view training progess and get training speed metrics, but it's not required. The code and Jupyter Notebook for this article is [on GitHub](https://github.com/Steven-Hewitt/QA-with-Tensorflow), and I encourage you to grab it and follow along. If this is your first time working with TensorFlow, I recommend that you first check out Aaron Schumacher's [\"Hello, TensorFlow\"](https://www.oreilly.com/learning/hello-tensorflow) for a quick overview of what TensorFlow is and how it works. If this is your first time using TensorFlow for natural language tasks, I would also encourage you to check out [\"Textual Entailment with TensorFlow\"](#link not live yet), as it introduces several concepts that will be used to help construct this network.\n",
    "\n",
    "Let's start by importing all of the relevant libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "import urllib\n",
    "import sys\n",
    "import os\n",
    "import zipfile\n",
    "import tarfile\n",
    "import json \n",
    "import hashlib\n",
    "import re\n",
    "import itertools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring bAbI\n",
    "\n",
    "For this project, we will be using the [bAbI data set](https://research.fb.com/downloads/babi) created by Facebook. This data set, like all QA data sets, contains questions. Questions in bAbI are very straightforward, although some are trickier than others. All of the questions in this data set have an associated context, which is a sequence of sentences guaranteed to have the details necessary to answer the question. In addition, the data set provides the correct answer to each question.\n",
    "\n",
    "Questions in the bAbI data set are partitioned into 20 different tasks based on what skills are required to answer the question. Each task has its own set of questions for training, and a separate set for testing. These tasks test a variety of standard natural language processing abilities, including time reasoning (task #14) and inductive logic (task #16). To get a better idea of this, let's consider a concrete example of a question that our QA system will be expected to answer, as shown in Figure 1.\n",
    "\n",
    "<img src=\"questionex.png\">\n",
    "<center><em>Figure 1.</em> An example of bAbI's data, with the context in blue, question in gold, and answer in green. Credit: Steven Hewitt.</center>\n",
    "\n",
    "This task (#5) tests the network's understanding of actions where there are relationships between three objects. Grammatically speaking, it tests to see if the system can distinguish between the subject, direct object, and indirect object. In this case, the question asks for the indirect object in the last sentence -- the person who received the milk from Jeff. The network must make the distinction between the fifth sentence, in which Bill was the subject and Jeff was the indirect object, and the sixth sentence, in which Jeff was the subject. Of course, our network doesn't receive any explicit training on what a subject or object is, and has to extrapolate this understanding from the examples in the training data.\n",
    "\n",
    "Another minor problem the system must solve is understanding the various synonyms used throughout the data set. Jeff \"handed\" the milk to Bill, but he could have just as easily \"gave\" it or \"passed\" it to him. In this regard, though, the network doesn't have to start from scratch. It gets some assistance in the form of word vectorization, which can store infomation about the definition of words and their relations to other words. Similar words have similar vectorizations, which means that the network can treat them as nearly the same word. For word vectorization, we'll use Stanford’s Global Vectors for Word Representation (GloVe), which I’ve discussed previously in more detail [here](https://www.oreilly.com/learning/textual-entailment-with-tensorflow).\n",
    "\n",
    "Many of the tasks have a restriction that forces the context to contain the exact word used for the answer. In our example, the answer \"Bill\" can be found in the context. We will use this restriction to our advantage, as we can search the context for the word closest in meaning to our final result.\n",
    "\n",
    "Note: It might take a few minutes to download and unpack all of this data, so run the next three code snippets to get that started as quickly as possible. As you run the code, it will download bAbI and GloVe, and unpack the necessary files from those data sets so they can be used in our network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "glove_zip_file = \"glove.6B.zip\"\n",
    "glove_vectors_file = \"glove.6B.50d.txt\"\n",
    "\n",
    "# 15 MB\n",
    "data_set_zip = \"tasks_1-20_v1-2.tar.gz\"\n",
    "\n",
    "#Select \"task 5\"\n",
    "train_set_file = \"qa5_three-arg-relations_train.txt\"\n",
    "test_set_file = \"qa5_three-arg-relations_test.txt\"\n",
    "\n",
    "train_set_post_file = \"tasks_1-20_v1-2/en/\"+train_set_file\n",
    "test_set_post_file = \"tasks_1-20_v1-2/en/\"+test_set_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "try: from urllib.request import urlretrieve, urlopen\n",
    "except ImportError: \n",
    "    from urllib import urlretrieve\n",
    "    from urllib2 import urlopen\n",
    "#large file - 862 MB\n",
    "if (not os.path.isfile(glove_zip_file) and\n",
    "    not os.path.isfile(glove_vectors_file)):\n",
    "    urlretrieve (\"http://nlp.stanford.edu/data/glove.6B.zip\", \n",
    "                 glove_zip_file)\n",
    "if (not os.path.isfile(data_set_zip) and\n",
    "    not (os.path.isfile(train_set_file) and os.path.isfile(test_set_file))):\n",
    "    urlretrieve (\"https://s3.amazonaws.com/text-datasets/babi_tasks_1-20_v1-2.tar.gz\", \n",
    "                 data_set_zip)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def unzip_single_file(zip_file_name, output_file_name):\n",
    "    \"\"\"\n",
    "        If the output file is already created, don't recreate\n",
    "        If the output file does not exist, create it from the zipFile\n",
    "    \"\"\"\n",
    "    if not os.path.isfile(output_file_name):\n",
    "        with open(output_file_name, 'wb') as out_file:\n",
    "            with zipfile.ZipFile(zip_file_name) as zipped:\n",
    "                for info in zipped.infolist():\n",
    "                    if output_file_name in info.filename:\n",
    "                        with zipped.open(info) as requested_file:\n",
    "                            out_file.write(requested_file.read())\n",
    "                            return\n",
    "def targz_unzip_single_file(zip_file_name, output_file_name, interior_relative_path):\n",
    "    if not os.path.isfile(output_file_name):\n",
    "        with tarfile.open(zip_file_name) as un_zipped:\n",
    "            un_zipped.extract(interior_relative_path+output_file_name)    \n",
    "unzip_single_file(glove_zip_file, glove_vectors_file)\n",
    "targz_unzip_single_file(data_set_zip, train_set_file, \"tasks_1-20_v1-2/en/\")\n",
    "targz_unzip_single_file(data_set_zip, test_set_file, \"tasks_1-20_v1-2/en/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "### Parsing GloVe and handling unknown tokens\n",
    "\n",
    "In \"[Textual Entailment with TensorFlow](https://www.oreilly.com/learning/textual-entailment-with-tensorflow),\" I discuss `sentence2sequence`, which is a function that would turn a string into a matrix, based on the mapping defined by GloVe. This function split up the string into tokens, which are smaller strings that are roughly equivalent to punctuation, words, or parts of words. For example, in \"Bill traveled to the kitchen,\" there are six tokens: five that correspond to each of the words, and the last for the period at the end. Each token gets individually vectorized, resulting in a list of vectors corresponding to each sentence, as shown in Figure 2.\n",
    "\n",
    "<img src=\"vectorize.png\">\n",
    "<center><em>Figure 2.</em> The process of turning a sentence into vectors. Credit: Steven Hewitt.</center>\n",
    "\n",
    "In some tasks in bAbI, the system will encounter words that are not in the GloVe word vectorization. In order for the network to be capable of processing these unknown words, we need to maintain a consistent vectorization of those words. Common practice is to replace all unknown tokens with a single `<UNK>` vector, but this isn't always effective. Instead, we can use randomization to draw a new vectorization for each unique unknown token.\n",
    "\n",
    "The first time we run across a new unknown token, we simply draw a new vectorization from the (Gaussian-approximated) distribution of the original GloVe vectorizations, and add that vectorization back to the GloVe word map . To gather the distribution hyperparameters, Numpy has functions that automatically calculate variance and mean.\n",
    "\n",
    "`fill_unk` will take care of giving us a new word vectorization whenever we need one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Deserialize GloVe vectors\n",
    "glove_wordmap = {}\n",
    "with open(glove_vectors_file, \"r\", encoding=\"utf8\") as glove:\n",
    "    for line in glove:\n",
    "        name, vector = tuple(line.split(\" \", 1))\n",
    "        glove_wordmap[name] = np.fromstring(vector, sep=\" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "wvecs = []\n",
    "for item in glove_wordmap.items():\n",
    "    wvecs.append(item[1])\n",
    "s = np.vstack(wvecs)\n",
    "\n",
    "# Gather the distribution hyperparameters\n",
    "v = np.var(s,0) \n",
    "m = np.mean(s,0) \n",
    "RS = np.random.RandomState()\n",
    "\n",
    "def fill_unk(unk):\n",
    "    global glove_wordmap\n",
    "    glove_wordmap[unk] = RS.multivariate_normal(m,np.diag(v))\n",
    "    return glove_wordmap[unk]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Known or unknown\n",
    "\n",
    "The limited vocabulary of bAbI tasks means the network can learn the relationships between words even without knowing what the words mean. However, for speed of learning, we should choose vectorizations that have inherent meaning when we can. To do this, we use a greedy search for words that exist in Stanford's GLoVe word vectorization data set, and if the word does not exist, then we fill in the entire word with an unknown, randomly created, new representation.\n",
    "\n",
    "Under that model of word vectorization, we can define a new `sentence2sequence`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sentence2sequence(sentence):\n",
    "    \"\"\"\n",
    "     \n",
    "    - Turns an input paragraph into an (m,d) matrix, \n",
    "        where n is the number of tokens in the sentence\n",
    "        and d is the number of dimensions each word vector has.\n",
    "    \n",
    "      TensorFlow doesn't need to be used here, as simply\n",
    "      turning the sentence into a sequence based off our \n",
    "      mapping does not need the computational power that\n",
    "      TensorFlow provides. Normal Python suffices for this task.\n",
    "    \"\"\"\n",
    "    tokens = sentence.strip('\"(),-').lower().split(\" \")\n",
    "    rows = []\n",
    "    words = []\n",
    "    #Greedy search for tokens\n",
    "    for token in tokens:\n",
    "        i = len(token)\n",
    "        while len(token) > 0:\n",
    "            word = token[:i]\n",
    "            if word in glove_wordmap:\n",
    "                rows.append(glove_wordmap[word])\n",
    "                words.append(word)\n",
    "                token = token[i:]\n",
    "                i = len(token)\n",
    "                continue\n",
    "            else:\n",
    "                i = i-1\n",
    "            if i == 0:\n",
    "                # word OOV\n",
    "                # https://arxiv.org/pdf/1611.01436.pdf\n",
    "                rows.append(fill_unk(token))\n",
    "                words.append(token)\n",
    "                break\n",
    "    return np.array(rows), words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can package all the data together needed for each question, including the vectorization of the contexts, questions, and answers. In bAbI, contexts are defined by a numbered sequence of sentences, which `contextualize` deserializes into a list of sentences associated with one context. Questions and answers are on the same line, separated by tabs, so we can use tabs as a marker of whether a specific line refers to a question or not. When the numbering resets, future questions will refer to the new context (note that often there is more than one question corresponding to a single context). Answers also contain one other piece of information that we keep but don't need to use: the number(s) corresponding to the sentences needed to answer the question, in the reference order. In our system, the network will teach itself which sentences are needed to answer the question. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def contextualize(set_file):\n",
    "    \"\"\"\n",
    "    Read in the dataset of questions and build question+answer -> context sets.\n",
    "    Output is a list of data points, each of which is a 7-element tuple containing:\n",
    "        The sentences in the context in vectorized form.\n",
    "        The sentences in the context as a list of string tokens.\n",
    "        The question in vectorized form.\n",
    "        The question as a list of string tokens.\n",
    "        The answer in vectorized form.\n",
    "        The answer as a list of string tokens.\n",
    "        A list of numbers for supporting statements, which is currently unused.\n",
    "    \"\"\"\n",
    "    data = []\n",
    "    context = []\n",
    "    with open(set_file, \"r\", encoding=\"utf8\") as train:\n",
    "        for line in train:\n",
    "            l, ine = tuple(line.split(\" \", 1))\n",
    "            # Split the line numbers from the sentences they refer to.\n",
    "            if l is \"1\":\n",
    "                # New contexts always start with 1, \n",
    "                # so this is a signal to reset the context.\n",
    "                context = []\n",
    "            if \"\\t\" in ine: \n",
    "                # Tabs are the separator between questions and answers,\n",
    "                # and are not present in context statements.\n",
    "                question, answer, support = tuple(ine.split(\"\\t\"))\n",
    "                data.append((tuple(zip(*context))+\n",
    "                             sentence2sequence(question)+\n",
    "                             sentence2sequence(answer)+\n",
    "                             ([int(s) for s in support.split()],)))\n",
    "                # Multiple questions may refer to the same context, so we don't reset it.\n",
    "            else:\n",
    "                # Context sentence.\n",
    "                context.append(sentence2sequence(ine[:-1]))\n",
    "    return data\n",
    "train_data = contextualize(train_set_post_file)\n",
    "test_data = contextualize(test_set_post_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "final_train_data = []\n",
    "def finalize(data):\n",
    "    \"\"\"\n",
    "    Prepares data generated by contextualize() for use in the network.\n",
    "    \"\"\"\n",
    "    final_data = []\n",
    "    for cqas in data:\n",
    "        contextvs, contextws, qvs, qws, avs, aws, spt = cqas\n",
    "\n",
    "        lengths = itertools.accumulate(len(cvec) for cvec in contextvs)\n",
    "        context_vec = np.concatenate(contextvs)\n",
    "        context_words = sum(contextws,[])\n",
    "        \n",
    "        # Location markers for the beginnings of new sentences.\n",
    "        sentence_ends = np.array(list(lengths)) \n",
    "        final_data.append((context_vec, sentence_ends, qvs, spt, context_words, cqas, avs, aws))\n",
    "    return np.array(final_data)\n",
    "final_train_data = finalize(train_data)   \n",
    "final_test_data = finalize(test_data)     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining hyperparameters\n",
    "\n",
    "At this point, we have fully prepared our training data and our testing data. The next task is to construct the network we'll use to understand the data. Let's start by clearing out the TensorFlow default graph so we always have the option to run the network again if we want to change something. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this is the beginning of the actual network, let's also define all the constants we'll need for the network. We call these \"hyperparameters,\" as they define how the network looks and trains:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Hyperparameters\n",
    "\n",
    "# The number of dimensions used to store data passed between recurrent layers in the network.\n",
    "recurrent_cell_size = 128\n",
    "\n",
    "# The number of dimensions in our word vectorizations.\n",
    "D = 50 \n",
    "\n",
    "# How quickly the network learns. Too high, and we may run into numeric instability \n",
    "# or other issues.\n",
    "learning_rate = 0.005\n",
    "\n",
    "# Dropout probabilities. For a description of dropout and what these probabilities are, \n",
    "# see Entailment with TensorFlow.\n",
    "input_p, output_p = 0.5, 0.5\n",
    "\n",
    "# How many questions we train on at a time.\n",
    "batch_size = 128\n",
    "\n",
    "# Number of passes in episodic memory. We'll get to this later.\n",
    "passes = 4\n",
    "\n",
    "# Feed Forward layer sizes: the number of dimensions used to store data passed from feed-forward layers.\n",
    "ff_hidden_size = 256\n",
    "\n",
    "weight_decay = 0.00000001\n",
    "# The strength of our regularization. Increase to encourage sparsity in episodic memory, \n",
    "# but makes training slower. Don't make this larger than leraning_rate.\n",
    "\n",
    "training_iterations_count = 400000\n",
    "# How many questions the network trains on each time it is trained. \n",
    "# Some questions are counted multiple times.\n",
    "\n",
    "display_step = 100\n",
    "# How many iterations of training occur before each validation check."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network structure\n",
    "\n",
    "With the hyperparameters out of the way, let's describe the network structure. The structure of this network is split loosely into four modules and is described in [Ask Me Anything: Dynamic Memory Networks for Natural Language Processing](https://arxiv.org/abs/1506.07285).\n",
    "\n",
    "The network is designed around having a recurrent layer's memory be set dynamically, based on other information in the text, hence the name dynamic memory network (DMN). DMNs are loosely based on an understanding of how a human tries to answer a reading-comprehension-type question. The person gets a chance, first of all, to read the context and create memories of the facts inside. With those facts in mind, they then read the question, and re-examine the context specifically searching for the answer to that question, comparing the question to each of the facts. \n",
    "\n",
    "Sometimes, one fact guides us to another. In the bAbI data set, the network might want to find the location of a football. It might search for sentences about the football to find that John was the last person to touch the football, then search for sentences about John to find that John had been in both the bedroom and the hallway. Once it realizes that John had been last in the hallway, it can then answer the question and confidently say that the football is in the hallway.\n",
    "\n",
    "<img src=\"model.png\">\n",
    "<center><em>Figure 2.</em> The modules inside the network as they work together to answer a bAbI question. In each episode, new facts are attended to so they can help come up with the answer. Kumar notes that the network incorrectly puts some weight in sentence 2, which makes sense since John has been there, even though at the time he did not have the football. Credit: Ankit Kumar, et al., used with permission.\n",
    "</center>\n",
    "\n",
    "#### Input\n",
    "\n",
    "The input module is the first of the four modules that a dynamic memory network uses to come up with its answer, and consists of a simple pass over the input with a gated recurrent unit, or GRU, (TensorFlow's [tf.contrib.nn.GRUCell](https://www.tensorflow.org/versions/r1.3/api_docs/python/tf/contrib/rnn/GRUCell)) to gather pieces of evidence. Each  piece of evidence, or *fact*, corresponds to a single sentence in the context, and is represented by the output at that timestep. This requires a bit of non-TensorFlow preprocessing so we can gather the locations of the ends of sentences and pass that in to TensorFlow for use in later modules. \n",
    "\n",
    "We'll take care of that external processing later on, when we get to training. We can use that processed data with TensorFlow's `gather_nd` to select the corresponding outputs. The function `gather_nd` is an extraordinarily useful tool, and I'd suggest you review the [API documentation](https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/gather_nd) to learn how it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Input Module\n",
    "\n",
    "# Context: A [batch_size, maximum_context_length, word_vectorization_dimensions] tensor \n",
    "# that contains all the context information.\n",
    "context = tf.placeholder(tf.float32, [None, None, D], \"context\")  \n",
    "context_placeholder = context # I use context as a variable name later on\n",
    "\n",
    "# input_sentence_endings: A [batch_size, maximum_sentence_count, 2] tensor that \n",
    "# contains the locations of the ends of sentences. \n",
    "input_sentence_endings = tf.placeholder(tf.int32, [None, None, 2], \"sentence\")\n",
    "\n",
    "# recurrent_cell_size: the number of hidden units in recurrent layers.\n",
    "input_gru = tf.contrib.rnn.GRUCell(recurrent_cell_size)\n",
    "\n",
    "# input_p: The probability of maintaining a specific hidden input unit.\n",
    "# Likewise, output_p is the probability of maintaining a specific hidden output unit.\n",
    "gru_drop = tf.contrib.rnn.DropoutWrapper(input_gru, input_p, output_p)\n",
    "\n",
    "# dynamic_rnn also returns the final internal state. We don't need that, and can\n",
    "# ignore the corresponding output (_). \n",
    "input_module_outputs, _ = tf.nn.dynamic_rnn(gru_drop, context, dtype=tf.float32, scope = \"input_module\")\n",
    "\n",
    "# cs: the facts gathered from the context.\n",
    "cs = tf.gather_nd(input_module_outputs, input_sentence_endings)\n",
    "# to use every word as a fact, useful for tasks with one-sentence contexts\n",
    "s = input_module_outputs "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question\n",
    "\n",
    "The question module is the second module, and arguably the simplest. It consists of another GRU pass, this time over the text of the question. Instead of pieces of evidence, we can simply pass forward the end state, as the question is guaranteed by the data set to be one sentence long. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Question Module\n",
    "\n",
    "# query: A [batch_size, maximum_question_length, word_vectorization_dimensions] tensor \n",
    "#  that contains all of the questions.\n",
    "\n",
    "query = tf.placeholder(tf.float32, [None, None, D], \"query\")\n",
    "\n",
    "# input_query_lengths: A [batch_size, 2] tensor that contains question length information. \n",
    "# input_query_lengths[:,1] has the actual lengths; input_query_lengths[:,0] is a simple range() \n",
    "# so that it plays nice with gather_nd.\n",
    "input_query_lengths = tf.placeholder(tf.int32, [None, 2], \"query_lengths\")\n",
    "\n",
    "question_module_outputs, _ = tf.nn.dynamic_rnn(gru_drop, query, dtype=tf.float32, \n",
    "                                               scope = tf.VariableScope(True, \"input_module\"))\n",
    "\n",
    "# q: the question states. A [batch_size, recurrent_cell_size] tensor.\n",
    "q = tf.gather_nd(question_module_outputs, input_query_lengths)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Episodic memory\n",
    "Our third module, the episodic memory module, is where things begin to get interesting. It uses attention to do multiple passes, each pass consisting of GRUs iterating over the input. Each iteration inside each pass has a weighted update on current memory, based on how much attention is being paid to the corresponding fact at that time.\n",
    "  \n",
    "##### Attention\n",
    "Attention in neural networks was originally designed for image analysis, especially for cases where parts of the image are far more relevant than others. Networks use attention to determine the best locations in which to do further analysis when performing tasks, such as finding locations of objects in images, tracking objects that move between images, facial recognition, or other tasks that benefit from finding the most pertinent information for the task within the image.\n",
    "\n",
    "The main problem is that attention, or at least hard attention (which attends to exactly one input location) is not easily optimizable. As with most other neural networks, our optimization scheme is to compute the derivative of a loss function with respect to our inputs and weights, and hard attention is simply not differentiable, thanks to its binary nature. Instead, we are forced to use the real-valued version known as “soft attention,” which combines all the input locations that could be attended to using some form of weighting. Thankfully, the weighting is fully differentiable and can be trained normally. While it is possible to learn hard attention, it’s much more difficult and sometimes performs worse than soft attention. Thus, we’ll stick with soft attention for this model. Don't worry about coding the derivative; TensorFlow's optimization schemes do it for us.\n",
    "\n",
    "We calculate attention in this model by constructing similarity measures between each fact, our current memory, and the original question. (Note that this is different from normal attention, which only constructs similarity measures between facts and current memory.) We pass the results through a two-layer feed-forward network to get an attention constant for each fact. We then modify the memory by doing a weighted pass with a GRU over the input facts (weighted by the corresponding attention constant). In order to avoid adding incorrect information into memory when the context is shorter than the full length of the matrix, we create a mask for which facts exist and don't attend at all (i.e., retain the same memory) when the fact does not exist.\n",
    "\n",
    "Another notable aspect is that the attention mask is nearly always wrapped around a representation used by a layer. For images, that wrapping is most likely to happen around a convolutional layer (most likely one with a direct mapping to locations in the image), and for natural language, that wrapping is most likely to happen around a recurrent layer. Wrapping attention around a feed-forward layer, while technically possible, is usually not useful -- at least not in a way that can’t be more easily simulated by further feed-forward layers. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Episodic Memory\n",
    "\n",
    "# make sure the current memory (i.e. the question vector) is broadcasted along the facts dimension\n",
    "size = tf.stack([tf.constant(1),tf.shape(cs)[1], tf.constant(1)])\n",
    "re_q = tf.tile(tf.reshape(q,[-1,1,recurrent_cell_size]),size)\n",
    "\n",
    "\n",
    "# Final output for attention, needs to be 1 in order to create a mask\n",
    "output_size = 1 \n",
    "\n",
    "# Weights and biases\n",
    "attend_init = tf.random_normal_initializer(stddev=0.1)\n",
    "w_1 = tf.get_variable(\"attend_w1\", [1,recurrent_cell_size*7, recurrent_cell_size], \n",
    "                      tf.float32, initializer = attend_init)\n",
    "w_2 = tf.get_variable(\"attend_w2\", [1,recurrent_cell_size, output_size], \n",
    "                      tf.float32, initializer = attend_init)\n",
    "\n",
    "b_1 = tf.get_variable(\"attend_b1\", [1, recurrent_cell_size], \n",
    "                      tf.float32, initializer = attend_init)\n",
    "b_2 = tf.get_variable(\"attend_b2\", [1, output_size], \n",
    "                      tf.float32, initializer = attend_init)\n",
    "\n",
    "# Regulate all the weights and biases\n",
    "tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, tf.nn.l2_loss(w_1))\n",
    "tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, tf.nn.l2_loss(b_1))\n",
    "tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, tf.nn.l2_loss(w_2))\n",
    "tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, tf.nn.l2_loss(b_2))\n",
    "\n",
    "def attention(c, mem, existing_facts):\n",
    "    \"\"\"\n",
    "    Custom attention mechanism.\n",
    "    c: A [batch_size, maximum_sentence_count, recurrent_cell_size] tensor \n",
    "        that contains all the facts from the contexts.\n",
    "    mem: A [batch_size, maximum_sentence_count, recurrent_cell_size] tensor that \n",
    "        contains the current memory. It should be the same memory for all facts for accurate results.\n",
    "    existing_facts: A [batch_size, maximum_sentence_count, 1] tensor that \n",
    "        acts as a binary mask for which facts exist and which do not.\n",
    "    \n",
    "    \"\"\"\n",
    "    with tf.variable_scope(\"attending\") as scope:\n",
    "        # attending: The metrics by which we decide what to attend to.\n",
    "        attending = tf.concat([c, mem, re_q, c * re_q,  c * mem, (c-re_q)**2, (c-mem)**2], 2)\n",
    "        \n",
    "        # m1: First layer of multiplied weights for the feed-forward network. \n",
    "        #     We tile the weights in order to manually broadcast, since tf.matmul does not\n",
    "        #     automatically broadcast batch matrix multiplication as of TensorFlow 1.2.\n",
    "        m1 = tf.matmul(attending * existing_facts, \n",
    "                       tf.tile(w_1, tf.stack([tf.shape(attending)[0],1,1]))) * existing_facts\n",
    "        # bias_1: A masked version of the first feed-forward layer's bias\n",
    "        #     over only existing facts.\n",
    "\n",
    "        bias_1 = b_1 * existing_facts\n",
    "        \n",
    "        # tnhan: First nonlinearity. In the original paper, this is a tanh nonlinearity; \n",
    "        #        choosing relu was a design choice intended to avoid issues with \n",
    "        #        low gradient magnitude when the tanh returned values close to 1 or -1. \n",
    "        tnhan = tf.nn.relu(m1 + bias_1)\n",
    "        \n",
    "        # m2: Second layer of multiplied weights for the feed-forward network. \n",
    "        #     Still tiling weights for the same reason described in m1's comments.\n",
    "        m2 = tf.matmul(tnhan, tf.tile(w_2, tf.stack([tf.shape(attending)[0],1,1])))\n",
    "        \n",
    "        # bias_2: A masked version of the second feed-forward layer's bias.\n",
    "        bias_2 = b_2 * existing_facts\n",
    "        \n",
    "        # norm_m2: A normalized version of the second layer of weights, which is used \n",
    "        #     to help make sure the softmax nonlinearity doesn't saturate.\n",
    "        norm_m2 = tf.nn.l2_normalize(m2 + bias_2, -1)\n",
    "        \n",
    "        # softmaxable: A hack in order to use sparse_softmax on an otherwise dense tensor. \n",
    "        #     We make norm_m2 a sparse tensor, then make it dense again after the operation.\n",
    "        softmax_idx = tf.where(tf.not_equal(norm_m2, 0))[:,:-1]\n",
    "        softmax_gather = tf.gather_nd(norm_m2[...,0], softmax_idx)\n",
    "        softmax_shape = tf.shape(norm_m2, out_type=tf.int64)[:-1]\n",
    "        softmaxable = tf.SparseTensor(softmax_idx, softmax_gather, softmax_shape)\n",
    "        return tf.expand_dims(tf.sparse_tensor_to_dense(tf.sparse_softmax(softmaxable)),-1)\n",
    "\n",
    "# facts_0s: a [batch_size, max_facts_length, 1] tensor \n",
    "#     whose values are 1 if the corresponding fact exists and 0 if not.\n",
    "facts_0s = tf.cast(tf.count_nonzero(input_sentence_endings[:,:,-1:],-1,keep_dims=True),tf.float32)\n",
    "\n",
    "\n",
    "with tf.variable_scope(\"Episodes\") as scope:\n",
    "    attention_gru = tf.contrib.rnn.GRUCell(recurrent_cell_size)\n",
    "    \n",
    "    # memory: A list of all tensors that are the (current or past) memory state \n",
    "    #   of the attention mechanism.\n",
    "    memory = [q]\n",
    "    \n",
    "    # attends: A list of all tensors that represent what the network attends to.\n",
    "    attends = []\n",
    "    for a in range(passes):\n",
    "        # attention mask\n",
    "        attend_to = attention(cs, tf.tile(tf.reshape(memory[-1],[-1,1,recurrent_cell_size]),size),\n",
    "                              facts_0s)\n",
    "        \n",
    "        # Inverse attention mask, for what's retained in the state.\n",
    "        retain = 1-attend_to\n",
    "        \n",
    "        # GRU pass over the facts, according to the attention mask.\n",
    "        while_valid_index = (lambda state, index: index < tf.shape(cs)[1])\n",
    "        update_state = (lambda state, index: (attend_to[:,index,:] * \n",
    "                                                 attention_gru(cs[:,index,:], state)[0] + \n",
    "                                                 retain[:,index,:] * state))\n",
    "        # start loop with most recent memory and at the first index\n",
    "        memory.append(tuple(tf.while_loop(while_valid_index,\n",
    "                          (lambda state, index: (update_state(state,index),index+1)),\n",
    "                           loop_vars = [memory[-1], 0]))[0]) \n",
    "        \n",
    "        attends.append(attend_to)\n",
    "        \n",
    "        # Reuse variables so the GRU pass uses the same variables every pass.\n",
    "        scope.reuse_variables()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer\n",
    "The final module is the answer module, which regresses from the question and episodic memory modules' outputs using a fully connected layer to a \"final result\" word vector, and the word in the context that is closest in distance to that result is our final output (to guarantee the result is an actual word). We calculate the closest word by creating a \"score\" for each word, which indicates the final result's distance from the word. While you can design an answer module that can return multiple words, it is not needed for the bAbI tasks we attempt in this article."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Answer Module\n",
    "\n",
    "# a0: Final memory state. (Input to answer module)\n",
    "a0 = tf.concat([memory[-1], q], -1)\n",
    "\n",
    "# fc_init: Initializer for the final fully connected layer's weights.\n",
    "fc_init = tf.random_normal_initializer(stddev=0.1) \n",
    "\n",
    "with tf.variable_scope(\"answer\"):\n",
    "    # w_answer: The final fully connected layer's weights.\n",
    "    w_answer = tf.get_variable(\"weight\", [recurrent_cell_size*2, D], \n",
    "                               tf.float32, initializer = fc_init)\n",
    "    # Regulate the fully connected layer's weights\n",
    "    tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, \n",
    "                     tf.nn.l2_loss(w_answer)) \n",
    "    \n",
    "    # The regressed word. This isn't an actual word yet; \n",
    "    #    we still have to find the closest match.\n",
    "    logit = tf.expand_dims(tf.matmul(a0, w_answer),1)\n",
    "    \n",
    "    # Make a mask over which words exist.\n",
    "    with tf.variable_scope(\"ending\"):\n",
    "        all_ends = tf.reshape(input_sentence_endings, [-1,2])\n",
    "        range_ends = tf.range(tf.shape(all_ends)[0])\n",
    "        ends_indices = tf.stack([all_ends[:,0],range_ends], axis=1)\n",
    "        ind = tf.reduce_max(tf.scatter_nd(ends_indices, all_ends[:,1],\n",
    "                                          [tf.shape(q)[0], tf.shape(all_ends)[0]]),\n",
    "                            axis=-1)\n",
    "        range_ind = tf.range(tf.shape(ind)[0])\n",
    "        mask_ends = tf.cast(tf.scatter_nd(tf.stack([ind, range_ind], axis=1), \n",
    "                                          tf.ones_like(range_ind), [tf.reduce_max(ind)+1, \n",
    "                                                                    tf.shape(ind)[0]]), bool)\n",
    "        # A bit of a trick. With the locations of the ends of the mask (the last periods in \n",
    "        #  each of the contexts) as 1 and the rest as 0, we can scan with exclusive or \n",
    "        #  (starting from all 1). For each context in the batch, this will result in 1s \n",
    "        #  up until the marker (the location of that last period) and 0s afterwards.\n",
    "        mask = tf.scan(tf.logical_xor,mask_ends, tf.ones_like(range_ind, dtype=bool))\n",
    "        \n",
    "    # We score each possible word inversely with their Euclidean distance to the regressed word.\n",
    "    #  The highest score (lowest distance) will correspond to the selected word.\n",
    "    logits = -tf.reduce_sum(tf.square(context*tf.transpose(tf.expand_dims(\n",
    "                    tf.cast(mask, tf.float32),-1),[1,0,2]) - logit), axis=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizing optimization\n",
    "\n",
    "Gradient descent is the default optimizer for a neural network. Its goal is to decrease the network's \"loss,\" which is a measure of how poorly the network performs. It does this by finding the derivative of loss with respect to each of the weights under the current input, and then \"descends\" the weights so they’ll reduce the loss. Most of the time, this works well enough, but often it’s not ideal. There are various schemes that use “momentum” or other approximations of the more direct path to the optimal weights. One of the most useful of these optimization schemes is known as adaptive moment estimation, or *Adam*. \n",
    "\n",
    "Adam estimates the first two moments of the gradient by calculating an exponentially decaying average of past iterations' gradients and squared gradients, which correspond to the estimated mean and the estimated variance of these gradients. The calculations use two additional hyperparameters  to indicate how quickly the averages decay with the addition of new information. The averages are initialized as zero, which leads to bias toward zero, especially when those hyperparameters near one.\n",
    "\n",
    "In order to counteract that bias, Adam computes bias-corrected moment estimates that are greater in magnitude than the originals. The corrected estimates are then used to update the weights throughout the network. The combination of these estimates make Adam one of the best choices overall for optimization, especially for complex networks. This applies doubly to data that is very sparse, such as is common in natural language processing tasks.\n",
    "\n",
    "In TensorFlow, we can use Adam by creating a `tf.train.AdamOptimizer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Training\n",
    "\n",
    "# gold_standard: The real answers.\n",
    "gold_standard = tf.placeholder(tf.float32, [None, 1, D], \"answer\")\n",
    "with tf.variable_scope('accuracy'):\n",
    "    eq = tf.equal(context, gold_standard)\n",
    "    corrbool = tf.reduce_all(eq,-1)\n",
    "    logloc = tf.reduce_max(logits, -1, keep_dims = True)\n",
    "    # locs: A boolean tensor that indicates where the score \n",
    "    #  matches the minimum score. This happens on multiple dimensions, \n",
    "    #  so in the off chance there's one or two indexes that match \n",
    "    #  we make sure it matches in all indexes.\n",
    "    locs = tf.equal(logits, logloc)\n",
    "    \n",
    "    # correctsbool: A boolean tensor that indicates for which \n",
    "    #   words in the context the score always matches the minimum score.\n",
    "    correctsbool = tf.reduce_any(tf.logical_and(locs, corrbool), -1)\n",
    "    # corrects: A tensor that is simply correctsbool cast to floats.\n",
    "    corrects = tf.where(correctsbool, tf.ones_like(correctsbool, dtype=tf.float32), \n",
    "                        tf.zeros_like(correctsbool,dtype=tf.float32))\n",
    "    \n",
    "    # corr: corrects, but for the right answer instead of our selected answer.\n",
    "    corr = tf.where(corrbool, tf.ones_like(corrbool, dtype=tf.float32), \n",
    "                        tf.zeros_like(corrbool,dtype=tf.float32))\n",
    "with tf.variable_scope(\"loss\"):\n",
    "    # Use sigmoid cross entropy as the base loss, \n",
    "    #  with our distances as the relative probabilities. There are\n",
    "    #  multiple correct labels, for each location of the answer word within the context.\n",
    "    loss = tf.nn.sigmoid_cross_entropy_with_logits(logits = tf.nn.l2_normalize(logits,-1),\n",
    "                                                   labels = corr)\n",
    "    \n",
    "    # Add regularization losses, weighted by weight_decay.\n",
    "    total_loss = tf.reduce_mean(loss) + weight_decay * tf.add_n(\n",
    "        tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES))\n",
    "\n",
    "# TensorFlow's default implementation of the Adam optimizer works. We can adjust more than \n",
    "#  just the learning rate, but it's not necessary to find a very good optimum.\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "\n",
    "# Once we have an optimizer, we ask it to minimize the loss \n",
    "#   in order to work towards the proper training.\n",
    "opt_op = optimizer.minimize(total_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Initialize variables\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "# Launch the TensorFlow session\n",
    "sess = tf.Session()\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Train the network\n",
    "\n",
    "With everything set and ready, we can begin batching our training data to train our network! While the system is training, we should check on how well the network is doing in terms of accuracy. We do this with a validation set, which is taken from testing data so it has no overlap with the training data. \n",
    "\n",
    "Using a validation set based on testing data allows us to get a better sense of how well the network can generalize what it learns and apply it to other contexts. If we validate on the training data, the network may overfit -- in other words, learn specific examples and memorize the answers to them, which doesn't help the network answer new questions.\n",
    "\n",
    "If you installed TQDM, you can use it to keep track of how long the network has been training and receive an estimate of when training will finish. You can stop the training at any time if you feel the results are good enough by interrupting the Jupyter Notebook kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prep_batch(batch_data, more_data = False):\n",
    "    \"\"\"\n",
    "        Prepare all the preproccessing that needs to be done on a batch-by-batch basis.\n",
    "    \"\"\"\n",
    "    context_vec, sentence_ends, questionvs, spt, context_words, cqas, answervs, _ = zip(*batch_data)\n",
    "    ends = list(sentence_ends)\n",
    "    maxend = max(map(len, ends))\n",
    "    aends = np.zeros((len(ends), maxend))\n",
    "    for index, i in enumerate(ends):\n",
    "        for indexj, x in enumerate(i):\n",
    "            aends[index, indexj] = x-1\n",
    "    new_ends = np.zeros(aends.shape+(2,))\n",
    "\n",
    "    for index, x in np.ndenumerate(aends):\n",
    "        new_ends[index+(0,)] = index[0]\n",
    "        new_ends[index+(1,)] = x\n",
    "\n",
    "    contexts = list(context_vec)\n",
    "    max_context_length = max([len(x) for x in contexts])\n",
    "    contextsize = list(np.array(contexts[0]).shape)\n",
    "    contextsize[0] = max_context_length\n",
    "    final_contexts = np.zeros([len(contexts)]+contextsize)\n",
    "\n",
    "    contexts = [np.array(x) for x in contexts]\n",
    "    for i, context in enumerate(contexts):\n",
    "        final_contexts[i,0:len(context),:] = context\n",
    "    max_query_length = max(len(x) for x in questionvs)\n",
    "    querysize = list(np.array(questionvs[0]).shape)\n",
    "    querysize[:1] = [len(questionvs),max_query_length]\n",
    "    queries = np.zeros(querysize)\n",
    "    querylengths = np.array(list(zip(range(len(questionvs)),[len(q)-1 for q in questionvs])))\n",
    "    questions = [np.array(q) for q in questionvs]\n",
    "    for i, question in enumerate(questions):\n",
    "        queries[i,0:len(question),:] = question\n",
    "    data = {context_placeholder: final_contexts, input_sentence_endings: new_ends, \n",
    "                            query:queries, input_query_lengths:querylengths, gold_standard: answervs}\n",
    "    return (data, context_words, cqas) if more_data else data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|                                                                                                                                       | 0/235 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 0.0, Minibatch Loss=  0.673046 Accuracy=  0.01875\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 43%|█████████████████████████████████████████████████████▏                                                                       | 100/235 [05:55<07:33,  3.36s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 100.0, Minibatch Loss=  0.672778 Accuracy=  0.384375\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 85%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▍                  | 200/235 [12:21<02:27,  4.21s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 200.0, Minibatch Loss=  0.672777 Accuracy=  0.449219\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 235/235 [14:45<00:00,  3.15s/it]\n"
     ]
    }
   ],
   "source": [
    "# Use TQDM if installed\n",
    "tqdm_installed = False\n",
    "try:\n",
    "    from tqdm import tqdm\n",
    "    tqdm_installed = True\n",
    "except:\n",
    "    pass\n",
    "\n",
    "\n",
    "# Prepare validation set\n",
    "batch = np.random.randint(final_test_data.shape[0], size=batch_size*10)\n",
    "batch_data = final_test_data[batch]\n",
    "\n",
    "validation_set, val_context_words, val_cqas = prep_batch(batch_data, True)\n",
    "\n",
    "# training_iterations_count: The number of data pieces to train on in total\n",
    "# batch_size: The number of data pieces per batch\n",
    "def train(iterations, batch_size):\n",
    "    training_iterations = range(0,iterations,batch_size)\n",
    "    if tqdm_installed:\n",
    "        # Add a progress bar if TQDM is installed\n",
    "        training_iterations = tqdm(training_iterations)\n",
    "\n",
    "    wordz = []\n",
    "    for j in training_iterations:\n",
    "\n",
    "        batch = np.random.randint(final_train_data.shape[0], size=batch_size)\n",
    "        batch_data = final_train_data[batch]\n",
    "\n",
    "        sess.run([opt_op], feed_dict=prep_batch(batch_data))\n",
    "        if (j/batch_size) % display_step == 0:\n",
    "\n",
    "            # Calculate batch accuracy\n",
    "            acc, ccs, tmp_loss, log, con, cor, loc  = sess.run([corrects, cs, total_loss, logit,\n",
    "                                                                context_placeholder,corr, locs], \n",
    "                                                               feed_dict=validation_set)\n",
    "            # Display results\n",
    "            print(\"Iter \" + str(j/batch_size) + \", Minibatch Loss= \",tmp_loss,\n",
    "                  \"Accuracy= \", np.mean(acc))\n",
    "train(30000,batch_size) # Small amount of training for preliminary results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After a little bit of training, let's peek inside and see what kinds of answers we're getting from the network.  In the diagrams below, we visualize attention over each of the episodes (rows) for all the sentences (columns) in our context; darker colors represent more attention paid to that sentence on that episode.\n",
    "\n",
    "You should see attention change between at least two episodes for each question, but sometimes attention will be able to find the answer within one, and sometimes it will take all four episodes. If the attention appears to be blank, it may be saturating and paying attention to everything at once. In that case, you can try training with a higher `weight_decay` in order to discourage that from happening. Later on in training, saturation becomes extremely common."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEPCAYAAABFpK+YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEJdJREFUeJzt3X2wXHV9x/HPJ0+YAIECapFIDFXKgJVAkYciuSDDlKLl\noS0PasBSZ+rgMII49MGBktCWOrb1Ae0DDMggCNIygKDEwRZNBCqFkEhKQMTw0IpAKQ9JDEJu8u0f\n51x63d69e+7Nnj279/t+zdy55+z+ds8337v57Nnfnj3riBAAII9pTRcAAOgtgh8AkiH4ASAZgh8A\nkiH4ASAZgh8AkulJ8NueZvsB27f2YnsAgPZ6tcd/jqS1PdoWAGActQe/7XmSjpN0Rd3bAgB01os9\n/s9JOl8SHxEGgD5Qa/Dbfp+kZyNitSSXPwCABrnOc/XYvkTSYknDkmZL2lHSTRFxRss4Xg0AwARF\nxKR2pmsN/l/YkD0k6ZMRcfwY18Wrw1t6Usdk/fnSpbrwoouaLqOjxed9TKd94mNNl9HRDZ//B516\n7llNlzGun25arxWXf7Xv65ToZ7cNQj9/d8H+kw5+juMHgGRm9GpDEbFc0vJebQ8AMDb2+CtaNDTU\ndAmV7HfoQU2XUAl1dhd1dteg1DlZPZvjH7eIAZjjHxS3PrVG08zBU93w003rtfucuU2XMWXQz+5i\njh8AUBnBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwA\nkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzB\nDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJ\nEPwAkAzBDwDJEPwAkAzBDwDJEPwAkAzBDwDJEPwAkMyMOu/c9naSVkiaVW7rxohYWuc2AQDjqzX4\nI+JV20dFxCbb0yXdbXtZRPx7ndsFALRX+1RPRGwqF7dT8UQTdW8TANBe7cFve5rtVZKekfTtiLiv\n7m0CANqrdapHkiJiq6QDbM+VdIvtfSNibeu4i5cueX150dCQFh15ZN2lTUkhaUvwoqobNg7/XI++\ntKHpMqaMrZquDa/Rz8lad/+DWnf/g125L0cPQ8L2hZJ+FhGfbbk8Xhke7lkdU9mWkKa76Sqmhkvu\nv0HTTTO75dGXntbeO7+l6TKmjCWHfFARMakHaK1TPbZ3s71TuTxb0jGSHqlzmwCA8dU91bO7pKtt\nT1PxJHNDRNxe8zYBAOOo+3DONZIOrHMbAICJ4ZO7AJAMwQ8AyRD8AJAMwQ8AyRD8AJAMwQ8AyRD8\nAJAMwQ8AyRD8AJAMwQ8AyRD8AJAMwQ8AyRD8AJAMwQ8AyRD8AJAMwQ8AyRD8AJAMwQ8AyRD8AJAM\nwQ8AyRD8AJAMwQ8AyRD8AJAMwQ8AyRD8AJAMwQ8AyRD8AJAMwQ8AyRD8AJAMwQ8AyRD8AJAMwQ8A\nyRD8AJAMwQ8AyVQKfhcW2/6zcn1P2wfXWxoAoA5V9/j/XtJhkj5Qrm+Q9He1VAQAqNWMiuMOiYgD\nba+SpIh40fasGusCANSk6h7/ZtvTJYUk2X6jpK21VQUAqE3V4L9U0s2S3mT7LyXdJemS2qoCANSm\n0lRPRHzV9kpJR0uypBMj4uFaKwMA1GLc4Le9y6jV5yRdP/q6iHihrsIAAPXotMe/UsW8viXtKenF\ncnlnSU9JWlBrdQCArht3jj8iFkTEXpL+RdJvR8RuEbGrpPdLuqMXBQIAuqvqm7uHRsTtIysRsUzS\nb9RTEgCgTlWP43/a9gWSri3XPyTp6XpKAgDUqeoe/wckvVHFIZ03S3qT/u9TvACAAVL1cM4XJJ1j\ne8diNTbWWxYAoC5VT9L2a+XpGv5D0kO2V9p+Z4XbzbN9p+2HbK+x/fFtLRgAsG2qzvFfJum8iPiO\nJNk+UtLl6vwG73B5u9W2d5C00vYdEfHIZAsGAGybqnP824+EviRFxHclbd/pRhHxTESsLpc3SnpY\n0h6TqBMA0CVV9/jX2b5Q0jXl+mJJ6yayIdtvk7RQ0r0TuR0AoLuqBv8fSFoq6aZyfUV5WSXlNM+N\nks5p98bwxUuWvr58xNCQjhgaqnr3aDEcTVcwNby0cb1+/PTjTZcxZczZ9Zc0HDw4J+uJlWv15APd\nOUWaY4J/iPL0zNtHxPqK42dI+oakZRHxhTZj4uXXNk+oDoxteGtoxjQ3XcaUcOqSj9DLLjr3lLP1\n7n0WNl3GlLHTrJmKiEk9QKse1XOd7bm2t5e0RtJa2+dX3MaXJa1tF/oAgN6q+ubuvuUe/omSlqk4\nOdvpnW5k+3AVn/J9r+1Vth+wfeykqwUAbLOqc/wzbc9UEfxfiojNtjvOEUXE3ZKmb0uBAIDuqrrH\nf5mkJ1QcwrnC9nxJleb4AQD9peopGy5V8fWLI560fVQ9JQEA6tTpG7gWR8S1ts9rM+SzNdQEAKhR\npz3+kU/n7lh3IQCA3hg3+CPisvL30vHGAQAGR9Xj+PeyfZvt/7b9nO2v296r7uIAAN1X9aie6yT9\nk6TdJb1F0j9Lur6uogAA9aka/HMi4pqIGC5/rpX0hjoLAwDUo+oHuJbZ/hNJX5MUkk6VdLvtXaTX\nv6ELADAAqgb/KeXvj7ZcfpqKJwLm+wFgQFT9ANeCugsBAPTGuHP8tv9o1PLJLdddUldRAID6dHpz\n97RRy3/ach1n2QSAAdQp+N1meax1AMAA6BT80WZ5rHUAwADo9Obu/rbXq9i7n10uq1znOH4AGECd\nztXDl6gAwBRT9ZO7AIApguAHgGQIfgBIhuAHgGQIfgBIhuAHgGQIfgBIhuAHgGQIfgBIhuAHgGQI\nfgBIhuAHgGQIfgBIhuAHgGQIfgBIhuAHgGQIfgBIhuAHgGQIfgBIhuAHgGQIfgBIhuAHgGQIfgBI\nhuAHgGQIfgBIhuAHgGQIfgBIhuAHgGQIfgBIhuAHgGRqDX7bV9p+1vaDdW4HAFBd3Xv8V0n6zZq3\nAQCYgFqDPyLukvRindsAAEwMc/wAkMyMpgsY8VcXX/z68nuGhnTE0FCD1QBAf/ne8uW6a/nyrtyX\nI6Ird9R2A/Z8SbdFxLvGGRMvv7a51jqyGN4amjHNTZcxJZy65CP0sovOPeVsvXufhU2XMWXsNGum\nImJSD9BeTPW4/AEA9IG6D+e8TtI9kva2/ZTtM+vcHgCgs1rn+CPig3XePwBg4jiqBwCSIfgBIBmC\nHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCS\nIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgB\nIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmC\nHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgBIBmCHwCSIfgr+t7y5U2XUMldA1LnoPTz+SeeabqESgal\nzkH5uw9KnZNF8Fc0KIF694rBqHNQ+vk/Tw5GoA5KnYPydx+UOieL4AeAZAh+AEjGEdF0DbLdfBEA\nMGAiwpO5XV8EPwCgd5jqAYBkCH4ASKZnwW/7WNuP2H7U9h+3GXOp7R/ZXm17Ya9qa6lh3DptD9l+\nyfYD5c8FDdR4pe1nbT84zph+6OW4dfZDL8s65tm+0/ZDttfY/nibcY32tEqdTffU9na277W9qqzx\nojbjmu5lxzqb7mVLLdPKGm5tc/3E+hkRtf+oeIJ5TNJ8STMlrZa0T8uY35L0zXL5EEnf70Vtk6hz\nSNKtva6tpYb3SFoo6cE21zfey4p1Nt7Lso5flrSwXN5B0g/79PFZpc7GeyppTvl7uqTvSzq433pZ\nsc7Gezmqlk9IunaseibTz17t8R8s6UcR8WREbJb0NUkntIw5QdJXJCki7pW0k+0396i+EVXqlKRJ\nvZPeLRFxl6QXxxnSD72sUqfUcC8lKSKeiYjV5fJGSQ9L2qNlWOM9rVin1Pzjc1O5uJ2kGZJajyBp\nvJfltjvVKfXB49P2PEnHSbqizZAJ97NXwb+HpP8ctf5f+v8P2NYxPxljTN2q1ClJh5Uvqb5pe9/e\nlDYh/dDLqvqql7bfpuJVyr0tV/VVT8epU2q4p+W0xCpJz0j6dkTc1zKkL3pZoU6pPx6fn5N0vsZ+\nYpIm0U/e3J24lZL2jIiFkr4k6ZaG6xlkfdVL2ztIulHSOeUedV/qUGfjPY2IrRFxgKR5kg7phyf0\nsVSos/Fe2n6fpGfLV3pWl16B9Cr4fyJpz1Hr88rLWse8tcOYunWsMyI2jrxEjIhlkmba3qV3JVbS\nD73sqJ96aXuGijC9JiK+PsaQvuhppzr7qacRsV7SdyQd23JVX/RyRLs6+6SXh0s63vY6SddLOsr2\nV1rGTLifvQr++yS93fZ827MknSap9d3pWyWdIUm2D5X0UkQ826P6RnSsc/Tcme2DVXwI7oXellls\nXu2f/fuhlyPa1tlHvZSkL0taGxFfaHN9v/R03Dqb7qnt3WzvVC7PlnSMpEdahjXeyyp1Nt1LSYqI\nT0XEnhGxl4o8ujMizmgZNuF+zqil2hYRscX22ZLuUPFkc2VEPGz7o8XVcXlE3G77ONuPSfqZpDN7\nUdtE65T0e7bPkrRZ0iuSTu11nbavk3SkpF1tPyXpIkmz1Ee9rFKn+qCXZZ2HS/qQpDXlnG9I+pSK\no7v6pqdV6lTzPd1d0tW2p6n4P3RD2bu++r9epU4138u2trWfnLIBAJLhzV0ASIbgB4BkCH4ASIbg\nB4BkCH4ASIbgB4BkCH4MJNt72L7FxemzHytPSzuzy9s4wfY+o9aX2n5vF+53FxenV95g+9JtvT9g\nogh+DKqbJN0UEXtLeoekOZL+usvbOFHSfiMrEXFRRNzZhfv9uaQLJH2yC/cFTBjBj4FT7nW/EhEj\np6INFecrP932HNsftv3FUeNvs72oXD7G9j2277d9g+055eWfdvEFJ6ttf8b2YZKOl/QZF1+AscD2\nVbZ/pxx/dHn5D2xfMfJqw/bjtpfYXllet3dr/RGxKSLukfRqvZ0CxkbwYxDtp+LMia+LiA2SnpD0\n9pGLWm9ke1cVe9pHR8RB5X2cV55468SI2K88E+NfRMS/qTgHyvkRcWBEPD7qfraTdJWkkyNifxVf\n2nPWqE09FxG/LukfVZxOF+grBD+mkk6nrD1U0r6S7i7PdXOGirOxvizplXLP/SQV52UZz69KWhcR\nPy7Xr5a0aNT1N5e/V6o4jw7QVwh+DKK1kg4afYHtuZLerOLrCIf1i4/tN4wMk3RHuQd/QES8MyL+\nMCK2qPj2tRslvV/StyrUMN6TzMgUzhb16ESIwEQQ/Bg4EfGvkmbbXixJtqdL+htJX4yIV1VM+Sx0\n4a0qQl0qvlf1cNu/Ut5uju132N5e0s4R8S1J50l6Vzl+g6S5Y5TwQ0nzbe9Vrp8u6buT/Oc0/tV+\nyIfgx6A6SdLJth+V9LykLRHxaUmKiLtVhP9Dkj6v8v2AiHhe0u9Lut72DyTdo2LaZkdJ3ygvW6Hi\njWKp+M7l88s3aheofN+gfHI5U9KN5W22SLqsvE2l093aflzS30r6sO2nRh82CtSN0zJj4JVfPnG9\npJNGvowcQHsEPwAkw1QPACRD8ANAMgQ/ACRD8ANAMgQ/ACRD8ANAMgQ/ACTzvyPFvRCduf3WAAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x22e5d37eda0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEPCAYAAACneLThAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEXNJREFUeJzt3XmwHNV5xuH31YYWEASJXUFIYIzBBCFhwGxiTYixWVIh\ngA0kuCrO5kAgRWxTBAwhxHEWBxziQMAKOwQFjE0gAQxIAWIQQgKBWBJkViH2TRaFti9/9LlwudzR\ntK6mb/ec+j1Vt6Z7pnv6q56575w53XPaESEAQJ6G1F0AAKA6hDwAZIyQB4CMEfIAkDFCHgAyRsgD\nQMYGJeRtD7H9sO0fD8b2AACFwWrJnyJp4SBtCwCQVB7ytidI+oKkS6veFgDg4wajJf89SadL4qe1\nADDIKg1524dJeiUi5kty+gMADBJXOXaN7fMlHS9ppaRRkjaQdGNEnNhnOVr5ALCWIqJtw7nSkP/Y\nhuzpkv40Ig7v57F4b/nKQaljoM479xydedbZdZfR1pQDp+oz++9adxltbfzGCF10wQ/qLmONjv72\nV/XMfz+iT0+fUncpa3TaMX+sO66/WWc0/P25KkJ//RfnNr7Obnndbznv8lIhz3nyAJCxYYO1oYiY\nJWnWYG0PAEBLvrR99ptedwmlbDxx87pLKGXK56bVXUIp47pkf+47vTven91SZ7e87mUQ8iXx5uys\nKZ/bre4SShm/TXfsz32n7193CaV0S53d8rqXQcgDQMYIeQDIGCEPABkj5AEgY4Q8AGSMkAeAjBHy\nAJAxQh4AMkbIA0DGCHkAyBghDwAZI+QBIGOEPABkjJAHgIwR8gCQMUIeADJGyANAxgh5AMgYIQ8A\nGSPkASBjhDwAZIyQB4CMEfIAkDFCHgAyRsgDQMYIeQDIGCEPABkj5AEgY4Q8AGSMkAeAjBHyAJAx\nQh4AMkbIA0DGCHkAyBghDwAZI+QBIGOEPABkjJAHgIwR8gCQMUIeADJGyANAxgh5AMgYIQ8AGSPk\nASBjhDwAZGxYlU9uez1JsyWNSNuaGRHnVLlNAMBHKg35iPjA9gERscz2UEn32b4tIh6scrsAgELl\n3TURsSxNrqfiQyWq3iYAoFB5yNseYnuepCWS7oiIOVVvEwBQqLS7RpIiYrWkXW2PlfQj2ztGxMK+\ny5137kdd9fvsN137Tp9edWlrJSStiuZ/CXl76TtatPjZustoa/nKFY3fn+zLzuJ/aN0sf3Wplr+2\ndK3Xqzzke0TEu7bvlnSopE+E/JlnnT1YpQzIqggNtesuo62N1t9Qu+8wpe4y2hoxbHjj9yf7srP4\nH1pHO3x89pbzLi+1WqXdNbbH294wTY+SdIikJ6vcJgDgI1W35LeQdLntISo+UK6PiFsr3iYAIKn6\nFMoFkqZWuQ0AQGv84hUAMkbIA0DGCHkAyBghDwAZI+QBIGOEPABkjJAHgIwR8gCQMUIeADJGyANA\nxgh5AMgYIQ8AGSPkASBjhDwAZIyQB4CMEfIAkDFCHgAyRsgDQMYIeQDIGCEPABkj5AEgY4Q8AGSM\nkAeAjBHyAJAxQh4AMkbIA0DGCHkAyBghDwAZI+QBIGOEPABkjJAHgIwR8gCQMUIeADJGyANAxkqF\nvAvH2z4rzW9te/dqSwMArKuyLfl/kvR5Scel+fckXVRJRQCAjhlWcrk9ImKq7XmSFBFv2R5RYV0A\ngA4o25JfYXuopJAk25tIWl1ZVQCAjigb8hdKuknSprb/UtK9ks6vrCoAQEeU6q6JiKttz5V0kCRL\nOjIinqi0MgDAOltjyNveuNfsq5Ku7f1YRLxZVWEAgHXXriU/V0U/vCVtLemtNL2RpOclTaq0OgDA\nOlljn3xETIqIyZLulPSliBgfEeMkfVHS7YNRIABg4MoeeN0zIm7tmYmI2yTtVU1JAIBOKXue/GLb\nZ0q6Ks1/RdLiakoCAHRK2Zb8cZI2UXEa5U2SNtVHv34FADRU2VMo35R0iu0NitlYWm1ZAIBOKDtA\n2c5pSIPHJD1ue67tz5ZYb4Ltu2w/bnuB7ZPXtWAAQHll++QvlnRaRNwtSbb3l3SJ2h98XZnWm297\nfUlzbd8eEU8OtGAAQHll++TH9AS8JEXEPZLGtFspIpZExPw0vVTSE5K2GkCdAIABKNuSX2T7zyVd\nmeaPl7RobTZkextJUyQ9sDbrAQAGrmzIf1XSOZJuTPOz032lpK6amZJOaXXQ9tRvnfrh9LS99tC0\nvfYs+/SDYtMx4/TqL96ou4y23v5gqV5b1vw6V6xepdURdZexRkuXL9PLv3it7jLa6oZ9KRU/ne+G\nOpv6ur//0tt6f/Hba71e2bNr3pJ0siSlIYfHRMS7Zda1PUxFwF8ZETe3Wu6XvvSZD6cX6V0teqpZ\nP6hd+MZz2nHcxLrLaGvcXp/SvuMOrruMtjbZfBONGNrsq0/uftiBGuq6q2ivG/alJK0OaUgX7M9u\ned3/Zv8/LLVc2bNrrrE91vYYSQskLbR9eslafihpYURcUHJ5AECHlP343zG13I+UdJuKgclOaLeS\n7b1V/Dr2QNvzbD9s+9ABVwsAWCtl++SH2x6uIuT/MSJW2G7buRYR90kaui4FAgAGrmxL/mJJz6o4\nbXK27YmSSvXJAwDqU/bA64UqLgHY4znbB1RTEgCgU9pdGer4iLjK9mktFvn7CmoCAHRIu5Z8z69a\nN6i6EABA560x5CPi4nR7zuCUAwDopLLnyU+2/RPbr9l+1fbNtidXXRwAYN2UPbvmGkn/JmkLSVtK\nukHStVUVBQDojLIhPzoiroyIlenvKkkjqywMALDuyv4Y6jbb35R0nYpxho6RdKvtjaUPrxwFAGiY\nsiH/W+n29/rcf6yK0Kd/HgAaqOyPoSZVXQgAoPPW2Cdv+896TR/d57HzqyoKANAZ7Q68Httr+lt9\nHmM0SQBouHYh7xbT/c0DABqmXchHi+n+5gEADdPuwOsutt9V0WoflaaV5jlPHgAart3YNVzwAwC6\nWPOv/gsAGDBCHgAyRsgDQMYIeQDIGCEPABkj5AEgY4Q8AGSMkAeAjBHyAJAxQh4AMkbIA0DGCHkA\nyBghDwAZI+QBIGOEPABkjJAHgIwR8gCQMUIeADJGyANAxgh5AMgYIQ8AGSPkASBjhDwAZIyQB4CM\nEfIAkDFCHgAyRsgDQMYIeQDIGCEPABkj5AEgY5WGvO3LbL9i+9EqtwMA6F/VLfkZkn6t4m0AAFqo\nNOQj4l5Jb1W5DQBAa/TJA0DGhtVdQI9518z+cHrybjtr2912rrGaT9oyhmv82M3qLqOtbqnzBwt/\nqu02bHad72m4tuuCfTly+MjuaK25O1qVm4ydoCGuu4pPeuahBVr00IK1Xq8xIX/I73+57hIAoLG2\n7dP4vfOS60qtNxgfrE5/AIBBVvUplNdIul/S9raft31SldsDAHxcpd01EUEfDADUqBuOgwAABoiQ\nB4CMEfIAkDFCHgAyRsgDQMYIeQDIGCEPABkj5AEgY4Q8AGSMkAeAjBHyAJAxQh4AMkbIA0DGCHkA\nyBghDwAZI+QBIGOEPABkjJAHgIwR8gCQMUIeADJGyANAxgh5AMgYIQ8AGSPkASBjhDwAZIyQB4CM\nEfIAkDFCHgAyRsgDQMYIeQDIGCEPABkj5AEgY4Q8AGSMkAeAjBHyAJAxQh4AMkbIA0DGCHkAyBgh\nDwAZI+QBIGOEPABkjJAHgIwR8gCQMUIeADJGyANAxgh5AMgYIQ8AGSPkS1o8/6m6SyiFOjvrmYcW\n1F1CKbPuuafuEkrpljq75XUvg5Av6eUuCSXq7KxFXfLPPnvWrLpLKKVb6uyW170MQh4AMkbIA0DG\nHBF11yDb9RcBAF0mItxumUaEPACgGnTXAEDGCHkAyFitIW/7UNtP2n7a9jfqrKUV25fZfsX2o3XX\nsia2J9i+y/bjthfYPrnumvpjez3bD9iel+o8u+6aWrE9xPbDtn9cdy2t2H7W9iNpfz5Ydz2t2N7Q\n9g22n0jv0T3qrqkv29un/fhwun2nwf9Hp9p+zPajtq+2PaLlsnX1ydseIulpSQdJWixpjqRjI+LJ\nWgpqwfY+kpZKuiIifqXuelqxvbmkzSNivu31Jc2VdETT9qck2R4dEctsD5V0n6STI6JxAWX7VEnT\nJI2NiMPrrqc/thdJmhYRb9Vdy5rY/ldJsyJihu1hkkZHxLs1l9VSyqcXJe0RES/UXU9vtreUdK+k\nHSJiue3rJf1HRFzR3/J1tuR3l/S/EfFcRKyQdJ2kI2qsp18Rca+kRv8DSVJELImI+Wl6qaQnJG1V\nb1X9i4hlaXI9ScMkNe7ov+0Jkr4g6dK6a2nDani3q+2xkvaNiBmSFBErmxzwycGSnmlawPcyVNKY\nng9MFQ3lftX55thKUu8d+KIaGkrdxvY2kqZIeqDeSvqXukHmSVoi6Y6ImFN3Tf34nqTT1cAPoD5C\n0h2259j+3bqLaWGSpNdtz0hdIZfYHlV3UW0cI+nauovoT0QslvR3kp6X9JKktyPizlbLN7oFgLWX\numpmSjoltegbJyJWR8SukiZI2sP2jnXX1JvtwyS9kr4ZOf011d4RMVXFt44/St2LTTNM0lRJF6Va\nl0n6Zr0ltWZ7uKTDJd1Qdy39sb2Ril6PiZK2lLS+7S+3Wr7OkH9J0ta95iek+zBA6avbTElXRsTN\nddfTTvrKfrekQ+uupY+9JR2e+ruvlXSA7X77O+sWES+n29ck3aSiG7RpXpT0QkQ8lOZnqgj9pvp1\nSXPTPm2igyUtiog3I2KVpBsl7dVq4TpDfo6k7WxPTEeGj5XU1LMYmt6a6/FDSQsj4oK6C2nF9njb\nG6bpUZIOkdSog8MRcUZEbB0Rk1W8L++KiBPrrqsv26PTNzfZHiPpVyU9Vm9VnxQRr0h6wfb26a6D\nJC2ssaR2jlNDu2qS5yXtaXukbavYn0+0WnjYoJXVR0Sssv11Sber+LC5LCJaFloX29dI2l/SONvP\nSzq75wBSk9jeW9JXJC1I/d0h6YyI+M96K/uELSRdns5eGCLp+oi4teaautVmkm5Kw4IMk3R1RNxe\nc02tnCzp6tQVskjSSTXX0y/bo1W0lL9Wdy2tRMSDtmdKmidpRbq9pNXyDGsAABnjwCsAZIyQB4CM\nEfIAkDFCHgAyRsgDQMYIeQDIGCGPrmR7K9s/SsNU/5/tC9M52J3cxhG2d+g1f47tAzvwvAfbfigN\nETzH9gHr+pxAK5wnj65k+wEVY6FckX719y+SlkbEn3RwGzMk3RIR/96p50zPu4uKsXGW2N5J0n9F\nxIRObgPoQUseXSe1pt/vGT87ipbKqZJOSD/1/23b3++1/E9s75emD7F9f2pJX59+4Sjb30kXs5hv\n+7u2P69ikKrvppETJ6VRFH8jLX9Quv8R25f2fIuw/XPb37Y9Nz22vfqIiEciYkmaflzSyE5/CwF6\nEPLoRjupuCjKhyLiPUnPStqu566+K9keJ+lMSQdFxG7pOU6zvbGkIyNip4iYIum8iPgfFWMpnR4R\nUyPi572eZz1JMyQdHRG7SBou6Q96berViJgm6Z9VDFfcku3flPRwuqYC0HGEPHLSbhC5PSXtKOm+\nNL7PiSpGQn1H0vupRX6UpPfbPM+nVYwC+Eyav1zSfr0evyndzlUxHGz/xRZdNX+lBo+Tgu5HyKMb\nLZS0W+870tWHNpP0lKSV+vh7e2TPYpJuTy3zXSPisxHxtTRc6+4qhsD9oqQyg7qt6QPlg3S7Si0G\nAUxXnrpR0gkR8WyJ7QEDQsij60TETyWNsn28JKVrxf6tpO9HxAcqum2muPDL+miM9Z9J2tv2tmm9\n0bY/lYbp3SiN2HmapJ5r+b4naWw/JTwlaaLtyWn+BEn3lK0/DbV8i6RvRMTPyq4HDAQhj251lKSj\nbT8t6XVJqyLiO5IUEfepCPrHJf2DUv99RLwu6XckXWv7EUn3q+h62UDSLem+2SoO4krFdYdPTwdR\nJyn186cPkpMkzUzrrJJ0cVqnzOlqX5e0raSzbM9LB3DHD3RHAGvCKZToerb3VHGRh6N6LmYOoEDI\nA0DG6K4BgIwR8gCQMUIeADJGyANAxgh5AMgYIQ8AGSPkASBj/w9gB2h/K3nZSwAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x22e5d95eeb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEPCAYAAACneLThAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEN9JREFUeJzt3XuQZGV9xvHvA8tyv5RcBCHcYoBCjIgKKEYQNCGKCLGM\noKghVTFapiCSIhpLQ6gylpVKYtSYEqMSVEAiiBcERYNKgQmLC4vcURFQCLAIClsYl8svf/QZGJaZ\nnd7d7jk9b30/VVNzTs85fZ7qnX36zNun305VIUlq03p9B5AkjY8lL0kNs+QlqWGWvCQ1zJKXpIZZ\n8pLUsHkp+STrJbkqyVfn43iSpIH5OpM/Ebhhno4lSeqMveST7AS8CvjUuI8lSXqq+TiT/zBwMuBb\nayVpno215JO8GrinqpYB6b4kSfMk45y7JskHgeOAR4GNgc2BL1XVW1bZzrN8SVpDVTXnifNYS/4p\nB0oOBv6qqo6c4Wd1xPveOi851tYl532DQ193eN8x5mTO0Xn3sSfwzS98hfefckrfUVbr6FOO50eX\nXsOeB+/bd5TVWnLTMra8B3OOyAUfOGOokvc6eUlq2KL5OlBVfQ/43nwdT5LkmfzQFm+7Wd8RhmLO\n0XrZwQf3HWEoW++yfd8RhmLO+WfJD2nxdgujlMw5WgcfckjfEYayza4Lo5TMOf8seUlqmCUvSQ2z\n5CWpYZa8JDXMkpekhlnyktQwS16SGmbJS1LDLHlJapglL0kNs+QlqWGWvCQ1zJKXpIZZ8pLUMEte\nkhpmyUtSwyx5SWqYJS9JDbPkJalhlrwkNcySl6SGWfKS1DBLXpIaZslLUsMseUlqmCUvSQ2z5CWp\nYZa8JDXMkpekhlnyktQwS16SGmbJS1LDLHlJapglL0kNs+QlqWGWvCQ1zJKXpIZZ8pLUMEtekhpm\nyUtSwyx5SWqYJS9JDbPkJalhlrwkNcySl6SGLRrnnSfZELgUWNwd69yqOnWcx5QkPWmsJV9Vv0ny\n8qp6OMn6wOVJLqqqJeM8riRpYOzDNVX1cLe4IYMnlRr3MSVJA2Mv+STrJbkauBv4VlVdOe5jSpIG\nxjpcA1BVjwPPT7IF8OUke1fVDatud8l533hiefG2m7F4u83GHW2NLP/VL1hy07K+Y8zJnKPzyKOP\n8njfIYbwyxUP8uM7f9p3jDkthH9zmNycK+9dwcrlK9Z4v7GX/JSqejDJd4DDgaeV/KGvO3y+oqyV\nJTctY/+99u07xpzMOTobLFq0IC4/22qzLSb+sYSF8W8OE5xzr6euXvCBM4babay/w0m2SbJlt7wx\n8ErgpnEeU5L0pHGfye8AnJFkPQZPKOdU1YVjPqYkqTPuSyivBfYb5zEkSbNbCEOOkqS1ZMlLUsMs\neUlqmCUvSQ2z5CWpYZa8JDXMkpekhlnyktQwS16SGmbJS1LDLHlJapglL0kNs+QlqWGWvCQ1zJKX\npIZZ8pLUMEtekhpmyUtSwyx5SWqYJS9JDbPkJalhlrwkNcySl6SGWfKS1DBLXpIaZslLUsMseUlq\nmCUvSQ2z5CWpYZa8JDXMkpekhlnyktQwS16SGmbJS1LDhir5DByX5G+79Z2T7D/eaJKkdTXsmfy/\nAS8Gju3WHwI+PpZEkqSRWTTkdgdU1X5JrgaoqgeSLB5jLknSCAx7Jv9IkvWBAkiyLfD42FJJkkZi\n2JL/KHA+sF2SvwcuAz44tlSSpJEYarimqs5MshQ4DAhwVFXdONZkkqR1ttqST/KMaav3AmdP/1lV\n3T+uYJKkdTfXmfxSBuPwAXYGHuiWtwLuAHYbazpJ0jpZ7Zh8Ve1WVbsD3wZeU1XbVNXWwBHAxfMR\nUJK09oZ94fXAqrpwaqWqLgJeMp5IkqRRGfY6+buSvA/4fLf+JuCu8USSJI3KsGfyxwLbMriM8nxg\nO55896skaUINewnl/cCJSTYfrNaK8caSJI3CsBOUPbeb0uA64PokS5PsM8R+OyW5JMn1Sa5NcsK6\nBpYkDW/YMfnTgJOq6jsASQ4BPsncL74+2u23LMlmwNIkF1fVTWsbWJI0vGHH5DedKniAqvousOlc\nO1XV3VW1rFteAdwI7LgWOSVJa2HYM/lbk7wf+Fy3fhxw65ocKMmuwL7AFWuynyRp7Q1b8n8KnAp8\nqVu/tLttKN1QzbnAibO9aHvJV7/5xPLiZ27O4u03H/bu58XyFfez5PZr+44xJ3OOzvIVD7Dkjuv6\njjGn5Sse4Pb77+w7xpwWwr85TG7OlXc/xMp7Hlrj/VJVa7bDYMrhTavqwSG3XwRcAFxUVR+ZZZs6\n4hPvXKMc823J7dey/y7P7TvGnMw5OgshI5hz1BZKzgve/nGqKnNtN+zVNWcl2SLJpsC1wA1JTh4y\ny2eAG2YreEnS+Az7wuve3Zn7UcBFDCYme/NcOyU5iMG7Yw9NcnWSq5IcvtZpJUlrZNgx+Q2SbMCg\n5P+1qh5JMuc4T1VdDqy/LgElSWtv2DP504DbGFw2eWmSXYChxuQlSf0ZdlqDjzL4CMAptyd5+Xgi\nSZJGZa5Phjquqj6f5KRZNvnnMWSSJI3IXGfyU+9qnayL1iVJQ1ltyVfVad33U+cnjiRplIa9Tn73\nJF9LsjzJvUm+kmT3cYeTJK2bYa+uOQv4T2AH4FnAF4GzxxVKkjQaw5b8JlX1uap6tPv6PLDROINJ\nktbdsG+GuijJe4AvAAW8AbgwyTPgiU+OkiRNmGFL/o+773++yu3HMCh9x+claQIN+2ao3cYdRJI0\neqsdk0/y19OWX7/Kzz44rlCSpNGY64XXY6Yt/80qP3M2SUmacHOVfGZZnmldkjRh5ir5mmV5pnVJ\n0oSZ64XX5yV5kMFZ+8bdMt2618lL0oSba+4aP/BDkhawYd/xKklagCx5SWqYJS9JDbPkJalhlrwk\nNcySl6SGWfKS1DBLXpIaZslLUsMseUlqmCUvSQ2z5CWpYZa8JDXMkpekhlnyktQwS16SGmbJS1LD\nLHlJapglL0kNs+QlqWGWvCQ1zJKXpIZZ8pLUMEtekhpmyUtSwyx5SWqYJS9JDbPkJalhlrwkNcyS\nl6SGjbXkk3w6yT1JfjjO40iSZjbuM/nTgT8Y8zEkSbMYa8lX1WXAA+M8hiRpdo7JS1LDFvUdYMrN\nX1vyxPLWe+zINnvu2GMaSZos9918J7+45c413m9iSn7P1+zfdwRJmljb7PnUk98fff3Kofabj+Ga\ndF+SpHk27ksozwK+D+yR5I4kx4/zeJKkpxrrcE1VvXGc9y9JWj2vrpGkhlnyktQwS16SGmbJS1LD\nLHlJapglL0kNs+QlqWGWvCQ1zJKXpIZZ8pLUMEtekhpmyUtSwyx5SWqYJS9JDbPkJalhlrwkNcyS\nl6SGWfKS1DBLXpIaZslLUsMseUlqmCUvSQ2z5CWpYZa8JDXMkpekhlnyktQwS16SGmbJS1LDLHlJ\napglL0kNs+QlqWGWvCQ1zJKXpIZZ8pLUMEtekhpmyUtSwyx5SWqYJS9JDbPkJalhlrwkNcySl6SG\nWfKS1DBLXpIaZslLUsMseUlqmCUvSQ2z5CWpYZb8kFbe/su+IwzFnKN138139h1hKOYcrYWScxiW\n/JBW3rEwSsmco/WLWxbGf3ZzjtZCyTkMS16SGmbJS1LDUlV9ZyBJ/yEkaYGpqsy1zUSUvCRpPByu\nkaSGWfKS1LBeSz7J4UluSnJLknf3mWU2ST6d5J4kP+w7y+ok2SnJJUmuT3JtkhP6zjSTJBsmuSLJ\n1V3OU/rONJsk6yW5KslX+84ymyS3JbmmezyX9J1nNkm2TPLFJDd2v6MH9J1pVUn26B7Hq7rvv5rg\n/0fvSnJdkh8mOTPJ4lm37WtMPsl6wC3AYcBdwJXAMVV1Uy+BZpHkpcAK4LNV9bt955lNku2B7atq\nWZLNgKXAayft8QRIsklVPZxkfeBy4ISqmriCSvIu4AXAFlV1ZN95ZpLkVuAFVfVA31lWJ8l/AN+r\nqtOTLAI2qaoHe441q66ffg4cUFU/6zvPdEmeBVwG7FVVK5OcA3y9qj470/Z9nsnvD/yoqm6vqkeA\nLwCv7THPjKrqMmCi/wMBVNXdVbWsW14B3Ajs2G+qmVXVw93ihsAiYOJe/U+yE/Aq4FN9Z5lDmPBh\n1yRbAL9XVacDVNWjk1zwnVcAP5m0gp9mfWDTqSdMBifKM+rzl2NHYPoD+HMmtJQWmiS7AvsCV/Sb\nZGbdMMjVwN3At6rqyr4zzeDDwMlM4BPQKgr4VpIrk/xZ32FmsRtwX5LTu6GQTybZuO9Qc3gDcHbf\nIWZSVXcB/wTcAdwJ/LKqvj3b9hN9BqA11w3VnAuc2J3RT5yqeryqng/sBByQZO++M02X5NXAPd1f\nRum+JtVBVbUfg7863tkNL06aRcB+wMe7rA8D7+k30uySbAAcCXyx7ywzSbIVg1GPXYBnAZsleeNs\n2/dZ8ncCO09b36m7TWup+9PtXOBzVfWVvvPMpfuT/TvA4X1nWcVBwJHdePfZwMuTzDje2beq+t/u\n+3LgfAbDoJPm58DPquoH3fq5DEp/Uv0hsLR7TCfRK4Bbq+r+qnoM+BLwktk27rPkrwSenWSX7pXh\nY4BJvYph0s/mpnwGuKGqPtJ3kNkk2SbJlt3yxsArgYl6cbiq3ltVO1fV7gx+Ly+pqrf0nWtVSTbp\n/nIjyabA7wPX9Zvq6arqHuBnSfbobjoMuKHHSHM5lgkdquncARyYZKMkYfB43jjbxovmLdYqquqx\nJH8BXMzgyebTVTVr0L4kOQs4BNg6yR3AKVMvIE2SJAcBbwKu7ca7C3hvVX2j32RPswNwRnf1wnrA\nOVV1Yc+ZFqpnAud304IsAs6sqot7zjSbE4Azu6GQW4Hje84zoySbMDhTflvfWWZTVUuSnAtcDTzS\nff/kbNs7rYEkNcwXXiWpYZa8JDXMkpekhlnyktQwS16SGmbJS1LDLHktSEl2TPLlbprqHyf5aHcN\n9iiP8doke01bPzXJoSO43xd1U9lOfR21rvcpzcbr5LUgJbmCwVwon+3e9ffvwIqq+ssRHuN04IKq\nOm9U99nd70bAyqp6vJsi+hpgh6p6fJTHkcAzeS1A3dn0r6fmz67Bmcq7gDd3b/V/a5KPTdv+a0le\n1i2/Msn3k/wgyTndOxxJ8qHuwyyWJfmHJC9mMEnVP3QzJ+7WzaL4R932h3W3X5PkU1N/RST5aZK/\nS7K0+9kerKKq/m9aoW8MWO4aG0teC9FzGHwoyhOq6iHgNuDZUzetulOSrYH3AYdV1Qu7+zgpyTOA\no6rqOVW1L/CBqvpvBnMpnVxV+1XVT6fdz4bA6cDrq+p5wAbAO6Yd6t6qegHwCQbTFT9Nkv2TXMfg\nLP7tnsVrXCx5tWSuSeQOBPYGLu/m93kLg5lQfwX8ujsjPxr49Rz3syeDWQB/0q2fAbxs2s/P774v\nZTAd7NNU1ZKq2gd4EfDe1X18m7QuLHktRDcAL5x+Q/fpQ88EbgYe5am/2xtNbQZc3J2ZP7+q9qmq\nt3XTte7PYArcI4BhJnVb3RPKb7rvjzHHJIBVdTODj5fcZ4hjSmvMkteCU1X/BWyc5DiA7rNi/xH4\nWFX9hsGwzb4Z+C2enGP9f4CDkvx2t98mSX6nm6Z3q27GzpOAqc/yfQjYYoYINwO7JNm9W38z8N1h\n8yfZtctMkl0Y/GVw27D7S2vCktdCdTTw+iS3APcBj1XVhwCq6nIGpXk98C904/dVdR/wJ8DZSa4B\nvs+gYDcHLuhuu5TBi7gw+Nzhk7sXUXejG+fvnkiOB87t9nkMOK3bZ5jL1V4KXJPkKuA84B1Vdf9a\nPg7SankJpRa8JAcy+JCHo6c+zFzSgCUvSQ1zuEaSGmbJS1LDLHlJapglL0kNs+QlqWGWvCQ1zJKX\npIb9P7l9efuPQKOYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x22e5db8d518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEPCAYAAABBUX+lAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEKtJREFUeJzt3X+QXWV9x/H3B4KRXwoKiEKTgApW/EEpAygtKpSR0cqP\ntrSgoNIZp9OpyoiDVWurTFuGsVqrqFP8xSAKWCmKOtAiRU3FViUmSAigLSZRGX5JLFAYhPDtH/eE\nLmGzeze7Z29unvdrZifnnHvv83xns/u5zz7n3OekqpAkbf22GXUBkqT5YeBLUiMMfElqhIEvSY0w\n8CWpEQa+JDViXgI/yTZJfpDkK/PRnyTpieZrhH86sGqe+pIkTaL3wE+yN/Aq4FN99yVJ2rT5GOF/\nCDgT8CO9kjRCvQZ+klcDd1TVCiDdlyRpBNLnWjpJzgZOAR4Btgd2Bi6rqtdv9DxH/5I0Q1U1o0F0\nr4H/uI6SlwFvr6pjJ3msjvvkW3rp97q1Kzl40Qt6aRvgmgu/ypGnvqa39q1/an3W33ft161dyU4r\n/pfnHXtob+1b/9Ttj3P9l7/p3BkHvtfhS1IjFsxXR1X1LeBb89WfJOnxHOHP0sLFu4y6hFmx/tHa\nbf+9Rl3CrFj/eDHwZ2nhkl1HXcKsWP9o7bb/3qMuYVasf7wY+JLUCANfkhph4EtSIwx8SWqEgS9J\njTDwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQI\nA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY+JLUCANfkhph4EtSIwx8SWqEgS9JjTDw\nJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+S\nGmHgS1IjDHxJaoSBL0mNWNBn40kWAkuBJ3V9XVpVZ/XZpyRpcr0GflU9lOQVVfVAkm2Ba5NcWVXf\n67NfSdIT9T6lU1UPdJsLGbzBVN99SpKeqPfAT7JNkuXA7cDXq+r7ffcpSXqiVM3PgDvJU4AvA2+u\nqlUbPVY7H7Hksf2Fi3dh4ZJd56Tfu+5fx+47zU1btt9W++Ncu+1vfe0/tHodD6355WP79y1dTVVl\nJm30Ooc/UVXdm+QbwDHAqo0fP/LU1/TS73VrV3Lwohf00rbtb93tj3Pttr8Vtr/o8buXLz13xk30\nOqWTZLckT+22tweOBm7us09J0uT6HuE/E7ggyTYM3ly+UFVX9NynJGkSfV+WeQNwUJ99SJKG4ydt\nJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+S\nGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY+JLUCANfkhph4EtSIwx8SWqEgS9JjTDwJakR\nBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDViqMDPwClJ/qrbX5TkkH5L\nkyTNpWFH+B8HXgKc3O3fB3ysl4okSb1YMOTzDq2qg5IsB6iqdUme1GNdkqQ5NuwI/+Ek2wIFkGR3\n4NHeqpIkzblhA/8jwJeAPZL8LfBt4OzeqpIkzbmhpnSq6vNJlgFHAQGOr6qbeq1MkjSnpgz8JE+b\nsHsncPHEx6rqnr4KkyTNrelG+MsYzNsHWASs67Z3AdYC+/RanSRpzkw5h19V+1TVvsDVwGuqareq\nejrwu8BV81GgJGluDHvS9rCqumLDTlVdCby0n5IkSX0Y9jr825K8B/hct/864LZ+SpIk9WHYEf7J\nwO4MLs38ErAH//+pW0nSGBj2ssx7gNOT7DzYrfv7LUuSNNeGXTzthd2yCiuBG5MsS/KCIV63d5Jr\nktyY5IYkb51twZKkzTPsHP55wBlV9Q2AJC8HPsH0J24f6V63IslOwLIkV1XVzZtbsCRp8ww7h7/j\nhrAHqKpvAjtO96Kqur2qVnTb9wM3AXttRp2SpFkadoR/a5K/BC7s9k8Bbp1JR0mWAAcC353J6yRJ\ncyNVNf2Tkl2Bs4Df6g4tBc6qqnVDdTKYzvkm8NdVdfkkj9fORyx5bH/h4l1YuGTXYZqe1l33r2P3\nneamLdtvq/1xrt32t772H1q9jofW/PKx/fuWrqaqMpM2hgr8x71gsEzyjlV175DPXwB8Dbiyqj68\niefUcZ98y4zqGNZ1a1dy8KJpzy/bvu3Pa9u2b/uzdfmbzp1x4A97lc5FSZ6SZEfgBmBVkjOH7OMz\nwKpNhb0kaX4Me9L2+d2I/njgSgaLpp063YuSHM7gU7lHJlme5AdJjtnsaiVJm23Yk7bbJdmOQeB/\ntKoeTjLtXFBVXQtsO5sCJUlzY9gR/nnAagaXYi5NshgYag5fkrRlGHZphY8wuM3hBmuSvKKfkiRJ\nfZjujlenVNXnkpyxiaf8fQ81SZJ6MN0If8OnaXfuuxBJUr+mDPyqOq/796z5KUeS1Jdhr8PfN8lX\nk9yV5M4klyfZt+/iJElzZ9irdC4C/gl4JvAs4IvAxX0VJUmae8MG/g5VdWFVPdJ9fQ54cp+FSZLm\n1rAfvLoyyTuBS4AC/gi4IsnT4LE7YkmStmDDBv4fdv/+yUbHT2LwBuB8viRt4Yb94NU+fRciSerX\nlHP4Sd4xYfvEjR47u6+iJElzb7qTtidN2H7XRo+56qUkjZHpAj+b2J5sX5K0BZsu8GsT25PtS5K2\nYNOdtH1xknsZjOa377bp9r0OX5LGyHRr6XjzEknaSgz7SVtJ0pgz8CWpEQa+JDXCwJekRhj4ktQI\nA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY+JLUCANfkhph4EtSIwx8SWqEgS9JjTDw\nJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+S\nGtFr4Cf5dJI7kvywz34kSdPre4R/PvDKnvuQJA2h18Cvqm8D6/rsQ5I0HOfwJakRC0ZdwAarPrv0\nse2nL96T3ZbsOSftPnzzPayve+ekLdvf8tq/5OQPcMj+B/bS9qP0OyI64b2nsX7N+H7vbX9+2797\n9e38Ys3ts2pjiwn8/V/Wzy+tJG0Ndlvy+IHwj//9+hm3MR9TOum+JEkj1PdlmRcB3wH2S7I2yWl9\n9idJ2rRep3Sq6rV9ti9JGp5X6UhSIwx8SWqEgS9JjTDwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBL\nUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1\nwsCXpEYY+JLUCANfkhph4EtSIwx8SWqEgS9JjTDwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMM\nfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCX\npEYY+JLUCAN/ln515/2jLmFWxr3+5d+7btQlzMrdq28fdQmzYv3jxcCfpV/dNd6BOe71L//+slGX\nMCu/WDPegWP948XAl6RGGPiS1IhU1ahrIMnoi5CkMVNVmcnzt4jAlyT1zykdSWqEgS9JjRhp4Cc5\nJsnNSX6U5M9HWctMJdk7yTVJbkxyQ5K3jrqmmUqyTZIfJPnKqGvZHEmemuSLSW7q/h8OHXVNw0ry\ntiQrk/wwyeeTPGnUNU0lyaeT3JHkhxOO7ZrkqiS3JPnXJE8dZY1T2UT97+9+dlYk+eckTxlljVOZ\nrP4Jj709yaNJnjZdOyML/CTbAB8FXgkcAJyc5HmjqmczPAKcUVUHAC8B/mzM6gc4HVg16iJm4cPA\nFVX168CLgZtGXM9QkjwLeAtwUFW9CFgAnDTaqqZ1PoPf1YneCVxdVfsD1wDvmveqhjdZ/VcBB1TV\ngcCPGb/6SbI3cDSwZphGRjnCPwT4cVWtqaqHgUuA40ZYz4xU1e1VtaLbvp9B2Ow12qqG1/2gvAr4\n1Khr2RzdaOy3q+p8gKp6pKruHXFZM7EtsGOSBcAOwG0jrmdKVfVtYN1Gh48DLui2LwCOn9eiZmCy\n+qvq6qp6tNv9T2DveS9sSJv4/gN8CDhz2HZGGfh7AT+dsP8zxigwJ0qyBDgQ+O5oK5mRDT8o43qZ\n1j7A3UnO76alPpFk+1EXNYyqug34ILAW+Dnwy6q6erRVbZY9quoOGAyAgD1GXM9s/DFw5aiLmIkk\nxwI/raobhn2NJ21nKclOwKXA6d1If4uX5NXAHd1fKOm+xs0C4CDgY1V1EPAAgymGLV6SXRiMjhcD\nzwJ2SvLa0VY1J8Zy8JDkL4CHq+qiUdcyrG5w827gvRMPT/e6UQb+z4FFE/b37o6Nje7P8UuBC6vq\n8lHXMwOHA8cmuRW4GHhFks+OuKaZ+hmD0c2G1dMuZfAGMA5+B7i1qu6pqvXAZcBLR1zT5rgjyTMA\nkuwJ3DniemYsyRsZTG2O2xvus4ElwPVJfsIgP5clmfKvrFEG/veB5yRZ3F2hcBIwbleLfAZYVVUf\nHnUhM1FV766qRVW1L4Pv+zVV9fpR1zUT3VTCT5Ps1x06ivE5Ab0WOCzJk5OEQe3jcMJ5478GvwK8\nsdt+A7ClD3oeV3+SYxhMax5bVQ+NrKrhPVZ/Va2sqj2rat+q2ofBAOg3qmrKN92RBX43snkzgzPl\nNwKXVNU4/NADkORw4HXAkUmWd/PIx4y6rsa8Ffh8khUMrtI5e8T1DKWqvsfgL5LlwPUMfok/MdKi\nppHkIuA7wH5J1iY5DTgHODrJLQzetM4ZZY1T2UT95wI7AV/vfn8/PtIip7CJ+icqhpjScWkFSWqE\nJ20lqREGviQ1wsCXpEYY+JLUCANfkhph4EtSIwx8jaUkeyX5cre09n8l+UiS7ea4j+MmroCa5Kwk\nR85h+4uS3JfkjLlqU5qKga9xdRlwWVXtBzyXwYqTfzfHfRzPYOluAKrqvVV1zRy2/0HgijlsT5qS\nga+x042yH6yqzwLU4NODbwNOTbJDkjckOXfC87+a5Ihu++gk30lyXZIvJNmhO35OdxOVFd2NMV4C\nHAu8v/sU5j7dypy/1z3/qO749Uk+teGviyQ/SfK+JMu6x/ZjEkmOA25l8ClzaV4Y+BpHBwDLJh6o\nqvuA1cBzNhza+EVJng68Bziqqg7u2jiju1PQ8VW14WYYf1NV/8FgrZgzq+qgqvrJhHYWMrghxYlV\n9WJgO+BPJ3R1Z1X9JvCPTLJWeZIdgXcAZzGeK5VqTBn42ppMF56HAc8Hrk2yHHg9gxVb/wd4sBup\nnwA8OE07+zNY7fK/u/0LgCMmPP6l7t9lDJZA3tj7gA9V1QND1i3NiQWjLkDaDKuAP5h4oLsD1jOA\nW4AX8vjBzJM3PA24qqpet3GDSQ5hsADYiQwW9TtqmhqmCukNKy+uZ/LfsUOB30/yfmBXYH2SB6tq\ni128S1sHR/gaO1X1b8D2SU4BSLIt8AHg3G6Z29XAgRn4NQa304TBbewOT/Ls7nU7JHluN8WyS1X9\nC3AG8KLu+fcBk93Y+hZgcZJ9u/1TgW/OoP4jumVt9wX+ATjbsNd8MPA1rk4ATkzyI+BuYH1VnQNQ\nVdcyCP0bGQTqsu743QzWb784yfUMlpvdH9gZ+Fp3bCmDE8AwuM/ymd0J2H3ozgt0byqnAZd2r1kP\nnNe9xuVntcVyeWSNvSSHMbhz1wkbbiwv6YkMfElqhFM6ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAl\nqREGviQ14v8AtcDJOKtv7e0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x22e5dd09c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEPCAYAAABBUX+lAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEVpJREFUeJzt3X2QXXV9x/HPZ5OACQIKSHiSkJBKBwgEBIqlwiJDZXzg\nQQd5KKg4Y9E+QMWhWkbFdAqDTKEFC1OqyPAMSuVxwILFDQKVxIRAJDw0PFlLE56iJIA8JN/+cc+G\nTbKbPTd7f3v33u/7NbOz55z93XO+5/52P/fs75x7riNCAIDu19PuAgAAo4PAB4AkCHwASILAB4Ak\nCHwASILAB4AkRiXwbffYnm/7ltHYHgBgXaN1hH+qpEWjtC0AwCCKB77tHSR9TNL3S28LADC00TjC\n/ydJp0viLb0A0EZFA9/2xyUtjYgFklx9AQDawCXvpWP7bEknSHpb0kRJm0r6cUR8dq12HP0DQJMi\noqmD6KKBv8aG7IMkfTUiDh/kZ3HO/JuHXcfi3y3V9M0n19peM21Lrvv6C76nY079YltrKPlcdNr+\nNdv26Wvv1qFfOq7l6/7kTvtq+mb12p730G3qqflnXWr/xsrfU8n++9pe60TTmGa76cDnOnwASGL8\naG0oImZLmj1a2wMArIkj/MK2nblLu0soqtv3b9o+u7e7hKLYv1wI/MK26/JA7Pb923mfGe0uoSj2\nLxcCHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkC\nHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCS\nIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSIPAB\nIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSIPABIAkCHwCSGF9y5bY3lnSPpI2qbd0QEbNKbhMAMLii\ngR8Rb9g+OCJesz1O0n2274iIOSW3CwBYV/EhnYh4rZrcWI0XmCi9TQDAuooHvu0e2w9KWiLproiY\nW3qbAIB1FR3SkaSIWCVpL9ubSbrJ9q4RsWjtdlefd9Hq6cl77KzJe05fZ12/j9DCN5bX2m4zbUuu\nu1TbCRMmaVXN/5WWv/mqFj6/uF7jJuvotLYln7dm1h0hrar5z25IRWou+fdU6nku2X9jXV9fn/r6\n+ka0DkeM3giL7W9KejUizl9refzFbWcN+/iFSxdrxuR1XwhG2rbkuku1/fxen67d9it3nKMeu1bb\nZuvotLYln7dm1r0ypHE1V91M22ZqLvn3VOp5Ltl/F338jNptxwLbioj6O6jCQzq2t7K9eTU9UdKh\nkh4ruU0AwOBKD+lsK+ly2z1qvLhcHxG3F94mAGAQpS/LXChp75LbAADUwzttASAJAh8AkiDwASAJ\nAh8AkiDwASAJAh8AkiDwASAJAh8AkiDwASAJAh8AkiDwASAJAh8AkiDwASAJAh8AkiDwASAJAh8A\nkiDwASAJAh8AkiDwASAJAh8AkiDwASAJAh8AkiDwASAJAh8AkiDwASAJAh8AkiDwASAJAh8AkiDw\nASAJAh8AkiDwASAJAh8AkiDwASAJAh8AkqgV+G44wfa3qvkdbe9XtjQAQCvVPcK/WNKHJB1XzS+X\ndFGRigAARYyv2e6PImJv2w9KUkQss71RwboAAC1W9wj/LdvjJIUk2X6fpFXFqgIAtFzdwL9Q0o2S\ntrZ9lqR7JZ1drCoAQMvVGtKJiKttz5N0iCRLOjIiHi1aGQCgpdYb+La3GDD7vKRrB/4sIl4uVRgA\noLWGO8Kfp8a4vSXtKGlZNf0eSb+WNLVodQCAllnvGH5ETI2IaZJ+KumTEbFVRGwp6ROS7hyNAgEA\nrVH3pO3+EXF7/0xE3CHpj8uUBAAooe51+M/Z/oakq6r5P5P0XJmSAAAl1D3CP07S+9S4NPNGSVvr\nnXfdAgA6QN3LMl+WdKrtTRuzsaJsWQCAVqt787QZ1W0VfiXpEdvzbO9e43E72L7b9iO2F9o+ZaQF\nAwA2TN0x/EsknRYRP5Mk272S/k3Dn7h9u3rcAtvvljTP9p0R8diGFgwA2DB1x/A36Q97SYqIPkmb\nDPegiFgSEQuq6RWSHpW0/QbUCQAYobpH+E/Z/qakK6v5EyQ91cyGbO8kaaakB5p5HACgNeoG/hck\nzZL042r+nmpZLdVwzg2STh3qhO8DV/109fR2M6ZquxnT1mkTEVq5Kmpt87evvqI5ix+uW6JWxO81\nZ3m99s20HT9xQpGaj9/jcK2seb/SZa8u15NLnqnXWOVqLvUcN9O25PPWzLpDoZXhlrdtpuZS/SyV\ne55L9t9Y19fXp76+vhGto+5VOssknSJJ1W2SN4mIV+o81vZ4NcL+yoi4eah2s6+4a9h1rYzQONf7\nxT/6vC9pXBMf4Dhn8dOaPn2Plrc97aNf1n4771mrbTM1v/nqm/UaSnrl6Ze0Tc+mtduXqrnUc9xM\n25LPWzPrXhXSuHq/yk21babmUv0slXueS/bfWNfb26ve3t7V87NmzWp6HXWv0rnG9ma2N5G0UNIi\n26fX3MYPJC2KiAuarg4A0DJ1X7N3rY7oj5R0hxo3TTtxuAfZPkCNd+V+xPaDtufbPmyDqwUAbLC6\nY/gTbE9QI/D/JSLesj3s4F9E3Cdp3EgKBAC0Rt0j/EskPaPGpZj32J4iqdYYPgBgbKh70vZCNT7m\nsN+ztg8uUxIAoIThPvHqhIi4yvZpQzQ5v0BNAIAChjvC7383bfdc2wQASa038CPikup78xd8AgDG\nlLrX4U+zfavtF2w/b/tm2+u+FRYAMGbVvUrnGkk/lLStpO0k/UjStaWKAgC0Xt3AnxQRV0bE29XX\nVZLeVbIwAEBr1X3j1R22vy7pOkkh6RhJt9veQlr9iVgAgDGsbuB/pvp+8lrLj1XjBYDxfAAY4+q+\n8Wpq6UIAAGWtdwzf9t8OmD56rZ+dXaooAEDrDXfS9tgB03+31s+46yUAdJDhAt9DTA82DwAYw4YL\n/BhierB5AMAYNtxJ2z1tv6LG0fzEalrVPNfhA0AHGe5eOnx4CQB0iSY+lhgA0MkIfABIgsAHgCQI\nfABIgsAHgCQIfABIgsAHgCQIfABIgsAHgCQIfABIgsAHgCQIfABIgsAHgCQIfABIgsAHgCQIfABI\ngsAHgCQIfABIgsAHgCQIfABIgsAHgCQIfABIgsAHgCQIfABIgsAHgCQIfABIgsAHgCQIfABIgsAH\ngCQIfABIomjg277U9lLbD5fcDgBgeKWP8C+T9NHC2wAA1FA08CPiXknLSm4DAFAPY/gAkMT4dhfQ\n76y/n7V6+sMHHaQPH9S7TpuovuqI5W9KPa5fwLI3pOVvtb7t21Gm5lLrLbju6065WPvuMrNW26O/\n/YXO6o9m160mfpebadvlv0NFax7j+vr61NfXN6J1jJnAP+NbZ7a7BAAYs3p7e9Xb27t6ftasWUM3\nHsJoDOm4+gIAtFHpyzKvkXS/pA/Y/rXtk0puDwAwtKJDOhFxfMn1AwDq4yodAEiCwAeAJAh8AEiC\nwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeA\nJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8\nAEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiC\nwAeAJAh8AEiCwAeAJAh8AEiCwAeAJAh8AEiCwC9swdxftruEol56dkm7SyjqxWe6e/9+Pruv3SUU\n1e391ywCv7AFc+e1u4SiXu7ywO/2F7Sfz57d7hKK6vb+axaBDwBJEPgAkIQjot01yHb7iwCADhMR\nbqb9mAh8AEB5DOkAQBIEPgAk0dbAt32Y7cdsP2H7a+2spQTbz9h+yPaDtue0u56Rsn2p7aW2Hx6w\n7L2277T9uO3/sL15O2sciSH270zbv7E9v/o6rJ01bijbO9i+2/YjthfaPqVa3hX9N8j+/XW1vFv6\nb2PbD1RZstD2mdXypvqvbWP4tnskPSHpEEnPSZor6diIeKwtBRVg+ylJH4yIZe2upRVs/4mkFZKu\niIg9qmXfkfRSRJxbvWi/NyK+3s46N9QQ+3empOURcX5bixsh29tI2iYiFth+t6R5ko6QdJK6oP/W\ns3/HqAv6T5JsT4qI12yPk3SfpFMkfVpN9F87j/D3k/TfEfFsRLwl6To1OqibWF00bBYR90pa+8Xr\nCEmXV9OXSzpyVItqoSH2T2r0Y0eLiCURsaCaXiHpUUk7qEv6b4j92776ccf3nyRFxGvV5MaSxksK\nNdl/7Qyj7SX9z4D53+idDuoWIeku23Ntf7HdxRSydUQslRp/dJK2bnM9JfyV7QW2v9+pQx4D2d5J\n0kxJv5A0udv6b8D+PVAt6or+s91j+0FJSyTdFRFz1WT/dc3R5xh1QETsLeljkv6yGjLodt12ne/F\nkqZFxEw1/tA6emigGu64QdKp1ZHw2v3V0f03yP51Tf9FxKqI2EuN/8z2s72bmuy/dgb+/0raccD8\nDtWyrhER/1d9f0HSjWoMY3WbpbYnS6vHUZ9vcz0tFREvxDsnur4nad921jMStserEYZXRsTN1eKu\n6b/B9q+b+q9fRLwiqU/SYWqy/9oZ+HMlTbc9xfZGko6VdEsb62kp25Oqow3Z3kTSn0r6VXuraglr\nzTHRWyR9vpr+nKSb135Ah1lj/6o/on6fUmf34Q8kLYqICwYs66b+W2f/uqX/bG/VPxxle6KkQ9U4\nT9FU/7X1nbbVJVIXqPHCc2lEnNO2YlrM9lQ1jupDjRMsV3f6/tm+RlKvpC0lLZV0pqSbJP1I0vsl\nPSvpMxHx23bVOBJD7N/BaowHr5L0jKST+8dMO4ntAyTdI2mhGr+TIekMSXMk/VAd3n/r2b/j1R39\nN0ONk7I91df1EXGW7S3URP9xawUASIKTtgCQBIEPAEkQ+ACQBIEPAEkQ+ACQBIEPAEkQ+OhItre3\nfVN1a+3Fti+0PaHF2zjC9h8OmJ9l+yMtWO8U268NuGXvxSNdJ1AH1+GjI9l+QNJFEXGFbavxtvkV\nEfE3LdzGZZJui4h/b9U6q/VOkXRr/y2YgdHCET46TnWU/XpEXCFJ1b1SviLpxOqWFp+z/d0B7W+1\nfWA1fajt+23/0vb1tidVy8+pPjxjge1zbX9I0uGSzq2Owqfavsz2p6r2h1TLH6ruwjihWv607W/b\nnlf97AND7Uap5wcYCoGPTrSbGh9wsVpELFfjrfPT+xet/SDbW0r6hqRDImKfah2nVW9PPzIidqvu\nqvgPEfFfatyn5PSI2Dsinh6wno0lXSbp6IjYU9IESV8esKnnI+KDkv5V0ulD7MNO1QvGz5LcRRVj\nAIGPbjLcUfP+knaVdF91X/HPqnHH1t9Jer06Uj9K0uvDrGcXSU9FxJPV/OWSDhzw8xur7/MkTRnk\n8c9J2rG6dfZXJV3Tf6M9oCQCH51okaR9Bi6wvZmkyZIel/S21vzdfld/M0l3Vkfse0XE7hHx5xGx\nUo1bV98g6ROSflKjhvW9uLxRfV+pxo3z1hARb/V/7GVEzJf0pKShhn6AliHw0XEi4j8lTbR9giRV\nn/H5j5K+GxFvqDG0M9MN79c7n0PwC0kH2N65etwk239Q3b76PRHxE0mnSeo/mbpc0maDlPC4pCm2\np1XzJ6pxf/Jaqlvd9lTT09QYhnqq7uOBDUXgo1MdJelo209IelHSyv7bT0fEfWqE/iOS/lnVeH9E\nvKjGvcOvtf2QpPvVGJ7ZVNJt1bJ71DgBLDU+Z/n06gTsVFXnBaoXlZMk3VA9ZqWkS6rH1Lns7UBJ\nD9uer8atbU/uxFsSo/NwWSY6nu39JV0r6aj+D7IGsC4CHwCSYEgHAJIg8AEgCQIfAJIg8AEgCQIf\nAJIg8AEgCQIfAJL4fxwIEMPnZs0EAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x22e5ed3bac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ancr = sess.run([corrbool,locs, total_loss, logits, facts_0s, w_1]+attends+\n",
    "                [query, cs, question_module_outputs],feed_dict=validation_set)\n",
    "a = ancr[0]\n",
    "n = ancr[1]\n",
    "cr = ancr[2]\n",
    "attenders = np.array(ancr[6:-3]) \n",
    "faq = np.sum(ancr[4], axis=(-1,-2)) # Number of facts in each context\n",
    "\n",
    "limit = 5\n",
    "for question in range(min(limit, batch_size)):\n",
    "    plt.yticks(range(passes,0,-1))\n",
    "    plt.ylabel(\"Episode\")\n",
    "    plt.xlabel(\"Question \"+str(question+1))\n",
    "    pltdata = attenders[:,question,:int(faq[question]),0] \n",
    "    # Display only information about facts that actually exist, all others are 0\n",
    "    pltdata = (pltdata - pltdata.mean()) / ((pltdata.max() - pltdata.min() + 0.001)) * 256\n",
    "    plt.pcolor(pltdata, cmap=plt.cm.BuGn, alpha=0.7)\n",
    "    plt.show()\n",
    "\n",
    "#print(list(map((lambda x: x.shape),ancr[3:])), new_ends.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to see what the answers for the above questions were, we can use the location of our distance score in the context as an index and see what word is at that index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TEXT:  mary travelled to the bedroom . mary journeyed to the bathroom . mary got the football there . mary passed the football to fred .\n",
      "QUESTION:  who received the football ?\n",
      "RESPONSE:  mary Incorrect\n",
      "EXPECTED:  fred\n",
      "\n",
      "TEXT:  bill grabbed the apple there . bill got the football there . jeff journeyed to the bathroom . bill handed the apple to jeff . jeff handed the apple to bill . bill handed the apple to jeff . jeff handed the apple to bill . bill handed the apple to jeff .\n",
      "QUESTION:  what did bill give to jeff ?\n",
      "RESPONSE:  apple Correct\n",
      "EXPECTED:  apple\n",
      "\n",
      "TEXT:  bill moved to the bathroom . mary went to the garden . mary picked up the apple there . bill moved to the kitchen . mary left the apple there . jeff got the football there . jeff went back to the kitchen . jeff gave the football to fred .\n",
      "QUESTION:  what did jeff give to fred ?\n",
      "RESPONSE:  apple Incorrect\n",
      "EXPECTED:  football\n",
      "\n",
      "TEXT:  jeff travelled to the bathroom . bill journeyed to the bedroom . jeff journeyed to the hallway . bill took the milk there . bill discarded the milk . mary moved to the bedroom . jeff went back to the bedroom . fred got the football there . bill grabbed the milk there . bill passed the milk to mary . mary gave the milk to bill . bill discarded the milk there . bill went to the kitchen . bill got the apple there .\n",
      "QUESTION:  who gave the milk to bill ?\n",
      "RESPONSE:  jeff Incorrect\n",
      "EXPECTED:  mary\n",
      "\n",
      "TEXT:  fred travelled to the bathroom . jeff went to the bathroom . mary went back to the bathroom . fred went back to the bedroom . fred moved to the office . mary went back to the bedroom . jeff got the milk there . bill journeyed to the garden . mary went back to the kitchen . fred went to the bedroom . mary journeyed to the bedroom . jeff put down the milk there . jeff picked up the milk there . bill went back to the office . mary went to the kitchen . jeff went back to the kitchen . jeff passed the milk to mary . mary gave the milk to jeff . jeff gave the milk to mary . mary got the football there . bill travelled to the bathroom . fred moved to the garden . fred got the apple there . mary handed the football to jeff . fred put down the apple . jeff left the football .\n",
      "QUESTION:  who received the football ?\n",
      "RESPONSE:  mary Incorrect\n",
      "EXPECTED:  jeff\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Locations of responses within contexts\n",
    "indices = np.argmax(n,axis=1)\n",
    "\n",
    "# Locations of actual answers within contexts \n",
    "indicesc = np.argmax(a,axis=1)\n",
    "\n",
    "for i,e,cw, cqa in list(zip(indices, indicesc, val_context_words, val_cqas))[:limit]:\n",
    "    ccc = \" \".join(cw)\n",
    "    print(\"TEXT: \",ccc)\n",
    "    print (\"QUESTION: \", \" \".join(cqa[3]))\n",
    "    print (\"RESPONSE: \", cw[i], [\"Correct\", \"Incorrect\"][i!=e])\n",
    "    print(\"EXPECTED: \", cw[e])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's keep training! In order to get good results, you may have to train for a long period of time (on my home desktop, it took about 12 hours), but you should eventually be able to reach very high accuracies (over 90%). Experienced users of Jupyter Notebook should note that at any time, you can interrupt training and still save the progress the network has made so far, as long as you keep the same `tf.Session`; this is useful if you want to visualize the attention and answers the network is currently giving."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|                                                                                                                                      | 0/3125 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 0.0, Minibatch Loss=  0.672774 Accuracy=  0.4625\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  3%|███▉                                                                                                                      | 100/3125 [06:58<2:53:58,  3.45s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 100.0, Minibatch Loss=  0.672757 Accuracy=  0.552344\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  6%|███████▊                                                                                                                  | 200/3125 [14:05<3:13:51,  3.98s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 200.0, Minibatch Loss=  0.67276 Accuracy=  0.563281\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 10%|███████████▋                                                                                                              | 300/3125 [21:16<2:50:22,  3.62s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 300.0, Minibatch Loss=  0.672724 Accuracy=  0.666406\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 13%|███████████████▌                                                                                                          | 400/3125 [28:43<3:17:32,  4.35s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 400.0, Minibatch Loss=  0.672714 Accuracy=  0.735156\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|███████████████████▌                                                                                                      | 500/3125 [36:04<3:31:08,  4.83s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 500.0, Minibatch Loss=  0.672723 Accuracy=  0.774219\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 19%|███████████████████████▍                                                                                                  | 600/3125 [43:55<3:31:30,  5.03s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 600.0, Minibatch Loss=  0.672722 Accuracy=  0.816406\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 22%|███████████████████████████▎                                                                                              | 700/3125 [51:34<3:11:04,  4.73s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 700.0, Minibatch Loss=  0.67267 Accuracy=  0.854688\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 26%|███████████████████████████████▏                                                                                          | 800/3125 [59:10<2:24:24,  3.73s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 800.0, Minibatch Loss=  0.672644 Accuracy=  0.842969\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 29%|██████████████████████████████████▌                                                                                     | 900/3125 [1:06:42<2:14:59,  3.64s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 900.0, Minibatch Loss=  0.672641 Accuracy=  0.839063\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 32%|██████████████████████████████████████                                                                                 | 1000/3125 [1:14:36<2:54:10,  4.92s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1000.0, Minibatch Loss=  0.672659 Accuracy=  0.866406\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 35%|█████████████████████████████████████████▉                                                                             | 1100/3125 [1:22:04<2:40:32,  4.76s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1100.0, Minibatch Loss=  0.672652 Accuracy=  0.873438\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 38%|█████████████████████████████████████████████▋                                                                         | 1200/3125 [1:29:43<2:23:06,  4.46s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1200.0, Minibatch Loss=  0.672654 Accuracy=  0.835156\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 42%|█████████████████████████████████████████████████▌                                                                     | 1300/3125 [1:37:45<2:15:52,  4.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1300.0, Minibatch Loss=  0.672625 Accuracy=  0.896094\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 45%|█████████████████████████████████████████████████████▎                                                                 | 1400/3125 [1:45:21<1:43:42,  3.61s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1400.0, Minibatch Loss=  0.672668 Accuracy=  0.895312\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 48%|█████████████████████████████████████████████████████████                                                              | 1500/3125 [1:53:24<2:10:53,  4.83s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1500.0, Minibatch Loss=  0.672647 Accuracy=  0.900781\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 51%|████████████████████████████████████████████████████████████▉                                                          | 1600/3125 [2:01:07<2:13:24,  5.25s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1600.0, Minibatch Loss=  0.67266 Accuracy=  0.908594\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 54%|████████████████████████████████████████████████████████████████▋                                                      | 1700/3125 [2:08:49<1:37:01,  4.09s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1700.0, Minibatch Loss=  0.672629 Accuracy=  0.927344\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 58%|████████████████████████████████████████████████████████████████████▌                                                  | 1800/3125 [2:16:30<1:51:59,  5.07s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1800.0, Minibatch Loss=  0.672622 Accuracy=  0.914063\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 61%|████████████████████████████████████████████████████████████████████████▎                                              | 1900/3125 [2:24:07<1:23:12,  4.08s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1900.0, Minibatch Loss=  0.672625 Accuracy=  0.932031\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 64%|████████████████████████████████████████████████████████████████████████████▏                                          | 2000/3125 [2:32:00<1:23:51,  4.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 2000.0, Minibatch Loss=  0.672632 Accuracy=  0.924219\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 67%|███████████████████████████████████████████████████████████████████████████████▉                                       | 2100/3125 [2:39:42<1:20:01,  4.68s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 2100.0, Minibatch Loss=  0.672626 Accuracy=  0.934375\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 70%|███████████████████████████████████████████████████████████████████████████████████▊                                   | 2200/3125 [2:47:07<1:11:49,  4.66s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 2200.0, Minibatch Loss=  0.672638 Accuracy=  0.922656\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 74%|███████████████████████████████████████████████████████████████████████████████████████▌                               | 2300/3125 [2:54:43<1:00:58,  4.43s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 2300.0, Minibatch Loss=  0.672625 Accuracy=  0.914844\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 77%|████████████████████████████████████████████████████████████████████████████████████████████▉                            | 2400/3125 [3:02:21<52:58,  4.38s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 2400.0, Minibatch Loss=  0.672631 Accuracy=  0.9375\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 80%|████████████████████████████████████████████████████████████████████████████████████████████████▊                        | 2500/3125 [3:10:05<42:46,  4.11s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 2500.0, Minibatch Loss=  0.672617 Accuracy=  0.94375\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 83%|████████████████████████████████████████████████████████████████████████████████████████████████████▋                    | 2600/3125 [3:17:30<32:40,  3.73s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 2600.0, Minibatch Loss=  0.67263 Accuracy=  0.940625\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 86%|████████████████████████████████████████████████████████████████████████████████████████████████████████▌                | 2700/3125 [3:25:15<30:29,  4.30s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 2700.0, Minibatch Loss=  0.672621 Accuracy=  0.944531\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 90%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▍            | 2800/3125 [3:32:55<24:21,  4.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 2800.0, Minibatch Loss=  0.672631 Accuracy=  0.93125\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 93%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎        | 2900/3125 [3:40:30<12:20,  3.29s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 2900.0, Minibatch Loss=  0.672638 Accuracy=  0.94375\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏    | 3000/3125 [3:48:00<10:35,  5.08s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 3000.0, Minibatch Loss=  0.67261 Accuracy=  0.941406\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 3100/3125 [3:55:40<02:06,  5.05s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 3100.0, Minibatch Loss=  0.672613 Accuracy=  0.953125\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3125/3125 [3:57:46<00:00,  4.14s/it]\n"
     ]
    }
   ],
   "source": [
    "train(training_iterations_count, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.95\n"
     ]
    }
   ],
   "source": [
    "# Final testing accuracy\n",
    "print(np.mean(sess.run([corrects], feed_dict= prep_batch(final_test_data))[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Once we're done viewing what our model is returning, we can close the session to free up system resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Looking for more? \n",
    "\n",
    "There's a lot still left to be done and experimented with:\n",
    "\n",
    "- *Other tasks in bAbI.* We've only sampled the many tasks that bAbI has to offer. Try changing the preprocessing to fit another task and see how our dynamic memory network performs on it.  You may want to retrain the network before you try running it on the new task, of course. If the task doesn't guarantee  the answer is inside the context, you may want to compare output to a dictionary of words and their corresponding vectors instead. (Those tasks are 6-10 and 17-20.)  I  recommend trying task 1 or 3, which you can do by changing the values of `test_set_file` and `train_set_file`.\n",
    "\n",
    "- *Supervised training.* Our attention mechanism is *unsupervised*, in that we don't specify explicitly which sentences should be attended to and instead let the network figure that out on its own. Try adding loss to the network that encourages the attention mechanism toward attending to the right sentences.\n",
    "\n",
    "- *Coattention.* Instead of attending simply over the input sentences, some researchers have found success in what they call [\"dynamic coattention networks\"](https://arxiv.org/pdf/1611.01604.pdf), which attends over a matrix representing two locations in two sequences simultaneously.\n",
    "\n",
    "- *Alternate vectorization schemes and sources.* Try making more intelligent mappings between sentences and vectors, or maybe use a different data set. GloVe offers larger corpi of up to eight-hundred-forty billion distinct tokens, of three hundred dimensions each.\n",
    "\n",
    "\n",
    "<p><em>This post is a collaboration between O'Reilly and </em><a href=\"https://www.tensorflow.org/\"><em>TensorFlow</em></a><em>. </em><a href=\"http://www.oreilly.com/about/editorial_independence.html\"><em>See our statement of editorial independence</em></a><em>.</em></p>"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}