2.2.2 Supervised Learning - ConvNet HandsOn Part II.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Convolution Nets for MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Deep Learning models can take quite a bit of time to run, particularly if GPU isn't used. \n",
    "\n",
    "In the interest of time, you could sample a subset of observations (e.g. $1000$) that are a particular number of your choice (e.g. $6$) and $1000$ observations that aren't that particular number (i.e. $\\neq 6$). \n",
    "\n",
    "We will build a model using that and see how it performs on the test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Theano backend.\n",
      "Using gpu device 0: GeForce GTX 760 (CNMeM is enabled with initial size: 90.0% of memory, cuDNN 4007)\n"
     ]
    }
   ],
   "source": [
    "#Import the required libraries\n",
    "import numpy as np\n",
    "np.random.seed(1338)\n",
    "\n",
    "from keras.datasets import mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.layers.core import Dense, Dropout, Activation, Flatten"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.layers.convolutional import Convolution2D\n",
    "from keras.layers.pooling import MaxPooling2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.utils import np_utils\n",
    "from keras.optimizers import SGD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Loading Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "path_to_dataset = \"euroscipy_2016_dl-keras/data/mnist.pkl.gz\"\n",
    "\n",
    "#Load the training and testing data\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data(path_to_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "X_test_orig = X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)\n",
    "X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)\n",
    "\n",
    "X_train = X_train.astype('float32')\n",
    "X_test = X_test.astype('float32')\n",
    "\n",
    "X_train /= 255\n",
    "X_test /= 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Seed for reproducibilty\n",
    "np.random.seed(1338)\n",
    "\n",
    "# Test data\n",
    "X_test = X_test.copy()\n",
    "Y = y_test.copy()\n",
    "\n",
    "# Converting the output to binary classification(Six=1,Not Six=0)\n",
    "Y_test = Y == 6\n",
    "Y_test = Y_test.astype(int)\n",
    "\n",
    "# Selecting the 5918 examples where the output is 6\n",
    "X_six = X_train[y_train == 6].copy()\n",
    "Y_six = y_train[y_train == 6].copy()\n",
    "\n",
    "# Selecting the examples where the output is not 6\n",
    "X_not_six = X_train[y_train != 6].copy()\n",
    "Y_not_six = y_train[y_train != 6].copy()\n",
    "\n",
    "# Selecting 6000 random examples from the data that \n",
    "# only contains the data where the output is not 6\n",
    "random_rows = np.random.randint(0,X_six.shape[0],6000)\n",
    "X_not_six = X_not_six[random_rows]\n",
    "Y_not_six = Y_not_six[random_rows]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Appending the data with output as 6 and data with output as <> 6\n",
    "X_train = np.append(X_six,X_not_six)\n",
    "\n",
    "# Reshaping the appended data to appropraite form\n",
    "X_train = X_train.reshape(X_six.shape[0] + X_not_six.shape[0], \n",
    "                          1, img_rows, img_cols)\n",
    "\n",
    "# Appending the labels and converting the labels to \n",
    "# binary classification(Six=1,Not Six=0)\n",
    "Y_labels = np.append(Y_six,Y_not_six)\n",
    "Y_train = Y_labels == 6 \n",
    "Y_train = Y_train.astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(11918, 1, 28, 28) (11918,) (10000, 1, 28, 28) (10000, 2)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape, Y_labels.shape, X_test.shape, Y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Converting the classes to its binary categorical form\n",
    "nb_classes = 2\n",
    "Y_train = np_utils.to_categorical(Y_train, nb_classes)\n",
    "Y_test = np_utils.to_categorical(Y_test, nb_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A simple CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#Initializing the values for the convolution neural network\n",
    "nb_epoch = 2\n",
    "batch_size = 128\n",
    "# number of convolutional filters to use\n",
    "nb_filters = 32\n",
    "# size of pooling area for max pooling\n",
    "nb_pool = 2\n",
    "# convolution kernel size\n",
    "nb_conv = 3\n",
    "\n",
    "sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Step 1: Model Definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
    "                        border_mode='valid',\n",
    "                        input_shape=(1, img_rows, img_cols)))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Step 2: Compile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Step 3: Fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.2890 - acc: 0.9326 - val_loss: 0.1251 - val_acc: 0.9722\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.1341 - acc: 0.9612 - val_loss: 0.1298 - val_acc: 0.9599\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6ccb68f630>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          nb_epoch=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Step 4: Evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.129807630396\n",
      "Test accuracy: 0.9599\n"
     ]
    }
   ],
   "source": [
    "# Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Let's plot our model Predictions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA50AAABVCAYAAADKf4AOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXm4VmW5/z+PgMIJA4UABYckPZoYaJSWeEmWph3MMicQ\nTpAoTimCx711b3NhbGWTECAKKhYdUBQnNAtSK0xIJcejRqeccjghDj9NTAXx+f1x32tP7I17eOf3\n+7mufT3rfdaz1nvfe71reu4pxBgRQgghhBBCCCGywTb5FkAIIYQQQgghROmil04hhBBCCCGEEFkj\nKy+d3bp1WxdCiG3969at27psyJNtpK/0lc7St5iQvtJXOkvfYkL6St9S0rnc9E0J2YjpDCHE9uw3\nhECMMWRcoCwjfVu9XVnp69uWlc7StziQvq3erqz09W3LSmfpWxxI31ZvV1b6+rZFp3O56Zsi91oh\nhBBCCCGEEFlDL51CCCGEEEIIIbJGzl46V6xYwd57781ee+1FbW1trr42b5xyyin07duXL3zhC/kW\nJWeU2zGWvqVNuekL5aez9C1tpG9po+es0j/G0rfEiDFm/M92W8/mzZvjwIED44svvhg3btwYBw8e\nHNeuXRub4ttlRaZs/jXVN8YYH3jggfj444/H/fbbb4t1pahva45xuelbSjpL39LWt7U6S9/i+NM1\nS/qWur4x6jlL16zi1bnc9E3/cmLpXLNmDXvuuSe77bYbXbp04aSTTuLOO+/MxVfnjWHDhrHDDjvk\nW4ycUW7HWPpK31Kj3HSWvtK3lCg3fUHPWaV+jKVv6embk5fOV199lV122aXu84ABA3j11Vdz8dUi\nR5TbMZa+0rfUKDedpa/0LSXKTd9ypNyOsfQtPX1z8tLppuRGhFCU2X5FC5TbMZa+0rfUKDedpa/0\nLSXKTd9ypNyOsfQtPX1z8tI5YMAAXnrppbrPr7zyCjvvvHMuvlrkiHI7xtJX+pYa5aaz9JW+pUS5\n6VuOlNsxlr4lqG8uAmQ/+uijuuDYDz/8MA4ePDj++c9/Lpng2Kb6przwwgtx0KBBza4rNX1bc4zL\nTd9S0ln6lra+rdVZ+hbHn65Z0rfU9U3Rc5auWcWoc7npm/51zsqbbBM6derE3LlzOeKII/j44485\n5ZRT2GeffZodG0KypX25QIgxabWde9SoUaxcuZI333yTXXfdlSlTpjBu3LgtxpWKvq09xuWmL5SG\nztK3tPUFncPSt55S0Fn6lra+oOcsXbPqKQWdy0Hf4G/cGSWEENuzX/NdviTj8mSKlv6R0rd1lJu+\nvi3lpLP0LSykryF9W0+56Sx9Cwvpa0jf1lOMOpebvik5iekUQgghhBBCCFGe6KVTCCGEEEIIIUTW\nyElMp2g7J8TPArALLwNwxcsXA5DsWj/mG/EAAI56bwUAG7pflUMJhRBCCCGKld0BiBMsDrT2Guut\neBzC/oXrvihEsSJLpxBCCCGEEEKIrCFLZ4ERX5sCQG2TMNxab7s16FsdHgPg3aF9AAikQclJ1uQr\nCPolAExdZ/+ke+NyAO4PD+VLogxhlusH4pS6Y1txrq0Js8vk2AohSoyjYdgXm1+16mEAkni7tc/4\nne50YFWSdclEC5yfWHvFTGv3nsTotdcBsGj2aQCMOffaRpss/q9TfZskBwJ2lN0BeCBeCcBMf97q\n4mvPH/Jj4KOcSyVEtqiOG+kULgcgmW994fTcW/OzYuns2rXrayEE2vq33Xa9siFO1pG+0lc6S99i\nQvpKX+ksfYsJ6St9S0nnctM3JSslUxp9QQHXkmkrram10xF942tTqO3b/LqKI6y95TcjOP6JuwGo\n3b/xmL/HGQDMC/9srwiN5cmyvu3mWZudmfE5mzOZ/JCLcFDSod22tpZS1nTunQAw+416Md739s74\nWwAeDH/I6Ffm9Bgfmdh3PmNfGV5uxyzb/MSsIEB7rL4F+5u+OwHgkhEm3j11x/sJoP3nc270HW/f\nNXcXAGafZZaQicEDpNptnf+GNXcPs3bEPO9/rcUtCvb4ZomC1PfWBIBHv/d5AA44aC21Dzc/tOJQ\na2++39oXG6yrbKEkQEHqnEVyo+9w+65VXwPgNj/l0gqBrwJrPmEPe3v7BWCvJ1ycIUmbJcmFviOj\nJccYGk4BYNJS3++vfbcLk47svk3o97wl2dP3aGummefF9RWjAHgpLAHqPQkrXoOf9RkJwCnhp96b\n3n/aTiEc4zh/CsnpjfumpF+ZJJn9LpVMEUIIIYQQQgiRDxTTWQh8JwFgXt8pdV0V37U23GEzEZX3\nLPeOh4FvAxDHmkWhdqGt+ip/tP0wKKvi5psrB1rsSJ2WHbRw5p/vAxB/aJNDc0o0ad5py2cD8GYH\nrjq/nfBVvnb8gwBs06sU/lEnALDxZDv2Nd57RPg6AA9yEx2xdGafo/krZuGcd7b1nHu8xXpNZKd2\n7tMsnHHQIQCsHGG9X+vts7JvJO3cb7YYDsDKWAHAobPXECZe7etatsoWPd0TAJa+a9aD7YJ54Dx7\nnN3H7vVh9zbdrgG192dJNtEmHowTAahpYp/4i7dVR8Dh3/YPQ5ts/IE1NcPrt/n9ENvR1w7yc/ah\nJHPCZoCjuQuAf/jnMX7N4oQkL/KIbNKXU+JmAOb1tB/vlZW25m1vUwtnGtM7sy+AWT+fi78HYGCY\nkH1Rs0hTKydA9SUXATA12TZncsjSKYQQQgghhBAiaxSWpfOJBID4vM2S/f27n+Ef7AzAV857wsYs\n9LFvJzkVLatYSBSbljW0cL7jK5Mthp8R3wVgYZNZyTE1t7a4TUkwNAFg12CKd/Y6pbQqErPwOCza\nrNtvF+wBwJxTWx77x/Vm+RqeZuq98EhbMS3JmnyZY0cArpluqXjf7MCeTuIm1o/fzT/58eexDuwx\nz6yyuLeaYY27KzdYO6X7szkWqLWcDED8+V7UWok7DoimRHArbXtZ6Bkl0wze18anbSEkHdpvpuny\nxiQANu7fA6iX13Joe7AiS3MtVu4w4xjPu4WzLaQ5Cvh5y2Mq+7ddpJzRM4Hv2OK1Px8DwKnnL7aO\n7azpdekrALzVr38BWucbMCihV5jSqKvKEuITzjRLZXUS4Z4VvrZpgK7ZiaqjPbPsEC5npa+J/+3x\n+3s94j2/zJzcHeDz/BmAzf558VHpzTfJhzj54xsJALE6UDvcutI8Eg3jGwHCv7vV+u27YJmbvb9z\ns49am1UxO8Kw+Dk+H+yCc2ULYyosMTMzm3kGWxbWZUewAmBqyJ2FM0WWTiGEEEIIIYQQWaOgLJ3x\neI9r+lva87r/wVQ3Zx3sa4b/qJ1f4m7Z4SyftVmWtHNHGWREAsC5RM6943+8c2aLw68+YTJQX7uz\nbFhgzVNDrK2++FFfkeRDmg5zRLBYqDmtGDvHsxofy1EArPSA1jAi1mU+LVjmngPAzLPN0tkpWixy\neyzUX2U1M+9IP+3nbTFaOncHIC6yf0LSZG23ZQVel3X+ngB1Vk6AI8J/+9Iv2r/fQQmvuCdD1V+t\nqzL8X/v3lxUslviZ3jZnm8bhNryZvr9hXwC6dX/Se27PjWgZxyzatdGuUhW/uBLGJrbKDQCeY5gD\nzKGBx97yz+7SMGnHy5h11YX24Wy7uVfe45aR/kVy7rqXzZo/2TWnd5jCjQtt1Xpva5psMumyAYD5\neZzhSTMfumswAF8JZ/moV7MhbdsYCkvdmSD9DX/1Ncue3TrvArONpVaTWfE0+gaLkazZy4fc6nVa\njysES+cFrHDLboUftDFVSf7EySn+IDHrDAA2XmLX2pnD6+MZUxrHN8Ir0cp09F/4FrVu5U/etpP8\ng54FaOkclACwJIQW/U0ejgsBCOFfAFwZ7ZlyY7g+y8KVLwX10hkm2IPWMZMtgPfOX4zkmO/b8s/4\nAQA79LKo9ZpLbZuqnaGmhWcSvwfy/U9Z+9P3AN9uUTwOgDGhkJLuJJ84YmTclcVNHtYr/AW8svq2\nzItUQMSfu+KftaZ66l35E6YDxN/YDS/55iePPcHbz9ozPrU+IVPrDwnTng4tlhYoCIYmPHO2HbfP\nH2hdIfgdaws3rU9m2VWjtjIdU0QcNxaA5JpxjbrTaxajk1xK0wYsFuDRCeYWfO/pMCP+3VaFn7V/\nt/6A8MbagNetpluf1BG7NdMyuWNZNNf227cyaTK7u7WPYC8Zl8dFANwWUue1AnjZ2Crmuh7H2ltD\n6jrMigYOagsSAIbiEyRvJb7C217ptOj7FOzkSUt4crpHH7Tf+b/8JeWeBsf8Ij9ZP3zJ2qs/ZQ/y\nnYOVVjjnAuuvmQ7/+p3vtr9NQgyO9vL6ZCiA38GA+sXjYz8AKjtQmmti2Ino9VNqPBPRX79nycb2\n4pR27zdTPBAPYXUmQnLGJ9ZObNCXzsCsSzLwBVlgtP1GZ0y0f0B6Np+3I2yz1M/jvzTeZI+zngHg\nv17bHYCnxnUn9Xz/YMGOFBx+L4kLTMfag+pfoCf57zJ0dl1Dw2sU/DD45MhJC1h3k21/g4+I37Vr\nQLijgJ+3tsLmeCGdwuWN+qrjRiC3brZyrxVCCCGEEEIIkTUKytLJ+QkAd56fdiTcOdaWelHhfY2L\nu1ZXroQrhje/vxetOXPuCwA8wx6knnlX8kNf+n0Hhc4R7gb8rRDq5sjHd7I2fFzgrngdJgGgZrbN\nNFWlPtYvFIlrVkOqEzjX9EgD9bs1M2wfT8qyx68esI5VvvnlluL6Mw1mrA6INr36WLgv09J2mI9/\nE7jBvHLYdxf/nT6ctGNPxwKQnD2l2f9XsXHlLWYxfKPJjPs57tV2bsdy8WSNN6OZ3OcE8zhJJkBl\neKLD+x38lKXfmR9gpFtbqkNhWTiNhB6h8UGrGukLx1tTe2z9uvSMvKazJZu5jTe8p6WUFvnGbBjx\nT8cAUPsl6w3R6+GE5rxpkk/4XFw8Gpeyg1s2b2xyflbdaO0tI0ewTTjRPjRJ9hXdVB+m2fUu/ihw\ns3tYnegJekaGrwDwJM/7Vh1wSe8gsWeocw0+htR76Fcd2ueqtWmSP7tH7/nNVzq0v0wyrP9jrPbl\nPhe5l0bVJ3tpPBiXAXBQX7NWL19gv5G1C+rHpOnDlsQfAzAjfNRheTOCW/9edje51N10knv9h2tj\nXVKhpjx/tpns41nmMjgT2Da6xbrAkrsBdH/IQvJq3dvkfeo9iI5bax4nLcvtWZNuSrg9Wim7zsHO\nzcRfHp6IdhEYEkZlTugc0NTKmS9k6RRCCCGEEEIIkTUKy9LZKjwQvbJBQPr5K7e+ySOJbTkUqnw2\nujrskHHJskn1MWbhahgBsuojzztfgLNNmWTHj1zr9NeaFqle3dzoQiUB4I3LA3M2Nz/iHA8vvu6p\n0RxRF/eYNBozddplALyznc1aLfwQpoZDALg62vzl3SG1puSxML2XP7qv1xRGu3PCmFuTdu/uxmiB\nyy8FqPAyC5XjVmxli8Lm7EstUUHin9MYmfDjwvZa2OwnYZ21eRxwTXvi0vYB4LJoFs7/cMvPcgq8\nCPd9sNrjtqpmWRsmeozPks8BMCAeyMv7WyxkjRuBr/RzPk7ubdvMKLQEQxZwPSPafTWN4TzZ1+6S\nWvUoPG+KjmPH7YL4HAD3hrWk9qnve3tNtERIIRxkHaMeA1ooZ/Qbb93SEnaOdYkQa17InNQdpmti\n7cIpdaGI1We4abuDls5DnrSELFMLqp6ZxdH+6x2ocItzZYtx6DvCSZYA771fevyjq5LeddLfxlFP\nAZ6fIfH8Ipf3vBiAGRTG9fzj+0346e51VGGXXUtECFsv6TP/3wCYeXp91w8vTs27W9kuT7yOHdyr\nG/Sd6pfZ00LrS5CdGXYHIKYeLLdYO3jc35odL1qHLJ1CCCGEEEIIIbJGEVo624LFTb081GZ5FgCh\nwmd2bknyI1IbiY9b3MDNPsv2InBS/AwAu4c0bVrbs4AWE7M7WakNj74g7FsYs4dtwgwc3PDGlqvO\nScMMxniNiHAD8FQLO0oA+PYHywE4NhxFWmTn8GC5bu/u7v+fDUlHJO4QcSdPxQ5U37XSOkN74qcT\nAA7zOLrbgfBaER7/hpyekFzSuBj7AenCyiTX0nSI2oMgzj/TPnisdVgdWxw/Y4KNnXSp6Z9a1Jb7\n+orXoLJvNiTNEAPqb5phSNPfoc2ivxKerbN2dW6aoXrPdKEAspY2oOvbZuncFL4FQIVnCA8vfOwj\npjSzVYmwcDQAw/0a8xhwQQ9bte1RaZbLxAc3l0fg095OAuDE2+2fF/vb/u49Hdb4iE3enhXNDFod\n8uetccH7dkxrAlR5Bl52TfImT7Y5LG4HWNz4lzekMadNR1n8YnWcyg7+e0izaV/ggYFz3rSSX3sE\nX7Nf/bkRH7LlmW4QZ6y3Czssfrt5Oi5iuuuZ3mfC2FZYOL28SvR/UprnteI6mHzq1rbLF/b80/Wg\nLa9V3Q5rfyb0E5cuBOCAMLadcomGyNIphBBCCCGEECJrlLSlM4mWIfAXPstzNDBlfP7kaRs2a/rC\n/jZr86L3ngzssv16/5TkWKY8MD7hCz7jONqPXfWIxfmTJ4NUTLY2jGm75e7+YLUCV14AtdMzK1fH\n8Oig/etnG9e0y8JpzIoW33eDn8OTjoYzK5N2768QOGLenfXT58790eqnFVQIVDP0OfpdADb2cEv2\nO1DrsT5pHNy0rSjxvo+tbdJfMcTaMDBSyNe1uCnUyf70oQMBGMSYLca90EIN3rkTPOvj6YXlnXJv\nj8OBBmHyv/Z2nxK2cKZ47OX2Dbq6uFXr0CVmifzPJeaF8oOHl9QPcmsw/puec4dd0E/2p6qaZmL3\n/Synz07v+lLSXqk7TO3X7bsfBsLMIvceaQVprffbgFoqvfeZRmMeiJYfYXWot2hP8lSv4dKmVu8t\nz40lBx7jS3d2XOAMse+Y5+vCjA/czhf+kmxlC7NwXh3tGjXTL+dprcvwaoFeo+dbTd2GsacAFXOh\nsmcGg6nvTxcmeVsSlcNzhiydQgghhBBCCCGyRmlaOs9OANjZLWSpXXBoZYRpSV5EaivxRxZUUntp\n4/4H4jEln622IROvu5zbPVHawLk+R7Kg9RnICo33GyyHGZ71siOzhh823ifAO29vC0CPzhe1f7/t\nZncAZnoWv/EfdWFyB64y566/1vbnnzctAnq2f3+FwC2bj6/T5yhvDwpX+VKBW5buTgDYlkfs89Qv\ncm2VWfrWB/NAGOtD74qjt9j8tP0tgHnaE42toese9yC6Ar+2hS6xzpK77yivsbh7Yu3d1ry/e+Ba\nt56lxzfNtX5YsKzF7O1Wk61aHHJH/9A4VvFmSy7MhdFiOi8/3G9E9yU5lCpHHHczAMM8jP6t/aDW\nDSPDgx3Bf/jQtJ7l9sC7NM/8JhbObsAkj5kMu75jC+uSjsmcQQ68CLgsybcYWWe3O15veWW/BIBP\nh/rr71fjYABCOMt7klZ/1yRLQ8Hk2de1QcJ8kwBwfbT6k2+HJY3WnufW/8lJYXlppJwxwe+qTSyd\nYV1mLbMzU6PpXI/lPjtju845U8O2Of/OknzpnHGlJatYP9c+pwWdq0fNy5NEbWBFAsDCIxs/fKZl\nIkKYQKknDmrIT1++qO5G3/2R9G6e5Ema9rPo9eMAeCtD7pOHRnP7mhPqy1ekbY+eG30pycyXtQlz\n0JuUVr0YuQk41j+0pTyE+VLX9m18Hhza47fAHzoiYP6YmAAwr3O9Tgfu4gsvF/jL5hb4a1T1Lzmt\neqD32SRKdTqkud96V2tSV9yqsT40PO49v8ikkJln7/+pKy9R689kaVmIzl7yaDZQkRZd/4G9XMZ9\nPRmH+6+uWWvlG74cvpdtiVtF+l9PryEvetsjWJH72Vi7TxzGf7xp5TTG9bKb0rXP+BP2cU12equ3\ngyKFPZmyFoCwXzoJeBRXR3tWqJju/xmvTrZiyKEA9GE9B1TYdjWfEN7w5XgAIaQFHArBFc/TybyV\nXylyjpcv2gR0qrsCGff8w9xql/s169TYlV51Zctan/Tr8/zZFnZMe/KfMOy9BdvAYps8uvlD64uj\ntzwfN/3SkyA1cadN+b83XamwnEJkd+xtcFPTFVOTjOy/6eS+aB9yrxVCCCGEEEIIkTVKzNJpPkGT\nTrUZm4XeG/5QLEHyJxB/1zg9dcqSsR6gPq5crJw/BOC1XadQZZPLVA9L8idOBxl90G1AexJ2p3gp\n6mrLXnFbZ/ud3NBgxFhPEnDuhnZ/SQaw2X/cG29mX4gXuZvSqJZLaST7VgBw3uafAvDpGT7rWtF4\n3GY6ZVDWHDPCmvdn1Xc989IetlDgCYQyxSvvW3Xyxa5vGOq/iYVJfgRqM7czdJbJ/MBEU+K+JiMG\nx8MIYZx/SgCYvcrKLHwUzF38S3s8bau7P5XX0kYpn41WbmBdWNrs+nSW/7Gwih/To9G6aUzccgMA\nt/yOBfp4xYJtel3S/NiCYnldYfgz3XpPxZaj1scZtjC98QXXKwcxNtq1sDr8mvqiQPlnQLTnpBo/\nB6u+lfnvWD/Y0jJdm3b0aHFo7vBbRxdgc5NH38NfWAXUFyrbcfEHbdy5lVpZ4e65g9MSNAXwc+/e\n9UbiqpMAqB1mfTO3koux8lb3zjjEk8V5CasB49KyI0kWpOw4F9xlro3Z8iVIvUCaWoBF25ClUwgh\nhBBCCCFE1igpS+cF8S9Agxk8zxM97ptJfgRqI7Pi7Lpi6SkVlqndYzmhXOI5T4kWMLUgwOiVn7HO\nMrEGNceyaNHxfw/3AI0tnBV7WxtGuNXoiiR3grVAONhnS18N3Nbf+mZc1vIBPNnbTT4bPaeZcgMA\na8K6DEmYe94ZbkH7M6lPMDPo+Od8KcmDRDnk6QSAhW4JSK1BafKdosJjcw9xi+dz5+4EwKMMBeDo\ncCZNr9MTw2EAxKfM9lNrIZ38I/Zkp9CCpTCHjA0H2oIn1Ykv+rnqeW/m/M3a9sQ1LQS6mIGbidHc\nMWaFD9slZ6EwOvbn1tC8S8nX11n/x+EnuRSpcBiR8FJoHDMYTvR70y1J7uVpBWEPK53yRqd9AVg8\nBs6Ilihmnpcng+a9AADiBZ8C4C6P7w0/KiTvurWEYW5yTRIATrikcez80pu/DyfZupXfs2tB+ixa\n4Z5mkxdmsOxIEfPldKEY710FgCydQgghhBBCCCGyRmlYOm9NAOjvJVI8NxvhlkKabfpkeoRraRpJ\nEPbyqeaCyHqXO8ZjdVJ+C6zhwPwKk0fib3zG2EsYNBsTOt/b4Un2BWotzyYAhP6XwDdsmStaHj55\niI9xC+c/ogU+3hjeaTJybYYEzCU2y9woa+3RvuDXrlLnuX3NGpgm4f/SBz7f2TXJizwZwS2eAydO\naLKiOW8Uz5C6vWWWnoZZvfv1fYf2ZXfONP+0xn+PoWkwWuo90Rvinz3vwCdkbW1ImlHyp09bGadZ\nhRDs1g72iMcD8J9hEGuarKvyc7q63y25FapQGJEAsOHWTszyLNVVnpyi+rj2ZzPoOBZvyeKtZVA2\nK2bvzXa/iWf1YH2YbMuzrA0T7/GxnoJ6WWLr/xi4zs+F/42W5blgSz+5pXNpssWKuqWHgv2y09jF\n11d6/adQ4JnFs8TNx44F6vOsXBYtP0fBHuMWSOZD0qScTHW0+1EuS6fI0imEEEIIIYQQImuUgKXz\nBGK1zbym9RyHpqEW3ZN8CJRZBnkB2qdP3sqgNH/ia956urG6inJA7z0BeOD1Lza7h/f5N46oiy16\nqtkxueSLPZ8EzNJ5Qm1aWj3Jlzgdxy14jWKiFieNhmw8y7PFNTDuJd+0thstE4YXuNUgLSg/pPWb\nzOEcAHp7fcA6hibwSJIJqXLHX+y4vr93fdf0u7yidJnEKffobLG4HspI9+8Ub83dDrG7eaykGaZr\nu8MF8Q4Aphfyb+H8pG5xfLwSgD2nW4bxtCThkGjpar98tN0/3llms+fzOm9ROa/48Hi3692banWD\nVWmMV9itODyrXqm1Z4GDP2Fc67FnlI2eknp6V7goLen4VCH8T9wM6aaqTcfCL1dbtuZQV0w2feax\n8zNcdUmdJXPy6MZ7eyKeBdRnqr2LegvnjNC4/mdxkXhreqW/6z5HvdtkfWESfmq/tXU0zuz/XLyG\ngaGpN0rr+eevrR3rnyvDv9q9L1HUL512oYtn7UvNVdZT5Q+1oV8hXOgyw7SnP/lJ5ALPZH3Hjpae\n5FvvWXr22d0bDHrDmtVb2d3Z0fyD5ubz4ccfbuZdUcjFxNvO0X8y953Dwwl1fdNG2z86faGc521z\nL5gtvXTuE4eV5ItLWrx7i8fVYnvhBA7491WNPp8M7BXO8U83bDG+1Bgcj2S+P6CNTztXJPkSJ8/Y\ntFM3T/p1K4FnUzet9L61Lsm9WG3g+qNswmSal7V6y/t/F6wMTFzkk2fNPF38btBXsi5fNrhsyXkA\nrL6pvi+dQDki8ePmrosFT2UCwPDD/PMjUJ/KrRXXo4Ns+zUP2n/gSxPtuNf0rB+yzZn+P5matF/O\nDBOO/RiAePw2daVDYs1xtu7rLu+C+vGXHWPH/ML/slCPmT63uyJNsOP/v3BpLDpXy+b4a9wFgNtd\nvy985A62nZN8iNN2ViYAfG7DegCqu/cB4OmwDo60dW257zwdFwEwz/8f6XM2vZ7toKA5xq9LyemF\n8Uwt91ohhBBCCCGEEFmjeC2d/SYBUHPV5LquMNVnqzygvdgYuxZq92n7dtN7pUtu4WywLnV9amop\nG+2euKP6XF/Xt+TUH/hS0nYhMsSNP/kOAM970pmq8VBdWfxJlO4OloDihu1gYTuqBYy1SgN82pMG\nhR/6bz3cRiG4Q2eatHh3KRRifvSJQ4D6s2rP8cCC+1oaXnI8MeordUkY/hkH2EKddT518jvc2yRX\nYuUXn5VfEhexfxgDWEISgLBHak4r0KRZK6ysyqS37ePMno1X145p/PnTwBl+iQqh0nsfy5p4mcV+\nn73DEQCsb7BmJ3cnLnYrV809ECfsZR9GfvL4vw03i8lSP4fT9DqjvP0w7kZ1uC2jMmYGkzvc8g7x\n4B4AzKyyNTOqGrsLbQK6uNWz6dPHMXE3209wv8vfJVmQNccMSnitSZmbvhvS8mT5TALVdjb0thJ7\nZ7ildl5QoWDYAAAGLUlEQVTnTYxYbsm97m6tV9i0hJB653gZt20OLk4PytjPlE7yK0YdsnQKIYQQ\nQgghhMgaRWjpTACIXeqTBz3uvteEmuY3KRLCPpcwOvYHoCf/r9kxP3mvonGsZgNGxD0AmE990HTl\nnp4u3EtYpIxJcw3xUoPexmNyi8WHjPRZ1PRITrhuFixo/n9RXCQA9DgvsvRyi519PrS+uvCRH/wW\ngAfDHxrtr1TZmf8DYKN//lz+ROkAfpJVNeneBeqTfpUX/3Kfiy5vmKfKxiqzOKybb+1OdcnMyoPb\nwsHcOtSWa/bwzoNOtPahJB8itQKzwG7b0ywhcWU/AN78uq1d4DmiKjxx1sVrLyTUpeQvFgunlena\n2MMsnNObVG6qGgkhVPinIovxcsKlZrmJewRqrvHOa1oe35TUe+q73tZ4gqnrwxsUtvfNTMLqhwAY\nFM3ucr7X8xoYLPfCi/F7bKZTo63GftYs/JPrLNtLsy9qjuj+0Ous9ufK1Lvogxd3bHF8QfNBAkCP\nzvb0EPcMHOy5NL4RTwNgYtip8Tb9bJsuT1vpqP/tHbjDV23/kWeSKjaPhq3Eck5J49DzoJMsnUII\nIYQQQgghskbRWTpnxDMBqGngm33bd9Oc1knO5ck0i8OrW10/dytFtSvr/icNUzonHRUpR3i0zKHW\nVFnCOEIYAC1YfYuSaQknTPOyNQutFEw80A7cHI/nPeevEP7gM1HpGVp3bFNLZ2lzfLBY4zRh3Hlx\nmS2Ex/MjULuwTKV//5XFmBBeB2Ddj3qwldO4pLk7/A2AH2OWzSejlW8YEv7qI5I8SJVPfkF4xM71\nqX6Sb1hpFpbuXS/Om1Stw/JtpyWbum+wUhJ3f2qE9adxfWHBlpsWOissE3zNkfaxaWx5GBxhSZJT\nkTLOsASAwHvE734KgJo7tjLeOSrajeoH/AyAyWGFrQhvZFzE7GH5L9LiAGNJk2n4hbnZ2L8kyzLl\nj69/6r663/gkSz/B5CFJvsTJEAkA4W/vEP9k9xvCtQBEf2XYb9EaAJZ7KaSlva3/DmCSP2qE7Rc1\n2l+xsjleyNTU4ySPmbZl6RRCCCGEEEIIkTWKx9Lp9Rv38xmJNXkURWQDi3ELl/pM46VpfyHHhnSQ\nsQkAoYnZ69y9oNhn1TpKXy9t2W9V/mIPOo7Fh+weXgTguTgQgGO5gXKxWAOE/pF4nV23Lx5/IQBT\nw7kAVIa7fFSSB8kKhQSAKg8Ru6ur1RNkb+vnL0muBWoXG7pbwezhHOU9RWjhdOL19TkjGtInFml8\n11aZTrij9a4X1XVWwBVZkUbknmVjRtVl6T1x6UJbCC/kS5wMM5PwpQds8SQr0BpPtx/xuPBlAJb7\neb29b3HayEVM3j/N2pvkSM4M49bMKVu13uceWTqFEEIIIYQQQmSNorF0nvYTqz655orG/VVDoPqJ\nPAgkhMgaYVg6857kU4wMMR2AgSHNKl0+Vk4Arkjqrfmnpp3z8iVNwRJO+D0A8bNfsw6Pa+fI/MhT\nzkRLFs4m/zze213CVb5U/LWjhUh5b8E2sPjjfIuRRbwu9k3WhpuaWPa3sAIm2RaobCmal86mVKVe\nLoufRD8QIYQQxc1KAMIL/kB0ZJI3Scqdb79pvs5f9FILI+M9tiLoZVOUHv0+Wse7F/QBIFz8fe9N\n8iaPKF3kXiuEEEIIIYQQImsUjaXz2pCWzbBZ4OrF6Zrb8yGOEEIIIUqQu8Mz1tYl4VidR2mEyC4b\nul/VIKFhkk9RRIkjS6cQQgghhBBCiKwRYoz5lkEIIYQQQgghRIkiS6cQQgghhBBCiKyhl04hhBBC\nCCGEEFlDL51CCCGEEEIIIbKGXjqFEEIIIYQQQmQNvXQKIYQQQgghhMgaeukUQgghhBBCCJE19NIp\nhBBCCCGEECJr6KVTCCGEEEIIIUTW0EunEEIIIYQQQoisoZdOIYQQQgghhBBZQy+dQgghhBBCCCGy\nhl46hRBCCCGEEEJkDb10CiGEEEIIIYTIGnrpFEIIIYQQQgiRNfTSKYQQQgghhBAia+ilUwghhBBC\nCCFE1tBLpxBCCCGEEEKIrKGXTiGEEEIIIYQQWUMvnUIIIYQQQgghsoZeOoUQQgghhBBCZI3/D4vq\nWbumY8f6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f6cab246898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "slice = 15\n",
    "predicted = model.predict(X_test[:slice]).argmax(-1)\n",
    "\n",
    "plt.figure(figsize=(16,8))\n",
    "for i in range(slice):\n",
    "    plt.subplot(1, slice, i+1)\n",
    "    plt.imshow(X_test_orig[i], interpolation='nearest')\n",
    "    plt.text(0, 0, predicted[i], color='black', \n",
    "             bbox=dict(facecolor='white', alpha=1))\n",
    "    plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding more Dense Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
    "                        border_mode='valid',\n",
    "                        input_shape=(1, img_rows, img_cols)))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.3044 - acc: 0.9379 - val_loss: 0.1469 - val_acc: 0.9625\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.1189 - acc: 0.9640 - val_loss: 0.1058 - val_acc: 0.9655\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6cf59f7358>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          nb_epoch=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.105762729073\n",
      "Test accuracy: 0.9655\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
    "                        border_mode='valid',\n",
    "                        input_shape=(1, img_rows, img_cols)))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.3128 - acc: 0.9097 - val_loss: 0.1438 - val_acc: 0.9624\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.1362 - acc: 0.9580 - val_loss: 0.1145 - val_acc: 0.9628\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6ccb180208>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          nb_epoch=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.11448907243\n",
      "Test accuracy: 0.9628\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding more Convolution Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
    "                        border_mode='valid',\n",
    "                        input_shape=(1, img_rows, img_cols)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Convolution2D(nb_filters, nb_conv, nb_conv))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n",
    "model.add(Dropout(0.25))\n",
    "    \n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.4707 - acc: 0.8288 - val_loss: 0.2307 - val_acc: 0.9399\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.1882 - acc: 0.9383 - val_loss: 0.1195 - val_acc: 0.9621\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6cc97b8748>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          nb_epoch=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.11954063682\n",
      "Test accuracy: 0.9621\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise\n",
    "\n",
    "The above code has been written as a function. \n",
    "\n",
    "Change some of the **hyperparameters** and see what happens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Function for constructing the convolution neural network\n",
    "# Feel free to add parameters, if you want\n",
    "\n",
    "def build_model():\n",
    "    \"\"\"\"\"\"\n",
    "    model = Sequential()\n",
    "    model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
    "                        border_mode='valid',\n",
    "                        input_shape=(1, img_rows, img_cols)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Convolution2D(nb_filters, nb_conv, nb_conv))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n",
    "    model.add(Dropout(0.25))\n",
    "    \n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(nb_classes))\n",
    "    model.add(Activation('softmax'))\n",
    "    \n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "    model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "              nb_epoch=nb_epoch,verbose=1,\n",
    "              validation_data=(X_test, Y_test))\n",
    "          \n",
    "\n",
    "    #Evaluating the model on the test data    \n",
    "    score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "    print('Test score:', score)\n",
    "    print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.5634 - acc: 0.7860 - val_loss: 0.3574 - val_acc: 0.9363\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.2372 - acc: 0.9292 - val_loss: 0.2253 - val_acc: 0.9190\n",
      "Test score: 0.225333989978\n",
      "Test accuracy: 0.919\n",
      "1 loop, best of 1: 5.45 s per loop\n"
     ]
    }
   ],
   "source": [
    "#Timing how long it takes to build the model and test it.\n",
    "%timeit -n1 -r1 build_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Batch Normalisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## How to BatchNorm in Keras\n",
    "\n",
    "```python\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "\n",
    "BatchNormalization(epsilon=1e-06, mode=0, \n",
    "                   axis=-1, momentum=0.99, \n",
    "                   weights=None, beta_init='zero', \n",
    "                   gamma_init='one')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Try to add a new BatchNormalization layer to the Model \n",
    "# (after the Dropout layer)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}