2. Define the Network Architecture.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the Convolutional Neural Network\n",
    "\n",
    "After you've looked at the data you're working with and, in this case, know the shapes of the images and of the keypoints, you are ready to define a convolutional neural network that can *learn* from this data.\n",
    "\n",
    "In this notebook and in `models.py`, you will:\n",
    "1. Define a CNN with images as input and keypoints as output\n",
    "2. Construct the transformed FaceKeypointsDataset, just as before\n",
    "3. Train the CNN on the training data, tracking loss\n",
    "4. See how the trained model performs on test data\n",
    "5. If necessary, modify the CNN structure and model hyperparameters, so that it performs *well* **\\***\n",
    "\n",
    "**\\*** What does *well* mean?\n",
    "\n",
    "\"Well\" means that the model's loss decreases during training **and**, when applied to test image data, the model produces keypoints that closely match the true keypoints of each face. And you'll see examples of this later in the notebook.\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNN Architecture\n",
    "\n",
    "Recall that CNN's are defined by a few types of layers:\n",
    "* Convolutional layers\n",
    "* Maxpooling layers\n",
    "* Fully-connected layers\n",
    "\n",
    "You are required to use the above layers and encouraged to add multiple convolutional layers and things like dropout layers that may prevent overfitting. You are also encouraged to look at literature on keypoint detection, such as [this paper](https://arxiv.org/pdf/1710.00977.pdf), to help you determine the structure of your network.\n",
    "\n",
    "\n",
    "### TODO: Define your model in the provided file `models.py` file\n",
    "\n",
    "This file is mostly empty but contains the expected name and some TODO's for creating your model.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyTorch Neural Nets\n",
    "\n",
    "To define a neural network in PyTorch, you define the layers of a model in the function `__init__` and define the feedforward behavior of a network that employs those initialized layers in the function `forward`, which takes in an input image tensor, `x`. The structure of this Net class is shown below and left for you to fill in.\n",
    "\n",
    "Note: During training, PyTorch will be able to perform backpropagation by keeping track of the network's feedforward behavior and using autograd to calculate the update to the weights in the network.\n",
    "\n",
    "#### Define the Layers in ` __init__`\n",
    "As a reminder, a conv/pool layer may be defined like this (in `__init__`):\n",
    "```\n",
    "# 1 input image channel (for grayscale images), 32 output channels/feature maps, 3x3 square convolution kernel\n",
    "self.conv1 = nn.Conv2d(1, 32, 3)\n",
    "\n",
    "# maxpool that uses a square window of kernel_size=2, stride=2\n",
    "self.pool = nn.MaxPool2d(2, 2)      \n",
    "```\n",
    "\n",
    "#### Refer to Layers in `forward`\n",
    "Then referred to in the `forward` function like this, in which the conv1 layer has a ReLu activation applied to it before maxpooling is applied:\n",
    "```\n",
    "x = self.pool(F.relu(self.conv1(x)))\n",
    "```\n",
    "\n",
    "Best practice is to place any layers whose weights will change during the training process in `__init__` and refer to them in the `forward` function; any layers or functions that always behave in the same way, such as a pre-defined activation function, should appear *only* in the `forward` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Why models.py\n",
    "\n",
    "You are tasked with defining the network in the `models.py` file so that any models you define can be saved and loaded by name in different notebooks in this project directory. For example, by defining a CNN class called `Net` in `models.py`, you can then create that same architecture in this and other notebooks by simply importing the class and instantiating a model:\n",
    "```\n",
    "    from models import Net\n",
    "    net = Net()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the usual resources\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "# watch for any changes in model.py, if it changes, re-load it automatically\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO: Define the Net in models.py\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "## TODO: Once you've define the network, you can instantiate it\n",
    "# one example conv layer has been provided for you\n",
    "from models import Net\n",
    "\n",
    "net = Net()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Transform the dataset \n",
    "\n",
    "To prepare for training, create a transformed dataset of images and keypoints.\n",
    "\n",
    "### TODO: Define a data transform\n",
    "\n",
    "In PyTorch, a convolutional neural network expects a torch image of a consistent size as input. For efficient training, and so your model's loss does not blow up during training, it is also suggested that you normalize the input images and keypoints. The necessary transforms have been defined in `data_load.py` and you **do not** need to modify these; take a look at this file (you'll see the same transforms that were defined and applied in Notebook 1).\n",
    "\n",
    "To define the data transform below, use a [composition](http://pytorch.org/tutorials/beginner/data_loading_tutorial.html#compose-transforms) of:\n",
    "1. Rescaling and/or cropping the data, such that you are left with a square image (the suggested size is 224x224px)\n",
    "2. Normalizing the images and keypoints; turning each RGB image into a grayscale image with a color range of [0, 1] and transforming the given keypoints into a range of [-1, 1]\n",
    "3. Turning these images and keypoints into Tensors\n",
    "\n",
    "These transformations have been defined in `data_load.py`, but it's up to you to call them and create a `data_transform` below. **This transform will be applied to the training data and, later, the test data**. It will change how you go about displaying these images and keypoints, but these steps are essential for efficient training.\n",
    "\n",
    "As a note, should you want to perform data augmentation (which is optional in this project), and randomly rotate or shift these images, a square image size will be useful; rotating a 224x224 image by 90 degrees will result in the same shape of output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset, DataLoader\n",
    "from torchvision import transforms, utils\n",
    "\n",
    "# the dataset we created in Notebook 1 is copied in the helper file `data_load.py`\n",
    "from data_load import FacialKeypointsDataset\n",
    "# the transforms we defined in Notebook 1 are in the helper file `data_load.py`\n",
    "from data_load import Rescale, RandomCrop, Normalize, ToTensor\n",
    "\n",
    "\n",
    "## TODO: define the data_transform using transforms.Compose([all tx's, . , .])\n",
    "# order matters! i.e. rescaling should come before a smaller crop\n",
    "data_transform = transforms.Compose([Rescale(250),\n",
    "                                     RandomCrop(224),\n",
    "                                     Normalize(),\n",
    "                                     ToTensor()])\n",
    "\n",
    "# testing that you've defined a transform\n",
    "assert(data_transform is not None), 'Define a data_transform'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the transformed dataset\n",
    "transformed_dataset = FacialKeypointsDataset(csv_file='data/training_frames_keypoints.csv',\n",
    "                                             root_dir='data/training/',\n",
    "                                             transform=data_transform)\n",
    "\n",
    "\n",
    "print('Number of images: ', len(transformed_dataset))\n",
    "\n",
    "# iterate through the transformed dataset and print some stats about the first few samples\n",
    "for i in range(4):\n",
    "    sample = transformed_dataset[i]\n",
    "    print(i, sample['image'].size(), sample['keypoints'].size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batching and loading data\n",
    "\n",
    "Next, having defined the transformed dataset, we can use PyTorch's DataLoader class to load the training data in batches of whatever size as well as to shuffle the data for training the model. You can read more about the parameters of the DataLoader, in [this documentation](http://pytorch.org/docs/master/data.html).\n",
    "\n",
    "#### Batch size\n",
    "Decide on a good batch size for training your model. Try both small and large batch sizes and note how the loss decreases as the model trains.\n",
    "\n",
    "**Note for Windows users**: Please change the `num_workers` to 0 or you may face some issues with your DataLoader failing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load training data in batches\n",
    "batch_size = 10\n",
    "\n",
    "train_loader = DataLoader(transformed_dataset, \n",
    "                          batch_size=batch_size,\n",
    "                          shuffle=True, \n",
    "                          num_workers=4)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before training\n",
    "\n",
    "Take a look at how this model performs before it trains. You should see that the keypoints it predicts start off in one spot and don't match the keypoints on a face at all! It's interesting to visualize this behavior so that you can compare it to the model after training and see how the model has improved.\n",
    "\n",
    "#### Load in the test dataset\n",
    "\n",
    "The test dataset is one that this model has *not* seen before, meaning it has not trained with these images. We'll load in this test data and before and after training, see how your model performs on this set!\n",
    "\n",
    "To visualize this test data, we have to go through some un-transformation steps to turn our images into python images from tensors and to turn our keypoints back into a recognizable range. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load in the test data, using the dataset class\n",
    "# AND apply the data_transform you defined above\n",
    "\n",
    "# create the test dataset\n",
    "test_dataset = FacialKeypointsDataset(csv_file='data/test_frames_keypoints.csv',\n",
    "                                             root_dir='data/test/',\n",
    "                                             transform=data_transform)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load test data in batches\n",
    "batch_size = 10\n",
    "\n",
    "test_loader = DataLoader(test_dataset, \n",
    "                          batch_size=batch_size,\n",
    "                          shuffle=True, \n",
    "                          num_workers=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply the model on a test sample\n",
    "\n",
    "To test the model on a test sample of data, you have to follow these steps:\n",
    "1. Extract the image and ground truth keypoints from a sample\n",
    "2. Make sure the image is a FloatTensor, which the model expects.\n",
    "3. Forward pass the image through the net to get the predicted, output keypoints.\n",
    "\n",
    "This function test how the network performs on the first batch of test data. It returns the images, the transformed images, the predicted keypoints (produced by the model), and the ground truth keypoints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test the model on a batch of test images\n",
    "\n",
    "def net_sample_output():\n",
    "    \n",
    "    # iterate through the test dataset\n",
    "    for i, sample in enumerate(test_loader):\n",
    "        \n",
    "        # get sample data: images and ground truth keypoints\n",
    "        images = sample['image']\n",
    "        key_pts = sample['keypoints']\n",
    "\n",
    "        # convert images to FloatTensors\n",
    "        images = images.type(torch.FloatTensor)\n",
    "\n",
    "        # forward pass to get net output\n",
    "        output_pts = net(images)\n",
    "        \n",
    "        # reshape to batch_size x 68 x 2 pts\n",
    "        output_pts = output_pts.view(output_pts.size()[0], 68, -1)\n",
    "        \n",
    "        # break after first image is tested\n",
    "        if i == 0:\n",
    "            return images, output_pts, key_pts\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Debugging tips\n",
    "\n",
    "If you get a size or dimension error here, make sure that your network outputs the expected number of keypoints! Or if you get a Tensor type error, look into changing the above code that casts the data into float types: `images = images.type(torch.FloatTensor)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# call the above function\n",
    "# returns: test images, test predicted keypoints, test ground truth keypoints\n",
    "test_images, test_outputs, gt_pts = net_sample_output()\n",
    "\n",
    "# print out the dimensions of the data to see if they make sense\n",
    "print(test_images.data.size())\n",
    "print(test_outputs.data.size())\n",
    "print(gt_pts.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the predicted keypoints\n",
    "\n",
    "Once we've had the model produce some predicted output keypoints, we can visualize these points in a way that's similar to how we've displayed this data before, only this time, we have to \"un-transform\" the image/keypoint data to display it.\n",
    "\n",
    "Note that I've defined a *new* function, `show_all_keypoints` that displays a grayscale image, its predicted keypoints and its ground truth keypoints (if provided)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_all_keypoints(image, predicted_key_pts, gt_pts=None):\n",
    "    \"\"\"Show image with predicted keypoints\"\"\"\n",
    "    # image is grayscale\n",
    "    plt.imshow(image, cmap='gray')\n",
    "    plt.scatter(predicted_key_pts[:, 0], predicted_key_pts[:, 1], s=20, marker='.', c='m')\n",
    "    # plot ground truth points as green pts\n",
    "    if gt_pts is not None:\n",
    "        plt.scatter(gt_pts[:, 0], gt_pts[:, 1], s=20, marker='.', c='g')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Un-transformation\n",
    "\n",
    "Next, you'll see a helper function. `visualize_output` that takes in a batch of images, predicted keypoints, and ground truth keypoints and displays a set of those images and their true/predicted keypoints.\n",
    "\n",
    "This function's main role is to take batches of image and keypoint data (the input and output of your CNN), and transform them into numpy images and un-normalized keypoints (x, y) for normal display. The un-transformation process turns keypoints and images into numpy arrays from Tensors *and* it undoes the keypoint normalization done in the Normalize() transform; it's assumed that you applied these transformations when you loaded your test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize the output\n",
    "# by default this shows a batch of 10 images\n",
    "def visualize_output(test_images, test_outputs, gt_pts=None, batch_size=10):\n",
    "\n",
    "    for i in range(batch_size):\n",
    "        plt.figure(figsize=(20,10))\n",
    "        ax = plt.subplot(1, batch_size, i+1)\n",
    "\n",
    "        # un-transform the image data\n",
    "        image = test_images[i].data   # get the image from it's wrapper\n",
    "        image = image.numpy()   # convert to numpy array from a Tensor\n",
    "        image = np.transpose(image, (1, 2, 0))   # transpose to go from torch to numpy image\n",
    "\n",
    "        # un-transform the predicted key_pts data\n",
    "        predicted_key_pts = test_outputs[i].data\n",
    "        predicted_key_pts = predicted_key_pts.numpy()\n",
    "        # undo normalization of keypoints  \n",
    "        predicted_key_pts = predicted_key_pts*50.0+100\n",
    "        \n",
    "        # plot ground truth points for comparison, if they exist\n",
    "        ground_truth_pts = None\n",
    "        if gt_pts is not None:\n",
    "            ground_truth_pts = gt_pts[i]         \n",
    "            ground_truth_pts = ground_truth_pts*50.0+100\n",
    "        \n",
    "        # call show_all_keypoints\n",
    "        show_all_keypoints(np.squeeze(image), predicted_key_pts, ground_truth_pts)\n",
    "            \n",
    "        plt.axis('off')\n",
    "\n",
    "    plt.show()\n",
    "    \n",
    "# call it\n",
    "visualize_output(test_images, test_outputs, gt_pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "#### Loss function\n",
    "Training a network to predict keypoints is different than training a network to predict a class; instead of outputting a distribution of classes and using cross entropy loss, you may want to choose a loss function that is suited for regression, which directly compares a predicted value and target value. Read about the various kinds of loss functions (like MSE or L1/SmoothL1 loss) in [this documentation](http://pytorch.org/docs/master/_modules/torch/nn/modules/loss.html).\n",
    "\n",
    "### TODO: Define the loss and optimization\n",
    "\n",
    "Next, you'll define how the model will train by deciding on the loss function and optimizer.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO: Define the loss and optimization\n",
    "import torch.optim as optim\n",
    "\n",
    "criterion = nn.SmoothL1Loss()\n",
    "\n",
    "optimizer = optim.Adam(net.parameters(), lr=0.001)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and Initial Observation\n",
    "\n",
    "Now, you'll train on your batched training data from `train_loader` for a number of epochs. \n",
    "\n",
    "To quickly observe how your model is training and decide on whether or not you should modify it's structure or hyperparameters, you're encouraged to start off with just one or two epochs at first. As you train, note how your the model's loss behaves over time: does it decrease quickly at first and then slow down? Does it take a while to decrease in the first place? What happens if you change the batch size of your training data or modify your loss function? etc. \n",
    "\n",
    "Use these initial observations to make changes to your model and decide on the best architecture before you train for many epochs and create a final model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_net(n_epochs):\n",
    "\n",
    "    # prepare the net for training\n",
    "    net.train()\n",
    "\n",
    "    for epoch in range(n_epochs):  # loop over the dataset multiple times\n",
    "        \n",
    "        running_loss = 0.0\n",
    "\n",
    "        # train on batches of data, assumes you already have train_loader\n",
    "        for batch_i, data in enumerate(train_loader):\n",
    "            # get the input images and their corresponding labels\n",
    "            images = data['image']\n",
    "            key_pts = data['keypoints']\n",
    "\n",
    "            # flatten pts\n",
    "            key_pts = key_pts.view(key_pts.size(0), -1)\n",
    "\n",
    "            # convert variables to floats for regression loss\n",
    "            key_pts = key_pts.type(torch.FloatTensor)\n",
    "            images = images.type(torch.FloatTensor)\n",
    "\n",
    "            # forward pass to get outputs\n",
    "            output_pts = net(images)\n",
    "\n",
    "            # calculate the loss between predicted and target keypoints\n",
    "            loss = criterion(output_pts, key_pts)\n",
    "\n",
    "            # zero the parameter (weight) gradients\n",
    "            optimizer.zero_grad()\n",
    "            \n",
    "            # backward pass to calculate the weight gradients\n",
    "            loss.backward()\n",
    "\n",
    "            # update the weights\n",
    "            optimizer.step()\n",
    "\n",
    "            # print loss statistics\n",
    "            # to convert loss into a scalar and add it to the running_loss, use .item()\n",
    "            running_loss += loss.item()\n",
    "            if batch_i % 10 == 9:    # print every 10 batches\n",
    "                print('Epoch: {}, Batch: {}, Avg. Loss: {}'.format(epoch + 1, batch_i+1, running_loss/1000))\n",
    "                running_loss = 0.0\n",
    "\n",
    "    print('Finished Training')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train your network\n",
    "n_epochs = 10 # start small, and increase when you've decided on your model structure and hyperparams\n",
    "\n",
    "train_net(n_epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test data\n",
    "\n",
    "See how your model performs on previously unseen, test data. We've already loaded and transformed this data, similar to the training data. Next, run your trained model on these images to see what kind of keypoints are produced. You should be able to see if your model is fitting each new face it sees, if the points are distributed randomly, or if the points have actually overfitted the training data and do not generalize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get a sample of test data again\n",
    "net.eval()\n",
    "test_images, test_outputs, gt_pts = net_sample_output()\n",
    "\n",
    "print(test_images.data.size())\n",
    "print(test_outputs.data.size())\n",
    "print(gt_pts.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO: visualize your test output\n",
    "# you can use the same function as before, by un-commenting the line below:\n",
    "\n",
    "net.eval()\n",
    "visualize_output(test_images, test_outputs, gt_pts)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've found a good model (or two), save your model so you can load it and use it later!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO: change the name to something uniqe for each new model\n",
    "model_dir = 'saved_models/'\n",
    "model_name = 'keypoints_model_1.pt'\n",
    "#model_name = 'keypoints_model_2.pt'\n",
    "\n",
    "# after training, save your model parameters in the dir 'saved_models'\n",
    "torch.save(net.state_dict(), model_dir+model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you've trained a well-performing model, answer the following questions so that we have some insight into your training and architecture selection process. Answering all questions is required to pass this project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1: What optimization and loss functions did you choose and why?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer**:\n",
    "criterion = nn.SmoothL1Loss()\n",
    "\n",
    "optimizer = optim.Adam(net.parameters(), lr=0.001)\n",
    "\n",
    "For the loss function, i chose Smooth L1 loss since it prevents exploding gradients and also it is less sensitive to outliers than MSELoss.\n",
    "\n",
    "For the optimizer , i chose Adam Optimizer since it is well suited for problems that are large in terms of data and/or parameters, it is computationally efficient with little memory requirements. Also it is invariant to diagonal rescale of the gradients and it is appropriate for problems with very noisy or sparse gradients.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2: What kind of network architecture did you start with and how did it change as you tried different architectures? Did you decide to add more convolutional layers or any layers to avoid overfitting the data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer**:\n",
    "\n",
    "To start i referred to the Naimish arhcitecture but i really did not attempt to replicate it. I started of using 2 convolution layers and one fully conencted layers and built this up to use more convolution layers and fully conencted layers since my training loss was high. I did use a kernel size of 5 and used three convolution layers with filter sizes of 32, 64 and 128. All these 3 convolution layers had dropout of 0.4 after max pooling.\n",
    "\n",
    "I then used to two fully conencted layers. Only the first fully connected layer had a dropout of 0.4 after the activation.\n",
    "\n",
    "I did several experiments and settled for the above. Here are some of the other observations I had along with some other architectures, I tried.\n",
    "\n",
    "- Using a kernel size of 3 for the convolution layers actually made the training loss a little worse\n",
    "\n",
    "- I tried using Xavier init for the weights and with this the training loss was to some extent oscilatting. Hence i did not proceed with this.\n",
    "\n",
    "- For my 3 convolution layers, i tried filters of 32, 48 and 64 instead of 32, 64 and 128. The results were worse.\n",
    "\n",
    "- Using three fully connected layers instead of two was only marginally worse. I was expecting this to be actually better. Also adding just an extra fully connected layer increased the amount of training time per epoch substantially.\n",
    "\n",
    "- I used a learning rate of 0.001. Bumping this up to 0.01 made the training loss worse. Even making the learning rate 0.003 did not help performance.\n",
    "\n",
    "These are some of the variations i tried; after much experimentation i settled for the current architecture that can be seen in models.py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3: How did you decide on the number of epochs and batch_size to train your model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer**:\n",
    "\n",
    "I kept the batch size at 10 and increased the number of epochs to 10. These were mainly decided by the amount of time the training was taking. I first started of with an epoch of 1 to ensure that i was atleast getting reasonable trainig loss. After that i increased the number to 5 and finally to 10. I am sure increasing the number of epochs to a higher number say 20 or more would in all likelihood have made my model a lot better but it was taking too much time to train, hence i stuck with 10 as the number of epochs. I tried one run with a batch size of 20 but that did not help to reduce the training loss so i reverted back to 10."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Visualization\n",
    "\n",
    "Sometimes, neural networks are thought of as a black box, given some input, they learn to produce some output. CNN's are actually learning to recognize a variety of spatial patterns and you can visualize what each convolutional layer has been trained to recognize by looking at the weights that make up each convolutional kernel and applying those one at a time to a sample image. This technique is called feature visualization and it's useful for understanding the inner workings of a CNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell below, you can see how to extract a single filter (by index) from your first convolutional layer. The filter should appear as a grayscale grid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the weights in the first conv layer, \"conv1\"\n",
    "# if necessary, change this to reflect the name of your first conv layer\n",
    "weights1 = net.conv1.weight.data\n",
    "\n",
    "w = weights1.numpy()\n",
    "\n",
    "filter_index = 5\n",
    "\n",
    "print(w[filter_index][0])\n",
    "print(w[filter_index][0].shape)\n",
    "\n",
    "# display the filter weights\n",
    "plt.imshow(w[filter_index][0], cmap='gray')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature maps\n",
    "\n",
    "Each CNN has at least one convolutional layer that is composed of stacked filters (also known as convolutional kernels). As a CNN trains, it learns what weights to include in it's convolutional kernels and when these kernels are applied to some input image, they produce a set of **feature maps**. So, feature maps are just sets of filtered images; they are the images produced by applying a convolutional kernel to an input image. These maps show us the features that the different layers of the neural network learn to extract. For example, you might imagine a convolutional kernel that detects the vertical edges of a face or another one that detects the corners of eyes. You can see what kind of features each of these kernels detects by applying them to an image. One such example is shown below; from the way it brings out the lines in an the image, you might characterize this as an edge detection filter.\n",
    "\n",
    "<img src='images/feature_map_ex.png' width=50% height=50%/>\n",
    "\n",
    "\n",
    "Next, choose a test image and filter it with one of the convolutional kernels in your trained CNN; look at the filtered output to get an idea what that particular kernel detects.\n",
    "\n",
    "### TODO: Filter an image to see the effect of a convolutional kernel\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##TODO: load in and display any image from the transformed test dataset\n",
    "import cv2\n",
    "image_num = 3\n",
    "\n",
    "im = test_images[image_num].data   # get the image from it's wrapper\n",
    "im = im.numpy()   # convert to numpy array from a Tensor\n",
    "im = np.transpose(im, (1, 2, 0)) \n",
    "\n",
    "plt.imshow(np.squeeze(im), cmap='gray')\n",
    "plt.show()\n",
    "## TODO: Using cv's filter2D function,\n",
    "## apply a specific set of filter weights (like the one displayed above) to the test image\n",
    "c = cv2.filter2D(im, -1, w[filter_index][0])\n",
    "plt.imshow(c, cmap='gray')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4: Choose one filter from your trained CNN and apply it to a test image; what purpose do you think it plays? What kind of feature do you think it detects?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer**: \n",
    "\n",
    "By carefully looking at the above images, it can be clearly seen that every time there is a change in intensity of the color, it gets detected. Let us look at the neck area along with the shirt and coat. We can clearly see that the filtered image detects the change in color intensity. Next let us look at the hair; we can clearly see where the hair mingles with the background in the filtered picture. This is behaving more like an edge detector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Moving on!\n",
    "\n",
    "Now that you've defined and trained your model (and saved the best model), you are ready to move on to the last notebook, which combines a face detector with your saved model to create a facial keypoint detection system that can predict the keypoints on *any* face in an image!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}