colab.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/google/applied-machine-learning-intensive/blob/master/content/05_deep_learning/01_recurrent_neural_networks/colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5b8JqGUoPG78"
      },
      "source": [
        "#### Copyright 2020 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aAAE3rDaPMGo"
      },
      "outputs": [],
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UnD_6o5-XdIb"
      },
      "source": [
        "# Recurrent Neural Networks (RNNs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9HLFzlpOXgq4"
      },
      "source": [
        "Recurrent Neural Networks (RNNs) are an interesting application of deep learning that allow models to predict the future. While regression models attempt to fit an equation to existing data and extend the predictive power of the equation into the future, RNNs fit a model and use sequences of time series data to make step-by-step predictions about the next most likely output of the model.\n",
        "\n",
        "In this colab we will create a recurrent neural network that can predict engine vibrations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vq5Zt4HUWiLt"
      },
      "source": [
        "## Exploratory Data Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2PJeAYSEagTs"
      },
      "source": [
        "We'll use the [Engine Vibrations data](https://www.kaggle.com/joshmcadams/engine-vibrations) from Kaggle. This dataset contains artificial engine vibration values we will use to train a model that can predict future values.\n",
        "\n",
        "To load the data, upload your `kaggle.json` file and run the code block below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1i53xIkD-t-v"
      },
      "outputs": [],
      "source": [
        "! chmod 600 kaggle.json && (ls ~/.kaggle 2>/dev/null || mkdir ~/.kaggle) && mv kaggle.json ~/.kaggle/ && echo 'Done'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KH3nEJSF_T6C"
      },
      "source": [
        "Next, download the data from Kaggle."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xD0OyLnq_WYZ"
      },
      "outputs": [],
      "source": [
        "!kaggle datasets download joshmcadams/engine-vibrations\n",
        "!ls"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q0gY53BBAE_q"
      },
      "source": [
        "Now load the data into a `DataFrame`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AHkcYf8IAB9a"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.read_csv('engine-vibrations.zip')\n",
        "df.describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cWW6wLrhtkGC"
      },
      "source": [
        "We know the data contains readings of engine vibration over time. Let's see how that looks on a line chart."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "geUfAGJZdNTE"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.figure(figsize=(24, 8))\n",
        "plt.plot(list(range(len(df['mm']))), df['mm'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rpb9dOHqt31W"
      },
      "source": [
        "That's quite a tough chart to read. Let's sample it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "khEnEOkgt8IS"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.figure(figsize=(24, 8))\n",
        "plt.plot(list(range(100)), df['mm'].iloc[:100])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nDzwV2rLucrc"
      },
      "source": [
        "See if any of the data is missing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3pzg5wM_uYjG"
      },
      "outputs": [],
      "source": [
        "df.isna().any()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N6Ez86Cmunox"
      },
      "source": [
        "Finally, we'll do a box plot to see if the data is evenly distributed, which it is."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Fd0Ha-VOufGG"
      },
      "outputs": [],
      "source": [
        "import seaborn as sns\n",
        "\n",
        "_ = sns.boxplot(df['mm'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q9K3fN5_utmQ"
      },
      "source": [
        "There is not much more EDA we need to do at this point. Let's move on to modeling."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z7_dmDYIa6bY"
      },
      "source": [
        "## Preparing the Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CMZXi7_pv-JM"
      },
      "source": [
        "Currently we have a series of data that contains a single list of vibration values over time. When training our model and when asking for predictions, we'll want to instead feed the model a subset of our sequence.\n",
        "\n",
        "We first need to determine our subsequence length and then create in-order subsequences of that length.\n",
        "\n",
        "We'll create a list of lists called `X` that contains subsequences. We'll also create a list called `y` that contains the next value after each subsequence stored in `X`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8kjythCQ-8xX"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "X = []\n",
        "y = []\n",
        "sseq_len = 50\n",
        "for i in range(0, len(df['mm']) - sseq_len - 1):\n",
        "  X.append(df['mm'][i:i+sseq_len])\n",
        "  y.append(df['mm'][i+sseq_len+1])\n",
        "\n",
        "y = np.array(y)\n",
        "X = np.array(X)\n",
        "\n",
        "X.shape, y.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aAhFLYpsyYsP"
      },
      "source": [
        "We also need to explicitly set the final dimension of the data in order to have it pass through our model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SSqMsXK4sYTE"
      },
      "outputs": [],
      "source": [
        "X = np.expand_dims(X, axis=2)\n",
        "y = np.expand_dims(y, axis=1)\n",
        "\n",
        "X.shape, y.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G4ARNFpJypCc"
      },
      "source": [
        "We'll also standardize our data for the model. Note that we don't normalize here because we need to be able to reproduce negative values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lnVcS3sxVgSF"
      },
      "outputs": [],
      "source": [
        "data_std = df['mm'].std()\n",
        "data_mean = df['mm'].mean()\n",
        "\n",
        "X = (X - data_mean) / data_std\n",
        "y = (y - data_mean) / data_std\n",
        "\n",
        "X.max(), y.max(), X.min(), y.min()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L7xD_PkVyyZR"
      },
      "source": [
        "And for final testing after model training, we'll split off 20% of the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4X2rePx2WjaZ"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    X, y, test_size=0.2, random_state=0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vI6KMXyry75a"
      },
      "source": [
        "## Setting a Baseline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8T7NIsw2y9qp"
      },
      "source": [
        "We are only training with 50 data points at a time. This is well within the bounds of what a standard deep neural network can handle, so let's first see what a very simple neural network can do."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b5787JSoWbS-"
      },
      "outputs": [],
      "source": [
        "import math\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "import tensorflow.keras as keras\n",
        "\n",
        "tf.random.set_seed(0)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "  keras.layers.Flatten(input_shape=[sseq_len, 1]),\n",
        "  keras.layers.Dense(1)\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "  loss='mse',\n",
        "  optimizer='Adam',\n",
        "  metrics=['mae', 'mse'],\n",
        ")\n",
        "\n",
        "stopping = tf.keras.callbacks.EarlyStopping(\n",
        "  monitor='loss',\n",
        "  min_delta=0,\n",
        "  patience=2)\n",
        "\n",
        "history = model.fit(X_train, y_train, epochs=50, callbacks=[stopping])\n",
        "\n",
        "y_pred = model.predict(X_test)\n",
        "rmse = math.sqrt(np.mean(keras.losses.mean_squared_error(y_test, y_pred)))\n",
        "print(\"RMSE Scaled: {}\\nRMSE Base Units: {}\".format(\n",
        "    rmse, rmse * data_std + data_mean))\n",
        "\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.plot(list(range(len(history.history['mse']))), history.history['mse'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XPr-FVCZ0BC-"
      },
      "source": [
        "We quickly converged and, when we ran the model, we got a baseline quality value of `0.03750885081060467`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OJSxRb6i0LPo"
      },
      "source": [
        "## The Most Basic RNN"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CaSl6sHz0SzH"
      },
      "source": [
        "Let's contrast a basic feedforward neural network with a basic RNN. To do this we simply need to use the [`SimpleRNN` layer](https://www.tensorflow.org/api_docs/python/tf/keras/layers/SimpleRNN) in our network in place of the `Dense` layer in our network above. Notice that, in this case, there is no need to flatten the data before we feed it into the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "btjmwN7eZvfP"
      },
      "outputs": [],
      "source": [
        "tf.random.set_seed(0)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "  keras.layers.SimpleRNN(1, input_shape=[None, 1])\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "  loss='mse',\n",
        "  optimizer='Adam',\n",
        "  metrics=['mae', 'mse'],\n",
        ")\n",
        "\n",
        "stopping = tf.keras.callbacks.EarlyStopping(\n",
        "  monitor='loss',\n",
        "  min_delta=0,\n",
        "  patience=2)\n",
        "\n",
        "history = model.fit(X_train, y_train, epochs=50, callbacks=[stopping])\n",
        "\n",
        "y_pred = model.predict(X_test)\n",
        "rmse = math.sqrt(np.mean(keras.losses.mean_squared_error(y_test, y_pred)))\n",
        "print(\"RMSE Scaled: {}\\nRMSE Base Units: {}\".format(\n",
        "    rmse, rmse * data_std + data_mean))\n",
        "\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.plot(list(range(len(history.history['mse']))), history.history['mse'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yEzJ2o5N1h4Y"
      },
      "source": [
        "Our model converged a little more slowly, but it got an error of only `0.8974118571865628`, which is not an improvement over the baseline model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "umjB2bG12AHi"
      },
      "source": [
        "## A Deep RNN"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q15DdJkM1-gm"
      },
      "source": [
        "Let's try to build a deep RNN and see if we can get better results.\n",
        "\n",
        "In the model below, we stick together four layers ranging in width from `50` nodes to our final output of `1`.\n",
        "\n",
        "Notice all of the layers except the output layer have `return_sequences=True` set. This causes the layer to pass outputs for all timestamps to the next layer. If you don't include this argument, only the output for the last timestamp is passed, and intermediate layers will complain about the wrong shape of input."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6BzFeGita_dx"
      },
      "outputs": [],
      "source": [
        "tf.random.set_seed(0)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.SimpleRNN(50, return_sequences=True, input_shape=[None, 1]),\n",
        "    keras.layers.SimpleRNN(20, return_sequences=True),\n",
        "    keras.layers.SimpleRNN(10, return_sequences=True),\n",
        "    keras.layers.SimpleRNN(1)\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "  loss='mse',\n",
        "  optimizer='Adam',\n",
        "  metrics=['mae', 'mse'],\n",
        ")\n",
        "\n",
        "stopping = tf.keras.callbacks.EarlyStopping(\n",
        "  monitor='loss',\n",
        "  min_delta=0,\n",
        "  patience=2)\n",
        "\n",
        "history = model.fit(X_train, y_train, epochs=50, callbacks=[stopping])\n",
        "\n",
        "y_pred = model.predict(X_test)\n",
        "rmse = math.sqrt(np.mean(keras.losses.mean_squared_error(y_test, y_pred)))\n",
        "print(\"RMSE Scaled: {}\\nRMSE Base Units: {}\".format(\n",
        "    rmse, rmse * data_std + data_mean))\n",
        "\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.plot(list(range(len(history.history['mse']))), history.history['mse'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jcWDYuDX3PmL"
      },
      "source": [
        "Woah! What happened? Our MSE during training looked nice: `0.0496`. But our final testing didn't perform much better than our simple model. We seem to have overfit!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KpFsyAT0ZGRp"
      },
      "source": [
        "We can try to simplify the model and add dropout layers to reduce overfitting, but even with a very basic model like the one below, we still get very different MSE between the training and test datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6nscnyybb06H"
      },
      "outputs": [],
      "source": [
        "tf.random.set_seed(0)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.SimpleRNN(2, return_sequences=True, input_shape=[None, 1]),\n",
        "    keras.layers.Dropout(0.3),\n",
        "    keras.layers.SimpleRNN(1),\n",
        "    keras.layers.Dense(1)\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "  loss='mse',\n",
        "  optimizer='Adam',\n",
        "  metrics=['mae', 'mse'],\n",
        ")\n",
        "\n",
        "stopping = tf.keras.callbacks.EarlyStopping(\n",
        "  monitor='loss',\n",
        "  min_delta=0,\n",
        "  patience=2)\n",
        "\n",
        "history = model.fit(X_train, y_train, epochs=50, callbacks=[stopping])\n",
        "\n",
        "y_pred = model.predict(X_test)\n",
        "rmse = math.sqrt(np.mean(keras.losses.mean_squared_error(y_test, y_pred)))\n",
        "print(\"RMSE Scaled: {}\\nRMSE Base Units: {}\".format(\n",
        "    rmse, rmse * data_std + data_mean))\n",
        "\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.plot(list(range(len(history.history['mse']))), history.history['mse'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HgStKfGWFnO0"
      },
      "source": [
        "Even with these measures, we still seem to be overfitting a bit. We could keep tuning, but let's instead look at some other types of neurons found in RNNs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eNKv2NFEZfGc"
      },
      "source": [
        "## Long Short Term Memory"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b4v2XkdYZmuY"
      },
      "source": [
        "The RNN layers we've been using are basic neurons that have a very short memory. They tend to learn patterns that they have recently seen, but they quickly forget older training data.\n",
        "\n",
        "The **Long Short Term Memory (LSTM)** neuron was built to combat this forgetfulness. The neuron outputs values for the next layer in the network, and it also outputs two other values: one for short-term memory and one for long-term memory. These weights are then fed back into the neuron at the next iteration of the network. This backfeed is similar to that of a `SimpleRNN`, except the `SimpleRNN` only has one backfeed.\n",
        "\n",
        "We can replace the `SimpleRNN` with an `LSTM` layer, as you can see below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C_VG4uC0KsKB"
      },
      "outputs": [],
      "source": [
        "tf.random.set_seed(0)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.LSTM(1, input_shape=[None, 1]),\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "  loss='mse',\n",
        "  optimizer=tf.keras.optimizers.Adam(),\n",
        "  metrics=['mae', 'mse'],\n",
        ")\n",
        "\n",
        "stopping = tf.keras.callbacks.EarlyStopping(\n",
        "  monitor='loss',\n",
        "  min_delta=0,\n",
        "  patience=2)\n",
        "\n",
        "history = model.fit(X_train, y_train, epochs=100, callbacks=[stopping])\n",
        "\n",
        "y_pred = model.predict(X_test)\n",
        "rmse = math.sqrt(np.mean(keras.losses.mean_squared_error(y_test, y_pred)))\n",
        "print(\"RMSE Scaled: {}\\nRMSE Base Units: {}\".format(\n",
        "    rmse, rmse * data_std + data_mean))\n",
        "\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.plot(list(range(len(history.history['mse']))), history.history['mse'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wkbr61Cq4B09"
      },
      "source": [
        "We got a test RMSE of `0.8989123704842217`, which is still not better than our `SimpleRNN`. And in the more complex model below, we got close to the baseline but still didn't beat it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w9uAJtpcc8Hg",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "tf.random.set_seed(0)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.LSTM(20, return_sequences=True, input_shape=[None, 1]),\n",
        "    keras.layers.Dropout(0.2),\n",
        "    keras.layers.LSTM(10),\n",
        "    keras.layers.Dropout(0.2),\n",
        "    keras.layers.Dense(1)\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "  loss='mse',\n",
        "  optimizer='Adam',\n",
        "  metrics=['mae', 'mse'],\n",
        ")\n",
        "\n",
        "stopping = tf.keras.callbacks.EarlyStopping(\n",
        "  monitor='loss',\n",
        "  min_delta=0,\n",
        "  patience=2)\n",
        "\n",
        "history = model.fit(X_train, y_train, epochs=50, callbacks=[stopping])\n",
        "\n",
        "y_pred = model.predict(X_test)\n",
        "rmse = math.sqrt(np.mean(keras.losses.mean_squared_error(y_test, y_pred)))\n",
        "print(\"RMSE Scaled: {}\\nRMSE Base Units: {}\".format(\n",
        "    rmse, rmse * data_std + data_mean))\n",
        "\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.plot(list(range(len(history.history['mse']))), history.history['mse'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S5qJ4WGa4vN6"
      },
      "source": [
        "LSTM neurons can be very useful, but as we have seen, they aren't always the best option.\n",
        "\n",
        "Let's look at one more neuron commonly found in RNN models, the GRU."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K8IkQTQ447N2"
      },
      "source": [
        "## Gated Recurrent Unit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "21OFa1UC5UgB"
      },
      "source": [
        "The Gated Recurrent Unit (GRU) is another special neuron that often shows up in Recurrent Neural Networks. The GRU is similar to the LSTM in that it feeds output back into itself. The difference is that the GRU feeds a single weight back into itself and then makes long- and short-term state adjustments based on that single backfeed.\n",
        "\n",
        "The GRU tends to train faster than LSTM and has similar performance. Let's see how a network containing one GRU performs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f9JjfseAdl2n"
      },
      "outputs": [],
      "source": [
        "%time\n",
        "tf.random.set_seed(0)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.GRU(1),\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "  loss='mse',\n",
        "  optimizer='Adam',\n",
        "  metrics=['mae', 'mse'],\n",
        ")\n",
        "\n",
        "stopping = tf.keras.callbacks.EarlyStopping(\n",
        "  monitor='loss',\n",
        "  min_delta=0,\n",
        "  patience=2)\n",
        "\n",
        "history = model.fit(X_train, y_train, epochs=50, callbacks=[stopping])\n",
        "\n",
        "y_pred = model.predict(X_test)\n",
        "rmse = math.sqrt(np.mean(keras.losses.mean_squared_error(y_test, y_pred)))\n",
        "print(\"RMSE Scaled: {}\\nRMSE Base Units: {}\".format(\n",
        "    rmse, rmse * data_std + data_mean))\n",
        "\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.plot(list(range(len(history.history['mse']))), history.history['mse'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MunF57N07iWG"
      },
      "source": [
        "We got a RMSE of `0.9668634342193015`, which isn't bad, but it still performs worse than our baseline."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EGZRgWNF7pJF"
      },
      "source": [
        "## Convolutional Layers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xELjm9uh7sEZ"
      },
      "source": [
        "Convolutional layers are limited to image classification models. They can also be really handy when training RNNs. For training on a sequence of data, we use the [`Conv1D` class](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1D) as shown below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QbYLGqn2eG9e"
      },
      "outputs": [],
      "source": [
        "tf.random.set_seed(0)\n",
        "\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Conv1D(filters=20, kernel_size=4, strides=2, padding=\"valid\",\n",
        "                        input_shape=[None, 1]),\n",
        "    keras.layers.GRU(2, input_shape=[None, 1], activation='relu'),\n",
        "    keras.layers.Dropout(0.2),\n",
        "    keras.layers.Dense(1),\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "  loss='mse',\n",
        "  optimizer='Adam',\n",
        "  metrics=['mae', 'mse'],\n",
        ")\n",
        "\n",
        "stopping = tf.keras.callbacks.EarlyStopping(\n",
        "  monitor='loss',\n",
        "  min_delta=0,\n",
        "  patience=2)\n",
        "\n",
        "history = model.fit(X_train, y_train, epochs=50, callbacks=[stopping])\n",
        "\n",
        "y_pred = model.predict(X_test)\n",
        "rmse = math.sqrt(np.mean(keras.losses.mean_squared_error(y_test, y_pred)))\n",
        "print(\"RMSE Scaled: {}\\nRMSE Base Units: {}\".format(\n",
        "    rmse, rmse * data_std + data_mean))\n",
        "\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.plot(list(range(len(history.history['mse']))), history.history['mse'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sVTGTJiP8Rea"
      },
      "source": [
        "Recurrent Neural Networks are a powerful tool for sequence generation and prediction. But they aren't the only mechanism for sequence prediction. If the sequence you are predicting is short enough, then a standard deep neural network might be able to provide the predictions you are looking for.\n",
        "\n",
        "Also note that we created a model that took a series of data and output one value. It is possible to create RNNs that input one or more values and output one or more values. Each use case is different."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ay3NBOQ39C4M"
      },
      "source": [
        "# Exercises"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5MRMSH_l9ETz"
      },
      "source": [
        "## Exercise 1: Visualization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y8v9_mQm9GF6"
      },
      "source": [
        "Create a plot containing a series of at least 50 predicted points. Plot that series against the actual.\n",
        "\n",
        "> *Hint: Pick a sequence of 100 values from the original data. Plot data points 50-100 as the actual line. Then predict 50 single values starting with the features 0-49, 1-50, etc.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BryuihNkBYhx"
      },
      "source": [
        "### **Student Solution**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mHp2MEpjBaPl"
      },
      "outputs": [],
      "source": [
        "# Your code goes here\n",
        "\n",
        "plt.figure(figsize=(50,5))\n",
        "plt.plot(list(range(50)), df['mm'].iloc[:50],)\n",
        "plt.plot(list(range(50)), y_pred[:50],)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zzZAmgPUBbf6"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xfAND89KJSPS"
      },
      "source": [
        "## Exercise 2: Stock Price Prediction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JdL2S7oVO7aV"
      },
      "source": [
        "Using the [`Stonks!`](https://www.kaggle.com/joshmcadams/stonks) dataset, create a recurrent neural network that can predict the stock price for the 'AAA' ticker. Calculate your RMSE with some holdout data.\n",
        "\n",
        "Use as many text and code cells as you need to complete this exercise.\n",
        "\n",
        "> *Hint: if predicting absolute prices doesn't yield a good model, look into other ways to represent the day-to-day change in data.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O0G2cMfuPK1T"
      },
      "outputs": [],
      "source": [
        "#1\n",
        "!kaggle datasets download joshmcadams/stonks\n",
        "!ls\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "7dzNaIQZx07I"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#2\n",
        "import pandas as pd \n",
        "import zipfile \n",
        "with zipfile.ZipFile('stonks.zip', 'r') as zip_ref:\n",
        "  zip_ref.extractall('stonks')\n"
      ],
      "metadata": {
        "id": "r_bow_aBrM8p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "#3\n",
        "df = pd.read_csv('stonks.csv')"
      ],
      "metadata": {
        "id": "QzuvqsMMx6-X"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#4\n",
        "import seaborn as sns \n",
        "\n",
        "_=sns.heatmap(df.corr(),annot=True)"
      ],
      "metadata": {
        "id": "MYwQZIpqvaWa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#5\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    X, y, test_size=0.2, random_state=0)"
      ],
      "metadata": {
        "id": "0-7ZGKyvvkQz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#6\n",
        "tf.random.set_seed(0)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.GRU(1),\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "  loss='mse',\n",
        "  optimizer='Adam',\n",
        "  metrics=['mae', 'mse'],\n",
        ")\n",
        "\n",
        "stopping = tf.keras.callbacks.EarlyStopping(\n",
        "  monitor='loss',\n",
        "  min_delta=0,\n",
        "  patience=2)\n",
        "\n",
        "history = model.fit(X_train, y_train, epochs=50, callbacks=[stopping])\n",
        "\n",
        "y_pred = model.predict(X_test)\n",
        "rmse = math.sqrt(np.mean(keras.losses.mean_squared_error(y_test, y_pred)))\n",
        "print(\"RMSE Scaled: {}\\nRMSE Base Units: {}\".format(\n",
        "    rmse, rmse * data_std + data_mean))\n",
        "\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.plot(list(range(len(history.history['mse']))), history.history['mse'])\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "w23jMcTQsItK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#7\n",
        "plt.figure(figsize=(50,5))\n",
        "plt.plot(list(range(50)), y_test[:50],)\n",
        "plt.plot(list(range(50)), y_pred[:50],)\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "6qnyrjCOwlPO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#8\n",
        "plt.plot(history.history['mse'])"
      ],
      "metadata": {
        "id": "qMCaIPIVvTM9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ESKR9NfEJu3o"
      },
      "source": [
        "---"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "5b8JqGUoPG78",
        "45xublipBcE5",
        "NNwQ-S2KPVRI"
      ],
      "name": "rec_neural_networks.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}