Creating_and_Evaluating_Predictors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating and Evaluating Predictors: Part 1 - Target Time Series\n",
    "\n",
    "This notebook will build off of the earlier data processing that was performed in the validation sessions. If you have not completed that part yet, go back to `Validating_and_Importing_Target_Time_Series_Data.ipynb` and complete it first before resuming.\n",
    "\n",
    "At this point you have target-time-series data loaded into Amazon Forecast inside a Dataset Group, this is what is required to use all of the models within Amazon Forecast. As an initial exploration we will evaluate the results from ARIMA, Prophet, and DeepAR+. We could have also included ETS but have left it out for time constraints, similarly NPTS was left out as it specializes on spiky data or large gaps which our dataset does not have.\n",
    "\n",
    "The very first thing to do is start with our imports, establish a connection to the Forecast service, and then restore our variables from before. The cells below will do that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from time import sleep\n",
    "import subprocess\n",
    "import pandas as pd\n",
    "import json\n",
    "import time\n",
    "import pprint\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.dates import DateFormatter\n",
    "import matplotlib.dates as mdates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('/opt/ml/metadata/resource-metadata.json') as notebook_info:\n",
    "    data = json.load(notebook_info)\n",
    "    resource_arn = data['ResourceArn']\n",
    "    region = resource_arn.split(':')[3]\n",
    "print(region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = boto3.Session(region_name=region)\n",
    "forecast = session.client(service_name='forecast')\n",
    "forecast_query = session.client(service_name='forecastquery')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating and Training Predictors\n",
    " \n",
    "Given that that our data is hourly and we want to generate a forecast on the hour, Forecast limits us to a horizon of 500 of whatever the slice is. This means we will be able to predict about 20 days into the future.\n",
    "\n",
    "The cells below will define a few variables to be used with all of our models. Then there will be an API call to create each `Predictor` where they are based on ARIMA, Prophet, and DeepAR+ respectfully.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecastHorizon = 480\n",
    "NumberOfBacktestWindows = 4\n",
    "BackTestWindowOffset = 480\n",
    "ForecastFrequency = \"H\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arima_algorithmArn = 'arn:aws:forecast:::algorithm/ARIMA'\n",
    "prophet_algorithmArn = 'arn:aws:forecast:::algorithm/Prophet'\n",
    "deepAR_Plus_algorithmArn = 'arn:aws:forecast:::algorithm/Deep_AR_Plus'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ARIMA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ARIMA Specifics\n",
    "arima_predictorName= project+'_arima_algo_1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build ARIMA:\n",
    "arima_create_predictor_response=forecast.create_predictor(PredictorName=arima_predictorName, \n",
    "                                                  AlgorithmArn=arima_algorithmArn,\n",
    "                                                  ForecastHorizon=forecastHorizon,\n",
    "                                                  PerformAutoML= False,\n",
    "                                                  PerformHPO=False,\n",
    "                                                  EvaluationParameters= {\"NumberOfBacktestWindows\": NumberOfBacktestWindows, \n",
    "                                                                         \"BackTestWindowOffset\": BackTestWindowOffset}, \n",
    "                                                  InputDataConfig= {\"DatasetGroupArn\": datasetGroupArn, \"SupplementaryFeatures\": [ \n",
    "                                                                     { \n",
    "                                                                        \"Name\": \"holiday\",\n",
    "                                                                        \"Value\": \"US\"\n",
    "                                                                     }\n",
    "                                                                  ]},\n",
    "                                                  FeaturizationConfig= {\"ForecastFrequency\": ForecastFrequency, \n",
    "                                                                        \"Featurizations\": \n",
    "                                                                        [\n",
    "                                                                          {\"AttributeName\": \"target_value\", \n",
    "                                                                           \"FeaturizationPipeline\": \n",
    "                                                                            [\n",
    "                                                                              {\"FeaturizationMethodName\": \"filling\", \n",
    "                                                                               \"FeaturizationMethodParameters\": \n",
    "                                                                                {\"frontfill\": \"none\", \n",
    "                                                                                 \"middlefill\": \"zero\", \n",
    "                                                                                 \"backfill\": \"zero\"}\n",
    "                                                                              }\n",
    "                                                                            ]\n",
    "                                                                          }\n",
    "                                                                        ]\n",
    "                                                                       }\n",
    "                                                 )\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prophet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prophet Specifics\n",
    "prophet_predictorName= project+'_prophet_algo_1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build Prophet:\n",
    "prophet_create_predictor_response=forecast.create_predictor(PredictorName=prophet_predictorName, \n",
    "                                                  AlgorithmArn=prophet_algorithmArn,\n",
    "                                                  ForecastHorizon=forecastHorizon,\n",
    "                                                  PerformAutoML= False,\n",
    "                                                  PerformHPO=False,\n",
    "                                                  EvaluationParameters= {\"NumberOfBacktestWindows\": NumberOfBacktestWindows, \n",
    "                                                                         \"BackTestWindowOffset\": BackTestWindowOffset}, \n",
    "                                                  InputDataConfig= {\"DatasetGroupArn\": datasetGroupArn, \"SupplementaryFeatures\": [ \n",
    "                                                                     { \n",
    "                                                                        \"Name\": \"holiday\",\n",
    "                                                                        \"Value\": \"US\"\n",
    "                                                                     }\n",
    "                                                                  ]},\n",
    "                                                  FeaturizationConfig= {\"ForecastFrequency\": ForecastFrequency, \n",
    "                                                                        \"Featurizations\": \n",
    "                                                                        [\n",
    "                                                                          {\"AttributeName\": \"target_value\", \n",
    "                                                                           \"FeaturizationPipeline\": \n",
    "                                                                            [\n",
    "                                                                              {\"FeaturizationMethodName\": \"filling\", \n",
    "                                                                               \"FeaturizationMethodParameters\": \n",
    "                                                                                {\"frontfill\": \"none\", \n",
    "                                                                                 \"middlefill\": \"zero\", \n",
    "                                                                                 \"backfill\": \"zero\"}\n",
    "                                                                              }\n",
    "                                                                            ]\n",
    "                                                                          }\n",
    "                                                                        ]\n",
    "                                                                       }\n",
    "                                                 )\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DeepAR+"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DeepAR+ Specifics\n",
    "prophet_predictorName= project+'_deeparp_algo_1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build DeepAR+:\n",
    "deeparp_create_predictor_response=forecast.create_predictor(PredictorName=prophet_predictorName, \n",
    "                                                  AlgorithmArn=deepAR_Plus_algorithmArn,\n",
    "                                                  ForecastHorizon=forecastHorizon,\n",
    "                                                  PerformAutoML= False,\n",
    "                                                  PerformHPO=False,\n",
    "                                                  EvaluationParameters= {\"NumberOfBacktestWindows\": NumberOfBacktestWindows, \n",
    "                                                                         \"BackTestWindowOffset\": BackTestWindowOffset}, \n",
    "                                                  InputDataConfig= {\"DatasetGroupArn\": datasetGroupArn, \"SupplementaryFeatures\": [ \n",
    "                                                                     { \n",
    "                                                                        \"Name\": \"holiday\",\n",
    "                                                                        \"Value\": \"US\"\n",
    "                                                                     }\n",
    "                                                                  ]},\n",
    "                                                  FeaturizationConfig= {\"ForecastFrequency\": ForecastFrequency, \n",
    "                                                                        \"Featurizations\": \n",
    "                                                                        [\n",
    "                                                                          {\"AttributeName\": \"target_value\", \n",
    "                                                                           \"FeaturizationPipeline\": \n",
    "                                                                            [\n",
    "                                                                              {\"FeaturizationMethodName\": \"filling\", \n",
    "                                                                               \"FeaturizationMethodParameters\": \n",
    "                                                                                {\"frontfill\": \"none\", \n",
    "                                                                                 \"middlefill\": \"zero\", \n",
    "                                                                                 \"backfill\": \"zero\"}\n",
    "                                                                              }\n",
    "                                                                            ]\n",
    "                                                                          }\n",
    "                                                                        ]\n",
    "                                                                       }\n",
    "                                                 )\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally in our notebooks we would have a while loop that polls for each of these to determine the status of the models in training. For simplicity sake here we are going to rely on you opening a new browser tab and following along in the console until a predictor has been created for each algorithm. \n",
    "\n",
    "Your previous tab from opening this session of Jupyter Lab should still be open, from there navigate to the Amazon Forecast service page, then select your dataset group. Lastly click `Predictors` and you should see the creation in progress. Once they are active you are ready to continue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examining the Predictors\n",
    "\n",
    "Once each of the Predictors is in an `Active` state you can get metrics about it to better understand its accuracy and behavior. These are computed based on the hold out periods we defined when building the Predictor. The metrics are meant to guide our decisions when we use a particular Predictor to generate a forecast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ARIMA\n",
    "\n",
    "ARIMA is one of the gold standards for time series forecasting. This algorithm is not particularly sophisticated but it is reliable and can help us understand a baseline of performance. To note it does not really understand seasonality very well and it does not support any item metadata or related time series information. Due to that we will explore it here but not after adding other datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ARIMA Metrics\n",
    "arima_arn = arima_create_predictor_response['PredictorArn']\n",
    "arima_metrics = forecast.get_accuracy_metrics(PredictorArn=arima_arn)\n",
    "pp = pprint.PrettyPrinter()\n",
    "pp.pprint(arima_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall our RMSE is 2315.573395... This will help you rank your other predictors where you look to see a reduction in RMSE. Overall performance looks like this:\n",
    "\n",
    "| Predictor | RMSE               | 10%                 | 50%                 | 90%                |\n",
    "|-----------|--------------------|---------------------|---------------------|--------------------|\n",
    "| ARIMA     | 2315.5733958045103 | 0.18638100224895027 | 0.35619575117523433 | 0.2906928828133737 |\n",
    "\n",
    "Again these particular values will help us evaluate the other predictors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prophet\n",
    "\n",
    "Same as ARIMA, now you should look at the metrics from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prophet Metrics\n",
    "prophet_arn = prophet_create_predictor_response['PredictorArn']\n",
    "prophet_metrics = forecast.get_accuracy_metrics(PredictorArn=prophet_arn)\n",
    "pp = pprint.PrettyPrinter()\n",
    "pp.pprint(prophet_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you see an RMSE of 2326.90723.... Just a bit higher than ARIMA, but at this point Prophet has not had a chance to use any of its abilities to integrate related time series information. Prophet's performance compared to ARIMA is:\n",
    "\n",
    "| Predictor | RMSE               | 10%                 | 50%                 | 90%                 |\n",
    "|-----------|--------------------|---------------------|---------------------|---------------------|\n",
    "| ARIMA     | 2315.5733958045103 | 0.18638100224895027 | 0.35619575117523433 | 0.2906928828133737  |\n",
    "| Prophet   | 2326.907237203276  | 0.15434241067213655 | 0.3880618345549856  | 0.28403268455977804 |\n",
    "\n",
    "What this tells us is that when querying the 10% quantile we see less of an error from Prophet, also in the 90% but we see a bit worse performance in the 50% quantile. Next will be DeepAR.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DeepAR+\n",
    "\n",
    "Same as Prophet and ARIMA, now you should look at the metrics from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DeepAR+ Metrics\n",
    "deeparp_arn = deeparp_create_predictor_response['PredictorArn']\n",
    "deeparp_metrics = forecast.get_accuracy_metrics(PredictorArn=deeparp_arn)\n",
    "pp = pprint.PrettyPrinter()\n",
    "pp.pprint(deeparp_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you see an RMSE of 2343.7694... Higher than both ARIMA and Prophet, but also lacking in related time series data. Diving a bit more deeply we see:\n",
    "\n",
    "\n",
    "| Predictor | RMSE               | 10%                 | 50%                 | 90%                 |\n",
    "|-----------|--------------------|---------------------|---------------------|---------------------|\n",
    "| ARIMA     | 2315.5733958045103 | 0.18638100224895027 | 0.35619575117523433 | 0.2906928828133737  |\n",
    "| Prophet   | 2326.907237203276  | 0.15434241067213655 | 0.3880618345549856  | 0.28403268455977804 |\n",
    "| DeepAR+   | 2343.76942963439   | 0.09470210806720818 | 0.27579575099681214 | 0.32289195793540876 |\n",
    "\n",
    "\n",
    "We are now seeing major improvements in accuracy for the 10 and 50% quantiles with a bit worse performance on the 90%. To explore what this all looks like in a visual format we will now create a Forecast with each Predictor and then export it to s3 where we can download and explore the results.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating and Exporting Forecasts\n",
    "\n",
    "Inside Amazon Forecast a Forecast is a rendered collection of all of your items, at every time interval, for all selected quantiles, for your given forecast horizon. This process takes the Predictor you just created and uses it to generate these inferences and to store them in a useful state. Once a Forecast exists within the service you can query it and obtain a JSON response or use another API call to export it to a CSV that is stored in S3. \n",
    "\n",
    "This tutorial will focus on the S3 Export as that is often an easy way to manually explore the data with many tools.\n",
    "\n",
    "These again will take some time to complete after you have executed the cells so explore the console to see when they have completed.\n",
    "\n",
    "To do that visit the Amazon Forecast Service page, then clck your Dataset Group, and then click `Forecasts` on the left. They will say `Create in progress...` initially and then `Active` when ready for export."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ARIMA\n",
    "arima_forecastName = project+'_arima_algo_forecast'\n",
    "arima_create_forecast_response=forecast.create_forecast(ForecastName=arima_forecastName,\n",
    "                                                  PredictorArn=arima_arn)\n",
    "arima_forecast_arn = arima_create_forecast_response['ForecastArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prophet\n",
    "prophet_forecastName = project+'_prophet_algo_forecast'\n",
    "prophet_create_forecast_response=forecast.create_forecast(ForecastName=prophet_forecastName,\n",
    "                                                  PredictorArn=prophet_arn)\n",
    "prophet_forecast_arn = prophet_create_forecast_response['ForecastArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DeepAR+\n",
    "deeparp_forecastName = project+'_deeparp_algo_forecast'\n",
    "deeparp_create_forecast_response=forecast.create_forecast(ForecastName=deeparp_forecastName,\n",
    "                                                  PredictorArn=deeparp_arn)\n",
    "deeparp_forecast_arn = deeparp_create_forecast_response['ForecastArn']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once they are `Active` you can start the export process. The code to do so is in the cells below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arima_path = \"s3://\" + bucket_name + \"/arima_1/\"\n",
    "arima_job_name = \"ArimaExport1\"\n",
    "forecast.create_forecast_export_job(ForecastExportJobName=arima_job_name,\n",
    "                                                        ForecastArn=arima_forecast_arn,\n",
    "                                                        Destination={\n",
    "                                                            \"S3Config\" : {\n",
    "                                                                \"Path\": arima_path,\n",
    "                                                                \"RoleArn\": role_arn\n",
    "                                                            }\n",
    "                                                        })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prophet_path = \"s3://\" + bucket_name + \"/prophet_1/\"\n",
    "prophet_job_name = \"ProphetExport1\"\n",
    "forecast.create_forecast_export_job(ForecastExportJobName=prophet_job_name,\n",
    "                                                        ForecastArn=prophet_forecast_arn,\n",
    "                                                        Destination={\n",
    "                                                            \"S3Config\" : {\n",
    "                                                                \"Path\": prophet_path,\n",
    "                                                                \"RoleArn\": role_arn\n",
    "                                                            }\n",
    "                                                        })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deeparp_path = \"s3://\" + bucket_name + \"/deeparp_1/\"\n",
    "deeparp_job_name = \"DeepARPExport1\"\n",
    "forecast.create_forecast_export_job(ForecastExportJobName=deeparp_job_name,\n",
    "                                                        ForecastArn=deeparp_forecast_arn,\n",
    "                                                        Destination={\n",
    "                                                            \"S3Config\" : {\n",
    "                                                                \"Path\": deeparp_path,\n",
    "                                                                \"RoleArn\": role_arn\n",
    "                                                            }\n",
    "                                                        })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This exporting process is another one of those items that will take several minutes to complete. Just poll for progress in the console. From the earlier page where you saw the status turn `Active` for a Forecast, click it and you can see the progress of the export."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Obtaining the Forecasts\n",
    "\n",
    "At this point they are all exported into S3 but you need to obtain the results locally so we can explore them, the cells below will do that starting with ARIMA, then Prophet, and lastly DeepAR+."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Arima\n",
    "s3 = boto3.resource('s3')\n",
    "poc_bucket = boto3.resource('s3').Bucket(bucket_name)\n",
    "arima_filename = \"\"\n",
    "arima_files = list(poc_bucket.objects.filter(Prefix=\"arima_1\"))\n",
    "for file in arima_files:\n",
    "    # There will be a collection of CSVs if the forecast is large, modify this to go get them all\n",
    "    if \"csv\" in file.key:\n",
    "        arima_filename = file.key.split('/')[1]\n",
    "        s3.Bucket(bucket_name).download_file(file.key, data_dir+\"/\"+arima_filename)\n",
    "print(arima_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prophet\n",
    "s3 = boto3.resource('s3')\n",
    "poc_bucket = boto3.resource('s3').Bucket(bucket_name)\n",
    "prophet_filename = \"\"\n",
    "prophet_files = list(poc_bucket.objects.filter(Prefix=\"prophet_1\"))\n",
    "for file in prophet_files:\n",
    "    # There will be a collection of CSVs if the forecast is large, modify this to go get them all\n",
    "    if \"csv\" in file.key:\n",
    "        prophet_filename = file.key.split('/')[1]\n",
    "        s3.Bucket(bucket_name).download_file(file.key, data_dir+\"/\"+prophet_filename)\n",
    "print(prophet_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DeepAR+\n",
    "s3 = boto3.resource('s3')\n",
    "poc_bucket = boto3.resource('s3').Bucket(bucket_name)\n",
    "deeparp_filename = \"\"\n",
    "deeparp_files = list(poc_bucket.objects.filter(Prefix=\"deeparp_1\"))\n",
    "for file in deeparp_files:\n",
    "    # There will be a collection of CSVs if the forecast is large, modify this to go get them all\n",
    "    if \"csv\" in file.key:\n",
    "        deeparp_filename = file.key.split('/')[1]\n",
    "        s3.Bucket(bucket_name).download_file(file.key, data_dir+\"/\"+deeparp_filename)\n",
    "print(deeparp_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ARIMA Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ARIMA Eval\n",
    "arima_predicts = pd.read_csv(data_dir+\"/\"+arima_filename)\n",
    "arima_predicts.sample()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arima_predicts.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the column to datetime\n",
    "arima_predicts['date'] = pd.to_datetime(arima_predicts['date'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arima_predicts.sample()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the timezone and make date the index\n",
    "arima_predicts['date'] = arima_predicts['date'].dt.tz_convert(None)\n",
    "arima_predicts.set_index('date', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arima_predicts.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (arima_predicts.index.min())\n",
    "print (arima_predicts.index.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see our prediction goes from Jan 01 to Jan 20 as expectged given our 480 interval forecast horizon. Also we can see the cyclical nature of the predictions over. the entire timeframe. \n",
    "\n",
    "Now we are going to create a dataframe of the prediction values from this Forecast and the actual values.\n",
    "\n",
    "First let us remove the column ID of item before continuing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arima_predicts = arima_predicts[['p10', 'p50', 'p90']]\n",
    "arima_predicts.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets slice validation to meet our needs\n",
    "validation_df = validation_time_series_df.copy()\n",
    "validation_df = validation_df.loc['2018-01-01':'2018-01-20']\n",
    "print (validation_df.index.min())\n",
    "print (validation_df.index.max())\n",
    "validation_df.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finally let us join the dataframes together\n",
    "arima_val_df = arima_predicts.join(validation_df, how='outer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot\n",
    "arima_val_df.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that this particular plot is hard to see, let us pick a random day January 5th to compare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arima_val_df_jan_5 = arima_val_df.loc['2018-01-05':'2018-01-06']\n",
    "arima_val_df_jan_5.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now this is pretty clear for p50 showcasing that it does a great job of predicting the volume. Let us now do this for Prophet and DeepAR+."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prophet Validation\n",
    "\n",
    "We will speed up the prep work to just a few cells this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prophet Eval\n",
    "prophet_predicts = pd.read_csv(data_dir+\"/\"+prophet_filename)\n",
    "prophet_predicts.sample()\n",
    "# Remove the timezone\n",
    "prophet_predicts['date'] = pd.to_datetime(prophet_predicts['date'])\n",
    "prophet_predicts['date'] = prophet_predicts['date'].dt.tz_convert(None)\n",
    "prophet_predicts.set_index('date', inplace=True)\n",
    "prophet_predicts = prophet_predicts[['p10', 'p50', 'p90']]\n",
    "# Finally let us join the dataframes together\n",
    "prophet_val_df = prophet_predicts.join(validation_df, how='outer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot\n",
    "prophet_val_df.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prophet_val_df_jan_5 = prophet_val_df.loc['2018-01-05':'2018-01-06']\n",
    "prophet_val_df_jan_5.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DeepAR+ Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DeepAR+ Eval\n",
    "deeparp_predicts = pd.read_csv(data_dir+\"/\"+deeparp_filename)\n",
    "deeparp_predicts.sample()\n",
    "# Remove the timezone\n",
    "deeparp_predicts['date'] = pd.to_datetime(deeparp_predicts['date'])\n",
    "deeparp_predicts['date'] = deeparp_predicts['date'].dt.tz_convert(None)\n",
    "deeparp_predicts.set_index('date', inplace=True)\n",
    "deeparp_predicts = deeparp_predicts[['p10', 'p50', 'p90']]\n",
    "# Finally let us join the dataframes together\n",
    "deeparp_val_df = deeparp_predicts.join(validation_df, how='outer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deeparp_val_df_jan_5 = deeparp_val_df.loc['2018-01-05':'2018-01-06']\n",
    "deeparp_val_df_jan_5.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is particularly interesting here is that we were below the actual numbers for a good portion of the day even with p90. We did see great performance from Prophet and the metrics indicate that DeepAR+ is objectively better here so now we will add related time series data to our project and see how the models behave then."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recap and Next Steps\n",
    "\n",
    "At this point we can now see through the 3 plots below that DeepAR+ does a really good job outside of the high ranges, and that perhaps adding related data could improve both Prophet and DeepAR+'s performance. The next thing to do is to move to the notebook for importing your related-time series data and then progress to the second Creating and Evaluating notebook that will explain how to leverage the related data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}