Validating_and_Importing_Target_Time_Series_Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Validating and Importing Target Time Series Data\n",
    "\n",
    "## Obtaining Your Data\n",
    "\n",
    "A critical requirement to use Amazon Forecast is to have access to time-series data for your selected use case. To learn more about time series data:\n",
    "\n",
    "1. [Wikipedia](https://en.wikipedia.org/wiki/Time_series)\n",
    "1. [Toward's Data Science Primer](https://towardsdatascience.com/the-complete-guide-to-time-series-analysis-and-forecasting-70d476bfe775)\n",
    "1. [O'Reilly Book](https://www.amazon.com/gp/product/1492041653/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1)\n",
    "\n",
    "As an exmaple for this POC guide we are going to select a dataset from the UCI repository of machine learning datasets. This is a great tool for finding datasets for various problems. In this particular case it is traffic data for a given section of interstate highway. More information on the dataset can be found [here](https://archive.ics.uci.edu/ml/datasets/Metro+Interstate+Traffic+Volume)\n",
    "\n",
    "Your specific data my come from a DB export, an existing spreadsheet, it really does not matter the source. Going forward the files should be uploaded into this notebook and stored in CSV format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To begin the cell below will complete the following:\n",
    "\n",
    "1. Create a directory for the data files.\n",
    "1. Download the sample data into the directory.\n",
    "1. Extract the archive file into the directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = \"data\"\n",
    "!mkdir $data_dir\n",
    "!cd $data_dir && wget https://archive.ics.uci.edu/ml/machine-learning-databases/00492/Metro_Interstate_Traffic_Volume.csv.gz\n",
    "!gunzip $data_dir/Metro_Interstate_Traffic_Volume.csv.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the data downloaded, now we will import the Pandas library as well as a few other data science tools in order to inspect the information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from time import sleep\n",
    "import subprocess\n",
    "import pandas as pd\n",
    "import json\n",
    "import time\n",
    "import pprint\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.dates import DateFormatter\n",
    "import matplotlib.dates as mdates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next open the file with Pandas and take a look at the contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_data = pd.read_csv(data_dir + '/Metro_Interstate_Traffic_Volume.csv')\n",
    "original_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point we can see a few things about the data:\n",
    "\n",
    "* Holidays seem to be specified\n",
    "* There is a value for temp, rainfall, snowfall, and a few other weather metrics.\n",
    "* The time series is hourly\n",
    "* Our value to predict is `traffic_volume down` at the end.\n",
    "\n",
    "Amazon Forecast relies on a concept called the target-time-series in order to start making predictions, this has a timestamp, an item identifier, and a value. The timestamp is pretty self explanatory, and the value to predict will be traffic_volume, given this is a singular time series an arbitrary item_ID of `1` will be applied later to all entries in the time series file.\n",
    "\n",
    "The other attributes provided can serve as a basis for related time series components when we get to that much later.\n",
    "\n",
    "Amazon Forecast also works well to fill in gaps for the target-time-series but not the related data, so before we input our data and get a prediction we should look to see where gaps are, and how we want to structure both inputs to address this issue. \n",
    "\n",
    "To get started we will manipulate our starting dataframe to determine the quality and consistency of this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_df = original_data.copy()\n",
    "target_df.plot()\n",
    "print(\"Start Date: \", min(target_df['date_time']))\n",
    "print(\"End Date: \", max(target_df['date_time']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly at this point we do not see an obvious gaps in this plot, but we should still check a bit deeper to confirm this. The next cell gives some basic information on the dataset size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell above we now see a range of October 2012 to nearly October 2018, almost 6 years of hourly data. Given there are around 8700 hours in a year we expect to see 52,000 time series. Immediately here we see 48,204. It looks like some data points are missing, next let us define the index, drop the duplicates and see where we are then."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_df.set_index('date_time', inplace=True)\n",
    "target_df = target_df.drop_duplicates(keep ='first')\n",
    "target_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That change dropped us to 48,175 unique entries. Given this is traffic data we could be dealing with a missing sensor, construction causing outages, or even severe weather delay damaging the recording equipment. Before we decide on how to fill any gaps, let us first take a look to see where they are, and how large the gaps themselves may be.\n",
    "\n",
    "We will do this by creating a new dataframe for the entire length of the dataset, that has no missing entries, then joining our data to it, and padding out 0's where anything is missing.\n",
    "\n",
    "*Note* the periods value below is the total number of entriess to make, I cheated and used WolframAlpha to sort out the number of days: https://www.wolframalpha.com/input/?i=days+from+2012-10-02+to+2018-09-30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_days = 2190\n",
    "# Build the index first\n",
    "idx = pd.date_range(start='10/02/2012', end='09/30/2018', freq='H')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_df = pd.DataFrame(index=idx)\n",
    "full_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (full_df.index.min())\n",
    "print (full_df.index.max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now perform the join\n",
    "full_historical_df = full_df.join(target_df, how='outer')\n",
    "print (full_historical_df.index.min())\n",
    "print (full_historical_df.index.max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take a look at 10 random entries\n",
    "full_historical_df.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample may or may not have shown values with NaNs or other nulls, in this instance it did but we will still want to look for these NaN entities to confirm if they exist and where they are.\n",
    "\n",
    "At this point we have done enough work to see where we may have any large portions of missing data. To that end we can plot the data below and see any gaps that may crop up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_historical_df.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows a large gap of missing data from late 2014 until mid 2016. If we just wanted to feed in the previously known value this may give us too long of a timeframe of data that is simply not relexive to the problem. \n",
    "\n",
    "Before making any decisions we will now step through each year and see what the gaps look like starting in 2013 as it is the first full year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2013 = full_historical_df.loc['2013-01-01':'2013-12-31']\n",
    "print (df_2013.index.min())\n",
    "print (df_2013.index.max())\n",
    "df_2013.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2014 = full_historical_df.loc['2014-01-01':'2014-12-31']\n",
    "print (df_2014.index.min())\n",
    "print (df_2014.index.max())\n",
    "df_2014.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2015 = full_historical_df.loc['2015-01-01':'2015-12-31']\n",
    "print (df_2015.index.min())\n",
    "print (df_2015.index.max())\n",
    "df_2015.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2016 = full_historical_df.loc['2016-01-01':'2016-12-31']\n",
    "print (df_2016.index.min())\n",
    "print (df_2016.index.max())\n",
    "df_2016.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2017 = full_historical_df.loc['2017-01-01':'2017-12-31']\n",
    "print (df_2017.index.min())\n",
    "print (df_2017.index.max())\n",
    "df_2017.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2018 = full_historical_df.loc['2018-01-01':'2018-12-31']\n",
    "print (df_2018.index.min())\n",
    "print (df_2018.index.max())\n",
    "df_2018.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few things to note here, clearly we are missing a large volume of data in 2014 and 2015 but also there are some missing patches in 2013 as well. 2016 had spotty data initially but 2017 and 2018 look pretty good.\n",
    "\n",
    "Given that the data is hourly we still have plenty of it within a single year, and an additional 10 months to use for broader validation if we choose to do that. \n",
    "\n",
    "To note, it seems approaches like DeepAR+ and Prophet work very well with > 1k measurements on a given time series. Assuming hourly data (24 measurements per day), that yields around 42 days before we have a solid base of data. Learning over an entire year should be plenty.\n",
    "\n",
    "Also we need to think about a Forecast horizon or how far into the future we are going to predict at once. Forecast currently limits us to 500 intervals of whatever granularity we have selected. For this exercise we will keep the data hourly and predict 480 hours into the future, or exactly 20 days."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Data Files\n",
    "\n",
    "Knowing that our above dataframe `full_historical_df` covers the entire time period we care about we start there reducing it to 2017 to end. Then we will use feed forward to plug in any missing holes before splitting into the 3 files described before. \n",
    "\n",
    "More info on techniques to patch missing information can be found here: https://pandas.pydata.org/pandas-docs/version/0.22/generated/pandas.DataFrame.fillna.html \n",
    "\n",
    "The risk of filling in values like this is that in smoothing out the data it may cause our predictions to resemble the smoother curve than is our historical data. This is why we selected 2017 to 2018 based on the lack of large gaps in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a copy\n",
    "target_df = full_historical_df.copy()\n",
    "# Slice to only 2017 onward\n",
    "target_df = target_df.loc['2017-01-01':]\n",
    "# Validate the dates\n",
    "print (target_df.index.min())\n",
    "print (target_df.index.max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill in any missing data with the method ffill\n",
    "targtet_df.ffill()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point we have all the data needed to make our target time series file and dataset. While we are doing this we will also make a validation file for later use as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building The Target Time Series File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_time_series_df = target_df.copy()\n",
    "target_time_series_df = target_time_series_df.loc['2017-01-01':'2017-12-31']\n",
    "# Validate the date range\n",
    "print (target_time_series_df.index.min())\n",
    "print (target_time_series_df.index.max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Restrict the columns to timestamp, traffic_volume\n",
    "target_time_series_df = target_time_series_df[['traffic_volume']]\n",
    "# Add in item_id\n",
    "target_time_series_df['item_ID'] = \"1\"\n",
    "# Validate the structure\n",
    "target_time_series_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# With the data in a great state, save it off as a CSV\n",
    "target_time_series_filename = \"target_time_series.csv\"\n",
    "target_time_series_path = data_dir + \"/\" + target_time_series_filename\n",
    "target_time_series_df.to_csv(target_time_series_path, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building The Validation File\n",
    "\n",
    "This is the last file we need to build before getting started with Forecast itself. This will be the same in structure as our target-time-series file but will only project into 2018 and includes no historical data from the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_time_series_df = targtet_df.copy()\n",
    "validation_time_series_df = validation_time_series_df.loc['2018-01-01':]\n",
    "# Validate the date range\n",
    "print (validation_time_series_df.index.min())\n",
    "print (validation_time_series_df.index.max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Restrict the columns to timestamp, traffic_volume\n",
    "validation_time_series_df = validation_time_series_df[['traffic_volume']]\n",
    "# Add in item_id\n",
    "validation_time_series_df['item_ID'] = \"1\"\n",
    "# Validate the structure\n",
    "validation_time_series_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# With the data in a great state, save it off as a CSV\n",
    "validation_time_series_filename = \"validation_time_series.csv\"\n",
    "validation_time_series_path = data_dir + \"/\" + validation_time_series_filename\n",
    "validation_time_series_df.to_csv(validation_time_series_path, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started With Forecast\n",
    "\n",
    "Now that all of the required data to get started exists, our next step is to build the dataset groups and datasets required for our problem. Inside Amazon Forecast a DatasetGroup is an abstraction that contains all the datasets for a particular collection of Forecasts. There is no information sharing between DatasetGroups so if you'd like to try out various alternatives to the schemas we create below, you could create a new DatasetGroup and make your changes inside its corresponding Datasets.\n",
    "\n",
    "The order of the process below will be as follows:\n",
    "\n",
    "1. Create a DatasetGroup for our POC.\n",
    "1. Create a `Target-Time-Series` Dataset.\n",
    "1. Attach the Dataset to the DatasetGroup.\n",
    "1. Import the data into the Dataset.\n",
    "\n",
    "Later you can use the other notebooks to build Predictors based off this information, or to add related time series data as well.\n",
    "\n",
    "The cell immediately below defines a few core aspects of our Dataset Group and info on our data. For example the timestamp format, the project name, and how frequent our time series data is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET_FREQUENCY = \"H\" \n",
    "TIMESTAMP_FORMAT = \"yyyy-MM-dd hh:mm:ss\"\n",
    "\n",
    "project = 'forecast_poc'\n",
    "datasetName= project+'_ds'\n",
    "datasetGroupName= project +'_dsg'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now using the metada stored on this instance of a SageMaker Notebook determine the region we are operating in. If you are using a Jupyter Notebook outside of SageMaker simply define `region` as the string that indicates the region you would like to use for Forecast and S3.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('/opt/ml/metadata/resource-metadata.json') as notebook_info:\n",
    "    data = json.load(notebook_info)\n",
    "    resource_arn = data['ResourceArn']\n",
    "    region = resource_arn.split(':')[3]\n",
    "print(region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Configure your AWS APIs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = boto3.Session(region_name=region)\n",
    "forecast = session.client(service_name='forecast')\n",
    "forecast_query = session.client(service_name='forecastquery')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the Dataset Group, this is the largest abstraction when using Forecast. There is no information sharing between Dataset Groups so if you want to try out new schemas, or completely different datasets for a problem this is a great isolation layer to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the DatasetGroup\n",
    "create_dataset_group_response = forecast.create_dataset_group(DatasetGroupName=datasetGroupName,\n",
    "                                                              Domain=\"CUSTOM\",\n",
    "                                                             )\n",
    "datasetGroupArn = create_dataset_group_response['DatasetGroupArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_dataset_group(DatasetGroupArn=datasetGroupArn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming you made no initial schema changes, the cell below should just be fine. If you have made any alterations, update the cell accordingly then execute it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify the schema of your dataset here. Make sure the order of columns matches the raw data files.\n",
    "schema ={\n",
    "   \"Attributes\":[\n",
    "      {\n",
    "         \"AttributeName\":\"timestamp\",\n",
    "         \"AttributeType\":\"timestamp\"\n",
    "      },\n",
    "      {\n",
    "         \"AttributeName\":\"target_value\",\n",
    "         \"AttributeType\":\"float\"\n",
    "      },\n",
    "      {\n",
    "         \"AttributeName\":\"item_id\",\n",
    "         \"AttributeType\":\"string\"\n",
    "      }\n",
    "   ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inside every DatasetGroup you can have 3 types of additional data:\n",
    "\n",
    "1. Target Time Series\n",
    "1. Related Time Series\n",
    "1. Item Metadata\n",
    "\n",
    "In this guide we are really only focusing on the target-time-series bit. The cells below will create this container for you and then add it to your DatasetGroup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response=forecast.create_dataset(\n",
    "                    Domain=\"CUSTOM\",\n",
    "                    DatasetType='TARGET_TIME_SERIES',\n",
    "                    DatasetName=datasetName,\n",
    "                    DataFrequency=DATASET_FREQUENCY, \n",
    "                    Schema = schema\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_datasetArn = response['DatasetArn']\n",
    "forecast.describe_dataset(DatasetArn=target_datasetArn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Attach the Dataset to the Dataset Group:\n",
    "forecast.update_dataset_group(DatasetGroupArn=datasetGroupArn, DatasetArns=[target_datasetArn])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also need a Role to interact with S3 and Forecast on our behalf going forward. This cell creates that role. Note that it does sleep for 60 seconds to ensure that the process has completed and all permissions have propagated before going forward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iam = boto3.client(\"iam\")\n",
    "\n",
    "role_name = \"ForecastRolePOC\"\n",
    "assume_role_policy_document = {\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "          \"Effect\": \"Allow\",\n",
    "          \"Principal\": {\n",
    "            \"Service\": \"forecast.amazonaws.com\"\n",
    "          },\n",
    "          \"Action\": \"sts:AssumeRole\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "create_role_response = iam.create_role(\n",
    "    RoleName = role_name,\n",
    "    AssumeRolePolicyDocument = json.dumps(assume_role_policy_document)\n",
    ")\n",
    "\n",
    "# AmazonPersonalizeFullAccess provides access to any S3 bucket with a name that includes \"personalize\" or \"Personalize\" \n",
    "# if you would like to use a bucket with a different name, please consider creating and attaching a new policy\n",
    "# that provides read access to your bucket or attaching the AmazonS3ReadOnlyAccess policy to the role\n",
    "policy_arn = \"arn:aws:iam::aws:policy/AmazonForecastFullAccess\"\n",
    "iam.attach_role_policy(\n",
    "    RoleName = role_name,\n",
    "    PolicyArn = policy_arn\n",
    ")\n",
    "\n",
    "# Now add S3 support\n",
    "iam.attach_role_policy(\n",
    "    PolicyArn='arn:aws:iam::aws:policy/AmazonS3FullAccess',\n",
    "    RoleName=role_name\n",
    ")\n",
    "time.sleep(60) # wait for a minute to allow IAM role policy attachment to propagate\n",
    "\n",
    "role_arn = create_role_response[\"Role\"][\"Arn\"]\n",
    "print(role_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point thie next thing to do is import a file into Amazon Forecast, however we do not yet have anything in S3, so we will create a bucket, and upload our target file there. Note this is only the target file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(region)\n",
    "s3 = boto3.client('s3')\n",
    "account_id = boto3.client('sts').get_caller_identity().get('Account')\n",
    "bucket_name = account_id + \"forecastpoc\"\n",
    "print(bucket_name)\n",
    "if region != \"us-east-1\":\n",
    "    s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={'LocationConstraint': region})\n",
    "else:\n",
    "    s3.create_bucket(Bucket=bucket_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload Target File\n",
    "boto3.Session().resource('s3').Bucket(bucket_name).Object(target_time_series_filename).upload_file(target_time_series_path)\n",
    "target_s3DataPath = \"s3://\"+bucket_name+\"/\"+target_time_series_filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point your data is now formatted correctly for Forecast and exists within S3, the last thing to do is to import it so you can get started actually generating models!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finally we can call import the dataset\n",
    "datasetImportJobName = 'DSIMPORT_JOB_TARGET_POC'\n",
    "ds_import_job_response=forecast.create_dataset_import_job(DatasetImportJobName=datasetImportJobName,\n",
    "                                                          DatasetArn=target_datasetArn,\n",
    "                                                          DataSource= {\n",
    "                                                              \"S3Config\" : {\n",
    "                                                                 \"Path\":target_s3DataPath,\n",
    "                                                                 \"RoleArn\": role_arn\n",
    "                                                              } \n",
    "                                                          },\n",
    "                                                          TimestampFormat=TIMESTAMP_FORMAT\n",
    "                                                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_import_job_arn=ds_import_job_response['DatasetImportJobArn']\n",
    "print(ds_import_job_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell below will run and poll every 30 seconds until the import process has completed. From there we will be able to view the metrics on the data and see that it is valid and ready for use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "while True:\n",
    "    dataImportStatus = forecast.describe_dataset_import_job(DatasetImportJobArn=ds_import_job_arn)['Status']\n",
    "    print(dataImportStatus)\n",
    "    if dataImportStatus != 'ACTIVE' and dataImportStatus != 'CREATE_FAILED':\n",
    "        sleep(30)\n",
    "    else:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the import shows a state of `ACTIVE` we are then ready to evaluate the data that exists within the system and call the importing process complete.\n",
    "\n",
    "## Evaluating the Target Time Series Data\n",
    "\n",
    "First let us take a look at the information provided in our target time series file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Validate the date range\n",
    "print (target_time_series_df.index.min())\n",
    "print (target_time_series_df.index.max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take a look at high level metrics:\n",
    "target_time_series_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are exactly 10,642 entries in this file with no null values at all. Let us now look at the metrics from the import."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_dataset_import_job(DatasetImportJobArn=ds_import_job_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At long last we see the same metrics from our import that we saw from our dataframe. From here we can now consider our work on target-time-series done. \n",
    "\n",
    "If you are running the POC process this is a great time to now explore sorting out the related data bits as well. If you are not you can move on to just building Predictors.\n",
    "\n",
    "The final cell below will use the store function of jupyter to save off a few variables for use in other notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store full_historical_df\n",
    "%store target_time_series_df\n",
    "%store validation_time_series_df\n",
    "%store datasetName\n",
    "%store bucket_name\n",
    "%store datasetGroupName\n",
    "%store datasetGroupArn\n",
    "%store target_datasetArn\n",
    "%store role_arn\n",
    "%store region\n",
    "%store target_time_series_filename\n",
    "%store targtet_df\n",
    "%store full_df\n",
    "%store data_dir\n",
    "%store DATASET_FREQUENCY\n",
    "%store TIMESTAMP_FORMAT\n",
    "%store project\n",
    "%store data_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}