next-purchase-date-with-deep-learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fc98867c",
   "metadata": {
    "papermill": {
     "duration": 0.008205,
     "end_time": "2024-03-08T08:43:09.747109",
     "exception": false,
     "start_time": "2024-03-08T08:43:09.738904",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Next Purchase Date with DL"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79284433",
   "metadata": {
    "papermill": {
     "duration": 0.007494,
     "end_time": "2024-03-08T08:43:09.762376",
     "exception": false,
     "start_time": "2024-03-08T08:43:09.754882",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this use case, we use deep learning approach to know the model's performance if trained with the routine transactions between the user and the item. We assume that routine transactions are when a user buys a certain item at least once a month. To avoid overfitting for the too-frequent transactions (e.g., buying an item every day), we choose two interactions of user and item with the biggest average interval of transactions.\n",
    "\n",
    "We also perform model training by some items with the highest number of transactions. Hopefully, we can achieve the best score because using the largest number of transactions compared to the others will make the model learn and perform better."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd3948fc",
   "metadata": {
    "papermill": {
     "duration": 0.007658,
     "end_time": "2024-03-08T08:43:41.068496",
     "exception": false,
     "start_time": "2024-03-08T08:43:41.060838",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Load and Preprocess Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcef3041",
   "metadata": {
    "papermill": {
     "duration": 0.007282,
     "end_time": "2024-03-08T08:43:41.083381",
     "exception": false,
     "start_time": "2024-03-08T08:43:41.076099",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We use more than 500k transaction data between users and items from the EPM database. The raw data still has some returning transactions with a negative amount, but we are only looking for buying transactions. Each transaction has a timestamp record daily. Because a user can buy the same item multiple times on the same day, we consider it a single data aggregating the sales quantity column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "650bc4c4",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:41.100608Z",
     "iopub.status.busy": "2024-03-08T08:43:41.100182Z",
     "iopub.status.idle": "2024-03-08T08:43:47.905012Z",
     "shell.execute_reply": "2024-03-08T08:43:47.903831Z"
    },
    "papermill": {
     "duration": 6.816893,
     "end_time": "2024-03-08T08:43:47.907879",
     "exception": false,
     "start_time": "2024-03-08T08:43:41.090986",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import time\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
    "from sklearn.preprocessing import MinMaxScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "afcda072",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:47.925509Z",
     "iopub.status.busy": "2024-03-08T08:43:47.924941Z",
     "iopub.status.idle": "2024-03-08T08:43:49.986312Z",
     "shell.execute_reply": "2024-03-08T08:43:49.985153Z"
    },
    "papermill": {
     "duration": 2.073271,
     "end_time": "2024-03-08T08:43:49.988929",
     "exception": false,
     "start_time": "2024-03-08T08:43:47.915658",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trx_date</th>\n",
       "      <th>customer_name</th>\n",
       "      <th>ship_to_id</th>\n",
       "      <th>branch_code</th>\n",
       "      <th>item_code</th>\n",
       "      <th>item_desc</th>\n",
       "      <th>principal_desc</th>\n",
       "      <th>gross_sales_amount</th>\n",
       "      <th>sales_qty</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-08-20</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>KCFMB</td>\n",
       "      <td>CEFIXIME 100MG 50 KAPSUL</td>\n",
       "      <td>HEXPHARM (PHARMAMED)</td>\n",
       "      <td>195000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-11-02</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>CKCOA</td>\n",
       "      <td>KALCINOL N CREAM  5 GR</td>\n",
       "      <td>KALBE NIMITZ (PHARMAMED)</td>\n",
       "      <td>28500</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-11-02</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>TMFNB</td>\n",
       "      <td>METFORMIN HCL 200 TABLET</td>\n",
       "      <td>HEXPHARM (PHARMAMED)</td>\n",
       "      <td>175000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-11-02</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>TBSVC</td>\n",
       "      <td>BRONSOLVAN 100 TABLET</td>\n",
       "      <td>HEXPHARM TSJ (PHARMAMED)</td>\n",
       "      <td>35000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-08-20</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>TALNF</td>\n",
       "      <td>AMLODIPINE BESILATE 10MG</td>\n",
       "      <td>HEXPHARM (PHARMAMED)</td>\n",
       "      <td>255000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     trx_date            customer_name ship_to_id branch_code item_code  \\\n",
       "0  2021-08-20  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     KCFMB   \n",
       "1  2021-11-02  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     CKCOA   \n",
       "2  2021-11-02  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     TMFNB   \n",
       "3  2021-11-02  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     TBSVC   \n",
       "4  2021-08-20  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     TALNF   \n",
       "\n",
       "                  item_desc            principal_desc  gross_sales_amount  \\\n",
       "0  CEFIXIME 100MG 50 KAPSUL      HEXPHARM (PHARMAMED)              195000   \n",
       "1    KALCINOL N CREAM  5 GR  KALBE NIMITZ (PHARMAMED)               28500   \n",
       "2  METFORMIN HCL 200 TABLET      HEXPHARM (PHARMAMED)              175000   \n",
       "3     BRONSOLVAN 100 TABLET  HEXPHARM TSJ (PHARMAMED)               35000   \n",
       "4  AMLODIPINE BESILATE 10MG      HEXPHARM (PHARMAMED)              255000   \n",
       "\n",
       "   sales_qty  \n",
       "0          3  \n",
       "1          3  \n",
       "2          5  \n",
       "3          1  \n",
       "4          3  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load Dataset\n",
    "df = pd.read_csv('/kaggle/input/epm-prep/EPM.csv')\n",
    "df = df.drop(['Unnamed: 0', 'principal_code'], axis=1)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c06e087e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:50.007028Z",
     "iopub.status.busy": "2024-03-08T08:43:50.006611Z",
     "iopub.status.idle": "2024-03-08T08:43:50.218336Z",
     "shell.execute_reply": "2024-03-08T08:43:50.217111Z"
    },
    "papermill": {
     "duration": 0.223755,
     "end_time": "2024-03-08T08:43:50.220770",
     "exception": false,
     "start_time": "2024-03-08T08:43:49.997015",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trx_date</th>\n",
       "      <th>customer_name</th>\n",
       "      <th>ship_to_id</th>\n",
       "      <th>branch_code</th>\n",
       "      <th>item_code</th>\n",
       "      <th>item_desc</th>\n",
       "      <th>principal_desc</th>\n",
       "      <th>gross_sales_amount</th>\n",
       "      <th>sales_qty</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-08-20</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>KCFMB</td>\n",
       "      <td>CEFIXIME 100MG 50 KAPSUL</td>\n",
       "      <td>HEXPHARM (PHARMAMED)</td>\n",
       "      <td>195000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-11-02</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>CKCOA</td>\n",
       "      <td>KALCINOL N CREAM  5 GR</td>\n",
       "      <td>KALBE NIMITZ (PHARMAMED)</td>\n",
       "      <td>28500</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-11-02</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>TMFNB</td>\n",
       "      <td>METFORMIN HCL 200 TABLET</td>\n",
       "      <td>HEXPHARM (PHARMAMED)</td>\n",
       "      <td>175000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-11-02</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>TBSVC</td>\n",
       "      <td>BRONSOLVAN 100 TABLET</td>\n",
       "      <td>HEXPHARM TSJ (PHARMAMED)</td>\n",
       "      <td>35000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-08-20</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>TALNF</td>\n",
       "      <td>AMLODIPINE BESILATE 10MG</td>\n",
       "      <td>HEXPHARM (PHARMAMED)</td>\n",
       "      <td>255000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    trx_date            customer_name ship_to_id branch_code item_code  \\\n",
       "0 2021-08-20  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     KCFMB   \n",
       "1 2021-11-02  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     CKCOA   \n",
       "2 2021-11-02  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     TMFNB   \n",
       "3 2021-11-02  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     TBSVC   \n",
       "4 2021-08-20  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     TALNF   \n",
       "\n",
       "                  item_desc            principal_desc  gross_sales_amount  \\\n",
       "0  CEFIXIME 100MG 50 KAPSUL      HEXPHARM (PHARMAMED)              195000   \n",
       "1    KALCINOL N CREAM  5 GR  KALBE NIMITZ (PHARMAMED)               28500   \n",
       "2  METFORMIN HCL 200 TABLET      HEXPHARM (PHARMAMED)              175000   \n",
       "3     BRONSOLVAN 100 TABLET  HEXPHARM TSJ (PHARMAMED)               35000   \n",
       "4  AMLODIPINE BESILATE 10MG      HEXPHARM (PHARMAMED)              255000   \n",
       "\n",
       "   sales_qty  \n",
       "0          3  \n",
       "1          3  \n",
       "2          5  \n",
       "3          1  \n",
       "4          3  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Filter negative transactions\n",
    "df = df[(df['sales_qty'] > 0) & (df['gross_sales_amount'] > 0)]\n",
    "df['trx_date'] = pd.to_datetime(df['trx_date'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b1419671",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:50.241071Z",
     "iopub.status.busy": "2024-03-08T08:43:50.239661Z",
     "iopub.status.idle": "2024-03-08T08:43:50.796428Z",
     "shell.execute_reply": "2024-03-08T08:43:50.795275Z"
    },
    "papermill": {
     "duration": 0.569368,
     "end_time": "2024-03-08T08:43:50.799088",
     "exception": false,
     "start_time": "2024-03-08T08:43:50.229720",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trx_date</th>\n",
       "      <th>customer_name</th>\n",
       "      <th>ship_to_id</th>\n",
       "      <th>branch_code</th>\n",
       "      <th>item_code</th>\n",
       "      <th>item_desc</th>\n",
       "      <th>principal_desc</th>\n",
       "      <th>gross_sales_amount</th>\n",
       "      <th>sales_qty</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-08-20</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>KCFMB</td>\n",
       "      <td>CEFIXIME 100MG 50 KAPSUL</td>\n",
       "      <td>HEXPHARM (PHARMAMED)</td>\n",
       "      <td>195000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-11-02</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>CKCOA</td>\n",
       "      <td>KALCINOL N CREAM  5 GR</td>\n",
       "      <td>KALBE NIMITZ (PHARMAMED)</td>\n",
       "      <td>28500</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-11-02</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>TMFNB</td>\n",
       "      <td>METFORMIN HCL 200 TABLET</td>\n",
       "      <td>HEXPHARM (PHARMAMED)</td>\n",
       "      <td>175000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-11-02</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>TBSVC</td>\n",
       "      <td>BRONSOLVAN 100 TABLET</td>\n",
       "      <td>HEXPHARM TSJ (PHARMAMED)</td>\n",
       "      <td>35000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-08-20</td>\n",
       "      <td>JK1-AP. DEVITA_GROUP_NA</td>\n",
       "      <td>EPM_34950</td>\n",
       "      <td>JK1</td>\n",
       "      <td>TALNF</td>\n",
       "      <td>AMLODIPINE BESILATE 10MG</td>\n",
       "      <td>HEXPHARM (PHARMAMED)</td>\n",
       "      <td>255000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    trx_date            customer_name ship_to_id branch_code item_code  \\\n",
       "0 2021-08-20  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     KCFMB   \n",
       "1 2021-11-02  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     CKCOA   \n",
       "2 2021-11-02  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     TMFNB   \n",
       "3 2021-11-02  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     TBSVC   \n",
       "4 2021-08-20  JK1-AP. DEVITA_GROUP_NA  EPM_34950         JK1     TALNF   \n",
       "\n",
       "                  item_desc            principal_desc  gross_sales_amount  \\\n",
       "0  CEFIXIME 100MG 50 KAPSUL      HEXPHARM (PHARMAMED)              195000   \n",
       "1    KALCINOL N CREAM  5 GR  KALBE NIMITZ (PHARMAMED)               28500   \n",
       "2  METFORMIN HCL 200 TABLET      HEXPHARM (PHARMAMED)              175000   \n",
       "3     BRONSOLVAN 100 TABLET  HEXPHARM TSJ (PHARMAMED)               35000   \n",
       "4  AMLODIPINE BESILATE 10MG      HEXPHARM (PHARMAMED)              255000   \n",
       "\n",
       "   sales_qty  \n",
       "0          1  \n",
       "1          1  \n",
       "2          1  \n",
       "3          1  \n",
       "4          2  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop duplicate transactions\n",
    "temp = df[['ship_to_id', 'item_code', 'trx_date', 'sales_qty']].groupby(['ship_to_id', 'item_code', 'trx_date']).sum().reset_index(drop=True)\n",
    "df = df.drop_duplicates(['ship_to_id', 'item_code', 'trx_date']).reset_index(drop=True)\n",
    "df['sales_qty'] = temp\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbb671c5",
   "metadata": {
    "papermill": {
     "duration": 0.009617,
     "end_time": "2024-03-08T08:43:50.817507",
     "exception": false,
     "start_time": "2024-03-08T08:43:50.807890",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Training and Evaluation Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a79aa59b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:50.837743Z",
     "iopub.status.busy": "2024-03-08T08:43:50.836862Z",
     "iopub.status.idle": "2024-03-08T08:43:50.870055Z",
     "shell.execute_reply": "2024-03-08T08:43:50.868735Z"
    },
    "papermill": {
     "duration": 0.046386,
     "end_time": "2024-03-08T08:43:50.872814",
     "exception": false,
     "start_time": "2024-03-08T08:43:50.826428",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "input_window = 3\n",
    "output_window = 1\n",
    "block_len = input_window + output_window\n",
    "\n",
    "# Mean Average Error\n",
    "criterion = nn.L1Loss() \n",
    "lr = 0.005\n",
    "\n",
    "def create_input_sequences(input_data, input_window, output_window):\n",
    "    input_seq = []\n",
    "    L = len(input_data)\n",
    "    block_num = L - block_len + 1\n",
    "\n",
    "    for i in range(block_num):\n",
    "        train_seq = input_data[i : i + input_window]\n",
    "        train_label = input_data[i + output_window : i + input_window + output_window]\n",
    "        input_seq.append((train_seq ,train_label))\n",
    "\n",
    "    return torch.FloatTensor(np.array(input_seq))\n",
    "\n",
    "def create_sliding_window(x):\n",
    "\n",
    "    data = pd.DataFrame(x).to_numpy().reshape(-1, 1).reshape(-1)\n",
    "    times = len(data)\n",
    "\n",
    "    sampels = int(times * 0.8)\n",
    "    train_data = data[:sampels]\n",
    "    test_data = data[sampels:]\n",
    "    test_data2 = test_data\n",
    "\n",
    "    train_data = create_input_sequences(train_data, input_window, output_window).to('cpu')\n",
    "    test_data = create_input_sequences(test_data, input_window, output_window).to('cpu')\n",
    "    \n",
    "    return train_data, test_data, test_data2\n",
    "\n",
    "class PositionalEncoding(nn.Module):\n",
    "\n",
    "    def __init__(self, d_model, max_len=5000):\n",
    "        super(PositionalEncoding, self).__init__()       \n",
    "        pe = torch.zeros(max_len, d_model)\n",
    "        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)\n",
    "        div_term = 1 / (10000 ** ((2 * np.arange(d_model)) / d_model))\n",
    "        pe[:, 0::2] = torch.sin(position * div_term[0::2])\n",
    "        pe[:, 1::2] = torch.cos(position * div_term[1::2])\n",
    "\n",
    "        pe = pe.unsqueeze(0).transpose(0, 1)\n",
    "        self.register_buffer('pe', pe)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return x + self.pe[:x.size(0), :].repeat(1,x.shape[1],1)\n",
    "          \n",
    "class Transformer(nn.Module):\n",
    "    def __init__(self, feature_size=250, num_layers=1, dropout=0.1):\n",
    "        super(Transformer, self).__init__()\n",
    "        self.model_type = 'Transformer'\n",
    "        self.input_embedding  = nn.Linear(1, feature_size)\n",
    "        self.src_mask = None\n",
    "\n",
    "        self.pos_encoder = PositionalEncoding(feature_size)\n",
    "        self.encoder_layer = nn.TransformerEncoderLayer(d_model=feature_size, nhead=10, dropout=dropout)\n",
    "        self.transformer_encoder = nn.TransformerEncoder(self.encoder_layer, num_layers=num_layers)\n",
    "        self.decoder = nn.Linear(feature_size, 1)\n",
    "        self.init_weights()\n",
    "\n",
    "    def init_weights(self):\n",
    "        initrange = 0.1    \n",
    "        self.decoder.bias.data.zero_()\n",
    "        self.decoder.weight.data.uniform_(-initrange, initrange)\n",
    "\n",
    "    def forward(self, src):\n",
    "        if self.src_mask is None or self.src_mask.size(0) != len(src):\n",
    "            device = src.device\n",
    "            mask = self._generate_square_subsequent_mask(len(src)).to(device)\n",
    "            self.src_mask = mask\n",
    "\n",
    "        src = self.input_embedding(src)\n",
    "        src = self.pos_encoder(src)\n",
    "        output = self.transformer_encoder(src, self.src_mask)\n",
    "        output = self.decoder(output)\n",
    "        return output\n",
    "\n",
    "    def _generate_square_subsequent_mask(self, sz):\n",
    "        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)\n",
    "        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n",
    "        return mask\n",
    "\n",
    "def get_batch(input_data, i, batch_size):\n",
    "\n",
    "    batch_len = min(batch_size, len(input_data) - i)\n",
    "    data = input_data[i:i + batch_len]\n",
    "    src = torch.stack([item[0] for item in data]).view((batch_len,input_window,1))\n",
    "    target = torch.stack([item[1] for item in data]).view((batch_len,input_window,1))\n",
    "    return src, target\n",
    "\n",
    "def train_one_epoch(epoch, train_data, model, optimizer):\n",
    "\n",
    "    model.train()\n",
    "    batch_size = len(train_data) // 5\n",
    "    total_loss = 0\n",
    "    start_time = time.time()\n",
    "\n",
    "    for i in range(0, len(train_data), batch_size):\n",
    "        data, targets = get_batch(train_data, i, batch_size)\n",
    "        optimizer.zero_grad()\n",
    "        output = model(data)\n",
    "\n",
    "        loss = criterion(output, targets)\n",
    "        loss.backward()\n",
    "\n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.7)\n",
    "        optimizer.step()\n",
    "\n",
    "        total_loss += loss.item()\n",
    "        log_interval = int(len(train_data) / batch_size / 5)\n",
    "        log_interval = max(log_interval, 1)\n",
    "        if (i // batch_size) % log_interval == 0 and i > 0:\n",
    "            cur_loss = total_loss / log_interval\n",
    "            elapsed = time.time() - start_time\n",
    "            print('| epoch {:3d} | {:5d}/{:5d} instances | {:5.2f} ms | loss {:5.5f}'.format(\n",
    "                    epoch, i, len(train_data), elapsed * 1000, cur_loss))\n",
    "            total_loss = 0\n",
    "            start_time = time.time()\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31fef338",
   "metadata": {
    "papermill": {
     "duration": 0.008567,
     "end_time": "2024-03-08T08:43:50.890248",
     "exception": false,
     "start_time": "2024-03-08T08:43:50.881681",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We use two usecases from the machine learning code (routine transactions and top items). After choosing a specific user and item, we put those data into the model. We perform 5 epochs for training the transformers model. The models are trained with 80% of the data and evaluated with the rest by the Mean Average Error and Mean Squared Error metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "585547d0",
   "metadata": {
    "papermill": {
     "duration": 0.008596,
     "end_time": "2024-03-08T08:43:50.907682",
     "exception": false,
     "start_time": "2024-03-08T08:43:50.899086",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Routine Transactions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "057f9903",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:50.927974Z",
     "iopub.status.busy": "2024-03-08T08:43:50.927198Z",
     "iopub.status.idle": "2024-03-08T08:43:51.334194Z",
     "shell.execute_reply": "2024-03-08T08:43:51.333090Z"
    },
    "papermill": {
     "duration": 0.420611,
     "end_time": "2024-03-08T08:43:51.337172",
     "exception": false,
     "start_time": "2024-03-08T08:43:50.916561",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ship_to_id</th>\n",
       "      <th>item_code</th>\n",
       "      <th>trx_date</th>\n",
       "      <th>sales_qty</th>\n",
       "      <th>Period</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>EPM_3041843</td>\n",
       "      <td>CKPXB</td>\n",
       "      <td>2021-01-20</td>\n",
       "      <td>1</td>\n",
       "      <td>30.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EPM_3041843</td>\n",
       "      <td>CKPXB</td>\n",
       "      <td>2021-02-19</td>\n",
       "      <td>1</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>EPM_3041843</td>\n",
       "      <td>CKPXB</td>\n",
       "      <td>2021-03-09</td>\n",
       "      <td>2</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>EPM_3041843</td>\n",
       "      <td>CKPXB</td>\n",
       "      <td>2021-03-19</td>\n",
       "      <td>1</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>EPM_3041843</td>\n",
       "      <td>CKPXB</td>\n",
       "      <td>2021-04-03</td>\n",
       "      <td>5</td>\n",
       "      <td>30.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    ship_to_id item_code   trx_date  sales_qty  Period\n",
       "0  EPM_3041843     CKPXB 2021-01-20          1    30.0\n",
       "1  EPM_3041843     CKPXB 2021-02-19          1    18.0\n",
       "2  EPM_3041843     CKPXB 2021-03-09          2    10.0\n",
       "3  EPM_3041843     CKPXB 2021-03-19          1    15.0\n",
       "4  EPM_3041843     CKPXB 2021-04-03          5    30.0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Taken from ML - Routine Transactions code\n",
    "lst = [('EPM_136080', 'TRGCA'), ('EPM_3041843', 'CKPXB')]\n",
    "series = []\n",
    "for l in lst:\n",
    "    dfU = df[(df['ship_to_id'] == l[0]) & (df['item_code'] == l[1])].sort_values('trx_date').reset_index(drop=True)[['ship_to_id', 'item_code', 'trx_date', 'sales_qty']]\n",
    "    dfU['trx_date'] = pd.to_datetime(dfU['trx_date'])\n",
    "    dfU = dfU.drop_duplicates('trx_date').reset_index(drop=True)\n",
    "    dfU['Period'] = dfU['trx_date'].diff().apply(lambda x: x.days)[1:].reset_index(drop=True)\n",
    "    series.append(list(dfU['Period'][:-1]))\n",
    "\n",
    "dfU.head() # We only use 'Period' column as the 'series' object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8b9dfa63",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:51.357249Z",
     "iopub.status.busy": "2024-03-08T08:43:51.356873Z",
     "iopub.status.idle": "2024-03-08T08:43:51.489159Z",
     "shell.execute_reply": "2024-03-08T08:43:51.487676Z"
    },
    "papermill": {
     "duration": 0.145153,
     "end_time": "2024-03-08T08:43:51.491529",
     "exception": false,
     "start_time": "2024-03-08T08:43:51.346376",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of the train and test data: torch.Size([28, 2, 3]) torch.Size([5, 2, 3])\n",
      "The first three data:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[[30., 18., 10.],\n",
       "         [18., 10., 15.]],\n",
       "\n",
       "        [[18., 10., 15.],\n",
       "         [10., 15., 30.]],\n",
       "\n",
       "        [[10., 15., 30.],\n",
       "         [15., 30., 15.]]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert series data to sliding window data\n",
    "input_window = 3\n",
    "output_window = 1\n",
    "block_len = input_window + output_window\n",
    "train_datas = []\n",
    "test_datas = []\n",
    "\n",
    "for s in series:\n",
    "    train, test, x = create_sliding_window(s)\n",
    "    train_datas.append(train)\n",
    "    test_datas.append(test)\n",
    "\n",
    "print(\"Size of the train and test data:\", train_datas[-1].shape, test_datas[-1].shape)\n",
    "print(\"The first three data:\")\n",
    "train_datas[-1][:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "480c9826",
   "metadata": {
    "papermill": {
     "duration": 0.010099,
     "end_time": "2024-03-08T08:43:51.511087",
     "exception": false,
     "start_time": "2024-03-08T08:43:51.500988",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Explanation of the sliding window data. Imagine we have time series data in the form of t1, t2, t3, t4, t5, ..., tn. Then the first data will be like this:\n",
    "\n",
    "Input -> Target\n",
    "\n",
    "[t1, t2, t3] -> [t2, t3, t4]\n",
    "\n",
    "It means that the first three data will be learned by the model to predict the fourth data (t4). Then we apply sliding method so the second data will be like this:\n",
    "\n",
    "Input -> Target\n",
    "\n",
    "[t2, t3, t4] -> [t3, t4, t5]\n",
    "\n",
    "It continues until we get the last three data that we can use to predict the next data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d2b97c81",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:51.531247Z",
     "iopub.status.busy": "2024-03-08T08:43:51.530838Z",
     "iopub.status.idle": "2024-03-08T08:43:56.126209Z",
     "shell.execute_reply": "2024-03-08T08:43:56.124931Z"
    },
    "papermill": {
     "duration": 4.608438,
     "end_time": "2024-03-08T08:43:56.128696",
     "exception": false,
     "start_time": "2024-03-08T08:43:51.520258",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EPM_136080 TRGCA\n",
      "-----------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/transformer.py:282: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)\n",
      "  warnings.warn(f\"enable_nested_tensor is True, but self.use_nested_tensor is False because {why_not_sparsity_fast_path}\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   1 |     7/   37 instances | 158.81 ms | loss 17.24604\n",
      "| epoch   1 |    14/   37 instances | 14.67 ms | loss 11.35866\n",
      "| epoch   1 |    21/   37 instances | 16.65 ms | loss 11.82564\n",
      "| epoch   1 |    28/   37 instances | 13.49 ms | loss 11.66547\n",
      "| epoch   1 |    35/   37 instances | 11.56 ms | loss 15.73454\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   2 |     7/   37 instances | 27.74 ms | loss 16.38223\n",
      "| epoch   2 |    14/   37 instances | 13.12 ms | loss 9.05232\n",
      "| epoch   2 |    21/   37 instances | 12.89 ms | loss 11.45190\n",
      "| epoch   2 |    28/   37 instances | 15.07 ms | loss 10.52405\n",
      "| epoch   2 |    35/   37 instances | 11.33 ms | loss 14.13780\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |     7/   37 instances | 26.85 ms | loss 19.90362\n",
      "| epoch   3 |    14/   37 instances | 12.53 ms | loss 7.68657\n",
      "| epoch   3 |    21/   37 instances | 14.09 ms | loss 10.92470\n",
      "| epoch   3 |    28/   37 instances | 15.96 ms | loss 9.39481\n",
      "| epoch   3 |    35/   37 instances | 11.98 ms | loss 13.46013\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |     7/   37 instances | 28.00 ms | loss 21.69312\n",
      "| epoch   4 |    14/   37 instances | 12.98 ms | loss 7.63748\n",
      "| epoch   4 |    21/   37 instances | 13.23 ms | loss 10.95950\n",
      "| epoch   4 |    28/   37 instances | 13.36 ms | loss 9.49998\n",
      "| epoch   4 |    35/   37 instances | 11.53 ms | loss 13.15010\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |     7/   37 instances | 30.80 ms | loss 21.90726\n",
      "| epoch   5 |    14/   37 instances | 13.15 ms | loss 7.67901\n",
      "| epoch   5 |    21/   37 instances | 12.97 ms | loss 10.95487\n",
      "| epoch   5 |    28/   37 instances | 12.90 ms | loss 9.53310\n",
      "| epoch   5 |    35/   37 instances | 11.41 ms | loss 13.15015\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_3041843 CKPXB\n",
      "-----------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/transformer.py:282: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)\n",
      "  warnings.warn(f\"enable_nested_tensor is True, but self.use_nested_tensor is False because {why_not_sparsity_fast_path}\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   1 |     5/   28 instances | 34.81 ms | loss 29.30421\n",
      "| epoch   1 |    10/   28 instances | 13.17 ms | loss 15.98377\n",
      "| epoch   1 |    15/   28 instances | 14.45 ms | loss 14.77681\n",
      "| epoch   1 |    20/   28 instances | 13.81 ms | loss 11.44895\n",
      "| epoch   1 |    25/   28 instances | 12.42 ms | loss 8.29645\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   2 |     5/   28 instances | 26.49 ms | loss 15.70588\n",
      "| epoch   2 |    10/   28 instances | 12.98 ms | loss 13.20625\n",
      "| epoch   2 |    15/   28 instances | 12.28 ms | loss 9.94258\n",
      "| epoch   2 |    20/   28 instances | 13.84 ms | loss 6.65999\n",
      "| epoch   2 |    25/   28 instances | 12.12 ms | loss 6.69000\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |     5/   28 instances | 27.44 ms | loss 17.78985\n",
      "| epoch   3 |    10/   28 instances | 14.34 ms | loss 12.34952\n",
      "| epoch   3 |    15/   28 instances | 12.46 ms | loss 9.66815\n",
      "| epoch   3 |    20/   28 instances | 13.52 ms | loss 5.28287\n",
      "| epoch   3 |    25/   28 instances | 14.53 ms | loss 6.43298\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |     5/   28 instances | 25.52 ms | loss 19.20883\n",
      "| epoch   4 |    10/   28 instances | 13.97 ms | loss 12.14402\n",
      "| epoch   4 |    15/   28 instances | 12.02 ms | loss 9.78305\n",
      "| epoch   4 |    20/   28 instances | 12.83 ms | loss 5.10741\n",
      "| epoch   4 |    25/   28 instances | 13.68 ms | loss 6.14438\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |     5/   28 instances | 26.49 ms | loss 18.36020\n",
      "| epoch   5 |    10/   28 instances | 12.95 ms | loss 12.38940\n",
      "| epoch   5 |    15/   28 instances | 12.51 ms | loss 9.58667\n",
      "| epoch   5 |    20/   28 instances | 13.40 ms | loss 5.59619\n",
      "| epoch   5 |    25/   28 instances | 12.00 ms | loss 6.42050\n",
      "-----------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Training Model\n",
    "\n",
    "models = []\n",
    "for idx in range(len(train_datas)):\n",
    "\n",
    "    print(lst[idx][0], lst[idx][1])\n",
    "    print('-' * 89)\n",
    "\n",
    "    # Mean Average Error\n",
    "    criterion = nn.L1Loss() \n",
    "    lr = 0.005 \n",
    "    model = Transformer().to('cpu')\n",
    "    optimizer = torch.optim.AdamW(model.parameters(), lr=lr)\n",
    "    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.95)\n",
    "\n",
    "    for i in range(5):\n",
    "        model = train_one_epoch(i + 1, train_datas[idx], model, optimizer)\n",
    "        print('-' * 89)\n",
    "        scheduler.step() \n",
    "    \n",
    "    models.append(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "de7c30ca",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:56.153950Z",
     "iopub.status.busy": "2024-03-08T08:43:56.153388Z",
     "iopub.status.idle": "2024-03-08T08:43:56.177214Z",
     "shell.execute_reply": "2024-03-08T08:43:56.176087Z"
    },
    "papermill": {
     "duration": 0.038307,
     "end_time": "2024-03-08T08:43:56.179754",
     "exception": false,
     "start_time": "2024-03-08T08:43:56.141447",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EPM_136080 TRGCA\n",
      "MAE: 10.046 MSE: 117.798\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_3041843 CKPXB\n",
      "MAE: 6.727 MSE: 58.512\n",
      "-----------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Evaluation\n",
    "for idx in range(len(test_datas)):\n",
    "\n",
    "    print(lst[idx][0], lst[idx][1])\n",
    "    eval_batch_size = len(test_datas[idx])\n",
    "\n",
    "    model = models[idx]\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        data, targets = get_batch(test_datas[idx], 0, eval_batch_size)\n",
    "        predictions = model(data)\n",
    "\n",
    "        mae = mean_absolute_error(targets.view(3, eval_batch_size).numpy(), predictions.view(3, eval_batch_size).numpy())\n",
    "        mse = mean_squared_error(targets.view(3, eval_batch_size).numpy(), predictions.view(3, eval_batch_size).numpy())\n",
    "\n",
    "    print(\"MAE: {:<5} MSE: {:<5}\".format(str(round(mae, 3)), str(round(mse, 3))))\n",
    "    print('-' * 89)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8c93e31",
   "metadata": {
    "papermill": {
     "duration": 0.009853,
     "end_time": "2024-03-08T08:43:56.199805",
     "exception": false,
     "start_time": "2024-03-08T08:43:56.189952",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Top Items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "959f5dca",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:56.222925Z",
     "iopub.status.busy": "2024-03-08T08:43:56.222258Z",
     "iopub.status.idle": "2024-03-08T08:43:58.176186Z",
     "shell.execute_reply": "2024-03-08T08:43:58.175052Z"
    },
    "papermill": {
     "duration": 1.968368,
     "end_time": "2024-03-08T08:43:58.178879",
     "exception": false,
     "start_time": "2024-03-08T08:43:56.210511",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ship_to_id</th>\n",
       "      <th>item_code</th>\n",
       "      <th>trx_date</th>\n",
       "      <th>sales_qty</th>\n",
       "      <th>Period</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>EPM_34923</td>\n",
       "      <td>TALNE</td>\n",
       "      <td>2021-01-20</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EPM_34923</td>\n",
       "      <td>TALNE</td>\n",
       "      <td>2021-01-23</td>\n",
       "      <td>5</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>EPM_34923</td>\n",
       "      <td>TALNE</td>\n",
       "      <td>2021-01-28</td>\n",
       "      <td>10</td>\n",
       "      <td>20.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>EPM_34923</td>\n",
       "      <td>TALNE</td>\n",
       "      <td>2021-02-17</td>\n",
       "      <td>10</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>EPM_34923</td>\n",
       "      <td>TALNE</td>\n",
       "      <td>2021-02-20</td>\n",
       "      <td>5</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  ship_to_id item_code   trx_date  sales_qty  Period\n",
       "0  EPM_34923     TALNE 2021-01-20          2     3.0\n",
       "1  EPM_34923     TALNE 2021-01-23          5     5.0\n",
       "2  EPM_34923     TALNE 2021-01-28         10    20.0\n",
       "3  EPM_34923     TALNE 2021-02-17         10     3.0\n",
       "4  EPM_34923     TALNE 2021-02-20          5     5.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Taken from ML - Top Items code\n",
    "item = 'TALNE'\n",
    "lst = ['EPM_35159', 'EPM_4334085', 'EPM_1807311', 'EPM_3564728', 'EPM_35002', 'EPM_34985', 'EPM_136080', 'EPM_1624002', 'EPM_3676050', 'EPM_34923']\n",
    "series = []\n",
    "for l in lst:\n",
    "    dfU = df[(df['ship_to_id'] == l) & (df['item_code'] == item)].sort_values('trx_date').reset_index(drop=True)[['ship_to_id', 'item_code', 'trx_date', 'sales_qty']]\n",
    "    dfU['trx_date'] = pd.to_datetime(dfU['trx_date'])\n",
    "    dfU = dfU.drop_duplicates('trx_date').reset_index(drop=True)\n",
    "    dfU['Period'] = dfU['trx_date'].diff().apply(lambda x: x.days)[1:].reset_index(drop=True)\n",
    "    series.append(list(dfU['Period'][:-1]))\n",
    "\n",
    "dfU.head() # We only use 'Period' column as the 'series' object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9bc28977",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:58.201901Z",
     "iopub.status.busy": "2024-03-08T08:43:58.201501Z",
     "iopub.status.idle": "2024-03-08T08:43:58.222197Z",
     "shell.execute_reply": "2024-03-08T08:43:58.221136Z"
    },
    "papermill": {
     "duration": 0.035149,
     "end_time": "2024-03-08T08:43:58.224646",
     "exception": false,
     "start_time": "2024-03-08T08:43:58.189497",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of the train and test data: torch.Size([95, 2, 3]) torch.Size([22, 2, 3])\n",
      "The first three data:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[[ 3.,  5., 20.],\n",
       "         [ 5., 20.,  3.]],\n",
       "\n",
       "        [[ 5., 20.,  3.],\n",
       "         [20.,  3.,  5.]],\n",
       "\n",
       "        [[20.,  3.,  5.],\n",
       "         [ 3.,  5., 16.]]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert series data to sliding window data\n",
    "train_datas = []\n",
    "test_datas = []\n",
    "\n",
    "for s in series:\n",
    "    train, test, x = create_sliding_window(s)\n",
    "    train_datas.append(train)\n",
    "    test_datas.append(test)\n",
    "\n",
    "print(\"Size of the train and test data:\", train_datas[-1].shape, test_datas[-1].shape)\n",
    "print(\"The first three data:\")\n",
    "train_datas[-1][:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d4708cd5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:43:58.248335Z",
     "iopub.status.busy": "2024-03-08T08:43:58.247642Z",
     "iopub.status.idle": "2024-03-08T08:44:05.014539Z",
     "shell.execute_reply": "2024-03-08T08:44:05.013157Z"
    },
    "papermill": {
     "duration": 6.783179,
     "end_time": "2024-03-08T08:44:05.018650",
     "exception": false,
     "start_time": "2024-03-08T08:43:58.235471",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EPM_35159 TALNE\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   1 |    27/  138 instances | 45.37 ms | loss 12.57479\n",
      "| epoch   1 |    54/  138 instances | 22.50 ms | loss 4.72645\n",
      "| epoch   1 |    81/  138 instances | 33.94 ms | loss 2.67864\n",
      "| epoch   1 |   108/  138 instances | 29.78 ms | loss 2.35305\n",
      "| epoch   1 |   135/  138 instances | 15.74 ms | loss 0.96102\n",
      "-----------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/transformer.py:282: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)\n",
      "  warnings.warn(f\"enable_nested_tensor is True, but self.use_nested_tensor is False because {why_not_sparsity_fast_path}\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   2 |    27/  138 instances | 59.50 ms | loss 4.49643\n",
      "| epoch   2 |    54/  138 instances | 27.76 ms | loss 3.40243\n",
      "| epoch   2 |    81/  138 instances | 25.50 ms | loss 2.44292\n",
      "| epoch   2 |   108/  138 instances | 24.88 ms | loss 2.41719\n",
      "| epoch   2 |   135/  138 instances | 12.98 ms | loss 0.82932\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |    27/  138 instances | 54.81 ms | loss 4.92077\n",
      "| epoch   3 |    54/  138 instances | 26.89 ms | loss 2.98112\n",
      "| epoch   3 |    81/  138 instances | 24.23 ms | loss 2.16505\n",
      "| epoch   3 |   108/  138 instances | 31.15 ms | loss 2.26282\n",
      "| epoch   3 |   135/  138 instances | 13.25 ms | loss 0.81894\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |    27/  138 instances | 51.62 ms | loss 4.66377\n",
      "| epoch   4 |    54/  138 instances | 25.14 ms | loss 3.12828\n",
      "| epoch   4 |    81/  138 instances | 24.50 ms | loss 2.28695\n",
      "| epoch   4 |   108/  138 instances | 23.85 ms | loss 2.31895\n",
      "| epoch   4 |   135/  138 instances | 12.46 ms | loss 0.97421\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |    27/  138 instances | 53.04 ms | loss 4.58307\n",
      "| epoch   5 |    54/  138 instances | 24.54 ms | loss 3.03838\n",
      "| epoch   5 |    81/  138 instances | 23.36 ms | loss 2.20280\n",
      "| epoch   5 |   108/  138 instances | 22.28 ms | loss 2.22572\n",
      "| epoch   5 |   135/  138 instances | 12.22 ms | loss 0.85290\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_4334085 TALNE\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   1 |    26/  130 instances | 37.99 ms | loss 11.75898\n",
      "| epoch   1 |    52/  130 instances | 23.79 ms | loss 6.20678\n",
      "| epoch   1 |    78/  130 instances | 24.22 ms | loss 1.73980\n",
      "| epoch   1 |   104/  130 instances | 25.53 ms | loss 4.66162\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   2 |    26/  130 instances | 44.19 ms | loss 2.94010\n",
      "| epoch   2 |    52/  130 instances | 20.26 ms | loss 2.53253\n",
      "| epoch   2 |    78/  130 instances | 23.11 ms | loss 1.86741\n",
      "| epoch   2 |   104/  130 instances | 27.85 ms | loss 1.81337\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |    26/  130 instances | 51.72 ms | loss 2.36687\n",
      "| epoch   3 |    52/  130 instances | 25.79 ms | loss 2.53524\n",
      "| epoch   3 |    78/  130 instances | 27.50 ms | loss 1.77340\n",
      "| epoch   3 |   104/  130 instances | 27.05 ms | loss 1.75694\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |    26/  130 instances | 54.36 ms | loss 2.64147\n",
      "| epoch   4 |    52/  130 instances | 24.07 ms | loss 2.62137\n",
      "| epoch   4 |    78/  130 instances | 25.87 ms | loss 1.66348\n",
      "| epoch   4 |   104/  130 instances | 27.15 ms | loss 1.73797\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |    26/  130 instances | 54.70 ms | loss 2.41794\n",
      "| epoch   5 |    52/  130 instances | 25.61 ms | loss 2.55399\n",
      "| epoch   5 |    78/  130 instances | 26.64 ms | loss 1.76568\n",
      "| epoch   5 |   104/  130 instances | 22.61 ms | loss 1.78683\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_1807311 TALNE\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   1 |    25/  128 instances | 44.18 ms | loss 13.62649\n",
      "| epoch   1 |    50/  128 instances | 23.77 ms | loss 5.22828\n",
      "| epoch   1 |    75/  128 instances | 24.75 ms | loss 3.67836\n",
      "| epoch   1 |   100/  128 instances | 25.29 ms | loss 1.98722\n",
      "| epoch   1 |   125/  128 instances | 13.53 ms | loss 4.64148\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   2 |    25/  128 instances | 51.22 ms | loss 5.31037\n",
      "| epoch   2 |    50/  128 instances | 23.22 ms | loss 2.50402\n",
      "| epoch   2 |    75/  128 instances | 22.56 ms | loss 3.48087\n",
      "| epoch   2 |   100/  128 instances | 27.48 ms | loss 2.55590\n",
      "| epoch   2 |   125/  128 instances | 13.88 ms | loss 4.62922\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |    25/  128 instances | 45.03 ms | loss 5.58971\n",
      "| epoch   3 |    50/  128 instances | 26.05 ms | loss 2.65399\n",
      "| epoch   3 |    75/  128 instances | 22.02 ms | loss 2.86066\n",
      "| epoch   3 |   100/  128 instances | 24.30 ms | loss 2.21158\n",
      "| epoch   3 |   125/  128 instances | 12.49 ms | loss 4.65814\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |    25/  128 instances | 46.16 ms | loss 5.35675\n",
      "| epoch   4 |    50/  128 instances | 25.22 ms | loss 2.55388\n",
      "| epoch   4 |    75/  128 instances | 22.61 ms | loss 2.96847\n",
      "| epoch   4 |   100/  128 instances | 24.29 ms | loss 2.28097\n",
      "| epoch   4 |   125/  128 instances | 13.00 ms | loss 4.64256\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |    25/  128 instances | 51.50 ms | loss 5.29427\n",
      "| epoch   5 |    50/  128 instances | 26.16 ms | loss 2.60366\n",
      "| epoch   5 |    75/  128 instances | 23.08 ms | loss 2.78289\n",
      "| epoch   5 |   100/  128 instances | 25.82 ms | loss 2.08881\n",
      "| epoch   5 |   125/  128 instances | 13.27 ms | loss 4.62690\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_3564728 TALNE\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   1 |    25/  127 instances | 45.06 ms | loss 13.24857\n",
      "| epoch   1 |    50/  127 instances | 20.05 ms | loss 4.15398\n",
      "| epoch   1 |    75/  127 instances | 26.04 ms | loss 5.99461\n",
      "| epoch   1 |   100/  127 instances | 21.52 ms | loss 8.17198\n",
      "| epoch   1 |   125/  127 instances | 15.07 ms | loss 6.22323\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   2 |    25/  127 instances | 53.42 ms | loss 4.23270\n",
      "| epoch   2 |    50/  127 instances | 27.46 ms | loss 3.58158\n",
      "| epoch   2 |    75/  127 instances | 24.50 ms | loss 7.11499\n",
      "| epoch   2 |   100/  127 instances | 23.82 ms | loss 9.69211\n",
      "| epoch   2 |   125/  127 instances | 12.11 ms | loss 6.08220\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |    25/  127 instances | 53.14 ms | loss 3.95393\n",
      "| epoch   3 |    50/  127 instances | 27.28 ms | loss 2.38752\n",
      "| epoch   3 |    75/  127 instances | 23.02 ms | loss 5.58544\n",
      "| epoch   3 |   100/  127 instances | 22.78 ms | loss 8.55744\n",
      "| epoch   3 |   125/  127 instances | 12.64 ms | loss 5.99667\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |    25/  127 instances | 51.04 ms | loss 4.73952\n",
      "| epoch   4 |    50/  127 instances | 25.79 ms | loss 2.38981\n",
      "| epoch   4 |    75/  127 instances | 23.04 ms | loss 5.11470\n",
      "| epoch   4 |   100/  127 instances | 23.42 ms | loss 8.43703\n",
      "| epoch   4 |   125/  127 instances | 12.10 ms | loss 6.08798\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |    25/  127 instances | 47.73 ms | loss 4.43655\n",
      "| epoch   5 |    50/  127 instances | 25.65 ms | loss 2.28069\n",
      "| epoch   5 |    75/  127 instances | 23.02 ms | loss 5.54501\n",
      "| epoch   5 |   100/  127 instances | 21.68 ms | loss 8.75193\n",
      "| epoch   5 |   125/  127 instances | 11.55 ms | loss 5.84384\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_35002 TALNE\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   1 |    23/  118 instances | 50.34 ms | loss 12.95737\n",
      "| epoch   1 |    46/  118 instances | 28.12 ms | loss 2.67044\n",
      "| epoch   1 |    69/  118 instances | 29.38 ms | loss 4.53952\n",
      "| epoch   1 |    92/  118 instances | 25.56 ms | loss 2.23768\n",
      "| epoch   1 |   115/  118 instances | 16.77 ms | loss 2.66434\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   2 |    23/  118 instances | 48.97 ms | loss 5.71742\n",
      "| epoch   2 |    46/  118 instances | 27.82 ms | loss 3.12574\n",
      "| epoch   2 |    69/  118 instances | 31.61 ms | loss 2.03371\n",
      "| epoch   2 |    92/  118 instances | 28.32 ms | loss 2.85923\n",
      "| epoch   2 |   115/  118 instances | 13.25 ms | loss 3.43632\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |    23/  118 instances | 50.38 ms | loss 4.53240\n",
      "| epoch   3 |    46/  118 instances | 27.40 ms | loss 2.46909\n",
      "| epoch   3 |    69/  118 instances | 27.39 ms | loss 1.93090\n",
      "| epoch   3 |    92/  118 instances | 30.61 ms | loss 2.35204\n",
      "| epoch   3 |   115/  118 instances | 13.57 ms | loss 2.92284\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |    23/  118 instances | 50.04 ms | loss 4.69664\n",
      "| epoch   4 |    46/  118 instances | 24.30 ms | loss 2.96125\n",
      "| epoch   4 |    69/  118 instances | 29.60 ms | loss 2.12009\n",
      "| epoch   4 |    92/  118 instances | 27.84 ms | loss 2.28940\n",
      "| epoch   4 |   115/  118 instances | 13.22 ms | loss 2.95031\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |    23/  118 instances | 45.65 ms | loss 4.45676\n",
      "| epoch   5 |    46/  118 instances | 23.88 ms | loss 2.56094\n",
      "| epoch   5 |    69/  118 instances | 27.23 ms | loss 1.90758\n",
      "| epoch   5 |    92/  118 instances | 27.92 ms | loss 2.54922\n",
      "| epoch   5 |   115/  118 instances | 13.24 ms | loss 3.23089\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_34985 TALNE\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   1 |    21/  105 instances | 48.16 ms | loss 17.54439\n",
      "| epoch   1 |    42/  105 instances | 20.79 ms | loss 5.12800\n",
      "| epoch   1 |    63/  105 instances | 22.58 ms | loss 4.36078\n",
      "| epoch   1 |    84/  105 instances | 23.35 ms | loss 6.17026\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   2 |    21/  105 instances | 51.89 ms | loss 10.61577\n",
      "| epoch   2 |    42/  105 instances | 23.67 ms | loss 3.12894\n",
      "| epoch   2 |    63/  105 instances | 22.39 ms | loss 4.65055\n",
      "| epoch   2 |    84/  105 instances | 26.74 ms | loss 6.57863\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |    21/  105 instances | 42.68 ms | loss 10.71339\n",
      "| epoch   3 |    42/  105 instances | 21.90 ms | loss 3.06897\n",
      "| epoch   3 |    63/  105 instances | 23.12 ms | loss 4.19928\n",
      "| epoch   3 |    84/  105 instances | 22.54 ms | loss 6.05212\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |    21/  105 instances | 45.67 ms | loss 10.81479\n",
      "| epoch   4 |    42/  105 instances | 21.33 ms | loss 3.27143\n",
      "| epoch   4 |    63/  105 instances | 21.39 ms | loss 4.15565\n",
      "| epoch   4 |    84/  105 instances | 23.23 ms | loss 6.04275\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |    21/  105 instances | 45.40 ms | loss 10.56038\n",
      "| epoch   5 |    42/  105 instances | 20.94 ms | loss 3.06841\n",
      "| epoch   5 |    63/  105 instances | 20.46 ms | loss 4.23246\n",
      "| epoch   5 |    84/  105 instances | 20.51 ms | loss 6.18798\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_136080 TALNE\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   1 |    20/  102 instances | 51.23 ms | loss 10.36160\n",
      "| epoch   1 |    40/  102 instances | 31.04 ms | loss 3.01422\n",
      "| epoch   1 |    60/  102 instances | 24.33 ms | loss 3.03100\n",
      "| epoch   1 |    80/  102 instances | 19.32 ms | loss 3.96119\n",
      "| epoch   1 |   100/  102 instances | 11.80 ms | loss 3.79654\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   2 |    20/  102 instances | 44.01 ms | loss 5.74765\n",
      "| epoch   2 |    40/  102 instances | 23.32 ms | loss 2.28336\n",
      "| epoch   2 |    60/  102 instances | 22.13 ms | loss 1.85218\n",
      "| epoch   2 |    80/  102 instances | 23.25 ms | loss 3.88131\n",
      "| epoch   2 |   100/  102 instances | 13.44 ms | loss 3.60176\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |    20/  102 instances | 50.57 ms | loss 5.74103\n",
      "| epoch   3 |    40/  102 instances | 24.11 ms | loss 2.20149\n",
      "| epoch   3 |    60/  102 instances | 24.19 ms | loss 1.94944\n",
      "| epoch   3 |    80/  102 instances | 28.14 ms | loss 3.94886\n",
      "| epoch   3 |   100/  102 instances | 12.59 ms | loss 3.29995\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |    20/  102 instances | 48.90 ms | loss 5.63407\n",
      "| epoch   4 |    40/  102 instances | 21.66 ms | loss 2.52664\n",
      "| epoch   4 |    60/  102 instances | 23.41 ms | loss 2.38214\n",
      "| epoch   4 |    80/  102 instances | 25.75 ms | loss 4.56717\n",
      "| epoch   4 |   100/  102 instances | 13.17 ms | loss 2.81910\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |    20/  102 instances | 48.03 ms | loss 5.65201\n",
      "| epoch   5 |    40/  102 instances | 21.81 ms | loss 2.44957\n",
      "| epoch   5 |    60/  102 instances | 22.28 ms | loss 2.02691\n",
      "| epoch   5 |    80/  102 instances | 24.90 ms | loss 4.04585\n",
      "| epoch   5 |   100/  102 instances | 13.79 ms | loss 3.06799\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_1624002 TALNE\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   1 |    19/   97 instances | 50.16 ms | loss 10.56959\n",
      "| epoch   1 |    38/   97 instances | 22.00 ms | loss 4.69875\n",
      "| epoch   1 |    57/   97 instances | 22.30 ms | loss 11.61507\n",
      "| epoch   1 |    76/   97 instances | 24.99 ms | loss 3.61071\n",
      "| epoch   1 |    95/   97 instances | 13.77 ms | loss 3.14138\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   2 |    19/   97 instances | 46.04 ms | loss 10.14058\n",
      "| epoch   2 |    38/   97 instances | 18.77 ms | loss 3.43354\n",
      "| epoch   2 |    57/   97 instances | 19.35 ms | loss 10.24498\n",
      "| epoch   2 |    76/   97 instances | 17.76 ms | loss 4.81409\n",
      "| epoch   2 |    95/   97 instances | 12.15 ms | loss 2.98662\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |    19/   97 instances | 46.22 ms | loss 7.76716\n",
      "| epoch   3 |    38/   97 instances | 19.60 ms | loss 2.88192\n",
      "| epoch   3 |    57/   97 instances | 20.38 ms | loss 10.36263\n",
      "| epoch   3 |    76/   97 instances | 17.71 ms | loss 4.55048\n",
      "| epoch   3 |    95/   97 instances | 12.77 ms | loss 2.63769\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |    19/   97 instances | 43.83 ms | loss 7.65776\n",
      "| epoch   4 |    38/   97 instances | 18.71 ms | loss 3.11952\n",
      "| epoch   4 |    57/   97 instances | 17.75 ms | loss 9.95412\n",
      "| epoch   4 |    76/   97 instances | 17.71 ms | loss 3.99378\n",
      "| epoch   4 |    95/   97 instances | 12.49 ms | loss 2.75185\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |    19/   97 instances | 43.76 ms | loss 7.91702\n",
      "| epoch   5 |    38/   97 instances | 18.77 ms | loss 3.47482\n",
      "| epoch   5 |    57/   97 instances | 17.48 ms | loss 9.65452\n",
      "| epoch   5 |    76/   97 instances | 24.83 ms | loss 3.58620\n",
      "| epoch   5 |    95/   97 instances | 13.67 ms | loss 2.62698\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_3676050 TALNE\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   1 |    19/   96 instances | 51.23 ms | loss 11.08163\n",
      "| epoch   1 |    38/   96 instances | 28.16 ms | loss 2.08550\n",
      "| epoch   1 |    57/   96 instances | 25.29 ms | loss 3.35112\n",
      "| epoch   1 |    76/   96 instances | 30.30 ms | loss 2.37965\n",
      "| epoch   1 |    95/   96 instances | 14.80 ms | loss 2.80598\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   2 |    19/   96 instances | 46.27 ms | loss 4.38584\n",
      "| epoch   2 |    38/   96 instances | 21.13 ms | loss 1.92170\n",
      "| epoch   2 |    57/   96 instances | 21.42 ms | loss 2.39924\n",
      "| epoch   2 |    76/   96 instances | 24.45 ms | loss 2.14542\n",
      "| epoch   2 |    95/   96 instances | 11.43 ms | loss 3.27591\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |    19/   96 instances | 45.66 ms | loss 3.91746\n",
      "| epoch   3 |    38/   96 instances | 21.50 ms | loss 1.86413\n",
      "| epoch   3 |    57/   96 instances | 21.30 ms | loss 2.30993\n",
      "| epoch   3 |    76/   96 instances | 20.38 ms | loss 2.11695\n",
      "| epoch   3 |    95/   96 instances | 11.72 ms | loss 3.00642\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |    19/   96 instances | 42.84 ms | loss 3.97118\n",
      "| epoch   4 |    38/   96 instances | 19.85 ms | loss 1.83632\n",
      "| epoch   4 |    57/   96 instances | 23.02 ms | loss 2.24466\n",
      "| epoch   4 |    76/   96 instances | 20.80 ms | loss 2.11071\n",
      "| epoch   4 |    95/   96 instances | 17.65 ms | loss 2.88506\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |    19/   96 instances | 46.79 ms | loss 4.02690\n",
      "| epoch   5 |    38/   96 instances | 21.83 ms | loss 1.83011\n",
      "| epoch   5 |    57/   96 instances | 31.25 ms | loss 2.25513\n",
      "| epoch   5 |    76/   96 instances | 31.54 ms | loss 2.10197\n",
      "| epoch   5 |    95/   96 instances | 17.60 ms | loss 3.09840\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_34923 TALNE\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   1 |    19/   95 instances | 44.26 ms | loss 13.98257\n",
      "| epoch   1 |    38/   95 instances | 27.21 ms | loss 4.05954\n",
      "| epoch   1 |    57/   95 instances | 23.37 ms | loss 6.20116\n",
      "| epoch   1 |    76/   95 instances | 21.04 ms | loss 4.57406\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   2 |    19/   95 instances | 48.88 ms | loss 7.64420\n",
      "| epoch   2 |    38/   95 instances | 24.82 ms | loss 4.18886\n",
      "| epoch   2 |    57/   95 instances | 20.07 ms | loss 5.46434\n",
      "| epoch   2 |    76/   95 instances | 18.87 ms | loss 3.91276\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   3 |    19/   95 instances | 45.26 ms | loss 7.08697\n",
      "| epoch   3 |    38/   95 instances | 21.93 ms | loss 2.82850\n",
      "| epoch   3 |    57/   95 instances | 17.94 ms | loss 4.91770\n",
      "| epoch   3 |    76/   95 instances | 19.11 ms | loss 4.25877\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   4 |    19/   95 instances | 39.05 ms | loss 7.19398\n",
      "| epoch   4 |    38/   95 instances | 21.09 ms | loss 2.95012\n",
      "| epoch   4 |    57/   95 instances | 19.32 ms | loss 5.00384\n",
      "| epoch   4 |    76/   95 instances | 17.63 ms | loss 4.02521\n",
      "-----------------------------------------------------------------------------------------\n",
      "| epoch   5 |    19/   95 instances | 37.78 ms | loss 7.07814\n",
      "| epoch   5 |    38/   95 instances | 21.70 ms | loss 3.20523\n",
      "| epoch   5 |    57/   95 instances | 19.06 ms | loss 5.12232\n",
      "| epoch   5 |    76/   95 instances | 18.21 ms | loss 4.03910\n",
      "-----------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Training Model\n",
    "\n",
    "models = []\n",
    "for idx in range(len(train_datas)):\n",
    "\n",
    "    print(lst[idx], item)\n",
    "    print('-' * 89)\n",
    "    # Mean Average Error\n",
    "    criterion = nn.L1Loss() \n",
    "    lr = 0.005 \n",
    "    model = Transformer().to('cpu')\n",
    "    optimizer = torch.optim.AdamW(model.parameters(), lr=lr)\n",
    "    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.95)\n",
    "\n",
    "    for i in range(5):\n",
    "        model = train_one_epoch(i + 1, train_datas[idx], model, optimizer)\n",
    "        print('-' * 89)\n",
    "        scheduler.step() \n",
    "    \n",
    "    models.append(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "fabd1749",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-08T08:44:05.054727Z",
     "iopub.status.busy": "2024-03-08T08:44:05.054322Z",
     "iopub.status.idle": "2024-03-08T08:44:05.117316Z",
     "shell.execute_reply": "2024-03-08T08:44:05.116453Z"
    },
    "papermill": {
     "duration": 0.083599,
     "end_time": "2024-03-08T08:44:05.120001",
     "exception": false,
     "start_time": "2024-03-08T08:44:05.036402",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EPM_35159 TALNE\n",
      "MAE: 2.084 MSE: 7.252\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_4334085 TALNE\n",
      "MAE: 1.807 MSE: 7.506\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_1807311 TALNE\n",
      "MAE: 5.579 MSE: 216.754\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_3564728 TALNE\n",
      "MAE: 2.445 MSE: 8.904\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_35002 TALNE\n",
      "MAE: 2.605 MSE: 9.068\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_34985 TALNE\n",
      "MAE: 2.411 MSE: 11.862\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_136080 TALNE\n",
      "MAE: 4.995 MSE: 76.317\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_1624002 TALNE\n",
      "MAE: 2.688 MSE: 10.559\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_3676050 TALNE\n",
      "MAE: 5.188 MSE: 82.339\n",
      "-----------------------------------------------------------------------------------------\n",
      "EPM_34923 TALNE\n",
      "MAE: 3.945 MSE: 22.391\n",
      "-----------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Evaluation\n",
    "for idx in range(len(test_datas)):\n",
    "\n",
    "    print(lst[idx], item)\n",
    "    eval_batch_size = len(test_datas[idx])\n",
    "\n",
    "    model = models[idx]\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        data, targets = get_batch(test_datas[idx], 0, eval_batch_size)\n",
    "        predictions = model(data)\n",
    "\n",
    "        mae = mean_absolute_error(targets.view(3, eval_batch_size).numpy(), predictions.view(3, eval_batch_size).numpy())\n",
    "        mse = mean_squared_error(targets.view(3, eval_batch_size).numpy(), predictions.view(3, eval_batch_size).numpy())\n",
    "\n",
    "    print(\"MAE: {:<5} MSE: {:<5}\".format(str(round(mae, 3)), str(round(mse, 3))))\n",
    "    print('-' * 89)"
   ]
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "none",
   "dataSources": [
    {
     "datasetId": 3947402,
     "sourceId": 7173504,
     "sourceType": "datasetVersion"
    }
   ],
   "dockerImageVersionId": 30646,
   "isGpuEnabled": false,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 60.14891,
   "end_time": "2024-03-08T08:44:06.664281",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2024-03-08T08:43:06.515371",
   "version": "2.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}