Part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c210cfb34f9dfbe681b8194c01b8d3c2a3441729"
   },
   "source": [
    "## **Αναγνώριση Προτύπων - 3η Εργαστηριακή Άσκηση** ##\n",
    "\n",
    "## Αναγνώριση Είδους και Εξαγωγή Συναισθήματος από Μουσική ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "0bbbcb90e79cea052c45cd5b8b14c3a0a09e51a1"
   },
   "source": [
    "Χρυσούλα Κοσμά - 03114025\n",
    "\n",
    "Λεωνίδας Αβδελάς - 03113182\n",
    "\n",
    "9ο Εξάμηνο ΣΗΜΜΥ ΕΜΠ\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "8ce6a3ec655990a554434f008037226306addb8e"
   },
   "source": [
    "Ακολουθούν τα **βήματα του κύριου μέρους (βήματα 9-12)** του 3ου Εργαστηρίου.\n",
    "\n",
    "Αρχικά, κάνουμε import ορισμένες από τις βιβλιοθήκες που είναι απαραίτητες για την εκτέλεση των βημάτων της εργασίας."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['CUDA_LAUNCH_BLOCKING'] = \"1\"\n",
    "\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "import gzip\n",
    "import matplotlib.pyplot as plt\n",
    "import librosa\n",
    "\n",
    "from librosa import display\n",
    "from librosa import beat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [],
   "source": [
    "import copy\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from torch.utils.data import Dataset\n",
    "from torch.utils.data import SubsetRandomSampler, DataLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "fd26fae7090e34fdef5a62091f58bed871bf8a73"
   },
   "source": [
    "Επαναλαμβάνουμε το **Βήμα 4 - Φόρτωση Δεδομένων** που βρίσκεται στο notebook της προπαρασκευής, για να δημιουργήσουμε τους Data Loaders που θα χρειαστούν για τα μοντέλα που υλοποιούμε στα υπόλοιπα βήματα."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_uuid": "bea0e1e0752e64b43adf23bdf23d44523b13913c"
   },
   "outputs": [],
   "source": [
    "def torch_train_val_split(dataset, batch_train, batch_eval,val_size=.2, shuffle=True, seed=42):\n",
    "    # Creating data indices for training and validation splits:\n",
    "    dataset_size = len(dataset)\n",
    "    indices = list(range(dataset_size))\n",
    "    val_split = int(np.floor(val_size * dataset_size))\n",
    "    if shuffle:\n",
    "        np.random.seed(seed)\n",
    "        np.random.shuffle(indices)\n",
    "    train_indices = indices[val_split:]\n",
    "    val_indices = indices[:val_split]\n",
    "\n",
    "    # Creating PT data samplers and loaders:\n",
    "    train_sampler = SubsetRandomSampler(train_indices)\n",
    "    val_sampler = SubsetRandomSampler(val_indices)\n",
    "\n",
    "    train_loader = DataLoader(dataset,\n",
    "                              batch_size=batch_train,\n",
    "                              sampler=train_sampler)\n",
    "    val_loader = DataLoader(dataset,\n",
    "                            batch_size=batch_eval,\n",
    "                            sampler=val_sampler)\n",
    "    return train_loader, val_loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_uuid": "2830483a7e9fe3a713aa7d5fd24a3c1fa5bf2d9d"
   },
   "outputs": [],
   "source": [
    "def read_spectrogram(spectrogram_file, chroma=True):\n",
    "    with gzip.GzipFile(spectrogram_file, 'r') as f:\n",
    "        spectrograms = np.load(f)\n",
    "    # spectrograms contains a fused mel spectrogram and chromagram\n",
    "    # Decompose as follows\n",
    "    return spectrograms.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_uuid": "332fbfe1d4f26de7b1d7a64e1c38eb0a69f56d94"
   },
   "outputs": [],
   "source": [
    "class LabelTransformer(LabelEncoder):\n",
    "    def inverse(self, y):\n",
    "        try:\n",
    "            return super(LabelTransformer, self).inverse_transform(y)\n",
    "        except:\n",
    "            return super(LabelTransformer, self).inverse_transform([y])\n",
    "\n",
    "    def transform(self, y):\n",
    "        try:\n",
    "            return super(LabelTransformer, self).transform(y)\n",
    "        except:\n",
    "            return super(LabelTransformer, self).transform([y])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_uuid": "4ba9e73cd5b94f322aac381102340cbb8cd274df"
   },
   "outputs": [],
   "source": [
    "class PaddingTransform(object):\n",
    "    def __init__(self, max_length, padding_value=0):\n",
    "        self.max_length = max_length\n",
    "        self.padding_value = padding_value\n",
    "\n",
    "    def __call__(self, s):\n",
    "        if len(s) == self.max_length:\n",
    "            return s\n",
    "\n",
    "        if len(s) > self.max_length:\n",
    "            return s[:self.max_length]\n",
    "\n",
    "        if len(s) < self.max_length:\n",
    "            s1 = copy.deepcopy(s)\n",
    "            pad = np.zeros((self.max_length - s.shape[0], s.shape[1]), dtype=np.float32)\n",
    "            s1 = np.vstack((s1, pad))\n",
    "            return s1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "_uuid": "fbf97f239b3f4f997954e21ac9710c481239773a"
   },
   "outputs": [],
   "source": [
    "class SpectrogramDataset(Dataset):\n",
    "    def __init__(self, path, class_mapping=None, train=True, max_length=-1):\n",
    "        t = 'train' if train else 'test'\n",
    "        p = os.path.join(path, t)\n",
    "        self.index = os.path.join(path, \"{}_labels.txt\".format(t))\n",
    "        self.files, labels = self.get_files_labels(self.index, class_mapping)\n",
    "        self.feats = [read_spectrogram(os.path.join(p, f)) for f in self.files]\n",
    "        self.feat_dim = self.feats[0].shape[1]\n",
    "        self.lengths = [len(i) for i in self.feats]\n",
    "        self.max_length = max(self.lengths) if max_length <= 0 else max_length\n",
    "        self.zero_pad_and_stack = PaddingTransform(self.max_length)\n",
    "        self.label_transformer = LabelTransformer()\n",
    "        if isinstance(labels, (list, tuple)):\n",
    "            self.labels = np.array(self.label_transformer.fit_transform(labels)).astype('int64')\n",
    "\n",
    "    def get_files_labels(self, txt, class_mapping):\n",
    "        with open(txt, 'r') as fd:\n",
    "            lines = [l.rstrip().split('\\t') for l in fd.readlines()[1:]]\n",
    "        files, labels = [], []\n",
    "        for l in lines:\n",
    "            label = l[1]\n",
    "            if class_mapping:\n",
    "                label = class_mapping[l[1]]\n",
    "            if not label:\n",
    "                continue\n",
    "            files.append(l[0])\n",
    "            labels.append(label)\n",
    "        return files, labels\n",
    "\n",
    "    def __getitem__(self, item):\n",
    "        l = min(self.lengths[item], self.max_length)\n",
    "        return self.zero_pad_and_stack(self.feats[item]), self.labels[item], l\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1b7e4a82d24202c126f3bc550c54f0c74002ca62"
   },
   "source": [
    "## **Βήμα 9 - 2D CNN** ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "334a2422040b1e9f6c091b20cc34970ab7a5c4db"
   },
   "source": [
    "Αρχικά, δημιουργούμε τους train_loader, val_loader, test_loader που θα χρησιμοποιήσουμε για την εκπαίδευση και την αξιολόγηση του 2D CNN. Επιλέγουμε για το λόγο αυτό τα fma_genre_spectrograms που βρίσκονται στο φάκελο '../input/data/data/fma_genre_spectrograms/', καθώς και το κατάλληλο batch_size. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "_uuid": "4ec866ec76170fbe7dd6f27f107d8f7fc6e71429"
   },
   "outputs": [],
   "source": [
    "specs = SpectrogramDataset('../input/data/data/fma_genre_spectrograms/', train=True, class_mapping=None, max_length=-1)\n",
    "train_loader, val_loader = torch_train_val_split(specs, 45, 45, val_size=.33)\n",
    "test_loader = DataLoader(SpectrogramDataset('../input/data/data/fma_genre_spectrograms/', train=False, class_mapping=None, max_length=-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "43e5b139e560860d9bc8b2a7ee4c0f1444d16e25"
   },
   "source": [
    "Xρησιμοποιώντας τα spectrograms της βάσης δεδομένων FMA εκπαιδεύουμε ένα 2D CNN δίκτυο, το οποίο θα δέχεται ως είσοδο τα\n",
    "σπεκτρογράμματα (train set) και θα προβλέπει τις 20 διαφορετικές κλάσεις (μουσικά είδη) του dataset. Για την εκπαίδευση χρησιμοποιούμε και validation set και ενεργοποιούμε τη GPU (χρήση cuda()).\n",
    "Συγκεκριμένα, υλοποιήσαμε:\n",
    "1. Την κλάση **CNN_d2** η οποία αποτελεί ένα 2D CNN με 4 layers που επεξεργάζεται το σπεκτρόγραμμα σαν μονοκάναλη εικόνα, και κάθε layer πραγματοποιεί με αυτή τη σειρά: 2D convolution, Batch normalization, ReLU activation, Max pooling. To 1o nn.Conv2dr έχει in_channels=1, out_channels=2, kernel_size=5, stride=1, padding=2, τo 2o nn.Conv2d έχει in_channels=2, out_channels=4, kernel_size=5, stride=1, padding=2, τo 3o nn.Conv2d έχει in_channels=4, out_channels=6, kernel_size=5, stride=1, padding=2 και τo 4o nn.Conv2d έχει in_channels=6, out_channels=8, kernel_size=5, stride=1, padding=2. Όλα τα nn.MaxPool2d έχουν kernel_size=2, stride=2. Μετά τα 4 con2d layers περνάμε την έξοδο από ένα nn.Linear για τον μετασχησμό των προβλέψεων των 20 κλάσεων. Τα παραπάνω layers εφαρμόζονται με τη σειρά στην είσοδο του νευρωνικού στη συνάρτηση forward από όπου επειστρέφεται και το output στη μορφή batch_size x num_classes (πλήθος κλάσεων).\n",
    "2. Τη συνάρτηση **eval_pred**, η οποία παράγει από την έξοδο του νευρωνικού τις προβλέψεις για τις ετικέτες κάθε δείγματος του batch. Για το σκοπό αυτό  στο output του νευρωνικού (που είναι της μορφής μέγεθος batch επί 20 κλάσεις) εφαρμόζει την torch.argmax ανά γραμμή (δηλαδή δείγμα) για να επιστραφεί η ετικέτα με την μεγαλύτερη τιμή. Επιστρέφονται οι προβλέψεις για όλα τα δείγματα, καθώς και η έξοδος του νευρωνικού.\n",
    "3. Τη συνάρτηση που περιλαμβάνει το **train loop - υπολογισμό train/validation loss ανά εποχή**. Για το πλήθος των εποχών που έχουμε ορίσει,  το μοντέλο, τον optimizer και loss function που έχουμε επιλέξει, σε κάθε μια από αυτές παίρνουμε ένα batch από τον train loader που έχουμε δημιουργήσει καλούμε το μοντέλο και βρικουμε το loss ανάμεσα στο output και τις πραγματικές ετικέτες των δειγμάτων του batch. Κάνουμε back propagation στο λάθος με την loss.backward(). Για την ανανέωση των βαρών του νευρωνικού χρησιμοποιούμε την optimizer.step(). Yπολογίζουμε το training loss σε κάθε εποχή, αθροίζοντας τα επιμέρους loss κάθε batch και διαιρώντας με το πλήθος τους και το τυπώνουμε. Σε κάθε εποχή, αφού γίνει η εκπαίδευση μέσω των batches του training set γίνεται αποτίμηση του μοντέλου για κάθε batch του validation set καλώντας την eval_pred και την accuracy_score για τις προβλέψεις των δειγμάτων. Eπιπλέον υπολογίζεται και τυπώνεται το loss στο validation set.\n",
    "\n",
    "Ορίζουμε το μοντέλο μας με num_classes = 20 (διαφορετικές ετικέτες), timesteps = 1293, num_features = 140. Χρησιμοποιούμε για loss function το nn.CrossEntropyLoss και ως optimizer το torch.optim.SGD και παραμέτρους epochs = 20, LR = 0.008, weight_decay=0.05. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "_uuid": "87e99445bf5033e171705fbb6de2b14a2b6b2f2f"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.autograd import Variable\n",
    "import math \n",
    "\n",
    "class CNN_d2(nn.Module):\n",
    "    def __init__(self, num_classes,timesteps,num_features):\n",
    "        super(CNN_d2, self).__init__()\n",
    "        self.layer1 = nn.Sequential(\n",
    "            nn.Conv2d(1, 2, kernel_size=11, stride=1, padding=5),\n",
    "            nn.BatchNorm2d(2),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(kernel_size=2)\n",
    "        )\n",
    "            \n",
    "        self.layer2 = nn.Sequential(\n",
    "            nn.Conv2d(2, 4, kernel_size=11, stride=1, padding=5),\n",
    "            nn.BatchNorm2d(4),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(kernel_size=2))\n",
    "        self.layer3 = nn.Sequential(\n",
    "            nn.Conv2d(4, 6, kernel_size=11, stride=1, padding=5),\n",
    "            nn.BatchNorm2d(6),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(kernel_size=2))\n",
    "        self.layer4 = nn.Sequential(\n",
    "            nn.Conv2d(6, 8, kernel_size=11, stride=1, padding=5),\n",
    "            nn.BatchNorm2d(8),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(kernel_size=2))\n",
    "        self.dropout = nn.Dropout(0.0001)\n",
    "        self.fc = nn.Linear(int(math.floor(timesteps/16)*math.floor(num_features/16))*8, num_classes)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        #print(x.shape)\n",
    "        out = self.layer1(x)\n",
    "        #print(out.shape)\n",
    "        out = self.layer2(out)\n",
    "        #print(out.shape)\n",
    "        out = self.layer3(out)\n",
    "        #print(out.shape)\n",
    "        out = self.layer4(out)\n",
    "        #print(out.shape)\n",
    "        out = out.reshape(out.size(0), -1)\n",
    "        out = self.dropout(out)\n",
    "        out = self.fc(out)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_uuid": "a1e82ef066d803a5fe2dca1d378e543534ecdd6d"
   },
   "outputs": [],
   "source": [
    "def eval_pred(features):\n",
    "    output_tensor = model(features.unsqueeze_(1))\n",
    "    batch_pred = torch.argmax(output_tensor.data, dim=1)\n",
    "    return batch_pred, output_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "_uuid": "11035c84b186c0578a3b28e4afd20040db91e154"
   },
   "outputs": [],
   "source": [
    "# Early stopping\n",
    "\n",
    "class EarlyStopping(object):\n",
    "    \n",
    "    \"\"\"\n",
    "    EarlyStopping can be used to stop te training if no improvement after a given number of events\n",
    "    \n",
    "    Args: \n",
    "        patience(int):\n",
    "            Number of events to wait if no improvement and then stop the training\n",
    "        \n",
    "        mode(string):\n",
    "            There are two modes:\n",
    "                min, for looking for minumums\n",
    "                max, for looking for maximums\n",
    "                \n",
    "        min_delta(float):\n",
    "            The threshold of improvement\n",
    "            \n",
    "        percentage(boolean):\n",
    "            Defines whether min_delta is a percentage or an absolute number\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, mode='min', min_delta=0, patience=10, percentage=False):\n",
    "        self.mode = mode\n",
    "        self.min_delta = min_delta\n",
    "        self.patience = patience\n",
    "        self.best = None\n",
    "        self.num_bad_epochs = 0 # counter of no events\n",
    "        self.is_better = None\n",
    "        self._init_is_better(mode, min_delta, percentage)\n",
    "\n",
    "        if patience == 0:\n",
    "            self.is_better = lambda a, b: True\n",
    "            self.step = lambda a: False\n",
    "\n",
    "    \"\"\"\n",
    "    Returns True if the Early Stopping has to be enforced, otherwise returns False.\n",
    "    \"\"\"\n",
    "    \n",
    "    def step(self, metrics):\n",
    "        if self.best is None:\n",
    "            self.best = metrics\n",
    "            return False\n",
    "\n",
    "        if np.isnan(metrics):\n",
    "            return True\n",
    "\n",
    "        if self.is_better(metrics, self.best):\n",
    "            self.num_bad_epochs = 0\n",
    "            self.best = metrics\n",
    "        else:\n",
    "            self.num_bad_epochs += 1\n",
    "\n",
    "        if self.num_bad_epochs >= self.patience:\n",
    "            return True\n",
    "\n",
    "        return False\n",
    "\n",
    "    def _init_is_better(self, mode, min_delta, percentage):\n",
    "        if mode not in {'min', 'max'}:\n",
    "            raise ValueError('mode ' + mode + ' is unknown!')\n",
    "        if not percentage:\n",
    "            if mode == 'min':\n",
    "                self.is_better = lambda a, best: a < best - min_delta\n",
    "            if mode == 'max':\n",
    "                self.is_better = lambda a, best: a > best + min_delta\n",
    "        else:\n",
    "            if mode == 'min':\n",
    "                self.is_better = lambda a, best: a < best - (\n",
    "                            best * min_delta / 100)\n",
    "            if mode == 'max':\n",
    "                self.is_better = lambda a, best: a > best + (\n",
    "                            best * min_delta / 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_uuid": "08954f84107e43a791e56983ca3f76d3a155fc8e"
   },
   "outputs": [],
   "source": [
    "def train_val_loop(epochs,model,criterion,optimizer,earlystopping):\n",
    "    \n",
    "    for epoch in range(epochs):\n",
    "        #train loop\n",
    "        train_loss = 0.0\n",
    "\n",
    "        for i, data in enumerate(train_loader):\n",
    "            features = torch.tensor(data[0]).float().cuda()\n",
    "            labels = torch.tensor(data[1]).long().cuda()\n",
    "\n",
    "            optimizer.zero_grad()           \n",
    "            output = model(features.unsqueeze_(1))\n",
    "\n",
    "            loss = criterion(output,labels)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            train_loss = train_loss + loss\n",
    "\n",
    "        num_batch_train = i+1\n",
    "\n",
    "        val_loss = 0.0\n",
    "        f1_val = 0.0\n",
    "\n",
    "        #validation loop\n",
    "        for j, data_val in enumerate(val_loader):\n",
    "            features_val = torch.tensor(data_val[0]).float().cuda()\n",
    "            labels_val = torch.tensor(data_val[1]).long().cuda()\n",
    "\n",
    "            batch_pred, output_tensor = eval_pred(features_val)\n",
    "\n",
    "            loss_val = criterion(output_tensor,labels_val)\n",
    "            val_loss = val_loss + loss_val\n",
    "\n",
    "            f1_val = f1_val + accuracy_score(labels_val.cpu(), batch_pred.cpu())\n",
    "\n",
    "        num_batch_val = j+1    \n",
    "        f1_val = f1_val/num_batch_val\n",
    "\n",
    "        print ('Epoch %d from %d, Train loss: %.2f' %(epoch + 1, epochs, train_loss/num_batch_train))\n",
    "        print ('Epoch %d from %d, Validation loss: %.2f' %(epoch + 1, epochs, val_loss/num_batch_val))\n",
    "        print('Score in validation set is: %d %%' % (100 * f1_val))\n",
    "        print('--------------------------------')\n",
    "        \n",
    "        if(earlystopping.step(f1_val) is True):\n",
    "            print('Early stopping the training cycle on epoch %d .' %(epoch+1))\n",
    "            break\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e9f2c1b4b4c4d093ea44cbf9efef8f7e6a98fe9b"
   },
   "source": [
    "Ακολουθεί το train loop, που τυπώνονται σε κάθε εποχή τα losses σε train και validation set καθώς και τα ποσοστά του μοντέλου στο validation set. Παρατηρούμε ότι το train loss μειώνεται ανά εποχή σταθερά, ενώ το validation loss μειώνεται αλλά πολύ πιο αργά και όχι τόσο σταθερά, γεγονός που μας προϊδεάζει για κακή γενίκευση του μοντέλου. Το ποσοστό στο validation loss αυξάνεται ανά εποχή και φτάνει μέχρι 27% ενώ μετά την 9 εποχή κυμαίνεται γύρω στο 25%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "_uuid": "4af5577ee72bee53d6821bc16d8de3599663b9c3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Loop for 2D CNN\n",
      "Epoch 1 from 30, Train loss: 2.85\n",
      "Epoch 1 from 30, Validation loss: 2.75\n",
      "Score in validation set is: 12 %\n",
      "--------------------------------\n",
      "Epoch 2 from 30, Train loss: 2.69\n",
      "Epoch 2 from 30, Validation loss: 2.62\n",
      "Score in validation set is: 18 %\n",
      "--------------------------------\n",
      "Epoch 3 from 30, Train loss: 2.64\n",
      "Epoch 3 from 30, Validation loss: 2.62\n",
      "Score in validation set is: 17 %\n",
      "--------------------------------\n",
      "Epoch 4 from 30, Train loss: 2.55\n",
      "Epoch 4 from 30, Validation loss: 2.59\n",
      "Score in validation set is: 18 %\n",
      "--------------------------------\n",
      "Epoch 5 from 30, Train loss: 2.46\n",
      "Epoch 5 from 30, Validation loss: 2.47\n",
      "Score in validation set is: 23 %\n",
      "--------------------------------\n",
      "Epoch 6 from 30, Train loss: 2.44\n",
      "Epoch 6 from 30, Validation loss: 2.50\n",
      "Score in validation set is: 22 %\n",
      "--------------------------------\n",
      "Epoch 7 from 30, Train loss: 2.30\n",
      "Epoch 7 from 30, Validation loss: 2.45\n",
      "Score in validation set is: 26 %\n",
      "--------------------------------\n",
      "Epoch 8 from 30, Train loss: 2.25\n",
      "Epoch 8 from 30, Validation loss: 2.41\n",
      "Score in validation set is: 23 %\n",
      "--------------------------------\n",
      "Epoch 9 from 30, Train loss: 2.19\n",
      "Epoch 9 from 30, Validation loss: 2.52\n",
      "Score in validation set is: 20 %\n",
      "--------------------------------\n",
      "Epoch 10 from 30, Train loss: 2.12\n",
      "Epoch 10 from 30, Validation loss: 2.40\n",
      "Score in validation set is: 26 %\n",
      "--------------------------------\n",
      "Epoch 11 from 30, Train loss: 2.08\n",
      "Epoch 11 from 30, Validation loss: 2.41\n",
      "Score in validation set is: 25 %\n",
      "--------------------------------\n",
      "Epoch 12 from 30, Train loss: 2.01\n",
      "Epoch 12 from 30, Validation loss: 2.42\n",
      "Score in validation set is: 25 %\n",
      "--------------------------------\n",
      "Epoch 13 from 30, Train loss: 1.94\n",
      "Epoch 13 from 30, Validation loss: 2.41\n",
      "Score in validation set is: 24 %\n",
      "--------------------------------\n",
      "Epoch 14 from 30, Train loss: 1.89\n",
      "Epoch 14 from 30, Validation loss: 2.38\n",
      "Score in validation set is: 26 %\n",
      "--------------------------------\n",
      "Epoch 15 from 30, Train loss: 1.82\n",
      "Epoch 15 from 30, Validation loss: 2.35\n",
      "Score in validation set is: 28 %\n",
      "--------------------------------\n",
      "Epoch 16 from 30, Train loss: 1.77\n",
      "Epoch 16 from 30, Validation loss: 2.35\n",
      "Score in validation set is: 28 %\n",
      "--------------------------------\n",
      "Epoch 17 from 30, Train loss: 1.70\n",
      "Epoch 17 from 30, Validation loss: 2.44\n",
      "Score in validation set is: 25 %\n",
      "--------------------------------\n",
      "Epoch 18 from 30, Train loss: 1.61\n",
      "Epoch 18 from 30, Validation loss: 2.60\n",
      "Score in validation set is: 22 %\n",
      "--------------------------------\n",
      "Epoch 19 from 30, Train loss: 1.55\n",
      "Epoch 19 from 30, Validation loss: 2.46\n",
      "Score in validation set is: 25 %\n",
      "--------------------------------\n",
      "Epoch 20 from 30, Train loss: 1.49\n",
      "Epoch 20 from 30, Validation loss: 2.52\n",
      "Score in validation set is: 24 %\n",
      "--------------------------------\n",
      "Epoch 21 from 30, Train loss: 1.38\n",
      "Epoch 21 from 30, Validation loss: 2.55\n",
      "Score in validation set is: 23 %\n",
      "--------------------------------\n",
      "Epoch 22 from 30, Train loss: 1.31\n",
      "Epoch 22 from 30, Validation loss: 2.51\n",
      "Score in validation set is: 24 %\n",
      "--------------------------------\n",
      "Epoch 23 from 30, Train loss: 1.26\n",
      "Epoch 23 from 30, Validation loss: 2.47\n",
      "Score in validation set is: 27 %\n",
      "--------------------------------\n",
      "Early stopping the training cycle on epoch 23 .\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "num_classes = 20\n",
    "timesteps = 1293\n",
    "num_features = 140\n",
    "\n",
    "model = CNN_d2(num_classes,timesteps,num_features)\n",
    "model.cuda()\n",
    "\n",
    "print('Training Loop for 2D CNN')\n",
    "\n",
    "epochs = 30\n",
    "LR = 0.008\n",
    "weight_decay=0.05\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(model.parameters(), weight_decay=weight_decay, lr=LR)\n",
    "earlystopping = EarlyStopping(mode='max', min_delta=0.01, patience=8)\n",
    "train_val_loop(epochs,model,criterion,optimizer,earlystopping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "839d5b026da882a484880d10b5d6431171282bad"
   },
   "source": [
    "\n",
    "Στη συνέχεια κάνουμε αποτίμηση του 2D CNN στο test set μετά την εκπαίδευση που προηγήθηκε και παρατηρούμε να πιάνει ένα ποσοστό της τάξης του 15%, παρά τα μεγαλύτερα ποσοστά που παρατηρήσαμε στο validation set. Η επιλογή του συγκεκριμένου μοντέλου έγινε μετά από tuning των παραμέτρων LR, weight_decay καθώς και των in, out channels τωv conv2d layers του νευρωνικού. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "_uuid": "9b1ed3a86ad66b4f5eeffff3b948dc6f39f27654"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score in test set is: 17 %\n"
     ]
    }
   ],
   "source": [
    "predictions, labels = [], []\n",
    "for j, data in enumerate(test_loader):\n",
    "    features_test = torch.tensor(data[0]).float().cuda()\n",
    "    labels_test = torch.tensor(data[1]).long().cuda()\n",
    "    \n",
    "    batch_pred, output_tensor = eval_pred(features_test)\n",
    "\n",
    "    predictions.append(batch_pred.cpu())\n",
    "    labels.append(labels_test.cpu())\n",
    "\n",
    "f1 = accuracy_score(labels, predictions)\n",
    "print('Score in test set is: %d %%' % (100 * f1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "_uuid": "72331759e5383ffbb6e39b5908aa93724150fa70"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "End of step 9\n"
     ]
    }
   ],
   "source": [
    "print(\"End of step 9\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}