Anna_KaRNNa.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anna KaRNNa\n",
    "\n",
    "In this notebook, we'll build a character-wise RNN trained on Anna Karenina, one of my all-time favorite books. It'll be able to generate new text based on the text from the book.\n",
    "\n",
    "This network is based off of Andrej Karpathy's [post on RNNs](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) and [implementation in Torch](https://github.com/karpathy/char-rnn). Also, some information [here at r2rt](http://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html) and from [Sherjil Ozair](https://github.com/sherjilozair/char-rnn-tensorflow) on GitHub. Below is the general architecture of the character-wise RNN.\n",
    "\n",
    "<img src=\"assets/charseq.jpeg\" width=\"500\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9.0\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "from collections import namedtuple\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we'll load the text file and convert it into integers for our network to use. Here I'm creating a couple dictionaries to convert the characters to and from integers. Encoding the characters as integers makes it easier to use as input in the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('anna.txt', 'r') as f:\n",
    "    text=f.read()\n",
    "vocab = set(text)\n",
    "vocab_to_int = {c: i for i, c in enumerate(vocab)}\n",
    "int_to_vocab = dict(enumerate(vocab))\n",
    "encoded = np.array([vocab_to_int[c] for c in text], dtype=np.int32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check out the first 100 characters, make sure everything is peachy. According to the [American Book Review](http://americanbookreview.org/100bestlines.asp), this is the 6th best first line of a book ever."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Chapter 1\\n\\n\\nHappy families are all alike; every unhappy family is unhappy in its own\\nway.\\n\\nEverythin'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can see the characters encoded as integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([80, 30, 76, 79,  2, 14, 42, 27, 19, 11, 11, 11, 64, 76, 79, 79, 65,\n",
       "       27, 75, 76, 47, 81, 63, 81, 14, 41, 27, 76, 42, 14, 27, 76, 63, 63,\n",
       "       27, 76, 63, 81, 68, 14, 53, 27, 14, 61, 14, 42, 65, 27,  5, 39, 30,\n",
       "       76, 79, 79, 65, 27, 75, 76, 47, 81, 63, 65, 27, 81, 41, 27,  5, 39,\n",
       "       30, 76, 79, 79, 65, 27, 81, 39, 27, 81,  2, 41, 27, 46,  1, 39, 11,\n",
       "        1, 76, 65, 70, 11, 11, 69, 61, 14, 42, 65,  2, 30, 81, 39], dtype=int32)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the network is working with individual characters, it's similar to a classification problem in which we are trying to predict the next character from the previous text.  Here's how many 'classes' our network has to pick from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "83"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making training mini-batches\n",
    "\n",
    "Here is where we'll make our mini-batches for training. Remember that we want our batches to be multiple sequences of some desired number of sequence steps. Considering a simple example, our batches would look like this:\n",
    "\n",
    "<img src=\"assets/sequence_batching@1x.png\" width=500px>\n",
    "\n",
    "\n",
    "<br>\n",
    "We have our text encoded as integers as one long array in `encoded`. Let's create a function that will give us an iterator for our batches. I like using [generator functions](https://jeffknupp.com/blog/2013/04/07/improve-your-python-yield-and-generators-explained/) to do this. Then we can pass `encoded` into this function and get our batch generator.\n",
    "\n",
    "The first thing we need to do is discard some of the text so we only have completely full batches. Each batch contains $N \\times M$ characters, where $N$ is the batch size (the number of sequences) and $M$ is the number of steps. Then, to get the number of batches we can make from some array `arr`, you divide the length of `arr` by the batch size. Once you know the number of batches and the batch size, you can get the total number of characters to keep.\n",
    "\n",
    "After that, we need to split `arr` into $N$ sequences. You can do this using `arr.reshape(size)` where `size` is a tuple containing the dimensions sizes of the reshaped array. We know we want $N$ sequences (`n_seqs` below), let's make that the size of the first dimension. For the second dimension, you can use `-1` as a placeholder in the size, it'll fill up the array with the appropriate data for you. After this, you should have an array that is $N \\times (M * K)$ where $K$ is the number of batches.\n",
    "\n",
    "Now that we have this array, we can iterate through it to get our batches. The idea is each batch is a $N \\times M$ window on the array. For each subsequent batch, the window moves over by `n_steps`. We also want to create both the input and target arrays. Remember that the targets are the inputs shifted over one character. You'll usually see the first input character used as the last target character, so something like this:\n",
    "```python\n",
    "y[:, :-1], y[:, -1] = x[:, 1:], x[:, 0]\n",
    "```\n",
    "where `x` is the input batch and `y` is the target batch.\n",
    "\n",
    "The way I like to do this window is use `range` to take steps of size `n_steps` from $0$ to `arr.shape[1]`, the total number of steps in each sequence. That way, the integers you get from `range` always point to the start of a batch, and each window is `n_steps` wide.\n",
    "\n",
    "> **Exercise:** Write the code for creating batches in the function below. The exercises in this notebook _will not be easy_. I've provided a notebook with solutions alongside this notebook. If you get stuck, checkout the solutions. The most important thing is that you don't copy and paste the code into here, **type out the solution code yourself.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_batches(arr, n_seqs, n_steps):\n",
    "    '''Create a generator that returns batches of size\n",
    "       n_seqs x n_steps from arr.\n",
    "       \n",
    "       Arguments\n",
    "       ---------\n",
    "       arr: Array you want to make batches from\n",
    "       n_seqs: Batch size, the number of sequences per batch\n",
    "       n_steps: Number of sequence steps per batch\n",
    "    '''\n",
    "    # Get the number of characters per batch and number of batches we can make\n",
    "    characters_per_batch = n_seqs*n_steps\n",
    "    n_batches = len(arr)//characters_per_batch\n",
    "    \n",
    "    # Keep only enough characters to make full batches\n",
    "    arr = arr[:n_batches * characters_per_batch]\n",
    "    \n",
    "    # Reshape into n_seqs rows\n",
    "    arr = arr.reshape((n_seqs, -1))\n",
    "    \n",
    "    for n in range(0, arr.shape[1], n_steps):\n",
    "        # The features\n",
    "        x = arr[:, n:n+n_steps]\n",
    "        # The targets, shifted by one\n",
    "        y = np.zeros_like(x)\n",
    "        y[:, :-1], y[:, -1] = x[:, 1:], x[:, 0] \n",
    "        yield x, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now I'll make my data sets and we can check out what's going on here. Here I'm going to use a batch size of 10 and 50 sequence steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batches = get_batches(encoded, 10, 50)\n",
    "x, y = next(batches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x\n",
      " [[80 30 76 79  2 14 42 27 19 11]\n",
      " [27 76 47 27 39 46  2 27 29 46]\n",
      " [61 81 39 70 11 11 52 20 14 41]\n",
      " [39 27 43  5 42 81 39 29 27 30]\n",
      " [27 81  2 27 81 41 74 27 41 81]\n",
      " [27  7  2 27  1 76 41 11 46 39]\n",
      " [30 14 39 27 34 46 47 14 27 75]\n",
      " [53 27  8  5  2 27 39 46  1 27]\n",
      " [ 2 27 81 41 39  6  2 70 27 45]\n",
      " [27 41 76 81 43 27  2 46 27 30]]\n",
      "\n",
      "y\n",
      " [[30 76 79  2 14 42 27 19 11 11]\n",
      " [76 47 27 39 46  2 27 29 46 81]\n",
      " [81 39 70 11 11 52 20 14 41 74]\n",
      " [27 43  5 42 81 39 29 27 30 81]\n",
      " [81  2 27 81 41 74 27 41 81 42]\n",
      " [ 7  2 27  1 76 41 11 46 39 63]\n",
      " [14 39 27 34 46 47 14 27 75 46]\n",
      " [27  8  5  2 27 39 46  1 27 41]\n",
      " [27 81 41 39  6  2 70 27 45 30]\n",
      " [41 76 81 43 27  2 46 27 30 14]]\n"
     ]
    }
   ],
   "source": [
    "print('x\\n', x[:10, :10])\n",
    "print('\\ny\\n', y[:10, :10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you implemented `get_batches` correctly, the above output should look something like \n",
    "```\n",
    "x\n",
    " [[55 63 69 22  6 76 45  5 16 35]\n",
    " [ 5 69  1  5 12 52  6  5 56 52]\n",
    " [48 29 12 61 35 35  8 64 76 78]\n",
    " [12  5 24 39 45 29 12 56  5 63]\n",
    " [ 5 29  6  5 29 78 28  5 78 29]\n",
    " [ 5 13  6  5 36 69 78 35 52 12]\n",
    " [63 76 12  5 18 52  1 76  5 58]\n",
    " [34  5 73 39  6  5 12 52 36  5]\n",
    " [ 6  5 29 78 12 79  6 61  5 59]\n",
    " [ 5 78 69 29 24  5  6 52  5 63]]\n",
    "\n",
    "y\n",
    " [[63 69 22  6 76 45  5 16 35 35]\n",
    " [69  1  5 12 52  6  5 56 52 29]\n",
    " [29 12 61 35 35  8 64 76 78 28]\n",
    " [ 5 24 39 45 29 12 56  5 63 29]\n",
    " [29  6  5 29 78 28  5 78 29 45]\n",
    " [13  6  5 36 69 78 35 52 12 43]\n",
    " [76 12  5 18 52  1 76  5 58 52]\n",
    " [ 5 73 39  6  5 12 52 36  5 78]\n",
    " [ 5 29 78 12 79  6 61  5 59 63]\n",
    " [78 69 29 24  5  6 52  5 63 76]]\n",
    " ```\n",
    " although the exact numbers will be different. Check to make sure the data is shifted over one step for `y`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the model\n",
    "\n",
    "Below is where you'll build the network. We'll break it up into parts so it's easier to reason about each bit. Then we can connect them up into the whole network.\n",
    "\n",
    "<img src=\"assets/charRNN.png\" width=500px>\n",
    "\n",
    "\n",
    "### Inputs\n",
    "\n",
    "First off we'll create our input placeholders. As usual we need placeholders for the training data and the targets. We'll also create a placeholder for dropout layers called `keep_prob`. This will be a scalar, that is a 0-D tensor. To make a scalar, you create a placeholder without giving it a size.\n",
    "\n",
    "> **Exercise:** Create the input placeholders in the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_inputs(batch_size, num_steps):\n",
    "    ''' Define placeholders for inputs, targets, and dropout \n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        batch_size: Batch size, number of sequences per batch\n",
    "        num_steps: Number of sequence steps in a batch\n",
    "        \n",
    "    '''\n",
    "    # Declare placeholders we'll feed into the graph\n",
    "    inputs = tf.placeholder(tf.int32, shape = (batch_size, num_steps), name='inputs')\n",
    "    targets = tf.placeholder(tf.int32, shape = (batch_size, num_steps), name='targets') \n",
    "    \n",
    "    # Keep probability placeholder for drop out layers\n",
    "    keep_prob = tf.placeholder(tf.float32, name='keep_prob') \n",
    "    \n",
    "    return inputs, targets, keep_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LSTM Cell\n",
    "\n",
    "Here we will create the LSTM cell we'll use in the hidden layer. We'll use this cell as a building block for the RNN. So we aren't actually defining the RNN here, just the type of cell we'll use in the hidden layer.\n",
    "\n",
    "We first create a basic LSTM cell with\n",
    "\n",
    "```python\n",
    "lstm = tf.contrib.rnn.BasicLSTMCell(num_units)\n",
    "```\n",
    "\n",
    "where `num_units` is the number of units in the hidden layers in the cell. Then we can add dropout by wrapping it with \n",
    "\n",
    "```python\n",
    "tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)\n",
    "```\n",
    "You pass in a cell and it will automatically add dropout to the inputs or outputs. Finally, we can stack up the LSTM cells into layers with [`tf.contrib.rnn.MultiRNNCell`](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/contrib/rnn/MultiRNNCell). With this, you pass in a list of cells and it will send the output of one cell into the next cell. For example,\n",
    "\n",
    "```python\n",
    "tf.contrib.rnn.MultiRNNCell([cell]*num_layers)\n",
    "```\n",
    "\n",
    "This might look a little weird if you know Python well because this will create a list of the same `cell` object. However, TensorFlow will create different weight matrices for all `cell` objects. Even though this is actually multiple LSTM cells stacked on each other, you can treat the multiple layers as one cell.\n",
    "\n",
    "We also need to create an initial cell state of all zeros. This can be done like so\n",
    "\n",
    "```python\n",
    "initial_state = cell.zero_state(batch_size, tf.float32)\n",
    "```\n",
    "\n",
    "> **Exercise:** Below, implement the `build_lstm` function to create these LSTM cells and the initial state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_lstm(lstm_size, num_layers, batch_size, keep_prob):\n",
    "    ''' Build LSTM cell.\n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        keep_prob: Scalar tensor (tf.placeholder) for the dropout keep probability\n",
    "        lstm_size: Size of the hidden layers in the LSTM cells\n",
    "        num_layers: Number of LSTM layers\n",
    "        batch_size: Batch size\n",
    "\n",
    "    '''\n",
    "    ### Build the LSTM Cell\n",
    "    # Use a basic LSTM cell\n",
    "    lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size)\n",
    "    \n",
    "    # Add dropout to the cell outputs\n",
    "    drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)\n",
    "    \n",
    "    # Stack up multiple LSTM layers, for deep learning\n",
    "    cell = tf.contrib.rnn.MultiRNNCell([drop]*num_layers)\n",
    "    initial_state = cell.zero_state(batch_size, tf.float32)\n",
    "    \n",
    "    return cell, initial_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RNN Output\n",
    "\n",
    "Here we'll create the output layer. We need to connect the output of the RNN cells to a full connected layer with a softmax output. The softmax output gives us a probability distribution we can use to predict the next character, so we want this layer to have size $C$, the number of classes/characters we have in our text.\n",
    "\n",
    "If our input has batch size $N$, number of steps $M$, and the hidden layer has $L$ hidden units, then the output is a 3D tensor with size $N \\times M \\times L$. The output of each LSTM cell has size $L$, we have $M$ of them, one for each sequence step, and we have $N$ sequences. So the total size is $N \\times M \\times L$. \n",
    "\n",
    "We are using the same fully connected layer, the same weights, for each of the outputs. Then, to make things easier, we should reshape the outputs into a 2D tensor with shape $(M * N) \\times L$. That is, one row for each sequence and step, where the values of each row are the output from the LSTM cells. We get the LSTM output as a list, `lstm_output`. First we need to concatenate this whole list into one array with [`tf.concat`](https://www.tensorflow.org/api_docs/python/tf/concat). Then, reshape it (with `tf.reshape`) to size $(M * N) \\times L$.\n",
    "\n",
    "One we have the outputs reshaped, we can do the matrix multiplication with the weights. We need to wrap the weight and bias variables in a variable scope with `tf.variable_scope(scope_name)` because there are weights being created in the LSTM cells. TensorFlow will throw an error if the weights created here have the same names as the weights created in the LSTM cells, which they will be default. To avoid this, we wrap the variables in a variable scope so we can give them unique names.\n",
    "\n",
    "> **Exercise:** Implement the output layer in the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_output(lstm_output, in_size, out_size):\n",
    "    ''' Build a softmax layer, return the softmax output and logits.\n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        \n",
    "        lstm_output: List of output tensors from the LSTM layer\n",
    "        in_size: Size of the input tensor, for example, size of the LSTM cells\n",
    "        out_size: Size of this softmax layer\n",
    "    \n",
    "    '''\n",
    "\n",
    "    # Reshape output so it's a bunch of rows, one row for each step for each sequence.\n",
    "    # Concatenate lstm_output over axis 1 (the columns)\n",
    "    seq_output = tf.concat(lstm_output, axis = 1)\n",
    "    # Reshape seq_output to a 2D tensor with lstm_size columns\n",
    "    x = tf.reshape(seq_output, [-1, in_size])\n",
    "    \n",
    "    # Connect the RNN outputs to a softmax layer\n",
    "    with tf.variable_scope('softmax'):\n",
    "        # Create the weight and bias variables here\n",
    "        softmax_w = tf.Variable(tf.truncated_normal(shape=(in_size, out_size),mean=0,stddev=0.1)) \n",
    "        softmax_b = tf.Variable(tf.zeros(out_size))\n",
    "    \n",
    "    # Since output is a bunch of rows of RNN cell outputs, logits will be a bunch\n",
    "    # of rows of logit outputs, one for each step and sequence\n",
    "    logits = tf.matmul(x,softmax_w) + softmax_b\n",
    "    \n",
    "    # Use softmax to get the probabilities for predicted characters\n",
    "    out = tf.nn.softmax(logits, name='predictions')\n",
    "    \n",
    "    return out, logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training loss\n",
    "\n",
    "Next up is the training loss. We get the logits and targets and calculate the softmax cross-entropy loss. First we need to one-hot encode the targets, we're getting them as encoded characters. Then, reshape the one-hot targets so it's a 2D tensor with size $(M*N) \\times C$ where $C$ is the number of classes/characters we have. Remember that we reshaped the LSTM outputs and ran them through a fully connected layer with $C$ units. So our logits will also have size $(M*N) \\times C$.\n",
    "\n",
    "Then we run the logits and targets through `tf.nn.softmax_cross_entropy_with_logits` and find the mean to get the loss.\n",
    "\n",
    ">**Exercise:** Implement the loss calculation in the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_loss(logits, targets, lstm_size, num_classes):\n",
    "    ''' Calculate the loss from the logits and the targets.\n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        logits: Logits from final fully connected layer\n",
    "        targets: Targets for supervised learning\n",
    "        lstm_size: Number of LSTM hidden units\n",
    "        num_classes: Number of classes in targets\n",
    "        \n",
    "    '''\n",
    "    \n",
    "    # One-hot encode targets and reshape to match logits, one row per sequence per step\n",
    "    y_one_hot = tf.one_hot(targets, num_classes)\n",
    "    y_reshaped = tf.reshape(y_one_hot, logits.get_shape())  \n",
    "    \n",
    "    # Softmax cross entropy loss\n",
    "    loss = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y_reshaped)\n",
    "    loss = tf.reduce_mean(loss)\n",
    "    \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizer\n",
    "\n",
    "Here we build the optimizer. Normal RNNs have have issues gradients exploding and disappearing. LSTMs fix the disappearance problem, but the gradients can still grow without bound. To fix this, we can clip the gradients above some threshold. That is, if a gradient is larger than that threshold, we set it to the threshold. This will ensure the gradients never grow overly large. Then we use an AdamOptimizer for the learning step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_optimizer(loss, learning_rate, grad_clip):\n",
    "    ''' Build optmizer for training, using gradient clipping.\n",
    "    \n",
    "        Arguments:\n",
    "        loss: Network loss\n",
    "        learning_rate: Learning rate for optimizer\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # Optimizer for training, using gradient clipping to control exploding gradients\n",
    "    tvars = tf.trainable_variables()\n",
    "    grads, _ = tf.clip_by_global_norm(tf.gradients(loss, tvars), grad_clip)\n",
    "    train_op = tf.train.AdamOptimizer(learning_rate)\n",
    "    optimizer = train_op.apply_gradients(zip(grads, tvars))\n",
    "    \n",
    "    return optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the network\n",
    "\n",
    "Now we can put all the pieces together and build a class for the network. To actually run data through the LSTM cells, we will use [`tf.nn.dynamic_rnn`](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/nn/dynamic_rnn). This function will pass the hidden and cell states across LSTM cells appropriately for us. It returns the outputs for each LSTM cell at each step for each sequence in the mini-batch. It also gives us the final LSTM state. We want to save this state as `final_state` so we can pass it to the first LSTM cell in the the next mini-batch run. For `tf.nn.dynamic_rnn`, we pass in the cell and initial state we get from `build_lstm`, as well as our input sequences. Also, we need to one-hot encode the inputs before going into the RNN. \n",
    "\n",
    "> **Exercise:** Use the functions you've implemented previously and `tf.nn.dynamic_rnn` to build the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class CharRNN:\n",
    "    \n",
    "    def __init__(self, num_classes, batch_size=64, num_steps=50, \n",
    "                       lstm_size=128, num_layers=2, learning_rate=0.001, \n",
    "                       grad_clip=5, sampling=False):\n",
    "    \n",
    "        # When we're using this network for sampling later, we'll be passing in\n",
    "        # one character at a time, so providing an option for that\n",
    "        if sampling == True:\n",
    "            batch_size, num_steps = 1, 1\n",
    "        else:\n",
    "            batch_size, num_steps = batch_size, num_steps\n",
    "\n",
    "        tf.reset_default_graph()\n",
    "        \n",
    "        # Build the input placeholder tensors\n",
    "        self.inputs, self.targets, self.keep_prob = build_inputs(batch_size, num_steps) \n",
    "\n",
    "        # Build the LSTM cell\n",
    "        cell, self.initial_state = build_lstm(lstm_size, num_layers, batch_size, keep_prob)\n",
    "\n",
    "        ### Run the data through the RNN layers\n",
    "        # First, one-hot encode the input tokens\n",
    "        x_one_hot = tf.one_hot(self.inputs, num_classes) \n",
    "        \n",
    "        # Run each sequence step through the RNN with tf.nn.dynamic_rnn \n",
    "        outputs, state = tf.nn.dynamic_rnn(cell, x_one_hot, initial_state=self.initial_state)\n",
    "        self.final_state = state\n",
    "        \n",
    "        # Get softmax predictions and logits\n",
    "        self.prediction, self.logits = build_output(outputs, lstm_size, num_classes) \n",
    "        \n",
    "        # Loss and optimizer (with gradient clipping)\n",
    "        self.loss = build_loss(self.logits, self.targets, lstm_size, num_classes)   \n",
    "        self.optimizer = build_optimizer(self.loss, learning_rate, grad_clip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters\n",
    "\n",
    "Here are the hyperparameters for the network.\n",
    "\n",
    "* `batch_size` - Number of sequences running through the network in one pass.\n",
    "* `num_steps` - Number of characters in the sequence the network is trained on. Larger is better typically, the network will learn more long range dependencies. But it takes longer to train. 100 is typically a good number here.\n",
    "* `lstm_size` - The number of units in the hidden layers.\n",
    "* `num_layers` - Number of hidden LSTM layers to use\n",
    "* `learning_rate` - Learning rate for training\n",
    "* `keep_prob` - The dropout keep probability when training. If you're network is overfitting, try decreasing this.\n",
    "\n",
    "Here's some good advice from Andrej Karpathy on training the network. I'm going to copy it in here for your benefit, but also link to [where it originally came from](https://github.com/karpathy/char-rnn#tips-and-tricks).\n",
    "\n",
    "> ## Tips and Tricks\n",
    "\n",
    ">### Monitoring Validation Loss vs. Training Loss\n",
    ">If you're somewhat new to Machine Learning or Neural Networks it can take a bit of expertise to get good models. The most important quantity to keep track of is the difference between your training loss (printed during training) and the validation loss (printed once in a while when the RNN is run on the validation data (by default every 1000 iterations)). In particular:\n",
    "\n",
    "> - If your training loss is much lower than validation loss then this means the network might be **overfitting**. Solutions to this are to decrease your network size, or to increase dropout. For example you could try dropout of 0.5 and so on.\n",
    "> - If your training/validation loss are about equal then your model is **underfitting**. Increase the size of your model (either number of layers or the raw number of neurons per layer)\n",
    "\n",
    "> ### Approximate number of parameters\n",
    "\n",
    "> The two most important parameters that control the model are `lstm_size` and `num_layers`. I would advise that you always use `num_layers` of either 2/3. The `lstm_size` can be adjusted based on how much data you have. The two important quantities to keep track of here are:\n",
    "\n",
    "> - The number of parameters in your model. This is printed when you start training.\n",
    "> - The size of your dataset. 1MB file is approximately 1 million characters.\n",
    "\n",
    ">These two should be about the same order of magnitude. It's a little tricky to tell. Here are some examples:\n",
    "\n",
    "> - I have a 100MB dataset and I'm using the default parameter settings (which currently print 150K parameters). My data size is significantly larger (100 mil >> 0.15 mil), so I expect to heavily underfit. I am thinking I can comfortably afford to make `lstm_size` larger.\n",
    "> - I have a 10MB dataset and running a 10 million parameter model. I'm slightly nervous and I'm carefully monitoring my validation loss. If it's larger than my training loss then I may want to try to increase dropout a bit and see if that helps the validation loss.\n",
    "\n",
    "> ### Best models strategy\n",
    "\n",
    ">The winning strategy to obtaining very good models (if you have the compute time) is to always err on making the network larger (as large as you're willing to wait for it to compute) and then try different dropout values (between 0,1). Whatever model has the best validation performance (the loss, written in the checkpoint filename, low is good) is the one you should use in the end.\n",
    "\n",
    ">It is very common in deep learning to run many different models with many different hyperparameter settings, and in the end take whatever checkpoint gave the best validation performance.\n",
    "\n",
    ">By the way, the size of your training and validation splits are also parameters. Make sure you have a decent amount of data in your validation set or otherwise the validation performance will be noisy and not very informative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 100         # Sequences per batch\n",
    "num_steps = 100          # Number of sequence steps per batch\n",
    "lstm_size = 512         # Size of hidden layers in LSTMs\n",
    "num_layers = 2          # Number of LSTM layers\n",
    "learning_rate = 0.001    # Learning rate\n",
    "keep_prob = 0.5         # Dropout keep probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Time for training\n",
    "\n",
    "This is typical training code, passing inputs and targets into the network, then running the optimizer. Here we also get back the final LSTM state for the mini-batch. Then, we pass that state back into the network so the next batch can continue the state from the previous batch. And every so often (set by `save_every_n`) I save a checkpoint.\n",
    "\n",
    "Here I'm saving checkpoints with the format\n",
    "\n",
    "`i{iteration number}_l{# hidden layer units}.ckpt`\n",
    "\n",
    "> **Exercise:** Set the hyperparameters above to train the network. Watch the training loss, it should be consistently dropping. Also, I highly advise running this on a GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/20...  Training Step: 1...  Training loss: 4.4180...  1.1464 sec/batch\n",
      "Epoch: 1/20...  Training Step: 2...  Training loss: 4.3272...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 3...  Training loss: 3.8027...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 4...  Training loss: 4.7899...  0.4798 sec/batch\n",
      "Epoch: 1/20...  Training Step: 5...  Training loss: 4.2755...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 6...  Training loss: 3.9487...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 7...  Training loss: 3.7821...  0.4780 sec/batch\n",
      "Epoch: 1/20...  Training Step: 8...  Training loss: 3.6120...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 9...  Training loss: 3.5143...  0.4793 sec/batch\n",
      "Epoch: 1/20...  Training Step: 10...  Training loss: 3.4748...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 11...  Training loss: 3.4247...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 12...  Training loss: 3.4088...  0.4780 sec/batch\n",
      "Epoch: 1/20...  Training Step: 13...  Training loss: 3.3876...  0.4778 sec/batch\n",
      "Epoch: 1/20...  Training Step: 14...  Training loss: 3.3854...  0.4776 sec/batch\n",
      "Epoch: 1/20...  Training Step: 15...  Training loss: 3.3469...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 16...  Training loss: 3.3416...  0.4780 sec/batch\n",
      "Epoch: 1/20...  Training Step: 17...  Training loss: 3.3147...  0.4789 sec/batch\n",
      "Epoch: 1/20...  Training Step: 18...  Training loss: 3.3417...  0.4796 sec/batch\n",
      "Epoch: 1/20...  Training Step: 19...  Training loss: 3.3040...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 20...  Training loss: 3.2595...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 21...  Training loss: 3.2871...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 22...  Training loss: 3.2750...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 23...  Training loss: 3.2738...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 24...  Training loss: 3.2605...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 25...  Training loss: 3.2504...  0.4780 sec/batch\n",
      "Epoch: 1/20...  Training Step: 26...  Training loss: 3.2569...  0.4788 sec/batch\n",
      "Epoch: 1/20...  Training Step: 27...  Training loss: 3.2529...  0.4775 sec/batch\n",
      "Epoch: 1/20...  Training Step: 28...  Training loss: 3.2253...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 29...  Training loss: 3.2277...  0.4786 sec/batch\n",
      "Epoch: 1/20...  Training Step: 30...  Training loss: 3.2368...  0.4776 sec/batch\n",
      "Epoch: 1/20...  Training Step: 31...  Training loss: 3.2564...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 32...  Training loss: 3.2231...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 33...  Training loss: 3.2022...  0.4786 sec/batch\n",
      "Epoch: 1/20...  Training Step: 34...  Training loss: 3.2354...  0.4790 sec/batch\n",
      "Epoch: 1/20...  Training Step: 35...  Training loss: 3.2072...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 36...  Training loss: 3.2297...  0.4789 sec/batch\n",
      "Epoch: 1/20...  Training Step: 37...  Training loss: 3.1859...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 38...  Training loss: 3.1919...  0.4786 sec/batch\n",
      "Epoch: 1/20...  Training Step: 39...  Training loss: 3.1827...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 40...  Training loss: 3.1950...  0.4778 sec/batch\n",
      "Epoch: 1/20...  Training Step: 41...  Training loss: 3.1827...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 42...  Training loss: 3.1885...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 43...  Training loss: 3.1765...  0.4799 sec/batch\n",
      "Epoch: 1/20...  Training Step: 44...  Training loss: 3.1761...  0.4786 sec/batch\n",
      "Epoch: 1/20...  Training Step: 45...  Training loss: 3.1683...  0.4792 sec/batch\n",
      "Epoch: 1/20...  Training Step: 46...  Training loss: 3.1801...  0.4779 sec/batch\n",
      "Epoch: 1/20...  Training Step: 47...  Training loss: 3.1835...  0.4791 sec/batch\n",
      "Epoch: 1/20...  Training Step: 48...  Training loss: 3.1882...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 49...  Training loss: 3.1858...  0.4789 sec/batch\n",
      "Epoch: 1/20...  Training Step: 50...  Training loss: 3.1734...  0.4777 sec/batch\n",
      "Epoch: 1/20...  Training Step: 51...  Training loss: 3.1710...  0.4801 sec/batch\n",
      "Epoch: 1/20...  Training Step: 52...  Training loss: 3.1636...  0.4788 sec/batch\n",
      "Epoch: 1/20...  Training Step: 53...  Training loss: 3.1768...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 54...  Training loss: 3.1523...  0.4790 sec/batch\n",
      "Epoch: 1/20...  Training Step: 55...  Training loss: 3.1690...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 56...  Training loss: 3.1492...  0.4775 sec/batch\n",
      "Epoch: 1/20...  Training Step: 57...  Training loss: 3.1539...  0.4777 sec/batch\n",
      "Epoch: 1/20...  Training Step: 58...  Training loss: 3.1574...  0.4780 sec/batch\n",
      "Epoch: 1/20...  Training Step: 59...  Training loss: 3.1466...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 60...  Training loss: 3.1544...  0.4789 sec/batch\n",
      "Epoch: 1/20...  Training Step: 61...  Training loss: 3.1536...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 62...  Training loss: 3.1712...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 63...  Training loss: 3.1754...  0.4788 sec/batch\n",
      "Epoch: 1/20...  Training Step: 64...  Training loss: 3.1299...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 65...  Training loss: 3.1367...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 66...  Training loss: 3.1581...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 67...  Training loss: 3.1562...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 68...  Training loss: 3.1158...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 69...  Training loss: 3.1300...  0.4777 sec/batch\n",
      "Epoch: 1/20...  Training Step: 70...  Training loss: 3.1548...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 71...  Training loss: 3.1405...  0.4773 sec/batch\n",
      "Epoch: 1/20...  Training Step: 72...  Training loss: 3.1570...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 73...  Training loss: 3.1377...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 74...  Training loss: 3.1509...  0.4776 sec/batch\n",
      "Epoch: 1/20...  Training Step: 75...  Training loss: 3.1479...  0.4779 sec/batch\n",
      "Epoch: 1/20...  Training Step: 76...  Training loss: 3.1522...  0.4793 sec/batch\n",
      "Epoch: 1/20...  Training Step: 77...  Training loss: 3.1452...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 78...  Training loss: 3.1377...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 79...  Training loss: 3.1335...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 80...  Training loss: 3.1230...  0.4766 sec/batch\n",
      "Epoch: 1/20...  Training Step: 81...  Training loss: 3.1252...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 82...  Training loss: 3.1386...  0.4786 sec/batch\n",
      "Epoch: 1/20...  Training Step: 83...  Training loss: 3.1393...  0.4777 sec/batch\n",
      "Epoch: 1/20...  Training Step: 84...  Training loss: 3.1258...  0.4794 sec/batch\n",
      "Epoch: 1/20...  Training Step: 85...  Training loss: 3.1103...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 86...  Training loss: 3.1124...  0.4790 sec/batch\n",
      "Epoch: 1/20...  Training Step: 87...  Training loss: 3.1066...  0.4780 sec/batch\n",
      "Epoch: 1/20...  Training Step: 88...  Training loss: 3.1125...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 89...  Training loss: 3.1250...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 90...  Training loss: 3.1195...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 91...  Training loss: 3.1176...  0.4790 sec/batch\n",
      "Epoch: 1/20...  Training Step: 92...  Training loss: 3.1064...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 93...  Training loss: 3.1091...  0.4779 sec/batch\n",
      "Epoch: 1/20...  Training Step: 94...  Training loss: 3.1240...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 95...  Training loss: 3.1202...  0.4794 sec/batch\n",
      "Epoch: 1/20...  Training Step: 96...  Training loss: 3.1284...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 97...  Training loss: 3.1411...  0.4786 sec/batch\n",
      "Epoch: 1/20...  Training Step: 98...  Training loss: 3.1198...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 99...  Training loss: 3.1288...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 100...  Training loss: 3.1106...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 101...  Training loss: 3.1211...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 102...  Training loss: 3.1135...  0.4788 sec/batch\n",
      "Epoch: 1/20...  Training Step: 103...  Training loss: 3.1191...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 104...  Training loss: 3.1116...  0.4786 sec/batch\n",
      "Epoch: 1/20...  Training Step: 105...  Training loss: 3.1017...  0.4777 sec/batch\n",
      "Epoch: 1/20...  Training Step: 106...  Training loss: 3.1010...  0.4779 sec/batch\n",
      "Epoch: 1/20...  Training Step: 107...  Training loss: 3.0815...  0.4789 sec/batch\n",
      "Epoch: 1/20...  Training Step: 108...  Training loss: 3.0892...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 109...  Training loss: 3.0916...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 110...  Training loss: 3.0569...  0.4776 sec/batch\n",
      "Epoch: 1/20...  Training Step: 111...  Training loss: 3.0826...  0.4791 sec/batch\n",
      "Epoch: 1/20...  Training Step: 112...  Training loss: 3.0911...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 113...  Training loss: 3.0699...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 114...  Training loss: 3.0635...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 115...  Training loss: 3.0529...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 116...  Training loss: 3.0579...  0.4778 sec/batch\n",
      "Epoch: 1/20...  Training Step: 117...  Training loss: 3.0574...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 118...  Training loss: 3.0699...  0.4790 sec/batch\n",
      "Epoch: 1/20...  Training Step: 119...  Training loss: 3.0692...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 120...  Training loss: 3.0442...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 121...  Training loss: 3.2308...  0.4777 sec/batch\n",
      "Epoch: 1/20...  Training Step: 122...  Training loss: 3.3101...  0.4791 sec/batch\n",
      "Epoch: 1/20...  Training Step: 123...  Training loss: 3.2899...  0.4779 sec/batch\n",
      "Epoch: 1/20...  Training Step: 124...  Training loss: 3.2644...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 125...  Training loss: 3.2081...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 126...  Training loss: 3.1752...  0.4794 sec/batch\n",
      "Epoch: 1/20...  Training Step: 127...  Training loss: 3.1471...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 128...  Training loss: 3.1378...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 129...  Training loss: 3.0557...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 130...  Training loss: 3.0443...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 131...  Training loss: 3.0448...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 132...  Training loss: 3.0169...  0.4781 sec/batch\n",
      "Epoch: 1/20...  Training Step: 133...  Training loss: 3.0285...  0.4776 sec/batch\n",
      "Epoch: 1/20...  Training Step: 134...  Training loss: 3.0058...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 135...  Training loss: 2.9941...  0.4789 sec/batch\n",
      "Epoch: 1/20...  Training Step: 136...  Training loss: 3.0239...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 137...  Training loss: 2.9983...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 138...  Training loss: 2.9945...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 139...  Training loss: 3.0144...  0.4792 sec/batch\n",
      "Epoch: 1/20...  Training Step: 140...  Training loss: 2.9813...  0.4780 sec/batch\n",
      "Epoch: 1/20...  Training Step: 141...  Training loss: 2.9982...  0.4790 sec/batch\n",
      "Epoch: 1/20...  Training Step: 142...  Training loss: 2.9552...  0.4797 sec/batch\n",
      "Epoch: 1/20...  Training Step: 143...  Training loss: 2.9591...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 144...  Training loss: 2.9535...  0.4788 sec/batch\n",
      "Epoch: 1/20...  Training Step: 145...  Training loss: 2.9586...  0.4791 sec/batch\n",
      "Epoch: 1/20...  Training Step: 146...  Training loss: 2.9433...  0.4793 sec/batch\n",
      "Epoch: 1/20...  Training Step: 147...  Training loss: 2.9493...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 148...  Training loss: 2.9647...  0.4794 sec/batch\n",
      "Epoch: 1/20...  Training Step: 149...  Training loss: 2.9075...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 150...  Training loss: 2.9233...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 151...  Training loss: 2.9459...  0.4791 sec/batch\n",
      "Epoch: 1/20...  Training Step: 152...  Training loss: 2.9346...  0.4793 sec/batch\n",
      "Epoch: 1/20...  Training Step: 153...  Training loss: 2.9060...  0.4791 sec/batch\n",
      "Epoch: 1/20...  Training Step: 154...  Training loss: 2.8946...  0.4780 sec/batch\n",
      "Epoch: 1/20...  Training Step: 155...  Training loss: 2.8680...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 156...  Training loss: 2.8629...  0.4790 sec/batch\n",
      "Epoch: 1/20...  Training Step: 157...  Training loss: 2.8455...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 158...  Training loss: 2.8561...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 159...  Training loss: 2.8279...  0.4797 sec/batch\n",
      "Epoch: 1/20...  Training Step: 160...  Training loss: 2.8418...  0.4788 sec/batch\n",
      "Epoch: 1/20...  Training Step: 161...  Training loss: 2.8470...  0.4799 sec/batch\n",
      "Epoch: 1/20...  Training Step: 162...  Training loss: 2.7915...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 163...  Training loss: 2.7827...  0.4795 sec/batch\n",
      "Epoch: 1/20...  Training Step: 164...  Training loss: 2.8034...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 165...  Training loss: 2.7971...  0.4786 sec/batch\n",
      "Epoch: 1/20...  Training Step: 166...  Training loss: 2.7742...  0.4790 sec/batch\n",
      "Epoch: 1/20...  Training Step: 167...  Training loss: 2.7759...  0.4791 sec/batch\n",
      "Epoch: 1/20...  Training Step: 168...  Training loss: 2.7651...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 169...  Training loss: 2.7621...  0.4794 sec/batch\n",
      "Epoch: 1/20...  Training Step: 170...  Training loss: 2.7386...  0.4790 sec/batch\n",
      "Epoch: 1/20...  Training Step: 171...  Training loss: 2.7523...  0.4796 sec/batch\n",
      "Epoch: 1/20...  Training Step: 172...  Training loss: 2.7686...  0.4788 sec/batch\n",
      "Epoch: 1/20...  Training Step: 173...  Training loss: 2.7752...  0.4786 sec/batch\n",
      "Epoch: 1/20...  Training Step: 174...  Training loss: 2.7931...  0.4794 sec/batch\n",
      "Epoch: 1/20...  Training Step: 175...  Training loss: 2.7695...  0.4790 sec/batch\n",
      "Epoch: 1/20...  Training Step: 176...  Training loss: 2.7128...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 177...  Training loss: 2.7040...  0.4798 sec/batch\n",
      "Epoch: 1/20...  Training Step: 178...  Training loss: 2.6801...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 179...  Training loss: 2.6679...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 180...  Training loss: 2.6592...  0.4792 sec/batch\n",
      "Epoch: 1/20...  Training Step: 181...  Training loss: 2.6669...  0.4791 sec/batch\n",
      "Epoch: 1/20...  Training Step: 182...  Training loss: 2.6647...  0.4792 sec/batch\n",
      "Epoch: 1/20...  Training Step: 183...  Training loss: 2.6384...  0.4793 sec/batch\n",
      "Epoch: 1/20...  Training Step: 184...  Training loss: 2.6534...  0.4785 sec/batch\n",
      "Epoch: 1/20...  Training Step: 185...  Training loss: 2.6810...  0.4784 sec/batch\n",
      "Epoch: 1/20...  Training Step: 186...  Training loss: 2.6246...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 187...  Training loss: 2.6146...  0.4797 sec/batch\n",
      "Epoch: 1/20...  Training Step: 188...  Training loss: 2.5862...  0.4786 sec/batch\n",
      "Epoch: 1/20...  Training Step: 189...  Training loss: 2.6006...  0.4788 sec/batch\n",
      "Epoch: 1/20...  Training Step: 190...  Training loss: 2.5998...  0.4783 sec/batch\n",
      "Epoch: 1/20...  Training Step: 191...  Training loss: 2.5984...  0.4787 sec/batch\n",
      "Epoch: 1/20...  Training Step: 192...  Training loss: 2.5601...  0.4788 sec/batch\n",
      "Epoch: 1/20...  Training Step: 193...  Training loss: 2.5769...  0.4792 sec/batch\n",
      "Epoch: 1/20...  Training Step: 194...  Training loss: 2.5812...  0.4794 sec/batch\n",
      "Epoch: 1/20...  Training Step: 195...  Training loss: 2.5574...  0.4790 sec/batch\n",
      "Epoch: 1/20...  Training Step: 196...  Training loss: 2.5562...  0.4780 sec/batch\n",
      "Epoch: 1/20...  Training Step: 197...  Training loss: 2.5506...  0.4782 sec/batch\n",
      "Epoch: 1/20...  Training Step: 198...  Training loss: 2.5469...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 199...  Training loss: 2.6118...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 200...  Training loss: 2.5166...  0.4788 sec/batch\n",
      "Epoch: 2/20...  Training Step: 201...  Training loss: 2.5300...  0.4798 sec/batch\n",
      "Epoch: 2/20...  Training Step: 202...  Training loss: 2.5350...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 203...  Training loss: 2.5376...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 204...  Training loss: 2.5335...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 205...  Training loss: 2.5316...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 206...  Training loss: 2.5272...  0.4772 sec/batch\n",
      "Epoch: 2/20...  Training Step: 207...  Training loss: 2.5480...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 208...  Training loss: 2.5212...  0.4779 sec/batch\n",
      "Epoch: 2/20...  Training Step: 209...  Training loss: 2.5027...  0.4792 sec/batch\n",
      "Epoch: 2/20...  Training Step: 210...  Training loss: 2.5181...  0.4787 sec/batch\n",
      "Epoch: 2/20...  Training Step: 211...  Training loss: 2.5081...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 212...  Training loss: 2.5359...  0.4777 sec/batch\n",
      "Epoch: 2/20...  Training Step: 213...  Training loss: 2.5058...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 214...  Training loss: 2.5026...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 215...  Training loss: 2.5008...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 216...  Training loss: 2.5348...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 217...  Training loss: 2.5008...  0.4797 sec/batch\n",
      "Epoch: 2/20...  Training Step: 218...  Training loss: 2.4686...  0.4789 sec/batch\n",
      "Epoch: 2/20...  Training Step: 219...  Training loss: 2.4893...  0.4778 sec/batch\n",
      "Epoch: 2/20...  Training Step: 220...  Training loss: 2.5215...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 221...  Training loss: 2.4872...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 222...  Training loss: 2.4791...  0.4791 sec/batch\n",
      "Epoch: 2/20...  Training Step: 223...  Training loss: 2.4655...  0.4777 sec/batch\n",
      "Epoch: 2/20...  Training Step: 224...  Training loss: 2.4693...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 225...  Training loss: 2.4735...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 226...  Training loss: 2.4618...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 227...  Training loss: 2.4717...  0.4789 sec/batch\n",
      "Epoch: 2/20...  Training Step: 228...  Training loss: 2.4677...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 229...  Training loss: 2.4815...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 230...  Training loss: 2.4350...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 231...  Training loss: 2.4339...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 232...  Training loss: 2.4481...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 233...  Training loss: 2.4342...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 234...  Training loss: 2.4573...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 235...  Training loss: 2.4333...  0.4792 sec/batch\n",
      "Epoch: 2/20...  Training Step: 236...  Training loss: 2.4068...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 237...  Training loss: 2.4194...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 238...  Training loss: 2.4336...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 239...  Training loss: 2.4104...  0.4788 sec/batch\n",
      "Epoch: 2/20...  Training Step: 240...  Training loss: 2.4071...  0.4787 sec/batch\n",
      "Epoch: 2/20...  Training Step: 241...  Training loss: 2.4062...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 242...  Training loss: 2.3910...  0.4781 sec/batch\n",
      "Epoch: 2/20...  Training Step: 243...  Training loss: 2.4021...  0.4787 sec/batch\n",
      "Epoch: 2/20...  Training Step: 244...  Training loss: 2.3680...  0.4787 sec/batch\n",
      "Epoch: 2/20...  Training Step: 245...  Training loss: 2.4253...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 246...  Training loss: 2.4064...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 247...  Training loss: 2.3990...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 248...  Training loss: 2.4231...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 249...  Training loss: 2.3841...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 250...  Training loss: 2.4037...  0.4800 sec/batch\n",
      "Epoch: 2/20...  Training Step: 251...  Training loss: 2.3948...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 252...  Training loss: 2.3796...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 253...  Training loss: 2.3748...  0.4794 sec/batch\n",
      "Epoch: 2/20...  Training Step: 254...  Training loss: 2.3892...  0.4794 sec/batch\n",
      "Epoch: 2/20...  Training Step: 255...  Training loss: 2.3799...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 256...  Training loss: 2.3702...  0.4789 sec/batch\n",
      "Epoch: 2/20...  Training Step: 257...  Training loss: 2.3641...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 258...  Training loss: 2.3895...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 259...  Training loss: 2.4214...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 260...  Training loss: 2.3989...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 261...  Training loss: 2.3921...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 262...  Training loss: 2.3762...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 263...  Training loss: 2.3674...  0.4777 sec/batch\n",
      "Epoch: 2/20...  Training Step: 264...  Training loss: 2.3847...  0.4779 sec/batch\n",
      "Epoch: 2/20...  Training Step: 265...  Training loss: 2.3657...  0.4794 sec/batch\n",
      "Epoch: 2/20...  Training Step: 266...  Training loss: 2.3327...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 267...  Training loss: 2.3400...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 268...  Training loss: 2.3623...  0.4789 sec/batch\n",
      "Epoch: 2/20...  Training Step: 269...  Training loss: 2.3777...  0.4776 sec/batch\n",
      "Epoch: 2/20...  Training Step: 270...  Training loss: 2.3625...  0.4778 sec/batch\n",
      "Epoch: 2/20...  Training Step: 271...  Training loss: 2.3510...  0.4774 sec/batch\n",
      "Epoch: 2/20...  Training Step: 272...  Training loss: 2.3394...  0.4791 sec/batch\n",
      "Epoch: 2/20...  Training Step: 273...  Training loss: 2.3424...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 274...  Training loss: 2.3780...  0.4787 sec/batch\n",
      "Epoch: 2/20...  Training Step: 275...  Training loss: 2.3317...  0.4781 sec/batch\n",
      "Epoch: 2/20...  Training Step: 276...  Training loss: 2.3538...  0.4787 sec/batch\n",
      "Epoch: 2/20...  Training Step: 277...  Training loss: 2.3118...  0.4778 sec/batch\n",
      "Epoch: 2/20...  Training Step: 278...  Training loss: 2.3291...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 279...  Training loss: 2.3063...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 280...  Training loss: 2.3395...  0.4789 sec/batch\n",
      "Epoch: 2/20...  Training Step: 281...  Training loss: 2.3042...  0.4776 sec/batch\n",
      "Epoch: 2/20...  Training Step: 282...  Training loss: 2.2963...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 283...  Training loss: 2.2821...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 284...  Training loss: 2.3081...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 285...  Training loss: 2.3135...  0.4774 sec/batch\n",
      "Epoch: 2/20...  Training Step: 286...  Training loss: 2.3087...  0.4791 sec/batch\n",
      "Epoch: 2/20...  Training Step: 287...  Training loss: 2.2859...  0.4776 sec/batch\n",
      "Epoch: 2/20...  Training Step: 288...  Training loss: 2.3096...  0.4787 sec/batch\n",
      "Epoch: 2/20...  Training Step: 289...  Training loss: 2.2972...  0.4779 sec/batch\n",
      "Epoch: 2/20...  Training Step: 290...  Training loss: 2.3081...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 291...  Training loss: 2.2837...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 292...  Training loss: 2.2827...  0.4779 sec/batch\n",
      "Epoch: 2/20...  Training Step: 293...  Training loss: 2.2834...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 294...  Training loss: 2.2756...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 295...  Training loss: 2.2850...  0.4793 sec/batch\n",
      "Epoch: 2/20...  Training Step: 296...  Training loss: 2.2917...  0.4773 sec/batch\n",
      "Epoch: 2/20...  Training Step: 297...  Training loss: 2.2707...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 298...  Training loss: 2.2586...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 299...  Training loss: 2.2920...  0.4775 sec/batch\n",
      "Epoch: 2/20...  Training Step: 300...  Training loss: 2.2887...  0.4777 sec/batch\n",
      "Epoch: 2/20...  Training Step: 301...  Training loss: 2.2614...  0.4778 sec/batch\n",
      "Epoch: 2/20...  Training Step: 302...  Training loss: 2.2593...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 303...  Training loss: 2.2585...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 304...  Training loss: 2.2714...  0.4788 sec/batch\n",
      "Epoch: 2/20...  Training Step: 305...  Training loss: 2.2621...  0.4776 sec/batch\n",
      "Epoch: 2/20...  Training Step: 306...  Training loss: 2.2973...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 307...  Training loss: 2.2766...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 308...  Training loss: 2.2497...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 309...  Training loss: 2.2651...  0.4794 sec/batch\n",
      "Epoch: 2/20...  Training Step: 310...  Training loss: 2.2801...  0.4788 sec/batch\n",
      "Epoch: 2/20...  Training Step: 311...  Training loss: 2.2651...  0.4776 sec/batch\n",
      "Epoch: 2/20...  Training Step: 312...  Training loss: 2.2515...  0.4768 sec/batch\n",
      "Epoch: 2/20...  Training Step: 313...  Training loss: 2.2444...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 314...  Training loss: 2.2069...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 315...  Training loss: 2.2496...  0.4779 sec/batch\n",
      "Epoch: 2/20...  Training Step: 316...  Training loss: 2.2491...  0.4779 sec/batch\n",
      "Epoch: 2/20...  Training Step: 317...  Training loss: 2.2775...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 318...  Training loss: 2.2496...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 319...  Training loss: 2.2722...  0.4781 sec/batch\n",
      "Epoch: 2/20...  Training Step: 320...  Training loss: 2.2321...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 321...  Training loss: 2.2322...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 322...  Training loss: 2.2649...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 323...  Training loss: 2.2306...  0.4779 sec/batch\n",
      "Epoch: 2/20...  Training Step: 324...  Training loss: 2.2103...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 325...  Training loss: 2.2522...  0.4781 sec/batch\n",
      "Epoch: 2/20...  Training Step: 326...  Training loss: 2.2446...  0.4781 sec/batch\n",
      "Epoch: 2/20...  Training Step: 327...  Training loss: 2.2411...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 328...  Training loss: 2.2307...  0.4787 sec/batch\n",
      "Epoch: 2/20...  Training Step: 329...  Training loss: 2.2087...  0.4788 sec/batch\n",
      "Epoch: 2/20...  Training Step: 330...  Training loss: 2.2005...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 331...  Training loss: 2.2370...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 332...  Training loss: 2.2341...  0.4788 sec/batch\n",
      "Epoch: 2/20...  Training Step: 333...  Training loss: 2.2269...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 334...  Training loss: 2.2179...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 335...  Training loss: 2.2260...  0.4779 sec/batch\n",
      "Epoch: 2/20...  Training Step: 336...  Training loss: 2.2137...  0.4790 sec/batch\n",
      "Epoch: 2/20...  Training Step: 337...  Training loss: 2.2514...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 338...  Training loss: 2.2127...  0.4787 sec/batch\n",
      "Epoch: 2/20...  Training Step: 339...  Training loss: 2.2453...  0.4792 sec/batch\n",
      "Epoch: 2/20...  Training Step: 340...  Training loss: 2.2088...  0.4788 sec/batch\n",
      "Epoch: 2/20...  Training Step: 341...  Training loss: 2.2082...  0.4787 sec/batch\n",
      "Epoch: 2/20...  Training Step: 342...  Training loss: 2.1958...  0.4775 sec/batch\n",
      "Epoch: 2/20...  Training Step: 343...  Training loss: 2.2107...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 344...  Training loss: 2.2332...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 345...  Training loss: 2.2084...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 346...  Training loss: 2.2313...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 347...  Training loss: 2.1968...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 348...  Training loss: 2.1881...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 349...  Training loss: 2.2050...  0.4792 sec/batch\n",
      "Epoch: 2/20...  Training Step: 350...  Training loss: 2.2280...  0.4795 sec/batch\n",
      "Epoch: 2/20...  Training Step: 351...  Training loss: 2.2096...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 352...  Training loss: 2.1963...  0.4788 sec/batch\n",
      "Epoch: 2/20...  Training Step: 353...  Training loss: 2.1870...  0.4795 sec/batch\n",
      "Epoch: 2/20...  Training Step: 354...  Training loss: 2.1850...  0.4781 sec/batch\n",
      "Epoch: 2/20...  Training Step: 355...  Training loss: 2.1839...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 356...  Training loss: 2.1843...  0.4801 sec/batch\n",
      "Epoch: 2/20...  Training Step: 357...  Training loss: 2.1575...  0.4789 sec/batch\n",
      "Epoch: 2/20...  Training Step: 358...  Training loss: 2.2134...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 359...  Training loss: 2.1922...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 360...  Training loss: 2.1729...  0.4791 sec/batch\n",
      "Epoch: 2/20...  Training Step: 361...  Training loss: 2.1805...  0.4788 sec/batch\n",
      "Epoch: 2/20...  Training Step: 362...  Training loss: 2.1768...  0.4794 sec/batch\n",
      "Epoch: 2/20...  Training Step: 363...  Training loss: 2.1853...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 364...  Training loss: 2.1755...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 365...  Training loss: 2.1828...  0.4778 sec/batch\n",
      "Epoch: 2/20...  Training Step: 366...  Training loss: 2.2003...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 367...  Training loss: 2.1677...  0.4786 sec/batch\n",
      "Epoch: 2/20...  Training Step: 368...  Training loss: 2.1582...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 369...  Training loss: 2.1562...  0.4790 sec/batch\n",
      "Epoch: 2/20...  Training Step: 370...  Training loss: 2.1711...  0.4781 sec/batch\n",
      "Epoch: 2/20...  Training Step: 371...  Training loss: 2.1841...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 372...  Training loss: 2.1783...  0.4782 sec/batch\n",
      "Epoch: 2/20...  Training Step: 373...  Training loss: 2.1708...  0.4790 sec/batch\n",
      "Epoch: 2/20...  Training Step: 374...  Training loss: 2.1534...  0.4781 sec/batch\n",
      "Epoch: 2/20...  Training Step: 375...  Training loss: 2.1433...  0.4789 sec/batch\n",
      "Epoch: 2/20...  Training Step: 376...  Training loss: 2.1586...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 377...  Training loss: 2.1354...  0.4778 sec/batch\n",
      "Epoch: 2/20...  Training Step: 378...  Training loss: 2.1171...  0.4792 sec/batch\n",
      "Epoch: 2/20...  Training Step: 379...  Training loss: 2.1365...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 380...  Training loss: 2.1499...  0.4788 sec/batch\n",
      "Epoch: 2/20...  Training Step: 381...  Training loss: 2.1482...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 382...  Training loss: 2.1661...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 383...  Training loss: 2.1557...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 384...  Training loss: 2.1291...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 385...  Training loss: 2.1318...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 386...  Training loss: 2.1150...  0.4790 sec/batch\n",
      "Epoch: 2/20...  Training Step: 387...  Training loss: 2.1192...  0.4789 sec/batch\n",
      "Epoch: 2/20...  Training Step: 388...  Training loss: 2.1286...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 389...  Training loss: 2.1400...  0.4778 sec/batch\n",
      "Epoch: 2/20...  Training Step: 390...  Training loss: 2.1130...  0.4789 sec/batch\n",
      "Epoch: 2/20...  Training Step: 391...  Training loss: 2.1428...  0.4780 sec/batch\n",
      "Epoch: 2/20...  Training Step: 392...  Training loss: 2.1229...  0.4783 sec/batch\n",
      "Epoch: 2/20...  Training Step: 393...  Training loss: 2.1037...  0.4784 sec/batch\n",
      "Epoch: 2/20...  Training Step: 394...  Training loss: 2.1370...  0.4779 sec/batch\n",
      "Epoch: 2/20...  Training Step: 395...  Training loss: 2.1179...  0.4785 sec/batch\n",
      "Epoch: 2/20...  Training Step: 396...  Training loss: 2.1105...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 397...  Training loss: 2.1975...  0.4779 sec/batch\n",
      "Epoch: 3/20...  Training Step: 398...  Training loss: 2.1044...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 399...  Training loss: 2.0937...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 400...  Training loss: 2.1146...  0.4792 sec/batch\n",
      "Epoch: 3/20...  Training Step: 401...  Training loss: 2.1092...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 402...  Training loss: 2.0806...  0.4791 sec/batch\n",
      "Epoch: 3/20...  Training Step: 403...  Training loss: 2.1174...  0.4777 sec/batch\n",
      "Epoch: 3/20...  Training Step: 404...  Training loss: 2.1078...  0.4778 sec/batch\n",
      "Epoch: 3/20...  Training Step: 405...  Training loss: 2.1474...  0.4791 sec/batch\n",
      "Epoch: 3/20...  Training Step: 406...  Training loss: 2.1099...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 407...  Training loss: 2.0894...  0.4780 sec/batch\n",
      "Epoch: 3/20...  Training Step: 408...  Training loss: 2.0912...  0.4794 sec/batch\n",
      "Epoch: 3/20...  Training Step: 409...  Training loss: 2.0998...  0.4792 sec/batch\n",
      "Epoch: 3/20...  Training Step: 410...  Training loss: 2.1394...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 411...  Training loss: 2.0915...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 412...  Training loss: 2.0843...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 413...  Training loss: 2.0887...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 414...  Training loss: 2.1334...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 415...  Training loss: 2.0951...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 416...  Training loss: 2.0986...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 417...  Training loss: 2.0833...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 418...  Training loss: 2.1264...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 419...  Training loss: 2.0945...  0.4780 sec/batch\n",
      "Epoch: 3/20...  Training Step: 420...  Training loss: 2.0775...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 421...  Training loss: 2.0872...  0.4789 sec/batch\n",
      "Epoch: 3/20...  Training Step: 422...  Training loss: 2.0607...  0.4791 sec/batch\n",
      "Epoch: 3/20...  Training Step: 423...  Training loss: 2.0664...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 424...  Training loss: 2.0895...  0.4791 sec/batch\n",
      "Epoch: 3/20...  Training Step: 425...  Training loss: 2.1080...  0.4778 sec/batch\n",
      "Epoch: 3/20...  Training Step: 426...  Training loss: 2.0961...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 427...  Training loss: 2.0882...  0.4789 sec/batch\n",
      "Epoch: 3/20...  Training Step: 428...  Training loss: 2.0546...  0.4788 sec/batch\n",
      "Epoch: 3/20...  Training Step: 429...  Training loss: 2.0764...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 430...  Training loss: 2.1057...  0.4791 sec/batch\n",
      "Epoch: 3/20...  Training Step: 431...  Training loss: 2.0639...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 432...  Training loss: 2.0715...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 433...  Training loss: 2.0690...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 434...  Training loss: 2.0304...  0.4789 sec/batch\n",
      "Epoch: 3/20...  Training Step: 435...  Training loss: 2.0277...  0.4780 sec/batch\n",
      "Epoch: 3/20...  Training Step: 436...  Training loss: 2.0337...  0.4775 sec/batch\n",
      "Epoch: 3/20...  Training Step: 437...  Training loss: 2.0456...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 438...  Training loss: 2.0657...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 439...  Training loss: 2.0486...  0.4780 sec/batch\n",
      "Epoch: 3/20...  Training Step: 440...  Training loss: 2.0308...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 441...  Training loss: 2.0619...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 442...  Training loss: 2.0030...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 443...  Training loss: 2.0613...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 444...  Training loss: 2.0326...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 445...  Training loss: 2.0358...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 446...  Training loss: 2.0865...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 447...  Training loss: 2.0047...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 448...  Training loss: 2.0906...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 449...  Training loss: 2.0352...  0.4789 sec/batch\n",
      "Epoch: 3/20...  Training Step: 450...  Training loss: 2.0428...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 451...  Training loss: 2.0293...  0.4794 sec/batch\n",
      "Epoch: 3/20...  Training Step: 452...  Training loss: 2.0492...  0.4792 sec/batch\n",
      "Epoch: 3/20...  Training Step: 453...  Training loss: 2.0403...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 454...  Training loss: 2.0471...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 455...  Training loss: 2.0297...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 456...  Training loss: 2.0757...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 457...  Training loss: 2.0359...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 458...  Training loss: 2.0775...  0.4792 sec/batch\n",
      "Epoch: 3/20...  Training Step: 459...  Training loss: 2.0687...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 460...  Training loss: 2.0519...  0.4796 sec/batch\n",
      "Epoch: 3/20...  Training Step: 461...  Training loss: 2.0279...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 462...  Training loss: 2.0588...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 463...  Training loss: 2.0395...  0.4794 sec/batch\n",
      "Epoch: 3/20...  Training Step: 464...  Training loss: 2.0076...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 465...  Training loss: 2.0206...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 466...  Training loss: 2.0264...  0.4774 sec/batch\n",
      "Epoch: 3/20...  Training Step: 467...  Training loss: 2.0663...  0.4773 sec/batch\n",
      "Epoch: 3/20...  Training Step: 468...  Training loss: 2.0309...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 469...  Training loss: 2.0532...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 470...  Training loss: 2.0234...  0.4780 sec/batch\n",
      "Epoch: 3/20...  Training Step: 471...  Training loss: 2.0228...  0.4788 sec/batch\n",
      "Epoch: 3/20...  Training Step: 472...  Training loss: 2.0645...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 473...  Training loss: 2.0250...  0.4779 sec/batch\n",
      "Epoch: 3/20...  Training Step: 474...  Training loss: 2.0301...  0.4776 sec/batch\n",
      "Epoch: 3/20...  Training Step: 475...  Training loss: 1.9833...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 476...  Training loss: 2.0122...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 477...  Training loss: 1.9802...  0.4789 sec/batch\n",
      "Epoch: 3/20...  Training Step: 478...  Training loss: 2.0367...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 479...  Training loss: 1.9840...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 480...  Training loss: 2.0101...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 481...  Training loss: 1.9786...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 482...  Training loss: 1.9933...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 483...  Training loss: 2.0104...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 484...  Training loss: 1.9825...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 485...  Training loss: 1.9757...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 486...  Training loss: 2.0103...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 487...  Training loss: 1.9858...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 488...  Training loss: 2.0014...  0.4788 sec/batch\n",
      "Epoch: 3/20...  Training Step: 489...  Training loss: 1.9684...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 490...  Training loss: 1.9784...  0.4780 sec/batch\n",
      "Epoch: 3/20...  Training Step: 491...  Training loss: 1.9716...  0.4779 sec/batch\n",
      "Epoch: 3/20...  Training Step: 492...  Training loss: 1.9957...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 493...  Training loss: 1.9888...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 494...  Training loss: 1.9805...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 495...  Training loss: 1.9674...  0.4793 sec/batch\n",
      "Epoch: 3/20...  Training Step: 496...  Training loss: 1.9475...  0.4800 sec/batch\n",
      "Epoch: 3/20...  Training Step: 497...  Training loss: 1.9945...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 498...  Training loss: 1.9990...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 499...  Training loss: 1.9741...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 500...  Training loss: 1.9742...  0.4777 sec/batch\n",
      "Epoch: 3/20...  Training Step: 501...  Training loss: 1.9769...  0.4789 sec/batch\n",
      "Epoch: 3/20...  Training Step: 502...  Training loss: 1.9857...  0.4791 sec/batch\n",
      "Epoch: 3/20...  Training Step: 503...  Training loss: 1.9787...  0.4795 sec/batch\n",
      "Epoch: 3/20...  Training Step: 504...  Training loss: 1.9810...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 505...  Training loss: 1.9950...  0.4788 sec/batch\n",
      "Epoch: 3/20...  Training Step: 506...  Training loss: 1.9823...  0.4788 sec/batch\n",
      "Epoch: 3/20...  Training Step: 507...  Training loss: 1.9838...  0.4788 sec/batch\n",
      "Epoch: 3/20...  Training Step: 508...  Training loss: 1.9678...  0.4792 sec/batch\n",
      "Epoch: 3/20...  Training Step: 509...  Training loss: 1.9834...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 510...  Training loss: 1.9694...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 511...  Training loss: 1.9602...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 512...  Training loss: 1.9292...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 513...  Training loss: 1.9701...  0.4788 sec/batch\n",
      "Epoch: 3/20...  Training Step: 514...  Training loss: 1.9580...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 515...  Training loss: 1.9791...  0.4789 sec/batch\n",
      "Epoch: 3/20...  Training Step: 516...  Training loss: 1.9688...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 517...  Training loss: 1.9831...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 518...  Training loss: 1.9488...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 519...  Training loss: 1.9421...  0.4791 sec/batch\n",
      "Epoch: 3/20...  Training Step: 520...  Training loss: 1.9915...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 521...  Training loss: 1.9624...  0.4792 sec/batch\n",
      "Epoch: 3/20...  Training Step: 522...  Training loss: 1.9242...  0.4788 sec/batch\n",
      "Epoch: 3/20...  Training Step: 523...  Training loss: 1.9783...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 524...  Training loss: 1.9831...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 525...  Training loss: 1.9675...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 526...  Training loss: 1.9764...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 527...  Training loss: 1.9397...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 528...  Training loss: 1.9375...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 529...  Training loss: 1.9691...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 530...  Training loss: 1.9709...  0.4789 sec/batch\n",
      "Epoch: 3/20...  Training Step: 531...  Training loss: 1.9669...  0.4794 sec/batch\n",
      "Epoch: 3/20...  Training Step: 532...  Training loss: 1.9660...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 533...  Training loss: 1.9698...  0.4795 sec/batch\n",
      "Epoch: 3/20...  Training Step: 534...  Training loss: 1.9552...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 535...  Training loss: 1.9967...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 536...  Training loss: 1.9437...  0.4789 sec/batch\n",
      "Epoch: 3/20...  Training Step: 537...  Training loss: 1.9800...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 538...  Training loss: 1.9431...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 539...  Training loss: 1.9615...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 540...  Training loss: 1.9493...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 541...  Training loss: 1.9398...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 542...  Training loss: 1.9668...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 543...  Training loss: 1.9593...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 544...  Training loss: 1.9757...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 545...  Training loss: 1.9537...  0.4789 sec/batch\n",
      "Epoch: 3/20...  Training Step: 546...  Training loss: 1.9302...  0.4779 sec/batch\n",
      "Epoch: 3/20...  Training Step: 547...  Training loss: 1.9240...  0.4779 sec/batch\n",
      "Epoch: 3/20...  Training Step: 548...  Training loss: 1.9761...  0.4780 sec/batch\n",
      "Epoch: 3/20...  Training Step: 549...  Training loss: 1.9493...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 550...  Training loss: 1.9527...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 551...  Training loss: 1.9348...  0.4798 sec/batch\n",
      "Epoch: 3/20...  Training Step: 552...  Training loss: 1.9293...  0.4795 sec/batch\n",
      "Epoch: 3/20...  Training Step: 553...  Training loss: 1.9440...  0.4789 sec/batch\n",
      "Epoch: 3/20...  Training Step: 554...  Training loss: 1.9363...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 555...  Training loss: 1.9047...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 556...  Training loss: 1.9671...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 557...  Training loss: 1.9690...  0.4788 sec/batch\n",
      "Epoch: 3/20...  Training Step: 558...  Training loss: 1.9327...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 559...  Training loss: 1.9560...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 560...  Training loss: 1.9370...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 561...  Training loss: 1.9285...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 562...  Training loss: 1.9289...  0.4767 sec/batch\n",
      "Epoch: 3/20...  Training Step: 563...  Training loss: 1.9510...  0.4788 sec/batch\n",
      "Epoch: 3/20...  Training Step: 564...  Training loss: 1.9738...  0.4793 sec/batch\n",
      "Epoch: 3/20...  Training Step: 565...  Training loss: 1.9203...  0.4774 sec/batch\n",
      "Epoch: 3/20...  Training Step: 566...  Training loss: 1.9256...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 567...  Training loss: 1.9033...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 568...  Training loss: 1.9144...  0.4792 sec/batch\n",
      "Epoch: 3/20...  Training Step: 569...  Training loss: 1.9467...  0.4792 sec/batch\n",
      "Epoch: 3/20...  Training Step: 570...  Training loss: 1.9276...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 571...  Training loss: 1.9274...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 572...  Training loss: 1.9121...  0.4788 sec/batch\n",
      "Epoch: 3/20...  Training Step: 573...  Training loss: 1.9073...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 574...  Training loss: 1.9303...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 575...  Training loss: 1.8941...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 576...  Training loss: 1.8958...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 577...  Training loss: 1.9051...  0.4796 sec/batch\n",
      "Epoch: 3/20...  Training Step: 578...  Training loss: 1.9269...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 579...  Training loss: 1.9217...  0.4775 sec/batch\n",
      "Epoch: 3/20...  Training Step: 580...  Training loss: 1.9333...  0.4778 sec/batch\n",
      "Epoch: 3/20...  Training Step: 581...  Training loss: 1.9234...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 582...  Training loss: 1.9076...  0.4779 sec/batch\n",
      "Epoch: 3/20...  Training Step: 583...  Training loss: 1.9190...  0.4780 sec/batch\n",
      "Epoch: 3/20...  Training Step: 584...  Training loss: 1.8855...  0.4781 sec/batch\n",
      "Epoch: 3/20...  Training Step: 585...  Training loss: 1.8981...  0.4786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 586...  Training loss: 1.9083...  0.4792 sec/batch\n",
      "Epoch: 3/20...  Training Step: 587...  Training loss: 1.9254...  0.4782 sec/batch\n",
      "Epoch: 3/20...  Training Step: 588...  Training loss: 1.8877...  0.4783 sec/batch\n",
      "Epoch: 3/20...  Training Step: 589...  Training loss: 1.9091...  0.4784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 590...  Training loss: 1.8918...  0.4790 sec/batch\n",
      "Epoch: 3/20...  Training Step: 591...  Training loss: 1.8783...  0.4787 sec/batch\n",
      "Epoch: 3/20...  Training Step: 592...  Training loss: 1.9104...  0.4791 sec/batch\n",
      "Epoch: 3/20...  Training Step: 593...  Training loss: 1.8996...  0.4785 sec/batch\n",
      "Epoch: 3/20...  Training Step: 594...  Training loss: 1.8829...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 595...  Training loss: 1.9855...  0.4789 sec/batch\n",
      "Epoch: 4/20...  Training Step: 596...  Training loss: 1.8860...  0.4793 sec/batch\n",
      "Epoch: 4/20...  Training Step: 597...  Training loss: 1.8867...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 598...  Training loss: 1.8981...  0.4789 sec/batch\n",
      "Epoch: 4/20...  Training Step: 599...  Training loss: 1.8864...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 600...  Training loss: 1.8554...  0.4795 sec/batch\n",
      "Epoch: 4/20...  Training Step: 601...  Training loss: 1.9060...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 602...  Training loss: 1.8842...  0.4793 sec/batch\n",
      "Epoch: 4/20...  Training Step: 603...  Training loss: 1.9269...  0.4781 sec/batch\n",
      "Epoch: 4/20...  Training Step: 604...  Training loss: 1.8950...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 605...  Training loss: 1.8663...  0.4777 sec/batch\n",
      "Epoch: 4/20...  Training Step: 606...  Training loss: 1.8744...  0.4788 sec/batch\n",
      "Epoch: 4/20...  Training Step: 607...  Training loss: 1.8949...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 608...  Training loss: 1.9213...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 609...  Training loss: 1.8802...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 610...  Training loss: 1.8651...  0.4778 sec/batch\n",
      "Epoch: 4/20...  Training Step: 611...  Training loss: 1.8940...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 612...  Training loss: 1.9250...  0.4791 sec/batch\n",
      "Epoch: 4/20...  Training Step: 613...  Training loss: 1.8836...  0.4779 sec/batch\n",
      "Epoch: 4/20...  Training Step: 614...  Training loss: 1.8890...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 615...  Training loss: 1.8721...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 616...  Training loss: 1.9207...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 617...  Training loss: 1.8804...  0.4781 sec/batch\n",
      "Epoch: 4/20...  Training Step: 618...  Training loss: 1.8861...  0.4791 sec/batch\n",
      "Epoch: 4/20...  Training Step: 619...  Training loss: 1.8791...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 620...  Training loss: 1.8575...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 621...  Training loss: 1.8547...  0.4779 sec/batch\n",
      "Epoch: 4/20...  Training Step: 622...  Training loss: 1.8916...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 623...  Training loss: 1.9009...  0.4779 sec/batch\n",
      "Epoch: 4/20...  Training Step: 624...  Training loss: 1.8809...  0.4780 sec/batch\n",
      "Epoch: 4/20...  Training Step: 625...  Training loss: 1.8783...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 626...  Training loss: 1.8505...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 627...  Training loss: 1.8811...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 628...  Training loss: 1.8950...  0.4792 sec/batch\n",
      "Epoch: 4/20...  Training Step: 629...  Training loss: 1.8585...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 630...  Training loss: 1.8653...  0.4801 sec/batch\n",
      "Epoch: 4/20...  Training Step: 631...  Training loss: 1.8539...  0.4791 sec/batch\n",
      "Epoch: 4/20...  Training Step: 632...  Training loss: 1.8363...  0.4789 sec/batch\n",
      "Epoch: 4/20...  Training Step: 633...  Training loss: 1.8255...  0.4790 sec/batch\n",
      "Epoch: 4/20...  Training Step: 634...  Training loss: 1.8398...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 635...  Training loss: 1.8398...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 636...  Training loss: 1.8760...  0.4780 sec/batch\n",
      "Epoch: 4/20...  Training Step: 637...  Training loss: 1.8379...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 638...  Training loss: 1.8296...  0.4788 sec/batch\n",
      "Epoch: 4/20...  Training Step: 639...  Training loss: 1.8710...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 640...  Training loss: 1.8148...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 641...  Training loss: 1.8554...  0.4788 sec/batch\n",
      "Epoch: 4/20...  Training Step: 642...  Training loss: 1.8358...  0.4792 sec/batch\n",
      "Epoch: 4/20...  Training Step: 643...  Training loss: 1.8464...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 644...  Training loss: 1.8955...  0.4777 sec/batch\n",
      "Epoch: 4/20...  Training Step: 645...  Training loss: 1.8316...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 646...  Training loss: 1.9090...  0.4775 sec/batch\n",
      "Epoch: 4/20...  Training Step: 647...  Training loss: 1.8450...  0.4781 sec/batch\n",
      "Epoch: 4/20...  Training Step: 648...  Training loss: 1.8589...  0.4781 sec/batch\n",
      "Epoch: 4/20...  Training Step: 649...  Training loss: 1.8452...  0.4780 sec/batch\n",
      "Epoch: 4/20...  Training Step: 650...  Training loss: 1.8568...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 651...  Training loss: 1.8579...  0.4792 sec/batch\n",
      "Epoch: 4/20...  Training Step: 652...  Training loss: 1.8472...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 653...  Training loss: 1.8199...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 654...  Training loss: 1.8918...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 655...  Training loss: 1.8450...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 656...  Training loss: 1.8859...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 657...  Training loss: 1.8821...  0.4780 sec/batch\n",
      "Epoch: 4/20...  Training Step: 658...  Training loss: 1.8679...  0.4789 sec/batch\n",
      "Epoch: 4/20...  Training Step: 659...  Training loss: 1.8387...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 660...  Training loss: 1.8739...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 661...  Training loss: 1.8608...  0.4796 sec/batch\n",
      "Epoch: 4/20...  Training Step: 662...  Training loss: 1.8226...  0.4781 sec/batch\n",
      "Epoch: 4/20...  Training Step: 663...  Training loss: 1.8300...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 664...  Training loss: 1.8353...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 665...  Training loss: 1.8832...  0.4779 sec/batch\n",
      "Epoch: 4/20...  Training Step: 666...  Training loss: 1.8539...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 667...  Training loss: 1.8590...  0.4798 sec/batch\n",
      "Epoch: 4/20...  Training Step: 668...  Training loss: 1.8334...  0.4781 sec/batch\n",
      "Epoch: 4/20...  Training Step: 669...  Training loss: 1.8422...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 670...  Training loss: 1.8760...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 671...  Training loss: 1.8418...  0.4778 sec/batch\n",
      "Epoch: 4/20...  Training Step: 672...  Training loss: 1.8487...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 673...  Training loss: 1.8065...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 674...  Training loss: 1.8286...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 675...  Training loss: 1.7935...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 676...  Training loss: 1.8471...  0.4789 sec/batch\n",
      "Epoch: 4/20...  Training Step: 677...  Training loss: 1.7876...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 678...  Training loss: 1.8324...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 679...  Training loss: 1.7984...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 680...  Training loss: 1.8106...  0.4788 sec/batch\n",
      "Epoch: 4/20...  Training Step: 681...  Training loss: 1.8090...  0.4790 sec/batch\n",
      "Epoch: 4/20...  Training Step: 682...  Training loss: 1.8102...  0.4789 sec/batch\n",
      "Epoch: 4/20...  Training Step: 683...  Training loss: 1.7948...  0.4777 sec/batch\n",
      "Epoch: 4/20...  Training Step: 684...  Training loss: 1.8388...  0.4789 sec/batch\n",
      "Epoch: 4/20...  Training Step: 685...  Training loss: 1.8056...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 686...  Training loss: 1.8121...  0.4792 sec/batch\n",
      "Epoch: 4/20...  Training Step: 687...  Training loss: 1.7854...  0.4791 sec/batch\n",
      "Epoch: 4/20...  Training Step: 688...  Training loss: 1.8077...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 689...  Training loss: 1.7860...  0.4780 sec/batch\n",
      "Epoch: 4/20...  Training Step: 690...  Training loss: 1.8190...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 691...  Training loss: 1.8193...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 692...  Training loss: 1.7867...  0.4791 sec/batch\n",
      "Epoch: 4/20...  Training Step: 693...  Training loss: 1.8042...  0.4791 sec/batch\n",
      "Epoch: 4/20...  Training Step: 694...  Training loss: 1.7753...  0.4788 sec/batch\n",
      "Epoch: 4/20...  Training Step: 695...  Training loss: 1.8196...  0.4790 sec/batch\n",
      "Epoch: 4/20...  Training Step: 696...  Training loss: 1.8148...  0.4796 sec/batch\n",
      "Epoch: 4/20...  Training Step: 697...  Training loss: 1.8061...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 698...  Training loss: 1.8013...  0.4797 sec/batch\n",
      "Epoch: 4/20...  Training Step: 699...  Training loss: 1.7940...  0.4788 sec/batch\n",
      "Epoch: 4/20...  Training Step: 700...  Training loss: 1.8079...  0.4799 sec/batch\n",
      "Epoch: 4/20...  Training Step: 701...  Training loss: 1.8156...  0.4789 sec/batch\n",
      "Epoch: 4/20...  Training Step: 702...  Training loss: 1.8168...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 703...  Training loss: 1.8247...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 704...  Training loss: 1.8228...  0.4790 sec/batch\n",
      "Epoch: 4/20...  Training Step: 705...  Training loss: 1.8104...  0.4793 sec/batch\n",
      "Epoch: 4/20...  Training Step: 706...  Training loss: 1.8080...  0.4789 sec/batch\n",
      "Epoch: 4/20...  Training Step: 707...  Training loss: 1.8023...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 708...  Training loss: 1.8048...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 709...  Training loss: 1.7804...  0.4794 sec/batch\n",
      "Epoch: 4/20...  Training Step: 710...  Training loss: 1.7674...  0.4781 sec/batch\n",
      "Epoch: 4/20...  Training Step: 711...  Training loss: 1.8098...  0.4795 sec/batch\n",
      "Epoch: 4/20...  Training Step: 712...  Training loss: 1.7960...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 713...  Training loss: 1.8051...  0.4788 sec/batch\n",
      "Epoch: 4/20...  Training Step: 714...  Training loss: 1.7964...  0.4777 sec/batch\n",
      "Epoch: 4/20...  Training Step: 715...  Training loss: 1.8164...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 716...  Training loss: 1.7733...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 717...  Training loss: 1.7718...  0.4781 sec/batch\n",
      "Epoch: 4/20...  Training Step: 718...  Training loss: 1.8237...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 719...  Training loss: 1.7974...  0.4779 sec/batch\n",
      "Epoch: 4/20...  Training Step: 720...  Training loss: 1.7547...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 721...  Training loss: 1.8199...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 722...  Training loss: 1.8070...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 723...  Training loss: 1.7937...  0.4793 sec/batch\n",
      "Epoch: 4/20...  Training Step: 724...  Training loss: 1.8010...  0.4773 sec/batch\n",
      "Epoch: 4/20...  Training Step: 725...  Training loss: 1.7602...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 726...  Training loss: 1.7604...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 727...  Training loss: 1.8097...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 728...  Training loss: 1.8028...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 729...  Training loss: 1.7973...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 730...  Training loss: 1.7938...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 731...  Training loss: 1.8112...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 732...  Training loss: 1.7980...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 733...  Training loss: 1.8186...  0.4792 sec/batch\n",
      "Epoch: 4/20...  Training Step: 734...  Training loss: 1.7911...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 735...  Training loss: 1.8368...  0.4781 sec/batch\n",
      "Epoch: 4/20...  Training Step: 736...  Training loss: 1.7843...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 737...  Training loss: 1.7946...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 738...  Training loss: 1.7994...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 739...  Training loss: 1.7810...  0.4788 sec/batch\n",
      "Epoch: 4/20...  Training Step: 740...  Training loss: 1.8152...  0.4794 sec/batch\n",
      "Epoch: 4/20...  Training Step: 741...  Training loss: 1.8012...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 742...  Training loss: 1.8284...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 743...  Training loss: 1.7952...  0.4790 sec/batch\n",
      "Epoch: 4/20...  Training Step: 744...  Training loss: 1.7778...  0.4789 sec/batch\n",
      "Epoch: 4/20...  Training Step: 745...  Training loss: 1.7623...  0.4774 sec/batch\n",
      "Epoch: 4/20...  Training Step: 746...  Training loss: 1.8141...  0.4777 sec/batch\n",
      "Epoch: 4/20...  Training Step: 747...  Training loss: 1.7920...  0.4779 sec/batch\n",
      "Epoch: 4/20...  Training Step: 748...  Training loss: 1.8001...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 749...  Training loss: 1.7848...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 750...  Training loss: 1.7723...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 751...  Training loss: 1.7990...  0.4796 sec/batch\n",
      "Epoch: 4/20...  Training Step: 752...  Training loss: 1.7895...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 753...  Training loss: 1.7529...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 754...  Training loss: 1.8023...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 755...  Training loss: 1.8130...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 756...  Training loss: 1.7859...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 757...  Training loss: 1.7907...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 758...  Training loss: 1.7769...  0.4793 sec/batch\n",
      "Epoch: 4/20...  Training Step: 759...  Training loss: 1.7723...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 760...  Training loss: 1.7762...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 761...  Training loss: 1.7870...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 762...  Training loss: 1.8414...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 763...  Training loss: 1.7613...  0.4794 sec/batch\n",
      "Epoch: 4/20...  Training Step: 764...  Training loss: 1.7697...  0.4783 sec/batch\n",
      "Epoch: 4/20...  Training Step: 765...  Training loss: 1.7576...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 766...  Training loss: 1.7495...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 767...  Training loss: 1.8044...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 768...  Training loss: 1.7845...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 769...  Training loss: 1.7821...  0.4791 sec/batch\n",
      "Epoch: 4/20...  Training Step: 770...  Training loss: 1.7591...  0.4780 sec/batch\n",
      "Epoch: 4/20...  Training Step: 771...  Training loss: 1.7617...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 772...  Training loss: 1.7892...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 773...  Training loss: 1.7406...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 774...  Training loss: 1.7361...  0.4779 sec/batch\n",
      "Epoch: 4/20...  Training Step: 775...  Training loss: 1.7422...  0.4793 sec/batch\n",
      "Epoch: 4/20...  Training Step: 776...  Training loss: 1.7681...  0.4792 sec/batch\n",
      "Epoch: 4/20...  Training Step: 777...  Training loss: 1.7637...  0.4784 sec/batch\n",
      "Epoch: 4/20...  Training Step: 778...  Training loss: 1.7776...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 779...  Training loss: 1.7629...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 780...  Training loss: 1.7500...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 781...  Training loss: 1.7770...  0.4786 sec/batch\n",
      "Epoch: 4/20...  Training Step: 782...  Training loss: 1.7374...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 783...  Training loss: 1.7580...  0.4788 sec/batch\n",
      "Epoch: 4/20...  Training Step: 784...  Training loss: 1.7608...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 785...  Training loss: 1.7607...  0.4796 sec/batch\n",
      "Epoch: 4/20...  Training Step: 786...  Training loss: 1.7236...  0.4781 sec/batch\n",
      "Epoch: 4/20...  Training Step: 787...  Training loss: 1.7640...  0.4782 sec/batch\n",
      "Epoch: 4/20...  Training Step: 788...  Training loss: 1.7391...  0.4785 sec/batch\n",
      "Epoch: 4/20...  Training Step: 789...  Training loss: 1.7233...  0.4781 sec/batch\n",
      "Epoch: 4/20...  Training Step: 790...  Training loss: 1.7621...  0.4787 sec/batch\n",
      "Epoch: 4/20...  Training Step: 791...  Training loss: 1.7438...  0.4788 sec/batch\n",
      "Epoch: 4/20...  Training Step: 792...  Training loss: 1.7316...  0.4782 sec/batch\n",
      "Epoch: 5/20...  Training Step: 793...  Training loss: 1.8385...  0.4780 sec/batch\n",
      "Epoch: 5/20...  Training Step: 794...  Training loss: 1.7507...  0.4793 sec/batch\n",
      "Epoch: 5/20...  Training Step: 795...  Training loss: 1.7304...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 796...  Training loss: 1.7437...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 797...  Training loss: 1.7390...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 798...  Training loss: 1.6971...  0.4794 sec/batch\n",
      "Epoch: 5/20...  Training Step: 799...  Training loss: 1.7454...  0.4800 sec/batch\n",
      "Epoch: 5/20...  Training Step: 800...  Training loss: 1.7369...  0.4802 sec/batch\n",
      "Epoch: 5/20...  Training Step: 801...  Training loss: 1.7657...  0.4800 sec/batch\n",
      "Epoch: 5/20...  Training Step: 802...  Training loss: 1.7475...  0.4781 sec/batch\n",
      "Epoch: 5/20...  Training Step: 803...  Training loss: 1.7302...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 804...  Training loss: 1.7225...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 805...  Training loss: 1.7303...  0.4790 sec/batch\n",
      "Epoch: 5/20...  Training Step: 806...  Training loss: 1.7858...  0.4799 sec/batch\n",
      "Epoch: 5/20...  Training Step: 807...  Training loss: 1.7395...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 808...  Training loss: 1.7178...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 809...  Training loss: 1.7421...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 810...  Training loss: 1.7642...  0.4790 sec/batch\n",
      "Epoch: 5/20...  Training Step: 811...  Training loss: 1.7410...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 812...  Training loss: 1.7549...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 813...  Training loss: 1.7308...  0.4781 sec/batch\n",
      "Epoch: 5/20...  Training Step: 814...  Training loss: 1.7685...  0.4793 sec/batch\n",
      "Epoch: 5/20...  Training Step: 815...  Training loss: 1.7377...  0.4790 sec/batch\n",
      "Epoch: 5/20...  Training Step: 816...  Training loss: 1.7470...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 817...  Training loss: 1.7493...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 818...  Training loss: 1.7032...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 819...  Training loss: 1.7014...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 820...  Training loss: 1.7487...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 821...  Training loss: 1.7773...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 822...  Training loss: 1.7452...  0.4781 sec/batch\n",
      "Epoch: 5/20...  Training Step: 823...  Training loss: 1.7420...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 824...  Training loss: 1.7090...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 825...  Training loss: 1.7437...  0.4789 sec/batch\n",
      "Epoch: 5/20...  Training Step: 826...  Training loss: 1.7498...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 827...  Training loss: 1.7131...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 828...  Training loss: 1.7398...  0.4790 sec/batch\n",
      "Epoch: 5/20...  Training Step: 829...  Training loss: 1.7028...  0.4792 sec/batch\n",
      "Epoch: 5/20...  Training Step: 830...  Training loss: 1.6877...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 831...  Training loss: 1.6762...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 832...  Training loss: 1.6986...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 833...  Training loss: 1.6968...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 834...  Training loss: 1.7415...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 835...  Training loss: 1.7083...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 836...  Training loss: 1.6877...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 837...  Training loss: 1.7379...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 838...  Training loss: 1.6806...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 839...  Training loss: 1.7112...  0.4789 sec/batch\n",
      "Epoch: 5/20...  Training Step: 840...  Training loss: 1.7101...  0.4776 sec/batch\n",
      "Epoch: 5/20...  Training Step: 841...  Training loss: 1.6967...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 842...  Training loss: 1.7682...  0.4789 sec/batch\n",
      "Epoch: 5/20...  Training Step: 843...  Training loss: 1.6867...  0.4793 sec/batch\n",
      "Epoch: 5/20...  Training Step: 844...  Training loss: 1.7705...  0.4782 sec/batch\n",
      "Epoch: 5/20...  Training Step: 845...  Training loss: 1.7097...  0.4789 sec/batch\n",
      "Epoch: 5/20...  Training Step: 846...  Training loss: 1.7188...  0.4782 sec/batch\n",
      "Epoch: 5/20...  Training Step: 847...  Training loss: 1.7123...  0.4777 sec/batch\n",
      "Epoch: 5/20...  Training Step: 848...  Training loss: 1.7227...  0.4794 sec/batch\n",
      "Epoch: 5/20...  Training Step: 849...  Training loss: 1.7305...  0.4781 sec/batch\n",
      "Epoch: 5/20...  Training Step: 850...  Training loss: 1.7099...  0.4792 sec/batch\n",
      "Epoch: 5/20...  Training Step: 851...  Training loss: 1.7028...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 852...  Training loss: 1.7553...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 853...  Training loss: 1.7155...  0.4773 sec/batch\n",
      "Epoch: 5/20...  Training Step: 854...  Training loss: 1.7721...  0.4790 sec/batch\n",
      "Epoch: 5/20...  Training Step: 855...  Training loss: 1.7621...  0.4782 sec/batch\n",
      "Epoch: 5/20...  Training Step: 856...  Training loss: 1.7331...  0.4774 sec/batch\n",
      "Epoch: 5/20...  Training Step: 857...  Training loss: 1.7125...  0.4780 sec/batch\n",
      "Epoch: 5/20...  Training Step: 858...  Training loss: 1.7468...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 859...  Training loss: 1.7307...  0.4782 sec/batch\n",
      "Epoch: 5/20...  Training Step: 860...  Training loss: 1.6923...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 861...  Training loss: 1.7105...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 862...  Training loss: 1.7209...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 863...  Training loss: 1.7580...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 864...  Training loss: 1.7147...  0.4801 sec/batch\n",
      "Epoch: 5/20...  Training Step: 865...  Training loss: 1.7450...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 866...  Training loss: 1.6959...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 867...  Training loss: 1.7127...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 868...  Training loss: 1.7401...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 869...  Training loss: 1.7134...  0.4790 sec/batch\n",
      "Epoch: 5/20...  Training Step: 870...  Training loss: 1.7092...  0.4777 sec/batch\n",
      "Epoch: 5/20...  Training Step: 871...  Training loss: 1.6790...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 872...  Training loss: 1.7007...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 873...  Training loss: 1.6676...  0.4780 sec/batch\n",
      "Epoch: 5/20...  Training Step: 874...  Training loss: 1.7328...  0.4789 sec/batch\n",
      "Epoch: 5/20...  Training Step: 875...  Training loss: 1.6665...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 876...  Training loss: 1.7019...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 877...  Training loss: 1.6746...  0.4778 sec/batch\n",
      "Epoch: 5/20...  Training Step: 878...  Training loss: 1.6924...  0.4789 sec/batch\n",
      "Epoch: 5/20...  Training Step: 879...  Training loss: 1.6828...  0.4794 sec/batch\n",
      "Epoch: 5/20...  Training Step: 880...  Training loss: 1.6736...  0.4781 sec/batch\n",
      "Epoch: 5/20...  Training Step: 881...  Training loss: 1.6611...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 882...  Training loss: 1.7095...  0.4794 sec/batch\n",
      "Epoch: 5/20...  Training Step: 883...  Training loss: 1.6735...  0.4781 sec/batch\n",
      "Epoch: 5/20...  Training Step: 884...  Training loss: 1.6866...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 885...  Training loss: 1.6780...  0.4777 sec/batch\n",
      "Epoch: 5/20...  Training Step: 886...  Training loss: 1.6725...  0.4779 sec/batch\n",
      "Epoch: 5/20...  Training Step: 887...  Training loss: 1.6673...  0.4798 sec/batch\n",
      "Epoch: 5/20...  Training Step: 888...  Training loss: 1.7061...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 889...  Training loss: 1.6891...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 890...  Training loss: 1.6643...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 891...  Training loss: 1.6767...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 892...  Training loss: 1.6390...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 893...  Training loss: 1.6886...  0.4780 sec/batch\n",
      "Epoch: 5/20...  Training Step: 894...  Training loss: 1.6865...  0.4778 sec/batch\n",
      "Epoch: 5/20...  Training Step: 895...  Training loss: 1.6829...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 896...  Training loss: 1.6846...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 897...  Training loss: 1.6839...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 898...  Training loss: 1.6785...  0.4795 sec/batch\n",
      "Epoch: 5/20...  Training Step: 899...  Training loss: 1.6903...  0.4790 sec/batch\n",
      "Epoch: 5/20...  Training Step: 900...  Training loss: 1.6901...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 901...  Training loss: 1.6875...  0.4790 sec/batch\n",
      "Epoch: 5/20...  Training Step: 902...  Training loss: 1.7021...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 903...  Training loss: 1.6735...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 904...  Training loss: 1.6832...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 905...  Training loss: 1.6725...  0.4790 sec/batch\n",
      "Epoch: 5/20...  Training Step: 906...  Training loss: 1.6701...  0.4798 sec/batch\n",
      "Epoch: 5/20...  Training Step: 907...  Training loss: 1.6593...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 908...  Training loss: 1.6473...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 909...  Training loss: 1.6900...  0.4782 sec/batch\n",
      "Epoch: 5/20...  Training Step: 910...  Training loss: 1.6708...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 911...  Training loss: 1.6862...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 912...  Training loss: 1.6843...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 913...  Training loss: 1.6821...  0.4792 sec/batch\n",
      "Epoch: 5/20...  Training Step: 914...  Training loss: 1.6427...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 915...  Training loss: 1.6515...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 916...  Training loss: 1.7076...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 917...  Training loss: 1.6740...  0.4781 sec/batch\n",
      "Epoch: 5/20...  Training Step: 918...  Training loss: 1.6271...  0.4799 sec/batch\n",
      "Epoch: 5/20...  Training Step: 919...  Training loss: 1.6952...  0.4792 sec/batch\n",
      "Epoch: 5/20...  Training Step: 920...  Training loss: 1.6897...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 921...  Training loss: 1.6769...  0.4795 sec/batch\n",
      "Epoch: 5/20...  Training Step: 922...  Training loss: 1.6568...  0.4789 sec/batch\n",
      "Epoch: 5/20...  Training Step: 923...  Training loss: 1.6388...  0.4790 sec/batch\n",
      "Epoch: 5/20...  Training Step: 924...  Training loss: 1.6432...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 925...  Training loss: 1.6901...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 926...  Training loss: 1.6830...  0.4801 sec/batch\n",
      "Epoch: 5/20...  Training Step: 927...  Training loss: 1.6735...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 928...  Training loss: 1.6768...  0.4779 sec/batch\n",
      "Epoch: 5/20...  Training Step: 929...  Training loss: 1.6912...  0.4794 sec/batch\n",
      "Epoch: 5/20...  Training Step: 930...  Training loss: 1.6791...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 931...  Training loss: 1.6937...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 932...  Training loss: 1.6746...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 933...  Training loss: 1.7168...  0.4794 sec/batch\n",
      "Epoch: 5/20...  Training Step: 934...  Training loss: 1.6634...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 935...  Training loss: 1.6677...  0.4773 sec/batch\n",
      "Epoch: 5/20...  Training Step: 936...  Training loss: 1.6830...  0.4792 sec/batch\n",
      "Epoch: 5/20...  Training Step: 937...  Training loss: 1.6635...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 938...  Training loss: 1.6830...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 939...  Training loss: 1.6845...  0.4777 sec/batch\n",
      "Epoch: 5/20...  Training Step: 940...  Training loss: 1.7041...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 941...  Training loss: 1.6701...  0.4780 sec/batch\n",
      "Epoch: 5/20...  Training Step: 942...  Training loss: 1.6567...  0.4796 sec/batch\n",
      "Epoch: 5/20...  Training Step: 943...  Training loss: 1.6338...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 944...  Training loss: 1.6805...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 945...  Training loss: 1.6768...  0.4781 sec/batch\n",
      "Epoch: 5/20...  Training Step: 946...  Training loss: 1.6676...  0.4780 sec/batch\n",
      "Epoch: 5/20...  Training Step: 947...  Training loss: 1.6602...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 948...  Training loss: 1.6696...  0.4793 sec/batch\n",
      "Epoch: 5/20...  Training Step: 949...  Training loss: 1.6623...  0.4792 sec/batch\n",
      "Epoch: 5/20...  Training Step: 950...  Training loss: 1.6641...  0.4790 sec/batch\n",
      "Epoch: 5/20...  Training Step: 951...  Training loss: 1.6319...  0.4793 sec/batch\n",
      "Epoch: 5/20...  Training Step: 952...  Training loss: 1.6853...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 953...  Training loss: 1.6940...  0.4779 sec/batch\n",
      "Epoch: 5/20...  Training Step: 954...  Training loss: 1.6650...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 955...  Training loss: 1.6756...  0.4793 sec/batch\n",
      "Epoch: 5/20...  Training Step: 956...  Training loss: 1.6589...  0.4789 sec/batch\n",
      "Epoch: 5/20...  Training Step: 957...  Training loss: 1.6568...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 958...  Training loss: 1.6583...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 959...  Training loss: 1.6902...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 960...  Training loss: 1.7302...  0.4794 sec/batch\n",
      "Epoch: 5/20...  Training Step: 961...  Training loss: 1.6514...  0.4782 sec/batch\n",
      "Epoch: 5/20...  Training Step: 962...  Training loss: 1.6541...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 963...  Training loss: 1.6451...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 964...  Training loss: 1.6329...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 965...  Training loss: 1.6772...  0.4777 sec/batch\n",
      "Epoch: 5/20...  Training Step: 966...  Training loss: 1.6653...  0.4782 sec/batch\n",
      "Epoch: 5/20...  Training Step: 967...  Training loss: 1.6704...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 968...  Training loss: 1.6492...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 969...  Training loss: 1.6478...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 970...  Training loss: 1.6901...  0.4792 sec/batch\n",
      "Epoch: 5/20...  Training Step: 971...  Training loss: 1.6290...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 972...  Training loss: 1.6314...  0.4781 sec/batch\n",
      "Epoch: 5/20...  Training Step: 973...  Training loss: 1.6352...  0.4789 sec/batch\n",
      "Epoch: 5/20...  Training Step: 974...  Training loss: 1.6488...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 975...  Training loss: 1.6472...  0.4791 sec/batch\n",
      "Epoch: 5/20...  Training Step: 976...  Training loss: 1.6560...  0.4785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 977...  Training loss: 1.6580...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 978...  Training loss: 1.6356...  0.4779 sec/batch\n",
      "Epoch: 5/20...  Training Step: 979...  Training loss: 1.6639...  0.4795 sec/batch\n",
      "Epoch: 5/20...  Training Step: 980...  Training loss: 1.6406...  0.4788 sec/batch\n",
      "Epoch: 5/20...  Training Step: 981...  Training loss: 1.6516...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 982...  Training loss: 1.6520...  0.4783 sec/batch\n",
      "Epoch: 5/20...  Training Step: 983...  Training loss: 1.6546...  0.4784 sec/batch\n",
      "Epoch: 5/20...  Training Step: 984...  Training loss: 1.6273...  0.4781 sec/batch\n",
      "Epoch: 5/20...  Training Step: 985...  Training loss: 1.6516...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 986...  Training loss: 1.6220...  0.4789 sec/batch\n",
      "Epoch: 5/20...  Training Step: 987...  Training loss: 1.6149...  0.4797 sec/batch\n",
      "Epoch: 5/20...  Training Step: 988...  Training loss: 1.6535...  0.4786 sec/batch\n",
      "Epoch: 5/20...  Training Step: 989...  Training loss: 1.6418...  0.4787 sec/batch\n",
      "Epoch: 5/20...  Training Step: 990...  Training loss: 1.6235...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 991...  Training loss: 1.7384...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 992...  Training loss: 1.6543...  0.4793 sec/batch\n",
      "Epoch: 6/20...  Training Step: 993...  Training loss: 1.6263...  0.4793 sec/batch\n",
      "Epoch: 6/20...  Training Step: 994...  Training loss: 1.6456...  0.4776 sec/batch\n",
      "Epoch: 6/20...  Training Step: 995...  Training loss: 1.6270...  0.4790 sec/batch\n",
      "Epoch: 6/20...  Training Step: 996...  Training loss: 1.6092...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 997...  Training loss: 1.6408...  0.4782 sec/batch\n",
      "Epoch: 6/20...  Training Step: 998...  Training loss: 1.6286...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 999...  Training loss: 1.6632...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1000...  Training loss: 1.6316...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1001...  Training loss: 1.6189...  0.4793 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1002...  Training loss: 1.6262...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1003...  Training loss: 1.6404...  0.4789 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1004...  Training loss: 1.6792...  0.4780 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1005...  Training loss: 1.6332...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1006...  Training loss: 1.6245...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1007...  Training loss: 1.6515...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1008...  Training loss: 1.6718...  0.4790 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1009...  Training loss: 1.6448...  0.4800 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1010...  Training loss: 1.6587...  0.4790 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1011...  Training loss: 1.6292...  0.4782 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1012...  Training loss: 1.6633...  0.4791 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1013...  Training loss: 1.6213...  0.4789 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1014...  Training loss: 1.6469...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1015...  Training loss: 1.6457...  0.4777 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1016...  Training loss: 1.6009...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1017...  Training loss: 1.6033...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1018...  Training loss: 1.6415...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1019...  Training loss: 1.6569...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1020...  Training loss: 1.6448...  0.4790 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1021...  Training loss: 1.6360...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1022...  Training loss: 1.5996...  0.4792 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1023...  Training loss: 1.6432...  0.4792 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1024...  Training loss: 1.6464...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1025...  Training loss: 1.6254...  0.4794 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1026...  Training loss: 1.6258...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1027...  Training loss: 1.6053...  0.4778 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1028...  Training loss: 1.5951...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1029...  Training loss: 1.5764...  0.4792 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1030...  Training loss: 1.6031...  0.4795 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1031...  Training loss: 1.6013...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1032...  Training loss: 1.6457...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1033...  Training loss: 1.6002...  0.4781 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1034...  Training loss: 1.5839...  0.4793 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1035...  Training loss: 1.6392...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1036...  Training loss: 1.5812...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1037...  Training loss: 1.6096...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1038...  Training loss: 1.6046...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1039...  Training loss: 1.6101...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1040...  Training loss: 1.6494...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1041...  Training loss: 1.5956...  0.4796 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1042...  Training loss: 1.6728...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1043...  Training loss: 1.6302...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1044...  Training loss: 1.6397...  0.4793 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1045...  Training loss: 1.6045...  0.4789 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1046...  Training loss: 1.6264...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1047...  Training loss: 1.6401...  0.4801 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1048...  Training loss: 1.6101...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1049...  Training loss: 1.5931...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1050...  Training loss: 1.6556...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1051...  Training loss: 1.6122...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1052...  Training loss: 1.6675...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1053...  Training loss: 1.6603...  0.4789 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1054...  Training loss: 1.6301...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1055...  Training loss: 1.6128...  0.4780 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1056...  Training loss: 1.6361...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1057...  Training loss: 1.6416...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1058...  Training loss: 1.6046...  0.4789 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1059...  Training loss: 1.6209...  0.4791 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1060...  Training loss: 1.6131...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1061...  Training loss: 1.6689...  0.4790 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1062...  Training loss: 1.6403...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1063...  Training loss: 1.6512...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1064...  Training loss: 1.6022...  0.4778 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1065...  Training loss: 1.6101...  0.4790 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1066...  Training loss: 1.6492...  0.4790 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1067...  Training loss: 1.6126...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1068...  Training loss: 1.6117...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1069...  Training loss: 1.5756...  0.4792 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1070...  Training loss: 1.6045...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1071...  Training loss: 1.5636...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1072...  Training loss: 1.6270...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1073...  Training loss: 1.5720...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1074...  Training loss: 1.6018...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1075...  Training loss: 1.5831...  0.4780 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1076...  Training loss: 1.6066...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1077...  Training loss: 1.5850...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1078...  Training loss: 1.5838...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1079...  Training loss: 1.5679...  0.4782 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1080...  Training loss: 1.6256...  0.4778 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1081...  Training loss: 1.5875...  0.4770 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1082...  Training loss: 1.5899...  0.4779 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1083...  Training loss: 1.5810...  0.4779 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1084...  Training loss: 1.5868...  0.4792 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1085...  Training loss: 1.5762...  0.4789 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1086...  Training loss: 1.6036...  0.4779 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1087...  Training loss: 1.6164...  0.4793 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1088...  Training loss: 1.5736...  0.4781 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1089...  Training loss: 1.5805...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1090...  Training loss: 1.5535...  0.4789 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1091...  Training loss: 1.6080...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1092...  Training loss: 1.5927...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1093...  Training loss: 1.5897...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1094...  Training loss: 1.6017...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1095...  Training loss: 1.5981...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1096...  Training loss: 1.5923...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1097...  Training loss: 1.5981...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1098...  Training loss: 1.5983...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1099...  Training loss: 1.5980...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1100...  Training loss: 1.6106...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1101...  Training loss: 1.5871...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1102...  Training loss: 1.5945...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1103...  Training loss: 1.5865...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1104...  Training loss: 1.5738...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1105...  Training loss: 1.5648...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1106...  Training loss: 1.5561...  0.4796 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1107...  Training loss: 1.6014...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1108...  Training loss: 1.5891...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1109...  Training loss: 1.5915...  0.4780 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1110...  Training loss: 1.5915...  0.4781 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1111...  Training loss: 1.5946...  0.4777 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1112...  Training loss: 1.5555...  0.4780 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1113...  Training loss: 1.5499...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1114...  Training loss: 1.5998...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1115...  Training loss: 1.5893...  0.4782 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1116...  Training loss: 1.5479...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1117...  Training loss: 1.6050...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1118...  Training loss: 1.6043...  0.4779 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1119...  Training loss: 1.5780...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1120...  Training loss: 1.5665...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1121...  Training loss: 1.5449...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1122...  Training loss: 1.5501...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1123...  Training loss: 1.6073...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1124...  Training loss: 1.5929...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1125...  Training loss: 1.5967...  0.4781 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1126...  Training loss: 1.5937...  0.4778 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1127...  Training loss: 1.6111...  0.4795 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1128...  Training loss: 1.6004...  0.4792 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1129...  Training loss: 1.5989...  0.4780 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1130...  Training loss: 1.5787...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1131...  Training loss: 1.6313...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1132...  Training loss: 1.5877...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1133...  Training loss: 1.5907...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1134...  Training loss: 1.6050...  0.4777 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1135...  Training loss: 1.5774...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1136...  Training loss: 1.6070...  0.4776 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1137...  Training loss: 1.5986...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1138...  Training loss: 1.6269...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1139...  Training loss: 1.6043...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1140...  Training loss: 1.5818...  0.4777 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1141...  Training loss: 1.5581...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1142...  Training loss: 1.5948...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1143...  Training loss: 1.5991...  0.4777 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1144...  Training loss: 1.5928...  0.4781 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1145...  Training loss: 1.5837...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1146...  Training loss: 1.5878...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1147...  Training loss: 1.5899...  0.4773 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1148...  Training loss: 1.5708...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1149...  Training loss: 1.5351...  0.4777 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1150...  Training loss: 1.6079...  0.4782 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1151...  Training loss: 1.6124...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1152...  Training loss: 1.5864...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1153...  Training loss: 1.5931...  0.4777 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1154...  Training loss: 1.5794...  0.4780 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1155...  Training loss: 1.5801...  0.4796 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1156...  Training loss: 1.5739...  0.4776 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1157...  Training loss: 1.6122...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1158...  Training loss: 1.6506...  0.4778 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1159...  Training loss: 1.5726...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1160...  Training loss: 1.5665...  0.4777 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1161...  Training loss: 1.5695...  0.4780 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1162...  Training loss: 1.5572...  0.4794 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1163...  Training loss: 1.5994...  0.4772 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1164...  Training loss: 1.5751...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1165...  Training loss: 1.5956...  0.4777 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1166...  Training loss: 1.5695...  0.4780 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1167...  Training loss: 1.5568...  0.4788 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1168...  Training loss: 1.5980...  0.4776 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1169...  Training loss: 1.5582...  0.4789 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1170...  Training loss: 1.5500...  0.4780 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1171...  Training loss: 1.5529...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1172...  Training loss: 1.5619...  0.4784 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1173...  Training loss: 1.5844...  0.4776 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1174...  Training loss: 1.5797...  0.4779 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1175...  Training loss: 1.5746...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1176...  Training loss: 1.5558...  0.4782 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1177...  Training loss: 1.5868...  0.4779 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1178...  Training loss: 1.5572...  0.4783 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1179...  Training loss: 1.5754...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1180...  Training loss: 1.5728...  0.4776 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1181...  Training loss: 1.5598...  0.4777 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1182...  Training loss: 1.5469...  0.4787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1183...  Training loss: 1.5618...  0.4786 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1184...  Training loss: 1.5482...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1185...  Training loss: 1.5402...  0.4779 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1186...  Training loss: 1.5734...  0.4781 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1187...  Training loss: 1.5643...  0.4785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1188...  Training loss: 1.5484...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1189...  Training loss: 1.6626...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1190...  Training loss: 1.5699...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1191...  Training loss: 1.5557...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1192...  Training loss: 1.5690...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1193...  Training loss: 1.5495...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1194...  Training loss: 1.5252...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1195...  Training loss: 1.5621...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1196...  Training loss: 1.5561...  0.4779 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1197...  Training loss: 1.5732...  0.4778 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1198...  Training loss: 1.5593...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1199...  Training loss: 1.5427...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1200...  Training loss: 1.5458...  0.4772 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1201...  Training loss: 1.5573...  0.4782 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1202...  Training loss: 1.5925...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1203...  Training loss: 1.5572...  0.4774 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1204...  Training loss: 1.5411...  0.4796 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1205...  Training loss: 1.5750...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1206...  Training loss: 1.5899...  0.4780 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1207...  Training loss: 1.5627...  0.4779 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1208...  Training loss: 1.5819...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1209...  Training loss: 1.5578...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1210...  Training loss: 1.5948...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1211...  Training loss: 1.5571...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1212...  Training loss: 1.5602...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1213...  Training loss: 1.5676...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1214...  Training loss: 1.5157...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1215...  Training loss: 1.5234...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1216...  Training loss: 1.5744...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1217...  Training loss: 1.5785...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1218...  Training loss: 1.5855...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1219...  Training loss: 1.5524...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1220...  Training loss: 1.5272...  0.4778 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1221...  Training loss: 1.5701...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1222...  Training loss: 1.5644...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1223...  Training loss: 1.5505...  0.4776 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1224...  Training loss: 1.5576...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1225...  Training loss: 1.5313...  0.4791 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1226...  Training loss: 1.5175...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1227...  Training loss: 1.4982...  0.4778 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1228...  Training loss: 1.5273...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1229...  Training loss: 1.5258...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1230...  Training loss: 1.5887...  0.4790 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1231...  Training loss: 1.5396...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1232...  Training loss: 1.5316...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1233...  Training loss: 1.5653...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1234...  Training loss: 1.5032...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1235...  Training loss: 1.5399...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1236...  Training loss: 1.5233...  0.4793 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1237...  Training loss: 1.5344...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1238...  Training loss: 1.5823...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1239...  Training loss: 1.5212...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1240...  Training loss: 1.6037...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1241...  Training loss: 1.5538...  0.4794 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1242...  Training loss: 1.5545...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1243...  Training loss: 1.5340...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1244...  Training loss: 1.5510...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1245...  Training loss: 1.5704...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1246...  Training loss: 1.5310...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1247...  Training loss: 1.5195...  0.4778 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1248...  Training loss: 1.5787...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1249...  Training loss: 1.5487...  0.4777 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1250...  Training loss: 1.6012...  0.4790 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1251...  Training loss: 1.5834...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1252...  Training loss: 1.5562...  0.4790 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1253...  Training loss: 1.5419...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1254...  Training loss: 1.5587...  0.4793 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1255...  Training loss: 1.5632...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1256...  Training loss: 1.5306...  0.4780 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1257...  Training loss: 1.5416...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1258...  Training loss: 1.5381...  0.4792 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1259...  Training loss: 1.5973...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1260...  Training loss: 1.5704...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1261...  Training loss: 1.5760...  0.4776 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1262...  Training loss: 1.5286...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1263...  Training loss: 1.5375...  0.4779 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1264...  Training loss: 1.5702...  0.4778 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1265...  Training loss: 1.5399...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1266...  Training loss: 1.5356...  0.4774 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1267...  Training loss: 1.5081...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1268...  Training loss: 1.5434...  0.4782 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1269...  Training loss: 1.4939...  0.4780 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1270...  Training loss: 1.5510...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1271...  Training loss: 1.5071...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1272...  Training loss: 1.5364...  0.4791 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1273...  Training loss: 1.5168...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1274...  Training loss: 1.5301...  0.4779 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1275...  Training loss: 1.5066...  0.4794 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1276...  Training loss: 1.5271...  0.4775 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1277...  Training loss: 1.4980...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1278...  Training loss: 1.5522...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1279...  Training loss: 1.5124...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1280...  Training loss: 1.5234...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1281...  Training loss: 1.5096...  0.4778 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1282...  Training loss: 1.5150...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1283...  Training loss: 1.5087...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1284...  Training loss: 1.5509...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1285...  Training loss: 1.5372...  0.4782 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1286...  Training loss: 1.5017...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1287...  Training loss: 1.5034...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1288...  Training loss: 1.4945...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1289...  Training loss: 1.5353...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1290...  Training loss: 1.5356...  0.4791 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1291...  Training loss: 1.5450...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1292...  Training loss: 1.5181...  0.4793 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1293...  Training loss: 1.5286...  0.4776 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1294...  Training loss: 1.5279...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1295...  Training loss: 1.5380...  0.4779 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1296...  Training loss: 1.5347...  0.4794 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1297...  Training loss: 1.5393...  0.4792 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1298...  Training loss: 1.5536...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1299...  Training loss: 1.5249...  0.4782 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1300...  Training loss: 1.5339...  0.4791 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1301...  Training loss: 1.5214...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1302...  Training loss: 1.5184...  0.4780 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1303...  Training loss: 1.5108...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1304...  Training loss: 1.4934...  0.4778 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1305...  Training loss: 1.5327...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1306...  Training loss: 1.5247...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1307...  Training loss: 1.5243...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1308...  Training loss: 1.5129...  0.4794 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1309...  Training loss: 1.5289...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1310...  Training loss: 1.4899...  0.4790 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1311...  Training loss: 1.4828...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1312...  Training loss: 1.5360...  0.4776 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1313...  Training loss: 1.5242...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1314...  Training loss: 1.4803...  0.4794 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1315...  Training loss: 1.5402...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1316...  Training loss: 1.5332...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1317...  Training loss: 1.5126...  0.4773 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1318...  Training loss: 1.4881...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1319...  Training loss: 1.4793...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1320...  Training loss: 1.4916...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1321...  Training loss: 1.5464...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1322...  Training loss: 1.5329...  0.4791 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1323...  Training loss: 1.5303...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1324...  Training loss: 1.5265...  0.4797 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1325...  Training loss: 1.5442...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1326...  Training loss: 1.5316...  0.4782 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1327...  Training loss: 1.5296...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1328...  Training loss: 1.5156...  0.4779 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1329...  Training loss: 1.5753...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1330...  Training loss: 1.5204...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1331...  Training loss: 1.5167...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1332...  Training loss: 1.5447...  0.4790 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1333...  Training loss: 1.5128...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1334...  Training loss: 1.5360...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1335...  Training loss: 1.5313...  0.4793 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1336...  Training loss: 1.5605...  0.4782 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1337...  Training loss: 1.5345...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1338...  Training loss: 1.5115...  0.4780 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1339...  Training loss: 1.4807...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1340...  Training loss: 1.5125...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1341...  Training loss: 1.5269...  0.4782 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1342...  Training loss: 1.5138...  0.4790 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1343...  Training loss: 1.5184...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1344...  Training loss: 1.5242...  0.4780 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1345...  Training loss: 1.5276...  0.4792 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1346...  Training loss: 1.5207...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1347...  Training loss: 1.4848...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1348...  Training loss: 1.5513...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1349...  Training loss: 1.5626...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1350...  Training loss: 1.5279...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1351...  Training loss: 1.5222...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1352...  Training loss: 1.5235...  0.4796 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1353...  Training loss: 1.5146...  0.4780 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1354...  Training loss: 1.5258...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1355...  Training loss: 1.5483...  0.4799 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1356...  Training loss: 1.5858...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1357...  Training loss: 1.5212...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1358...  Training loss: 1.5161...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1359...  Training loss: 1.5055...  0.4785 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1360...  Training loss: 1.4945...  0.4796 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1361...  Training loss: 1.5464...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1362...  Training loss: 1.5254...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1363...  Training loss: 1.5358...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1364...  Training loss: 1.4932...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1365...  Training loss: 1.4984...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1366...  Training loss: 1.5384...  0.4796 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1367...  Training loss: 1.4879...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1368...  Training loss: 1.4888...  0.4793 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1369...  Training loss: 1.4938...  0.4782 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1370...  Training loss: 1.5065...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1371...  Training loss: 1.5112...  0.4788 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1372...  Training loss: 1.5048...  0.4790 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1373...  Training loss: 1.5025...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1374...  Training loss: 1.5020...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1375...  Training loss: 1.5205...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1376...  Training loss: 1.4999...  0.4783 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1377...  Training loss: 1.5105...  0.4781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1378...  Training loss: 1.5152...  0.4794 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1379...  Training loss: 1.5067...  0.4787 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1380...  Training loss: 1.4896...  0.4784 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1381...  Training loss: 1.5124...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1382...  Training loss: 1.5006...  0.4789 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1383...  Training loss: 1.4824...  0.4786 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1384...  Training loss: 1.5084...  0.4782 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1385...  Training loss: 1.4999...  0.4782 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1386...  Training loss: 1.4855...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1387...  Training loss: 1.6080...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1388...  Training loss: 1.5209...  0.4791 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1389...  Training loss: 1.5100...  0.4787 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1390...  Training loss: 1.5150...  0.4778 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1391...  Training loss: 1.5003...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1392...  Training loss: 1.4747...  0.4782 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1393...  Training loss: 1.5160...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1394...  Training loss: 1.4904...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1395...  Training loss: 1.5187...  0.4791 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1396...  Training loss: 1.4893...  0.4795 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1397...  Training loss: 1.4768...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1398...  Training loss: 1.4940...  0.4795 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1399...  Training loss: 1.5028...  0.4779 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1400...  Training loss: 1.5365...  0.4782 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1401...  Training loss: 1.4920...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1402...  Training loss: 1.4900...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1403...  Training loss: 1.5047...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1404...  Training loss: 1.5281...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1405...  Training loss: 1.5046...  0.4783 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1406...  Training loss: 1.5253...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1407...  Training loss: 1.4989...  0.4777 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1408...  Training loss: 1.5237...  0.4776 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1409...  Training loss: 1.4849...  0.4793 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1410...  Training loss: 1.5028...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1411...  Training loss: 1.5096...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1412...  Training loss: 1.4600...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1413...  Training loss: 1.4694...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1414...  Training loss: 1.5196...  0.4794 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1415...  Training loss: 1.5230...  0.4779 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1416...  Training loss: 1.5295...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1417...  Training loss: 1.4986...  0.4779 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1418...  Training loss: 1.4833...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1419...  Training loss: 1.5132...  0.4778 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1420...  Training loss: 1.5071...  0.4792 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1421...  Training loss: 1.4943...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1422...  Training loss: 1.5028...  0.4775 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1423...  Training loss: 1.4831...  0.4780 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1424...  Training loss: 1.4617...  0.4774 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1425...  Training loss: 1.4523...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1426...  Training loss: 1.4801...  0.4773 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1427...  Training loss: 1.4731...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1428...  Training loss: 1.5446...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1429...  Training loss: 1.4849...  0.4787 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1430...  Training loss: 1.4660...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1431...  Training loss: 1.5041...  0.4795 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1432...  Training loss: 1.4553...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1433...  Training loss: 1.4907...  0.4791 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1434...  Training loss: 1.4821...  0.4794 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1435...  Training loss: 1.4868...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1436...  Training loss: 1.5192...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1437...  Training loss: 1.4661...  0.4782 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1438...  Training loss: 1.5475...  0.4783 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1439...  Training loss: 1.5007...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1440...  Training loss: 1.4943...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1441...  Training loss: 1.4837...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1442...  Training loss: 1.4982...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1443...  Training loss: 1.5240...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1444...  Training loss: 1.4715...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1445...  Training loss: 1.4580...  0.4791 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1446...  Training loss: 1.5292...  0.4773 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1447...  Training loss: 1.5023...  0.4790 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1448...  Training loss: 1.5488...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1449...  Training loss: 1.5234...  0.4783 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1450...  Training loss: 1.5075...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1451...  Training loss: 1.4903...  0.4794 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1452...  Training loss: 1.5023...  0.4780 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1453...  Training loss: 1.5206...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1454...  Training loss: 1.4638...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1455...  Training loss: 1.5010...  0.4779 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1456...  Training loss: 1.4896...  0.4793 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1457...  Training loss: 1.5494...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1458...  Training loss: 1.5125...  0.4779 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1459...  Training loss: 1.5300...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1460...  Training loss: 1.4807...  0.4787 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1461...  Training loss: 1.4857...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1462...  Training loss: 1.5150...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1463...  Training loss: 1.4920...  0.4782 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1464...  Training loss: 1.4846...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1465...  Training loss: 1.4515...  0.4780 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1466...  Training loss: 1.4874...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1467...  Training loss: 1.4482...  0.4783 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1468...  Training loss: 1.4887...  0.4792 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1469...  Training loss: 1.4460...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1470...  Training loss: 1.4788...  0.4787 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1471...  Training loss: 1.4653...  0.4778 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1472...  Training loss: 1.4815...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1473...  Training loss: 1.4575...  0.4775 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1474...  Training loss: 1.4645...  0.4779 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1475...  Training loss: 1.4553...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1476...  Training loss: 1.4886...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1477...  Training loss: 1.4632...  0.4780 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1478...  Training loss: 1.4657...  0.4777 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1479...  Training loss: 1.4582...  0.4789 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1480...  Training loss: 1.4637...  0.4787 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1481...  Training loss: 1.4720...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1482...  Training loss: 1.4804...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1483...  Training loss: 1.4919...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1484...  Training loss: 1.4417...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1485...  Training loss: 1.4531...  0.4783 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1486...  Training loss: 1.4506...  0.4787 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1487...  Training loss: 1.4800...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1488...  Training loss: 1.4706...  0.4790 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1489...  Training loss: 1.4803...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1490...  Training loss: 1.4712...  0.4787 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1491...  Training loss: 1.4720...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1492...  Training loss: 1.4808...  0.4783 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1493...  Training loss: 1.4946...  0.4796 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1494...  Training loss: 1.4757...  0.4791 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1495...  Training loss: 1.4623...  0.4790 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1496...  Training loss: 1.4989...  0.4783 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1497...  Training loss: 1.4692...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1498...  Training loss: 1.4822...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1499...  Training loss: 1.4762...  0.4778 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1500...  Training loss: 1.4784...  0.4792 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1501...  Training loss: 1.4556...  0.4791 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1502...  Training loss: 1.4572...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1503...  Training loss: 1.4866...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1504...  Training loss: 1.4978...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1505...  Training loss: 1.4856...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1506...  Training loss: 1.4702...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1507...  Training loss: 1.4837...  0.4787 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1508...  Training loss: 1.4533...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1509...  Training loss: 1.4341...  0.4783 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1510...  Training loss: 1.4875...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1511...  Training loss: 1.4745...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1512...  Training loss: 1.4391...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1513...  Training loss: 1.4911...  0.4792 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1514...  Training loss: 1.4936...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1515...  Training loss: 1.4776...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1516...  Training loss: 1.4532...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1517...  Training loss: 1.4291...  0.4795 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1518...  Training loss: 1.4544...  0.4787 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1519...  Training loss: 1.5028...  0.4792 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1520...  Training loss: 1.4831...  0.4787 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1521...  Training loss: 1.4828...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1522...  Training loss: 1.4854...  0.4795 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1523...  Training loss: 1.5048...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1524...  Training loss: 1.4908...  0.4773 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1525...  Training loss: 1.4775...  0.4783 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1526...  Training loss: 1.4763...  0.4790 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1527...  Training loss: 1.5224...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1528...  Training loss: 1.4771...  0.4789 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1529...  Training loss: 1.4674...  0.4794 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1530...  Training loss: 1.4967...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1531...  Training loss: 1.4531...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1532...  Training loss: 1.4938...  0.4780 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1533...  Training loss: 1.4821...  0.4794 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1534...  Training loss: 1.5088...  0.4797 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1535...  Training loss: 1.4892...  0.4780 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1536...  Training loss: 1.4618...  0.4789 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1537...  Training loss: 1.4283...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1538...  Training loss: 1.4529...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1539...  Training loss: 1.4769...  0.4796 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1540...  Training loss: 1.4651...  0.4791 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1541...  Training loss: 1.4670...  0.4783 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1542...  Training loss: 1.4672...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1543...  Training loss: 1.4726...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1544...  Training loss: 1.4702...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1545...  Training loss: 1.4259...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1546...  Training loss: 1.4886...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1547...  Training loss: 1.5017...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1548...  Training loss: 1.4721...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1549...  Training loss: 1.4697...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1550...  Training loss: 1.4658...  0.4790 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1551...  Training loss: 1.4643...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1552...  Training loss: 1.4680...  0.4783 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1553...  Training loss: 1.4954...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1554...  Training loss: 1.5457...  0.4789 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1555...  Training loss: 1.4716...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1556...  Training loss: 1.4634...  0.4789 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1557...  Training loss: 1.4510...  0.4796 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1558...  Training loss: 1.4454...  0.4791 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1559...  Training loss: 1.4943...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1560...  Training loss: 1.4666...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1561...  Training loss: 1.4757...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1562...  Training loss: 1.4385...  0.4789 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1563...  Training loss: 1.4478...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1564...  Training loss: 1.4897...  0.4790 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1565...  Training loss: 1.4403...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1566...  Training loss: 1.4353...  0.4789 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1567...  Training loss: 1.4298...  0.4795 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1568...  Training loss: 1.4563...  0.4790 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1569...  Training loss: 1.4527...  0.4793 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1570...  Training loss: 1.4543...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1571...  Training loss: 1.4495...  0.4789 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1572...  Training loss: 1.4434...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1573...  Training loss: 1.4729...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1574...  Training loss: 1.4468...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1575...  Training loss: 1.4495...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1576...  Training loss: 1.4525...  0.4785 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1577...  Training loss: 1.4500...  0.4790 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1578...  Training loss: 1.4463...  0.4788 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1579...  Training loss: 1.4564...  0.4781 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1580...  Training loss: 1.4417...  0.4782 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1581...  Training loss: 1.4211...  0.4784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1582...  Training loss: 1.4679...  0.4782 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1583...  Training loss: 1.4425...  0.4786 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1584...  Training loss: 1.4351...  0.4784 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1585...  Training loss: 1.5759...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1586...  Training loss: 1.4603...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1587...  Training loss: 1.4547...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1588...  Training loss: 1.4650...  0.4791 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1589...  Training loss: 1.4374...  0.4794 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1590...  Training loss: 1.4205...  0.4784 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1591...  Training loss: 1.4508...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1592...  Training loss: 1.4356...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1593...  Training loss: 1.4543...  0.4798 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1594...  Training loss: 1.4355...  0.4780 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1595...  Training loss: 1.4268...  0.4793 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1596...  Training loss: 1.4398...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1597...  Training loss: 1.4392...  0.4790 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1598...  Training loss: 1.4758...  0.4788 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1599...  Training loss: 1.4430...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1600...  Training loss: 1.4265...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1601...  Training loss: 1.4554...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1602...  Training loss: 1.4710...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1603...  Training loss: 1.4579...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1604...  Training loss: 1.4656...  0.4778 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1605...  Training loss: 1.4382...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1606...  Training loss: 1.4709...  0.4797 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1607...  Training loss: 1.4374...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1608...  Training loss: 1.4539...  0.4782 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1609...  Training loss: 1.4504...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1610...  Training loss: 1.4117...  0.4779 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1611...  Training loss: 1.4065...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1612...  Training loss: 1.4698...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1613...  Training loss: 1.4521...  0.4795 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1614...  Training loss: 1.4601...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1615...  Training loss: 1.4324...  0.4784 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1616...  Training loss: 1.4239...  0.4782 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1617...  Training loss: 1.4634...  0.4788 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1618...  Training loss: 1.4506...  0.4780 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1619...  Training loss: 1.4381...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1620...  Training loss: 1.4506...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1621...  Training loss: 1.4225...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1622...  Training loss: 1.4039...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1623...  Training loss: 1.3969...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1624...  Training loss: 1.4303...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1625...  Training loss: 1.4184...  0.4790 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1626...  Training loss: 1.4737...  0.4782 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1627...  Training loss: 1.4208...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1628...  Training loss: 1.4099...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1629...  Training loss: 1.4501...  0.4794 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1630...  Training loss: 1.4092...  0.4784 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1631...  Training loss: 1.4271...  0.4784 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1632...  Training loss: 1.4205...  0.4793 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1633...  Training loss: 1.4406...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1634...  Training loss: 1.4536...  0.4797 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1635...  Training loss: 1.4115...  0.4794 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1636...  Training loss: 1.4776...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1637...  Training loss: 1.4348...  0.4791 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1638...  Training loss: 1.4482...  0.4790 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1639...  Training loss: 1.4323...  0.4793 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1640...  Training loss: 1.4445...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1641...  Training loss: 1.4561...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1642...  Training loss: 1.4311...  0.4774 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1643...  Training loss: 1.4140...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1644...  Training loss: 1.4736...  0.4782 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1645...  Training loss: 1.4622...  0.4784 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1646...  Training loss: 1.4911...  0.4791 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1647...  Training loss: 1.4758...  0.4780 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1648...  Training loss: 1.4544...  0.4788 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1649...  Training loss: 1.4297...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1650...  Training loss: 1.4645...  0.4788 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1651...  Training loss: 1.4536...  0.4780 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1652...  Training loss: 1.4218...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1653...  Training loss: 1.4436...  0.4782 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1654...  Training loss: 1.4260...  0.4778 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1655...  Training loss: 1.4887...  0.4779 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1656...  Training loss: 1.4589...  0.4778 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1657...  Training loss: 1.4691...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1658...  Training loss: 1.4117...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1659...  Training loss: 1.4240...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1660...  Training loss: 1.4610...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1661...  Training loss: 1.4281...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1662...  Training loss: 1.4289...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1663...  Training loss: 1.3944...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1664...  Training loss: 1.4395...  0.4782 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1665...  Training loss: 1.3908...  0.4790 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1666...  Training loss: 1.4389...  0.4791 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1667...  Training loss: 1.3938...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1668...  Training loss: 1.4282...  0.4778 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1669...  Training loss: 1.4097...  0.4792 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1670...  Training loss: 1.4343...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1671...  Training loss: 1.4003...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1672...  Training loss: 1.4022...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1673...  Training loss: 1.3962...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1674...  Training loss: 1.4457...  0.4775 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1675...  Training loss: 1.4113...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1676...  Training loss: 1.4130...  0.4794 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1677...  Training loss: 1.3994...  0.4774 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1678...  Training loss: 1.3937...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1679...  Training loss: 1.4030...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1680...  Training loss: 1.4331...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1681...  Training loss: 1.4209...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1682...  Training loss: 1.3885...  0.4779 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1683...  Training loss: 1.4026...  0.4782 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1684...  Training loss: 1.3848...  0.4777 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1685...  Training loss: 1.4432...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1686...  Training loss: 1.4143...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1687...  Training loss: 1.4176...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1688...  Training loss: 1.4098...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1689...  Training loss: 1.4231...  0.4788 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1690...  Training loss: 1.4232...  0.4791 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1691...  Training loss: 1.4216...  0.4794 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1692...  Training loss: 1.4233...  0.4790 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1693...  Training loss: 1.4078...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1694...  Training loss: 1.4350...  0.4799 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1695...  Training loss: 1.4021...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1696...  Training loss: 1.4220...  0.4790 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1697...  Training loss: 1.4155...  0.4793 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1698...  Training loss: 1.4094...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1699...  Training loss: 1.3999...  0.4779 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1700...  Training loss: 1.3735...  0.4795 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1701...  Training loss: 1.4307...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1702...  Training loss: 1.4263...  0.4796 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1703...  Training loss: 1.4125...  0.4795 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1704...  Training loss: 1.4049...  0.4791 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1705...  Training loss: 1.4263...  0.4790 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1706...  Training loss: 1.3856...  0.4794 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1707...  Training loss: 1.3777...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1708...  Training loss: 1.4288...  0.4797 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1709...  Training loss: 1.4048...  0.4792 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1710...  Training loss: 1.3776...  0.4782 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1711...  Training loss: 1.4281...  0.4780 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1712...  Training loss: 1.4279...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1713...  Training loss: 1.4160...  0.4795 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1714...  Training loss: 1.3849...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1715...  Training loss: 1.3716...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1716...  Training loss: 1.3890...  0.4780 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1717...  Training loss: 1.4415...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1718...  Training loss: 1.4213...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1719...  Training loss: 1.4334...  0.4791 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1720...  Training loss: 1.4167...  0.4788 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1721...  Training loss: 1.4436...  0.4780 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1722...  Training loss: 1.4298...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1723...  Training loss: 1.4285...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1724...  Training loss: 1.4223...  0.4793 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1725...  Training loss: 1.4772...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1726...  Training loss: 1.4218...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1727...  Training loss: 1.4203...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1728...  Training loss: 1.4491...  0.4779 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1729...  Training loss: 1.4060...  0.4795 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1730...  Training loss: 1.4374...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1731...  Training loss: 1.4272...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1732...  Training loss: 1.4582...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1733...  Training loss: 1.4352...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1734...  Training loss: 1.4061...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1735...  Training loss: 1.3819...  0.4780 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1736...  Training loss: 1.4013...  0.4784 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1737...  Training loss: 1.4338...  0.4784 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1738...  Training loss: 1.4120...  0.4791 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1739...  Training loss: 1.4124...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1740...  Training loss: 1.4156...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1741...  Training loss: 1.4154...  0.4788 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1742...  Training loss: 1.4131...  0.4788 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1743...  Training loss: 1.3753...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1744...  Training loss: 1.4413...  0.4782 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1745...  Training loss: 1.4361...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1746...  Training loss: 1.4178...  0.4781 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1747...  Training loss: 1.4213...  0.4791 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1748...  Training loss: 1.4219...  0.4796 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1749...  Training loss: 1.4070...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1750...  Training loss: 1.4145...  0.4799 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1751...  Training loss: 1.4403...  0.4792 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1752...  Training loss: 1.4894...  0.4797 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1753...  Training loss: 1.4225...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1754...  Training loss: 1.4113...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1755...  Training loss: 1.4037...  0.4784 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1756...  Training loss: 1.3957...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1757...  Training loss: 1.4500...  0.4793 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1758...  Training loss: 1.4166...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1759...  Training loss: 1.4187...  0.4793 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1760...  Training loss: 1.3953...  0.4784 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1761...  Training loss: 1.3987...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1762...  Training loss: 1.4514...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1763...  Training loss: 1.3962...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1764...  Training loss: 1.3796...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1765...  Training loss: 1.3922...  0.4787 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1766...  Training loss: 1.3969...  0.4788 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1767...  Training loss: 1.4150...  0.4783 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1768...  Training loss: 1.4105...  0.4803 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1769...  Training loss: 1.4071...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1770...  Training loss: 1.3929...  0.4793 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1771...  Training loss: 1.4395...  0.4797 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1772...  Training loss: 1.4090...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1773...  Training loss: 1.4066...  0.4784 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1774...  Training loss: 1.4091...  0.4788 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1775...  Training loss: 1.3922...  0.4793 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1776...  Training loss: 1.3911...  0.4782 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1777...  Training loss: 1.4104...  0.4780 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1778...  Training loss: 1.4019...  0.4786 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1779...  Training loss: 1.3837...  0.4785 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1780...  Training loss: 1.4158...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1781...  Training loss: 1.4002...  0.4789 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1782...  Training loss: 1.3984...  0.4790 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1783...  Training loss: 1.5537...  0.4782 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1784...  Training loss: 1.4193...  0.4782 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1785...  Training loss: 1.4106...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1786...  Training loss: 1.4304...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1787...  Training loss: 1.4044...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1788...  Training loss: 1.3840...  0.4791 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1789...  Training loss: 1.4157...  0.4791 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1790...  Training loss: 1.3947...  0.4784 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1791...  Training loss: 1.4099...  0.4793 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1792...  Training loss: 1.3995...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1793...  Training loss: 1.3915...  0.4790 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1794...  Training loss: 1.3990...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1795...  Training loss: 1.4129...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1796...  Training loss: 1.4439...  0.4791 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1797...  Training loss: 1.4069...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1798...  Training loss: 1.3886...  0.4792 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1799...  Training loss: 1.4217...  0.4784 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1800...  Training loss: 1.4340...  0.4791 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1801...  Training loss: 1.4081...  0.4794 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1802...  Training loss: 1.4304...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1803...  Training loss: 1.3997...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1804...  Training loss: 1.4348...  0.4784 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1805...  Training loss: 1.4010...  0.4793 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1806...  Training loss: 1.4100...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1807...  Training loss: 1.4081...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1808...  Training loss: 1.3786...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1809...  Training loss: 1.3894...  0.4784 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1810...  Training loss: 1.4228...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1811...  Training loss: 1.4244...  0.4781 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1812...  Training loss: 1.4233...  0.4799 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1813...  Training loss: 1.3815...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1814...  Training loss: 1.3796...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1815...  Training loss: 1.4110...  0.4790 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1816...  Training loss: 1.4195...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1817...  Training loss: 1.3946...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1818...  Training loss: 1.4063...  0.4777 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1819...  Training loss: 1.3895...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1820...  Training loss: 1.3644...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1821...  Training loss: 1.3600...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1822...  Training loss: 1.3825...  0.4801 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1823...  Training loss: 1.3788...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1824...  Training loss: 1.4284...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1825...  Training loss: 1.3855...  0.4792 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1826...  Training loss: 1.3657...  0.4794 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1827...  Training loss: 1.4132...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1828...  Training loss: 1.3746...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1829...  Training loss: 1.3903...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1830...  Training loss: 1.3892...  0.4782 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1831...  Training loss: 1.3865...  0.4790 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1832...  Training loss: 1.4187...  0.4781 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1833...  Training loss: 1.3686...  0.4777 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1834...  Training loss: 1.4458...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1835...  Training loss: 1.3940...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1836...  Training loss: 1.4031...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1837...  Training loss: 1.3957...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1838...  Training loss: 1.3962...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1839...  Training loss: 1.4178...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1840...  Training loss: 1.3779...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1841...  Training loss: 1.3732...  0.4779 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1842...  Training loss: 1.4273...  0.4784 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1843...  Training loss: 1.3992...  0.4781 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1844...  Training loss: 1.4491...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1845...  Training loss: 1.4249...  0.4791 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1846...  Training loss: 1.4060...  0.4796 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1847...  Training loss: 1.3893...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1848...  Training loss: 1.4033...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1849...  Training loss: 1.4181...  0.4791 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1850...  Training loss: 1.3831...  0.4793 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1851...  Training loss: 1.3992...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1852...  Training loss: 1.3863...  0.4795 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1853...  Training loss: 1.4522...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1854...  Training loss: 1.4228...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1855...  Training loss: 1.4384...  0.4782 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1856...  Training loss: 1.3875...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1857...  Training loss: 1.4022...  0.4777 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1858...  Training loss: 1.4227...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1859...  Training loss: 1.3933...  0.4794 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1860...  Training loss: 1.3885...  0.4781 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1861...  Training loss: 1.3590...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1862...  Training loss: 1.3954...  0.4779 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1863...  Training loss: 1.3547...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1864...  Training loss: 1.3990...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1865...  Training loss: 1.3574...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1866...  Training loss: 1.3787...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1867...  Training loss: 1.3726...  0.4779 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1868...  Training loss: 1.4009...  0.4796 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1869...  Training loss: 1.3704...  0.4784 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1870...  Training loss: 1.3730...  0.4794 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1871...  Training loss: 1.3597...  0.4795 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1872...  Training loss: 1.4028...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1873...  Training loss: 1.3677...  0.4777 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1874...  Training loss: 1.3833...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1875...  Training loss: 1.3667...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1876...  Training loss: 1.3666...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1877...  Training loss: 1.3733...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1878...  Training loss: 1.3981...  0.4791 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1879...  Training loss: 1.3982...  0.4781 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1880...  Training loss: 1.3646...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1881...  Training loss: 1.3657...  0.4779 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1882...  Training loss: 1.3570...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1883...  Training loss: 1.4035...  0.4779 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1884...  Training loss: 1.3872...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1885...  Training loss: 1.3859...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1886...  Training loss: 1.3879...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1887...  Training loss: 1.3824...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1888...  Training loss: 1.3897...  0.4795 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1889...  Training loss: 1.3932...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1890...  Training loss: 1.3814...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1891...  Training loss: 1.3743...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1892...  Training loss: 1.4072...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1893...  Training loss: 1.3795...  0.4778 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1894...  Training loss: 1.3901...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1895...  Training loss: 1.3840...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1896...  Training loss: 1.3696...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1897...  Training loss: 1.3657...  0.4781 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1898...  Training loss: 1.3368...  0.4782 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1899...  Training loss: 1.3844...  0.4782 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1900...  Training loss: 1.3889...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1901...  Training loss: 1.3751...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1902...  Training loss: 1.3775...  0.4781 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1903...  Training loss: 1.3790...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1904...  Training loss: 1.3592...  0.4804 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1905...  Training loss: 1.3414...  0.4782 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1906...  Training loss: 1.3875...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1907...  Training loss: 1.3776...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1908...  Training loss: 1.3364...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1909...  Training loss: 1.3894...  0.4782 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1910...  Training loss: 1.3901...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1911...  Training loss: 1.3729...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1912...  Training loss: 1.3454...  0.4784 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1913...  Training loss: 1.3362...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1914...  Training loss: 1.3595...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1915...  Training loss: 1.4068...  0.4793 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1916...  Training loss: 1.3854...  0.4776 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1917...  Training loss: 1.3860...  0.4778 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1918...  Training loss: 1.3840...  0.4782 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1919...  Training loss: 1.4145...  0.4776 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1920...  Training loss: 1.3980...  0.4791 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1921...  Training loss: 1.3873...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1922...  Training loss: 1.3802...  0.4790 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1923...  Training loss: 1.4336...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1924...  Training loss: 1.3853...  0.4778 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1925...  Training loss: 1.3840...  0.4792 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1926...  Training loss: 1.4103...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1927...  Training loss: 1.3689...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1928...  Training loss: 1.4050...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1929...  Training loss: 1.3932...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1930...  Training loss: 1.4206...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1931...  Training loss: 1.4017...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1932...  Training loss: 1.3759...  0.4793 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1933...  Training loss: 1.3510...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1934...  Training loss: 1.3707...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1935...  Training loss: 1.3900...  0.4784 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1936...  Training loss: 1.3751...  0.4791 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1937...  Training loss: 1.3881...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1938...  Training loss: 1.3825...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1939...  Training loss: 1.3883...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1940...  Training loss: 1.3793...  0.4793 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1941...  Training loss: 1.3395...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1942...  Training loss: 1.4094...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1943...  Training loss: 1.4157...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1944...  Training loss: 1.3919...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1945...  Training loss: 1.3834...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1946...  Training loss: 1.3851...  0.4791 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1947...  Training loss: 1.3768...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1948...  Training loss: 1.3874...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1949...  Training loss: 1.4058...  0.4794 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1950...  Training loss: 1.4545...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1951...  Training loss: 1.3802...  0.4793 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1952...  Training loss: 1.3744...  0.4797 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1953...  Training loss: 1.3677...  0.4781 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1954...  Training loss: 1.3567...  0.4781 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1955...  Training loss: 1.4045...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1956...  Training loss: 1.3800...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1957...  Training loss: 1.3919...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1958...  Training loss: 1.3528...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1959...  Training loss: 1.3626...  0.4784 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1960...  Training loss: 1.4051...  0.4790 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1961...  Training loss: 1.3639...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1962...  Training loss: 1.3514...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1963...  Training loss: 1.3626...  0.4772 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1964...  Training loss: 1.3707...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1965...  Training loss: 1.3828...  0.4782 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1966...  Training loss: 1.3686...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1967...  Training loss: 1.3746...  0.4794 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1968...  Training loss: 1.3635...  0.4785 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1969...  Training loss: 1.3989...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1970...  Training loss: 1.3833...  0.4782 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1971...  Training loss: 1.3729...  0.4780 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1972...  Training loss: 1.3793...  0.4787 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1973...  Training loss: 1.3532...  0.4783 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1974...  Training loss: 1.3588...  0.4788 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1975...  Training loss: 1.3807...  0.4779 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1976...  Training loss: 1.3581...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1977...  Training loss: 1.3455...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1978...  Training loss: 1.3839...  0.4786 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1979...  Training loss: 1.3658...  0.4789 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1980...  Training loss: 1.3657...  0.4779 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1981...  Training loss: 1.5217...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1982...  Training loss: 1.3900...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1983...  Training loss: 1.3716...  0.4780 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1984...  Training loss: 1.3966...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1985...  Training loss: 1.3571...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1986...  Training loss: 1.3471...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1987...  Training loss: 1.3802...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1988...  Training loss: 1.3734...  0.4778 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1989...  Training loss: 1.3826...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1990...  Training loss: 1.3758...  0.4775 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1991...  Training loss: 1.3491...  0.4790 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1992...  Training loss: 1.3669...  0.4789 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1993...  Training loss: 1.3759...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1994...  Training loss: 1.3886...  0.4795 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1995...  Training loss: 1.3551...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1996...  Training loss: 1.3403...  0.4778 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1997...  Training loss: 1.3691...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1998...  Training loss: 1.3931...  0.4779 sec/batch\n",
      "Epoch: 11/20...  Training Step: 1999...  Training loss: 1.3731...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2000...  Training loss: 1.4051...  0.4781 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2001...  Training loss: 1.3702...  0.4791 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2002...  Training loss: 1.3852...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2003...  Training loss: 1.3579...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2004...  Training loss: 1.3821...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2005...  Training loss: 1.3729...  0.4781 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2006...  Training loss: 1.3309...  0.4794 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2007...  Training loss: 1.3521...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2008...  Training loss: 1.3920...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2009...  Training loss: 1.3838...  0.4794 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2010...  Training loss: 1.3925...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2011...  Training loss: 1.3603...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2012...  Training loss: 1.3462...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2013...  Training loss: 1.3737...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2014...  Training loss: 1.3809...  0.4791 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2015...  Training loss: 1.3648...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2016...  Training loss: 1.3760...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2017...  Training loss: 1.3551...  0.4780 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2018...  Training loss: 1.3253...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2019...  Training loss: 1.3194...  0.4795 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2020...  Training loss: 1.3492...  0.4780 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2021...  Training loss: 1.3496...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2022...  Training loss: 1.4046...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2023...  Training loss: 1.3582...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2024...  Training loss: 1.3383...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2025...  Training loss: 1.3827...  0.4789 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2026...  Training loss: 1.3480...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2027...  Training loss: 1.3619...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2028...  Training loss: 1.3572...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2029...  Training loss: 1.3622...  0.4793 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2030...  Training loss: 1.3872...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2031...  Training loss: 1.3459...  0.4789 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2032...  Training loss: 1.4087...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2033...  Training loss: 1.3642...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2034...  Training loss: 1.3852...  0.4792 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2035...  Training loss: 1.3657...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2036...  Training loss: 1.3699...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2037...  Training loss: 1.3874...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2038...  Training loss: 1.3525...  0.4798 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2039...  Training loss: 1.3449...  0.4797 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2040...  Training loss: 1.4030...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2041...  Training loss: 1.3779...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2042...  Training loss: 1.4123...  0.4781 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2043...  Training loss: 1.3908...  0.4789 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2044...  Training loss: 1.3821...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2045...  Training loss: 1.3644...  0.4777 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2046...  Training loss: 1.3774...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2047...  Training loss: 1.3814...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2048...  Training loss: 1.3463...  0.4776 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2049...  Training loss: 1.3707...  0.4779 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2050...  Training loss: 1.3590...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2051...  Training loss: 1.4151...  0.4781 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2052...  Training loss: 1.3857...  0.4778 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2053...  Training loss: 1.3988...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2054...  Training loss: 1.3506...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2055...  Training loss: 1.3566...  0.4798 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2056...  Training loss: 1.3848...  0.4776 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2057...  Training loss: 1.3620...  0.4776 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2058...  Training loss: 1.3509...  0.4790 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2059...  Training loss: 1.3245...  0.4792 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2060...  Training loss: 1.3659...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2061...  Training loss: 1.3300...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2062...  Training loss: 1.3697...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2063...  Training loss: 1.3269...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2064...  Training loss: 1.3594...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2065...  Training loss: 1.3421...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2066...  Training loss: 1.3610...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2067...  Training loss: 1.3365...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2068...  Training loss: 1.3500...  0.4797 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2069...  Training loss: 1.3325...  0.4793 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2070...  Training loss: 1.3756...  0.4791 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2071...  Training loss: 1.3378...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2072...  Training loss: 1.3539...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2073...  Training loss: 1.3279...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2074...  Training loss: 1.3432...  0.4791 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2075...  Training loss: 1.3442...  0.4778 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2076...  Training loss: 1.3714...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2077...  Training loss: 1.3698...  0.4793 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2078...  Training loss: 1.3250...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2079...  Training loss: 1.3348...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2080...  Training loss: 1.3275...  0.4795 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2081...  Training loss: 1.3723...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2082...  Training loss: 1.3532...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2083...  Training loss: 1.3575...  0.4780 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2084...  Training loss: 1.3537...  0.4781 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2085...  Training loss: 1.3545...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2086...  Training loss: 1.3554...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2087...  Training loss: 1.3622...  0.4777 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2088...  Training loss: 1.3628...  0.4780 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2089...  Training loss: 1.3510...  0.4789 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2090...  Training loss: 1.3727...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2091...  Training loss: 1.3448...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2092...  Training loss: 1.3613...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2093...  Training loss: 1.3530...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2094...  Training loss: 1.3482...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2095...  Training loss: 1.3386...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2096...  Training loss: 1.3168...  0.4781 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2097...  Training loss: 1.3684...  0.4789 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2098...  Training loss: 1.3599...  0.4792 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2099...  Training loss: 1.3513...  0.4780 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2100...  Training loss: 1.3509...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2101...  Training loss: 1.3518...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2102...  Training loss: 1.3194...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2103...  Training loss: 1.3111...  0.4790 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2104...  Training loss: 1.3534...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2105...  Training loss: 1.3419...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2106...  Training loss: 1.3005...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2107...  Training loss: 1.3658...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2108...  Training loss: 1.3607...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2109...  Training loss: 1.3453...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2110...  Training loss: 1.3199...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2111...  Training loss: 1.2956...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2112...  Training loss: 1.3326...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2113...  Training loss: 1.3780...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2114...  Training loss: 1.3616...  0.4781 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2115...  Training loss: 1.3676...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2116...  Training loss: 1.3560...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2117...  Training loss: 1.3876...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2118...  Training loss: 1.3749...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2119...  Training loss: 1.3617...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2120...  Training loss: 1.3585...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2121...  Training loss: 1.4039...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2122...  Training loss: 1.3599...  0.4779 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2123...  Training loss: 1.3493...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2124...  Training loss: 1.3898...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2125...  Training loss: 1.3394...  0.4793 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2126...  Training loss: 1.3772...  0.4777 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2127...  Training loss: 1.3599...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2128...  Training loss: 1.3960...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2129...  Training loss: 1.3804...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2130...  Training loss: 1.3593...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2131...  Training loss: 1.3276...  0.4791 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2132...  Training loss: 1.3302...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2133...  Training loss: 1.3630...  0.4781 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2134...  Training loss: 1.3524...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2135...  Training loss: 1.3497...  0.4778 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2136...  Training loss: 1.3465...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2137...  Training loss: 1.3588...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2138...  Training loss: 1.3459...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2139...  Training loss: 1.3195...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2140...  Training loss: 1.3844...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2141...  Training loss: 1.3899...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2142...  Training loss: 1.3698...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2143...  Training loss: 1.3470...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2144...  Training loss: 1.3566...  0.4794 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2145...  Training loss: 1.3599...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2146...  Training loss: 1.3484...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2147...  Training loss: 1.3725...  0.4790 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2148...  Training loss: 1.4189...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2149...  Training loss: 1.3588...  0.4798 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2150...  Training loss: 1.3578...  0.4797 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2151...  Training loss: 1.3437...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2152...  Training loss: 1.3397...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2153...  Training loss: 1.3787...  0.4792 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2154...  Training loss: 1.3435...  0.4779 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2155...  Training loss: 1.3599...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2156...  Training loss: 1.3300...  0.4797 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2157...  Training loss: 1.3415...  0.4792 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2158...  Training loss: 1.3850...  0.4791 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2159...  Training loss: 1.3434...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2160...  Training loss: 1.3244...  0.4800 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2161...  Training loss: 1.3344...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2162...  Training loss: 1.3451...  0.4791 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2163...  Training loss: 1.3469...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2164...  Training loss: 1.3400...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2165...  Training loss: 1.3476...  0.4783 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2166...  Training loss: 1.3387...  0.4787 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2167...  Training loss: 1.3707...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2168...  Training loss: 1.3460...  0.4789 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2169...  Training loss: 1.3431...  0.4790 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2170...  Training loss: 1.3395...  0.4788 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2171...  Training loss: 1.3303...  0.4789 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2172...  Training loss: 1.3287...  0.4784 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2173...  Training loss: 1.3510...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2174...  Training loss: 1.3308...  0.4782 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2175...  Training loss: 1.3193...  0.4791 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2176...  Training loss: 1.3507...  0.4786 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2177...  Training loss: 1.3429...  0.4785 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2178...  Training loss: 1.3302...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2179...  Training loss: 1.4966...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2180...  Training loss: 1.3817...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2181...  Training loss: 1.3544...  0.4780 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2182...  Training loss: 1.3716...  0.4777 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2183...  Training loss: 1.3258...  0.4776 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2184...  Training loss: 1.3246...  0.4780 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2185...  Training loss: 1.3554...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2186...  Training loss: 1.3496...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2187...  Training loss: 1.3623...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2188...  Training loss: 1.3431...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2189...  Training loss: 1.3392...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2190...  Training loss: 1.3402...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2191...  Training loss: 1.3573...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2192...  Training loss: 1.3702...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2193...  Training loss: 1.3320...  0.4789 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2194...  Training loss: 1.3292...  0.4791 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2195...  Training loss: 1.3614...  0.4794 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2196...  Training loss: 1.3721...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2197...  Training loss: 1.3589...  0.4791 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2198...  Training loss: 1.3778...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2199...  Training loss: 1.3468...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2200...  Training loss: 1.3587...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2201...  Training loss: 1.3427...  0.4790 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2202...  Training loss: 1.3578...  0.4781 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2203...  Training loss: 1.3518...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2204...  Training loss: 1.3037...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2205...  Training loss: 1.3164...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2206...  Training loss: 1.3762...  0.4777 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2207...  Training loss: 1.3608...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2208...  Training loss: 1.3544...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2209...  Training loss: 1.3427...  0.4790 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2210...  Training loss: 1.3263...  0.4793 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2211...  Training loss: 1.3585...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2212...  Training loss: 1.3517...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2213...  Training loss: 1.3391...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2214...  Training loss: 1.3431...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2215...  Training loss: 1.3235...  0.4781 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2216...  Training loss: 1.3008...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2217...  Training loss: 1.2974...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2218...  Training loss: 1.3323...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2219...  Training loss: 1.3228...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2220...  Training loss: 1.3829...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2221...  Training loss: 1.3244...  0.4788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2222...  Training loss: 1.3187...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2223...  Training loss: 1.3677...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2224...  Training loss: 1.3161...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2225...  Training loss: 1.3372...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2226...  Training loss: 1.3322...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2227...  Training loss: 1.3413...  0.4788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2228...  Training loss: 1.3534...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2229...  Training loss: 1.3124...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2230...  Training loss: 1.3789...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2231...  Training loss: 1.3436...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2232...  Training loss: 1.3623...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2233...  Training loss: 1.3428...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2234...  Training loss: 1.3434...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2235...  Training loss: 1.3633...  0.4790 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2236...  Training loss: 1.3335...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2237...  Training loss: 1.3216...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2238...  Training loss: 1.3692...  0.4789 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2239...  Training loss: 1.3585...  0.4781 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2240...  Training loss: 1.3938...  0.4794 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2241...  Training loss: 1.3642...  0.4781 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2242...  Training loss: 1.3469...  0.4778 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2243...  Training loss: 1.3319...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2244...  Training loss: 1.3509...  0.4789 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2245...  Training loss: 1.3618...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2246...  Training loss: 1.3233...  0.4795 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2247...  Training loss: 1.3455...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2248...  Training loss: 1.3307...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2249...  Training loss: 1.3937...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2250...  Training loss: 1.3639...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2251...  Training loss: 1.3811...  0.4788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2252...  Training loss: 1.3257...  0.4788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2253...  Training loss: 1.3386...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2254...  Training loss: 1.3608...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2255...  Training loss: 1.3393...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2256...  Training loss: 1.3292...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2257...  Training loss: 1.3009...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2258...  Training loss: 1.3368...  0.4777 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2259...  Training loss: 1.3049...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2260...  Training loss: 1.3422...  0.4791 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2261...  Training loss: 1.3120...  0.4793 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2262...  Training loss: 1.3306...  0.4788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2263...  Training loss: 1.3207...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2264...  Training loss: 1.3423...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2265...  Training loss: 1.3120...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2266...  Training loss: 1.3113...  0.4788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2267...  Training loss: 1.3074...  0.4788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2268...  Training loss: 1.3342...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2269...  Training loss: 1.3252...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2270...  Training loss: 1.3212...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2271...  Training loss: 1.3039...  0.4793 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2272...  Training loss: 1.3178...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2273...  Training loss: 1.3158...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2274...  Training loss: 1.3502...  0.4800 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2275...  Training loss: 1.3447...  0.4789 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2276...  Training loss: 1.3038...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2277...  Training loss: 1.3114...  0.4794 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2278...  Training loss: 1.3094...  0.4792 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2279...  Training loss: 1.3425...  0.4797 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2280...  Training loss: 1.3314...  0.4788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2281...  Training loss: 1.3397...  0.4791 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2282...  Training loss: 1.3313...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2283...  Training loss: 1.3300...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2284...  Training loss: 1.3341...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2285...  Training loss: 1.3314...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2286...  Training loss: 1.3397...  0.4778 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2287...  Training loss: 1.3207...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2288...  Training loss: 1.3562...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2289...  Training loss: 1.3233...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2290...  Training loss: 1.3452...  0.4796 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2291...  Training loss: 1.3320...  0.4790 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2292...  Training loss: 1.3211...  0.4790 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2293...  Training loss: 1.3085...  0.4792 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2294...  Training loss: 1.2898...  0.4793 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2295...  Training loss: 1.3442...  0.4792 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2296...  Training loss: 1.3419...  0.4781 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2297...  Training loss: 1.3211...  0.4790 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2298...  Training loss: 1.3300...  0.4793 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2299...  Training loss: 1.3386...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2300...  Training loss: 1.2963...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2301...  Training loss: 1.2908...  0.4796 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2302...  Training loss: 1.3377...  0.4789 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2303...  Training loss: 1.3237...  0.4790 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2304...  Training loss: 1.2877...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2305...  Training loss: 1.3403...  0.4791 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2306...  Training loss: 1.3391...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2307...  Training loss: 1.3185...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2308...  Training loss: 1.2957...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2309...  Training loss: 1.2798...  0.4801 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2310...  Training loss: 1.3134...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2311...  Training loss: 1.3594...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2312...  Training loss: 1.3460...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2313...  Training loss: 1.3385...  0.4792 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2314...  Training loss: 1.3313...  0.4789 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2315...  Training loss: 1.3649...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2316...  Training loss: 1.3457...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2317...  Training loss: 1.3409...  0.4790 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2318...  Training loss: 1.3475...  0.4793 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2319...  Training loss: 1.3805...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2320...  Training loss: 1.3439...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2321...  Training loss: 1.3312...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2322...  Training loss: 1.3685...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2323...  Training loss: 1.3215...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2324...  Training loss: 1.3498...  0.4793 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2325...  Training loss: 1.3470...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2326...  Training loss: 1.3733...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2327...  Training loss: 1.3609...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2328...  Training loss: 1.3322...  0.4788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2329...  Training loss: 1.2998...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2330...  Training loss: 1.3104...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2331...  Training loss: 1.3364...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2332...  Training loss: 1.3276...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2333...  Training loss: 1.3298...  0.4781 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2334...  Training loss: 1.3231...  0.4790 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2335...  Training loss: 1.3349...  0.4791 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2336...  Training loss: 1.3240...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2337...  Training loss: 1.2881...  0.4778 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2338...  Training loss: 1.3513...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2339...  Training loss: 1.3622...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2340...  Training loss: 1.3404...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2341...  Training loss: 1.3288...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2342...  Training loss: 1.3362...  0.4780 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2343...  Training loss: 1.3428...  0.4784 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2344...  Training loss: 1.3374...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2345...  Training loss: 1.3477...  0.4788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2346...  Training loss: 1.3960...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2347...  Training loss: 1.3405...  0.4791 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2348...  Training loss: 1.3269...  0.4794 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2349...  Training loss: 1.3207...  0.4788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2350...  Training loss: 1.3130...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2351...  Training loss: 1.3551...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2352...  Training loss: 1.3305...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2353...  Training loss: 1.3368...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2354...  Training loss: 1.3107...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2355...  Training loss: 1.3169...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2356...  Training loss: 1.3701...  0.4791 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2357...  Training loss: 1.2994...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2358...  Training loss: 1.3007...  0.4778 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2359...  Training loss: 1.3122...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2360...  Training loss: 1.3178...  0.4783 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2361...  Training loss: 1.3277...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2362...  Training loss: 1.3179...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2363...  Training loss: 1.3168...  0.4786 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2364...  Training loss: 1.3100...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2365...  Training loss: 1.3474...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2366...  Training loss: 1.3211...  0.4789 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2367...  Training loss: 1.3186...  0.4779 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2368...  Training loss: 1.3225...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2369...  Training loss: 1.3081...  0.4781 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2370...  Training loss: 1.3033...  0.4787 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2371...  Training loss: 1.3342...  0.4780 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2372...  Training loss: 1.3171...  0.4793 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2373...  Training loss: 1.2912...  0.4782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2374...  Training loss: 1.3324...  0.4785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2375...  Training loss: 1.3186...  0.4775 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2376...  Training loss: 1.3041...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2377...  Training loss: 1.4881...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2378...  Training loss: 1.3604...  0.4779 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2379...  Training loss: 1.3356...  0.4779 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2380...  Training loss: 1.3488...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2381...  Training loss: 1.3159...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2382...  Training loss: 1.2930...  0.4778 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2383...  Training loss: 1.3187...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2384...  Training loss: 1.3230...  0.4775 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2385...  Training loss: 1.3262...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2386...  Training loss: 1.3137...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2387...  Training loss: 1.3123...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2388...  Training loss: 1.3288...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2389...  Training loss: 1.3313...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2390...  Training loss: 1.3359...  0.4788 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2391...  Training loss: 1.3209...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2392...  Training loss: 1.3019...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2393...  Training loss: 1.3395...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2394...  Training loss: 1.3515...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2395...  Training loss: 1.3329...  0.4789 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2396...  Training loss: 1.3535...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2397...  Training loss: 1.3233...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2398...  Training loss: 1.3454...  0.4783 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2399...  Training loss: 1.3155...  0.4788 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2400...  Training loss: 1.3443...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2401...  Training loss: 1.3273...  0.4791 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2402...  Training loss: 1.2798...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2403...  Training loss: 1.2957...  0.4799 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2404...  Training loss: 1.3366...  0.4793 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2405...  Training loss: 1.3307...  0.4783 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2406...  Training loss: 1.3339...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2407...  Training loss: 1.3097...  0.4779 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2408...  Training loss: 1.2974...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2409...  Training loss: 1.3295...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2410...  Training loss: 1.3266...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2411...  Training loss: 1.3100...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2412...  Training loss: 1.3257...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2413...  Training loss: 1.2980...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2414...  Training loss: 1.2809...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2415...  Training loss: 1.2848...  0.4783 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2416...  Training loss: 1.3055...  0.4795 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2417...  Training loss: 1.3096...  0.4789 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2418...  Training loss: 1.3567...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2419...  Training loss: 1.3115...  0.4779 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2420...  Training loss: 1.2944...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2421...  Training loss: 1.3357...  0.4791 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2422...  Training loss: 1.2993...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2423...  Training loss: 1.3037...  0.4791 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2424...  Training loss: 1.3102...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2425...  Training loss: 1.3226...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2426...  Training loss: 1.3427...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2427...  Training loss: 1.2922...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2428...  Training loss: 1.3566...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2429...  Training loss: 1.3257...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2430...  Training loss: 1.3329...  0.4796 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2431...  Training loss: 1.3156...  0.4790 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2432...  Training loss: 1.3175...  0.4791 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2433...  Training loss: 1.3417...  0.4788 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2434...  Training loss: 1.3071...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2435...  Training loss: 1.2992...  0.4789 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2436...  Training loss: 1.3487...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2437...  Training loss: 1.3263...  0.4789 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2438...  Training loss: 1.3796...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2439...  Training loss: 1.3515...  0.4778 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2440...  Training loss: 1.3395...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2441...  Training loss: 1.3172...  0.4783 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2442...  Training loss: 1.3401...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2443...  Training loss: 1.3482...  0.4783 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2444...  Training loss: 1.3122...  0.4788 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2445...  Training loss: 1.3250...  0.4798 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2446...  Training loss: 1.3132...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2447...  Training loss: 1.3678...  0.4795 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2448...  Training loss: 1.3521...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2449...  Training loss: 1.3549...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2450...  Training loss: 1.3076...  0.4792 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2451...  Training loss: 1.3187...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2452...  Training loss: 1.3373...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2453...  Training loss: 1.3147...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2454...  Training loss: 1.3095...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2455...  Training loss: 1.2895...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2456...  Training loss: 1.3205...  0.4783 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2457...  Training loss: 1.2858...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2458...  Training loss: 1.3195...  0.4774 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2459...  Training loss: 1.2944...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2460...  Training loss: 1.3157...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2461...  Training loss: 1.2996...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2462...  Training loss: 1.3235...  0.4790 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2463...  Training loss: 1.3007...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2464...  Training loss: 1.2956...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2465...  Training loss: 1.2852...  0.4774 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2466...  Training loss: 1.3215...  0.4791 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2467...  Training loss: 1.3016...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2468...  Training loss: 1.3059...  0.4776 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2469...  Training loss: 1.2871...  0.4779 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2470...  Training loss: 1.2880...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2471...  Training loss: 1.2985...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2472...  Training loss: 1.3186...  0.4793 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2473...  Training loss: 1.3189...  0.4779 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2474...  Training loss: 1.2777...  0.4788 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2475...  Training loss: 1.2881...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2476...  Training loss: 1.2819...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2477...  Training loss: 1.3248...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2478...  Training loss: 1.2973...  0.4790 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2479...  Training loss: 1.3164...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2480...  Training loss: 1.3053...  0.4794 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2481...  Training loss: 1.3085...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2482...  Training loss: 1.3142...  0.4783 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2483...  Training loss: 1.3226...  0.4783 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2484...  Training loss: 1.3195...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2485...  Training loss: 1.2969...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2486...  Training loss: 1.3270...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2487...  Training loss: 1.3025...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2488...  Training loss: 1.3217...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2489...  Training loss: 1.3237...  0.4778 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2490...  Training loss: 1.3056...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2491...  Training loss: 1.2879...  0.4796 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2492...  Training loss: 1.2788...  0.4781 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2493...  Training loss: 1.3153...  0.4788 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2494...  Training loss: 1.3126...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2495...  Training loss: 1.2996...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2496...  Training loss: 1.3046...  0.4788 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2497...  Training loss: 1.3108...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2498...  Training loss: 1.2755...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2499...  Training loss: 1.2612...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2500...  Training loss: 1.3184...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2501...  Training loss: 1.3104...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2502...  Training loss: 1.2645...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2503...  Training loss: 1.3258...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2504...  Training loss: 1.3202...  0.4776 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2505...  Training loss: 1.3023...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2506...  Training loss: 1.2788...  0.4783 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2507...  Training loss: 1.2648...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2508...  Training loss: 1.2865...  0.4777 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2509...  Training loss: 1.3367...  0.4791 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2510...  Training loss: 1.3165...  0.4776 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2511...  Training loss: 1.3134...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2512...  Training loss: 1.3175...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2513...  Training loss: 1.3449...  0.4776 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2514...  Training loss: 1.3259...  0.4800 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2515...  Training loss: 1.3194...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2516...  Training loss: 1.3198...  0.4796 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2517...  Training loss: 1.3606...  0.4790 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2518...  Training loss: 1.3219...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2519...  Training loss: 1.3044...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2520...  Training loss: 1.3486...  0.4790 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2521...  Training loss: 1.3000...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2522...  Training loss: 1.3327...  0.4790 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2523...  Training loss: 1.3261...  0.4788 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2524...  Training loss: 1.3484...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2525...  Training loss: 1.3427...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2526...  Training loss: 1.3153...  0.4789 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2527...  Training loss: 1.2823...  0.4788 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2528...  Training loss: 1.2904...  0.4789 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2529...  Training loss: 1.3250...  0.4789 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2530...  Training loss: 1.3016...  0.4791 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2531...  Training loss: 1.3093...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2532...  Training loss: 1.3038...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2533...  Training loss: 1.3106...  0.4797 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2534...  Training loss: 1.3040...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2535...  Training loss: 1.2754...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2536...  Training loss: 1.3262...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2537...  Training loss: 1.3334...  0.4789 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2538...  Training loss: 1.3202...  0.4787 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2539...  Training loss: 1.3126...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2540...  Training loss: 1.3083...  0.4777 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2541...  Training loss: 1.3167...  0.4794 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2542...  Training loss: 1.3131...  0.4788 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2543...  Training loss: 1.3374...  0.4789 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2544...  Training loss: 1.3802...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2545...  Training loss: 1.3258...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2546...  Training loss: 1.3182...  0.4796 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2547...  Training loss: 1.3045...  0.4788 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2548...  Training loss: 1.3050...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2549...  Training loss: 1.3480...  0.4783 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2550...  Training loss: 1.3163...  0.4780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2551...  Training loss: 1.3232...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2552...  Training loss: 1.2874...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2553...  Training loss: 1.3013...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2554...  Training loss: 1.3410...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2555...  Training loss: 1.2885...  0.4784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2556...  Training loss: 1.2857...  0.4777 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2557...  Training loss: 1.3001...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2558...  Training loss: 1.3009...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2559...  Training loss: 1.3098...  0.4790 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2560...  Training loss: 1.2914...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2561...  Training loss: 1.3042...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2562...  Training loss: 1.2960...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2563...  Training loss: 1.3319...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2564...  Training loss: 1.3050...  0.4789 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2565...  Training loss: 1.3023...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2566...  Training loss: 1.2988...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2567...  Training loss: 1.2904...  0.4786 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2568...  Training loss: 1.2856...  0.4782 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2569...  Training loss: 1.3085...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2570...  Training loss: 1.2915...  0.4777 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2571...  Training loss: 1.2774...  0.4785 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2572...  Training loss: 1.3121...  0.4794 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2573...  Training loss: 1.3014...  0.4774 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2574...  Training loss: 1.2999...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2575...  Training loss: 1.4617...  0.4777 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2576...  Training loss: 1.3245...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2577...  Training loss: 1.3102...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2578...  Training loss: 1.3274...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2579...  Training loss: 1.2878...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2580...  Training loss: 1.2618...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2581...  Training loss: 1.2997...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2582...  Training loss: 1.3100...  0.4797 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2583...  Training loss: 1.3132...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2584...  Training loss: 1.2926...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2585...  Training loss: 1.2911...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2586...  Training loss: 1.2974...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2587...  Training loss: 1.2996...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2588...  Training loss: 1.3231...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2589...  Training loss: 1.2910...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2590...  Training loss: 1.2836...  0.4794 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2591...  Training loss: 1.3168...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2592...  Training loss: 1.3280...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2593...  Training loss: 1.3061...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2594...  Training loss: 1.3355...  0.4780 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2595...  Training loss: 1.2989...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2596...  Training loss: 1.3250...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2597...  Training loss: 1.3054...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2598...  Training loss: 1.3149...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2599...  Training loss: 1.3116...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2600...  Training loss: 1.2629...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2601...  Training loss: 1.2756...  0.4799 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2602...  Training loss: 1.3272...  0.4793 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2603...  Training loss: 1.3211...  0.4795 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2604...  Training loss: 1.3177...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2605...  Training loss: 1.2800...  0.4779 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2606...  Training loss: 1.2712...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2607...  Training loss: 1.3116...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2608...  Training loss: 1.3090...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2609...  Training loss: 1.2989...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2610...  Training loss: 1.3165...  0.4777 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2611...  Training loss: 1.2951...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2612...  Training loss: 1.2682...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2613...  Training loss: 1.2586...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2614...  Training loss: 1.2934...  0.4779 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2615...  Training loss: 1.2825...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2616...  Training loss: 1.3422...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2617...  Training loss: 1.3005...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2618...  Training loss: 1.2829...  0.4792 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2619...  Training loss: 1.3128...  0.4793 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2620...  Training loss: 1.2778...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2621...  Training loss: 1.2968...  0.4786 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2622...  Training loss: 1.2925...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2623...  Training loss: 1.2949...  0.4777 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2624...  Training loss: 1.3175...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2625...  Training loss: 1.2728...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2626...  Training loss: 1.3456...  0.4796 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2627...  Training loss: 1.3086...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2628...  Training loss: 1.3186...  0.4779 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2629...  Training loss: 1.2979...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2630...  Training loss: 1.3049...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2631...  Training loss: 1.3103...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2632...  Training loss: 1.2952...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2633...  Training loss: 1.2808...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2634...  Training loss: 1.3309...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2635...  Training loss: 1.3128...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2636...  Training loss: 1.3669...  0.4778 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2637...  Training loss: 1.3346...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2638...  Training loss: 1.3080...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2639...  Training loss: 1.3028...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2640...  Training loss: 1.3173...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2641...  Training loss: 1.3219...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2642...  Training loss: 1.2986...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2643...  Training loss: 1.3079...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2644...  Training loss: 1.2948...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2645...  Training loss: 1.3448...  0.4794 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2646...  Training loss: 1.3331...  0.4776 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2647...  Training loss: 1.3417...  0.4778 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2648...  Training loss: 1.2782...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2649...  Training loss: 1.2994...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2650...  Training loss: 1.3200...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2651...  Training loss: 1.3035...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2652...  Training loss: 1.2873...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2653...  Training loss: 1.2593...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2654...  Training loss: 1.3077...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2655...  Training loss: 1.2717...  0.4792 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2656...  Training loss: 1.3066...  0.4777 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2657...  Training loss: 1.2781...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2658...  Training loss: 1.2939...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2659...  Training loss: 1.2686...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2660...  Training loss: 1.2957...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2661...  Training loss: 1.2738...  0.4786 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2662...  Training loss: 1.2848...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2663...  Training loss: 1.2697...  0.4774 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2664...  Training loss: 1.3131...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2665...  Training loss: 1.2830...  0.4790 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2666...  Training loss: 1.2936...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2667...  Training loss: 1.2720...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2668...  Training loss: 1.2786...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2669...  Training loss: 1.2817...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2670...  Training loss: 1.3076...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2671...  Training loss: 1.3053...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2672...  Training loss: 1.2736...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2673...  Training loss: 1.2695...  0.4792 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2674...  Training loss: 1.2719...  0.4769 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2675...  Training loss: 1.2970...  0.4780 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2676...  Training loss: 1.2895...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2677...  Training loss: 1.2934...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2678...  Training loss: 1.2873...  0.4778 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2679...  Training loss: 1.2937...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2680...  Training loss: 1.2843...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2681...  Training loss: 1.3000...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2682...  Training loss: 1.3008...  0.4780 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2683...  Training loss: 1.2863...  0.4779 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2684...  Training loss: 1.3048...  0.4793 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2685...  Training loss: 1.2742...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2686...  Training loss: 1.3043...  0.4775 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2687...  Training loss: 1.2960...  0.4786 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2688...  Training loss: 1.2888...  0.4777 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2689...  Training loss: 1.2713...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2690...  Training loss: 1.2685...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2691...  Training loss: 1.2962...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2692...  Training loss: 1.3077...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2693...  Training loss: 1.2907...  0.4775 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2694...  Training loss: 1.2797...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2695...  Training loss: 1.2973...  0.4779 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2696...  Training loss: 1.2553...  0.4786 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2697...  Training loss: 1.2449...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2698...  Training loss: 1.2970...  0.4777 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2699...  Training loss: 1.2894...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2700...  Training loss: 1.2493...  0.4779 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2701...  Training loss: 1.3139...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2702...  Training loss: 1.3036...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2703...  Training loss: 1.2796...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2704...  Training loss: 1.2517...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2705...  Training loss: 1.2514...  0.4779 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2706...  Training loss: 1.2757...  0.4779 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2707...  Training loss: 1.3111...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2708...  Training loss: 1.2987...  0.4792 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2709...  Training loss: 1.3037...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2710...  Training loss: 1.2930...  0.4778 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2711...  Training loss: 1.3280...  0.4773 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2712...  Training loss: 1.3010...  0.4786 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2713...  Training loss: 1.3068...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2714...  Training loss: 1.2996...  0.4779 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2715...  Training loss: 1.3464...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2716...  Training loss: 1.3013...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2717...  Training loss: 1.2889...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2718...  Training loss: 1.3249...  0.4786 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2719...  Training loss: 1.2781...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2720...  Training loss: 1.3207...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2721...  Training loss: 1.2989...  0.4794 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2722...  Training loss: 1.3249...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2723...  Training loss: 1.3176...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2724...  Training loss: 1.2988...  0.4794 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2725...  Training loss: 1.2676...  0.4774 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2726...  Training loss: 1.2791...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2727...  Training loss: 1.3036...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2728...  Training loss: 1.2926...  0.4780 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2729...  Training loss: 1.2864...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2730...  Training loss: 1.2804...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2731...  Training loss: 1.2874...  0.4777 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2732...  Training loss: 1.2799...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2733...  Training loss: 1.2601...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2734...  Training loss: 1.3161...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2735...  Training loss: 1.3249...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2736...  Training loss: 1.3065...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2737...  Training loss: 1.2911...  0.4780 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2738...  Training loss: 1.2984...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2739...  Training loss: 1.2923...  0.4786 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2740...  Training loss: 1.2973...  0.4795 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2741...  Training loss: 1.3157...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2742...  Training loss: 1.3572...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2743...  Training loss: 1.3048...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2744...  Training loss: 1.3006...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2745...  Training loss: 1.2830...  0.4786 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2746...  Training loss: 1.2824...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2747...  Training loss: 1.3251...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2748...  Training loss: 1.2952...  0.4781 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2749...  Training loss: 1.3012...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2750...  Training loss: 1.2640...  0.4797 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2751...  Training loss: 1.2861...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2752...  Training loss: 1.3227...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2753...  Training loss: 1.2787...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2754...  Training loss: 1.2639...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2755...  Training loss: 1.2771...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2756...  Training loss: 1.2907...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2757...  Training loss: 1.2934...  0.4794 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2758...  Training loss: 1.2824...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2759...  Training loss: 1.2851...  0.4783 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2760...  Training loss: 1.2762...  0.4790 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2761...  Training loss: 1.3234...  0.4778 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2762...  Training loss: 1.2891...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2763...  Training loss: 1.2918...  0.4784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2764...  Training loss: 1.2887...  0.4786 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2765...  Training loss: 1.2702...  0.4787 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2766...  Training loss: 1.2795...  0.4782 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2767...  Training loss: 1.2908...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2768...  Training loss: 1.2715...  0.4795 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2769...  Training loss: 1.2510...  0.4785 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2770...  Training loss: 1.3053...  0.4789 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2771...  Training loss: 1.2808...  0.4788 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2772...  Training loss: 1.2841...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2773...  Training loss: 1.4192...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2774...  Training loss: 1.3255...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2775...  Training loss: 1.2967...  0.4791 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2776...  Training loss: 1.3202...  0.4779 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2777...  Training loss: 1.2689...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2778...  Training loss: 1.2621...  0.4793 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2779...  Training loss: 1.2987...  0.4793 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2780...  Training loss: 1.2844...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2781...  Training loss: 1.2907...  0.4791 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2782...  Training loss: 1.2829...  0.4782 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2783...  Training loss: 1.2727...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2784...  Training loss: 1.2844...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2785...  Training loss: 1.2924...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2786...  Training loss: 1.3053...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2787...  Training loss: 1.2720...  0.4793 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2788...  Training loss: 1.2658...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2789...  Training loss: 1.2965...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2790...  Training loss: 1.3197...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2791...  Training loss: 1.2908...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2792...  Training loss: 1.3124...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2793...  Training loss: 1.2773...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2794...  Training loss: 1.3021...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2795...  Training loss: 1.2840...  0.4793 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2796...  Training loss: 1.2945...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2797...  Training loss: 1.2941...  0.4791 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2798...  Training loss: 1.2471...  0.4782 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2799...  Training loss: 1.2653...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2800...  Training loss: 1.3064...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2801...  Training loss: 1.2887...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2802...  Training loss: 1.3056...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2803...  Training loss: 1.2741...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2804...  Training loss: 1.2572...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2805...  Training loss: 1.2900...  0.4792 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2806...  Training loss: 1.2939...  0.4798 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2807...  Training loss: 1.2700...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2808...  Training loss: 1.2937...  0.4799 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2809...  Training loss: 1.2776...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2810...  Training loss: 1.2473...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2811...  Training loss: 1.2545...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2812...  Training loss: 1.2630...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2813...  Training loss: 1.2599...  0.4783 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2814...  Training loss: 1.3243...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2815...  Training loss: 1.2678...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2816...  Training loss: 1.2495...  0.4783 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2817...  Training loss: 1.2840...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2818...  Training loss: 1.2555...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2819...  Training loss: 1.2699...  0.4792 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2820...  Training loss: 1.2740...  0.4775 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2821...  Training loss: 1.2825...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2822...  Training loss: 1.2984...  0.4791 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2823...  Training loss: 1.2492...  0.4783 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2824...  Training loss: 1.3197...  0.4777 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2825...  Training loss: 1.2910...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2826...  Training loss: 1.3018...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2827...  Training loss: 1.2676...  0.4795 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2828...  Training loss: 1.2974...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2829...  Training loss: 1.2968...  0.4798 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2830...  Training loss: 1.2685...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2831...  Training loss: 1.2561...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2832...  Training loss: 1.3241...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2833...  Training loss: 1.2954...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2834...  Training loss: 1.3323...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2835...  Training loss: 1.3102...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2836...  Training loss: 1.2981...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2837...  Training loss: 1.2789...  0.4777 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2838...  Training loss: 1.3011...  0.4781 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2839...  Training loss: 1.3052...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2840...  Training loss: 1.2821...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2841...  Training loss: 1.2891...  0.4798 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2842...  Training loss: 1.2738...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2843...  Training loss: 1.3222...  0.4792 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2844...  Training loss: 1.3164...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2845...  Training loss: 1.3247...  0.4779 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2846...  Training loss: 1.2625...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2847...  Training loss: 1.2847...  0.4795 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2848...  Training loss: 1.3094...  0.4779 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2849...  Training loss: 1.2838...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2850...  Training loss: 1.2618...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2851...  Training loss: 1.2458...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2852...  Training loss: 1.2862...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2853...  Training loss: 1.2519...  0.4779 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2854...  Training loss: 1.2900...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2855...  Training loss: 1.2543...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2856...  Training loss: 1.2880...  0.4794 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2857...  Training loss: 1.2631...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2858...  Training loss: 1.2737...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2859...  Training loss: 1.2625...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2860...  Training loss: 1.2622...  0.4783 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2861...  Training loss: 1.2567...  0.4794 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2862...  Training loss: 1.2887...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2863...  Training loss: 1.2567...  0.4798 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2864...  Training loss: 1.2720...  0.4793 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2865...  Training loss: 1.2585...  0.4796 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2866...  Training loss: 1.2530...  0.4780 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2867...  Training loss: 1.2641...  0.4777 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2868...  Training loss: 1.2907...  0.4782 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2869...  Training loss: 1.2917...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2870...  Training loss: 1.2514...  0.4778 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2871...  Training loss: 1.2646...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2872...  Training loss: 1.2607...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2873...  Training loss: 1.2875...  0.4781 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2874...  Training loss: 1.2706...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2875...  Training loss: 1.2742...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2876...  Training loss: 1.2659...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2877...  Training loss: 1.2820...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2878...  Training loss: 1.2777...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2879...  Training loss: 1.2856...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2880...  Training loss: 1.2774...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2881...  Training loss: 1.2705...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2882...  Training loss: 1.3018...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2883...  Training loss: 1.2694...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2884...  Training loss: 1.2863...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2885...  Training loss: 1.2771...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2886...  Training loss: 1.2774...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2887...  Training loss: 1.2644...  0.4782 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2888...  Training loss: 1.2531...  0.4776 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2889...  Training loss: 1.2911...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2890...  Training loss: 1.2861...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2891...  Training loss: 1.2744...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2892...  Training loss: 1.2762...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2893...  Training loss: 1.2790...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2894...  Training loss: 1.2474...  0.4792 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2895...  Training loss: 1.2326...  0.4791 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2896...  Training loss: 1.2833...  0.4791 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2897...  Training loss: 1.2644...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2898...  Training loss: 1.2328...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2899...  Training loss: 1.2803...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2900...  Training loss: 1.2887...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2901...  Training loss: 1.2689...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2902...  Training loss: 1.2452...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2903...  Training loss: 1.2294...  0.4794 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2904...  Training loss: 1.2662...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2905...  Training loss: 1.2985...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2906...  Training loss: 1.2822...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2907...  Training loss: 1.2875...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2908...  Training loss: 1.2837...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2909...  Training loss: 1.2986...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2910...  Training loss: 1.2912...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2911...  Training loss: 1.2859...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2912...  Training loss: 1.2814...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2913...  Training loss: 1.3291...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2914...  Training loss: 1.2877...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2915...  Training loss: 1.2760...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2916...  Training loss: 1.3152...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2917...  Training loss: 1.2667...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2918...  Training loss: 1.3043...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2919...  Training loss: 1.2931...  0.4781 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2920...  Training loss: 1.3138...  0.4777 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2921...  Training loss: 1.3064...  0.4776 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2922...  Training loss: 1.2672...  0.4797 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2923...  Training loss: 1.2524...  0.4795 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2924...  Training loss: 1.2552...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2925...  Training loss: 1.2854...  0.4792 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2926...  Training loss: 1.2712...  0.4792 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2927...  Training loss: 1.2709...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2928...  Training loss: 1.2795...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2929...  Training loss: 1.2762...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2930...  Training loss: 1.2614...  0.4790 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2931...  Training loss: 1.2488...  0.4782 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2932...  Training loss: 1.3070...  0.4783 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2933...  Training loss: 1.3032...  0.4793 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2934...  Training loss: 1.2806...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2935...  Training loss: 1.2786...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2936...  Training loss: 1.2828...  0.4780 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2937...  Training loss: 1.2776...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2938...  Training loss: 1.2877...  0.4791 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2939...  Training loss: 1.3044...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2940...  Training loss: 1.3423...  0.4780 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2941...  Training loss: 1.2938...  0.4793 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2942...  Training loss: 1.2759...  0.4783 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2943...  Training loss: 1.2638...  0.4779 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2944...  Training loss: 1.2824...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2945...  Training loss: 1.3098...  0.4791 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2946...  Training loss: 1.2775...  0.4773 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2947...  Training loss: 1.2926...  0.4783 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2948...  Training loss: 1.2470...  0.4791 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2949...  Training loss: 1.2701...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2950...  Training loss: 1.3226...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2951...  Training loss: 1.2555...  0.4794 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2952...  Training loss: 1.2410...  0.4792 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2953...  Training loss: 1.2718...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2954...  Training loss: 1.2806...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2955...  Training loss: 1.2786...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2956...  Training loss: 1.2677...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2957...  Training loss: 1.2741...  0.4781 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2958...  Training loss: 1.2530...  0.4794 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2959...  Training loss: 1.3061...  0.4779 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2960...  Training loss: 1.2754...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2961...  Training loss: 1.2727...  0.4788 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2962...  Training loss: 1.2633...  0.4786 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2963...  Training loss: 1.2463...  0.4789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2964...  Training loss: 1.2523...  0.4798 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2965...  Training loss: 1.2775...  0.4785 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2966...  Training loss: 1.2596...  0.4787 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2967...  Training loss: 1.2416...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2968...  Training loss: 1.2752...  0.4784 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2969...  Training loss: 1.2707...  0.4791 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2970...  Training loss: 1.2628...  0.4784 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2971...  Training loss: 1.4026...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2972...  Training loss: 1.2965...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2973...  Training loss: 1.2700...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2974...  Training loss: 1.2976...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2975...  Training loss: 1.2473...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2976...  Training loss: 1.2563...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2977...  Training loss: 1.2780...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2978...  Training loss: 1.2814...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2979...  Training loss: 1.2801...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2980...  Training loss: 1.2646...  0.4784 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2981...  Training loss: 1.2627...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2982...  Training loss: 1.2695...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2983...  Training loss: 1.2719...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2984...  Training loss: 1.2886...  0.4784 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2985...  Training loss: 1.2618...  0.4796 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2986...  Training loss: 1.2573...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2987...  Training loss: 1.2878...  0.4784 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2988...  Training loss: 1.2916...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2989...  Training loss: 1.2842...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2990...  Training loss: 1.3027...  0.4784 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2991...  Training loss: 1.2784...  0.4792 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2992...  Training loss: 1.2945...  0.4795 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2993...  Training loss: 1.2690...  0.4795 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2994...  Training loss: 1.2829...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2995...  Training loss: 1.2750...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2996...  Training loss: 1.2368...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2997...  Training loss: 1.2377...  0.4796 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2998...  Training loss: 1.2884...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 2999...  Training loss: 1.2864...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3000...  Training loss: 1.2871...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3001...  Training loss: 1.2507...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3002...  Training loss: 1.2509...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3003...  Training loss: 1.2779...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3004...  Training loss: 1.2878...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3005...  Training loss: 1.2623...  0.4792 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3006...  Training loss: 1.2825...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3007...  Training loss: 1.2427...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3008...  Training loss: 1.2370...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3009...  Training loss: 1.2321...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3010...  Training loss: 1.2620...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3011...  Training loss: 1.2441...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3012...  Training loss: 1.3089...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3013...  Training loss: 1.2586...  0.4781 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3014...  Training loss: 1.2412...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3015...  Training loss: 1.2701...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3016...  Training loss: 1.2503...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3017...  Training loss: 1.2563...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3018...  Training loss: 1.2584...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3019...  Training loss: 1.2733...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3020...  Training loss: 1.2815...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3021...  Training loss: 1.2479...  0.4792 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3022...  Training loss: 1.3086...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3023...  Training loss: 1.2755...  0.4784 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3024...  Training loss: 1.2831...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3025...  Training loss: 1.2629...  0.4795 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3026...  Training loss: 1.2696...  0.4792 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3027...  Training loss: 1.2898...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3028...  Training loss: 1.2640...  0.4793 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3029...  Training loss: 1.2572...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3030...  Training loss: 1.3076...  0.4798 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3031...  Training loss: 1.2791...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3032...  Training loss: 1.3192...  0.4799 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3033...  Training loss: 1.2865...  0.4780 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3034...  Training loss: 1.2777...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3035...  Training loss: 1.2685...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3036...  Training loss: 1.2808...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3037...  Training loss: 1.2920...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3038...  Training loss: 1.2548...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3039...  Training loss: 1.2776...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3040...  Training loss: 1.2640...  0.4794 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3041...  Training loss: 1.3152...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3042...  Training loss: 1.3016...  0.4784 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3043...  Training loss: 1.3150...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3044...  Training loss: 1.2578...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3045...  Training loss: 1.2745...  0.4798 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3046...  Training loss: 1.2834...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3047...  Training loss: 1.2640...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3048...  Training loss: 1.2654...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3049...  Training loss: 1.2289...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3050...  Training loss: 1.2739...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3051...  Training loss: 1.2395...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3052...  Training loss: 1.2760...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3053...  Training loss: 1.2456...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3054...  Training loss: 1.2687...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3055...  Training loss: 1.2485...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3056...  Training loss: 1.2660...  0.4781 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3057...  Training loss: 1.2465...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3058...  Training loss: 1.2521...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3059...  Training loss: 1.2415...  0.4799 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3060...  Training loss: 1.2775...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3061...  Training loss: 1.2490...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3062...  Training loss: 1.2606...  0.4779 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3063...  Training loss: 1.2352...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3064...  Training loss: 1.2456...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3065...  Training loss: 1.2411...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3066...  Training loss: 1.2805...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3067...  Training loss: 1.2721...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3068...  Training loss: 1.2333...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3069...  Training loss: 1.2491...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3070...  Training loss: 1.2504...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3071...  Training loss: 1.2701...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3072...  Training loss: 1.2606...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3073...  Training loss: 1.2688...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3074...  Training loss: 1.2534...  0.4797 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3075...  Training loss: 1.2546...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3076...  Training loss: 1.2553...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3077...  Training loss: 1.2676...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3078...  Training loss: 1.2762...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3079...  Training loss: 1.2596...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3080...  Training loss: 1.2861...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3081...  Training loss: 1.2480...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3082...  Training loss: 1.2758...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3083...  Training loss: 1.2682...  0.4776 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3084...  Training loss: 1.2543...  0.4784 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3085...  Training loss: 1.2426...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3086...  Training loss: 1.2348...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3087...  Training loss: 1.2855...  0.4794 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3088...  Training loss: 1.2750...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3089...  Training loss: 1.2657...  0.4782 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3090...  Training loss: 1.2681...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3091...  Training loss: 1.2676...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3092...  Training loss: 1.2281...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3093...  Training loss: 1.2164...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3094...  Training loss: 1.2662...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3095...  Training loss: 1.2547...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3096...  Training loss: 1.2197...  0.4781 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3097...  Training loss: 1.2702...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3098...  Training loss: 1.2656...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3099...  Training loss: 1.2433...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3100...  Training loss: 1.2230...  0.4796 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3101...  Training loss: 1.2141...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3102...  Training loss: 1.2487...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3103...  Training loss: 1.2819...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3104...  Training loss: 1.2739...  0.4780 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3105...  Training loss: 1.2647...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3106...  Training loss: 1.2661...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3107...  Training loss: 1.2895...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3108...  Training loss: 1.2730...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3109...  Training loss: 1.2660...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3110...  Training loss: 1.2712...  0.4780 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3111...  Training loss: 1.3225...  0.4797 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3112...  Training loss: 1.2860...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3113...  Training loss: 1.2670...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3114...  Training loss: 1.2949...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3115...  Training loss: 1.2556...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3116...  Training loss: 1.2820...  0.4780 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3117...  Training loss: 1.2794...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3118...  Training loss: 1.3014...  0.4782 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3119...  Training loss: 1.3030...  0.4799 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3120...  Training loss: 1.2644...  0.4793 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3121...  Training loss: 1.2355...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3122...  Training loss: 1.2468...  0.4792 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3123...  Training loss: 1.2786...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3124...  Training loss: 1.2562...  0.4793 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3125...  Training loss: 1.2554...  0.4798 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3126...  Training loss: 1.2641...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3127...  Training loss: 1.2605...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3128...  Training loss: 1.2594...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3129...  Training loss: 1.2346...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3130...  Training loss: 1.2914...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3131...  Training loss: 1.2950...  0.4782 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3132...  Training loss: 1.2707...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3133...  Training loss: 1.2700...  0.4796 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3134...  Training loss: 1.2625...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3135...  Training loss: 1.2632...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3136...  Training loss: 1.2688...  0.4780 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3137...  Training loss: 1.2892...  0.4779 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3138...  Training loss: 1.3247...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3139...  Training loss: 1.2794...  0.4788 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3140...  Training loss: 1.2720...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3141...  Training loss: 1.2606...  0.4800 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3142...  Training loss: 1.2508...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3143...  Training loss: 1.2997...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3144...  Training loss: 1.2697...  0.4794 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3145...  Training loss: 1.2629...  0.4792 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3146...  Training loss: 1.2398...  0.4787 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3147...  Training loss: 1.2498...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3148...  Training loss: 1.3026...  0.4785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3149...  Training loss: 1.2430...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3150...  Training loss: 1.2513...  0.4793 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3151...  Training loss: 1.2607...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3152...  Training loss: 1.2613...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3153...  Training loss: 1.2642...  0.4799 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3154...  Training loss: 1.2449...  0.4796 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3155...  Training loss: 1.2563...  0.4800 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3156...  Training loss: 1.2495...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3157...  Training loss: 1.2861...  0.4795 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3158...  Training loss: 1.2620...  0.4791 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3159...  Training loss: 1.2526...  0.4783 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3160...  Training loss: 1.2692...  0.4798 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3161...  Training loss: 1.2418...  0.4789 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3162...  Training loss: 1.2458...  0.4786 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3163...  Training loss: 1.2620...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3164...  Training loss: 1.2495...  0.4784 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3165...  Training loss: 1.2172...  0.4790 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3166...  Training loss: 1.2714...  0.4792 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3167...  Training loss: 1.2592...  0.4800 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3168...  Training loss: 1.2508...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3169...  Training loss: 1.3957...  0.4780 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3170...  Training loss: 1.2842...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3171...  Training loss: 1.2688...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3172...  Training loss: 1.2788...  0.4783 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3173...  Training loss: 1.2411...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3174...  Training loss: 1.2437...  0.4796 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3175...  Training loss: 1.2652...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3176...  Training loss: 1.2630...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3177...  Training loss: 1.2699...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3178...  Training loss: 1.2557...  0.4794 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3179...  Training loss: 1.2563...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3180...  Training loss: 1.2716...  0.4792 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3181...  Training loss: 1.2580...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3182...  Training loss: 1.2731...  0.4791 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3183...  Training loss: 1.2503...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3184...  Training loss: 1.2451...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3185...  Training loss: 1.2803...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3186...  Training loss: 1.2888...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3187...  Training loss: 1.2687...  0.4794 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3188...  Training loss: 1.2871...  0.4791 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3189...  Training loss: 1.2658...  0.4791 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3190...  Training loss: 1.2723...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3191...  Training loss: 1.2601...  0.4796 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3192...  Training loss: 1.2794...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3193...  Training loss: 1.2767...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3194...  Training loss: 1.2325...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3195...  Training loss: 1.2362...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3196...  Training loss: 1.2777...  0.4800 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3197...  Training loss: 1.2685...  0.4793 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3198...  Training loss: 1.2851...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3199...  Training loss: 1.2463...  0.4789 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3200...  Training loss: 1.2300...  0.4792 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3201...  Training loss: 1.2718...  0.4798 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3202...  Training loss: 1.2651...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3203...  Training loss: 1.2487...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3204...  Training loss: 1.2698...  0.4791 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3205...  Training loss: 1.2423...  0.4781 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3206...  Training loss: 1.2239...  0.4796 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3207...  Training loss: 1.2216...  0.4783 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3208...  Training loss: 1.2463...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3209...  Training loss: 1.2352...  0.4796 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3210...  Training loss: 1.2947...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3211...  Training loss: 1.2448...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3212...  Training loss: 1.2316...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3213...  Training loss: 1.2698...  0.4781 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3214...  Training loss: 1.2226...  0.4778 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3215...  Training loss: 1.2370...  0.4795 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3216...  Training loss: 1.2525...  0.4792 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3217...  Training loss: 1.2535...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3218...  Training loss: 1.2727...  0.4780 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3219...  Training loss: 1.2396...  0.4791 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3220...  Training loss: 1.2934...  0.4792 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3221...  Training loss: 1.2627...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3222...  Training loss: 1.2615...  0.4776 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3223...  Training loss: 1.2556...  0.4789 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3224...  Training loss: 1.2697...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3225...  Training loss: 1.2728...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3226...  Training loss: 1.2408...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3227...  Training loss: 1.2407...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3228...  Training loss: 1.2932...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3229...  Training loss: 1.2673...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3230...  Training loss: 1.3115...  0.4789 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3231...  Training loss: 1.2819...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3232...  Training loss: 1.2699...  0.4791 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3233...  Training loss: 1.2578...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3234...  Training loss: 1.2673...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3235...  Training loss: 1.2772...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3236...  Training loss: 1.2460...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3237...  Training loss: 1.2576...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3238...  Training loss: 1.2516...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3239...  Training loss: 1.2963...  0.4783 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3240...  Training loss: 1.2837...  0.4792 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3241...  Training loss: 1.2941...  0.4777 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3242...  Training loss: 1.2337...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3243...  Training loss: 1.2646...  0.4794 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3244...  Training loss: 1.2764...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3245...  Training loss: 1.2571...  0.4798 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3246...  Training loss: 1.2488...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3247...  Training loss: 1.2133...  0.4789 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3248...  Training loss: 1.2615...  0.4791 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3249...  Training loss: 1.2274...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3250...  Training loss: 1.2639...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3251...  Training loss: 1.2199...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3252...  Training loss: 1.2491...  0.4793 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3253...  Training loss: 1.2371...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3254...  Training loss: 1.2602...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3255...  Training loss: 1.2377...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3256...  Training loss: 1.2381...  0.4790 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3257...  Training loss: 1.2248...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3258...  Training loss: 1.2541...  0.4796 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3259...  Training loss: 1.2460...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3260...  Training loss: 1.2394...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3261...  Training loss: 1.2336...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3262...  Training loss: 1.2333...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3263...  Training loss: 1.2359...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3264...  Training loss: 1.2715...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3265...  Training loss: 1.2653...  0.4793 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3266...  Training loss: 1.2298...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3267...  Training loss: 1.2399...  0.4789 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3268...  Training loss: 1.2375...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3269...  Training loss: 1.2617...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3270...  Training loss: 1.2441...  0.4783 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3271...  Training loss: 1.2515...  0.4781 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3272...  Training loss: 1.2453...  0.4781 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3273...  Training loss: 1.2515...  0.4778 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3274...  Training loss: 1.2529...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3275...  Training loss: 1.2522...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3276...  Training loss: 1.2580...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3277...  Training loss: 1.2479...  0.4791 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3278...  Training loss: 1.2780...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3279...  Training loss: 1.2369...  0.4781 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3280...  Training loss: 1.2708...  0.4783 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3281...  Training loss: 1.2677...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3282...  Training loss: 1.2571...  0.4792 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3283...  Training loss: 1.2349...  0.4796 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3284...  Training loss: 1.2201...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3285...  Training loss: 1.2578...  0.4780 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3286...  Training loss: 1.2623...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3287...  Training loss: 1.2557...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3288...  Training loss: 1.2431...  0.4774 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3289...  Training loss: 1.2568...  0.4781 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3290...  Training loss: 1.2219...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3291...  Training loss: 1.2103...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3292...  Training loss: 1.2499...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3293...  Training loss: 1.2409...  0.4791 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3294...  Training loss: 1.2064...  0.4794 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3295...  Training loss: 1.2553...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3296...  Training loss: 1.2530...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3297...  Training loss: 1.2332...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3298...  Training loss: 1.2150...  0.4796 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3299...  Training loss: 1.2086...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3300...  Training loss: 1.2330...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3301...  Training loss: 1.2673...  0.4789 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3302...  Training loss: 1.2706...  0.4790 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3303...  Training loss: 1.2598...  0.4781 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3304...  Training loss: 1.2530...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3305...  Training loss: 1.2802...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3306...  Training loss: 1.2702...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3307...  Training loss: 1.2617...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3308...  Training loss: 1.2584...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3309...  Training loss: 1.2972...  0.4776 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3310...  Training loss: 1.2662...  0.4781 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3311...  Training loss: 1.2411...  0.4780 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3312...  Training loss: 1.2824...  0.4774 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3313...  Training loss: 1.2412...  0.4790 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3314...  Training loss: 1.2743...  0.4791 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3315...  Training loss: 1.2669...  0.4792 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3316...  Training loss: 1.2857...  0.4792 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3317...  Training loss: 1.2828...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3318...  Training loss: 1.2412...  0.4792 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3319...  Training loss: 1.2193...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3320...  Training loss: 1.2254...  0.4783 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3321...  Training loss: 1.2677...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3322...  Training loss: 1.2455...  0.4783 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3323...  Training loss: 1.2380...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3324...  Training loss: 1.2438...  0.4789 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3325...  Training loss: 1.2536...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3326...  Training loss: 1.2407...  0.4789 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3327...  Training loss: 1.2100...  0.4790 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3328...  Training loss: 1.2769...  0.4793 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3329...  Training loss: 1.2725...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3330...  Training loss: 1.2583...  0.4781 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3331...  Training loss: 1.2525...  0.4778 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3332...  Training loss: 1.2505...  0.4777 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3333...  Training loss: 1.2617...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3334...  Training loss: 1.2577...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3335...  Training loss: 1.2846...  0.4790 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3336...  Training loss: 1.3161...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3337...  Training loss: 1.2642...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3338...  Training loss: 1.2535...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3339...  Training loss: 1.2444...  0.4779 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3340...  Training loss: 1.2505...  0.4782 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3341...  Training loss: 1.2857...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3342...  Training loss: 1.2559...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3343...  Training loss: 1.2628...  0.4791 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3344...  Training loss: 1.2204...  0.4790 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3345...  Training loss: 1.2459...  0.4783 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3346...  Training loss: 1.2829...  0.4784 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3347...  Training loss: 1.2295...  0.4778 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3348...  Training loss: 1.2284...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3349...  Training loss: 1.2347...  0.4796 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3350...  Training loss: 1.2502...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3351...  Training loss: 1.2558...  0.4786 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3352...  Training loss: 1.2408...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3353...  Training loss: 1.2383...  0.4787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3354...  Training loss: 1.2343...  0.4775 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3355...  Training loss: 1.2777...  0.4789 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3356...  Training loss: 1.2562...  0.4794 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3357...  Training loss: 1.2406...  0.4794 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3358...  Training loss: 1.2525...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3359...  Training loss: 1.2267...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3360...  Training loss: 1.2291...  0.4779 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3361...  Training loss: 1.2486...  0.4780 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3362...  Training loss: 1.2371...  0.4785 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3363...  Training loss: 1.2143...  0.4783 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3364...  Training loss: 1.2532...  0.4788 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3365...  Training loss: 1.2399...  0.4792 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3366...  Training loss: 1.2364...  0.4782 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3367...  Training loss: 1.3862...  0.4775 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3368...  Training loss: 1.2722...  0.4790 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3369...  Training loss: 1.2612...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3370...  Training loss: 1.2809...  0.4779 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3371...  Training loss: 1.2284...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3372...  Training loss: 1.2200...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3373...  Training loss: 1.2529...  0.4782 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3374...  Training loss: 1.2574...  0.4790 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3375...  Training loss: 1.2551...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3376...  Training loss: 1.2412...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3377...  Training loss: 1.2387...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3378...  Training loss: 1.2570...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3379...  Training loss: 1.2561...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3380...  Training loss: 1.2594...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3381...  Training loss: 1.2340...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3382...  Training loss: 1.2288...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3383...  Training loss: 1.2627...  0.4788 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3384...  Training loss: 1.2750...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3385...  Training loss: 1.2517...  0.4790 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3386...  Training loss: 1.2756...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3387...  Training loss: 1.2555...  0.4789 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3388...  Training loss: 1.2702...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3389...  Training loss: 1.2517...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3390...  Training loss: 1.2710...  0.4779 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3391...  Training loss: 1.2551...  0.4793 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3392...  Training loss: 1.2062...  0.4791 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3393...  Training loss: 1.2200...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3394...  Training loss: 1.2679...  0.4794 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3395...  Training loss: 1.2699...  0.4793 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3396...  Training loss: 1.2687...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3397...  Training loss: 1.2303...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3398...  Training loss: 1.2319...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3399...  Training loss: 1.2489...  0.4788 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3400...  Training loss: 1.2583...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3401...  Training loss: 1.2345...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3402...  Training loss: 1.2666...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3403...  Training loss: 1.2261...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3404...  Training loss: 1.2169...  0.4788 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3405...  Training loss: 1.2029...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3406...  Training loss: 1.2310...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3407...  Training loss: 1.2219...  0.4793 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3408...  Training loss: 1.2846...  0.4778 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3409...  Training loss: 1.2354...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3410...  Training loss: 1.2223...  0.4790 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3411...  Training loss: 1.2554...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3412...  Training loss: 1.2250...  0.4790 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3413...  Training loss: 1.2416...  0.4788 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3414...  Training loss: 1.2420...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3415...  Training loss: 1.2517...  0.4782 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3416...  Training loss: 1.2584...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3417...  Training loss: 1.2196...  0.4792 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3418...  Training loss: 1.2916...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3419...  Training loss: 1.2478...  0.4794 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3420...  Training loss: 1.2496...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3421...  Training loss: 1.2347...  0.4795 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3422...  Training loss: 1.2434...  0.4789 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3423...  Training loss: 1.2550...  0.4778 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3424...  Training loss: 1.2314...  0.4788 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3425...  Training loss: 1.2237...  0.4788 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3426...  Training loss: 1.2831...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3427...  Training loss: 1.2568...  0.4795 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3428...  Training loss: 1.3072...  0.4804 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3429...  Training loss: 1.2733...  0.4793 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3430...  Training loss: 1.2560...  0.4791 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3431...  Training loss: 1.2401...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3432...  Training loss: 1.2585...  0.4795 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3433...  Training loss: 1.2637...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3434...  Training loss: 1.2429...  0.4790 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3435...  Training loss: 1.2577...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3436...  Training loss: 1.2466...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3437...  Training loss: 1.2835...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3438...  Training loss: 1.2675...  0.4789 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3439...  Training loss: 1.2774...  0.4797 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3440...  Training loss: 1.2332...  0.4792 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3441...  Training loss: 1.2497...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3442...  Training loss: 1.2640...  0.4794 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3443...  Training loss: 1.2447...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3444...  Training loss: 1.2273...  0.4791 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3445...  Training loss: 1.2053...  0.4790 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3446...  Training loss: 1.2468...  0.4799 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3447...  Training loss: 1.2212...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3448...  Training loss: 1.2409...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3449...  Training loss: 1.2242...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3450...  Training loss: 1.2457...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3451...  Training loss: 1.2149...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3452...  Training loss: 1.2477...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3453...  Training loss: 1.2262...  0.4776 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3454...  Training loss: 1.2291...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3455...  Training loss: 1.2250...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3456...  Training loss: 1.2467...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3457...  Training loss: 1.2264...  0.4791 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3458...  Training loss: 1.2369...  0.4788 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3459...  Training loss: 1.2111...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3460...  Training loss: 1.2248...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3461...  Training loss: 1.2178...  0.4782 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3462...  Training loss: 1.2626...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3463...  Training loss: 1.2596...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3464...  Training loss: 1.2107...  0.4789 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3465...  Training loss: 1.2225...  0.4792 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3466...  Training loss: 1.2221...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3467...  Training loss: 1.2515...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3468...  Training loss: 1.2268...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3469...  Training loss: 1.2415...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3470...  Training loss: 1.2290...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3471...  Training loss: 1.2425...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3472...  Training loss: 1.2424...  0.4778 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3473...  Training loss: 1.2500...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3474...  Training loss: 1.2552...  0.4796 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3475...  Training loss: 1.2258...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3476...  Training loss: 1.2582...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3477...  Training loss: 1.2301...  0.4794 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3478...  Training loss: 1.2539...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3479...  Training loss: 1.2431...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3480...  Training loss: 1.2384...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3481...  Training loss: 1.2179...  0.4791 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3482...  Training loss: 1.2144...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3483...  Training loss: 1.2423...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3484...  Training loss: 1.2483...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3485...  Training loss: 1.2393...  0.4797 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3486...  Training loss: 1.2367...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3487...  Training loss: 1.2380...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3488...  Training loss: 1.2047...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3489...  Training loss: 1.2035...  0.4791 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3490...  Training loss: 1.2360...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3491...  Training loss: 1.2368...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3492...  Training loss: 1.1977...  0.4782 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3493...  Training loss: 1.2476...  0.4796 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3494...  Training loss: 1.2444...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3495...  Training loss: 1.2186...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3496...  Training loss: 1.2051...  0.4794 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3497...  Training loss: 1.1942...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3498...  Training loss: 1.2206...  0.4798 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3499...  Training loss: 1.2605...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3500...  Training loss: 1.2555...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3501...  Training loss: 1.2535...  0.4791 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3502...  Training loss: 1.2409...  0.4798 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3503...  Training loss: 1.2639...  0.4782 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3504...  Training loss: 1.2684...  0.4782 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3505...  Training loss: 1.2462...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3506...  Training loss: 1.2503...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3507...  Training loss: 1.2844...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3508...  Training loss: 1.2590...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3509...  Training loss: 1.2443...  0.4796 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3510...  Training loss: 1.2805...  0.4772 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3511...  Training loss: 1.2152...  0.4782 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3512...  Training loss: 1.2705...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3513...  Training loss: 1.2549...  0.4779 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3514...  Training loss: 1.2684...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3515...  Training loss: 1.2735...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3516...  Training loss: 1.2408...  0.4771 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3517...  Training loss: 1.2170...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3518...  Training loss: 1.2214...  0.4783 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3519...  Training loss: 1.2559...  0.4802 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3520...  Training loss: 1.2340...  0.4789 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3521...  Training loss: 1.2308...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3522...  Training loss: 1.2423...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3523...  Training loss: 1.2476...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3524...  Training loss: 1.2250...  0.4789 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3525...  Training loss: 1.2124...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3526...  Training loss: 1.2698...  0.4789 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3527...  Training loss: 1.2618...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3528...  Training loss: 1.2577...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3529...  Training loss: 1.2321...  0.4770 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3530...  Training loss: 1.2454...  0.4795 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3531...  Training loss: 1.2445...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3532...  Training loss: 1.2479...  0.4788 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3533...  Training loss: 1.2645...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3534...  Training loss: 1.3001...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3535...  Training loss: 1.2527...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3536...  Training loss: 1.2420...  0.4779 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3537...  Training loss: 1.2309...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3538...  Training loss: 1.2408...  0.4790 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3539...  Training loss: 1.2662...  0.4797 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3540...  Training loss: 1.2423...  0.4795 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3541...  Training loss: 1.2527...  0.4787 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3542...  Training loss: 1.2139...  0.4791 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3543...  Training loss: 1.2350...  0.4789 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3544...  Training loss: 1.2804...  0.4796 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3545...  Training loss: 1.2178...  0.4789 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3546...  Training loss: 1.2210...  0.4788 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3547...  Training loss: 1.2449...  0.4793 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3548...  Training loss: 1.2494...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3549...  Training loss: 1.2403...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3550...  Training loss: 1.2315...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3551...  Training loss: 1.2318...  0.4784 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3552...  Training loss: 1.2248...  0.4785 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3553...  Training loss: 1.2693...  0.4798 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3554...  Training loss: 1.2469...  0.4802 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3555...  Training loss: 1.2375...  0.4794 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3556...  Training loss: 1.2401...  0.4781 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3557...  Training loss: 1.2202...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3558...  Training loss: 1.2197...  0.4791 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3559...  Training loss: 1.2382...  0.4795 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3560...  Training loss: 1.2302...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3561...  Training loss: 1.2023...  0.4780 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3562...  Training loss: 1.2388...  0.4788 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3563...  Training loss: 1.2345...  0.4786 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3564...  Training loss: 1.2277...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3565...  Training loss: 1.3694...  0.4778 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3566...  Training loss: 1.2584...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3567...  Training loss: 1.2423...  0.4780 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3568...  Training loss: 1.2645...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3569...  Training loss: 1.2269...  0.4778 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3570...  Training loss: 1.2123...  0.4790 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3571...  Training loss: 1.2397...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3572...  Training loss: 1.2436...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3573...  Training loss: 1.2433...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3574...  Training loss: 1.2170...  0.4777 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3575...  Training loss: 1.2186...  0.4789 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3576...  Training loss: 1.2400...  0.4781 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3577...  Training loss: 1.2329...  0.4777 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3578...  Training loss: 1.2435...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3579...  Training loss: 1.2232...  0.4795 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3580...  Training loss: 1.2172...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3581...  Training loss: 1.2559...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3582...  Training loss: 1.2556...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3583...  Training loss: 1.2370...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3584...  Training loss: 1.2621...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3585...  Training loss: 1.2396...  0.4797 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3586...  Training loss: 1.2568...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3587...  Training loss: 1.2301...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3588...  Training loss: 1.2528...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3589...  Training loss: 1.2420...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3590...  Training loss: 1.2003...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3591...  Training loss: 1.2065...  0.4779 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3592...  Training loss: 1.2575...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3593...  Training loss: 1.2540...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3594...  Training loss: 1.2542...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3595...  Training loss: 1.2209...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3596...  Training loss: 1.2145...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3597...  Training loss: 1.2386...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3598...  Training loss: 1.2406...  0.4796 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3599...  Training loss: 1.2196...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3600...  Training loss: 1.2488...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3601...  Training loss: 1.2218...  0.4795 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3602...  Training loss: 1.1986...  0.4789 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3603...  Training loss: 1.1992...  0.4779 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3604...  Training loss: 1.2206...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3605...  Training loss: 1.2104...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3606...  Training loss: 1.2818...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3607...  Training loss: 1.2269...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3608...  Training loss: 1.2088...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3609...  Training loss: 1.2374...  0.4789 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3610...  Training loss: 1.2083...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3611...  Training loss: 1.2199...  0.4799 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3612...  Training loss: 1.2219...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3613...  Training loss: 1.2326...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3614...  Training loss: 1.2509...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3615...  Training loss: 1.2064...  0.4800 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3616...  Training loss: 1.2766...  0.4789 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3617...  Training loss: 1.2431...  0.4789 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3618...  Training loss: 1.2478...  0.4789 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3619...  Training loss: 1.2269...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3620...  Training loss: 1.2382...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3621...  Training loss: 1.2431...  0.4780 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3622...  Training loss: 1.2275...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3623...  Training loss: 1.2224...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3624...  Training loss: 1.2615...  0.4793 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3625...  Training loss: 1.2456...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3626...  Training loss: 1.2831...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3627...  Training loss: 1.2597...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3628...  Training loss: 1.2495...  0.4794 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3629...  Training loss: 1.2286...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3630...  Training loss: 1.2458...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3631...  Training loss: 1.2571...  0.4794 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3632...  Training loss: 1.2207...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3633...  Training loss: 1.2466...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3634...  Training loss: 1.2248...  0.4781 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3635...  Training loss: 1.2781...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3636...  Training loss: 1.2575...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3637...  Training loss: 1.2633...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3638...  Training loss: 1.2222...  0.4792 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3639...  Training loss: 1.2377...  0.4796 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3640...  Training loss: 1.2552...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3641...  Training loss: 1.2349...  0.4778 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3642...  Training loss: 1.2240...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3643...  Training loss: 1.2002...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3644...  Training loss: 1.2346...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3645...  Training loss: 1.2032...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3646...  Training loss: 1.2444...  0.4778 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3647...  Training loss: 1.2056...  0.4791 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3648...  Training loss: 1.2331...  0.4781 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3649...  Training loss: 1.1978...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3650...  Training loss: 1.2377...  0.4779 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3651...  Training loss: 1.2155...  0.4792 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3652...  Training loss: 1.2086...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3653...  Training loss: 1.2120...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3654...  Training loss: 1.2371...  0.4791 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3655...  Training loss: 1.2148...  0.4790 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3656...  Training loss: 1.2217...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3657...  Training loss: 1.1953...  0.4780 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3658...  Training loss: 1.2081...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3659...  Training loss: 1.2121...  0.4793 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3660...  Training loss: 1.2429...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3661...  Training loss: 1.2384...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3662...  Training loss: 1.1999...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3663...  Training loss: 1.2141...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3664...  Training loss: 1.2165...  0.4796 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3665...  Training loss: 1.2341...  0.4779 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3666...  Training loss: 1.2195...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3667...  Training loss: 1.2347...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3668...  Training loss: 1.2180...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3669...  Training loss: 1.2326...  0.4779 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3670...  Training loss: 1.2283...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3671...  Training loss: 1.2372...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3672...  Training loss: 1.2388...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3673...  Training loss: 1.2177...  0.4779 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3674...  Training loss: 1.2399...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3675...  Training loss: 1.2197...  0.4780 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3676...  Training loss: 1.2411...  0.4776 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3677...  Training loss: 1.2243...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3678...  Training loss: 1.2290...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3679...  Training loss: 1.2037...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3680...  Training loss: 1.2054...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3681...  Training loss: 1.2313...  0.4790 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3682...  Training loss: 1.2427...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3683...  Training loss: 1.2331...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3684...  Training loss: 1.2346...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3685...  Training loss: 1.2330...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3686...  Training loss: 1.1975...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3687...  Training loss: 1.1908...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3688...  Training loss: 1.2398...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3689...  Training loss: 1.2228...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3690...  Training loss: 1.1838...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3691...  Training loss: 1.2498...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3692...  Training loss: 1.2356...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3693...  Training loss: 1.2133...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3694...  Training loss: 1.1826...  0.4796 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3695...  Training loss: 1.1807...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3696...  Training loss: 1.2274...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3697...  Training loss: 1.2490...  0.4795 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3698...  Training loss: 1.2335...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3699...  Training loss: 1.2396...  0.4773 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3700...  Training loss: 1.2383...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3701...  Training loss: 1.2490...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3702...  Training loss: 1.2503...  0.4789 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3703...  Training loss: 1.2375...  0.4797 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3704...  Training loss: 1.2317...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3705...  Training loss: 1.2742...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3706...  Training loss: 1.2313...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3707...  Training loss: 1.2321...  0.4791 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3708...  Training loss: 1.2598...  0.4791 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3709...  Training loss: 1.2138...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3710...  Training loss: 1.2515...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3711...  Training loss: 1.2414...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3712...  Training loss: 1.2675...  0.4791 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3713...  Training loss: 1.2578...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3714...  Training loss: 1.2345...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3715...  Training loss: 1.2055...  0.4781 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3716...  Training loss: 1.2052...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3717...  Training loss: 1.2363...  0.4781 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3718...  Training loss: 1.2254...  0.4793 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3719...  Training loss: 1.2168...  0.4789 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3720...  Training loss: 1.2277...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3721...  Training loss: 1.2336...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3722...  Training loss: 1.2163...  0.4791 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3723...  Training loss: 1.1941...  0.4802 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3724...  Training loss: 1.2439...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3725...  Training loss: 1.2501...  0.4792 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3726...  Training loss: 1.2375...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3727...  Training loss: 1.2284...  0.4789 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3728...  Training loss: 1.2289...  0.4793 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3729...  Training loss: 1.2307...  0.4797 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3730...  Training loss: 1.2311...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3731...  Training loss: 1.2518...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3732...  Training loss: 1.2820...  0.4791 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3733...  Training loss: 1.2447...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3734...  Training loss: 1.2437...  0.4791 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3735...  Training loss: 1.2226...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3736...  Training loss: 1.2241...  0.4792 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3737...  Training loss: 1.2680...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3738...  Training loss: 1.2335...  0.4785 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3739...  Training loss: 1.2451...  0.4778 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3740...  Training loss: 1.2094...  0.4780 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3741...  Training loss: 1.2399...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3742...  Training loss: 1.2708...  0.4789 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3743...  Training loss: 1.2144...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3744...  Training loss: 1.2084...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3745...  Training loss: 1.2338...  0.4786 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3746...  Training loss: 1.2314...  0.4782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3747...  Training loss: 1.2279...  0.4784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3748...  Training loss: 1.2212...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3749...  Training loss: 1.2369...  0.4792 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3750...  Training loss: 1.2083...  0.4790 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3751...  Training loss: 1.2571...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3752...  Training loss: 1.2337...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3753...  Training loss: 1.2269...  0.4779 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3754...  Training loss: 1.2298...  0.4789 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3755...  Training loss: 1.2202...  0.4790 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3756...  Training loss: 1.2113...  0.4788 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3757...  Training loss: 1.2424...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3758...  Training loss: 1.2209...  0.4790 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3759...  Training loss: 1.1957...  0.4783 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3760...  Training loss: 1.2379...  0.4775 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3761...  Training loss: 1.2330...  0.4787 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3762...  Training loss: 1.2252...  0.4772 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3763...  Training loss: 1.3704...  0.4767 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3764...  Training loss: 1.2466...  0.4770 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3765...  Training loss: 1.2288...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3766...  Training loss: 1.2488...  0.4780 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3767...  Training loss: 1.2077...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3768...  Training loss: 1.1979...  0.4783 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3769...  Training loss: 1.2187...  0.4792 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3770...  Training loss: 1.2282...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3771...  Training loss: 1.2331...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3772...  Training loss: 1.2148...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3773...  Training loss: 1.2187...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3774...  Training loss: 1.2246...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3775...  Training loss: 1.2251...  0.4790 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3776...  Training loss: 1.2413...  0.4786 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3777...  Training loss: 1.2163...  0.4797 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3778...  Training loss: 1.2029...  0.4791 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3779...  Training loss: 1.2427...  0.4786 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3780...  Training loss: 1.2479...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3781...  Training loss: 1.2266...  0.4789 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3782...  Training loss: 1.2579...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3783...  Training loss: 1.2228...  0.4798 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3784...  Training loss: 1.2433...  0.4792 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3785...  Training loss: 1.2189...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3786...  Training loss: 1.2449...  0.4791 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3787...  Training loss: 1.2246...  0.4786 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3788...  Training loss: 1.1837...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3789...  Training loss: 1.2042...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3790...  Training loss: 1.2441...  0.4792 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3791...  Training loss: 1.2365...  0.4790 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3792...  Training loss: 1.2539...  0.4783 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3793...  Training loss: 1.2044...  0.4789 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3794...  Training loss: 1.2015...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3795...  Training loss: 1.2324...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3796...  Training loss: 1.2436...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3797...  Training loss: 1.2146...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3798...  Training loss: 1.2414...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3799...  Training loss: 1.2104...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3800...  Training loss: 1.1952...  0.4790 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3801...  Training loss: 1.1964...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3802...  Training loss: 1.2134...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3803...  Training loss: 1.2029...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3804...  Training loss: 1.2697...  0.4789 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3805...  Training loss: 1.2161...  0.4774 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3806...  Training loss: 1.1974...  0.4775 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3807...  Training loss: 1.2295...  0.4789 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3808...  Training loss: 1.2100...  0.4783 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3809...  Training loss: 1.2190...  0.4786 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3810...  Training loss: 1.2195...  0.4779 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3811...  Training loss: 1.2264...  0.4798 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3812...  Training loss: 1.2349...  0.4789 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3813...  Training loss: 1.2041...  0.4778 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3814...  Training loss: 1.2677...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3815...  Training loss: 1.2323...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3816...  Training loss: 1.2475...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3817...  Training loss: 1.2232...  0.4778 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3818...  Training loss: 1.2205...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3819...  Training loss: 1.2310...  0.4780 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3820...  Training loss: 1.2151...  0.4792 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3821...  Training loss: 1.2085...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3822...  Training loss: 1.2657...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3823...  Training loss: 1.2376...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3824...  Training loss: 1.2741...  0.4777 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3825...  Training loss: 1.2452...  0.4786 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3826...  Training loss: 1.2419...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3827...  Training loss: 1.2282...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3828...  Training loss: 1.2398...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3829...  Training loss: 1.2488...  0.4783 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3830...  Training loss: 1.2181...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3831...  Training loss: 1.2349...  0.4792 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3832...  Training loss: 1.2207...  0.4796 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3833...  Training loss: 1.2669...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3834...  Training loss: 1.2498...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3835...  Training loss: 1.2618...  0.4792 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3836...  Training loss: 1.2177...  0.4790 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3837...  Training loss: 1.2281...  0.4780 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3838...  Training loss: 1.2531...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3839...  Training loss: 1.2385...  0.4789 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3840...  Training loss: 1.2188...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3841...  Training loss: 1.1869...  0.4774 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3842...  Training loss: 1.2321...  0.4803 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3843...  Training loss: 1.1982...  0.4794 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3844...  Training loss: 1.2229...  0.4780 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3845...  Training loss: 1.1968...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3846...  Training loss: 1.2180...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3847...  Training loss: 1.2023...  0.4791 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3848...  Training loss: 1.2300...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3849...  Training loss: 1.2020...  0.4778 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3850...  Training loss: 1.2178...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3851...  Training loss: 1.1992...  0.4804 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3852...  Training loss: 1.2293...  0.4791 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3853...  Training loss: 1.2034...  0.4780 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3854...  Training loss: 1.2170...  0.4790 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3855...  Training loss: 1.2009...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3856...  Training loss: 1.1956...  0.4780 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3857...  Training loss: 1.2063...  0.4799 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3858...  Training loss: 1.2289...  0.4794 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3859...  Training loss: 1.2352...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3860...  Training loss: 1.1950...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3861...  Training loss: 1.2006...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3862...  Training loss: 1.2056...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3863...  Training loss: 1.2268...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3864...  Training loss: 1.2134...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3865...  Training loss: 1.2176...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3866...  Training loss: 1.2105...  0.4778 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3867...  Training loss: 1.2274...  0.4790 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3868...  Training loss: 1.2180...  0.4780 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3869...  Training loss: 1.2255...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3870...  Training loss: 1.2278...  0.4780 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3871...  Training loss: 1.2143...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3872...  Training loss: 1.2411...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3873...  Training loss: 1.2117...  0.4790 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3874...  Training loss: 1.2281...  0.4789 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3875...  Training loss: 1.2277...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3876...  Training loss: 1.2201...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3877...  Training loss: 1.1999...  0.4790 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3878...  Training loss: 1.1964...  0.4780 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3879...  Training loss: 1.2210...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3880...  Training loss: 1.2325...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3881...  Training loss: 1.2091...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3882...  Training loss: 1.2189...  0.4791 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3883...  Training loss: 1.2263...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3884...  Training loss: 1.1889...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3885...  Training loss: 1.1815...  0.4796 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3886...  Training loss: 1.2218...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3887...  Training loss: 1.2019...  0.4774 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3888...  Training loss: 1.1827...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3889...  Training loss: 1.2249...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3890...  Training loss: 1.2227...  0.4779 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3891...  Training loss: 1.2044...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3892...  Training loss: 1.1750...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3893...  Training loss: 1.1734...  0.4783 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3894...  Training loss: 1.2073...  0.4783 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3895...  Training loss: 1.2383...  0.4783 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3896...  Training loss: 1.2204...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3897...  Training loss: 1.2254...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3898...  Training loss: 1.2284...  0.4786 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3899...  Training loss: 1.2456...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3900...  Training loss: 1.2348...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3901...  Training loss: 1.2269...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3902...  Training loss: 1.2301...  0.4786 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3903...  Training loss: 1.2648...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3904...  Training loss: 1.2372...  0.4793 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3905...  Training loss: 1.2204...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3906...  Training loss: 1.2528...  0.4788 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3907...  Training loss: 1.2043...  0.4795 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3908...  Training loss: 1.2369...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3909...  Training loss: 1.2243...  0.4783 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3910...  Training loss: 1.2552...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3911...  Training loss: 1.2448...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3912...  Training loss: 1.2215...  0.4778 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3913...  Training loss: 1.1916...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3914...  Training loss: 1.1956...  0.4783 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3915...  Training loss: 1.2335...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3916...  Training loss: 1.2178...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3917...  Training loss: 1.2225...  0.4795 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3918...  Training loss: 1.2223...  0.4786 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3919...  Training loss: 1.2243...  0.4779 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3920...  Training loss: 1.2078...  0.4789 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3921...  Training loss: 1.1830...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3922...  Training loss: 1.2489...  0.4779 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3923...  Training loss: 1.2456...  0.4786 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3924...  Training loss: 1.2306...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3925...  Training loss: 1.2185...  0.4798 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3926...  Training loss: 1.2280...  0.4791 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3927...  Training loss: 1.2161...  0.4794 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3928...  Training loss: 1.2168...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3929...  Training loss: 1.2491...  0.4796 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3930...  Training loss: 1.2793...  0.4777 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3931...  Training loss: 1.2338...  0.4778 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3932...  Training loss: 1.2312...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3933...  Training loss: 1.2183...  0.4779 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3934...  Training loss: 1.2173...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3935...  Training loss: 1.2619...  0.4776 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3936...  Training loss: 1.2301...  0.4777 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3937...  Training loss: 1.2305...  0.4786 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3938...  Training loss: 1.1977...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3939...  Training loss: 1.2258...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3940...  Training loss: 1.2624...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3941...  Training loss: 1.1941...  0.4793 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3942...  Training loss: 1.1913...  0.4776 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3943...  Training loss: 1.2164...  0.4782 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3944...  Training loss: 1.2209...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3945...  Training loss: 1.2196...  0.4791 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3946...  Training loss: 1.2053...  0.4795 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3947...  Training loss: 1.2193...  0.4779 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3948...  Training loss: 1.2035...  0.4783 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3949...  Training loss: 1.2537...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3950...  Training loss: 1.2232...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3951...  Training loss: 1.2079...  0.4779 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3952...  Training loss: 1.2162...  0.4792 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3953...  Training loss: 1.1901...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3954...  Training loss: 1.2140...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3955...  Training loss: 1.2235...  0.4787 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3956...  Training loss: 1.2083...  0.4785 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3957...  Training loss: 1.1933...  0.4783 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3958...  Training loss: 1.2244...  0.4781 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3959...  Training loss: 1.2128...  0.4784 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3960...  Training loss: 1.2065...  0.4785 sec/batch\n"
     ]
    }
   ],
   "source": [
    "epochs = 20\n",
    "# Save every N iterations\n",
    "save_every_n = 200\n",
    "\n",
    "model = CharRNN(len(vocab), batch_size=batch_size, num_steps=num_steps,\n",
    "                lstm_size=lstm_size, num_layers=num_layers, \n",
    "                learning_rate=learning_rate)\n",
    "\n",
    "saver = tf.train.Saver(max_to_keep=100)\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # Use the line below to load a checkpoint and resume training\n",
    "    #saver.restore(sess, 'checkpoints/______.ckpt')\n",
    "    counter = 0\n",
    "    for e in range(epochs):\n",
    "        # Train network\n",
    "        new_state = sess.run(model.initial_state)\n",
    "        loss = 0\n",
    "        for x, y in get_batches(encoded, batch_size, num_steps):\n",
    "            counter += 1\n",
    "            start = time.time()\n",
    "            feed = {model.inputs: x,\n",
    "                    model.targets: y,\n",
    "                    model.keep_prob: keep_prob,\n",
    "                    model.initial_state: new_state}\n",
    "            batch_loss, new_state, _ = sess.run([model.loss, \n",
    "                                                 model.final_state, \n",
    "                                                 model.optimizer], \n",
    "                                                 feed_dict=feed)\n",
    "            \n",
    "            end = time.time()\n",
    "            print('Epoch: {}/{}... '.format(e+1, epochs),\n",
    "                  'Training Step: {}... '.format(counter),\n",
    "                  'Training loss: {:.4f}... '.format(batch_loss),\n",
    "                  '{:.4f} sec/batch'.format((end-start)))\n",
    "        \n",
    "            if (counter % save_every_n == 0):\n",
    "                saver.save(sess, \"checkpoints/i{}_l{}.ckpt\".format(counter, lstm_size))\n",
    "    \n",
    "    saver.save(sess, \"checkpoints/i{}_l{}.ckpt\".format(counter, lstm_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Saved checkpoints\n",
    "\n",
    "Read up on saving and loading checkpoints here: https://www.tensorflow.org/programmers_guide/variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "model_checkpoint_path: \"checkpoints/i3960_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i200_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i400_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i600_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i800_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i1000_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i1200_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i1400_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i1600_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i1800_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i2000_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i2200_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i2400_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i2600_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i2800_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i3000_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i3200_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i3400_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i3600_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i3800_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i3960_l512.ckpt\""
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.train.get_checkpoint_state('checkpoints')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sampling\n",
    "\n",
    "Now that the network is trained, we'll can use it to generate new text. The idea is that we pass in a character, then the network will predict the next character. We can use the new one, to predict the next one. And we keep doing this to generate all new text. I also included some functionality to prime the network with some text by passing in a string and building up a state from that.\n",
    "\n",
    "The network gives us predictions for each character. To reduce noise and make things a little less random, I'm going to only choose a new character from the top N most likely characters.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pick_top_n(preds, vocab_size, top_n=5):\n",
    "    p = np.squeeze(preds)\n",
    "    p[np.argsort(p)[:-top_n]] = 0\n",
    "    p = p / np.sum(p)\n",
    "    c = np.random.choice(vocab_size, 1, p=p)[0]\n",
    "    return c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sample(checkpoint, n_samples, lstm_size, vocab_size, prime=\"The \"):\n",
    "    samples = [c for c in prime]\n",
    "    model = CharRNN(len(vocab), lstm_size=lstm_size, sampling=True)\n",
    "    saver = tf.train.Saver()\n",
    "    with tf.Session() as sess:\n",
    "        saver.restore(sess, checkpoint)\n",
    "        new_state = sess.run(model.initial_state)\n",
    "        for c in prime:\n",
    "            x = np.zeros((1, 1))\n",
    "            x[0,0] = vocab_to_int[c]\n",
    "            feed = {model.inputs: x,\n",
    "                    model.keep_prob: 1.,\n",
    "                    model.initial_state: new_state}\n",
    "            preds, new_state = sess.run([model.prediction, model.final_state], \n",
    "                                         feed_dict=feed)\n",
    "\n",
    "        c = pick_top_n(preds, len(vocab))\n",
    "        samples.append(int_to_vocab[c])\n",
    "\n",
    "        for i in range(n_samples):\n",
    "            x[0,0] = c\n",
    "            feed = {model.inputs: x,\n",
    "                    model.keep_prob: 1.,\n",
    "                    model.initial_state: new_state}\n",
    "            preds, new_state = sess.run([model.prediction, model.final_state], \n",
    "                                         feed_dict=feed)\n",
    "\n",
    "            c = pick_top_n(preds, len(vocab))\n",
    "            samples.append(int_to_vocab[c])\n",
    "        \n",
    "    return ''.join(samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, pass in the path to a checkpoint and sample from the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'checkpoints/i3960_l512.ckpt'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.train.latest_checkpoint('checkpoints')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Farred. Stepan Arkadyevitch saw that their husband, and he hurt\n",
      "hever, the condition-ought to him by her, and his health, shaking her\n",
      "from the cleamoust. He stouded was signing on the tale, and the sillid\n",
      "chief conscirunces an ang hands,.\n",
      "\n",
      "\"Was it work to me?\" he repeased. She did not happy she strained that\n",
      "he was to be so as to concerning her wife and the sensiless and\n",
      "service, heard a bircal. The serroce admits the prince saw that were when\n",
      "he came out on his leastress and a cattle, and that the one had been\n",
      "any secine and happy, he was in this soul.\n",
      "\n",
      "The ceaping peasants walked intonerizuleately so far his hand,\n",
      "that some money treated him, as so, and that this she made to holt\n",
      "that some thoughant and prove that he had been allowed home. He\n",
      "suched by that steas of all this, as though he was not astance to around\n",
      "her that she was telling her had been to hinde him. She was inquiringly\n",
      "about the toperous, which she did not wear her. He could not be spiritsel,\n",
      "but and who had altaked all a sitting of that steas for himself, which the\n",
      "room, with the book, at the contimueatine. And they, all the sacred\n",
      "came again with him, and he was a carviage, and starting which had\n",
      "now he classed to the station.\n",
      "\n",
      "\"And that I'd vocknage in married,, then I'm a sport of cappition,\" she went\n",
      "on to this steps, with his mother, and stood shipting his brotsee, wion\n",
      "wite the table, hopeing his starg of tatting.\n",
      "\n",
      "The stepar and were\n",
      "felt than enceressed feits for the children the matters.\n",
      "\n",
      "When he was saking her flowers, and which had beloesed that why some\n",
      "special small banker in a share and points and the corner of the\n",
      "shamesh of the driv roan, he he scuped her and hushal.\n",
      "\n",
      "\"You must come for you anotherer.\" He flung her true again.\n",
      "\n",
      "\"I see helling more and drinking up it in their life in the words and had been\n",
      "to complite this attempts about your tried.t\" stoo ally the solar and\n",
      "way of her striting. \"If it was to miserult for...\n",
      " \"We are not going, I must be able anything about to the minut\n"
     ]
    }
   ],
   "source": [
    "checkpoint = tf.train.latest_checkpoint('checkpoints')\n",
    "samp = sample(checkpoint, 2000, lstm_size, len(vocab), prime=\"Far\")\n",
    "print(samp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Farg,\n",
      ".\n",
      "\n",
      "Hha stise hite\n",
      "heed thos he pand tath se seind, bhete af ar os he ale hing ton too she tim ood the tontin hor this hhe birtes on he he sintersesar hise\n",
      "sand tas ante\n",
      "ang ins oto nes th oth ad ars lisig\n",
      "ond the whe wh thant tou hat at tim he hesrit, the ho bel ole ote ant ohe hin an tes oti sos heas an ot ar on tithe tith wathe he an tisisilt\n",
      "os hat his as tores and ood male srand sathar.\n",
      "\"\n",
      "\"\n",
      "he wor wothe\n",
      ", hhe hend hee thins and or her ho he ad thir sis hithe sare se are to ha ta sot the sarete ast this whas win lhe and che meed the ther wot her ser ereret sale wean an te hams al tas ha tinses sos te the sand at iod antis to he an who ile tor otherind hiter or shit an he whor ond ot se ante he tarsetithe\n",
      "seas and wral hhr wor so th tis ther th mang bot ho wher the terersinto he chel sein te ter thr sat ta his whe se al oo the win she coos hon won the thee serere ang. aud thererese helt her he set the wet ant tiog as anse ot te thes at whe cherim sim oe site. hit int ot he to him\n",
      "a\n"
     ]
    }
   ],
   "source": [
    "checkpoint = 'checkpoints/i200_l512.ckpt'\n",
    "samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime=\"Far\")\n",
    "print(samp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Farcy to life at how her as he headdes\n",
      "the thomer had to hord ap sure to tree\n",
      "tome thould all the seaness that it he cralttens at hes hed been som and neal ther elang, she\n",
      "the pored to the moteres\n",
      "for his\n",
      "faricion, and the pesprings of the crilling to mant of\n",
      "the puserss and she suppithers.\n",
      "\n",
      "\"It sout of the mone sone that hirs and tone it her.\"\n",
      "\"This alough to\n",
      "at that heast she samigalion it that he dad to seem his\n",
      "to all wo she to hore and tanthare on a he pance and so sprepantelly of theme sume her shanges that her salpeed ot whith she said she a shougn her, at the sering of still. And thatene so chestod on the preening.\"\n",
      "\n",
      "\"Wam as you thask in seesill, as in that the colden of and a campinest the sreest wher the mettrepening and and the spaters the men shamang the tould, she heas not were theme that whith the parices itten tur the peass anding her bested tor hen to soe take the more. The pirss and and\n",
      "he war of the sappans the\n",
      "momat, her, sut thes the some his wirestly with a\n",
      "dromn the h\n"
     ]
    }
   ],
   "source": [
    "checkpoint = 'checkpoints/i600_l512.ckpt'\n",
    "samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime=\"Far\")\n",
    "print(samp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Farizh and take him the tright, but the pare what he had not been she said an inveroading,\n",
      "the pitson, she had notes to her. He was she had getered to mother to her hand.\n",
      "\"And whe child you\n",
      "know I should be it tham,\" she hand, and the honding worrs on the\n",
      "head, who take all the will was through what he with a latyer saye of child\n",
      "have the same of her seeming\n",
      "saying her thought as though the same time it was in preterver at her suct altere the see he could think to\n",
      "gree mary for a cannecs on the mene of who sen how have teen having her, as the sure and taking her her.\" \"I did' lletge tow.\"\n",
      "\n",
      "\"An that?\" said Levin, should he had\n",
      "feeling to the wird wind was to showt her would. He would have leenged to their\n",
      "condictynon waired thought, his\n",
      "back to\n",
      "be his eare to the morter, and highert. But the she was ask it to trisken of in the stronging hir bights.\n",
      "\n",
      "\"At the persaim is to her. He was took the stalt.\" And a she wanted it would ask and the some sain to her soft a penity of trea harts the most \n"
     ]
    }
   ],
   "source": [
    "checkpoint = 'checkpoints/i1200_l512.ckpt'\n",
    "samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime=\"Far\")\n",
    "print(samp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}