4_tf.keras_model_Sequential.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第4天：TensorFlow2建立模型的三种方式之Sequential API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensorflow版本： 2.1.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "print('tensorflow版本：', tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**tf.keras**提供**[Sequential API](https://tensorflow.google.cn/versions/r2.2/api_docs/python/tf/keras/Sequential)**(Sequential按层顺序创建模型 )，这种方式比较简单。下面对手写数字数据集mnist建立基于MLP(多层感知机)的分类模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1、数据准备"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 引入数据集，该数据集的获取方式已经集成，直接引入后，会自动下载数据集\n",
    "from tensorflow.keras.datasets import mnist\n",
    "mnistdata = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "mnist数据集分为两部分，前一部分是训练数据集，数据条数60000条，数据包括训练特征、训练标签；后一部分是测试数据集，数据条数10000条，数据包括测试特征、测试标签。其中每一条特征数据是一个维度为(28, 28)的矩阵，代表一张黑白的图片；标签数据就是该图片中的数字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练数据集\n",
      "特征：类型<class 'numpy.ndarray'>, 维度(60000, 28, 28)\n",
      "标签：类型<class 'numpy.ndarray'>, 维度(60000,)\n",
      "测试数据集\n",
      "特征：类型<class 'numpy.ndarray'>, 维度(10000, 28, 28)\n",
      "标签：类型<class 'numpy.ndarray'>, 维度(10000,)\n"
     ]
    }
   ],
   "source": [
    "(train_features, train_labels), (test_features, test_labels) = mnistdata\n",
    "print('训练数据集')\n",
    "print('特征：类型{}, 维度{}'.format(type(train_features), train_features.shape))\n",
    "print('标签：类型{}, 维度{}'.format(type(train_labels), train_labels.shape))\n",
    "print('测试数据集')\n",
    "print('特征：类型{}, 维度{}'.format(type(test_features), test_features.shape))\n",
    "print('标签：类型{}, 维度{}'.format(type(test_labels), test_labels.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "数据维度： (28, 28)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAEICAYAAACZA4KlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAPq0lEQVR4nO3dfaxUdX7H8fdHy2Kr4PpQlVUUaqnd7pKw2xvU1LQ0FMUHoljdLKsJZm0v6Wqrja1VNFkTTB+23TUaEzcQZbFdcdeuj9SuS0nxITarKFZA6moRleUWNLhB0yYr8O0fc3Cv15kzl5kzD9zv55XczMz5zjnne4f74ZyZc878FBGY2dh3SK8bMLPucNjNknDYzZJw2M2ScNjNknDYzZJw2McYSVsl/cEonheSfr3FdbQ8r/WOw26VkvRpSSsk7Sx+bul1T1bzS71uwMac24BfAaYAxwFrJL0ZEct72pV5yz5WSZop6T8k/UzSkKQ7JX1qxNPOk7RF0ruS/l7SIcPm/6qkzZLek/SEpFNGuep5wDci4n8jYitwN/DVan4ra4fDPnbtBf4cOBY4E5gNfG3Ec+YDA8AXgQspQinpImAxcDHwq8DTwMp6K5H0FUkvj5w84v7n2/lFrBryufFji6StwB9FxL+NmH4t8HsRMb94HMC5EfHD4vHXgD+MiNmS/hX454i4u6gdAnwAfDYi3izmnRYRr9dZ/z9R241fCBwPPAGcFBHjO/Mb22h5yz5GSfoNSask/Y+k3cBfU9vKD/f2sPtvAp8p7p8C3F68BfgZsIvaFvrEUaz6z4D/A14DHqG2R7Ct9d/EquKwj113Af9FbQs8kdpuuUY8Z/Kw+ycD24v7bwOLIuLTw35+OSKebbbSiNgVEZdFxAkR8Tlqf2PPtf3bWNsc9rFrArAb+EDSbwJ/Uuc5fynpKEmTgWuA7xXTvw3cKOlzAJKOlHTpaFYq6VRJx0g6VNK5wCBwa7u/jLXPYR+7/gL4CvA+sIxfBHm4R4AXgJeAf6H2yTkR8RDwd8D9xVuAjcC59VYi6TJJm4ZN+m1gQ7HevwEui4hN9ea17vIHdGZJeMtuloTDbpaEw26WhMNulkRXL4Qpzrwysw6KiJHnUwBtbtklzZX0qqTXJd3QzrLMrLNaPvQm6VDgJ8AcaqdDPg8siIhXSubxlt2swzqxZZ8JvB4RWyLi58D91K6cMrM+1E7YT+TjF1Jso86FEpIGJa2TtK6NdZlZm9r5gK7ersIndtMjYimwFLwbb9ZL7WzZt/Hxq6ZO4hdXTZlZn2kn7M8D0yRNLb7u6MvAo9W0ZWZVa3k3PiL2SLqa2jeRHArc46ubzPpXV69683t2s87ryEk1ZnbwcNjNknDYzZJw2M2ScNjNknDYzZJw2M2ScNjNknDYzZJw2M2ScNjNknDYzZJw2M2ScNjNknDYzZJw2M2ScNjNknDYzZJw2M2ScNjNknDYzZJw2M2ScNjNknDYzZJw2M2ScNjNknDYzZJw2M2ScNjNknDYzZJoeXx2AElbgfeBvcCeiBiooikzq15bYS/8fkS8W8FyzKyDvBtvlkS7YQ/gR5JekDRY7wmSBiWtk7SuzXWZWRsUEa3PLH0mIrZLOg5YDfxpRDxV8vzWV2ZmoxIRqje9rS17RGwvbncCDwEz21memXVOy2GXdLikCfvvA2cDG6tqzMyq1c6n8ccDD0nav5z7IuKHlXRlZpVr6z37Aa/M79nNOq4j79nN7ODhsJsl4bCbJeGwmyXhsJslUcWFMHYQmzVrVmn9xhtvLK3PmTOn5XVv2rSptH722WeX1oeGhlped0bespsl4bCbJeGwmyXhsJsl4bCbJeGwmyXhsJsl4avexoAjjjiiYW3RokWl8956662l9fHjx5fWO/n3s2HDhtL6BRdcUFrftm1ble0cNHzVm1lyDrtZEg67WRIOu1kSDrtZEg67WRIOu1kSvp79IHDkkUeW1u+7776Gtblz51bdTtdMnz69tD5t2rTSetbj7I14y26WhMNuloTDbpaEw26WhMNuloTDbpaEw26WhK9n7wMLFiworS9ZsqS0PnXq1Ia1PXv2lM67atWq0vpNN91UWm/297N48eKGtcsvv7x03mYef/zx0vq8efPaWv7BquXr2SXdI2mnpI3Dph0tabWk14rbo6ps1syqN5rd+O8AI0/DugFYExHTgDXFYzPrY03DHhFPAbtGTL4QWFHcXwFcVHFfZlaxVs+NPz4ihgAiYkjScY2eKGkQGGxxPWZWkY5fCBMRS4Gl4A/ozHqp1UNvOyRNAihud1bXkpl1QqthfxRYWNxfCDxSTTtm1ilNj7NLWgnMAo4FdgBfBx4Gvg+cDLwFXBoRIz/Eq7eslLvxy5cvL63Pnz+/tD5hwoSW171+/frS+sDAQGl9ypQppfVmx8pvvvnmhrVx48aVztvM9u3bS+vnnHNOw9orr7zS1rr7WaPj7E3fs0dEozM+ZrfVkZl1lU+XNUvCYTdLwmE3S8JhN0vCYTdLwl8lXYFml6h28tBaM88880xpffXq1aX1sstnR1PvpGZfFT2WD6+1wlt2syQcdrMkHHazJBx2syQcdrMkHHazJBx2syT8VdIVaHY897TTTutSJ9WT6l4t+ZFXX321tP7ss882rF1xxRWttPSRa665prR+5513trX8g1XLXyVtZmODw26WhMNuloTDbpaEw26WhMNuloTDbpaEr2evwAMPPFBaL/s65U5rdh5Fs69jbtb7ypUrS+sPPvhgab0dmzZt6tiyxyJv2c2ScNjNknDYzZJw2M2ScNjNknDYzZJw2M2S8HH2CqxYsaK0fvHFF5fWJ0+e3Nb6d+/e3bC2ZMmS0nmXLVvW1rqbmTFjRsvz7t27t7S+b9++lpedUdMtu6R7JO2UtHHYtFsk/VTSS8XPeZ1t08zaNZrd+O8Ac+tMvy0iZhQ/j1fblplVrWnYI+IpYFcXejGzDmrnA7qrJb1c7OYf1ehJkgYlrZO0ro11mVmbWg37XcCpwAxgCPhmoydGxNKIGIiIgRbXZWYVaCnsEbEjIvZGxD5gGTCz2rbMrGothV3SpGEP5wMbGz3XzPpD0+PsklYCs4BjJW0Dvg7MkjQDCGArsKiDPfa9LVu2lNanT5/epU6674wzziitT5w4seVlr127trT+5JNPtrzsjJqGPSIW1Jl8dwd6MbMO8umyZkk47GZJOOxmSTjsZkk47GZJ+BJXa8t1111XWp8wYULLy168eHHL89onectuloTDbpaEw26WhMNuloTDbpaEw26WhMNuloSPs1upgYHyLxg6//zzS+tlQ0Y/9thjpfOuX7++tG4Hxlt2syQcdrMkHHazJBx2syQcdrMkHHazJBx2syR8nN1KXX/99aX18ePHt7zsdevKRwRrNmSzHRhv2c2ScNjNknDYzZJw2M2ScNjNknDYzZJw2M2SGM2QzZOBe4ETgH3A0oi4XdLRwPeAKdSGbf5SRLzXuVatFePGjSut33HHHaX1Sy65pLRedr06wFtvvdWwtnz58tJ5rVqj2bLvAa6LiM8CZwBXSfot4AZgTURMA9YUj82sTzUNe0QMRcSLxf33gc3AicCFwIriaSuAizrVpJm174Des0uaAnwB+DFwfEQMQe0/BOC4qpszs+qM+tx4SUcAPwCujYjdkkY73yAw2Fp7ZlaVUW3ZJY2jFvTvRsSDxeQdkiYV9UnAznrzRsTSiBiIiPJvLjSzjmoadtU24XcDmyPiW8NKjwILi/sLgUeqb8/MqqJmh04knQU8DWygdugNYDG19+3fB04G3gIujYhdTZZVvjKr3MSJE0vr771XfrS02du1Zn8/Z555ZsPac889VzqvtSYi6v6jNX3PHhHPAI3+xWe305SZdY/PoDNLwmE3S8JhN0vCYTdLwmE3S8JhN0vCXyU9xl155ZUdXf6OHTtK62+88UZH12+j5y27WRIOu1kSDrtZEg67WRIOu1kSDrtZEg67WRI+zj4GHHPMMQ1rV111VVvLbnY9++Bg+TeOvfPOO22t36rjLbtZEg67WRIOu1kSDrtZEg67WRIOu1kSDrtZEj7OPgYcdthhDWtTp07t6LpPP/300vqqVas6un4bPW/ZzZJw2M2ScNjNknDYzZJw2M2ScNjNknDYzZJoepxd0mTgXuAEauOzL42I2yXdAvwxsP+C5cUR8XinGrXGysZI//DDD0vnHTduXGn9iSeeKK0vWbKktG79YzQn1ewBrouIFyVNAF6QtLqo3RYR/9C59sysKk3DHhFDwFBx/31Jm4ETO92YmVXrgN6zS5oCfAH4cTHpakkvS7pH0lEN5hmUtE7SurY6NbO2jDrsko4AfgBcGxG7gbuAU4EZ1Lb836w3X0QsjYiBiBiooF8za9Gowi5pHLWgfzciHgSIiB0RsTci9gHLgJmda9PM2tU07Kp9vejdwOaI+Naw6ZOGPW0+sLH69sysKio7bAMg6SzgaWADtUNvAIuBBdR24QPYCiwqPswrW1b5yqxy8+bNK60//PDDpfXZs2eX1teuXXugLVmHRUTd7/8ezafxzwD1ZvYxdbODiM+gM0vCYTdLwmE3S8JhN0vCYTdLwmE3S6LpcfZKV+bj7GYd1+g4u7fsZkk47GZJOOxmSTjsZkk47GZJOOxmSTjsZkl0e8jmd4E3hz0+tpjWj/q1t37tC9xbq6rs7ZRGha6eVPOJlUvr+vW76fq1t37tC9xbq7rVm3fjzZJw2M2S6HXYl/Z4/WX6tbd+7QvcW6u60ltP37ObWff0estuZl3isJsl0ZOwS5or6VVJr0u6oRc9NCJpq6QNkl7q9fh0xRh6OyVtHDbtaEmrJb1W3NYdY69Hvd0i6afFa/eSpPN61NtkSf8uabOkTZKuKab39LUr6asrr1vX37NLOhT4CTAH2AY8DyyIiFe62kgDkrYCAxHR8xMwJP0u8AFwb0R8vpj2DWBXRPxt8R/lURHxV33S2y3AB70exrsYrWjS8GHGgYuAK+jha1fS15fowuvWiy37TOD1iNgSET8H7gcu7EEffS8ingJ2jZh8IbCiuL+C2h9L1zXorS9ExFBEvFjcfx/YP8x4T1+7kr66ohdhPxF4e9jjbfTXeO8B/EjSC5IGe91MHcfvH2aruD2ux/2M1HQY724aMcx437x2rQx/3q5ehL3e92P10/G/34mILwLnAlcVu6s2OqMaxrtb6gwz3hdaHf68Xb0I+zZg8rDHJwHbe9BHXRGxvbjdCTxE/w1FvWP/CLrF7c4e9/ORfhrGu94w4/TBa9fL4c97EfbngWmSpkr6FPBl4NEe9PEJkg4vPjhB0uHA2fTfUNSPAguL+wuBR3rYy8f0yzDejYYZp8evXc+HP4+Irv8A51H7RP6/gZt60UODvn4N+M/iZ1OvewNWUtut+5DaHtGVwDHAGuC14vboPurtH6kN7f0ytWBN6lFvZ1F7a/gy8FLxc16vX7uSvrryuvl0WbMkfAadWRIOu1kSDrtZEg67WRIOu1kSDrtZEg67WRL/D6Oy2mqESaLPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 需要安装matplotlib：打开Anaconda Prompt，激活环境(activate tf2)，安装matplotlib(conda install matplotlib)\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# 显示index为n的数据\n",
    "def showfig(features, label, n):\n",
    "    plt.title('label:{}'.format(label[n]))\n",
    "    figdata = features[n]\n",
    "    print('数据维度：', figdata.shape)\n",
    "    plt.imshow(figdata, 'gray')  # 因为原始数据是(28, 28)的，只有一个通道，所以是黑白色的\n",
    "\n",
    "showfig(train_features, train_labels, 1122)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "每个数字以及对应的图片个数：\n",
      "[(0, 5923), (1, 6742), (2, 5958), (3, 6131), (4, 5842), (5, 5421), (6, 5918), (7, 6265), (8, 5851), (9, 5949)]\n"
     ]
    }
   ],
   "source": [
    "# 看下训练数据中的标签的分布\n",
    "from collections import Counter\n",
    "print('每个数字以及对应的图片个数：\\n{}'.format(sorted(Counter(train_labels).items(), key=lambda s:s[0])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为是构建MLP模型，需要对数据进行归一化。因为图片的数字矩阵中的数字类型是无符号8位整数 uint8最大值为255，因此归一化需要除以255，也就是将数值变为[0, 1]范围内的数字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 归一化\n",
    "trainfeatures = train_features / 255\n",
    "testfeatures = test_features / 255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2、 Sequential按层顺序创建模型 \n",
    "\n",
    "下面给出MLP模型的一些参数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "Epoch = 100  # 模型迭代的次数\n",
    "Batch_Size = 128  # 批量训练的样本的个数\n",
    "Out_Class = 10  # 输出的类别的个数,0-9共10类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.  构建模型\n",
    "\n",
    "利用Sequential构建模型，就是像搭积木一样，按照层顺序建立模型。对于MLP模型而言，第一层就是输入层，接下来一个或者多个隐层，最后是输出层。实现方式可以通过下面的 **.add** 实现，也可以通过列表形式实现。利用**tf.keras.models.Sequential.Dense()** 构建全连接层。\n",
    "\n",
    "**tf.keras.layers.Dense(\n",
    "    units, activation=None, use_bias=True, kernel_initializer='glorot_uniform',\n",
    "    bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None,\n",
    "    activity_regularizer=None, kernel_constraint=None, bias_constraint=None,\n",
    "    **kwargs\n",
    ")**\n",
    "\n",
    "该层神经单元的个数units；激活函数activation，比较常用的linear,softmax,tanh,sigmoid,sinh,relu,selu,softplus等，需要根据输出来确定用哪个激活函数比较合适；是否添加偏置use_bias，默认为True，相当于线性回归中的b；权重的初始化kernel_initializer，选择合适的有利于获得成本函数的最优值；偏置的初始化bias_initializer；权重、偏置的正则化kernel_regularizer，bias_regularizer，有助于防止过拟合；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "transpose_1 (Flatten)        (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "hidden_layer_1 (Dense)       (None, 128)               100480    \n",
      "_________________________________________________________________\n",
      "hidden_layer_2 (Dense)       (None, 256)               33024     \n",
      "_________________________________________________________________\n",
      "hidden_layer_3 (Dense)       (None, 10)                2570      \n",
      "=================================================================\n",
      "Total params: 136,074\n",
      "Trainable params: 136,074\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# 实现一：通过.add方式添加\n",
    "\n",
    "def build_model_1(name='python_fan'):  # name:模型的名称\n",
    "    # 首先定义一个Sequential\n",
    "    model = tf.keras.models.Sequential(name=name)\n",
    "\n",
    "    # 添加输入层：平铺，也就是将每条维度为(28, 28)的数据变为(28*28=784,)的。\n",
    "    model.add(keras.layers.Flatten(input_shape=(28, 28), name='transpose_1'))\n",
    "\n",
    "    # 添加第一个隐层：全连接，128表示该层神经元的个数\n",
    "    model.add(keras.layers.Dense(128, name='hidden_layer_1', activation='relu', kernel_initializer='glorot_normal'))\n",
    "    # 添加第二个隐层：全连接，256表示该层神经元的个数\n",
    "    model.add(keras.layers.Dense(256, name='hidden_layer_2', activation='relu'))\n",
    "\n",
    "    # 添加输出层：全连接\n",
    "    model.add(keras.layers.Dense(Out_Class, name='hidden_layer_3', activation='softmax'))\n",
    "    return model\n",
    "\n",
    "model = build_model_1('sequential_1')\n",
    "# 模型结构描述\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "transpose_1 (Flatten)        (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "hidden_layer_1 (Dense)       (None, 128)               100480    \n",
      "_________________________________________________________________\n",
      "hidden_layer_2 (Dense)       (None, 256)               33024     \n",
      "_________________________________________________________________\n",
      "hidden_layer_3 (Dense)       (None, 10)                2570      \n",
      "=================================================================\n",
      "Total params: 136,074\n",
      "Trainable params: 136,074\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# 实现二:列表形式\n",
    "def build_model_2(name='python_fan'):  # name:模型的名称\n",
    "    model = keras.Sequential([\n",
    "        keras.layers.Flatten(input_shape=(28, 28), name='transpose_1'),  # 输入层\n",
    "        keras.layers.Dense(128, name='hidden_layer_1', activation='relu'), # 隐层1\n",
    "        keras.layers.Dense(256, name='hidden_layer_2', activation='relu'), # 隐层2\n",
    "        keras.layers.Dense(Out_Class, name='hidden_layer_3', activation='softmax') # 输出层\n",
    "        ], name=name)\n",
    "    return model\n",
    "model = build_model_2('sequential_2')\n",
    "# 模型结构描述\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参数个数说明：\n",
    " + transpose_1：不涉及参数\n",
    " + hidden_layer_1：$100480 = 784 \\times 128 + 128$\n",
    " + hidden_layer_2：$33024= 128 \\times 256 + 256$\n",
    " + hidden_layer_3：$2570= 256 \\times 10 + 10$\n",
    " + Total params：$136074=100480+33024+2570$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2  模型编译 **tf.keras.models.Sequential.compile()**\n",
    "\n",
    "**compile(optimizer='rmsprop', loss=None, metrics=None, loss_weights=None,sample_weight_mode=None, weighted_metrics=None, **kwargs)**\n",
    "\n",
    "设置求解模型参数的优化器optimizer，包括Adadelta，Adagrad，Adam，Adamax，Ftrl，Nadam，RMSprop，Sgd等；损失函数loss，回归问题一般采用MSE；分类问题可采用categorical_crossentropy，sparse_categorical_crossentropy，binary_crossentropy；用于监控训练，输出当前模型metrics中的指标，例如分类问题的准确率accuracy，回归问题中的mse。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 模型编译\n",
    "model.compile(optimizer='Sgd', loss='categorical_crossentropy', \n",
    "              metrics=['accuracy'])  \n",
    "\n",
    "# 或者将字符串写成函数的形式，这种方式可以更改优化器的参数，例如学习率之类的\n",
    "\"\"\"\n",
    "# model.compile(optimizer=tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.0), \n",
    "loss=tf.keras.losses.CategoricalCrossentropy(),metrics=[tf.keras.metrics.CategoricalAccuracy()]) \n",
    "\"\"\"\n",
    "# 标签数据要进行one-hot编码，因为loss设置的是categorical_crossentropy，如果设置的是sparse_categorical_crossentropy，则不需要下面的转化。\n",
    "train_label_cate = tf.keras.utils.to_categorical(train_labels, Out_Class)\n",
    "testlabels = tf.keras.utils.to_categorical(test_labels, Out_Class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 模型回调保存 **tf.keras.callbacks.ModelCheckpoint()**\n",
    "\n",
    "**tf.keras.callbacks.ModelCheckpoint(\n",
    "    filepath, monitor='val_loss', verbose=0, save_best_only=False,\n",
    "    save_weights_only=False, mode='auto', save_freq='epoch', **kwargs\n",
    ")**\n",
    "\n",
    "可以使用训练好的模型而无需从头开始重新训练，或在打断的地方开始训练，以防止训练过程没有保存。\n",
    "\n",
    "保存的文件名filepath，文件名可自定义为迭代次数或者训练中的指标值，例如**para.{epoch:03d}-{val_accuracy:.5f}.ckpt**；参数monitor和参数mode是对应设置的，如果前者为误差成本，后者就设置为最小；前者为准确率，后者就设置为最大或者自动。save_best_only如果设置为True，并且文件名是自定义的，则新的文件会覆盖旧的文件；如果名称包含自定义的，则不会覆盖；如果设置为Fasle，则会全部保留；\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "# 在文件名中包含 epoch (使用 `str.format`)\n",
    "checkpoint_path = \"./cp-{val_accuracy:.5f}.ckpt\"\n",
    "checkpoint_dir = os.path.dirname(checkpoint_path)\n",
    "\n",
    "# 创建一个回调，保证验证数据集准确率最大\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(\n",
    "    filepath=checkpoint_path,\n",
    "    save_weights_only=True,\n",
    "    monitor='val_accuracy',\n",
    "    mode='max',\n",
    "    verbose=2,\n",
    "    save_best_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4  模型训练 **tf.keras.models.Sequential.fit()**\n",
    "\n",
    "**fit(\n",
    "    x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None,\n",
    "    validation_split=0.0, validation_data=None, shuffle=True, class_weight=None,\n",
    "    sample_weight=None, initial_epoch=0, steps_per_epoch=None,\n",
    "    validation_steps=None, validation_batch_size=None, validation_freq=1,\n",
    "    max_queue_size=10, workers=1, use_multiprocessing=False, **kwargs\n",
    ")**\n",
    "\n",
    "设置特征数据x，输出数据y，批量训练中一次训练输入的数据条数batch_size，循环的次数epochs，训练过程中如何输出训练数据和验证数据的模型的评估数据verbose，0是不输出，1是进度条输出，2是一次训练输出一次，建议值2；验证数据集占特征数据的比例validation_split，如果给出了验证数据集validation_data，则值设置为0。是否随机打乱数据集shuffle，默认为True。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 54000 samples, validate on 6000 samples\n",
      "Epoch 1/100\n",
      "\n",
      "Epoch 00001: val_accuracy did not improve from 0.97783\n",
      "54000/54000 - 1s - loss: 0.0656 - accuracy: 0.9817 - val_loss: 0.0821 - val_accuracy: 0.9775\n",
      "Epoch 2/100\n",
      "\n",
      "Epoch 00002: val_accuracy did not improve from 0.97783\n",
      "54000/54000 - 1s - loss: 0.0645 - accuracy: 0.9822 - val_loss: 0.0815 - val_accuracy: 0.9778\n",
      "Epoch 3/100\n",
      "\n",
      "Epoch 00003: val_accuracy improved from 0.97783 to 0.97833, saving model to ./cp-0.97833.ckpt\n",
      "54000/54000 - 1s - loss: 0.0636 - accuracy: 0.9825 - val_loss: 0.0808 - val_accuracy: 0.9783\n",
      "Epoch 4/100\n",
      "\n",
      "Epoch 00004: val_accuracy improved from 0.97833 to 0.97867, saving model to ./cp-0.97867.ckpt\n",
      "54000/54000 - 1s - loss: 0.0628 - accuracy: 0.9827 - val_loss: 0.0806 - val_accuracy: 0.9787\n",
      "Epoch 5/100\n",
      "\n",
      "Epoch 00005: val_accuracy did not improve from 0.97867\n",
      "54000/54000 - 1s - loss: 0.0618 - accuracy: 0.9831 - val_loss: 0.0806 - val_accuracy: 0.9777\n",
      "Epoch 6/100\n",
      "\n",
      "Epoch 00006: val_accuracy did not improve from 0.97867\n",
      "54000/54000 - 1s - loss: 0.0611 - accuracy: 0.9833 - val_loss: 0.0796 - val_accuracy: 0.9787\n",
      "Epoch 7/100\n",
      "\n",
      "Epoch 00007: val_accuracy improved from 0.97867 to 0.97883, saving model to ./cp-0.97883.ckpt\n",
      "54000/54000 - 1s - loss: 0.0601 - accuracy: 0.9838 - val_loss: 0.0799 - val_accuracy: 0.9788\n",
      "Epoch 8/100\n",
      "\n",
      "Epoch 00008: val_accuracy did not improve from 0.97883\n",
      "54000/54000 - 1s - loss: 0.0592 - accuracy: 0.9839 - val_loss: 0.0790 - val_accuracy: 0.9780\n",
      "Epoch 9/100\n",
      "\n",
      "Epoch 00009: val_accuracy did not improve from 0.97883\n",
      "54000/54000 - 1s - loss: 0.0585 - accuracy: 0.9844 - val_loss: 0.0783 - val_accuracy: 0.9783\n",
      "Epoch 10/100\n",
      "\n",
      "Epoch 00010: val_accuracy did not improve from 0.97883\n",
      "54000/54000 - 1s - loss: 0.0577 - accuracy: 0.9844 - val_loss: 0.0790 - val_accuracy: 0.9778\n",
      "Epoch 11/100\n",
      "\n",
      "Epoch 00011: val_accuracy improved from 0.97883 to 0.97917, saving model to ./cp-0.97917.ckpt\n",
      "54000/54000 - 1s - loss: 0.0569 - accuracy: 0.9845 - val_loss: 0.0785 - val_accuracy: 0.9792\n",
      "Epoch 12/100\n",
      "\n",
      "Epoch 00012: val_accuracy did not improve from 0.97917\n",
      "54000/54000 - 1s - loss: 0.0561 - accuracy: 0.9844 - val_loss: 0.0774 - val_accuracy: 0.9787\n",
      "Epoch 13/100\n",
      "\n",
      "Epoch 00013: val_accuracy did not improve from 0.97917\n",
      "54000/54000 - 1s - loss: 0.0553 - accuracy: 0.9850 - val_loss: 0.0778 - val_accuracy: 0.9782\n",
      "Epoch 14/100\n",
      "\n",
      "Epoch 00014: val_accuracy improved from 0.97917 to 0.97933, saving model to ./cp-0.97933.ckpt\n",
      "54000/54000 - 1s - loss: 0.0546 - accuracy: 0.9851 - val_loss: 0.0769 - val_accuracy: 0.9793\n",
      "Epoch 15/100\n",
      "\n",
      "Epoch 00015: val_accuracy improved from 0.97933 to 0.97967, saving model to ./cp-0.97967.ckpt\n",
      "54000/54000 - 1s - loss: 0.0539 - accuracy: 0.9852 - val_loss: 0.0768 - val_accuracy: 0.9797\n",
      "Epoch 16/100\n",
      "\n",
      "Epoch 00016: val_accuracy did not improve from 0.97967\n",
      "54000/54000 - 1s - loss: 0.0531 - accuracy: 0.9859 - val_loss: 0.0764 - val_accuracy: 0.9793\n",
      "Epoch 17/100\n",
      "\n",
      "Epoch 00017: val_accuracy did not improve from 0.97967\n",
      "54000/54000 - 1s - loss: 0.0524 - accuracy: 0.9859 - val_loss: 0.0764 - val_accuracy: 0.9783\n",
      "Epoch 18/100\n",
      "\n",
      "Epoch 00018: val_accuracy did not improve from 0.97967\n",
      "54000/54000 - 1s - loss: 0.0516 - accuracy: 0.9861 - val_loss: 0.0766 - val_accuracy: 0.9788\n",
      "Epoch 19/100\n",
      "\n",
      "Epoch 00019: val_accuracy did not improve from 0.97967\n",
      "54000/54000 - 1s - loss: 0.0510 - accuracy: 0.9864 - val_loss: 0.0760 - val_accuracy: 0.9790\n",
      "Epoch 20/100\n",
      "\n",
      "Epoch 00020: val_accuracy did not improve from 0.97967\n",
      "54000/54000 - 1s - loss: 0.0503 - accuracy: 0.9865 - val_loss: 0.0763 - val_accuracy: 0.9792\n",
      "Epoch 21/100\n",
      "\n",
      "Epoch 00021: val_accuracy did not improve from 0.97967\n",
      "54000/54000 - 1s - loss: 0.0497 - accuracy: 0.9868 - val_loss: 0.0749 - val_accuracy: 0.9787\n",
      "Epoch 22/100\n",
      "\n",
      "Epoch 00022: val_accuracy did not improve from 0.97967\n",
      "54000/54000 - 1s - loss: 0.0490 - accuracy: 0.9869 - val_loss: 0.0749 - val_accuracy: 0.9790\n",
      "Epoch 23/100\n",
      "\n",
      "Epoch 00023: val_accuracy improved from 0.97967 to 0.97983, saving model to ./cp-0.97983.ckpt\n",
      "54000/54000 - 1s - loss: 0.0484 - accuracy: 0.9871 - val_loss: 0.0748 - val_accuracy: 0.9798\n",
      "Epoch 24/100\n",
      "\n",
      "Epoch 00024: val_accuracy did not improve from 0.97983\n",
      "54000/54000 - 1s - loss: 0.0478 - accuracy: 0.9872 - val_loss: 0.0744 - val_accuracy: 0.9798\n",
      "Epoch 25/100\n",
      "\n",
      "Epoch 00025: val_accuracy improved from 0.97983 to 0.98017, saving model to ./cp-0.98017.ckpt\n",
      "54000/54000 - 1s - loss: 0.0471 - accuracy: 0.9876 - val_loss: 0.0750 - val_accuracy: 0.9802\n",
      "Epoch 26/100\n",
      "\n",
      "Epoch 00026: val_accuracy did not improve from 0.98017\n",
      "54000/54000 - 1s - loss: 0.0465 - accuracy: 0.9877 - val_loss: 0.0744 - val_accuracy: 0.9795\n",
      "Epoch 27/100\n",
      "\n",
      "Epoch 00027: val_accuracy did not improve from 0.98017\n",
      "54000/54000 - 1s - loss: 0.0459 - accuracy: 0.9877 - val_loss: 0.0738 - val_accuracy: 0.9788\n",
      "Epoch 28/100\n",
      "\n",
      "Epoch 00028: val_accuracy did not improve from 0.98017\n",
      "54000/54000 - 1s - loss: 0.0453 - accuracy: 0.9881 - val_loss: 0.0746 - val_accuracy: 0.9798\n",
      "Epoch 29/100\n",
      "\n",
      "Epoch 00029: val_accuracy did not improve from 0.98017\n",
      "54000/54000 - 1s - loss: 0.0448 - accuracy: 0.9881 - val_loss: 0.0742 - val_accuracy: 0.9788\n",
      "Epoch 30/100\n",
      "\n",
      "Epoch 00030: val_accuracy did not improve from 0.98017\n",
      "54000/54000 - 1s - loss: 0.0443 - accuracy: 0.9886 - val_loss: 0.0734 - val_accuracy: 0.9800\n",
      "Epoch 31/100\n",
      "\n",
      "Epoch 00031: val_accuracy did not improve from 0.98017\n",
      "54000/54000 - 1s - loss: 0.0435 - accuracy: 0.9885 - val_loss: 0.0739 - val_accuracy: 0.9785\n",
      "Epoch 32/100\n",
      "\n",
      "Epoch 00032: val_accuracy did not improve from 0.98017\n",
      "54000/54000 - 1s - loss: 0.0431 - accuracy: 0.9889 - val_loss: 0.0736 - val_accuracy: 0.9797\n",
      "Epoch 33/100\n",
      "\n",
      "Epoch 00033: val_accuracy did not improve from 0.98017\n",
      "54000/54000 - 1s - loss: 0.0425 - accuracy: 0.9890 - val_loss: 0.0741 - val_accuracy: 0.9795\n",
      "Epoch 34/100\n",
      "\n",
      "Epoch 00034: val_accuracy did not improve from 0.98017\n",
      "54000/54000 - 1s - loss: 0.0420 - accuracy: 0.9890 - val_loss: 0.0731 - val_accuracy: 0.9788\n",
      "Epoch 35/100\n",
      "\n",
      "Epoch 00035: val_accuracy improved from 0.98017 to 0.98050, saving model to ./cp-0.98050.ckpt\n",
      "54000/54000 - 1s - loss: 0.0415 - accuracy: 0.9893 - val_loss: 0.0723 - val_accuracy: 0.9805\n",
      "Epoch 36/100\n",
      "\n",
      "Epoch 00036: val_accuracy did not improve from 0.98050\n",
      "54000/54000 - 1s - loss: 0.0407 - accuracy: 0.9893 - val_loss: 0.0731 - val_accuracy: 0.9788\n",
      "Epoch 37/100\n",
      "\n",
      "Epoch 00037: val_accuracy did not improve from 0.98050\n",
      "54000/54000 - 1s - loss: 0.0404 - accuracy: 0.9896 - val_loss: 0.0721 - val_accuracy: 0.9797\n",
      "Epoch 38/100\n",
      "\n",
      "Epoch 00038: val_accuracy did not improve from 0.98050\n",
      "54000/54000 - 1s - loss: 0.0399 - accuracy: 0.9899 - val_loss: 0.0720 - val_accuracy: 0.9788\n",
      "Epoch 39/100\n",
      "\n",
      "Epoch 00039: val_accuracy did not improve from 0.98050\n",
      "54000/54000 - 1s - loss: 0.0395 - accuracy: 0.9901 - val_loss: 0.0719 - val_accuracy: 0.9792\n",
      "Epoch 40/100\n",
      "\n",
      "Epoch 00040: val_accuracy did not improve from 0.98050\n",
      "54000/54000 - 1s - loss: 0.0389 - accuracy: 0.9902 - val_loss: 0.0724 - val_accuracy: 0.9797\n",
      "Epoch 41/100\n",
      "\n",
      "Epoch 00041: val_accuracy did not improve from 0.98050\n",
      "54000/54000 - 1s - loss: 0.0384 - accuracy: 0.9901 - val_loss: 0.0715 - val_accuracy: 0.9802\n",
      "Epoch 42/100\n",
      "\n",
      "Epoch 00042: val_accuracy improved from 0.98050 to 0.98083, saving model to ./cp-0.98083.ckpt\n",
      "54000/54000 - 1s - loss: 0.0379 - accuracy: 0.9903 - val_loss: 0.0732 - val_accuracy: 0.9808\n",
      "Epoch 43/100\n",
      "\n",
      "Epoch 00043: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0375 - accuracy: 0.9904 - val_loss: 0.0720 - val_accuracy: 0.9790\n",
      "Epoch 44/100\n",
      "\n",
      "Epoch 00044: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0370 - accuracy: 0.9908 - val_loss: 0.0718 - val_accuracy: 0.9797\n",
      "Epoch 45/100\n",
      "\n",
      "Epoch 00045: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0366 - accuracy: 0.9906 - val_loss: 0.0722 - val_accuracy: 0.9793\n",
      "Epoch 46/100\n",
      "\n",
      "Epoch 00046: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0361 - accuracy: 0.9907 - val_loss: 0.0714 - val_accuracy: 0.9795\n",
      "Epoch 47/100\n",
      "\n",
      "Epoch 00047: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0357 - accuracy: 0.9911 - val_loss: 0.0711 - val_accuracy: 0.9790\n",
      "Epoch 48/100\n",
      "\n",
      "Epoch 00048: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0352 - accuracy: 0.9915 - val_loss: 0.0721 - val_accuracy: 0.9788\n",
      "Epoch 49/100\n",
      "\n",
      "Epoch 00049: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0349 - accuracy: 0.9913 - val_loss: 0.0721 - val_accuracy: 0.9800\n",
      "Epoch 50/100\n",
      "\n",
      "Epoch 00050: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0343 - accuracy: 0.9916 - val_loss: 0.0726 - val_accuracy: 0.9800\n",
      "Epoch 51/100\n",
      "\n",
      "Epoch 00051: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0339 - accuracy: 0.9917 - val_loss: 0.0722 - val_accuracy: 0.9793\n",
      "Epoch 52/100\n",
      "\n",
      "Epoch 00052: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0335 - accuracy: 0.9919 - val_loss: 0.0714 - val_accuracy: 0.9795\n",
      "Epoch 53/100\n",
      "\n",
      "Epoch 00053: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0330 - accuracy: 0.9919 - val_loss: 0.0712 - val_accuracy: 0.9795\n",
      "Epoch 54/100\n",
      "\n",
      "Epoch 00054: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0327 - accuracy: 0.9921 - val_loss: 0.0719 - val_accuracy: 0.9800\n",
      "Epoch 55/100\n",
      "\n",
      "Epoch 00055: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0323 - accuracy: 0.9923 - val_loss: 0.0725 - val_accuracy: 0.9793\n",
      "Epoch 56/100\n",
      "\n",
      "Epoch 00056: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0320 - accuracy: 0.9926 - val_loss: 0.0709 - val_accuracy: 0.9788\n",
      "Epoch 57/100\n",
      "\n",
      "Epoch 00057: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0316 - accuracy: 0.9925 - val_loss: 0.0708 - val_accuracy: 0.9788\n",
      "Epoch 58/100\n",
      "\n",
      "Epoch 00058: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0312 - accuracy: 0.9926 - val_loss: 0.0709 - val_accuracy: 0.9792\n",
      "Epoch 59/100\n",
      "\n",
      "Epoch 00059: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0307 - accuracy: 0.9931 - val_loss: 0.0715 - val_accuracy: 0.9792\n",
      "Epoch 60/100\n",
      "\n",
      "Epoch 00060: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0305 - accuracy: 0.9926 - val_loss: 0.0716 - val_accuracy: 0.9802\n",
      "Epoch 61/100\n",
      "\n",
      "Epoch 00061: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0301 - accuracy: 0.9929 - val_loss: 0.0713 - val_accuracy: 0.9797\n",
      "Epoch 62/100\n",
      "\n",
      "Epoch 00062: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0297 - accuracy: 0.9932 - val_loss: 0.0710 - val_accuracy: 0.9788\n",
      "Epoch 63/100\n",
      "\n",
      "Epoch 00063: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0294 - accuracy: 0.9932 - val_loss: 0.0708 - val_accuracy: 0.9800\n",
      "Epoch 64/100\n",
      "\n",
      "Epoch 00064: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0291 - accuracy: 0.9935 - val_loss: 0.0708 - val_accuracy: 0.9795\n",
      "Epoch 65/100\n",
      "\n",
      "Epoch 00065: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0286 - accuracy: 0.9935 - val_loss: 0.0706 - val_accuracy: 0.9805\n",
      "Epoch 66/100\n",
      "\n",
      "Epoch 00066: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0284 - accuracy: 0.9935 - val_loss: 0.0706 - val_accuracy: 0.9797\n",
      "Epoch 67/100\n",
      "\n",
      "Epoch 00067: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0280 - accuracy: 0.9933 - val_loss: 0.0704 - val_accuracy: 0.9792\n",
      "Epoch 68/100\n",
      "\n",
      "Epoch 00068: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0277 - accuracy: 0.9937 - val_loss: 0.0710 - val_accuracy: 0.9790\n",
      "Epoch 69/100\n",
      "\n",
      "Epoch 00069: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0273 - accuracy: 0.9938 - val_loss: 0.0720 - val_accuracy: 0.9787\n",
      "Epoch 70/100\n",
      "\n",
      "Epoch 00070: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0271 - accuracy: 0.9938 - val_loss: 0.0703 - val_accuracy: 0.9797\n",
      "Epoch 71/100\n",
      "\n",
      "Epoch 00071: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0267 - accuracy: 0.9941 - val_loss: 0.0713 - val_accuracy: 0.9805\n",
      "Epoch 72/100\n",
      "\n",
      "Epoch 00072: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0264 - accuracy: 0.9941 - val_loss: 0.0707 - val_accuracy: 0.9797\n",
      "Epoch 73/100\n",
      "\n",
      "Epoch 00073: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0260 - accuracy: 0.9943 - val_loss: 0.0715 - val_accuracy: 0.9795\n",
      "Epoch 74/100\n",
      "\n",
      "Epoch 00074: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0258 - accuracy: 0.9943 - val_loss: 0.0712 - val_accuracy: 0.9788\n",
      "Epoch 75/100\n",
      "\n",
      "Epoch 00075: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0255 - accuracy: 0.9946 - val_loss: 0.0714 - val_accuracy: 0.9793\n",
      "Epoch 76/100\n",
      "\n",
      "Epoch 00076: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0251 - accuracy: 0.9946 - val_loss: 0.0704 - val_accuracy: 0.9793\n",
      "Epoch 77/100\n",
      "\n",
      "Epoch 00077: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0249 - accuracy: 0.9946 - val_loss: 0.0711 - val_accuracy: 0.9795\n",
      "Epoch 78/100\n",
      "\n",
      "Epoch 00078: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0246 - accuracy: 0.9947 - val_loss: 0.0711 - val_accuracy: 0.9792\n",
      "Epoch 79/100\n",
      "\n",
      "Epoch 00079: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0243 - accuracy: 0.9948 - val_loss: 0.0713 - val_accuracy: 0.9800\n",
      "Epoch 80/100\n",
      "\n",
      "Epoch 00080: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0239 - accuracy: 0.9949 - val_loss: 0.0710 - val_accuracy: 0.9795\n",
      "Epoch 81/100\n",
      "\n",
      "Epoch 00081: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0238 - accuracy: 0.9950 - val_loss: 0.0709 - val_accuracy: 0.9797\n",
      "Epoch 82/100\n",
      "\n",
      "Epoch 00082: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0234 - accuracy: 0.9953 - val_loss: 0.0719 - val_accuracy: 0.9797\n",
      "Epoch 83/100\n",
      "\n",
      "Epoch 00083: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0232 - accuracy: 0.9953 - val_loss: 0.0709 - val_accuracy: 0.9798\n",
      "Epoch 84/100\n",
      "\n",
      "Epoch 00084: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0229 - accuracy: 0.9952 - val_loss: 0.0710 - val_accuracy: 0.9798\n",
      "Epoch 85/100\n",
      "\n",
      "Epoch 00085: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0226 - accuracy: 0.9955 - val_loss: 0.0711 - val_accuracy: 0.9793\n",
      "Epoch 86/100\n",
      "\n",
      "Epoch 00086: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0224 - accuracy: 0.9955 - val_loss: 0.0711 - val_accuracy: 0.9797\n",
      "Epoch 87/100\n",
      "\n",
      "Epoch 00087: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0222 - accuracy: 0.9955 - val_loss: 0.0706 - val_accuracy: 0.9792\n",
      "Epoch 88/100\n",
      "\n",
      "Epoch 00088: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0219 - accuracy: 0.9957 - val_loss: 0.0713 - val_accuracy: 0.9800\n",
      "Epoch 89/100\n",
      "\n",
      "Epoch 00089: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0216 - accuracy: 0.9959 - val_loss: 0.0711 - val_accuracy: 0.9798\n",
      "Epoch 90/100\n",
      "\n",
      "Epoch 00090: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0214 - accuracy: 0.9958 - val_loss: 0.0714 - val_accuracy: 0.9795\n",
      "Epoch 91/100\n",
      "\n",
      "Epoch 00091: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0212 - accuracy: 0.9959 - val_loss: 0.0713 - val_accuracy: 0.9795\n",
      "Epoch 92/100\n",
      "\n",
      "Epoch 00092: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0209 - accuracy: 0.9963 - val_loss: 0.0708 - val_accuracy: 0.9797\n",
      "Epoch 93/100\n",
      "\n",
      "Epoch 00093: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0207 - accuracy: 0.9961 - val_loss: 0.0711 - val_accuracy: 0.9790\n",
      "Epoch 94/100\n",
      "\n",
      "Epoch 00094: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0204 - accuracy: 0.9962 - val_loss: 0.0711 - val_accuracy: 0.9793\n",
      "Epoch 95/100\n",
      "\n",
      "Epoch 00095: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0202 - accuracy: 0.9963 - val_loss: 0.0708 - val_accuracy: 0.9793\n",
      "Epoch 96/100\n",
      "\n",
      "Epoch 00096: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0200 - accuracy: 0.9963 - val_loss: 0.0712 - val_accuracy: 0.9790\n",
      "Epoch 97/100\n",
      "\n",
      "Epoch 00097: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0197 - accuracy: 0.9964 - val_loss: 0.0710 - val_accuracy: 0.9797\n",
      "Epoch 98/100\n",
      "\n",
      "Epoch 00098: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0195 - accuracy: 0.9965 - val_loss: 0.0715 - val_accuracy: 0.9797\n",
      "Epoch 99/100\n",
      "\n",
      "Epoch 00099: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0192 - accuracy: 0.9967 - val_loss: 0.0721 - val_accuracy: 0.9788\n",
      "Epoch 100/100\n",
      "\n",
      "Epoch 00100: val_accuracy did not improve from 0.98083\n",
      "54000/54000 - 1s - loss: 0.0190 - accuracy: 0.9967 - val_loss: 0.0716 - val_accuracy: 0.9797\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0xe56c0c8>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 训练模型\n",
    "model.fit(trainfeatures, train_label_cate, batch_size=Batch_Size, \\\n",
    "          epochs=Epoch, verbose=2, validation_split=0.1, callbacks=[cp_callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 模型评估 **tf.keras.models.Sequential.evaluate()**\n",
    "\n",
    "**evaluate(\n",
    "    x=None, y=None, batch_size=None, verbose=1, sample_weight=None, steps=None,\n",
    "    callbacks=None, max_queue_size=10, workers=1, use_multiprocessing=False,\n",
    "    return_dict=False\n",
    ")**\n",
    "\n",
    "设置测试的特征数据x；测试对应的输出数据y；其中两种数据的格式要和训练的数据保持一致，只允许样本数可以不同；callbacks设置对哪个已经保存的模型进行评估。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 33us/sample - loss: 0.0707 - accuracy: 0.9779\n",
      "测试数据集成本：0.07070848,准确率97.79000282%%\n"
     ]
    }
   ],
   "source": [
    "# 评估模型,按照模型最后的参数计算\n",
    "test_loss, test_acc = model.evaluate(testfeatures, testlabels)\n",
    "\n",
    "print('测试数据集成本：{:.8f},准确率{:.8f}%%'. format(test_loss, 100*test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6 参数加载 **tf.train.latest_checkpoint()**\n",
    "\n",
    "   利用**tf.train.latest_checkpoint()** 可以获得最新的保存的参数文件，或者直接使用保存的参数文件名称，然后用 **.load_weights**来读取保存的参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 44us/sample - loss: 2.3192 - accuracy: 0.1082\n",
      "使用初始参数 成本: 2.319191443634033 准确率 0.1082\n",
      "最优的参数文件： .\\cp-0.98083.ckpt\n",
      "10000/10000 [==============================] - 0s 32us/sample - loss: 0.0787 - accuracy: 0.9758\n",
      "使用训练后的参数 成本: 0.07869966647587717 准确率 0.9758\n"
     ]
    }
   ],
   "source": [
    "# 6. 参数加载\n",
    "# 新的模型结构保持一致。\n",
    "model_new = build_model_2('sequential_new')\n",
    "# 需要经过编译，参数也要和原来的一致\n",
    "model_new.compile(optimizer='Sgd', loss='categorical_crossentropy', \n",
    "              metrics=['accuracy'])\n",
    "# 此时没有加载已经训练好的参数，也就是根据初始的参数值计算，因为有10类，准确率大概十分之一。\n",
    "predict_loss, predict_acc = model_new.evaluate(testfeatures, testlabels)\n",
    "print('使用初始参数','成本:', predict_loss, '准确率', predict_acc)\n",
    "\n",
    "# 加载已经训练好的参数\n",
    "best_para = tf.train.latest_checkpoint(checkpoint_dir)\n",
    "print('最优的参数文件：', best_para)\n",
    "model_new.load_weights(best_para)\n",
    "predict_loss, predict_acc = model_new.evaluate(testfeatures, testlabels)\n",
    "print('使用训练后的参数','成本:', predict_loss, '准确率', predict_acc)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3、保存模型\n",
    "\n",
    "除了上面说的在模型训练中保存模型以外，还可以在模型训练结束后保存模型参数或者整个训练好的模型结构。\n",
    "+ 3.1 手动保存模型参数 **save_weights**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 44us/sample - loss: 0.0707 - accuracy: 0.9779\n",
      "成本: 0.07070848318163771 准确率 0.9779\n"
     ]
    }
   ],
   "source": [
    "# 手动保存权重，保存的是最后的参数\n",
    "model.save_weights('./mnist_4_checkpoint')\n",
    "\n",
    "# 构建模型\n",
    "model_save = build_model_1('newmodel')\n",
    "model_save.load_weights('./mnist_4_checkpoint')\n",
    "\n",
    "model_save.compile(optimizer='Sgd', loss='categorical_crossentropy', metrics=['accuracy'])  \n",
    "loss, acc = model_save.evaluate(testfeatures, testlabels)\n",
    "print('成本:', loss, '准确率', acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ 3.2 保存整个模型结构 **save**\n",
    "\n",
    "保存整个模型结构，包括参数、结构、配置，这样就可以在不访问原始python代码的情况下使用它。如果使用TensorFlow.js加载它们，可在Web浏览器中训练和运行它们；如果使用TensorFlow Lite，可以在移动设备上运行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "transpose_1 (Flatten)        (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "hidden_layer_1 (Dense)       (None, 128)               100480    \n",
      "_________________________________________________________________\n",
      "hidden_layer_2 (Dense)       (None, 256)               33024     \n",
      "_________________________________________________________________\n",
      "hidden_layer_3 (Dense)       (None, 10)                2570      \n",
      "=================================================================\n",
      "Total params: 136,074\n",
      "Trainable params: 136,074\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "10000/10000 [==============================] - 0s 43us/sample - loss: 0.0707 - accuracy: 0.9779\n",
      "成本: 0.07070848318163771 准确率 0.9779\n"
     ]
    }
   ],
   "source": [
    "# 保存整个模型到HDF5文件 \n",
    "model.save('./miaier.h5')\n",
    "# 加载模型\n",
    "new_model = keras.models.load_model('./miaier.h5')\n",
    "new_model.summary()\n",
    "loss, acc = new_model.evaluate(testfeatures, testlabels)\n",
    "print('成本:', loss, '准确率', acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4、模型预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "输出的数字结果: [4 9 6 9 0 6 9 0 1 5 9 7 3 4 9 6 6]\n",
      "真实的数字结果: [4 9 5 9 0 6 9 0 1 5 9 7 3 4 9 6 6]\n"
     ]
    }
   ],
   "source": [
    "# 因为没有预测数据，我们在测试数据中选取一部分作为预测数据\n",
    "predict_features = testfeatures[6:23, :, :]\n",
    "predict_labels = testlabels[6:23]\n",
    "output_label = new_model.predict(predict_features)\n",
    "# 获取每一行中最大数字的索引\n",
    "digit_label = tf.argmax(output_label, axis=1)\n",
    "print('输出的数字结果:', digit_label.numpy())\n",
    "print('真实的数字结果:', tf.argmax(predict_labels, axis=1).numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5、模型优化\n",
    "\n",
    "下面介绍几种MLP模型优化的方式：\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ 防止过拟合\n",
    "\n",
    "  + L1/L2正则化：kernel_regularizer=keras.regularizers.l2(0.001)\n",
    "    + 例如：keras.layers.Dense(128, name='hidden_layer_1', activation='relu', kernel_regularizer=keras.regularizers.l2(0.001))\n",
    "   \n",
    "  + 丢弃：添加DropOut层\n",
    "    + 例如：model.add(keras.layers.Dropout(0.2)) \n",
    "   \n",
    "  + 提前结束训练：在模型的回调中使用\n",
    "    + callbacks = [tf.keras.callbacks.EarlyStopping(patience=4, monitor='val_loss')]，当验证数据集的成本连续4次不在变化时，就停止训练\n",
    "   \n",
    "+ 有助于得到最优解\n",
    "\n",
    "  + 动态更改学习率：在模型的回调中使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 动态更改学习率：在模型的回调中使用\n",
    "def scheduler(epoch):  # 根据epoch动态更改学习率的参数\n",
    "    if epoch < 10:\n",
    "        return 0.001\n",
    "    else:\n",
    "        return 0.001 * tf.math.exp(0.1 * (10 - epoch))\n",
    "callbacks = [tf.keras.callbacks.LearningRateScheduler(scheduler)]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "TensorFlow2",
   "language": "python",
   "name": "tf2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}