rnn_tang_poetry.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CvFDf5nLhtzx"
   },
   "source": [
    "# 唐詩三百首藏頭詩產生器\n",
    "\n",
    "### 中文的困難點\n",
    "> 一般常見的範例是用英文的，然後拆成字元的形式．但是中文沒辦法這樣做，它每個字都是單獨的．\n",
    "> 一般的英文字元分類大概3, 40個，但光是用唐詩三百首為資料的詞彙就有2千多個．所以在分類訓練上來說比較難達到精確的結果．\n",
    "\n",
    "### 以唐詩三百首當訓練的優點\n",
    "> 資料量適中\n",
    "> 詩講究的是意境，人腦會自動腦補其句子含義．即使句子不精確但好像還像回事．\n",
    "\n",
    "### 設計時遇到的問題\n",
    "> 這原本是Udacity AIND的一個作業，他是用tensorflow實作．我想改成keras來應該是很簡單阿．\n",
    "在作業裡它已經幫忙把資料處理的架構的寫好，所以只要指定window size, step size就會將資料拆成批次．\n",
    "但Keras LSTM相對簡單，它只需要決定input一次要給幾個字、output幾個字．\n",
    "\n",
    "> 因為中文字是方塊字，不是字符組成．而且唐詩的每個字的相關性不需要很久遠前的字來影響．\n",
    "所以一般英文字的window size設個100，在中文裡就不適合．\n",
    "又因為最後的結果是要以藏頭詩的形式產生後續的字詞，所以input也只會有一個字．\n",
    "最後決定就是以1個input對1個output的形式作為LSTM的輸入．\n",
    "\n",
    "### 資料處理\n",
    "> 這個部分是最花時間的，除了等待訓練以外．\n",
    "我記得是從中國的一個網站下載的(資料來源已經忘了，但不得不說中國在資料整理、搜集這方面做的真的比台灣好太多)，\n",
    "但文字檔裡包含了詩名稱、作者等資訊．但我只要唐詩的部分，所以只好用人工的方式把不要的刪除．\n",
    "\n",
    "> 因為原始文字檔是簡體編碼，我想看繁體，所以就資料轉成繁體再轉utf8．\n",
    "我記得有用一些工具還有簡單的pyhon轉檔，但忘記放哪了．網路上應該可以找到方法，如果你要自己整理資料的話．\n",
    "\n",
    "### 字 vs 詞\n",
    "> 一開始我有jieba做分詞，感覺會讓詞彙表小一點．但單一字跟jieba分詞的詞彙表數量差不多．\n",
    "而且分詞沒辦法做藏頭詩．所以最後就用單一字的方式．\n",
    "\n",
    "### 執行參數\n",
    "\n",
    "* 如果在本機跑的話，folder='data/'．如果在Google Colab跑要記得指向放訓練文字檔的目錄．\n",
    "* window_size = 1．也可以改成其他較大的數字，但predict_next_chars可能需要調整．\n",
    "* one_hot = True．input資料可以指定用one hot vector或是用vocab index．這個例子裡one hot vector的效果比較好．\n",
    "\n",
    "### Model\n",
    "> LSTM -> BatchNormalization -> LSTM -> BatchNormalization -> Dense -> Softmax\n",
    "\n",
    "### 訓練參數\n",
    "\n",
    "* hidden_units = 512．一層有512個LSTM．\n",
    "* lr=0.01．learning rate．\n",
    "* batch_size=128．一個批次的數量．\n",
    "* epochs=200．執行多少次．\n",
    "* 目前loss到3.54就降不下去，或許可以調整model或是訓練參數．但很花時間．\n",
    "\n",
    "### 測試藏頭詩\n",
    "\n",
    "```\n",
    "prefix_ary = ['絕', '妙', '好', '詩']\n",
    "poetry = gen_poetry(predict_modell, prefix_ary, 5)\n",
    "print('5言')\n",
    "print(*poetry, sep='\\n')\n",
    "poetry = gen_poetry(predict_modell, prefix_ary, 7)\n",
    "print('\\n7言')\n",
    "print(*poetry, sep='\\n')\n",
    "```\n",
    "\n",
    "```\n",
    "5言\n",
    "絕頂誰為我\n",
    "妙獨立揚生\n",
    "好以橫波上\n",
    "詩文藻道旁\n",
    "\n",
    "7言\n",
    "絕倫自來歸遠去\n",
    "妙獨釣春草群塔\n",
    "好以菲薄幸李斯\n",
    "詩不得清瑟玳筵\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 2366
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 83794,
     "status": "ok",
     "timestamp": 1534862400810,
     "user": {
      "displayName": "Prins Wu",
      "photoUrl": "//lh5.googleusercontent.com/-e2-hwPV_U-g/AAAAAAAAAAI/AAAAAAAABTU/wj46QM5yHpc/s50-c-k-no/photo.jpg",
      "userId": "111176040429626006479"
     },
     "user_tz": -480
    },
    "id": "boMs0_dBhtzy",
    "outputId": "0cb878eb-c4df-4a9e-b4f0-469a794dbf65"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preconfiguring packages ...\n",
      "Selecting previously unselected package cron.\n",
      "(Reading database ... 18408 files and directories currently installed.)\n",
      "Preparing to unpack .../00-cron_3.0pl1-128ubuntu5_amd64.deb ...\n",
      "Unpacking cron (3.0pl1-128ubuntu5) ...\n",
      "Selecting previously unselected package libapparmor1:amd64.\n",
      "Preparing to unpack .../01-libapparmor1_2.11.0-2ubuntu17.1_amd64.deb ...\n",
      "Unpacking libapparmor1:amd64 (2.11.0-2ubuntu17.1) ...\n",
      "Selecting previously unselected package libdbus-1-3:amd64.\n",
      "Preparing to unpack .../02-libdbus-1-3_1.10.22-1ubuntu1_amd64.deb ...\n",
      "Unpacking libdbus-1-3:amd64 (1.10.22-1ubuntu1) ...\n",
      "Selecting previously unselected package dbus.\n",
      "Preparing to unpack .../03-dbus_1.10.22-1ubuntu1_amd64.deb ...\n",
      "Unpacking dbus (1.10.22-1ubuntu1) ...\n",
      "Selecting previously unselected package dirmngr.\n",
      "Preparing to unpack .../04-dirmngr_2.1.15-1ubuntu8.1_amd64.deb ...\n",
      "Unpacking dirmngr (2.1.15-1ubuntu8.1) ...\n",
      "Selecting previously unselected package distro-info-data.\n",
      "Preparing to unpack .../05-distro-info-data_0.36ubuntu0.2_all.deb ...\n",
      "Unpacking distro-info-data (0.36ubuntu0.2) ...\n",
      "Selecting previously unselected package libkmod2:amd64.\n",
      "Preparing to unpack .../06-libkmod2_24-1ubuntu2_amd64.deb ...\n",
      "Unpacking libkmod2:amd64 (24-1ubuntu2) ...\n",
      "Selecting previously unselected package kmod.\n",
      "Preparing to unpack .../07-kmod_24-1ubuntu2_amd64.deb ...\n",
      "Unpacking kmod (24-1ubuntu2) ...\n",
      "Selecting previously unselected package lsb-release.\n",
      "Preparing to unpack .../08-lsb-release_9.20160110ubuntu5_all.deb ...\n",
      "Unpacking lsb-release (9.20160110ubuntu5) ...\n",
      "Selecting previously unselected package libgirepository-1.0-1:amd64.\n",
      "Preparing to unpack .../09-libgirepository-1.0-1_1.54.1-1_amd64.deb ...\n",
      "Unpacking libgirepository-1.0-1:amd64 (1.54.1-1) ...\n",
      "Selecting previously unselected package gir1.2-glib-2.0:amd64.\n",
      "Preparing to unpack .../10-gir1.2-glib-2.0_1.54.1-1_amd64.deb ...\n",
      "Unpacking gir1.2-glib-2.0:amd64 (1.54.1-1) ...\n",
      "Selecting previously unselected package iso-codes.\n",
      "Preparing to unpack .../11-iso-codes_3.75-1_all.deb ...\n",
      "Unpacking iso-codes (3.75-1) ...\n",
      "Selecting previously unselected package libdbus-glib-1-2:amd64.\n",
      "Preparing to unpack .../12-libdbus-glib-1-2_0.108-2_amd64.deb ...\n",
      "Unpacking libdbus-glib-1-2:amd64 (0.108-2) ...\n",
      "Selecting previously unselected package python-apt-common.\n",
      "Preparing to unpack .../13-python-apt-common_1.4.0~beta3build2_all.deb ...\n",
      "Unpacking python-apt-common (1.4.0~beta3build2) ...\n",
      "Selecting previously unselected package python3-apt.\n",
      "Preparing to unpack .../14-python3-apt_1.4.0~beta3build2_amd64.deb ...\n",
      "Unpacking python3-apt (1.4.0~beta3build2) ...\n",
      "Selecting previously unselected package python3-dbus.\n",
      "Preparing to unpack .../15-python3-dbus_1.2.4-1build3_amd64.deb ...\n",
      "Unpacking python3-dbus (1.2.4-1build3) ...\n",
      "Selecting previously unselected package python3-gi.\n",
      "Preparing to unpack .../16-python3-gi_3.24.1-2build1_amd64.deb ...\n",
      "Unpacking python3-gi (3.24.1-2build1) ...\n",
      "Selecting previously unselected package module-init-tools.\n",
      "Preparing to unpack .../17-module-init-tools_24-1ubuntu2_all.deb ...\n",
      "Unpacking module-init-tools (24-1ubuntu2) ...\n",
      "Selecting previously unselected package python-apt.\n",
      "Preparing to unpack .../18-python-apt_1.4.0~beta3build2_amd64.deb ...\n",
      "Unpacking python-apt (1.4.0~beta3build2) ...\n",
      "Selecting previously unselected package python-pycurl.\n",
      "Preparing to unpack .../19-python-pycurl_7.43.0-2build2_amd64.deb ...\n",
      "Unpacking python-pycurl (7.43.0-2build2) ...\n",
      "Selecting previously unselected package python-software-properties.\n",
      "Preparing to unpack .../20-python-software-properties_0.96.24.17_all.deb ...\n",
      "Unpacking python-software-properties (0.96.24.17) ...\n",
      "Selecting previously unselected package python3-software-properties.\n",
      "Preparing to unpack .../21-python3-software-properties_0.96.24.17_all.deb ...\n",
      "Unpacking python3-software-properties (0.96.24.17) ...\n",
      "Selecting previously unselected package software-properties-common.\n",
      "Preparing to unpack .../22-software-properties-common_0.96.24.17_all.deb ...\n",
      "Unpacking software-properties-common (0.96.24.17) ...\n",
      "Selecting previously unselected package unattended-upgrades.\n",
      "Preparing to unpack .../23-unattended-upgrades_0.98ubuntu1.1_all.deb ...\n",
      "Unpacking unattended-upgrades (0.98ubuntu1.1) ...\n",
      "Setting up python-apt-common (1.4.0~beta3build2) ...\n",
      "Setting up python3-apt (1.4.0~beta3build2) ...\n",
      "Setting up iso-codes (3.75-1) ...\n",
      "Setting up distro-info-data (0.36ubuntu0.2) ...\n",
      "Setting up python-pycurl (7.43.0-2build2) ...\n",
      "Setting up lsb-release (9.20160110ubuntu5) ...\n",
      "Setting up libgirepository-1.0-1:amd64 (1.54.1-1) ...\n",
      "Setting up libkmod2:amd64 (24-1ubuntu2) ...\n",
      "Setting up gir1.2-glib-2.0:amd64 (1.54.1-1) ...\n",
      "Processing triggers for libc-bin (2.26-0ubuntu2.1) ...\n",
      "Setting up libapparmor1:amd64 (2.11.0-2ubuntu17.1) ...\n",
      "Setting up unattended-upgrades (0.98ubuntu1.1) ...\n",
      "\n",
      "Creating config file /etc/apt/apt.conf.d/20auto-upgrades with new version\n",
      "\n",
      "Creating config file /etc/apt/apt.conf.d/50unattended-upgrades with new version\n",
      "invoke-rc.d: could not determine current runlevel\n",
      "invoke-rc.d: policy-rc.d denied execution of start.\n",
      "Setting up dirmngr (2.1.15-1ubuntu8.1) ...\n",
      "Setting up cron (3.0pl1-128ubuntu5) ...\n",
      "Adding group `crontab' (GID 102) ...\n",
      "Done.\n",
      "update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults\n",
      "update-rc.d: warning: stop runlevel arguments (1) do not match cron Default-Stop values (none)\n",
      "invoke-rc.d: could not determine current runlevel\n",
      "invoke-rc.d: policy-rc.d denied execution of start.\n",
      "Setting up libdbus-1-3:amd64 (1.10.22-1ubuntu1) ...\n",
      "Setting up kmod (24-1ubuntu2) ...\n",
      "Setting up libdbus-glib-1-2:amd64 (0.108-2) ...\n",
      "Setting up python3-gi (3.24.1-2build1) ...\n",
      "Setting up module-init-tools (24-1ubuntu2) ...\n",
      "Setting up python3-software-properties (0.96.24.17) ...\n",
      "Setting up dbus (1.10.22-1ubuntu1) ...\n",
      "Setting up python-apt (1.4.0~beta3build2) ...\n",
      "Setting up python3-dbus (1.2.4-1build3) ...\n",
      "Setting up python-software-properties (0.96.24.17) ...\n",
      "Setting up software-properties-common (0.96.24.17) ...\n",
      "Processing triggers for libc-bin (2.26-0ubuntu2.1) ...\n",
      "Processing triggers for dbus (1.10.22-1ubuntu1) ...\n",
      "gpg: keybox '/tmp/tmpsqtwwbcf/pubring.gpg' created\n",
      "gpg: /tmp/tmpsqtwwbcf/trustdb.gpg: trustdb created\n",
      "gpg: key AD5F235DF639B041: public key \"Launchpad PPA for Alessandro Strada\" imported\n",
      "gpg: Total number processed: 1\n",
      "gpg:               imported: 1\n",
      "Warning: apt-key output should not be parsed (stdout is not a terminal)\n",
      "Selecting previously unselected package libfuse2:amd64.\n",
      "(Reading database ... 19816 files and directories currently installed.)\n",
      "Preparing to unpack .../libfuse2_2.9.7-1ubuntu1_amd64.deb ...\n",
      "Unpacking libfuse2:amd64 (2.9.7-1ubuntu1) ...\n",
      "Selecting previously unselected package fuse.\n",
      "Preparing to unpack .../fuse_2.9.7-1ubuntu1_amd64.deb ...\n",
      "Unpacking fuse (2.9.7-1ubuntu1) ...\n",
      "Selecting previously unselected package google-drive-ocamlfuse.\n",
      "Preparing to unpack .../google-drive-ocamlfuse_0.6.21-0ubuntu2_amd64.deb ...\n",
      "Unpacking google-drive-ocamlfuse (0.6.21-0ubuntu2) ...\n",
      "Setting up libfuse2:amd64 (2.9.7-1ubuntu1) ...\n",
      "Processing triggers for libc-bin (2.26-0ubuntu2.1) ...\n",
      "Setting up fuse (2.9.7-1ubuntu1) ...\n",
      "Setting up google-drive-ocamlfuse (0.6.21-0ubuntu2) ...\n",
      "Please, open the following URL in a web browser: https://accounts.google.com/o/oauth2/auth?client_id=32555940559.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&response_type=code&access_type=offline&approval_prompt=force\n",
      "··········\n",
      "Please, open the following URL in a web browser: https://accounts.google.com/o/oauth2/auth?client_id=32555940559.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&response_type=code&access_type=offline&approval_prompt=force\n",
      "Please enter the verification code: Access token retrieved correctly.\n"
     ]
    }
   ],
   "source": [
    "# run it to mount google drive if on Google Colab\n",
    "!apt-get install -y -qq software-properties-common python-software-properties module-init-tools\n",
    "!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null\n",
    "!apt-get update -qq 2>&1 > /dev/null\n",
    "!apt-get -y install -qq google-drive-ocamlfuse fuse\n",
    "from google.colab import auth\n",
    "auth.authenticate_user()\n",
    "from oauth2client.client import GoogleCredentials\n",
    "creds = GoogleCredentials.get_application_default()\n",
    "import getpass\n",
    "!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL\n",
    "vcode = getpass.getpass()\n",
    "!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}\n",
    "\n",
    "!mkdir -p drive\n",
    "!google-drive-ocamlfuse drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VI7HxLZDhtz2"
   },
   "outputs": [],
   "source": [
    "import time\n",
    "from collections import namedtuple\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RlbvH-dDhtz6"
   },
   "outputs": [],
   "source": [
    "class DataGenerator:\n",
    "    \"\"\"\n",
    "    Poetry Data Generator.\n",
    "    \n",
    "    \n",
    "    # Arguments\n",
    "        path: the data file path\n",
    "        single_word: used single word or used vocabulary by jieba. False is default.\n",
    "        remove_words: list of remove words. ex: '。', '，'\n",
    "    \"\"\"\n",
    "    def __init__(self, path, single_word=False, remove_words=[]):\n",
    "        with open(path, 'r') as f:\n",
    "            self.text=f.read()\n",
    "        \n",
    "        if remove_words:\n",
    "            print(\"remove the words:\", remove_words)\n",
    "            self.text = [t for t in self.text if t not in remove_words]\n",
    "        \n",
    "        if single_word:\n",
    "            self.vocab = sorted(set(self.text))\n",
    "            self.vocab_to_int = {c: i for i, c in enumerate(self.vocab)}\n",
    "            self.int_to_vocab = dict(enumerate(self.vocab))\n",
    "            self.encoded = np.array([self.vocab_to_int[c] for c in self.text], dtype=np.int32)\n",
    "        else:\n",
    "            import jieba\n",
    "            seg_list = jieba.lcut(self.text, cut_all=False)\n",
    "            self.vocab = sorted(set(seg_list))\n",
    "            self.vocab_to_int = {c: i for i, c in enumerate(self.vocab)}\n",
    "            self.int_to_vocab = dict(enumerate(self.vocab))\n",
    "            self.encoded = np.array([self.vocab_to_int[c] for c in seg_list if c not in remove_word], dtype=np.int32)\n",
    "    \n",
    "    \"\"\"\n",
    "    To sperate data to input / output by window size and step size.\n",
    "    # Arguments\n",
    "        window_size: input sentence size, word numbers each stentence.\n",
    "        step_size: shift size for next sentence along all text.\n",
    "    # return\n",
    "        inputs: input data list. It is a two-dimensional list (, window_size).\n",
    "        outputs: output data list. It is a two-dimensional list (, 1).\n",
    "    \"\"\"\n",
    "    def window_transform_text(self, window_size, step_size):\n",
    "        total_len = len(self.text)\n",
    "        data_size = int(np.ceil((total_len - window_size) / step_size))\n",
    "        x_start, x_end = 0, (total_len - window_size)\n",
    "        y_start, y_end = window_size, total_len\n",
    "        print(\"x_start:{} x_end:{} y_start:{} y_end:{} total_len:{} data_size:{}\".format(x_start, x_end, y_start, y_end, total_len, data_size))\n",
    "        # containers for input/output pairs\n",
    "        inputs = []\n",
    "        outputs = []\n",
    "        for i in range(x_start, x_end, step_size):\n",
    "            inputs.append(self.text[i : (i + window_size)])\n",
    "        for i in range(y_start, y_end, step_size):\n",
    "            outputs.append(self.text[i])\n",
    "        return inputs,outputs\n",
    "    \n",
    "    # transform character-based input/output into equivalent numerical versions\n",
    "    \"\"\"\n",
    "    To encode data for RNN style.\n",
    "    # Arguments\n",
    "        window_size: input sentence size, word numbers each stentence.\n",
    "        step_size: shift size for next sentence along all text.\n",
    "        one_hot: used one hot vector or not. False is default.\n",
    "    # return\n",
    "        X: input data narray. one hot is (, window_size, number of vocabulary), int is (, windows_size, 1).\n",
    "        y: output data narray. y is one hot (, number of vocabulary).\n",
    "    \"\"\"\n",
    "    def encode_io_pairs(self, window_size, step_size, one_hot=False):\n",
    "        # number of vocabulary\n",
    "        num_vocab = len(self.vocab)\n",
    "\n",
    "        # cut up text into character input/output pairs\n",
    "        inputs, outputs = self.window_transform_text(window_size, step_size)\n",
    "\n",
    "        # create empty vessels for one-hot encoded input/output\n",
    "        if one_hot:\n",
    "            X = np.zeros((len(inputs), window_size, num_vocab), dtype=np.bool)\n",
    "            for i, sentence in enumerate(inputs):\n",
    "                for t, char in enumerate(sentence):\n",
    "                    X[i, t, self.vocab_to_int[char]] = 1\n",
    "        else:\n",
    "            inputs_i = []\n",
    "            for data in inputs:\n",
    "                inputs_i.append([self.vocab_to_int[char] for char in data])\n",
    "            X = np.reshape(inputs_i, ((len(inputs), window_size, 1)))\n",
    "            X = X / float(len(self.vocab))\n",
    "            \n",
    "        y = np.zeros((len(inputs), num_vocab), dtype=np.bool)\n",
    "        for i, sentence in enumerate(inputs):\n",
    "            y[i, self.vocab_to_int[outputs[i]]] = 1\n",
    "\n",
    "        return X,y\n",
    "    \n",
    "    \"\"\"\n",
    "    To predict next char.\n",
    "    # Arguments\n",
    "        model: Instance of `Model`\n",
    "        input_chars: input chars list\n",
    "        num_to_predict: number of chars to predict\n",
    "        window_size: input sentence size, word numbers each stentence.\n",
    "        one_hot: used one hot vector or not. False is default.\n",
    "    # return\n",
    "        predicted_chars: input_chars + [predict chars]\n",
    "    \"\"\"\n",
    "    def predict_next_chars(self, model, input_chars, num_to_predict, window_size, one_hot=False):\n",
    "        inputs = input_chars[:]\n",
    "        # create output\n",
    "        predicted_chars = ''.join(inputs)\n",
    "        # number of vocabulary\n",
    "        num_vocab = len(self.vocab)\n",
    "        for i in range(num_to_predict):\n",
    "            if one_hot:\n",
    "                # convert this round's predicted characters to numerical input\n",
    "                x_test = np.zeros((1, window_size, num_vocab))\n",
    "                for t, char in enumerate(inputs):\n",
    "                    x_test[0, t, self.vocab_to_int[char]] = 1.\n",
    "            else:\n",
    "                x_test = np.zeros((1, window_size, 1))\n",
    "                for t, char in enumerate(inputs):\n",
    "                    x_test[0, t, 0] = self.vocab_to_int[char]\n",
    "\n",
    "            # make this round's prediction\n",
    "            test_predict = model.predict(x_test,verbose = 0)[0]\n",
    "            r = self.pick_top_n(test_predict, num_vocab)\n",
    "            d = self.int_to_vocab[r] \n",
    "\n",
    "            # update predicted_chars and input\n",
    "            predicted_chars += d\n",
    "            inputs += d\n",
    "            inputs = inputs[1:]\n",
    "        return predicted_chars\n",
    "      \n",
    "    \"\"\"\n",
    "    To pick up one of top n.\n",
    "    \"\"\"\n",
    "    def pick_top_n(self, preds, vocab_size, top_n=5):\n",
    "        p = np.squeeze(preds)\n",
    "        p[np.argsort(p)[:-top_n]] = 0\n",
    "        p = p / np.sum(p)\n",
    "        c = np.random.choice(vocab_size, 1, p=p)[0]\n",
    "        return c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3690,
     "status": "ok",
     "timestamp": 1534862411322,
     "user": {
      "displayName": "Prins Wu",
      "photoUrl": "//lh5.googleusercontent.com/-e2-hwPV_U-g/AAAAAAAAAAI/AAAAAAAABTU/wj46QM5yHpc/s50-c-k-no/photo.jpg",
      "userId": "111176040429626006479"
     },
     "user_tz": -480
    },
    "id": "dS37It9lhtz8",
    "outputId": "b6e4940e-2316-43de-b122-f0a5717d791f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "remove the words: ['\\n', '。', '，', '！', ' ']\n",
      "2449\n",
      "['一', '丁', '七', '丈', '三', '上', '下', '不', '且', '世', '丘', '丞', '並', '中', '丹', '主', '乃', '久', '之', '乍', '乎', '乖', '乘', '乙', '九', '乞', '也', '乳', '乾', '亂', '了', '予', '事', '二', '五', '井', '亞', '亡', '交', '亦', '京', '亭', '人', '仁', '仆', '今', '仍', '他', '仗', '仙', '仞', '代', '令', '以', '仰', '仲', '任', '伊', '伏', '伐', '休', '伯', '估', '伴', '伶', '伺', '似', '但', '佇', '位', '低', '住', '何', '佗', '余', '佛', '作', '佩', '佳', '使', '來', '侍', '供', '依', '侯', '侵', '便', '促', '俊', '俗', '俜', '保', '俠', '信', '修', '俯', '俱', '俸', '倍', '倏']\n",
      "['絕', '代', '有', '佳', '人', '幽', '居', '在', '空', '谷', '自', '雲', '良', '家', '子', '零', '落', '依', '草', '木', '關', '中', '昔', '喪', '亂', '兄', '弟', '遭', '殺', '戮', '官', '高', '何', '足', '論', '不', '得', '收', '骨', '肉', '世', '情', '惡', '衰', '歇', '萬', '事', '隨', '轉', '燭', '夫', '婿', '輕', '薄', '兒', '新', '人', '美', '如', '玉', '合', '昏', '尚', '知', '時', '鴛', '鴦', '不', '獨', '宿', '但', '見', '新', '人', '笑', '那', '聞', '舊', '人', '哭', '在', '山', '泉', '水', '清', '出', '山', '泉', '水', '濁', '侍', '婢', '賣', '珠', '回', '牽', '蘿', '補', '茅', '屋']\n",
      "18994\n"
     ]
    }
   ],
   "source": [
    "remove_words = ['\\n', '。', '，', '！', ' ']\n",
    "# folder = 'data/'\n",
    "folder = 'drive/data/3390_utf8_big5/'\n",
    "gen = DataGenerator(folder + '3390_utf8_big5.txt', single_word=True, remove_words=remove_words)\n",
    "print(len(gen.vocab)) # vocabulary size\n",
    "print(gen.vocab[:100]) # vocabulary\n",
    "print(gen.text[:100])\n",
    "print(len(gen.encoded)) # encod all text to int\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 833,
     "status": "ok",
     "timestamp": 1534862412175,
     "user": {
      "displayName": "Prins Wu",
      "photoUrl": "//lh5.googleusercontent.com/-e2-hwPV_U-g/AAAAAAAAAAI/AAAAAAAABTU/wj46QM5yHpc/s50-c-k-no/photo.jpg",
      "userId": "111176040429626006479"
     },
     "user_tz": -480
    },
    "id": "TSSn7i9hht0A",
    "outputId": "b8ac0977-1a1e-41f4-b898-19a8bda9ad82"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_start:0 x_end:18993 y_start:1 y_end:18994 total_len:18994 data_size:18993\n"
     ]
    }
   ],
   "source": [
    "# run your text window-ing function \n",
    "window_size = 1\n",
    "step_size = 1\n",
    "inputs, outputs = gen.window_transform_text(window_size,step_size)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 725,
     "status": "ok",
     "timestamp": 1534862412986,
     "user": {
      "displayName": "Prins Wu",
      "photoUrl": "//lh5.googleusercontent.com/-e2-hwPV_U-g/AAAAAAAAAAI/AAAAAAAABTU/wj46QM5yHpc/s50-c-k-no/photo.jpg",
      "userId": "111176040429626006479"
     },
     "user_tz": -480
    },
    "id": "dHOHlKzbht0C",
    "outputId": "50eb97b9-cd3a-4e98-9b31-de9ffd284cee"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input =  ['絕']\n",
      "output =  代\n",
      "--------------\n",
      "input =  ['代']\n",
      "output =  有\n"
     ]
    }
   ],
   "source": [
    "# print out a few of the input/output pairs to verify that we've made the right kind of stuff to learn from\n",
    "print('input = ', inputs[0])\n",
    "print('output = ', outputs[0])\n",
    "print('--------------')\n",
    "print('input = ', inputs[1])\n",
    "print('output = ', outputs[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1046,
     "status": "ok",
     "timestamp": 1534862414113,
     "user": {
      "displayName": "Prins Wu",
      "photoUrl": "//lh5.googleusercontent.com/-e2-hwPV_U-g/AAAAAAAAAAI/AAAAAAAABTU/wj46QM5yHpc/s50-c-k-no/photo.jpg",
      "userId": "111176040429626006479"
     },
     "user_tz": -480
    },
    "id": "hy7zuR-Hht0F",
    "outputId": "5fec2f21-4228-4632-f5fb-b7771f592e22"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_start:0 x_end:18993 y_start:1 y_end:18994 total_len:18994 data_size:18993\n"
     ]
    }
   ],
   "source": [
    "# window_size = 50\n",
    "# step_size = 10\n",
    "one_hot = True\n",
    "X,y = gen.encode_io_pairs(window_size,step_size, one_hot=one_hot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 726,
     "status": "ok",
     "timestamp": 1534862414907,
     "user": {
      "displayName": "Prins Wu",
      "photoUrl": "//lh5.googleusercontent.com/-e2-hwPV_U-g/AAAAAAAAAAI/AAAAAAAABTU/wj46QM5yHpc/s50-c-k-no/photo.jpg",
      "userId": "111176040429626006479"
     },
     "user_tz": -480
    },
    "id": "B-7lGJRBht0H",
    "outputId": "d2157933-5587-4162-e1b2-b9b474f70235"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(18993, 1, 2449)\n",
      "[[False False False ... False False False]]\n"
     ]
    }
   ],
   "source": [
    "print(X.shape)\n",
    "print(X[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 717,
     "status": "ok",
     "timestamp": 1534862415702,
     "user": {
      "displayName": "Prins Wu",
      "photoUrl": "//lh5.googleusercontent.com/-e2-hwPV_U-g/AAAAAAAAAAI/AAAAAAAABTU/wj46QM5yHpc/s50-c-k-no/photo.jpg",
      "userId": "111176040429626006479"
     },
     "user_tz": -480
    },
    "id": "nH5_7HLpht0M",
    "outputId": "adece2aa-0fc0-41eb-e7b5-e1ac3a5b1e3e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation, LSTM, BatchNormalization, Input\n",
    "from keras.optimizers import RMSprop\n",
    "from keras.utils.data_utils import get_file\n",
    "import keras\n",
    "import random\n",
    "\n",
    "def build_model(hidden_units, window_size, features, output_dim):\n",
    "    model = Sequential()\n",
    "    # two LSTM layers\n",
    "    model.add(LSTM(hidden_units, input_shape=(window_size, features), return_sequences=True, name='lstm_1'))\n",
    "    model.add(BatchNormalization(name='bn_rnn_1'))\n",
    "    model.add(LSTM(hidden_units, name='lstm_2'))\n",
    "    model.add(BatchNormalization(name='bn_rnn_2'))\n",
    "    model.add(Dense(output_dim, name='dense_1'))\n",
    "    model.add(Activation('softmax', name='softmax_1'))\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 340
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4093,
     "status": "ok",
     "timestamp": 1534862419864,
     "user": {
      "displayName": "Prins Wu",
      "photoUrl": "//lh5.googleusercontent.com/-e2-hwPV_U-g/AAAAAAAAAAI/AAAAAAAABTU/wj46QM5yHpc/s50-c-k-no/photo.jpg",
      "userId": "111176040429626006479"
     },
     "user_tz": -480
    },
    "id": "Vtf1gBxSht0P",
    "outputId": "d824122b-b764-4e6b-e333-6e62a1f6d7ad"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm_1 (LSTM)                (None, 1, 512)            6066176   \n",
      "_________________________________________________________________\n",
      "bn_rnn_1 (BatchNormalization (None, 1, 512)            2048      \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 512)               2099200   \n",
      "_________________________________________________________________\n",
      "bn_rnn_2 (BatchNormalization (None, 512)               2048      \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 2449)              1256337   \n",
      "_________________________________________________________________\n",
      "softmax_1 (Activation)       (None, 2449)              0         \n",
      "=================================================================\n",
      "Total params: 9,425,809\n",
      "Trainable params: 9,423,761\n",
      "Non-trainable params: 2,048\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "hidden_units = 512\n",
    "features = (len(gen.vocab) if one_hot else 1) # 1 for vocab_for_int, len(gen.vocab) for one hot encoding\n",
    "output_dim = len(gen.vocab)\n",
    "model = build_model(hidden_units, window_size, features, output_dim)\n",
    "# initialize optimizer\n",
    "# optimizer = keras.optimizers.RMSprop(lr=0.03, rho=0.9, epsilon=1e-08, decay=0.0)\n",
    "optimizer = keras.optimizers.Adam(lr=0.01)\n",
    "\n",
    "# compile model --> make sure initialized optimizer and callbacks - as defined above - are used\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 6817
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1497135,
     "status": "ok",
     "timestamp": 1534863917036,
     "user": {
      "displayName": "Prins Wu",
      "photoUrl": "//lh5.googleusercontent.com/-e2-hwPV_U-g/AAAAAAAAAAI/AAAAAAAABTU/wj46QM5yHpc/s50-c-k-no/photo.jpg",
      "userId": "111176040429626006479"
     },
     "user_tz": -480
    },
    "id": "CRL2y7Syht0S",
    "outputId": "e8eb58a9-e91f-44d3-ef95-d41c7e3df7ca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/200\n",
      "18993/18993 [==============================] - 14s 714us/step - loss: 7.4996\n",
      "Epoch 2/200\n",
      "18993/18993 [==============================] - 7s 355us/step - loss: 6.7617\n",
      "Epoch 3/200\n",
      "18993/18993 [==============================] - 7s 355us/step - loss: 6.4618\n",
      "Epoch 4/200\n",
      "18993/18993 [==============================] - 7s 348us/step - loss: 6.0819\n",
      "Epoch 5/200\n",
      "18993/18993 [==============================] - 8s 409us/step - loss: 5.7470\n",
      "Epoch 6/200\n",
      "18993/18993 [==============================] - 7s 374us/step - loss: 5.5344\n",
      "Epoch 7/200\n",
      "18993/18993 [==============================] - 7s 369us/step - loss: 5.3530\n",
      "Epoch 8/200\n",
      "18993/18993 [==============================] - 7s 369us/step - loss: 5.1784\n",
      "Epoch 9/200\n",
      "18993/18993 [==============================] - 7s 373us/step - loss: 5.0620\n",
      "Epoch 10/200\n",
      "18993/18993 [==============================] - 7s 369us/step - loss: 4.9611\n",
      "Epoch 11/200\n",
      "18993/18993 [==============================] - 7s 352us/step - loss: 4.8755\n",
      "Epoch 12/200\n",
      "18993/18993 [==============================] - 7s 364us/step - loss: 4.7954\n",
      "Epoch 13/200\n",
      "18993/18993 [==============================] - 7s 393us/step - loss: 4.7501\n",
      "Epoch 14/200\n",
      "18993/18993 [==============================] - 7s 365us/step - loss: 4.7172\n",
      "Epoch 15/200\n",
      "18993/18993 [==============================] - 7s 372us/step - loss: 4.6807\n",
      "Epoch 16/200\n",
      "18993/18993 [==============================] - 7s 389us/step - loss: 4.6149\n",
      "Epoch 17/200\n",
      "18993/18993 [==============================] - 7s 378us/step - loss: 4.5794\n",
      "Epoch 18/200\n",
      "18993/18993 [==============================] - 7s 388us/step - loss: 4.5323\n",
      "Epoch 19/200\n",
      "18993/18993 [==============================] - 7s 389us/step - loss: 4.5082\n",
      "Epoch 20/200\n",
      "18993/18993 [==============================] - 8s 395us/step - loss: 4.4975\n",
      "Epoch 21/200\n",
      "18993/18993 [==============================] - 7s 378us/step - loss: 4.4773\n",
      "Epoch 22/200\n",
      "18993/18993 [==============================] - 7s 379us/step - loss: 4.4434\n",
      "Epoch 23/200\n",
      "18993/18993 [==============================] - 7s 385us/step - loss: 4.4191\n",
      "Epoch 24/200\n",
      "18993/18993 [==============================] - 7s 363us/step - loss: 4.3955\n",
      "Epoch 25/200\n",
      "18993/18993 [==============================] - 7s 376us/step - loss: 4.3597\n",
      "Epoch 26/200\n",
      "18993/18993 [==============================] - 7s 359us/step - loss: 4.3365\n",
      "Epoch 27/200\n",
      "18993/18993 [==============================] - 7s 370us/step - loss: 4.3106\n",
      "Epoch 28/200\n",
      "18993/18993 [==============================] - 7s 372us/step - loss: 4.2610\n",
      "Epoch 29/200\n",
      "18993/18993 [==============================] - 7s 374us/step - loss: 4.2431\n",
      "Epoch 30/200\n",
      "18993/18993 [==============================] - 7s 372us/step - loss: 4.2290\n",
      "Epoch 31/200\n",
      "18993/18993 [==============================] - 7s 379us/step - loss: 4.2464\n",
      "Epoch 32/200\n",
      "18993/18993 [==============================] - 7s 382us/step - loss: 4.2269\n",
      "Epoch 33/200\n",
      "18993/18993 [==============================] - 7s 385us/step - loss: 4.1937\n",
      "Epoch 34/200\n",
      "18993/18993 [==============================] - 7s 388us/step - loss: 4.1881\n",
      "Epoch 35/200\n",
      "18993/18993 [==============================] - 8s 399us/step - loss: 4.1689\n",
      "Epoch 36/200\n",
      "18993/18993 [==============================] - 8s 411us/step - loss: 4.1508\n",
      "Epoch 37/200\n",
      "18993/18993 [==============================] - 8s 415us/step - loss: 4.1640\n",
      "Epoch 38/200\n",
      "18993/18993 [==============================] - 8s 414us/step - loss: 4.1469\n",
      "Epoch 39/200\n",
      "18993/18993 [==============================] - 8s 415us/step - loss: 4.1183\n",
      "Epoch 40/200\n",
      "18993/18993 [==============================] - 7s 388us/step - loss: 4.0969\n",
      "Epoch 41/200\n",
      "18993/18993 [==============================] - 7s 377us/step - loss: 4.0759\n",
      "Epoch 42/200\n",
      "18993/18993 [==============================] - 7s 387us/step - loss: 4.0624\n",
      "Epoch 43/200\n",
      "18993/18993 [==============================] - 7s 376us/step - loss: 4.0549\n",
      "Epoch 44/200\n",
      "18993/18993 [==============================] - 7s 367us/step - loss: 4.0321\n",
      "Epoch 45/200\n",
      "18993/18993 [==============================] - 7s 379us/step - loss: 4.0145\n",
      "Epoch 46/200\n",
      "18993/18993 [==============================] - 7s 383us/step - loss: 4.0335\n",
      "Epoch 47/200\n",
      "18993/18993 [==============================] - 7s 366us/step - loss: 4.0190\n",
      "Epoch 48/200\n",
      "18993/18993 [==============================] - 7s 369us/step - loss: 3.9988\n",
      "Epoch 49/200\n",
      "18993/18993 [==============================] - 7s 371us/step - loss: 3.9907\n",
      "Epoch 50/200\n",
      "18993/18993 [==============================] - 7s 357us/step - loss: 3.9823\n",
      "Epoch 51/200\n",
      "18993/18993 [==============================] - 7s 363us/step - loss: 3.9509\n",
      "Epoch 52/200\n",
      "18993/18993 [==============================] - 7s 381us/step - loss: 3.9700\n",
      "Epoch 53/200\n",
      "18993/18993 [==============================] - 7s 375us/step - loss: 3.9511\n",
      "Epoch 54/200\n",
      "18993/18993 [==============================] - 7s 380us/step - loss: 3.9340\n",
      "Epoch 55/200\n",
      "18993/18993 [==============================] - 7s 383us/step - loss: 3.9477\n",
      "Epoch 56/200\n",
      "18993/18993 [==============================] - 7s 379us/step - loss: 3.9406\n",
      "Epoch 57/200\n",
      "18993/18993 [==============================] - 7s 363us/step - loss: 3.9201\n",
      "Epoch 58/200\n",
      "18993/18993 [==============================] - 7s 356us/step - loss: 3.9062\n",
      "Epoch 59/200\n",
      "18993/18993 [==============================] - 7s 376us/step - loss: 3.9090\n",
      "Epoch 60/200\n",
      "18993/18993 [==============================] - 7s 365us/step - loss: 3.9115\n",
      "Epoch 61/200\n",
      "18993/18993 [==============================] - 7s 376us/step - loss: 3.8935\n",
      "Epoch 62/200\n",
      "18993/18993 [==============================] - 7s 386us/step - loss: 3.8820\n",
      "Epoch 63/200\n",
      "18993/18993 [==============================] - 7s 371us/step - loss: 3.8838\n",
      "Epoch 64/200\n",
      "18993/18993 [==============================] - 7s 368us/step - loss: 3.8639\n",
      "Epoch 65/200\n",
      "18993/18993 [==============================] - 7s 380us/step - loss: 3.8851\n",
      "Epoch 66/200\n",
      "18993/18993 [==============================] - 7s 358us/step - loss: 3.8702\n",
      "Epoch 67/200\n",
      "18993/18993 [==============================] - 7s 359us/step - loss: 3.8567\n",
      "Epoch 68/200\n",
      "18993/18993 [==============================] - 7s 378us/step - loss: 3.8577\n",
      "Epoch 69/200\n",
      "18993/18993 [==============================] - 6s 336us/step - loss: 3.8603\n",
      "Epoch 70/200\n",
      "18993/18993 [==============================] - 7s 351us/step - loss: 3.8545\n",
      "Epoch 71/200\n",
      "18993/18993 [==============================] - 7s 355us/step - loss: 3.8305\n",
      "Epoch 72/200\n",
      "18993/18993 [==============================] - 7s 345us/step - loss: 3.8418\n",
      "Epoch 73/200\n",
      "18993/18993 [==============================] - 7s 352us/step - loss: 3.8356\n",
      "Epoch 74/200\n",
      "18993/18993 [==============================] - 7s 380us/step - loss: 3.8318\n",
      "Epoch 75/200\n",
      "18993/18993 [==============================] - 7s 380us/step - loss: 3.8001\n",
      "Epoch 76/200\n",
      "18993/18993 [==============================] - 7s 364us/step - loss: 3.8124\n",
      "Epoch 77/200\n",
      "18993/18993 [==============================] - 6s 341us/step - loss: 3.7973\n",
      "Epoch 78/200\n",
      "18993/18993 [==============================] - 7s 351us/step - loss: 3.8064\n",
      "Epoch 79/200\n",
      "18993/18993 [==============================] - 6s 337us/step - loss: 3.8160\n",
      "Epoch 80/200\n",
      "18993/18993 [==============================] - 7s 364us/step - loss: 3.7965\n",
      "Epoch 81/200\n",
      "18993/18993 [==============================] - 7s 346us/step - loss: 3.8098\n",
      "Epoch 82/200\n",
      "18993/18993 [==============================] - 7s 368us/step - loss: 3.8198\n",
      "Epoch 83/200\n",
      "18993/18993 [==============================] - 7s 365us/step - loss: 3.8315\n",
      "Epoch 84/200\n",
      "18993/18993 [==============================] - 7s 365us/step - loss: 3.7966\n",
      "Epoch 85/200\n",
      "18993/18993 [==============================] - 7s 360us/step - loss: 3.7792\n",
      "Epoch 86/200\n",
      "18993/18993 [==============================] - 6s 318us/step - loss: 3.7778\n",
      "Epoch 87/200\n",
      "18993/18993 [==============================] - 6s 332us/step - loss: 3.7782\n",
      "Epoch 88/200\n",
      "18993/18993 [==============================] - 7s 353us/step - loss: 3.7810\n",
      "Epoch 89/200\n",
      "18993/18993 [==============================] - 7s 362us/step - loss: 3.7602\n",
      "Epoch 90/200\n",
      "18993/18993 [==============================] - 7s 369us/step - loss: 3.7520\n",
      "Epoch 91/200\n",
      "18993/18993 [==============================] - 7s 363us/step - loss: 3.7489\n",
      "Epoch 92/200\n",
      "18993/18993 [==============================] - 8s 398us/step - loss: 3.7327\n",
      "Epoch 93/200\n",
      "18993/18993 [==============================] - 7s 364us/step - loss: 3.7191\n",
      "Epoch 94/200\n",
      "18993/18993 [==============================] - 7s 357us/step - loss: 3.7124\n",
      "Epoch 95/200\n",
      "18993/18993 [==============================] - 7s 375us/step - loss: 3.7331\n",
      "Epoch 96/200\n",
      "18993/18993 [==============================] - 7s 352us/step - loss: 3.7062\n",
      "Epoch 97/200\n",
      "18993/18993 [==============================] - 7s 349us/step - loss: 3.7155\n",
      "Epoch 98/200\n",
      "18993/18993 [==============================] - 7s 370us/step - loss: 3.7139\n",
      "Epoch 99/200\n",
      "18993/18993 [==============================] - 7s 385us/step - loss: 3.7123\n",
      "Epoch 100/200\n",
      "18993/18993 [==============================] - 8s 410us/step - loss: 3.7238\n",
      "Epoch 101/200\n",
      "18993/18993 [==============================] - 7s 387us/step - loss: 3.7306\n",
      "Epoch 102/200\n",
      "18993/18993 [==============================] - 7s 367us/step - loss: 3.7132\n",
      "Epoch 103/200\n",
      "18993/18993 [==============================] - 7s 377us/step - loss: 3.7111\n",
      "Epoch 104/200\n",
      "18993/18993 [==============================] - 8s 398us/step - loss: 3.7002\n",
      "Epoch 105/200\n",
      "18993/18993 [==============================] - 8s 396us/step - loss: 3.6901\n",
      "Epoch 106/200\n",
      "18993/18993 [==============================] - 7s 365us/step - loss: 3.6867\n",
      "Epoch 107/200\n",
      "18993/18993 [==============================] - 7s 384us/step - loss: 3.6769\n",
      "Epoch 108/200\n",
      "18993/18993 [==============================] - 7s 380us/step - loss: 3.6828\n",
      "Epoch 109/200\n",
      "18993/18993 [==============================] - 7s 357us/step - loss: 3.6748\n",
      "Epoch 110/200\n",
      "18993/18993 [==============================] - 7s 347us/step - loss: 3.7031\n",
      "Epoch 111/200\n",
      "18993/18993 [==============================] - 7s 366us/step - loss: 3.7006\n",
      "Epoch 112/200\n",
      "18993/18993 [==============================] - 7s 363us/step - loss: 3.6824\n",
      "Epoch 113/200\n",
      "18993/18993 [==============================] - 7s 351us/step - loss: 3.6787\n",
      "Epoch 114/200\n",
      "18993/18993 [==============================] - 7s 360us/step - loss: 3.6782\n",
      "Epoch 115/200\n",
      "18993/18993 [==============================] - 6s 334us/step - loss: 3.6704\n",
      "Epoch 116/200\n",
      "18993/18993 [==============================] - 7s 377us/step - loss: 3.6652\n",
      "Epoch 117/200\n",
      "18993/18993 [==============================] - 7s 388us/step - loss: 3.6609\n",
      "Epoch 118/200\n",
      "18993/18993 [==============================] - 7s 379us/step - loss: 3.6688\n",
      "Epoch 119/200\n",
      "18993/18993 [==============================] - 7s 374us/step - loss: 3.6591\n",
      "Epoch 120/200\n",
      "18993/18993 [==============================] - 7s 382us/step - loss: 3.6507\n",
      "Epoch 121/200\n",
      "18993/18993 [==============================] - 8s 405us/step - loss: 3.6453\n",
      "Epoch 122/200\n",
      "18993/18993 [==============================] - 8s 417us/step - loss: 3.6494\n",
      "Epoch 123/200\n",
      "18993/18993 [==============================] - 7s 393us/step - loss: 3.6596\n",
      "Epoch 124/200\n",
      "18993/18993 [==============================] - 7s 393us/step - loss: 3.6446\n",
      "Epoch 125/200\n",
      "18993/18993 [==============================] - 8s 418us/step - loss: 3.6318\n",
      "Epoch 126/200\n",
      "18993/18993 [==============================] - 8s 416us/step - loss: 3.6362\n",
      "Epoch 127/200\n",
      "18993/18993 [==============================] - 8s 424us/step - loss: 3.6400\n",
      "Epoch 128/200\n",
      "18993/18993 [==============================] - 8s 404us/step - loss: 3.6301\n",
      "Epoch 129/200\n",
      "18993/18993 [==============================] - 7s 391us/step - loss: 3.6351\n",
      "Epoch 130/200\n",
      "18993/18993 [==============================] - 8s 410us/step - loss: 3.6359\n",
      "Epoch 131/200\n",
      "18993/18993 [==============================] - 7s 384us/step - loss: 3.6204\n",
      "Epoch 132/200\n",
      "18993/18993 [==============================] - 7s 377us/step - loss: 3.6332\n",
      "Epoch 133/200\n",
      "18993/18993 [==============================] - 7s 359us/step - loss: 3.6189\n",
      "Epoch 134/200\n",
      "18993/18993 [==============================] - 7s 368us/step - loss: 3.6222\n",
      "Epoch 135/200\n",
      "18993/18993 [==============================] - 7s 346us/step - loss: 3.6319\n",
      "Epoch 136/200\n",
      "18993/18993 [==============================] - 7s 347us/step - loss: 3.6274\n",
      "Epoch 137/200\n",
      "18993/18993 [==============================] - 7s 381us/step - loss: 3.6405\n",
      "Epoch 138/200\n",
      "18993/18993 [==============================] - 7s 375us/step - loss: 3.6182\n",
      "Epoch 139/200\n",
      "18993/18993 [==============================] - 7s 382us/step - loss: 3.5999\n",
      "Epoch 140/200\n",
      "18993/18993 [==============================] - 7s 384us/step - loss: 3.5991\n",
      "Epoch 141/200\n",
      "18993/18993 [==============================] - 8s 405us/step - loss: 3.5959\n",
      "Epoch 142/200\n",
      "18993/18993 [==============================] - 8s 404us/step - loss: 3.6042\n",
      "Epoch 143/200\n",
      "18993/18993 [==============================] - 8s 413us/step - loss: 3.6089\n",
      "Epoch 144/200\n",
      "18993/18993 [==============================] - 7s 388us/step - loss: 3.5975\n",
      "Epoch 145/200\n",
      "18993/18993 [==============================] - 8s 402us/step - loss: 3.6030\n",
      "Epoch 146/200\n",
      "18993/18993 [==============================] - 7s 394us/step - loss: 3.6018\n",
      "Epoch 147/200\n",
      "18993/18993 [==============================] - 8s 406us/step - loss: 3.5910\n",
      "Epoch 148/200\n",
      "18993/18993 [==============================] - 8s 395us/step - loss: 3.5898\n",
      "Epoch 149/200\n",
      "18993/18993 [==============================] - 8s 411us/step - loss: 3.6011\n",
      "Epoch 150/200\n",
      "18993/18993 [==============================] - 8s 415us/step - loss: 3.5860\n",
      "Epoch 151/200\n",
      "18993/18993 [==============================] - 8s 439us/step - loss: 3.6004\n",
      "Epoch 152/200\n",
      "18993/18993 [==============================] - 8s 432us/step - loss: 3.5948\n",
      "Epoch 153/200\n",
      "18993/18993 [==============================] - 8s 417us/step - loss: 3.6007\n",
      "Epoch 154/200\n",
      "18993/18993 [==============================] - 7s 381us/step - loss: 3.5896\n",
      "Epoch 155/200\n",
      "18993/18993 [==============================] - 7s 390us/step - loss: 3.6005\n",
      "Epoch 156/200\n",
      "18993/18993 [==============================] - 8s 411us/step - loss: 3.6035\n",
      "Epoch 157/200\n",
      "18993/18993 [==============================] - 7s 391us/step - loss: 3.5953\n",
      "Epoch 158/200\n",
      "18993/18993 [==============================] - 8s 406us/step - loss: 3.5941\n",
      "Epoch 159/200\n",
      "18993/18993 [==============================] - 8s 441us/step - loss: 3.5797\n",
      "Epoch 160/200\n",
      "18993/18993 [==============================] - 8s 428us/step - loss: 3.5997\n",
      "Epoch 161/200\n",
      "18993/18993 [==============================] - 8s 421us/step - loss: 3.5812\n",
      "Epoch 162/200\n",
      "18993/18993 [==============================] - 8s 418us/step - loss: 3.5693\n",
      "Epoch 163/200\n",
      "18993/18993 [==============================] - 7s 389us/step - loss: 3.5758\n",
      "Epoch 164/200\n",
      "18993/18993 [==============================] - 7s 382us/step - loss: 3.5690\n",
      "Epoch 165/200\n",
      "18993/18993 [==============================] - 8s 399us/step - loss: 3.5729\n",
      "Epoch 166/200\n",
      "18993/18993 [==============================] - 7s 390us/step - loss: 3.5509\n",
      "Epoch 167/200\n",
      "18993/18993 [==============================] - 8s 411us/step - loss: 3.5853\n",
      "Epoch 168/200\n",
      "18993/18993 [==============================] - 9s 451us/step - loss: 3.5844\n",
      "Epoch 169/200\n",
      "18993/18993 [==============================] - 9s 458us/step - loss: 3.5677\n",
      "Epoch 170/200\n",
      "18993/18993 [==============================] - 9s 465us/step - loss: 3.5739\n",
      "Epoch 171/200\n",
      "18993/18993 [==============================] - 8s 440us/step - loss: 3.5582\n",
      "Epoch 172/200\n",
      "18993/18993 [==============================] - 8s 416us/step - loss: 3.5645\n",
      "Epoch 173/200\n",
      "18993/18993 [==============================] - 8s 425us/step - loss: 3.5591\n",
      "Epoch 174/200\n",
      "18993/18993 [==============================] - 8s 414us/step - loss: 3.5783\n",
      "Epoch 175/200\n",
      "18993/18993 [==============================] - 8s 430us/step - loss: 3.5705\n",
      "Epoch 176/200\n",
      "18993/18993 [==============================] - 9s 454us/step - loss: 3.5573\n",
      "Epoch 177/200\n",
      "18993/18993 [==============================] - 8s 440us/step - loss: 3.5596\n",
      "Epoch 178/200\n",
      "18993/18993 [==============================] - 8s 427us/step - loss: 3.5555\n",
      "Epoch 179/200\n",
      "18993/18993 [==============================] - 8s 435us/step - loss: 3.5456\n",
      "Epoch 180/200\n",
      "18993/18993 [==============================] - 8s 432us/step - loss: 3.5411\n",
      "Epoch 181/200\n",
      "18993/18993 [==============================] - 8s 427us/step - loss: 3.5432\n",
      "Epoch 182/200\n",
      "18993/18993 [==============================] - 8s 397us/step - loss: 3.5482\n",
      "Epoch 183/200\n",
      "18993/18993 [==============================] - 8s 421us/step - loss: 3.5456\n",
      "Epoch 184/200\n",
      "18993/18993 [==============================] - 8s 417us/step - loss: 3.5376\n",
      "Epoch 185/200\n",
      "18993/18993 [==============================] - 8s 426us/step - loss: 3.5681\n",
      "Epoch 186/200\n",
      "18993/18993 [==============================] - 8s 445us/step - loss: 3.5566\n",
      "Epoch 187/200\n",
      "18993/18993 [==============================] - 9s 478us/step - loss: 3.5383\n",
      "Epoch 188/200\n",
      "18993/18993 [==============================] - 9s 487us/step - loss: 3.5505\n",
      "Epoch 189/200\n",
      "18993/18993 [==============================] - 8s 444us/step - loss: 3.5630\n",
      "Epoch 190/200\n",
      "18993/18993 [==============================] - 8s 431us/step - loss: 3.5636\n",
      "Epoch 191/200\n",
      "18993/18993 [==============================] - 8s 431us/step - loss: 3.5629\n",
      "Epoch 192/200\n",
      "18993/18993 [==============================] - 9s 452us/step - loss: 3.5615\n",
      "Epoch 193/200\n",
      "18993/18993 [==============================] - 8s 407us/step - loss: 3.5506\n",
      "Epoch 194/200\n",
      "18993/18993 [==============================] - 8s 438us/step - loss: 3.5480\n",
      "Epoch 195/200\n",
      "18993/18993 [==============================] - 8s 420us/step - loss: 3.5472\n",
      "Epoch 196/200\n",
      "18993/18993 [==============================] - 8s 445us/step - loss: 3.5457\n",
      "Epoch 197/200\n",
      "18993/18993 [==============================] - 8s 438us/step - loss: 3.5458\n",
      "Epoch 198/200\n",
      "18993/18993 [==============================] - 9s 464us/step - loss: 3.5474\n",
      "Epoch 199/200\n",
      "18993/18993 [==============================] - 8s 419us/step - loss: 3.5405\n",
      "Epoch 200/200\n",
      "18993/18993 [==============================] - 8s 412us/step - loss: 3.5404\n"
     ]
    }
   ],
   "source": [
    "# train the model\n",
    "model.fit(X, y, batch_size=128, epochs=200,verbose = 1)\n",
    "\n",
    "# save weights\n",
    "model.save_weights(folder + 'rnn_tang_poetry_weights.hdf5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6700,
     "status": "ok",
     "timestamp": 1534863923823,
     "user": {
      "displayName": "Prins Wu",
      "photoUrl": "//lh5.googleusercontent.com/-e2-hwPV_U-g/AAAAAAAAAAI/AAAAAAAABTU/wj46QM5yHpc/s50-c-k-no/photo.jpg",
      "userId": "111176040429626006479"
     },
     "user_tz": -480
    },
    "id": "L5GG-kdPht0W",
    "outputId": "3cc49c3b-9daa-4044-d91d-240555412dca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predict: 絕峨眉仍欲去來\n"
     ]
    }
   ],
   "source": [
    "# load weights\n",
    "predict_modell = build_model(hidden_units, window_size, features, output_dim)\n",
    "predict_modell.load_weights(folder + 'rnn_tang_poetry_weights.hdf5')\n",
    "input_chars = ['絕']\n",
    "predict_input = gen.predict_next_chars(model,input_chars,6, window_size, one_hot=one_hot)\n",
    "print('predict:', predict_input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xMtFQmjFENxo"
   },
   "outputs": [],
   "source": [
    "\n",
    "def gen_poetry(model, prefix_ary, poetry_type=5):\n",
    "#     model.load_weights(folder + 'rnn_tang_poetry1_weights.hdf5')\n",
    "    poetry_sentence_len = poetry_type - 1\n",
    "    poetry = []\n",
    "    for char in prefix_ary:\n",
    "        predict_input = gen.predict_next_chars(model, [char], poetry_sentence_len, window_size, one_hot=one_hot)\n",
    "        poetry.append(predict_input)\n",
    "#         poetry.append('\\n')\n",
    "#         print(predict_input)\n",
    "    return poetry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 938,
     "status": "ok",
     "timestamp": 1534864346595,
     "user": {
      "displayName": "Prins Wu",
      "photoUrl": "//lh5.googleusercontent.com/-e2-hwPV_U-g/AAAAAAAAAAI/AAAAAAAABTU/wj46QM5yHpc/s50-c-k-no/photo.jpg",
      "userId": "111176040429626006479"
     },
     "user_tz": -480
    },
    "id": "HdI9KtJuFjcE",
    "outputId": "53a184f0-d55e-4be7-8514-d81daa133adb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5言\n",
      "絕頂誰為我\n",
      "妙獨立揚生\n",
      "好以橫波上\n",
      "詩文藻道旁\n",
      "\n",
      "7言\n",
      "絕倫自來歸遠去\n",
      "妙獨釣春草群塔\n",
      "好以菲薄幸李斯\n",
      "詩不得清瑟玳筵\n"
     ]
    }
   ],
   "source": [
    "prefix_ary = ['絕', '妙', '好', '詩']\n",
    "# prefix_ary = ['一', '二', '三', '四']\n",
    "# prefix_ary = ['日', '月', '天', '地']\n",
    "poetry = gen_poetry(predict_modell, prefix_ary, 5)\n",
    "print('5言')\n",
    "print(*poetry, sep='\\n')\n",
    "poetry = gen_poetry(predict_modell, prefix_ary, 7)\n",
    "print('\\n7言')\n",
    "print(*poetry, sep='\\n')"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "rnn_tang_poetry.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}