Best_Practices.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Best Practices",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ii_Vk8PnLyKt",
        "colab_type": "text"
      },
      "source": [
        "# Convolutonal Neural Network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S2xxpV0sL403",
        "colab_type": "text"
      },
      "source": [
        "01. Always use transfer learning. No point in training from scratch. Even if domain is completly different, still low level feathure can be fine tuned with low learning rates while higher level of abstraction can be trained by usaul procedure. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eh7jHpZ0L4zR",
        "colab_type": "text"
      },
      "source": [
        "02. *Use* different learning rate for different layers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UXtxnHTQMz1P",
        "colab_type": "text"
      },
      "source": [
        "# Recurrent Neural Netowork"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZfKMjq3_M2RO",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q4x6DTGbM2hD",
        "colab_type": "text"
      },
      "source": [
        "# Generative Advesarial Networks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p2r6fCrEM2fN",
        "colab_type": "text"
      },
      "source": [
        "01. Sample from Standard Normal distribution or Standard-T distribution. Avoid using uniform distribution. Standard Laplacian distribution works fine but a large fraction of sample is zero."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "usGN5F7iOEgn",
        "colab_type": "text"
      },
      "source": [
        "02. Use dropout in both train and test phase. Keep probability between 0.5 to 0.6"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1ZEyHFzdO3S7",
        "colab_type": "text"
      },
      "source": [
        "03. ADAM for generator and SGD for discrimintor."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SVToeYlFO3Qw",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gi4ONO79NMse",
        "colab_type": "text"
      },
      "source": [
        "# Generative Network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GONtswtPNUvn",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Bbx_UbEM8ll",
        "colab_type": "text"
      },
      "source": [
        "# Bayesian Neural Network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jHyjuyCPM8ju",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N8kzQ9_YNHuG",
        "colab_type": "text"
      },
      "source": [
        "# Gaussain Processes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8wH46_lxNV8B",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    }
  ]
}