jupyter/transfer_learning_on_cifar10.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transfer Learning on CIFAR-10 Dataset\n",
    "\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this tutorial, you learn how to train an image classification model using [Transfer Learning](https://en.wikipedia.org/wiki/Transfer_learning). Transfer learning is a popular machine learning technique that uses a model trained on one problem and applies it to a second related problem. Compared to training from scratch or designing a model for your specific problem, transfer learning can leverage the features already learned on a similar problem and produce a more robust model in a much shorter time.\n",
    "\n",
    "Train your model with the [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset which consists of 60,000 32x32 color images in 10 classes. As for the pre-trained model, use the ResNet50v1[1] model. It's a 50 layer deep model already trained on [ImageNet](http://www.image-net.org/), a much larger dataset consisting of over 1.2 million images in 1000 classes. Modify it to classify 10 classes from the CIFAR-10 dataset.\n",
    "\n",
    "![The CIFAR-10 Dataset](https://resources.djl.ai/images/cifar-10.png)\n",
    "<center>the CIFAR10 dataset</center>\n",
    "\n",
    "\n",
    "## Pre-requisites\n",
    "This tutorial assumes you have the following knowledge. Follow the READMEs and tutorials if you are not familiar with:\n",
    "1. How to setup and run [Java Kernel in Jupyter Notebook](https://github.com/deepjavalibrary/djl/blob/master/jupyter/README.md)\n",
    "2. Basic components of Deep Java Library, and how to [train your first model](https://github.com/deepjavalibrary/djl/blob/master/jupyter/tutorial/02_train_your_first_model.ipynb).\n",
    "\n",
    "\n",
    "## Getting started\n",
    "Load the Deep Java Libarary and its dependencies from Maven:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "// %mavenRepo snapshots https://oss.sonatype.org/content/repositories/snapshots/\n",
    "\n",
    "%maven ai.djl:api:0.12.0\n",
    "%maven ai.djl:basicdataset:0.12.0\n",
    "%maven ai.djl.mxnet:mxnet-engine:0.12.0\n",
    "%maven ai.djl.mxnet:mxnet-model-zoo:0.12.0\n",
    "%maven org.slf4j:slf4j-api:1.7.26\n",
    "%maven org.slf4j:slf4j-simple:1.7.26\n",
    "%maven net.java.dev.jna:jna:5.3.0\n",
    "        \n",
    "// See https://github.com/deepjavalibrary/djl/blob/master/mxnet/mxnet-engine/README.md\n",
    "// for more MXNet library selection options\n",
    "%maven ai.djl.mxnet:mxnet-native-auto:1.8.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's import the necessary modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ai.djl.*;\n",
    "import ai.djl.basicdataset.cv.classification.*;\n",
    "import ai.djl.modality.*;\n",
    "import ai.djl.modality.cv.*;\n",
    "import ai.djl.modality.cv.transform.*;\n",
    "import ai.djl.ndarray.*;\n",
    "import ai.djl.ndarray.types.*;\n",
    "import ai.djl.nn.*;\n",
    "import ai.djl.nn.core.*;\n",
    "import ai.djl.repository.zoo.*;\n",
    "import ai.djl.training.*;\n",
    "import ai.djl.training.dataset.*;\n",
    "import ai.djl.training.initializer.*;\n",
    "import ai.djl.training.listener.*;\n",
    "import ai.djl.training.loss.*;\n",
    "import ai.djl.training.evaluator.*;\n",
    "import ai.djl.training.optimizer.*;\n",
    "import ai.djl.training.tracker.*;\n",
    "import ai.djl.training.util.*;\n",
    "import ai.djl.translate.*;\n",
    "import java.nio.file.*;\n",
    "import java.util.*;\n",
    "import java.util.concurrent.*;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Construct your model\n",
    "\n",
    "Load the pre-trained ResNet50V1 model. You can find it in the [Model Zoo](https://github.com/deepjavalibrary/djl/blob/master/docs/model-zoo.md). First construct the `criteria` to specify which ResNet model to load, then call `loadModel` to get a ResNet50V1 model with pre-trained weights. Note this model was trained on ImageNet with 1000 classes; the last layer is a Linear layer with 1000 output channels. Because you are repurposing it on CIFAR10 with 10 classes, you need to remove the last layer and add a new Linear layer with 10 output channels. After you are done modifying the block, set it back to model using `setBlock`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "// load model and change last layer\n",
    "Criteria<Image, Classifications> criteria = Criteria.builder()\n",
    "    .setTypes(Image.class, Classifications.class)\n",
    "    .optProgress(new ProgressBar())\n",
    "    .optArtifactId(\"resnet\")\n",
    "    .optFilter(\"layers\", \"50\")\n",
    "    .optFilter(\"flavor\", \"v1\").build();\n",
    "Model model = criteria.loadModel();\n",
    "SequentialBlock newBlock = new SequentialBlock();\n",
    "SymbolBlock block = (SymbolBlock) model.getBlock();\n",
    "block.removeLastBlock();\n",
    "newBlock.add(block);\n",
    "newBlock.add(Blocks.batchFlattenBlock());\n",
    "newBlock.add(Linear.builder().setUnits(10).build());\n",
    "model.setBlock(newBlock);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare Dataset\n",
    "\n",
    "After you have the model, the next step is to prepare the dataset for training. You can construct a CIFAR10 builder with your own specifications. You have the options to get the train or test dataset, specify desired batch size, specify whether to shuffle your data during training, and most importantly, specify the pre-process pipeline. \n",
    "\n",
    "A pipeline consists of a series of transformations to apply on the input data before feeding it to the model. \n",
    "\n",
    "For example, `ToTensor` can be used to transform colored image NDArrays with shape (32, 32, 3) and values from 0 to 256 to NDArrays with shape (3, 32, 32) and values from 0 to 1. This operation is transposing image data from channels last to channels first format, which is more suitable for GPU computation. \n",
    "\n",
    "The `Normalize` transformation can normalize input data according to their mean and standard deviation values. This will make different features have similar range and help our model perform better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "int batchSize = 32;\n",
    "int limit = Integer.MAX_VALUE; // change this to a small value for a dry run\n",
    "// int limit = 160; // limit 160 records in the dataset for a dry run\n",
    "Pipeline pipeline = new Pipeline(\n",
    "    new ToTensor(),\n",
    "    new Normalize(new float[] {0.4914f, 0.4822f, 0.4465f}, new float[] {0.2023f, 0.1994f, 0.2010f}));\n",
    "Cifar10 trainDataset = \n",
    "    Cifar10.builder()\n",
    "    .setSampling(batchSize, true)\n",
    "    .optUsage(Dataset.Usage.TRAIN)\n",
    "    .optLimit(limit)\n",
    "    .optPipeline(pipeline)\n",
    "    .build();\n",
    "trainDataset.prepare(new ProgressBar());"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up training configuration\n",
    "\n",
    "You are leveraging a pre-trained model, so you can expect the model to converge quickly. You will only train only ten epochs. As the model converges, you need to reduce the learning rate to get better results. You can use a `Tracker` to reduce the learning rate by 0.1 after two, five, and eight epochs. \n",
    "\n",
    "Deep Java Library supports training on multiple GPUs. You can use `setDevices` and pass an array of devices you want the model to be trained on. For example, `new Device[]{Device.gpu(0), Device.gpu(1)}` for training on GPU0 and GPU1. You can also call `Device.getDevices(4)` and pass the number of GPUs you want to train. It will start with GPU0, and use CPU if no GPU is available. To learn more about multi-GPU training, read our multi-GPU [documentation](https://github.com/deepjavalibrary/djl/tree/master/examples/docs).\n",
    "\n",
    "To complete the training configuration set up, use the `Adam` optimizer, `SoftmaxCrossEntropyLoss`, and `Accuracy` for classification problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DefaultTrainingConfig config = new DefaultTrainingConfig(Loss.softmaxCrossEntropyLoss())\n",
    "    //softmaxCrossEntropyLoss is a standard loss for classification problems\n",
    "    .addEvaluator(new Accuracy()) // Use accuracy so we humans can understand how accurate the model is\n",
    "    .optDevices(Device.getDevices(1)) // Limit your GPU, using more GPU actually will slow down coverging\n",
    "    .addTrainingListeners(TrainingListener.Defaults.logging());\n",
    "\n",
    "// Now that we have our training configuration, we should create a new trainer for our model\n",
    "Trainer trainer = model.newTrainer(config);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train your model\n",
    "Now you can start training. This procedure is similar to the one in [Train Your First Model](https://github.com/deepjavalibrary/djl/blob/master/jupyter/tutorial/02_train_your_first_model.ipynb). Training requires the following steps:\n",
    "1. Initialize a new trainer using the training config you just set up\n",
    "2. Initialize the weights in trainer\n",
    "3. Using a `for` loop to iterate through the whole dataset 10 times (epochs), resetting the evaluators at the end of each epoch\n",
    "4. During each epoch, using a `for` loop to iterate through the dataset in batches and train batch by batch while printing the training accuracy on the progress bar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "int epoch = 10;\n",
    "Shape inputShape = new Shape(1, 3, 32, 32);\n",
    "trainer.initialize(inputShape);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for (int i = 0; i < epoch; ++i) {\n",
    "    int index = 0;\n",
    "    for (Batch batch : trainer.iterateDataset(trainDataset)) {\n",
    "        EasyTrain.trainBatch(trainer, batch);\n",
    "        trainer.step();\n",
    "        batch.close();\n",
    "    }\n",
    "\n",
    "    // reset training and validation evaluators at end of epoch\n",
    "    trainer.notifyListeners(listener -> listener.onEpoch(trainer));\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save your model\n",
    "\n",
    "Finally, you can save your model after training is done and use it for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Path modelDir = Paths.get(\"build/resnet\");\n",
    "Files.createDirectories(modelDir);\n",
    "\n",
    "model.setProperty(\"Epoch\", String.valueOf(epoch));\n",
    "model.save(modelDir, \"resnet\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's next\n",
    "\n",
    "1. Try inference using the model you just trained. You can find an airplane image in [test resources](https://github.com/deepjavalibrary/djl/blob/master/examples/src/test/resources/airplane1.png) and follow the inference tutorials in the [Jupyter module](https://github.com/deepjavalibrary/djl/tree/master/jupyter).\n",
    "\n",
    "2. Follow the complete example with multi-GPU support, a validation dataset, and the fit API in the [examples module](https://github.com/deepjavalibrary/djl/tree/master/examples/docs).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "[1] [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385)\n",
    "\n",
    "[2] [Gluon CV model zoo](https://gluon-cv.mxnet.io/model_zoo/classification.html) for pre-trained ResNet50 models"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "14.0.2+12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}