03_nbdev.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a55fea8a-ca20-4dcf-a452-67cae672b555",
   "metadata": {},
   "source": [
    "# nbdev"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "808130cf-0f9b-41d0-b1ea-31e23236fd54",
   "metadata": {},
   "source": [
    "## What is nbdev?\n",
    "\n",
    "nbdev is a notebook-driven development platform. Simply write notebooks with lightweight markup and get high-quality documentation, tests, continuous integration, and packaging"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70d31e2b-05fa-47f5-86b2-efdb781361ba",
   "metadata": {},
   "source": [
    "## What will we learn in this notebook?\n",
    "\n",
    "`nbdev` sounds pretty cool, but there's also a lot going on there. Maybe you haven't even heard of continuous integration before. In this notebook we will go over how to use modular nbdev to create a python module. This is a much less complex introduction than the nbdev end-to-end walkthrough, because we won't create a whole package that can be uploaded to PyPi or Anaconda. We will simply use notebooks to create clean python modules that we can use to build our dashboards and web apps. \n",
    "\n",
    "\n",
    "### Export code to modules with nb_export\n",
    "- `#| default_exp` directive to signal which module to export to\n",
    "- `#| export` directive to signal that a cell is to be included in the export\n",
    "- `nbdev.export.nb_export` with the name of the notebook and export directory\n",
    "\n",
    "### End Goal\n",
    "\n",
    "The end goal of this tutorial notebook is to understand how we can use nbdev to create a `dashboard` python module that contains all the files we need for our dashboard.  We can use some of the cells in this notebook to create a python module that we can import to other notebooks. \n",
    "\n",
    "## Export cells to __init__.py\n",
    "\n",
    "What will the content of this notebook be exactly? We are going to create a few line and cell magics that we will use through the tutorial. We want these magic functions to be accessible anywhere. Therefore, if we want them to go in `__init__.py` the default export for this notebook needs to be `__init__`. Let's set the `default_exp` directive below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c386eed-7542-4ff4-aa6e-cc3036c35beb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| default_exp __init__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9f1341d-d52e-4e12-a293-a3e510bb8a71",
   "metadata": {},
   "source": [
    "## #| default_exp\n",
    "\n",
    "The `default_exp` directive is required of all notebooks used by nbdev. Notice that nbdev directives all start with \"#|\". \n",
    "This requires that we add a default export directive to our notebook. It tells nbdev what module to export to. You can also give it a “dotted module names” as is done with python packages. For example, a default export to `some.module` will create a file called `some/module.py`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bf7ccc7-7160-430e-b73d-3b68f543ed96",
   "metadata": {},
   "source": [
    "## #| export\n",
    "\n",
    "This directive tells nbdev that we want to export this code to the module specified by `default_exp`. Notice that we need to export everything that we want to end up in the python file, so we can't forget to export our imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a54f937-2bd7-473d-8dc9-b575de271b45",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| export\n",
    "\n",
    "from IPython.core.magic import register_line_magic, register_cell_magic, needs_local_scope\n",
    "from IPython import get_ipython\n",
    "import re, sys, os"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f9cf0eb-4e6d-4f05-b3c5-c85b6ba4f26f",
   "metadata": {},
   "source": [
    "## Literate Programming \n",
    "\n",
    "You might be wondering, other than markdown cells, why wouldn't we want to always export our code cells? This is a fair question. If you clean up your notebooks after you are done developing the code, you might not see the utitlity here yet. But the real value of being able to exclude code cells lies in the ability to tell the story of how your code came into it's final form. Instead of cleaning up your notebooks when you have achieved your coding goal, you can leave in all the valuable information you left behind along the way. This could include design planning and decision making. \n",
    "\n",
    "Have you ever been looking at old code and scoffed at the way you designed something? Maybe you decide to go clean it up, only to gradually realize that you've made this mistake before. There was a reason you left in this ugly solution, wasn't there! Now you've wasted a lot of time because you didn't take the time to record your decision-making the first time. Of course, you may log your thoughts in GitHub issues, but leaving behind your logical process in the notebook will make sure that you don't loose track of it down the road. Your thoughts and decisions remain transparent for your future self, or that lucky person that inherits all your code. \n",
    "\n",
    "Another reason we might want to have non-exported cells is that we want to be able to use or test our code without having to switch context to the terminal or another file. And again, this is helpful for another developer or our future-selves who may be trying to read our code. They don't have to hold as many variables in their head, because they can see the steps broken down and read the intermediate values without having to use a debugger. Let's try out this implicit benefit for ourselves below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "768db916-323c-4f36-8d12-e4d5505c17af",
   "metadata": {},
   "source": [
    "### %answer line magic\n",
    "\n",
    "Below is the code that defines the cell magic function called `exception`. Often in tutorials, we want to demonstrate an intentional error. The default behavior of a Jupyter Notebook is to print the full traceback, which can be quite lengthy and distracting. In addition, if the participant wants to \"Restart and Run All Cells\" due to some problem they encountered, the execution will get hung up where the error is. In our case, when running a cell with an error for demonstrative purposes, we want to display the error, but we don't want to display the traceback or prevent the execution of the next cell. This cell magic achieves this desired behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f2bd252-8ec7-416e-8491-3f6e9b9baacd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| export\n",
    "@register_cell_magic('exception')\n",
    "def exception(line, cell):\n",
    "    ip = get_ipython()\n",
    "    try:\n",
    "        exec(cell, None, ip.user_ns)\n",
    "    except Exception as e:\n",
    "        etype, value, tb = sys.exc_info()\n",
    "        value.__cause__ = None  # suppress chained exceptions\n",
    "        ip._showtraceback(etype, value, ip.InteractiveTB.get_exception_only(etype, value))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a822ea0b-cb3b-4504-913e-ec372c9e6fe0",
   "metadata": {},
   "source": [
    "As I said, the benefit of literate programming is that we can test out this function in-line. We don't have to delete the exploratory or explanitory code. The next line shows what happens when we use the cell magic. Notice that there isn't an export directive at the top, so this breaking error isn't exported to our python module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bab48bcc-9a44-4ad5-8e30-8ba332e0a16c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%exception\n",
    "\n",
    "5 / 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f13d0efc-f3e0-4fd0-87bb-b7478211dfe5",
   "metadata": {},
   "source": [
    "You can compare that output to the output below, which includes a traceback by default because the cell magic wasn't used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04f6f9ff-6631-48e0-9d20-066640a20005",
   "metadata": {},
   "outputs": [],
   "source": [
    "5 / 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49568f0d-4ad9-4727-9fa0-bcdf3005e0bb",
   "metadata": {},
   "source": [
    "Isn't literate programming great! It also means that we can build a very real output from a series of tutorial-style notebooks. All the python modules and packages in this tutorial were produced from the tutorial notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca63138c-13ff-445e-8a55-7176def3366f",
   "metadata": {},
   "source": [
    "### %answer line magic\n",
    "\n",
    "The code below allows us to import code from the nbdev-generated python files like the one we loaded above. You can think of this function working like the `%load` line magic, which pulls in code from an external file. But instead of loading the entire file, this function loads only the python code that came from an exported notebook cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf826708-05c7-48a6-b8ce-8423fe897faf",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| export\n",
    "@register_line_magic('answer')\n",
    "def answer(inputs):\n",
    "    '''\n",
    "    This is a cell magic called answer that allows tutorial goers to import the correct answer from the key. \n",
    "    '''\n",
    "    words = []\n",
    "    for word in inputs.split(' '):\n",
    "        if not word.startswith('#') and len(word) != 0:\n",
    "            words.append(word)\n",
    "        else:\n",
    "            break\n",
    "    \n",
    "    flag = False\n",
    "    if len(words) == 2:\n",
    "        if words[1] == '-e':\n",
    "            flag = True\n",
    "        else:\n",
    "            filepath = words[0]\n",
    "            cell_number = int(words[1])\n",
    "\n",
    "            with open(filepath, 'r') as file:\n",
    "                lines = file.readlines()\n",
    "\n",
    "            pattern = r'# %%\\s+(.+)\\s+(\\d+)'\n",
    "            start_line = None\n",
    "            end_line = None\n",
    "\n",
    "            for i, line in enumerate(lines):\n",
    "                if re.match(pattern, line):\n",
    "                    match = re.search(pattern, line)\n",
    "                    if match and int(match.group(2)) == cell_number:\n",
    "                        start_line = i + 1\n",
    "                        break\n",
    "            if start_line is not None:\n",
    "                for i in range(start_line, len(lines)):\n",
    "                    if re.match(pattern, lines[i]):\n",
    "                        end_line = i\n",
    "                        break\n",
    "                else:\n",
    "                    end_line = len(lines)\n",
    "\n",
    "            if start_line is not None and end_line is not None:\n",
    "                code_chunk = f\"#| export\\n# %answer {inputs}\\n\\n\" + ''.join(lines[start_line:end_line])\n",
    "                code_chunk = code_chunk.rstrip(\"\\n\")\n",
    "                get_ipython().set_next_input(code_chunk, replace=True)\n",
    "            else:\n",
    "                raise Exception(f\"Cell number {cell_number} not found in the Python file.\")\n",
    "        \n",
    "    if len(words) == 1 or words[1] == '-e':\n",
    "        filepath = words[0]\n",
    "        with open(filepath, 'r') as file:\n",
    "            lines = file.readlines()\n",
    "        code_chunk = ''.join(lines[:])\n",
    "        if flag:\n",
    "            code_chunk = f\"# %%export {filepath}\\n\\n\" + code_chunk\n",
    "        else: \n",
    "            code_chunk = f\"# %answer {filepath}\\n\\n\" + code_chunk\n",
    "        get_ipython().set_next_input(code_chunk, replace=True)\n",
    "    \n",
    "    with open(filepath, 'r') as file:\n",
    "        lines = file.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8550a7c2-5176-4a9e-b66d-db1ccd44fdf3",
   "metadata": {},
   "source": [
    "If at any point in this tutorial you have trouble coming up with an answer on your own, you can always import it from the answer key.\n",
    "\n",
    "### Pull from exported module\n",
    "\n",
    "This will be the case when we are pulling in an answer to a cell that starts with `#| export`. The path to the answer key files is the same as the default, but is preceeded by the `key` directory. For example, the next line will load the code from the 6th cell of this notebook, which is the import statements. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2cb3849-b5f0-4313-928e-1cf3bcc425d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %answer key/dashboard/__init__.py 6"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c9ff319-d009-4d66-8eea-e663b3f41d62",
   "metadata": {},
   "source": [
    "### Pull from file\n",
    "\n",
    "Sometimes we will want to check an answer from a cell that isn't a module exported by nbdev. In this case, we will provide the file path only.\n",
    "\n",
    "**In both cases, these paths will be provided for you in comments.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c7ecc7a-0058-4d2f-be99-0d50a6fe0519",
   "metadata": {},
   "source": [
    "## nb_export\n",
    "\n",
    "Next we can use the nb_export function to compile the exported cells into a python file. The second parameter tells nbdev which package to export the module to. Let's go ahead and export this notebook to the `dashboard` package. \n",
    "\n",
    "> **IMPORTANT**: do not forget to save before exporting the notebook, or your most recent changes will not make it into the py file. It is also a good habit to \"Restart Kernel and Clear All Outputs\" and then save your notebook at the same time as you export. That way, when you get around to committing your code, your git diff isn't filled up with a bunch of serial numbers and other unintelligible things."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b8b5029-0636-4dda-ae11-d5641450ec35",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nbdev.export import nb_export\n",
    "\n",
    "nb_export('03_nbdev.ipynb', 'dashboard')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcd23a2c-ec98-4ab5-a362-1ff9e1e5f37d",
   "metadata": {},
   "source": [
    "Usually, we will put `nb_export` at the end of our file for convenience, but we will continue on from here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7a42e2c-1d26-4771-8380-f05d1446cccc",
   "metadata": {},
   "source": [
    "Another way to verify that this worked is to import the function we just wrote to the python file. Can you guess what that import statement would look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c0f19f1-f4a8-4740-a069-d038e5d8b503",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %answer key/03/01.py\n",
    "\n",
    "# import ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7060754-a91a-42e1-8aa0-0455741edb66",
   "metadata": {},
   "source": [
    "So for most modules, the import statement would look like `dashboard.__init__` but the `__init__` file is special, and will be imported any time we make an import from `dashboard` package.\n",
    "\n",
    "Great job! Now we know the basic idea of what nbdev does. Go ahead and use the `%load` line magic inspect the exported file below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c1de961-d19b-41b7-a893-992d8fb32422",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %answer key/03/02.py\n",
    "\n",
    "# %load pkg/module.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ef6b07d-1023-48ec-bdb6-879d1c7d216d",
   "metadata": {},
   "source": [
    "There are a few thing to notice here. One is that we get a warning about this file being autogenerated. That is, if we make changes to `dashboard/__init__.py`, those changes will be overwritten the next time we run `nb_export`.\n",
    "\n",
    "The second thing to notice is that there is a handy comment to tell us where the code came from. `# %% ../_nbdev.ipynb 9` tells us that the code came from the 9th code cell. We will use this information to write our next magic function."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de626c60-1a66-4e28-9285-3cf779b7dbee",
   "metadata": {},
   "source": [
    "## Literate Programming\n",
    "\n",
    "Of course, nbdev has a lot of other nice features, but the notebook export is the core of literate programming. You can do your work incrementally, leave in a lot of markdown, and read your code like a book. At the same time, you can keep the code in clean, importable python files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fa634ee-53f9-41d1-8ad4-8d55f5e154be",
   "metadata": {},
   "source": [
    "### The End\n",
    "\n",
    "We will not go over the code below. Proceed to the next notebook.\n",
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97fb40d0-242e-4aea-aa50-d054d38f8b55",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| export\n",
    "@register_cell_magic('export')\n",
    "def export(line, cell=None):\n",
    "    line_args = line.split()\n",
    "    export_filepath = None\n",
    "\n",
    "    if len(line_args):\n",
    "        export_filepath = line_args[0]\n",
    "        directory = os.path.dirname(export_filepath)\n",
    "        os.makedirs(directory, exist_ok=True)\n",
    "        with open(export_filepath, 'w') as file:\n",
    "            file.write(cell)\n",
    "        print(f\"exported to {export_filepath}\")\n",
    "            \n",
    "    processed_lines = []\n",
    "    for line in cell.split('\\n'):\n",
    "        comment_match = re.search(r'#', line)\n",
    "        if comment_match:\n",
    "            line = line[comment_match.start():]\n",
    "            processed_lines.append(line)\n",
    "\n",
    "    processed_cell = '\\n'.join(processed_lines)\n",
    "    if len(line_args):\n",
    "        processed_cell = '# %answer ' + line_args[0] + '\\n\\n' + processed_cell\n",
    "    get_ipython().set_next_input(processed_cell, replace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f38ff439-b7c6-4f66-bcea-ca87a079d7cd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}