notebooks/graphical-models-exercises.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examples and exercises for graphical models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Only needed to generate graphs, may be safely ommitted \n",
    "# once you comment out relevant cells below\n",
    "from graphviz import Digraph, Graph\n",
    "\n",
    "# The below are packages I used to solve these exercises, but may be safely removed\n",
    "from fake_data_for_learning.fake_data_for_learning import (\n",
    "    BayesianNodeRV, SampleValue, FakeDataBayesianNetwork\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "datadir = Path(os.getcwd()) / 'data'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphical models recap\n",
    "\n",
    "Main assumptions:\n",
    "\n",
    "1. The graph $G$ is a directed, acyclical graph (DAG)\n",
    "2. The random variable(s) represented by $G$ follow the *Markov property*\n",
    "\n",
    "Recall that a *path* is undirected, and $G$ has no loops, as it is a DAG.\n",
    "\n",
    "Given a path, recall that a node $Z$ on the path is *blocked* by conditioning on a set of nodes $S$ in $G$ if\n",
    "\n",
    "1. $Z$ is in $S$ and is a non-collider\n",
    "2. neither $Z$ nor its descendents is in $S$ and $Z$ is a collider\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphical models example: Credit risk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 8.0.5 (20230430.1635)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"324pt\" height=\"369pt\"\n",
       " viewBox=\"0.00 0.00 324.41 368.66\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 364.66)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-364.66 320.41,-364.66 320.41,4 -4,4\"/>\n",
       "<!-- A -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>A</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"127.97\" cy=\"-305.21\" rx=\"55.45\" ry=\"55.45\"/>\n",
       "<text text-anchor=\"middle\" x=\"127.97\" y=\"-300.16\" font-family=\"Times,serif\" font-size=\"14.00\">Automotive</text>\n",
       "</g>\n",
       "<!-- b -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>b</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"34.97\" cy=\"-158.83\" rx=\"34.97\" ry=\"34.97\"/>\n",
       "<text text-anchor=\"middle\" x=\"34.97\" y=\"-153.78\" font-family=\"Times,serif\" font-size=\"14.00\">BMW</text>\n",
       "</g>\n",
       "<!-- A&#45;&#45;b -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>A&#45;&#45;b</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M98.29,-258.13C83.62,-235.36 66.37,-208.57 53.58,-188.72\"/>\n",
       "</g>\n",
       "<!-- z -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>z</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"127.97\" cy=\"-158.83\" rx=\"39.58\" ry=\"39.58\"/>\n",
       "<text text-anchor=\"middle\" x=\"127.97\" y=\"-153.78\" font-family=\"Times,serif\" font-size=\"14.00\">Zastava</text>\n",
       "</g>\n",
       "<!-- A&#45;&#45;z -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>A&#45;&#45;z</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M127.97,-249.63C127.97,-232.7 127.97,-214.4 127.97,-198.76\"/>\n",
       "</g>\n",
       "<!-- T -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>T</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"240.97\" cy=\"-158.83\" rx=\"54.93\" ry=\"54.93\"/>\n",
       "<text text-anchor=\"middle\" x=\"240.97\" y=\"-153.78\" font-family=\"Times,serif\" font-size=\"14.00\">Technology</text>\n",
       "</g>\n",
       "<!-- A&#45;&#45;T -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>A&#45;&#45;T</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M161.88,-260.9C176.31,-242.46 193.1,-221.01 207.48,-202.62\"/>\n",
       "</g>\n",
       "<!-- x -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>x</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"197.97\" cy=\"-33.95\" rx=\"33.95\" ry=\"33.95\"/>\n",
       "<text text-anchor=\"middle\" x=\"197.97\" y=\"-28.9\" font-family=\"Times,serif\" font-size=\"14.00\">Xerox</text>\n",
       "</g>\n",
       "<!-- T&#45;&#45;x -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>T&#45;&#45;x</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M222.99,-106.43C218.25,-92.88 213.27,-78.67 209.02,-66.51\"/>\n",
       "</g>\n",
       "<!-- a -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>a</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"282.97\" cy=\"-33.95\" rx=\"33.44\" ry=\"33.44\"/>\n",
       "<text text-anchor=\"middle\" x=\"282.97\" y=\"-28.9\" font-family=\"Times,serif\" font-size=\"14.00\">Apple</text>\n",
       "</g>\n",
       "<!-- T&#45;&#45;a -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>T&#45;&#45;a</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M258.54,-106.43C263.22,-92.75 268.13,-78.39 272.31,-66.16\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Graph at 0x1370fa7d0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dot = Graph()\n",
    "dot.attr('node', shape='circle')\n",
    "dot.node('A', 'Automotive')\n",
    "dot.node('b', 'BMW')\n",
    "dot.node('z', 'Zastava')\n",
    "\n",
    "dot.node('T', 'Technology')\n",
    "dot.node('x', 'Xerox')\n",
    "dot.node('a', 'Apple')\n",
    "\n",
    "dot.edges(['Ab', 'Az', 'AT', 'Tx', 'Ta'])\n",
    "\n",
    "dot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphical models exercise: DAGs\n",
    "\n",
    "Consider a directed, acyclical graph (DAG) $G$ with nodes $X_0, \\ldots, X_4$ and [adjacency matrix](https://en.wikipedia.org/wiki/Adjacency_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array([\n",
    "        [0, 0, 1, 0, 0],\n",
    "        [0, 0, 1, 0, 1],\n",
    "        [0, 0, 0, 1, 1],\n",
    "        [0, 0, 0, 0, 0],\n",
    "        [0, 0, 0, 0, 0]\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Draw the graph for $G$. Difficulty: *\n",
    "2. What row or column properties of the matrix $X$ can be used to identify the nodes with no incoming edges? Difficulty: *\n",
    "3. Write pseudo- (or better, real) code to implement your method to extract nodes with no incoming edges from $X$. Difficulty: **\n",
    "4. Write pseudo- (or better, real) code to determine the parents of given node from $X$. Difficulty: **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphical models exercise: conditional independence\n",
    "\n",
    "Take K to be your Karma, H to be the hours you spend in the gym lifting weight each day, and then W be the weight you can bench press on a given day. For simplicity, all random variables are binary, with each row representing a day in your life."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>karma</th>\n",
       "      <th>hours</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   karma  hours  weight\n",
       "0      1      0       1\n",
       "1      1      1       1\n",
       "2      1      1       1\n",
       "3      1      1       1\n",
       "4      1      0       1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(datadir / 'karma_weights.csv').head()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are also the parent of a very young child, so perhaps Karma will punish you for devoting too much time to your triceps and neglecting your partner and baby. Let $G$ be a graphical model for $(K, H, W)$, as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 8.0.5 (20230430.1635)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"62pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 62.00 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 58,-184 58,4 -4,4\"/>\n",
       "<!-- K -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>K</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-162\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-156.95\" font-family=\"Times,serif\" font-size=\"14.00\">K</text>\n",
       "</g>\n",
       "<!-- H -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>H</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-84.95\" font-family=\"Times,serif\" font-size=\"14.00\">H</text>\n",
       "</g>\n",
       "<!-- K&#45;&gt;H -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>K&#45;&gt;H</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M27,-143.7C27,-136.24 27,-127.32 27,-118.97\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"30.5,-119.1 27,-109.1 23.5,-119.1 30.5,-119.1\"/>\n",
       "</g>\n",
       "<!-- W -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>W</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-12.95\" font-family=\"Times,serif\" font-size=\"14.00\">W</text>\n",
       "</g>\n",
       "<!-- H&#45;&gt;W -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>H&#45;&gt;W</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M27,-71.7C27,-64.24 27,-55.32 27,-46.97\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"30.5,-47.1 27,-37.1 23.5,-47.1 30.5,-47.1\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x137118fd0>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dot = Digraph(comment='d-sep-exercise-1')\n",
    "dot.attr('node')\n",
    "dot.node('K')\n",
    "dot.node('H')\n",
    "dot.node('W')\n",
    "\n",
    "dot.edges(['KH', 'HW'])\n",
    "dot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Difficulty: *\n",
    "\n",
    "Determine all conditionally independent relationships $K \\perp W | S$ for possible values of $S$, and justify your answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphical models exercise: conditional independence 2\n",
    "\n",
    "Difficulty: *\n",
    "\n",
    "Same as above, but for $K \\perp H | S$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 8.0.5 (20230430.1635)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"134pt\" height=\"116pt\"\n",
       " viewBox=\"0.00 0.00 134.00 116.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 112)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-112 130,-112 130,4 -4,4\"/>\n",
       "<!-- K -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>K</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-84.95\" font-family=\"Times,serif\" font-size=\"14.00\">K</text>\n",
       "</g>\n",
       "<!-- W -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>W</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"63\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"63\" y=\"-12.95\" font-family=\"Times,serif\" font-size=\"14.00\">W</text>\n",
       "</g>\n",
       "<!-- K&#45;&gt;W -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>K&#45;&gt;W</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M35.35,-72.76C39.62,-64.46 44.92,-54.15 49.73,-44.79\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"53.23,-46.64 54.7,-36.15 47.01,-43.44 53.23,-46.64\"/>\n",
       "</g>\n",
       "<!-- H -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>H</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"99\" y=\"-84.95\" font-family=\"Times,serif\" font-size=\"14.00\">H</text>\n",
       "</g>\n",
       "<!-- H&#45;&gt;W -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>H&#45;&gt;W</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M90.65,-72.76C86.38,-64.46 81.08,-54.15 76.27,-44.79\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"78.99,-43.44 71.3,-36.15 72.77,-46.64 78.99,-43.44\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x137101690>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dot = Digraph()\n",
    "dot.attr('node')\n",
    "dot.node('K')\n",
    "dot.node('H')\n",
    "dot.node('W')\n",
    "\n",
    "dot.edges(['HW', 'KW'])\n",
    "\n",
    "dot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphical models exercise: conditional independence 3\n",
    "\n",
    "Difficulty: **\n",
    "\n",
    "Determine all conditional independence relationships implied by the graphical model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 8.0.5 (20230430.1635)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"134pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 134.00 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 130,-184 130,4 -4,4\"/>\n",
       "<!-- X -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>X</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-162\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-156.95\" font-family=\"Times,serif\" font-size=\"14.00\">X</text>\n",
       "</g>\n",
       "<!-- Y -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>Y</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-84.95\" font-family=\"Times,serif\" font-size=\"14.00\">Y</text>\n",
       "</g>\n",
       "<!-- X&#45;&gt;Y -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>X&#45;&gt;Y</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M27,-143.7C27,-136.24 27,-127.32 27,-118.97\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"30.5,-119.1 27,-109.1 23.5,-119.1 30.5,-119.1\"/>\n",
       "</g>\n",
       "<!-- Z -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>Z</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"63\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"63\" y=\"-12.95\" font-family=\"Times,serif\" font-size=\"14.00\">Z</text>\n",
       "</g>\n",
       "<!-- Y&#45;&gt;Z -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>Y&#45;&gt;Z</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M35.35,-72.76C39.62,-64.46 44.92,-54.15 49.73,-44.79\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"53.23,-46.64 54.7,-36.15 47.01,-43.44 53.23,-46.64\"/>\n",
       "</g>\n",
       "<!-- W -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>W</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"99\" y=\"-84.95\" font-family=\"Times,serif\" font-size=\"14.00\">W</text>\n",
       "</g>\n",
       "<!-- W&#45;&gt;Z -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>W&#45;&gt;Z</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M90.65,-72.76C86.38,-64.46 81.08,-54.15 76.27,-44.79\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"78.99,-43.44 71.3,-36.15 72.77,-46.64 78.99,-43.44\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x13711a790>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dot = Digraph(comment='d-sep-exercise-1')\n",
    "dot.attr('node')\n",
    "dot.node('X')\n",
    "dot.node('Y')\n",
    "dot.node('Z')\n",
    "dot.node('W')\n",
    "\n",
    "dot.edges(['XY', 'YZ', 'WZ'])\n",
    "\n",
    "dot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphical models exercise: d-separation for hit-rate\n",
    "\n",
    "Difficulty: **\n",
    "\n",
    "Find all conditional independence relations implied by the graph structure of the below graphical model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 8.0.5 (20230430.1635)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"146pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 146.22 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 142.22,-184 142.22,4 -4,4\"/>\n",
       "<!-- p -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>p</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"71.41\" cy=\"-162\" rx=\"54.93\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"71.41\" y=\"-156.95\" font-family=\"Times,serif\" font-size=\"14.00\">producttype</text>\n",
       "</g>\n",
       "<!-- r -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>r</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"32.41\" cy=\"-90\" rx=\"32.41\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"32.41\" y=\"-84.95\" font-family=\"Times,serif\" font-size=\"14.00\">rating</text>\n",
       "</g>\n",
       "<!-- p&#45;&gt;r -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>p&#45;&gt;r</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M61.97,-144.05C57.4,-135.85 51.82,-125.83 46.74,-116.71\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"49.42,-115.34 41.5,-108.31 43.31,-118.75 49.42,-115.34\"/>\n",
       "</g>\n",
       "<!-- d -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>d</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"110.41\" cy=\"-90\" rx=\"27.81\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"110.41\" y=\"-84.95\" font-family=\"Times,serif\" font-size=\"14.00\">days</text>\n",
       "</g>\n",
       "<!-- p&#45;&gt;d -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>p&#45;&gt;d</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M80.86,-144.05C85.42,-135.85 91.01,-125.83 96.09,-116.71\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"99.52,-118.75 101.33,-108.31 93.4,-115.34 99.52,-118.75\"/>\n",
       "</g>\n",
       "<!-- h -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>h</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"71.41\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"71.41\" y=\"-12.95\" font-family=\"Times,serif\" font-size=\"14.00\">hit</text>\n",
       "</g>\n",
       "<!-- r&#45;&gt;h -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>r&#45;&gt;h</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M41.66,-72.41C46.29,-64.1 52,-53.85 57.18,-44.56\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"60.7,-46.43 62.51,-35.99 54.58,-43.02 60.7,-46.43\"/>\n",
       "</g>\n",
       "<!-- d&#45;&gt;h -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>d&#45;&gt;h</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M101.37,-72.76C96.69,-64.37 90.88,-53.93 85.62,-44.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"88.34,-43.18 80.41,-36.15 82.22,-46.59 88.34,-43.18\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x13711b050>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dot = Digraph()\n",
    "dot.attr('node')\n",
    "dot.node('p', 'producttype')\n",
    "dot.node('r', 'rating')\n",
    "dot.node('d', 'days')\n",
    "dot.node('h', 'hit')\n",
    "\n",
    "dot.edges(['pd', 'pr', 'rh', 'dh'])\n",
    "\n",
    "dot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphical models exercise: model selection\n",
    "\n",
    "Consider the following Bayesian network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 8.0.5 (20230430.1635)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"196pt\" height=\"116pt\"\n",
       " viewBox=\"0.00 0.00 195.79 116.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 112)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-112 191.79,-112 191.79,4 -4,4\"/>\n",
       "<!-- X -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>X</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"28.32\" cy=\"-90\" rx=\"28.32\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"28.32\" y=\"-84.95\" font-family=\"Times,serif\" font-size=\"14.00\">Rain</text>\n",
       "</g>\n",
       "<!-- Z -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>Z</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"79.32\" cy=\"-18\" rx=\"47.26\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"79.32\" y=\"-12.95\" font-family=\"Times,serif\" font-size=\"14.00\">GrassWet</text>\n",
       "</g>\n",
       "<!-- X&#45;&gt;Z -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>X&#45;&gt;Z</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M39.89,-73.12C46.04,-64.68 53.74,-54.1 60.71,-44.55\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"64.02,-46.94 67.08,-36.8 58.36,-42.82 64.02,-46.94\"/>\n",
       "</g>\n",
       "<!-- Y -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>Y</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"131.32\" cy=\"-90\" rx=\"56.47\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"131.32\" y=\"-84.95\" font-family=\"Times,serif\" font-size=\"14.00\">SprinklerOn</text>\n",
       "</g>\n",
       "<!-- Y&#45;&gt;Z -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>Y&#45;&gt;Z</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M118.73,-72.05C112.6,-63.8 105.1,-53.7 98.29,-44.54\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"100.56,-42.73 91.79,-36.79 94.95,-46.9 100.56,-42.73\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x13711a610>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dot = Digraph()\n",
    "dot.attr('node')\n",
    "dot.node('X', 'Rain')\n",
    "dot.node('Y', 'SprinklerOn')\n",
    "dot.node('Z', 'GrassWet')\n",
    "\n",
    "dot.edges(['XZ', 'YZ'])\n",
    "\n",
    "dot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where \n",
    "$$\n",
    "\\begin{equation*}\n",
    "\\mathrm{Rain} \\sim B(p_R) \\\\\n",
    "\\mathrm{SprinklerOn} \\sim B(p_S)\n",
    "\\end{equation*}\n",
    "$$\n",
    "\n",
    "and $\\mathrm{GrassWet}$ is Bernoulli conditioned on the values of $\\mathrm{Rain}$ and $\\mathrm{SprinklerOn}$.\n",
    "\n",
    "1. Write down the probability distribution equation / decomposition for $(\\mathrm{Rain}, \\mathrm{SprinklerOn}, \\mathrm{GrassWet})$. Difficulty: *\n",
    "2. Given the data in `data/wet_grass.csv`, estimate the probability distribution of $(\\mathrm{Rain}, \\mathrm{SprinklerOn}, \\mathrm{GrassWet})$. Difficulty: **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rain</th>\n",
       "      <th>SprinklerOn</th>\n",
       "      <th>GrassWet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Rain  SprinklerOn  GrassWet\n",
       "0     0            0         0\n",
       "1     0            0         0\n",
       "2     0            0         0\n",
       "3     0            0         1\n",
       "4     1            1         1"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv(datadir / 'wet_grass.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphical models exercise: model selection, continued\n",
    "\n",
    "Consider the same data as above, `data/wet_grass.csv`, but with the graphical model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 8.0.5 (20230430.1635)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"91pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 90.75 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 86.75,-184 86.75,4 -4,4\"/>\n",
       "<!-- X -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>X</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"68.38,-180 14.38,-180 14.38,-144 68.38,-144 68.38,-180\"/>\n",
       "<text text-anchor=\"middle\" x=\"41.38\" y=\"-156.95\" font-family=\"Times,serif\" font-size=\"14.00\">Rain</text>\n",
       "</g>\n",
       "<!-- Y -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>Y</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"82.75,-108 0,-108 0,-72 82.75,-72 82.75,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"41.38\" y=\"-84.95\" font-family=\"Times,serif\" font-size=\"14.00\">SprinklerOn</text>\n",
       "</g>\n",
       "<!-- X&#45;&gt;Y -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>X&#45;&gt;Y</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M41.38,-143.7C41.38,-136.24 41.38,-127.32 41.38,-118.97\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"44.88,-119.1 41.38,-109.1 37.88,-119.1 44.88,-119.1\"/>\n",
       "</g>\n",
       "<!-- Z -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>Z</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"76,-36 6.75,-36 6.75,0 76,0 76,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"41.38\" y=\"-12.95\" font-family=\"Times,serif\" font-size=\"14.00\">GrassWet</text>\n",
       "</g>\n",
       "<!-- Y&#45;&gt;Z -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>Y&#45;&gt;Z</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M41.38,-71.7C41.38,-64.24 41.38,-55.32 41.38,-46.97\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"44.88,-47.1 41.38,-37.1 37.88,-47.1 44.88,-47.1\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x137119f10>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dot = Digraph()\n",
    "dot.attr('node', shape='box')\n",
    "dot.node('X', 'Rain')\n",
    "dot.node('Y', 'SprinklerOn')\n",
    "dot.node('Z', 'GrassWet')\n",
    "\n",
    "dot.edges(['XY', 'YZ'])\n",
    "\n",
    "dot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate which of the two graphical models is better for the given data.\n",
    "\n",
    "Difficulty: **"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}