mobile_game_testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "3"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "## 1. Of cats and cookies\n",
    "<p><a href=\"https://www.facebook.com/cookiecatsgame\">Cookie Cats</a> is a hugely popular mobile puzzle game developed by <a href=\"http://tactile.dk\">Tactile Entertainment</a>. It's a classic \"connect three\"-style puzzle game where the player must connect tiles of the same color to clear the board and win the level. It also features singing cats. We're not kidding! Check out this short demo:</p>\n",
    "<p><a href=\"https://youtu.be/GaP5f0jVTWE\"><img src=\"https://s3.amazonaws.com/assets.datacamp.com/production/project_184/img/cookie_cats_video.jpeg\" style=\"width: 500px\"></a></p>\n",
    "<p>As players progress through the levels of the game, they will occasionally encounter gates that force them to wait a non-trivial amount of time or make an in-app purchase to progress. In addition to driving in-app purchases, these gates serve the important purpose of giving players an enforced break from playing the game, hopefully resulting in that the player's enjoyment of the game being increased and prolonged.</p>\n",
    "<p><img src=\"https://s3.amazonaws.com/assets.datacamp.com/production/project_184/img/cc_gates.png\" alt=\"\"></p>\n",
    "<p>But where should the gates be placed? Initially the first gate was placed at level 30, but in this notebook we're going to analyze an AB-test where we moved the first gate in Cookie Cats from level 30 to level 40. In particular, we will look at the impact on player retention. But before we get to that, a key step before undertaking any analysis is understanding the data. So let's load it in and take a look!</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {
    "dc": {
     "key": "3"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userid</th>\n",
       "      <th>version</th>\n",
       "      <th>sum_gamerounds</th>\n",
       "      <th>retention_1</th>\n",
       "      <th>retention_7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>116</td>\n",
       "      <td>gate_30</td>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>337</td>\n",
       "      <td>gate_30</td>\n",
       "      <td>38</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>377</td>\n",
       "      <td>gate_40</td>\n",
       "      <td>165</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>483</td>\n",
       "      <td>gate_40</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>488</td>\n",
       "      <td>gate_40</td>\n",
       "      <td>179</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userid  version  sum_gamerounds  retention_1  retention_7\n",
       "0     116  gate_30               3        False        False\n",
       "1     337  gate_30              38         True        False\n",
       "2     377  gate_40             165         True        False\n",
       "3     483  gate_40               1        False        False\n",
       "4     488  gate_40             179         True         True"
      ]
     },
     "execution_count": 240,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Importing pandas\n",
    "import pandas as pd\n",
    "# Reading in the data\n",
    "df = pd.read_csv('cookie_cats.csv')\n",
    "\n",
    "# Showing the first few rows\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "10"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "## 2. The AB-test data\n",
    "<p>The data we have is from 90,189 players that installed the game while the AB-test was running. The variables are:</p>\n",
    "<ul>\n",
    "<li><code>userid</code> - a unique number that identifies each player.</li>\n",
    "<li><code>version</code> - whether the player was put in the control group (<code>gate_30</code> - a gate at level 30) or the group with the moved gate (<code>gate_40</code> - a gate at level 40).</li>\n",
    "<li><code>sum_gamerounds</code> - the number of game rounds played by the player during the first 14 days after install.</li>\n",
    "<li><code>retention_1</code> - did the player come back and play <strong>1 day</strong> after installing?</li>\n",
    "<li><code>retention_7</code> - did the player come back and play <strong>7 days</strong> after installing?</li>\n",
    "</ul>\n",
    "<p>When a player installed the game, he or she was randomly assigned to either <code>gate_30</code> or <code>gate_40</code>. As a sanity check, let's see if there are roughly the same number of players in each AB group. </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {
    "dc": {
     "key": "10"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gate_40    45489\n",
       "gate_30    44700\n",
       "Name: version, dtype: int64"
      ]
     },
     "execution_count": 242,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Counting the number of players in each AB group.\n",
    "df['version'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "17"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "## 3. The distribution of game rounds\n",
    "<p><img src=\"https://s3.amazonaws.com/assets.datacamp.com/production/project_184/img/mr_waffles_smiling.png\" style=\"width:200px; float:left\"> </p>\n",
    "<p>It looks like there is roughly the same number of players in each group, nice!</p>\n",
    "<p>The focus of this analysis will be on how the gate placement affects player retention, but just for fun: Let's plot the distribution of the number of game rounds players played during their first week playing the game.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {
    "dc": {
     "key": "17"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sum_gamerounds\n",
      "0    3994\n",
      "1    5538\n",
      "2    4606\n",
      "3    3958\n",
      "4    3629\n",
      "Name: userid, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Players')"
      ]
     },
     "execution_count": 244,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3XmcXFWZ//HP09X7vqazdRayAAmRrdkCbuzgAuOoAwMalTHjuDGjjoMz/mRGxxmdxV1RFDSoI6CIIoNgBlEQDSRhS0iAbITs6SS971X1/P64t5tKpzvdnXR1dVd9369Xv7ruqdtVz01Bf/ucc++55u6IiIiMVFaqCxARkclFwSEiIqOi4BARkVFRcIiIyKgoOEREZFQUHCIiMioKDhERGRUFh4iIjIqCQ0RERiU71QUkQ3V1tc+ZMyfVZYiITCpr16494O41w+2XlsExZ84c1qxZk+oyREQmFTPbPpL9NFQlIiKjouAQEZFRUXCIiMioKDhERGRUFBwiIjIqCg4RERkVBYeIiIyKgmMI7s49a3fS3h1NdSkiIhOKgmMI2w928PGfPsv9z+1OdSkiIhOKgmMIzZ29AOxv6U5xJSIiE4uCYwht4RDVgTYFh4hIIgXHEFq7+oKjJ8WViIhMLAqOIfRNijeoxyEichgFxxA0VCUiMjgFxxD6g6NVwSEikkjBMYS+OY6Wrijd0ViKqxERmTgUHENo6+7tf3xQE+QiIv0UHENo63r1ivEGDVeJiPRTcAyhrTuKWfBYE+QiIq9ScAyhtSvK9LICQMEhIpJIwTGEtu4oc6uLAF0EKCKSSMExhLbuKNXFuRTnZWuOQ0QkgYJjCG1dUYrysqkpydNQlYhIAgXHEFq7oxTnZ1NdnKvgEBFJkNTgMLOXzWydmT1jZmvCtkozW2lmm8LvFWG7mdnXzGyzmT1nZmckvM6ycP9NZrYsmTUDdEdj9ETjlORlU12cpzkOEZEE49HjeKO7n+bu9eH2TcDD7r4AeDjcBrgCWBB+LQdugSBogJuBc4CzgZv7wiZZ2ruDK8WLw+DQHIeIyKtSMVR1FbAifLwCuDqh/Q4PrALKzWwacBmw0t0PuXsjsBK4PJkF9l38V5yfQ3VxHs2dvfRE48l8SxGRSSPZweHAb8xsrZktD9tq3X1P+HgvUBs+ngHsSPjZnWHbUO1J0xouN1Kcl011SS4AB9vV6xARAchO8utf4O67zGwKsNLMXkh80t3dzHws3igMpuUAs2bNOq7X6utxlORnv3r1eGsP08ILAkVEMllSexzuviv8vh+4l2COYl84BEX4fX+4+y6gLuHHZ4ZtQ7UPfK9b3b3e3etramqOq+6+JdX75jhAV4+LiPRJWnCYWZGZlfQ9Bi4F1gP3AX1nRi0Dfhk+vg94d3h21blAczik9RBwqZlVhJPil4ZtSdMfHPnZTCkJgkN3AhQRCSRzqKoWuNeCsZ5s4H/c/UEzWw3cbWY3ANuBd4b7PwBcCWwGOoD3Arj7ITP7HLA63O+z7n4oiXX3B0dJXjYl+TmAehwiIn2SFhzuvhU4dZD2g8BFg7Q78KEhXut24PaxrnEor55VlU1BboSi3IhOyRURCenK8UG0dUfJMijIiQBQXaKLAEVE+ig4BtHaFaU4L5twmC24elw9DhERQMExqLbuaP/cBqD1qkREEig4BtEW9jj6BOtVKThEREDBMai2cGXcPtXFeTR29NIb07IjIiIKjkG0dh/e46gJr+U41K4JchERBccg2rp6j+hxADolV0QEBceg2rqjlBzW4wgWOtTV4yIiCo5BDTY5DuiUXBERFBxHiMWd9p7YYUNVU0ryAdiv4BARUXAM1N7z6sq4fQpyI1QX57LjUEeqyhIRmTAUHAP0r1OVd/gyXjMrCtnRqOAQEVFwDJC4pHqiuspCdhzqTEVJIiITioJjgNYhehyzKgvY1dRJVBcBikiGU3AM0H8vjoE9jopCYnFnT3NXKsoSEZkwFBwDvDrHkXNYe11lIYDmOUQk4yk4Bmgfao6jIgiOnZrnEJEMp+AYoLV78DmOaeX5RLKMV3RKrohkOAXHAEOdjpsTyWJaWb6GqkQk4yk4Bmjr7qUwN0Iky454rq6iUBcBikjGU3AM0DZgSfVEsyoL2dGoOQ4RyWwKjgFau6JHTIz3qassoKG1m86e2DhXJSIycSg4Bhi4pHqivlNyd2qeQ0QymIJjgLaj9DhmVuhaDhERBccAw81xAFqzSkQymoJjgNau6BFXjfepLs6lICeiazlEJKMpOAZo644esU5VHzNjZkWBTskVkYym4Ejg7kcdqoJweXWdkisiGSzpwWFmETN72szuD7fnmtkTZrbZzO4ys9ywPS/c3hw+PyfhNT4Vtr9oZpclq9au3jixuA85OQ7BPMfOQx24e7LKEBGZ0Majx3EjsDFh+4vAl919PtAI3BC23wA0hu1fDvfDzBYB1wCLgcuBb5lZJBmFtnb3AkcuN5JoZkUBrd1Rmjp6k1GCiMiEl9TgMLOZwJuA74XbBlwI/CzcZQVwdfj4qnCb8PmLwv2vAu5092533wZsBs5ORr1961QNNccBWl5dRCTZPY6vAJ8E+m6bVwU0uXs03N4JzAgfzwB2AITPN4f797cP8jP9zGy5ma0xszUNDQ3HVGzbECvjJtIpuSKS6ZIWHGb2ZmC/u69N1nskcvdb3b3e3etramqO6TWmlxfwxT9fwsnTSofcp6/HoVNyRSRTDf2n9fE7H3irmV0J5AOlwFeBcjPLDnsVM4Fd4f67gDpgp5llA2XAwYT2Pok/M6aqi/P4i7NmHXWf4rxsZlYU8MyOxmSUICIy4SWtx+Hun3L3me4+h2By+7fufh3wCPD2cLdlwC/Dx/eF24TP/9aDU5fuA64Jz7qaCywAnkxW3SNx3glVrNp6iHhcZ1aJSOZJxXUc/wB8zMw2E8xh3Ba23wZUhe0fA24CcPfngbuBDcCDwIfcPaXL0y6dX0VzZy8b9rSksgwRkZRI5lBVP3f/HfC78PFWBjkryt27gHcM8fOfBz6fvApH57wTqgFYtfUgp8woS3E1IiLjS1eOH4OpZfmcUF3EH7ccTHUpIiLjTsFxjM6dV8WT2w4RjcWH31lEJI0oOI7R0nlVtHVHWberOdWliIiMKwXHMTr3hCoA/rRVw1UiklkUHMeoujiPE2tL+JPmOUQkwyg4jsN586pY/fIheqKa5xCRzKHgOA7nzauiqzfOMzuaUl2KiMi4UXAch3PnVmGGhqtEJKMoOI5DWWEOc6qKeHGfriAXkcyh4DhOdZWFWilXRDKKguM4zaos0L05RCSjKDiO06zKQpo7e2nWrWRFJEMoOI7TLN1KVkQyjILjOOmOgCKSaRQcx0nBISKZRsFxnErzcygvzFFwiEjGUHCMgVmVhexQcIhIhlBwjIE6BYeIZBAFxxiYVVnIzsZOYnFPdSkiIkmn4BgDsyoLicadPc26EFBE0p+CYwzM0plVIpJBFBxjoC84dmrpERHJACMKDjO70cxKLXCbmT1lZpcmu7jJYlpZPpEsU49DRDLCSHsc73P3FuBSoAJ4F/CFpFU1yWRHsphenq/gEJGMMNLgsPD7lcAP3f35hDYhGK5ScIhIJhhpcKw1s98QBMdDZlYC6EbbCXQRoIhkiuzhdjAzAz4D1ABb3b3DzKqA9ya7uMmkrrKQg+09tHdHKcob9p9VRGTSGrbH4e4OPODuT7l7U9h20N2fO9rPmVm+mT1pZs+a2fNm9i9h+1wze8LMNpvZXWaWG7bnhdubw+fnJLzWp8L2F83ssuM43qTR8uoikilGOlT1lJmdNcrX7gYudPdTgdOAy83sXOCLwJfdfT7QCNwQ7n8D0Bi2fzncDzNbBFwDLAYuB75lZpFR1pJ0dRXhtRwHFRwikt5GGhznAKvMbIuZPWdm68zsqD0OD7SFmznhlwMXAj8L21cAV4ePrwq3CZ+/KBwmuwq409273X0bsBk4e4R1jxtdBCgimWKkg/HHNDwU9gzWAvOBbwJbgCZ3j4a77ARmhI9nADsA3D1qZs1AVdi+KuFlE39mwigvzKEkL1vBISJpb0Q9DnffDtQRDD1tBzpG8rPuHnP304CZBL2Ek46j1qMys+VmtsbM1jQ0NCTrbY72/iyZWcYD6/bQ1NEz7u8vIjJeRnrl+M3APwCfCptygB+N9E3CSfVHgPOAcjPr6+nMBHaFj3cRhBPh82XAwcT2QX4m8T1udfd6d6+vqakZaWlj6tNvWkRjRy+f/9+NKXl/EZHxMNI5jj8D3gq0A7j7bqDkaD9gZjVmVh4+LgAuATYSBMjbw92WAb8MH98XbhM+/9vwjK77gGvCs67mAguAJ0dY97haNL2Uv37dCfx07U7+sOlAqssREUmKkQZHT/hL3AHMrGgEPzMNeCScRF8NrHT3+wl6Lh8zs80Ecxi3hfvfBlSF7R8DbgIIr1K/G9gAPAh8yN1jI6x73H30ogXMrS7iH+9dR2fPhC1TROSYWZAHw+xk9gmCv/QvAf4deB/wP+7+9eSWd2zq6+t9zZo1KXv/VVsPcs2tq/ibN8zjHy5P2rSOiMiYMrO17l4/3H4jnRz/L4JTZO8BTgQ+M1FDYyI494Qq3nBiDf+3YV+qSxERGXMjOh3XzD4C/MjdVya5nrRxyvQyHtt0gO5ojLzsCXe9oojIMRvpHEctsNrM7jazy8ML8+QoFk4tIRZ3tuxvT3UpIiJjaqRDVZ8mmOO4DXgPsMnM/s3M5iWxtkntpKnBSWcv7WtNcSUiImNrxLeODc+q2ht+RQlu6PQzM/uPJNU2qc2tLiInYrywV8EhIullpHMcNwLvBg4A3wP+3t17zSwL2AR8MnklTk45kSzm1RTz4t6WVJciIjKmRrpWVSXwtnC5kX7uHjezN499WenhxKklrN52KNVliIiMqZHOcdzs7tvNbIqZzer7Cp/T+hpDOHFqCbubu2jp6k11KSIiY2aka1W9xcw2AduA3wMvA79OYl1poX+CXPMcIpJGRjo5/q/AucBL7j4XuIjDlzqXQSysDYJDE+Qikk5GGhy97n4QyDKzLHd/BBj2svRMN6O8gJK8bF5UcIhIGhnp5HiTmRUDjwI/NrP9hCvlytDMjIVTS3hR13KISBoZaY/jKqAT+DuCFWq3AG9JVlHpZGFtCS/ubWUki0mKiEwGIz2rqj28m1/U3Ve4+9fCoSsZxklTS2ju7GVfS3eqSxERGRNHHaoys1aCe3D0rU3V92ezEVxMXprE2tLCiVP7JshbmFqWn+JqRESO31F7HO5e4u6l4fe+x33bCo0ROLFWa1aJSHoZrseRD3wAmA88B9zu7tHxKCxdVBTlMqUkT6fkikjaGG6OYwXBabfrgCuB/056RWno5Gml/GnLQVp1BbmIpIHhgmORu1/v7t8B3g68dhxqSjsfvnA++1u7uenn63R2lYhMesMFR/+fyBqiOnZnzank45cu5H+f28OPnngl1eWIiByX4S4APNXM+tYFN6Ag3NZZVaP0gdfN48lth/jcrzZwel05p8woS3VJIiLHZLizqiIDzqTK1llVxyYry/jSO0+jsiiXj939jIasRGTSGvEdAOX4VRbl8tGLFvDSvjY27NENnkRkclJwjLPLFtcSyTIeWLcn1aWIiBwTBcc4qyrO49wTKnlg3V4NV4nIpKTgSIErl0xj24F2Nu7RRYEiMvkoOFLgssVTyTI0XCUik5KCIwWqi/M494QqHli3R8NVIjLpJC04zKzOzB4xsw1m9ryZ3Ri2V5rZSjPbFH6vCNvNzL5mZpvN7DkzOyPhtZaF+28ys2XJqnk8XblkGlsPtOsmTyIy6SSzxxEFPu7uiwjuV/4hM1sE3AQ87O4LgIfDbYArgAXh13LgFgiCBrgZOAc4G7i5L2wms/7hquc0XCUik0vSgsPd97j7U+HjVmAjMIPgboIrwt1WAFeHj68C7vDAKqDczKYBlwEr3f2QuzcCK4HLk1X3eKkpyePsuZX8r+Y5RGSSGZc5DjObA5wOPAHUunvfb8u9QG34eAawI+HHdoZtQ7UPfI/lZrbGzNY0NDSMaf3J8qYl09jS0M4mDVeJyCSS9OAws2LgHuBv3f2wy6U9mBkek9lhd7/V3evdvb6mpmYsXjLpLl08FYAH1+9NcSUiIiOX1OAwsxyC0Pixu/88bN4XDkERft8ftu8C6hJ+fGbYNlT7pFdbms+Zsyv4tYJDRCaRZJ5VZcBtwEZ3/1LCU/cBfWdGLQN+mdD+7vDsqnOB5nBI6yHgUjOrCCfFLw3b0sIVp0xlw54WXjnYkepSRERGJJk9jvOBdwEXmtkz4deVwBeAS8xsE3BxuA3wALAV2Ax8F/gggLsfAj4HrA6/Phu2pYXL+oarntckuYhMDsPdj+OYufsfCO7bMZiLBtnfgQ8N8Vq3A7ePXXUTR11lIafMKOXB9XtZ/rp5qS5HRGRYunJ8Arh88VSeeqWJvc1dqS5FRGRYCo4J4PJTpgHwmw2aJBeRiU/BMQHMn1LM/CnF/HqdgkNEJj4FxwRx5ZJpPLHtII9tmhwXL4pI5lJwTBDLX3cCC2tL+OCPnuLFvbqSXEQmLgXHBFGcl83t7zmLgtwI7/vBava3aqJcRCYmBccEMr28gNvfcxaH2nt4/4o1tHVHU12SiMgRFBwTzCkzyvj6taezfncL77n9SYWHiEw4Co4J6OJFtXzj2tN5ekcTy8LwiMed9bua+cXTu+joUZiISOok7cpxOT5XLJnGN4AP/+Rp3vr1P9DSFeVAWzcAp9aVc/uyeqqK81JbpIhkJPU4JrArlkzjm395OjmRLJbOq+K/33EqX/mL03hhTwt/fssf2X6wPdUlikgGsmCJqPRSX1/va9asSXUZSbN2eyM3rFhNdpZx27KzOLWuPNUliUgaMLO17l4/3H7qcUxCZ86u4J6/WUp+ToRrbl3Fyg37Ul2SiGQQBcckNa+mmHs/eD4La4tZ/sM1/ODxbakuSUQyhIJjEqspyePO5edxycm1/POvNvCd329JdUkikgEUHJNcQW6EW64/kzctmcYXHnyB3724f/gfEhE5DgqONBDJMv7zHa/hpKmlfOQnT7PtgM62EpHkUXCkicLcbG5915lkZxnvv2MNrV29qS5JRNKUgiON1FUW8s3rzmDbgXauv+1JGlq7U12SiKQhBUeaWTqvmm9ffyYv7m3hz771OJv3a4l2ERlbCo40dMmiWu5afh5dvXHe9q0/8qtndxOPp9+FniKSGgqONHVqXTn3fnApMyoK+chPnubSrzzKL57eRTQWT3VpIjLJKTjSWF1lIfd/5AK+fu3pRMz427ue4S9uXaW5DxE5LgqONBfJMt5y6nR+feNr+dI7T+X53c1c/c3H2bC7JdWlicgkpeDIEFlZxtvOmMlP/3opsbjz9m//kUde0MWCIjJ6Co4Ms2RmGfd9+HzmVBXx0TufZk9zZ6pLEpFJRsGRgaaU5nPL9WcQjTn/cM860nFpfRFJnqQFh5ndbmb7zWx9Qlulma00s03h94qw3czsa2a22cyeM7MzEn5mWbj/JjNblqx6M83sqiI+deVJPPpSA3eu3pHqckRkEklmj+MHwOUD2m4CHnb3BcDD4TbAFcCC8Gs5cAsEQQPcDJwDnA3c3Bc2cvyuP2c2S+dV8a/3b2BnY0eqyxGRSSJpweHujwKHBjRfBawIH68Ark5ov8MDq4ByM5sGXAasdPdD7t4IrOTIMJJjlJVlfPHPXwPAG//rdyz+zIMs+eeHeO/3n6SrN5bi6kRkosoe5/erdfc94eO9QG34eAaQOF6yM2wbql3GSF1lISvedza/2bCPWNzp6Ily5+odfOzuZ/jGtWeQlWWpLlFEJpjxDo5+7u5mNmazsma2nGCYi1mzZo3Vy2aE+jmV1M+p7N8+obqYzz+wkf+ofJGbrjgphZWJyEQ03mdV7QuHoAi/911IsAuoS9hvZtg2VPsR3P1Wd6939/qampoxLzyT/NVr53LdObP49u+38OMntqe6HBGZYMY7OO4D+s6MWgb8MqH93eHZVecCzeGQ1kPApWZWEU6KXxq2SRKZGf/y1sW8fmEN/3Tvej521zM0dfSkuiwRmSCSeTruT4A/ASea2U4zuwH4AnCJmW0CLg63AR4AtgKbge8CHwRw90PA54DV4ddnwzZJsuxIFre++0w+euF87nt2Nxd/6VEeXL831WWJyARg6XjxV319va9ZsybVZaSNDbtb+OQ9z7J+Vwv/9mdL+MtzNIckko7MbK271w+3n64cl2Etml7KPX+zlAtPmsI/3ruOO598JdUliUgKpeysKplc8rIj3HL9Gfz1D9dy08/X0d4To352BTmRLKqLc5lSmp/qEkVknCg4ZMTysiN8+/ozWf7DtXzu/g397ZEs45brzuDSxVNTWJ2IjBfNccio9cbirH75EF29MXqizrd+t5kt+9u490Pns7C2JNXlicgx0hyHJE1OJIul86q58KRaLj9lKt9515kU5GbzVyvW0NjeQzzu/P6lBv79gY28tK811eWKyBhTj0PGxNrtjVx76yoW1BbT0tXLjkPBfT7ysrP4f29exHXnzMJMy5eITGTqcci4OnN2BZ//s1PYuKeFGeUFfO3a03n8pgs554QqPv2L9XzgR2s50KZ7nYukA/U4ZEx19sQoyI30b8fjzu2Pb+OLD75Afk6EGy9awLKlcwD4w6YDrNy4j+ll+VyxZBrzaorp6Inyq2d387O1OynJz+GGC+aydF6Veisi42CkPQ4Fh4yLzfvb+Nz9G/j9Sw3MriqkubOXpo5eCnMjdPQES7gvrC1mT1MXrd1RFkwpprGjlwNt3SyaVsqfnzmTU6aXcvL0Ukrzc1J8NCLpScGh4Jhw3J3fvrCfb/1uCzMrCnjLa6bz2oXVHGrv4dfr9rJywz6mluVz3TmzOHN2Bd3ROPc9s5vvPraVTfvb+l9nSkkeU0rzmFKSz/wpxVx12nQWTy9L4ZGJpAcFh4Ijrexv6eL5PS1s2N3CywfaaWjrZn9LN5v3t9ETi3PS1BJet7CGnmicjp4oedkR3nradOpnVxw2zOXuGvYSGYKCQ8GREZo6evjVc3u4Z+1O1u9qpiA3QlFuNi1dvXT0xDihpoiLT67llYMdPL+nmX0t3bxpyTRuuGAup8xQL0UkkYJDwZHROnqi3P/cHu5avYOnXmlkdmUhi6eXUZyXzf3P7aa9J8bZcyr58IXzee2CavVCRFBwKDikXzQWJzvy6pnnLV293L16B7f/YRu7m7s4e04lH7loPpVFuRxq76G1K8r586opK9QkvGQWBYeCQ4bRHY1x1+odfOO3m9nfevg1JtXFuXzmLYt5y2umqTciGUPBoeCQEerqjfF/G/eRnWVUFuURiztf+PVGnt3ZzOsX1rB0XhVdvXG6ozFmVhRy9twK5tUU9wdKPO60dkVp6uyhqaOX9u4oPbE4vTHnhJoi5tUUp/gIRUZGwaHgkOMQizs//NPL/NdvXqKtOwoEqwDH4sH/L5VFuZTmZ9PU2UtzZy9D/W+Um53Fd99dz+sX1oxT5SLHTsGh4JAx0B2NEY05edlZRLKMbQfaWf3yIda83EhXNE5FYQ7lBTmUFuRQUZhLeWEOhbnZ5GZnYQafvnc9mxvaDguPaCzOjsZOGjt6aO7opTcWZ2FtCbMqC8nK0rCYpI6CQ8EhE0Bjew/Xfe8JNje0ceNFC9i4p4XHNh2gubP3iH2L87I5aWoJdZWFTC/PZ1pZAdXFeVQX51JWkEN7T4ymjh7auqOcOrOcusrCFByRpDMFh4JDJoi+8Niwp4Wakjxev7CGc+ZWUl2SR3lBDmbGi3tbeH53Cy/sbWVXYyd7W7r6h8WGctLUEi5ZVMvi6WXMqixkZmUB3b1x9jZ3sa+liznVRcyfovkVGTkFh4JDJpDuaIydjZ3MrSoa0XBULO40tHZzoK2bg+09NHf2UpwXoawgl7zsLFZtPchvNuxjzcuHOFq+nDytlLecOo2Tp5bSE4vTE40TT/h/vq6ykNPrynXmmAAKDgWHZISWrl5ePtDOjkOd7GzsID8nwtSyfGpK8nh2RxO/enY3T73SdNTXWDKjjPddMIfXLqhh24F2XtrXSmtXsNDkSdNKmV6Wf1iwNHf0sn53MzsbO1g6r/qoQ2atXb0U5WZr7maSUHAoOEQA2NXUyYHWbnIiWeRErP+XuDus2nqQ2x/fxtaG9iF/Pj8ni+K8HIryIsTizs7GzsOeP31WOVecMpUpJfnkRLKIufPU9kb+uOUAL+1ro6ool/PmVbF0XjW1pXlkZRnZWcaM8gLmjLAHJuNDwaHgEBmReNx5dFMDm/e3Ma+mmAW1xZTk57BpXysv7G1l+8F22ntidHRHiXkwt/KamWXUlubz8Mb93PfsbjbuaTnsNfNzsjhrTiX1syvZfrCdx7ccYF/LkTfyKsnLZvGMUqaXF5CfE6EgJ0J25NUgKciJUFOSR01xHlPL8pleXkBVUW5/D8jdiXtwqvRQ3J0DbT1UFeUqpIah4FBwiIyb/S1dtPfE6I3FicWDCx/zsl+9oZe7s/1gBy1dvcTiTjTubGto57ldTazb2cyBth66ozE6e2JEEyZtuqPxI94rNzuLsoIcOntitPdEyTJjZkUBs6uKmF1Z2L/sfiQrmAt6bFMD+1q6Kc3P5tS6cl4zs4yppfmUFeZSkpfN7uZOtja0s/1gB7OrCjl/fhVnz62iOC/7qMfc1BEsT1Oan0NxfvZRw+tYtXdHKcyNjNsclIJDwSEy6fXG4hxq76GhtZs9zV3sbupkV1MnLZ29FOZmU5QXIR6G0vaDHbxyqOOwU53LCnK4YH41p9WVs/VAO8/saOLFvS1HnFBQkBNhZkUB2w910BONE8myw4KjrCCH6eVBj6e1K8qG3S3sajp8yK6iMIe6ykLqKgupLMylqzdGVzRONBYny4IhwsKcCPOmFLGgtoQTqosozc+hMC9CbiSrPxxiceeRF/Zzx6rtPPpSAzMrCrj45FouPrmWyqJcemNxemPx/tUJeqNxCnIjlBXkUF6YQ2VRLoW5Rw+9oSg4FBwiGak7GqOhtZuOnhjzaoqP6An0xuI0dfTS3NlDc2eUqWX5TCu3zSaYAAAKn0lEQVTNJyvL6OqN8dT2RlZtPUhLV7BigLvT2NHLrqZOdjd1UpAbYfH0MhZPL6WyMJeWrl5au6I0tHWz41AHO8Lwys+JkJed1T/vE487bd1RDrT1HFFzdpZRmBuhKC+b3licA2091JbmcfVpM9jS0MZjmw4M2vsazJVLpvKt6848pn+7kQbHscWSiMgElZcdYWbF0Gd65USygnmTkrwjnsvPibB0fjVL51cnrb7G9h42N7Tx8oF22rujtPfEaO+O0tETo6MnSjTmXLyolksW1ZITrurc0RPliW2H6O6Nk5ttZGdlkRuGUk7E6OqN09QRrJU2tSw/abX3mTTBYWaXA18FIsD33P0LKS5JRGTUKopyOauokrPmVI74Zwpzs3njiVOSWNXoZA2/S+qZWQT4JnAFsAi41swWpbYqEZHMNCmCAzgb2OzuW929B7gTuCrFNYmIZKTJEhwzgB0J2zvDNhERGWeTJTiGZWbLzWyNma1paGhIdTkiImlrsgTHLqAuYXtm2NbP3W9193p3r6+p0U1zRESSZbIEx2pggZnNNbNc4BrgvhTXJCKSkSbF6bjuHjWzDwMPEZyOe7u7P5/iskREMtKkCA4Ad38AeCDVdYiIZLq0XHLEzBqA7cfxEtXAgTEqZ7LIxGOGzDxuHXPmGO1xz3b3YSeJ0zI4jpeZrRnJei3pJBOPGTLzuHXMmSNZxz1ZJsdFRGSCUHCIiMioKDgGd2uqC0iBTDxmyMzj1jFnjqQct+Y4RERkVNTjEBGRUVFwJDCzy83sRTPbbGY3pbqeZDCzOjN7xMw2mNnzZnZj2F5pZivNbFP4vSLVtSaDmUXM7Gkzuz/cnmtmT4Sf+V3hygRpw8zKzexnZvaCmW00s/My4bM2s78L//teb2Y/MbP8dPyszex2M9tvZusT2gb9fC3wtfD4nzOzM471fRUcoQy650cU+Li7LwLOBT4UHudNwMPuvgB4ONxORzcCGxO2vwh82d3nA43ADSmpKnm+Cjzo7icBpxIce1p/1mY2A/goUO/upxCsNnEN6flZ/wC4fEDbUJ/vFcCC8Gs5cMuxvqmC41UZcc8Pd9/j7k+Fj1sJfpHMIDjWFeFuK4CrU1Nh8pjZTOBNwPfCbQMuBH4W7pJWx21mZcDrgNsA3L3H3ZvIgM+aYFWMAjPLBgqBPaThZ+3ujwKHBjQP9fleBdzhgVVAuZlNO5b3VXC8KuPu+WFmc4DTgSeAWnffEz61F6hNUVnJ9BXgk0A83K4Cmtw9Gm6n22c+F2gAvh8Oz33PzIpI88/a3XcB/wW8QhAYzcBa0vuzTjTU5ztmv+MUHBnKzIqBe4C/dfeWxOc8ONUurU63M7M3A/vdfW2qaxlH2cAZwC3ufjrQzoBhqTT9rCsI/rqeC0wHijhyOCcjJOvzVXC8ath7fqQLM8shCI0fu/vPw+Z9fd3W8Pv+VNWXJOcDbzWzlwmGIS8kGP8vD4czIP0+853ATnd/Itz+GUGQpPtnfTGwzd0b3L0X+DnB55/On3WioT7fMfsdp+B4VUbc8yMc178N2OjuX0p46j5gWfh4GfDL8a4tmdz9U+4+093nEHy2v3X364BHgLeHu6XVcbv7XmCHmZ0YNl0EbCDNP2uCIapzzaww/O+977jT9rMeYKjP9z7g3eHZVecCzQlDWqOiCwATmNmVBOPgfff8+HyKSxpzZnYB8BiwjlfH+v+RYJ7jbmAWwcrC73T3gZNuacHM3gB8wt3fbGYnEPRAKoGngevdvTuV9Y0lMzuN4GSAXGAr8F6CPxjT+rM2s38B/oLgLMKngb8iGM9Pq8/azH4CvIFgFdx9wM3ALxjk8w1D9BsEw3YdwHvdfc0xva+CQ0RERkNDVSIiMioKDhERGRUFh4iIjIqCQ0RERkXBISIio6LgkHFnZm5m/52w/Qkz++cxeu0fmNnbh9/zuN/nHeFqs48MaJ9jZp1m9ky4AvEd4QWXya6nbRT7/i5cBfpZM3u87zqPsD1p9+U2szf0rUosk5uCQ1KhG3ibmVWnupBECVcVj8QNwPvd/Y2DPLfF3U8DlhBcnfvOsahvjF3n7qcSLIL3n6kuRiYXBYekQpTglpZ/N/CJgT2Gvr+kw79Wf29mvzSzrWb2BTO7zsyeNLN1ZjYv4WUuNrM1ZvZSuEZV3304/tPMVof3IvjrhNd9zMzuI7i6eGA914avv97Mvhi2fQa4ALjNzIb8pevuMeBJwoXkLLgnxPfD13vazN4Ytr/HzL6R8J73hxcpYmZtZvb5sHewysxqw/a5Zvan8LX+NeFnp5nZo2GPZ72ZvfaonwQ8Cswf5LhvCf8Nnw8vpsPMLjSzXyTsc4mZ3Rs+vjSs5ykz+6kFa6H13ePmBTN7CnjbMLXIJKHgkFT5JnCdBUt/j9SpwAeAk4F3AQvd/WyCK6M/krDfHIJl8t8EfNvM8gl6CM3ufhZwFvB+M5sb7n8GcKO7L0x8MzObTnAPhwuB04CzzOxqd/8ssIbgr/a/H6rY8H3PAR4Mmz5EsO7cEuBaYEW4z9EUAavC3sGjwPvD9q8SLF64hGAF2D5/CTwU9nhOBZ4Z5vXfQrCKwED/5O71wGuA15vZawiW7DjJzGrCfd4L3B72HD8NXOzuZxD823wsPLbvhu9xJjB1mFpkklBwSEqEK/LeQXDDnZFaHd5PpBvYAvwmbF9HEBZ97nb3uLtvIlhm4yTgUoJ1ep4hWF6liuCGNgBPuvu2Qd7vLOB34WJ5UeDHBPe3GM688H32AXvc/bmw/QLgRwDu/gLBchALB3+Jfj1A37zA2oTjPB/4Sfj4hwn7rwbeG84ZLQnvuTKYH4c1ng98YpDn3xn2Ep4GFgOLwpVWfwhcb2blwHnArwluCLYIeDx8zWXAbIJ/923uvin82R8Nc6wySYxmTFdkrH0FeAr4fkJblPAPGjPLIlhjqU/iukLxhO04h/+3PHAdHQcM+Ii7P5T4RDgk1H5s5Q9pi7ufFv4l/riZvdXdj7ZgZv8xhxJ7Ib3+6rpAMY5+nLj7o2b2OoLe1g/M7Evufscg73ndUOsUhT2xTwBnuXujmf0goabvA78CuoCfuns0XANppbtfO+B1Thv8cGWyU49DUiZcWO9uDr+F58sEwxoAbwWO5Yykd5hZVjjvcQLwIvAQ8Dd9ZziZ2UILbmp0NE8SDNNUW3Br4WuB34+0CHc/QHD/i0+FTY8B1/W9P8EidC8SHPNpYc11BMNsw3mcYJVf+l4zfN3ZwD53/y7BEN6x3Fe6lCBMm8M5lSsSjmk3sJtgaKov8FcB55vZ/LCGovD4XgDmJMw/HRYsMnkpOCTV/ptgZc8+3yX4Zf0swVDIsfQGXiH4pf9r4APu3kXwS3QD8JSZrQe+wzA97nDJ6ZsIxvafBda6+2iX4v4FUBhOUn8LyDKzdcBdwHvCYbfHgW1hfV8j6IUN50aC+8Wv4/C7uL0BeNbMniZYHfaro6wXd3+WYIjqBeB/wvoS/RjY4e4bw/0bgPcAPzGz54A/ASeF/+7Lgf8Nh73S7b4fGUur44rIqIRngD3t7reluhZJDQWHiIyYma0l6AVeMtnvZSHHTsEhIiKjojkOEREZFQWHiIiMioJDRERGRcEhIiKjouAQEZFRUXCIiMio/H95sd6bqk46EwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f44e556bb70>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# This command makes plots appear in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "# Counting the number of players for each number of gamerounds \n",
    "plot_df = df.groupby('sum_gamerounds')['userid'].count()\n",
    "print(plot_df.head())\n",
    "# Plotting the distribution of players that played 0 to 100 game rounds\n",
    "ax = plot_df.head(n=100).plot(x='sum_gamerounds', y='userid')\n",
    "ax.set_xlabel(\"Number of Rounds Played\")\n",
    "ax.set_ylabel(\"Players\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "25"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "## 4. Overall 1-day retention\n",
    "<p>In the plot above we can see that some players install the game but then never play it (0 game rounds), some players just play a couple of game rounds in their first week, and some get really hooked!</p>\n",
    "<p>What we want is for players to like the game and to get hooked. A common metric in the video gaming industry for how fun and engaging a game is <em>1-day retention</em>: The percentage of players that comes back and plays the game <em>one day</em> after they have installed it.  The higher 1-day retention is, the easier it is to retain players and build a large player base. </p>\n",
    "<p>As a first step, let's look at what 1-day retention is overall.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {
    "dc": {
     "key": "25"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4452095044850259"
      ]
     },
     "execution_count": 246,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The % of users that came back the day after they installed\n",
    "df['retention_1'].sum() / df['retention_1'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "32"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "## 5. 1-day retention by AB-group\n",
    "<p><img src=\"https://s3.amazonaws.com/assets.datacamp.com/production/project_184/img/belle_cookie.png\" style=\"width:200px; float:right\"> </p>\n",
    "<p>So, a little less than half of the players come back one day after installing the game. Now that we have a benchmark, let's look at how 1-day retention differs between the two AB-groups.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {
    "dc": {
     "key": "32"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "version\n",
       "gate_30    0.448188\n",
       "gate_40    0.442283\n",
       "Name: retention_1, dtype: float64"
      ]
     },
     "execution_count": 248,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculating 1-day retention for each AB-group\n",
    "df.groupby('version')['retention_1'].sum() / df.groupby('version')['retention_1'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "39"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "## 6. Should we be confident in the difference?\n",
    "<p>It appears that there was a slight decrease in 1-day retention when the gate was moved to level 40 (44.2%) compared to the control when it was at level 30 (44.8%). It's a small change, but even small changes in retention can have a large impact. But while we are certain of the difference in the data, how certain should we be that a gate at level 40 will be worse in the future?</p>\n",
    "<p>There are a couple of ways we can get at the certainty of these retention numbers. Here we will use bootstrapping: We will repeatedly re-sample our dataset (with replacement) and calculate 1-day retention for those samples. The variation in 1-day retention will give us an indication of how uncertain the retention numbers are.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {
    "dc": {
     "key": "39"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "version       gate_30   gate_40\n",
      "retention_1  0.448253  0.441670\n",
      "retention_1  0.447601  0.442712\n",
      "retention_1  0.450571  0.443210\n",
      "retention_1  0.448687  0.439539\n",
      "retention_1  0.448825  0.444698\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f44e59dceb8>"
      ]
     },
     "execution_count": 250,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAD8CAYAAABthzNFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8VOW5wPHfk52QQEISIEAgCWHfIQKKonWpWivaWiu2WrELdeu9t/be1t4uelv7sVXb3lqtVq9b6y601n2tiFUR2ZewBEKAhJCNLQtZ57l/nEkIkJBtZs7M5Pl+PvM5mTPnvOfJZHnmXc77iqpijDHGnCjC7QCMMcYEJ0sQxhhj2mUJwhhjTLssQRhjjGmXJQhjjDHtsgRhjDGmXZYgjDHGtMsShDHGmHZZgjDGGNOuKLcD6I3U1FTNzMx0OwxjjAkpq1evrlDVtM6OC+kEkZmZyapVq9wOwxhjQoqI7O7KcdbEZIwxpl2WIIwxxrTLEoQxxph2hXQfhDGm72lsbKSoqIi6ujq3Qwl6cXFxjBgxgujo6B6d77cEISKPAV8EylR1snff88A47yFJwCFVnS4imcAWYJv3tRWqeoO/YjPGhK6ioiISExPJzMxERNwOJ2ipKpWVlRQVFZGVldWjMvxZg3gCuB/4S8sOVb2q5WsR+S1wuM3xO1V1uh/jMcaEgbq6OksOXSAipKSkUF5e3uMy/JYgVHW5t2ZwEnF+sl8FzvXX9Y0x4cuSQ9f09n1yq5P6LKBUVfPb7MsSkbUi8oGInNXRiSKyWERWiciq3mRGY05Svg0+ug/W/BXqq9yOxhjXuZUgrgaebfO8BBipqjOAW4FnRGRAeyeq6sOqmququWlpnd4IaEzXfPxH+NNceOdn8PIt8OA8qMjv/Dxj2vHzn/+cd9991+0wei3gCUJEooAvA8+37FPVelWt9H69GtgJjA10bKaPWv8cvP1TmHAp/GA7XPcqNNbCM1+FusOdn2/6JFXF4/G0+9ovfvELzj///ABH5Htu1CDOB7aqalHLDhFJE5FI79fZwBigwIXYTF9zuAhe/T5kngVXPAqJQyDrLLjqKTiwC96/y+0IjZ/ddtttPPDAA63P77jjDu69917uueceTjvtNKZOncrtt98OQGFhIePGjeMb3/gGkydPZu/evSxatIjJkyczZcoUfv/73wOwaNEilixZAsB7773HjBkzmDJlCt/85jepr68HnKmCbr/9dmbOnMmUKVPYunVrgL/zzvktQYjIs8AnwDgRKRKRb3lfWsjxzUsA84ENIrIOWALcoKoH/BWbMa3e+TmoBy57ACLbjBUfORdmLYKVD8Ohva6FZ/zvqquu4oUXXmh9/sILL5CWlkZ+fj4rV65k3bp1rF69muXLlwOQn5/PTTfdxObNm6moqKC4uJhNmzaxceNGrr/++uPKrqurY9GiRTz//PNs3LiRpqYmHnzwwdbXU1NTWbNmDTfeeCP33ntvYL7hbvBbglDVq1U1XVWjVXWEqj7q3b9IVR864dilqjpJVaer6kxVfcVfcRnTqnw7bFoKp98MyaNOfv2sHzjbFQ+e/JoJGzNmzKCsrIx9+/axfv16kpOT2bhxI2+//TYzZsxg5syZbN26lfx8p09q1KhRzJ07F4Ds7GwKCgr43ve+x5tvvsmAAcd3nW7bto2srCzGjnVazK+77rrWRAPw5S9/GYBZs2ZRWFgYgO+2e2yqDdN3ffJHiIqDOTe2/3pSBky+AtY8CfXVgY3NBNSVV17JkiVLeP7557nqqqtQVX784x+zbt061q1bx44dO/jWt5xGkP79+7eel5yczPr16znnnHN46KGH+Pa3v92t68bGxgIQGRlJU1OT774hH7EEYfqm6nJY/zxMuxoSTjEaLveb0FANW18LXGwm4K666iqee+45lixZwpVXXsmFF17IY489RnW188GguLiYsrKyk86rqKjA4/FwxRVXcOedd7JmzZrjXh83bhyFhYXs2LEDgL/+9a+cffbZ/v+GfMTmYjJ908YXobke5nz31MdlzIGkkbDhOZh21amPNSFr0qRJVFVVMXz4cNLT00lPT2fLli2cfvrpACQkJPDUU08RGRl53HnFxcVcf/31raOZ7rrr+EENcXFxPP7441x55ZU0NTVx2mmnccMNoTOLkKiq2zH0WG5urtqCQaZH/jwfEPjuB50f+94v4F+/h//Mh/6pfg/NnNqWLVuYMGGC22GEjPbeLxFZraq5nZ1rTUym7ynbAiXrYdrCrh0//hJnpNOO9/wblzFBxhKE6Xs2LQWJgMlf6drx6TOg/2DIf8u/cRkTZCxBmL5n62sw8oxTd063FREBYy6AHe9Cc/CNNDHGXyxBmL7lwC4oy4PxX+jeeWM+70y7UfSZf+IyJghZgjB9y7bXne24biaI7LMBgV3LOz3UmHBhCcL0LVtfh8ETYVA3V9jqlwxDJsPuj/wTlzFByBKE6TvqDsOeT2DcxT07P3Me7F0JTQ2+jcuYIGUJwvQduz4EbYbR5/Xs/FHzoOko7Fvr27hM2HvppZfIy8vr0bkPPfQQU6ZMYfr06Zx55pnHlXPXXXeRk5PDuHHjeOst34+yswRh+o6CZRDdH0ac1rPzR81ztrv/5bOQTN/QmwTxta99jY0bN7Ju3Tp++MMfcuuttwKQl5fHc889x+bNm3nzzTe56aabaG5u9mXYNtWG6UMK3neaiaJienZ+/xRIHQt7bSRTsPifVzaTt++IT8ucOGwAt186qdPjfvnLX/LUU0+RlpZGRkYGs2bNYuDAgTz88MM0NDSQk5PDX//6V9atW8fLL7/MBx98wJ133snSpUsBuPnmmykvLyc+Pp5HHnmE8ePHt3udtjPE1tTUtK4z/Y9//IOFCxcSGxtLVlYWOTk5rFy5snV6EF+wGoTpGw7thcodkH1O78oZngvFqyGEp6gxvffZZ5+xdOlS1q9fzxtvvEHLlD9f/vKX+eyzz1i/fj0TJkzg0Ucf5YwzzmDBggXcc889rFu3jtGjR7N48WL++Mc/snr1au69915uuummU17vgQceYPTo0fzwhz/kvvvuA5x5oDIyMlqPGTFiBMXFxT79Pq0GYfqGgmXONvtzvStn+ExY/4yzEl1SRufHG7/qyid9f/joo4+47LLLiIuLIy4ujksvvRSATZs28dOf/pRDhw5RXV3NhRdeeNK51dXVfPzxx1x55ZWt+1pWmevIzTffzM0338wzzzzDnXfeyZNPPunbb6gDliBM31CwzJkuY3AvJ3kbPsvZFq+2BGFOsmjRIl566SWmTZvGE088wbJly046xuPxkJSUxLp167pd/sKFC7nxRmf9kuHDh7N377HVDouKihg+fHiPY2+PNTGZ8OfxOAki+xzwtt/22JDJEBnjJAjTZ82bN49XXnmFuro6qqurefXVVwGoqqoiPT2dxsZGnn766dbjExMTqaqqApw+haysLF588UUAVJX169d3eK2WlewAXnvtNcaMGQPAggULeO6556ivr2fXrl3k5+cze/Zsn36fVoMw4a9sM9RWwOheNi+B08E9dCoUr+n8WBO2TjvtNBYsWMDUqVMZMmQIU6ZMYeDAgfzyl79kzpw5pKWlMWfOnNaksHDhQr7zne9w3333sWTJEp5++mluvPFG7rzzThobG1m4cCHTpk1r91r3338/7777LtHR0SQnJ7c2L02aNImvfvWrTJw4kaioKB544IGT1qvoNVX1ywN4DCgDNrXZdwdQDKzzPr7Q5rUfAzuAbcCFXbnGrFmz1JhOfXSf6u0DVA8X+6a81/5L9c501eYm35RnuiUvL8/tEFRVtaqqSlVVa2pqdNasWbp69WqXI2pfe+8XsEq78D/Wn01MTwAXtbP/96o63ft4HUBEJgILgUnec/4kIj5OhabP2vk+pI6DAcN8U97wmdBYAxXbfVOeCUmLFy9m+vTpzJw5kyuuuIKZM2e6HZLP+a2JSVWXi0hmFw+/DHhOVeuBXSKyA5gNfOKn8Exf0VQPuz+Gmd/wXZlDJjvb/Zt63+ltQtYzzzzj0/J+9atftfZLtLjyyiv5yU9+4tPrdIcbfRC3iMg3gFXAD1T1IDAcWNHmmCLvPmN6Z++nzvQYvuh/aJE61umoLt0IXNnp4cZ0xU9+8hNXk0F7Aj2K6UFgNDAdKAF+290CRGSxiKwSkVXl5eW+js+Em53vg0QemybDF6JiIG2cU4MwJowFNEGoaqmqNquqB3gEpxkJnI7rtoPKR3j3tVfGw6qaq6q5aWldXBHM9F0Fy5y5l+IGdHpotwyZAqWWIEx4C2iCEJH0Nk+/BLT8hb0MLBSRWBHJAsYAKwMZmwlDtQecmVezz/F92UMnQ3UpVFst1oQvv/VBiMizwDlAqogUAbcD54jIdECBQuC7AKq6WUReAPKAJuBmVfXttISm7yn8EFDf9j+0aOmoLt0ICef6vnxjgoDfahCqerWqpqtqtKqOUNVHVfVaVZ2iqlNVdYGqlrQ5/leqOlpVx6nqG/6Ky/QhO9+HmMRj02P40tApztb6IUwX9Ga67xZLly5FRFonBgRbD8KYnitYBplnQmS078uOHwSJw2D/Rt+XbcJObxNEVVUVf/jDH5gzZ07rPlsPwpieOlgIB3fB3Bv9d42hgeuobmz2sLuyhviYKNIHxrWuCdDnvXGb75P00Clw8a87PSxQ60EA/OxnP+NHP/oR99xzT+s+Ww/CmJ7a+b6zzT7Hf9cYMhEq8qG50X/XAJasLmL2r97l/N8t54xf/5PP3buMB5ft5Eidf69rOhbI9SDWrFnD3r17ueSSS47bb+tBGNNTBcucJqDUsf67Rtp48DTCgV2Q5p/rvPDZXn64dAOzMwexcHYGR4428tbmUn7z5lb+9P4Orp+XybfnZzMgzg/NaKGgC5/0/SFQ60F4PB5uvfVWnnjiCb98H52xBGHCj6cZdn0A477Q++m9TyVtnLMt3+qXBLH3QC23v7yZeTkpPL5oNjFRToV/0bwsNhUf5oH3d3DfP3fw5Ce7ueHs0Sw6I5N+MTaFmZt8vR5EVVUVmzZt4pxzzgFg//79LFiwgJdfftnWgzCmR/ZvgKMH/du8BMdqJ+Xb/FL8fe/l06zKPV+Z1pocWkwePpAHr5nFq987k5kjk/jNm1uZf8/7PPlxIfVNNkLc3wK1HsTAgQOpqKigsLCQwsJC5s6dy8svv0xubm5A1oOwBGHCTyD6HwBi+kPSSKcG4WPlVfX8bW0xX5s9kmFJ/To8bvLwgTx+/WyW3HA6Wan9uf3lzZx77we8umGfz2Myx7RdD+Liiy8+aT2IefPmHdfpvHDhQu655x5mzJjBzp07efrpp3n00UeZNm0akyZN4h//+Ee3Y2i7HsRFF13kl/UgREN48fXc3FxtOybYGACeXAA1FXDTx/6/1tNXwpESuPFfPi32/z4s4M7XtvDurfPJGZzYpXNUlQ/zK7j7ra1sKj7CgmnD+OXlkxnYL7z6J7Zs2cKECe7PoltdXU1CQgK1tbXMnz+fhx9+OCin/G7v/RKR1aqa29m51gdhwkvjUdizAmZ/JzDXSxsHBR84/R4Rvvv0tmR1EdMzkrqcHABEhPlj0zhjdAoPfbCT/303n637j/DYotMYkRzvs9iMY/HixeTl5VFXV8d1110XlMmhtyxBmPCy5xNorodsP0yv0Z608c71Du2GQdk+KbKwooat+6u4/dKJPTo/KjKCW84dw8yRyXz3qdV86U8f8+x35pIzOMEn8RmHrQdhTKjZ+b6zVsMo390sdEqpLSOZtvksQXyw3ZkA8Nzxg3tVzhk5qSy98Qy+9sgKrn30U1684fSwqUmoatjdLOiP9SB624VgndQmvBS8DxlznA7kQGgZ3urDjupl28rITIlnVErvv4exQxL567fmUFPfxLWPruRwbejfXBcXF0dlZWWv//mFO1WlsrKSuLi4HpdhNQgTPmoqnGkXzv1p4K4ZN9C5Ic9HQ10bmjx8UlDJVbkZnR/cRRPSB/DYotO4+pEV3PLsGh5fdBpRkaH72XDEiBEUFRVhC4Z1Li4ujhEjRvT4fEsQJnwULHO22QGefjttnM9qEHklR6hr9DA7K8Un5bXIzRzEnZdP5kdLN/KbN7fyk0t61r8RDKKjo8nKynI7jD4hdD9GGHOigmXOJ/ph0wN73bTxUL4dfNDksWb3QQBmjkrqdVknuuq0kVx3+ige+XAXf19b5PPyTfixBGHCg6qTILLm+3S4aZek5kBjDVTt73VRa/YcZNjAONIHdnxzXG/89IsTmZM1iNuWbmRT8WG/XMOED0sQJjwcKIDDewM3vLWtQaOdbeWOXhe1ds8hZoxK7nU5HYmOjOCBr89kUP8YvvvX1RyoafDbtUzoswRhwsPOfzrb7HMCf+2UHGfbywRReqSO4kNHmTnSfwkCIDUhloeumUV5dT3fe3YNTc0ev17PhC5LECY8FCxz5kXy0b0I3TJgOETF9TpBrN97CIDpGb7vfzjRtIwkfnX5ZD7aUcndb/lnskET+vyWIETkMREpE5FNbfbdIyJbRWSDiPxdRJK8+zNF5KiIrPM+HvJXXCYMeZqh8EPIOtu/03t3JCLCaWaq3NmrYrbur0IExg/t+vQavXFlbgbfOH0UDy8v4IXP9nZ+gulz/FmDeAK46IR97wCTVXUqsB34cZvXdqrqdO/jBj/GZcLN/g1Qd9id5qUWKaN7XYPYUnKEUYPi6R8buNHnP71kImeNSeW2v23glfU2A6w5nt8ShKouBw6csO9tVW3yPl0B9PwODmNaFHzgbDPPci+GlBxnDezmps6P7cCWkiNMSB/gw6A6FxMVwcPX5pI7ahDff36dJQlzHDf7IL4JvNHmeZaIrBWRD0TExb90E3J2LYe0CZA4xL0YUnLA0+RM2tcDNfVN7D5QG/AEAdAvJpJHF+Uyc2Qy33t2Lf/3YUHAYzDByZUEISI/AZqAliWXSoCRqjoDuBV4RkTa/UsRkcUiskpEVtmt9oamBmcG16z57saR0jLUtWf9ENtKq1ANXP/DiRLjovnLt2bzhSlDufO1LfzPK5tp9thcR31dwBOEiCwCvgh8Xb2zbalqvapWer9eDewE2l3kV1UfVtVcVc1NS0sLUNQmaBV9Bo21kH22u3G0DHU90LMEsaXkCIArNYgWcdGR3H/1TL51ZhaPf1TITU+v5miDLV/alwU0QYjIRcAPgQWqWttmf5qIRHq/zgbGAFbPNZ3btRwkAkbNczeO+BRnmo8edlTvKKsmPiaSEcn+uYO6qyIihJ99cSK3XzqRt/NKufqRFXYzXR/mz2GuzwKfAONEpEhEvgXcDyQC75wwnHU+sEFE1gFLgBtU9UC7BRvTVuGHMHQq9PP/vQOnJOLUInqYIAorahiV0j9o1ji4fl4WD10ziy0lR7j+8ZVU1/e8892ELn+OYrpaVdNVNVpVR6jqo6qao6oZJw5nVdWlqjrJu2+mqr7ir7hMGGlqgOLV7tceWqTk9LgPorCylqzU4FrM58JJQ7n/azPZtO8Itz6/ztZf6IPsTmoTukrWQ1MdjJzjdiSOlBxnPqjGo906rbHZw94DtWSlBmiRo264YOIQbrtoPG/nlfL0p3vcDscEmCUIE7r2rnC2GXPdjaNFy0imA93rPis6eJQmj5LpgxXk/OFbZ2Zx1phU7np9C2VH6twOxwSQJQgTuvasgORMd+9/aKuHs7oWVtQABGUNApyO619eNpmGZg/3vm3zNvUlliBMaFKFvZ8GT+0BelyD2OVNEJlBmiDAie3auZksXVPM3gO1nZ9gwoIlCBOaDhRATXnw9D8AxCZC/8Hd7qgurKwhMTaKlP4xfgrMN74zP4sIwe607kMsQZjQtPdTZxtMNQhwphvvQQ0iMzV4hrh2JH1gPy6dNowlq4uobbBhr32BJQgTmoo+g9gBznrQwSSl+9N+F1bWBHXzUltXzx5JTUMzr2/s/fKqJvhZgjChad9aGDbdWYshmAzKhur9UF/dpcMbmjwUHzxKVkpw3QPRkdxRyWSl9ufFVbZ+RF8QZH9dxnRBUz3s3wTp092O5GTd7Kjec6AWjwZ3B3VbIsKCacP4rPAAFdX1bodj/MwShAk9ZXngaYRhM9yO5GSDupcgCkNgBNOJLpg4BI/CP7eUuR2K8TNLECb07FvnbIMyQWQ52y7O6lpY6b0HIkhvkmvPpGEDGJ7Uj7fzSt0OxfiZJQgTevathbgk5ya5YBObCAlDoLJrNYhdFTUM7BdNcpAPcW1LRLhg4hA+zC+36cDDnCUIE3paOqiDdVjooNHdqkGEUvNSi7PHpVHf5GHVbpt0OZxZgjChpbHO6YMIxualFinZXR7qWlhRGzIjmNo6LXMQURHCJzsr3Q7F+JElCBNayjY7az8Hc4IYNBpqyqC+6pSH1TU2U3zoKFmpCQEKzHcSYqOYOmIgH1uCCGuWIExo2bfW2QbjENcWXRzqurvSmdMoM8jWgeiqM0ansrH4sC0mFMYsQZjQUrIe+iVD0ki3I+nYoGxn20kz064gn8W1M6ePTqHZo3y2y/ohwpUlCBNaSjfDkMnB20ENxxJEJx3VLUNcQ7GTGmDGyCQiBNbuOeh2KMZPLEGY0OHxQNkWGDLJ7UhOLaY/JKZ3OtS1sKKGlP4xDIiLDlBgvhUfE8W4oQNYu/eQ26EYP7EEYULHwV3QWAuDJ7odSee6MNS1ZRbXUDZjZBLr9h7C47H1qsORXxOEiDwmImUisqnNvkEi8o6I5Hu3yd79IiL3icgOEdkgIjP9GZsJQWV5zjbYaxDgDHXtpJO6sLImaJcZ7arpGUlU1TVRUNG1yQlNaOlSghCRv4nIJSLS3YTyBHDRCftuA95T1THAe97nABcDY7yPxcCD3byWCXeleYAE3xTf7RmU7SxoVHek3ZdrG5ooPVJPVoiOYGoxc2QSAGv3WDNTOOrqP/w/AV8D8kXk1yIyrisnqepy4MQhDpcBT3q/fhK4vM3+v6hjBZAkIuldjM/0BWWbnek1YkPgvoHWSfvab2YqrGgZ4hraNYjs1AQSY6NYZ/0QYalLCUJV31XVrwMzgULgXRH5WESuF5Hu9rANUdUS79f7gZYV54cDbSeZL/LuO46ILBaRVSKyqry8vJuXNiGtNC80mpfg2L0QHQx1bR3BFOJNTBERwoRhA9hS0n5NyYS2LjcZiUgKsAj4NrAW+ANOwninpxdXVQW61bulqg+raq6q5qalpfX00ibUNB51Po2HQgc1QHLLrK7t90PsCsFpvjsyMX0AW0qqaLaO6rDT1T6IvwMfAvHApaq6QFWfV9XvAd2t75e2NB15ty2TyhcDGW2OG+HdZwyUbwX1hE4NIiYeBgzvMEEUVtSQlhhLQmxUgAPzvUnDBnC0sbm1VmTCR1drEI+o6kRVvauleUhEYgFUNbeb13wZuM779XXAP9rs/4Z3NNNc4HCbpijT15WG0AimFoM6nrSvsLImpNaAOJWJwwYAkLfPmpnCTVcTxJ3t7Puks5NE5FnvceNEpEhEvgX8GrhARPKB873PAV4HCoAdwCPATV2MzfQFZXkQFXfsLuVQMCi7w07qXRW1ITsH04nGDE4kOlLIs36IsHPK+q2IDMXpKO4nIjOAlvkNBuA0N52Sql7dwUvntXOsAjd3Vqbpo0o3Qdo4iIh0O5KuSxkNtZVw9BD0S2rdXVXXSEV1fUjO4tqemKgIcgYnWg0iDHXWAHohTsf0COB3bfZXAf/tp5iMOVlpHoy5wO0ouqftUNfhs1p3twxxDfV7INqamD6A5fk2qjDcnDJBqOqTwJMicoWqLg1QTMYcr6bCWV8hVEYwtWgd6lpwXILYFeKT9LVn4rABLF1TREV1PakJsW6HY3yksyama1T1KSBTRG498XVV/V07pxnjW6Wbne2QEEsQyVmAnDSSqdA7xHXUoPBJEGOHOM1l20urLEGEkc46qVt+gxOAxHYexvhfyxxMg0NoBBNAdJx3qOvxHdWFFTWkD4yjX0wI9ad0YuwQ59/BjjKbkymcdNbE9Gfv9n8CE44x7SjdBPGpkDDY7Ui6r531qXeFwSR9JxqcGEtiXBTbS0+9zKoJLV29Ue5uERkgItEi8p6IlIvINf4OzhjAO8XGxOBeJKgj7Uz7XRgG03yfSEQYOySR7aVWgwgnXb0P4vOqegT4Is5cTDnAf/krKGNaeTzOXdSh1rzUImU0HD0Itc6clYdrGzlY2xhWI5hajBmcYE1MYaarCaKlKeoS4EVVPeyneIw5XssiQaHWQd2idajrLqDNCKYwa2ICGDMkkQM1DVRU17sdivGRriaIV0VkKzALeE9E0oA6/4VljFfrCKYQrkFAazPTLu/COtlp4XGTXFstI5nyrZkpbHR1uu/bgDOAXFVtBGpw1m8wxr/KWhYJmuB2JD2TNAqQ1o7qgvIaIgRGDgrHJiZnJFN+mXVUh4vuTCU5Hud+iLbn/MXH8RhzvNLNMCjLmR01FEXHwcCM1hpEQUUNGYPiiYkKv+XghwywkUzhpksJQkT+CowG1gHN3t2KJQjjb2V5oXcH9YnaDHXdVV5DVpiNYGohIowZnGBNTGGkqzWIXGCid0I9YwKjoda5C3nyV9yOpHcGjYZNS/A0e9hVUcPc7BS3I/KbsUMSeTuv1O0wjI90tZ67CRjqz0CMOUnrIkGhXoMYDXWHKSsv4WhjM1lp4VmDAMgZnMCBmgYqbSRTWOhqDSIVyBORlUDrT15VF/glKmPg2BQbQya7G0dveYe6lu1yvp/sMG1iAidBgDPlRorNyRTyupog7vBnEMa0q3QzRPWD5Ey3I+kd7yJHVfu2AVlh2wcBzr0QADvKq5kTxk1pfUWXEoSqfiAio4AxqvquiMQD4TPTmAlOpZtg8ITQWiSoPcmZIBE0V+ygX3QOQwfEuR2R3wwbGEd8TKR1VIeJrs7F9B1gCfBn767hwEv+CsoYVJ0aRKjeINdWVAwMzCDm8C4yU/sTERGCc0p1kYgwOi2BneWWIMJBVzupbwbmAUcAVDUfCMGpNU3IqC5zlusMhwQBkDKapLqisO5/aJFjczKFja4miHpVbWh54r1ZrkdDXkVknIisa/M4IiL/ISJ3iEhxm/1f6En5JkwNyO5UAAAdQ0lEQVSUbnK2YZIgmpOzGda8j+wwnKTvRDmDEyg5XEd1fZPboZhe6mqC+EBE/hvoJyIXAC8Cr/Tkgqq6TVWnq+p0nLmdaoG/e1/+fctrqvp6T8o3YSJUFwnqwIHYEQyQWsYmNnR+cIgb7Z1naqfVIkJeVxPEbUA5sBH4LvA68FMfXP88YKeq7vZBWSaclG6GhKHQPzxGwuzx3kY0PqbM5Uj8r+1QVxPaujqKySMiLwEvqWq5D6+/EHi2zfNbROQbwCrgB6p60IfXMqGkdFPYNC8BbK5LYxYwUkvcDsXvRqXEEx0p5FuCCHmnrEGI4w4RqQC2Adu8q8n9vLcXFpEYYAFOcxXAgzjzPU0HSoDfdnDeYhFZJSKryst9matM0GhuhPJtYZUgVh0ZQBMRxB4pdDsUv4uOjCAzpb/VIMJAZ01M38cZvXSaqg5S1UHAHGCeiHy/l9e+GFijqqUAqlqqqs2q6gEeAWa3d5KqPqyquaqam5aW1ssQTFCq3AnNDWGVILaV1VEZNRQqd7gdSkDkDLahruGgswRxLXC1qu5q2aGqBcA1wDd6ee2radO8JCLpbV77Es78T6YvCrMRTE3NHgoqqjmcMNqZX6oPyBmcwO7KGuqbmjs/2AStzhJEtKpWnLjT2w8R3dOLikh/4ALgb2123y0iG0VkA/A5nNqL6YtKN0NEFKSOdTsSnyisrKWxWfGkToCKfGgK/4nscgYn4FEorKh1OxTTC511Up9qTF6Px+upag2QcsK+a3tangkzZXmQMgaiwmOyt3zvAjrxGVNhRzNUbIehU1yOyr9ahrruKKtm3NBEl6MxPdVZDWKa90a2Ex9VQHj/hhv3hMsUG17bS6sRgcE5M5wdZVvcDSgARqclIGLLj4a6U9YgVDXEZ0kzIefoQTi8F3K/6XYkPrO9rIoRyf2IGzoOIqKdBBjm+sVEMiK5n41kCnHhtzCuCW0lG5ztsOnuxuFDO0qrGTs4ESKjIW3csbvEw1xOms3JFOosQZjgUrLe2Q6d5m4cPtLQ5IxgGtvSDj94ApT2kQQxOIGCihqaPbZScaiyBGGCS8l6GJgRNlNsbC+torFZmTRsgLNj8EQ4UgRHD7kbWADkDE6goclD0UEbyRSqLEGY4FKyHtLDo/YAkFdyBICJ6d4E0dL53gfuh7A5mUKfJQgTPOqrnDuNwylB7DtCfEwkmSnedSAGT3S2peF/H2hOmnf5UUsQIcsShAke+zcBCkOnuh2Jz+TtO8KE9AHHVpEbOALiko51xoexgfHRpCbE2qR9IcwShAkeLR3UYVKD8HiUvJIjx/ofAERg2AzYt9a9wAJojK0uF9IsQZjgUbIe+g+GxKFuR+ITew/WUl3fdKz/ocWw6c7Nco117gQWQDmDE9hZVo2qjWQKRZYgTPBo6aAWcTsSn9i8z+mgnjRs4PEvDJsBnkYoC/8b5nIGJ1BV30RZVfjPPxWOLEGY4NB41BnZEybNSwAbig4THSmMGZJw/Avp3psA+0Azk41kCm2WIExwKM0DbQ6rBLFu70Empg8gLvqEGWuSRkK/QbBvnTuBBZAliNBmCcIEh31rnO2wGe7G4SPNHmVD0WGmZySd/KKI0w/RBxLE4MRYEmOjbNK+EGUJwgSHos8gYYgzDDQMbC+torahmRkjk9s/YNgMZ06mhprABhZgIsJoG8kUsixBmOBQtApGnBY2HdTr9jpTabRbgwDImOs0qRWvDmBU7nCGuoZ3IgxXliCM+2oPwIGdMHyW25H4zNo9B0mOj2ZUSnz7B2ScBgjsWRHQuNyQMziBiup6Dtc2uh2K6SZLEMZ9LZ+iR5zmbhw+tG7vIaZnJCEd1Yj6JTvTbuz5JLCBuaC1o7rc+iFCjSUI476iz0AiwqaD+lBtA/ll1czsqP+hxci5sHclNDcFJjCX2Eim0GUJwrivaBWkTYDYhM6PDQErdx1AFeZkdzJl+cjToaE67G+YG5EcT0xUhCWIEORaghCRQhHZKCLrRGSVd98gEXlHRPK9204+gpmQ5/FA8SoYket2JD6zouAAsVERTMsYeOoDR53ubHeHdzNTZISQndrfJu0LQW7XID6nqtNVteW/w23Ae6o6BnjP+9yEswM7oe5wWCWIT3dVMnNkMrFRnSzpPnAEJI2CXR8EJjAX5dhQ15DkdoI40WXAk96vnwQudzEWEwh7Vzrb4eGRIA4fbSSv5Ahzsgd17YSc82HXcmhq8G9gLssZnEDxoaMcbWh2OxTTDW4mCAXeFpHVIrLYu2+IqpZ4v94PDDnxJBFZLCKrRGRVeXl5oGI1/rLnY2d9hLTxbkfiE6sKvf0PWV1cMjXnPKcfYu+n/g3MZWMGJ6IKO8utFhFK3EwQZ6rqTOBi4GYRmd/2RXXmBz5pjmBVfVhVc1U1Ny0tLUChGr/Z/YnTWRsRbJXZnvkwv4LYqAhmjOzgBrkTZc2HiCjY8a5/A3PZuKHOAISt+22oayhx7a9SVYu92zLg78BsoFRE0gG82zK34jMBUFXq9EGMOsPtSHxm2bYyzhidcvIEfR2JTXQS5I73/BuYy7JSE4iLjmDzvsNuh2K6wZUEISL9RSSx5Wvg88Am4GXgOu9h1wH/cCM+EyB7Pna2YZIgCitqKKys5Zxxg7t3Ys55ULoRDhf5J7AgEBkhjB86oHWNDBMa3KpBDAH+JSLrgZXAa6r6JvBr4AIRyQfO9z434Wr3JxAdHzZTfC/b5lR4zxnXzabPCQucbV54fx6aNGwAW/YdweOx1eVChSsJQlULVHWa9zFJVX/l3V+pquep6hhVPV9VD7gRnwmQPR8702tERrsdiU8s215OVmp/RqX0796JKaNh6FTY9Df/BBYkJg0bSFV9E3sP1rodiumi8OgZNKHn6CHYvylsmpdq6pv4ZGdl92sPLSZ/2blh8OBu3wYWRCYNc9bmtmam0GEJwrhj70pAnQ7aMPD+tjLqmzxcPDm9ZwVM9N7ys/nvvgsqyIwbmkhkhFhHdQixBGHcUbgcImPCZgbX1zeWkJYYy6xRPZwdZlAWjJgNa58CDc82+rjoSHLSEqwGEUIsQRh3FCyDjDkQ08F6CSGktqGJ97eWc9GkoURG9GLBo9xvQmU+FH7ou+CCzKRhNpIplFiCMIFXUwH7N0L22W5H4hMfbCvnaGMzF08Z2ruCJl3urBPx2aO+CSwITR4+kPKqekoOH3U7FNMFliBM4O1a7myzP+duHD7y6sYSBvWPYXZmF+df6kh0P5j+ddj6KhzZ55vggkzLHebr9hxyORLTFZYgTOAVLIPYAZA+3e1Ieu1wbSPv5JWyYNowoiJ98Oc0e7HTB/HxH3tfVhCaOGwAMZERrN1rCSIUWIIwgVewDDLPgsgotyPptZfXF9PQ5OErs0b4psDkUTBtIax6HKrDbzLK2KhIJg8fwNo9B90OxXSBJQgTWAd2waHdYdP/sGR1ERPSBzB5eCeLA3XHmbdCUx18cr/vygwiM0Yms6HoMI3NHrdDMZ2wBGECq2VxnOxz3IzCJ7aXVrG+6LDvag8tUnOcG+dWPuJ06IeZGSOTqG/ysLXEZnYNdpYgTGDtfB8ShkLqWLcj6bUXV+0lKkK4fPow3xd+9o+g6Sh89L++L9tlM0c694qssWamoGcJwgROc5OTIHLOB+nF/QJB4GhDMy+sKuLzk4aQkhDr+wukjYOpVzm1iKr9vi/fRekD4xgyINYSRAiwBGECp2gl1B+GMRe4HUmvvbJ+H4ePNnLd6Zn+u8jZP4TmRvjwd/67hgtEhFmjkllVeBAN07vGw4UlCBM4+W87q6eNDu37H1SVJz4uZPzQRGZn9fLeh1MZlA0zroHVj8Ohvf67jgtOz06h+NBR9h6wG+aCmSUIEzj57ziT88X5cMSPC9bsOUheyRG+cXom4u+msvn/5WyX3+Pf6wTY6aOdNbs/KQi/TvhwYgnCBMbhYijdFBbNS09+vJvEuCgun+GHzukTJWXArEWw7mk4UOD/6wXI6LQE0hJj+WRnpduhmFOwBGECY8c7znbM592No5fKjtTx+sYSvpqbQXxMgG70O+sHTtPcB3cH5noBICLMzU7hk4JK64cIYpYgTGDkvwMDMyBtvNuR9MpTK3bTrMo1c0cF7qKJQ2H2d2DD81C+LXDX9bPTs1MoPVJPQUWN26GYDliCMP7XVO9MrzHmgpAe3nq0oZm/rNjN+ROGkJXazWVFe2vef0BUP1h2V2Cv60dn5qQCsHx7+E0pEi4CniBEJENE3heRPBHZLCL/7t1/h4gUi8g67+MLgY7N+Mnuj6ChOuSbl5as3suh2kYWz88O/MX7p8LcG50V5/ZvCvz1/WBkSjyj0/rzz61lbodiOuBGDaIJ+IGqTgTmAjeLyETva79X1enex+suxGb8YetrEB0f0tNrNHuU//vXLqZnJJHb01XjeuuMWyB2YFjVIj43bjCfFhygpr7J7VBMOwKeIFS1RFXXeL+uArYAwwMdhwkQj8dJEDnnOesdhKh38krZXVnL4vnZ/h/a2pF+yTD3Bme9iDDpizh3/GAamj18tMOGuwYjV/sgRCQTmAF86t11i4hsEJHHRMSlj2nGp/athaoSGP9FtyPplUc+LCBjUD8unNTLVeN6a/Z3nb6Ij+5zNw4fyc0cREJsFO9vs2amYORaghCRBGAp8B+qegR4EBgNTAdKgN92cN5iEVklIqvKy61zK+htfQUkEsZe6HYkPbZ69wFW7z7It8/M7t2a077QPwVmXuuMaDpc7G4sPhATFcHZ49J4J6+UJpv+O+i4kiBEJBonOTytqn8DUNVSVW1WVQ/wCDC7vXNV9WFVzVXV3LS0tMAFbXpm62uQeabTPBKiHvqggIH9orky18fTevfU6TeDeuDTB92OxCe+OCWdiuoGPt11wO1QzAncGMUkwKPAFlX9XZv96W0O+xIQHkM1+rLy7VCxHSZc6nYkPZa37wjv5JXyzXlZgbsxrjPJmTDpS7DqCTga+kt3njNuMPExkby6ocTtUMwJ3KhBzAOuBc49YUjr3SKyUUQ2AJ8Dvu9CbMaXtr7qbMdd7G4cvXD/+/kkxkaxaF6m26Ecb96/Q0MVrHrM7Uh6rV9MJOdPGMKbm0pslbkgE/CPRKr6L6C9hlwb1hputrwCw2bAwCBpmumm7aVVvL5xP987N4eB/aLdDud46VNh9Lmw4kGYexNEx7kdUa98cWo6L6/fx4f55Zw7fojb4Rgvu5Pa+EflTti3xmkKCVH3/3MH/WMi+ea8LLdDad+8/4CaMlj/rNuR9No54waT0j+G51aG17Tmoc4ShPGPTX9ztpO+7G4cPbS9tIpXNuzjmtNHkdw/xu1w2pc136mhffxH8DS7HU2vxERF8JVZI3hvaxllR+rcDsd4WYIwvqcKm5Y4az8kZbgdTY/85o2tJMRGccP80W6H0jERpy/iwM5j/T0h7KrTMmj2KEvWFLkdivGyBGF8r3QzlG+FyVe4HUmPfLKzkve2lnHTOTnBW3toMWEBJGfBv/7XScwhLDstgbnZg3h6xR7rrA4SliCM721a4twcF4L9Dx6P8us3tpA+MI7rg23kUnsiIuGM7zn9PYX/cjuaXvv2mdkUHzrK6xttyGswsARhfMvTDBuXOOtO9091O5pue2HVXtYXHeY/Pz+OuOhIt8Ppmulfg/5p8NH/uh1Jr507fjCj0/rz5w8KbCGhIGAJwvjWzvfh8F6YcY3bkXRbZXU9d72xldmZg/jyzBCaPzK6H8y5AXa8C/s3uh1Nr0RECN+dP5q8kiM2P1MQsARhfGvNExCfCuMucTuSbrvztS3UNjTxqy9Ndm/G1p467VsQNxDeuC3k+yIunzGckYPiufvNbXg8of29hDpLEMZ3qkph2xsw/WqICvLO3RO8tqGEv68t5sazRzNmSKLb4XRfv2S44Jew+1+w9im3o+mVmKgIfvD5sWzdX8VL60J/QsJQZgnC+M7qJ8DTBDOvczuSbtl36Cg//tsGpmUk8b3zxrgdTs/NuBZGzYO3fgIHdrkdTa9cOnUYk4cP4O43t1FV1+h2OH2WJQjjG41HYeWfYexFkBo6/2Trm5q55Zk1NHmUP1w1nejIEP6TiIiAy//k3B/x/LXQUON2RD0WESHcefkUSqvquPet8FgcKRSF8F+DCSrrnobaSjjj39yOpMtUlf/+2ybW7DnEb6+cRmZqf7dD6r3kTLji/6BsMzx7tZO4Q9T0jCS+MXcUf1mxm08LKt0Op0+yBGF6r6EWlt8LI2bDqDPcjqbLfv9uPkvXFPEf54/h4inpnZ8QKsZcAJf9CXYth79+CWpCdznP/7poPKMGxfPvz63jQE2D2+H0OZYgTO+teMBZVvSC/3GaN0LAn5bt4L738rly1gj+7dzQaRLrsulXw1cedZZ8ffhzsGeF2xH1SEJsFPd/bSYHahr49+fW2h3WAWYJwvTOwUL48PfOsNYQqD14PMpdb2zh7je3sWDaMH59xVQi3F5G1F8mXwGLXneS9uMXw7t3QFO921F12+ThA7nzS5P5ML+CHy3dYDfQBZAlCNNzzY3w0k3OdA8X/8btaDp1+GgjNz+zhj9/UMA1c0fyu69Oc3+NaX8bMQtu/Aimfx3+9Xt46CzY86nbUXXbV3Mz+P75Y/nbmmJ+/o/Ndn9EgATJGoom5Hg88NoPYPdH8KWHg37W1o93VvBfL25g/5E6fvKFCXz7rKzQuxmup2IT4bL7nYn9Xv0+PHYhzP4OnPdz57UQ8W/n5VDT0MTDyws4fLSRu78yNXSmQwlRliBM99Uddv7RbFoKZ/0nTLvK7Yg6VFhRw91vbeX1jfsZOSieF284nZkjk90Oyx1jPw83r4B/3gmf/hm2vg5f/B2MvdDtyLpERPjvL0wgOT6G37y5lZ3l1TzwtZnhMfosSEkot+fl5ubqqlWr3A4jfKk6zUhNdU7b9YGdsPOfsOpxqK2Ac38KZ94adB3THo+ysvAAT3xUyFt5+4mNiuCmc3JYPD/bPnG22LsSXv6ed1r2r8BFv4aENLej6rJ380r5wYvraWjy8G/njeFbZ2YRE2Ut5l0lIqtVNbfT44ItQYjIRcAfgEjg/1T11x0dawmih1ThSLEzwqVsCxzaDYf2QnWpc3NVQ7Uzfr6pg5W9xnwezr7Nad8OEs0eZUPRId7YtJ9X1u+j5HAdA/tFc83ckVx3eiaDB4T2ms1+0dTg9EssvwdiE+DCu2DawqBL+B0pPnSUO17ezDt5pWSmxHPD2aP50szhxEbZh4DOhGSCEJFIYDtwAVAEfAZcrap57R1vCaKLaiqgeA0Ur3aSwr61zlrGLRKGQNJISEyHmASIiXdmCI3qB1GxziMyFgaOgBG5kDDYve/Fy+NRCiqqWVV4kA93VPDRjgoO1TYSFSGcPTaNBdOHccHEIcTHWCtqp8q2OrWJopXOEqan3wLjvwjRoZFU399axm/f2cam4iOk9I/hi1PTWTB9GNNGJBEVynfG+1GoJojTgTtU9ULv8x8DqOpd7R1vCcJLFeqPwNGDTjI4UACVO6Biu5MYDu32HiiQNt75J9DyGDLJSQhBqK6xmcNHGzlU20jJ4aMUVtRQWFnL9tIqNhYdpqq+CYAhA2I5a0wa88emMX9MKknxoTVRYFDweGDdU/DRH5zfnZhEyDkPMmZD+jQYMNz5ABGkSUNV+TC/guc+28O7W8poaPKQGBtFbmYyk4YNJGdwAqNS4klNiCUlIabPf3AI1QTxFeAiVf229/m1wBxVvaW943ucIEo3w4vXA97vvfU96MLzk16j42M7fN6D654UR5vrNtSAHr9ovQehVNLYHpnD1sgxbIkYS35ENrUS3zqOvLU0BfU+0xMvBV07/rhztM1xJ5dx7Bptyj3h7Wxo9tDQdPJNUfExkYxOS2DqiIFMz0hixsgkRqcl9J0RSf7m8UDBPyHvH5D/LlTtO/711lplnLONaGnO8b7/rT+Hzp77T7MqNfVNHG1o5mhjc7u/Ry/oeTwrlxATFdH6iPDG1hqxSOvXreG3ea3tczecMy6Nn1wysUfndjVBhFwaFZHFwGKAkSNH9qyQqDgYPKGlwJaSu/G8q8fS8es9um4Hz2Pineme+w2C+EG8X9aflwpjaIqIcS7lPTK7zS+1nBBi+38M0nqpY8cde01ODKXtvjbHty3n2PWP/9M68bioCGFAv2iS4qNJ6hdDWmIsmanxpCXEWjLwp4gIyDnfeQBU7Xc+UFWVwJESp6baVO8duFAH6un+Bx0/iwQGeB8AHsVJGI3N1Dc5HzymDcyBpJGtzxuaPajqCR+CWr4+/gPSsQ8z7n64HhKAfrVgq0FYE5MxxvhZV2sQwdaD8xkwRkSyRCQGWAi87HJMxhjTJwVVE5OqNonILcBbODXFx1R1s8thGWNMnxRUCQJAVV8HXnc7DmOM6euCrYnJGGNMkLAEYYwxpl2WIIwxxrTLEoQxxph2WYIwxhjTrqC6Ua67RKQc2N3pgYGVCoTaKvEWc2BYzIFhMXdulKp2Or97SCeIYCQiq7pyh2IwsZgDw2IODIvZd6yJyRhjTLssQRhjjGmXJQjfe9jtAHrAYg4MizkwLGYfsT4IY4wx7bIahDHGmHZZgjgFEblIRLaJyA4Rue0Ux10hIioiud7ns0VknfexXkS+1N0ygyzmQhHZ6H3N5wtw9DTmNvtHiki1iPxnd8sMspj9+j73Jm4RyRSRo21+Rx5qc+wsb9w7ROQ+8fGKTn6KeZm3zJbXfLrQem9+P0Rkqoh8IiKbve9rnHe/X9/ndqmqPdp54Ew3vhPIBmKA9cDEdo5LBJYDK4Bc7754IMr7dTpQhjNzbpfKDKaYvc8LgdRge5/bvLYEeBH4z+6UGUwx+/t99sHvRyawqYNyVwJzcRYLfAO4OARiXnbizyRIYo4CNgDTvM9TgEh/v88dPawG0bHZwA5VLVDVBuA54LJ2jvsl8BugrmWHqtaqapP3aRzHVivsapnBFLO/9ThmABG5HNgFtF03JGjf51PEHAi9irs9IpIODFDVFer8F/sLcHkwxxwAvYn588AGVV0PoKqVqtocgPe5XZYgOjYc2NvmeZF3XysRmQlkqOprJ54sInNEZDOwEbjB+8+30zKDMGZwksXbIrJanDXBfanHMYtIAvAj4H+6W2Yv+SNm8O/7DL38/QCyRGStiHwgIme1KbPoVGUGYcwtHvc2L/3Mx801vYl5LKAi8paIrBGRH7Yp05/vc7uCbsGgUCEiEcDvgEXtva6qnwKTRGQC8KSIvBHA8NrVk5hVtQ44U1WLve2074jIVlVdHgQx3wH8XlWrA9Ec21W9iNm19xk6jbsEGKmqlSIyC3hJRCYFKraO9CRmVT0CfN37XicCS4FrcT6Vux1zFHAmcBpQC7wnIquBw4GIrb1gTPuKgYw2z0d497VIBCYDy7x/6EOBl0Vkgaq2djCq6hYRqfYe21mZwRjzKlUt9u4vE5G/41ShffWPq8cxA3OAr4jI3UAS4BGROmB1J2UGXcyqer+f3+dexe39/aj3xrdaRHbifNot9pbTUZnBGHPb3+kqEXkG5732VYLoze9HEbBcVSsAROR1YCbwFP59n9vn706OUH3gJM8CIItjHU2TTnH8Mo51NGVxrIN3FLAPZzKubpUZJDH3BxK9+/sDHwMXBUPMJ+y/g2Od1EH7Pp8iZr++zz74/UjjWGdpNs4/p0He5yd2nn4hmGP2lpnq3R+NM2DghiCJORlYg3fQCPAucIm/3+eOHlaD6ICqNonILcBbOKMSHlPVzSLyC5xPIC+f4vQzgdtEpBHwADfpsU8EJ5UZzDGLSDbwd+8nnSjgGVV9M0hi7laZwRwzMAQ/vs/Q67jnA79o8/txg6oe8L52E/AE0A/nH5fPmlP9EbOI9AfeEpFob5nvAo8EQ8yqelBEfgd8htMn9boe66fw2/vcEbuT2hhjTLtsFJMxxph2WYIwxhjTLksQxhhj2mUJwhhjTLssQRhjjGmXJQhjjDHtsgRhjDGmXZYgjDHGtOv/AVV65PsSYN3rAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f44e582fbe0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "# Creating an list with bootstrapped means for each AB-group\n",
    "boot_1d = []\n",
    "iterations=500\n",
    "for i in range(iterations):\n",
    "    boot_mean = df.sample(frac=1 , replace=True).groupby('version')['retention_1'].mean()\n",
    "    boot_1d.append(boot_mean)\n",
    "    \n",
    "# Transforming the list to a DataFrame\n",
    "boot_1d = pd.DataFrame(boot_1d)\n",
    "print(boot_1d.head())\n",
    "boot_1d\n",
    "    \n",
    "# A Kernel Density Estimate plot of the bootstrap distributions\n",
    "boot_1d.plot(kind='kde')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "47"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "## 7. Zooming in on the difference\n",
    "<p>These two distributions above represent the bootstrap uncertainty over what the underlying 1-day retention could be for the two AB-groups. Just eyeballing this plot, we can see that there seems to be some evidence of a difference, albeit small. Let's zoom in on the difference in 1-day retention</p>\n",
    "<p>(<em>Note that in this notebook we have limited the number of bootstrap replication to 500 to keep the calculations quick. In \"production\" we would likely increase this to a much larger number, say, 10 000.</em>)</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {
    "dc": {
     "key": "47"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,0,'% difference in means')"
      ]
     },
     "execution_count": 252,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8XGd97/HPT6PV2ixbm9fYsWXZshNncWyIyQZZm5DALZCEpTfQXkovIXS9DRRSyvIqhFugt+TeJqVAytIUQpo6ELIvZLNjO953Rd5kW7Zsa7NlbTO/+8eMFEWRLFnWzBnNfN+vl16eOXPmzE9jab46z/Oc5zF3R0REBCAj6AJERCR5KBRERKSPQkFERPooFEREpI9CQURE+igURESkj0JBRET6KBRERKSPQkFERPpkBl3AmSotLfVZs2YFXYaIyLiydu3ao+5eNtx+4y4UZs2axZo1a4IuQ0RkXDGzvSPZT81HIiLSR6EgIiJ9FAoiItJHoSAiIn0UCiIi0kehICIifRQKIiLSZ9xdpyCSTHY0tPG7nY2EMozrF1UydWJe0CWJnBWFgsgodIcjfP3XW/m3lXvpXeb8W09s59sfXszNi6cGW5zIWVAoiJyhcMT53M/X8cSWBu64dBafvWou7V09/NXDG/n8Q+sozM3kquryoMsUGZW49imY2fVmtsPMas3s7kEev8PMGs1sfezrj+JZj8hYuPeJ7TyxpYEv3biAr9y8kLLCHM6ZnM+Dn1xKdUUhf/XLDRw/2RV0mSKjErdQMLMQcB9wA1AD3G5mNYPs+h/ufkHs6wfxqkdkLKysO8b9v6vjY8tm8keXnfu2x/KyQ3z31gs4frKL+56vDahCkbMTzzOFpUCtu9e5exfwEHBLHF9PJK46usN88ZFNzJw0gS/dONjfN7BgShEfung6P3ltL4dbOxJcocjZi2coTAP297tfH9s20O+b2UYze9jMZsSxHpGzct/ztdQdPck3PriIvOzQkPt99qq5dEci/GzliCalFEkqQV+n8Bgwy93PB54GHhxsJzP7tJmtMbM1jY2NCS1QBKJDT//fC2/y3y6axmVVp5+S/pzJ+by3upyfrdpHV08kQRWKjI14hsIBoP9f/tNj2/q4+zF374zd/QFw8WAHcvcH3H2Juy8pKxt2jQiRMRWOOHc/spGivKwhm40Gun3pTI6d7OLlWv0RI+NLPENhNVBlZrPNLBu4DVjRfwczm9Lv7s3AtjjWIzIqP125l3X7mrnnphom5WeP6DmXzyujOC+LxzYcinN1ImMrbtcpuHuPmd0JPAmEgB+6+xYz+yqwxt1XAHeZ2c1AD3AcuCNe9YiMxsHmU9z7xHYun1fGLReM/KK07MwMblhUyWMbDnKqK3zaPgiRZBLXi9fc/XHg8QHb7ul3+wvAF+JZg8houTtfeGQTEYdvfGARZnZGz7/p/Kk8tHo/L+1q5NqFlXGqUmRsBd3RLJK0frmmnhd3NnL3DfOZMWnCGT9/6exJ5GeHeGGn+hVk/FAoiAyivqmdr/16K+86dxKfeNc5ozpGdmYGy+eW8uKORrx3giSRJKdQEBmgozvMZ366FoBvf2gxGRln1mzU35XV5RxoPsWbjSfGqjyRuFIoiPQTiThffGQTmw+08t1bLxhVs1F/V1RHh1C/sENNSDI+KBREYtydr/1mK4+sO8CfXzOPq2sqzvqY0ybmMWvyBFbtPj4GFYrEn0JBJOY7T+/kR6/s4VPLZ/O5984ds+MunT2J1XuOE4moX0GSn0JBBLj/xTf5p+dque2SGXz5pgVnPPz0dJbOnkxzeze7jqhfQZKfQkHS3r+/vo+//+12bjp/Ct/44HljGggAy2ZPAmDV7mNjelyReFAoSFp7Y18TX350M1dWl/HdWy8gdBYjjYYyvSSPKcW56leQcUGhIGnrZGcPn/v5OqZMzOUfb7uQrFB8fh3MjItmlrBhf3Ncji8ylhQKkrb++cU3OdB8iu9+5AKK87Li+lqLZxRT33SKoyc6h99ZJEAKBUlLB5tPcf/v6rjlgqksmTUp7q+3ePpEADbW62xBkptCQdLSg6/uIRxx/uq66oS83qJpxWQYbNjfkpDXExkthYKknfauHv799X1cv7CS6SVnd8XySOXnZFJVXsgGnSlIklMoSNr5r/UHae3o4ZPLZyX0dRfPKGbD/mZNjidJTaEgaeexDQc5tyyfi88pSejrLp4xkab2bvYdb0/o64qcCYWCpJXGtk5W1h3jpvOmjPlFasM5b1oxAFsOtib0dUXOhEJB0soTWxqIONx4/siX1hwr8yoKyTDYdkihIMlLoSBp5aktDZxbls+8ioKEv3ZuVog5ZQUKBUlqCgVJGx3dYV7ffZwr5pUlvOmo14IpRWxV85EkMYWCpI21e5vo7IlwWVVpYDUsmFLEwZYOmtu7AqtB5HQUCpI2Xtp1lMwMY+nsyYHVUDO1CIBth9oCq0HkdBQKkjZeqT3KRTNLKMjJDKyGBVMKAdiqfgVJUgoFSQsnO3vYcrCFZefGf56j0ykvzKW0IFudzZK0FAqSFjbWtxBxuGhmYi9YG4w6myWZKRQkLbyxrwmAC2ZMDLgSqJlSRO2RE3SHI0GXIvIOCgVJC+v2NXNuaT4l+dlBl0J1ZSFd4Qh7jp4MuhSRd1AoSMpzd9bvb+LCJGg6guiVzQA7D58IuBKRd1IoSMqLrnjWxYUzg286AphTVoAZ7DqiYamSfOIaCmZ2vZntMLNaM7v7NPv9vpm5mS2JZz2Snnr7E5KhkxkgLzvEjJIJ7NKZgiShuIWCmYWA+4AbgBrgdjOrGWS/QuDzwKp41SLpbevBVrJDGVQFMN/RUOZVFLDzsM4UJPnE80xhKVDr7nXu3gU8BNwyyH5fA74FdMSxFklj2xvamFteQFYoeVpLqyoK2X30JF09GoEkySWevyXTgP397tfHtvUxs4uAGe7+mzjWIWluR0Mb8ysLgy7jbeZVFNATcfYe0wgkSS6B/elkZhnAd4C/GMG+nzazNWa2prGxMf7FScpoae+mobWD6iQLhapyjUCS5BTPUDgAzOh3f3psW69CYBHwgpntAd4FrBiss9ndH3D3Je6+pKysLI4lS6rZ3hC9cjjZQqF3BJL6FSTZxDMUVgNVZjbbzLKB24AVvQ+6e4u7l7r7LHefBawEbnb3NXGsSdLMjtiH7vzKooArebu87BAzJ03QsFRJOnELBXfvAe4EngS2Ab9w9y1m9lUzuzleryvS37ZDbRTnZVFRlBN0Ke9QVV6oYamSdOI6h7C7Pw48PmDbPUPse2U8a5H0tKOhlerKwsBWWjudeRUFvLDjCF09EbIzk2dklKQ3/SRKynJ3dh4+wYIk60/oNa+ikJ6Is0cjkCSJKBQkZdU3neJEZw/VSdaf0GtuefRiOnU2SzJRKEjK2tEQ/bBNtpFHveaWF5BhGpYqyUWhICmrd+RRsoZCblZ0BNKbRxQKkjwUCpKyth1qZXpJXqBrMg9nbnmhhqVKUlEoSMpKxuktBqqqKGD30ZNahU2ShkJBUlJnT5i6oyeTtumoV1V5Ad1hZ++x9qBLEQEUCpKi3jxyknDEk+5K5oF650CqVROSJAmFgqSkHYejcx4le/PRnPJ8AF3ZLElDoSApaXtDG9mhDGaV5gddymlNyM5kekkeuzQCSZKEQkFS0o6GNuYk2cI6Q6kqL1AoSNJI/t8YkVHYfij5Rx71qqoo5M3GE4QjHnQpIgoFST3JurDOUOaWF9DVE2H/cY1AkuApFCTlJOvCOkOpis2BpCYkSQYKBUk5by2sMz5CYW5fKGhYqgRPoSApZ3tDdGGdyqLcoEsZkcLcLKYU51KrYamSBBQKknJ2NLQl7cI6Q5mrEUiSJBQKklLcfVzMeTRQVXkhtUdOENEIJAmYQkFSylsL64yzUKgo4FR3mAPNp4IuRdKcQkFSSu/COuPtTKG3s7lWTUgSMIWCpJTekUfzKsZZKJQpFCQ5KBQkpfQurFOYmxV0KWekJD+b0oIcDUuVwCkUJKVsO9TKginJPV32UDQHkiQDhYKkjI7uMLuPnhy/oVBRQO3hE7hrBJIER6EgKWNHQxsRh5op46s/oVdVeQFtnT0cbu0MuhRJYwoFSRlbD0XnPBqvZwpzY6uwqV9BgqRQkJSx7VArBTmZzCiZEHQpo1JVEZsDSdNdSIAUCpIyth1qZX5lIRkZ42d6i/4m52dTMiFLnc0SKIWCpAR3Z/uhtnHbdARgZrHpLtR8JMFRKEhKqG86RVtnz7gOBYC5FQXs1AgkCVBcQ8HMrjezHWZWa2Z3D/L4Z8xsk5mtN7OXzawmnvVI6nqrk3l8jjzqVVVeQMupbo6e6Aq6FElTcQsFMwsB9wE3ADXA7YN86P/c3c9z9wuAe4HvxKseSW2bD7QQyjDmV47vM4UqjUCSgMXzTGEpUOvude7eBTwE3NJ/B3dv7Xc3H9A5s4zK+v3NVFcUkpcdCrqUs9I7AklzIElQRhQKZvaImd1oZmcSItOA/f3u18e2DTz2Z83sTaJnCnedwfFFAIhEnA37m1k8Y2LQpZy18sIcCnMzNSxVAjPSD/n/C3wU2GVm3zSz6rEqwN3vc/c5wF8DXxpsHzP7tJmtMbM1jY2NY/XSkiJ2HztJa0cPF6ZAKERHIBWo+UgCM6JQcPdn3P1jwEXAHuAZM3vVzD5pZkNNR3kAmNHv/vTYtqE8BHxgiNd/wN2XuPuSsrKykZQsaWTD/maAlDhTgGi/gs4UJCgjbg4ys8nAHcAfAeuAfyQaEk8P8ZTVQJWZzTazbOA2YMWAY1b1u3sjsGvElYvErN/fTH52qG+hmvGuurKQYye7ONLWEXQpkoYyR7KTmf0nUA38BHi/ux+KPfQfZrZmsOe4e4+Z3Qk8CYSAH7r7FjP7KrDG3VcAd5rZ1UA30AT897P7diQdbdjfzHnTiwmN0yuZB6qZGh1BtfVgK+XVuQFXI+lmRKEA/Iu7P95/g5nluHunuy8Z6kmx5zw+YNs9/W5//kyKFRmovauHrYda+aPLzg26lDHTewHetkNtXFldHnA1km5G2nz09UG2vTaWhYiMxpo9TXSHnXedOznoUsZMcV4W0ybm9V2QJ5JIpz1TMLNKosNI88zsQqD3/LwIGJ9TUUpKea3uGJkZxpJzSoIuZUzVTC1i68GWoMuQNDRc89F1RDuXp/P2q43bgC/GqSaREXvtzWMsnjGR/JyRtoSODzVTinhm22Hau3qYkJ1a35skt9P+tLn7g8CDZvb77v6rBNUkMiInOnvYdKCFz1yROv0JvWqmFuEeXU3uwpmpdRYkyW245qOPu/tPgVlm9ucDH3d3zVUkgXl511HCEWf53NKgSxlzNf06mxUKkkjDnZfmx/5NjQHgklKe3nqYotxMLpk1KehSxtz0kjwKczPZekj9CpJYwzUf3R/79+8SU47IyIQjznPbD/Pe+eVkhVJvWRAzY8GUIrYe1AgkSayRToh3r5kVmVmWmT1rZo1m9vF4FycylLV7m2hq7+bqmoqgS4mbmilFbG9oIxzR5MGSOCP9E+va2DTXNxGd+2gu8FfxKkpkOI+uP0BuVgZXzEvdubBqphbR3hVm91HNgySJM9JQ6G1muhH4pburoVMC09Ed5rENB7lh0RQKc4eaj3H8uyA2wd+G/fp1k8QZaSj82sy2AxcDz5pZGaDZuiQQT25poK2jhw9fPD3oUuJqTlkB+dkhNtY3B12KpJGRTp19N3ApsMTdu4GTDFhFTSQR3J0fv7qHGZPyUmpqi8GEMozzphezvl5nCpI4Z3Kp5Hyi1yv0f86/jXE9Iqe1avdx1u1r5mu3LCQjRWZFPZ3F0yfyo1f20NkTJidzfC81KuPDSKfO/gkwB1gPhGObHYWCJJC780/P7aK0IJsPL5kx/BNSwOIZE+kKR9h+qC1lFhGS5DbSM4UlQI27a2ycBOaJzQ28UnuMe26qITcrPf5q7g2CDfWpsQa1JL+RdjRvBirjWYjI6Zzs7OHvHtvKgilF/MG7zwm6nISZWpxLaUEO6/ers1kSY6RnCqXAVjN7Hejs3ejuN8elKpEB/s+zu2ho7eC+j11EZgpewTwUM+OCGcUKBUmYkYbCV+JZhMjp7Gho419f3s2tS2ZwcYqtmzASF84s4ZltR2g62UVJfnbQ5UiKG+mQ1BeJXsmcFbu9GngjjnWJANHO5S8/upmC3Ez++ob5QZcTiN4J/9bsbQq4EkkHI5376H8ADwP3xzZNAx6NV1EivVZsOMjre47z19fPZ1Ka/pV8/vRiskMZrN5zPOhSJA2MtHH2s8ByoBXA3XcBWlFc4qq9q4dv/nY7i6YVcWuaDEEdTG5WiPOnF/P6boWCxN9IQ6HT3bt678QuYNPwVImr+1+s41BLB3/7/vS4UO10Lpk9ic0HWmjv6gm6FElxIw2FF83si0CemV0D/BJ4LH5lSbprOdXND1/ezQ2LKlNyEZ0ztXTWJHoizvp9GoUk8TXSULgbaAQ2AX8MPA58KV5Fifzbq3to6+zhzvfODbqUpHDROSWYwevqV5A4G9GQVHePmNmjwKPu3hjnmiTNdfaE+dGre3jv/HIWTi0OupykUJyXxfzKIvUrSNyd9kzBor5iZkeBHcCO2Kpr9ySmPElHT2xu4PjJLj65fFbQpSSVS+dMZs3eJjq6w8PvLDJKwzUf/RnRUUeXuPskd58ELAOWm9mfxb06SUs/W7mPcyZPYPmc0qBLSSqXVZXS1RPR2YLE1XCh8Angdnff3bvB3euAjwN/EM/CJD3VNZ7g9T3HuX3pzLQfcTTQstmTyQ5l8HLt0aBLkRQ2XChkufs7fgJj/Qqpuw6iBObXGw8B8IELpgVcSfLJyw5x8TklvLRLoSDxM1wodI3yMQDM7Hoz22FmtWZ29yCP/7mZbTWzjWb2rJmlz/SXMqjfbDzEJbNKqCzODbqUpPSeqlK2HWqlsa1z+J1FRmG4UFhsZq2DfLUB553uiWYWAu4DbgBqgNvNrGbAbuuILvF5PtFpNO4d3bchqWDX4TZ2HG7jpvOnBl1K0rqsKtrP8uqbOluQ+DhtKLh7yN2LBvkqdPfhmo+WArXuXhe7GvohBqzr7O7Pu3t77O5KILVXYpfT+vXGQ5jBDYu0dMdQFk4tZuKELH63U6Eg8RHPiemnAfv73a+PbRvKHwK/jWM9kuSe2XaYJeeUUF6kpqOhhDKM5XNLeWlXI1oIUeIhKVYrMbOPE13y89tDPP5pM1tjZmsaG3XtXCo60tbBloOtXFmteRaHc1V1OUfaOtl8oDXoUiQFxTMUDgD9p7acHtv2NmZ2NfA3wM3uPmjvmbs/4O5L3H1JWVlZXIqVYL24Ixr2V8zT/+9wrqouwyx6ZiUy1uIZCquBKjObbWbZwG3Aiv47mNmFRNdouNndj8SxFklyL+xspKwwh4VTi4IuJelNLsjhopklPLtdoSBjL26h4O49wJ3Ak8A24BfuvsXMvmpmvWs7fxsoAH5pZuvNbMUQh5MU1hOO8PKuo1wxrwwzXbA2Eu9bUM7mA600tHQEXYqkmJGu0Twq7v440RlV+2+7p9/tq+P5+jI+bKhvpuVUN1dWq+lopK5eUMG9T+zg2e2H+dgyXd4jYycpOpolvb248ygZBpfNVSiMVFV5ATMm5fHsNrW6ythSKEjgVtYdY9G0YoonaOaUkTIz3je/gldqj3KqS7OmythRKEigOrrDrN/fzLLZWl3tTF29oILOngivaII8GUMKBQnU+v3NdPVEWDZ7ctCljDtLZ0+iICdTQ1NlTCkUJFCr6o5jFl2YXs5MdmYGV1aX8cy2w4QjurpZxoZCQQK1su4YCyqLKM5Tf8JoXLuwkqMnuli3rynoUiRFKBQkMJ09Yd7Y18S7zlXT0WhdVV1GVsh4cktD0KVIilAoSGA21rfQ2RNh2blqOhqtwtwsLp1TylNbD2uCPBkTCgUJzMo3jwGwdJZC4Wxcu7CCvcfa2XG4LehSJAUoFCQwq3YfZ35lISX52UGXMq5dU1OBGTy1RaOQ5OwpFCQQ3eEIa/eqP2EslBfmcuGMiTy1Vf0KcvYUChKIjfUtnOoO66K1MXLdwko2H2ilvql9+J1FTkOhIIFYWRfrT1AojIlrF0aXMH16q5qQ5OwoFCQQq3Yfp6q8gMkFOUGXkhJml+Yzr6JAQ1PlrCkUJOG6wxHW7jmu/oQxdm1NJa/vPk7Tya6gS5FxTKEgCbf5QAsnu8K6PmGMXbewkohrmU45OwoFSbhVu48D6k8Ya4umFTGlOJen1K8gZ0GhIAm3qu4Y55blU16YG3QpKcXMuLamgt/tbKS9qyfocmScUihIQoUjzpo9uj4hXq5bWElnT4Tf7dQaCzI6CgVJqK0HW2nr7NH1CXFyyexJFOdl8ZRGIckoKRQkoXqvT9CZQnxkhTJ434Jynt1+hO5wJOhyZBxSKEhCrdp9jNml+VQUqT8hXq6tqaTlVDerYx36ImdCoSAJE444r+8+rqajOLtiXhm5WRm6kE1GRaEgCbO9oZXWjh5dnxBnedkhLqsq0xoLMioKBUmYVXXR5oxls9WfEG/X1lRwqKWDTQdagi5FxhmFgiTMqt3HmDlpAlMn5gVdSsq7ekEFGVpjQUZBoSAJEY44q9SfkDAl+dksnT1J/QpyxhQKkhCbD7TQ3N7Ne6pKgy4lbVy3sJJdR05Q13gi6FJkHFEoSEK8tKsRgPfMVSgkyjU1FQCaC0nOiEJBEuJ3u46yaFqR1k9IoOklE1g0rUhXN8sZiWsomNn1ZrbDzGrN7O5BHr/czN4wsx4z+1A8a5HgnOjsYd2+Jt4ztyzoUtLOtTWVvLGvmSOtHUGXIuNE3ELBzELAfcANQA1wu5nVDNhtH3AH8PN41SHBW1V3jO6wc7n6ExLuut5lOrXGgoxQPM8UlgK17l7n7l3AQ8At/Xdw9z3uvhHQJC0p7KVdR8nNyuDiWSVBl5J25lUUcM7kCTypoakyQvEMhWnA/n7362PbzpiZfdrM1pjZmsbGxjEpThLnpV2NLJs9mZzMUNClpJ3eNRZee/MorR3dQZcj48C46Gh29wfcfYm7LykrU7v0eLLvWDtvNp7k8nn6fwvKdQsr6Q47L+zQH1QyvHiGwgFgRr/702PbJI30tmVfvaA84ErS14UzSygtyNaFbDIi8QyF1UCVmc02s2zgNmBFHF9PktAzWw/H2rXzgy4lbYUyjGtqKnhh+xE6e8JBlyNJLm6h4O49wJ3Ak8A24BfuvsXMvmpmNwOY2SVmVg98GLjfzLbEqx5JvJb2bl7fc5yrF1QEXUrau7amkpNdYV6tPRZ0KZLkMuN5cHd/HHh8wLZ7+t1eTbRZSVLQCzuPEI5435W1EpxL504mPzvEU1sbuGq+mvJkaOOio1nGp6e2Hqa0IIfF0ycGXUray8kMceX8cp7eephwRGssyNAUChIXp7rCPL/9CNfUlJORYUGXI0RHIR090cW6fU1BlyJJTKEgcfHc9iO0d4V5//lTgy5FYq6sLiMrZBqFJKelUJC4WLHhAGWFOSw7V6usJYui3CzePadUy3TKaSkUZMy1dnTz/I5GbjxvCiE1HSWV6xdWsvdYO1sOtgZdiiQphYKMuSc3N9DVE+HmC9R0lGx+77xKskMZPLy2PuhSJEkpFGTMPbr+ANNL8rhwhkYdJZuJE7K5ZmEF/7X+AF09modS3kmhIGNq77GTvFJ7jI8smYGZmo6S0Ycunk5TezfPbT8SdCmShBQKMqb+Y/V+Mgw+vETXJCary+aWUl6YoyYkGZRCQcZMdzjCL9fWc1V1OVOK84IuR4aQGcrggxdN4/kdR2hs6wy6HEkyCgUZM89tj37I3LZ0ZtClyDA+smQG4YjzH6v3BV2KJBmFgoyZH7+yhynFuVxVrbUTkt2csgIuqyrlJyv30h1Wh7O8RaEgY2JTfQuv1R3jU8tnkxnSj9V48Knlsznc2slvN+sKZ3mLfntlTPzLS3UU5mRy29IZw+8sSeGKeWXMmjyBH7+yO+hSJIkoFOSs1Te185tNh7h92UwKc7OCLkdGKCPDuOPSWbyxr5lVdVpnQaIUCnLWvv9cLSEzPrl8VtClyBm69ZKZlBbk8L1ndgVdiiQJhYKclbrGE/xybT0fe9dMDUMdh/KyQ/zJlXN4re4YK3W2ICgU5Cx995ld5GRm8D+vnBt0KTJKH1s2k/LCHL7z1E7NnioKBRm99fubeWzDQT61fDZlhTlBlyOjlJsV4q73VfH6nuP8ZtOhoMuRgCkUZFTCEedLj26ioiiHP77i3KDLkbN0+9KZLJxaxNd/vY2TnT1BlyMBUijIqPzktT1sPtDKl2+q0YijFBDKML56yyIaWju494ntQZcjAVIoyBl7s/EE33xiO5fPK+PG86YEXY6MkYvPKeFTy2fz4Gt7eV4zqKYthYKcka6eCH/60HryskJ8+0Pna3rsFPO/rq9mfmUhf/nLDew/3h50ORIAhYKMmHu0H2HTgRb+/r+dT0VRbtAlyRjLzQrx/Y9eRHc4wqd+vJqWU91BlyQJplCQEfvnF+v4xZp67nrvXK5fVBl0ORInc8sLuP8TS9hz7CSf+NdVNJ3sCrokSSCFgozID16q41tPbOf9i6fyp1fPC7ocibN3z5nM/Z+4mO0NbXz4/teoPXIi6JIkQRQKclrhiPPN327n67/Zxo3nTeE7H1lMRob6EdLBe+dX8OAnl3L8ZBc3f/9lfrpyL+GILm5LdQoFGdLB5lPc8aPX+ecX3+Sjy2byvdsuIEvTYqeVd8+ZzG/ueg+Lp0/kS49u5qZ/epkVGw5qDYYUZuPtsvYlS5b4mjVrgi4jpXV0h/npyr1875ldRNz50o01fHSZVlNLZ+7Orzce4jtP72T30ZOUFeZw3cIKrqmp5KKZE3WtyjhgZmvdfcmw+8UzFMzseuAfgRDwA3f/5oDHc4B/Ay4GjgG3uvue0x1ToRA/R0908vDaeh58dQ+HWjq4rKqUb3zgPGZOnhB0aZIkIhHn2e1H+M919Ty/vZFT3WHMoLqikOrKQmZNzufcsnwqinIpLchmcn4OxXlZanJMAiMNhcw4FhAC7gOuAeqB1Wa2wt239tvtD4Emd59rZrcB3wJujVfTgRFOAAAJ/0lEQVRN8nbuTn3TKV7Y2chz2w7zcu1RusPOstmT+IePLObSOaVBlyhJJiPDuKamgmtqKjjVFWb1nuO8sa+JdfuaWbu3iRUbDjLw78xQhjExL4uJE7KYOCGbkti/E/OyKMnPZuKELEoLcphSnEtlcS6l+TkKkQDFLRSApUCtu9cBmNlDwC1A/1C4BfhK7PbDwPfNzHy8tWklKXenoztCa0c3rae6aWjtYP/xU+xvamdHQxsb9jdzLDbccOakCdxx6SxuvWQGc8sLA65cxoO87BCXzyvj8nlvrcnd0R1m3/F2Gts6OXqik2Mnujh2spOm9m6a27tobu/mQHMHWw620tTeRUf3O/smMjOMiqJoQFQW51JZlNsXGJWx7eWFuWRnqn8rHuIZCtOA/f3u1wPLhtrH3XvMrAWYDBwd62J+sXo/D7xU1zc18NtSx9/2zzv28b7H/W33B94eyXPfvm3gMXzQx0973IHfT7/ndvaE6Q6/M18zM4zZpflcWV3O4hnFXDqnlDll+bo6Wc5ablaIeRWFzKsY2R8WHd1hmtu7aWzr5FDLKRpaO2hoiX4daulg28FWntt2hFPd4Xc8Nyczg+xQBpkhIyuUQVbsdu9Pcf+f57f9ZNs7tw25b5K5631VvH/x1Li+RjxDYcyY2aeBTwPMnDm6Ds+S/Gyqe39QT/ND8dYPFAPuv/3x/k/u/TF853MGf/ztt0f23NPV9M7HozeyMzMozsuiKC+TotzoKfrMyROoLMolpNNzSQK5WSEqi0NUFudy3vTiQfdxd1o7emJBcYrDrR00tHRyqjtMdzjS99XV4/REIrHn9Hv+gGMN3Mbb9k3uRorivPh36MczFA4A/Vdxnx7bNtg+9WaWCRQT7XB+G3d/AHgAoh3Noymmtx1URMYXM6M4L4vivCyqK9W0GW/xbJRbDVSZ2WwzywZuA1YM2GcF8N9jtz8EPKf+BBGR4MTtTCHWR3An8CTRIak/dPctZvZVYI27rwD+FfiJmdUCx4kGh4iIBCSufQru/jjw+IBt9/S73QF8OJ41iIjIyGlMl4iI9FEoiIhIH4WCiIj0USiIiEgfhYKIiPQZd1Nnm1kjsDfoOs5CKXGYxmOc0nvxFr0Xb9F7ETXW78M57l423E7jLhTGOzNbM5Lpa9OB3ou36L14i96LqKDeBzUfiYhIH4WCiIj0USgk3gNBF5BE9F68Re/FW/ReRAXyPqhPQURE+uhMQURE+igUAmBm3zaz7Wa20cz+08wmBl1TopnZ9Wa2w8xqzezuoOsJgpnNMLPnzWyrmW0xs88HXVPQzCxkZuvM7NdB1xIkM5toZg/HPie2mdm7E/XaCoVgPA0scvfzgZ3AFwKuJ6HMLATcB9wA1AC3m1lNsFUFogf4C3evAd4FfDZN34f+Pg9sC7qIJPCPwBPuPh9YTALfE4VCANz9KXfvid1dSXRVunSyFKh19zp37wIeAm4JuKaEc/dD7v5G7HYb0V/8acFWFRwzmw7cCPwg6FqCZGbFwOVE15vB3bvcvTlRr69QCN6ngN8GXUSCTQP297tfTxp/GAKY2SzgQmBVsJUE6nvA/wIiQRcSsNlAI/CjWFPaD8wsP1EvrlCIEzN7xsw2D/J1S799/oZoE8LPgqtUgmZmBcCvgD9199ag6wmCmd0EHHH3tUHXkgQygYuA/+fuFwIngYT1u8V15bV05u5Xn+5xM7sDuAl4XxquS30AmNHv/vTYtrRjZllEA+Fn7v5I0PUEaDlws5n9HpALFJnZT9394wHXFYR6oN7de88aHyaBoaAzhQCY2fVET5Nvdvf2oOsJwGqgysxmm1k20bW5VwRcU8KZmRFtN97m7t8Jup4gufsX3H26u88i+vPwXJoGAu7eAOw3s+rYpvcBWxP1+jpTCMb3gRzg6ejnAivd/TPBlpQ47t5jZncCTwIh4IfuviXgsoKwHPgEsMnM1se2fTG2trmkt88BP4v90VQHfDJRL6wrmkVEpI+aj0REpI9CQURE+igURESkj0JBRET6KBRERKSPQkESxszKzOzl2JXdH+i3/b/MbOoIj3Ei9u9UM3u43/Z/j806+2dmNt/M1semCJgz9t/J2TOzz5jZHwRdh8hAGpIqCWNmdwHHgUeAx939SjN7P3Cxu39lhMc44e4FA7ZVAi+7+9zY/buBTHf/+hnUFnL38Ej3F0lVOlOQROoGJhC9cC9sZpnAnwL3DvWE2FXPr5nZJjP7er/ts8xsc+zuU8C02NnB38aO+Sdm9nxs34+b2euxx++PTd2NmZ0ws38wsw3Au83sYjN70czWmtmTZjYltt8LZvat2DF2mtllse0hM/vfsTOfjWb2udj2QY8z4Pv6ipn95emOP2D/K2PH/C8zqzOzb5rZx2LP2dR7RhQ7G/uVma2OfS2PbV8aex/XmdmrvVfLmtkdZvaImT1hZrvM7N5+39uPY9/bJjP7s5H+J8s45+760ldCvoBi4DfAGqKX7t8F3DHMc1YAfxC7/VngROz2LGDzwNux+18B/jJ2ewHwGJAVu/9/+x3PgY/EbmcBrwJlsfu3Er3SGuAF4B9it38PeCZ2+0+IzkuTGbs/6XTHGfB99a9x0OMP2P9KoBmYQjRUDwB/F3vs88D3Yrd/Drwndnsm0Sk0AIr61Xk18KvY7TuIXjFbTHTOob1E56W6GHi63+tPDPrnR1+J+dI0F5Iw7t5CdL58zKyE6CRfHzSzfwFKiH4wvjbgacuB34/d/gnwrTN82fcR/YBbHZtSJA84EnssTHQyOoBqYBFvTT0SAg71O07vZHVriYYQRD9c/9lja2O4+3EzWzTMcYYy2PEHWu3uhwDM7E2iZ0gAm4Cr+tVUE3ttiE4sV0D0Q/9BM6siGoZZ/Y77bOz/BjPbCpwDbAHONbN/IhrkTyFpQaEgQfky8A3gduBlon9xPwJcN8i+Z9PxZcCD7j7Y6nYd/lY/ggFb3H2oZQ87Y/+GOf3vzXDHGcpIjt/Z73ak3/1Iv+dkAO9y9463FWX2feB5d/+gRddueGGI44aJnlE0mdliov8fnwE+QnTtD0lx6lOQhIv9tTrd3V8g2scQIfrBnzfI7q8QnTUT4GOjeLlngQ+ZWXnstSeZ2TmD7LcDKLPYWrhmlmVmC4c59tPAH8f6RjCzSaM8zlh6iuhkasRe/4LYzWLemp78juEOYmalQIa7/wr4EtH5/SUNKBQkCN8A/iZ2+9+Jts2vJrou7UCfJ7p28SZGsTqbu28l+qH2lJltJPpB/o6OX48uC/oh4Fuxjuf1wKXDHP4HwD5gY+w5Hx3lccbSXcCSWMf3VqJ/5UO0M//vzWwdI2shmAa8YNHZW39Kmq0jns40JFVERProTEFERPooFEREpI9CQURE+igURESkj0JBRET6KBRERKSPQkFERPooFEREpM//B1UQFFCUp08SAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f44e58b5240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Adding a column with the % difference between the two AB-groups\n",
    "boot_1d['diff'] = boot_1d['diff'] = (boot_1d['gate_30'] - boot_1d['gate_40']) /  boot_1d['gate_40'] * 100\n",
    "# Ploting the bootstrap % difference\n",
    "ax = boot_1d['diff'].plot(kind='kde')\n",
    "ax.set_xlabel('% difference in means')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "54"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "## 8. The probability of a difference\n",
    "<p><img src=\"https://s3.amazonaws.com/assets.datacamp.com/production/project_184/img/ziggy_smiling.png\" style=\"width:200px; float:left\"> </p>\n",
    "<p>From this chart, we can see that the most likely % difference is around 1% - 2%, and that most of the distribution is above 0%, in favor of a gate at level 30. But what is the <em>probability</em> that the difference is above 0%? Let's calculate that as well.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {
    "dc": {
     "key": "54"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "96.39999999999999 %\n"
     ]
    }
   ],
   "source": [
    "# Calculating the probability that 1-day retention is greater when the gate is at level 30\n",
    "prob = (boot_1d['diff'] > 0).sum() / len(boot_1d['diff'])\n",
    "\n",
    "# Pretty printing the probability\n",
    "print((prob*100), \"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "61"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "## 9. 7-day retention by AB-group\n",
    "<p>The bootstrap analysis tells us that there is a high probability that 1-day retention is better when the gate is at level 30. However, since players have only been playing the game for one day, it is likely that most players haven't reached level 30 yet. That is, many players won't have been affected by the gate, even if it's as early as level 30. </p>\n",
    "<p>But after having played for a week, more players should have reached level 40, and therefore it makes sense to also look at 7-day retention. That is: What percentage of the people that installed the game also showed up a week later to play the game again.</p>\n",
    "<p>Let's start by calculating 7-day retention for the two AB-groups.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {
    "dc": {
     "key": "61"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "version\n",
       "gate_30    0.190201\n",
       "gate_40    0.182000\n",
       "Name: retention_7, dtype: float64"
      ]
     },
     "execution_count": 256,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculating 7-day retention for both AB-groups\n",
    "df.groupby('version')['retention_7'].sum() / df.groupby('version')['retention_7'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "68"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "## 10. Bootstrapping the difference again\n",
    "<p>Like with 1-day retention, we see that 7-day retention is slightly lower (18.2%) when the gate is at level 40 than when the gate is at level 30 (19.0%). This difference is also larger than for 1-day retention, presumably because more players have had time to hit the first gate. We also see that the <em>overall</em> 7-day retention is lower than the <em>overall</em> 1-day retention; fewer people play a game a week after installing than a day after installing.</p>\n",
    "<p>But as before, let's use bootstrap analysis to figure out how certain we should be of the difference between the AB-groups.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {
    "dc": {
     "key": "68"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "version       gate_30   gate_40\n",
      "retention_7  0.188971  0.183543\n",
      "retention_7  0.190301  0.181607\n",
      "retention_7  0.190511  0.181461\n",
      "retention_7  0.188144  0.181485\n",
      "retention_7  0.192803  0.180219\n",
      "100.0 %\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xt8XXWd7//XJ/c2lzZpkt7bNL23YKGEQgEBR0DwAnpkEBCF8QKOMhf9zW8Oo2dQcTzj5eec8TfjDKByxstRRECtCCK3ggiF3u9Nem+TpmmSpk3StLl+zh97BTYhadI0e6+dvd/Px2M/svZa37Xyyc7O/uR7Wd+vuTsiIiKnkxZ2ACIikviULEREZFBKFiIiMiglCxERGZSShYiIDErJQkREBqVkISIig1KyEBGRQSlZiIjIoDLCDmCkFBcXe1lZWdhhiIiMKmvXrm1w95LByiVNsigrK2PNmjVhhyEiMqqY2f6hlFMzlIiIDErJQkREBqVkISIig1KyEBGRQSlZiIjIoJQsRERkUEoWIiIyqKS5z0Ik0bk76w40sXZ/E5npaVw+r4TZJXlhhyUyJEoWInFQ3dTG3/1yI6v2HH1jnxncvryML753IVkZquRLYlOyEImxysMtfOyHr3Gyo5uvXr+YG86bwsnObu5fuZv/emUf9a3t/NvN55OWZmGHKjIgJQuRGKo9fpLbfvgaaQaPffYS5k3MB2A88NUbzmHy+DF846kdnDNlHH955exwgxU5DdV9RWKko6uHO3+8lpMd3fzkkxe9kSii3XV5OdedM4n/9UwVu+tbQ4hSZGiULERi5LvPVbG55jjfuWlJv4kCwMz42gfPISsjjW/9fkecIxQZOiULkRhYf6CJ/1y5m5sqpvGexZNOW7Y4L5tPv7Ocp7fWsaXmeJwiFDkzShYiI6ynx7n3N1spzc/hH9+/aEjn3HFpGWMy0/mvV/bFNjiRYVKyEBlhv1pfw+aa4/z36+aTn5M5pHPGjcnkwxdMZcXGQzS2tsc4QpEzp2QhMoLaOrr41tM7WDJtHDcsmXpG596+vIyOrh4eW1cdo+hEhk/JQmQEPfDiHuqa2/nH9y864/sm5k7MZ8m0cazYeChG0YkMn5KFyAipPX6SB17azfveMZmKsqJhXeMDS6awpaaZPRpGKwkmpsnCzK41s0oz22Vm9/Rz/Atmts3MNpnZc2Y2M+pYt5ltCB4rYhmnyEj49u8r6XG459oFw77G+94xGYDfbqwdqbBERkTMkoWZpQPfA64DFgG3mFnfoSHrgQp3fwfwKPCtqGMn3f284HF9rOIUGQmbqo/x+PoaPnnZLKYXjR32dSaPG8PSGeN5dnvdCEYncvZiWbNYBuxy9z3u3gE8DNwQXcDdX3D3tuDpKmBaDOMRiQl352tPbKM4L4vPjsCUHX+2oJTNNcepb9GoKEkcsUwWU4GDUc+rg30D+STwVNTzHDNbY2arzOyD/Z1gZncGZdbU19effcQiw/Dk5sOs3tfEF64e+lDZ07lyfikAL1bpPS2JIyE6uM3sNqAC+HbU7pnuXgHcCvyrmb3tXzZ3f9DdK9y9oqSkJE7RirzpZEc3//PJ7SycXMBHLpw+ItdcNLmAkvxsVlYeGZHriYyEWCaLGiD6r2dasO8tzOwq4EvA9e7+Rr3b3WuCr3uAlcD5MYxVZFgeeGk3NcdO8pUPLCJ9hKYYT0szLp9bwp92NdDT4yNyTZGzFctksRqYa2azzCwLuBl4y6gmMzsfeIBIojgStb/QzLKD7WLgUmBbDGMVOWPVTW3858rIUNmLyieM6LUvLi+iqa2TnUc0hFYSQ8yShbt3AXcDTwPbgUfcfauZ3WdmvaObvg3kAb/sM0R2IbDGzDYCLwDfcHclC0kY7pH5n8zgi+9dOOLXvzhIPq/tbRzxa4sMR0wXP3L3J4En++y7N2r7qgHOewU4N5axiZyNR9dW8/yOI9z7/kVMHT9mxK8/rXAMk8fl8Nqeo3x8edmIX1/kTCVEB7fIaLKnvpX7fruNZbOKuOOSsph8DzPjollFvLa3EXf1W0j4lCxEzkDLqU4+/eM1ZGak8S83LYnputnLZk2gobWD/Y1tgxcWiTElC5EhOtHexad+tIb9jW38x0eXMq1w+HdqD8V508cDsLH6WEy/j8hQKFmIDEFd8yk+9sPXWLO/ie/ctOSNDuhYmjcxj5zMNDYcVLKQ8MW0g1skGTyzrY6/f3Qjpzp7+Pdbzue6cyfH5ftmpKdx7tRxbFSykASgZCEygFOd3Xz9d9v5yar9LJ5SwHdvPp85pXlxjWHJtPH8eNV+Ort7yExXQ4CER+8+kX4cPNrGjfe/wk9W7edTl83i8c9eEvdEAbBk+ng6unqoPNwS9+8tEk01C5E+Dh5t46YHXuVEexff/3gFVy+aGFosvZ3cGw4e45yp40KLQ0TJQiRKa3sXH3/odU52dvOLu5azcHJBqPFMKxxDUW4Wm6qPATMHLS8SK2qGEony1RVb2d94ggc/VhF6ooDIzXmLJhewvVbNUBIuJQuRwLoDTfxybTV3XTGbZbOGt4Z2LCycnE9lXQtd3T1hhyIpTMlCJPDt31dSkp/N3e+aE3Yob7FgUgEdXT3sbTgRdiiSwpQsRIisof3qnkbufGc5udmJ1ZXX2xy2rbY55EgklSlZiAD/9co+8rIzuHnZyKx2N5LmlOaRmW7qt5BQKVlIymvr6OL3Ww7zgSWTR2QN7ZGWlZHG7JI8tqtmISFSspCU94etdbR1dPPB86aGHcqAIiOilCwkPEoWkvJ+vaGGqePHcGFZ4oyA6mvh5AKOtLTT2No+eGGRGFCykJR2or2LV3Y1ct05k2K6NsXZ6u3k3qFpPyQkShaS0l7Z3UhHdw/vWlAadiintWByPoCaoiQ0ShaS0lZWHiE3K52KssKwQzmt4rxsinKz2HWkNexQJEUpWUjKcndWVtZzyZxisjPSww5nUHNK85QsJDRKFpKyDhxto+bYSS6fWxx2KEMypzSPnUdacfewQ5EUpGQhKWv1viYAls2K/RKpI2FuaR7HT3ZSrxFREgIlC0lZa/YdpSAng7khLGo0HHNLI53caoqSMChZSMpave8oFWVFCT1kNtrciZGkpmQhYVCykJTU2NrO7voTCT8KKlppfjb52RnsrFOykPhTspCUtHZ/pL8ike/a7svMmDNRI6IkHDFNFmZ2rZlVmtkuM7unn+NfMLNtZrbJzJ4zs5lRx243s53B4/ZYximpZ+3+JrLS0zh3lK1rPTcYESUSbzFLFmaWDnwPuA5YBNxiZov6FFsPVLj7O4BHgW8F5xYBXwYuApYBXzaz0dNeIAlvY/UxFk7OJycz8e+viDanNI+G1naaTnSEHYqkmFjWLJYBu9x9j7t3AA8DN0QXcPcX3L0teLoKmBZsvwd4xt2PunsT8AxwbQxjlRTS0+NsrWnmnFFWq4CoEVH1ql1IfMUyWUwFDkY9rw72DeSTwFPDPFdkyPYfbaOlvWvUNUFBpGYBGhEl8ZcQ60ea2W1ABXDFGZ53J3AnwIwZM2IQmSSjzTXHAUZlzWLq+DGMyUzXiCiJu1jWLGqA6DUqpwX73sLMrgK+BFzv7u1ncq67P+juFe5eUVJSMmKBS3LbUnOcrPQ05k3MDzuUM5aWZpSX5LJbzVASZ7FMFquBuWY2y8yygJuBFdEFzOx84AEiieJI1KGngWvMrDDo2L4m2Cdy1rbUHGfB5HyyMkbnyPE5pXlKFhJ3Mftrcfcu4G4iH/LbgUfcfauZ3Wdm1wfFvg3kAb80sw1mtiI49yjwNSIJZzVwX7BP5Ky4O1tqjrN4yuhrguo1uySPmmMnOdnRHXYokkJi2mfh7k8CT/bZd2/U9lWnOfch4KHYRSepqLrpJM2nujhnakHYoQzb7JI83GFPQ+uoTnoyuozOerjIMFXVRZYlXTBp9PVX9JpdmgvA7voTIUciqUTJQlJKZZAs5o7Czu1eZRNySTMNn5X4UrKQlFJ1uIUp43IoyMkMO5Rhy8lMZ3rRWHVyS1wpWUhKqaxrZd4oboLqNbskj92qWUgcKVlIyujq7mH3kVbmj+ImqF5zSvPY03CC7h4tsSrxoWQhKWNfYxsd3T2j8ma8vmaX5NLR1UNN08mwQ5EUoWQhKWNn0LmdHMkiMkeU+i0kXpQsJGVU1rVg9uZkfKNZb7LQiCiJFyULSRlVdS3MLBrLmKzRtYZFfwpzs5iQm6WahcSNkoWkjMrDLUnRBNVrdonmiJL4UbKQlNDe1c2+xjbmJ8Gw2V6zS7Uet8SPkoWkhD31kWGmyVWzyKWprZOjWmJV4kDJQlJCVRKNhOo1u1QjoiR+lCwkJVQebiEjzZhVnBt2KCNmTu/wWTVFSRwoWUhKqKprobwkd9QueNSfqePHkJ2Rpn4LiYvk+csROY3KuuQaCQW9S6xqRJTEh5KFJL0T7V0cPHoyKeaE6iuyxKrWtZDYU7KQpNfbTJMMs832Nbskl4NNbZzq1BKrEltKFpL0ehc8SsaaRe8Sq3sbVLuQ2FKykKRXdbiF7Iw0pheNDTuUETdHw2clTpQsJOlV1rUwd2Ie6WkWdigjblZxLqYlViUOlCwk6VUl4UioXjmZ6UwrHKNObok5JQtJasfbOqlrbk/K/opeWmJV4kHJQpJa1ZFgmo8kHAnVa05JHnsaWunREqsSQ0oWktQqDyfvSKhes0vzONXZQ80xLbEqsaNkIUmtqq6F/OwMJo/LCTuUmNESqxIPQ0oWZva4mb3PzJRcZFTZcTgyEsos+UZC9eodPqsRURJLQ/3w/w/gVmCnmX3DzObHMCaREeHuVNW1MH9SQdihxFRRbhaFYzM1IkpiakjJwt2fdfePAkuBfcCzZvaKmf2FmWUOdJ6ZXWtmlWa2y8zu6ef45Wa2zsy6zOzGPse6zWxD8FhxZj+WCBxpaedYWycLkrhzu5eWWJVYG3KzkplNAO4APgWsB75LJHk8M0D5dOB7wHXAIuAWM1vUp9iB4Jo/6+cSJ939vOBx/VDjFOnV27mdrPdYRJtTmsceJQuJoaH2WfwK+CMwFviAu1/v7r9w978C8gY4bRmwy933uHsH8DBwQ3QBd9/n7puAnmH/BCIDeGMkVIrULBpaOzjWpiVWJTaGWrP4vrsvcvd/dvdaADPLBnD3igHOmQocjHpeHewbqhwzW2Nmq8zsg2dwnggQ6dwuyc+mKDcr7FBibnZpZAVANUVJrAw1WfxTP/teHclA+jEzSES3Av9qZrP7FjCzO4OEsqa+vj7G4choU1XXkhL9FQBzSiI/5+4j6uSW2DhtsjCzSWZ2ATDGzM43s6XB40oiTVKnUwNMj3o+Ldg3JO5eE3zdA6wEzu+nzIPuXuHuFSUlJUO9tKSA7h5P6jmh+ppaOIasjDR2qWYhMZIxyPH3EOmAngb8S9T+FuCLg5y7GphrZrOIJImbidQSBmVmhUCbu7ebWTFwKfCtoZwrAnDgaBvtXT0p0V8BkJ5mlBfnao4oiZnTJgt3/xHwIzP7sLs/diYXdvcuM7sbeBpIBx5y961mdh+wxt1XmNmFwK+AQuADZvZVd18MLAQeMLMeIrWfb7j7tjP/8SRVVR5uBkiZZiiITPuxteZ42GFIkjptsjCz29z9p0CZmX2h73F3/5d+Tos+/iTwZJ9990ZtryZSa+l73ivAuacPXWRgOw63YAZzS1MoWZTk8dTmWk51dpOTmR52OJJkBuvgzg2+5gH5/TxEElJVXQszi8YyJit1PjRnl+TS47C/sS3sUCQJDdYM9UDw9avxCUdkZOw4nDqd272il1hNlb4aiZ+h3pT3LTMrMLNMM3vOzOrN7LZYBycyHKc6u9nXcCKl+isAyos1oaDEzlDvs7jG3ZuB9xOZG2oO8P/GKiiRs1FV10KPw4LJyT2BYF9jstKZUTT2jTvXRUbSUJNFb3PV+4BfuruGXEjC2nooMhJq8ZTUShYAiyYXsK22OewwJAkNNVk8YWY7gAuA58ysBDgVu7BEhm/roePkZ2cwvXCw+0aTz+IpBextOEFre1fYoUiSGeoU5fcAlwAV7t4JnKDPpIAiiWLroWYWTikgLS15FzwayKKgNrVDtQsZYYPdwR1tAZH7LaLP+fEIxyNyVrp7nO21zdy6bGbYoYRi8ZRxQCRhVpQVhRyNJJMhJQsz+wkwG9gAdAe7HSULSTB76ls51dmTkv0VABMLIrPsbjukmoWMrKHWLCqARe7usQxG5Gy90bk9NTWThZmxeEoBW2s1BkVG1lA7uLcAk2IZiMhI2HroOFkZacwuGWhNruS3aHIBVYdb6ezWmmIycoZasygGtpnZ60B7704tdyqJZuuhZhZMyiczfcgrBiedRVMK6OjuYdeRVham2L0mEjtDTRZfiWUQIiPB3dl6qJn3npvaleDe/pqth5qVLGTEDHXo7ItE7tzODLZXA+tiGJfIGdvbcILjJztZMm182KGEalZxHnnZGWw8eCzsUCSJDHVuqE8DjwIPBLumAr+OVVAiw7GxOvLheN6M1E4W6WnGkunjWH+wKexQJIkMtWH3c0RWq2sGcPedQGmsghIZjg0HjpGblZ5Sa1gM5Lzp49lR28LJju7BC4sMwVCTRbu7d/Q+CW7M0zBaSSgbDh7j3GnjSE/BO7f7On96IV09zpZDGkIrI2OoyeJFM/siMMbMrgZ+Cfw2dmGJnJlTnd1sq23mvOmFYYeSEHqb4tYfUFOUjIyhJot7gHpgM3AXkaVS/0esghI5U9tqm+nsds6bntr9Fb2K87KZXjSGDerklhEypKGz7t5jZr8Gfu3u9TGOSeSM9Y78UbJ40/nTC3l971HcHTM1zcnZOW3NwiK+YmYNQCVQGaySd298whMZmg0HjzGpIIdJ43LCDiVhXDiriMPNpzh49GTYoUgSGKwZ6vNERkFd6O5F7l4EXARcamafj3l0IkO0Zl8TS2eqVhFteXlk1tlX9zSEHIkkg8GSxceAW9x9b+8Od98D3AZ8PJaBiQxVdVMbNcdOskxTcr/F7JI8ivOyWLXnaNihSBIYLFlkuvvb/i0J+i0yYxOSyJlZvS/yYbhs1oSQI0ksZsZF5RNYtacRTRgtZ2uwZNExzGMicfP63ibyczKYP0k34/V1cfkEao+f4sDRtrBDkVFusNFQS8ysv1VUDFBPoiSE1/c2cmFZkW7G68fy8kht65XdjcyckBtyNDKanbZm4e7p7l7QzyPf3dUMJaFraG1nd/0Jls1Sf0V/ZpfkMnlcDisrj4QdioxyqTvpvySF1Xt7+yuULPpjZrxrQSkv72ygvUvzRMnwxTRZmNm1ZlZpZrvM7J5+jl9uZuvMrMvMbuxz7HYz2xk8bo9lnDJ6vbb3KDmZaZwzZVzYoSSsP5tfyomOblbv1dQfMnwxSxZmlg58D7gOWATcYmaL+hQ7ANwB/KzPuUXAl4nc07EM+LKZadIfeZvX9x5l6YxCsjJUSR7IJXMmkJWRxvM71BQlwxfLv7BlwC533xPMWPswcEN0AXff5+6bgL6LBb8HeMbdj7p7E/AMcG0MY5VRqKG1nW21zW904kr/xmZlsLx8As/vqNMQWhm2WCaLqcDBqOfVwb4RO9fM7jSzNWa2pr5eU1almpd3Rm4BunxeSciRJL6rF01kX2MbOw63hB2KjFKjuu7u7g+6e4W7V5SU6AMj1bxUVU9RbhbnTlV/xWCuO2cS6WnGbzceCjsUGaVimSxqgOlRz6cF+2J9rqSAnh7npZ0NXDanmDTdXzGoCXnZXDJ7Ak9sqlVTlAxLLJPFamCumc0ysyzgZmDFEM99GrjGzAqDju1rgn0iAGw/3ExDa7uaoM7AB5ZM4cDRNjZVa/U8OXMxSxbu3gXcTeRDfjvwiLtvNbP7zOx6ADO70MyqgT8HHjCzrcG5R4GvEUk4q4H7gn0iALxYFemjunxucciRjB7vWTyJrPQ0frVelXQ5c0Na/Gi43P1JIqvqRe+7N2p7NZEmpv7OfQh4KJbxyej1UlU9CyblU1qgWWeGatyYTK49ZxKPr6vmnusWkJOZHnZIMoqM6g5uSU0n2rtYu7+JK+arCepM3bJsBs2nunhyc23Yocgoo2Qho86ruxvp7HaumKtkcaYuLi9iVnEuP3/9QNihyCijZCGjzks76xmTmc4FZbqp/0yZGbcsm87qfU1U1emeCxk6JQsZdV6qqmf57AlkZ6jNfTg+vHQa2Rlp/O8/7R28sEhAyUJGlf2NJ9jX2KZRUGdhQl42H75gGo+tq6G+pT3scGSUULKQUeWl3iGzur/irHzqsll0dvfw41f3hR2KjBJKFjKqvFjVwPSiMcwq1qpvZ6O8JI+rF07kJ6v209bRFXY4MgooWcio0dHVw6u7G7h8bglmmuLjbN15eTnH2jp5dG112KHIKKBkIaPGugNNnOjoVhPUCKkoK2LpjPF8/4976Oruu0qAyFspWcio8VJVPRlpxiWztX7FSLnritkcPHqSJ7ccDjsUSXBKFjJqvLSznqUzCsnPyQw7lKRx9cKJlJfkcv/K3ZqNVk5LyUJGhfqWdrbUNGuKjxGWlmbcdXk522qbeXlXQ9jhSAJTspBR4eVdvbPMKlmMtA+eP5WJBdnc/+LusEORBKZkIaPCS1UNTMjNYvGUgrBDSTrZGel84tJZ/GlXI5u11oUMQMlCEl5Pj/PHnfVcNler4sXKLRfNID87Q7ULGZCShSS8bbXNNLR2qAkqhgpyMvnoxTN5akst+xpOhB2OJCAlC0l4L+2M9Fe8c57mg4qlT1xaRkZaGg/+cU/YoUgCUrKQhPdiZT2LJhdQmq9V8WKptCCHD18wlUfXVnP0REfY4UiCUbKQhNYarIqnu7bj4y8unUVHVw+PrDkYdiiSYJQsJKG9uruRrh7ncjVBxcW8iflcNKuIn67aT3ePbtKTNylZSEJ7qaqesVnpVMwsCjuUlPHx5WVUN51kZeWRsEORBKJkIQntpZ31LC+fQFaG3qrxcs3iiZTmZ/OTVfvDDkUSiP4CJWHtazjB/sY2TfERZ5npadyybAYvVtWzv1HDaCVCyUISVu+QWd1fEX+3XjSDNDN+9vqBsEORBKFkIQlrZWU9ZRPGUqZV8eJuYkEOf7aglMfW1tCptS4EJQtJUKc6u3lldwNXzi8NO5SUdVPFdBpa23mxsj7sUCQBKFlIQnpt71FOdfaovyJEV84voTgvm1/ongtByUIS1MrKI2RnpLG8XKvihSUzPY0PL53K8zuOcKTlVNjhSMhimizM7FozqzSzXWZ2Tz/Hs83sF8Hx18ysLNhfZmYnzWxD8Lg/lnFK4llZWc/y2RPIyUwPO5SU9ucV0+nucX69vibsUCRkMUsWZpYOfA+4DlgE3GJmi/oU+yTQ5O5zgP8FfDPq2G53Py94fCZWcUri2ddwgr0NJ3iX+itCN6c0jwtmFvLImmotu5riYlmzWAbscvc97t4BPAzc0KfMDcCPgu1HgXebmRYsSHG9dw5fqf6KhHBTxTR2HWll3YFjYYciIYplspgKRPeMVQf7+i3j7l3AcaC3kXqWma03sxfN7J39fQMzu9PM1pjZmvp6jdhIFiur6plVnMvMCRoymwje944pjM1K55HV6uhOZYnawV0LzHD384EvAD8zs7etp+nuD7p7hbtXlJTov9BkcKqzm1d3N6pWkUDysjN437mTeWLTIU60d4UdjoQklsmiBpge9XxasK/fMmaWAYwDGt293d0bAdx9LbAbmBfDWCVBvLqnkfauHt1fkWA+cuF0TnR087vNtWGHIiGJZbJYDcw1s1lmlgXcDKzoU2YFcHuwfSPwvLu7mZUEHeSYWTkwF9DyXSngxcp6cjLTuGiWZplNJBfMLKS8JJdfqCkqZcUsWQR9EHcDTwPbgUfcfauZ3Wdm1wfFfghMMLNdRJqbeofXXg5sMrMNRDq+P+PuR2MVqyQGd+e5HXVcMrtYQ2YTjJnxkYrprN3fxK4jLWGHIyGIaZ+Fuz/p7vPcfba7fz3Yd6+7rwi2T7n7n7v7HHdf5u57gv2PufviYNjsUnf/bSzjlMSw43ALB4+e5OpFE8MORfrx35ZOIyPNeGRNddihSAgStYNbUtAz2+owg3cvVH9FIirJz+bdC0t5fF01HV2aXDDVKFlIwnhmWx3nTx9PaX5O2KHIAD5y4XQaWjt4fkdd2KFInClZSEI4dOwkm2uOc/WiSWGHIqdx+dwSJhZkq6M7BSlZSEJ4dnvkP1X1VyS2jPQ0brxgGi9W1XP4uCYXTCVKFpIQntlWR3lJLnNK88IORQZxU8V0ehweXq1V9FKJkoWE7uiJDl7d3cg1aoIaFWZOyOXK+SX8dNUB2ru6ww5H4kTJQkL35OZaunqc65dMCTsUGaJPXDqLhtZ2ntioO7pThZKFhG7FxkPMKc1j4eT8sEORIXrn3GLmlObx0J/2auryFKFkIaGqPX6S1fuOcv2SKWh2+tHDzPjEpbPYeqiZ1fuawg5H4kDJQkL1xMZa3FET1Cj0ofOnUjg2k/tf3B12KBIHShYSGnfn8fU1vGPaOMqKtXbFaDMmK51PvbOc53ccYcNBLYyU7JQsJDQbq4+zvbaZj1w4ffDCkpBuv6SMwrGZfPfZqrBDkRhTspDQ/Py1A4zNSlcT1CiWl53Bpy8v54XKetYdUN9FMlOykFC0nOpkxcZDXL9kCvk5mWGHI2fh9uVlFOdl87UnttHTo5FRyUrJQkLxi9UHOdnZza0XzQg7FDlLudkZ/Pdr57P+wDF+tb7vYpiSLJQsJO46unr4wR/3cnF5Ee+YNj7scGQEfHjpNM6bPp5/fmo7R090hB2OxICShcTdrzfUcLj5FJ+5YnbYocgISUsz/ueHzuX4yU7+8ddbdKNeElKykLjq6OrhP17YxcLJBVwxryTscGQELZpSwN9eNY/fba7l8XVqjko2ShYSVz97bT/7Gtv4+/fM1x3bSeiuy8u5uLyIL/5qM5uqde9FMlGykLhpbG3nu8/t5NI5E7hyvmoVySgjPY3v3bqU4rxsPv3jNRxobAs7JBkhShYSN1/57TZa27v48gcWq1aRxCbkZfPDOyo41dnDLd9fRXWTEkYyULKQuHh8XTW/3XiIv/qzucybqNllk92U74f7AAANqUlEQVSCSQX89JMX0Xyqk5vuf5Wth46HHZKcJSULibmNB4/xD49v5uLyIv7ySo2AShXnThvHw3dejAM3/uerPLHpUNghyVlQspCY2lx9nI/98DVK8rP591uXkpmut1wqWTxlHL/53KUsmJzP3T9bz988vJ4m3YcxKukvV2LC3fnNhhpueuBV8nMy+fmnL6Y4LzvssCQEpQU5PHLXcj5/1Tx+t6mWK779Aj/44x4tyTrKWLLcPFNRUeFr1qwJOwwBqupa+OZTO3huxxGWzhjP/bddQGlBTthhSQKoqmvh67/bzotV9ZTmZ3P7JWV89KIZjB+bFXZoKcvM1rp7xaDllCxkJNQeP8lz24/w1JZa/rSrkbFZ6Xz+qnn8xaVlZKjpSfp4eWcDD7y0mz/ubCA7I42rFk7k+vOmcOX8ErIz0sMOL6UMNVlkxDiIa4HvAunAD9z9G32OZwM/Bi4AGoGPuPu+4Ng/AJ8EuoG/dvenYxmrDE1Pj3OwqY0tNc1sPXScrYea2XqomYbWdgBmFefy/1w9j9sunklhrv5blP5dNreYy+YWs722mZ+/foAnNtXyu821jM1K55LZxVwxv4Qr5pYwvWiMhlkniJjVLMwsHagCrgaqgdXALe6+LarMZ4F3uPtnzOxm4EPu/hEzWwT8HFgGTAGeBea5+4CNnKpZjLyWU53sPNJK1eEWKuta2Hqome2Hmmlp7wIgI82YU5rH4injWDylgMvmFjO3NE9/3HLGOrt7eHlXA89tr2NlZT3VTScBKM3PZumMQi6YWcjSmeNZMKmA3OyY/o+bchKhZrEM2OXue4KAHgZuALZFlbkB+Eqw/Sjw7xb5pLkBeNjd24G9ZrYruN6rMYw3pZzq7KblVBctpzppauvk0LGTbzz2H21jZ10rNcdOvlF+TGY6Cybnc8P5UzhnyjgWTxnH3Il55GSqyUDOXmZ6Gu+aX8q75pfi7uxtOMGfdjWw7sAx1u5v4vdbD79RdlrhGOZNzGduaR5TC8cwsSCHSQU5lORnk5udQW5Wupo+YyCWyWIqcDDqeTVw0UBl3L3LzI4DE4L9q/qcOzUWQR5r6+DG+yM5qLeW9UZdy9/y5W3HoytlHuzt3de3wnZG576tjPdzzumvO9DPAtDe1UNHdw/9KcjJYFrhWC4sK+TWiTOYNzGf+RPzmVY4hrQ01Rgk9syM8pI8ykvy+NjyyL76lnbWH2ii8nALVUda2VnXwss7GwZ8H2dlpJGblU56WhppBulpRppZ8DUyS2683s3xqGkvnFzAv91yfky/x6iuz5nZncCdADNmDG8RnfQ0Y370HcX2li9v/KLffN7/8eh9b17DBjgn+tv1KdPnImd0bj8xvfV4ZCMzwyjIySQ/J4P8nAzGj8liyvgxTBmfo1XrJCGV5GdzzeJJXLN40hv7enqchhPt1B1v53DzKRpb22lt76Kto5sTHV20tXfT1eO4O909Trc77ryxHRdx+jbTC8fE/HvEMlnUANOjnk8L9vVXptrMMoBxRDq6h3Iu7v4g8CBE+iyGE2R+Tibf++jS4ZwqIiFKSzNK83Mozc/hXMaFHU7Si2XD3mpgrpnNMrMs4GZgRZ8yK4Dbg+0bgec90n6yArjZzLLNbBYwF3g9hrGKiMhpxKxmEfRB3A08TWTo7EPuvtXM7gPWuPsK4IfAT4IO7KNEEgpBuUeIdIZ3AZ873UgoERGJLd2UJyKSwoY6dFbjy0REZFBKFiIiMiglCxERGZSShYiIDErJQkREBpU0o6HMrB7YH3YcUYqBhrCDGAbFHV+KO74U99vNdPeSwQolTbJINGa2ZijD0RKN4o4vxR1finv41AwlIiKDUrIQEZFBKVnEzoNhBzBMiju+FHd8Ke5hUp+FiIgMSjULEREZlJLFCDGzb5vZDjPbZGa/MrPxA5TbZ2abzWyDmYU286GZXWtmlWa2y8zu6ed4tpn9Ijj+mpmVxT/Kt8U03cxeMLNtZrbVzP6mnzJXmtnx4PXdYGb3hhFrX4P93i3i/w9e701mFvoiK2Y2P+p13GBmzWb2t33KJMTrbWYPmdkRM9sSta/IzJ4xs53B18IBzr09KLPTzG7vr0ysDBB3Yn6WuLseI/AArgEygu1vAt8coNw+oDjkWNOB3UA5kAVsBBb1KfNZ4P5g+2bgFwnwGk8Glgbb+UBVP3FfCTwRdqxn+nsH3gs8RWTBw4uB18KOuZ/3zGEiY/IT7vUGLgeWAlui9n0LuCfYvqe/v0mgCNgTfC0MtgtDjjshP0tUsxgh7v4Hd+8Knq4isrpfoloG7HL3Pe7eATwM3NCnzA3Aj4LtR4F3WzwWEz4Nd69193XBdguwnRitzR6CG4Afe8QqYLyZTQ47qCjvBna7eyLd+PoGd3+JyJo40aLfwz8CPtjPqe8BnnH3o+7eBDwDXBuzQPvoL+5E/SxRsoiNTxD5L7E/DvzBzNYGa4iHYSpwMOp5NW//0H2jTPDGPQ5MiEt0QxA0i50PvNbP4eVmttHMnjKzxXENbGCD/d6H8jsJ083Azwc4loivN8BEd68Ntg8DE/spk+ive8J8lsRyDe6kY2bPApP6OfQld/9NUOZLRFb3+z8DXOYyd68xs1LgGTPbEfx3IUNkZnnAY8Dfuntzn8PriDSVtJrZe4FfE1mWN2yj9vceLIt8PfAP/RxO1Nf7LdzdzWxUDf1MtM8S1SzOgLtf5e7n9PPoTRR3AO8HPupBo2I/16gJvh4BfkWkSSjeaoDpUc+nBfv6LWNmGcA4oDEu0Z2GmWUSSRT/x90f73vc3ZvdvTXYfhLINLPiOIf5NkP4vQ/ldxKW64B17l7X90Civt6But6mvODrkX7KJOTrnoifJUoWI8TMrgX+Hrje3dsGKJNrZvm920Q6srb0VzbGVgNzzWxW8F/jzcCKPmVWAL0jQ24Enh/oTRsvQZ/JD4Ht7v4vA5SZ1Nu3YmbLiLzHQ01yQ/y9rwA+HoyKuhg4HtWEErZbGKAJKhFf7yjR7+Hbgd/0U+Zp4BozKwxGS10T7AtNwn6WxKsnPdkfwC4ibZ8bgkfvSKIpwJPBdjmRkUcbga1Emq/Cive9REYT7e6NA7iPyBsUIAf4ZfBzvQ6UJ8BrfBmRdtpNUa/ze4HPAJ8JytwdvLYbiXQOXpIAcff7e+8TtwHfC34fm4GKsOMO4sol8uE/Lmpfwr3eRJJZLdBJpN/hk0T62J4DdgLPAkVB2QrgB1HnfiJ4n+8C/iIB4k7IzxLdwS0iIoNSM5SIiAxKyUJERAalZCEiIoNSshARkUEpWYiIyKCULCQhmFmJmb1sZlvM7INR+39jZlOGeI3W4OsUM3s0av/Pgxk8P29mC4JZOteb2eyR/0nOnpl9xsw+HnYcItE0dFYSgpn9NZEJ1R4nMpb8SjP7AHCBu39liNdodfe8PvsmAS+7+5zg+T1EZvT8pzOILd3du4daXiQZqWYhiaITGAtkA93BFCN/S2Sa6X4Fd6C/Gszp/09R+8ui1gf4AzA1qE18ObjmX5rZC0HZ28zs9eD4A2aWHuxvNbPvmNlGIhPlXWBmLwaTtj0dNY3ESjP7ZnCNKjN7Z7A/3cz+v6CmtMnM/irY3+91+vxcXzGzvzvd9fuUvzK45m/MbI+ZfcPMPhqcs7m3BhXU3h4zs9XB49Jg/7LgdVxvZq+Y2fxg/x1m9riZ/d4iaz18K+pn+6/gZ9tsZp8f6i9ZRrGw7xDVQw93h8jcU78D1hCZDvuvgTsGOWcF8PFg+3NAa7BdRrA+QPR28PwrwN8F2wuB3wKZwfP/iLqeAzcF25nAK0BJ8PwjwEPB9krgO8H2e4Fng+2/JDK1e++6BEWnu06fnys6xn6v36f8lcAxIut9ZBOZ2+irwbG/Af412P4ZkcnnAGYQmTYFoCAqzquAx4LtO4is7zCOyB39+4nMo3QBkWm9e7//+LDfP3rE/qFZZyUhuPtx4H0AwRw99wAfMrPvE1mU5jvu/mqf0y4FPhxs/4TIQjFn4t1EPvhWB9MbjeHNyea6iUxYCDAfOIfIzJ4QWQgoet6m3gkN1xJJThD50L3fg3UJ3P2omZ0zyHUG0t/1+1rtwVxSZrabSI0KIlOHvCsqpkX25rIkBRaZwXcc8CMzm0skSWZGXfe54HeDmW0DZhKZXqLczP6NSIL/A5L0lCwkEf0j8HUiE9i9TOQ/9MeJLFTT19l0uhnwI3fvb+rtU/5mP4UBW919+QDXaQ++dnP6v6nBrjOQoVy/PWq7J+p5T9Q5acDF7n7qLUGZ/Tvwgrt/yCLrhKwc4LrdRGogTWa2hMjv4zPATUTmV5Ikpj4LSSjBf7fT3H0lkT6MHiIJYUw/xf9EZMZcgI8O49s9B9xokfUAetdsntlPuUqgxMyWB+UybfBFfp4B7gr6XjCzomFeZyT9Afir3idmdl6wOY43p+W+Y7CLWGQK8jR3fwz4H0SWBZUkp2QhiebrwJeC7Z8TaftfDXy3n7J/A3zOzDYzjNXN3H0bkQ+7P5jZJiIf8G/rcPbI0rM3At8MOrw3AJcMcvkfAAeATcE5tw7zOiPpr4GKoMN9G5FaAUQGEfyzma1naK0NU4GVZrYB+Cn9L4okSUZDZ0VEZFCqWYiIyKCULEREZFBKFiIiMiglCxERGZSShYiIDErJQkREBqVkISIig1KyEBGRQf1fVgqG98y4YMUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f44d7dc7d68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Creating a list with bootstrapped means for each AB-group\n",
    "boot_7d = []\n",
    "for i in range(500):\n",
    "    boot_mean = df.sample(frac=1 , replace=True).groupby('version')['retention_7'].mean()\n",
    "    boot_7d.append(boot_mean)\n",
    "    \n",
    "# Transforming the list to a DataFrame\n",
    "boot_7d = pd.DataFrame(boot_7d)\n",
    "print(boot_7d.head())\n",
    "# Adding a column with the % difference between the two AB-groups\n",
    "boot_7d['diff'] = (boot_7d['gate_30'] - boot_7d['gate_40']) /  boot_7d['gate_40'] * 100\n",
    "\n",
    "# Ploting the bootstrap % difference\n",
    "ax = boot_7d['diff'].plot(kind='kde')\n",
    "ax.set_xlabel(\"% difference in means\")\n",
    "\n",
    "# Calculating the probability that 7-day retention is greater when the gate is at level 30\n",
    "prob = (boot_7d['diff'] > 0).sum() / len(boot_7d['diff'])\n",
    "\n",
    "# Pretty printing the probability\n",
    "print((prob*100), \"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "75"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "## 11.  The conclusion\n",
    "<p>The bootstrap result tells us that there is strong evidence that 7-day retention is higher when the gate is at level 30 than when it is at level 40. The conclusion is: If we want to keep retention high — both 1-day and 7-day retention — we should <strong>not</strong> move the gate from level 30 to level 40. There are, of course, other metrics we could look at, like the number of game rounds played or how much in-game purchases are made by the two AB-groups. But retention <em>is</em> one of the most important metrics. If we don't retain our player base, it doesn't matter how much money they spend in-game.</p>\n",
    "<p><img src=\"https://s3.amazonaws.com/assets.datacamp.com/production/project_184/img/cookie_yellow.png\" style=\"width:100px; float:center\"> </p>\n",
    "<p>So, why is retention higher when the gate is positioned earlier? One could expect the opposite: The later the obstacle, the longer people are going to engage with the game. But this is not what the data tells us. The theory of <em>hedonic adaptation</em> can give one explanation for this. In short, hedonic adaptation is the tendency for people to get less and less enjoyment out of a fun activity over time if that activity is undertaken continuously. By forcing players to take a break when they reach a gate, their enjoyment of the game is prolonged. But when the gate is moved to level 40, fewer players make it far enough, and they are more likely to quit the game because they simply got bored of it. </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {
    "collapsed": true,
    "dc": {
     "key": "75"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# So, given the data and the bootstrap analysis\n",
    "# Should we move the gate from level 30 to level 40 ?\n",
    "move_to_level_40 = False"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}