ab_test_dating_project_aeremina.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Проект: вариант 3\n",
    "\n",
    "### Представьте, что вы работаете в крупном дейтинговом приложении.\n",
    "Помимо базовых функций, в приложении также имеется премиум-подписка, которая дает доступ к ряду важных дополнительных возможностей. Был проведен A/B тест, в рамках которого для новых пользователей из нескольких стран была изменена стоимость премиум-подписки* при покупке через две новые платежные системы. При этом стоимость пробного периода оставалась прежней.\n",
    "\n",
    "**Проверьте:**\n",
    "Был ли эксперимент успешен в целом.\n",
    "\n",
    "_*Деньги за подписку списываются ежемесячно до тех пор, пока пользователь её не отменит_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Данные\n",
    "\n",
    "**Всего есть три группы: тестовая (test), контрольная 1 (control_1) и контрольная 2 (control_2). Для каждой из них:**\n",
    "\n",
    "users_*.csv – информация о пользователях:\n",
    "\n",
    "- uid – идентификатор пользователя\n",
    "- age – возраст\n",
    "- attraction_coeff – коэффициент привлекательности (от 0 до 1000, лайки / просмотры * 1000)\n",
    "- coins – число монеток (внутренняя валюта)\n",
    "- country – страна  \n",
    "- visit_days – в какие дни после регистрации пользователь посещал приложение (напр. в 1, затем в 7)\n",
    "- gender – пол\n",
    "- age_filter_start  – фильтр поиска, мин. значение \n",
    "- age_filter_end  – фильтр поиска, макс. значение \n",
    "- views_count – число полученных оценок \n",
    "- was_premium – был ли когда-либо премиум (либо пробный период премиум-статуса, либо купленный за деньги)\n",
    "- is_premium –  является ли премиум\n",
    "- total_revenue – нормированная выручка\n",
    "\n",
    "transactions_*.csv – информация о платежах пользователей:\n",
    "\n",
    "- uid – идентификатор пользователя\n",
    "- country – страна\n",
    "- joined_at – дата и время регистрации\n",
    "- paid_at – дата и время покупки\n",
    "- revenue – нормированная выручка\n",
    "- payment_id – идентификатор платежа\n",
    "- from_page – откуда пользователь перешел на страницу оплаты\n",
    "- product_type – тип продукта (trial_premium – пробная премиум-подписка, premium_no_trial – премиум-подписка без пробной, coins – подписка за внутреннюю валюту, other_type – другое) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Файлы\n",
    "\n",
    "- users_test – информация о пользователях в тестовой группе\n",
    "- users_control_1 – информация о пользователях в первой контрольной группе\n",
    "- users_control_2 – информация о пользователях во второй контрольной группе \n",
    "- transactions_test – информация о платежах пользователей в тестовой группе \n",
    "- transactions_control_1 – информация о платежах пользователей в первой контрольной группе\n",
    "- transactions_control_2 – информация о платежах пользователей во второй контрольной группе "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### План\n",
    "\n",
    "1. **Сначала сделаем небольшой предварительный анализ:**\n",
    "    - Посмотрим на данные и изучим их, внесем необходимые изменения\n",
    "    - Отберем нужных пользователей (новых, из определенных стран и с определенными платежными системами оплаты)\n",
    "    \n",
    "    \n",
    "2. **Общая оценка данных:**\n",
    "    - Сравним группы для определения репрезентативности выборок\n",
    "    - Построим графики\n",
    "    \n",
    "       \n",
    "3. **A/A/B тесты:**\n",
    "     - Выбираем гипотезы \n",
    "     - Выбираем подходящие статистические тесты\n",
    "     - Тестируем\n",
    "     \n",
    "         \n",
    "4. **Подводим итоги:**\n",
    "    - Формулируем вывод \n",
    "    - Отвечаем на заданный вопрос: был ли эксперимент по изменению стоимости премиум-подписки в приложении успешен в целом"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1: Предварительный анализ\n",
    "\n",
    "**Загружаем данные**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Импортируем нужные библиотеки\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import requests\n",
    "from io import BytesIO\n",
    "import datetime \n",
    "\n",
    "import scipy.stats as ss\n",
    "from scipy.stats import chi2_contingency, f_oneway, kruskal, ttest_ind, mannwhitneyu\n",
    "import pingouin as pg\n",
    "\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "%matplotlib inline\n",
    "sns.set_style(\"white\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Скачиваем данные с Яндекс.Диска\n",
    "def get_data(public_key):\n",
    "    base_url = 'https://cloud-api.yandex.net/v1/disk/public/resources/download?'\n",
    "    response = requests.get(base_url + 'public_key=' + public_key)\n",
    "    download_response = requests.get(response.json()['href'])\n",
    "    df = pd.read_csv(BytesIO(download_response.content), sep = ';')\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Читаем наши файлы\n",
    "users_test             = get_data('https://disk.yandex.ru/d/4XXIME4osGrMRA')\n",
    "users_control_1        = get_data('https://disk.yandex.ru/d/yJFydMNNGkEKfg')\n",
    "users_control_2        = get_data('https://disk.yandex.ru/d/br6KkQupzzTGoQ')\n",
    "transactions_test      = get_data('https://disk.yandex.ru/d/gvCWpZ55ODzs2g')\n",
    "transactions_control_1 = get_data('https://disk.yandex.ru/d/VY5W0keMX5TZBQ')\n",
    "transactions_control_2 = get_data('https://disk.yandex.ru/d/th5GL0mGOc-qzg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Посмотрим на наши df, изучим их тип данных и пропущенные значения, внесем необходимые изменения**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>age</th>\n",
       "      <th>attraction_coeff</th>\n",
       "      <th>coins</th>\n",
       "      <th>country</th>\n",
       "      <th>visit_days</th>\n",
       "      <th>gender</th>\n",
       "      <th>age_filter_start</th>\n",
       "      <th>age_filter_end</th>\n",
       "      <th>views_count</th>\n",
       "      <th>was_premium</th>\n",
       "      <th>is_premium</th>\n",
       "      <th>total_revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>892309896</td>\n",
       "      <td>27</td>\n",
       "      <td>685</td>\n",
       "      <td>NaN</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>1,2,3,4,5,6,7,8,9,10,11,12,16,17</td>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>30</td>\n",
       "      <td>89</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>892044516</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Germany</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>892185708</td>\n",
       "      <td>45</td>\n",
       "      <td>44</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Israel</td>\n",
       "      <td>1,3,4,6,7,8,10,11,12,13,14,15,16,17,18,19,20,2...</td>\n",
       "      <td>1</td>\n",
       "      <td>42</td>\n",
       "      <td>48</td>\n",
       "      <td>68</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>892130292</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>1,2,3,4,5,6,7,8,9,10,11,12</td>\n",
       "      <td>1</td>\n",
       "      <td>29</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>891406035</td>\n",
       "      <td>27</td>\n",
       "      <td>1000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>France</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         uid  age  attraction_coeff  coins                   country  \\\n",
       "0  892309896   27               685    NaN  United States of America   \n",
       "1  892044516   27                 0    NaN                   Germany   \n",
       "2  892185708   45                44    NaN                    Israel   \n",
       "3  892130292   32                 0    NaN  United States of America   \n",
       "4  891406035   27              1000    NaN                    France   \n",
       "\n",
       "                                          visit_days  gender  \\\n",
       "0                   1,2,3,4,5,6,7,8,9,10,11,12,16,17       1   \n",
       "1                                                NaN       1   \n",
       "2  1,3,4,6,7,8,10,11,12,13,14,15,16,17,18,19,20,2...       1   \n",
       "3                         1,2,3,4,5,6,7,8,9,10,11,12       1   \n",
       "4                                                NaN       1   \n",
       "\n",
       "   age_filter_start  age_filter_end  views_count  was_premium  is_premium  \\\n",
       "0                24              30           89          NaN         NaN   \n",
       "1                24              30            0          NaN         NaN   \n",
       "2                42              48           68          NaN         NaN   \n",
       "3                29              35            0          NaN         NaN   \n",
       "4                24              30            1          1.0         NaN   \n",
       "\n",
       "   total_revenue  \n",
       "0              0  \n",
       "1              0  \n",
       "2              0  \n",
       "3              0  \n",
       "4              0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Пользователи тестовой группы\n",
    "users_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4308 entries, 0 to 4307\n",
      "Data columns (total 13 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   uid               4308 non-null   int64  \n",
      " 1   age               4308 non-null   int64  \n",
      " 2   attraction_coeff  4308 non-null   int64  \n",
      " 3   coins             49 non-null     float64\n",
      " 4   country           4308 non-null   object \n",
      " 5   visit_days        2978 non-null   object \n",
      " 6   gender            4308 non-null   int64  \n",
      " 7   age_filter_start  4308 non-null   int64  \n",
      " 8   age_filter_end    4308 non-null   int64  \n",
      " 9   views_count       4308 non-null   int64  \n",
      " 10  was_premium       408 non-null    float64\n",
      " 11  is_premium        157 non-null    float64\n",
      " 12  total_revenue     4308 non-null   int64  \n",
      "dtypes: float64(3), int64(8), object(2)\n",
      "memory usage: 437.7+ KB\n"
     ]
    }
   ],
   "source": [
    "users_test.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Есть пропуски в колонках:\n",
    "- coins (у пользователя нет внутренней валюты, т.е. она равно 0)\n",
    "- visit_days (пользователь не посещал приложение после регистрации, т.е. 0 дней)\n",
    "- was_premium (у пользователя не было премиум подписки, т.е. 0, т.к. наличие подписки обозначается как 1)\n",
    "- is_premium (у пользователя на настоящий момент нет премиум подписки, т.е. 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Обозначение гендера\n",
    "users_test.gender.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Можно предположить, что gender 1 - мужчина, 0 - женщина"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4308"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Число уникальных пользователей\n",
    "users_test.uid.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>age</th>\n",
       "      <th>attraction_coeff</th>\n",
       "      <th>coins</th>\n",
       "      <th>country</th>\n",
       "      <th>visit_days</th>\n",
       "      <th>gender</th>\n",
       "      <th>age_filter_start</th>\n",
       "      <th>age_filter_end</th>\n",
       "      <th>views_count</th>\n",
       "      <th>was_premium</th>\n",
       "      <th>is_premium</th>\n",
       "      <th>total_revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>892319115</td>\n",
       "      <td>25</td>\n",
       "      <td>435</td>\n",
       "      <td>NaN</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>1,2,3,4,5,6</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>32</td>\n",
       "      <td>982</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>891248523</td>\n",
       "      <td>29</td>\n",
       "      <td>500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>1,2</td>\n",
       "      <td>1</td>\n",
       "      <td>26</td>\n",
       "      <td>32</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>891670932</td>\n",
       "      <td>33</td>\n",
       "      <td>800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>France</td>\n",
       "      <td>1,2,3,5,6,8,9,11,13,16</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>36</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>891060786</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Argentina</td>\n",
       "      <td>1,2,3,4,5</td>\n",
       "      <td>1</td>\n",
       "      <td>23</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>892006554</td>\n",
       "      <td>35</td>\n",
       "      <td>294</td>\n",
       "      <td>NaN</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>1,2,3,5,6,7,8,9,10,12,15,16,17,19</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>40</td>\n",
       "      <td>17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         uid  age  attraction_coeff  coins                   country  \\\n",
       "0  892319115   25               435    NaN  United States of America   \n",
       "1  891248523   29               500    NaN  United States of America   \n",
       "2  891670932   33               800    NaN                    France   \n",
       "3  891060786   26                 0    NaN                 Argentina   \n",
       "4  892006554   35               294    NaN  United States of America   \n",
       "\n",
       "                          visit_days  gender  age_filter_start  \\\n",
       "0                        1,2,3,4,5,6       0                22   \n",
       "1                                1,2       1                26   \n",
       "2             1,2,3,5,6,8,9,11,13,16       1                30   \n",
       "3                          1,2,3,4,5       1                23   \n",
       "4  1,2,3,5,6,7,8,9,10,12,15,16,17,19       1                30   \n",
       "\n",
       "   age_filter_end  views_count  was_premium  is_premium  total_revenue  \n",
       "0              32          982          NaN         NaN              0  \n",
       "1              32           12          NaN         NaN              0  \n",
       "2              36            5          NaN         NaN              0  \n",
       "3              29            0          NaN         NaN              0  \n",
       "4              40           17          NaN         NaN              0  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Пользователи 1 контрольной группы\n",
    "users_control_1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4340 entries, 0 to 4339\n",
      "Data columns (total 13 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   uid               4340 non-null   int64  \n",
      " 1   age               4340 non-null   int64  \n",
      " 2   attraction_coeff  4340 non-null   int64  \n",
      " 3   coins             60 non-null     float64\n",
      " 4   country           4340 non-null   object \n",
      " 5   visit_days        3016 non-null   object \n",
      " 6   gender            4340 non-null   int64  \n",
      " 7   age_filter_start  4340 non-null   int64  \n",
      " 8   age_filter_end    4340 non-null   int64  \n",
      " 9   views_count       4340 non-null   int64  \n",
      " 10  was_premium       436 non-null    float64\n",
      " 11  is_premium        192 non-null    float64\n",
      " 12  total_revenue     4340 non-null   int64  \n",
      "dtypes: float64(3), int64(8), object(2)\n",
      "memory usage: 440.9+ KB\n"
     ]
    }
   ],
   "source": [
    "users_control_1.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4340"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Число уникальных пользователей\n",
    "users_control_1.uid.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>age</th>\n",
       "      <th>attraction_coeff</th>\n",
       "      <th>coins</th>\n",
       "      <th>country</th>\n",
       "      <th>visit_days</th>\n",
       "      <th>gender</th>\n",
       "      <th>age_filter_start</th>\n",
       "      <th>age_filter_end</th>\n",
       "      <th>views_count</th>\n",
       "      <th>was_premium</th>\n",
       "      <th>is_premium</th>\n",
       "      <th>total_revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>892035504</td>\n",
       "      <td>37</td>\n",
       "      <td>137</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Israel</td>\n",
       "      <td>1,2,3,4,5,6,7,8,9,11,13</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>40</td>\n",
       "      <td>51</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>891782112</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Italy</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>54</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>891110337</td>\n",
       "      <td>30</td>\n",
       "      <td>769</td>\n",
       "      <td>NaN</td>\n",
       "      <td>France</td>\n",
       "      <td>1,2,3,4,5</td>\n",
       "      <td>1</td>\n",
       "      <td>27</td>\n",
       "      <td>33</td>\n",
       "      <td>13</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>891796320</td>\n",
       "      <td>48</td>\n",
       "      <td>750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>France</td>\n",
       "      <td>1,4,6,8,10,11,12,14,15,16,18</td>\n",
       "      <td>1</td>\n",
       "      <td>45</td>\n",
       "      <td>51</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>891880212</td>\n",
       "      <td>54</td>\n",
       "      <td>638</td>\n",
       "      <td>NaN</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "      <td>53</td>\n",
       "      <td>94</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         uid  age  attraction_coeff  coins                   country  \\\n",
       "0  892035504   37               137    NaN                    Israel   \n",
       "1  891782112   57                 0    NaN                     Italy   \n",
       "2  891110337   30               769    NaN                    France   \n",
       "3  891796320   48               750    NaN                    France   \n",
       "4  891880212   54               638    NaN  United States of America   \n",
       "\n",
       "                     visit_days  gender  age_filter_start  age_filter_end  \\\n",
       "0       1,2,3,4,5,6,7,8,9,11,13       1                30              40   \n",
       "1                             1       1                54              60   \n",
       "2                     1,2,3,4,5       1                27              33   \n",
       "3  1,4,6,8,10,11,12,14,15,16,18       1                45              51   \n",
       "4                             1       1                35              53   \n",
       "\n",
       "   views_count  was_premium  is_premium  total_revenue  \n",
       "0           51          NaN         NaN              0  \n",
       "1            0          NaN         NaN              0  \n",
       "2           13          NaN         NaN              0  \n",
       "3           12          NaN         NaN              0  \n",
       "4           94          NaN         NaN              0  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Пользователи 2 контрольной группы\n",
    "users_control_2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4264 entries, 0 to 4263\n",
      "Data columns (total 13 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   uid               4264 non-null   int64  \n",
      " 1   age               4264 non-null   int64  \n",
      " 2   attraction_coeff  4264 non-null   int64  \n",
      " 3   coins             61 non-null     float64\n",
      " 4   country           4264 non-null   object \n",
      " 5   visit_days        2900 non-null   object \n",
      " 6   gender            4264 non-null   int64  \n",
      " 7   age_filter_start  4264 non-null   int64  \n",
      " 8   age_filter_end    4264 non-null   int64  \n",
      " 9   views_count       4264 non-null   int64  \n",
      " 10  was_premium       411 non-null    float64\n",
      " 11  is_premium        191 non-null    float64\n",
      " 12  total_revenue     4264 non-null   int64  \n",
      "dtypes: float64(3), int64(8), object(2)\n",
      "memory usage: 433.2+ KB\n"
     ]
    }
   ],
   "source": [
    "users_control_2.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4264"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Число уникальных пользователей\n",
    "users_control_2.uid.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверим, уникальны ли пользователи в каждой группе, нет ли одних и тех же юзеров в разных выборках:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_test.query('uid in @users_control_1.uid').shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_test.query('uid in @users_control_2.uid').shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_control_1.query('uid in @users_control_2.uid').shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Во всех df строки с уникальными пользователями. Но можем дополнительно проверить df на дубликаты:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_test.duplicated().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_control_1.duplicated().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_control_2.duplicated().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В контрольных группах пропуски в тех же колонках, что и в тестовой, все их можно заменить на нули, смысл от этого не изменится:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_test      = users_test.fillna(0)\n",
    "users_control_1 = users_control_1.fillna(0)\n",
    "users_control_2 = users_control_2.fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Изучим датасеты с платежами пользователей**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>country</th>\n",
       "      <th>joined_at</th>\n",
       "      <th>paid_at</th>\n",
       "      <th>revenue</th>\n",
       "      <th>payment_id</th>\n",
       "      <th>from_page</th>\n",
       "      <th>product_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>891345942</td>\n",
       "      <td>Italy</td>\n",
       "      <td>2017-05-11 13:00:00</td>\n",
       "      <td>2017-11-13 15:04:00</td>\n",
       "      <td>12909</td>\n",
       "      <td>147</td>\n",
       "      <td>trial_vip_popup</td>\n",
       "      <td>trial_premium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>892054251</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>2017-10-22 00:33:00</td>\n",
       "      <td>2017-10-30 01:37:00</td>\n",
       "      <td>13923</td>\n",
       "      <td>147</td>\n",
       "      <td>trial_vip_popup</td>\n",
       "      <td>trial_premium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>892236423</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>2017-10-18 01:09:00</td>\n",
       "      <td>2017-10-23 00:15:00</td>\n",
       "      <td>3783</td>\n",
       "      <td>67</td>\n",
       "      <td>menu</td>\n",
       "      <td>other_type</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>892236423</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>2017-10-18 01:09:00</td>\n",
       "      <td>2017-10-27 22:38:00</td>\n",
       "      <td>3783</td>\n",
       "      <td>67</td>\n",
       "      <td>menu</td>\n",
       "      <td>other_type</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>892168170</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>2017-10-19 17:10:00</td>\n",
       "      <td>2017-10-27 19:10:00</td>\n",
       "      <td>9087</td>\n",
       "      <td>147</td>\n",
       "      <td>trial_vip_popup</td>\n",
       "      <td>trial_premium</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         uid                   country            joined_at  \\\n",
       "0  891345942                     Italy  2017-05-11 13:00:00   \n",
       "1  892054251  United States of America  2017-10-22 00:33:00   \n",
       "2  892236423  United States of America  2017-10-18 01:09:00   \n",
       "3  892236423  United States of America  2017-10-18 01:09:00   \n",
       "4  892168170  United States of America  2017-10-19 17:10:00   \n",
       "\n",
       "               paid_at  revenue  payment_id        from_page   product_type  \n",
       "0  2017-11-13 15:04:00    12909         147  trial_vip_popup  trial_premium  \n",
       "1  2017-10-30 01:37:00    13923         147  trial_vip_popup  trial_premium  \n",
       "2  2017-10-23 00:15:00     3783          67             menu     other_type  \n",
       "3  2017-10-27 22:38:00     3783          67             menu     other_type  \n",
       "4  2017-10-27 19:10:00     9087         147  trial_vip_popup  trial_premium  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Платежи пользователей в тестовой группе\n",
    "transactions_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 273 entries, 0 to 272\n",
      "Data columns (total 8 columns):\n",
      " #   Column        Non-Null Count  Dtype \n",
      "---  ------        --------------  ----- \n",
      " 0   uid           273 non-null    int64 \n",
      " 1   country       273 non-null    object\n",
      " 2   joined_at     273 non-null    object\n",
      " 3   paid_at       273 non-null    object\n",
      " 4   revenue       273 non-null    int64 \n",
      " 5   payment_id    273 non-null    int64 \n",
      " 6   from_page     273 non-null    object\n",
      " 7   product_type  273 non-null    object\n",
      "dtypes: int64(3), object(5)\n",
      "memory usage: 17.2+ KB\n"
     ]
    }
   ],
   "source": [
    "transactions_test.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "146"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Число уникальных uid в тестовой группе\n",
    "transactions_test.uid.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>country</th>\n",
       "      <th>joined_at</th>\n",
       "      <th>paid_at</th>\n",
       "      <th>revenue</th>\n",
       "      <th>payment_id</th>\n",
       "      <th>from_page</th>\n",
       "      <th>product_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>891319275.0</td>\n",
       "      <td>France</td>\n",
       "      <td>2017-05-11 19:57:00</td>\n",
       "      <td>2017-12-11 21:57:00</td>\n",
       "      <td>12727.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>trial_vip_popup</td>\n",
       "      <td>trial_premium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>892421826.0</td>\n",
       "      <td>Israel</td>\n",
       "      <td>2017-10-14 12:46:00</td>\n",
       "      <td>2017-10-23 12:54:00</td>\n",
       "      <td>14586.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>trial_vip_popup</td>\n",
       "      <td>trial_premium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>891822480.0</td>\n",
       "      <td>Italy</td>\n",
       "      <td>2017-10-26 22:27:00</td>\n",
       "      <td>2017-12-11 20:59:00</td>\n",
       "      <td>1911.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>none</td>\n",
       "      <td>coins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>891367521.0</td>\n",
       "      <td>Italy</td>\n",
       "      <td>2017-05-11 02:37:00</td>\n",
       "      <td>2017-06-11 19:15:00</td>\n",
       "      <td>1456.0</td>\n",
       "      <td>127.0</td>\n",
       "      <td>sympathy</td>\n",
       "      <td>other_type</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>892207959.0</td>\n",
       "      <td>France</td>\n",
       "      <td>2017-10-18 18:30:00</td>\n",
       "      <td>2017-12-11 12:03:00</td>\n",
       "      <td>1261.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>empty_likes</td>\n",
       "      <td>coins</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           uid country            joined_at              paid_at  revenue  \\\n",
       "0  891319275.0  France  2017-05-11 19:57:00  2017-12-11 21:57:00  12727.0   \n",
       "1  892421826.0  Israel  2017-10-14 12:46:00  2017-10-23 12:54:00  14586.0   \n",
       "2  891822480.0   Italy  2017-10-26 22:27:00  2017-12-11 20:59:00   1911.0   \n",
       "3  891367521.0   Italy  2017-05-11 02:37:00  2017-06-11 19:15:00   1456.0   \n",
       "4  892207959.0  France  2017-10-18 18:30:00  2017-12-11 12:03:00   1261.0   \n",
       "\n",
       "   payment_id        from_page   product_type  \n",
       "0       147.0  trial_vip_popup  trial_premium  \n",
       "1       147.0  trial_vip_popup  trial_premium  \n",
       "2        19.0             none          coins  \n",
       "3       127.0         sympathy     other_type  \n",
       "4        19.0      empty_likes          coins  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Платежи пользователей в 1 контрольной группе\n",
    "transactions_control_1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1007 entries, 0 to 1006\n",
      "Data columns (total 8 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   uid           377 non-null    float64\n",
      " 1   country       377 non-null    object \n",
      " 2   joined_at     1007 non-null   object \n",
      " 3   paid_at       1007 non-null   object \n",
      " 4   revenue       377 non-null    float64\n",
      " 5   payment_id    377 non-null    float64\n",
      " 6   from_page     377 non-null    object \n",
      " 7   product_type  377 non-null    object \n",
      "dtypes: float64(3), object(5)\n",
      "memory usage: 63.1+ KB\n"
     ]
    }
   ],
   "source": [
    "transactions_control_1.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "193"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Число уникальных uid в 1 контрольной группе\n",
    "transactions_control_1.uid.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>country</th>\n",
       "      <th>joined_at</th>\n",
       "      <th>paid_at</th>\n",
       "      <th>revenue</th>\n",
       "      <th>payment_id</th>\n",
       "      <th>from_page</th>\n",
       "      <th>product_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>891266616</td>\n",
       "      <td>Argentina</td>\n",
       "      <td>2017-06-11 15:25:00</td>\n",
       "      <td>2017-10-11 17:35:00</td>\n",
       "      <td>6305</td>\n",
       "      <td>19</td>\n",
       "      <td>autorefill</td>\n",
       "      <td>coins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>892186737</td>\n",
       "      <td>Chile</td>\n",
       "      <td>2017-10-19 06:03:00</td>\n",
       "      <td>2017-04-11 08:53:00</td>\n",
       "      <td>4732</td>\n",
       "      <td>147</td>\n",
       "      <td>promo_09</td>\n",
       "      <td>premium_no_trial</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>891959004</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>2017-10-24 00:12:00</td>\n",
       "      <td>2017-10-31 02:12:00</td>\n",
       "      <td>10153</td>\n",
       "      <td>147</td>\n",
       "      <td>trial_vip_popup</td>\n",
       "      <td>trial_premium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>892115478</td>\n",
       "      <td>Spain</td>\n",
       "      <td>2017-10-20 20:39:00</td>\n",
       "      <td>2017-10-26 06:26:00</td>\n",
       "      <td>2366</td>\n",
       "      <td>67</td>\n",
       "      <td>empty_likes</td>\n",
       "      <td>other_type</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>891592941</td>\n",
       "      <td>Switzerland</td>\n",
       "      <td>2017-10-31 12:40:00</td>\n",
       "      <td>2017-03-11 16:25:00</td>\n",
       "      <td>6292</td>\n",
       "      <td>147</td>\n",
       "      <td>promo_09</td>\n",
       "      <td>premium_no_trial</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         uid                   country            joined_at  \\\n",
       "0  891266616                 Argentina  2017-06-11 15:25:00   \n",
       "1  892186737                     Chile  2017-10-19 06:03:00   \n",
       "2  891959004  United States of America  2017-10-24 00:12:00   \n",
       "3  892115478                     Spain  2017-10-20 20:39:00   \n",
       "4  891592941               Switzerland  2017-10-31 12:40:00   \n",
       "\n",
       "               paid_at  revenue  payment_id        from_page      product_type  \n",
       "0  2017-10-11 17:35:00     6305          19       autorefill             coins  \n",
       "1  2017-04-11 08:53:00     4732         147         promo_09  premium_no_trial  \n",
       "2  2017-10-31 02:12:00    10153         147  trial_vip_popup     trial_premium  \n",
       "3  2017-10-26 06:26:00     2366          67      empty_likes        other_type  \n",
       "4  2017-03-11 16:25:00     6292         147         promo_09  premium_no_trial  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Платежи пользователей в 2 контрольной группе\n",
    "transactions_control_2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 328 entries, 0 to 327\n",
      "Data columns (total 8 columns):\n",
      " #   Column        Non-Null Count  Dtype \n",
      "---  ------        --------------  ----- \n",
      " 0   uid           328 non-null    int64 \n",
      " 1   country       328 non-null    object\n",
      " 2   joined_at     328 non-null    object\n",
      " 3   paid_at       328 non-null    object\n",
      " 4   revenue       328 non-null    int64 \n",
      " 5   payment_id    328 non-null    int64 \n",
      " 6   from_page     328 non-null    object\n",
      " 7   product_type  328 non-null    object\n",
      "dtypes: int64(3), object(5)\n",
      "memory usage: 20.6+ KB\n"
     ]
    }
   ],
   "source": [
    "transactions_control_2.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "187"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Число уникальных uid во 2 контрольной группе\n",
    "transactions_control_2.uid.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверим уникальность юзеров в датасетах с транзакциями:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_test.query('uid in @transactions_control_1.uid').shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_test.query('uid in @transactions_control_2.uid').shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_control_2.query('uid in @transactions_control_1.uid').shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверим данные на дубликаты и удалим их при наличии:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_test.duplicated().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "650"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_control_1.duplicated().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_control_2.duplicated().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions_test      = transactions_test.drop_duplicates()\n",
    "transactions_control_1 = transactions_control_1.drop_duplicates()\n",
    "transactions_control_2 = transactions_control_2.drop_duplicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В тестовой группе и 2 контрольной группе нет пропусков.\n",
    "В 1 контрольной группе странная ситуация - количество строк значительно больше, чем в предыдущих датафреймах, но также много пропусков во всех колонках кроме joined_at и paid_at, предполагаем, что это сбой в системе и удаляем эти строки:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions_control_1 = transactions_control_1.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Заменим в df transactions_control_1 тип данных на int у колонок с id для унификации\n",
    "transactions_control_1[['uid', 'payment_id']] = transactions_control_1[['uid', 'payment_id']].astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Исправим некорректный для дат тип данных в df по платежам\n",
    "transactions_test[['joined_at', 'paid_at']]      = transactions_test[['joined_at', 'paid_at']].apply(pd.to_datetime)\n",
    "transactions_control_1[['joined_at', 'paid_at']] = transactions_control_1[['joined_at', 'paid_at']].apply(pd.to_datetime)\n",
    "transactions_control_2[['joined_at', 'paid_at']] = transactions_control_2[['joined_at', 'paid_at']].apply(pd.to_datetime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Отберем нужных нам пользователей (новых, из определенных стран и с определенными платежными системами оплаты)**\n",
    "\n",
    "Новых пользователей найдем по датам регистрации, нужно чтобы они сходились во всех группах:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Timestamp('2017-01-11 03:07:00'), Timestamp('2017-10-31 23:40:00'))"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Период проведения эксперимента в тестовой группе\n",
    "transactions_test.joined_at.min(), transactions_test.joined_at.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Timestamp('2015-01-19 11:49:00'), Timestamp('2017-10-31 22:17:00'))"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Период проведения эксперимента в 1 контрольной группе\n",
    "transactions_control_1.joined_at.min(), transactions_control_1.joined_at.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Timestamp('2017-01-11 08:36:00'), Timestamp('2017-10-31 17:43:00'))"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Период проведения эксперимента во 2 контрольной группе\n",
    "transactions_control_2.joined_at.min(), transactions_control_2.joined_at.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Окончание эксперимента плюс минус совпадает у всех групп (не будем сверять всех по времени, даты достаточно).\n",
    "\n",
    "А вот минимальная дата в 1 контрольной группе не совпадает с минимальными датами в других группах, отфильтруем лишнее:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>country</th>\n",
       "      <th>joined_at</th>\n",
       "      <th>paid_at</th>\n",
       "      <th>revenue</th>\n",
       "      <th>payment_id</th>\n",
       "      <th>from_page</th>\n",
       "      <th>product_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>960936960</td>\n",
       "      <td>Russian Federation</td>\n",
       "      <td>2015-01-19 11:49:00</td>\n",
       "      <td>2016-12-14 17:30:00</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>refund_VP</td>\n",
       "      <td>other_type</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>960936960</td>\n",
       "      <td>Russian Federation</td>\n",
       "      <td>2015-01-19 11:49:00</td>\n",
       "      <td>2016-12-14 17:35:00</td>\n",
       "      <td>637.0</td>\n",
       "      <td>0</td>\n",
       "      <td>refung_AP</td>\n",
       "      <td>other_type</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224</th>\n",
       "      <td>960936960</td>\n",
       "      <td>Russian Federation</td>\n",
       "      <td>2015-01-19 11:49:00</td>\n",
       "      <td>2016-12-14 16:16:00</td>\n",
       "      <td>143.0</td>\n",
       "      <td>0</td>\n",
       "      <td>refund_WP</td>\n",
       "      <td>other_type</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           uid             country           joined_at             paid_at  \\\n",
       "65   960936960  Russian Federation 2015-01-19 11:49:00 2016-12-14 17:30:00   \n",
       "101  960936960  Russian Federation 2015-01-19 11:49:00 2016-12-14 17:35:00   \n",
       "224  960936960  Russian Federation 2015-01-19 11:49:00 2016-12-14 16:16:00   \n",
       "\n",
       "     revenue  payment_id  from_page product_type  \n",
       "65      26.0           0  refund_VP   other_type  \n",
       "101    637.0           0  refung_AP   other_type  \n",
       "224    143.0           0  refund_WP   other_type  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_control_1.query('joined_at < \"2017-01-11\"')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Уберем пользователя из обоих df\n",
    "transactions_control_1 = transactions_control_1.query('uid != \"960936960.0\"')\n",
    "users_control_1        = users_control_1.query('uid != \"960936960\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Отбираем нужные страны. Составим список стран, которые есть во всех датасетах users_* и transactions_* (т.к. изменения коснулись премиум пользователей). Будем считать, что именно в этих странах производился эксперимент."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "country_ls = list(transactions_test \\\n",
    "                       .query('country in @transactions_control_1.country.unique() and country in @transactions_control_2.country.unique() and country in @users_control_1.country.unique() and country in @users_control_2.country.unique() and country in @users_test.country.unique()') \\\n",
    "                       .country.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Оставляем записи пользователей только из этих стран\n",
    "users_test      = users_test.query('country in @country_ls')\n",
    "users_control_1 = users_control_1.query('country in @country_ls')\n",
    "users_control_2 = users_control_2.query('country in @country_ls')\n",
    "\n",
    "transactions_test      = transactions_test.query('country in @country_ls')\n",
    "transactions_control_1 = transactions_control_1.query('country in @country_ls')\n",
    "transactions_control_2 = transactions_control_2.query('country in @country_ls')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Отбираем 2 нужные платежные системы. Ищем записи в датасетах transaction_*, где присутствует премиум подписка (premium_no_trial) (т.к. изменили именно стоимость премиум подписки)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "147    61\n",
       "68     12\n",
       "0       1\n",
       "Name: payment_id, dtype: int64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_test.query('product_type == \"premium_no_trial\"').payment_id.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "147    85\n",
       "68     16\n",
       "0       2\n",
       "Name: payment_id, dtype: int64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_control_1.query('product_type == \"premium_no_trial\"').payment_id.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "147    88\n",
       "68     11\n",
       "0       1\n",
       "Name: payment_id, dtype: int64"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_control_2.query('product_type == \"premium_no_trial\"').payment_id.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Платежная система 0 имеет всего 4 записи суммарно, поэтому мы можем исключить эту систему, т.к. по условию у нас их должно быть всего 2 - это системы 147 и 68.\n",
    "\n",
    "Отбираем транзакции с этими платежными системами:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions_test      = transactions_test.query('payment_id == 147 or payment_id == 68')\n",
    "transactions_control_1 = transactions_control_1.query('payment_id == 147 or payment_id == 68')\n",
    "transactions_control_2 = transactions_control_2.query('payment_id == 147 or payment_id == 68')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(150, 8)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_control_2.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Среди юзеров также отберем только этих пользователей, чтобы соблюсти все условия эксперимента, т.к. изменения коснулись только платящих пользователей."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_test      = users_test.query('uid in @transactions_test.uid.unique()')\n",
    "users_control_1 = users_control_1.query('uid in @transactions_control_1.uid.unique()')\n",
    "users_control_2 = users_control_2.query('uid in @transactions_control_2.uid.unique()')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Посмотрим на соотношение оставшихся пользователей"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.061784897025174"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Процент тестовой группы\n",
    "users_test.shape[0]/(users_control_1.shape[0] + users_control_2.shape[0] + users_test.shape[0])*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "37.52860411899314"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Процент 1 контрольной группы\n",
    "users_control_1.shape[0]/(users_control_1.shape[0] + users_control_2.shape[0] + users_test.shape[0])*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "33.40961098398169"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Процент 2 контрольной группы\n",
    "users_control_2.shape[0]/(users_control_1.shape[0] + users_control_2.shape[0] + users_test.shape[0])*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для убеждения в равном разделении пользователей между группами добавим статистический анализ.\n",
    "\n",
    "Используем критерий хи-квадрат с ожидаемыми и наблюдаемыми частотами:\n",
    "\n",
    "- $H_0$: ожидаемые и наблюдаемые частоты данных статистически значимо не различаются в группах (p-value > 0.05)\n",
    "- $H_1$: ожидаемые и наблюдаемые частоты данных статистически значимо различаются в группах"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Power_divergenceResult(statistic=4.700228832951947, pvalue=0.09535825103631365)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs = [users_test.shape[0], users_control_1.shape[0], users_control_2.shape[0]]\n",
    "obs_sum = sum(obs)\n",
    "exp = [obs_sum/3, obs_sum/3,  obs_sum/3]\n",
    "\n",
    "ss.chisquare(obs, f_exp=exp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "p-value > 0.05, нулевую гипотезу не отклоняем, т.е. размеры итоговых выборок в контрольных и тестовой группы статистически значимо не различаются."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(127, 13)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Итоговые размеры датасетов\n",
    "users_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(164, 13)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_control_1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(146, 13)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_control_2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(137, 8)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(169, 8)"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_control_1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(150, 8)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_control_2.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Можно заметить, что даже после фильтрации на итоговых выборках такого размера нормальность распределений не играет важной роли."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Таким образом, были отобраны необходимые новые пользователи (по одинаковой минимальной дате) из некоторых стран (общие страны из всех датасетов), которые воспользовались двумя платежными системами (147 и 68 - общие системы из всех групп с оплатой премиум подписки)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2: Общая оценка данных\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Исследуем наши данные по некоторым имеющимся признакам, чтобы удостовериться в репрезентативности выборок для последующего проведения А/А/В тестов, а также сделать некоторые предварительные выводы."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Создадим в датасетах колонки с названием группы\n",
    "users_test['group']      = 'test'\n",
    "users_control_1['group'] = 'control_1'\n",
    "users_control_2['group'] = 'control_2'\n",
    "\n",
    "transactions_test['group']      = 'test'\n",
    "transactions_control_1['group'] = 'control_1'\n",
    "transactions_control_1['group'] = 'control_2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Объединим датасеты для удобства\n",
    "users_merged        = pd.concat([users_test, users_control_1, users_control_2])\n",
    "transactions_merged = pd.concat([transactions_test, transactions_control_1, transactions_control_2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAEYCAYAAABBWFftAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3df1SUdf7//8cAkiaLBjqQRPteTMujaZaKk2YHFDRFQZTSypLYw5Y/SMkM9FO7WbrtZma5bcXa9ibXUnNzsPKdCpayamr+yNq3eo72Y9X48Rb5IZr8GOb7h8f5ij8HmpmLGe63czxnuOaa63qM1/A8POe6rtfLZLfb7QIAAAAA/GJ+RgcAAAAAAF9BgwUAAAAALkKDBQAAAAAuQoMFAAAAAC5CgwUAAAAALkKDBQAAAAAuQoMFAGi1duzYoSFDhhgdA4CPosa0TjRYuETfvn0d/2677Tb17t3b8fPatWubvL1Jkybpww8/dENSAEb6xz/+oeTkZPXq1UtZWVmGZLj11lv1448/GrJvAO5TW1urOXPmKCYmRn379lViYqI2b97s8RzUGDRHgNEB0PLs3bvX8Tg2NlYvvvii7r77bgMTNVZfX6+AAD66gNHMZrOmTJmiwsJC1dTUGB3nsjxZL6hNgOvU19frxhtv1LJly9SlSxdt3rxZM2bM0Mcff6ybbrrJ6HgO1BhcDmew4LSGhgbl5ORo2LBhio6O1pNPPqmKigpJUk1NjWbNmqXo6Gj169dP48aN04kTJ/Tqq6/qq6++0rx589S3b1/Nmzfvstu2Wq2KiYlRdHS03njjDcXGxmrbtm2SpCVLligjI0OzZs3SnXfeqTVr1qikpESPP/64BgwYoLi4OK1atcqxraysLL366quOny8+PR8bG6u3335bI0eOVP/+/ZWdnd1i/zgEWrL4+HgNGzZMHTt2dGr9VatW6b777lPfvn01cuRI/fvf/5YkHTlyRJMmTVK/fv00atQoFRQUOF6TlZWl559/Xunp6erbt69SUlL0n//8R5L00EMPSZISExPVt29frVu3zvH7npOTo0GDBik7O1u1tbWaP3++Bg8erMGDB2v+/Pmqra11KvO//vUvDR8+XHfddZf+8Ic/6OGHH3ackf/oo480YcIELViwQNHR0VqyZIlOnTql2bNna+DAgYqJidFf//pXNTQ0SDpXy2bNmuXY9rFjx3Trrbeqvr5e0rmz/a+88orGjx+vO++8U0888YSjxgKtzfXXX6/p06frpptukp+fn2JiYnTTTTc56sblUGOoMS0FDRactmzZMuXn5+sf//iHCgsL1aFDB0fDtGbNGlVXV+uLL77Qjh079Pzzz6tt27aaOXOm+vXrp+eee0579+7Vc889d8l2Dx8+rOeff14vv/yyCgsLVV1drZKSkkbrFBQUaMSIEfrqq680evRoZWZmKjw8XIWFhXr99de1aNEibd++3en38vHHH+udd97Rxo0b9f333+uvf/3rL/vPAXBV//M//6MlS5boT3/6k/bs2aM333xTHTt2VF1dnR5//HENGjRI27Zt0//7f/9Ps2bN0nfffed47bp16zRt2jTt2rVLN998s+MLlOXLl0uS8vLytHfvXo0cOVKSdOLECVVWVurzzz/XCy+8oDfffFNff/218vLytHbtWn3zzTdO/c6fPHlSGRkZeuqpp7Rjxw795je/aXSGX5L279+vyMhIbd26VU888YReeOEFnTp1Svn5+Vq2bJny8vL0z3/+0+n/J6vVqgULFuhf//qXAgIC9OKLLzr9WsCXnThxQj/88INuueWWyz5PjXEONcYzaLDgtBUrVmjmzJkKDw9XYGCgpk2bpvXr1ztOWVdUVOjHH3+Uv7+/evXqpaCgIKe2+9lnnykmJkb9+vVTYGCgMjIyZDKZGq1zxx13aNiwYfLz81N5ebn27NmjWbNm6brrrlOPHj2UkpKivLw8p9/LQw89pBtvvFEdO3bUE088oU8//bRJ/xcAmmb16tX67W9/q969e8tkMunXv/61IiIi9PXXX+vMmTNKT09XYGCgLBaLYmJiGv1ODhs2TL1791ZAQIDGjBmjAwcOXHVffn5+ysjIUGBgoNq2bauPP/5YU6dOVWhoqEJCQjR16lSn7ifdsmWLunXrpvj4eAUEBOiRRx5Rp06dGq1jNps1adIkBQQEqE2bNlq3bp2eeuopBQUF6aabblJqamqT7l1NTExU9+7ddf311+vJJ5/UZ599JpvN5vTrAV9UV1enWbNmaezYseratetl16HGOIca4xlcyAmn/fTTT5o6dar8/P7/vtzPz09lZWVKTExUcXGxMjMzVVVVpTFjxmjmzJlq06bNNbdbWlqq8PBwx8/t2rW75JKjC58vLS1Vhw4dGjVwXbp00bfffuv0e7nxxhsbvba0tNTp1wJouqKiIt18882XLD//+39hXenSpUujs9gX/sHRtm1bnTlz5qr7uuGGG3Tdddc12keXLl0abd+Z3/mLa5PJZGr0s9S4NpWXl6uuru6SfV18Rv5qLq5NdXV1Ki8vv+SPLqC1aGho0OzZs9WmTRs9++yzV1yPGuMcaoxncAYLTgsPD9ff/vY3ffXVV45/33zzjcLCwtSmTRtNmzZN69at04oVK/TFF1/IarU6tV2z2dyoOJw9e/aSa4IvPKNlNptVWVmp6upqx7KioiKFhYVJOtegnT171vHciRMnLtlnUVGR4/FPP/0ks9nsVFYAzXPjjTc67mu4kNlsVnFxseMeAqnx73NzXHwG3Gw266effmq0fWd+5zt37tyoNtntdhUXF19xXzfccIPatGlzyb6aW5uKiorUpk0b3XDDDdfMCvgiu92uuXPn6sSJE1qyZMlVv7SlxlBjWhIaLDht4sSJWrx4sY4fPy7p3LXD+fn5kqQvv/xShw4dks1mU1BQkAICAhzfFnXq1ElHjx694naHDx+uTZs2ac+ePaqtrdWSJUtkt9uvuP6NN96ovn37atGiRaqpqdHBgwe1evVqjRkzRpLUo0cPbd68WRUVFfq///s/5ebmXrKN999/X8XFxaqoqNBbb73luK4agPPq6+tVU1OjhoYG2Ww21dTUOG6mvtj48eP197//Xd9++63sdrt+/PFHHT9+XL1791bbtm21dOlS1dXVaceOHdq0aZPTv5PXqi+SNGrUKL355ps6efKkTp48qTfeeEOjR4++5rbvvfdeHTp0SPn5+aqvr9fy5csv+wfLef7+/hoxYoReffVVVVdX6/jx43r33Xcb1aZdu3bpp59+0qlTp/T2229fso21a9fq8OHD+vnnn/Xaa69p+PDh8vf3v2ZWwBf9/ve/15EjR/TWW2+pbdu2V12XGkONaUlosOC0Rx55RLGxsXrsscfUt29f3X///dq/f7+kc9+SZGRk6K677tLIkSM1YMAAJSYmOl63fv169e/f/7I3U3br1k3PPvusMjMzdc899+j6669XSEiIAgMDr5hl0aJFOn78uO655x5NmzZN06dPdwwln5iYqNtuu82R9XJFNCEhQY899piGDRumm2++WU888YQr/ouAVuXNN99U7969lZOTo7Vr16p379568803L7vufffdp8cff1xPPfWU7rzzTk2dOlWVlZUKDAzUW2+9pS1btmjgwIF6/vnn9ec///mK91lcbNq0acrKylK/fv20bt26y64zZcoU9erVS2PGjNGYMWPUs2dPTZky5ZrbDgkJ0WuvvaaXX35Z0dHROnz4sHr16nXVb9GfffZZtWvXTsOGDdODDz6ohIQEjRs3TpI0aNAgjRw5UmPGjFFycrJiYmIueX1iYqKysrI0aNAg1dbWau7cuU79PwC+5vjx41q5cqUOHDigwYMHX3M+TmoMNaYlMdmvdqoAMMDp06fVv39/rV+/XpGRkS7ffkuc2wtAy9fQ0KAhQ4Zo4cKFGjhwoMu3P2nSJI0ZM0YpKSku3zaAlo8a4zs4g4UWYdOmTfr555915swZ/elPf1L37t1b1ESCAFqnwsJCVVVVqba2Vm+99Zakc6OaAoArUGN8E6MIokUoKCjQ7NmzZbfb1atXLy1atOiSm0gBwNP27dunWbNmqba2VrfccoveeOONa94LAgDOosb4Ji4RBAAAAAAX4QwWAK8VGxur9u3by8/PT/7+/vroo49UUVGhmTNn6vjx44qIiNDixYvVoUMHo6MCAIBWwivuwUpLSzM6AoAWKjc3V3l5efroo48kSTk5ObJYLNqwYYMsFotycnKuuQ1qDAB3ob4ArY9XNFjl5eVGRwDgJQoKCpSUlCRJSkpKcszVdjXUGADuQn0BWh+vaLAA4ErS0tKUnJyslStXSpLKyspkNpslSZ07d1ZZWZmR8QAAQCvDPVgAvNYHH3ygsLAwlZWVKTU1VVFRUY2eN5lMjEYJAAA8ijNYALxWWFiYJCk0NFRxcXHav3+/QkNDVVpaKkkqLS1VSEiIkREBAEArQ4MFwCudOXNG1dXVjsdbt25Vt27dFBsbK6vVKkmyWq0aOnSokTEBAEArwyWCALxSWVmZpk6dKkmy2WxKSEjQkCFDdPvtt2vGjBlavXq1unTposWLFxucFAAAtCY0WAC8UmRkpNauXXvJ8htuuEG5ubkGJAIAAOASQQAAAABwGRosAAAAAHARGiwAAAAAcBHuwfKw2voGBQY0ra9tzmsAeJ+zdTb5+zk/b5etwa62bfzdmAgAWiZP/G3E319oLhosDwsM8NOEnO1Nes2KdIub0gBoSfz9TJr0zg6n11+WFu3GNADQcjXn76mm4u8vNBdtOQAAAAC4CA0WAAAAALgIDRYAAAAAuAgNFgAAAAC4CA0WAAAAALgIDRYAAAAAuAgNFgAAAFymtr7B6AiAoZgHCwAAAC7DHFVo7TiDBQAAAAAuQoMFAAAAAC7CJYIAAAAXKCoq0uzZs1VWViaTyaT7779fjz76qJYsWaJVq1YpJCREkpSZmal7773X4LQAWhoaLAAAgAv4+/srKytLPXv2VHV1tcaNG6dBgwZJkiZPnqy0tDSDEwJoyWiwAAAALmA2m2U2myVJQUFBioqKUklJicGpAHgL7sECAAC4gmPHjunAgQPq06ePJGn58uUaPXq0srOzVVlZaXA6AC2R2xqsmpoajR8/XmPGjNGoUaP0+uuvS5KOHj2qlJQUxcXFacaMGaqtrXVXBAAAgGY7ffq0MjIyNGfOHAUFBWnixInauHGj8vLyZDab9dJLLxkdEUAL5LYGKzAwULm5uVq7dq2sVqsKCwu1b98+LVy4UJMnT9bGjRsVHBys1atXuysCAABAs9TV1SkjI0OjR49WfHy8JKlTp07y9/eXn5+fUlJS9M033xicEkBL5LYGy2QyqX379pKk+vp61dfXy2Qy6csvv9Tw4cMlSWPHjlVBQYG7IgAAADSZ3W7X3LlzFRUVpdTUVMfy0tJSx+P8/Hx169bNiHgAWji3DnJhs9mUnJys//znP3rwwQcVGRmp4OBgBQSc2214eDg3jQIAgBZl9+7dysvLU/fu3ZWYmCjp3JDsn3zyiQ4ePChJioiI0Lx584yMCaCFcmuD5e/vr7y8PFVVVWnq1Kn67rvv3Lk7AACAX6xfv346dOjQJcuZ8wqAMzwyimBwcLCio6O1b98+VVVVqb6+XpJUXFyssLAwT0QAAAAAALdzW4N18uRJVVVVSZLOnj2rbdu2qWvXroqOjtb69eslSWvWrFFsbKy7IgAAAACAR7ntEsHS0lJlZWXJZrPJbrdrxIgRiomJ0S233KKZM2dq8eLF6tGjh1JSUtwVAQAAABeorW9QYADToALu5LYG67bbbpPVar1keWRkJEOzAwAAGCAwwE8Tcra7dR8r0i1u3T7Q0vEVBgAAAAC4CA0WAAAAALgIDRYAAAAAuAgNFgAAAAC4CA0WAAAAALgIDRYAAAAAuAgNFgCvZrPZlJSUpN/97neSpKNHjyolJUVxcXGaMWOGamtrDU4IAABaExosAF7tvffeU9euXR0/L1y4UJMnT9bGjRsVHBzMvHsAAMCjaLAAeK3i4mJ98cUXGj9+vCTJbrfryy+/1PDhwyVJY8eOVUFBgZERAQBAK0ODBcBrLViwQE8//bT8/M6VsvLycgUHBysgIECSFB4erpKSEiMjAgCAVoYGC4BX+vzzzxUSEqJevXoZHQUAAMAhwOgAANAce/bs0aZNm7RlyxbV1NSourpa8+fPV1VVlerr6xUQEKDi4mKFhYUZHRUAALQinMEC4JWeeuopbdmyRZs2bdKiRYs0cOBAvfLKK4qOjtb69eslSWvWrFFsbKzBSQEAQGtCgwXApzz99NN69913FRcXp4qKCqWkpBgdCQAAtCJcIgjA60VHRys6OlqSFBkZydDsAADAMJzBAgAAAAAXocECAAAAABehwQIAAAAAF6HBAgAAAAAXocECAAAAABehwQIAAAAAF6HBAgAAAAAXcds8WEVFRZo9e7bKyspkMpl0//3369FHH9WSJUu0atUqhYSESJIyMzN17733uisGAAAAAHiM2xosf39/ZWVlqWfPnqqurta4ceM0aNAgSdLkyZOVlpbmrl0DAAAAgCHc1mCZzWaZzWZJUlBQkKKiolRSUuKu3QEAAACA4TxyD9axY8d04MAB9enTR5K0fPlyjR49WtnZ2aqsrPREBAAAAABwO7c3WKdPn1ZGRobmzJmjoKAgTZw4URs3blReXp7MZrNeeukld0cAAABwWlFRkSZNmqSRI0dq1KhRys3NlSRVVFQoNTVV8fHxSk1N5UtiAJfl1garrq5OGRkZGj16tOLj4yVJnTp1kr+/v/z8/JSSkqJvvvnGnREAAACa5Px95OvWrdPKlSv1/vvv6/Dhw8rJyZHFYtGGDRtksViUk5NjdFQALZDbGiy73a65c+cqKipKqampjuWlpaWOx/n5+erWrZu7IgAAADSZ2WxWz549JTW+j7ygoEBJSUmSpKSkJOXn5xsZE0AL5bZBLnbv3q28vDx1795diYmJks4Nyf7JJ5/o4MGDkqSIiAjNmzfPXREAAAB+kQvvIy8rK3MM4NW5c2eVlZUZnA5AS+S2Bqtfv346dOjQJcuZ8woAAHiDi+8jv5DJZJLJZDIoGYCWzCOjCAIAAHiTy91HHhoa6rjVobS0VCEhIUZGBNBC0WABAABc4Er3kcfGxspqtUqSrFarhg4dalREAC2Y2y4RBAAA8EZXuo88PT1dM2bM0OrVq9WlSxctXrzY4KQAWiIaLAAAgAtc6T5ySY45sQDgSrhEEAAAAABchAYLAAAAAFyEBgsAAAAAXIQGCwAAAABchAYLAAAAAFyEBgsAAAAAXIQGCwAAAABchAYLAAAAAFyEBgsAAAAAXIQGS1JtfYNHXwcAAADANwUYHaAlCAzw04Sc7U1+3Yp0ixvSAAAAAPBWnMECAAAAABehwQIAAAAAF6HBAgAAAAAXocECAAAAABdhkAsAXqmmpkYPPfSQamtrZbPZNHz4cGVkZOjo0aPKzMxURUWFevbsqT//+c8KDAw0Oi4AAGglOIMFwCsFBgYqNzdXa9euldVqVWFhofbt26eFCxdq8uTJ2rhxo4KDg7V69WqjowIAgFaEBguAVzKZTGrfvr0kqb6+XvX19TKZTPryyy81fPhwSdLYsWNVUFBgZEwAANDK0GAB8Fo2m02JiYm6++67dffddysyMlLBwcEKCDh39XN4eLhKSkoMTgkAAFoTtzVYRUVFmjRpkkaOHKlRo0YpNzdXklRRUaHU1FTFx8crNTVVlZWV7ooAwMf5+/srLy9Pmzdv1v79+/Xdd98ZHQkAALRybmuw/P39lZWVpXXr1mnlypV6//33dfjwYeXk5MhisWjDhg2yWCzKyclxVwQArURwcLCio6O1b98+VVVVqb6+XpJUXFyssLAwg9MBAIDWxG0NltlsVs+ePSVJQUFBioqKUklJiQoKCpSUlCRJSkpKUn5+vrsiAPBhJ0+eVFVVlSTp7Nmz2rZtm7p27aro6GitX79ekrRmzRrFxsYaGRMAALQyHhmm/dixYzpw4ID69OmjsrIymc1mSVLnzp1VVlbmiQgAfExpaamysrJks9lkt9s1YsQIxcTE6JZbbtHMmTO1ePFi9ejRQykpKUZHBQAArYjbG6zTp08rIyNDc+bMUVBQUKPnTCaTTCaTuyMA8EG33XabrFbrJcsjIyMZmh0AABjGraMI1tXVKSMjQ6NHj1Z8fLwkKTQ0VKWlpZLOfQMdEhLizggAAAAA4DFua7Dsdrvmzp2rqKgopaamOpbHxsY6vnW2Wq0aOnSouyIAAAB4jdr6BqMjAHABt10iuHv3buXl5al79+5KTEyUJGVmZio9PV0zZszQ6tWr1aVLFy1evNhdEQAAALxGYICfJuRsd+s+VqRb3Lp9AE42WLt379Zdd911zWUX6tevnw4dOnTZ587PiQUAAAAAvsSpSwRffPFFp5YBAAAAQGt21TNYe/fu1d69e3Xy5Em9++67juXV1dWy2WxuDwcAAAAA3uSqZ7Dq6up05swZ2Ww2nT592vEvKChIr7/+uqcyAgAAeFR2drYsFosSEhIcy5YsWaJ77rlHiYmJSkxM1ObNmw1MCKCluuoZrAEDBmjAgAEaO3asIiIiPJUJAADAUMnJyXr44Yf1zDPPNFo+efJkpaWlGZQKgDdwapCL2tpaPfvsszp+/Ljq6+sdy9977z23BQMAADBK//79dezYMaNjAPBCTjVYTz75pCZMmKCUlBT5+bl1bmIAAIAWa/ny5bJarerVq5eysrLUoUMHoyMBaGGcarACAgL04IMPujsLAABAizVx4kRNmTJFJpNJr732ml566SX98Y9/NDoWgBbGqdNRMTExWr58uUpLS1VRUeH4BwAA0Fp06tRJ/v7+8vPzU0pKir755hujIwFogZw6g7VmzRpJ0jvvvONYZjKZVFBQ4J5UAAAALUxpaanMZrMkKT8/X926dTM4EYCWyKkGa9OmTe7OAQAA0GJkZmZq586dKi8v15AhQzR9+nTt3LlTBw8elCRFRERo3rx5BqcE0BI51WBZrdbLLk9KSnJpGAAAgJZg0aJFlyxLSUkxIAkAb+NUg3XhNcY1NTXavn27evbsSYMFAAAAABdwqsF69tlnG/1cVVWlmTNnuiUQAPiSs3U2+fuZjI4BAGii2voGBQa4d3oi9uGbnGqwLtauXbsWO/meJw+wp/bVnP3wQQdaBn8/kya9s8OpdZelRbs5DQDAWYEBfpqQs92t+1iRbnHr9iXfeR/exKkG6/HHH3c8bmho0JEjR3Tfffe5LdQv0ZwPUXM/FJ7alyffEwAAAIDmc6rBeuyxxxyP/f39FRERofDwcLeFAgAAAABv5FSDNWDAAJ04ccIx2MV//dd/uTMTAMAJtga7pIYmrd+2jb/7AgEAAOcarHXr1unll1/WgAEDZLfb9cILL2j27NkaMWKEu/MBAK6gKfd3SdzjBQCAJzjVYL311ltavXq1QkNDJUknT57U5MmTabAAAAAA4AJODTNnt9sdzZUkdezYUXa73W2hAAAAAMAbOXUGa/DgwUpLS9OoUaMknbtkcMiQIW4NBgAAAPgyptXxTVdtsH788UedOHFCzzzzjDZs2KDdu3dLku644w6NGTPGIwEBAAAAX8QcVb7pqi3zggULFBQUJEmKj49Xdna2srOzFRcXpwULFngkIAAAAAB4i6s2WCdOnNCtt956yfJbb71Vx48fv+qGs7OzZbFYlJCQ4Fi2ZMkS3XPPPUpMTFRiYqI2b97czNgAAAAA0PJc9RLBU6dOXfG5s2fPXnXDycnJevjhh/XMM880Wj558mSlpaU1ISIAAAAAeIernsHq1auXVq1adcnyDz/8UD179rzqhvv3768OHTr8snQAAAAA4EWuegZrzpw5mjZtmj7++GNHQ/Xtt9+qrq5Of/nLX5q1w+XLl8tqtapXr17KysqiCQMAAADgM656BqtTp05asWKFpk6dqoiICEVERGjq1KlauXKlOnfu3OSdTZw4URs3blReXp7MZrNeeumlZgcHAAAAgJbGqXmwBg4cqIEDB/7inXXq1MnxOCUlRY8//vgv3iYAAAAAtBQendmstLTU8Tg/P1/dunXz5O4BAAAAwK2cOoPVHJmZmdq5c6fKy8s1ZMgQTZ8+XTt37tTBgwclSREREZo3b567dg8AAAAAHue2BmvRokWXLEtJSXHX7gC0MkVFRZo9e7bKyspkMpl0//3369FHH1VFRYVmzpyp48ePKyIiQosXL2YwHQAA4DEevUQQAFzF399fWVlZWrdunVauXKn3339fhw8fVk5OjiwWizZs2CCLxaKcnByjowIAgFaEBguAVzKbzY7pI4KCghQVFaWSkhIVFBQoKSlJkpSUlKT8/HwjYwIAgFaGBguA1zt27JgOHDigPn36qKysTGazWZLUuXNnlZWVGZwOAAC0JjRYALza6dOnlZGRoTlz5igoKKjRcyaTSSaTyaBkAACgNaLBAuC16urqlJGRodGjRys+Pl6SFBoa6pgSorS0VCEhIUZGBAAArQwNFgCvZLfbNXfuXEVFRSk1NdWxPDY2VlarVZJktVo1dOhQoyICAIBWyG3DtAOAO+3evVt5eXnq3r27EhMTJZ2bfy89PV0zZszQ6tWr1aVLFy1evNjgpAAAoDWhwQLglfr166dDhw5d9rnc3FwPpwEAADiHSwQBAAAukp2dLYvFooSEBMeyiooKpaamKj4+XqmpqaqsrDQwIYCWigYLAADgIsnJyVq6dGmjZUxkDsAZNFgAAAAX6d+/vzp06NBoGROZA3AGDRYAAIATmMgcgDNosAAAAJqIicwBXAkNFgAAgBOYyByAM2iwAAAAnMBE5gCcQYMFAABwkczMTE2YMEHff/+9hgwZog8//FDp6enaunWr4uPjtW3bNqWnpxsdE0ALxETDAAAAF1m0aNFllzOROYBr4QwWAAAAALgIDRYAAAAAuAgNFgAAAAC4CA0WAAAAALgIDRYAAAAAuIjbGqzs7GxZLBYlJCQ4llVUVCg1NVXx8fFKTU1VZWWlu3YPAAAAAB7ntgYrOTlZS5cubbQsJydHFotFGzZskMViUU5OjjJRvO4AAA+fSURBVLt2DwAAAAAe57YGq3///urQoUOjZQUFBUpKSpIkJSUlKT8/3127BwC3OFtnU52twel/AACgdfHoRMNlZWUym82SpM6dO6usrMyTuweAX8zfz6RJ7+xwev1ladFuTAMAAFoawwa5MJlMMplMRu0eAAAAAFzOow1WaGioSktLJUmlpaUKCQnx5O4BAAAAwK082mDFxsbKarVKkqxWq4YOHerJ3QMAmqCp95udrbMZHRkAAMO57R6szMxM7dy5U+Xl5RoyZIimT5+u9PR0zZgxQ6tXr1aXLl20ePFid+0eAPALcb8ZAABN57YGa9GiRZddnpub665dAgAAAIChDBvkAgAAAAB8DQ2Wj6qtb978O819HQAAAAAPz4MFzwkM8NOEnO1Nft2KdIsb0gAAAACtA2ewAAAAAMBFaLAAAAAANJsnbjHxpttYuEQQAAAAQLM199aUpvCm21hosACglbA12CV5zzeAAAB4IxosAGglmDgYAAD34x4sAAAAAHARGiwAAAAAcBEaLAAAAABwERosAF4pOztbFotFCQkJjmUVFRVKTU1VfHy8UlNTVVlZaWBCAADQGtFgAfBKycnJWrp0aaNlOTk5slgs2rBhgywWi3JycgxKBwAAWisaLABeqX///urQoUOjZQUFBUpKSpIkJSUlKT8/34hoAACgFaPBAuAzysrKZDabJUmdO3dWWVmZwYkAAEBrQ4MFwCeZTCaZTCajYwAAgFaGiYYB+IzQ0FCVlpbKbDartLRUISEhRkcC4INiY2PVvn17+fn5yd/fXx999JHRkQC0IDRYAHxGbGysrFar0tPTZbVaNXToUKMjAfBRubm5fIkD4LK4RBCAV8rMzNSECRP0/fffa8iQIfrwww+Vnp6urVu3Kj4+Xtu2bVN6errRMQEAQCvDGSwAXmnRokWXXZ6bm+vhJDjP1mCX1NCk9du28XdfIMCN0tLSZDKZ9MADD+iBBx4wOg7g82rrGxQY4N5zQ67aBw0WAMAl/P1MmvTODqfXX5YW7cY0gPt88MEHCgsLU1lZmVJTUxUVFaX+/fsbHQvwaYEBfpqQs92t+1iRbnHJdrhEEAAAoAnCwsIknRtYJy4uTvv37zc4EYCWhAYLAADASWfOnFF1dbXj8datW9WtWzeDUwFoSQy5RJDhTQEAgDcqKyvT1KlTJUk2m00JCQkaMmSIwakAtCSG3YPF8KYAAMDbREZGau3atUbHANCCcYkgAAAAALiIYQ1WWlqakpOTtXLlSqMi4DJq650fYtnTrwEAAABaOkMuEWR405arOUNgrki3NOs1AFo35s0CAPgiQxqsyw1vSoMFAK0L82YBAHyRxy8RZHhTAAAAAL7K42ewGN4UAAAAgK/yeIPF8KYAAAAAfJVh82ABANAUTRkUgwExAABGocECAHiFpgyKwYAYAACjMNEwAAAAALgIDRYAAAAAuAgNFgAAAAC4CA0WAAAAALgIDRYAAAAAuAgNFgAAAAC4CA0WAAAAALgIDRYAAAAAuAgNFgAAAAC4CA0WAAAAALgIDRYAAAAAuEiA0QEAAHA1W4NdUkOT1m/bxt99gQAArQYNFgDA5/j7mTTpnR1Or78sLdqNaQAArQmXCAIAAACAi9BgAQAAAICL0GABAAAAgIvQYAEAAACAi9BgwRC19c6P7vVLXoNfhuMEAADQNIwiCEMEBvhpQs72Jr1mRbrFTWlwJRwnAACApqHBAgAAhjlbZ5O/n8np9ZmzDEBLR4MFAAAMw5xlAHyNIfdgbdmyRcOHD1dcXJxycnKMiADAh1FjALgTNQbA1Xi8wbLZbJo3b56WLl2qTz/9VJ988okOHz7s6RgAfBQ1BoA7UWMAXIvHG6z9+/fr17/+tSIjIxUYGKhRo0apoKDA0zEA+ChqDAB3osYAuBaT3W63e3KHn332mQoLCzV//nxJktVq1f79+/Xcc89d8TXR0dGKiIjwVEQAHnLDDTfonXfecek2qTEAzmsJNYb6AviuK9UYrxjkYscO529+BYCmosYAcBfqC9D6ePwSwbCwMBUXFzt+LikpUVhYmKdjAPBR1BgA7kSNAXAtHm+wbr/9dv3www86evSoamtr9emnnyo2NtbTMQD4KGoMAHeixgC4Fo9fIhgQEKDnnntOv/3tb2Wz2TRu3Dh169bN0zEA+ChqDAB3osYAuBaPD3IBAAAAAL7KkImGAQAAAMAX0WABAAAAgIv4dINVVFSkSZMmaeTIkRo1apRyc3MlSRUVFUpNTVV8fLxSU1NVWVlpcNLmsdlsSkpK0u9+9ztJ0tGjR5WSkqK4uDjNmDFDtbW1BidsvqqqKmVkZGjEiBG67777tHfvXp84bv/93/+tUaNGKSEhQZmZmaqpqfHq45adnS2LxaKEhATHsisdJ7vdrhdffFFxcXEaPXq0/v3vfxsV2+WoNd7zmT3PV2uM5Dt1hvrSfL5ek86rqanR+PHjNWbMGI0aNUqvv/66JN+oURfzxTp8sdjYWI0ePVqJiYlKTk6W5L2fWZ9usPz9/ZWVlaV169Zp5cqVev/993X48GHl5OTIYrFow4YNslgsysnJMTpqs7z33nvq2rWr4+eFCxdq8uTJ2rhxo4KDg7V69WoD0/0y8+fP1z333KPPPvtMeXl56tq1q9cft5KSEr333nv65z//qU8++UQ2m02ffvqpVx+35ORkLV26tNGyKx2nLVu26IcfftCGDRv0wgsv6A9/+IMBid2DWuM9n9nzfLHGSL5VZ6gvzefrNem8wMBA5ebmau3atbJarSosLNS+ffu88vN+Lb5Yhy8nNzdXeXl5+uijjyRd+Xe+pfPpBstsNqtnz56SpKCgIEVFRamkpEQFBQVKSkqSJCUlJSk/P9/ImM1SXFysL774QuPHj5d07tu7L7/8UsOHD5ckjR07VgUFBUZGbLZTp05p165djvcWGBio4OBgnzhuNptNZ8+eVX19vc6ePavOnTt79XHr37+/OnTo0GjZlY7T+eUmk0l33HGHqqqqVFpa6vHM7kCt8Z7PrOTbNUbynTpDfWk+X65JFzKZTGrfvr0kqb6+XvX19TKZTF75eb8aX6zDzvLWz6xPN1gXOnbsmA4cOKA+ffqorKxMZrNZktS5c2eVlZUZnK7pFixYoKefflp+fucOYXl5uYKDgxUQcG7k/fDwcJWUlBgZsdmOHTumkJAQZWdnKykpSXPnztWZM2e8/riFhYXpscceU0xMjAYPHqygoCD17NnTZ47beVc6TiUlJQoPD3es5wvv9XKoNS2fr9YYyffrTGuvL83hazXpYjabTYmJibr77rt19913KzIy0mc+7+f5Yh2+krS0NCUnJ2vlypWSrvw739K1igbr9OnTysjI0Jw5cxQUFNToOZPJJJPJZFCy5vn8888VEhKiXr16GR3FLerr6/W///u/mjhxoqxWq9q1a3fJKWFvPG6VlZUqKChQQUGBCgsL9fPPP6uwsNDoWG7ljcfpl6DWeAdfrTFS66oz3nqMPMnXatLl+Pv7Ky8vT5s3b9b+/fv13XffGR3JpXy1Dl/OBx98oDVr1uhvf/ubli9frl27djV63ps+sx6faNjT6urqlJGRodGjRys+Pl6SFBoaqtLSUpnNZpWWliokJMTglE2zZ88ebdq0SVu2bFFNTY2qq6s1f/58VVVVqb6+XgEBASouLlZYWJjRUZslPDxc4eHh6tOnjyRpxIgRysnJ8frjtm3bNt10002O3PHx8dqzZ4/PHLfzrnScwsLCVFxc7FjPF97rhag13sNXa4zk+3WmtdaX5vDFmnQ1wcHBio6O1r59+3zm8y75bh2+nPPvITQ0VHFxcdq/f7/XfmZ9+gyW3W7X3LlzFRUVpdTUVMfy2NhYWa1WSZLVatXQoUONitgsTz31lLZs2aJNmzZp0aJFGjhwoF555RVFR0dr/fr1kqQ1a9YoNjbW4KTN07lzZ4WHhzu+hdq+fbu6du3q9cetS5cu+vrrr/Xzzz/Lbrdr+/btuuWWW3zmuJ13peN0frndbte+ffv0q1/9ynHa39tRa7zrM+urNUby/TrTGutLc/hqTbrYyZMnVVVVJUk6e/astm3bpq5du/rM513y3Tp8sTNnzqi6utrxeOvWrerWrZvXfmZNdrvdbnQId/nqq6/00EMPqXv37o7rVjMzM9W7d2/NmDFDRUVF6tKlixYvXqyOHTsanLZ5duzYob///e96++23dfToUc2cOVOVlZXq0aOHFi5cqMDAQKMjNsuBAwc0d+5c1dXVKTIyUn/84x/V0NDg9cft9ddf17p16xQQEKAePXpo/vz5Kikp8drjlpmZqZ07d6q8vFyhoaGaPn26hg0bdtnjZLfbNW/ePBUWFqpdu3ZasGCBbr/9dqPfgktQa7znM3uer9YYyXfqDPWl+VpDTZKkgwcPKisrSzabTXa7XSNGjNC0adN8okZdjq/V4QsdPXpUU6dOlXTuvrqEhAQ98cQTKi8v98rPrE83WAAAAADgST59iSAAAAAAeBINFgAAAAC4CA0WAAAAALgIDRYAAAAAuAgNFgAAAAC4CA0WAAAAALgIDRYAAAAAuEiA0QHQ+kyZMkXFxcWqqanRI488ogceeEAffvihli5dql/96le67bbbFBgYqOeee04nT57U73//e/3000+SpDlz5uiuu+4y+B0AaMmoMQDchfoCp9gBDysvL7fb7Xb7zz//bB81apS9uLjYHhMTYy8vL7fX1tbaJ06caH/++eftdrvdnpmZad+1a5fdbrfbjx8/bh8xYoRhuQF4B2oMAHehvsAZnMGCxy1btkwbN26UJBUVFSkvL0/9+/dXx44dJUkjRozQDz/8IEnatm2bDh8+7HhtdXW1Tp8+rfbt23s8NwDvQI0B4C7UFziDBgsetWPHDm3btk0rV65Uu3btNGnSJEVFRenIkSOXXb+hoUGrVq3Sdddd5+GkALwRNQaAu1Bf4CwGuYBHnTp1Sh06dFC7du105MgR7du3T2fOnNGuXbtUWVmp+vp6bdiwwbH+4MGDtWzZMsfPBw4cMCI2AC9BjQHgLtQXOMtkt9vtRodA61FbW6spU6bo+PHj+s1vfqNTp05p2rRp+uGHH/TOO++oQ4cOioqKUnh4uGbOnKmTJ09q3rx5OnLkiGw2m/r166d58+YZ/TYAtFDUGADuQn2Bs2iw0CKcvya5vr5e06ZN07hx4xQXF2d0LAA+ghoDwF2oL7gY92ChRfjLX/6ibdu2qaamRoMHD9awYcOMjgTAh1BjALgL9QUX4wwWAAAAALgIg1wAAAAAgIvQYAEAAACAi9BgAQAAAICL0GABAAAAgIvQYAEAAACAi/x/DJrG4aNUi4sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x288 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Распределение пользователей в зависимости от возраста\n",
    "fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))\n",
    "\n",
    "sns.histplot(x=users_test.age, ax = axes[0])\n",
    "axes[0].set(title='Test group')\n",
    "\n",
    "sns.histplot(x=users_control_1.age, ax = axes[1])\n",
    "axes[1].set(title='1 control group', ylabel='')\n",
    "\n",
    "sns.histplot(x=users_control_2.age, ax = axes[2])\n",
    "axes[2].set(title='2 control group', ylabel='')\n",
    "\n",
    "sns.despine()\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32.83464566929134"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Посмотрим на средние значения возраста по группам\n",
    "users_test.age.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "31.79268292682927"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_control_1.age.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "31.472602739726028"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_control_2.age.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверим группы на гомоскедастичность:\n",
    "\n",
    "- $H_0$: дисперсии равны, equal_var = True\n",
    "- $H_1$: дисперсии различны"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>W</th>\n",
       "      <th>pval</th>\n",
       "      <th>equal_var</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>levene</th>\n",
       "      <td>0.107369</td>\n",
       "      <td>0.898218</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               W      pval  equal_var\n",
       "levene  0.107369  0.898218       True"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.homoscedasticity(data = users_merged, dv = 'age', group = 'group')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Тест на гомоскедастичность показал, что дисперсии в группах статистически не различны"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Т.к. у нас 3 группы с количественными переменными, сравним их распределения при помощи критерия Краскела-Уоллиса:\n",
    "\n",
    "- $H_0$: нет статистически значимых различий между распределениями возраста в тестовой и контрольных группах (p-value > 0.05)\n",
    "- $H_1$: есть статистически значимые различия между распределениями возраста в тестовой и контрольных группах"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KruskalResult(statistic=1.8616835221800525, pvalue=0.39422173015390216)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kruskal(users_merged.query('group == \"test\"').age, users_merged.query('group == \"control_1\"').age, users_merged.query('group == \"control_2\"').age)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Основная возрастная категория - молодежь (25-35 лет). Средний возраст пользователей - 32 года.\n",
    "\n",
    "Согласно статистике (p-value > 0.05) нулевую гипотезу об отсутствии различий распределений возраста в тестовой и контрольных группах мы отклонить также не можем. Мы можем считать возрастное распределение в группах одинаковым, что говорит о репрезентативности выборок по этому признаку."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1YAAAEYCAYAAABfkWdeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de1hVdb7H8c8GZLLICyp4rSOF1WheGm94oQEVL1zVMK0oyXk4ZYpJZqDHZnS81ZRpnso4No2ZM2qWYOZkgaOS10pLnbTp7iWBIwLiDdiwzh9O+0QqbtyufWG/X8/j88CC/VtfVvF9+Ozf+v2WxTAMQwAAAACAq+bj6gIAAAAAwNMRrAAAAADAQQQrAAAAAHAQwQoAAAAAHESwAgAAAAAHEawAAAAAwEEEKwCAV9q1a5fCw8NdXQaAeobe4r0IVqihW7dutn+33367OnfubPt83bp1dR4vKSlJb731lgmVAnClN998UyNGjFCnTp2Unp7ukhpuu+02/fDDDy45N4Brr6KiQtOmTVNERIS6deum+Ph4bdmyxel10FtwtfxcXQDcy969e20fR0ZGavbs2erTp48LK6rJarXKz4//bQFXCwoK0vjx45WXl6fy8nJXl3NJzuwX9CbAcVarVa1atdLy5cvVunVrbdmyRY8//rjeffddtW3b1tXl2dBbcDnMWMEu1dXVyszM1MCBA9WrVy9NmjRJJSUlkqTy8nJNmTJFvXr1Uvfu3TVy5EidOHFCL7zwgj755BPNmjVL3bp106xZsy45dlZWliIiItSrVy+99NJLioyM1Pbt2yVJixcvVmpqqqZMmaK77rpLa9euVUFBgR555BH17NlTgwYN0urVq21jpaen64UXXrB9/svp+MjISL366qsaNmyYevTooYyMDLf9oxBwZ1FRURo4cKCaNGli1/evXr1aQ4cOVbdu3TRs2DD985//lCR98803SkpKUvfu3RUdHa3c3Fzba9LT0zVz5kylpKSoW7duSkxM1OHDhyVJ999/vyQpPj5e3bp104YNG2y/75mZmerbt68yMjJUUVGhOXPmqF+/furXr5/mzJmjiooKu2r+6KOPNHjwYP3mN7/RH/7wBz3wwAO2Gfh33nlHo0eP1ty5c9WrVy8tXrxYZWVlmjp1qnr37q2IiAi9/PLLqq6ulnShl02ZMsU29tGjR3XbbbfJarVKujC7//zzz+uee+7RXXfdpUcffdTWYwFvcf3112vixIlq27atfHx8FBERobZt29r6xaXQW+gt7oRgBbssX75cOTk5evPNN5WXl6fGjRvbgtLatWt1+vRpbd68Wbt27dLMmTN13XXXafLkyerevbuefvpp7d27V08//fRF43799deaOXOm/vSnPykvL0+nT59WQUFBje/Jzc3VkCFD9Mknnyg2NlZpaWlq2bKl8vLy9OKLL2rBggXasWOH3T/Lu+++q9dee00ffvihvvvuO7388suOXRwAtfr73/+uxYsX65lnntGePXv0yiuvqEmTJqqsrNQjjzyivn37avv27fqv//ovTZkyRd9++63ttRs2bNCECRP08ccf66abbrK9cbJixQpJUnZ2tvbu3athw4ZJkk6cOKHS0lL94x//0B//+Ee98sor+vzzz5Wdna1169Zp//79dv3Onzx5UqmpqXriiSe0a9cutW/fvsaMviTt27dP7dq107Zt2/Too4/qj3/8o8rKypSTk6Ply5crOztbb7/9tt3XKSsrS3PnztVHH30kPz8/zZ492+7XAvXRiRMn9P333+vWW2+95NfpLfahtzgPwQp2WblypSZPnqyWLVvK399fEyZM0MaNG21T1CUlJfrhhx/k6+urTp06KSAgwK5x33//fUVERKh79+7y9/dXamqqLBZLje/p2rWrBg4cKB8fHxUXF2vPnj2aMmWKfvWrX+mOO+5QYmKisrOz7f5Z7r//frVq1UpNmjTRo48+qvfee69O1wJA3axZs0a/+93v1LlzZ1ksFt18881q06aNPv/8c509e1YpKSny9/dXWFiYIiIiavxODhw4UJ07d5afn5/i4uJ08ODBWs/l4+Oj1NRU+fv767rrrtO7776rxx57TM2aNVNgYKAee+wxu9aLbt26VaGhoYqKipKfn58efPBBNW/evMb3BAUFKSkpSX5+fmrQoIE2bNigJ554QgEBAWrbtq2Sk5PrtDY1Pj5eHTp00PXXX69Jkybp/fffV1VVld2vB+qTyspKTZkyRcOHD9ctt9xyye+ht9iH3uI83LQJu/z444967LHH5OPz/1ncx8dHRUVFio+PV35+vtLS0nTq1CnFxcVp8uTJatCgwRXHLSwsVMuWLW2fN2zY8KJbi37+9cLCQjVu3LhGcGvdurUOHDhg98/SqlWrGq8tLCy0+7UA6u748eO66aabLjr+0+//z/tK69ata8xa//wPjuuuu05nz56t9VxNmzbVr371qxrnaN26dY3x7fmd/2VvslgsNT6Xavam4uJiVVZWXnSuX87A1+aXvamyslLFxcUX/dEF1HfV1dWaOnWqGjRooBkzZlz2++gt9qG3OA8zVrBLy5Yt9T//8z/65JNPbP/279+v4OBgNWjQQBMmTNCGDRu0cuVKbd68WVlZWXaNGxQUVKM5nD9//qJ7f38+gxUUFKTS0lKdPn3aduz48eMKDg6WdCGYnT9/3va1EydOXHTO48eP2z7+8ccfFRQUZFetAK5Oq1atbOsXfi4oKEj5+fm2tQJSzd/nq/HLGe+goCD9+OOPNca353e+RYsWNXqTYRjKz8+/7LmaNm2qBg0aXHSuq+1Nx48fV4MGDdS0adMr1grUJ4ZhaPr06Tpx4oQWL15c65u09BZ6i7shWMEuY8aM0cKFC3Xs2DFJF+4RzsnJkSTt3LlTX375paqqqhQQECA/Pz/bu0TNmzfXkSNHLjvu4MGDtWnTJu3Zs0cVFRVavHixDMO47Pe3atVK3bp104IFC1ReXq5Dhw5pzZo1iouLkyTdcccd2rJli0pKSvS///u/WrZs2UVj/PWvf1V+fr5KSkq0ZMkS2/3TAOxntVpVXl6u6upqVVVVqby83LZY+pfuuece/fnPf9aBAwdkGIZ++OEHHTt2TJ07d9Z1112npUuXqrKyUrt27dKmTZvs/p28Un+RpOjoaL3yyis6efKkTp48qZdeekmxsbFXHPvuu+/Wl19+qZycHFmtVq1YseKSf7D8xNfXV0OGDNELL7yg06dP69ixY3r99ddr9KaPP/5YP/74o8rKyvTqq69eNMa6dev09ddf69y5c1q0aJEGDx4sX1/fK9YK1Ce///3v9c0332jJkiW67rrrav1eegu9xd0QrGCXBx98UJGRkXr44YfVrVs3jRo1Svv27ZN04d2R1NRU/eY3v9GwYcPUs2dPxcfH2163ceNG9ejR45KLJUNDQzVjxgylpaWpf//+uv766xUYGCh/f//L1rJgwQIdO3ZM/fv314QJEzRx4kTblvDx8fG6/fbbbbVeqonGxMTo4Ycf1sCBA3XTTTfp0UcfvRaXCPAqr7zyijp37qzMzEytW7dOnTt31iuvvHLJ7x06dKgeeeQRPfHEE7rrrrv02GOPqbS0VP7+/lqyZIm2bt2q3r17a+bMmXr22Wcvu57ilyZMmKD09HR1795dGzZsuOT3jB8/Xp06dVJcXJzi4uLUsWNHjR8//opjBwYGatGiRfrTn/6kXr166euvv1anTp1qffd8xowZatiwoQYOHKj77rtPMTExGjlypCSpb9++GjZsmOLi4jRixAhFRERc9Pr4+Hilp6erb9++qqio0PTp0+26DkB9cezYMa1atUoHDx5Uv379rvgcTXoLvcXdWIzapgcAJztz5ox69OihjRs3ql27dtd8fHd8NhcA91ddXa3w8HA999xz6t279zUfPykpSXFxcUpMTLzmYwNwX/SW+oUZK7jcpk2bdO7cOZ09e1bPPPOMOnTo4FYPAgTgnfLy8nTq1ClVVFRoyZIlki7sUgoAjqC31F/sCgiXy83N1dSpU2UYhjp16qQFCxZctEgUAJzts88+05QpU1RRUaFbb71VL7300hXXfADAldBb6i9uBQQAAAAAB3ErIAAAAAA4yCOC1bhx41xdAoB6iv4CwAz0FsD7eESwKi4udnUJAOop+gsAM9BbAO/jEcEKAAAAANwZwQoAAAAAHESwAgAAAAAHEawAAAAAwEEEKwAAAABwEMEKAAAAABxEsAIAAAAABxGsAAAAAMBBBCsAAAAAcBDByk1VWKvrxTkAuI6Zv+P0D8Cz1Je/K+g9cGd+ri4Al+bv56PRmTtMPcfKlDBTxwfgWmb2EfoH4Fmc9XcFf7vAmxGsAHic8vJy3X///aqoqFBVVZUGDx6s1NRUpaena/fu3brxxhslSfPnz9cdd9zh4moBAIA3IFgB8Dj+/v5atmyZbrjhBlVWVuq+++5TeHi4JGnq1KkaMmSIiysEAADehjVWADyOxWLRDTfcIEmyWq2yWq2yWCwurgoAAHgzghUAj1RVVaX4+Hj16dNHffr0UZcuXSRJL7zwgmJjYzV37lxVVFS4uEoAAOAtCFYAPJKvr6+ys7O1ZcsW7du3T//617+Ulpam999/X2+//bZKS0uVmZnp6jIBAICXIFgB8GiNGjVSr169lJeXp6CgIFksFvn7+2vEiBHav3+/q8sDAABegmAFwOOcPHlSp06dkiSdP39e27dvV0hIiAoLCyVJhmEoJydHoaGhriwTAAB4EXYFBOBxCgsLlZ6erqqqKhmGoSFDhigiIkIPPvigiouLZRiGbr/9ds2cOdPVpQIAAC9BsALgcW6//XZlZWVddPyNN95wQTUAAADcCggAAAAADjNtxur48eOaOnWqioqKZLFYNGrUKD300ENavHixVq9ercDAQElSWlqa7r77brPKAAAAAADTmRasfH19lZ6ero4dO+r06dMaOXKk+vbtK0kaO3asxo0bZ9apAQAAAMCpTAtWQUFBCgoKkiQFBAQoJCREBQUFZp0OAAAAAFzGKWusjh49qoMHD6pLly6SpBUrVig2NlYZGRkqLS11RgnXVIW12tUlAAAAAHAjpu8KeObMGaWmpmratGkKCAjQmDFjNH78eFksFi1atEjz58/XvHnzzC7jmvL389HozB2mnmNlSpip4wMAAAC4dkydsaqsrFRqaqpiY2MVFRUlSWrevLl8fX3l4+OjxMRE7d+/38wSAAAAAMB0pgUrwzA0ffp0hYSEKDk52Xa8sLDQ9nFOTo5CQ0PNKgEAAAAAnMK0WwE//fRTZWdnq0OHDoqPj5d0YWv19evX69ChQ5KkNm3aaNasWWaVAAAAYDceFQPAEaYFq+7du+vLL7+86DiNCAAAuCMeFQPAEaZvXgEAAOAJeFQMAEc4Zbt1AAAAT1LfHhUDwHwEKwAAgJ+51KNiPvzwQ2VnZysoKEjz5893dYkA3BDBCgAA4N94VAyAq0WwAgAAEI+KAeAYNq8AAAAQj4oB4BiCFQAAgHhUDADHcCsgAAAAADiIYAUAAAAADiJYAQAAAICDWGMFwOOUl5fr/vvvV0VFhaqqqjR48GClpqbqyJEjSktLU0lJiTp27Khnn31W/v7+ri4XAAB4AWasAHgcf39/LVu2TOvWrVNWVpby8vL02Wef6bnnntPYsWP14YcfqlGjRlqzZo2rSwUAAF6CYAXA41gsFt1www2SJKvVKqvVKovFop07d2rw4MGSpOHDhys3N9eVZQIAAC9CsALgkaqqqhQfH68+ffqoT58+ateunRo1aiQ/vwt3OLds2VIFBQUurhIAAHgLghUAj+Tr66vs7Gxt2bJF+/bt07fffuvqkgAAgBcjWAHwaI0aNVKvXr302Wef6dSpU7JarZKk/Px8BQcHu7g6AADgLQhWADzOyZMnderUKUnS+fPntX37dt1yyy3q1auXNm7cKElau3atIiMjXVkmAADwImy37sUqrNXy9zMvW5s9PrxXYWGh0tPTVVVVJcMwNGTIEEVEROjWW2/V5MmTtXDhQt1xxx1KTEx0dakAAMBLEKy8mL+fj0Zn7jBt/JUpYaaNDe92++23Kysr66Lj7dq1Y4t1AADgEkwnAAAAAICDCFYAAAAA4CCCFQAAAAA4iGAFAAAAAA4iWAEAAACAgwhWAAAAAOAgghUAAAAAOIhgBQAAAAAOIlgBAAAAgIMIVgAAAADgIIIVAAAAADjIz6yBjx8/rqlTp6qoqEgWi0WjRo3SQw89pJKSEk2ePFnHjh1TmzZttHDhQjVu3NisMgAAAADAdKbNWPn6+io9PV0bNmzQqlWr9Ne//lVff/21MjMzFRYWpg8++EBhYWHKzMw0qwQAAAAAcArTglVQUJA6duwoSQoICFBISIgKCgqUm5urhIQESVJCQoJycnLMKgEAAAAAnMIpa6yOHj2qgwcPqkuXLioqKlJQUJAkqUWLFioqKnJGCQAAAABgGtOD1ZkzZ5Samqpp06YpICCgxtcsFossFovZJQAAAACAqUwNVpWVlUpNTVVsbKyioqIkSc2aNVNhYaEkqbCwUIGBgWaWAAAAYLfjx48rKSlJw4YNU3R0tJYtWyZJKikpUXJysqKiopScnKzS0lIXVwrA3ZgWrAzD0PTp0xUSEqLk5GTb8cjISGVlZUmSsrKyNGDAALNKAAAAqBM23wJwtUwLVp9++qmys7O1c+dOxcfHKz4+Xlu2bFFKSoq2bdumqKgobd++XSkpKWaVAAAAUCdsvgXgapn2HKvu3bvryy+/vOTXfppWBwAAcFfO3Hyrwlotfz+n7CkGwCSmBSsAMMvlHkC+ePFirV692rZ2My0tTXfffbeLqwXgiZy9+Za/n49GZ+64pmP+3MqUMNPGBnABwQqAx/lpDUTHjh11+vRpjRw5Un379pUkjR07VuPGjXNxhQA8WW2bbwUFBbH5FoBLYs4ZgMe53BoIAHAUm28BuFoEKwAe7edrICRpxYoVio2NVUZGBtshm6jCWu1R4wL2YvMtAFeLWwEBeKxfroEYM2aMxo8fL4vFokWLFmn+/PmaN2+eq8usl8xaD8I6ELgam28BuFrMWAHwSJdaA9G8eXP5+vrKx8dHiYmJ2r9/v4urBAAA3oJgBcDjXG4NRGFhoe3jnJwchYaGuqI8AADghbgVEIDH+WkNRIcOHRQfHy/pwtbq69ev16FDhyRJbdq00axZs1xZJgDgGnPG8754phiuFsEKgMe53BoInlkFAPWb2c/7kljriatHHAcAAAAABxGsAAAAAMBBBCsAAAAAcBDBCgAAAAAcRLACAAAAAAcRrAAAAADAQQQrALBThbXao8YFAADOw3OsAMBOZj0/ZWVKmGnjAgAA52DGCqZxxrvwvNMPAAAAd8CMFUzD09EBAADgLZixAgAAAAAHEawAAAAAwEEEKwAAAABwEMEKAAAA+Dc238LVYvMKAAAA4N/YfAtXixkrAAAAAHAQwQoAAAAAHGRXsPr000/tOgYAAAAA3siuYDV79my7jgEAAACAN6p184q9e/dq7969OnnypF5//XXb8dOnT6uqqsr04gAAAADAE9Q6Y1VZWamzZ8+qqqpKZ86csf0LCAjQiy++WOvAGRkZCgsLU0xMjO3Y4sWL1b9/f8XHxys+Pl5btmy5Nj8FAK9y/PhxJSUladiwYYqOjtayZcskSSUlJUpOTlZUVJSSk5NVWlrq4koBAIC3qHXGqmfPnurZs6eGDx+uNm3a1GngESNG6IEHHtBTTz1V4/jYsWM1bty4ulcKAP/m6+ur9PR0dezYUadPn9bIkSPVt29fvfPOOwoLC1NKSooyMzOVmZmpJ5980tXlAgAAL2DXc6wqKio0Y8YMHTt2TFar1Xb8jTfeuOxrevTooaNHjzpeIQD8QlBQkIKCgiRJAQEBCgkJUUFBgXJzc7V8+XJJUkJCgpKSkghWAADAKewKVpMmTdLo0aOVmJgoHx/HdmhfsWKFsrKy1KlTJ6Wnp6tx48YOjfdLFdZq+fuxizzgLY4ePaqDBw+qS5cuKioqsgWuFi1aqKioyMXVAQAAb2FXsPLz89N9993n8MnGjBmj8ePHy2KxaNGiRZo/f77mzZvn8Lg/x9OyAe9x5swZpaamatq0aQoICKjxNYvFIovF4qLKAHiqjIwMbd68Wc2aNdP69eslXVgjvnr1agUGBkqS0tLSdPfdd7uyTABuyK6pnYiICK1YsUKFhYUqKSmx/aur5s2by9fXVz4+PkpMTNT+/fvrPAYASBc210lNTVVsbKyioqIkSc2aNVNhYaEkqbCw0PZHEADYa8SIEVq6dOlFx8eOHavs7GxlZ2cTqgBckl0zVmvXrpUkvfbaa7ZjFotFubm5dTpZYWGh7TadnJwchYaG1un1ACBJhmFo+vTpCgkJUXJysu14ZGSksrKylJKSoqysLA0YMMCFVQLwRKwRB3C17ApWmzZtqvPAaWlp2r17t4qLixUeHq6JEydq9+7dOnTokCSpTZs2mjVrVp3HBYBPP/1U2dnZ6tChg+Lj4yVd6DkpKSl6/PHHtWbNGrVu3VoLFy50caUA6guz14gD8Hx2BausrKxLHk9ISLjsaxYsWHDRscTERDvLAoDL6969u7788stLfu2nZ1oBwLXijDXiADyfXcHq52uhysvLtWPHDnXs2LHWYAUAQF2ZtbMrO8bCEc2bN7d9nJiYqEceecSF1QBwV3YFqxkzZtT4/NSpU5o8ebIpBQEAvJdZO7uymyscwRpxAPawK1j9UsOGDVnYCQAA6h3WiAO4WnYFq59PeVdXV+ubb77R0KFDTSsKAADAFVgjDuBq2RWsHn74YdvHvr6+atOmjVq2bGlaUQAAAADgSexayduzZ0+FhITozJkzOnXqlBo0aGB2XQAAAADgMewKVhs2bFBiYqLef/99/f3vf7d9DACAJ6iwVnvk2AAAz2HXrYBLlizRmjVr1KxZM0nSyZMnNXbsWA0ZMsTU4gAAuBbM2m1QYsdBAMAFds1YGYZhC1WS1KRJExmGYVpRAAAAAOBJ7Jqx6tevn8aNG6fo6GhJF24NDA8PN7UwAAAAAPAUtQarH374QSdOnNBTTz2lDz74QJ9++qkkqWvXroqLi3NKgQAAAADg7mq9FXDu3LkKCAiQJEVFRSkjI0MZGRkaNGiQ5s6d65QCAQAAAMDd1RqsTpw4odtuu+2i47fddpuOHTtmWlEAAAAA4ElqDVZlZWWX/dr58+eveTEAAAAA4IlqDVadOnXS6tWrLzr+1ltvqWPHjqYVBQAAAACepNbNK6ZNm6YJEybo3XfftQWpAwcOqLKyUv/93//tlAIBAAAAwN3VGqyaN2+ulStXaufOnfrqq68kSXfffbfCwngYIgAAAAD8xK7nWPXu3Vu9e/c2uxYAAAAA8Ei1rrECAAAAAFwZwQoAAAAAHESwAuBxMjIyFBYWppiYGNuxxYsXq3///oqPj1d8fLy2bNniwgoBAIC3sWuNFQC4kxEjRuiBBx7QU089VeP42LFjNW7cOBdVBQAAvBkzVgA8To8ePdS4cWNXlwEAAGBDsAJQb6xYsUKxsbHKyMhQaWmpq8sBAABehGAFoF4YM2aMPvzwQ2VnZysoKEjz5893dUkAAMCLEKwA1AvNmzeXr6+vfHx8lJiYqP3797u6JAAA4EUIVgDqhcLCQtvHOTk5Cg0NdWE1AADA27ArIACPk5aWpt27d6u4uFjh4eGaOHGidu/erUOHDkmS2rRpo1mzZrm4SgAA4E0IVgA8zoIFCy46lpiY6IJKAAAALuBWQAAAAABwkGnBKiMjQ2FhYYqJibEdKykpUXJysqKiopScnMx2yAAAAADqBdOC1YgRI7R06dIaxzIzMxUWFqYPPvhAYWFhyszMNOv0AAAAAOA0pgWrHj16qHHjxjWO5ebmKiEhQZKUkJCgnJwcs04PAABQZ9xxA+BqOXWNVVFRkYKCgiRJLVq0UFFRkTNPDwAAUCvuuAFwtVy2eYXFYpHFYnHV6QEAAC7CHTcArpZTg1WzZs1sD/EsLCxUYGCgM08PAABQZ9xxA8AeTg1WkZGRysrKkiRlZWVpwIABzjw9AACAQ7jjBp6gwlrNOVxwDtMeEJyWlqbdu3eruLhY4eHhmjhxolJSUvT4449rzZo1at26tRYuXGjW6QEAAK6Jn+64CQoK4o4beAR/Px+Nztxh6jlWpoSZOr7keT+HacFqwYIFlzy+bNkys04JAABwzf10x01KSgp33AC4LJdtXgEAAOBu0tLSNHr0aH333XcKDw/XW2+9pZSUFG3btk1RUVHavn27UlJSXF0mADdk2owVAACAp+GOGwBXixkr4Ao8beEkAAAAnI8ZK+AKPG3hJAAAAJyPGSsAAAAAcBDBCgAAAAAcRLACAAAAAAcRrAAAAADAQQQrAAAAAHAQwQoAAAAAHESwAuBxMjIyFBYWppiYGNuxkpISJScnKyoqSsnJySotLXVhhQAAwNsQrAB4nBEjRmjp0qU1jmVmZiosLEwffPCBwsLClJmZ6aLqAACoXYW12tUlwAQ8IBiAx+nRo4eOHj1a41hubq6WL18uSUpISFBSUpKefPJJV5QHAECt/P18NDpzh2njr0wJM21sXB4zVgDqhaKiIgUFBUmSWrRooaKiIhdXBAAAvAnBCkC9Y7FYZLFYXF0GAADwIgQrAPVCs2bNVFhYKEkqLCxUYGCgiysCAADehGAFoF6IjIxUVlaWJCkrK0sDBgxwcUUAAMCbEKwAeJy0tDSNHj1a3333ncLDw/XWW28pJSVF27ZtU1RUlLZv366UlBRXlwkAALwIuwIC8DgLFiy45PFly5Y5uRIAAIALmLECAAAAAAcRrAAAAADAQQQrAAAAAHAQwQoAAAAAHESwAgAAAAAHEawAAAAAwEEEK3i0Cmu1q0sAAAAAeI4VPJu/n49GZ+4w9RwrU8JMHR8AAMDTVFir5e/HHM3PEawAAAAA1Alvbl+MmAkAAAAADnLJjFVkZKRuuOEG+fj4yNfXV++8844rygAAAACAa8JltwIuW7ZMgYGBrjo9AAC4xsxac8FaDgCegDVWAADgmjBrzYW7rLPgjhsAtXFZsBo3bpwsFovuvVEEoRsAAA+mSURBVPde3Xvvva4qAwAAwG7ccQPgclwSrP72t78pODhYRUVFSk5OVkhIiHr06OGKUgAAAADAYS65YTk4OFiS1KxZMw0aNEj79u1zRRkAAAB1Mm7cOI0YMUKrVq1ydSkA3IzTZ6zOnj2r6upqBQQE6OzZs9q2bZvGjx/v7DIAAADqhDtuANTG6cGqqKhIjz32mCSpqqpKMTExCg8Pd3YZAABcE+yE5z0udccNwQrAT5werNq1a6d169Y5+7QAAJiivu+Ehwu44wbAlbDdOgAAwBVwxw2AKyFYAahXeM4MADNwxw2AKyFYAW7AGWspvGm9Bs+ZAQAAzkawAtyAWWs0fo71GgAAAObxjrevAXgVnjMDAACcjRkrAPUKz5kBAACuwIwVgHrlUs+ZAQAAMBvBCkC9cfbsWZ0+fdr28bZt2xQaGuriqgAAgDfgVkAA9QbPmQEAAK5CsAJQb/CcGQAA4CrcCggAgBuqsFZ71LgA4O2YsQIAwA2Z9Xw7nmkHAOZgxgoAAAAAHESwAgAAAAAHEawAAAAAwEEEKwAAAABwEMEKAAAvwq6AAGAOdgUEvESFtVr+fua9l2L2+ACuDbN2G5TYcRCAdyNYAV7CzD+mJP6gAgAA3o23lwEAAADAQQQrAAAAAHAQwQoAAAAAHESwAgAAAAAHEawAAAAAwEEEKwAAAABwEMEKAAAAABxEsAIAAAAABxGsAAAAAMBBBCsAAAAAcBDBCgAAAAAc5JJgtXXrVg0ePFiDBg1SZmamK0oAUE/RXwCYhf4CoDZOD1ZVVVWaNWuWli5dqvfee0/r16/X119/7ewyANRD9BcAZqG/ALgSpwerffv26eabb1a7du3k7++v6Oho5ebmOrsMAPUQ/QWAWegvAK7EYhiG4cwTvv/++8rLy9OcOXMkSVlZWdq3b5+efvrpy76mV69eatOmjbNKBOAETZs21WuvvXZNx6S/ADCjt0h17y/0FqD+uVJ/8XNiLVdt165dri4BQD1FfwFgBnoL4H2cfitgcHCw8vPzbZ8XFBQoODjY2WUAqIfoLwDMQn8BcCVOD1Z33nmnvv/+ex05ckQVFRV67733FBkZ6ewyANRD9BcAZqG/ALgSp98K6Ofnp6efflq/+93vVFVVpZEjRyo0NNTZZQCoh+gvAMxCfwFwJU7fvAIAAAAA6huXPCAYAAAAAOoTghUAAAAAOKheBautW7dq8ODBGjRokDIzM11djiTp+PHjSkpK0rBhwxQdHa1ly5ZJkkpKSpScnKyoqCglJyertLRUkmQYhmbPnq1BgwYpNjZW//znP11Sd1VVlRISEvSf//mfkqQjR44oMTFRgwYN0uOPP66KigpJUkVFhR5//HENGjRIiYmJOnr0qEvqPXXqlFJTUzVkyBANHTpUe/fudftr/Je//EXR0dGKiYlRWlqaysvL3e46Z2RkKCwsTDExMbZjV3Nd165dq6ioKEVFRWnt2rVOqf1acsfeItFfnIX+Yg76ywX0l2uL/mIuessVGPWE1Wo1BgwYYBw+fNgoLy83YmNjja+++srVZRkFBQXGgQMHDMMwjLKyMiMqKsr46quvjGeeecZ49dVXDcMwjFdffdV49tlnDcMwjM2bNxvjxo0zqqurjb179xr33HOPS+r+85//bKSlpRkpKSmGYRhGamqqsX79esMwDGPGjBnGihUrDMMwjDfffNOYMWOGYRiGsX79emPSpEkuqXfq1KnG6tWrDcMwjPLycqO0tNStr3F+fr4RERFhnDt3zjCMC9f37bffdrvrvHv3buPAgQNGdHS07Vhdr2txcbERGRlpFBcXGyUlJUZkZKRRUlLilPqvBXftLYZBf3EW+os56C/0FzPQX8xDb7lyb6k3M1b79u3TzTffrHbt2snf31/R0dHKzc11dVkKCgpSx44dJUkBAQEKCQlRQUGBcnNzlZCQIElKSEhQTk6OJNmOWywWde3aVadOnVJhYaFTa87Pz9fmzZt1zz33SLqQ5nfu3KnBgwdLkoYPH267tps2bdLw4cMlSYMHD9aOHTtkOHk/lLKyMn388ce2ev39/dWoUSO3vsbShXfVzp8/L6vVqvPnz6tFixZud5179Oihxo0b1zhW1+v60UcfqW/fvmrSpIkaN26svn37Ki8vz/TarxV37S0S/cUZ6C/mob/QX641+ov56C21qzfBqqCgQC1btrR9HhwcrIKCAhdWdLGjR4/q4MGD6tKli4qKihQUFCRJatGihYqKiiRd/HO0bNnS6T/H3Llz9eSTT8rH58L/HsXFxWrUqJH8/PwuqqmgoECtWrWSdGEr2htvvFHFxcVOrffo0aMKDAxURkaGEhISNH36dJ09e9atr3FwcLAefvhhRUREqF+/fgoICFDHjh3d+jr/pK7X1RN+N2vjKfXTX8xBf3Eu+ot71k9/MYen9Rd6y5Wvdb0JVu7uzJkzSk1N1bRp0xQQEFDjaxaLRRaLxUWV1fSPf/xDgYGB6tSpk6tLsZvVatUXX3yhMWPGKCsrSw0bNrzoPnV3usaSVFpaqtzcXOXm5iovL0/nzp3zqHdZf+Ju19Vb0V/MQ39xHXe7rt6K/mIeT+sv9JYrqzfBKjg4WPn5+bbPCwoKFBwc7MKK/l9lZaVSU1MVGxurqKgoSVKzZs1s07eFhYUKDAyUdPHPkZ+f79SfY8+ePdq0aZMiIyOVlpamnTt3as6cOTp16pSsVutFNQUHB+v48eOSLjSIsrIyNW3a1Gn1ShfeXWjZsqW6dOkiSRoyZIi++OILt73GkrR9+3a1bdtWgYGBatCggaKiorRnzx63vs4/qet1deffTXu4e/30F3PRX5yL/uJe9dNfzOVp/YXecuVrXW+C1Z133qnvv/9eR44cUUVFhd577z1FRka6uiwZhqHp06crJCREycnJtuORkZHKysqSJGVlZWnAgAE1jhuGoc8++0w33nijberSGZ544glt3bpVmzZt0oIFC9S7d289//zz6tWrlzZu3Cjpwi4pP13byMhI204pGzduVO/evZ3+zkqLFi3UsmVLffvtt5KkHTt26JZbbnHbayxJrVu31ueff65z587JMAzt2LFDt956q1tf55/U9br269dPH330kUpLS1VaWqqPPvpI/fr1c0ntV8Nde4tEf3EG+otz0V/oL1eL/mI+eosdvcWBTTfczubNm42oqChjwIABxssvv+zqcgzDMIyPP/7Y6NChgxETE2PExcUZcXFxxubNm42TJ08aDz74oDFo0CDjoYceMoqLiw3DMIzq6mrjD3/4gzFgwAAjJibG2Ldvn8tq37lzp21XncOHDxsjR440Bg4caEycONEoLy83DMMwzp8/b0ycONEYOHCgMXLkSOPw4cMuqfWLL74whg8fbsTExBiPPvqoUVJS4vbXeNGiRcbgwYON6OhoY8qUKUZ5ebnbXefJkycbffv2NX79618b/fv3N1avXn1V1/Wtt94yBg4caAwcONBYs2aNU2q/ltyxtxgG/cVZ6C/moL9cQH+59ugv5qG31M5iGE7eAgUAAAAA6pl6cysgAAAAALgKwQoAAAAAHESwAgAAAAAHEawAAAAAwEEEKwAAAABwEMEKAAAAABxEsILNkiVLbB+fOnVKK1asuGZj79q1S3v27LF9/re//c32oDZ38s033yg+Pl4JCQk6fPiw3njjDQ0dOlRPPPGEq0sDPBr9hf4CmIX+Qn9xFzzHCjbdunXT3r17JUlHjx7VI488ovXr11/0fVarVX5+fnUae/Hixbr++us1bty4a1KrWTIzM2W1WjV+/HhJ0pAhQ/SXv/xFLVu2dHFlgGejv9BfALPQX+gv7oJg5aXGjx+v/Px8lZeX68EHH9SRI0f02muvqUOHDrr11ltVXV2t3NxctW/fXn369NFvf/tbLVq0SI0aNdJ3332njRs3XjTGvffeK0naunWrXnjhBVVVValp06aaM2eO7r33Xvn4+CgwMFAzZszQjh07bI3q4MGD+v3vf69z587ppptu0ty5c9W4cWMlJSWpc+fO2rVrl8rKyjRnzhx17979kj9PVVWVnnvuOeXl5clisWjUqFFKSkrSjh079Mwzz6iqqkqdOnXSzJkz5e/vrwMHDmj+/Pk6e/asmjZtqnnz5ungwYOaNm2afHx89B//8R9q37693nnnHbVv314jR47U2LFjnfhfCPBc9Bf6C2AW+gv9xa0Z8ErFxcWGYRjGuXPnjOjoaOPkyZNG165dbV8/cuSIER0dbft8586dRpcuXYzDhw/XOkZRUZERHh5u+76fvufFF180li5danvtzz+PiYkxdu3aZRiGYSxcuNCYPXu2YRiG8cADDxjz5s0zDMMwNm/ebDz00EOX/XlWrFhhTJw40aisrLSd9/z580Z4eLjx7bffGoZhGE8++aTx+uuvGxUVFca9995rFBUVGYZhGO+9956Rnp5+yTojIiJs3wfAPvQX+gtgFvoL/cWd1W0+FPXG8uXL9eGHH0qSjh8/rh9++OGKr7nzzjvVrl27Wsc4efKkunfvbvu+Jk2a1DpmWVmZysrK1LNnT0nS8OHDNWnSJNvXBw0aJEnq2LGjjh07dtlxduzYodGjR9um+Js0aaJDhw6pbdu2at++vW3sFStWqE+fPvrXv/6l5ORkSVJ1dbVatGhxxZ8fgH3oL/QXwCz0F/qLOyNYeaFdu3Zp+/btWrVqlRo2bKikpCSVl5df8XXXX3+9w2PUlb+/vyTJx8dHVVVV12RMwzAUGhqqVatWXZPxAPw/+gv9BTAL/YX+4u7YFdALlZWVqXHjxmrYsKG++eYbffbZZ5IkPz8/VVZWSpJuuOEGnTlzps5jdO3aVZ988omOHDkiSSopKal1vBtvvFGNGjXSJ598IknKzs5Wjx496vwz9enTR6tWrZLVarWdt3379jp27Jjt3ayfxm7fvr1OnjxpW+haWVmpr776qs7nBHAx+gv9BTAL/YX+4u6YsfJC4eHhWrlypYYOHar27dura9eukqRRo0YpLi5Ov/71r/X888/rrrvuUkxMjPr376/f/va3do0RGBioWbNmaeLEiaqurlazZs30+uuvKyIiQqmpqcrNzdWMGTNqjPXMM8/YFn+2a9dO8+bNq/PPlJiYqO+//15xcXHy8/PTqFGj9MADD2jevHmaNGmSbfHnmDFj5O/vrxdffFGzZ89WWVmZqqqq9NBDDyk0NPTqLigAG/oL/QUwC/2F/uLu2BUQAAAAABzErYAAAAAA4CBuBYRHycvL03PPPVfjWNu2bfXSSy+5qCIA9QX9BYBZ6C/egVsBAQAAAMBB3AoIAAAAAA4iWAEAAACAgwhWAAAAAOAgghUAAAAAOOj/APi0XVf7F9eBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Распределение пользователей в зависимости от коэффициента привлекательности\n",
    "fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))\n",
    "\n",
    "sns.histplot(x=users_test.attraction_coeff, ax = axes[0])\n",
    "axes[0].set(title='Test group')\n",
    "\n",
    "sns.histplot(x=users_control_1.attraction_coeff, ax = axes[1])\n",
    "axes[1].set(title='1 control group', ylabel='')\n",
    "\n",
    "sns.histplot(x=users_control_2.attraction_coeff, ax = axes[2])\n",
    "axes[2].set(title='2 control group', ylabel='')\n",
    "\n",
    "sns.despine()\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "353.13385826771656"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Посмотрим на средние значения коэффициента привлекательности по группам\n",
    "users_test.attraction_coeff.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "339.6219512195122"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_control_1.attraction_coeff.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "381.7260273972603"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_control_2.attraction_coeff.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>W</th>\n",
       "      <th>pval</th>\n",
       "      <th>equal_var</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>levene</th>\n",
       "      <td>2.508938</td>\n",
       "      <td>0.082534</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               W      pval  equal_var\n",
       "levene  2.508938  0.082534       True"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Проверка на гомоскедастичность\n",
    "pg.homoscedasticity(data = users_merged, dv = 'attraction_coeff', group = 'group')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Т.к. у нас 3 группы с количественными переменными, сравним их распределения при помощи критерия Краскела-Уоллиса:\n",
    "\n",
    "- $H_0$: нет статистически значимых различий между распределениями коэффициента привлекательности в тестовой и контрольных группах (p-value > 0.05)\n",
    "- $H_1$: есть статистически значимые различия между распределениями коэффициента привлекательности в тестовой и контрольных группах"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KruskalResult(statistic=1.3258233628683038, pvalue=0.5153486168360484)"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kruskal(users_merged.query('group == \"test\"').attraction_coeff, users_merged.query('group == \"control_1\"').attraction_coeff, users_merged.query('group == \"control_2\"').attraction_coeff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Средний коэффициент привлекательности пользователей в группах - 340-382 (max - 1000)\n",
    "\n",
    "Согласно статистике (p-value > 0.05) нулевую гипотезу об отсутствии различий распределений коэффициента привлекательности в тестовой и контрольных группах мы отклонить не можем. Т.е. мы можем считать распределения в группах одинаковыми, что говорит о репрезентативности выборок по этому признаку."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAEYCAYAAABBWFftAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de3RU5b3/8c9khojlzkAmgCnrILG6uLfQGEAoYyYJCSEhJD3aFkKqK0dEUkDwB1K5iVRbihSwSkoPJyi2CEKCDQWSUEMAkVbFUBZyFm1jAcmkDrlwUcJM5vdHFnOMXAx2JnuSvF9ruVb2M7P3fHdKns5nP89+tsnr9XoFAAAAAPi3hRhdAAAAAAC0FgQsAAAAAPATAhYAAAAA+AkBCwAAAAD8hIAFAAAAAH5CwAIAAAAAPyFgAQAg6d1339WYMWOMLgNAK0Y/0zYQsHBTw4YN8/137733avDgwb7tnTt33vbxpkyZoq1btwagUgBGe+2115SamqqBAwdq/vz5htTwrW99Sx9//LEhnw0gsOrq6vT0009r3LhxGjZsmJKTk1VSUtLsddDPoCksRheA4PXBBx/4frbb7Vq+fLlGjhxpYEWNud1uWSz8EwaCQVhYmB5//HGVlpbqypUrRpdzQ83ZZ9A/Af7ldrvVq1cvvfrqq+rdu7dKSko0a9YsvfXWW7rrrruMLs+HfgYSI1j4Gurr65WTk6OYmBhFRUXpJz/5iaqrqyVJV65c0dy5cxUVFaXhw4dr8uTJ+vTTT/Xiiy/qL3/5i5YtW6Zhw4Zp2bJlNzx2Xl6exo0bp6ioKL300kuy2+06dOiQJGnt2rXKzs7W3Llz9e1vf1s7duyQ0+nUY489pu9+97tyOBx64403fMeaP3++XnzxRd/2l4fl7Xa71q9fr4SEBI0YMUILFiwI2i+GQLCLjY1VTEyMunbt2qT3v/HGGxo/fryGDRumhIQEHT9+XJL0t7/9TVOmTNHw4cOVmJio4uJi3z7z58/X0qVLlZWVpWHDhik9PV3//Oc/JUk//OEPJUnJyckaNmyYdu3a5fubz8nJ0ahRo7RgwQLV1dXpueee0+jRozV69Gg999xzqqura1LNBw4cUFxcnL7zne9oyZIl+tGPfuQbld++fbseeughrVixQlFRUVq7dq0uXLigp556Svfff7/GjRunX//616qvr5fU0J/NnTvXd+wzZ87oW9/6ltxut6SGEf9f/vKXSktL07e//W1Nnz7d188CbdE3vvENzZw5U3fddZdCQkI0btw43XXXXb6+40boZ+hnjELAwm179dVXVVRUpNdee02lpaXq0qWLLzDt2LFDFy9e1Ntvv613331XS5cuVfv27TV79mwNHz5cixYt0gcffKBFixZdd9xTp05p6dKl+sUvfqHS0lJdvHhRTqez0XuKi4sVHx+vv/zlL0pKStKcOXMUHh6u0tJSrVmzRqtWrdI777zT5HN566239Nvf/laFhYX6xz/+oV//+tf/3i8HwFf64x//qLVr1+qFF17Q+++/r5dfflldu3bV1atX9dhjj2nUqFE6dOiQfvrTn2ru3Ln6+9//7tt3165deuKJJ/TnP/9Z3/zmN30XUTZv3ixJys/P1wcffKCEhARJ0qeffqqamhr96U9/0rPPPquXX35ZH374ofLz87Vz504dO3asSX/358+fV3Z2tp588km9++67+o//+I9Go/ySVFZWpoiICB08eFDTp0/Xs88+qwsXLqioqEivvvqq8vPz9eabbzb595SXl6cVK1bowIEDslgsWr58eZP3BVq7Tz/9VOXl5erfv/8NX6efaRr6mcAgYOG2/f73v9fs2bMVHh6u0NBQPfHEE9qzZ49vqLq6uloff/yxzGazBg4cqI4dOzbpuLt379a4ceM0fPhwhYaGKjs7WyaTqdF7hg4dqpiYGIWEhKiqqkrvv/++5s6dqzvuuEP33Xef0tPTlZ+f3+Rz+eEPf6hevXqpa9eumj59ugoKCm7rdwHg9m3btk2PPvqoBg8eLJPJpL59+6pPnz768MMPdfnyZWVlZSk0NFTR0dEaN25co7/LmJgYDR48WBaLRRMnTtSJEydu+VkhISHKzs5WaGio2rdvr7feekszZsyQ1WpV9+7dNWPGjCbdU7p//35FRkYqNjZWFotFU6dOVY8ePRq9JywsTFOmTJHFYlG7du20a9cuPfnkk+rYsaPuuusuZWZm3tb9q8nJybrnnnv0jW98Qz/5yU+0e/dueTyeJu8PtFZXr17V3LlzNWnSJN199903fA/9TNPQzwQGEzdx2z755BPNmDFDISH/l89DQkLkcrmUnJysiooKzZkzR7W1tZo4caJmz56tdu3afeVxKysrFR4e7tu+8847r5tu9MXXKysr1aVLl0YBrnfv3vrrX//a5HPp1atXo30rKyubvC+Ar+fcuXP65je/eV37tT7gi31L7969G41kf/HLRvv27XX58uVbfla3bt10xx13NPqM3r17Nzp+U/7uv9w/mUymRttS4/6pqqpKV69eve6zvjwqfytf7p+uXr2qqqqq675wAW1JfX29nnrqKbVr107PPPPMTd9HP9M09DOBwQgWblt4eLh+85vf6C9/+Yvvv2PHjslms6ldu3Z64okntGvXLv3+97/X22+/rby8vCYdNywsrFGn8Pnnn183F/iLI1phYWGqqanRxYsXfW3nzp2TzWaT1BDQPv/8c99rn3766XWfee7cOd/Pn3zyicLCwppUK4Cvr1evXr57Gr4oLCxMFRUVvvsHpMZ/01/Hl0fBw8LC9MknnzQ6flP+7nv27Nmof/J6vaqoqLjpZ3Xr1k3t2rW77rO+bv907tw5tWvXTt26dfvKWoHWyuv1auHChfr000+1du3aW168pZ+hnzESAQu37eGHH9bq1at19uxZSQ1zhouKiiRJhw8f1smTJ+XxeNSxY0dZLBbfVaIePXro9OnTNz1uXFyc9u3bp/fff191dXVau3atvF7vTd/fq1cvDRs2TKtWrdKVK1f00Ucfadu2bZo4caIk6b777lNJSYmqq6v1r3/9S7m5udcd4/XXX1dFRYWqq6v1yiuv+OZTA7g9brdbV65cUX19vTwej65cueK7kfrL0tLS9N///d/661//Kq/Xq48//lhnz57V4MGD1b59e23YsEFXr17Vu+++q3379jX57/Kr+hhJSkxM1Msvv6zz58/r/Pnzeumll5SUlPSVxx47dqxOnjypoqIiud1ubd68+YZfVq4xm82Kj4/Xiy++qIsXL+rs2bPauHFjo/7pz3/+sz755BNduHBB69evv+4YO3fu1KlTp/TZZ5/pV7/6leLi4mQ2m7+yVqC1Wrx4sf72t7/plVdeUfv27W/5XvoZ+hkjEbBw26ZOnSq73a4f//jHGjZsmL7//e+rrKxMUsPVkezsbH3nO99RQkKCvvvd7yo5Odm33549ezRixIgb3kQZGRmpZ555RnPmzNEDDzygb3zjG+revbtCQ0NvWsuqVat09uxZPfDAA3riiSc0c+ZM31LyycnJuvfee3213qjznDBhgn784x8rJiZG3/zmNzV9+nR//IqANufll1/W4MGDlZOTo507d2rw4MF6+eWXb/je8ePH67HHHtOTTz6pb3/725oxY4ZqamoUGhqqV155Rfv379f999+vpUuX6uc///lN77H4sieeeELz58/X8OHDtWvXrhu+5/HHH9fAgQM1ceJETZw4UQMGDNDjjz/+lcfu3r27fvWrX+kXv/iFoqKidOrUKQ0cOPCWV9CfeeYZ3XnnnYqJidEPfvADTZgwQZMnT5YkjRo1SgkJCZo4caJSU1M1bty46/ZPTk7W/PnzNWrUKNXV1WnhwoVN+j0ArdHZs2e1ZcsWnThxQqNHj/7K53LSz9DPGMnkvdUQAWCgS5cuacSIEdqzZ48iIiL8fvxgfLYXgJahvr5eY8aM0cqVK3X//ff7/fhTpkzRxIkTlZ6e7vdjA2gZ6GdaLkawEFT27dunzz77TJcvX9YLL7yge+65J6geIAig7SotLVVtba3q6ur0yiuvSGpY2RQA/IV+pnVgFUEEleLiYj311FPyer0aOHCgVq1add3NowBghKNHj2ru3Lmqq6tT//799dJLL33lfSAAcDvoZ1oHpggCAAAAgJ8wRRAAAAAA/KRFB6xHHnnE6BIAtHL0MwACjX4GaF1adMCqqqoyugQArRz9DIBAo58BWpcWHbAAAAAAIJgQsAAAAADATwhYAAAAAOAnBCwAAAAA8BMCFgAAAAD4CQELAAAAAPyEgIWg43K5lJ2dLZfLZXQpAFop+hkAgUY/03YRsBB0cnNzdezYMW3atMnoUgC0UvQzAAKNfqbtImAhqLhcLu3evVter1e7d+/mqg8Av6OfARBo9DNtGwELQSU3N1f19fWSJI/Hw1UfAH5HPwMg0Ohn2jYCFoJKUVGR3G63JMntdquwsNDgigC0NvQzAAKNfqZtI2AhqMTExMhisUiSLBaLHA6HwRUBaG3oZwAEGv1M20bAQlDJyMhQSEjDP0uz2aypU6caXBGA1oZ+BkCg0c+0bQQsBBWr1ar4+HiZTCbFx8fLarUaXRKCwJUrV5SWlqaJEycqMTFRa9askSTNnz9fdrtdycnJSk5O1okTJyRJXq9Xy5cvl8PhUFJSko4fP+471o4dOxQbG6vY2Fjt2LHDkPOBsehnAAQa/UzbZjG6AODLMjIyVF5eztUe+ISGhio3N1cdOnTQ1atX9YMf/EBjxoyRJD311FOKj49v9P79+/ervLxce/fu1YcffqglS5Zo69atqq6u1rp16/Tmm2/KZDIpNTVVdrtdXbp0MeK0YCD6GQCBRj/TdhGwEHSsVqtvhAKQJJPJpA4dOkhquFnY7XbLZDLd9P3FxcVKSUmRyWTS0KFDVVtbq8rKSh05ckSjRo1S165dJUmjRo1SaWmpJkyY0CzngeBBPwMg0Ohn2i6mCAJoETwej5KTkzVy5EiNHDlSQ4YMkSS9+OKLSkpK0ooVK1RXVydJcjqdCg8P9+0bHh4up9N5XbvNZpPT6WzeEwEAAK0aAQtAi2A2m5Wfn6+SkhKVlZXpf//3fzVnzhzt3r1bb775pmpqapSTk2N0mQAAoI0jYAFoUTp37qyoqCiVlpYqLCxMJpNJoaGhSk1N1bFjxyQ1jExVVFT49qmoqJDNZruu3el0ymazNfs5AACA1iugActutyspKUnJyclKTU2VJFVXVyszM1OxsbHKzMxUTU2NpFuv+gWgbTt//rxqa2slSZ9//rkOHTqkfv36qbKyUlJD/1FUVKTIyEhJDX1PXl6evF6vjh49qk6dOiksLEyjR4/WgQMHVFNTo5qaGh04cECjR4827LwAAEDrE/BFLnJzc9W9e3ffdk5OjqKjo5WVlaWcnBzl5ORo3rx5N131CwAqKys1f/58eTweeb1excfHa9y4cZo6daqqqqrk9Xp17733aunSpZKksWPHqqSkRA6HQ3feeadWrFghSeratasef/xxpaWlSZJmzJjhW/ACAADAH5p9FcHi4mK9+uqrkqSUlBRNmTJF8+bNu+mqX2FhYc1dIoAgc++99yovL++69k2bNt3w/SaTSYsXL77ha2lpab6ABQAA4G8BvwfrkUceUWpqqrZs2SJJcrlcvtDUs2dPuVwuSTdf9QsAAAAAWoqAjmD97ne/k81mk8vlUmZmpvr169fodZPJdMtn2QAAAABASxLQEaxrq3NZrVY5HA6VlZXJarX6bkyvrKz03Z91s1W/AAAAjHLu3DlNmTJFCQkJSkxMVG5uriRp7dq1euCBB5ScnKzk5GSVlJT49lm/fr0cDofi4uJUWlpqVOkADBKwgHX58mVdvHjR9/PBgwcVGRnpW91LkvLy8vTggw9KuvmqXwAAAEYxm82aP3++du3apS1btuj111/XqVOnJEnTpk1Tfn6+8vPzNXbsWEnSqVOnVFBQoIKCAm3YsEFLly6Vx+Mx8hQANLOATRF0uVyaMWOGJMnj8WjChAkaM2aMBg0apFmzZmnbtm3q3bu3Vq9eLenmq34BAAAYJSwszHfBt2PHjurXr98t7xEvLi5WYmKiQkNDFRERob59+6qsrEzDhg1rrpIBGCxgASsiIkI7d+68rr1bt26+4fUvutWqXwAAAEY7c+aMTpw4oSFDhuj999/X5s2blZeXp4EDB2r+/Pnq0qWLnE6nhgwZ4tvHZrOxaBfQxgR8FUEAAICW7tKlS8rOztbTTz+tjh076uGHH1ZhYaHy8/MVFham559/3ugSAQQJAhYAAMAtXL16VdnZ2UpKSlJsbKwkqUePHjKbzQoJCVF6erqOHTsm6fpFu5xOJ4t2AW0MAQsAAOAmvF6vFi5cqH79+ikzM9PXfm1FZEkqKipSZGSkpIZFuwoKClRXV6fTp0+rvLxcgwcPbva6ARgnoM/BAgAAaMnee+895efn65577lFycrIkac6cOfrDH/6gjz76SJLUp08fLVu2TJIUGRmp8ePHKyEhQWazWYsWLZLZbDasfgDNj4AFAABwE8OHD9fJkyeva7+2LPuNTJ8+XdOnTw9kWQCCGFMEAQAAAMBPCFgAAAAA4CcELAAAAADwEwIWAAAAAPgJAQsAAAAA/ISABQAAAAB+QsACAAAAAD8hYAEAAACAnxCwAAAAAMBPCFgAAAAA4CcELAAAAADwEwIWAAAAAPgJAQtA0Lty5YrS0tI0ceJEJSYmas2aNZKk06dPKz09XQ6HQ7NmzVJdXZ0kqa6uTrNmzZLD4VB6errOnDnjO9b69evlcDgUFxen0tJSQ84HAAC0XgQsAEEvNDRUubm52rlzp/Ly8lRaWqqjR49q5cqVmjZtmgoLC9W5c2dt27ZNkrR161Z17txZhYWFmjZtmlauXClJOnXqlAoKClRQUKANGzZo6dKl8ng8Rp4aAABoZQhYAIKeyWRShw4dJElut1tut1smk0mHDx9WXFycJGnSpEkqLi6WJO3bt0+TJk2SJMXFxemdd96R1+tVcXGxEhMTFRoaqoiICPXt21dlZWXGnBQAAGiVCFgAWgSPx6Pk5GSNHDlSI0eOVEREhDp37iyLxSJJCg8Pl9PplCQ5nU716tVLkmSxWNSpUydVVVXJ6XQqPDzcd0ybzebbBwAAwB8IWABaBLPZrPz8fJWUlKisrEx///vfjS4JAADgOgQsAC1K586dFRUVpaNHj6q2tlZut1uSVFFRIZvNJqlhZOrcuXOSGqYUXrhwQd26dZPNZlNFRYXvWE6n07cPAACAPxCwAAS98+fPq7a2VpL0+eef69ChQ7r77rsVFRWlPXv2SJJ27Nghu90uSbLb7dqxY4ckac+ePbr//vtlMplkt9tVUFCguro6nT59WuXl5Ro8eLAxJwUAAFoli9EFAMBXqays1Pz58+XxeOT1ehUfH69x48apf//+mj17tlavXq377rtP6enpkqS0tDTNmzdPDodDXbp00YsvvihJioyM1Pjx45WQkCCz2axFixbJbDYbeWoAAKCVMXm9Xq/RRXxdqamp2r59u9FlAGjF6GcABBr9DNC6MEUQAAAAAPyEgAUAAAAAfkLAAgAAAAA/IWABAAAAgJ8QsAAAAADATwhYAAAAAOAnBCwAAAAA8BMCFgAAAAD4ScADlsfjUUpKiv7rv/5LknT69Gmlp6fL4XBo1qxZqqurkyTV1dVp1qxZcjgcSk9P15kzZwJdGgAAAAD4VcAD1qZNm3T33Xf7tleuXKlp06apsLBQnTt31rZt2yRJW7duVefOnVVYWKhp06Zp5cqVgS4NAAAAAPwqoAGroqJCb7/9ttLS0iRJXq9Xhw8fVlxcnCRp0qRJKi4uliTt27dPkyZNkiTFxcXpnXfekdfrDWR5AAAAAOBXAQ1YK1as0Lx58xQS0vAxVVVV6ty5sywWiyQpPDxcTqdTkuR0OtWrVy9JksViUadOnVRVVRXI8gAAAADArwIWsP70pz+pe/fuGjhwYKA+AgAAIKDOnTunKVOmKCEhQYmJicrNzZUkVVdXKzMzU7GxscrMzFRNTY2khtk6y5cvl8PhUFJSko4fP25k+QAMYAnUgd9//33t27dP+/fv15UrV3Tx4kU999xzqq2tldvtlsViUUVFhWw2myTJZrPp3LlzCg8Pl9vt1oULF9StW7dAlQcAAPCVzGaz5s+frwEDBujixYuaPHmyRo0ape3btys6OlpZWVnKyclRTk6O5s2bp/3796u8vFx79+7Vhx9+qCVLlmjr1q1GnwaAZhSwEawnn3xS+/fv1759+7Rq1Srdf//9+uUvf6moqCjt2bNHkrRjxw7Z7XZJkt1u144dOyRJe/bs0f333y+TyRSo8gAAAL5SWFiYBgwYIEnq2LGj+vXrJ6fTqeLiYqWkpEiSUlJSVFRUJEm+dpPJpKFDh6q2tlaVlZWG1Q+g+TX7c7DmzZunjRs3yuFwqLq6Wunp6ZKktLQ0VVdXy+FwaOPGjZo7d25zlwYAAHBTZ86c0YkTJzRkyBC5XC6FhYVJknr27CmXyyWp4Z7y8PBw3z5fvN8cQNsQsCmCXxQVFaWoqChJUkREhG9p9i+64447tGbNmuYoBwAA4LZcunRJ2dnZevrpp9WxY8dGr5lMJmbdAPBp9hEs4Ku4XC5lZ2f7rgYCAGCkq1evKjs7W0lJSYqNjZUkWa1W39S/yspKde/eXVLDPeUVFRW+fb94vzmAtoGAhaCTm5urY8eOadOmTUaXAgBo47xerxYuXKh+/fopMzPT126325WXlydJysvL04MPPtio3ev16ujRo+rUqZNvKiGAtoGAhaDicrm0e/dueb1e7d69m1EsAICh3nvvPeXn5+vw4cNKTk5WcnKySkpKlJWVpYMHDyo2NlaHDh1SVlaWJGns2LGKiIiQw+HQM888o8WLFxt8BgCaW7PcgwU0VW5ururr6yVJHo9HmzZt0uzZsw2uCgDQVg0fPlwnT5684WvXnon1RSaTiVAFtHGMYCGoFBUVye12S5LcbrcKCwsNrggAAABoOgIWgkpMTIwsloaBVYvFIofDYXBFMNq5c+c0ZcoUJSQkKDEx0XfFeO3atXrggQcaTdm5Zv369XI4HIqLi1Npaamvff/+/YqLi5PD4VBOTk6znwsAAGj9mCKIoJKRkaHdu3dLksxms6ZOnWpwRTCa2WzW/PnzNWDAAF28eFGTJ0/WqFGjJEnTpk3TI4880uj9p06dUkFBgQoKCuR0OpWZmel7uPmyZcu0ceNG2Ww2paWlyW63q3///s1+TgAAoPViBAtBxWq1Kj4+XiaTSfHx8bJarUaXBIOFhYVpwIABkqSOHTuqX79+t3xoZ3FxsRITExUaGqqIiAj17dtXZWVlKisrU9++fRUREaHQ0FAlJiaquLi4uU4DAAC0EQQsBJ2MjAwNGjSI0Stc58yZMzpx4oSGDBkiSdq8ebOSkpK0YMEC1dTUSJKcTqfCw8N9+9hsNjmdzpu2AwAA+BMBC0HHarVqzZo1jF6hkUuXLik7O1tPP/20OnbsqIcffliFhYXKz89XWFiYnn/+eaNLBAAAIGABCH5Xr15Vdna2kpKSFBsbK0nq0aOHzGazQkJClJ6ermPHjklqGJmqqKjw7et0OmWz2W7aDgAA4E8ELABBzev1auHCherXr58yMzN97ZWVlb6fi4qKFBkZKUmy2+0qKChQXV2dTp8+rfLycg0ePFiDBg1SeXm5Tp8+rbq6OhUUFMhutzf7+QAAgNaNVQQBBLX33ntP+fn5uueee5ScnCxJmjNnjv7whz/oo48+kiT16dNHy5YtkyRFRkZq/PjxSkhIkNls1qJFi2Q2myVJixYt0qOPPiqPx6PJkyf7QhkAAIC/ELAABLXhw4fr5MmT17WPHTv2pvtMnz5d06dPv+E+t9oPAADg38UUQQAAAADwEwIWAAAAAPgJAQsAAAAA/ISABQAAAAB+QsACAAAAAD8hYAEAAACAnxCwAAAAAMBPCFgAAAAA4CcELAAAAADwEwIWAAAAAPgJAQsAAAAA/ISABQAAAAB+QsACAAAAAD8hYAEAAACAnxCwAAAAAMBPCFgAAAAA4CcELAAAAADwEwIWAAAAAPgJAQsAAAAA/ISABQAAAAB+YgnUga9cuaIf/vCHqqurk8fjUVxcnLKzs3X69GnNmTNH1dXVGjBggH7+858rNDRUdXV1euqpp3T8+HF17dpVL774ou66665AlQcAAAAAfhewEazQ0FDl5uZq586dysvLU2lpqY4ePaqVK1dq2rRpKiwsVOfOnbVt2zZJ0tatW9W5c2cVFhZq2rRpWrlyZaBKAwAAAICACFjAMplM6tChgyTJ7XbL7XbLZDLp8OHDiouLkyRNmjRJxcXFkqR9+/Zp0qRJkqS4uDi988478nq9gSoPAAAAAPwuoPdgeTweJScna+TIkRo5cqQiIiLUuXNnWSwNMxPDw8PldDolSU6nU7169ZIkWSwWderUSVVVVYEsD0ALce7cOU2ZMkUJCQlKTExUbm6uJKm6ulqZmZmKjY1VZmamampqJEler1fLly+Xw+FQUlKSjh8/7jvWjh07FBsbq9jYWO3YscOQ8wHQcixYsEDR0dGaMGGCr23t2rV64IEHlJycrOTkZJWUlPheW79+vRwOh+Li4lRaWmpEyQAMFtCAZTablZ+fr5KSEpWVlenvf/97ID8OQCtlNps1f/587dq1S1u2bNHrr7+uU6dOKScnR9HR0dq7d6+io6OVk5MjSdq/f7/Ky8u1d+9ePfvss1qyZImkhkC2bt06vfHGG9q6davWrVvnC2UAcCOpqanasGHDde3Tpk1Tfn6+8vPzNXbsWEnSqVOnVFBQoIKCAm3YsEFLly6Vx+Np7pIBGKxZVhHs3LmzoqKidPToUdXW1srtdkuSKioqZLPZJEk2m03nzp2T1DCl8MKFC+rWrVtzlAcgyIWFhWnAgAGSpI4dO6pfv35yOp0qLi5WSkqKJCklJUVFRUWS5Gs3mUwaOnSoamtrVVlZqQMHDmjUqFHq2rWrunTpolGjRnGFGcAtjRgxQl26dGnSe4uLi5WYmKjQ0FBFRESob9++KisrC3CFAIJNwALW+f7W3voAABmgSURBVPPnVVtbK0n6/PPPdejQId19992KiorSnj17JDVM1bHb7ZIku93um66zZ88e3X///TKZTIEqD0ALdebMGZ04cUJDhgyRy+VSWFiYJKlnz55yuVySGqYch4eH+/a5Nh35y+02m803TRkAbsfmzZuVlJSkBQsW+EbC6WMASAEMWJWVlZo6daqSkpKUlpamkSNHaty4cZo3b542btwoh8Oh6upqpaenS5LS0tJUXV0th8OhjRs3au7cuYEqDUALdenSJWVnZ+vpp59Wx44dG71mMpm4KAOgWTz88MMqLCxUfn6+wsLC9PzzzxtdEoAg0qTnYP3zn/9UeHi4QkND9e677+rkyZNKSUlR586db7rPvffeq7y8vOvaIyIifEuzf9Edd9yhNWvW3EbpANqSq1evKjs7W0lJSYqNjZUkWa1WVVZWKiwsTJWVlerevbukhqvGFRUVvn2vTUe22Ww6cuSIr93pdOq73/1u854IgBavR48evp/T09P12GOPSbq+73E6nb5bIQC0HU0awZo5c6ZCQkL08ccfa9GiRTp37pyefPLJQNcGAJIaVgVcuHCh+vXrp8zMTF+73W73XcjJy8vTgw8+2Kjd6/Xq6NGj6tSpk8LCwjR69GgdOHBANTU1qqmp0YEDBzR69GhDzglAy1VZWen7uaioSJGRkZIa+p6CggLV1dXp9OnTKi8v1+DBg40qE4BBmjSCFRISIovFosLCQv3oRz/SlClTfDeWA0Cgvffee8rPz9c999yj5ORkSdKcOXOUlZWlWbNmadu2berdu7dWr14tSRo7dqxKSkrkcDh05513asWKFZKkrl276vHHH1daWpokacaMGeratasxJwWgRZgzZ46OHDmiqqoqjRkzRjNnztSRI0f00UcfSZL69OmjZcuWSZIiIyM1fvx4JSQkyGw2a9GiRTKbzUaWD8AAJm8Tnuabnp6ujIwMvfLKK3r55ZcVERGhCRMm6A9/+ENz1HhTqamp2r59u6E1AGjd6GcABBr9DNC6NGmK4M9+9jMdPXpUjz32mCIiInT69GlNnDgx0LUBAAAAQIvSpCmC/fv3109/+lPfdkREhLKysgJWFAAAAAC0RE0KWO+9957WrVunTz75RG63W16vVyaTScXFxYGuDwAAAABajCYFrIULF2rBggUaOHCgQkIC9ugsAAAAAGjRmhSwOnXqpLFjxwa6FgAAAABo0ZoUsKKiovTCCy8oNjZWoaGhvvYBAwYErDAAAAAAaGmaFLA+/PBDSdJf//pXX5vJZNKmTZsCUxUAAAAAtEBNClivvvpqoOsAAAAAgBbvlgErPz9fycnJ2rhx4w1fz8zMDEhRAAAAANAS3TJgffbZZ5KkS5cuNUsxAAAAANCS3TJgPfTQQ5KkJ554olmKASTJ5XJp6dKlWrx4saxWq9HlAAAAAE3WpIdaVVRUaMaMGYqOjlZ0dLRmzpypioqKQNeGNio3N1fHjh1jERUAAAC0OE0KWAsWLJDdbldpaalKS0s1btw4LViwINC1oQ1yuVzavXu3vF6vdu/eLZfLZXRJAAAAQJM1KWCdP39ekydPlsVikcViUWpqqs6fPx/o2tAG5ebmqr6+XpLk8XgYxQIAAECL0qSA1bVrV+Xn58vj8cjj8Sg/P19du3YNdG1og4qKiuR2uyVJbrdbhYWFBlcEAAAANF2TAtaKFSv0xz/+UaNGjdLo0aO1Z88ePf/884GuDW1QTEyMLJaGtVcsFoscDofBFQEAAABN16QHDa9Zs0YvvPCCunTpIkmqrq7WCy+8oJ/97GcBLQ5tT0ZGhnbv3i1JMpvNmjp1qsEVAQAAAE3XpBGskydP+sKV1DBl8MSJEwErCm2X1WpVfHy8TCaT4uPjWaYdAAAALUqTAlZ9fb1qamp829XV1fJ4PAErCm1bRkaGBg0axOgVAAAAWpwmTRH88Y9/rP/8z/9UfHy8JGn37t167LHHAloY2i6r1ao1a9YYXQYAAABw25oUsFJSUjRw4EAdPnxYkrRu3Tr1798/oIUBAAAAQEvTpIAlSf379ydUAQAAAMAtNOkeLAAAAADAVyNgAQh6CxYsUHR0tCZMmOBrW7t2rR544AElJycrOTlZJSUlvtfWr18vh8OhuLg4lZaW+tr379+vuLg4ORwO5eTkNOs5AACAtqHJUwQBwCipqan60Y9+pP/3//5fo/Zp06bpkUceadR26tQpFRQUqKCgQE6nU5mZmdqzZ48kadmyZdq4caNsNpvS0tJkt9uZ+gwAAPyKgAUg6I0YMUJnzpxp0nuLi4uVmJio0NBQRUREqG/fviorK5Mk9e3bVxEREZKkxMREFRcXE7AAAIBfMUUQQIu1efNmJSUlacGCBb5n9TmdToWHh/veY7PZ5HQ6b9oOAADgTwQsAC3Sww8/rMLCQuXn5yssLEzPP/+80SUBAAAQsAC0TD169JDZbFZISIjS09N17NgxSQ0jUxUVFb73OZ1O2Wy2m7YDAAD4EwELQcflcik7O1sul8voUhDEKisrfT8XFRUpMjJSkmS321VQUKC6ujqdPn1a5eXlGjx4sAYNGqTy8nKdPn1adXV1KigokN1uN6p8AADQSrHIBYJObm6ujh07pk2bNmn27NlGl4MgMGfOHB05ckRVVVUaM2aMZs6cqSNHjuijjz6SJPXp00fLli2TJEVGRmr8+PFKSEiQ2WzWokWLZDabJUmLFi3So48+Ko/Ho8mTJ/tCGQAAgL+YvF6vNxAHPnfunJ566im5XC6ZTCZ9//vfV0ZGhqqrqzV79mydPXtWffr00erVq9WlSxd5vV4999xzKikpUfv27fX8889rwIABt/yM1NRUbd++PRDlwyAul0sPP/yw6urqdMcdd+j111+X1Wo1uiy0YfQzAAKNfgZoXQI2RdBsNmv+/PnatWuXtmzZotdff12nTp1STk6OoqOjtXfvXkVHR/se9rl//36Vl5dr7969evbZZ7VkyZJAlYYglpubq/r6ekmSx+PRpk2bDK4IAAAAaLqABaywsDDfCFTHjh3Vr18/OZ1OFRcXKyUlRZKUkpKioqIiSfK1m0wmDR06VLW1tY3usUDbUFRUJLfbLUlyu90qLCw0uCIAQFu2YMECRUdHa8KECb626upqZWZmKjY2VpmZmb7HRHi9Xi1fvlwOh0NJSUk6fvy4UWUDMFCzLHJx5swZnThxQkOGDJHL5VJYWJgkqWfPnr6FDL78jJrw8HCeUdMGxcTE+O6XMZvNcjgcBlcEAGjLUlNTtWHDhkZtzMYBcCsBD1iXLl1Sdna2nn76aXXs2LHRayaTSSaTKdAloAXJyMjQtdsCvV6vpk6danBFAIC2bMSIEerSpUujNmbjALiVgAasq1evKjs7W0lJSYqNjZUkWa1WX2dTWVmp7t27S7r+2TUVFRU8owYAAAQdZuMAuJWABSyv16uFCxeqX79+yszM9LXb7Xbl5eVJkvLy8vTggw82avd6vTp69Kg6derk67zQduTm5iokpOGfZUhICItcAACCGrNxAHxZwALWe++9p/z8fB0+fFjJyclKTk5WSUmJsrKydPDgQcXGxurQoUPKysqSJI0dO1YRERFyOBx65plntHjx4kCVhiDGIhcAgGDHbBwAtxKwBw0PHz5cJ0+evOFrubm517WZTCZCFRQTE6Ndu3bJ7XbLYrGwyAUAIOhcm3WTlZV13Wyc1157TYmJifrwww+ZjQO0Uc2yiiDQVBkZGb4pgmazmUUuAACGmjNnjh566CH94x//0JgxY7R161Zm4wC4pYCNYAFfh9VqVXx8vN566y3Fx8fLarUaXRIAoA1btWrVDduZjQPgZghYCDoZGRkqLy9n9AoAAAAtDgELQcdqtWrNmjVGlwEAAADcNu7BAgAAAAA/IWABAAAAgJ8QsAAAAADATwhYAAAAAOAnBCwAAAAA8BMCFgAAAAD4CQELAAAAAPyEgAUAAAAAfkLAAgAAAAA/IWABAAAAgJ8QsAAEvQULFig6OloTJkzwtVVXVyszM1OxsbHKzMxUTU2NJMnr9Wr58uVyOBxKSkrS8ePHffvs2LFDsbGxio2N1Y4dO5r9PAAAQOtHwAIQ9FJTU7Vhw4ZGbTk5OYqOjtbevXsVHR2tnJwcSdL+/ftVXl6uvXv36tlnn9WSJUskNQSydevW6Y033tDWrVu1bt06XygDAADwFwIWgKA3YsQIdenSpVFbcXGxUlJSJEkpKSkqKipq1G4ymTR06FDV1taqsrJSBw4c0KhRo9S1a1d16dJFo0aNUmlpabOfCwAAaN0IWABaJJfLpbCwMElSz5495XK5JElOp1Ph4eG+94WHh8vpdF7XbrPZ5HQ6m7doAADQ6hGwALR4JpNJJpPJ6DIAAAAIWABaJqvVqsrKSklSZWWlunfvLqlhZKqiosL3voqKCtlstuvanU6nbDZb8xYNAGgzXC6XsrOzfTMs0HYQsAC0SHa7XXl5eZKkvLw8Pfjgg43avV6vjh49qk6dOiksLEyjR4/WgQMHVFNTo5qaGh04cECjR4828hQAAK1Ybm6ujh07pk2bNhldCpqZxegCAOCrzJkzR0eOHFFVVZXGjBmjmTNnKisrS7NmzdK2bdvUu3dvrV69WpI0duxYlZSUyOFw6M4779SKFSskSV27dtXjjz+utLQ0SdKMGTPUtWtXw84JANB6uVwu7d69W16vV7t379bUqVNltVqNLgvNhIAFIOitWrXqhu25ubnXtZlMJi1evPiG709LS/MFLAAAAiU3N1f19fWSJI/Ho02bNmn27NkGV4XmwhRBAAAAwI+KiorkdrslSW63W4WFhQZXhOZEwAIAAAD8KCYmRhZLw0Qxi8Uih8NhcEVoTgQsAAAAwI8yMjIUEtLwNdtsNmvq1KkGV4TmRMACAAAA/MhqtSo+Pl4mk0nx8fEscNHGsMgFAAAA4GcZGRkqLy9n9KoNImABAAAAfma1WrVmzRqjy4ABmCIIAAAAAH5CwAIAAAAAPyFgAQAAAICfELAAAAAAwE8IWAAAAADgJwELWAsWLFB0dLQmTJjga6uurlZmZqZiY2OVmZmpmpoaSZLX69Xy5cvlcDiUlJSk48ePB6osAAAAAAiYgAWs1NRUbdiwoVFbTk6OoqOjtXfvXkVHRysnJ0eStH//fpWXl2vv3r169tlntWTJkkCVBQAA4Bd2u11JSUlKTk5WamqqpJtfTAbQdgQsYI0YMUJdunRp1FZcXKyUlBRJUkpKioqKihq1m0wmDR06VLW1taqsrAxUaQAAAH6Rm5ur/Px8bd++XdLNLyYDaDua9R4sl8ulsLAwSVLPnj3lcrkkSU6nU+Hh4b73hYeHy+l0NmdpAAAA/7abXUwG0HYYtsiFyWSSyWQy6uMBAAD+bY888ohSU1O1ZcsWSTe/mAyg7WjWgGW1Wn1T/yorK9W9e3dJks1mU0VFhe99FRUVstlszVkagojL5VJ2djb/pwQACGq/+93vtGPHDv3mN7/R5s2b9ec//7nR61xMBtqmZg1YdrtdeXl5kqS8vDw9+OCDjdq9Xq+OHj2qTp06+a7+oO1Zv369ysrKmLcOAAhq1y4GW61WORwOlZWV3fRiMtoeLhi3XQELWHPmzNFDDz2kf/zjHxozZoy2bt2qrKwsHTx4ULGxsTp06JCysrIkSWPHjlVERIQcDoeeeeYZLV68OFBlIci5XC7ffPXCwkI6JQBAULp8+bIuXrzo+/ngwYOKjIy86cVktD1cMG67LIE68KpVq27Ynpube12byWQiVEFSQ2dUX18vSaqvr1dOTo4WLFhgcFUAADTmcrk0Y8YMSZLH49GECRM0ZswYDRo0SLNmzdK2bdvUu3dvrV692uBKYYQvXzDOysqS1Wo1uCo0l4AFLODrKC4ubrRdVFREwAIABJ2IiAjt3LnzuvZu3brd8GIy2hYuGLdthq0iCAAAALRGN7pgjLaDgIWg0rt371tuAwAABLsvrx7JapJtC1MEEVT+9a9/3XIb+DK73a4OHTooJCREZrNZ27dvV3V1tWbPnq2zZ8+qT58+Wr16tbp06SKv16vnnntOJSUlat++vZ5//nkNGDDA6FMAALQyDz74oPbs2dNoG20HI1gIKj169LjlNnAjubm5ys/P1/bt2yVJOTk5io6O1t69exUdHe1bwWn//v0qLy/X3r179eyzz2rJkiUGVg0AaK2ysrIUEtLwNTskJMS3cjbaBgIWgsq5c+duuQ00RXFxsVJSUiRJKSkpvrnv19pNJpOGDh2q2tpa3/NqAADwl2vPRpMkh8PBCoJtDAELQYU5y/g6HnnkEaWmpmrLli2SGpbHvfaw8p49e/qep+Z0OhUeHu7bLzw8XE6ns/kLBgC0ellZWRo8eDCjV20Q92AhqDBnGbfrd7/7nWw2m1wulzIzM9WvX79Gr5tMJoI6AKDZVVVV6W9/+5uqqqoYwWpjGMFCUElPT7/lNvBlNptN0v9NxygrK5PVavVN/ausrFT37t19762oqPDtW1FR4dsfAAB/Wr58uS5duqTly5cbXQqaGQELQWXnzp2+0QaTyaS33nrL4IoQzC5fvqyLFy/6fj548KAiIyNlt9uVl5cnScrLy/ONhF5r93q9Onr0qDp16uSbSggAgL+cOnVK5eXlkqTy8nKdOnXK2ILQrJgiiKBSVFQkr9crSfJ6vSosLNTs2bMNrgrByuVyacaMGZIkj8ejCRMmaMyYMRo0aJBmzZqlbdu2qXfv3lq9erUkaezYsSopKZHD4dCdd96pFStWGFk+AKCV+vKo1fLly/U///M/xhSDZkfAQlCJiYnRrl275Ha7ZbFYfCvwADcSERGhnTt3XtferVs35ebmXtduMpm0ePHi5igNANCGXRu9utk2WjemCCKoZGRk+J4bYTabNXXqVIMrAgAAuD1fnn7O/b5tCwELQcVqtWrcuHGSpO9973usugMAAFqca/cHX3PhwgWDKoERCFgIOtfuwQIAAGiJLl++fMtttG4ELAQVl8ult99+W5L09ttv+x4QCwAA0FJ07Njxltto3QhYCCq5ubmqr6+X1LAq3KZNmwyuCAAA4PYsWbKk0fbSpUuNKQSGIGAhqBQVFcntdkuS3G63CgsLDa4IAADg9gwfPlwdOnSQJHXo0EHf+c53DK4IzYmAhaASExMji6Xh6QEs0w4AAFqqe++9V5J03333GVwJmhsBC0ElIyPDN0Wwvr6eZdoBAECL43K59MEHH0iS3n//fe4pb2MIWAAAAIAf5eTkNLpgnJOTY3BFaE4ELASV3NxcmUwmSZLJZGKRCwAA0OIUFxffchutGwELQaWoqEgej0dSwyqCLHIBAABammvfZW62jdaNgIWgEhMTI7PZLEkym80scgEAAFqca99lbraN1o2AhaCSkZEhr9crSfJ6vSxyAQAAWpxrj5y52TZaNwIWgs4XAxYAAEBLwwhW20bAQlDJzc1ttM0iFwAAoKX58kViLhq3LQQsBJXCwsJGI1h79+41uCIAAIDbc21F5Jtto3UjYCGo2Gy2W24DgD+4XC5lZ2fz8E8AAdG+fftbbqN1I2AhqJw9e/aW2wDgDzk5OSorK+PhnwAC4tKlS7fcRutGwEJQYdUdAIHmcrl8z9grLCxkFAuA3zFFsG0jYCGocFMogEDLyclRfX29JKm+vp5RLAB+x/eZto2AhaDCFR8AgVZUVHTLbQAA/h0ELASVdu3a3XIbAP5dHo/nltsAAPw7LEYX8EX79+/Xc889p/r6eqWnpysrK8voklqsPXv2aNeuXUaXcds6derU6H6ITp066Sc/+YmBFX09CQkJiouLM7oM3AD9DIBAo58B2ragCVgej0fLli3Txo0bZbPZlJaWJrvdrv79+xtSz9q1a3Xq1ClDPtsfzp8/r/Pnzxtdxm27dl/ENZcvX26R/zu89tprLTLgXtO/f3/NnDnT6DL8Ltj6GQCtD/0MgKAJWGVlZerbt68iIiIkSYmJiSouLjasQzpy5IhOnz5tyGfj/3z22WdGl/C1XLp0qUX/+2mJ4bwpgq2f2bNnj9asWWPIZ/vDlStXWs1Kn9/73veMLuG2WSwW3XHHHUaX8bVlZ2e3ypF++hn/aU19jEQ/YwSj+pmgCVhOp1Ph4eG+bZvNprKyMsPqsdlsLfpLptvt1tWrV40u42urr69XSEjLvUWwXbt2sliC5s/rtrXWBzwHWz8DoPWhnwHQcr8BBtjKlSuNLgFAKxcXF9cqr+AHuxtdRX777bebvQ6gOdDPGGPVqlV666235PV6ZTKZNHHiRM2ePdvostBMgmaIwGazqaKiwrftdDpb7VV0AMagn4EkdevWrdG21Wo1qBK0RvQzkKSMjAzfSsihoaGaOnWqwRWhOQVNwBo0aJDKy8t1+vRp1dXVqaCgQHa73eiyALQi9DOQpB07djTafvPNNw2qBK0R/Qykhgs38fHxMplMio+P50JOGxM0UwQtFosWLVqkRx99VB6PR5MnT1ZkZKTRZQFoRehncE23bt1UVVXFlx74Hf0MrsnIyFB5eTmjV22Qyev1eo0u4utKTU3V9u3bjS4DQCtGPwMg0OhngNYlaKYIAgAAAEBLR8ACAAAAAD8hYAEAAACAnxCwAAAAAMBPCFgAAAAA4CcELAAAAADwkxa9THtUVJT69OljdBkAblO3bt3029/+1ugymoR+BmiZ6GcABNrN+pkWHbAAAAAAIJgwRRAAAAAA/ISABQAAAAB+QsACAAAAAD8hYAEAAACAnxCwAAAAAMBP/j9avO7q4JX69gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Распределение пользователей в зависимости от количества внутренней валюты (coins)\n",
    "fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))\n",
    "\n",
    "sns.boxplot(y=users_test.coins, ax = axes[0])\n",
    "axes[0].set(title='Test group')\n",
    "\n",
    "sns.boxplot(y=users_control_1.coins, ax = axes[1])\n",
    "axes[1].set(title='1 control group', ylabel='')\n",
    "\n",
    "sns.boxplot(y=users_control_2.coins, ax = axes[2])\n",
    "axes[2].set(title='2 control group', ylabel='')\n",
    "\n",
    "sns.despine()\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверим наш экстремальный выброс в 1 контрольной группе:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>age</th>\n",
       "      <th>attraction_coeff</th>\n",
       "      <th>coins</th>\n",
       "      <th>country</th>\n",
       "      <th>visit_days</th>\n",
       "      <th>gender</th>\n",
       "      <th>age_filter_start</th>\n",
       "      <th>age_filter_end</th>\n",
       "      <th>views_count</th>\n",
       "      <th>was_premium</th>\n",
       "      <th>is_premium</th>\n",
       "      <th>total_revenue</th>\n",
       "      <th>group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1427</th>\n",
       "      <td>892333905</td>\n",
       "      <td>22</td>\n",
       "      <td>175</td>\n",
       "      <td>3608.0</td>\n",
       "      <td>Italy</td>\n",
       "      <td>1,2,3,4,5,6,7,15</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>25</td>\n",
       "      <td>575</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>486330</td>\n",
       "      <td>control_1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            uid  age  attraction_coeff   coins country        visit_days  \\\n",
       "1427  892333905   22               175  3608.0   Italy  1,2,3,4,5,6,7,15   \n",
       "\n",
       "      gender  age_filter_start  age_filter_end  views_count  was_premium  \\\n",
       "1427       1                19              25          575          1.0   \n",
       "\n",
       "      is_premium  total_revenue      group  \n",
       "1427         1.0         486330  control_1  "
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_control_1.query('coins == coins.max()')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>country</th>\n",
       "      <th>joined_at</th>\n",
       "      <th>paid_at</th>\n",
       "      <th>revenue</th>\n",
       "      <th>payment_id</th>\n",
       "      <th>from_page</th>\n",
       "      <th>product_type</th>\n",
       "      <th>group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>892333905</td>\n",
       "      <td>Italy</td>\n",
       "      <td>2017-10-16 01:38:00</td>\n",
       "      <td>2017-10-16 13:10:00</td>\n",
       "      <td>63037.0</td>\n",
       "      <td>147</td>\n",
       "      <td>no_sympathy</td>\n",
       "      <td>premium_no_trial</td>\n",
       "      <td>control_2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           uid country           joined_at             paid_at  revenue  \\\n",
       "143  892333905   Italy 2017-10-16 01:38:00 2017-10-16 13:10:00  63037.0   \n",
       "\n",
       "     payment_id    from_page      product_type      group  \n",
       "143         147  no_sympathy  premium_no_trial  control_2  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transactions_control_1.query('uid == 892333905')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Примем решение не удалять данного пользователя, он не должен испортить результат нашего анализа"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>W</th>\n",
       "      <th>pval</th>\n",
       "      <th>equal_var</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>levene</th>\n",
       "      <td>0.547366</td>\n",
       "      <td>0.57887</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               W     pval  equal_var\n",
       "levene  0.547366  0.57887       True"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Проверка на гомоскедастичность\n",
    "pg.homoscedasticity(data = users_merged, dv = 'coins', group = 'group')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Т.к. у нас 3 группы с количественными переменными, сравним их распределения при помощи критерия Краскела-Уоллиса:\n",
    "\n",
    "- $H_0$: нет статистически значимых различий между распределениями количества внутренней валюты в тестовой и контрольных группах (p-value > 0.05)\n",
    "- $H_1$: есть статистически значимые различия между распределениями количества внутренней валюты в тестовой и контрольных группах"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KruskalResult(statistic=1.2617925744049447, pvalue=0.5321146596513002)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kruskal(users_merged.query('group == \"test\"').coins, users_merged.query('group == \"control_1\"').coins, users_merged.query('group == \"control_2\"').coins)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Наше p-value > 0.05, мы не можем отвергнуть нулевую гипотезу, т.е. различий между распределениями количества внутренней валюты пользователей разных групп нет, что говорит о репрезентативности выборок по данному признаку."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAEYCAYAAABBWFftAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzde1hU1eLG8S8gICAJpKiEl0qNTE0UD5XWOSLeuRhe0sq0rE4qaYYV6dHCNLNSM7xlVl6y0hBFUrLUNE8qXistPZWaiRcUEeWSXIb5/eHj/ETuOAwMvp/n6XmcmbXXWnvrvO01a++9bIxGoxERERERERG5YbZV3QEREREREZGaQgMsERERERERM9EAS0RERERExEw0wBIRERERETETDbBERERERETMRAMsERERERERM9EAS0RE5CaWmJjIQw89VNXdEJGbwM2SNxpgiQC+vr6m/3x8fGjbtq3p9dq1a8td35AhQ/jyyy8roaciUp18+umnhIWF0bp1ayIjI6ukD3fddRfHjx+vkrZFxDJycnIYP348Xbp0wdfXl9DQULZu3WrxfihvyqZWVXdApDrYv3+/6c8BAQFMmTKFBx54oAp7VFBeXh61aunrKlLdeHp6MnLkSLZt20Z2dnZVd6dIlswPZZVI5cjLy6NRo0YsW7YMLy8vtm7dygsvvEB8fDze3t5V3T0T5c0VmsESKUF+fj4LFy4kMDAQf39/xowZQ1paGgDZ2dmMGzcOf39//Pz86NevHykpKcyaNYs9e/YwefJkfH19mTx5cpF1r1mzhi5duuDv78/cuXMJCAhg+/btAERHRzN69GjGjRtH+/btWb16NcnJyTz33HP84x//oFu3bqxcudJUV2RkJLNmzTK9vn4KPiAggA8++IDevXvTsWNHXn311Wp7MihiTbp3705gYCBubm5lKr9y5Up69eqFr68vvXv35pdffgHgyJEjDBkyBD8/P/r06cOmTZtM20RGRhIVFcWzzz6Lr68vAwYM4K+//gLgscceAyA0NBRfX1/Wr19v+v4vXLiQTp068eqrr5KTk8PUqVPp3LkznTt3ZurUqeTk5JSpz//973/p0aMHHTp04PXXX+fxxx83zdDHxsYyaNAg3nzzTfz9/YmOjiY9PZ2XX36Z++67jy5dujBv3jzy8/OBK9k2btw4U91JSUncdddd5OXlAVdm/2fMmEH//v1p3749I0aMMGWuyM3M2dmZ559/Hm9vb2xtbenSpQve3t6mDCmK8qbq8kYDLJESLFu2jI0bN/Lpp5+ybds26tataxowrV69moyMDLZs2UJiYiJRUVHUrl2bsWPH4ufnx6RJk9i/fz+TJk0qVO8ff/xBVFQU77zzDtu2bSMjI4Pk5OQCZTZt2kTPnj3Zs2cPwcHBvPjiizRs2JBt27bx/vvvM3PmTHbs2FHmfYmPj+ejjz7i22+/5dixY8ybN+/GDo6IlEtCQgLR0dFMnz6dffv2MX/+fNzc3MjNzeW5556jU6dObN++nf/85z+MGzeOo0ePmrZdv3494eHh7N69myZNmph+UFm+fDkAcXFx7N+/n969ewOQkpLCxYsX+e6773jjjTeYP38+P/30E3Fxcaxdu5YDBw6UKQNSU1MZPXo0ERERJCYmcvvttxeY8Qf4+eefady4MT/88AMjRozgjTfeID09nY0bN7Js2TLi4uJYtWpVmY/TmjVrePPNN/nvf/9LrVq1mDJlSpm3FblZpKSk8Oeff9K8efMiP1felE1l5Y0GWCIl+OKLLxg7diwNGzbEwcGB8PBwNmzYYJqWTktL4/jx49jZ2dG6dWvq1KlTpnq//vprunTpgp+fHw4ODowePRobG5sCZdq1a0dgYCC2trZcuHCBffv2MW7cOBwdHbn77rsZMGAAcXFxZd6Xxx57jEaNGuHm5saIESNYt25duY6FiNyYmJgYnn76adq2bYuNjQ1Nmzbltttu46effiIrK4tnn30WBwcH7r//frp06VLgOxoYGEjbtm2pVasWISEhHDp0qMS2bG1tGT16NA4ODtSuXZv4+HhGjRrFrbfeioeHB6NGjSrT/aXff/89LVq0oHv37tSqVYsnnniCevXqFSjj6enJkCFDqFWrFvb29qxfv56IiAjq1KmDt7c3Tz75ZLnuZQ0NDaVly5Y4OzszZswYvv76awwGQ5m3F6npcnNzGTduHA8//DB33nlnkWWUN2VTWXlTPS9cFKkmTp06xahRo7C1/f/fImxtbTl//jyhoaGcOXOGF198kUuXLhESEsLYsWOxt7cvtd6zZ8/SsGFD02snJ6dClxhd+/nZs2epW7dugQGcl5cXBw8eLPO+NGrUqMC2Z8+eLfO2InLjTp8+TZMmTQq9fzUPrs0ZLy+vArPa155k1K5dm6ysrBLbcnd3x9HRsUAbXl5eBeovSwZcn1U2NjYFXkPBrLpw4QK5ubmF2rp+hr4k12dVbm4uFy5cKHSiJXIzys/P5+WXX8be3p6JEycWW055UzaVlTeawRIpQcOGDfnwww/Zs2eP6b8DBw7QoEED7O3tCQ8PZ/369XzxxRds2bKFNWvWlKleT0/PAgFw+fLlQtf9Xjuj5enpycWLF8nIyDC9d/r0aRo0aABcGaBdvnzZ9FlKSkqhNk+fPm3686lTp/D09CxTX0XEPBo1amS6l+Fanp6enDlzxnTfABT8flfE9TPinp6enDp1qkD9ZcmA+vXrF8gqo9HImTNnim3L3d0de3v7Qm1VNKtOnz6Nvb097u7upfZVpKYzGo1MmDCBlJQUoqOjS/xBV3lTtXmjAZZICQYPHsx7773HyZMngSvXB2/cuBGAnTt38r///Q+DwUCdOnWoVauW6RehevXqceLEiWLr7dGjB5s3b2bfvn3k5OQQHR2N0WgstnyjRo3w9fVl5syZZGdnc/jwYWJiYggJCQHg7rvvZuvWraSlpXHu3DmWLFlSqI7PPvuMM2fOkJaWxoIFC0zXTotIxeXl5ZGdnU1+fj4Gg4Hs7GzTDdTX69+/Px9//DEHDx7EaDRy/PhxTp48Sdu2balduzaLFi0iNzeXxMRENm/eXObvaGl5A9CnTx/mz59PamoqqampzJ07l+Dg4FLr/uc//8n//vc/Nm7cSF5eHsuXLy/yJOUqOzs7evbsyaxZs8jIyODkyZN88sknBbJq9+7dnDp1ivT0dD744INCdaxdu5Y//viDv//+m9mzZ9OjRw/s7OxK7atITffaa69x5MgRFixYQO3atUssq7yp2rzRAEukBE888QQBAQE89dRT+Pr6MnDgQH7++Wfgyi8ho0ePpkOHDvTu3Zt//OMfhIaGmrbbsGEDHTt2LPKGyRYtWjBx4kRefPFFHnzwQZydnfHw8MDBwaHYvsycOZOTJ0/y4IMPEh4ezvPPP296lHxoaCg+Pj6mvhYVlEFBQTz11FMEBgbSpEkTRowYYY5DJHJTmz9/Pm3btmXhwoWsXbuWtm3bMn/+/CLL9urVi+eee46IiAjat2/PqFGjuHjxIg4ODixYsIDvv/+e++67j6ioKN5+++1i7624Xnh4OJGRkfj5+bF+/foiy4wcOZLWrVsTEhJCSEgI99xzDyNHjiy1bg8PD2bPns0777yDv78/f/zxB61bty7xl/OJEyfi5OREYGAgjz76KEFBQfTr1w+ATp060bt3b0JCQggLC6NLly6Ftg8NDSUyMpJOnTqRk5PDhAkTynQcRGqykydPsmLFCg4dOkTnzp1LXatTeVO1eWNjLOlncxGxiMzMTDp27MiGDRto3Lix2euvjmt7iYj1yc/P56GHHuLdd9/lvvvuM3v9Q4YMISQkhAEDBpi9bhGxLtacN5rBEqkimzdv5u+//yYrK4vp06fTsmXLarVYoIgIwLZt27h06RI5OTksWLAAuPKUUxERc6speaOnCIpUkU2bNvHyyy9jNBpp3bo1M2fOLHSjqIhIVfvxxx8ZN24cOTk5NG/enLlz55Z6/4eISEXUlLzRJYIiIiIiIiJmoksERUREREREzEQDLKn2hg8fXtVdEJGbiDJHRCxFeVMzaYAl1d6FCxequgsichNR5oiIpShvaiYNsERERERERMxEAywREREREREz0QBLRERERETETDTAEhERERERMRMNsERERERERMxEAywREREREREz0QBLRERERETETDTAEhERERERMRMNsETELAz5xqrugojcRJQ5ImIp5c2bWpXUDxGzmp5wiOT07KruhhSjgasjr/S6u6q7IWI2ypzqTZkjNYnypnqrSN5ogCVWITk9m1Npf1d1N0TkJqHMERFLUd7UPLpEUERERERExEyq7QArKSmJoKCgAu9FR0fz0UcflbjdgQMHmDJlCgCJiYns27ev3G0HBASQmppa6P2YmBiCg4MJDg4mKCiIjRs3AhAbG0tycnKp9Za13I148cUXCQ4OZvHixUV+HhoaytixYyu1DxMmTOCPP/6o1DZEzEl5UzHKG5HyU95UjPJGrEmNu0SwTZs2tGnTBoBdu3bh7OxM+/btb7jeM2fOsGDBAlavXo2rqyuZmZmmkFq9ejUtWrSgQYMGJdZR1nIVde7cOQ4cOMC3335b5OdHjhwhPz+fPXv2kJWVhbOzs9n7YDAYmDp1qtnrFamOlDfKGxFLUd4ob8R6VNsZrNIMGTKEd955h/79+9OjRw/27NkDXPlV59///jdJSUl88cUXLF68mNDQUPbs2UNqairPP/88/fr1o1+/fuzduxeACxcu8NRTT9GnTx8mTJiA0Vj4SSHnz5/HxcXF9KV1cXGhcePGfP311xw8eJBx48YRGhrK5cuXmTNnDv369SMoKIiJEydiNBqLLHfw4EEef/xxwsLCGD58OGfPngVg6dKl9O7dm+Dg4CJ/jcnOzubVV18lODiYvn37snPnTgCeeuopkpOTTft7va+++oqQkBA6d+7Mpk2bChzLN998k7CwMHr16sXPP/9MeHg43bt3Z9asWaZycXFx9O/fn9DQUCZNmoTBYADA19eXt956i5CQEPbv38+QIUM4cOAAAN9//z0PP/wwISEhDB06FICff/6ZRx55hL59+zJo0CCOHj1azr99EctS3ihvRCxFeaO8Eetn1TNYBoOBmJgYtm7dypw5cwpMG3t7ezNo0CCcnZ0ZPnw4ABEREQwdOhQ/Pz9OnTrF8OHDSUhIYO7cubRv357w8HC2bNlCTExMobZ8fHyoV68eXbt25f7776dbt24EBATQs2dPli9fzssvv2z6Zenxxx8nPDwcgJdeeonvvvuuULnc3FymTJnCvHnz8PDwYP369cyaNYtp06axcOFCNm/ejIODA5cuXSrUl+XLlwMQHx/PkSNHGD58OBs2bGD+/Pk899xzxMXFFXm81q9fzyeffMLRo0f59NNPCQ4ONn1mb29PbGwsS5YsYeTIkcTGxuLm5kZgYCDDhg0jNTWVhIQEPv/8c+zt7Xn99deJj4+nb9++ZGVl0bZtWyIjIwu0l5qaysSJE/n0009p3LgxaWlpANxxxx0sX76cWrVqsX37dmbNmkV0dHRZ/9pFqoTyRnkjYinKG+WNWLdqO8CysbEp9f1u3boBcM8993Dy5MlS69y+fXuBa2czMjLIzMxk9+7dzJkzB4B//etf1K1bt9C2dnZ2LFq0iAMHDrBjxw6mTZvGL7/8wvPPP1+obGJiIosWLeLy5cukpaXRokULAgICCpQ5duwYv/32G08++SQA+fn51K9fH4C77rqLcePG0bVrVwIDAwvVv3fvXh5//HEA7rzzTry8vDh27Bh16tQpdt8PHDiAu7s7Xl5eNGjQgPHjx5OWloabmxuAqX8tW7akRYsWeHp6AtC4cWPOnDnD3r17OXjwIP379wfg8uXL3HrrraZj06NHj0Jt/vjjj/j5+dG4cWMAU1vp6em88sorHD9+HBsbG3Jzc4vtt4glKG+UNyKWorxR3kjNV20HWG5ubly8eLHAexcvXsTb29v02sHBAQBbW1vTdG5J8vPzWblyJY6OjhXqk42NDW3btqVt27Y88MADjB8/vlAAZWdnExUVxapVq2jUqBHR0dFkZxde28BoNNKiRQtWrFhR6LOFCxeye/duvvvuOxYsWEB8fDy1at3YX9W6des4duyYKWgyMjL45ptvGDhwIFDwWF7989XXeXl5GI1GHn74YSIiIgrV7ejoiJ2dXZn7Mnv2bPz9/Zk7dy5JSUk88cQTN7JrIjdMeaO8EbEU5Y3yRmq+ansPlouLC/Xr12fHjh0ApKWlsW3bNjp06FCuOjIzM02vO3fuzLJly0yvDx06BEDHjh2Jj48HYOvWrYWCDyA5OZlffvnF9Prw4cN4eXkVaudq2Li7u5OZmcmGDRuK7M/tt99Oamoq+/fvByA3N5fff/+d/Px8Tp8+zX333ce4ceNIT08nKyurQF/8/PxM/T127BinT5/mjjvuKPY45Ofnk5CQwNq1a9m8eTObN29m3rx5fPXVVyUev2vdf//9bNiwgfPnzwNX/j5K+1WtXbt27NmzhxMnTpi2gSu/8Fy9EXb16tVl7oNIZVHeKG9ELEV5o7yRmq/azmABvP3220RFRfHWW28BMGrUKJo0aVLm7bt06cLo0aPZtGkTEydOZMKECUyePJng4GAMBgN+fn5MnjyZUaNGERERQZ8+ffD19TUFy7Xy8vKYPn06Z8+exdHREQ8PD6KiogB4+OGHee2116hduzYrVqxgwIABBAUFUa9ePdN1y0WVe//995kyZQrp6ekYDAaGDh1Ks2bNeOmll8jIyMBoNPLEE09wyy23FOjLo48+yuuvv05wcDB2dnZMmzatwK8y19uzZw8NGjQo8HSfjh07cuTIEdONp6Vp3rw5L7zwAk899RT5+fnY29szadIkbrvttmK38fDwYPLkyTz//PPk5+dz66238sknn/D0008TGRnJ/Pnz+ec//1mm9kUqm/JGeSNiKcob5Y3UbDbGoh4pI1KNhIWF0WzQJK1yXo15uTkxc2C7qu6GiFkoc6o/ZY7UFMqb6q8ieVOtZ7BErmrV0BVP14pdW35VVo6BjOw8M/VIrtXgBv9uRKqbimaOcsYylDlSk5jjHOcqZZD5VSRvNMASq/D0Q3fecB2GfCN2tkU/vUlunI6v1CQVzRx9DyxHx1pqCnOc41yl70XlKO9x1QBLrML0hEMkpxd+WlFZNXB15JVed5uxR3I9BbrUJBXJHOWMZSlzpKa40XOcq5RBlae8eaMBlliF5PRsXZ8sIhajzBERS1He1DwlPqY9KSmJoKCgAu9FR0fz0UcflVjpgQMHmDJlCnBlUbp9+/aVu2MBAQGkpqaW+P7BgwcJCAjg119/ZdOmTSxcuLDc7RQlMTGRf//732ap63pTp05l9+7dwJUn98ycOZPu3bsTGhpKaGgo8+fPN1tbCxYsKPazgIAAgoODCQ0NJTg4mI0bNxZb9plnnuHSpUtcunTJtMp6aQYNGlRqmbFjx/Lnn3+WqT65OShzzE+Z8/+UOXIt5Y35KW/+n/Lm5lYp62C1adOG//znPwDs2rXLtBaCOR0+fJjRo0fz3nvv0apVK7p27cqzzz5r9nbM6cKFC/z000907NgRgPfee4+zZ88SHx9PXFwcy5cvJy+v8I2JRqOR/Pz8crf3wQcflPj5kiVLiIuLY/bs2UydOrXYdj/88ENuueUWLl26xOeff16mtr/44otSywwePJhFixaVqT6RkihziqbMKUiZI+agvCma8qYg5c3N7YYuERwyZAht27YlMTGR9PR0pk6dip+fH4mJiXz88cdMnDiRL774AltbW9auXcvEiRO54447eO211zh16hQA48ePp0OHDly4cIGIiAiSk5Np164dJT09/ujRo7zyyiu8/fbbtG3bFoDY2FgOHjzIpEmTiIyMpE6dOhw8eJBz587x0ksv0bNnT/Lz85k8eTI7d+6kUaNG1KpVi379+tGzZ0++//573nzzTZycnAos9peWlsb48eM5ceIETk5OTJ48GR8fH6Kjo0lKSuLEiROcPn2aV199lR9//JFt27bh6enJggULsLe3L9Dvb775hgcffBCAv//+my+//JJNmzaZVl6vU6eOaeX0pKQkhg8fzr333ssvv/zCwoULSUhIICEhgZycHLp168bo0aMBGDlyJGfOnCE7O5snnniCRx55hHfffZfLly8TGhpK8+bNmTFjRrHHMyMjw7QWRVHtDhkyhJiYGGbMmMFff/1FaGgoDzzwAOHh4YwcOZJLly6Rl5fHmDFjCAwMBMDX15f9+/eTmJjInDlzcHd357fffuOee+7h3XffxcbGBj8/PyIjI8nLy7vhldzl5qDMUeYoc8RSlDfKG+WNVNQN/40bDAZiYmLYunUrc+bMYfHixabPvL29GTRoEM7OzgwfPhyAiIgIhg4dip+fH6dOnWL48OEkJCQwd+5c2rdvT3h4OFu2bCEmJqbYNkeOHMk777yDn59fsWXOnj3LZ599xtGjRxkxYgQ9e/bkm2++4eTJk6xfv57z58/Tu3dv+vXrR3Z2NhMnTmTJkiU0bdqUF154wVRPdHQ0rVq1Yt68eezYsYNXXnmFuLg4AP766y+WLl3KkSNHeOSRR3j//fd5+eWXGTVqFFu3bjV9Ea/at28fPXr0AOD48eM0atSIOnXqFLsPx48fZ/r06bRr147//ve/HD9+nJiYGIxGIyNGjGD37t107NiRN998Ezc3Ny5fvkz//v3p3r0748aNY/ny5aa+FmXo0KEYjUZOnDjBe++9V2S714qIiOD333831ZmXl8fcuXOpU6cOqampPPLII3Tt2hUbm4I3Av7666+sW7cOT09PBg8ezN69e/Hz88PW1pamTZty+PBhWrduXWw/Ra6lzFHmKHPEUpQ3yhvljVREiQOs6/8RFfV+t27dALjnnns4efJkqQ1u376dP/74w/Q6IyODzMxMdu/ezZw5cwD417/+Rd26dYut4/777+fLL7+kc+fO2NnZFVkmMDAQW1tbmjdvTkpKCgB79+6lZ8+e2NraUr9+ffz9/YErvxZ5e3vTrFkzAEJCQli5cqVpm+joaFO7aWlpZGRkAPDQQw9hb29Py5YtMRgMPPTQQwC0bNmSpKSkQn06d+4cHh4eRfZ31apVLF26lLS0NNPUs5eXlykAfvjhB3744Qf69u0LQFZWFn/++ScdO3Zk2bJlfPvttwCcPn2a48eP4+7uXuzxu2rJkiV4eHjw119/MWzYMP7xj38UarckRqORmTNnsnv3bmxtbUlOTiYlJYX69esXKNe2bVsaNmwIgI+PDydPnjT9j8PDw6PMq61LzafMUeaURJkj5qS8Ud6URHkjN6LEAZabmxsXL14s8N7Fixfx9vY2vXZwcADA1tYWg8FQaoP5+fmsXLnSNGVcEZMmTeK1114jKiqKyZMnF1nmar8q07X7bm9vbwrl4o6Fo6Mj2dlXHsPZtGlTTp8+TUZGBnXq1KFfv37069ePoKAg07bOzs6mbY1GI88++2yhGysTExPZvn07K1aswMnJiSFDhpjaKKsmTZpw6623cuTIETw8PAq0W5L4+HhSU1OJjY3F3t6egICAItu+9u/Czs6uwLHJycmhdu3a5eqv1FzKnJIpc5Q5Yj7Km5Ipb5Q3UnElPuTCxcWF+vXrs2PHDuDKtbrbtm0rcP1uaVxcXMjMzDS97ty5M8uWLTO9PnToEAAdO3YkPj4egK1btxYKvWvZ2NgwY8YMjh49yuzZs8vcl/bt2/PNN9+Qn59PSkoKu3btAuCOO+7g5MmT/PXXXwCsW7fOtI2fnx9r164FrnzR3d3dS5zyLsmdd95pasPJyYl+/frxxhtvmL6wBoOB3NzcIrft3Lkzq1atMh3L5ORkzp8/T3p6OnXr1sXJyYkjR47w448/mrapVatWsfVd6/z58yQlJeHl5VViuev/LtPT07n11luxt7dn586dZfp173p//vknLVq0KPd2UjMpc5Q511LmSGVS3ihvrqW8EXMq9R6st99+m6ioKN566y0ARo0aRZMmTcrcQJcuXRg9ejSbNm1i4sSJTJgwgcmTJxMcHIzBYMDPz4/JkyczatQoIiIi6NOnD76+vqV+ERwdHZk/fz6PP/449erVw8nJqdS+9OjRgx07dtC7d28aNWpEq1atcHV1xdHRkcmTJ/Pss8+abgC9+iULDw9n/PjxBAcH4+TkZDoOFfGvf/2LL774ggEDBgBXHuE5e/ZsgoKCcHFxoXbt2vTt2xdPT89CU8qdO3fmyJEjpl93nJ2deeedd3jooYf44osv6NWrF7fffnuBae+BAwcSEhJCq1atirwBdOjQodja2pKXl0dERAT16tUrctr/Knd3d9q3b09QUBAPPvggzzzzDCNGjCA4OJjWrVtzxx13lOt4pKSk4OjoWGi6XW5uyhxlzlXKHKlsyhvlzVXKGzEnG2NJj7KpgTIzM3FxceHChQsMGDCAzz//3KL/+AcPHswHH3xgeqLNzWzx4sW4uLiYwrg4YWFhNBs06YYW4fNyc2LmwNKvuRYxN2VO9VHZmaOckaqmvKk+LHmOc5UyqPq46Z4b+dxzz3Hp0iVyc3MZOXKkxX9ZiIyM5NSpUwofwNXVldDQ0DKVbdXQFU/Xil/TXs+l8q9XFymKMqf6qKzMycoxkJGdR4MbyCgRc1DeVB/mzpurOVMSZVD1cdPNYIn1CQsLIzY2tlLqNuQbsbMt+klSInJzKm/mXJsjyhQRKY+y5k1Zs0UZVD3cdDNYYp2mJxwiOb18Tw4qTQNXR17pdbdZ6xSRmqGsmXN9jujERkTKq7S8Kc/5ijKoetAAS6xCcnq2Wa5PFhEpC2WOiFiK8qbmKfEx7dYuKSmJoKCgAu9FR0fz0UcflbjdgQMHmDJlCnDlsaX79u0rd9sBAQGkpqYW+dmhQ4e46667+P7778tdb2l1XxUbG8t9991HaGio6b9rFz8szeeff86aNWvK1a+KHiuRmkKZo8wRsRTljfJGqi/NYBWhTZs2tGnTBoBdu3bh7OxM+/btzVb/V199RYcOHVi3bp1pZfRrGY1GjEYjtrY3Nv7t3bs3kyZNqtC2gwcPLvL9vLw8atUq+p9NZRwrkZuBMkeZIyKKxjkAACAASURBVGIpyhvljVS+m3qANWTIENq2bUtiYiLp6elMnToVPz8/EhMT+fjjj5k4cSJffPEFtra2rF27lokTJ3LHHXfw2muvcerUKQDGjx9Phw4duHDhAhERESQnJ9OuXTuKe3aI0Wjk66+/5pNPPuHRRx8lOzsbR0dHkpKSGD58OPfeey+//PILCxcuZOHChRw4cIDs7Gx69OjB6NGjTfUsWrSIbdu24ejoyIwZM2jatGmZ9jkxMZHo6GhcXV357bff6NWrFy1btmTp0qVkZ2czd+5cmjRpQnR0NM7OzgwfPpwhQ4bg4+PD3r17CQoKolmzZsyfP5/c3Fzc3Nx49913uXz5cpmP1a5du5g6dSpwZUHFTz/9tMILG4pYE2WOMkfEUpQ3yhupOjf1AAuurCweExPD1q1bmTNnDosXLzZ95u3tzaBBg0xfQoCIiAiGDh2Kn58fp06dYvjw4SQkJDB37lzat29PeHg4W7ZsISYmpsj29u3bh7e3N02aNMHf358tW7bQo0cPAI4fP8706dNNC+mNHTsWNzc3DAYDw4YN4/Dhw/j4+ABXHv8ZHx/PmjVrePPNN/nggw8KtbV+/Xr27t1rer1ixQoADh8+zPr163Fzc6Nr164MGDCAmJgYlixZwrJly5gwYUKhunJzc01Publ48SIrV67ExsaGL7/8kkWLFhEZGVnmY/Xxxx8zadIk02KHjo56rKjcPJQ5yhwRS1HeKG+katToAZaNTdFPUrn2/W7dugFwzz33cPLkyVLr3L59e4HrfDMyMsjMzGT37t3MmTMHuLKaed26dYvcft26dfTp0we4Mr0dFxdnCh8vL68Cq5QnJCSwcuVK8vLyOHfuHEeOHDGFz9Xrrvv06cO0adOKbKu46fM2bdrg6ekJQJMmTejUqRMALVu2JDExsdi6rjpz5gxjx47l3Llz5OTk4O3tXeQ2xR2r9u3b89ZbbxEcHEz37t1xcXEpcnsRa6PMUeaIWIryRnkj1VeNHmC5ublx8eLFAu9dvHixwJfFweHKArS2trYYDIZS68zPz2flypUV+kXCYDDwzTffsGnTJhYsWIDRaCQtLY2MjAwAnJ2dTWVPnDjBxx9/TExMDHXr1iUyMpLsbPM8pvzqPsOV/S7LMXBycjL9ecqUKQwbNoyuXbuSmJhoCt3rFXesnn32Wf75z3+ydetWBg8ezKJFi7jzzjtvdLdEqpwyp2jKHBHzU94UTXkj1UGNfoqgi4sL9evXZ8eOHQCkpaWxbds2OnToUK46MjMzTa87d+7MsmXLTK8PHToEQMeOHYmPjwdg69athUIPYMeOHdx1111s3bqVzZs3891339G9e3c2btxYqGxmZiZOTk64urqSkpJS6Gk8CQkJwJUpcl9f3zLvjzmkp6fToEEDgAJP4Snrsfrrr7+46667ePbZZ2nTpg3Hjh2zUM9FKpcyp3Ioc0QKU95UDuWNmEONHmABvP3228ybN4/Q0FCGDh3KqFGjaNKkSZm379KlC99++y2hoaHs2bOHCRMmcPDgQYKDg+nduzeff/45AKNGjWLPnj306dOHb7/9Fi8vr0J1rVu3jsDAwALvde/ena+++qpQWR8fH1q1akWvXr2IiIgo9NSaixcvEhwczNKlS3n11VeL7Pv69esLPMLUXI8XDQ8PZ8yYMYSFheHm5mZ6v6zHasmSJQQFBREcHEytWrWKfMqQiLVS5ihzRCxFeaO8kerJxljco2BEqomwsDCaDZpk9kX4vNycmDmwXekFReSmUp7MUY6IyI0oS94oZ6xPjb4HS2qOVg1d8XR1JCvHQEZ2nlnqbOCqJ/uISNGuZs61isof5YiI3KhWDV2p41ir2PMb5Yz10QBLrMLTD125QdSQb8TOtugnJ1WEuesTkZrhauZcq7i8UI6IyI14+qE7S80R5Yx10QBLrML0hCs3j77S626z1quwEpGiTE84RHL6/z/VrIGrY7H5oxwRkRvxyQ/HeLLT7SWWUc5YFw2wxCpce6IjIlLZktOzzX7fp4hIUVIzc6q6C2JmNf4pgiIiIiIiIpaiGSwB4MKFCwwbNgyAlJQUbG1t8fDwAODLL78ssHDf9ZKSknjuueeKfBSriMj1lDciYknKHLE0DbAEAHd3d+Li4gCIjo7G2dmZ4cOHl7pdXp55nugnIjcP5Y2IWJIyRyxNlwhKsSIjI/n6669Nr6+upp6YmMijjz7Kc889R58+fQpsc+LECfr27cvPP//MX3/9xfDhwwkLC+PRRx/lyJEjZGRkEBAQQG5uLkCh1yJyc1LeiIglKXOkMmkGSyrk119/JT4+nsaNG5OUlATA0aNHefHFF3nrrbfw8fFh6NChREVF0axZM3766SeioqJYunQp/v7+bN26lcDAQNatW0f37t2xt7ev4j0SkepKeSMilqTMkRulAZZUSJs2bWjcuLHpdWpqKiNHjmTOnDk0b96czMxM9u/fz5gxY0xlcnKuPCWnf//+LFq0iMDAQGJjY3njjTcs3n8RsR7KGxGxJGWO3CgNsKRYdnZ25OfnA5Cfn19gitvZ2blAWVdXV7y8vNi7dy/NmzfHaDRyyy23mK55vlaHDh2IiooiMTERg8FAy5YtK3dHRKTaU96IiCUpc6Qy6R4sKdZtt93GL7/8AsDmzZtLvIbY3t6eOXPmsGbNGuLj46lTpw7e3t4kJCQAYDQaOXz4sKl83759iYiIICwsrHJ3QkSsgvJGRCxJmSOVSQMsKdbAgQPZvXs3ISEh7N+/v9AvOtdzdnbmgw8+YPHixWzatIl33nmHmJgYQkJC6NOnDxs3bjSVDQ4O5tKlSwQFBVX2boiIFVDeiIglKXOkMtkYjUZjVXdCbj5ff/21KaBKExYWRrNBkwCYObBdZXdNRGqY8uQN/H/mnEr72/Sel5uT8kdEyqS85zidnptGRPe7LNAzsRTdgyUW98Ybb/D999+zcOHCMm/TqqFrJfZIRGqqiuQNXMkcT1dH0+t6LsUvRCoiclVFMuf2ei6V2COpCprBkmovLCyM2NjYAu8Z8o3Y2dpUUY9EpCYrKnNAuSMi5qe8qZk0gyVWYXrCIZLTswFo4OrIK73uruIeiUhNdm3mgHJHRCqP8qbm0QBLrEJyenaB+yFERCqTMkdELEV5U/PoKYJWYv78+fTp04fg4GBCQ0P56aefSt1m9uzZbN++HYDFixfz99/m+fJGR0fz0UcfmaWuyMhIvv76a7PUJSLmobwREUtR3khNpBksK7B//362bNnC6tWrcXBwIDU1tcT1Gq66doXxpUuXEhISgpOT0w31JS8v74a2F5HqTXkjIpaivJGaSgMsK3Du3Dnc3d1xcLjyFCsPDw9+/vln3njjDebMmcPGjRt58cUX2bNnD0ajkd69e7Np0yYiIyP517/+xdmzZzl79ixDhw7Fzc2NYcOG8f777wNw+fJlcnNz2bx5MwcPHuStt94iKysLd3d3pk2bhqenJ0OGDMHHx4e9e/cWWtNh5cqVrFixgtzcXJo2bcrbb7+Nk5MTkZGR1KlTh4MHD3Lu3DleeuklevbsidFo5I033uCHH36gUaNG2NvbW/x4ikjxlDciYinKG6mpdImgFejUqROnT5+mR48evP766+zatYtWrVpx6NAhAPbu3UuLFi04cOAAP/30E/fee2+B7Z944gk8PT1ZsmQJy5Yto2vXrsTFxREXF4ePjw/Dhw8nNzeXKVOm8P777xMbG0u/fv2YNWuWqY7c3FxiY2N56qmnCtTdrVs3Vq1axdq1a7njjjuIiYkxfXb27Fk+++wzPvjgA2bMmAHAt99+y7Fjx1i/fj3Tp09n//79lXXYRKQClDciYinKG6mpNINlBVxcXIiNjWXPnj0kJiYyduxYIiIiaNKkCUeOHOHnn3/mySefZM+ePRgMBjp06FCmej/88ENq167NY489xm+//cZvv/3Gk08+CUB+fj7169c3le3du3eRdfz++++89957pKenk5mZSefOnU2fBQYGYmtrS/PmzUlJSQFg9+7d9OnTBzs7Oxo0aMB9991X0cMiIpVAeSMilqK8kZpKAywrYWdnh7+/P/7+/rRs2ZI1a9bg5+fH999/T61atXjggQeIjIzEYDDw8ssvl1rf9u3b+frrr1m+fDkARqORFi1asGLFiiLLF3dtc2RkJPPmzcPHx4fY2Fh27dpl+uzqlL+IWBfljYhYivJGaiJdImgFjh49yp9//ml6fejQIby8vPDz82PJkiW0a9cODw8P0tLSOHbsGC1btixUh4uLC5mZmQCcPHmSqKgoZs+eTe3atQG4/fbbSU1NNU1p5+bm8vvvv5fat8zMTOrXr09ubi7x8fGllu/YsSMJCQkYDAbOnj1LYmJiWQ6BiFiI8kZELEV5IzWVZrCsQFZWFlOmTOHSpUvY2dnRtGlTJk+ejLOzMykpKXTs2BGAu+66Cw8PD2xsCq/8PXDgQJ5++mk8PT3x9/cnLS2NUaNGAeDp6cmHH37I+++/z5QpU0hPT8dgMDB06FBatGhRYt/GjBnDgAED8PDw4N577zWFXHG6devGzp076d27N15eXrRr166CR0VEKoPyRkQsRXkjNZWN0Wg0VnUnREoSFhZGs0GTTIvwebk5MXOggktEKsf1mQPKHRGpHMqbmkkzWGIVWjV0xdPVEYB6Lrr2WUQq17WZA8odEak8ypuaRwMssQpPP3RnlbZvyDdiZ1v40gQRqZkqmjnKChEpr8o8x1EmVQ0NsMQqTE84RHJ6dpW03cDVkVd63V0lbYtI1ahI5igrRKQiKuscR5lUdTTAEquQnJ5d4PpkEZHKpMwREUtR3tQ8eky7lGj+/Pn06dOH4OBgQkND+emnn8pdx6ZNm1i4cGEl9E5EahLljYhYkjJHKotmsKRY+/fvZ8uWLaxevRoHBwdSU1PJzc0tdz1du3ala9euldBDEakplDciYknKHKlMGmBJsc6dO4e7u7tpxXIPDw8AAgIC6NmzJ9u2bcPR0ZEZM2bQtGlTNm/ezPz588nNzcXNzY13332XevXqERsby8GDB5k0aRKRkZHUqVOHgwcPcu7cOV566SV69uxZlbspItWA8kZELEmZI5VJlwhKsTp16sTp06fp0aMHr7/+Ort27TJ95urqSnx8PI8//jhvvvkmAB06dGDlypWsWbOGPn36sGjRoiLrPXv2LJ999hkffPABM2bMsMi+iEj1prwREUtS5khl0gyWFMvFxYXY2Fj27NlDYmIiY8eOJSIiAoCgoCAA+vTpw7Rp0wA4c+YMY8eO5dy5c+Tk5ODt7V1kvYGBgdja2tK8eXNSUlIsszMiUq0pb0TEkpQ5Upk0wJIS2dnZ4e/vj7+/Py1btmTNmjXFlp0yZQrDhg2ja9euJCYmMmfOnCLLXZ2OFxG5lvJGRCxJmSOVRZcISrGOHj3Kn3/+aXp96NAhvLy8AEhISABg/fr1+Pr6ApCenk6DBg0ASgwpEZHrKW9ExJKUOVKZNIMlxcrKymLKlClcunQJOzs7mjZtyuTJk9myZQsXL14kODgYBwcHZs6cCUB4eDhjxoyhbt26+Pv7k5SUVMV7ICLWQnkjIpakzJHKZGM0Go1V3QmxLgEBAcTExJieuFPZwsLCaDZoUpUtwufl5sTMge2qpG2Rm52l8wYqnjnKChHrV5POcZRJVUczWGIVWjV0xdPV0Wz1ZeUYyMjOK1PZBmZsV0SsQ0Uyp56L7r0QkfIrT97o/MU6aIAl5bZ582aLt/n0Q3eatT5DvhE7W5tKKy8i5lEVeQPmz5zSKGNEqofqfo6j8xfroAGWWIXpCYdITs82S10NXB15pdfd5dpG4SRyczFn5pSmIpkkIjVHWfNG5y/WQwMssQrJ6dlVdg+WiNx8lDkiYinKm5pHj2kXERERERExEw2wpEKurguRlJREfHx8qeWTkpJMK6OLiJSH8kZELEV5I+agAZbckJMnT/LVV19VdTdE5CagvBERS1HeyI3QAEtuyIwZM9izZw+hoaEsXryYpKQkHn30UR5++GEefvhh9u3bV2ibxx57jEOHDpleDx48mMOHD1uy2yJihZQ3ImIpyhu5ERpgyQ2JiIjAz8+PuLg4hg0bxq233sonn3zC6tWrmTVrFlOmTCm0Tf/+/YmNjQXg2LFjZGdn4+PjY+mui4iVUd6IiKUob+RGaIAlZpWXl8d//vMfgoODGTNmDEeOHClUpmfPnmzZsoXc3FxWrVpFWFhYFfRURKyd8kZELEV5I+Whx7SLWS1evJh69eoRFxdHfn4+bdu2LVTGycmJBx54gE2bNpGQkGD6tUdEpDyUNyJiKcobKQ8NsOSGuLi4kJmZaXqdnp5Ow4YNsbW1ZfXq1RgMhiK3GzBgAM899xx+fn7UrVvXUt0VESumvBERS1HeyI3QJYJyQ+666y5sbW0JCQlh8eLFPProo6xevZqQkBCOHj2Ks7Nzkdu1bt2aOnXqaPpcRMpMeSMilqK8kRthYzQajVXdCbn5JCcn88QTT5CQkICtbcnj/LCwMHq/8A4pmTllrj8rx0BGdl6RnzVwdeSVXneXq78iYr3KkzdQscwpq6KySZkkUnOYO2+uzQxlhfXQJYJicWvWrGHWrFlERkaWKXwAnn7oznK1Ycg3YmdrU+HPRaRmqEjeQPkzp6yKyx5lkoj1q4y8uT4blBXWQTNYUu2FhYXh/8xUktOzy1Rev/CIyI0ob+aUlbJJRK5XUt4oM6yXZrDEKiSnZ3Mq7e+q7oaI3CSUOSJiKcqbmkcPuRARERERETETDbCkRL6+vpVaf3R0NB999FGltiEi1kF5IyKWpMyRyqIBlphNcWtCiIiYm/JGRCxJmSPloXuwpEzOnj3L2LFjycjIwGAw8Prrr+Pn54evry+PPPII27dvZ9KkSezcuZPvvvuO7OxsfH19mTx5MjY2Nvz1119ERUVx4cIFateuzRtvvMGdd1bOU7pExLopb0TEkpQ5Ym4aYEmZfPXVV3Tu3JkRI0ZgMBj4++8rN2NmZWXRtm1bIiMjAWjevDnh4eEAvPTSS3z33XcEBAQwceJEoqKiaNasGT/99BNRUVEsXbq0yvZHRKov5Y2IWJIyR8xNAywpkzZt2jB+/Hjy8vIIDAzk7ruvPDbUzs6OHj16mMolJiayaNEiLl++TFpaGi1atMDf35/9+/czZswYU7mcHPMv4CkiNYPyRkQsSZkj5qYBlpRJx44d+fTTT9m6dSuRkZE8+eST9O3bF0dHR+zs7ADIzs4mKiqKVatW0ahRI6Kjo8nOzsZoNHLLLbcQFxdXxXshItZAeSMilqTMEXPTQy6kTE6ePEm9evUYOHAgAwYM4JdffilUJjv7yiJ57u7uZGZmsmHDBgDq1KmDt7c3CQkJABiNRg4fPmy5zouIVVHeiIglKXPE3DSDJWWya9cuPvroI2rVqoWzszPTp08vVOaWW25hwIABBAUFUa9ePdq0aWP67J133uH1119n/vz55OXl0bt3b3x8fCy5CyJiJZQ3ImJJyhwxNxuj0Wis6k6IlCQsLIxmgyaVeZVzLzcnZg5sV8m9EpGaqryZU1bKJhG5Xkl5o8ywXprBEqvQqqErnq6OZSpbz8WhknsjIjVdSZmTlWMgIzuv3HU2KGOGicjNpbi80fmM9dIAS6zC0w9VbD0JQ74RO1sbM/dGRGq6kjLnRnJFmSQi1yvLOY6yw7pogCVWYXrCIZLTs8u1TQNXR17pdXcl9UhEarLiMudGc0UnSCJyvdLOcXQ+Y300wBKrkJyebfb7IUREiqPMERFLUd7UPBpg1VApKSlMmzaNH3/8kbp162Jvb8/TTz9Nt27dqrprIlLDKG9ExJKUOVLdaYBVAxmNRkaNGkXfvn2ZMWMGcGWNh82bN5dp+7y8PGrV0j8NESmd8kZELEmZI9ZA/8JqoJ07d2Jvb8/gwYNN7912220MGTIEg8HAu+++y65du8jJyeGxxx5j0KBBJCYmMnv2bG655RaOHTvG5MmTiY6OxtXVld9++41evXrRsmVLli5dSnZ2NnPnzqVJkyZs3ryZ+fPnk5ubi5ubG++++y716tUjOjqaU6dOkZSUxKlTpxg6dChPPPEEs2fPpm7dugwbNgyAWbNm4eHhwdChQ6voaInIjVDeiIglKXPEGthWdQfE/H7//XdatWpV5GcxMTG4urqyatUqVq1axcqVKzlx4gQAv/76KxMmTDCtTn748GGioqJISEggLi6OP//8k5iYGPr378+yZcsA6NChAytXrmTNmjX06dOHRYsWmdo6duwYH330EV9++SVz584lNzeXfv36ERcXB0B+fj7r1q0jJCSkMg+HiFQi5Y2IWJIyR6yBZrBuAlFRUezduxd7e3tuu+02/ve//5kCJj09nePHj2Nvb0+bNm1o3Lixabs2bdrg6ekJQJMmTejUqRMALVu2JDExEYAzZ84wduxYzp07R05ODt7e3qbt//nPf+Lg4ICHhwceHh6cP38eb29v3Nzc+PXXX0lJSaFVq1a4u7tb6lCISCVT3oiIJSlzpDrSAKsGatGiBd98843p9WuvvUZqair9+/fHy8uL//znPzz44IMFtklMTMTZ2bnAew4O/7/Ana2trem1ra0tBoMBgClTpjBs2DC6du1KYmIic+bMKXJ7Ozs78vKuLMw5YMAAYmNjSUlJoV+/fmbaaxGpCsobEbEkZY5YA10iWAPdd999ZGdn89lnn5neu3z5MgCdO3fm888/Jzc3F7gyxZ2VlVXhttLT02nQoAEAa9asKdM2gYGBbNu2jQMHDtC5c+cKty0iVU95IyKWpMwRa6AZrBrIxsaGuXPnMm3aNBYtWoSHhwdOTk6MGzeOnj17cvLkScLCwjAajbi7uzNv3rwKtxUeHs6YMWOoW7cu/v7+JCUllbqNg4MD/v7+3HLLLdjZ2VW4bRGpesobEbEkZY5YAxuj0Wis6k7IzSU/P5+HH36Y2bNn06xZs1LLh4WF0WzQpHIvwufl5sTMge0q2EsRqQnKmzdQcuYoV0SkJJVxjqPcsT6awRKL+uOPP/j3v/9Nt27dynyyA9CqoSuero7laquei0PphUSkxqpo3kDxmaNcEZHiVNY5jnLH+mgGS6q9sLAwYmNjq7QPhnwjdrY2VdoHEbEMZY6IWIql80bZYhmawRKrMD3hEMnp2VXSdgNXR17pdXeVtC0iVUOZIyKWYqm8UbZYjgZYYhWS07PLfQ+WiEhFKXNExFKUNzWPBlgCwN13303Lli1Nr+fOnVtgQT0REXNR3oiIJSlzxNI0wBIAateuTVxcXJGfGY1GjEYjtrZaNk1EbpzyRkQsSZkjlqYBlhQpKSmJ4cOHc++99/LLL7+wcOFCFi5cyIEDB8jOzqZHjx6MHj0agICAAPr27ct3331HXl4e7733HnfeeSeZmZlMmTKFgwcPAlfWk+jRowf//e9/iY6OJicnh8aNGzNt2jRcXFyqcndFpAopb0TEkpQ5Utk0wBLgyirooaGhAHh7e/Pqq69y/Phxpk+fTrt2V9ZeGDt2LG5ubhgMBoYNG8bhw4fx8fEBwN3dndWrV7N8+XI+/vhjpk6dyrx586hTpw7x8fEAXLx4kdTUVObPn88nn3yCs7MzCxcu5JNPPiE8PLxqdlxELE55IyKWpMwRS9MAS4DC0+dJSUl4eXmZggcgISGBlStXkpeXx7lz5zhy5IgpfLp37w5A69at+fbbbwHYsWMHM2fONG1ft25dvvvuO/744w8GDx4MQG5uboE2RKTmU96IiCUpc8TSNMCSYjk7O5v+fOLECT7++GNiYmKoW7cukZGRZGf//yNF7e3tAbC1tcVgMBRbp9FopFOnTgVCSUREeSMilqTMkcqkO/qkTDIzM3FycsLV1ZWUlBS+//77Urd54IEHWL58uen1xYsXadeuHfv27eP48eMAZGVlcezYsUrrt4hYH+WNiFiSMkfMTQMsKRMfHx9atWpFr169iIiIoH379qVuM2LECC5dukRQUBAhISEkJibi4eHBtGnTePHFFwkODuaRRx7h6NGjFtgDEbEWyhsRsSRljpibjdFoNFZ1J0RKEhYWRrNBk6psET4vNydmDtQ11CI3C2WOiFiKJfNG2WI5ugdLrEKrhq54ujqWWCYrx0BGdp7Z225QSrsiUvOUJXOKYo4cUuaI3FwqmjfXKkv2KFssRwMssQpPP3RnqWUM+UbsbG0qpf3KrFtEqp+yZE5RzJUVyhyRm0dF8+ZaZc0MZYtlaIAlVmF6wiGS07OL/byBqyOv9Lq70tpXGIncXErLnKKYM4eUOSI3j4rkzbXKkz3KFsvQAEusQnJ6dpXdDyEiNx9ljohYivKm5tFTBEVERERERMxEAywps3PnzjF27FgCAwMJCwvjmWeeYcWKFfz73/8usvyECRP4448/AAgICCA1NdWS3RURK6a8ERFLUd6IuekSQSkTo9FIeHg4ffv2ZdasWQAcPnyYTZs2FbvN1KlTLdU9EalBlDciYinKG6kMmsGSMtm5cye1atVi8ODBpvd8fHzw8/MjKyuL0aNH07NnTyIiIri6tNqQIUM4cOBAobri4uLo378/oaGhTJo0CYPBYLH9EJHqT3kjIpaivJHKoAGWlMnvv//OPffcU+Rnv/76K+PHj2f9+vUkJSWxd+/eYus5cuQICQkJfP7558TFxWFra0t8fHxldVtErJDyRkQsRXkjlUGXCMoNa9u2LQ0bNgSu/Opz8uRJ/Pz8iiy7Y8cODh48SP/+/QG4fPkyt956q8X6KiLWTXkjIpaivJGK0gBLyqRFixZs2LChyM8cHBxMf7azsytxStxoNPLwww8TERFh9j6KSM2gvBERS1HeSGXQJYJSJvfddx85OTmsWLHC9N7hw4fZs2dPueq5//772bBhA+fPnwcgLS2NkydPmrWvImLdlDciYinKG6kMmsGSMrGxsWHOnDm8+eabfPjh34nBdQAAF2tJREFUhzg6OnLbbbcRGBhYrnqaN2/OCy+8wFNPPUV+fj729vZMmjSJ2267rZJ6LiLWRnkjIpaivJHKYGO8+kgUkWoqLCyMZoMmlbjKuZebEzMHtrNgr0SkpipL5hRFOSQi5VXRvLmWsqf60QyWWIVWDV3xdHUs9vN6Lg7FfiYiUl6lZU5RlEMiUhFF5U1WjoGM7Lwybd+gnFkllU8DLLEKTz90Z7nKG/KN2NnaVFJvRKSmK2/mXKXsEZHyKipvypslyp7qRQMssQrTEw6RnJ5dprINXB15pdfdldwjEanJypM5Vyl7RKQirs+bimSJBlfViwZYYhWS07Nv6PpkEZHyUOaIiKUob2oePab9JnXu3DnGjh1LYGAgYWFhPPPMMxw7dqzS2vP19a20ukWkelPeiIglKXOkqmkG6yZkNBoJDw+nb9++zJo1C7iy5sP58+e5/fbbq7h3IlKTKG9ExJKUOVIdaIB1E9q5cye1atVi8ODBpvd8fHzIzMxk6NChXLp0iby8PMaMGUNgYCBJSUk888wzdOjQgf3799OgQQPmzZtH7dq1WblyJStWrCA3N5emTZvy9ttv4+TkxIkTJxg3bhxZWVkEBASY2snMzGTkyJGF2hCRmkl5IyKWpMyR6kCXCN6Efv/9d+65555C7zs6OjJ37lxWr17NkiVLmD59OleXSTt+/DiPPfYY69atw9XVlQ0bNgDQrVs3Vq1axdq1a7njjjuIiYmB/2vv/mOivu84jr+On6sIExTQimm1dSr+qroGm9hWRZChFETpNItNTbf6Iw22a1wJpv2DFKsbQ6fOps6Z1nY6OkZ1cKhUsLp1AkpdVjacytpFaT0VUPCcgMftD+OtVoUDz7vvHc/HX97xvft+TpJneN997/uVlJubq0WLFqm4uFhRUVFO7QOA76E3ANyJ5sAI+AQLDna7Xfn5+Tp69Kj8/PxksVh08eJFSVJMTIzGjLlxRpuxY8eqoaFB0o2QbdiwQa2trbJarZo2bZok6fjx49q0aZMkKTU1VXl5eV3uIzIy0t0vF4AH0RsA7kRz4E4MWH3QyJEjHe/OfFNxcbGamppUVFSkwMBAzZw5U21tN04bGhT0/wto+vv7O+7PysrSli1bNHr0aBUVFam6utqxncl0+ylDu9oHAN9DbwC4E82BEXCIYB80depUtbe3q6CgwHHfiRMn9NVXX2ngwIEKDAxUZWWl4x2crlitVkVGRqqjo0PFxcWO+ydNmiSz2SxJ+tOf/uS4v7W1tcf7AOC96A0Ad6I5MAIGrD7IZDJp8+bN+utf/6pZs2Zpzpw5ys/P11NPPaXa2lqlpKRoz549GjFiRLfPtXLlSmVkZGjRokW3bL969Wrt3LlTKSkpslgsjvtTUlJ6vA8A3oveAHAnmgMjMNn59h0MLj09XQ8vfMPpi/A9OOAB5T/72H1eFQBf1dPm3ER7APTUnXpDS7wf38GCV4gdHKqo0OAut7nabtOVtuuK7mY7AOhOV8252Zpvoz0AeuPbvRkUEtTF1vAGDFjwCj9+6pFut7F12uXvZ7rt3wDQU101p6u+0B4APeXM3zjdoT3GwoAFr7Bub50srXc/E090aLBe+8EYx20iA+Be3K05327Nt9EeAD3V3d843emuS3A/Bix4BUtrW4+/DwEAvUVzALgLvfE9nEWwjxgzZoxSU1P1zDPPaN68efrss8+6fcykSZO63Wb16tU6ffq0K5YIwIfQHADuQm9gNHyC1Ud85zvf0Z49eyRJf/7zn5Wfn68PPvjgnp83Nzf3np8DgO+hOQDchd7AaBiw+qArV64oLCzMcXvbtm3au3ev2tvblZCQoMzMzFu27+zsVE5OjiorKzVkyBAFBARo/vz5SkpK0uLFi/Wzn/1M48eP16RJk3T8+HFJ0r59+/TJJ59o7dq1ysrKUnBwsOrq6tTY2Kg1a9Zo9+7d+tvf/qaJEydq7dq1bn39ANyL5gBwF3oDI2DA6iOuXbum1NRUtbW16cKFC3rvvfckSX/5y1/0n//8R4WFhbLb7Vq+fLmOHj2qxx9/3PHYsrIyNTQ0qLS0VI2NjUpOTtb8+fN7tP+WlhYVFBSovLxcy5cv165duzRy5EgtWLBAdXV1GjOGL2cCvoTmAHAXegOjYcDqI7758fnx48f12muvqaSkRJ9++qk+/fRTpaWlSZKuXr2qL7/88pb41NTUKCkpSX5+foqMjFRcXFyP9z9jxgyZTCaNGjVKgwYN0qhRoyRJjz76qBoaGogP4GNoDgB3oTcwGgasPmjSpElqbm5WU1OT7Ha7XnzxRS1cuNCl+2hru/V0o0FBNy6aZzKZHP+WJD8/P12/fvsFOwH4DpoDwF3oDYyAswj2QfX19bLZbBowYICmTZumP/7xj7JarZIki8WixsbGW7afPHmyysrK1NnZqYsXL6q6uvqOzzto0CDV19ers7NTBw4cuO+vA4B3oDkA3IXewAj4BKuPuHl8siTZ7XatW7dO/v7+mjZtmurr6x3v7vTr10+/+MUvNHDgQMdjZ8+erSNHjig5OVlDhgxRbGysQkNDb9vHq6++qqVLlyoiIkLjxo3T1atX3fPiABgOzQHgLvQGRmOy2+12Ty8Cxme1WhUSEqLm5mZlZGRo165dioyMdMu+09PT9fDCN7q8CN+DAx5Q/rOPuWU9AO4/ozaH1gC+x6i9cRZdMh4+wYJTli1bppaWFnV0dGjFihVuC89NsYNDFRUarKvtNl1pu/145ujQYLeuB8D9ZZTmSLqlO7QG8D1G7Y2z6JLxMGDBKe+//75H9//jpx6RJNk67fL3M91xm65+BsC7GKU50u1toTWAbzFyb5xFl4yFAQteYd3eOknSaz+4+6lOCQsAV1m3t06W1jZFhwbf1h1aA8CVuuqNs+iSsTBgwStYWtu63wgAXMTS2nZP34kAAGfRG9/DadoBAAAAwEUYsHzcgQMHNGrUKNXX1/f68adPn+7x44qKipSTkyNJ2rVrl3bv3t2r/QPwHvQGgDvRHBgVA5aPKykp0ZQpU2Q2m3v1+K7i4+zVyRctWqS0tLRe7R+A96A3ANyJ5sCo+A6WD7NaraqpqdGOHTu0bNkyZWZmqqqqStu3b9c777wjScrJydG4ceOUnp6uvLw8VVRUOC7Ol5CQoIqKClVXV+vtt9/Wpk2btHr1ao0ePVo1NTWaO3euHn74Yb399tvq6OjQgAEDlJeXp0GDBt2yjk2bNqlfv3564YUX9OGHH6qgoEAdHR166KGH9POf/1wPPPCAJ/57ALgQvQHgTjQHRsaA5cPKy8v15JNPavjw4QoPD1dtbe1dt21ubtbHH3+sffv2yWQyqaWlRWFhYZo5c6amT5+upKQkx7YdHR0qKiqSJF2+fFkffvihTCaT/vCHP2jbtm3Kysq6634SEhL07LPPSpLWr1+vwsJCLV682EWvGICn0BsA7kRzYGQMWD7MbDbrueeekyQlJyfLbDZr+vTpd9w2NDRUwcHBys7O1owZM+663c3nuuncuXN65ZVXdOHCBbW3tysmJqbLNZ06dUobNmxQa2urrFarpk2b1uPXBcB46A0Ad6I5MDIGLB916dIlVVZW6uTJkzKZTLLZbDKZTIqPj1dnZ6dju7a2G6c/DwgIUGFhoY4cOaJ9+/bpgw8+0I4dO+743N/8uPvNN9/U888/r/j4eFVVVWnz5s1drisrK0tbtmzR6NGjVVRUpOrqahe8WgCeRG8AuBPNgdFxkgsftX//fqWmpurgwYOqqKjQoUOHFBMTo87OTtXX16u9vV0tLS06cuSIpBvHMre2turpp59Wdna2/vWvf0mSQkJCZLVa77qf1tZWRUdHS5JTZ9GxWq2KjIxUR0eHiouLXfBKAXgavQHgTjQHRscnWD6qpKREP/nJT265LzExUWazWUlJSZo7d65iYmIUGxsr6UYUVqxY4Xi35+YxxsnJyXr99df1/vvva+PGjbft56WXXtLKlSv13e9+V3FxcTp79myX61q5cqUyMjIUERGhiRMndhk2AN6B3gBwJ5oDozPZ7Xa7pxcBdCU9PV0PL3xDkpT/7GMeXg0AX3ezOV9d+q8eHPAA3QFw39Ab38QnWPAKsYNDPb0EAH1I7OBQRYUGa1BIkKeXAsDH0Rvfw4AFr/Djpx655+ewddrl72dywWoA+DpXNOdO6BCAb+NvHN/DgAWvsG5vnSytbb1+fHRosF77wRgXrgiAL7vX5twJHQJwJ/yN43sYsOAVLK1t+urSfz29DAB9BM0B4C70xvdwmnYfcuDAAY0aNUr19fX3fV91dXU6dOiQ43Z5ebm2bt163/cLwDhoDgB3oTfwJgxYPqSkpERTpkyR2Wy+7WfXr1936b6+HZ/4+Hi9+OKLLt0HAGOjOQDchd7Am3CIoI+wWq2qqanRjh07tGzZMmVmZqqqqkq/+tWvFBYWpi+++EJ79+5VTk6OKisrNWTIEAUEBGj+/PlKSkpSbW2t1q5dq6tXryo8PFxvvfWWoqKitHjxYk2YMEFVVVVqbW1Vbm6uJkyYoI0bN+ratWuqqanR0qVLde3aNdXW1uqNN95QVlaW+vfvr9raWl24cEGrVq1SUlKS4zoULS0tun79ulauXKlZs2Z5+r8OQC/QHADuQm/gbRiwfER5ebmefPJJDR8+XOHh4aqtrZUk/fOf/1RxcbGGDRumffv2qaGhQaWlpWpsbFRycrLmz5+vjo4Ovfnmm9qyZYsiIiJUWlqq9evX66233pIk2Ww2FRYW6tChQ9q8ebPeffddZWZmOmIjSUVFRbes5/z589q5c6f+/e9/a/ny5UpKSlJwcLB+/etfq3///mpqatIPf/hDxcfHy2TirDeAt6E5ANyF3sDbMGD5CLPZrOeee07SjSuTm81mTZ8+XePHj9ewYcMkSTU1NUpKSpKfn58iIyMVFxcnSfriiy908uRJLVmyRJLU2dmpyMhIx3MnJCRIksaOHauGhgan1jNr1iz5+fnp0Ucf1cWLFyVJdrtd+fn5Onr0qPz8/GSxWHTx4sVb9gXAO9AcAO5Cb+BtGLB8wKVLl1RZWamTJ0/KZDLJZrPJZDLp6aefVr9+/bp9vN1u18iRI1VQUHDHnwcF3bjwnZ+fn2w2m1NruvmYbyouLlZTU5OKiooUGBiomTNnqq3NtadBBnD/0RwA7kJv4I04yYUP2L9/v1JTU3Xw4EFVVFTo0KFDiomJ0bFjx27ZbvLkySorK1NnZ6cuXryo6upqSdLw4cPV1NSk48ePS5I6Ojp06tSpLvcZEhIiq9Xao3W2trZq4MCBCgwMVGVlpdPvFAEwFpoDwF3oDbwRA5YPKCkpue2LlImJibedaWf27NmKjo5WcnKyVq1apdjYWIWGhiooKEgbN25UXl6ennnmGaWlpTlCdDdxcXE6ffq0UlNTVVpa6tQ6U1JSVFtbq5SUFO3Zs0cjRozo2QsFYAg0B4C70Bt4I5Pdbrd7ehFwH6vVqpCQEDU3NysjI0O7du0y/PHB6enpenjhG/d0Eb4HBzyg/Gcfc+GqADijrzbnTugQcH/11d7QFuPhO1h9zLJly9TS0qKOjg6tWLHC8OG5KTo02KOPB9A7fbU57npOAP/XV3tDW4yHT7BgeOnp6bedIrU3bJ12+ftxulQAXXNVc+6EDgH4Jv7G8U18Bwt9BuEB4Gl0CMD9QFuMhQELAAAAAFyEAQsAAAAAXIQBCwAAAABchAELAAAAAFyEAQsAAAAAXIQBCwAAAABchAELAAAAAFyEAQsAAAAAXMRkt9vtnl4E0JW4uDgNHTrU08tAL4WHh+u3v/2tp5cBOI3meC96A29Db7zb3ZrDgAUAAAAALsIhggAAAADgIgxYAAAAAOAiDFgAAAAA4CIMWAAAAADgIgxYAAAAAOAiDFgAAAAA4CIMWDC0w4cPa/bs2UpISNDWrVs9vRynff3111q8eLGSk5M1Z84cvffee5KkS5cuacmSJUpMTNSSJUt0+fJlD6/UeTabTWlpaVq6dKkk6cyZM8rIyFBCQoJefvlltbe3e3iFwL0xUm+8pSF0Aeg9mtMz3tQbBiwYls1mU05OjrZt2yaz2aySkhKdPn3a08tyir+/v7KyslRaWqqCggLt3LlTp0+f1tatW/XEE0+orKxMTzzxhMeD2hM7duzQI4884ridl5en559/Xh9//LHCwsJUWFjowdUB98ZovfGWhtAFoHdoTs95U28YsGBYf//73/XQQw9p2LBhCgoK0pw5c1ReXu7pZTklKipKY8eOlST1799fI0aMkMViUXl5udLS0iRJaWlpOnDggCeX6bRz587pk08+0YIFCyRJdrtdlZWVmj17tiRp3rx5XvO7Ae7EaL3xhobQBaD3aE7PeFtvGLBgWBaLRYMHD3bcjo6OlsVi8eCKeufs2bOqq6vTxIkT1djYqKioKElSZGSkGhsbPbw656xZs0arVq2Sn9+NZDQ3NyssLEwBAQGSpMGDB3vl7wa4yci9MWpD6ALQezSnZ7ytNwxYwH1ktVqVmZmp7Oxs9e/f/5afmUwmmUwmD63MeQcPHlRERITGjRvn6aUAfY5RG0IXAN9kxOZ4Y28CPL0A4G6io6N17tw5x22LxaLo6GgPrqhnOjo6lJmZqZSUFCUmJkqSBg4cqPPnzysqKkrnz59XRESEh1fZvc8++0wVFRU6fPiw2tradOXKFeXm5qqlpUXXr19XQECAzp0751W/G+DbjNgbIzeELgD3huY4zxt7wydYMKzx48fryy+/1JkzZ9Te3i6z2ayZM2d6ellOsdvtWr16tUaMGKElS5Y47p85c6Z2794tSdq9e7fi4+M9tUSnvfrqqzp8+LAqKiqUn5+vqVOn6pe//KXi4uK0f/9+SdJHH33kNb8b4E6M1hujN4QuAPeG5jjPG3tjstvtdk8vAribQ4cOac2aNbLZbJo/f76WL1/u6SU55dixY/rRj36k733ve47jhX/6059qwoQJevnll/X111/rwQcf1IYNGzRgwAAPr9Z5VVVV2r59u9555x2dOXNGr7zyii5fvqwxY8YoLy9PQUFBnl4i0GtG6o03NYQuAL1Dc3rOW3rDgAUAAAAALsIhggAAAADgIgxYAAAAAOAiDFgAAAAA4CIMWAAAAADgIgxYAAAAAOAiXGgYcKELFy5ozZo1+vzzzxUWFqaBAwcqOztbw4cPd8nzV1VVKTAwUJMnT3bJ8wHwXvQGgDvRHOcxYAEuYrfb9dJLLyktLU3r16+XJJ04cUKNjY0ui091dbX69evnE/EB0Hv0BoA70Zye4RBBwEUqKysVEBCgRYsWOe4bPXq0pkyZonXr1mnu3LlKSUlRaWmppBvv1CxdutSxbU5OjoqKiiTduHL6xo0bNW/ePKWkpKi+vl5nz57V73//e7377rtKTU3VsWPH3PsCARgGvQHgTjSnZ/gEC3CRU6dOaezYsbfdX1ZWphMnTmjPnj1qbm7WggUL9P3vf7/b5wsPD9dHH32k3/3ud9q+fbtyc3O1cOFC9evXTy+88ML9eAkAvAS9AeBONKdn+AQLuM9qamo0Z84c+fv7a9CgQXr88cf1+eefd/u4xMRESdK4cePU0NBwv5cJwAfQGwDuRHPujAELcJGRI0fqH//4h9Pb+/v7q7Oz03G7ra3tlp8HBgZKkvz8/GSz2VyzSAA+gd4AcCea0zMMWICLTJ06Ve3t7SooKHDcd+LECYWFhWnv3r2y2WxqamrSsWPHNGHCBA0dOlT19fVqb29XS0uLjhw50u0+QkJCZLVa7+fLAOAF6A0Ad6I5PcN3sAAXMZlM2rx5s9asWaPf/OY3Cg4O1tChQ5WdnS2r1arU1FSZTCatWrVKkZGRkqSkpCTNnTtXMTExio2N7XYfM2bMUGZmpsrLy/X66687dZwzAN9DbwC4E83pGZPdbrd7ehEAAAAA4As4RBAAAAAAXIQBCwAAAABchAELAAAAAFyEAQsAAAAAXIQBCwAAAABchAELAAAAAFyEAQsAAAAAXOR/HJvioxbqgSAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x288 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Распределение пользователей в зависимости от страны \n",
    "fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))\n",
    "\n",
    "sns.histplot(y=users_test.country.sort_values(ascending=False), ax = axes[0])\n",
    "axes[0].set(title='Test group', ylabel='')\n",
    "\n",
    "sns.histplot(y=users_control_1.country.sort_values(ascending=False), ax = axes[1])\n",
    "axes[1].set(title='1 control group', ylabel='')\n",
    "\n",
    "sns.histplot(y=users_control_2.country.sort_values(ascending=False), ax = axes[2])\n",
    "axes[2].set(title='2 control group', ylabel='')\n",
    "\n",
    "sns.despine()\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Сравним наши распределения с помощью статистического теста.\n",
    "\n",
    "Есть категориальные переменные (страна, группа пользователя) → нужен хи-квадрат\n",
    "\n",
    "- $H_0$: нет статистически значимой разницы между распределениями пользователей по странам в тестовой и контрольных группах (p-value > 0.05)\n",
    "- $H_1$: есть статистически значимая разница между распределениями пользователей по странам в тестовой и контрольных группах"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(21.86910075724715, 0.7873150709907116)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stat, p, dof, expected = chi2_contingency(pd.crosstab(users_merged.country, users_merged.group))\n",
    "stat, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Распределение пользователей в зависимости от страны согласно графикам примерно одинаково в тестовой и контрольных группах. Превалируют пользователи из США.\n",
    "\n",
    "Согласно статистическому тесту (p-value > 0.05) мы не смогли отклонить нулевую гипотезу, не обнаружена статистически значимая разница между распределениями пользователей по странам в тестовой и контрольных группах, что говорит о репрезентативности выборок по этому признаку."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAEYCAYAAABBWFftAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXSU9b3H8c+QBeKFIEmTGSIRW9acUBaFYi6bDISEACEEUq9VrOH2cEUkQgRLStk3F4ppPTaSm1rQtrdKgLAEATMISZXNsqmot1SjgGRGQhZQIAtz/6DOJRAFkyeZGeb9OsdzyG+ePM93EvM9+eT3e56fyel0OgUAAAAAaLQW7i4AAAAAAG4VBCwAAAAAMAgBCwAAAAAMQsACAAAAAIMQsAAAAADAIAQsAAAAADAIAQsA4FP27dunwYMHu7sMAD6AfuObCFg+qk+fPq7/unfvrp49e7o+3rRp0/c+38SJE7V27domqBSAJ/nTn/6k5ORk9ejRQ7Nnz3ZLDd26ddNnn33mlmsDaB5VVVX61a9+paFDh6pPnz4aO3asdu/e3ex10G/QEP7uLgDucejQIde/rVarlixZon//9393Y0V11dTUyN+f/z0BTxMeHq7HHntMRUVFunTpkrvLqVdz9g96FdA0ampq1L59e7366quKiIjQ7t27NX36dG3evFkdOnRwd3ku9BvUhxks1HH58mVlZ2dr+PDh6t+/v5544gmVl5dLki5duqSZM2eqf//+6tu3r8aPH68zZ87o+eef17vvvqtFixapT58+WrRoUb3nzsvL09ChQ9W/f3+9+OKLslqteueddyRJL7zwgtLS0jRz5kzdfffd2rBhg+x2ux599FH95Cc/UWxsrF5//XXXuWbPnq3nn3/e9fG1U/BWq1WrVq1SQkKC+vXrp4yMDI/9ZRDwJiNGjNDw4cN1++2339Txr7/+ukaOHKk+ffooISFBH3zwgSTpn//8pyZOnKi+fftq1KhRstlsrs+ZPXu2Fi5cqMmTJ6tPnz5KSUnR559/Lkl68MEHJUljx45Vnz59tHXrVtfPf3Z2tgYMGKCMjAxVVVVp6dKlGjhwoAYOHKilS5eqqqrqpmr+29/+pri4ON1zzz1asGCBHnroIdcM/fr16/Uf//EfWrZsmfr3768XXnhB586d01NPPaV7771XQ4cO1e9//3tdvnxZ0pXeNnPmTNe5T548qW7duqmmpkbSldn/3/zmN5owYYLuvvtuTZkyxdVzAV922223adq0aerQoYNatGihoUOHqkOHDq4eUh/6Df3GUxCwUMerr76qgoIC/elPf1JRUZHatm3rCkwbNmzQ+fPntWvXLu3bt08LFy5Uq1atNGPGDPXt21fz5s3ToUOHNG/evOvOe/z4cS1cuFDPPfecioqKdP78ednt9jrH2Gw2xcfH691339WYMWOUnp4ui8WioqIi/e53v9PKlSu1Z8+em34vmzdv1h/+8Ae9+eab+vTTT/X73/++cV8cAN/LG2+8oRdeeEHPPPOMDh48qKysLN1+++2qrq7Wo48+qgEDBuidd97Rr3/9a82cOVOffPKJ63O3bt2qxx9/XAcOHNCdd97p+oPKn//8Z0nSxo0bdejQISUkJEiSzpw5o4qKCr311ltavHixsrKydOTIEW3cuFGbNm3Se++9d1M94OzZs0pLS9OTTz6pffv26Yc//GGdGX9JOnr0qCIjI/X2229rypQpWrx4sc6dO6eCggK9+uqr2rhxo9atW3fTX6e8vDwtW7ZMf/vb3+Tv768lS5bc9OcCvuLMmTMqLi5W586d632dfnNz6DfNg4CFOv76179qxowZslgsCgwM1OOPP67t27e7pqXLy8v12Wefyc/PTz169FDr1q1v6rzbtm3T0KFD1bdvXwUGBiotLU0mk6nOMb1799bw4cPVokULlZWV6eDBg5o5c6ZatmypqKgopaSkaOPGjTf9Xh588EG1b99et99+u6ZMmaL8/Pzv9bUA0Di5ubn6xS9+oZ49e8pkMqljx4664447dOTIEX399deaPHmyAgMDFRMTo6FDh9b5GR0+fLh69uwpf39/JSYm6sMPP/zOa7Vo0UJpaWkKDAxUq1attHnzZk2dOlWhoaEKCQnR1KlTb+r+0sLCQnXp0kUjRoyQv7+/Hn74Yf3gBz+oc0x4eLgmTpwof39/BQQEaOvWrXryySfVunVrdejQQampqd/rXtaxY8eqa9euuu222/TEE09o27Ztqq2tvenPB2511dXVmjlzpsaNG6dOnTrVewz95ubQb5oHCzlRxxdffKGpU6eqRYv/z94tWrRQaWmpxo4dq5KSEqWnp6uyslKJiYmaMWOGAgICbnheh8Mhi8Xi+jgoKOi6JUZXv+5wONS2bds6AS4iIkLvv//+Tb+X9u3b1/lch8Nx058LoPFOnz6tO++887rxb/rB1X0mIiKizqz21b9ktGrVSl9//fV3Xqtdu3Zq2bJlnWtERETUOf/N9IBre5XJZKrzsVS3V5WVlam6uvq6a107Q/9dru1V1dXVKisru+4XLcAXXb58WU899ZQCAgI0d+7cbz2OfnNz6DfNgxks1GGxWPTf//3fevfdd13/vffeezKbzQoICNDjjz+urVu36q9//at27dqlvLy8mzpveHh4nQZw8eLF69b9Xj2jFR4eroqKCp0/f941dvr0aZnNZklXAtrFixddr505c+a6a54+fdr17y+++ELh4eE3VSsAY7Rv3951L8PVwsPDVVJS4rpvQKr7890Q186Ih4eH64svvqhz/pvpAWFhYXV6ldPpVElJybdeq127dgoICLjuWg3tVadPn1ZAQIDatWt3w1qBW53T6dScOXN05swZvfDCC9/5B136Df3GkxCwUMcDDzygzMxMnTp1StKV9cEFBQWSpL179+rjjz9WbW2tWrduLX9/f9dfhH7wgx/oxIkT33reuLg47dy5UwcPHlRVVZVeeOEFOZ3Obz2+ffv26tOnj1auXKlLly7po48+Um5urhITEyVJUVFR2r17t8rLy/Xll19qzZo1153jL3/5i0pKSlReXq6XXnrJtXYaQMPV1NTo0qVLunz5smpra3Xp0iXXDdTXmjBhgl5++WW9//77cjqd+uyzz3Tq1Cn17NlTrVq1Uk5Ojqqrq7Vv3z7t3Lnzpn9Gb9RvJGnUqFHKysrS2bNndfbsWb344osaM2bMDc89ZMgQffzxxyooKFBNTY3+/Oc/1/tLyjf8/PwUHx+v559/XufPn9epU6f0xz/+sU6vOnDggL744gudO3dOq1atuu4cmzZt0vHjx3XhwgX99re/VVxcnPz8/G5YK3Crmz9/vv75z3/qpZdeUqtWrb7zWPoN/caTELBQx8MPPyyr1apJkyapT58++ulPf6qjR49KuvKXkLS0NN1zzz1KSEjQT37yE40dO9b1edu3b1e/fv3qvWGyS5cumjt3rtLT0zVo0CDddtttCgkJUWBg4LfWsnLlSp06dUqDBg3S448/rmnTprkeJT927Fh1797dVWt9jXL06NGaNGmShg8frjvvvFNTpkwx4ksE+LSsrCz17NlT2dnZ2rRpk3r27KmsrKx6jx05cqQeffRRPfnkk7r77rs1depUVVRUKDAwUC+99JIKCwt17733auHChXr22We/9d6Kaz3++OOaPXu2+vbtq61bt9Z7zGOPPaYePXooMTFRiYmJio6O1mOPPXbDc4eEhOi3v/2tnnvuOfXv31/Hjx9Xjx49vvMv53PnzlVQUJCGDx+un/3sZxo9erTGjx8vSRowYIASEhKUmJio5ORkDR069LrPHzt2rGbPnq0BAwaoqqpKc+bMuamvA3ArO3XqlF577TV9+OGHGjhw4A336qTf0G88icn5XdMIQBP56quv1K9fP23fvl2RkZGGn98T9/YC4H0uX76swYMHa8WKFbr33nsNP//EiROVmJiolJQUw88NwLvQb24dzGCh2ezcuVMXLlzQ119/rWeeeUZdu3b1qM0CAUCSioqKVFlZqaqqKr300kuSrjzlFACMRr+5NfEUQTQbm82mp556Sk6nUz169NDKlSuvu1EUANzt8OHDmjlzpqqqqtS5c2e9+OKLN7z/AwAagn5za2KJIAAAAAAYhCWCAAAAAGAQjwhY//mf/+nuEgD4EHoOgOZCvwF8j0cErLKyMneXAMCH0HMANBf6DeB7PCJgAQAAAMCtgIAFAAAAAAYhYAEAAACAQQhYAAAAAGAQAhYAAAAAGISABQAAAAAGIWAB8CgZGRmKiYnR6NGj64y/+uqrio+P16hRo/Tss8+6xletWqXY2FjFxcWpqKioucsF4MW+rd9I0ssvv6xu3brp7NmzkiSn06klS5YoNjZWY8aM0QcffNDc5QLwEgQsAB4lOTlZOTk5dcb27t0rm82mTZs2KT8/37Vx5/Hjx5Wfn6/8/Hzl5ORo4cKFqq2tdUfZALxQff1Gkk6fPq23335bERERrrHCwkIVFxdrx44dWrx4sRYsWNCMlQLwJgQsAB6lX79+atu2bZ2x//mf/9HkyZMVGBgoSQoNDZUk2Ww2jRo1SoGBgYqMjFTHjh119OjRZq8ZgHeqr99I0vLlyzVr1iyZTCbXmM1mU1JSkkwmk3r37q3Kyko5HI7mLBeAlyBgweM5ay65uwT8i7u+F8XFxXr33XeVkpKihx56yBWi7Ha7LBaL6ziz2Sy73e6WGnHroOd4Dnd8LwoKChQeHq7u3bvXGb+231gsFvoNGo1+4zmM/F74G3YmoImY/Fvq80U/dncZkHTnvPfcct3a2lpVVFTo9ddf13vvvafp06fLZrO5pRbc+ug5nqO5e86FCxe0atUqvfzyy816Xfgu+o3nMLLfELAAeDyz2azY2FiZTCb17NlTLVq0UFlZmcxms0pKSlzH2e12mc1mN1YKwJt9/vnnOnnypMaOHStJKikpUXJystauXXtdvykpKaHfAKgXSwQBeLzhw4dr3759kqRPP/1U1dXVateunaxWq/Lz81VVVaUTJ06ouLhYPXv2dHO1ALxVt27dtGfPHu3cuVM7d+6UxWLR+vXrFRYWJqvVqry8PDmdTh0+fFht2rRReHi4u0sG4IGYwQLgUdLT07V//36VlZVp8ODBmjZtmsaPH69f/epXGj16tAICAvT000/LZDKpS5cuGjlypBISEuTn56d58+bJz8/P3W8BgJeor9+kpKTUe+yQIUO0e/duxcbGKigoSMuWLWvmagF4ixsGrIyMDO3atUuhoaHasmWLJOmZZ57RW2+9pYCAAN15551avny5goODJV3ZkyY3N1ctWrTQr3/9aw0aNKhp3wGAW8rKlSvrHV+xYkW941OmTNGUKVOasiQAt6hv6zff2Llzp+vfJpNJ8+fPb+qSANwCbrhEsL49IgYMGKAtW7Zo8+bNuuuuu7Rq1SpJ7EkDAAAAwLfdMGDVt0fEwIED5e9/ZfKrd+/erps+2ZMGAAAAgC9r9EMu1q1bp8GDB0tiTxoAAAAAvq1RASsrK0t+fn5KTEw0qh4AAAAA8FoNforg+vXrtWvXLq1evVomk0mS2JMGAAAAgE9r0AxWYWGhcnJylJWVpaCgINc4e9IAAAAA8GU3nMGqb4+I7OxsVVVVKTU1VZLUq1cvLVq0iD1pAAAAAPi0Gwas+vaI+LZN+CT2pAEAAADguxr9FEEAAAAAwBUELAAAAAAwCAELAAAAAAxCwAIAAAAAgxCwAAAAAMAgBCwAAAAAMAgBCwAAAAAMQsACAAAAAIMQsAAAAADAIAQsAB4lIyNDMTExGj169HWvvfzyy+rWrZvOnj0rSXI6nVqyZIliY2M1ZswYffDBB81dLgAAQB0ELAAeJTk5WTk5OdeNnz59Wm+//bYiIiJcY4WFhSouLtaOHTu0ePFiLViwoBkrBQAAuB4BC4BH6devn9q2bXvd+PLlyzVr1iyZTCbXmM1mU1JSkkwmk3r37q3Kyko5HI7mLBcAAKAOAhYAj1dQUKDw8HB17969zrjdbpfFYnF9bLFYZLfbm7s8AF6qviXJzzzzjOLj4zVmzBhNnTpVlZWVrtdWrVql2NhYxcXFqaioyB0lA/ACBCwAHu3ChQtatWqVnnjiCXeXAuAWU9+S5AEDBmjLli3avHmz7rrrLq1atUqSdPz4ceXn5ys/P185OTlauHChamtr3VE2AA9HwALg0T7//HOdPHlSY8eOldVqVUlJiZKTk/Xll1/KbDarpKTEdWxJSYnMZrMbqwXgTepbkjxw4ED5+/tLknr37u3qMTabTaNGjVJgYKAiIyPVsWNHHT16tNlrBuD5CFgAPFq3bt20Z88e7dy5Uzt37pTFYtH69esVFhYmq9WqvLw8OZ1OHT58WG3atFF4eLi7SwZwi1i3bp0GDx4s6folyWazmSXJAOrl7+4CAOBq6enp2r9/v8rKyjR48GBNmzZNKSkp9R47ZMgQ7d69W7GxsQoKCtKyZcuauVoAt6qsrCz5+fkpMTHR3aUA8DIELAAeZeXKld/5+s6dO13/NplMmj9/flOXBMDHrF+/Xrt27dLq1atdTy69dkmy3W5nSTKAerFEEAAA4F8KCwuVk5OjrKwsBQUFucatVqvy8/NVVVWlEydOqLi4WD179nRjpQA8FTNYAADAJ9W3JDk7O1tVVVVKTU2VJPXq1UuLFi1Sly5dNHLkSCUkJMjPz0/z5s2Tn5+fm98BAE9EwAIAAD6pviXJ33bPpyRNmTJFU6ZMacqSANwCWCIIAAAAAAYhYAEAAACAQQhYAAAAAGCQGwasjIwMxcTEaPTo0a6x8vJypaamasSIEUpNTVVFRYUkyel0asmSJYqNjdWYMWP0wQcfNF3lAAAAAOBhbhiwkpOTlZOTU2csOztbMTEx2rFjh2JiYpSdnS3pyqNNi4uLtWPHDi1evFgLFixokqIBAAAAwBPdMGD169dPbdu2rTNms9mUlJQkSUpKSlJBQUGdcZPJpN69e6uyslIOh6MJygYAAAAAz9Oge7BKS0sVHh4uSQoLC1NpaamkK7uaWywW13EWi0V2u92AMgEAAADA8zX6IRcmk0kmk8mIWgAAAADAqzUoYIWGhrqW/jkcDoWEhEiSzGazSkpKXMeVlJTIbDYbUCYAAAAAeL4GBSyr1aq8vDxJUl5enoYNG1Zn3Ol06vDhw2rTpo1rKSEAAAAA3Or8b3RAenq69u/fr7KyMg0ePFjTpk3T5MmTNX36dOXm5ioiIkKZmZmSpCFDhmj37t2KjY1VUFCQli1b1uRvAAAAAAA8xQ0D1sqVK+sdX7NmzXVjJpNJ8+fPb3xVAAAAAOCFGv2QCwAAAADAFQQsAAAAADAIAQuAR8nIyFBMTIxGjx7tGnvmmWcUHx+vMWPGaOrUqaqsrHS9tmrVKsXGxiouLk5FRUXuKBkAAMCFgAXAoyQnJysnJ6fO2IABA7RlyxZt3rxZd911l1atWiVJOn78uPLz85Wfn6+cnBwtXLhQtbW17igbAABAEgELgIfp16+f2rZtW2ds4MCB8ve/8kye3r17u/bbs9lsGjVqlAIDAxUZGamOHTvq6NGjzV4zAADANwhYALzKunXrNHjwYEmS3W6XxWJxvWY2m2W3291VGgAAAAELgPfIysqSn5+fEhMT3V0KAABAvW64DxYAeIL169dr165dWr16tUwmk6QrM1bfLBeUrsxomc1md5UIAADADBYAz1dYWKicnBxlZWUpKCjINW61WpWfn6+qqiqdOHFCxcXF6tmzpxsrBQAAvo4ZLAAeJT09Xfv371dZWZkGDx6sadOmKTs7W1VVVUpNTZUk9erVS4sWLVKXLl00cuRIJSQkyM/PT/PmzZOfn5+b3wEAb5GRkaFdu3YpNDRUW7ZskSSVl5drxowZOnXqlO644w5lZmaqbdu2cjqdWrp0qXbv3q1WrVrp6aefVnR0tJvfAQBPRMAC4FFWrlx53VhKSsq3Hj9lyhRNmTKlKUsCcItKTk7WQw89pF/+8peusezsbMXExGjy5MnKzs5Wdna2Zs2apcLCQhUXF2vHjh06cuSIFixYoLVr17qxegCeiiWCAADAJ9W3LYTNZlNSUpIkKSkpSQUFBXXGTSaTevfurcrKSjkcjmavGYDnI2ABAAD8S2lpqcLDwyVJYWFhKi0tlXT9thAWi4VtIQDUi4AFAABQD5PJ5HpqKQDcLAIWAADAv4SGhrqW/jkcDoWEhEi6fluIkpIStoUAUC8CFgAAwL9YrVbl5eVJkvLy8jRs2LA6406nU4cPH1abNm1cSwkB4Go8RRAAAPik+raFmDx5sqZPn67c3FxFREQoMzNTkjRkyBDt3r1bsbGxCgoK0rJly9xcPQBPRcACAAA+qb5tISRpzZo1142ZTCbNnz+/qUsCcAtgiSAAAAAAGISABQAAAAAGIWABAAAAgEEIWAAAAABgEAIWAAAAABiEgAUAAAAABiFgAQAAAIBBGrUP1urVq7V27VqZTCZ17dpVy5cvl8PhUHp6usrLyxUdHa1nn31WgYGBRtULAAAAAB6rwTNYdrtdr7zyitatW6ctW7aotrZW+fn5WrFihR555BG9+eabCg4OVm5urpH1AgAAAIDHatQSwdraWl28eFE1NTW6ePGiwsLCtHfvXsXFxUmSxo0bJ5vNZkihAAAAAODpGhywzGazJk2apKFDh2rgwIFq3bq1oqOjFRwcLH//KysPLRaL7Ha7YcUCuPVlZGQoJiZGo0ePdo2Vl5crNTVVI0aMUGpqqioqKiRJTqdTS5YsUWxsrMaMGaMPPvjAXWUDAABIakTAqqiokM1mk81mU1FRkS5cuKCioiIjawPgg5KTk5WTk1NnLDs7WzExMdqxY4diYmKUnZ0tSSosLFRxcbF27NihxYsXa8GCBW6oGAAA4P81OGC988476tChg0JCQhQQEKARI0bo4MGDqqysVE1NjSSppKREZrPZsGIB3Pr69euntm3b1hmz2WxKSkqSJCUlJamgoKDOuMlkUu/evVVZWSmHw9HsNQMAAHyjwQErIiJCR44c0YULF+R0OrVnzx517txZ/fv31/bt2yVJGzZskNVqNaxYAL6ptLRU4eHhkqSwsDCVlpZKuvKwHYvF4jqOZckAAMDdGvyY9l69eikuLk7jxo2Tv7+/oqKidP/99+u+++7TjBkzlJmZqaioKKWkpBhZLwAfZzKZZDKZ3F0GAABAvRq1D1ZaWprS0tLqjEVGRvJodgCGCg0NlcPhUHh4uBwOh0JCQiRdedhOSUmJ6ziWJQMAAHdr1GPaAaA5WK1W5eXlSZLy8vI0bNiwOuNOp1OHDx9WmzZtXEsJAQAA3KFRM1gAYLT09HTt379fZWVlGjx4sKZNm6bJkydr+vTpys3NVUREhDIzMyVJQ4YM0e7duxUbG6ugoCAtW7bMzdUDAABfR8AC4FFWrlxZ7/iaNWuuGzOZTJo/f35TlwQAAHDTWCIIAAAAAAYhYAEAAACAQVgiCAAAcI3Vq1dr7dq1MplM6tq1q5YvXy6Hw6H09HSVl5crOjpazz77rAIDA91dKgAPwwwWAADAVex2u1555RWtW7dOW7ZsUW1trfLz87VixQo98sgjevPNNxUcHMy2NADqRcACAAC4Rm1trS5evKiamhpdvHhRYWFh2rt3r+Li4iRJ48aNk81mc3OVADwRSwQBAACuYjabNWnSJA0dOlQtW7bUgAEDFB0dreDgYPn7X/nVyWKxyG63u7lSAJ6IGSwAAICrVFRUyGazyWazqaioSBcuXFBRUZG7ywLgJZjBAgAAuMo777yjDh06KCQkRJI0YsQIHTx4UJWVlaqpqZG/v79KSkpkNpvdXCkAT8QMFgAAwFUiIiJ05MgRXbhwQU6nU3v27FHnzp3Vv39/bd++XZK0YcMGWa1WN1cKwBMxgwUAAHCVXr16KS4uTuPGjZO/v7+ioqJ0//3367777tOMGTOUmZmpqKgopaSkuLtUAB6IgAUAAHCNtLQ0paWl1RmLjIzk0ewAboglggAAAABgEAIWAAAAABiEgAUAAAAABiFgAQAAAIBBCFgAAAAAYBACFgAAAAAYhMe0A/Aaq1ev1tq1a2UymdS1a1ctX75cDodD6enpKi8vV3R0tJ599lkFBga6u1QAAOCjmMEC4BXsdrteeeUVrVu3Tlu2bFFtba3y8/O1YsUKPfLII3rzzTcVHBzMHjUAAMCtCFgAvEZtba0uXryompoaXbx4UWFhYdq7d6/i4uIkSePGjZPNZnNzlQAAwJexRBCAVzCbzZo0aZKGDh2qli1basCAAYqOjlZwcLD8/a+0MovFIrvd7uZKAQCAL2vUDFZlZaXS0tIUHx+vkSNH6tChQyovL1dqaqpGjBih1NRUVVRUGFUrAB9WUVEhm80mm82moqIiXbhwQUVFRe4uCwAAoI5GBaylS5dq0KBB2rZtmzZu3KhOnTopOztbMTEx2rFjh2JiYpSdnW1UrQB82DvvvKMOHTooJCREAQEBGjFihA4ePKjKykrV1NRIkkpKSmQ2m91cKQAA8GUNDljnzp3TgQMHNGHCBElSYGCggoODZbPZlJSUJElKSkpSQUGBMZUC8GkRERE6cuSILly4IKfTqT179qhz587q37+/tm/fLknasGGDrFarmysFAAC+rMH3YJ08eVIhISHKyMjQRx99pOjoaM2ZM0elpaUKDw+XJIWFham0tNSwYgH4rl69eikuLk7jxo2Tv7+/oqKidP/99+u+++7TjBkzlJmZqaioKKWkpLi7VAAA4MMaHLBqamp07NgxzZ07V7169dKSJUuuWw5oMplkMpkaXSQASFJaWprS0tLqjEVGRvJodgAA4DEavETQYrHIYrGoV69ekqT4+HgdO3ZMoaGhcjgckiSHw6GQkBBjKgUAAAAAD9fggBUWFiaLxaJPPvlEkrRnzx516tRJVqtVeXl5kqS8vDwNGzbMmEoBAAAAwMM1ah+suXPnaubMmaqurlZkZKSWL1+uy5cva/r06crNzVVERIQyMzONqhUAAAAAPFqjAlZUVJTWr19/3fiaNWsac1oAAAAA8EqN2gcLAAAAAPD/CFgAAAAAYBACFgAAwDUqKyuVlpam+Ph4jRw5UsWKW5MAAA7SSURBVIcOHVJ5eblSU1M1YsQIpaamqqKiwt1lAvBABCwAAIBrLF26VIMGDdK2bdu0ceNGderUSdnZ2YqJidGOHTsUExNz3f6fACARsAAAAOo4d+6cDhw4oAkTJkiSAgMDFRwcLJvNpqSkJElSUlKSCgoK3FkmAA/VqKcIAgAA3GpOnjypkJAQZWRk6KOPPlJ0dLTmzJmj0tJShYeHS7qyH2hpaambKwXgiZjBAgAAuEpNTY2OHTumBx54QHl5eQoKCrpuOaDJZJLJZHJThQA8GQELAADgKhaLRRaLRb169ZIkxcfH69ixYwoNDZXD4ZAkORwOhYSEuLNMAB6KgAUAAHCVsLAwWSwWffLJJ5KkPXv2qFOnTrJarcrLy5Mk5eXladiwYe4sE4CH4h4sAACAa8ydO1czZ85UdXW1IiMjtXz5cl2+fFnTp09Xbm6uIiIilJmZ6e4yAXggAhYAAMA1oqKitH79+uvG16xZ44ZqAHgTlggCAAAAgEEIWAAAAABgEAIWAK9RWVmptLQ0xcfHa+TIkTp06JDKy8uVmpqqESNGKDU1VRUVFe4uEwAA+DACFgCvsXTpUg0aNEjbtm3Txo0b1alTJ2VnZysmJkY7duxQTEzMdXvVAAAANCcCFgCvcO7cOR04cEATJkyQJAUGBio4OFg2m01JSUmSpKSkJBUUFLizTAAA4ON4iiAAr3Dy5EmFhIQoIyNDH330kaKjozVnzhyVlpYqPDxc0pW9a0pLS91cKQAA8GXMYAHwCjU1NTp27JgeeOAB5eXlKSgo6LrlgCaTSSaTyU0VAgAAELAAeAmLxSKLxaJevXpJkuLj43Xs2DGFhobK4XBIkhwOh0JCQtxZJgAA8HEELABeISwsTBaLRZ988okkac+ePerUqZOsVqvy8vIkSXl5eRo2bJg7ywQAAD6Oe7AAeI25c+dq5syZqq6uVmRkpJYvX67Lly9r+vTpys3NVUREhDIzM91dJgAA8GEELABeIyoqSuvXr79ufM2aNW6oBgAA4HosEQQAAAAAgxCwAAAAAMAgBCwAAAAAMEijA1Ztba2SkpL0X//1X5KkEydOKCUlRbGxsZo+fbqqqqoaXSQAAAAAeINGB6xXXnlFnTp1cn28YsUKPfLII3rzzTcVHBys3Nzcxl4CAAAAALxCowJWSUmJdu3apQkTJkiSnE6n9u7dq7i4OEnSuHHjZLPZGl8lAAAAAHiBRgWsZcuWadasWWrR4sppysrKFBwcLH//K09/t1gsstvtja8SAAAAALxAgwPWW2+9pZCQEPXo0cPIegAAAADAazV4o+GDBw9q586dKiws1KVLl3T+/HktXbpUlZWVqqmpkb+/v0pKSmQ2m42sFwAAAAA8VoNnsJ588kkVFhZq586dWrlype6991795je/Uf/+/bV9+3ZJ0oYNG2S1Wg0rFgAAAAA8meH7YM2aNUt//OMfFRsbq/LycqWkpBh9CQAAgCbHVjQAGqLBSwSv1r9/f/Xv31+SFBkZyaPZAQCA1/tmK5rz589L+v+taEaNGqV58+YpNzdXP/vZz9xcJQBPY/gMFgAAgLdjKxoADUXAAgAAuAZb0QBoKAIWAADAVdiKBkBjGHIPFgAAwK2CrWgANAYzWAAAAFdhKxoAjUHAAuBVeGwyAHdhKxoAN4MlggC8Co9NBtCc2IoGwPfFDBYAr8FjkwEAgKcjYAHwGjw2GQAAeDoCFgCvwGOTAQCAN+AeLABegccmAwAAb8AMFgCvwGOTAQCANyBgAfBqPDYZAAB4EpYIAvA6PDYZAAB4KmawAAAAAMAgBCwAAAAAMAgBCwAAAAAMQsACAAAAAIMQsAAAAADAIAQsAAAAADAIAQsAAAAADELAAgAAAACDELAAAAAAwCAELAAAAAAwiH9DP/H06dN66qmnVFpaKpPJpJ/+9Kf6+c9/rvLycs2YMUOnTp3SHXfcoczMTLVt29bImgEAAADAIzV4BsvPz0+zZ8/W1q1b9dprr+kvf/mLjh8/ruzsbMXExGjHjh2KiYlRdna2kfUCAAAAgMdqcMAKDw9XdHS0JKl169b60Y9+JLvdLpvNpqSkJElSUlKSCgoKjKkUAAAAADycIfdgnTx5Uh9++KF69eql0tJShYeHS5LCwsJUWlpqxCUAAAAAwOM1+B6sb3z11VdKS0vTr371K7Vu3brOayaTSSaTqbGXAAAAaDbcZw6gMRo1g1VdXa20tDSNGTNGI0aMkCSFhobK4XBIkhwOh0JCQhpfJQAAQDPhPnMAjdHggOV0OjVnzhz96Ec/UmpqqmvcarUqLy9PkpSXl6dhw4Y1vkoAAIBmwn3mABqjwUsE//73v2vjxo3q2rWrxo4dK0lKT0/X5MmTNX36dOXm5ioiIkKZmZmGFQvAd7FkB4A7cJ85gO+rwQGrb9+++vjjj+t9bc2aNQ0uCADq882SnejoaJ0/f17jx4/XgAEDtH79esXExGjy5MnKzs5Wdna2Zs2a5e5yAdwCuM8cQEMY8hRBAGhqLNkB0Jy4zxxAQxGwAHgdluwAaErcZw6gMRr9mHYAaE4s2QHQ1LjPHEBjELAAeI3vWrITHh7Okh0AhuA+cwCNwRJBAF6BJTsAAMAbMIMFwCuwZAcAAHgDAhYAr8CSHQAA4A1YIggAAAAABiFgAQAAAIBBCFgAAAAAYBACFgAAAAAYhIAFAAAAAAYhYAEAAACAQQhYAAAAAGAQAhYAAAAAGISABQAAAAAGIWABAAAAgEEIWAAAAABgEAIWAAAAABiEgAUAAOAhLlXXursE/AvfCzSUv7sLAAAAwBUtA/x0z6xX3F0GJP39uYfdXQK8FDNYAAAAAGAQAhYAAAAAGISABQA3wDp8z8H3AgDg6bz2HqxL1bVqGeDn7jIgvhe49XFPhOfgnggAgKdrsoBVWFiopUuX6vLly0pJSdHkyZMNPT+/8HgOfuGBuzV1vwGAb9BvANxIkywRrK2t1aJFi5STk6P8/Hxt2bJFx48fb4pLAfBx9BsAzYV+A+BmNEnAOnr0qDp27KjIyEgFBgZq1KhRstlsTXEpAD6OfgOgudBvANyMJlkiaLfbZbFYXB+bzWYdPXr0W48/deqUkpOTv/d1OjaoOhgtOTmvGa7SpRmugRtqwM9pu3bt9Ic//KEJirni+/YbqWE9h37jGZqn30j0HA/xPX9O6TcwEv3Gxxj4O45HPORi37597i4BgA+h5wBoLvQbwPc0yRJBs9mskpIS18d2u11ms7kpLgXAx9FvADQX+g2Am9EkAevHP/6xiouLdeLECVVVVSk/P19Wq7UpLgXAx9FvADQX+g2Am9EkSwT9/f01b948/eIXv1Btba3Gjx+vLl1YXwrAePQbAM2FfgPgZpicTqfT3UUAAAAAwK2gSZYIAgAAAIAvImABAAAAgEEIWG5SWFiouLg4xcbGKjs7293loAllZGQoJiZGo0ePdncp8FH0G99Bv4G70W98Cz2nfgQsN6itrdWiRYuUk5Oj/Px8bdmyRcePH3d3WWgiycnJysnJcXcZ8FH0G99Cv4E70W98Dz2nfgQsNzh69Kg6duyoyMhIBQYGatSoUbLZbO4uC02kX79+atu2rbvLgI+i3/gW+g3ciX7je+g59SNguYHdbpfFYnF9bDabZbfb3VgRgFsV/QZAc6HfAFcQsAAAAADAIAQsNzCbzSopKXF9bLfbZTab3VgRgFsV/QZAc6HfAFcQsNzgxz/+sYqLi3XixAlVVVUpPz9fVqvV3WUBuAXRbwA0F/oNcIXJ6XQ63V2EL9q9e7eWLVum2tpajR8/XlOmTHF3SWgi6enp2r9/v8rKyhQaGqpp06YpJSXF3WXBh9BvfAf9Bu5Gv/Et9Jz6EbAAAAAAwCAsEQQAAAAAgxCwAAAAAMAgBCwAAAAAMAgBCwAAAAAMQsACAAAAAIMQsOARZs+erW3btrm7DAA+gH4DoDnRc3wPAQteqaamxt0lAPAR9BsAzYme4/383V0AvM+LL76oTZs2KSQkRO3bt1d0dLRiY2O1cOFClZWVqVWrVlq8eLE6deqk2bNnq3Xr1nr//ff15ZdfatasWYqPj5fT6dTixYv19ttvq3379goICHCd//3339fTTz+tr7/+Wu3atdPy5csVHh6uiRMnqnv37vr73/+u0aNHa9KkSW78KgBoDvQbAM2JngNDOIHv4ciRI87ExETnxYsXnefOnXPGxsY6c3JynA8//LDz008/dTqdTufhw4edEydOdDqdTucvf/lL57Rp05y1tbXOf/zjH87hw4c7nU6nc/v27c5HHnnEWVNT4ywpKXHec889zjfeeMNZVVXlvP/++52lpaVOp9PpzM/Pd86ePdvpdDqdDz30kHP+/PnN/p4BuAf9BkBzoufAKMxg4Xs5ePCghg0bppYtW6ply5YaOnSoLl26pEOHDumJJ55wHVdVVeX69/Dhw9WiRQt17txZZ86ckSQdOHBAo0aNkp+fn8xms+69915J0qeffqr//d//VWpqqiTp8uXLCgsLc50rISGhOd4mAA9AvwHQnOg5MAoBC412+fJlBQcHa+PGjfW+HhgYeNPncjqd6tKli1577bV6Xw8KCmpQjQBuDfQbAM2JnoOG4CEX+F7uvvtuvfXWW7p06ZK++uor7dq1S0FBQerQoYPeeOMNSVcayEcfffSd5+nXr5/eeOMN1dbWyuFwaN++fZKkH/7whzp79qwOHTokSaqurtY//vGPpn1TADwS/QZAc6LnwCjMYOF76dmzp6xWqxITExUaGqquXbuqTZs2eu6557RgwQJlZWWppqZGCQkJ6t69+7eeJzY2Vnv37lVCQoIiIiLUu3dvSVf+EvS73/1OS5Ys0blz51RbW6uf//zn6tKlS3O9RQAegn4DoDnRc2AUk9PpdLq7CHiXr776Sv/2b/+mCxcu6MEHH9TixYsVHR3t7rIA3ILoNwCaEz0HRmAGC9/bvHnzdPz4cV26dEnjxo2j8QBoMvQbAM2JngMjMIMFAAAAAAbhIRcAAAAAYBACFgAAAAAYhIAFAAAAAAYhYAEAAACAQQhYAAAAAGCQ/wMQ2mp7OIfm2AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Распределение пользователей в зависимости от пола\n",
    "fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))\n",
    "\n",
    "sns.countplot(data=users_test, x = 'gender', ax = axes[0])\n",
    "axes[0].set(title='Test group', ylabel='')\n",
    "\n",
    "sns.countplot(data=users_control_1, x = 'gender', ax = axes[1])\n",
    "axes[1].set(title='1 control group', ylabel='')\n",
    "\n",
    "sns.countplot(data=users_control_2, x = 'gender', ax = axes[2])\n",
    "axes[2].set(title='2 control group', ylabel='')\n",
    "\n",
    "sns.despine()\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Кажется, ранее было верно предположено, что 1 - это мужчины, а 0 - женщины, т.к. в дейтинг приложениях согласно статистике превалирует мужской пол.\n",
    "Распределение пользователей во всех группах примерно одинаковое."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Также сравним наши распределения с помощью статистического теста.\n",
    "\n",
    "Есть категориальные переменные (пол, группа пользователя) → нужен хи-квадрат\n",
    "\n",
    "- $H_0$: нет статистически значимой разницы между распределениями пользователей по половому признаку в тестовой и контрольных группах (p-value > 0.05)\n",
    "- $H_1$: есть статистически значимая разница между распределениями пользователей по половому признаку в тестовой и контрольных группах"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.72669415296182, 0.4217480941477504)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stat, p, dof, expected = chi2_contingency(pd.crosstab(users_merged.gender, users_merged.group))\n",
    "stat, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Согласно статистическому тесту (p-value > 0.05) также не обнаружена статистически значимая разница между распределениями пользователей по половому признаку в тестовой и контрольных группах, что говорит о репрезентативности выборок по этому признаку."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXxN597//9dOIqQhJCGJsYe7ZkqOROKYQ8gxJVG0tKbWrXdxYipNedRpKaqtSs3N4zZTdyslMUsR4VYzRVv3OaipSEIkppBkJ/v3h6/9a2roNuzsRd7Pf2Svta61Pmuvbb/3uta11zZZLBYLIiIiBuPk6AJERETuRwElIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJICih5bB07dmTPnj2OLsOwfvvtN2rWrInZbHZ0KQXMmDGDd999F4ALFy7g7+9PXl6eg6sSuZcCSu7rrbfe4ssvv7xn+ubNm2natClms5l169YRFBTkgOoco2bNmpw5c8bRZTxVFSpU4NChQzg7Ozu6FJF7KKDkviIjI1m9ejV//B736tWr6dy5My4uLg6qTOThjHbGKo9PASX31bZtWzIzM9m/f7912tWrV0lKSiIiIgKAkJAQfvjhBwDy8/OJjY2lbdu2BAUFMXToUDIzMwF47733mD9/PgCpqanUrFmTZcuWAXD27FkaN25Mfn4+V65c4e233yYgIIDGjRvTq1cv8vPz71vf8ePH6d+/P40bN+Zvf/sbc+fOBSAnJ4eJEyfSrFkzmjVrxsSJE8nJyQFg5cqV9OzZs8B6fn9WFB0dzUcffcTAgQPx9/ene/funD17FoDXX38dgPDwcPz9/Vm/fv09NeXl5TFlyhSCgoJo06YNycnJBeZfv36dMWPG0KxZM5o3b860adOsXWsrV67ktddeY/z48TRq1IiwsDB27dplc9uePXsyZcoUAgMDCQkJKbDtc+fO8cYbb+Dv70///v3JyMiwzvtjN2Tv3r2JiYnhtddew9/fnzfffJMrV65Yl4+Pj6d169YEBQUxa9asAq+BP7p+/TqjR48mODiY1q1bM3v2bOvx/LOa/+jnn38mIiICf39/oqKiGDZsGNOmTQNgz549tGjRgtjYWJo2bcr777//xK+DcePG0b9/f/z9/XnjjTc4f/78A2sT+1FAyX2VKFGCv//978THx1unbdiwgWrVqlGrVq17ll+yZAmbN29m6dKl7Nixg9KlSzN+/HgAAgMD2bt3LwB79+6lcuXK7Nu3z/q4UaNGODk5sWDBAnx9fdm1axc7d+5kxIgRmEyme7Z148YN+vfvT/PmzdmxYweJiYk0adIEgDlz5nD48GESEhJYvXo1R48eZfbs2Tbv9/r16xkyZAj79u2jSpUq1jfBu4GakJDAoUOH6NChwz1tv/32W5KSkoiPj+e7775j48aNBeZHR0fj4uJCYmIi8fHx7Ny5kxUrVljnHzlyhCpVqrB7926ioqIYMmSINeRtaVu1alV2797NgAEDGDt2rPXs991336Vu3brs2bOHQYMGsWrVqoc+B2vXrmXy5Mns2rWL3Nxc64eLEydO8NFHH/HZZ5+xY8cObty4QWpq6gPXM2HCBK5fv87mzZtZsmQJCQkJfPfddzbV/Hs5OTkMGTKEyMhI9u7dS6dOndi8eXOBZS5fvmz9ADVhwoQnfh2sWbOGQYMGsWfPHmrVqmW9ZieFSwElDxQREcGmTZvIzs4G7nx6joyMvO+y//M//8Pw4cPx8/PD1dWVIUOGsGnTJsxmM40bN+bAgQPk5+ezb98+BgwYwMGDBwHYt28fjRs3BsDFxYVLly5x4cIFihUrRkBAwH0Datu2bZQtW5Y333yT4sWLU7JkSRo0aADceWMZPHgw3t7eeHl5MXjwYFavXm3zPrdt25aXX34ZFxcXunTpwrFjx2xuu2HDBvr27Uv58uUpU6YMb7/9tnXe5cuXSU5OZsyYMbzwwgt4e3vTr18/1q1bZ13Gy8uLvn37UqxYMTp06EDVqlXZtm2bTW0rVKhAjx49cHZ2JjIykkuXLnH58mUuXLjA0aNHGTp0KK6urtazlYfp2rUrVatWpUSJEoSFhVmfg40bN9K6dWsCAgJwdXUlKirqvscH7pxNrl+/npEjR1KyZEkqVapE//79CxyLB9X8R4cPH8ZsNtOnTx+KFStGu3btqF+/foFlnJyciIqKwtXVlRIlSjzx66BVq1YEBgbi6urK8OHD+fHHH7l48aLN7eXp0IUEeaCAgAA8PT3ZvHkz9evX5+jRo8ycOfO+y164cIHBgwfj5PT/f+ZxcnIiPT2dKlWq4ObmxrFjxzhw4ACDBw8mLi6OX3/9lX379tG7d2/gzsCMmTNn8uabbwLw6quvMnDgwHu2dfHiRapUqXLfOtLS0qhQoYL1cYUKFUhLS7N5n8uWLWv9u0SJEmRlZdncNi0tjfLlyxfY9l0XLlzAbDbTrFkz67T8/PwCy/v6+hZ4w79buy1tf1+3m5sbAFlZWWRkZODh4cELL7xQYL0Pe7MtV65cgXXdfQ7S0tLw8/MrMK9MmTL3XUdGRga5ubn3HIvfn3E9qOY/SktLu+e5+f2+A3h6elK8ePECbZ7kdfD7/XR3d6d06dL3HF+xPwWUPFR4eDjx8fGcOnWKZs2aFXhT+T0/Pz8mTZpEo0aN7js/MDCQTZs2kZubi6+vL4GBgcTHx3P16lVq164NQMmSJYmOjiY6Opp///vf9O3bl/r161u77+4qX778fa8BAfj4+HDhwgWqV68O3AkzHx8f4M6b4O3bt63LXrp06dGejD9Rrly5Am/8v//77pnl7t27HzjAJDU1FYvFYn0jvnjxIiEhITa1fVhN165dIysryxpSFy5ceOCZz8P4+Phw6tQp6+Pbt29buyD/yNPTk2LFinHhwgVeeukl6/74+vo+8nbLlSt33+emcuXK1mX+uD9P+jpISUmx/n3z5k2uXr1qbS+FR1188lARERHs2rWLb7/91jo44n569uxJTEyM9WLylStXClwnaNy4MUuXLiUgIACAoKAgli5dSqNGjaxDnJOSkjhz5gwWi4VSpUrh7Ox83zfSVq1acenSJRYuXEhOTg43btzg8OHDwJ3vZs2ZM4crV65w5coVZs2aRefOnQGoVasWx48f59ixY2RnZzNjxoxHei7Kli3LuXPnHjj/73//O0uWLCElJYWrV68SGxtrnefj40PTpk355JNPuHHjBvn5+Zw9e9Z6be7uc7Z48WJyc3PZsGEDJ0+epGXLlja1fZCKFStSr149ZsyYQU5ODvv37ycpKemR9vuu9u3bs3XrVg4ePEhOTg4zZsy47zUjAGdnZ8LCwpg2bRo3btzg/PnzLFiwgC5dujzydhs2bIizszNLly7FbDazefNmjh49+tA2T/o6SE5OZv/+/eTk5PDll1/SoEEDnT05gAJKHqpSpUr4+/tz69Yt2rRp88Dl+vTpQ0hICG+++Sb+/v706NGDI0eOWOcHBgZy8+ZNAgMDAWjUqBG3b9+2BhbAmTNnrCOnXn31VXr27ElwcPA92ypZsiTz588nKSmJpk2b0r59e+sXhgcNGkS9evXo0qULXbp0oW7dugwaNAiAqlWrMnjwYPr160e7du0eeLb3IEOGDCE6OpqAgID7nsH16NGDZs2aER4eTmRkJO3atSsw/9NPPyU3N5cOHToQGBhIVFRUgU/vL7/8MmfOnCE4OJiYmBimT5+Op6enTW0fZurUqRw+fNg68u5hHzQepnr16nzwwQeMGDGC5s2b88ILL+Dl5YWrq+t9l//ggw9wc3Ojbdu29OrVi06dOvHKK6888nZdXV2ZMWMGcXFxBAYGsnr1alq1avXA7cKTvw46derErFmzCAoK4ueff+azzz575LrlyZn0g4Uijrdy5UpWrFjB8uXLHV2Kze5+4Ni0aVOB7rbC0L17d1577bXHCrw/Ex0dja+vL8OHD3/q65ZHozMoEbHZ1q1buXXrFllZWUyZMoUaNWpQqVIlu2937969XLp0CbPZzKpVq/jXv/5F8+bN7b5dcSwNkhARm23ZsoXRo0djsVioV68eX3zxxWMNuHhUp06dYtiwYdy6dYtKlSoxffp0DVooAtTFJyIihqQuPhERMSQFlIiIGJICSkREDEkBJSIihqSAEhERQ1JAiYiIISmgRETEkBRQIiJiSAooERExJAWUiIgYkgJKREQMSQElIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJIz3xAvfXWW44uQURE7OCZD6iMjAxHlyAiInbwzAeUiIg8nxRQIiJiSAooERExJAWUiIgYkgJKREQMycXeGwgJCcHd3R0nJyecnZ1ZuXIlmZmZDB8+nPPnz1OxYkViYmIoXbo0FouFiRMnkpycTIkSJfjkk0+oW7euvUsUEREDKpQzqEWLFpGQkMDKlSsBiI2NpUmTJiQmJtKkSRNiY2MB2L59O6dPnyYxMZEJEybw4YcfFkZ5IiJiQA7p4tuyZQsREREAREREsHnz5gLTTSYTDRs25Nq1a6SlpTmiRBGRQrV7925GjBjB7t27HV2KYdi9iw/u3O3BZDLx6quv8uqrr5Keno6Pjw8A5cqVIz09HYDU1FT8/Pys7fz8/EhNTbUuKyLyvFq4cCHHjx8nKyuL4OBgR5djCHYPqOXLl+Pr60t6ejr9+/enWrVqBeabTCZMJpO9yxARMbSsrKwC/0ohdPH5+voC4O3tTWhoKEeOHMHb29vadZeWloaXl5d12ZSUFGvblJQUa3sRESla7BpQWVlZ3Lhxw/r3zp07qV69OiEhIcTHxwMQHx9PmzZtAKzTLRYLP/74I6VKlVL3nohIEWXXLr709HQGDx4MQF5eHp06daJFixbUr1+fYcOGERcXR4UKFYiJiQGgZcuWJCcnExoaipubG5MmTbJneSIiYmB2DajKlSuzevXqe6Z7enqyaNGie6abTCb++c9/2rMkERF5RuhOEiIiYkgKKBERMSQFlIiIGJICSkREDEkBJSIihqSAEhERQ1JAiYiIISmgRETEkBRQIiJiSAooERExJAWUiIgYkgJKREQMSQElIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJICigRETEkBZSIiBiSAkpERAxJASUiIoakgBIREUNSQImIiCEpoERExJAUUCIiYkgKKBERMSQFlIiIGJICSkREDEkBJSIihlQoAZWXl0dERARvv/02AOfOnaN79+6EhoYybNgwcnJyAMjJyWHYsGGEhobSvXt3fvvtt8IoT0REDKhQAmrx4sX8x3/8h/Xx559/Tr9+/fj+++/x8PAgLi4OgBUrVuDh4cH3339Pv379+PzzzwujPBERMSC7B1RKSgrbtm2jW7duAFgsFnbv3k379u0BiIyMZMuWLQBs3bqVyMhIANq3b8+uXbuwWCz2LlFERAzI7gE1adIkRo0ahZPTnU1lZGTg4eGBi4sLAH5+fqSmpgKQmppK+fLlAXBxcaFUqVJkZGTYu0QRETEguwZUUlISXl5e1KtXz56bERGR55CLPVd+8OBBtm7dyvbt28nOzubGjRtMnDiRa9euYTabcXFxISUlBV9fXwB8fX25ePEifn5+mM1mrl+/jqenpz1LFBERg7LrGdTIkSPZvn07W7du5YsvviA4OJipU6cSFBTEpk2bAFi1ahUhISEAhISEsGrVKgA2bdpEcHAwJpPJniWKiIhBOeR7UKNGjWLBggWEhoaSmZlJ9+7dAejWrRuZmZmEhoayYMEC3n33XUeUJyIiBmDXLr7fCwoKIigoCIDKlStbh5b/XvHixZk+fXphlSQiIgamO0mIiIghKaBERMSQFFAiImJICigRETEkBZSIiBiSAkpERAxJASUiIoakgBIREUNSQImIiCEpoERExJAUUCIiYkgKKBERMSQFlIiIGJICSkREDEkBJSIihqSAEhERQ1JAiYiIISmgRETEkBRQIiJiSAooERExJJsC6ty5czZNExEReVpsCqioqKh7pg0dOvSpFyMiInKXy8Nmnjx5khMnTnD9+nUSExOt02/cuEF2drbdixMRkaLroQF16tQptm3bxvXr10lKSrJOd3d3Z8KECXYvTkREiq6HBlTbtm1p27Ythw4dwt/fv7BqEhEReXhA3fXiiy8yd+5czp8/j9lstk6fPHmy3QoTEZGizaaAGjRoEI0aNaJJkyY4OzvbuyYRERHbAurWrVuMGjXK3rWIiIhY2TTMvFWrViQnJ9u7FhERESubzqAWL17MV199haurKy4uLlgsFkwmEwcPHrR3fSIiUkTZFFCHDh16rJVnZ2fz+uuvk5OTQ15eHu3btycqKopz584xYsQIMjMzqVu3Lp9++imurq7k5OQwevRofv75Z8qUKcO0adOoVKnSY21bRESebTYF1L59++47PTAw8KHtXF1dWbRoEe7u7uTm5tKrVy9atGjBggUL6NevHx07dmTcuHHExcXRq1cvVqxYgYeHB99//z3r1q3j888/JyYm5tH3SkREnnk2BdS8efOsf2dnZ3PkyBHq1q3L4sWLH9rOZDLh7u4OgNlsxmw2YzKZ2L17N1OnTgUgMjKSmTNn0qtXL7Zu3cqQIUMAaN++PePHj7d2J4qISNFiU0DNnTu3wOOLFy8yadIkmzaQl5dH165dOXv2LL169aJy5cp4eHjg4nJn035+fqSmpgKQmppK+fLl7xTm4kKpUqXIyMjAy8vL5h0SEZHnw2P93Iafnx8nT560aVlnZ2cSEhJITk7myJEj/Prrr4+zSRERKWJsOoOaMGGCtZstPz+fY8eOUadOnUfakIeHB0FBQfz4449cu3YNs9mMi4sLKSkp+Pr6AuDr68vFixfx8/PDbDZz/fp1PD09H3GXRETkeWBTQNWrV8/6t7OzMx07dqRRo0Z/2u7KlSu4uLjg4eHB7du3+eGHH/jP//xPgoKC2LRpEx07dmTVqlWEhIQAEBISwqpVq/D392fTpk0EBwfr+pOISBFlU0BFRkaSk5PD6dOnAahatapNK09LSyM6Opq8vDwsFgthYWG0bt2al156ieHDhxMTE0Pt2rXp3r07AN26dWPUqFGEhoZSunRppk2b9nh7JSIizzybAmrPnj1ER0dTsWJFLBYLFy9eZMqUKX86zLxWrVrEx8ffM71y5crExcXdM7148eJMnz7dxtJFROR5ZlNATZkyhXnz5lGtWjXgzu9EjRw5kpUrV9q1OBERKbpsGsWXm5trDSe408WXm5trt6JERERsHiQxduxYunTpAsCaNWsKDJwQERF52mwKqI8++ohly5axZMkSAAICAujVq5ddCxMRkaLNpoAym8306dOH/v37A3fuDpGTk2PXwkREpGiz6RpUv379uH37tvXx7du3rWElIiJiDzYFVHZ2tvWmrwDu7u7cunXLbkWJiIjYFFBubm78/PPP1sc//fQTJUqUsFtRIiIiNl2DGjNmDEOHDsXHxweLxcLly5d1lwcREbErmwLq5ZdfZsOGDZw6dQq48z2oYsWKWefv3LmTpk2b2qdCEREpkmz+uY1ixYpRo0YNatSoUSCcAD7//POnXpiIiBRtj/V7UH9ksViexmpERESsnkpA6ScxRETkaXsqASUiIvK0PZWAqlix4tNYjYiIiJVNAbVhwwZu3LgBwOzZsxkyZEiB70XNnDnTPtWJiEiRZVNAzZ49m5IlS7J//3527dpFt27d+PDDD+1cmoiIFGU2BZSzszMAycnJ9OjRg1atWun3oERExK5sCihfX1/GjRvH+vXradmyJTk5OeTn59u7NhERKcJsCqiYmBiaNWvGvHnz8PDwIDMzk9GjR9u7NhERKcJsCqivvvoKd3d3fHx8APDx8aFZs2Z2LUxE5FFkm7MdXYLwdI+DTffiq1y5MmvXruXjjz/G3d2dgIAAAgICaNu27VMrRETkSRR3KU7TGc/uPUFdM11xwolzmeee6f3Y+Y+dT21dNgXUK6+8wiuvvMKlS5fYsGED8+fP55tvvuHQoUNPrRAREZHfsymgxo4dy8mTJ/H29iYgIIDp06dTp04de9cmIiJFmE3XoDIzM8nLy8PDw4PSpUvj6emJi4tN2SYiIvJYbEqZWbNmAXDy5El27NhBnz59yMvLY/v27XYtTkREii6bAiopKYn9+/ezf/9+rl27RnBwMI0aNbJ3bSIiUoTZFFA7duwgICCAPn364Ovra++aREREbLsGNW7cOBo0aMDJkycBuH37tvXmsSIiIvZgU0B9++23REVFMW7cOABSUlIYPHiwXQsTEZGizaaAWrZsGcuXL6dkyZIA/OUvf+HKlSt/2u7ixYv07t2bDh060LFjRxYtWgTcGRXYv39/2rVrR//+/bl69Spw56fjP/74Y0JDQ+ncuXOBn/QQEZGixaaAcnV1xdXV1frYbDbbtHJnZ2eio6NZv34933zzDV9//TUnTpwgNjaWJk2akJiYSJMmTYiNjQVg+/btnD59msTERCZMmKCf9BARKcJsCqjAwEDmzp3L7du32blzJ0OHDiUkJORP2/n4+FC3bl0ASpYsSbVq1UhNTWXLli1EREQAEBERwebNmwGs000mEw0bNuTatWukpaU97r6JiMgzzKaAevfdd/Hy8qJGjRp88803tGzZkmHDhj3Shn777TeOHTtGgwYNSE9Pt954tly5cqSnpwOQmpqKn5+ftY2fnx+pqamPtB0REXk+2DTM3MnJiR49etCjR4/H2sjNmzeJiopizJgx1utYd5lMJkwm02OtV0REnl8PDaihQ4fy5Zdf0rlz5/vOX7NmzZ9uIDc3l6ioKDp37ky7du0A8Pb2Ji0tDR8fH9LS0vDy8gLu/DBiSkqKtW1KSoq+dyUiUkQ9NKDGjh0LQGRkJA0aNCjQ/WYLi8XC2LFjqVatGv3797dODwkJIT4+noEDBxIfH0+bNm2s05cuXUrHjh05fPgwpUqVsnYFiohI0fLQgLobDllZWYwbN47SpUvToUMHwsLCKFu27J+u/MCBAyQkJFCjRg3Cw8MBGDFiBAMHDmTYsGHExcVRoUIFYmJiAGjZsiXJycmEhobi5ubGpEmTnnT/RETkGWWyWCwWWxf+v//7PzZs2MCmTZvw8/Nj4cKFdizNNl27dmXlypWOLkNEDOBZ/qE/1+9dcbrpRL57PjmhOY4u57E9zR8stGkU313e3t6ULVuWMmXKWEfeiYiI2INNo/iWLVvGxo0buXLlCmFhYXz88ce89NJL9q5NRESKMJsCKiUlhTFjxlC7dm171yMiIgLYGFAjR460dx0iIiIFPNI1KBERkcKigBIREUNSQImIiCEpoERExJAUUCIiYkgKKBERMSQFlIiIGJICSkREDEkBJSIihqSAEhERQ1JAPYN2797NiBEj2L17t6NLERGxG5vuxSfGsnDhQo4fP05WVhbBwcGOLkdExC50BvUMysrKKvCviMjzSAElIiKGpIASERFDUkCJGJgGxEhRpkESIgamATFSlOkMSsTANCBGijIFlIiIGJICSkREDEkBJSIihqSAEhERQ1JAiYiIISmgRETEkBRQIiJiSEUuoLJz8xxdgvw/OhYi8jB2vZPE+++/z7Zt2/D29mbt2rUAZGZmMnz4cM6fP0/FihWJiYmhdOnSWCwWJk6cSHJyMiVKlOCTTz6hbt26T72m4sWcaTRq8VNfb2Eqdfk6zsDZy9ef6X058Fkfu2/DYs7G5FLc7tuRh9NxkMdh14Dq2rUrb7zxBu+99551WmxsLE2aNGHgwIHExsYSGxvLqFGj2L59O6dPnyYxMZHDhw/z4YcfsmLFCnuWJ0WAyaU4Z8fXd3QZj818xQtwwXzlzDO9H1XGHXV0CfIMsmsXX2BgIKVLly4wbcuWLURERAAQERHB5s2bC0w3mUw0bNiQa9eukZaWZs/yRETEwAr9GlR6ejo+Pj4AlCtXjvT0dABSU1Px8/OzLufn50dqamphlyciIgbh0EESJpMJk8nkyBJERMSgCj2gvL29rV13aWlpeHl5AeDr60tKSop1uZSUFHx9fQu7PBERMYhCD6iQkBDi4+MBiI+Pp02bNgWmWywWfvzxR0qVKmXtChQRkaLHrqP4RowYwd69e8nIyKBFixb84x//YODAgQwbNoy4uDgqVKhATEwMAC1btiQ5OZnQ0FDc3NyYNGmSPUsTERGDs2tAffHFF/edvmjRonummUwm/vnPf9qzHBER43L5w79S9O4kIfIsKeFsKfCvPL/Mtc3klc3DXNvs6FIMQ1n9DLI4uRT4V55fkX+5ycZzLxBWWT/5/rzL98sn3y/f0WUYit7hnkG3K/hTPPVnsn2f/q2gxFgaeOfQwDvH0WWIOIQC6hlkLl0Jc+lKji5DRMSudA1KREQMSQElIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJICigRETEkBZSIiBiSAkpERAxJASUiIoakgBIREUNSQImIiCEpoERExJAUUCIiYkgKKBERMSQFlIiIGJICSkREDEkBJSIihqSAEhERQ1JAiYiIISmgRETEkBRQIiJiSAooERExJAWUiIgYkgJKREQMSQElIiKGZLiA2r59O+3btyc0NJTY2FhHlyMiIg5iqIDKy8tj/Pjx/Pd//zfr1q1j7dq1nDhxwtFliYiIAxgqoI4cOcKLL75I5cqVcXV1pWPHjmzZssXRZYmIiAO4OLqA30tNTcXPz8/62NfXlyNHjjy0zfnz5+natesjbefFx6pOnrauXeMLaUvVC2k78kCP+H/0cfniWyjbkQfrmvTox9rT05N58+bdM91QAfU49uzZ4+gSRETEDgzVxefr60tKSor1cWpqKr6++kQkIlIUGSqg6tevz+nTpzl37hw5OTmsW7eOkJAQR5clIiIOYKguPhcXF8aNG8eAAQPIy8vjlVdeoXp1XT8QESmKTBaLxeLoIkRERP7IUF18IiIidymgRETEkBRQBnPt2jWWLVv2WG0XLlzIrVu3nnJFIiKOoYAymGvXrrF8+fLHart48WIF1DPst99+Y82aNY/cbuXKlYwfP/6B8/ft20dkZCR16tRh48aNT1KiPEX2Ot4LFiygQ4cOdO7cmb59+3L+/PknKdOhDMimddMAAAaRSURBVDWKT2Dq1KmcPXuW8PBw/va3v+Ht7c2GDRvIyckhNDSUqKgosrKyGDZsGCkpKeTn5zNo0CAuX75MWloaffv2pUyZMixZssTRuyKP6Pz586xdu5bOnTvfM89sNuPi8nj/XcuXL8/kyZOZP3/+k5YoT5G9jnft2rX57rvvcHNz4+uvv+azzz4jJibmSct1CAWUwYwcOZLjx4+TkJDA//7v/7Jp0ybi4uKwWCy888477Nu3jytXruDj42O92/v169cpVaoUCxcuZNGiRXh5eTl4L4qm+Ph45s2bh8lkombNmgwdOpQxY8aQkZGBl5cXkydPpkKFCkRHR1OyZEl++uknLl26xKhRowgLC2Pq1KmcPHmS8PBwIiMj8fDwIDExkaysLPLz85k5cyZjxozh3LlzuLm5MX78eGrVqvWndVWqVAkAJyd1mDxNRj3ewcHB1r8bNmzI6tWr7fk02JUCysB27tzJzp07iYiIACArK4vTp08TEBDAlClT+Oyzz2jdujUBAQEOrlSOHz/OnDlzWL58OV5eXmRmZhIdHU1kZCSRkZHExcXx8ccfM3v2bADS0tL4+uuv+fXXX3nnnXcICwtj5MiRzJ8/n6+++gq405Xzyy+/sHr1asqUKcOECROoU6cOs2fPZteuXbz33nskJCQ4creLrGfleMfFxdGiRYunvv+FRR+pDMxisTBw4EASEhJISEjg+++/p3v37lStWpWVK1dSo0YNYmJimDlzpqNLLfJ2795NWFiY9ey1TJkyHDp0iE6dOgEQHh7OgQMHrMu3bdsWJycnXnrpJS5fvvzA9TZt2pQyZcoAcODAAcLDwwFo0qQJmZmZ3Lhxw167JA/xLBzvhIQEfvrpJwYMGPDI+2cUCiiDcXd35+bNmwA0a9aM7777zvo4NTWV9PR0UlNTcXNzIzw8nLfeeotffvnlnrZibK6urjYt5+bmZudKpDAU9vH+4YcfmDt3LnPmzLF520akgDIYT09P/vrXv9KpUyd27txJp06deO211+jcuTNRUVHcvHmTf//733Tr1o3w8HBmzpzJO++8A0CPHj0YMGAAvXv3dvBeFD3BwcFs3LiRjIwMADIzM/H392fdunUArFmz5k+7Yv/sA0ZAQID1esKePXvw9PSkZMmST2kP5FEY+Xj/8ssvjBs3jjlz5uDt7W3rLhmSrkEZ0NSpUws87tu3b4HHVapUoXnz5ve06927t8LJQapXr85//dd/0bt3b5ycnKhTpw4ffPAB77//PvPmzbNeNH+YmjVr4uTkRJcuXejatSseHh4F5g8ZMoQxY8bQuXNn3Nzc+OSTT2yq7ciRIwwZMoRr166RlJTEjBkzrG+k8niMfLw//fRTsrKyGDp0KHBnFOfcuXMfb0cdTPfiExERQ1IXn4iIGJK6+ESeE3PmzLnnThFhYWHWa5TyfCkKx1tdfCIiYkjq4hMREUNSQImIiCEpoERExJAUUCIGYjabHV2CiGFoFJ9IIZo1axarV6/Gy8uL8uXLU7duXbZt20atWrU4cOAAnTp1onbt2kyZMoW8vDzq1avHRx99hKurKyEhIcTFxeHl5cXRo0f59NNPWbJkCTNmzODs2bOcPXuWjIwMBgwYQI8ePRy9qyJPTAElUkiOHDlCYmIiq1evJjc3l65du1K3bl0AcnNzWblyJdnZ2bRr146FCxdStWpVRo8ezddff02/fv0euu5//etffPvtt2RlZREZGUnLli3x9fUthL0SsR918YkUkoMHD9KmTRuKFy9OyZIlad26tXVehw4dADh16hSVKlWiatWqAERGRrJ///4/XXebNm0oUaIEXl5eBAUFcfToUfvshEghUkCJGIAtd7F2dnbm7tcWs7OzC8wzmUx2qUvEkRRQIoXkr3/9K0lJSWRnZ3Pz5k22bdt2zzJVq1bl/PnznDlzBrjzmz6BgYEAVKxYkZ9++gmAxMTEAu22bNlCdnY2GRkZ7N27l/r169t3Z0QKga5BiRSSl19+mZCQELp06YK3tzc1atSgVKlSBZYpXrw4kydPZujQodZBEj179gTu3N167NixfPnllwQFBRVoV7NmTfr06UNGRgaDBg3S9Sd5LuhWRyKF6ObNm7i7u3Pr1i1ef/11JkyYYB0o8bhmzJjBCy+8wFtvvfWUqhQxBp1BiRSicePGceLECbKzs4mMjHzicBJ5nukMSkREDEmDJERExJAUUCIiYkgKKBERMSQFlIiIGJICSkREDOn/A86KfiA9/BL8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Посмотрим на число оценок пользователей в зависимости от группы\n",
    "sns.barplot(data = users_merged, x = 'group', y = 'views_count')\n",
    "sns.despine()\n",
    "plt.title('Views count depending on group\\n')\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>W</th>\n",
       "      <th>pval</th>\n",
       "      <th>equal_var</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>levene</th>\n",
       "      <td>1.271269</td>\n",
       "      <td>0.281518</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               W      pval  equal_var\n",
       "levene  1.271269  0.281518       True"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Тест на гомоскедастичность\n",
    "pg.homoscedasticity(data = users_merged, dv = 'views_count', group = 'group')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Т.к. у нас 3 группы, сравним их распределения при помощи критерия Краскела-Уоллиса:\n",
    "\n",
    "- $H_0$: нет статистически значимых различий между распределениями количества просмотров в тестовой и контрольных группах (p-value > 0.05)\n",
    "- $H_1$: есть статистически значимые различия между распределениями количества просмотров в тестовой и контрольных группах"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KruskalResult(statistic=1.9364385074056973, pvalue=0.37975869022377584)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kruskal(users_merged.query('group == \"test\"').views_count, users_merged.query('group == \"control_1\"').views_count, users_merged.query('group == \"control_2\"').views_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Наше p-value > 0.05, мы не можем отвергнуть нулевую гипотезу, т.е. статистически значимых различий между распределениями количества просмотров пользователей разных групп нет, по данному признаку выборки репрезентативны."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Итак, в ходе проверки наших выборок по нескольких признакам, мы удостоверились в их репрезентативности, имеем качественное разделение пользователей по группам."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: Проводим А/А/В тесты. Тестируем гипотезы"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для оценки успешности эксперимента по изменению стоимости премиум-подписки будем полагаться в первую очередь на изменение **ARPPU_pr (средний доход на платящего премиум пользователя)**, т.к. эксперимент направлен на премиум пользователей, и доход от них - важный для бизнеса показатель. \n",
    "\n",
    "- $H_0$: ARPPU_pr в тестовой группе статистически значимо больше, чем в контрольных группах\n",
    "- $H_1$: ARPPU_pr в тестовой группе статистически значимо меньше, чем в контрольных группах, либо статистически значимо не изменился\n",
    "\n",
    "Но также обратим внимание на такие экономические метрики как:\n",
    "\n",
    "- CR_pr (конверсия из пользователя в покупателя премиум подписки, т.к. в наших итоговых выборках уже все пользователи = покупатели)\n",
    "- ARPPU (средний доход на платящего пользователя, без премиум подписки)\n",
    "(т.к. в наших выборках все пользователи - платящие, будем рассчитывать ARPPU и ARPPU_pr как доход на платящего пользователя и доход на платящего именно за премиум подписку пользователя соответственно)\n",
    "\n",
    "Эти метрики непосредственно связаны с метрикой ARPPU_pr, и при внесении изменений нельзя не обращать на них внимание.\n",
    "\n",
    "И посмотрим на такие метрики как:\n",
    "- соотношение пользователей, продливших/не продливших премиум подписку\n",
    "\n",
    "Пользователи могут менее охотно продлевать премиум подписку, если ее стоимость будет слишком высокая (или более охотно, если низкая), т.е. изменится число премиум пользователей и впоследствии может измениться доход от них.\n",
    "\n",
    "- количество дней посещения (visit days) \n",
    "\n",
    "Есть предположение, что есть взаимосвязь между стоимостью подписки и количеством посещений приложения (можно сравнить это с абонементом в спорт зал - чем выше его стоимость, тем больше мотивация его посещать). То есть заплатив весомую сумму, клиент захочет, чтобы эта трата была оправдана, и будет чаще посещать приложение. А большая возвращаемость клиента может способствовать увеличению дохода от него.\n",
    "\n",
    "Эти метрики хоть и опосредовано, но все же влияют на основую метрику ARPPU_pr, и могут внести вклад в успех эксперимента."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проведем **А/А тест для контрольных групп** перед А/В тестированием."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для начала проверим группы на гомоскедастичность:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>W</th>\n",
       "      <th>pval</th>\n",
       "      <th>equal_var</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>levene</th>\n",
       "      <td>0.737499</td>\n",
       "      <td>0.391133</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               W      pval  equal_var\n",
       "levene  0.737499  0.391133       True"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.homoscedasticity(data = users_merged.query('total_revenue > 0 and group != \"test\"'), dv = 'total_revenue', group = 'group')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Будем использовать t-тест, который сравнивает средние значения ARPPU двух выборок:\n",
    "\n",
    "- $H_0$: нет статистически значимых различий между средними значениями ARPPU в контрольных группах\n",
    "- $H_1$: есть статистически значимые различия между средними значениями ARPPU в контрольных группах"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAEYCAYAAABRMYxdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3df1RVdb7/8deBE4alFo4cxMvkjLcxl5aiMEgW6kFEfhzlh85gpSNjcfXeJC+OTU7qzDV/NK5JnZuNydCMNnVvt0gOKeYvnMSKJIx0smnKSUbJOCSgkJoInO8ffj1rCJQTHH60eT7Wai3P/uyzP+/Nx+R1Pvtz9jY5nU6nAAAADMSrqwsAAADwNAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIO8C12+vRpBQcHq6GhwePH3rZtm2bOnNnm9z/44IPKycnxYEUA4D4CDuBBs2bNUmhoqOrq6lpsj46O1okTJ1yvn376aQ0dOlRHjhxpU3+BgYEqKSmRt7d3m97vKU8//bR+9rOfNdmWlZWlxMTELqpIOnTokCIiIq67T1lZmYYOHar6+vp29/fYY49p/fr1re7ndDoVGRmp2NjYZm2zZs3SnXfeqeDgYIWFhenhhx9WRUWFq/3pp5/W8OHDFRwcrJCQEKWkpKikpETSlUA6bNgwBQcHa/To0Zo2bZr+/Oc/u9paCqtWq1Vvv/12W08Z6NYIOICHlJWVqbi4WCaTSfn5+c3aT548qcbGRn3ve9+TdOUXnd1u1y233CK73d7Z5aKLvPvuu6qqqtKpU6d09OjRZu3Lly9XSUmJ9u7dqwsXLujXv/51k/aYmBiVlJSosLBQo0eP1oIFC3T1hvSjRo1SSUmJiouLNX36dC1cuFDnzp3rlPMCuhsCDuAhdrtdI0eOVGJiYouB5Y033tD48eNdr4uLi/XFF1/o8ccf186dO6856yNJR48eVVJSkkaPHq27775ba9askdR8BmLWrFlav369UlJSFBwcrHnz5qm6ulqLFi3S6NGjlZycrLKyshbfe/X9r7zySos1rFy5UuPHj9fo0aOVlJSk4uJiSVJBQYE2b96s119/XcHBwZo6dWqzYzU2Nup3v/udJk6cqPDwcD366KOqra1tUkdOTo4mTJigsLAwbdq0qdVzv54LFy7ooYceUkVFhYKDgxUcHCyHw9FsvwceeECSFBoaquDgYNdsSHZ2tmJiYhQaGqq5c+fqs88+k3QllK5evVrh4eEaPXq0bDabPv74Y/3f//2ftm/frueee871c7+WnJwcWa1WjR8//rrBtm/fvoqMjNRHH33UYvsNN9ygxMREffHFF6qurm7S5uXlpeTkZH311Vc6efLk9X9YgEERcAAPyc3Nlc1mk81m05tvvqkzZ840aT9w4ECTgJOTk6OJEycqJiZGklyXE1qyatUqzZ49W++995727t3rek9Ldu7cqbVr16qgoEAnT55USkqKkpOTVVRUpCFDhuiZZ55p0/ndeeedstvtKioqUnx8vB555BFdunRJERER+rd/+zfXzMJrr73W7L3btm1TTk6Onn/+ee3bt08XLlzQihUrmuxz+PBh7dq1S1u3btUzzzyjv//9762eu81m0/bt25v117t3b/3+97+Xv7+/SkpKVFJSIovF0my/F154QdKVWZWSkhIFBwdr37592rx5szZu3KjCwkKNGTNGixYtkiS9+eabKi4u1u7du3X48GFt2LBBt9xyi3784x/LZrNp7ty5Kikp0bPPPtviz/DixYvavXu3pk6dKpvNpry8vGsG2+rqau3du1ff/e53W2yvq6vTtm3bNHDgQPn5+TVpq6+v1yuvvKLevXtr8ODBLb4fMDoCDuABxcXFOn36tGJiYjRixAgFBQVpx44drvaLFy/qgw8+UFhYmOv1rl27ZLPZdMMNNyg6Ovq6n+bNZrNOnjypqqoq3XTTTRo1atQ1901KStJ3v/td9enTRxEREQoKCtLdd98ts9msKVOm6MMPP2zTOU6bNk233nqrzGazfvrTn6qurq7JeqLr2b59u+bMmaOgoCDddNNNysjI0M6dO5vMHj388MO68cYbdccdd+iOO+5wzVxc79y3b98um83WpvO5lpdeeklpaWkaMmSIzGaz5s2bp7/+9a/67LPPZDabdf78eX366adyOp0aMmSI/P393T72nj175OPjo3HjxmnChAmqr6/XgQMHmuyzcuVKjRkzRmPHjlV1dbWWLVvWpH3Xrl0KCQnR+PHjdezYMW3cuNHVduTIEYWEhGjcuHHKy8vTM888oz59+rTvBwJ8SxFwAA+w2+0aN26c65N0fHx8k28QFRYWKjg4WD4+PpKkvXv3ymw2uxbB2mw2HTx4UFVVVS0ef9WqVSotLVVMTIySk5OvO9vzne98x/XnXr16NXl944036sKFC206x+eee04xMTEaM2aMQkJCVFtb2+zSyLVUVFRo0KBBrteDBg1SfX29KisrW6zb19fXVec3OffWXL1cFRwcrNOnT7e4z+nTp7V69WqFhIQoJCREP/zhD+V0OuVwOBQeHq77779fK1asUHh4uJYtW6Yvv/zS7f7tdrtiYmJkNpvVq1cvTZ48udk3zZYuXarDhw/rtddeU01NjcrLy5u0T5kyRcXFxSosLNTzzz+vESNGuNpGjhyp4uJiHTp0SC+//LLuvvtuSZK3t3eLC6kvX74ss9nsdv3Atwl/s4F2+uqrr/T666+rsbFR48aNk3Tl8kFNTY0++ugj3XHHHTpw4ECTb/TY7XZduHBBEydOlHRlbcfly5e1fft2/eQnP2nWx+DBg7Vu3To1NjZqz549Sk9P16FDh9pVd+/evV3133zzzZKkL774osV9i4uLlZWVpS1btuj222+Xl5eXQkNDXYtbTSbTdfvy9/d3rWORroQIs9ms/v37N/sF/nXXOver9V9LSzVdXWNz1T/XdNXAgQM1b94811qir5s9e7Zmz56tyspKLVy4UFlZWVq4cGGrP4Py8nK98847Onr0qPbs2SPpykxeXV2dqqqqml1mGjp0qObPn68VK1YoJyen1eNfT2BgoD7//HM5nU7XcS5evKiqqioFBga2+bhAd8YMDtBO+/btk7e3t/Ly8mS322W327Vz506FhIS4LjsdPHhQEyZMkCQ5HA4VFhbq2Wefde2fm5urhx56SLm5uS32kZubq6qqKnl5ealv376SriwkbQ8/Pz9ZLBbl5uaqoaFB2dnZOnXqVIv7nj9/Xt7e3vLz81N9fb02btzYZOaif//++uyzz9TY2Nji++Pj47V161adOnVK58+f1/r1610zGa1p67n3799fZ8+edS1mbomfn5+8vLyanHdKSooyMzP1ySefSJJqa2v1+uuvS7qy4PnIkSO6fPmyfH195ePj46qlf//+rgXc1zqPwYMHa9euXa5x3717tywWi/Ly8lp8T0JCgs6cOdPit/K+iZEjR8rHx0eZmZm6dOmSLly4oKeeekojRoxoMrMGGAkBB2innJwcJSUlKTAwUAMGDHD9d//992v79u36+OOP1bt3b9cn5dzcXA0bNkz33HNPk/1nzZqlv/3tb/r444+b9XHw4EHFxcUpODhYq1at0vr163XjjTe2u/YnnnhCzz33nMLCwnT8+HEFBwe3uN8999yje++9V9HR0bJarerVq5cGDhzoap8yZYokKSwsrMV73yQnJ2vq1Kl64IEHFBkZKR8fn2ZrS67leuceFxfX4qJmSRoyZIji4uI0adIkhYSEtPgtKl9fX82bN08zZ85USEiI3n//fUVFRenBBx9URkaGRo8erfj4eBUUFEi6EvSWLl2qH/7wh5o4caJuueUWzZ07V5I0ffp0HT9+XCEhIfr3f//3Zn3l5OTovvvuazLmAwYMUEpKyjVviOjj46PZs2frd7/7nVs/q2u5Gm6KiooUERGhSZMmqaKiQhs2bGjXzBDQnZmcV+eYAXSI3//+96qurtajjz7a1aUAQI/BGhyggw0aNEhWq7WrywCAHoUZHAAAYDiswQEAAIbTLQLO1UV6AAAAntAtAo67NwsDAABwR7cIOAAAAJ5EwAEAAIZDwAEAAIZDwAEAAIbTasBZsmSJwsPDFR8f32T7n/70J02ZMkVxcXFau3ata/vmzZsVFRWl6OhoHTx40PMVAwAAtKLVOxknJSXpgQce0M9//nPXtnfeeUf5+fl67bXX5OPjo8rKSknS8ePHlZeXp7y8PDkcDqWmpmr37t3y9vbuuDMAAAD4mlZncEJDQ9WvX78m2/73f/9XaWlp8vHxkXTlKbqSlJ+fr7i4OPn4+CgoKEi33Xabjh492gFlAwAAXFub1uCUlpaquLhYM2bM0AMPPOAKMQ6HQwEBAa79LBZLi0/wBQAA6EhtethmQ0ODzp07p5dffll/+ctftHDhQuXn53u6NgAAgDZp0wyOxWJRVFSUTCaT7rrrLnl5eam6uloWi0Xl5eWu/RwOhywWi8eKBVpSV9/Yo/oFALSuTTM4kyZN0qFDhzR27FidOHFCly9f1q233iqr1apFixYpNTVVDodDpaWluuuuuzxdM9CEj9lLKZmFnd7vS2nhnd4nAMA9rQacjIwMFRUVqbq6WhEREVqwYIGSk5P1i1/8QvHx8brhhhv05JNPymQy6fbbb1dMTIxiY2Pl7e2t5cuX8w0qAADQ6UxOp9PZ1UUkJSVp27ZtXV0GvsWYwQEA/DPuZAwAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyHgAMAAAyn1YCzZMkShYeHKz4+vlnbH/7wBw0dOlRVVVWSJKfTqZUrVyoqKko2m03Hjh3zfMUAAACtaDXgJCUlKSsrq9n2zz//XG+99ZYCAwNd2woKClRaWqo9e/boiSee0K9+9SuPFgsAAOCOVgNOaGio+vXr12z7mjVrtHjxYplMJte2/Px8JSQkyGQyadSoUaqpqVFFRYVnKwYAAGhFm9bg7Nu3T/7+/rrjjjuabHc4HAoICHC9DggIkMPhaF+FAAAA35D5m77h4sWL2rx5s/7whz90RD0AAADt9o0DzsmTJ1VWVqZp06ZJksrLy5WUlKRXXnlFFotF5eXlrn3Ly8tlsVg8Vy0AAIAbvnHAGTp0qAoLC12vrVarsrOz5efnJ6vVqhdeeEFxcXE6cuSI+vTpI39/f48WDAAA0JpWA05GRoaKiopUXV2tiIgILViwQDNmzGhx3/Hjx+vAgQOKioqSr6+vVq9e7fGCAQAAWtNqwFm3bt112/fv3+/6s8lk0i9/+cv2VwUAANAO3MkYAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYTqsBZ8mSJQoPD1d8fLxr269//WtNmTJFNptN//Ef/6GamhpX2+bNmxUVFaXo6GgdPHiwY6oGAAC4jlYDTlJSkrKysppsGzdunHbs2KHt27dr8ODB2rx5syTp+PHjysvLU15enrKysvRf//Vfamho6JjKAQAArqHVgBMaGqp+/fo12XbPPffIbDZLkkaNGqXy8nJJUn5+vuLi4uTj46OgoCDddtttOnr0aAeUDXS9uvrGHtUvAHybmNt7gFdffVUxMTGSJIfDoZEjR7raLBaLHA5He7sAuiUfs5dSMgs7vd+X0sI7vU8A+LZp1yLjTZs2ydvbW1OnTvVUPQAAAO3W5hmcbdu26Y033tCWLVtkMpkkXZmxuXq5Sroyo2OxWNpfJQAAwDfQphmcgoICZWVladOmTfL19XVtt1qtysvLU11dnU6dOqXS0lLdddddHisWAADAHa3O4GRkZKioqEjV1dWKiIjQggULlJmZqbq6OqWmpkqSRo4cqRUrVuj2229XTEyMYmNj5e3treXLl8vb27vDTwIAAOCftRpw1q1b12zbjBkzrrn//PnzNX/+/PZVBQAA0A7cyRgAABgOAQcAABgOAQcAABgOAQcAABgOAQcAABgOAQcAABgOAQcAABgOAQcAABgOAQcAABgOAQcAABgOAQcAABgOAQcAABgOAQcAABgOAQcAABgOAQcAABgOAQceU1ff2NUlAAAgSTJ3dQEwDh+zl1IyCzu935fSwju9TwBA99bqDM6SJUsUHh6u+Ph417azZ88qNTVVkydPVmpqqs6dOydJcjqdWrlypaKiomSz2XTs2LGOqxwAAOAaWg04SUlJysrKarItMzNT4eHh2rNnj8LDw5WZmSlJKigoUGlpqfbs2aMnnnhCv/rVrzqkaAAAgOtpNeCEhoaqX79+Tbbl5+crISFBkpSQkKB9+/Y12W4ymTRq1CjV1NSooqKiA8oGAAC4tjYtMq6srJS/v78kacCAAaqsrJQkORwOBQQEuPYLCAiQw+HwQJkAAADua/e3qEwmk0wmkydqAQAA8Ig2BZz+/fu7Lj1VVFTIz89PkmSxWFReXu7ar7y8XBaLxQNlAgAAuK9NAcdqtcput0uS7Ha7IiMjm2x3Op16//331adPH9elLAAAgM7S6n1wMjIyVFRUpOrqakVERGjBggVKS0vTwoULlZ2drcDAQG3YsEGSNH78eB04cEBRUVHy9fXV6tWrO/wEAAAAvq7VgLNu3boWt2/durXZNpPJpF/+8pftrwoAAKAdeFQDAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHHN73rxlyxa98sorMplM+sEPfqA1a9aooqJCGRkZOnv2rIYPH661a9fKx8fHU/UCAAC0qs0zOA6HQ88//7xeffVV7dixQw0NDcrLy9NvfvMbzZkzR3v37lXfvn2VnZ3tyXoBAABa1a5LVA0NDfrqq69UX1+vr776SgMGDNA777yj6OhoSVJiYqLy8/M9UigAAIC72nyJymKx6Kc//akmTpyoXr16ady4cRo+fLj69u0rs/nKYQMCAuRwODxWLAAAgDvaPINz7tw55efnKz8/XwcPHtTFixd18OBBT9YGAADQJm2ewXn77bf1L//yL/Lz85MkTZ48We+9955qampUX18vs9ms8vJyWSwWjxULAADgjjbP4AQGBurIkSO6ePGinE6nCgsL9a//+q8KCwvT7t27JUk5OTmyWq0eKxYAAMAdbZ7BGTlypKKjo5WYmCiz2axhw4bpxz/+sSZMmKD//M//1IYNGzRs2DDNmDHDk/UCAAC0ql33wUlPT1d6enqTbUFBQXw1HAAAdCnuZAwAAAyHgAN8y9TVN/aofgGgLdp1iQpA5/Mxeykls7DT+30pLbzT+wSAtmIGBwAAGA4BBwAAGA4BBwAAGA4BBwAAGA4BBwAAGA4BBwAAGA4BBwAAGA4BBwAAGA4BBwAAGA4BBwAAGA4BBwAAGA4BBwAAGA4BBwAAGA4BBwAAGA4BBwAAGE67Ak5NTY3S09M1ZcoUxcTEqKSkRGfPnlVqaqomT56s1NRUnTt3zlO1AgAAuKVdAWfVqlW69957tWvXLuXm5mrIkCHKzMxUeHi49uzZo/DwcGVmZnqqVgAAALe0OeDU1tbq3Xff1fTp0yVJPj4+6tu3r/Lz85WQkCBJSkhI0L59+zxTKQAAgJvMbX1jWVmZ/Pz8tGTJEn300UcaPny4Hn/8cVVWVsrf31+SNGDAAFVWVnqsWAAAAHe0eQanvr5eH374oWbOnCm73S5fX99ml6NMJpNMJlO7iwQAAPgm2hxwAgICFBAQoJEjR0qSpkyZog8//FD9+/dXRUWFJKmiokJ+fn6eqRQAAMBNbQ44AwYMUEBAgD799FNJUmFhoYYMGSKr1Sq73S5JstvtioyM9EylAAAAbmrzGhxJWrZsmX72s5/p8uXLCgoK0po1a9TY2KiFCxcqOztbgYGB2rBhg6dqhZvq6hvlY+YWRwCAnqtdAWfYsGHatm1bs+1bt25tz2HRTj5mL6VkFnZ6vy+lhXd6nwAAtISP+QAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOAAAwHAIOADcUlff2KP6BfDtZm7vARoaGpScnCyLxaLNmzfr1KlTysjI0NmzZzV8+HCtXbtWPj4+nqgVQBfyMXspJbOw0/t9KS280/sE8O3X7hmc559/XkOGDHG9/s1vfqM5c+Zo79696tu3r7Kzs9vbBQDAoLpyho7ZQWNr1wxOeXm53njjDc2bN09btmyR0+nUO++8o6eeekqSlJiYqI0bN+q+++7zSLEAAGPpqplBidlBo2vXDM7q1au1ePFieXldOUx1dbX69u0rs/lKbgoICJDD4Wh/lQAAAN9AmwPOn//8Z/n5+WnEiBGerAcAAKDd2nyJ6r333tP+/ftVUFCgS5cu6csvv9SqVatUU1Oj+vp6mc1mlZeXy2KxeLJeAACAVrV5BmfRokUqKCjQ/v37tW7dOo0dO1ZPPfWUwsLCtHv3bklSTk6OrFarx4oFAABwh8fvg7N48WL98Y9/VFRUlM6ePasZM2Z4ugsAAIDravd9cCQpLCxMYWFhkqSgoCC+Gg4AALoUdzIGAACGQ8ABAACGQ8ABAACGQ8ABAACGQ8ABAACGQ8ABAKATddVDPnvaw0U98jVxAADgnq56wGhPe7goMzgA0AI+ZQPfbszgAEAL+JQNfLsxgwMAAAyHgAMAAAyHgAOgW2NNCoC2YA0OgG6NtTAA2oIZHAAAYDgEHAAAlwJ7gJ526wMuUQEAuBTYA/S0MWYGBwAAGE6bA87nn3+uWbNmKTY2VnFxcdq6dask6ezZs0pNTdXkyZOVmpqqc+fOeaxYADA6LhUBntHmS1Te3t567LHHNHz4cH355ZdKTk7WuHHjtG3bNoWHhystLU2ZmZnKzMzU4sWLPVkzABhWT7uMAHSUNs/g+Pv7a/jw4ZKkm2++Wd///vflcDiUn5+vhIQESVJCQoL27dvnmUoBAPAgZsuMzSOLjMvKyvTXv/5VI0eOVGVlpfz9/SVJAwYMUGVlpSe6AADAo5gtM7Z2LzI+f/680tPT9Ytf/EI333xzkzaTySSTydTeLgAAAL6RdgWcy5cvKz09XTabTZMnT5Yk9e/fXxUVFZKkiooK+fn5tb9KAACAb6DNAcfpdOrxxx/X97//faWmprq2W61W2e12SZLdbldkZGT7qwQAAPgG2rwG5/Dhw8rNzdUPfvADTZs2TZKUkZGhtLQ0LVy4UNnZ2QoMDNSGDRs8ViwAAIA72hxwQkJC9Le//a3Ftqv3xAEAAOgK3MkYAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgGnA9XVN3Z1CQAA9EhtflQDWudj9lJKZmGn9/tSWnin9wkAQHfCDA4AADCcHhFwuFQEAEDP0iMuUXGpCACAnqVHzOAAAICehYADAAAMh4ADAAAMh4ADAAAMp8MCTkFBgaKjoxUVFaXMzMyO6gYAAKCZDgk4DQ0NWrFihbKyspSXl6cdO3bo+PHjHdEVAABAMx0ScI4eParbbrtNQUFB8vHxUVxcnPLz8zuiKwAAgGZMTqfT6emD7tq1SwcPHtSqVaskSXa7XUePHtXy5ctb3D8sLEyDBg3ydBkAAMDgbr31Vj333HPNtneLG/0dOnSoq0sAAAAG0iGXqCwWi8rLy12vHQ6HLBZLR3QFAADQTIcEnDvvvFOlpaU6deqU6urqlJeXJ6vV2hFdAQAANNMhl6jMZrOWL1+uBx98UA0NDUpOTtbtt9/eEV0BAAA00yGLjAEAALoSdzIGAACGQ8ABAACG0+0CTmuPeKirq9PChQsVFRWlGTNmqKysrAuq7BlaG4s//vGPio2Nlc1m009+8hN99tlnXVBlz+Duo092796toUOH6i9/+UsnVtezuDMWO3fuVGxsrOLi4rRo0aJOrrDnaG0sTp8+rVmzZikhIUE2m00HDhzogip7hiVLlig8PFzx8fEttjudTq1cuVJRUVGy2Ww6duxYxxfl7Ebq6+udkZGRzpMnTzovXbrktNlszk8++aTJPi+88IJz2bJlTqfT6dyxY4fzkUce6YpSDc+dsSgsLHReuHDB6XQ6nS+++CJj0UHcGQun0+msra113nfffc4ZM2Y4jx492gWVGp87Y3HixAnntGnTnGfPnnU6nU7nmTNnuqJUw3NnLJYuXep88cUXnU6n0/nJJ584J06c2BWl9ghFRUXODz74wBkXF9di+xtvvOGcO3eus7Gx0VlSUuKcPn16h9fUrWZw3HnEw/79+5WYmChJio6OVmFhoZysk/Y4d8Zi7Nix8vX1lSSNGjWqyb2P4DnuPvrkt7/9rR566CH16tWrC6rsGdwZi5dffln333+/+vXrJ0nq379/V5RqeO6Mhclk0pdffilJqq2tlb+/f1eU2iOEhoa6/s63JD8/XwkJCTKZTBo1apRqampUUVHRoTV1q4DjcDgUEBDgem2xWORwOJrtM3DgQElXvo7ep08fVVdXd2qdPYE7Y/HPsrOzFRER0Rml9TjujMWxY8dUXl6uCRMmdHJ1PYs7Y1FaWqoTJ04oJSVFP/rRj1RQUNDZZfYI7ozFww8/rO3btysiIkJpaWlaunRpZ5eJ/+/r4xUQEHDd3yme0K0CDr6dcnNz9cEHH+jBBx/s6lJ6pMbGRj355JP6+c9/3tWlQFJDQ4P+8Y9/6E9/+pOeeuopLVu2TDU1NV1dVo+Ul5enxMREFRQUKDMzU48++qgaGxu7uix0km4VcNx5xIPFYtHnn38uSaqvr1dtba1uvfXWTq2zJ3D3cRtvv/22nn32WW3atMZukygAAARjSURBVEk+Pj6dWWKP0dpYnD9/Xh9//LFmz54tq9Wq999/X/Pnz2ehcQdw998oq9WqG264QUFBQRo8eLBKS0s7uVLjc2cssrOzFRMTI0kKDg7WpUuXmPHvIl8fr/Ly8g5/hFO3CjjuPOLBarUqJydH0pVvjIwdO1Ymk6kryjU0d8biww8/1PLly7Vp0ybWGXSg1saiT58+OnTokPbv36/9+/dr1KhR2rRpk+68884urNqY3Pn/YtKkSSoqKpIkVVVVqbS0VEFBQV1RrqG5MxYDBw5UYWGhJOnvf/+7Ll26JD8/v64ot8ezWq2y2+1yOp16//331adPnw5fE9UtniZ+1bUe8fDb3/5WI0aMUGRkpKZPn67FixcrKipK/fr10/r167u6bENyZyzWrl2rCxcu6JFHHpF05R+TZ599tosrNx53xgKdw52xuPfee/XWW28pNjZW3t7eevTRR5ll7gDujMVjjz2mpUuXasuWLTKZTHryySf5QNxBMjIyVFRUpOrqakVERGjBggWqr6+XJM2cOVPjx4/XgQMHFBUVJV9fX61evbrDa+JRDQAAwHC61SUqAAAATyDgAAAAwyHgAAAAwyHgAAAAwyHgAAAAwyHgAPhW2bZtm1asWNHVZQDo5gg4AADAcAg4ADpMWVmZpkyZokWLFikmJkbp6em6ePGiq72xsVFWq7XJs5omT56sM2fOaP/+/ZoxY4YSEhI0Z84cnTlzptnxH3vsMe3atcv1Ojg42PXnrKwsJScny2az6b//+7876AwBdFcEHAAd6sSJE7rvvvv0+uuv66abbtL//M//uNq8vLxktVq1d+9eSdKRI0cUGBio73znOxozZoxefvll2e12xcXFKSsry+0+33zzTf3jH/9Qdna2cnNzdezYMb377rsePzcA3RcBB0CHGjhwoMaMGSNJmjp1qg4fPtykPTY2Vjt37pR05enPsbGxkq48jG/u3Lmy2WzKysrSJ5984nafb731lt566y0lJCQoMTFRn376KQ+8BHqYbvUsKgDG8/Vn/9TW1mratGmSpPT0dFmtVp08eVJVVVXat2+f5s+fL0lauXKl5syZo8jISB06dEgbN25sdmxvb281NjZKunK56/Lly5Ikp9OptLQ0paSkdOSpAejGmMEB0KFOnz6tkpISSdKOHTs0ceJE5ebmKjc3V5GRkTKZTJo0aZLWrFmjIUOGuB5MWVtbK4vFIkmy2+0tHnvQoEE6duyYJGn//v2ugHPPPffo1Vdf1fnz5yVJDodDlZWVHXqeALoXAg6ADvW9731PL774omJiYlRTU6OZM2c22yc2Nlavvfaa6/KUJD388MN65JFHlJSUpFtuuaXFY//oRz/Su+++q6lTp6qkpES9e/eWdCXgxMfHKyUlRTabTenp6a6wA6Bn4GniADpMWVmZ5s2bpx07dnR1KQB6GGZwAACA4TCDAwAADIcZHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDgEHAAAYDj/Dx7LTy7xegOwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = []\n",
    "\n",
    "for i in range(800):\n",
    "    c_1 = users_control_1.query('total_revenue > 0').total_revenue.sample(100, replace = False).values\n",
    "    c_2 = users_control_2.query('total_revenue > 0').total_revenue.sample(100, replace = False).values\n",
    "    df.append(ttest_ind(c_1, c_2, equal_var = False)[1])\n",
    "\n",
    "fig, axes = plt.subplots(figsize=(8, 4))\n",
    "sns.histplot(df)\n",
    "axes.set(title = 'A/A simulations: t-test ARPPU', xlabel = 'p-value', ylabel = '')\n",
    "sns.despine()\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Проверим, что количество ложноположительных случаев не превышает уровень значимости (0.05)\n",
    "((np.array(df) < 0.05) / 800).sum() <= 0.05"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Нулевую гипотезу мы не можем отвергнуть, т.е. статистически значимых различий по ARPPU в контрольных группах не наблюдается.\n",
    "\n",
    "Таким образом, в А/В тестах мы может использовать только одну из контольных групп, предпочтем группу \"control_2\", т.к. эта выборка претерпевала меньше изменений в предобработке, чем группа \"control_1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>was_premium</th>\n",
       "      <th>is_premium</th>\n",
       "      <th>group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>891152790</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>test</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>891589239</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>test</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>891343671</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>test</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>892430988</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>test</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>891919368</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>test</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           uid  was_premium  is_premium group\n",
       "20   891152790          1.0         1.0  test\n",
       "44   891589239          1.0         1.0  test\n",
       "100  891343671          1.0         1.0  test\n",
       "105  892430988          1.0         1.0  test\n",
       "134  891919368          1.0         1.0  test"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Соберем информацию по пользователям, продливших / не продливших подписку\n",
    "users_premium = users_merged[['uid', 'was_premium', 'is_premium', 'group']].query('group != \"control_1\"')\n",
    "users_premium.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_premium.query('was_premium == 0.0 and is_premium == 1.0').shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Пользователей, у которых не было подписки, и она появилась, нет (что логично), поэтому у нас будет разделение на 3 группы:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>was_premium</th>\n",
       "      <th>is_premium</th>\n",
       "      <th>group</th>\n",
       "      <th>premium</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>891152790</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>test</td>\n",
       "      <td>not canceled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>891589239</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>test</td>\n",
       "      <td>not canceled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>891343671</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>test</td>\n",
       "      <td>not canceled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>892430988</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>test</td>\n",
       "      <td>not canceled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>891919368</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>test</td>\n",
       "      <td>not canceled</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           uid  was_premium  is_premium group       premium\n",
       "20   891152790          1.0         1.0  test  not canceled\n",
       "44   891589239          1.0         1.0  test  not canceled\n",
       "100  891343671          1.0         1.0  test  not canceled\n",
       "105  892430988          1.0         1.0  test  not canceled\n",
       "134  891919368          1.0         1.0  test  not canceled"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def f(row):\n",
    "    if row['was_premium'] == 0.0 and row['is_premium'] == 0.0:\n",
    "        x = 'no'\n",
    "    elif row['was_premium'] == 1.0 and row['is_premium'] == 0.0:\n",
    "        x = 'canceled'\n",
    "    else:\n",
    "        x = 'not canceled'\n",
    "    return x\n",
    "\n",
    "users_premium['premium'] = users_premium.apply(f, axis = 1)\n",
    "users_premium.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "group      premium     \n",
       "control_2  canceled         10\n",
       "           not canceled    136\n",
       "test       canceled          7\n",
       "           not canceled    120\n",
       "dtype: int64"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_premium.groupby(['group', 'premium']).size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deWAN9/7/8efJHmuaILG22tpKEaKk1BJCLGkWVdxWUb0uSuy9aC9XtFTVvjaX2tpqSSOpPRWUqq211dIFtQRJKos1sv/+8HO+UkkFOcmQ1+OvnDmf+XzeMwmvM5+ZM2PKysrKQkRExGCsCrsAERGRnCigRETEkBRQIiJiSAooERExJAWUiIgYkgJKREQMSQElubpw4QLu7u5kZGQUdim5mj17NiNGjLD4OO7u7pw7d+6B1v3xxx9p165dPleUMy8vL3744YcCGSuv9uzZQ/Pmzc2vO3bsyJ49ewqxInlU2BR2AXJvXl5eXLp0CWtraxwdHWnevDn/+c9/KF68uEXHrVChAgcOHLDoGI+K+9kPNWrUIDIykieffBIADw8PNm3aZKnSHjnr1q0r7BLkEaEjqEfEggULOHDgAKtXr+bIkSPMnz//rjbp6emFUNnjTftU7kV/I5ajgHrEuLq68tJLL/H7778Dtz6tf/7557Rt25a2bdsCsHXrVvz8/PDw8KBbt2788ssv5vW9vLxYuHAhvr6+1K9fnzFjxnDp0iXeeust3N3d6dWrF5cvXwYgOjqaGjVqmP8B/nX66M7ptdttv/76a1q0aEGjRo1YsWIFhw8fxtfXFw8PD4KDg3PdrsOHDxMYGEiDBg148cUXmTRpEnD39FBOdaSmpjJkyBDc3d0JCAjItr0hISG89NJLuLu7065dO3bt2gVARkYGCxYsoE2bNri7uxMYGMjFixdz3ac1atTgzJkzAIwaNYqxY8fSu3dv3N3def311zl//jwAr732GgB+fn64u7uzfv36u7bh5MmT9OjRAw8PDzp27EhUVJT5vVGjRjF+/Hj69u2Lu7s7Xbp04ezZs7nut/DwcFq1akXjxo3v+tCSmZlJSEgIbdq0oXHjxgwePJikpKRsv6+vvvqKZs2a0axZMxYtWnRf665evZqWLVveNfbNmzcZNWoUjRo1okOHDvz888+5/v5mz57N4MGDeeedd3B3d6djx47Z2h89ehR/f3/c3d0JCgpiyJAhTJ8+Pcd9kZmZybx582jVqhWenp688847XL16NU81/1ViYiL9+vWjQYMGdO7cmenTp9O9e3fz+zn9jaxcuRJvb29eeOEF+vXrR2xsbLax7wyyHj16sGrVKgDCwsLo1q0bwcHBNGzYEB8fH/PfaVGngHrEXLx4ke3bt1OrVi3zss2bN7Ny5UrWr1/PsWPHGDNmDMHBwezZs4euXbsyYMAAUlNTze0jIyNZvHgxmzZtYuvWrfzzn/9k2LBh7N69m8zMTJYvX/7A9R06dIjIyEimT5/OxIkTWbBgAUuWLGHdunVs2LCBvXv35rjeBx98wBtvvMH+/fv59ttvad++fZ7HjIqKwsfHh71799KpUycGDBhAWloap06d4vPPPyc0NJQDBw6waNEiKlasCMDixYtZt24dISEh7N+/n4kTJ+Lg4GDu8859mpM1a9YwYMAA9uzZQ82aNc1B/fnnnwMQERHBgQMH6NChQ7b10tLS6NevH02bNuWHH37gvffeY8SIEZw6dcrcZv369QwcOJB9+/ZRpUqVXP9DPnHiBOPHj+ejjz5ix44dJCUlERMTY35/+fLlbN68mc8++4wdO3ZQunTpuz4k7Nmzh8jISBYtWsT//vc/c3DkZd2ffvqJjRs3snTpUubOncvJkycBmDNnDmfPnuXbb79l0aJFhIeH5/Kbu2XLli107NiRH3/8ES8vLyZMmADc+uAxcOBAAgICzL/bzZs359pPWFgYq1evZtmyZWzevJkbN27kuea/Cg4OxtHRkZ07dzJ58uQct+HOv5Fdu3YxdepUZsyYwffff0/FihUZNmzY3273nQ4fPkyVKlXYvXs3QUFBDBw40PyBoChTQD0i3n77bTw8PPjHP/5Bo0aN6Nevn/m9vn374uTkhIODA1999RVdu3alXr16WFtbExAQgK2tLQcPHjS3f/311ylTpgyurq54eHhQt25dnnvuOezt7fH29ubYsWMPVae9vT3NmjWjWLFidOrUCRcXF/NYufVtY2PD2bNnSUhIoHjx4tSvXz/PY9auXRsfHx9sbW3p3bs3qampHDp0CGtra1JTUzl58iRpaWlUqlSJKlWqALBq1SoGDx7M008/jclkombNmjzxxBPmPu/cpzlp2bIljRo1ws7OjqFDh3Lw4EHzEdjfOXToEDdu3KBv377Y2dnh6elJq1atsp2XadOmDXXr1sXGxoaXX36Z48eP59jXxo0bs9UxePBgrKz+75/0l19+ydChQ3Fzc8POzo6BAweyadOmbJ/k3377bYoVK0aNGjUIDAxk7dq1eV534MCBODg4ULNmTWrWrGk+ct2wYQP9+vXDycmJ8uXL06NHj7/dJw0bNqRFixZYW1vj5+dn7ufQoUOkp6fzxhtvYGtrS9u2bXn++edz7WfNmjX06tWLypUrU7x4cYYNG8b69evzVPOdMjIyiIyMZNCgQTg6OvLss8/i7+9/V7s7/0bWrFlD586dqV27NnZ2dgwbNoyDBw8SHR39t9t+m7OzMz179sTW1pYOHTpQtWpVtm3blqd1H2e6SOIRMXfuXF588cUc3ytfvrz55wsXLhAeHs5nn31mXpaWlkZcXJz5dZkyZcw/29vbZ3vt4ODAjRs3HrhOFxeXbH3/9XVufX/wwQfMmjWL9u3bU6lSJQYOHEirVq3yNKabm5v5ZysrK1xdXYmLi8PDw4MxY8Ywe/ZsTpw4QbNmzRg1ahSurq7ExMSYwyond+7Te41ZvHhxSpcuTVxc3D3Xi4uLw83NLVuQVKhQwTwdBOT593G7r9uKFSuGk5OT+fWFCxd4++23s41lZWVFfHy8+fWd9VasWJHffvstz+veWaejo6O5zr/uhwoVKuRYf27bm5KSQnp6OnFxcbi6umIymXKs96/i4uLMR8i3tyc9PT1PNd8pISGB9PT0bGPlNO6dy+Li4qhdu7b5dfHixXFyciI2NhZXV9dca77tr9tZoUKFbP9miyoF1GPgr/+A+/XrR//+/fN9HEdHR5KTk82v//zzz3zr+6mnnmLatGlkZmYSGRlJUFAQe/bswdHRkZs3b5rbZWRkkJCQkG3dO6e1MjMziY2NpVy5cgD4+vri6+vLtWvXGDt2LB9//DFTpkzBzc2Ns2fPUr169RzruXOf5uTOMa9fv87ly5fNY/6dcuXKERMTQ2Zmpvk//4sXL/LUU0/dc92c+rpziio5OTnbtJCbmxsTJ06kYcOGd617+5P9xYsXeeaZZ4BboXR7G/Kybm7Kli3LxYsXqVatmnmMB1G2bFliY2PJysoy/z4uXrxI5cqVc2xfrlw587nA29tjY2ODi4tLtt/XvTg7O2NjY0NMTAxVq1bNdRvu/Bv569g3btwgKSkJV1dXihUrBtw6N1eiRAng7n87OW2nl5dXnmt+XGmK7zHTpUsXvvzySw4dOkRWVhY3btxg27ZtXLt27aH7rlmzJuvXryctLY2ff/45Xy+djoiIICEhASsrK0qVKgXc+sRetWpVUlJS2LZtG2lpacyfPz/b+TS4dSI9MjKS9PR0li5dip2dHfXq1ePUqVPs2rWL1NRU7OzssLe3N4dCly5dmDlzJqdPnyYrK4tffvmFxMTEPNf73Xff8eOPP5KamsrMmTOpV6+e+RN1mTJlcv3OVN26dXFwcGDhwoWkpaWxZ88etmzZcte5qrxo164d27ZtM9cxa9YsMjMzze93796dGTNmmP/jTEhIuOsczrx580hOTub3338nLCzMXEde1s1N+/btCQkJ4fLly8TExDzwOc369etjbW3NZ599Rnp6Ops3b77rgos7derUiaVLl3Lu3DmuX7/O9OnTad++PTY29/c53NraGm9vb+bMmUNycjInT54kIiLib9fp1KkTYWFhHD9+nNTUVKZNm0bdunWpVKkSzs7OuLq6EhERQUZGBqGhoXf9fSQkJLBs2TLS0tLYsGEDJ0+epEWLFvdV9+NIR1CPmeeff54JEyYQHBzMmTNncHBwoEGDBnh4eDx030OGDGHYsGG88MILNGrUCF9f33w7kbtjxw4+/PBDbt68SYUKFZg+fToODg44ODgwbtw43nvvPTIyMnjrrbeyTWsBtG7dmvXr1/Pvf/+bJ598ktmzZ2Nra0tqaipTp07l5MmT2Nra4u7ubj5pfvtc1ZtvvkliYiJPP/00c+fOzXO9nTp1Yu7cuRw8eJDnnnuOKVOmmN8bOHAgo0aN4ubNmwQHB2eb5rSzs2PBggWMHz+eTz75BFdXVz766CPzUcz9qFatGmPHjmXEiBEkJyfTq1evbPvmjTfeICsrizfffJO4uDhcXFzo0KEDbdq0Mbd54YUX8Pb2Nrdr1qxZntfNzcCBAxk3bhytW7emXLlyBAYGsmzZsvvePjs7O2bPns17773HtGnTeOmll2jZsiV2dnY5tu/cuTOxsbG8/vrrpKSk0KxZM/7zn//c97gAY8eOZdSoUTRt2pSqVavSsWNHjhw5kmv7F198kcGDBzNo0CCuXLmCu7t7totbJkyYwPjx45k+fTqvvPIK7u7u2davW7cuZ86coUmTJpQpU4ZZs2ZlOydaVJn0wEKR+3P7PNbQoUMLu5QHFh0dTevWrTl69Oh9H2EUpi5dutCtWzc6d+5coONOmTKFS5cuMXny5HzvOywsjFWrVrFixYp87/tRpyk+ETGsvXv38ueff5Kens7q1av59ddfeemllyw+7smTJ/nll1/Iysri8OHDhIaG4u3tbfFxJbtH56OTiBQ5f/zxB0OGDCE5OZlKlSoxa9asPF2M8rCuX7/O8OHDzdObb775Jq1bt7b4uJKdpvhERMSQNMUnIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJICigRETEkBZSIiBiSAkpERAxJASUiIoakgBIREUNSQImIiCEpoERExJAUUCIiYkgKKBERMSQFlIiIGJICSkREDOmRDqg+ffoUdgkiImIhFguo0aNH4+npSadOne5679NPP6VGjRokJCQAkJWVxfvvv4+3tze+vr4cPXo0T2MkJibma80iImIcFguowMBAFi5ceNfyixcvsnPnTipUqGBetn37dk6fPk1kZCQTJkzgv//9r6XKEhGRR4TFAqpRo0aULl36ruWTJk1i5MiRmEwm87KoqCj8/f0xmUzUr1+fK1euEBcXZ6nSRETkEVCg56A2b95MuXLlqFmzZrblsbGxuLm5mV+7ubkRGxtbkKWJiIjB2BTUQMnJyXzyySd8+umnBTWkiBRxaWlpREdHc/PmzcIuRQAHBwcqVaqEra1tntoXWECdPXuW6Oho/Pz8AIiJiSEwMJBVq1bh6upKTEyMuW1MTAyurq4FVZqIPKaio6MpWbIkTz31VLbTClLwsrKyiI+PJzo6mqpVq+ZpnQKb4qtRowa7du1iy5YtbNmyBTc3N8LCwihbtixeXl6Eh4eTlZXFwYMHKVmyJOXKlSuo0kTkMXXz5k1cXFwUTgZgMplwcXG5r6NZix1BDRs2jL1795KYmEjz5s0ZNGgQXbp0ybFtixYt+O677/D29sbR0ZGJEydaqiwRKWIUTsZxv78LiwXUtGnT/vb9LVu2mH82mUyMGzfOUqWIiMgj6JG+k4SI5F1KWkZhl/BIyMzMKuwS7ik2NpagoKDCLsPiCuwiCREpXPa21jQcuaywyyhQUwJqkXXu0n2t81zlMhaq5v9kZGRgbW39wOu7uroya9asfKzImHQEJSKSj6Kjo/Hx8WH48OG0b9+eoKAgkpOT8fLyYsqUKQQEBLBx40a+//57unbtSkBAAEFBQVy/fh0ALy8vpk6dip+fH4GBgRw9epQ+ffrQpk0bVqxYYR7j9m3kwsLCCA4ONo//r3/9iz179gDg7u7O5MmT6dixI7169eLw4cP06NGD1q1bExUVVcB75v4poERE8tkff/zBP/7xDzZs2EDx4sX54osvAHBycmL16tV4enoyf/58Fi9ezOrVq6lTpw6LFy82r1++fHkiIiLw8PBg1KhRzJw5k5UrVzJ79uz7quPGjRs0adKEdevWUbx4cWbMmMGnn37K3LlzH4kjME3xiYjks/Lly9OwYUMAXn75ZZYvXw5Ahw4dADh06BAnTpyge/fuwK0vFNevX9+8fuvWrQGoXr06N27coESJEgDY2dlx5cqVPNdha2tL8+bNzX3Z2dlha2tL9erVOX/+/ENupeUpoERE8tlfL6e+/drR0RG49aXVpk2b5nq18+07LVhZWWFnZ2debmVlRXp6era21tbWZGZmml+npKRk6+f22Hf2ZWVlRUaG8S+a0RSfiEg+u3DhAgcOHABg7dq15qOp2+rXr8/+/fs5c+YMcGsq7o8//nigsSpWrMgvv/xCZmYmFy9e5PDhww9XvIEooERE8lnVqlX5/PPPad++PVeuXDFP5d3m7OzMpEmTGDZsGL6+vnTt2pVTp0490FgNGzakYsWKdOjQgffff5/atWvnxyYYgikrK8v4F/3nIjAwkLCwsMIuQ+SRURQvM3ernLf7vt32sJeZR0dH069fP9auXftQ/Tyujh8/Tq1atfLUVkdQIiJiSAooEZF8VKlSJR095RMFlIiIGJICSkREDEkBJSIihqSAEhERQ9KdJESkyHiybGmKO9jmW38paRnY2z74XclzEhYWRtOmTXF1dc3Xfh+Gu7u7+YvHeTF79myKFStGnz59HmpcBZSIFBnFHWzz9btgP015I9/6um316tVUq1bNUAFVWBRQIiIWEh0dzT//+U8aNmzIgQMHcHV1Zd68eTg4OHD8+HHGjRtHcnIyVapUYeLEiezatYsjR44wYsQIHBwc+Oqrr3BwcDD3d+bMGcaNG0dCQgLW1tbMnDkTFxcXBgwYwJUrV0hPT2fw4MG0adPmb8fOqZ8qVaqwcOFCNmzYQGpqKt7e3jk+FDG3NvPnzyc8PBxnZ2fKly+fL3e00DkoERELOnPmDK+99hrr1q2jZMmSbNq0CYB33nmHESNGsGbNGqpXr86cOXPw8fGhTp06fPzxx0RERGQLJ4ARI0bw2muv8c033/Dll19StmxZ7O3tmTt3LqtXr2bp0qVMnjyZ2zcIym3snPr5/vvvOXPmDKGhoURERHD06FH27duXbfzc2hw5coT169cTHh7O//73P37++ed82Xc6ghIRsaBKlSqZb+1Tu3Ztzp8/z9WrV7l69SovvPACAAEBAQwePPhv+7l27RqxsbF4e3sDYG9vD9x6VMe0adPYt28fVlZWxMbGcunSpVzHzq2fnTt3snPnTvz9/YFbN7A9ffo0jRo1MteQW5vr16/Tpk0b893avby8HnKv3WKxgBo9ejTbtm3DxcXF/K3qyZMns3XrVmxtbalSpQqTJk2iVKlSAHzyySeEhoZiZWXFe++9x0svvWSp0kRECsydj8uwtrbO9jiM/LBmzRoSEhIICwvD1tYWLy8v8xj3M3ZWVhZ9+/alW7du991myZIlD7cRubDYFF9gYCALFy7Mtqxp06asXbuWNWvW8NRTT/HJJ58AcOLECdatW8e6detYuHAh48ePfySeVSIi8iBKlixJqVKl+PHHHwGIiIgwH6kUL17c/Pj3O5UoUQI3Nzc2b94MQGpqKsnJyVy9ehUXFxdsbW3ZvXv3PR9EmFs/zZo14+uvvzaPHRsbS3x8fLZ1c2vTqFEjNm/ezM2bN7l27Rpbt259iL3zfyx2BNWoUSOio6OzLWvWrJn55/r167Nx40YAoqKi6NixI3Z2dlSuXJknn3ySw4cP4+7ubqnyRKQIun4zLV+vvHuYy8wnT55svkiicuXKTJo0Cbg13Tdu3LgcL5L46KOPGDt2LDNnzsTW1paZM2fi6+tL//798fX1pU6dOjz99NP3HDunfpo1a8bJkyfNR0fFihVjypQpuLi4mNfLrU3t2rXp0KEDfn5+ODs78/zzzz/QPvkriz5u4+9uO9+vXz/at2+Pn58fwcHB1KtXDz8/PwDGjBlD8+bN8fHx+dv+8/NxG5b4PsPjSPvp0abHbdzbwz5uQ/7e/Txuo1Aukpg/fz7W1ta8/PLLhTF8juxtrYvcP94HYYnvfYiI5KTAAyosLIxt27axZMkSTCYTAK6ursTExJjbxMbG6ktqIiJFXIF+D2r79u0sXLiQ+fPnmy9HhFuXJK5bt47U1FTOnTvH6dOnqVu3bkGWJiIiBmOxI6hhw4axd+9eEhMTad68OYMGDSIkJITU1FR69+4NQL169QgODqZatWq0b9+eDh06YG1tzdixY7G21nkOEZGizGIBNW3atLuWdenSJdf2/fv3p3///pYqR0REHjG61ZGIiBiSbnUkIkVGtXLFsbV3vHfDPMpKT8FkY59v/eWXsLAwjhw5wtixY/O8jpeXF6GhoTg7O1uwsvujgBKRIsPW3pGzwfnzJVKAKmPz56aokjMFlIiIhYWHh7No0SJMJhM1atSgffv2zJ8/n7S0NJycnPj4448pU6YMs2fP5sKFC0RHR3PhwgV69uzJG2+8kWMfU6ZMISEhgXHjxnHhwgXg1k0OGjZsmG3s3NokJiYyfPhwYmNjqV+/Pha8Z8MDU0CJiFjQ77//zvz581mxYgXOzs4kJSVhMplYuXIlJpOJVatWsXDhQkaNGgXAH3/8wbJly7h27Rrt27ene/funD59+q4+AD744AN69uyJh4cHFy5coE+fPmzYsCHb+Lm1mTt3Lg0aNGDgwIFs27aN0NDQAt8396KAEhGxoN27d+Pj42M+t+Pk5MSvv/7K0KFD+fPPP0lNTaVSpUrm9i1atMDOzg5nZ2ecnZ2Jj4/PsQ+AH374gRMnTpjXvXbt2l03ms2tzb59+5gzZw4ALVu2pHTp0pbZAQ9BASUiUsDef/99evXqRevWrdmzZ485KODuR2Skp6fn2k9mZiYrV640P9PpQdsYlS4zFxGxoCZNmrBx40YSExMBSEpK4urVq+bbuYWHhz9QH3Dr7uLLly83tzt+/Phd6+bWplGjRqxZswaA7777jsuXLz/I5lmUjqBEpMhIS0nO1yvv8nKZebVq1ejXrx89evTAysqK5557joEDBzJ48GBKly5N48aN73o0UV76+PDDD3n33XcJDg7G19eXjIwMPDw8CA4OzrZubm3efvtthg8fTseOHXF3d6dChQoPvT/ym0Uft2Fp+fm4DSh6jyJ4ELqb+aOtqP2N63EbxnM/j9vQFJ+IiBiSAkpERAxJASUij62sLAz5BdSi6n5/FwooEXlsRSfdJPXGVYWUAWRlZREfH4+Dg0Oe19FVfCLy2Fq86yy9gUpODvz/B3jfk+nanxatqShzcHDI9qXke1FAichj62pKBrO2/XFf6+hKVePQFJ+IiBiSAkpERAxJASUiIoZksYAaPXo0np6edOrUybwsKSmJ3r1707ZtW3r37m2+91NWVhbvv/8+3t7e+Pr6cvToUUuVJSIijwiLBVRgYCALFy7MtiwkJARPT08iIyPx9PQkJCQEgO3bt3P69GkiIyOZMGEC//3vfy1VloiIPCIsFlCNGjW66/kiUVFR+Pv7A+Dv78/mzZuzLTeZTNSvX58rV64QFxdnqdJEROQRUKDnoOLj4ylXrhwAZcuWJT4+HoDY2Fjc3NzM7dzc3IiNjS3I0kRExGAK7SIJk8mEKa/fnBMRkSKnQAPKxcXFPHUXFxdnfnyxq6srMTEx5nYxMTHmh3mJiEjRVKAB5eXlZX56ZHh4OK1bt862PCsri4MHD1KyZEnzVKCIiBRNFrvV0bBhw9i7dy+JiYk0b96cQYMG0bdvX4YMGUJoaCgVKlRgxowZALRo0YLvvvsOb29vHB0dmThxoqXKEhGRR4TFAmratGk5Ll+6dOldy0wmE+PGjbNUKSIi8gjSnSRERMSQFFAiImJICigRETEkBZSIiBiSAkpERAxJASUiIoakgBIREUNSQImIiCEpoERExJAUUCIiYkgKKBERMSQFlIiIGJICSkREDEkBJSIihqSAEhERQ1JAiYiIISmgRETEkBRQIiJiSAooERExJAWUiIgYkk1hDLpkyRJWrVqFyWSievXqTJo0ibi4OIYNG0ZSUhK1a9fmo48+ws7OrjDKExERAyjwI6jY2FiWLVvG119/zdq1a8nIyGDdunV8/PHH9OrVi2+//ZZSpUoRGhpa0KWJiIiBFMoUX0ZGBjdv3iQ9PZ2bN29StmxZdu/eTbt27QAICAggKiqqMEoTERGDKPApPldXV958801atWqFvb09TZs2pXbt2pQqVQobm1vluLm5ERsbW9CliYiIgeTpCKpnz555WpYXly9fJioqiqioKHbs2EFycjI7dux4oL5EROTx9bdHUCkpKSQnJ5OYmMjly5fJysoC4Nq1aw98hPPDDz9QqVIlnJ2dAWjbti379+/nypUrpKenY2NjQ0xMDK6urg/Uv4iIPB7+NqC+/PJLli5dSlxcHIGBgeaAKlGiBK+//voDDVihQgUOHTpEcnIyDg4O7Nq1izp16tC4cWM2bdpEx44dWb16NV5eXg/Uv4iIPB7+NqB69uxJz549Wb58OT169MiXAevVq0e7du0ICAjAxsaGWrVq0bVrV1q2bMnQoUOZMWMGtWrVokuXLvkynoiIPJrydJFEjx492L9/P+fPnycjI8O83N/f/4EGDQoKIigoKNuyypUr69JyERExy1NAjRw5knPnzlGzZk2sra0BMJlMDxxQIiIi95KngDpy5Ajr16/HZDJZuh4REREgj5eZV6tWjT///NPStYiIiJjl6QgqMTGRjh07UrduXWxtbc3LFyxYYLHCRESkaMtTQA0aNMjSdYiIiGSTp4B64YUXLF2HiIhINnkKKHd3d/MFEmlpaaSnp+Po6Mj+/fstWpyIiBRdeQqoAwcOmH/OysoiKiqKgwcPWqwoERGR+37chslkok2bNnz//TlOtOUAAA6cSURBVPeWqEdERATI4xFUZGSk+efMzEyOHDmCvb29xYoSERHJU0Bt3brV/LO1tTUVK1Zk3rx5FitKREQkTwE1adIkS9chIiKSTZ7OQcXExPD222/j6emJp6cngwYNIiYmxtK1iYhIEZangBo9ejReXl7s2LGDHTt20KpVK0aPHm3p2kREpAjLU0AlJCTQuXNnbGxssLGxITAwkISEBEvXJiIiRVieAsrJyYmIiAgyMjLIyMggIiICJycnS9cmIiJFWJ4CauLEiWzYsIGmTZvSrFkzNm3axIcffmjp2kREpAjL01V8s2bNYvLkyZQuXRqApKQkJk+erKv7RETEYvJ0BPXrr7+awwluTfkdP37cYkWJiIjkKaAyMzO5fPmy+XVSUhIZGRkPPOiVK1cICgrCx8eH9u3bc+DAAZKSkujduzdt27ald+/e2cYTEZGiJ09TfG+++SZdu3bFx8cHgI0bN9KvX78HHvSDDz7gpZdeYtasWaSmpnLz5k0WLFiAp6cnffv2JSQkhJCQEEaOHPnAY4iIyKMtT0dQ/v7+zJkzhzJlylCmTBnmzJmDv7//Aw149epV9u3bxyuvvAKAnZ0dpUqVIioqytynv78/mzdvfqD+RUTk8ZCnIyiAZ599lmefffahB4yOjsbZ2ZnRo0fzyy+/ULt2bd59913i4+MpV64cAGXLliU+Pv6hxxIRkUfXfT9u42Glp6dz7NgxunfvTnh4OI6OjoSEhGRrYzKZzA9IFBGRoqnAA8rNzQ03Nzfq1asHgI+PD8eOHcPFxYW4uDgA4uLicHZ2LujSRETEQAo8oMqWLYubmxunTp0CYNeuXTzzzDN4eXkRHh4OQHh4OK1bty7o0kRExEDyfA4qP/3nP/9hxIgRpKWlUblyZSZNmkRmZiZDhgwhNDSUChUqMGPGjMIoTUREDKJQAqpWrVqEhYXdtXzp0qWFUI2IiBhRgU/xiYiI5IUCSkREDEkBJSIihqSAEhERQ1JAiYiIISmgRETEkBRQIiJiSAooERExJAWUiIgYkgJKREQMSQElIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJICigRETEkBZSIiBiSAkpERAxJASUiIoakgBIREUMqtIDKyMjA39+ff/3rXwCcO3eOLl264O3tzZAhQ0hNTS2s0kRExAAKLaCWLVvGM888Y3798ccf06tXL7799ltKlSpFaGhoYZUmIiIGUCgBFRMTw7Zt23jllVcAyMrKYvfu3bRr1w6AgIAAoqKiCqM0ERExiEIJqIkTJzJy5EisrG4Nn5iYSKlSpbCxsQHAzc2N2NjYwihNREQMosADauvWrTg7O1OnTp2CHlpERB4hNgU94P79+9myZQvbt28nJSWFa9eu8cEHH3DlyhXS09OxsbEhJiYGV1fXgi5NREQMpMCPoIYPH8727dvZsmUL06ZNo0mTJkydOpXGjRuzadMmAFavXo2Xl1dBlyYiIgZimO9BjRw5ksWLF+Pt7U1SUhJdunQp7JJERKQQFfgU350aN25M48aNAahcubIuLRcRETPDHEGJiIjcSQElIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJICigRETEkBZSIiBiSAkpERAxJASUiIoakgBIREUNSQImIiCEpoERExJAUUCIiYkgKKBERMSQFlIiIGJICSkREDEkBJSIihqSAEhERQ7Ip6AEvXrzIO++8Q3x8PCaTiVdffZWePXuSlJTE0KFDOX/+PBUrVmTGjBmULl26oMsTERGDKPAjKGtra0aNGsX69ev56quv+OKLLzhx4gQhISF4enoSGRmJp6cnISEhBV2aiIgYSIEHVLly5ahduzYAJUqU4OmnnyY2NpaoqCj8/f0B8Pf3Z/PmzQVdmoiIGEihnoOKjo7m+PHj1KtXj/j4eMqVKwdA2bJliY+PL8zSRESkkBVaQF2/fp2goCDGjBlDiRIlsr1nMpkwmUyFVJmIiBhBoQRUWloaQUFB+Pr60rZtWwBcXFyIi4sDIC4uDmdn58IoTUREDKLAAyorK4t3332Xp59+mt69e5uXe3l5ER4eDkB4eDitW7cu6NJERMRACvwy859++omIiAiqV6+On58fAMOGDaNv374MGTKE0NBQKlSowIwZMwq6NBERMZACDygPDw9+/fXXHN9bunRpAVcjIiJGpTtJiIiIISmgRETEkBRQIiJiSAooERExJAWUiIgYkgJKREQMSQElIiKGpIASERFDUkCJiIghKaBERO6QlZ5S2CU8EgpiPxX4rY5ERIzMZGPP2eDnC7sMw6sy9meLj6EjKBERMSQFlIiIGJICSu6L5ufzRvtJ5OHpHJTcF83P501BzM+LPO50BCUiIoakgBIREUNSQImIiCEpoERExJAUUCIiYkiGC6jt27fTrl07vL29CQkJKexyRESkkBgqoDIyMggODmbhwoWsW7eOtWvXcuLEicIuS0RECoGhAurw4cM8+eSTVK5cGTs7Ozp27EhUVFRhlyUiIoXAUF/UjY2Nxc3Nzfza1dWVw4cP59r+/PnzBAYG5tv4T+ZbT4+vwMBwoFphl2F8+fh3mZ/0N35v+hvPo3z8G3/iiSdYtGjRXcsNFVD3a8+ePYVdgoiIWIihpvhcXV2JiYkxv46NjcXV1bUQKxIRkcJiqIB6/vnnOX36NOfOnSM1NZV169bh5eVV2GWJiEghMNQUn42NDWPHjuWtt94iIyODzp07U62a5oJFRIoiU1ZWVlZhFyEiIvJXhpriExERuU0BJSIihqSAEgCuXLnC559//kDrLlmyhOTk5HyuSESKOgWUALcCasWKFQ+07rJlyxRQ8kiKjo5mzZo1971eWFgYwcHBub6/ePFiOnTogK+vLz179uT8+fMPU2aRZair+KTwTJ06lbNnz+Ln58eLL76Ii4sLGzZsIDU1FW9vb4KCgrhx4wZDhgwhJiaGzMxMBgwYwKVLl4iLi6Nnz544OTmxfPnywt4UkTw7f/48a9euxdfX96730tPTsbF5sP8ia9Wqxddff42joyNffPEFU6ZMYcaMGQ9bbpGjgBIAhg8fzu+//05ERATff/89mzZtIjQ0lKysLPr378++fftISEigXLly5rvMX716lZIlS7JkyRKWLl2Ks7NzIW+FFDXh4eEsWrQIk8lEjRo1GDx4MGPGjCExMRFnZ2cmTZpEhQoVGDVqFCVKlODIkSP8+eefjBw5Eh8fH6ZOncrJkyfx8/MjICCAUqVKERkZyY0bN8jMzGTOnDmMGTOGc+fO4ejoSHBwMDVr1rxnXU2aNDH/XL9+fb755htL7obHlgJK7rJz50527tyJv78/ADdu3OD06dN4eHgwefJkpkyZQqtWrfDw8CjkSqUo+/3335k/fz4rVqzA2dmZpKQkRo0aRUBAAAEBAYSGhvL+++8zb948AOLi4vjiiy84deoU/fv3x8fHh+HDh/Ppp5/yySefALem7o4dO8Y333yDk5MTEyZM4LnnnmPevHns2rWLf//730RERNxXnaGhoTRv3jzft78oUEDJXbKysujbty/dunW7672wsDC+++47ZsyYQZMmTRg4cGAhVCgCu3fvxsfHx3zk7uTkxIEDB5g9ezYAfn5+TJkyxdy+TZs2WFlZ8eyzz3Lp0qVc+23atClOTk4A/PTTT+b+PD09SUpK4tq1a3muMSIigiNHjvDZZ5/d9/aJLpKQ/6948eJcv34dgGbNmvH111+bX8fGxhIfH09sbCyOjo74+fnRp08fjh07dte6IkZlZ2eXp3aOjo75Mt4PP/zAggULmD9/fp7HluwUUALcut19gwYN6NSpEzt37qRTp05069YNX19fgoKCuH79Or/99huvvPIKfn5+zJkzh/79+wPw6quv8tZbb9GjR49C3gopSpo0acLGjRtJTEwEICkpCXd3d9atWwfAmjVr7jkNfa8PVx4eHubzR3v27OGJJ56gRIkS96zt2LFjjB07lvnz5+Pi4pLXTZK/0K2OROSRtXr1ahYtWoSVlRXPPfccgwYNYvTo0TleJNGyZUt8fHwAcHd358CBA6SlpdGnTx+SkpIIDAykVKlSHDlyhLFjxwK3Qi+niyTCwsKytfurXr168dtvv1G2bFkAypcvz4IFCwpmpzxGFFAiImJImuITERFD0lV8IiIPaP78+WzcuDHbMh8fH/P5WXk4muITERFD0hSfiIgYkgJKREQMSQElIiKGpIASMYj09PTCLkHEUHQVn0gBmTt3Lt988w3Ozs6UL1+e2rVrs23bNmrWrMlPP/1Ep06dqFWrFpMnTyYjI4M6deowfvx47Ozs8PLyIjQ0FGdnZ37++Wc++ugjli9fzuzZszl79ixnz54lMTGRt956i1dffbWwN1UkXyigRArA4cOHiYyM5JtvviEtLY3AwEBq164NQFpaGmFhYaSkpNC2bVuWLFlC1apVeeedd/jiiy/o1avX3/b966+/snLlSm7cuEFAQAAtWrTA1dW1ALZKxLI0xSdSAPbv30/r1q2xt7enRIkStGrVyvxehw4dAPjjjz+oVKkSVatWBSAgIIAff/zxnn23bt0aBwcHnJ2dady4MT///LNlNkKkgCmgRApZXu6ebW1tze2vLKakpGR7z2QyWaQukcKmgBIpAA0aNGDr1q2kpKRw/fp1tm3bdlebqlWrcv78ec6cOQPcepZQo0aNAKhYsSJHjhwBIDIyMtt6UVFRpKSkkJiYyN69e3n++ectuzEiBUTnoEQKQN26dfHy8uLll1/GxcWF6tWrU7JkyWxt7O3tmTRpEoMHDzZfJNG9e3cABg4cyLvvvsvMmTNp3LhxtvVq1KjBG2+8QWJiIgMGDND5J3ls6FZHIgXk+vXrFC9enOTkZF577TUmTJhgvlDiQc2ePZtixYrRp0+ffKpSxDh0BCVSQMaOHcuJEydISUkhICDgocNJ5HGnIygRETEkXSQhIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJI/w8xTJXiNQynhAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Визуацизация распределений\n",
    "sns.countplot(data=users_premium, x = 'group', hue = 'premium')\n",
    "\n",
    "sns.despine()\n",
    "plt.title('Premium subscription depending on group\\n')\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Визуально различия не велики. Проверим данные статистикой:\n",
    "Есть категориальные переменные (категория в зависимости от подписки, группа пользователя) → нужен хи-квадрат\n",
    "\n",
    "- $H_0$: нет статистически значимой разницы между распределениями пользователей в зависимости от подписки (продлили/ не продлили) в тестовой и контрольной группе (p-value > 0.05)\n",
    "- $H_1$: есть статистически значимая разница между распределениями пользователей в зависимости от подписки (продлили/ не продлили) в тестовой и контрольной группе\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.04205977241354288, 0.8375058771665312)"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stat, p, dof, expected = chi2_contingency(pd.crosstab(users_premium.premium, users_premium.group))\n",
    "stat, p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test</th>\n",
       "      <th>lambda</th>\n",
       "      <th>chi2</th>\n",
       "      <th>dof</th>\n",
       "      <th>pval</th>\n",
       "      <th>cramer</th>\n",
       "      <th>power</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pearson</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.042060</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.837506</td>\n",
       "      <td>0.012412</td>\n",
       "      <td>0.054832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cressie-read</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.042093</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.837443</td>\n",
       "      <td>0.012417</td>\n",
       "      <td>0.054836</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>log-likelihood</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.042163</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.837309</td>\n",
       "      <td>0.012428</td>\n",
       "      <td>0.054844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>freeman-tukey</td>\n",
       "      <td>-0.500000</td>\n",
       "      <td>0.042221</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.837199</td>\n",
       "      <td>0.012436</td>\n",
       "      <td>0.054851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mod-log-likelihood</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.042283</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.837081</td>\n",
       "      <td>0.012445</td>\n",
       "      <td>0.054858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>neyman</td>\n",
       "      <td>-2.000000</td>\n",
       "      <td>0.042418</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.836824</td>\n",
       "      <td>0.012465</td>\n",
       "      <td>0.054873</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 test    lambda      chi2  dof      pval    cramer     power\n",
       "0             pearson  1.000000  0.042060  1.0  0.837506  0.012412  0.054832\n",
       "1        cressie-read  0.666667  0.042093  1.0  0.837443  0.012417  0.054836\n",
       "2      log-likelihood  0.000000  0.042163  1.0  0.837309  0.012428  0.054844\n",
       "3       freeman-tukey -0.500000  0.042221  1.0  0.837199  0.012436  0.054851\n",
       "4  mod-log-likelihood -1.000000  0.042283  1.0  0.837081  0.012445  0.054858\n",
       "5              neyman -2.000000  0.042418  1.0  0.836824  0.012465  0.054873"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Посмотрим статистику еще через пингвин\n",
    "exp, obs, stats = pg.chi2_independence(data=users_premium, \n",
    "                                       x='group', \n",
    "                                       y='premium') \n",
    "stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Согласно статистике (p-value > 0.05) нулевую гипотезу об отсутствии разницы между распределениями пользователей в зависимости от подписки (продлили/ не продлили) в тестовой и контрольной группе мы отклонить не можем."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Сравним количество дней посещения пользователей в зависимости от группы (visit days)\n",
    "\n",
    "# Преобразуем колонку с номерами дней в нужный формат\n",
    "users_merged['visit_days'] = users_merged['visit_days'].str.split(',\\s*')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Добавим колонку с количеством дней посещения\n",
    "users_merged['days_count'] = users_merged.visit_days.str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Изменим NaN на нули и поменяем тип данных на int\n",
    "users_merged['days_count'] = users_merged['days_count'].fillna(0).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>group</th>\n",
       "      <th>days_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>control_1</td>\n",
       "      <td>9.707317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>control_2</td>\n",
       "      <td>9.287671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>test</td>\n",
       "      <td>9.244094</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       group  days_count\n",
       "0  control_1    9.707317\n",
       "1  control_2    9.287671\n",
       "2       test    9.244094"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Среднее количество дней посещения\n",
    "users_merged.groupby('group', as_index = False).agg({'days_count' : 'mean'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de1QV9f7/8ecGJBAEhALz0jFPRh3SssRbJgniJbkIhkp51+M3FW94MqyTld3U0kwrU/KUlVFqCIKlpqaW1/BXoedUxyxLLEHxgoqKwP794bf9lUDcIHvvAV+PtVxrz8xnZt6zh7Vfzu0zJrPZbEZERMRgnBxdgIiISEUUUCIiYkgKKBERMSQFlIiIGJICSkREDEkBJSIihqSAErGB0NBQtm3b5ugyyti5cyddunSxDPfu3ZudO3c6sCKRyrk4ugCpHUJDQzl69CjOzs44Oztzyy23EB0dTf/+/XFyqtv/zxk0aBBRUVHExcU5upQatXr1akeXIFIpBZRY7c0336RTp06cOnWKXbt28fzzz5Odnc2LL77o6NJEKC4uxsVFP2l1Sd3+r6/YRIMGDQgLC2Pu3LmsXLmS//73vwBs2rSJPn36cPfddxMSEsL8+fMt84waNYr33nuvzHIiIyP57LPPMJvNvPDCC3Ts2JG7776byMhIyzL/7MSJE0ydOpXOnTsTHBzMmDFjLNOWLVtGeHg47dq145FHHiE3NxeAnJwcAgMDKS4utrQdNGgQy5cvByA1NZX4+HhmzpxJcHAwoaGhbN68GYBXXnmFrKwspk+fTps2bZg+fXqFdaWlpdG1a1fat2/PggULykwrLS1l0aJFdOvWjfbt2zNhwgROnDhRpraPPvqIzp0707lzZxYvXlyleVeuXMn9999fbt3nzp0jKSmJ4OBgHnjgAfbs2VOmrktPQ86fP58JEyYwZcoU2rRpQ+/evcu0//e//02fPn1o06YN48ePZ+LEibzyyisVfhelpaW88cYbdO3alY4dOzJlyhROnTplVc1/dvz4cR555BHuvvtu+vbtyyuvvEJ8fLxlemBgIEuXLqV79+50794duLq/gwEDBjB9+nTuueceevbsyfbt2y9bm9ieAkqqrXXr1jRq1IisrCwA3N3dmTlzJllZWSxcuJCUlBTWr18PQJ8+fVi1apVl3u+//568vDxCQkL48ssvycrKYu3atezevZu5c+fi4+NT4TqnTJnC2bNnWb16Ndu2bWPo0KEAbN++ndmzZzN37ly+/PJLmjRpQmJiotXbkp2dzc0338yOHTsYOXIkTzzxBGazmUmTJtG2bVumTZvG119/zbRp08rN++OPP/LMM88wa9YsvvjiC06cOMHhw4ct09977z3Wr1/P+++/zxdffIG3t3e5oNu5cyfr1q1j8eLFJCcnW4LDmnl3797NmjVrWLJkCa+//jr79+8H4LXXXuPXX3/ls88+Y/HixaSlpVX6HWzcuJHevXuTlZVFaGgozz77LABFRUUkJCQQExPDrl27iIiIsOzXiqSmprJy5Ureffdd1q9fT2FhodU1/9n06dNxd3dn69atzJw5s8JtWL9+PcuWLeOTTz6pkb+Dm266iR07djB+/HgSEhIs/yEQ+1NAyVXx9/fn5MmTALRv357AwECcnJy47bbb6N27N7t27QIgLCyMAwcOcODAAQDS09Pp1asXrq6uuLi4cObMGX766SfMZjN//etf8ff3L7euvLw8tmzZwjPPPIO3tzf16tWjXbt2AGRkZNC3b1+CgoJwdXUlMTGRb775hpycHKu2o3HjxvTr1w9nZ2diYmI4cuQIR48etWreNWvWcP/99xMcHIyrqysTJkwoc13uww8/ZNKkSTRq1AhXV1cSEhJYu3Ztmf/Jjx07lvr16xMYGEhsbCyZmZlWz5uQkICbmxu33XYbt912G99//z0An376KY888gg+Pj7ceOONDBo0qNLtuOeeewgJCcHZ2Zno6GjLcr799luKi4sZPHgw9erVo3v37rRq1eqyy8nIyGDo0KE0a9YMDw8PEhMT+eSTT6yq+VIlJSWsW7eOcePG4e7uzi233EKfPn3KtRs1ahQ+Pj64ubld9d+Br68vQ4YMoV69ejzwwAPcfPPNbNq0yap5pebphK1cldzcXLy9vYGLP2Qvv/wy+/bt48KFCxQVFdGzZ08ArrvuOnr16sWqVatISEggMzOTefPmAdCxY0cefvhhpk+fzqFDh+jevTuPPfYYnp6eZdZ1+PBhvL29Leu7VF5eHkFBQZZhDw8PfHx8yM3NJSAg4Irbcf3111s+u7u7A1BYWGjVd5CXl0ejRo0sw/Xr1y9zBPjbb78xduzYMqHl5OREfn6+ZfjGG2+0fG7SpInlFKc18/659j/qzsvLK7Pcxo0bV7odly7Hzc2N8+fPU1xcTF5eHgEBAZhMpgrr/bO8vDyaNGlSZnuKi4utqvlSx44do7i4uMy6KlrvpeOu9u/gz9vZuHFj8vLyrjif2IaOoKTasrOzyc3N5Z577gFg8uTJhIWFsXnzZnbv3s2AAQO4tLP8mJgYMjIy2L59O+7u7rRp08YybfDgwaSmpvLJJ59w4MAB3nrrrXLra9SoESdPnqSgoKDcNH9/fw4dOmQZLiws5MSJEwQEBFC/fn3g4jWZPxw5cuTqv4BL1n3pKb2zZ8+WOS3UqFEjkpOTycrKsvzbs2dPmR/M33//3fL5t99+sxxBWjPv5dxwww1llnvp56q44YYbyM3NLbMvK1vWn/fFb7/9houLC35+flVar6+vLy4uLmW+24rWe2mgXO3fQUXbWdHRvNiHAkqq7PTp03z++eckJiYSFRVFYGAgAGfOnMHb25vrrruO7Oxsy2mqP7Rp0wYnJydmzJhBVFSUZXx2djbffvstFy5cwN3dHVdX1wpvXff396dLly4888wznDx5kgsXLvDVV18BEBERQWpqKt999x1FRUXMmTOH1q1b07RpU3x9fQkICCA9PZ2SkhJWrFjBwYMHrd7e66+/vtL2PXr0YNOmTWRlZVFUVMS8efMoLS21TI+Pj2fu3LmWH85jx46Vu4bzxhtvcPbsWfbt20dqaioPPPCA1fNeTq9evVi0aBEnT57k8OHD5W5SsdZdd92Fs7Mz77//PsXFxaxfv77cDReXioiIYMmSJRw8eJAzZ87wyiuv0KtXryrfYefs7Ex4eDivvfYaZ8+eZf/+/aSnp1c6z9X+HRw7dox3332XCxcu8Omnn7J//35CQkKqVLfUHAWUWO2RRx6hTZs2hISE8OabbzJs2LAyt5g/9dRTzJs3jzZt2vD666/Tq1evcsuIjo7mv//9L9HR0ZZxZ86c4Z///Cft2rWja9eu+Pj4MGLEiAprmDVrFi4uLvTq1YtOnTqxZMkSADp16sSECRMYN24cnTt35uDBg2XuMnv22WdZvHgx7du358cffyxz9HYlgwcPZu3atQQHB/Pcc8+Vm96yZUumTZvGP/7xD+677z68vLzKnPIbPHgwoaGhDB8+nDZt2tCvXz+ys7PLLKNdu3aEh4czdOhQhg8fTufOna2e93ISEhJo3LgxYWFhDB8+vMx3XhWurq7Mnz+fFStWEBwczKpVq7j//vtxdXWtsH3fvn2Jiopi4MCBhIWF4erqypNPPlmtdU+bNo1Tp05x7733MmXKFHr37n3Z9cLV/x20bt2aX375hQ4dOjB37lzmzZtHw4YNq1W7XD2TXlgo9pSWlsZHH31ESkqKo0sxhJycHMLCwvj3v/9dq57hiYuLY8CAAfTt29eu633ppZc4evQoM2fOrPFlp6amsnz5cv1tGoiOoMRuzp49ywcffED//v0dXYpU0a5duzhy5AjFxcWsXLmSH374gfvuu8/m692/fz/ff/89ZrOZ7OxsVqxYQXh4uM3XK8ZQe/7LJrXaF198wbhx4+jYsSMRERGOLkeq6Oeff2bixImcPXuWpk2bMm/ePLvcPHDmzBkmT55MXl4efn5+DB8+nLCwMJuvV4xBp/hERMSQdIpPREQMSQElIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJICigRETEkBZSIiBiSAkpERAxJASUiIoakgBIREUNSQImIiCEpoERExJAUUCIiYkgKKBERMSQFlIiIGFKdCqgRI0Y4ugQREakhdSqgjh8/7ugSRESkhtSpgBIRkbpDASUiIoakgBIREUNycXQBIiJGduHCBXJycjh37pyjS6m13NzcaNq0KfXq1avSfAooEZFK5OTk0KBBA5o3b47JZHJ0ObWO2WwmPz+fnJwcbr755irNq1N8IiKVOHfuHH5+fgqnajKZTPj5+VXrCFQBJSJyBQqnq1Pd708BJSIihqRrUNU14yY4d9K6tm7ekPSrbesREaljFFDVde4kDMm0ru2SCNvWIiLXjOLiYlxcro2f7mtjK0VEaonXX3+dVatW4evry4033khQUBCbNm3itttuY/fu3URERHD77bczc+ZMSkpKuOOOO3jmmWdwdXUlNDSUFStW4Ovry549e5g1axbvvfce8+fP59dff+XXX3/l+PHjjBw5kn79+jl6U69IASUiYhDZ2dmsW7eOVatWceHCBWJjYwkKCgIuPo+VmprK+fPn6d69O++88w4333wzU6ZM4YMPPmDo0KGVLvuHH35g2bJlFBYWEhMTQ0hICAEBAXbYqurTTRIiIgbx//7f/yMsLIzrrrsOT09Punbtapn2wAMPAPDzzz/TtGlTyzNFMTExZGVlXXHZYWFhuLm54evrS/v27dmzZ49tNqIGKaBERGoBd3f3K7ZxdnbGbDYDcP78+TLTauOt8gooERGDuPvuu/n88885f/48Z86cYdOmTeXa3HzzzRw6dIhffvkFgPT0dIKDgwFo0qQJe/fuBWDdunVl5tuwYQPnz5/n+PHj7Nq1i1atWtl2Y2qArkGJiBhE69atCQ0NJSoqCj8/P2699VYaNGhQps11113Hiy++yIQJEyw3ScTHxwOQkJDAE088wauvvkr79u3LzBcYGMjgwYM5fvw4Y8aMMfz1J1BAiYgYyvDhwxk3bhxnz57l4YcfJigoqNwddx07diQtLa3cvG3btmXt2rUVLjcwMJBZs2bZpGZbUUCJiBjItGnT+PHHHzl//jwxMTGWu/iuRQqoS1Wld4jrGly5jYhIFc2ePbvGlzlu3LgaX6Y9KKAuVZXeIURExKZsFlBTp05l06ZN+Pn5kZl58Ud/4sSJ/PzzzwCcOnWKBg0akJ6eXm7e0NBQPDw8cHJywtnZmdTUVFuVKSIiBmWzgIqNjWXgwIE89thjlnFz5861fJ4xYwaenp6XnX/JkiX4+vraqjwRETE4mz0HFRwcjLe3d4XTzGYzn376KRER6kRVREQq5pBrUFlZWfj5+dG8efPLthkxYgQmk4n+/fvTv39/+xUnIlKJTi9u4LeTVX877OU09nZj29Swy04vKCggIyODhx9+uMrLfuedd+jfv79VvVAYkUMCKjMzs9Kjp5SUFAICAsjPz2fYsGG0aNHC8qS0iIgj/XbyHCl/71Bjy4tP3lHp9IKCAlJSUqoVUO+++y5RUVEKKGsVFxfz2WefVXrjwx9POPv5+REeHk52drYCSkSuSbNnz+bXX38lOjqaTp064efnx6effkpRURHh4eGMHz+ewsJCJk6cyOHDhyktLWXMmDEcPXqUvLw8hgwZgo+PD++9956jN6XK7B5Q27Zto0WLFjRq1KjC6YWFhZSWluLp6UlhYSFbt25lzJgxdq5SRMQYJk+ezL59+0hPT+fLL79k7dq1rFixArPZzOjRo/nqq684duwY/v7+LFq0CPi/u6TfeeedWn3Dmc1ukkhMTGTAgAH8/PPPdOnSheXLlwPwySef0Lt37zJtc3Nz+fvf/w5Afn4+Dz30EFFRUcTFxRESEkKXLl1sVaaISK2xdetWtm7dSp8+fYiJieGnn37iwIED3HrrrWzbto2XXnqJrKyscv331VY2O4KaM2dOheNnzJhRblxAQADJyckANGvWjFWrVtmqrEq1PpdMwcICq9p6uUL2MC8bVyQi8n/MZjOjRo1iwIAB5aalpqayefNm5s6dS4cOHUhISHBAhTVLPUlcogAPUiLrW9U2PqPQxtWIiICHhwdnzpwBoHPnzrz66qtERkbi4eFBbm4uLi4uFBcX4+PjQ3R0NF5eXpYzVn/MW1tP8SmgRESqoLG32xXvvKvq8irTsGFD7r77biIiIrjvvvuIiIiwHEHVr1+fl156iV9++YVZs2bh5OSEi4sLTz/9NAD9+vVj5MiR+Pv76yYJEZG6rrJnlmzlzx3IDhkypMzwTTfdxH333VduvkGDBjFo0CCb1mZLeqOuiIgYkgJKREQMSQElIiKGpIASERFDUkCJiIghKaBERMSQdJu5iEhVzAmCgpyaW55XU0j8d80trw5RQImIVEVBDgzJrLnlLbHPi1tzcnL4+uuviYyMrNJ8qamp7N27l2nTplU4/e2332b58uU4Ozvj6+vLCy+8QJMmTWqiZJ3iExG5Fhw6dIjMzIqDtbi4uNrLvf322/n444/JyMigR48evPTSS9Ve1p/pCEpEpBZIS0tj8eLFmEwmAgMDmTBhAo8//jjHjx/H19eXF198kcaNG5OUlISnpyd79+7lyJEjPProo/Ts2ZPZs2ezf/9+oqOjiYmJwcvLi3Xr1llecfTaa6/x+OOPc/DgQdzd3Zk+fTq33XbbFevq0OH/Xt5411131Whn3wooERGD27dvHwsWLCAlJQVfX19OnDhBUlISMTExxMTEsGLFCp577jneeOMNAPLy8vjggw/46aefGD16ND179mTy5Mn861//YuHChcDFU3f/+c9/WLVqFT4+Pjz77LP87W9/44033mD79u089thjpKenV6nOFStW1OjrkXSKT0TE4Hbs2EHPnj0tvZL7+Pjw9ddfExFx8fpVdHQ0u3fvtrTv1q0bTk5O3HLLLRw9evSyy7333nvx8fEBYPfu3URHRwPQsWNHTpw4wenTp62uMT09nb179zJy5Mgqb9/l6AhKRKSOcXV1taqdu7t7jaxv27ZtvPnmm7z//vtWr9saCigRkarwalqzd955Nb1ikz9eQDh06FAaNmzIiRMnaNOmDatXr6ZPnz5kZGTQtm3bSpdx6XulKtK2bVtWrVrF2LFj2blzJw0bNsTT0/OKtf3nP/9h2rRpvPXWW/j5+V2xfVXYLKCmTp3Kpk2b8PPzs9w5Mn/+fJYtW2Y5TE1MTCQkJKTcvFu2bOH555+ntLSUuLg4Ro0aZasyRUSqxgHPLLVs2ZJHHnmEQYMG4eTkxN/+9jeefPJJpk6dyuLFiy03SVQmMDAQJycnoqKiiI2Nxcur7BvBExISePzxx4mMjMTd3b3Ct59XZNasWRQWFjJhwgQAbrzxRt58883qbeifmMxms7lGlvQnX331FfXr1+exxx4rE1D169dnxIgRl52vpKSEHj168PbbbxMQEMCDDz7InDlzuOWWW664ztjYWFJTU6tdc/Ok1VV6o+6B/7Hyle9LIuDpk9WuS0Qc57vvvuP22293dBm1XnW+R5vdJBEcHIy3t3eV58vOzuYvf/kLzZo1w9XVld69e7NhwwYbVCgiIkZm92tQS5cuJS0tjTvuuIOkpKRyIZabm0ujRo0swwEBAWRnZ9u7TBERucSCBQtYs2ZNmXE9e/Zk9OjRNlunXQMqPj6eMWPGYDKZePXVV5kxY8YVz5uKiDia2WzGZDI5ugyHGj16dLXDqLpXkuz6HNT111+Ps7MzTk5OxMXFsWfPnnJtAgICOHz4sGU4NzeXgIAAe5YpImLh5uZGfn5+tX9kr3Vms5n8/Hzc3NyqPK9dj6Dy8vLw9/cHYP369bRs2bJcm1atWnHgwAEOHjxIQEAAq1evZvbs2fYsU0TEomnTpuTk5HDkyBFHl1Jrubm50bTplW+n/zObBVRiYiK7du3i+PHjdOnShXHjxrFr1y6+//57AJo0acL06dOBi0dJ//znP0lOTsbFxYVp06YxcuRISkpK6Nu3b4VBJiJiD/Xq1ePmm292dBnXJJsF1Jw5c8qNi4uLq7BtQEAAycnJluGQkJAKn48SEZFrh/riExERQ1JAiYiIISmgRETEkBRQIiJiSAooERExJAWUiIgYkgJKREQMSQElIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJICigRETEkBZSIiBiSXV9YeK1qfS6ZgqTVVrX1cnMh++keNq5IRMT4FFB2UIAHKX/vYFXb+OQdNq5GRKR2sFlATZ06lU2bNuHn50dmZiYAM2fO5PPPP6devXrcdNNNvPjii3h5eZWbNzQ0FA8PD5ycnHB2diY1NdVWZYqIiEHZ7BpUbGwsb731Vplx9957L5mZmWRkZNC8eXMWLlx42fmXLFlCenq6wklE5Bpls4AKDg7G29u7zLjOnTvj4nLxoO2uu+7i8OHDtlq9iIjUcg67i+/jjz+mS5cul50+YsQIYmNj+eijj+xYlYiIGIVDbpJYsGABzs7OREVFVTg9JSWFgIAA8vPzGTZsGC1atCA4ONjOVYqIiCPZ/QgqNTWVTZs28fLLL2MymSpsExAQAICfnx/h4eFkZ2fbs0QRETEAuwbUli1beOutt1iwYAHu7u4VtiksLOT06dOWz1u3bqVly5b2LFNERAzAZqf4EhMT2bVrF8ePH6dLly6MGzeORYsWUVRUxLBhwwC48847mT59Orm5ufzzn/8kOTmZ/Px8xo4dC0BJSQkRERGVXqsSEZG6yWYBNWfOnHLj4uLiKmwbEBBAcnIyAM2aNWPVqlW2KktERGoJ9cUnIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJICigRETEkBZSIiBiSAkpERAzJqoDavXu3VeNERERqilUB9dxzz1k1TkREpKZU2tXR119/zddff82xY8d4++23LeNPnz5NSUmJzYsTEZFrV6UBdeHCBQoLCykpKeHMmTOW8Z6ensybN8/mxYmIyLWr0oBq164d7dq1IyYmhiZNmtirJhEREet6My8qKuLJJ5/k0KFDFBcXW8a/++67NitMRESubVYF1IQJExgwYABxcXE4OenOdBERsT2rAsrFxYWHHnrI1rWIiIhYWHU41LVrV5YuXUpeXh4nTpyw/BMREbEVq46gVq5cCcDixYst40wmExs2bKh0vqlTp7Jp0yb8/PzIzMwE4MSJE0yaNIlDhw7RpEkT5s6di7e3d4XrXLBgAQCjR48mJibGui0SEZE6waqA2rhxY7UWHhsby8CBA3nssccs4xYtWkTHjh0ZNWoUixYtYtGiRTz66KNl5jtx4gSvvfYaH3/8MSaTidjYWEJDQysMMhERqZusCqi0tLQKx/fp06fS+YKDg8nJySkzbsOGDbz33nuW+QcNGlQuoL788kvuvfdefHx8ALj33nv54osviIiIsKZcERGpA6wKqD179lg+nz9/nu3btxMUFHTFgKpIfn4+/v7+ANxwww3k5+eXa5Obm0ujRo0swwEBAeTm5lZ5XVK7tX56LQXniq/cEPBycyH76R42rkhE7MmqgHryySfLDBcUFDBp0qSrXrnJZMJkMl31cqRuKjhXTMrfO1jVNj55h42rERF7q9ZDTe7u7uVO3VnLz8+PvLw8APLy8vD19S3XJiAggMOHD1uGc3NzCQgIqNb6RESkdrLqCOqRRx6xfC4tLWX//v306tWrWisMDQ0lLS2NUaNGkZaWRlhYWLk2nTt3Zs6cOZw8eRK4eE0qMTGxWusTEZHayaqAGj58uOWzs7MzTZo0KXON6HISExPZtWsXx48fp0uXLowbN45Ro0YxceJEVqxYQePGjZk7dy5w8TrXhx9+yPPPP4+Pjw9jxozhwQcfBGDs2LGWGyZEROTaYFVAtWvXjqNHj1pulmjevLlVC58zZ06F45csWVJuXKtWrWjVqpVl+MEHH7QElIiIXHusugb1ySefEBcXx5o1a/j0008tn0VERGzFqiOoN998kxUrVuDn5wfAsWPHGDp0KD179rRpcSIicu2y6gjKbDZbwgnAx8cHs9lss6JERESsOoLq3LkzI0aMoHfv3sDFU35dunSxaWEiInJtqzSgfvnlF44ePcpjjz3GunXr2L17NwB33XUXUVFRdinwWuN5nQvNk1Zb1Va9J4hIXVZpQL3wwguW54+6d+9O9+7dAfjhhx944YUXePPNN21f4TUmeXBbq9uq9wQRqcsqvQZ19OhRAgMDy40PDAzk0KFDNitKRESk0oA6derUZaedO3euxosRERH5Q6UBdccdd7Bs2bJy45cvX05QUJDNihIREan0GtTjjz9OQkICGRkZlkDau3cvFy5c4LXXXrNLgSIicm2qNKCuv/56PvzwQ3bs2MG+ffsACAkJoWPHjnYpTkRErl1WPQfVoUMHOnSw7r08IiIiNaFa74MSERGxNQWUiIgYklWn+KQGvNPbunauHvBQ+Tsn7W7GTXDupHVt3bwh6Vfb1GHt98aTtlm/iDiMAspeer5oXbs1U21bh7XOnYQhmda1XRJhuzqs/d4yCm1Xg4g4hN0D6qeffmLSpEmW4YMHDzJ+/HiGDh1qGbdz507GjBlD06ZNAQgPDychIcHepYqIiAPZPaBatGhBeno6ACUlJXTp0oXw8PBy7dq2bcvChQvtXZ6IiBiEQ2+S2L59O82aNaNJkyaOLENERAzIoQG1evVqIiIqvn7xzTffEBUVxciRIy0PCYuIyLXDYQFVVFTExo0bK3xtfFBQEBs3bmTVqlUMGjSIsWPHOqBCERFxJIcF1JYtWwgKCuL6668vN83T0xMPDw/gYtdKxcXFHDt2zN4lioiIAzksoFavXm15hfyfHTlyBLPZDEB2djalpaU0bNjQnuWJiIiDOeQ5qMLCQrZt28b06dMt41JSUgCIj49n7dq1pKSk4OzsjJubG3PmzMFkMjmiVBERcRCHBFT9+vXZuXNnmXHx8fGWzwMHDmTgwIH2LssYXOvXvt4TrmsAT3tb19ZGvU54UkjzpNVWtfVycyH76R41XkOtU5XeQsC2PYaIVEA9SRhNaBVCxyi9JwxIsb6tjXqdSK43G4ZaF1DxyWvWdaMAABANSURBVDtsUkOtU5XeQsC2PYaIVECdxYqIiCEpoERExJAUUCIiYkgKKBERMSQFlIiIGJICSkREDEkBJSIihqSAEhERQ1JAiYiIIakniWtI66fXUnCu2Kq2XiSTbeN6xAaq0n3RdQ2MUYe6UJLLUEBdQwrOFZPy9w5WtVV3QLVUVbsvMkId6kJJLkOn+ERExJAUUCIiYkgKKBERMSQFlIiIGJICSkREDMlhd/GFhobi4eGBk5MTzs7OpKamlpluNpt5/vnn2bx5M25ubsyYMYOgoCAHVSsiIvbm0NvMlyxZgq+vb4XTtmzZwoEDB1i3bh3ffvstTz/9NMuXL7dzhSIi4iiGPcW3YcMG+vTpg8lk4q677qKgoIC8vDxHlyUiInbi0COoESNGYDKZ6N+/P/379y8zLTc3l0aNGlmGGzVqRG5uLv7+/vYus0Ke9aD5wgKr29Zlrd8uoKDI2tYfQNJqq1p6UgjUr25ZNUM9Iog4jMMCKiUlhYCAAPLz8xk2bBgtWrQgODjYUeVUWXJPB/9wGkhBEaREWvl9rJkKQ60LKN7pDbxY7bpqhHpEEHEYh53iCwgIAMDPz4/w8HCys7PLTT98+LBl+PDhw5Z5RESk7nNIQBUWFnL69GnL561bt9KyZcsybUJDQ0lLS8NsNvPNN9/QoEEDw5zeExER23PIKb78/HzGjh0LQElJCREREXTp0oWUlBQA4uPjCQkJYfPmzYSHh+Pu7s4LL7zgiFJFRMRBHBJQzZo1Y9WqVeXGx8fHWz6bTCaeeuope5YlIiIGYtjbzEVE5NqmgBIREUNSQImIiCEpoERExJD0yvdazJOzNLeyV4aL7Qv/9+FXazxZvaKuxLW+9TW41q6HoVufS6agCvvDWl5uLmQ/3aPGl2tLrc8lU2BlTyteJJN95WaG0TppOQU26OGkNu5nW1NA1WLJkX5VnKM+VvfMkFFY1XKsE2qj4DOAAjxI+XuHGl9ufPKOGl+mrRXgYXXvIvEZNi6mhhVQ3yY9p9TG/WxrOsUnIiKGpIASERFDUkCJiIghKaBERMSQFFAiImJICigRETEkBZSIiBiSAkpERAxJD+pKhTzrQXMrewLwrGfjYqxRlR4qqtBLRlV6RPDkbBVqqArbPNzc+u0CCoqsb+9FMtlPe1vZ+oNq1XQlrZ9eS8G5YqvaGqJnBhv9XV4rFFBSoeSetauboSr1UFGFXjKq0iNClXrqqAob9epRUEQVtu1/e3wYkmldYytDvaoKzhVb3VuHIXpmsNHf5bXC7gH1+++/M2XKFPLz8zGZTPTr148hQ4aUabNz507GjBlD06ZNAQgPDychIcHepYqIiAPZPaCcnZ1JSkoiKCiI06dP07dvX+69915uueWWMu3atm3LwoUL7V2eiIgYhN1vkvD39ycoKAgAT09PWrRoQW5urr3LEBERg3PoXXw5OTl899133HnnneWmffPNN0RFRTFy5Ej27dvngOpERMSRHHaTxJkzZxg/fjyPP/44np6eZaYFBQWxceNGPDw82Lx5M2PHjmXdunUOqlRERBzBIUdQFy5cYPz48URGRtK9e/dy0z09PfHw8AAgJCSE4uJijh07Zu8yRUTEgeweUGazmSeeeIIWLVowbNiwCtscOXIEs9kMQHZ2NqWlpTRs2NCeZYqIiIPZ/RTf7t27SU9P59ZbbyU6OhqAxMREfvvtNwDi4+NZu3YtKSkpODs74+bmxpw5czCZTPYuVUREHMjuAdW2bVt++OGHStsMHDiQgQMH2qkiudZ4cpbmSda9htuTs1x8ANdxqlKvF8lk26oOG/UuUvX9UQVW93xRFbbpJaNq+7mQ7Blx1i14xk1w7qR1bd28IelX69ragXqSkGtOcqRfFVo7vkeNqtQbn2HDOmzUu0iV9seaZ4Gu1re3tueLqrBRLxk228/nTlr/PSyJqMKCbU+dxYqIiCEpoERExJAUUCIiYkgKKBERMSQFlIiIGJICSkREDEkBJSIihqSAEhERQ1JAiYiIIaknCRGpPVzrwzu9rWz8pNWLbf12AQVF1rWtSjdOtlKVbpHggyr0fvEBWNvdkpsL2U/3sHK51aOAEpHaI9T60CGj0OqmBUWQEun4bq2sVbXuuqpgzVQYal1AxSfvsE0Nl9ApPhERMSQFlIiIGJICSkREDEkBJSIihqSAEhERQ3JIQG3ZsoUePXoQHh7OokWLyk0vKipi4sSJhIeHExcXR05OjgOqFBERR7J7QJWUlDB9+nTeeustVq9eTWZmJj/++GOZNsuXL8fLy4vPPvuMoUOH8vLLL9u7TBERcTC7B1R2djZ/+ctfaNasGa6urvTu3ZsNGzaUabNx40ZiYmIA6NGjB9u3b8dsNtu7VBERcSC7P6ibm5tLo0aNLMMBAQFkZ2eXa3PjjTcC4OLiQoMGDTh+/Di+vr6VLvvQoUPExsZWuza3Q8cZ/Lkuy0ntVQ9o+bn1bQdb2bY20ndRHR6wrp9VLd0oJTZ28VWtrWHDhixefPll1KmeJHbu3OnoEkREpIbY/XAhICCAw4cPW4Zzc3MJCAgo1+b3338HoLi4mFOnTtGwYUO71ikiIo5l94Bq1aoVBw4c4ODBgxQVFbF69WpCQ0PLtAkNDWXlypUArF27lg4dOmAymexdqoiIOJDJ7IC7DzZv3swLL7xASUkJffv2ZfTo0bz66qvccccdhIWFcf78eR599FG+++47vL29eeWVV2jWrJm9yxQREQdySECJiIhciW5ZExERQ1JAiYiIIdWp28yvxpYtW3j++ecpLS0lLi6OUaNGObqkGhcaGoqHhwdOTk44OzuTmprq6JJqxNSpU9m0aRN+fn5kZmYCcOLECSZNmsShQ4do0qQJc+fOxdvb28GVVl9F2zh//nyWLVtmeT4wMTGRkJAQR5Z5VX7//XemTJlCfn4+JpOJfv36MWTIkDq1Ly+3jXVpX54/f56HH36YoqIiSkpK6NGjB+PHj+fgwYMkJiZy4sQJgoKCmDVrFq6urpUvzCzm4uJic1hYmPnXX381nz9/3hwZGWnet2+fo8uqcV27djXn5+c7uowat2vXLvPevXvNvXv3toybOXOmeeHChWaz2WxeuHChedasWY4qr0ZUtI3z5s0zv/XWWw6sqmbl5uaa9+7dazabzeZTp06Zu3fvbt63b1+d2peX28a6tC9LS0vNp0+fNpvNZnNRUZH5wQcfNH/99dfm8ePHmzMzM81ms9n85JNPmpcuXXrFZekUH9Z1vyTGFRwcXO5/1Bs2bKBPnz4A9OnTh/Xr1zuitBpT0TbWNf7+/gQFBQHg6elJixYtyM3NrVP78nLbWJeYTCY8PDyAi8+xFhcXYzKZ2LFjBz169AAgJibGqt9YBRQVd79U1/5o/jBixAhiY2P56KOPHF2KTeXn5+Pv7w/ADTfcQH5+voMrso2lS5cSGRnJ1KlTOXnypKPLqTE5OTl899133HnnnXV2X166jVC39mVJSQnR0dF06tSJTp060axZM7y8vHBxuXhVqVGjRlb9xiqgriEpKSmsXLmS5ORkli5dyldffeXokuzCZDLVyQe94+Pj+eyzz0hPT8ff358ZM2Y4uqQacebMGcaPH8/jjz+Op6dnmWl1ZV/+eRvr2r50dnYmPT2dzZs3k52dzU8//VSt5SigsK77pbrgj23y8/MjPDy8XCe9dYmfnx95eXkA5OXlXbGj4dro+uuvx9nZGScnJ+Li4tizZ4+jS7pqFy5cYPz48URGRtK9e3eg7u3LiraxLu5LAC8vL9q3b88333xDQUEBxcXFABw+fNiq31gFFNZ1v1TbFRYWcvr0acvnrVu30rJlSwdXZTuhoaGkpaUBkJaWRlhYmIMrqnl//GgDrF+/vtbvT7PZzBNPPEGLFi0YNmyYZXxd2peX28a6tC+PHTtGQUEBAOfOnWPbtm389a9/pX379qxduxaAlStXWvUbq54k/ldF3S/VJQcPHmTs2LHAxfPDERERdWYbExMT2bVrF8ePH8fPz49x48bRrVs3Jk6cyO+//07jxo2ZO3cuPj4+ji612iraxl27dvH9998D0KRJE6ZPn265VlMbZWVl8fDDD3Prrbfi5HTx/86JiYm0bt26zuzLy21jZmZmndmX33//PUlJSZSUlGA2m+nZsycJCQkcPHiQSZMmcfLkSW6//XZefvnlK95mroASERFD0ik+ERExJAWUiIgYkgJKREQMSQElIiKGpIASERFDUkCJiIghKaBErtL8+fNZvHixo8uotpycHDIyMhxdhkg5CiiRa9yhQ4cs75gSMRIFlEg1LFiwgB49ehAfH8/PP/8MwLJly+jbty9RUVGMGzeOs2fPcvr0aUJDQ7lw4QJAmeF3332XBx54gMjISCZNmnTZdZ05c4apU6cSGRlJZGSkpbuYzMxMIiMjiYiI4KWXXrK0b9OmjeXzmjVrSEpKAiApKYnnnnuOAQMGEBYWxpo1awCYPXs2WVlZREdH884779To9yRyNfRGXZEq2rt3L5988glpaWmUlJQQExNDUFAQ4eHh9OvXD4BXXnmFFStWMGjQINq3b8/mzZvp1q0bq1evpnv37tSrV49FixaxceNGXF1dLX2XVeSNN97A09PTchru5MmT5Obm8vLLL5OamoqXlxfDhw9n/fr1dOvWrdLa8/Ly+OCDD/jpp58YPXo0PXv2ZPLkyfzrX/9i4cKFNfclidQAHUGJVFFWVhbdunXD3d0dT09PS6eX+/bt46GHHiIyMpKMjAz27dsHwIMPPsjHH38MQGpqKrGxsQAEBgbyj3/8g/T0dJydnS+7vu3bt/Pwww9bhr29vdmzZw/t2rXD19cXFxcXIiMjrXp9Srdu3XBycuKWW27h6NGj1f4OROxBASVSQ5KSkpg2bRoZGRkkJCRQVFQEwD333MOhQ4fYuXMnJSUl3HrrrQAsWrSIhx56iP/85z88+OCDllcR1KTz58+XGb5S55wiRqKAEqmi4OBg1q9fz7lz5zh9+jSff/45cPFa0Q033MCFCxfK3RXXp08fJk+ebDl6Ki0t5ffff6dDhw784x//4NSpUxQWFla4vk6dOrF06VLL8MmTJ2ndujVfffUVx44do6SkhNWrVxMcHAxcfLfQ/v37KS0tter16B4eHpw5c6Za34WILekalEgVBQUF8cADDxAdHY2vry+tWrUCYMKECcTFxeHr68udd95Z5kc/MjKSuXPnEhERAVx85cmjjz7K6dOnMZvNDB48GC8vrwrXN3r0aKZPn05ERAROTk4kJCTQvXt3Jk+ezJAhQzCbzYSEhFiuP02ePJn/+Z//wdfXlzvuuOOywfeHwMBAnJyciIqKIjY2lqFDh9bAtyRy9fS6DRE7WLNmDRs2bChzt52IVE5HUCI29uyzz7JlyxYWLVrk6FJEahUdQYkYxMcff8y7775bZtzdd9/NU0895aCKRBxLASUiIoaku/hERMSQFFAiImJICigRETEkBZSIiBjS/wfFvJ2EJaslFAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Визуализация\n",
    "sns.histplot(data = users_merged.query('group != \"control_1\"'), hue = 'group', x = 'days_count', bins = 30, element='step')\n",
    "sns.despine()\n",
    "plt.title('Days count depending on group\\n')\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Визуально сильные различия не наблюдаются. Посмотрим статистику"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>W</th>\n",
       "      <th>pval</th>\n",
       "      <th>equal_var</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>levene</th>\n",
       "      <td>0.156987</td>\n",
       "      <td>0.854764</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               W      pval  equal_var\n",
       "levene  0.156987  0.854764       True"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Тест на гомоскедастичность (equal_var = True)\n",
    "pg.homoscedasticity(data = users_merged, dv = 'days_count', group = 'group')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Т.к. у нас 2 группы, сравним их распределения при помощи критерия Манна-Уитни:\n",
    "\n",
    "- $H_0$: нет статистически значимых различий между распределениями количества посещений в тестовой и контрольной группе (p-value > 0.05)\n",
    "- $H_1$: есть статистически значимые различия между распределениями количества посещений в тестовой и контрольной группе"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MannwhitneyuResult(statistic=9358.0, pvalue=0.8940167190188826)"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mannwhitneyu(users_merged.query('group == \"test\"').days_count, users_merged.query('group == \"control_2\"').days_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Наше p-value > 0.05, мы не можем отвергнуть нулевую гипотезу, т.е. статистически значимых различий между распределениями количества посещений в тестовой и контрольной группе нет, пользователи не стали чаще или реже заходить в приложение в связи с изменением стоимости премиум-подписки."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Рассчитаем метрики экономики для тестовой и контрольной группы и создадим сводную таблицу:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>users</th>\n",
       "      <th>premium_users</th>\n",
       "      <th>CR_pr</th>\n",
       "      <th>ARPPU</th>\n",
       "      <th>ARPPU_pr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>test</th>\n",
       "      <td>127</td>\n",
       "      <td>65</td>\n",
       "      <td>0.511811</td>\n",
       "      <td>12868.259843</td>\n",
       "      <td>25142.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      users  premium_users     CR_pr         ARPPU  ARPPU_pr\n",
       "test    127             65  0.511811  12868.259843   25142.6"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics = []\n",
    "\n",
    "users = users_test.uid.nunique()\n",
    "premium_users = transactions_test.query('revenue > 0 and product_type == \"premium_no_trial\"').uid.nunique()\n",
    "revenue = transactions_test.revenue.sum()\n",
    "CR_pr = (premium_users/users)\n",
    "ARPPU = (revenue/users)\n",
    "ARPPU_pr = (revenue/premium_users)\n",
    "\n",
    "metrics.append(pd.DataFrame(index = ['test'], data = {'users':users, 'premium_users':premium_users, \n",
    "                                                      'CR_pr':CR_pr, 'ARPPU':ARPPU, 'ARPPU_pr':ARPPU_pr}))\n",
    "metrics = pd.concat(metrics)\n",
    "metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>users</th>\n",
       "      <th>premium_users</th>\n",
       "      <th>CR_pr</th>\n",
       "      <th>ARPPU</th>\n",
       "      <th>ARPPU_pr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>control_2</th>\n",
       "      <td>146</td>\n",
       "      <td>95</td>\n",
       "      <td>0.650685</td>\n",
       "      <td>9038.650685</td>\n",
       "      <td>13890.978947</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           users  premium_users     CR_pr        ARPPU      ARPPU_pr\n",
       "control_2    146             95  0.650685  9038.650685  13890.978947"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics_1 = []\n",
    "\n",
    "users = users_control_2.uid.nunique()\n",
    "premium_users = transactions_control_2.query('revenue > 0 and product_type == \"premium_no_trial\"').uid.nunique()\n",
    "revenue = transactions_control_2.revenue.sum()\n",
    "CR_pr = (premium_users/users)\n",
    "ARPPU = (revenue/users)\n",
    "ARPPU_pr = (revenue/premium_users)\n",
    "\n",
    "metrics_1.append(pd.DataFrame(index = ['control_2'], data = {'users':users, 'premium_users':premium_users,\n",
    "                                                      'CR_pr':CR_pr, 'ARPPU':ARPPU, 'ARPPU_pr':ARPPU_pr}))\n",
    "\n",
    "metrics_1 = pd.concat(metrics_1)\n",
    "metrics_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>users</th>\n",
       "      <th>premium_users</th>\n",
       "      <th>CR_pr</th>\n",
       "      <th>ARPPU</th>\n",
       "      <th>ARPPU_pr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>test</th>\n",
       "      <td>127</td>\n",
       "      <td>65</td>\n",
       "      <td>0.5118</td>\n",
       "      <td>12868.26</td>\n",
       "      <td>25142.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>control_2</th>\n",
       "      <td>146</td>\n",
       "      <td>95</td>\n",
       "      <td>0.6507</td>\n",
       "      <td>9038.65</td>\n",
       "      <td>13890.98</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           users  premium_users   CR_pr     ARPPU  ARPPU_pr\n",
       "test         127             65  0.5118  12868.26  25142.60\n",
       "control_2    146             95  0.6507   9038.65  13890.98"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics_merged = pd.concat([metrics, metrics_1])\n",
    "\n",
    "# Округлим значения\n",
    "metrics_merged[['CR_pr']] = metrics_merged[['CR_pr']].round(4)\n",
    "metrics_merged[['ARPPU', 'ARPPU_pr']] = metrics_merged[['ARPPU', 'ARPPU_pr']].round(2)\n",
    "metrics_merged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "#T_b4055654_972d_11ee_bdbb_96000040dfb6row0_col2,#T_b4055654_972d_11ee_bdbb_96000040dfb6row1_col3,#T_b4055654_972d_11ee_bdbb_96000040dfb6row1_col4{\n",
       "            background-color:  coral;\n",
       "        }#T_b4055654_972d_11ee_bdbb_96000040dfb6row0_col3,#T_b4055654_972d_11ee_bdbb_96000040dfb6row0_col4,#T_b4055654_972d_11ee_bdbb_96000040dfb6row1_col2{\n",
       "            background-color:  yellowgreen;\n",
       "        }</style><table id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >users</th>        <th class=\"col_heading level0 col1\" >premium_users</th>        <th class=\"col_heading level0 col2\" >CR_pr</th>        <th class=\"col_heading level0 col3\" >ARPPU</th>        <th class=\"col_heading level0 col4\" >ARPPU_pr</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6level0_row0\" class=\"row_heading level0 row0\" >test</th>\n",
       "                        <td id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6row0_col0\" class=\"data row0 col0\" >127</td>\n",
       "                        <td id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6row0_col1\" class=\"data row0 col1\" >65</td>\n",
       "                        <td id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6row0_col2\" class=\"data row0 col2\" >0.5118</td>\n",
       "                        <td id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6row0_col3\" class=\"data row0 col3\" >12868.26</td>\n",
       "                        <td id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6row0_col4\" class=\"data row0 col4\" >25142.60</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6level0_row1\" class=\"row_heading level0 row1\" >control_2</th>\n",
       "                        <td id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6row1_col0\" class=\"data row1 col0\" >146</td>\n",
       "                        <td id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6row1_col1\" class=\"data row1 col1\" >95</td>\n",
       "                        <td id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6row1_col2\" class=\"data row1 col2\" >0.6507</td>\n",
       "                        <td id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6row1_col3\" class=\"data row1 col3\" >9038.65</td>\n",
       "                        <td id=\"T_b4055654_972d_11ee_bdbb_96000040dfb6row1_col4\" class=\"data row1 col4\" >13890.98</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fa7340ae0b8>"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Выделим максимальные и минимальные значения метрик цветом\n",
    "(metrics_merged\n",
    " .style\n",
    " .format('{:.4f}', subset = ['CR_pr'])\n",
    " .format('{:.2f}', subset = ['ARPPU', 'ARPPU_pr'])\n",
    " .highlight_max(color='yellowgreen', subset = ['CR_pr', 'ARPPU', 'ARPPU_pr'])\n",
    " .highlight_min(color='coral', subset = ['CR_pr', 'ARPPU', 'ARPPU_pr'])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "По результатам вычислений метрик экономики можно сделать следующие выводы:\n",
    "\n",
    "В тестовой группе по сравнению с контрольной уменьшилась метрика CR_pr (конверсия в покупку премиум подписки) на 21%. Т.е. в ходе эксперимента покупать премиум подписку стали реже, видимо стоимость подписки изменилась в большую сторону.\n",
    "\n",
    "В тестовой группе по сравнению с контрольными увеличились метрики: \n",
    "- средний доход на платящего пользователя (+42%)\n",
    "- средний доход на платящего премиум пользователя (+82%)\n",
    "\n",
    "Т.е. хоть количество платящих пользователей уменьшилось, но их \"качество\" - доход от них - значительно увеличился. Пользователи стали покупать на большую сумму (потдверждение того, что стоимость премиум подписки увеличилась).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Выручка выросла, но она включает выручку от всех продуктов (пробная премиум-подписка, премиум-подписка, другой тип), тогда как цена менялась только на один (премиум-подписка). \n",
    "\n",
    "Посмотрим, как этот эффект роста среднего распределён по видам продуктов, за счёт какого продукта в итоге стали больше зарабатывать."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ARPPU_product_type = transactions_merged.groupby(['group', 'product_type'], as_index = False).agg({'revenue' : 'sum', 'uid' : 'nunique'})\n",
    "ARPPU_product_type['ARPPU'] = ARPPU_product_type.revenue / ARPPU_product_type.uid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbkAAAEYCAYAAADBFIhjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deVgVZf8/8PdhB3EBExXUIhVUFD2IgbuyuLCDy6OPkeGWmKmZmktZWmm5VZK5JJWaWikgKFpuuWWaKC64PIGpoCAUq4Ls9+8Pv86PI9tBDx4c3q/r8ro4s9zzmTlnfJ97Zs6MQgghQEREJEM62i6AiIiotjDkiIhIthhyREQkWww5IiKSLYYcERHJFkOOiIhkiyFHGhEeHo7Ro0dLr5VKJZKSkjTS9rp167BgwQIAwO3bt2Fra4vi4mKNtJ2cnAylUomSkhKNtFeWra0tbt26pfF2NS0kJASzZs3SdhnP1OOf17oiMDAQO3bsAABERUVh3LhxWq7o+ceQq4MCAwPRo0cPFBYWqgyfO3cuOnfuDKVSiVdeeQVBQUG4fv26ND48PBwdO3aEUqmEg4MDfH198dtvvwEATp8+jQ4dOkCpVEKpVGLw4MEICwuTxvXr16/COh7tcDUVGxuL1q1bVzlNZct93OTJk/HJJ588UR2Pc3FxwcmTJ6XXlpaWiI2Nha6urkbar8+qC/W6Gix1lY+PD7799lttl/HcY8jVMbdv30ZMTAwUCgUOHTpUbvz48eMRGxuLY8eOoXnz5lIP55Fu3bohNjYWMTExGD58OGbMmIHs7GwAgIWFBWJjY3Hu3DnMnj0b77//PhISEp7Jej0pTfXY6gshBEpLS7VdBlGFtLE/M+TqmF27dqFr167w9/fHrl27Kp3OyMgIQ4cOxbVr1yocr6Ojg2HDhiE/Px+JiYkq4xQKBdzc3NCoUaMnDrnMzExMnjwZDg4OGD58eLlllP1Wf/ToUXh4eECpVKJv374IDQ1FXl4eJk6ciLS0NKl3mZqaipCQEEybNg2zZs2Cg4MDIiIiKjycFhYWhj59+qBPnz4IDQ2Vhs+dOxeff/659Lpsb3H27NlITk7G5MmToVQq8c0335Q7/JmamorJkyfjlVdegbu7O37++WeprZCQEEyfPh1z5syBUqmEp6cnLl26VOV2Onr0KFxdXeHk5ITPPvsMpaWlKCwsxCuvvIL//e9/0nTp6eno2rUrMjIyyrURHh6OUaNGYfHixejevTuGDBmCP/74QxofGBiIzz//HKNGjULXrl2RlJSEc+fOYdiwYejevTuGDRuGc+fOSdMnJSXh1VdfhVKpRFBQEDIzMyvcXo+U7f2WlJRg3bp1cHNzg1KpREBAAFJSUjBmzBgAgK+vL5RKJfbu3avSxvXr1/HBBx/g/PnzUCqVcHR0xMWLF9GrVy+VQ8X79++Hj4+PtL2nTZuGGTNmQKlUwt/fX+XznpqairfeegvOzs5wcXHB5s2bK30fqvu8Xr9+HUFBQXjllVcwePBglfrnzp2LhQsXIigoCEqlEq+++iru3Lmj9ryLFi3CpEmToFQqMWLECJVl//777xgyZAi6d++OxYsXo+wNqB7v+dra2mL79u0YNGgQHB0dsWjRImn6kpISfPrpp3BycoKLiwt++OGHKg/rX79+HYGBgXB0dISnp6fKF+rqan7crl27MHDgQDg5OWHNmjUqn5eK9ueq9rGq9l/g4Wdx/fr18PDwQI8ePTBv3jwUFBRUWhvAkKtzIiMj4e3tDW9vb5w4cQL//vtvhdPl5eVhz549aNOmTYXji4uLsWPHDpiYmOCll15SGVdaWooDBw7g3r17sLGxeaI6Fy9eDENDQ5w4cQJLliyRDn1WZMGCBVi8eDFiY2OxZ88eODs7w8TEBN98843Uu4yNjUXz5s0BAIcOHcKQIUMQExMDb2/vCts8ffo09u/fj9DQUHzzzTcqhyArs3z5clhaWmLdunWIjY3FxIkTy00zc+ZMtGjRAsePH8fq1auxatUqlUA5fPgwPD09ERMTAxcXF3z00UdVLvPAgQMICwtDREQEDh8+jLCwMBgYGMDDwwNRUVHSdHv27EHPnj1hbm5eYTsXL15EmzZtcOrUKUybNg1Tp05FVlaWND4yMhIfffQRzp07hwYNGuCNN95AYGAgTp8+jaCgILzxxhtSmM2aNQt2dnY4ffo0pkyZgoiIiGq33SPfffcdoqOjsWHDBpw7dw5LliyBkZERtm7dKtURGxsLDw8Plfnatm2LRYsWqRxpsLe3R5MmTXDixAmV9fDz85NeP/os/Pnnn/Dy8sKUKVNQVFSE0tJSBAcHw9bWFseOHcOmTZuwadMmHD9+vMK6q/q85uXlYdy4cfDy8sLJkyfx+eefY9GiRSpfAHfv3o0pU6ZIh/0ffelSZ969e/di6tSpOHPmDNq0aSP9J56RkYGpU6dixowZOHXqFNq0aaPyZaQiR44cwc6dOxEVFYV9+/ZJ6/vzzz/j2LFjiIyMREREBA4ePFhpG0VFRZg8eTJ69+6NkydP4r333sOsWbPw999/V1vz4xISErBo0SIsX74cx48fx/3795GamqoyzeP7c3X7WHV2796N0NBQHDhwADdu3MDXX39d5fQMuTokJiYGycnJGDp0KDp37ozWrVtjz549KtN8++23cHR0hIODA86ePYtly5apjL9w4QIcHR3Ru3dvREdHY82aNWjYsCEAIC0tDY6OjnB2dsZXX32FZcuW4eWXX65xnSUlJdi/fz+mTZsGExMT2NjYwN/fv9Lp9fT0kJCQgPv376Nx48aws7Orsv1u3brBzc0NOjo6MDIyqnCaN998EyYmJrC1tUVAQEC57fQkUlJScO7cOcyaNQuGhobo2LEjRowYgcjISGma7t27o3///tDV1YWvr2+lPelHJk6ciCZNmsDS0hKvvfaaVKe/vz+io6Olb+KRkZFSD6Yi5ubmGDt2LPT19eHh4QFra2scOXJEGu/v74/27dtDT08PJ06cwIsvvgg/Pz/o6enBy8sLL7/8Mn777TckJyfj0qVLmD59OgwMDNCjRw+4uLiovY127NiB6dOn4+WXX4ZCoUCHDh1gZmam9vyP8/Pzk8I+KysLJ06cgJeXlzTezs4OQ4YMgb6+PoKCglBYWIgLFy7g0qVLUkgYGBigdevWGDlyZLkeJFD95/XIkSOwsrLCsGHDoKenh06dOmHw4MH45ZdfpGkGDBiAHj16wMDAAG+//TbOnz+PlJQUteZ1c3ODvb099PT04OPjg6tXrwIAjh07hvbt20vrN3bsWLzwwgtVbq+JEyeiUaNGsLS0hJOTk/T527dvH1577TW0aNECjRs3xqRJkypt48KFC8jLy8OkSZNgYGCAnj17YuDAgYiOjq625sf98ssvGDhwIBwdHWFgYIBp06ZBoVCoTFN2f87MzKx2H6vOmDFj0LJlSzRp0gTBwcEqdVdET+2Wqdbt2rULvXv3lr7Ne3l5ISIiAq+//ro0zbhx4/D2228jOTkZEyZMwI0bN9ChQwdpfNeuXbF9+/YK27ewsMCxY8fKDdfV1a3wsEZRURH09Mp/RDIyMlBcXIyWLVtKwywtLStdr9WrV2Pt2rVYuXIlbG1t8c4770CpVFY6fYsWLSod90jZZVtZWeGvv/6qdp7qpKWloXHjxjA1NZWGWVpaIi4uTnpd9j8hIyMjFBQUoLi4uMLtVFGdaWlpAB6+T0ZGRjh9+jSaNWuGxMREuLq6Vlpb8+bNVf7zsLS0lNp6fDlpaWnl3g9LS0ukpqYiLS0NjRo1gomJicq4lJSUSpdd1t27dys9evAkfH19MXToUOTl5WHfvn1wdHSEhYWFNL7sZ0FHRwfNmzeX1vvRl7ZHSkpKVF4/Ut3n9c6dO7h48WK5tsp+6ShbR4MGDdC4cWOkpaWpNe/jn5m8vDyp/rLtKhQKlRor0qxZM+lvY2Nj5ObmSm2VnbeqfejRcnV0/n8f59Hno7qaK2urbE1NmjRRmabseHX2seo8/j6W3Q8qwpCrI/Lz87Fv3z6Ulpaid+/eAIDCwkLk5OTg2rVrKkEGPHxzFyxYgHfffRcDBw6stMejDktLS2RmZiI3NxcNGjQA8PAChuTk5ArDy9zcHHp6ekhJSUHbtm0BoMr/JO3t7bF27VoUFRVh69atmDFjBo4ePVruG98jlQ0vq+yyk5OTpf8YjY2NkZ+fL01X2eHeilhYWCA7Oxv379+XdsKUlBTpMOqTSElJQfv27cvVCTzsfUVFRaFZs2YYPHgwDA0NK20nNTUVQghp26SkpKj0wMpuMwsLCyQnJ5ero2/fvmjWrBlycnKQl5cnBV1ycrI0/+Pbr6SkROU8YYsWLZCYmPhEh7krel+bN28OpVKJ/fv3IzIystzVl3fv3pX+Li0tRWpqKiwsLKCrq4tWrVph//791S63us9ry5Yt0aNHD3z33XeVtlG2jtzcXGRnZ8PCwkKteSvTrFkzlXaFEGp/2aiurbJ/P87CwgJ3795FaWmpFHQpKSnlTmuow8LCAjdu3JBe5+fnqxxGB8p/Nqvax9TZf8tuo8f3qYrwcGUdcfDgQejq6iI6Ohq7du3Crl27sHfvXjg6OlZ6AUrv3r1hYWGBn3766amWbWlpia5du2LFihXIzc1FYWEhNm7cCD09PXTr1q3c9Lq6unB3d8dXX32FBw8eICEhodLzOoWFhYiKisK9e/egr6+PBg0aSDtW06ZNkZWVhXv37tW45q+//hoPHjxAfHw8wsPDpXNAHTt2xNGjR5GVlYV//vkHmzZtUpnvhRdeqPT3ey1btoRSqcSqVatQUFCAa9euYefOnVUeRqxOaGgosrOzkZKSgs2bN6ucq/Lx8cHBgwcRFRWlch6qIhkZGdi8eTOKioqwb98+XL9+Hf37969w2v79++PmzZvYvXs3iouLsXfvXiQkJGDAgAGwsrJC586dERISgsLCQsTExEg/MwEAa2trFBQU4MiRIygqKsLatWtVfsoyYsQIfPnll7h58yaEELh27Zp0rq+qbQs8fL9TU1PL/TTG19cXoaGh+OuvvzBo0CCVcZcvX8b+/ftRXFyMTZs2wcDAAF27doW9vT0aNGiADRs2ID8/HyUlJfjrr79w8eLFcsut7vM6YMAA3Lx5E7t27UJRURGKiopw8eJFlZ/nHD16FDExMSgsLMSXX36Jrl27omXLlmrNW5n+/fsjPj5eWr/NmzfX6EtZWUOHDsXmzZuRmpqKnJwcfPPNN5VOa29vDyMjI2zcuBFFRUU4ffo0Dh8+XO48qjoGDx6Mw4cP49y5cygsLERISAiqenpbdftYdfsvAGzbtg13795FVlYW1q1bV23dDLk6IiIiAgEBAbC0tESzZs2kf2PGjJH+s6rIhAkTsHHjxnL/cdTU559/jvT0dAwaNAh9+/bFqVOnsGHDhkp7FwsXLkReXh569+6NuXPnIiAgoNK2IyMj4eLiAgcHB/z4449Yvnw5gIcXI3h6esLNzQ2Ojo7lTlhX5dGVWa+//jrGjRuHPn36AHj4H2aHDh3g4uKCcePGldsBJk2ahLVr18LR0VHlqsxHVq1ahTt37qBv376YOnUq3nrrLfTq1Uvtuh7n6uqKgIAA+Pn5YcCAARg+fLg0rmXLlujUqRMUCkWFh9nKsre3x61bt+Ds7IwvvvgCq1evrvRcmJmZGdatW4fvvvsOTk5O2LhxI9atWycdBl+5ciUuXLggXQ1XNmAbNmyIDz74AO+99x769esHY2NjlcNNQUFBGDp0KMaNGwcHBwcsWLBAurpt6tSpmDt3LhwdHSs8N+bs7Ix27dqhT58+cHJykoa7u7vjzp07cHd3h7Gxcbntt3fvXvTo0QORkZEICQmBvr4+dHV1sW7dOly7dg2urq5wdnbGe++9h/v371e4Tar6vJqamiI0NBR79+5F37590adPH6xYsUJln/Ly8sKaNWvg5OSEy5cvS59hdeatjLm5Ob788kusXLkSTk5OuHXrFhwcHKqdryIjR45E79694ePjAz8/P/Tv3x96enoV/v7TwMAA69atw7Fjx+Ds7IxFixZh2bJlUi+3Jtq3b4/3338fM2fORN++fWFiYgJzc3MYGBhUOk9V+1h1+y/w8L0YN24c3Nzc0KZNGwQHB1ddpCAirZk7d65YtWpVldOEhYWJUaNGPaOKtMPV1VX8/vvvKsNWr14t3nnnHS1V9P+9++671b5Hdc2RI0fEgAEDnvly79+/Lzp27CgSExNrpf2BAweW+5xUhz05Ii25ffs2Dhw4oNK7q49+/fVXKBQKODs7a7uU51Z+fj6OHj2K4uJipKamYs2aNXBzc3smyz58+DAePHiAvLw8fPbZZ7CxsUGrVq2eybLVwQtPiLTgiy++wKZNmzBp0qRqb38mZ4GBgUhISMCyZctUrvajmhFCYPXq1ZgxYwaMjIwwYMAATJ8+/Zks+9ChQ5gzZw6EEOjcuTNWrVql1sVjz4pCiCrOEhIRET3H+NWJiIhkiyFHRESyxZAjIiLZYsgREZFsMeSIiEi2GHJERCRbDDkiIpIthhwREckWQ46IiGSLIUdERLLFkCMiItliyBERkWwx5IiISLYYckREJFsMOSIiki2GHBERyRZDTmbGjx+v7RKIiOoMhpzMZGZmarsEIqI6gyFHRESyxZAjIiLZYsgREZFsMeSIiEi2GHJERCRbDDmZEULbFRBRTRUUlWi7BNnS03YBpFkKBdB99mZtl0FENXB2+WvaLkG22JMjIiLZYsgREZFsMeSIiEi2GHJERCRbDDkiIpIthhwREckWQ46IiGSLIUdERLLFkCMiItliyBERkWwx5IiISLYYckREJFsMOSIiki2GHBERyRZDjoiIZIshR0REssWQIyIi2WLIERGRbDHkiIhIthhyREQkWww5IiKSLYYcERHJFkOOiIhkq16GXE5ODrZu3Sq9Pn36NN544w2NL+fgwYNISEjQeLtERKSeehty27dv11h7xcXFFQ5nyBERaZdCCCG0XURt++677xAWFgYAGD58OC5cuIBDhw7B2toavXr1woABA/DVV1/BzMwMf/31F+zs7LBixQooFArExcXh008/RV5eHszMzLB06VJYWFggMDAQHTp0wNmzZ+Hl5YVx48apLPPcuXOYPHkyTE1N0bBhQ4SEhGD69OmIiIgAANy8eRNvv/02IiIi4OLigiFDhuD48eMwNDTEypUr8eKLLyIjIwMffPABkpOTAQDz589H9+7dq1zXAH9/hP/fMoiIngVRXACFnqG2y6iQnrYLqG1xcXEIDw/Hzz//DCEERo4cieXLlyM+Ph6RkZEAHh6uvHLlCqKjo2FhYYHRo0fj7Nmz6Nq1Kz7++GN8/fXXMDc3x969e/H5559j6dKlAICioiKEh4dXuFwHBwe4uLhgwIABGDJkCADA1NQUV69eRceOHREeHo6AgABp+oYNG2L37t3YtWsXlixZgvXr1+OTTz7B2LFj4ejoiOTkZIwfPx779u2reoUVCiQu7qKBLUdEpJ42Cy9pu4RKyT7kzp49Czc3N5iYmAAA3N3dERMTU246e3t7tGjRAgDQoUMH3LlzB40aNcJff/2FoKAgAEBpaSmaNWsmzePh4VGjWkaMGIGwsDDMmzcPe/fuxY4dO6RxXl5eAABPT08pRE+ePKlyuPP+/fvIzc1FgwYNarRcIqL6SvYhpy4DAwPpb11dXZSUlEAIgfbt2+Onn36qcB5jY+MaLWPw4MFYs2YNnJ2dYWdnBzMzsyqnLy0txc8//wxDw7p5GICIqK6T/YUnjo6OOHjwIB48eIC8vDwcPHgQDg4OyM3NrXZea2trZGRkIDY2FsDDw5Px8fFqL7tBgwYqyzE0NESfPn3w4YcfqhyqBCAdhty7dy+USiUAoE+fPtiyZYs0zdWrV9VeNhER1YOQs7OzQ0BAAEaMGIGRI0di+PDh6Ny5MxwcHODl5YXPPvus0nkNDAywevVqrFixAj4+PvDz85MCTx0eHh4IDQ2Fn58fEhMTAQDe3t7Q0dFBnz59VKbNzs6Gt7c3Nm/ejHnz5gEAFixYgLi4OHh7e8PDw0OjV4QSEdUH9eLqyrokNDQU9+7dw4wZM6RhLi4u2LlzJ8zNzZ+6/YCAAHzRTf3eJhHR0+KFJwQAePPNN5GYmIhNmzZpuxQionqBIacBa9euxS+//KIybMiQIQgODlYZtmbNmgrnP3z4cK3VRkRUnzHkNCA4OLhcoBERkfbJ/sITIiKqvxhyREQkWww5IiKSLYYcERHJFkOOiIhkiyFHRESyxZAjIiLZYsgREZFsMeSIiEi2GHJERCRbDDkiIpIthhwREckWQ46IiGSLIUdERLLFkCMiItliyBERkWwx5IiISLYYckREJFt62i6ANE2gzcJL2i6CiOoRUVwAhZ6htsuoEHtysqPQdgFEVM/U1YADGHJERCRjDDkiIpIthhwREckWQ46IiGSLIUdERLLFkCMiItliyBERkWwx5IiISLYYckREJFtqh9y///6L+fPnY8KECQCAhIQE7Nixo9YKIyIielpqh9zcuXPRp08fpKWlAQBeeuklbN68udYKIyIielpqh1xmZiY8PDygo/NwFj09PelvqjuE0HYF9EhBUYm2SyCq99R+CoGJiQkyMzOhUDy8AfD58+fRsGHDWiuMnoxCAXSfzR52XXB2+WvaLoGo3lM75ObOnYvg4GAkJiZi1KhRyMzMxJdfflmbtRERET0VtUPOzs4OP/zwA27cuAEhBKytraGvr1+btRERET0VtUOuoKAA27Ztw9mzZ6FQKNC9e3eMHj0ahoZ19zlCRERUv6l95cicOXMQHx+PV199FWPGjEFCQgJmz55dm7URERE9FbV7cvHx8di7d6/02tnZGR4eHrVSFBERkSao3ZPr1KkTzp8/L72+cOECOnfuXCtFERERaYLaPbnLly9j1KhRsLS0BAAkJyfD2toa3t7eAIDdu3fXToVERERPSO2Q27hxY23WQUREpHFqh9yWLVswfPhwtGvXrjbrISIi0hi1Q65t27Z47733UFJSgoCAAHh5efGOJ0REVKepHXIjRozAiBEj8PfffyM8PBw+Pj5wcHDAiBEj4OzsXJs1EhERPZEa3WG5pKQEf//9N/7++2+YmZnB1tYW33//Pd5+++3aqo+IiOiJqd2TW7JkCY4cOQJnZ2dMnjwZ9vb20rjBgwfXSnFERERPQ+2Qs7W1xYwZM2BiYlJu3M6dOzVaFBERkSaofbgyKiqqXMCNHTsWAHgBChER1UnV9uQKCgrw4MEDZGZmIjs7G+L/nsp5//59pKam1nqBRERET6rakPvxxx+xadMmpKWlISAgQAo5U1NTvPrqq7VeIBER0ZOqNuTGjh2LsWPHYsuWLQgMDKx0ut9//x29e/fWaHFERERPQ+1zclUFHACsWLHiqYshIiLSpBr9Tq4qjw5jEhER1RUaCzmFQqGppoiIiDRCYyFHRERU16gdcoWFhVUOs7Ky0kxFREREGqJ2yP3nP/+pcthXX32lmYpqKDU1FdOmTdPKsmti3bp1VY6fOHEicnJyqpzGxcUFGRkZmiyLiEjWqv0JwT///IPU1FTk5+fjypUrKj8Gf/DgwVMXUFJSAl1d3Seev3nz5li9evVT11Hb1q9fj8mTJ5cbLoSAEALffPONFqoiIpK3akPuxIkTCA8Px927d/Hpp5+q/Bh85syZVc57+/ZtTJgwAXZ2drhy5Qrat2+Pzz77DJ6enhg6dChOnjyJCRMmoHHjxggJCUFhYSFat26NpUuXokGDBnBxcYGnpyeOHTsGXV1dfPTRR1i1ahVu3bqF8ePHY/To0bh9+zYmT56MPXv2IDw8HHFxcVi4cCEA4I033sC4cePg5OQEpVKJUaNG4dixY2jWrBlmzpyJ5cuXIzk5GfPnz4erq2uF6xAeHo7Dhw/jwYMHSEpKgpubG+bMmQMA2LNnD9avXw8hBPr374/Zs2dX2MaKFSuQn58PX19ftGvXDm+//TbGjx+Prl274vLly9iwYQMCAwOxc+dOmJubY8qUKbh79y4KCgrw2muvVdiLJiIiNQg1/fLLL+pOKklKShI2NjYiJiZGCCHE3LlzxcaNG8XAgQPFhg0bhBBCpKeni//+978iNzdXCCHE+vXrRUhIiBBCiIEDB4qtW7cKIYT45JNPhJeXl7h3755IT08XPXv2lJbh6ekphBAiLCxMLFq0SFr+pEmTxKlTp4QQQtjY2IgjR44IIYSYMmWKCAoKEoWFheLq1avCx8en0nUICwsTLi4uIicnR+Tn54sBAwaI5ORkcffuXdG/f3+Rnp4uioqKRGBgoDhw4ECl7XTr1k1lu9ja2orY2Fhp2MCBA0V6eroQQojMzEwhhBAPHjwQnp6eIiMjo9w0lfH386tyPJEmlBbla7sEIrWo/RSCy5cvo2fPnmjUqBEAIDs7G99++221z5Jr2bIlunfvDgDw8fHBli1bAAAeHh4AgAsXLiAhIQGjR48GABQVFaFbt27S/I96WDY2NsjLy4OpqSkAwMDAoNpzWGXp6+ujX79+UlsGBgbQ19eHjY0N7ty5U+W8PXv2lG5C3bZtW9y5cwdZWVl45ZVXYG5uDgDw9vbGmTNn4ObmplY9lpaWKutZ1pYtW3DgwAEAQEpKCm7dugUzMzO12oVCgcTFXdSblugJtVl4SdslEKlF7ZA7duyYyuHJxo0b49ixY9WG3OO/n3v02tjYGMDDc1K9e/fGqlWrKpxfX18fAKCjowMDAwNpuI6ODoqLi1Wm1dXVRWlpqfS6oKBApZ1Hyy7blo6ODkpKSqpch7LL1dXVrXZ6dVT0yCIAOH36NE6ePImffvoJxsbGCAwMVFkPIiJSn9pXV5aUlKj8ZCA/P7/CnxU8Ljk5GbGxsQAensN61Kt7pFu3bjh37hxu3boFAMjLy8ONGzfULUuFlZUVrl27htLSUqSkpODixYtP1I467O3tcebMGWRkZKCkpATR0dHo0aNHpdPr6emhqKio2nbv3buHxo0bw9jYGNevX8f58+c1WTYRUb2idk/O29sbY8eORUBAAICHF2T4+flVO5+1tTW2bt2K+fPno127dhg9ejR++BNEdl0AABjESURBVOEHaby5uTmWLl2KmTNnSqE5Y8YMWFtb13Rd0L17d1hZWcHDwwNt27aFnZ1djdtQl4WFBd555x2MHTtWuvCkqkOVI0eOhI+PDzp16lRl77dfv3748ccfMXToUFhbW1d6SJOIiKqnEEL9m04ePXoUp06dAgD06tULffv2rXL6slc+0rMREBCAL7rFa7sMkjmek6Pnhdo9OQDo378/+vfvX1u1EBERaZTaIadUKqULN4qKilBcXAxjY2OcO3eu0nlatWr13PTijh8/Xu5xQa1atcKaNWtq1M6IESPKnatctmwZbG1tn7pGIiKqGbVD7tHFI8DDKyIPHTokq4si+vbtW+3hV3Xs2LFDA9UQEZEmPNFTCBQKBdzc3HDixAlN10NERKQxavfk9u/fL/1dWlqKuLg4GBoa1kpRREREmqB2yP3222/S37q6urCyssLXX39dK0URERFpgtoht3Tp0tqsg4iISOOqDbmPPvqo3K25ynrvvfc0WhAREZGmVHvhSefOnWFnZ4eCggJcvnwZL774Il588UVcvXpVrdt6ERERaUu1PTl/f38AwPbt27Ft2zbo6T2cZdSoURgzZkztVkdERPQU1P4JQXZ2Nu7fvy+9zsvLQ3Z2dq0URUREpAlqX3gyadIk+Pv7w8nJCUIInDlzBm+99VZt1kZERPRU1A65YcOGoV+/frhw4QIUCgVmzZqFZs2a1WZtRERET6VGN2i+ePEizp49C+DhXU9cXFxqpSgiIiJNUPuc3IoVK7B582a0bdsWbdu2xZYtWyp9mjcREVFdoHZP7ujRo4iMjISOzsNc9Pf3h5+fH2bOnFlrxRERET2NGt2gOScnR/r73r17Gi+GiIhIk9TuyU2ePLnc1ZWzZs2qzdqIiIieilohV1paCoVCgZ9++gmXLj187D2vriQiorpOrZDT0dHBxo0b4eHhAVdX19quiYiISCPUPifXq1cvhIaGIiUlBVlZWdI/IiKiukohhBDqTOji4lLh0wgOHTqk8aLoyQUE+CM8PELbZZDMieICKPT40GSq+9S+8GTv3r3Ytm0bzp49C4VCAUdHR4waNao2a6MnUvljkYg0hQFHzwu1D1e+++67uH79OgIDA/Hqq68iISEB7777bm3WRkRE9FTU7snFx8dj79690mtnZ2d4eHjUSlFERESaoHZPrlOnTjh//rz0+sKFC+jcuXOtFEVERKQJavfkLl++jFGjRsHS0hIAkJycDGtra3h7ewMAdu/eXTsVEhERPSG1Q27jxo21WQcREZHGqR1yVlZWtVkHERGRxtXoBs1ERETPE4YcERHJFkOOiIhkiyFHRESyxZAjIiLZYsjJjHq32yZST0FRibZLIHoqav+EgJ4PCgXQffZmbZdBMnF2+WvaLoHoqbAnR0REssWQIyIi2WLIERGRbDHkiIhIthhyREQkWww5IiKSLf6EgIioGkVFRbh9+zby8/O1Xcpzy8jICK1atYK+vv4zXS5DjoioGrdv30bDhg3x0ksvQaFQaLuc544QAunp6bh9+zasra2f6bJ5uJKIqBr5+flo2rQpA+4JKRQKNG3aVCs9YYYcEZEaGHBPR1vbjyFHRESyxZAjIiLZYsgRET1niouLtV3Cc4NXVxIR1TFr1qxBVFQUzM3N0bJlS9jZ2eHIkSPo0KEDzp49Cy8vL3Ts2BGfffYZSkpK0LlzZyxatAgGBgZwcXHBzp07YW5ujkuXLmHZsmXYsmULQkJCkJiYiMTERGRmZmLChAkYOXKktle11jHkiIjqkIsXL2L//v2IiopCUVERAgICYGdnB+Dh7/XCw8NRUFCAQYMG4fvvv4e1tTXmzJmDbdu24fXXX6+y7f/973/4+eefkZeXB39/f/Tv3x/Nmzd/BmulPTxcSURUh5w7dw6urq4wNDSEqakpBg4cKI3z8PAAANy4cQOtWrWSfnPm7++PmJiYatt2dXWFkZERzM3N4eTkhEuXLtXOStQhDDkioueEsbFxtdPo6upCCAEAKCgoUBlXH38GwZAjIqpDHBwc8Ntvv6GgoAC5ubk4cuRIuWmsra1x584d3Lp1CwAQGRmJHj16AACsrKwQFxcHANi/f7/KfIcOHUJBQQEyMzPx559/okuXLrW7MnUAz8kREdUh9vb2cHFxgY+PD5o2bQobGxs0bNhQZRpDQ0MsXboU06dPly48GT16NABg6tSpWLBgAb788ks4OTmpzGdra4vXXnsNmZmZmDJliuzPxwEMOSKiOmfcuHF466238ODBA4wZMwZ2dnblroTs2bMndu3aVW5eR0dH/PrrrxW2a2tri2XLltVKzXUVQ46IqI5ZuHAhEhISUFBQAH9/f+nqSqo5hhwRUR2zcuVKjbf51ltvabzN5wEvPCEiItmq8yGXk5ODrVu3Vjp+1KhR1bahVCo1WdITuXTpEj7++GNtl0FEVK88FyG3ffv2csMf3bvtxx9/fCZ1lJSUPNX8Xbp0wXvvvaehaoiISB11/pzcypUrkZiYCF9fX+jp6cHQ0BCNGjXCjRs38Ouvv0KpVCI2Nha5ubmYMmUKcnJyUFxcjOnTp8PNza3a9k+fPo3Vq1ejQYMGuHXrFpycnPDhhx9CR0cHSqUS//nPf3Dy5EksXLgQd+7cwZYtW1BUVISuXbvigw8+gK6uLpRKJUaNGoVjx46hWbNmmDlzJpYvX47k5GTMnz8frq6uOH36NL799lusX78eISEhMDExwfjx4wEAXl5eWLduHQBgwoQJ6NatG2JjY9G5c2cMGzYMq1evRkZGBlasWAF7e/ta3d5ERLIi6rikpCTh6ekphBDi1KlTomvXriIxMVEa361bNyGEEEVFReLevXtCCCHS09OFm5ubKC0tVZmmIqdOnRKdO3cWiYmJori4WLz++uti3759QgghbGxsRHR0tBBCiISEBPHGG2+IwsJCIYQQH3zwgYiIiJCmO3LkiBBCiClTpoigoCBRWFgorl69Knx8fKTlTJo0SQghxOrVq8XGjRulGjw9PUVSUpJISkoSHTt2FNeuXRMlJSXC399fzJ07V5SWlooDBw6I4ODgareXv59ftdMQPWulRfnaLuGpXLlyReV1fmGxRtvXdHt11ePb8Vmo8z25x3Xp0gWtW7cuN1wIgVWrVuHMmTPQ0dFBamoq/v33XzRr1qzaNu3t7aU2PT09cfbsWQwZMgS6uroYPHgwAOCPP/5AXFwchg8fDgDIz89H06ZNAQD6+vro168fAMDGxgYGBgbQ19eHjY0N7ty5U6P1a9WqFWxtbQEA7dq1Q8+ePaFQKGBra6teWwoFEhfL/y4G9Hxps1Be90g01NdF99mbNdbe2eWvaaytyty+fRuxsbHw9vau0Xzh4eGIi4vDwoULKxz/3XffYceOHdDV1YW5uTmWLFkCKysrTZSsEc9dyJmYmFQ4fPfu3cjIyEB4eDj09fXh4uJS7r5tlXn8fm6PXhsaGkJXVxfAwxD19/fHO++8U25+fX19aR4dHR0YGBhIf1d0Lk9XVxelpaXS67J1Ppr38bYUCsVTnxckovrrzp072LNnT4UhV1xcDD29J4uDjh07IiwsDMbGxti2bRuWL1+OL7744mnL1Zg6H3INGjRAbm5utdPdu3cPTZs2hb6+Pk6dOlWjHtTFixeRlJQEKysr7Nu3r8JnLPXs2RNTpkzB66+/jqZNmyIrKwu5ublP9I3FyspKuh/d5cuXcfv27Rq3QUT1y65duxAaGiod2Zk+fTrmz5+PzMxMmJubY+nSpbC0tMTcuXNhamqKuLg4/PPPP5g9ezaGDBmClStX4vr16/D19YW/vz8aNWqE/fv3Iy8vD6Wlpfjqq68wf/58JCUlwdjYGIsXL0aHDh2qrcvZ2Vn6u1u3boiKiqrNzVBjdT7kzMzM4ODgAC8vLxgaGuKFF16ocDpvb28EBwfD29sbnTt3xssvv6z2Mrp06YKPPvpIuvDE3d293DTt2rXDjBkzMG7cOJSWlkJfXx8LFy58opAbPHgwIiMj4enpCXt7e7z00ks1boOI6o/4+HisXbsW27dvh7m5ObKysjB37lz4+/vD398fO3fuxMcff4yvv/4aAJCWloZt27bh77//RnBwMIYMGYJ33nlHuvgNeHgY8sqVK4iKikKTJk3w0UcfoVOnTvj666/xxx9/4N1330VkZGSN6ty5c6d06qauqPMhB1T96//Y2FgAgLm5OX766acqp6mMqamp9MZXNZ+Hh4f0PKfKpnv8rgKPxjk5OUk3SzUyMsK3335bYS179uyR/v7000+lv1u1aqUyjojqj1OnTmHIkCEwNzcHADRp0gSxsbEICQkBAPj6+mL58uXS9G5ubtDR0UG7du3w77//Vtpu79690aRJEwDA2bNnpfZ69uyJrKws3L9/X+0aIyMjERcXhx9++KHG61ebnouQIyIi9ZU9t18VdZ5Pp46TJ09i3bp1+OGHH9Re9rNSb0Luf//7H+bMmaMyzMDAADt27Cj3OAoioqoUFJVo9IrIgqISGOrrVjre2dkZU6dOxeuvvw4zMzNkZWVBqVQiOjoafn5+2L17NxwdHatcRnXXNzg6OiIqKgpvvvkmTp8+DTMzM5iamlZb+5UrV7Bw4UJs3LhRuuK8Lqk3IWdra1vj48tERBWpKpBqo7327dtj8uTJCAwMhI6ODjp16oT3338f8+bNQ2hoqHThSVVsbW2ho6MDHx8fBAQEoFGjRirjp06divnz58Pb2xvGxsYqp0uqsmzZMuTl5WH69OkAgJYtW0o3t6gLFEL833PSSRYCAgLwRbd4bZdBpOJ5/53c1atX0bFjR22X8dzTxnas8/euJCIielL15nAlERE9ubVr1+KXX35RGTZkyBAEBwdrqSL1MOSIiKhawcHBdT7QKsLDlUREJFsMOSIiki2GHBERyRZDjoiohkSxek840VR7OTk52Lp16xO1/f333+PBgwdPNK8c8MITIqIaUugZavS5jdX9jjAnJwfbt2/HmDFjatz25s2b4ePjo7FbeD1vGHJERHXcypUrkZiYCF9fX/Tq1QtNmzbFvn37UFhYCHd3d0ybNg15eXmYMWMG7t69i9LSUkyZMgX//vsv0tLSMHbsWDRp0gRbtmzR9qo8cww5IqI67p133kF8fDwiIyNx4sQJ/Prrr9i5cyeEEAgODsaZM2eQkZEBCwsLbNiwAcDDZ2w2bNgQ33//PTZt2iQ9waC+4Tk5IqLnyO+//47ff/8dfn5+8Pf3x99//42bN2/CxsYGJ0+exPLlyxETE4OGDRtqu9Q6gT05IqLniBACkyZNwqhRo8qNCw8Px9GjR/HFF19ITy6o79iTIyKq48o+JqdPnz4ICwuTXqempiI9PR2pqakwNjaGr68vxo8fjytXrpSbtz5iT46IqIZEcYFGn6wgigug0DOsdLyZmRkcHBzg5eWFvn37wsvLS+rJmZiYYPny5bh16xaWLVsGHR0d6Onp4cMPPwQAjBw5EhMmTICFhUW9vPCEj9qRGT5qh+oiPmqHAD5qh4iISKMYckREJFs8Jyc74rk/NETyU905p+eBEAIKhULbZTy3tHVmjD052eFOSHXP8x5wRkZGSE9P19p/1M87IQTS09NhZGT0zJfNnhwRUTVatWqF27dv459//tF2Kc8tIyMjtGrV6pkvlyFHRFQNfX19WFtba7sMegI8XElERLLFkCMiItliyBERkWzxjicy4+TkBCsrK22XQUSkFjMzM4SGhtZa+ww5IiKSLR6uJCIi2WLIERGRbDHkiIhIthhyREQkWww5IiKSLYYcERHJFkNORo4dO4bBgwfD3d0dGzZs0HY5GuPi4gJvb2/4+voiICAAAJCVlYWgoCAMGjQIQUFByM7OBvDwbucff/wx3N3d4e3tjcuXL0vtREREYNCgQRg0aBAiIiK0si7VmTdvHnr27AkvLy9pmCbXNS4uDt7e3nB3d8fHH39c5+6qX9H6h4SEoG/fvvD19YWvry+OHj0qjVu/fj3c3d0xePBgHD9+XBpe2b6QlJSEESNGwN3dHTNmzEBhYeGzWTE1pKSkIDAwEB4eHvD09MSmTZsA1K/3v1YIkoXi4mLh6uoqEhMTRUFBgfD29hbx8fHaLksjBg4cKNLT01WGffbZZ2L9+vVCCCHWr18vli1bJoQQ4siRI2L8+PGitLRUxMbGiuHDhwshhMjMzBQuLi4iMzNTZGVlCRcXF5GVlfVsV0QNf/75p4iLixOenp7SME2u67Bhw0RsbKwoLS0V48ePF0eOHHnGa1i1itZ/9erVYuPGjeWmjY+PF97e3qKgoEAkJiYKV1dXUVxcXOW+MG3aNLFnzx4hhBDvv/++2Lp167NZMTWkpqaKuLg4IYQQ9+7dE4MGDRLx8fH16v2vDezJycTFixfx4osvonXr1jAwMICnpycOHTqk7bJqzaFDh+Dn5wcA8PPzw8GDB1WGKxQKdOvWDTk5OUhLS8OJEyfQu3dvNGnSBI0bN0bv3r1VvvnXFT169EDjxo1VhmlqXdPS0nD//n1069YNCoUCfn5+de4zUtH6V+bQoUPw9PSEgYEBWrdujRdffBEXL16sdF8QQuDUqVMYPHgwAMDf379Orb+FhQXs7OwAAKampnj55ZeRmppar97/2sCQk4nU1FS0aNFCet28eXOkpqZqsSLNGj9+PAICAvDTTz8BANLT02FhYQEAaNasGdLT0wGU3w4tWrRAamrqc719NLWulU3/PNi6dSu8vb0xb9486XCduuv5aHhmZiYaNWoEPb2HTxiry+t/+/ZtXL16FV27duX7/5QYclTnbd++HREREfjmm2+wdetWnDlzRmW8QqGAQlE/nohen9b1kdGjR+PAgQOIjIyEhYUFPv30U22XVKtyc3Mxbdo0zJ8/H6ampirj6uP7/7QYcjLRvHlz3L17V3qdmpqK5s2ba7EizXm0Hk2bNoW7uzsuXryIpk2bIi0tDQCQlpYGc3Nzadqy2+Hu3bto3rz5c719NLWulU1f173wwgvQ1dWFjo4ORowYgUuXLgGo/DNf2XAzMzPk5OSguLgYQN1c/6KiIkybNg3e3t4YNGgQAL7/T4shJxNdunTBzZs3kZSUhMLCQkRHR8PFxUXbZT21vLw83L9/X/r7999/R/v27eHi4oJdu3YBAHbt2gVXV1cAkIYLIXD+/Hk0bNgQFhYW6NOnD06cOIHs7GxkZ2fjxIkT6NOnj9bWqyY0ta4WFhYwNTXF+fPnIYRQaasue/QfPAAcPHgQ7du3B/Bw/aOjo1FYWIikpCTcvHkT9vb2le4LCoUCTk5O+PXXXwE8vAKxLu0jQggsWLAAL7/8MoKCgqTh9f39f1p8CoGMHD16FEuWLEFJSQmGDRuG4OBgbZf01JKSkvDmm28CAEpKSuDl5YXg4GBkZmZixowZSElJgaWlJb744gs0adIEQggsXrwYx48fh7GxMZYsWYIuXboAAHbu3In169cDACZPnoxhw4Zpbb0qM3PmTPz555/IzMxE06ZN8dZbb8HNzU1j63rp0iXMmzcP+fn56NevH95///06dfirovX/888/ce3aNQCAlZUVFi9eLJ2jWrt2LcLCwqCrq4v58+ejf//+ACrfF5KSkvD2228jOzsbHTt2xIoVK2BgYKCdlX1MTEwMxowZAxsbG+joPOx/zJw5E/b29vXm/a8NDDkiIpItHq4kIiLZYsgREZFsMeSIiEi2GHJERCRbDDkiIpIthhxRPXLw4EHY2tri+vXrAB7ePsre3h6+vr7w8PDAnDlzUFRUBAA4ffo0unfvDl9fXwwdOhRfffVVlcPDw8OxePFileUFBgZKP94m0gaGHFE9smfPHnTv3h3R0dHSsDZt2iAyMhK7d+/G3bt3sW/fPmmco6MjIiMjERYWhqioKOlxLpUNJ6prGHJE9URubi7Onj2LTz75RCXkHtHV1YW9vX2FN+01MTGBnZ0dbt26pdZworqCIUdUTxw6dAh9+/aFtbU1zMzMEBcXpzK+oKAAFy5cQN++fcvNm5mZiQsXLki31KpuOFFdwZAjqieio6Ph6ekJAPDw8JB6c4mJifD19UWvXr1gYWGBDh06SPPExMTAz88P48ePx8SJE6Uwq2h4ZbeHkvtto6hu09N2AURU+7KysnDq1Cn89ddfUCgUKCkpgUKhwH//+1/pnFxGRgZGjx6NQ4cOSTfudXR0lO6BWFZFw5s0aSI9663scs3MzGpvxYiqwZ4cUT3w66+/wtfXF7/99hsOHz6Mo0ePolWrViqPXjE3N8esWbOwYcOGJ1pGly5dEBsbi3/++QfAw5sBFxYWomXLlhpZB6InwZ4cUT2wZ88eTJw4UWXYoEGDyvXG3NzcEBISgpiYmBov44UXXsD8+fMxadIklJaWwsTEBKtWrZLuqE+kDXwKARERyRa/YhERkWwx5IiISLYYckREJFsMOSIiki2GHBERyRZDjoiIZIshR0REsvX/AEcu3PHnFHhuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.barplot(data=ARPPU_product_type, y='product_type', x='ARPPU', hue='group')\n",
    "sns.despine()\n",
    "plt.title('ARPPU distribution by product type depending on group\\n')\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "По графику можем видеть, что ни по одному виду product_type наша выручка не просела, везде идет ее рост в тестовой группе по сравнению с контрольной. \n",
    "\n",
    "Причем для премиум подписки идет самый большой рост, пользователи не начали в большей степени переходить на другие платные опции."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Сравним конверсию из пользователя в покупателя премиум подписки (CR_pr) в группах.\n",
    "Т.к. переменные номинативные - используем хи-квадрат:\n",
    "\n",
    "- $H_0$: нет статистически значимой разницы между распределением CR_pr в тестовой и контрольной группе (p-value > 0.05)\n",
    "- $H_1$: есть статистически значимая разница между распределением CR_pr в тестовой и контрольной группе"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 489,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Добавим в данные колонку с конверсией в покупателя премиум подписки в зависимости от типа подписки (product_type)\n",
    "users_merged['CR_pr'] = users_merged.uid.isin(transactions_merged.query('product_type == \"premium_no_trial\"').uid.unique()).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 490,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4.842316410131557, 0.02776956942901534)"
      ]
     },
     "execution_count": 490,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stat, p, dof, expected = chi2_contingency(pd.crosstab(users_merged.query('group != \"control_1\"').CR_pr, users_merged.query('group != \"control_1\"').group))\n",
    "stat, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Так как p < 0.05, мы отвергаем нулевую гипотезу, т.е. есть статистически значимая разница в распределении CR_pr в тестовой и контрольной группе, конверсия в покупателя премиум подписки уменьшилась."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Сравним средний доход на платящего пользователя (ARPPU) в группах при помощи статистического F теста:\n",
    "\n",
    "- $H_0$: нет статистически значимых различий между средними значениями ARPPU в тестовой и контрольной группе (p-value > 0.05)\n",
    "- $H_1$: есть статистически значимые различия между средними значениями ARPPU в тестовой и контрольной группе"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверим группы на гомоскедастичность:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 491,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>W</th>\n",
       "      <th>pval</th>\n",
       "      <th>equal_var</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>levene</th>\n",
       "      <td>0.543017</td>\n",
       "      <td>0.581387</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               W      pval  equal_var\n",
       "levene  0.543017  0.581387       True"
      ]
     },
     "execution_count": 491,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.homoscedasticity(data = users_merged, dv = 'total_revenue', group = 'group')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 492,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F Test: statistic=10.4186, p-value=0.0014\n"
     ]
    }
   ],
   "source": [
    "stat, p_value = f_oneway(users_test.total_revenue, users_control_2.total_revenue)\n",
    "print(f\"F Test: statistic={stat:.4f}, p-value={p_value:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Согласно статистике F-теста (p-value < 0.05) нулевую гипотезу об отсутствии различий между средними значениями ARPPU в тестовой и контрольной группе мы отклоняем, т.е. различия статистически не значимы, значение ARPPU увеличилось."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Сравним средний доход на платящего премиум пользователя (ARPPU_pr) в группах при помощи статистического F теста:\n",
    "\n",
    "- $H_0$: нет статистически значимых различий между средними значениями ARPPU_pr в тестовой и контрольной группе (p-value > 0.05)\n",
    "- $H_1$: есть статистически значимые различия между средними значениями ARPPU_pr в тестовой и контрольной группе"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверим группы на гомоскедастичность:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 493,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>W</th>\n",
       "      <th>pval</th>\n",
       "      <th>equal_var</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>levene</th>\n",
       "      <td>0.538057</td>\n",
       "      <td>0.584272</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               W      pval  equal_var\n",
       "levene  0.538057  0.584272       True"
      ]
     },
     "execution_count": 493,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.homoscedasticity(data = users_merged.query('total_revenue > 0'), dv = 'total_revenue', group = 'group')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 494,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F Test: statistic=10.4186, p-value=0.0014\n"
     ]
    }
   ],
   "source": [
    "stat, p_value = f_oneway(users_test.query('total_revenue > 0').total_revenue, \n",
    "                         users_control_2.query('total_revenue > 0').total_revenue)\n",
    "print(f\"F Test: statistic={stat:.4f}, p-value={p_value:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "p < 0.05, мы отвергаем нулевую гипотезу и делаем вывод, что ARPPU_pr в тестовой группе статистически значимо больше ARPPU_pr в контрольной группе. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4: Подводим итоги\n",
    "Из предоставленных датасетов были отобраны только новые пользователи из нескольких стран, в которых была изменена стоимость премиум-подписки при покупке через две новые платежные системы.\n",
    "\n",
    "**По итоговым выборкам можно сказать следующее:**\n",
    "- Выборки для анализа репрезентативные, группы пользователей распределены одинаково по приведенным признакам, но были произведены некоторые манипуляции по удалению неподходящих данных и выбросов (в особенности в 1 контрольной группе, поэтому в А/В тестах использовалась 2 контрольная группа)\n",
    "- Основная возрастная категория - молодежь (25-35 лет). Средний возраст пользователей - 32 года\n",
    "- Средний коэффициент привлекательности пользователей - 340-382 (max - 1000)\n",
    "- По географическому признаку превалируют пользователи из США\n",
    "- По половому признаку превалируют мужчины"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Согласно проведенным исследованиям:**\n",
    "- Статистически значимо не изменилось количество дней посещения приложения\n",
    "- Статистически значимо не изменилось количество пользователей, продливших премиум подписку\n",
    "\n",
    "- Конверсия в покупку премиум подписки (CR_pr) уменьшилась на 21% (стат значимо). Т.е. в ходе эксперимента покупать стали реже, видимо стоимость подписки изменилась в большую сторону, и часть пользователей отказалась от платных услуг\n",
    "\n",
    "- Средний доход на платящего пользователя (ARPPU) увеличился на 42% (стат значимо) (причем доход увеличился по всем видам продукта)\n",
    "- Средний доход на платящего за премиум подписку пользователя (ARPPU_pr) увеличился на 82% (стат значимо). Т.е. хоть количество платящих пользователей уменьшилось, но их \"качество\" - доход от них - значимо увеличился. Платящие пользователи стали покупать на большую сумму (что логично при увеличении стоимости премиум подписки)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Таким образом, судя по нашей основной метрике - ARPPU_pr - эксперимент оказался удачным, изменение (предполагаемо повышение) стоимости премиум подписки при оплате через 2 новые платежные системы статистически значимо увеличило метрику ARPPU_pr.\n",
    "\n",
    "Но есть опасение, что в долгосрочной перспективе падениеCR_pr может привести к большим потерям платящих за премиум подписку пользователей, и ARPPU_pr не сможет нивелировать эти потери. \n",
    "\n",
    "Рекомендуется раскатывать данное обновление, но также необходимо периодически мониторить данные экономические показатели."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}