diff --git a/notebooks/images/add_the_repository.jpg b/notebooks/images/add_the_repository.jpg new file mode 100644 index 0000000..98dcb0e Binary files /dev/null and b/notebooks/images/add_the_repository.jpg differ diff --git a/notebooks/images/governance.jpg b/notebooks/images/governance.jpg new file mode 100644 index 0000000..7863cfe Binary files /dev/null and b/notebooks/images/governance.jpg differ diff --git a/notebooks/images/monitoring_job.jpg b/notebooks/images/monitoring_job.jpg new file mode 100644 index 0000000..06818b3 Binary files /dev/null and b/notebooks/images/monitoring_job.jpg differ diff --git a/notebooks/images/monitoring_result.jpg b/notebooks/images/monitoring_result.jpg new file mode 100644 index 0000000..bf1f337 Binary files /dev/null and b/notebooks/images/monitoring_result.jpg differ diff --git a/notebooks/images/repository.jpg b/notebooks/images/repository.jpg new file mode 100644 index 0000000..0c552ec Binary files /dev/null and b/notebooks/images/repository.jpg differ diff --git a/notebooks/images/runtime.jpg b/notebooks/images/runtime.jpg new file mode 100644 index 0000000..f50d219 Binary files /dev/null and b/notebooks/images/runtime.jpg differ diff --git a/notebooks/images/studio_model.jpg b/notebooks/images/studio_model.jpg new file mode 100644 index 0000000..cbd0d82 Binary files /dev/null and b/notebooks/images/studio_model.jpg differ diff --git a/notebooks/images/violations.jpg b/notebooks/images/violations.jpg new file mode 100644 index 0000000..01a172e Binary files /dev/null and b/notebooks/images/violations.jpg differ diff --git a/notebooks/model_bias_monitor.ipynb b/notebooks/model_bias_monitor.ipynb new file mode 100644 index 0000000..c307aa6 --- /dev/null +++ b/notebooks/model_bias_monitor.ipynb @@ -0,0 +1,2319 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "d6eccd85-1c0e-4f7b-9da0-dd473f8a9d95", + "metadata": {}, + "source": [ + "# Model Bias Monitoring with AWS SageMaker Clarity" + ] + }, + { + "cell_type": "markdown", + "id": "67557cb8-fe01-473a-a256-442ab3131197", + "metadata": {}, + "source": [ + "This Jupyter notebook shows how to perform model bias observability with AWS SageMaker (based on [docs](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker_model_monitor/fairness_and_explainability/SageMaker-Model-Monitor-Fairness-and-Explainability.html))\n", + "\n", + "1. Configuration\n", + "1. Dataset description\n", + "1. Deploying model for ML Observability\n", + "1. Setting up monitoring job - Creates Monitoring tasks by creating baseline and scheduling regular monitoring\n", + "1. Generate traffic - Provides traffic (examples and ground truth) to the endpoint based on which bias metrics will be calculated\n", + "1. Cleaning up - Removes all the created resources.\n", + "\n", + "Prerequisites:\n", + "\n", + "- Existing Roles with all needed permissions (S3, SageMaker, etc.);\n", + "- Configured SageMaker Domain;\n", + "- SageMaker Studio user.\n", + "\n", + "One can use SageMaker Studio (Jupyter-like environment) to run this notebook. To do that, follow the next steps:\n", + "\n", + "1. Run the SageMaker Studio\n", + "2. Clone this repository (https://github.com/griddynamics/gd-ml-observability.git)\n", + "\n", + "\n", + "\"Add\n", + "\n", + "\"repository.png\"\n", + "\n", + "3. Run this notebook cell by cell paying attention to comments" + ] + }, + { + "cell_type": "markdown", + "id": "9096958f-3dbf-41b6-96bd-dce1fe2bb299", + "metadata": {}, + "source": [ + "### Background" + ] + }, + { + "cell_type": "markdown", + "id": "30192a0e-66ad-4f5f-b1fa-10296809ef81", + "metadata": {}, + "source": [ + "A computer system powered by machine learning model might contain bias, i.e., discriminate against certain individuals or groups of individuals. The models learn from data, so it might memorize bias that appears in that data. Biased judgements impact on people badly and unfairly, and automation of such process could lead to disasterous consequences. Hence, people developing such systems must be aware of it, and be able to address biased models. [\\*](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-detect-data-bias.html)\n", + "\n", + "AWS provides SageMaker Clarity services to identify and measure bias level by calculating [pre-training](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-measure-data-bias.html) and [post-training](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-measure-post-training-bias.html) bias metrics based on model input, output, and ground truth information gathered and added during monitoring process and comparing it to calculated in advance baseline metrics." + ] + }, + { + "cell_type": "markdown", + "id": "4d6d91b3-4a1a-4384-b836-cdf69c6382f4", + "metadata": {}, + "source": [ + "## 1. Configuration" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "db0aa694-f45f-4051-b253-1870d150a9ab", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: boto3 in /opt/conda/lib/python3.7/site-packages (1.26.80)\n", + "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from boto3) (0.6.0)\n", + "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3) (1.0.1)\n", + "Requirement already satisfied: botocore<1.30.0,>=1.29.80 in /opt/conda/lib/python3.7/site-packages (from boto3) (1.29.80)\n", + "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.7/site-packages (from botocore<1.30.0,>=1.29.80->boto3) (1.26.14)\n", + "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/lib/python3.7/site-packages (from botocore<1.30.0,>=1.29.80->boto3) (2.8.2)\n", + "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.30.0,>=1.29.80->boto3) (1.14.0)\n", + "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n", + "\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.0.1\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n" + ] + } + ], + "source": [ + "!pip install --upgrade boto3" + ] + }, + { + "cell_type": "markdown", + "id": "3dccd774-9ef4-477a-a409-4e8a4606ee94", + "metadata": {}, + "source": [ + "### Importing necessary libraries" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "23465867-bed7-4cef-8699-3eeab0903008", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "import copy\n", + "import json\n", + "import random\n", + "import time\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "import pandas as pd\n", + "import sagemaker\n", + "\n", + "from datetime import datetime\n", + "\n", + "from IPython.display import display\n", + "from sagemaker import get_execution_role, image_uris\n", + "from sagemaker.clarify import (\n", + " BiasConfig,\n", + " DataConfig,\n", + " ModelConfig,\n", + " ModelPredictedLabelConfig,\n", + ")\n", + "from sagemaker.model import Model\n", + "from sagemaker.model_monitor import (\n", + " BiasAnalysisConfig,\n", + " CronExpressionGenerator,\n", + " DataCaptureConfig,\n", + " EndpointInput,\n", + " ModelBiasMonitor,\n", + ")\n", + "from sagemaker.s3 import S3Downloader, S3Uploader\n" + ] + }, + { + "cell_type": "markdown", + "id": "735c2543-a393-467c-a0f8-c7c6d6c3a9a1", + "metadata": {}, + "source": [ + "### Setting up necessary variables and constants" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "be24abcc-7a7f-429b-b635-a33c30a39625", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "sagemaker_session = sagemaker.session.Session()\n", + "sagemaker_client = sagemaker_session.sagemaker_client\n", + "sagemaker_runtime_client = sagemaker_session.sagemaker_runtime_client\n", + "role = get_execution_role()\n", + "\n", + "DATA_BUCKET = 'adp-rnd-ml-datasets'\n", + "# Change the following bucket names if you want to run this code outside GridDynamics to the ones you have accees to.\n", + "STAGE_BUCKET = 'adp-rnd-ml-stage'\n", + "MODEL_BUCKET = 'adp-rnd-ml-models'\n", + "MODEL_PATH = f\"s3://{MODEL_BUCKET}/xgboost-for-loan-default-data/output/xgboost-for-loan-default-data-2023-02-24-08-18-30-629/output/model.tar.gz\"\n", + "DATASET_PATH = f\"s3://{DATA_BUCKET}/loan_default/dataset.zip\"\n", + "TEST_PATH = f\"s3://{DATA_BUCKET}/loan_default/test.csv\"\n", + "VALIDATION_PATH = f\"s3://{DATA_BUCKET}/loan_default/eval.csv\"\n", + "\n", + "NOW = datetime.now()\n", + "\n", + "%matplotlib inline\n", + "plt.style.use(\"seaborn-pastel\")" + ] + }, + { + "cell_type": "markdown", + "id": "f06b3c0a-117f-415f-b30b-a52938c0c623", + "metadata": {}, + "source": [ + "## 2. Dataset" + ] + }, + { + "cell_type": "markdown", + "id": "6414debb-84e5-4f6a-9914-fe83ad09300b", + "metadata": {}, + "source": [ + "Loan Default Dataset was used for this work. It's published under CC0: Public Domain license, hence both non-profit, and commercial usage is permitted. It consists of 33 features of different types (categorical/binary, continuous/decimal) including the binary target feature **Status** which is used for the classification task. The dataset has set of columns with missing values, which needs to be handled before using with the machine learning models. It has outliers, which needs to be handled, it is likely to have data leakege, which solves the task for 100% accuracy, so such features are to remove." + ] + }, + { + "cell_type": "markdown", + "id": "6c5acb59-42b8-49d9-986d-1790c3fe0c9d", + "metadata": {}, + "source": [ + "### Basic EDA" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "957b1e1e-004a-40fb-b96c-300e12153d00", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Int64Index: 148670 entries, 24890 to 173559\n", + "Data columns (total 33 columns):\n", + " # Column Non-Null Count Dtype \n", + "--- ------ -------------- ----- \n", + " 0 year 148670 non-null int64 \n", + " 1 loan_limit 145326 non-null object \n", + " 2 Gender 148670 non-null object \n", + " 3 approv_in_adv 147762 non-null object \n", + " 4 loan_type 148670 non-null object \n", + " 5 loan_purpose 148536 non-null object \n", + " 6 Credit_Worthiness 148670 non-null object \n", + " 7 open_credit 148670 non-null object \n", + " 8 business_or_commercial 148670 non-null object \n", + " 9 loan_amount 148670 non-null int64 \n", + " 10 rate_of_interest 112231 non-null float64\n", + " 11 Interest_rate_spread 112031 non-null float64\n", + " 12 Upfront_charges 109028 non-null float64\n", + " 13 term 148629 non-null float64\n", + " 14 Neg_ammortization 148549 non-null object \n", + " 15 interest_only 148670 non-null object \n", + " 16 lump_sum_payment 148670 non-null object \n", + " 17 property_value 133572 non-null float64\n", + " 18 construction_type 148670 non-null object \n", + " 19 occupancy_type 148670 non-null object \n", + " 20 Secured_by 148670 non-null object \n", + " 21 total_units 148670 non-null object \n", + " 22 income 139520 non-null float64\n", + " 23 credit_type 148670 non-null object \n", + " 24 Credit_Score 148670 non-null int64 \n", + " 25 co-applicant_credit_type 148670 non-null object \n", + " 26 age 148470 non-null object \n", + " 27 submission_of_application 148470 non-null object \n", + " 28 LTV 133572 non-null float64\n", + " 29 Region 148670 non-null object \n", + " 30 Security_Type 148670 non-null object \n", + " 31 Status 148670 non-null int64 \n", + " 32 dtir1 124549 non-null float64\n", + "dtypes: float64(8), int64(4), object(21)\n", + "memory usage: 38.6+ MB\n" + ] + } + ], + "source": [ + "dataset = pd.read_csv(DATASET_PATH, compression='zip', index_col='ID')\n", + "dataset.info()" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "f770040e-9539-4709-b744-72b522d2dc73", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
yearloan_limitGenderapprov_in_advloan_typeloan_purposeCredit_Worthinessopen_creditbusiness_or_commercialloan_amount...credit_typeCredit_Scoreco-applicant_credit_typeagesubmission_of_applicationLTVRegionSecurity_TypeStatusdtir1
ID
248902019cfSex Not Availablenopretype1p1l1nopcnob/c116500...EXP758CIB25-34to_inst98.728814southdirect145.0
248912019cfMalenopretype2p1l1nopcb/c206500...EQUI552EXP55-64to_instNaNNorthdirect1NaN
248922019cfMalepretype1p1l1nopcnob/c406500...EXP834CIB35-44to_inst80.019685southdirect046.0
248932019cfMalenopretype1p4l1nopcnob/c456500...EXP587CIB45-54not_inst69.376900Northdirect042.0
248942019cfJointpretype1p1l1nopcnob/c696500...CRIF602EXP25-34not_inst91.886544Northdirect039.0
\n", + "

5 rows × 33 columns

\n", + "
" + ], + "text/plain": [ + " year loan_limit Gender approv_in_adv loan_type \\\n", + "ID \n", + "24890 2019 cf Sex Not Available nopre type1 \n", + "24891 2019 cf Male nopre type2 \n", + "24892 2019 cf Male pre type1 \n", + "24893 2019 cf Male nopre type1 \n", + "24894 2019 cf Joint pre type1 \n", + "\n", + " loan_purpose Credit_Worthiness open_credit business_or_commercial \\\n", + "ID \n", + "24890 p1 l1 nopc nob/c \n", + "24891 p1 l1 nopc b/c \n", + "24892 p1 l1 nopc nob/c \n", + "24893 p4 l1 nopc nob/c \n", + "24894 p1 l1 nopc nob/c \n", + "\n", + " loan_amount ... credit_type Credit_Score co-applicant_credit_type \\\n", + "ID ... \n", + "24890 116500 ... EXP 758 CIB \n", + "24891 206500 ... EQUI 552 EXP \n", + "24892 406500 ... EXP 834 CIB \n", + "24893 456500 ... EXP 587 CIB \n", + "24894 696500 ... CRIF 602 EXP \n", + "\n", + " age submission_of_application LTV Region Security_Type \\\n", + "ID \n", + "24890 25-34 to_inst 98.728814 south direct \n", + "24891 55-64 to_inst NaN North direct \n", + "24892 35-44 to_inst 80.019685 south direct \n", + "24893 45-54 not_inst 69.376900 North direct \n", + "24894 25-34 not_inst 91.886544 North direct \n", + "\n", + " Status dtir1 \n", + "ID \n", + "24890 1 45.0 \n", + "24891 1 NaN \n", + "24892 0 46.0 \n", + "24893 0 42.0 \n", + "24894 0 39.0 \n", + "\n", + "[5 rows x 33 columns]" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Few instances of dataset\n", + "dataset.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "1e542bba-9c2e-4df8-aaad-0ae5d8c085dd", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
yearloan_amountrate_of_interestInterest_rate_spreadUpfront_chargestermproperty_valueincomeCredit_ScoreLTVStatusdtir1
count148670.01.486700e+05112231.000000112031.000000109028.000000148629.0000001.335720e+05139520.000000148670.000000133572.000000148670.000000124549.000000
mean2019.03.311177e+054.0454760.4416563224.996127335.1365824.978935e+056957.338876699.78910372.7464570.24644537.732932
std0.01.839093e+050.5613910.5130433251.12151058.4090843.599353e+056496.586382115.87585739.9676030.43094210.545435
min2019.01.650000e+040.000000-3.6380000.00000096.0000008.000000e+030.000000500.0000000.9674780.0000005.000000
25%2019.01.965000e+053.6250000.076000581.490000360.0000002.680000e+053720.000000599.00000060.4748600.00000031.000000
50%2019.02.965000e+053.9900000.3904002596.450000360.0000004.180000e+055760.000000699.00000075.1358700.00000039.000000
75%2019.04.365000e+054.3750000.7754004812.500000360.0000006.280000e+058520.000000800.00000086.1842110.00000045.000000
max2019.03.576500e+068.0000003.35700060000.000000360.0000001.650800e+07578580.000000900.0000007831.2500001.00000061.000000
\n", + "
" + ], + "text/plain": [ + " year loan_amount rate_of_interest Interest_rate_spread \\\n", + "count 148670.0 1.486700e+05 112231.000000 112031.000000 \n", + "mean 2019.0 3.311177e+05 4.045476 0.441656 \n", + "std 0.0 1.839093e+05 0.561391 0.513043 \n", + "min 2019.0 1.650000e+04 0.000000 -3.638000 \n", + "25% 2019.0 1.965000e+05 3.625000 0.076000 \n", + "50% 2019.0 2.965000e+05 3.990000 0.390400 \n", + "75% 2019.0 4.365000e+05 4.375000 0.775400 \n", + "max 2019.0 3.576500e+06 8.000000 3.357000 \n", + "\n", + " Upfront_charges term property_value income \\\n", + "count 109028.000000 148629.000000 1.335720e+05 139520.000000 \n", + "mean 3224.996127 335.136582 4.978935e+05 6957.338876 \n", + "std 3251.121510 58.409084 3.599353e+05 6496.586382 \n", + "min 0.000000 96.000000 8.000000e+03 0.000000 \n", + "25% 581.490000 360.000000 2.680000e+05 3720.000000 \n", + "50% 2596.450000 360.000000 4.180000e+05 5760.000000 \n", + "75% 4812.500000 360.000000 6.280000e+05 8520.000000 \n", + "max 60000.000000 360.000000 1.650800e+07 578580.000000 \n", + "\n", + " Credit_Score LTV Status dtir1 \n", + "count 148670.000000 133572.000000 148670.000000 124549.000000 \n", + "mean 699.789103 72.746457 0.246445 37.732932 \n", + "std 115.875857 39.967603 0.430942 10.545435 \n", + "min 500.000000 0.967478 0.000000 5.000000 \n", + "25% 599.000000 60.474860 0.000000 31.000000 \n", + "50% 699.000000 75.135870 0.000000 39.000000 \n", + "75% 800.000000 86.184211 0.000000 45.000000 \n", + "max 900.000000 7831.250000 1.000000 61.000000 " + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Shows statistics of numerical features\n", + "dataset.describe() " + ] + }, + { + "cell_type": "markdown", + "id": "f2b8a2dd-5ddd-4a76-b6a1-1f1e9c0e21a3", + "metadata": {}, + "source": [ + "Here one can get intuition of sensitive features distribution before preprocessing (handling missing values, and one-hot encoding):" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "2e2da775-d508-4583-8212-5884bd0193e5", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3kAAASlCAYAAAAVqxAPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXgNd9/H8c/JHtlISCLEvostVBu0dkHo4i6tokI3VUpVe5duuoaW7i1udUvaUq2tT1tVUhKtUntrrSpiqcQuCSpIfs8fHufpkSAhyZHxfl3XuS5n5jcz3/maaD6dOb9jM8YYAQAAAAAswcXZBQAAAAAACg8hDwAAAAAshJAHAAAAABZCyAMAAAAACyHkAQAAAICFEPIAAAAAwEIIeQAAAABgIYQ8AAAAALAQQh4AAAAAWAghDwCAIrZhwwY98MADql69ury9veXt7a2aNWvqkUce0Zo1a5xSU5UqVRQbG+uUYwMAipabswsAAMDKJk+erCFDhqh27doaNmyY6tevL5vNpq1bt+rzzz/XTTfdpD///FPVq1d3dqkAAIsg5AEAUER+/vlnDR48WDExMZo9e7Y8PDzs69q1a6fHHntMs2bNkre3txOrvHbZ2dk6d+6cPD09nV0KAEA8rgkAQJF5/fXX5erqqsmTJzsEvH/q2bOnwsLC7O/XrFmj22+/XYGBgfLy8lKTJk305ZdfOmwTHx8vm82mpKQkPfrooypbtqyCgoLUo0cP7d+/32Hs2bNn9fTTTys0NFSlSpVSq1attGrVqjxrSUtL0yOPPKKKFSvKw8NDVatW1UsvvaRz587Zx6SkpMhms+mNN97Qq6++qqpVq8rT01NJSUlX2yYAQCHjTh4AAEUgOztbSUlJatasmcqXL5+vbZKSktS5c2fdfPPNmjRpkgICAjRz5kzdc889OnXqVK7P0D344IOKiYnRjBkztHfvXj311FPq27evlixZYh/z0EMP6ZNPPtHIkSPVsWNHbdq0ST169FBmZqbDvtLS0tS8eXO5uLjohRdeUPXq1bVixQq9+uqrSklJ0bRp0xzGv/fee6pVq5bGjx8vf39/1axZ8+oaBQAodIQ8AACKwOHDh/X333+rcuXKudZlZ2fLGGN/7+rqKpvNpsGDB6t+/fpasmSJ3NzO/yc6Ojpahw8f1ujRo3X//ffLxeX/H8Lp3Lmz3nvvPfv7o0eP6umnn1ZaWppCQ0P1+++/KyEhQU888YTeeOMNSVLHjh0VEhKiPn36ONQ0ZswYHTt2TJs3b1alSpUkSe3bt5e3t7dGjhypp556SvXq1bOP9/Ly0sKFC+Xu7l4I3QIAFCYe1wQAoJg1bdpU7u7u9teECRP0559/6vfff7eHr3PnztlfXbt2VWpqqrZt2+awn9tvv93hfcOGDSVJu3fvliT7I5QXB7pevXrZQ+QF3377rdq2bauwsDCHY3fp0kWStHTp0lzHJuABwPWJO3kAABSBsmXLytvb2x64/mnGjBk6deqUUlNT7UHtwIEDkqSRI0dq5MiRee7z8OHDDu+DgoIc3l+Y+OTvv/+WJB05ckSSFBoa6jDOzc0t17YHDhzQN998c8ngdvGx8/sIKgCg+BHyAAAoAq6urmrXrp0WLVqk1NRUh1B04bHHlJQU+7KyZctKkkaNGqUePXrkuc/atWsXqIYLQS4tLU0VKlSwLz937pw9AP7z+A0bNtRrr72W577+OTmMJNlstgLVAgAoPoQ8AACKyKhRo7RgwQINGjRIs2fPvuzjjbVr11bNmjX122+/6fXXXy+U47dp00aSNH36dDVt2tS+/Msvv3SYMVOSunXrpu+++07Vq1dXmTJlCuX4AADnIOQBAFBEWrZsqQ8//FBDhw5VZGSkHn74YdWvX18uLi5KTU3VnDlzJEn+/v6Szn9xepcuXRQdHa3Y2FhVqFBBR48e1datW7Vu3TrNmjWrQMevW7eu+vbtq3feeUfu7u7q0KGDNm3aZJ8R859efvllJSYmqkWLFnr88cdVu3ZtnT59WikpKfruu+80adIkVaxYsXAaAwAoUoQ8AACK0KBBgxQVFaV3331Xb7/9tvbv3y+bzaaKFSuqRYsWWrx4sdq1aydJatu2rVatWqXXXntNw4cP17FjxxQUFKR69eqpV69eV3X8qVOnKiQkRPHx8XrvvffUuHFjzZkzR/fee6/DuPLly2vNmjV65ZVX9Oabb2rfvn3y8/NT1apV1blzZ+7uAUAJYjP/nMMZAAAAAFCi8RUKAAAAAGAhPK6JAsvJydH+/fvl5+fH7GoAAADAFRhjlJmZqbCwMLm4FP19NkIeCmz//v0KDw93dhkAAABAibJ3795imcSKkIcC8/Pzk3T+Ir14djYAAAAAjjIyMhQeHm7/PbqoEfJQYBce0fT39yfkAQAAAPlUXB91YuIVAAAAALAQQh4AAAAAWAghDwAAAAAshJAHAAAAABZCyAMAAAAACyHkAQAAAICFEPIAAAAAwEIIeQAAAABgIXwZOq7apFWSt4+zqwAAoPANjXJ2BQBw9biTBwAAAAAWQsgDAAAAAAsh5AEAAACAhRDyAAAAAMBCCHkAAAAAYCGEPAAAAACwEEIeAAAAAFgIIQ8AAAAALISQBwAAAAAWQsgDAAAAAAsh5AEAAACAhRDyAAAAAMBCCHkAAAAAYCGEPAAAAACwEEIeAAAAAFgIIQ8AAAAALISQBwAAAAAWQsgDAAAAAAsh5FlcSkqKbDabfv31V2eXAgAAAKAYEPIKUWxsrGw2W67Xn3/+6ezSAAAAANwg3JxdgNV07txZ06ZNc1hWrlw5J1UDAAAA4EbDnbxC5unpqdDQUIeXq6urjDF64403VK1aNXl7e6tRo0aaPXu2fbvk5GTZbDYtXLhQTZo0kbe3t9q1a6eDBw9qwYIFqlu3rvz9/dW7d2+dOnXKvt3333+vVq1aqXTp0goKClK3bt20Y8eOy9a4ZcsWde3aVb6+vgoJCVG/fv10+PDhS47PyspSRkaGwwsAAADA9YmQV0yee+45TZs2TRMnTtTmzZv1xBNPqG/fvlq6dKnDuDFjxuiDDz7Q8uXLtXfvXvXq1UvvvPOOZsyYofnz5ysxMVHvv/++ffzJkyc1YsQIrV69WosXL5aLi4vuuusu5eTk5FlHamqqWrdurcaNG2vNmjX6/vvvdeDAAfXq1euStcfFxSkgIMD+Cg8PL5ymAAAAACh0NmOMcXYRVhEbG6vPPvtMXl5e9mVdunRRfHy8ypYtqyVLligqKsq+7sEHH9SpU6c0Y8YMJScnq23btvrhhx/Uvn17SdLYsWM1atQo7dixQ9WqVZMkDRo0SCkpKfr+++/zrOHQoUMKDg7Wxo0bFRERoZSUFFWtWlXr169X48aN9cILL2jlypVauHChfZt9+/YpPDxc27ZtU61atXLtMysrS1lZWfb3GRkZCg8P17jEdHn7+F9b0wAAuA4NjbryGADIr4yMDAUEBCg9PV3+/kX/+zOfyStkbdu21cSJE+3vfXx8tGXLFp0+fVodO3Z0GHvmzBk1adLEYVnDhg3tfw4JCVGpUqXsAe/CslWrVtnf79ixQ88//7x++eUXHT582H4Hb8+ePYqIiMhV39q1a5WUlCRfX99c63bs2JFnyPP09JSnp+eVTh0AAADAdYCQV8h8fHxUo0YNh2V79uyRJM2fP18VKlRwWHdxeHJ3d7f/2WazOby/sOyfj2J2795d4eHhmjJlisLCwpSTk6OIiAidOXMmz/pycnLUvXt3jRs3Lte68uXL5+MMAQAAAFzPCHnFoF69evL09NSePXvUunXrQtvvkSNHtHXrVk2ePFm33nqrJGnZsmWX3SYyMlJz5sxRlSpV5ObGXz8AAABgNUy8Ugz8/Pw0cuRIPfHEE0pISNCOHTu0fv16ffjhh0pISLjq/ZYpU0ZBQUH6z3/+oz///FNLlizRiBEjLrvNY489pqNHj6p3795atWqVdu7cqUWLFmngwIHKzs6+6loAAAAAXB+4lVNMXnnlFQUHBysuLk47d+5U6dKlFRkZqdGjR1/1Pl1cXDRz5kw9/vjjioiIUO3atfXee++pTZs2l9wmLCxMP//8s/79738rOjpaWVlZqly5sjp37iwXFzI/AAAAUNIxuyYK7MLsQMyuCQCwKmbXBFCYint2TW7dAAAAAICFEPIAAAAAwEIIeQAAAABgIYQ8AAAAALAQQh4AAAAAWAghDwAAAAAshJAHAAAAABZCyAMAAAAACyHkAQAAAICFEPIAAAAAwEIIeQAAAABgIYQ8AAAAALAQQh4AAAAAWAghDwAAAAAshJAHAAAAABbi5uwCUHINai75+zu7CgAAAAD/xJ08AAAAALAQQh4AAAAAWAghDwAAAAAshJAHAAAAABZCyAMAAAAACyHkAQAAAICFEPIAAAAAwEIIeQAAAABgIYQ8AAAAALAQQh4AAAAAWAghDwAAAAAsxM3ZBaDkmrRK8vZxdhUAAAA3lqFRzq4A1zvu5AEAAACAhRDyAAAAAMBCCHkAAAAAYCGEPAAAAACwEEIeAAAAAFgIIQ8AAAAALISQBwAAAAAWQsgDAAAAAAsh5AEAAACAhRDyAAAAAMBCCHkAAAAAYCGEPAAAAACwEEIeAAAAAFgIIQ8AAAAALISQBwAAAAAWQsgDAAAAAAsh5AEAAACAhRDynKxNmzYaPny4/X2VKlX0zjvv5Hv7+Ph4lS5d+rJjxowZo8aNG191jQAAAABKjgKFvIMHD+qRRx5RpUqV5OnpqdDQUEVHR2vFihVFVZ+k80HIZrNp5syZDsvfeecdValSpUD7stls+uqrr/I9/uGHH5arq2uuYxeWuXPn6pVXXimSfQMAAAC48RQo5P3rX//Sb7/9poSEBP3xxx/6+uuv1aZNGx09erSo6rPz8vLSc889p7Nnzxb5sS44deqUvvjiCz311FOaOnVqkRwjMDBQfn5+RbJvAAAAADeefIe848ePa9myZRo3bpzatm2rypUrq3nz5ho1apRiYmLs49LT0/Xwww8rODhY/v7+ateunX777TdJkjFGHTp0UOfOnWWMse+3UqVKevbZZy97/N69eys9PV1Tpky57LiJEyeqevXq8vDwUO3atfXpp5/a112463fXXXfJZrNd8S7grFmzVK9ePY0aNUo///yzUlJS7OsWLlwoLy8vHT9+3GGbxx9/XK1bt5YkHTlyRL1791bFihVVqlQpNWjQQJ9//rnD+Isf17zYW2+9pQYNGsjHx0fh4eEaPHiwTpw4kWvcV199pVq1asnLy0sdO3bU3r17L3tu06ZNU926deXl5aU6deroo48+uux4AAAAACVDvkOer6+vfH199dVXXykrKyvPMcYYxcTEKC0tTd99953Wrl2ryMhItW/fXkePHpXNZlNCQoJWrVql9957T5I0aNAghYSEaMyYMZc9vr+/v0aPHq2XX35ZJ0+ezHPMvHnzNGzYMD355JPatGmTHnnkEQ0YMEBJSUmSpNWrV0s6H3BSU1Pt7y9l6tSp6tu3rwICAtS1a1dNmzbNvq5Dhw4qXbq05syZY1+WnZ2tL7/8Un369JEknT59Wk2bNtW3336rTZs26eGHH1a/fv20cuXKyx73n1xcXPTee+9p06ZNSkhI0JIlS/T00087jDl16pRee+01JSQk6Oeff1ZGRobuvffeS+5zypQpevbZZ/Xaa69p69atev311/X8888rISEhz/FZWVnKyMhweAEAAAC4PuU75Lm5uSk+Pl4JCQkqXbq0WrZsqdGjR2vDhg32MUlJSdq4caNmzZqlZs2aqWbNmho/frxKly6t2bNnS5IqVKigyZMn69///rdGjx6tb775RtOnT5e7u/sVaxg8eLC8vLz01ltv5bl+/Pjxio2N1eDBg1WrVi2NGDFCPXr00Pjx4yVJ5cqVkySVLl1aoaGh9vd52b59u3755Rfdc889kqS+fftq2rRpysnJkSS5urrqnnvu0YwZM+zbLF68WMeOHVPPnj3t5zpy5Eg1btxY1apV09ChQxUdHa1Zs2Zd8VwvGD58uNq2bauqVauqXbt2euWVV/Tll186jDl79qw++OADRUVFqWnTpkpISNDy5cu1atWqPPf5yiuvaMKECerRo4eqVq2qHj166IknntDkyZPzHB8XF6eAgAD7Kzw8PN/1AwAAACheBf5M3v79+/X1118rOjpaycnJioyMVHx8vCRp7dq1OnHihIKCgux3/nx9fbVr1y7t2LHDvp+ePXuqR48eiouL04QJE1SrVq18Hd/T01Mvv/yy3nzzTR0+fDjX+q1bt6ply5YOy1q2bKmtW7cW5DQlnb+LFx0drbJly0qSunbtqpMnT+qHH36wj+nTp4+Sk5O1f/9+SdL06dPVtWtXlSlTRtL5O3uvvfaaGjZsaO/JokWLtGfPnnzXkZSUpI4dO6pChQry8/PT/fffryNHjjjczXRzc1OzZs3s7+vUqaPSpUvned6HDh3S3r179cADDzj8Hb366qsOf0f/NGrUKKWnp9tfV3oUFAAAAIDzuBV0gwuf+erYsaNeeOEFPfjgg3rxxRcVGxurnJwclS9fXsnJybm2++c0/6dOndLatWvl6uqq7du3F+j4ffv21fjx4/Xqq6/m+Zk6m83m8N4Yk2vZlWRnZ+uTTz5RWlqa3NzcHJZPnTpVnTp1kiQ1b95c1atX18yZM/Xoo49q3rx5Do90TpgwQW+//bbeeecd++fqhg8frjNnzuSrjt27d6tr164aNGiQXnnlFQUGBmrZsmV64IEHck1Ak9c55rXswp3IKVOm6Oabb3ZY5+rqmmcdnp6e8vT0zFfNAAAAAJyrwCHvYvXq1bN/JUFkZKQ9GF1uUpMnn3xSLi4uWrBggbp27aqYmBi1a9cuX8dzcXFRXFycevTooUcffdRhXd26dbVs2TLdf//99mXLly9X3bp17e/d3d2VnZ192WN89913yszM1Pr16x2Cz++//64+ffroyJEjCgoKkiTdd999mj59uipWrCgXFxeHSWh++ukn3XHHHerbt6+k8wFr+/btDvVczpo1a3Tu3DlNmDBBLi7nb7pe/KimJJ07d05r1qxR8+bNJUnbtm3T8ePHVadOnVxjQ0JCVKFCBe3cudP+2UEAAAAA1pHvxzWPHDmidu3a6bPPPtOGDRu0a9cuzZo1S2+88YbuuOMOSecnI4mKitKdd96phQsXKiUlRcuXL9dzzz2nNWvWSJLmz5+v//73v5o+fbo6duyoZ555Rv3799exY8fyXXRMTIxuvvnmXJ8he+qppxQfH69JkyZp+/bteuuttzR37lyNHDnSPqZKlSpavHix0tLSLnnMqVOnKiYmRo0aNVJERIT99a9//UvlypXTZ599Zh/bp08frVu3Tq+99pruvvtueXl52dfVqFFDiYmJWr58ubZu3apHHnlEaWlp+T7P6tWr69y5c3r//fe1c+dOffrpp5o0aVKuce7u7ho6dKhWrlypdevWacCAAbrlllvsoe9iY8aMUVxcnN5991398ccf2rhxo6ZNm3bJzzoCAAAAKDkKNLvmzTffrLffflu33XabIiIi9Pzzz+uhhx7SBx98IOn844HfffedbrvtNg0cOFC1atXSvffeq5SUFIWEhOjQoUN64IEHNGbMGEVGRkqSXnzxRYWFhWnQoEEFKnzcuHE6ffq0w7I777xT7777rt58803Vr19fkydP1rRp09SmTRv7mAkTJigxMVHh4eFq0qRJrv0eOHBA8+fP17/+9a9c62w2m3r06OHwnXk1a9bUTTfdpA0bNuS6M/b8888rMjJS0dHRatOmjUJDQ3XnnXfm+xwbN26st956S+PGjVNERISmT5+uuLi4XONKlSqlf//737rvvvsUFRUlb2/vy355+4MPPqiPP/5Y8fHxatCggVq3bq34+HhVrVo137UBAAAAuD7ZzIUvrAPyKSMjQwEBARqXmC5vH39nlwMAAHBDGRrl7ApQUBd+f05PT5e/f9H//lyg2TUBAAAAANc3Qh4AAAAAWAghDwAAAAAshJAHAAAAABZCyAMAAAAACyHkAQAAAICFEPIAAAAAwEIIeQAAAABgIYQ8AAAAALAQQh4AAAAAWAghDwAAAAAshJAHAAAAABZCyAMAAAAACyHkAQAAAICFEPIAAAAAwEIIeQAAAABgIW7OLgAl16Dmkr+/s6sAAAAA8E/cyQMAAAAACyHkAQAAAICFEPIAAAAAwEIIeQAAAABgIYQ8AAAAALAQQh4AAAAAWAghDwAAAAAshJAHAAAAABZCyAMAAAAACyHkAQAAAICFEPIAAAAAwELcnF0ASq5JqyRvH2dXAQAAACsbGuXsCkoe7uQBAAAAgIUQ8gAAAADAQgh5AAAAAGAhhDwAAAAAsBBCHgAAAABYCCEPAAAAACyEkAcAAAAAFkLIAwAAAAALIeQBAAAAgIUQ8gAAAADAQgh5AAAAAGAhhDwAAAAAsBBCHgAAAABYCCEPAAAAACyEkAcAAAAAFkLIAwAAAAALIeQBAAAAgIUQ8iwiOTlZNptNx48fd3YpAAAAAJyIkHcdi42N1Z133pmvsS1atFBqaqoCAgKKZP8AAAAASgY3ZxeAwuHh4aHQ0FBnlwEAAADAybiTV0JkZWXp8ccfV3BwsLy8vNSqVSutXr3avv7ixzXj4+NVunRpLVy4UHXr1pWvr686d+6s1NRUSdKYMWOUkJCg//mf/5HNZpPNZlNycrIzTg0AAABAISLklRBPP/205syZo4SEBK1bt041atRQdHS0jh49esltTp06pfHjx+vTTz/Vjz/+qD179mjkyJGSpJEjR6pXr1724JeamqoWLVrkuZ+srCxlZGQ4vAAAAABcnwh5JcDJkyc1ceJEvfnmm+rSpYvq1aunKVOmyNvbW1OnTr3kdmfPntWkSZPUrFkzRUZGasiQIVq8eLEkydfXV97e3vL09FRoaKhCQ0Pl4eGR537i4uIUEBBgf4WHhxfJeQIAAAC4doS8EmDHjh06e/asWrZsaV/m7u6u5s2ba+vWrZfcrlSpUqpevbr9ffny5XXw4MECH3/UqFFKT0+3v/bu3VvgfQAAAAAoHky8UgIYYyRJNpst1/KLl/2Tu7u7w3ubzWbfV0F4enrK09OzwNsBAAAAKH7cySsBatSoIQ8PDy1btsy+7OzZs1qzZo3q1q171fv18PBQdnZ2YZQIAAAA4DrBnbwSwMfHR48++qieeuopBQYGqlKlSnrjjTd06tQpPfDAA1e93ypVqmjhwoXatm2bgoKCFBAQkOvuHwAAAICShZB3HcvJyZGb2/m/orFjxyonJ0f9+vVTZmammjVrpoULF6pMmTJXvf+HHnpIycnJatasmU6cOKGkpCS1adOmkKoHAAAA4Aw2czUf0kKx6Ny5s2rUqKEPPvjA2aU4yMjIUEBAgMYlpsvbx9/Z5QAAAMDChkY5u4Jrd+H35/T0dPn7F/3vz3wm7zp07NgxzZ8/X8nJyerQoYOzywEAAABQgvC45nVo4MCBWr16tZ588kndcccdzi4HAAAAQAlCyLsOzZs3z9klAAAAACiheFwTAAAAACyEkAcAAAAAFkLIAwAAAAALIeQBAAAAgIUQ8gAAAADAQgh5AAAAAGAhhDwAAAAAsBBCHgAAAABYCCEPAAAAACyEkAcAAAAAFkLIAwAAAAALIeQBAAAAgIUQ8gAAAADAQtycXQBKrkHNJX9/Z1cBAAAA4J+4kwcAAAAAFkLIAwAAAAALIeQBAAAAgIUQ8gAAAADAQgh5AAAAAGAhhDwAAAAAsBBCHgAAAABYCCEPAAAAACyEkAcAAAAAFkLIAwAAAAALcXN2ASi5Jq2SvH2cXQUAAABwaUOjnF1B8eNOHgAAAABYCCEPAAAAACyEkAcAAAAAFkLIAwAAAAALIeQBAAAAgIUQ8gAAAADAQgh5AAAAAGAhhDwAAAAAsBBCHgAAAABYCCEPAAAAACyEkAcAAAAAFkLIAwAAAAALIeQBAAAAgIUQ8gAAAADAQgh5AAAAAGAhhDwAAAAAsBBCHgAAAABYCCEPAAAAACyEkGdxKSkpstls+vXXX51dCgAAAIBiQMi7DsXGxspms2nQoEG51g0ePFg2m02xsbHFXxgAAACA6x4h7zoVHh6umTNn6u+//7YvO336tD7//HNVqlTJiZUBAAAAuJ4R8q5TkZGRqlSpkubOnWtfNnfuXIWHh6tJkyb2Zd9//71atWql0qVLKygoSN26ddOOHTsuu+8tW7aoa9eu8vX1VUhIiPr166fDhw9fcnxWVpYyMjIcXgAAAACuT4S869iAAQM0bdo0+/v//ve/GjhwoMOYkydPasSIEVq9erUWL14sFxcX3XXXXcrJyclzn6mpqWrdurUaN26sNWvW6Pvvv9eBAwfUq1evS9YRFxengIAA+ys8PLxwThAAAABAobMZY4yzi4Cj2NhYHT9+XB9//LEqVqyo33//XTabTXXq1NHevXv14IMPqnTp0oqPj8+17aFDhxQcHKyNGzcqIiJCKSkpqlq1qtavX6/GjRvrhRde0MqVK7Vw4UL7Nvv27VN4eLi2bdumWrVq5dpnVlaWsrKy7O8zMjIUHh6ucYnp8vbxL5IeAAAAAIVhaJSzKzj/+3NAQIDS09Pl71/0vz+7FfkRcNXKli2rmJgYJSQkyBijmJgYlS1b1mHMjh079Pzzz+uXX37R4cOH7Xfw9uzZo4iIiFz7XLt2rZKSkuTr65tr3Y4dO/IMeZ6envL09CykswIAAABQlAh517mBAwdqyJAhkqQPP/ww1/ru3bsrPDxcU6ZMUVhYmHJychQREaEzZ87kub+cnBx1795d48aNy7WufPnyhVs8AAAAgGJHyLvOde7c2R7YoqOjHdYdOXJEW7du1eTJk3XrrbdKkpYtW3bZ/UVGRmrOnDmqUqWK3Nz46wcAAACsholXrnOurq7aunWrtm7dKldXV4d1ZcqUUVBQkP7zn//ozz//1JIlSzRixIjL7u+xxx7T0aNH1bt3b61atUo7d+7UokWLNHDgQGVnZxflqQAAAAAoBoS8EsDf3z/PD2i6uLho5syZWrt2rSIiIvTEE0/ozTffvOy+wsLC9PPPP3pn2GAAACAASURBVCs7O1vR0dGKiIjQsGHDFBAQIBcXLgcAAACgpGN2TRTYhdmBmF0TAAAA17sbcXZNbt0AAAAAgIUQ8gAAAADAQgh5AAAAAGAhhDwAAAAAsBBCHgAAAABYCCEPAAAAACyEkAcAAAAAFkLIAwAAAAALIeQBAAAAgIUQ8gAAAADAQgh5AAAAAGAhhDwAAAAAsBBCHgAAAABYCCEPAAAAACyEkAcAAAAAFkLIAwAAAAALcXN2ASi5BjWX/P2dXQUAAACAf+JOHgAAAABYCCEPAAAAACyEkAcAAAAAFkLIAwAAAAALIeQBAAAAgIUQ8gAAAADAQgh5AAAAAGAhhDwAAAAAsBBCHgAAAABYiJuzC0DJY4yRJGVkZDi5EgAAAOD6d+H35gu/Rxc1Qh4K7MiRI5Kk8PBwJ1cCAAAAlByZmZkKCAgo8uMQ8lBggYGBkqQ9e/YUy0VqRRkZGQoPD9fevXvl7+/v7HJKJHp47ejhtaOH144eXjt6eO3o4bWjh5dnjFFmZqbCwsKK5XiEPBSYi8v5j3IGBATwQ3yN/P396eE1oofXjh5eO3p47ejhtaOH144eXjt6eGnFeXOEiVcAAAAAwEIIeQAAAABgIa5jxowZ4+wiUPK4urqqTZs2cnPjid+rRQ+vHT28dvTw2tHDa0cPrx09vHb08NrRw+uHzRTXPJ4AAAAAgCLH45oAAAAAYCGEPAAAAACwEEIeAAAAAFgIIQ8AAAAALISQhwL76KOPVLVqVXl5ealp06b66aefnF2SU4wZM0Y2m83hFRoaal9vjNGYMWMUFhYmb29vtWnTRps3b3bYx7Fjx9SvXz8FBAQoICBA/fr10/Hjxx3GbNy4Ua1bt5a3t7cqVKigl19+WSV1vqQff/xR3bt3V1hYmGw2m7766iuH9cXZszlz5qhevXry9PRUvXr1NG/evKI56UJ2pR7Gxsbmui5vueUWhzFZWVkaOnSoypYtKx8fH91+++3at2+fw5g9e/aoe/fu8vHxUdmyZfX444/rzJkzDmOWLl2qpk2bysvLS9WqVdOkSZOK5qQLUVxcnG666Sb5+fkpODhYd955p7Zt2+Ywpjj7UxL/Pc1PD9u0aZPrOrz33nsdxtzIP8sTJ05Uw4YN7V8aHRUVpQULFtjXcw1e2ZV6yDVYMHFxcbLZbBo+fLh9GddhCWeAApg5c6Zxd3c3U6ZMMVu2bDHDhg0zPj4+Zvfu3c4urdi9+OKLpn79+iY1NdX+OnjwoH392LFjjZ+fn5kzZ47ZuHGjueeee0z58uVNRkaGfUznzp1NRESEWb58uVm+fLmJiIgw3bp1s69PT083ISEh5t577zUbN240c+bMMX5+fmb8+PHFeq6F5bvvvjPPPvusmTNnjpFk5s2b57C+uHq2fPly4+rqal5//XWzdetW8/rrrxs3Nzfzyy+/FH0TrtGVeti/f3/TuXNnh+vyyJEjDmMGDRpkKlSoYBITE826detM27ZtTaNGjcy5c+eMMcacO3fOREREmLZt25p169aZxMREExYWZoYMGWLfx86dO02pUqXMsGHDzJYtW8yUKVOMu7u7mT17dtE34RpER0ebadOmmU2bNplff/3VxMTEmEqVKpkTJ07YxxRXf0rqv6f56WHr1q3NQw895HAdHj9+3GE/N/LP8tdff23mz59vtm3bZrZt22ZGjx5t3N3dzaZNm4wxXIP5caUecg3m36pVq0yVKlVMw4YNzbBhw+zLuQ5LNkIeCqR58+Zm0KBBDsvq1KljnnnmGSdV5DwvvviiadSoUZ7rcnJyTGhoqBk7dqx92enTp01AQICZNGmSMcaYLVu2GEkO/yFYsWKFkWR+//13Y4wxH330kQkICDCnT5+2j4mLizNhYWEmJyenKE6r2FwcUIqzZ7169TKdO3d2qCc6Otrce++9hX+iRehSIe+OO+645DbHjx837u7uZubMmfZlf/31l3FxcTHff/+9MeZ8kHRxcTF//fWXfcznn39uPD09TXp6ujHGmKefftrUqVPHYd+PPPKIueWWW675vIrTwYMHjSSzdOlSY0zx9scq/55e3ENjzv+C/c9fFi/Gz3JuZcqUMR9//DHX4DW40ENjuAbzKzMz09SsWdMkJiY69IzrsOTjcU3k25kzZ7R27Vp16tTJYXmnTp20fPlyJ1XlXNu3b1dYWJiqVq2qe++9Vzt37pQk7dq1S2lpaQ698vT0VOvWre29WrFihQICAnTzzTfbx9xyyy0KCAhwGNO6dWt5enrax0RHR2v//v1KSUkphjMsPsXZsxUrVuS6jqOjoy1zHScnJys4OFi1atXSQw89pIMHD9rXrV27VmfPnnU4/7CwMEVERDj0MCIiQmFhYfYx0dHRysrK0tq1a+1j8urhmjVrdPbs2aI8vUKVnp4uSQoMDJRUfP2x0r+nF/fwgunTp6ts2bKqX7++Ro4cqczMTPs6fpb/X3Z2tmbOnKmTJ08qKiqKa/AqXNzDC7gGr+yxxx5TTEyMOnTo4LCc67Dk4+vokW+HDx9Wdna2QkJCHJaHhIQoLS3NSVU5z80336xPPvlEtWrV0oEDB/Tqq6+qRYsW2rx5s70fefVq9+7dkqS0tDQFBwfn2m9wcLB9+7S0NFWpUiXXPi6sq1q1amGfltMUZ8/S0tIsex136dJFPXv2VOXKlbVr1y49//zzateundauXStPT0+lpaXJw8NDZcqUcdjun+efV3/KlCkjDw+Py44JCQnRuXPndPjwYZUvX74Iz7JwGGM0YsQItWrVShEREZJUbP0xxlji39O8eihJffr0UdWqVRUaGqpNmzZp1KhR+u2335SYmCiJn2Xp/Ge9oqKidPr0afn6+mrevHmqV6+efv31V67BfLpUDyWuwfyYOXOm1q1bp9WrV+dax7+FJR8hDwVms9kc3htjci27EXTp0sX+5wYNGigqKkrVq1dXQkKCfaKLK/Uqr75daYz5vw98W7XnxdUzq17H99xzj/3PERERatasmSpXrqz58+erR48el9zuRrw2hwwZog0bNmjZsmVXHFvY/blUr0radXipHj700EP2P0dERKhmzZpq1qyZ1q1bp8jISEn8LNeuXVu//vqrjh8/rjlz5qh///5aunTpJcdzDeZ2qR7Wq1ePa/AK9u7dq2HDhmnRokXy8vLK93ZchyUHj2si38qWLStXV9dc/2fl4MGDuf4PzI3Ix8dHDRo00Pbt2+2zbF6uV6GhoTpw4ECu/Rw6dMhhTF77kHLf8SrpirNnlxpjtZ5KUvny5VW5cmVt375d0vlzP3PmjI4dO+Yw7uI+X9yfY8eO6ezZs1fsoZubm4KCgorqdArN0KFD9fXXXyspKUkVK1a0Ly+u/ljh39NL9TAvkZGRcnd3d7gOb/SfZQ8PD9WoUUPNmjVTXFycGjVqpHfffZdrsAAu1cO8cA06Wrt2rQ4ePKimTZvKzc1Nbm5uWrp0qd577z25ubkpJCSE67CEI+Qh3zw8PNS0aVP7ow4XJCYmqkWLFk6q6vqRlZWlrVu3qnz58vZHRP7ZqzNnzmjp0qX2XkVFRSk9PV2rVq2yj1m5cqXS09Mdxvz4448O0xEvWrRIYWFhuR4hKemKs2dRUVG5ruNFixZZ8jo+cuSI9u7da398smnTpnJ3d3c4/9TUVG3atMmhh5s2bVJqaqp9zKJFi+Tp6ammTZvax+TVw2bNmsnd3b2oT+uqGWM0ZMgQzZ07V0uWLMn1yHNx9ack/3t6pR7mZfPmzTp79qz9OuRnOTdjjLKysrgGr8GFHuaFa9BR+/bttXHjRv3666/2V7NmzdSnTx/7n7kOS7gim9IFlnRhmtupU6eaLVu2mOHDhxsfHx+TkpLi7NKK3ZNPPmmSk5PNzp07zS+//GK6detm/Pz87L0YO3asCQgIMHPnzjUbN240vXv3zvPrABo2bGhWrFhhVqxYYRo0aOAwffPx48dNSEiI6d27t9m4caOZO3eu8ff3L7FfoZCZmWnWr19v1q9fbySZt956y6xfv94+TXJx9eznn382rq6uZuzYsWbr1q1m7NixJWbK68v1MDMz0zz55JNm+fLlZteuXSYpKclERUWZChUqOPRw0KBBpmLFiuaHH34w69atM+3atctzWuz27dubdevWmR9++MFUrFgxz2mxn3jiCbNlyxYzderUEvEVCo8++qgJCAgwycnJDlOrnzp1yj6muPpTUv89vVIP//zzT/PSSy+Z1atXm127dpn58+ebOnXqmCZNmth7aMyN/bM8atQo8+OPP5pdu3aZDRs2mNGjRxsXFxezaNEiYwzXYH5crodcg1fn4hlJuQ5LNkIeCuzDDz80lStXNh4eHiYyMtJh2uwbyYXvcHN3dzdhYWGmR48eZvPmzfb1OTk55sUXXzShoaHG09PT3HbbbWbjxo0O+zhy5Ijp06eP8fPzM35+fqZPnz7m2LFjDmM2bNhgbr31VuPp6WlCQ0PNmDFjSuzXJyQlJRlJuV79+/c3xhRvz2bNmmVq165t3N3dTZ06dcycOXOK9NwLy+V6eOrUKdOpUydTrlw54+7ubipVqmT69+9v9uzZ47CPv//+2wwZMsQEBgYab29v061bt1xjdu/ebWJiYoy3t7cJDAw0Q4YMcZhG3BhjkpOTTZMmTYyHh4epUqWKmThxYpGf/7XKq3eSzLRp0+xjirM/JfHf0yv1cM+ePea2224zgYGBxsPDw1SvXt08/vjjub6v8Ub+WR44cKD9771cuXKmffv29oBnDNdgflyuh1yDV+fikMd1WLLZjPm/TzwCAAAAAEo8PpMHAAAAABZCyAMAAAAACyHkAQAAAICFEPIAAAAAwEIIeQAAAABgIYQ8AAAAALAQQh4AAAAAWAghDwAAAAAshJAHAAAAABZCyAMAAAAACyHkAQAAAICFEPIAAAAAwEIIeQAAAABgIYQ8AAAAALAQQh4AAAAAWAghDwAAAAAshJAHAAAAABZCyAMAAIXmo48+Unx8vLPLAIAbms0YY5xdBAAAsIaIiAiVLVtWycnJzi4FAG5Y3MkDAAAAAAsh5AEALGXZsmVq3769/Pz8VKpUKbVo0ULz5893GPPXX3/p4YcfVnh4uDw8PBQWFqa7775bBw4csI85fvy4nnzySVWrVk2enp4KDg5W165d9fvvv0uSkpOTZbPZct2xSklJkc1mc3hkMTY2Vr6+vtq8ebPat28vHx8flStXTkOGDNGpU6cctv/www912223KTg4WD4+PmrQoIHeeOMNnT171mFcmzZtFBERodWrV+vWW29VqVKlVK1aNY0dO1Y5OTkOYy93LsYY1axZU9HR0bl6eeLECQUEBOixxx7LV++rVKmizZs3a+nSpbLZbLLZbKpSpYpOnDih0qVL65FHHsm1TUpKilxdXfXmm29KkuLj42Wz2ZSYmKgBAwYoMDBQPj4+6t69u3bu3Jlr+x9++EHt27eXv7+/SpUqpZYtW2rx4sX5qhcArIqQBwCwjKVLl6pdu3ZKT0/X1KlT9fnnn8vPz0/du3fXF198Iel8wLvppps0b948jRgxQgsWLNA777yjgIAAHTt2TJKUmZmpVq1aafLkyRowYIC++eYbTZo0SbVq1VJqaupV1Xb27Fl17dpV7du311dffaUhQ4Zo8uTJuueeexzG7dixQ/fdd58+/fRTffvtt3rggQf05ptv5hmQ0tLS1KdPH/Xt21dff/21unTpolGjRumzzz6zj7nSudhsNg0dOlSJiYnavn27w/4/+eQTZWRk5DvkzZs3T9WqVVOTJk20YsUKrVixQvPmzZOvr68GDhyo6dOnKz093WGbjz76SB4eHho4cKDD8gceeEAuLi6aMWOG3nnnHa1atUpt2rTR8ePH7WM+++wzderUSf7+/kpISNCXX36pwMBARUdHE/QA3NgMAAAWccstt5jg4GCTmZlpX3bu3DkTERFhKlasaHJycszAgQONu7u72bJlyyX38/LLLxtJJjEx8ZJjkpKSjCSTlJTksHzXrl1Gkpk2bZp9Wf/+/Y0k8+677zqMfe2114wks2zZsjyPkZ2dbc6ePWs++eQT4+rqao4ePWpf17p1ayPJrFy50mGbevXqmejo6AKdS0ZGhvHz8zPDhg3Lta+2bdtecru81K9f37Ru3TrX8h07dhgXFxfz9ttv25f9/fffJigoyAwYMMC+bNq0aUaSueuuuxy2//nnn40k8+qrrxpjjDl58qQJDAw03bt3dxiXnZ1tGjVqZJo3b16gugHASriTBwCwhJMnT2rlypW6++675evra1/u6uqqfv36ad++fdq2bZsWLFigtm3bqm7dupfc14IFC1SrVi116NChUGvs06ePw/v77rtPkpSUlGRftn79et1+++0KCgqSq6ur3N3ddf/99ys7O1t//PGHw/ahoaFq3ry5w7KGDRtq9+7dBToXPz8/DRgwQPHx8Tp58qQkacmSJdqyZYuGDBlydSd7kWrVqqlbt2766KOPZP5vzrcZM2boyJEjeR7j4l61aNFClStXtvdq+fLlOnr0qPr3769z587ZXzk5OercubNWr15tPxcAuNEQ8gAAlnDs2DEZY1S+fPlc68LCwiRJR44c0aFDh1SxYsXL7is/YwrKzc1NQUFBDstCQ0PtdUnSnj17dOutt+qvv/7Su+++q59++kmrV6/Whx9+KEn6+++/Hba/eH+S5Onp6TAuv+cydOhQZWZmavr06ZKkDz74QBUrVtQdd9xRgLO8vGHDhmn79u1KTEyUdP7zh1FRUYqMjMw19kJvLl52oVcXPj959913y93d3eE1btw4GWN09OjRQqsdAEoSN2cXAABAYShTpoxcXFzy/Mzc/v37JUlly5ZVuXLltG/fvsvuKz9jvLy8JElZWVkOyw8fPpzn+HPnzunIkSMOwSwtLU3S/4e1r776SidPntTcuXNVuXJl+7hff/31srVcTn7ORZJq1KihLl266MMPP1SXLl309ddf66WXXpKrq+tVH/ti7dq1U0REhD744AP5+vpq3bp1Dp8f/KcLvbl4WY0aNSSd/7uUpPfff1+33HJLnvsICQkppMoBoGThTh4AwBJ8fHx08803a+7cuQ53snJycvTZZ5+pYsWKqlWrlrp06aKkpCRt27btkvvq0qWL/vjjDy1ZsuSSY6pUqSJJ2rBhg8Pyr7/++pLbXLhLdsGMGTMknZ8pU5JsNpuk83fjLjDGaMqUKZfc55Xk51wuGDZsmDZs2KD+/fvL1dVVDz30UIGPd/GdxIs9/vjjmj9/vkaNGqWQkBD17Nkzz3EX92r58uXavXu3vVctW7ZU6dKltWXLFjVr1izPl4eHR4HrBwAr4E4eAMAy4uLi1LFjR7Vt21YjR46Uh4eHPvroI23atEmff/65bDabXn75ZS1YsEC33XabRo8erQYNGuj48eP6/vvvNWLECNWpU0fDhw/XF198oTvuuEPPPPOMmjdvrr///ltLly5Vt27d1LZtW4WGhqpDhw6Ki4tTmTJlVLlyZS1evFhz587NszYPDw9NmDBBJ06c0E033aTly5fr1VdfVZcuXdSqVStJUseOHeXh4aHevXvr6aef1unTpzVx4kT7rJ9XIz/nckHHjh1Vr149JSUlqW/fvgoODi7w8Ro0aKCZM2fqiy++ULVq1eTl5aUGDRrY1/ft21ejRo3Sjz/+qOeee+6SQWzNmjV68MEH1bNnT+3du1fPPvusKlSooMGDB0uSfH199f7776t///46evSo7r77bgUHB+vQoUP67bffdOjQIU2cOLHA9QOAJTh33hcAAArXTz/9ZNq1a2d8fHyMt7e3ueWWW8w333zjMGbv3r1m4MCBJjQ01Li7u5uwsDDTq1cvc+DAAfuYY8eOmWHDhplKlSoZd3d3ExwcbGJiYszvv/9uH5OammruvvtuExgYaAICAkzfvn3NmjVr8pxd08fHx2zYsMG0adPGeHt7m8DAQPPoo4+aEydOONT2zTffmEaNGhkvLy9ToUIF89RTT5kFCxbkmsmzdevWpn79+rnOv3///qZy5coOy/JzLheMGTPGSDK//PJLftqdS0pKiunUqZPx8/MzknLVYowxsbGxxs3Nzezbty/Xuguzay5atMj069fPlC5d2nh7e5uuXbua7du35xq/dOlSExMTYwIDA427u7upUKGCiYmJMbNmzbqq+gHACmzG/N8UVwAAoEjExsZq9uzZOnHihLNLuaJmzZrJZrNp9erVRbL/M2fOqEqVKmrVqpW+/PLLXOvj4+M1YMAArV69Ws2aNSuSGgDA6nhcEwCAG1xGRoY2bdqkb7/9VmvXrtW8efMK/RiHDh3Stm3bNG3aNB04cEDPPPNMoR8DAHAeIQ8FlpOTo/3798vPz88+SQAA4NLOnj0r6XyYuh799NNP6tatmwIDA/XMM8+oXbt2uWrNzs7W5R7+sdlsl52Jc/bs2Ro8eLBCQ0M1YcIE1ahRI89+XJi05cSJE9dtvwCgoIwxyszMVFhYmFxcin7uSx7XRIHt27dP4eHhzi4DAAAAKFH27t1b6N/Dmhfu5KHA/Pz8JJ2/SP39/Z1cDQCgOGzevDnXdwL+k5+fn2rWrFmMFQFAyZGRkaHw8HD779FFjZCHArvwiKa/vz8hDwBuEFFRUc4uAQBKvOL6qBNfhg4AAAAAFkLIAwAAAAALIeQBAAAAgIUQ8gAAAADAQgh5AAAAAGAhhDwAAAAAsBBCHgAAAABYCCEPAAAAACyEL0PHVZu0SvL2ubpth/KdugAAAECR4E4eAAAAAFgIIQ8AAAAALISQBwAAAAAWQsgDAAAAAAsh5AEAAACAhRDyAAAAAMBCCHkAAAAAYCGEPAAAAACwEEIeAAAAAFgIIQ8AAAAALISQBwAAAAAWQsgDAAAAAAsh5AEAAACAhRDyAAAAAMBCCHkAAAAAYCGEPAAAAACwEEIeAAAAAFgIIQ8AAAAALISQBwAAAAAWQsgDAAAAAAsh5AEAAACAhRDyAAAAAMBCCHkAAAAAYCGEPAAAAACwEEIeAAAAAFgIIQ8AAAAALISQBwAAAAAWQsgDAAAAAAsh5MHuzJkzzi4BAAAAwDUi5N0AZs+erQYNGsjb21tBQUHq0KGDTp48qdjYWN15552Ki4tTWFiYatWq5exSAQAAAFwjN2cXgKKVmpqq3r1764033tBdd92lzMxM/fTTTzLGSJIWL14sf39/JSYm2pddLCsrS1lZWfb3GRkZxVI7AAAAgIIj5Flcamqqzp07px49eqhy5cqSpAYNGtjX+/j46OOPP5aHh8cl9xEXF6eXXnqpyGsFAAAAcO14XNPiGjVqpPbt26tBgwbq2bOnpkyZomPHjtnXN2jQ4LIBT5JGjRql9PR0+2vv3r1FXTYAAACAq0TIszhXV1clJiZqwYIFqlevnt5//33Vrl1bu3btknT+Tt6VeHp6yt/f3+EFAAAA4PpEyLsB2Gw2tWzZUi+99JLWr18vDw8PzZs3z9llAQAAACgCfCbP4lauXKnFixerU6dOCg4O1sqVK3Xo0CHVrVtXGzZscHZ5AAAAAAoZd/Iszt/fXz/++KO6/i97dx5lVXXmDfgtqqAohioUuhhLQCNhKI0IDoAKgq0g0STaURBFWptIf0JwSpRoPgxtgy7MoFmK0RjUSDStOJCYiCBCTEBF0MiQIE4RGYKgVEFCSqDO94eft70yG6hbHJ5nrbOW55x9933P2SL35z533zPPjA4dOsQNN9wQ3//+92PAgAG5Lg0AANgPzOSlXKdOneLpp5/e4bn77ruvZosBAAD2OzN5AAAAKSLkAQAApIiQBwAAkCJCHgAAQIoIeQAAACki5AEAAKSIkAcAAJAiQh4AAECKCHkAAAApIuQBAACkiJAHAACQIkIeAABAigh5AAAAKSLkAQAApIiQBwAAkCJCHgAAQIoIeQAAACki5AEAAKSIkAcAAJAiQh4AAECKCHkAAAApIuQBAACkiJAHAACQIkIeAABAihTkugAOXCOOjyguznUVAADAp5nJAwAASBEhDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEgRIQ8AACBFhDwAAIAUEfIAAABSRMgDAABIESEPAAAgRYQ8AACAFBHyAAAAUkTIAwAASBEhDwAAIEWEPAAAgBQpyHUBHLjueimiqGGuq/hfo3rkugIAAMg9M3kAAAApIuQBAACkiJAHAACQIkIeAABAigh5AAAAKSLkAQAApIiQBwAAkCJCHgAAQIoIeQAAACki5AEAAKSIkAcAAJAiQh4AAECKCHkAAAApIuQBAACkiJAHAACQIkIeAABAigh5AAAAKSLkAQAApIiQBwAAkCJCHgAAQIoIeQAAACki5AEAAKSIkAcAAJAiQh4AAECKCHkAAAApIuQBAACkiJB3kOnTp09cccUVuS4DAADYTwpyXQA167HHHou6devmugwAAGA/EfIOMoceeuhOz3300UdRr169GqwGAADY1zyueZD59OOa7dq1i5tuuimGDRsWJSUlMXz48B2+pqqqKiorK7M2AACgdhLyDnITJ06M8vLyWLBgQXz3u9/dYZsJEyZESUlJZisrK6vhKgEAgD0l5B3k+vbtG9dcc0184QtfiC984Qs7bDNmzJioqKjIbCtWrKjhKgEAgD3lO3kHue7du++2TWFhYRQWFtZANQAAwD/LTN5BrmHDhrkuAQAA2IeEPAAAgBQR8gAAAFJEyAMAAEgRC68cZGbPnp3553feeSdndQAAAPuHmTwAAIAUEfIAAABSRMgDAABIESEPAAAgRYQ8AACAFBHyAAAAUkTIAwAASBEhDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEgRIQ8AACBFhDwAAIAUEfIAAABSRMgDAABIESEPAAAgRYQ8AACAFBHyAAAAUkTIAwAASBEhDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEiRglwXwIFrxPERUN2Y7gAAIABJREFUxcW5rgIAAPg0M3kAAAApIuQBAACkiJAHAACQIkIeAABAigh5AAAAKSLkAQAApIiQBwAAkCJCHgAAQIoIeQAAACki5AEAAKSIkAcAAJAiQh4AAECKCHkAAAApIuQBAACkiJAHAACQIgW5LoAD110vRRQ1zHUVAACwc6N65LqCmmcmDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEgRIQ8AACBFhDwAAIAUEfIAAABSRMgDAABIESEPAAAgRYQ8AACAFBHyAAAAUkTIAwAASBEhDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEgRIQ8AACBFhDwAAIAUEfIAAABSRMgDAABIESEPAAAgRYQ8AACAFBHyAAAAUkTIAwAASBEhDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEgRIY+Mjz76KNclAAAA/yQhL8X69OkTI0eOjJEjR0aTJk2iadOmccMNN0SSJBER0a5du7jpppti2LBhUVJSEsOHD99hP1VVVVFZWZm1AQAAtZOQl3L3339/FBQUxIsvvhi33357/PCHP4yf/vSnmfMTJ06M8vLyWLBgQXz3u9/dYR8TJkyIkpKSzFZWVlZT5QMAAHspL/lkWofU6dOnT6xduzaWLFkSeXl5ERFx3XXXxbRp02Lp0qXRrl276Nq1azz++OO77Keqqiqqqqoy+5WVlVFWVha3zKiIoobF+/UaAADgnzGqR64r+Pjzc0lJSVRUVERx8f7//GwmL+VOPPHETMCLiOjRo0csX748tm3bFhER3bt3320fhYWFUVxcnLUBAAC1k5B3kGvYsGGuSwAAAPYhIS/lXnjhhe32jzzyyMjPz89RRQAAwP4k5KXcihUr4qqrroply5bFQw89FD/+8Y9j9OjRuS4LAADYTwpyXQD719ChQ2Pz5s1x/PHHR35+fowaNSq+8Y1v5LosAABgPxHyUq5u3brxox/9KCZNmrTduXfeeafmCwIAAPYrj2sCAACkiJAHAACQIh7XTLHZs2fnugQAAKCGmckDAABIESEPAAAgRYQ8AACAFBHyAAAAUkTIAwAASBEhDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEgRIQ8AACBFhDwAAIAUEfIAAABSRMgDAABIESEPAAAgRYQ8AACAFBHyAAAAUkTIAwAASBEhDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEiRglwXwIFrxPERxcW5rgIAAPg0M3kAAAApIuQBAACkiJAHAACQIkIeAABAigh5AAAAKSLkAQAApIiQBwAAkCJCHgAAQIoIeQAAAClSkOsCOPAkSRIREZWVlTmuBAAAar9PPjd/8jl6fxPy2Gvr16+PiIiysrIcVwIAAAeOjRs3RklJyX5/HyGPvXbooYdGRMS7775bI/+SsmuVlZVRVlYWK1asiOLi4lyXc1AzFrWHsag9jEXtYSxqD2NRu9TEeCRJEhs3boxWrVrtl/4/S8hjr9Wp8/FXOUtKSvyHqRYpLi42HrWEsag9jEXtYSxqD2NRexiL2mV/j0dNTo5YeAUAACBFhDwAAIAUyb/xxhtvzHURHHjy8/OjT58+UVDgid/awHjUHsai9jAWtYexqD2MRe1hLGqXtI1HXlJT63gCAACw33lcEwAAIEWEPAAAgBQR8gAAAFJEyAMAAEgRIY+9duedd0b79u2jfv360a1bt3j++edzXdIBZcKECXHcccdF48aNo7S0NL761a/GsmXLstpUVVXFqFGjolmzZtGwYcM4++yz47333stq8+6778ZZZ50VDRs2jGbNmsU3v/nN+Oijj7LazJkzJ7p16xb169ePww8/PO66667t6jGe/2vChAmRl5cXV1xxReaYsag5K1eujAsvvDCaNm0aDRo0iGOOOSYWLFiQOZ8kSdx4443RqlWrKCoqij59+sSSJUuy+vjwww/joosuipKSkigpKYmLLrooNmzYkNVm0aJF0bt37ygqKorWrVvHuHHj4rNrkE2dOjU6d+4chYWF0blz53j88cf334XXMlu3bo0bbrgh2rdvH0VFRXH44YfHuHHjorq6OtPGWOwfv/vd7+Kss86KVq1aRV5eXjzxxBNZ52vTfd+TWg50uxqPLVu2xLXXXhtHHXVUNGzYMFq1ahVDhw6NVatWZfVhPPaN3f3Z+LTLLrss8vLy4kc/+lHW8YNuLBLYCw8//HBSt27d5J577kmWLl2ajB49OmnYsGHyl7/8JdelHTDOOOOMZPLkycnixYuTV199NRk4cGBy2GGHJZs2bcq0GTFiRNK6detkxowZycKFC5NTTz01+dKXvpRs3bo1SZIk2bp1a1JeXp6ceuqpycKFC5MZM2YkrVq1SkaOHJnp46233koaNGiQjB49Olm6dGlyzz33JHXr1k0effTRTBvj+b9eeumlpF27dsnRRx+djB49OnPcWNSMDz74IGnbtm0ybNiw5MUXX0zefvvtZObMmckbb7yRaXPzzTcnjRs3TqZOnZosWrQoOf/885OWLVsmlZWVmTb9+/dPysvLk7lz5yZz585NysvLky9/+cuZ8xUVFUnz5s2TQYMGJYsWLUqmTp2aNG7cOLn11lszbebOnZvk5+cn48ePT/70pz8l48ePTwoKCpIXXnihZm5Gjt10001J06ZNk1//+tfJ22+/nTzyyCNJo0aNkh/96EeZNsZi//jNb36TXH/99cnUqVOTiEgef/zxrPO16b7vSS0Hul2Nx4YNG5LTTjst+eUvf5n8+c9/TubNm5eccMIJSbdu3bL6MB77xu7+bHzi8ccfT770pS8lrVq1Sn74wx9mnTvYxkLIY68cf/zxyYgRI7KOdezYMbnuuutyVNGBb+3atUlEJHPmzEmS5OO/OOrWrZs8/PDDmTYrV65M6tSpkzz99NNJknz8H7s6deokK1euzLR56KGHksLCwqSioiJJkiT59re/nXTs2DHrvS677LLkxBNPzOwbz49t3LgxOfLII5MZM2YkvXv3zoQ8Y1Fzrr322uSkk07a6fnq6uqkRYsWyc0335w59o9//CMpKSlJ7rrrriRJkmTp0qVJRGT9ZTtv3rwkIpI///nPSZIkyZ133pmUlJQk//jHPzJtJkyYkLRq1Sqprq5OkiRJzjvvvKR///5Z73/GGWckgwYN+ucv9AAwcODA5JJLLsk6ds455yQXXnhhkiTGoqZ89oNsbbrve1JL2uwqWHzipZdeSiIi8z/njMf+sbOxeO+995LWrVsnixcvTtq2bZsV8g7GsfC4Jnvso48+igULFsTpp5+edfz000+PuXPn5qiqA19FRUVERBx66KEREbFgwYLYsmVL1n1u1apVlJeXZ+7zvHnzory8PFq1apVpc8YZZ0RVVVXm8bZ58+ZtN1ZnnHFGvPzyy7Flyxbj+SmXX355DBw4ME477bSs48ai5kybNi26d+8eX//616O0tDS6du0a99xzT+b822+/HWvWrMm6R4WFhdG7d++ssSgpKYkTTjgh0+bEE0+MkpKSrDa9e/eOwsLCTJszzjgjVq1aFe+8806mzY7G62AZi5NOOimeffbZeP311yMi4o9//GP8/ve/jzPPPDMijEWu1Kb7vie1HIwqKioiLy8vmjRpEhHGoyZVV1fHRRddFN/61reiS5cu250/GMdCyGOPrVu3LrZt2xbNmzfPOt68efNYs2ZNjqo6sCVJEldddVWcdNJJUV5eHhERa9asiXr16sUhhxyS1fbT93nNmjXbjcMhhxwS9erV22Wb5s2bx9atW2PdunXG8/97+OGHY+HChTFhwoTtzhmLmvPWW2/FpEmT4sgjj4zp06fHiBEj4pvf/GY88MADERGZ+7Cre7RmzZooLS3dru/S0tLdjsWn32NnbQ6Wsbj22mtj8ODB0bFjx6hbt2507do1rrjiihg8eHBEGItcqU33fU9qOdj84x//iOuuuy4uuOCCKC4ujgjjUZNuueWWKCgoiG9+85s7PH8wjkVBjb4bqZCXl5e1nyTJdsfYMyNHjozXXnstfv/73++27Wfv847u+e7aJP//y8N5eXlZ/7yrPtJsxYoVMXr06HjmmWeifv36e/w6Y7HvVVdXR/fu3WP8+PEREdG1a9dYsmRJTJo0KYYOHZppt7t79M+OxZ6+T5r98pe/jAcffDB+8YtfRJcuXeLVV1+NK664Ilq1ahUXX3xxpp2xyI3adN+Nzce2bNkSgwYNiurq6rjzzjuzzhmP/W/BggVx2223xcKFC3d5vQfbWJjJY481a9Ys8vPzt/s/EWvXrt3u/1iwe6NGjYpp06bFc889F23atMkcb9GiRXz00Ufx4YcfZrX/9H1u0aLFduPw4YcfxpYtW3bZZu3atVFQUBBNmzY1nvHxXwxr166Nbt26RUFBQRQUFMScOXPi9ttvj4KCgmjevLmxqCEtW7aMzp07Zx3r1KlTvPvuuxHx8T2MiF3eoxYtWsRf//rX7fp+//33dzsWEbHbNgfLWHzrW9+K6667LgYNGhRHHXVUXHTRRXHllVdmZruNRW7Upvu+J7UcLLZs2RLnnXdevP322zFjxozMLF6E8agpzz//fKxduzYOO+ywzN/lf/nLX+Lqq6+Odu3aRcTBORZCHnusXr160a1bt5gxY0bW8RkzZkTPnj1zVNWBJ0mSGDlyZDz22GMxa9asaN++fdb5bt26Rd26dbPu8+rVq2Px4sWZ+9yjR49YvHhxrF69OtPmmWeeicLCwujWrVumzWfH6plnnonu3btH3bp1jWdE9OvXLxYtWhSvvvpqZuvevXsMGTIk88/Gomb06tVru58Sef3116Nt27YREdG+ffto0aJF1j366KOPYs6cOVljUVFRES+99FKmzYsvvhgVFRVZbX73u99l/cTFM888E61atcp8GNjZeB0sY/H3v/896tTJ/niQn5+f+QkFY5Ebtem+70ktB4NPAt7y5ctj5syZ0bRp06zzxqNmXHTRRfHaa69l/V3eqlWr+Na3vhXTp0+PiIN0LPb3yi6kyyfLvN97773J0qVLkyuuuCJp2LBh8s477+S6tAPGf/7nfyYlJSXJ7Nmzk9WrV2e2v//975k2I0aMSNq0aZPMnDkzWbhwYdK3b98dLtvfr1+/ZOHChcnMmTOTNm3a7HDZ/iuvvDJZunRpcu+99+502X7j+b8+vbpmkhiLmvLSSy8lBQUFyX//938ny5cvT6ZMmZI0aNAgefDBBzNtbr755qSkpCR57LHHkkWLFiWDBw/e4fLxRx99dDJv3rxk3rx5yVFHHZW1RPaGDRuS5s2bJ4MHD04WLVqUPPbYY0lxcXHWEtl/+MMfkvz8/OTmm29O/vSnPyU333xzqpft/6yLL744ad26deYnFB577LGkWbNmybe//e1MG2Oxf2zcuDF55ZVXkldeeSWJiOQHP/hB8sorr2RWa6xN931PajnQ7Wo8tmzZkpx99tlJmzZtkldffTXr7/OqqqpMH8Zj39jdn43P+uzqmkly8I2FkMdeu+OOO5K2bdsm9erVS4499tjM0v/smYjY4TZ58uRMm82bNycjR45MDj300KSoqCj58pe/nLz77rtZ/fzlL39JBg4cmBQVFSWHHnpoMnLkyKxlf5MkSWbPnp107do1qVevXtKuXbtk0qRJ29VjPLN9NuQZi5rzq1/9KikvL08KCwuTjh07JnfffXfW+erq6mTs2LFJixYtksLCwuSUU05JFi1alNVm/fr1yZAhQ5LGjRsnjRs3ToYMGZJ8+OGHWW1ee+215OSTT04KCwuTFi1aJDfeeGNmeexPPPLII8kXv/jFpG7duknHjh2TqVOn7p+LroUqKyuT0aNHJ4cddlhSv3795PDDD0+uv/76rA+uxmL/eO6553b498PFF1+cJEntuu97UsuBblfj8fbbb+/07/Pnnnsu04fx2Dd292fjs3YU8g62schLks/8jDsAAAAHLN/JAwAASBEhDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEgRIQ8AACBFhDwAAIAUEfIAAABSRMgDAABIESEPAAAgRYQ8AACAFBHyAAAAUkTIAwAASBEhDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEgRIQ8AACBFhDwAAIAUEfIAAABSRMgDgFrojTfeiH//93+PI488Mho0aBCtW7eOs846KxYtWrRd2yVLlsTpp58eDRo0iH/5l3+Jyy+/PJ566qnIy8uL2bNnZ7WdOXNm9OvXL4qLi6NBgwbRq1evePbZZ2voqgCoCUIeANRCq1atiqZNm8bNN98cTz/9dNxxxx1RUFAQJ5xwQixbtizTbvXq1dG7d+9YtmxZTJo0KR544IHYuHFjjBw5crs+H3zwwTj99NOjuLg47r///vif//mfOPTQQ+OMM84Q9ABSJC9JkiTXRQAAu7Zt27aorq6OLl26xJe//OX4wQ9+EBER3/72t+PWW2+NxYsXR+fOnTPt+/fvH9OnT4/nnnsu+vTpE3//+9+jrKwsevXqFdOmTcu0q66ujmOPPTYKCwvjxRdfrPHrAmDfM5MHALXQ1q1bY/z48dG5c+eoV69eFBQURL169WL58uXxpz/9KdNuzpw5UV5enhXwIiIGDx6ctT937tz44IMP4uKLL46tW7dmturq6ujfv3/Mnz8//va3v9XItQGwfxXkugAAYHtXXXVV3HHHHXHttddG796945BDDok6derEf/zHf8TmzZsz7davXx/t27ff7vXNmzfP2v/rX/8aERH/9m//ttP3/OCDD6Jhw4b76AoAyBUhDwBqoQcffDCGDh0a48ePzzq+bt26aNKkSWa/adOmmQD3aWvWrMnab9asWURE/PjHP44TTzxxh+/52WAIwIFJyAOAWigvLy8KCwuzjj311FOxcuXK+MIXvpA51rt377j11ltj6dKlWY9sPvzww1mv7dWrVzRp0iSWLl26w0VZAEgPIQ8AaqEvf/nLcd9990XHjh3j6KOPjgULFsTEiROjTZs2We2uuOKK+NnPfhYDBgyIcePGRfPmzeMXv/hF/PnPf46IiDp1Pv76faNGjeLHP/5xXHzxxfHBBx/Ev/3bv0VpaWm8//778cc//jHef//9mDRpUo1fJwD7noVXAKAWuu222+LCCy+MCRMmxFlnnRXTpk2Lxx57LI444oisdq1atYo5c+ZEhw4dYsSIETFkyJCoV69ejBs3LiIi69HOCy+8MJ577rnYtGlTXHbZZXHaaafF6NGjY+HChdGvX78avT4A9h8/oQAAKfSNb3wjHnrooVi/fn3Uq1cv1+UAUIM8rgkAB7hx48ZFq1at4vDDD49NmzbFr3/96/jpT38aN9xwg4AHcBAS8gDgAFe3bt2YOHFivPfee7F169Y48sgj4wc/+EGMHj0616UBkAMe12SvVVdXx6pVq6Jx48aRl5eX63IAAKBWS5IkNm7cGK1atcosiLU/mcljr61atSrKyspyXQYAABxQVqxYsd0qyfuDkMdea9y4cUR8/C9pcXFxjqsBAIDarbKyMsrKyjKfo/c3IY+99skjmsXFxUIeAADsoZr6qpPfyQMAAEgRIQ8AACBFhDwAAIAUEfIAAABSRMgDAABIESEPAAAgRYQ8AACAFBHyAAAAUsSPofO53fVSRFHD/f8+o3rs//cAAIC0MJMHAACQIkIeAABAigh5AAAAKSLkAQAApIiQBwAAkCJCHgAAQIoIeQAAACki5AEAAKSIkJci77zzTlx66aXRvn37KCoqiiOOOCLGjh0bH330UVabvLy87bann346h5UDAAD7SkGuC+Cf9+GHH0bdunXjz3/+c1RXV8dPfvKT+MIXvhCLFy+O4cOHx9/+9re49dZbs14zc+bM6NKlS2b/0EMPremyAQCA/UDIO0Bt3bo1pk+fHvfff39MmzYtXnzxxejfv3/0798/0+bwww+PZcuWxaRJk7YLeU2bNo0WLVrUdNkAAMB+5nHNA8yiRYvimmuuiTZt2sTQoUOjadOm8dxzz8WXvvSlHbavqKjY4Szd2WefHaWlpdGrV6949NFHd/meVVVVUVlZmbUBAAC1k5B3AFi/fn3cfvvtceyxx0b37t3jjTfeiDvvvDNWr14dkyZNih49euzwdW+++Wb8+Mc/jhEjRmSONWrUKH7wgx/Eo48+Gr/5zW+iX79+cf7558eDDz640/efMGFClJSUZLaysrJ9fo0AAMC+kZckSZLrIti1G2+8Mb73ve/FySefHFOmTNmjkLVq1aro3bt39O7dO37605/usu2oUaNizpw58dprr+3wfFVVVVRVVWX2Kysro6ysLG6ZURFFDYv37mI+h1E7zrAAAHBAqKysjJKSkqioqIji4v3/+dlM3gHgG9/4Rtx0002xZs2a6Ny5cwwbNiyeffbZqK6u3mH7VatWxamnnho9evSIu+++e7f9n3jiibF8+fKdni8sLIzi4uKsDQAAqJ2EvANAq1at4vrrr4/XX389pk+fHoWFhXHuuedG27Zt47rrroslS5Zk2q5cuTL69OkTxx57bEyePDnq1Nn9EL/yyivRsmXL/XkJAABADbG65gGmZ8+e0bNnz7jtttviiSeeiPvvvz9uvfXWeOWVV6Jp06bRp0+fOOyww+LWW2+N999/P/O6T1bSvP/++6Nu3brRtWvXqFOnTvzqV7+K22+/PW655ZZcXRIAALAPCXkHqPr168egQYNi0KBBsWrVqmjUqFE89thj8cYbb8Qbb7wRbdq0yWr/6a9e3nTTTfGXv/wl8vPzo0OHDvGzn/0sLrzwwpq+BAAAYD+w8Ap77ZMvjlp4BQAAds/CKwAAAHxuQh4AAECKCHkAAAApIuQBAACkiJAHAACQIn5Cgc9txPERNbA4EAAAsBfM5AEAAKSIkAcAAJAiQh4AAECKCHkAAAApIuQBAACkiJAHAACQIn5Cgc/trpciihrmugoOJKN65LoCAID0M5MHAACQIkIeAABAigh5AAAAKSLkAQAApIiQBwAAkCJCHgAAQIoIeQAAACki5AEAAKSIkJcis2fPjry8vB1u8+fP3679G2+8EY0bN44mTZrkoFoAAGB/EPIOAKtWrYqtW7futl3Pnj1j9erVWdt//Md/RLt27aJ79+5Zbbds2RKDBw+Ok08+eX+VDQAA5ICQdwC45557ok2bNnH11VfHokWLdtquXr160aJFi8zWtGnTmDZtWlxyySWRl5eX1faGG26Ijh07xnnnnbe/ywcAAGqQkHcAuPbaa+P222+PZcuWxbHHHhvHHnts3HbbbfH+++/v8nXTpk2LdevWxbBhw7KOz5o1Kx555JG444479uj9q6qqorKyMmsDAABqJyHvAFC/fv0477zz4te//nWsXLkyhg4dGvfff3+0bt06vvrVr8bjjz++w8c577333jjjjDOirKwsc2z9+vUxbNiwuO+++6K4uHiP3n/ChAlRUlKS2T7dHwAAULsIeQeY0tLSuOKKK2LhwoXx5JNPxrx58+Kcc86JxYsXZ7V77733Yvr06XHppZdmHR8+fHhccMEFccopp+zxe44ZMyYqKioy24oVK/bJtQAAAPuekHeA2bhxY0yePDn69u0bZ511VpSXl8f9998fnTt3zmo3efLkaNq0aZx99tlZx2fNmhW33nprFBQUREFBQVx66aVRUVERBQUF8bOf/WyH71lYWBjFxcVZGwAAUDsV5LoAdm/btm3xzDPPxM9//vN44oknok2bNjF06NC477774rDDDtuufZIkMXny5Bg6dGjUrVs369y8efNi27Ztmf0nn3wybrnllpg7d260bt16v18LAACwfwl5B4Dx48fH97///TjvvPNi5syZ0bNnz122nzVrVrz99tvbPaoZEdGpU6es/Zdffjnq1KkT5eXl+7RmAAAgN4S8A8BFF10U3/rWt6J+/fp71P7ee++Nnj17bhfoAACA9MtLkiTJdREcWCorK6OkpCRumVERRQ19P489N6pHrisAAKh5n3x+rqioqJH1LSy8AgAAkCJCHgAAQIoIeQAAACki5AEAAKSIkAcAAJAiQh4AAECK+J08PrcRx0fUwAqwAADAXjCTBwAAkCJCHgAAQIoIeQAAACki5AEAAKSIkAcAAJAiVtfkc7vrpYiihrmuAoD9YVSPXFcAwOdlJg8AACBFhDwAAIAUEfIAAABSRMgDAABIESEPAAAgRYQ8AACAFBHyAAAAUkTIAwAASBEhDwAAIEWEvByZMGFCHHfccdG4ceMoLS2Nr371q7Fs2bKsNn369Im8vLysbdCgQbvsd/369dG/f/9o1apVFBYWRllZWYwcOTIqKyt32P4Pf/hDFBQUxDHHHLPPrg0AAMgdIS9H5syZE5dffnm88MILMWPGjNi6dWucfvrp8be//S2r3fDhw2P16tWZ7Sc/+cku+61Tp0585StfiWnTpsXrr78e9913X8ycOTNGjBixXduKiooYOnRo9OvXb59eGwAAkDsFuS7gYPX0009n7U+ePDlKS0tjwYIFccopp2SON2jQIFq0aLHH/R5yyCHxn//5n5n9tm3bxv/5P/8nJk6cuF3byy67LC644ILIz8+PJ554Yqd9VlVVRVVVVWZ/Z7OCAABA7pnJqyUqKioiIuLQQw/NOj5lypRo1qxZdOnSJa655prYuHHjXvW7atWqeOyxx6J3795ZxydPnhxvvvlmjB07drd9TJgwIUpKSjJbWVnZXtUAAADUHCGvFkiSJK666qo46aSTory8PHN8yJAh8dBDD8Xs2bPju9/9bkydOjXOOeecPepz8ODB0aBBg2jdunUUFxfHT3/608y55cuXx3XXXRdTpkyJgoLdT+aOGTMmKioqMtuKFSv2/iIBAIAaIeTVAiNHjozXXnstHnrooazjw4cPj9NOOy3Ky8tj0KBB8eijj8bMmTNj4cKFERExYMCAaNSoUTRq1Ci6dOmS9dof/vCHsXDhwnjiiSfizTffjKuuuioiIrZt2xa4ER+3AAAgAElEQVQXXHBBfO9734sOHTrsUX2FhYVRXFyctQEAALVTXpIkSa6LOJiNGjUqnnjiifjd734X7du332XbJEmisLAwfv7zn8f5558fK1eujM2bN0dERN26daNt27Y7fN3vf//7OPnkk2PVqlVRVFQUhxxySOTn52fOV1dXR5IkkZ+fH88880z07dt3l3VUVlZGSUlJ3DKjIooaCnwAaTSqR64rAEiPTz4/V1RU1MiEiYVXciRJkhg1alQ8/vjjMXv27N0GvIiIJUuWxJYtW6Jly5YREdG6des9fq+IjxdQad68eSxatCjr/J133hmzZs2KRx99dI/qAAAAai8hL0cuv/zy+MUvfhFPPvlkNG7cONasWRMRESUlJVFUVBRvvvlmTJkyJc4888xo1qxZLF26NK6++uro2rVr9OrVa6f9/uY3v4m//vWvcdxxx0WjRo1i6dKl8e1vfzt69eoV7dq1i4jI+t5fRERpaWnUr19/u+MAAMCBR8jLkUmTJkXExz94/mmTJ0+OYcOGRb169eLZZ5+N2267LTZt2hRlZWUxcODAGDt2bNajlp9VVFQU99xzT1x55ZVRVVUVZWVlcc4558R11123Py8HAACoJXwnj73mO3kA6ec7eQD7Tk1/J8/qmgAAACki5AEAAKSIkAcAAJAiQh4AAECKCHkAAAAp4icU+NxGHB9RA4sDAQAAe8FMHgAAQIoIeQAAACki5AEAAKSIkAcAAJAiQh4AAECKWF2Tz+2ulyKKGua6CgDSaFSPXFcAcOAykwcAAJAiQh4AAECKCHkAAAApIuQBAACkiJAHAACQIkIeAABAigh5AAAAKSLkAQAApIiQBwAAkCJCXg6tXLkyLrzwwmjatGk0aNAgjjnmmFiwYEHm/LBhwyIvLy9rO/HEE3fZ5+zZs7d7zSfb/Pnzt2v/xhtvROPGjaNJkyb7/PoAAICaV5DrAg5WH374YfTq1StOPfXU+O1vfxulpaXx5ptvbhe2+vfvH5MnT87s16tXb5f99uzZM1avXp117Lvf/W7MnDkzunfvnnV8y5YtMXjw4Dj55JNj7ty5/+QVAQAAtYGQlyO33HJLlJWVZQW4du3abdeusLAwWrRoscf91qtXL6v9li1bYtq0aTFy5MjIy8vLanvDDTdEx44do1+/frsMeVVVVVFVVZXZr6ys3ON6AACAmuVxzRyZNm1adO/ePb7+9a9HaWlpdO3aNe65557t2s2ePTtKS0ujQ4cOMXz48Fi7du1ev8+6deti2LBhWcdnzZoVjzzySNxxxx277WPChAlRUlKS2crKyvaqBgAAoOYIeTny1ltvxaRJk+LII4+M6dOnx4gRI+Kb3/xmPPDAA5k2AwYMiClTpsSsWbPi+9//fsyfPz/69u2bNau2O/fee2+cccYZWcFs/fr1MWzYsLjvvvuiuLh4t32MGTMmKioqMtuKFSv27mIBAIAa43HNHKmuro7u3bvH+PHjIyKia9eusWTJkpg0aVIMHTo0IiLOP//8TPvy8vLo3r17tG3bNp566qk455xzYsSIEfHggw9m2mzatCnrPd57772YPn16/M///E/W8eHDh8cFF1wQp5xyyh7VWlhYGIWFhZ/rOgEAgJplJi9HWrZsGZ07d8461qlTp3j33Xd3+Zq2bdvG8uXLIyJi3Lhx8eqrr2a2z5o8eXI0bdo0zj777Kzjs2bNiltvvTUKCgqioKAgLr300qioqIiCgoL42c9+tg+uDgAAyBUzeTnSq1evWLZsWdax119/Pdq2bbvT16xfvz5WrFgRLVu2jIiI0tLSKC0t3WHbJEli8uTJMXTo0Khbt27WuXnz5sW2bdsy+08++WTccsstMXfu3GjduvXnvSQAAKAWEPJy5Morr4yePXvG+PHj47zzzouXXnop7r777rj77rsj4uNHL2+88cY499xzo2XLlvHOO+/Ed77znWjWrFl87Wtf223/s2bNirfffjsuvfTS7c516tQpa//ll1+OOnXqRHl5+b65OAAAIGc8rpkjxx13XDz++OPx0EMPRXl5efzXf/1X/OhHP4ohQ4ZERER+fn4sWrQovvKVr0SHDh3i4osvjg4dOsS8efOicePGu+3/3nvvjZ49e24X6AAAgHTLS5IkyXURHFgqKyujpKQkbplREUUNd786JwDsrVE9cl0BwL7zyefnioqKPVrd/p9lJg8AACBFhDwAAIAUEfIAAABSRMgDAABIESEPAAAgRfxOHp/biOMjamBxIAAAYC+YyQMAAEgRIQ8AACBFhDwAAIAUEfIAAABSRMgDAABIESEPAAAgRfyEAp/bXS9FFDXMdRUAANS0UT1yXQG7YiYPAAAgRYQ8AACAFBHyAAAAUkTIAwAASBEhDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEgRIS8HbrzxxsjLy8vaWrRokTk/bNiw7c6feOKJe9T3U089FSeccEIUFRVFs2bN4pxzztlhu/Xr10ebNm0iLy8vNmzYsE+uCwAAyL2CXBdwsOrSpUvMnDkzs5+fn591vn///jF58uTMfr169Xbb59SpU2P48OExfvz46Nu3byRJEosWLdph20svvTSOPvroWLly5ee8AgAAoDYS8nKkoKAga/buswoLC3d5/rO2bt0ao0ePjokTJ8all16aOf7FL35xu7aTJk2KDRs2xP/9v/83fvvb3+5d4QAAQK3mcc0cWb58ebRq1Srat28fgwYNirfeeivr/OzZs6O0tDQ6dOgQw4cPj7Vr1+6yv4ULF8bKlSujTp060bVr12jZsmUMGDAglixZktVu6dKlMW7cuHjggQeiTp09G/6qqqqorKzM2gAAgNpJyMuBE044IR544IGYPn163HPPPbFmzZro2bNnrF+/PiIiBgwYEFOmTIlZs2bF97///Zg/f3707ds3qqqqdtrnJyHxxhtvjBtuuCF+/etfxyGHHBK9e/eODz74ICI+DmuDBw+OiRMnxmGHHbbH9U6YMCFKSkoyW1lZ2T9x9QAAwP4k5OXAgAED4txzz42jjjoqTjvttHjqqaciIuL++++PiIjzzz8/Bg4cGOXl5XHWWWfFb3/723j99dcz7UaMGBGNGjXKbBER1dXVERFx/fXXx7nnnhvdunWLyZMnR15eXjzyyCMRETFmzJjo1KlTXHjhhXtV75gxY6KioiKzrVixYp/cBwAAYN8T8mqBhg0bxlFHHRXLly/f4fmWLVtG27ZtM+fHjRsXr776amb7pE1EROfOnTOvKywsjMMPPzzefffdiIiYNWtWPPLII1FQUBAFBQXRr1+/iIho1qxZjB07dqf1FRYWRnFxcdYGAADUThZeqQWqqqriT3/6U5x88sk7PL9+/fpYsWJFJsiVlpZGaWlpVptu3bpFYWFhLFu2LE466aSIiNiyZUu888470bZt24j4ePXNzZs3Z14zf/78uOSSS+L555+PI444Yn9cGgAAUMOEvBy45ppr4qyzzorDDjss1q5dGzfddFNUVlbGxRdfHJs2bYobb7wxzj333GjZsmW888478Z3vfCeaNWsWX/va13baZ3FxcYwYMSLGjh0bZWVl0bZt25g4cWJERHz961+PiNguyK1bty4iIjp16hRNmjTZT1cLAADUJCEvB957770YPHhwrFu3Lv7lX/4lTjzxxHjhhReibdu2sXnz5li0aFE88MADsWHDhmjZsmWceuqp8ctf/jIaN268y34nTpwYBQUFcdFFF8XmzZvjhBNOiFmzZsUhhxxSQ1cGAADkWl6SJEmui+DAUllZGSUlJXHLjIooauj7eQAAB5tRPXJdwYHlk8/PFRUVNbK+hYVXAAAAUkTIAwAASBEhDwAAIEWEPAAAgBQR8gAAAFJEyAMAAEgRv5PH5zbi+IgaWAEWAADYC2byAAAAUkTIAwAASBEhDwAAIEWEPAAAgBQR8gAAAFLE6pp8bne9FFHUMNdVAABQG4zqkesK+ISZPAAAgBQR8gAAAFJEyAMAAEgRIQ8AACBFhDwAAIAUEfIAAABSRMgDAABIESEPAAAgRYS8HJk0aVIcffTRUVxcHMXFxdGjR4/47W9/mznfp0+fyMvLy9oGDRq0x/2vX78+2rRpE3l5ebFhw4YdtvnDH/4QBQUFccwxx/zT1wMAANQOQl6OtGnTJm6++eZ4+eWX4+WXX46+ffvGV77ylViyZEmmzfDhw2P16tWZ7Sc/+cke93/ppZfG0UcfvdPzFRUVMXTo0OjXr98/dR0AAEDtIuTlyFlnnRVnnnlmdOjQITp06BD//d//HY0aNYoXXngh06ZBgwbRokWLzFZSUrJHfU+aNCk2bNgQ11xzzU7bXHbZZXHBBRdEjx49/ulrAQAAag8hrxbYtm1bPPzww/G3v/0tK3RNmTIlmjVrFl26dIlrrrkmNm7cuNu+li5dGuPGjYsHHngg6tTZ8fBOnjw53nzzzRg7duwe1VdVVRWVlZVZGwAAUDsV5LqAg9miRYuiR48e8Y9//CMaNWoUjz/+eHTu3DkiIoYMGRLt27ePFi1axOLFi2PMmDHxxz/+MWbMmLHT/qqqqmLw4MExceLEOOyww+Ktt97ars3y5cvjuuuui+effz4KCvZs+CdMmBDf+973Pt9FAgAANUrIy6EvfvGL8eqrr8aGDRti6tSpcfHFF8ecOXOic+fOMXz48Ey78vLyOPLII6N79+6xcOHCOPbYY2PAgAHx/PPPR0RE27ZtY8mSJTFmzJjo1KlTXHjhhTt8v23btsUFF1wQ3/ve96JDhw57XOeYMWPiqquuyuxXVlZGWVnZ57xqAABgf8pLkiTJdRF87LTTTosjjjhihwusJEkShYWF8fOf/zzOP//8WLlyZWzevDkiIurWrRtt27aNY445JhYtWhR5eXmZ11RXV0d+fn5cf/31ceWVV8YhhxwS+fn5mX6rq6sjSZLIz8+PZ555Jvr27bvbOisrK6OkpCRumVERRQ2L99HVAwBwIBtlqYed+uTzc0VFRRQX7//Pz2byapEkSaKqqmqH55YsWRJbtmyJli1bRkRE69att2szderUTPCLiJg/f35ccskl8fzzz8cRRxwRxcXFsWjRoqzX3HnnnTFr1qx49NFHo3379vvwagAAgFwQ8nLkO9/5TgwYMCDKyspi48aN8fDDD8fs2bPj6aefjjfffDOmTJkSZ555ZjRr1iyWLl0aV199dXTt2jV69eq10z6POOKIrP1169ZFRESnTp2iSZMmEfHxo5+fVlpaGvXr19/uOAAAcGAS8nLkr3/9a1x00UWxevXqKCkpiaOPPjqefvrp+Nd//ddYsWJFPPvss3HbbbfFpk2boqysLAYOHBhjx47NetQSAADgs3wnj73mO3kAAHyW7+TtXE1/J8/v5AEAAKSIkAcAAJAiQh4AAECKCHkAAAApIuQBAACkiJAHAACQIn4nj89txPERNbACLAAAsBfM5AEAAKSIkAcAAJAiQh4AAECKCHkAAAApIuQBAACkiNU1+dzueimiqGGuqwAAgJ0b1SPXFdQ8M3kAAAApIuQBAACkiJAHAACQIkIeAABAigh5AAAAKSLkAQAApIiQBwAAkCJCHgAAQIoIeQAAACki5NUCEyZMiLy8vLjiiisyx/r06RN5eXlZ26BBg3bb12dfk5eXF3fdddcO277xxhvRuHHjaNKkyT67FgAAILcKcl3AwW7+/Plx9913x9FHH73dueHDh8e4ceMy+0VFRXvU5+TJk6N///6Z/ZKSku3abNmyJQYPHhwnn3xyzJ0793NUDgAA1EZCXg5t2rQphgwZEvfcc0/cdNNN251v0KBBtGjRYq/7bdKkyW5fd8MNN0THjh2jX79+uw15VVVVUVVVldmvrKzc65oAAICa4XHNHLr88stj4MCBcdppp+3w/JQpU6JZs2bRpUuXuOaaa2Ljxo171O/IkSOjWbNmcdxxx8Vdd90V1dXVWednzZoVjzzySNxxxx171N+ECROipKQks5WVle3R6wAAgJpnJi9HHn744Vi4cGHMnz9/h+eHDBkS7du3jxYtWsTixYtjzJgx8cc//jFmzJixy37/67/+K/r16xdFRUXx7LPPxtVXXx3r1q2LG264ISIi1q9fH8OGDYsHH3wwiouL96jWMWPGxFVXXZXZr6ysFPQAAKCWEvJyYMWKFTF69Oh45plnon79+jtsM3z48Mw/l5eXx5FHHhndu3ePhQsXxrHHHhsDBgyI559/PiIi2rZtG0uWLImIyIS5iIhjjjkmIiLGjRuXOT58+PC44IIL4pRTTtnjegsLC6OwsHDvLhIAAMiJvCRJklwXcbB54okn4mtf+1rk5+dnjm3bti3y8vKiTp06UVVVlXUuIiJJkigsLIyf//zncf7558fKlStj8+bNERFRt27daNu27Q7f6w9/+EOcdNJJsWbNmmjevHk0adIkNm3alNVvdXV15Ofnx9133x2XXHLJbuuvrKyMkpKSuGVGRRQ13LPZQAAAyIVRPXJdwf9+fq6oqNjjp+n+GWbycqBfv36xaNGirGP//u//Hh07doxrr712u4AXEbFkyZLYsmVLtGzZMiIiWrduvUfv9corr0T9+vUzP5Mwb9682LZtW+b8k08+GbfcckvMnTt3j/sEAABqLyEvBxo3bhzl5eVZxxo2bBhNmzaN8vLyePPNN2PKlClx5plnRrNmzWLp0qVx9dVXR9euXaNXr1477fdXv/pVrFmzJnr06BFFRUXx3HPPxfXXXx/f+MY3Mo9bdurUKes1L7/8ctSpU2e7egAAgAOTkFcL1atXL5599tm47bbbYtOmTVFWVhYDBw6MsWPH7nCW7xN169aNO++8M6666qqorq6Oww8/PMaNGxeXX355DVYPAADkku/ksdd8Jw8AgAPFwfidPL+TBwAAkCJCHgAAQIoIeQAAACki5AEAAKSIkAcAAJAifkKBz23E8RE1sDgQAACwF8zkAQAApIiQBwAAkCJCHgAAQIoIeQAAACki5AEAAKSIkAcAAJAiQh4AAECKCHkAAAApIuQBAACkSEGuC+DA8//au//oqOo7/+OvkB/DD5MpIZIQfsTYKhgDKAlCqBQibTAKlmXbggcxbtU1bUBo9NgiZxdkW8OpXQ9dGXBBj+j2B9QirK20ElcgaFAhQAkELSzBACYGIiQBJYHw/v7hMl+HJDBAkpm5eT7OmXOYez+5933vJ3cmL+69n2tmkqS6uroAVwIAAAAEv/N/N5//O7q9EfJw2WpqaiRJ/fv3D3AlAAAAQOior6+X2+1u9/UQ8nDZYmNjJUkVFRUd8kuK1tXV1al///46dOiQYmJiAl1Op0ZfBAf6IXjQF8GDvggO9EPwCERfmJnq6+uVmJjYIesj5OGydeny5a2cbrebD6kgERMTQ18ECfoiONAPwYO+CB70RXCgH4JHR/dFR54cYeAVAAAAAHAQQh4AAAAAOEj4/Pnz5we6CISe8PBwjR07VhERXPEbaPRF8KAvggP9EDzoi+BBXwQH+iF4OL0vwqyjxvEEAAAAALQ7LtcEAAAAAAch5AEAAACAgxDyAAAAAMBBCHkAAAAA4CCEPAAAAABwEEIeLtuSJUuUnJysrl27Ki0tTZs3bw50SSFt/vz5CgsL83klJCR455uZ5s+fr8TERHXr1k1jx47Vnj17fJZx/PhxTZ8+XW63W263W9OnT9eJEyd82pSWlmrMmDHq1q2b+vbtqwULFqgzD65bVFSkiRMnKjExUWFhYVq7dq3P/I7c76tXr1ZKSopcLpdSUlK0Zs2a9tnoIHWpvnjggQeaHSMjR470adPQ0KCZM2cqLi5OPXr00D333KPDhw/7tKmoqNDEiRPVo0cPxcXF6dFHH1VjY6NPm02bNiktLU1du3bV9ddfr+eff759NjoIFRQUaPjw4YqOjlbv3r01adIkffTRRz5tOnI/d+bvGn/6YuzYsc2Oi6lTp/q04TPq6i1dulRDhgxRTEyMYmJilJGRob/85S/e+RwTHeNS/cDx0AIDLsPKlSstMjLSli9fbmVlZTZr1izr0aOHffzxx4EuLWTNmzfPbr75ZqusrPS+qqurvfMXLlxo0dHRtnr1aistLbUpU6ZYnz59rK6uztvmzjvvtNTUVCsuLrbi4mJLTU21CRMmeOfX1tZafHy8TZ061UpLS2316tUWHR1tv/rVrzp0W4PJunXrbO7cubZ69WqTZGvWrPGZ31H7vbi42MLDw+3pp5+2vXv32tNPP20RERH23nvvtf9OCBKX6oucnBy78847fY6Rmpoanza5ubnWt29fKywstO3bt1tmZqYNHTrUzp49a2ZmZ8+etdTUVMvMzLTt27dbYWGhJSYm2owZM7zLOHDggHXv3t1mzZplZWVltnz5couMjLQ//vGP7b8TgsD48ePtpZdest27d9vOnTvt7rvvtgEDBtjJkye9bTpqP3f27xp/+mLMmDH28MMP+xwXJ06c8FkOn1FX7/XXX7c33njDPvroI/voo4/sySeftMjISNu9e7eZcUx0lEv1A8dDc4Q8XJbbbrvNcnNzfaYNGjTIfvaznwWootA3b948Gzp0aIvzzp07ZwkJCbZw4ULvtNOnT5vb7bbnn3/ezMzKyspMks8HzJYtW0ySffjhh2ZmtmTJEnO73Xb69Glvm4KCAktMTLRz5861x2aFlAuDRUfu9x/84Ad25513+tQzfvx4mzp1attvaAhoLeR997vfbfVnTpw4YZGRkbZy5UrvtCNHjliXLl3sr3/9q5l9GSS7dOliR44c8bb5/e9/by6Xy2pra83M7IknnrBBgwb5LPuRRx6xkSNHXvV2haLq6mqTZJs2bTKzjt3PfNf4urAvzL78o3bWrFmt/gyfUe2nZ8+e9sILL3BMBNj5fjDjeGgJl2vCb42NjSopKVFWVpbP9KysLBUXFweoKmfYt2+fEhMTlZycrKlTp+rAgQOSpPLyclVVVfnsc5fLpTFjxnj3+ZYtW+R2uzVixAhvm5EjR8rtdvu0GTNmjFwul7fN+PHj9cknn+jgwYMdsIWhpSP3+5YtW5odU+PHj+eYusDGjRvVu3dv3XjjjXr44YdVXV3tnVdSUqIzZ8747MfExESlpqb69EVqaqoSExO9bcaPH6+GhgaVlJR427TUF9u2bdOZM2fac/OCUm1trSQpNjZWUsftZ75rmruwL8777W9/q7i4ON188816/PHHVV9f753HZ1Tba2pq0sqVK3Xq1CllZGRwTATIhf1wHseDr4hAF4DQcezYMTU1NSk+Pt5nenx8vKqqqgJUVegbMWKEXnnlFd1444369NNP9fOf/1yjRo3Snj17vPu1pX3+8ccfS5KqqqrUu3fvZsvt3bu39+erqqp03XXXNVvG+XnJycltvVkhrSP3e1VVFcfUJWRnZ+v73/++kpKSVF5ern/5l3/RHXfcoZKSErlcLlVVVSkqKko9e/b0+bmv7seW9nPPnj0VFRV10Tbx8fE6e/asjh07pj59+rTjVgYXM1N+fr5uv/12paamSlKH7Wcz47vmK1rqC0maNm2akpOTlZCQoN27d2vOnDn629/+psLCQkl8RrWl0tJSZWRk6PTp07rmmmu0Zs0apaSkaOfOnRwTHai1fpA4HlpCyMNlCwsL83lvZs2mwX/Z2dnefw8ePFgZGRn6+te/rpdfftk7uMSl9nlL+/9Sbez/biSm71rXUfudY+ripkyZ4v13amqq0tPTlZSUpDfeeEOTJ09u9ec4Tq7cjBkztGvXLr3zzjuXbNvW+7m1fd5Zj4vW+uLhhx/2/js1NVU33HCD0tPTtX37dg0bNkwSn1FtZeDAgdq5c6dOnDih1atXKycnR5s2bWq1PcdE+2itH1JSUjgeWsDlmvBbXFycwsPDm/1vRXV1dbP/1cCV69GjhwYPHqx9+/Z5R9m82D5PSEjQp59+2mw5R48e9WnT0jKk5meroA7d7621oV9a16dPHyUlJWnfvn2SvtyHjY2NOn78uE+7C/vrwv18/PhxnTlz5pJ9ERERoV69erXX5gSdmTNn6vXXX9eGDRvUr18/7/SO2s981/x/rfVFS4YNG6bIyEif44LPqLYRFRWlb3zjG0pPT1dBQYGGDh2qX//61xwTHay1fmgJxwMhD5chKipKaWlp3lPf5xUWFmrUqFEBqsp5GhoatHfvXvXp08d76cFX93ljY6M2bdrk3ecZGRmqra3VBx984G3z/vvvq7a21qdNUVGRz5DN69evV2JiYrNLE6AO3e8ZGRnNjqn169dzTF1ETU2NDh065L18Mi0tTZGRkT77sbKyUrt37/bpi927d6uystLbZv369XK5XEpLS/O2aakv0tPTFRkZ2d6bFXBmphkzZui1117T22+/3ewy7o7az3zXXLovWrJnzx6dOXPGe1zwGdV+zEwNDQ0cEwF2vh9awvEgHqGAy3N+CN8XX3zRysrKbPbs2dajRw87ePBgoEsLWY899pht3LjRDhw4YO+9955NmDDBoqOjvft04cKF5na77bXXXrPS0lK79957WxzKf8iQIbZlyxbbsmWLDR482GdY4BMnTlh8fLzde++9Vlpaaq+99prFxMR06kco1NfX244dO2zHjh0myZ599lnbsWOHdzjqjtrv7777roWHh9vChQtt7969tnDhwqAdjrm9XKwv6uvr7bHHHrPi4mIrLy+3DRs2WEZGhvXt29enL3Jzc61fv3721ltv2fbt2+2OO+5ocRjzcePG2fbt2+2tt96yfv36tTiM+U9+8hMrKyuzF198sVM9QuFHP/qRud1u27hxo88w5J9//rm3TUft587+XXOpvti/f7899dRTtnXrVisvL7c33njDBg0aZLfeequ3L8z4jGoLc+bMsaKiIisvL7ddu3bZk08+aV26dLH169ebGcdER7lYP3A8tIyQh8vm8XgsKSnJoqKibNiwYT5DOuPynX/+WmRkpCUmJn1X7R4AACAASURBVNrkyZNtz5493vnnzp2zefPmWUJCgrlcLvvWt75lpaWlPsuoqamxadOmWXR0tEVHR9u0adPs+PHjPm127dplo0ePNpfLZQkJCTZ//vxO/fiEDRs2mKRmr5ycHDPr2P3+6quv2sCBAy0yMtIGDRpkq1evbtdtDzYX64vPP//csrKy7Nprr7XIyEgbMGCA5eTkWEVFhc8yvvjiC5sxY4bFxsZat27dbMKECc3afPzxx3b33Xdbt27dLDY21mbMmOEzVLaZ2caNG+3WW2+1qKgou+6662zp0qXtvv3BoqU+kGQvvfSSt01H7ufO/F1zqb6oqKiwb33rWxYbG2tRUVH29a9/3R599NFmz4/kM+rq/fCHP/T+Hl577bU2btw4b8Az45joKBfrB46HloWZXfAYdwAAAABAyOKePAAAAABwEEIeAAAAADgIIQ8AAAAAHISQBwAAAAAOQsgDAAAAAAch5AEAAACAgxDyAAAAAMBBCHkAAAAA4CCEPAAAAABwEEIeAAAAADgIIQ8AAAAAHISQBwAAAAAOQsgDAAAAAAch5AEAAACAgxDyAAAAAMBBCHkAAAAA4CCEPAAAAABwEEIeAADtZMWKFQoLC9PBgwcDXQoAoBMJMzMLdBEAADjR0aNH9b//+7+69dZb5XK5Al0OAKCTIOQBAAAAgINwuSYAAO3kwss1x44dq9TUVG3dulWjR49W9+7ddf3112vhwoU6d+6cz8+eOHFCjz32mK6//nq5XC717t1bd911lz788ENvm88++0w//vGP1bdvX0VFRen666/X3Llz1dDQ4LOssLAwzZgxQy+99JIGDhyobt26KT09Xe+9957MTM8884ySk5N1zTXX6I477tD+/fubbctbb72lcePGKSYmRt27d9c3v/lN/c///E/b7zQAwFUj5AEA0IGqqqo0bdo03XfffXr99deVnZ2tOXPm6De/+Y23TX19vW6//Xb953/+p/7pn/5Jf/rTn/T888/rxhtvVGVlpSTp9OnTyszM1CuvvKL8/Hy98cYbuu+++/TLX/5SkydPbrbeP//5z3rhhRe0cOFC/f73v1d9fb3uvvtuPfbYY3r33Xe1ePFiLVu2TGVlZfrHf/xHffVCn9/85jfKyspSTEyMXn75Zf3hD39QbGysxo8fT9ADgCAUEegCAADoTGpqarRu3TrddtttkqRvf/vb2rhxo373u9/p/vvvlyQtWrRIe/bsUWFhob797W97f/ar4e3ll1/Wrl279Ic//EHf//73JUnf+c53dM011+inP/2pCgsL9Z3vfMfbvqGhQevXr1ePHj0kfXl2b9KkSdqwYYO2b9+usLAwSV/eRzh79mzt3r1bgwcP1ueff65Zs2ZpwoQJWrNmjXd5d911l4YNG6Ynn3xS77//fjvtLQDAleBMHgAAHSghIcEb8M4bMmSIPv74Y+/7v/zlL7rxxht9At6F3n77bfXo0UPf+973fKY/8MADktTsDFtmZqY34EnSTTfdJEnKzs72BryvTj9fT3FxsT777DPl5OTo7Nmz3te5c+d05513auvWrTp16pS/mw8A6ACcyQMAoAP16tWr2TSXy6UvvvjC+/7o0aMaMGDARZdTU1OjhIQEn4AmSb1791ZERIRqamp8psfGxvq8j4qKuuj006dPS5I+/fRTSWoWJr/qs88+8wmQAIDAIuQBABBkrr32Wh0+fPiibXr16qX3339fZuYT9Kqrq3X27FnFxcW1SS3nl/Pcc89p5MiRLbaJj49vk3UBANoGl2sCABBksrOz9fe//11vv/12q23GjRunkydPau3atT7TX3nlFe/8tvDNb35TX/va11RWVqb09PQWX+fP/gEAggNn8gAACDKzZ8/WqlWr9N3vflc/+9nPdNttt+mLL77Qpk2bNGHCBGVmZur++++Xx+NRTk6ODh48qMGDB+udd97R008/rbvuuuui9/NdjmuuuUbPPfeccnJy9Nlnn+l73/ueevfuraNHj+pvf/ubjh49qqVLl7bJugAAbYOQBwBAkImOjtY777yj+fPna9myZXrqqafUs2dPDR8+XP/8z/8sSeratas2bNiguXPn6plnntHRo0fVt29fPf7445o3b16b1nPfffdpwIAB+uUvf6lHHnlE9fX16t27t2655RbvQC8AgOARZl99EA4AAAAAIKRxTx4AAAAAOAiXa3ZiERERSk1NlSSlp6frhRde8Ovnzp07p08++UTR0dHNhu4GAAAA4MvMVF9fr8TERHXp0v7n2bhcsxOLi4vTsWPHLvvnDh8+rP79+7dDRQAAAIBzHTp0SP369Wv39XAmD5ctOjpa0pe/pDExMQGuBgAAAAhudXV16t+/v/fv6PZGyAtRRUVFeuaZZ1RSUqLKykqtWbNGkyZN8mmzZMkSPfPMM6qsrNTNN9+sRYsWafTo0d75dXV1SktLU7du3fSLX/xCY8aM8Wvd5y/RjImJIeQBAAAAfuqoW50YeCVEnTp1SkOHDtXixYtbnL9q1SrNnj1bc+fO1Y4dOzR69GhlZ2eroqLC2+bgwYMqKSnR888/r/vvv191dXUtLquhoUF1dXU+LwAAAADBiXvyHCAsLKzZmbwRI0Zo2LBhPg+ovemmmzRp0iQVFBQ0W0Z2drb+7d/+Tenp6c3mzZ8/X0899VSz6bW1tZzJAwAAAC6hrq5Obre7w/5+5kyeAzU2NqqkpERZWVk+07OyslRcXCxJOn78uBoaGiR9OZBKWVmZrr/++haXN2fOHNXW1npfhw4dat8NAAAAAHDFuCfPgY4dO6ampibFx8f7TI+Pj1dVVZUkae/evXrkkUfUpUsXhYWF6de//rViY2NbXJ7L5ZLL5ZLH45HH41FTU1O7bwMAAACAK0PIc7ALb+w0M++0UaNGqbS09LKWl5eXp7y8PO/pZgAAAADBh8s1HSguLk7h4eHes3bnVVdXNzu7dzk8Ho9SUlI0fPjwqy0RAAAAQDsh5DlQVFSU0tLSVFhY6DO9sLBQo0aNuuLl5uXlqaysTFu3br3aEgEAAAC0Ey7XDFEnT57U/v37ve/Ly8u1c+dOxcbGasCAAcrPz9f06dOVnp6ujIwMLVu2TBUVFcrNzQ1g1QAAAADaGyEvRG3btk2ZmZne9/n5+ZKknJwcrVixQlOmTFFNTY0WLFigyspKpaamat26dUpKSrridQbbwCvPbWl93syMjqsDAAAACCY8Jw+XraOf89EaQh4AAABCAc/JAwAAAABcMUIe/MbomgAAAEDwI+TBb4yuCQAAAAQ/Qh4AAAAAOAija8KRWhuUhQFZAAAA4HScyYPfuCcPAAAACH6EPPiNe/IAAACA4EfIAwAAAAAHIeQBAAAAgIMQ8uA37skDAAAAgh8hD37jnjwAAAAg+BHyAAAAAMBBCHkAAAAA4CCEPAAAAABwEEIeAAAAADgIIQ9+Y3RNAAAAIPgR8uA3RtcEAAAAgh8hDwAAAAAchJAHAAAAAA5CyAMAAAAAByHkAQAAAICDEPIAAAAAwEEIeQAAAADgIIQ8+I3n5AEAAADBj5AHv/GcPAAAACD4EfIAAAAAwEEIeQAAAADgIIQ8AAAAAHAQQh4AAAAAOAghDwAAAAAchJAHAAAAAA5CyOvEPv/8cyUlJenxxx8PdCkAAAAA2gghrxP7xS9+oREjRgS6DAAAAABtiJDXSe3bt08ffvih7rrrrkCXAgAAAKANEfJCUFFRkSZOnKjExESFhYVp7dq1zdosWbJEycnJ6tq1q9LS0rR582af+Y8//rgKCgo6quSg8dyW1l8AAACAExDyQtCpU6c0dOhQLV68uMX5q1at0uzZszV37lzt2LFDo0ePVnZ2tioqKiRJ//3f/60bb7xRN954o1/ra2hoUF1dnc8LAAAAQHCKCHQBuHzZ2dnKzs5udf6zzz6rBx98UA899JAkadGiRXrzzTe1dOlSFRQU6L333tPKlSv16quv6uTJkzpz5oxiYmL0r//6ry0ur6CgQE899VS7bAsAAACAtsWZPIdpbGxUSUmJsrKyfKZnZWWpuLhY0peh7dChQzp48KB+9atf6eGHH2414EnSnDlzVFtb630dOnSoXbcBAAAAwJXjTJ7DHDt2TE1NTYqPj/eZHh8fr6qqqitapsvlksvlksfjkcfjUVNTU1uUCgAAAKAdEPIcKiwszOe9mTWbJkkPPPCA38vMy8tTXl6e6urq5Ha7r7ZEAAAAAO2AyzUdJi4uTuHh4c3O2lVXVzc7uwcAAADAeQh5DhMVFaW0tDQVFhb6TC8sLNSoUaOuatkej0cpKSkaPnz4VS0HAAAAQPvhcs0QdPLkSe3fv9/7vry8XDt37lRsbKwGDBig/Px8TZ8+Xenp6crIyNCyZctUUVGh3Nzcq1ovl2sCAAAAwY+QF4K2bdumzMxM7/v8/HxJUk5OjlasWKEpU6aopqZGCxYsUGVlpVJTU7Vu3TolJSVd1XoZeAUAAAAIfmFmZoEuAqHl/Jm82tpaxcTEBKyO57a07fJmZrTt8gAAAACp4/9+5p48AAAAAHAQQh78xsArAAAAQPAj5MFveXl5Kisr09atWwNdCgAAAIBWEPIAAAAAwEEIeQAAAADgIIQ8+I178gAAAIDgR8iD37gnDwAAAAh+hDwAAAAAcBBCHgAAAAA4CCEPfuOePAAAACD4EfLgN+7JAwAAAIIfIQ8AAAAAHISQBwAAAAAOQsgDAAAAAAch5AEAAACAgxDy4DdG1wQAAACCX5iZWaCLQGipq6uT2+1WbW2tYmJiAlbHc1s6Zj0zMzpmPQAAAHCmjv77mTN5AAAAAOAghDwAAAAAcBBCHgAAAAA4CCEPAAAAAByEkAcAAAAADkLIAwAAAAAHIeTBbzwnDwAAAAh+hDz4LS8vT2VlZdq6dWugSwEAAADQCkIeAAAAADgIIQ8AAAAAHISQBwAAAAAOQsgDAAAAAAch5AEAAACAgxDyAAAAAMBBCHmdUH19vYYPH65bbrlFgwcP1vLlywNdEgAAAIA2EhHoAtDxunfvrk2bNql79+76/PPPlZqaqsmTJ6tXr16BLg0AAADAVeJMXicUHh6u7t27S5JOnz6tpqYmmVmAqwIAAADQFgh5IaioqEgTJ05UYmKiwsLCtHbt2mZtlixZouTkZHXt2lVpaWnavHmzz/wTJ05o6NCh6tevn5544gnFxcV1VPkAAAAA2hEhLwSdOnVKQ4cO1eLFi1ucv2rVKs2ePVtz587Vjh07NHr0aGVnZ6uiosLb5mtf+5r+9re/qby8XL/73e/06aefdlT5AAAAANoRIS8EZWdn6+c//7kmT57c4vxnn31WDz74oB566CHddNNNWrRokfr376+lS5c2axsfH68hQ4aoqKio1fU1NDSorq7O5wUAAAAgOBHyHKaxsVElJSXKysrymZ6VlaXi4mJJ0qeffuoNanV1dSoqKtLAgQNbXWZBQYHcbrf31b9///bbAAAAAABXhdE1HebYsWNqampSfHy8z/T4+HhVVVVJkg4fPqwHH3xQZiYz04wZMzRkyJBWlzlnzhzl5+d739fV1XWqoPfcltbnzczouDoAAAAAfxDyHCosLMznvZl5p6WlpWnnzp1+L8vlcsnlcsnj8cjj8aipqalNawUAAADQdrhc02Hi4uIUHh7uPWt3XnV1dbOze5crLy9PZWVl2rp161UtBwAAAED7IeQ5TFRUlNLS0lRYWOgzvbCwUKNGjbqqZXs8HqWkpGj48OFXtRwAAAAA7YfLNUPQyZMntX//fu/78vJy7dy5U7GxsRowYIDy8/M1ffp0paenKyMjQ8uWLVNFRYVyc3Ovar15eXnKy8tTXV2d3G731W4GAAAAgHZAyAtB27ZtU2Zmpvf9+UFRcnJytGLFCk2ZMkU1NTVasGCBKisrlZqaqnXr1ikpKSlQJQMAAADoIGFmZoEuAqHhqwOv/P3vf1dtba1iYmICVs/FRr3sKIyuCQAAgEs5fyVcR/39zD158BsDrwAAAADBj5AHAAAAAA5CyIPfGF0TAAAACH6EPPiNyzUBAACA4EfIAwAAAAAHIeTBb1yuCQAAAAQ/Qh78xuWaAAAAQPAj5AEAAACAgxDyAAAAAMBBIgJdABDKntvS8vSZGR1bBwAAAHAeZ/LgNwZeAQAAAIIfIQ9+Y+AVAAAAIPgR8gAAAADAQQh5AAAAAOAghDwAAAAAcBBCHvzGwCsAAABA8CPkwW8MvAIAAAAEP0IeAAAAADgIIQ8AAAAAHISQBwAAAAAOEhHoAgAnem5L6/NmZnRcHQAAAOh8OJMHAAAAAA5CyAMAAAAAByHkwW88Jw8AAAAIfoQ8+I3n5AEAAADBj5AHAAAAAA5CyAMAAAAAByHkAQAAAICDEPIAAAAAwEEIeQAAAADgIIQ8AAAAAHCQiEAXAHQ2z21pefrMjI6tAwAAAM7EmbxO6tChQxo7dqxSUlI0ZMgQvfrqq4EuCQAAAEAb4ExeJxUREaFFixbplltuUXV1tYYNG6a77rpLPXr0CHRpAAAAAK4CIa+T6tOnj/r06SNJ6t27t2JjY/XZZ58R8gAAAIAQx+WaIaqoqEgTJ05UYmKiwsLCtHbt2mZtlixZouTkZHXt2lVpaWnavHlzi8vatm2bzp07p/79+7d32QAAAADaGSEvRJ06dUpDhw7V4sWLW5y/atUqzZ49W3PnztWOHTs0evRoZWdnq6KiwqddTU2N7r//fi1btqzVdTU0NKiurs7nBQAAACA4hZmZBboIXJ2wsDCtWbNGkyZN8k4bMWKEhg0bpqVLl3qn3XTTTZo0aZIKCgokfRnevvOd7+jhhx/W9OnTW13+/Pnz9dRTTzWbXltbq5iYmDbcksvT2iiVoYrRNQEAAJyprq5Obre7w/5+5kyeAzU2NqqkpERZWVk+07OyslRcXCxJMjM98MADuuOOOy4a8CRpzpw5qq2t9b4OHTrUbrUDAAAAuDqEPAc6duyYmpqaFB8f7zM9Pj5eVVVVkqR3331Xq1at0tq1a3XLLbfolltuUWlpaYvLc7lciomJ0X/9139p5MiRGjduXLtvAwAAAIArw+iaDhYWFubz3sy8026//XadO3fuspaXl5envLw87+lmtK2LXX7KpZwAAADwF2fyHCguLk7h4eHes3bnVVdXNzu7dzk8Ho9SUlI0fPjwqy0RAAAAQDsh5DlQVFSU0tLSVFhY6DO9sLBQo0aNuuLl5uXlqaysTFu3br3aEgEAAAC0Ey7XDFEnT57U/v37ve/Ly8u1c+dOxcbGasCAAcrPz9f06dOVnp6ujIwMLVu2TBUVFcrNzQ1g1QAAAADaGyEvRG3btk2ZmZne9/n5+ZKknJwcrVixQlOmTFFNTY0WLFigyspKpaamat26dUpKSrridXo8Hnk8HjU1NV11/bg8rd2vx716AAAAuBDPycNl6+jnfLTGac/JuxKEPAAAgODHc/IAAAAAAFeMkAe/MbomAAAAEPwIefAbo2sCAAAAwY+QBwAAAAAOQsiD37hcEwAAAAh+hDz4jcs1AQAAgOBHyAMAAAAAByHkAQAAAICDEPIAAAAAwEEiAl0AQofH45HH41FTU1OgS8H/eW5L6/NmZnRcHQAAAAgenMmD3xh4BQAAAAh+hDwAAAAAcBBCHgAAAAA4CPfkAQ7F/XoAAACdE2fy4DePx6OUlBQNHz480KUAAAAAaAUhD35j4BUAAAAg+BHyAAAAAMBBCHkAAAAA4CCEPAAAAABwEEIeAAAAADgIIQ8AAAAAHISQBwAAAAAOwsPQ4TePxyOPx6OmpqZAl4Kr1NqD0nlIOgAAQOjjTB78xnPyAAAAgOBHyAMAAAAAByHkAQAAAICDcE8eAK/W7tWTuF8PAAAgVHAmDwAAAAAchJAHAAAAAA5CyAMAAAAAByHkAQAAAICDEPI6sX/4h39Qz5499b3vfS/QpQAAAABoI4S8TuzRRx/VK6+8EugyAAAAALQhQl4nlpmZqejo6ECXAQAAAKANEfJCVFFRkSZOnKjExESFhYVp7dq1zdosWbJEycnJ6tq1q9LS0rR58+YAVAqneG5Lyy8AAAAEF0JeiDp16pSGDh2qxYsXtzh/1apVmj17tubOnasdO3Zo9OjRys7OVkVFxWWvq6GhQXV1dT4vAAAAAMGJkBeisrOz9fOf/1yTJ09ucf6zzz6rBx98UA899JBuuukmLVq0SP3799fSpUsve10FBQVyu93eV//+/a+2fAAAAADthJDnQI2NjSopKVFWVpbP9KysLBUXF1/28ubMmaPa2lrv69ChQ21VKgAAAIA2FhHoAtD2jh07pqamJsXHx/tMj4+PV1VVlff9+PHjtX37dp06dUr9+vXTmjVrNHz48GbLc7lccrlc8ng88ng8ampqavdtAAAAAHBlCHkOFhYW5vPezHymvfnmm5e1vLy8POXl5amurk5ut7tNagQAAADQtrhc04Hi4uIUHh7uc9ZOkqqrq5ud3bscHo9HKSkpLZ7tAwAAABAcCHkOFBUVpbS0NBUWFvpMLyws1KhRo654uXl5eSorK9PWrVuvtkQAAAAA7YTLNUPUyZMntX//fu/78vJy7dy5U7GxsRowYIDy8/M1ffp0paenKyMjQ8uWLVNFRYVyc3MDWDUAAACA9kbIC1Hbtm1TZmam931+fr4kKScnRytWrNCUKVNUU1OjBQsWqLKyUqmpqVq3bp2SkpKueJ0MvIKO0NoD1mdmdGwdAAAAoSrMzCzQRSC0nB94pba2VjExMQGro7UwgI7V1uGLkAcAAJymo/9+5p48AAAAAHAQLteE37hcEy25kjOqHXVW7mK1cWYQAAA4FWfy4DdG1wQAAACCHyEPAAAAAByEkAcAAAAADsI9efAb9+QhWF3JfYFtfb8eo4ICAIBgwZk8+I178gAAAIDgR8gDAAAAAAch5AEAAACAgxDy4DePx6OUlBQNHz480KUAAAAAaAUhD37jnjwAAAAg+BHyAAAAAMBBCHkAAAAA4CCEPAAAAABwEEIeAAAAADhIRKALQOjweDzyeDxqamoKdCnohJ7bEvh1zczouBoAAACuFGfy4DdG1wQAAACCHyEPAAAAAByEkAcAAAAADkLIAwAAAAAHIeQBAAAAgIMQ8gAAAADAQQh5AAAAAOAgPCcPfuM5eWgrHfnMu7bU1nUH+nl8F9sengmIYMHvKQBcPs7kwW88Jw8AAAAIfoQ8AAAAAHAQQh4AAAAAOAghDwAAAAAchJAHAAAAAA5CyAMAAAAAByHkAQAAAICDEPI6qT//+c8aOHCgbrjhBr3wwguBLgcAAABAG+Fh6J3Q2bNnlZ+frw0bNigmJkbDhg3T5MmTFRsbG+jSAAAAAFwlzuR1Qh988IFuvvlm9e3bV9HR0brrrrv05ptvBrosAAAAAG2AkBeCioqKNHHiRCUmJiosLExr165t1mbJkiVKTk5W165dlZaWps2bN3vnffLJJ+rbt6/3fb9+/XTkyJEOqR0AAABA+yLkhaBTp05p6NChWrx4cYvzV61apdmzZ2vu3LnasWOHRo8erezsbFVUVEiSzKzZz4SFhbW6voaGBtXV1fm8AAAAAAQn7skLQdnZ2crOzm51/rPPPqsHH3xQDz30kCRp0aJFevPNN7V06VIVFBSob9++PmfuDh8+rBEjRrS6vIKCAj311FNttwFAJ/LclsAvb2ZGx6y/LddzpQK9f5yotX16sf3WmfqhM20rEAqu5DPLiTiT5zCNjY0qKSlRVlaWz/SsrCwVFxdLkm677Tbt3r1bR44cUX19vdatW6fx48e3usw5c+aotrbW+zp06FC7bgMAAACAK8eZPIc5duyYmpqaFB8f7zM9Pj5eVVVVkqSIiAj9+7//uzIzM3Xu3Dk98cQT6tWrV6vLdLlccrlc8ng88ng8ampqatdtAAAAAHDlCHkOdeE9dmbmM+2ee+7RPffcc1nLzMvLU15enurq6uR2u9ukTgAAAABti8s1HSYuLk7h4eHes3bnVVdXNzu7BwAAAMB5CHkOExUVpbS0NBUWFvpMLyws1KhRo65q2R6PRykpKRo+fPhVLQcAAABA++FyzRB08uRJ7d+/3/u+vLxcO3fuVGxsrAYMGKD8/HxNnz5d6enpysjI0LJly1RRUaHc3NyrWi+XawIAAADBj5AXgrZt26bMzEzv+/z8fElSTk6OVqxYoSlTpqimpkYLFixQZWWlUlNTtW7dOiUlJV3Vehl4BQAAAAh+hLwQNHbs2BYfaP5VP/7xj/XjH/+4TdfLmTwAAAAg+HFPHgAAAAA4CGfy4Lfzl2uePXtWklRXVxfQer44FdDVA+3mYofWlfzet7a8iy2rLX+mI7Xl/sGXWtunwfx72pH4nQOCy5V8ZnWE8383X+pqvLYSZh21JjjG4cOH1b9//0CXAQAAAISUQ4cOqV+/fu2+HkIeLtu5c+f0ySefKDo6utlD1ztKXV2d+vfvr0OHDikmJiYgNeDq0Iehjz4MffRh6KMPQx99GPr86UMzU319vRITE9WlS/vfMcflmrhsXbp06ZD/gfBHTEwMH4ghjj4MffRh6KMPQx99GProw9B3qT7syIELGXgFAAAAAByEkAcAAAAADhI+f/78+YEuArgS4eHhGjt2rCIiuOo4VNGHoY8+DH30YeijD0MffRj6gq0PGXgFAAAAAByEyzUBAAAAwEEIeQAAAADgIIQ8AAAAAHAQQh4AAAAAOAghDwAAAAAchJCHkLRkyRIlJyera9euSktL0+bNmwNdkuMUFRVp4sSJSkxMVFhYmNauXesz38w0f/58JSYmqlu3bho7dqz27Nnj0+b48eOaPn263G633G63pk+frhMnTvi0KS0t1ZgxY9StWzf17dtXCxYs0IWD/q5evVopKSlyuVxKSUnRmjVrLruWzqigoEDDhw9XdHS0evfurUmTJumjjz7yadPQ0KCZM2cqTgEFmwAAClNJREFULi5OPXr00D333KPDhw/7tKmoqNDEiRPVo0cPxcXF6dFHH1VjY6NPm02bNiktLU1du3bV9ddfr+eff75ZPZc6bv2ppbNZunSphgwZopiYGMXExCgjI0N/+ctfvPPpv9BTUFCgsLAwzZ492zuNfgxu8+fPV1hYmM8rISHBO5/vw9Bw5MgR3XffferVq5e6d++uW265RSUlJd75jutHA0LMypUrLTIy0pYvX25lZWU2a9Ys69Gjh3388ceBLs1R1q1bZ3PnzrXVq1ebJFuzZo3P/IULF1p0dLStXr3aSktLbcqUKdanTx+rq6vztrnzzjstNTXViouLrbi42FJTU23ChAne+bW1tRYfH29Tp0610tJSW716tUVHR9uvfvUrb5vi4mILDw+3p59+2vbu3WtPP/20RURE2HvvvXdZtXRG48ePt5deesl2795tO3futLvvvtsGDBhgJ0+e9LbJzc21vn37WmFhoW3fvt0yMzNt6NChdvbsWTMzO3v2rKWmplpmZqZt377dCgsLLTEx0WbMmOFdxoEDB6x79+42a9YsKysrs+XLl1tkZKT98Y9/9Lbx57i9VC2d0euvv25vvPGGffTRR/bRRx/Zk08+aZGRkbZ7924zo/9CzQcffGDXXXedDRkyxGbNmuWdTj8Gt3nz5tnNN99slZWV3ld1dbV3Pt+Hwe+zzz6zpKQke+CBB+z999+38vJye+utt2z//v3eNk7rR0IeQs5tt91mubm5PtMGDRpkP/vZzwJUkfNdGPLOnTtnCQkJtnDhQu+006dPm9vttueff97MzMrKykySz4fWli1bTJJ9+OGHZma2ZMkSc7vddvr0aW+bgoICS0xMtHPnzpmZ2Q9+8AO78847feoZP368TZ061e9a8KXq6mqTZJs2bTIzsxMnTlhkZKStXLnS2+bIkSPWpUsX++tf/2pmX4b9Ll262JEjR7xtfv/735vL5bLa2lozM3viiSds0KBBPut65JFHbOTIkd73lzpu/akFX+rZs6e98MIL9F+Iqa+vtxtuuMEKCwttzJgx3pBHPwa/efPm2dChQ1ucx/dhaPjpT39qt99+e6vzndiPXK6JkNLY2KiSkhJlZWX5TM/KylJxcXGAqup8ysvLVVVV5dMPLpdLY8aM8fbDli1b5Ha7NWLECG+bkSNHyu12+7QZM2aMXC6Xt8348eP1ySef6ODBg942F/b3+PHjvcvwpxZ8qba2VpIUGxsrSSopKdGZM2d89l1iYqJSU1N9+ig1NVWJiYneNuPHj1dDQ4P3MpfW+mjbtm06c+aMX8etP7V0dk1NTVq5cqVOnTqljIwM+i/E5OXl6e6779a3v/1tn+n0Y2jYt2+fEhMTlZycrKlTp+rAgQOS+D4MFa+//rrS09P1/e9/X71799att96q5cuXe+c7sR8JeQgpx44dU1NTk+Lj432mx8fHq6qqKkBVdT7n9/XF+qGqqkq9e/du9rO9e/f2adPSMr66jtbafHX+pWrBl9f35+fn6/bbb1dqaqqkL/ddVFSUevbs6dP2wv174b7t2bOnoqKiLtlHZ8+e1bFjx/w6bv2ppbMqLS3VNddcI5fLpdzcXK1Zs0YpKSn0XwhZuXKltm/froKCgmbz6MfgN2LECL3yyit68803tXz5clVVVWnUqFGqqanh+zBEHDhwQEuXLtUNN9ygN998U7m5uXr00Uf1yiuvSHLm3zURfrcEgkhYWJjPezNrNg3t71L90FKfXKqN/d/NyZdqc+E0ficubsaMGdq1a5feeeedS7Zt635sqU9bWoY/tXRGAwcO1M6dO3XixAmtXr1aOTk52rRpU6vt6b/gcujQIc2aNUvr169X165d/f45+jF4ZGdne/89ePBgZWRk6Otf/7pefvlljRw5UhLfh8Hu3LlzSk9P19NPPy1JuvXWW7Vnzx4tXbpU999/v7edk/qRM3kIKXFxcQoPD2/2PxnV1dXN/scD7ef8qGIX64eEhAR9+umnzX726NGjPm1aWoakS7b56vxL1dLZzZw5U6+//ro2bNigfv36eacnJCSosbFRx48f92l/4f69cN8eP35cZ86cuWQfRUREqFevXn4dt/7U0llFRUXpG9/4htLT01VQUKChQ4fq17/+Nf0XIkpKSlRdXa20tDRFREQoIiJCmzZt0n/8x38oIiJC8fHx9GOI6dGjhwYPHqx9+/bxfRgi+vTpo5SUFJ9pN910kyoqKiQ58+8aQh5CSlRUlNLS0lRYWOgzvbCwUKNGjQpQVZ1PcnKyEhISfPqhsbFRmzZt8vZDRkaGamtr9cEHH3jbvP/++6qtrfVpU1RU5DMM+Pr165WYmKjrrrvO2+bC/l6/fr13Gf7U0lmZmWbMmKHXXntNb7/9tpKTk33mp6WlKTIy0mffVVZWavfu3T59tHv3blVWVnrbrF+/Xi6XS2lpad42LfVRenq6IiMj/Tpu/akFXzIzNTQ00H8hYty4cSotLdXOnTu9r/T0dE2bNs37b/oxtDQ0NGjv3r3q06cP34ch4pvf/GazRwj9/e9/V1JSkiSH/l3j9xAtQJA4PwT0iy++aGVlZTZ79mzr0aOHHTx4MNClOUp9fb3t2LHDduzYYZLs2WeftR07dniH2l64cKG53W577bXXrLS01O69994WhxoeMmSIbdmyxbZs2WKDBw/2GWr4xIkTFh8fb/fee6+Vlpbaa6+9ZjExMT5DDb/77rsWHh5uCxcutL1799rChQtbHGr4UrV0Rj/60Y/M7Xbbxo0bfYb+/vzzz71tcnNzrV+/fvbWW2/Z9u3b7Y477mhx6PZx48bZ9u3b7a233rJ+/fq1OHT7T37yEysrK7MXX3yx1aHbL3bcXqqWzmjOnDlWVFRk5eXltmvXLnvyySetS5cutn79ejOj/0LVV0fXNKMfg91jjz1mGzdutAMHDth7771nEyZMsOjoaO9+4/sw+H3wwQcWERFhv/jFL2zfvn3229/+1rp3726/+c1vvG2c1o+EPIQkj8djSUlJFhUVZcOGDfMOCY+2s2HDBpPU7JWTk2NmXw7xO2/ePEtISDCXy2Xf+ta3rLS01GcZNTU1Nm3aNIuOjrbo6GibNm2aHT9+3KfNrl27bPTo0eZyuSwhIcHmz5/vHWb4vFdffdUGDhxokZGRNmjQIFu9erXPfH9q6Yxa6j9J9tJLL3nbfPHFFzZjxgyLjY21bt262YQJE6yiosJnOR9//LHdfffd1q1bN4uNjbUZM2b4DA9tZrZx40a79dZbLSoqyq677jpbunRps3ouddz6U0tn88Mf/tC7z6699lobN26cN+CZ0X+h6sKQRz8Gt/PPKIuMjLTExESbPHmy7dmzxzuf78PQ8Kc//clSU1PN5XLZoEGDbNmyZT7zndaPYWYXPIIdAAAAABCyuCcPAAAAAByEkAcAAAAADkLIAwAAAAAHIeQBAAAAgIMQ8gAAAADAQQh5AAAAAOAghDwAAAAAcBBCHgAAAAA4CCEPAAAAAByEkAcAAAAADkLIAwAAAAAH+X8+YPoODRu6EQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "SENSITIVE_COLUMNS = ['Gender', 'occupancy_type', 'age'] # + \"income\", but processed separately, as different type\n", + "\n", + "figs, axs = plt.subplots(4, 1, figsize=(9, 12))\n", + "for i, column in enumerate(SENSITIVE_COLUMNS):\n", + " data = dataset[column].value_counts()\n", + " axs[i].barh(data.index, data.values)\n", + " axs[i].set_title(column, pad=3)\n", + "\n", + "axs[-1].hist(dataset['income'], bins=100, log=True)\n", + "axs[-1].set_title('income', pad=3)\n", + "plt.tight_layout()" + ] + }, + { + "cell_type": "markdown", + "id": "6d2b35a3-6e7d-4850-b091-9a1c006a18ce", + "metadata": {}, + "source": [ + "## 3. Deploying model" + ] + }, + { + "cell_type": "markdown", + "id": "99413c71-e1e1-4a76-ae4b-582a25c0344c", + "metadata": {}, + "source": [ + "Setting up data capture config to be able to monitor model bias based on the stored data" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "7176b64d-d8cc-4c80-83fa-977a9c9b0c83", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "S3 key: s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28\n" + ] + } + ], + "source": [ + "container = sagemaker.image_uris.retrieve(\"xgboost\", sagemaker_session.boto_region_name, \"1.5-1\")\n", + "\n", + "prefix = f'xgboost-for-loan-default-data-{NOW:%Y-%m-%d}'\n", + "s3_key = f\"s3://{STAGE_BUCKET}/{prefix}\"\n", + "print(f\"S3 key: {s3_key}\")\n", + "s3_capture_upload_path = f\"{s3_key}/datacapture\"\n", + "\n", + "data_capture_config = DataCaptureConfig(\n", + " enable_capture=True,\n", + " sampling_percentage=100,\n", + " destination_s3_uri=s3_capture_upload_path,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "e509f304-701d-435e-9062-86564f893d86", + "metadata": {}, + "source": [ + "Creating and deploying Model instance fox XGBoost classifier model which then will be used for ML Observability" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "aa52d932-3ec3-479b-8c62-24eadb74945b", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "-------!" + ] + } + ], + "source": [ + "xgb_model_name = f'loan-default-xgboost-{NOW:%Y-%m-%d}'\n", + "xgb_endpoint_name = f'{xgb_model_name}-endpoint'\n", + "\n", + "xgb_model = Model(\n", + " image_uri=container,\n", + " model_data=MODEL_PATH,\n", + " role=role,\n", + " name=xgb_model_name,\n", + " sagemaker_session=sagemaker_session\n", + ")\n", + "\n", + "xgb_model.deploy(\n", + " endpoint_name=xgb_endpoint_name,\n", + " model_name=xgb_model_name,\n", + " initial_instance_count=1, \n", + " instance_type=\"ml.m4.xlarge\",\n", + " data_capture_config=data_capture_config,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "fef6ee74-b0fe-4b4b-a463-06eacd2c30dd", + "metadata": {}, + "source": [ + "### Results" + ] + }, + { + "cell_type": "markdown", + "id": "9bbec4fa-af38-4985-ac47-d35f9798e850", + "metadata": { + "tags": [] + }, + "source": [ + "As a result, one will get a Load Default model and an endpoint. You can find them in the AWS Console -> SageMaker -> Governance -> Model cards. The new model should appear with specified endpoint (and that's all for now)\n", + "\n", + "\"governance.png\"\n", + "\n", + "\\* here and later all screenshots are given for illustration. You can have different names/values" + ] + }, + { + "cell_type": "markdown", + "id": "5bcab79a-8deb-4734-8838-37ce22bccc53", + "metadata": {}, + "source": [ + "## 4. Setting up monitoring job" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "653cb277-1a13-47c0-9030-b390dc52c6be", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Capture path: s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28/datacapture\n", + "Ground truth path: s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28/ground_truth_data\n", + "Report path: s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28/reports\n", + "Baseline results uri: s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28/baselining\n" + ] + } + ], + "source": [ + "monitoring_prefix = f\"{prefix}/ClarifyModelMonitor-{NOW:%Y-%m-%d}\"\n", + "\n", + "ground_truth_upload_path = f\"{s3_key}/ground_truth_data\"\n", + "s3_report_path = f\"{s3_key}/reports\"\n", + "\n", + "print(f\"Capture path: {s3_capture_upload_path}\")\n", + "print(f\"Ground truth path: {ground_truth_upload_path}\")\n", + "print(f\"Report path: {s3_report_path}\")\n", + "\n", + "baseline_results_uri = f\"{s3_key}/baselining\"\n", + "print(f\"Baseline results uri: {baseline_results_uri}\")\n", + "model_bias_baselining_job_result_uri = f\"{baseline_results_uri}/model_bias\"\n" + ] + }, + { + "cell_type": "markdown", + "id": "d1a9ee97-66cb-44a2-82eb-6ba9fb258759", + "metadata": {}, + "source": [ + "After preprocessing some columns were dropped, some were transformed into numerical by converting binary columns to 0/1 columns, and some were one-hot encoded:" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "9c456e85-3d8c-4e6d-b740-e4bf7177f117", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "TARGET_COLUMN = 'Status'\n", + "ALL_INPUT_COLUMNS = ['Status', 'loan_amount', 'term', 'income', 'Credit_Score', 'loan_limit',\n", + " 'approv_in_adv', 'Credit_Worthiness', 'open_credit',\n", + " 'business_or_commercial', 'Neg_ammortization', 'interest_only',\n", + " 'lump_sum_payment', 'Secured_by', 'co-applicant_credit_type',\n", + " 'submission_of_application', 'Gender_Female', 'Gender_Joint',\n", + " 'Gender_Male', 'Gender_Sex Not Available', 'loan_purpose_p1',\n", + " 'loan_purpose_p2', 'loan_purpose_p3', 'loan_purpose_p4',\n", + " 'occupancy_type_ir', 'occupancy_type_pr', 'occupancy_type_sr',\n", + " 'total_units_1U', 'total_units_2U', 'total_units_3U', 'total_units_4U',\n", + " 'age_25-34', 'age_35-44', 'age_45-54', 'age_55-64', 'age_65-74',\n", + " 'age_gt74', 'age_lt25']\n", + "DATASET_TYPE = \"text/csv\"" + ] + }, + { + "cell_type": "markdown", + "id": "a92fd3c2-48e9-4c2f-9906-854d1875a480", + "metadata": {}, + "source": [ + "Creating and configuring monitor:" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "1978e0a4-2324-4ee0-901d-4a3fcf5591dd", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "model_bias_monitor = ModelBiasMonitor(\n", + " role=role,\n", + " sagemaker_session=sagemaker_session,\n", + " max_runtime_in_seconds=1800,\n", + ")\n", + "\n", + "model_bias_data_config = DataConfig(\n", + " s3_data_input_path=VALIDATION_PATH,\n", + " s3_output_path=model_bias_baselining_job_result_uri,\n", + " label=TARGET_COLUMN,\n", + " headers=ALL_INPUT_COLUMNS,\n", + " dataset_type=DATASET_TYPE,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "eb2a0667-9f49-4254-9d99-c3dfa299239f", + "metadata": {}, + "source": [ + "Setting up facets - sensitive features based on which model bias will be calculated and later observed" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "2398faa8-73c9-411f-add7-9eea111978bd", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "FACET_COLUMNS_VALUES = {\n", + " facet: values \n", + " for facet, values\n", + " in zip(\n", + " [\n", + " 'Gender_Female', 'Gender_Joint', 'Gender_Male', 'Gender_Sex Not Available', \n", + " 'income',\n", + " 'occupancy_type_ir', 'occupancy_type_pr', 'occupancy_type_sr',\n", + " 'age_25-34', 'age_35-44', 'age_45-54', 'age_55-64', 'age_65-74', 'age_gt74', 'age_lt25'\n", + " ],\n", + " [\n", + " [1], [1], [1], [1], # Gender - Binary data, assessing only positive values of one-hot-encoded features\n", + " [25000.], # income - threshold splitting into 2 groups with income level less and more then 30000.0 \n", + " [1], [1], [1], # occupancy_pyte - Binary data, assessing only positive values of one-hot-encoded features \n", + " [1], [1], [1], [1], [1], [1], [1] # age - Binary data, assessing only positive values of one-hot-encoded features\n", + " ]\n", + " )\n", + "}" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "3740e90b-cb2a-4b12-912c-c4f1b2ab7a99", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "model_bias_config = BiasConfig(\n", + " # Loan default value of target feature Status is 1. \n", + " # Thus, we'll choose 0 as positive outcome, so Bias metrics could make sense\n", + " label_values_or_threshold=[0],\n", + " facet_name=list(FACET_COLUMNS_VALUES.keys()),\n", + " facet_values_or_threshold=list(FACET_COLUMNS_VALUES.values()),\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "35d4de0a-0df7-47d3-8197-f5693f7cecef", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "model_predicted_label_config = ModelPredictedLabelConfig(probability_threshold=.5)\n", + "model_config = ModelConfig(\n", + " instance_count=1,\n", + " instance_type=\"ml.m4.xlarge\",\n", + " content_type=DATASET_TYPE,\n", + " accept_type=DATASET_TYPE,\n", + " endpoint_name=xgb_endpoint_name,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "3738f518-c0d6-4515-ba22-3e52a8bcba25", + "metadata": {}, + "source": [ + "### Creating baseline" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "222ad0c1-ef20-479c-8bf2-038ddea23425", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:sagemaker:Creating processing-job with name baseline-suggestion-job-2023-02-28-15-46-10-999\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + ".............................................................................!\n" + ] + } + ], + "source": [ + "model_bias_monitor.suggest_baseline(\n", + " model_config=model_config,\n", + " data_config=model_bias_data_config,\n", + " bias_config=model_bias_config,\n", + " model_predicted_label_config=model_predicted_label_config,\n", + ")\n", + "\n", + "model_bias_monitor.latest_baselining_job.wait(logs=False)\n", + "model_bias_constraints = model_bias_monitor.suggested_constraints()" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "6a70d948-f54a-4643-9495-564c7f80e28b", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "ModelBiasMonitor suggested constraints: s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28/baselining/model_bias/analysis.json\n", + "Visualizing part of suggested constrains:\n", + "{\n", + " \"version\": \"1.0\",\n", + " \"post_training_bias_metrics\": {\n", + " \"label\": \"Status\",\n", + " \"facets\": {\n", + " \"Gender_Female\": [\n", + " {\n", + " \"value_or_threshold\": \"1\",\n", + " \"metrics\": [\n", + " {\n", + " \"name\": \"AD\",\n", + " \"description\": \"Accuracy Difference (AD)\",\n", + " \"value\": -0.012274862880233273\n", + " },\n", + " {\n", + " \"name\": \"CDDPL\",\n", + " \"description\": \"Conditional Demographic Disparity in Predicted Labels (CDDPL)\",\n", + " \"value\": null,\n", + " \"error\": \"Group variable is empty or not provided\"\n", + " },\n", + " {\n", + " \"name\": \"DAR\",\n", + " \"description\": \"Difference in Acceptance Rates (DAR)\",\n", + " \"value\": -0.019687953899706012\n", + " },\n", + " {\n", + " \"name\": \"DCA\",\n", + " \"description\": \"Difference in Conditional Acceptance (DCA)\",\n", + " \"value\": -0.035234696157777856\n", + " },\n", + " {\n", + " \"name\": \"DCR\",\n", + " \"description\": \"Difference in Conditional Rejection (DCR)\",\n", + " \"value\": -0.5736707206241256\n", + " },\n", + " {\n", + " \"name\": \"DI\",\n", + " \"description\": \"Disparate Impact (DI)\",\n", + " \"value\": 0.9544013145312146\n", + " },\n", + " {\n", + " \"name\": \"DPPL\",\n", + " \"description\": \"Difference in Positive Proportions in Predicted Labels (DPPL)\",\n", + " \"value\": 0.040580765079404446\n", + " },\n", + " {\n", + " \"name\": \"DRR\",\n", + " \"description\": \"Difference in Rejection Rates (DRR)\",\n", + " \"value\": -0.010994514578744008\n", + " },\n", + " {\n", + " \"name\": \"FT\",\n", + " \"description\": \"Flip Test (FT)\",\n", + " \"value\": -0.11717435993272285\n", + " },\n", + " {\n", + " \"name\": \"GE\",\n", + " \"description\": \"Generalized Entropy (GE)\",\n", + " \"value\": 0.06776540220552933\n", + " },\n", + " {\n", + " \"name\": \"RD\",\n", + " \"description\": \"Recall Difference (RD)\",\n", + " \"value\": 0.01614804633418354\n", + " },\n", + " {\n", + " \"name\": \"SD\",\n", + " \"description\": \"Specificity Difference (SD)\",\n", + " \"value\": 0.10832164460419041\n", + " },\n", + " {\n", + " \"name\": \"TE\",\n", + " \"description\": \"Treatment Equality (TE)\",\n", + " \"value\": 0.11461989439613904\n", + " }\n", + " ]\n", + " }\n", + " ],\n", + " \"income\": \"...\"\n", + " }\n", + " },\n", + " \"pre_training_bias_metrics\": {\n", + " \"label\": \"Status\",\n", + " \"facets\": {\n", + " \"Gender_Female\": [\n", + " {\n", + " \"value_or_threshold\": \"1\",\n", + " \"metrics\": [\n", + " {\n", + " \"name\": \"CDDL\",\n", + " \"description\": \"Conditional Demographic Disparity in Labels (CDDL)\",\n", + " \"value\": null,\n", + " \"error\": \"Group variable is empty or not provided\"\n", + " },\n", + " {\n", + " \"name\": \"CI\",\n", + " \"description\": \"Class Imbalance (CI)\",\n", + " \"value\": 0.6389460544516042\n", + " },\n", + " {\n", + " \"name\": \"DPL\",\n", + " \"description\": \"Difference in Positive Proportions in Labels (DPL)\",\n", + " \"value\": 0.004437718747468233\n", + " },\n", + " {\n", + " \"name\": \"JS\",\n", + " \"description\": \"Jensen-Shannon Divergence (JS)\",\n", + " \"value\": 1.3179244474955353e-05\n", + " },\n", + " {\n", + " \"name\": \"KL\",\n", + " \"description\": \"Kullback-Liebler Divergence (KL)\",\n", + " \"value\": 5.261255142023462e-05\n", + " },\n", + " {\n", + " \"name\": \"KS\",\n", + " \"description\": \"Kolmogorov-Smirnov Distance (KS)\",\n", + " \"value\": 0.004437718747468261\n", + " },\n", + " {\n", + " \"name\": \"LP\",\n", + " \"description\": \"L-p Norm (LP)\",\n", + " \"value\": 0.006275882038666939\n", + " },\n", + " {\n", + " \"name\": \"TVD\",\n", + " \"description\": \"Total Variation Distance (TVD)\",\n", + " \"value\": 0.004437718747468247\n", + " }\n", + " ]\n", + " }\n", + " ],\n", + " \"income\": \"...\"\n", + " }\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "print(f\"ModelBiasMonitor suggested constraints: {model_bias_constraints.file_s3_uri}\")\n", + "print(\"Visualizing part of suggested constrains:\")\n", + "suggested_constrains = json.loads(S3Downloader.read_file(model_bias_constraints.file_s3_uri))\n", + "res = {k: v for k, v in suggested_constrains.items() if 'metrics' not in k}\n", + "res['post_training_bias_metrics'] = dict(label='Status', facets=dict(Gender_Female=suggested_constrains['post_training_bias_metrics']['facets']['Gender_Female']))\n", + "res['pre_training_bias_metrics'] = dict(label='Status', facets=dict(Gender_Female=suggested_constrains['pre_training_bias_metrics']['facets']['Gender_Female']))\n", + "res['post_training_bias_metrics']['facets']['income'] = '...'\n", + "res['pre_training_bias_metrics']['facets']['income'] = '...'\n", + "print(json.dumps(res, indent=4))" + ] + }, + { + "cell_type": "markdown", + "id": "67cc455b-fed0-493f-8ce2-fa78aa8bd7c4", + "metadata": {}, + "source": [ + "### Setting up monitoring job" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "9444f5ad-4a5a-4ea5-8677-32930a1cf101", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:sagemaker.model_monitor.clarify_model_monitoring:Uploading analysis config to {s3_uri}.\n", + "INFO:sagemaker.model_monitor.model_monitoring:Creating Monitoring Schedule with name: monitoring-schedule-2023-02-28-15-53-44-998\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Model bias monitoring schedule: monitoring-schedule-2023-02-28-15-53-44-998\n" + ] + } + ], + "source": [ + "model_bias_analysis_config = None\n", + "if not model_bias_monitor.latest_baselining_job:\n", + " model_bias_analysis_config = BiasAnalysisConfig(\n", + " model_bias_config,\n", + " headers=all_headers,\n", + " label=label_header,\n", + " )\n", + "\n", + "model_bias_monitor.create_monitoring_schedule(\n", + " analysis_config=model_bias_analysis_config,\n", + " output_s3_uri=s3_report_path,\n", + " endpoint_input=EndpointInput(\n", + " endpoint_name=xgb_endpoint_name,\n", + " destination=\"/opt/ml/processing/input/endpoint\",\n", + " start_time_offset=\"-PT1H\",\n", + " end_time_offset=\"-PT0H\",\n", + " probability_threshold_attribute=0.5,\n", + " ),\n", + " ground_truth_input=ground_truth_upload_path,\n", + " schedule_cron_expression=CronExpressionGenerator.hourly(),\n", + ")\n", + "print(f\"Model bias monitoring schedule: {model_bias_monitor.monitoring_schedule_name}\")" + ] + }, + { + "cell_type": "markdown", + "id": "c01ea526-ef99-4adb-9b97-2ee3a3a1d90e", + "metadata": {}, + "source": [ + "### Results" + ] + }, + { + "cell_type": "markdown", + "id": "54f4f8d1-5466-434e-9bee-2dc7600c73b2", + "metadata": {}, + "source": [ + "Model monitoring job will be created. It might be found in the model dashboard if we go to the card or in the SageMaker Studio:\n", + "SageMaker Studio -> Home -> Deployments -> Endpoint -> _Model name_ -> Model bias\n", + "\n", + "\n", + "![studio_model](images/studio_model.jpg)\n", + "\n", + "\n", + "\"monitoring_job\"\n", + "\n", + "After some time, when several reports will be generated (after the 5th chapter) one will be able to see bias metrics charts below" + ] + }, + { + "cell_type": "markdown", + "id": "19261039-b5df-4a45-8f6d-7d6d5f3c8c87", + "metadata": {}, + "source": [ + "## 5. Generating traffic" + ] + }, + { + "cell_type": "markdown", + "id": "0bf5d84b-4fcb-49bc-be27-155944a51cea", + "metadata": {}, + "source": [ + "Preparing data for checking for model bias. It can be any data supported by model." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "1a7ed27b-94d2-4f09-8f84-099d3549032e", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "test_df = pd.read_csv(TEST_PATH)\n", + "for col in FACET_COLUMNS_VALUES:\n", + " if col == 'income':\n", + " continue\n", + " test_df[col] = test_df[col].astype(int)\n", + "\n", + "gender_aspects = [col for col in FACET_COLUMNS_VALUES if 'Gender' in col]\n", + "descr_colnames = ['metric_kind', 'good', 'min', 'max', 'description',]" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "a3921c14-32da-4b89-871b-480d916e390a", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "# Serialization function which will make Clarity understand types of attributes\n", + "def cast_str(x):\n", + " if x in [1.0, 1, 0, 0.0]:\n", + " return(str(int(x)))\n", + " return str(x)\n", + "\n", + "def serialize_example(example):\n", + " return ','.join(map(cast_str, example))\n", + "\n", + "# For reproducibility\n", + "np.random.seed(42)" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "be41b804-1bfc-4c73-b356-089f7257b55d", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "import threading\n", + "\n", + "class WorkerThread(threading.Thread):\n", + " def __init__(self, do_run, *args, **kwargs):\n", + " super(WorkerThread, self).__init__(*args, **kwargs)\n", + " self.__do_run = do_run\n", + " self.__terminate_event = threading.Event()\n", + "\n", + " def terminate(self):\n", + " self.__terminate_event.set()\n", + "\n", + " def run(self):\n", + " while not self.__terminate_event.is_set():\n", + " self.__do_run(self.__terminate_event)" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "f0d9f9f1-3d80-483b-b33f-dcb1197de03b", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "def invoke(terminate_event):\n", + " # Getting predictions from deployed model for test dataframe\n", + " for i, row in test_df.drop(columns=[TARGET_COLUMN]).iterrows():\n", + " payload = serialize_example(row)\n", + " response = sagemaker_runtime_client.invoke_endpoint(\n", + " EndpointName=xgb_endpoint_name,\n", + " Body=payload,\n", + " ContentType=DATASET_TYPE,\n", + " InferenceId=str(i)\n", + " )\n", + " prediction = response[\"Body\"].read()\n", + " time.sleep(0.01)\n", + " if terminate_event.is_set():\n", + " break\n", + "\n", + "def real_ground_truth_with_id(inference_id):\n", + " # Ground truth requires following format:\n", + " return {\n", + " \"groundTruthData\": {\n", + " \"data\": str(int(test_df.iloc[inference_id][TARGET_COLUMN])),\n", + " \"encoding\": \"CSV\",\n", + " },\n", + " \"eventMetadata\": {\n", + " \"eventId\": str(inference_id),\n", + " },\n", + " \"eventVersion\": \"0\",\n", + " }\n", + "\n", + "\n", + "def upload_ground_truth(upload_time):\n", + " records = [real_ground_truth_with_id(i) for i in range(len(test_df))]\n", + " records = [json.dumps(r) for r in records]\n", + " data_to_upload = \"\\n\".join(records)\n", + " target_s3_uri = f\"{ground_truth_upload_path}/{upload_time:%Y/%m/%d/%H/%M%S}.jsonl\"\n", + " S3Uploader.upload_string_as_file_body(data_to_upload, target_s3_uri)\n", + "\n", + "def generate_ground_truth(terminate_event):\n", + " invoke(terminate_event)\n", + " upload_ground_truth(datetime.utcnow())\n", + " for _ in range(0, 60):\n", + " time.sleep(60)\n", + " if terminate_event.is_set():\n", + " break" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "28ba35fd-2c42-4799-8664-a671190dc0fd", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "ground_truth_thread = WorkerThread(do_run=generate_ground_truth)\n", + "ground_truth_thread.start()" + ] + }, + { + "cell_type": "markdown", + "id": "b7d0e8d9-f546-4a67-b67d-bc6305aa338a", + "metadata": {}, + "source": [ + "### Results" + ] + }, + { + "cell_type": "markdown", + "id": "544a1b16-7f68-4ba7-9264-a4a238880b6c", + "metadata": {}, + "source": [ + "After generating some trafic we need to wait until it is processed, so we could visualize the results" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "f3ab7723-75a3-42ee-a32b-22b084e164f5", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "def wait_for_execution_to_start(model_monitor):\n", + " print(\n", + " \"A hourly schedule will kick off executions ON the hour (plus 0 - 20 min buffer).\"\n", + " )\n", + "\n", + " print(\"Waiting for the first execution to happen\", end=\"\")\n", + " schedule_desc = model_monitor.describe_schedule()\n", + " while \"LastMonitoringExecutionSummary\" not in schedule_desc:\n", + " schedule_desc = model_monitor.describe_schedule()\n", + " print(\".\", end=\"\", flush=True)\n", + " time.sleep(60)\n", + " print()\n", + " print(\"Done! Execution has been created\")\n", + "\n", + " print(\"Now waiting for execution to start\", end=\"\")\n", + " while schedule_desc[\"LastMonitoringExecutionSummary\"][\"MonitoringExecutionStatus\"] in \"Pending\":\n", + " schedule_desc = model_monitor.describe_schedule()\n", + " print(\".\", end=\"\", flush=True)\n", + " time.sleep(10)\n", + "\n", + " print()\n", + " print(\"Done! Execution has started\")" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "aca68c10-401d-42f9-9495-56dcc859fbc8", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "# Waits for the schedule to have last execution in a terminal status.\n", + "def wait_for_execution_to_finish(model_monitor):\n", + " schedule_desc = model_monitor.describe_schedule()\n", + " execution_summary = schedule_desc.get(\"LastMonitoringExecutionSummary\")\n", + " if execution_summary is not None:\n", + " print(\"Waiting for execution to finish\", end=\"\")\n", + " while execution_summary[\"MonitoringExecutionStatus\"] not in [\n", + " \"Completed\",\n", + " \"CompletedWithViolations\",\n", + " \"Failed\",\n", + " \"Stopped\",\n", + " ]:\n", + " print(\".\", end=\"\", flush=True)\n", + " time.sleep(60)\n", + " schedule_desc = model_monitor.describe_schedule()\n", + " execution_summary = schedule_desc[\"LastMonitoringExecutionSummary\"]\n", + " print()\n", + " print(\"Done! Execution has finished\")\n", + " else:\n", + " print(\"Last execution not found\")\n", + " return schedule_desc" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "490a911a-0d69-4d63-b3eb-7e2e9cf0e0ed", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "A hourly schedule will kick off executions ON the hour (plus 0 - 20 min buffer).\n", + "Waiting for the first execution to happen........\n", + "Done! Execution has been created\n", + "Now waiting for execution to start\n", + "Done! Execution has started\n", + "Waiting for execution to finish...........\n", + "Done! Execution has finished\n" + ] + } + ], + "source": [ + "wait_for_execution_to_start(model_bias_monitor)\n", + "schedule_desc = wait_for_execution_to_finish(model_bias_monitor)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c7ceb6d1-2984-4942-8826-eed29041bf2e", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "metrix_values = pd.DataFrame.from_dict({\n", + " # Post\n", + " 'AD': [0, -1, 1, 'Accuracy Difference (AD)'],\n", + " 'DAR': [0, -1, 1, 'Difference in Acceptance Rates (DAR)'],\n", + " 'DCA': [0, -np.Inf, np.Inf, 'Difference in Conditional Acceptance (DCA)'],\n", + " 'DCR': [0, -np.Inf, np.Inf, 'Difference in Conditional Rejection (DCR)'],\n", + " 'DI': [1, 0, np.Inf, 'Disparate Impact (DI)'],\n", + " 'DPPL': [0, -1, 1, 'Difference in Positive Proportions in Predictected Labels (DPPL)'],\n", + " 'DRR': [0, -1, 1, 'Difference in Rejection Rates (DRR)'], \n", + " 'FT': [0, -1, 1, 'Flip Test (FT)'],\n", + " 'GE': [0, 0, .5, 'Generalized Entropy (GE)'],\n", + " 'RD': [0, -1, 1, 'Recall Difference (RD)'],\n", + " 'SD': [0, -1, 1, 'Specificity Difference (SD)'],\n", + " 'TE': [0, -np.Inf, np.Inf, 'Treatment Equality (TE)'],\n", + " # Pre\n", + " 'CI': [0, -1, 1, 'Class Imbalance (CI)'],\n", + " 'DPL': [0, -1, 1, 'Difference in Positive Proportions in Labels (DPL)'],\n", + " 'JS': [0, 0, np.Inf, 'Jensen-Shannon Divergence (JS)'],\n", + " 'KL': [0, 0, np.Inf, 'Kullback-Liebler Divergence (KL)'],\n", + " 'KS': [0, 0, 1, 'Kolmogorov-Smirnov Distance (KS)'],\n", + " 'LP': [0, 0, np.Inf, 'L-p Norm (LP)'],\n", + " 'TVD': [0, 0, np.Inf, 'Total Variation Distance (TVD)'],\n", + "}, orient='index', columns=['good', 'min', 'max', 'description'])\n", + "\n", + "def extract_metrix_from_analysis(analysis_dict):\n", + " post_training = analysis_dict['post_training_bias_metrics']\n", + " pre_training = analysis_dict['pre_training_bias_metrics']\n", + " facets = post_training['facets'].keys()\n", + " frames = dict()\n", + " \n", + " for metric_kind in ['post_training_bias_metrics', 'pre_training_bias_metrics']:\n", + " metric = analysis_dict[metric_kind]\n", + " frames[metric_kind] = dict()\n", + " for facet in facets:\n", + " frames[metric_kind][facet] = []\n", + " for facet_analysis in metric['facets'][facet]:\n", + " frames[metric_kind][facet].append(\n", + " pd.DataFrame(facet_analysis['metrics'])\n", + " .drop(columns=['error', 'description'])\n", + " .set_index('name')\n", + " # as all features are binary:\n", + " .rename(columns={'value': facet})\n", + " )\n", + " frames[metric_kind][facet] = pd.concat(frames[metric_kind][facet], axis=1)\n", + " frames[metric_kind] = pd.concat(frames[metric_kind].values(), axis=1)\n", + " frames[metric_kind]['metric_kind'] = metric_kind\n", + " return pd.concat(frames.values()).join(metrix_values, how='inner')" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "305c2eae-6df8-48a0-8317-755e85ceee46", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Report URI: s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28/reports/loan-default-xgboost-2023-02-28-endpoint/monitoring-schedule-2023-02-28-15-53-44-998/2023/02/28/16\n", + "Found Report Files:\n", + "s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28/reports/loan-default-xgboost-2023-02-28-endpoint/monitoring-schedule-2023-02-28-15-53-44-998/2023/02/28/16/analysis.json\n", + " s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28/reports/loan-default-xgboost-2023-02-28-endpoint/monitoring-schedule-2023-02-28-15-53-44-998/2023/02/28/16/constraint_violations.json\n", + " s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28/reports/loan-default-xgboost-2023-02-28-endpoint/monitoring-schedule-2023-02-28-15-53-44-998/2023/02/28/16/report.html\n", + " s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28/reports/loan-default-xgboost-2023-02-28-endpoint/monitoring-schedule-2023-02-28-15-53-44-998/2023/02/28/16/report.ipynb\n", + " s3://adp-rnd-ml-stage/xgboost-for-loan-default-data-2023-02-28/reports/loan-default-xgboost-2023-02-28-endpoint/monitoring-schedule-2023-02-28-15-53-44-998/2023/02/28/16/report.pdf\n", + "\n", + "Gender aspects pre-training bias metrics:\n" + ] + }, + { + "data": { + "text/html": [ + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
 Gender_FemaleGender_JointGender_MaleGender_Sex Not Availablemetric_kindgoodminmaxdescription
CI0.6311070.4396810.4336990.495513pre_training_bias_metrics0-1.0000001.000000Class Imbalance (CI)
DPL-0.029181-0.0988800.0256910.101349pre_training_bias_metrics0-1.0000001.000000Difference in Positive Proportions in Labels (DPL)
JS0.0005730.0068490.0004270.006377pre_training_bias_metrics00.000000infJensen-Shannon Divergence (JS)
KL0.0023220.0289280.0016910.024665pre_training_bias_metrics00.000000infKullback-Liebler Divergence (KL)
KS0.0291810.0988800.0256910.101349pre_training_bias_metrics00.0000001.000000Kolmogorov-Smirnov Distance (KS)
LP0.0412690.1398380.0363330.143329pre_training_bias_metrics00.000000infL-p Norm (LP)
TVD0.0291810.0988800.0256910.101349pre_training_bias_metrics00.000000infTotal Variation Distance (TVD)
\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Gender aspects post-training bias metrics:\n" + ] + }, + { + "data": { + "text/html": [ + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
 Gender_FemaleGender_JointGender_MaleGender_Sex Not Availablemetric_kindgoodminmaxdescription
AD-0.036073-0.053750-0.0220030.109918post_training_bias_metrics0-1.0000001.000000Accuracy Difference (AD)
DAR-0.054765-0.046989-0.0283260.123350post_training_bias_metrics0-1.0000001.000000Difference in Acceptance Rates (DAR)
DCA-0.079445-0.009282-0.0533510.125789post_training_bias_metrics0-infinfDifference in Conditional Acceptance (DCA)
DCR-0.8688902.435185-1.0647911.119565post_training_bias_metrics0-infinfDifference in Conditional Rejection (DCR)
DI0.9487941.1256800.9075141.013672post_training_bias_metrics10.000000infDisparate Impact (DI)
DPPL0.045510-0.1068800.083611-0.011995post_training_bias_metrics0-1.0000001.000000Difference in Positive Proportions in Predictected Labels (DPPL)
DRR-0.0466090.0277780.034953-0.013975post_training_bias_metrics0-1.0000001.000000Difference in Rejection Rates (DRR)
FT-0.145946-0.017794-0.158451-0.102767post_training_bias_metrics0-1.0000001.000000Flip Test (FT)
GE0.0711140.0711140.0711140.071114post_training_bias_metrics00.0000000.500000Generalized Entropy (GE)
RD0.023645-0.0449270.0260700.004010post_training_bias_metrics0-1.0000001.000000Recall Difference (RD)
SD0.152032-0.2074110.213436-0.149188post_training_bias_metrics0-1.0000001.000000Specificity Difference (SD)
TE0.228039-0.1664530.171154-0.110849post_training_bias_metrics0-infinfTreatment Equality (TE)
\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "analisys_df = None\n", + "\n", + "execution_summary = schedule_desc.get(\"LastMonitoringExecutionSummary\")\n", + "if execution_summary and execution_summary[\"MonitoringExecutionStatus\"] in [\n", + " \"Completed\",\n", + " \"CompletedWithViolations\",\n", + "]:\n", + " last_model_bias_monitor_execution = model_bias_monitor.list_executions()[-1]\n", + " last_model_bias_monitor_execution_report_uri = (\n", + " last_model_bias_monitor_execution.output.destination\n", + " )\n", + " print(f\"Report URI: {last_model_bias_monitor_execution_report_uri}\")\n", + " last_model_bias_monitor_execution_report_files = sorted(\n", + " S3Downloader.list(last_model_bias_monitor_execution_report_uri)\n", + " )\n", + " print(\"Found Report Files:\")\n", + " print(\"\\n \".join(last_model_bias_monitor_execution_report_files))\n", + " if execution_summary[\"MonitoringExecutionStatus\"] == \"CompletedWithViolations\":\n", + " file = [name for name in last_model_bias_monitor_execution_report_files if 'analysis.json' in name]\n", + " assert len(file) == 1, 'Error: analysis file was not generated'\n", + " file = file[0]\n", + " analysis = json.loads(S3Downloader.read_file(file))\n", + " analysis_df = extract_metrix_from_analysis(analysis)\n", + " print('Gender aspects pre-training bias metrics:')\n", + " display(analysis_df[gender_aspects + descr_colnames].query(\"metric_kind == 'pre_training_bias_metrics'\")\n", + " .style.applymap(lambda v: 'opacity: 20%;' if (v < 0.1) and (v > -0.1) else None, subset=gender_aspects))\n", + " print('Gender aspects post-training bias metrics:')\n", + " idx = pd.IndexSlice\n", + " display(analysis_df[gender_aspects + descr_colnames].query(\"metric_kind == 'post_training_bias_metrics'\")\n", + " .style.applymap(lambda v: 'opacity: 20%;' if (v < 0.1) and (v > -0.1) else None, subset=gender_aspects)\n", + " .applymap(lambda v: 'opacity: 20%;' if (v < 1.1) and (v > -0.9) else None, subset=idx[idx['DI'], idx[gender_aspects]]))\n", + " \n", + "else:\n", + " last_model_bias_monitor_execution = None\n", + " print(\n", + " \"====STOP==== \\n No completed executions to inspect further. Please wait till an execution completes or investigate previously reported failures.\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "9e78d581-bc25-4767-9af3-771f0e765b5d", + "metadata": {}, + "source": [ + "We can see some abnormal values (without opacity). Especially in **Difference in Conditional Rejection (DCR)** metrics. That means that model tends to give negative prediction differently across genders, i.e., we can see bias and need to handle that. At the same time, pre-training metrics, overall, looks normal." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "8d1bd888-f68e-4c12-af36-9d1686a3a366", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Let's print the first few of violations:\n", + "{\n", + " \"version\": \"1.0\",\n", + " \"violations\": [\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"DPL\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value -0.029181259499107992 doesn't meet the baseline constraint requirement 0.004437718747468233\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"JS\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value 0.0005725270156258485 doesn't meet the baseline constraint requirement 1.3179244474955353e-05\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"KL\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value 0.0023215021891724794 doesn't meet the baseline constraint requirement 5.261255142023462e-05\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"KS\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value 0.029181259499107992 doesn't meet the baseline constraint requirement 0.004437718747468261\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"LP\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value 0.041268532950767156 doesn't meet the baseline constraint requirement 0.006275882038666939\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"TVD\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value 0.029181259499107937 doesn't meet the baseline constraint requirement 0.004437718747468247\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"AD\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value -0.036073481794753226 doesn't meet the baseline constraint requirement -0.012274862880233273\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"DAR\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value -0.05476492787359355 doesn't meet the baseline constraint requirement -0.019687953899706012\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"DCA\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value -0.07944485592353545 doesn't meet the baseline constraint requirement -0.035234696157777856\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"DCR\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value -0.8688897309586965 doesn't meet the baseline constraint requirement -0.5736707206241256\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"DPPL\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value 0.045509812991475496 doesn't meet the baseline constraint requirement 0.040580765079404446\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"DRR\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value -0.04660856384994316 doesn't meet the baseline constraint requirement -0.010994514578744008\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"FT\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value -0.14594594594594595 doesn't meet the baseline constraint requirement -0.11717435993272285\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"GE\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value 0.07111418898122515 doesn't meet the baseline constraint requirement 0.06776540220552933\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"RD\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value 0.023645182352392546 doesn't meet the baseline constraint requirement 0.01614804633418354\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"SD\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value 0.15203216692023475 doesn't meet the baseline constraint requirement 0.10832164460419041\"\n", + " },\n", + " {\n", + " \"facet\": \"Gender_Female\",\n", + " \"facet_value\": \"1\",\n", + " \"metric_name\": \"TE\",\n", + " \"constraint_check_type\": \"bias_drift_check\",\n", + " \"description\": \"Metric value 0.228039041703638 doesn't meet the baseline constraint requirement 0.11461989439613904\"\n", + " }\n", + " ]\n", + "}\n" + ] + } + ], + "source": [ + "if last_model_bias_monitor_execution:\n", + " model_bias_violations = last_model_bias_monitor_execution.constraint_violations()\n", + " if model_bias_violations:\n", + " print(\"Let's print the first few of violations:\")\n", + " violations_part = copy.deepcopy(model_bias_violations.body_dict)\n", + " violations_part['violations'] = [violation for violation in violations_part['violations'] if violation['facet'] == 'Gender_Female']\n", + " print(json.dumps(violations_part, indent=4))" + ] + }, + { + "cell_type": "markdown", + "id": "7074f498-bb0a-408f-8c8d-469deb0dbd58", + "metadata": {}, + "source": [ + "After some time, we'll see the monitoring job results in the Monitoring Job History tab\n", + "\n", + "\"monitoring_result\"\n", + "\n", + "Clicking on them we can see the additional details: which metrics constrains were violated compared to the baseline \n", + "\n", + "\"violations\"" + ] + }, + { + "cell_type": "markdown", + "id": "88b9969c-cc15-49af-8e57-c5af9e950775", + "metadata": { + "tags": [] + }, + "source": [ + "## 6. Cleaning up " + ] + }, + { + "cell_type": "markdown", + "id": "cd0bf892-8bf7-431a-a9f1-843f4a9c776c", + "metadata": {}, + "source": [ + "Now we know how to set up model bias monitoring with AWS SageMaker tools. And we can remove all created resourses, in order not to recieve unexpected bills at the end of the month." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "ebf8c362-688e-422c-8c37-ae07c0313231", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Stopping Monitoring Schedule with name: monitoring-schedule-2023-02-28-15-53-44-998\n" + ] + } + ], + "source": [ + "# At first we must wait untill monitoring job finishes, \n", + "# as we can't delete monitoring schedule before\n", + "while model_bias_monitor.describe_schedule()['LastMonitoringExecutionSummary']['MonitoringExecutionStatus'] in ['Pending', 'InProgress']:\n", + " time.sleep(60)\n", + "\n", + "model_bias_monitor.stop_monitoring_schedule()" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "4e373fc3-3fc1-4632-bdbe-00e6dea10d4c", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Deleting Monitoring Schedule with name: monitoring-schedule-2023-02-28-15-53-44-998\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:sagemaker.model_monitor.clarify_model_monitoring:Deleting Model Bias Job Definition with name: model-bias-job-definition-2023-02-28-15-53-44-998\n" + ] + } + ], + "source": [ + "# Turning off traffic generation\n", + "ground_truth_thread.terminate()\n", + "# Deleting monitoring job\n", + "model_bias_monitor.delete_monitoring_schedule()\n", + "# Removing endpoint and model\n", + "sagemaker_client.delete_endpoint(EndpointName=xgb_endpoint_name)\n", + "sagemaker_client.delete_endpoint_config(EndpointConfigName=xgb_endpoint_name)\n", + "sagemaker_client.delete_model(ModelName=xgb_model_name);" + ] + }, + { + "cell_type": "markdown", + "id": "8b9a2e0b-7454-4109-bdec-1dd00fa0a290", + "metadata": {}, + "source": [ + "Don't forget to stop the current runtime (if you work from the SageMaker Studio):\n", + "\n", + "\"runtime\"" + ] + } + ], + "metadata": { + "instance_type": "ml.t3.medium", + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.15" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}