dia.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License \n",
    "\n",
    "Copyright 2019-2020 Patrick Hall and the H2O.ai team\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "\n",
    "    http://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**DISCLAIMER:** This notebook is not legal compliance advice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Increase Fairness in Your Machine Learning Project with Disparate Impact Analysis using Python and H2O\n",
    "\n",
    "#### Assess if your machine learning model is treating different groups of people similarly or differently\n",
    "\n",
    "Fairness is an incredibly important, but highly complex entity. So much so that leading scholars have yet to agree on a strict definition. However, there is a practical way to discuss and handle *observational* fairness, or how your model predictions affect different groups of people. This procedure is often known as disparate impact analysis (DIA). DIA is far from perfect, as it relies heavily on user-defined thresholds and reference levels for disparity and does not attempt to remediate disparity or provide information on sources of disparity, but it is a fairly straightforward method to quantify your model’s behavior across sensitive demographic segments or other potentially interesting groups of observations. DIA is also an accepted, regulation-compliant tool for fair-lending purposes in the U.S. financial services industry. If it’s good enough for multibillion-dollar credit portfolios, it’s probably good enough for your project. \n",
    "\n",
    "This example DIA notebook starts by training a constrained, monotonic gradient boosting machine (GBM) classifier on the UCI credit card default data using the popular open source library, h2o. A probability cutoff for making credit decisions is selected by maximizing the F1 statistic and confusion matrices are generated to summarize the GBM’s decisions across male and female customers. A basic DIA procedure is then conducted using the information stored in the confusion matrices then several traditional fair lending measures are calculated. \n",
    "\n",
    "Because DIA really only considers groups of people, it's important to pair DIA with constrained models that can't treat similar people too differently. When using complex models with DIA, it's also important to look for any local, or individual, dicrimination that would not be flagged in the group quantities.\n",
    "\n",
    "For an excellent and free reference on the topics of fairness and machine learning be sure to follow the book being written by Solon Barocas, Moritz Hardt and Arvind Narayanan: https://fairmlbook.org/. \n",
    "\n",
    "For Python fairness libraries that go far beyond the scope of this notebook see: \n",
    "* aequitas: https://github.com/dssg/aequitas\n",
    "* AI Fairness 360: http://aif360.mybluemix.net/\n",
    "* Themis: https://github.com/LASER-UMASS/Themis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Python imports\n",
    "In general, NumPy and Pandas will be used for data manipulation purposes and h2o will be used for modeling tasks. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/patrickh/anaconda3/lib/python3.6/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.\n",
      "  from numpy.core.umath_tests import inner1d\n"
     ]
    }
   ],
   "source": [
    "# imports\n",
    "\n",
    "# h2o Python API with specific classes\n",
    "import h2o                                        \n",
    "from h2o.estimators.gbm import H2OGradientBoostingEstimator # for GBM\n",
    "from h2o.grid.grid_search import H2OGridSearch \n",
    "\n",
    "import numpy as np   # array, vector, matrix calculations\n",
    "import pandas as pd  # DataFrame handling\n",
    "import shap          # for visualizing Shapley values\n",
    "\n",
    "import matplotlib.pyplot as plt # general plotting\n",
    "pd.options.display.max_columns = 999 # enable display of all columns in notebook\n",
    "\n",
    "# display plots in-notebook\n",
    "%matplotlib inline   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Start h2o\n",
    "H2o is both a library and a server. The machine learning algorithms in the library take advantage of the multithreaded and distributed architecture provided by the server to train machine learning algorithms extremely efficiently. The API for the library was imported above in cell 1, but the server still needs to be started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321 . connected.\n",
      "Warning: Your H2O cluster version is too old (5 months and 24 days)! Please download and install the latest version from http://h2o.ai/download/\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O cluster uptime:</td>\n",
       "<td>48 mins 12 secs</td></tr>\n",
       "<tr><td>H2O cluster timezone:</td>\n",
       "<td>America/New_York</td></tr>\n",
       "<tr><td>H2O data parsing timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O cluster version:</td>\n",
       "<td>3.26.0.3</td></tr>\n",
       "<tr><td>H2O cluster version age:</td>\n",
       "<td>5 months and 24 days !!!</td></tr>\n",
       "<tr><td>H2O cluster name:</td>\n",
       "<td>H2O_from_python_patrickh_h0f7ya</td></tr>\n",
       "<tr><td>H2O cluster total nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O cluster free memory:</td>\n",
       "<td>1.532 Gb</td></tr>\n",
       "<tr><td>H2O cluster total cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O cluster allowed cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O cluster status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O connection url:</td>\n",
       "<td>http://localhost:54321</td></tr>\n",
       "<tr><td>H2O connection proxy:</td>\n",
       "<td>None</td></tr>\n",
       "<tr><td>H2O internal security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>H2O API Extensions:</td>\n",
       "<td>Amazon S3, XGBoost, Algos, AutoML, Core V3, Core V4</td></tr>\n",
       "<tr><td>Python version:</td>\n",
       "<td>3.6.3 final</td></tr></table></div>"
      ],
      "text/plain": [
       "--------------------------  ---------------------------------------------------\n",
       "H2O cluster uptime:         48 mins 12 secs\n",
       "H2O cluster timezone:       America/New_York\n",
       "H2O data parsing timezone:  UTC\n",
       "H2O cluster version:        3.26.0.3\n",
       "H2O cluster version age:    5 months and 24 days !!!\n",
       "H2O cluster name:           H2O_from_python_patrickh_h0f7ya\n",
       "H2O cluster total nodes:    1\n",
       "H2O cluster free memory:    1.532 Gb\n",
       "H2O cluster total cores:    8\n",
       "H2O cluster allowed cores:  8\n",
       "H2O cluster status:         locked, healthy\n",
       "H2O connection url:         http://localhost:54321\n",
       "H2O connection proxy:\n",
       "H2O internal security:      False\n",
       "H2O API Extensions:         Amazon S3, XGBoost, Algos, AutoML, Core V3, Core V4\n",
       "Python version:             3.6.3 final\n",
       "--------------------------  ---------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "h2o.init(max_mem_size='2G')       # start h2o\n",
    "h2o.remove_all()                  # remove any existing data structures from h2o memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Download, explore, and prepare UCI credit card default data\n",
    "\n",
    "UCI credit card default data: https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients\n",
    "\n",
    "The UCI credit card default data contains demographic and payment information about credit card customers in Taiwan in the year 2005. The data set contains 23 input variables: \n",
    "\n",
    "* **`LIMIT_BAL`**: Amount of given credit (NT dollar)\n",
    "* **`SEX`**: 1 = male; 2 = female\n",
    "* **`EDUCATION`**: 1 = graduate school; 2 = university; 3 = high school; 4 = others \n",
    "* **`MARRIAGE`**: 1 = married; 2 = single; 3 = others\n",
    "* **`AGE`**: Age in years \n",
    "* **`PAY_0`, `PAY_2` - `PAY_6`**: History of past payment; `PAY_0` = the repayment status in September, 2005; `PAY_2` = the repayment status in August, 2005; ...; `PAY_6` = the repayment status in April, 2005. The measurement scale for the repayment status is: -1 = pay duly; 1 = payment delay for one month; 2 = payment delay for two months; ...; 8 = payment delay for eight months; 9 = payment delay for nine months and above. \n",
    "* **`BILL_AMT1` - `BILL_AMT6`**: Amount of bill statement (NT dollar). `BILL_AMNT1` = amount of bill statement in September, 2005; `BILL_AMT2` = amount of bill statement in August, 2005; ...; `BILL_AMT6` = amount of bill statement in April, 2005. \n",
    "* **`PAY_AMT1` - `PAY_AMT6`**: Amount of previous payment (NT dollar). `PAY_AMT1` = amount paid in September, 2005; `PAY_AMT2` = amount paid in August, 2005; ...; `PAY_AMT6` = amount paid in April, 2005. \n",
    "\n",
    "Demographic variables will not be used as model inputs as is common in credit scoring models. However, demographic variables will be used after model training to test for disparate impact.\n",
    "\n",
    "Because h2o accepts both numeric and character inputs, some variables will be recoded into more transparent character values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import data and clean\n",
    "The credit card default data is available as an `.xls` file. Pandas reads `.xls` files automatically, so it's used to load the credit card default data and give the prediction target a shorter name: `DEFAULT_NEXT_MONTH`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import XLS file\n",
    "path = 'default_of_credit_card_clients.xls'\n",
    "data = pd.read_excel(path,\n",
    "                     skiprows=1)\n",
    "\n",
    "# remove spaces from target column name \n",
    "data = data.rename(columns={'default payment next month': 'DEFAULT_NEXT_MONTH'}) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assign modeling roles\n",
    "The shorthand name `y` is assigned to the prediction target. `X` is assigned to all other non-demographic input variables in the credit card default data except the row indentifier, `ID`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y = DEFAULT_NEXT_MONTH\n",
      "X = ['LIMIT_BAL', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']\n"
     ]
    }
   ],
   "source": [
    "# assign target and inputs for GBM\n",
    "y = 'DEFAULT_NEXT_MONTH'\n",
    "X = [name for name in data.columns if name not in [y, 'ID', 'AGE', 'EDUCATION', 'MARRIAGE', 'SEX']]\n",
    "print('y =', y)\n",
    "print('X =', X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Helper function for recoding values in the UCI credict card default data\n",
    "This simple function maps longer, more understandable character string values from the UCI credit card default data dictionary to the original integer values of the input variables found in the dataset. These character values can be used directly in h2o decision tree models, and the function returns the original Pandas DataFrame as an h2o object, an H2OFrame. H2o models cannot run on Pandas DataFrames. They require H2OFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "def recode_cc_data(frame):\n",
    "    \n",
    "    \"\"\" Recodes numeric categorical variables into categorical character variables\n",
    "    with more transparent values. \n",
    "    \n",
    "    Args:\n",
    "        frame: Pandas DataFrame version of UCI credit card default data.\n",
    "        \n",
    "    Returns: \n",
    "        H2OFrame with recoded values.\n",
    "        \n",
    "    \"\"\"\n",
    "    \n",
    "    # define recoded values\n",
    "    sex_dict = {1:'male', 2:'female'}\n",
    "    education_dict = {0:'other', 1:'graduate school', 2:'university', 3:'high school', \n",
    "                      4:'other', 5:'other', 6:'other'}\n",
    "    marriage_dict = {0:'other', 1:'married', 2:'single', 3:'divorced'}\n",
    "    \n",
    "    # recode values using Pandas apply() and anonymous function\n",
    "    frame['SEX'] = frame['SEX'].apply(lambda i: sex_dict[i])\n",
    "    frame['EDUCATION'] = frame['EDUCATION'].apply(lambda i: education_dict[i])    \n",
    "    frame['MARRIAGE'] = frame['MARRIAGE'].apply(lambda i: marriage_dict[i])           \n",
    "                \n",
    "    return h2o.H2OFrame(frame)\n",
    "\n",
    "data = recode_cc_data(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ensure target is handled as a categorical variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In h2o, a numeric variable can be treated as numeric or categorical. The target variable `DEFAULT_NEXT_MONTH` takes on values of `0` or `1`. To ensure this numeric variable is treated as a categorical variable, the `asfactor()` function is used to explicitly declare that it is a categorical variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data[y] = data[y].asfactor() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display descriptive statistics\n",
    "The h2o `describe()` function displays a brief description of the credit card default data. Basic descriptive statistics are displayed for numeric inputs and the target. There are no missing values in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows:30000\n",
      "Cols:20\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>       </th><th>LIMIT_BAL         </th><th>PAY_0                </th><th>PAY_2              </th><th>PAY_3               </th><th>PAY_4              </th><th>PAY_5               </th><th>PAY_6              </th><th>BILL_AMT1        </th><th>BILL_AMT2        </th><th>BILL_AMT3         </th><th>BILL_AMT4         </th><th>BILL_AMT5        </th><th>BILL_AMT6        </th><th>PAY_AMT1         </th><th>PAY_AMT2         </th><th>PAY_AMT3         </th><th>PAY_AMT4          </th><th>PAY_AMT5          </th><th>PAY_AMT6         </th><th>DEFAULT_NEXT_MONTH  </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>type   </td><td>int               </td><td>int                  </td><td>int                </td><td>int                 </td><td>int                </td><td>int                 </td><td>int                </td><td>int              </td><td>int              </td><td>int               </td><td>int               </td><td>int              </td><td>int              </td><td>int              </td><td>int              </td><td>int              </td><td>int               </td><td>int               </td><td>int              </td><td>enum                </td></tr>\n",
       "<tr><td>mins   </td><td>10000.0           </td><td>-2.0                 </td><td>-2.0               </td><td>-2.0                </td><td>-2.0               </td><td>-2.0                </td><td>-2.0               </td><td>-165580.0        </td><td>-69777.0         </td><td>-157264.0         </td><td>-170000.0         </td><td>-81334.0         </td><td>-339603.0        </td><td>0.0              </td><td>0.0              </td><td>0.0              </td><td>0.0               </td><td>0.0               </td><td>0.0              </td><td>                    </td></tr>\n",
       "<tr><td>mean   </td><td>167484.32266666673</td><td>-0.016700000000000118</td><td>-0.1337666666666667</td><td>-0.16619999999999988</td><td>-0.2206666666666661</td><td>-0.26620000000000105</td><td>-0.2910999999999988</td><td>51223.33090000043</td><td>49179.07516666629</td><td>47013.154799999655</td><td>43262.94896666659 </td><td>40311.40096666682</td><td>38871.76039999976</td><td>5663.580500000035</td><td>5921.163499999979</td><td>5225.681499999981</td><td>4826.076866666652 </td><td>4799.387633333336 </td><td>5215.50256666665 </td><td>                    </td></tr>\n",
       "<tr><td>maxs   </td><td>1000000.0         </td><td>8.0                  </td><td>8.0                </td><td>8.0                 </td><td>8.0                </td><td>8.0                 </td><td>8.0                </td><td>964511.0         </td><td>983931.0         </td><td>1664089.0         </td><td>891586.0          </td><td>927171.0         </td><td>961664.0         </td><td>873552.0         </td><td>1684259.0        </td><td>896040.0         </td><td>621000.0          </td><td>426529.0          </td><td>528666.0         </td><td>                    </td></tr>\n",
       "<tr><td>sigma  </td><td>129747.66156720246</td><td>1.1238015279973332   </td><td>1.1971859730345495 </td><td>1.1968675684465686  </td><td>1.1691386224023357 </td><td>1.1331874060027525  </td><td>1.149987625607897  </td><td>73635.86057552966</td><td>71173.76878252832</td><td>69349.38742703677 </td><td>64332.856133916444</td><td>60797.15577026471</td><td>59554.1075367459 </td><td>16563.28035402577</td><td>23040.87040205719</td><td>17606.96146980311</td><td>15666.159744032062</td><td>15278.305679144742</td><td>17777.46577543531</td><td>                    </td></tr>\n",
       "<tr><td>zeros  </td><td>0                 </td><td>14737                </td><td>15730              </td><td>15764               </td><td>16455              </td><td>16947               </td><td>16286              </td><td>2008             </td><td>2506             </td><td>2870              </td><td>3195              </td><td>3506             </td><td>4020             </td><td>5249             </td><td>5396             </td><td>5968             </td><td>6408              </td><td>6703              </td><td>7173             </td><td>                    </td></tr>\n",
       "<tr><td>missing</td><td>0                 </td><td>0                    </td><td>0                  </td><td>0                   </td><td>0                  </td><td>0                   </td><td>0                  </td><td>0                </td><td>0                </td><td>0                 </td><td>0                 </td><td>0                </td><td>0                </td><td>0                </td><td>0                </td><td>0                </td><td>0                 </td><td>0                 </td><td>0                </td><td>0                   </td></tr>\n",
       "<tr><td>0      </td><td>20000.0           </td><td>2.0                  </td><td>2.0                </td><td>-1.0                </td><td>-1.0               </td><td>-2.0                </td><td>-2.0               </td><td>3913.0           </td><td>3102.0           </td><td>689.0             </td><td>0.0               </td><td>0.0              </td><td>0.0              </td><td>0.0              </td><td>689.0            </td><td>0.0              </td><td>0.0               </td><td>0.0               </td><td>0.0              </td><td>1                   </td></tr>\n",
       "<tr><td>1      </td><td>120000.0          </td><td>-1.0                 </td><td>2.0                </td><td>0.0                 </td><td>0.0                </td><td>0.0                 </td><td>2.0                </td><td>2682.0           </td><td>1725.0           </td><td>2682.0            </td><td>3272.0            </td><td>3455.0           </td><td>3261.0           </td><td>0.0              </td><td>1000.0           </td><td>1000.0           </td><td>1000.0            </td><td>0.0               </td><td>2000.0           </td><td>1                   </td></tr>\n",
       "<tr><td>2      </td><td>90000.0           </td><td>0.0                  </td><td>0.0                </td><td>0.0                 </td><td>0.0                </td><td>0.0                 </td><td>0.0                </td><td>29239.0          </td><td>14027.0          </td><td>13559.0           </td><td>14331.0           </td><td>14948.0          </td><td>15549.0          </td><td>1518.0           </td><td>1500.0           </td><td>1000.0           </td><td>1000.0            </td><td>1000.0            </td><td>5000.0           </td><td>0                   </td></tr>\n",
       "<tr><td>3      </td><td>50000.0           </td><td>0.0                  </td><td>0.0                </td><td>0.0                 </td><td>0.0                </td><td>0.0                 </td><td>0.0                </td><td>46990.0          </td><td>48233.0          </td><td>49291.0           </td><td>28314.0           </td><td>28959.0          </td><td>29547.0          </td><td>2000.0           </td><td>2019.0           </td><td>1200.0           </td><td>1100.0            </td><td>1069.0            </td><td>1000.0           </td><td>0                   </td></tr>\n",
       "<tr><td>4      </td><td>50000.0           </td><td>-1.0                 </td><td>0.0                </td><td>-1.0                </td><td>0.0                </td><td>0.0                 </td><td>0.0                </td><td>8617.0           </td><td>5670.0           </td><td>35835.0           </td><td>20940.0           </td><td>19146.0          </td><td>19131.0          </td><td>2000.0           </td><td>36681.0          </td><td>10000.0          </td><td>9000.0            </td><td>689.0             </td><td>679.0            </td><td>0                   </td></tr>\n",
       "<tr><td>5      </td><td>50000.0           </td><td>0.0                  </td><td>0.0                </td><td>0.0                 </td><td>0.0                </td><td>0.0                 </td><td>0.0                </td><td>64400.0          </td><td>57069.0          </td><td>57608.0           </td><td>19394.0           </td><td>19619.0          </td><td>20024.0          </td><td>2500.0           </td><td>1815.0           </td><td>657.0            </td><td>1000.0            </td><td>1000.0            </td><td>800.0            </td><td>0                   </td></tr>\n",
       "<tr><td>6      </td><td>500000.0          </td><td>0.0                  </td><td>0.0                </td><td>0.0                 </td><td>0.0                </td><td>0.0                 </td><td>0.0                </td><td>367965.0         </td><td>412023.0         </td><td>445007.0          </td><td>542653.0          </td><td>483003.0         </td><td>473944.0         </td><td>55000.0          </td><td>40000.0          </td><td>38000.0          </td><td>20239.0           </td><td>13750.0           </td><td>13770.0          </td><td>0                   </td></tr>\n",
       "<tr><td>7      </td><td>100000.0          </td><td>0.0                  </td><td>-1.0               </td><td>-1.0                </td><td>0.0                </td><td>0.0                 </td><td>-1.0               </td><td>11876.0          </td><td>380.0            </td><td>601.0             </td><td>221.0             </td><td>-159.0           </td><td>567.0            </td><td>380.0            </td><td>601.0            </td><td>0.0              </td><td>581.0             </td><td>1687.0            </td><td>1542.0           </td><td>0                   </td></tr>\n",
       "<tr><td>8      </td><td>140000.0          </td><td>0.0                  </td><td>0.0                </td><td>2.0                 </td><td>0.0                </td><td>0.0                 </td><td>0.0                </td><td>11285.0          </td><td>14096.0          </td><td>12108.0           </td><td>12211.0           </td><td>11793.0          </td><td>3719.0           </td><td>3329.0           </td><td>0.0              </td><td>432.0            </td><td>1000.0            </td><td>1000.0            </td><td>1000.0           </td><td>0                   </td></tr>\n",
       "<tr><td>9      </td><td>20000.0           </td><td>-2.0                 </td><td>-2.0               </td><td>-2.0                </td><td>-2.0               </td><td>-1.0                </td><td>-1.0               </td><td>0.0              </td><td>0.0              </td><td>0.0               </td><td>0.0               </td><td>13007.0          </td><td>13912.0          </td><td>0.0              </td><td>0.0              </td><td>0.0              </td><td>13007.0           </td><td>1122.0            </td><td>0.0              </td><td>0                   </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data[X + [y]].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Train an H2O GBM classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split data into training and test sets for early stopping\n",
    "The credit card default data is split into training and test sets to monitor and prevent overtraining. Reproducibility is also an important factor in creating trustworthy models, and randomly splitting datasets can introduce randomness in model predictions and other results. A random seed is used here to ensure the data split is reproducible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data rows = 21060, columns = 25\n",
      "Test data rows = 8940, columns = 25\n"
     ]
    }
   ],
   "source": [
    "# split into training and validation\n",
    "train, test = data.split_frame([0.7], seed=12345)\n",
    "\n",
    "# summarize split\n",
    "print('Train data rows = %d, columns = %d' % (train.shape[0], train.shape[1]))\n",
    "print('Test data rows = %d, columns = %d' % (test.shape[0], test.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Establish monotonicity constraints based on Pearson correlation\n",
    "In general for high-stakes situations, it's best to use constrained machine learning models. Constrained models are easier to explain and to test for disparate impact. (Constrained models may also generalize better in low signal-to-noise scenarios.) Very complex models may be nearly impossible to test for disparate impact, as they can treat similar individuals quite differently, making it possible for local instances of discrimination to occur and not be detected by disparate impact testing. \n",
    "\n",
    "This GBM model will be constrained using the pairwise Pearson correlation between the inputs and the targets to establish the direction of the monotonicity constraints. H2o expects a dictionary with input variable names as keys and values of 1 for a positive monotonic constrain, 0 for no constraint, and -1 for a negative monotonic constraint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "           DEFAULT_NEXT_MONTH\n",
      "LIMIT_BAL           -0.151732\n",
      "PAY_0                0.318419\n",
      "PAY_2                0.250650\n",
      "PAY_3                0.220672\n",
      "PAY_4                0.201708\n",
      "PAY_5                0.193368\n",
      "PAY_6                0.183716\n",
      "BILL_AMT1           -0.023252\n",
      "BILL_AMT2           -0.019563\n",
      "BILL_AMT3           -0.021105\n",
      "BILL_AMT4           -0.018092\n",
      "BILL_AMT5           -0.012072\n",
      "BILL_AMT6           -0.011350\n",
      "PAY_AMT1            -0.085398\n",
      "PAY_AMT2            -0.062419\n",
      "PAY_AMT3            -0.069603\n",
      "PAY_AMT4            -0.055825\n",
      "PAY_AMT5            -0.051942\n",
      "PAY_AMT6            -0.047694\n",
      "\n",
      " {'LIMIT_BAL': -1, 'PAY_0': 1, 'PAY_2': 1, 'PAY_3': 1, 'PAY_4': 1, 'PAY_5': 1, 'PAY_6': 1, 'BILL_AMT1': -1, 'BILL_AMT2': -1, 'BILL_AMT3': -1, 'BILL_AMT4': -1, 'BILL_AMT5': -1, 'BILL_AMT6': -1, 'PAY_AMT1': -1, 'PAY_AMT2': -1, 'PAY_AMT3': -1, 'PAY_AMT4': -1, 'PAY_AMT5': -1, 'PAY_AMT6': -1}\n"
     ]
    }
   ],
   "source": [
    "# display Pearson correlation between inputs and targets\n",
    "print(pd.DataFrame(test.as_data_frame()[X + [y]].corr()[y]).iloc[:-1])\n",
    "\n",
    "# positive Pearson correlation w/ P_DEFAULT_NEXT_MONTH == positive constraint == 1\n",
    "# negative Pearson correlation w/ P_DEFAULT_NEXT_MONTH == negative constraint == -1\n",
    "names = list(test.as_data_frame()[X + [y]].corr()[y].index)[:-1]\n",
    "signs = list([int(i) for i in np.sign(test.as_data_frame()[X + [y]].corr()[y].values[:-1])])\n",
    "\n",
    "mc = {}\n",
    "for i, name in enumerate(names):\n",
    "    mc[name] = signs[i]\n",
    "\n",
    "print('\\n', mc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train h2o GBM classifier with monotonicity constraints\n",
    "Many tuning parameters must be specified to train a GBM using h2o. Typically a grid search would be performed to identify the best parameters for a given modeling task using the `H2OGridSearch` class. For brevity's sake, a previously-discovered set of good tuning parameters are specified here. Because gradient boosting methods typically resample training data, an additional random seed is also specified for the h2o GBM using the `seed` parameter to create reproducible predictions, error rates, and variable importance values. To avoid overfitting, the `stopping_rounds` parameter is used to stop the training process after the test error fails to decrease for 5 iterations. Monotonicity constraints are passed to the GBM using the `monotone_constraints` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gbm Model Build progress: |███████████████████████████████████████████████| 100%\n",
      "GBM Test AUC = 0.77\n"
     ]
    }
   ],
   "source": [
    "# initialize GBM model\n",
    "model = H2OGradientBoostingEstimator(ntrees=150,              # maximum 150 trees in GBM\n",
    "                                     max_depth=4,             # trees can have maximum depth of 4\n",
    "                                     sample_rate=0.9,         # use 90% of rows in each iteration (tree)\n",
    "                                     col_sample_rate=0.9,     # use 90% of variables in each iteration (tree)\n",
    "                                     #balance_classes=True,   # sample to balance 0/1 distribution of target - can help LOCO\n",
    "                                     stopping_rounds=5,       # stop if validation error does not decrease for 5 iterations (trees)\n",
    "                                     score_tree_interval=1,   # for reproducibility, set higher for bigger data\n",
    "                                     monotone_constraints=mc, # enforces monotonicity wrt each input\n",
    "                                     model_id='dia_gbm',      # for locating the model in Flow UI \n",
    "                                     seed=12345)              # for reproducibility\n",
    "\n",
    "# train a GBM model\n",
    "model.train(y=y, x=X, training_frame=train, validation_frame=test)\n",
    "\n",
    "# print AUC\n",
    "print('GBM Test AUC = %.2f' % model.auc(valid=True))\n",
    "\n",
    "# uncomment to see model details\n",
    "# print(model) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Shapley variable importance\n",
    "Shapley values are a locally-accurate and globally consistent variable importance metric. Instead of relying on traditional single-value variable importance measures, local Shapley values for each input will be calculated and aggregated below to get a more holistic and consisent measurement for the global importance of each input variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh0AAAIiCAYAAABsTHqfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3X+8X1V95/tXiEVLI0RDUUoRK97A\nWGGYeXzEQcZCGcFSPVevUaE3KVAgOa1cfJRH2wtjyNWSkDqDrXUGsTXhhxKCmgtTzCClag1IB4pv\nkIBXLr0YacPQ6FBgmgpiILl/7H3sl+NJcs73fLNzDuf1fDzOI2evvfZaa+ef836stfbes3bs2IEk\nSdKets/eHoAkSZoZDB2SJKkThg5JktQJQ4ckSeqEoUOSJHXC0CFJkjph6JAkSZ0wdEiSpE4YOiRJ\nUidesrcH8GKzfv36HUNDQ3t7GJIkdWXWeCs60yFJkjph6JAkSZ0wdEiSpE4YOiRJUicMHZIkqROG\nDkmS1AlDhyRJ6oShQ5IkdcLQIUmSOmHokCRJnTB0SJKkThg6JElSJwwdkiSpE4YOSZLUCUOHJEnq\nxKwdO3bs7TG8qJz0gb/zP1SSNCX95RWv2RPNzhpvRWc6JElSJwwdkiSpE4YOSZLUCUOHJEnqhKFD\nkiR14iV7ewC7UlUbgOOAbcDzwCZgRZIbeuqsBs4BTkhye1u2H3APsDbJ8p66hwAPAMNJ1u2m74uA\nDwIHAH8FLEnyyMBuTpKkGWY6zHQsTzIHmAdcD3y+quYDVNXLgdOBJ4DhkQuSPA0sBC6qqje1dWcB\nVwM3jSNwnAlcAPwqcBDwMHBTVU2H/y9JkqakafNHNMlzwBXAbOCotngR8CxwPrCgqub11L8XuARY\n0858nA8cTjN7sTtLgCuS3JfkB8BFwBE0sy6SJKkP0yZ0VNW+wHk0Sy0b2+IlwHXAOmArcOaoyy4D\nHgPWAiuAhUm2jqO7f0mzPANAkn8EvtOWS5KkPkyH0LG0qp4CHgXeBSxI8nBVHQscA1yVZBtwLU0I\n+bEk24EzgJOBy5PctbvO2mWYnwH+56hTTwH7T/ZmJEmaqaZD6Lg0ydwkByV5S5L1bfkw8M0k97XH\nVwJHVNWJvRcn2Qw8Cdw/ns6S7AB+QLOBtNdc4B/7vAdJkma8Kf30ys5U1f7AacA+VbWl59QOmtmO\nDZPsYiPwr4H/2tPf4fzzso4kSZqgaRk6aDaQbgeOBp7uKX8n8MmqOjDJ45No/9PAR6vqz2ieXFkJ\nPATcOYk2JUma0aZr6FgCrEqyqbewqq4BlgFnAR/rt/Ekn6mqnwNu5Z/f0/Hudo+IJEnqg5+2HzA/\nbS9Jmqr8tL0kSZoRpuvyyqRU1TLgwp2cPjmJezckSRowl1cGbP369TuGhob29jAkSeqKyyuSJGlq\nMXRIkqROGDokSVInDB2SJKkThg5JktQJQ4ckSeqEj8wOmG8kffHZQ2/wk6QXCx+ZlSRJU4uhQ5Ik\ndcLQIUmSOmHokCRJnTB0SJKkTvT1ldmq2gB8JcmKUeXXAM8lObc9fgQ4DHhzkrt76p0GfA64LcmJ\nvW0CtwG39DS7H/Aj4Ln2+OtJTt3F2D4CXAz8sC3aCnwR+O0kz4yquwi4FvhwkkvGc4+SJKk/XXza\n/kFgMXB3T9nitvwnJPk6MGfkuKoeBlYkuWYCfW5I8rb2+kNoQswy4EOj6i0BngDOrapLkzw/gT4k\nSdIEdLG8cg3w3qqaA1BVrwOOAW7soG+S/HfgVuCNveVV9S+AtwJnAgcDO509kSRJk9dF6HgMuB34\ntfb4XGAN/7z8sUdV1WE0geKOUaeGgQeS/FfgSzSzHpIkaQ/paiPpKmBJVb0EOKs93pNOqKqnquof\ngUdo9nVcNXKyql4G/HpP2ZXAr1bVz+/hcUmSNGN1FTpuAV4N/F/AI0n+nz3c321J5ibZH9ifZj/J\nX1XVS9vz76PZN7KmPf4S8H2aWRhJkrQHdBI62g2aV9E8VfLpLvrs6XsrzczKfP55X8cwMBv4VlVt\nAR4FXgmcU1WzuxyfJEkzxWSeXnlJu0wxXn8MfJ2f3FuxR1XVfsA5wA+A71TVG4Djgf8V+EZP1YOA\ne4BfBda3ZT9xj0k62YsiSdKLzWRCx4fbn1630swa/IQkT9K8h6MLJ1bVP7W/bwPuB341yVNV9fvA\nvUnWj7pmS1Wto5kFGTn3E/dYVQcn2bIHxy5J0ouSn7YfMD9t/+Ljp+0laZf8tL0kSZpaungj6UBV\n1ULgT3dyejjJdV2OZ7QLTt3I0NDQ3hyCJElT0rQLHW2o2KvBQpIkTZzLK5IkqROGDkmS1AlDhyRJ\n6oShQ5IkdcLQIUmSOmHokCRJnfCNpAPmG0mnD980KkkD4RtJJUnS1GLokCRJnTB0SJKkThg6JElS\nJwwdkiSpE1P6g29VtQE4DtgGPA9sAlYkuaGnzmrgHOCEJLe3ZfsB9wBrkyzvqXsI8ADN12jX7aLf\n/xP4NeB1wDPABuD3kmwe5P1JkjSTTIeZjuVJ5gDzgOuBz1fVfICqejlwOvAEMDxyQZKngYXARVX1\nprbuLOBq4KZdBY7WTwHnAa8C5gM/Av5skDclSdJMMx1CBwBJngOuAGYDR7XFi4BngfOBBVU1r6f+\nvcAlwJp25uN84HDgg+Po69Ik/y3JD5P8I/CHwL+uqv0HeU+SJM0k0yZ0VNW+NLMP24CNbfES4Dpg\nHbAVOHPUZZcBjwFrgRXAwiRb++j+3wGPtAFEkiT1YTqEjqVV9RTwKPAuYEGSh6vqWOAY4Kok24Br\naULIjyXZDpwBnAxcnuSuiXZeVW8Ffh/4rcndhiRJM9uU3kjaujTJijHKh4FvJrmvPb4SuKCqTkyy\nYaRSks1V9SRw/0Q7rqoTgRuBc5L8+YRHLkmSfmw6hI6f0O6tOA3Yp6q29JzaQTPbsWEAffwqzbLM\nmUlummx7kiTNdNNheWUsi4DtwBtpllhGfpbQbCg9cDKNV9X7aZ6U+TUDhyRJgzEtZzpowsWqJJt6\nC6vqGmAZcBbwsUm0/4fAzwDrqqq3fH6SxybRriRJM5afth8wP20/ffhpe0kaCD9tL0mSppbpurwy\nKVW1DLhwJ6dPTnJnv21fcOpGhoaG+r1ckqQXrRkZOtrvsSzfbUVJkjQwLq9IkqROGDokSVInDB2S\nJKkThg5JktQJQ4ckSeqELwcbMF8ONli+wEuSpjxfDiZJkqYWQ4ckSeqEoUOSJHXC0CFJkjph6JAk\nSZ0wdEiSpE5M6Q++VdUG4DhgG/A8sAlYkeSGnjqrgXOAE5Lc3pbtB9wDrG0/7jZS9xDgAWA4ybpd\n9PvbwHnAQcBzwN3A7yX51kBvUJKkGWQ6zHQsTzIHmAdcD3y+quYDVNXLgdOBJ4DhkQuSPA0sBC6q\nqje1dWcBVwM37SpwtNYDxyU5ADgY+Bpw80DvSpKkGWY6hA4AkjwHXAHMBo5qixcBzwLnAwuqal5P\n/XuBS4A17czH+cDhwAfH0dd3kjzeHs4CtgOHtu1IkqQ+TOnllV5VtS/Nksc2YGNbvAS4DlgHfAI4\nE/ijnssuA34FWAucBJySZOs4+zsBuAk4ANgBfLSdQZEkSX2YDjMdS6vqKeBR4F3AgiQPV9WxwDHA\nVUm2AdfShJAfS7IdOAM4Gbg8yV3j7TTJbUnmAq8EfodmX4ckSerTdAgdlyaZm+SgJG9Jsr4tHwa+\nmeS+9vhK4IiqOrH34iSbgSeB+/vpPMmTwH8CPjuyl0SSJE3ctFle6VVV+wOnAftU1ZaeUztoZjs2\nDLjLfYB9gdcBfzPgtiVJmhGmw0zHWBbRbO58I80Sy8jPEpoNpQdOpvGq+q328Vratq4A/gn4xmTa\nlSRpJpuWMx004WJVkk29hVV1DbAMOAv42CTafzOwrKoOALYCfw28Lck/TKJNSZJmtFk7duzY22N4\nUTnpA3/nf+gA/eUVr9nbQ5Ak7dqs8VacrssrkiRpmpmuyyuTUlXLgAt3cvrkJHf22/YFp25kaGio\n38slSXrRmpGho/0ey/LdVpQkSQPj8ookSeqEoUOSJHXC0CFJkjph6JAkSZ0wdEiSpE74crAB8+Vg\n4+NLvyTpRcOXg0mSpKnF0CFJkjph6JAkSZ0wdEiSpE4YOiRJUicm9O2VqtoAHAdsA54HNgErktzQ\nU2c1cA5wQpLb27L9gHuAte13T0bqHgI8AAwnWTeO/s8CrgaWJlk56twdwPHAgiQ39pQfD9wBfCfJ\n66vqIeCQ9vS+NMHrhz1NzQc+AFw0qvwTSZbuboySJGls/cx0LE8yB5gHXA98vqrmA1TVy4HTgSeA\n4ZELkjwNLAQuqqo3tXVn0QSIm8YTOFpL2rYXV9VYY38QWDyqbHFbPjKWI5LMae9hJbBh5Lj9eayt\n+pVR5QYOSZImoe/llSTPAVcAs4Gj2uJFwLPA+cCCqprXU/9e4BJgTTvzcT5wOPDB8fRXVUfRzLKc\nARwKnDJGtXXAsVV1WHvNAcC7gc9M9P4kSdJg9R06qmpf4DyapZaNbfES4DqaP/5bgTNHXXYZ8Biw\nFlgBLEyydZxdDgPfTHIzcGvb12jPAJ8Dzm6PFwJfAb43zj56HV9Vj1fVpqr606o6sI82JElSq5/Q\nsbSqngIeBd5Fs4fi4ao6FjgGuCrJNuBaRgWDJNtpZipOBi5Pctd4OmxnRhYBV7VFVwJDVXXwGNVX\nAWdX1WyapZVVE71BmuDyBuBngbcBrwX+Sx/tSJKkVj+h49Ikc5MclOQtSda35SMzEfe1x1cCR1TV\nib0XJ9kMPAncP4E+TwNeRjOLArCeZm/HOaMrtv1vAT4MvAL48gT6GWnjW0k2J9mRZBNNePq3I8s2\nkiRp4gbyyGxV7U8TDI6sqi1VtQX4KrCDsZdBJmqYZu/Ig23bm4G5wLk72VD6aeBiYHU7uzJZI22M\n+/3ykiTphSb0yOwuLKL5w3w08HRP+TuBT1bVgUke76fhqjoaeDPwDuDenlMHA98A3g7cMuqyNcB3\ngfTZ5wLgtiSPV9XPA58C7k7ySD/tSZKkwYWOJcCqdinix6rqGmAZcBbwsT7bHqb5g/+lUeVbqurG\n9vwLQkeSZ2g2kPbrfcCfVNXPAP9As3H13Em0J0nSjOen7QfMT9uPj5+2l6QXDT9tL0mSppZBLa9M\nSlUtAy7cyemTk9zZ5Xgm44JTNzI0NLS3hyFJ0pQzJUJH+z2W5butKEmSpi2XVyRJUicMHZIkqROG\nDkmS1AlDhyRJ6oShQ5IkdcLQIUmSOuEbSQfsxfZGUt8cKknaDd9IKkmSphZDhyRJ6oShQ5IkdcLQ\nIUmSOmHokCRJnZgSH3zbmaraABwHbAOeBzYBK5Lc0FNnNXAOcEKS29uy/YB7gLXtx+RG6h4CPAAM\nJ1m3i35XABcBP+wp/kSSpQO6NUmSZpzpMNOxPMkcYB5wPfD5qpoPUFUvB04HngCGRy5I8jSwELio\nqt7U1p0FXA3ctKvA0eMrSeb0/Bg4JEmahOkQOgBI8hxwBTAbOKotXgQ8C5wPLKiqeT317wUuAda0\nMx/nA4cDH+xy3JIkqTGll1d6VdW+wHk0Sy0b2+IlwHXAOuATwJnAH/VcdhnwK8Ba4CTglCRbx9nl\n8VX1OPCPwJeBpUken+x9SJI0U02HmY6lVfUU8CjwLmBBkoer6ljgGOCqJNuAa2lCyI8l2Q6cAZwM\nXJ7krnH2+TngDcDPAm8DXgv8lwHciyRJM9Z0mOm4NMmKMcqHgW8mua89vhK4oKpOTLJhpFKSzVX1\nJHD/eDtM8q2ew01VtQR4pKoOS/K3E78FSZI0HULHT6iq/YHTgH2qakvPqR00sx0bBtzl9vbfcb9f\nXpIkvdB0WF4ZyyKaIPBGmiWWkZ8lNBtKD5xM41X14zaq6ueBTwF3J3lkMu1KkjSTTcuZDppwsSrJ\npt7CqroGWAacBXxsEu2/D/iTqvoZ4B+AW4FzJ9GeJEkznp+2HzA/bS9JmmH8tL0kSZpapuvyyqRU\n1TLgwp2cPjnJnf22fcGpGxkaGur3ckmSXrRmZOhov8eyfLcVJUnSwLi8IkmSOmHokCRJnTB0SJKk\nThg6JElSJwwdkiSpE4YOSZLUCd9IOmAvpjeS+jZSSdI4+EZSSZI0tRg6JElSJwwdkiSpE4YOSZLU\nCUOHJEnqxG4/+FZVG4DjgG1t0Rbg8iR/3J5/BLg4yZqqei3wXeDQJI+O0dYO4K1J7pjoQKvqNW3b\ntyU5adS5jwAfBj6V5AM95S8DHgNeAfwCcBGwqD29D/DTwA96mhoGNgBXAMcArwF+PcmaiY5XkiS9\n0HhnOpYnmZNkDs0f7Uur6pQ9OK6xnAs8BfxyVc0f4/zfAKdX1X49Ze+lCUkAJPnNnvs4pS2b0/Nz\nHbAd+Avgfwd+IjhJkqT+THh5JcldwLeBNw5+OGOrqtnA2cAfAN8CloxRbTNwF/D+nrLFwKqJ9JXk\n75N8MslfAc/3N2JJkjTahEJHVc2qquOBI4E798yQxjQEvAq4FrgKOLOqXjpGvVU0QYN2NuRI4Kau\nBilJknZuvKFjaVU9RbP/4Q7gOuDuPTaqn7QEuDnJ92iCx/7Ae8aotx54XVX9YnvNZ4EfdTZKSZK0\nU+MNHZcmmZtkP+BQ4A00Mw57XFUdBrx9pL8kjwNfpNn0+QJJngOuAc4DzgBWdzFGSZK0e/3s6XgU\n+AJjzzTsCYtpxrm6qrZU1RaaEHJCVR0xRv1VNIHk20ke6miMkiRpN3b7yOxoVfVq4H3Axl1Ue2n7\nuOqI55OMPHK776hz25OMuQRSVS8BfgP4KPCJUae/RrOE8ju9hUk2VdUvAX+/25vZiZ7xzQJ+qj1+\nrp1JkSRJfRhv6FhWVRe1v/8AuA343V3Uf3jU8a3Ar7S/f3XUuYdoNnyOZQh4JfDxJN/vPVFVHwdW\nVtWHRl/UPnkyGc/0/H5V+/P7wEcm2a4kSTOWn7YfMD9tL0maYfy0vSRJmlomvKdj0KrqrcAtOzm9\nMsnKLsczWRecupGhoaG9PQxJkqacvR46knwdmLO3xyFJkvYsl1ckSVInDB2SJKkThg5JktQJQ4ck\nSeqEoUOSJHXCl4MN2HR/OZgvBJMkTZAvB5MkSVOLoUOSJHXC0CFJkjph6JAkSZ0wdEiSpE4YOiRJ\nUif2+gffdqWqNgDHAduA54FNwIokN/TUWQ2cA5yQ5Pa2bD/gHmBtkuU9dQ8BHgCGk6wb5xhuAN4D\nHJfkrkHclyRJM9F0mOlYnmQOMA+4Hvh8Vc0HqKqXA6cDTwDDIxckeRpYCFxUVW9q684CrgZumkDg\nOA04YID3IknSjDUdQgcASZ4DrgBmA0e1xYuAZ4HzgQVVNa+n/r3AJcCadubjfOBw4IPj6a+qfhb4\nKD1hRpIk9W/ahI6q2hc4j2apZWNbvAS4DlgHbAXOHHXZZcBjwFpgBbAwydZxdnkF8EfA305u5JIk\nCaZH6FhaVU8BjwLvAhYkebiqjgWOAa5Ksg24liaE/FiS7cAZwMnA5ePdk1FV7wUOAT45uNuQJGlm\nmw6h49Ikc5MclOQtSda35cPAN5Pc1x5fCRxRVSf2XpxkM/AkcP94OmuXaD4OnNuGFkmSNABT+umV\nnamq/YHTgH2qakvPqR00sx0bJtH8vwJeDdxeVb3lf15Vlye5eBJtS5I0Y03L0EGzgXQ7cDTwdE/5\nO4FPVtWBSR7vs+2vA7/QczwbeAT4deD2PtuUJGnGm66hYwmwKsmm3sKqugZYBpwFfKyfhpM8S7N/\nZKTNkf+j/5Hkf/bTpiRJglk7duzY22N4UTnpA383rf9D//KK1+ztIUiSppdZ4604HTaSSpKkF4Hp\nurwyKVW1DLhwJ6dPTnJnv21fcOpGhoaG+r1ckqQXrRkZOtrvsSzfbUVJkjQwLq9IkqROGDokSVIn\nDB2SJKkThg5JktQJQ4ckSeqELwcbMF8OJkmaYXw5mCRJmloMHZIkqROGDkmS1AlDhyRJ6oShQ5Ik\ndWJKf3ulqjYAxwHbgOeBTcCKJDf01FkNnAOckOT2tmw/4B5gbfudlZG6hwAPAMNJ1u2m7wL+A/Bm\n4EfAhiTvGdzdSZI0s0yHmY7lSeYA84Drgc9X1XyAqno5cDrwBDA8ckGSp4GFwEVV9aa27izgauCm\ncQSOXwS+CnwOOAh4FbBywPclSdKMMh1CBwBJngOuAGYDR7XFi4BngfOBBVU1r6f+vcAlwJp25uN8\n4HDgg+Po7iPA+iSrkjydZFuSDOxmJEmagaZN6KiqfYHzaJZaNrbFS4DrgHXAVuDMUZddBjwGrAVW\nAAuTbB1Hd78MfL+qbq+qf6iqv66qtw3gNiRJmrGmQ+hYWlVPAY8C7wIWJHm4qo4FjgGuSrINuJYm\nhPxYku3AGcDJwOVJ7tpdZ+0yzDzgXOBDwKuBPwG+WFWvHdhdSZI0w0yH0HFpkrlJDkryliTr2/Jh\n4JtJ7muPrwSOqKoTey9Oshl4Erh/PJ0l2QH8E3BjkjvapZWraTaxnjKA+5EkaUaa0k+v7ExV7Q+c\nBuxTVVt6Tu2gme3YMMku7mvbGm1af1dFkqS9aVqGDpoNpNuBo4Gne8rfCXyyqg5M8vgk2r8C+NOq\n+lPgbponYV4L/MUk2pQkaUabrqFjCbAqyabewqq6BlgGnAV8rN/Gk1xfVQcBXwDmAg8C70jyt/22\nKUnSTOen7QfMT9tLkmYYP20vSZKmlum6vDIpVbUMuHAnp09Ocme/bV9w6kaGhob6vVySpBetGRk6\n2u+xLN9tRUmSNDAur0iSpE4YOiRJUicMHZIkqROGDkmS1AlDhyRJ6oShQ5IkdcI3kg7YVHwjqW8Z\nlSTtQb6RVJIkTS2GDkmS1AlDhyRJ6oShQ5IkdcLQIUmSOjGhD75V1QbgOGAb8DywCViR5IaeOquB\nc4ATktzelu0H3AOsbT+2NlL3EOABYDjJunH0fxZwNbA0ycpR5+4AjgcWJLmxp/x44A7gO0leX1UP\nAYe0p/elCV4/7GlqfpLHqqqA/wC8GfgRsCHJe3Y3RkmSNLZ+ZjqWJ5kDzAOuBz5fVfMBqurlwOnA\nE8DwyAVJngYWAhdV1ZvaurNoAsRN4wkcrSVt24uraqyxPwgsHlW2uC0fGcsRSea097CSJkzM6fl5\nrKp+Efgq8DngIOBVbV1JktSnvpdXkjwHXAHMBo5qixcBzwLnAwuqal5P/XuBS4A17czH+cDhwAfH\n019VHUUzy3IGcChwyhjV1gHHVtVh7TUHAO8GPjPB2/sIsD7JqiRPJ9mWJBNsQ5Ik9eg7dFTVvsB5\nNEstG9viJcB1NH/8twJnjrrsMuAxYC2wAliYZOs4uxwGvpnkZuDWtq/RnqGZnTi7PV4IfAX43jj7\nGPHLwPer6vaq+oeq+uuqetsE25AkST36CR1Lq+op4FHgXTR7KB6uqmOBY4CrkmwDrmVUMEiynWam\n4mTg8iR3jafDdmZkEXBVW3QlMFRVB49RfRVwdlXNpllaWTWRm2uXfeYB5wIfAl4N/Anwxap67UTa\nkiRJ/6yf0HFpkrlJDkryliTr2/KRmYj72uMrgSOq6sTei5NsBp4E7p9An6cBL6OZRQFYT7O345zR\nFdv+twAfBl4BfHkC/ZBkB/BPwI1J7miXVq6m2TQ71pKOJEkah4E8MltV+9MEgyOraktVbaHZiLmD\nsZdBJmqYZu/Ig23bm4G5wLk72VD6aeBiYHU7uzJR99GMfbQp910VSZKmiwk9MrsLi4DtwNHA0z3l\n7wQ+WVUHJnm8n4ar6miax1bfAdzbc+pg4BvA24FbRl22Bvgu0O/mzyuAP62qPwXuptkb8lrgL/ps\nT5KkGW9QoWMJsCrJpt7CqroGWAacBXysz7aHgbuTfGlU+ZaqurE9/4LQkeQZmg2kfUlyfVUdBHyB\nZkblQeAdSf623zYlSZrp/LT9gPlpe0nSDOOn7SVJ0tQyqOWVSamqZcCFOzl9cpI7uxzPZFxw6kaG\nhob29jAkSZpypkToaL/Hsny3FSVJ0rTl8ookSeqEoUOSJHXC0CFJkjph6JAkSZ0wdEiSpE4YOiRJ\nUid8I+mA+UZSSdIM4xtJJUnS1GLokCRJnTB0SJKkThg6JElSJwwdkiSpExP64FtVbQCOA7YBzwOb\ngBVJbuipsxo4Bzghye1t2X7APcDa9uNuI3UPAR4AhpOsG0f/ZwFXA0uTrBx17g7geGBBkht7yo8H\n7gC+k+T1VfUQcEh7el+a4PXDnqbmA28ELgKOAV4BHJxky+7GJ0mSdq6fmY7lSeYA84Drgc9X1XyA\nqno5cDrwBDA8ckGSp4GFwEVV9aa27iyaAHHTeAJHa0nb9uKqGmvsDwKLR5UtbstHxnJEkjntPawE\nNowctz+PAf8EXAOcNc5xSZKk3eh7eSXJc8AVwGzgqLZ4EfAscD6woKrm9dS/F7gEWNPOfJwPHA58\ncDz9VdVRNLMsZwCHAqeMUW0dcGxVHdZecwDwbuAzE7y3/5bks8C3J3KdJEnaub5DR1XtC5xHs9Sy\nsS1eAlxH88d/K3DmqMsuAx4D1gIrgIVJto6zy2Hgm0luBm5t+xrtGeBzwNnt8ULgK8D3xtmHJEna\nQ/oJHUur6ingUeBdNHsoHq6qY2n2QFyVZBtwLaOCQZLtNDMVJwOXJ7lrPB22MyOLgKvaoiuBoao6\neIzqq4Czq2o2zdLKqoneoCRJGrx+QselSeYmOSjJW5Ksb8tHZiLua4+vBI6oqhN7L06yGXgSuH8C\nfZ4GvIxmFgVgPc3ejnNGV2z73wJ8mGYT6Jcn0I8kSdpDBvLIbFXtTxMMjqyqLVW1BfgqsIOxl0Em\naphm78iDbdubgbnAuTvZUPpp4GJgdTu7IkmS9rIJPTK7C4uA7cDRwNM95e8EPllVByZ5vJ+Gq+po\n4M3AO4B7e04dDHwDeDtwy6jL1gDfBdJnn/vQPE67b1v00qp6GfBskin3QTdJkqaDQYWOJcCqJJt6\nC6vqGmAZzaOnH+uz7WHg7iRfGlW+papubM+/IHQkeYZmA2m/TuKFyzKPtP++leadH5IkaYL8tP2A\n+Wl7SdIM46ftJUnS1DKo5ZVJqaplwIU7OX1ykju7HM9kXHDqRoaGhvb2MCRJmnKmROhov8eyfLcV\nJUnStOXyiiRJ6oShQ5IkdcKrDa5OAAAgAElEQVTQIUmSOmHokCRJnTB0SJKkTvhysAGbKi8H84Vg\nkqSO+HIwSZI0tRg6JElSJwwdkiSpE4YOSZLUCUOHJEnqxIS+vVJVG4DjgG3A88AmYEWSG3rqrAbO\nAU5Icntbth9wD7C2/c7KSN1DgAeA4STrxtH/WcDVwNIkK0eduwM4HliQ5Mae8uOBO4DvJHl9VT0E\nHNKe3pcmeP2wp6n5wCLg14DXAc8AG4DfS7J5d2OUJElj62emY3mSOcA84Hrg81U1H6CqXg6cDjwB\nDI9ckORpYCFwUVW9qa07iyZA3DSewNFa0ra9uKrGGvuDwOJRZYvb8pGxHJFkTnsPK4ENI8ftz2PA\nTwHnAa+iCSE/Av5snGOUJElj6Ht5JclzwBXAbOCotngR8CxwPrCgqub11L8XuARY0858nA8cDnxw\nPP1V1VE0syxnAIcCp4xRbR1wbFUd1l5zAPBu4DMTvLdLk/y3JD9M8o/AHwL/uqr2n0g7kiTpn/Ud\nOqpqX5rZgG3AxrZ4CXAdzR//rcCZoy67DHgMWAusABYm2TrOLoeBbya5Gbi17Wu0Z4DPAWe3xwuB\nrwDfG2cfO/PvgEfaACJJkvrQT+hYWlVPAY8C76LZQ/FwVR0LHANclWQbcC2jgkGS7TQzFScDlye5\nazwdtjMji4Cr2qIrgaGqOniM6quAs6tqNs3SyqqJ3uCovt8K/D7wW5NpR5Kkma6f0HFpkrlJDkry\nliTr2/KRmYj72uMrgSOq6sTei9vNmE8C90+gz9OAl9HMogCsp9nbcc7oim3/W4APA68AvjyBfl6g\nHftNwDlJ/rzfdiRJ0oAemW33OpwGHFlVW6pqC/BVYAdjL4NM1DDN3pEH27Y3A3OBc3eyofTTwMXA\n6nZ2ZcKq6ldpNo/+RpIv9DdsSZI0YkKPzO7CImA7cDTwdE/5O4FPVtWBSR7vp+GqOhp4M/AO4N6e\nUwcD3wDeDtwy6rI1wHeB9Nnn+2mWZU5PMrptSZLUh0GFjiXAqiSbegur6hpgGXAW8LE+2x4G7k7y\npVHlW6rqxvb8C4JBkmdoNpD26w+BnwHWVVVv+fz2kVpJkjRBftp+wPy0vSRphvHT9pIkaWoZ1PLK\npFTVMuDCnZw+OcmdXY5HkiQNnssrA7Z+/fodQ0NDe3sYkiR1xeUVSZI0tRg6JElSJwwdkiSpE4YO\nSZLUCUOHJEnqhKFDkiR1wkdmB8w3kkqSZhgfmZUkSVOLoUOSJHXC0CFJkjph6JAkSZ0wdEiSpE5M\n6CuzVbUBOA7YBjwPbAJWJLmhp85q4BzghCS3t2X7AfcAa5Ms76l7CPAAMJxk3Tj6Pwu4GliaZOWo\nc3cAxwMLktzYU348cAfwnSSvr6qHgEPa0/vSBK8f9jQ1H3g/cB5wEPAccDfwe0m+tbsxSpKksfUz\n07E8yRxgHnA98Pmqmg9QVS8HTgeeAIZHLkjyNLAQuKiq3tTWnUUTIG4aT+BoLWnbXlxVY439QWDx\nqLLFbfnIWI5IMqe9h5XAhpHj9ucxYD1wXJIDgIOBrwE3j3OMkiRpDH0vryR5DrgCmA0c1RYvAp4F\nzgcWVNW8nvr3ApcAa9qZj/OBw4EPjqe/qjqKZpblDOBQ4JQxqq0Djq2qw9prDgDeDXxmgvf2nSSP\nt4ezgO3Aoe24JUlSH/oOHVW1L80SxDZgY1u8BLiO5o//VuDMUZddBjwGrAVWAAuTbB1nl8PAN5Pc\nDNza9jXaM8DngLPb44XAV4DvjbOPH6uqE6rqKZqll/8IfLSdsZEkSX3oJ3Qsbf8YPwq8i2YPxcNV\ndSxwDHBVkm3AtYwKBkm208xUnAxcnuSu8XTYzjAsAq5qi64Ehqrq4DGqrwLOrqrZNEsrqyZ6g+1Y\nb0syF3gl8Ds0+zokSVKf+gkdlyaZm+SgJG9Jsr4tH5mJuK89vhI4oqpO7L04yWbgSeD+CfR5GvAy\nmlkUaPZcPEGzYfUF2v63AB8GXgF8eQL9/IQkTwL/CfjsyN4VSZI0cQN5ZLaq9qcJBkdW1Zaq2gJ8\nFdjB2MsgEzVMs3fkwbbtzcBc4NydbCj9NHAxsLqdXZmsfWiedHndANqSJGlGGtR7OhbRbLZ8I80S\ny8jPEpoNpQf223BVHQ28mWYpp7ftfwP8PPD2MS5bQ7PR9PI++/yt9nFe2rFfAfwT8I1+2pMkSRN8\nT8cuLAFWJdnUW1hV1wDLgLOAj/XZ9jBwd5IvjSrfUlU3tudv6T2R5BmaDaT9ejOwrH36ZSvw18Db\nkvzDJNqUJGlG89P2A+an7SVJM4yftpckSVPLoJZXJqWqlgEX7uT0yUnu7HI8k3HBqRsZGhra28OQ\nJGnKmRKho/0ey/LdVpQkSdOWyyuSJKkThg5JktQJQ4ckSeqEoUOSJHXC0CFJkjph6JAkSZ3wjaQD\n1vUbSX3zqCRpL/ONpJIkaWoxdEiSpE4YOiRJUicMHZIkqROGDkmS1IlxffCtqjYAxwHb2qItwOVJ\n/rg9/whwcZI1VfVa4LvAoUkeHaOtHcBbk9wx0cFW1Wvatm9LctKocx8BPgx8KskHespfBjwGvAL4\nBeAiYFF7eh/gp4Ef9DQ1DPx34Gujyu9P8paJjlmSJDUm8pXZ5UlWAFTVvwG+WlXfTvIXe2ZoYzoX\neAr45aqan+RvRp3/G+D0qvrdJE+3Ze+lCUmvAEjym8BvAlTVvwW+nmRObyNVdSLw/OhySZLUv76W\nV5LcBXwbeONgh7NzVTUbOBv4A+BbwJIxqm0G7gLe31O2GFi1xwcoSZJ2acKho6pmVdXxwJHAnYMf\n0k4NAa8CrgWuAs6sqpeOUW8VTdCgqubTjPOmPvqbXVWbq2pLVd1cVf+yz3FLkiQmFjqWVtVTNPsc\n7gCuA+7eI6Ma2xLg5iTfowke+wPvGaPeeuB1VfWL7TWfBX40wb7+X+AYmj0gRwL3A39ZVT/X59gl\nSZrxJhI6Lk0yN8l+wKHAG2hmHPa4qjoMePtIf0keB75Is+nzBZI8B1wDnAecAayeaH9JtiTZmOS5\nJE8l+ffAE8Cpfd+EJEkzXL97Oh4FvsDYMw17wmKasa5ulzu20ISQE6rqiDHqr6IJJN9O8tCAxrCd\nCbxfXpIkvdBEnl75sap6NfA+YOMuqr20fVx1xPNJRh653XfUue1JxlwCqaqXAL8BfBT4xKjTX6NZ\nQvmd3sIkm6rql4C/3+3NjN3nScDfAZuA/YDfpdlPcms/7UmSpImFjmVVdVH7+w+A22j+GO/Mw6OO\nbwV+pf39q6POPUSzd2IsQ8ArgY8n+X7viar6OLCyqj40+qIkf7WLse3OvwSuBg6kudd7gZOTbJ5E\nm5IkzWh+2n7A/LS9JGmG8dP2kiRpaulrT8egVdVbgVt2cnplkpVdjmcyLjh1I0NDQ3t7GJIkTTlT\nInQk+TrgK8clSXoRc3lFkiR1wtAhSZI6YeiQJEmdMHRIkqROGDokSVInfDnYgO2Jl4P5AjBJ0hTm\ny8EkSdLUYuiQJEmdMHRIkqROGDokSVInDB2SJKkThg5JktSJCX3wrao2AMcB24DngU3AiiQ39NRZ\nDZwDnJDk9rZsP+AeYG2S5T11DwEeAIaTrBtH/2cBVwNLR395tqruAI4HFiS5saf8eOAO4DtJXl9V\nDwGHtKf3pQleP+xpan6Sx3quvwF4D3Bckrt2N0ZJkjS2fmY6lieZA8wDrgc+X1XzAarq5cDpwBPA\n8MgFSZ4GFgIXVdWb2rqzaALETeMJHK0lbduLq2qssT8ILB5VtrgtHxnLEUnmtPewEtgwctz+9AaO\n04ADxjk2SZK0C30vryR5DrgCmA0c1RYvAp4FzgcWVNW8nvr3ApcAa9qZj/OBw4EPjqe/qjqKZpbl\nDOBQ4JQxqq0Djq2qw9prDgDeDXxmovdXVT8LfJSe8CRJkvrXd+ioqn2B82iWWja2xUuA62j++G8F\nzhx12WXAY8BaYAWwMMnWcXY5DHwzyc3ArW1foz0DfA44uz1eCHwF+N44++h1BfBHwN/2ca0kSRql\nn9CxtKqeAh4F3kWzh+LhqjoWOAa4Ksk24FpGBYMk22lmKk4GLh/vHol2ZmQRcFVbdCUwVFUHj1F9\nFXB2Vc2mWVpZNdEbrKr30uz7+OREr5UkSWPrJ3RcmmRukoOSvCXJ+rZ8ZCbivvb4SuCIqjqx9+Ik\nm4Engfsn0OdpwMtoZlEA1tPs7ThndMW2/y3Ah4FXAF+eQD+0S0IfB85tQ5IkSRqAgTwyW1X70wSD\nI6tqS1VtAb4K7GDsZZCJGqbZO/Jg2/ZmYC5w7k42lH4auBhY3Udw+FfAq4Hbq+pxmgAD8OdVtaKv\n0UuSpIk9MrsLi4DtwNHA0z3l7wQ+WVUHJnm8n4ar6mjgzcA7gHt7Th0MfAN4O3DLqMvWAN8F0keX\nXwd+oed4NvAI8OvA7X20J0mSGFzoWAKsSrKpt7CqrgGWAWcBH+uz7WHg7iRfGlW+papubM+/IHQk\neYZmA+mEJXmWZr8KAFU18n/0P5L8z37alCRJMGvHjh17ewwvKid94O8G/h/6l1e8ZtBNSpI0KLPG\nW9HXoEuSpE4ManllUqpqGXDhTk6fnOTOLsczGRecupGhoaG9PQxJkqacKRE62u+xLN9tRUmSNG25\nvCJJkjph6JAkSZ0wdEiSpE4YOiRJUicMHZIkqRO+HGzAfDmYJGmG8eVgkiRpajF0SJKkThg6JElS\nJwwdkiSpE4YOSZLUiXF9e6WqNgDHAdvaoi3A5Un+uD3/CHBxkjVV9Vrgu8ChSR4do60dwFuT3DHR\nwVbVa9q2b0ty0qhzHwE+DHwqyQd6yl8GPAa8AvgF4CJgUXt6H+CngR/0NDUMPAn8LnA0MBv4FvCh\nJF+f6JglSVJjIjMdy5PMSTKH5o/2pVV1yh4a186cCzwF/HJVzR/j/N8Ap1fVfj1l76UJSQAk+c2e\n+zilLZvT83MdTUD5z8DrgZ8F1gK3VNWhe+SuJEmaAfpaXklyF/Bt4I2DHc7OVdVs4GzgD2hmHpaM\nUW0zcBfw/p6yxcCqifSV5Lok/yXJU0meS/Ip4Bmg+hq8JEmaeOioqllVdTxwJHDn4Ie0U0PAq4Br\ngauAM6vqpWPUW0UTNGhnQ44EbppMx1V1NDCPJuxIkqQ+TCR0LK2qp2j2P9wBXAfcvUdGNbYlwM1J\nvkcTPPYH3jNGvfXA66rqF9trPgv8qN9Oq+og4P8G/mOS/6/fdiRJmukmEjouTTI3yX7AocAbaGYc\n9riqOgx4+0h/SR4Hvkiz6fMFkjwHXAOcB5wBrJ5Evz8HfA34C+Df99uOJEnqf0/Ho8AXGHumYU9Y\nTDPW1VW1paq20ISQE6rqiDHqr6IJJN9O8lA/HbZP4XwduCXJ/5HEj9RIkjQJ43pkdrSqejXwPmDj\nLqq9tH1cdcTzSUYeud131LntScZcAqmqlwC/AXwU+MSo01+jWUL5nd7CJJuq6peAv9/tzYzd55HA\nV4BrklzcTxuSJOmFJhI6llXVRe3vPwBuo3mXxc48POr4VuBX2t+/OurcQzQbPscyBLwS+HiS7/ee\nqKqPAyur6kOjL0ryV7sY2+5cCBwC/HZV/XZP+XD7SK0kSZogP20/YH7aXpI0w/hpe0mSNLX0tadj\n0KrqrcAtOzm9MsnKLscjSZIGz+WVAVu/fv2OoaGhvT0MSZK64vKKJEmaWgwdkiSpE4YOSZLUCUOH\nJEnqhKFDkiR1wtAhSZI64SOzA9bvG0l966gkaZrykVlJkjS1GDokSVInDB2SJKkThg5JktQJQ4ck\nSerEuL4yW1UbgOOAbW3RFuDyJH/cnn8EuDjJmqp6LfBd4NAkj47R1g7grUnumOhgq+o1bdu3JTlp\n1LmPAB8GPpXkAz3lLwMeA14B/AJwEbCoPb0P8NPAD3qaGgb+DvgE8FpgNvAdYEWSGyc6ZkmS1JjI\nTMfyJHOSzKH5o31pVZ2yh8a1M+cCTwG/XFXzxzj/N8DpVbVfT9l7aUISAEl+s+c+TmnL5vT8XAc8\nBPxvwDxgLvDbwJqq+hd75K4kSZoB+lpeSXIX8G3gjYMdzs5V1WzgbOAPgG8BS8aothm4C3h/T9li\nYNVE+kry/SR/m2QHzfPH22n+r17fx9AlSRJ9hI6qmlVVxwNHAncOfkg7NQS8CrgWuAo4s6peOka9\nVTRBg3Y25Ejgpn46rKqngGeBrwN/DfxFP+1IkqSJhY6l7R/hHwB3ANcBd++RUY1tCXBzku/RBI/9\ngfeMUW898Lqq+sX2ms8CP+qnwyRzgTk0Sy1fAp7rpx1JkjSx0HFpkrlJ9gMOBd5AM+Owx1XVYcDb\nR/pL8jjwRZpNny+Q5DngGuA84Axg9WT6TvJskj8DTqDZUyJJkvrQ756OR4EvMPZMw56wmGasq6tq\nS1VtoQkhJ1TVEWPUX0UTSL6d5KEBjeElwP8yoLYkSZpxxvXI7GhV9WrgfcDGXVR7afu46ojnk4w8\ncrvvqHPbk4y5BFJVLwF+A/gozWOsvb5Gs4TyO72FSTZV1S8Bf7/bmxm7zwU0T8I8SPN/9OvAScBl\n/bQnSZImFjqWVdVF7e8/AG4DfncX9R8edXwr8Cvt718dde4hmg2fYxkCXgl8PMn3e09U1ceBlVX1\nodEXJfmrXYxtdw6mCTkH0+wHeQj4tSRfnkSbkiTNaH7afsD8tL0kaYbx0/aSJGlq6WtPx6BV1VuB\nW3ZyemWSlV2OZzIuOHUjQ0NDe3sYkiRNOVMidCT5Os37MCRJ0ouUyyuSJKkThg5JkvT/t3fncXdU\n9R3HPz+yADFGoCFuJCSUTWVROLRSFoNUQCWKCgqyR0MC+GqLYMWwmAoE6tKoVQTCEllksa0ooERF\noMCrSH+VIoiQkhCWQEwCCRBQQ8jpH+dcMhnuvc+9N/eZ5+F5vu/X63k9d+bMcmbOmZnfnDl3biUU\ndIiIiEglFHSIiIhIJRR0iIiISCUUdIiIiEgl9EbSLuvkjaR6G6mIiLyO6Y2kIiIi0r8o6BAREZFK\nKOgQERGRSijoEBERkUoo6BAREZFKtPSDbyGE24DdgZfzqMXAd9z9mzl9IXC6u18ZQhgPPAqMdfcn\n6ywrAnu5+53tZjaEMC4v+3Z3f38pbQbwZeB77n5CYfxGwFPApsAE4FTgiJy8AbAx8GJhUVPd/arC\n/McD5wNnuPvZ7eZZREREknZaOs5y95HuPpJ00T4nhLBfL+Wrkc8CK4B9Qgjb1kmfBxwaQhhRGHcw\nKUgCwN2nFbZjvzxuZOGvGHBsCZwM3N8L2yIiIjKodPR4xd3vBh4EduhudhoLIQwBJgPnAg8Ax9WZ\n7AngbuCThXFTgNkdrvYS4DTg2Q7nFxERkaztoCOEYCGEPYDtgf/qfpYamgS8GbgCuBQ4OoSwYZ3p\nZpMCDXJryPbAj9tdWQhhKvCSu1/bcY5FRETkVe0EHaeFEFaQ+j/cCVwF3NMruarvOOAmd/8DKfAY\nBXy8znQ3AFuFEN6V57kcWNXOinLfkdOB49crxyIiIvKqdoKOc9x9E3cfAYwF3klqceh1uW/F/rX1\nufsy4CfA1PK07r4amAOcCBwFXNzBKi8Gznb3RR1mWUREREo67dPxJHAd9VsaesMUUl4vDiEsDiEs\nJgUh7wshbFdn+tmkgORBd3+4g/V9AJgZQlgWQlgG7AF8KYRwR4f5FxERGfRa+spsWQjhLcAhwH1N\nJtswf1215hV3r33ldngpbY27130EEkIYChwLnAd8q5R8K+kRysnFke6+IISwN/B0jxtT39jS8A+B\nO4BvdLg8ERGRQa+doOOMEMKp+fOLwO3AKU2mf6Q0PBc4IH++pZT2MKnDZz2TgM2AWe6+pJgQQphF\napGYXp7J3e9qkremyu8XCSH8GXg+9ycRERGRDuin7btMP20vIiKDjH7aXkRERPqXjvp0dFsIYS/g\nZw2SZ7r7zCrzsz5O+uB9TJo0qa+zISIi0u/0i6DD3e8ARvZ1PkRERKT36PGKiIiIVEJBh4iIiFRC\nQYeIiIhUQkGHiIiIVEJBh4iIiFRCLwfrsp5eDqYXgYmIyACjl4OJiIhI/6KgQ0RERCqhoENEREQq\noaBDREREKqGgQ0RERCqhoENEREQq0dIPvoUQbgN2B17OoxYD33H3b+b0hcDp7n5lCGE88Cgw1t2f\nrLOsCOzl7ne2m9kQwri87Nvd/f2ltBnAl4HvufsJhfEbAU8BmwITgFOBI3LyBsDGwIuFRU1196tC\nCGOArwEHAsOABcCH3P2pdvMtIiIi7bV0nOXuI919JOmifU4IYb9eylcjnwVWAPuEELatkz4PODSE\nMKIw7mBSkASAu08rbMd+edzIwt9VOVC5BVgFbAdsAhwOrOyVrRIRERkEOnq84u53Aw8CO3Q3O42F\nEIYAk4FzgQeA4+pM9gRwN/DJwrgpwOw2V3c0KdA4wd2Xufsad/+duz/ffs5FREQEOgg6QggWQtgD\n2B74r+5nqaFJwJuBK4BLgaNDCBvWmW42KdAgt4ZsD/y4zXXtQwqqLgwhPBNCeCiE8PmOcy4iIiJt\nBR2nhRBWkPo/3AlcBdzTK7mq7zjgJnf/AynwGAV8vM50NwBbhRDelee5nPSYpB2jSY9e7gPeSnqc\nND2EcHiHeRcRERn02gk6znH3Tdx9BDAWeCepxaHXhRC2BPavrc/dlwE/AaaWp3X31cAc4ETgKODi\nDlb5ArDI3b/l7qvc3YErgY92tAEiIiLScZ+OJ4HrqN/S0BumkPJ6cQhhcQhhMSkIeV8IYbs6088m\nBSQPuvvDHazvf4F6P9ymX8cTERHpUEtfmS0LIbwFOIT0+KGRDfO3QGpecffaV26Hl9LWuHvdRyAh\nhKHAscB5wLdKybeSHqGcXBzp7gtCCHsDT/e4MfXNAb4YQjgRuIDUYfZw4HMdLk9ERGTQayfoOCOE\ncGr+/CJwO3BKk+kfKQ3PBQ7In28ppT1M6vBZzyRgM2CWuy8pJoQQZgEzQwjTyzO5+11N8taUuz8W\nQvgQMAv4Kuk9HzPc/dpOlykiIjLYWYx6YtBN7z/h8aY79Ffnj6sqKyIiIlWwVifUa9BFRESkEh31\n6ei2EMJewM8aJM9095lV5md9nPTB+5g0aVJfZ0NERKTf6RdBh7vfAYzs63yIiIhI79HjFREREamE\ngg4RERGphIIOERERqYSCDhEREamEgg4RERGphIIOERERqYSCDhEREamEgg4RERGphIIOERERqYSC\nDhEREamEgg4RERGphIIOERERqYSCDhEREamEgg4RERGphMUY+zoPA8qGG274wKpVq/7U1/kYzIYO\nHTp69erVy/o6H4OdyqHvqQz63iApg2UxxgNamXBob+dksNlxxx3/5O6hr/MxmIUQXGXQ91QOfU9l\n0PdUBuvS4xURERGphIIOERERqYSCju67qK8zICqDfkLl0PdUBn1PZVCgjqQiIiJSCbV0iIiISCX0\n7ZUOhBC2Bb4P/AXwDHCUu/9faZohwLeBA4AInOfuF1ed14GqxTLYD5gJ7Aj8q7ufUnlGB7gWy+EM\n4FBgdf6b7u5zq87rQNViGRwLnASsAYYAs93921XndaBqpQwK024H3AucPxjPSWrp6MwFwHfdfVvg\nu8CFdaY5HNga2AbYHZgRQhhfWQ4HvlbKYAEwBfhalRkbZFoph3uA3dx9Z2AycG0IYeMK8zjQtVIG\n/w7s7O7vBv4GODmEsFOFeRzoWimD2s3ohcD1FeatX1HQ0aYQwhhgF+DqPOpqYJcQwualST9FuptY\n4+5LSZXskOpyOnC1Wgbu/oi730u6u5Yua6Mc5rr7S3nwt4CR7ghlPbVRBs+7e60D3whgGKkFVtZT\nG9cEgFOBG4F5FWWv31HQ0b6xwCJ3fwUg/38qjy8aBzxWGH68zjTSmVbLQHpXJ+VwFDDf3Z+sIH+D\nQctlEEL4SAjhd6Tz0tfc/f5KczpwtVQGuWVpf2BW5TnsRxR0iEglQgjvA84CDuvrvAxG7v4Td38X\nsC1wZO5bIBUIIQwDZgPTasHJYKWgo31PAG/Pz+Zqz+jelscXPQ5sWRgeV2ca6UyrZSC9q+VyCCHs\nDlwJHOTuD1eay4Gt7WPB3R8n9bM5sJIcDnytlMFbgb8EfhpCWAj8AzAlhDDo3uGhoKNN7r4E+F/W\n3q0dBtyb+20U/ZBUqTbIz/YOInXmkvXURhlIL2q1HEIIuwHXAge7+2+qzeXA1kYZbF/4PBrYB9Dj\nlS5opQzc/XF3H+3u4919PPBNUp+/4yrPcB/TV2Y7Mw34fgjhTGA56Tk1IYSfAme6uwNXAH8N1L42\n9RV3X9AXmR2geiyDEMKewDXAKMBCCIcCn9HXNbuqlWPhfGBj4MIQXv3dqyPVp6BrWimDqfkr5C+T\nOvJ+x91/3lcZHoBaKQNBbyQVERGRiujxioiIiFRCQYeIiIhUQkGHiIiIVEJBh4iIiFRCQYeIiIhU\nQkGHYGb7m9kdheGJZrawD7NUGTObY2Zd+/VfMxtvZrEwvLmZPWZmo1uYd5qZXdGtvLwemNleZrai\nr/MxGJnZEe0c590+VqS53jo2Oij3fzazs7q1fgUdg5yZGem3AL7cw3THm9kDZva8mS03MzezTxXS\nF5rZEXXme814S+blZY0spU00s2hmK/PfU2Z2mZlttn5b2jdijEuBH9Dz/n0D8BVgRgXZ6jdijHfE\nGDfp63w0YmYzzOyXfZ2PwaC39rWZ3WZmp3d7ub2tfGz0YV08DzjRzN7ejYUp6JD9gOHArY0mMLPD\nSBfNzwBvIr3i9yTSS3A6sQ+wFbCG+r/D8UqMcWSMcSSwJ7A76Q1+r1eXAsea2agm0xwB3B9jnF9R\nntZhZkPMTOcDEVlHjHE58DNgajeWp5NMhfJd/+lmdmu+i7/fzHYys8PM7BEze87MLjazoYV5xpnZ\nv5nZ0/nvIjN7YyF9ppktyMubb2b/UEgbn1sNjjSzB83sBTP7uZm9tZCtg4BfxuZvifsb4D9jjL+O\nyR9zFN7pGw2nAjeT3qw8bK0AAAxBSURBVNratCLHGBeQfgr6PeU0Mxua98lHS+O/b2aX5s/7mtmv\nc+vMUjO7xszGNFpf3l97FoYnmtnq0jqn55aaFWZ2l5nt2sM2/B+wDPjbJpMdBPyilJe/N7OHcrk9\nbmbnmtmQnPZ1M/tRafp98rRvyMM7mNlcM1tWmH9YTqvVjc+Y2YPAS8AYMzvUzO7LrVBPm9mFteXl\n+d5iZjfkujovzx/NbHxhmim5Vew5M7vXzPZrtNF19u8cM7vCzC7N+3dRPj7ebWb/nbfvVjN7W2Ge\nhWZ2ppndmY8DN7PdCulN64CZDctl+nBe/nwz+4SllrzpwERb2/K2VYPteF9ex3O5zKYW0iaa2Woz\n+1Re9nNmdl3xOK6zvE7OFTuZ2a/ydi7I8w8ppP9V3jcrzexOUuBfXOeIXK8eNbNnzexmM9u6UR7r\n5PkvzOzyXG8WWzoONyukr9PqWaiDWzTa12Z2TN7eL+blLjGzb9Spx1sUlnuMmT2SP38H2As4Iy+z\n7u/+WGpFuMXSo4SlZvaMmX3ezLbM+/QFM/sfM3tHYZ71OlYKdX12oa6/pt7kz033T2lb1nkM1qVy\n/wXpHLX+Yoz6q+gPWEh6Lfo7gGGkH8CaD1wEvIH0o3BLgE/n6TcCHiE1u28MbAr8FLi0sMwjSC0P\nBrwf+COwf04bD0TSRXs06XXgdwGzC/P/Gvi7Uj4nAgsLw4cAfwLOBvYFNmmwbUf0NB7YHPgz8HHg\n3Tl/u5bWvbowvDXwcHGbS8v/KnB9YXgksBLYKw/vCexGeuX/W4D/BK4uTD8HuLgwHIE9m+RnZt5n\nWwFDSK0/y4BNi/u8Tj5vAM5uUjf+AHykNO4TwIRctu/J00zNae8EVgGbF6b/PnBJ/jwGeIYU1A0H\n3g44cGapbtyS98vwvD0fBN5FuiHZGngQOLewjltIvyE0Kq/jtryc8Tn9OFKd3Tkv40O5PLZusN3l\n/TuHVIc/nOefluf/CbAFMAL4FXBRqY49Beyat+NUYCkwqsU68M95O3fK+3oLYKecNoMUlDc7rifk\nPB+b1/Fe4FngkMI2RuASUv18M+k8cFoXzxVvyvXjDGDDPN8C4AuF9Gfyvhme98di1j3Of0A6V7w5\nT/NPwEPAsHrHSp0830yq55vmv5uAm5qcC8bn/bJFo30NHEN6dft3SefAvwTmAV+qt4zCPI8Uhm8D\nTu+hDGfk9XyWtcfBK8AvS2Xw88I863uszCHVm4/kZXw852HLBsdGo/3zSGncq+XUjXLP0+xKapke\n3mw/tvJX6UV3sP/lg+4LheEP5UpYvHBcB8zKnw8G5peWsSvpoj2kwTr+Dfhq/lw7IHcrpJ8I3FsY\nngccU1rGxGKlzOMOBP6DdGJ7hfQ4ZofStr0IrCj9rWHdE80/kk6WtRPZb4ALS+uOed7lwKPABdQJ\ndPL07yBdfMfk4cnAvCZlcCCwpDD86gGahxsGHaQL0gvA3qVl3l/bRhoHHVcB5zfJ1ypgYg/15+vA\ndYXhXwMn5c9vJF2c98jDpwC/Ks3/CfIJqlA39u5hnZ8D7smft8jzbFVI35d1T6QPAEeVlnEDDU76\n1A86iheqEXn5hxTGncC6dXghcFZh2Ei/8vzpnupAnnYl8OEG086g56BjOnBXady5wNxSnS4e518D\nftRkmQtp71zxadKvmlohfSrwcP58eN4nxfRzyMc56aYkAuMK6RsAz5GPB5oEHaQbnwhsUxi3XR73\n1sI2dRJ0/BkYURj3WfIxXl5GYZ5Ogo7flcYtqVMGy7t4rMyhUNfzuKXARxscG432T7OgY73LPY/b\nJk83ptl+bOVPP/hWvacLn18i9V9YWhpXa3adAIyz1/ZgjqQ7tkVm9nfAFFIlN9LdwA+arPPFwvIh\nXdib9TVIK4zxRlI0jJltT/oRrxvNbELMtZJ0F35lcT4r9JI2M8t5vTLG+HIefQlwnpmdHGNcmce9\nElvsXBhj/L2Z/YbU4vMvpLvNywrr3JXUOrEz6QJmpLvNTozO895ghW+okO6Ctqg/y6tGkQKoRl5T\nDpb60nye1KoylHQXcndhkstIF+BZwCeBRTHGu3LaBGCPUt0x0l1c0cLSOj8AnAlsT7pjHkI6+UJq\nLYF0Eqt5rLS8CcB3zezbhXFDgSdp3av1Ncb4Uqo2rzluyo8mFhbmiWb2OLlMeqgDm5NaDua1kb+y\nsaRWhaL5QPGxX/k4Lx+H9bRzrhhLupAU6+X8PB7SvnislF6sjxPy/9/m/V0zrLCMZmrTFJc5v5D2\nNJ1bEmN8qTC8kJ6Pt06U8/gSTepdF46VeutspV60o1vlPoq1N4PrRX06+rfHSBH9JqW/jWKMi8xs\nD1LT8FRgdL5Q30A6qbbqXlJTfctijA+RLnRbkppRW7UvqRlycn7mu5jUlDeSdKfWqcuAY/JzyPcC\nlxfSriG1pmwbYxxF/Y6rRS+SLkI1byt8XpbT/7ZUHm+IMZ7Xw3J3IO3rRtYpBzMbS2rOPZt0p/gm\nUhNzsWyvAbYxs11IdzyXFdIeI90VFfP5ppg65xatKaxzOHB9Xu64vL++WFjnovx/XGH+4ufaeieX\n1jsyxnh8k23vhvG1Dzm4HcfaQKdZHVhKKtNtGix3TYPxRU+w9uRds1UeX5UngC1t3StHMQ+L6qQX\n81y7IG5TKrsRMcarW1w/FMqBtX0HamkraXxsQeN9PcbMRhSGx7O2bGs3Kp0st2NdOlbaVW87yvsU\n1t3+bpX7DqSWoFWdZr5GQUf/diNQ6+T2RkvebmYfy+mjSI86lgLRzD5Mes7YjutJwUBDZjbZzA6x\n/K6J3GlrGvBgjPHZNtZ1HOl5+vak/hzvJlXmy1i/ntHXkIKZbwO/iDEuKqSNIjUVvmBm40jPNptx\n4GgzG547fH2+lpDvFr4FfN3MtgEws5GW3nNSPtG9KgdDm5OeDzdyPet2NB1JOj6XAi+b2XuBI4sz\nxBhXAD8iBSblYOtyIOSy28jMNsgdzw5okofhpH5Ey2OMfzSzd5KajGvre5LUVH1ero9jgPJXEWcB\nMyx1/DQz29jM9sytY71pspntYqmD4RdILRo35bSGdSCX6feAr1rqeFs7xnbMkywmtTYOb7Luq4Fd\nzewoSx2N/4pUny/p6hY2dxOp7Kbnursd6SJYy8ONpDr1BUsdZ3chPYoEIMa4hNRCer7lr0aa2SZm\n9jErfa29nhjjU8DPgW/k+TYFvgH8LMZYu5t34LB8zGxO6n9S1Ghfb0Cqcxtb6sh7Cqn/EjHGZeRA\n19I3sHYktaaWl9tyh9gWdeNYaVe9/XMvKSg7MB/jHwP2LqR3q9w/QDpHrTcFHf1YblLcl3QH/BDp\nxHkL6WINMJf0DZB7SHfhB5MuQu2YC6w2s4lNpllOasb/vZm9SOpLsIL0bLwl+aA7CPh6jHFx8Y/U\nWvMeMwtt5h2AGONzpO3+IOnrqUXHkZ4Bv0Dqk/LDHhb3OdIJ6lnSM/M5pfQvAz8Gfmxmz5M6+02j\n+bE0GZiT89nIFcDO+aRKjPH3hXWtIF0o691xXkba7rn5xE+efzHpq8kHkZqjl5P2Ud1vX+R5VgLH\nky7AK0ktK+VHdZ8mXdCfBO5k7f78c17GbFLn3svyOh8nXVyGNdn2briIFHQuBz5F6qNR29891YHT\nSGV9fZ7mdta2fPyQdKe+2NI3DMotGsQYHyU97/8cqdPeFaQOu9d1bet6kLd1P1Lg+gfScX056ZFj\nLUD9MGnfLCftq++VFjOF1Gn7NjN7gdRX6RBSs3orjiDtv4fy3wrgqEL66aSbpKdJF+RrSvM32teP\nke7YHyWde24m1bGao0nnoufy9paDvVmkAHyFmf2uxW1pqhvHSgdes39i+or935Pq/7PAAaTOq7V8\nrne5m9kmpPp9QYf5Xoet+6hHBqN89zs9xrh3Hp5IukiO78t8vR7l1pFHY4yWh0cD/wOE0vP4evNO\nI3UEPbLZdP2Jme1PCow2jn10MrHUb+j0cn8ief0zs2NIZdvtlorK9YdjpRNmdi6pP1FXXrCmjqRC\njPFm0t2DdFlu/t2yxWkvoEt3E73FzHYm3QHdT3o2fDZw7evpJCpShYFyrMQYv9TN5enxitSzkNf3\nG0D70gpS59iBajPSI4qVpCbj35Kad0VkXTpW6tDjFREREamEWjpERESkEgo6REREpBIKOkRERKQS\nCjpERESkEgo6REREpBIKOkRERKQS/w/KoGuAiA3cogAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0623b47a20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "contributions = model.predict_contributions(test)\n",
    "contributions_matrix = contributions.as_data_frame().values\n",
    "shap_values = contributions_matrix[:,:-1]\n",
    "shap.summary_plot(shap_values, X, plot_type='bar', color='royalblue')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repayment `PAY_*` variables appear to be most important in this model along with a credit limit, `LIMIT_BAL`.\n",
    "\n",
    "(In a more realistic scenario, credit limit would likely not be used as a model input because it could cause target leakage. It's used here in this small data example to improve model accuracy.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Select a Probability Cutoff by Maximizing F1 Statistic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bind model predictions to test data for further calculations\n",
    "Model predictions will be used in numerous subsequent calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gbm prediction progress: |████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "# cbind predictions to training frame\n",
    "# give them a nice name\n",
    "yhat = 'p_DEFAULT_NEXT_MONTH'\n",
    "preds1 = test['ID'].cbind(model.predict(test).drop(['predict', 'p0']))\n",
    "preds1.columns = ['ID', yhat]\n",
    "test_yhat = test.cbind(preds1[yhat]).as_data_frame()\n",
    "test_yhat.reset_index(drop=True, inplace=True) # necessary for later searches/joins"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function to calculate PR-AUC Curve\n",
    "\n",
    "Predictive models often produce probabilities, not decisions. So to make a decision with a model-generated predicted probability for any one customer, you must specify a numeric cutoff above which we say a customer will default and below which we say they will not default. Cutoffs play a crucial role in DIA as they impact the underlying measurements used to calculate diparity. In fact, tuning cutoffs carefully is a potential remdiation tactic for any discovered disparity. There are many accepted ways to select a cutoff (besides simply using 0.5) and in this notebook the cutoff will be selected by striking a balance between the model's recall (true positive rate) and it's precision using the F1 statistic. Using precision and recall to select a cutoff is sometimes seen as more robust to imbalanced data than the standard ROC approach. Maximizing F1 typically results in a good balance between sensitivity and precision. \n",
    "\n",
    "Selecting a cutoff can be done intervactively with h2o Flow as well. Enter the url: http://localhost:54321/flow/index.html (or your H2O Connection URL displayed in cell 2) into your browser. Select `Models` -> `List all models` -> `dia_gbm` and you should see an interactive ROC curve where you can pick your own cutoff.\n",
    "\n",
    "To learn more about confusion matrices see: https://en.wikipedia.org/wiki/Confusion_matrix.\n",
    "\n",
    "**Note:** h2o supports `pr_auc` directly: http://docs.h2o.ai/h2o/latest-stable/h2o-py/docs/metrics.html#h2o.model.metrics_base.MetricsBase.pr_auc. This utility function is used here to give more control over the plotting of the PR curve below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_prauc(frame, y, yhat, pos=1, neg=0, res=0.01):\n",
    "    \n",
    "    \"\"\" Calculates precision, recall, and f1 for a pandas dataframe of y and yhat values.\n",
    "    \n",
    "    Args:\n",
    "        frame: Pandas dataframe of actual (y) and predicted (yhat) values.\n",
    "        y: Name of actual value column.\n",
    "        yhat: Name of predicted value column.\n",
    "        pos: Primary target value, default 1.\n",
    "        neg: Secondary target value, default 0.\n",
    "        res: Resolution by which to loop through cutoffs, default 0.01.\n",
    "    \n",
    "    Returns:\n",
    "        Pandas dataframe of precision, recall, and f1 values. \n",
    "    \"\"\"\n",
    "    \n",
    "    frame_ = frame.copy(deep=True) # don't destroy original data\n",
    "    dname = 'd_' + str(y) # column for predicted decisions\n",
    "    eps = 1e-20 # for safe numerical operations\n",
    "    \n",
    "    # init p-r roc frame\n",
    "    prauc_frame = pd.DataFrame(columns=['cutoff', 'recall', 'precision', 'f1'])\n",
    "    \n",
    "    # loop through cutoffs to create p-r roc frame\n",
    "    for cutoff in np.arange(0, 1 + res, res):\n",
    "\n",
    "        # binarize decision to create confusion matrix values\n",
    "        frame_[dname] = np.where(frame_[yhat] > cutoff , 1, 0)\n",
    "        \n",
    "        # calculate confusion matrix values\n",
    "        tp = frame_[(frame_[dname] == pos) & (frame_[y] == pos)].shape[0]\n",
    "        fp = frame_[(frame_[dname] == pos) & (frame_[y] == neg)].shape[0]\n",
    "        tn = frame_[(frame_[dname] == neg) & (frame_[y] == neg)].shape[0]\n",
    "        fn = frame_[(frame_[dname] == neg) & (frame_[y] == pos)].shape[0]\n",
    "        \n",
    "        # calculate precision, recall, and f1\n",
    "        recall = (tp + eps)/((tp + fn) + eps)\n",
    "        precision = (tp + eps)/((tp + fp) + eps)\n",
    "        f1 = 2/((1/(recall + eps)) + (1/(precision + eps)))\n",
    "        \n",
    "        # add new values to frame\n",
    "        prauc_frame = prauc_frame.append({'cutoff': cutoff,\n",
    "                                          'recall': recall,\n",
    "                                          'precision': precision,\n",
    "                                          'f1': f1}, \n",
    "                                          ignore_index=True)\n",
    "    \n",
    "    # housekeeping\n",
    "    del frame_\n",
    "    \n",
    "    return prauc_frame\n",
    "        \n",
    "prauc_frame = get_prauc(test_yhat, 'DEFAULT_NEXT_MONTH', 'p_DEFAULT_NEXT_MONTH')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Select best cutoff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.27\n"
     ]
    }
   ],
   "source": [
    "best_cut = prauc_frame.loc[prauc_frame['f1'].idxmax(), 'cutoff'] # Find cutoff w/ max F1\n",
    "### !!! UNCOMMENT LINES BELOW TO REMEDIATE MINOR FAIRNESS ISSUES !!! ###\n",
    "# best_cut = 0.3  # min threshold with overall fairness\n",
    "# best_cut = 0.46 # max accuracy\n",
    "# best_cut = 0.35 # max MCC\n",
    "print('%.2f' % best_cut)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot PR-AUC Curve\n",
    "\n",
    "An area under the curve for precision and recall (PR-AUC) plot is a typical way to visualize recall and precision for a predictive model. The F1 statistic is the harmonic mean of recall and precision, and we can visualize where F1 is maximized on with the AUC-PR curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmUXHWd9/H3p3qDLBBIAkgSNknE\nwBDktMAYhYA4BoTgM6ASwVEHzeMSdQaB4OiAgzPOAG6D4ILIKAyImhwlaJThYRkBAWkkaQkKtIik\nE4QQISYh6a2+zx+3qlOprqqu7vTt7up8XufUSd1bv7r17UtT3/7tigjMzMzKyYx0AGZmNro5UZiZ\nWUVOFGZmVpEThZmZVeREYWZmFTlRmJlZRU4UZjVA0mskPSppk6SPS9pd0m2SNkr64UjHZ2ObE4VV\nTdIzkrZK2izpeUn/JWlCmbLvk9STK/sXSaskndbP9feQ9BVJz+be15Y7npLOT7RzJL1bUksu1uck\n/UzSG6t87zOSTh7Ax10E3BMREyPiKuAsYF9gckS8YxDhl4tLki6XtCH3uEKSypR9m6T7JL0s6U+S\nviVpYsHrq3P3Jv/olnTbUMVqw8eJwgbq9IiYABwNvB74TIWyD+TKTgK+BtwiaVKpgpIagTuBw4H5\nwB7AG4ANwDEDDVJS/UDfM8Drnw98Bfg8yRf2ASQ/4xkpfeSBwOqi4ycjonuIP2cR8HZgDnAkcBrw\nf8uU3RP4V2B/4LXAdODK/IsRcXhETMj9DkwEngVc+6lFEeGHH1U9gGeAkwuOrwR+Uqbs+4D7Co7H\nAQG8vkz5DwDPAxMqfH4AhxYcfwf419zzeUA7sAT4E3Aj8FvgtILy9cCLwNG54+OAXwIvA6uAeVXe\nhz2BzcA7KpTpja0wvtzzG4EssDV3nYty5xeQJIOXgXuA1+bO3wX0ANty5b8HdAJduePzhvC/8S+B\nRQXH5wEPVvnevwV+U+a1E3Kxjh/p32M/Bv5wjcIGRdIM4FTg0SrK1gHvJ/li+2OZYicDP4+IzTsR\n1n7A3iR/bS8i+UJdWPD6W4EXI+LXkqYBPyX5i3hv4AJgmaSpuZgvlvSTMp/z18BuwI8GE2REvIfk\nr+vTI/mL+wpJs3Lx/gMwFVgB3CapMSJOAu4FFufKLySpyXw/d/zt4s/INYu9XOFxQJnwDidJmnmr\ncueqcTw71noKvRdYGhFbqryWjSKpVs9tTPqxpG5gI8kX7ecrlD1O0svAeKAbODciXihTdjLwyE7G\nlgUujYgOAEk3A49KGhcRrwDvBm7OlT0XWBERK3LHd0hqIUl+342I/6jwOZNJEs5QNvu8C/hpRNyR\ni/0LwCdImt/uGejFIuJmtv+sAzGB5L9t3kZggiRFRNmF4SS9hSQZHFvitXEkfSoLBhGPjQKuUdhA\nvT0iJkXEgRHxkYjYKulNBR2WhX9RPhgRk4C9gOXAmypcdwPwqp2MbX1EbMsfREQbSfPT6bkvqwVs\n//I8EHhH4V/ZwBurjGEDMGWI+0H2p6C2FRFZYA0wbQg/oxqbSfqH8vYANveTJI4jua9nRcSTJYr8\nLfBn4H+HMlAbPk4UttMi4t5cE8iEiOjTTJFrTvoI8B5Jrytzmf8HvFXS+Aof9QpJX0fefsUfVeI9\n+eanM4DHc8kDki/hG3NJL/8Y309NIu8Bkv6Ct1cos2WAsa4jSV5AMvoImAGsrSKePiSdUzTiqPhR\nrulpNUlHdt4cyjcnkfvvuRz4+4i4s0yx9wI3VEo2Nro5UdiwiIgNwHXAJWWK3Ejy5b1M0mGSMpIm\nS/onSafmyqwE3i2pTtJ8kg7S/twC/A3wYXZsivlvkprGW3PX203SPEnTq/hZNuZ+jmskvV3SOEkN\nkk6RdEVBrKdK2lvSfiR9D4WeBw4pOP4B8DZJb5bUAHwS6CDpXB6wiLipIHmXejxb5q03AOdLmiZp\n/1wc3ylVUNIRwM+Bj0VEyWGvuft5IvDdwfwcNjo4Udhw+grJl+eRxS/k+hVOBn4H3AH8BfgVMAV4\nKFfsE8DpJKOCzgF+3N8HRsRzJDWANwDfLzi/hqSW8U/AepIkdSG5/ydyCepnFa77JeB8kuHB+fcv\nLojpRpKO4GeA/yn87Jx/Bz6Ta/a6ICKeIOk3+SrJyKzTSTq7O/v7GYfYN4HbgN8Aj5H0Q30z/2Ku\nNpJvQvwkScf7t8s0PQK8h2SY9O/TD93SItcGzcysEtcozMysIicKMzOryInCzMwqcqIwM7OKam5m\n9pQpU+Kggw4a6TDGtieeSP59zWtGNg4zGzKPPPLIixExdTDvrblEcdBBB9HS0jLSYYxt8+Yl/95z\nz0hGYWZDSFK5ddb65aYnMzOryInCzMwqcqIwM7OKnCjMzKwiJwozM6vIicLMzCpyojAzs4qcKMzM\nrKLUEoWk6yW9IOmxMq9L0lWS2iS1Sjq6mus++fwmPvLfD9P2/KahDdjMzEpKs0bxHWB+hddPAWbm\nHouAr1dz0Y7uLCsee4GTv/wLLrn1NzsdpJmZVZZaooiIX5BsqF7OGeT20Y2IB4FJkqrZ2L7XDQ88\n65qFmVnKRrKPYhrJ9pF57blzfUhaJKlFUkvPKxt3eG3lmpfTi9DMzEY0UajEuZL7skbEtRHRHBHN\ndeP23OG1o2ZMSiM2MzPLGclE0Q7MKDieDqwbyAX+7q8P4NB9Jw5pUGZmtqORXGZ8ObBY0i3AscDG\niHiuvzc11Wc49Yh9OP8thzlJmJkNg9QShaTvAfOAKZLagUuBBoCI+AawAjgVaANeAd5fzXVn7TuR\nr537+jRCNjOzElJLFBGxsJ/XA/hoWp9vZmZDwzOzzcysIicKMzOryInCzMwqcqIwM7OKnCjMzKwi\nJwozM6vIicLMzCpyojAzs4qcKMzMrCInCjMzq8iJwszMKnKiMDOzipwozMysIicKMzOryInCzMwq\nSjVRSJov6QlJbZIuLvH6gZLulNQq6R5J09OMx8zMBi61RCGpDrgGOAWYDSyUNLuo2BeAGyLiSOAy\n4N/TisfMzAYnzRrFMUBbRDwdEZ3ALcAZRWVmA3fmnt9d4nUzMxthaSaKacCaguP23LlCq4Azc8//\nDzBR0uRKF+3OBqvWvMyGzR1DFqiZmZWXZqJQiXNRdHwBcIKkR4ETgLVAd58LSYsktUhqWf37ds69\n7iHmXn4Xy1euHfqozcxsB2kminZgRsHxdGBdYYGIWBcRfxsRrwM+nTu3sfhCEXFtRDRHRHNm3B5s\n6uhmW1eWi5a1umZhZpayNBPFw8BMSQdLagTOBpYXFpA0RVI+hk8B1w/kAxoyGdpf2jokwZqZWWmp\nJYqI6AYWA7cDvwV+EBGrJV0maUGu2DzgCUlPAvsC/zaQz+jKZpm+1+5DGLWZmRWrT/PiEbECWFF0\n7pKC50uBpQO5pgQTm+rpyma54swjmTyhaWiCNTOzklJNFGk4bL89+O4HjmX6Xrs7SZiZDYOaSxT1\nGTFnxqSRDsPMbJfhtZ7MzKwiJwozM6uophPFhs0dnqVtZpaymuujyLt15VqWLGulIZPpHQG14Kji\nFULMzGxn1WSNYsPmDpYsa2VbV9aztM3MUlaTiaL9pa00ZHYM3bO0zczSUZOJYvpeu9PZ07PDOc/S\nNjNLR00mivvaXiRbsA5tfQbP0jYzS0nNJYrubLBkWStdPdszRV0mw9xDp4xgVGZmY1fNJYqu7myf\n/omO7iw3P/TsCEVkZja21VyiaKjP0NmT7XP+6ruf8qgnM7MU1FyiqM+INx+2T5/zjXV1HvVkZpaC\nmksU3dngzt+90Od8Z0+PRz2ZmaWg5hJFV3eWxrq+YS8+caZHPZmZpSDVRCFpvqQnJLVJurjE6wdI\nulvSo5JaJZ3a3zUb6pMlOwo11Wd497EHDGHkZmaWl1qikFQHXAOcAswGFkqaXVTsMyRbpL6OZE/t\nr/V33fqMuOLMI9mtIcPEpnp2a8hw5VmeQ2FmlpY0FwU8BmiLiKcBJN0CnAE8XlAmgD1yz/cE1lVz\n4QVHTWPuoVNof2kr4xvr2NLZw4bNHU4WZmYpSDNRTAPWFBy3A8cWlfks8D+SPgaMB04udSFJi4BF\nAAcckDQxTZ7QxH1tL3oFWTOzlKXZR6ES56LoeCHwnYiYDpwK3CipT0wRcW1ENEdE89SpU4HSK8he\nsLSVtuc3DfXPYWa2S0szUbQDMwqOp9O3aek84AcAEfEAsBtQ1VocpVaQ7ezOcupV97J85drBxmxm\nZkXSTBQPAzMlHSypkaSzenlRmWeBNwNIei1JolhfzcWn77V7n9FPAJ09Mer2pvBOfGZWy1Lro4iI\nbkmLgduBOuD6iFgt6TKgJSKWA58EviXpH0mapd4XEcXNUyVNntDEFWceyQVLW+ns3jFh5PemGMnO\n7Q2bO2h/aSuPrd3I5376uPtRzKxmpboVakSsAFYUnbuk4PnjwNyBXjf/JTz30Clc8JZZfP5nv9vh\n9ZHcm2LD5g5ueuhZrrn7KRrqMmzuSPbN2EaSzC5a1srcQ6d4hJaZ1Yya2zP75Ve6mHv5XTRkksUB\ne0o0P/3z22YP+xdxPkFcfddTdOaWQO/o7ulTLiOxet1Gjp+1zw7vbX9pK9P32t0JxMxGnZpLFO0v\nv8J+Xdnev9CL7d5Qx4y9h6c2Udi8dNlPHqeju3RMhV7p7OGDN7RwyemHc8T+e7ppysxGvZpLFCo5\n6na7rV3JF/GVZ81J9Qv31pVrWbKslfqMepuXqtXRHXz6R48xvjHDls4kubhpysxGq5pbFDD6TMXo\nq6N7aEY+lRutVDiHo78k0VQvPnzCIYxrqOvzWj5JFGrIZFi9bqNHSZnZqFFzNYrpk8ZR35AhI/FK\nZ/kv6Z0d+ZSvMZRqEsrP4SjX/AXQWCc+dtLM3sUK/+uXz1T1udu6kxpRY12dm6LMbFSouUQxaVwD\nty85idXrNvLBG1ro6C5dw9iZkU+FNYbiJiGAjVs76ezpm6TGN9bRnQ0Wn3go7z72gB2S1BVnHslF\ny1qpy4gtJWoh+ff2ZLN09EBHdzcAFy5tZdK4Rg7ffw83R5nZiKi5RAHJHIrjZ+3DlWfN4aLcX/3b\nunuICHZvqO/9S3ywX6ylagwNmQw3PfQsX7unjYZMhmxAfYbez/vn02ZzxP57lh25VLiQYXEHdv69\nG7d28dGbfs2mju7e93V0Z/nQjY+QJVy7MLMRUZOJIq/wyzdfexiKYaalZn139mS55u6n6OiO3gTS\nVJ/hmnOOrvqv/ckTmpg8oYk5MyYx/4j9+sS6YXNHydnmr3QlNRDXLsxsJNRcZ3ax/Bdv4Zfwzn6J\n5md9F+55sfjEQ2ms27FDurEuw567Nwzq80rFWvi54xr7dn7naxdzL7+rdz2rwg73apYKGWh5M7Oa\nrlGkqVRt5Zp72nYok8YM8PznluuDydcuLlrWyqZt3b1NWMVNb4VNYUCfJq9STXX5Zq2uniwd3Vk6\nvMeHmQGqcmmlUaO5uTlaWloG9d6dnQG9fOXa3j6R4RiRlP+8UiO8xjfV0dWd7Z0FXsr4xjo6e7JE\nBLs11FUcyrtbQ4b7l5zEfW0v8qozTkGC97zn8h1+Rs8gN6tdkh6JiObBvHdM1ygKv9iGYpOj4lpG\n2l+WlWoXXT1BQ12m5OirvC0FyaW/+R75+RtLlrXynQgI2NaV7R3t5U2izHZdYzZRFM6DyK8J1Z3d\n+RnQ+b6Q4VJqhFe+aelzP3m8/wtUKelEV589PgoTSOFwYXesm+06xmSiKDUPothoWIp8IErVZiY2\n1fcZHtzUUFdynkahwiapwj6Kw/ffo8+oq8IEUngvPWzXbNcxJhNFNTOnR3Ip8sEqrs2UGx5cqdO6\nVCd3YTPaFWceSeYGISX9FuUSCOzYsT77VXuwpbPH/RdmY9CY7MzesLmDuZffxbaugglzdSIjdpml\nMQr7Z2Bg80u63nR8Murpjjt7y1fqWN+tIUM2GzTVJzWVUjPTzWxk7UxndqqJQtJ84D9Jdri7LiL+\no+j1LwMn5g7HAftExKRK16x21FOpEUrD2RFd0+bNS/69554dTm/Y3NHv0il5TfXqXUrd99ts5I3K\nRCGpDngSeAvQTrKH9sLcrnalyn8MeF1E/H2l6w5keKyHcw5SmUSRV5iEO3qyKIKOMsN0C9e/OuWI\n/Vi3cSsgd4KbDbPROjz2GKAtIp4GkHQLcAZQbqjOQuDSoQxguEco7SoK+0bGN9Zx2tX3QZlEkR+i\n+8U7nuSLdzzZe74+A/9yxhHM2Gt3ChOHk7vZ6JNmopgGrCk4bgeOLVVQ0oHAwcBdZV5fBCwCOOCA\nA4Y2ShuUwiR8xZlHcuHS1qp2+MvrzsKnf/RY73Gd4K2H78edv3s+Nz8kuPT02Zxz7IFDHruZDUya\naz2V2oquXDvX2cDSiCg5rjMiro2I5ohonjp16pAFaENjwVHT+OXFJ/HJt8yiqV6Mb+q7TlV/egJW\nPPYnOrqDzR09dHZn+fSPHuPaX/w+hYjNbCDSrFG0AzMKjqcD68qUPRv4aIqxWMomT2jiY29ONmoa\n6D7ilXx+xe8Y31jPOce5ZmE2UqpKFJKagDOBgwrfExGXVXjbw8BMSQcDa0mSwbtLXPs1wF7AA1VH\nbaNW8VLqNz/0LFff/RSSdhiuPBCXLl/NsQfvzaH7ThziaM2sGtXWKG4FNgKPAFWtSR0R3ZIWA7eT\nDI+9PiJWS7oMaImI5bmiC4FbotYmdFi/imsZ4xvrekc9rfnzK3z2ttV0VVjUMK87G7z1K7/gy+86\nakzPfTEbraoaHivpsYg4Yhji6dfOrB5rVepneOxQyc/LeOD3G7j+/j/QWFfH1q4eurOlfycb68QD\nn3qzR0OZDcJwDI/9paS/iojfDOZDzErJL3h4/Kx9+MCbDukdFnvdvU/z9f99uk/5zp7gS//zJOf/\nzSwnC7NhVO2opzcCj0h6QlKrpN9Iak0zMNu1FO7494E3HUJjXalBc3DTr57lDf9xZ+8Of2aWvmpr\nFKekGoVZgckTmvjCO+Zw/g9WlWyG6ugOLliaLEToDm6z9FVVo4iIPwKTgNNzj0m5c2apWHDUNH7+\niTfRUGZKRmd3llOuupev3vmU9/w2S1lViULSJ4CbgH1yj//Orc1klppD953IF99xFE31pX9Nu3qC\nL97xJMf9+53c9JD/bjFLS7VNT+cBx0bEFgBJl5PMe/hqWoGZwfZ1pW5+6FmuuuupksNpu3oiWQ4k\n8MQ8sxRU25ktoHB5jR5KL9FhNuTy8zF+9vE3le3kBrj0ttW0Pb9pGCMz2zVUmyj+C3hI0mclfRZ4\nEPh2alGZlXDovhP5wjvmlG2K6u4J91uYpaDazuwvAe8H/gy8BLw/Ir6SZmBmpeQXIPzwvENKvp7v\ntygcQrthcwer1rzs5GE2SBX7KCTtERF/kbQ38EzukX9t74j4c7rhmfU1eUITS+a/lumTxnHpbavp\nLtFv0dEdXLSslU3bunv3D98VtsA1S0N/NYqbc/8+ArQUPPLHZiPmnOMO5OcV+i3qMuJfblvNtq4s\nmzq62daV5YKlre7HMBugiokiIk7L/XtwRBxS8Dg4IkrX/c2GUaV+i66eoKFux/Od3VlOvepez+w2\nG4Bq51HMlTQ+9/xcSV+S5K3mbFQo3jhpYlM9uzVkuPT02fSUWPSysydcszAbgGrnUXwdmCNpDnAR\nyYinG4ET0grMbCCKlzTP77k9sameC5a20lm0gVK+ZvGFd8xxn4VZP6odHtud2y/iDOA/I+I/AS+y\nY6NO4eKCkNQ2VnzsjSX7MTp7kg5vj4Yyq6zaRLFJ0qeAc4GfSqoDGtILy2zo5PsxGkv0Y9RlxN2/\ne8HJwqyCahPFu0h2tjsvIv4ETAOu7O9NkubnliZvk3RxmTLvlPS4pNWSbi5VxmxnlatZbOno4dLl\nq5l7+V3u4DYro9oJd3+KiC9FxL2542cj4oZK78nVOq4hWaJ8NrBQ0uyiMjOBTwFzI+Jw4B8G8TOY\nVSVfs9itIcP4pu3L0m7p7GFbV9bNUGZlVEwUku7L/btJ0l8KHpsk/aWfax8DtEXE0xHRCdxC0sdR\n6IPANRHxEkBEvDC4H8OsOguOmsb9S07iX04/nPGNO65h3pDJ0P7S1hGKzGz06m8exRtz/06MiD0K\nHhMjYo9+rj0NWFNw3J47V2gWMEvS/ZIelDS/1IUkLZLUIqll/fr1/XysWWWTJzRx4mH79Bk625XN\nMr6xzst9mBWpdh7FcZImFhxPkHRsf28rca54UHs9MBOYBywErpM0qc+bIq6NiOaIaJ46dWo1IZtV\nNHlCE1eceSS7NWR65128s3k6p119H+de95D7LMwKDGQexdEFx6+UOFesHZhRcDwdWFeizIMR0QX8\nQdITJInj4SrjMhu0/F4X7S9tZXxjHaddfR/burJsI5lz4e1WzRJV70eRm0cBQERk6T/JPAzMlHSw\npEbgbGB5UZkfAycCSJpC0hT1dJUxme20/LyLLZ09NGS83IdZKdUmiqclfVxSQ+7xCfr5Qo+IbmAx\ncDvwW+AHEbFa0mWSFuSK3Q5skPQ4cDdwYURsGNyPYjZ40/fana5sts/5/KS8tuc3ue/CdlmKEmvh\n9Ckk7QNcBZxE0s9wJ/APIzFKqbm5OVpavHBtqubNS/69556RjGLYLV+5tuRyH7s1ZMhmg6b6ut6l\nyvNNVvmlQsxGO0mPRETzYN5bVR9FLiGcPZgPMKsVC46axuxX7cGpV91LZ8EeF9u6ksTR2dMNwCd/\nuIqMoLGuzntc2C6h2lFPsyTdKemx3PGRkj6Tbmhmw69wUt7Epnoa6zM0Fc3m7uoJOrqjd48LT9Sz\nsa7aUU/fAi4EvgkQEa255Tb+Na3AzEZKqdFQlNhFLy8/Uc9NUDZWVduZPS4iflV0rnuogzEbLfKj\noQ7dd+IO8y2a6jMUry3Ylc0yfa/d+72m9+62WlVtjeJFSa8mN2FO0lnAc6lFZTaKFNYwpu+1O/e3\nvchFy1p32Ie7v9rErSvXsqToPe7XsFpRbaL4KHAtcJiktcAfgHNSi8pslJk8oWmHPS4GMuppw+YO\nlixr3WEy30XLWpl76BQ3V1lN6DdRSMoAzRFxcm471ExEeA9J26UVJo7+tL+0lYZMpjdJgPs1rLb0\n20eRm4W9OPd8i5OE2cCUmszX2ZNl49Yu91dYTai2M/sOSRdImiFp7/wj1cjMxojiBQgb6kRPNstH\nb/q1Fx+0mlBtH8Xfk3Rkf6To/CFDG47Z2JTv11i9biMfvKGFjh7Y1JEMHHR/hY121dYoZpPsVrcK\nWAl8FTg8raDMxqLJE5rYc/dGGut23DDJ+3bbaFdtovgu8FqS9Z6+mnv+3bSCMhurSvVXeN9uG+2q\nbXp6TUTMKTi+W9KqNAIyG8vy/RUXLWulLiO2dPQAyb7dABcubWXSuEYO338PN0XZqFFtjeJRScfl\nD3K7292fTkhmY1ulfbs7urN86MZHXLuwUaXaRHEs8EtJz0h6BngAOEHSbyS1phad2RhVbt9ugFe6\netjWleXCpa384sn17ruwEVdt09P8VKMw2wUVNkNlJF7JNT/l5WsXWcJLftiIqqpGERF/rPQo9z5J\n8yU9IalN0sUlXn+fpPWSVuYeH9iZH8as1uSbob5x7tE01avP6/nahZcyt5FUbdPTgEmqIxlSewrJ\n8NqFkmaXKPr9iDgq97gurXjMRqvJE5o4ftY+XHlWsg/GuKJ+C9i+5IdXoLWRUG3T02AcA7RFxNMA\nkm4BzgAeT/EzzWpWn0l53dv7L7qyWR5bu5F3XfuAV6C1YZdajQKYBqwpOG7PnSt2pqRWSUslzSh1\nIUmLJLVIalm/fn0asZqNCsW1i4lN9ezWkOGfT5vN5376ONu6st5Zz4ZdmjWKvg2uuf0sCtwGfC8i\nOiR9iGQS30l93hRxLcky5zQ3N5ffasxsjCheytwr0NpISjNRtAOFNYTpwLrCAhGxoeDwW8DlKcZj\nVlOKlzIvntHdlc0yvrGOVWtermpfDLPBSrPp6WFgpqSDJTUCZwPLCwtIelXB4QLgtynGY1azileg\n3a0hwzubp3Pa1fdx7nUPeYKepSq1GkVEdEtaDNwO1AHXR8RqSZcBLRGxHPi4pAUk+2//GXhfWvGY\n1brC5qjxjXWcdvV93jXPhkWaTU9ExApgRdG5Swqefwr4VJoxmI0l+eaoVWtedp+FDZs0m57MLCWl\nVqHtymaZvtfuvceec2FDJdUahZmlo3D5j8J5FfnaxK0r17Kk6DXPubDBcqIwq1HFQ2jzSWLD5g6W\nLGt1/4UNGScKsxpWPIQW6HfOxYbNHX2Si1klThRmY0yl/gs3SdlguDPbbIwpNefiijOPBOhtkvIy\nIDYQrlGYjUGl+i88pNYGy4nCbIwq7r+oZkitWSluejLbRZRrkqq2NuF5Gbsu1yjMdiHlhtT2x53g\nuzYnCrNdTKkhtZV4Xoa56cnMKsrPyyiU7wS3XYMThZlV5E5wc6Iws4p2thPcap/7KMysX4PtBLex\nwYnCzKoy0E7wPK8tVftSbXqSNF/SE5LaJF1codxZkkJSc5rxmNnwunXlWuZefpe3a61xqSUKSXXA\nNcApwGxgoaTZJcpNBD4OPJRWLGY2/AqH1XptqdqWZo3iGKAtIp6OiE7gFuCMEuU+B1wBbEsxFjMb\nZh5WO3akmSimAWsKjttz53pJeh0wIyJ+UulCkhZJapHUsn79+qGP1MyGnIfVjh1pJgqVOBe9L0oZ\n4MvAJ/u7UERcGxHNEdE8derUIQzRzNLiYbVjR5qjntqBGQXH04F1BccTgSOAeyQB7Acsl7QgIlpS\njMvMhomH1Y4NaSaKh4GZkg4G1gJnA+/OvxgRG4Ep+WNJ9wAXOEmYjS3VDKv1ENrRLbVEERHdkhYD\ntwN1wPURsVrSZUBLRCxP67PNrHZ4ZdrRL9UJdxGxAlhRdO6SMmXnpRmLmY0+Xpm2NnitJzMbMR5C\nWxucKMxsxHgIbW1wojCzEeMhtLXBiwKa2YjyENrRz4nCzEbcQFam9VDa4edEYWY1w0NpR4b7KMys\nJng12pHjRGFmNcFDaUeOE4WZ1YSdGUq7YXMHq9a87NrHILmPwsxqQn4o7UVFfRT9dWi7X2PnOVGY\nWc0Y6FBaLxEyNJwozKymDGSSOgRvAAAJGUlEQVQobb5fI58kYHu/hhNF9dxHYWZj1kD6NdyPUZ5r\nFGY2ZlXbr+F+jMqcKMxsTOuvX8P9GP1LtelJ0nxJT0hqk3Rxidc/JOk3klZKuk/S7DTjMbNd0+QJ\nTcyZMankF7/nZ/QvtUQhqQ64BjgFmA0sLJEIbo6Iv4qIo4ArgC+lFY+ZWSmDnZ+xK/VppFmjOAZo\ni4inI6ITuAU4o7BARPyl4HA8ECnGY2bWx2CWOr915VrmXn4X5173EHMvv4vlK9cOY8TDL80+imnA\nmoLjduDY4kKSPgqcDzQCJ6UYj5lZSQOZn7Er9mmkWaNQiXN9agwRcU1EvBpYAnym5IWkRZJaJLWs\nX79+iMM0M6vcj1FoV+zTSDNRtAMzCo6nA+sqlL8FeHupFyLi2ohojojmqVOnDmGIZmYDU22fxljq\nw0gzUTwMzJR0sKRG4GxgeWEBSTMLDt8GPJViPGZmO62aPo2x1oeRWh9FRHRLWgzcDtQB10fEakmX\nAS0RsRxYLOlkoAt4CXhvWvGYmQ2VSn0aY7EPI9UJdxGxAlhRdO6SguefSPPzzczSUm7NqWrXl6ql\nLV09M9vMbAhV04dRa0uGeFFAM7Mh1F8fRi1u6eoahZnZEKvUh1FN09Roa5ZyojAzS0G5Poz+mqZG\nY7OUm57MzIZRpaap0dos5RqFmdkwK9c0NVp35HOiMDMbAaWapgazku1w9Gc4UZiZjRLV7siXN1z9\nGU4UZmajSLUr2fY3A3woaxpOFGZmo0y5EVOFKvVn3Nf2IkuWtVKfEZ3dWc5748GQqRv0970ThZlZ\nDSrXnzG+sa63ppH39f99msapB88Z7Gd5eKyZWQ0qN8x2S2cP9ZkS2wGV2iGoSq5RmJnVqFL9GRs2\nd9DZM7S7SrtGYWZWw4p35ps8oYlLT589pJ/hRGFmNsacc+yB/NOphw3Z9ZwozMzGoEXHv5p/e/sR\nQ3KtVBOFpPmSnpDUJuniEq+fL+lxSa2S7pR0YJrxmJntSs457kAe+czJfObUw8hu27JhsNdJLVFI\nqgOuAU4BZgMLJRU3nD0KNEfEkcBS4Iq04jEz2xVNntDEB45/Nd0vP/fMYK+RZo3iGKAtIp6OiE7g\nFuCMwgIRcXdEvJI7fBCYnmI8ZmY2CGkmimnAmoLj9ty5cs4DflbqBUmLJLVIalm/fv0QhmhmZv1J\nM1GUmt5RcnCvpHOBZuDKUq9HxLUR0RwRzVOnTh3CEM3MrD9pTrhrB2YUHE8H1hUXknQy8GnghIgY\nvZvGmpntotKsUTwMzJR0sKRG4GxgeWEBSa8DvgksiIgXUozFzMwGKbVEERHdwGLgduC3wA8iYrWk\nyyQtyBW7EpgA/FDSSknLy1zOzMxGSKprPUXECmBF0blLCp6fnObnm5nZzvPMbDMzq0gRQ7vKYNok\nbQKeGOk4RokpwIsjHcQo4Xuxne/Fdr4X270mIiYO5o21uMz4ExHRPNJBjAaSWnwvEr4X2/lebOd7\nsZ2klsG+101PZmZWkROFmZlVVIuJ4tqRDmAU8b3YzvdiO9+L7Xwvthv0vai5zmwzMxtetVijMDOz\nYeREYWZmFY3aRFHF7nhNkr6fe/0hSQcNf5TDwzsFbtffvSgod5akkDRmh0ZWcy8kvTP3u7Fa0s3D\nHeNwqeL/kQMk3S3p0dz/J6eORJxpk3S9pBckPVbmdUm6KnefWiUdXdWFI2LUPYA64PfAIUAjsAqY\nXVTmI8A3cs/PBr4/0nGP4L04ERiXe/7hXfle5MpNBH5BshlW80jHPYK/FzNJdpHcK3e8z0jHPYL3\n4lrgw7nns4FnRjrulO7F8cDRwGNlXj+VZN8fAccBD1Vz3dFao+h3d7zc8Xdzz5cCb5ZUag+MWued\nArer5vcC4HMk2+puG87ghlk19+KDwDUR8RJAjN0Vmqu5FwHskXu+JyW2PBgLIuIXwJ8rFDkDuCES\nDwKTJL2qv+uO1kRRze54vWUiWal2IzB5WKIbXkO2U+AY0O+9yC1dPyMifjKcgY2Aan4vZgGzJN0v\n6UFJ84ctuuFVzb34LHCupHaShUo/NjyhjToD/T4BRu8SHtXsjlf1Dno1bjA7BZ6QakQjp+K9kJQB\nvgy8b7gCGkHV/F7UkzQ/zSOpZd4r6YiIeDnl2IZbNfdiIfCdiPiipL8Gbszdi2z64Y0qg/reHK01\nimp2x+stI6mepDpZqcpVqwa6U+CCGLs7BfZ3LyYCRwD3SHqGpA12+Rjt0K72/5FbI6IrIv5Aspjm\nzGGKbzhVcy/OA34AEBEPALuRLBi4q6nq+6TYaE0U/e6Olzt+b+75WcBdkeutGWO8U+B2Fe9FRGyM\niCkRcVBEHETSX7MgIga9GNooVs3/Iz8mGeiApCkkTVFPD2uUw6Oae/Es8GYASa8lSRTrhzXK0WE5\n8He50U/HARsj4rn+3jQqm54ioltSfne8OuD6yO2OB7RExHLg2yTVxzaSmsTZIxdxeqq8F4U7BQI8\nGxELyl60RlV5L3YJVd6L24G/kfQ40ANcGBEbRi7qdFR5Lz4JfEvSP5I0tbxvLP5hKel7JE2NU3L9\nMZcCDQAR8Q2S/plTgTbgFeD9VV13DN4rMzMbQqO16cnMzEYJJwozM6vIicLMzCpyojAzs4qcKMzM\nrCInCrNhJOmg/MqekuZJGutLjdgY4ERhVoXcBCX//2K7JP/im5WR++v/t5K+BvwaeI+kByT9WtIP\nJU3IlXu9pF9KWiXpV5Im5t57b67sryW9YWR/GrPBc6Iwq+w1wA3AW0jWCzo5Io4GWoDzc0tGfB/4\nRETMAU4GtgIvAG/JlX0XcNVIBG82FEblEh5mo8gfI+JBSaeRbHhzf26ZlEbgAZJE8lxEPAwQEX8B\nkDQeuFrSUSTLZ8waieDNhoIThVllW3L/CrgjIhYWvijpSEov0/yPwPPAHJKa+1jeRMnGODc9mVXn\nQWCupEMBJI2TNAv4HbC/pNfnzk8sWPb+udx+B+8hWazOrCY5UZhVISLWk2yI9D1JrSSJ47Dc1pvv\nAr4qaRVwB8kS1l8D3ivpQZJmpy0lL2xWA7x6rJmZVeQahZmZVeREYWZmFTlRmJlZRU4UZmZWkROF\nmZlV5ERhZmYVOVGYmVlF/x+dnDomrvk4VAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f06268dde10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot P-R AUC w/ best cutoff\n",
    "title_ = 'P-R Curve: Cutoff = ' + str(best_cut)\n",
    "ax = prauc_frame.plot(x='recall', y='precision', kind='scatter', title=title_, xlim=[0,1])\n",
    "_ = ax.axvline(best_cut, color='r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In h2o Flow a traditional ROC curve is displayed, in which true positive rate is plotted against false positive rate. You can still use h2o Flow and the traditional ROC curve to verify that F1 is maximized at a probability cutoff of ~ 0.27."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Report Raw Confusion Matrices\n",
    "\n",
    "The basic DIA procedure in this notebook is based on measurements found commonly in confusion matrices, so confusion matrices are calculated as a precursor to DIA and to provide a basic summary of the GBM's behavior in general and across men and women."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function to print confusion matrices by an input variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>actual: 1</th>\n",
       "      <th>actual: 0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>predicted: 1</th>\n",
       "      <td>1051</td>\n",
       "      <td>859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>predicted: 0</th>\n",
       "      <td>952</td>\n",
       "      <td>6078</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             actual: 1 actual: 0\n",
       "predicted: 1      1051       859\n",
       "predicted: 0       952      6078"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_confusion_matrix(frame, y, yhat, by=None, level=None, cutoff=0.5):\n",
    "\n",
    "    \"\"\" Creates confusion matrix from pandas dataframe of y and yhat values, can be sliced \n",
    "        by a variable and level.\n",
    "    \n",
    "    Args:\n",
    "        frame: Pandas dataframe of actual (y) and predicted (yhat) values.\n",
    "        y: Name of actual value column.\n",
    "        yhat: Name of predicted value column.\n",
    "        by: By variable to slice frame before creating confusion matrix, default None.\n",
    "        level: Value of by variable to slice frame before creating confusion matrix, default None.\n",
    "        cutoff: Cutoff threshold for confusion matrix, default 0.5. \n",
    "\n",
    "    Returns:\n",
    "        Confusion matrix as pandas dataframe. \n",
    "    \"\"\"\n",
    "    \n",
    "    # determine levels of target (y) variable\n",
    "    # sort for consistency\n",
    "    level_list = list(frame[y].unique())\n",
    "    level_list.sort(reverse=True)\n",
    "\n",
    "    # init confusion matrix\n",
    "    cm_frame = pd.DataFrame(columns=['actual: ' +  str(i) for i in level_list], \n",
    "                            index=['predicted: ' + str(i) for i in level_list])\n",
    "    \n",
    "    # don't destroy original data\n",
    "    frame_ = frame.copy(deep=True)\n",
    "    \n",
    "    # convert numeric predictions to binary decisions using cutoff\n",
    "    dname = 'd_' + str(y)\n",
    "    frame_[dname] = np.where(frame_[yhat] > cutoff , 1, 0)\n",
    "    \n",
    "    # slice frame\n",
    "    if (by is not None) & (level is not None):\n",
    "        frame_ = frame_[frame[by] == level]\n",
    "    \n",
    "    # calculate size of each confusion matrix value\n",
    "    for i, lev_i in enumerate(level_list):\n",
    "        for j, lev_j in enumerate(level_list):\n",
    "            cm_frame.iat[j, i] = frame_[(frame_[y] == lev_i) & (frame_[dname] == lev_j)].shape[0]\n",
    "            # i, j vs. j, i nasty little bug ... updated 8/30/19\n",
    "    \n",
    "    # output results\n",
    "    if by is None:\n",
    "        print('Confusion matrix:')\n",
    "    else:\n",
    "        print('Confusion matrix by ' + by + '=' + level)\n",
    "    \n",
    "    return cm_frame\n",
    "    \n",
    "    \n",
    "get_confusion_matrix(test_yhat, 'DEFAULT_NEXT_MONTH', 'p_DEFAULT_NEXT_MONTH', cutoff=best_cut)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The general confusion matrix shows that the GBM is more accurate than not because the true positive and true negative cells contain the largest values by far. But the GBM seems to make a larger number of type II errors or false negative predictions. False negatives can be a disparity issue, because for complex reasons, many credit scoring and other models tend to over-estimate the likelihood of non-reference groups - typically people other than white males - to default. This is both a sociological fairness problem and a financial problem if an unpriviledged group is not recieving the credit they deserve, in favor of undeserving white males. Deserving people miss out on potentially life-changing credit and lenders incur large write-off costs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Report confusion matrices by `SEX`\n",
    "\n",
    "The only values for `SEX` in the dataset are `female` and `male`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "['female', 'male']"
      ],
      "text/plain": [
       "['female', 'male']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sex_levels = list(test_yhat['SEX'].unique())\n",
    "sex_levels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Confusion matrix for `SEX = male`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix by SEX=male\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>actual: 1</th>\n",
       "      <th>actual: 0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>predicted: 1</th>\n",
       "      <td>453</td>\n",
       "      <td>345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>predicted: 0</th>\n",
       "      <td>418</td>\n",
       "      <td>2261</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             actual: 1 actual: 0\n",
       "predicted: 1       453       345\n",
       "predicted: 0       418      2261"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "male_cm = get_confusion_matrix(test_yhat, 'DEFAULT_NEXT_MONTH', 'p_DEFAULT_NEXT_MONTH', by='SEX', level='male', cutoff=best_cut)\n",
    "male_cm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Confusion matrix for `SEX = female`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix by SEX=female\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>actual: 1</th>\n",
       "      <th>actual: 0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>predicted: 1</th>\n",
       "      <td>598</td>\n",
       "      <td>514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>predicted: 0</th>\n",
       "      <td>534</td>\n",
       "      <td>3817</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             actual: 1 actual: 0\n",
       "predicted: 1       598       514\n",
       "predicted: 0       534      3817"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "female_cm = get_confusion_matrix(test_yhat, 'DEFAULT_NEXT_MONTH', 'p_DEFAULT_NEXT_MONTH', by='SEX', level='female', cutoff=best_cut)\n",
    "female_cm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both confusion matrices reflect the global confusion matrix: more accurate than not with a larger number of false negative predictions (type II errors) than false positive predictions (type I errors)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Disparate Impact Analysis (DIA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform the following basic DIA many different values from the confusion matrices reflecting different prediction behavior are calculated. These metrics essentially help us understand the GBM's overall performance and how it behaves when predicting:\n",
    "\n",
    "* Default correctly\n",
    "* Non-default correctly\n",
    "* Default incorrectly (type I errors)\n",
    "* Non-default incorrectly (type II errors)\n",
    "\n",
    "In a real-life lending scenario, type I errors essentially amount to false accusations of financial impropriety and type II errors result in awarding loans to undeserving customers. Both types of errors can be costly to the lender too. Type I errors likely result in lost interest and fees. Type II errors often result in write-offs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dictionary of metrics used to assess disparity and parity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# represent metrics as dictionary for use later\n",
    "metric_dict = {\n",
    "\n",
    "#### overall performance\n",
    "'Prevalence': '(tp + fn) / (tp + tn +fp + fn)', # how much default actually happens for this group\n",
    "#'Adverse Impact': '(tp + fp) / (tp + tn + fp + fn)', # how often the model predicted default for each group   \n",
    "'Accuracy':       '(tp + tn) / (tp + tn + fp + fn)', # how often the model predicts default and non-default correctly for this group\n",
    "\n",
    "#### predicting default will happen\n",
    "# (correctly)\n",
    "'True Positive Rate': 'tp / (tp + fn)',  # out of the people in the group *that did* default, how many the model predicted *correctly* would default              \n",
    "'Precision':          'tp / (tp + fp)',  # out of the people in the group the model *predicted* would default, how many the model predicted *correctly* would default\n",
    "\n",
    "#### predicting default won't happen\n",
    "# (correctly)\n",
    "'Specificity':              'tn / (tn + fp)', # out of the people in the group *that did not* default, how many the model predicted *correctly* would not default\n",
    "'Negative Predicted Value': 'tn / (tn + fn)', # out of the people in the group the model *predicted* would not default, how many the model predicted *correctly* would not default  \n",
    "\n",
    "#### analyzing errors - type I\n",
    "# false accusations \n",
    "'False Positive Rate':  'fp / (tn + fp)', # out of the people in the group *that did not* default, how many the model predicted *incorrectly* would default\n",
    "'False Discovery Rate': 'fp / (tp + fp)', # out of the people in the group the model *predicted* would default, how many the model predicted *incorrectly* would default\n",
    "\n",
    "#### analyzing errors - type II\n",
    "# costly ommisions\n",
    "'False Negative Rate': 'fn / (tp + fn)', # out of the people in the group *that did* default, how many the model predicted *incorrectly* would not default\n",
    "'False Omissions Rate':'fn / (tn + fn)'  # out of the people in the group the model *predicted* would not default, how many the model predicted *incorrectly* would not default\n",
    "}    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Utility function to translate metrics into Pandas statements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# small utility function\n",
    "# translates abbreviated metric expressions into executable Python statements\n",
    "\n",
    "def cm_exp_parser(expression):\n",
    "    \n",
    "    # tp | fp       cm_dict[level].iat[0, 0] | cm_dict[level].iat[0, 1]\n",
    "    # -------  ==>  --------------------------------------------\n",
    "    # fn | tn       cm_dict[level].iat[1, 0] | cm_dict[level].iat[1, 1]\n",
    "\n",
    "    expression = expression.replace('tp', 'cm_dict[level].iat[0, 0]')\\\n",
    "                           .replace('fp', 'cm_dict[level].iat[0, 1]')\\\n",
    "                           .replace('fn', 'cm_dict[level].iat[1, 0]')\\\n",
    "                           .replace('tn', 'cm_dict[level].iat[1, 1]')\n",
    "\n",
    "    return expression\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate and report metrics\n",
    "This nested loop calculates all the metrics defined above for men and women."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Prevalence</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>True Positive Rate</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Specificity</th>\n",
       "      <th>Negative Predicted Value</th>\n",
       "      <th>False Positive Rate</th>\n",
       "      <th>False Discovery Rate</th>\n",
       "      <th>False Negative Rate</th>\n",
       "      <th>False Omissions Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>0.207212</td>\n",
       "      <td>0.808164</td>\n",
       "      <td>0.528269</td>\n",
       "      <td>0.537770</td>\n",
       "      <td>0.881321</td>\n",
       "      <td>0.877270</td>\n",
       "      <td>0.118679</td>\n",
       "      <td>0.462230</td>\n",
       "      <td>0.471731</td>\n",
       "      <td>0.122730</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>0.250503</td>\n",
       "      <td>0.780558</td>\n",
       "      <td>0.520092</td>\n",
       "      <td>0.567669</td>\n",
       "      <td>0.867613</td>\n",
       "      <td>0.843972</td>\n",
       "      <td>0.132387</td>\n",
       "      <td>0.432331</td>\n",
       "      <td>0.479908</td>\n",
       "      <td>0.156028</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Prevalence  Accuracy  True Positive Rate  Precision  Specificity  \\\n",
       "female    0.207212  0.808164            0.528269   0.537770     0.881321   \n",
       "male      0.250503  0.780558            0.520092   0.567669     0.867613   \n",
       "\n",
       "        Negative Predicted Value  False Positive Rate  False Discovery Rate  \\\n",
       "female                  0.877270             0.118679              0.462230   \n",
       "male                    0.843972             0.132387              0.432331   \n",
       "\n",
       "        False Negative Rate  False Omissions Rate  \n",
       "female             0.471731              0.122730  \n",
       "male               0.479908              0.156028  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# initialize dict of confusion matrices and corresponding rows of dataframe\n",
    "cm_dict = {'male': male_cm, \n",
    "           'female': female_cm} \n",
    "\n",
    "metrics_frame = pd.DataFrame(index=sex_levels) # frame for metrics\n",
    "\n",
    "# nested loop through:\n",
    "# - sex levels\n",
    "# - metrics \n",
    "for level in sex_levels:\n",
    "    for metric in metric_dict.keys():\n",
    "              \n",
    "        # parse metric expressions into executable pandas statements\n",
    "        expression = cm_exp_parser(metric_dict[metric])\n",
    "\n",
    "        # dynamically evaluate metrics to avoid code duplication\n",
    "        metrics_frame.loc[level, metric] = eval(expression)  \n",
    "\n",
    "# display results                \n",
    "metrics_frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From eyeballing the raw metrics it appears that the model is treating men and women roughly similarly as groups. (These metrics do not indicate very much about specific, individual cases of bias, but only give information for the groups as a whole.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot false omissions rate by `SEX`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the confusion matrices indicated there might be a problem with non-default predictions, false omissions rate will be examined closely. False omissions measures how many customers the model predicted *incorrectly* would not default, out of the customers in the group the model *predicted* would not default. (Of course other metrics are important from a fairness and business perspective. For an excellent in-depth discussion and example regarding numerous disparity metrics, see: https://github.com/dssg/aequitas/blob/master/docs/source/examples/compas_demo.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEhCAYAAACTNXDdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAGYBJREFUeJzt3X20XXV95/H3h0RAHgSFa9U8mFBi\n24syVg7RtootVE0cIc6a2CbUJXToSlvNckZbK0zrlMZOfWqlnSVTSYuKOBqRUSejaOrA1E59wJwg\nBS404zUiuQTGi+FBihgCn/lj76uHw0nOvjeHnJv7+7zWOitn//Zv7/093M3n7PPbZ+8j20RERBkO\nG3YBERFx8CT0IyIKktCPiChIQj8ioiAJ/YiIgiT0IyIKktCPg0LSxyRdPOw6+pH0DkkfPIDlz5P0\nhUHWFDFICf2YFkm3S/qhpAc7Hs8ZYj0vlfT3dR33S/ofkn52puuz/U7bv3MAy19he+VMl58JSRMd\nf5O7JV0u6eiGy54sKRfrFCShHzNxtu1jOh67hlGEpJcBXwSuBp4FnATcCnxF0pJh1DREK20fA7wI\neDHwB0OuJ2aphH4MhKTDJF1dH2neVx99/9w++j5T0jV1v92S/qFj3kJJn5E0Kek7kt60n82+F/iQ\n7Q/YftD2921fBNwA/Kd6fb9afzq5qF7nLklnS3qNpG/V2/9xQEr6U0kfqZ8fJenjkr5f1/oNSSfW\n8y6o1/sDSTskranbf0vS33es76WS2vWnkG9IenHHvH+U9CeSvlqv54uSntFv2/tTvwH/HfDCju2c\nI+nGeht3SHpHxyL/UPeZ+tR2esfr+GdJ90r6gqRF/bYdh4aEfgzS54BlVEfdtwBX7qPf24AdwEjd\n9x0AkubV69gKLABeAbxN0lndK5B0LNUR7ad6rP+qetkpC6n29ecA7wQuB9YAPw/8MrBB0uIe6/lN\n4Kh6+ROANwIPS3oa8H7gFbaPBX4JuKlHjScCnwf+ol7+vwDXSHp6R7dzgfOAnwKOBt66v233qLF7\nm4uAFcB4R/ODwOuB44CzgX8v6TX1vDMAOj61bZW0mupvtIrqb3Q98PF+245DQ0I/ZuKz9dHnfZI+\nC2D7Mdsfsf0D2w8DFwOn7WNs+RGqAF5se4/tL9ftLwGeZvvP6vZxfhLQ3U4ABNzVY95dQOdR8cPA\nu20/AmyiCrJL6k8HNwHbgVP3UeeJwMm2H7Xdtv1gPc/A8yUdafsu27f2WP5sYMz2J2zvtf0xqje7\nf93R53Lb37L9ENUb2NQR+v623cvnJP0AuAOYADZMzbB9ne1b6r/RP9X/DV6+n3X9NvBntrfb3gv8\nKbBc0oL9LBOHiIR+zMRrbR9fP14L1VG6pPfWQx0P8JMjzV5DEu8GvgtcK+nbkt5Wtz8XWNzxhnIf\n1dj0s3qsYzdV8D67x7xnA/d0TN9j+9H6+Q/rf/9fx/wfAsf0WM9HgP8FXCXpTknvljTf9gPAWuBN\nwN2SPifpeT2Wf079Ojt9l+pTzJS7O54/1FFHz2332MaU19SfOs4CTgGeMTVD0i/Uw22Tku4Hfove\nf5cpzwUu7fgb3AM8RvWpIw5xCf0YlDcArwbOpBpGOLluV3dH2w/YfovtJcBrgbdLejmwE/hWxxvK\n8baPtX12r3UA3wBe16OWXwOuPdAXVH/auNj2zwEvBf4N8Bv1vC/Y/lWqN5hx4LIeq9hFFaCdFgN3\nHsi2+yx3HfAx4H0dzZuA/w4ssn0c8Lf85O/S65s7O4ELuv4OT7V9fb/tx+yX0I9BORb4EfB9qrHo\n/7yvjvWJ1J+WJOB+4NH68TVgj6Tfk3Rk/enhBZJO28eq3g5cIOlNko6R9AxJ7wJadAxvzJSkMyU9\nX9JhwANUQy6PSnp2/RqOAvYA/1LX3+1zwCmSfl3SfEnnUr0ZXjPTbTcs/RLg1ZKeX08fC+y2/bCk\nl/D44bLvAZZ0UkfbB4E/VH0iXtLx9Th/zAEJ/RiUD1Md2e4CxoCv7qfvzwDXUZ1g/ArwV7b/sR4/\nfjWwHLidaljhMuBpvVZSnwtYSXVkf3e9zPOBX7K944BfUTU882mq0B2jGm75BDCP6kTnXVRvcr8I\nrO9R3yRwDtWb0/eBt1ANw+w+gG33Zftu4L9RnyAHfhd4Vz3m/x+pTnRP9f0B8C7g+no4p2X7U1Qn\nqj9VD9XdBLyqybZj9lN+RCUiohw50o+IKEhCPyKiIAn9iIiCJPQjIgqS0I+IKMj+rvAbihNPPNFL\nliwZdhkREYeUbdu23WN7pF+/RqEvaQXwV1TfT/5b2+/umn8G8JdU9y9ZY/vqjnmLqa4AXER19d+r\nbd++r20tWbKEdrvdpKyIiKhJ6r7lR099h3fqOx9eSnURzCiwVtJoV7c7gPPpfSe+jwLvqy8nX051\nBWBERAxBkyP95cD41BWOkjZR3XL1x3cVnDpyl/RY54L1m8N821+q++3vLoEREfEka3IidwHVDZim\nTPD4uwTuz/OA+yR9WtI3Jb2v/uQQERFD0CT0n3CXRHrfma+X+cDLgN8HTqf6Obvzn7ABaV3960Lt\nycnJhquOiIjpahL6E1QnYacspLqpVhMTwDdt76hvpvVZqt/wfBzbG223bLdGRvqefI6IiBlqEvpb\ngWWSlko6nOq2rJsbrn8r8HRJU0l+Jh3nAiIi4uDqG/r1Efp6YAtwG3CV7TFJGySdAyDpdEkTVD9o\ncZmksXrZR6mGdq6VdDPVUNHfPDkvJSIi+pl1t1ZutVrO9/QjIqZH0jbbrX79Zt0VuRFx4NTr6xcx\nI7PsuPiA5d47EREFSehHRBQkoR8RUZCEfkREQRL6EREFSehHRBQkoR8RUZCEfkREQRL6EREFSehH\nRBQkoR8RUZCEfkREQRL6EREFSehHRBQkoR8RUZCEfkREQRL6EREFaRT6klZI2i5pXNKFPeafIekG\nSXslre4x/2mS7pT0gUEUHRERM9M39CXNAy4FVgKjwFpJo13d7gDOBz6+j9W8E/jyzMuMiIhBaHKk\nvxwYt73D9h5gE7Cqs4Pt223fBDzWvbCk04CfAv5uAPVGRMQBaBL6C4CdHdMTdVtfkg4D/gJ4W59+\n6yS1JbUnJyebrDoiImagSeirR1vT34d/I3CN7Z3762R7o+2W7dbIyEjDVUdExHTNb9BnAljUMb0Q\n2NVw/b8AvEzSG4FjgMMlPWj7CSeDIyLiydck9LcCyyQtBe4E1gDnNlm57d+Yei7pfKCVwI+IGJ6+\nwzu29wLrgS3AbcBVtsckbZB0DoCk0yVNAK8DLpM09mQWHRERMyO76fD8wdFqtdxut4ddRsQhTb3O\nxMWMzLKI3CdJ22y3+vXLFbkREQVJ6EdEFCShHxFRkIR+RERBEvoREQVJ6EdEFCShHxFRkIR+RERB\nEvoREQVJ6EdEFCShHxFRkIR+RERBEvoREQVJ6EdEFCShHxFRkIR+RERBEvoREQVpFPqSVkjaLmlc\n0hN+41bSGZJukLRX0uqO9hdK+pqkMUk3Sfr1QRY/TFIeg3xExMHRN/QlzQMuBVYCo8BaSaNd3e4A\nzgc+3tX+EPAG26cAK4C/lHT8gRYdEREzM79Bn+XAuO0dAJI2AauAW6c62L69nvdY54K2/2/H812S\nvgeMAPcdcOURETFtTYZ3FgA7O6Yn6rZpkbQcOBz49nSXjYiIwWgS+r1GXKf1+/CSng1cCfym7cd6\nzF8nqS2pPTk5OZ1VR0TENDQJ/QlgUcf0QmBX0w1IehrweeCPbH+9Vx/bG223bLdGRkaarjoiIqap\nSehvBZZJWirpcGANsLnJyuv+nwE+avtTMy8zIiIGoW/o294LrAe2ALcBV9kek7RB0jkAkk6XNAG8\nDrhM0li9+K8BZwDnS7qxfrzwSXklERHRl+xpDc8/6Vqtltvt9rDL6CvfLR+sWbYbHvKyfw7OobJv\nStpmu9WvX67IjYgoSEI/IqIgCf2IiIIk9CMiCpLQj4goSEI/IqIgCf2IiIIk9CMiCpLQj4goSEI/\nIqIgCf2IiIIk9CMiCpLQj4goSEI/IqIgCf2IiIIk9CMiCpLQj4goSEI/IqIgjUJf0gpJ2yWNS7qw\nx/wzJN0gaa+k1V3zzpP0rfpx3qAKj4iI6esb+pLmAZcCK4FRYK2k0a5udwDnAx/vWvYZwB8DLwaW\nA38s6ekHXnZERMxEkyP95cC47R229wCbgFWdHWzfbvsm4LGuZV8FfMn2btv3Al8CVgyg7oiImIEm\nob8A2NkxPVG3NXEgy0ZExIA1CX31aHPD9TdaVtI6SW1J7cnJyYarjoiI6WoS+hPAoo7phcCuhutv\ntKztjbZbtlsjIyMNVx0REdPVJPS3AsskLZV0OLAG2Nxw/VuAV0p6en0C95V1W0REDEHf0Le9F1hP\nFda3AVfZHpO0QdI5AJJOlzQBvA64TNJYvexu4J1UbxxbgQ11W0REDIHspsPzB0er1XK73R52GX2p\n19mKmLFZthse8rJ/Ds6hsm9K2ma71a9frsiNiChIQj8ioiAJ/YiIgiT0IyIKktCPiChIQj8ioiAJ\n/YiIgiT0IyIKktCPiChIQj8ioiAJ/YiIgiT0IyIKktCPiChIQj8ioiAJ/YiIgiT0IyIKktCPiChI\no9CXtELSdknjki7sMf8ISZ+s518vaUnd/hRJV0i6WdJtki4abPkRETEdfUNf0jzgUmAlMAqslTTa\n1e0C4F7bJwOXAO+p218HHGH7BcBpwG9PvSFERMTB1+RIfzkwbnuH7T3AJmBVV59VwBX186uBsyQJ\nMHC0pPnAU4E9wAMDqTwiIqatSegvAHZ2TE/UbT372N4L3A+cQPUG8C/AXcAdwJ/b3n2ANUdExAw1\nCX31aOv+ffh99VkOPAo8B1gK/J6kk56wAWmdpLak9uTkZIOSIiJiJpqE/gSwqGN6IbBrX33qoZzj\ngN3AucAXbT9i+3vAV4BW9wZsb7Tdst0aGRmZ/quIiIhGmoT+VmCZpKWSDgfWAJu7+mwGzqufrwau\ns22qIZ0zVTkaeAnwz4MpPSIipqtv6Ndj9OuBLcBtwFW2xyRtkHRO3e1y4ARJ48BbgamvdV4KHAPc\nQvXm8WHbNw34NUREREOqDshnj1ar5Xa7Pewy+lKvsxgxY7NsNzzkZf8cnENl35S0zfYThs+75Yrc\niIiCJPQjIgqS0I+IKEhCPyKiIAn9iIiCJPQjIgqS0I+IKEhCPyKiIAn9iIiCJPQjIgqS0I+IKEhC\nPyKiIAn9iIiCJPQjIgqS0I+IKEhCPyKiIAn9iIiCJPQjIgrSKPQlrZC0XdK4pAt7zD9C0ifr+ddL\nWtIx71RJX5M0JulmSUcOrvyIiJiOvqEvaR7VD5yvBEaBtZJGu7pdANxr+2TgEuA99bLzgY8Bv2P7\nFOCXgUcGVn1ERExLkyP95cC47R229wCbgFVdfVYBV9TPrwbOkiTglcBNtv8JwPb3bT86mNIjImK6\nmoT+AmBnx/RE3dazj+29wP3ACcDzAEvaIukGSX9w4CVHRMRMzW/QRz3a3LDPfOClwOnAQ8C1krbZ\nvvZxC0vrgHUAixcvblBSRETMRJMj/QlgUcf0QmDXvvrU4/jHAbvr9i/bvsf2Q8A1wIu6N2B7o+2W\n7dbIyMj0X0VERDTSJPS3AsskLZV0OLAG2NzVZzNwXv18NXCdbQNbgFMlHVW/GbwcuHUwpUdExHT1\nHd6xvVfSeqoAnwd8yPaYpA1A2/Zm4HLgSknjVEf4a+pl75X0fqo3DgPX2P78k/RaIiKiD1UH5LNH\nq9Vyu90edhl9qddZjJixWbYbHvKyfw7OobJv1udLW/365YrciIiCJPQjIgqS0I+IKEhCPyKiIAn9\niIiCJPQjIgqS0I+IKEhCPyKiIAn9iIiCJPQjIgqS0I+IKEhCPyKiIAn9iIiCJPQjIgqS0I+IKEhC\nPyKiIAn9iIiCJPQjIgrSKPQlrZC0XdK4pAt7zD9C0ifr+ddLWtI1f7GkByX9/mDKjoiImegb+pLm\nAZcCK4FRYK2k0a5uFwD32j4ZuAR4T9f8S4AvHHi5ERFxIJoc6S8Hxm3vsL0H2ASs6uqzCriifn41\ncJZU/TSzpNcCO4CxwZQcEREz1ST0FwA7O6Yn6raefWzvBe4HTpB0NPB24E/2twFJ6yS1JbUnJyeb\n1h4REdPUJPTVo80N+/wJcIntB/e3Adsbbbdst0ZGRhqUFBERMzG/QZ8JYFHH9EJg1z76TEiaDxwH\n7AZeDKyW9F7geOAxSQ/b/sABVx4REdPWJPS3AsskLQXuBNYA53b12QycB3wNWA1cZ9vAy6Y6SLoY\neDCBHxExPH1D3/ZeSeuBLcA84EO2xyRtANq2NwOXA1dKGqc6wl/zZBYdEREzo+qAfPZotVput9vD\nLqMv9TqLETM2y3bDQ172z8E5VPZNSdtst/r1yxW5EREFSehHRBQkoR8RUZCEfkREQRL6EREFSehH\nRBQkoR8RUZCEfkREQRL6EREFSehHRBQkoR8RUZCEfkREQRL6EREFSehHRBQkoR8RUZCEfkREQRL6\nEREFaRT6klZI2i5pXNKFPeYfIemT9fzrJS2p218haZukm+t/zxxs+RERMR19Q1/SPOBSYCUwCqyV\nNNrV7QLgXtsnA5cA76nb7wHOtv0Cqh9Ov3JQhUdExPQ1OdJfDozb3mF7D7AJWNXVZxVwRf38auAs\nSbL9Tdu76vYx4EhJRwyi8IiImL4mob8A2NkxPVG39exjey9wP3BCV59/C3zT9o9mVmpERByo+Q36\nqEdb9+/D77ePpFOohnxe2XMD0jpgHcDixYsblBQRETPR5Eh/AljUMb0Q2LWvPpLmA8cBu+vphcBn\ngDfY/navDdjeaLtluzUyMjK9VxAREY01Cf2twDJJSyUdDqwBNnf12Ux1ohZgNXCdbUs6Hvg8cJHt\nrwyq6IiImJm+oV+P0a8HtgC3AVfZHpO0QdI5dbfLgRMkjQNvBaa+1rkeOBl4h6Qb68czB/4qIiKi\nEdndw/PD1Wq13G63h11GX+p1FiNmbJbthoe87J+Dc6jsm5K22W7165crciMiCpLQj4goSEI/IqIg\nCf2IiIIk9CMiCpLQj4goSEI/IqIgCf2IiIIk9CMiCpLQj4goSEI/IqIgCf2IiIIk9CMiCpLQj4go\nSEI/IqIgCf2IiIIk9CMiCpLQj4goSKPQl7RC0nZJ45Iu7DH/CEmfrOdfL2lJx7yL6vbtkl41uNIj\nImK6+oa+pHnApcBKYBRYK2m0q9sFwL22TwYuAd5TLzsKrAFOAVYA/7VeX0REDEGTI/3lwLjtHbb3\nAJuAVV19VgFX1M+vBs6SpLp9k+0f2f4OMF6vLyIihmB+gz4LgJ0d0xPAi/fVx/ZeSfcDJ9TtX+9a\ndkH3BiStA9bVkw9K2t6o+mjiROCeYRfRjzTsCmJIZv3+eQjtm89t0qlJ6Pd6yW7Yp8my2N4IbGxQ\nS0yTpLbt1rDriOgl++fB12R4ZwJY1DG9ENi1rz6S5gPHAbsbLhsREQdJk9DfCiyTtFTS4VQnZjd3\n9dkMnFc/Xw1cZ9t1+5r62z1LgWXANwZTekRETFff4Z16jH49sAWYB3zI9pikDUDb9mbgcuBKSeNU\nR/hr6mXHJF0F3ArsBd5k+9En6bVEbxk2i9ks++dBpuqAPCIiSpArciMiCpLQj4goSEI/IqIgCf2I\nOKgkPVXSzwy7jlIl9OcYSc+TdK2kW+rpUyX90bDrigCQdDZwI/DFevqFkrq/Ah5PooT+3PM3wEXA\nIwC2b6L+Cm3ELHAx1f237gOwfSOwZIj1FCehP/ccZbv7Ari9Q6kk4on22r5/2EWUrMm9d+LQco+k\nn6a+x5Gk1cBdwy0p4sdukXQuME/SMuDNwFeHXFNRcnHWHCPpJKqrHH8RuBf4DvB627cPs64IAElH\nAX8IvJLqhoxbgHfafniohRUkoT9HSToaOMz2D4ZdS0TMHgn9OULSW/c33/b7D1YtEd0k/U963FZ9\niu1zDmI5RcuY/txx7LALiNiPPx92AVHJkX5EREFypD/HSDqS6ofqTwGOnGq3/e+GVlRErf7GzruA\nUR6/f540tKIKk+/pzz1XAs8CXgV8merXynIyN2aLDwN/TXXtyK8AH6XaZ+MgyfDOHCPpm7Z/XtJN\ntk+V9BRgi+0zh11bhKRttk+TdLPtF9Rt/8f2y4ZdWykyvDP3PFL/e5+k5wN3k8vcY/Z4WNJhwLfq\nX+S7E3jmkGsqSoZ35p6Nkp4OvIPqN4pvBd473JIifuw/AEdRXYl7GvB64A1DragwGd6JiINGUovq\nitznAk+pm2371OFVVZaE/hwj6XiqI6cldAzf2X7zsGqKmCJpO/A24Gbgsal2298dWlGFyZj+3HMN\n8HW6/qeKmCUmbef++UOUI/05RtINtl807DoiepF0FrAWuBb40VS77U8PrajCJPTnGElvAR4EPsfj\n/6faPbSiImqSPgb8LDDGTz6JOhcPHjwZ3pl79gDvozpZNvWObiBXPMZs8K+mvp8fw5HQn3veCpxs\n+55hFxLRw9cljdq+ddiFlCqhP/eMAQ8Nu4iIfXgpcJ6k71ANP4p8ZfOgSujPPY8CN0r63zx+TD9f\n2YzZYMWwCyhdQn/u+Wz9iJh18n384cu3d+YgSU8FFtvePuxaImJ2yb135hhJZwM3Al+sp18oKRfD\nRASQ0J+LLgaWA/cB2L4RWDrMgiJi9kjozz17bd/f1ZYxvIgAciJ3LrpF0rnAvPqn6d4MfHXINUXE\nLJEj/TlC0tRPzn2b6vdxfwR8AniA6h7mERH59s5cIelWYCXVD6f8Svf83HsnIiDDO3PJB6m+sXMS\n0O5oF7n3TkTUcqQ/x0j6a9u/O+w6ImJ2SuhHRBQkJ3IjIgqS0I+IKEhCPyKiIAn9iIiCJPQjIgry\n/wFdVopeMSSjMAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f06267d3dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = metrics_frame['False Omissions Rate'].plot(kind='bar', color='b', title='False Omissions Rate')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate and report disparity\n",
    "To calculate disparity we compare the confusion matrix for each sex to the metrics for a user-defined reference level and to user-defined thresholds. In this case, we take the class of people who seem most priviledged as the reference level, i.e. `SEX = male`. (Usually the reference level would be `race = white` or `sex = male`.) According to the four-fifths rule (https://en.wikipedia.org/wiki/Disparate_impact#The_80%_rule) thresholds are set such that metrics 20% lower or higher than the reference level metric will be flagged as disparate. **Technically, the four-fifths rule only applies to the adverse impact ratio, discussed further below, but it will be applied to all other displayed metrics as a rule of thumb.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col0 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col1 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col2 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col3 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col4 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col5 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col6 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col7 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col8 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col9 {\n",
       "            color:  red;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col0 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col1 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col2 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col3 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col4 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col5 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col6 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col7 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col8 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col9 {\n",
       "            color:  blue;\n",
       "        }</style>  \n",
       "<table id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54e\" > \n",
       "<thead>    <tr> \n",
       "        <th class=\"blank level0\" ></th> \n",
       "        <th class=\"col_heading level0 col0\" >Prevalence Disparity</th> \n",
       "        <th class=\"col_heading level0 col1\" >Accuracy Disparity</th> \n",
       "        <th class=\"col_heading level0 col2\" >True Positive Rate Disparity</th> \n",
       "        <th class=\"col_heading level0 col3\" >Precision Disparity</th> \n",
       "        <th class=\"col_heading level0 col4\" >Specificity Disparity</th> \n",
       "        <th class=\"col_heading level0 col5\" >Negative Predicted Value Disparity</th> \n",
       "        <th class=\"col_heading level0 col6\" >False Positive Rate Disparity</th> \n",
       "        <th class=\"col_heading level0 col7\" >False Discovery Rate Disparity</th> \n",
       "        <th class=\"col_heading level0 col8\" >False Negative Rate Disparity</th> \n",
       "        <th class=\"col_heading level0 col9\" >False Omissions Rate Disparity</th> \n",
       "    </tr></thead> \n",
       "<tbody>    <tr> \n",
       "        <th id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54elevel0_row0\" class=\"row_heading level0 row0\" >female</th> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col0\" class=\"data row0 col0\" >0.827183</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col1\" class=\"data row0 col1\" >1.03537</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col2\" class=\"data row0 col2\" >1.01572</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col3\" class=\"data row0 col3\" >0.94733</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col4\" class=\"data row0 col4\" >1.0158</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col5\" class=\"data row0 col5\" >1.03945</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col6\" class=\"data row0 col6\" >0.896459</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col7\" class=\"data row0 col7\" >1.06916</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col8\" class=\"data row0 col8\" >0.982962</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow0_col9\" class=\"data row0 col9\" >0.78659</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54elevel0_row1\" class=\"row_heading level0 row1\" >male</th> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col0\" class=\"data row1 col0\" >1</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col1\" class=\"data row1 col1\" >1</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col2\" class=\"data row1 col2\" >1</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col3\" class=\"data row1 col3\" >1</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col4\" class=\"data row1 col4\" >1</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col5\" class=\"data row1 col5\" >1</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col6\" class=\"data row1 col6\" >1</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col7\" class=\"data row1 col7\" >1</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col8\" class=\"data row1 col8\" >1</td> \n",
       "        <td id=\"T_5c18ff32_5199_11ea_bb38_28b2bdeda54erow1_col9\" class=\"data row1 col9\" >1</td> \n",
       "    </tr></tbody> \n",
       "</table> "
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f06268149b0>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ref_level = 'male' # user-defined reference level\n",
    "\n",
    "parity_threshold_low = 0.8    # user-defined low threshold value\n",
    "parity_threshold_hi = 1.25    # user-defined high threshold value\n",
    "\n",
    "# init frame to store disparity measures\n",
    "disp_frame = pd.DataFrame(index=sex_levels)\n",
    "\n",
    "# compare all metrics to reference level\n",
    "disp_frame = metrics_frame/metrics_frame.loc[ref_level, :]\n",
    "\n",
    "# change column names\n",
    "disp_frame.columns=[col + ' Disparity' for col in metrics_frame.columns]\n",
    "\n",
    "# small utility function to format pandas table output\n",
    "def disparate_red(val):\n",
    "    \n",
    "    color = 'blue' if (parity_threshold_low < val < parity_threshold_hi) else 'red'\n",
    "    return 'color: %s' % color \n",
    "\n",
    "# display results\n",
    "disp_frame.style.applymap(disparate_red)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the selected thresholds, the GBM appears to have only one value for metrics that is low out-of-range, false omissions rate. The flagged false omissions rate disparity indicates males may be receiving too many loans they cannot pay back, potentially preventing females from recieving these loans."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot false omissions rate disparity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEhCAYAAACEF+AUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAGgVJREFUeJzt3X+cVPV97/HXW9AKETUB0hgB0Qok\nxCgmW38leYiBewMUML1ShRSjrQle1OsDQ3IjBq0RK6k/qu2j/oAmKQZvVIKJRQqah0YaE9S4JsQK\nFt2iCRs0Aipq0QDmc/84Z83s7OzO7O7A7Hz3/Xw85rHnnO93zvnM7Jn3nPnOmRlFBGZmlpb9al2A\nmZlVn8PdzCxBDnczswQ53M3MEuRwNzNLkMPdzCxBDveESbpD0pW1rqMcSZdLuq0b1z9H0upq1pQC\nSRslfWofb7OPpDclDduX27W2HO51QNILkt7KHzQtlw/WsJ5PSlqT17FD0r9K+lBX1xcRCyLif3fj\n+rdHxMSuXr8rJDUX/E9ekvQtSe+p8LpHS+ryB0xarl+wL7wk6T5J4wr7RcSoiHikq9vpioh4JyIO\niohf57XWxQFGihzu9WNK/qBpuWypRRH5keD9wHLgA8BRwAbgp5KG16KmGpoYEQcBHwNOBP7vvtx4\ny74AHA/8CFghaea+rKGQpL612ra15XCvY5L2k7Q8P3J7LT+a/nA7fd8vaVXe7xVJPy5oGyLpB5K2\nSnpe0oUdbPZa4NsR8U8R8WZEbI+IecDPgSvy9Y3PX23My9e5RdIUSZMlPZdv/90glHS1pCX5dH9J\n35W0Pa/1Z5IG5W3n5et9Q9ImSdPz5V+QtKZgfZ+U1Ji/qviZpBML2n4i6euS1ubruV/S+8ptuyP5\nE+0PgTEF25kqaV2+jV9LurzgKj/O+7Qcef9pwe34T0mvSlotaWi5befbfzEibgQWANdKUr6+Zklj\n8+mTJP1c0uuSfivpunx5y6uAL+b/py2SLim4HSdLeiy/P16U9I+S9s/b+ubXvUBSE/CfBcuGS7oA\nOAu4LL+dP8j3ibsL65d0q6TrK7mt1gkR4UsPvwAvAONLLN8POBcYABwI/BPQWNB+B3BlPn1d3r4/\ncABwar68D7AOuCxffnS+vXEltjcA+D3wqRJtXwQ259PjgT3A1/LtzQZezus5CDgWeBsYlve/GliS\nT18I3Av0y2tryK9zMLADGJH3OwwYnU9/AViTTw/K+80A+gIzge3Ae/P2nwDPASOA/sAjwNUdbbud\n/0kzMDafHkr26uWGgvZPA8fk/6PjgG3A5Lzt6Oyh12p904CNwKi87iuBR9rZdpvr58tHAlFwHxXW\n+AQwo+D/eGLhuoCl+f1xXH5/tVzvT8lelfQle5X2LHBR3tY3v+79wHvz+61l2fDifTCfHwK8CRyc\nzx+Q3zfH1fpxltrFR+7149786Ok1SfcCRMTvI2JJRLwREW+TBcLH2xn73Q18kCxQd0XEv+fLTyJ7\noF2TL28CvgVML7GOgYCAF0u0vUgWrC3eBr4REbuBu4DBwI2RHe0/RRZkx7ZT5yDg6MjGbxsj4s28\nLYBjJB0Y2dHqhhLXnwKsj4g7I2JPRNwBbAL+rKDPtyLiuYjYCXyPPxxxd7TtUlZKegP4NVmQXtXS\nEBE/ioin8//RL/P74NQO1nU+cE1EbIyIPWRPeCdIOryD6xRrGap7X4m23cAISQPz/eXxovavR8TO\nvNbbyZ4ciYgnIuLx/L7cBCwucTuuiYhXI+KtcgVGRDPwKHBGvmgSsCXfrlWRw71+fDYiDs0vn4V3\nz0y4Nh+ieB1oyvuWGkr4BvAr4CFJ/yXpK/nyI4BhBU8cr5GNHX+gxDpeIQvYw0q0HUZ2BNZiW0S8\nk0+3POh/W9D+FtkRebElwIPAMkm/kfQNSX0j4nWywLkQeEnSSkkjS1z/g/ntLPQroDAkXyqY3llQ\nR8ltl9hGi8kRMQAYB3yEglDNhzPW5MNSO8heXXQ0xHMEcHPB/2Ab2aukIR1cp1jLbXylRNtfAaOB\njflw06Si9s0F078iux+R9CFJ/6Zs6O91siew4tuxmc65newVFfnfpZ28vlXA4V7fPk925PNp4BCy\nl9iQHV23EhGvR8QlETEc+CzwVUmnkj0wnyt44jg0IgZExJRS6wB+BvxFiVrOBB7q7g3KXz1cGREf\nBj4J/Dnwl3nb6ogYT/ZE0gQsKrGKLWRBWWgY8JvubLvM9X5ENvxwXcHiu4B7gKERcQjwTf7wfyl1\npsxm4Lyi/0O/EkfYHflzsieupuKG/BXBdOD9wA3APZIOLOhSOL4/jD+8ClgEPE32auZgsvdVivev\njs78KdX2fbJXmB8BJgLf7eD61kUO9/o2APgd2Rhpf+Bv2+uYv6H5J/mbbTuAd/LLo8AuSXMlHZi/\nGviopI+3s6qvAudJulDSQZLeJ2kh2fj0Ve1cp2KSPi3pGEn7Aa+TDSe8I+mw/Db0B3YB/53XX2wl\n8BFJZ+Vv7n2O7ElvVVe3XWHpNwKTJB2Tzw8AXomItyWdROthrpeBkHRUwbLbgK8pf0Nc0qGSplWy\nYUl/LOliYD7w1YhoE6iSzpY0KCJ+T/b/D7JXBi0ul9RP0keBc4CWNz0H5P3/O6/t/EpqKvBbsrH6\nd+XDYT8A7gR+GhFln3it8xzu9e1fyI6wtgDrgbUd9B1Fdrrcm8BPgX+IiJ/k47uTgBPI3kjdRna0\ndnCpleRj9RPJjtRfyq9zDPCJfEy2uz5IdmT3OtltepAsBPoAXyEb298OnAJcVKK+rcBUsieh7cAl\nZMMnpYYqKt12WRHxEvD/gJazYmYDC/Mx+cuAZQV93wAWAo/nwzANEfE94O+B7+XDH08Bn+lomy1n\n2xT0/V8R8Z12uk8CnsnruR44KyJ2FbT/hOy9iR8CC/NXIwBzycL+DbL9otWZLhX4JnBcfgbQ8oLl\ntwMfxUMye41KPMmbWS8h6WiyYbk2Q3l7ebtHkT0pfaDMm9bWRT5yN7N9Kh/2+hLwXQf73uNPlJnZ\nPiPpELI3t1+gzLCTdY+HZczMEuRhGTOzBDnczcwSVLMx90GDBsXw4cNrtXkzs7r05JNPbouIweX6\n1Szchw8fTmNjY602b2ZWlyQVf71GSR6WMTNLkMPdzCxBDnczswQ53M3MEuRwNzNLkMPdzCxBDncz\nswTV7ovDNm6EsWNbLzvzTLjgAti5EyYV/woYcO652WXbNphW4ncMZs+Gs86CzZvh7LPbts+dC1Om\nZNs+v8RvDsyfD+PHw7p1MGdO2/ZrroFTToG1a+Gyy9q233QTjBkDDz4IV1/dtn3RIhg1Cu67D264\noW370qUwdCjcfTfcemvb9uXLYdAgWLIkuxRbtQr694dbboFly9q2r1mT/b3+eli5snVbv36wenU2\nvWABPFT0o0oDB8I992TT8+bBo4+2bh8yBO64I5ueMye7DwuNHAmLF2fTs2bBs8+2bh8zJrv/AGbO\nhObm1u0nnwwLF2bTZ5wB27e3bh83Di7Pv0p94kR4q+jnPCdPhi9/OZsu3u/A+573vWy6Hve9dvjI\n3cwsQTX7VsiGhobwJ1TNzDpH0pMR0VCun4/czcwS5HA3M0uQw93MLEFlw13StyW9LOnpdtr/UtJT\n+WWtpOOqX6aZmXVGJUfuS4AJHbQ/D5waEccCC4DFVajLzMy6oex57hHxY0nDO2hfWzD7GDCk+2WZ\nmVl3VHvM/TxgdXuNkmZJapTUuHXr1ipv2szMWlQt3CWdRhbuX22vT0QsjoiGiGgYPLjsr0SZmVkX\nVeXrByQdC3wTmBgR28v1NzOzvavbR+6ShgHfB86OiGfL9Tczs72v7JG7pDuBscAgSc3A3wD7A0TE\nbcAVwEDgFkkAeyr5aKyZme09lZwtM6NM+xeAL1StIjMz6zZ/QtXMLEEOdzOzBDnczcwS5HA3M0uQ\nw93MLEEOdzOzBDnczcwS5HA3M0uQw93MLEEOdzOzBDnczcwS5HA3M0uQw93MLEEOdzOzBDnczcwS\n5HA3M0uQw93MLEEOdzOzBDnczcwS5HA3M0uQw93MLEEOdzOzBDnczcwS5HA3M0tQ2XCX9G1JL0t6\nup12SfpHSU2SnpL0seqXaWZmnVHJkfsSYEIH7ROBEfllFnBr98syM7PuKBvuEfFj4JUOupwOfCcy\njwGHSjqsWgWamVnnVWPM/XBgc8F8c77MzMxqpG8V1qESy6JkR2kW2dANw4YNq8KmzXovlXrkWZdF\nydSqX9U4cm8GhhbMDwG2lOoYEYsjoiEiGgYPHlyFTZuZWSnVCPcVwOfzs2ZOAnZExItVWK+ZmXVR\n2WEZSXcCY4FBkpqBvwH2B4iI24BVwCSgCdgJ/NXeKtbMzCpTNtwjYkaZ9gAurFpFZmbWbf6EqplZ\nghzuZmYJcribmSXI4W5mliCHu5lZghzuZmYJcribmSXI4W5mliCHu5lZghzuZmYJcribmSXI4W5m\nliCHu5lZghzuZmYJcribmSXI4W5mliCHu5lZghzuZmYJcribmSXI4W5mliCHu5lZghzuZmYJcrib\nmSXI4W5mlqCKwl3SBEkbJTVJurRE+zBJD0v6haSnJE2qfqlmZlapsuEuqQ9wMzARGA3MkDS6qNt8\nYFlEHA9MB26pdqFmZla5So7cTwCaImJTROwC7gJOL+oTwMH59CHAluqVaGZmndW3gj6HA5sL5puB\nE4v6XAn8UNL/Ad4DjC+71o0bYezY1svOPBMuuAB27oRJJUZ2zj03u2zbBtOmtW2fPRvOOgs2b4az\nz27bPncuTJmSbfv889u2z58P48fDunUwZ07b9muugVNOgbVr4bLL2rbfdBOMGQMPPghXX922fdEi\nGDUK7rsPbrihbfvSpTB0KNx9N9x6a9v25cth0CBYsiS7FFu1Cvr3h1tugWXL2ravWZP9vf56WLmy\ndVu/frB6dTa9YAE89FDr9oED4Z57sul58+DRR1u3DxkCd9yRTc+Zk92HhUaOhMWLs+lZs+DZZ1u3\njxmT3X8AM2dCc3Pr9pNPhoULs+kzzoDt21u3jxsHl1+eTU+cCG+91bp98mT48pez6eL9Dupy33u4\noHkON/FLxjCOB5lP233vfBbxLKOYzH3Mpe2+dzZLaWYoZ3I3s2m7701jOdsZxDks4VyWtGmfxCre\noj+zuYUzabvvncaa7G7geibTet97i35MItv35rOAcbTe97YzkGlk+941zONkWu97zQzhbLJ970bm\nMIbW+96zjOR8sn1vEbMYSet9bx1juIQ63vfaUcmRu0osi6L5GcCSiBgCTAKWSmqzbkmzJDVKaty9\ne3fFRZqZWecoojinizpIJwNXRsRn8vl5ABGxsKDPemBCRGzO5zcBJ0XEy+2tt6GhIRobG7t/C8x6\nKZU67LIuKxOFPYakJyOioVy/SoZlngBGSDoS+A3ZG6afK+rza2AcsETSh4EDga2dK7ln8gOouurl\nAWRW78oOy0TEHuAi4AHgGbKzYtZLukrS1LzbXOCLkn4J3AmcG+VeEpiZ2V5TyZE7EbEKWFW07IqC\n6Q3AJ6pbmpmZdZU/oWpmliCHu5lZghzuZmYJcribmSXI4W5mliCHu5lZghzuZmYJcribmSXI4W5m\nliCHu5lZghzuZmYJcribmSXI4W5mliCHu5lZghzuZmYJcribmSXI4W5mliCHu5lZghzuZmYJcrib\nmSXI4W5mliCHu5lZghzuZmYJcribmSWoonCXNEHSRklNki5tp8+ZkjZIWi/pu9Ut08zMOqNvuQ6S\n+gA3A/8DaAaekLQiIjYU9BkBzAM+ERGvSnr/3irYzMzKq+TI/QSgKSI2RcQu4C7g9KI+XwRujohX\nASLi5eqWaWZmnVFJuB8ObC6Yb86XFRoJjJT0U0mPSZpQrQLNzKzzyg7LACqxLEqsZwQwFhgCPCLp\nmIh4rdWKpFnALIBhw4Z1ulgzM6tMJUfuzcDQgvkhwJYSff41InZHxPPARrKwbyUiFkdEQ0Q0DB48\nuKs1m5lZGZWE+xPACElHSjoAmA6sKOpzL3AagKRBZMM0m6pZqJmZVa5suEfEHuAi4AHgGWBZRKyX\ndJWkqXm3B4DtkjYADwNfiYjte6toMzPrmCKKh8/3jYaGhmhsbKzJtjtDpd5xsC6r0e6WJO+b1VUv\n+6akJyOioVw/f0LVzCxBDnczswQ53M3MEuRwNzNLkMPdzCxBDnczswQ53M3MEuRwNzNLkMPdzCxB\nDnczswQ53M3MEuRwNzNLkMPdzCxBDnczswQ53M3MEuRwNzNLkMPdzCxBDnczswQ53M3MEuRwNzNL\nkMPdzCxBDnczswQ53M3MEuRwNzNLUEXhLmmCpI2SmiRd2kG/aZJCUkP1SjQzs84qG+6S+gA3AxOB\n0cAMSaNL9BsAXAw8Xu0izcyscyo5cj8BaIqITRGxC7gLOL1EvwXAtcDbVazPzMy6oJJwPxzYXDDf\nnC97l6TjgaERsbKKtZmZWRdVEu4qsSzebZT2A24E5pZdkTRLUqOkxq1bt1ZepZmZdUol4d4MDC2Y\nHwJsKZgfABwDrJH0AnASsKLUm6oRsTgiGiKiYfDgwV2v2szMOlRJuD8BjJB0pKQDgOnAipbGiNgR\nEYMiYnhEDAceA6ZGRONeqdjMzMoqG+4RsQe4CHgAeAZYFhHrJV0laereLtDMzDqvbyWdImIVsKpo\n2RXt9B3b/bLMzKw7/AlVM7MEOdzNzBLkcDczS5DD3cwsQQ53M7MEOdzNzBLkcDczS5DD3cwsQQ53\nM7MEOdzNzBLkcDczS5DD3cwsQQ53M7MEOdzNzBLkcDczS5DD3cwsQQ53M7MEOdzNzBLkcDczS5DD\n3cwsQQ53M7MEOdzNzBLkcDczS5DD3cwsQQ53M7MEVRTukiZI2iipSdKlJdq/JGmDpKckPSTpiOqX\namZmlSob7pL6ADcDE4HRwAxJo4u6/QJoiIhjgeXAtdUu1MzMKlfJkfsJQFNEbIqIXcBdwOmFHSLi\n4YjYmc8+BgypbplmZtYZlYT74cDmgvnmfFl7zgNWl2qQNEtSo6TGrVu3Vl6lmZl1SiXhrhLLomRH\naSbQAFxXqj0iFkdEQ0Q0DB48uPIqzcysU/pW0KcZGFowPwTYUtxJ0njga8CpEfG76pRnZmZdUcmR\n+xPACElHSjoAmA6sKOwg6XhgETA1Il6ufplmZtYZZcM9IvYAFwEPAM8AyyJivaSrJE3Nu10HHAR8\nT9I6SSvaWZ2Zme0DlQzLEBGrgFVFy64omB5f5brMzKwb/AlVM7MEOdzNzBLkcDczS5DD3cwsQQ53\nM7MEOdzNzBLkcDczS5DD3cwsQQ53M7MEOdzNzBLkcDczS5DD3cwsQQ53M7MEOdzNzBLkcDczS5DD\n3cwsQQ53M7MEOdzNzBLkcDczS5DD3cwsQQ53M7MEOdzNzBLkcDczS5DD3cwsQRWFu6QJkjZKapJ0\naYn2P5J0d97+uKTh1S7UzMwqVzbcJfUBbgYmAqOBGZJGF3U7D3g1Io4GbgT+rtqFmplZ5So5cj8B\naIqITRGxC7gLOL2oz+nA7fn0cmCcJFWvTDMz64y+FfQ5HNhcMN8MnNhen4jYI2kHMBDYVthJ0ixg\nVj77pqSNXSnaShpE0f3dE/kpv1fyvlldR1TSqZJwL3WTowt9iIjFwOIKtmmdJKkxIhpqXYdZMe+b\ntVHJsEwzMLRgfgiwpb0+kvoChwCvVKNAMzPrvErC/QlghKQjJR0ATAdWFPVZAZyTT08DfhQRbY7c\nzcxs3yg7LJOPoV8EPAD0Ab4dEeslXQU0RsQK4FvAUklNZEfs0/dm0VaSh7usp/K+WQPyAbaZWXr8\nCVUzswQ53M3MEuRwNzNLkMPdzPYKSf0kjap1Hb2Vw71OSRop6SFJT+fzx0qaX+u6zAAkTQHWAffn\n82MkFZ9CbXuRw71+/TMwD9gNEBFP4VNQree4kux7qV4DiIh1wPAa1tPrONzrV/+I+FnRsj01qcSs\nrT0RsaPWRfRmlXy3jPVM2yT9Cfl3+EiaBrxY25LM3vW0pM8BfSSNAC4G1ta4pl7FH2KqU5KOIvvk\n3ynAq8DzwMyIeKGWdZkBSOoPfA34n2RfLPgAsCAi3q5pYb2Iw73OSXoPsF9EvFHrWsys53C41xlJ\nX+qoPSL+fl/VYlZM0n2U+LrvFhExdR+W06t5zL3+DKh1AWYduL7WBVjGR+5mZgnykXudknQg2Q+T\nfwQ4sGV5RPx1zYoyy+VnyCwERtN6/zyqZkX1Mj7PvX4tBT4AfAb4d7JfyPKbqtZT/AtwK9lnL04D\nvkO2z9o+4mGZOiXpFxFxvKSnIuJYSfsDD0TEp2tdm5mkJyPi45L+IyI+mi97JCI+VevaegsPy9Sv\n3fnf1yQdA7yEP95tPcfbkvYDnst/ye03wPtrXFOv4mGZ+rVY0nuBy8l+w3YDcG1tSzJ71xygP9kn\nUz8OzAQ+X9OKehkPy5hZ1UlqIPuE6hHA/vniiIhja1dV7+Jwr1OSDiU7EhpOwfBaRFxcq5rMWkja\nCHwF+A/g9y3LI+JXNSuql/GYe/1aBTxG0YPHrIfYGhH+/vYa8pF7nZL084j4WK3rMCtF0jhgBvAQ\n8LuW5RHx/ZoV1cs43OuUpEuAN4GVtH7wvFKzosxyku4APgSs5w+vLMMfstt3PCxTv3YB15G9adXy\nDB2APwFoPcFxLee3W2043OvXl4CjI2JbrQsxK+ExSaMjYkOtC+mtHO71az2ws9ZFmLXjk8A5kp4n\nGzYUPhVyn3K41693gHWSHqb1mLtPhbSeYEKtC+jtHO716978Ytbj+Hz22vPZMnVMUj9gWERsrHUt\nZtaz+Ltl6pSkKcA64P58fowkf2jEzACHez27EjgBeA0gItYBR9ayIDPrORzu9WtPROwoWuYxNjMD\n/IZqPXta0ueAPvlPml0MrK1xTWbWQ/jIvc5Iavmpsv8i+/3U3wF3Aq+TfYe2mZnPlqk3kjYAE8l+\noOO04nZ/t4yZgYdl6tFtZGfIHAU0FiwX/m4ZM8v5yL1OSbo1ImbXug4z65kc7mZmCfIbqmZmCXK4\nm5klyOFuZpYgh7uZWYIc7mZmCfr/mr/fEAzclMgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f06267d3630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = disp_frame['False Omissions Rate Disparity'].plot(kind='bar', color='b', title='False Omissions Rate Disparity')\n",
    "_ = ax.axhline(parity_threshold_low, color='r', linestyle='--')\n",
    "_ = ax.axhline(parity_threshold_hi, color='r', linestyle='--')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assess and report parity\n",
    "A binary indication of parity for metrics is reported by simply checking whether disparity values are within the user-defined thresholds. Further parity indicators are defined as combinations of other disparity values:\n",
    "\n",
    "* Type I Parity: Both FDR Parity and FPR Parity\n",
    "* Type II Parity: Both FOR Parity and FNR Parity\n",
    "* Equalized Odds: Both FPR Parity and TPR Parity\n",
    "* Supervised Fairness: Both Type I and Type II Parity\n",
    "* Overall Fairness: Fairness across all metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col0 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col1 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col2 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col3 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col4 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col5 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col6 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col7 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col8 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col9 {\n",
       "            color:  red;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col10 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col11 {\n",
       "            color:  red;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col12 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col13 {\n",
       "            color:  red;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col14 {\n",
       "            color:  red;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col0 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col1 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col2 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col3 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col4 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col5 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col6 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col7 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col8 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col9 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col10 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col11 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col12 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col13 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col14 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col0 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col1 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col2 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col3 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col4 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col5 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col6 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col7 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col8 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col9 {\n",
       "            color:  red;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col10 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col11 {\n",
       "            color:  red;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col12 {\n",
       "            color:  blue;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col13 {\n",
       "            color:  red;\n",
       "        }    #T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col14 {\n",
       "            color:  red;\n",
       "        }</style>  \n",
       "<table id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54e\" > \n",
       "<thead>    <tr> \n",
       "        <th class=\"blank level0\" ></th> \n",
       "        <th class=\"col_heading level0 col0\" >Prevalence Parity</th> \n",
       "        <th class=\"col_heading level0 col1\" >Accuracy Parity</th> \n",
       "        <th class=\"col_heading level0 col2\" >True Positive Rate Parity</th> \n",
       "        <th class=\"col_heading level0 col3\" >Precision Parity</th> \n",
       "        <th class=\"col_heading level0 col4\" >Specificity Parity</th> \n",
       "        <th class=\"col_heading level0 col5\" >Negative Predicted Value Parity</th> \n",
       "        <th class=\"col_heading level0 col6\" >False Positive Rate Parity</th> \n",
       "        <th class=\"col_heading level0 col7\" >False Discovery Rate Parity</th> \n",
       "        <th class=\"col_heading level0 col8\" >False Negative Rate Parity</th> \n",
       "        <th class=\"col_heading level0 col9\" >False Omissions Rate Parity</th> \n",
       "        <th class=\"col_heading level0 col10\" >Type I Parity</th> \n",
       "        <th class=\"col_heading level0 col11\" >Type II Parity</th> \n",
       "        <th class=\"col_heading level0 col12\" >Equalized Odds</th> \n",
       "        <th class=\"col_heading level0 col13\" >Supervised Fairness</th> \n",
       "        <th class=\"col_heading level0 col14\" >Overall Fairness</th> \n",
       "    </tr></thead> \n",
       "<tbody>    <tr> \n",
       "        <th id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54elevel0_row0\" class=\"row_heading level0 row0\" >female</th> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col0\" class=\"data row0 col0\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col1\" class=\"data row0 col1\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col2\" class=\"data row0 col2\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col3\" class=\"data row0 col3\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col4\" class=\"data row0 col4\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col5\" class=\"data row0 col5\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col6\" class=\"data row0 col6\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col7\" class=\"data row0 col7\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col8\" class=\"data row0 col8\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col9\" class=\"data row0 col9\" >False</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col10\" class=\"data row0 col10\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col11\" class=\"data row0 col11\" >False</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col12\" class=\"data row0 col12\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col13\" class=\"data row0 col13\" >False</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow0_col14\" class=\"data row0 col14\" >False</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54elevel0_row1\" class=\"row_heading level0 row1\" >male</th> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col0\" class=\"data row1 col0\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col1\" class=\"data row1 col1\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col2\" class=\"data row1 col2\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col3\" class=\"data row1 col3\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col4\" class=\"data row1 col4\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col5\" class=\"data row1 col5\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col6\" class=\"data row1 col6\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col7\" class=\"data row1 col7\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col8\" class=\"data row1 col8\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col9\" class=\"data row1 col9\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col10\" class=\"data row1 col10\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col11\" class=\"data row1 col11\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col12\" class=\"data row1 col12\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col13\" class=\"data row1 col13\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow1_col14\" class=\"data row1 col14\" >True</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54elevel0_row2\" class=\"row_heading level0 row2\" >all</th> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col0\" class=\"data row2 col0\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col1\" class=\"data row2 col1\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col2\" class=\"data row2 col2\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col3\" class=\"data row2 col3\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col4\" class=\"data row2 col4\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col5\" class=\"data row2 col5\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col6\" class=\"data row2 col6\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col7\" class=\"data row2 col7\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col8\" class=\"data row2 col8\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col9\" class=\"data row2 col9\" >False</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col10\" class=\"data row2 col10\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col11\" class=\"data row2 col11\" >False</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col12\" class=\"data row2 col12\" >True</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col13\" class=\"data row2 col13\" >False</td> \n",
       "        <td id=\"T_5c18ff33_5199_11ea_bb38_28b2bdeda54erow2_col14\" class=\"data row2 col14\" >False</td> \n",
       "    </tr></tbody> \n",
       "</table> "
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f06705af5c0>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# parity checks\n",
    "# low_threshold (0.8) < *_metric/white_metric < (high_threshold) 1.25 => parity, else disparity \n",
    "\n",
    "# init frame for parity\n",
    "par_frame = pd.DataFrame(index=sex_levels, \n",
    "                         columns=[col + ' Parity' for col in metrics_frame.columns])\n",
    "# nested loop through: \n",
    "# - races\n",
    "# - disparity metrics\n",
    "for i, _ in enumerate(sex_levels):\n",
    "    for j, _ in enumerate(par_frame.columns):\n",
    "        par_frame.iat[i, j] = (parity_threshold_low < disp_frame.iat[i, j] < parity_threshold_hi)\n",
    "\n",
    "# add overall parity checks\n",
    "# Type I Parity: Fairness in both FDR Parity and FPR Parity\n",
    "# Type II Parity: Fairness in both FOR Parity and FNR Parity\n",
    "# Equalized Odds: Fairness in both FPR Parity and TPR Parity\n",
    "# Supervised Fairness: Fairness in both Type I and Type II Parity\n",
    "# Overall Fairness: Fairness across all parities for all metrics\n",
    "par_frame['Type I Parity'] = (par_frame['False Discovery Rate Parity']) & (par_frame['False Positive Rate Parity'])\n",
    "par_frame['Type II Parity'] = (par_frame['False Omissions Rate Parity']) & (par_frame['False Negative Rate Parity'])\n",
    "par_frame['Equalized Odds'] = (par_frame['False Positive Rate Parity']) & (par_frame['True Positive Rate Parity'])\n",
    "par_frame['Supervised Fairness'] = (par_frame['Type I Parity']) & (par_frame['Type II Parity'])\n",
    "par_frame['Overall Fairness'] = par_frame.all(axis='columns')\n",
    "par_frame.loc['all', :] = par_frame.all(axis='index')\n",
    "    \n",
    "# small utility function to format pandas table output    \n",
    "def color_false_red(val):\n",
    "\n",
    "    color = 'red' if not val else 'blue'\n",
    "    return 'color: %s' % color \n",
    "    \n",
    "par_frame.style.applymap(color_false_red)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is still suffering from a minor disparity problem due to it's propensity to make false negative predictions for males. To address such bias, users could tune the GBM cutoff or regularization, could try new methods for reweighing data prior to model training, try new modeling methods specifically designed for fairness, or post-process the decisions. Checkout the IBM AIF360 package for some of these techniques: http://aif360.mybluemix.net/. Before attempting remediation here, more traditional fair lending measures will be assessed and, local, or individual, fairness will also be investigated. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Traditional Fair Lending Measures\n",
    "\n",
    "Along with adverse impact ratio (AIR), several measures have long-standing legal precedence in fair lending, including marginal effect and standardized mean difference. These measures are calculated and discussed here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Simple function to calculate adverse impact ratio (AIR)\n",
    "AIR is perhaps the most well-known discrimination measure. It was first delineated by the U.S. Equal Employment Opportunity Commission (EEOC) and AIR is associated with the convenient 4/5ths, or 0.8, cutoff threshold. AIR values below 0.8 can be considered evidence of illegal discrimination in many lending or employment scenarios in the U.S."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Male proportion accepted: 0.770\n",
      "Female proportion accepted: 0.796\n",
      "Adverse impact ratio: 1.034\n"
     ]
    }
   ],
   "source": [
    "def air(reference, protected):\n",
    "\n",
    "    \"\"\" Calculates the adverse impact ratio as a quotient between protected and \n",
    "        reference group acceptance rates: protected_prop/reference_prop. \n",
    "        Prints intermediate values. Tightly coupled to cm_dict. \n",
    "    \n",
    "    Args:\n",
    "        reference: name of reference group in cm_dict as a string.\n",
    "        protected: name of protected group in cm_dict as a string.\n",
    "\n",
    "    Returns:\n",
    "        AIR as a formatted string. \n",
    "        \n",
    "    \"\"\"\n",
    "\n",
    "    # reference group summary\n",
    "    reference_accepted = float(cm_dict[reference].iat[1,0] + cm_dict[reference].iat[1,1]) # predicted 0's\n",
    "    reference_total = float(cm_dict[reference].sum().sum())\n",
    "    reference_prop = reference_accepted/reference_total\n",
    "    print(reference.title() + ' proportion accepted: %.3f' % reference_prop)\n",
    "    \n",
    "    # protected group summary\n",
    "    protected_accepted = float(cm_dict[protected].iat[1,0] + cm_dict[protected].iat[1,1]) # predicted 0's\n",
    "    protected_total = float(cm_dict[protected].sum().sum())\n",
    "    protected_prop = protected_accepted/protected_total\n",
    "    print(protected.title() + ' proportion accepted: %.3f' % protected_prop)\n",
    "\n",
    "    # return adverse impact ratio\n",
    "    return 'Adverse impact ratio: %.3f' % (protected_prop/reference_prop)\n",
    "\n",
    "print(air('male', 'female'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Typical desirable ranges of AIR are above the 0.8 marker set by the 4/5ths rule. Here we see an almost ideal result where the protected and reference groups have very similar acceptance rates and AIR is near 1. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Simple function to calculate marginal effect\n",
    "Marginal effect describes the difference between the percent of the reference group awarded a loan and the percent of the protected group awarded a loan under a model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Male accepted: 77.05%\n",
      "Female accepted: 79.64%\n",
      "Marginal effect: -2.60%\n"
     ]
    }
   ],
   "source": [
    "def marginal_effect(reference, protected):\n",
    "\n",
    "    \"\"\" Calculates the marginal effect as a percentage difference between a reference and \n",
    "        a protected group: reference_percent - protected_percent. Prints intermediate values. \n",
    "        Tightly coupled to cm_dict. \n",
    "    \n",
    "    Args:\n",
    "        reference: name of reference group in cm_dict as a string.\n",
    "        protected: name of protected group in cm_dict as a string.\n",
    "\n",
    "    Returns:\n",
    "        Marginal effect as a formatted string. \n",
    "        \n",
    "    \"\"\"\n",
    "    \n",
    "    # reference group summary\n",
    "    reference_accepted = float(cm_dict[reference].iat[1,0] + cm_dict[reference].iat[1,1]) # predicted 0's\n",
    "    reference_total = float(cm_dict[reference].sum().sum())\n",
    "    reference_percent = 100*(reference_accepted/reference_total)\n",
    "    print(reference.title() + ' accepted: %.2f%%' % reference_percent)\n",
    "    \n",
    "    # protected group summary\n",
    "    protected_accepted = float(cm_dict[protected].iat[1,0] + cm_dict[protected].iat[1,1]) # predicted 0's\n",
    "    protected_total = float(cm_dict[protected].sum().sum())\n",
    "    protected_percent = 100*(protected_accepted/protected_total)\n",
    "    print(protected.title() + ' accepted: %.2f%%' % protected_percent)\n",
    "    \n",
    "    # return marginal effect\n",
    "    return 'Marginal effect: %.2f%%' % (reference_percent - protected_percent)\n",
    "\n",
    "print(marginal_effect('male', 'female'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "About 77% of men are awarded a loan by the model. About 79.6% of women are awarded a loan. This results in a marginal effect of -2.60%. Given that the marginal effect is negative, indicating that a higher percentage of individuals in the protected group were awarded a loan than in the reference group, this value would likely not indicate a discrimination problem in most scenarios. The magnitude of the marginal effect is also relatively small, another sign that discrimination concerning SEX is low under the model. Generally, larger marginal effects may be tolerated in newer credit products, whereas smaller marginal effects are expected in established credit products."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Standardized mean difference\n",
    "The standardized mean difference (SMD), i.e. Cohen's D, is the mean value of the prediction for the protected group minus the mean prediction for the reference group, all divided by the standard deviation of the prediction. Like AIR, SMD has some prescribed thresholds: 0.2, 0.5, and 0.8 for small, medium, and large differences, respectively. The standardized mean difference can also be used on continuous values like credit limits or loan amounts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Male mean yhat: 0.22\n",
      "Female mean yhat: 0.21\n",
      "P_Default_Next_Month std. dev.:  0.18\n",
      "Standardized Mean Difference: -0.08\n"
     ]
    }
   ],
   "source": [
    "def smd(frame, yhat, j, reference, protected):\n",
    "    \n",
    "    \"\"\" Calculates standardized mean difference between a protected and reference group:\n",
    "    (mean(yhat | j=protected) - mean(yhat | j=reference))/sigma(yhat). \n",
    "    Prints intermediate values. \n",
    "    \n",
    "    Args:\n",
    "        frame: Pandas dataframe containing j and predicted (yhat) values.\n",
    "        yhat: Name of predicted value column.\n",
    "        j: name of demographic column containing reference and protected group labels. \n",
    "        reference: name of reference group in j.\n",
    "        protected: name of protected group in j. \n",
    "\n",
    "    Returns:\n",
    "       Standardized mean difference as a formatted string.\n",
    "       \n",
    "    \"\"\"\n",
    "    \n",
    "    # yhat mean for j=reference\n",
    "    reference_yhat_mean = frame[frame[j] == reference][yhat].mean()\n",
    "    print(reference.title() + ' mean yhat: %.2f' % reference_yhat_mean)\n",
    "    \n",
    "    #yhat mean for j=protected \n",
    "    protected_yhat_mean = frame[frame[j] == protected][yhat].mean()\n",
    "    print(protected.title() + ' mean yhat: %.2f' % protected_yhat_mean)\n",
    "    \n",
    "    # std for yhat\n",
    "    sigma = frame[yhat].std()\n",
    "    print(yhat.title() + ' std. dev.:  %.2f' % sigma)\n",
    "    \n",
    "    return 'Standardized Mean Difference: %.2f' % ((protected_yhat_mean - reference_yhat_mean)/sigma)\n",
    "    \n",
    "print(smd(test_yhat, 'p_DEFAULT_NEXT_MONTH', 'SEX', 'male', 'female'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this model, in the test set, men receive a higher average probability of default than do women. This difference is evident even after standardizing with the standard deviation of the predictions. However, the difference is quite small, below the 0.2 threshold for a small difference. SMD also points to low disparity between men and women under this model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Investigate Individual Disparity "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In nonlinear models, similar people can be treated differenly by the model, so even if the model is mostly fair for most kinds of people, there could still be people the model treated unfairly. This could occur for multiple reasons, including the functional form of the learned model or because different variables are combined by the model to represent strong signals. If a variable is important in a dataset, model, or problem domain it's likely that a nonlinear model will find combinations of other variables to act as proxies for the problematic variable -- potentially even different combinations for different rows of data! So by simply testing for group fairness, you may miss instances of individual discrimination."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Augment predictions with decisions and logloss residuals for women with false positive predictions\n",
    "In this notebook, residuals for false positive predictions for women will be examined in an attempt to locate any individual instances of model discrimination. These are women who the model said would default, but they did not default. So they may have experienced some discrimination under the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_yhat_female = test_yhat[test_yhat['SEX'] == 'female'].copy(deep=True)\n",
    "\n",
    "\n",
    "test_yhat_female['d_DEFAULT_NEXT_MONTH'] = 0\n",
    "test_yhat_female.loc[test_yhat_female[yhat] > best_cut, 'd_DEFAULT_NEXT_MONTH'] = 1\n",
    "\n",
    "test_yhat_female['r_DEFAULT_NEXT_MONTH'] = -test_yhat_female[y]*np.log(test_yhat_female[yhat]) -\\\n",
    "                       (1 - test_yhat_female[y])*np.log(1 - test_yhat_female[yhat]) \n",
    "    \n",
    "test_yhat_female_fp = test_yhat_female[(test_yhat_female[y] == 0) &\\\n",
    "                                       (test_yhat_female['d_DEFAULT_NEXT_MONTH'] == 1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot logloss redisuals\n",
    "There are clear high-magnitude outliers in the logloss residuals for women who were issued false positive predictions. Given that those people were the people the model was most wrong about, those predictions will be checked first, just in case something very strange happened."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfYAAAHxCAYAAACf7p4EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3XmcZHdd7//X55xau3rvnqUnM5OZ\nyT4JTAIJ2RQDMQJBJYIgUbwGlIgrisi98kCvP/np9aLARQE1Aj9+LjcqIMvFoMRADBBJMglpkgxk\nYWbS05NKz/R0V6+1n8/9o6pJZzJLzaSrq7v6/Xw8+jFV55w69enwePDu7/d8F3N3REREpD0ErS5A\nRERElo6CXUREpI0o2EVERNqIgl1ERKSNKNhFRETaiIJdRESkjSjYRVYZM7vJzL7e6jqeDzPbamaz\nZhYe5/zvm9nfLdF3uZmdvRT3ElkNFOwiTWRm+83sh1tdx+kys231YJyt/+w3s//2fO/r7iPu3unu\n1aWoU0SeEWt1ASKyKvS6e8XMLgX+w8zud/fbW12UiDyXWuwiLWJmbzWzJ8xswsy+YGabFp37ETN7\n1MymzOyjZvYfZvYLx7nPVWZ2X/3a+8zsqkXnbjKzvWY2Y2b7zOxn6sfPrt9zyszGzewfG6nZ3XcD\njwAXL/qOTWb2GTM7XP+OX1907iVmttvMps1szMw+UD++0BMQq7/fXq9nxsxuBwYX3eMaMxs96nf+\nfk9I/Tv+08xyZpY1sw+bWeI4/62uN7M99e85aGbvbOT3FllNFOwiLWBmLwf+B/AGYAh4EviH+rlB\n4NPA7wADwKPAVce5Tz/wL8Cf1a/9APAvZjZgZpn68Ve5e1f9Hg/WP/pe4MtAH7AZ+PMG674CuAh4\nov4+AP4PMAycAVwL/IaZvaL+kQ8BH3L3buAs4J+Oc+v/DdxPLdDfC/xcI/XUVYHfrH/2ynoNv3yc\naz8O/GL9v8dFwFdO4XtEVgUFu0hr/AzwCXd/wN2L1EL8SjPbBlwPPOLu/+zuFWrh/PRx7vNq4HF3\n/1t3r7j7rcB3gR+rn4+Ai8ws7e5Zd3+kfrwMnAlscveCu59sMN64meWB/wQ+CnyufvwyYJ27/4G7\nl9x9L/DXwBsXfc/ZZjbo7rPu/s2jb2xmW+v3+V13L7r7XdT+WGiIu9/v7t+s//77gb8Cfug4l5eB\nnWbW7e6T7v5Ao98jsloo2EVaYxO1VjoA7j4LHKHW6t0EHFh0zoHRo29wrPvUPQmc4e5zwE8BbwOy\nZvYvZnZ+/Zp3AQbca2aPmNlbTlLvINAJvBO4BojXj58JbKp3g+fMLAe8G9hQP//zwLnAd+uPCX70\nOL/DZL3exb9DQ8zsXDP7opk9bWbTwB+xqCv/KK+j9ofTk/Wu/ysb/R6R1ULBLtIaT1ELRQDq3eYD\nwEEgS617fOGcLX5/ovvUba3fB3f/N3e/jlp3/3eptaZx96fd/a3uvgn4ReCjJ5sS5u5Vd38/UOCZ\nru4DwD5371300+Xu19c/87i73wisB/4n8On677pYFug76vjWRa/ngI5F/z1CYN2i839R/93OqXf5\nv5vaHy3H+h3uc/fX1Ov5HMd/NCCyainYRZovbmapRT8xas+U32xmF5tZklor8556V/K/AC8wsxvq\n1/4KsPE4974NONfMftrMYmb2U8BO4ItmtsHMfrwemEVgltrzaMzs9Wa28MfCJOAL5xrwx8C7zCwF\n3AtMm9l/NbO0mYVmdpGZXVb/njeZ2Tp3j4Bc/fPP+h53fxLYDfw/ZpYwsx/gmUcJAI8BKTN7tZnF\ngfcAyUXnu4BpYLbeI/FLxyq6fu+fMbMedy/XP6PpdtJ2FOwizXcbkF/08/vufgfwu8BnqLVYz6L+\nXNrdx4HXA++j1j2/k1rwFY++sbsfAX4U+K36te8CfrR+j6B+/Clggtpz54WW9mXAPWY2C3wBeLu7\n72vw9/kXan8MvLU+D/3HqI2S3weMAx8DeurXvhJ4pP49HwLe6O6FY9zzp4HL63X+d+BvFv2OU/W6\nP0atJ2KOZz+aeGf98zPUeiRONML/Z4H99S77twFvavB3Flk1rPb4TkRWqvrI81HgZ9z9q62uR0RW\nNrXYRVYgM3uFmfXWu+kXnhk/Z0S5iMjRFOwiK9OVwPeodW3/GHCDu+dbW5KIrAbqihcREWkjarGL\niIi0EQW7iIhIG1mVu7sNDg76tm3bWl2GiIjIsrn//vvH3X3dya5blcG+bds2du/e3eoyRERElo2Z\nNbTUsrriRURE2khTg93MtpjZV83sO/WNJt5+jGuuqe8J/WD95/eaWZOIiEg7a3ZXfAX4LXd/wMy6\ngPvN7HZ333PUdV9z92Pt+iQiIiKnoKkt9vr+zw/UX88A36G2LaWIiIg0wbI9YzezbcAlwD3HOH2l\nmQ2b2ZfM7MLlqklERKTdLMuoeDPrpLaL1W+4+/RRpx8AznT3WTO7ntoeyecc4x43AzcDbN269ejT\nIiIiwjK02Ov7J38G+Ht3/+ejz7v7tLvP1l/fRm3v6sFjXHeLu1/q7peuW3fSaXwiIiJrUrNHxRvw\nceA77v6B41yzsX4dZvaSek1HmlmXiIhIu2p2V/zVwM8CD5nZg/Vj7wa2Arj7XwI/CfySmVWAPPBG\n1840IiIip6Wpwe7uX6e2j/SJrvkw8OFm1iEiIrJWaOU5ERGRNqJgFxERaSMKdhERkTaiYBcREWkj\nCnYREZE2sir3YxcREVmJsrk8w6M5JuZK9GcS7Nrcy1BvellrUItdRERkCWRzeW7fM0a+VGWwM0m+\nVOX2PWNkc/llrUPBLiIisgSGR3N0pWJ0peIEZnSl4nSlYgyP5pa1DgW7iIjIEpiYK5FJPvsJdyYZ\nY2KutKx1KNhFRESWQH8mwVyx8qxjc8UK/ZnEstahYBcREVkCuzb3MlOoMFMoE7kzUygzU6iwa3Pv\nstahYBcREVkCQ71prtu5gXQiZHy2SDoRct3ODcs+Kl7T3URERJbIUG962YP8aGqxi4iItBEFu4iI\nSBtRV7yIiMjzsBJWm1tMLXYREZHTtFJWm1tMwS4iInKaVspqc4sp2EVERE7TSlltbjEFu4iIyGla\nKavNLaZgFxEROU0rZbW5xRTsIiIip2mlrDa3mKa7iYiIPA8rYbW5xdRiFxERaSNqsYuIiJyilbYo\nzWJqsYuIiJyClbgozWIKdhERkVOwEhelWUzBLiIicgpW4qI0iynYRURETsFKXJRmMQW7iIjIKViJ\ni9IspmAXERE5BStxUZrFNN1NRETkFK20RWkWU4tdRESkjSjYRURE2oiCXUREpI0o2EVERNqIgl1E\nRKSNKNhFRETaiIJdRESkjSjYRURE2oiCXUREpI0o2EVERNqIgl1ERKSNKNhFRETaiIJdRESkjSjY\nRURE2oiCXUREpI0o2EVERNqIgl1ERKSNKNhFRETaiIJdRESkjSjYRURE2oiCXUREpI0o2EVERNqI\ngl1ERKSNKNhFRETaiIJdRESkjSjYRURE2oiCXUREpI0o2EVERNqIgl1ERKSNKNhFRETaiIJdRESk\njSjYRURE2oiCXUREpI0o2EVERNqIgl1ERKSNKNhFRETaiIJdRESkjSjYRURE2oiCXUREpI0o2EVE\nRNqIgl1ERKSNKNhFRETaiIJdRESkjSjYRURE2oiCXUREpI0o2EVERNqIgl1ERKSNKNhFRETaiIJd\nRESkjSjYRURE2oiCXUREpI0o2EVERNpIrJk3N7MtwN8AG4EIuMXdP3TUNQZ8CLgemAducvcHmlmX\niIi0v2wuz/Bojom5Ev2ZBLs29zLUm251WU3X7BZ7Bfgtd78AuAL4FTPbedQ1rwLOqf/cDPxFk2sS\nEZE2l83luX3PGPlSlcHOJPlSldv3jJHN5VtdWtM1NdjdPbvQ+nb3GeA7wBlHXfYa4G+85ptAr5kN\nNbMuERFpb8OjObpSMbpScQIzulJxulIxhkdzrS6t6ZbtGbuZbQMuAe456tQZwIFF70d5bviLiIg0\nbGKuRCb57KfNmWSMiblSiypaPssS7GbWCXwG+A13nz769DE+4se4x81mttvMdh8+fLgZZYqISJvo\nzySYK1aedWyuWKE/k2hRRcun6cFuZnFqof737v7Px7hkFNiy6P1m4KmjL3L3W9z9Une/dN26dc0p\nVkRE2sKuzb3MFCrMFMpE7swUyswUKuza3Nvq0pquqcFeH/H+ceA77v6B41z2BeC/WM0VwJS7Z5tZ\nl4iItLeh3jTX7dxAOhEyPlsknQi5bueGNTEqvqnT3YCrgZ8FHjKzB+vH3g1sBXD3vwRuozbV7Qlq\n093e3OSaRERkDRjqTa+JID9aU4Pd3b/OsZ+hL77GgV9pZh0iIiJrhVaeExERaSPN7ooXERFpurW6\nytyxqMUuIiKr2lpeZe5YFOwiIrKqreVV5o5FwS4iIqvaWl5l7lgU7CIisqqt5VXmjkXBLiIiq9pa\nXmXuWBTsIiKyqq3lVeaORdPdRERkVdIUt2NTsIuIyKozPDLJrfcdoBo5A5kEhXKVQ9PFNd1SX6Cu\neBERWVWyuTy33jdCGMD6riTlqvPEoTkqUbRmp7gtpha7iIisKnc+eognj8yTjAVMzpVY35UmHQ85\nNF0kFQ9bXV7LqcUuIiKrRjaX5+7vHSEdD4gFAeWqs3d8lko14kj9Wftap2AXEZFVY3g0x2BngvVd\nKUrVCDASMWPv+CxhYGt2itti6ooXEZFVY+/hWcqViO+Nz5IIA0oVpxw55UrEjZdtWfMD50DBLiIi\nq0Q2l2dkYp4wMC7Y2M3ByQLTxTLbBzq4ZGsvu7b2tbrEFUFd8SIisioMj+Y4b0M37kY8FnD+UBfn\nbugkCAKuOW9Dq8tbMdRiFxGRVWFirsQZfWkyyRj7j8wyXSjTnYrRnUqoC34RBbuIiKwKC5u99GcS\n9Gf6AZgplEknNMVtMXXFi4jIqqDNXhqjYBcRkVVBm700Rl3xIiKyagz1phXkJ6EWu4iISBtRsIuI\niLQRBbuIiEgbUbCLiIi0EQW7iIhIG1Gwi4iItBEFu4iISBtRsIuIiLQRBbuIiEgbUbCLiIi0EQW7\niIhIG1Gwi4iItBEFu4iISBtRsIuIiLQRBbuIiEgbUbCLiIi0EQW7iIhIG1Gwi4iItBEFu4iISBtR\nsIuIiLQRBbuIiEgbUbCLiIi0EQW7iIhIG1Gwi4iItJFYqwsQEZH2ls3lGR7NMTFXoj+TYNfmXoZ6\n060uq22pxS4iIk2TzeW5fc8Y+VKVwc4k+VKV2/eMkc3lW11a21KLXUREmmZ4NEc1ch4/NMNMoUJX\nKsa6zhTDozm12ptEwS4iIk2z7/AcIxPzdCRDulNxCpUqj43NUChXW11a21Kwi4hIU2Rzeb41Mkmu\nUGIgk2R9V4quVJxCuUouX2p1eW3rpMFuZp8F/Hjn3f21S1qRiIisegvP1uMxIxmGzBcj9hZm2dST\nJgiMnnS81SW2rUZa7B+u/2vAXwBva145IiLSDoZHc3SlYpw5kOHIXImZ+Qq5QpVcvszlO/rZpOfr\nTXPSYHf3OxZem9ns4vciIiJHy+by3PXYYQKDwIxiqcpQX4ptYQeHZ0vEgoBdm3tbXWbbOtVn7Mft\nkhcREVnogk/GAgzDAsCMciViOl+mP5Pgup0bNCK+iRp5xt696G1oZl3UuuUBcPfpZhQmIiKrz0IX\n/AVD3Tx4YIp0PKQ/E8cdXnBGr0J9GTTSYn+EWkt9Icz3LHrvwNbmlCYiIqvNxFyJwc4kgRkXb+lh\n//g804UqjivUl0kjwX6lu482vRIREVn1+jMJ5ooVulJx+jNJ+jNJZgpl0olQob5MGllS9gtNr0JE\nRNrCrs29zBQqzBTKRO7MFMrMFCoaLLeMGgl2O/klIiIiMNSb5rqdG0gnQsZni6QTobrgl1kjXfFn\nmNkHjnfS3d+xhPWIiMgqc6zd21550VCry1qzGgn2PLUBdCIiIs8yPDLJrfcdoBo5A5kEhXKVQ9NF\ntdJbqJFgP+LuH296JSIisqpkc3luvW+EMDAGMkkK5YgnDs1x9vqMdm9roUaCXVvwiIjIcyxsyTqQ\nSWJmpBMhAIemi6TiYYurW7saCfbXmNmm451096eWsB4REVkFhkcm+ft7niSby5OOh5y/sZsNPWlS\n8YBDM0Uu3qpR8K3SSLDfwbMXqKH+vh9YB+jPMhGRNWR4ZJI/u+NxxqeLOHB4pkguP8FLtveRioeE\ngWl6WwuddLqbu1/g7jvr/14A/Ai1sJ8F3tnsAkVEZGX51P0HmCqUGehMkozF6O6IU6pU+NZIrWv+\nxsu26Pl6CzW8CYyZ7QDeDfwg8EHgt9y91KzCRERkZXrs0Aw9qTipRIxELCCXLxMAFXfefu25CvUW\na2QTmAuoBfqLgD8B3ubulWYXJiIiK1MyDClHESkgnYiRTsSYKYS4o1BfARppsT8MHAA+D7wQeJ/Z\nM4/btUCNiMjactVZA/zrI2MYRkciZL5UZSpf4ZUXbmh1aUJjwX5z06sQEZFV44ZLNjMxX+aJsRmO\nzBfJxGO8ZHs/N1yyudWlCQ0E++LFacwsVT9WaGZRIiKycg31pnnL1dufs4ysuuFXhoYGz5nZW4Hf\noTbFDTM7AvxPd7+libWJiMgKNdSbVpCvUI0Mnvsd4Brgle7+WP3YucCHzGzA3f9Hc0sUEZFWOdYG\nLwr0la2RbVtvAm5YCHWA+uvXAW9uUl0iItJi2Vye2/eMkS9VGexMki9VuX3PGNlcvtWlyQk00hXv\n7v6c/xXdfd7MoibUJCIiK8Cdj46xb3yWSuR0JeNsG+ygKxXTBi8rXCPBnjWza9z9zsUHzeyHgKeb\nUpWIiLTU8Mgkn7n/IIl4QE8qRrni5PJlXri5m3xZe4OtZI0E+68DnzOzrwL3U1sn/jJqz91vaF5p\nIiLSCnc8kuV9X36Midki6WSMSneKuVLEpp40jz49w+U7BlpdopxAI2vFPwRcBNwLnA/srL9+Qf2c\niIi0iTseyfL/fum7HJktkowH5MtVnhyfI1+ucGgmz/hsSRu8rHANTXerP2PX1DYRkTaWzeX5xN37\nMaArFaMaGTGrQGBkcwXiAx1ct3NAz9dXuEamu01S635/zilqA+v6l7wqERFZdu/70h7u3TdBNap1\n56YT0JfpIAycyGFrfwfXnLe+1WXKSTTSYv8GtX3XPwP8A3CwqRWJiMiye/+XHuH/PPQ0UX2uUxWY\nKUFAnng8xkBnkhsv26rW+irQyDP2HwVeCUwCnwBuB34e6HJ3DY0UEVnlhkcm+cTdI+AQCw0zCINa\nt+xMyelKx3nXj5zLrq19rS5VGtDIAjW4+6S7/zVwHbVn7X9ELdxFRGQVGx6Z5H3/9l3y5YjQqIX6\noh08zeA9rzqfay8camGVcioaXSv+JcCNwMuAbwKvB+5sXlkiItJs2Vyej319H4emiwRAxSF0CAMj\nxIgFzkAmoVBfZU7aYjezJ6i10seotdI/ChwBXmBmLzzJZz9hZofM7OHjnL/GzKbM7MH6z++d+q8g\nIiKn485HxxidzJOOh/RnYrhDpepEUUTkEWDcdOXWVpcpp6iRFvvT1EbFvxq4ntpjlwUOvPQEn/0k\n8GHgb05wzdfqz/FFRGQZPfzUNH0dcUqViKp3YMxzZL5CKYKOeMBbrtrKW685t9VlyilqZD/2H2jk\nRmb2cnf/ylGfvcvMtp1eaSIi0izZXJ6RiXnmCxXylYhMIuSM/gzdHRUidz7whos1WG6VamjwXIP+\n9DQ/d6WZDZvZl8zswiWsR0REjmF4ZJIP3fE488UqM6UKidCYLVYJzBjoSvL6F29WqK9iSxnsdvJL\nnuMB4Ex33wX8OfC5497c7GYz221muw8fPny6NYqIrGnZXJ5b7xshDOCiTd10JuLkS1XScaMrHefq\nswd5zSWbW12mPA9LGezHWp3uxB9wn3b32frr24C4mQ0e59pb3P1Sd7903bp1z7NUEZG1aXg0RzVy\netMJutIJXri5hy19GTpTCTb1pvjJF23WIjSrXEPT3ZrFzDYCY+7u9Sl1AbUR9yIissSyuTx3PXaY\np6cKTMyV2NLXQWcqznlDMQ7NFPnBc9Yp1NtAI2vFX+bu9zVwrwPH+Oyt1LZ3HTSzUeC/A3EAd/9L\n4CeBXzKzCpAH3ujup9zyFxGRE8vm8ty+Z4xkLGCoN82BiXkeHZvhnA2dVCMnDEy7trUJO1mOmtkD\n7v6iZaqnIZdeeqnv3r271WWIiKwa//pwlnypSrka8eCBKaLIOTSTp1CJ2NrfwY2XbdWAuRXOzO53\n90tPdl1Lu+JFRGR5TMyVGOxMEphx8ZYe9o/PEwsNx/nVl52jLvg20kiw7zCzfz7eSXd/7RLWIyIi\nTdCfSTBXrNCVitOfSdKfSTJTKJNOhAr1NtNIsB8GPtLsQkREpHl2be7l9j1jAGSSMeaKFWYKFa7Y\nMdDiymSpNRLss+5+R9MrERGRphnqTXPdzg0Mj+YYny3Sn0lwxY4BtdbbUCPBPtL0KkREpOmGetMK\n8jWgkQVqvmJmz7nOzPrM7K+aUJOIiIicpkaCfRew28wuXzhgZjcD3wIeb1ZhIiIicuoa2d3tLWb2\ng8AtZvYAcAG17vmr3f1gswsUERGRxjU6j/1bwDeAH6u/f4dCXURk5RgemeS2h7OMTRfY0J3i+ouG\ntODMGnXSrngzeyMwDBwEzgJeB3zIzD5xvA1bRERk+QyPTHLLXfuYLVbY1JNmtljhlrv2MTwy2erS\npAUaecb+s8APu/sfunvJ3e8FLqfWir+3qdWJiMhJ3fZwlt5MjL6OJEEQ0NeRpDcT47aHs60uTVrg\npMHu7q92931HHYvc/c+BK5tWmYiINGRsukBPKv6sYz2pOGPThRZVJK3USFf8+xe9/tWjTv/Rklck\nIiKnZEN3iqlC+VnHpgplNnSnWlSRtFIjXfEvW/T6LUedu2QJaxERkdNw/UVD5OYqTM4XiaKIyfki\nubkK11801OrSpAUaCXY7zmsREVkBdm3t4+aXbqczGeOpqTydyRg3v3S7RsWvUY1MdwvMrIvaHwEL\nrxcCPmxaZSIi8hzZXJ7h0RwTcyX6Mwl2be5lqDfNrq19CnIBGgv2AeARngnzPYvO+ZJXJCIix5TN\n5bl9zxhdqRiDnUnmihVu3zPGdTs3aA14+b5GVp7bvByFiIjIiQ2P5uhKxeiqj4Bf+Hd4NKdgl+87\nabCb2aYTnXf3p5auHBEROdpC9/u/PpxlqCfF9sEu+jMJoLa3+vhsscUVykrSSFf8HdS63BcPnHOg\nH1iHnrOLiDTN4u73jd1ppgtlHjyQ4+ItvfRnEswVK98PeRFobIGaC9x9Z/3fC4AfoRb2s8A7m12g\niMhatrj7fce6DO6GmbNvfIaZQpmZQoVdm3tbXaasII1MdwPAzHaY2ceAf6c2mG6nu3+waZWJiAgT\ncyUyyVrnan8mycVbeuhKxslOFUgnQg2ck+do5Bn7BcC7gRcBfwK8zd0rzS5MRET4fnf7wkC5/kyS\neBhw8dZeXqkFaOQYGnnG/jBwAPg88ELgfWbPPG5393c0pzQRkbVp8Vx1wxmfK7O5tzZQbq5YYaZQ\n4YodA60uU1aoRoL95qZXISIiwLHnqhOVKJSr5MtV+jMJrtgxoO53Oa5G5rF/vJEbmdn/cvffeP4l\niYisXceaq765v4N0IlTXuzSk4cFzDXjpEt5LRGRNWjxYbkEmGWNirtSiimS1WcpgFxGR52lhsNxi\nmqsup0LBLiKyguza3MtMocJMoUzkrrnqcsqWMti1pauIyPM01Jvmup0bSCdCxmeLmqsup6yReezv\ndfffbeBeH16CekRE1ryh3rSCXE5bI9PdXg2cNNgbHT0vIrLWHW9PdZGl0EhXfGhmXWbWfayfplco\nItJGFuap50tVBjuT5EtVbt8zRjaXb3Vp0iYaabGfT21t+KN3d7P6v1ubUJeISFvSnurSbI0E+x53\nv6TplYiIrAETcyUGO5PPOqY91WUpNRLsIiLyPC08V3/44BTJWMAFQ930Z2oBr3nqspQaCfbjjnY3\nsz91d+3JLiJyAsMjk9x63wjVyEnFAp6eKjBdqHDZtj6SsVCbusiSOunguZOMdn/DEtYiItJ2srk8\nt953gDAw1nelSMRDkvGQ0OChg1Oapy5L7vl2xWtRGhGRExgezVGNnIFMEjMjHY8x0AmJ0NjSn9HG\nLrLkGlmgpv94p1Cwi4gc08Iz9X99OEu+VCU0Y7Cr9kw9FQs5NFNg15a+Flcp7aiRFvv9PDO97Wja\nbkhE5CiL91Tf2J3m6ek8TxyaAWCgM0EuXyIMTOu/S1M0sh/79uUoRESkXSyeq75jXYbpQoXN/TA5\nX6TqThgYN162Vc/VpSka6Yp/0VGHHBh39wPNKUlEZHVbPFe9P5Pk4i097D08x9PTea69YL2WkJWm\naqQr/v3HONZvZgngRnd/cIlrEhFZ1Rb2VF9YVa4/kyQeBly8tVeD5aTpGumKf9mxjpvZpcCfAS9d\n6qJERFaTozd1GepOMTw6BdRWlZsrVjRXXZbNaU93c/fdZta5lMWIiKw22VyeT+8+wMR8iXLViYdG\nf0eCl567jux0gfHZIv2ZBFfsGFD3uyyL0w52M9tA7Xm7iMiadeejY+yfmKc3naAnHVAoR+yfmGcw\nO8WNl29rdXmyBjUyeO7PeW6A9wNXAW9vRlEiIivdQvf7Zx88SGc8TmcyhllIOhHiHufhp6ZbXaKs\nUY202Hcf9d6BI8A73P3Q0pckIrKyLZ6nngxDKh6xb3yO7YOZ2oA5c1z9mdIijQye+/+PddzMtpjZ\nb7v7nyx9WSIiK9fieerbBzM8fmiWRGiMTReIhUZuvrbBi0grnHQTmMXMbNDMfsnM7gLuBDY0pSoR\nkRVsYq5EJllrF110Ri8bulOEAUzMF4kiZ/u6DNect77FVcpa1cgz9i7gJ4CfBs4FPgvscPfNTa5N\nRGRFWjxPvT+T4KqzBvlOdopSNeLyHQNagEZaqpFn7IeAe4H3AF93dzezn2huWSIiK9euzb3cvmcM\nqM1Tj4fG9sFObb8qK0IjXfHvBlLAXwC/Y2ZnNbckEZGVbag3zXU7N5BOhIzPFrWnuqwojQye+yDw\nQTPbAdwIfA7YZGb/Ffisuz8t5JOrAAAgAElEQVTW5BpFRFri6BXlFnexD/WmFeSyIjU8eM7d97r7\nH7r7C4DLgB7gS02rTESkhRamtOVLVQY7k+RLVW7fM0Y2l291aSIndNJgN7Mrjj7m7g+5+7vdXd3y\nItKWFk9pC8zoSsXpSsUYHs21ujSRE2pk8NxHgRcBmNl/uvuVzS1JRKR17ngkyz/uHmVPdoqBTJKX\nnjPIrq39QG2g3PhsscUVipxYI8Fui16nmlWIiEgrZXN5PvKVR/niQ2Mkw4B4zMjly3x+OAvArq39\nzBUr9GcSLa5U5MQaecYemFmfmQ0set2/8NPsAkVEmi2by/OJr+3li98eI3IncqdQqpIvVzGDux4f\nZ6ZQZqZQYdfm3laXK3JCjbTYe4D7eabl/sCicw7sWOqiRESWSzaX58NfeYxvfO8I+UqVzkRIEASU\no4gYjgFH5mpT2rT1qqwGjUx327YMdYiILLuFke97x+dJxQISgZEvV+kMAmJmVCInCI2dG3p45UVD\nrS5XpCGNjIp/06LXVx917lebUZSIyHK489Ex9o3PMj5TZL5UpSsVI8KYK1WIPKJYiSiVI37qUq2g\nLatHI13x7wD+rv76z6mPkK97C/DhpS5KRKSZPnXvfj7xjf3sP5Inkwjp7wgBqLjRn44xlS8zV4ro\nTsd5+8vP4toL1VqX1eNUR8XbCc6JiKx4n7p3Px/49+/hRGQSAaUoYiRXYag7SU8yZLZU5Yz+Ds7f\n2M3PX72dXVu1/aqsLo0Eux/n9bHei4isaH93zwEyqZBSxUjFQmaLVQBy+SqXbO3k8GyJn37JVq45\nb70Gysmq1Eiwn29m36bWOj+r/pr6e42IF5FVYWHhmUey06RjAR3JkI5Egs6kMV9yZktVzhzIcN3O\njdx4+ZmtLlfktDUS7Bc0vQoRkSa645EsH/j3J+jpiNERCyhFTm6+QiWC7lSCIDAGMgm2D3ZyzXnr\nW12uyPPSyHS3JwHMrBc4p374MXefamZhIiLPVzaX53PfOsAtd+2jHDl9hTgDmQRPTRcxg0KpQjw0\n5ksRr3nhem29Km3hpMFuZgngFuAGYB+1LvgzzeyzwNvcvdTcEkVETt0dj2R535cfZXQyz3wpIhnA\nVB5SsYBN3UmOzJWYr0Sc3ZXiTZdv4fUv2dbqkkWWRCNd8e8B4sAWd58BMLMu4CPA79Z/RERWjOGR\nSd73b48yMjGPU1uwoxQBlSphYBAGXLSll85EjFt+7rIWVyuytBpZK/61wFsXQh2g/vqXgZ9oVmEi\nIqcjm8vz/tsfZTQ3TzWCeGAkYwERUK5AqVxlJl9mar6ihWekLTXSYo/cff7og+4+a2aa7iYiK8bw\nyCS33neAPU9N4w6YU6pCKh6ScqdUdcpVZzAR8o4fPlsLz0hbamgeu5n1cezFaKIlrkdE5LR86t79\nfPCOJyiUI+aKZYza/2lV3SlWqoRB7ZniORu7+cPXXKSFZ6Rtnc7uboupxS4iLfepe/fzx//2GIVy\nlUwiJBYY+bITDyBmUI2cKrCuM8Fvvvxshbq0tSXb3c3MLnT3R553RSIipyCby/Oxb+wnADqTMaoR\nJOMxIq8QRU48FtKXinHhpm5+7WXnKNSl7TXSYm/U3/LsDWJERJoqm8vzybv3MTZdJB4aAVAFAjM6\nEiGFSsT5G7t47YvO4JrzNEdd1oalDHZtCCMiy2ZhL/WJuRK96RjzxSqFqpOKGREwW4zoScd5z6t3\nqpUua8pSBruet4tI02VzeYZHc3zt8cMkwoB0POTs9V08eCBHPICKO3htJPxvXqvn6bL2NBTsZmbA\nZnc/0OR6RESOqzadbYRq5IxNFxjqSVOsRHQmY1y8pZcnDs2Qy1fY0J3kF67eptXkZE1qKNjd3c3s\nc8CLT3CZlpYVkaZZWE2uXI3ozySoRs7IxDxnDmRIhMamvjR9mQT9mQQ3XbVdz9NlzWpk5bkF3zSz\n46696O5XLEE9IiLPkc3lufW+EcpRlYFMgmoE5WpEsRJxeLpA5M4567t4wRm9CnVZ807lGfvLgF80\nsyeBOWqD5dzdX9iUykRkzRsemeRT9x/gm3snmC6U6U6GJIKQno4EPekkpUqF+XKFyCGdCLlix4BC\nXda8Uwn2VzWtChGRRYZHJvnk3fv4xt4jGJAIjGQYcGSuzFx5hrMGu+hKxZgqR+xYl+HXXn6OAl2k\nruFgX9iX/VSY2SeAHwUOuftFxzhvwIeA64F54CZ3f+BUv0dE2sfwyCS33LWPvYdnSAQBjjNTrNKb\njpOMB0QRzJYqlKIq8TDkxsu2KtRFFjmVZ+yn45PAK09w/lXAOfWfm4G/aHI9IrKCZXN53v/l7/Ld\nsWkO5vLMFcvEgoCOZMhMsUJvRwIzIzBjx2An73rFeZrOJnKUpga7u98FTJzgktcAf+M13wR6zUzb\nLYmsQdlcnk/vPsDe8Xky8YBELKBYiZjKl4mZEQuMEOhOx7hsWx9vv/ZchbrIMTS7xX4yZwCL58aP\n1o+JyBozPJpjYr5Ef0eCyGGwM0EYBJQqVabzZeJhQCwW8kNnr9PId5ETaHWwN7xjnJndbGa7zWz3\n4cOHm1yWiCy3ibkS5apz7oZOCpWIWBCwsTtBLDCKVWcgk+CVF67nLT+4Q6EucgJLuaTs6RgFtix6\nvxl46lgXuvstwC0Al156qZavFVnlhkcmue3hLGPTBTZ0p+hLx4mHRrIjycVbetl7eJZ82dg60Mmr\nX7CRX3rZOa0uWWRVaHWwfwH4VTP7B+ByYMrdsy2uSUSabGHke28mxqaeNFOFMo8/PctQT5JCtUJP\nKs7FW3vJzVfYvi7DDZdsbnXJIqtGU4PdzG4FrgEGzWwU+O9AHMDd/xK4jdpUtyeoTXd7czPrEZGV\n4baHs/RmYvR1JAG+/y/uXHpmLw8/NY07XLatj2vOW6+ud5FT0NRgd/cbT3LegV9pZg0isvKMTRfY\n1PPssO5JxXlqKs+Nl29rTVEibaLVXfEisgYsbLU6MVeiP5MgkwiZKpSfaakDU4UyG7pTLaxSpD0o\n2EWkabK5PHc+eoi7v3eEwc4E523sIl+q0pmM8/jYLFBrqU8VyuTmKrzhxVtOckcRORkFu4g0xcKC\nMw+MTFKsRswXq0zMlbj67EHOH+qmryPOZL7MU1N5NnSneMOLt2jBGZEloGAXkaa489Ex9k/MU6xG\n9KTiVCI4NF3iodEpfvDcdXR3JHibprCJLDkFu4gsqYXn6Z/91kE6k3ESsYBK5CRiIZlkyJOT87yo\nWKE/k2h1qSJtScEuIkvmjkeyfOLuJ8mXq4xN5RnsrJKKx8lTpTsN7k65EjFTqHDFjoFWlyvSllq9\npKyItInhkUk+8h97cZwNXUk6kjFGcwWKlTLpREjkzsR8ibPWZbhu5wbNTRdpErXYReR5Weh6/9/3\nPMlUvkxPuoMgCNja10Gp4kzMlUnEQi7c1MPFW/v4yRdtVqiLNJGCXUROy9FT2abzZTrixlO5PJt7\nO0glYuwYTLN/Is/Z67u4fMcAuzb3KtRFmkzBLiKnbHhkko99fS+Pjs1gwMRciplChe50jFI14vBs\ngc19HRSrzqaeNL/28nMU6CLLRM/YReSUDI9M8gdf3MPw6BSzhSpmMD5bIDA4MluityNGpRoxPlug\nUI54y1VnKtRFlpFa7CLSsDseyfInX36MA5PzpOMh8dCYKUB3Ok4yBh3JkI54jMjhkq19XH/RkBad\nEVlmCnYROalsLs/nvzXK399zgFK1Sjw0qpFTqkQk4pAvVojiAf2ZJC87f4NGvYu0kIJdRI4rm8vz\nuW+Ncsd3DnEwl6dcqZJMhBTK4DixMMCjiELFMExT2URWAAW7iBzTHY9k+dBXnuB747PEzChXq8SC\nEMoRsaDWYg8N5ksR/Z1JXn7eem66ertCXaTFFOwi8hzDI5N88I4nODRVIGZGLDDmSxB5bbBcZzJG\nFDkR0JmK87pLzuA1l2h+ushKoGAXke8bHpnkn3aP8O/fPcTUfJl4aCRjIUFgdCRD8sUq7uAOmXSM\n7mSCX7lmB9deONTq0kWkTsEuIgD8/mcf5NbdBylXwYHAoBo5kUMiDEgEAZXQCcyIxY3Lt/Xzhku3\natS7yAqjYBdZ47K5PH/whW/zpT3jAMQMKg5Vr50vR04yDhV3wgDO2ZDhPa++UIEuskIp2EXWqGwu\nz9/evY8vPpTlwGQBAAMir61cFVEL91h9R7ZELOTCzb389ivOV6iLrGAKdpE1aHhkko989Ql275/A\nqXW9U//XADMIHapAKh6yqTfNj+/axA0aICey4inYRdaY4ZFJ3vflR3ni0AxVd8xqYb4Q7hFgXjsW\nD+CNl23VNDaRVURrxYusIdlcnlvvO0C5UiWgNiiuXIXkUX/iO7WA/4Gz+hTqIquMWuwia0Q2l+eT\nd+/jsbEZIneCICBmEfHIcA9IxiJKlVqod6cCfu7yrfzWqy5sddkicooU7CJt7J3/sJvPPzhGuf6+\nM25ceEYPBAEGlCuQjgVUo4iQgP6OOK+9eBNvukqtdJHVSsEu0qZe+2df5YGn5p91bLbsDI/kOH9T\nN2f0p5mYKTJdrJCMhfzA2QO8+eodGvEussop2EXa0O9/9sHnhPqCQgSFUpUN3WnKFefF2wa48bIt\nCnSRNqFgF2kzn7p3P39378ETXlOltof6tsEOblK3u0hbUbCLtIlsLs+djx7if/37E99fNe54Nvem\necEZPdpiVaQNKdhFVrnhkUk+efc+7h+ZpFxxpuaLBNRa5cfSHTdesr1PoS7SphTsIqvY8Mgkf/aV\nJ9g/PktnImSyUqFaX1xm8aIzCwZTAR9/yxV6ni7SxhTsIqvU8Mgkv/f5hzmYK1D1iPWdCdKJGF3J\nam2ku0ElemaZ2De95Ax+/ycubnXZItJkCnaRVWah6/2e/ZPM5MukY4ZjPD1dYjATI5OKEzlgThgE\nrOtM8vNXn8nrX7Kt1aWLyDJQsIusIp+6dz8f/Y99HJktEgsN3JkpOl2pkAiYLkT0dsSIkjGGelK8\n7sVncM15epYuspYo2EVWgTseyfJXd32PBw9MEYaGOcTDEICIiNki9KZDZopV8uWACzZ28WvXnqNn\n6SJrkIJdZAVb2DP9sw9myZdrC8MGZuTLEUEYkYrHCKMqUQTlKnSnYvzM5Vu44ZItaqWLrFEKdpEV\nKJvL87lvHeAr3z3M42OzJGNGFEEQGDgkYlAoRyRDp1qFzX0pdqzr5uaXblcrXWSNU7CLrCALi8zc\n8Z0xRibm6UvHqbpTipyqO7HAKFedeBhQrUZUoogI57yN3bz5aoW6iCjYRVaM4ZFJbr1vhCePzDOT\nL1GuRswUKyRiAVV34iGUKl57H0WEodHfmeKXX7pNI95F5PsU7CIrwPDIJO/7t0cpVyPmihUKZa+3\nzJ3uVMiR2TJmkI7XQt4s4Mrt/fzyyzRATkSeTcEu0mLZXJ5b7xuhHFUZyCQplqvkKxWSgTFfqtKZ\njLFtIM6ByQIRzgvP6OHnrtzGtRcOtbp0EVmBFOwiLTY8mqMaOf0dSSoRDHammClWKJSrxEMjFhph\nEPKD5w7y83qOLiInoWAXabGJuRKDnQkCg+xUkWQs5Mz+DvaOzxEPA85a18kVOwa45rz1msImIiel\nYBdZRsMjk9z2cJax6QIbulNcf9EQ/ZkExXLEVL7KUE+SqXyFqUKFzX0dvOsV56mFLiKnJGh1ASJr\nxfDIJLfctY/ZYoVNPWlmixVuuWsfcYMwMM5Z30l/JkFXKsaOwU6FuoicFrXYRZqottDMKHd/7wiP\nj83SmQy5eGsvQUeSvo4kAPc+OclNV21neDRHMh6wa0sfuzb3qttdRE6Lgl2kCRYWmvnit59idDLP\nxu4kkUfMFJ379k9y2TbY2NNBTyrOU1N5hnrTCnIRWRIKdpEldscjWT7xn0+SncozNV8mk4wxW6yS\niodU3Yki2Ht4jo09HUwVymzoTrW6ZBFpIwp2kSXyqXv38+Gvfo8DkwXCADKJkLJDtVoiGQakEyFT\n8xWqVmWmUGFyvkhursIbXryl1aWLSBtRsIs8T9lcnvd+4SG+/N3DVKLascBhulglNAgTIdOFMuu6\nUmzpT/BULo+b05mM8YYXb9EAORFZUgp2kdM0PDLJP+0e4Z59k+w9PAeAAQ5UHGIBVCIoV50oqrC+\n03Gc8zd28evaK11EmkTBLnIaFqauzRRLlCpVoBbotuiaKILAIBkaDgShcf7GLl6vVrqINJGCXeQU\nZHN5hkdz/P09T2LUdlsrRxFhCPV8B2oh70DcoK8zyXtedb7WdheRZaEFakQalM3luX3PGPlSlWK5\nSiIImMqXMQI6E7W/kSMgtqjZvrUvrVAXkWWlFrvISSy00r/2+GESYcAFQz30ZZIUylUGMgkK5QrV\nSkAmYeRLThWIB/DD5w/yez/+Qs1PF5FlpWAXOY7hkUk+df8Bvn1wioGOBIlYyBl9KR48kOPcdRn+\nc+8kHcmA9V1JCmlndNLZ0pfgRWf28YZLt+o5uoi0hIJd5BgWBsdNF0qsyySpRM4Th2ZIxox13Smq\nwCsu2sDd3zuCU+WKHf1cf9GFCnMRaTkFu8gx3PZwlt5MjLlymWQ8JGXGYGeCxw/PMdiZYjpf5pz1\nXbzsvPVct3ODuttFZMVQsIvwzHP0ibkS/ZkE+8bnOXtdhslYmXLVScSM/kySYiWP40QO6UTIFTsG\nFOoisqIo2GXNWxjt3pWKMdiZZK5YYa5YITtVYH13kr3jtcVnipUKfR0Jtg92qpUuIiuWgl3WpMUt\n9CePzLGhO0VXKg5AVyrOpdt6+drjR7hgqIszB9LsH59ncr7Mq1+wUaEuIiuagl3WnOGRSW697wDV\nyBnIJBidmGc6X6EzGaM/U9sj/cJNvRTLEQ6MTRfYuamb6y8a0uA4EVnxFOyypmRzeW69b4QwMAYy\nSQrliOlChXgsYP/4/PeDfa5Y4eKtfbzyIi0sIyKri4Jd2t7R3e4zhTLbBjoxM9KJkDN6OxidnCce\nhkTuzBUrzBQqXLFjoNWli4icMgW7tKVsLs+djx7inn1HODiZZ9tABy/eNsDEXJmpfJnxuSLrOlMA\nDHQmmCmU6c/EGZ8t0p9JaLS7iKxaCnZpO9lcnk9+Yx/DB3NMzBaJBQHfG5+nVHV60gnWdaZ4Kleg\nMxkjFQvJ5Ut0pePcdNV2hbmIrHoKdmkbC630zz44yv7xebpTMQyIhwFzxTLjsyHd6RipeEhPKkYi\nNA7NFAgD48bLtirURaQtKNilLQyPTPLxb+zjwMQ8T0/n8cjJlyKqUUQ8DGst8/kSkOGcDRnGpgts\n6c+wa0sfuzb3KtRFpG0o2GVVq7XSx/jnBw4yWygz0JUiMKNCRMwhFhgzhTLdqThlj4gFRiwI1O0u\nIm1LwS6rzvDIJP/fN/by4IEpZosVulIhmWScCJiYK9GZjFGuOoVKmY54jGQsoFiNSMcDzt3QxTXn\nrVeoi0jbUrDLqpHN5fm7u/fx6QcOMluqEDNIxUMOTZeIhxUGuhLEAkjEAnrScXJzTgSEoXHehm5+\n4Qe2a4EZEWl7CnZZFbK5PJ9+YJQvf/cQxUqVZCykUK5iFSceN6JKxHyxQhgYFkF/JgHAmQMdXHv+\neq45T8vAisjaoGCXFWthYZlvH8jxjSfGmZgrMV2sUI0gEzcqkVF1J0FAZBGxIGAwk2B8tkQ1cl57\nyRnccMlmBbqIrCkKdlmRFnZcm5wvcs/eCWaKFcrVCMMpVqrEQiMZBsyXIypEdCRDetJxBjpT/Niu\nM/QcXUTWLAW7rCgLc9FveyhL1Z3cXInudILejgTlSgRAsRIxV6zQlYwRMyeqQjod8uoXbuSGS7Yo\n0EVkTVOwy4owPDLJ+7/8He4fmaJcjUjGArYNdDA2U6RcddZ3JkklQkrzERu7U4xNFyhFETEzrtre\nzy+/7BwNjBMRQcEuLbTQOr/z0UPs3neE6WKV0CAeGPlSxOOH5+lMBMyVKxSqcTZ0pUiEAaVKxJkD\nGf7LlWdqUJyIyFEU7NISwyOTvPeLD/P42Bxz5Sr1XnYIIG4ByZhTrkaUK+BuTM6X6OhNc86GTsoV\nuPmlmromInIsCnZZdtlcnvf96x6+k50lHguwRefKEcTdScQCqlFEhDHUmyI0I50I2NCd4vqLhhTq\nIiLHoWCXZTM8MsltD2f51sgkDx+cxsxJxULK1YhKKcLr11Uix6oRQWBkkjFesXMjP3mpBsWJiDRC\nwS5NtXgu+jf3TrB1IE06HqMaOdXIScSqpOIhxXJEpZ7soUG5GpEIA374/HUKdRGRU6Bgl6bI5vK8\n9wsPcefjRyhXaw/QN3Yn6EzFCMzpSITMlSrMl6r0pOJ0pmJM5ys4kI6HnL2+k1986Q6uvXCotb+I\niMgqo2CXJbWwQctXvnuI6WJECKTitYVkDuZKJGJzbOnPsLE7wchEVBsgV61SrjqDXXF++7pzef1L\ntrX61xARWbUU7LIksrk8f/qlPXzpO09TKMHCIPcIqLgTBhA5HJzMs6W/k5fsWAcc5unpEt3pOGcO\ndPBzV5ypFrqIyPOkYJfnJZvL8/lvjfLpBw6y9/AcgUEshFL1mWuqVScMjajilKqQjAVUoogd63r4\n49dp2pqIyFIKmv0FZvZKM3vUzJ4ws/92jPM3mdlhM3uw/vMLza5JlsYdj2T59Vsf4K+/vpeRiTms\nPm+tsijUHah67UUsgHgITkRnMqa56CIiTdDUFruZhcBHgOuAUeA+M/uCu+856tJ/dPdfbWYtsjQW\nj3L/v+3de3jU5Z338fd3kkkGQgAJiugQAggIAkYLQrvghgL7WGw5qPuAbV2lUrTA0mpXpGt3y9On\nBR8vLbZSagXrgboJi88l7XY9bKV0txUPRSFIZDkHiASFhCDBJJLk3j9mEiYhIZNJZiaZfF7XlYuZ\n3/H7zST5ct+/3+++X9l1HKjF503idEU1zgUKuTWxX41zJAELJmbx3S9dE9ugRUS6kGh3xd8A7HfO\nHQQwszxgJtC4sEsHFxj+9SPePFhKRloKB0+UYx6oqApMl+pN8vBZbS01BB5XS3JQ13D3GGR09zJ/\nYhbfzBkWzzRERBJetAv7lcDRkPdFwPgmtrvVzG4E9gL3OeeONrGNxFjdWO5vHSzhWFkF3iTD3ycN\nj8c4XPopaanJnDtXS62D7ilJVNfUUlsbuL7jDHxJHm4edRn/cNNIPYcuIhIj0S7sTfXKukbv/w3I\ndc5Vmdm9wHPAFy84kNkCYAFAZmZme8cpIYrLKti0vYhXdh3nTMU5vMlG724pFH9SRXWNY+QVvUlP\nTabyXC3dUjycrqjm0h6p1NbWUl5VQ7cUD1df3pN7Juk5dBGRWIt2YS8CBoS89wPHQjdwzpWEvF0L\n/L+mDuScewp4CmDs2LGN/3Mg7aC4rILntx5iw1+OUFZRgwNSkiDV68Fh+JI8lFdV8/GZCoZd3pO/\nFJbSt0cKSUkezlU7evi8zB2Xyd99YZBa6CIicRLtwv4XYKiZDQI+BOYCXw3dwMz6O+eKg29nALuj\nHJM0UtdC37jtCIdLKgm5qZ1zNYHhXZOp5IpLulNZU0tZxTku7+VjzJU9OVVRTd8eKQzq20OTs4iI\ndABRLezOuWozWwy8BiQBv3LOFZjZD4FtzrnfAkvMbAZQDZQCd0UzJjlvc0Exj/7HHvadOHt+2tQQ\ndddRzKD8sxqqqmu5rIeX1CQPJ858xl9ddSk5wy9T61xEpAOJ+gA1zrmXgZcbLfvnkNffA74X7Tik\nocdeKeDJPxVyromCDoGiXne9wxzgCRT47qlepozop4IuItJBaeS5LiR02tRthWU0U9OBQFH3BP+t\nBbole5h93RXMuk4zrYmIdGQq7F1A/pFTPPbaf/PukTI8SUZ1dc1Fi3qdWgLF/ZLuySy7abgmZxER\n6QRU2BNY/pFT/HzLXrYeLOVsVS1eD6QlJ1Ne3fK+BqQmG8MvT2fJ5Kv02JqISCehwp6ANr5TyOOb\n93Hs9Gfnr5MDNbVQVV2Lx4Ljtzcjq4+PueMymXmdX93uIiKdjAp7ggh0t+/m7UOnqGqinz04DwvV\nNbV4k6CmiVa7B5h17eU88CWNFCci0lmpsHdyG98pZPUfD3C4tLLFbetmWks2w5fscLXgS0nCl5LM\ntVf2ZNHkoXoOXUSkk1Nh76SKyyq4P3cbbx7+JOx96nrfq2sdGWmp3D/1Kt0QJyKSYFTYO5nNBcX8\n8r8OsLPoNJU1LW8P559JNyAtNYnPD+7Dwhy1zkVEEpEKeydQXFbBL/6wl43vFlERZjEP5YDuXuNv\nr/dz7+Shun4uIpLAVNg7sPwjp3h26yFe3VVMRRiPqDXFlwTjs/pw//+6Wi10EZEuQIW9A8o/coof\nbNpJ/rHyC+a4DVdvXxKLcgbzzZxh7RqbiIh0bCrsHUTdYDJv7D/J2XORHyezdwrfmjyUnOH91OUu\nItIFqbDH2cZ3Cnns93s5fibyau71QFZGdxZMGqS73EVEujgV9jjZXFDMt3PfC2t41+Z4gL8Z0Zcf\nzByj1rmIiAAq7DG18Z1CVm85wOFTLQ8m05L+6cn8aNYYjeEuIiINqLDHwLRHfs++0s/afJzUJLjW\n35t7bhysgi4iIk1SYY+i/COnmLNmK21tn6d5YfZ1fhZ9cZi63EVE5KJU2NtZcVkFv956iE07PuTY\nJ59F/Lia1wMj+vXg21OHqXUuIiJhU2FvJ8VlFfxmexHPvlnI8U8i73b3AituuUZ3t4uISERU2Nto\nc0ExK17+gEMllTQxW2qr5Azpzcq/vV7d7SIiEjEV9gjlHznFin8v4J3DpyPubgfIvjKd/zNztIZ7\nFRGRdqHC3gob3ynkV28Usuejs21unWf2TuEHXxml6+ciItKuVNjDUFxWwaOvfsDv3j9OVQSzq9VJ\nTYIR/dL5+ylDVdBFRCQqVNibsbmgmJWvfMCBk5Vt6mr3ACMvT+Nrn8/S+O0iIhJ1KuwhissqWP6b\nnby2+2Sbj5XigewBvdnPiTkAABC4SURBVLjnxiFqnYuISMyosBO4dv745v18eLqqzcdKT/EwO/sK\nvqXBZEQ6jXPnzlFUVERlZduHexZpK5/Ph9/vx+v1RrR/ly7smwuK+e6G9yhr+2ivZPb08sTXx+nu\ndpFOqKioiPT0dLKysjCzeIcjXZhzjpKSEoqKihg0aFBEx+iyhX1zQTFLX9zZ5qLezQtzrr+S5bOz\n2ycwEYm5yspKFXXpEMyMjIwMTpw4EfExumxh37CtiNMVrZ8z1ZcM4wb2YfqY/roZTiSBqKhLR9HW\nn8UuW9g/OlNJa59cGz+wF/948zXqbhcRkQ7LE+8A4qVfuo+kVmxf+PDNbPjWRBV1EaG4rIJXdxXz\nL28f5tVdxRSXVcTkvFlZWYwePZrRo0czcuRIvv/971NVFbjpt7CwkG7dupGdnV3/9fzzzzfYr275\n1q1b64+5atUqfD4fp0+frl/27LPPsnjx4gbnzsnJYdu2bfXHO3ny/NND69atqz92SkpK/bkeeuih\nJvNYt24dHo+HgoKC+mVXX301RUVFAPj9/gbx3nfffQBMnz6d3Nzc+n3mzZvHqlWrmDFjBtnZ2Vx1\n1VX06tWrfr+33367yfNPnDiRwYMHN1j25S9/md69e9e/f//998nJyWH48OEMHTqUFStWhBX/2LFj\nyc7OJjMzk0svvbQ+lqNHj+L3+ykrK6vf5/XXX2fWrFlNxtgWXbbFPmesn3cLSylppjvegDvH69q5\niDRUXFbB7z/4iHRfMn17pHK2qprff/AR00a236U55xzOOTyeC9teW7ZsoW/fvpSXl7NgwQIWLFjA\nc889B8CQIUPYsWNHk8es26+x3Nxcxo0bx0svvcRdd90VUbzz589n/vz5QKAo/+lPf2pQJJvi9/tZ\nsWIFL7zwQpPrmzrG6tWrmTp1KjfffDP5+fls376dtWvX1hf+119/ndWrV7Np06YWY+7RowdvvfUW\nEyZMoLS0lI8//rh+3aeffsrMmTNZu3YtU6ZM4ezZs8yePZuMjAzuueeei8Zf95+fdevWsWvXLh5/\n/PEWY2lvXbbFPuWa/jxy2xiyeqc2WJ7mhYduGsqhh29WUReRC+QXlZHuSybd58VjRrrPS7ovmfyi\nspZ3vojCwkJGjBjBwoULuf766zl69OhFt+/RowdPPvkkmzZtorS0NKJzHjhwgPLycn70ox81aAnH\nwqxZs3jvvffYv39/2PsMHjyYefPm8eCDD7Jw4ULWrFlDcnJk7dO5c+eSl5cHwIsvvsitt95av279\n+vXk5OQwZcoUANLS0njiiSd4+OGH2xR/rHTZFjsEirsGjxGR1ig9+xl9ezRqEKQmc7K87eNg7Nmz\nh2eeeYY1a9aEtX3Pnj0ZNGgQ+/bto1+/fhw4cIDs7PMNkieeeIJJkyYBMHnyZJKSkkhNTa3vos7N\nzeX2229n0qRJ7Nmzh48//pjLLruszXmEw+Px8MADD7By5UqefvrpC9ZPmjSJpKTABdNvfOMbLFmy\nBIClS5cyZMgQpkyZwhe+8IWIzz9t2jTuvvtuamtr2bBhA08//TQrV64EoKCggM997nMNth8+fDgl\nJSV8+umnYcXfnNC8ysvLGTVqVMQ5NKdLF3YRkdbqk5bC2apq0n3nBw85W1VNn7SUNh974MCBTJgw\noVX7OHd+0OvWdsXn5eXx0ksv4fF4uOWWW9i4cSOLFi1q9q7s9n5y4I477mDlypUcOXLkgnXNdefn\n5+cDsHv3bmpra5u8XBEOr9fLhAkT2LBhAzU1Nfj9/vp1zrmwcr1Y/M0Jzavu0kF767Jd8SIikbjW\n35szldWcqTxHrXOcqTzHmcpqrvVf/JpyONLS0lq1/ZkzZygsLGTYsGGtPtfOnTvZt28f06ZNIysr\ni7y8vPru+IyMDE6dOtVg+9LS0iav0beF1+vlvvvu45FHHglr+5qaGhYuXEheXh6ZmZmsXbu2Teef\nO3cuixcvZs6cOQ2WX3PNNfXXyuvs3buXjIwMunfvHnH8saLCLiLSCv17d2PayH50S0niZHkV3VKS\n2vXGuXCVl5ezcOFCZs2axSWXtP5pndzcXJYvX05hYSGFhYUcO3aMDz/8kMOHDzNu3DjeeOMNjh8/\nDgRuCKuqqmLAgAHtnQZ33303r7zySlj3CaxZs4ZRo0YxceJEHn/8cVasWEFJSUnE587JyWHZsmUX\nFPY77riDLVu2sGXLFiBwM92SJUtYunRpm+KPFRV2EZFW6t+7GzeN6s9Xxw/kplH9Y1rUJ0+ezKhR\no7jhhhvIzMzkl7/8Zf26umvsdV8/+9nPmj1OXl4es2fPbrBs9uzZ5OXl0a9fP376058yffp0srOz\n+c53vkNubm6Dbu8xY8bg9/vx+/3cf//9EeeTmprKokWLLhhpbdKkSfV5zJs3j+PHj/PYY4/Vt479\nfj+LFi1i2bJlEZ+77jp5nz59GixPS0tj06ZNLF++nOHDhzNmzBgmTpzIvffeG3b88WSh12c6i7Fj\nx7rG3SQiIpHavXs3I0aMiHcYIvWa+pk0s3edc2Nb2lctdhERkQSiu+JFRDqY8ePH148oV2f9+vWM\nHj06ThFFZt26dRfc9X3jjTde9BJBe5oxY8YFd6w/+uijTJ06NSbnjxd1xYtIl6eueOlo1BUvItJG\nnbGRI4mprT+LKuwi0uX5fD5KSkpU3CXunHOUlJTg8/kiPoausYtIl+f3+ykqKupQjyxJ1+Xz+RqM\nhNdaKuwi0uV5vV4GDRoU7zBE2oW64kVERBKICruIiEgCUWEXERFJIJ3yOXYzOwEcjncczegLnIx3\nEHHUlfNX7l2Tcu+6Yp3/QOfcpS1t1CkLe0dmZtvCGUAgUXXl/JW7cu9qunLu0HHzV1e8iIhIAlFh\nFxERSSAq7O3vqXgHEGddOX/l3jUp966rQ+ava+wiIiIJRC12ERGRBKLCHiEzu8nM9pjZfjNb1sT6\ne83sfTPbYWZ/NrOR8YgzGlrKPWS728zMmVmHu2s0UmF87neZ2Yng577DzObHI85oCeezN7P/bWYf\nmFmBmf1LrGOMljA++1Uhn/teMyuLR5zREEbumWa2xcy2m9lOM5sejzijIYzcB5rZ5mDefzSzyAd5\nby/OOX218gtIAg4Ag4EUIB8Y2WibniGvZwCvxjvuWOUe3C4d+C/gLWBsvOOO4ed+F7A63rHGMf+h\nwHbgkuD7y+Idd6xyb7T93wO/infcMfzcnwK+FXw9EiiMd9wxzH0jcGfw9ReB9fGOWy32yNwA7HfO\nHXTOfQbkATNDN3DOfRLyNg1IlJsZWsw96P8CjwCVsQwuysLNPVGFk/83gZ87504BOOc+jnGM0dLa\nz/52IDcmkUVfOLk7oGfwdS/gWAzji6Zwch8JbA6+3tLE+phTYY/MlcDRkPdFwWUNmNkiMztAoMAt\niVFs0dZi7mZ2HTDAOfe7WAYWA2F97sCtwW65F81sQGxCi4lw8h8GDDOzN8zsLTO7KWbRRVe4nz1m\nNhAYBPwhBnHFQji5Lwe+bmZFwMsEeiwSQTi55wO3Bl/PBtLNLCMGsTVLhT0y1sSyC1rkzrmfO+eG\nAA8C3496VLFx0dzNzAOsAr4bs4hiJ5zP/d+ALOfcGOB14LmoRxU74eSfTKA7PodAq3WdmfWOclyx\nENbvfNBc4EXnXE0U44mlcHK/HXjWOecHpgPrg38LOrtwcv8H4K/NbDvw18CHQHW0A7uYRPjGx0MR\nENoS83Pxrqc8YFZUI4qdlnJPB0YBfzSzQmAC8NsEuYGuxc/dOVfinKsKvl0LfC5GscVCOD/3RcBv\nnHPnnHOHgD0ECn1n15rf+bkkTjc8hJf73cC/Ajjn3gR8BMZR7+zC+Z0/5py7xTl3HfBQcNnp2IV4\nIRX2yPwFGGpmg8wshcAv8m9DNzCz0D9mNwP7YhhfNF00d+fcaedcX+dclnMui8DNczOcc9viE267\nCudz7x/ydgawO4bxRVuL+QObgMkAZtaXQNf8wZhGGR3h5I6ZDQcuAd6McXzRFE7uR4ApAGY2gkBh\nPxHTKKMjnN/5viG9E98DfhXjGC+gwh4B51w1sBh4jcAf7n91zhWY2Q/NbEZws8XBx312APcDd8Yp\n3HYVZu4JKczclwQ/93wC91XcFZ9o21+Y+b8GlJjZBwRuJHrAOVcSn4jbTyt+7m8H8lzwFulEEGbu\n3wW+Gfy5zwXuSoTvQZi55wB7zGwv0A/4cVyCDaGR50RERBKIWuwiIiIJRIVdREQkgaiwi4iIJBAV\ndhERkQSiwi4iIpJAVNhFREQSiAq7SJQEp3DcExw3/r/NbHXo8KpmVhMyzeeOuikhQ/arW35byD6z\nLTAV7tUhy3LM7HeNzv1s3X7B440NWTc95NjlIed6ppk8pgbP+aWQZa+a2cTg6z83indDcPkaM/te\nyD4/MLOfmtmTwe0+MLOKkP1mN3P+XwfjTAtZ9vNgTL2D7weY2W/NbJ+ZHTCzn5iZt6X4g/vssMCU\nnKdDYhkfzCs7ZJ+rguNSiHRoyfEOQCTBfc05ty04atVK4DcExpMGqHDOZV9svyaW3w78mcAIWMsj\nCcg59zKBiTowsz8Di51zLRWsowTmO3ilmfVzmjjGPwLbzSyXwN+aO4HsupkPzewqAmOqN/c9CHUQ\n+AqQZ2ZJwCTgePA4RuD7uso5t97MkoGngR8SGAms2fidczOCx5hK4PtQP/Rz4LAinY9a7CKtYGZZ\nwdb3c3Z+BrfuLe0XnPJxKZBpZtdGeO4ewF8RGJd7biTHaIP3gEozmxzuDs65MuCfgZ8Ba4CHGk1n\n3Bq5wJzg6ynAfwJ1k6z8DVDmnFsfPG818G1ggZn5Io1fpLNSYRdpveHAU8EZ3D4BFoazU3C2r3yg\nrhu9W6Ou+Dkhm78QsrxuCshZwKvOub1AqZld3z7phO3HND9L4YaQeB+uWxgstv2BVOdcWyZG2Q1c\naWa9CA7bGrLuGuDd0I2D/6k4BgwOM/7m1OdFE2PDi3REKuwirXfUOfdG8PWvgYmt2De0f7fCOZcd\n8rUhZN3XQpbXjbUeWtDygu+h+elD23W8aOfcHwCfmX2+idVzQuJdVrfQzDKBSwF/OD0bLdhEoKfi\nemBryHKj6VwbLG8h/ubU50VgUh+RDk/X2EVar3ERCauABq8NjyaCGd+CrfYvAqPMzAFJgDOzpUAJ\ngRnFQvUBTrb2PGH4McGpKcP0RHD764B/4vw170jkEZhta51zzoVcAy8gMINiveBNdVcAh4ArQ1a1\nNn6RTkctdpHWywxp9dXdzHZRwTu0VxJo7e+M4Jy3Ac875wYGp8QdQKBoTSQwJfAVFpguEzMbCFwL\ntPsd3MEb7y4n0P19UWb2FaCXc+4FAjf6zbHAtKaRnvsgga70Jxut+g/gEjP7WvC8ycBPgLXOucpI\n4xfprFTYRVpvN3Cnme0k0DL+xUW2fSG43S4gDZgZsq7xNfaHmz4EEPgPxEuNlv1/4KvOuSrg68Az\nwWvBLwLznXOnQ7b9dzMrCn5tDCvL5q0A/I2WhV5jf83MugGPEbz/wDlXDiwj0IKPmHPuF865Q42W\n1RK4/+B2M9sH7AHOEOghCDd+kYShaVtFWsHMsoDfOedGxTkUEZEmqcUuIiKSQNRiF2kjM3sJGNRo\n8YPOudfiEU+kzGw6gW7qUPudc7c1tX0Uzv8kMKHR4p84556PxflFEoUKu4iISAJRV7yIiEgCUWEX\nERFJICrsIiIiCUSFXUREJIGosIuIiCSQ/wFRWaLanB61SQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f06267ad3c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# initialize figure\n",
    "fig, ax_ = plt.subplots(figsize=(8, 8))         \n",
    "\n",
    "ax_.plot(test_yhat_female_fp[yhat],\n",
    "         test_yhat_female_fp['r_DEFAULT_NEXT_MONTH'],\n",
    "         marker='o', linestyle='', alpha=0.3)\n",
    "\n",
    "# annotate plot\n",
    "_ = plt.xlabel(yhat)\n",
    "_ = plt.ylabel('r_DEFAULT_NEXT_MONTH')\n",
    "_ = ax_.legend(loc=4)\n",
    "_ = plt.title('Logloss Residuals')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examine top 3 outlying individuals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>LIMIT_BAL</th>\n",
       "      <th>SEX</th>\n",
       "      <th>EDUCATION</th>\n",
       "      <th>MARRIAGE</th>\n",
       "      <th>AGE</th>\n",
       "      <th>PAY_0</th>\n",
       "      <th>PAY_2</th>\n",
       "      <th>PAY_3</th>\n",
       "      <th>PAY_4</th>\n",
       "      <th>PAY_5</th>\n",
       "      <th>PAY_6</th>\n",
       "      <th>BILL_AMT1</th>\n",
       "      <th>BILL_AMT2</th>\n",
       "      <th>BILL_AMT3</th>\n",
       "      <th>BILL_AMT4</th>\n",
       "      <th>BILL_AMT5</th>\n",
       "      <th>BILL_AMT6</th>\n",
       "      <th>PAY_AMT1</th>\n",
       "      <th>PAY_AMT2</th>\n",
       "      <th>PAY_AMT3</th>\n",
       "      <th>PAY_AMT4</th>\n",
       "      <th>PAY_AMT5</th>\n",
       "      <th>PAY_AMT6</th>\n",
       "      <th>DEFAULT_NEXT_MONTH</th>\n",
       "      <th>p_DEFAULT_NEXT_MONTH</th>\n",
       "      <th>d_DEFAULT_NEXT_MONTH</th>\n",
       "      <th>r_DEFAULT_NEXT_MONTH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1760</th>\n",
       "      <td>5916</td>\n",
       "      <td>110000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>married</td>\n",
       "      <td>41</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>150</td>\n",
       "      <td>150</td>\n",
       "      <td>150</td>\n",
       "      <td>150</td>\n",
       "      <td>150</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.911895</td>\n",
       "      <td>1</td>\n",
       "      <td>2.429222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>929</th>\n",
       "      <td>3087</td>\n",
       "      <td>30000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>24</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>300</td>\n",
       "      <td>300</td>\n",
       "      <td>300</td>\n",
       "      <td>300</td>\n",
       "      <td>300</td>\n",
       "      <td>300</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.907065</td>\n",
       "      <td>1</td>\n",
       "      <td>2.375853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5731</th>\n",
       "      <td>19316</td>\n",
       "      <td>110000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>married</td>\n",
       "      <td>41</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>150</td>\n",
       "      <td>150</td>\n",
       "      <td>150</td>\n",
       "      <td>150</td>\n",
       "      <td>150</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.902306</td>\n",
       "      <td>1</td>\n",
       "      <td>2.325911</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         ID  LIMIT_BAL     SEX        EDUCATION MARRIAGE  AGE  PAY_0  PAY_2  \\\n",
       "1760   5916     110000  female  graduate school  married   41      2      2   \n",
       "929    3087      30000  female       university   single   24      2      2   \n",
       "5731  19316     110000  female  graduate school  married   41      3      2   \n",
       "\n",
       "      PAY_3  PAY_4  PAY_5  PAY_6  BILL_AMT1  BILL_AMT2  BILL_AMT3  BILL_AMT4  \\\n",
       "1760      7      7      7      7        150        150        150        150   \n",
       "929       7      7      7      7        300        300        300        300   \n",
       "5731      2      7      7      7        150        150        150        150   \n",
       "\n",
       "      BILL_AMT5  BILL_AMT6  PAY_AMT1  PAY_AMT2  PAY_AMT3  PAY_AMT4  PAY_AMT5  \\\n",
       "1760        150        150         0         0         0         0         0   \n",
       "929         300        300         0         0         0         0         0   \n",
       "5731        150        150         0         0         0         0         0   \n",
       "\n",
       "      PAY_AMT6  DEFAULT_NEXT_MONTH  p_DEFAULT_NEXT_MONTH  \\\n",
       "1760         0                   0              0.911895   \n",
       "929          0                   0              0.907065   \n",
       "5731         0                   0              0.902306   \n",
       "\n",
       "      d_DEFAULT_NEXT_MONTH  r_DEFAULT_NEXT_MONTH  \n",
       "1760                     1              2.429222  \n",
       "929                      1              2.375853  \n",
       "5731                     1              2.325911  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_yhat_female_fp.sort_values(by='r_DEFAULT_NEXT_MONTH', ascending=False).head(n=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the top 3 highest residual predictions, it's clear these were customers with a horrible payment track record that somehow were able to make a last-minute payment. Although it's not possible to say they were not discrimated against by the model, it seems very clear that the model was justified to issue default predictions for these women. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adverse Action Notices \n",
    "Anyone that is denied further credit due to this model in the U.S. must be given reasons why. Shapley values provide a mechanism to rank the contributions of input variables to any given model decision and may be suitable for adverse action notice generation. Because the highest residual false positive female seems like a probable candidate to be rejected for further credit at some point, Shapley values can be used to say why according to this model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sum of Shapley contributions and bias: 2.3369910083711147\n",
      "Model prediction in margin space: 2.336991189841194\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAE0CAYAAADXDHM8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmUHXWZ//H3xyTsIEtaERIICoyg\noGgElPFnEPUEUXAYZRUHRgl6xNEBF2bGAQQXxGVQBnUAEZQhiMJAhAg4yjLIYprFsEQ0hECagDSL\nskSBwPP74/ttuLm53bequzpdXXxe59xz6lbVfeqppZ+u+tamiMDMzJrlJWOdgJmZVc/F3cysgVzc\nzcwayMXdzKyBXNzNzBrIxd3MrIFc3MeYpIMlXTNKsc+U9MXRiD3WJN0uaUbuPlbS2RXG/ldJp1cV\nb7xr30YlPSHplcOIc6Cky6vNzgbj4r4KSPpbSddK+rOkRyT9WtKbxjqvkZB0gKTe/Id+v6SfS/rb\nCuIW+ocUEa+JiCsrmN4MSX1tsb8cER8ZaeySeYSkLYf522n590/kz2JJR1Wd44CIWCciFhXMaWLL\n7/47It41WnnZiiZ2H8VGQtJ6wMXAx4DzgNWAtwJPjWVeIyHpCOAo4KPAZcDTwExgL2BUjkJapj0x\nIpaP5jTGsfUjYrmkNwO/lHRLRFzaOoKX34uH99xH39YAETE7Ip6NiL9ExOURMb91JElfl/SopLsl\n7d7S/xBJCyQ9LmmRpMNahs2Q1JebER7Ke2wHDpaIpPdIukXSn/KRxPa5/2cknd827smSTuoQ46XA\nccDHI+KCiHgyIp6JiJ9FxGfyOKtLOknS0vw5SdLqbTkfKenBvNd/SB42CzgQ+GzeA/1Z7r9Y0uck\nzQeelDQx93tHS2prSPpxXk43SXpdS84r7BUPHB1IWhv4ObBJy17vJu3NPJL2zM1Af5J0paRtWoYt\nlvRpSfPzkdmPJa2Rh02WdHH+3SOS/k/SSn9zkq7Onb/NOeyb+x8qaWH+7RxJmwy2bltFxHXA7cBr\nW+b/45L+APwh93u1pF/k2HdK2qcln43y9B6T9BvgVW35Pr88Ja0p6RuS7snzf42kNYGBefpTnqc3\na+XmnbdImpd/N0/SW1qGXSnpeKWj3MclXS5pch62hqSzJT2cl+08SS8vsmxeVCLCn1H8AOsBDwNn\nAbsDG7QNPxh4BjgUmEDaw18KKA/fg/THJeBtwDLgDXnYDGA58E1g9Tz8SeBv8vAzgS/m7jcADwI7\n5en8A7A4/+4V+Xfr53En5nHf2GF+ZuZpThxino8DrgdeBvQA1wLHt+V8HDAJeHeepw3ac26Jtxi4\nBZgKrNnS7x25+9i8DN+fY34auBuYlIcHsGVLvNblMgPoa5vescDZuXvrvGzemWN/FlgIrNaSx2+A\nTYANgQXAR/OwrwDfy7+bRDpi0yDLrD3HtwMP5fW2OnAycPUgv52Wfz+RtJ3skpfpbi2xf5HzWxNY\nG1gCHJJ/84Y8rdfk8c8lHWWuTfoHcR9wTadcgVOAK4FNSdvVW3K+z+fUtq1fk7s3BB4FDso57J+/\nb5SHXwnclZf/mvn7CXnYYcDPgLXyNN8IrDfWf+t1+3jPfZRFxGPA35I29NOA/rxX1LqncU9EnBYR\nz5L+CbwCeHn+/SURcVckVwGXk4pEq3+PiKfy8EuAfVjZocB/RcQNkY4gziI1De0cEfeT9rQ+kMed\nCTwUETd2iLNRHjbUof2BwHER8WBE9ANfIP0RD3gmD38mIuYCTwB/M0Q8gG9HxJKI+Msgw2+MiJ9G\nxDOkf3ZrADt3iVnEvsAlEfGLHPvrpGLzlpZxvh0RSyPiEVLReX3u/wxpXW6e5/X/IlenAg4EzoiI\nmyLiKeBfgDdLmjbEbx4CHgFOB46KiF+2DPtKRDySl997gMUR8YOIWB4RNwHnA++XNAH4e+DoSEdl\nt5G2yZXko5B/BD4ZEffl7eranG83ewB/iIgf5RxmA78D3tsyzg8i4vc55/NYcbluRPoH82xE3Jj/\nzqyFi/sqEBELIuLgiJhC2hPaBGht8nigZdxluXMdAEm7S7o+Hz7/ibSnO7nlt49GxJMt3+/J8dtt\nDhyZD2P/lGNNbRn3LOCDufuDwI8GmZ2HgclqOVHWwSY5j8Fyerjtn8My8vwOYUnR4RHxHNBH5+VQ\n1grzkmMvIe2pDnigpbt1Xr5G2su/XKlJrcxJzvbpPkFa9psO+guYHBEbRMQ2EfHttmGty29zYKe2\nbeFAYGPSkdbEtvFb1+UK0yP9E72ryAy1ad9GBqZTZLn+iHSu59zc7HeipEnDyKHRXNxXsYj4HalZ\n4LXdxs3t1OeT9hZfHhHrA3NJh94DNshtxwM2IzXrtFsCfCki1m/5rJX3mAAuBLaX9FrSnt1/D5LW\ndcBfgfcNkfpSUgHpllMng+3ZdtvjnTrQkfcop7RMcxnpEH7AxiXirjAvkpSndV+X3xERj0fEkRHx\nStIe6RGSduv2u0GmuzZpb7XrdAdLp6V7CXBV27awTkR8DOgnNZtNbRl/s0FiPkTaFl7VYVip5doy\nnSLL9ZmI+EJEbEs6gnoP8KFuv3uxcXEfZfnE1ZGSpuTvU0nti9cX+PlqpPbLfmC50onWTpeSfUHS\napLeStrQf9JhnNOAj0raScnakvaQtC5ARPwV+ClwDvCbiLi3U0IR8WfgaOAUSe+TtJakSfkI48Q8\n2mzg85J68kmwo4Gi16H/ESh9DTXwRkl75yOKT5GanAaW8S3AAZImSJpJOjfROr2NlE4Ud3IesIek\n3fLe4ZE59rXdElI6gb1l/ofwGPBs/nTSPt/nAIdIen3+J/9l4IaIWNxtugVcDGwt6aC87iZJepOk\nbXLT4AXAsXndbks6P7OSfBRzBvBNpRPRE/KJ04Ft9jkGX5dzcw4HKJ0g3xfYNuc2JEm7StouNyE9\nRmqmGWy5vmi5uI++x0knMW+Q9CSp4NxGKhJDiojHgX8iFZhHgQOAOW2jPZCHLSXtbX80Hx20x+ol\ntbv/Zx5/IekEV6uzgO0YvElmINY3gSOAz5P+iJcAh5P2/gG+CPQC84FbgZtyvyK+D2ybmwsu7Dr2\nCy4itY8PnKTbO7eRA3yStOc80PzwfNy8rGYDi/I0V2jKiYg7Sc1UJ5P2VN8LvDcini6Q01bA/5LO\nKVwHfCcGvzb/WOCsnMM+ub3830lHbveT9o73KzDNrvJ29a4cbylpG/oqaUcC0rpcJ/c/E/jBEOE+\nTVrH80jt/V8FXpKbF78E/DrP0wrnPyLiYdKOyJGk5qbPAu+JiIcKzMLGpB2Rx0gnsK+i+M7Di8bA\nFRk2DindoXl2bsuvIt5mpJNaG/sEldn45j13A55vpz4CONeF3Wz88x2qNnCy7o+kqxVmjnE6ZlYB\nN8uYmTWQm2XMzBrIxd3MrIHGrM198uTJMW3atLGavJnZuHTjjTc+FBE93cYbs+I+bdo0ent7x2ry\nZmbjkqTBHgexAjfLmJk1kIu7mVkDubibmTWQi7uZWQN1Le6SzlB6HdptQ4wzQ+n1bbdLuqraFM3M\nrKwie+5nMsQt6ZLWB74D7BkRr+GFt/mYmdkY6VrcI+Jq0qM8B3MAcMHA878j4sGKcjMzs2Gqos19\na9LbgK6UdKOkQd+IImmWpF5Jvf39/RVM2szMOqniJqaJpLeP70Z6cfB1kq6PiN+3jxgRpwKnAkyf\nPn2lJ5ZNO+qSISe0+IQ9KkjXzKz5qijufcBD+SXNT0q6GngdsFJxNzOzVaOKZpmLgLfm9yCuRXql\n3IIK4pqZ2TB13XOXNBuYAUyW1AccA0wCiIjvRcQCSZeS3pf5HHB6RAx62aSZmY2+rsU9IvYvMM7X\ngK9VkpGZmY2Y71A1M2sgF3czswZycTczayAXdzOzBnJxNzNrIBd3M7MGcnE3M2sgF3czswZycTcz\nayAXdzOzBnJxNzNrIBd3M7MGcnE3M2sgF3czswZycTczayAXdzOzBnJxNzNroK7FXdIZkh6UNOSr\n8yS9SdKzkt5fXXpmZjYcRfbczwRmDjWCpAnAV4HLKsjJzMxGqGtxj4irgUe6jPYJ4HzgwSqSMjOz\nkRlxm7ukTYG/A75XYNxZknol9fb394900mZmNogqTqieBHwuIp7tNmJEnBoR0yNiek9PTwWTNjOz\nTiZWEGM6cK4kgMnAuyUtj4gLK4htZmbDMOLiHhFbDHRLOhO42IXdzGxsdS3ukmYDM4DJkvqAY4BJ\nABHRtZ3dzMxWva7FPSL2LxosIg4eUTZmZlYJ36FqZtZALu5mZg3k4m5m1kAu7mZmDeTibmbWQC7u\nZmYN5OJuZtZALu5mZg3k4m5m1kAu7mZmDeTibmbWQC7uZmYN5OJuZtZALu5mZg3k4m5m1kAu7mZm\nDeTibmbWQF2Lu6QzJD0o6bZBhh8oaX7+XCvpddWnaWZmZRTZcz8TmDnE8LuBt0XE9sDxwKkV5GVm\nZiNQ5B2qV0uaNsTwa1u+Xg9MGXlaZmY2ElW3uX8Y+PlgAyXNktQrqbe/v7/iSZuZ2YDKirukXUnF\n/XODjRMRp0bE9IiY3tPTU9WkzcysTddmmSIkbQ+cDuweEQ9XEdPMzIZvxHvukjYDLgAOiojfjzwl\nMzMbqa577pJmAzOAyZL6gGOASQAR8T3gaGAj4DuSAJZHxPTRStjMzLorcrXM/l2GfwT4SGUZmZnZ\niPkOVTOzBnJxNzNrIBd3M7MGcnE3M2sgF3czswZycTczayAXdzOzBnJxNzNrIBd3M7MGcnE3M2sg\nF3czswZycTczayAXdzOzBnJxNzNrIBd3M7MGcnE3M2sgF3czswbqWtwlnSHpQUm3DTJckr4taaGk\n+ZLeUH2aZmZWRpE99zOBmUMM3x3YKn9mAd8deVpmZjYSXYt7RFwNPDLEKHsBP4zkemB9Sa+oKkEz\nMyuvijb3TYElLd/7cr+VSJolqVdSb39/fwWTNjOzTqoo7urQLzqNGBGnRsT0iJje09NTwaTNzKyT\nKop7HzC15fsUYGkFcc3MbJiqKO5zgA/lq2Z2Bv4cEfdXENfMzIZpYrcRJM0GZgCTJfUBxwCTACLi\ne8Bc4N3AQmAZcMhoJWtmZsV0Le4RsX+X4QF8vLKMzMxsxHyHqplZA7m4m5k1kIu7mVkDubibmTWQ\ni7uZWQO5uJuZNZCLu5lZA7m4m5k1kIu7mVkDubibmTWQi7uZWQO5uJuZNZCLu5lZA7m4m5k1kIu7\nmVkDubibmTWQi7uZWQMVKu6SZkq6U9JCSUd1GL6ZpCsk3SxpvqR3V5+qmZkV1bW4S5oAnALsDmwL\n7C9p27bRPg+cFxE7APsB36k6UTMzK67InvuOwMKIWBQRTwPnAnu1jRPAern7pcDS6lI0M7OyihT3\nTYElLd/7cr9WxwIflNQHzAU+0SmQpFmSeiX19vf3DyNdMzMrokhxV4d+0fZ9f+DMiJgCvBv4kaSV\nYkfEqRExPSKm9/T0lM/WzMwKKVLc+4CpLd+nsHKzy4eB8wAi4jpgDWByFQmamVl5RYr7PGArSVtI\nWo10wnRO2zj3ArsBSNqGVNzd7mJmNka6FveIWA4cDlwGLCBdFXO7pOMk7ZlHOxI4VNJvgdnAwRHR\n3nRjZmaryMQiI0XEXNKJ0tZ+R7d03wHsUm1qZmY2XL5D1cysgVzczcwayMXdzKyBXNzNzBrIxd3M\nrIFc3M3MGsjF3cysgVzczcwayMXdzKyBXNzNzBrIxd3MrIFc3M3MGsjF3cysgVzczcwayMXdzKyB\nXNzNzBrIxd3MrIEKFXdJMyXdKWmhpKMGGWcfSXdIul3SOdWmaWZmZXR9zZ6kCcApwDuBPmCepDn5\n1XoD42wF/AuwS0Q8Kullo5WwmZl1V2TPfUdgYUQsioingXOBvdrGORQ4JSIeBYiIB6tN08zMyihS\n3DcFlrR878v9Wm0NbC3p15KulzSzUyBJsyT1Surt7+8fXsZmZtZVkeKuDv2i7ftEYCtgBrA/cLqk\n9Vf6UcSpETE9Iqb39PSUzdXMzAoqUtz7gKkt36cASzuMc1FEPBMRdwN3koq9mZmNgSLFfR6wlaQt\nJK0G7AfMaRvnQmBXAEmTSc00i6pM1MzMiuta3CNiOXA4cBmwADgvIm6XdJykPfNolwEPS7oDuAL4\nTEQ8PFpJm5nZ0LpeCgkQEXOBuW39jm7pDuCI/DEzszHmO1TNzBrIxd3MrIFc3M3MGsjF3cysgVzc\nzcwayMXdzKyBXNzNzBrIxd3MrIFc3M3MGsjF3cysgVzczcwayMXdzKyBCj04bDyZdtQlXcdZfMIe\nqyATM7Ox4z13M7MGcnE3M2sgF3czswZycTcza6BCJ1QlzQS+BUwATo+IEwYZ7/3AT4A3RURvZVmO\nAZ+YNbPxrOueu6QJwCnA7sC2wP6Stu0w3rrAPwE3VJ2kmZmVU6RZZkdgYUQsioingXOBvTqMdzxw\nIvDXCvMzM7NhKFLcNwWWtHzvy/2eJ2kHYGpEXDxUIEmzJPVK6u3v7y+drJmZFVOkuKtDv3h+oPQS\n4D+AI7sFiohTI2J6REzv6ekpnqWZmZVSpLj3AVNbvk8BlrZ8Xxd4LXClpMXAzsAcSdOrStLMzMop\nUtznAVtJ2kLSasB+wJyBgRHx54iYHBHTImIacD2w53i/WsbMbDzrWtwjYjlwOHAZsAA4LyJul3Sc\npD1HO0EzMyuv0HXuETEXmNvW7+hBxp0x8rTMzGwkfIeqmVkDubibmTWQi7uZWQO5uJuZNZCLu5lZ\nA7m4m5k1kIu7mVkDNe4F2XXiZ8Kb2VhxcR8H/E/CzMpys4yZWQO5uJuZNZCLu5lZA7m4m5k1kIu7\nmVkDubibmTWQi7uZWQO5uJuZNVCh4i5ppqQ7JS2UdFSH4UdIukPSfEm/lLR59amamVlRXe9QlTQB\nOAV4J9AHzJM0JyLuaBntZmB6RCyT9DHgRGDf0UjYhs93upq9eBTZc98RWBgRiyLiaeBcYK/WESLi\niohYlr9eD0ypNk0zMyujSHHfFFjS8r0v9xvMh4GfdxogaZakXkm9/f39xbM0M7NSihR3degXHUeU\nPghMB77WaXhEnBoR0yNiek9PT/EszcyslCJPhewDprZ8nwIsbR9J0juAfwPeFhFPVZOemZkNR5E9\n93nAVpK2kLQasB8wp3UESTsA/wXsGREPVp+mmZmV0bW4R8Ry4HDgMmABcF5E3C7pOEl75tG+BqwD\n/ETSLZLmDBLOzMxWgUIv64iIucDctn5Ht3S/o+K8rKZ8OaXZ+OA7VM3MGsjF3cysgVzczcwayMXd\nzKyBXNzNzBrIxd3MrIFc3M3MGsjF3cysgVzczcwayMXdzKyBXNzNzBrIxd3MrIEKPTjMrGp+AJnZ\n6HJxt3Gt2z+JIv8g/I/GmsjNMmZmDeQ9d7OK+AjA6sR77mZmDVRoz13STOBbwATg9Ig4oW346sAP\ngTcCDwP7RsTialM1a76q9v5XVRwfidRX1+IuaQJwCvBOoA+YJ2lORNzRMtqHgUcjYktJ+wFfBfYd\njYTNbHyp2z+sF4sie+47AgsjYhGApHOBvYDW4r4XcGzu/inwn5IUEVFhrmZmI1anK6xG8x+WutVf\nSe8HZkbER/L3g4CdIuLwlnFuy+P05e935XEeaos1C5iVv/4NcGeX/CYDD3UZp4gq4jiX0Y1Tp1yq\niuNcRjdOnXKpKk6RGJtHRE+3QEX23NWhX/t/hCLjEBGnAqcWmGYKKvVGxPSi449mHOcyunHqlEtV\ncZzL6MapUy5VxakqFyh2tUwfMLXl+xRg6WDjSJoIvBR4pIoEzcysvCLFfR6wlaQtJK0G7AfMaRtn\nDvAPufv9wK/c3m5mNna6NstExHJJhwOXkS6FPCMibpd0HNAbEXOA7wM/krSQtMe+X0X5FW7CWQVx\nnMvoxqlTLlXFcS6jG6dOuVQVp6pcup9QNTOz8cd3qJqZNZCLu5lZA7m4m5k1kIu72SogaZOxzqFO\nJG1UUZyXVRGniWpV3CW9VNK+ko6Q9M+5e/1hxFlP0qs69N++RIzNJK2RuyXpEEknS/pYvpa/tHw5\n6d6SXj2M324saePc3ZPjvGY4ebTF/fJIY7TEKnOD2syW7pdK+r6k+ZLOkfTygjG2b+meJOnzkuZI\n+rKktUrksrGk70o6RdJGko6VdKuk8yS9omicLq4vmMsESYdJOl7SLm3DPl9FIpJ+XmLcwyVNzt1b\nSrpa0p8k3SBpu4IxTmiJMV3SIuAGSfdIeluJXDZs+2wE/EbSBpI2LBqnQ9zfD+M3a0n6rKTPSFpD\n0sF52ztR0jol4qwn6SuSfiTpgLZh3ymb1wq/r8vVMpI+BBwDXA7cl3tPIT2w7AsR8cOCcfYBTgIe\nBCYBB0fEvDzspoh4Q8E4twE7RsQySV8FXgVcCLwdICL+sUCMCyPifbl7r5zXlcBbgK9ExJkFczkM\nOIp0J/BXgYOB24FdgBMj4vsF43y7vRdwEOmJnkTEPxWIMdgfkYDfRsSUgrk8vy4knQ48AJwG7A28\nbWC5lYjxDWAj4AfA+4CNIuJDBXO5FLgEWBs4APhvYDbpmUnviIi9isTpMo0lETG1wHinA2sBvyGt\nm6si4og8rMz2O9h4Ai6OiEL/tCTdHhGvyd2XkJ4K+z+SZgBfiohdhgyQfndrRGyXu68APhsR8yRt\nDZxT9I5MSc8B97T1nkK6iTIi4pUFYjzOC3fPD9xZvxawLMdYr2Au5wFLgDVJj1JZAJwHvBfYOCIO\nKhjnfOAPpH/+/wg8AxwQEU+VWd8dRUQtPqTnzKzfof8GwO9LxLkFeEXu3hH4HbB3/n5ziTh3tHTf\nCLyk5ftvC8a4uaX7WmCL3D25aIw8/q2kDXAj4AnSxjOwbG4pEacPOBv4EOmms38A+ge6C8Z4FlgE\n3N3yGfj+dIlcbmpdZ+3rcBjL9xZgUu4WML9ELq1x7h1OLgWmcW/B8ea3dE8kXfd8AbB6ye33WeBX\nwBUdPn8pEefOlu55g+XaJcbvgIm5+/q2YbeWyOXTwKXAdi397i65Hk4m7cy8fLgxWreLvK09wAs7\nymW3vfZt/9+AX+e/9ZvK5tX6qdObmESH59EAz9H52TWDmRAR9wNExG8k7QpcLGnKIPEHs0TS2yPi\nV8Bi0uMV7inZVtg6vYkRcXfO66G8F1LUMxGxDFgm6a6IeCDHeVRSmXnaBjgemAl8JiLuk3RMRJxV\nIsYiYLeIuLd9gKQlJeK8TNIRpHW7nrTCU0SLNhe+VNLf5fFXj4hnIO1+lVwurdNrP0Is3HQp6WQ6\nb2MCijYvrjbQERHLgVmSjiYV6sKH+6Q9ycMi4g8d8iyznn4q6UzgOOB/JH2K9M9mN2ClbWAQpwBz\nJZ0AXCrppJYYtxRNJCK+rvRU2v/I83AM5f6miYhPSHojMFvShcB/lo3RFi8kzR3Ydoex7a0u6SUR\n8Vz+/Zck9QFXU259r6ROxf1LwE2SLicd7gBsRmqWOb5EnMclvSoi7gKIiPvzIeSFQJk26o8AP5R0\nLPBn4BZJN5P2lo8oGON1kh4j/XGvLmnjiHhA6TEOE0rk8pykSbl4Pf/8T6VzAoWLT0Q8Dnwqb9xn\n58PssuddTiItg05/2CeWiHMasG7uPot0NNOvdF6h6B/8VcCeuft6SS+PiD/mGGWezneRpHUi4omI\neL5dW9KWQJn22N5hDlthPEkzI+LSgR4RcZykpcB3S+RyLIOv208UDRIR/ybpYFIz1atIRxCzSH9P\nBxaMcbKkW4GPAVuT6s7WOcYXi+aSY/UBH5D0XuAXpCPaUiLiRknvAA4nbUNrlI1BWk8D28zzTbRK\n5/oeLxHnZ6Sm3v9tye8sSX8kHWUM30h2+6v+kIrGfsCRpEOw/YANSsZ4HbBlh/6TgAOHkdM2pLbX\nvwd2oqV5ZgTzuT7w5hLjb0Y+rG3rvympTXg4OQj4OHB2yd/tXNG63ruCGLXJpUv8NYAPjMd5qiKf\nCnPZu+37msBrRzI/wCuAd1e5XMhNNKty2XSMPVqBRy1huK5pceqUS7c4jLAdsMo4dcqlQ8wJwO6k\nZp4/Aj8dj/PUtPVUp1yqjNPpU6dmmaKGcwhV9zh1yqXKOC86kv4f6YqbPUhXvOxCOpG+bEwTsxed\n8Vjcq7p2s05x6pRLtzivlNT+yOcXfhix52DD2rxa0vwO/ZXCRJF7EuqUC/lE2L2ktvHPRMTjku4u\nWdhrNU8V5VNVLnXaZuq2nlYyHou7ja1+4BsVxLmbdE1wU3IBOJ90jf2+wLOSLqL8P9y6zVMV+VSV\nS522mbqtp5WMx+Je5rLI8RKnTrl0i/NERFxVwTSejoj2G1LKqlMuRMQn86WCuwL7A18jXea5DzA3\nIp4oEKZW80Q1+VSVS522mbqtp5XU5vEDkr4jqcjdYUPe+VWnOHXKpcI4dxf4fRG/riBGnXIB0nF0\nRPwqIg4FppEuF3wf6V6JIuo2T1XkU1Uuddpm6raeVlKnxw98FjgUOCYizmlCnDrlUlUcSXsPNTwi\nLigYZ8h7BSLim+MplwLTWDMi/lJgvFrNUxX5VJhLnbaZWq2njrHrUtwBJG0KfJN0Q8t3SXenAsUX\nVt3i1CmXKuLkO2tv4YUbjVqbcCIKPHOnLc7Pgafa4hARXxhPueQ4W5FuH3+EtIxPA94K3AV8OCK6\n3shUw3kacT6jkEudtplarKeOsetU3OH5B4h9iXS79UDhKbyw6hinTrmMNE6+3X9fYEvgImB2RCws\nM/0cZ4ccZybp2T2zgV9GiQ2yTrnkONeQrmtfD/hn4FOkOxDfCnwxInYqEKNu8zTifCrMpU7bTK3W\nU0cxShfQl/2QHg1wNXAu+cFf4z1OnXKpMk6ONfAExYuAa0hPchxurLeQbrVeAOw5XnOh5SFQwMLB\nho2neao6nypyqdM2U7f11Pqp09UyPwU+GRGXNyhOnXKpMg7AX0nP3HmM9HiEYd34JKkH2AHYjvTU\nygfHcS6tD4N7bIhhRdRlnirLp6pc6rTNVBWnwvX0gpH+d6jqQ3qqX6f+uwCnjMc4dcqlwnnalfQY\n2luArwPTh7m+DyE9vvVK0gOcXjaMGLXJJcdZBswnPaJ5oHvg+5PjdJ5GnE+FudRpm6nVeur0qV2b\nO4Ck15MOdfYhXXJ0QUSUfkI/8IHIAAAHaUlEQVRaneLUKZeRxMkngOaTDkGDtpt0osALP1ri3MoL\nT5dsj9P1Dr865ZLjbD7U8ChwPXMN52nE+VScS522mdqsp05q0yyj9FaW/Ug3fzwM/Jh0wnfX8Rqn\nTrlUGOeQMtMcQqncB1GnXAoVbwBJ10XEmwcZXKt5opp8qsqlTttM3dbTyqo6BBjph9QmeRUtj+sF\nFo3nOHXKpco4g8Qu/FjbLnGmkp7L0ohcBolb+I1KdZ2nKvKpMJc6bTO1WU+1uUOV9Lz0B4ArJJ0m\naTeGdzt9neLUKZcq4wCg9DLn3SX9kPRuy32HGWey0ovHrya1PRZ6QXZdcymgUFto3eapinwqzKVO\n20yt1tPzRvofpuoP6dKiA4GLSSelvgu8azzHqVMuVcQB/h/wPdIbs84n/cNYq2QO65Le5Xop6dV9\n3wD6hjEvtcmlxPSGfIZ33eZppPlUuK5rs83UcT2tFHu0NuBKkoMNgcOAXzUlTp1yGU4c0mVa15Ke\nP7Nu7nf3MKb7F1IT0Vt54Wa6ss1etcml5PQGbZap2zxVkU+FudRpm6nVeuoYu4oglSSS2qo+RXph\n7WF0eK3ceItTp1wqnKdvkQ49LyZdbbP2MP9Q/xm4AbgN+FfS+znL/qHWJpcc5/KC4w36argaztOI\n86kwlzptM7VaTx1jVxGkkkTSlRtn56JzIfCt8R6nTrlUHEekl/qeBtxHeiHwPsA6w4j1StLzWG4l\n3RDyOWDrcZpL6ROldZ+nKvOpIpc6bTN1W08rxaxiY6xog761pXsiw3y3YJ3i1CmXKuO0xZxEetnA\nOcBDI4y1HfBl4K7xmAupzXTvwT7jcZ5GK58qcqnTNlO39RRRr+J+01Dfx2OcOuVSZZwh4q/Z0n1+\nRTGH9dLvsciFdO/AGcAPOnzOaNLyHY18RpJLnbaZuqyn2tzEBLxO0sDzOASsmb8PvEuwyEsm6han\nTrlUGaejWPF55a8cSawWw3pWxxjlck+UfEJnGXVavjAq+YzpC96rmp+6rKfaFPeImNC0OHXKpco4\nRSdXozirKpeqXnNYRJ2Wb1VxnEuFcep0E5PZePfBsU7AbEBt9tytcer00u9Vlcv1kjrtYVXS7FUy\nl/EYx7lUGMd77lYZST9u+fq5LuNW8tLvOuUSEetGxHodPutWUdjrtHzL5FNVLnXaZqqKM5rz5OJu\nVXr+SYfR/YUgi4EbJR0w1EgRcdt4yUXShkN9Cmc+uDot3zL5VJVLVXEGU2b5VhVnMaM0T7V8nruN\nT5LujYjNSoxfyUu/65JLfjZ3H7B8oFfL4IiIEV05UaflWzafqnKp0zZTVZzRmie3uVspkt4w2CDS\njRyFRcR9ki4hvaz7vbS8rBvoulHXKZfsZGAG8GvSi46viZJ7T3Wbp6ryqWj51mabqdt66sTF3cr6\nxhDDflc0iKTXkPZSlgI7RsT94zwXIuKTkkQq8AcBJ0u6HPhuRNxdMEyt5qmKfKrKpU7bTFVxKlxP\nK6vi7il//IkIgJ1KjLuAYTyueLzkAqwPfBToBw5twjwNN5+qcqnTNjMe1pPb3K0yJdtgV4+Ipzr0\n3wU4ICI+Pt5ykbQ2sBfpZQ09pEPqH0fEkuKZDxm/Nsu3TD4VLt/abDNVxRnNeXKzjFWp8LW4rRu0\nOryse5zm8iDwB1J7+0JSm+mbJL0pT2ek81Wn5Vs4n6pyqdM2U1Wc0ZwnF3erUuHDQFX00u+a5fKT\nPN1X5097PiMtQHVavoXzqSqXOm0zVcUZzXlys4yVIulndN54Bbw9ItYuGOc54P+AD0fEwtxvUZS4\nXLBOuVSlbvNURT4V5lKnbaZW66kT77lbWV8f5rB2f0/aY7lC0qXAuZQ/LK5TLkg6YqjhEfHNAmFq\nNU8V5VNVLnXaZuq2nlbiPXcbU/kk5PtIh6VvB84C/idGdofgmOQi6ZghBkdEHDfyLMtp0vKtOk6d\njMY8ubhbKZJuZYg2xYjYfgSxNwQ+AOwbEW8fT7kUiPepiDipwHi1mqfRyqeq5TtW20zd1lPHOC7u\nVoakzYcaHhH3FIyzBuk68C1J7438fkQsH/pX9c2lwDSKXjZYq3mqIp8Kc6nTNlOr9dQxtou7jZSk\nycDDUWJjUnpy3jOkk0m7k95i9Mkm5dI2jSURMXWYv63VPJXNp6pc6rTNVBVnVLe9qu6G8ufF8QF2\nBq4kXda3A3Ab8ADpGu+ZJeJU8QLy2uRSYBr3jsd5qiKfCnOp0zZTq/XUMXZVgfx5cXyAXuBdpDbB\nR4Gdc/9XAzeXiFPFC8hrk0v+3ePAYx0+jwPLx+k8jTifCnOp0zZTq/XU6eNmGStF0i0R8frcvSAi\ntmkZdnNE7FAwzrPAkwNfgTWBZbk7osDLLeqUS1XqNk9V5FNhLnXaZmq1njrxde5W1nMt3X9pG1Z4\nTyGqeVl3nXKpSt3macT5VJVLnbaZquKM5rbnPXcrpWVPo3Uvg/x9jYgo9SzrpuRSlbrNU93yGamq\n5mc8LBcXdzOzBvI7VM3MGsjF3cysgVzczcwayMXdzKyBXNzNzBro/wNGr6qMV9jiEgAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f06267e9e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# select highest residual female false positive\n",
    "row = test_yhat_female_fp.sort_values(by='r_DEFAULT_NEXT_MONTH', ascending=False).head(n=1)\n",
    "\n",
    "# search for her index in shap_values array\n",
    "# create Pandas DataFrame and plot\n",
    "s_df = pd.DataFrame(shap_values[row.index[0], :].reshape(len(X), 1), columns=['Reason Codes'], index=X)\n",
    "_ = s_df.sort_values(by='Reason Codes', ascending=False).plot(kind='bar', legend=False, \n",
    "                                                              title='Shapley Contributions to Predictions')\n",
    "\n",
    "# check that Shapley is locally accurate\n",
    "print('Sum of Shapley contributions and bias:', s_df.sum()[0] + contributions_matrix[0, -1])\n",
    "print('Model prediction in margin space:', np.log(row[yhat].values/(1 - row[yhat].values))[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combining the Shapley values with the individual's data values allows ranked reasons to be given for denial of credit:\n",
    "\n",
    "1. Two months late on most recent payment\n",
    "2. Two months late on second most recent payment\n",
    "3. Seven months late on sixth most recent payment\n",
    "\n",
    "Because monotonicity contraints were used during model training, these reasons should be consistent across all denial decisions meaning that no one who was denied credit would have a better value for an input variable than someone who was granted credit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examine low logloss residual individuals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that high-residual individuals were not clearly discriminated against, their low-residual counter parts could be interesting to investigate next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>LIMIT_BAL</th>\n",
       "      <th>SEX</th>\n",
       "      <th>EDUCATION</th>\n",
       "      <th>MARRIAGE</th>\n",
       "      <th>AGE</th>\n",
       "      <th>PAY_0</th>\n",
       "      <th>PAY_2</th>\n",
       "      <th>PAY_3</th>\n",
       "      <th>PAY_4</th>\n",
       "      <th>PAY_5</th>\n",
       "      <th>PAY_6</th>\n",
       "      <th>BILL_AMT1</th>\n",
       "      <th>BILL_AMT2</th>\n",
       "      <th>BILL_AMT3</th>\n",
       "      <th>BILL_AMT4</th>\n",
       "      <th>BILL_AMT5</th>\n",
       "      <th>BILL_AMT6</th>\n",
       "      <th>PAY_AMT1</th>\n",
       "      <th>PAY_AMT2</th>\n",
       "      <th>PAY_AMT3</th>\n",
       "      <th>PAY_AMT4</th>\n",
       "      <th>PAY_AMT5</th>\n",
       "      <th>PAY_AMT6</th>\n",
       "      <th>DEFAULT_NEXT_MONTH</th>\n",
       "      <th>p_DEFAULT_NEXT_MONTH</th>\n",
       "      <th>d_DEFAULT_NEXT_MONTH</th>\n",
       "      <th>r_DEFAULT_NEXT_MONTH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5869</th>\n",
       "      <td>19776</td>\n",
       "      <td>110000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>5174</td>\n",
       "      <td>6184</td>\n",
       "      <td>8383</td>\n",
       "      <td>8394</td>\n",
       "      <td>7709</td>\n",
       "      <td>8084</td>\n",
       "      <td>1106</td>\n",
       "      <td>2393</td>\n",
       "      <td>300</td>\n",
       "      <td>0</td>\n",
       "      <td>500</td>\n",
       "      <td>500</td>\n",
       "      <td>0</td>\n",
       "      <td>0.270166</td>\n",
       "      <td>1</td>\n",
       "      <td>0.314938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2959</th>\n",
       "      <td>9916</td>\n",
       "      <td>20000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>24</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>379</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.270243</td>\n",
       "      <td>1</td>\n",
       "      <td>0.315044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3508</th>\n",
       "      <td>11755</td>\n",
       "      <td>160000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6204</td>\n",
       "      <td>6204</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6204</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.270293</td>\n",
       "      <td>1</td>\n",
       "      <td>0.315112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>321</th>\n",
       "      <td>1057</td>\n",
       "      <td>190000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>115056</td>\n",
       "      <td>108137</td>\n",
       "      <td>89722</td>\n",
       "      <td>69861</td>\n",
       "      <td>49894</td>\n",
       "      <td>49727</td>\n",
       "      <td>0</td>\n",
       "      <td>4413</td>\n",
       "      <td>2942</td>\n",
       "      <td>2000</td>\n",
       "      <td>1583</td>\n",
       "      <td>1500</td>\n",
       "      <td>0</td>\n",
       "      <td>0.271024</td>\n",
       "      <td>1</td>\n",
       "      <td>0.316114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8353</th>\n",
       "      <td>28028</td>\n",
       "      <td>240000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>262</td>\n",
       "      <td>0</td>\n",
       "      <td>1389</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>262</td>\n",
       "      <td>0</td>\n",
       "      <td>1389</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.271053</td>\n",
       "      <td>1</td>\n",
       "      <td>0.316154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5993</th>\n",
       "      <td>20181</td>\n",
       "      <td>170000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>married</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>185</td>\n",
       "      <td>185</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>185</td>\n",
       "      <td>0</td>\n",
       "      <td>4578</td>\n",
       "      <td>0</td>\n",
       "      <td>0.271532</td>\n",
       "      <td>1</td>\n",
       "      <td>0.316812</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1513</th>\n",
       "      <td>5066</td>\n",
       "      <td>200000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>married</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>900</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>900</td>\n",
       "      <td>880</td>\n",
       "      <td>0</td>\n",
       "      <td>0.271769</td>\n",
       "      <td>1</td>\n",
       "      <td>0.317137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2531</th>\n",
       "      <td>8476</td>\n",
       "      <td>360000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>51</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>390</td>\n",
       "      <td>540</td>\n",
       "      <td>540</td>\n",
       "      <td>390</td>\n",
       "      <td>0</td>\n",
       "      <td>780</td>\n",
       "      <td>540</td>\n",
       "      <td>540</td>\n",
       "      <td>390</td>\n",
       "      <td>390</td>\n",
       "      <td>0</td>\n",
       "      <td>0.271795</td>\n",
       "      <td>1</td>\n",
       "      <td>0.317172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2389</th>\n",
       "      <td>8018</td>\n",
       "      <td>230000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>185399</td>\n",
       "      <td>198941</td>\n",
       "      <td>201358</td>\n",
       "      <td>204067</td>\n",
       "      <td>200720</td>\n",
       "      <td>212403</td>\n",
       "      <td>18150</td>\n",
       "      <td>7200</td>\n",
       "      <td>7500</td>\n",
       "      <td>0</td>\n",
       "      <td>15079</td>\n",
       "      <td>8000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.271860</td>\n",
       "      <td>1</td>\n",
       "      <td>0.317262</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3986</th>\n",
       "      <td>13358</td>\n",
       "      <td>100000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>3473</td>\n",
       "      <td>0</td>\n",
       "      <td>1012</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3473</td>\n",
       "      <td>0</td>\n",
       "      <td>1012</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.272039</td>\n",
       "      <td>1</td>\n",
       "      <td>0.317508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2881</th>\n",
       "      <td>9638</td>\n",
       "      <td>20000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5473</td>\n",
       "      <td>7492</td>\n",
       "      <td>4988</td>\n",
       "      <td>11680</td>\n",
       "      <td>6590</td>\n",
       "      <td>0</td>\n",
       "      <td>2300</td>\n",
       "      <td>0</td>\n",
       "      <td>102</td>\n",
       "      <td>1500</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.272170</td>\n",
       "      <td>1</td>\n",
       "      <td>0.317688</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>434</th>\n",
       "      <td>1390</td>\n",
       "      <td>20000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>11192</td>\n",
       "      <td>15677</td>\n",
       "      <td>15133</td>\n",
       "      <td>15582</td>\n",
       "      <td>16677</td>\n",
       "      <td>16259</td>\n",
       "      <td>5000</td>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>1500</td>\n",
       "      <td>0</td>\n",
       "      <td>700</td>\n",
       "      <td>0</td>\n",
       "      <td>0.272171</td>\n",
       "      <td>1</td>\n",
       "      <td>0.317689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2890</th>\n",
       "      <td>9660</td>\n",
       "      <td>140000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>144922</td>\n",
       "      <td>141289</td>\n",
       "      <td>137377</td>\n",
       "      <td>93261</td>\n",
       "      <td>95368</td>\n",
       "      <td>97063</td>\n",
       "      <td>17</td>\n",
       "      <td>5547</td>\n",
       "      <td>3242</td>\n",
       "      <td>3512</td>\n",
       "      <td>3333</td>\n",
       "      <td>4383</td>\n",
       "      <td>0</td>\n",
       "      <td>0.272392</td>\n",
       "      <td>1</td>\n",
       "      <td>0.317993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3996</th>\n",
       "      <td>13390</td>\n",
       "      <td>340000</td>\n",
       "      <td>female</td>\n",
       "      <td>high school</td>\n",
       "      <td>married</td>\n",
       "      <td>36</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>2566</td>\n",
       "      <td>1689</td>\n",
       "      <td>-11</td>\n",
       "      <td>35947</td>\n",
       "      <td>83360</td>\n",
       "      <td>82516</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>35958</td>\n",
       "      <td>50000</td>\n",
       "      <td>0</td>\n",
       "      <td>10000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.272520</td>\n",
       "      <td>1</td>\n",
       "      <td>0.318169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8420</th>\n",
       "      <td>28250</td>\n",
       "      <td>150000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>married</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-3</td>\n",
       "      <td>350</td>\n",
       "      <td>0</td>\n",
       "      <td>863</td>\n",
       "      <td>863</td>\n",
       "      <td>0</td>\n",
       "      <td>353</td>\n",
       "      <td>0</td>\n",
       "      <td>863</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.272878</td>\n",
       "      <td>1</td>\n",
       "      <td>0.318661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1900</th>\n",
       "      <td>6369</td>\n",
       "      <td>40000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>20936</td>\n",
       "      <td>20647</td>\n",
       "      <td>21561</td>\n",
       "      <td>18438</td>\n",
       "      <td>20619</td>\n",
       "      <td>17582</td>\n",
       "      <td>3000</td>\n",
       "      <td>4000</td>\n",
       "      <td>0</td>\n",
       "      <td>5000</td>\n",
       "      <td>0</td>\n",
       "      <td>8000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.272988</td>\n",
       "      <td>1</td>\n",
       "      <td>0.318812</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7703</th>\n",
       "      <td>25869</td>\n",
       "      <td>350000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>39</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>590</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>590</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.273208</td>\n",
       "      <td>1</td>\n",
       "      <td>0.319115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8532</th>\n",
       "      <td>28613</td>\n",
       "      <td>360000</td>\n",
       "      <td>female</td>\n",
       "      <td>high school</td>\n",
       "      <td>single</td>\n",
       "      <td>39</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>271</td>\n",
       "      <td>-1</td>\n",
       "      <td>560</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>271</td>\n",
       "      <td>0</td>\n",
       "      <td>561</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.273208</td>\n",
       "      <td>1</td>\n",
       "      <td>0.319115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>309</th>\n",
       "      <td>1012</td>\n",
       "      <td>410000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>married</td>\n",
       "      <td>31</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>2744</td>\n",
       "      <td>2041</td>\n",
       "      <td>630</td>\n",
       "      <td>-736</td>\n",
       "      <td>-1591</td>\n",
       "      <td>-2303</td>\n",
       "      <td>2051</td>\n",
       "      <td>5</td>\n",
       "      <td>266</td>\n",
       "      <td>266</td>\n",
       "      <td>253</td>\n",
       "      <td>6384</td>\n",
       "      <td>0</td>\n",
       "      <td>0.273953</td>\n",
       "      <td>1</td>\n",
       "      <td>0.320140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5925</th>\n",
       "      <td>19928</td>\n",
       "      <td>330000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>39</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>1380</td>\n",
       "      <td>0</td>\n",
       "      <td>404</td>\n",
       "      <td>404</td>\n",
       "      <td>24420</td>\n",
       "      <td>1380</td>\n",
       "      <td>0</td>\n",
       "      <td>404</td>\n",
       "      <td>0</td>\n",
       "      <td>24420</td>\n",
       "      <td>480</td>\n",
       "      <td>0</td>\n",
       "      <td>0.274621</td>\n",
       "      <td>1</td>\n",
       "      <td>0.321061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1322</th>\n",
       "      <td>4397</td>\n",
       "      <td>80000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>25</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>390</td>\n",
       "      <td>5640</td>\n",
       "      <td>390</td>\n",
       "      <td>780</td>\n",
       "      <td>390</td>\n",
       "      <td>390</td>\n",
       "      <td>5250</td>\n",
       "      <td>0</td>\n",
       "      <td>780</td>\n",
       "      <td>0</td>\n",
       "      <td>390</td>\n",
       "      <td>390</td>\n",
       "      <td>0</td>\n",
       "      <td>0.274808</td>\n",
       "      <td>1</td>\n",
       "      <td>0.321319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7789</th>\n",
       "      <td>26144</td>\n",
       "      <td>300000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>43</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>499</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>507</td>\n",
       "      <td>0</td>\n",
       "      <td>629</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>507</td>\n",
       "      <td>0</td>\n",
       "      <td>629</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.275279</td>\n",
       "      <td>1</td>\n",
       "      <td>0.321969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>854</th>\n",
       "      <td>2876</td>\n",
       "      <td>140000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>811</td>\n",
       "      <td>1633</td>\n",
       "      <td>1666</td>\n",
       "      <td>1700</td>\n",
       "      <td>0</td>\n",
       "      <td>811</td>\n",
       "      <td>1000</td>\n",
       "      <td>33</td>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.275537</td>\n",
       "      <td>1</td>\n",
       "      <td>0.322325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2220</th>\n",
       "      <td>7423</td>\n",
       "      <td>60000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>24066</td>\n",
       "      <td>24786</td>\n",
       "      <td>27050</td>\n",
       "      <td>26317</td>\n",
       "      <td>28200</td>\n",
       "      <td>28762</td>\n",
       "      <td>1414</td>\n",
       "      <td>2678</td>\n",
       "      <td>0</td>\n",
       "      <td>2310</td>\n",
       "      <td>1173</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.275588</td>\n",
       "      <td>1</td>\n",
       "      <td>0.322394</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4462</th>\n",
       "      <td>14924</td>\n",
       "      <td>160000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>43</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>695</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>695</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.275657</td>\n",
       "      <td>1</td>\n",
       "      <td>0.322491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3654</th>\n",
       "      <td>12194</td>\n",
       "      <td>220000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>married</td>\n",
       "      <td>54</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>788</td>\n",
       "      <td>0</td>\n",
       "      <td>349</td>\n",
       "      <td>600</td>\n",
       "      <td>0</td>\n",
       "      <td>788</td>\n",
       "      <td>0</td>\n",
       "      <td>349</td>\n",
       "      <td>600</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.275657</td>\n",
       "      <td>1</td>\n",
       "      <td>0.322491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3243</th>\n",
       "      <td>10884</td>\n",
       "      <td>160000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>30</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>6548</td>\n",
       "      <td>3200</td>\n",
       "      <td>0</td>\n",
       "      <td>4775</td>\n",
       "      <td>456</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>4775</td>\n",
       "      <td>456</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.275847</td>\n",
       "      <td>1</td>\n",
       "      <td>0.322752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>206</th>\n",
       "      <td>665</td>\n",
       "      <td>40000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>24114</td>\n",
       "      <td>25417</td>\n",
       "      <td>28777</td>\n",
       "      <td>29107</td>\n",
       "      <td>29693</td>\n",
       "      <td>30456</td>\n",
       "      <td>2000</td>\n",
       "      <td>4110</td>\n",
       "      <td>1100</td>\n",
       "      <td>1200</td>\n",
       "      <td>1400</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.276142</td>\n",
       "      <td>1</td>\n",
       "      <td>0.323160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4658</th>\n",
       "      <td>15646</td>\n",
       "      <td>130000</td>\n",
       "      <td>female</td>\n",
       "      <td>high school</td>\n",
       "      <td>married</td>\n",
       "      <td>42</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.276634</td>\n",
       "      <td>1</td>\n",
       "      <td>0.323840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306</th>\n",
       "      <td>4347</td>\n",
       "      <td>30000</td>\n",
       "      <td>female</td>\n",
       "      <td>high school</td>\n",
       "      <td>married</td>\n",
       "      <td>44</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1072</td>\n",
       "      <td>5122</td>\n",
       "      <td>4440</td>\n",
       "      <td>4050</td>\n",
       "      <td>3660</td>\n",
       "      <td>3660</td>\n",
       "      <td>4440</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.276657</td>\n",
       "      <td>1</td>\n",
       "      <td>0.323871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4238</th>\n",
       "      <td>14201</td>\n",
       "      <td>290000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>18099</td>\n",
       "      <td>18912</td>\n",
       "      <td>18249</td>\n",
       "      <td>20305</td>\n",
       "      <td>26615</td>\n",
       "      <td>27063</td>\n",
       "      <td>1400</td>\n",
       "      <td>0</td>\n",
       "      <td>2305</td>\n",
       "      <td>6615</td>\n",
       "      <td>1896</td>\n",
       "      <td>2358</td>\n",
       "      <td>0</td>\n",
       "      <td>0.300226</td>\n",
       "      <td>1</td>\n",
       "      <td>0.356998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1951</th>\n",
       "      <td>6527</td>\n",
       "      <td>20000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>24</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>632</td>\n",
       "      <td>316</td>\n",
       "      <td>948</td>\n",
       "      <td>632</td>\n",
       "      <td>316</td>\n",
       "      <td>1096</td>\n",
       "      <td>0</td>\n",
       "      <td>948</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1096</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.301021</td>\n",
       "      <td>1</td>\n",
       "      <td>0.358135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3820</th>\n",
       "      <td>12799</td>\n",
       "      <td>100000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>105033</td>\n",
       "      <td>102513</td>\n",
       "      <td>100693</td>\n",
       "      <td>71253</td>\n",
       "      <td>70206</td>\n",
       "      <td>71060</td>\n",
       "      <td>0</td>\n",
       "      <td>3509</td>\n",
       "      <td>3047</td>\n",
       "      <td>3500</td>\n",
       "      <td>3000</td>\n",
       "      <td>2500</td>\n",
       "      <td>0</td>\n",
       "      <td>0.302226</td>\n",
       "      <td>1</td>\n",
       "      <td>0.359859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>832</th>\n",
       "      <td>2784</td>\n",
       "      <td>280000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>married</td>\n",
       "      <td>47</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2002</td>\n",
       "      <td>1311</td>\n",
       "      <td>0</td>\n",
       "      <td>430</td>\n",
       "      <td>430</td>\n",
       "      <td>-550</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>430</td>\n",
       "      <td>0</td>\n",
       "      <td>550</td>\n",
       "      <td>1304</td>\n",
       "      <td>0</td>\n",
       "      <td>0.302589</td>\n",
       "      <td>1</td>\n",
       "      <td>0.360380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1680</th>\n",
       "      <td>5665</td>\n",
       "      <td>50000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>349</td>\n",
       "      <td>350</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>350</td>\n",
       "      <td>351</td>\n",
       "      <td>4076</td>\n",
       "      <td>0</td>\n",
       "      <td>0.302610</td>\n",
       "      <td>1</td>\n",
       "      <td>0.360411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2926</th>\n",
       "      <td>9803</td>\n",
       "      <td>20000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>15034</td>\n",
       "      <td>16285</td>\n",
       "      <td>20498</td>\n",
       "      <td>19509</td>\n",
       "      <td>20038</td>\n",
       "      <td>18982</td>\n",
       "      <td>1500</td>\n",
       "      <td>4500</td>\n",
       "      <td>0</td>\n",
       "      <td>1600</td>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.303381</td>\n",
       "      <td>1</td>\n",
       "      <td>0.361517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1903</th>\n",
       "      <td>6383</td>\n",
       "      <td>80000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>82891</td>\n",
       "      <td>80997</td>\n",
       "      <td>81380</td>\n",
       "      <td>75374</td>\n",
       "      <td>77158</td>\n",
       "      <td>78710</td>\n",
       "      <td>0</td>\n",
       "      <td>3299</td>\n",
       "      <td>2699</td>\n",
       "      <td>3000</td>\n",
       "      <td>2987</td>\n",
       "      <td>3100</td>\n",
       "      <td>0</td>\n",
       "      <td>0.303567</td>\n",
       "      <td>1</td>\n",
       "      <td>0.361784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>420</th>\n",
       "      <td>1346</td>\n",
       "      <td>120000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>77785</td>\n",
       "      <td>75984</td>\n",
       "      <td>78109</td>\n",
       "      <td>79198</td>\n",
       "      <td>80865</td>\n",
       "      <td>82859</td>\n",
       "      <td>0</td>\n",
       "      <td>4000</td>\n",
       "      <td>2988</td>\n",
       "      <td>3100</td>\n",
       "      <td>3500</td>\n",
       "      <td>6382</td>\n",
       "      <td>0</td>\n",
       "      <td>0.303567</td>\n",
       "      <td>1</td>\n",
       "      <td>0.361784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>836</th>\n",
       "      <td>2806</td>\n",
       "      <td>120000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>married</td>\n",
       "      <td>39</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>915</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>915</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.304701</td>\n",
       "      <td>1</td>\n",
       "      <td>0.363414</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5058</th>\n",
       "      <td>17006</td>\n",
       "      <td>150000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>-200</td>\n",
       "      <td>200</td>\n",
       "      <td>200</td>\n",
       "      <td>200</td>\n",
       "      <td>550</td>\n",
       "      <td>550</td>\n",
       "      <td>600</td>\n",
       "      <td>200</td>\n",
       "      <td>200</td>\n",
       "      <td>550</td>\n",
       "      <td>200</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.305159</td>\n",
       "      <td>1</td>\n",
       "      <td>0.364072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4169</th>\n",
       "      <td>13971</td>\n",
       "      <td>50000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>34361</td>\n",
       "      <td>44804</td>\n",
       "      <td>44800</td>\n",
       "      <td>43576</td>\n",
       "      <td>44315</td>\n",
       "      <td>34361</td>\n",
       "      <td>10443</td>\n",
       "      <td>2001</td>\n",
       "      <td>1</td>\n",
       "      <td>2001</td>\n",
       "      <td>2001</td>\n",
       "      <td>0</td>\n",
       "      <td>0.306716</td>\n",
       "      <td>1</td>\n",
       "      <td>0.366315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>586</th>\n",
       "      <td>1902</td>\n",
       "      <td>110000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>42</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>57046</td>\n",
       "      <td>51005</td>\n",
       "      <td>48572</td>\n",
       "      <td>49299</td>\n",
       "      <td>47126</td>\n",
       "      <td>48215</td>\n",
       "      <td>0</td>\n",
       "      <td>2100</td>\n",
       "      <td>2500</td>\n",
       "      <td>2000</td>\n",
       "      <td>2250</td>\n",
       "      <td>1200</td>\n",
       "      <td>0</td>\n",
       "      <td>0.307312</td>\n",
       "      <td>1</td>\n",
       "      <td>0.367176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5244</th>\n",
       "      <td>17657</td>\n",
       "      <td>20000</td>\n",
       "      <td>female</td>\n",
       "      <td>high school</td>\n",
       "      <td>married</td>\n",
       "      <td>44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>20313</td>\n",
       "      <td>20111</td>\n",
       "      <td>18057</td>\n",
       "      <td>18536</td>\n",
       "      <td>11532</td>\n",
       "      <td>19450</td>\n",
       "      <td>3030</td>\n",
       "      <td>0</td>\n",
       "      <td>2175</td>\n",
       "      <td>0</td>\n",
       "      <td>10000</td>\n",
       "      <td>931</td>\n",
       "      <td>0</td>\n",
       "      <td>0.308034</td>\n",
       "      <td>1</td>\n",
       "      <td>0.368218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4187</th>\n",
       "      <td>14025</td>\n",
       "      <td>50000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>married</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>50175</td>\n",
       "      <td>48275</td>\n",
       "      <td>49646</td>\n",
       "      <td>44051</td>\n",
       "      <td>45912</td>\n",
       "      <td>45383</td>\n",
       "      <td>0</td>\n",
       "      <td>7775</td>\n",
       "      <td>3000</td>\n",
       "      <td>3000</td>\n",
       "      <td>5000</td>\n",
       "      <td>3000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.308502</td>\n",
       "      <td>1</td>\n",
       "      <td>0.368895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8468</th>\n",
       "      <td>28386</td>\n",
       "      <td>100000</td>\n",
       "      <td>female</td>\n",
       "      <td>high school</td>\n",
       "      <td>married</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>249</td>\n",
       "      <td>317</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>249</td>\n",
       "      <td>317</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.309639</td>\n",
       "      <td>1</td>\n",
       "      <td>0.370540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6790</th>\n",
       "      <td>22819</td>\n",
       "      <td>360000</td>\n",
       "      <td>female</td>\n",
       "      <td>high school</td>\n",
       "      <td>single</td>\n",
       "      <td>35</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>2500</td>\n",
       "      <td>2500</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.310088</td>\n",
       "      <td>1</td>\n",
       "      <td>0.371191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7619</th>\n",
       "      <td>25590</td>\n",
       "      <td>360000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>37</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>5748</td>\n",
       "      <td>550</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.310088</td>\n",
       "      <td>1</td>\n",
       "      <td>0.371191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6581</th>\n",
       "      <td>22148</td>\n",
       "      <td>100000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3788</td>\n",
       "      <td>1263</td>\n",
       "      <td>2394</td>\n",
       "      <td>2839</td>\n",
       "      <td>32934</td>\n",
       "      <td>31056</td>\n",
       "      <td>0</td>\n",
       "      <td>2000</td>\n",
       "      <td>3000</td>\n",
       "      <td>31000</td>\n",
       "      <td>2000</td>\n",
       "      <td>5000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.310116</td>\n",
       "      <td>1</td>\n",
       "      <td>0.371232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3878</th>\n",
       "      <td>12983</td>\n",
       "      <td>180000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>28</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3775</td>\n",
       "      <td>5570</td>\n",
       "      <td>4722</td>\n",
       "      <td>8643</td>\n",
       "      <td>7783</td>\n",
       "      <td>12398</td>\n",
       "      <td>2005</td>\n",
       "      <td>31</td>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>5000</td>\n",
       "      <td>5000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311136</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1738</th>\n",
       "      <td>5848</td>\n",
       "      <td>120000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>married</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>200</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311165</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3896</th>\n",
       "      <td>13032</td>\n",
       "      <td>120000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>28</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311165</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1788</th>\n",
       "      <td>6010</td>\n",
       "      <td>80000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>42</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311165</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7766</th>\n",
       "      <td>26074</td>\n",
       "      <td>80000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311165</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1040</th>\n",
       "      <td>3469</td>\n",
       "      <td>100000</td>\n",
       "      <td>female</td>\n",
       "      <td>high school</td>\n",
       "      <td>married</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311165</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3560</th>\n",
       "      <td>11912</td>\n",
       "      <td>120000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>197</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>197</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311165</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2570</th>\n",
       "      <td>8625</td>\n",
       "      <td>80000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>300</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>300</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>500</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311165</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3227</th>\n",
       "      <td>10827</td>\n",
       "      <td>120000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>34</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311165</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2246</th>\n",
       "      <td>7517</td>\n",
       "      <td>80000</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>single</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311165</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3292</th>\n",
       "      <td>11040</td>\n",
       "      <td>110000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>married</td>\n",
       "      <td>48</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311165</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6807</th>\n",
       "      <td>22870</td>\n",
       "      <td>120000</td>\n",
       "      <td>female</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.311165</td>\n",
       "      <td>1</td>\n",
       "      <td>0.372754</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         ID  LIMIT_BAL     SEX        EDUCATION MARRIAGE  AGE  PAY_0  PAY_2  \\\n",
       "5869  19776     110000  female       university  married   42      0      0   \n",
       "2959   9916      20000  female  graduate school   single   24     -1     -1   \n",
       "3508  11755     160000  female       university   single   27      1     -2   \n",
       "321    1057     190000  female  graduate school   single   33      1      2   \n",
       "8353  28028     240000  female       university  married   32      1     -2   \n",
       "5993  20181     170000  female  graduate school  married   33      1     -2   \n",
       "1513   5066     200000  female  graduate school  married   32      1     -2   \n",
       "2531   8476     360000  female       university  married   51      1     -2   \n",
       "2389   8018     230000  female       university   single   28      0      0   \n",
       "3986  13358     100000  female  graduate school   single   25      1     -1   \n",
       "2881   9638      20000  female       university   single   22      0      0   \n",
       "434    1390      20000  female       university   single   22      0      0   \n",
       "2890   9660     140000  female  graduate school   single   29      1      2   \n",
       "3996  13390     340000  female      high school  married   36     -1      2   \n",
       "8420  28250     150000  female  graduate school  married   33      1     -1   \n",
       "1900   6369      40000  female  graduate school   single   26      0      0   \n",
       "7703  25869     350000  female       university  married   39      1     -2   \n",
       "8532  28613     360000  female      high school   single   39      1     -2   \n",
       "309    1012     410000  female  graduate school  married   31     -1     -1   \n",
       "5925  19928     330000  female       university  married   39      1     -1   \n",
       "1322   4397      80000  female  graduate school   single   25     -1     -1   \n",
       "7789  26144     300000  female       university   single   43      1     -2   \n",
       "854    2876     140000  female       university   single   26      1     -1   \n",
       "2220   7423      60000  female  graduate school   single   25      0      0   \n",
       "4462  14924     160000  female       university   single   43      1     -2   \n",
       "3654  12194     220000  female  graduate school  married   54      1     -2   \n",
       "3243  10884     160000  female  graduate school   single   30     -1      2   \n",
       "206     665      40000  female       university  married   32      0      0   \n",
       "4658  15646     130000  female      high school  married   42      1     -2   \n",
       "1306   4347      30000  female      high school  married   44     -1     -1   \n",
       "...     ...        ...     ...              ...      ...  ...    ...    ...   \n",
       "4238  14201     290000  female       university   single   26      1      2   \n",
       "1951   6527      20000  female  graduate school   single   24     -1      2   \n",
       "3820  12799     100000  female  graduate school   single   30      1      2   \n",
       "832    2784     280000  female  graduate school  married   47     -1      2   \n",
       "1680   5665      50000  female  graduate school   single   26      1     -2   \n",
       "2926   9803      20000  female       university   single   22      0      0   \n",
       "1903   6383      80000  female       university  married   30      1      2   \n",
       "420    1346     120000  female  graduate school   single   35      1      2   \n",
       "836    2806     120000  female  graduate school  married   39      1     -2   \n",
       "5058  17006     150000  female  graduate school   single   24      1     -1   \n",
       "4169  13971      50000  female       university   single   23      1     -1   \n",
       "586    1902     110000  female       university   single   42      1      2   \n",
       "5244  17657      20000  female      high school  married   44      0      0   \n",
       "4187  14025      50000  female  graduate school  married   50      1      2   \n",
       "8468  28386     100000  female      high school  married   35      1     -2   \n",
       "6790  22819     360000  female      high school   single   35     -1      2   \n",
       "7619  25590     360000  female       university  married   37     -1      2   \n",
       "6581  22148     100000  female  graduate school   single   29      1      2   \n",
       "3878  12983     180000  female  graduate school   single   28      1      2   \n",
       "1738   5848     120000  female  graduate school  married   30      1     -2   \n",
       "3896  13032     120000  female       university  married   28      1     -2   \n",
       "1788   6010      80000  female       university  married   42      1     -2   \n",
       "7766  26074      80000  female       university  married   35      1     -2   \n",
       "1040   3469     100000  female      high school  married   53      1     -2   \n",
       "3560  11912     120000  female       university  married   33      1     -2   \n",
       "2570   8625      80000  female  graduate school   single   25      1     -2   \n",
       "3227  10827     120000  female  graduate school   single   34      1     -2   \n",
       "2246   7517      80000  female  graduate school   single   52      1     -2   \n",
       "3292  11040     110000  female       university  married   48      1     -2   \n",
       "6807  22870     120000  female       university   single   29      1     -2   \n",
       "\n",
       "      PAY_3  PAY_4  PAY_5  PAY_6  BILL_AMT1  BILL_AMT2  BILL_AMT3  BILL_AMT4  \\\n",
       "5869      0      2      2      0       5174       6184       8383       8394   \n",
       "2959     -2     -2     -2     -2        379          0          0          0   \n",
       "3508     -1      0      0     -2          0          0       6204       6204   \n",
       "321       0      0      0      0     115056     108137      89722      69861   \n",
       "8353     -2     -1     -1     -1          0          0          0        262   \n",
       "5993     -2     -2     -1      0          0          0          0          0   \n",
       "1513     -2     -2     -2     -1          0          0          0          0   \n",
       "2531     -1     -1     -1     -1          0          0        390        540   \n",
       "2389      2      2      2      2     185399     198941     201358     204067   \n",
       "3986     -1     -1     -1     -2          0       3473          0       1012   \n",
       "2881      2      0      0      0       5473       7492       4988      11680   \n",
       "434       2      0      0      2      11192      15677      15133      15582   \n",
       "2890      0      0      0      0     144922     141289     137377      93261   \n",
       "3996      2     -1     -1      2       2566       1689        -11      35947   \n",
       "8420     -1     -1      0      0         -3        350          0        863   \n",
       "1900      0      2      2      2      20936      20647      21561      18438   \n",
       "7703     -2     -2     -2     -2          0          0          0          0   \n",
       "8532     -2     -2     -2     -2          0          0        271         -1   \n",
       "309       2      2     -2     -2       2744       2041        630       -736   \n",
       "5925     -1     -1      2     -1          0       1380          0        404   \n",
       "1322      2     -1      0     -1        390       5640        390        780   \n",
       "7789     -2     -1     -1     -1        499          0          0        507   \n",
       "854       0      0      0      0          0        811       1633       1666   \n",
       "2220      0      2      2      2      24066      24786      27050      26317   \n",
       "4462     -2     -2     -2     -1          0          0          0          0   \n",
       "3654     -1     -1     -1     -1          0          0        788          0   \n",
       "3243      2     -1     -1     -1       6548       3200          0       4775   \n",
       "206       0      2      2      2      24114      25417      28777      29107   \n",
       "4658     -2     -2     -2     -2          0          0          0          0   \n",
       "1306      2      0      0      0       1072       5122       4440       4050   \n",
       "...     ...    ...    ...    ...        ...        ...        ...        ...   \n",
       "4238      2      0      0      0      18099      18912      18249      20305   \n",
       "1951     -1      0      0     -1        632        316        948        632   \n",
       "3820      0      0      0      0     105033     102513     100693      71253   \n",
       "832       2     -1      0      0       2002       1311          0        430   \n",
       "1680     -2     -2     -1     -1         -1         -1         -1         -1   \n",
       "2926      0      2      2      2      15034      16285      20498      19509   \n",
       "1903      0      0      0      0      82891      80997      81380      75374   \n",
       "420       0      0      0      0      77785      75984      78109      79198   \n",
       "836      -2     -2     -1     -1          0          0          0          0   \n",
       "5058     -1     -1     -1      2       -200        200        200        200   \n",
       "4169     -1      2      2      0          3      34361      44804      44800   \n",
       "586       0      0      0      0      57046      51005      48572      49299   \n",
       "5244      2      2      2      0      20313      20111      18057      18536   \n",
       "4187      0      0      0      0      50175      48275      49646      44051   \n",
       "8468     -2     -2     -1     -1          0          0          0          0   \n",
       "6790      2     -2     -2     -2       2500       2500          0          0   \n",
       "7619      2     -2     -2     -2       5748        550          0          0   \n",
       "6581      0     -1      0      0       3788       1263       2394       2839   \n",
       "3878      2      0      0      0       3775       5570       4722       8643   \n",
       "1738     -2     -2     -2     -1          0          0          0          0   \n",
       "3896     -2     -2     -2     -2          0          0          0          0   \n",
       "1788     -2     -2     -2     -2          0          0          0          0   \n",
       "7766     -2     -2     -2     -2          0          0          0          0   \n",
       "1040     -2     -2     -2     -2          0          0          0          0   \n",
       "3560     -2     -2     -1     -1          0          0          0          0   \n",
       "2570     -1     -1     -2     -2          0          0        300          0   \n",
       "3227     -2     -2     -2     -2          0          0          0          0   \n",
       "2246     -2     -2     -2     -2          0          0          0          0   \n",
       "3292     -2     -2     -2     -2          0          0          0          0   \n",
       "6807     -2     -2     -2     -2          0          0          0          0   \n",
       "\n",
       "      BILL_AMT5  BILL_AMT6  PAY_AMT1  PAY_AMT2  PAY_AMT3  PAY_AMT4  PAY_AMT5  \\\n",
       "5869       7709       8084      1106      2393       300         0       500   \n",
       "2959          0          0         0         0         0         0         0   \n",
       "3508          0          0         0      6204         0         0         0   \n",
       "321       49894      49727         0      4413      2942      2000      1583   \n",
       "8353          0       1389         0         0       262         0      1389   \n",
       "5993        185        185         0         0         0       185         0   \n",
       "1513          0        900         0         0         0         0       900   \n",
       "2531        540        390         0       780       540       540       390   \n",
       "2389     200720     212403     18150      7200      7500         0     15079   \n",
       "3986          0          0      3473         0      1012         0         0   \n",
       "2881       6590          0      2300         0       102      1500         0   \n",
       "434       16677      16259      5000         0      1000      1500         0   \n",
       "2890      95368      97063        17      5547      3242      3512      3333   \n",
       "3996      83360      82516         8         0     35958     50000         0   \n",
       "8420        863          0       353         0       863         0         0   \n",
       "1900      20619      17582      3000      4000         0      5000         0   \n",
       "7703        590          0         0         0         0       590         0   \n",
       "8532        560          0         0       271         0       561         0   \n",
       "309       -1591      -2303      2051         5       266       266       253   \n",
       "5925        404      24420      1380         0       404         0     24420   \n",
       "1322        390        390      5250         0       780         0       390   \n",
       "7789          0        629         0         0       507         0       629   \n",
       "854        1700          0       811      1000        33        34         0   \n",
       "2220      28200      28762      1414      2678         0      2310      1173   \n",
       "4462          0        695         0         0         0         0       695   \n",
       "3654        349        600         0       788         0       349       600   \n",
       "3243        456          0         7         0      4775       456         0   \n",
       "206       29693      30456      2000      4110      1100      1200      1400   \n",
       "4658          0          0         0         0         0         0         0   \n",
       "1306       3660       3660      4440         0         0         0         0   \n",
       "...         ...        ...       ...       ...       ...       ...       ...   \n",
       "4238      26615      27063      1400         0      2305      6615      1896   \n",
       "1951        316       1096         0       948         0         0      1096   \n",
       "3820      70206      71060         0      3509      3047      3500      3000   \n",
       "832         430       -550         0         0       430         0       550   \n",
       "1680        349        350         0         0         0       350       351   \n",
       "2926      20038      18982      1500      4500         0      1600         0   \n",
       "1903      77158      78710         0      3299      2699      3000      2987   \n",
       "420       80865      82859         0      4000      2988      3100      3500   \n",
       "836         915          0         0         0         0       915         0   \n",
       "5058        550        550       600       200       200       550       200   \n",
       "4169      43576      44315     34361     10443      2001         1      2001   \n",
       "586       47126      48215         0      2100      2500      2000      2250   \n",
       "5244      11532      19450      3030         0      2175         0     10000   \n",
       "4187      45912      45383         0      7775      3000      3000      5000   \n",
       "8468        249        317         0         0         0       249       317   \n",
       "6790          0          0         0         0         0         0         0   \n",
       "7619          0          0         0         0         0         0         0   \n",
       "6581      32934      31056         0      2000      3000     31000      2000   \n",
       "3878       7783      12398      2005        31      5000      1000      5000   \n",
       "1738          0        150         0         0         0         0       150   \n",
       "3896          0          0         0         0         0         0         0   \n",
       "1788          0          0         0         0         0         0         0   \n",
       "7766          0          0         0         0         0         0         0   \n",
       "1040          0          0         0         0         0         0         0   \n",
       "3560        197          0         0         0         0       197         0   \n",
       "2570          0          0         0       300         0         0         0   \n",
       "3227          0          0         0         0         0         0         0   \n",
       "2246          0          0         0         0         0         0         0   \n",
       "3292          0          0         0         0         0         0         0   \n",
       "6807          0          0         0         0         0         0         0   \n",
       "\n",
       "      PAY_AMT6  DEFAULT_NEXT_MONTH  p_DEFAULT_NEXT_MONTH  \\\n",
       "5869       500                   0              0.270166   \n",
       "2959         0                   0              0.270243   \n",
       "3508         0                   0              0.270293   \n",
       "321       1500                   0              0.271024   \n",
       "8353         0                   0              0.271053   \n",
       "5993      4578                   0              0.271532   \n",
       "1513       880                   0              0.271769   \n",
       "2531       390                   0              0.271795   \n",
       "2389      8000                   0              0.271860   \n",
       "3986         0                   0              0.272039   \n",
       "2881         0                   0              0.272170   \n",
       "434        700                   0              0.272171   \n",
       "2890      4383                   0              0.272392   \n",
       "3996     10000                   0              0.272520   \n",
       "8420         0                   0              0.272878   \n",
       "1900      8000                   0              0.272988   \n",
       "7703         0                   0              0.273208   \n",
       "8532         0                   0              0.273208   \n",
       "309       6384                   0              0.273953   \n",
       "5925       480                   0              0.274621   \n",
       "1322       390                   0              0.274808   \n",
       "7789         0                   0              0.275279   \n",
       "854          0                   0              0.275537   \n",
       "2220         0                   0              0.275588   \n",
       "4462         0                   0              0.275657   \n",
       "3654         0                   0              0.275657   \n",
       "3243         0                   0              0.275847   \n",
       "206          0                   0              0.276142   \n",
       "4658      5000                   0              0.276634   \n",
       "1306         0                   0              0.276657   \n",
       "...        ...                 ...                   ...   \n",
       "4238      2358                   0              0.300226   \n",
       "1951         0                   0              0.301021   \n",
       "3820      2500                   0              0.302226   \n",
       "832       1304                   0              0.302589   \n",
       "1680      4076                   0              0.302610   \n",
       "2926      1000                   0              0.303381   \n",
       "1903      3100                   0              0.303567   \n",
       "420       6382                   0              0.303567   \n",
       "836          0                   0              0.304701   \n",
       "5058         0                   0              0.305159   \n",
       "4169      2001                   0              0.306716   \n",
       "586       1200                   0              0.307312   \n",
       "5244       931                   0              0.308034   \n",
       "4187      3000                   0              0.308502   \n",
       "8468         0                   0              0.309639   \n",
       "6790         0                   0              0.310088   \n",
       "7619         0                   0              0.310088   \n",
       "6581      5000                   0              0.310116   \n",
       "3878      5000                   0              0.311136   \n",
       "1738       200                   0              0.311165   \n",
       "3896         0                   0              0.311165   \n",
       "1788         0                   0              0.311165   \n",
       "7766         0                   0              0.311165   \n",
       "1040         0                   0              0.311165   \n",
       "3560         0                   0              0.311165   \n",
       "2570       500                   0              0.311165   \n",
       "3227         0                   0              0.311165   \n",
       "2246         0                   0              0.311165   \n",
       "3292         0                   0              0.311165   \n",
       "6807         0                   0              0.311165   \n",
       "\n",
       "      d_DEFAULT_NEXT_MONTH  r_DEFAULT_NEXT_MONTH  \n",
       "5869                     1              0.314938  \n",
       "2959                     1              0.315044  \n",
       "3508                     1              0.315112  \n",
       "321                      1              0.316114  \n",
       "8353                     1              0.316154  \n",
       "5993                     1              0.316812  \n",
       "1513                     1              0.317137  \n",
       "2531                     1              0.317172  \n",
       "2389                     1              0.317262  \n",
       "3986                     1              0.317508  \n",
       "2881                     1              0.317688  \n",
       "434                      1              0.317689  \n",
       "2890                     1              0.317993  \n",
       "3996                     1              0.318169  \n",
       "8420                     1              0.318661  \n",
       "1900                     1              0.318812  \n",
       "7703                     1              0.319115  \n",
       "8532                     1              0.319115  \n",
       "309                      1              0.320140  \n",
       "5925                     1              0.321061  \n",
       "1322                     1              0.321319  \n",
       "7789                     1              0.321969  \n",
       "854                      1              0.322325  \n",
       "2220                     1              0.322394  \n",
       "4462                     1              0.322491  \n",
       "3654                     1              0.322491  \n",
       "3243                     1              0.322752  \n",
       "206                      1              0.323160  \n",
       "4658                     1              0.323840  \n",
       "1306                     1              0.323871  \n",
       "...                    ...                   ...  \n",
       "4238                     1              0.356998  \n",
       "1951                     1              0.358135  \n",
       "3820                     1              0.359859  \n",
       "832                      1              0.360380  \n",
       "1680                     1              0.360411  \n",
       "2926                     1              0.361517  \n",
       "1903                     1              0.361784  \n",
       "420                      1              0.361784  \n",
       "836                      1              0.363414  \n",
       "5058                     1              0.364072  \n",
       "4169                     1              0.366315  \n",
       "586                      1              0.367176  \n",
       "5244                     1              0.368218  \n",
       "4187                     1              0.368895  \n",
       "8468                     1              0.370540  \n",
       "6790                     1              0.371191  \n",
       "7619                     1              0.371191  \n",
       "6581                     1              0.371232  \n",
       "3878                     1              0.372712  \n",
       "1738                     1              0.372754  \n",
       "3896                     1              0.372754  \n",
       "1788                     1              0.372754  \n",
       "7766                     1              0.372754  \n",
       "1040                     1              0.372754  \n",
       "3560                     1              0.372754  \n",
       "2570                     1              0.372754  \n",
       "3227                     1              0.372754  \n",
       "2246                     1              0.372754  \n",
       "3292                     1              0.372754  \n",
       "6807                     1              0.372754  \n",
       "\n",
       "[150 rows x 28 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_yhat_female_fp.sort_values(by='r_DEFAULT_NEXT_MONTH', ascending=True).head(n=150)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examing the low-residual false positives, it can be seen that perhaps the F1-selected cutoff is a bit too conservative. Many women just above the cutoff have missed 0-2 payments, and only been late 1-2 months on the few payments they missed, if any. This potential bias problem can be remediated by increasing the cutoff in cell 14."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shutdown H2O\n",
    "After using h2o, it's typically best to shut it down. However, before doing so, users should ensure that they have saved any h2o data structures, such as models and H2OFrames, or scoring artifacts, such as POJOs and MOJOs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Are you sure you want to shutdown the H2O instance running at http://localhost:54321 (Y/N)? y\n",
      "H2O session _sid_b933 closed.\n"
     ]
    }
   ],
   "source": [
    "# be careful, this can erase your work!\n",
    "h2o.cluster().shutdown(prompt=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook outlines a basic approach to DIA (and over-generalizes that phrase). In a complex, real-world machine learning project the hard-to-define phenomenas of sociological bias and unfairness can materialize in many ways and from many different sources. Although far from a flawless technique, the beauty of DIA is it is straightforward to implement, functions in a model-agnostic fashion on known labels and model predictions, and is applied in complex real-world fair lending situations, so it can probably be applied to your model too! \n",
    "\n",
    "Why risk being called out in the media for training an unfair model? Or why not investigate the monetary opportunity costs of type I errors and potential losses from type II errors? Why not do the right thing and investigate how your model treats people?"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}