t81_558_class_04_2_multi_class.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_2_multi_class.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 4: Training for Tabular Data**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Material\n",
    "\n",
    "* Part 4.1: Encoding a Feature Vector for Keras Deep Learning [[Video]](https://www.youtube.com/watch?v=Vxz-gfs9nMQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_1_feature_encode.ipynb)\n",
    "* **Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC** [[Video]](https://www.youtube.com/watch?v=-f3bg9dLMks&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_2_multi_class.ipynb)\n",
    "* Part 4.3: Keras Regression for Deep Neural Networks with RMSE [[Video]](https://www.youtube.com/watch?v=wNhBUC6X5-E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_3_regression.ipynb)\n",
    "* Part 4.4: Backpropagation, Nesterov Momentum, and ADAM Neural Network Training [[Video]](https://www.youtube.com/watch?v=VbDg8aBgpck&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_4_backprop.ipynb)\n",
    "* Part 4.5: Neural Network RMSE and Log Loss Error Calculation from Scratch [[Video]](https://www.youtube.com/watch?v=wmQX1t2PHJc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_5_rmse_logloss.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC\n",
    "\n",
    "* **Binary Classification** - Classification between two possibilities (positive and negative).  Common in medical testing, does the person have the disease (positive) or not (negative).\n",
    "* **Classification** - Classification between more than 2.  The iris dataset (3-way classification).\n",
    "* **Regression** - Numeric prediction.  How many MPG does a car get? (covered in next video)\n",
    "\n",
    "In this class session we will look at some visualizations for all three.\n",
    "\n",
    "\n",
    "It is important to evaluate the level of error in the results produced by a neural network.  In this part we will look at how to evaluate error for both classification and regression neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Binary classification is used to create a model that classifies between only two classes.  These two classes are often called \"positive\" and \"negative\".  Consider the following program that uses the [wcbreast_wdbc dataset](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/datasets_wcbc.ipynb) to classify if a breast tumor is cancerous (malignant) or not (benign).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>mean_radius</th>\n",
       "      <th>mean_texture</th>\n",
       "      <th>mean_perimeter</th>\n",
       "      <th>mean_area</th>\n",
       "      <th>mean_smoothness</th>\n",
       "      <th>mean_compactness</th>\n",
       "      <th>mean_concavity</th>\n",
       "      <th>mean_concave_points</th>\n",
       "      <th>...</th>\n",
       "      <th>worst_radius</th>\n",
       "      <th>worst_texture</th>\n",
       "      <th>worst_perimeter</th>\n",
       "      <th>worst_area</th>\n",
       "      <th>worst_smoothness</th>\n",
       "      <th>worst_compactness</th>\n",
       "      <th>worst_concavity</th>\n",
       "      <th>worst_concave_points</th>\n",
       "      <th>worst_symmetry</th>\n",
       "      <th>worst_fractal_dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>84300903</td>\n",
       "      <td>M</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84348301</td>\n",
       "      <td>M</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84358402</td>\n",
       "      <td>M</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id diagnosis  mean_radius  mean_texture  mean_perimeter  mean_area  \\\n",
       "0    842302         M        17.99         10.38          122.80     1001.0   \n",
       "1    842517         M        20.57         17.77          132.90     1326.0   \n",
       "2  84300903         M        19.69         21.25          130.00     1203.0   \n",
       "3  84348301         M        11.42         20.38           77.58      386.1   \n",
       "4  84358402         M        20.29         14.34          135.10     1297.0   \n",
       "\n",
       "   mean_smoothness  mean_compactness  mean_concavity  mean_concave_points  \\\n",
       "0          0.11840           0.27760          0.3001              0.14710   \n",
       "1          0.08474           0.07864          0.0869              0.07017   \n",
       "2          0.10960           0.15990          0.1974              0.12790   \n",
       "3          0.14250           0.28390          0.2414              0.10520   \n",
       "4          0.10030           0.13280          0.1980              0.10430   \n",
       "\n",
       "            ...             worst_radius  worst_texture  worst_perimeter  \\\n",
       "0           ...                    25.38          17.33           184.60   \n",
       "1           ...                    24.99          23.41           158.80   \n",
       "2           ...                    23.57          25.53           152.50   \n",
       "3           ...                    14.91          26.50            98.87   \n",
       "4           ...                    22.54          16.67           152.20   \n",
       "\n",
       "   worst_area  worst_smoothness  worst_compactness  worst_concavity  \\\n",
       "0      2019.0            0.1622             0.6656           0.7119   \n",
       "1      1956.0            0.1238             0.1866           0.2416   \n",
       "2      1709.0            0.1444             0.4245           0.4504   \n",
       "3       567.7            0.2098             0.8663           0.6869   \n",
       "4      1575.0            0.1374             0.2050           0.4000   \n",
       "\n",
       "   worst_concave_points  worst_symmetry  worst_fractal_dimension  \n",
       "0                0.2654          0.4601                  0.11890  \n",
       "1                0.1860          0.2750                  0.08902  \n",
       "2                0.2430          0.3613                  0.08758  \n",
       "3                0.2575          0.6638                  0.17300  \n",
       "4                0.1625          0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/wcbreast_wdbc.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "display(df[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Curves\n",
    "\n",
    "ROC curves can be a bit confusing.  However, they are very common.  It is important to know how to read them.  Even their name is confusing.  Do not worry about their name, it comes from electrical engineering (EE).\n",
    "\n",
    "Binary classification is common in medical testing.  Often you want to diagnose if someone has a disease.  This can lead to two types of errors, know as false positives and false negatives:\n",
    "\n",
    "* **False Positive** - Your test (neural network) indicated that the patient had the disease; however, the patient did not have the disease.\n",
    "* **False Negative** - Your test (neural network) indicated that the patient did not have the disease; however, the patient did have the disease.\n",
    "* **True Positive** - Your test (neural network) correctly identified that the patient had the disease.\n",
    "* **True Negative** - Your test (neural network) correctly identified that the patient did not have the disease.\n",
    "\n",
    "Types of errors:\n",
    "\n",
    "![Type of Error](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_4_errors.png \"Type of Error\")\n",
    "\n",
    "Neural networks classify in terms of probability of it being positive. However, at what probability do you give a positive result?  Is the cutoff 50%? 90%?  Where you set this cutoff is called the threshold.  Anything above the cutoff is positive, anything below is negative.  Setting this cutoff allows the model to be more sensitive or specific:\n",
    "\n",
    "More info on Sensitivity vs Specificity: [Khan Academy](https://www.youtube.com/watch?v=Z5TtopYX1Gc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAAEJCAYAAACjcV2kAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8FNX6x/HPSaeEXiUUkSJJgBiaUgTpCALSEgEpSlOvihcL1/bT67V7FVEpAqJIMAn9SgcRlN4MJYD0EhDpHULK/P44G0AFScLuntnd5/167SvLZnfmm8ny5MzZmWeUZVkIIYQwz890ACGEEJoUZCGEsAkpyEIIYRNSkIUQwiakIAshhE1IQRZCCJuQguwGSqlXlFLJSqlNSqkkpVQ9Jy57heNrBaVU9+ser62UGu6s9TiLUirDsQ02KqU2KKXqZ+M1592R7QbrfUMp9byJdf8dF7+f5iilCjnuP6OU2qaUilNKtVdKDXXWesSNBZgO4O2UUvcB7YBoy7JSlVLFgCBnLd+yrKyCVgHoDkxyPL4OWOes9TjRJcuyogCUUq2Ad4HGZiN5Dje8nx687p9PAs0ty0px/Pt/zlqPuDEZIbteaeC4ZVmpAJZlHbcs67BSqpZSaqlSar1Sar5SqjSAUmqJUup9pdQapdQOpVQjx+MRjseSHCOjyo7Hs0aP7wGNHN9/TinVRCk1Synlp5TalzXqcbxmp1KqpFKquFJqqlJqrePWwK1bBgoApxyZ8iulfnCMmjcrpTr8+ck3e45j72CbUmqMY+S4QCmVx/G9SkqpRdeNyO9yPP6C42fepJR687p1vOLY7suAqu7YCDl0s/fTPqXUB47tskYpVQngZr9jx7Yc73j+JqVUZ8fj+5RSxZRSo4CKwFzH+6mPUupzx3NKKqWmO7bpxuzs5YhssixLbi68AfmBJGAHMAI9GgwEVgDFHc+JAb5y3F8C/Ndx/0FgkeP+Z0APx/0gII/j/nnH1ybArOvWe/XfwKdAX8f9etctcxLQ0HG/HLDNDdsjw7E9tgNngFqOxwOAAo77xYBdgPrTz3jD56D3DtKBKMf3EoGejvurgYcd90OAvEBL4EvHa/2AWcD9QC1gs+M5BRzLf970e+hW7yfH4/uAVxz3e133u7/h7xh4Hxh23XILX7ecYje43wf43HE/ARjsuO8PFDS9XbzlJlMWLmZZ1nmlVC2gEfAA+s38HyASWKiUAv2m/u26l01zfF2PLjYAK4FXlFJhwDTLsnbmIEYC8DowHoh1/BugORDuyABQQCmV37IsV87ZXj9lcR8wQSkViS6O7yil7gcygTJASeDIda+92XMA9lqWleS4vx6ooJQKBcpYljUdwLKsy471tkQX5V8cz88PVAZCgemWZV10PM92u+g3ej9dN7f73XVfP3Hcv+Hv2PF47HXLPZWDGE3RRR/LsjLQf1iFE0hBdgPHm3YJsEQptRl4Cki2LOu+m7wk1fE1A8fvyLKsSUqp1UBbYI5SaqBlWYuzGWElUEkpVRzoiP6DAHp0eG9WoXI3y7JWOuZAi6P3BoqjR8xpSql96BHt9Xr8zXNSr3teBpDnb1atgHctyxr9hweVGpzbn8WdbvB+6p31reuf5vh6w9/xdQVa2IjMIbuYUqpq1nyvQxSwDSjuGCGilApUSkXcYjkVgT2WZQ0HZgI1/vSUc+gR3l9Yet9yOvAxepf1hONbC4Cnr1tHVLZ/MCdQSt2N3js4ARQEjjoK7QNA+Ru8JDvPucqyrHNAilKqo2N9wUqpvMB84DHHSBGlVBmlVAngJ6CjUiqPY3T9kHN+Uue5yftpv+N+zHVfVzru3+x3vBA9MMh6vHAOYvwAPOF4nb9SqmAOXiv+hhRk18sPfKOU2qqU2gSEo6cPugDvK6U2oucEb/XBSDdgi1IqCT3dMeFP398EZDg+ZHnuBq9PAHpybboC4BmgtuNDna3AoBz+bLmRx/HBY5IjS2/HiC/OkWUzend4+w1em53n/NmjwDOObb8CKGVZ1gL03OpKx7KmAKGWZW1wZNoIzAXW3s4P6iI3ej+94fheYcdjzwJZ74Gb/Y7/43j+Fsd78IEcZHgWeMCx7dY7MggnyPrQRAjhwRzTN7UtyzpuOovIPRkhCyGETcgIWQghbEJGyEIIYRNSkIUQwiakIAshhE1IQRZCCJuQgiyEEDaRo1OnixUrZlWoUMFFUURO/PrrrwBUrWrHhmTuJdviGtkW9rR+/frjlmUVv9XzclSQK1SowLp1dmyx63uaNGkCwJIlS4zmsAPZFtfItrAnpdT+Wz9LpiyEEMI2pCALIYRNSEEWQgibkH7IQoir0tLSSElJ4fJlIy2yPV5ISAhhYWEEBgbm6vVSkIUQV6WkpBAaGkqFChWkiX0OWZbFiRMnSElJ4c4778zVMmTKQghx1eXLlylatKgU41xQSlG0aNHb2ruQgiyE+AMpxrl3u9tOCrIQwlaUUgwZMuTqvz/66CPeeOMNp6/nnXfe+cO/69e/1UV7XE8KshDCVoKDg5k2bRrHj7v24id/LsgrVqxw6fqyQwqyEMJWAgICGDBgAJ988slfvnfs2DE6d+5MnTp1qFOnDsuXL7/6eIsWLYiIiKBfv36UL1/+akHv2LEjtWrVIiIigi+//BKAoUOHcunSJaKioujRowcA+fPnByA2NpbZs2dfXWefPn2YMmUKGRkZvPDCC9SpU4caNWowevQfLlruFFKQhRC289RTTxEXF8eZM2f+8Pizzz7Lc889x9q1a5k6dSr9+vUD4M0336Rp06YkJyfTpUsXDhw4cPU1X331FevXr2fdunUMHz6cEydO8N5775EnTx6SkpKIi4v7wzpiYmJITEwE4MqVK/zwww+0bduWcePGUbBgQdauXcvatWsZM2YMe/fuderPLYe9CSFuaPC8wSQdSXLqMqNKRTGs9bBbPq9AgQL06tWL4cOHkydPnquPL1q0iK1bt17999mzZzl//jzLli1j+vTpALRu3ZrChQtffc7w4cOvfu/gwYPs3LmTokWL3nTdbdq04dlnnyU1NZV58+Zx//33kydPHhYsWMCmTZuYMmUKAGfOnGHnzp25PsTtRqQgCyFsafDgwURHR9O3b9+rj2VmZrJq1SpCQkKytYwlS5awaNEiVq5cSd68eWnSpMktD0sLCQmhSZMmzJ8/n4SEBGJjYwF9nPFnn31Gq1atcv9D3YIUZCHEDWVnJOtKRYoUoVu3bowbN47HHnsMgJYtW/LZZ5/xwgsvAJCUlERUVBQNGjQgMTGRl156iQULFnDq1ClAj2ILFy5M3rx52b59O6tWrbq6/MDAQNLS0m54Vl1MTAxjx45l3bp1fP311wC0atWKkSNH0rRpUwIDA9mxYwdlypQhX758TvuZZQ5ZCGFbQ4YM+cPRFsOHD2fdunXUqFGD8PBwRo0aBcD//d//sWDBAiIjI5k8eTKlSpUiNDSU1q1bk56eTrVq1Rg6dCj33nvv1WUNGDCAGjVqXP1Q73otW7Zk6dKlNG/enKCgIAD69etHeHg40dHRREZGMnDgQNLT05368yrLsrL95Nq1a1vSD9kepO/tNbItrrndbbFt2zaqVavmvEBukpqair+/PwEBAaxcuZInnniCpCTnzn9n1422oVJqvWVZtW/1WpmyEEJ4vAMHDtCtWzcyMzMJCgpizJgxpiPlihRkIYTHq1y5Mr/88ovpGLdN5pCFEMImpCALIYRNSEEWQgibkIIshBA2IQVZCGEr/v7+REVFERkZSdeuXbl48WKOl9GvX7+rp1jbsc3mzUhBFkLYSlbTny1bthAUFHT15I+cGDt2LOHh4YA922zejBRkIYRtNWrUiF27dgHw8ccfExkZSWRkJMOG6dO6L1y4QNu2balZsyaRkZEkJCQA+gSZdevW2bbN5s3IcchCCFtKT09n7ty5tG7dmvXr1zN+/HhWr16NZVnUq1ePxo0bs2fPHu64446rhfXP7Trfe+89Pv/88xuetZfVZrNt27ZX22yOHDnyD202U1NTadCgAS1btnRqV7ebkYIshLixwYPB2acfR0XBsL9vWpQ1ogU9Qn788ccZOXIkDz/88NVGPp06deLnn3+mdevWDBkyhJdeeol27drRqFGjbEcx2WbzZqQgCyFsJWsOOTuqVKnChg0bmDNnDq+++irNmjXj9ddfz9ZrTbbZvBkpyEKIG7vFSNadGjVqRJ8+fRg6dCiWZTF9+nS+/fZbDh8+TJEiRejZsyeFChVi7Nixf3mtHdts3owUZCGE7UVHR9OnTx/q1q0L6MPa7rnnHubPn88LL7yAn58fgYGBjBw58i+vzWqzGR0d/ZfLNbVs2ZJHH32UDh06/KHN5r59+4iOjsayLIoXL86MGTNc/0Mi7Tc9lrScvEa2xTW+2n7TTm6n/aYc9iaEEDYhBVkIIWxCCrIQQtiEFGQhxB/k5HMl8Ue3u+2kIAshrgoJCeHEiRNSlHPBsixOnDhBSEhIrpchh70JIa4KCwsjJSWFY8eOmY7ikUJCQggLC8v166UgCyGuCgwMdMspwuLGZMpCCCFsQgqyEELYhBRkIYSwCSnIQghhE1KQhRDCJqQgCyGETUhBFkIIm5CCLIQQNiEFWQghbEIKshBC2IQUZCGEsAkpyEIIYRNSkIUQwiakIHuYU5dOMW/XPPad3sfuk7uJ3xLP/tP7pX+trzt+HGbNgr17Yc8emDoVDh82nUrkkLTf9BApZ1N4as5T/O/X/+kHzoBC8cjURwCoVqwao9uNplH5RgZTCrfbtQsGDoTFi689phR06aLv16oFX34J0dFm8okckRGyzVmWxbgN44gYEcHC3Qt5pdEr/NDrBxqWbUijco1Y138dn7X5jMvpl2n8dWOemfsMF65cMB1buFpGBnzyCdSoAevXw1tvwdKl0KgRNGwIq1bBf/+rR8l168Krr0JqqunU4hZkhGxjGZkZ9JjWg4TkBBqXb8y49uO4q8hdAPzb798A1LqjFrXuqEWfqD68/MPLfLbmM+bvns+S3ksoHVraZHzhKpcvQ4cOsGABtGsHo0ZBmTL6e36OMVa9evrWty889xy8/TbMmQM//ACFC5vLLv6WjJBtyrIsBs4aSEJyAm83fZvFvRdfLcY3kj8oP8PbDGdxr8UcOnuIVhNbcerSKTcmFm6Rng7du+tiPHIk/O9/14rxjRQuDF9/DTNmQHKyLuAXL7otrsgZKcg2NXTRUMb9Mo5XG73Ky41exk9l71f1wJ0PMDN2Jr+e+JV237WT6QtvYlkwaBBMnw6ffqrvK5W913boAJMm6amMLl3gyhXXZhW5IgXZhj5Z+QkfrPiAJ2o/wb8f+HeOX9+sYjO+6/wdq1JW0XVyVzKtTBekFG736qswbhy89ho880zOX9+5s57emDsXHntMF3hhK1KQbeaX337hxUUv8vDdD/NZm89Q2R0B/Umnap344sEvmLtrLp+u+tTJKYXb/fgjvPMO9OsHb76Z++X07w///jfExcHEic7LJ5xCCrKNpKan0mtGL4rnLc7Y9mPx9/O/reUNrDWQ9lXb868f/sX249udlFK43dmz+sO5ypX1VEUu/0hf9fLL0KABPP00pKQ4J6NwCinINvLm0jfZcnQLYx4aQ5E8RW57eUopRrcbTb6gfPSe0Zv0zHQnpBRu9/zzcPCg/nAub97bX56/v15WWpoeccvUhW1IQbaJ1SmreX/5+zwW9Rhtq7R12nJL5S/FiAdHsObQGj5Y/oHTlivcZN48GDNGF+X69Z233EqV4IMPYP58GDvWecsVt0UKsg1kZGbQ//v+lAktw8etPnb68mMiY+gW0Y03lrzB7pO7nb584SKXLsGAARAefnvzxjfzxBPQrBkMGQK//+785Ysck4JsAxM2TmDz0c183OpjCoYUdMk6hrUaRqB/IK8sfsUlyxcu8Nlneqriiy8gJMT5y/fz08u+eFGf6SeMk4Js2MW0i7z242vUK1OPztU6u2w9pUNLM+S+ISQkJ7D20FqXrUc4yYkT+qiKtm2hSRPXradqVT0KHz0adu503XpEtkhBNmz46uEcOneID1p8kOtD3LLrhfovUDxvcV5c9KJ0h7O7d96Bc+fgvfdcv67/+z8IDtZHXwijpCAbdPzicd5d9i4PVXmI+8vf7/L1hQaH8nrj11mybwlzd811+fpELu3bB59/Dr17Q2Sk69dXsqT+0HDKFFi92vXrEzclBdmgt396m/NXzvNeczeMghwG1BpApSKVeGnRS2RkZrhtvSIHXntNz+/+O+dnaebakCG6ML/4ohwGZ5AUZEN+P/87I9eNpFfNXoQXD3fbeoP8g/jPA/9hy9EtTN8+3W3rFdm0c6c+i+7ppyEszH3rDQ3Vfwh++knfhBFSkA35bM1nXMm4wtAGQ92+7i7hXahUpBIfLP9A5pLt5r//haAg+Oc/3b/uxx6DEiXg/ffdv24BSEE24lzqOb5Y+wUd7+5I1WJV3b5+fz9/nr/vedYeXsuSfUvcvn5xE0eO6DPoeveGUqXcv/48eXTTorlzYdMm969fSEE2YcyGMZy+fJqXGrxkLEPvqN6UzFeS95fLaMg2hg/XbTGff95chiefhHz59Fl8wu2kILvZlYwrfLLqExqXb0y9sHrGcoQEhPBsvWeZv3s+SUeSjOUQDmfPwogRukVm5crmchQurI9Ljo+H/fvN5fBRUpDd7LvN35FyNsXo6DjLE3WeIDQoVHpc2MGXX8KZM/CS+fcFzz2nO8p97PzT+MXfk4LsRpZl8dHKj6heojqtK7U2HYdCIYUYWEtfJmr/aRkNGZOWBsOGQdOmULu26TRQtiz06KGbDp08aTqNT5GC7EY/7f+JLUe3MPjewS4/Ky+7nq73NACj1482nMSHzZwJhw7B4MGmk1zz3HO6x8U335hO4lOkILvRyHUjKRRSiNjIWNNRripXsBztqrRj3C/jSE2Xy8QbMXIklC8PDz5oOsk1NWvCfffpbJlyCTB3kYLsJkfOH2Hqtqn0jepL3kAnNBl3oidrP8nRC0eZtm2a6Si+Z/t2WLwYBg7UjePt5Mkn9YkqixebTuIzpCC7ydgNY0nPTGdQ7UGmo/xFi7tacFfhuxixboTpKL5n1CgIDITHHzed5K+6dIFixfTRH8ItpCC7QXpmOl+u/5LmFZtTpWgV03H+wk/5Maj2IJYdWMbm3zebjuM7LlzQJ4J06aLPkLObkBB99t7//ifX3nMTKchuMHvHbA6ePciTtZ80HeWm+kb1Jdg/mJHrRpqO4jvi4/Whbk88YTrJzQ0cqOeQx4wxncQnSEF2gxHrRlAmtAwPVX3IdJSbKpq3KDGRMXy76VvOpp41Hcc3jBih22s2bGg6yc1VrAht2ujjpNPSTKfxelKQXWz/6f0s3L2QftH9CPALMB3nbw2qNYjzV86TmJxoOor3++UX2LBBj0BtcgjkTQ0apPtszJljOonXk4LsYt9s/AYLiz5RfUxHuaV7w+6latGqjE8abzqK9xs/Xnd1697ddJJba9NG90oeL+8LV5OC7EKZViZfJ31N0zubUqFQBdNxbkkpRd+ovqw4uIJfj/9qOo73Sk3VPY87doQiRUynubWAAHj0UZg9G44eNZ3Gq0lBdqGf9v/E3tN76RvV13SUbHu05qP4KT++2ShnaLnM99/rU5L7es77gr59IT0dJk40ncSrSUF2ofFJ4ykQXIBO1TqZjpJtd4TeQetKrZmwcYJc4slVxo+HMmWgRQvTSbIvPBzq1tXZ5aIGLiMF2UXOpZ5jytYpxETE2O7MvFvpG9WXQ+cOsXDPQtNRvM/hwzBvHvTqZb8z826lb1/YsgXWrzedxGtJQXaRxORELqZd9KjpiiwPVXmIInmKyId7rvDtt/q43j59TCfJudhYfbKIfLjnMlKQXWR80niqFq3KvWH3mo6SY8EBwfSs3pMZ22dw8pK0X3Qay9Jn5jVsCFXsd8bmLRUqBJ06waRJcPmy6TReSQqyC+w5tYflB5fTu2Zv27TZzKk+UX24knGFycmTTUfxHuvW6WZCvXubTpJ7ffrA6dP6iAvhdFKQXSBuUxwAPWr0MJwk96JKRRFePJyJm+VTdaeZOBGCg3XvCk/VtKm+AKscbeESUpCdzLIs4jbHcX/5+ylXsJzpOLmmlKJH9R4sO7CMfaf3mY7j+dLTde+Kdu30rr+n8veHRx7RZ+3J1UScTgqyk63/bT2/nviVntV7mo5y27pX12eRTdo8yXASL7BokT6poqfnvy/o2VNfHXvKFNNJvI4UZCeL2xRHkH8QXcI9eLfUoUKhCjQs15CJmyZiybGntycuTo+M27QxneT23XMP3H23/pmEU0lBdqL0zHTik+NpW7kthfMUNh3HKXpU78G249tIOpJkOornunABpk+Hrl31HLKnU0pfBPWnn+DAAdNpvIoUZCdavHcxR84foWcNL9gtdega3pVAv0DiNstoKNdmztRF2RumK7JkNUWaJNNZziQF2YniNsdRMLggD1a20cUqb1PRvEVpU7kNkzZPklOpcysuDsqWtXff45yqWBHq19dHW8h0ltNIQXaSi2kXmbZtGl3CuxASEGI6jlP1rN6T387/xpJ9S0xH8TzHjsH8+XpE6edl/9169oTkZNi0yXQSr+Fl7xBz5uycw/kr568emeBN2lVpR/6g/MRviTcdxfNMnQoZGZ7R9zinunTRh8ElJJhO4jWkIDtJ/JZ4SuYrSePyjU1Hcbo8gXnoeHdHpm6bypWMK6bjeJb4eN0prXp100mcr3hxaN5c/4wybeEUUpCd4GzqWWbtmEW3iG74+3lYB69sio2I5dTlUyzcLR3gsu3QIX0kQmys/S/TlFuxsbB3L6xZYzqJV5CC7AQzt88kNSOV2MhY01FcpsVdLSgcUpj4ZJm2yLbERD1yjIkxncR1OnbUl6KKl/eFM0hBdoL45HjKFyzPfWH3mY7iMkH+QXSu1pkZ22dwKe2S6TieIT4eoqM9s7NbdhUqBA8+qOeRM+QonNslBfk2nbh4ggW7FxATEeOxnd2yKzYylvNXzjNnp1x9+Jb27NG78bHeu9d0VWws/PYbLFtmOonHk4J8m6Ztm0Z6ZrpXT1dkaVKhCSXzlZRpi+zIOvKgWzezOdyhXTvIm1emLZxACvJtik+Op0rRKkSVijIdxeX8/fzpGt6VWTtmcS71nOk49hYfr0+cKF/edBLXy5cP2rfXzYbS0kyn8WhSkG/DkfNHWLJviU9MV2R5pPojXE6/zMxfZ5qOYl/btumTJXxhuiJLbCwcPw4//GA6iUeTgnwbpmydQqaVSUyEF3+K/if3ht1LWIEwEpMTTUexr4QEfZibJzeiz6lWraBAAX1kicg1Kci3ITE5kYjiEUSUiDAdxW38lB/dwrsxb9c8Tl8+bTqO/ViWLkqNG0Pp0qbTuE9ICHTooLvaXZGTh3JLCnIuHTp7iGUHlvnU6DhLTGQMaZlpzNwu0xZ/sWWLnrLw5mOPbyYmRl9vb6GcPJRbUpBzacrWKVhYdIvwgU/R/6TOHXWoUKgCCcnSw+AvEhN1E6FOnUwncb8WLfRxyTJtkWtSkHMpITmBmiVrUrVYVdNR3E4pRbfwbizcs5ATF0+YjmMflqXnj5s2hRIlTKdxv6AgePhhmDEDLl82ncYjSUHOhQNnDrAyZaVPjo6zdIvoRnpmOjO2zzAdxT42boSdO33j2OOb6dYNzp6FBQtMJ/FIUpBzYXLyZACfLsjRpaO5q/BdMm1xvYQE3Y7SF6crsjRrBkWKSEvOXJKCnAuJWxOJLh1NpSKVTEcxRilFt4huLN67mGMXjpmOY17W0RXNm0PRoqbTmBMYqP8g/e9/cEl6nuSUFOQc2ntqL2sOraFbuO+OjrPERMSQYWUwbds001HMW79e96/w5emKLDExcP48zJ1rOonHkYKcQ1knRPjydEWWGiVrUKVoFZm2AL2LHhioP9TydU2a6Ob1Mm2RY1KQcyhxayJ1y9TlzsJ3mo5inFKKmIgYlu5fypHzR0zHMSdruqJlSyhc2HQa8wICoHNnmDVLX21bZJsU5BzYdXIXG37b4JMng9xMTEQMmVYmU7dONR3FnNWr4cAB3zwZ5GZiYuDiRZg923QSjyIFOQeypiu6hnc1nMQ+IkpEEF48nMStPnwyQGKiPga3fXvTSeyjUSMoVUpOEskhKcg5kJCcQP2y9SlbsKzpKLYSExHDz/t/5vC5w6ajuF9mpi46bdpAwYKm09iHv79urjR7NpyTVq3ZJQU5m7Yf386m3zfJ0RU30C2iGxbW1eOzfcqKFfpipnJ0xV/FxOgz9r7/3nQSjyEFOZsSkxNRKLqE+1BLxWy6u9jd1ChZwzenLRITdaezhx4yncR+6teHMmVk2iIHpCBnU0JyAg3LNaRMgTKmo9hSt/BurDi4goNnDpqO4j4ZGTB5sr7IZ2io6TT24+cHXbvq45HPnDGdxiNIQc6GLUe3sPXYVjm64m/EROpt41ON63/+GY4ckaMr/k5MjO6PPFNatWaHFORsSNiSgJ/yk+mKv1GpSCVqla7lWyeJJCTo68m1bWs6iX3Vq6evKygniWSLFORbsCyL+OR4mt7ZlJL5S5qOY2uxkbGsPbyW3Sd3m47iemlp+qKe7dvroixuTCk9Sl6wAE5Iq9ZbkYJ8Cxt+28Cuk7uIjfChC1bmUtbp5D4xSl68WF/U05cuZJpbsbGQng7TpOfJrUhBvoX4LfEE+gXycDXpUXAr5QqWo0HZBsRviTcdxfXi4/Vxx61amU5if1FRUKWK3mbib0lB/huZViYJyQm0qtSKInmKmI7jEWIjY9l8dDPJR5NNR3Gd1FQ92uvUCYKDTaexP6X0KPnHH+G330ynsTUpyH9j5cGVHDx7UKYrcqBLeBf8lJ93T1vMm6eviiHTFdkXE6ObME2ZYjqJrUlB/hvxW+IJCQihfVXpUZBdpfKX4oEKD5CQnIBlWabjuEZ8PBQrpq+dJ7InPBxq1JBpi1uQgnwT6ZnpJG5NpG3ltoQGy0H/ORETEcOOEztIOpJkOorzXbigr4bRpYtuMymyLzZWn2q+f7/pJLYlBfkmluxbwtELR4mNlN3SnOpUrRMBfgFM2jzJdBTn+/6tA0LCAAAYzElEQVR73VZSpityLusEGhkl35QU5JuI2xxHaFAobSvLQf85VTRvUVpXak18cjyZVqbpOM4VFwdhYbq9pMiZihX1iSKTvPAPtZNIQb6BS2mXmLp1Kp3DO5MnMI/pOB6pR/UepJxN4af9P5mO4jzHj+sP9B55RPdpEDnXowds2gRbtphOYkvyrrqB2Ttnc+7KOXpU72E6isdqX7U9+QLzEbcpznQU55k8WZ/g0EPeF7nWrZvulSyj5BuSgnwDcZvjrh4tIHInb2BeHq72MFO2TSE1PdV0HOeYNOna0QIid0qWhObN9bbM9LLpLCeQgvwnpy6dYs7OOcRGxOLv5286jkfrUb0Hpy+fZu4uL7gc/P79sGyZHh0rZTqNZ+vRQ2/PlStNJ7EdKch/MnXbVK5kXKFHDdktvV3NKzanRL4SxG32gmmLrF3sRx4xm8MbdOwIefLoD0jFH0hB/pO4zXFUKVqFWqVrmY7i8QL8AoiJiOH7X7/nzGUPblBuWbp41K8Pd95pOo3nCw2FDh30lUTS0kynsRUpyNdJOZvC0n1L6R7ZHSW7pU7RvXp3UjNSmbbNgzt9bd4MycnQvbvpJN6je3fdjnP+fNNJbEUK8nXiNsVhYcl0hRPVK1OPSkUq8e2mb01Hyb0JE/RZeXIhU+dp1Uqffv6tB78vXEAKsoNlWXyz8RsalG1ApSKVTMfxGkopetXoxY/7fmT/aQ88ZTY9HSZO1FcFKV7cdBrvERSk5+NnzoRTp0ynsQ0pyA7rDq9j2/Ft9KrZy3QUr/NozUcBPHOUvGAB/P479O5tOon36dVLtzKVq1JfJQXZYcLGCQT7B1+96oVwngqFKtC4fGMmbJzgeR3gJkyAIkX0laWFc9WqpY/rnjDBdBLbkIIMXMm4wndbvqPj3R0pFFLIdByv1Ltmb3ae3MmqlFWmo2Tf6dMwY4betZZG9M6nlN7zWLECdu40ncYWpCADs3fM5sSlE/SuKbulrtIlvAt5AvLwzcZvTEfJvsREvUst0xWu07On7gsio2RACjIAEzZNoFT+UrS4q4XpKF4rNDiUTtU6kZCcwOX0y6bjZM+ECVCtGtSubTqJ97rjDn0q9bffyqnUSEHm+MXjzN4xmx7VexDgJw3HXal3zd6cvnya73/93nSUW9u1C5Yv16NjOSbdtXr31qdS/+RFnQFzyecL8sRNE0nLTJPpCjdoemdTwgqE8VXSV6aj3Nr48XpXumdP00m8X8eOUKAAfOUB7wsX8+mCbFkWYzeMpW6ZulQvWd10HK/n7+dP36i+zN81nwNnDpiOc3Pp6bogt2kDZcqYTuP98ubVZ+5Nnqw/SPVhPl2QV6WsIvlYMv2j+5uO4jMeu+cxAMb/Mt5wkr8xd66+XH1/eV+4Tf/+cPmyzzcc8umCPHbDWPIF5iMmIsZ0FJ9RoVAFWtzVgq+SviIjM8N0nBsbOxZKlZJjj90pOhruuQfGjNHNnHyUzxbks6lniU+O55HIR+Sq0m7WP7o/B84cYOGehaaj/NXhwzB7NvTpA4GBptP4lv79YeNG2LDBdBJjfLYgx2+J52LaRfpF9zMdxee0r9qeYnmLMXbDWNNR/urrryEjAx5/3HQS3/PII7pP8pgxppMY47MFeeyGsUSWiKRumbqmo/icIP8getfszcxfZ3L0wlHTca7JzIRx4+CBB6CSNJhyu0KFoGtXfTGACxdMpzHCJwvyxiMbWXt4Lf2j+0vfY0P6RfcjPTOdb5JsdObejz/Cnj3QT/aajOnfH86d89mGQz5ZkEesHUFIQAg9a8gxpqbcXexu7i9/P6PWjyLTsskZWiNGQNGi8PDDppP4rgYNdMOhESNMJzHC5wry6cunmbh5It0ju1MkTxHTcXzaU3WeYs+pPczbNc90FDh4UPfmffxxPY8pzFAKnnwS1q2DNWtMp3E7nyvIXyd9zcW0izxV9ynTUXzew3c/TOn8pfl8zeemo8Do0XoO+YknTCcRvXrp6+59boP3hZv5VEHOtDIZsXYE94XdR3TpaNNxfF6gfyADaw1k7q657Dq5y1yQ1FT9yf5DD0GFCuZyCC00VBflhAQ4dsx0GrfyqYK8cPdCdp7cyT/q/sN0FOEwoNYAAvwCGLl2pLkQU6bA0aPwlOw12cZTT8GVK/okHR/iUwX5i7VfUCJfCTpX62w6inAoHVqaztU681XSV1y4YuhQpy++gCpVdBtIYQ/VqkHTpjBqlO4t4iN8piDvPbWXWTtmMSB6AMEBcvUHO/lH3X9w+vJpJm2e5P6Vb9gAK1fqEZmfz/x38Az/+AccOACzZplO4jY+8w4cvno4/n7+DKw90HQU8ScNyjYgqlQUw1YPc/8hcJ98Avnzy1VB7Oihh6BcOf078hE+UZBPXTrFmA1jiI2MJaxAmOk44k+UUgy5bwhbj2117yFwBw9CfLw+GaFgQfetV2RPQAAMHqwb1/vIIXA+UZBHrx/NhbQLPH/f86ajiJuIiYghrEAYH6740H0r/fRT3Vns2Wfdt06RM/366T+WH31kOolbeH1BTk1P5dPVn9KiYgtqlqppOo64iUD/QAbXG8ySfUtYd3id61d45gx8+SXExED58q5fn8id0FAYNAimTtWntXs5ry/IkzZP4sj5I7xQ/wXTUcQt9K/VnwLBBfhohRtGQ6NH654Jz8tek+098wz4+8PHH5tO4nJeXZAzrUw+WvkRNUvWpHlFOaTJ7goEF2BQrUFM3jqZvaf2um5FV67o6YpmzXRTdGFvd9yhr2341Vdw/LjpNC7l1QV5zs45bD22lefrPy9d3TzEM/WewV/58/FKF46G4uJ0I3oZHXuOIUPg0iV9zLgX89qCbFkW/176byoUqiCXaPIgZQqUoVfNXozZMIbD5w47fwXp6fD22xAVBa1aOX/5wjUiIqB9exg2TM//eymvLchzd81l7eG1vNroVQL95VI8nuTlRi+TYWXw/rL3nb/wuDjYvRveeEN3FhOe4//+T1+Vevhw00lcxisLsmVZvLHkDSoUqkCvmr1MxxE5VLFwRXrX7M3o9aOdO0pOT4e33tKj4/btnbdc4R7R0fr39vHHXjtK9sqCLKNjz+eSUbKMjj2fl4+Sva4gy+jYOzh9lCyjY+/g5aNkryvIs3fOltGxl8gaJb/787u3v7Bvv5XRsbfIGiV/+qnpJE7nVQU5PTOdlxa9RKUilWR07AUqFq7IY1GPMWr9KHae2Jn7BV28CK+/DrVry+jYG0RH6+sefvgh/P676TRO5VUFefwv49l6bCvvN39fRsde4o0mbxDsH8y/fvhX7hcybBikpOh+CDI69g7vvQeXL+s9Hi/iNQX5/JXzvPbjazQo24CH75arBnuL0qGlebHBi0zdNpXlB5bnfAG//w7vvgsdOkDjxs4PKMyoUkVf/3DMGNi61XQap/Gagvzh8g/5/cLv/Lflf+WsPC8z5L4h3BF6B0MWDMGyrJy9+M039Ujqgw9cE06Y8/rrkC8fvPSS6SRO4xUF+dDZQ3y44kNiImKoF1bPdBzhZPmC8vGfB/7D6kOrmbx1cvZfuG2b7ug2aJAeUQnvUqwYvPKKvqLI4sWm0ziFVxTkFxe9SIaVwTvN3jEdRbhIr5q9qFmyJs8veJ7zV87f+gVZfY7z5dMjKeGdnnlGt0999llISzOd5rZ5fEFesHsBkzZP4l8N/0XFwhVNxxEu4u/nzxcPfsHBswd5Y8kbt37Bd9/BwoW6b0Xx4i7PJwwJCdEniWzZ4hXtOT26IF9Ku8STs5+kcpHKDG041HQc4WINyjWgf3R/hq0axsYjG2/+xFOn4LnnoE4d/cGP8G7t20PHjvrzgr0ubNvqBh5dkN/++W12n9rNqHajCAkIMR1HuMF7zd+jSJ4iDJw1kIzMjBs/6V//0n1zR4/Wjc2F9xs+XP+un3xST1d5KI8tyFuPbeWD5R/Qq2Yvmt7Z1HQc4SZF8hThk1afsPrQakavH/3XJ6xYoQvx4MHSfN6XlC0L//kPzJsHk3Pwwa/NeGRBTstIo/eM3oQGh/JRC9+4+KG4pnv17jSv2JyXFr3E7pO7r33jwgXo21f/53zzTXMBhRn/+AfUqgVPPQVHjphOkyseWZDfWPIG6w6v48t2X1I8n3xg42uUUoxrPw5/5U/P6T2xcOyi/vOfsHMnfPMN5M9vNqRwP39/mDABzp/Xf5g9cOrC4wryz/t/5t1l79I3qi+dwzubjiMMKVewHKPajWJVyir2n96v54y//FJflumBB0zHE6aEh+seF/PmwYgRptPkmEcV5DOXz/Do9EepWLgin7b2vk5PImdiI2PpWaMnv53Yj/Xrdt1a8623TMcSpj31FLRpo/84e9hp1R5TkC3Lot/3/Ug5m8LEThMJDQ41HUnYwOcthhF+QmFlZHBq3BcQHGw6kjBNKX2F6vz5ISZGT2F4CI8pyO8ue5cpW6fwXvP3uDfsXtNxhE0UfO0/FLxksasodN78KmkZnn+2lnCCUqVg0iQ9Qu7Tx2Pmkz2iIM/aMYtXF79Kj+o9GHLfENNxhF1MmKBba5YpQ4HyVflx3488v+B506mEXbRooZtKTZ2qz9j0AAGmA9zK9uPb6T61O/eUvocxD42RTm5CW7sWBgzQH+BlZFBKKVrXa82nqz8lqlQUfe/pazqhsIN//hOSkuC116BGDdtfoMDWI+SUsym0ntiakIAQpsdMJ09gHtORhB3s2AFt20Lp0pCYeLXp/EctP6LZnc0YOGsgc3fONRxS2IJS+uibWrXgkUdg5UrTif6WbQvysQvHaPFtC05eOsncHnMpV7Cc6UjCDg4cgObN9f3583ULRocAvwCmdptKZIlIOid25uf9PxsKKWwlTx7dovOOO+DBB2Hj3/RBMcyWBfnM5TO0jmvNvtP7mNV9FrXuqGU6krCD33/X84Jnz8KCBTfscVwwpCDzes6jXMFytPuuHRt+22AgqLCdUqVg0SJ95EXLlvoEIhuyXUE+cfEErSa2YtPvm5jabSr3l7/fdCRhBykp0LSp/jp7tj7m+CZK5CvBwkcXUiikEC2/bcmaQ2vcGFTYVvnyuiWrZenPHmx4jLKtCvLBMwdpNL4RSUeSmNJ1Cg9WftB0JGEH27dDgwZw8KAuxg0a3PIlZQuWZXGvxRQMKUjTb5qyYPcCNwQVtnf33fDDD5CRAY0a2W5O2TYFeeuxrdT/qj6Hzx1mwaML6HB3B9ORhB2sWQMNG+rr4i1dCk2aZPuldxW5i+WPLady0cq0ndSWSZsnuS6n8BzVq+uugEWKQLNmMGeO6URX2aIgT9k6hXvH3kt6ZjpL+yyVaQqhjRsH998PBQrA8uW5aqdZKn8plvReQoOyDegxrQcvLnyR9Mx0F4QVHuXOO/V7qlo1eOgheOcdyMw0ncpsQU7LSOOf8/9J18ldCS8ezpp+a6hZqqbJSMIOLl2Cxx+Hfv30buXq1VCpUq4XVzCkIPN7zueJ2k/w4YoPaTahGUfOe2Z7RuFEJUrova6YGH2x1A4d9NVmDDJWkJOPJtNwfEM+WfUJT9d9mp/6/kTZgmVNxRF2sXYt1K2rexG8+qru2uWEa+IFBwQzou0IJnScwNpDa4kaFcW0bdOcEFh4tPz5IS4OPv9cH0YZFaWP4DHE7QX5SsYV3lzyJveMvofdJ3eT0CWB4W2GE+Qf5O4owk4uXtTdue69F06e1PN6b73l9EswPVrzUVb3W03p0NJ0TuxMl8QuMlr2dUrpDnE//6yPWW7VSvdTPnnS7VHcVpAty2L6tunUGFmDN5a+QdeIrmx7ahvdIrq5K4Kwo8xMmDhR97H973+hf399OFKbNi5bZfWS1VnTbw3vNnuXWTtmcffnd/P+sve5lHbJZesUHqBePX2a9csvw7ff6iMyPv8crlxxWwSXF2TLsli8dzH1v6pPp8ROKKWY9cgs4jrFydU+fFlmpj57KjoaHn1Uf+L9448wahQULOjy1Qf6BzK04VA2DtpIo/KNGPrDUCp/Vpkx68eQmp7q8vULmwoJ0Y2I1q2DiAh4+mn9wV9cHKS7/sNglxXk1PRUJmycQPSX0TSb0IyDZw4y9qGxbH5iM22rtHXVaoXdXbyoL0IaEaE/3T53Dr77Tv8HyMEhbc5StVhVvn/ke5b2WUrZgmUZMGsA5YeV562lb3HswjG35xE2ERUFixfD3LkQGgo9e0LFivpqJKdPu2y1Ti3IlmWx5tAanp7zNGGfhNF7Rm+uZFxhzENj2Pn0Th6PfpwAP9s3mBPOlpmp5+cGDIAyZWDQIMiXT09VbNsGsbHgZ/YIzPvL38+Kx1awoOcCoktH8/qS1wn7JIwuiV2YsX0GVzLct9sqbEIpaN0aNmyAmTPhrrvgxRf1e7hHD/2Bs5NHzbddHS+nX2bpvqXM2TmHWTtnsefUHoL9g2lftT39ovvRomILaZnpiy5c0COMOXP02XUHD+oi3KnTtcPZbPa+UErR4q4WtLirBduObWPUulHEJ8czddtUCocUpnWl1rSt3JZWlVpRLG+xWy9QeAc/P922s317+OUXvYeXmKgb4BcvrhsWPfig7pFRqNBtrUpZOeikX7t2bWvaD9NIOpLEyoMrWX5wOWsOrSE1I5WQgBCa3tmUTnd3okt4FwqGuH4e0Jc1cezeL1myxGgOQPcG2LdPv1lXrNAH3K9fD2lpugi3aAFdukDHjvrfTubKbZGemc6iPYuI3xLP3F1zOXrhKApFRIkIGpZtSINyDbin1D1ULVbVFnt/tnpfeLPUVD2dkZioR8qnTunCXbOmPrW/QQN9IlOlSuDvj1JqvWVZtW+12BwV5ICwACujf4a+7xdArdK1aFC2Ac0qNuOBCg9Iv2I3cvt/vIwMOHJEj3QPHIDdu3XHrB07YPNm3YENICgI6tTRb8gWLfRI2MXXuXPXtsi0Mll/eD3zd89n2YFlrExZydlU/XMH+wcTUSKCqkWrUrlIZSoXrUz5guUJKxBGmQJl3HZYpxRkA9LT9clLCxbAsmWwapX+rAQgb16IjEStWZOtgpyjP+mF8xTmzQffpGbJmtxT+h7yBubN3Q8gnCc9Xc/RZmT89Zaerm9pafrQnbQ03RMiNVV/vXRJ3y5c0BeCPHdOF9ZTp/Tt5Ek4dgyOHtVfMzL+uO5SpaByZf2BR82a124hIWa2hYv5KT/qlKlDnTJ1AMjIzGDrsa0kHUli4+8b2fT7JlamrCR+SzwWfxzoFM1TlBL5SlAiXwmK5i1K4ZDCFA4pTIHgAoQGhxIaFEq+oHzkCchD3sC8hASEEBwQTLB/MEH+QQT5BxHoH0iAX8DVm7/yx9/PHz/lh7/SXy0sFPaaCvJ6AQHXRsWg/89t3qz7Lm/cCJs2ZXtROZ6yWLduXU7jepejR/XkfpbsbL+bPef6x7PuX//1z/evuzVxvGxJDqJnS2AgFC587VaixLVbWBiULatvd92lz3KyAbuNClPTU9l7ei8Hzhzg4JmDHDx7kKMXjl69nbx0klOXT3Hq0ikupTv52OfxkC8oH+d3eM6Vln2BS6YslFLHgP23E8wJigHHDWewC9kW18i2uEa2xTV22RblLcu65YkXOSrIdqCUWpedvzS+QLbFNbItrpFtcY2nbQtbtN8UQgghBVkIIWzDEwvyl6YD2Ihsi2tkW1wj2+Iaj9oWHjeHLIQQ3soTR8hCCOGVPLogK6WGKKUspZTPNhZQSn2olNqulNqklJqulLq9k+k9kFKqtVLqV6XULqXUUNN5TFFKlVVK/aiU2qqUSlZKPWs6k0lKKX+l1C9KqVmms2SXxxZkpVRZoCVwwHQWwxYCkZZl1QB2AP8ynMetlFL+wBdAGyAceEQpFW42lTHpwBDLssKBe4GnfHhbADwLbDMdIic8tiADnwAvAj49CW5Z1gLLsrJ6AK4CwkzmMaAusMuyrD2WZV0B4oEOhjMZYVnWb5ZlbXDcP4cuRmXMpjJDKRUGtAXGms6SEx5ZkJVSHYBDlmVtNJ3FZh4D5poO4WZlgIPX/TsFHy1C11NKVQDuAVabTWLMMPSALdN0kJww3y/wJpRSi4BSN/jWK8DL6OkKn/B328KyrJmO57yC3mWNc2c2YT9KqfzAVGCwZVlnTedxN6VUO+CoZVnrlVJNTOfJCdsWZMuymt/ocaVUdeBOYKOj8X0YsEEpVdeyLK+8fPDNtkUWpVQfoB3QzPK94xgPAWWv+3eY4zGfpJQKRBfjOMuyppnOY0gDoL1S6kEgBCiglJpoWVZPw7luyeOPQ1ZK7QNqW5ZlhwYibqeUag18DDS2LMvnLgKnlApAf5jZDF2I1wLdLctKNhrMAKVHKN8AJy3LGmw6jx04RsjPW5bVznSW7PDIOWTxB58DocBCpVSSUmqU6UDu5PhA8x/AfPSHWIm+WIwdGgCPAk0d74UkxyhReAiPHyELIYS3kBGyEELYhBRkIYSwCSnIQghhE1KQhRDCJqQgCyGETUhBFkIIm5CCLIQQNiEFWQghbOL/ARtAgKTR3uQrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import scipy.stats as stats\n",
    "import math\n",
    "\n",
    "mu1 = -2\n",
    "mu2 = 2\n",
    "variance = 1\n",
    "sigma = math.sqrt(variance)\n",
    "x1 = np.linspace(mu1 - 5*sigma, mu1 + 4*sigma, 100)\n",
    "x2 = np.linspace(mu2 - 5*sigma, mu2 + 4*sigma, 100)\n",
    "plt.plot(x1, stats.norm.pdf(x1, mu1, sigma)/1,color=\"green\")\n",
    "plt.plot(x2, stats.norm.pdf(x2, mu2, sigma)/1,color=\"red\")\n",
    "plt.axvline(x=-2,color=\"black\")\n",
    "plt.axvline(x=0,color=\"black\")\n",
    "plt.axvline(x=+2,color=\"black\")\n",
    "plt.text(-2.7,0.55,\"Sensitive\")\n",
    "plt.text(-0.7,0.55,\"Balanced\")\n",
    "plt.text(1.7,0.55,\"Specific\")\n",
    "plt.ylim([0,0.53])\n",
    "plt.xlim([-5,5])\n",
    "plt.legend(['Negative','Positive'])\n",
    "plt.yticks([])\n",
    "#plt.set_yticklabels([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import zscore\n",
    "\n",
    "# Prepare data - apply z-score to ALL x columns\n",
    "# Only do this if you have no categoricals (and are sure you want to use z-score across the board)\n",
    "x_columns = df.columns.drop('diagnosis').drop('id')\n",
    "for col in x_columns:\n",
    "    df[col] = zscore(df[col])\n",
    "\n",
    "# Convert to numpy - Regression\n",
    "x = df[x_columns].values\n",
    "y = df['diagnosis'].map({'M':1,\"B\":0}).values # Binary classification, M is 1 and B is 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "# Plot a confusion matrix.\n",
    "# cm is the confusion matrix, names are the names of the classes.\n",
    "def plot_confusion_matrix(cm, names, title='Confusion matrix', cmap=plt.cm.Blues):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(names))\n",
    "    plt.xticks(tick_marks, names, rotation=45)\n",
    "    plt.yticks(tick_marks, names)\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    \n",
    "\n",
    "# Plot an ROC. pred - the predictions, y - the expected output.\n",
    "def plot_roc(pred,y):\n",
    "    fpr, tpr, _ = roc_curve(y, pred)\n",
    "    roc_auc = auc(fpr, tpr)\n",
    "\n",
    "    plt.figure()\n",
    "    plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.05])\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.title('Receiver Operating Characteristic (ROC)')\n",
    "    plt.legend(loc=\"lower right\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Chart Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/anaconda/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:143: calling RandomNormal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "WARNING:tensorflow:From /usr/local/anaconda/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/usr/local/anaconda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 426 samples, validate on 143 samples\n",
      "Epoch 1/1000\n",
      "426/426 - 0s - loss: 0.6862 - acc: 0.7160 - val_loss: 0.6709 - val_acc: 0.8811\n",
      "Epoch 2/1000\n",
      "426/426 - 0s - loss: 0.6311 - acc: 0.9131 - val_loss: 0.5482 - val_acc: 0.9650\n",
      "Epoch 3/1000\n",
      "426/426 - 0s - loss: 0.4305 - acc: 0.9413 - val_loss: 0.2529 - val_acc: 0.9720\n",
      "Epoch 4/1000\n",
      "426/426 - 0s - loss: 0.1852 - acc: 0.9413 - val_loss: 0.0916 - val_acc: 0.9790\n",
      "Epoch 5/1000\n",
      "426/426 - 0s - loss: 0.1062 - acc: 0.9671 - val_loss: 0.0567 - val_acc: 0.9860\n",
      "Epoch 6/1000\n",
      "426/426 - 0s - loss: 0.0796 - acc: 0.9765 - val_loss: 0.0535 - val_acc: 0.9860\n",
      "Epoch 7/1000\n",
      "426/426 - 0s - loss: 0.0675 - acc: 0.9812 - val_loss: 0.0530 - val_acc: 0.9860\n",
      "Epoch 8/1000\n",
      "426/426 - 0s - loss: 0.0617 - acc: 0.9789 - val_loss: 0.0510 - val_acc: 0.9930\n",
      "Epoch 9/1000\n",
      "426/426 - 0s - loss: 0.0564 - acc: 0.9836 - val_loss: 0.0500 - val_acc: 0.9860\n",
      "Epoch 10/1000\n",
      "426/426 - 0s - loss: 0.0504 - acc: 0.9859 - val_loss: 0.0478 - val_acc: 0.9860\n",
      "Epoch 11/1000\n",
      "426/426 - 0s - loss: 0.0465 - acc: 0.9859 - val_loss: 0.0472 - val_acc: 0.9860\n",
      "Epoch 12/1000\n",
      "426/426 - 0s - loss: 0.0438 - acc: 0.9930 - val_loss: 0.0455 - val_acc: 0.9860\n",
      "Epoch 13/1000\n",
      "426/426 - 0s - loss: 0.0384 - acc: 0.9906 - val_loss: 0.0557 - val_acc: 0.9860\n",
      "Epoch 14/1000\n",
      "426/426 - 0s - loss: 0.0350 - acc: 0.9906 - val_loss: 0.0519 - val_acc: 0.9860\n",
      "Epoch 15/1000\n",
      "426/426 - 0s - loss: 0.0311 - acc: 0.9930 - val_loss: 0.0530 - val_acc: 0.9860\n",
      "Epoch 16/1000\n",
      "426/426 - 0s - loss: 0.0293 - acc: 0.9930 - val_loss: 0.0611 - val_acc: 0.9860\n",
      "Epoch 17/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "426/426 - 0s - loss: 0.0264 - acc: 0.9953 - val_loss: 0.0572 - val_acc: 0.9860\n",
      "Epoch 00017: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f7794932f98>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification neural network\n",
    "import numpy as np\n",
    "import tensorflow.keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(1,activation='sigmoid',kernel_initializer='random_normal'))\n",
    "model.compile(loss='binary_crossentropy', \n",
    "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
    "              metrics =['accuracy'])\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "    patience=5, verbose=1, mode='auto', restore_best_weights=True)\n",
    "\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4FGX38PHvIfQqRVGJCkgLJYB0RAwixQJYEFH0ASwYBQsiCFh4QBSxACJNfqjoi4KKAhZ8aIpiAaR3DEZKEAVCDQZMyHn/mElcQsoSsjsp53Nde2Vn5t6Zs3dm58zcM3OPqCrGGGNMegp4HYAxxpiczRKFMcaYDFmiMMYYkyFLFMYYYzJkicIYY0yGLFEYY4zJkCWKPE5EeojIQq/jyElEJE5Eqnqw3MoioiJSMNjLDgQR2SwiEVn4XJbXSRHpICJzs/LZrBKRIiKyTUQuDOZycxJLFEEkIjtFJN7dUP0pItNFpGQgl6mqH6hq+0Auw5eItBSRb0TkuIgcFZEvRKR2sJafRjxLReQB33GqWlJVowO0vBoi8omIHHS//wYReVJEQgKxvKxyE1a185mHqtZR1aWZLOes5Hie6+SLwMs+81cROeH+pvaKyJjUdS0iN4vISrdcrIh8ICKhqcpcIiJvi8g+d93dJiLDRaSEqp4C3gEGZzHmXM8SRfB1UtWSQAOgITDE43iyJK29YhFpASwE5gGXAlWA9cCPgdiDz2l75iJyJbAC2APUU9UywB1AI6BUNi/Ls+/u1bJFpAlQRlWXp5pU3/1NXQvcCdzn85muwIfAOKACUAc4BfwgImXdMuWAn4FiQAtVLQW0A8oAV7qz+hDoKSJFAvT1cjZVtVeQXsBO4Hqf4VeAr3yGiwCvAbuBv4ApQDGf6V2AdcAx4Degozu+DPA2sA/YC4wEQtxpvYAf3PeTgddSxTQPeNJ9fynwKXAA+B14zKfcf4HZwAx3+Q+k8f2WAZPSGP818L77PgKIAYYCB9066eFPHfh89mngT+D/AWWBL92YD7vvQ93yLwKngZNAHDDBHa9ANff9dGAi8BVwHGdDf6VPPO2B7cBRYBLwXVrf3S07w/f/mcb0yu6ye7rf7yDwjM/0pjgbrCPu/3ICUNhnugJ9gSjgd3fcGziJ6RiwGrjGp3yIW8+/ud9tNXAZ8L07rxNuvdzplr8ZZ/06AvwEhKdad58GNuBsaAvisz67sa9y4/gLGOOO3+0uK859tcBnnXTL1AEWAYfczw5Np/6eB6alGpfyv3SHPwYmuu8F2AUMSvWZAsAmYIQ7PBLYCBTI5PcbBVzr9XbEi5fnAeSnV6ofVqi7cr7hM30s8DlQDmcP9AtglDutqbuxaueu6JWAWu60OcBbQAngImAl8JA7LeVHCbR2NyriDpcF4nESRAF3Q/I8UBioCkQDHdyy/wUSgFvcssVSfbfiOBvlNml8797APvd9BJAIjMFJCtfibLBq+lEHyZ8d7X62GFAeuN1dfingE2Cuz7KXkmrDztmJItat34LAB8Asd1oFnA3fbe60x906SC9R/An0zuD/X9ld9v+5sdfH2eiGudMbAc3dZVUGtgJPpIp7kVs3ycnzHrcOCgID3BiKutMG4qxjNXE2mvWB8qnrwB1uCOwHmuEkmJ4462sRn3V3HU6iKeYzLnl9/hm4131fEmie6jsX9FlWL/5dJ0vhJMUBQFF3uFk69fcJMDCD/2Utd179fYYVqJLGvIYDP7vvlwPD/fj9fo7PzlN+enkeQH56uT+sOJy9OwWWABe40wRng+m7N9uCf/cc3wLGpjHPiu7GxvfI4y7gW/e9749ScPbwWrvDDwLfuO+bAbtTzXsI8K77/r/A9xl8t1D3O9VKY1pHIMF9H4GzsS/hM/1j4Dk/6iAC+Ad3Q5hOHA2Awz7DS8k8UUzzmXYjsM19/5/kjYlP/e1JPT+f6Qm4R3npTK/sLjvUZ9xKoHs65Z8A5qSK+7pM1rHDOE0x4BwJdUmnXOpEMRl4IVWZ7bh70O66e18a63NyovgeZ+NbIZ3vnF6iuAtY6+fvZxEQmcb3OOauNwrM5N/k1sodd9b6AkQCUe77qNTzTWf5HwDP+xNrXnvZOYrgu0WdNtAInD2eCu74C3H2ileLyBEROQL8zx0Pzp7cb2nM7wqgELDP53Nv4RxZnEGdtX0Wzo8T4G6clT95Ppcmz8Odz1CcRJRsTwbf6zCQBFySxrRLcJpZUsqq6gmf4V04RzWZ1QHAAVU9mTwgIsVF5C0R2SUix3A2WBec48njP33e/42zR4wbU8p3dusvJoP5xJL29/dree6J8C/dCx2OAS/x7/qR7Iz/gYg8JSJb3RPnR3CaIZM/k946k5YrgAGp/v+X4dRBmstO5X6gBrBNRH4RkZv9XO65xHiYtM/1XIVTh3fi7PCUcMcnr3OZrZP+/t9K4TTL5TuWKDyiqt/h7M2+5o46iNMMVEdVL3BfZdQ5SQfOj/TKs+fEHpwjigo+nyutqnXSWfRMoKuIXIHzo/rUZz6/+8zjAlUtpao3+oadwfc5gdP8cEcak7vhHD0lKysiJXyGLwf+8KMO0ophAE7TSjNVLY3TvAbO3n+GMfthH86RkjNDEfEdTsNinGawrJoMbAOqu99lKP9+j2Qp30dErgEG4dRvWVW9AKd5Mvkz6a0zadkDvJjq/19cVWemtezUVDVKVe/C2UEZDcx2/8eZ1f8enGZOf2zASUZpLV9V9WOcdfB5d/R2nMR+xjopIgVw/k/J6+Ri4FZ3fEbCcC7OyHcsUXhrHNBOROqrahJO2/VYEbkIQEQqiUgHt+zbQG8RaSsiBdxptVR1H86VRq+LSGl32pUicm1aC1TVtTgb5GnAAlVN3kNaCRwXkadFpJiIhIhIXfdKE38Nxrky5DERKSUiZUVkJE7z0fBUZYeLSGF3Y3cz8IkfdZCWUjjJ5Yh79cqwVNP/wv8NUWpfAfVE5Bb3Sp++wMUZlB8GtBSRV0XkYjf+aiIyQ0Qu8GN5pXCaUeJEpBbwsB/lE3FO5BcUkeeB0j7TpwEviEh1cYSLSHl3Wup6+T8gUkSauWVLiMhNIuLX1Voico+IXOj+D5PXqSQ3tiTS/x98CVwiIk+Ic79CKRFplk7Z+TjntDLyMvCgiFzsHgE+BTwrIneLSFH3/zINp57Gup8Z4w6/5+5AJa93Y0QkPHkY59xQ6iuu8gVLFB5S1QPA+/y7B/Q0sANY7jY9LMbZW0ZVV+KcFB6Ls9f4HU5zATht6YWBLTiH57PJ+FD6Q+B6929yLKdxNtgNcK54Sk4mZc7h+/wAdMA5+bsPp0mpIdBKVaN8iv7pxvkHTtNXpKpuy6wO0jEO58TwQZwf8f9STX8D5wjqsIiM9/e7uN/nIM7e6Cs4zRO1ca7sOZVO+d9wkmJlYLOIHMU5YluFc14qM0/hNAcex9lwf5RJ+QU43/dXnLo+yZnNQ2Nwzv8sxElAb+PUFTjnnN5zm5m6qeoqnHNWE3D+NztwziX4qyPOd47DqfPuqhqvqn/jXH32o7us5r4fUtXjOBdodMJZL6KANmktQFXXAEczSCSo6kac5seB7vBHwL1Af5z/4Ra3Dq5W1Vi3zCGgJc45phUichznaOOoWw/g/F/eU+eeinwn+eoXY4JCnDt5Z6hqRk04OZLbNBGDcznvt17Hkx+JSHvgEVW9JYjLLILT5NRaVfcHa7k5SY66YcmYnMZt9lqB07w1EKf9P182P+QEqroQ5wgpmMs8hXPhSb5lTU/GZKwFzlU5B3GaR25R1XhvQzImuKzpyRhjTIbsiMIYY0yGct05igoVKmjlypW9DsMYY3KV1atXH1TVLHWVnusSReXKlVm1apXXYRhjTK4iIruy+llrejLGGJMhSxTGGGMyZInCGGNMhixRGGOMyZAlCmOMMRmyRGGMMSZDAUsUIvKOiOwXkU3pTBcRGS8iO0Rkg4hcFahYjDHGZF0gjyim43Q9nJ4bgOruqw/OQ1uMMcbkMAG74U5VvxeRyhkU6QK87z5cZLmIXCAil7gP4snRPlyxm3nr9nodhjHGZGr/r2v5dcms85qHl3dmV+LMh6zEuOPOShQi0gfnqIPLL788KMFlZN66vWzZd4zal5TOvLAxxnjg5PHDbPh0IjuXz6dEeX8eCZ6+XNGFh6pOBaYCNG7cOEd0d1v7ktJ89FALr8Mwxpg03X777cSsWsiQIUN49tlnKVGiROYfSoeXVz3tBS7zGQ51xxljjMmCzZs3s3evsxkdPXo069at46WXXqJ48eLnNV8vjyg+B/qJyCygGXDUn/MT0QdOcOdbPwc8uIxYs5MxJic5ceIEL7zwAq+//jo9evRg+vTpVKtWLdvmH7BEISIzgQiggojEAMOAQgCqOgWYD9yI8/Dyv4He/sw3PuF0IMI9J7UvKU2XBpW8DsMYY/jqq6/o27cvu3bt4r777mP06NHZvoxc94S7cleE6aFdW70OwxhjPDdp0iT69u1L7dq1mTJlCtdcc026ZUVktao2zspycsXJbGOMMY7ExEQOHDjAJZdcQrdu3YiPj+fRRx+lcOHCAVumHVEYY0wusXLlSh566CEKFizI8uXLCQkJ8fuz53NEYX09GWNMDnfkyBEeeeQRmjdvzv79+3n66acpUCB4m29rejLGmBxs48aNtGvXjgMHDvDYY48xYsQISpcO7lWXliiMMSYHSkhIoFChQtSoUYM2bdowcOBArrrKm75TrenJGGNykFOnTjFixAjq1KlDXFwcRYoUYebMmZ4lCbBEYYwxOcY333xDeHg4w4YN46qrruLUqVNehwRYojDGGM/Fx8dz77330rZtWxITE/nf//7HrFmzKF++vNehAZYojDHGc0WLFuXgwYM8++yzbNq0iQ4dOngd0hksURhjjAc2bNhAhw4diImJQUT46quveOGFFyhWrJjXoZ3FEoUxxgTRiRMnUq5gWrNmDVFRUQBBvS/iXOXcyIwxJo/5/PPPqV27Nq+99hr33Xcf27dvp02bNl6HlSm7j8IYY4Jk7ty5lC5dmh9++IGrr77a63D8Zn09GWNMgCQkJDB+/HjatGnDVVddxfHjxylatCiFChUKeizW15MxxuQwy5cvp3Hjxjz11FN8/PHHAJQqVcqTJHG+LFEYY0w2Onz4MJGRkbRs2ZJDhw4xZ84cRo0a5XVY58UShTHGZKOpU6cybdo0+vfvz5YtW7jlllsQEa/DOi92jsIYY87T9u3bOXDgAK1ateLUqVNs376d8PBwr8M6g52jMMYYD5w8eZJhw4YRHh5O3759UVWKFCmS45LE+bJEYYwxWbBo0SLq1avHiBEj6Nq1KwsWLMj1TUzpsfsojDHmHH3//fe0b9+e6tWrs2jRIq6//nqvQwooO6Iwxhg/nD59mo0bNwJwzTXX8Pbbb7Nhw4Y8nyTAEoUxxmRq7dq1tGzZkquvvpq//voLEeG+++6jaNGiXocWFJYojDEmHcePH+fJJ5+kcePG7Ny5k8mTJ3PRRRd5HVbQ2TkKY4xJw9GjR6lXrx579uzhoYceYtSoUZQtW9brsDxhicIYY3wcO3aM0qVLU6ZMGfr06UPbtm1p0aKF12F5ypqejDEGpwO/V155hdDQUNasWQPAs88+m++TBNgRhTHG8OOPPxIZGcmmTZu45ZZbuPDCC70OKUexIwpjTL726KOP0qpVK44ePcq8efOYM2cOl112mddh5SiWKIwx+Y5vH3cXX3wxTz31FFu2bKFz584eRpVzWaeAxph8Zdu2bURGRtK/f3+6dOnidThBY50CGmNMJuLj43nuuecIDw9n/fr1xMfHex1SrhHQRCEiHUVku4jsEJHBaUwvIyJfiMh6EdksIr0DGY8xJn9asmQJ9erVY+TIkXTv3p3t27fTvXt3r8PKNQJ21ZOIhAATgXZADPCLiHyuqlt8ivUFtqhqJxG5ENguIh+o6j+BissYk//ExMRQsGBBlixZwnXXXed1OLlOIC+PbQrsUNVoABGZBXQBfBOFAqXE6Zu3JHAISAxgTMaYfOD06dNMmTKFwoUL8+CDD/Kf//yH7t27U6RIEa9Dy5UC2fRUCdjjMxzjjvM1AQgD/gA2Ao+ralLqGYlIHxFZJSKrEhISAhWvMSYPWLNmDc2bN6dfv34sWLAAABGxJHEevD6Z3QFYB1wKNAAmiEjp1IVUdaqqNlbVxoUKFQp2jMaYXODYsWM8/vjjNGnShD179jBz5kw++eQTr8PKEwKZKPYCvnethLrjfPUGPlPHDuB3oFYAYzLG5FHr169nwoQJREZGsm3bNrp3755nnzgXbIFMFL8A1UWkiogUBroDn6cqsxtoCyAiFYGaQHQAYzLG5CG///4777zzDuA8TGjHjh1MnDiRCy64wOPI8paAJQpVTQT6AQuArcDHqrpZRCJFJNIt9gLQUkQ2AkuAp1X1YKBiMsbkDf/88w+jRo2idu3aDBgwgMOHDwNQpUoVjyPLm+zObGNMrrJs2TIiIyPZsmULt912G2+88QahoaFeh5Xjnc+d2dZ7rDEm1zhw4ADt27enYsWKfPHFF9x8881eh5QveH3VkzHGZEhVWbRoEQAXXnghX375JZs3b7YkEUSWKIwxOdbmzZu59tprad++PUuXLgWgbdu2lChRwtvA8hlLFMaYHOfvv/9m6NChNGjQgM2bNzNt2jRat27tdVj5lp2jMMbkKKpKmzZtWLlyJT179uTVV1+1J855zBKFMSZH2LdvHxdddBEhISEMHTqUMmXKEBER4XVYBmt6MsZ47PTp04wfP56aNWsyadIkALp06WJJIgfxK1GISGERqRboYIwx+cuqVato2rQpjz/+OC1btuTGG2/0OiSThkwThYjchNOz6yJ3uIGIzAl0YMaYvO2VV16hadOm7Nu3j48++oivv/6aK6+80uuwTBr8OaIYATQDjgCo6jrAji6MMedMVUl+VEDTpk3p27cvW7dupVu3btaBXw7mT6JIUNUjqcblrn4/jDGe++233+jYsSODBztPRY6IiODNN9+kTJkyHkdmMuNPotgqIt2AAm5PsGOB5QGOyxiTR5w6dYqRI0dSt25dfv75Z2teyoX8SRT9gEZAEvAZcAp4PJBBGWPyhtWrV9OgQQOee+45OnXqxLZt23jkkUe8DsucI3/uo+igqk8DTyePEJHbcJKGMcakq2TJkogI8+fP54YbbvA6HJNFmXYzLiJrVPWqVONWq2qjgEaWDutm3JicKykpiXfffZeff/6ZadOmpYwrUMBu2fJaQLoZF5EOQEegkoiM8ZlUGqcZyhhjUmzatInIyEh+/PFHWrduzYkTJyhRooQliTwgo//gfmATcBLY7PNaCNgxpDEGgBMnTvD000/TsGFDtm3bxrvvvsvSpUuth9c8xJ+mp6KqejJI8WTKmp6MyVliY2MJCwujU6dOvPLKK5QvX97rkEwaAv2Eu0oi8iJQGyiaPFJVa2RlgcaY3C8mJobx48czatQoypcvz7Zt2yhXrpzXYZkA8afxcDrwLiA4TU4fAx8FMCZjTA6VmJjI2LFjCQsLY8KECaxbtw7AkkQe50+iKK6qCwBU9TdVfRY7R2FMvrNixQoaN27Mk08+SevWrdm8eTONGnly8aMJMn+ank6JSAHgNxGJBPYCpQIbljEmJ0lKSqJ3794cPXqU2bNnc9ttt1nfTPmIP4miP1ACeAx4ESgD3BfIoIwx3lNVZs+eTceOHSlVqhSfffYZlSpVolQp20/MbzJtelLVFap6XFV3q+q9qtoZ2Bn40IwxXomKiqJDhw5069aNqVOnAlCrVi1LEvlUholCRJqIyC0iUsEdriMi7wMrghKdMSaoTp06xYgRI6hXrx4rVqxgwoQJPPHEE16HZTyWbqIQkVHAB0AP4H8i8l/gW2A9YJfGGpMH9e3bl2HDhnHrrbeybds2+vbtS0hIiNdhGY+le8OdiGwBGqlqvIiUA/YA9VQ1OpgBpmY33BmTvfbv309SUhIXX3wxUVFRREdH06FDB6/DMtnsfG64y6jp6aSqxgOo6iHgV6+ThDEm+yQlJTF16lRq1qzJ4487Tw6oXr26JQlzloyueqoqIsldiQtQxWcYVb0toJEZYwJmw4YNREZG8vPPPxMREcHw4cO9DsnkYBklittTDU8IZCDGmOCYPXs23bt3p2zZsrz//vvcc889dk+EyVC6iUJVlwQzEGNMYB07dozSpUsTERGRctLaut4w/si099icxk5mG3Nudu/ezaOPPsoff/zB8uXL7SqmfCpQJ7PPm4h0FJHtIrJDRAanUyZCRNaJyGYR+S6Q8RiTnyQkJPDaa68RFhbG4sWL6datG7ltx9DkDP504QGAiBRR1VPnUD4EmAi0A2KAX0Tkc1Xd4lPmAmAS0FFVd4vIRf6HboxJz65du+jcuTMbNmygU6dOvPnmm1xxxRVeh2VyqUyPKESkqYhsBKLc4foi8qYf824K7FDVaFX9B5gFdElV5m7gM1XdDaCq+88pemPMGZKPGC6++GIqVqzInDlzmDdvniUJc178aXoaD9wMxAKo6nqgjR+fq4Rzk16yGHecrxpAWRFZKiKrReQ/fszXGJOKqjJjxgyaNGlCXFwcRYoUYeHChdxyyy12RZM5b/4kigKquivVuNPZtPyCQCPgJqAD8JyInNU9iIj0EZFVIrIqISEhmxZtTN6wfft22rZty7333kvBggWJjY31OiSTx/iTKPaISFNARSRERJ4AfvXjc3uBy3yGQ91xvmKABap6QlUPAt8D9VPPSFWnqmpjVW1cqFAhPxZtTN6XmJjIsGHDCA8PZ82aNUyePJmffvrJmplMtvMnUTwMPAlcDvwFNHfHZeYXoLqIVBGRwkB34PNUZeYBrUSkoIgUB5oBdu2rMX4ICQlh2bJldO3ale3btxMZGUmBAgG9kNHkU/5c9ZSoqt3Pdcaqmigi/YAFQAjwjqpudp+Sh6pOUdWtIvI/YAOQBExT1U3nuixj8os///yToUOHMnz4cC677DLmz59P0aJFvQ7L5HGZ3nAnIr8B24GPcK5QOh6MwNJjN9yZ/Oj06dNMnTqVIUOGEB8fz4wZM7jjjju8DsvkIgG94U5VrwRG4px03igic0XknI8wjDFZs3btWlq2bMkjjzxC48aN2bhxoyUJE1R+NWiq6k+q+hhwFXAM54FGxpggmDBhAjt37uSDDz5g0aJF1Khhzw0zweVP01NJnBvlugNhOCegP1ZVTx6Hak1PJq9TVebOnUvlypVp2LAhhw8fBqBs2bIeR2Zys0D39bQJ50qnV1S1mqoO8CpJGJPX7dy5k86dO3Pbbbcxbtw4wEkQliSMl/y56qmqqiYFPBJj8rGEhATGjBnD8OHDKVCgAK+99lrKU+eM8Vq6iUJEXlfVAcCnInJW+5Q94c6Y7PPWW28xePBgbrnlFt544w0uv/xyr0MyJkVGRxQfuX/tyXbGBEBsbCw7d+6kUaNGPPjgg1SrVo2OHTt6HZYxZ0n3HIWqrnTfhqnqEt8XzkltY0wWqCrvvfcetWrV4o477iAxMZEiRYpYkjA5lj8ns+9LY9z92R2IMfnB1q1badOmDb169aJ69erMnTuXggX9fiyMMZ7I6BzFnTiXxFYRkc98JpUCjgQ6MGPymvXr19OkSRNKlizJ1KlTuf/++61vJpMrZLQrsxLnGRShOE+qS3YcWBvIoIzJS2JiYggNDSU8PJzhw4dz//33c9FF9jBHk3tkesNdTmM33Jnc4o8//qB///7Mnz+fbdu2UalS6ud2GRM8AbnhTkS+c/8eFpFDPq/DInIoq8Eak9edPn2aCRMmEBYWxrx58xg0aBAVKlTwOixjsiyjpqfkx53aGm6Mn06ePEnr1q355ZdfaNeuHZMmTaJatWpeh2XMecno8tjku7EvA0JU9TTQAngIKBGE2IzJNZIf0Vu0aFHatGnDzJkzWbBggSUJkyf4c8nFXJzHoF4JvAtUBz4MaFTG5BKqyuzZs6lWrRpr1qwBYPTo0XTv3h0R8Tg6Y7KHP4kiSVUTgNuAN1W1P2Bn5Uy+Fx0dzU033cQdd9xB+fLl7VJXk2f5s2YnisgdwL3Al+64QoELyZicb8yYMdSpU4dly5Yxbtw4Vq5cSYMGDbwOy5iA8OeW0PuAR3C6GY8WkSrAzMCGZUzOFhcXx4033sgbb7xBaGio1+EYE1B+3UchIgWB5LNyO1Q1MaBRZcDuozBeOHjwIAMHDuTWW2+lc+fOJCUlWVOTyVXO5z6KTI8oROQa4P8BewEBLhaRe1X1x6ws0JjcJCkpienTpzNw4ECOHTtGvXr1ACxJmHzFn6anscCNqroFQETCcBJHljKTMbnFli1biIyMZNmyZbRq1YopU6ZQp04dr8MyJuj8SRSFk5MEgKpuFZHCAYzJmBxh1apVbN68mbfffptevXrZUYTJtzI9RyEi04GTwAx3VA+guKr2DGxoabNzFCaQ5s+fT2xsLPfeey+qyuHDhylXrpzXYRlz3gLS15OPSCAaGOS+onHuzjYmz4iJiaFr167cdNNNTJgwAVVFRCxJGEMmTU8iUg+4Epijqq8EJyRjgicxMZGJEyfy7LPPkpiYyIsvvshTTz1ld1Ub4yOj3mOH4nTf0QNYJCJpPenOmFxt9erVPPHEE7Rq1YrNmzczdOhQChe2U3DG+Mqo6akHEK6qdwBNgIeDE5IxgXX06FE++8x5aGOzZs1YsWIF8+fPp2rVqh5HZkzOlFGiOKWqJwBU9UAmZY3J8VSVjz76iFq1atG9e3f++OMPAJo2bWpNTcZkIKNzFFV9npUtwJW+z85W1dsCGpkx2ei3336jb9++LFiwgEaNGvHFF19w6aWXeh2WMblCRoni9lTDEwIZiDGBcvz4cRo1akRSUhLjx4/nkUceISQkxOuwjMk10k0UqrokmIEYk902bNhAeHg4pUqV4u2336Z58+b23GpjssDOO5g858CBA/Ts2ZP69eszf/58AG6//XZLEsZkUUAThYh0FJHtIrJDRAZnUK6JiCSKSNdAxmPytqSkJKZNm0bNmjWZOXMmQ4cOJSIiwuuwjMn1/OnrCQARKaKqp86hfAgwEWgHxAC/iMjnvv1G+ZQbDSz0d97GpOVHWCqTAAAb1klEQVT2229n7ty5tG7dmsmTJ1O7dm2vQzImT8j0iEJEmorIRiDKHa4vIm/6Me+mOM+uiFbVf4BZQJc0yj0KfArs9z9sYxwnTpwgMdF5PMpdd93F9OnTWbp0qSUJY7KRP01P44GbgVgAVV0PtPHjc5WAPT7DMaR61raIVAJuBSZnNCMR6SMiq0RkVUJCgh+LNvnBF198Qe3atZk0aRIA3bp1o2fPnnZPhDHZzJ9EUUBVd6Uadzqblj8OeFpVkzIqpKpTVbWxqjYuVMge153f7dmzh9tuu43OnTtTqlQpGjVq5HVIxuRp/pyj2CMiTQF1zyc8Cvzqx+f2Apf5DIe643w1Bma5e4AVgBtFJFFV5/oxf5MPzZgxg8jISJKSknj55Zfp37+/9c1kTID5kygexml+uhz4C1iMf/0+/QJUF5EqOAmiO3C3bwFVrZL83n3uxZeWJExakrv9Dg0NJSIigjfffJMqVapk/kFjzHnLNFGo6n6cjfw5UdVEEekHLABCgHdUdbOIRLrTp5zrPE3+c+TIEYYMGUKJEiV47bXXiIiIsEtejQmyTBOFiPwfcNZj8FS1T2afVdX5wPxU49JMEKraK7P5mfxDVZk5cyZPPvkkBw4coH///ilHFcaY4PKn6Wmxz/uiOFcp7UmnrDHn7ffff6dPnz4sXryYJk2a8PXXX9OwYUOvwzIm3/Kn6ekj32ER+X/ADwGLyOR7CQkJbNiwgYkTJ/LQQw9ZB37GeMzvO7N9VAEqZncgJn9bsmQJX331FWPGjKFGjRrs2rWLokWLeh2WMQb/7sw+LCKH3NcRYBEwJPChmfzgr7/+4p577uH666/n888/JzY2FsCShDE5SIaJQpwzh/WBC91XWVWtqqofByM4k3clJSXx1ltvUatWLT7++GOee+45Nm7cSPny5b0OzRiTiqiedUHTmQVENqlq3SDFk6lyV4TpoV1bvQ7DnKfDhw9To0YN6taty+TJk6lVq5bXIRmTp4nIalVtnJXP+tOFxzoRsUtOzHmLi4tjzJgxnD59mrJly7JixQq++eYbSxLG5HDpJgoRST7R3RCni/DtIrJGRNaKyJrghGfyinnz5lG7dm0GDBjAd999B0DVqlXtvghjcoGMrnpaCVwFdA5SLCYP2rVrF4899hiff/459erVY9asWbRs2dLrsIwx5yCjRCEAqvpbkGIxeYyq0rVrV7Zs2cIrr7zCE088gfX+a0zuk1GiuFBEnkxvoqqOCUA8Jg9Yvnw5derUoVSpUkydOpVy5cpxxRVXeB2WMSaLMjqZHQKUBEql8zLmDIcOHeKhhx6iRYsWvPbaawA0bNjQkoQxuVxGRxT7VHVE0CIxuZaqMmPGDAYMGMChQ4cYMGAAAwcO9DosY0w2yfQchTGZGTp0KC+//DLNmzdn0aJF1K9f3+uQjDHZKKNE0TZoUZhc5+TJk8TFxVGhQgV69+7NFVdcQZ8+fShQwJ9bc4wxuUm6v2pVPRTMQEzusWjRIurVq8eDDz4IQI0aNYiMjLQkYUweZb9s47c///yTu+++m/bt2yMi9OvXz+uQjDFBkJVuxk0+9O2333LrrbcSHx/Pf//7X55++mnr4dWYfMIShclQQkIChQoVIjw8nHbt2vHiiy9So0YNr8MyxgSRNT2ZNB0/fpz+/ftzzTXXcPr0acqXL88nn3xiScKYfMgShTmDqvLZZ58RFhbGG2+8QcOGDTl16pTXYRljPGSJwqQ4ePAgnTp14vbbb6dChQr89NNPTJ48meLFi3sdmjHGQ5YoTIpSpUrx119/MWbMGFatWkXz5s29DskYkwNYosjnfvjhB2644Qbi4uIoUqQIK1asoH///hQsaNc5GGMclijyqdjYWB544AGuueYatmzZQnR0NIDdNGeMOYttFfIZVWX69OnUrFmT6dOnM3DgQLZs2UJ4eLjXoRljcihrX8iH3n//fWrWrMmUKVOoV6+e1+EYY3I4O6LIB+Lj4xk2bBgxMTGICJ9++inLli2zJGGM8YslijxuwYIF1K1blxEjRjBv3jwAypYta+cijDF+s61FHvXHH39w55130rFjRwoVKsQ333xD3759vQ7LGJMLWaLIo0aOHMm8efMYMWIE69evp02bNl6HZIzJpURVvY7hnFxavY7+EbXZ6zBypNWrV6d04BcbG8vhw4epVq2a12EZY3IAEVmtqo2z8tmAHlGISEcR2S4iO0RkcBrTe4jIBhHZKCI/iUimz9C8tEyxwASbix07dozHHnuMpk2bMnToUADKly9vScIYky0ClihEJASYCNwA1AbuEpHaqYr9DlyrqvWAF4CpgYonL1JVPvnkE2rVqsWECRN4+OGHmTFjhtdhGWPymEDeR9EU2KGq0QAiMgvoAmxJLqCqP/mUXw6EBjCePOfDDz/knnvuoWHDhsybN48mTZp4HZIxJg8KZKKoBOzxGY4BmmVQ/n7g67QmiEgfoA/A5Zdfnl3x5Ur//PMP0dHR1KpVi65duxIfH0+vXr2sbyZjTMDkiKueRKQNTqJ4Oq3pqjpVVRurauMLL7wwuMHlIN9//z0NGjSgffv2nDx5kiJFivDAAw9YkjDGBFQgE8Ve4DKf4VB33BlEJByYBnRR1dgAxpNrHTx4kN69e3PttdcSHx/PlClT7HnVxpigCeSu6C9AdRGpgpMgugN3+xYQkcuBz4B7VfXXAMaSa0VHR9OkSROOHTvG4MGDee655+xBQsaYoApYolDVRBHpBywAQoB3VHWziES606cAzwPlgUkiApCY1et885pjx45RunRpqlSpQu/evenVqxd169b1OixjTD6U6264a9y4sa5atcrrMALm77//5oUXXmDq1KmsX7+e0FC7EMwYc/7O54Y7Owuag3z11Vf069ePnTt30rt3b4oVs5sLjTHes0SRAyQmJnLXXXcxe/ZswsLC+O6772jdurXXYRljDJBDLo/Nr5Kb/QoWLEjFihV56aWXWLdunSUJY0yOYonCI7/88gvNmjVjzZo1AEyYMIEhQ4ZQuHBhjyMzxpgzWaIIsqNHj9KvXz+aNWtGTEwMsbF264gxJmezRBFEyR34TZ48mX79+rFt2zbatWvndVjGGJMhO5kdRFu3bqVSpUp88cUXNG5st4sYY3IHu48igE6dOsWrr75K/fr16dSpEwkJCRQoUICQkBCvQzPG5DM59sFF+dm3335L/fr1ee6551iyZAkAhQoVsiRhjMl1LFFks/3799OzZ0+uu+46EhIS+Prrrxk3bpzXYRljTJZZoshmCxcuZObMmTzzzDNs2rSJjh07eh2SMcacFzuZnQ02btzI9u3b6dq1Kz169KBly5ZUrVrV67CMMSZb2BHFeThx4gSDBg2iYcOGDBo0iISEBETEkoQxJk+xI4os+uKLL+jXrx+7d+/m/vvvZ/To0RQqVMjrsEwOkpCQQExMDCdPnvQ6FJOPFC1alNDQ0GzdHlmiyIJNmzbRuXNn6tSpw7Jly2jVqpXXIZkcKCYmhlKlSlG5cmXc560YE1CqSmxsLDExMVSpUiXb5mtNT35KTExk6dKlANStW5cvv/yStWvXWpIw6Tp58iTly5e3JGGCRkQoX758th/FWqLww4oVK2jcuDFt27YlKioKgJtuusmamkymLEmYYAvEOmeJIgOHDx/m4YcfpkWLFhw8eJBPPvmEatWqeR2WMcYElSWKdJw6dYqGDRsydepUnnjiCbZu3cptt91me4gmVwkJCaFBgwbUrVuXTp06ceTIkZRpmzdv5rrrrqNmzZpUr16dF154Ad8ufb7++msaN25M7dq1adiwIQMGDPDiK2Ro7dq13H///V6Hka7Y2FjatGlDyZIl6devX7rlDh06RLt27ahevTrt2rXj8OHDKdNGjRpFtWrVqFmzJgsWLEgZf/31159RLqBUNVe9GjVqpIEUExOT8v7dd9/VNWvWBHR5Ju/asmWL1yFoiRIlUt7/5z//0ZEjR6qq6t9//61Vq1bVBQsWqKrqiRMntGPHjjphwgRVVd24caNWrVpVt27dqqqqiYmJOmnSpGyNLSEh4bzn0bVrV123bl1Ql3ku4uLidNmyZTp58mTt27dvuuUGDhyoo0aNUlXVUaNG6aBBg1RVdfPmzRoeHq4nT57U6OhorVq1qiYmJqqq6vTp01P+n6mlte4BqzSL21276sl18uRJRo8ezUsvvcTHH39Mly5d6NWrl9dhmTxi+Beb2fLHsWydZ+1LSzOsUx2/y7do0YINGzYA8OGHH3L11VfTvn17AIoXL86ECROIiIigb9++vPLKKzzzzDPUqlULcI5MHn744bPmGRcXx6OPPsqqVasQEYYNG8btt99OyZIliYuLA2D27Nl8+eWXTJ8+nV69elG0aFHWrl3L1VdfzWeffca6deu44IILAKhevTo//PADBQoUIDIykt27dwMwbtw4rr766jOWffz4cTZs2ED9+vUBWLlyJY8//jgnT56kWLFivPvuu9SsWZPp06fz2WefERcXx+nTp/nuu+949dVX+fjjjzl16hS33norw4cPB+CWW25hz549nDx5kscff5w+ffr4Xb9pKVGiBK1atWLHjh0Zlps3b17KxTI9e/YkIiKC0aNHM2/ePLp3706RIkWoUqUK1apVY+XKlbRo0YLOnTtzzTXX8Mwzz5xXjP6wRAEsWbKEhx9+mKioKO666y6aNWvmdUjGZKvTp0+zZMmSlGaazZs306hRozPKXHnllcTFxXHs2DE2bdrkV1PTCy+8QJkyZdi4cSOAX00hMTEx/PTTT4SEhHD69GnmzJlD7969WbFiBVdccQUVK1bk7rvvpn///rRq1Yrdu3fToUMHtm7desZ8Vq1aRd26dVOGa9WqxbJlyyhYsCCLFy9m6NChfPrppwCsWbOGDRs2UK5cORYuXEhUVBQrV65EVencuTPff/89rVu35p133qFcuXLEx8fTpEkTbr/9dsqXL3/Gcvv378+333571vfq3r07gwcPzvT7p+Wvv/7ikksuAeDiiy/mr7/+AmDv3r00b948pVxoaCh79+4FoGzZspw6dYrY2NizYsxu+T5RPPHEE7zxxhtUq1aNhQsX2oOETECcy55/doqPj6dBgwbs3buXsLCwbF+/Fy9ezKxZs1KGy5Ytm+ln7rjjjpRelO+8805GjBhB7969mTVrFnfeeWfKfLds2ZLymWPHjhEXF0fJkiVTxu3bt48LL7wwZfjo0aP07NmTqKgoRISEhISUae3ataNcuXKA0x/bwoULadiwIeAcFUVFRdG6dWvGjx/PnDlzANizZw9RUVFnbYTHjh3rX+VkkYj4fS70oosu4o8//rBEEQhJSUmoKiEhITRt2pTnn3+eIUOGULRoUa9DMyZbFStWjHXr1vH333/ToUMHJk6cyGOPPUbt2rX5/vvvzygbHR1NyZIlKV26NHXq1GH16tUpzTrnyndDl/qa/hIlSqS8b9GiBTt27ODAgQPMnTuXZ599FnB+o8uXL8/wN1msWLEz5v3cc8/Rpk0b5syZw86dO4mIiEhzmarKkCFDeOihh86Y39KlS1m8eDE///wzxYsXJyIiIs37EQJxRFGxYkX27dvHJZdcwr59+7jooosAqFSpEnv27EkpFxMTQ6VKlVKGk5vZAi3fXfW0fv16WrZsycSJEwG4++67GT58uCUJk6cVL16c8ePH8/rrr5OYmEiPHj344YcfWLx4MeAceTz22GMMGjQIgIEDB/LSSy/x66+/As6Ge8qUKWfNt127dim/Jfi36alixYps3bqVpKSklD30tIgIt956K08++SRhYWEpe8bt27fnzTffTCm3bt26sz4bFhZ2Rtv/0aNHUzai06dPT3eZHTp04J133kk5h7J3717279/P0aNHKVu2LMWLF2fbtm0sX748zc+PHTuWdevWnfXKapIA6Ny5M++99x4A7733Hl26dEkZP2vWLE6dOsXvv/9OVFQUTZs2BZyE9+eff1K5cuUsL9df+SZRxMXFMWDAABo1akR0dDQXX3yx1yEZE1QNGzYkPDycmTNnUqxYMebNm8fIkSOpWbMm9erVo0mTJimXcIaHhzNu3DjuuusuwsLCqFu3LtHR0WfN89lnn+Xw4cPUrVuX+vXrp+xpv/zyy9x88820bNkype09PXfeeSczZsxIaXYCGD9+PKtWrSI8PJzatWunmaRq1arF0aNHOX78OACDBg1iyJAhNGzYkMTExHSX1759e+6++25atGhBvXr16Nq1K8ePH6djx44kJiYSFhbG4MGDzzg3cD4qV67Mk08+yfTp0wkNDU1pUnvggQdIflrn4MGDWbRoEdWrV2fx4sUpSadOnTp069aN2rVr07FjRyZOnJjSbLd69WqaN29OwYKBbxjKF49CXbx4Mb179yYmJoY+ffrw8ssv+9WWasz52Lp1K2FhYV6HkaeNHTuWUqVK8cADD3gdStA9/vjjdO7cmbZt2541La11zx6FmonChQtTrlw5fvzxR9566y1LEsbkEQ8//DBFihTxOgxP1K1bN80kEQh58ogiISGBcePGcfToUUaOHAk4bawFCuSLvGhyCDuiMF6xI4pM/PTTTzRq1IhBgwalnEwDLEkYT+S2HTGT+wVincszW89Dhw7Rp08frr76ao4cOcLcuXP59NNPLUEYzxQtWpTY2FhLFiZo1H0eRXZfxZln7qOIjY3lww8/5KmnnmLYsGFn3JhjjBdCQ0OJiYnhwIEDXodi8pHkJ9xlp1ydKLZv385HH33E888/T/Xq1dm1a1fA71A0xl+FChXK1qeMGeOVgLbLiEhHEdkuIjtE5Ky7UcQx3p2+QUSu8me+8fHxPP/884SHhzN27NiUOxctSRhjTPYL2BGFiIQAE4F2QAzwi4h8rqpbfIrdAFR3X82Aye7fdB07dox69erx22+/0aNHD15//XUqVqwYmC9hjDEmoEcUTYEdqhqtqv8As4Auqcp0Ad53u0tfDlwgIhnexvn7779ToEABFi9ezIwZMyxJGGNMgAXyHEUlYI/PcAxnHy2kVaYSsM+3kIj0AZI7hj8VFRW16frrr8/eaHOnCsBBr4PIIawu/mV18S+ri3/VzOoHc8XJbFWdCkwFEJFVWb1pJK+xuviX1cW/rC7+ZXXxLxE5t76PfASy6WkvcJnPcKg77lzLGGOM8VAgE8UvQHURqSIihYHuwOepynwO/Me9+qk5cFRV96WekTHGGO8ErOlJVRNFpB+wAAgB3lHVzSIS6U6fAswHbgR2AH8Dvf2Y9dQAhZwbWV38y+riX1YX/7K6+FeW6yLXdQpojDEmuKwjJGOMMRmyRGGMMSZDOTZRBKr7j9zIj7ro4dbBRhH5SUTqexFnMGRWFz7lmohIooh0DWZ8weRPXYhIhIisE5HNIvJdsGMMFj9+I2VE5AsRWe/WhT/nQ3MdEXlHRPaLyKZ0pmdtu6mqOe6Fc/L7N6AqUBhYD9ROVeZG4GtAgObACq/j9rAuWgJl3fc35Oe68Cn3Dc7FEl29jtvD9eICYAtwuTt8kddxe1gXQ4HR7vsLgUNAYa9jD0BdtAauAjalMz1L282cekQRkO4/cqlM60JVf1LVw+7gcpz7UfIif9YLgEeBT4H9wQwuyPypi7uBz1R1N4Cq5tX68KcuFCglIgKUxEkUicENM/BU9Xuc75aeLG03c2qiSK9rj3Mtkxec6/e8H2ePIS/KtC5EpBJwK04Hk3mZP+tFDaCsiCwVkdUi8p+gRRdc/tTFBCAM+APYCDyuqknBCS9HydJ2M1d04WH8IyJtcBJFK69j8dA44GlVTXJ2HvO1gkAjoC1QDPhZRJar6q/ehuWJDsA64DrgSmCRiCxT1WPehpU75NREYd1//Muv7yki4cA04AZVjQ1SbMHmT100Bma5SaICcKOIJKrq3OCEGDT+1EUMEKuqJ4ATIvI9UB/Ia4nCn7roDbysTkP9DhH5HagFrAxOiDlGlrabObXpybr/+FemdSEilwOfAffm8b3FTOtCVauoamVVrQzMBh7Jg0kC/PuNzANaiUhBESmO03vz1iDHGQz+1MVunCMrRKQiTk+q0UGNMmfI0nYzRx5RaOC6/8h1/KyL54HywCR3TzpR82CPmX7WRb7gT12o6lYR+R+wAUgCpqlqmpdN5mZ+rhcvANNFZCPOFT9Pq2qe635cRGYCEUAFEYkBhgGF4Py2m9aFhzHGmAzl1KYnY4wxOYQlCmOMMRmyRGGMMSZDliiMMcZkyBKFMcaYDFmiMDmOiJx2ezxNflXOoGzl9HrKPMdlLnV7H10vIj+KSM0szCMyuZsMEeklIpf6TJsmIrWzOc5fRKSBH595wr2PwpgssURhcqJ4VW3g89oZpOX2UNX6wHvAq+f6YffehffdwV7ApT7THlDVLdkS5b9xTsK/OJ8ALFGYLLNEYXIF98hhmYiscV8t0yhTR0RWukchG0Skujv+Hp/xb4lISCaL+x6o5n62rYisFedZH++ISBF3/MsissVdzmvuuP+KyFPiPAOjMfCBu8xi7pFAY/eoI2Xj7h55TMhinD/j06GbiEwWkVXiPG9huDvuMZyE9a2IfOuOay8iP7v1+ImIlMxkOSafs0RhcqJiPs1Oc9xx+4F2qnoVcCcwPo3PRQJvqGoDnA11jIiEueWvdsefBnpksvxOwEYRKQpMB+5U1Xo4PRk8LCLlcXqoraOq4cBI3w+r6mxgFc6efwNVjfeZ/Kn72WR34vRNlZU4OwK+3ZM8496RHw5cKyLhqjoep8fUNqraRkQqAM8C17t1uQp4MpPlmHwuR3bhYfK9eHdj6asQMMFtkz+N04V2aj8Dz4hIKM5zGKJEpC1OD6q/uN2bFCP951R8ICLxwE6cZ1rUBH736T/rPaAvTpfVJ4G3ReRL4Et/v5iqHhCRaLefnSicjul+dOd7LnEWxnmugm89dRORPji/60uA2jjdd/hq7o7/0V1OYZx6MyZdlihMbtEf+Aun99MCOBvqM6jqhyKyArgJmC8iD+H06/Oeqg7xYxk9VHVV8oCIlEurkNu3UFOcTua6Av1wuq/21yygG7ANmKOqKs5W2+84gdU45yfeBG4TkSrAU0ATVT0sItOBoml8VoBFqnrXOcRr8jlrejK5RRlgn/uwmXtxOn87g4hUBaLd5pZ5OE0wS4CuInKRW6aciFzh5zK3A5VFpJo7fC/wndumX0ZV5+MksLSeUX4cKJXOfOfgPGnsLpykwbnG6XaX/RzQXERqAaWBE8BRcXpHvSGdWJYDVyd/JxEpISJpHZ0Zk8IShcktJgE9RWQ9TnPNiTTKdAM2icg6oC7OIx+34LTJLxSRDcAinGaZTKnqSZzeNT9xex1NAqbgbHS/dOf3A2m38U8HpiSfzE4138M43X1foaor3XHnHKd77uN1YKCqrgfW4hylfIjTnJVsKvA/EflWVQ/gXJE1013Ozzj1aUy6rPdYY4wxGbIjCmOMMRmyRGGMMSZDliiMMcZkyBKFMcaYDFmiMMYYkyFLFMYYYzJkicIYY0yG/j/yI5uCz0ZiIgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred = model.predict(x_test)\n",
    "plot_roc(pred,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiclass Classification Error Metrics\n",
    "\n",
    "The following sections will examine several metrics for evaluating classification error. The following classification neural network will be used to evaluate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1500 samples, validate on 500 samples\n",
      "Epoch 1/1000\n",
      "1500/1500 - 0s - loss: 1.5313 - acc: 0.4627 - val_loss: 1.1385 - val_acc: 0.4980\n",
      "Epoch 2/1000\n",
      "1500/1500 - 0s - loss: 1.1401 - acc: 0.4780 - val_loss: 1.0856 - val_acc: 0.5460\n",
      "Epoch 3/1000\n",
      "1500/1500 - 0s - loss: 1.0477 - acc: 0.5320 - val_loss: 0.9720 - val_acc: 0.6180\n",
      "Epoch 4/1000\n",
      "1500/1500 - 0s - loss: 0.8949 - acc: 0.6273 - val_loss: 0.8497 - val_acc: 0.6680\n",
      "Epoch 5/1000\n",
      "1500/1500 - 0s - loss: 0.8082 - acc: 0.6780 - val_loss: 0.8116 - val_acc: 0.6840\n",
      "Epoch 6/1000\n",
      "1500/1500 - 0s - loss: 0.7640 - acc: 0.6920 - val_loss: 0.7951 - val_acc: 0.6760\n",
      "Epoch 7/1000\n",
      "1500/1500 - 0s - loss: 0.7541 - acc: 0.6873 - val_loss: 0.7828 - val_acc: 0.7000\n",
      "Epoch 8/1000\n",
      "1500/1500 - 0s - loss: 0.7395 - acc: 0.6967 - val_loss: 0.7811 - val_acc: 0.6980\n",
      "Epoch 9/1000\n",
      "1500/1500 - 0s - loss: 0.7301 - acc: 0.6947 - val_loss: 0.7613 - val_acc: 0.6900\n",
      "Epoch 10/1000\n",
      "1500/1500 - 0s - loss: 0.7095 - acc: 0.7000 - val_loss: 0.7541 - val_acc: 0.7000\n",
      "Epoch 11/1000\n",
      "1500/1500 - 0s - loss: 0.6918 - acc: 0.7093 - val_loss: 0.7680 - val_acc: 0.6800\n",
      "Epoch 12/1000\n",
      "1500/1500 - 0s - loss: 0.6873 - acc: 0.7073 - val_loss: 0.7513 - val_acc: 0.7020\n",
      "Epoch 13/1000\n",
      "1500/1500 - 0s - loss: 0.6811 - acc: 0.7093 - val_loss: 0.7720 - val_acc: 0.6680\n",
      "Epoch 14/1000\n",
      "1500/1500 - 0s - loss: 0.6732 - acc: 0.7087 - val_loss: 0.7444 - val_acc: 0.6960\n",
      "Epoch 15/1000\n",
      "1500/1500 - 0s - loss: 0.6703 - acc: 0.7093 - val_loss: 0.7518 - val_acc: 0.6940\n",
      "Epoch 16/1000\n",
      "1500/1500 - 0s - loss: 0.6653 - acc: 0.7147 - val_loss: 0.7559 - val_acc: 0.6900\n",
      "Epoch 17/1000\n",
      "1500/1500 - 0s - loss: 0.6625 - acc: 0.7160 - val_loss: 0.7521 - val_acc: 0.6960\n",
      "Epoch 18/1000\n",
      "1500/1500 - 0s - loss: 0.6653 - acc: 0.7260 - val_loss: 0.7367 - val_acc: 0.7060\n",
      "Epoch 19/1000\n",
      "1500/1500 - 0s - loss: 0.6807 - acc: 0.7033 - val_loss: 0.7396 - val_acc: 0.6900\n",
      "Epoch 20/1000\n",
      "1500/1500 - 0s - loss: 0.6458 - acc: 0.7213 - val_loss: 0.7363 - val_acc: 0.7100\n",
      "Epoch 21/1000\n",
      "1500/1500 - 0s - loss: 0.6413 - acc: 0.7293 - val_loss: 0.7375 - val_acc: 0.7120\n",
      "Epoch 22/1000\n",
      "1500/1500 - 0s - loss: 0.6474 - acc: 0.7253 - val_loss: 0.7528 - val_acc: 0.6980\n",
      "Epoch 23/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "1500/1500 - 0s - loss: 0.6350 - acc: 0.7293 - val_loss: 0.7680 - val_acc: 0.6880\n",
      "Epoch 00023: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f778c558da0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification neural network\n",
    "import numpy as np\n",
    "import tensorflow.keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(y.shape[1],activation='softmax',kernel_initializer='random_normal'))\n",
    "model.compile(loss='categorical_crossentropy', \n",
    "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
    "              metrics =['accuracy'])\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, \n",
    "                        verbose=1, mode='auto', restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Classification Accuracy\n",
    " \n",
    "Accuracy is the number of rows where the neural network correctly predicted the target class.  Accuracy is only used for classification, not regression.\n",
    "\n",
    "$ accuracy = \\frac{c}{N} $\n",
    "\n",
    "Where $c$ is the number correct and $N$ is the size of the evaluated set (training or validation). Higher accuracy numbers are desired.\n",
    "\n",
    "As we just saw, by default, Keras will return the percent probability for each class. We can change these prediction probabilities into the actual iris predicted with **argmax**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model.predict(x_test)\n",
    "pred = np.argmax(pred,axis=1) # raw probabilities to chosen class (highest probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the actual iris flower predicted, we can calculate the percent accuracy (how many were correctly classified)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score: 0.706\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "y_compare = np.argmax(y_test,axis=1) \n",
    "score = metrics.accuracy_score(y_compare, pred)\n",
    "print(\"Accuracy score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Classification Log Loss\n",
    "\n",
    "Accuracy is like a final exam with no partial credit.  However, neural networks can predict a probability of each of the target classes.  Neural networks will give high probabilities to predictions that are more likely.  Log loss is an error metric that penalizes confidence in wrong answers. Lower log loss values are desired.\n",
    "\n",
    "The following code shows the output of predict_proba:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numpy array of predictions\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0.    , 0.1085, 0.7909, 0.0994, 0.0011, 0.    , 0.    ],\n",
       "       [0.    , 0.8047, 0.1913, 0.    , 0.0038, 0.0001, 0.    ],\n",
       "       [0.    , 0.7322, 0.2608, 0.0001, 0.0066, 0.0002, 0.    ],\n",
       "       [0.    , 0.3643, 0.6275, 0.0045, 0.0036, 0.0001, 0.    ],\n",
       "       [0.    , 0.0268, 0.6301, 0.343 , 0.0002, 0.    , 0.    ]],\n",
       "      dtype=float32)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As percent probability\n",
      "[ 0.0006 10.8487 79.0939  9.9386  0.1146  0.0035  0.    ]\n",
      "Log loss score: 0.7366913137659431\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import display\n",
    "\n",
    "# Don't display numpy in scientific notation\n",
    "np.set_printoptions(precision=4)\n",
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "# Generate predictions\n",
    "pred = model.predict(x_test)\n",
    "\n",
    "print(\"Numpy array of predictions\")\n",
    "display(pred[0:5])\n",
    "\n",
    "print(\"As percent probability\")\n",
    "print(pred[0]*100)\n",
    "\n",
    "score = metrics.log_loss(y_test, pred)\n",
    "print(\"Log loss score: {}\".format(score))\n",
    "\n",
    "pred = np.argmax(pred,axis=1) # raw probabilities to chosen class (highest probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Log loss](https://www.kaggle.com/wiki/LogarithmicLoss) is calculated as follows:\n",
    "\n",
    "$ \\mbox{log loss} = -\\frac{1}{N}\\sum_{i=1}^N {( {y}_i\\log(\\hat{y}_i) + (1 - {y}_i)\\log(1 - \\hat{y}_i))} $\n",
    "\n",
    "The log function is useful to penalizing wrong answers.  The following code demonstrates the utility of the log function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/anaconda/lib/python3.6/site-packages/ipykernel_launcher.py:12: RuntimeWarning: divide by zero encountered in log\n",
      "  if sys.path[0] == '':\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAE0CAYAAAASSJRcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xuc1PV97/H3Z+/3XdgrsCwggshFDS5eiImLMVbbWJvrsWlM60likzRtbZukac21bdrGXk7ax+lp66NJT3qahKY1t0diTGIaNK2iAmoAEbkjsLAX2Pt95nP+mIGssLC/2dnf/mZ2X8/HYx4z8/vNb+YzH8cfb7585zvm7gIAAAAwvXKiLgAAAACYjQjaAAAAQAgI2gAAAEAICNoAAABACAjaAAAAQAgI2gAAAEAICNoAkOHM7LCZ3ToNz/NzZvbNgI99xszWpPuaADCXEbQBYO74rKQ/D/jYv5T0RyHWAgCzHkEbAOYAM9sgqdLdtwY85NuSNplZQ4hlAcCsRtAGgCxhZoVm9nkzO5G8fN7MCsft/6iZtSb3vdfM3MwuT+6+Q9Lj4x670cw6zGxx8v7VZnbGzFZJkrsPSdou6edm7h0CwOxC0AaA7PGApBskXSPpaknXSfq4JJnZ7ZJ+V9Ktki6X1HLesesk7T17x92flPSPkr5kZsWS/lXSJ9z9pXHH7Em+DgBgCgjaAJA9fkXSH7l7m7u3S/qMpHuS+94h6Z/dfbe7D0j69HnHVknqPW/bpyVVSnpG0nFJf3fe/t7kcQCAKSBoA0D2WCjpyLj7R5Lbzu57Zdy+8bcl6Yyk8vEb3H1U0v+VtFbSX7m7n3dMuaSu9EoGgLmLoA0A2eOEpCXj7jclt0lSq6TGcfsWn3fsTyWtHL/BzBZJ+pSkf5b0V+PneyddKemFNGsGgDmLoA0A2eOrkj5uZrVmViPpk0rMrZakr0m618yuNLMSSZ8479hHJN189o6ZmRKj2V+Q9B4lgvofj9tfJOlaST8M560AwOxH0AaA7PEnkrYpMTq9U9KO5Da5+/ck/a2kH0vaL+nsMn7Dyf07JHWb2fXJ7b8lqU6JL0C6pHuVCOqvS+6/U9IWdz87Yg4ASJFdOCUPAJDtzOxKSbskFbr7WHLbbZI+6O6/FOD4pyW9x913hVspAMxeBG0AmCXM7M1KTBEpkfQlSfEgoRoAEA6mjgDA7PHrktokHZAUk/SBaMsBgLmNEW0AAAAgBIxoAwAAACEgaAMAAAAhyIu6gFTU1NT40qVLI3v9/v5+lZaWRvb62Y7+TR29Sw/9Sw/9mzp6lx76lx76l57t27d3uHttOs+RVUF76dKl2rZtW2Svv2XLFrW0tET2+tmO/k0dvUsP/UsP/Zs6epce+pce+pceMzuS7nMwdQQAAAAIAUEbAAAACAFBGwAAAAgBQRsAAAAIAUEbAAAACAFBGwAAAAgBQRsAAAAIAUEbAAAACAFBGwAAAAgBQRsAAAAIAUEbAAAACAFBGwAAAAgBQRsAAAAIAUEbAAAACAFBGwAAAAgBQRsAAAAIAUEbAAAACAFBGwAAAAgBQRsAAAAIAUEbAAAACAFBGwAAAAgBQRsAAAAIAUEbAAAACAFBGwAAAAgBQRsAAAAIAUEbAAAACAFBGwAAAAgBQRsAAAAIAUEbAAAACAFBGwAAAAgBQRsAAAAIAUEbAAAACAFBGwAAAAhBpEHbzG43s71mtt/MPhZlLQAAAMB0iixom1mupL+TdIek1ZJ+2cxWR1UPAAAAMJ2iHNG+TtJ+dz/o7iOSNku6K8J6AADnGRmLazTuUZcBAFkpL8LXXiTplXH3j0m6PqJaAGBOiMddXYOj6uwbVkffiDr7h9XZN5K435+47uwbUWfyds/QmN51ZYHeGHXhAJCFzD2akQoze5uk2939vcn790i63t0/dN7j7pN0nyTV19dfu3nz5hmv9ay+vj6VlZVF9vrZjv5NHb1Lz2zun7trKCb1jrh6hl29o4nrnpHEpTd5ndiWeNxEZ32TVFYgVRSYKgpM5clLRYFpeemI1i6Ynf0L22z+7M0E+pce+peeTZs2bXf35nSeI8oR7eOSFo+735jc9iru/pCkhySpubnZW1paZqS4iWzZskVRvn62o39TR+/Sk239c3f1DI6pvW9Ibb3Daj976fvZ6HNixHlEHX3DGh6LT/g8ZYV5qi4rVHVpgRbVF6qmrEDVpYWqLitQdVmhakoT19VlBZpXUqDcHJvwebKtf5mE3qWH/qWH/kUvyqD9rKQVZrZMiYB9t6R3RlgPAIRqaDR2LjC39w6/OkQnt3ckb4/ELgzP+bmmmmQwri4t1OV1ZYn74wLzudulBSrKz43gXQIAzoosaLv7mJl9SNL3JeVK+qK7746qHgCYinjcdXpgRG09w2rrHXpVkB4foNt7h9U7NHbB8WZSdWmBasoKVVteqOW1paotL1Rt8n5teaHqygtVW1akiuI8mU086gwAyDxRjmjL3R+R9EiUNQDARNxdvcNjausZ0snuYZ3qGdKp3iGd6h7SqZ7hc7fbeoc1NsGqHGWFeecC85UNFXr9isILAnRteWLkOS+X3w4DgNko0qANAFE4O4XjZM+QTvUM6WQyMJ9/e2AkdsGx5UV5aqgoUn1FkW5YXn3udl15oeoqEiPPNeUFKing9AoAcx1/EgCYVQZHYjrRPajWriGd6B7Uye4hbd8zrH8++EwiSPcMqWtg9ILjCvJykqG5UGsWVuiWVXWqryhUfTJIN1QUqa6ikAANAAiMPzEAZI3hsZhOdg/pRNeQWrsH1dqdvO4a0onk7YlCdEWBtKR2RI3zStS8dJ7qy4tUX/mzAF1fUajK4nzmPwMAphVBG0BGcHe19w3r2JlBHTszqBNdg2rtGtSJ7sR0jtbuQXX0jVxwXFVJvhZUFmthZZHWN1VpYVWxFlQWJbZVJcL01v/+iVpaborgXQEA5jKCNoAZEY+72nqHdezMgI53DSYD9YCOnRnU8TODOtY1qJHz1oMuL8zTgqpEaF6zsEILKou1oKpIC5PXCyqLmMoBAMhY/AkFYFrE4q6TPUOJ0PyqAJ243do1dMHa0NWlBWqcV6xVC8p16+p6LaoqVuO8YjXOK9GCqiJVFOVH9G4AAEgfQRtAYP3DYzp6eiBx6RzQkdP9Onp6UEc7+3XszOAFy9zVlheqcV6x1i2q1B1rF2jRvESQXjyvWAurihmNBgDMavwpB+Ccs/Okj3YmwvSR5PXZ2x19w696fEVRnpZUl2rNokrdsW6BFs8rSY5IJ4I0v0wIAJjLCNrAHOPuau8d1sGOfh0ad3klGajHrx1tJi2sLNbi+cV6w6o6NVWXqGl+iZZUl2jJ/FJVljC1AwCAiyFoA7NUz9CoDrUnQvTPQnWfDrX3q39cmC7Iy9GSZHjeuLxGS5Jhuqk6MTpdmMeoNAAAU0HQBrLYaCyuI5392t82LkgnQ/X4pfByTGqcV6JlNaVqXjJfl9WWallN4rKgsli5OawfDQDAdCNoA1lgJOZ68USP9rX16kBbn/YlL4c7+l/1BcTa8kItqynVrVfWnwvSl9WWavH8EkamAQCYYQRtIIP0DY+dC9L72/q0v61X+9r6dLRzQP7Dn0hKjE4vrS7V5XVlum11vVbUl2l5bZmW1ZSqnOXwAADIGARtIAKjsbgOtvfrpZM9eulkr15q7dHLp/p0vGvw3GPyc02X1ZRp7aJKXVM1qjdev1Yr6sq1tIbRaQAAsgFBGwiRe+LXEPe0JgL13pO92tPaowPtfRqNJaZ85OealteWacPSeXpnfZMuryvTiroyNc0vUV5ujiRpy5YtarlqYZRvBQAApIigDUyTkbG49rX1avfxHr3Y2nNutLprYPTcYxZUFumKhnK1XFGnKxeU64qGcl1WU6aCvJwIKwcAAGEgaANTMDQa096Tvdp1olu7jndr1/Ee7T3Ze+4nxksKcrWyvlx3rG3QqoYKXdFQrlUN5aoqKYi4cgAAMFMI2sAk+ofHtKe1JxGoTySu97X1KZZc7aOyOF9rF1Xo3tcu1ZpFlVq7sEJLq0uVw5J5AADMaQRtYJxY3LWvrVfPH+3S868kLi+f6tXZFfSqSwu0dlGl3nBlndYtqtSahZVqnFcsM0I1AAB4NYI25rRTPUN67lyoPqOdx7rP/WpiZXG+rl5cpdtW12tdY5XWLapUfUUhoRoAAARC0MacMRaL68XWHj17+Iy2Hzmt5452qbV7SFJi5Y/VCyr01msbdc3iKl2zuErLakoJ1QAAYMoI2pi1+ofH9PwrXXr28GltO3xGO46e0UBytLpxXrE2LJ2fCNVNVVq9oEJF+axNDQAApg9BG7PGmf4RPX2oU88ePqNth09r14kexeIuM+nKhgq9/dpGbVg2X81L5quhsijqcgEAwCxH0EbW6h0a1TOHTuvJA5168kCn9rT2SJIK83L0mqYqfbBluZqXztdrmqpUwU+TAwCAGUbQRtYYHIlp+5EzevJAh5480Kmdx7sVi7sK83LUvHSePnzbSt24vFrrFlXxAzAAACByBG1kLHfX3lO9enxvux5/uV3bDp/RSCyuvBzT1YsTI9Y3Lq/W+qZ5zK8GAAAZh6CNjNI9OKr/3t9xLlyf7EmsCrKqoVy/unGJNl5eow1L56uskI8uAADIbKQVRMrd9fKpPj2255S27G3TjqNdisVd5YV5umlFjVquqNXrV9ZqQWVx1KUCAACkhKCNGReLu7YfOaMf7D6pH+45pSOdA5KkNQsr9P6bL9PNK+v0mqYq5ecyzxoAAGQvgjZmxOBITM+1jem7//6CfvRSm073j6ggN0c3Lq/W+153md64ul71FSy5BwAAZg+CNkIzNBrTlr3t+s5PT+hHe9o0OBpTedFJ3bKqTm9cXa+bV9aqnGX3AADALEXQxrQaGYvrv/a36zsvtOoHL55S3/CY5pcW6M3rF2lRrE3v+6VNLL0HAADmBII20ubueubQaX3jueP63q6T6h4cVUVRnn5+XYPedNVCbVxerbzcHG3ZsoWQDQAA5gyCNqbsldMD+vqO43p4xzEdPT2gkoJc/dyaBr3pqgV63YpaQjUAAJjTCNpIycDImL6386T+Y/sxPXWwU5K0cXm17r91hW5f26CSAj5SAAAAEkEbAe1v69W/bj2qh7cfU+/wmJrml+h337hSb37NIi2eXxJ1eQAAABmHoI2LGhmL6wcvntS/bj2irQdPqyA3R3esa9A7r2vSdcvmy8yiLhEAACBjEbRxgc6+Yf3LU0f0lWeOqr13WI3zivX7t6/S25sbVVNWGHV5AAAAWYGgjXMOtPfpC/91SA9vP6bhsbhuWVWne25YotevrFVuDqPXAAAAqSBoQ88ePq2Hnjiox/acUn5ujt66vlHvuWmZLq8ri7o0AACArEXQnsOeOtCpzz/2sp4+dFrzSvL1m7es0LtvXML0EAAAgGlA0J5j3F1PHezU5x/bp2cOnVZdeaE+dedq3b2hScUFuVGXBwAAMGsQtOeQ7UdO63OP7j0XsD9952rdfV2TivIJ2AAAANMtkqBtZn8h6U5JI5IOSLrX3buiqGUuONTRrwcffUnf23VSteWF+swvrtH/2LCYgA0AABCiqEa0fyjpD9x9zMw+J+kPJP1+RLXMWp19w/rbH+3Tl58+qoK8HP3OrSv1vtcv49cbAQAAZkAkicvdfzDu7lZJb4uijtkqFnd9+ekj+ovv79XASEx3b1is+29dqdpyvuQIAAAwUzJhaPN/Svq3qIuYLZ47ekaf+NYu7Treo9deXq3P/OIaXV5XHnVZAAAAc465ezhPbPaYpIYJdj3g7t9KPuYBSc2S3uIXKcTM7pN0nyTV19dfu3nz5lDqDaKvr09lZZm5tvTAqOtrL4/o8VfGVFlo+uVVBbquITejfiY9k/uX6ehdeuhfeujf1NG79NC/9NC/9GzatGm7uzen8xyhBe1JX9js1yT9uqQ3uPtAkGOam5t927ZtodZ1KVu2bFFLS0tkr38xj7/cro89/FOd6hnSva9dpvtvXaHyovyoy7pApvYvG9C79NC/9NC/qaN36aF/6aF/6TGztIN2VKuO3C7po5JuDhqycaHeoVF99rt7tPnZV3R5XZm+/sHX6prFVVGXBQAAAEU3R/t/SyqU9MPk1Iat7v7+iGrJStuPnNFvffU5tXYP6v03L9f9t65guT4AAIAMEtWqI5dH8bqzQTzu+scnDuovf7BXC6uK9B8f2Kj1TfOiLgsAAADnyYRVRxBQZ9+wfudrL+iJl9v1C1ct0J+9ZZ0qMnAuNgAAAAjaWWP3iW6970vb1NE/os++ea3eeV1TRq0oAgAAgFcjaGeBR3a26ve+9oKqSvL18Ps3al1jZdQlAQAAYBIE7Qzm7vqbH+3T5x/bp/VNVfqHe65VXXlR1GUBAAAgAIJ2horFXR//5i599Zmjeuv6Rv3pW9aqMI9VRQAAALIFQTsDDY3GdP/m5/Xo7pP6jU3L9eHbrmA+NgAAQJYhaGeYwZGY3vOlZ/XkgU594k2r9Z6blkVdEgAAAKaAoJ1BhkYTIXvrwU799Tuu1lvWN0ZdEgAAAKaIoJ0hhkZjet+/bNNTBzv1V28nZAMAAGS7nKgLgDQWi+tDX9mhn+zr0OfeehUhGwAAYBYgaEfM3fXJb+/WY3va9Md3rdE7mhdHXRIAAACmAUE7Yn//+AF95emj+kDLct1z49KoywEAAMA0IWhH6JGdrXrw0b2665qF+shtV0RdDgAAAKYRQTsi+9t69eF/f0Hrm6r04NuuUk4O62QDAADMJgTtCPQOjeq+/7ddJQV5+vt3XcsvPgIAAMxCLO83w9xdH/v6Th3pHNCX33u96iuKoi4JAAAAIWBEe4Z947nj+u5PW/V7t63UDZdVR10OAAAAQkLQnkHHuwb1qW/t1oal8/Trr18edTkAAAAIEUF7hri7Pvy1FxR311+/4xrl8uVHAACAWY2gPUMe3nFcTx3s1AO/sFqL55dEXQ4AAABCRtCeAV0DI/rTR/ZofVOV7t7ALz8CAADMBQTtGfC5R/eqe3BUn33zOtbLBgAAmCMI2iHb09qjzc8e1a/euFRXLqiIuhwAAADMEIJ2yB589CWVF+bpt9+wIupSAAAAMIMI2iHaerBTP97brg9uulyVJflRlwMAAIAZRNAOibvrwUdfUkNFkX5t49KoywEAAMAMmzRom9lvmtm8mShmNtl68LR2HO3Sb2xarqL83KjLAQAAwAwLMqJdL+lZM/uamd1uZiybEcDfP35ANWUFenszy/kBAADMRZMGbXf/uKQVkr4g6dck7TOzPzUzfkP8Inaf6NYTL7fr3tcuYzQbAABgjgo0R9vdXdLJ5GVM0jxJ/2FmD4ZYW9b6p58cUllhnt51w5KoSwEAAEBE8iZ7gJn9tqR3S+qQ9E+SPuLuo2aWI2mfpI+GW2J2OdM/ou/ubNXdGxarspiVRgAAAOaqSYO2pPmS3uLuR8ZvdPe4mb0pnLKy18M7jmlkLK53Xt8UdSkAAACI0KRB290/dYl9e6a3nOzm7vrKM0e1vqlKqxr4FUgAAIC5jHW0p9G2I2d0sL1f77yeudkAAABzHUF7Gn37+RMqys/RHWsboi4FAAAAESNoT5OxWFyP7GzVG66sV2lhkKnvAAAAmM0I2tPkqYOd6uwf0Z1XLYy6FAAAAGQAgvY0+c4LrSorzFPLFbVRlwIAAIAMQNCeBvG460cvtanlilp+CRIAAACSCNrTYveJHnX0DeuWVXVRlwIAAIAMQdCeBj/e2yYz6eaVTBsBAABAAkF7Gvx4b5uubqxSdVlh1KUAAAAgQxC009Q9MKrnX+liNBsAAACvEmnQNrPfMzM3s5oo60jHtiOn5S7duLw66lIAAACQQSIL2ma2WNJtko5GVcN0eObQaRXk5uiaxVVRlwIAAIAMEuWI9v+S9FFJHmENaXvm8Gld1VjJsn4AAAB4lUiCtpndJem4u78QxetPl4GRMe081q3rls2PuhQAAABkGHMPZ0DZzB6T1DDBrgck/aGk29y928wOS2p2946LPM99ku6TpPr6+ms3b94cSr1B9PX1qays7Nz9PZ0xfe7ZIf3OtYW6ujYvsrqyxfn9Q3D0Lj30Lz30b+roXXroX3roX3o2bdq03d2b03mO0NKhu9860XYzWydpmaQXzEySGiXtMLPr3P3kBM/zkKSHJKm5udlbWlrCKnlSW7Zs0fjX3/fEQUl79K47XsfSfgGc3z8ER+/SQ//SQ/+mjt6lh/6lh/5Fb8aHYd19p6RzP6E42Yh2Jtt5vFsLK4sI2QAAALgA62inYdeJbq1dVBl1GQAAAMhAkQdtd1+ajaPZfcNjOtTRT9AGAADAhCIP2tnqxRM9cpfWEbQBAAAwAYL2FO091StJWrWgPOJKAAAAkIkI2lN0oK1PpQW5aqgoiroUAAAAZCCC9hQdaO/T8royJZcoBAAAAF6FoD1FB9r6tLyWReABAAAwMYL2FPQPj+lE95CW15ZGXQoAAAAyFEF7Cg519EsSI9oAAAC4KIL2FJwN2ktrGNEGAADAxAjaU3DszKAkqXFeccSVAAAAIFMRtKfgeNeAKovzVV6UH3UpAAAAyFAE7Sk4fmZQi6oYzQYAAMDFEbSn4HjXoBYxbQQAAACXQNBOkbszog0AAIBJEbRT1D04qv6RGF+EBAAAwCURtFN0omtIkrSQEW0AAABcAkE7RR19w5Kk2vLCiCsBAABAJiNop6i9NxG0a8oI2gAAALg4gnaKzo5o15QVRFwJAAAAMhlBO0UdfcMqys9RWWFe1KUAAAAggxG0U9TRN6KaskKZWdSlAAAAIIMRtFPU0TfM/GwAAABMiqCdovZegjYAAAAmR9BOUWLqCF+EBAAAwKURtFPg7uoZHFVlSX7UpQAAACDDEbRTMBqXRmJxVRYTtAEAAHBpBO0U9I+6JBG0AQAAMCmCdgoGxhLXFUUEbQAAAFwaQTsFA4xoAwAAICCCdgrOTh2pIGgDAABgEgTtFJydOsKINgAAACZD0E7B2akjFUV5EVcCAACATEfQTgFTRwAAABAUQTsFg2Ou4vxc5efSNgAAAFwaiTEFwzGppCA36jIAAACQBQjaKRiOScUEbQAAAARA0E7BcMwZ0QYAAEAgBO0UJEa0WXEEAAAAkyNop2Ak5irJZ0QbAAAAkyNop4A52gAAAAiKoJ2C4ZgTtAEAABAIQTsFIzExdQQAAACBELRTwKojAAAACIqgnYLhMVYdAQAAQDCRBW0z+00ze8nMdpvZg1HVEdRYLK4xl4qZOgIAAIAAIhmeNbNNku6SdLW7D5tZXRR1pGJwNCZJKi7gHwEAAAAwuahS4wck/bm7D0uSu7dFVEdgI2NxSVJhHiPaAAAAmFxUQXulpNeZ2dNm9riZbYiojsBGYy5Jys9lRBsAAACTM3cP54nNHpPUMMGuByR9VtKPJf2WpA2S/k3SZT5BMWZ2n6T7JKm+vv7azZs3h1LvZNoH4vrIE4N677oC3bQoP5Iasl1fX5/KysqiLiMr0bv00L/00L+po3fpoX/poX/p2bRp03Z3b07nOUKbo+3ut15sn5l9QNLXk8H6GTOLS6qR1D7B8zwk6SFJam5u9paWlnAKnsT+tj7pice1bs1qtVyzKJIast2WLVsU1X+/bEfv0kP/0kP/po7epYf+pYf+RS+qeRDflLRJksxspaQCSR0R1RLIaCwxR7uAqSMAAAAIIKpFob8o6YtmtkvSiKRfnWjaSCY5F7TzCNoAAACYXCRB291HJL0riteeqrOrjvBlSAAAAARBagxoJEbQBgAAQHCkxoDOLu/H1BEAAAAEQWoMaHSML0MCAAAgOFJjQOemjuRZxJUAAAAgGxC0A2J5PwAAAKSC1BgQq44AAAAgFaTGgEZYRxsAAAApIDUGxJchAQAAkApSY0Bnl/fLy+XLkAAAAJgcQTugsXgyaOfQMgAAAEyO1BhQ3BNBm5wNAACAIIiNAcWSI9q5xtQRAAAATI6gHdC5oJ1D0AYAAMDkCNoBxd1lkowRbQAAAARA0A4oFncxmA0AAICgCNoBxdzFYDYAAACCImgHFGdEGwAAACkgaAcUi9MsAAAABEd2DCjujGgDAAAgOIJ2QHwZEgAAAKkgaAcUY0QbAAAAKSBoB5T4MiRJGwAAAMEQtANi6ggAAABSQdAOKJb8ZUgAAAAgCIJ2QKyjDQAAgFQQtAOKuQjaAAAACIygHRAj2gAAAEgFQTsgvgwJAACAVBC0A0qso03SBgAAQDAE7YCYOgIAAIBUELQDirnTLAAAAARGdgwoFncxcwQAAABBEbQDcpb3AwAAQAoI2gHF+WVIAAAApICgHZC7mDoCAACAwAjaAbk86hIAAACQRQjaKWBAGwAAAEERtANyBrQBAACQAoJ2QORsAAAApIKgnQK+DAkAAICgCNpBMaQNAACAFBC0A3KxjjYAAACCI2gHxJchAQAAkIpIgraZXWNmW83seTPbZmbXRVFHqpijDQAAgKCiGtF+UNJn3P0aSZ9M3s9oDGgDAAAgFVEFbZdUkbxdKelERHUE5s4cbQAAAASXF9Hr3i/p+2b2l0qE/Y0R1QEAAACEwjykb/mZ2WOSGibY9YCkN0h63N0fNrN3SLrP3W+9yPPcJ+m+5N0rJO0No96AaiR1RPj62Y7+TR29Sw/9Sw/9mzp6lx76lx76l54r3L08nScILWhf8kXNuiVVububmUnqdveKyY6Lmpltc/fmqOvIVvRv6uhdeuhfeujf1NG79NC/9NC/9ExH/6Kao31C0s3J27dI2hdRHQAAAEAoopqj/T5Jf2NmeZKG9LOpIQAAAMCsEEnQdvf/knRtFK+dpoeiLiDL0b+po3fpoX/poX9TR+/SQ//SQ//Sk3b/IpmjDQAAAMx2/AQ7AAAAEAKCtiQzu93M9prZfjP72AT7zcz+Nrn/p2a2Puixc0GA/v1Ksm87zexJM7t63L7Dye3Pm9m2ma08MwToX4uZdSd79LyZfTLosbNdgN59ZFzfdplZzMzmJ/fx2TP7opm1mdmui+zn3HcRAXrHee8SAvSP894lBOgf576LMLPFZvZjM3vRzHab2W9P8JjpO/e5+5y+SMqVdECTD+LFAAAETElEQVTSZZIKJL0gafV5j/l5Sd+TZJJukPR00GNn+yVg/zZKmpe8fcfZ/iXvH5ZUE/X7yPD+tUj6zlSOnc2XVN+/pDsl/ee4+3P6s5fsweslrZe06yL7OfdNvXec99LrH+e9NPp33mM59726HwskrU/eLpf0cpi5jxFt6TpJ+939oLuPSNos6a7zHnOXpH/xhK2SqsxsQcBjZ7tJe+DuT7r7meTdrZIaZ7jGTJbOZ2iuf/5Sff+/LOmrM1JZlnD3JySdvsRDOPddxGS947x3aQE+excz5z97Usr949w3jru3uvuO5O1eSXskLTrvYdN27iNoJ5r7yrj7x3Rhwy/2mCDHznap9uA9Svwt8SyX9JiZbbfEr4DONUH7tzH5z1ffM7M1KR47WwV+/2ZWIul2SQ+P2zzXP3tBcO6bHpz3pobzXpo4912amS2V9BpJT5+3a9rOfVGto405yMw2KfEHzk3jNt/k7sfNrE7SD83speTf1PEzOyQ1uXufmf28pG9KWhFxTdnmTkn/7e7jR4D47CF0nPemjPPe9ODcdxFmVqbEX0Dud/eesF6HEW3puKTF4+43JrcFeUyQY2e7QD0ws6sk/ZOku9y98+x2dz+evG6T9A0l/llmLpm0f+7e4+59yduPSMo3s5ogx85yqbz/u3XeP53y2QuEc18aOO9NHee9acO5bwJmlq9EyP6yu399godM27mPoC09K2mFmS0zswIlPpTfPu8x35b07uS3UG+Q1O3urQGPne0m7YGZNUn6uqR73P3lcdtLzaz87G1Jt0ma8BvUs1iQ/jWYmSVvX6fE/7edQY6d5QK9fzOrlHSzpG+N28ZnLxjOfVPEeS89nPfSx7lvYsnP1Rck7XH3v77Iw6bt3Dfnp464+5iZfUjS95X4NukX3X23mb0/uf8fJD2ixDdQ90sakHTvpY6N4G1EJmD/PimpWtL/SZ43x9y9WVK9pG8kt+VJ+oq7PxrB24hMwP69TdIHzGxM0qCkuz3x9ec5/fkL2DtJerOkH7h7/7jD5/xnT5LM7KtKrO5QY2bHJH1KUr7EuW8yAXrHee8SAvSP894lBOifxLnvYl4r6R5JO83s+eS2P5TUJE3/uY9fhgQAAABCwNQRAAAAIAQEbQAAACAEBG0AAAAgBARtAAAAIAQEbQAAACAEBG0AAAAgBARtAAAAIAQEbQCYhcxsg5n91MyKkr8Gt9vM1kZdFwDMJfxgDQDMUmb2J5KKJBVLOubufxZxSQAwpxC0AWCWMrMCSc9KGpK00d1jEZcEAHMKU0cAYPaqllQmqVyJkW0AwAxiRBsAZikz+7akzZKWSVrg7h+KuCQAmFPyoi4AADD9zOzdkkbd/StmlivpSTO7xd3/M+raAGCuYEQbAAAACAFztAEAAIAQELQBAACAEBC0AQAAgBAQtAEAAIAQELQBAACAEBC0AQAAgBAQtAEAAIAQELQBAACAEPx/hYJokx1wxAIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib.pyplot import figure, show\n",
    "from numpy import arange, sin, pi\n",
    "\n",
    "#t = arange(1e-5, 5.0, 0.00001)\n",
    "#t = arange(1.0, 5.0, 0.00001) # computer scientists\n",
    "t = arange(0.0, 1.0, 0.00001)  # data     scientists\n",
    "\n",
    "fig = figure(1,figsize=(12, 10))\n",
    "\n",
    "ax1 = fig.add_subplot(211)\n",
    "ax1.plot(t, np.log(t))\n",
    "ax1.grid(True)\n",
    "ax1.set_ylim((-8, 1.5))\n",
    "ax1.set_xlim((-0.1, 2))\n",
    "ax1.set_xlabel('x')\n",
    "ax1.set_ylabel('y')\n",
    "ax1.set_title('log(x)')\n",
    "\n",
    "show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[ 21   1   0   0   0   0   0]\n",
      " [  5 197  47   0   0   0   0]\n",
      " [  0  54 133   0   0   0   0]\n",
      " [  0   0  12   2   0   0   0]\n",
      " [  0   8   0   0   0   0   0]\n",
      " [ 10   7   0   0   0   0   0]\n",
      " [  3   0   0   0   0   0   0]]\n",
      "Normalized confusion matrix\n",
      "[[0.95 0.05 0.   0.   0.   0.   0.  ]\n",
      " [0.02 0.79 0.19 0.   0.   0.   0.  ]\n",
      " [0.   0.29 0.71 0.   0.   0.   0.  ]\n",
      " [0.   0.   0.86 0.14 0.   0.   0.  ]\n",
      " [0.   1.   0.   0.   0.   0.   0.  ]\n",
      " [0.59 0.41 0.   0.   0.   0.   0.  ]\n",
      " [1.   0.   0.   0.   0.   0.   0.  ]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAEmCAYAAADr3bIaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAH6hJREFUeJzt3XucHFWd9/HPdxLCxSCXJAYICQENKPBIJJFVEIyCCC5XFeQiiqIhoKCyrILighceXFF0FbyEBcEbgmKUBVxEXEV4QElCiISbBMkDGEkCCsgl5PLbP+oMNMNMd/VMV1dXz/fNq17TXV19zi8BfnPOqXNOKSIwM7P6esoOwMysCpwszcxycLI0M8vBydLMLAcnSzOzHJwszcxycLIcRiStL+m/JD0m6cdDKOdISb9sZWxlkbS7pLvLjsM6nzzPsvNIOgI4CXgl8ASwADgzIm4YYrlHAScAu0bE6iEH2uEkBTAlIu4tOxarPrcsO4ykk4CvAv8XGA9MAs4DDmhB8VsB9wyHRJmHpJFlx2AVEhE+OuQANgL+ARxS55p1yZLpX9LxVWDd9NkM4EHgX4BlwFLgfemzzwDPAqtSHccAZwDfryl7MhDAyPT+aOA+stbtn4Eja87fUPO9XYFbgMfSz11rPvsN8DngxlTOL4GxA/zZeuP/eE38BwFvA+4BHgU+WXP9LsBNwN/TtecCo9Jn16c/y5Ppz/uumvI/AfwV+F7vufSdl6c6dk7vtwCWAzPK/m/DR/mHW5ad5fXAesCcOtd8CngdMBXYiSxhnFbz+WZkSXcCWUI8T9ImEXE6WWv10ogYHREX1AtE0kuArwH7RsSGZAlxQT/XbQpcla4dA5wDXCVpTM1lRwDvA14GjAJOrlP1ZmR/BxOAfwPOB94NTAN2Bz4taet07RrgY8BYsr+7PYHjASJij3TNTunPe2lN+ZuStbJn1lYcEYvJEun3JW0AfAe4OCJ+UydeGyacLDvLGGBF1O8mHwl8NiKWRcRyshbjUTWfr0qfr4qIq8laVdsNMp61wI6S1o+IpRGxqJ9r/hn4U0R8LyJWR8QlwF3A/jXXfCci7omIp4HLyBL9QFaRjc+uAn5Elgj/IyKeSPXfQfZLgoiYFxE3p3rvB74NvDHHn+n0iFiZ4nmBiDgfuBf4PbA52S8nMyfLDvMIMLbBWNoWwJKa90vSuefK6JNsnwJGNxtIRDxJ1nWdBSyVdJWkV+aIpzemCTXv/9pEPI9ExJr0ujeZPVzz+dO935e0raQrJf1V0uNkLeexdcoGWB4RzzS45nxgR+DrEbGywbU2TDhZdpabgJVk43QD+QtZF7LXpHRuMJ4ENqh5v1nthxFxTUS8hayFdRdZEmkUT29MDw0ypmZ8kyyuKRHxUuCTgBp8p+70D0mjycaBLwDOSMMMZk6WnSQiHiMbpztP0kGSNpC0jqR9JX0xXXYJcJqkcZLGpuu/P8gqFwB7SJokaSPg1N4PJI2XdGAau1xJ1p1f208ZVwPbSjpC0khJ7wK2B64cZEzN2BB4HPhHavUe1+fzh4FtmizzP4C5EfEBsrHYbw05SusKTpYdJiK+TDbH8jSyO7EPAB8GfpYu+TwwF1gI/BGYn84Npq5rgUtTWfN4YYLrSXH8hewO8Rt5cTIiIh4B9iO7A/8I2Z3s/SJixWBiatLJZDePniBr9V7a5/MzgIsl/V3SoY0Kk3QgsA/P/zlPAnaWdGTLIrbK8qR0M7Mc3LI0M8vBydLMLAcnSzOzHJwszcxy6KiNBDYdMza2nNh3yl57jRzRaJpe8cqPwOyFliy5nxUrVrT0P80RL90qYvWLFlENKJ5efk1E7NPKGJrRUclyy4lbceWv/1+pMYzbcFSp9QNITpfWWXb7p+ktLzNWP8262zWc0fWcZxac12h1VqE6Klma2XAiUHVGAp0szawcAirUi3KyNLPyuGVpZtaIoGdE2UHk5mRpZuVxN9zMrAHhbriZWWNyy9LMLBe3LM3McnDL0sysEU9KNzNrzJPSzcxycsvSzKwRwQhPSjczq69i8ywLjVTSzyTNk7RI0swi6zKzCpLyHyUrumX5/oh4VNL6wC2SLk+PTn1OSqIzASZsObHgcMysc1TrbnjRkZ4o6TbgZmAiMKXvBRExOyKmR8T0TceMKzgcM+soblmCpBnAXsDrI+IpSb8B1iuqPjOroAq1LIvshm8E/C0lylcCryuwLjOrmg5pMeZVZLL8b2CWpDuBu8m64mZmz3PLEiJiJbBvUeWbWRdwy9LMrJHW3g2XdCGwH7AsInZM5y4FtkuXbAz8PSKmSpoM9PZ6AW6OiFn1yneyNLNyiFY/VuIi4Fzgu70nIuJdz1UnfRl4rOb6xRExNW/hTpZmVpLWtiwj4vrUYnxxTZKAQ4E3D7b86oyumln3aW6e5VhJc2uOZlYF7g48HBF/qjm3taQFkn4rafdGBbhlaWblaa5luSIipg+ypsOBS2reLwUmRcQjkqYBP5O0Q0Q8PlABTpZmVp423A2XNBJ4OzCt91yarbMyvZ4naTGwLTB3oHKcLM2sHGrb2vC9gLsi4sHnq9Y44NGIWCNpG7Kl2PfVK8RjlmZWnhauDZd0CXATsJ2kByUdkz46jBd2wQH2ABZKWgD8BJgVEY/WK98tSzMrjVrYDY+Iwwc4f3Q/5y4HLm+mfCdLMytF9gger+AZlJEjxCYbrFNqDJvuckKp9QP85iefLzsEdtpq47JDsG4noR4nSzOzhtyyNDPLwcnSzCwHJ0szs0aUjopwsjSzUgi5ZWlmloeTpZlZDk6WZmY5OFmamTXiGzxmZo0J0dNTnb18nCzNrDTuhpuZ5VGdXOlkaWYlUbValoUNGEiaLOn2oso3s+qTlPsom1uWZlaaTkiCeRV9K2qkpB9IulPSTyRtUHB9ZlYRvcsdq9KyLDpZbgd8IyJeBTwOHN/3Akkze58DvGL58oLDMbOOoiaOkhWdLB+IiBvT6+8Db+h7QUTMjojpETF97LhxBYdjZh1DHrOsFQ3em9kw1glJMK+iW5aTJL0+vT4CuKHg+sysQtSj3EfDsqQLJS2rnYUj6QxJD0lakI631Xx2qqR7Jd0t6a2Nyi86Wd4NfEjSncAmwDcLrs/MKqTF3fCLgH36Of+ViJiajqtTvduTPU98h/Sdb0gaUa/wwrrhEXE/8Mqiyjezamv1WGREXC9pcs7LDwR+FBErgT9LuhfYBbhpoC9UZxW7mXWdJluWY3tnzqRjZs5qTpC0MHXTN0nnJgAP1FzzYDo3IE9KN7PSNNmyXBER05us4pvA58huLn8O+DLw/ibLANyyNLMyFTzPMiIejog1EbEWOJ+sqw3wEDCx5tIt07kBOVmaWWmKnmcpafOatwcDvXfKrwAOk7SupK2BKcAf6pXlbriZlaPFuw5JugSYQTa2+SBwOjBD0lSybvj9wLEAEbFI0mXAHcBq4EMRsaZe+U6WZlYKAa2ckx4Rh/dz+oI6158JnJm3fCdLMyuJ6Mkx2bxTOFmaWWmqtNzRydLMyqHWdsOL5mRpZqUQuBs+WALWGVnubKarL/lMqfUDfPhHC8oOgd99YkbZIdgw4JalmVkOHrM0M2vEY5ZmZo1l8yyrky2dLM2sJJ3xuIi8nCzNrDQVypVOlmZWEnnqkJlZQx6zNDPLqUK50snSzMrjlqWZWQ4VypVOlmZWkhZv/ls0J0szK0WrN/8tmpOlmZXEk9LNzHKpUK50sjSzklRsUnrhm0dKeo+khZJuk/S9ouszs2ronZTeqkfhSrpQ0jJJt9ecO1vSXSkHzZG0cTo/WdLTkhak41uNyi80WUraATgNeHNE7AR8pJ9rZkqaK2nu8hXLiwzHzDpMi58bfhGwT59z1wI7RsSrgXuAU2s+WxwRU9Mxq1HhRbcs3wz8OCJWAETEo30viIjZETE9IqaPGzuu4HDMrJNI+Y9GIuJ64NE+534ZEavT25uBLQcba7nPcDCzYa3JluXY3l5oOmY2Wd37gV/UvN86dcF/K2n3Rl8u+gbPr4E5ks6JiEckbdpf69LMhqHmd0pfERHTB1WV9ClgNfCDdGopMCnlpWnAzyTtEBGPD1RGockyIhZJOhP4raQ1wK3A0UXWaWbVoDbNs5R0NLAfsGdEBEBErARWptfzJC0GtgXmDlRO4VOHIuJi4OKi6zGz6ik6V0raB/g48MaIeKrm/Djg0YhYI2kbYApwX72yPM/SzErT08JsKekSYAbZ2OaDwOlkd7/XBa5Nrdib053vPYDPSloFrAVmNRoidLI0s9K0smUZEYf3c/qCAa69HLi8mfKdLM2sFBKMqNAKHidLMytNV2ykIeml9b5Y7xa7mVkeFcqVdVuWi4AgW8LZq/d9AJMKjMvMupzIpg9VxYDJMiImtjMQMxt+KjRkmW+5o6TDJH0yvd4yzXg3Mxu8JpY6dsLYZsNkKelc4E3AUenUU0DD7YzMzBpp5UYaRctzN3zXiNhZ0q2Q7RwkaVTBcZlZlxOtnZRetDzJcpWkHrKbOkgaQzbjvSu9/hVjyg6BX3zkDWWHwNq1UXYIldpF2wanQrky15jleWQz3cdJ+gxwA/DvhUZlZsNClcYsG7YsI+K7kuYBe6VTh0TE7fW+Y2bWSLeu4BkBrCLrinvDYDNrieqkynx3wz8FXAJsQbYl+w8lnVr/W2ZmjXVVNxx4D/Ca3r3g0ma+twJnFRmYmXW37G542VHklydZLu1z3ch0zsxs8DqkxZhXvY00vkI2RvkosEjSNen93sAt7QnPzLpZhXJl3ZZl7x3vRcBVNedvLi4cMxtOuqJlGRH97jBsZtYKXTdmKenlwJnA9sB6vecjYtsC4zKzYaBKLcs8cyYvAr5D9otgX+Ay4NICYzKzYUCCEVLuo2x5kuUGEXENQEQsjojTyJKmmdmQtHLXIUkXSlom6faac5tKulbSn9LPTWo+O1XSvZLulvTWRuXnSZYr00YaiyXNkrQ/sGGO75mZ1dXiSekXAfv0OXcKcF1ETAGuS++RtD1wGLBD+s43JI2oV3ieZPkx4CXAicBuwAeB9+eJvJakMySd3Oz3zKx7tbJlGRHXk011rHUgcHF6fTFwUM35H0XEyoj4M3AvsEu98vNspPH79PIJnt8A2MxsSITasZ/l+IjoXUTzV2B8ej2BF06DfDCdG1C9SelzSHtY9ici3t4oyrSu/L3AMuABYF6j75jZMNH8DuhjJc2teT87Imbn/XJEhKRBb9Rar2V57mALBUjP6TkMmJrqmU8/yVLSTGAmwMRJfmCk2XDS5NShFRExvckqHpa0eUQslbQ5WcMN4CGg9qGMW6ZzA6o3Kf26JoPqa3dgTs0GHFcMUM9sYDbAtGnTy9+e28zapg37PV5B1rv9Qvr585rzP5R0DtmOalOAP9QrKO9+lmZmLSVaOyld0iXADLLu+oPA6WRJ8jJJxwBLgEMBImKRpMuAO4DVwIciYk298otMltcDF0k6K9WzP/DtAuszs4pp5XLHiDh8gI/2HOD6M8lWJ+aSO1lKWjciVua9PiLmS7oUuI1snMA7FZnZc7rusRKSdgEuADYCJknaCfhARJzQ6LvNZm4zG14qlCtzja9+DdgPeAQgIm4D3lRkUGY2PLRyUnrR8nTDeyJiSZ+B2LoDoWZmjWRbtHVAFswpT7J8IHXFI62dPAG4p9iwzGw4qNKjYvMky+PIuuKTgIeBX6VzZmZDUqGGZa614cvIVuKYmbWM1Ja14S2T5274+fSzRjwiZhYSkZkNGxXKlbm64b+qeb0ecDDZphhmZkNSpalDebrhL3iEhKTvATcUFpGZDQuiyyal92Nrnt8TzsxscNRlLUtJf+P5Mcsesp2ITykyKDMbHkR1smXdZKlsJvpOPL/P29qI8DZqZjZkXfXc8LSz8NURsWO7AirbM8+Wvzhp9HreOc+GhyolyzwT6BdIek3hkZjZsNPipzsWqt4zeEZGxGrgNcAtkhYDT5K1niMidm5TjGbWhbqpG/4HYGfggDbFYmbDSYfsJpRXvWQpgIhY3KZYzGyY6ZbljuMknTTQhxFxTgHxmNkw0U3d8BHAaKjQRCgzqxAxoktalksj4rNti8TMhpXs6Y5lR5FfwzFLM7NCtHC5o6TtgNp9LLYB/g3YGPggsDyd/2REXD2YOuoly34fH2lm1iqtusETEXcDUwHSEx0eAuYA7wO+EhFfGmodAybLiHh0qIWbmQ2kwG74nsDifp4dNiRVegSGmXWZnrRbep4DGCtpbs0x0AbkhwGX1Lw/QdJCSRdK2mTQsQ72i2ZmQ9Xko3BXRMT0mmP2i8vTKLKFND9Op75JNn45FVgKfHmwsXrHBjMrhSiktbYvMD8iHgbo/QnPPSLnysEW7GRpZuUQRWyQcTg1XXBJm0fE0vT2YOD2wRZcaLKU9G7gRGAU8Hvg+Igofw80M+sIrUyVkl4CvAU4tub0FyVNJdvA/P4+nzWlsGQp6VXAu4DdImKVpG8ARwLf7XPdTGAmwMRJk4oKx8w6jKClK3gi4klgTJ9zR7Wq/CJblnsC08i2dwNYH1jW96I0SDsbYNq06d6F3WwY6ZYVPEMl4OKIOLXAOsyssjpjU9+8ipw6dB3wTkkvA5C0qaStCqzPzCqk92543qNshbUsI+IOSacBv5TUA6wCPgQsKapOM6uWKrUsC70bHhGX8sLF7WZmz6lOqvQ8SzMrSzHzLAvjZGlmpShoBU9hnCzNrDRuWZqZ5dAtz+AxMytM1g2vTrZ0sjSz0lSoF+5kaWZlEXLL0sysMbcszcwa8JilmVkecsvSzCwXJ8tBWhPBUytXlxpDT5UmfplVnG/wmJk1IDwp3cwsl54K9cOdLM2sNO6Gm5k10OpuuKT7gSeANcDqiJguaVOyPXUnkz3d8dCI+Ntgyq/SDklm1lXU1D85vSkipkbE9PT+FOC6iJhC9qibUwYbrZOlmZUjzbPMewzSgcDF6fXFwEGDLcjJ0sxKoyYOYKykuTXHzD7FBfArSfNqPhsfEUvT678C4wcbq8cszawU2ZhlU03GFTXd6/68ISIeSk+UvVbSXbUfRkRIikGECrhlaWYlarJlWVdEPJR+LgPmALsAD0vaHCD9XDbYWJ0szaw8LcqWkl4iacPe18DewO3AFcB702XvBX4+2FDdDTez0rRwUvp4YE56ps9I4IcR8d+SbgEuk3QMsAQ4dLAVOFmaWWlalSoj4j5gp37OPwLs2Yo62tINl3SipDsl/aAd9ZlZRbRy0LJg7WpZHg/sFREPtqk+M+twWQ7sgCyYU+EtS0nfArYBfiHpY0XXZ2YV0Z5J6S1TeMsyImZJ2odsGdKKouszs+rogByYW+lThyTN7J2R/8gK51KzYaVCY5alJ8uImB0R0yNi+pixY8sOx8zappCNNArjqUNmVppOGIvMy8nSzErRIb3r3NqSLCNicjvqMbNqUYWalm5ZmllpKpQrnSzNrDwVypVOlmZWkooNWjpZmllpOmFKUF5OlmZWCuExSzOzXCqUK50szaxEFcqWTpZmVhqPWZqZ5dBTnVzpZGlmJXKyNDOrr2o7pXdUsuyRWHedEaXGMKJK/QKzKuuQHdDzKn0/SzMbvlq196+kiZL+R9IdkhZJ+kg6f4akhyQtSMfbBhtrR7UszWyYaV3LcjXwLxExX9KGwDxJ16bPvhIRXxpqBU6WZlaS1u2AHhFLgaXp9ROS7gQmtKTwxN1wMytNk093HNv7vK50zOy/TE0GXgP8Pp06QdJCSRdK2mSwsTpZmlkpmhmvTO3PFb3P60rH7BeVKY0GLgc+GhGPA98kexT3VLKW55cHG6+TpZmVp4VPd5S0Dlmi/EFE/BQgIh6OiDURsRY4H9hlsKF6zNLMStPTorlDyp5PcQFwZ0ScU3N+8zSeCXAwcPtg63CyNLPStHCa5W7AUcAfJS1I5z4JHC5pKhDA/cCxg63AydLMytHCSekRcQP9596rW1ODk6WZlao6S3icLM2sFN4p3cwspwrlSidLMyuPW5ZmZjl4i7ZE0qeBdwPLgQeAea1Y0G5mXaI6ubK4ZCnptcA7gJ2AdYD5wLyi6jOz6qlQriy0Zbkb8POIeAZ4RtJ/9XdRWgw/E2DipEkFhmNmnURq3Qqedih9bXhEzO5dGD927LiywzGzdmrh2vCiFZksbwT2l7Re2glkvwLrMrMKqlCuLK4bHhG3SLoCWAg8DPwReKyo+syseirUCy+8G/6liNgWeCuwFb7BY2bPUVP/lK3oeZazJW0PrAdcHBHzC67PzCrCyx1rRMQRRZZvZtYuXsFjZqVxy9LMLIdOGIvMy8nSzEqRTUovO4r8nCzNrDxOlmZmjbkbbmaWQ5Vu8JS+NtzMhq9WLneUtI+kuyXdK+mUVsfqZGlm5WlRtpQ0AjgP2BfYnuwRuNu3MlQnSzMrTQuXO+4C3BsR90XEs8CPgANbGWtHjVneOn/eitHr9iwZQhFjgRWtiscxVD4G6Iw4uiGGrVoVSK9b58+7ZoNRGtvEV9aTNLfm/eyImJ1eTyB7GkOvB4F/GmqMtToqWUbEkDa0lDQ3Iqa3Kh7HUO0YOiUOx9C/iNin7Bia4W64mXWDh4CJNe+3TOdaxsnSzLrBLcAUSVtLGgUcBlzRygo6qhveArMbX1I4x5DphBigM+JwDAWLiNWSPgxcA4wALoyIRa2sQxHRyvLMzLqSu+FmZjk4WZoVQNIGZcdgreVkaS0labxUpRW/rSfpAODr6UaDdQknyxaQ1JN+DvckMQE4jWypWal/F2XVL2kMcCLw78CWkjYtIw5rva5JlpLWL6vuiFibXm4laWSZiULSq8qqG/gL2RM8XwO8veSEuQWApHbP+HgWWA2cDpwDrK1/efEkbVb2L69u0BXJMk0Z+KKksyRt1MZ6d5V0WHp9AvBT4ELglN7WZjtJOg44W9L4EupWZFMresg2MvgEcGAZ/5Om/x6+JekLwPGS1m1X3RHxBPBrsnXJCyPi7yX/8nw18FngHU6YQ1P5eZaSjgcOAY4A5gMTJH0uIv7Uhuo3Ac6StAPw8hTHVsAM4AuSTqlpdRYqjZPNAg6IiIfbUWetiAhJRwInAEcD7wPeBIyUdHm0aY6apIOAQ4EDgDnAoohY2Y66a1xK1sI+V9LfIuIrba4fAEn7AyeT/X8+OZ1r27+LblPplqWklwI7k83Wfwdwa/roa5KmFF1/RFwFzAQOzt7GYuAG4HvAeLIWVrtsAVwaEUskrdPGemttB/wwIm4DPg7cC3wYOKSNrZqNgK8CBwGrgJMAJG3bpvqJiCURcR3ZL/DjUku3rVLv4hPAsRGxG1mvZwYltfa7QaWTZUQ8DnwIeBlwcFqY/17gtcBR7bgbGRHXAp8C3ibpXRHxbGrVbgi0c/xwCbCHpO0iYhWApKNSS6td5gO7Sdoh/T18HVgPmAaMblMM9wNnA8dExN4R8aykE4EPtPuXSETcCrwTOEPSse2sm2zstAcYk95fQNbC/Bjw1jbH0hUq3w2PiJWSniLr7v0fsm7wdcB/pn3t2hHDzyUdRdaifRWwAJhCljza5UZgV+BoSTeSJesTgcPbGMNvyH5RHSHp18D6wD+Ar6WxvHaYB/wcWCtpBjCJ7Bfoe3t/ibRTRCxMcTzd5nr/July4M2SHouI2yXNIdtg4jBJ/1PC8ESldcVyxzSA/1FgL7Lu6CERcUcJcRwEXA5cCXwsIu5rc/2bk91YOAB4DDgrIha2OYYtgLenYzVwcgkxbE72d3AA8AhwdkT8sZ0xdAJJWwLHAtPJfnG/EziKbHrXp9NwieXUFckSIHWxNgPWRkRLt2ZqMo43Aksi4v4SYxgF0K6W9QAxvITsv69/lBjDOgBltCg7RRrX3xXYCbga2AA4H3hLGTcCq6xrkqWZ1SfpTcBZZDd93KpskpOl2TCRhidGRcRQHt0ybDlZmpnlUOmpQ2Zm7eJkaWaWg5OlmVkOTpZdQtIaSQsk3S7px0PZfFbSDElXptcHSDqlzrUbp/X5zdZxhqST857vc81Fkt7ZRF2TJd3ebIxmtZwsu8fTETE1InYkW+o2q/ZDZZr+9x0RV0TEF+pcsjHQdLI0qxony+70O+AVqUV1t6TvArcDEyXtLekmSfNTC3Q0gKR9JN0laT7Z6hvS+aMlnZtej5c0R9Jt6dgV+ALw8tSqPTtd96+SbpG0UNJnasr6lKR7JN1AtulGXZI+mMq5TdLlfVrLe0mam8rbL10/QtLZNXW3ez22dTEnyy6jbLPbfYHe5X1TgG9ExA7Ak2RL3faKiJ2BucBJktYjW9WxP9mmF5sNUPzXgN9GxE5kuz0tAk4BFqdW7b9K2jvVuQswFZgmaQ9J08h2h5oKvI1sDXkjP42I16b67gSOqflscqrjn8n2rlwvff5YRLw2lf9BSVvnqMesocpvpGHPWV/SgvT6d2S7zGxBtvTy5nT+dWTbxt2YdukaBdwEvBL4c+8eoJK+T7b1XF9vBt4DEBFrgMckbdLnmr3T0btd3miy5LkhMCcinkp1XJHjz7SjpM+TdfVHkz0Tutdlaa/QP0m6L/0Z9gZeXTOeuVGq+54cdZnV5WTZPZ6OiKm1J1JCfLL2FHBtRBze57oXfG+IRLaBx7f71PHRQZR1EXBQRNwm6Wiy/Rh79V1NEanuEyKiNqkiafIg6jZ7AXfDh5ebyfabfAVkm12kTXHvAiZLenm6bqBt3a4DjkvfHaHsER5PkLUae10DvL9mLHSCpJcB1wMHSVpf0oZkXf5GNgSWpg0xjuzz2SGSelLM2wB3p7qP691AQ9K2aUMPsyFzy3IYiYjlqYV2iZ5/Ls1pEXGPpJnAVcr2Bv0dL0yAvT4CzJZ0DLAGOC4ibpJ0Y5qa84s0bvkq4KbUsv0H8O6ImC/pUuA2YBlwS46QPw38HlieftbG9P+BPwAvBWZFxDOS/pNsLHO+ssqXk+2YbjZkXhtuZpaDu+FmZjk4WZqZ5eBkaWaWg5OlmVkOTpZmZjk4WZqZ5eBkaWaWw/8CJQ2ARUsOhOYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAEmCAYAAAAA6gkZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xu8VHW9//HXe29AIBBNzAvgJcVSO2mKVlYeTTPtqNjF8p6pEealy7Hi/LQyy6Mdq1OmRpwyo5tYalFRVPoz0yRBRBQ5GJIIaAp4xyvyOX+s79ZhnJk9mz1r1sze76ePeTiz1prv9zOzN5/9/X7X+n6XIgIzM3uljqIDMDNrVU6QZmZVOEGamVXhBGlmVoUTpJlZFU6QZmZVOEG2AUnnSvpxer6NpKckdTa4jvskHdjIMuuo81RJD6XPs1kvynlK0msbGVtRJC2QtF/RcVjGCZKXksPDkl5Vsu0USTcUGFZFEXF/RAyLiBeLjqU3JA0EvgEclD7P6g0tK71/SeOiazxJV0j6SnfHRcSuEXFDE0KyOjhBvqwT+ERvC1HG32v3tgAGAwuKDqQVSBpQdAz2Sv6H/LKLgLMkbVJpp6R9JM2W9Hj6/z4l+26QdL6km4GngdembV+R9NfUBfy1pM0k/UTSE6mM7UrK+JakZWnfbZLeUSWO7SSFpAGS3prK7no8K+m+dFyHpEmS7pW0WtJVkl5dUs7xkpamfWfX+mIkDZH09XT845JukjQk7Ts8dQsfS59555L33SfpLEnz0/umSRosaSdgUTrsMUnXl36usu/1lPR8R0l/TuWskjSt5LiQtGN6PkLSVEkrU7zndP3BknRiiv1rkh6V9A9Jh9T43PdJ+kyKf42k70vaQtLvJD0p6U+SNi05/ueS/plivFHSrmn7BOBY4LNdvwsl5X9O0nxgTfqZvjTUIWmGpK+XlH+lpMtr/ayswSKi3z+A+4ADgWuAr6RtpwA3pOevBh4FjgcGAEen15ul/TcA9wO7pv0D07bFwA7ACOBu4J5UzwBgKvCDkhiOAzZL+/4d+CcwOO07F/hxer4dEMCAss8wEPgzcEF6/QlgFjAa2Aj4LvCztG8X4Clg37TvG8Ba4MAq38+l6fOMImtp75PetxOwBnhXqv+z6TMPKvlebwW2Tt/hQmBipc9R6XOlOk9Jz38GnE32R30w8PaS4wLYMT2fCvwKGJ7KvAc4Oe07EXgB+Gj6HKcCDwCq8Xsxi6y1Owp4GJgLvCnFcD3wxZLjT0r1bgR8E5hXsu8K0u9WWfnzgDHAkNLfxfR8y1TnO8kS7BJgeNH/XvrTo/AAWuHBywnyDcDjwOasnyCPB24te88twInp+Q3AeWX7bwDOLnn9deB3Ja8PK/0HVCGmR4Hd0vNz6T5Bfgf4DdCRXi8EDijZv1VKDgOALwBXlux7FfA8FRJkSkjPdMVStu/zwFVlx64A9iv5Xo8r2f9fwORKn6PS52L9BDkVmAKMrhBHADuSJb3ngV1K9n2s5Od4IrC4ZN/Q9N4ta/xeHFvy+mrgOyWvzwB+WeW9m6SyR6TXV1A5QZ5U6Xex5PX7gWXAKkr+KPjRnIe72CUi4i6yJDOpbNfWwNKybUvJWhVdllUo8qGS589UeD2s60Xqii5M3bPHyFqdI+uJW9LHgP2AYyJiXdq8LXBt6vo+RpYwXyRrDW1dGm9ErAGqnSQZSdZaurfCvvW+l1T3Mtb/Xv5Z8vxpSj5zD30WEHBr6tKfVCXWgaz/syr/Ob0UT0Q8nZ7Wiqmun6GkTkkXpiGNJ8gSXVdMtVT6vSn1a7LEvygiburmWGswJ8hX+iJZF6z0H9UDZAmn1DZkraUuG7wsUhpv/CzwQWDTiNiErCWrOt/7ZWB8RDxRsmsZcEhEbFLyGBwRK4AHybp1XWUMJeveV7IKeJZsqKDcet+LJKVyV1Q4tjtr0v+HlmzbsutJRPwzIj4aEVuTtQov6xp3LIv1Bdb/WZX/nPJyDDCerCcygqxFDC//DKv9fnT3e3M+2R+3rSQd3csYrYecIMtExGJgGnBmyeYZwE6SjkkD6R8iG8f7TYOqHU42BrgSGCDpC8DG3b1J0hjgKuCEiLinbPdk4HxJ26ZjN5c0Pu37BXCopLdLGgScR5XfhdQqvBz4hqStU0vprZI2SnX/m6QDlF228+/Ac8Bfe/Tps3pWkiWy41IdJ1GSlCUdKWl0evkoWWJZV1bGiymm8yUNT5/908CPexrPBhhO9tlXkyX5/yzb/xDQo2s1Je0LfAQ4Afgw8G1Jo2q/yxrJCbKy88jG5QCI7Bq9Q8kSwGqy1t6hEbGqQfXNBH5PdkJhKVmLrbuuF8ABZF3mX+jlM9ldl818C5gO/EHSk2QnG96cPs8C4DTgp2StyUeB5TXqOQu4E5gNPAJ8lWyscxHZyaVvk7XeDgMOi4jn6/zc5T4KfIbsO96V9RPtXsDfJD2VPtcnovK1j2eQtUaXADelz9iMM79TyX52K8hOyM0q2/99YJc05PHL7gqTtHEq8/SIWBERf0ll/CC11K0JlAaCzcysjFuQZmZVOEGaWZ8g6XJlU4bvqrJfki6WtDhd/L9Hd2U6QZpZX3EFcHCN/YcAY9NjAtm1wzU5QZpZnxARN5KdRKxmPDA1MrOATSRtVavMlpogr4FDQxuNKDSG3XYq/iqKDp+jtBazdOl9rFq1qqG/mZ0bbxux9pm6j49nVi4gu8Kjy5SImNKDKkex/tUhy9O2B6u9obUS5EYj2Gj3kwuN4fo/fKnQ+gGGDGroUo9mvfa2N49reJmx9hk2et0H6z7+2XmXPhsRjQ+khpZKkGbWnwiauzLgCkpmkJEt5FJzlpXHIM2sGAKk+h+9Nx04IZ3NfgvweERU7V6DW5BmVqQGtiAl/Yxs0ZaRkpaTraswECAiJpNNGX4P2ZJ8T5NN46zJCdLMCiLoaNx4e0TUXMwjsmmDp/WkTCdIMytOi08rd4I0s2KIZp+k6TEnSDMrSMNOvuTGCdLMiuMWpJlZFW5BmplV0vQLxXvMCdLMitF1oXgLc4I0s+K4BWlmVomgs7UXZnGCNLNitMF1kLlGJ+mXkm5LN3qfkGddZtaGmrtYRY/l3YI8KSIekTQEmC3p6nQL1ZekxJklz426vRW0mfUZPot9pqT3pudjyO4FsV6CTCsCTwHoGLaV70Fr1p/017PYkvYDDgTeGhFPS7oBGJxXfWbWhvpxC3IE8GhKjq8H3pJjXWbWbgocW6xXngny98BESQuBRcCsHOsys3bUX1uQEfEc2X1ozcwq68ctSDOzGnwW28ysMtHQWy7kwQnSzAriFqSZWXUegzQzq8ItSDOzKtyCNDOrQB6DNDOrzi1IM7PK5ARpZvZK2S1pnCDrtvvrRnHjdecVGsPYM68ttH6AmeccVHQI7LTV8KJDsL5OQh1OkGZmFbkFaWZWhROkmVkVTpBmZpUoPVqYE6SZFULILUgzs2qcIM3MqnCCNDOrwgnSzKwSn6QxM6tMiI6O1l7Np7WjM7M+TVLdjzrKOljSIkmLJU2qsH+EpF9LukPSAkkf6a5MJ0gzK4568KhVjNQJXEp2q+ldgKMl7VJ22GnA3RGxG7Af8HVJg2qV6wRpZsVQQ1uQewOLI2JJRDwPXAmMLzsmgOHKChsGPAKsrVVobglS0naS7sqrfDNrfz1MkCMlzSl5TCgpahSwrOT18rSt1CXAzsADwJ3AJyJiXa34fJLGzArTw8t8VkXEuF5U925gHvBOYAfgj5L+EhFPVHtD3l3sAZJ+ImmhpF9IGppzfWbWJrqmGjaoi70CGFPyenTaVuojwDWRWQz8A3h9rULzTpCvAy6LiJ2BJ4CPlx8gaUJXk3nVypU5h2NmLaVBJ2mA2cBYSdunEy9HAdPLjrkfOABA0hZk+WlJrULzTpDLIuLm9PzHwNvLD4iIKRExLiLGjdx885zDMbOW0cCTNBGxFjgdmAksBK6KiAWSJkqamA77MrCPpDuB64DPRcSqWuXmPQYZ3bw2s36skVMNI2IGMKNs2+SS5w8APbqfSd4tyG0kvTU9Pwa4Kef6zKyNqEN1P4qQd4JcBJwmaSGwKfCdnOszszbSyJk0ecitix0R99HNGSIz67+KTHz18nWQZlYYJ0gzsyqcIM3Mqmnt/OgEaWbFcQvSzKwSOUGamVUkoMXzoxOkmRVFdBR0AXi9nCDNrDDuYpuZVSJ3sc3MKhK4i90TAgZ0FnubnKmf3LfQ+gGOnnxL0SFw25d6tOiJ2QZxC9LMrAqPQZqZVeIxSDOzyrLrIFs7QzpBmllBvNyZmVlVLZ4fnSDNrCDyZT5mZhV5DNLMrIYWz49OkGZWHLcgzcyqaPH86ARpZgXxgrlmZpV5wVwzs6p8obiZWVUtnh+dIM2sIG1woXjuiy9KOkHSfEl3SPpR3vWZWXvoulC83kcRcm1BStoVOAfYJyJWSXp1hWMmABMAxmyzTZ7hmFmLafUxyLxbkO8Efh4RqwAi4pHyAyJiSkSMi4hxm4/cPOdwzKyVSPU/iuAxSDMrTH9vQV4PHClpM4BKXWwz66d60Hrsky3IiFgg6Xzgz5JeBG4HTsyzTjNrD/J1kBARPwR+mHc9ZtZ+Wjw/egzSzIrT0eIZstibUJtZv9bIMUhJB0taJGmxpElVjtlP0jxJCyT9ubsy3YI0s0JI0NmgmTSSOoFLgXcBy4HZkqZHxN0lx2wCXAYcHBH3S3pNd+W6BWlmhWngTJq9gcURsSQingeuBMaXHXMMcE1E3A8QEQ93V2jVFqSkjWu9MSKe6K5wM7NaejgEOVLSnJLXUyJiSno+ClhWsm858Oay9+8EDJR0AzAc+FZETK1VYa0u9gIgyKZMdul6HYDnBZrZBhPZpT49sCoixvWiygHAnsABwBDgFkmzIuKeWm+oKCLG9CIQM7NuNXAxnxVAac4anbaVWg6sjog1wBpJNwK7AVUTZF1jkJKOkvT/0vPRkvbsSeRmZq/Qg/HHOsYgZwNjJW0vaRBwFDC97JhfAW+XNEDSULIu+MJahXZ7FlvSJcBAYF/gP4GngcnAXt2918yslkZdBhkRayWdDswEOoHL00y+iWn/5IhYKOn3wHxgHfC9iLirVrn1XOazT0TsIen2VNEjKUObmW0w0dgLxSNiBjCjbNvkstcXARfVW2Y9CfIFSR1kJ2ZIC0+sq7eCdvOOscUvubb07w8UHQLLVj9ddAiM2Wxo0SFYzlp8Ik1dY5CXAlcDm0v6EnAT8NVcozKzfqHtVxSPiKmSbgMOTJuO7K7fbmbWnUbOpMlLvVMNO4EXyLrZnn1jZg3R2umxjmQn6WzgZ8DWZNcW/VTSf+QdmJn1fW3fxQZOAN4UEU8DpAVwbwcuyDMwM+vbsrPYRUdRWz0J8sGy4wakbWZmG67AlmG9ai1W8d9kY46PAAskzUyvDyK7at3MrFdaPD/WbEF2naleAPy2ZPus/MIxs/6kbVuQEfH9ZgZiZv1LnxiDlLQDcD6wCzC4a3tE7JRjXGbWD7R6C7KeaxqvAH5AlvAPAa4CpuUYk5n1AxJ0SnU/ilBPghwaETMBIuLeiDiHLFGamfVKI2/alYd6LvN5Li1WcW9aOmgF2XLlZma90upd7HoS5KeAVwFnko1FjgBO6mlFks4FnoqIr/X0vWbWN7V4fqxrsYq/padPAsfnG46Z9RdCDV0PMg+1LhS/lrQGZCUR8b7uCk/zuD8MPEx2x7HbNiBGM+uLChxbrFetFuQlvSk43bfmKGD3VM9cKiRISROACQBjtvGNEs36k7Ydg4yI63pZ9juAa0sWuSi/gU5XPVOAKQB77jmuaovVzPqeVl87sd71IM3MGkq0fgsyzwR+I3CEpCGShgOH5ViXmbWhDtX/KELdLUhJG0XEc/UeHxFzJU0D7iA7SeMVgMzsJX3ilguS9ga+T3b94zaSdgNOiYgzuntvRJxPdu2kmdkrtHh+rKuLfTFwKLAaICLuAPbPMygz6x/6wlTDjohYWjaY+mJO8ZhZP5Etd9baTch6EuSy1M0OSZ3AGcA9+YZlZv1BX7jM51SybvY2wEPAn9I2M7NeafEGZF1zsR8mmxFjZtYwUhvPxe4i6X+oMCc7IibkEpGZ9Rstnh/r6mL/qeT5YOC9ZAtPmJn1Sqtf5lNPF3u92ytI+hFwU24RmVm/IPrAheIVbA9s0ehAzKyfKXAKYb3qGYN8lJfHIDuAR4BJeQZlZv2DaO0MWTNBKrs6fDey+9AArIsIL0lmZr3W9vfFjoiQNCMi3tCsgIq26V6nFx0Cj87u1VrFZm2j1RNkPReyz5P0ptwjMbN+R1LdjyLUuifNgIhYC7wJmC3pXmANWcs4ImKPJsVoZn1Qu3exbwX2AA5vUixm1p80eJUeSQcD3wI6ge9FxIVVjtsLuAU4KiJ+UavMWglSABFx74aFa2ZWW6OmGqaFdC4F3gUsJ+v1To+Iuysc91XgD/WUWytBbi7p09V2RsQ36qnAzKySBnex9wYWR8QSAElXAuOBu8uOOwO4GtirnkJrJchOYBi0+IVKZtamRGfPWpAjJc0peT0l3RUVYBTrT4FeDrx5vdqkUWRTpfenAQnywYg4r55CzMx6KrurYY/esioixvWiym8Cn4uIdfWeFe92DNLMLBeNnWq4AhhT8no0L09w6TIOuDIlx5HAeyStjYhfViu0VoI8YAMDNTOrSwPXg5wNjJW0PVliPAo4pvSAiNi+67mkK4Df1EqOUCNBRsQjvYnWzKyWDehiVxURayWdDswkO39yeUQskDQx7Z+8IeVuyGo+ZmYN0cgVxSNiBjCjbFvFxBgRJ9ZTphOkmRWmL6wobmbWcKJv3NXQzKzxRGGLUNQr1wQu6ThJt0qaJ+m7aZqPmRmQTtTU+ShCbglS0s7Ah4C3RcTuwIvAsRWOmyBpjqQ5K1etzCscM2sxAjqluh9FyLOLfQCwJ9mkcYAhwMPlB6WpQlMA9txznFcrN+tHWryHnWuCFPDDiPiPHOsws7ZV3EK49cpzDPI64AOSXgMg6dWSts2xPjNrI11nset9FCG3FmRE3C3pHOAPkjqAF4DTgKV51Wlm7aXVW5C5XuYTEdOAaXnWYWbtq7XTo6+DNLOitMF1kE6QZlYIz6QxM6vBLUgzsyra+bavZma5ybrYrZ0hnSDNrDAt3sN2gjSzogi5BWlmVplbkGZmFXgM0sysGrkFaWZWlRNkD9y7eg0f+sHsQmO44FufLrR+s/7EJ2nMzCoQvlDczKyqRt4XOw9OkGZWGHexzcwqcBfbzKwqz6QxM6vM10GamVXX4vnRCdLMipGNQbZ2inSCNLPCtHZ6dII0syK1eIZ0gjSzwriLbWZWRWunxybddVHSmZIWSvpJM+ozszahHjwK0KwW5MeBAyNieZPqM7MWl+W91m5D5t6ClDQZeC3wO0mfyrs+M2sT6ULxeh9FyL0FGRETJR0M7B8Rq/Kuz8zaR2u3H5s0BlmLpAmS5kia8/yTjxUdjpk1UwPHICUdLGmRpMWSJlXYf6yk+ZLulPRXSbt1V2bhCTIipkTEuIgYN2j4JkWHY2ZNox79V7MkqRO4FDgE2AU4WtIuZYf9A/jXiPgX4MvAlO4i9GU+ZlaYBo4t7g0sjoglWbm6EhgP3N11QET8teT4WcDo7gotvAVpZv1TT3rXKY+O7BqOS48JJcWNApaVvF6etlVzMvC77mJsSgsyIrZrRj1m1l7UsybkqogY14A69ydLkG/v7lh3sc2sMA3sYq8AxpS8Hp22ldWnNwLfAw6JiNXdFeoutpkVpoEnsWcDYyVtL2kQcBQwfb26pG2Aa4DjI+KeeuJzC9LMitHAKYQRsVbS6cBMoBO4PCIWSJqY9k8GvgBsBlyWuvZru+uyO0GaWWEaOdUwImYAM8q2TS55fgpwSk/KdII0s0II35PGzKyqFs+PTpBmVqAWz5BOkGZWmFZf7swJ0swK09Ha+dEJ0swK5ARpZvZK7bCieEslyCceXs0fLvthoTFMm31JofWb9RsFrhRer5ZKkGbWv7R4fnSCNLMCtXiGdII0s4J0v1J40ZwgzawwHoM0M6uggYv55MYJ0syK0+IZ0gnSzArT0eJ9bCdIMytMa6dHJ0gzK4ovFDczq6W1M6QTpJkVwiuKm5nV0OL50QnSzIrjFqSZWRX9eqqhpM8DxwErgWXAbRHxtTzrNLM20tr5Mb8EKWkv4P3AbsBAYC5wW171mVn7afH8mGsL8m3AryLiWeBZSb+udJCkCcAEAAYOyzEcM2slUuvPpOkoOoCImBIR4yJinAYMKTocM2sm9eBRgDwT5M3AYZIGSxoGHJpjXWbWhlo8P+bXxY6I2ZKmA/OBh4A7gcfzqs/M2k+L97Bz72J/LSJ2At4NbItP0pjZS9Sj/4qQ93WQUyTtAgwGfhgRc3Ouz8zaRL+fahgRx+RZvplZnjyTxswK069bkGZmtfTrqYZmZtVkF4oXHUVtTpBmVhwnSDOzytzFNjOrotVP0hQ+F9vM+q9GTjWUdLCkRZIWS5pUYb8kXZz2z5e0R3dlOkGaWXEalCEldQKXAocAuwBHp0kqpQ4BxqbHBOA73YXnBGlmhWngVMO9gcURsSQingeuBMaXHTMemBqZWcAmkraqVWhLjUHGMytXPTvv0qW9KGIksKo3MQwZeGlv3t6QGBrAMbysFeLoCzFs26hAutw+97aZQwdpZA/eMljSnJLXUyJiSno+iuyuBV2WA28ue3+lY0YBD1arsLUSZMTmvXm/pDkRMa5R8TiG9o6hVeJwDJVFxMFFx9Add7HNrC9YAYwpeT06bevpMetxgjSzvmA2MFbS9pIGAUcB08uOmQ6ckM5mvwV4PCKqdq+hxbrYDTCl+0Ny5xgyrRADtEYcjiFnEbFW0unATKATuDwiFkiamPZPBmYA7wEWA08DH+muXEVEflGbmbUxd7HNzKpwgjTLgaShRcdgvecEaQ0laQup1WfY5kvS4cC308kCa2NOkA0gqSP9v78nhlHAOWTTvAr9LoqqX9JmwJnAV4HRkl5dRBzWGH0mQUoaUlTdEbEuPd1W0oAik4OknYuqG3iA7M6VbwLeV3CS3BpAUrOv1HgeWAt8EfgGsK724fmTtGXRf7DaVZ9IkOn0/n9JukDSiCbWu4+ko9LzM4BrgMuBSV2tymaSdCpwkaQtCqhbkV0S0UG2WMDngPFF/MNMvw+TJV0IfFzSRs2qOyKeBK4nm/c7PyIeK/gP5huB84D3O0n2XNtfBynp48CRwDHAXGCUpC9HxN+bUP2mwAWSdgV2SHFsC+wHXChpUknrMldp3GsicHhEPNSMOktFREg6FjgDOJHsGrP9gQGSro4mXU8m6Qjgg8DhwLXAgoh4rhl1l5hG1pK+RNKjEfHfTa4fAEmHAWeR/TvfLm1r2s+iL2jrFqSkjYE9yK6afz9we9p1saSxedcfEb8lWzbpvdnLuBe4CfgRsAVZS6pZtgamRcRSSQObWG+p1wE/jYg7gM+SXZB7OnBkE1svI4BvAkcALwCfBpC0U5PqJyKWRsR1ZH+0T00t2qZKvYjPAR+LiLeR9W72o6BWfbtq6wQZEU8ApwGvAd6bJr9/GNgLOL4ZZxEj4o/A2cB7JH0oIp5PrdfhQDPHA5cC+0p6XUS8ACDp+NSiapa5wNsk7Zq+h28Dg4E9gWFNiuE+4CLg5Ig4KCKel3QmcEqz/3BExO3AB4BzJX2smXWTjYV2AJul198na0l+Cnh3k2NpW23fxY6I5yQ9TdaV+xeyLu51wPfSunDNiOFXko4na7nuDMwjW5RzbjPqT24G9gFOlHQzWYI+Ezi6iTHcQPbH6RhJ1wNDgKeAi9PYXDPcBvwKWCdpP2Absj+aH+76w9FMETE/xfFMk+t9VNLVwDslPR4Rd0m6lmyBhqMk/f8Chh7aTp+YapgG4T8JHEjW1TwyIu4uII4jgKuB3wCfioglTa5/K7KTA4cDjwMXRMT8JsewNfC+9FgLnFVADFuRfQeHA6uBiyLizmbG0AokjQY+Bowj+2P9AeB4skuxPp+GQqyGPpEgAVL3aUtgXUTUXMIo5zj+FVgaEfcVGMMggGa1oKvE8Cqy36+nCoxhIEARLcdWkcbp9wF2I1usYSjwP8C7ijiZ1276TII0s9ok7Q9cQHbixq3HOjhBmvUTaehhUET05rYm/YoTpJlZFW19mY+ZWZ6cIM3MqnCCNDOrwgmyj5D0oqR5ku6S9PPeLNgqaT9Jv0nPD5c0qcaxm6T58D2t41xJZ9W7veyYKyR9oAd1bSfprp7GaOYE2Xc8ExG7R8QbyKaZTSzdqUyPf94RMT0iLqxxyCZAjxOkWTtwguyb/gLsmFpOiyRNBe4Cxkg6SNItkuamluYwAEkHS/pfSXPJZsGQtp8o6ZL0fAtJ10q6Iz32AS4Edkit14vScZ+RNFvSfElfKinrbEn3SLqJbGGLmiR9NJVzh6Sry1rFB0qak8o7NB3fKemikrqbPf/Z+hgnyD5G2QKxhwBdU+vGApdFxK7AGrJpZgdGxB7AHODTkgaTza44jGxhiS2rFH8x8OeI2I1sFaUFwCTg3tR6/Yykg1KdewO7A3tK2lfSnmSrLu1OduvNver4ONdExF6pvoXAySX7tkt1/BvZ2o+D0/7HI2KvVP5HJW1fRz1mFbX9YhX2kiGS5qXnfyFbvWVrsmmPs9L2t5AtwXZzWvFqEHAL8HrgH11raEr6MdkybuXeCZwAEBEvAo9L2rTsmIPSo2vpuWFkCXM4cG1EPJ3qKL+peyVvkPQVsm78MLJ7Hne5Kq21+XdJS9JnOAh4Y8n45IhU9z111GX2Ck6QfcczEbF76YaUBNeUbgL+GBFHlx233vt6SWSLZHy3rI5PbkBZVwBHRMQdkk4kW8+wS/kMh0h1nxERpYkUSdttQN1m7mL3M7PI1mvcEbIFJdJCsv8LbCdph3RctSXSrgNOTe/tVHZ7iyfJWoddZgInlYxtjpL0GuBG4AhJQyQNJ+vOd2c48GBadOLYsn1HSupIMb8WWJTqPrVrkQpJO6VFM8w2iFuQ/UhErEwtsZ/p5fu0nBMR90iaAPxW2dqaf2H9pNflE8AUSScDLwKnRsQtkm5Ol9H8Lo1D7gzcklqwTwE/M/v9AAAAY0lEQVTHRcRcSdOAO4CHgdl1hPx54G/AyvT/0pjuB24FNgYmRsSzkr5HNjY5V1nlK8lWFjfbIJ6LbWZWhbvYZmZVOEGamVXhBGlmVoUTpJlZFU6QZmZVOEGamVXhBGlmVsX/AQucR2jyEHxgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn import svm, datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "# Compute confusion matrix\n",
    "cm = confusion_matrix(y_compare, pred)\n",
    "np.set_printoptions(precision=2)\n",
    "print('Confusion matrix, without normalization')\n",
    "print(cm)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, products)\n",
    "\n",
    "# Normalize the confusion matrix by row (i.e by the number of samples\n",
    "# in each class)\n",
    "cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "print('Normalized confusion matrix')\n",
    "print(cm_normalized)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm_normalized, products, title='Normalized confusion matrix')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}