19_Hyper-Parameters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Tutorial #19\n",
    "# Hyper-Parameter Optimization\n",
    "\n",
    "by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "/ [GitHub](https://github.com/Hvass-Labs/TensorFlow-Tutorials) / [Videos on YouTube](https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "There are many parameters you can select when building and training a Neural Network in TensorFlow. These are often called Hyper-Parameters. For example, there is a hyper-parameter for how many layers the network should have, and another hyper-parameter for how many nodes per layer, and another hyper-parameter for the activation function to use, etc. The optimization method also has one or more hyper-parameters you can select, such as the learning-rate.\n",
    "\n",
    "One way of searching for good hyper-parameters is by hand-tuning, where you try one set of parameters and see how they perform, and then you try another set of parameters and see if they improve the performance. You try and build an intuition for what works well and guide your parameter-search accordingly. Not only is this extremely time-consuming for a human researcher, but the optimal parameters are often counter-intuitive to humans so you will not find them!\n",
    "\n",
    "Another way of searching for good hyper-parameters is to divide each parameter's valid range into evenly spaced values, and then simply have the computer try all combinations of parameter-values. This is called Grid Search. Although it is run entirely by the computer, it quickly becomes extremely time-consuming because the number of parameter-combinations increases exponentially as you add more hyper-parameters. This problem is known as the Curse of Dimensionality. For example, if you have just 4 hyper-parameters to tune and each of them is allowed 10 possible values, then there is a total of 10^4 parameter-combinations. If you add just one more hyper-parameter then there are 10^5 parameter-combinations, and so on.\n",
    "\n",
    "Yet another way of searching for good hyper-parameters is by random search. Instead of systematically trying every single parameter-combination as in Grid Search, we now try a number of parameter-combinations completely at random. This is like searching for \"a needle in a haystack\" and as the number of parameters increases, the probability of finding the optimal parameter-combinations by random sampling decreases to zero.\n",
    "\n",
    "This tutorial uses a clever method for finding good hyper-parameters known as Bayesian Optimization. You should be familiar with TensorFlow, Keras and Convolutional Neural Networks, see Tutorials #01, #02 and #03-C."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flowchart\n",
    "\n",
    "The problem with hyper-parameter optimization is that it is extremely costly to assess the performance of a set of parameters. This is because we first have to build the corresponding neural network, then we have to train it, and finally we have to measure its performance on a test-set. In this tutorial we will use the small MNIST problem so this training can be done very quickly, but on more realistic problems the training may take hours, days or even weeks on a very fast computer. So we need an optimization method that can search for hyper-parameters as efficiently as possible, by only evaluating the actual performance when absolutely necessary.\n",
    "\n",
    "The idea with Bayesian optimization is to construct another model of the search-space for hyper-parameters. One kind of model is known as a Gaussian Process. This gives us an estimate of how the performance varies with changes to the hyper-parameters. Whenever we evaluate the actual performance for a set of hyper-parameters, we know for a fact what the performance is - except perhaps for some noise. We can then ask the Bayesian optimizer to give us a new suggestion for hyper-parameters in a region of the search-space that we haven't explored yet, or hyper-parameters that the Bayesian optimizer thinks will bring us most improvement. We then repeat this process a number of times until the Bayesian optimizer has built a good model of how the performance varies with different hyper-parameters, so we can choose the best parameters.\n",
    "\n",
    "The flowchart of the algorithm is roughly:\n",
    "\n",
    "![Flowchart](images/19_flowchart_bayesian_optimization.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/magnus/anaconda3/envs/tf-test/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to import several things from Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# from tf.keras.models import Sequential  # This does not work!\n",
    "from tensorflow.python.keras import backend as K\n",
    "from tensorflow.python.keras.models import Sequential\n",
    "from tensorflow.python.keras.layers import InputLayer, Input\n",
    "from tensorflow.python.keras.layers import Reshape, MaxPooling2D\n",
    "from tensorflow.python.keras.layers import Conv2D, Dense, Flatten\n",
    "from tensorflow.python.keras.callbacks import TensorBoard\n",
    "from tensorflow.python.keras.optimizers import Adam\n",
    "from tensorflow.python.keras.models import load_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** We will save and load models using Keras so you need to have [h5py](http://docs.h5py.org/en/latest/build.html#install) installed. You also need to have [scikit-optimize](https://scikit-optimize.github.io/) installed for doing the hyper-parameter optimization.\n",
    "\n",
    "You should be able to run the following command in a terminal to install them both:\n",
    "\n",
    "`pip install h5py scikit-optimize`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** This Notebook requires plotting functions in `scikit-optimize` that have not been merged into the official release at the time of this writing. If this Notebook cannot run with the version of `scikit-optimize` installed by the command above, you may have to install `scikit-optimize` from a development branch by running the following command instead:\n",
    "\n",
    "`pip install git+git://github.com/Hvass-Labs/scikit-optimize.git@dd7433da068b5a2509ef4ea4e5195458393e6555`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import skopt\n",
    "from skopt import gp_minimize, forest_minimize\n",
    "from skopt.space import Real, Categorical, Integer\n",
    "from skopt.plots import plot_convergence\n",
    "from skopt.plots import plot_objective, plot_evaluations\n",
    "from skopt.plots import plot_histogram, plot_objective_2D\n",
    "from skopt.utils import use_named_args"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was developed using Python 3.6 (Anaconda) and package versions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.4.0'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.0.8-tf'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.4'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skopt.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyper-Parameters\n",
    "\n",
    "In this tutorial we want to find the hyper-parametes that makes a simple Convolutional Neural Network perform best at classifying the MNIST dataset for hand-written digits.\n",
    "\n",
    "For this demonstration we want to find the following hyper-parameters:\n",
    "\n",
    "* The learning-rate of the optimizer.\n",
    "* The number of fully-connected / dense layers.\n",
    "* The number of nodes for each of the dense layers.\n",
    "* Whether to use 'sigmoid' or 'relu' activation in all the layers.\n",
    "\n",
    "We will use the Python package `scikit-optimize` (or `skopt`) for finding the best choices of these hyper-parameters. Before we begin with the actual search for hyper-parameters, we first need to define the valid search-ranges or search-dimensions for each of these parameters.\n",
    "\n",
    "This is the search-dimension for the learning-rate. It is a real number (floating-point) with a lower bound of `1e-6` and an upper bound of `1e-2`. But instead of searching between these bounds directly, we use a logarithmic transformation, so we will search for the number `k` in `1ek` which is only bounded between -6 and -2. This is better than searching the entire exponential range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dim_learning_rate = Real(low=1e-6, high=1e-2, prior='log-uniform',\n",
    "                         name='learning_rate')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the search-dimension for the number of dense layers in the neural network. This is an integer and we want at least 1 dense layer and at most 5 dense layers in the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dim_num_dense_layers = Integer(low=1, high=5, name='num_dense_layers')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the search-dimension for the number of nodes for each dense layer. This is also an integer and we want at least 5 and at most 512 nodes in each layer of the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dim_num_dense_nodes = Integer(low=5, high=512, name='num_dense_nodes')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the search-dimension for the activation-function. This is a combinatorial or categorical parameter which can be either 'relu' or 'sigmoid'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dim_activation = Categorical(categories=['relu', 'sigmoid'],\n",
    "                             name='activation')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then combine all these search-dimensions into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dimensions = [dim_learning_rate,\n",
    "              dim_num_dense_layers,\n",
    "              dim_num_dense_nodes,\n",
    "              dim_activation]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is helpful to start the search for hyper-parameters with a decent choice that we have found by hand-tuning. But we will use the following parameters that do not perform so well, so as to better demonstrate the usefulness of hyper-parameter optimization: A learning-rate of 1e-5, a single dense layer with 16 nodes, and relu activation-functions.\n",
    "\n",
    "Note that these hyper-parameters are packed in a single list. This is how `skopt` works internally on hyper-parameters. You therefore need to ensure that the order of the dimensions are consistent with the order given in `dimensions` above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "default_parameters = [1e-5, 1, 16, 'relu']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function for log-dir-name\n",
    "\n",
    "We will log the training-progress for all parameter-combinations so they can be viewed and compared using TensorBoard. This is done by setting a common parent-dir and then have a sub-dir for each parameter-combination with an appropriate name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def log_dir_name(learning_rate, num_dense_layers,\n",
    "                 num_dense_nodes, activation):\n",
    "\n",
    "    # The dir-name for the TensorBoard log-dir.\n",
    "    s = \"./19_logs/lr_{0:.0e}_layers_{1}_nodes_{2}_{3}/\"\n",
    "\n",
    "    # Insert all the hyper-parameters in the dir-name.\n",
    "    log_dir = s.format(learning_rate,\n",
    "                       num_dense_layers,\n",
    "                       num_dense_nodes,\n",
    "                       activation)\n",
    "\n",
    "    return log_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting data/MNIST/train-images-idx3-ubyte.gz\n",
      "Extracting data/MNIST/train-labels-idx1-ubyte.gz\n",
      "Extracting data/MNIST/t10k-images-idx3-ubyte.gz\n",
      "Extracting data/MNIST/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "data = input_data.read_data_sets('data/MNIST/', one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST data-set has now been loaded and consists of 70,000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of:\n",
      "- Training-set:\t\t55000\n",
      "- Test-set:\t\t10000\n",
      "- Validation-set:\t5000\n"
     ]
    }
   ],
   "source": [
    "print(\"Size of:\")\n",
    "print(\"- Training-set:\\t\\t{}\".format(len(data.train.labels)))\n",
    "print(\"- Test-set:\\t\\t{}\".format(len(data.test.labels)))\n",
    "print(\"- Validation-set:\\t{}\".format(len(data.validation.labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test-set, so we calculate it now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data.test.cls = np.argmax(data.test.labels, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the performance on the validation-set as an indication of which choice of hyper-parameters performs the best on previously unseen data. The Keras API needs the validation-set as a tuple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "validation_data = (data.validation.images, data.validation.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We know that MNIST images are 28 pixels in each dimension.\n",
    "img_size = 28\n",
    "\n",
    "# Images are stored in one-dimensional arrays of this length.\n",
    "img_size_flat = img_size * img_size\n",
    "\n",
    "# Tuple with height and width of images used to reshape arrays.\n",
    "# This is used for plotting the images.\n",
    "img_shape = (img_size, img_size)\n",
    "\n",
    "# Tuple with height, width and depth used to reshape arrays.\n",
    "# This is used for reshaping in Keras.\n",
    "img_shape_full = (img_size, img_size, 1)\n",
    "\n",
    "# Number of colour channels for the images: 1 channel for gray-scale.\n",
    "num_channels = 1\n",
    "\n",
    "# Number of classes, one class for each of 10 digits.\n",
    "num_classes = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function for plotting images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_images(images, cls_true, cls_pred=None):\n",
    "    assert len(images) == len(cls_true) == 9\n",
    "    \n",
    "    # Create figure with 3x3 sub-plots.\n",
    "    fig, axes = plt.subplots(3, 3)\n",
    "    fig.subplots_adjust(hspace=0.3, wspace=0.3)\n",
    "\n",
    "    for i, ax in enumerate(axes.flat):\n",
    "        # Plot image.\n",
    "        ax.imshow(images[i].reshape(img_shape), cmap='binary')\n",
    "\n",
    "        # Show true and predicted classes.\n",
    "        if cls_pred is None:\n",
    "            xlabel = \"True: {0}\".format(cls_true[i])\n",
    "        else:\n",
    "            xlabel = \"True: {0}, Pred: {1}\".format(cls_true[i], cls_pred[i])\n",
    "\n",
    "        # Show the classes as the label on the x-axis.\n",
    "        ax.set_xlabel(xlabel)\n",
    "        \n",
    "        # Remove ticks from the plot.\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "    \n",
    "    # Ensure the plot is shown correctly with multiple plots\n",
    "    # in a single Notebook cell.\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot a few images to see if data is correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUMAAAD5CAYAAAC9FVegAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHihJREFUeJzt3XmUFNXZx/HvA0LYVQQFFWdOwAVCFBWDu0aBKCogccG4\nEGM0osEtAaNx1xglKBzRE7YD4QQNigKCUVFAEV8EJIIi4wYiCsRlhLggIsJ9/5i5XdUzPXtXVU/7\n+5zjmequ6qpnvPSdp27dxZxziIj80DVIOgARkVygylBEBFWGIiKAKkMREUCVoYgIoMpQRARQZSgi\nAqgyFBEBVBmKiACwS00ObtOmjSssLIwolNzzwQcfUFxcbEnHESeVcf5TGWdWo8qwsLCQZcuW1T6q\neqZ79+5JhxA7lXH+UxlnpttkERFUGYqIAKoMRUQAVYYiIoAqQxERoIZPk0Vqa8SIEQBs3boVgDfe\neAOAxx9/vNyxgwcPBuCoo44C4MILL4wjRPmBU2YoIoIyQ4nYueeeC8C0adMy7jcr3xd2zJgxAMyd\nOxeAE044AYD99tsvihAlQe+++y4ABx54IAAPPPAAAEOGDIk9FmWGIiIoM5QI+GwQKs4IDzroIABO\nOeUUAN5///3UvlmzZgGwevVqAKZMmQLAjTfemP1gJVHLly8HoEGDkrxsn332SSwWZYYiIigzlCzy\n411nzJhRbl/Xrl2BIOtr06YNAC1atADgu+++Sx3bo0cPAF5//XUAPv/884gilqStWLECCP4dDBgw\nILFYlBmKiBBDZuj7kY0fPx6AvffeO7WvSZMmAJx//vkAtGvXDoBOnTpFHZZE4L///S8AzrnUez4j\nnDNnDgDt27fP+FnfDxHgrbfeStt3+umnZzVOSd7KlSsBGD16NAAXXXRRkuEAygxFRIAYMsOhQ4cC\nJRMsVsT3K2vVqhUAXbp0ycq1O3ToAMCwYcOAH+bcdXE644wzgOApMEDLli0BaN26daWfffTRR1Pb\n4fZDyU/vvPMOAFu2bAHSeyAkRZmhiAiqDEVEgBhukydMmAAE3STCt8BFRUVA0PHyxRdfBGDx4sVA\nMPzqww8/rPD8jRo1AoKuGr4RP3wef7us2+R4FBQUVPvYv/3tb0AwLCvMd7HxPyV/DB8+HChZggBy\n47upzFBEhBgyw5NPPjntZ5gfiuVt3rwZCDJF/9fi1VdfrfD8P/rRj4BgoLcf5gWwadMmADp27Fir\n2CU6Tz31FAC33HILANu2bUvt22uvvQC45557AGjWrFnM0UkUwg9R/Xfaf2+bN2+eREhplBmKiJBj\nw/F23313AE466aS09zNllWU98cQTQJBdAhx88MEADBw4MFshSpb4oXvhjNDz3Sz81F2SHxYsWFDu\nvbZt2yYQSWbKDEVEyLHMsDY+/fRTAK644gogfSiYb4+qqsOvxKd///5AMDzPGzRoUGr7rrvuijUm\niYdf6iHMD4jIBcoMRUTIg8zwoYceAoIMcbfddkvt80+qJHm+/+eiRYuAoK3QtxnddNNNqWP9dE6S\nH1555RUAJk2alHrv0EMPBaBXr16JxJSJMkMREepxZvjyyy8DQV8078knn0xt++mjJHl+0s7i4uK0\n9/30beoLmr/mzZsHpPf08H2M/TR+uUCZoYgIqgxFRIB6fJv89NNPA8Hcdz179gTgqKOOSiwmKc+v\neeKHWHonnngiAHfccUfcIUnM/CQtYWeffXYCkVROmaGICPUwM9y6dSsAzz77LBBM1HD77bcDwZRe\nkpzwanZ33303UH726m7dugHqRpPPPv74YwAWLlwIpE+icuaZZyYSU2WUGYqIUA8zQz8ZqG+DOvXU\nUwE4+uijE4tJ0t13332p7aVLl6bt88Px1FaY//7xj38A8MknnwDBdzVXKTMUEaGeZIZ+IlCAO++8\nE4Bdd90VgJtvvjmRmKRi999/f4X7/PBJtRXmv3Xr1qW99lP05SplhiIi5Hhm6J9KXnXVVan3vv/+\newD69OkDqF9hfePLtDpP/X3274/dvn07AF988UW5Y/1Qr5EjR2Y8V8OGDVPb9957L6DlBKI2e/bs\ntNenn356QpFUjzJDERFUGYqIADl6m7xjxw4gmNli7dq1qX2dOnUCggcpUr/4dWmq45xzzgGgffv2\nQNBFY+rUqXWKwa++F55DUbLHd7L25VVfKDMUESFHM8M1a9YAwQpqYb7bhua/y13+4RbAzJkza32e\nxx57rMpj/MOVBg3S/6737dsXCNbeDjv22GNrHZNUbcaMGUDwsNPPap3rqx0qMxQRIccyQ99Js3fv\n3mnvjxgxIrWd64/nBaZPn57aHj58OFB+ogavqKgIqLwd8JJLLgGgoKCg3L5f/vKXAHTu3Ll2wUrW\nfPPNNwA888wzae/76brC3ZtykTJDERFyLDMcO3YsUH4YT7itwcxijUnqprrr4j7yyCMRRyJR8+23\nfoXKfv36AXD11VcnFlNNKDMUESFHMkPfL+nBBx9MOBIRqS2fGfp1kusbZYYiIuRIZujXQP7qq6/S\n3vejTTTdk4hETZmhiAiqDEVEgBy5TS7Lr5w2b948AFq3bp1kOCLyA6DMUESEHMkMb7jhhrSfIiJx\nU2YoIgKYc676B5t9Bqyr8sD8UeCca5t0EHFSGec/lXFmNaoMRUTylW6TRURQZSgiAkT8NNnM9gDm\nlb5sB+wAPit9/TPnXOYZP+t2zS5AeD6ojsANzjnNAhGBhMq4AJgM7Ak44O8q3+gkUcal150M9AE2\nOOe6RXGNtOvF1WZoZrcBXzvnRpR530rj2BnBNRsBG4DDnHPrs31+SRdXGZvZ3sCezrkVZtYKWA6c\n6px7Nxvnl4rF+T02sxOArcC4OCrDRG6TzayTmRWZ2cPAKqCDmf0vtH+gmU0o3d7LzKab2TIzW2pm\nR9bgUr2At1QRxi/KMnbObXTOrSjd/hJ4G9gnut9GMon6e+ycWwBsiuwXKCPJNsODgJHOuS6UZG8V\neQAY7pzrDpwD+P+5PcxsTBXXGAj8KxvBSq1EXsZm9mOgK/BqdkKWGorjexyLJEegrHHOlV8LtLye\nwIGh6f53N7OmzrklwJKKPmRmTYDTgOvqHKnUVtRl3Ap4AhjinPu6ztFKbURaxnFKsjLcEtreCYQX\nN2kS2jZq10h7GrDEOVdcy/ik7iIrYzNrDEwHJjnnZtUpSqmLqL/HscmJrjWlja6bzWx/M2sAnBna\nPRe40r8ws+o2pJ6HbpFzRjbLuLSx/h/ACufcAxGEK7UQ0fc4NjlRGZa6HpgDLALCDzyuBI4xszfM\nrAi4FCpvazCzlsDPgZnRhiw1lK0yPoGSP3a9zGxF6X+/iDh2qZ5sfo+nAQuBLma23sx+HWXgGo4n\nIkJuZYYiIolRZSgigipDERFAlaGICKDKUEQEqGGn6zZt2rjCwsKIQsk9H3zwAcXFxVb1kflDZZz/\nVMaZ1agyLCwsZNmy6oy8yQ/du3dPOoTYqYzzn8o4M90mi4igylBEBFBlKCICqDIUEQFUGYqIAKoM\nRUSAZCd3rdCWLSXzRQ4dOhSAMWOCGX78Y/Jp06YBUFBQEHN0IpKPlBmKiJCjmeHGjRsBGD9+PAAN\nGzZM7fOdRWfPng3A73//+5ijk9p47bXXABgwYABQMiqgtp577rnUdufOnQHo0KFD7YOTxPjvcd++\nfQEYPXo0AIMHD04dE/7+R0mZoYgIOZYZfvbZZwAMGjQo4Ugk2+bMmQPAtm3b6nyuWbOC9Z8mTpwI\nwNSpU+t8XonP559/DqRngABDhgwB4JJLLkm917Rp01hiUmYoIkKOZIYPPFCywNnMmSXrN736atXr\ngS9cuBAAv4bLIYccAsDxxx8fRYhSS99//z0ATz/9dNbOGR54f//99wNBD4TmzZtn7ToSnZdeegmA\nDRvS150/77zzAGjSpEm5z0RNmaGICDmSGV5zzTVAzZ4aTZ8+Pe3nfvvtB8Bjjz2WOubwww/PVohS\nSy+88AIAixYtAuD666+v8zk3bdqU2l61ahUA33zzDaDMMJeF24vvuuuujMdceOGFAJQsjR0vZYYi\nIqgyFBEBEr5N7tOnDxA8BNmxY0eVn2nTpg0Q3A6tW7cOgLVr1wJwxBFHpI7duXNn9oKValu5cmVq\ne+DAgQB06tQJgBtvvLHO5w93rZH644033kht+0743i67lFRFp556aqwxhSkzFBEhgcxwwYIFqe23\n334bCBpLK3qAcvnll6e2e/fuDcCuu+4KwPz58wH4y1/+Uu5zf//734HyHTslWuGy8A82pkyZAkCL\nFi1qfV7/4CT8byiJhnapHf+wM5NevXrFGElmygxFRIgxM/QD830bEkBxcXHGY303mbPOOguAW2+9\nNbWvWbNmacf6KbzGjh1b7pzDhg0D4NtvvwWCSR0aNWpUu19CKvX4448D6R2sfVthuC23tnx3jHA2\neOKJJwKw22671fn8Eq1wRu81btwYgLvvvjvucMpRZigiQoyZ4fbt24GKs0EIhtI9+uijQPDkuDI+\nM/RPKa+77rrUPj9Ey2eIfpqgjh071ih2qR4/4a7//w7Zaa/1dxWPPPIIEDx5BLjpppsAZfu5zHe4\nf+WVV8rt83d63bp1izWmTJQZioiQI8PxfHvSpEmTgOplhGX5rO/hhx9Ovbd06dIsRCdV+eKLLwBY\nvHhxuX1XXHFFnc8/btw4IJjirUuXLql9J510Up3PL9GqbOKVXOrpocxQRIQEMsNMo0yWLFlS5/P6\nUSzhUSdlR7b4p9K+z5tkhx+Av379eiCYhilb1qxZk/a6a9euWT2/RCtTZuif/mfjziFblBmKiKDK\nUEQEiPE22a99HNVKV36VreXLl6feKzvM7/bbb4/k2j90LVu2BILuEeGJGvwQutatW9f4vJ9++ikQ\ndNnxjjnmmFrFKfF6+eWXgaBLVJgfTrvvvvvGGlNllBmKiBBjZvjUU09l9Xy+m0VRURFQ+XAe31VH\nHXOj4Vcv80Pv/LA8gNNOOw1I7wyfyZtvvpna9g9M/PRsZSdjaNBAf8PrA78Cnn+QGZYLEzOUpX9V\nIiLkSKfr2vDTRD300EMVHlNYWAjA5MmTgWACCInGbbfdBqRnAv6OIDxBRyZt27ZNbftMsKKhmxdf\nfHFdwpSYlG3rDU+mcdlll8UdTpWUGYqIUA8zQ79UgJ8YtjJ+2NZxxx0XaUxSonPnzkD6CoX+6X7Z\njtNl+enawgYNGgSU7yTv2yglN/nO92WfIoefHGdjSrdsU2YoIkKMmWFliz4988wzaa8vvfRSADZu\n3Fjheaoz3Xu2n2BLzR166KFpP2vixz/+ccb3w/0Yf/rTn9YuMImMn7Kr7FPkfv36JRFOtSkzFBFB\nlaGICBDjbbKft8zPOh3mO+aWHaqXaeiev82uzkp6Ur/526yyt1u6Nc5tvrO15wc9XHPNNUmEU23K\nDEVEiDEzHDBgAADDhw9PvVfZeihV8X9tfHeO8ePHA9C+fftan1Nyi39IprWR65c5c+akve7QoQMQ\nTM6Qq5QZiogQY2boV7HzK98BzJw5E4BRo0bV+Hx//vOfgWAtZMk/fr1rT52tc5tfAXP16tVp7zdp\n0gTI/YlSlBmKiJDAcDy/NnJ4u3fv3kCwCpqfqPWMM84A4He/+13qM/7JYniFNMlPfrVEP8D/lltu\nSTIcqYKfWs0PtVu1ahUA+++/f2Ix1YQyQxERcmSihlNOOSXtpwgEGca1114LaI3kXOf7/vrp9Xwv\ngMMOOyyxmGpCmaGICDmSGYpk4tuOpX7Ze++9AZg4cWLCkdSMMkMREVQZiogAqgxFRABVhiIigCpD\nERFAlaGICACWabX7Cg82+wxYF104OafAOde26sPyh8o4/6mMM6tRZSgikq90mywigipDERFAlaGI\nCBDx2GQz2wOYV/qyHbAD+Kz09c+cc99FdN0+wEigITDWOfe3KK4jyZVx6bV3AV4D3nfO9Y/qOj90\nCX6PJwN9gA3OuW5RXCPtenE9QDGz24CvnXMjyrxvpXHszNJ1GgHvAD8HPgaWAb90zr2bjfNLxeIq\n49B5hwHdgGaqDOMRZxmb2QnAVmBcHJVhIrfJZtbJzIrM7GFgFdDBzP4X2j/QzCaUbu9lZtPNbJmZ\nLTWzI6s4/ZHAW865dc65bcBjQL+ofhfJLOIyxswKgF7ApKh+B6lc1GXsnFsAbIrsFygjyTbDg4CR\nzrkuwIZKjnsAGO6c6w6cA/j/uT3MbEyG4/cBPgq9Xl/6nsQvqjIGGAUMBdQ3LFlRlnGskpzPcI1z\nblk1jusJHBhaO3d3M2vqnFsCLIksOsmGSMrYzPoDHznnVphZz+yFK7WQN9/jJCvDLaHtnUB4pfAm\noW2jZo20G4AOodf7UvlfLIlOVGV8NDDAzPqWnqeVmU12zg2qU7RSG1GVcexyomtNaaPrZjPb38wa\nAGeGds8FrvQvzKyqhtTFQBczKzCzH1GSks/KdsxSM9ksY+fcMOfcvs65QuAC4DlVhMnL8vc4djlR\nGZa6HpgDLKKknc+7EjjGzN4wsyLgUqi4rcE5tx24CngeKAKmOOfeiTp4qZaslLHktKyVsZlNAxZS\nktysN7NfRxm4xiaLiJBbmaGISGJUGYqIoMpQRARQZSgiAtSwn2GbNm1cYWFhRKHkng8++IDi4mKr\n+sj8oTLOfyrjzGpUGRYWFrJsWXU6m+eH7t27Jx1C7FTG+U9lnJluk0VEUGUoIgKoMhQRAVQZiogA\nqgxFRABVhiIigCpDEREg2cldRUQA2Lx5MwAffvhhhccUFBQAMHLkSAC6du0KwAEHHADAIYccUqcY\nlBmKiJBwZvjpp58CcM455wBw9NFHA3DZZZcBJT3ls+GLL74A4KWXXgLglFNOAaBRo0ZZOb+I1MxT\nTz0FwOzZswF48cUXAXjvvfcq/MyBBx4IlAyvA9i2bVva/p0767ZKqTJDERESyAx92wDAT37yEyDI\n3Pbaay8g+xnhYYcdBkBxcTFAalzm/vvvn5XrSPV9+eWXAPzpT38CYNWqVQDMnTs3dYwy9vywZs0a\nAB566CEAxo0bl9q3detWAGoy0/4770S7eocyQxERYswMfVbm2wcBPv/8cwCuvLJk0azRo0dn9Zp3\n3XUXAGvXrgWCv0zKCOM3ZcoUAG666Sag/FNDnzEC7LHHHvEFJpFZv75kPahRo0bV6TwHHXQQEDw9\njooyQxERYswMX3vtNSB4ahR2yy23ZO06b775Zmp7xIgRAJx5Zsnyreeee27WriPV47ODa6+9Fgju\nEMzS59ocMmRIavvBBx8EoHXr1nGEKLXgyxGCzO/YY48Fgt4ajRs3BmDXXXcFoEWLFqnPfP311wD8\n4he/AIKsr0ePHgAceuihqWObNm0KQPPmzbP8W6RTZigigipDEREghttk37H6iSeeKLdv4sSJALRt\n27bO1/G3x7169Sq3b8CAAQC0bNmyzteRmvFNFf5hWUWmTp2a2n7mmWeA4GGLv4X2t12SnC1btgDp\n37PXX38dgJkzZ6Yde9RRRwGwfPlyIL3LnH+Atu+++wLQoEHyeVnyEYiI5IDIM8M//OEPQNC1wneA\nBjj77LOzdp2XX34ZgI8//jj13sUXXwzABRdckLXrSNXWrVuX2p40aVLaPj+Y3newf/7558t93neW\n91nl+eefD0C7du2yH6xUy3fffQfAr371KyDIBgFuvPFGAHr27Jnxs5kGUey3335ZjrDulBmKiBBD\nZui7UPif++yzT2pfXdqA/HCeu+++GwiG/IS7bPg2SYnXihUrUtu+M/Xxxx8PwIIFCwD49ttvAXjk\nkUcA+Otf/5r6zOrVq4Egy+/Xrx8QtCWqy018fBcY/z3zEyuE2/mHDh0KQLNmzWKOLruUGYqIkMBE\nDX7qHoDevXsDsNtuuwEwePDgKj/vO237n4sXL07bn812SKmd8NRKPlP3na69Jk2aAPCb3/wGgMcf\nfzy1zw/w94P4fcahp8nx80+I77nnHiCYYHXhwoWpY3yn6vpOmaGICDFkhldffTUA8+fPB2Djxo2p\nfb79yGcATz75ZJXn88eWHc7VsWNHIGjbkOT861//Kvfev//9bwD69++f8TN+WrVMjjzySCB9OJfE\nY9GiRWmv/TA53z8wnygzFBEhhszw8MMPB2DlypVA+pPGZ599FoDhw4cDsOeeewIwaNCgCs934YUX\nAnDwwQenve+XDPAZoiTnvPPOS237bP/VV18F4O233waCfw8zZswA0if99W3I/j0/9Zov+y5dukQW\nu6QLt+VC8ET/9ttvT73Xt29fIH1yhfpImaGICKoMRUQAsJqsQdC9e3dXWUN3HN5//30guB3u1q0b\nAM899xyQnUkfvO7du7Ns2TKr+sj8kY0y3rRpU2rbl5MfYlfRA7DwwH/fgf70008H4N133wWCVRPH\njBlTp/jCVMaVKztoIpOGDRsCcPnllwPBnIQfffQRAJ06dQKCNY/C/Bo4flKHKB7MVLeMlRmKiJDw\nusm1cccddwDBXyr/8CWbGaHUTXi43LRp0wA466yzgPIZ4lVXXQXAvffem/qM75Dtp17zQ/XmzJkD\nBJ2yQQ/MovbHP/4RgPvuu6/CY3bs2AEEGb3/WRP+4emJJ54IpE/pFhdlhiIi1JPM0GcXAJMnTwag\nVatWgFZSy3V+WiffRcNPzOC7z/hM32eDYTfffDMAb731FhB00/GfgeDfg0TDD8Pzq1r66dS2b9+e\nOsavc+MzxNrwk0D773p4JTw/yW/UlBmKiFBPMkPf0TPstNNOA9Ini5Xc5TPEiiYAzcSviuZXNfSZ\n4QsvvJA6xj+51rRe0fBPio844gggeLIfNm/ePCDIFm+77TYAli5dWuPr+bbk//znPzX+bF0pMxQR\noR5mhn7tVP+US/Kfb6+aNWsWkP6k0a+xnM21t6VmTj755LTXfsitzwwbNWoEBMtwAFx66aUAjBw5\nEgjakpOkzFBEBFWGIiJAjt8m+2FX4RXv/KpqenDyw+HX1B02bBiQvj6vb6wfOHAgAAcccEC8wUk5\nfgZ7v2qef7DiZx8CeO+994BgxvqywmslxUWZoYgI9SQzDA8S79OnT9oxX331FRDMfZeL67FKdvhJ\nOe68887Ue/5B2g033AAE63P7bjkSv86dOwNBl6hHH3203DHh7lEAu+xSUhX5LnPh4ZlxUWYoIkKO\nZ4aZ+L8gPgPwj+b98B0Nz8p/F110UWp77NixAEyfPh0I2qLKzoQu8fFZ+ahRo4Dg7i3ckfqTTz4B\noLCwEAjK1LcBJ0GZoYgI9TAzHD9+PAATJkwA4Le//S0QDOqX/Beerm3u3LlAsJ6vn1ggFzrx/tD5\nnh9+rfR//vOfqX2vvPIKEGSCfgqvJCkzFBEhxzPD0aNHA3Drrbem3jv++OMBGDx4MAC77747AI0b\nN445OskFvveAXzbAD9krKioCtJJeLvGrG5bdzhXKDEVEyPHM8LjjjgNg/vz5CUciuc5PHnvIIYcA\nsHr1akCZoVSfMkMREVQZiogAOX6bLFJdfk2ctWvXJhyJ1FfKDEVEUGUoIgKoMhQRAcD8alTVOtjs\nM2BddOHknALnXNuqD8sfKuP8pzLOrEaVoYhIvtJtsogIqgxFRICI+xma2R7AvNKX7YAdwGelr3/m\nnPsuwmvvArwGvO+c6x/VdX7okipjM7sOuKT05Rjn3OgoriOJlvF6YHPp9bY553pEcZ3U9eJqMzSz\n24CvnXMjyrxvpXHszPL1hgHdgGaqDOMRVxmbWTdgMnAk8D3wHPAb55x6XEcszu9xaWXY1Tn3v2yd\nszKJ3CabWSczKzKzh4FVQAcz+19o/0Azm1C6vZeZTTezZWa21MyOrMb5C4BewKSofgepXMRl3BlY\n7Jzb6pzbDrwEnBnV7yKZRf09jluSbYYHASOdc12ADZUc9wAw3DnXHTgH8P9ze5jZmAo+MwoYCuhR\nebKiKuOVwAlm1trMmgOnAh2yG7pUU5TfYwfMN7P/mNklFRyTNUmOTV7jnFtWjeN6AgeGlgvd3cya\nOueWAEvKHmxm/YGPnHMrzKxn9sKVWoikjJ1zb5rZ/cBc4GtgOSXtShK/SMq41JHOuQ1m1g543sze\ncs4tykLMGSVZGW4Jbe8ELPS6SWjbqFkj7dHAADPrW3qeVmY22Tk3qE7RSm1EVcY458YB4wDMbDiw\nug5xSu1FWcYbSn9+bGZPAj8DIqsMc6JrTWmj62Yz29/MGpDe/jMXuNK/KG08r+xcw5xz+zrnCoEL\ngOdUESYvm2VcesyepT8Lgb7A1GzGKzWXzTI2sxZm1qJ0uzklzwDezH7UgZyoDEtdD8yhpOZfH3r/\nSuAYM3vDzIqAS6HKtgbJTdks45mlx84ELnfOfRlh3FJ92Srj9sD/mdnrwFJghnNubpSBazieiAi5\nlRmKiCRGlaGICKoMRUQAVYYiIoAqQxERQJWhiAigylBEBFBlKCICwP8D3P5bzM0W5d8AAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc289fca7f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get the first images from the test-set.\n",
    "images = data.test.images[0:9]\n",
    "\n",
    "# Get the true classes for those images.\n",
    "cls_true = data.test.cls[0:9]\n",
    "\n",
    "# Plot the images and labels using our helper-function above.\n",
    "plot_images(images=images, cls_true=cls_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function to plot example errors\n",
    "\n",
    "Function for plotting examples of images from the test-set that have been mis-classified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_example_errors(cls_pred):\n",
    "    # cls_pred is an array of the predicted class-number for\n",
    "    # all images in the test-set.\n",
    "\n",
    "    # Boolean array whether the predicted class is incorrect.\n",
    "    incorrect = (cls_pred != data.test.cls)\n",
    "\n",
    "    # Get the images from the test-set that have been\n",
    "    # incorrectly classified.\n",
    "    images = data.test.images[incorrect]\n",
    "    \n",
    "    # Get the predicted classes for those images.\n",
    "    cls_pred = cls_pred[incorrect]\n",
    "\n",
    "    # Get the true classes for those images.\n",
    "    cls_true = data.test.cls[incorrect]\n",
    "    \n",
    "    # Plot the first 9 images.\n",
    "    plot_images(images=images[0:9],\n",
    "                cls_true=cls_true[0:9],\n",
    "                cls_pred=cls_pred[0:9])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyper-Parameter Optimization\n",
    "\n",
    "There are several steps required to do hyper-parameter optimization.\n",
    "\n",
    "### Create the Model\n",
    "\n",
    "We first need a function that takes a set of hyper-parameters and creates the Convolutional Neural Network corresponding to those parameters. We use Keras to build the neural network in TensorFlow, see Tutorial #03-C for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def create_model(learning_rate, num_dense_layers,\n",
    "                 num_dense_nodes, activation):\n",
    "    \"\"\"\n",
    "    Hyper-parameters:\n",
    "    learning_rate:     Learning-rate for the optimizer.\n",
    "    num_dense_layers:  Number of dense layers.\n",
    "    num_dense_nodes:   Number of nodes in each dense layer.\n",
    "    activation:        Activation function for all layers.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Start construction of a Keras Sequential model.\n",
    "    model = Sequential()\n",
    "\n",
    "    # Add an input layer which is similar to a feed_dict in TensorFlow.\n",
    "    # Note that the input-shape must be a tuple containing the image-size.\n",
    "    model.add(InputLayer(input_shape=(img_size_flat,)))\n",
    "\n",
    "    # The input from MNIST is a flattened array with 784 elements,\n",
    "    # but the convolutional layers expect images with shape (28, 28, 1)\n",
    "    model.add(Reshape(img_shape_full))\n",
    "\n",
    "    # First convolutional layer.\n",
    "    # There are many hyper-parameters in this layer, but we only\n",
    "    # want to optimize the activation-function in this example.\n",
    "    model.add(Conv2D(kernel_size=5, strides=1, filters=16, padding='same',\n",
    "                     activation=activation, name='layer_conv1'))\n",
    "    model.add(MaxPooling2D(pool_size=2, strides=2))\n",
    "\n",
    "    # Second convolutional layer.\n",
    "    # Again, we only want to optimize the activation-function here.\n",
    "    model.add(Conv2D(kernel_size=5, strides=1, filters=36, padding='same',\n",
    "                     activation=activation, name='layer_conv2'))\n",
    "    model.add(MaxPooling2D(pool_size=2, strides=2))\n",
    "\n",
    "    # Flatten the 4-rank output of the convolutional layers\n",
    "    # to 2-rank that can be input to a fully-connected / dense layer.\n",
    "    model.add(Flatten())\n",
    "\n",
    "    # Add fully-connected / dense layers.\n",
    "    # The number of layers is a hyper-parameter we want to optimize.\n",
    "    for i in range(num_dense_layers):\n",
    "        # Name of the layer. This is not really necessary\n",
    "        # because Keras should give them unique names.\n",
    "        name = 'layer_dense_{0}'.format(i+1)\n",
    "\n",
    "        # Add the dense / fully-connected layer to the model.\n",
    "        # This has two hyper-parameters we want to optimize:\n",
    "        # The number of nodes and the activation function.\n",
    "        model.add(Dense(num_dense_nodes,\n",
    "                        activation=activation,\n",
    "                        name=name))\n",
    "\n",
    "    # Last fully-connected / dense layer with softmax-activation\n",
    "    # for use in classification.\n",
    "    model.add(Dense(num_classes, activation='softmax'))\n",
    "    \n",
    "    # Use the Adam method for training the network.\n",
    "    # We want to find the best learning-rate for the Adam method.\n",
    "    optimizer = Adam(lr=learning_rate)\n",
    "    \n",
    "    # In Keras we need to compile the model so it can be trained.\n",
    "    model.compile(optimizer=optimizer,\n",
    "                  loss='categorical_crossentropy',\n",
    "                  metrics=['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and Evaluate the Model\n",
    "\n",
    "The neural network with the best hyper-parameters is saved to disk so it can be reloaded later. This is the filename for the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path_best_model = '19_best_model.keras'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the classification accuracy for the model saved to disk. It is a global variable which will be updated during optimization of the hyper-parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "best_accuracy = 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the function that creates and trains a neural network with the given hyper-parameters, and then evaluates its performance on the validation-set. The function then returns the so-called fitness value (aka. objective value), which is the negative classification accuracy on the validation-set. It is negative because `skopt` performs minimization instead of maximization.\n",
    "\n",
    "Note the function decorator `@use_named_args` which wraps the fitness function so that it can be called with all the parameters as a single list, for example: `fitness(x=[1e-4, 3, 256, 'relu'])`. This is the calling-style `skopt` uses internally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "@use_named_args(dimensions=dimensions)\n",
    "def fitness(learning_rate, num_dense_layers,\n",
    "            num_dense_nodes, activation):\n",
    "    \"\"\"\n",
    "    Hyper-parameters:\n",
    "    learning_rate:     Learning-rate for the optimizer.\n",
    "    num_dense_layers:  Number of dense layers.\n",
    "    num_dense_nodes:   Number of nodes in each dense layer.\n",
    "    activation:        Activation function for all layers.\n",
    "    \"\"\"\n",
    "\n",
    "    # Print the hyper-parameters.\n",
    "    print('learning rate: {0:.1e}'.format(learning_rate))\n",
    "    print('num_dense_layers:', num_dense_layers)\n",
    "    print('num_dense_nodes:', num_dense_nodes)\n",
    "    print('activation:', activation)\n",
    "    print()\n",
    "    \n",
    "    # Create the neural network with these hyper-parameters.\n",
    "    model = create_model(learning_rate=learning_rate,\n",
    "                         num_dense_layers=num_dense_layers,\n",
    "                         num_dense_nodes=num_dense_nodes,\n",
    "                         activation=activation)\n",
    "\n",
    "    # Dir-name for the TensorBoard log-files.\n",
    "    log_dir = log_dir_name(learning_rate, num_dense_layers,\n",
    "                           num_dense_nodes, activation)\n",
    "    \n",
    "    # Create a callback-function for Keras which will be\n",
    "    # run after each epoch has ended during training.\n",
    "    # This saves the log-files for TensorBoard.\n",
    "    # Note that there are complications when histogram_freq=1.\n",
    "    # It might give strange errors and it also does not properly\n",
    "    # support Keras data-generators for the validation-set.\n",
    "    callback_log = TensorBoard(\n",
    "        log_dir=log_dir,\n",
    "        histogram_freq=0,\n",
    "        batch_size=32,\n",
    "        write_graph=True,\n",
    "        write_grads=False,\n",
    "        write_images=False)\n",
    "   \n",
    "    # Use Keras to train the model.\n",
    "    history = model.fit(x=data.train.images,\n",
    "                        y=data.train.labels,\n",
    "                        epochs=3,\n",
    "                        batch_size=128,\n",
    "                        validation_data=validation_data,\n",
    "                        callbacks=[callback_log])\n",
    "\n",
    "    # Get the classification accuracy on the validation-set\n",
    "    # after the last training-epoch.\n",
    "    accuracy = history.history['val_acc'][-1]\n",
    "\n",
    "    # Print the classification accuracy.\n",
    "    print()\n",
    "    print(\"Accuracy: {0:.2%}\".format(accuracy))\n",
    "    print()\n",
    "\n",
    "    # Save the model if it improves on the best-found performance.\n",
    "    # We use the global keyword so we update the variable outside\n",
    "    # of this function.\n",
    "    global best_accuracy\n",
    "\n",
    "    # If the classification accuracy of the saved model is improved ...\n",
    "    if accuracy > best_accuracy:\n",
    "        # Save the new model to harddisk.\n",
    "        model.save(path_best_model)\n",
    "        \n",
    "        # Update the classification accuracy.\n",
    "        best_accuracy = accuracy\n",
    "\n",
    "    # Delete the Keras model with these hyper-parameters from memory.\n",
    "    del model\n",
    "    \n",
    "    # Clear the Keras session, otherwise it will keep adding new\n",
    "    # models to the same TensorFlow graph each time we create\n",
    "    # a model with a different set of hyper-parameters.\n",
    "    K.clear_session()\n",
    "    \n",
    "    # NOTE: Scikit-optimize does minimization so it tries to\n",
    "    # find a set of hyper-parameters with the LOWEST fitness-value.\n",
    "    # Because we are interested in the HIGHEST classification\n",
    "    # accuracy, we need to negate this number so it can be minimized.\n",
    "    return -accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Run\n",
    "\n",
    "Before we run the hyper-parameter optimization, let us first check that the various functions above actually work, when we pass the default hyper-parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "learning rate: 1.0e-05\n",
      "num_dense_layers: 1\n",
      "num_dense_nodes: 16\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.2525 - acc: 0.1995 - val_loss: 2.1754 - val_acc: 0.3578\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 2s - loss: 2.0279 - acc: 0.4612 - val_loss: 1.8432 - val_acc: 0.5558\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 4s - loss: 1.6227 - acc: 0.5998 - val_loss: 1.3877 - val_acc: 0.6654\n",
      "\n",
      "Accuracy: 66.54%\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "-0.66539999999999999"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitness(x=default_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the Hyper-Parameter Optimization\n",
    "\n",
    "Now we are ready to run the actual hyper-parameter optimization using Bayesian optimization from the scikit-optimize package. Note that it first calls `fitness()` with `default_parameters` as the starting point we have found by hand-tuning, which should help the optimizer locate better hyper-parameters faster.\n",
    "\n",
    "There are many more parameters you can experiment with here, including the number of calls to the `fitness()` function which we have set to 40. But `fitness()` is very expensive to evaluate so it should not be run too many times, especially for larger neural networks and datasets.\n",
    "\n",
    "You can also experiment with the so-called acquisition function which determines how to find a new set of hyper-parameters from the internal model of the Bayesian optimizer. You can also try using another Bayesian optimizer such as Random Forests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "learning rate: 1.0e-05\n",
      "num_dense_layers: 1\n",
      "num_dense_nodes: 16\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 2s - loss: 2.2287 - acc: 0.1868 - val_loss: 2.1264 - val_acc: 0.3182\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 2s - loss: 1.9607 - acc: 0.4438 - val_loss: 1.7713 - val_acc: 0.5082\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 5s - loss: 1.5763 - acc: 0.5579 - val_loss: 1.3832 - val_acc: 0.6166\n",
      "\n",
      "Accuracy: 61.66%\n",
      "\n",
      "learning rate: 6.1e-04\n",
      "num_dense_layers: 2\n",
      "num_dense_nodes: 474\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 1.3354 - acc: 0.5258 - val_loss: 0.3002 - val_acc: 0.9112\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 0.2336 - acc: 0.9269 - val_loss: 0.1626 - val_acc: 0.9538\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.1403 - acc: 0.9563 - val_loss: 0.1113 - val_acc: 0.9692\n",
      "\n",
      "Accuracy: 96.92%\n",
      "\n",
      "learning rate: 6.1e-06\n",
      "num_dense_layers: 2\n",
      "num_dense_nodes: 333\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.1702 - acc: 0.5067 - val_loss: 1.9186 - val_acc: 0.6892\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 1.4878 - acc: 0.7480 - val_loss: 1.0546 - val_acc: 0.7940\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.8226 - acc: 0.8264 - val_loss: 0.6324 - val_acc: 0.8514\n",
      "\n",
      "Accuracy: 85.14%\n",
      "\n",
      "learning rate: 1.7e-04\n",
      "num_dense_layers: 4\n",
      "num_dense_nodes: 252\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.3075 - acc: 0.1058 - val_loss: 2.2968 - val_acc: 0.1126\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 1.5272 - acc: 0.4944 - val_loss: 0.8210 - val_acc: 0.7386\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.6595 - acc: 0.7967 - val_loss: 0.4940 - val_acc: 0.8544\n",
      "\n",
      "Accuracy: 85.44%\n",
      "\n",
      "learning rate: 7.3e-03\n",
      "num_dense_layers: 3\n",
      "num_dense_nodes: 166\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.1821 - acc: 0.9431 - val_loss: 0.0705 - val_acc: 0.9808\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.0605 - acc: 0.9829 - val_loss: 0.0678 - val_acc: 0.9848\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.0549 - acc: 0.9855 - val_loss: 0.0736 - val_acc: 0.9846\n",
      "\n",
      "Accuracy: 98.46%\n",
      "\n",
      "learning rate: 6.1e-05\n",
      "num_dense_layers: 2\n",
      "num_dense_nodes: 209\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 2s - loss: 2.3187 - acc: 0.1073 - val_loss: 2.3030 - val_acc: 0.0924\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 2s - loss: 2.3016 - acc: 0.1121 - val_loss: 2.2993 - val_acc: 0.1126\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 5s - loss: 2.2858 - acc: 0.1573 - val_loss: 2.2243 - val_acc: 0.2898\n",
      "\n",
      "Accuracy: 28.98%\n",
      "\n",
      "learning rate: 1.8e-04\n",
      "num_dense_layers: 4\n",
      "num_dense_nodes: 453\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.3601 - acc: 0.8920 - val_loss: 0.1234 - val_acc: 0.9640\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.0850 - acc: 0.9741 - val_loss: 0.0576 - val_acc: 0.9830\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.0566 - acc: 0.9824 - val_loss: 0.0535 - val_acc: 0.9856\n",
      "\n",
      "Accuracy: 98.56%\n",
      "\n",
      "learning rate: 5.5e-06\n",
      "num_dense_layers: 4\n",
      "num_dense_nodes: 186\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.3129 - acc: 0.1039 - val_loss: 2.3025 - val_acc: 0.1100\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 2.3016 - acc: 0.1106 - val_loss: 2.3010 - val_acc: 0.1126\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 2.3013 - acc: 0.1123 - val_loss: 2.3011 - val_acc: 0.1126\n",
      "\n",
      "Accuracy: 11.26%\n",
      "\n",
      "learning rate: 3.1e-05\n",
      "num_dense_layers: 3\n",
      "num_dense_nodes: 427\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.3132 - acc: 0.1070 - val_loss: 2.3007 - val_acc: 0.1126\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 2.3029 - acc: 0.1080 - val_loss: 2.3020 - val_acc: 0.1126\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 2.3021 - acc: 0.1093 - val_loss: 2.3016 - val_acc: 0.1126\n",
      "\n",
      "Accuracy: 11.26%\n",
      "\n",
      "learning rate: 1.4e-04\n",
      "num_dense_layers: 2\n",
      "num_dense_nodes: 29\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.8474 - acc: 0.7524 - val_loss: 0.2954 - val_acc: 0.9190\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.2392 - acc: 0.9315 - val_loss: 0.1741 - val_acc: 0.9512\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 5s - loss: 0.1643 - acc: 0.9517 - val_loss: 0.1346 - val_acc: 0.9612\n",
      "\n",
      "Accuracy: 96.12%\n",
      "\n",
      "learning rate: 3.7e-04\n",
      "num_dense_layers: 4\n",
      "num_dense_nodes: 338\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.1610 - acc: 0.1844 - val_loss: 1.0813 - val_acc: 0.6678\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 0.5982 - acc: 0.8131 - val_loss: 0.3252 - val_acc: 0.9100\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.2712 - acc: 0.9201 - val_loss: 0.1858 - val_acc: 0.9468\n",
      "\n",
      "Accuracy: 94.68%\n",
      "\n",
      "learning rate: 1.7e-06\n",
      "num_dense_layers: 4\n",
      "num_dense_nodes: 512\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.2568 - acc: 0.3895 - val_loss: 2.1984 - val_acc: 0.6048\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 2.0854 - acc: 0.6719 - val_loss: 1.9276 - val_acc: 0.7052\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 1.7106 - acc: 0.7158 - val_loss: 1.4589 - val_acc: 0.7290\n",
      "\n",
      "Accuracy: 72.90%\n",
      "\n",
      "learning rate: 1.4e-03\n",
      "num_dense_layers: 2\n",
      "num_dense_nodes: 62\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.2396 - acc: 0.9249 - val_loss: 0.0643 - val_acc: 0.9822\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.0587 - acc: 0.9819 - val_loss: 0.0536 - val_acc: 0.9838\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 6s - loss: 0.0427 - acc: 0.9867 - val_loss: 0.0480 - val_acc: 0.9842\n",
      "\n",
      "Accuracy: 98.42%\n",
      "\n",
      "learning rate: 2.7e-03\n",
      "num_dense_layers: 2\n",
      "num_dense_nodes: 364\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 1.3014 - acc: 0.5223 - val_loss: 0.2531 - val_acc: 0.9232\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.1956 - acc: 0.9386 - val_loss: 0.1221 - val_acc: 0.9650\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 5s - loss: 0.1138 - acc: 0.9646 - val_loss: 0.0846 - val_acc: 0.9758\n",
      "\n",
      "Accuracy: 97.58%\n",
      "\n",
      "learning rate: 5.6e-04\n",
      "num_dense_layers: 5\n",
      "num_dense_nodes: 13\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.9357 - acc: 0.6775 - val_loss: 0.3024 - val_acc: 0.9184\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.2416 - acc: 0.9338 - val_loss: 0.1749 - val_acc: 0.9520\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.1685 - acc: 0.9525 - val_loss: 0.1541 - val_acc: 0.9570\n",
      "\n",
      "Accuracy: 95.70%\n",
      "\n",
      "learning rate: 1.0e-02\n",
      "num_dense_layers: 5\n",
      "num_dense_nodes: 352\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.3316 - acc: 0.1049 - val_loss: 2.3019 - val_acc: 0.1070\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.3024 - acc: 0.1090 - val_loss: 2.3017 - val_acc: 0.1126\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 8s - loss: 2.3020 - acc: 0.1104 - val_loss: 2.3014 - val_acc: 0.1126\n",
      "\n",
      "Accuracy: 11.26%\n",
      "\n",
      "learning rate: 1.5e-03\n",
      "num_dense_layers: 1\n",
      "num_dense_nodes: 5\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 2s - loss: 1.7072 - acc: 0.4784 - val_loss: 1.2153 - val_acc: 0.6980\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.9949 - acc: 0.7914 - val_loss: 0.7749 - val_acc: 0.8564\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 6s - loss: 0.6663 - acc: 0.8663 - val_loss: 0.5469 - val_acc: 0.9014\n",
      "\n",
      "Accuracy: 90.14%\n",
      "\n",
      "learning rate: 1.0e-03\n",
      "num_dense_layers: 3\n",
      "num_dense_nodes: 496\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.1843 - acc: 0.9426 - val_loss: 0.0483 - val_acc: 0.9852\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 5s - loss: 0.0506 - acc: 0.9840 - val_loss: 0.0471 - val_acc: 0.9856\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.0347 - acc: 0.9889 - val_loss: 0.0451 - val_acc: 0.9856\n",
      "\n",
      "Accuracy: 98.56%\n",
      "\n",
      "learning rate: 3.7e-03\n",
      "num_dense_layers: 5\n",
      "num_dense_nodes: 512\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 4s - loss: 0.2060 - acc: 0.9377 - val_loss: 0.0739 - val_acc: 0.9832\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 5s - loss: 0.0781 - acc: 0.9814 - val_loss: 0.0765 - val_acc: 0.9842\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 8s - loss: 0.0908 - acc: 0.9818 - val_loss: 0.1368 - val_acc: 0.9766\n",
      "\n",
      "Accuracy: 97.66%\n",
      "\n",
      "learning rate: 1.0e-02\n",
      "num_dense_layers: 5\n",
      "num_dense_nodes: 512\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.3199 - acc: 0.1105 - val_loss: 2.3015 - val_acc: 0.1126\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 2.3020 - acc: 0.1104 - val_loss: 2.3011 - val_acc: 0.1126\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 8s - loss: 2.3018 - acc: 0.1110 - val_loss: 2.3013 - val_acc: 0.1126\n",
      "\n",
      "Accuracy: 11.26%\n",
      "\n",
      "learning rate: 1.9e-04\n",
      "num_dense_layers: 4\n",
      "num_dense_nodes: 418\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.3595 - acc: 0.8999 - val_loss: 0.0888 - val_acc: 0.9732\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.0868 - acc: 0.9738 - val_loss: 0.0686 - val_acc: 0.9782\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 8s - loss: 0.0584 - acc: 0.9821 - val_loss: 0.0478 - val_acc: 0.9850\n",
      "\n",
      "Accuracy: 98.50%\n",
      "\n",
      "learning rate: 2.4e-03\n",
      "num_dense_layers: 4\n",
      "num_dense_nodes: 144\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.1906 - acc: 0.9390 - val_loss: 0.0576 - val_acc: 0.9834\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.0550 - acc: 0.9840 - val_loss: 0.0402 - val_acc: 0.9890\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.0380 - acc: 0.9885 - val_loss: 0.0459 - val_acc: 0.9880\n",
      "\n",
      "Accuracy: 98.80%\n",
      "\n",
      "learning rate: 6.8e-03\n",
      "num_dense_layers: 2\n",
      "num_dense_nodes: 105\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.1552 - acc: 0.9507 - val_loss: 0.0498 - val_acc: 0.9860\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.0485 - acc: 0.9853 - val_loss: 0.0534 - val_acc: 0.9836\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 6s - loss: 0.0417 - acc: 0.9875 - val_loss: 0.0496 - val_acc: 0.9852\n",
      "\n",
      "Accuracy: 98.52%\n",
      "\n",
      "learning rate: 2.5e-04\n",
      "num_dense_layers: 2\n",
      "num_dense_nodes: 435\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.3258 - acc: 0.9131 - val_loss: 0.1024 - val_acc: 0.9676\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 0.0856 - acc: 0.9742 - val_loss: 0.0603 - val_acc: 0.9812\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.0601 - acc: 0.9819 - val_loss: 0.0477 - val_acc: 0.9868\n",
      "\n",
      "Accuracy: 98.68%\n",
      "\n",
      "learning rate: 2.5e-06\n",
      "num_dense_layers: 1\n",
      "num_dense_nodes: 409\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.2504 - acc: 0.3689 - val_loss: 2.1796 - val_acc: 0.5498\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.0835 - acc: 0.6384 - val_loss: 1.9688 - val_acc: 0.6812\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 6s - loss: 1.8409 - acc: 0.7098 - val_loss: 1.6977 - val_acc: 0.7404\n",
      "\n",
      "Accuracy: 74.04%\n",
      "\n",
      "learning rate: 4.4e-03\n",
      "num_dense_layers: 3\n",
      "num_dense_nodes: 311\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.1504 - acc: 0.9523 - val_loss: 0.0746 - val_acc: 0.9800\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 0.0559 - acc: 0.9842 - val_loss: 0.0751 - val_acc: 0.9812\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.0431 - acc: 0.9884 - val_loss: 0.0500 - val_acc: 0.9870\n",
      "\n",
      "Accuracy: 98.70%\n",
      "\n",
      "learning rate: 2.1e-03\n",
      "num_dense_layers: 5\n",
      "num_dense_nodes: 436\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.1884 - acc: 0.9418 - val_loss: 0.0664 - val_acc: 0.9840\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 0.0598 - acc: 0.9837 - val_loss: 0.0454 - val_acc: 0.9880\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 8s - loss: 0.0435 - acc: 0.9887 - val_loss: 0.0553 - val_acc: 0.9864\n",
      "\n",
      "Accuracy: 98.64%\n",
      "\n",
      "learning rate: 1.9e-04\n",
      "num_dense_layers: 3\n",
      "num_dense_nodes: 441\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.3664 - acc: 0.8989 - val_loss: 0.1076 - val_acc: 0.9698\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 0.0872 - acc: 0.9736 - val_loss: 0.0626 - val_acc: 0.9816\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.0583 - acc: 0.9824 - val_loss: 0.0504 - val_acc: 0.9856\n",
      "\n",
      "Accuracy: 98.56%\n",
      "\n",
      "learning rate: 1.7e-03\n",
      "num_dense_layers: 1\n",
      "num_dense_nodes: 512\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 1.2528 - acc: 0.5598 - val_loss: 0.2764 - val_acc: 0.9186\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.2010 - acc: 0.9369 - val_loss: 0.1251 - val_acc: 0.9592\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 6s - loss: 0.1203 - acc: 0.9629 - val_loss: 0.0916 - val_acc: 0.9734\n",
      "\n",
      "Accuracy: 97.34%\n",
      "\n",
      "learning rate: 1.5e-03\n",
      "num_dense_layers: 5\n",
      "num_dense_nodes: 285\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.1378 - acc: 0.1588 - val_loss: 1.2723 - val_acc: 0.4116\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 0.5670 - acc: 0.7991 - val_loss: 0.2616 - val_acc: 0.9266\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 8s - loss: 0.1877 - acc: 0.9460 - val_loss: 0.1365 - val_acc: 0.9618\n",
      "\n",
      "Accuracy: 96.18%\n",
      "\n",
      "learning rate: 3.3e-04\n",
      "num_dense_layers: 5\n",
      "num_dense_nodes: 5\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.3570 - acc: 0.0907 - val_loss: 2.3175 - val_acc: 0.0868\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.3074 - acc: 0.0952 - val_loss: 2.3029 - val_acc: 0.1126\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 8s - loss: 2.3019 - acc: 0.1123 - val_loss: 2.3013 - val_acc: 0.1126\n",
      "\n",
      "Accuracy: 11.26%\n",
      "\n",
      "learning rate: 2.3e-04\n",
      "num_dense_layers: 4\n",
      "num_dense_nodes: 512\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 2.1591 - acc: 0.1861 - val_loss: 1.0381 - val_acc: 0.6422\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 5s - loss: 0.6686 - acc: 0.7868 - val_loss: 0.4403 - val_acc: 0.8662\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 8s - loss: 0.3814 - acc: 0.8831 - val_loss: 0.2920 - val_acc: 0.9090\n",
      "\n",
      "Accuracy: 90.90%\n",
      "\n",
      "learning rate: 2.6e-03\n",
      "num_dense_layers: 1\n",
      "num_dense_nodes: 126\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 2s - loss: 1.1633 - acc: 0.5922 - val_loss: 0.1928 - val_acc: 0.9422\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.1490 - acc: 0.9550 - val_loss: 0.0859 - val_acc: 0.9778\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 5s - loss: 0.0885 - acc: 0.9729 - val_loss: 0.0735 - val_acc: 0.9786\n",
      "\n",
      "Accuracy: 97.86%\n",
      "\n",
      "learning rate: 5.7e-04\n",
      "num_dense_layers: 1\n",
      "num_dense_nodes: 246\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 6s - loss: 0.2579 - acc: 0.9261 - val_loss: 0.0748 - val_acc: 0.9782\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 6s - loss: 0.0691 - acc: 0.9787 - val_loss: 0.0502 - val_acc: 0.9858\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 6s - loss: 0.0465 - acc: 0.9854 - val_loss: 0.0423 - val_acc: 0.9880\n",
      "\n",
      "Accuracy: 98.80%\n",
      "\n",
      "learning rate: 2.4e-04\n",
      "num_dense_layers: 1\n",
      "num_dense_nodes: 164\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.4321 - acc: 0.8849 - val_loss: 0.1429 - val_acc: 0.9608\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.1163 - acc: 0.9654 - val_loss: 0.0821 - val_acc: 0.9766\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 4s - loss: 0.0794 - acc: 0.9762 - val_loss: 0.0679 - val_acc: 0.9796\n",
      "\n",
      "Accuracy: 97.96%\n",
      "\n",
      "learning rate: 1.0e-06\n",
      "num_dense_layers: 2\n",
      "num_dense_nodes: 5\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 2s - loss: 2.3000 - acc: 0.1046 - val_loss: 2.2987 - val_acc: 0.1122\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 2s - loss: 2.2981 - acc: 0.1124 - val_loss: 2.2965 - val_acc: 0.1224\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 4s - loss: 2.2959 - acc: 0.1221 - val_loss: 2.2941 - val_acc: 0.1290\n",
      "\n",
      "Accuracy: 12.90%\n",
      "\n",
      "learning rate: 1.3e-05\n",
      "num_dense_layers: 2\n",
      "num_dense_nodes: 512\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 1.6243 - acc: 0.6472 - val_loss: 0.7587 - val_acc: 0.8260\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 0.5184 - acc: 0.8724 - val_loss: 0.3656 - val_acc: 0.9038\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 7s - loss: 0.3292 - acc: 0.9091 - val_loss: 0.2724 - val_acc: 0.9268\n",
      "\n",
      "Accuracy: 92.68%\n",
      "\n",
      "learning rate: 7.6e-05\n",
      "num_dense_layers: 1\n",
      "num_dense_nodes: 241\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.7636 - acc: 0.8233 - val_loss: 0.2393 - val_acc: 0.9368\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 2s - loss: 0.1961 - acc: 0.9448 - val_loss: 0.1449 - val_acc: 0.9612\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 6s - loss: 0.1309 - acc: 0.9617 - val_loss: 0.1068 - val_acc: 0.9688\n",
      "\n",
      "Accuracy: 96.88%\n",
      "\n",
      "learning rate: 2.0e-03\n",
      "num_dense_layers: 4\n",
      "num_dense_nodes: 512\n",
      "activation: relu\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.1668 - acc: 0.9474 - val_loss: 0.0605 - val_acc: 0.9832\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 4s - loss: 0.0548 - acc: 0.9845 - val_loss: 0.0419 - val_acc: 0.9902\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 8s - loss: 0.0408 - acc: 0.9890 - val_loss: 0.0596 - val_acc: 0.9844\n",
      "\n",
      "Accuracy: 98.44%\n",
      "\n",
      "learning rate: 2.2e-03\n",
      "num_dense_layers: 2\n",
      "num_dense_nodes: 326\n",
      "activation: sigmoid\n",
      "\n",
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/3\n",
      "55000/55000 [==============================] - 3s - loss: 1.1358 - acc: 0.5865 - val_loss: 0.2104 - val_acc: 0.9356\n",
      "Epoch 2/3\n",
      "55000/55000 [==============================] - 3s - loss: 0.1576 - acc: 0.9505 - val_loss: 0.0999 - val_acc: 0.9712\n",
      "Epoch 3/3\n",
      "55000/55000 [==============================] - 6s - loss: 0.1011 - acc: 0.9679 - val_loss: 0.0856 - val_acc: 0.9726\n",
      "\n",
      "Accuracy: 97.26%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "search_result = gp_minimize(func=fitness,\n",
    "                            dimensions=dimensions,\n",
    "                            acq_func='EI', # Expected Improvement.\n",
    "                            n_calls=40,\n",
    "                            x0=default_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimization Progress\n",
    "\n",
    "The progress of the hyper-parameter optimization can be easily plotted. The best fitness value found is plotted on the y-axis, remember that this is the negated classification accuracy on the validation-set.\n",
    "\n",
    "Note how few hyper-parameters had to be tried before substantial improvements were found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fc2830202e8>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEYCAYAAACQgLsAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYXFWZ7/HvL7fuQAMhCTRRkHCJCgoT6chluKUhYLyd\nAKKoqJkRDOCNOR48MMIo45ERBm/jERVEJCoSvIM6eAKxIzKCTgKBBJCbEAViIiExdEI6t/f8sVd1\nqjtV3VXVVV3V6d/nefbTe++1dtXbBV1v1lp7r6WIwMzMrJpG1DsAMzPb+Ti5mJlZ1Tm5mJlZ1Tm5\nmJlZ1Tm5mJlZ1Tm5mJlZ1Tm5mFlJJE2WFJJG1TsWa3xOLrZTkPRuSYskdUpaIel2ScfVO67hStLl\nkr5b7zisfpxcbMiT9DHgS8C/Aa3AK4CvArPqGVc+/2vfhhsnFxvSJO0BfBr4UET8OCLWR8TmiPhZ\nRHw81WmS9CVJz6XtS5KaUtl0Sc9I+l+SVqVWzz+msqMk/UXSyLz3O13Sg2l/hKRLJD0pabWk70sa\nn8pyXUjnSPoT8Kt0/n2Slqf6/yLpaUkzyni92ZL+JOl5SZfmxTVS0ifStS9KWixpv1T2akl3SHpB\n0qOS3tHH57lQ0mcl/V7SOkm35mIoUPdlkm5Lr/uEpA+k8zOBTwBnpZbkAxX9x7UhzcnFhrpjgGbg\nJ33UuRQ4GpgK/B1wJHBZXvk+wB7Ay4FzgGsk7RkRvwPWAyfl1X038L20/xHgNOBE4GXAGuCaXu99\nInAI8AZJh5K1qM4GJuW9Z04pr3cc8CrgZOCTkg5J5z8GvAt4E7A78H5gg6RdgTtSzHsD7wS+mmIp\n5n3p+knAFuDLRerNA55JsZ4J/JukkyLil2StyFsioiUi/q6P97KdVUR48zZkN7Iv6r/0U+dJ4E15\nx28Ank7704GXgFF55auAo9P+Z4Ab0v5uZMlm/3T8CHBy3nWTgM3AKGAyEMCBeeWfBG7OO94F2ATM\nKOP19s0r/z3wzrT/KDCrwO9+FvCbXueuBT5V5LNaCFyZd3xoinFkXgyjgP2ArcBueXU/C9yY9i8H\nvlvv/z+81W9zP7ANdauBiZJGRcSWInVeBizPO16eznW/Rq9rNwAtaf97wG8lXQCcAdwXEbnX2h/4\niaRtedduJRv3yflzrzi6jyNig6TVeeWlvN5fisS5H1kS7W1/4ChJa/POjQK+U6BuoZiXA6OBib3q\nvAx4ISJe7FV3Wh+va8OIu8VsqLsH6CLrTirmObIv2ZxXpHP9ioiHyb4030jPLjHIvoTfGBHj8rbm\niHg2/yXy9lcA++YOJI0FJpT5esX8GTioyPlf93rNloi4oI/X2i9v/xVkrafne9V5DhgvabdedXOx\nerr1Yc7JxYa0iPgbWXfTNZJOk7SLpNGS3ijp31O1m4HLJO0laWKqX85tst8DLgROAH6Qd/7rwBWS\n9gdIr9/XHWo/BN4q6e8ljSHrOtIAXi/f9cD/kTRFmcMlTQB+DrxS0nvT5zJa0uvzxmoKeY+kQyXt\nQnazxA8jYmt+hYj4M/Bb4LOSmiUdTjZelftcVwKTJfk7Zpjyf3gb8iLi82QD2pcBfyX71/qHgZ+m\nKp8BFgEPAkuB+9K5Ut1MNsj+q4jI/xf8fwC3AfMlvQjcCxzVR5wPkQ3azyNrxXSSje90VfJ6vXwB\n+D4wH1gHfBMYm7qtTiUbyH+OrFvtKqCpj9f6DnBjqtsMfLRIvXeRjcM8R3ZDxaci4s5UlkvCqyXd\nV+LvYDsRRbj1alYPklqAtcCUiHiq3vFAdisy2UD89fWOxYY2t1zMBpGkt6auu12Bz5G1pJ6ub1Rm\n1efkYja4ZpF1Iz0HTCG7ldjdB7bTqXu3WHr69xayvtungXdExJoC9V5BNmi5H9mdKG+KiKclHUDW\nhz0BWAy8NyI2DU70ZmZWSCO0XC4BFkTEFGBBOi7k28DVEXEI2RPWq9L5q4AvRsTBZE80n1PjeM3M\nrB+N0HJ5FJgeESskTQIWRsSretU5FLguIo7rdV5kdwftExFbJB0DXB4Rb+jvfSdOnBiTJ08uWLZ+\n/Xp23XXXyn6hGnNslXFslXFsldmZY1u8ePHzEbFXf/Ua4Qn91ohYkfb/Qs+nkXNeCayV9GPgAOBO\nshbOnsDavKern6HnXE09SJoDzAFobW3lc5/7XMF6nZ2dtLS0FCyrN8dWGcdWGcdWmZ05tvb29uX9\n12Jw5hYjSwbLCmyzyJJDft01Ba4/E/gbcCBZQvwRWffXROCJvHr7ActKiamtrS2K6ejoKFpWb46t\nMo6tMo6tMjtzbMCiaJS5xSJiRrEySSslTYrt3WKrClR7BlgSEX9M1/yUbJbbG4BxefNK7cv26SfM\nzKxOGmFA/zZgdtqfDdxaoM5/kyWRXD/fScDDKYt2kLVs+rrezMwGUSMklyuBUyQ9DsxIx0iaJul6\ngMjmNboIWCBpKdl8TN9I118MfEzSE2S3I39zkOM3M7Ne6j6gHxGryRY+6n1+EXBu3vEdwOEF6v2R\n7NZkMzNrEHVPLkPJ/Lse5tqb7mbV6nXsPWF3zjv7OE49oa8F/czMhicnlxLNv+thrvr6fLq6srue\nVz6/jqu+Ph/ACcbMrJdGGHMZEq696e7uxJLT1bWFa2+6u04RmZk1LieXEq1ava6s82Zmw5mTS4n2\nnrB7WefNzIYzJ5cSnXf2cTQ19RyiamoaxXlnH1fkCjOz4csD+iXKDdr/21d+yZat25iw56586H0n\nejDfzKwAt1zKcOoJh/LKA7N5Na/437OcWMzMinByKVNz6hrbuHFznSMxM2tcTi5lam4aDcDGLicX\nM7NinFzKNLY5l1y29FPTzGz4cnIpU1Ou5eJuMTOzopxcyjQ2l1w2ObmYmRXj5FKm3JjLS265mJkV\n5eRSpu67xTygb2ZWlJNLmZqbfbeYmVl/nFzK1DzGd4uZmfXHyaVM3S0Xj7mYmRVV9+QiabykOyQ9\nnn7uWaTeKyTNl/SIpIclTU7nb5T0lKQlaZtay3jH+iFKM7N+1T25AJcACyJiCrAgHRfybeDqiDgE\nOBJYlVf28YiYmrYltQw2NzPyS04uZmZFNUJymQXMTftzgdN6V5B0KDAqIu4AiIjOiNgweCFu55aL\nmVn/FBH1DUBaGxHj0r6ANbnjvDqnAecCm4ADgDuBSyJiq6QbgWOALlLLJyK6irzXHGAOQGtra9u8\nefMKxtTZ2UlLS0vBsmdXredrtzzCy/bahQ++c/BnRe4rtnpzbJVxbJVxbJUZaGzt7e2LI2JavxUj\nouYbWTJYVmCbBaztVXdNgevPBP4GHEi2Bs2PgHNS2SRAQBNZy+eTpcTU1tYWxXR0dBQte+rPz8ex\nZ1wd7/rwN4vWqaW+Yqs3x1YZx1YZx1aZgcYGLIoSvmMHZbGwiJhRrEzSSkmTImKFpEn0HEvJeQZY\nEhF/TNf8FDga+GZErEh1uiR9C7ioyuH34FmRzcz61whjLrcBs9P+bODWAnX+Gxgnaa90fBLwMEBK\nSLkutdPIWkQ1M9YPUZqZ9asRksuVwCmSHgdmpGMkTZN0PUBEbCVrkSyQtJSsG+wb6fqb0rmlwETg\nM7UMtnmMp38xM+vPoHSL9SUiVgMnFzi/iGwQP3d8B3B4gXon1TTAXsaMGYUEmzZvZevWbYwc2Qj5\n2cyssfibsUySusddujZ5Chgzs0KcXCrgaffNzPrm5FIB3zFmZtY3J5cKeE0XM7O+OblUIDczsucX\nMzMrzMmlArn5xbq8pouZWUFOLhXwgL6ZWd+cXCrgAX0zs745uVSg2VPAmJn1ycmlAp4Cxsysb04u\nFei+W8xjLmZmBTm5VGD73WJOLmZmhTi5VKCpyc+5mJn1xcmlAmO77xbzcy5mZoU4uVTAd4uZmfXN\nyaUC3XOLeUDfzKwgJ5cKNHvMxcysT3VPLpLGS7pD0uPp554F6rRLWpK3bZR0Wio7QNLvJD0h6RZJ\nY2odc7PnFjMz61PdkwtwCbAgIqYAC9JxDxHRERFTI2IqcBKwAZifiq8CvhgRBwNrgHNqHfBYt1zM\nzPrUCMllFjA37c8FTuun/pnA7RGxQZLIks0Py7h+wLoH9D3mYmZWUCMkl9aIWJH2/wK09lP/ncDN\naX8CsDYicv1TzwAvr36IPXVP/7LJycXMrBBFRO3fRLoT2KdA0aXA3IgYl1d3TUTsMO6SyiYBDwIv\ni4jNkiYC96YuMSTtR9aqeW2R6+cAcwBaW1vb5s2bVzDezs5OWlpaiv4+61/azGevf4CxzSO59AOv\nK1qvFvqLrZ4cW2UcW2UcW2UGGlt7e/viiJjWb8WIqOsGPApMSvuTgEf7qHshcF3esYDngVHp+Bjg\n/5Xyvm1tbVFMR0dH0bKIiI0bN8WxZ1wd7Wd9oc96tdBfbPXk2Crj2Crj2Coz0NiARVHCd2wjdIvd\nBsxO+7OBW/uo+y62d4mRftEOsnGYUq6vijFjRiHBps1b2bp1W63fzsxsyGmE5HIlcIqkx4EZ6RhJ\n0yRdn6skaTKwH/DrXtdfDHxM0hNkYzDfrHXAkrbfjrzJtyObmfU2qt4BRMRq4OQC5xcB5+YdP02B\nwfqI+CNwZA1DLKi5aTQvbdzMxq7N7DK25o/WmJkNKY3QchmSclPAeE0XM7MdOblUqLnJk1eamRXj\n5FKh7TMje8zFzKy3kpOLpLdL2i3tXybpx5KOqF1oja15jFsuZmbFlNNy+ZeIeFHScWR3dX0T+Fpt\nwmp8Y1PLxWMuZmY7Kie5bE0/30z2IOMvgGF7m1RTmgKmy1PAmJntoJzk8qyk68jm9vpPSU1lXr9T\nccvFzKy4cpLD24HbgVMiYi2wJ3BRTaIaAny3mJlZcf0+RCnpRSA3u6WAyGa6z/aB3WsWXQNzcjEz\nK67f5BIRuw1GIEON13QxMytu2I6ZDFSzV6M0MyuqnG4xFSiOiBim3WLpbjE/RGlmtgN3i1VorFsu\nZmZFlTUrsqQ9gSlAc+5cRNxV7aCGAo+5mJkVV3JykXQu2UqQ+wJLgKOBe4CTahNaY+ue/sUPUZqZ\n7aCcAf0LgdcDyyOiHXgdsLYmUQ0B21suHnMxM+utnOSyMSI2Akhqiog/AK+qTViNz2MuZmbFlTPm\n8oykccBPgTskrQGW1yasxtfUfbeYk4uZWW8lJ5eIOD3tXi6pA9gD+OVAA5A0HrgFmAw8DbwjItb0\nqtMOfDHv1KuBd0bETyXdCJwI/C2V/UNELBloXP1xy8XMrLiKHqKMiF9HxG0RsakKMVwCLIiIKcCC\ndNz7/ToiYmpETCW7gWADMD+vysdz5YORWACaUnLxcy5mZjsqZ7GwualbLHe8p6QbqhDDLGBu2p8L\nnNZP/TOB2yNiQxXeu2LdsyK75WJmtgNFRP+1AEn3R8Tr+jtXdgDS2ogYl/YFrMkdF6n/K+ALEfHz\ndHwjcAzQRWr5RERXkWvnAHMAWltb2+bNm1fwPTo7O2lpaekz7ojgX76yGIBPf6iNESMKTWBQfaXE\nVi+OrTKOrTKOrTIDja29vX1xREzrt2JElLQBDwB75h2PB5aWeO2dwLIC2yxgba+6a/p4nUnAX4HR\nvc4JaCJr+XyylJja2tqimI6OjqJl+U5+1xfj2DOujvUbukqqXw2lxlYPjq0yjq0yjq0yA40NWBQl\nfMeWc7fY54F7JP0gHb8duKKUCyNiRrEySSslTYqIFZImAav6eKl3AD+JiO6+qIhYkXa7JH2LQVxj\nprlpNBu7trCxazO7jB22i3Kame2g5DGXiPg2cAawMm1nRMR3qhDDbcDstD8buLWPuu8Cbs4/kRJS\nrkvtNLIW0aDwapRmZoWVNbdYRDwMPFzlGK4Evi/pHLLnZt4BIGkacH5EnJuOJwP7Ab/udf1NkvYi\n6xpbApxf5fiKakpTwHR5Chgzsx7KSi61EBGrgZMLnF8EnJt3/DTw8gL16ja32faWi29HNjPL58XC\nBsBLHZuZFVbOrMgnAWeTTVa5DHgQWBZFbvsdDnILhjm5mJn1VE632A3APwGjgcPJBs9fAxxcg7iG\nBLdczMwKKye5LI+In6b9H/RZc5jw3WJmZoWVM+Zyl6T/mW75NfLnF3NyMTPLV07L5VDgMOBiSYvJ\nbvtdEhHDthXjmZHNzAorZ8r9twFIGsv2RHMUw7iLLLemy0Z3i5mZ9VD2cy4R8RKwOG3DWq7lsnGT\nn3MxM8vn51wGoDkN6LvlYmbWk5PLADSPybVcnFzMzPKVlFyU2a/WwQw1brmYmRVWUnJJc/j/Z41j\nGXJ8t5iZWWHldIvdJ+n1NYtkCMrdLdbV5QF9M7N85dwtdhTwHklPA+vJpriPiDi8FoENBd1P6Lvl\nYmbWQznJ5Q01i2KI6p5bzGMuZmY9lNMt9ifgeGB2RCwHAmitSVRDhCeuNDMrrJzk8lXgGLKlhgFe\nBK6pekRDyPbk4jEXM7N8ZY25RMQRku4HiIg1ksbUKK4hwWMuZmaFldNy2SxpJFl3GGnd+m3VCELS\neEl3SHo8/dyzSL1/l/SQpEckfTk3Q7OkNklLJT2Rf77WxozOcvOmTVvYti0G4y3NzIaEcpLLl4Gf\nAHtLugK4G/hsleK4BFgQEVOABem4B0l/DxxLtlDZa4HXAyem4q8BHwCmpG1mleLq04gR8mqUZmYF\nlDMr8k1pqv2TyW5DPi0iHqlSHLOA6Wl/LrAQuLh3CEAzMCa9/2hgpaRJwO4RcS+ApG+TrZJ5e5Vi\n61Nz02g2dm1hY9dmdhk7rHsJzcy6KXv4voSK0lURcXF/5yoKQlobEePSvoA1ueNe9T4HnEuWXL4S\nEZdKmgZcGREzUp3jgYsj4i0Frp8DzAFobW1tmzdvXsF4Ojs7aWlpKSn2z934IGtf3MTH3ncY4/do\nKumagSgntsHm2Crj2Crj2Coz0Nja29sXR8S0fitGREkbcF+Bcw+Wcf2dwLIC2yxgba+6awpcfzDw\nC6AlbfeQ3Ro9Dbgzr97xwM/7i6etrS2K6ejoKFrW29kfvSGOPePqeHL5qpKvGYhyYhtsjq0yjq0y\njq0yA40NWBQlfOf32y0m6QLgg8CBkh7MK9oN+K9+s9f2JDajj/dYKWlSRKxI3VyrClQ7Hbg3IjrT\nNbeT3Rr9HWDfvHr7As+WGtdA5cZcXtro25HNzHJKGdB/E/AWYCTw1rytLSLeU6U4bgNmp/3ZwK0F\n6vwJOFHSKEmjyQbzH4mIFcA6SUenLrX3Fbm+JnIzI3d5QN/MrFspyeUgYDPwKLCO7OHJFyG7hbhK\ncVwJnCLpcWBGOkbSNEnXpzo/BJ4ElgIPAA9ExM9S2QeB64EnUp1BGcwHz4xsZlZIKXeLfZ3s9uAD\nyJY2zn+GJIADBxpERKwmuwut9/lFZAP4RMRW4Lwi1y8iuz150DV5Chgzsx3023KJiC9HxCHAtyLi\nwIg4IG8bcGIZ6sY6uZiZ7aCc51wuSE/OTyF73iR3/q5aBDZUeDVKM7MdlZxcJJ0LXEh2N9YS4Giy\n24FPqk1oQ0PzmPSE/ibfLWZmllPO9C8Xkk25sjwi2oHXAWtrEtUQ4paLmdmOykkuGyNiI4Ckpoj4\nA/Cq2oQ1dPhuMTOzHZUz5f4zksYBPwXukLQGWF6bsIYO3y1mZrajcgb0T0+7l0vqAPYAflmTqIYQ\n3y1mZrajclou3SLi19UOZKhqyk257zEXM7Nu5Yy5WAG51Si91LGZ2XZOLgPU7G4xM7MdlJ1cJO2a\nljs2ticX3y1mZrZdv8lF0ghJ75b0C0mrgD8AKyQ9LOlqSQfXPszGlUsunhXZzGy7UlouHWQzI/8z\nsE9E7BcRewPHAfcCV0mq1tT7Q05uzOUlD+ibmXUr5W6xGRGxwzdnRLwA/Aj4UVpfZVjKLRbmAX0z\ns+1KmRV5M4Ck/0iLcRWtMxx5QN/MbEflDOi/CNwmaVcASW+QVPIyxzurpjFpzGXTFrZtizpHY2bW\nGMp5Qv8ySe8GFkraBHQCl9QssiFixAjRNGYUXZu2sLFrM7uMHVPvkMzM6q7kloukk4EPAOuBicBH\nI+I3Aw1A0nhJd0h6PP3cs0i9f5f0kKRHJH0510UnaaGkRyUtSdveA42pXLlB/a5N7hozM4PyusUu\nBf4lIqYDZwK3SKrGWi6XAAsiYgrZcso7tIYk/T1wLHA42XLGrwdOzKtydkRMTduqKsRUlu5nXXzH\nmJkZUEZyiYiTIuLutL8UeCPwmSrEMAuYm/bnAqcVenuy1S/HAE3AaGBlFd67KrbfMebkYmYGoIi+\nB6ElKYpUkjQ2Il7qq06/AUhrI2Jc7r2ANbnjXvU+B5wLCPhKRFyazi8EJgBbyW6N/kwf8c4B5gC0\ntra2zZs3r2BMnZ2dtLS0lPw7fO2Wh3l21QbOf/ur2Xef0q+rRLmxDSbHVhnHVhnHVpmBxtbe3r44\nIqb1WzEi+tyAhcBHgFf0Oj+GbInjucA/9PMadwLLCmyzgLW96q4pcP3BwC+AlrTdAxyfyl6efu4G\nzAfe19/vFBG0tbVFMR0dHUXLCvnQZTfHsWdcHYsfXF7WdZUoN7bB5Ngq49gq49gqM9DYgEVRwnds\nKXeLzQTeD9ws6QCypY2bgZHpy/xLEXF/PwlsRrEySSslTYqIFZImAYXGTE4H7o2IznTN7cAxwG8i\n4tn0Hi9K+h5wJPDtEn6vqul+1sUD+mZmQGljLldFxFeBU4D9gZOBIyJi/4j4QH+JpQS3AbPT/mzg\n1gJ1/gScKGlUmg3gROCRdDwRIJ1/C1mLaFA1ewoYM7MeSkkuJ6Sfv4mIzRGxIiLWVjGGK4FTJD0O\nzEjHSJom6fpU54fAk8BS4AHggYj4Gdng/v+T9CCwBHgW+EYVYyuJB/TNzHoqpVtsgaR7gH0kvZ/s\ny31ZRHRVI4CIWE3WGup9fhHZAD4RsRU4r0Cd9UBbNeIYiO1TwHh+MTMzKCG5RMRFkg4imx35AOB/\nAK9JT+kvi4izahxjwxvr+cXMzHooafqXiHhS0oyIeCx3TlIL2QONw153y8VjLmZmQBlziwHL09xi\nk3tdd29VIxqCcgP6brmYmWXKSS63An8DFgNVGW/ZWXipYzOznspJLvtGxMyaRTKE+W4xM7Oeypm4\n8reSDqtZJEOY7xYzM+upnJbLccA/SHqKrFtMQETE4TWJbAjJTbnvAX0zs0w5yeWNNYtiiGse4wF9\nM7N85axEubyWgQxlvlvMzKynfsdcJN2dfr4oaV36mdvW1T7ExucBfTOznkp5Qv+49HO32oczNHkl\nSjOznkruFpM0DfgEvR6i9ID+9gH9rk2+W8zMDMob0L8J+DjZzMTbahPO0NQ0xi0XM7N85SSXv0bE\nbTWLZAjLdYt1bdrCtm3BiBGqc0RmZvVVTnL5VFpfZQF5079ExI+rHtUQM2KEaBoziq5NW+jatJmx\nzWPqHZKZWV2Vk1z+EXg1MJrt3WIBDPvkAlnrpWvTFjZ2ObmYmZWTXF4fEa+qWSRDXHPTKP72oqeA\nMTOD8ucWO7RmkQxxuTvGPKhvZlZecjkaWCLpUUkPSlqa1q4fEEnjJd0h6fH0c88i9a6StCxtZ+Wd\nP0DS7yQ9IekWSXXpk2ryapRmZt3KSS4zgSnAqcBbgbeknwN1CbAgIqaQ3SxwSe8Kkt4MHAFMBY4C\nLpK0eyq+CvhiRBwMrAHOqUJMZfNSx2Zm25WcXCJieaGtCjHMAuam/bnAaQXqHArcFRFbImI98CAw\nU5KAk4Af9nN9zTU7uZiZdVNE1DcAaW1EjEv7AtbkjvPqnAp8CjgF2AX4PXANWTK5N7VakLQfcHtE\nvLbIe80B5gC0tra2zZs3r2BMnZ2dtLS0lPV73PyfT/LQk2s4a+aBHDZlfFnXlqOS2AaLY6uMY6uM\nY6vMQGNrb29fHBHT+qtXzt1iFZN0J7BPgaJL8w8iIiTtkO0iYr6k1wO/Bf4K3ANsLTeOiLgOuA5g\n2rRpMX369IL1Fi5cSLGyYu5euoGHnlzDQQe9kunTC+a2qqgktsHi2Crj2Crj2CozWLENSnKJiBnF\nyiStlDQpIlZImgSsKvIaVwBXpGu+BzwGrAbGSRoVEVuAfYFnq/4LlCC3pstL7hYzMytrQL9WbgNm\np/3ZwK29K0gaKWlC2j8cOByYH1mfXgdwZl/XDwav6WJmtl0jJJcrgVMkPQ7MSMdImpamm4FsVoDf\nSHqYrFvrPamlAnAx8DFJTwATgG8OavSJ7xYzM9tuULrF+hIRq4GTC5xfBJyb9jeS3TFW6Po/AkfW\nMsZSNOUWDPNDlGZmDdFy2SnkWi4eczEzc3KpmtyYS5fnFjMzc3Kplma3XMzMujm5VEn3E/oeczEz\nc3Kplu67xTY5uZiZOblUSZNbLmZm3ZxcqmRsc7oV2WMuZmZOLtWyfVZk3y1mZubkUiWect/MbDsn\nlypxcjEz287JpUqaxuTGXLawbVt918gxM6s3J5cqGTlyBGNSguny7chmNsw5uVTRWA/qm5kBTi5V\n1T0zssddzGyYc3Kpou6Zkf0gpZkNc04uVdQ9M7LHXMxsmHNyqaLmMW65mJmBk0tVNXsKGDMzoAGS\ni6Txku6Q9Hj6uWeReldJWpa2s/LO3yjpKUlL0jZ18KLvyXeLmZll6p5cgEuABRExBViQjnuQ9Gbg\nCGAqcBRwkaTd86p8PCKmpm3JYARdiGdGNjPLNEJymQXMTftzgdMK1DkUuCsitkTEeuBBYOYgxVey\nsV6N0swMAEXUd6oSSWsjYlzaF7Amd5xX51TgU8ApwC7A74FrIuLzkm4EjgG6SC2fiOgq8l5zgDkA\nra2tbfPmzSsYU2dnJy0tLWX/Lrff/Wf+6/6VvOHYfTn+iH3Kvr4UlcY2GBxbZRxbZRxbZQYaW3t7\n++KImNZvxYio+QbcCSwrsM0C1vaqu6bIa1wKLAHuAG4C/imdnwQIaCJr+XyylJja2tqimI6OjqJl\nffnG934Tx55xdVw/7+6Kri9FpbENBsdWGcdWGcdWmYHGBiyKEr5jR1WcvsoQETOKlUlaKWlSRKyQ\nNAlYVeQ1rgCuSNd8D3gsnV+RqnRJ+hZwUVWDL0PuORePuZjZcNcIYy63AbPT/mzg1t4VJI2UNCHt\nHw4cDszF32AmAAANfUlEQVRPx5PST5GN1ywbhJgL6p52f5PvFjOz4W1QWi79uBL4vqRzgOXAOwAk\nTQPOj4hzgdHAb7L8wTrgPRGR+wa/SdJeZF1jS4DzBzn+bs2+W8zMDGiA5BIRq4GTC5xfBJyb9jeS\n3TFW6PqTahpgGcY2+24xMzNojG6xnUZTmv6ly8nFzIa5urdcdiYPPfYcAPfc9xRvO+86zjv7OE49\nYXuDa/5dD3PtTXezavU69p6we4/yvsrMzIYaJ5cqmX/Xw9zys0XdxyufX8dVX5tP5/ouph/zShbe\n8xjXfPvXdKXB/vxyYMeyr88HcIIxsyHJyaVKrr3pbjZt3trjXNemLXzh+gV84foFBa/JlRcs69rC\ntTfd7eRiZkOSx1yqZNXqdUXL9txjl6q/pplZI3NyqZK9J+xe8HzrxN352Q0fpHVi8fJiZbu3NFct\nPjOzweTkUiXnnX0cTU09exmbmkZx3tnH9VteqAxg3Ysb+fXvHq9d0GZmNeLkUiWnnnAoF59/Kq0T\nd0fKWiQXn39q95hJX+WFyo4/8iAC+NQXfsa99z9V31/OzKxMHtCvolyiqKS8d1lE8H9vXMj3f76Y\nT/z7rXz+srfxutfsV/WYzcxqwcmlQUniI/8wnZc2buJndy7lY5/+Abu1NPPC2g203vxYWc/QmJkN\nNieXBiaJi+acwlN/Xs2yR5/jhbUbgB2fg5l/18Nc9fX5dHUVf06mv+QzkAc8c+Urn19XUeLbWWMr\nNfZKYzNrZHVfLKxepk2bFosWLSpYtnDhQqZPnz64AfXhbeddy8rnX9zh/IgRYvy4XXlh7Xq2bdvx\nv2PTmFHMnP4ann/hRX53/9Ns2bqtu2z0qBG8+eTDOOxVL2fpo8/yiwVL2bxlx3KgaFl/19a6vJFj\nq0XsTWNGcfEFp1blHwz9lfdIfBMHN+k6tvrEVipJJS0W5uRSQKMll+PP/BzD9D+T9TJq5AhOOGoK\nW7dt47eL/sjmLdsf3G0aM4oPz57ePSPEV+Yu7J71oZxyoOJra13u2KoYW9OoHjcdlcrJpR9DKbm8\n7bzrWPn8jg9U7jW+heuuPJs5l9zEX1/o3KF8j93Gcu47j+Xz37iz6GufesIhzL/rkYri6u/aWpc3\ncmy1jN2sWlon7s6Prp1T1jWlJhffijwEFHtG5oL3nsBeE3bjgveeULD8wve3c/rMqX0+wPnJC99c\n0QOepVxb6/JGjq1WsY8ftyuXfuSNBctyxu0+dkDltXxtx1af8mJqOQuIk8sQkP8cDJT3DA1U/wHP\nUq+tdXkjx1ar2D88+0TeOP01fSann3/rQwMqr+VrO7bGiq3YzCLVMPLyyy+v2Ys3suuuu+7yOXMK\nNweffvppJk+ePLgB9eOg/ffirLe2cWDrJi7+yNs4aP+9Cpa//x1/z1lvbetRftD+ezFpr935w5Mr\n2fBSF60Td+fC97d3J5++ysu5dv2G8l57Z46tnNgriW3PPcZy75Kn2Jp3k0autXrQ/nsNqHza4a+o\n2Ws7tsaMrRz/+q//uuLyyy+/rr96dR9zkfR24HLgEODItAJloXozgf8ARgLXR8SV6fwBwDxgArAY\neG9EbOrvfYfSmEs+x1aZnTE23/Xk2Br5bjEioq4bWVJ5FbAQmFakzkjgSeBAYAzwAHBoKvs+8M60\n/3XgglLet62tLYrp6OgoWlZvjq0yjq0yjq0yO3NswKIo4Tu27mMuEfFIRDzaT7UjgSci4o+RtUrm\nAbMkCTgJ+GGqNxc4rXbRmplZKereLZYjaSFwURToFpN0JjAzIs5Nx+8FjiLrTrs3Ig5O5/cDbo+I\n1xZ5jznAHIDW1ta2efPmFYyls7OTlpaWgf5KNeHYKuPYKuPYKrMzx9be3l5St9igTP8i6U5gnwJF\nl0bErYMRA0BEXAdcB9mYS7F+7p2xf34wOLbKOLbKOLbKDFZsg5JcImLGAF/iWSB/SuB907nVwDhJ\noyJiS955MzOro7qPuZTov4Epkg6QNAZ4J3BbGlzqAM5M9WYDg9YSMjOzwuo+5iLpdOD/AnsBa4El\nEfEGSS8ju+X4Tanem4Avkd05dkNEXJHOH0g2wD8euB94T0R0lfC+fwWWFymeCDw/oF+sdhxbZRxb\nZRxbZXbm2PaPiH4fjql7cmlEkhaVMmBVD46tMo6tMo6tMo5t6HSLmZnZEOLkYmZmVefkUli/8+bU\nkWOrjGOrjGOrzLCPzWMuZmZWdW65mJlZ1Tm5mJlZ1Tm59CJppqRHJT0h6ZJ6x5NP0tOSlkpaIqnw\negGDF8sNklZJWpZ3brykOyQ9nn7u2UCxXS7p2fTZLUnPTdUjtv0kdUh6WNJDki5M5+v+2fURW90/\nO0nNkn4v6YEU27+m8wdI+l36e70lPWTdKLHdKOmpvM9t6mDHlhfjSEn3S/p5Oq755+bkkkfSSOAa\n4I3AocC7JJW/4EFttUfE1Aa4h/5GYGavc5cACyJiCrAgHdfDjewYG8AX02c3NSL+c5BjytkC/K+I\nOBQ4GvhQ+n+sET67YrFB/T+7LuCkiPg7YCowU9LRwFUptoOBNcA5DRQbwMfzPrcldYgt50Lgkbzj\nmn9uTi49FZzav84xNaSIuAt4odfpWWTLHkAdlz8oEltDiIgVEXFf2n+R7A/+5TTAZ9dHbHWXlhLp\nTIej0xY0wJIbfcTWECTtC7wZuD4dD8pSJU4uPb0c+HPe8TM0yB9XEsB8SYvT8gGNpjUiVqT9vwCt\n9QymgA9LejB1m9Wlyy6fpMnA64Df0WCfXa/YoAE+u9S1swRYBdxBtoDg2jRpLdTx77V3bBGR+9yu\nSJ/bFyU11SM2smmz/jeQW+N4AoPwuTm5DC3HRcQRZN12H5J0Qr0DKiZNKtow/3oDvgYcRNZtsQL4\nfD2DkdQC/Aj4p4hYl19W78+uQGwN8dlFxNaImEo2+/mRwKvrEUchvWOT9Frgn8lifD3Z3IcXD3Zc\nkt4CrIqIxYP93k4uPRWb2r8hRMSz6ecq4Cdkf2CNZKWkSQDp56o6x9MtIlamL4BtwDeo42cnaTTZ\nl/dNEfHjdLohPrtCsTXSZ5fiWUs2G/oxpCU3UlHd/17zYpuZuhkjTaT7LerzuR0L/A9JT5N1858E\n/AeD8Lk5ufRUcGr/OscEgKRdJe2W2wdOBZb1fdWgu41s2QNosOUPcl/cyenU6bNL/d3fBB6JiC/k\nFdX9sysWWyN8dpL2kjQu7Y8FTiEbE6r7khtFYvtD3j8WRDamMeifW0T8c0TsGxGTyb7PfhURZzMY\nn1tEeMvbgDcBj5H1515a73jy4joQeCBtD9U7NuBmsi6SzWR9tueQ9eUuAB4H7gTGN1Bs3wGWAg+S\nfZFPqlNsx5F1eT0ILEnbmxrhs+sjtrp/dsDhZEtqPEj2Jf3JdP5A4PfAE8APgKYGiu1X6XNbBnwX\naKnH/3N5cU4Hfj5Yn5unfzEzs6pzt5iZmVWdk4uZmVWdk4uZmVWdk4uZmVWdk4uZmVWdk4uZmVWd\nk4uZmVWdk4sNG5JC0ufzji+SdHkVXndy/toxtSTpo5IekXTTAF+ns9C+WbU4udhw0gWcIWlivQPJ\np0ypf4sfBE6JbAoPs4bl5GLDyRbgOuB/5p/s3fLItWjS+T+kFQUfk3STpBmS/kvZipH5ExGOSuWP\nSPqhpF3Sa70nrVK4RNK1aUG63Hs+KunbZNOD7Ncrpo9JWpa2f0rnvk42bcftknr8Dqn8fWl69wck\nfSed+2laouGh/pZpSPPX/SJdv0zSWQXq/FjSZyTdJelPkmb09Zo2fDm52HBzDXC2pD1KrH8w2RTz\nr07bu8nm4LoI+ERevVcBX42IQ4B1wAclHQKcBRwb2XTsW4H8FseUdM1rImJ57qSkNuAfgaPIVoT8\ngKTXRcT5wHNkq5F+MT9ISa8BLmP7iogXpqL3R0QbMA34qKQJffyuM4HnIuLvIuK1wC8L1DmMbC2Q\nE9J7uAVlBTm52LAS2fok3wY+WuIlT0XE0simm3+IbCniIJuQcHJevT9HxH+l/e+SJaCTgTbgv9NC\nUieTtTxylkfEvQXe8zjgJxGxPrIVDn8MHN9PnCcBP4iI59PvmVuJ86OSHgDuJWsdTenjNZYCp0i6\nStLxEfG3/MLUGtsDyCW20cDafuKyYWpU/1XMdjpfAu4jW2MDsu6y/H9oNeftd+Xtb8s73kbPv5/e\nM8AGIGBuRPxzkTjWlxFz2SRNB2YAx0TEBkkL6fm79RARj0k6gmwm5M9IWhARn86rciiwOCK2puPD\nabxlH6xBuOViw076V/33yabiB1gJ7C1pQlqK9i0VvOwrJB2T9t8N3E02hf6ZkvYGkDRe0v4lvNZv\ngNMk7ZLW7jk9nevLr4C357q9JI0na2WsSYnl1WRdbEVJehmwISK+C1wNHNGrymFk0/DnHE42zbzZ\nDtxyseHq88CHASJis6RPk61v8Szwhwpe71GypadvAB4Gvpa+1C8D5qe7wTYDHwKW9/E6RMR9km5M\n8QBcHxH393PNQ5KuAH4taSvZ+iLnAedLeiTFV6gLLt9hwNWStqVYLyhQ/ru849filosV4fVczMys\n6twtZmZmVefkYmZmVefkYmZmVefkYmZmVefkYmZmVefkYmZmVefkYmZmVff/AeWG+IOGdYFxAAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc285dc5cf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_convergence(search_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Best Hyper-Parameters\n",
    "\n",
    "The best hyper-parameters found by the Bayesian optimizer are packed as a list because that is what it uses internally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0023584457378584664, 4, 144, 'relu']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_result.x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert these parameters to a dict with proper names for the search-space dimensions.\n",
    "\n",
    "First we need a reference to the search-space object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "space = search_result.space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can use it to create a dict where the hyper-parameters have the proper names of the search-space dimensions. This is a bit awkward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'activation': 'relu',\n",
       " 'learning_rate': 0.0023584457378584664,\n",
       " 'num_dense_layers': 4,\n",
       " 'num_dense_nodes': 144}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "space.point_to_dict(search_result.x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the fitness value associated with these hyper-parameters. This is a negative number because the Bayesian optimizer performs minimization, so we had to negate the classification accuracy which is posed as a maximization problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.98799999999999999"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_result.fun"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also see all the hyper-parameters tried by the Bayesian optimizer and their associated fitness values (the negated classification accuracies). These are sorted so the highest classification accuracies are shown first.\n",
    "\n",
    "It appears that 'relu' activation was generally better than 'sigmoid'. Otherwise it can be difficult to see a pattern of which parameter choices are good. We really need to plot these results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(-0.98799999999999999, [0.00057102338020535671, 1, 246, 'relu']),\n",
       " (-0.98799999999999999, [0.0023584457378584664, 4, 144, 'relu']),\n",
       " (-0.98699999999999999, [0.0043924439217142824, 3, 311, 'relu']),\n",
       " (-0.98680000000000001, [0.00025070302453255417, 2, 435, 'relu']),\n",
       " (-0.98640000000000005, [0.0020904801989242469, 5, 436, 'relu']),\n",
       " (-0.98560000000000003, [0.00017567744133971055, 4, 453, 'relu']),\n",
       " (-0.98560000000000003, [0.00018871091218374878, 3, 441, 'relu']),\n",
       " (-0.98560000000000003, [0.0010013922052631494, 3, 496, 'relu']),\n",
       " (-0.98519999999999996, [0.006752254693985822, 2, 105, 'relu']),\n",
       " (-0.98499999999999999, [0.0001905308801138268, 4, 418, 'relu']),\n",
       " (-0.98460000000000003, [0.0073224617473678331, 3, 166, 'relu']),\n",
       " (-0.98440000000000005, [0.0020143982003767271, 4, 512, 'relu']),\n",
       " (-0.98419999999999996, [0.0014193250864683331, 2, 62, 'relu']),\n",
       " (-0.97960000000000003, [0.00023735076383216567, 1, 164, 'relu']),\n",
       " (-0.97860000000000003, [0.0026064900033469073, 1, 126, 'sigmoid']),\n",
       " (-0.97660000000000002, [0.0037123587226393501, 5, 512, 'relu']),\n",
       " (-0.9758, [0.0027230837381696737, 2, 364, 'sigmoid']),\n",
       " (-0.97340000000000004, [0.0016597651372777609, 1, 512, 'sigmoid']),\n",
       " (-0.97260000000000002, [0.0022460993827137423, 2, 326, 'sigmoid']),\n",
       " (-0.96919999999999995, [0.00060563429543890952, 2, 474, 'sigmoid']),\n",
       " (-0.96879999999999999, [7.5808558985641429e-05, 1, 241, 'relu']),\n",
       " (-0.96179999999999999, [0.0014963322170155162, 5, 285, 'sigmoid']),\n",
       " (-0.96120000000000005, [0.00013559943302194881, 2, 29, 'relu']),\n",
       " (-0.95699999999999996, [0.00056441093780360571, 5, 13, 'relu']),\n",
       " (-0.94679999999999997, [0.00036704404112128516, 4, 338, 'sigmoid']),\n",
       " (-0.92679999999999996, [1.3066947342663859e-05, 2, 512, 'relu']),\n",
       " (-0.90900000000000003, [0.00023277413216549582, 4, 512, 'sigmoid']),\n",
       " (-0.90139999999999998, [0.001544493082361837, 1, 5, 'sigmoid']),\n",
       " (-0.85440000000000005, [0.00016937303683800523, 4, 252, 'sigmoid']),\n",
       " (-0.85140000000000005, [6.1458838378363633e-06, 2, 333, 'relu']),\n",
       " (-0.74039999999999995, [2.4847514577863683e-06, 1, 409, 'relu']),\n",
       " (-0.72899999999999998, [1.7068698743151031e-06, 4, 512, 'relu']),\n",
       " (-0.61660000000000004, [1e-05, 1, 16, 'relu']),\n",
       " (-0.2898, [6.1011365846453456e-05, 2, 209, 'sigmoid']),\n",
       " (-0.129, [9.9999999999999995e-07, 2, 5, 'relu']),\n",
       " (-0.11260000000000001, [5.4599879082087208e-06, 4, 186, 'sigmoid']),\n",
       " (-0.11260000000000001, [3.1218037895598157e-05, 3, 427, 'sigmoid']),\n",
       " (-0.11260000000000001, [0.00033099542158994725, 5, 5, 'sigmoid']),\n",
       " (-0.11260000000000001, [0.01, 5, 352, 'sigmoid']),\n",
       " (-0.11260000000000001, [0.01, 5, 512, 'relu'])]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(zip(search_result.func_vals, search_result.x_iters))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plots\n",
    "\n",
    "There are several plotting functions available in the `skopt` library. For example, we can plot a histogram for the `activation` parameter, which shows the distribution of samples during the hyper-parameter optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEetJREFUeJzt3XuwXWV9xvHvI2ChwihMMgxa4qGKWrwhBIqiCF46qLTq\ngNi0VazUeMVLtVPG1opFK1SUmdYKhkIRC4gKeINaHQRRW8QkAgkiqAgqRQjFIhErt1//2OvUQziX\nzUnW3uG838/Mmb3Xu9flR+awn/O+a613paqQJLXrIeMuQJI0XgaBJDXOIJCkxhkEktQ4g0CSGmcQ\nSFLjDAJJapxBIEmNMwgkqXFbjruAYSxatKgmJibGXYYkPaisWrXqlqpaPNd6D4ogmJiYYOXKleMu\nQ5IeVJJcP8x6Dg1JUuMMAklqnEEgSY0zCCSpcQaBJDWutyBIsnOSC5N8J8mVSd7StR+V5IYkl3U/\nL+yrBknS3Pq8fPRu4O1VtTrJdsCqJF/uPju+qo7r8diSpCH1FgRVdSNwY/f+9iRXAY/q63iSpPkZ\nyTmCJBPA04Bvdk1vSnJFklOSbD+KGiRJ0+v9zuIk2wJnA2+tqp8nOQE4Gqju9YPAq6fZbjmwHGDJ\nkiXzPv7EkefNe1stbNcd86JxlyBtFnrtESTZikEInF5V5wBU1U1VdU9V3QucBOw93bZVtaKqllbV\n0sWL55wqQ5I0T31eNRTgZOCqqvrQlPadpqz2UmBtXzVIkubW59DQvsArgDVJLuva3gksS7I7g6Gh\n64DX9liDJGkOfV419HUg03x0fl/HlCQ9cN5ZLEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS\n4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXO\nIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwC\nSWpcb0GQZOckFyb5TpIrk7yla98hyZeTfK973b6vGiRJc+uzR3A38Paq2g3YB3hjkt2AI4ELqmpX\n4IJuWZI0Jr0FQVXdWFWru/e3A1cBjwJeDHysW+1jwEv6qkGSNLeRnCNIMgE8DfgmsGNV3dh99FNg\nx1HUIEmaXu9BkGRb4GzgrVX186mfVVUBNcN2y5OsTLJy3bp1fZcpSc3qNQiSbMUgBE6vqnO65puS\n7NR9vhNw83TbVtWKqlpaVUsXL17cZ5mS1LQ+rxoKcDJwVVV9aMpHnwMO694fBny2rxokSXPbssd9\n7wu8AliT5LKu7Z3AMcAnkxwOXA8c2mMNkqQ59BYEVfV1IDN8/Ny+jitJemC8s1iSGmcQSFLjDAJJ\napxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTG\nGQSS1DiDQJIaZxBIUuPmDIIkxw7TJkl6cBqmR/D8adpesKkLkSSNx4wPr0/yeuANwG8nuWLKR9sB\n3+i7MEnSaMwYBMAZwL8B7weOnNJ+e1Xd2mtVkqSRmTEIquo24DZgWZItgB279bdNsm1V/WhENUqS\nejRbjwCAJG8CjgJuAu7tmgt4Sn9lSZJGZc4gAN4KPL6q/rvvYiRJozfMVUM/ZjBEJElagIbpEVwL\nXJTkPOBXk41V9aHeqpIkjcwwQfCj7ueh3Y8kaQGZMwiq6j2jKESSNB7DXDV0IYOrhO6jqp7TS0WS\npJEaZmjoHVPebw0cDNzdTzmSpFEbZmho1QZN30hyaU/1SJJGbJihoR2mLD4E2BN4eG8VSZJGapih\noVUMzhGEwZDQD4HD59ooySnAQcDNVfWkru0o4DXAum61d1bV+Q+8bEnSpjLM0NAu89z3qcCHgdM2\naD++qo6b5z4lSZvYMENDWwGvB/brmi4CPlpVd822XVVdnGRiI+uTJPVsmCkmTmBwXuAj3c+eXdt8\nvSnJFUlOSbL9RuxHkrQJDHOOYK+qeuqU5a8kuXyexzsBOJrBOYejgQ8Cr55uxSTLgeUAS5Ysmefh\npM3fxJHnjbsEbcauO+ZFvR9jmB7BPUkeM7mQ5LeBe+ZzsKq6qaruqap7gZOAvWdZd0VVLa2qpYsX\nL57P4SRJQximR/AXwIVJrmVw5dCjgT+dz8GS7FRVN3aLLwXWzmc/kqRNZ5irhi5Isivw+K7p6qr6\n1WzbACQ5E9gfWJTkJ8C7gf2T7M5gaOg64LXzrFuStInM9vD6PwFSVR/vvviv6NpfkeSeqjpjth1X\n1bJpmk/eqGolSZvcbOcIjgDOnab9HODt/ZQjSRq12YJgq6pav2FjVf0C2Kq/kiRJozRbEGyT5GEb\nNibZDh9QI0kLxmxBcDLw6SSPnmzo7hT+BI71S9KCMePJ4qo6Lsl64OIk23bN64Fjqmpj7iyWJG1G\nZr18tKpOBE7shoOoqttHUpUkaWSGuaHMAJCkBWyYKSYkSQuYQSBJjZszCJL8ZpJ3JTmpW941yUH9\nlyZJGoVhegT/AvwKeHq3fAPw3t4qkiSN1DBB8Jiq+nvgLoCquoPBLKSSpAVgmCC4M8k2DGYMpXs2\nwZyzj0qSHhyGuXz03cAXgZ2TnA7sC7yqz6IkSaMzzPMIvpxkNbAPgyGht1TVLb1XJkkaidmeR7DH\nBk2TTxZbkmRJVa3uryxJ0qjM1iP44CyfFfCcTVyLJGkMZpt07oBRFiJJGo85zxEk2Rp4A/BMBj2B\nrwEnVtX/9lybJGkEhrlq6DTgduAfu+U/Aj4OvKyvoiRJozNMEDypqnabsnxhku/0VZAkabSGuaFs\ndZJ9JheS/C6wsr+SJEmjNEyPYE/gP5L8qFteAlydZA1QVfWU3qqTJPVumCA4sPcqJEljM8ydxdcn\n2R7Yeer63lAmSQvDMJePHs1gbqEf0E08hzeUSdKCMczQ0KEMpqK+s+9iJEmjN8xVQ2uBR/RdiCRp\nPIbpEbwf+HaStUx5DkFV/UFvVUmSRmaYIPgYcCywBri333IkSaM2TBDcUVX/0HslkqSxGCYIvpbk\n/cDnuO/QkJePStICMEwQPK173WdK25yXjyY5BTgIuLmqntS17QCcBUwA1wGHVtXPHljJkqRNac6r\nhqrqgGl+hrmH4FTuf1fykcAFVbUrcEG3LEkao2F6BCR5EfBEYOvJtqr629m2qaqLk0xs0PxiYP/u\n/ceAi4C/HKpSSVIv5uwRJDkReDlwBIOH178MePQ8j7djVU0++/inwI7z3I8kaRMZ5oayZ1TVK4Gf\nVdV7gKcDj9vYA1dV8espK+4nyfIkK5OsXLdu3cYeTpI0g2GC4Jfd6x1JHgncBew0z+PdlGQngO71\n5plWrKoVVbW0qpYuXrx4noeTJM1lmCD4QpJHAB8AVjO42ueMeR7vc8Bh3fvDgM/Ocz+SpE1kmGmo\nj+7enp3kC8DWVXXbXNslOZPBieFFSX4CvBs4BvhkksOB6xlMaCdJGqMZgyDJXsCPq+qn3fIrgYOB\n65McVVW3zrbjqlo2w0fPnW+xkqRNb7ahoY8CdwIk2Y/BX/OnAbcBK/ovTZI0CrMNDW0x5a/+lwMr\nqupsBkNEl/VfmiRpFGbrEWyRZDIongt8ZcpnQ92IJkna/M32hX4m8NUktzC4hPRrAEkey2B4SJK0\nAMwYBFX1viQXMLhn4EvdDWAw6EUcMYriJEn9m3WIp6oumabtmv7KkSSN2jA3lEmSFjCDQJIaZxBI\nUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1\nziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuO2\nHMdBk1wH3A7cA9xdVUvHUYckaUxB0Dmgqm4Z4/ElSTg0JEnNG1cQFPClJKuSLJ9uhSTLk6xMsnLd\nunUjLk+S2jGuIHhmVe0BvAB4Y5L9NlyhqlZU1dKqWrp48eLRVyhJjRhLEFTVDd3rzcC5wN7jqEOS\nNIYgSPKwJNtNvgd+D1g76jokSQPjuGpoR+DcJJPHP6OqvjiGOiRJjCEIqupa4KmjPq4kaXpePipJ\njTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4\ng0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMI\nJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklq3FiCIMmBSa5O8v0kR46jBknSwMiDIMkWwD8B\nLwB2A5Yl2W3UdUiSBsbRI9gb+H5VXVtVdwKfAF48hjokSYwnCB4F/HjK8k+6NknSGGw57gJmkmQ5\nsLxbXJ/k6nHWs4AsAm4ZdxGbgxw77go0A39Hp9jI39NHD7PSOILgBmDnKcu/1bXdR1WtAFaMqqhW\nJFlZVUvHXYc0E39HR28cQ0PfAnZNskuShwJ/CHxuDHVIkhhDj6Cq7k7yJuDfgS2AU6rqylHXIUka\nGMs5gqo6Hzh/HMeWw23a7Pk7OmKpqnHXIEkaI6eYkKTGGQQNSnJREq/KUK+S/HPfswYkOT/JI6Zp\nPyrJO/o89kKy2d5HoI2TJAyG/u4ddy1qU1X92QiO8cK+j9ECewQLSJKJbjK/04C1wCuS/GeS1Uk+\nlWTbabZZP+X9IUlOHWHJWiCSPCzJeUkuT7I2ycun9jyTHJ7kmiSXJjkpyYe79lOTnJDkkiTXJtk/\nySlJrpr6u5hkWZI13b6PndJ+XZJF3fu/6o7xdeDxo/0XeHAzCBaeXYGPAM8GDgeeV1V7ACuBPx9n\nYVrQDgT+q6qeWlVPAr44+UGSRwLvAvYB9gWesMG22wNPB97G4J6i44EnAk9Osnu3/bHAc4Ddgb2S\nvGTqDpLsyeCepN2BFwJ7bfL/wgXMIFh4rq+qSxj8T7cb8I0klwGHMeTt5tI8rAGen+TYJM+qqtum\nfLY38NWqurWq7gI+tcG2n6/B5YtrgJuqak03pHklMMHgS/2iqlpXVXcDpwP7bbCPZwHnVtUdVfVz\nvEn1AfEcwcLzi+41wJeratkc60+9fnjrfkrSQldV1yTZg8Ff4+9NcsED2PxX3eu9U95PLm8J3LVp\nqtRM7BEsXJcA+yZ5LPz/GO7jplnvpiS/k+QhwEtHWqEWjG745o6q+lfgA8AeUz7+FvDsJNsn2RI4\n+AHu/tJu+0Xd80yWAV/dYJ2LgZck2SbJdsDvz+s/pFH2CBaoqlqX5FXAmUl+o2v+a+CaDVY9EvgC\nsI7BeYT7nVCWhvBk4ANJ7mXwF/zrgeMAquqGJH/H4Av9VuC7wG0z7WhDVXVj9yTDCxn0dM+rqs9u\nsM7qJGcBlwM3MwgfDck7iyX1Lsm2VbW+6xGcy2COsXPHXZcGHBqSNApHdRctrAV+CHxmzPVoCnsE\nktQ4ewSS1DiDQJIaZxBIUuMMAqnTzXPzjCnLr0vyynnu61XdtfWTy73PxCnNlyeLpU6So4D1VXXc\nJtjXRcA7qmrlxu5L6ps9Ai14ST6TZFWSK5Ms79oO7GZlvTzJBUkmgNcBb0tyWZJnTc5pn+QJSS6d\nsr+JJGu693+T5FvdrJgrMnAIsBQ4vdvXNhvMxDnTTJrrk7yvq+mSJDuO7l9JLTMI1IJXV9WeDL6c\n39x9wZ4EHFxVTwVeVlXXAScCx1fV7lX1tcmNq+q7wEOT7NI1vRw4q3v/4araq5txcxvgoKr6NIO7\ntP+429cvJ/c1x0yaDwMu6Wq6GHjNpv+nkO7PIFAL3pzkcgbzL+0MLAcurqofAlTVrUPs45MMAgDu\nGwQHJPlm10N4DoPpk2cz20yadzKY7gNgFYOZN6XeGQRa0JLsDzwPeHr3l/a3gcvmsauzgEO7ifuq\nqr6XZGsGz344pKqezKCXsTEzuN5Vvz5pdw/OBaYRMQi00D0c+FlV3ZHkCQye07A1sN/kUE+SHbp1\nbwe2m24nVfUDBl/O7+LXvYHJL/1buqe/HTJlk5n2NcxMmtJI+ReHFrovAq9LchVwNYPhoXUMhofO\n6abfvhl4PvB54NNJXgwcMc2+zmIwxfIuAFX1P0lOYjB/zk+574yXpwInJvklg6dv0W0z50ya0qh5\n+agkNc6hIUlqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLj/g+27IpfuBANXgAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc2704bb860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plot_histogram(result=search_result,\n",
    "                         dimension_name='activation')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also make a landscape-plot of the estimated fitness values for two dimensions of the search-space, here taken to be `learning_rate` and `num_dense_layers`.\n",
    "\n",
    "The Bayesian optimizer works by building a surrogate model of the search-space and then searching this model instead of the real search-space, because it is much faster. The plot shows the last surrogate model built by the Bayesian optimizer where yellow regions are better and blue regions are worse. The black dots show where the optimizer has sampled the search-space and the red star shows the best parameters found.\n",
    "\n",
    "Several things should be noted here. Firstly, this surrogate model of the search-space may not be accurate. It is built from only 40 samples of calls to the `fitness()` function for training a neural network with a given choice of hyper-parameters. The modelled fitness landscape may differ significantly from its true values especially in regions of the search-space with few samples. Secondly, the plot may change each time the hyper-parameter optimization is run because of random noise in the training process of the neural network. Thirdly, this plot shows the effect of changing these two parameters `num_dense_layers` and `learning_rate` when averaged over all other dimensions in the search-space, this is also called a Partial Dependence plot and is a way of visualizing high-dimensional spaces in only 2-dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEPCAYAAABGP2P1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucHGWd7/HPd+4zCSRLQiQJxHBJUAkXIQIBTCKGgJIT\ndNWzeHC9rhwVFNfLKrLHXV09rrruusgLEVdfGLyAgrIxB+UmJCABmUACIUCIAQJJcEggCXO/9O/8\n0T0wPZmZTM10TfdMvu/Xq1/d9dRTVb+uqelfP/VU16OIwMzMrFtZsQMwM7PS4sRgZmZ5nBjMzCyP\nE4OZmeVxYjAzszxODGZmlseJwczM8jgxmJlZHicGMzPL48RgZmZ5KoodwFBMnjw5Zs6cWewwzMxG\nlTVr1uyIiIP3VW9UJoaZM2dSX19f7DDMzEYVSc8Mpp5PJZmZWR4nBjMzy+PEYGZmeZwYzMwsjxOD\nmZnlST0xSHpa0iOS1kra61IiZV0uaZOkhyWdmHZMZmbWv5G6XPUtEbGjn3lvA2blHqcA3889m5lZ\nEZTCqaTzgGWRdR8wUdLUYgdlNlotX76ciy++mOXLl5fEeoYraRyFjrtU9sNll13Gsccey2WXXZb6\nthQR6W5Aegp4CQjgBxFxda/5K4B/jYh7ctN3AF+IiH5/wXagDopTKxanGHXxldXWFnX7qq0Z/kqq\nqwdft2bgulFT2f+86r7nZar3bhB3VZf3ms7/bpTpMd1VpR71eryuYq86mVxZV49QussyVZCp7K6f\n/X+L3PTmSz7TZ+z9+dADHxpw/pZVW1j5jyvpau2ioqacc75xGkcsmJ5oGwCbV27l95feS2drF5U1\n5bzzm3OZ/ZapTKhozqs3sTx/ekKvaYCJZb2XaelzmxPL2vcqu+PWVv7+4pdobYHaWvjBlQdxzuK9\nj80DlT1+VtzSxPs/8QLNLUFdrVh25cEsOXvcwG+2hz3R9srrXZnI235Nrbj0P6cxb9EB7MrUsbur\nLluvq47dnXXs6axhT2c2tsaOGho7q2jqqKKlvZLWjkraOirobC8n01EOjeWUdZShDihvF2UdUNae\nfVQ2QXl7UN4O5W1BeXvwzMM38+zmO1+J7Utf+hJf//rXB/2+uklaExFz91VvJE4lnRERWyVNAW6T\n9HhErEq6EkkXAhcC1FBX6BjNUlXW8Wpy6Gnmlf9GVAWqymTrVXZRUdVFdWUnADWVHdRWdTCusp3x\nFe2M7z8/ArDt/m10tXYB0NnaxZb7tg8pMWy5bzudufV0tHaxeXUDs98y8g35e1a10ZrLIy0tcNfK\n1j4TQ7fbV7XQ3JJNvs0twe2rWhIlhoG239oSrLm7iXmLDhhwmcaOgb9URXsZ6mdeeUff5S/+ZUPe\n9PLly4eUGAYr9VNJEbE199wA/AY4uVeVrcBhPaYPzZX1Xs/VETE3IuZWkuCbqNl+ZNop0yivybaK\nKmrKmXHq0D7MZ5w6lYrceipryjli3pSCxZjEGfOrqck1nmtrYeGCgT90F82vpa42+7FbVysWzR9e\ny7vn9mtqxUlvHnqSGY7Jk1+fN7106dJUt5dqi0HSOKAsIl7OvV4MfLVXteXAxZKuI9vpvDsitqcZ\nl9lYNWP+DBZ8bQE7HtjCjFOnDqm1AHDEgumc843TeP7+5zhi3pSitBYA3rq4hv+44q944O42Fi6o\nGbC1ALDk7HEsuzLbclg0v3ZYrYWe2//Dqg5OeXMtx505cGshLUcceTYABxywg6VLl6baWoD0TyW9\nBviNpO5t/Twifi/pYwARcRVwM/B2YBPQDAx8EtXMBjRj/gzmnDn8b/hHLJjOCW+dVICIhueti2t4\n1zmD/+a/5Oxxw04Ivbd/0qIDAdjVVbDVJnbEkWfzhzu+OCLbSjUxRMRm4Pg+yq/q8TqAi9KMw8zM\nBq8ULlc1M7MS4sRgZmZ5nBjMzCyPE4OZmeVxYjAzszxODGZmlseJwczM8jgxmJlZHicGMzPL48Rg\nZmZ5nBjMzCyPE4OZmeVxYjAzszxODGZmlseJwczM8oxIYpBULukhSSv6mPdBSS9IWpt7/N1IxGRm\nZn1LewS3bpcAjwEH9jP/+oi4eIRiMTOzAaTeYpB0KHAu8F9pb8vMzIZvJE4lfRf4ByAzQJ13SXpY\n0g2SDhuBmMzMrB+pJgZJS4CGiFgzQLXfAjMj4jjgNuAn/azrQkn1kuo7aEshWjMzg/RbDKcDSyU9\nDVwHnCnppz0rRMTOiOj+pP8v4KS+VhQRV0fE3IiYW0l1mjGbme3XUk0MEXFpRBwaETOB84E/RMT7\netaRNLXH5FKyndRmZlYkI3VVUh5JXwXqI2I58ClJS4FO4EXgg8WIyczMskYsMUTEXcBduddf7lF+\nKXDpSMVhZmYD8y+fzcwsjxODmZnlcWIwM7M8TgxmZpbHicHMzPI4MZiZWR4nBjMzy+PEYGZmeZwY\nzMwsjxODmZnlcWIwM7M8TgxmZpbHicHMzPI4MZiZWR4nBjMzy+PEYGZmeUYkMUgql/SQpBV9zKuW\ndL2kTZLulzRzJGIys/1EBGW/boSIYkcyaoxUi+ES+h/L+SPASxFxFPAfwDdHKCYz2w/o4XYqL34B\nPdJe7FBGjdSH9pR0KHAu8HXgM31UOQ/459zrG4ArJCki/fTekNnKznieSTqEKWXT095cYg2dW9jR\nuY3JFdOYUjGj2OFYATWtX8/Lf1xNlMH4haegiqBl/ZPUnXAkE+bNzqu7895NPPPQZg49ZRqzzzwk\n0XY2r9zKlvu2M+PUqRyxoPSO8UJacUsTt69qYdH8WpacPQ5e6IIItnz7JY4Etnz7JQ77zmSQ4ODy\nYodb0kZizOfvAv8AHNDP/OnAswAR0SlpNzAJ2JFmUA2ZrTySWU2GLrbFUxzLvJJKDg2dW1jXspIM\nXWzteJLjaxc4OYwRTY+up+HaZdDVBUDrYxuhTNDZRdPd9ZRXvIfq044CYNd9T7L5myvItHWy9eYN\n1FTMZ8b8wR0Hm1du5feX3ktnaxcb/vspzvnGaWM2Oay4pYn3f+IFmluCZdc18t9f6GDRP79ECKYF\nCJh2RwtVb3wWBbTfMo04trrYYZesVE8lSVoCNETEmgKs60JJ9ZLqO2gbdmw743kyZP8xM3SxM54f\n9joLaUfntrz4dnRuK3JEVigtGze+khQAyGSgMzsd7R00r9v0yqzdDz5Fpq0TgK62TrbdP/jjYMt9\n2+lsza63s7WLLfdtL0D0pen2VS00t2RPMjS3BDc+00nHT15Da8Wr334rgNYK6Fj2GieFfUi7j+F0\nYKmkp4HrgDMl/bRXna3AYQCSKoAJwM7eK4qIqyNibkTMrWT4f9RJOoQyss3JMsqZpGRN9LRNrpiW\nF9/kimlFjsgKpXb2bCjvcSqjrAwqstOqqqTu+KNemTXhxMMpq85+tJVXVzDtlMEfBzNOnUpFTXa9\nFTXlzDh1agGiL02L5tdSVysA6mrFovm1ZM6q45nFdQjoJNtqeGZxHZlFdcUMdVRI9VRSRFwKXAog\naSHwuYh4X69qy4EPAKuBdwN/GIn+hSll0zmWeSXbxzClYgbH1y5wH8MYNO6YOUz50Pv77WMY/6bZ\nZD/KYOKps3jdZUtozPUxzJg/+C8wRyyYzjnfOG2/6GNYcvY4ll1Jfh8DcPTaNhA8NKOCN27p5Oh1\n7bgLet9Goo9hL5K+CtRHxHLgR8C1kjYBLwLnj1QcU8qmM4XS/WeZUjHDCWGMGjdnDuPmzCFTmSGq\nAlVlqDvxDZRVdgFdeXUnnXYUMxbMYHxFO9CaaDtHLJg+phNCT0vOHvdKQgCgK8gcUUnXD6ZwzIk1\ndKxppfxbL0FXQLmKF+goMOjEIOlI4LmIaMt9+z8OWBYRuwazfETcBdyVe/3lHuWtwHsGH7KZ2SCU\ni87rXz19FifV5E1b/5L0MdwIdEk6CriabL/Az1OJyszMiiZJYshERCfwTuB7EfF5wOnXzGyMSZIY\nOiS9l2xHcfetLSoLH5KZmRVTksTwIWAe8PWIeErS4cC16YRlZmbFMqjOZ0nlwGURcUF3WUQ8he9r\nZGY25gyqxRARXcBrJVWlHI+ZmRVZkt8xbAb+KGk50NRdGBH/XvCozMysaJIkhj/nHmX0f0M8MzMb\n5QadGCLiKwCS6iKiOb2QzMysmAZ9VZKkeZI2AI/npo+XdGVqkZmZWVEkuVz1u8DZ5O58GhHrgPlp\nBGVmZsWT6LbbEfFsr6KuPiuamdmolaTz+VlJpwEhqZKBx3E2M7NRKkmL4WPARWSH4twKnJCbNjOz\nMSRJiyF6/vLZzMzGpiQthvsk/UrS2yQNapQLSTWS/iRpnaRHJX2ljzoflPSCpLW5x98liMnMzAos\nSYthNrAI+DDwPUm/BK6JiI0DLNMGnBkRjbl+iXsk/S4i7utV7/qIuDhR5GZmlopBtxgi67aIeC/w\nUbK33/6TpJWS5g2wTGNusjL3SH08ZzMzG7okP3CbJOkSSfXA54BPApOBzzLASG6SyiWtBRqA2yLi\n/j6qvUvSw5JukHRYsrdgZmaFlKSPYTVwIPCOiDg3In4dEZ0RUQ9c1d9CEdEVEScAhwInS5rTq8pv\ngZkRcRxwG/CTvtYj6UJJ9ZLqO2hLELaZmSWRpI/h6Ijo8zRQROxzXIaI2CXpTuAcYH2P8p09qv0X\n8K1+lr+a7FjTHKiDfDrKzCwlSRLDZEn/ABwD1HQXRsSZ/S0g6WCgI5cUaoGz6DW4j6SpEbE9N7kU\n/2jOzKyokpxK+hnZG+gdDnwFeBp4YB/LTAXulPRwru5tEbFC0lclLc3V+VTuUtZ1wKeADyaIyczM\nCixJi2FSRPxI0iURsRJYKWnAxBARDwNv7KP8yz1eXwpcmiAOMzNLUZLE0JF73i7pXGAbcFDhQzIz\ns2JKkhi+JmkC2ctTv0f2CqW/TyUqMzMrmiQjuK3IvdwNvCWdcMzMrNj2mRgkfY8Bfq0cEZ8qaERm\nZlZUg2kx1KcehZmZlYx9JoaI6POXyL1J+l5EfHL4IZmZWTElGtpzH04v4LrMzKxICpkYzMxsDHBi\nMDOzPIVMDIMa1c3MzEpb4sQgqa6fWf85zFjMzKwEJBmo5zRJG8jeSA9Jx0u6snt+RFxT+PDMzGyk\nJWkx/AdwNrATICLWAfPTCMrMzIon0amkiHi2V1FXAWMxM7MSkOQmes9KOg0ISZXAJXhQHTOzMSdJ\ni+FjwEXAdGArcEJuul+SaiT9SdK63GA8X+mjTrWk6yVtknS/pJkJYjIzswJLcnfVHcAFCdffBpwZ\nEY25VsY9kn4XEff1qPMR4KWIOErS+WSH/vybhNsxM7MCSXJV0rckHSipUtIdkl6Q9L6Blomsxtxk\nZe7R+06t5wHd92O6AXirJP8mYj/U0PgkGxpuo6HxyWKHMqKa1z7KzmX/TfODG4odSqpW3/4yV/zT\n86y6rbnYoQxb/e27+PlX/8zaO3YOaz1Nj66nYfmNND6+vkCRFUaSU0mLI2IPsITseM9HAZ/f10KS\nyiWtBRrIjvl8f68q04FnASKik+x4D5MSxGVjQEPjk6x9/rds2f0Qa5//7X6THJoffpSdP/w5jXes\nZsf3f0HTmrHZbbf69pf5v5dsY/m1u/jHT+4Y1clh9e0v891PP8WdP9vODz/zBI/+oWFI62lav56G\nn/2U3ff/kW03XsvLG0snOSRJDN2nnc4FfhURuwezUER0RcQJwKHAyZLmJIwRAEkXSqqXVN9B21BW\nYSVsR/PTZKITgEx0sqP56eIGNEJaHttItGdHzY32DloeGZsJcc3dTbS1ZE8WtLYE99/dUuSIhm7N\n3U20t2bfS3trho33Dq3V0PL4RqIj97fv6KDpqScKFuNwJUkMKyQ9DpwE3CHpYKB1sAtHxC7gTuCc\nXrO2AocBSKoAJpD7rUSv5a+OiLkRMbeS6gRh22gwuW4mZcp+9yhTBZPrZhY3oBFS+/rZqKoSAFVV\nUnvsrCJHlI6T3jyO6trsGeKaWnHKm2uLHNHQnfTmcVTVZN9LVU0Zs08b2gmO2tfNRpW5v31lJeMO\nP7pgMQ5Xks7nL0r6FrA7IrokNZHtH+hXLnl0RMQuSbXAWWQ7l3taDnwAWA28G/hDRPQ7YpyNTVPG\nz+KEQ/4HO5qfZnLdTKaMH5sfkL3VHXcMkz76v2h9fCO1c2Yx7qSjGYs/D5q36AC+9J/TWHN3E/MX\nVDL/rP7urFP65i06gE9/93AeuLuFN5w+kZkLp7C7M/l6xs2Zgy54H62PbWT84UdzwBFzoKnw8Q5F\nkt8xALwOmJn7Zt9t2QD1pwI/kVROtnXyy4hYIemrQH1ELAd+BFwraRPwInB+wphsjJgyftZ+kxB6\nqjvhGMad/Prc1NhLCt3mLTqAeYsOYGL56D2N1G3uoonMess0AHYN40827pg5HDjrWMo6gPbCxFYI\ng04Mkq4FjgTW8urRGwyQGCLiYeCNfZR/ucfrVuA9g43DzMzSlaTFMBd4g0/zmJmNbUk6n9cDh6QV\niJmZlYYkLYbJwAZJf4JXrxeNiKUFj8rMzIomSWL457SCMDOz0pHkctWVkl4LzIqI23MjuZWnF5qZ\nmRVDknslfZTsvYx+kCuaDtyURlBmZlY8STqfLwJOB/YARMSTwJQ0gjIzs+JJkhjaIuKVn2DkfuTm\nS1fNzMaYJIlhpaQvAbWSzgJ+Bfw2nbDMzKxYkiSGLwIvAI8A/xu4GfjHNIIyM7PiSXJVUgb4Ye5h\nZmZj1D4Tg6RHGKAvISKOK2hEZmZWVINpMSzJPV+Ue7429/w+3PlsZjbm7DMxRMQzAJLOioied0r9\ngqQHyfY9mJnZGJGk81mSTu8xcVrC5c3MbBRIcq+kjwA/ljQhN70L+HDhQzIzs2Ia9Df+iFgTEccD\nxwPHR8QJEfFg93xJH+i9jKTDJN0paYOkRyVd0kedhZJ2S1qbe3y5dx0zMxs5SYf2JCJ29zPrEuAn\nvco6gc9GxIOSDgDWSLotIjb0qnd3RCzBzMyKrpB9BOpdEBHbu1sVEfEy8BjZm++ZmVmJKmRiGPDS\nVUkzyY7/fH8fs+dJWifpd5KOKWBMZmaWUOJTSQPYq8XwygxpPHAj8OmI2NNr9oPAayOiUdLbyd7K\ne1Yf67gQuBCghrqCBW1mZvkK2WL4Y1+FkirJJoWfRcSve8+PiD0R0Zh7fTNQKWlyH/Wujoi5ETG3\nkuoChm1mZj0NusUgaSLwfmBmz+Ui4lO554v7WEbAj4DHIuLf+1nvIcBfIiIknUw2We1M8B7MzKyA\nkpxKuhm4j+zdVTODXOZ04G+BRyStzZV9CZgBEBFXAe8GPi6pE2gBzo8I32rDzKxIkiSGmoj4TJKV\nR8Q9DND3kKtzBXBFkvWamVl6kvQxXCvpo5KmSjqo+5FaZGZmVhRJWgztwLeBy3j10tQAjih0UGZm\nVjxJEsNngaMiYkdawZiZWfElOZW0CWhOKxAzMysNSVoMTcBaSXcCbd2F3ZermpnZ2JAkMdyUe5iZ\n2Rg26MQQEb3vnGpmZmNQkl8+P0UfN8qLCF+VZGY2hiQ5lTS3x+sa4D2Af8dgZjbGJBnBbWePx9aI\n+C5wboqxmZlZESQ5lXRij8kysi2IQt6228zMSkCSD/bv8GofQyfwNNnTSWZmNoYkSQxvA95F/m23\nzwe+WuCYzMysiJL+jmEX2RHXWtMJx8zMii1JYjg0Is5JLRIzMysJSe6VdK+kY5OsXNJhku6UtEHS\no5Iu6aOOJF0uaZOkh3t1cpuZ2QhL0mI4A/hg7odubWQH4ImIOG6AZTqBz0bEg5IOANZIui0iNvSo\n8zZgVu5xCvD93LOZmRVB0s7nRCJiO7A99/plSY8B04GeieE8YFluOM/7JE2UNDW3bJ9aaaYhs5Up\nZdOThjSqNXRuYUfnNiZXTGNKxYxUt7Nzz1+YXH0YU6pnpradkfTCzsfYuWsTf3XwbCYf/IbUt7dn\n03oan32CcYcfzbg5cwqyzm13P82La55m2inTmDE/vb9/XzbeuZ3NqxuYc8YEjjlzyohs845bW7ln\nVRtnzK/mXefUJl5+xS1N3L6qhUXza1ly9rhBz0tb89pHaX3kSepmH834owtzbBRaknslPTOcDUma\nCbwRuL/XrOnAsz2mn8uV9ZsYOmjjkcxqjmXefpMcGjq3sK5lJRm62NrxJMfXLkglOfTcznOtT3DC\nhEWjPjm8sPMxHtn4SzKZDrY1PMgxx5yfanLYs2k9z624lujsYNe6PzGt4m8Z/7rhfQA0/HEz675y\nO11tnTz52ydZ8LUFI5YcNt65nd98oZ6O1i7W3VTGBf92XOrJ4Y5bW/n7i1+itQVu/GUz464U5yyu\nGfTyK25p4v2feIHmlmDZdY0su5JXEsBA89LWtOYxdv7weqK9g6bVD6AL3seBsxKdoR8RSfoYhkzS\neOBG4NMRsWeI67hQUr2keoAMXeyM5wsZZknb0bmNDF1A9r3v6Nw2AtvpZEfbs/tYovTt3LWJTKYD\ngEymgxdffDLV7TU+/QTRmd1edHbQ9Ocnhr3OHQ9soautE4Cu1i623Z/O378vm1c30NGaPSY6WjNs\nvHdn6tu8Z1UbrS3Z160tcNfKZBdC3r6qheaW7M+umluC21e1DGpe2loeeZJozx0bHR20bNw4YttO\nIvXEIKmSbFL4WUT8uo8qW4HDekwfmivLExFXR8TciJgLUEY5k3RIGiGXpMkV0yijHMi+98kV00Zg\nOxVMrj5sH0uUvkkTj6KsrBKAsrJKDjpoVqrbGz/zaFSR3Z4qKhl35NHDXufkN82gvDrbwC+vKWfa\nKen8/ftyxLwpVNZkj4nKmjJmnzYp9W2eMb+amtzZo5paWLhg8K0FgEXza6mrFQB1tWLR/NpBzUtb\n7bGzUFXu2KispHb27BHbdhLKntpPaeWSgJ8AL0bEp/upcy5wMfB2sp3Ol0fEyQOtt0o18YayN43p\n00hltXsfrCPaxxAF6GOorh583ZqB60ZNZf/zqvuel6l+9Uxpf30MXdVlvZZ5dbqrSj3q9XhdxV51\nMrmyrlwoffUxZCqhqyr7/xa5epnKDFEVqCoDQFllFxVVXVRXZlsHNZUd1FZ1MK6ynT2rNw66j+HA\nisL81Kh7Pf31MUwszx/UcUL53oM8TizLL5tY3vc39Ill7XnTSfoYDtTex89Q+hj2xCtjkLErEz1e\nZ//Au7pqc9N17O6qy5XVsbuzjj2dNezprKGxI5vEGjuraOqooqW9ktaOSto6KuhsL+flP27M62Mo\nbxdlHVDWDpVN2efy9qC8HcrbgvL2oHJP9nj4wx1fHHA/7IukNd1frgesl3JiOAO4G3gEyOSKvwTM\nAIiIq3LJ4wrgHLJDh34oIuoHWu+BOihOrVicWtyloK/EMJJUm+wbWp9KKDF066ou7zWdTmLoWfbK\n8zATw/iKdsZXDu4Dv9CJoduEit4f8uklhvx56nce9J0YhmIkEkNXUyVqF2UdZaiDkkwMqd4ELyLu\nIXtZ60B1ArgozTjMzGzwRqTz2czMRg8nBjMzy+PEYGZmeZwYzMwsjxODmZnlcWIwM7M8TgxmZpbH\nicHMzPI4MZiZWR4nBjMzy+PEYGZmeZwYzMwsjxODmZnlcWIwM7M8TgxmZpYn1cQg6ceSGiSt72f+\nQkm7Ja3NPb6cZjxmZrZvqQ7UA1xDdnS2ZQPUuTsilqQch5mZDVKqLYaIWAW8mOY2zMyssEqhj2Ge\npHWSfifpmGIHY2a2v0v7VNK+PAi8NiIaJb0duAmY1VdFSRcCFwLUUDdyEZpZv3Z11TGxvLnYYYyo\nXZmx//lT1BZDROyJiMbc65uBSkmT+6l7dUTMjYi5lVSPaJxmZvuToiYGSYdIUu71ybl4dhYzJjOz\n/V2qp5Ik/QJYCEyW9BzwT0AlQERcBbwb+LikTqAFOD8iIs2YzMxsYKkmhoh47z7mX0H2clYzMysR\npXBVkpmZlRAnBjMzy+PEYGZmeZwYzMwsjxODmZnlcWIwM7M8TgxmZpbHicHMzPI4MZiZWR4nBjMz\ny+PEYGZmeZwYzMwsjxODmZnlcWIwM7M8TgxmZpbHicHMzPKkmhgk/VhSg6T1/cyXpMslbZL0sKQT\n04zHzMz2Le0WwzXAOQPMfxswK/e4EPh+yvGYmdk+pD205ypJMweoch6wLDfO832SJkqaGhHb04xr\nOBoyW9kZzzNJhzClbHqxw0msoXMLOzq3MbliGlMqZhQ7nJKx8y8beGnHRiZMPZpJU48pdjh5dt67\niWce2syhp0xj9pmHjOi2N965nc2rGzhi3hRmv2XqiG67p9/f2spdK1tZuKAG4JXX5yyuSbSeFbc0\ncfuqFhbNr2XJ2ePSCHVMSDUxDMJ04Nke08/lykoyMTRktvJIZjUZutgWT3Es80ZVcmjo3MK6lpVk\n6GJrx5McX7vAyYFsUnhs7c/JZDp4fms9R8+9oGSSw677nmTzN1eQaetk680bqKmYz4z5I/M327xy\nK7dcWk9HaxfrbtrCO785lzedNWFEtt3THbe28pmLX6KlBX7682YA2tvhF9c384MrDxp0clhxSxPv\n/8QLNLcEy65rZNmVDCk5TCxrZlemLvFyo0pEpPoAZgLr+5m3Ajijx/QdwNx+6l4I1AP1M2bMiGK4\n6KKLAnjlcdFFFxUljqEa7fGnpZT3SzFjK5X90juOocZUKu+nmID6GMTndrGvStoKHNZj+tBc2V4i\n4uqImBsRcw8++OARCa63xYsXU1eX/aZQV1fH4sWLixLHUI32+NNSyvulmLGVyn7pGUdVVRXV1dVD\niqlU3s9ooGwSSXED2T6GFRExp4955wIXA28HTgEuj4iT97XOuXPnRn19fYEjHZzly5dz6623snjx\nYpYuXVqUGIZjtMefllLeL8WMrVT2S884gCHHVCrvp1gkrYmIufusl2ZikPQLYCEwGfgL8E9AJUBE\nXCVJwBVkr1xqBj4UEfv8xC9mYjAzG60GmxjSvirpvfuYH8BFacZgZmbJFLuPwczMSowTg5mZ5XFi\nMDOzPE4MZmaWx4nBzMzyODGYmVme1H/glgZJLwNPpLiJCcDuFJfbV73+5vdVPpiy3tOTgR2DiHOo\nirn/ks4tEcpZAAAGNklEQVRLuv/G8r4baP5gy/fnY2+g+aXyv/vaiNj3rSMGc9+MUnswyPt9DGP9\nV6e53L7q9Te/r/LBlPUxPWb3X9J5SfffWN53A80fbPn+fOwl3X+l+L/b/fCppL79NuXl9lWvv/l9\nlQ+mbKjvZ6iKuf+Sziu1/Teajr2+yvfnY2+g+aPh2HvFaD2VVB+D+Fm39c37b+i874bH+294Rmr/\njdYWw9XFDmCU8/4bOu+74fH+G54R2X+jssVgZmbpGa0tBjMzS4kTg5mZ5XFiMDOzPGMuMUgqk/R1\nSd+T9IFixzPaSFoo6W5JV0laWOx4RhtJ4yTVS1pS7FhGG0mvzx13N0j6eLHjGW0kvUPSDyVdL2lY\n45aWVGKQ9GNJDZLW9yo/R9ITkjZJ+uI+VnMe2bGjO4Dn0oq1FBVo/wXQCNSwH+2/Au07gC8Av0wn\nytJViP0XEY9FxMeA/wmcnma8paZA+++miPgo8DHgb4YVTyldlSRpPtkPpWWRGyNaUjmwETiL7AfV\nA8B7gXLgG71W8eHc46WI+IGkGyLi3SMVf7EVaP/tiIiMpNcA/x4RF4xU/MVUoH13PDCJbFLdEREr\nRib64ivE/ouIBklLgY8D10bEz0cq/mIr1P7LLfcd4GcR8eBQ40l1aM+kImKVpJm9ik8GNkXEZgBJ\n1wHnRcQ3gL2a65KeA9pzk13pRVt6CrH/engJqE4jzlJUoGNvITAOeAPQIunmiMikGXepKNSxFxHL\ngeWS/h+w3ySGAh1/Av4V+N1wkgKUWGLox3Tg2R7TzwGnDFD/18D3JL0ZWJVmYKNEov0n6a+Bs4GJ\nwBXphlbyEu27iLgMQNIHybW8Uo2u9CU99hYCf032C8nNqUY2OiT97PsksAiYIOmoiLhqqBseDYkh\nkYhoBj5S7DhGq4j4NdnkakMUEdcUO4bRKCLuAu4qchijVkRcDlxeiHWVVOdzP7YCh/WYPjRXZoPj\n/Td03nfD4/03PEXbf6MhMTwAzJJ0uKQq4HxgeZFjGk28/4bO+254vP+Gp2j7r6QSg6RfAKuBoyU9\nJ+kjEdEJXAzcAjwG/DIiHi1mnKXK+2/ovO+Gx/tveEpt/5XU5apmZlZ8JdViMDOz4nNiMDOzPE4M\nZmaWx4nBzMzyODGYmVkeJwYzM8vjxGBmZnmcGGxMkdQ4AttYOsixGdLY9jskvaEY27b9h3/gZmOK\npMaIGF+A9ZRHRFFu2z7QtiVdA6yIiBtGNirbn7jFYGOWpM9LekDSw5K+0qP8JklrJD0q6cIe5Y2S\nviNpHTBP0tOSviLpQUmPSHpdrt4HJV2Re32NpMsl3Stps6R358rLJF0p6XFJt0m6uXteP7E+Lemb\nkh4E3iPpo7nY10m6UVKdpNOApcC3Ja2VdGTu8fvc+7m7O0az4XBisDFJ2TFvZ5Ed7OQE4KTcKFmQ\nHe3qJGAu8ClJk3Ll44D7I+L4iLgnV7YjIk4Evg98rp/NTQXOIDt4yr/myv4amEl20J6/BeYNIuyd\nEXFiRFwH/Doi3hQRx5O9T85HIuJesjdR+3xEnBARfwauBj6Zez+fA64cxHbMBjTmxmMwy1mcezyU\nmx5PNlGsIpsM3pkrPyxXvpPsiH839lpP99gUa8h+2PflptygPBtyQ6JCNlH8Klf+vKQ7BxHz9T1e\nz5H0NbIDJo0neyO1PJLGA6cBv8oO3gXsR6PuWXqcGGysEvCNiPhBXmF2lLBFwLyIaJZ0F9kxmgFa\n+zi335Z77qL//5e2Hq/VT53BaOrx+hrgHRGxLjci3MI+6pcBuyLihGFs02wvPpVkY9UtwIdz36qR\nNF3SFGAC8FIuKbwOODWl7f8ReFeur+E19P3BPpADgO2SKoELepS/nJtHROwBnpL0HsiO+Svp+GFH\nbvs9JwYbkyLiVrKDya+W9AhwA9kP1N8DFZIeI9sfcF9KIdxIdozeDcBPgQeB3QmW/z/A/WQTzOM9\nyq8DPi/pIUlHkk0aH8l1mD8KnFeA2G0/58tVzVIiaXxENOY6t/8EnB4Rzxc7LrN9cR+DWXpWSJoI\nVAH/4qRgo4VbDGYjSNJvgMN7FX8hIva66sisWJwYzMwsjzufzcwsjxODmZnlcWIwM7M8TgxmZpbH\nicHMzPL8fy6cRQja3MY6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc270556320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plot_objective_2D(result=search_result,\n",
    "                        dimension_name1='learning_rate',\n",
    "                        dimension_name2='num_dense_layers',\n",
    "                        levels=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We cannot make a landscape plot for the `activation` hyper-parameter because it is a categorical variable that can be one of two strings `relu` or `sigmoid`. How this is encoded depends on the Bayesian optimizer, for example, whether it is using Gaussian Processes or Random Forests. But it cannot currently be plotted using the built-in functions of `skopt`.\n",
    "\n",
    "Instead we only want to use the real- and integer-valued dimensions of the search-space which we identify by their names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dim_names = ['learning_rate', 'num_dense_nodes', 'num_dense_layers']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then make a matrix-plot of all combinations of these dimensions.\n",
    "\n",
    "The diagonal shows the influence of a single dimension on the fitness. This is a so-called Partial Dependence plot for that dimension. It shows how the approximated fitness value changes with different values in that dimension.\n",
    "\n",
    "The plots below the diagonal show the Partial Dependence for two dimensions. This shows how the approximated fitness value changes when we are varying two dimensions simultaneously.\n",
    "\n",
    "These Partial Dependence plots are only approximations of the modelled fitness function - which in turn is only an approximation of the true fitness function in `fitness()`. This may be a bit difficult to understand. For example, the Partial Dependence is calculated by fixing one value for the `learning_rate` and then taking a large number of random samples for the remaining dimensions in the search-space. The estimated fitness for all these points is then averaged. This process is then repeated for other values of the `learning_rate` to show how it affects the fitness on average. A similar procedure is done for the plots that show the Partial Dependence plots for two dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAGECAYAAADKqHwZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl4VPX1+PH3yUJWSIAECGEJAoKIghIsqFhFbKvi1mpr\nf+5at7rWtmprW+32ra3WpdWquG+tinXFXeqCO0FWERVZZIdAEiAh65zfH/dOmISZyWT2TM7ree6T\nO3fu3HuG0Tnz2UVVMcYYY0KRlugAjDHGdB2WNIwxxoTMkoYxxpiQWdIwxhgTMksaxhhjQmZJwxhj\nTMgsaRhjjAmZJQ0/RGRnHO5xvIhcG+v7BLj3iSIyJhH3NsZ0bWKD+/YkIjtVNT8K10lX1ZZoxBTN\ne4vIQ8AsVX06vlEZY7o6K2l0QER+KSJzRWSRiPze5/hzIjJPRD4TkQt8ju8Ukb+LyEJgsoisEpHf\ni8inIrJYREa7550tIne4+w+JyD9E5AMRWSEiJ7vH00TkXyKyTETeEJGXvc8FiHWViPxVRD4FThGR\n893YF4rIf0UkV0QOBo4HbhKRBSIy3N1edd/PHG+MxhjTniWNIETkO8BI4CBgPDBBRA5znz5XVScA\n5cDlItLXPZ4HfKyq41T1PfdYpaoeCNwF/CLA7UqAQ4HpwI3use8DZcAY4Axgcghhb1XVA1X1CeAZ\nVZ2oquOAz4HzVPUD4AXgl6o6XlW/BmYAl7nv5xfAv0K4jzGmG8pIdABJ7jvuNt99nI+TRN7FSRQn\nuccHu8e3Ai3Af9td5xn37zycRODPc6rqAZaKSH/32KHATPf4RhF5K4SYn/TZHysifwIK3dhfa3+y\niOQDBwMzRcR7OCuE+xhjuiFLGsEJ8BdVvafNQZHDgWnAZFWtE5G3gWz36Xo/bQkN7t8WAv+bN/js\nS4BzQlHrs/8QcKKqLhSRs4HD/ZyfBlSr6vgI7mmM6Saseiq414Bz3V/jiEipiPQDCoAqN2GMBibF\n6P7vAz9w2zb64/9LP5iewAYRyQRO8zm+w30OVd0OrBSRUwDEMS7iyI0xKcmSRhCq+jrwb+BDEVkM\nPI3zZfsqkCEin+O0P3wUoxD+C6wFlgKPAZ8CNZ14/W+Bj3GSzzKf408AvxSR+SIyHCehnOc23n8G\nnBCF2I0xKci63CY5EclX1Z1uQ/snwCGqujHRcRljuidr00h+s0SkEOgB/NEShjEmkayk0QWJyLPA\nsHaHr1HVPXpHGWNMNHWYNEQkF/g5MERVzxeRkcAoVZ0VjwA7q6ioSMvKyhIdhgHmzZtXqarFiY7D\nGBM9oVRPPYgzvsA7sGwdMBNIyqRRVlZGRUVFosMwgIisTnQMxpjoCqX31HBV/RvQBKCqdUQ2jsB0\nFevXJzoCY0ySCSVpNIpIDqAAbhfNhuAvSZxN2+tpbPYkOozUUFqa6AiMMUkmlKRxPc64hMEi8jgw\nG7g6plFFYPOOBn7z3GKsgd8YY6KvwzYNVX3DnTV1Ek611BWqWhnzyMLUr2cWT1WsZWjfPC45YkSi\nwzHGmJTSYUnDnZSvWVVfcntMNYvIibEPLTz9e2VzwviB3PTaF7y40OrkjTEmmkKqnlLV1qkrVLUa\np8oqbCLSx10f4iv3b+8g5/YSkbXetSdC8beT92diWW9+PnMh81ZviyRUY4wxPkJJGv7OiXQk+bXA\nbFUdidNGEmzZ0z/iTEUesqyMdGacUc7AgmzOf2Qeq7fWdvwis6frI/ptYIxJQaEkjQoRucVnhbdb\ncMZtROIE4GF3/2HAb3WXiEwA+gOvd/YGvfN68OA5B+FR5ZyH5lJd1xh2sN3WDTckOgJjTJIJJWlc\nBjTiLO7zJE5320sivG9/Vd3g7m/ESQxtiEga8HcCr3TXoWFFecw4o5y123Zx4aPzrCtuZ9k4DWNM\nO6H0nqolePWRXyLyJjDAz1PXtbu+ioi//rE/BV5W1bU+K8oFutcFwAUAQ4YMafPcQcP68LeT9+fK\nJxdw7TOL+Psp4+joesZVWgrWddkY46PDpCEie+P82i/zPV9VpwZ7napOC3LNTSJSoqobRKQE2Ozn\ntMnAFBH5Kc5SpT1EZKeq7pHAVHUGzjrXlJeX7/Etd+IBpazeWsetb35JWd88Lj9yZLDQjTHGBBBK\ng/ZM4G7gPpzlSqPhBeAsnAWMzgKeb3+CqrauNOcuVVruL2GE6vIjR7B6ay23vPElQ/vmcsJ4G+1s\njDGdFUrSaFbVu6J83xuBp0TkPGA18EMAESkHLlLVn0T5fogIf/nBfqyr3sUvZy5iYGEOE8v6RPs2\nxhiT0kKZGv0GnOqjZ/GZc0pVk3IARHl5uQab5ba6rpHv/+sDquoaeeanhzCsKC+O0XUxIhG1aYjI\nPFUtj2JExpgEC6X31FnAL4EPcLrazgO67Nzjhbk9ePCciYgI5z40l6pa64obkI3TMMa002HSUNVh\nfra94hFcrAztm8eMMyawrtrpitvQHK2mmhQTwTiNml1N0YvDGJM0Qpl7KldEfiMiM9zHI0VkeuxD\ni63ysj7cfMo4Plm1jWueXmSz4voTwTiNU2d8FMVAjDHJIpTqqQdxBvcd7D5eB/wpZhHF0fHjBvLL\n747iuQXrue3NrxIdTvIJcz0NVWXttrooB2OMSQbdfuW+nx4+nFMmDOL22V/x33lrEx1OSqiua2JH\nQ3OiwzDGxEAoXW671Mp9nSUi/Pkkpyvutc8sorR3DpP26pvosLq0b6yUYUzKSrmV+8LRIyONu06f\nwNC+eVz46Dy+3rIz0SF1aWuqLGkYk6pC6T31BvB94GzgPzgjs9+ObVjxV5CTyYNnTyQjTTjnwbls\n3Zkyham4W7NtV6JDMMbESMCkISIHejdgKLABWA8McY+lnMF9crn3rHI2ba/ngkfnUd/UzbvihjlO\nY01VHb1zM6McjDEmGQQrafzd3e4EPsaZEPBed//O2IeWGAcO6c2tPxrPvNVV/GLmQjyebtwVN8xx\nGmu21TG4T250YzHGJIWASUNVj1DVI3BKGAeqarmqTgAOwOl2m7KO2a+Ea48ezaxFG7jljS8THU7i\nhDlOY23VLgb3tqRhTCoKpSF8lKou9j5Q1SXAPrELKTlceNhe/Pigwdzx1nKeqliT6HASI4xxGh6P\nsq5qF4P65MQgIGNMooXS5XaRiNwHPOY+Pg1YFLuQkoOI8IcTxrK2ahe/fmYxgwpzOHhEUaLDSnqb\ndzTQ2OKxkoYxKSqUksY5wGfAFe621D2W8jLT07jztAMZVpTHhY/NY/nmHYkOKel5u9sO6m0lDWNS\nUShdbutV9VZVPcndblXV+ngElwx6ZWfywNkTycpI5+wH51JpXXGDWtuaNKykYUwqCmXCwkNE5A0R\n+VJEVni3eASXLAb3yeW+s8qp3NnA+Y9UWFfcILxjNKykYUxqCqV66n7gFuBQYKLP1q2MH1zIbT86\ngAVrqvn5U92kK24Y4zTWVtVR3DOL7Mz0GARkjEm0UJJGjaq+oqqbVXWrd4t5ZEnoe2MH8Ouj9+Gl\nxRu46fUvEh1O7IUxTmPNtl0MtlKGMSkrlN5Tb4nITcAztF3u9dOYRZXEfjJlGKu21nLX218ztE8u\npx40JNEhxc769TBwYKdesra6jgOH9I5RQMaYRAslaXzL/eu71rMCU8O9qYj0AZ4EyoBVwA9VtcrP\neUOA+4DB7j2PUdVV4d43GkSE3x+/L2urdnHdc0so7Z3DlJHFiQwpdkpLO7VGeHOLh/XV9Rw/zkoa\nxqSqUHpPHeFnCzthuK4FZqvqSJxZc68NcN4jwE2qug9wELA5wvtGRUZ6Gnf8vwMY2S+fnz72KV9u\nsq64ABtq6mnxqI3RMCaFhdJ7qr+I3C8ir7iPx4jIeRHe9wTgYXf/YeBEP/cdA2S4s+yiqjvdBaCS\nQk+3K25Oj3TOeXAum3d0m17IAa2t8vacsqRhTKoKpSH8IeA1wFu5/SVwZYT37a+qG9z9jUB/P+fs\nDVSLyDMiMl9EbhIRv11yROQCEakQkYotW7ZEGFroBhbmcP9ZE9lW28j5D1ewq7F7d8X1DuwbbFOI\nGJOyQkkaRar6FOABUNVmoMNvRxF5U0SW+NlO8D1PVRV3VcB2MoApwC9wuvjuhbOmxx5UdYY7oWJ5\ncXF82xf2G1TAP358AIvW1XDlk/O7R1fcANZuqyNNnGRqjElNoSSNWhHpy+7lXicBNR29SFWnqepY\nP9vzwCYRKXGvV4L/toq1wAJVXeEmqueApFzH46gx/fntsWN47bNN3PjqskSHEz2dHKexpmoXA3pl\nk5keyn9WxpiuKJTeU1cBLwDDReR9oBg4OcL7vgCcBdzo/n3ezzlzgUIRKVbVLTi9tSoivG/MnHNI\nGau31jLj3RUM6ZPL6ZOGJjqkyHVynMaabXUMsnU0jElpofSe+hT4NnAwcCGwr6pGOsvtjcBRIvIV\nMM19jIiUuzPqoqotOFVTs0VkMSA4i0AlJRHht9PHMHV0P65/4TPe/iIpOnpFppPraaypqrOeU8ak\nuA5LGiKSDfwUZxoRBeaIyN2RTFrojig/0s/xCuAnPo/fAPYP9z7xlpGexj9/fACn3P0hl/57PjMv\nmsw+Jb0SHVb4OjFOo76phU3bG6wR3JgUF0rl8yPAvsA/gTvc/UdjGVRXlpeVwQNnTyQ/K4NzH5rL\npu3doyvuumqnu+0Qq54yJqWFkjTGqup5qvqWu52PkzhMAAMKsrn/7HJqdjVx3sNzqWtsTnRIMbdm\nm7e7rSUNY1JZKEnjU7fHFAAi8i2SuEE6Wew7sIA7/t8BLF2/ncv/s4CWFO+K25o0rE3DmJQWStKY\nAHwgIqtEZBXwITBRRBaLSMov+xqJqaP7c/1x+/Lm55v480ufJzqcmFpTtYseGWn065mV6FCMMTEU\nSpfb78U8ihR21sFlrNpaywPvr6SsKJczJ5clOqTQdWKcxjdb6xjcO4e0NIlhQMaYRAuly+1qnFlm\np7r7tUCaqq52H5sO/ObYMUzbpx83vPAZby3rQl1xOzFO45ttddaeYUw3EMqEhdcD1wC/cg/1AB6L\nZVCpJj1NuP3UAxgzsBeX/vtTPlvf4YD65BDiOA1VZc22OoZa0jAm5YXSpnEScDxOCQNVXQ/0jGVQ\nqSgvK4P7z5pIr5xMznuogo01XaArbmlpSKdV1zWxo6HZShrGdAOhJI1G30kFRSQvtiGlrv69snng\n7InsqG/i3IfmUtuQGl1xV7s9p4b2tf80jEl1oSSNp0TkHpx5oM4H3iSJp/NIdvuU9OLO0w7ki007\nuOw/81OiK+7qrbUADO1rJQ1jUl0oDeE3A08D/wVGAb9T1X/GOrBUdvioftxw/L78b9lm/jhraaLD\nidg3W52Sho0GNyb1hTL3VCFQDTwFfKmqXaQVN7mdMWko32yt5d45KxnaN5dzDhmW6JDCtnpbHf17\nZZGd6XeNLGNMCgmYNEQkC7gHZynWFTilkqEi8ixwkao2xifE1PWro/fhm211/GHWUgb1zuWoMf4W\nMEygEMdprN5aS5m1ZxjTLQSrnroOyAQGq+qBqjoeGIKTaH4bj+BSXVqacNuPDmD/0gIu/898Fq9N\nskJciOM0Vm2ts6RhTDcRLGl8HzhfVXd4D7j7P8XphmuiIKdHOveeVU6fvB6c9/Bc1ruzxSaFEMZp\n1DY0s2VHA0OsEdyYbiFY0vCoal37g6q6E/9repsw9evpdMXd1djCuQ/NZUd9U6JDcoQwTmNlpdNz\naq8iK2kY0x0ESxoqIr1FpE/7DfDEK8DuYtSAnvzr9AP5avNOLv33fJpbusY/8Sq3u22ZJQ1juoVg\nSaMAmBdgsxHhMTBlZDF/OnEs73y5hetf+AwNcdW8RFq5xU0a1qZhTLcQsPeUqpbF6qZuaeVJoAxY\nBfxQVav8nPc34Fic5PYGcIV2hW/SCPz4oCGs3lrH3e98zbCiPH4yZa9EhxTUyspaBhZkk9PDutsa\n0x2EMiI8Fq4FZqvqSGC2+7gNETkYOARnjfCxwETg2/EMMlGu/u4ojtlvAH9++XNeXbIx0eEE9XVl\nLcOKrZRhTHeRqKRxAvCwu/8wzliQ9hTIxplVNwun+++muESXYGlpwi0/HM+4QYVc+eR8Fq6pTkwg\nHYzTUFVWbNnJMGvPMKbbSFTS6K+qG9z9jcAeo9pU9UPgLWCDu72mqqm9/J2P7Mx07jurnKL8LM57\nuIK1VXt0ZIu9DsZpbNnZwI76ZkYU58cnHmNMwgVMGv56TbXrQRWUiLwpIkv8bCf4nuc7g267148A\n9gEGAaXAVBGZEuBeF4hIhYhUbNmypaPQuoyi/CweOmciDc0tnPdQBdvj3RW3g3EayzfvBGB4P0sa\nxnQXweaemofzZe5v/U4FgrbQquq0QM+JyCYRKVHVDSJSAvhbzu4k4CN3XAgi8gowGZjj514zgBkA\n5eXlKdVQPqJfT+45fQJnPvAJlzz+KQ+cPZHM9DgVEEtLIUi/g6/dpDHCkoYx3UbAbx9VHaaqe7l/\n22+Rdul5ATjL3T8LeN7POd8A3xaRDBHJxGkE7zbVU74OHlHE/520H3O+quR3zy9Jmq64X23eSc+s\nDAb0yk50KMaYOOlwllsAEekNjMRpmAZAVd+N4L434qzTcR6wGvihe59ynMkQf4IzHftUYDFOyeZV\nVX0xgnt2aT+cOJjV22q5862vKeubx4XfHp7okPhy0w5G9M9HxF9h1BiTikKZGv0nwBU4bQsLgEnA\nhzhf6GFR1a3AkX6OVwA/cfdbgAvDvUcq+vlRo1i9tY6/vLKMwX1yOWa/koTG89WmnUzbJ8lm5jXG\nxFQoleNX4IyRWK2qRwAH4KyvYeIsLU24+ZRxTBjam589uYD53+wxHjJutuxoYGttI3sPsMkBjOlO\nQkka9apaD84aG6q6DGcFP5MA2ZnpzDhjAv17ZXP+IxW8++WW2E1wGGScxhcbncmP97GkYUy3Ekqb\nxlp39b7ngDdEpAqnHcIkSN/8LB48ZyIn3/UBZz7wCSIwojif8YMLGTe4kPGDCxk1oGfkvayCjNNY\ntnE74Ey0aIzpPjpMGqrqXTvjBhF5C2ciw1djGpXp0PDifN65+gg+XV3FwjU1LFxbzexlm5k5by0A\nWRlpTNunP7f+aDw9MsJMHuvXw8CBfp9aun47/Xtl0Tc/K9y3YIzpgoIt99pLVbe3G8i32P2bD2yL\naWSmQ72yMzl8VD8OH9UPcKb1WFu1iwVrqvloxVYe//gbBhZmc92xY8K7QZBxGkvW17DvwIJwQzfG\ndFHBShr/BqbTdpCf79/knn61GxIRBvfJZXCfXI4bNxARuHfOSibt1Zcjo9jLqb6pha+31PK9fQdE\n7ZrGmK4h2OC+6e5f30F+e0VpcJ+Jg98cO4YxJb34+cyFUV1G9rP1NbR4lLGlVtIwprvpsLJbRGaH\ncswkn+zMdO487UCamj1c9p/5NEVpNcD53zg9rscPLozK9YwxXUewCQuz3faMonbLvpbhTCBouoBh\nRXn85Qf7M291FTe//kVUrrlgTTUDC7LpZ9OHGNPtBGvTuBC4EhiI067hnStiO3BHjOMyUXT8uIF8\ntGIr97yzgoPK+oTevuFnnIaqUrGqionDOpzo2BiTgoK1adwOjAD+1G7iwnGqakmji/nd9DHsO7AX\nP3tyAau31ob2Ij/jNNZs28XG7fUcZEnDmG4paJuGO//T9+MUi4mh7Mx07j59AiLCRY99yq7Glo5f\n5Gc9jY9WbAXgoDJLGsZ0R6GM+potIj8Qm8q0yxvcJ5fbTh3Pso3bue7ZxR1PsV66Z9PVO19toV/P\nLPbub2toGNMdhZI0LgRmAg0isl1EdojI9hjHZWLkiFH9uOLIkTwzfx3/mL28U69tbvHw3leVfHvv\nYpsO3ZhuKpRpRGxyoRRz+dSRfLOtjlvf/BJFuXLa3iG97qMV26jZ1cTU0f1iHKExJlklahEmk0Bp\nacJNJ48jTYTb3vyKXU0tXP3d0aSnBS89zFq0nrwe6RxhScOYbishizCZxEtPE/72g/3JykjjnndW\n8OXGHdz2owMoyM30e/72+iZeXLie744dQHZmepyjNcYkC1uEqRtLSxP+dOJY/njiWOZ8Vcmx/5xD\nxSqfeSh9xmn85+NvqG1s4ZyDhyUgUmNMsrBFmLo5EeGMSUN56qLJiMAP7/mQv7zyOfVNLa3jNNZX\n7+Kf/1vOt/cuZr9BNt+UMd1ZKEmj/SJMzxPhIkwicoqIfCYiHhEpD3Le90TkCxFZLiLXRnJPE9yB\nQ3rz8uVT+GH5YO55ZwVTb36bx575gOcXrOP/3fsRHlX+cMK+iQ7TGJNg0mFffd+TRb6NuwiTqjaG\nfVORfQAPcA/wC1Wt8HNOOvAlcBSwFpgL/FhVlwa7dnl5uVZU7HE50wkffr2Vm1//gv/+9BDKrplF\ncc8s7j79QCYM7dyAPhGZp6oBfxQYY7qeYIswZQMX4Uwlshi4X1XficZNVfVz9x7BTjsIWK6qK9xz\nnwBOAIImDRO5ycP78vRFk+Gn8OKlh7JPSU8yIl061hiTEoJ9EzwMlOMkjKOBv8clot1KgTU+j9cS\nYHZdEblARCpEpGLLli1xCS7VeRP6foMKLGEYY1oF63I7RlX3AxCR+4FPOnNhEXkT8Le023Wq+nxn\nrtURVZ0BzACneiqa1zbGGLNbsKTR5N1R1ebOThuhqtPCDcq1Dhjs83iQe8wYY0yCBEsa43zmmBIg\nx30sgKpqrxjHNhcYKSLDcJLFqcD/6+hF8+bN2ykiwVYbKgBqAjz2t+/tY9p+P9aKgMoAzwV7D76P\nI38Pkc0xZV2zjUk1qhr3DTgJp42iAdgEvOYeHwi87HPeMTg9qL7GqdYK5doVHTw/I9Bjf/s41V57\n7Mfh3yjg+wj2HtrHnqzvwTbbbOuaW0hzT0Wbqj4LPOvn+HqcROF9/DLwcpRv/2KQx/72Az2fSMHe\ng+/jZH4PxpguqFPjNLoCEanQFBgbkArvIxXegzGmrVTsSzkj0QFESSq8j1R4D8YYHylX0jDGGBM7\nqVjSMMYYEyOWNIwxxoTMkoYxnSAiZSKyJNFxhEtEbhCRXyQ6DtN1WdIwxhgTMksapktyf/F/LiL3\numuzvC4iOSLytneNFhEpEpFV7v7ZIvKciLwhIqtE5FIRuUpE5ovIRyIScN53EZkgIgtFZCFwic/x\ndBG5SUTmisgiEbnQPX64G8fTIrJMRB4Xdx4eEblRRJa659/sHisWkf+615krIocEieUGEXnAvf4K\nEbnc57mrRGSJu13pc/w6EflSRN7DZ5S+iAwXkVdFZJ6IzBGR0e7xU9xrLBSRdzv50ZhUl+jRhbbZ\nFs4GlAHNwHj38VPA6cDbQLl7rAhY5e6fDSwHegLFONOoXOQ+dytwZZB7LQIOc/dvApa4+xcAv3H3\ns4AKYBhwuHv9QTg/zD4EDgX6Al+wu9diofv338Ch7v4Q4PMgsdwAfODerwjYCmQCE3BmpM4D8oHP\ncJZm9h7PBXq5/wa/cK81Gxjp7n8L+J+7vxgo9Y3RNtu8W0JGhBsTJStVdYG7Pw8nkQTzlqruAHaI\nSA27R8cvBvb39wJ31cpCVfX+4n4UZ6kAgO8A+4vIye7jAmAk0Ah8oqpr3WsscGP7CKgH7heRWcAs\n93XTgDE+k4L2EpF8Vd0Z4H28pKoNQIOIbAb64ySlZ1W11r3nM8AUnKT1rKrWucdfcP/mAwcDM33u\nm+X+fR94SESeAp4JEIPppixpmK6swWe/BcjBKX14q12zg5zv8XnsIbz/FwS4TFVfa3NQ5HA/sWWo\nM1v0QcCRwMnApcBUN95Jqlof4n33uHYYsacB1ao6vv0TqnqRiHwLOBaYJyITVHVrGPcwKcjaNEyq\nWYVTJQPOF3NEVLUaqBaRQ91Dp/k8/RpwsYhkAojI3iKSF+ha7q/7AnXmVPsZMM596nXgMp/z9vgi\nD8Ec4EQRyXVjOMk99q57PEdEegLHue9rO7BSRE5x7ykiMs7dH66qH6vq74AttF2iwHRzVtIwqeZm\n4CkRuQB4KUrXPAd4QEQU5wve6z6caqdP3YbuLcCJQa7TE3jeXUpZgKvc45cDd4rIIpz/J9/FWWo5\nZKr6qYg8xO7F0u5T1fkAIvIksBDYjLPkgNdpwF0i8hucdpEn3PNuEpGRboyz3WPGADaNiDHGmE6w\n6iljjDEhs+opY1wicifQfozE7ar6YAJiOQe4ot3h91X1En/nGxMvKVc9lderpzb3KWh9rB6lac36\n1seZgwciaREtYdrhddMzPRFdOyejifzmPHZm1LY53jPd6Vzj8ShrPt/93Jh9M0iPsMzY4oGlnzXv\ncdx77fbP9967L/WeHs5rm9KQJiGtCWdraGbHro0AqGrk/9iuoqIiLSsri9blTJjmzZtXqarFiY7D\nJEbKlTT6FvdDfn1hm2OVdz9OXcUicsv3p+ii0wK8svOCXbf3gB1hX3e/og1MrZzC/4rmtDn+7d67\nlz6/58plVLxSybHTs7njrsJO36O21kNeXttMc+nF1bw0q56SkjQ2bPDscW3v8+VHFzH6+hNYXFkC\nQNXGnmRtzCR3I+SvayFvTS2LFj3CxsYVnY4rmLKyMioqKqJ6TdN5IrI60TGYxEm5ksaQ4cO1fdIA\n8NQ3kJad5ecVkenouuEkj1CSBsBk/XKPL/5QeL/8/SUcbzLxl1QAZm0ZRnZeOu9UjQqaNGTlOl7b\neu98VT2w0wEGUF5ernFLGuvXw8CB8blXFyMi89RWZOy2uk1DeCwSRiyv25FpucvDShi1tR5emuVU\nc700q57a2rZVad5rBrp2dl566/5+RRs6ul1k9XSJVFqa6AiMSUpRSRoi0ltE/E7D0N1VbeyZ6BDa\nyMtL49jpzkDpY6dnh5V4jDHdV9jfGO4sm73c2UE/Be4VkVuiF1rqSHTiaF+auOOuQpYs6xdWW8i0\n3OUhnafD7Je6Makokp+ZBe5UBN8HHlHVb+FMvGb86Ezi8LYVRMOlF1czdvRmLr24us1xK2EYY8IR\nyTdHhoiUAD9k92ydJol01H5hjDGdFUnS+APOhG1fq+pcEdkL+Co6YaWmeFdTWftFBK6/PtERGJOU\nwh6noapxirs+AAAgAElEQVQzgZk+j1cAP4hGUKmsamPPiMZw1Ne2tOnB1JE77irkrzf77z4bK7WD\n88hbU9vxicnshhsSHYExSSmShvC9RWS2iCxxH+/vzpZpYuSeK5dx2YEfcs+Vyzr1OithhGH9+o7P\nMaYbiuTb5F7gV0ATgKouAk6NRlCpLpxqqqa6JipeqQSg4pXKpG2f2FkaeikoFCJygYhUiEjFli1b\nonrtoGychjF+RZI0clX1k3bH9py8yERFZm4mvQc4cz31HtAj5qWHjpJS+9HpsaKqM1S1XFXLi4tt\nuiNjEi2Sb55KERkOKIC7TnKHQ4RNaHY1Z7Z53FTXRNXGRgCqNja2fqnHosQRqJtuR+oGRD0UY0yS\niSRpXALcA4wWkXXAlcDFUYkqxXjqG/Y41tkqqszcTMqPLgKg/Ogi8vLSwv5yD8a66Rpjggk7aajq\nClWdBhQDo1X1UFVdFbXIUkTl3Y+z9tLfUXn34xFf68LbRvPPTydz4W2jY/blbt10jTHBdLrLrYhc\nFeA4AKqa2KlEFLI27q7aaRjQlLBQPPUN1FUsAqCuYhGe+pPbTHAYTvfb7Lx06mtbyCt2vty9s9VG\n88s9nG66DQOa2vy7d3k2TsMYv8IZp+GtVxkFTARecB8fx+5F7ZNGKF9ksUosadlZ5Jbv37rmRjRm\nxPWuo/GqO615rMZgdPsSho3TMMavTicNVf09gIi8CxyoqjvcxzcAL4VyDRFJByqAdao6XUSGAU8A\nfYF5wBmq2igiWcAjwARgK/CjaFWBeRoaSMtyvsQDJZZoJJOii07bo4QRrvraltZuty/Nqo/7oL1A\neg/YkfBJGaPO1tMwxq9IvnH6A40+jxvdY6G4Avjc5/FfgVtVdQRQBZznHj8PqHKP3+qeF7ENTzzC\n13/8FRueeCToeVkbM/fYwhEsYXTmyzY7L721MTzZ2xtqB+clOoTI2DgNY/yK5FvnEeATEblBRH4P\nfAw81NGLRGQQcCxwn/tYgKnA0+4pDwMnuvsnuI9xnz9SvI0nYfI0NLBzyQIAdi5ZgKdhz55NwUSS\nPKLB2xgezrTmxhgTqUjmnvqziLwCTMEZq3GOqs4P4aW3AVezu22kL1Ctqt6BgWsB78+8UmCNe79m\nEalxz6/0vaCIXABcAFBUVMzVw4P/Smy68194dtWRlpNLZp++IYQcmGZEvlxuenPLHseKyWbyxqnk\nZOyuIutZPbF1/8vKziW7aCr0OCWnqc3ZTHbHk7QUpCN5QtoQSG90/k3efzZhIRpjYiTspOFqwVnS\nUwlhaU8RmQ5sVtV5InJ4hPdupaozgBkAQ/Yarv/4el2Hr/E0NJFWXw9VHZ8bikjbP9r3orqgeR9m\nZHy+x5Kq3pHY5SEuhhQLb9aNAGi7Tnjl7nXCwVkr3BiTeiKZsPAK4HGgCOgHPCYil3XwskOA40Vk\nFU7D91TgdqBQRLwJbBDg/SZfBwx275cBFOA0iEfM2wgeLYmutkqUENYJN8akkEjaNM4DvqWq16vq\n74BJwPnBXqCqv1LVQapahjO54f9U9TTgLeBk97SzgOfd/Rfcx7jP/09VI68PiqHumDhSko3TMMav\nSJKG4FRPebW4x8JxDXCViCzHabO43z1+P9DXPX4VcG2Y14+r7pw4Umb+KRunYYxfkbRpPAh8LCLe\n5s4T2f1l3yFVfRt4291fARzk55x64JQIYkyYrI2ZnWrniHRxpliqrU2O8SBxZeM0jPErkrmnbgHO\nBba52zmqelu0AksFqVDiiMWkiOFq9sSxZtLGaRjjV6Q/HxfgjJ94DtgqIkMiDym1xCpxeHswxVJn\nJ0X0U7KKavHk8w3bOe+huby6ZCNNLTb7rjGJEEnvqcuATcAbwCycKURmRSmulBJq4vA3OtzbpTUR\nIpnxdsnixwEOiGY8xT2zWLK+hosem8fkv/yPG19ZxuqtXXwtcmO6mEjaNK4ARqlqVLrAmuQUzqSI\nLY0N1CxfEPVYBvTK5v1rpvLOl1v4zydruHfOCu5+52sO27uYMyYNZerofqSnRTRhgDGmA5EkjTVA\nTbQCSXWdbRgPVywarUO5nu+khek9sigYMT4miSMjPY0j9+nPkfv0Z2NNPU/M/Yb/fPIN5z9SwaDe\nOZwxaSg/mjiYwtweUb+3MSayOucVwNsi8isRucq7RSuwZNHZuamCiXXDeLwbrYOtE172vTMBQplW\nJmwDCrK5ctrevHfNVP512oGUFubwl1eWMekvs/n1s4v5alMEvdFsnIYxfkVS0vjG3Xq4W8rZ8MQj\n7FyygPyx4yk59cxEhxNU+0braEybHm6ppW4A3ulE4tJanZmexjH7lXDMfiV8vmE7D72/iqfnreXf\nH3/Dt/cu5idThnHoiCI6NdeljdMwxq9Iutz+3t/mfV5E/hmdEBMj0tlwA+motBHuuhTRXqY1mbra\ndsY+Jb3468n78+G1U/n5UXuzdMN2zrj/E46+fQ4zK9bQ4GdySL/Wr49toMZ0UbEcsXVIDK8dc2lZ\nWeSPHQ9A/tjxUZ+rKhbuuKuQJcv6RTxteqzWH4+nvvlZXHbkSN675ghuPmUcAL98ehGH/vUt7nxr\nOTV1HbQv2TgNY/zqZsN8O6fk1DMZ/tu/RL1qKpK2jXeqRgV9PhqN4B2VWqblLqe+dvcvdt9JCxO5\nJrs/WRnpnDxhEK9cMYVHzj2I0QN6ctNrXzD5xtn8/sXPWFtVl+gQjelSIp0aPemIttanB9SZ+ZGS\noYSxuLIk7rPJButqe+nF1bw060PKjy5i9PXBk1iyEBEO27uYw/YuZun67dw7ZwWPfriaRz5czfT9\nSzh/yl6MLS1IdJjGJL1YljSStsN87sbgWzwEK20ky3rb/hKGb9VVxSuVNHVUzRMhEblARCpEpGLL\nli1RueaYgb249UfjeffqIzj3kDLeXLqJ6f98j9Pu+4i3v9hMkk+kbExCRZw0RCQ3wFO3R3rtRElU\nIukKfKuuyo8uIjPXf/KL1my3qjpDVctVtby4uDg6F3UNLMzhumPH8MGvjuTao0ezfPNOzn5wLt+7\nbQ4Ajc1dry3HmFiLZBqRg0VkKbDMfTxORP7lfV5VH4o8vOQRiwTSVSc0vOOuQv756WQuvG10m+PJ\nOktvRwpyMrno28OZc/VU/n7KOETgtkN+zKF//R93vrWcqtrGRIdoTNKIpKRxK/Bd3JX0VHUhcFg0\ngkp2qVoC6Uwvqey89BhGkhg9MtL4gdtofuADtzPKp9H8umcXs3zzzkSHaEzCRVQ9papr2h3qdgtD\np0ri6KrjMmJBRDgsv4lHz/sWr145hRPGlTJz3lqm3fIOZz3wCW99sRlPPKdpNyaJRJI01ojIwYCK\nSKaI/AL4PEpxdSmRlDoCVVG1NMXvl3wqjMuIOnecxugBzmDBD66dylVH7c3nG7ZzzoNzOfKWd7j/\nvZXU7EquLsbGxFokSeMi4BKgFFgHjHcfd1uxLHX4myI9WmtqRHs0ebKN1YiGovwsLj9yJO9dM5Xb\nTx1Pn7we/HHWUib932yueXoRi9ZaCc10D2GP01DVSuC0KMaSEnI3dr7nULxmwA0mnCnQu6MeGWmc\nML6UE8aXsmRdDY99tJrnF6znyYo1jC3txakTh3D8+IH0yu6anRyM6Ugkvaf+JiK93Kqp2SKyRURO\nj2ZwXVWsG8k7GhUernAShnem23gPPkwGY0sLuPEH+/PxdUfyhxP2pcUDv3luCQf9+U1+9uQCPlhe\naW0fJuVEMiL8O6p6tYicBKwCvg+8CzwWjcCMM8jPtxtrU11TwHERJnF6ZWdy5uQyzpg0lMXranhy\n7hpeWLieZ+evo7QwhxPGD+SkA0oZ2T85Bm0aE4lIkob3tccCM1W1plNTT3cDnamq6qiKatVfn2Hh\nnKUMPXIYU/48NeQYYrEoUzC+izF1aWGspyEi7D+okP0HFfLb6WN47bONPDt/Hfe8u4J/vf01+5T0\n4rhxJUzfbyBD+gYaE2tMcovk22SWiCwDJgCzRaQYqI9OWKmjM9VUnnr/06+37GqkZs5SAFbPXhnS\n1B1v1o3glAuaE9aNNtFtNBGLcD2N7Mx0ThhfykPnHMRHvzqS648bQ05mGn979QsOu+ktjv3HHO58\naznLN++waUtMlxJJQ/i1IvI3oEZVW0SkFjgheqGljlBKHN4Fn3LL96foorb9C9JzelAwZQw1bkkj\nWBWVt0dVfW0LFa9UAtFblKlbWb8eBg6MyqWKe2ZxziHDOOeQYazZVserSzby8pIN3PTaF9z02hcM\nK8pj2j79OGJ0PyaW9SEz3T4nk7wineV2NFAmIr7XeSTCa6akYInDd8GnuopFeOpPhnYTrpZd831a\nLp/O+MFb/V6jfffb7Lx0yo8uouKVyqh0o+12SkshBiWAwX1yOf+wvTj/sL3YWFPPG59v4vXPNvLQ\nB6u4d85KemZlcPCIvhy2dzFTRhRbNZZJOmEnDRF5FBgOLGD3SHDFkkaneRd88pY00rJ3T8fu2xie\nnrPnqrrBxmpceNtozvpzizvlx/Kox20iM6AgmzMmDeWMSUPZ2dDM+8sreWvZZt79cguvfbYJgEG9\nczh4eF++NawvBw3rw6DeOZ1bttaYKIukpFEOjFGrkA1ZsNJGyaln4mn4EU1Do1siSMU5olJRflYG\n3913AN/ddwCqyorKWt77qpIPvq7ktc828VTFWgBKCrI5cGhvJgzpzfghhYwp6UV2pn3GJn4iSRpL\ngAFA9+ugH4FgicNZ8Cm0BuR3qka1jpEIxZt1I5iWG9vSxn5FG/yOXDedIyIML85neHE+Zx1chsej\nfLFpB5+s3MbcVduY/001Ly1y/rfLSBNGl/Rk7MAC9i0tYExJL0YN6El+Vsqtr2aSRCT/ZRUBS0Xk\nE6C124+qHh/oBSIyGKf6qj9OVdYMVb1dRPoATwJlOGM+fqiqVeKUw28HjgHqgLNV9dMIYk46noaG\nNqsDJsPo8EilTLfbJJGWJuxT0ot9Snpx1sFlAGyo2cXCNTUsXFvNorXVvPrZRp6Yu3v+0EG9c9i7\nf09G9s93E1Aew4ry6Z2badVbJiKRJI0bwnhNM/BzVf1URHoC80TkDeBsYLaq3igi1wLXAtcARwMj\n3e1bwF3u3y7NW9rw9pjKHzs+6uuQ+xPN0sa03OWt7Snf7v1FzEapJ0wY4zTiqaQgh5KCHL431im2\nqirrqnexbMMOlm3czrKNO1i+eSfvfVVJY8vuCSh7ZmcwtG8uQ/rkMqh3LqWFOQwszKGkIJuSgmx6\n5/YgLc2Sigkski6374jIUGCkqr7pruAXtHJVVTfgVmep6g4R+RxnwsMTgMPd0x4G3sZJGicAj7jt\nJh+JSKGIlLjXCXATyF8XfIb2naWJrwP27TG1c8kCPA0/CrgeuW9jeCLWC++WIhynEW8iwqDeTiKY\nNqZ/6/HmFg9rq3axonInK7bU8s22OlZvrWPZxh28+fnmPVYnzEgT+vXMoqhnFkX5WfTN60Gf/B70\nzetBYU4PCmxGgm4vkt5T5wMXAH1welGVAncDR4b4+jLgAOBjoL9PItiIU32Fe03fNTvWusci+tbs\nKKlA7BNLftXuHlP5Y8fvUUXVvsutibMojtNIpIz0NMqK8igrymNq24UW8XiUytoGNlTXs6FmFxtr\n6tm0o4FN2+up3NnIxpp6lq7fzrbaxjalFdO9RVI9dQlwEM6XPqr6lYj0C+WFIpIP/Be4UlW3+9ax\nqqqKSKd6ZInIBTgJjKKiIs44dnBnXt6hlh6xKa57DvkVeDyQtmePqX7pmfyiYGjr4/Tm3Ykup9Kp\nFupZPbHT9/wyzf+o83AUenYnuqnNztTqk5udX6ItBelcHrU7JUCMxmkkk7Q0oV/PbPr1zGbc4MKA\n56kqtY0t1Oxqoqq2kf3+GscgTdKJJGk0qGqj9wvfHeDX4f9lIpKJkzAeV9Vn3MObvNVOIlICbHaP\nrwN8M8Ag91gbqjoDmAEwtGy4PvpS+wUFw9PS3EB6Rtsqo2iXQAL1pLps1EBurlnd+th34kJv9VRn\nek95VUPU2jV8x4h42zS8vaeqKq0hPFWICPlZGeRnZVBamJPocEyCRTIo4B0R+TWQIyJHATOBF4O9\nwO0NdT/wuare4vPUC8BZ7v5ZwPM+x88UxyScKUviUqG/7JPH+OjF37Dsk7aT9uava2ndYkmaOy7d\nJHPjs2+SM8akjkiSxrXAFmAxcCHwMvCbDl5zCHAGMFVEFrjbMcCNwFEi8hUwzX2Me80VOMOZ7wV+\nGkG8IWtpbmDruoUAbF23kJZm/1U60UgcoU5o6NuF1cZCGGMSJZLeUx6cL/J7O/Ga94BAP6H3aEB3\ne03FfQnZ9Iws+paOY+u6hfQtHbdHFZUvb+JIhh5ZycAG+BmT2jqdNERkMUHaLlR1/4giShKjDzqd\nluZTgiYMX/nrWuKeOJrqmqB3518Xy9HhoUzb3iUk+TgNYxIlnJLGdPevtwTwqPv3dEJoCO9KQk0Y\nXuGWOgJNLRJsdPic6/7H6tkr+froIi68bbTfcyKVts2Dp0/oNZjemAqmjKHsmu/HJKa46WLjNIyJ\nl04nDVVdDSAiR6nqAT5PXSMin+K0dXRrsSp1eAf5texqZPXslQBUvFLJ8F/uFXCNjXB6WHn1+d12\nKu8I3BXTV1NdU2tMNXOW0nL59A5eERrf7tRDhgyJyjVDkiLjNIyJtkgawkVEDvF5cHCE10spnW0k\n78wKf+k5PRh65DCADhdlClfGimbyn68nY2VzSOdn5ma2xlQwZYzfadzDoaozVLVcVcuLi4ujcs2Q\nlJbG717GdCGRjNM4D3hARLxjl6uBcyMPKXXEsp2j8GenMem6bzpMGCHPhutRZNfu2sW8F5yVe/Oe\nr2f7+bsXAtIcgQBzE03581QmXdfEsro4lgiMMXEVdslAVeep6jhgHDBOVcf7zkArImcFfnX3EWm3\n3KyNgZNCVEsYCgX31DJk380MHb2Z3n/fCUDvv+9k6OjNDNl3M71m1AVstfImpliUeowxySPi6iRV\nrVHVGj9PXRHptbubzlRRRWqPFf/SheqrerLxiT40lrT9z6KxJI2NT/ah5mf5kG4zoBrTncWyDcK+\nXVzRHD0ezjoVnRk53jCpBzvPzWtzbNvpuTR8KzptFMaYri2WSSOlut9GKtbTjkRT7qv1eLJh26k5\nNKTB1zft5NKLq0N+fUpM3W7jNIzxy0oacRRK4vBXRRWoXSMWI6/TN7SQVqtseLEvG27oyQQP9AQ+\nnVVPbW03mh7bxmkY41csk8b7Mby2iZUW2PBCX5pGZ5KXl0bZ9Gy+BXz3yCzy8rpRj+r16xMdgTFJ\nKZJFmAqBM3HW9W69jqpe7v69NNLgUlEiphuB0LvetgxqG9sddxVSe7MnpITRVNeUOr2nusF6GsaE\nI5Kfji/jJIzFwDyfzUSoo15U4TSGRyKUhHHPlct4cuojzLnuf3GIyBiTKJEM7stW1auiFkk3Ek5p\nI9A8VN52jUQ2PtfXtlDxSiUAq2evZNJ1KTJpoTFmD5GUNB4VkfNFpERE+ni3qEVmOmVxZUmnG8b3\nGKsRpuy8dMqPLgJiN62JMSY5RFLSaARuAq5jd/daBfaKNKjuoKPSRlqYP9YXV5YkpNRx4W2jg06c\naIxJDZGUNH4OjFDVMlUd5m6WMOIkWLtGoFJHrJeHTamEYeM0jPErkqSxHKiLViDdUWcH/AWbh8of\nW0EvAjZOwxi/IqmeqgUWiMhbQOsi2t4utyZygRZn6gr2K9rAwkQHEQlbT8MYvyJJGs+5W1IRVfLW\n1LY5Vjs4L8DZiZeocRvt1daGNhaj27BxGsb4FXbSUNWHoxlILLVPIl7JnExC4V3JL5hQGsYvvbia\nl2bVc+z0bO64K7SV+owx3VMkI8JX4mdSwq7UGN4Vkkn7Kqpg64aHo7bWw0uznAWXXppVz19DHP1t\njOmeIvl2KAcmutsU4B/AY9EIKtHy1tS2bvEQzxlw2/egystL49jp2QAcOz3bEoYxJqhIqqe2tjt0\nm4jMA34XWUjJxZs4kqn0EU1v1o3gjruWd7qEEa2BgcaYriWS6qkDfR6m4ZQ8ImlYT2qxTh7hNoiH\n0q4RCithtGPjNIzxK5Iv+b+zu02jGVgFnBJpQMkuESWPrG8aaBiStftxu3aNjhJHokaJd2k2TsMY\nvyL5eXk0cD8wG2ftjHXAqdEIqiuIV3vHqlcfYcmMX7HhiUeCnhfvmW9Tnq2nYYxfkSSN54DjgCZg\np7vF55s0SUQ7cbRvEG9pbKBm+QIAdi5ZgKehwd/LWlniiKLS0kRHYExSiqR6apCqfi9qkXQhzS0N\nZKQ71UV5a2pjVlWV3iOLghHjqVm+gPyx40nLyurwNe2rqlp2NZKe0yMm8Rljup9IksYHIrKfqi6O\nWjRdwMKvZ7Kp6jP6996XccOdJpxYJo6y751JS+OPSO+R1Wair2DjNbyJY9Vfn6FmzlIKpoxhv5sm\ntz4f6ip+xhjTXiTVU4cC80TkCxFZJCKLRWRRtAILVywnfmhuaWBT1WcAbKr6jOaW3dVF0aqq8jdm\nI71HxyUMT33bqquWXY3UzFkKQM2cpSxY07fDa9TWekKMMn5E5AIRqRCRii1btiQ6HGO6vUgbwkcC\n38Fp25ju/k0oj6eJhV/PjMm1M9Kz6N97XwD69963tYrKK16N4+1V3v04ay/9HZV3P956bHtNXwqm\njAGgYMqYDquoLr24mrGjN3PpxdUxjbWzVHWGqparanlxcXGiwzGm2xNNsUnZRMT7huYD/n46FwA1\nAR772y9wH/vu7whw7WgqAioDPOcbZxpwgM9z84Ge7I53h/sY9nw/Xv6uEY33N0pVo9Y6LyJbgNXR\nul6UBPucEi1WsQ1VVcvg3ZWqptQGVHTw/IxAj/3tAzP87SfyfQR7D+1jT9b3kCpbMr/HZI7Ntq67\npewI7iBeDPLY336g5xMp2HvwfZzM78EY0wWlYvVUhaqWJzqOSKXC+0iF99CRZH6PyRyb6bpSccKh\nGYkOIEpS4X2kwnvoSDK/x2SOzXRRKVfSMMYYEzupWNIwxhgTI5Y0jDHGhMyShjGm2xORMhFZksD7\nny0idyTq/p1hScMYY1KciERteIUlDWNM0nB/8X8uIveKyGci8rqI5IjI2yJS7p5TJCKr3P2zReQ5\nEXlDRFaJyKUicpWIzBeRj0SkT5B7TRCRhSKyELjE53i6iNwkInPdefUudI8f7sbxtIgsE5HHRUTc\n524UkaXu+Te7x4pF5L/udeaKyCEh/hscJyIfu+/hTRHpLyJpIvKViBS756SJyHL3Hn7vIyI3iMij\nIvI+8KiI7Csin4jIAjfOkWF8RJY0jDFJZyRwp6ruC1QDP+jg/LHA94GJwJ+BOlU9APgQODPI6x4E\nLlPVce2OnwfUqOpE95rni8gw97kDgCuBMcBewCEi0hc4CdhXVfcH/uSeeztwq3udHwD3dfA+vN4D\nJrnv4QngalX1AI8Bp7nnTAMWquqWDu4zBpimqj8GLgJuV9XxOMtzrw0xnja644hwY0xyW6mqC9z9\neUBZB+e/pao7gB0iUsPuWQ8WA/v7e4GIFAKFqvque+hRnElYwZmEdX8ROdl9XICTyBqBT1R1rXuN\nBW5sHwH1wP0iMguY5b5uGjDGLYwA9BKRfFXd2cH7GQQ8KSIlQA9gpXv8AeB54DbgXJykF/A+7v4L\nqrrL3f8QuE5EBgHPqOpXHcThl5U0jDHJxnee/xacH7fN7P6+yg5yvsfnsYfwfhgLTglkvLsNU9XX\nA8Wmqs3AQcDTOLN9v+o+n4ZTYvBepzSEhAHwT+AOVd0PuBD3/arqGmCTiEx17/dKCPdpnXpbVf8N\nHA/sAl52r9NpljSMMV3BKmCCu39ykPNCoqrVQLWIHOoeOs3n6deAi0UkE0BE9haRgKusub/qC1T1\nZeBngLe663XgMp/zxocYXgGwzt0/q91z9+FUU81UVe/iOyHdR0T2Alao6j9wSix+S2Ed6RJJw23g\nWuw24FQkOh5jTNzdjPNFPh9nyvdoOAe4061mEp/j9wFLgU/dbrj3ELzE0hOY5S5C9x5wlXv8cqDc\nbXReitOmEIobgJkiMo89p7Z/Achnd9VUZ+7zQ2CJ+37HAo+EGE8bXWIaEbenRLmqJuu6BcYYE3Nu\nD7JbVXVKomKwhnBjjOkCRORa4GLaVqXFP44uUtJYCVThLAF+j6ra7J3GmJCIyJ1A+zESt6vqg/7O\nj3Es5wBXtDv8vqpe4u/8ZNRVkkapqq4TkX7AGzg9G971ef4C4AKArKysCQOKSyK/aZp0fE4nabBL\nis+TAhkiNKvS5tMRwF3NNk2UNJznm+tbWk/JyE5HwO/xdAnwWSs01Ptf3TUnZ3dcu3btfn1Gdjrq\nVgN7VJw3p27FsEfxtDSxdu1aVIO+6w75fraZmZkT+vfvD0AaGc69AnxOIf9btznu/3DA/0N8Pqc9\nnwv8/1Wa9zmFlobm1uPpWRmtMaQFuGugzzvN3/00A6S53bG2n3VWdtoe7zvgvRXq6zUqn6uvoqIi\nLSsri9blTJjmzZtXqSEs49slkoYvEbkB2KmqN/t7vmzwMB2988jIbzR4QOTXaKehJPBy2XX9M3fv\nFwkXH1jKXZ+uo97nI2wqaia7r9PleljRVvYrWA/AC1d/xBevr2XUdwZx/N8mtZ7f/vjE/BUB73/r\n5Sv48OVqjjvO6c344ov1HHdcNnff1bv1nJMv2MWHL1cz+ZhCDv2/aSyuGQjAysq+1G/NIbMyg+wt\nkFupLPjvH2isq4nql4t3/fcBmcMYl+f2FgzwOYX6b93meJH/UOsD/G/UVNTMz3sP4e9V3+zxXE5R\nXcD7l/Xd1rq/4IZX2PjWVww4YiTjbzi69bj3s/XH3+ft77PNWP4TmkfsOZ7M+1lPPqaQn/1jrz2e\nPyR7Q8B7TzxoE+vXe6L6uZaXl2tFhfVvSTQRmRfKol1JnzTcrm5pqrrD3X8D+IOqvurv/O6YNAAa\n6177qRYAACAASURBVJrpkbtnE5Xv8WBJA2B8yzry8pwOdbW1ntZ9r/frS6ivbSE7L525O/cKmjQA\nPnn05wvd0adR0SujSA/KP5YM8fnS7+JJA6C5rpGM3B5tjgVLGrDn592ZpAG0fo7+BEsaAKWDNkT1\nc7WkEaL162HgwJhdPtSk0RUawvsDz7qjHTOAfwdKGN2Zv4QR7Lg/vkmifcLwCvRFE0Bzx6eETpC2\nCSNFtE8YoejM5+pPJz/H9qL6uZoQlZY6dYQJFtekISLDgbWq2iAih+MMLnnEHWjjl6quYPdgGWOM\nMQkU78F9/wVaRGQEzvrFg4F/xzkGY4wxYYp30vC487ScBPxTVX8JRKGrkzHGmHiId9JoEpEf48yn\n4p0JMvUqqY0xJkXFO2mcA0wG/qyqK9056h+NcwzGGNP1XH99oiMA4tgQLiLpwHWq2joEXlVXAn+N\nVwzGGNNl3XBDoiMA4ljScKfxHSoine9faIwx3d364GN34iXe4zRWAO+LyAu0XRzkljjHYYxJIN/p\nYYYMGZLgaLqIJBmnEe82ja9xGsDTcOag927GmG5EVWeoarmqlhcXdzjdkUkicS1pqOrvAUQkV1UD\nz7NgjDEmKcW1pCEik92VpZa5j8eJyL/iGYMxxpjwxbt66jbgu8BWAFVdCBwW5xiMMcaEKe5rhKvq\nmnaHWvyeaIwxZrfuNk7DtUZEDgZURDJxVrD6PM4xGGNM19Pdxmm4LgIuAUqBdcB497Exxphguuk4\nDfUdEW6SW9o2D54+ca/BTAqFu3ZSnZOf6DCM2a2bjtP4SERmisjRIoEWavZPRNJFZL6IzOr4bBMN\nvX5bk+gQEuZXbz2b6BCMSUrxThp746yjcSbwlYj8n4jsHeJrrf0jjtJXNJP7fD3pK7rfIm1DqzZz\nzBfzGVK1JdGhGJN04po01PGGqv4YOB9nivRPROQdEZkc6HUiMgg4FvC/4HEImrWp869paQz3djHh\nqW/As8vZwFknuj1/x0K7uCK1ntYt42ln7GXOC7taj6XXtoAn8cXjaBP1kNPUQE6js33viwUAHP3F\nfHIaG8htcDbxeFpf42lo6PR9vJ9buML+bF31tYE7KgZ7zg+bP64bi/dyr32B04EzgE3AZcALOA3i\nM4FhAV56G3A1YU45srD2f2xsWsmAzGGMy5sa2mu+eYaNNZ8zoGAfxg35fji3jaqN/3mEnYsXtD5e\n0y+fVzfvZNR3BnH83yYB8MLVH/HF62vbHAuZQv7dteT/Yyfi8/3R6+ad9Lp5J5oOoy5JY+klA6Lx\ndpKKKJy29G3OWfwmGbo7MVz64atc+uGrNKelcecRR/GvI45C2f1Z5JbvD9f8OqR7rLt5Jjs++Iye\nB+9L2V+O7HSMEX22wK2Xr+DDl6uZfEwhP/vHXiE/197wERsA9ut0ACZliMaxYUVEvsRZP+NBVV3b\n7rlrVHWPadJFZDpwjKr+1F1X/BeqOr3dOa2TnxUVFU34v+tubn1OUXa0bG193DO9L0Lw5pQ9XpM9\ngE42wfi/bmbggp0nc/f1PRlCcW4mW+qa8GQA6qFh/bqAr+03uhCAzcuq2xxLS2sbc25a8F+6+WlN\nsMODrmxp81NSM6FlSAY7cnYfrfNkUdfirJ/V+P/bO/f4usoq739/uTTpvbQJvaQtLVKBcilCVKDK\nMIAoaKEvgy8UsYpopx0VqHQUFOctn9FBBQYFK1LFKQUFESpS5KbIRbm3paWlBenbYqEXeqHXhKRJ\nzpo/9j7JSXJuSU52zknX9/M5n7P3s59n7/Xsdc5e+7mt1VhCrLEINYqiRihqDH5Tsy65cKmZVae9\naAYSdXvQ4KEn/HBuG9+WfZLH8Mr2XidSHttP1e4dlMZarGZDUTHvDB1GTVlZeOLWuhhz6HjebWrf\nii0qaTE+xGLUrd/SvDtoQiUUpf899StuOWcsZp3SreorsLLtWMxYv/r95vTxE/uisHyyYwNLkrdo\n9u83Xn+9kTlz5mBmXf9DhFRXV9uSJUtydbrey9y53TrtVlJW/9eojYasgxeUdB1By6QRKAcGAYvM\n7OJk+ceNGW9H7Gv9JteploY9l/OWRv3I1A2l2uEtD7/aCjHr+CpuXbaRutCXW9uWRvnBA6jrYEvj\nwwPWpZVvcvlmAB77xDYuWdPy4Nh9zUBqZg7g2bqWyLwv7zuUlbtHAbB++zDqdvSldHsJ5dug3/ZA\nxS/deWWXjUYig0sq7aSB57ZOHJO85ZPtvW6VXiG+uOQvXPnXlrkWN3x8Cred9c+t8iW2NH7wrW9z\n484N7c7Vt6K1a7XElsbkLFoaxwxuPb2yM7otWftlGg8LenQ70tKI/w6S8YHDNlNXhxuNXki2RiPq\nKbcVkr4JHEVgAAAws5RPcjO7GrgaIKGlkdRgpGJS/9M4yhooUfaRZSeNPY+jmvZTUpwf3bcjpk2n\n/iv/QvnQ4K3wA2P2cXjpBvr0a1HhOT86kf1zG1uldYbP9ROxctgztS+Df/8+fR+to2bmgTH99PS1\nq3i/pJQ/HH48U19fyulrV3IbrY3GiGnTiZ13AU1VxVmft2rOZ4m9fw5FfcuA9zosV1d1O/vmQ5l1\nXRPl/dvLnO5YW/7/2pFUjd68slNCOF1j0yYYNaqnpYh89tSvCZwVjgeuBd4CXo7iwh0xGM1l8sRg\nxCkqL6Oob1n44CHpA6SrBqNocxOqMbY/VEHtDUPY9scKtM8o2tL7vb0cvG8X/RrquWjaFXz/zAuY\ndtFs+jfUc/Ce9lOPi+LdVR0grrfO0lXdpjMK2RiMBPJrhsiBQlVVT0sARN/SGGZmt0u63MyeBp6W\nlLXRMLOngKe6SzgH1GRsW1wBfYPeh8YjStm2uILi92IZShY+xbEYF027nPqS4GVhbcVILpp2Of1i\nNRlKOs6BQ9RGIz66t1nSp4FNwNCIZXDS0DQ6yU+ir4KumLro5YmSzYPa/xTrS/qwe0h+tTgdpyeJ\n2mh8T9Jg4ErgFoJB7dkRy+A4juN0kqgj98WnpeyGNqOLjuM4Tt4TidGQdAuQcqqtmV0WhRyO4zgF\nywEWT8MnYTuO43SFPImnEYnRMLM7sskn6RYz+3p3y+M4jlNwHKDrNDIxuacFcBzHyUvyZJ1GvhkN\nx3EcJ49xo+E4juNkTb4ZjZw5QXMcx3FyT48YDUn9Uhz6SaSCOI7TI0iaIWmJpCXbtnmExEIiUqMh\n6WRJqwmcFiJpkqSfxY+b2YIo5XEcp2cws/lmVm1m1ZWVlT0tTmGQJ+s0om5p3AR8EtgBYGYrgFMi\nlsFxHKfwyJN1GpF3T5nZ222Ser/PbcdxnK6yaVPmPBEQtcPCtyWdDJikUuByYE3EMjiO4xQeVVUQ\nYaTVVETd0pgJfBWoAjYCx4X7KZFULuklSSskvSbp2gjkdBzHcZIQtZfb7cDnOlisHjjNzPaFrZO/\nSXrEzF7IvYSO4zhOOiI1GpJ+BHwPeB94FDgWmG1md6UqY2YG7At3S8NPz7fROkBjDmONx96v73LY\nUIC6muxiQqdif21j0vSm/fVA7wtaFKuv71SI186yv7brsd5TEdd9R34DNTUx+vfvno6JddtquOC2\n57vl3L2J30Je3Keou6fONLM9wGcI4oMfBvx7pkKSiiUtB7YCfzKzF7tVyhyyYsMinlh9PSs2LOry\nubbNv4s3L/4vNt14b5fOc9Nl65g+aQU3Xbau0+V/cvIDLJ/7SKv0LXcvZM28q1n7zMIuyZdvbLl7\nIeuuvZotd0dTr+VzH+EnJz/Ag9/MfWM6rvtZH1+Z9W9g5qydfPDwd5k5a2fO5XEKD1mEAyuSVpnZ\n0ZJ+CdxnZo9KWmFmk7IsPwT4PfB1M1uVkD4DmAFQUVFxwn9954auC9untMunMDP21m1p3h8wcBRS\n8kXvsdKW9FiJqOxXyrbaBmLxl02LUb9pY3OeQRMq6Vea/G0/Ff2K6rGYsX71+81p4yf2RUXBtQcU\nNaQqCsC+WGm78oMmVLI/VkpsP+x/u2V2R7+hVfzbl6YtNbPqDgnZhkTdHjR46Ak/nPvfrTOk0JOV\npn4fSrzXrdJL2qebxajb2nLfy0ZVgYJzW4kxvLgP7zbtb1euqCR1TPU+JWn0FjP2vNmy2O3gI4ZQ\nVJTZUUK/ovp2aaqvwMq2N++31V2cxN8AtP4dxGKwalXL/tFHl/KpT13eZb0mUl1dbUuWePSEjMyd\n263TbiVlpdeoZ089JOl1gu6pWZIq6UDkaTPbJelJ4FPAqoT0+cB8gHFjxtv91/6165KOGdH1cxC0\nNLbsXsOIwUdyxEe/kDJf7fCWh19thZh1fBW3LttIXcK6p02LFlK75FUGnnwUk687nWMGd2wK3ocH\nBG+V9/90Hc8/vIuTzh7C7KmHNh8/qnxz2vLP1o1sVX7EP0/guLlnsX77MOp292XHHb9h38rlDD1k\nEoedMr1DsqUiUbeDSyrb6zaFnupHDkx5zsR73Sq9IvnD+a0/38W+lcsZcMxxjJjWUq+GikauPGgs\nN+7c0K5M34ralNcfN+i9lMcAlv/xEbY8+SaHnzmac844MW3eOHHdJlKy9ss0HvbLVmlx3Q0bWcqO\nzQ3tfgPQ/ncwb95OFi+uY8qUcqZdeFBW8jjdQJ6s04h6IPyqcFxjt5k1SaoBzk1XJjQsDaHB6At8\nAvhhBOLmhEljz+OocEyj/btgx6iccTF9rtgVjmns6PR5Zt98KLOu6/yYxuybD+Uj14zljYaxrdJH\nTJtO6SkXMHBP7xrTGDFtOrHzLohsTOO4uWdx+PeP6pYxjUTdZzum8fNbD+LGG7pvTMPJkjyJpxF1\nSwPgCGCcpMRrp+ssHgncIamYYAzm3oRY4wVBrgbBgZwMggNdGgQHggfa7vbpxX3KKLB5ClkR5SA4\n0G2D4NCi+478Btxg5AF5sk4j6tlTdwIfAJbTshLcSGM0zOxV4EPdL53jOI6TiahbGtXARIty9N1x\nHMfJGVG3OVcBuRlhdhzHcSIn6pZGBbBa0kvQMi5sZudELIfjOI7TCaI2GnMjvp7jOE7vIE/iaUQ9\n5fZpSYcAE8zsz2EEv65N43EcxzkQyJN1GlFH7vsKcB9wW5hUBTwQpQyO4zgFSZ7E04h6IPyrwGRg\nD4CZvQkcHLEMjuM4hUdVVU9LAERvNOrNrNlRT7jAz6ffOo7jFAhRG42nJX0b6CvpE8DvgMURy+A4\njuN0kqiNxlXANmAl8K/Aw8A1EcvgOI7jdJKoZ0/FgF+EH8dxHKfAiMRoSFpJmrELMzs2Cjkcx3EK\nlgNsncZnwu+vht93ht8X4wPhjuM4mcmTdRqRGA0z+weApE+YWaLH2m9JWkYw1uE4zgFCYkTGsWPH\nZsjtAHkTTyPqgXBJmpywc3IPyOA4Tg9jZvPNrNrMqisrKzMXcPJmnUbUvqcuBX4laXC4vwv4UroC\nksYQxNsYTtCVNd/MftKtUjqO4zhJiXr21FJgUtxomFmr2G+SvmBmd7Qp1ghcaWbLJA0Elkr6k5mt\njkZqx3EcJ06PdA2Z2e62BiPk8iR5N5vZsnB7L7CGwGdV8nPnTMqAxqb9XTqe8fyNmSOHx+pT59lf\n25ixfDZ56mqa2qXV1MRafWcinZy5wNpot9EaUubNdF+bGnIra6yu++q+v7YxpQ6z0W0ykuk7kQw6\n9y7lA5ieiBGeDqU9KI0jCP36Yqo8MRpZUfMXJvU/rcvCrNiwiC271zBi8JFMGnteh49n4rUVv2Hb\nu69SOfxYxp/5haR5tty9kH0rlzPgmOMY9vWLWh178Jsv8Mbj73D4maM550cnJi2fmOfDP0vu5uum\ny9bx/MO7mDKlnJ/fehAAM2ftZPHiOkaNKmLTphhTppTz+ZtGpqzLtvl3UbvkVQZ98DiO+Ojns6l+\nh9nbtKNZtytq/sKWhvWM2ND+3q/YsIgtK9dQOfxYjpp0UbvzrH1mIe/9YwVDD5nEYadM77Jc8br3\nqz6WyhkXd/l8icT1B7TTcza6TUZc3yedPYTZNx/a7nhc94m/h8RjePjlAxrlU+RVScvM7PgUxwYA\nTwPfN7NFbY41z8SoqKg44aqrrmJg8TCU3galxUpL2Fu3pXl/YPkIpJbzmVna48nP2fKCZmbs29vi\ntbLf0Krm8rESUdmvlK019dRt3dicp8+YURT1CbeLGtjz5rbmYwcfMYSiotbXj8WMra/vat4fP7Ev\napPHYsb61e837x99dCkAq1a1f4tPLF8bK6O2Kchbv7+Y+vWb29Vl1iUXLjWz6nT3JBOJuh08ePAJ\n3/3udxlYPJS9Te8150m89231MmDgqHZ6S3ffkxFL8WplJcbw4lLeXre+Oa3PmFFQVERRSeo39T4l\nmVsH/Yob2ukPWvScjW5VX4GVbW8tcxt9ty2X7PdQFP5sY7HgdzFnzhzMrPN/rjZUV1fbkiVLcnW6\n3svcud067VZSVv/XgmhpSCoF7gd+3dZgQDATA5gPMGbMGLvh6nldb2mMGZHzlkb9yIGt9lO1NGor\nxKzjq/j5K5t56893tWpplA8L/tDjh+5g/eOLWloaZ6Roadzc8jb6n1OTv43e/9OWlsa0C4M3y3nz\n2rc0xk6d2FL3fYeycncw/W/99mG8fc/93dLSSNStpGbdNrc0ktz7uF5StTRefePupC2N2orkz8G6\nFJN7GioaufKgsfzHPQvbtTT6VtSmrNO4Qe+lPBbnmMGBYYvrD2in50y6LVn7ZRoP+2W79Li+Tzp7\nCLOntm9pPH3r6uaWRvz3EGfevJ0ZZXe6iTxZp5FvLY2fmtnX2qQJuAN4z8yuyHSOQ8aMtyP3nd51\nYcYEocwbm/ZTUtwnZbZMxxNpazQg6HsvKSmjdnhpc1rcaNy6bCN1lcFYQVFZGQ0VjS1Go2IHxwze\nxP7aRvr0S2/743k+PGBdyjx1NU2cPmxrq7Samhj9+xc1fz9b19I99XIbo1G3oy/FG5vot6eMftuD\n39RLd17Z5ZZGIoNKKuzkgVOb9xutgZKxY5LmransQ0lJWdJjtcNLaWqop7i09fHOGo0bd24gVldP\nUXnL+dIajWHZGw1oGbdIpud0uk1lNCDQd3n/5PHPJpdvbtZ5MqpGb34lVY9AZ/CWRpZ08zqNvGxp\nSBoCTAfGJV7bzC4Lv7+WpNhk4PPASknLw7Rvm9nDSa+RS4Eho0HI1mCkLJ/iwZZIUVnqPJkMRrZ5\nkj1A4g+NVA+PtqSTMxe07W4sUWmKnJnva1uD0VUSDUauSae/bHSbjFQGI04GnWc3M8LJLVVVkAcv\n+VF3Tz0MvEDg5TarH56Z/Y3c2wLHcRynE0RtNMrN7BsRX9NxHMfJEVHPt75T0lckjZQ0NP6JWAbH\ncRynk0Td0tgPXA98h5Z1eAa0n8LhOI7j5B1RG40rgcPMbHvGnI7jOE4LeRJPI+ruqbVA6rmIjuM4\nTnLyZJ1G1C2NGmC5pCeBZmc98Sm3juM4TgryJJ5G1EbjgfDjOI7jdIQDcZ1GErfnjuM4TgER9Yrw\n9STxXm5mPnvKcRynAIi6eyrRr0k58FnA12k4juMUCJHOnjKzHQmfjWb2Y+DTUcrgOI7jdJ6ou6cS\nPWMWEbQ88s09u+M4Tv6RJ+s0on5g30jLmEYj8BZBF5XjOI6TjjxZpxH14r6zgNuBJ4BngY3AhRHL\n4DiOU3hs2pQ5TwT0xDqNXcAyoC7iazuO4xQuB+I6DWC0mX2qIwUk/Qr4DLDVzI7uHrEcx3GcbIi6\ne+o5Scd0sMwCoEOGxnEcx+keojYaHwOWSnpD0quSVkp6NV0BM3sGyBxUuYM0WkPnyzbtz6EkLTQ1\n1GfOlEA8dnRX88SpqYlRU9M+oGKytJ7AwjkUXdFdKtLd+1h9x/TSquz79cTe73z5XFBX05TrU3b5\nuSFphqQlkpZs27YtFzI5ERF199RZEV8vKStq/sKWhvWMKB3PpP6ndazshkVs2b2GEYOPZNLY83Im\n09pnFvLeP1Yw9JBJjDrvCxnzL5/7CI8++SaHnzmac350YtI8D37zBd54/J20eeLcdNk6nn94FwBT\nppTz81sPAmDmrJ0sXlzHlCnlfP6mkR2sVW7Z27SDp3ffQ53VdEp3qYjf+0EfPI6xn57e6tiGPy5k\nz9+XM+CY4xgxbXqKMyRn4w2/Y+9zrwEw8OSjqJoT/UTBuF5POnsIs2/uuuOFmbN2Anyoq+cxs/nA\nfIDq6uqe76h3skaWBwMrmZA0Dngo1ZiGpBnAjHD3aGBVmtMdROugT+uAneH2YGB3m+3B4f7uJGVf\nIctY5xkoovUf8RWClfKp4o6kqwMEMu9tc854nsT6pLp+XAaSyNWR+h5uZgM7kL8dbXR7QpvDubj/\nye59LItjcSpIrqdU9zTKZtvBwJgcXr+5TmamLpynFZK2Af/I1fmSkEpHhUZ31+MQM6vMmMvM8v4D\njANWZZl3SYbj81PtJ9smeBtqtx1BnVPWI10d2sqer3XoLZ98rmM+y+b3oXDrcSCuxl6cZj/Zdqrj\nPUm6OiTu53MdHMcpQPK+e0rS3cCpBE2zd4H/Z2a3p8m/xMyqUx0vFHpDPXpDHTKRz3XMZ9mipLfc\nh3ypR963NMxsWgeLzO8WQaKnN9SjN9QhE/lcx3yWLUp6y33Ii3rkfUvDcRzHyR+iXqfhOI7jFDBu\nNBwnDZLGSHpS0mpJr0m6PEwfKulPkt4Mvw8K0yXpZklrwwWsx6e/QpflK5b0iqSHwv3xkl4Mr/9b\nSX3C9LJwf214fFx3ypUPpNJdIdJWzz2JGw3HSU8jcKWZTQROBL4qaSJwFfCEmU0g8Np8VZj/LGBC\n+JkB3NrN8l0OrEnY/yFwk5kdRrAu59Iw/VJgZ5h+U5ivt5NKd4VIWz33GAeU0ZBUJOn7km6RlHnZ\ndZ4iqX/oguEzPS1LZ5E0VdIvwrffM3tanlSY2WYzWxZu7yX441YB5wJ3hNnuAKaG2+cCCy3gBWCI\npG5ZSi9pNEHky1+G+wJOA+5LIVdc3vuA08P8vZY0uiso2uq5pykYoyHpV5K2SlrVJv1ToS+rtZKu\nSlU+5FxgNNAAvNNdsqYiR3UA+BZwb/dImZlc1MPMHjCzrwAzgQu6U95cEXbpfAh4ERhuZpvDQ1uA\n4eF2FfB2QrF36L4H1Y+Bb9KyynsYsMvM4g7HEq/dLFd4fHeY/4Cgje4KjbZ67lHyfsptAguAnwIL\n4wmSioF5wCcI/iAvS3oQKAaua1P+S8DhwHNmdpuk+wi6FaJkAV2vwyRgNVAegbypWEAX62FmW8Pt\na8JyeY2kAcD9wBVmtifxJd3MTFKk0xDDVuZWM1sq6dQor11otNVdT8vTEfJRzwVjNMzsmSSDdx8B\n1prZOgBJ9wDnmtl1BDE4WiHpHSDuojbnrj8zkaM6nAr0ByYC70t62MwifQPJUT0E/AB4JN6FkK9I\nKiV46PzazBaFye9KGmlmm8Pup7gR3Ehrf0+jw7RcMxk4R9LZBC8Qg4CfEHSHlYSticRrx+V6R1IJ\ngQ+yHd0gV16RQneFRDs9S7rLzC7uKYEKpnsqBR3tClgEfFLSLcAz3SlYB+hQHczsO2Z2BfAb4BdR\nG4w0dFQXXwfOAM6XNLM7BesKoXG7HVhjZv+dcOhBID4u9gXgDwnp08NZVCcCuxO6sXKGmV1tZqPN\nbBxByOS/mNnngCeB81PIFZf3/DB/r16klUZ3BUMKPfeYwYACamnkAjOrpWU2SUFjZgt6WoauYGY3\nAzf3tBxZMBn4PLBS0vIw7dsEraR7JV1K4KH1/4bHHgbOBtYCtcAl0YrLt4B7JH2PwKtt3OXO7cCd\nktYSxKe5MGK5eoKkujOzh3tQpoKn0I1GVF0B3UlvqAP0nnq0wsz+BqSaZXR6kvwGfLVbhWp/zaeA\np8LtdQRdhW3z1AHRB/ToQTLoruBI1HNPUujdUy8DE8IFTX0I3p4e7GGZOkpvqAP0nno4jpOGgjEa\nCrzdPg8cLukdSZeGg31fAx4jmIN9r5m91pNypqM31AF6Tz0cx+k47rDQcRzHyZqCaWk4juM4PY8b\nDcdxHCdr3Gg4juM4WeNGw3E6gKRxbX1uFRKS5kqa09NyOIWLGw3HcRwna9xoOAVJ+Ma/JnSv/pqk\nxyX1lfSUpOowT4Wkt8LtL0p6QEHApLckfU3SNxQEtnlB0tA01zpB0gpJK0hYuKcgMM71kl5WEHDp\nX8P0U0M57pP0uqRfhy4tkPQDBUGBXpV0Q5hWKen+8DwvS5qcRpa5oZfhpyStk3RZwrFvSFoVfq5I\nSP+OpL9L+huB0854+gckPSppqaS/SjoiTP9seI4VkvLF3Y6TL5iZf/xTcB9gHEGQnePC/XuBiwlW\nzFaHaRXAW+H2FwlcewwEKglcg88Mj91E4AE11bVeBU4Jt68HVoXbM4Brwu0yYAkwHjg1PP9oghez\n54GPEbgif4OWqe5Dwu/fAB8Lt8cS+EpKJctc4LnwehUETgdLgROAlQTOLAcArxG4Ao+n9yNwargW\nmBOe6wlgQrj9UQK/RoT5qxJl9I9/4p9CdyPSLUjaZ2YDuvka5wATzewH3XmdFNeeCvzdzFZHfe0c\ns97M4j6FlhIYknQ8aUEwnr2SdgOLw/SVwLHJCkgaQvDgjL9x30kQnQ/gTOBYSXEHgYMJIvbtB14y\ns3fCcywPZXsBqANuVxC2Mx668wxgolrcrQ+SNMDM9qWoxx/NrB6ol7SVIJbHx4Dfm1lNeM1FwMcJ\njNbvLfC7hgJ39XF34ScDv0u4bln4/SywQNK9BE4+HacZNxrdiKRiM0vqgt3MHqQb3WykuzZBNLeH\nCOJyFDL1CdtNQF+C1ke827VtzJHE/LGE/Rid+y8I+LqZPdYqMXBf31a2EjNrlPQRAp9V5xOsoD8t\nlPdEC/xDZUO7c3dC9iKCgE3HtT1gZjMlfZQgWtxSSSeYWa93o+5kh49pZEDSvyf0WV+bkP5A2Bf8\nmqQZCen7JN0Y9n+fFPafXytpmaSVCf3GX5T003B7gaSbJT0X9lOfH6YXSfpZ2C/+J0kPJ7zVEZLN\ncAAAA0pJREFUJpP1LUk/lLQM+Kykr4Syrwj7zPtJOhk4B7he0vKwXztp33aB8hZBlwy0uAjvNGa2\nC9gl6WNh0ucSDj8GzFIQswFJH5TUP9W5wrf7wRZ4WZ1NEFAL4HECV/HxfO0e5FnwV2BqqOP+wP8J\n054J0/tKGghMCeu1B1gv6bPhNSVpUrj9ATN70cz+A9hGa0eUzgGOtzTSoCB29QQCr6ECHpR0SthV\n8SUze09SX4IodfeHb2P9gRfN7MrwHADbzex4Sf8GzAG+nORyIwm6GI4gaIHcB5xH0K0xETiYwKfT\nrzKIvcPMjg+vPczMfhFufw+41MxuCbsoHjKz+8JjTxD0778ZvmH+jOANuBC5gcBl+Qzgjzk65yXA\nrxRE53s8If2XBPpZpkDR22iJyZ2MgcAfJJUT/J6+EaZfBsyT9CrBf/IZgjC4WWNmyyQtAF6Ky2Zm\nrwBI+i2wgiBQ1MsJxT4H3CrpGoJxkXvCfNdLmhDK+ESY5jiA+55KSnxMQ8HslvOBXeGhAcB1Zna7\npLkEb3MQPDg+aWYvSGoEyuJdQwpm70w2s43hA/n7ZnaGpC8SDNh+Lfyz/8nMfh2W2WtmAyX9GFhh\nZv8Tpi8CfhN/2CeR+y3gn8zsH+H+PwHfA4aEsj8Wdj0sIDQa4dvvNoIB2jhlZnZk5++g4zi9FW9p\npEcERuK2VolBn/UZwElmVivpKVr6z+uSjCXE+6DT9T8n9lN3JQZATcL2AmCqma0IjdSpSfKn7Nt2\nHMdpi49ppOcx4Evh2ziSqiQdTDBLZmdoMI4ATuym6z8L/Es4tjGc5A/9dAwENod97ol98XvDY2n7\ntg80JM0Lx3kSP1FH3ovLckkSWeb1hCyOk4i3NNJgZo9LOhJ4Phyb2EewFuBRYKakNQTdOi90kwj3\nE8y0WU0Qf3sZwfz/bPku8CJB99OLhIaCoO/6FwoWhp1P6r7tAwozizTiXjrCLsn/6Wk5HKctPqaR\n5yicry9pGMEg52Qz29LTcjmOc2DiLY385yEFC8z6AP/pBsNxnJ7EWxoFiKTfE7irSORbbReZOY7j\n5Bo3Go7jOE7W+Owpx3EcJ2vcaDiO4zhZ40bDcRzHyRo3Go7jOE7WuNFwHMdxsuZ/AQ57Qnkteh1t\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc27c3010b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plot_objective(result=search_result, dimension_names=dim_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also show another type of matrix-plot. Here the diagonal shows histograms of the sample distributions for each of the hyper-parameters during the Bayesian optimization. The plots below the diagonal show the location of samples in the search-space and the colour-coding shows the order in which the samples were taken. For larger numbers of samples you will likely see that the samples eventually become concentrated in a certain region of the search-space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAGECAYAAADDQ9xjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8FHX6wPHPk14IBEhApAVCExBQQKWoqNixndjO3ttZ\nTj1P7/R3eqd3enqW09MT+6nYEBVBBRViBwwg0hHpRSCUkJC++/z+2ElIsrtJNrub3YTn/Xrta2e+\nM/OdZ7Kwz87Md75fUVWMMcYYX2IiHYAxxpjoZUnCGGOMX5YkjDHG+GVJwhhjjF+WJIwxxvhlScIY\nY4xfliSMMcb4ZUkCEJHCJtjH6SJyV7j342ffZ4pI/0js2xjTvIk9TOdJEqraKgT1xKqqKxQxhXLf\nIvIKMFVVJzVtVMaY5s7OJGoRkT+IyA8i8pOI3F+t/AMRmSciS0TkmmrlhSLyLxFZCIwQkbUicr+I\nzBeRRSLSz1nvMhF52pl+RUT+LSLfichqERnvlMeIyDMislxEPhORjyuX+Yl1rYg8LCLzgXNE5Gon\n9oUi8p6IpIjISOB04BER+VFEsp3Xp87xfF0ZozHG1GZJohoROQHoDRwGDAGGishRzuIrVHUoMAy4\nWUTaO+WpwBxVHayq3zhleap6KPAscIef3XUCRgPjgIecst8AWUB/4GJgRAPC3qGqh6rqW8BkVR2u\nqoOBZcCVqvodMAX4g6oOUdVfgAnATc7x3AE804D9GGP2Q3GRDiDKnOC8FjjzrfAkja/wJIaznPKu\nTvkOwAW8V6ueyc77PDxf/L58oKpuYKmIdHTKRgPvOuW/isisBsT8drXpgSLyAJDuxD699soi0goY\nCbwrIpXFiQ3YjzFmP2RJoiYB/qGqz9UoFBkDjAVGqGqRiOQASc7iEh/3Akqddxf+/8al1abFzzoN\nsbfa9CvAmaq6UEQuA8b4WD8G2K2qQ4LYpzFmP2GXm2qaDlzh/NpGRDqLSAegDbDLSRD9gCPCtP9v\ngbOdexMd8f0lX5c0YIuIxAMXVisvcJahqnuANSJyDoB4DA46cmNMi2RJohpVnQFMBL4XkUXAJDxf\nrp8CcSKyDM/9g9lhCuE9YCOwFHgdmA/kB7D9vcAcPMlmebXyt4A/iMgCEcnGk0CudG62LwHOCEHs\nxpgWyJrARhkRaaWqhc6N8bnAKFX9NdJxGWP2T3ZPIvpMFZF0IAH4myUIY0wk2ZlEMyAi7wM9ahX/\nUVW9Wi8ZY0woNfskkZGRoVlZWZEOY783b968PFXNjHQcxpjQavaXm7KyssjNzY10GPs9EVkX6RiM\nMaFnrZuMMcb4ZUnCGGOMX5YkjDHG+GVJwhhjjF+WJIwxxvhlScIYY4xfliSMMcb4FZEkISIvicg2\nEVlcraydMxrbz85720jEZowxZp9InUm8ApxUq+wu4AtV7Q184cwbY4yJoIgkCVX9CthZq/gM4FVn\n+lXgzCYNyhhjjJdouifRUVW3ONO/Ah3rWtkYY0z4RVOSqKKeXgf99jwoIteISK6I5G7fvr0JIzPG\nmP1LUElCRB5uSFkDbRWRTk4dnYBt/lZU1QmqOkxVh2VmWsejxhgTLsGeSRzvo+zkRtY1BbjUmb4U\n+LCR9RhjjAmRRnUVLiLXAzcAPUXkp2qL0vCMr1zf9m8CY4AMEdkI/AXP2NHviMiVwDrg3MbEZowx\nJnQaO57EROAT4B/UbKpaoKq1Wy15UdUL/Cw6rpHxGGOMCYNGJQlVzQfygQtEJBZPS6Q4oJWItFLV\n9SGM0RhjTIQENTKdiPwOuA/YCridYgUGBReWMcaYaBDs8KW3An1VdUcogjHGGBNdgm3dtAHPZSdj\njDEtULBnEquBHBGZBpRWFqrqY0HWa4wxJgoEmyTWO68E52WMMaYFCSpJqOr9oQrEGGNM9Am2ddMs\nfPSxpKrHBlOvMcaY6BDs5aY7qk0nAWcDFUHWaYwxJkoEe7lpXq2ib0VkbjB1GmOMiR7BXm5qV202\nBhgKtAkqImOMMVEj2MtN8/DckxA8l5nWAFcGG5QxxpjoEOzlph6hCsQYY0z0CfZyUzxwPXCUU5QD\nPKeq5UHGZYwxJgoEe7npWSAeeMaZv9gpuyrIeo0xxkSBYJPEcFUdXG1+pogsDLJOY4wxUSLYDv5c\nIpJdOSMiPQFXkHUaY4yJEsGeSfwBmCUiq/G0cOoOXB50VMYYY6JCsK2bvhCR3kBfp2iFqpbWtY0x\nxpjmo1FJQkQuAkRVX3OSwk9O+cUi4lLViaEM0hhjTGQ09p7ETcD7PsonA7c3PhxjjDHRpLFJIl5V\nC2sXqupePE1ijTHGtACNTRLJIpJau1BE0rDBh4wxpsVobJJ4EZgkIt0rC0QkC3jLWWaMMaYFaNSN\na1V9VEQKga9EpJVTXAg8pKrPhiw6Y4wxEdXoJrCq+l/gv84lJlS1IGRRGWOMiQrBPkxnycEYY1qw\nYLvlMMYY04IFfSZhmqesu6Z5la196NQIRGKMiWZBnUmISIqI3CsizzvzvUVkXGhCM8YYE2nBXm56\nGSgFRjjzm4AHgqzTGGNMlAg2SWSr6j+BcgBVLcLTG6wxxpgWINgkUSYiyYACOGNLWC+wxhjTQgR7\n4/ovwKdAVxF5AxgFXBZMhSKyFijAM3hRhaoOCzJGY4wxjRTseBKfich84Ag8l5luUdW8EMR1TIjq\nMcYYE4TGjidxaK2iLc57NxHppqrzgwvLGGNMNGjsmcS/6limwLGNrLdy+xkiosBzqjqh9goicg1w\nDUC3bt2C2FX0secXjDHRpLEd/B0T6kCqGa2qm0SkA/CZiCxX1a9q7X8CMAFg2LBhGsZYjDFmvxbU\nPQkRSQJuAEbjOQP4GvivqpY0tk5V3eS8bxOR94HDgK/q3soYY0w4BNsE9n/AAOAp4Gln+rXGViYi\nqZW9yjqDGp0ALA4yRmOMMY0UbBPYgarav9r8LBFZGkR9HYH3RQQ8sU1U1U+DCdAYY0zjBZsk5ovI\nEao6G0BEDgdyG1uZqq4GBgcZkzHGmBAJNkkMBb4TkfXOfDdghYgsAlRVBwVZvzHGmAgKNkmcFJIo\njDHGRKVgn7heJyJtga7V67KH6YwxpmUItgns3/D01fQLTid/BP8wnamlqR6wa+h+fK1njGmZgr3c\ndC6e7sLLQhGMMcaY6BLscxKLgfRQBGKMMSb6BHsm8Q9ggYgspto4Eqp6epD1GmOMiQLBJolXgYeB\nRYA7+HCMMcZEk2CTRJGq/jskkRhjjIk6wSaJr0XkH8AUal5usiawxhjTAgSbJA5x3o+oVmZNYI0x\npoUI9mG6cI4rYQLk7/kFG7TIGNNYwZ5JICKn4ukiPKmyTFX/Gmy9xhhjIi+o5yRE5L/AecBNgADn\nAN1DEJcxxpgoEOzDdCNV9RJgl6reD4wA+gQfljHGmGgQbJIodt6LRORAoBzoFGSdxhhjokSw9ySm\nikg68AgwH0/LpueDjsoYY0xUCLZ109+cyfdEZCqQpKr5wYdljDEmGjTqcpOIDBeRA6rNXwK8A/xN\nRNqFKjhjjDGR1dgzieeAsQAichTwEJ4WTkOACcD4kETXSIGMv9AUYzVE+vkFG//BGNNYjU0Ssaq6\n05k+D5igqu/huez0Y2hCM8YYE2mNbd0UKyKVCeY4YGa1ZUE/oGeMMSY6NPYL/U3gSxHJw9MM9msA\nEekF2I1rY4xpIRqVJFT1QRH5As8zETNUtXJ86xg89yaMMca0AI2+NKSqs32UrQwuHGOMMdEk2Ceu\njTHGtGCWJIwxxvhlScIYY4xfzb656qJN+SF/WKyh9fl6GK6pHlxrafsxxkQnO5MwxhjjlyUJY4wx\nflmSMMYY41fUJQkROUlEVojIKhG5K9LxGGPM/iyqkoSIxAL/AU4G+gMXiEj/yEZljDH7r6hKEsBh\nwCpVXa2qZcBbwBkRjskYY/Zb0ZYkOgMbqs1vdMqMMcZEgOzrmy/yRGQ8cJKqXuXMXwwcrqq/q7Xe\nNcA1zuxAYLGfKtvg3Stt9bLKaV/vlZqiV9sMIK+O5bWPI5BjyPexfTj0VdW0MO/DGNPUVDVqXsAI\nYHq1+buBu+vZJreOZRPqKquc9vVe+Wqi4/Z7DL6OI5Bj8Pd3aOpjsJe97NU8X9H2xPUPQG8R6QFs\nAs4HfhtEfR/VU/ZRPe/RonY8gR5DtB2PMaaZiKrLTQAicgrwBBALvKSqD9azfq6qDmuS4MLEjsEY\nE62i7UwCVf0Y+DiATSaEK5YmZMdgjIlKUXcmYYwxJnpEWxNYY4wxUcSShDHGGL8sSRhTBxHJEhF/\nz+FEPRG5T0TuiHQcpvmyJGGMMcYvSxKmWXB+0S8TkedFZImIzBCRZBHJEZFhzjoZIrLWmb5MRD4Q\nkc9EZK2I/E5EbhORBSIyW0Ta1bGvoSKyUEQWAjdWK48VkUdE5AcR+UlErnXKxzhxTBKR5SLyhoiI\ns+whEVnqrP+oU5YpIu859fwgIqPqiOU+EXnJqX+1iNxcbdltIrLYed1arfzPIrJSRL4B+lYrzxaR\nT0Vknoh8LSL9nPJznDoWishXAX40pqWL9NN89rJXQ15AFlABDHHm3wEuAnKAYU5ZBrDWmb4MWAWk\nAZl4uiW5zln2OHBrHfv6CTjKmX4EWOxMXwPc40wnArlAD2CMU38XPD+8vgdGA+2BFexrRZjuvE8E\nRjvT3YBldcRyH/Cds78MYAcQDwwFFgGpQCtgCXBItfIUoLXzN7jDqesLoLczfTgw05leBHSuHqO9\n7FX5irrnJIypwxpV/dGZnocncdRllqoWAAUiks++J88XAYN8bSAi6Xi+KCt/Ub+Gp+t6gBOAQU4f\nY+DpE6s3UAbMVdWNTh0/OrHNBkqAF0VkKjDV2W4s0N852QBoLSKtVLXQz3FMU9VSoFREtgEd8SSh\n91V1r7PPycCReJLU+6pa5JRPcd5bASOBd6vtN9F5/xZ4RUTeASb7icHspyxJmOaktNq0C0jGc3ZR\nedk0qY713dXm3TTu374AN6nq9BqFImN8xBanqhUichhwHDAe+B1wrBPvEapa0sD9etXdiNhjgN2q\nOqT2AlW9TkQOB04F5onIUFXd0Yh9mBbI7kmY5m4tnkss4PkiDoqq7gZ2i8hop+jCaounA9eLSDyA\niPQRkVR/dTm/3tuopxeB3wODnUUzgJuqref1xd0AXwNnikiKE8NZTtlXTnmyiKQBpznHtQdYIyLn\nOPsUERnsTGer6hxV/T9gO9C1EfGYFsrOJExz9yjwjtN9/LQQ1Xk58JKIKJ4v9Eov4LmMNN+5Mb0d\nOLOOetKAD0UkCc9ZyG1O+c3Af0TkJzz/B78CrgskQFWdLyKvAHMrY1PVBQAi8jawENiGp9PMShcC\nz4rIPXjua7zlrPeIiPR2YvzCKTMGsG45jDHG1MEuNxljjPHLLjeZ/ZaI/Aeo/YzCk6r6cgRiuRy4\npVbxt6p6o6/1jWkqzf5yU3p6uvbq1Ytde4rYuqPAa3liQhw9OrdvdP2lFetxuXd7lcfFZJAQd2Cj\n661u7969pKZ63//8tWQ9Li33Km+b0JGU2Fb11ltW4WJz/h4Ky8qIQUhPSaY8roByreDA+AISY1xe\n25RrR1bt9K5LRBjQsUONsvX5+eSXeBrolG3cmKeqmfUG1UAZGRmalZUVqupMEObNmxfSz9Y0L83+\nTKJjx47k5uaSt6uQ39z2ImXlNb/4bjhvNJecdlij699dNJ3VeVd7lffpOJlWiaEZYycnJ4cxY8Z4\nlU//dSKzttVstp4Qk8TdBz1HcqzfRjUAuFU55ZlXKd2xi+op8tjRCWyMX87JbX7mtxlLam0VT3LG\nVxz3/CdsLajZZP+sgf3557gTa8b388/c8JHn0YPVt9++rs6AApSVlUVubm4oqzSNJCIh/WxN89Ji\n7klktG3F3248lXZtUgCIjRFOO3oAvz0luC/y9JQT6dTm93gaqECMtKJL2/tCliDqcmyH8QxOH404\nH1Pr+HZc1P2OehMEwPz1m1izY5dXee485ajMoczI783M/Cxc6nmwSmIySGn7FIkJXXj2N6fRvW16\n1TZH9czinrFHe9V1Yu/e3HTEESTFNfvfGsYYP5r95aa+ffvqihUrquYrKlys3rSD9umptG9T/5dp\nQ1W48ymr2EBiXA9iY0JXL/g/k6i0p3wnhRV76JjUlViJbVCd3/6yjive8H54NikujoV/uokdpfns\nKS+gS1I8onnExvVGJKFqPVXl57wdpCYk0LlN6zr3taekhDbJyfM0hMOXDhs2TO1MIjqISEg/W9O8\nNOonoIi0Bbqq6k8hjidocXGx9Oneof4VA603pg1xCW1CXm9DtI5vR+t4v/3RAbDk+5X874HJrFu6\niezB3fntn88gIzWFvL1FNdY7qX9vANontqF9YuXxdPSqT0Tok5nRsPiSaj/obIxpKRqcJEQkBzjd\n2WYesE1EvlXV2+rc0ITdumWbuGvcw5SVeG5y5372E0u+X8n9H93KA999x5Y9nhv6I3p05e4Tx0Qw\nUmNMcxPImUQbVd0jIlcB/1PVvzhPjJoIm/rCF1UJolJxYQkbP1nKF/dewdJft5GWmEhW+7YRitAY\n01wFkiTiRKQTcC7w5zDFYxph17Y9vsu35hMbE8PBBx7QxBEZY1qKQFo3/RVPB2e/qOoPItIT+Dk8\nYe3fPv5gPlef9yzjj3+Ef9w7me1bfSeBSsPGHuy7/Hjf5cYY01ANThKq+q6qDlLV65351ap6dvhC\n2z99OmUBT/x9KuvWbGdPfjGzpi/mjze+hqvC7Xeb4y8czagzajY+Oe6CUV5lxhgTqEBuXPcBngU6\nqupAERkEnK6qD4Qtuv3QB297OvV0Z1fgPqQMgPULKsidvYrDR/fxuU1sXCz/N/Fmluf+wrolG+k1\nJIvswd2bLGZjTMsVyD2J54E/AM8BqOpPIjIRsCQRQvm7i3CNLMV9+r7xaFzDyvn37ukcvmgrp3Q7\niF5tfDdN7Tcsm37Dshu8rxJXOe+tn8MPO1aRmdSac7uPoHdap6CPwRjTcgSSJFJUdW61oQ/BMyqY\nqcblchMb2/gH2YeO7sm0gZu8yjckrWP2oj38e/E3/PPwcfymZ3D3G1SVW3NfYf6uNVVln2z+kWeG\nX8Wgtt2CqtsY03IE8m2WJyLZgAI44/xuCUtUzdCGjTu580/vcPwpj3DWOU/x+sTvaMzT7Kdefiik\nem8XG6PExrpxq/Lggs8pdQWXn+fuWFUjQQCUuSt4efWsoOoNhohcIyK5IpK7ffv2iMVhjNknkCRx\nI55LTf1EZBNwK3B9WKJqZsrLXfzhrrf5IXcNqrA7v4gXX/maSZMD71Yiu0Mn2sR7d/vhcgsVLs/H\ntau0mF/2BDcE8dq9vr+E1/kpbwqqOkFVh6nqsMxM63TUmGgQSOum1ao6FsgE+qnqaFVdG7bImpG5\nuavZ6uNZhY+m/RhwXQkxcfzmwDFe5bsLUvCMLgkJMbEcmFJ3f0r1GdDG9zDG/Vt3CapeY0zLUu89\nCRHx2e1G5b0JVX0sxDGFjEvdfLpxGd9vW8MBKa0ZnzWEA5KD+3L1paiozGd5cbHv8vrMXLKLLdsz\nSG1TBLFKUUU8peXxVcsv7jOU9MTkRtVdaWB6V04+cAifbN6XyNLjU7i613FB1WuMaVkacuM6zXnv\nCwwHpjjzp7FvEPaodOuc95i+aXnV/Ks/z+HNMZfRq3VoL2UcNrwnCQlxlJXVvE8wepTvJqt1KXVV\n8NX6tSgJlBY7vbImuCDBzaCOHbmwz6Gc03NQKMLmvoPPYewBBzN3xyoyE9swrvOhtEusfzAjY8z+\no97LTap6v6reD3QBDlXV21X1dmAo0KBmMCISKyILRGSqM99DROaIyCoReVucPqpFJNGZX+Usz2rs\ngf2Qt75GggDILy/h6WVfNbZKv9q0TuZPfxxHasq+rrYPGdyNKy8/KuC64iSG1PiEmoVlsVAYzz8P\nO41zswdTq4VZo4kIR3Y4iNsPOo1Leh5lCcIY4yWQJrAdgerXT8rw1ce0b7cAy4DKaz0PA4+r6lsi\n8l/gSjwP6l0J7FLVXiJyvrPeeQHEWGXpLt8Nr5bs+rUx1dXr6CP7ctiwHixdtpn09BSyezauu/LY\nmBguGDiI5xfUvOk9/MDO9G3fsK67jTEmVAJp3fQ/YK6I3Cci9wNzgFfq20hEugCnAi848wIcC0xy\nVnkVONOZPsOZx1l+nDTyZ7O/S0q924Sv1UxycgJDD81qdIKodOeII7lh6OG0T04mOS6Os/r257mT\nzwhRlMYY03ANPpNQ1QdF5BPgSDzPSlyuqgsasOkTwJ3su7fRHtitqpUX8DcCnZ3pzsAGZ38VIpLv\nrJ9XvUIRuQa4BiAzM5OcnByfO/5T4gAKK0qr5mMQeha19bt+pBQWFnrFNBwYnt2/an7hnDlNG5Qx\nxhD4yHQuwI0nSfjvcc4hIuOAbao6T0TGBB6eb6o6AZgAnuFL/Q39Weaq4N21P/L9tjV0TE7jwuxh\n9EyLvks29Q1faowxkRJIB3+3AFcD7+FpsP+6iExQ1afq2GwUcLqInAIk4bkn8SSQLiJxztlEF6Cy\nH4pNQFdgo4jEAW2ARj81lhAbx4XZw7gw23pDNcaYxgjknsSVwOGq+hdV/T/gCDxJwy9VvVtVu6hq\nFnA+MFNVLwRmAeOd1S4FPnSmpzjzOMtnamP6tjCN4nK7ycvfS4Wr3pNEY8x+IpDLTYLnclMlF5WP\nAAfuj8BbIvIAsAB40Sl/EXhNRFYBO/EkFtMEpueu4MnJX/PrrgLat07h2nEjGH9kaJ7HMMY0X4Ek\niZeBOSLyvjN/Jvu+3OulqjlAjjO9GjjMxzolwDkBxGRCYOXG7dzz8ie43J6Tth17ivj7xC/olpnO\nYf2sR1hj9meBtG56TES+xHOfARreuslEuamzl1YliOqmfL+k3iSxYvN2np8Z1Q/eG2OCEGjrph/x\ndA8eByAi3VR1fcijMk3K3z2I+u5NrN2+i4ufeZui0vJwhGWMiQINvnEtIjcBW4HPgKnANOfdRMC2\nHQWs27QzJHWdMMx3H1MnDutb53YTv/3REoQxLVwgZxK3AH1VNbiBDKLUzl/z+XV9Hln9O5PSKinS\n4fi1p7CEv/57Gt/P9wwY1KNre+67ZRy9shr/JPmQ7M784ZwxPPPRd+wtKSMpPo6Ljx/KMUN61bnd\nll3e3aOb5i3rrmmRDsFEmUCSxAYgP1yBRIrb7ebZu97mk9e+xlXhJjk1kSv+8hvGXX50pEPz6fEX\nv6hKEABrNuzg7kc+4O2nriImpvEd/11w7CGcMXIA67btoktGG9JS6k+Uw7O7Mmvp6kbv0xgT/QJ5\nTmI1kCMid4vIbZWvcAXWVGa88R1TX/4SV4Xn+nvx3lKe+eNbrF68McKReatwuZn5/Qqv8s1b81my\ncnOj6tSyBbh3Xo572yiSiq6nX6etDUoQAOcccTCHZB3YqP0aY5qHQJLEejz3IxLw9MNU+WrWvp4y\n36tMVfnmI+/ySBMgJsb3R+avvC5a8Qu681Io+xbc26Hsa3TnJWhFw9oiJCfE88r15/D05db5oDEt\nVSBNYO+va7mIPKWqNwUfUtNKSPL9J0hIivdZHkmxsTGcMLofU2curlHevXM7+vc+IOD6tOgtoKR2\nIVr8DpJ2R1XRnvJiUuMSiRXvRBQbE8OY/j0D3rcxpnkItAlsXUbVv0r0OfHCUcz+9KcaZQlJ8Rxz\n9vAIRVS3Wy4/ltKyCmZ+twKXWxnY50Duuenkxg1E5M6rszx3x1oeWjyNlXu20jYhhct7jeay7NFB\nRG+MaW5CmSSapSNOGsyND5/PxH99zK5te+jWtxPX/O0cOnaLvt5iAVKSE7jv1nHcfvVYyspctG+b\n2ui6JOFItMS7NYskHMmO0kJunPM6xS7POFO7yop4bOkMMhPTOLXL4Ebv0xjTvOz3SQJg3BVjOPnS\noyjZW0pq6+RIh9MgaalJ0Pj84JF8OpTmQOmn+8qSToOkk/l0zdyqBFHd5PXzLUkYsx8JZZIIzcDL\nERIbG9NsEkSoiMQhbf+Nli2EipUQ3x+JHwBAqdv3Q3Jl7gqf5caYlingJjEikuJn0ZNBxmKayJ7y\nAraVbK+al4TBSMo5VQkCYGyn/j5vVJ9w4ACvMmNMyxXIoEMj8YxT3QroJiKDgWtV9QYAVX0lLBGa\nkCl2lfD86leYu3M+itItpQvXZ19Jt5QuXut2S23PfYPP4J+LP6GgooQYhNO7DuG3PY6IQOTGmEgJ\n5HLT48CJeAYGQlUXishRYYnKhMUb695hzs55VfPrizbyrxVP8/iQvxPj46zhjK6HcEKnAaws2MoB\nSW3omNy6KcM1xkSBgC43qeqGWkUunyuaqPTdjjleZXllO1hRsMrvNslxCQxu29UShDH7qYD6bnIu\nOamIxOPp8G9ZeMIyxhgTDQI5k7gOuBHoDGwChjjzppkY2f5wr7LMxPb0Tau7t1djzP4rkG458oAL\nwxiLCbOLup9Lkauoxo3rG7Kv8nk/whhjILDWTf8EHgCKgU+BQcDvVfX1MMW2XyrML+J/j37M7BmL\nSGmVxKkXj+a0y44MSd1JsUnc3Ps69pQXUOouJTMxOp8qB1i0Kd9rbIO1D50aoWiM2X8Fck/iBFW9\nU0TOAtYCvwG+AixJhNBfr3qRRbP33Uh+5t5JlJaUM/66Y/1u49IKluyazNrCb0mITaV/+hl0S/W+\ntFSpdXyL6MDXGNMEArnOUJlQTgXeVdUWNwBRUygqmcWW7RewcetYduz+Ky73vj/jqkUbaiSISh+8\nkFNnnTO3/I3vt/+HLcU/sq7wWz7ZeCc/7/ks1KEbY/ZDgZxJTBWR5XguN10vIpl49TNt6rK3eAZb\nd1wGKABl5UsoKf0OuAuAXXkFPrfzVw6wq3QtqwtyvMrn5b1K79bHBxmxMWZ/1+AzCVW9CxgJDFPV\ncmAvYKPNBGB3wdNUJohKpeULcWshAP2H9SQ5NdFru6FH9/NbZ375Jp/le/yUG2NMIAJt1tIPOE9E\nLgHGAyeEPqSWq8Ll54tbPb2tpqYlcfND5xGXEFu1qEOXdlx731l+6+yQ1I8YHyeEHZMHBhesMcYQ\nWOum14CqpmHjAAAgAElEQVRs4Ef2PWmtwP/CEFeLlJw4gsKi92qVCiL7+vwec+ZQhozuww+zlpGa\nlsTwY/sTn+D/Y0qJa8/wjCuZk/dcVVlCTCojMm8IdfjGmP1QIPckhgH9VVXrXdP41Lb1HykpnV3j\njCI97WbWS81LTOkZaRx/zmENrndI+9/SOXUoawu/ITGmFb1aH09KXLuQxW2M2X8FkiQWAwcAW8IU\nS4vldrv5dPI8vpq+iKTk6xh3YQG9+ieQnHg0iQkDgZyg95GZ1JfMpL5B1xNJInINcA1AbOvMCEdj\njIHAkkQGsFRE5gKllYWqerq/DUSkK57LUR3xXJqaoKpPikg74G0gC88zF+eq6i7xDNT8JHAKUARc\npqrzAzqiKPTfhz9mypuzq+Zn58DVd5zM2ZfYfYPqVHUCMAEgsVNvO2M1JgoEkiTua0T9FcDtqjpf\nRNKAeSLyGXAZ8IWqPiQid+FpA/pH4GSgt/M6HHjWeW+2du0oZNq7c73K33nxS8787RHExsX62MoY\nY6JDIE1gv8Tzqz/emf4BqPNXvqpuqTwTUNUCPL3GdsbTdPZVZ7VXgTOd6TOA/6nHbCBdRDo1/HCi\nz7bNu3FVuL3K83cVUVhgj5kYY6Jbg5OEiFwNTAIqm9F0Bj4IYPss4BBgDtBRVSvvbfyK53JUZZ3V\nx6zY6JSFTUlJObO+WMK0jxawo46H1hqre3YHUtOSvMq7ZGXQpm2qjy2MMSZ6BHK56UbgMDxf8qjq\nzyLSoSEbikgr4D3gVlXd47n14KGqKiIBXX+ufoMzMzOTnJycQDavUlZWwcYNO6lwfulPnryCAzql\nk+bjSz0Yl/7hULZt3l31GJ2IcGC3dlVxFxYWNvoYjDEmnAJJEqWqWlb5BS8icdR+fNgHZ4Ci94A3\nVHWyU7xVRDqp6hbnctI2p3wT0LXa5l2cshqq3+Ds27evjhkzJoDD2OeOW17nxwXrapSlpCTw9uRb\nSE5JaFSd/qz7ZRtfz1hMfEIsx5wymA6d0quW5eTk0NhjMMaYcArkiesvReRPQLKIHA+8C3xU1wZO\na6UXgWWq+li1RVOAS53pS4EPq5VfIh5HAPnVLkuFlMvl9koQAEVFZSxdsjHk++ue3YGLrj+W8648\nukaCMMaYaBbImcRdwJXAIuBa4GPghXq2GQVcDCwSkR+dsj8BDwHviMiVwDrgXGfZx3iav67C0wT2\n8gDiC0hsbAxt26Wya+der2UZmdHVjXZxxXYKytfRJiGbxNi2kQ6HNcu3MP29HyjZW8bIEwZy2Bj/\nfUsZY5q3QEamcwPPO6+GbvMNIH4WH+djfaUJh0Qdf97hPP/szBplww/vSfes6HmQ68e8x1iV/xaK\nixhJ4KD0K+jf7uqIxTN75lIe+N1rVS22pk/6gfOuPSZi8RhjwqveJCEii6jj3oOqDgppRE3ovAtG\nkJqSyNQp8ykuKmP0Uf246LLRkQ6ryobCGfyc/0bVvFvLWLLrv2QkD6FD8vCIxPTSI594Nel976Wv\nIhKLMSb8GnImMc55r/yF/5rzfhENuHEd7cadcSjjzjg00mH4tLHwCz/lM8OTJPLyIMP/kKblZRVs\n+GWbV3lFucvH2saYlqDeG9equk5V1wHHq+qdqrrIef0R6yo8rGJjfDfFjfNT3ljbigp5dfF8Vl9y\nEduKCv2uF58QR5ce3pfiYuMC7XHeGNNcBPK/W0RkVLWZkQFubwLUI+1Mat/SEeLISjstZPuYtX41\nR775PK+8/xY9P5nOhU88yMz1v/hd/7LbTiImtubHfsYlo/ysbYxp7gJp3XQl8JKItHHmdwNXhD4k\nUykz+RAO7/A3Fu98lr0Vm0iL78Hg9rfQOqFn8JW73bj37uXBzz8idu9exuUuBOCkOfN5sONHHH3O\nFcTGxEBKCsTsSwqjThzI4+/cyPRJP1C8t5RRxw9k1IkDuebu4EMyxkSfQFo3zQMGVyYJVc2vvlxE\nLlXVV31ubBqtW9rJdEs7GZe7xO/lp0ZRZc/fH+DTfz5KnHvfjejbp8zg9ikz0GvvgHvvhXvu8dq0\nz8Fd6HNwl9DFYoyJWgFfLlLV/NoJwnFLCOIxfoQ0QQDExhJ3/9+47I7r2dyuTY1FOzuksOOTSfCX\nv0Cs9VJrzP4skMtN9fH3PISJUq0SEuh/1ni+3j2H8ybkVpUvvKQrm3p9xUVVnfOa+mTdNc2rbO1D\np4Z924bW50sw+zD7j1DeeG72zWH3R3cMP4LRc1ZRnhTDgvO6Up4YQ+/Pt7K5eBk7StdHOjxjTISF\nMknYmURztHkj8cUuXntrBJ/93wBee3sECUUVtNpWQoW7zOcmJa4KdpYUNXGgxphICOXlpm9DWJdp\nIgnueL756EbyKjxda+X1TuP1N0fQsbAtHZKya6zrVuXheTm8tmIBRRXlDGjXkX+MOJFBGc16XChj\nTB0CGXQoXURuFpHHROTfla/K5ar6u/CEaMKqe3eO73E7nZL6VhW1at2V4w/7B9XH/QCYsGQuzy2Z\nS1FFOQBLdm7l0s/fpajc9xmHMab5C+RM4mNgNp5eYL3H4zTNVlp8Bhf3fIK8knVUaBkdk3p5JQiA\nd1f95FW2q7SYzzeuaoowjTEREEiSSFLV28IWiYm4jKTudS6vcPv+bVDup9wY0/wFcuP6NRG5WkQ6\niUi7ylfYIjNRZ1zWQV5lyXHxHNcl28faxpiWIJAziTLgEeDP7GvuqkAI+ogwzcFNg0eyrmAXH69b\ngVuVDsmteHjkSaQnJkc6tJAJ9TMLzT0OYwJJErcDvVQ1L1zBmOiWFBvH00efwZa9e9hRUkTftpnE\nx9gT2ca0ZIEkicohRc1+rlNqazqlto50GMaYJhBIktgL/Cgis4DSykJVvTnkUZmwWLLmV/4z6RuW\nrN5C907tuPr0ERw5xK4WGmP8CyRJfOC8TDO0dWcBNz4yib0lnmcalq3dyh+e/pAJd53HoF4HRjg6\nY0y0CqSrcOsGvBmb9u3SqgRRyeVWJs1aaEnCGONXg5OEiKzBRyd+qmrXK5qB/L3FvssLfZcbYwwE\ndrlpWLXpJOAcwJ6TaCZGD+7JxBnzfZYbY4w/DX6YTlV3VHttUtUnAGu43UwMP6gbF500lJhq3W0c\nO7Q3Zx11cASjMsZEu0AuNx1abTYGz5lFKHuR3a9t31HApI/ms2Z9Hr17dmD8uENpm54a0n3ccu7R\nnD1mMMvWbiWrUzt6d80Maf2R0NABdqJlH8HUFepjbYq/nWn+AvmS/xf77klUAGvxXHIyQXK53Fx7\n++vk7SwEYHbuaj7/chkvPH4Jaa1CO2xplw7pdOmQHtI6jTEtVyB9N50MvAh8gWfsiE3A+eEIKloU\n7t7Lwi+XsH3jjrDuZ1d+UVWCqLRlaz5TP/PuddUYY5pSoM9J7AbmAyXhCSd6TH5yGi/9aSKlxWXE\nxMZw8hXHcvOzVxMTE8rB/DzKylw+y9dt2Nmw7Ssq+GD2EmavWE+H9FacN3owPTpamwJjTPACSRJd\nVPWksEXSaKEfWnvlvF949vevVM27XW6mPf85fYb34pSrjgv5/pKSfH8M/Xp19Fm+ozQPRclI9NxT\nuPn5KXy/fF3V8vdnL+alm85hQLcDfG5fVFbO5t176NK2DUnxdlvJGONfIN8Q34nIwaq6KGzRNIJb\nS/hhy8UcnPkoSXG+v1QD9fV7c/yUfx+WJJHeOoUe3TJYs35f34n9eh3ASccNrLHerrKdvLjmGVYV\nrgCgZ2pvhslZNRIEQElZBROmz+HJq8/w2tdL3+Ty7Kw57C0to3VSIjeNHcmFRwwJ+TEZY1qGQJLE\naOAy56G6UkAAVdVBYYksALtLclmy/S6Gdno5JPUlpST6LE/0Ux6smBjh2UcuZMasJaxen0efnh05\n/uiDSEyMr7HeK2v/W5UgAFbv/Zk81+uA9xnDqi3e91G+/Xkdj376ddX8npJSHpw6i34HZDI0q3Po\nDsgY02IEkiRODlsUIbCzZDalrjwSYzOCruu4i45k4t/fo6ykvEb5KVeNDbpuf1KSEzjzlEP8Ls8v\n382KgmVe5XtiN5OQ2payvTUTWP+uHbzWnfbTcp91T/tpuSUJY4xPohr6a/pNSUQKgBV+FrcB8uso\nq5z29V6p9vbhkAHUNU5H7eMI5BjyfWwfDn1VNS2YCkTkGuCayvrw/7lGUn2fVSSFK7buqtr8H6ox\njaOqzfoF5NaxbEJdZZXTvt4rX5E+Bl/HEcgx+Ps7NPUxtJRXNB9nNMdmr+b7aulNWz6qp+yjet6j\nRe14Aj2GaDseY0wz0RIuN+Wq6rD614xedgzNRzQfZzTHZpqv0D8Z1vQmRDqAELBjaD6i+TijOTbT\nTDX7MwljjDHh0xLOJIwxxoSJJQljjDF+WZIwxux3RCRLRBZHcP+XicjTkdp/ICxJGGNMCyMiIXu8\nwZKEMSZinF/0y0TkeRFZIiIzRCRZRHJEZJizToaIrHWmLxORD0TkMxFZKyK/E5HbRGSBiMwWEb99\n5IvIUBFZKCILgRurlceKyCMi8oOI/CQi1zrlY5w4JonIchF5Q8Qz/q+IPCQiS531H3XKMkXkPaee\nH0RkVAP/BqeJyBznGD4XkY4iEiMiP4tIprNOjIiscvbhcz8icp+IvCYi3wKvicgAEZkrIj86cfZu\nxEdkScIYE3G9gf+o6gA8Y9acXc/6A4HfAMOBB4EiVT0E+B64pI7tXgZuUtXBtcqvBPJVdbhT59Ui\n0sNZdghwK9Af6AmMEpH2wFnAAPV0cPqAs+6TwONOPWcDL9RzHJW+AY5wjuEt4E5VdQOvAxc664wF\nFqrq9nr20x8Yq6oXANcBT6rqEDzDTW9sYDw1tPQnro0x0W+Nqv7oTM8DsupZf5aqFgAFIpLPvh4F\nFgE+e6UWkXQgXVW/copeY1+npScAg0RkvDPfBk/iKgPmqupGp44fndhm4xl47UURmQpMdbYbC/R3\nTjYAWotIK1WtOeykty7A2yLSCUgA1jjlLwEfAk8AV+BJcn7340xPUdViZ/p74M8i0gWYrKo/1xOH\nT3YmYYyJtNJq0y48P14r2Pf9VHug9+rru6vNu2ncD1/Bc4YxxHn1UNUZ/mJT1QrgMGASMA741Fke\ng+eMoLKezg1IEABPAU+r6sHAtTjHq6obgK0icqyzv08asJ+9lZWq6kTgdKAY+NipJ2CWJIwx0Wgt\nMNSZHl/Heg2iqruB3SIy2im6sNri6cD1IhIPICJ9RCTVX13Or/Y2qvox8Hug8vLVDOCmaus1dDSv\nNsAmZ/rSWstewHPZ6V1VrRznuEH7EZGewGpV/TeeM5JGjf0TdUnCuRm1yLnZkhvpeIwxEfEoni/u\nBXi6QA+Fy4H/OJeNpFr5C8BSYL7TLPY56j4jSQOmishPeO4n3OaU3wwMc24SL8VzT6Ah7gPeFZF5\neHf1PgVoxb5LTYHs51xgsXO8A4H/NTCeGqKuWw6nFcMwVY3WPvuNMaZJOC28HlfVIyMVg924NsaY\nKCQidwHXU/PSWNPHEYVnEmuAXYACz6mq9WxpjGkwEfkPUPsZhSdV9WVf64c5lsuBW2oVf6uqN/pa\nPxpFY5LorKqbRKQD8BmeVgdf1VqnapjLpKSkod26dYtApKHjdruJian/9lCZu9RnudsVAyo1ykQg\nMb7miWKZuwxfn7YiVC4QAbdLvFcSSI6rWV9JRUVVfRtWr87TIIe4rP65JiYlDu3U5QDAc3yCeB1P\nJDT0s6quQitwqdurPFZiiPPzYGy5y43L7WObGCE+Nrbe2NyqlLkrfK6XFBtfZ7ylLhfuat8Lofhs\nK2VkZGhWVlYoqjJBmjdvXoM+16hLEtWJyH1Aoao+6m+dvn376ooV0TgUcsPl5OQwZsyYetd7cOk9\nbCheV6MsM/4Apr3cGZe75ud42bFD+f0ZR9Uoe3zlM+TuWlCjzOUW1uS39yQK4IJuo5g4Yz27S0pq\nrHd8n14885vTapTdM3UGb670dH+z+vbb54VywJvM/u317NdOoaI0lnmTDuaKY4/gpnENeoA1rBr6\nWVU3f9dC/rXSu5ueG7OvYmTG4T63+Wb5Wq578X2v8n+cfxKnDT2o3thKXOWMy3mIPeXFNdY5puMA\nHj6k7qsX7y1ewp0zplfNh/KzHTZsmObmWnuUaCAiDfpco6p1k4ikikha5TSeh1wi1glXtLk06xra\nxrevmk+Pb8vVvW7kz+ccR2L8vl+Xw3t35ZoTj/Da/pLu59M1uXPVfLwksKM4vSpBDG3Xk2v6jOXR\n006idWJi1Xp9MzO4d+wYr/ruO3ksPVLS8Xl6EgIVpbH8/HUWh2VnceXxh4VnJ03g0LaDOemAsYjz\ndxaEYzKPZER7/8c0ul8Wlx41lBjngSkROGv4AE49pF+D9pkUG89fB51LWty+Rwx6tTqA2w4aV++2\nZw8cwKiu3cL2uZpmJtKDbFd/4XnsfaHzWgL8ub5t+vTpo83drFmzGrxuhbtCl+Uv1iX5P2mFu7yq\nfHdhsc76aZUuWf9rndu73W5dvmelLtj1k5ZUlOju0r365dalunT3xhrrFZWV6axVq/WH9RvV7XbX\nWedXq9YokKsh/LfQb1A//WzhUl26YWsD/zJNI5DPqrZtJdv1hx0LdEtxw49p0858/WLRKl27bWej\nYisqL9Vvti3XBTvX1Ps51rZk61Z95MuvQ/rZDh06NKAYTPg09HMN20VeEckGNqpqqYiMwfMgx//U\n81CLv4S1mn0PphgfYiWWfq0HeJW3SU1izMHZ9W4vIvRN29fPV2IsHNXB+/JFcnw8Y7J7eJX7cmR2\nVoPWC0RqfCpjB/m+rNJcZSZmkJkYWJP/A9u25sC2rRu9z+S4BEZl9m3Utv07dKB/hw78odF7Ny1B\nOO8EvofngY9eeMbe/RCYCJwSxn0aY/ZzWXdN8ypb+9CpEYikZQjnPQm3evo4OQt4SlX/AHQK4/6M\nMcaEWDiTRLmIXICnL5LKXhLrbntnjDEmqoQzSVwOjAAeVNU1Tv/sr4Vxf8YYY0IsLPckRCQWT8uk\nqgbZqroGeDgc+zPGGBMeYTmTUE+Xtt1FJCEc9RtjjGka4WzdtBr4VkSmUHMgjMfCuE9jjDEhFM4k\n8YvzisHT/7oxxphmJmxJQlXvBxCRFFUtCtd+jDHRrXrHjc29M879UdhaN4nICGfUpOXO/GAReSZc\n+zPGRCdVnaCqw1R1WGZmSDqTNU0onE1gnwBOBHYAqOpC4Kg6tzDGGBNVwtoLrKpuqFXk8rmiMcaY\nqBTOG9cbRGQkoCISj2d0pmVh3J8xxpgQC+eZxHXAjUBnYBMwxJk3xhjTTITzTEKrP3FtmoG8PMgI\nrCvrFmF/PW5jGiCcZxKzReRdETlZRHwMmuyfiMSKyAIRmVr/2iZkbr450hFExv563MY0QDjPJPoA\nY4ErgKdE5B3gFVVd2YBtK+9fBDTayqoN23nitRzmL9tIh3atuGjccMYfP8Tv+q4KN6+89CVTpyyg\npLiMkaP7cOPNJ9CufatAdhtSO37N5+4bXiY3s4j8Pgn06bmFgd23ERdXSu+0QZzS4VLef38NH37+\nEyUl5Rx9eG9uufwY2qWnBryv0r2vU1zwNOrazLL57Rn55o88Om4s142/iFYJLbtHlQq3myfmfcvX\nMz/jwzff5C8nHMmN515Mh5R9n/2Hk3N5+83vydtewOAh3bn+puPrrrPcxf8encYnb3xPSXEpo04a\nzLX3nUXbzIb/M96at4cnXprJt/NWk5aayNknHcJl40cQE9Ow31nvLl3M07mz2bAnn6GdOnPP6DEM\n7ngALpeb11/4ko8m5VK0t5QjjurLDbedSEYH37HN+fkVkuUxuibvanDsJnSiaUyMsJ1JOCPkfaaq\nFwBX4+kyfK6IfCkiI/xtJyJdgFOBFwLZ397iMm76x3vkLt2AW5VfdxTw6Ksz+eSbpX63een5HN58\n/TsK9hRTXu7iy1nLuPfudwLZbcjdfP5/+OGAEnYdnEjPrK0Mzl5HbFwxipuVBT/y1LJ7eeOj2RQU\nllBe4eLzb5dz1z8/CHg/ZXvfp3jzH6FgA1Lk4pCvVgOQ+O4b3PvpZCgs9Lzc7lAfYmS53VBYyNNf\nfcZL33/JUd/NAaD1+x9ww/sTq477s08W8tQT09m2dQ9ut7Jg/lruvG0i6vY/8PMrD0/l3We+oDC/\niIoyF19Omc9fr3oxgNCU2x54j6/mrsLlcrN7TzEvvvMdr70/p0HbF5SVcufM6azfk48CuVs2cfGH\nk9hRXMTrL3zJGy9+zZ78Yioq3Hwzcxn3/P7NymGDa1i3/Ueyku+jW8ouArsGYFqicD5M115EbhGR\nXOAO4CYgA7gdzwh1/jwB3AkE9O2U88PP7Nrj/WD3+zN/8rm+qjL1o/le5SuWb2Hlii2B7Dpkls1f\ny7Y9JeT39pzgZXfa6rWOK6GQdr1rjgC7ZOUWVq7ZFtC+SgvfIOm/BbQ5aDPpfTaT/OgeAG76YCaP\nn3EBmp4O//oX+PgSadZU4dFH+d1xp7D0lnu5fcoMAG6fMoNJF11bddzTPvT+t7F7114KCkt8Vut2\nu/lk4nde5cvnr+WXJRsbFNrCZRtZu3GHV/n7039s0PY7iou9ygrKSpmycjlTJ8/zWrb6560sXeQd\n28otj5EYY63VjUc4Lzd9j2f8iDNVtfq/xFwR+a+vDURkHLBNVec542L7VP0x/8zMTHJycijbU8Rl\nx3X2WjcxIY6cnByf9Yw7s4vP78BfflnE5i0r/O0+5AoLC8nJyaGosIRzru5HaftYEEjf3Y6YfO8A\new9IpKJnzY/ulxUL2byu4WM6uStORE86GjnSTcz6Cijft6w8Nhbp2ZO41q3h668bfVyBqv65duzY\n0e/nFrQxY1jTuyddtu8k3rXvy7A8NpaKrO4kp7flkLV59D/Ee8xwt7vcb1wnXz8QXyl15dolbNi+\nqt6wCotKueSUrl7lMSIN+lukKdzWsYtXeeqmLZz4my6+zxo2LGX7zl9qlJUUH828rcOrldg9m/1Z\nOJNEX/X1rxJQVX/jSowCTheRU4AkoLWIvK6qF9XafgKecbPp27evjhkzhs3b8xl/20u4a+3y8jMO\nZ8yYUT53NnPGu3z3Tc1bJOnpKUycNJ6EhHD+aWrKyclhzJgxlBaXcfaoB1hzQipFXeI4JHsNfTr/\nWmNddcUw560hlO/dd8+gXXoKk58dT3x8bIP3WVKwgJICT4e8ifMKSH4gv2rZ8xefy9W33hrkUQWu\n+uc6bNgwHTNmTNj29fZnH7Jl8tv8afLHVWWPnXcmN/zuXZLi4njp+RzeeePbGtvExAh3/vlw/MWV\n8+rzzPl8cY2ytplpvDr7AuIb8O+pqLiMM6/9L3uLymqUn3RUf65uwN9i0scf89jGX7zKPzn/Et77\n5ku+/Lzmpde01klMnDqexKSaPy5yf/mVXsn31bs/s38IZ+umDBF5REQ+FpGZla+6NlDVu1W1i6pm\nAecDM2snCH8OzGzDnVccR2K1/4wjBmdx6emH+d3mpltPpGd2h6r51m2S+dNfzmzSBFFdYnICd/99\nPJ2+KyZhl5vFa7uydfe+G4sJMUmc0v4qurU/sKosvXUy9906LqAEAZDY6jriEscCEP9JMZokfH9q\nb0rj47lwxZrQHFAUu2/kcZy5eCXF8fG8ftQRlMTHc+UvG0iK83z2v714FMMP71m1fmJiHLfecTJx\ndfydb3xwPFn99g3j3rpdKn98+tIGJQiAlOQE/nLzqaSlJlaVHdTrAH536dEN2r59SgonZfeumk+I\njeWe0WPol5HJ9bedRK9qsaW1TuKuv/3GK0EADMs+nx/yjqZC7YaECe+ZxBvA28A4PA/WXQpsD+P+\nOPOYQRx7WB8Wr9pCx3ZpZHetu+17ZofWPPfSVSxbupniolIOHtSNhMTIJIhKo04axLSj+zF9ci6r\nyvYwuNPZZHeCUi2ge0pfkmJTOOpfypKVWyguLWfwQZ1JiA88ZpFkWrV/Gdfar5DSy1g87THodyix\nebtJvOgi2LwZDjyw/oqaqY678+mYmMyyzz6le3ZPJG83bS65pOq4k5Li+ccjF7Bm9Ta2b9vDQQM6\nk5aWXOdln8wD2/LMjD+yfP5aSorKGHBYdsD/nkYNy+aDCdexcNlGWrdK4qBenerfyCHAsyefzi+7\ndrA+P5/BHQ+gXXIKAO0yWvHM/65m2eKN7C0s5eAh3XwmiErHD3qdjTuW8svWKcDdAR2DaVnC+Y3Y\nXlVfFJFbVPVL4EsR+aGhG6tqDpAT6E5bpyYxcnCPBq8vIvQf4H0vI5ISkxM4/cKRfpeLCAP7huYL\nPFa6w9wlHJyc7Ck4sBvMmQPbw5rPI6+iAubM4aDK4+6Cz+Pu0bMDPXp28N7eDxHhoKEN//fnS1Ji\nPIcPaXwd2W3bk922vc9lBw30vmfhT5f2/enSvj+WJPZv4UwSlbdCt4jIqcBmoF0Y92cao3t377Lk\nZGjp/f7vr8dtTIDCmSQeEJE2eJq8PoXnwbjfh3F/xpgot2hTvteDYpF6SMw0TDhHpqvsUiMfOCZc\n+zHGGBM+IU8SIvIU+GwuDoCqWqNrY4xpJsJxJpEbhjqNMcZEQMiThKq+2pD1ROQpVb0p1Ps3xhgT\nOmEdvrQevh+DNsYYEzUimSSMMcZEOUsSxhhj/IpkHxTWMYwxxjQBX4MYNVTYzyREJMXPoifDvW9j\njDHBCeegQyNFZCmw3JkfLCLPVC5X1VfCtW9jjDGhEc4ziceBE4EdAKq6EDgqjPszxhgTYmG93KSq\nG2oV2ZiIxhjTjITzxvUGERkJqIjEA7cAy8K4P2OMMSEWzjOJ64Abgc7AJmCIM++XiCSJyFwRWSgi\nS0Tk/jDGFxYulztkdbnVjVuDr8/ldvsc37ihKtzeJ4CqGtJjjSauiqY94fX19w2F6p+RSxv+b6Dc\nZSf8Zp9w9gKbB1wY4GalwLGqWuicfXwjIp+o6uzQRxhaGzbu5N//+Yx589fSunUyZ585jIt+OwKR\nwIx0idsAACAASURBVFv6FlWU8PTPH/LF1vkAHNvxEH7X+0xS45ICqufXPQX89ZNZ5KxcTUpCAucP\nPZhbjx1FXEzDfhss3r2Jhxd/zMJdG+iY1Jqrex/NWQcewiv/+pRP351LSXE5I47rzw3/d0bAxxiN\nZk5ZwGtPzuDXDTvpNaAzV991KoMOzw7b/ubmreZfS6ezLH8LXVPacUPfYzm1y6Cg6y0vdzHhxRw+\n/vQnShJLST/Hxe52u0iNS+KMzqO4rOeJxIr3v4FZK1fz6Bdfs2r7TrIz2nHbcaMZ2zd8x2+ah7Al\nCRH5J/AAUAx8CgwCfq+qr/vbRj0/dQqd2Xjn1fifwE2kvPz/2zvvOKmq64F/z85WtsLu0quwgtio\nFjBKjC1WLET9aewSe4yJiSHGnxqNGjWaxJifNRg0UUTpKBIBQVB6W5q0BRYWdtned2fm/P54b3dn\ndsrW2YL3+/nMZ96775Zz587Meffc+85x8ehjH3E0uwiAwsJy3n1vOdHREUy6dmyT6/vTjul8lb2p\n9vzzrDWUOit4+tTbGl2HqnL3v2fxXfYxAIorK3lr5VocYWH84vyGPaLkV5bys2/eo9hZAcDRiiKe\n2TKXdfO3s/mfW2vzrfginezDBY2Wq6OyYeVuXnr0o9q77d1bD/HE3f/kjQWP0KNv68fKyizL5/5V\n71PpdgJwsCyPKRs+ITU6jjNSTmigdHDeeHsJn8xcByj8Tx553ayZQbGznPf3/xeHhHHbCRd7ldlx\nJIcHps/F6bZmHnuO5fHQ9Ll8fNeNLZLF0PkJpbnpIlUtwopxnQEMAR5tqJCIOERkI5ANLFLVVSGU\nsVVYvXZvrYLwZO78jU2uq7CqhOXZm33SV+Skk19V3Oh6NmRm1SoITz5c51u3PxYeTq9VEJ4srd7t\nk7YrPbPRcnVUPp++ysccU1lRzZez14ekvbkHN9YqiBoUZcb+dS2q1+VyM/8ze4z7VUOKr+lo7uFv\nfNJmbEyvVRC1dakyY8NWn7yG7xehXLiuqfsy4GNVLWyM6UVVXcAIEUkCZorIKaqa7plHRCYDkwFS\nU1ODBqdvC4qLK7hxkm84zPBwR6NkKykpqc3nVBfXl47wm2/titVEiKNRMpVUVvFQmm/sbhEaJVN4\nZQn3uIf5lu8D4Q/5Tu4+bwV/vp7j2qNHjzYd1wEjo7lmiK+pJzKlzEsOz7FqCd0qiv1+vvFHG/ed\n8UdJSQnLli3j6iv6AApRCqW+SiKsTHzaGFRZ7Pf7klRV4pNm+H4RSiUxT0R2YJmb7hWRVMD31jQA\nqlogIkuAS4D0etfeBN4EGDp0qE6YMKHVhG4OhUXlXH/T61RWet8ZXn3VKBoj29KlS73yTV7zCruK\nve/OB8f1ZvIZNzdapopqJ+e9+hYF5d4f+WUnD+W+Rsi0rziHiUtfQ+tZ+07N6EbeW1leaV1T4hot\nVzA8x3XMmDFtOq6ffbSKac9+6pP+5+n3cdKIuhuA+mPVXDbk7efWFe/4pD91ykQm9B/VrDprZFu0\neAbfrNoDEQr3H4No7zG8sOdo7ho+wStt8c49PP3RHJ86//6TK5oli+H4IWTmJlV9DBgHjFHVaqAU\nCLrCKSKp9gwCEYkBLsR+Yrsjk5gQw5TfXE5sbFRt2sgRA7jjtuY9Ozhl+I30jkmuPe8dnczvhjdt\nD0B0RDivXnsZ3brE1Kad1rsnUy4+r1HlB8WnMuXUy4gOi6hNu6jXybx00y0MHt67Nq1rShy//UtT\n9yd0PC6eNJaLrxtbu9EgPMLBbY9c4qUgWpOR3Qbw4LAfERFmzQzDEK7pP5qr+vmfRTaFXzx0EUMG\nd4dqgdkJSEXdz/zkhAHcN+RKnzLnDx3MXePG1G5qcIhw+1mjuGDYkBbLY+jchNrB3zBgoIh4tvOv\nIPl7Ae+JiANLgU33iJXdoTn3nKGMHT2IbdsP07VrLCcMSm12XQNjezLtrMfYWrgfUE5OHEiYn90o\nDXH2Cf356uG7WHfwMAnRUZzcq0eTyl8/8Ax+3PtUthYeondMVwbEWYrrtVk/57stmZSXVjJ81AAi\nItvTT2TrEBYWxsN/vI4b7j2fQ/uPMXh4b5K6tc4MKRB3p53HNf1Hs7PwCAPikunTpWur1JuamsBb\n/7idHTuzqKio5sSTurOj5CDxETGkxfcNWO7RC37ALWeOZFd2LmmpyfRICG3/DZ2DUO5umgYMBjZS\n96S1EkRJqOpmYGSoZAo1MTGRjB41sFXqCpMwTk0a1OJ6IsPDOXtQ/2aXT4iM4exU37vJE08N/GfT\nmenZrxs9+7X+bqZAJEfFMa57aO7Whw3tVXs8qltao8r0iI+jR7xRDoY6QnkLOAYYri15istgMBgM\n7Uoot8CmAz1DWL/BYDAYQkwoZxIpwDYRWY31JDUAquq7amYwGI5bPLc2OxKav1bXkagfxCfj+cva\nSZLQE0ol8WQI6zYYDJ0Ez63NUb3SjPm5kxFK301ficgAIE1V/2tHqGvck2AGg8Fg6BCEMjLd3cAM\n4A07qQ8wK1TtGQwGg6H1CeXC9f3AeKAIQFV3Ad1D2J7BYDAYWplQKolKVa2qObEfqDP2SIPBYOhE\nhFJJfCUiU4AYEbkQ+BiYG8L2DAaDwdDKhFJJPAbkAFuAnwELgMdD2J7BYDAYWplQ7m5yA2/ZL4PB\nYDB0QlpdSYjIFoKsPahqy+MzGgwGg6FNCMVM4nL7/X77fZr9fjNm4dpgMBg6Fa2uJFR1P4CIXKiq\nnh5dfyMi67HWKgwGg8HQCQjlwrWIyHiPk3Ehbs9gMBgMrUwo/7TvBF4XkQwRyQBeB+4IVkBE+onI\nEhHZJiJbReTnIZTPC1WlqqI6aB63Njr6ql+c1S5cLnfQPJVVTr/pTrebardvvOL6VFVUNZinstq3\nDdXKgNfq41al0tVwvpbhbZmsrnbhdvtaK91upaoBmStaWdaqKidud/BxbC6qFVRXOf1+T6qdLlzN\naLfS5SSQx363201VZeDvfUu/84bOTyh3N60DTheRRPu80PO6iNyqqu/VK+YEfqmq60UkHlgnIotU\ndVugdtxawdacKQxNnkJ4WPOCpSx8fwUfvDSPnEP5DD61H5P/cB2njR9aez2vdA5ZhS9S6dxPdEQa\nfZJ+R2LM+Y2uvyCvlL//cS4rF28jPNzBj64YweRf/ZjomMjaPIu/3sFb05ZzKKuAAf2SuefWcxl/\nxhBKq6v4w/pFzM7YilvdXNxvGE+NuYiuUV282vjqk1VMfWoGh/ccpf+w3tz5h+s561Lv+E2rtu7n\nLx99xa7MY/ROSWDyVeO4dPQBtPjP4MrgYH4yrywcy568Edx/xXguHjvUq7yq8trmb3h3+1ryK8s5\ns0c/nj7zQoZ2bX3PnkWV29hw5B6SXD/ntTfTWbsxg9guUVx96Uju+J/xiAhvzl7Jx4s3UlxWydiT\n+vPoTeczsFddwKBlh/bxx3VL2JGfQ9+4RB4ZcQ7XDD6l2TId2JvNof3HuPKhJ4lPjOGq/zmbm+75\nYW3I05agFQtx5v8Jhxwka18XPvi/YaQMuI7bf381uSVlPPefxXydvo/oyHCuGncKD1/zAyLCg7tC\n23DsEE+vW8Sm3MN0j4nj3uFnc+vQsVZ7qnzw/Gxmvb6Q4rxSTj/3JB545Rb6D+sDQHHFt2TmP015\ndXqwJgzfA0Ju/lHVwvoKwsZnlqCqWaq63j4uBrZj+XwK1gKHS2ay7djvmyXf6kVbePUX08g5lA/A\nni0HeeLG18g5lAdASeUaMnIfpNK5H4CK6l3szbmb8updjW7j2V/9h+VfpONyuqmsqGbBx2t4/fm6\nqKwVFdU8/dI8DmUVALD/YC6PPzebvftzeHzNZ0zfs4lKl5Nqt5t5+7fx8IrZXvVvX72b5297ncN7\njgJwYMdhnr7xr2RszazNk5ldwCN/mcWuzGMAHD5WxPSF7+HOfxhcGQD07ZrLC5O+ICZsD1P+uYCN\new55tTN1xzpe3ric/MpyAFYdPchPF02nwhl8BtZcjpV/xerMyazZsA9VKCmtZNrH3/L+x9/yr8/W\n8M7cVRSVVqIKq7cd4KE/f4rTac229hXlcdfiT9iRn2P1v6SQX349n5VZ+5slS1WVkyk/m0pZqTVT\nKy4s5/1/LObTaStb3E+t3oIWPIxDDgLQd1AZv3p2AxsXz+Rfz83h4ddns2zLXtyqlFVW858lG/jb\nrK+D1plXUcatSz5kU+5hALLLS3hq3SLm7bfutz756+dMe+ZTivNKAdi0bDu/vfJFqiqrqXJmsSfn\nVqMgDED7rhEEvf0SkYFYoUxXNaay7NJFVLsKmizE59N8f2yV5dUs/ng1AMdKPqS+6UOpJq/k40bV\nf2j/MbaszfBJXzJvExXl1h9OQVG5jynF5XIz878bmb9/u0/Z5Uf2kVlS19fPp37lW97pYuG0ZbXn\n81duo8rpba66YvRORLzNF+EON1eM2IEqzFzh/Sfxn+82+ciSXV7Cl5l7fNJbi27dj9FnYI5X2pwv\nNjNr2RafvFm5RXy71VICn+5Jp6qeeU6Bj3ZtbpYca7/+jmNHi3zSP5uxpln1eaJlM6iL8GvhCFcu\nvDqLmbNXsTMzx6fMzBXpAU1IAPMObKOkutIn/T+7NwDw2dSlPteOHcpj7RebySubiVvLm9YJw3GL\ntFd0URFZr6qjAlyLA74CnlXVT/1crw1ikpqaMvqd958EIC5yKNJEC9qhvdmUFfvaXbv1SCS5ZyJV\nzgM43b4ToQhHMhGO3g3WX1lRzYG9vj9ygMHDehEWJuTlF5KT6ytDQmI0uQ7/P9a0xBSiHFZfj2Tk\nUJxf6pMnMSWe7v2SAcjOLyG3yDtP767FJHbxbbegLIasgngSukTRJyWxNn1nQQ5VLt91kb5xiVxz\nyaXrVHWMX2Ebide4dk8a/c60ZwDIzU6gsqLONOdwCBouOP3Y5/umJhHfJYqssmKOlft+JgmR0QyI\nT2qybMWF5Rw5lE/X1Gjyc+o+s/AIB4PSejS5Pi9ch0DzfZKL8iM4mhOLMy7S55oAw/p7+8ssKSkh\nLs4yuR6rKOVIWbFPuZjwCAYnJLMv/SDOat+x7DkwleiEEqpd2bVpl138UIvHtoaoXmna69ZXvdJa\nO2BP/YBAbdFGR++Dv/r2v3B5o8Y1lEGHGsLvTEJEIoBPgA/8KQjwDmKSdmJ/jU57jcSo0zmj9/3+\nsgdlwf5lvP/Mv33S//bfKQw5vT95pXPIyPX1JpLW/UPio8c1WL+qcsflr5B1MM8rfeRZg7n7vh8C\nMG/+Qj6Y851P2ZefmsTs3OWszj7olT4kIZkvfnRD7fnyWWt45v6/+ZT/45xfM3qCZYPfvPswd/7x\nQ6/r5560jxd/utCn3IPvX8Y3e/rz3J2XMmFM3brEijWLeWub951zZJiDby5rnWCDnuN64qkxGp32\nGmUl0bz3j+txuers75dfdBqaGM6Mr7xnNl2iIpj/8kTiukSx5mgmv/38A582/nzOZUxoxrpEcWEZ\nN1/4IpfflsYn/1e3RDbxprOZMGFCk+vzRCv+ixY85ZP+v/edRkzqJSxNKuVYUZnXtQtGpXFPvXaX\nLl1aK8u+ojwunP8G7no3gY+NOJ8Jw89i+/z3mfX6Yq9rUTGRvP/dq4THHWDHkUta1CfD8UN7mptW\n1E8QawXwHWC7qv65sRXFRgzm5NTnmiXExTefw8U3jScszNJZUTERTP7DJIac3h+AbrFX0j3+Tmr0\nqRBFr8RHGqUgAESEKS/eQPfedXevg07sycNPTqw9j4uN4vqJY3A4rOGICHdw2w3jGDtyIH8663JO\nTEypzds/Lom/jL/aq40fTBzLtQ/9GIe9kBkRGc7NUyYy+kd1f4anDenNg5N+QFSElccRJqT2nAhd\n7qrtW5UzjLeXjWbNvoHc+MORXDzGe+H64RHn8MM+J9SeJ0RG8ZdzryA52nsRvbWIdKSQ4n6KuNi6\nDQkjT+3Pvbedx/3XnsOZJw+oTU+Mi+aZn11GXJcoAMb26MuvR51XO9tyiHDL0FFcfcLJzZIlPrEL\nj73wk9oxAhg9Lo1bHrigWfV5ItEXQOzduNWStbpa+OjNAZRVnMF9z17PC3dfTkpibG3+Uwb25Nc/\n+WHQOgcldOPZsT8mNtyahQhwxYDh3D7MWri+9YlrGXNhnfOD+G6xPPbPe0noFkeXyOH07fokYRLT\n4r4ZOj8hMzeJSBJwCzAQjxmLqj4UpMw5wHIsp4A1toQpqrogUJmhQ9N0x47vWrzDJDszj6yMHE44\npS/xSbE+16ucR6h07iU6YigRjuQm1+9yudm5JZOISAdpw73X4mvuAI/llXAgM49B/ZPpWk+GzblZ\nuNTN6cm9CQvQ19ysAg5+d5iBJ/UlqXuC3zwFJeXsPphDvx5d6dEtHgB15YBzD4WVfdmVpQzs0ZXU\npMA7xfYU5pJTXsqIlF5Eh0cAICKtZpIAGDl6uK5bu4kwiaCq2sm2nVkkJsQwqH+KV759h3PJLy7n\n5BN6EhXhOzHOryhne342JyR0o2dsfIvlWrJkCcnxA+maHEe/Qa27q0tdObgqd7FrSzjRcb0ZdHLf\n2mvVLhfp+44QGx3JiX39t+s5k6ihuLqS9Lws+sYm0S/O18x2YOdhCrKLGDrmBKJivM1aTnch5VVb\nSYgZb8xNDbTR0fvQUc1NC4Bv8f7DD4qqfk0DC9q+hLXKFsTufbvRvW+3gNcjw3sSGd6z2fU7HGEM\nH9E/aJ6UbnGkdPP/53xacq8G20julURyr+D29qS4GMac5C2HOFLBkUpSFIz1r1u8GJyYzODEpivK\npuCQLoSJpYAiI8IZcUo/v/kG9U5mUJB6ukbHMK7XgCA5moaIcNqYYC22oG5HKuFdUjnpTN9rEQ4H\nI4c0sNHPD/ERUZzdY2DA6/2H9qb/UP9ra+FhiY2eMRuOX0KpJKJV9ZEQ1m8wGAyGEBPKNYlpInK3\niPQSkW41rxC2ZzAYDIZWJpQziSrgReB31D1ooMAJAUsYDAaDoUMRSiXxS2CIqh4LYRsGg8FgCCGh\nNDftBsoazGUwGAyGDksoZxKlwEYRWQLU+gcItgXWYDAYDB2LUCqJWfbLYDAYDJ2UULoKr+8G3GAw\nGAydjJApCRHZh5+Y1qpqdjcZDAZDJyGU5ibPx72jgUmAeU7CYDAYOhEh292kqrker0Oq+irQug5O\nDAaDwRBSQmlu8owVEYY1s2hP1+QGg8FgaCKh/NN+mbo1CSeQgWVyMhgMBkMnIZRK4sfAtXi7Cr8B\neDqEbRoMBoOhFQn1cxIFwHrAN0amwWAwGDo8oVQSfVW1yTEQReRd4HIgW1WbHmfSD6XOKqId4Tik\n6ev0ZcXlRMdGERbWemv8qk5UKwkL8w1u5D+/4tQyIhrIX+2uIEzCcUjwYVVViisrCA8XuoRH16a7\ntBq3uokIi2qUXKHErYrL5aKqvIqYuNaNkFbpKiMyLBrx831waRWo4mjGZ+DWClzuKsIkGkeYb1zq\ntqCkupIu4ZEBA1M1lTKnub/7vhNKJbFSRE5V1S1NLDcVeA34V0sFSM/P4skNC9icf5ikyBhuTzuT\ne4f9oFFl1y3axP898h4ZWw+S3LsrP31iEpdNvrBF8qgq+UUvU1TyNm4tJCpyLClJwcOu7i9eQHre\n65Q5s4iLGMDpyT+nd+x5XnmKqrJYfvQlMsvWES7RDEu6jLNS7/WrLOZv2sFr22YgPXJwRLjpFd6L\nR0+5gd3Fc9hWuBi3uhgcfyYX9XqAuPD227G8/XA2Z93xLHFzvuOUYf158LU7OXH04BbVebA0nS+P\n/oPsir3EOrpyZspPGJNshZGtdpewPucFDpZ8ASh9Yn/IqNTfEuUIHsQJwOXO52jeI5RXLERVOepK\npDryBsb1/DWRYaEJ7Vqfldn72F2cw+Q5f6J7dBz3n3QuN54wutn1bSrYydt7PiWj7HArSmnojITS\nwd85wDoR2Skim0Vki4hsbqiQqi4D8lraeEl1JXd8/QGb860veUFVOa9sXcqHe9c1WDZr31GeuOoF\nMrYeBCD3cD6v3vMmq+Y3XDYYRSVvUVD8Mm4tBKCyag1Hjt1IoMB9OeXrWZ39BGXOLLtP+1l55FEK\nq/bU5lF189mhX5NZthZQnFpOev4M1hx726e+jQcO8+yaDwnvexRHhNVmljOLt3c/zpaChbi0GsXN\n7uJvmH3w2Rb1tTUo7RdP7k9OYvuqXfz2kmcpKy5vdl0lzjxmHPg92RV7rbpd+Sw++gbbC78CYG32\n0xwoWYDiRHGRWfpfVh19vFF1Z+c9QEXF5whKmECv8EKiqj5g+ZGXmy1vUzhcVsg9Kz+kwuW05Kko\n4X83LODLwzubVV9OZT5Pb33DKAgDEFol8WMgDbgIuALLhHRFCNvz4ovDOyio8v1TmZ6xocGyiz/4\nmqqKap/0z95d3CKZikr/7ZPmcufgdhf7zb+veDb1H1pXXOwvnld7fqR8CwVVB3zK7ijwjWk7c/1W\n4vsUeaUJbrpG5vvkPVS+jdzKg37lakucqV2o6hNPUW4xK2atbnY9OwqXUa2+ppMtBQupchWRWbrE\n59rR8m8ocx4JLp/rCOUVvt+L7o4SDhQvospV2myZG8vcg+m1CsKTGRkbm1XfV9lrqXL7fv8N309E\n1cdzRrsjIgOBeYHWJERkMjDZPj0FSA9QVSJQGCSt5tjfew31y4eCFCBY3I36/WhKHwr9lA8FQ1U1\nviUV1BvXoUDzboVDS0Nj1Z6ESrYBqpra3MLtMK4deYwaS1v0oXHjqqod7oW1bTa9kXnXBrn2ZrC0\nmmN/7zWvNupvwD7460dT+hDoc2jrPhwvr47cz44sm/kcOm8fjvcnoOc2kDa3gfeOQn15mtqHjtYf\ng8HQSehw5iYR+Q8wAWu6dRT4X1V9J0j+tao6JtD1zoDpQ+ehI/ezI8vWlhwPn0NH6kOHm0mo6o1N\nLPJmSARpW0wfOg8duZ8dWba25Hj4HDpMHzrcTMJgMBgMHYdQboE1GAwGQyfHKAmDwUZE+onIEhHZ\nJiJbReTndno3EVkkIrvs9652uojIX0Vkt/3A6KjgLbSKjA4R2SAi8+zzQSKyypbhIxGJtNOj7PPd\n9vWBoZatPQk0dp2R+mPc3hglYTDU4QR+qarDgbOA+0VkOPAY8KWqpgFf2udQ98BoGtZzAP9oAxl/\nDmz3OH8BeEVVhwD5wJ12+p1Avp3+ip3veCbQ2HVG6o9xu3JcKwkRCRORZ0XkbyJya3vL01xEJFZE\n1orI5e0tS3MQkYki8pZ9Z3tRe8sTCFXNUtX19nEx1g+1D3AV8J6d7T1gon18FfAvtfgWSBKRXqGS\nT0T6YkV3fNs+F+B8YEYA2WpkngH8yM5/XBJk7DoV9ce4I9BhlYSIvCsi2SKSXi/9Etsf1G4ReSxQ\neZurgL5ANZAZKlkD0Up9APgNMD00UganNfqgqrNU9W7gHuD6UMrbWtjmmZHAKqCHqmbZl44APezj\nPoCn75JMQvvH9Crwa+qcfSUDBapa45PDs/1a2ezrhXb+4556Y9fZqD/G7U6H2wLrwVTqeYMVEQfw\nd+BCrB/EGhGZAziA+u5U78ByAbBSVd8QkRlYpoK2ZCot78PpwDYgmvZhKi3sg6pm28eP2+U6NCIS\nB3wCPKyqRZ434KqqItLmWwLtWWS2qq4TkQlt3X5nof7Ytbc8TaGjjnGHVRKquszPYtsZwG5V3Qsg\nIh8CV6nqc1gOBL0QkUygyj51hU5a/7RSHyYAscBwoFxEFqhqm91ltFIfBHge+KzGJNBREZEIrD+Z\nD1T1Uzv5qIj0UtUs25xUo/QOAf08ive100LBeOBKEbkU64YhAfgLlokr3J4teLZfI1umiIRj+e/K\nDZFsHYIAY9eZ8BljEXlfVW9uT6E6rLkpAE2d3n8KXCwifwOWhVKwJtCkPqjq71T1YeDfwFttqSCC\n0NRxeBC4ALhORO4JpWAtwVZm7wDbVfXPHpfmADVrWrcCsz3Sb7F3OZ0FFHqYpVoVVf2tqvZV1YFY\nYYAXq+pNwBLgugCy1ch8nZ3/uH0oKsjYdRoCjHG7KgjowDOJ1kBVy6jb7dGpUdWp7S1Dc1HVvwJ/\nbW85GsF44KfAFhGp8bM9BWsWNF1E7gT2Az+xry0ALgV2A2XA7W0rLmCtV30oIs8AG7D+KLHfp4nI\nbqz4LDe0g2xtid+xU9UF7SjTcUFnUxJtOb0PFaYPHRRV/RoItAPoR37yK3B/SIXyg6ouBZbax3ux\nzH/181QAk9pUsHakgbHrdHiOcXvT2cxNa4A0+wGiSKy7ozntLFNTMX0wGAydhg6rJMTyBvsNMFRE\nMkXkTntx7gFgIdY+6OmqurU95QyG6YPBYOjsGAd/BoPBYAhIh51JGAwGg6H9MUrCYDAYDAExSsJg\nMBgMATFKwmAIgogMrO+3qjMhIk+KyK/aWw5D58UoCYPBYDAExCgJQ6fAvqPfbrsc3yoiX4hIjIgs\nFZExdp4UEcmwj28TkVliBQnKEJEHROQRsYK5fCsi3YK0NVpENonIJjwelhMrGMyLIrJGrCBDP7PT\nJ9hyzBCRHSLyge0mAhF5XqxAOJtF5CU7LVVEPrHrWSMi44PI8qTtiXepiOwVkYc8rj0iIun262GP\n9N+JyHci8jWWk8ua9MEi8rmIrBOR5SIyzE6fZNexSUQ6ivsaQ0dBVc3LvDr8CxiIFVhmhH0+HbgZ\n66nUMXZaCpBhH9+G5S4jHkjFcpV9j33tFSwvoYHa2gycax+/CKTbx5OBx+3jKGAtMAiYYNffF+vG\n6xvgHCzX3Dup22qeZL//GzjHPu6P5W8okCxPAivt9lKwnPRFAKOBLVjOH+OArVjusWvSu2A5AdwN\n/Mqu60sgzT4+E8s3EHb+Pp4ympd51bw6m1uOkCAiJaoaF+I2rgSGq+rzoWwnQNsTge9UdVtbt93K\n7FPVGr8867AURzCWqBWAplhECoG5dvoW4DR/BUQkCeuPsuaOehpWBDqAi4DTRKTGoV4iVlS6JlSu\npwAABG9JREFUKmC1qmbadWy0ZfsWqADeESsUZU04yguA4VLngjxBROJUtSRAP+araiVQKSLZWPEs\nzgFmqmqp3eanwA+wlNRMtfyWIZYL9xoX2uOAjz3ajbLfVwBTRWQ6llNMg6EWoyRaERFxqKpfl+Sq\nOocQuq4I1jZWtLJ5WHEpOjOVHscuIAZrdlFjNq0fc8Mzv9vj3E3zvvsCPKiqC70SLXfu9WULV1Wn\niJyB5ffpOqyn1M+35T1LLf9KjcGn7mbIHoYVoGhE/Quqeo+InIkVEW2diIxW1eParbih8Zg1iXqI\nyKMeNuenPNJn2bbcrSIy2SO9RERetu3XZ9v276dEZL2IbPGw+94mIq/Zx1NF5K8istK2M19np4eJ\nyOu2XXuRiCzwuGv1J2uGiLwgIuuBSSJyty37Jtvm3UVExgFXAi+KyEbbLu3XNt1JycAysUCdy+xm\no6oFQIGInGMn3eRxeSFwr1hxCxCRE0UkNlBd9t17olqeSH+BFUAK4Ass9+k1+Xz+uBvBcmCiPcax\nwNV22jI7PUZE4oEr7H4VAftEZJLdpojI6fbxYFVdpapPADl4O280fM8xMwkPxIq/nIblVVOAOSJy\nrm16uENV80QkBisS2yf23VYssEpVf2nXAXBMVUeJyH3Ar4C7/DTXC8tkMAxrhjEDuAbLTDEc6I7l\nF+ndBsTOVdVRdtvJqvqWffwMcKeq/s02OcxT1Rn2tS+x7PO77DvI17HucDsjL2G58Z4MzG+lOm8H\n3hUrAt0XHulvY43PerEGOoe6mNL+iAdmi0g01vfpETv9IeDvIrIZ6ze4DCu0a6NR1fUiMhVYXSOb\nqm4AEJGPgE1YwZHWeBS7CfiHiDyOta7xoZ3vRRFJs2X80k4zGADjuwmoW5MQa/fJdUCBfSkOeE5V\n3xGRJ7Hu1sD6o7hYVb8VEScQVWPqEWt3zXhVPWT/AT+rqheIyG1YC6wP2D/uRar6gV2mWFXjReRV\nYJOq/tNO/xT4d82fux+5M4DzVHW/fX4e8AyQZMu+0DYlTMVWEvbdbQ7WgmoNUap6UvM/QYPBcLxi\nZhLeCJZSeMMr0bI5XwCcraplIrKUOvt3hZ+1gBobcjD7saeduSV+8Es9jqcCE1V1k62UJvjJH9A2\nbTAYDPUxaxLeLATusO+2EZE+ItIdaxdLvq0ghgFnhaj9FcC19tpED/z/yQcjHsiybeaetvRi+1pQ\n2/T3DRH5u71O4/lqj+hyiMjtfmT5e3vIYjB4YmYSHqjqFyJyEvCNvbZQgrUX/3PgHhHZjmWm+TZE\nInyCtRNmG1YM6fVY++8by++BVVjmpFXYigHL9vyWWA9iXUdg2/T3ClVt86hygbBNjP9sbzkMhvqY\nNYkOhtj75UUkGWtRcryqHmlvuQwGw/cTM5PoeMwT64GuSOAPRkEYDIb2xMwkOgEiMhPL/YMnv6n/\nUJfBYDC0NkZJGAwGgyEgZneTwWAwGAJilITBYDAYAmKUhMFgMBgCYpSEwWAwGAJilITBYDAYAvL/\nRTV5kc06i7IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc28784fa58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plot_evaluations(result=search_result, dimension_names=dim_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate Best Model on Test-Set\n",
    "\n",
    "We can now use the best model on the test-set. It is very easy to reload the model using Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = load_model(path_best_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then evaluate its performance on the test-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 8960/10000 [=========================>....] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "result = model.evaluate(x=data.test.images,\n",
    "                        y=data.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print all the performance metrics for the test-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.0363312054525\n",
      "acc 0.9888\n"
     ]
    }
   ],
   "source": [
    "for name, value in zip(model.metrics_names, result):\n",
    "    print(name, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can just print the classification accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc: 98.88%\n"
     ]
    }
   ],
   "source": [
    "print(\"{0}: {1:.2%}\".format(model.metrics_names[1], result[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict on New Data\n",
    "\n",
    "We can also predict the classification for new images. We will just use some images from the test-set but you could load your own images into numpy arrays and use those instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "images = data.test.images[0:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the true class-number for those images. This is only used when plotting the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cls_true = data.test.cls[0:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the predicted classes as One-Hot encoded arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = model.predict(x=images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the predicted classes as integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe8VNW5//HPg2IQVBCxU86NWECjmGCvV5EoFpQYxViI\nsUQxlngDSYwSTJAYbPwsV1FfoglWEBGJimJXQISASrGAokEvIqJGiWJ7fn/MXjN7Tp19ph++79eL\n19kzs8uj68w6z1p77bXM3RERkdy0KncAIiLVRJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpT\nRCQBVZoiIgmo0hQRSWDdfA7u1KmT19TUFCiU6jBnzpyV7r5pueMoFZVxy6cyTiavSrOmpobZs2fn\nc4qqY2bvlDuGUlIZt3wq42TUPBcRSUCVpohIAqo0RUQSUKUpIpKAKk0RkQTyunsu0lxXXnklAF98\n8QUAr7zyCgATJkyos+/ZZ58NwF577QXAySefXIoQReqlTFNEJAFlmlJSxx9/PADjx4+v93Mzq/Pe\nTTfdBMC0adMAOOCAAwDo2rVrMUKUMnrjjTcA2H777QG49tprATj33HPLFlNtyjRFRBJQpilFF7JL\naDjD3GGHHQA49NBDAXjrrbfSn02ePBmAxYsXAzBu3DgALrroosIHK2U1d+5cAFq1SuVzW2+9dTnD\nqZcyTRGRBJRpStGE55kfeOCBOp/ttNNOQCaL7NSpEwAbbLABAF999VV63z322AOAl19+GYCPPvqo\nSBFLuc2bNw/I/B4MGDCgnOHUS5mmiEgCJc80wzi8W265BYCtttoq/VmbNm0AOPHEEwHYYostAOje\nvXspQ5QC+b//+z8A3D39Xsgwp06dCsCWW25Z77FhHCfAokWLsj474ogjChqnlN+rr74KwHXXXQfA\nKaecUs5wGqVMU0QkgZJnmkOGDAFg6dKlDe4TxuVttNFGAPTs2bMg1+7SpQsAQ4cOBaB3794FOa/U\n78gjjwQyd70BNtxwQwA6duzY6LH33ntvejvevykt0+uvvw7A6tWrgewRF5VGmaaISAKqNEVEEih5\n8/zWW28FMsNH4k3vhQsXApkBrk8//TQAM2fOBDKPzb377rsNnr9169ZAZghLuBkRP09opqt5Xhrd\nunXLed8rrrgCyDxOFxeGHoWf0nKMGjUKSC29AZX93VSmKSKSQMkzzYMPPjjrZ1x4hC74+OOPgUzm\nGf76vPTSSw2e/3vf+x6QeeA/PJ4HsGrVKgC22WabZsUuxTNlyhQAhg0bBsCaNWvSn22++eYAXH75\n5QC0bdu2xNFJMcRvBofvdPjetmvXrhwh5USZpohIAhX9GOXGG28MwEEHHZT1fn1Zam33338/kMlW\nAXbeeWcABg4cWKgQpUDCI5fxDDMIw0/ClHDSMjzzzDN13tt008pfbl6ZpohIAhWdaTbHihUrABg8\neDCQ/Qhf6C9ramC1lM7RRx8NZB6rDAYNGpTeHjFiREljktIIS5zEhQdPKpkyTRGRBFpcpnnDDTcA\nmYyzQ4cO6c/CnTkpvzB+dvr06UCmLzP0aV188cXpfcM0YdIyzJgxA4CxY8em39t1110BOOSQQ8oS\nUxLKNEVEEmgxmebzzz8PZMbyBQ8++GB6O0xLJuUXJpdduXJl1vthWkCNpW25nnjiCSB7ZEsYox2m\nh6xkyjRFRBJQpSkikkCLaZ4//PDDQGbuxT59+gCw1157lS0mqSusCRQejQ0OPPBAAP70pz+VOiQp\nsTBZT9xPf/rTMkTSPMo0RUQSqPpM84svvgDg0UcfBTITdlx66aVAZqo4KZ/46pEjR44E6s7G3qtX\nL0DDi1qy5cuXA/Dcc88B2ZPpHHPMMWWJqTmUaYqIJFD1mWaYtDb0kR122GEA7L333mWLSbJdddVV\n6e1Zs2ZlfRYeo1RfZst3++23A/DBBx8Ame9qtVGmKSKSQFVmmmHCWoA///nPALRv3x6ASy65pCwx\nScOuvvrqBj8Lj72qL7Ple+edd7Jeh6kfq40yTRGRBKoq0wx3Yc8777z0e9988w0A/fr1AzQus9qE\nMs1llENoTYR9v/76awA+/fTTOvuGR/Suueaaes+1zjrrpLf/+te/AlpGo9geeuihrNdHHHFEmSLJ\njzJNEZEEVGmKiCRQFc3zb7/9FsjMhPL222+nP+vevTuQuSEk1SWs25SL4447DoAtt9wSyAxdueee\ne/KKIax2GZ/DUwonDGYP5VXtlGmKiCRQFZnmkiVLgMyKhXFhOIvmX6xc4SYdwKRJk5p9nvvuu6/J\nfcJNolatsvOBo446CoDevXvXOWbfffdtdkzStAceeADI3LQNs7RX6+qiyjRFRBKo6EwzDIbt27dv\n1vtXXnllertahy2sTSZOnJjeHjVqFFB3wo5g4cKFQOP9lKeddhoA3bp1q/PZT37yEwB69OjRvGCl\nYP7zn/8A8Mgjj2S9H6aBiw/7qibKNEVEEqjoTHPMmDFA3cev4n0hZlbSmCQ/ua5rfddddxU5Eim2\n0L8cVoTt378/AOeff37ZYioEZZoiIglUZKYZxnVdf/31ZY5ERJorZJphnfOWQpmmiEgCFZlphjXM\nP/vss6z3w9M/mkZMRMpFmaaISAKqNEVEEqjI5nltYaXCJ554AoCOHTuWMxwRWYsp0xQRSaAiM83f\n//73WT9FRCqFMk0RkQTM3Zt/sNmHwDtN7tiydHP3TcsdRKmojFs+lXEyeVWaIiJrGzXPRUQSUKUp\nIpJAo5WmmW1iZvOif8vN7L3Y6/WKEZCZ9YxdY56ZfWZmv2rimNPN7MNo/0Vm9os8YxhnZkc3sc/v\nYjEuMLNvzKx9PtcthzKVcTcze9rMFkb/7xot3+iYcpTxKWb2qpm9YmYvmNkP8rlmuZSjjKPr3hHK\nLMf9y1HGO5rZDDNbY2YX5HRid8/pHzAc+E097xvQKtfzJPkHtAZWAJ2b2O90YHS0vQWwEuhUa591\nE1x3HHB0gv2PAR4rxv+DUv4rVRkDWwG9ou2NgCXAdpVWxsA+QIdo+0jghXKXUbWUcXTOA4DdgXk5\n7l+OMt4c6A1cDlyQy3mb1Tw3s+5RlnAnsADoYmafxD4faGa3Rtubm9lEM5ttZrPMbM8ElzoEWOTu\ny3I9wN2XA0uBrmY2wsz+ZmYvALeb2bpmdnUUxytmdnoUYysz+18ze83MHgc6JYgR4ATg7oTHVLRi\nlrG7v+/u86LtfwOvAVvnGlupytjdX3D38N88E+ica4zVoNjfY3d/BljVnNhKWMYfuPts4JtcY8tn\ncPsOwCnuPtvMGjvPtcAod59pZjXAFGAnM9sDONXdz2rk2IEkrIzMrDvQDXgrFuf+7v6lmQ0GVrj7\n7mb2PWCmmT0G7An8F9CTVBa0ELgpOt9lpDKMhxu43gZAH+CMJHFWiaKXsZl9H9gJeCnXoEpdxpHT\ngEca+bxaleJ7nFiZyjgn+VSaS6Iauil9gO0tsyzFxma2vru/CLzY0EFm1gY4HLgwx3hONLMDgTXA\n6e7+SXTNB939y2ifvkAPMxsYvW4PbAvsD9zt7t8By8zs6XBSd/9DE9ftDzzj7p/mGGc1KXYZbwTc\nD5zr7p/ncJ2ylLGZ9QFOBlriWr9FLeNmKNf3OGf5VJqrY9vfkeoTCdrEtg3Y3d3rX36wYYcDL7r7\nyhz3v9Pd6+vIjcdpwGB3fyK+g5kdkzC2uIHA3/M4vpIVrYwtdQNiIjDW3SfneFjJy9jMegFjgB+7\n+8fNOUeFK/b3OKlyfY9zVpAhR1HN/rGZbWtmrUjdGAmmAeeEF9EvYS7q9BOa2flmlk8zYCowODRD\nzGx7M1sfeBY4PuoT2ZpUB3aTzGxjYG/goTxiqgqFLGNLpQ63k7pBcG2tzyqmjKNm6ATgZ+6+OI+Y\nqkKRvsd1VFIZN0chx2n+ltR/zHQgfuPmHGCfqMN2IVHfn5ntYWY31XciM9sQ+G9gUq2PegAf5RHj\nGOBNYJ6ZzQduJJVtTwDeJdUHMhZIL2piZpeZWb8GzvcT4BF3/yKPmKpJocr4AFJ/FA+xzNCXH0ef\nVVIZDwc6AmOiGAvZDK1UhfwejweeA3qa2TIz+3n0UcWUsZl1NrNlwHnA8CjOto1dvKoeozSzfwD9\n3T3nO11SXVTGLV+1l3FVVZoiIuWmxyhFRBJQpSkikoAqTRGRBFRpiogkkNcaQZ06dfKampoChVId\n5syZs9LXolm9VcYtn8o4mbwqzZqaGmbPzuUJrJbDzNaqZQFUxi2fyjgZNc9FRBJQpSkikoAqTRGR\nBFRpiogkoEpTRCQBVZoiIgnkNeSoVFavTs0/OmTIEABuuikzE1Xv3r0BGD9+PADdunUrcXQisjZR\npikikkBVZJrvv/8+ALfccgsA66yzTvqzMCj3oYdSk6f/6ldNLqEtFeCf//wnAAMGDABg6dKlzT7X\nY489lt7u0aMHAF26dGl+cFI24Xt81FFHAXDdddcBcPbZZ6f3iX//y0GZpohIAhWdaX744YcADBo0\nqMyRSKFNnToVgDVr1uR9rsmTM+uy3XbbbQDcc889eZ9XSuejj1KrX8QzSoBzzz0XgNNOOy393vrr\nr1+6wOqhTFNEJIGKzDSvvTa1QOGkSal11V566aUmj3nuuecACMt37LLLLgDsv//+xQhRmumbb1LL\nwjz88MMFO2cYQQFw9dVXA5kRF+3atSvYdaR4nn32WQDee++9rPdPOOEEANq0aVPnmHJRpikikkBF\nZpoXXJBaKz7JXbKJEydm/ezatSsA9913X3qfH/3oR4UKUZrpqaeeAmD69OkA/Pa3v837nKtWrUpv\nL1iwAID//Oc/gDLNShbvzx4xYkS9+5x88skAmFlJYsqFMk0RkQRUaYqIJFBRzfN+/foBmZs53377\nbZPHdOrUCcg0w955JzUh89tvvw3Abrvtlt73u+++K1ywkrNXX301vT1w4EAAunfvDsBFF12U9/nj\nQ46kerzyyivp7fCwQ7Duuqmq6bDDDitpTLlQpikikkDZM81nnnkmvf3aa68BmU7fhm4EnXXWWent\nvn37AtC+fXsAnnzySQAuu+yyOsfdeOONQN0BtFJc8bIIN2jGjRsHwAYbbNDs84YbQPHfoUq6YSCN\nCzdt63PIIYeUMJJklGmKiCRQtkwzTNAQ+rgAVq5cWe++YfjQscceC8Af//jH9Gdt27bN2jdMDTdm\nzJg65xw6dCgAX375JZCZ3KN169bN+4+QRk2YMAHIHsge+jLjfc3NFYapxLPLAw88EIAOHTrkfX4p\nrngLIVhvvfUAGDlyZKnDyZkyTRGRBMqWaX799ddAw9klZB6BvPfee4HMnfLGhEwz3JW98MIL05+F\nR+tCxhmmn9pmm20SxS65CRNDh//vUJj+5NBKueuuu4DMnVaAiy++GFDroZKFBxtmzJhR57PQcuzV\nq1dJY0pCmaaISAJlv3ten9DfNXbsWCC3DLO2kEXeeeed6fdmzZpVgOikKZ9++ikAM2fOrPPZ4MGD\n8z7/zTffDGSmDuzZs2f6s4MOOijv80txNTYBTzWMbFGmKSKSQNkzzfqe+nnxxRfzPm94qij+FFDt\nJ43CXfgwZlAKI0zEsGzZMiAzvVehLFmyJOv1TjvtVNDzS3HVl2mG0Q6FaIkUmzJNEZEEVGmKiCRQ\ntuZ5WLu8WCvLhVXt5s6dm36v9uOZl156aVGuvbbbcMMNgcywkfiEHeHRx44dOyY+74oVK4DMUKZg\nn332aVacUlrPP/88kBkqFhceg+7cuXNJY2oOZZoiIgmULdOcMmVKQc8Xhp8sXLgQaPwxrDCESQOg\niyOsFhgemQyPUwIcfvjhQPZDB/WZP39+ejvc+AnT/tWelKNVK/3trwZhxclwQzaukifoqE2/bSIi\nCZR9yFGhhOnHbrjhhgb3qampAeCOO+4AMhOBSHEMHz4cyM4sQgsjPlFLfTbddNP0dsgsG3rk9tRT\nT80nTCmR2n3R8UlVzjzzzFKH02zKNEVEEqj6TDMskREmMG5MeNxuv/32K2pMktKjRw8ge0XQMJqh\n9gD12sI0gHGDBg0C6j6MEPpQpTKFhxxq3zWP3ykvxFSBpaJMU0QkgbJlmo0tnvbII49kvT7jjDMA\neP/99xs8Ty7LHBT6jr0kt+uuu2b9TOL73/9+ve/Hx4H+4Ac/aF5gUjRhKrjad8379+9fjnDypkxT\nRCQBVZoiIgmUrXke5s0Ls6jHhQHQtR+xrO+Ry9C8z2XlSqluoXlXu5mnJnllC4Pag/BwyQUXXFCO\ncPKmTFNEJIGyZZoDBgwAYNSoUen3GlsvqCnhr1cY5nLLLbcAsOWWWzb7nFJZws0+rW1eXaZOnZr1\nukuXLkBmko5qo0xTRCSBsmWaYdXIsNIkwKRJkwAYPXp04vP94Q9/ADJrmUvLE9arDzSovbKFFWcX\nL16c9X6bNm2A6p0wR5mmiEgCZX+MMqxtHt/u27cvkFl1MEwofOSRRwLwy1/+Mn1MuJMaX5FQWqaw\nOmmY6GHYsGHlDEeaEKbsC49ILliwAIBtt922bDEVgjJNEZEEyp5p1ufQQw/N+ikCmYzl17/+NaA1\nzitdGDsdpm0Mox5++MMfli2mQlCmKSKSQEVmmiL1CX3bUl222morAG677bYyR1IYyjRFRBJQpSki\nkoAqTRGRBFRpiogkoEpTRCQBVZoiIglY7QldEx1s9iHwTuHCqQrd3H3TpndrGVTGLZ/KOJm8Kk0R\nkbWNmuciIgmo0hQRSUCVpohIAo1Wmma2iZnNi/4tN7P3Yq/XK1ZQZtbPzF43s8VmNiSH/UfEYnvV\nzA7P8/rPm1mvJvZpY2YTohhnmFnXfK5ZLuUq4+ja65rZK2Y2KYd9S17GsX2PNzPPdf9KU8bv8R1m\n9qGZzctx/9PD/ma2yMx+kef1x5nZ0U3ss2P0/V1jZjktj9nohB3u/hHQKzr5cOBzd7+y1kWN1A2l\n73K5YFPMrDVwPfDfwHJgtpk96O5vNHHoFe4+2sx2Ap4ys808dpfLzNZ1928KEWPkTGC5u3c3s5OA\nvwAnFvD8JVGOMo65EJgPtM1x/1KXMWa2ETAYmF3I85ZSGcv4NuAG4OYEx9zp7heY2RbAfDOb7O7p\nFReLUMYrgXOBY3M9oFnNczPrbmYLzexOYAHQxcw+iX0+0MxujbY3N7OJZjbbzGaZ2Z5NnH5PYJG7\nv+Pua4D7gP65xubu8wEDNo7+0txoZrOAkWa2gZndHsUx18yOjGJsa2bjo79u9wNtcrhUf+COaPs+\n4Me5xlgNilzGmFk34BBgbNLYSljGACOjf2uSxlnpil3G7v4MsKo5sbn7cmAp0DVqZfzNzF4Abo9a\nKFdHcbxiZqdHMbYys/81s9fM7HGgUw7X+cDdZwM5V8T59GnuAFzj7j2B9xrZ71pglLv3Bo4DQiHs\nYWY31bP/1sC/Yq+XRe/lxMz2Br5091BYWwJ7uvtQYBjwqLvvDhwEXGVmbYBfAR+7ew9gBLBr7Hxj\nG2iWpeN096+A1WbWIdc4q0SxyhhgNDAESDzmrVRlbGa7AZu5+9Tan7UgxSzjZjOz7kA34K1YnAe7\n+0mkWnkrojLeDTjHUt1jxwL/BfQETgX2jp3vMjPrV4jY8plPc0lUQzelD7C9Zdaq3tjM1nf3F4EX\n87h+bUPM7OfAZ8DxsffHx5ocfYHDzOx30es2QFdgf2AUgLvPNbMF4WB3P7WAMVabopRx1M/0L3ef\nZ2Z9EsRTsjI2s1bAVVRhl0tClfY9PtHMDiSV2Z/u7p9E13zQ3cNypH2BHmY2MHrdHtiWVBnfHf0u\nLDOzp8NJ3f0PhQown0pzdWz7O1LNpSDe9DFg9ygby8V7QJfY6840/hcwuMLd61v7Nx6nAUe7+5L4\nDrFfhCRCnMst1Znezt0/aeKYalOsMt4bGGBmR0Xn2cjM7nD3QU0cV8oy7kAqY3kuOnYL4GEzO9zd\n5yY9WQUrVhk3153uXt8NmdplPNjdn4jvYGbHFDWySEGGHEU1+8dmtm30Fzoe/DTgnPCigaZu3Eyg\np5l1M7PvkWoKTI6OHRX6qJppKqlO3xBLaKI9C/wsem8XYMcczjUZCF/y44DH8oir4hWyjN19qLt3\ndvca4CTgsVBhVkoZu/sqd+/k7jVRnLOBfi2swsxS4O9xg8zsfDM7q/mRMhUYbGbrRufb3szWJ1XG\nx0d9m1sDB+RxjQYVcpzmb0n9x0wn1Q8ZnAPsE3XYLgTOgIb7Qtz9a+A84HFgITDO3V+PPt6Z1B31\n5roUaGepISsLgOHR+9cDm5jZIuASIP3FaKRP82ZgSzNbTKq/7KI84qoWBSnjJlRSGa+NClbGZjYe\neI5UErQs6loB6AF8lEeMY4A3gXlmNh+4kVSreQLwLql6YywwIxZLvX2aZtbZzJaRqnOGR3E2Opqj\nap49t1Qb6RF31xKVLZTKeO1gZv8A+hd6eFipVE2lKSJSCfQYpYhIAqo0RUQSUKUpIpJAPuM06dSp\nk9fU1BQolOowZ86clWvTrN4q45ZPZZxMXpVmTU0Ns2dX7TwGzWJma9WyACrjlk9lnIya5yIiCajS\nFBFJQJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkkNc4TRGRYvj4448BePfddxvcp1u3bgBcc801\nAOy0004AbLfddgDssssuRYlNmaaISAIVlWmuWLECgOOOOw6AvfdOrYt05plnAqknFwrh008/BeDZ\nZ58F4NBDU9M3tm7duiDnF5FkpkyZAsBDDz0EwNNPPw3Am2++2eAx22+/PQBLly4FYM2a7AVDv/uu\n0CtOpyjTFBFJoOyZZui7ANhxx9SyLSET3HzzzYHCZ5g//OEPAVi5MrUGfXjudtttty3IdSR3//73\nvwH43e9Si0cuWJBaJHLatGnpfdQCaBmWLEmtdXfDDTcAcPPNN6c/++KLLwBIMin666+/3vRORaBM\nU0QkgbJlmiHLC/2XAB99lFpr6ZxzUoveXXfddQW95ogRIwB4++23gcxfOmWYpTdu3DgALr74YqDu\nXdKQgQJssskmpQtMimbZstQ6baNH17cKc+522GEHIHO3vNSUaYqIJFC2TPOf//wnkLlLFjds2LCC\nXWf+/Pnp7SuvvBKAY45JLed8/PHHF+w6kpuQbfz6178GMi2O1EKUGeeem166nOuvvx6Ajh07liJE\naYZQjpDJJPfdd18gMzplvfXWA6B9+/YAbLDBBuljPv/8cwB+/OMfA5ksco899gBg1113Te+7/vrr\nA9CuXbsC/1fkRpmmiEgCqjRFRBIoefM8DGC///7763x22223AbDppvkvzxKa5YccckidzwYMGADA\nhhtumPd1JJnQRRJu+jXknnvuSW8/8sgjQOamUWi6h+aelM/q1auB7O/Zyy+/DMCkSZOy9t1rr70A\nmDt3LpA9lDDcCOzcuTMArVpVbj5XuZGJiFSgkmea//M//wNkhpyEgeYAP/3pTwt2neeffx6A5cuX\np9879dRTATjppJMKdh1p2jvvZNawGjt2bNZnYVKF8CDD448/Xuf48FBCyFJPPPFEALbYYovCBys5\n+eqrrwD42c9+BmSyS4CLLroIgD59+tR7bH0Pq3Tt2rXAERaPMk0RkQRKnmmGoSXh59Zbb53+LJ8+\nqvAY1siRI4HMo1rxoSyhz1RKa968eentMGh9//33B+CZZ54B4MsvvwTgrrvuAuAvf/lL+pjFixcD\nmVZD//79gUxfp4YilU4YGhS+Z2GCjfh9iCFDhgDQtm3bEkdXGso0RUQSKPuEHWFKKIC+ffsC0KFD\nBwDOPvvsJo8Pg+PDz5kzZ2Z9Xsh+Umme+JRdIfMPg9uDNm3aAPCLX/wCgAkTJqQ/CxM9hMkcQgaj\nu+elF+6IX3755UBmIuDnnnsuvU8YvN5SKdMUEUmg5Jnm+eefD8CTTz4JwPvvv5/+LPRvhYziwQcf\nbPJ8Yd/aj+Fts802QKbvRcrn7rvvrvPeP/7xDwCOPvroeo8J0/XVZ8899wSyH8OT0pg+fXrW6/B4\nYxhfuTZQpikikkDJM80f/ehHALz66qtA9p3VRx99FIBRo0YBsNlmmwEwaNCgBs938sknA7Dzzjtn\nvR+WyggZp5TPCSeckN4OrYeXXnoJgNdeew3I/D488MADQPbk1KGPO7wXpvQLZd+zZ8+ixS7Z4n3N\nkBnBcOmll6bfO+qoo4DsSTZaEmWaIiIJqNIUEUnAkqzJUVvv3r29sQ77UnjrrbeATDO8V69eADz2\n2GNAYSb/iDOzOe7eu6AnrWCFKONVq1alt0M5hUcjG7qRF58AIjyocMQRRwDwxhtvAJlVSm+66aa8\n4qtNZdyw2g+n1GedddYB4KyzzgIyc2L+61//AqB79+5AZk2wuLBGVJjco1g3mPIpY2WaIiIJlH1w\ne77+9Kc/AZm/fOEmUqEzTGm++GOO48ePB+DYY48F6mac5513HgB//etf08eEge9hSr/wiOXUqVOB\nzOB30I2/YvvNb34DwFVXXdXgPt9++y2QaSGEn0mEm8AHHnggkD1VYLkp0xQRSaAqM82QrQDccccd\nAGy00UaAVi6sdGG6sDB0JUzQEYYVhZZDyC7jLrnkEgAWLVoEZIYvhWMg8/sgxREenwyryIZp+r7+\n+uv0PmEdqJBxNkeYrDx81+MrT4bJqMtFmaaISAJVmWmGAbVxhx9+OJA9qbFUrpBxNjRRbX3CKoRh\nFdGQaT711FPpfcKdek0XVxzhzvhuu+0GZEYyxD3xxBNAJvscPnw4ALNmzUp8vdDXPWfOnMTHFosy\nTRGRBKo+0wxrH4e7etLyhf60yZMnA9l3VsMa6cOGDSt9YALAwQcfnPU6PCodMs3WrVsDmeVnAM44\n4wwArrnmGiDT112JlGmKiCSgSlNEJIGqap6Hx+XiK0yGVQx1A2jtEdbEHjp0KJC9vna46TBw4EAA\ntttuu9IGJ3WEFRnCKpXhBlGYrQrgzTffBDIrMNQWX0us3JRpiogkUJWZZnyygH79+mXt89lnnwGZ\nuReraT1lSSZMzvLnP/85/V64Ifj73/8egHHjxgGZ4UpSej169AAyQ8XuvffeOvvEh40BrLtuqmoK\nQwnjj9XY76NRAAAG3klEQVSWmzJNEZEEqirTrE/4ixQyijBkITx2pcfqWr5TTjklvT1mzBgAJk6c\nCGT6ymrP7C+lE7L80aNHA5nWYHzA+gcffABATU0NkCnT0EddSZRpiogkUPWZ5i233ALArbfeCsDp\np58OZCZ3kJYvPg3gtGnTgMx63GGCiUoeLL22CCNdpkyZAsDf//739GczZswAMpllmBquEinTFBFJ\noKoyzeuuuw6AP/7xj+n39t9/fwDOPvtsADbeeGMA1ltvvRJHJ5UgjJYIy2WERy0XLlwIaOXKShJW\nE629XemUaYqIJFBVmeZ+++0HwJNPPlnmSKTShUmOd9llFwAWL14MKNOU/CnTFBFJQJWmiEgCVdU8\nF8lVWDPq7bffLnMk0tIo0xQRSUCVpohIAqo0RUQSsLDaW7MONvsQeKdw4VSFbu6+adO7tQwq45ZP\nZZxMXpWmiMjaRs1zEZEEVGmKiCTQaKVpZpuY2bzo33Izey/2uqgzYpjZumb2iplNymHfEbHYXjWz\nw/O89vNm1ivHfY83M891/0pTrjI2swvNbEH079wc9j/dzD6M4lpkZr/I8/rjzOzoJvbpaGaTo9/D\nF82sKp/BLGMZL4u+j/PM7MUc9i9HGfcxs09j/z/+0NR5Gx3c7u4fAb2ikw8HPnf3K2td1Ej1jX7X\n1MUSuhCYD7TNcf8r3H20me0EPGVmm3msw9bM1nX3bwoZoJltBAwGZhfyvKVUjjKO/sAMAnoD3wCP\nmdkUd29qJPqd7n6BmW0BzDezye6+MnbeQpfxJcCL7n6Ume0I/D/gkAKevyTK/D3ez90/SbB/qcsY\n4Cl3b7RyjWtW89zMupvZQjO7E1gAdDGzT2KfDzSzW6Ptzc1sopnNNrNZZrZnDufvRuqXc2zS2Nx9\nPmDAxtFfmhvNbBYw0sw2MLPbozjmmtmR0fXamtn46K/b/UCbHC83Mvq3Jmmcla7IZdwDmOnuX7j7\n18CzwDG5xubuy4GlQNeolfE3M3sBuD1qoVwdxfGKmZ0exdjKzP7XzF4zs8eBTjlcqifwZHTNBcB2\nZrZJrnFWumJ/j/NRwjJOLJ8+zR2Aa9y9J/BeI/tdC4xy997AcUAohD3M7KYGjhkNDAES39o3s72B\nL919VfTWlsCe7j4UGAY86u67AwcBV5lZG+BXwMfu3gMYAewaO99Yq6fpbWa7AZu5+9SkMVaRYpXx\nq8ABlmr+tgMOA7rkGpSZdQe6AW/F4jzY3U8CzgRWRGW8G3COmXUFjgX+i1RFeCqwd+x8l5lZ9rKm\nKS8DA6J99gI6R/9akmJ+jx140szmmNlpSYIqYRkD7GtmL5vZw5ZDF0w+z54vcfdcmqV9gO0ts+zu\nxma2vru/CNTp54j6IP7l7vPMrE+CeIaY2c+Bz4DjY++PjzU5+gKHmdnvotdtgK7A/sAoAHefa2YL\nwsHufmo9MbYCrgJOTBBfNSpKGbv7fDO7GpgGfA7MBb7N4TonmtmBpDL70939k+iaD7r7l9E+fYEe\nZjYwet0e2JZUGd8d/S4sM7OnY/E01I91GXCtmc0jVYG+nGOc1aQoZRzZ093fi5raj5vZInef3sR1\nSl3GLwE17v551PKcSKqCblA+lebq2PZ3pJrEQbx5a8Du7v5VjufdGxhgZkdF59nIzO5w90FNHHeF\nu49uIk4Djnb3JfEdYr8IuepA6q/Zc9GxWwAPm9nh7j436ckqWLHKGHe/GbgZwMxGAYtzOOxOd7+g\niTgNGOzuT8R3MLOcm/+xGD8l1fca/lAuBVraDCDFLOP3op/LzexBYHegqUqzHGUcth+KuvM6NNYP\nW5AhR1HN/rGZbRv9csWDnwacE17U19Stda6h7t7Z3WuAk4DHQoVpZqNCP2QzTQXSd2rNLDTDnwV+\nFr23C7BjEzGucvdO7l4TxTkb6NfCKswshSzjaJ/Nop81wFHAPdHr883srDxCnQoMNrN1o/Ntb2br\nkyrj46N+r62BA3KIsYOZtY5e/hKY5u6rGzummhWyjC11/2CDaLsdqXsU86PXlVTGW8S29wS+aerG\nVSHHaf6W1H/MdGBZ7P1zgH2iDtuFwBlRgI31hTRkZ2B5HjFeCrSz1DCIBcDw6P3rgU3MbBGpO6bp\nyq+hPs21VCHLeFK07yTgLHf/d/R+D+CjPGIcA7wJzDOz+cCNpFpUE4B3gYWkbjDOCAc00t/1A2Ch\nmb0OHExqREdLV6gy3hJ4wcxeBmYBD7j7tOizSirjgZYa9jYPuIbsrr16Vc1jlJZqBz/i7oeWOxYp\nHjP7B9C/CMNKpEJUexlXTaUpIlIJ9BiliEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoi\nIgn8f6S1hmwDdPmXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc287ae2fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_images(images=images,\n",
    "            cls_true=cls_true,\n",
    "            cls_pred=cls_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of Mis-Classified Images\n",
    "\n",
    "We can plot some examples of mis-classified images from the test-set.\n",
    "\n",
    "First we get the predicted classes for all the images in the test-set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = model.predict(x=data.test.images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we convert the predicted class-numbers from One-Hot encoded arrays to integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred,axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot some of the mis-classified images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XncVeP+//HXJ1ORFCUh3c5pUIaTTjKEzENSScgUkupk\n6sTX7JRDZg45knBMOXF0EjJVh/STSoNKknksSRQNhuj6/bHXtde+x73Xvee79/PxuB/32ntfa61P\nLvu6P+ta17ouc84hIiKpqZXvAEREiokaTRGRCNRoiohEoEZTRCQCNZoiIhGo0RQRiUCNpohIBGo0\nRUQiUKMpIhLBpuns3LBhQ1dSUpKhUIrDnDlzVjjnGuU7jlxRHdd8quNo0mo0S0pKmD17djqHKDpm\n9nm+Y8gl1XHNpzqORpfnIiIRqNEUEYlAjaaISARqNEVEIlCjKSISQVp3z0Vy4YMPPgCgf//+AJx2\n2mkAnHfeeXmLSTZeyjRFRCJQpikFyWeXAMcddxwAn3zyCQCfffYZoExT8kOZpohIBMo0paDcfffd\nANx1113x97744otSZZo1a5bTmKR61q5dC8DixYsBeOCBB0p9vnz58vj2+PHjAejXr1+pMr7/+uCD\nD85anFEp0xQRiaCgM81Zs2YBMGrUKCDs5/rjH/8YL9OjRw8A9t13XwAaNdpo5lmoUX777TcAFi1a\nBMDnn4ePBpsZAC1btgRg9OjROY5Okrnhhhvi288++ywQZprvv/8+AH65cF+ficuH+/f8d92/njNn\nDgAvvfRSvGzDhg0z/w+IQJmmiEgEBZlpfv311wD07NkTCPu0Nt00Fu7UqVPjZR9++GEA2rdvD8Cd\nd94JwEEHHZSbYCUjRo4cCcCDDz5YaRmfYey88845iUkq57P9Sy65BCjdP1k2k2zdujUQ9kWfcMIJ\n5Y7nrxj9Ph06dACIz76U2K+tTFNEpIgUZKZZq1asLV+9ejUA9evXB2DMmDEArFq1Kl72yiuvBMK/\nSM899xygTLNYLF26FICHHnoICDONxP4u77bbbstdYFKlFStWlPo9YMCAcmX8ONrddtsNgC233DLp\ncd97771Sx/VZayFRpikiEoEaTRGRCAry8rxx48ZAeIntL7n9jaBevXrFyx544IEAjBgxAoD77rsP\ngAMOOACouNNZCocfWrRgwQKg4suxrl27AtCuXbvcBSZVGjRoUKnfmbJu3TogHK7kb/rk++ZPImWa\nIiIRFGSm6Z1++ulAmGmee+65AAwdOjRexn82d+5cIPwL5X9LYdt6662BMJPwNwASTZ8+HQgfbthj\njz1yFJ3k2jPPPAOEVxy6ESQiUuQKOtP0QxT8Xxs/wLVPnz7lytauXRsIB0efccYZuQhR0uSzxu7d\nuwMVD2732afvt/a/pebxj9H6IWe77LJLqd8V8f3i/v8TXzZbj1Qr0xQRicAqGkScqvbt27tcLDI/\nbtw4AObNmweUnhzAx+/vtCc+YpkNZjbHOdc+qycpILmq4y+//BKAkpISoOLJHHbccUcAJkyYAMCf\n/vSnrMSiOs4f/2CLr/OLL74YCAfIJ/L9n/5+RtlM0z9SDeVH0aRTx8o0RUQiKOg+Tc8/zO9/33zz\nzfHP1q9fD0Dv3r1zH5hkTNOmTYEws/jHP/5RrsySJUuAcNxm4vRxUtz81WPZK1///0HiXfSyE4H4\nCVyOPvpoIHy02n+eaco0RUQiKIpMMxWbbbZZvkOQDBgyZAgQTvUH4dK9/mmRZcuWAXDRRRcBpUdT\ntG3bNidxSvrOPPPM+LZf7qLs+Ey/zEVi1lidiUAySZmmiEgEajRFRCIoqsvz7777Dqh4rsXtttsu\n1+FIFmyzzTZAuAohwNNPPw3AlClTAPjxxx8BuPfeewEYO3ZsvOz8+fMBrRVVSPyQwauvvhoov2YQ\nhJfj/lL78ccfBwpzwh1lmiIiERRVpvn8888D4cqFAFtssQUAXbp0yUtMkn1+EPP9998PwMCBA0t9\n7m8MAfz666+5C0wq5DNLX2/+Jo+fROfEE08E4L///W98H59p+mGFhZhheso0RUQiKKpMMzGj8Cqa\nvENqpr322ivfIUgZ3377LQDXXntt/D2fYfrM0mePZQedVzTtm59UvJAp0xQRiaAoMk1/19z3aSU6\n6aSTch2ORPT666+Xe69Tp04p7//AAw8AcOONNwLlR0+kM+mMVI9fNbJz585A6Uda27RpA4Rr2Zft\nn/T7JmaaZfs0C5kyTRGRCIoi01y+fDkAn332WbnP/Lg+KTx+TfNu3brF3/OPxfk6LcsvX5KYnX7z\nzTdAOGrCZyX+kUm/D8AOO+yQkdilaj4j9BmmzyoTP6tsMbSKrhj8GM5CWkCtMso0RUQiUKMpIhJB\nUVyeS3H6/fffAVi9enX8PT/r+gsvvFDlvhU9YlevXj0AbrnlFgCOP/54AJo0aZKhiCVV/lFIXzeJ\nl9WVXWL7Qe9lZzSCwh7MXpYyTRGRCJRpStZssskmQJghAvzwww8p7etncgfYe++9gXBW90MPPTRT\nIUo1+clTfNbob+QAXHPNNUB4Q8jPfzl69GggHPQ+aNCg+D7t2rXLbsAZpExTRCQCZZqSNX71SJ+N\nALz99tulygwfPhyAQw45BAgflUzMQqTw+KFj/nfiCrGLFy8G4IwzzgDCtez9MDPfl3nVVVflJtgM\nU6YpIhKBMk3JusRHJss+PqmMsmZIXKfH908uWrQIgMGDBwPhypI+Oy2GgewVUaYpIhJBUWSajRs3\nBsIVCmfPnh3/rGPHjkA4pdSkSZNyHJ2IVMWvGun7MotpTGZFlGmKiERQFJnmtttuC8CLL74IhHdl\nAX755RcALrjggtwHJiJJ9evXr9TvYqdMU0QkAjWaIiIRFMXluefXsl6/fn2eIxGRjZUyTRGRCNRo\niohEoEZTRCQCS2clPzP7Fvg8acGapZlzrlG+g8gV1XHNpzqOJq1GU0RkY6PLcxGRCNRoiohEUGWj\naWbbmdm84GeZmS1JeL15toIys8Fm9m7wc2EK5fua2bdBXO+ZWZ80zz/azLonKdPAzF4ws/lBnL3T\nOWe+5LGOvzKzd4LzzEyhfD7quIeZLQjOOcvMDkjnnPmSjzo2s2ZmNsXMFgXfj6TPOeepjq9I+G/x\nrpn9ZmbbVHlg51xKP8BQ4NIK3jegVqrHSeE8bYH5QB1gM+A1YNck+/QF7gq2dwBWAA3LlNk0Qgyj\nge5JyvwNGBZsNwZWRjlHIf7kqo6DY34F1I9QPh91XJew378dsDDfdVQsdQzsCLQNtusBHwMtC62O\ny5Q/AZiYrFy1Ls/NrHnwF+QJ4F2gqZmtSvi8l5k9GGw3NrNxZjbbzN4ys/2SHL41MMM595Nzbj0w\nNfjHpMQ5twz4DNjFzG4ws8fMbBrwiJltamZ3BnEsMLO+QYy1zGyEmS02s0lAKrOjOmDrYLsusQr+\nPdU4C12W6zgtuapj59waF3ybgK2I1XmNkc06ds4tdc7NC7Z/BBYDO6UaWw6/x4lOBcYkK5TOY5S7\nAb2dc7PNrKrjDAdudc7NMLMSYAKwh5ntC5zjnBtQpvw7wBAz2xb4BTgWmJZqUGbWHGgGfJIQ58HO\nuZ/NbCCw3DnXwcy2AGaY2URgP2BXoA2xv5CLgJHB8YYB05xzL5Y51d3ABDNbSuwvac+EL1hNka06\nhlgD9KqZOWCEc+6hVIPKYR1jZj2BYcS+gJ1TjbGIZLOOATCzPwB7ALNSDSqXdRx8Xhc4AjgvWWzp\nNJofO+dmJy/GEUArCxeGb2BmdZxzM4FyfVnOuYVmdicwGVgDvE1qGdzpZnYIsYa2r3NuVXDOZ51z\nPwdljgJam1mv4PU2QAvgYGCMc24D8JWZTUmIJ1ybtLTOwFtAJ6Al8LKZ7emcW5NCrMUiK3Uc2M85\nt8TMdgAmmdl7zrk3k5wn13WMc24sMNbMDgWuD45fk2SzjjGzesB/gQtT/G7kvI4D3YDXnXNJ15hO\np9Fcm7C9gVifiFc7YduADs65X1M9sHNuFDAKwMxuBT5KYbcnnHMVLTiTGKcBA51z/0ssYGbVmUr6\nHGBokF2+b2ZfEms851bjWIUqm3W8JPi9zMyeBToAyRrNXNdxYryvmdmjZlbfObcq+R5FI2t1bLGb\nTOOAh51zz6W4W77quBfweCoFMzLkKGjZV5pZCzOrRek+yMnA+f6FmbVNdjwz2z74XQJ0BZ4MXl9s\nZpVeBqTgFWCgvwwxs1ZmVodYv+kpQZ/ITsSyx2S+AA4PjtMEaA58mkZsBS2TdWxmdYPLIcxsK+BI\nYGHwumDqOOjzs2C7PbGbQjWpwSwlw3VswCPAPOfc8DKfFUwdB/s3AA4Ank+lfCbHaV5O7B/zJrE7\no975QMegw3YRQZ+Bme1rZiMrOdb4oOx4YEDQkQyxm0TfpRHj/cCHwDwzWwjcRyzbHkusEVwEPAxM\n9zuY2TAzq6gvayjQycwWAJOI3ZFcmUZsxSBTddwEmGZm84l1cTzjnJscfFZIdXwysNDM5hHr0zsl\njbiKRabquBOxGytHWjik5+jgs0KqY4ATgZeccz+lcvKieozSzF4Aujnnfst3LJIdquOar9jruKga\nTRGRfNNjlCIiEajRFBGJQI2miEgEajRFRCJIazXKhg0bupKSkgyFUhzmzJmzwm1Es3qrjms+1XE0\naTWaJSUlzJ6dyhNYNYeZbVTLAqiOaz7VcTS6PBcRiUCNpohIBGo0RUQiUKMpIhKBGk0RkQjUaIqI\nRJDWkCORVIwePTq+fdZZZ5X67PHHY/O+nnbaaTmNSbJv8uTYbH/r168H4PnnY9NV3nfffUn39eNG\n69evD8CFF4aL0vbpk9YilWlTpikiEoEyTcm6xOxyk002KfXZ2WefDcDq1asBaNOmDQAHHXRQboKT\njHvyyScBOP300wEoO/2kX2co8f169eoBcMUVVwDQsWNHAPbdd18AtthiiyxGHI0yTRGRCJRpSkEY\nOHAgALvvvjsAI0aMiH924IEH5iUmqZ45c+YA0LRpUwC++OKLpPv4DPPKK6/MXmAZokxTRCSCgsw0\nH3nkEQBuuukmAD744IOk+9StWxeAk046CYBBg2KrgO61115ZiFCi8HfIIezDrMzixYtL/QZlmsXm\ntttuA2DIkCEAvPLKK0D43fQS+yn32WefHEWXPmWaIiIRFFSmedFFFwHwz3/+Eyh/161169YA9OjR\nI/6ev9s6ceJEAN566y0gvPv27LPPxssedthh2QhbkmjZsmV8+/fff6+wzIYNG0q97t+/f3x7yy23\nBDSWs9j4qz/fx1lW4vfxiCOOyElMmaBMU0QkAjWaIiIRFNTluX/cruxl+a677grAm2++CYSPViXy\nl26vv/46AIcccggAxxxzTLzMBRdcAMCdd96ZwaglmUaNwlUFOnXqBMAbb7xRYdmyg98hvHmky/Pi\n8tFHHwFw7733AuW/18Van8o0RUQiKKhMs3HjxgCsXLkSCDNK/4jV5ptvXum+06dPB6BLly6l3veT\nBQA8+uijgDLNXGvWrFl82w9a94PZK8s4pfi9/PLLQPiIrH980l/9lR2CVCyUaYqIRFBQmeYll1wC\nwHnnnQfA2rVrAahduzYAvXr1AsIJASAcjvLaa68BsGbNGiDMUqdOnRove88992QtdknNbrvtBkCr\nVq0AZZo12R133FHh+35oYSFNwhGFMk0RkQgKKtP0fR3+bvmnn34KwK+//grApEmTADj33HPj+/Tu\n3RuACRMmlDrWVlttBcCee+4Zf2/UqFHZCFuqwU9E69fbTmXdbf9wg+8rS+wrleKxYMECoPSjkw0a\nNACgVq3Cz+MKP0IRkQJSUJnmzjvvDISPRB599NEADB06FIDzzz8fCB+VBHjnnXcAePfdd0sd65xz\nzslqrJIZ/pHYuXPnAhWP0/T8xC233HILUHr6OCk8b7/9NhBe7S1ZsgQIp4G7/PLL42VPPfVUADp3\n7lzqGH7Zi0KatEWZpohIBAWVaXrNmzcHwn4u/xfrq6++Srqvz1avueaaLEUnmeQnnVV91Tx+BMuY\nMWMAGDBgAFD+qjCxjP9dlr+6ALjssssyGmdUyjRFRCJQoykiEkFBXp57fhiCnxvTDzH5/PPPK91n\n7733Bop34Kwk98wzzwBw1FFHAdC9e/d8hiNJ+Js4vrtt3bp1ADzxxBPxMv4Ry+eeew6AVatWldrn\n2muvjZfdZZddgPBhl1xTpikiEkFBZ5qezxqvu+46IOxQBvj5559LlfWvv/76awCaNGmSixAlTWVn\nbq+qzPLlywFYsWJFVmOSzPKPQ/vfF154YbkyfvrG559/HoBu3boB4QMuAHfffTegTFNEpCgURabp\nnXLKKUCYcUL4qKXPRv3jk9999x2gTLPQjRs3Dggfn6tqcLvny/jJPhL7NBs2bJjpECUH/OQ8vu/y\noYceqrRsvleYVaYpIhJBUWWafi1sn10m8pM5+DurUhxuvvnmau/773//G4CLL744/p4yzeLhH6sE\n6Nu3LxCukV7WySefHN/2q9XmizJNEZEIiirT/PLLL8u955fA0ATDxemxxx4DYPfdd89zJJIrs2bN\nAkpnj5999lmFZX2Zp556KutxpUqZpohIBGo0RUQiKIrLcz+wdc6cOeU++8Mf/gAU1nx7kjq/ZlDL\nli2BcM7MiqQyAF5yy68D5L+HAAcddBAQ3pTzs5ONHDkSgNtvvx2AX375Jb6Pf4zSD3w/88wzgfzf\n9KmIMk0RkQiKItP0a4okDmr3/F8mKW6jR48GYP/9909a1g9u91cXGmaUP2PHjgVg5syZ8fcaN24M\nQL169QD4/vvvgfCBk4q0adMGCIegdenSJfPBZogyTRGRCAo60/zmm2+AcD30ipx44om5CkeyyE/7\nd9pppwHhwPWq+LJNmzbNXmBSpa5duwLh6goAy5YtA8Lvb1n+kWe/9hfA2WefDYRZaiFTpikiEkFB\nZ5qffPIJAFOnTi31vp+cGKBPnz45jUmyw/dLHnDAAUBqmabkn1/jqUWLFvH3/Bo+ZQes+2ne/L2J\nfE+8UV3KNEVEIijoTLMyfvo3gC233DKPkUim9evXr9RvKQ49e/ascLsmUqYpIhJBQWeafpLZshKn\nlFq6dCkQrrEsIpJNyjRFRCJQoykiEkFBX55vv/32QLi+9cSJEwE45phj4mX841ciIrmgTFNEJIKC\nzjTPOuusUr9FRPJNmaaISATmnKv+zmbfAp9nLpyi0Mw51yjfQeSK6rjmUx1Hk1ajKSKysdHluYhI\nBGo0RUQiUKMpIhJBlY2mmW1nZvOCn2VmtiTh9ebZCsrMBpvZu8HPhSmU72tm3wZxvWdmaU2yaWaj\nzax7kjINzOwFM5sfxNk7nXPmSx7r+FFfZymWVx1XUx7r+Cszeyc4z8wUyuejjnuY2YLgnLPM7ICk\nB3bOpfQDDAUureB9A2qlepwUztMWmA/UATYDXgN2TbJPX+CuYHsHYAXQsEyZTSPEMBronqTM34Bh\nwXZjYGWUcxTiT67qODhmJ6ADMC/F8qrj4qvjr4D6Ecrno47rEt4QbwcsTHbcal2em1lzM1tkZk8A\n7wJNzWxVwue9zOzBYLuxmY0zs9lm9paZ7Zfk8K2BGc65n5xz64GpwAmpxuacWwZ8BuxiZjeY2WNm\nNg14xMw2NbM7gzgWmFnfIMZaZjbCzBab2SQgleUNHbB1sF2XWAX/nmqchS7LdYxz7nXg++rEpjrO\njGzXcTpyVcfOuTUuaDGBrYjVeZXSeSJoN6C3c262mVV1nOHArc65GWZWAkwA9jCzfYFznHMDypR/\nBxhiZtsCvwDHAtNSDcrMmgPNgE8S4jzYOfezmQ0EljvnOpjZFsAMM5sI7AfsCrQBdgQWASOD4w0D\npjnnXixzqruBCWa2FKgH9Ez4j19TZKuO06I6zqhs1rEDXjUzB4xwzj2UalA5rGPMrCcwjFgj2zlZ\nbOk0mh8752anUO4IoJWZ+dcNzKyOc24mUK6fwzm30MzuBCYDa4C3Se2v++lmdgixhravc25VcM5n\nnXM/B2WOAlqbWa/g9TZAC+BgYIxzbgPwlZlNSYjn6krO1xl4i9hlZkvgZTPb0zm3JoVYi0VW6jgN\nquPMy2Yd7+ecW2JmOwCTzOw959ybSc6T6zrGOTcWGGtmhwLXB8evVDqN5tqE7Q3E+kS82gnbBnRw\nzv2a6oGdc6OAUQBmdivwUQq7PeGcG5QkTgMGOuf+l1jAzFK+/E9wDjA0yDzeN7MviX2x5lbjWIUq\na3VcTarjzMvm93hJ8HuZmT1LrA87WaOZ6zpOjPc1i92grO+cW1VZuYwMOQpa9pVm1sLMalG6D3Iy\ncL5/YWZtkx3PzLYPfpcAXYEng9cXm1k6l3qvAAP9ZYiZtTKzOsT6TU8J+kR2IpZZJPMFcHhwnCZA\nc+DTNGIraJmu48qojvMnk3VsZnXNrG6wvRVwJLAweF0wdRz061qw3Z7YTaFKG0zI7DjNy4n9Y94k\ndtfMOx/oGHTYLgLOCwLc18xGVnKs8UHZ8cAA59yPwfutge/SiPF+4ENgnpktBO4jlm2PJfYFWQQ8\nDEz3O5jZMDOrqJ9jKNDJzBYAk4jdkVyZRmzFIGN1bGZPA/8PaGOxoSlnBx+pjvMrU3XcBJhmZvOJ\ndXE845ybHHxWSHV8MrDQYkPfhgOnJDt5UT17bmYvAN2cc7/lOxbJDtVxzVfsdVxUjaaISL7pMUoR\nkQjUaIqIRKBGU0QkgrTWCGrYsKErKSnJUCjFYc6cOSvcRjSrt+q45lMdR5NWo1lSUsLs2ak8TFBz\nmNlGtSyA6rjmUx1Ho8tzEZEI1GiKiESgRlNEJAI1miIiEajRFBGJQI2miEgEajRFRCJQoykiEoEa\nTRGRCNJ6Iqgq69evB+C772JzjS5atAiAFStWADBr1iwAXnrppfg+a9fGZrQ/6aSTSh1r8ODBAGyz\nzTYA1KlTJ1thi4hUSZmmiEgEGc00ly5dGt8ePnw4ALfddluFZf3kxwmr28XdcccdpV7ffvvtABx4\n4IEAXHfddfHPDj300DQilnz75JPYCq2TJ08u9f6CBQvi2/7/kT333BOA0047DYC6devmIkRJk7/q\n/PLLLwF45JFHAHj00UfjZb744osK9x05MraSRr9+/eLvVdRm5JIyTRGRCDKaad51113xbZ8tNmzY\nEIB27dqVKuszzTVrwiWkp0+fTlWmTZsGwOWXXx5/73//i63iufXWW1c3bMmhn3+OLV09bNgwAJ58\n8kkAPvoolVWaY6ZMmQLAiBEjAKhfv34GI5R0bdiwAYDHH38cgBtvvBGADz/8sNJ9Ksse//KXv5T7\nvG/fvgDUqpWfnE+ZpohIBBnNNP/617/Gt88880wg7HfaddddK9znp59+im/7fi3fD+ozy7LmzJkT\n337hhRcA6NWrV3XDlhy6+eabAbjhhhsq/Pz4448H4Kijjoq/N2rUKADeeecdAMaMGQOEWeu4ceOy\nE6yk7Pfff49v+yvO//u//ytVZrPNNgNgjz32AKB3796VHs/fE/n009gy8wMGhMukH3nkkUDlbUq2\nKdMUEYlAjaaISARprXvevn17l41p8v0l+8knnwyEl+AVdRb7ge5PP/00AMcee2zG40lkZnOcc+2z\nepICkok6rlevXnzb3/jz/9/5tWmeeeYZIBxWtMkmm8T38UNW/A2GFi1aALB69WoAVq5cGS976623\nAvC3v/0NgMMOOwyAJ554AoAGDRokjVd1HN19990X3z7//PNLfda/f38gvKzu0aNH0uP5G4NHH300\nEF6mQ3h5728cb7XVVpHjTaeOlWmKiESQtcco0+Gzx+effx6Azp07A/Dyyy+XK7tu3ToAjjvuOCDM\nPqrz10cy65tvvgHgl19+ib/nM8ztt98eCG/itG3bttLj+BsI3tixYwF4//33ATj44IPjn/nHc/05\nfTayfPlyILVMU1L322+/ATB16tRyn+2+++4AXHHFFQA0a9Ys5eM2b94cgCFDhgBw9tlnxz9buHAh\nENZxrr/ryjRFRCIoyEyzrH/9618A7LTTTknL+iErr776alZjkuT8sKJff/01/p7PMH0/9d577x35\nuH7iFn/8Dz74oNKyfuhbq1atIp9Hklu8eDEATz31VPy9HXbYAQivFKNkmMVAmaaISARFkWn6fqjD\nDz8cCB+drIgf+O6nomvTpk2Wo5Oy/CB0Pyg9ka/L9u2rf3O6W7duQMUZZqNGjYBwcPwtt9xS7fNI\ncv5R1kS+/9GPjKhplGmKiERQFJnmFltsAYRj7qrKNP1D/LVr185+YFIhP54ysS8zHf7O93/+8x8A\nPv/880rL+ukDR48enZFzS3StW7fO2LGWLVtW7r1jjjkGyN/UgMo0RUQiKIpM07vkkkuA0k+A+AmK\nvR9++AEIJwN44403chSdeP7u6X777QfAjBkz4p99++23AFxzzTWlfpe9Mrjpppvi2w8//DBQfmox\nv8/VV18df2/gwIHp/wMkZaeeeioQTqICmRkL68d9Xn/99eU+u+qqqwDYfPPN0z5PdSjTFBGJQI2m\niEgERXF57juD/cqViQ/vl51wxL/2l+mJj/D5G0qSXY0bNwZg4sSJQDh7P8D3338PhDO3+2FJZSdj\n8ZfxUL6O/QB5PwTNX+JL7nXo0AGA/fffP/7epZdeCkCXLl2qfVzf7eZXqPVdPpD/YYTKNEVEIiio\nTNOvTDhz5kwgnKDDrzVS0dRwla0t4ge3+8cqIZw1vOx6RZIdft2myy67LP6eX11wxYoVQOmMMlUP\nPfQQkF4mI5nhr94S1wf78ccfq308v0Lla6+9BsCOO+4IwPjx4+Nltt1222ofPxOUaYqIRJD3TNNP\n5QZwyimnADB37tyMHT9xILx//M6vqa1pwnIjcdjIxRdfDIRrwHh+MHpif3VZBx10EBAOYJfC0bJl\ny7T292uhX3jhhUDYl+lXnk3nsdtMU6YpIhJB3jPNxIHqlfVV/PnPfwbCSU0T+emnVq1aVeG+vk8E\nwr9aiUspSG75O+l///vfgXBkxIQJEyrdx08NeMIJJwBa57ym8P2XUD7D9FckZVe0LATKNEVEIsh7\nprnLLrufpF0mAAAG30lEQVTEt5977jkgHGPp+ensK5rW/v777wcqf3zOL8IEcMEFF6QXrGScf1zu\n7bffLvV+4hIXTZo0AZRhFjs/YqJPnz5AeIccwoml/WgX/8h0IY6tVqYpIhKBGk0RkQjyfnmeyKfi\n/jG5VHTs2BEI59YrO7DWr2EC8PXXXwPh5Z4UrsT16/38iVKc/GW4H3o2ZcoUoPRNWr/u/T777JPb\n4KpBmaaISAQFlWlWh7/R49cj8evT+McrE4c0rVmzJrfBSaU+/vhjIFwT2/P1qLV9ipe/kevXD/KT\ns6xbtw4IJ99IfDSykAavJ6NMU0QkgqLPND2/vnXi5BBQ+vEuP4t7ixYtcheYVMhP3FH2sclzzz0X\ngN122y3nMUn1JQ4f8uvRJ74H0L9/fyDMPPM98UZ1KdMUEYmgxmSaZ511FhA+muWnhvProEM4OWrX\nrl0B2G677XIZokiN4x979Ov2QDi1o39k1n/vBg0aBORvbZ9MUaYpIhJBjck0GzVqBISrGI4bNw4I\n79hBuH52IT6atbEZPHgwEPZt+pEN9957LwBPP/10vKy/Inj11VdzGaJU4aeffgLguOOOA8LsEsKV\nYP3d8y233DLH0WWXMk0RkQhqTKbp+SUQtBRCYfNPZZ1zzjkA3HPPPUA4VZz/DeGUflI4/BWcn3DF\nXzEA9O3bF4BatWpmTlYz/1UiIlmiRlNEJIIad3kuxcWvFVR2zSApbP7m3IYNG/IcSe4p0xQRiUCN\npohIBGo0RUQiMOdc9Xc2+xb4PHPhFIVmzrlG+Q4iV1THNZ/qOJq0Gk0RkY2NLs9FRCJQoykiEkGV\njaaZbWdm84KfZWa2JOF1VuZ3MrM2CeeYZ2arzazKBcvNrK+ZfRuUf8/M+qQZw2gz656kTA8zWxCc\nc5aZHZDOOfMlH3UcnHewmb0b/FyYQvl81HEDM3vBzOYHcfZO55z5ou9xlWW2NbPngu/yTDNrk/TA\nzrmUfoChwKUVvG9ArVSPE+UH2AxYDuycpFxf4K5gewdgBdCwTJlNI5x3NNA9SZm6hH3C7YCF2fhv\nkMufXNUx0BaYD9QJ6vg1YNcCrOO/AcOC7cbAyijnKMQffY/LlfkHcHWwvTswKdlxq3V5bmbNzWyR\nmT0BvAs0NbNVCZ/3MrMHg+3GZjbOzGab2Vtmtl+EUx0JvOec+yrVHZxzy4DPgF3M7AYze8zMpgGP\nmNmmZnZnEMcCM+sbxFjLzEaY2WIzmwQ0TOE8a1zwXxrYCqhRd9SyXMetgRnOuZ+cc+uBqcAJqcaW\nqzomVqdbB9t1iX2Jf081zkKn7zEAbYBXg3O+C7Q0sypnJ0/nMcrdgN7OudlmVtVxhgO3OudmmFkJ\nMAHYw8z2Bc5xzg2oYt9ewJgoQZlZc6AZ8ElCnAc75342s4HAcudcBzPbAphhZhOB/YBdif0H3BFY\nBIwMjjcMmOace7GCc/UEhhGrnM5R4iwS2arjd4AhZrYt8AtwLDAt1aByWMd3AxPMbClQD+iZ8Iey\nptjYv8fzgR7AdDPbH9g5+PmustjSaTQ/ds7NTqHcEUArC5bUBRqYWR3n3ExgZmU7mVlt4DhgcIrx\nnG5mhxD7EvZ1zq0Kzvmsc+7noMxRQGsz6xW83gZoARwMjHHObQC+MrMp/qDOuasrO6Fzbiww1swO\nBa4Pjl+TZKWOnXMLzexOYDKwBnib1DK4XNdxZ+AtoBPQEnjZzPZ0ztWktaA39u/xMGC4mc0j1oDO\nJ8n/i+k0mmsTtjcQ6xPxaidsG9DBOfdrxOMfB8x0zq1IsfwTzrlBFbyfGKcBA51z/0ssYGYpXxpW\nxDn3mpk9amb1nXOrku9RNLJWx865UcAoADO7Ffgohd1yXcfnAEOD7PJ9M/uSWOM5txrHKlQb9ffY\nOfcDcFawfy1iXQKfVrVPRoYcBS37SjNrEZw4MfjJwPn+hZm1TfGwp1ImpTezi82sqsuAZF4BBvrL\nEDNrZWZ1iPWpnRL0iexELLOoUtAfZMF2e2I3hWpSg1lKpuvYzLYPfpcAXYEng9cFU8fAF8DhwXGa\nAM1J8oUqZhvp97i+mW0WvOwPTHbOra1qn0yO07yc2D/mTSCxw/d8oGPQYbsIOC8Idl8zG1n+MGBm\nWwOHAuPLfNSaKvoaUnA/8CEwz8wWAvcRy7bHEvuCLAIeBqYnxDLMzCrqrzwZWBik9cOBU9KIq1hk\nrI6B8UHZ8cAA59yPwfuFVMdDgU5mtgCYROyu88o0YisGG9v3eE9gkZm9T+wPZNJuhKJ6jNLMXgC6\nOed+y3cskh2q45qv2Ou4qBpNEZF802OUIiIRqNEUEYlAjaaISARqNEVEIlCjKSISgRpNEZEI1GiK\niETw/wHzcqwbcSXHTgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc285c98128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_example_errors(cls_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This tutorial showed how to optimize the hyper-parameters of a neural network using Bayesian optimization. We used the scikit-optimize (`skopt`) library which is still under development, but it is already an extremely powerful tool. It was able to substantially improve on hand-tuned hyper-parameters in a small number of iterations. This is vastly superior to Grid Search and Random Search of the hyper-parameters, which would require far more computational time, and would most likely find inferior hyper-parameters, especially for more difficult problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "These are a few suggestions for exercises that may help improve your skills with TensorFlow. It is important to get hands-on experience with TensorFlow in order to learn how to use it properly.\n",
    "\n",
    "You may want to backup this Notebook before making any changes.\n",
    "\n",
    "* Try and run 100 or 200 iterations of the optimization instead of just 40 iterations. What happens to the plotted landscapes?\n",
    "* Try some of the other optimization methods from scikit-optimize such as `forest_minimize` instead of `gp_minimize`. How do they perform?\n",
    "* Try using another acquisition function for the optimizer e.g. Probability of Improvement.\n",
    "* Try optimizing more hyper-parameters with the Bayesian optimization. For example, the kernel-size and number of filters in the convolutional-layers, or the batch-size used in training.\n",
    "* Add a hyper-parameter for the number of convolutional layers and implement it in `create_model()`. Note that if you have pooling-layers after the convolution then the images are downsampled, so there is a limit to the number of layers you can have before the images become too small.\n",
    "* Look at the plots. Do you think that some of the hyper-parameters may be irrelevant? Try and remove these parameters and redo the optimization of the remaining hyper-parameters.\n",
    "* Use another and more difficult dataset with image-files.\n",
    "* Train for more epochs. Does it improve the classification accuracy on the validiation- and test-sets? How does it affect the time-usage?\n",
    "* Explain to a friend how the program works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License (MIT)\n",
    "\n",
    "Copyright (c) 2016-2018 by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}