Basic Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic classifier with TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code in this notebook is copyright 2018 <a href='https://comind.org/'>coMind</a>. Licensed under the Apache License, Version 2.0; you may not use this code except in compliance with the License. You may obtain a copy of the <a href='http://www.apache.org/licenses/LICENSE-2.0'>License</a>.\n",
    "\n",
    "Join the <a href='https://comindorg.slack.com/join/shared_invite/enQtNDMxMzc0NDA5OTEwLWIyZTg5MTg1MTM4NjhiNDM4YTU1OTI1NTgwY2NkNzZjYWY1NmI0ZjIyNWJiMTNkZmRhZDg2Nzc3YTYyNGQzM2I'>conversation</a> at Slack.\n",
    "\n",
    "This a series of three tutorials you are in the first one: \n",
    "* [Basic Classifier](https://github.com/coMindOrg/federated-averaging-tutorials/blob/master/Basic%20Classifier.ipynb)\n",
    "* [Basic Distributed Classifier](https://github.com/coMindOrg/federated-averaging-tutorials/blob/master/Basic%20Distributed%20Classifier.ipynb)\n",
    "* [Basic Federated Classifier](https://github.com/coMindOrg/federated-averaging-tutorials/blob/master/Basic%20Federated%20Classifier.ipynb)\n",
    "\n",
    "## What is TensorFlow?\n",
    "\n",
    "TensorFlow™ is an open source library developed by Google, for high performance numerical computation which has become mainstream within the Deep Learning community. The basic architecture of Tensorflow could be summarized as follows: \n",
    "\n",
    "- Operations are defined (layers of the neural network, sums, multiplications, optimizers ...) as nodes of a graph. \n",
    "\n",
    "- The data is in the form of a tensor. These tensors go through the nodes of the graph, where the defined operations are executed on them (hence the name TensorFlow). \n",
    "\n",
    "- Once the graph and the input tensors have been defined, the entire process described above is executed in a Session.\n",
    "\n",
    "If you would like to get a better idea of how TensorFlow works, we highly reccomed you to read the following article:\n",
    "\n",
    "<a href=\"https://medium.com/@camrongodbout/tensorflow-in-a-nutshell-part-one-basics-3f4403709c9d\">TensorFlow in a nutshell</a>\n",
    "\n",
    "You could also refer to the <a href=\"https://www.tensorflow.org\">official TensorFlow web page</a> for more information, tutorials and instalation guides.\n",
    "\n",
    "This basic classifier tutorial is based on the official TensorFlow tutorial, we have added some additional concepts such as the Datasets API and the Monitored Training Session. This is because we will need those concepts in the more advanced tutorials and we hope showing them here will make the series easier to follow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why only TensorFlow and not keras?\n",
    "\n",
    "TensorFlow incorporates a high level module called keras. It simplifies the usage a TensorFlow to a great extent.\n",
    "\n",
    "These tutorials could also use Keras, but native TensorFlow is more transparent allowing us to demonstrate more easily all the steps and decisions that are taken in the creation of the graph. We will only use Keras to import the data and define the cost function for convenience."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First step: import libraries\n",
    "\n",
    "The first one to be imported will be tensorflow, which by agreement is usually given the abbreviated name of tf.\n",
    "\n",
    "Although we really do not need to do to import keras (we could use it from tf.keras) we will also import it separately for convenience.\n",
    "\n",
    "Finally, three traditional basic Python libraries: numpy, matplotlib and time.\n",
    "\n",
    "In the next cell, we also define some basic constant variables for our algorithm. The number of epochs to train and the size of the batches we are going to use for trainning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TensorFlow and tf.keras\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "# Helper libraries\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from time import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 32\n",
    "EPOCHS = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and normalize the data\n",
    "\n",
    "We are going to work with a database called fashion mnist kindly put together by Zalando to substitute the more traditional handwritten digit database. It contains photographs consisting of 28x28 pixels in black and white of 10 different types of clothing. The features on which we will train our model will be the values of each pixel, ranging from 0 to 255.\n",
    "\n",
    "After loading we will divide these values by 255 so that they are normalized between 0 and 1.\n",
    "\n",
    "We have also added a list with the names of the different types of clothing, so that we can visualize the images with their names later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fashion_mnist = keras.datasets.fashion_mnist\n",
    "(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()\n",
    "print('Data loaded')\n",
    "print('Local dataset size: {}'.format(train_images.shape[0]))\n",
    "\n",
    "class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',\n",
    "               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']\n",
    "\n",
    "train_images = train_images / 255.0\n",
    "test_images = test_images / 255.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global Step\n",
    "\n",
    "Next we create a global step for the optimizer. This is a variable that increases every time the parameters of the model are updated. It is not necessary to create it always in TensorFlow, and it could also be created as a normal variable. However, by creating it through this function we guarantee that any other function that requires the global step will find it through its standardized name (a name that the variable carries below, its context or name scope). In addition, the function checks if a global step already exists, and it associates it with this variable instead of creating a new one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "global_step = tf.train.get_or_create_global_step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start to build the graph. Order it through name_scope.\n",
    "\n",
    "Getting a little bit ahead of ourselves, here we show a Tensorboard visualization of the final graph we will build in this tutorial. Notice that all operations are nicely arranged and it is easy to see the different elements of our neural network:\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/coMindOrg/federated-averaging-tutorials/master/images/graph_tensorboard.png\" alt=\"Tensoboard Graph\" border=\"0\">\n",
    "\n",
    "If we are not careful while defining our graph, each operation we add would be represented by a single node and the final visualization could end up being a rather confusing tangle.\n",
    "\n",
    "To avoid this, we will make use of the Context Manager tensorflow provides, the name_scope. Through name_scopes we will be able to merge operations together and visualize them as single blocks with Tensorboard. This will also force us to think on the structure of the graph will we code our neural network.\n",
    "\n",
    "In the next cell, we group every node related with data inputs into a single name_scope called 'dataset'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.name_scope('dataset'), tf.device('/cpu:0'):\n",
    "    images_placeholder = tf.placeholder(train_images.dtype, [None, train_images.shape[1], train_images.shape[2]], \n",
    "                                        name='images_placeholder')\n",
    "    labels_placeholder = tf.placeholder(train_labels.dtype, [None], name='labels_placeholder')\n",
    "    batch_size = tf.placeholder(tf.int64, name='batch_size')\n",
    "    shuffle_size = tf.placeholder(tf.int64, name='shuffle_size')\n",
    "\n",
    "    dataset = tf.data.Dataset.from_tensor_slices((images_placeholder, labels_placeholder))\n",
    "    dataset = dataset.shuffle(shuffle_size, reshuffle_each_iteration=True)\n",
    "    dataset = dataset.repeat(EPOCHS)\n",
    "    dataset = dataset.batch(batch_size)\n",
    "    iterator = tf.data.Iterator.from_structure(dataset.output_types, dataset.output_shapes)\n",
    "    dataset_init_op = iterator.make_initializer(dataset, name='dataset_init')\n",
    "    X, y = iterator.get_next()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Placeholders\n",
    "\n",
    "Placeholders are a way to initialize variables, without giving them a value, or even a certain shape if you do not want to. We simply create a structure, to which we tell what kind of data it will contain, and the dimensions that we know. For example, the first of them represents the image database. We know that the type of data that will contain will be the same as that of the images. In terms of its dimensions we know what dimension each image has (2nd and 3rd placeholder dimension) but we do not know how many images there will be (first dimension of the placeholder), by putting a <b>None</b> we are letting it know that in that dimension there can be any number of elements.\n",
    "\n",
    "This type of variables serve as input for operations that we define in the graph. A kind of promise that in the future (once the training session begins) we will pass on real values of this type.\n",
    "\n",
    "The first placeholder will be the input features to the neural network, the second will be the true label of each processed example. The third placeholder will be used to indicate how big we want each batch to be.\n",
    "\n",
    "We use a placeholder for the batch size because we want different batch sizes during training and testing. In fact, we don't need batches at all during testing.\n",
    "\n",
    "### Create batches through an iterator\n",
    "\n",
    "The function <b>from_tensor_slices</b> creates a database by dividing the tensors given as input by rows. The <b>shuffle</b> method will shuffle the dataset in chunks of <i>shuffle size</i> examples, which means that a <i>shuffle size</i> of 1 will not shuffle at all and a <i>shuffle size</i> equal or greater than the dataset size will provide an uniform sampling, however, the bigger the <i>shuffle size</i> the greater memory consumption.The <b>batch</b> function is responsible for grouping those rows in batches. And with <b>repeat</b> we make the database cyclical, that is to say that when we iterate over the whole dataset once it does not stop there but can start again, allowing us to train over the dataset as many times (<b>EPOCHS</b>) as we want.\n",
    "\n",
    "Finally we define the iterator, that will initialize and iterate over the dataset we just created.\n",
    "\n",
    "If we were to execute the graph in a normal session we could use one of the default iterators and initialize it in the session itself. We are going to use a <b>MonitoredTrainingSession</b> and we will later discuss its pros and cons. For now what concerns us is that the usage of a monitored session makes it harder to use a pre defined iterator as the monitored session takes care of all the initialization ops. Therefore creating an iterator explicitly and manually initializing it within a hook (a type of function that is executed while the training session is running) is our best option."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the architecture of the neural network\n",
    "\n",
    "Next we define the layers of our neural network.\n",
    "\n",
    "- The first of them, flatten type. The only thing this layer is going to do is turn our image (a matrix) into a vector, that is, take the rows of the matrix and place them one after another. The input to this layer is going to be the placeholder that we created for the features (our data).\n",
    "\n",
    "- The second layer is a fully connected or dense layer, which means that all the neurons of the previous layer are connected to all the neurons of the current one. The input is the previous layer itself (rather its output), and in this way we are chaining layers with each other. The number of neurons will be 128 and the activation function will be a <a href=\"https://en.wikipedia.org/wiki/Rectifier_(neural_networks)\">rectified linear unit function</a>  (ReLU).\n",
    "\n",
    "- The third and last layer will also be dense. Once more the input will be the output of the previous layer and this time the layer will consist of 10 neurons with a <a href=\"https://en.wikipedia.org/wiki/Softmax_function\">softmax function</a>. This is because we have 10 classes in our dataset. Thus, each of these 10 neurons will have as output the probability that the sample belongs to a certain group."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the training is done on batches, if we logged any metric during our training it would be the metric evaluated in a single batch which does not provide us with any useful information. Therefore, we will use the exponential moving average of the metrics in each batch to evaluate the evolution of training.\n",
    "\n",
    "With the following line we will create an object that will be dedicated to saving the moving averages of whichever metrics we want to define."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_averages = tf.train.ExponentialMovingAverage(0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flatten_layer = tf.layers.flatten(X, name='flatten')\n",
    "\n",
    "dense_layer = tf.layers.dense(flatten_layer, 128, activation=tf.nn.relu, name='relu')\n",
    "\n",
    "predictions = tf.layers.dense(dense_layer, 10, activation=tf.nn.softmax, name='softmax')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The loss function\n",
    "\n",
    "Once again, when defining several functions related to one operation, in this case the loss function, we group them all within a name_scope.\n",
    "\n",
    "With </b>keras.losses.sparse_categorical_crossentropy</b> we define the cost function, which takes as arguments the placeholder** of the labels (in the training session through this placeholder, the function will receive the correct label in which it had to classify the training example) and the output of the neural network. \n",
    "\n",
    "Since we are training by batches, we will have the cost for every example in the batch. To have a single cost value we average those costs with <b>tf.reduce_mean</b>.\n",
    "\n",
    "In the following line we use the object created previously to save the moving average of the cost.\n",
    "\n",
    "Finally we save this average in <b>summary</b> to be able to show it later in Tensorboard.\n",
    "\n",
    "** WATCH OUT! Again, as we train by batches, this is not the placeholder that we have manually created, but it is the one that the iterator returns (dividing into batches the one that we have created)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.name_scope('loss'):\n",
    "    loss = tf.reduce_mean(keras.losses.sparse_categorical_crossentropy(y, predictions))\n",
    "    loss_averages_op = summary_averages.apply([loss])\n",
    "    tf.summary.scalar('cross_entropy', summary_averages.average(loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The metric\n",
    "\n",
    "In these lines we will define some functions to know what is the precision of our model.\n",
    "\n",
    "Instead of using a predefined function, we will create the logic to obtain the precision.\n",
    "\n",
    "- From predictions (the output of the neural network) and through argmax we obtain which position of the vector has the highest probability.\n",
    "- We compare the real values that we will provide to y. If it has guessed correctly tf.equal will return True.\n",
    "- Then we average the correct predictions over the whole batch through tf.reduce_mean.\n",
    "- Finally we do the same thing as we did with the cost, save the moving averages instead of the prediction on this particular batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.name_scope('accuracy'):\n",
    "    with tf.name_scope('correct_prediction'):\n",
    "        correct_prediction = tf.equal(tf.argmax(predictions, 1), tf.cast(y, tf.int64))\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "    accuracy_averages_op = summary_averages.apply([accuracy])\n",
    "    tf.summary.scalar('accuracy', summary_averages.average(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the type of training\n",
    "\n",
    "Next we define the training operation. We use the ADAM optimizer to minimize the loss.\n",
    "\n",
    "Since loss_averages_op and accuracy_averages_op are disconnected from the rest of the graph, they will not be executed when train_op is executed. That's why we make it dependent on these two operations with tf.control_dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.name_scope('train'):\n",
    "    with tf.control_dependencies([loss_averages_op, accuracy_averages_op]):\n",
    "        train_op = tf.train.AdamOptimizer(0.001).minimize(loss, global_step=global_step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of the graph and start of the definition of the training session\n",
    "\n",
    "So far we have defined the graph. Once this is finished we will start to define the training session.\n",
    "\n",
    "We will start with a few simple operations.\n",
    "\n",
    "We calculate the number of batches in each epoch and how many batches we will train in total (last_step).\n",
    "\n",
    "Finally we define a directory where we want to save the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_batches = int(train_images.shape[0] / BATCH_SIZE)\n",
    "last_step = int(n_batches * EPOCHS)\n",
    "\n",
    "checkpoint_dir='logs_dir/{}'.format(time())\n",
    "print('Checkpoint directory: ' + checkpoint_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring the session\n",
    "\n",
    "We create a configuration object for the session in which we configure 2 parameters:\n",
    "\n",
    "- <b>allow_soft_placement</b>: if True it allows the algorithm to relegate in the CPU operations that do not have implementation for the GPU, even if we had specifically said that we wanted them in the GPU.\n",
    "- <b>log_device_placement</b>: if activated, it shows on the screen each of the operations that will be carried out and on which device they are placed in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess_config = tf.ConfigProto(allow_soft_placement=True, log_device_placement=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hooks\n",
    "\n",
    "Hooks are sets of functions, which will be executed during the lifetime of a session. This allow us to introduce pythonic functions within the training of our tensorflow session.\n",
    "\n",
    "For example, in the following hook we will define a class that inherits from <b>tf.train.SessionRunHook</b>. As it can be seen in <a href=\"https://www.tensorflow.org/api_docs/python/tf/train/SessionRunHook\">the documentation</a>, tf.train.SessionRunHook has methods that run automatically at certain points in the training session.\n",
    "- <b>begin</b> is executed at the beginning of the session.\n",
    "- <b>before_run</b> is executed at the beginning of each call to run (we will see it later), that is before you train with each batch. Within this function, what we are doing is storing certain values (loss, accuracy, and global step) in order to recover them later.\n",
    "- <b>after_run</b>. The first thing is to recover the values that we previously saved and that are stored in run_values.results. Then we accumulate the cost and accuracy after each batch so that we can print a rough approximation of both metrics averaged over each epoch. Finally we check if we have completed a full epoch with the current batch, and if so we show the values of the average cost and accuracy at this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class _LoggerHook(tf.train.SessionRunHook):\n",
    "    def begin(self):\n",
    "        self._total_loss = 0\n",
    "        self._total_acc = 0\n",
    "\n",
    "    def before_run(self, run_context):\n",
    "        return tf.train.SessionRunArgs([loss, accuracy, global_step])\n",
    "\n",
    "    def after_run(self, run_context, run_values):\n",
    "        loss_value, acc_value, step_value = run_values.results\n",
    "        self._total_loss += loss_value\n",
    "        self._total_acc += acc_value\n",
    "        if (step_value + 1) % n_batches == 0 and not step_value == 0:\n",
    "            print(\"Epoch {}/{} - loss: {:.4f} - acc: {:.4f}\".format(\n",
    "                int(step_value / n_batches) + 1, EPOCHS, self._total_loss / n_batches, self._total_acc / n_batches))\n",
    "            self._total_loss = 0\n",
    "            self._total_acc = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize iterator\n",
    "\n",
    "Next, we need to initialize our iterator. Once again this hook inherits from <b>SessionRunHook</b> and we use the <b>after_create_session</b> method, to execute the initialization just at the beginning, but once the graph is already created and the session started so that we can already pass the object that will be our training session (<b>session</b>) as an argument.\n",
    "\n",
    "<b>IMPORTANT!</b> In this step is when we give the placeholders the real values that they will have in <b>THIS SESSION</b>. We do it through a <b>feed_dict</b> dictionary. The place of <b>images_placeholder</b> will be taken by the training images and thus with each placeholder that we have defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class _InitHook(tf.train.SessionRunHook):\n",
    "    def after_create_session(self, session, coord):\n",
    "        session.run(dataset_init_op, feed_dict={\n",
    "            images_placeholder: train_images, labels_placeholder: \n",
    "            train_labels, batch_size: BATCH_SIZE, shuffle_size: train_images.shape[0]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run session to train our model\n",
    "\n",
    "The final step to train our model! There are different types of sessions with which we can not only train but do any other type of operations with tensorflow. The most basic of these is tf.Session. We will present this session in the next step, when we see how to use a trained model to make inference and thus evaluate its performance.\n",
    "\n",
    "For now we are going to use <b>tf.train.MonitoredTrainingSession</b>. This type of session will do many things for us that otherwise we would have to code manually. It will save the checkpoints for us, save the summaries (the metrics and variables), restore the variables if the training is interrupted and several devices that train simultaneously  can be synchronized if you configure a cluster for it.\n",
    "\n",
    "As arguments we pass the directory where we want to save the checkpoints, a list with all the hooks that we want to run, the configuration of the session that we have defined above and after how many steps we want to save the checkpoint.\n",
    "\n",
    "Once the session is created as mon_sess, we run it in a while loop until it receives the stop command once mon_sess.run() has been called last_step times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.name_scope('monitored_session'):\n",
    "    with tf.train.MonitoredTrainingSession(\n",
    "            checkpoint_dir=checkpoint_dir,\n",
    "            hooks=[_LoggerHook(), _InitHook()],\n",
    "            config=sess_config,\n",
    "            save_checkpoint_steps=n_batches) as mon_sess:\n",
    "        while not mon_sess.should_stop():\n",
    "            mon_sess.run(train_op)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate our model\n",
    "\n",
    "Because of how tensorflow works, the model trained in the previous session only exists while the session is open. Once closed that trained model no longer exists. So now to evaluate it we are going to open a simple session with <b>tf.Session</b> and we will load the checkpoints that have been saved. Once the model is restored. We have to re-initialize the iterator but this time to <b>feed_dict</b> we will pass other values, since in this session we want the placeholders to take the values of the test images and test labels.\n",
    "\n",
    "Once this is done we do a <b>sess.run</b> on the operation <b>accuracy</b> which is the one that will tell us what precision our model has and then again on <b>predictions</b> (which is the name we have given to the last layer of the neural network) to save the predictions that the model has made on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    ckpt = tf.train.get_checkpoint_state(checkpoint_dir)\n",
    "    tf.train.Saver().restore(sess, ckpt.model_checkpoint_path)\n",
    "    print('Model restored')\n",
    "    graph = tf.get_default_graph()\n",
    "    sess.run(dataset_init_op, feed_dict={\n",
    "        images_placeholder: test_images, labels_placeholder: test_labels, \n",
    "        batch_size: test_images.shape[0], shuffle_size: 1})\n",
    "    print('Test accuracy: {:4f}'.format(sess.run(accuracy)))\n",
    "    predicted = sess.run(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Show result\n",
    "\n",
    "Finally we can show the first 25 images and the prediction of the network together with the real label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 10))\n",
    "for i in range(25):\n",
    "    plt.subplot(5, 5, i + 1)\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.grid(False)\n",
    "    plt.imshow(test_images[i], cmap=plt.cm.binary)\n",
    "    predicted_label = np.argmax(predicted[i])\n",
    "    true_label = test_labels[i]\n",
    "    if predicted_label == true_label:\n",
    "      color = 'green'\n",
    "    else:\n",
    "      color = 'red'\n",
    "    plt.xlabel(\"{} ({})\".format(class_names[predicted_label],\n",
    "                                class_names[true_label]),\n",
    "                                color=color)\n",
    "\n",
    "plt.show(True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}