TF_demo1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TF_demo1.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python2",
      "display_name": "Python 2"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "AC8adBmw-5m3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This is an Earth Engine <> TensorFlow demonstration notebook.  The default public runtime already has the tensorflow libraries we need installed.  The first step is to verify that by importing the TensorFlow library.  Run the code in the cell by clicking the run button on the left (hover on the `[ ]`).  \n",
        "\n",
        "**NOTE: The cloud storage buckets used here are no longer writable. (Temporary access was for the Earth Engine Users Summit 2018 only).  Please substitute your own cloud storage bucket.**\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "i1PrYRLaVw_g",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Import tensorflow library\n",
        "\n",
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "46iNFM8lV6kP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Check the TensorFlow install by running a hello world operation (from the [Cloud ML example](https://cloud.google.com/ml-engine/docs/tensorflow/getting-started-training-prediction#run_a_simple_tensorflow_python_program)):"
      ]
    },
    {
      "metadata": {
        "id": "R27Tc9zqerkG",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Hello World TensorFlow\n",
        "\n",
        "hello = tf.constant('Hello world!')\n",
        "with tf.Session() as sess:\n",
        "  print sess.run(hello)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "--JtxLXsUpVA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note that you can use \"magic\" commands by prepending an `!` to a bash command.  For example, here we will install a python library to enable us to connect to Google Drive.  Learn more about magic functions from **Code snippets** to the left.  The objective here is to enable access to thinhs in Drive that you may have exported from Earth Engine."
      ]
    },
    {
      "metadata": {
        "id": "sYyTIPLsvMWl",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Install the PyDrive library\n",
        "\n",
        "# This only needs to be done once per notebook.\n",
        "!pip install -U PyDrive"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qptAXhKmXo_J",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We need to import some authentication APIs so that we can read from Drive and/or a cloud storage bucket.  See the **Code snippets** to the left."
      ]
    },
    {
      "metadata": {
        "id": "5qMKG1hEXuML",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Import authentication libraries\n",
        "\n",
        "from google.colab import auth\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from oauth2client.client import GoogleCredentials"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dEM3FP4YakJg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Authentication**.  The following will trigger the browser dance to authenticate.  Follow the link, copy the code from another browser window to the indicated field, then press return.  You should use the same account to authenticate here that you used to join the training group (which is hopefully the same account you use to login to Earth Engine, otherwise the exports will end up in the Drive of another account.)"
      ]
    },
    {
      "metadata": {
        "id": "CHsEU90Xyjmo",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Authenticate\n",
        "\n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZcjQnHH8zT4q",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We've already generated some training data in Earth Engine.  Specifically, these are exported testing and training data from a very simple classification demo.  The script exports a training dataset, a testing dataset and the image data (in TFRecord format) on which to make predictions:\n",
        "\n",
        "https://code.earthengine.google.com/a7ed957f3034825a54b6b546b8c5ce83\n",
        "\n",
        "RUN THE EXPORTS\n",
        "\n",
        "---\n",
        "\n",
        "Note that the script exports to two places: Your drive account and a public cloud storage bucket.  You can grab the files you need from either place, as demonstrated in the following sections.  Here we'll use Drive, but note that code to use Cloud Storage is also here in case you need it."
      ]
    },
    {
      "metadata": {
        "id": "EazQzf8lzLF3",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Load training/testing data from Earth Engine exports\n",
        "\n",
        "# Specify the training file exported from EE.\n",
        "# If you wish to use your own data, then\n",
        "# replace the file ID, below, with your own file.\n",
        "trainFileId = '1bLHhjGjKYXtdK_XAwC9636ZuxAKuGlmO' # nclinton version!\n",
        "trainDownload = drive.CreateFile({'id': trainFileId})\n",
        "\n",
        "# Create a local file of the specified name.\n",
        "tfrTrainFile = 'training.tfrecord.gz'\n",
        "trainDownload.GetContentFile(tfrTrainFile)\n",
        "print 'Successfully downloaded training file?'\n",
        "print tf.gfile.Exists(tfrTrainFile)\n",
        "\n",
        "# Specify the test file.\n",
        "# If you wish to use your own data, then\n",
        "# replace the file ID, below, with your own file.\n",
        "testFileId = '1PWakg7ygx-vRm5O_QKup6GIJup8LIvLy' # nclinton version!\n",
        "testDownload = drive.CreateFile({'id': testFileId})\n",
        "\n",
        "# Creates a local file of the specified name.\n",
        "tfrTestFile = 'testing.tfrecord.gz'\n",
        "testDownload.GetContentFile(tfrTestFile)\n",
        "print 'Successfully downloaded testing file?'\n",
        "print tf.gfile.Exists(tfrTestFile)\n",
        "\n",
        "print 'Content of the working directory:'\n",
        "!ls"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "LS4jGTrEfz-1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Here we are going to read from the Drive file into a `tf.data.Dataset`.  ([Slide](https://docs.google.com/presentation/d/1fEf-oScgbC9zjbzI3K3jUlHf4JdmoSLFiG_H491FUmk/edit#slide=id.g3b76860e75_0_63)).  Check that you can read examples from the file.  The purpose here is to ensure that we can read from the file without an error.  The actual content is not necessarily human readable."
      ]
    },
    {
      "metadata": {
        "id": "T3PKyDQW8Vpx",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Inspect the TFRecord dataset\n",
        "\n",
        "driveDataset = tf.data.TFRecordDataset(tfrTrainFile, compression_type='GZIP')\n",
        "iterator = driveDataset.make_one_shot_iterator()\n",
        "foo = iterator.get_next()\n",
        "with tf.Session() as sess:\n",
        "    print sess.run([foo])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "BrDYm-ibKR6t",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Define the structure of your data.  This includes the names of the bands you originally exported from Earth Engine and the name of the class property.  Unfortunately, these are called *features* in the TensorFlow context (not to be confused with an `ee.Feature`).  ([Slide](https://docs.google.com/presentation/d/1fEf-oScgbC9zjbzI3K3jUlHf4JdmoSLFiG_H491FUmk/edit#slide=id.g3b76860e75_0_67)).  Think of `columns` as a placeholder for the data that you're going to read in."
      ]
    },
    {
      "metadata": {
        "id": "-6JVQV5HKHMZ",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Define the structure of the training/testing data\n",
        "\n",
        "# Names of the features.\n",
        "bands = ['B2', 'B3', 'B4', 'B5', 'B6', 'B7']\n",
        "label = 'landcover'\n",
        "featureNames = list(bands)\n",
        "featureNames.append(label)\n",
        "\n",
        "# Feature columns\n",
        "columns = [\n",
        "  tf.FixedLenFeature(shape=[1], dtype=tf.float32) for k in featureNames\n",
        "]\n",
        "\n",
        "# Dictionary with names as keys, features as values.\n",
        "featuresDict = dict(zip(featureNames, columns))\n",
        "print featuresDict"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QNfaUPbcjuCO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now we need to make a parsing function.  The parsing function reads data from a serialized example proto into a dictionary in which the keys are the feature names and the values are the tensors storing the value of the feature for that example.  ([TF reference](https://www.tensorflow.org/programmers_guide/datasets#parsing_tfexample_protocol_buffer_messages), [Cloud ML reference](https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/cloudml-template/template/trainer/input.py#L61)).  \n",
        "\n",
        "Here we make a parsing function for the TFRecord files we've been generating.  The check at the end is to print a single parsed example."
      ]
    },
    {
      "metadata": {
        "id": "x2Q0g3fBj2kD",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Make and test a parsing function\n",
        "\n",
        "def parse_tfrecord(example_proto):\n",
        "  parsed_features = tf.parse_single_example(example_proto, featuresDict)\n",
        "  labels = parsed_features.pop(label)\n",
        "  return parsed_features, tf.cast(labels, tf.int32)\n",
        "\n",
        "# Map the function over the dataset\n",
        "parsedDataset = driveDataset.map(parse_tfrecord, num_parallel_calls=5)\n",
        "\n",
        "iterator = parsedDataset.make_one_shot_iterator()\n",
        "foo = iterator.get_next()\n",
        "with tf.Session() as sess:\n",
        "    print sess.run([foo])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xLCsxWOuEBmE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Another thing we might want to do as part of the input process is to create new features, for example NDVI.  Here are some helper functions for that.  Note that a and b are expected to be shape=[1] tensors and features is s dictionary of input tensors keyed by feature name."
      ]
    },
    {
      "metadata": {
        "id": "lT6v2RM_EB1E",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Make functions to add additional features\n",
        "\n",
        "# Compute normalized difference of two inputs.  If denomenator is zero, add a small delta.\n",
        "def normalizedDifference(a, b):\n",
        "  nd = (a - b) / (a + b)\n",
        "  nd_inf = (a - b) / (a + b + 0.000001)\n",
        "  return tf.where(tf.is_finite(nd), nd, nd_inf)\n",
        "\n",
        "# Add normalized differences and 3-D coordinates to the dataset.  Shift the label to zero.\n",
        "def addFeatures(features, label):\n",
        "  features['NDVI'] = normalizedDifference(features['B5'], features['B4'])\n",
        "  return features, label"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sz7kX5alFi4K",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now we need to define an input function that will feed data from a file into a TensorFlow model.  Putting together what we've done so far, here is the complete function for input, parsing and feature engineering:"
      ]
    },
    {
      "metadata": {
        "id": "rktz1DqqFlh_",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Make an input function\n",
        "\n",
        "def tfrecord_input_fn(fileName,\n",
        "                      numEpochs=None,\n",
        "                      shuffle=True,\n",
        "                      batchSize=None):\n",
        "\n",
        "  dataset = tf.data.TFRecordDataset(fileName, compression_type='GZIP')\n",
        "\n",
        "  # Map the parsing function over the dataset\n",
        "  dataset = dataset.map(parse_tfrecord, num_parallel_calls=5)\n",
        "\n",
        "  # Add additional features.\n",
        "  dataset = dataset.map(addFeatures)\n",
        "\n",
        "  # Shuffle, batch, and repeat.\n",
        "  if shuffle:\n",
        "    dataset = dataset.shuffle(buffer_size=batchSize * 10)\n",
        "  dataset = dataset.batch(batchSize)\n",
        "  dataset = dataset.repeat(numEpochs)\n",
        "\n",
        "  # Make a one-shot iterator.\n",
        "  iterator = dataset.make_one_shot_iterator()\n",
        "  features, labels = iterator.get_next()\n",
        "  return features, labels"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "t9pWa54oG-xl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The classifier we will use is a deep neural network (DNN) from the [`tf.estimator` package](https://www.tensorflow.org/api_docs/python/tf/estimator).  ([Slide](https://docs.google.com/presentation/d/1fEf-oScgbC9zjbzI3K3jUlHf4JdmoSLFiG_H491FUmk/edit#slide=id.g3b76860e75_0_59)).  First, define the input features, including the newly created NDVI column.  Here we specify an optimizer so that we can also set the learning rate.  Specify 7 nodes in the first hidden layer, 7 in the second and 5 in the third.  These are arbitrary demonstration numbers.  \n",
        "\n",
        "Lastly, train the classifier.  In order to pass the classifier a single argument input function, use a lambda function to specify the number of epochs and batch size.  You could also specify the number of training steps here where steps = N / batchSize for a single epoch ([reference](https://developers.google.com/machine-learning/glossary/#epoch))."
      ]
    },
    {
      "metadata": {
        "id": "OCZq3VNpG--G",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Make and train a classifier\n",
        "\n",
        "inputColumns = {tf.feature_column.numeric_column(k) for k in ['B2', 'B3', 'B4', 'B5', 'B6', 'B7', 'NDVI']}\n",
        "\n",
        "learning_rate = 0.05\n",
        "optimizer = tf.train.AdagradOptimizer(learning_rate)\n",
        "\n",
        "classifier = tf.estimator.DNNClassifier(feature_columns=inputColumns,\n",
        "                                  hidden_units=[5, 7, 5],\n",
        "                                  n_classes=3,\n",
        "                                  model_dir='output',\n",
        "                                  optimizer=optimizer)\n",
        "\n",
        "classifier.train(input_fn=lambda: tfrecord_input_fn(fileName=tfrTrainFile, numEpochs=8, batchSize=1))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Pa4ex_4eKiyb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now that we have a trained classifier, we can evaluate it using the test set.  To do that, use the same input function on a different file.  Since this is the test set, just use one epoch and don't shuffle.  Here we just print the overall accuracy."
      ]
    },
    {
      "metadata": {
        "id": "tE6d7FsrMa1p",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Evaluate the classifier\n",
        "\n",
        "accuracy_score = classifier.evaluate(\n",
        "    input_fn=lambda: tfrecord_input_fn(fileName=tfrTestFile, numEpochs=1, batchSize=1, shuffle=False)\n",
        ")['accuracy']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kUYADjdrc4Ie",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Training an estimator triggers storage of the state of the final model.  Unless you want subsequent runs to update previous model state, you may want to run the following (you will have to uncomment it first) to get rid of old model output.  Use with caution!"
      ]
    },
    {
      "metadata": {
        "id": "WSEnOAdGcYxL",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Optionally delete model output\n",
        "\n",
        "# !rm -rf output"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lGD4hzhAb4PR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Optional**.  The following code cell checks that the classifier can work by training it and testing it on the files stored in the cloud storage bucket.  To see the code, toggle the form with the control to the right."
      ]
    },
    {
      "metadata": {
        "id": "9nBrxvvCUDbi",
        "colab_type": "code",
        "colab": {},
        "cellView": "form"
      },
      "cell_type": "code",
      "source": [
        "#@title Optional Cloud Storage way (No need to run)\n",
        "\n",
        "inputColumns = {tf.feature_column.numeric_column(k) for k in ['B2', 'B3', 'B4', 'B5', 'B6', 'B7', 'NDVI']}\n",
        "\n",
        "learning_rate = 0.05\n",
        "optimizer = tf.train.AdagradOptimizer(learning_rate)\n",
        "\n",
        "classifier = tf.estimator.DNNClassifier(feature_columns=inputColumns,\n",
        "                                  hidden_units=[5, 7, 5],\n",
        "                                  n_classes=3,\n",
        "                                  model_dir='output',\n",
        "                                  optimizer=optimizer)\n",
        "\n",
        "# TensorFlow can read directly from a cloud storage location, so all we need to do is specify the path.\n",
        "tfrTrainFileCloud = 'gs://nclinton-training-temp/tf_demo_train_9a26cef21ab34f6257d0a250882124fcee_export.tfrecord.gz'\n",
        "tfrTestFileCloud = 'gs://nclinton-training-temp/tf_demo_test_9a26cef21ab34f6257d0a250882124fcee_export.tfrecord.gz'\n",
        "\n",
        "# Just check that you can see file(s):\n",
        "print tf.gfile.Exists(tfrTrainFileCloud)\n",
        "print tf.gfile.Exists(tfrTestFileCloud)\n",
        "\n",
        "# Train and test, passing the cloud storage path into the input function. \n",
        "classifier.train(input_fn=lambda: tfrecord_input_fn(fileName=tfrTrainFileCloud, numEpochs=8, batchSize=1))\n",
        "accuracy_score = classifier.evaluate(\n",
        "    input_fn=lambda: tfrecord_input_fn(fileName=tfrTestFileCloud, numEpochs=1, batchSize=1, shuffle=False)\n",
        ")['accuracy']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Nej2HDZM-G0H",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Get predictions on the evaluation dataset.  Note that we're going to make two iterators for this dataset.  The first one is just to see what's in there, to do a sanity check on the output.  We'll use the second one, below, to write the predictions to a TFRecord file.  \n",
        "\n",
        "Note that you can get both the predicted class and support probabilities for that classification. "
      ]
    },
    {
      "metadata": {
        "id": "12jRDVG4BpJ8",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Make predictions on the test data\n",
        "\n",
        "import itertools\n",
        "\n",
        "# Do the prediction from the trained classifier.\n",
        "checkPredictions = classifier.predict(\n",
        "  input_fn=lambda: tfrecord_input_fn(fileName=tfrTestFile, numEpochs=1, batchSize=1, shuffle=False)\n",
        ")\n",
        "\n",
        "# Make a couple iterators.\n",
        "iterator1, iterator2 = itertools.tee(checkPredictions, 2)\n",
        "\n",
        "# Iterate over the predictions, printing the class_ids and posteriors.\n",
        "for pred_dict in iterator1:\n",
        "  class_id = pred_dict['class_ids']\n",
        "  probability = pred_dict['probabilities']\n",
        "  print class_id, probability"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YRJsMMq_h7ZM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Optional**.  To write into a TFRecord file, it helps to have alittle understanding of how the records are stored.   This next example is to practice building a single record and writing it.  Specifically, define a `tf.train.Example` [protocol buffer](https://developers.google.com/protocol-buffers/) and write it to a file."
      ]
    },
    {
      "metadata": {
        "id": "66V-P88AjKRb",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Demonstration of writing an Example\n",
        "\n",
        "checkFilename = 'check.TFRecord'\n",
        "writer = tf.python_io.TFRecordWriter(checkFilename)\n",
        "\n",
        "example = tf.train.Example(\n",
        "    features=tf.train.Features(\n",
        "      feature={\n",
        "          'prediction': tf.train.Feature(\n",
        "              int64_list=tf.train.Int64List(\n",
        "                  value=[1])),\n",
        "          'posteriors': tf.train.Feature(\n",
        "              float_list=tf.train.FloatList(\n",
        "                  value=[0.1, 0.2, 0.3]))\n",
        "      }\n",
        "))\n",
        "\n",
        "writer.write(example.SerializeToString())\n",
        "writer.flush()\n",
        "writer.close()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_BtyFe6Oc7jo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now let's check that we can read our example back out of the file."
      ]
    },
    {
      "metadata": {
        "id": "No8107mzrpbc",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Demonstration of reading an Example\n",
        "\n",
        "checkDataset = tf.data.TFRecordDataset('check.TFRecord')\n",
        "\n",
        "checkDict = {\n",
        "    'prediction': tf.FixedLenFeature(shape=[1], dtype=tf.int64),\n",
        "    'posteriors': tf.FixedLenFeature(shape=[3], dtype=tf.float32),\n",
        "}\n",
        "\n",
        "checkParsed = checkDataset.map(\n",
        "    lambda example_proto: tf.parse_single_example(example_proto, checkDict))\n",
        "\n",
        "iterator = checkParsed.make_one_shot_iterator()\n",
        "foo = iterator.get_next()\n",
        "with tf.Session() as sess:\n",
        "    print sess.run([foo])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8iW4iPSkrrHb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now iterate over the predictions on the test data and try writing all those into a file.  For each prediction, we make a new `tf.Example` proto out of the prediction data, then write it.  Finally execute a shell command to see if we've successfully written the file.\n",
        "\n",
        "See:\n",
        "\n",
        "https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/core/example/feature.proto\n",
        "\n",
        "https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/how_tos/reading_data/convert_to_records.py"
      ]
    },
    {
      "metadata": {
        "id": "maMPR8yurm-j",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Demonstration of writing predictions to a file\n",
        "\n",
        "outputFilename = 'checkPredictions.TFRecord'\n",
        "writer = tf.python_io.TFRecordWriter(outputFilename)\n",
        "  \n",
        "for pred_dict in iterator2:\n",
        "  example = tf.train.Example(\n",
        "    features=tf.train.Features(\n",
        "      \n",
        "      feature={\n",
        "          'prediction': tf.train.Feature(\n",
        "              int64_list=tf.train.Int64List(\n",
        "                  value=pred_dict['class_ids'])),\n",
        "          'probabilities': tf.train.Feature(\n",
        "              float_list=tf.train.FloatList(\n",
        "                  value=pred_dict['probabilities']))\n",
        "      }\n",
        "  ))\n",
        "  writer.write(example.SerializeToString())\n",
        "         \n",
        "writer.close()\n",
        "!ls -Al"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nhHrnv3VR0DU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now it's time to classify the image from Earth Engine.  The way this happens is by exporting an image as a TFRecord file [announcement doc](https://docs.google.com/document/d/1njY_MKvXELEWvDaXmA56TFSteeiSytkziHBD9Pr8Q9I/edit?usp=sharing).  [The script for exporting the training and testing data](https://code.earthengine.google.com/a7ed957f3034825a54b6b546b8c5ce83) also exports a piece of the composite for classification.  Specifically, `Export.image` now accepts `'TFRecord'` for `fileFormat`. \n",
        "\n",
        "Theres some other new stuff in that export.  Specifically, note that we're exporting pixels in 256x256 patches for efficiency.  Also note that the image gets split into multiple TFRecord files in its destination folder."
      ]
    },
    {
      "metadata": {
        "id": "BdQrQIzKvEqp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Because there are multiple files that make up the image, use the Google PyDrive library to search for the files that match a particular prefix string.  Specifically, this is the name you specified in the JavaScript for the exported files.  Download all the Drive files that match that field, one of which is the JSON that we don't need as input to the model (but will need for import to Earth Engine after we've made predictions).  Lastly, print the list of filenames for a reality check.\n",
        "\n",
        "See https://pythonhosted.org/PyDrive/filelist.html for pyDrive docs.  Here's where you can find the info on that query expression: https://developers.google.com/drive/api/v2/search-parameters#file_fields"
      ]
    },
    {
      "metadata": {
        "id": "R_7a_r9tuo1U",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Find the exported image and JSON files in Drive\n",
        "\n",
        "file_list = drive.ListFile({\n",
        "    # You have to know this base filename from wherever you did the export.\n",
        "    'q': 'title contains \"tf_demo_image_9a26cef21ab34f6257d0a250882124fc\"'\n",
        "}).GetList()\n",
        "\n",
        "fileNames = []\n",
        "jsonFile = None\n",
        "for gDriveFile in file_list:\n",
        "  title = gDriveFile['title']\n",
        "  # Download to the notebook server VM.\n",
        "  gDriveFile.GetContentFile(title)\n",
        "  # If the filename contains .gz, it's part of the image.\n",
        "  if (title.find('gz') > 0):\n",
        "    fileNames.append(gDriveFile['title'])\n",
        "  if (title.find('json') > 0):\n",
        "    jsonFile = title\n",
        "\n",
        "# Make sure the files are in the right order.\n",
        "fileNames.sort()\n",
        "\n",
        "# Check the list of filenames to ensure there's nothing unintentional in there.\n",
        "print(fileNames)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PCnHyu53rQs4",
        "colab_type": "code",
        "colab": {},
        "cellView": "form"
      },
      "cell_type": "code",
      "source": [
        "#@title Optional Cloud Storage way (No need to run)\n",
        "# We'll need this for importing the classified image back to Earth Engine.\n",
        "jsonFile = 'gs://nclinton-training-temp/tf_demo_image_9a26cef21ab34f6257d0a250882124fcmixer.json'\n",
        "\n",
        "# TensorFlow can read directly from a cloud storage bucket.\n",
        "# Ensure that the files are in order.\n",
        "fileNames = [\n",
        "    'gs://nclinton-training-temp/tf_demo_image_9a26cef21ab34f6257d0a250882124fc00000.tfrecord.gz',\n",
        "    'gs://nclinton-training-temp/tf_demo_image_9a26cef21ab34f6257d0a250882124fc00001.tfrecord.gz'\n",
        "]\n",
        "print(fileNames)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6xyzyPPJwpVI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can feed this list of files directly to the Dataset constructor to make a combined dataset.  However, the the input function is slightly different from the previous ones.  Mainly, this is because the pixels are written into records as patches, we need to read the patches in as one big tensor (one patch for each band), then flatten them into lots of little tensors.  Once the input function is defined that can handle the shape of the image data, all you need to do is feed it directly to the trained model to make predictions."
      ]
    },
    {
      "metadata": {
        "id": "tn8Kj3VfwpiJ",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Make an input function for exported image data\n",
        "\n",
        "# You have to know the following from your export.\n",
        "PATCH_WIDTH = 256\n",
        "PATCH_HEIGHT = 256\n",
        "PATCH_DIMENSIONS_FLAT = [PATCH_WIDTH * PATCH_HEIGHT, 1]\n",
        "\n",
        "bands = ['B2', 'B3', 'B4', 'B5', 'B6', 'B7']\n",
        "\n",
        "# Note that the tensors are in the shape of a patch, one patch for each band.\n",
        "columns = [\n",
        "  tf.FixedLenFeature(shape=PATCH_DIMENSIONS_FLAT, dtype=tf.float32) for k in bands\n",
        "]\n",
        "\n",
        "featuresDict = dict(zip(bands, columns))\n",
        "\n",
        "# This function adds NDVI to a feature that doesn't have a label.\n",
        "def addServerFeatures(features):\n",
        "  return addFeatures(features, None)[0]\n",
        "    \n",
        "# This input function reads in the TFRecord files exported from an image.\n",
        "# Note that because the pixels are arranged in patches, we need some additional\n",
        "# code to reshape the tensors.\n",
        "def predict_input_fn(fileNames):\n",
        "  \n",
        "  # Note that you can make one dataset from many files by specifying a list.\n",
        "  dataset = tf.data.TFRecordDataset(fileNames, compression_type='GZIP')\n",
        "  \n",
        "  def parse_image(example_proto):\n",
        "    parsed_features = tf.parse_single_example(example_proto, featuresDict)\n",
        "    return parsed_features\n",
        "  \n",
        "  dataset = dataset.map(parse_image, num_parallel_calls=5)\n",
        "\n",
        "  # Break our long tensors into many littler ones\n",
        "  dataset = dataset.flat_map(lambda features: tf.data.Dataset.from_tensor_slices(features))\n",
        "  \n",
        "  # Add additional features (NDVI).\n",
        "  dataset = dataset.map(addServerFeatures)\n",
        "  \n",
        "  # Read in batches corresponding to patch size.\n",
        "  dataset = dataset.batch(PATCH_WIDTH * PATCH_HEIGHT)\n",
        "  \n",
        "  # Make a one-shot iterator.\n",
        "  iterator = dataset.make_one_shot_iterator()\n",
        "  return iterator.get_next()\n",
        "\n",
        "# Do the prediction from the trained classifier.\n",
        "predictions = classifier.predict(\n",
        "  input_fn=lambda: predict_input_fn(fileNames)\n",
        ")\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RNwwKeGtkhh_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Name the TFRecord file you're going to create with a unique identifier for you (like your username).  We'll write this file directly into a temporary cloud storage bucket created for this training.  *The bucket will be deleted daily, so don't store anything in there*."
      ]
    },
    {
      "metadata": {
        "id": "xuQ1s6YJhwOP",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Define output names\n",
        "\n",
        "# INSERT YOUR USERNAME HERE (e.g. nclinton):\n",
        "username = ''\n",
        "baseName = 'gs://nclinton-training-temp/' + username\n",
        "outputImageFile = baseName + '_predictions.TFRecord'\n",
        "outputJsonFile = baseName + '_predictions.json'\n",
        "print 'Writing to: ' + outputImageFile"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bPU2VlPOikAy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We already have the predictions as a list.  Iterate over them as we did previously, except with some additional code to handle the shape.  Specifically, we need to write the pixels into the file as patches in the same order they came out.  (Note: 5,620,989 pixels)"
      ]
    },
    {
      "metadata": {
        "id": "kATMknHc0qeR",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Make predictions on the image data, write to a file\n",
        "\n",
        "iter1, iter2 = itertools.tee(predictions, 2)\n",
        "\n",
        "# Iterate over the predictions, printing the class_ids and posteriors.\n",
        "# This is just to examine the first prediction.\n",
        "for pred_dict in iter1:\n",
        "  print pred_dict\n",
        "  break # OK\n",
        "\n",
        "# Instantiate the writer.\n",
        "writer = tf.python_io.TFRecordWriter(outputImageFile)\n",
        "  \n",
        "# Every patch-worth of predictions we'll dump an example into the output\n",
        "# file with a single feature that holds our predictions. Since are predictions\n",
        "# are already in the order of the exported data, our patches we create here\n",
        "# will also be in the right order.\n",
        "patch = [[], [], [], []]\n",
        "curPatch = 1\n",
        "for pred_dict in iter2:\n",
        "  patch[0].append(pred_dict['class_ids'])\n",
        "  patch[1].append(pred_dict['probabilities'][0])\n",
        "  patch[2].append(pred_dict['probabilities'][1])\n",
        "  patch[3].append(pred_dict['probabilities'][2])\n",
        "  # Once we've seen a patches-worth of class_ids...\n",
        "  if (len(patch[0]) == PATCH_WIDTH * PATCH_HEIGHT):\n",
        "    print('Done with patch ' + str(curPatch) + '...')\n",
        "    # Create an example\n",
        "    example = tf.train.Example(\n",
        "      features=tf.train.Features(\n",
        "        feature={\n",
        "          'prediction': tf.train.Feature(\n",
        "              int64_list=tf.train.Int64List(\n",
        "                  value=patch[0])),\n",
        "          'bareProb': tf.train.Feature(\n",
        "              float_list=tf.train.FloatList(\n",
        "                  value=patch[1])),\n",
        "          'vegProb': tf.train.Feature(\n",
        "              float_list=tf.train.FloatList(\n",
        "                  value=patch[2])),\n",
        "          'waterProb': tf.train.Feature(\n",
        "              float_list=tf.train.FloatList(\n",
        "                  value=patch[3])),\n",
        "            \n",
        "        }\n",
        "      )\n",
        "    )\n",
        "    # Write the example to the file and clear our patch array so it's ready for\n",
        "    # another batch of class ids\n",
        "    writer.write(example.SerializeToString())\n",
        "    patch = [[], [], [], []]\n",
        "    curPatch += 1 \n",
        "\n",
        "writer.close()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "M6sNZXWOSa82",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note that you should also move the JSON file downloaded earlier and give it the same base name as the TFRecord file with the predictions in it.  It's not necessary to do this, but will be helpful in the upload to Earth Engine command.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "q6yLaut2UYPp",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Copy the JSON file to a cloud storage bucket\n",
        "\n",
        "# Copy the JSON file so it has the same base name as the image.\n",
        "!gsutil cp {jsonFile} {outputJsonFile}\n",
        "!gsutil ls gs://nclinton-training-temp"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ScjihzmEn1EV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Almost there!  Now we have a predictions image, sitting in a cloud storage bucket.  The purpose of doing it this way is to enable us to upload the image to Earth Engine from the cloud storage bucket.  This can be accomplished with the [Earth Engine command line tool](https://developers.google.com/earth-engine/command_line#upload).  But first we need to install the Earth Engine API and authenticate."
      ]
    },
    {
      "metadata": {
        "id": "8TLqch_Bjz92",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Install the Earth Engine API\n",
        "\n",
        "!pip install earthengine-api\n",
        "!earthengine authenticate --quiet"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ejxa1MQjEGv9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Follow the link in the output above, copy the authorization link into the code cell below and run it to authenticate Earth Engine."
      ]
    },
    {
      "metadata": {
        "id": "LtLqeBZMljga",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Authentication for Earth Engine\n",
        "\n",
        "!earthengine authenticate --authorization-code=<YOUR CODE HERE>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "cgyvGbBxD9-Z",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's just test the `earthengine` command by looking for help on the upload command."
      ]
    },
    {
      "metadata": {
        "id": "HzwiVqbcmJIX",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Get earthengine upload help\n",
        "\n",
        "!earthengine upload image -h"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2ZyCo297Clcx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now we're ready to move the image file back to Earth Engine.  Note that we give both the image TFRecord file and the JSON file as arguments to `earthengine upload`.  Here's where it's useful to copy the JSON file to have a consistent basename with the image."
      ]
    },
    {
      "metadata": {
        "id": "NXulMNl9lTDv",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Upload the classified image to Earth Engine\n",
        "\n",
        "# Change the filenames to match your personal user folder in Earth Engine.\n",
        "# e.g. users/nclinton/TF_nclinton_predictions\n",
        "outputAssetID = 'users/nclinton/TF_foobar_predictions' \n",
        "\n",
        "!earthengine upload image --asset_id={outputAssetID} {outputImageFile} {outputJsonFile}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_vB-gwGhl_3C",
        "colab_type": "code",
        "colab": {},
        "cellView": "code"
      },
      "cell_type": "code",
      "source": [
        "#@title Check the status of the asset ingestion\n",
        "\n",
        "import ee\n",
        "ee.Initialize()\n",
        "\n",
        "tasks = ee.batch.Task.list()\n",
        "print tasks"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qMaPrK6OCxxA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Check the output in Earth Engine (nclinton version): https://code.earthengine.google.com/47ba19eedba20fad5d3df28fa2c4be1c  "
      ]
    }
  ]
}