20_Natural_Language_Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Tutorial #20\n",
    "# Natural Language Processing\n",
    "\n",
    "by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "/ [GitHub](https://github.com/Hvass-Labs/TensorFlow-Tutorials) / [Videos on YouTube](https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This tutorial is about a basic form of Natural Language Processing (NLP) called Sentiment Analysis, in which we will try and classify a movie review as either positive or negative.\n",
    "\n",
    "Consider a simple example: \"This movie is not very good.\" This text ends with the words \"very good\" which indicates a very positive sentiment, but it is negated because it is preceded by the word \"not\", so the text should be classified as having a negative sentiment. How can we teach a Neural Network to do this classification?\n",
    "\n",
    "Another problem is that neural networks cannot work directly on text-data, so we need to convert text into numbers that are compatible with a neural network.\n",
    "\n",
    "Yet another problem is that a text may be arbitrarily long. The neural networks we have worked with in previous tutorials use fixed data-shapes - except for the first dimension of the data which varies with the batch-size. Now we need a type of neural network that can work on both short and long sequences of text.\n",
    "\n",
    "You should be familiar with TensorFlow and Keras in general, see Tutorials #01 and #03-C."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flowchart\n",
    "\n",
    "To solve this problem we need several processing steps. First we need to convert the raw text-words into so-called tokens which are integer values. These tokens are really just indices into a list of the entire vocabulary. Then we convert these integer-tokens into so-called embeddings which are real-valued vectors, whose mapping will be trained along with the neural network, so as to map words with similar meanings to similar embedding-vectors. Then we input these embedding-vectors to a Recurrent Neural Network which can take sequences of arbitrary length as input and output a kind of summary of what it has seen in the input. This output is then squashed using a Sigmoid-function to give us a value between 0.0 and 1.0, where 0.0 is taken to mean a negative sentiment and 1.0 means a positive sentiment. This whole process allows us to classify input-text as either having a negative or positive sentiment.\n",
    "\n",
    "The flowchart of the algorithm is roughly:\n",
    "\n",
    "<img src=\"images/20_natural_language_flowchart.png\" alt=\"Flowchart NLP\" style=\"width: 300px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent Neural Network\n",
    "\n",
    "The basic building block in a Recurrent Neural Network (RNN) is a Recurrent Unit (RU). There are many different variants of recurrent units such as the rather clunky LSTM (Long-Short-Term-Memory) and the somewhat simpler GRU (Gated Recurrent Unit) which we will use in this tutorial. Experiments in the literature suggest that the LSTM and GRU have roughly similar performance. Even simpler variants also exist and the literature suggests that they may perform even better than both LSTM and GRU, but they are not implemented in Keras which we will use in this tutorial.\n",
    "\n",
    "The following figure shows the abstract idea of a recurrent unit, which has an internal state that is being updated every time the unit receives a new input. This internal state serves as a kind of memory. However, it is not a traditional kind of computer memory which stores bits that are either on or off. Instead the recurrent unit stores floating-point values in its memory-state, which are read and written using matrix-operations so the operations are all differentiable. This means the memory-state can store arbitrary floating-point values (although typically limited between -1.0 and 1.0) and the network can be trained like a normal neural network using Gradient Descent.\n",
    "\n",
    "The new state-value depends on both the old state-value and the current input. For example, if the state-value has memorized that we have recently seen the word \"not\" and the current input is \"good\" then we need to store a new state-value that memorizes \"not good\" which indicates a negative sentiment.\n",
    "\n",
    "The part of the recurrent unit that is responsible for mapping old state-values and inputs to the new state-value is called a gate, but it is really just a type of matrix-operation. There is another gate for calculating the output-values of the recurrent unit. The implementation of these gates vary for different types of recurrent units. This figure merely shows the abstract idea of a recurrent unit. The LSTM has more gates than the GRU but some of them are apparently redundant so they can be omitted.\n",
    "\n",
    "In order to train the recurrent unit, we must gradually change the weight-matrices of the gates so the recurrent unit gives the desired output for an input sequence. This is done automatically in TensorFlow.\n",
    "\n",
    "![Recurrent unit](images/20_recurrent_unit.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unrolled Network\n",
    "\n",
    "Another way to visualize and understand a Recurrent Neural Network is to \"unroll\" the recursion. In this figure there is only a single recurrent unit denoted RU, which will receive a text-word from the input sequence in a series of time-steps.\n",
    "\n",
    "The initial memory-state of the RU is reset to zero internally by Keras / TensorFlow every time a new sequence begins.\n",
    "\n",
    "In the first time-step the word \"this\" is input to the RU which uses its internal state (initialized to zero) and its gate to calculate the new state. The RU also uses its other gate to calculate the output but it is ignored here because it is only needed at the end of the sequence to output a kind of summary.\n",
    "\n",
    "In the second time-step the word \"is\" is input to the RU which now uses the internal state that was just updated from seeing the previous word \"this\".\n",
    "\n",
    "There is not much meaning in the words \"this is\" so the RU probably doesn't save anything important in its internal state from seeing these words. But when it sees the third word \"not\" the RU has learned that it may be important for determining the overall sentiment of the input-text, so it needs to be stored in the memory-state of the RU, which can be used later when the RU sees the word \"good\" in time-step 6.\n",
    "\n",
    "Finally when the entire sequence has been processed, the RU outputs a vector of values that summarizes what it has seen in the input sequence. We then use a fully-connected layer with a Sigmoid activation to get a single value between 0.0 and 1.0 which we interpret as the sentiment either being negative (values close to 0.0) or positive (values close to 1.0).\n",
    "\n",
    "Note that for the sake of clarity, this figure doesn't show the mapping from text-words to integer-tokens and embedding-vectors, as well as the fully-connected Sigmoid layer on the output.\n",
    "\n",
    "![Unrolled network](images/20_unrolled_flowchart.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-Layer Unrolled Network\n",
    "\n",
    "In this tutorial we will use a Recurrent Neural Network with 3 recurrent units (or layers) denoted RU1, RU2 and RU3 in the \"unrolled\" figure below.\n",
    "\n",
    "The first layer is much like the unrolled figure above for a single-layer RNN. First the recurrent unit RU1 has its internal state initialized to zero by Keras / TensorFlow. Then the word \"this\" is input to RU1 and it updates its internal state. Then it processes the next word \"is\", and so forth. But instead of outputting a single summary value at the end of the sequence, we use the output of RU1 for every time-step. This creates a new sequence that can then be used as input for the next recurrent unit RU2. The same process is repeated for the second layer and this creates a new output sequence which is then input to the third layer's recurrent unit RU3, whose final output is passed to a fully-connected Sigmoid layer that outputs a value between 0.0 (negative sentiment) and 1.0 (positive sentiment).\n",
    "\n",
    "Note that for the sake of clarity, the mapping of text-words to integer-tokens and embedding-vectors has been omitted from this figure.\n",
    "\n",
    "![Unrolled 3-layer network](images/20_unrolled_3layers_flowchart.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploding & Vanishing Gradients\n",
    "\n",
    "In order to train the weights for the gates inside the recurrent unit, we need to minimize some loss-function which measures the difference between the actual output of the network relative to the desired output.\n",
    "\n",
    "From the \"unrolled\" figures above we see that the reccurent units are applied recursively for each word in the input sequence. This means the recurrent gate is applied once for each time-step. The gradient-signals have to flow back from the loss-function all the way to the first time the recurrent gate is used. If the gradient of the recurrent gate is multiplicative, then we essentially have an exponential function.\n",
    "\n",
    "In this tutorial we will use texts that have more than 500 words. This means the RU's gate for updating its internal memory-state is applied recursively more than 500 times. If a gradient of just 1.01 is multiplied with itself 500 times then it gives a value of about 145. If a gradient of just 0.99 is multiplied with itself 500 times then it gives a value of about 0.007. These are called exploding and vanishing gradients. The only gradients that can survive recurrent multiplication are 0 and 1.\n",
    "\n",
    "To avoid these so-called exploding and vanishing gradients, care must be made when designing the recurrent unit and its gates. That is why the actual implementation of the GRU is more complicated, because it tries to send the gradient back through the gates without this distortion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/magnus/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from scipy.spatial.distance import cdist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to import several things from Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from tf.keras.models import Sequential  # This does not work!\n",
    "from tensorflow.python.keras.models import Sequential\n",
    "from tensorflow.python.keras.layers import Dense, GRU, Embedding\n",
    "from tensorflow.python.keras.optimizers import Adam\n",
    "from tensorflow.python.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.python.keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was developed using Python 3.6 (Anaconda) and package versions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.5.0'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.1.2-tf'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "\n",
    "We will use a data-set consisting of 50000 reviews of movies from IMDB. Keras has a built-in function for downloading a similar data-set (but apparently half the size). However, Keras' version has already converted the text in the data-set to integer-tokens, which is a crucial part of working with natural languages that will also be demonstrated in this tutorial, so we download the actual text-data.\n",
    "\n",
    "NOTE: The data-set is 84 MB and will be downloaded automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import imdb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change this if you want the files saved in another directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imdb.data_dir = \"data/IMDB/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Automatically download and extract the files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data has apparently already been downloaded and unpacked.\n"
     ]
    }
   ],
   "source": [
    "imdb.maybe_download_and_extract()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the training- and test-sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_text, y_train = imdb.load_data(train=True)\n",
    "x_test_text, y_test = imdb.load_data(train=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train-set size:  25000\n",
      "Test-set size:   25000\n"
     ]
    }
   ],
   "source": [
    "print(\"Train-set size: \", len(x_train_text))\n",
    "print(\"Test-set size:  \", len(x_test_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine into one data-set for some uses below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_text = x_train_text + x_test_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print an example from the training-set to see that the data looks correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A simple comment...<br /><br />What can I say... this is a wonderful film that I can watch over and over. It is definitely one of the top ten comedies made. With a great cast, Jack Lemmon and Walter Matthau wording a perfect script by Neil Simon, based on his play.<br /><br />It is real to life situation done perfectly. If you have digital cable, one gets the menu on bottom of screen to give what is on. It usually gives this film ***% stars but in reality it deserves **** stars. If you really watch this film, one can tell that it will be as funny and fresh a hundred years from now.'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_text[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The true \"class\" is a sentiment of the movie-review. It is a value of 0.0 for a negative sentiment and 1.0 for a positive sentiment. In this case the review is positive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenizer\n",
    "\n",
    "A neural network cannot work directly on text-strings so we must convert it somehow. There are two steps in this conversion, the first step is called the \"tokenizer\" which converts words to integers and is done on the data-set before it is input to the neural network. The second step is an integrated part of the neural network itself and is called the \"embedding\"-layer, which is described further below.\n",
    "\n",
    "We may instruct the tokenizer to only use e.g. the 10000 most popular words from the data-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_words = 10000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(num_words=num_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tokenizer can then be \"fitted\" to the data-set. This scans through all the text and strips it from unwanted characters such as punctuation, and also converts it to lower-case characters. The tokenizer then builds a vocabulary of all unique words along with various data-structures for accessing the data.\n",
    "\n",
    "Note that we fit the tokenizer on the entire data-set so it gathers words from both the training- and test-data. This is OK as we are merely building a vocabulary and want it to be as complete as possible. The actual neural network will of course only be trained on the training-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 10.6 s, sys: 16 ms, total: 10.6 s\n",
      "Wall time: 10.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "tokenizer.fit_on_texts(data_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to use the entire vocabulary then set `num_words=None` above, and then it will automatically be set to the vocabulary-size here. (This is because of Keras' somewhat awkward implementation.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "if num_words is None:\n",
    "    num_words = len(tokenizer.word_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then inspect the vocabulary that has been gathered by the tokenizer. This is ordered by the number of occurrences of the words in the data-set. These integer-numbers are called word indices or \"tokens\" because they uniquely identify each word in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'the': 1,\n",
       " 'and': 2,\n",
       " 'a': 3,\n",
       " 'of': 4,\n",
       " 'to': 5,\n",
       " 'is': 6,\n",
       " 'br': 7,\n",
       " 'in': 8,\n",
       " 'it': 9,\n",
       " 'i': 10,\n",
       " 'this': 11,\n",
       " 'that': 12,\n",
       " 'was': 13,\n",
       " 'as': 14,\n",
       " 'for': 15,\n",
       " 'with': 16,\n",
       " 'movie': 17,\n",
       " 'but': 18,\n",
       " 'film': 19,\n",
       " 'on': 20,\n",
       " 'not': 21,\n",
       " 'you': 22,\n",
       " 'are': 23,\n",
       " 'his': 24,\n",
       " 'have': 25,\n",
       " 'be': 26,\n",
       " 'one': 27,\n",
       " 'he': 28,\n",
       " 'all': 29,\n",
       " 'at': 30,\n",
       " 'by': 31,\n",
       " 'an': 32,\n",
       " 'they': 33,\n",
       " 'so': 34,\n",
       " 'who': 35,\n",
       " 'from': 36,\n",
       " 'like': 37,\n",
       " 'or': 38,\n",
       " 'just': 39,\n",
       " 'her': 40,\n",
       " 'out': 41,\n",
       " 'about': 42,\n",
       " 'if': 43,\n",
       " \"it's\": 44,\n",
       " 'has': 45,\n",
       " 'there': 46,\n",
       " 'some': 47,\n",
       " 'what': 48,\n",
       " 'good': 49,\n",
       " 'when': 50,\n",
       " 'more': 51,\n",
       " 'very': 52,\n",
       " 'up': 53,\n",
       " 'no': 54,\n",
       " 'time': 55,\n",
       " 'my': 56,\n",
       " 'even': 57,\n",
       " 'would': 58,\n",
       " 'she': 59,\n",
       " 'which': 60,\n",
       " 'only': 61,\n",
       " 'really': 62,\n",
       " 'see': 63,\n",
       " 'story': 64,\n",
       " 'their': 65,\n",
       " 'had': 66,\n",
       " 'can': 67,\n",
       " 'me': 68,\n",
       " 'well': 69,\n",
       " 'were': 70,\n",
       " 'than': 71,\n",
       " 'much': 72,\n",
       " 'we': 73,\n",
       " 'bad': 74,\n",
       " 'been': 75,\n",
       " 'get': 76,\n",
       " 'do': 77,\n",
       " 'great': 78,\n",
       " 'other': 79,\n",
       " 'will': 80,\n",
       " 'also': 81,\n",
       " 'into': 82,\n",
       " 'people': 83,\n",
       " 'because': 84,\n",
       " 'how': 85,\n",
       " 'first': 86,\n",
       " 'him': 87,\n",
       " 'most': 88,\n",
       " \"don't\": 89,\n",
       " 'made': 90,\n",
       " 'then': 91,\n",
       " 'its': 92,\n",
       " 'them': 93,\n",
       " 'make': 94,\n",
       " 'way': 95,\n",
       " 'too': 96,\n",
       " 'movies': 97,\n",
       " 'could': 98,\n",
       " 'any': 99,\n",
       " 'after': 100,\n",
       " 'think': 101,\n",
       " 'characters': 102,\n",
       " 'watch': 103,\n",
       " 'films': 104,\n",
       " 'two': 105,\n",
       " 'many': 106,\n",
       " 'seen': 107,\n",
       " 'character': 108,\n",
       " 'being': 109,\n",
       " 'never': 110,\n",
       " 'plot': 111,\n",
       " 'love': 112,\n",
       " 'acting': 113,\n",
       " 'life': 114,\n",
       " 'did': 115,\n",
       " 'best': 116,\n",
       " 'where': 117,\n",
       " 'know': 118,\n",
       " 'show': 119,\n",
       " 'little': 120,\n",
       " 'over': 121,\n",
       " 'off': 122,\n",
       " 'ever': 123,\n",
       " 'does': 124,\n",
       " 'your': 125,\n",
       " 'better': 126,\n",
       " 'end': 127,\n",
       " 'man': 128,\n",
       " 'scene': 129,\n",
       " 'still': 130,\n",
       " 'say': 131,\n",
       " 'these': 132,\n",
       " 'here': 133,\n",
       " 'why': 134,\n",
       " 'scenes': 135,\n",
       " 'while': 136,\n",
       " 'something': 137,\n",
       " 'such': 138,\n",
       " 'go': 139,\n",
       " 'through': 140,\n",
       " 'back': 141,\n",
       " 'should': 142,\n",
       " 'those': 143,\n",
       " 'real': 144,\n",
       " \"i'm\": 145,\n",
       " 'now': 146,\n",
       " 'watching': 147,\n",
       " 'thing': 148,\n",
       " \"doesn't\": 149,\n",
       " 'actors': 150,\n",
       " 'though': 151,\n",
       " 'funny': 152,\n",
       " 'years': 153,\n",
       " \"didn't\": 154,\n",
       " 'old': 155,\n",
       " 'another': 156,\n",
       " '10': 157,\n",
       " 'work': 158,\n",
       " 'before': 159,\n",
       " 'actually': 160,\n",
       " 'nothing': 161,\n",
       " 'makes': 162,\n",
       " 'look': 163,\n",
       " 'director': 164,\n",
       " 'find': 165,\n",
       " 'going': 166,\n",
       " 'same': 167,\n",
       " 'new': 168,\n",
       " 'lot': 169,\n",
       " 'every': 170,\n",
       " 'few': 171,\n",
       " 'again': 172,\n",
       " 'part': 173,\n",
       " 'cast': 174,\n",
       " 'down': 175,\n",
       " 'us': 176,\n",
       " 'things': 177,\n",
       " 'want': 178,\n",
       " 'quite': 179,\n",
       " 'pretty': 180,\n",
       " 'world': 181,\n",
       " 'horror': 182,\n",
       " 'around': 183,\n",
       " 'seems': 184,\n",
       " \"can't\": 185,\n",
       " 'young': 186,\n",
       " 'take': 187,\n",
       " 'however': 188,\n",
       " 'got': 189,\n",
       " 'thought': 190,\n",
       " 'big': 191,\n",
       " 'fact': 192,\n",
       " 'enough': 193,\n",
       " 'long': 194,\n",
       " 'both': 195,\n",
       " \"that's\": 196,\n",
       " 'give': 197,\n",
       " \"i've\": 198,\n",
       " 'own': 199,\n",
       " 'may': 200,\n",
       " 'between': 201,\n",
       " 'comedy': 202,\n",
       " 'right': 203,\n",
       " 'series': 204,\n",
       " 'action': 205,\n",
       " 'must': 206,\n",
       " 'music': 207,\n",
       " 'without': 208,\n",
       " 'times': 209,\n",
       " 'saw': 210,\n",
       " 'always': 211,\n",
       " 'original': 212,\n",
       " \"isn't\": 213,\n",
       " 'role': 214,\n",
       " 'come': 215,\n",
       " 'almost': 216,\n",
       " 'gets': 217,\n",
       " 'interesting': 218,\n",
       " 'guy': 219,\n",
       " 'point': 220,\n",
       " 'done': 221,\n",
       " \"there's\": 222,\n",
       " 'whole': 223,\n",
       " 'least': 224,\n",
       " 'far': 225,\n",
       " 'bit': 226,\n",
       " 'script': 227,\n",
       " 'minutes': 228,\n",
       " 'feel': 229,\n",
       " '2': 230,\n",
       " 'anything': 231,\n",
       " 'making': 232,\n",
       " 'might': 233,\n",
       " 'since': 234,\n",
       " 'am': 235,\n",
       " 'family': 236,\n",
       " \"he's\": 237,\n",
       " 'last': 238,\n",
       " 'probably': 239,\n",
       " 'tv': 240,\n",
       " 'performance': 241,\n",
       " 'kind': 242,\n",
       " 'away': 243,\n",
       " 'yet': 244,\n",
       " 'fun': 245,\n",
       " 'worst': 246,\n",
       " 'sure': 247,\n",
       " 'rather': 248,\n",
       " 'hard': 249,\n",
       " 'girl': 250,\n",
       " 'anyone': 251,\n",
       " 'each': 252,\n",
       " 'played': 253,\n",
       " 'day': 254,\n",
       " 'found': 255,\n",
       " 'looking': 256,\n",
       " 'woman': 257,\n",
       " 'screen': 258,\n",
       " 'although': 259,\n",
       " 'our': 260,\n",
       " 'especially': 261,\n",
       " 'believe': 262,\n",
       " 'having': 263,\n",
       " 'trying': 264,\n",
       " 'course': 265,\n",
       " 'dvd': 266,\n",
       " 'everything': 267,\n",
       " 'set': 268,\n",
       " 'goes': 269,\n",
       " 'comes': 270,\n",
       " 'put': 271,\n",
       " 'ending': 272,\n",
       " 'maybe': 273,\n",
       " 'place': 274,\n",
       " 'book': 275,\n",
       " 'shows': 276,\n",
       " 'three': 277,\n",
       " 'worth': 278,\n",
       " 'different': 279,\n",
       " 'main': 280,\n",
       " 'once': 281,\n",
       " 'sense': 282,\n",
       " 'american': 283,\n",
       " 'reason': 284,\n",
       " 'looks': 285,\n",
       " 'effects': 286,\n",
       " 'watched': 287,\n",
       " 'play': 288,\n",
       " 'true': 289,\n",
       " 'money': 290,\n",
       " 'actor': 291,\n",
       " \"wasn't\": 292,\n",
       " 'job': 293,\n",
       " 'together': 294,\n",
       " 'war': 295,\n",
       " 'someone': 296,\n",
       " 'plays': 297,\n",
       " 'instead': 298,\n",
       " 'high': 299,\n",
       " 'during': 300,\n",
       " 'year': 301,\n",
       " 'said': 302,\n",
       " 'half': 303,\n",
       " 'everyone': 304,\n",
       " 'later': 305,\n",
       " 'takes': 306,\n",
       " '1': 307,\n",
       " 'seem': 308,\n",
       " 'audience': 309,\n",
       " 'special': 310,\n",
       " 'beautiful': 311,\n",
       " 'left': 312,\n",
       " 'himself': 313,\n",
       " 'seeing': 314,\n",
       " 'john': 315,\n",
       " 'night': 316,\n",
       " 'black': 317,\n",
       " 'version': 318,\n",
       " 'shot': 319,\n",
       " 'excellent': 320,\n",
       " 'idea': 321,\n",
       " 'house': 322,\n",
       " 'mind': 323,\n",
       " 'star': 324,\n",
       " 'wife': 325,\n",
       " 'fan': 326,\n",
       " 'death': 327,\n",
       " 'used': 328,\n",
       " 'else': 329,\n",
       " 'simply': 330,\n",
       " 'nice': 331,\n",
       " 'budget': 332,\n",
       " 'poor': 333,\n",
       " 'completely': 334,\n",
       " 'short': 335,\n",
       " 'second': 336,\n",
       " \"you're\": 337,\n",
       " '3': 338,\n",
       " 'read': 339,\n",
       " 'less': 340,\n",
       " 'along': 341,\n",
       " 'top': 342,\n",
       " 'help': 343,\n",
       " 'home': 344,\n",
       " 'men': 345,\n",
       " 'either': 346,\n",
       " 'line': 347,\n",
       " 'boring': 348,\n",
       " 'dead': 349,\n",
       " 'friends': 350,\n",
       " 'kids': 351,\n",
       " 'try': 352,\n",
       " 'production': 353,\n",
       " 'enjoy': 354,\n",
       " 'camera': 355,\n",
       " 'use': 356,\n",
       " 'wrong': 357,\n",
       " 'given': 358,\n",
       " 'low': 359,\n",
       " 'classic': 360,\n",
       " 'father': 361,\n",
       " 'need': 362,\n",
       " 'full': 363,\n",
       " 'stupid': 364,\n",
       " 'until': 365,\n",
       " 'next': 366,\n",
       " 'performances': 367,\n",
       " 'school': 368,\n",
       " 'hollywood': 369,\n",
       " 'rest': 370,\n",
       " 'truly': 371,\n",
       " 'awful': 372,\n",
       " 'video': 373,\n",
       " 'couple': 374,\n",
       " 'start': 375,\n",
       " 'sex': 376,\n",
       " 'recommend': 377,\n",
       " 'women': 378,\n",
       " 'let': 379,\n",
       " 'tell': 380,\n",
       " 'terrible': 381,\n",
       " 'remember': 382,\n",
       " 'mean': 383,\n",
       " 'came': 384,\n",
       " 'understand': 385,\n",
       " 'getting': 386,\n",
       " 'perhaps': 387,\n",
       " 'moments': 388,\n",
       " 'name': 389,\n",
       " 'keep': 390,\n",
       " 'face': 391,\n",
       " 'itself': 392,\n",
       " 'wonderful': 393,\n",
       " 'playing': 394,\n",
       " 'human': 395,\n",
       " 'style': 396,\n",
       " 'small': 397,\n",
       " 'episode': 398,\n",
       " 'perfect': 399,\n",
       " 'others': 400,\n",
       " 'person': 401,\n",
       " 'doing': 402,\n",
       " 'often': 403,\n",
       " 'early': 404,\n",
       " 'stars': 405,\n",
       " 'definitely': 406,\n",
       " 'written': 407,\n",
       " 'head': 408,\n",
       " 'lines': 409,\n",
       " 'dialogue': 410,\n",
       " 'gives': 411,\n",
       " 'piece': 412,\n",
       " \"couldn't\": 413,\n",
       " 'went': 414,\n",
       " 'finally': 415,\n",
       " 'mother': 416,\n",
       " 'case': 417,\n",
       " 'title': 418,\n",
       " 'absolutely': 419,\n",
       " 'live': 420,\n",
       " 'boy': 421,\n",
       " 'yes': 422,\n",
       " 'laugh': 423,\n",
       " 'certainly': 424,\n",
       " 'liked': 425,\n",
       " 'become': 426,\n",
       " 'entertaining': 427,\n",
       " 'worse': 428,\n",
       " 'oh': 429,\n",
       " 'sort': 430,\n",
       " 'loved': 431,\n",
       " 'lost': 432,\n",
       " 'hope': 433,\n",
       " 'called': 434,\n",
       " 'picture': 435,\n",
       " 'felt': 436,\n",
       " 'overall': 437,\n",
       " 'entire': 438,\n",
       " 'several': 439,\n",
       " 'mr': 440,\n",
       " 'based': 441,\n",
       " 'supposed': 442,\n",
       " 'cinema': 443,\n",
       " 'friend': 444,\n",
       " 'guys': 445,\n",
       " 'sound': 446,\n",
       " '5': 447,\n",
       " 'problem': 448,\n",
       " 'drama': 449,\n",
       " 'against': 450,\n",
       " 'waste': 451,\n",
       " 'white': 452,\n",
       " 'beginning': 453,\n",
       " '4': 454,\n",
       " 'fans': 455,\n",
       " 'totally': 456,\n",
       " 'dark': 457,\n",
       " 'care': 458,\n",
       " 'direction': 459,\n",
       " 'humor': 460,\n",
       " 'wanted': 461,\n",
       " \"she's\": 462,\n",
       " 'seemed': 463,\n",
       " 'under': 464,\n",
       " 'game': 465,\n",
       " 'children': 466,\n",
       " 'despite': 467,\n",
       " 'lives': 468,\n",
       " 'lead': 469,\n",
       " 'guess': 470,\n",
       " 'example': 471,\n",
       " 'already': 472,\n",
       " 'final': 473,\n",
       " 'throughout': 474,\n",
       " \"you'll\": 475,\n",
       " 'turn': 476,\n",
       " 'evil': 477,\n",
       " 'becomes': 478,\n",
       " 'unfortunately': 479,\n",
       " 'able': 480,\n",
       " 'quality': 481,\n",
       " \"i'd\": 482,\n",
       " 'days': 483,\n",
       " 'history': 484,\n",
       " 'fine': 485,\n",
       " 'side': 486,\n",
       " 'wants': 487,\n",
       " 'heart': 488,\n",
       " 'horrible': 489,\n",
       " 'writing': 490,\n",
       " 'amazing': 491,\n",
       " 'b': 492,\n",
       " 'flick': 493,\n",
       " 'killer': 494,\n",
       " 'run': 495,\n",
       " 'son': 496,\n",
       " '\\x96': 497,\n",
       " 'michael': 498,\n",
       " 'works': 499,\n",
       " 'close': 500,\n",
       " \"they're\": 501,\n",
       " 'act': 502,\n",
       " 'art': 503,\n",
       " 'matter': 504,\n",
       " 'kill': 505,\n",
       " 'etc': 506,\n",
       " 'tries': 507,\n",
       " \"won't\": 508,\n",
       " 'past': 509,\n",
       " 'town': 510,\n",
       " 'turns': 511,\n",
       " 'enjoyed': 512,\n",
       " 'brilliant': 513,\n",
       " 'gave': 514,\n",
       " 'behind': 515,\n",
       " 'parts': 516,\n",
       " 'stuff': 517,\n",
       " 'genre': 518,\n",
       " 'eyes': 519,\n",
       " 'car': 520,\n",
       " 'favorite': 521,\n",
       " 'directed': 522,\n",
       " 'late': 523,\n",
       " 'hand': 524,\n",
       " 'expect': 525,\n",
       " 'soon': 526,\n",
       " 'hour': 527,\n",
       " 'obviously': 528,\n",
       " 'themselves': 529,\n",
       " 'sometimes': 530,\n",
       " 'killed': 531,\n",
       " 'actress': 532,\n",
       " 'thinking': 533,\n",
       " 'child': 534,\n",
       " 'girls': 535,\n",
       " 'viewer': 536,\n",
       " 'starts': 537,\n",
       " 'city': 538,\n",
       " 'myself': 539,\n",
       " 'decent': 540,\n",
       " 'highly': 541,\n",
       " 'stop': 542,\n",
       " 'type': 543,\n",
       " 'self': 544,\n",
       " 'god': 545,\n",
       " 'says': 546,\n",
       " 'group': 547,\n",
       " 'anyway': 548,\n",
       " 'voice': 549,\n",
       " 'took': 550,\n",
       " 'known': 551,\n",
       " 'blood': 552,\n",
       " 'kid': 553,\n",
       " 'heard': 554,\n",
       " 'happens': 555,\n",
       " 'except': 556,\n",
       " 'fight': 557,\n",
       " 'feeling': 558,\n",
       " 'experience': 559,\n",
       " 'coming': 560,\n",
       " 'slow': 561,\n",
       " 'daughter': 562,\n",
       " 'writer': 563,\n",
       " 'stories': 564,\n",
       " 'moment': 565,\n",
       " 'leave': 566,\n",
       " 'told': 567,\n",
       " 'extremely': 568,\n",
       " 'score': 569,\n",
       " 'violence': 570,\n",
       " 'involved': 571,\n",
       " 'police': 572,\n",
       " 'strong': 573,\n",
       " 'chance': 574,\n",
       " 'lack': 575,\n",
       " 'cannot': 576,\n",
       " 'hit': 577,\n",
       " 'roles': 578,\n",
       " 'hilarious': 579,\n",
       " 's': 580,\n",
       " 'wonder': 581,\n",
       " 'happen': 582,\n",
       " 'particularly': 583,\n",
       " 'ok': 584,\n",
       " 'including': 585,\n",
       " 'living': 586,\n",
       " 'save': 587,\n",
       " 'looked': 588,\n",
       " \"wouldn't\": 589,\n",
       " 'crap': 590,\n",
       " 'simple': 591,\n",
       " 'please': 592,\n",
       " 'murder': 593,\n",
       " 'cool': 594,\n",
       " 'obvious': 595,\n",
       " 'happened': 596,\n",
       " 'complete': 597,\n",
       " 'cut': 598,\n",
       " 'serious': 599,\n",
       " 'age': 600,\n",
       " 'gore': 601,\n",
       " 'attempt': 602,\n",
       " 'hell': 603,\n",
       " 'ago': 604,\n",
       " 'song': 605,\n",
       " 'shown': 606,\n",
       " 'taken': 607,\n",
       " 'english': 608,\n",
       " 'james': 609,\n",
       " 'robert': 610,\n",
       " 'david': 611,\n",
       " 'seriously': 612,\n",
       " 'released': 613,\n",
       " 'reality': 614,\n",
       " 'opening': 615,\n",
       " 'interest': 616,\n",
       " 'jokes': 617,\n",
       " 'across': 618,\n",
       " 'none': 619,\n",
       " 'hero': 620,\n",
       " 'possible': 621,\n",
       " 'today': 622,\n",
       " 'exactly': 623,\n",
       " 'alone': 624,\n",
       " 'sad': 625,\n",
       " 'brother': 626,\n",
       " 'number': 627,\n",
       " 'saying': 628,\n",
       " 'career': 629,\n",
       " \"film's\": 630,\n",
       " 'usually': 631,\n",
       " 'hours': 632,\n",
       " 'cinematography': 633,\n",
       " 'talent': 634,\n",
       " 'view': 635,\n",
       " 'annoying': 636,\n",
       " 'running': 637,\n",
       " 'yourself': 638,\n",
       " 'relationship': 639,\n",
       " 'documentary': 640,\n",
       " 'wish': 641,\n",
       " 'huge': 642,\n",
       " 'order': 643,\n",
       " 'whose': 644,\n",
       " 'shots': 645,\n",
       " 'ridiculous': 646,\n",
       " 'taking': 647,\n",
       " 'important': 648,\n",
       " 'light': 649,\n",
       " 'body': 650,\n",
       " 'middle': 651,\n",
       " 'level': 652,\n",
       " 'ends': 653,\n",
       " 'started': 654,\n",
       " 'call': 655,\n",
       " 'female': 656,\n",
       " \"i'll\": 657,\n",
       " 'husband': 658,\n",
       " 'four': 659,\n",
       " 'power': 660,\n",
       " 'word': 661,\n",
       " 'turned': 662,\n",
       " 'major': 663,\n",
       " 'opinion': 664,\n",
       " 'change': 665,\n",
       " 'mostly': 666,\n",
       " 'usual': 667,\n",
       " 'silly': 668,\n",
       " 'scary': 669,\n",
       " 'rating': 670,\n",
       " 'beyond': 671,\n",
       " 'somewhat': 672,\n",
       " 'happy': 673,\n",
       " 'ones': 674,\n",
       " 'words': 675,\n",
       " 'room': 676,\n",
       " 'knows': 677,\n",
       " 'knew': 678,\n",
       " 'country': 679,\n",
       " 'disappointed': 680,\n",
       " 'talking': 681,\n",
       " 'novel': 682,\n",
       " 'apparently': 683,\n",
       " 'non': 684,\n",
       " 'strange': 685,\n",
       " 'upon': 686,\n",
       " 'attention': 687,\n",
       " 'finds': 688,\n",
       " 'basically': 689,\n",
       " 'single': 690,\n",
       " 'cheap': 691,\n",
       " 'modern': 692,\n",
       " 'due': 693,\n",
       " 'jack': 694,\n",
       " 'musical': 695,\n",
       " 'television': 696,\n",
       " 'problems': 697,\n",
       " 'miss': 698,\n",
       " 'episodes': 699,\n",
       " 'clearly': 700,\n",
       " 'local': 701,\n",
       " '7': 702,\n",
       " 'british': 703,\n",
       " 'thriller': 704,\n",
       " 'talk': 705,\n",
       " 'events': 706,\n",
       " 'five': 707,\n",
       " 'sequence': 708,\n",
       " \"aren't\": 709,\n",
       " 'class': 710,\n",
       " 'french': 711,\n",
       " 'moving': 712,\n",
       " 'ten': 713,\n",
       " 'fast': 714,\n",
       " 'review': 715,\n",
       " 'earth': 716,\n",
       " 'tells': 717,\n",
       " 'predictable': 718,\n",
       " 'songs': 719,\n",
       " 'team': 720,\n",
       " 'comic': 721,\n",
       " 'straight': 722,\n",
       " 'whether': 723,\n",
       " '8': 724,\n",
       " 'die': 725,\n",
       " 'add': 726,\n",
       " 'dialog': 727,\n",
       " 'entertainment': 728,\n",
       " 'above': 729,\n",
       " 'sets': 730,\n",
       " 'future': 731,\n",
       " 'enjoyable': 732,\n",
       " 'appears': 733,\n",
       " 'near': 734,\n",
       " 'space': 735,\n",
       " 'easily': 736,\n",
       " 'hate': 737,\n",
       " 'soundtrack': 738,\n",
       " 'bring': 739,\n",
       " 'giving': 740,\n",
       " 'lots': 741,\n",
       " 'similar': 742,\n",
       " 'romantic': 743,\n",
       " 'george': 744,\n",
       " 'supporting': 745,\n",
       " 'release': 746,\n",
       " 'mention': 747,\n",
       " 'filmed': 748,\n",
       " 'within': 749,\n",
       " 'message': 750,\n",
       " 'sequel': 751,\n",
       " 'clear': 752,\n",
       " 'falls': 753,\n",
       " 'needs': 754,\n",
       " \"haven't\": 755,\n",
       " 'dull': 756,\n",
       " 'suspense': 757,\n",
       " 'eye': 758,\n",
       " 'bunch': 759,\n",
       " 'surprised': 760,\n",
       " 'showing': 761,\n",
       " 'sorry': 762,\n",
       " 'tried': 763,\n",
       " 'certain': 764,\n",
       " 'easy': 765,\n",
       " 'working': 766,\n",
       " 'ways': 767,\n",
       " 'theme': 768,\n",
       " 'theater': 769,\n",
       " 'named': 770,\n",
       " 'among': 771,\n",
       " \"what's\": 772,\n",
       " 'storyline': 773,\n",
       " 'monster': 774,\n",
       " 'king': 775,\n",
       " 'stay': 776,\n",
       " 'effort': 777,\n",
       " 'stand': 778,\n",
       " 'fall': 779,\n",
       " 'minute': 780,\n",
       " 'gone': 781,\n",
       " 'rock': 782,\n",
       " 'using': 783,\n",
       " '9': 784,\n",
       " 'feature': 785,\n",
       " 'comments': 786,\n",
       " 'buy': 787,\n",
       " \"'\": 788,\n",
       " 'typical': 789,\n",
       " 't': 790,\n",
       " 'sister': 791,\n",
       " 'editing': 792,\n",
       " 'tale': 793,\n",
       " 'avoid': 794,\n",
       " 'deal': 795,\n",
       " 'mystery': 796,\n",
       " 'dr': 797,\n",
       " 'doubt': 798,\n",
       " 'fantastic': 799,\n",
       " 'kept': 800,\n",
       " 'nearly': 801,\n",
       " 'subject': 802,\n",
       " 'okay': 803,\n",
       " 'feels': 804,\n",
       " 'viewing': 805,\n",
       " 'elements': 806,\n",
       " 'oscar': 807,\n",
       " 'check': 808,\n",
       " 'points': 809,\n",
       " 'realistic': 810,\n",
       " 'greatest': 811,\n",
       " 'means': 812,\n",
       " 'herself': 813,\n",
       " 'parents': 814,\n",
       " 'famous': 815,\n",
       " 'imagine': 816,\n",
       " 'rent': 817,\n",
       " 'viewers': 818,\n",
       " 'crime': 819,\n",
       " 'richard': 820,\n",
       " 'form': 821,\n",
       " 'peter': 822,\n",
       " 'actual': 823,\n",
       " 'lady': 824,\n",
       " 'general': 825,\n",
       " 'dog': 826,\n",
       " 'follow': 827,\n",
       " 'believable': 828,\n",
       " 'period': 829,\n",
       " 'red': 830,\n",
       " 'brought': 831,\n",
       " 'move': 832,\n",
       " 'material': 833,\n",
       " 'forget': 834,\n",
       " 'somehow': 835,\n",
       " 'begins': 836,\n",
       " 're': 837,\n",
       " 'reviews': 838,\n",
       " 'animation': 839,\n",
       " 'paul': 840,\n",
       " \"you've\": 841,\n",
       " 'leads': 842,\n",
       " 'weak': 843,\n",
       " 'figure': 844,\n",
       " 'surprise': 845,\n",
       " 'sit': 846,\n",
       " 'hear': 847,\n",
       " 'average': 848,\n",
       " 'open': 849,\n",
       " 'sequences': 850,\n",
       " 'killing': 851,\n",
       " 'atmosphere': 852,\n",
       " 'eventually': 853,\n",
       " 'tom': 854,\n",
       " 'learn': 855,\n",
       " 'premise': 856,\n",
       " '20': 857,\n",
       " 'wait': 858,\n",
       " 'sci': 859,\n",
       " 'deep': 860,\n",
       " 'fi': 861,\n",
       " 'expected': 862,\n",
       " 'whatever': 863,\n",
       " 'indeed': 864,\n",
       " 'particular': 865,\n",
       " 'note': 866,\n",
       " 'poorly': 867,\n",
       " 'lame': 868,\n",
       " 'dance': 869,\n",
       " 'imdb': 870,\n",
       " 'situation': 871,\n",
       " 'shame': 872,\n",
       " 'third': 873,\n",
       " 'york': 874,\n",
       " 'box': 875,\n",
       " 'truth': 876,\n",
       " 'decided': 877,\n",
       " 'free': 878,\n",
       " 'hot': 879,\n",
       " \"who's\": 880,\n",
       " 'difficult': 881,\n",
       " 'needed': 882,\n",
       " 'season': 883,\n",
       " 'acted': 884,\n",
       " 'leaves': 885,\n",
       " 'unless': 886,\n",
       " 'emotional': 887,\n",
       " 'possibly': 888,\n",
       " 'romance': 889,\n",
       " 'sexual': 890,\n",
       " 'gay': 891,\n",
       " 'boys': 892,\n",
       " 'footage': 893,\n",
       " 'write': 894,\n",
       " 'western': 895,\n",
       " 'forced': 896,\n",
       " 'credits': 897,\n",
       " 'memorable': 898,\n",
       " 'doctor': 899,\n",
       " 'became': 900,\n",
       " 'reading': 901,\n",
       " 'otherwise': 902,\n",
       " 'begin': 903,\n",
       " 'air': 904,\n",
       " 'crew': 905,\n",
       " 'de': 906,\n",
       " 'question': 907,\n",
       " 'meet': 908,\n",
       " 'society': 909,\n",
       " 'male': 910,\n",
       " 'meets': 911,\n",
       " \"let's\": 912,\n",
       " 'plus': 913,\n",
       " 'cheesy': 914,\n",
       " 'hands': 915,\n",
       " 'superb': 916,\n",
       " 'screenplay': 917,\n",
       " 'beauty': 918,\n",
       " 'interested': 919,\n",
       " 'street': 920,\n",
       " 'features': 921,\n",
       " 'perfectly': 922,\n",
       " 'masterpiece': 923,\n",
       " 'whom': 924,\n",
       " 'laughs': 925,\n",
       " 'stage': 926,\n",
       " 'nature': 927,\n",
       " 'effect': 928,\n",
       " 'comment': 929,\n",
       " 'forward': 930,\n",
       " 'nor': 931,\n",
       " 'badly': 932,\n",
       " 'sounds': 933,\n",
       " 'previous': 934,\n",
       " 'e': 935,\n",
       " 'japanese': 936,\n",
       " 'weird': 937,\n",
       " 'island': 938,\n",
       " 'inside': 939,\n",
       " 'personal': 940,\n",
       " 'quickly': 941,\n",
       " 'total': 942,\n",
       " 'keeps': 943,\n",
       " 'towards': 944,\n",
       " 'result': 945,\n",
       " 'america': 946,\n",
       " 'battle': 947,\n",
       " 'crazy': 948,\n",
       " 'worked': 949,\n",
       " 'setting': 950,\n",
       " 'incredibly': 951,\n",
       " 'earlier': 952,\n",
       " 'background': 953,\n",
       " 'mess': 954,\n",
       " 'cop': 955,\n",
       " 'writers': 956,\n",
       " 'fire': 957,\n",
       " 'copy': 958,\n",
       " 'unique': 959,\n",
       " 'dumb': 960,\n",
       " 'realize': 961,\n",
       " 'powerful': 962,\n",
       " 'mark': 963,\n",
       " 'lee': 964,\n",
       " 'business': 965,\n",
       " 'rate': 966,\n",
       " 'dramatic': 967,\n",
       " 'older': 968,\n",
       " 'pay': 969,\n",
       " 'following': 970,\n",
       " 'directors': 971,\n",
       " 'girlfriend': 972,\n",
       " 'joke': 973,\n",
       " 'plenty': 974,\n",
       " 'directing': 975,\n",
       " 'various': 976,\n",
       " 'creepy': 977,\n",
       " 'baby': 978,\n",
       " 'development': 979,\n",
       " 'appear': 980,\n",
       " 'brings': 981,\n",
       " 'front': 982,\n",
       " 'ask': 983,\n",
       " 'dream': 984,\n",
       " 'water': 985,\n",
       " 'admit': 986,\n",
       " 'bill': 987,\n",
       " 'rich': 988,\n",
       " 'apart': 989,\n",
       " 'joe': 990,\n",
       " 'political': 991,\n",
       " 'fairly': 992,\n",
       " 'reasons': 993,\n",
       " 'leading': 994,\n",
       " 'portrayed': 995,\n",
       " 'spent': 996,\n",
       " 'telling': 997,\n",
       " 'cover': 998,\n",
       " 'outside': 999,\n",
       " 'wasted': 1000,\n",
       " ...}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.word_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use the tokenizer to convert all texts in the training-set to lists of these tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_tokens = tokenizer.texts_to_sequences(x_train_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, here is a text from the training-set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A simple comment...<br /><br />What can I say... this is a wonderful film that I can watch over and over. It is definitely one of the top ten comedies made. With a great cast, Jack Lemmon and Walter Matthau wording a perfect script by Neil Simon, based on his play.<br /><br />It is real to life situation done perfectly. If you have digital cable, one gets the menu on bottom of screen to give what is on. It usually gives this film ***% stars but in reality it deserves **** stars. If you really watch this film, one can tell that it will be as funny and fresh a hundred years from now.'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_text[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This text corresponds to the following list of tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   3,  591,  929,    7,    7,   48,   67,   10,  131,   11,    6,\n",
       "          3,  393,   19,   12,   10,   67,  103,  121,    2,  121,    9,\n",
       "          6,  406,   27,    4,    1,  342,  713, 1317,   90,   16,    3,\n",
       "         78,  174,  694, 4910,    2, 2556, 3599,    3,  399,  227,   31,\n",
       "       4033, 2628,  441,   20,   24,  288,    7,    7,    9,    6,  144,\n",
       "          5,  114,  871,  221,  922,   43,   22,   25, 3639, 1897,   27,\n",
       "        217,    1, 9206,   20, 1306,    4,  258,    5,  197,   48,    6,\n",
       "         20,    9,  631,  411,   11,   19,  405,   18,    8,  614,    9,\n",
       "       1003,  405,   43,   22,   62,  103,   11,   19,   27,   67,  380,\n",
       "         12,    9,   80,   26,   14,  152,    2, 1451,    3, 2997,  153,\n",
       "         36,  146])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(x_train_tokens[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to convert the texts in the test-set to tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test_tokens = tokenizer.texts_to_sequences(x_test_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Padding and Truncating Data\n",
    "\n",
    "The Recurrent Neural Network can take sequences of arbitrary length as input, but in order to use a whole batch of data, the sequences need to have the same length. There are two ways of achieving this: (A) Either we ensure that all sequences in the entire data-set have the same length, or (B) we write a custom data-generator that ensures the sequences have the same length within each batch.\n",
    "\n",
    "Solution (A) is simpler but if we use the length of the longest sequence in the data-set, then we are wasting a lot of memory. This is particularly important for larger data-sets.\n",
    "\n",
    "So in order to make a compromise, we will use a sequence-length that covers most sequences in the data-set, and we will then truncate longer sequences and pad shorter sequences.\n",
    "\n",
    "First we count the number of tokens in all the sequences in the data-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_tokens = [len(tokens) for tokens in x_train_tokens + x_test_tokens]\n",
    "num_tokens = np.array(num_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The average number of tokens in a sequence is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "221.27716"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(num_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The maximum number of tokens in a sequence is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2209"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.max(num_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The max number of tokens we will allow is set to the average plus 2 standard deviations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "544"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_tokens = np.mean(num_tokens) + 2 * np.std(num_tokens)\n",
    "max_tokens = int(max_tokens)\n",
    "max_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This covers about 95% of the data-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.94534"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(num_tokens < max_tokens) / len(num_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When padding or truncating the sequences that have a different length, we need to determine if we want to do this padding or truncating 'pre' or 'post'. If a sequence is truncated, it means that a part of the sequence is simply thrown away. If a sequence is padded, it means that zeros are added to the sequence.\n",
    "\n",
    "So the choice of 'pre' or 'post' can be important because it determines whether we throw away the first or last part of a sequence when truncating, and it determines whether we add zeros to the beginning or end of the sequence when padding. This may confuse the Recurrent Neural Network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "pad = 'pre'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_pad = pad_sequences(x_train_tokens, maxlen=max_tokens,\n",
    "                            padding=pad, truncating=pad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test_pad = pad_sequences(x_test_tokens, maxlen=max_tokens,\n",
    "                           padding=pad, truncating=pad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now transformed the training-set into one big matrix of integers (tokens) with this shape:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 544)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_pad.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The matrix for the test-set has the same shape:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 544)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test_pad.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, we had the following sequence of tokens above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   3,  591,  929,    7,    7,   48,   67,   10,  131,   11,    6,\n",
       "          3,  393,   19,   12,   10,   67,  103,  121,    2,  121,    9,\n",
       "          6,  406,   27,    4,    1,  342,  713, 1317,   90,   16,    3,\n",
       "         78,  174,  694, 4910,    2, 2556, 3599,    3,  399,  227,   31,\n",
       "       4033, 2628,  441,   20,   24,  288,    7,    7,    9,    6,  144,\n",
       "          5,  114,  871,  221,  922,   43,   22,   25, 3639, 1897,   27,\n",
       "        217,    1, 9206,   20, 1306,    4,  258,    5,  197,   48,    6,\n",
       "         20,    9,  631,  411,   11,   19,  405,   18,    8,  614,    9,\n",
       "       1003,  405,   43,   22,   62,  103,   11,   19,   27,   67,  380,\n",
       "         12,    9,   80,   26,   14,  152,    2, 1451,    3, 2997,  153,\n",
       "         36,  146])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(x_train_tokens[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This has simply been padded to create the following sequence. Note that when this is input to the Recurrent Neural Network, then it first inputs a lot of zeros. If we had padded 'post' then it would input the integer-tokens first and then a lot of zeros. This may confuse the Recurrent Neural Network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    3,  591,  929,    7,    7,   48,   67,   10,\n",
       "        131,   11,    6,    3,  393,   19,   12,   10,   67,  103,  121,\n",
       "          2,  121,    9,    6,  406,   27,    4,    1,  342,  713, 1317,\n",
       "         90,   16,    3,   78,  174,  694, 4910,    2, 2556, 3599,    3,\n",
       "        399,  227,   31, 4033, 2628,  441,   20,   24,  288,    7,    7,\n",
       "          9,    6,  144,    5,  114,  871,  221,  922,   43,   22,   25,\n",
       "       3639, 1897,   27,  217,    1, 9206,   20, 1306,    4,  258,    5,\n",
       "        197,   48,    6,   20,    9,  631,  411,   11,   19,  405,   18,\n",
       "          8,  614,    9, 1003,  405,   43,   22,   62,  103,   11,   19,\n",
       "         27,   67,  380,   12,    9,   80,   26,   14,  152,    2, 1451,\n",
       "          3, 2997,  153,   36,  146], dtype=int32)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_pad[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenizer Inverse Map\n",
    "\n",
    "For some strange reason, the Keras implementation of a tokenizer does not seem to have the inverse mapping from integer-tokens back to words, which is needed to reconstruct text-strings from lists of tokens. So we make that mapping here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = tokenizer.word_index\n",
    "inverse_map = dict(zip(idx.values(), idx.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Helper-function for converting a list of tokens back to a string of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokens_to_string(tokens):\n",
    "    # Map from tokens back to words.\n",
    "    words = [inverse_map[token] for token in tokens if token != 0]\n",
    "    \n",
    "    # Concatenate all words.\n",
    "    text = \" \".join(words)\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, this is the original text from the data-set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A simple comment...<br /><br />What can I say... this is a wonderful film that I can watch over and over. It is definitely one of the top ten comedies made. With a great cast, Jack Lemmon and Walter Matthau wording a perfect script by Neil Simon, based on his play.<br /><br />It is real to life situation done perfectly. If you have digital cable, one gets the menu on bottom of screen to give what is on. It usually gives this film ***% stars but in reality it deserves **** stars. If you really watch this film, one can tell that it will be as funny and fresh a hundred years from now.'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_text[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can recreate this text except for punctuation and other symbols, by converting the list of tokens back to words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a simple comment br br what can i say this is a wonderful film that i can watch over and over it is definitely one of the top ten comedies made with a great cast jack lemmon and walter matthau a perfect script by neil simon based on his play br br it is real to life situation done perfectly if you have digital cable one gets the menu on bottom of screen to give what is on it usually gives this film stars but in reality it deserves stars if you really watch this film one can tell that it will be as funny and fresh a hundred years from now'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_to_string(x_train_tokens[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the Recurrent Neural Network\n",
    "\n",
    "We are now ready to create the Recurrent Neural Network (RNN). We will use the Keras API for this because of its simplicity. See Tutorial #03-C for a tutorial on Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first layer in the RNN is a so-called Embedding-layer which converts each integer-token into a vector of values. This is necessary because the integer-tokens may take on values between 0 and 10000 for a vocabulary of 10000 words. The RNN cannot work on values in such a wide range. The embedding-layer is trained as a part of the RNN and will learn to map words with similar semantic meanings to similar embedding-vectors, as will be shown further below.\n",
    "\n",
    "First we define the size of the embedding-vector for each integer-token. In this case we have set it to 8, so that each integer-token will be converted to a vector of length 8. The values of the embedding-vector will generally fall roughly between -1.0 and 1.0, although they may exceed these values somewhat.\n",
    "\n",
    "The size of the embedding-vector is typically selected between 100-300, but it seems to work reasonably well with small values for Sentiment Analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_size = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The embedding-layer also needs to know the number of words in the vocabulary (`num_words`) and the length of the padded token-sequences (`max_tokens`). We also give this layer a name because we need to retrieve its weights further below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Embedding(input_dim=num_words,\n",
    "                    output_dim=embedding_size,\n",
    "                    input_length=max_tokens,\n",
    "                    name='layer_embedding'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now add the first Gated Recurrent Unit (GRU) to the network. This will have 16 outputs. Because we will add a second GRU after this one, we need to return sequences of data because the next GRU expects sequences as its input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/magnus/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/backend.py:1456: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n"
     ]
    }
   ],
   "source": [
    "model.add(GRU(units=16, return_sequences=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This adds the second GRU with 8 output units. This will be followed by another GRU so it must also return sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(GRU(units=8, return_sequences=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This adds the third and final GRU with 4 output units. This will be followed by a dense-layer, so it should only give the final output of the GRU and not a whole sequence of outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(GRU(units=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add a fully-connected / dense layer which computes a value between 0.0 and 1.0 that will be used as the classification output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the Adam optimizer with the given learning-rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = Adam(lr=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compile the Keras model so it is ready for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/magnus/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/backend.py:1557: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer=optimizer,\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "layer_embedding (Embedding)  (None, 544, 8)            80000     \n",
      "_________________________________________________________________\n",
      "gru_1 (GRU)                  (None, None, 16)          1200      \n",
      "_________________________________________________________________\n",
      "gru_2 (GRU)                  (None, None, 8)           600       \n",
      "_________________________________________________________________\n",
      "gru_3 (GRU)                  (None, 4)                 156       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 5         \n",
      "=================================================================\n",
      "Total params: 81,961\n",
      "Trainable params: 81,961\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Recurrent Neural Network\n",
    "\n",
    "We can now train the model. Note that we are using the data-set with the padded sequences. We use 5% of the training-set as a small validation-set, so we have a rough idea whether the model is generalizing well or if it is perhaps over-fitting to the training-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 23750 samples, validate on 1250 samples\n",
      "Epoch 1/3\n",
      "23750/23750 [==============================]23750/23750 [==============================] - 464s 20ms/step - loss: 0.6517 - acc: 0.6002 - val_loss: 0.6218 - val_acc: 0.6752\n",
      "\n",
      "Epoch 2/3\n",
      "23750/23750 [==============================]23750/23750 [==============================] - 447s 19ms/step - loss: 0.4292 - acc: 0.8102 - val_loss: 0.6701 - val_acc: 0.6512\n",
      "\n",
      "Epoch 3/3\n",
      "23750/23750 [==============================]23750/23750 [==============================] - 445s 19ms/step - loss: 0.3092 - acc: 0.8765 - val_loss: 0.3182 - val_acc: 0.8752\n",
      "\n",
      "CPU times: user 35min 19s, sys: 2min 41s, total: 38min\n",
      "Wall time: 22min 37s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras._impl.keras.callbacks.History at 0x7ff79f0d6cf8>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "model.fit(x_train_pad, y_train,\n",
    "          validation_split=0.05, epochs=3, batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance on Test-Set\n",
    "\n",
    "Now that the model has been trained we can calculate its classification accuracy on the test-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================]25000/25000 [==============================] - 175s 7ms/step\n",
      "\n",
      "CPU times: user 2min 59s, sys: 340 ms, total: 2min 59s\n",
      "Wall time: 2min 55s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "result = model.evaluate(x_test_pad, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 86.71%\n"
     ]
    }
   ],
   "source": [
    "print(\"Accuracy: {0:.2%}\".format(result[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example of Mis-Classified Text\n",
    "\n",
    "In order to show an example of mis-classified text, we first calculate the predicted sentiment for the first 1000 texts in the test-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.01 s, sys: 0 ns, total: 7.01 s\n",
      "Wall time: 6.88 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "y_pred = model.predict(x=x_test_pad[0:1000])\n",
    "y_pred = y_pred.T[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These predicted numbers fall between 0.0 and 1.0. We use a cutoff / threshold and say that all values above 0.5 are taken to be 1.0 and all values below 0.5 are taken to be 0.0. This gives us a predicted \"class\" of either 0.0 or 1.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_pred = np.array([1.0 if p>0.5 else 0.0 for p in y_pred])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The true \"class\" for the first 1000 texts in the test-set are needed for comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_true = np.array(y_test[0:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then get indices for all the texts that were incorrectly classified by comparing all the \"classes\" of these two arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "incorrect = np.where(cls_pred != cls_true)\n",
    "incorrect = incorrect[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of the 1000 texts used, how many were mis-classified?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "121"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(incorrect)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us look at the first mis-classified text. We will use its index several times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx = incorrect[0]\n",
    "idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mis-classified text is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I would like to start by saying I can only hope that the makers of this movie and it\\'s sister film The Intruder (directed by the great unheralded stylist auteur that is Jopi Burnama) know in their hearts just how much pleasure they have brought to me and my friends in the sleepy north eastern town of Jarrow.<br /><br />From the opening pre credit sequence which manages to drag ever so slightly despite containing a man crashing through a window on a motorbike, the pitiless destruction of a silence lab, the introduction of one of the most simultaneously annoying and anaemic bad guys in movie history and costume design that Jean Paul Gautier would find ott and garish. Make no mistake; this is a truly unique experience. Early highlight - an explosion (get used to it, plenty more where that came from!) followed by a close up of our chubby heroine and the most hilarious line reading of the word \"dad\" in living memory. And then... the theme song...<br /><br />Yeah, this deserves its own paragraph. Sung by AJ, written by people who really should wish to remain anonymous, it makes the songs written for the Rocky films sound like Schubert. This is crap 80\\'s hero motivation narcissism at an all time high, with choice lyrics such as \"its only me and you, its come down to the wire\" and much talk of having to \"cross the line\" (it\\'ll make sense in time - our hero cares little for the boundaries of bona fida police work) abounding. Not to mention the Indonesian Supremes cooing the film\\'s title seductively. At this point anyone wishing to switch off officially has no pulse.<br /><br />Our hero is Semitic cop Peter Goldson (essayed brilliantly by Intruder star Peter O\\'Brien), the \"stabilizer\" of the title. The man\\'s bull in a china shop approach to crime fighting and particularly his less than inconspicuous undercover work truly leaves much to be desired, but he is without question an entertaining guide through the mean streets of downtown Jakarta, with local sleaze ball connection Captain Johnny in tow, as well as Peter\\'s own waste of space partner in fashion crime Sylvia Nash, who does little. So many highlights, so little time - the \"slide please\" arrogance of Peter\\'s not all too convincingly argued case against chief baddie Greg Rainmaker (Intruder fans will know hirsute slimy bastard Craig Gavin as the monstrous John White - helluva name eh? No! Oh well...), the x marks the spot location map stupidity, our hero taking horrible advantage of heroine Tina Probost during a moment of weakness on her behalf, the latter turning up at a sting operation dressed like a member of a particularly flamboyant dancing troop. And believe me that barely covers it.<br /><br />There wasn\\'t even time to go into the plot revolving around the hunt for a drug detection system and a kidnapped professor with an alarming but commendable amount of national pride. Or our hero turning up at a funeral dressed as if an extra on Boogie Nights. Or the absolutely hysterical craic between Captain Johnny and Goldson - two guys have never made more heavy weather of buddy buddy shtick than these clowns. The trowel was possibly too subtle me thinks.<br /><br />Ah it tails off people, and you never thought scenes of wanton destruction and general mayhem could be so unbelievably boring, but the character interaction is stupendous, the dialogue truly priceless and the incompetence on show somehow endearing. Oh and the shoes people - watch out for the shoes!'"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = x_test_text[idx]\n",
    "text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the predicted and true classes for the text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08332923"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cls_true[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New Data\n",
    "\n",
    "Let us try and classify new texts that we make up. Some of these are obvious, while others use negation and sarcasm to try and confuse the model into mis-classifying the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1 = \"This movie is fantastic! I really like it because it is so good!\"\n",
    "text2 = \"Good movie!\"\n",
    "text3 = \"Maybe I like this movie.\"\n",
    "text4 = \"Meh ...\"\n",
    "text5 = \"If I were a drunk teenager then this movie might be good.\"\n",
    "text6 = \"Bad movie!\"\n",
    "text7 = \"Not a good movie!\"\n",
    "text8 = \"This movie really sucks! Can I get my money back please?\"\n",
    "texts = [text1, text2, text3, text4, text5, text6, text7, text8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first convert these texts to arrays of integer-tokens because that is needed by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = tokenizer.texts_to_sequences(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To input texts with different lengths into the model, we also need to pad and truncate them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8, 544)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_pad = pad_sequences(tokens, maxlen=max_tokens,\n",
    "                           padding=pad, truncating=pad)\n",
    "tokens_pad.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use the trained model to predict the sentiment for these texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.868934  ],\n",
       "       [0.72526425],\n",
       "       [0.33099633],\n",
       "       [0.49190348],\n",
       "       [0.3054021 ],\n",
       "       [0.14959489],\n",
       "       [0.5235635 ],\n",
       "       [0.21565402]], dtype=float32)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(tokens_pad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A value close to 0.0 means a negative sentiment and a value close to 1.0 means a positive sentiment. These numbers will vary every time you train the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "\n",
    "The model cannot work on integer-tokens directly, because they are integer values that may range between 0 and the number of words in our vocabulary, e.g. 10000. So we need to convert the integer-tokens into vectors of values that are roughly between -1.0 and 1.0 which can be used as input to a neural network.\n",
    "\n",
    "This mapping from integer-tokens to real-valued vectors is also called an \"embedding\". It is essentially just a matrix where each row contains the vector-mapping of a single token. This means we can quickly lookup the mapping of each integer-token by simply using the token as an index into the matrix. The embeddings are learned along with the rest of the model during training.\n",
    "\n",
    "Ideally the embedding would learn a mapping where words that are similar in meaning also have similar embedding-values. Let us investigate if that has happened here.\n",
    "\n",
    "First we need to get the embedding-layer from the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_embedding = model.get_layer('layer_embedding')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then get the weights used for the mapping done by the embedding-layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights_embedding = layer_embedding.get_weights()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the weights are actually just a matrix with the number of words in the vocabulary times the vector length for each embedding. That's because it is basically just a lookup-matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 8)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_embedding.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us get the integer-token for the word 'good', which is just an index into the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "49"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_good = tokenizer.word_index['good']\n",
    "token_good"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us also get the integer-token for the word 'great'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "78"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_great = tokenizer.word_index['great']\n",
    "token_great"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These integertokens may be far apart and will depend on the frequency of those words in the data-set.\n",
    "\n",
    "Now let us compare the vector-embeddings for the words 'good' and 'great'. Several of these values are similar, although some values are quite different. Note that these values will change every time you train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.86528164, 0.6867993 , 0.4362397 , 0.66128314, 0.11546915,\n",
       "       0.94507647, 0.32628497, 0.535881  ], dtype=float32)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_embedding[token_good]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.0691622 ,  1.124244  , -0.04477464, -0.05861434,  0.16965319,\n",
       "        1.2626944 ,  0.76136374, -0.00998422], dtype=float32)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_embedding[token_great]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can compare the embeddings for the words 'bad' and 'horrible'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_bad = tokenizer.word_index['bad']\n",
    "token_horrible = tokenizer.word_index['horrible']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.31903917,  0.53934103,  1.3727672 ,  1.4083829 ,  0.8475107 ,\n",
       "       -0.22946651,  0.0251075 ,  0.77032244], dtype=float32)"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_embedding[token_bad]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.47915924,  0.12226178,  0.90192014,  0.742338  ,  0.58730644,\n",
       "        0.32736972, -0.17633988,  1.3744307 ], dtype=float32)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_embedding[token_horrible]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sorted Words\n",
    "\n",
    "We can also sort all the words in the vocabulary according to their \"similarity\" in the embedding-space. We want to see if words that have similar embedding-vectors also have similar meanings.\n",
    "\n",
    "Similarity of embedding-vectors can be measured by different metrics, e.g. Euclidean distance or cosine distance.\n",
    "\n",
    "We have a helper-function for calculating these distances and printing the words in sorted order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_sorted_words(word, metric='cosine'):\n",
    "    \"\"\"\n",
    "    Print the words in the vocabulary sorted according to their\n",
    "    embedding-distance to the given word.\n",
    "    Different metrics can be used, e.g. 'cosine' or 'euclidean'.\n",
    "    \"\"\"\n",
    "\n",
    "    # Get the token (i.e. integer ID) for the given word.\n",
    "    token = tokenizer.word_index[word]\n",
    "\n",
    "    # Get the embedding for the given word. Note that the\n",
    "    # embedding-weight-matrix is indexed by the word-tokens\n",
    "    # which are integer IDs.\n",
    "    embedding = weights_embedding[token]\n",
    "\n",
    "    # Calculate the distance between the embeddings for\n",
    "    # this word and all other words in the vocabulary.\n",
    "    distances = cdist(weights_embedding, [embedding],\n",
    "                      metric=metric).T[0]\n",
    "    \n",
    "    # Get an index sorted according to the embedding-distances.\n",
    "    # These are the tokens (integer IDs) for words in the vocabulary.\n",
    "    sorted_index = np.argsort(distances)\n",
    "    \n",
    "    # Sort the embedding-distances.\n",
    "    sorted_distances = distances[sorted_index]\n",
    "    \n",
    "    # Sort all the words in the vocabulary according to their\n",
    "    # embedding-distance. This is a bit excessive because we\n",
    "    # will only print the top and bottom words.\n",
    "    sorted_words = [inverse_map[token] for token in sorted_index\n",
    "                    if token != 0]\n",
    "\n",
    "    # Helper-function for printing words and embedding-distances.\n",
    "    def _print_words(words, distances):\n",
    "        for word, distance in zip(words, distances):\n",
    "            print(\"{0:.3f} - {1}\".format(distance, word))\n",
    "\n",
    "    # Number of words to print from the top and bottom of the list.\n",
    "    k = 10\n",
    "\n",
    "    print(\"Distance from '{0}':\".format(word))\n",
    "\n",
    "    # Print the words with smallest embedding-distance.\n",
    "    _print_words(sorted_words[0:k], sorted_distances[0:k])\n",
    "\n",
    "    print(\"...\")\n",
    "\n",
    "    # Print the words with highest embedding-distance.\n",
    "    _print_words(sorted_words[-k:], sorted_distances[-k:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then print the words that are near and far from the word 'great' in terms of their vector-embeddings. Note that these may change each time you train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distance from 'great':\n",
      "0.000 - great\n",
      "0.016 - touching\n",
      "0.017 - arguments\n",
      "0.025 - nevertheless\n",
      "0.031 - elmer\n",
      "0.032 - 8\n",
      "0.036 - ritter\n",
      "0.037 - juliet\n",
      "0.041 - randy\n",
      "0.045 - afterward\n",
      "...\n",
      "1.057 - rubbish\n",
      "1.060 - dull\n",
      "1.064 - disappointing\n",
      "1.069 - unlikeable\n",
      "1.078 - uninspired\n",
      "1.083 - lacks\n",
      "1.188 - worst\n",
      "1.225 - waste\n",
      "1.247 - awful\n",
      "1.282 - terrible\n"
     ]
    }
   ],
   "source": [
    "print_sorted_words('great', metric='cosine')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can print the words that are near and far from the word 'worst' in terms of their vector-embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distance from 'worst':\n",
      "0.000 - worst\n",
      "0.047 - embarrassingly\n",
      "0.053 - terrible\n",
      "0.094 - retarded\n",
      "0.095 - poor\n",
      "0.095 - stereotyping\n",
      "0.096 - uninspired\n",
      "0.099 - awful\n",
      "0.100 - severed\n",
      "0.108 - lacks\n",
      "...\n",
      "1.167 - restraint\n",
      "1.168 - available\n",
      "1.176 - foremost\n",
      "1.188 - great\n",
      "1.193 - mesmerizing\n",
      "1.222 - highly\n",
      "1.229 - exploration\n",
      "1.239 - delightful\n",
      "1.268 - wonderfully\n",
      "1.323 - 7\n"
     ]
    }
   ],
   "source": [
    "print_sorted_words('worst', metric='cosine')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This tutorial showed the basic methods for doing Natural Language Processing (NLP) using a Recurrent Neural Network with integer-tokens and an embedding layer. This was used to do sentiment analysis of movie reviews from IMDB. It works reasonably well if the hyper-parameters are chosen properly. But it is important to understand that this is not human-like comprehension of text. The system does not have any real understanding of the text. It is just a clever way of doing pattern-recognition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "These are a few suggestions for exercises that may help improve your skills with TensorFlow. It is important to get hands-on experience with TensorFlow in order to learn how to use it properly.\n",
    "\n",
    "You may want to backup this Notebook before making any changes.\n",
    "\n",
    "* Run more training-epochs. Does it improve performance?\n",
    "* If your model overfits the training-data, try using dropout-layers and dropout inside the GRU.\n",
    "* Increase or decrease the number of words in the vocabulary. This is done when the `Tokenizer` is initialized. Does it affect performance?\n",
    "* Increase the size of the embedding-vectors to e.g. 200. Does it affect performance?\n",
    "* Try varying all the different hyper-parameters for the Recurrent Neural Network.\n",
    "* Use Bayesian Optimization from Tutorial #19 to find the best choice of hyper-parameters.\n",
    "* Use 'post' for padding and truncating in `pad_sequences()`. Does it affect the performance?\n",
    "* Use individual characters instead of tokenized words as the vocabulary. You can then use one-hot encoded vectors for each character instead of using the embedding-layer.\n",
    "* Use `model.fit_generator()` instead of `model.fit()` and make your own data-generator, which creates a batch of data using a random subset of `x_train_tokens`. The sequences must be padded so they all match the length of the longest sequence.\n",
    "* Explain to a friend how the program works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License (MIT)\n",
    "\n",
    "Copyright (c) 2018 by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}