IMDB_Movie_review.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "IMDB Movie review.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dipanjannC/Deep-Learning/blob/master/IMDB_Movie_review.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uUyguAf0aBVW",
        "colab_type": "text"
      },
      "source": [
        "<h3>Problem Statement</h3>\n",
        "The problem that we will use is IMDB movie review sentiment classification problem. Each movie review is a variable sequence of words and the sentiment of each movie review must be classified.\n",
        "\n",
        "The Large Movie Review Dataset (often referred to as the IMDB dataset) contains 25,000 highly-polar movie reviews (good or bad) for training and the same amount again for testing. The problem is to determine whether a given movie review has a positive or negative sentiment.\n",
        "\n",
        "The data was collected by Stanford researchers and was used in a 2011 paper where a split of 50-50 of the data was used for training and test. An accuracy of 88.89% was achieved.\n",
        "\n",
        "Keras provides access to the IMDB dataset built-in. The imdb.load_data() function allows you to load the dataset in a format that is ready for use in neural network and deep learning models.\n",
        "\n",
        "The words have been replaced by integers that indicate the ordered frequency of each word in the dataset. The sentences in each review are therefore comprised of a sequence of integers.\n",
        "\n",
        "credits : https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iQxvkCQ-aIue",
        "colab_type": "code",
        "outputId": "e1c10f5a-e5ce-49c3-d62e-a83e9685d780",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "!pip install numpy\n",
        "import numpy as np\n",
        "from keras.datasets import imdb\n",
        "from keras.layers import Dense\n",
        "from keras.layers import LSTM\n",
        "from keras.models import Sequential\n",
        "from keras.layers.embeddings import Embedding\n",
        "from keras.preprocessing import sequence\n",
        "# fix random seed for reproducibility\n",
        "np.random.seed(7)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (1.16.4)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iMcfoIs9upJQ",
        "colab_type": "code",
        "outputId": "4e96cf56-64a5-4008-f1a4-12c33e70219c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# save np.load\n",
        "np_load_old = np.load\n",
        "\n",
        "# modify the default parameters of np.load\n",
        "np.load = lambda *a,**k: np_load_old(*a, allow_pickle=True, **k)\n",
        "\n",
        "# load the dataset but only keep the top n words, zero the rest\n",
        "top_words = 5000\n",
        "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=top_words)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz\n",
            "17465344/17464789 [==============================] - 1s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9lF6MDADHqN_",
        "colab_type": "code",
        "outputId": "66bc10f9-4959-4a4e-a0de-39d2ee706683",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 92
        }
      },
      "source": [
        "print(X_train[1])\n",
        "print(type(X_train[1]))\n",
        "print(len(X_train[1]))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[1, 194, 1153, 194, 2, 78, 228, 5, 6, 1463, 4369, 2, 134, 26, 4, 715, 8, 118, 1634, 14, 394, 20, 13, 119, 954, 189, 102, 5, 207, 110, 3103, 21, 14, 69, 188, 8, 30, 23, 7, 4, 249, 126, 93, 4, 114, 9, 2300, 1523, 5, 647, 4, 116, 9, 35, 2, 4, 229, 9, 340, 1322, 4, 118, 9, 4, 130, 4901, 19, 4, 1002, 5, 89, 29, 952, 46, 37, 4, 455, 9, 45, 43, 38, 1543, 1905, 398, 4, 1649, 26, 2, 5, 163, 11, 3215, 2, 4, 1153, 9, 194, 775, 7, 2, 2, 349, 2637, 148, 605, 2, 2, 15, 123, 125, 68, 2, 2, 15, 349, 165, 4362, 98, 5, 4, 228, 9, 43, 2, 1157, 15, 299, 120, 5, 120, 174, 11, 220, 175, 136, 50, 9, 4373, 228, 2, 5, 2, 656, 245, 2350, 5, 4, 2, 131, 152, 491, 18, 2, 32, 2, 1212, 14, 9, 6, 371, 78, 22, 625, 64, 1382, 9, 8, 168, 145, 23, 4, 1690, 15, 16, 4, 1355, 5, 28, 6, 52, 154, 462, 33, 89, 78, 285, 16, 145, 95]\n",
            "<class 'list'>\n",
            "189\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TGaL0HYhDeBu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# truncate and pad input sequences\n",
        "max_review_length = 500\n",
        "X_train = sequence.pad_sequences(X_train, maxlen=max_review_length)\n",
        "X_test = sequence.pad_sequences(X_test, maxlen=max_review_length)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GbBwrrwDEpwJ",
        "colab_type": "text"
      },
      "source": [
        "We can now define, compile and fit our LSTM model.\n",
        "\n",
        "The first layer is the Embedded layer that uses 32 length vectors to represent each word. The next layer is the LSTM layer with 100 memory units (smart neurons). Finally, because this is a classification problem we use a Dense output layer with a single neuron and a sigmoid activation function to make 0 or 1 predictions for the two classes (good and bad) in the problem.\n",
        "\n",
        "Because it is a binary classification problem, log loss is used as the loss function (binary_crossentropy in Keras). The efficient ADAM optimization algorithm is used. The model is fit for only 2 epochs because it quickly overfits the problem. A large batch size of 64 reviews is used to space out weight updates."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TUUhTzShEgxt",
        "colab_type": "code",
        "outputId": "b96ca9d6-b545-43ef-965e-884391e5aa7a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 715
        }
      },
      "source": [
        "print(X_train[1])\n",
        "print(type(X_train[1]))\n",
        "print(len(X_train[1]))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[   0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    1  194 1153  194    2   78  228    5    6 1463 4369\n",
            "    2  134   26    4  715    8  118 1634   14  394   20   13  119  954\n",
            "  189  102    5  207  110 3103   21   14   69  188    8   30   23    7\n",
            "    4  249  126   93    4  114    9 2300 1523    5  647    4  116    9\n",
            "   35    2    4  229    9  340 1322    4  118    9    4  130 4901   19\n",
            "    4 1002    5   89   29  952   46   37    4  455    9   45   43   38\n",
            " 1543 1905  398    4 1649   26    2    5  163   11 3215    2    4 1153\n",
            "    9  194  775    7    2    2  349 2637  148  605    2    2   15  123\n",
            "  125   68    2    2   15  349  165 4362   98    5    4  228    9   43\n",
            "    2 1157   15  299  120    5  120  174   11  220  175  136   50    9\n",
            " 4373  228    2    5    2  656  245 2350    5    4    2  131  152  491\n",
            "   18    2   32    2 1212   14    9    6  371   78   22  625   64 1382\n",
            "    9    8  168  145   23    4 1690   15   16    4 1355    5   28    6\n",
            "   52  154  462   33   89   78  285   16  145   95]\n",
            "<class 'numpy.ndarray'>\n",
            "500\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MqZCdUYZjTl8",
        "colab_type": "text"
      },
      "source": [
        "<h3>Model Description</h3>\n",
        "\n",
        "1.   The first layer is the Embedded layer that uses 32 length vectors to represent each word. The next layer is the LSTM layer with 100 memory units (smart neurons). Finally, because this is a classification problem we use a Dense output layer with a single neuron and a sigmoid activation function to make 0 or 1 predictions for the two classes (good and bad) in the problem.\n",
        "\n",
        "\n",
        "2.   Because it is a binary classification problem, log loss is used as the loss function (binary_crossentropy in Keras). The efficient ADAM optimization algorithm is used. The model is fit for only 2 epochs because it quickly overfits the problem. A large batch size of 64 reviews is used to space out weight updates.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L0y6pKlrHfHS",
        "colab_type": "code",
        "outputId": "428f85ae-7774-49e2-8a0b-6147d6a90694",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 274
        }
      },
      "source": [
        "\t# create the model\n",
        "embedding_vecor_length = 32\n",
        "model = Sequential()\n",
        "model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))\n",
        "model.add(LSTM(100))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "print(model.summary())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_3 (Embedding)      (None, 500, 32)           160000    \n",
            "_________________________________________________________________\n",
            "lstm_3 (LSTM)                (None, 100)               53200     \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 1)                 101       \n",
            "=================================================================\n",
            "Total params: 213,301\n",
            "Trainable params: 213,301\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p6lA4CHcjI62",
        "colab_type": "code",
        "outputId": "5b55469a-c151-44b9-e9ba-960082802333",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 439
        }
      },
      "source": [
        "model.fit(X_train, y_train, nb_epoch=10, batch_size=64)\n",
        "# Final evaluation of the model\n",
        "scores = model.evaluate(X_test, y_test, verbose=0)\n",
        "print(\"Accuracy: %.2f%%\" % (scores[1]*100))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
            "  \"\"\"Entry point for launching an IPython kernel.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "25000/25000 [==============================] - 195s 8ms/step - loss: 0.4490 - acc: 0.7738\n",
            "Epoch 2/10\n",
            "25000/25000 [==============================] - 194s 8ms/step - loss: 0.3360 - acc: 0.8533\n",
            "Epoch 3/10\n",
            "25000/25000 [==============================] - 194s 8ms/step - loss: 0.2731 - acc: 0.8924\n",
            "Epoch 4/10\n",
            "25000/25000 [==============================] - 194s 8ms/step - loss: 0.2247 - acc: 0.9124\n",
            "Epoch 5/10\n",
            "25000/25000 [==============================] - 193s 8ms/step - loss: 0.1980 - acc: 0.9255\n",
            "Epoch 6/10\n",
            "25000/25000 [==============================] - 194s 8ms/step - loss: 0.2624 - acc: 0.8856\n",
            "Epoch 7/10\n",
            "25000/25000 [==============================] - 194s 8ms/step - loss: 0.1998 - acc: 0.9212\n",
            "Epoch 8/10\n",
            "25000/25000 [==============================] - 193s 8ms/step - loss: 0.3208 - acc: 0.8614\n",
            "Epoch 9/10\n",
            "25000/25000 [==============================] - 194s 8ms/step - loss: 0.1670 - acc: 0.9372\n",
            "Epoch 10/10\n",
            "25000/25000 [==============================] - 194s 8ms/step - loss: 0.1511 - acc: 0.9454\n",
            "Accuracy: 86.01%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tM6Lbt30n02o",
        "colab_type": "text"
      },
      "source": [
        "<h3>LSTM Model with dropout </h3>\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6w0wm7yCjzkv",
        "colab_type": "code",
        "outputId": "7d1cdb2f-7dfc-48ae-ba55-08ba57bce637",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        }
      },
      "source": [
        "from keras.layers import Dropout\n",
        "model = Sequential()\n",
        "model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))\n",
        "model.add(Dropout(0.2))\n",
        "model.add(LSTM(100))\n",
        "model.add(Dropout(0.2))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "print(model.summary())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "W0625 17:46:17.303641 139706937862016 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_5 (Embedding)      (None, 500, 32)           160000    \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 500, 32)           0         \n",
            "_________________________________________________________________\n",
            "lstm_4 (LSTM)                (None, 100)               53200     \n",
            "_________________________________________________________________\n",
            "dropout_2 (Dropout)          (None, 100)               0         \n",
            "_________________________________________________________________\n",
            "dense_4 (Dense)              (None, 1)                 101       \n",
            "=================================================================\n",
            "Total params: 213,301\n",
            "Trainable params: 213,301\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VmvTIyqeoqYh",
        "colab_type": "code",
        "outputId": "d1db3fc4-23ca-4ae1-a6f0-89b663cdfc9d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 164
        }
      },
      "source": [
        "model.fit(X_train, y_train, epochs=3, batch_size=64)\n",
        "# Final evaluation of the model\n",
        "scores = model.evaluate(X_test, y_test, verbose=0)\n",
        "print(scores)\n",
        "print(\"Accuracy: %.2f%%\" % (scores[1]*100))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "25000/25000 [==============================] - 200s 8ms/step - loss: 0.5071 - acc: 0.7504\n",
            "Epoch 2/3\n",
            "25000/25000 [==============================] - 199s 8ms/step - loss: 0.3165 - acc: 0.8695\n",
            "Epoch 3/3\n",
            "25000/25000 [==============================] - 197s 8ms/step - loss: 0.2819 - acc: 0.8907\n",
            "[0.34639126547813415, 0.86008]\n",
            "Accuracy: 86.01%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_GeIyJNXvvHz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7kLIgt0Z0RBU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}