12. Sequence Labeling/12-5 ~ 12-7. ner_using_bilstm_cnn_crf.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Named Entity Recognition using BiLSTM-CNN-CRF (2021-12-27).ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7FrWBlgOqqWf"
      },
      "source": [
        "이 자료는 위키독스 딥 러닝을 이용한 자연어 처리 입문의 BiLSTM + CRF + 문자 임베딩의 튜토리얼 자료입니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E2QPT2-kqgWs"
      },
      "source": [
        "링크 : https://wikidocs.net/147219  \n",
        "링크 : https://wikidocs.net/147234  \n",
        "링크 : https://wikidocs.net/147299"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "이 자료는 2021년 12월 27일에 마지막으로 테스트되었습니다."
      ],
      "metadata": {
        "id": "-GpAkY7OQNE1"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "BLaXWXiV2ypI",
        "outputId": "4df5374e-726d-46fb-f8ee-fed40b6809fc"
      },
      "source": [
        "import tensorflow as tf\n",
        "tf.__version__"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'2.7.0'"
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pPthmTBbODpC"
      },
      "source": [
        "# CRF layer 설치"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "H9S1I2HIlfAH",
        "outputId": "10a8bf74-b0fa-41cd-85f3-8fe2cfa41fa6"
      },
      "source": [
        "!pip install keras-crf==0.3.0"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting keras-crf==0.3.0\n",
            "  Downloading keras_crf-0.3.0-py3-none-any.whl (8.3 kB)\n",
            "Collecting tensorflow-addons\n",
            "  Downloading tensorflow_addons-0.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1 MB 5.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tensorflow in /usr/local/lib/python3.7/dist-packages (from keras-crf==0.3.0) (2.7.0)\n",
            "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (0.22.0)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (1.15.0)\n",
            "Requirement already satisfied: libclang>=9.0.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (12.0.0)\n",
            "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (3.1.0)\n",
            "Requirement already satisfied: absl-py>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (0.12.0)\n",
            "Requirement already satisfied: flatbuffers<3.0,>=1.12 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (2.0)\n",
            "Requirement already satisfied: tensorflow-estimator<2.8,~=2.7.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (2.7.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (3.10.0.2)\n",
            "Requirement already satisfied: gast<0.5.0,>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (0.4.0)\n",
            "Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (3.17.3)\n",
            "Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (1.13.3)\n",
            "Requirement already satisfied: tensorboard~=2.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (2.7.0)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (0.2.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (1.1.0)\n",
            "Requirement already satisfied: numpy>=1.14.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (1.19.5)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (1.1.2)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (3.3.0)\n",
            "Requirement already satisfied: keras<2.8,>=2.7.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (2.7.0)\n",
            "Requirement already satisfied: wheel<1.0,>=0.32.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (0.37.0)\n",
            "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (1.6.3)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow->keras-crf==0.3.0) (1.42.0)\n",
            "Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py>=2.9.0->tensorflow->keras-crf==0.3.0) (1.5.2)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (1.8.0)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (1.35.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (3.3.6)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (2.23.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (0.4.6)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (57.4.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (0.6.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (1.0.1)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (4.2.4)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (4.8)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (1.3.0)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (4.8.2)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (3.6.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (0.4.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (2021.10.8)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (1.24.3)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->tensorflow->keras-crf==0.3.0) (3.1.1)\n",
            "Requirement already satisfied: typeguard>=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow-addons->keras-crf==0.3.0) (2.7.1)\n",
            "Installing collected packages: tensorflow-addons, keras-crf\n",
            "Successfully installed keras-crf-0.3.0 tensorflow-addons-0.15.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "329OHwxK-1if",
        "outputId": "36437295-b18e-4fd8-c29b-20a00623d9ba"
      },
      "source": [
        "# 모델 평가\n",
        "!pip install seqeval"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting seqeval\n",
            "  Downloading seqeval-1.2.2.tar.gz (43 kB)\n",
            "\u001b[?25l\r\u001b[K     |███████▌                        | 10 kB 27.1 MB/s eta 0:00:01\r\u001b[K     |███████████████                 | 20 kB 8.8 MB/s eta 0:00:01\r\u001b[K     |██████████████████████▌         | 30 kB 7.6 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████  | 40 kB 7.2 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 43 kB 828 kB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from seqeval) (1.19.5)\n",
            "Requirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.7/dist-packages (from seqeval) (1.0.1)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval) (1.1.0)\n",
            "Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval) (1.4.1)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval) (3.0.0)\n",
            "Building wheels for collected packages: seqeval\n",
            "  Building wheel for seqeval (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for seqeval: filename=seqeval-1.2.2-py3-none-any.whl size=16181 sha256=487266b3ca3a8ba64c2e4741beffd6bbf3a7f3836117a70c4fa7f4ce54510c12\n",
            "  Stored in directory: /root/.cache/pip/wheels/05/96/ee/7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7\n",
            "Successfully built seqeval\n",
            "Installing collected packages: seqeval\n",
            "Successfully installed seqeval-1.2.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L9gruUeWo-tI",
        "outputId": "e5352579-da2c-4a3e-9109-cea79fc34ca4"
      },
      "source": [
        "!pip list | grep keras-crf"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "keras-crf                     0.3.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NqD8wMcd7evE"
      },
      "source": [
        "ner_dataset.csv를 Colab에 업로드하세요."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p7yAfzbN7aXk"
      },
      "source": [
        "# 데이터 전처리"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BDfQP9b6yXPw"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "from sklearn.model_selection import train_test_split\n",
        "from tensorflow.keras.utils import to_categorical"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qo5Q_PDwycAo"
      },
      "source": [
        "data = pd.read_csv(\"ner_dataset.csv\", encoding=\"latin1\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "esAWxTIizDtS",
        "outputId": "da377b07-d535-404c-ab58-263f79c61a68"
      },
      "source": [
        "data[:5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-8e82288f-b591-450d-89f9-04616ca866d1\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Sentence #</th>\n",
              "      <th>Word</th>\n",
              "      <th>POS</th>\n",
              "      <th>Tag</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Sentence: 1</td>\n",
              "      <td>Thousands</td>\n",
              "      <td>NNS</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "      <td>of</td>\n",
              "      <td>IN</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>NNS</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NaN</td>\n",
              "      <td>have</td>\n",
              "      <td>VBP</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NaN</td>\n",
              "      <td>marched</td>\n",
              "      <td>VBN</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8e82288f-b591-450d-89f9-04616ca866d1')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-8e82288f-b591-450d-89f9-04616ca866d1 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-8e82288f-b591-450d-89f9-04616ca866d1');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "    Sentence #           Word  POS Tag\n",
              "0  Sentence: 1      Thousands  NNS   O\n",
              "1          NaN             of   IN   O\n",
              "2          NaN  demonstrators  NNS   O\n",
              "3          NaN           have  VBP   O\n",
              "4          NaN        marched  VBN   O"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "31B1kMVYzFMd",
        "outputId": "e9dccb79-7f37-4042-a150-91fb7ad36034"
      },
      "source": [
        "print('데이터프레임 행의 개수 : {}'.format(len(data)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "데이터프레임 행의 개수 : 1048575\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5qgElyBDzlXu",
        "outputId": "f0fc9773-7146-4754-c89b-15c474adb3d5"
      },
      "source": [
        "print('데이터에 Null 값이 있는지 유무 : ' + str(data.isnull().values.any()))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "데이터에 Null 값이 있는지 유무 : True\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mUKvYcMg3HCH",
        "outputId": "347648c2-1598-4e33-b7ef-581f747e281c"
      },
      "source": [
        "print('어떤 열에 Null값이 있는지 출력')\n",
        "print('==============================')\n",
        "data.isnull().sum()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "어떤 열에 Null값이 있는지 출력\n",
            "==============================\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Sentence #    1000616\n",
              "Word                0\n",
              "POS                 0\n",
              "Tag                 0\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D6V0rIuF3Jiv",
        "outputId": "be2f9f2a-77a6-4a20-802a-2d165cc37198"
      },
      "source": [
        "print('sentence # 열의 중복을 제거한 값의 개수 : {}'.format(data['Sentence #'].nunique()))\n",
        "print('Word 열의 중복을 제거한 값의 개수 : {}'.format(data.Word.nunique()))\n",
        "print('Tag 열의 중복을 제거한 값의 개수 : {}'.format(data.Tag.nunique()))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "sentence # 열의 중복을 제거한 값의 개수 : 47959\n",
            "Word 열의 중복을 제거한 값의 개수 : 35178\n",
            "Tag 열의 중복을 제거한 값의 개수 : 17\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5pb88O9o3LRo",
        "outputId": "73179732-15fe-4b39-82ca-883c80c8cba6"
      },
      "source": [
        "print('Tag 열의 각각의 값의 개수 카운트')\n",
        "print('================================')\n",
        "print(data.groupby('Tag').size().reset_index(name='count'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tag 열의 각각의 값의 개수 카운트\n",
            "================================\n",
            "      Tag   count\n",
            "0   B-art     402\n",
            "1   B-eve     308\n",
            "2   B-geo   37644\n",
            "3   B-gpe   15870\n",
            "4   B-nat     201\n",
            "5   B-org   20143\n",
            "6   B-per   16990\n",
            "7   B-tim   20333\n",
            "8   I-art     297\n",
            "9   I-eve     253\n",
            "10  I-geo    7414\n",
            "11  I-gpe     198\n",
            "12  I-nat      51\n",
            "13  I-org   16784\n",
            "14  I-per   17251\n",
            "15  I-tim    6528\n",
            "16      O  887908\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kobBX39g3Mfc"
      },
      "source": [
        "data = data.fillna(method=\"ffill\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "95-AHZjn3Nso",
        "outputId": "4048d9cf-2518-407d-cfe7-f53bdaf4534e"
      },
      "source": [
        "print(data.tail())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              Sentence #       Word  POS Tag\n",
            "1048570  Sentence: 47959       they  PRP   O\n",
            "1048571  Sentence: 47959  responded  VBD   O\n",
            "1048572  Sentence: 47959         to   TO   O\n",
            "1048573  Sentence: 47959        the   DT   O\n",
            "1048574  Sentence: 47959     attack   NN   O\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EOdOzFb43QzV",
        "outputId": "545eb889-74c9-4f98-b603-ea0d3b99be57"
      },
      "source": [
        "print('데이터에 Null 값이 있는지 유무 : ' + str(data.isnull().values.any()))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "데이터에 Null 값이 있는지 유무 : False\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9ez83REt3TNT",
        "outputId": "6e93a7e1-29f7-407c-d985-df8036aa0d6e"
      },
      "source": [
        "data['Word'] = data['Word'].str.lower()\n",
        "print('Word 열의 중복을 제거한 값의 개수 : {}'.format(data.Word.nunique()))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Word 열의 중복을 제거한 값의 개수 : 31817\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n1VSrB3z3UeP",
        "outputId": "32c66c81-e28a-4d53-96ff-57c8c85df4d8"
      },
      "source": [
        "print(data[:5])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "    Sentence #           Word  POS Tag\n",
            "0  Sentence: 1      thousands  NNS   O\n",
            "1  Sentence: 1             of   IN   O\n",
            "2  Sentence: 1  demonstrators  NNS   O\n",
            "3  Sentence: 1           have  VBP   O\n",
            "4  Sentence: 1        marched  VBN   O\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5v7gEF8u3Vyw",
        "outputId": "bb96e4b5-3aa4-45e8-bc5d-32a136c54f20"
      },
      "source": [
        "func = lambda temp: [(w, t) for w, t in zip(temp[\"Word\"].values.tolist(), temp[\"Tag\"].values.tolist())]\n",
        "tagged_sentences=[t for t in data.groupby(\"Sentence #\").apply(func)]\n",
        "print(\"전체 샘플 개수: {}\".format(len(tagged_sentences)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "전체 샘플 개수: 47959\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BP3eD9h53W_D",
        "outputId": "d5118cbe-8a8d-4eb0-e7db-468061dfe887"
      },
      "source": [
        "print(tagged_sentences[0]) # 첫번째 샘플 출력"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('thousands', 'O'), ('of', 'O'), ('demonstrators', 'O'), ('have', 'O'), ('marched', 'O'), ('through', 'O'), ('london', 'B-geo'), ('to', 'O'), ('protest', 'O'), ('the', 'O'), ('war', 'O'), ('in', 'O'), ('iraq', 'B-geo'), ('and', 'O'), ('demand', 'O'), ('the', 'O'), ('withdrawal', 'O'), ('of', 'O'), ('british', 'B-gpe'), ('troops', 'O'), ('from', 'O'), ('that', 'O'), ('country', 'O'), ('.', 'O')]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BDIr2WoF3YKu"
      },
      "source": [
        "sentences, ner_tags = [], [] \n",
        "for tagged_sentence in tagged_sentences: # 47,959개의 문장 샘플을 1개씩 불러온다.\n",
        "    sentence, tag_info = zip(*tagged_sentence) # 각 샘플에서 단어들은 sentence에 개체명 태깅 정보들은 tag_info에 저장.\n",
        "    sentences.append(list(sentence)) # 각 샘플에서 단어 정보만 저장한다.\n",
        "    ner_tags.append(list(tag_info)) # 각 샘플에서 개체명 태깅 정보만 저장한다."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L0Qcw-VN3aZp",
        "outputId": "8e44f1b6-9a68-4f08-a864-70ccc0138a4c"
      },
      "source": [
        "print(sentences[0])\n",
        "print(ner_tags[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['thousands', 'of', 'demonstrators', 'have', 'marched', 'through', 'london', 'to', 'protest', 'the', 'war', 'in', 'iraq', 'and', 'demand', 'the', 'withdrawal', 'of', 'british', 'troops', 'from', 'that', 'country', '.']\n",
            "['O', 'O', 'O', 'O', 'O', 'O', 'B-geo', 'O', 'O', 'O', 'O', 'O', 'B-geo', 'O', 'O', 'O', 'O', 'O', 'B-gpe', 'O', 'O', 'O', 'O', 'O']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zLglFWkC3blL",
        "outputId": "f6c42a2b-ab67-4c16-bdf5-10feff60b34f"
      },
      "source": [
        "print(sentences[98])\n",
        "print(ner_tags[98])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['she', 'had', 'once', 'received', 'a', 'kidney', 'transplant', '.']\n",
            "['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 315
        },
        "id": "tVvW72m63c0L",
        "outputId": "dd9c08d8-163e-409b-aaf1-44845b1a5cb6"
      },
      "source": [
        "print('샘플의 최대 길이 : %d' % max(len(l) for l in sentences))\n",
        "print('샘플의 평균 길이 : %f' % (sum(map(len, sentences))/len(sentences)))\n",
        "plt.hist([len(s) for s in sentences], bins=50)\n",
        "plt.xlabel('length of samples')\n",
        "plt.ylabel('number of samples')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "샘플의 최대 길이 : 104\n",
            "샘플의 평균 길이 : 21.863988\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAZXklEQVR4nO3de7BlZXnn8e9PUPCCAoIUNsSGkfKWRMQWsCQOagIojuiMIkZDiygVxwTMeAlER7xGKBPxNhJRiK2jIuUNRimxB0HiqEg3MHLTgkgz0EFpbeQiEQWe+WO9R7eHPr12d599zj77fD9Vu85a77rsZ7Ga85z3Xe9631QVkiRtzAPmOwBJ0vgzWUiSepksJEm9TBaSpF4mC0lSr63nO4BR2GmnnWrp0qXzHYYkLSirV6/+WVXtvKFtI00WSdYAdwD3AvdU1bIkOwKfB5YCa4DDq+rWJAE+CDwPuAt4ZVVd2s6zHHhrO+27q2rFxr536dKlrFq1avYvSJImWJIbZto2F81Qz6qqvatqWVs/Hji/qvYCzm/rAM8F9mqfY4BTAVpyORHYD9gXODHJDnMQtySpmY9nFocBUzWDFcALB8o/VZ3vAdsn2RU4GFhZVeur6lZgJXDIXActSYvZqJNFAd9IsjrJMa1sl6q6uS3/BNilLS8Bbhw49qZWNlP570lyTJJVSVatW7duNq9Bkha9UT/gPqCq1iZ5FLAyyQ8HN1ZVJZmV8Uaq6jTgNIBly5Y5hokkzaKR1iyqam37eQvwZbpnDj9tzUu0n7e03dcCuw8cvlsrm6lckjRHRpYskjw0yXZTy8BBwJXAOcDyttty4Oy2fA5wZDr7A7e15qrzgIOS7NAebB/UyiRJc2SUzVC7AF/uesSyNfDZqvp6kkuAs5IcDdwAHN72P5eu2+x1dF1njwKoqvVJ3gVc0vZ7Z1WtH2HckqRpMolDlC9btqx8z0KSNk2S1QOvOfweh/uQJPWayOE+tGFLj//aBsvXnHToHEciaaGxZiFJ6mWykCT1MllIknqZLCRJvUwWkqRe9obSjL2kwJ5SkjrWLCRJvUwWkqReJgtJUi+ThSSpl8lCktTL3lATaGO9myRpc1izkCT1MllIknqZLCRJvUwWkqRePuDWRjlhkiSwZiFJGoLJQpLUy2QhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiYLSVIvk4UkqZfJQpLUy2QhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiNPFkm2SnJZkq+29T2SXJzkuiSfT/KgVr5NW7+ubV86cI4TWvmPkhw86pglSb9vLmoWxwHXDKyfDJxSVY8FbgWObuVHA7e28lPafiR5InAE8CTgEOCjSbaag7glSc1Ik0WS3YBDgU+09QDPBr7QdlkBvLAtH9bWaduf0/Y/DDizqu6uquuB64B9Rxm3JOn3jbpm8QHgzcB9bf2RwC+q6p62fhOwpC0vAW4EaNtva/v/tnwDx/xWkmOSrEqyat26dbN9HZK0qI1sDu4kzwduqarVSQ4c1fdMqarTgNMAli1bVqP+vnEw0/zYkjTbRpYsgGcAL0jyPGBb4OHAB4Htk2zdag+7AWvb/muB3YGbkmwNPAL4+UD5lMFjJElzYGTNUFV1QlXtVlVL6R5Qf7OqXg5cALy47bYcOLstn9PWadu/WVXVyo9ovaX2APYCvj+quCVJ9zfKmsVM/hY4M8m7gcuA01v56cCnk1wHrKdLMFTVVUnOAq4G7gFeV1X3zn3YkrR4zUmyqKoLgQvb8o/ZQG+mqvoV8JIZjn8P8J7RRShJ2hjf4JYk9TJZSJJ6zcczC02Ambrtrjnp0DmORNJcsGYhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiYLSVIvk4UkqVdvskjykiTbteW3JvlSkn1GH5okaVwMU7P471V1R5IDgD+lG/Dv1NGGJUkaJ8Mki6kRXg8FTquqrwEPGl1IkqRxM0yyWJvkY8BLgXOTbDPkcZKkCTHML/3DgfOAg6vqF8COwJtGGpUkaaz0DiRYVXcluQU4ALiWbgKia0cdmH7HubYlzbdhekOdSDe73Qmt6IHA/xxlUJKk8TJMM9SLgBcAvwSoqn8DthtlUJKk8TJMsvh1VRVQAEkeOtqQJEnjZphkcVbrDbV9ktcA/xv4+GjDkiSNk2EecP9Dkj8DbgceB7ytqlaOPDJJ0tgYalrVlhxMEJK0SM2YLJLcQXtOMX0TUFX18JFFJUkaKzMmi6qyx5MkCRiyGaqNMnsAXU3j21V12UijkiSNlWFeynsbsAJ4JLAT8Mkkbx11YJKk8TFMzeLlwJOr6lcASU4CLgfePcrAJEnjY5j3LP4N2HZgfRtg7WjCkSSNo2FqFrcBVyVZSffM4s+A7yf5EEBVHTvC+CRJY2CYZPHl9ply4WhCkSSNq2He4F4xF4FIksbXML2hnp/ksiTrk9ye5I4kt89FcJKk8TBMM9QHgP8MXNFGn5VmNNNETWtOOnSOI5E0m4bpDXUjcKWJQpIWr2FqFm8Gzk3yLeDuqcKqev/GDkqyLXARXVfbrYEvVNWJSfYAzqR7yW818BdV9esk2wCfAp4K/Bx4aVWtaec6ATgauBc4tqrO26SrlCRtkWFqFu8B7qJ712K7gU+fu4FnV9WTgb2BQ5LsD5wMnFJVjwVupUsCtJ+3tvJT2n4keSJwBPAk4BDgo0m2Gu7yJEmzYZiaxaOr6g839cSt2erOtvrA9ing2cCft/IVwNuBU4HD2jLAF4CPJEkrP7Oq7gauT3IdsC/w3U2NSZK0eYapWZyb5KDNOXmSrZJcDtxCNx/GvwK/qKp72i43AUva8hK65yO07bfRNVX9tnwDxwx+1zFJViVZtW7dus0JV5I0g2GSxWuBryf5903tOltV91bV3sBudLWBx29BrH3fdVpVLauqZTvvvPOovkaSFqVhXsrb4nktquoXSS4Ank43l/fWrfawG78bZ2otsDtwU5KtgUfQPeieKp8yeIwkaQ4MU7MgyQ5J9k3yzKnPEMfsnGT7tvxgujGlrgEuAF7cdlsOnN2Wz2nrtO3fbM89zgGOSLJN60m1F/D94S5PkjQbemsWSV4NHEf3F/3lwP50D5ef3XPorsCK1nPpAcBZVfXVJFcDZyZ5N3AZcHrb/3Tg0+0B9nq6HlBU1VVJzgKuBu4BXldV927aZUqStsQwvaGOA54GfK+qnpXk8cDf9x1UVT8AnrKB8h/TPb+YXv4r4CUznOs9dF14JUnzYJhmqF8NTHy0TVX9EHjcaMOSJI2TYWoWN7VnD18BVia5FbhhtGFJksbJML2hXtQW3956ND0C+PpIo1qkZhqET5Lm2zBDlP+HNm4TQIClwENGGZQkabwM88zii8C9SR4LnEb3zsNnRxqVJGmsDJMs7msv0L0I+HBVvYmuW6wkaZEYJln8JsnL6F6Y+2ore+DoQpIkjZthksVRdMN0vKeqrm9vUX96tGFJksbJML2hrgaOHVi/njbXhCRpcRhqbChJ0uJmspAk9ZoxWST5dPt53NyFI0kaRxurWTw1yaOBV7Uhyncc/MxVgJKk+bexB9z/BJwP7Amspnt7e0q1cknSIjBjzaKqPlRVTwDOqKo9q2qPgY+JQpIWkWG6zr42yZOBP2lFF7W5KiRJi8QwAwkeC3wGeFT7fCbJX486MEnS+BhmPotXA/tV1S8BkpxMN63qh0cZmCRpfAyTLAIMznl9L7//sFvqNdNcHWtOOnSOI5G0OYZJFv8MXJzky239hcDpowtJkjRuhnnA/f4kFwIHtKKjquqykUYlSRorw9QsqKpLgUtHHIskaUw5NpQkqZfJQpLUa6PJIslWSS6Yq2AkSeNpo8miqu4F7kvyiDmKR5I0hoZ5wH0ncEWSlcAvpwqr6tiZD5EkTZJhksWX2keStEgN857FiiQPBv6gqn40BzFJksbMMAMJ/ifgcuDrbX3vJOeMOjBJ0vgYphnq7cC+wIUAVXV5Euez2AIzjZMkSeNqmPcsflNVt00ru28UwUiSxtMwNYurkvw5sFWSvYBjge+MNixJ0jgZpmbx18CTgLuBzwG3A6/vOyjJ7kkuSHJ1kquSHNfKd0yyMsm17ecOrTxJPpTkuiQ/SLLPwLmWt/2vTbJ8cy5UkrT5hukNdRfwljbpUVXVHUOe+x7gDVV1aZLtgNXtXY1XAudX1UlJjgeOB/4WeC6wV/vsB5wK7JdkR+BEYBlQ7TznVNWtm3KhkqTNN0xvqKcluQL4Ad3Lef83yVP7jquqm9totbQEcw2wBDgMWNF2W0E3Pwat/FPV+R6wfZJdgYOBlVW1viWIlcAhm3SVkqQtMkwz1OnAf62qpVW1FHgd3YRIQ0uyFHgKcDGwS1Xd3Db9BNilLS8Bbhw47KZWNlP59O84JsmqJKvWrVu3KeFJknoMkyzurap/mVqpqm/TNTENJcnDgC8Cr6+q2we3VVXRNS1tsao6raqWVdWynXfeeTZOKUlqZnxmMfCA+VtJPkb3cLuAl9LeueiT5IF0ieIzVTU1ZMhPk+xaVTe3ZqZbWvlaYPeBw3drZWuBA6eVD/X9kqTZsbEH3P84bf3EgeXe2kCS0DVhXVNV7x/YdA6wHDip/Tx7oPyvkpxJ94D7tpZQzgP+fqrXFHAQcELf90uSZs+MyaKqnrWF534G8Bd0D8Uvb2V/R5ckzkpyNHADcHjbdi7wPOA64C7gqBbH+iTvAi5p+72zqtZvYWySpE3Q23U2yfbAkcDSwf37hihvzzYyw+bnbGD/ont4vqFznQGc0RerJGk0hnmD+1zge8AVOMyHJC1KwySLbavqv408EknS2Bqm6+ynk7wmya5tqI4d21vVkqRFYpiaxa+B9wFv4Xe9oApwmHJJWiSGSRZvAB5bVT8bdTDSlJnm/Fhz0qFzHIkkGK4ZaqorqyRpkRqmZvFL4PIkF9ANUw70d52VJE2OYZLFV9pHmnVOMSstDMPMZ7Gibx9J0mQb5g3u69nAWFBVZW8oSVokhmmGWjawvC3wEsD3LCRpEentDVVVPx/4rK2qDwD2X5SkRWSYZqh9BlYfQFfTGKZGIkmaEMP80h+c1+IeYA2/G1ZckrQIDNMbakvntZAkLXDDNENtA/wX7j+fxTtHF5YkaZwM0wx1NnAbsJqBN7glSYvHMMlit6o6ZOSRSJLG1jADCX4nyR+NPBJJ0tgapmZxAPDK9ib33XTzaldV/fFII5MkjY1hksVzRx6FJGmsDdN19oa5CESSNL58E3uEHH5b0qQY5gG3JGmRM1lIknqZLCRJvUwWkqReJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1MllIknqZLCRJvUaWLJKckeSWJFcOlO2YZGWSa9vPHVp5knwoyXVJfpBkn4Fjlrf9r02yfFTxSpJmNsqaxSeB6TPsHQ+cX1V7Aee3deiGQd+rfY4BToUuuQAnAvsB+wInTiUYSdLcGVmyqKqLgPXTig8DVrTlFcALB8o/VZ3vAdsn2RU4GFhZVeur6lZgJfdPQJKkEZvrZxa7VNXNbfknwC5teQlw48B+N7WymcrvJ8kxSVYlWbVu3brZjVqSFrl5e8BdVQXULJ7vtKpaVlXLdt5559k6rSSJuU8WP23NS7Sft7TytcDuA/vt1spmKpckzaG5ThbnAFM9mpYDZw+UH9l6Re0P3Naaq84DDkqyQ3uwfVArkyTNoZFNq5rkc8CBwE5JbqLr1XQScFaSo4EbgMPb7ucCzwOuA+4CjgKoqvVJ3gVc0vZ7Z1VNf2guSRqxkSWLqnrZDJues4F9C3jdDOc5AzhjFkOTJG0i3+CWJPUyWUiSepksJEm9TBaSpF4mC0lSL5OFJKmXyUKS1Gtk71ksJkuP/9p8hyBJI2XNQpLUy5qFFpSZanFrTjp0jiORFhdrFpKkXiYLSVIvk4UkqZfJQpLUy2QhSeplspAk9TJZSJJ6+Z6FJoLvX0ijZc1CktTLZCFJ6mWykCT1MllIknqZLCRJvUwWkqReJgtJUi/fs9BE8/0LaXZYs5Ak9TJZSJJ6mSwkSb18ZrEJZmr/1sLjswxp01izkCT1MllIknrZDCUNsHlK2jBrFpKkXgumZpHkEOCDwFbAJ6rqpHkOSYvI5nRusDaiSbIgahZJtgL+B/Bc4InAy5I8cX6jkqTFY6HULPYFrquqHwMkORM4DLh6FF9mF1nNhtn6dzRTDcXnK5pLCyVZLAFuHFi/CdhvcIckxwDHtNU7k/xoE79jJ+Bnmx3hwuK1LiA5eehddwJ+tgn7L2QL/r5ugrm81sfMtGGhJIteVXUacNrmHp9kVVUtm8WQxpbXOpm81sk0Lte6IJ5ZAGuB3QfWd2tlkqQ5sFCSxSXAXkn2SPIg4AjgnHmOSZIWjQXRDFVV9yT5K+A8uq6zZ1TVVbP8NZvdhLUAea2TyWudTGNxramq+Y5BkjTmFkozlCRpHpksJEm9Fn2ySHJIkh8luS7J8fMdz2xKsnuSC5JcneSqJMe18h2TrExybfu5w3zHOluSbJXksiRfbet7JLm43d/Ptw4SC16S7ZN8IckPk1yT5OmTel+T/E3793tlks8l2XaS7muSM5LckuTKgbIN3st0PtSu+wdJ9pmrOBd1slgEw4jcA7yhqp4I7A+8rl3f8cD5VbUXcH5bnxTHAdcMrJ8MnFJVjwVuBY6el6hm3weBr1fV44En013zxN3XJEuAY4FlVfWHdB1cjmCy7usngUOmlc10L58L7NU+xwCnzlGMiztZMDCMSFX9GpgaRmQiVNXNVXVpW76D7hfKErprXNF2WwG8cH4inF1JdgMOBT7R1gM8G/hC22UirjXJI4BnAqcDVNWvq+oXTOh9peu1+eAkWwMPAW5mgu5rVV0ErJ9WPNO9PAz4VHW+B2yfZNe5iHOxJ4sNDSOyZJ5iGakkS4GnABcDu1TVzW3TT4Bd5ims2fYB4M3AfW39kcAvquqetj4p93cPYB3wz63J7RNJHsoE3teqWgv8A/D/6JLEbcBqJvO+DprpXs7b76zFniwWhSQPA74IvL6qbh/cVl3f6QXffzrJ84Fbqmr1fMcyB7YG9gFOraqnAL9kWpPTBN3XHej+mt4DeDTwUO7fZDPRxuVeLvZkMfHDiCR5IF2i+ExVfakV/3Sq6tp+3jJf8c2iZwAvSLKGrjnx2XTt+tu35guYnPt7E3BTVV3c1r9Alzwm8b7+KXB9Va2rqt8AX6K715N4XwfNdC/n7XfWYk8WEz2MSGuzPx24pqreP7DpHGB5W14OnD3Xsc22qjqhqnarqqV09/GbVfVy4ALgxW23SbnWnwA3JnlcK3oO3XD9E3df6Zqf9k/ykPbveepaJ+6+TjPTvTwHOLL1itofuG2guWqkFv0b3EmeR9fWPTWMyHvmOaRZk+QA4F+AK/hdO/7f0T23OAv4A+AG4PCqmv6AbcFKciDwxqp6fpI96WoaOwKXAa+oqrvnM77ZkGRvugf5DwJ+DBxF98ffxN3XJO8AXkrXu+8y4NV07fQTcV+TfA44kG4o8p8CJwJfYQP3siXMj9A1xd0FHFVVq+YkzsWeLCRJ/RZ7M5QkaQgmC0lSL5OFJKmXyUKS1MtkIUnqZbLQgpfkzhGcc+/WrXpq/e1J3rgF53tJGx32gtmJcLPjWJNkp/mMQQuTyULasL2B5/XuNbyjgddU1bNm8ZzSnDFZaKIkeVOSS9pY/+9oZUvbX/Ufb/MifCPJg9u2p7V9L0/yvjZnwoOAdwIvbeUvbad/YpILk/w4ybEzfP/LklzRznNyK3sbcABwepL3Tdt/1yQXte+5MsmftPJTk6xq8b5jYP81Sd7b9l+VZJ8k5yX51yR/2fY5sJ3za+nmavmnJPf7fz3JK5J8v53rY+nmAtkqySdbLFck+ZstvCWaFFXlx8+C/gB3tp8H0U1uH7o/hL5KN5T3Urq3f/du+51F98YvwJXA09vyScCVbfmVwEcGvuPtwHeAbejetP058MBpcTyabniKnekG+/sm8MK27UK6ORmmx/4G4C1teStgu7a840DZhcAft/U1wGvb8inAD4Dt2nf+tJUfCPwK2LMdvxJ48cDxOwFPAP7X1DUAHwWOBJ4KrByIb/v5vr9+xuNjzUKT5KD2uQy4FHg83SQx0A1Gd3lbXg0sTbI93S/n77byz/ac/2tVdXdV/YxuYLfpQ4A/DbiwukHv7gE+Q5esNuYS4Kgkbwf+qLp5RwAOT3Jpu5Yn0U3ONWVq/LIrgIur6o6qWgfc3a4J4PvVzdNyL/A5uprNoOfQJYZLklze1vekGzpkzyQfTnIIcDsS3V8/0qQI8N6q+tjvFXZzeQyOG3Qv8ODNOP/0c2zx/z9VdVGSZ9JN2vTJJO+nG8/rjcDTqurWJJ8Ett1AHPdNi+m+gZimj+MzfT3Aiqo6YXpMSZ4MHAz8JXA48KpNvS5NHmsWmiTnAa9q83eQZEmSR820c3Wzy92RZL9WdMTA5jvomnc2xfeB/5hkp3RT9r4M+NbGDkjyGLrmo4/TDQy4D/BwujkqbkuyC91Umptq3zaa8gPoBuH79rTt5wMvnvrvk27O58e0nlIPqKovAm9t8UjWLDQ5quobSZ4AfLcbnJM7gVfQ1QJmcjTw8ST30f1iv62VXwAc35po3jvk99+c5Ph2bOiarfqGzj4QeFOS37R4j6yq65NcBvyQbla0/zPM909zCd3opI9t8Xx5WqxXJ3kr8I2WUH4DvA74d7oZ+Kb+kLxfzUOLk6POalFL8rCqurMtHw/sWlXHzXNYW2RwiPb5jkWTw5qFFrtDk5xA9//CDXS9oCRNY81CktTLB9ySpF4mC0lSL5OFJKmXyUKS1MtkIUnq9f8BsUlv7VCTRNEAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y1Vj7idf3d6X"
      },
      "source": [
        "src_tokenizer = Tokenizer(oov_token='OOV') # 모든 단어를 사용하지만 인덱스 1에는 단어 'OOV'를 할당한다.\n",
        "src_tokenizer.fit_on_texts(sentences)\n",
        "tar_tokenizer = Tokenizer(lower=False) # 태깅 정보들은 내부적으로 대문자를 유지한채로 저장\n",
        "tar_tokenizer.fit_on_texts(ner_tags)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-KrEtQ3j3fTk",
        "outputId": "fe08766e-270b-45e0-9c9c-767e94dfac52"
      },
      "source": [
        "vocab_size = len(src_tokenizer.word_index) + 1\n",
        "tag_size = len(tar_tokenizer.word_index) + 1\n",
        "print('단어 집합의 크기 : {}'.format(vocab_size))\n",
        "print('개체명 태깅 정보 집합의 크기 : {}'.format(tag_size))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "단어 집합의 크기 : 31819\n",
            "개체명 태깅 정보 집합의 크기 : 18\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9Uir00Gn3nv4",
        "outputId": "f48a7b87-1eba-44f6-9a11-0fd4d82a3afd"
      },
      "source": [
        "print('단어 OOV의 인덱스 : {}'.format(src_tokenizer.word_index['OOV']))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "단어 OOV의 인덱스 : 1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CbThZwgm3pIy"
      },
      "source": [
        "X_data = src_tokenizer.texts_to_sequences(sentences)\n",
        "y_data = tar_tokenizer.texts_to_sequences(ner_tags)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8KW1_b7v3qW-",
        "outputId": "3ef71250-6658-494a-f585-3d26d72320b4"
      },
      "source": [
        "print(X_data[0])\n",
        "print(y_data[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[254, 6, 967, 16, 1795, 238, 468, 7, 523, 2, 129, 5, 61, 9, 571, 2, 833, 6, 186, 90, 22, 15, 56, 3]\n",
            "[1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 8, 1, 1, 1, 1, 1]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FsKckwN-3rh3"
      },
      "source": [
        "word_to_index = src_tokenizer.word_index\n",
        "index_to_word = src_tokenizer.index_word\n",
        "ner_to_index = tar_tokenizer.word_index\n",
        "index_to_ner = tar_tokenizer.index_word\n",
        "index_to_ner[0] = 'PAD'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IFmQxwzW3uMO",
        "outputId": "af07b3ab-b76e-42b5-f9e2-ff42c3c2b774"
      },
      "source": [
        "print(index_to_ner)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{1: 'O', 2: 'B-geo', 3: 'B-tim', 4: 'B-org', 5: 'I-per', 6: 'B-per', 7: 'I-org', 8: 'B-gpe', 9: 'I-geo', 10: 'I-tim', 11: 'B-art', 12: 'B-eve', 13: 'I-art', 14: 'I-eve', 15: 'B-nat', 16: 'I-gpe', 17: 'I-nat', 0: 'PAD'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n4ySMUSt3vY5",
        "outputId": "7961c015-c960-4c03-d0d9-bba1df0ace79"
      },
      "source": [
        "decoded = []\n",
        "for index in X_data[0] : # 첫번째 샘플 안의 인덱스들에 대해서\n",
        "    decoded.append(index_to_word[index]) # 다시 단어로 변환\n",
        "\n",
        "print('기존의 문장 : {}'.format(sentences[0]))\n",
        "print('디코딩 문장 : {}'.format(decoded))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "기존의 문장 : ['thousands', 'of', 'demonstrators', 'have', 'marched', 'through', 'london', 'to', 'protest', 'the', 'war', 'in', 'iraq', 'and', 'demand', 'the', 'withdrawal', 'of', 'british', 'troops', 'from', 'that', 'country', '.']\n",
            "디코딩 문장 : ['thousands', 'of', 'demonstrators', 'have', 'marched', 'through', 'london', 'to', 'protest', 'the', 'war', 'in', 'iraq', 'and', 'demand', 'the', 'withdrawal', 'of', 'british', 'troops', 'from', 'that', 'country', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9mZDlPqo3wlX"
      },
      "source": [
        "max_len = 70\n",
        "X_data = pad_sequences(X_data, padding='post', maxlen=max_len)\n",
        "y_data = pad_sequences(y_data, padding='post', maxlen=max_len)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SyB_WRor7S0Q",
        "outputId": "ebbfffa6-9f40-4278-e4c1-2f736db605ca"
      },
      "source": [
        "print(X_data.shape)\n",
        "print(y_data.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(47959, 70)\n",
            "(47959, 70)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gJtU5Oz03zNW"
      },
      "source": [
        "X_train, X_test, y_train_int, y_test_int = train_test_split(X_data, y_data, test_size=.2, random_state=777)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9FfY0d5f30Qv"
      },
      "source": [
        "y_train = to_categorical(y_train_int, num_classes=tag_size)\n",
        "y_test = to_categorical(y_test_int, num_classes=tag_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print('훈련 샘플 문장의 크기 : {}'.format(X_train.shape))\n",
        "print('훈련 샘플 레이블(정수 인코딩)의 크기 : {}'.format(y_train_int.shape))\n",
        "print('훈련 샘플 레이블(원-핫 인코딩)의 크기 : {}'.format(y_train.shape))\n",
        "print('테스트 샘플 문장의 크기 : {}'.format(X_test.shape))\n",
        "print('테스트 샘플 레이블(정수 인코딩)의 크기 : {}'.format(y_test_int.shape))\n",
        "print('테스트 샘플 레이블(원-핫 인코딩)의 크기 : {}'.format(y_test.shape))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Npq4bXUA7V8k",
        "outputId": "0afb71f9-6b3a-4e64-fd9e-472616251ac9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "훈련 샘플 문장의 크기 : (38367, 70)\n",
            "훈련 샘플 레이블(정수 인코딩)의 크기 : (38367, 70)\n",
            "훈련 샘플 레이블(원-핫 인코딩)의 크기 : (38367, 70, 18)\n",
            "테스트 샘플 문장의 크기 : (9592, 70)\n",
            "테스트 샘플 레이블(정수 인코딩)의 크기 : (9592, 70)\n",
            "테스트 샘플 레이블(원-핫 인코딩)의 크기 : (9592, 70, 18)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9maF6Hg7ngI"
      },
      "source": [
        "# char 정보를 사용하기 위한 추가 전처리"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UAmtu5LG4m-2",
        "outputId": "fded19de-1451-46d9-bfd8-ab49b9433e26"
      },
      "source": [
        "# char_vocab 만들기\n",
        "words = list(set(data[\"Word\"].values))\n",
        "chars = set([w_i for w in words for w_i in w])\n",
        "chars = sorted(list(chars))\n",
        "print(chars)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['!', '\"', '#', '$', '%', '&', \"'\", '(', ')', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '?', '@', '[', ']', '_', '`', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '~', '\\x85', '\\x91', '\\x92', '\\x93', '\\x94', '\\x96', '\\x97', '\\xa0', '°', 'é', 'ë', 'ö', 'ü']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tjahDonE5Zyn"
      },
      "source": [
        "char_to_index = {c: i + 2 for i, c in enumerate(chars)}\n",
        "char_to_index[\"OOV\"] = 1\n",
        "char_to_index[\"PAD\"] = 0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L3_Mf9HlB_p4"
      },
      "source": [
        "index_to_char = {}\n",
        "for key, value in char_to_index.items():\n",
        "    index_to_char[value] = key"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UbQYSZAl-uWQ",
        "outputId": "3a28c5fc-02f3-4147-fed6-ad7d8b5a6b20"
      },
      "source": [
        "print(sentences[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['thousands', 'of', 'demonstrators', 'have', 'marched', 'through', 'london', 'to', 'protest', 'the', 'war', 'in', 'iraq', 'and', 'demand', 'the', 'withdrawal', 'of', 'british', 'troops', 'from', 'that', 'country', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rNZZqDN99_lU"
      },
      "source": [
        "max_len_char = 15\n",
        "\n",
        "def padding_char_indice(char_indice, max_len_char):\n",
        "  return pad_sequences(\n",
        "        char_indice, maxlen=max_len_char, padding='post', value = 0)\n",
        "\n",
        "def integer_coding(sentences):\n",
        "  char_data = []\n",
        "  for ts in sentences:\n",
        "    word_indice = [word_to_index[t] for t in ts]\n",
        "    char_indice = [[char_to_index[char] for char in t]  \n",
        "                                          for t in ts]\n",
        "    char_indice = padding_char_indice(char_indice, max_len_char)\n",
        "\n",
        "    for chars_of_token in char_indice:\n",
        "      if len(chars_of_token) > max_len_char:\n",
        "        continue\n",
        "    char_data.append(char_indice)\n",
        "  return char_data"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KE7aX8UFC7MT"
      },
      "source": [
        "X_char_data = integer_coding(sentences)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 정수 인코딩 이전의 기존 문장\n",
        "print(sentences[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fvbZZ4pV7jkl",
        "outputId": "391f3308-a184-475f-ca14-1d41fdf5a812"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['thousands', 'of', 'demonstrators', 'have', 'marched', 'through', 'london', 'to', 'protest', 'the', 'war', 'in', 'iraq', 'and', 'demand', 'the', 'withdrawal', 'of', 'british', 'troops', 'from', 'that', 'country', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "424UcyumBi2Y",
        "outputId": "3df91d58-1b7c-4ec5-b270-75aae206449f"
      },
      "source": [
        "print(X_data[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[ 254    6  967   16 1795  238  468    7  523    2  129    5   61    9\n",
            "  571    2  833    6  186   90   22   15   56    3    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HuSVDOHhD8Ec",
        "outputId": "cdc10751-ee00-4dc4-9109-c263318c5a5d"
      },
      "source": [
        "print(X_char_data[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[53 41 48 54 52 34 47 37 52  0  0  0  0  0  0]\n",
            " [48 39  0  0  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [37 38 46 48 47 52 53 51 34 53 48 51 52  0  0]\n",
            " [41 34 55 38  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [46 34 51 36 41 38 37  0  0  0  0  0  0  0  0]\n",
            " [53 41 51 48 54 40 41  0  0  0  0  0  0  0  0]\n",
            " [45 48 47 37 48 47  0  0  0  0  0  0  0  0  0]\n",
            " [53 48  0  0  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [49 51 48 53 38 52 53  0  0  0  0  0  0  0  0]\n",
            " [53 41 38  0  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [56 34 51  0  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [42 47  0  0  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [42 51 34 50  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [34 47 37  0  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [37 38 46 34 47 37  0  0  0  0  0  0  0  0  0]\n",
            " [53 41 38  0  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [56 42 53 41 37 51 34 56 34 45  0  0  0  0  0]\n",
            " [48 39  0  0  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [35 51 42 53 42 52 41  0  0  0  0  0  0  0  0]\n",
            " [53 51 48 48 49 52  0  0  0  0  0  0  0  0  0]\n",
            " [39 51 48 46  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [53 41 34 53  0  0  0  0  0  0  0  0  0  0  0]\n",
            " [36 48 54 47 53 51 58  0  0  0  0  0  0  0  0]\n",
            " [14  0  0  0  0  0  0  0  0  0  0  0  0  0  0]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZUuaNklnEHzN"
      },
      "source": [
        "X_char_data = pad_sequences(X_char_data, maxlen=max_len, padding='post', value = 0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dC9agARj6uCa"
      },
      "source": [
        "X_char_train, X_char_test, _, _ = train_test_split(X_char_data, y_data, test_size=.2, random_state=777)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_ukEbj2tBc5r"
      },
      "source": [
        "X_char_train = np.array(X_char_train)\n",
        "X_char_test = np.array(X_char_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Odl26mVdEfPm",
        "outputId": "fbb72c9f-a8a3-4fc3-9055-50088b01a4be"
      },
      "source": [
        "print(X_train[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[ 150  928  361   17 2624    9 4131 3567    9    8 2893 1250  880  107\n",
            "    3    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "    0    0    0    0    0    0    0    0    0    0    0    0    0    0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(index_to_word[150])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_5zCFvf58Q1b",
        "outputId": "e103740a-6aad-4326-e1db-bb15e39ee195"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "soldiers\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NIGaJI-_RCA7",
        "outputId": "aa88bf51-2b1b-47e1-a9df-9ccdf321aea8"
      },
      "source": [
        "print(X_char_train[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[52 48 45 ...  0  0  0]\n",
            " [51 38 53 ...  0  0  0]\n",
            " [39 42 51 ...  0  0  0]\n",
            " ...\n",
            " [ 0  0  0 ...  0  0  0]\n",
            " [ 0  0  0 ...  0  0  0]\n",
            " [ 0  0  0 ...  0  0  0]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EdeauTKoRIqT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f2acf438-9606-43a7-9b59-7dfde7896a83"
      },
      "source": [
        "print(X_char_train[0][0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[52 48 45 37 42 38 51 52  0  0  0  0  0  0  0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2nvdylKuE0_Z",
        "outputId": "e7457b33-ff98-4c42-d972-3bafc98963d1"
      },
      "source": [
        "print(' '.join([index_to_char[index] for index in X_char_train[0][0]]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "s o l d i e r s PAD PAD PAD PAD PAD PAD PAD\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QgBR0y8mFWmn",
        "outputId": "e7294bc7-0522-4829-c74b-d6856ede6d31"
      },
      "source": [
        "print('훈련 샘플 문장의 크기 : {}'.format(X_train.shape))\n",
        "print('훈련 샘플 레이블의 크기 : {}'.format(y_train.shape))\n",
        "print('훈련 샘플 char 데이터의 크기 : {}'.format(X_char_train.shape))\n",
        "print('테스트 샘플 문장의 크기 : {}'.format(X_test.shape))\n",
        "print('테스트 샘플 레이블의 크기 : {}'.format(y_test.shape))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "훈련 샘플 문장의 크기 : (38367, 70)\n",
            "훈련 샘플 레이블의 크기 : (38367, 70, 18)\n",
            "훈련 샘플 char 데이터의 크기 : (38367, 70, 15)\n",
            "테스트 샘플 문장의 크기 : (9592, 70)\n",
            "테스트 샘플 레이블의 크기 : (9592, 70, 18)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M5HlLxZEV7kn"
      },
      "source": [
        "# BiLSTM을 이용한 개체명 인식"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8P0QOs4KOayO"
      },
      "source": [
        "## 모델링"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cwvOg6XiWDCO"
      },
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense, LSTM, InputLayer, Bidirectional, TimeDistributed, Embedding\n",
        "from tensorflow.keras.optimizers import Adam"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pKE60YszWD-e"
      },
      "source": [
        "embedding_dim = 128\n",
        "hidden_units = 256\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Embedding(vocab_size, embedding_dim, mask_zero=True))\n",
        "model.add(Bidirectional(LSTM(hidden_units, return_sequences=True)))\n",
        "model.add(TimeDistributed(Dense(tag_size, activation=('softmax'))))\n",
        "model.compile(loss='categorical_crossentropy', optimizer=Adam(0.001), metrics=['accuracy'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tQJb-y9ab0pg",
        "outputId": "aed0db35-3e34-43f6-cfb3-3323a207cd24"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding (Embedding)       (None, None, 128)         4072832   \n",
            "                                                                 \n",
            " bidirectional (Bidirectiona  (None, None, 512)        788480    \n",
            " l)                                                              \n",
            "                                                                 \n",
            " time_distributed (TimeDistr  (None, None, 18)         9234      \n",
            " ibuted)                                                         \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 4,870,546\n",
            "Trainable params: 4,870,546\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WRAKp9mSWF5J",
        "outputId": "25a0ad6e-8988-427f-fd0d-f02928266a48"
      },
      "source": [
        "history = model.fit(X_train, y_train, batch_size=128, epochs=6, validation_split=0.1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/6\n",
            "270/270 [==============================] - 20s 38ms/step - loss: 0.1768 - accuracy: 0.8739 - val_loss: 0.0820 - val_accuracy: 0.9291\n",
            "Epoch 2/6\n",
            "270/270 [==============================] - 7s 24ms/step - loss: 0.0558 - accuracy: 0.9491 - val_loss: 0.0489 - val_accuracy: 0.9534\n",
            "Epoch 3/6\n",
            "270/270 [==============================] - 6s 21ms/step - loss: 0.0375 - accuracy: 0.9642 - val_loss: 0.0443 - val_accuracy: 0.9572\n",
            "Epoch 4/6\n",
            "270/270 [==============================] - 6s 21ms/step - loss: 0.0304 - accuracy: 0.9699 - val_loss: 0.0427 - val_accuracy: 0.9585\n",
            "Epoch 5/6\n",
            "270/270 [==============================] - 6s 21ms/step - loss: 0.0260 - accuracy: 0.9737 - val_loss: 0.0434 - val_accuracy: 0.9587\n",
            "Epoch 6/6\n",
            "270/270 [==============================] - 6s 21ms/step - loss: 0.0228 - accuracy: 0.9765 - val_loss: 0.0444 - val_accuracy: 0.9587\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5qE4zaXuWI_M"
      },
      "source": [
        "model.save('bilstm.h5')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lEyHvNZ4WXE9",
        "outputId": "24a9c94c-ac86-4247-ea9c-0f198e5d14d1"
      },
      "source": [
        "i = 13 # 확인하고 싶은 테스트용 샘플의 인덱스.\n",
        "y_predicted = model.predict(np.array([X_test[i]])) # 입력한 테스트용 샘플에 대해서 예측 y를 리턴\n",
        "y_predicted = np.argmax(y_predicted, axis=-1) # 확률 벡터를 정수 인코딩으로 변경함.\n",
        "true = np.argmax(y_test[i], -1) # 원-핫 인코딩을 다시 정수 인코딩으로 변경함.\n",
        "\n",
        "print(\"{:15}|{:5}|{}\".format(\"단어\", \"실제값\", \"예측값\"))\n",
        "print(35 * \"-\")\n",
        "\n",
        "for word, tag, pred in zip(X_test[i], true, y_predicted[0]):\n",
        "    if word != 0: # PAD값은 제외함.\n",
        "        print(\"{:17}: {:7} {}\".format(index_to_word[word], index_to_ner[tag], index_to_ner[pred]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "단어             |실제값  |예측값\n",
            "-----------------------------------\n",
            "the              : O       O\n",
            "statement        : O       O\n",
            "came             : O       O\n",
            "as               : O       O\n",
            "u.n.             : B-org   B-org\n",
            "secretary-general: I-org   I-org\n",
            "kofi             : B-per   B-per\n",
            "annan            : I-per   I-per\n",
            "met              : O       O\n",
            "with             : O       O\n",
            "officials        : O       O\n",
            "in               : O       O\n",
            "amman            : B-geo   B-geo\n",
            "to               : O       O\n",
            "discuss          : O       O\n",
            "wednesday        : B-tim   B-tim\n",
            "'s               : O       O\n",
            "attacks          : O       O\n",
            ".                : O       O\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "ubLZsdeAWcM5",
        "outputId": "9c6ce50f-1461-453c-9864-89359ec62e85"
      },
      "source": [
        "epochs = range(1, len(history.history['val_loss']) + 1)\n",
        "plt.plot(epochs, history.history['loss'])\n",
        "plt.plot(epochs, history.history['val_loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'val'], loc='upper left')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxddZ3/8dcnNzdJs3VJ0y1JF2RtC7Q0qSCy/ECZAtIipU0VFR1HdIRRxmXEmXF0GP39cHRGxxFHKjgDLpTSAhYFK7IOytK0UKC0QIGWJt2SdMvS7J/fH+e0pMltm7T35GZ5Px+P+8i9Z7uf2+W+c77fc75fc3dERES6Skt1ASIi0j8pIEREJCEFhIiIJKSAEBGRhBQQIiKSkAJCREQSUkCIJIGZ/Y+ZfbuH224ysw8c73FEoqaAEBGRhBQQIiKSkAJChoywaeerZvaSmTWY2R1mNtbMHjazOjP7o5mN7LT9XDNbZ2Z7zOwJMzut07qZZrYm3O8eIKvLe33IzF4M9/2zmZ1xjDV/xsw2mtkuM1thZhPC5WZmPzCznWa2z8xeNrPp4brLzOzVsLYqM/vKMf2ByZCngJChZj7wQeBk4ArgYeDvgUKC/w9fADCzk4G7gRvDdQ8BD5pZhpllAA8AvwBGAfeGxyXcdybwc+CzQAFwG7DCzDJ7U6iZXQT8P2AhMB7YDCwJV18CnB9+juHhNrXhujuAz7p7HjAdeKw37ytygAJChpr/dPcd7l4F/C/wnLu/4O5NwP3AzHC7cuB37v6Iu7cC3weGAe8DzgbiwA/dvdXdlwGrOr3HdcBt7v6cu7e7+51Ac7hfb1wD/Nzd17h7M/B14Bwzmwy0AnnAqYC5+3p33xbu1wpMNbN8d9/t7mt6+b4igAJChp4dnZ7vT/A6N3w+geA3dgDcvQPYAhSF66r80JEuN3d6Pgn4cti8tMfM9gAl4X690bWGeoKzhCJ3fwz4MXArsNPMFptZfrjpfOAyYLOZPWlm5/TyfUUABYTI4Wwl+KIHgjZ/gi/5KmAbUBQuO2Bip+dbgO+4+4hOj2x3v/s4a8ghaLKqAnD3H7n7LGAqQVPTV8Plq9x9HjCGoClsaS/fVwRQQIgczlLgcjO72MziwJcJmon+DDwDtAFfMLO4mV0FzO6078+Az5nZe8PO5Bwzu9zM8npZw93Ap8xsRth/8X8JmsQ2mVlZePw40AA0AR1hH8k1ZjY8bBrbB3Qcx5+DDGEKCJEE3P014GPAfwI1BB3aV7h7i7u3AFcBnwR2EfRX3Ndp3wrgMwRNQLuBjeG2va3hj8A3gOUEZy3vARaFq/MJgmg3QTNULfC9cN3HgU1mtg/4HEFfhkivmSYMEhGRRHQGISIiCSkgREQkIQWEiIgkpIAQEZGE0lNdQLKMHj3aJ0+enOoyREQGlNWrV9e4e2GidYMmICZPnkxFRUWqyxARGVDMbPPh1qmJSUREEoo0IMxsjpm9Fg5XfFOC9eeHQya3mdnVXdb9azjU8noz+1GXYQ1ERCRikQWEmcUIBhK7lGCsmI+Y2dQum71DcIfpr7vs+z7gXOAMguGKy4ALoqpVRES6i7IPYjaw0d3fAjCzJcA84NUDG7j7pnBd17FinGAClgzACIZW3kEvtba2UllZSVNT07HUP6BkZWVRXFxMPB5PdSkiMkhEGRBFBKNaHlAJvLcnO7r7M2b2OMH4Mwb82N3Xd93OzK4jGHufiRMndl1NZWUleXl5TJ48mcHcQuXu1NbWUllZyZQpU1JdjogMEv2yk9rMTgROA4oJguYiMzuv63buvtjdS929tLCw+1VaTU1NFBQUDOpwADAzCgoKhsSZkoj0nSgDoopg/PwDisNlPfFh4Fl3rw8nSXkYOKZJTwZ7OBwwVD6niPSdKANiFXCSmU0J5/BdBKzo4b7vABeYWXo43v0FQLcmpmRoa+9gx74mmlrbozi8iMiAFVlAuHsbcAOwkuDLfam7rzOzm81sLkA46UklsAC4zczWhbsvA94EXgbWAmvd/cGoat1Z10xtQ0skx96zZw8/+clPer3fZZddxp49eyKoSESkZyK9k9rdHwIe6rLsnzo9X0XQ9NR1v3bgs1HWdkB6LI3hWXH2NLYwPj+LtLTkNtUcCIjPf/7zhyxva2sjPf3wf/wPPfTQYdeJiPSFftlJ3ddG5cRp73D2NbUm/dg33XQTb775JjNmzKCsrIzzzjuPuXPnMnVqcEvIlVdeyaxZs5g2bRqLFy8+uN/kyZOpqalh06ZNnHbaaXzmM59h2rRpXHLJJezfvz/pdYqIdDVoxmI6mn9+cB2vbt132PWNLe2kGWTFYz0+5tQJ+XzzimlH3OaWW27hlVde4cUXX+SJJ57g8ssv55VXXjl4OerPf/5zRo0axf79+ykrK2P+/PkUFBQccow33niDu+++m5/97GcsXLiQ5cuX87GPfazHdYqIHAudQYTiMaO9w4l6CtbZs2cfcq/Cj370I84880zOPvtstmzZwhtvvNFtnylTpjBjxgwAZs2axaZNmyKtUUQEhtAZxNF+029p62DD9n2Mycti3PCsyOrIyck5+PyJJ57gj3/8I8888wzZ2dlceOGFCe9lyMzMPPg8FoupiUlE+oTOIEIZ6WnkZcXZ3diS1LOIvLw86urqEq7bu3cvI0eOJDs7mw0bNvDss88m7X1FRI7XkDmD6IlR2XE272qlrqmN/GHJGdOooKCAc889l+nTpzNs2DDGjh17cN2cOXP46U9/ymmnncYpp5zC2WefnZT3FBFJBou6zb2vlJaWetcJg9avX89pp53W42N0uLNhWx05mTEmFeQcfYd+prefV0TEzFa7e2midWpi6iTNjJHZcfbtb6O1vesAsyIiQ4sCoouRORk4zp7GaO6sFhEZKBQQXWTFY2RnpLOroTXyS15FRPozBUQCo3LiNLe109iiAfxEZOhSQCQwfFgGaWbsjmgAPxGRgUABkUAszRgxLM6e/a20d6iZSUSGJgXEYYzMyaDDnb37+/YsIjc3t0/fT0TkcBQQh5GdESMrPcauhuSP8CoiMhDoTurDMDNG5mSwbe9+mlrbezXKa2c33XQTJSUlXH/99QB861vfIj09nccff5zdu3fT2trKt7/9bebNm5fM8kVEjlukAWFmc4D/AGLA7e5+S5f15wM/BM4AFrn7sk7rJgK3E8xr7cBl7r7pmIt5+CbY/nKvdhmNM6ylHUszSE8QEONOh0tv6b68k/Lycm688caDAbF06VJWrlzJF77wBfLz86mpqeHss89m7ty5mldaRPqVyALCzGLArcAHgUpglZmtcPdXO232DvBJ4CsJDnEX8B13f8TMcoE+v7XZMNLTjLYOJwPH6P0X+MyZM9m5cydbt26lurqakSNHMm7cOP72b/+Wp556irS0NKqqqtixYwfjxo2L4FOIiBybKM8gZgMb3f0tADNbAswDDgbEgTMCMzvky9/MpgLp7v5IuF39cVdzlN/0D6e1qZW3axqYNCqb4dkZx3SMBQsWsGzZMrZv3055eTm/+tWvqK6uZvXq1cTjcSZPnpxwmG8RkVSKspO6CNjS6XVluKwnTgb2mNl9ZvaCmX0vPCM5hJldZ2YVZlZRXV2dhJK7y81MJx5LY1fjsXdWl5eXs2TJEpYtW8aCBQvYu3cvY8aMIR6P8/jjj7N58+YkViwikhz99SqmdOA8gqanMuAEgqaoQ7j7YncvdffSwsLCSAoxM0ZmZ1DX1EpL27G1ck2bNo26ujqKiooYP34811xzDRUVFZx++uncddddnHrqqUmuWkTk+EXZxFRF0MF8QHG4rCcqgRc7NU89AJwN3JHUCntoVE6cnXVN7G5sYWz+sc029/LL73aQjx49mmeeeSbhdvX1x9+aJiKSDFGeQawCTjKzKWaWASwCVvRi3xFmduC04CI69V30tYz0GLmZ6exuSO5scyIi/VlkAeHubcANwEpgPbDU3deZ2c1mNhfAzMrMrBJYANxmZuvCfdsJmpceNbOXAQN+FlWtPTEqJ4OW9g7qm9tSWYaISJ+J9D4Id38IeKjLsn/q9HwVQdNTon0fIbg/4nhrSMr9BflZcWJpwQB+eVnJmY40mXRmIyLJ1l87qZMiKyuL2trapHx5pqUFndV7m9po62ezzbk7tbW1ZGUdW/+IiEgig3qojeLiYiorK0nWJbCt7R3s2NdM0844uVn9648uKyuL4uKEJ2MiIsekf33LJVk8HmfKlClJPebcHz9NS1sHD3/xPA2NISKD2qBuYopCeVkJG7bX8VLl3lSXIiISKQVEL11x5gSy4mksWbXl6BuLiAxgCoheys+Kc9np43lw7VYaW3TJq4gMXgqIY1BeWkJ9cxsPvbw91aWIiERGAXEMZk8ZxZTROSxVM5OIDGIKiGNgZiwsLeH5Tbt4s1pjJ4nI4KSAOEbzZxURSzOWVugsQkQGJwXEMRqTl8VFp45h+eoqWvvZndUiIsmggDgO5aUl1NQ389iGnakuRUQk6RQQx+HCUwoZk5epzmoRGZQUEMchPZbG/FnFPP7aTrbv1ZzSIjK4KCCO08LSEjoclq+pTHUpIiJJpYA4TlNG5/DeKaNYWrGFjg7NySAig4cCIgnKy0rYXNvIc2/vSnUpIiJJE2lAmNkcM3vNzDaa2U0J1p9vZmvMrM3Mrk6wPt/MKs3sx1HWebwunT6evKx03RMhIoNKZAFhZjHgVuBSYCrwETOb2mWzd4BPAr8+zGH+BXgqqhqTZVhGjHkzJvDQy9vYu7811eWIiCRFlGcQs4GN7v6Wu7cAS4B5nTdw903u/hLQ7U4zM5sFjAX+EGGNSVNeOpHmtg5WvFiV6lJERJIiyoAoAjq3uVSGy47KzNKAfwO+cpTtrjOzCjOrSNa0osdqelE+p43P5x41M4nIINFfO6k/Dzzk7ke8dtTdF7t7qbuXFhYW9lFpiZkZi8pKeKVqH69UabY5ERn4ogyIKqCk0+vicFlPnAPcYGabgO8DnzCzW5JbXvJdOaOIjPQ0dVaLyKAQZUCsAk4ysylmlgEsAlb0ZEd3v8bdJ7r7ZIJmprvcvdtVUP3N8Ow4c6aN44EXqmhqbU91OSIixyWygHD3NuAGYCWwHljq7uvM7GYzmwtgZmVmVgksAG4zs3VR1dNXFpWVsK+pjZXrNNuciAxs5j447v4tLS31ioqKVJdBR4dzwfcfp3hENndfd3aqyxEROSIzW+3upYnW9ddO6gErLc1YOKuEZ96qZXNtQ6rLERE5ZgqICFxdWkyawb0VGsBPRAYuBUQExg8fxgUnF3Lv6i20abY5ERmgFBARKS8rYce+Zp56I7U38ImIHCsFREQuOnUsBTkZ3KPZ5kRkgFJARCQjPZht7tH1O6mua051OSIivaaAiNDC0hLaOpz7NNuciAxACogInTgml1mTRnJPxRYGy/0mIjJ0KCAiVl5WwlvVDazevDvVpYiI9IoCImKXnz6enIwYS9RZLSIDjAIiYjmZ6Vxx5gR+99I26po025yIDBwKiD6wsKyE/a3t/PalbakuRUSkxxQQfWBmyQhOHpureyJEZEBRQPQBM2NhaQkvbtnDa9vrUl2OiEiPKCD6yFVnFROPmc4iRGTAUED0kVE5GVwydRz3v1BJc5tmmxOR/k8B0YcWlpWwu7GVR17dkepSRESOKtKAMLM5ZvaamW00s25zSpvZ+Wa2xszazOzqTstnmNkzZrbOzF4ys/Io6+wr7z9xNBOGZ6mZSUQGhMgCwsxiwK3ApcBU4CNmNrXLZu8AnwR+3WV5I/AJd58GzAF+aGYjoqq1r8TSjKtLS3h6Yw2VuxtTXY6IyBFFeQYxG9jo7m+5ewuwBJjXeQN33+TuLwEdXZa/7u5vhM+3AjuBwghr7TMLZhUDsGy1BvATkf4tyoAoAjq3pVSGy3rFzGYDGcCbCdZdZ2YVZlZRXT0wJuYpGZXN+08czb0VlbR3aAA/Eem/+nUntZmNB34BfMrdu83d6e6L3b3U3UsLCwfOCcbC0hKq9uznTxtrUl2KiMhhRRkQVUBJp9fF4bIeMbN84HfAP7j7s0muLaUumTaWEdlx7qlQZ7WI9F9RBsQq4CQzm2JmGcAiYEVPdgy3vx+4y92XRVhjSmSmx/jwzCL+sG47uxpaUl2OiEhCkQWEu7cBNwArgfXAUndfZ2Y3m9lcADMrM7NKYAFwm5mtC3dfCJwPfNLMXgwfM6KqNRXKy0pobXfuf6HHJ1UiIn3KBstMZ6WlpV5RUZHqMnpl3o+fpqm1g9/feB5mlupyRGQIMrPV7l6aaF2/7qQe7MrLJvLajjrWVu5NdSkiIt0oIFLoijPHMywe455V76S6FBGRbhQQKZSXFeey08fz4NptNLa0pbocEZFDKCBSbNHsEuqb2/idZpsTkX5GAZFipZNGckJhjgbwE5F+RwGRYgdmm6vYvJuNO+tTXY6IyEEKiH7gqrOKiKUZ9+rOahHpRxQQ/cCYvCwuPnUMy9dU0trebcgpEZGUUED0E+VlJdTUt/Do+p2pLkVEBFBA9BsXnFzImLxMlqqZSUT6CQVEP5EeS2NBaTFPvLaT7XubUl2OiIgCoj9ZWFpCh8Oy1TqLEJHUU0D0I5MKcjj7hFEsraikQ7PNiUiKKSD6mfKyEt7Z1cizb9emuhQRGeJ6FBBm9kUzy7fAHWa2xswuibq4oejS6ePJy0rXndUiknI9PYP4S3ffB1wCjAQ+DtwSWVVDWFY8xpUzinj4le3sbWxNdTkiMoT1NCAOzGZzGfALd1/XadnhdzKbY2avmdlGM7spwfrzw7ORNjO7usu6a83sjfBxbQ/rHBTKy0poaevgN2s125yIpE5PA2K1mf2BICBWmlkecMRbfs0sBtwKXApMBT5iZlO7bPYO8Eng1132HQV8E3gvMBv4ppmN7GGtA970ouFMm5CvZiYRSameBsSngZuAMndvBOLAp46yz2xgo7u/5e4twBJgXucN3H2Tu79E97D5C+ARd9/l7ruBR4A5Pax1UCgvK2Hd1n28UqXZ5kQkNXoaEOcAr7n7HjP7GPCPwNG+uYqAzr8CV4bLeqJH+5rZdWZWYWYV1dXVPTz0wDDvzCIy0tN0FiEiKdPTgPgvoNHMzgS+DLwJ3BVZVT3k7ovdvdTdSwsLC1NdTlINz45z6fRxPPBiFU2t7akuR0SGoJ4GRJu7O0ET0Y/d/VYg7yj7VAElnV4Xh8t64nj2HTTKy0qoa2rj4Vc025yI9L2eBkSdmX2d4PLW35lZGkE/xJGsAk4ysylmlgEsAlb08P1WApeY2ciwc/qScNmQcvaUAiaOylYzk4ikRE8DohxoJrgfYjvBb/TfO9IO7t4G3EDwxb4eWOru68zsZjObC2BmZWZWCSwAbjOzdeG+u4B/IQiZVcDN4bIhJS3NWFhazLNv7WJzbUOqyxGRIcaClqMebGg2FigLXz7v7v1q4oLS0lKvqKhIdRlJt31vE++75VH++sL38NW/ODXV5YjIIGNmq929NNG6ng61sRB4nuA3/YXAc11vbJNojBuexYWnjOHeikraNNuciPShnjYx/QPBPRDXuvsnCO5x+EZ0ZUlnC0tL2FnXzJOvD65LeUWkf+tpQKR1aVKq7cW+cpwuPm0Mo3Mz1FktIn2qp1/yvzezlWb2STP7JPA74KHoypLO4rE05p9VzKMbdrKzTrPNiUjf6FFAuPtXgcXAGeFjsbt/LcrC5FALSkto73DuWzPkbgcRkRRJ7+mG7r4cWB5hLXIEJ47JpXTSSJau2sJnzz8Bs6MOpisiclyOeAZhZnVmti/Bo87M9vVVkRIoLyvhrZoGKjbvTnUpIjIEHDEg3D3P3fMTPPLcPb+vipTA5WeMJzcznSXPq7NaRKKnK5EGkOyMdK44czwPvbyNuibNNici0VJADDALS0vY39rOg2s1gJ+IREsBMcDMKBnBKWPzuGfVO6kuRUQGOQXEAGNmLCwrYW3lXjZs13UCIhIdBcQA9OGZRcRjpjurRSRSCogBaFROBpdMG8f9L1TR3KbZ5kQkGgqIAaq8tIQ9ja38Yd2OVJciIoOUAmKAev+JoykaMYylFWpmEpFoKCAGqLQ04+pZxTy9sYYtuxpTXY6IDEKRBoSZzTGz18xso5ndlGB9ppndE65/zswmh8vjZnanmb1sZuvD+bCliwWlxQDcu7oyxZWIyGAUWUCYWQy4FbgUmAp8xMymdtns08Budz8R+AHw3XD5AiDT3U8HZgGfPRAe8q7ikdm8/8TRLKvYQntHz6aOFRHpqSjPIGYDG939LXdvAZYA87psMw+4M3y+DLjYgmFKHcgxs3RgGNAC6KL/BMrLSti6t4mnN9akuhQRGWSiDIgioHMPamW4LOE27t4G7AUKCMKiAdgGvAN83913dX0DM7vOzCrMrKK6+hin4+zogN9+CXasO7b9U+yDU8cyMjuuO6tFJOn6ayf1bKAdmABMAb5sZid03cjdF7t7qbuXFhYWHts77X4bXv0N3HY+PPYdaGs+nrr7XGZ6jA/PLOaRV3dQWz+waheR/i3KgKgCSjq9Lg6XJdwmbE4aTjDf9UeB37t7azgX9p+A0kiqLHgP3LAKpl8NT/0r/PQ82PJ8JG8VlfKyElrbnftf0GxzIpI8UQbEKuAkM5tiZhnAImBFl21WANeGz68GHnN3J2hWugjAzHKAs4ENkVWaPQquug2uWQ6tjXDHJfDwTdBcH9lbJtMp4/I4s2QESyu2EPzxiYgcv8gCIuxTuAFYCawHlrr7OjO72czmhpvdARSY2UbgS8CBS2FvBXLNbB1B0Py3u78UVa0HnfQB+PwzMPsz8NxP4b/OgTcfi/xtk2FRWQmv76jnhS17Ul2KiAwSNlh+4ywtLfWKiorkHfCdZ+E3N0DtGzDjGrjk28GZRj9V19TK7O88yrwZE7hl/hmpLkdEBggzW+3uCZvw+2sndepNPBs+9zSc9xVYuwRufW/Qmd1P5WXFufyM8Ty4disNzW2pLkdEBgEFxJHEs+Dib8B1T0D+eFj6CbjnY1C3PdWVJbSorISGlnZ+95JmmxOR46eA6InxZ8BfPQYf+Gd44xG4dTa88EvoZ81zsyaN5ITCHO7RAH4ikgQKiJ6KpcP7b4TP/QnGToffXA+/uBJ2vZ3qyg4yM8pLS1i9eTcbd9aluhwRGeAUEL01+kS49rdw+b9D5Wr4r/fBMz+Bjv4xcc9VZxWTnmYsrdAAfiJyfBQQxyItDco+Ddc/B5PPg5VfD+6d2Lk+1ZVRmJfJxaeNYfnqSlraOlJdjogMYAqI4zG8CD56D8y/Ixiy46fnwRPfhbaWlJZVXlZCbUMLj23QbHMicuwUEMfLDE6/Gq5/HqZdCU/8X1h8QdD8lCLnn1TI2PxM7lmlzmoROXYKiGTJGQ3zb4eP3AP798AdH4CV/wAtDX1eSnosjQWzSnjy9Wq27d3f5+8vIoODAiLZTpkT9E3M+hQ88+OgE/utJ/u8jIWlJXQ4LFNntYgcIwVEFLLy4UP/Dp98CCwGd82FFX8TnFn0kYkF2ZxzQgFLV2+hQ7PNicgxUEBEafK58Nd/gnNvhBd+FQzXsf63ffb25WUlbNm1n2ffqu2z9xSRwUMBEbX4MPjgP8NnHoPcQrjnGlh6LdTvjPyt50wfR35WOkvUWS0ix0AB0VcmzIDPPA4X/xO89jD8uAxevDvS4Tqy4jGunFnE79dtZ29ja2TvIyKDkwKiL8XicN6Xg1FiC0+FBz4Hv7wKdm+O7C0XlpbQ0tbBAy9qtjkR6R0FRCoUngyfehgu+34wvelPzoHnboOO5N/5PL1oONOL8lmySrPNiUjvRBoQZjbHzF4zs41mdlOC9Zlmdk+4/jkzm9xp3Rlm9oyZrTOzl80sK8pa+1xaWjBz3eefhUnnwMN/B/89B6pfS/pblZeWsH7bPl6p2pf0Y4vI4BVZQJhZjGDq0EuBqcBHzGxql80+Dex29xOBHwDfDfdNB34JfM7dpwEXAoOzEX1ECVyzDD68GGpeh5++H576HrQn7+POnVFEZnoa91S8k7RjisjgF+UZxGxgo7u/5e4twBJgXpdt5gF3hs+XARebmQGXAC+5+1oAd6919/4xXGoUzODMcrh+FZz6IXjs27D4Qqhak5TDDx8W59Lp4/jNi1vZ3zJ4/xhFJLmiDIgioPP1lZXhsoTbuHsbsBcoAE4G3MxWmtkaM/u7COvsP3ILYcF/w6JfQ2Mt3H4x/OEb0NJ43IcuL5tIXVMbD7+i2eZEpGf6ayd1OvB+4Jrw54fN7OKuG5nZdWZWYWYV1dXVfV1jdE69POibmPlx+POP4Kfnwtv/e1yHPPuEUUwqyNYAfiLSY1EGRBVQ0ul1cbgs4TZhv8NwoJbgbOMpd69x90bgIeCsrm/g7ovdvdTdSwsLCyP4CCk0bATM/RFc+2Bwr8SdH4IHb4Smvcd0ODNjYWkJz729i7dr+n4AQREZeKIMiFXASWY2xcwygEXAii7brACuDZ9fDTzmwbWYK4HTzSw7DI4LgFcjrLX/mnI+/PWf4X1/A2vuhFvPDm60OwZXzyomzWCp5qwWkR6ILCDCPoUbCL7s1wNL3X2dmd1sZnPDze4ACsxsI/Al4KZw393AvxOEzIvAGnf/XVS19nsZ2XDJt+Gv/gjDRsLdi2DZX0J975rVxuZn8X9OCWaba2vXbHMicmQ2WG6eKi0t9YqKilSXEb22FvjTD+HJf4XMPLj0u3D6guBKqB5YuW47n/3Fam7/RCkfmDo24mJFpL8zs9XuXppoXX/tpJbDSc+AC/4uGK6j4D1w32fg1wthb8/mfbjo1DGMzs3kHjUzichRKCAGqjGnwl+uhDnfhU1PB0OJP/+zow7XEY+lMX9WEY9t2MlTr1drrggROSwFxECWFoOzPxdcEltcBg99Bf7ncqh544i7fey9kxg+LM4nfv487//uY3xv5Qbeqq7vo6JFZKBQH8Rg4Q5r74bffx1a98OFNwVXPsXiCTdvam1n5brt3Lemiv99o5oOh5kTRzD/rGKuOGMCw7MT7ycig8uR+iAUEINN3Q54+Kvw6m9g3Okw71YYf+YRd9mxr4kHXqhi+ZpKXt9RT0YsjQ9MHcP8s4o5/+RC4jGdaABzIwcAABEgSURBVIoMVgqIoejVFUGTU0MNnPsFuOBrwex2R+DuvFK1j+VrKlmxdiu7GloYnZvBvBlFXHVWEdMmDO+j4kWkrygghqr9u+EP/wgv/BIKToS5/wmT3tejXVvaOnjitZ0sX1PJYxt20trunDouj6tnFTNvRhGFeZkRFy8ifUEBMdS9+Tg8+EXYsxnK/gou/iZk5fd4990NLTz40laWr65kbeVeYmnGBScXMv+sYi4+bQxZ8ViExYtIlBQQAi0N8Nh34NmfQH4RXP59eM9FkN67M4E3dtSxfE0VD7xQxfZ9TeRnpfOhMycw/6xizpo4AuvhDXsi0j8oIORdW1bBihugekPwOrsA8iZA3jjIHx88zx8PeeEjfwIMGxXMgNdJe4fz5zdrWL66kt+v205TawdTRudw1cwiPnxWEcUjs1Pw4USktxQQcqi25uAqp92boG4b7NsGdVuDnw3VQJd/E7EMyB33bnDkh4EShkl9ZiErNxv3rK3l+bd3AXDOCQXMn1XMpdPHkZOZ3ucfUUR6RgEhPdfeCvU7Dg2NuvCxb+u7gdKaYMjwrOG0Zo9ja8dIXq3PZmNTPrvSCiie9B5Kp5/G6aeeRlremOAGPxHpF44UEPrVTg4Vi8Pw4uBxJE37OoXG9oNhEq/bxqS6bUxs3wDtOzBvD+YM3AI8DB3E6MgdQ/rwCZ3ORg6cmXRq4srM64tPKyJHoICQY5OVHzwKT0m42gA62qF+J827q1j76nrWv/4a9dXvMHbvbk5srmPy3vUMb30Ka97X/QAZuYeGRt64d8PkwM/csRDTP2HpBzo6oL05aL5taw6ft7y7rL2ly7pOy9pboK0p8fYH13Vd1uV9xk6Djy5J+sfS/y6JTloM8seTmT+e2ZNKmX0p7NzXxAMvVvG11VW8tqOOjFgal52ax8JT4swuaCa9YXu3sxI2/yk4W+lo6/IGBrljEvaLHHJWkjWix8OhywDQ0RF8Sba3BE2i3Z53/nLu+qV7mC/iHn2pH+FY3f5tHisLriyMZQYjN8cyg9fpmUFf4IGf2TmHrit4T5Lev0s16oOQVHB31m3dx7LVPbxru6MDGms79YuEIdK5X6RuG+zf1f3N0ocFQZEzJvhPl5be6RELf8a7vO60PhZPsP2RHuE2sSMcM+0Ix4zFuy+ztGhD7qhfukd53tF6DPsebn3bkbf19uR+9tgRvojTMyE9q/uyo67L6sX2YRikZ4V/1337y4w6qaVfa2nr4MnXq1m+upJHN+w4vru2W5ve7VTvHBz7tgZXaHW0BY/21qAJ7MDrjrYurxOs9xTPwncsQYaFX95dv4xbo/3S7VxzLCOoLZaR+HlavMvyI2x7xOfxd58f/OLtwZf0ED+7TFlAmNkc4D+AGHC7u9/SZX0mcBcwC6gFyt19U6f1Ewnmov6Wu3//SO+lgBgcDt61vaaKtVv2EEszzj9pNPNnFfOB08am9q7tjo7gi7S99TChkujRZX37kbZpPcwxj/ae7Z327bLeO7p/oR780j6eL+OMoP/nSNukxbvdPyP9T0oCwsxiwOvAB4FKgvmlP+Lur3ba5vPAGe7+OTNbBHzY3cs7rV9GcFH+cwqIoWfjzuCu7fvX6K5tkaikKiDOIfjN/y/C118HcPf/12mbleE2z5hZOrAdKHR3N7MrgXOBBqBeATF0Hbhr+741VTz8yjbdtS2SRKm6D6KI4Or3AyqB9x5uG3dvM7O9QIGZNQFfIzj7+Mrh3sDMrgOuA5g4cWLyKpd+JZZmnHdSIeedVMi/XDmdh17exvLVlfzbI6/zb4+8rru2RSLSX/83fQv4gbvXH6kZwd0XA4shOIPom9IklXIz01lYWsLC0hK27Grk/nCio6/cu5ZvPPAKl04fx/xZxZxzQgFpaWqCEjkeUQZEFVDS6XVxuCzRNpVhE9Nwgs7q9wJXm9m/AiOADjNrcvcfR1ivDDAlo7L5wsUn8TcXncjqzbtZvqaS367dxn0vVDFheBYfPquIq84q5j2FuakuVWRAirIPIp2gk/pigiBYBXzU3dd12uZ64PROndRXufvCLsf5FuqDkB5qam3nkVd3sHxNJU+9Hsy1PaNkBPNnFXPFGeMZkZ2R6hJF+pVUXuZ6GfBDgstcf+7u3zGzm4EKd19hZlnAL4CZwC5gkbu/1eUY30IBIcfgwF3by8O7ts1gwvBhTB6dzeSCnOAxOofJBdmUjMrWxEcyJOlGORnSDty1/cf1O9hU08Cm2kY21Tawp7H14DYHwmNSQTaTCnKYMjr4Obkgh0kFCg8ZvDSaqwxpZsb0ouFMLzp0+I49jS1sDsNiU00jm2sbeLu2gZXrtrOroeWQbccPz2JSQTZTRueEwfFugAzLUHjI4KSAkCFrRHYGI7IzOLNkRLd1e/e3srk2ONvYXBMEx+baRv6wbge1XcJjbH5mcNZRkMOkTs1XkwqyddmtDGj61yuSwPBhcc4oHsEZxd3DY19TK+/UNvJ2TcPBENlU08CjG3ZSU998yLaFeZlBcBRkh/0d7z7PVXhIP6d/oSK9lJ8VT9hkBVDf3MammoZOTVfB8ydfr+be1ZWHbDs6N/NgU1XnPo/Jo7PJy4r31ccROSwFhEgS5WamHzY8Gprb2Fz7bl/H5pogRJ7eWM3yNYeeeRTkZDB5dHi2EZ51HOj/GD5M4SF9QwEh0kdyMtOZOiGfqRPyu61rbGnjnV2NB6+y2lzbwNs1DTzzZi33rTn0/tKR2fFDmqs6d5zrPg9JJgWESD+QnZHOqePyOXVc9/Boam0/pMnqQIA891Yt979waHiMyI4fDIvJBTmUjMqmMC+TwtxMRudlMCo7g/SYhuCWnlFAiPRzWfEYp4zL45Rxed3WNbW2s2XXgQ7zMERqG6jYtJsVa7fS9TYnMxiVncHo3EwK8zIZnRs8H52XeciywtxMRuUoTIY6BYTIAJYVj3HS2DxOGts9PJrb2tm6p4ma+mZq6pqpqW+muq6Z6vqWYFl9M5s2N1BT30xTa/fZ8sxgZHbGwbOP0bmZBx+dw6UwL5MChcmgpIAQGaQy02NMGZ3DlNE5R9zO3WloaT8kRGrqOwVJXTPV9c2seWc3NXUt7G9NPD3pqJyMd89IDoZIGCRhM1dhXnBmEleYDAgKCJEhzszIzUwnNzOdyUcJEwiuxuoWJF3C5cUte6ipb6axJXGYjMyOdw+S8CylsNOyglyFSSopIESkV3Iy08nJTGdSQc/DJAiPlk7Pm8PnLayt3ENNXTMNhwmTEWGYFB7sK+keJKPzMijIySQjXWGSTAoIEYlMb8KksaWNmroWquubuwdJGC4vV+6h+ghhMnxYnILcDIYPi3d75GeFPzsvzw5+5mTENMd5AgoIEekXsjPSmViQzsSCo88xvr+lPWzeereP5ECI7GpoYe/+VmrrW3iruoG9+1vZ19Ta7YquztLT7GBwHBIgw9ITBk1+p4DJy0wftOGigBCRAWdYRoySUcE8Hj3R0eHUt7Sxt7E1CIz9wc/DPhpbeKf2QLi00d5x+HRJM7qESteQSfzIHxaES3+eGlcBISKDXlqaBb/5Z8UPmQe5Jw5c5RUEx7shcqSQqdq9/+DztiOEixnkZaYfbOrqTcjkZcWJRRwuCggRkSPofJVX0YhhvdrX3WkMw2Vf06EBc7iQ2b63ib3729i3v5WW9u73p3SWlxU0gc2cOJL//MjM4/mYCUUaEGY2B/gPgilHb3f3W7qszwTuAmYBtUC5u28ysw8CtwAZQAvwVXd/LMpaRUSSzcwOdtRPoPfh0tTacdRA2be/lfEjsiKpP7KAMLMYcCvwQaASWGVmK9z91U6bfRrY7e4nmtki4LtAOVADXOHuW81sOrASKIqqVhGR/sbMGJYRY1hGjHHDowmAo4nyouHZwEZ3f8vdW4AlwLwu28wD7gyfLwMuNjNz9xfcfWu4fB0wLDzbEBGRPhJlQBQBWzq9rqT7WcDBbdy9DdgLFHTZZj6wxt2buyzHzK4zswozq6iurk5a4SIiEm1AHDczm0bQ7PTZROvdfbG7l7p7aWFhYd8WJyIyyEUZEFVwyBVlxeGyhNuYWTownKCzGjMrBu4HPuHub0ZYp4iIJBBlQKwCTjKzKWaWASwCVnTZZgVwbfj8auAxd3czGwH8DrjJ3f8UYY0iInIYkQVE2KdwA8EVSOuBpe6+zsxuNrO54WZ3AAVmthH4EnBTuPwG4ETgn8zsxfAxJqpaRUSkO/MjDVAygJSWlnpFRUWqyxARGVDMbLW7lyZa1687qUVEJHUGzRmEmVUDm4/jEKMJbtAbSobaZx5qnxf0mYeK4/nMk9w94WWggyYgjpeZVRzuNGuwGmqfeah9XtBnHiqi+sxqYhIRkYQUECIikpAC4l2LU11ACgy1zzzUPi/oMw8VkXxm9UGIiEhCOoMQEZGEFBAiIpLQkA8IM/u5me00s1dSXUtfMLMSM3vczF41s3Vm9sVU1xQ1M8sys+fNbG34mf851TX1FTOLmdkLZvbbVNfSF8xsk5m9HA7PMySGVjCzEWa2zMw2mNl6Mzsnacce6n0QZnY+UA/c5e7TU11P1MxsPDDe3deYWR6wGriyy0x/g4qZGZDj7vVmFgeeBr7o7s+muLTImdmXgFIg390/lOp6omZmm4BSdx8yN8qZ2Z3A/7r77eHAqNnuvicZxx7yZxDu/hSwK9V19BV33+bua8LndQQDKQ7q6Vw9UB++jIePQf+bUThk/uXA7amuRaJhZsOB8wkGPsXdW5IVDqCAGNLMbDIwE3gutZVEL2xqeRHYCTzi7oP+MwM/BP4O6Eh1IX3IgT+Y2Wozuy7VxfSBKUA18N9hU+LtZpaTrIMrIIYoM8sFlgM3uvu+VNcTNXdvd/cZBBNXzTazQd2caGYfAna6++pU19LH3u/uZwGXAteHTciDWTpwFvBf7j4TaODdaROOmwJiCArb4ZcDv3L3+1JdT18KT78fB+akupaInQvMDdvklwAXmdkvU1tS9Ny9Kvy5k2BGytmprShylUBlpzPiZQSBkRQKiCEm7LC9A1jv7v+e6nr6gpkVhrMUYmbDgA8CG1JbVbTc/evuXuzukwlmc3zM3T+W4rIiZWY54YUXhM0slwCD+upEd98ObDGzU8JFFwNJu+AkPVkHGqjM7G7gQmC0mVUC33T3O1JbVaTOBT4OvBy2yQP8vbs/lMKaojYeuNPMYgS/FC119yFx2ecQMxa4P/gdiHTg1+7++9SW1Cf+BvhVeAXTW8CnknXgIX+Zq4iIJKYmJhERSUgBISIiCSkgREQkIQWEiIgkpIAQEZGEFBAi/YCZXThURlyVgUMBISIiCSkgRHrBzD4Wzi3xopndFg4CWG9mPwjnmnjUzArDbWeY2bNm9pKZ3W9mI8PlJ5rZH8P5KdaY2XvCw+d2Gtf/V+Fd7yIpo4AQ6SEzOw0oB84NB/5rB64BcoAKd58GPAl8M9zlLuBr7n4G8HKn5b8CbnX3M4H3AdvC5TOBG4GpwAkEd72LpMyQH2pDpBcuBmYBq8Jf7ocRDB/eAdwTbvNL4L5wnP4R7v5kuPxO4N5wrKAid78fwN2bAMLjPe/uleHrF4HJBJMbiaSEAkKk5wy4092/fshCs2902e5Yx69p7vS8Hf3/lBRTE5NIzz0KXG1mYwDMbJSZTSL4f3R1uM1HgafdfS+w28zOC5d/HHgynMWv0syuDI+RaWbZffopRHpIv6GI9JC7v2pm/0gwY1ka0ApcTzBJy+xw3U6CfgqAa4GfhgHQeZTNjwO3mdnN4TEW9OHHEOkxjeYqcpzMrN7dc1Ndh0iyqYlJREQS0hmEiIgkpDMIERFJSAEhIiIJKSBERCQhBYSIiCSkgBARkYT+P/Od3U1D4NxvAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from seqeval.metrics import f1_score, classification_report"
      ],
      "metadata": {
        "id": "sTq2LfgZ9RAt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4cpU0SDGWerH"
      },
      "source": [
        "def sequences_to_tag(sequences): # 예측값을 index_to_ner를 사용하여 태깅 정보로 변경하는 함수.\n",
        "    result = []\n",
        "    for sequence in sequences: # 전체 시퀀스로부터 시퀀스를 하나씩 꺼낸다.\n",
        "        temp = []\n",
        "        for pred in sequence: # 시퀀스로부터 예측값을 하나씩 꺼낸다.\n",
        "            pred_index = np.argmax(pred) # 예를 들어 [0, 0, 1, 0 ,0]라면 1의 인덱스인 2를 리턴한다.\n",
        "            temp.append(index_to_ner[pred_index].replace(\"PAD\", \"O\")) # 'PAD'는 'O'로 변경\n",
        "        result.append(temp)\n",
        "    return result"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JHhSt-loWhc-"
      },
      "source": [
        "y_predicted = model.predict([X_test])\n",
        "pred_tags = sequences_to_tag(y_predicted)\n",
        "test_tags = sequences_to_tag(y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1djczWAQWlSH",
        "outputId": "876fc212-d8ef-433e-c3b8-83309995a3eb"
      },
      "source": [
        "print(classification_report(test_tags, pred_tags))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         art       0.16      0.05      0.07        63\n",
            "         eve       0.38      0.27      0.31        52\n",
            "         geo       0.81      0.85      0.83      7620\n",
            "         gpe       0.95      0.94      0.95      3145\n",
            "         nat       0.45      0.24      0.32        37\n",
            "         org       0.62      0.55      0.58      4033\n",
            "         per       0.72      0.71      0.72      3545\n",
            "         tim       0.87      0.83      0.85      4067\n",
            "\n",
            "   micro avg       0.80      0.78      0.79     22562\n",
            "   macro avg       0.62      0.55      0.58     22562\n",
            "weighted avg       0.79      0.78      0.78     22562\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FbQE34bSOdhP"
      },
      "source": [
        "## 성능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M00m3njrWoNd",
        "outputId": "96a098f5-9e3e-44ec-dc91-414cec2ea45c"
      },
      "source": [
        "print(\"F1-score: {:.1%}\".format(f1_score(test_tags, pred_tags)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "F1-score: 78.7%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MJIMCtl9Acxp"
      },
      "source": [
        "# BiLSTM-CRF를 이용한 개체명인식"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o9Xi26XHOhbv"
      },
      "source": [
        "## 모델링"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras import Model\n",
        "from tensorflow.keras.layers import Dense, LSTM, Input, Bidirectional, TimeDistributed, Embedding, Dropout\n",
        "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
        "from keras_crf import CRFModel\n",
        "from seqeval.metrics import f1_score, classification_report"
      ],
      "metadata": {
        "id": "ccSUZlLYGga1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CVvgFVt3llwU",
        "outputId": "97fb67ef-d6fa-4787-cc10-5d171a4f8ab2"
      },
      "source": [
        "embedding_dim = 128\n",
        "hidden_units = 64\n",
        "dropout_ratio = 0.3\n",
        "\n",
        "sequence_input = Input(shape=(max_len,),dtype=tf.int32, name='sequence_input')\n",
        "\n",
        "model_embedding = Embedding(input_dim=vocab_size,\n",
        "                            output_dim=embedding_dim,\n",
        "                            input_length=max_len)(sequence_input)\n",
        "\n",
        "model_bilstm = Bidirectional(LSTM(units=hidden_units, return_sequences=True))(model_embedding)\n",
        "\n",
        "model_dropout = TimeDistributed(Dropout(dropout_ratio))(model_bilstm)\n",
        "\n",
        "model_dense = TimeDistributed(Dense(tag_size, activation='relu'))(model_dropout)\n",
        "\n",
        "base = Model(inputs=sequence_input, outputs=model_dense)\n",
        "model = CRFModel(base, tag_size)\n",
        "model.compile(optimizer=tf.keras.optimizers.Adam(0.001), metrics='accuracy')\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"crf_model\"\n",
            "__________________________________________________________________________________________________\n",
            " Layer (type)                   Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            " sequence_input (InputLayer)    [(None, 70)]         0           []                               \n",
            "                                                                                                  \n",
            " embedding_2 (Embedding)        (None, 70, 128)      4072832     ['sequence_input[0][0]']         \n",
            "                                                                                                  \n",
            " bidirectional_2 (Bidirectional  (None, 70, 128)     98816       ['embedding_2[0][0]']            \n",
            " )                                                                                                \n",
            "                                                                                                  \n",
            " time_distributed_3 (TimeDistri  (None, 70, 128)     0           ['bidirectional_2[0][0]']        \n",
            " buted)                                                                                           \n",
            "                                                                                                  \n",
            " time_distributed_4 (TimeDistri  (None, 70, 18)      2322        ['time_distributed_3[0][0]']     \n",
            " buted)                                                                                           \n",
            "                                                                                                  \n",
            " crf (CRF)                      [(None, 70),         702         ['time_distributed_4[0][0]']     \n",
            "                                 (None, 70, 18),                                                  \n",
            "                                 (None,),                                                         \n",
            "                                 (18, 18)]                                                        \n",
            "                                                                                                  \n",
            " decode_sequence (Lambda)       (None, 70)           0           ['crf[0][0]']                    \n",
            "                                                                                                  \n",
            " potentials (Lambda)            (None, 70, 18)       0           ['crf[0][1]']                    \n",
            "                                                                                                  \n",
            " sequence_length (Lambda)       (None,)              0           ['crf[0][2]']                    \n",
            "                                                                                                  \n",
            " kernel (Lambda)                (18, 18)             0           ['crf[0][3]']                    \n",
            "                                                                                                  \n",
            "==================================================================================================\n",
            "Total params: 4,174,672\n",
            "Trainable params: 4,174,672\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jv3m451klwxt"
      },
      "source": [
        "es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=4)\n",
        "mc = ModelCheckpoint('bilstm_crf/cp.ckpt', monitor='val_decode_sequence_accuracy', mode='max', verbose=1, save_best_only=True, save_weights_only=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NYUs5NYDl4_x",
        "outputId": "427eb8e8-f721-4c14-a79c-1d9312b08496"
      },
      "source": [
        "history = model.fit(X_train, y_train_int, batch_size=128, epochs=15, validation_split=0.1, callbacks=[mc, es])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9204 - loss: 28.7448\n",
            "Epoch 00001: val_decode_sequence_accuracy improved from -inf to 0.96133, saving model to bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 46s 147ms/step - decode_sequence_accuracy: 0.9204 - loss: 28.6784 - val_decode_sequence_accuracy: 0.9613 - val_loss: 10.0355\n",
            "Epoch 2/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9702 - loss: 6.8184\n",
            "Epoch 00002: val_decode_sequence_accuracy improved from 0.96133 to 0.97886, saving model to bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 39s 146ms/step - decode_sequence_accuracy: 0.9702 - loss: 6.8108 - val_decode_sequence_accuracy: 0.9789 - val_loss: 5.2493\n",
            "Epoch 3/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9817 - loss: 3.9034\n",
            "Epoch 00003: val_decode_sequence_accuracy improved from 0.97886 to 0.98280, saving model to bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 38s 141ms/step - decode_sequence_accuracy: 0.9817 - loss: 3.9048 - val_decode_sequence_accuracy: 0.9828 - val_loss: 4.1232\n",
            "Epoch 4/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9853 - loss: 2.8616\n",
            "Epoch 00004: val_decode_sequence_accuracy improved from 0.98280 to 0.98349, saving model to bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 37s 138ms/step - decode_sequence_accuracy: 0.9853 - loss: 2.8610 - val_decode_sequence_accuracy: 0.9835 - val_loss: 3.4773\n",
            "Epoch 5/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9874 - loss: 2.3106\n",
            "Epoch 00005: val_decode_sequence_accuracy improved from 0.98349 to 0.98475, saving model to bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 40s 148ms/step - decode_sequence_accuracy: 0.9874 - loss: 2.3088 - val_decode_sequence_accuracy: 0.9847 - val_loss: 3.3564\n",
            "Epoch 6/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9887 - loss: 1.9530\n",
            "Epoch 00006: val_decode_sequence_accuracy improved from 0.98475 to 0.98522, saving model to bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 49s 180ms/step - decode_sequence_accuracy: 0.9887 - loss: 1.9524 - val_decode_sequence_accuracy: 0.9852 - val_loss: 3.2281\n",
            "Epoch 7/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9898 - loss: 1.6867\n",
            "Epoch 00007: val_decode_sequence_accuracy improved from 0.98522 to 0.98530, saving model to bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 37s 138ms/step - decode_sequence_accuracy: 0.9898 - loss: 1.6860 - val_decode_sequence_accuracy: 0.9853 - val_loss: 3.2156\n",
            "Epoch 8/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9906 - loss: 1.4833\n",
            "Epoch 00008: val_decode_sequence_accuracy did not improve from 0.98530\n",
            "270/270 [==============================] - 37s 136ms/step - decode_sequence_accuracy: 0.9906 - loss: 1.4830 - val_decode_sequence_accuracy: 0.9852 - val_loss: 3.2492\n",
            "Epoch 9/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9913 - loss: 1.3182\n",
            "Epoch 00009: val_decode_sequence_accuracy did not improve from 0.98530\n",
            "270/270 [==============================] - 38s 140ms/step - decode_sequence_accuracy: 0.9913 - loss: 1.3198 - val_decode_sequence_accuracy: 0.9851 - val_loss: 3.3742\n",
            "Epoch 10/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9920 - loss: 1.1570\n",
            "Epoch 00010: val_decode_sequence_accuracy did not improve from 0.98530\n",
            "270/270 [==============================] - 41s 151ms/step - decode_sequence_accuracy: 0.9920 - loss: 1.1570 - val_decode_sequence_accuracy: 0.9849 - val_loss: 3.5492\n",
            "Epoch 11/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9925 - loss: 1.0284\n",
            "Epoch 00011: val_decode_sequence_accuracy did not improve from 0.98530\n",
            "270/270 [==============================] - 36s 133ms/step - decode_sequence_accuracy: 0.9925 - loss: 1.0281 - val_decode_sequence_accuracy: 0.9847 - val_loss: 3.5498\n",
            "Epoch 00011: early stopping\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p5UqeRqwre2t",
        "outputId": "c7d80748-85d5-437c-88cd-adb01a57f344"
      },
      "source": [
        "model.load_weights('bilstm_crf/cp.ckpt')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7fc6c0ea0f50>"
            ]
          },
          "metadata": {},
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "i = 13 # 확인하고 싶은 테스트용 샘플의 인덱스.\n",
        "y_predicted = model.predict(np.array([X_test[i]]))[0] # 입력한 테스트용 샘플에 대해서 예측 y를 리턴\n",
        "true = np.argmax(y_test[i], -1) # 원-핫 인코딩을 다시 정수 인코딩으로 변경함.\n",
        "\n",
        "print(\"{:15}|{:5}|{}\".format(\"단어\", \"실제값\", \"예측값\"))\n",
        "print(35 * \"-\")\n",
        "\n",
        "for word, tag, pred in zip(X_test[i], true, y_predicted[0]):\n",
        "    if word != 0: # PAD값은 제외함.\n",
        "        print(\"{:17}: {:7} {}\".format(index_to_word[word], index_to_ner[tag], index_to_ner[pred]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y2ojsYdHH4vA",
        "outputId": "852855be-e993-41a2-bb00-5778d7bebfa6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "단어             |실제값  |예측값\n",
            "-----------------------------------\n",
            "the              : O       O\n",
            "statement        : O       O\n",
            "came             : O       O\n",
            "as               : O       O\n",
            "u.n.             : B-org   B-org\n",
            "secretary-general: I-org   I-org\n",
            "kofi             : B-per   B-per\n",
            "annan            : I-per   I-per\n",
            "met              : O       O\n",
            "with             : O       O\n",
            "officials        : O       O\n",
            "in               : O       O\n",
            "amman            : B-geo   B-geo\n",
            "to               : O       O\n",
            "discuss          : O       O\n",
            "wednesday        : B-tim   B-tim\n",
            "'s               : O       O\n",
            "attacks          : O       O\n",
            ".                : O       O\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QyrXGm8sm9Xl"
      },
      "source": [
        "y_predicted = model.predict(X_test)[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fz3ataXrnFNj"
      },
      "source": [
        "def sequences_to_tag(sequences): # 예측값을 index_to_ner를 사용하여 태깅 정보로 변경하는 함수.\n",
        "    result = []\n",
        "    for sequence in sequences: # 전체 시퀀스로부터 시퀀스를 하나씩 꺼낸다.\n",
        "        temp = []\n",
        "        for pred in sequence: # 시퀀스로부터 예측값을 하나씩 꺼낸다.\n",
        "            pred_index = np.argmax(pred) # 예를 들어 [0, 0, 1, 0 ,0]라면 1의 인덱스인 2를 리턴한다.\n",
        "            temp.append(index_to_ner[pred_index].replace(\"PAD\", \"O\")) # 'PAD'는 'O'로 변경\n",
        "        result.append(temp)\n",
        "    return result"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UHgc7HhZm-cs"
      },
      "source": [
        "def sequences_to_tag_for_crf(sequences): # 예측값을 index_to_ner를 사용하여 태깅 정보로 변경하는 함수.\n",
        "    result = []\n",
        "    for sequence in sequences: # 전체 시퀀스로부터 시퀀스를 하나씩 꺼낸다.\n",
        "        temp = []\n",
        "        for pred in sequence: # 시퀀스로부터 예측값을 하나씩 꺼낸다.\n",
        "            # pred_index = np.argmax(pred) # 예를 들어 [0, 0, 1, 0 ,0]라면 1의 인덱스인 2를 리턴한다.\n",
        "            pred_index = pred\n",
        "            temp.append(index_to_ner[pred_index].replace(\"PAD\", \"O\")) # 'PAD'는 'O'로 변경\n",
        "        result.append(temp)\n",
        "    return result"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GRjWgGwVm_y7"
      },
      "source": [
        "pred_tags = sequences_to_tag_for_crf(y_predicted)\n",
        "test_tags = sequences_to_tag(y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J62vAVj-nBmP",
        "outputId": "565a8fea-6764-4c09-e331-88eee5c4e6ad"
      },
      "source": [
        "print(classification_report(test_tags, pred_tags))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         art       0.00      0.00      0.00        63\n",
            "         eve       0.90      0.17      0.29        52\n",
            "         geo       0.83      0.84      0.84      7620\n",
            "         gpe       0.95      0.93      0.94      3145\n",
            "         nat       0.00      0.00      0.00        37\n",
            "         org       0.64      0.58      0.61      4033\n",
            "         per       0.76      0.70      0.73      3545\n",
            "         tim       0.86      0.83      0.85      4067\n",
            "\n",
            "   micro avg       0.81      0.78      0.80     22562\n",
            "   macro avg       0.62      0.51      0.53     22562\n",
            "weighted avg       0.81      0.78      0.79     22562\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pY8rEL_tOjCe"
      },
      "source": [
        "## 성능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7Udxj7vLnHGe",
        "outputId": "56638477-3a13-404b-9633-4cd160dd1f38"
      },
      "source": [
        "print(\"F1-score: {:.1%}\".format(f1_score(test_tags, pred_tags)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "F1-score: 79.6%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BSiCMNSs7ynl"
      },
      "source": [
        "# BiLSTM-CNN을 이용한 개체명인식"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HWUVra7SOfLk"
      },
      "source": [
        "## 모델링"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OsijEbzI99sN"
      },
      "source": [
        "from tensorflow.keras.layers import Embedding, Input, TimeDistributed, Dropout, concatenate, Bidirectional, LSTM, Conv1D, Dense, MaxPooling1D, Flatten\n",
        "from tensorflow.keras import Model\n",
        "from tensorflow.keras.initializers import RandomUniform\n",
        "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
        "from tensorflow.keras.models import load_model\n",
        "from seqeval.metrics import f1_score, classification_report"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Cl_f3LMS-IzV",
        "outputId": "149bc6f4-931f-4588-ad27-120584df180f"
      },
      "source": [
        "embedding_dim = 128\n",
        "char_embedding_dim = 64\n",
        "dropout_ratio = 0.5\n",
        "hidden_units = 256\n",
        "num_filters = 30\n",
        "kernel_size = 3\n",
        "\n",
        "# 단어 임베딩\n",
        "words_input = Input(shape=(None,),dtype='int32',name='words_input')\n",
        "words = Embedding(input_dim=vocab_size, output_dim=embedding_dim)(words_input)\n",
        "\n",
        "# char 임베딩\n",
        "character_input = Input(shape=(None, max_len_char,),name='char_input')\n",
        "embed_char_out = TimeDistributed(Embedding(len(char_to_index), char_embedding_dim, embeddings_initializer=RandomUniform(minval=-0.5, maxval=0.5)), name='char_embedding')(character_input)\n",
        "dropout = Dropout(dropout_ratio)(embed_char_out)\n",
        "\n",
        "# char 임베딩에 대해서는 Conv1D 수행\n",
        "conv1d_out= TimeDistributed(Conv1D(kernel_size=kernel_size, filters=num_filters, padding='same', activation='tanh', strides=1))(dropout)\n",
        "maxpool_out=TimeDistributed(MaxPooling1D(max_len_char))(conv1d_out)\n",
        "char = TimeDistributed(Flatten())(maxpool_out)\n",
        "char = Dropout(dropout_ratio)(char)\n",
        "\n",
        "# char 임베딩을 Conv1D 수행한 뒤에 단어 임베딩과 연결\n",
        "output = concatenate([words, char])\n",
        "\n",
        "# 연결한 벡터를 가지고 문장의 길이만큼 LSTM을 수행\n",
        "output = Bidirectional(LSTM(hidden_units, return_sequences=True, dropout=dropout_ratio))(output)\n",
        "\n",
        "# 출력층\n",
        "output = TimeDistributed(Dense(tag_size, activation='softmax'))(output)\n",
        "\n",
        "model = Model(inputs=[words_input, character_input], outputs=[output])\n",
        "model.compile(loss='categorical_crossentropy', optimizer='nadam',  metrics=['acc'])\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_1\"\n",
            "__________________________________________________________________________________________________\n",
            " Layer (type)                   Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            " char_input (InputLayer)        [(None, None, 15)]   0           []                               \n",
            "                                                                                                  \n",
            " char_embedding (TimeDistribute  (None, None, 15, 64  4736       ['char_input[0][0]']             \n",
            " d)                             )                                                                 \n",
            "                                                                                                  \n",
            " dropout_2 (Dropout)            (None, None, 15, 64  0           ['char_embedding[0][0]']         \n",
            "                                )                                                                 \n",
            "                                                                                                  \n",
            " time_distributed_6 (TimeDistri  (None, None, 15, 30  5790       ['dropout_2[0][0]']              \n",
            " buted)                         )                                                                 \n",
            "                                                                                                  \n",
            " time_distributed_7 (TimeDistri  (None, None, 1, 30)  0          ['time_distributed_6[0][0]']     \n",
            " buted)                                                                                           \n",
            "                                                                                                  \n",
            " words_input (InputLayer)       [(None, None)]       0           []                               \n",
            "                                                                                                  \n",
            " time_distributed_8 (TimeDistri  (None, None, 30)    0           ['time_distributed_7[0][0]']     \n",
            " buted)                                                                                           \n",
            "                                                                                                  \n",
            " embedding_4 (Embedding)        (None, None, 128)    4072832     ['words_input[0][0]']            \n",
            "                                                                                                  \n",
            " dropout_3 (Dropout)            (None, None, 30)     0           ['time_distributed_8[0][0]']     \n",
            "                                                                                                  \n",
            " concatenate (Concatenate)      (None, None, 158)    0           ['embedding_4[0][0]',            \n",
            "                                                                  'dropout_3[0][0]']              \n",
            "                                                                                                  \n",
            " bidirectional_4 (Bidirectional  (None, None, 512)   849920      ['concatenate[0][0]']            \n",
            " )                                                                                                \n",
            "                                                                                                  \n",
            " time_distributed_9 (TimeDistri  (None, None, 18)    9234        ['bidirectional_4[0][0]']        \n",
            " buted)                                                                                           \n",
            "                                                                                                  \n",
            "==================================================================================================\n",
            "Total params: 4,942,512\n",
            "Trainable params: 4,942,512\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iNxMCvYYYzYm"
      },
      "source": [
        "es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=4)\n",
        "mc = ModelCheckpoint('bilstm_cnn.h5', monitor='val_acc', mode='max', verbose=1, save_best_only=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ywZ-xsUHBGn9",
        "outputId": "234b5eaa-c629-49ec-a56a-703e6043b044"
      },
      "source": [
        "history = model.fit([X_train, X_char_train], y_train, batch_size=128, epochs=15, validation_split=0.1, verbose=1, callbacks=[es, mc])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/15\n",
            "270/270 [==============================] - ETA: 0s - loss: 0.2072 - acc: 0.9474\n",
            "Epoch 00001: val_acc improved from -inf to 0.97665, saving model to bilstm_cnn.h5\n",
            "270/270 [==============================] - 23s 47ms/step - loss: 0.2072 - acc: 0.9474 - val_loss: 0.0852 - val_acc: 0.9766\n",
            "Epoch 2/15\n",
            "269/270 [============================>.] - ETA: 0s - loss: 0.0635 - acc: 0.9820\n",
            "Epoch 00002: val_acc improved from 0.97665 to 0.98477, saving model to bilstm_cnn.h5\n",
            "270/270 [==============================] - 12s 43ms/step - loss: 0.0635 - acc: 0.9820 - val_loss: 0.0521 - val_acc: 0.9848\n",
            "Epoch 3/15\n",
            "270/270 [==============================] - ETA: 0s - loss: 0.0453 - acc: 0.9867\n",
            "Epoch 00003: val_acc improved from 0.98477 to 0.98625, saving model to bilstm_cnn.h5\n",
            "270/270 [==============================] - 12s 44ms/step - loss: 0.0453 - acc: 0.9867 - val_loss: 0.0454 - val_acc: 0.9862\n",
            "Epoch 4/15\n",
            "270/270 [==============================] - ETA: 0s - loss: 0.0381 - acc: 0.9886\n",
            "Epoch 00004: val_acc did not improve from 0.98625\n",
            "270/270 [==============================] - 12s 45ms/step - loss: 0.0381 - acc: 0.9886 - val_loss: 0.0451 - val_acc: 0.9861\n",
            "Epoch 5/15\n",
            "270/270 [==============================] - ETA: 0s - loss: 0.0337 - acc: 0.9897\n",
            "Epoch 00005: val_acc improved from 0.98625 to 0.98696, saving model to bilstm_cnn.h5\n",
            "270/270 [==============================] - 12s 44ms/step - loss: 0.0337 - acc: 0.9897 - val_loss: 0.0419 - val_acc: 0.9870\n",
            "Epoch 6/15\n",
            "269/270 [============================>.] - ETA: 0s - loss: 0.0306 - acc: 0.9905\n",
            "Epoch 00006: val_acc improved from 0.98696 to 0.98718, saving model to bilstm_cnn.h5\n",
            "270/270 [==============================] - 12s 43ms/step - loss: 0.0306 - acc: 0.9905 - val_loss: 0.0415 - val_acc: 0.9872\n",
            "Epoch 7/15\n",
            "269/270 [============================>.] - ETA: 0s - loss: 0.0280 - acc: 0.9912\n",
            "Epoch 00007: val_acc did not improve from 0.98718\n",
            "270/270 [==============================] - 12s 43ms/step - loss: 0.0281 - acc: 0.9912 - val_loss: 0.0428 - val_acc: 0.9869\n",
            "Epoch 8/15\n",
            "270/270 [==============================] - ETA: 0s - loss: 0.0263 - acc: 0.9917\n",
            "Epoch 00008: val_acc improved from 0.98718 to 0.98749, saving model to bilstm_cnn.h5\n",
            "270/270 [==============================] - 12s 45ms/step - loss: 0.0263 - acc: 0.9917 - val_loss: 0.0409 - val_acc: 0.9875\n",
            "Epoch 9/15\n",
            "269/270 [============================>.] - ETA: 0s - loss: 0.0247 - acc: 0.9921\n",
            "Epoch 00009: val_acc did not improve from 0.98749\n",
            "270/270 [==============================] - 11s 42ms/step - loss: 0.0247 - acc: 0.9921 - val_loss: 0.0417 - val_acc: 0.9875\n",
            "Epoch 10/15\n",
            "270/270 [==============================] - ETA: 0s - loss: 0.0231 - acc: 0.9926\n",
            "Epoch 00010: val_acc improved from 0.98749 to 0.98749, saving model to bilstm_cnn.h5\n",
            "270/270 [==============================] - 12s 44ms/step - loss: 0.0231 - acc: 0.9926 - val_loss: 0.0420 - val_acc: 0.9875\n",
            "Epoch 11/15\n",
            "269/270 [============================>.] - ETA: 0s - loss: 0.0218 - acc: 0.9929\n",
            "Epoch 00011: val_acc did not improve from 0.98749\n",
            "270/270 [==============================] - 11s 42ms/step - loss: 0.0218 - acc: 0.9929 - val_loss: 0.0441 - val_acc: 0.9873\n",
            "Epoch 12/15\n",
            "269/270 [============================>.] - ETA: 0s - loss: 0.0207 - acc: 0.9932\n",
            "Epoch 00012: val_acc did not improve from 0.98749\n",
            "270/270 [==============================] - 11s 42ms/step - loss: 0.0207 - acc: 0.9932 - val_loss: 0.0441 - val_acc: 0.9869\n",
            "Epoch 00012: early stopping\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r6BxnS62znhV"
      },
      "source": [
        "model = load_model('bilstm_cnn.h5')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZBNL0dX-C6Bd",
        "outputId": "43cc52cc-40ba-4e53-c609-fd1942290217"
      },
      "source": [
        "i = 13 # 확인하고 싶은 테스트용 샘플의 인덱스.\n",
        "\n",
        "# 입력한 테스트용 샘플에 대해서 예측 y를 리턴\n",
        "y_predicted = model.predict([np.array([X_test[i]]), np.array([X_char_test[i]])])\n",
        "y_predicted = np.argmax(y_predicted, axis=-1) # 확률 벡터를 정수 인코딩으로 변경함.\n",
        "true = np.argmax(y_test[i], -1) # 원-핫 인코딩을 다시 정수 인코딩으로 변경함.\n",
        "\n",
        "print(\"{:15}|{:5}|{}\".format(\"단어\", \"실제값\", \"예측값\"))\n",
        "print(35 * \"-\")\n",
        "\n",
        "for word, tag, pred in zip(X_test[i], true, y_predicted[0]):\n",
        "    if word != 0: # PAD값은 제외함.\n",
        "        print(\"{:17}: {:7} {}\".format(index_to_word[word], index_to_ner[tag], index_to_ner[pred]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "단어             |실제값  |예측값\n",
            "-----------------------------------\n",
            "the              : O       O\n",
            "statement        : O       O\n",
            "came             : O       O\n",
            "as               : O       O\n",
            "u.n.             : B-org   B-org\n",
            "secretary-general: I-org   I-org\n",
            "kofi             : B-per   B-per\n",
            "annan            : I-per   I-per\n",
            "met              : O       O\n",
            "with             : O       O\n",
            "officials        : O       O\n",
            "in               : O       O\n",
            "amman            : B-geo   B-geo\n",
            "to               : O       O\n",
            "discuss          : O       O\n",
            "wednesday        : B-tim   B-tim\n",
            "'s               : O       O\n",
            "attacks          : O       O\n",
            ".                : O       O\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "alMyDhYJDhU5",
        "outputId": "6095dfd7-2206-4f2f-ab38-66181b1e8d98"
      },
      "source": [
        "epochs = range(1, len(history.history['val_loss']) + 1)\n",
        "plt.plot(epochs, history.history['loss'])\n",
        "plt.plot(epochs, history.history['val_loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'val'], loc='upper left')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3xdZZ3v8c8vOzvZSXPrJW1C70JLUkDLoSAO2nFQsIICM1yKAqKDomdExVGO5XgdRs8wxzmjoowDKoqKXASRjsIUUaAooC1YoaWlN0qb3pJe0ibNPfmdP9ZKupOmbXazV3f2zvf9eu1X1nrWZT+rlHz7PM9azzJ3R0REZKjyMl0BERHJLgoOERFJiYJDRERSouAQEZGUKDhERCQlCg4REUmJgkMkQmb2IzP76hD33WRm7xzueUSipuAQEZGUKDhERCQlCg4Z9cIuopvM7CUzO2BmPzCzSWb2mJk1mdkTZjY2af+LzGyVmTWa2VNmVpu07XQzezE87n4gMeC73mNmK8JjnzWzNx5jnT9iZuvNbI+ZLTazE8JyM7NvmFm9me03s5fN7NRw2wVm9kpYt61m9tlj+gOTUU/BIRK4FDgPmA28F3gM+N9AJcH/J58EMLPZwL3AjeG2R4H/MrMCMysAfgn8BBgH/Dw8L+GxpwN3AR8FxgN3AIvNrDCViprZucC/AFcA1cDrwH3h5vOB+eF1lIf77A63/QD4qLuXAqcCv0vle0V6KThEAt92953uvhV4Bviju//Z3duAh4HTw/0WAr9299+4eyfwb0AR8FfA2UAc+Ka7d7r7g8CypO+4HrjD3f/o7t3ufjfQHh6XiquAu9z9RXdvB24G3mJmM4BOoBSoAczdV7v79vC4TmCOmZW5+153fzHF7xUBFBwivXYmLbcOsl4SLp9A8C98ANy9B9gCTA63bfX+M4e+nrQ8HfhM2E3VaGaNwNTwuFQMrEMzQatisrv/DvgOcDtQb2Z3mllZuOulwAXA62b2tJm9JcXvFQEUHCKp2kYQAEAwpkDwy38rsB2YHJb1mpa0vAX4mrtXJH2K3f3eYdZhDEHX11YAd7/N3c8A5hB0Wd0Uli9z94uBiQRdag+k+L0igIJDJFUPABea2TvMLA58hqC76VngOaAL+KSZxc3s74Czko79HvAxM3tzOIg9xswuNLPSFOtwL/AhM5sbjo/8H4KutU1mdmZ4/jhwAGgDesIxmKvMrDzsYtsP9Azjz0FGMQWHSArc/VXgauDbwC6CgfT3unuHu3cAfwd8ENhDMB7yi6RjlwMfIehK2gusD/dNtQ5PAF8EHiJo5ZwIXBluLiMIqL0E3Vm7ga+H264BNpnZfuBjBGMlIikzvchJRERSoRaHiIikRMEhIiIpUXCIiEhKFBwiIpKS/ExX4HiYMGGCz5gxI9PVEBHJKi+88MIud68cWD4qgmPGjBksX74809UQEckqZvb6YOXqqhIRkZQoOEREJCUKDhERScmoGOMYTGdnJ3V1dbS1tWW6KpFKJBJMmTKFeDye6aqISI4YtcFRV1dHaWkpM2bMoP9kprnD3dm9ezd1dXXMnDkz09URkRwxaruq2traGD9+fM6GBoCZMX78+JxvVYnI8TVqgwPI6dDoNRquUUSOr1EdHEfT2NLB7ub2TFdDRGREUXAcwb7WTnY1d0Ry7sbGRv7jP/4j5eMuuOACGhsbI6iRiMjQKDiOIBGP0d7VTXdP+t9Zcrjg6OrqOuJxjz76KBUVFWmvj4jIUI3au6qGIhGPAdDe2U1xYXr/qBYtWsSGDRuYO3cu8XicRCLB2LFjWbNmDWvXruWSSy5hy5YttLW18alPfYrrr78eODh9SnNzM+9+97t561vfyrPPPsvkyZN55JFHKCoqSms9RUQGUnAA//Rfq3hl2/5Dyt2dlo5uCvPzyI+l1jibc0IZX37vKYfdfuutt7Jy5UpWrFjBU089xYUXXsjKlSv7bpu96667GDduHK2trZx55plceumljB8/vt851q1bx7333sv3vvc9rrjiCh566CGuvvrqlOopIpIqBccRmBlmEEFP1SHOOuusfs9a3HbbbTz88MMAbNmyhXXr1h0SHDNnzmTu3LkAnHHGGWzatCn6iorIqBdpcJjZAuBbQAz4vrvfOmD7PwIfBrqABuDv3f31cNu1wBfCXb/q7neH5WcAPwKKgEeBT/kwX5x+pJbBhvpmAE6cWDKcrziqMWPG9C0/9dRTPPHEEzz33HMUFxfz9re/fdBnMQoLC/uWY7EYra2tkdZRRAQiHBw3sxhwO/BuYA7wPjObM2C3PwPz3P2NwIPA/w2PHQd8GXgzcBbwZTMbGx7zXeAjwKzwsyCqawBIxPNo6+xmmNl0iNLSUpqamgbdtm/fPsaOHUtxcTFr1qzh+eefT+t3i4gMR5R3VZ0FrHf3je7eAdwHXJy8g7s/6e4t4erzwJRw+V3Ab9x9j7vvBX4DLDCzaqDM3Z8PWxk/Bi6J8BpIxGN0u9PZ3ZPW844fP55zzjmHU089lZtuuqnftgULFtDV1UVtbS2LFi3i7LPPTut3i4gMR5RdVZOBLUnrdQQtiMO5DnjsCMdODj91g5QfwsyuB64HmDZtWir17qf3zqq2zh4K8mPHfJ7B/OxnPxu0vLCwkMcee2zQbb3jGBMmTGDlypV95Z/97GfTWjcRkcMZEc9xmNnVwDzg6+k6p7vf6e7z3H1eZeUhbz4cst7gaO3sTlfVRESyWpTBsRWYmrQ+JSzrx8zeCXweuMjd249y7FYOdmcd9pzpFMszCvKDcQ4REYk2OJYBs8xsppkVAFcCi5N3MLPTgTsIQqM+adMS4HwzGxsOip8PLHH37cB+Mzvbgtn7PgA8EuE1AFAUj9HWmd4xDhGRbBXZGIe7d5nZDQQhEAPucvdVZnYLsNzdFxN0TZUAPw9ncd3s7he5+x4z+2eC8AG4xd33hMv/wMHbcR/j4LhIZBLxGPtaO+nucWJ5mm1WREa3SJ/jcPdHCZ61SC77UtLyO49w7F3AXYOULwdOTWM1jyrKqUdERLLNiBgcH+kS8eCPqbVL4xwiIgqOISiI5REzy+g4R0lJtE+ui4gMlYJjCMyMwniMtg61OERE1GE/REXxPBpbO3H3tLyOddGiRUydOpWPf/zjAHzlK18hPz+fJ598kr1799LZ2clXv/pVLr744qOcSUTk+FJwADy2CHa8fMRdJnb3UN7VgxfEhhYcVafBu2897OaFCxdy44039gXHAw88wJIlS/jkJz9JWVkZu3bt4uyzz+aiiy7Se8NFZERRcAxRXngbbo87eWn4RX766adTX1/Ptm3baGhoYOzYsVRVVfHpT3+apUuXkpeXx9atW9m5cydVVVXD/j4RkXRRcMARWwZ9epyN2/YxqSzBpLJEWr728ssv58EHH2THjh0sXLiQe+65h4aGBl544QXi8TgzZswYdDp1EZFMUnAMURRTjyxcuJCPfOQj7Nq1i6effpoHHniAiRMnEo/HefLJJ3n99dfT9l0iIumi4EhBuqceOeWUU2hqamLy5MlUV1dz1VVX8d73vpfTTjuNefPmUVNTk7bvEhFJFwVHCnqnHunp8b4xj+F6+eWDg/ITJkzgueeeG3S/5ubmtHyfiMhw6TmOFPS9m0NPkIvIKKbgSEHv1COaYl1ERrNRHRypvke8IJZHnhmtWTTFerrflS4iMmqDI5FIsHv37pR+sZoZiXgsa1oc7s7u3btJJNJz+7CICIziwfEpU6ZQV1dHQ0NDSsc1tnTQ0tFNx66iiGqWXolEgilTphx9RxGRIRq1wRGPx5k5c2bKx/3k+df54iMr+cOic5lckR3hISKSTqO2q+pYzakuBWD1tv0ZromISGZEGhxmtsDMXjWz9Wa2aJDt883sRTPrMrPLksr/xsxWJH3azOyScNuPzOy1pG1zo7yGgU6uKgNgzQ4Fh4iMTpF1VZlZDLgdOA+oA5aZ2WJ3fyVpt83AB4HPJh/r7k8Cc8PzjAPWA48n7XKTuz8YVd2PpKQwn2njilm9vSkTXy8iknFRjnGcBax3940AZnYfcDHQFxzuvincdqT7Wy8DHnP3luiqmpqaqlJWq8UhIqNUlF1Vk4EtSet1YVmqrgTuHVD2NTN7ycy+YWaFgx1kZteb2XIzW57qnVNHU1tdxqZdB2jVGwFFZBQa0YPjZlYNnAYsSSq+GagBzgTGAZ8b7Fh3v9Pd57n7vMrKyrTWq7a6lB6HtTvVXSUio0+UwbEVmJq0PiUsS8UVwMPu3tlb4O7bPdAO/JCgS+y4qq0OBshXb1d3lYiMPlEGxzJglpnNNLMCgi6nxSme430M6KYKWyFY8D7VS4CVaahrSqaOLWZMQYw1O9TiEJHRJ7LgcPcu4AaCbqbVwAPuvsrMbjGziwDM7EwzqwMuB+4ws1W9x5vZDIIWy9MDTn2Pmb0MvAxMAL4a1TUcTl6ecXJVKa+oxSEio1CkT467+6PAowPKvpS0vIygC2uwYzcxyGC6u5+b3loem5rqMn71l224O5aGd5CLiGSLET04PpLVVpexv62Lbfv0TnARGV0UHMeotiqYemSNuqtEZJRRcByjk8Pg0J1VIjLaKDiOUWkiztRxRazWnVUiMsooOIahtqpMLQ4RGXUUHMNQo6lHRGQUUnAMw5xw6pF19equEpHRQ8ExDDVVmnpEREYfBccwTBtXTHFBTO/mEJFRRcExDL1Tj6jFISKjiYJjmGqry1izowl3z3RVRESOCwXHMNVWlbKvtZPtmnpEREYJBccw9b6bY41eJSsio4SCY5gOTj2iAXIRGR0UHMPUN/WIBshFZJRQcKRBjaYeEZFRRMGRBrXVZby26wBtnZp6RERyn4IjDWqrgqlH1u7UOIeI5L5Ig8PMFpjZq2a23swWDbJ9vpm9aGZdZnbZgG3dZrYi/CxOKp9pZn8Mz3m/mRVEeQ1D0XdnlQbIRWQUiCw4zCwG3A68G5gDvM/M5gzYbTPwQeBng5yi1d3nhp+Lksr/FfiGu58E7AWuS3vlU9Q79cgrGucQkVEgyhbHWcB6d9/o7h3AfcDFyTu4+yZ3fwnoGcoJzcyAc4EHw6K7gUvSV+Vj0zv1iJ7lEJHRIMrgmAxsSVqvC8uGKmFmy83seTPrDYfxQKO7dx3tnGZ2fXj88oaGhlTrnrLgzipNPSIiuW8kD45Pd/d5wPuBb5rZiakc7O53uvs8d59XWVkZTQ2TzKkOph7ZsV9Tj4hIbosyOLYCU5PWp4RlQ+LuW8OfG4GngNOB3UCFmeUfyzmjVFOtd3OIyOgQZXAsA2aFd0EVAFcCi49yDABmNtbMCsPlCcA5wCse9AM9CfTegXUt8Ejaa34MNPWIiIwWkQVHOA5xA7AEWA084O6rzOwWM7sIwMzONLM64HLgDjNbFR5eCyw3s78QBMWt7v5KuO1zwD+a2XqCMY8fRHUNqShLxJkyVlOPiEjuyz/6LsfO3R8FHh1Q9qWk5WUE3U0Dj3sWOO0w59xIcMfWiNP7bg4RkVw2kgfHs05tVSkbG5o19YiI5DQFRxrVVpfR47BuZ3OmqyIiEhkFRxrpzioRGQ0UHGk0bVwxRfEYq/UEuYjkMAVHGsXCqUfU4hCRXKbgSLPa6lLW7NDUIyKSuxQcaVZbXUZji6YeEZHcpeBIs5oqvZtDRHKbgiPNaqqDqUf0bg4RyVUKjjQrS8SZXFGkJ8hFJGcpOCJQW13GGrU4RCRHKTgiUFtdysZdBzT1iIjkJAVHBGqry+jucdbXa+oREck9Co4I1FRpgFxEcpeCIwLTx4+hKB7TLbkikpMUHBGI5RmzNfWIiOQoBUdE5lSXsmbHfk09IiI5J9LgMLMFZvaqma03s0WDbJ9vZi+aWZeZXZZUPtfMnjOzVWb2kpktTNr2IzN7zcxWhJ+5UV7DsaqpKmNvSyc797dnuioiImkV2atjzSwG3A6cB9QBy8xscdK7wwE2Ax8EPjvg8BbgA+6+zsxOAF4wsyXu3hhuv8ndH4yq7ulQ2/tujh37qSpPZLg2IiLpE2WL4yxgvbtvdPcO4D7g4uQd3H2Tu78E9AwoX+vu68LlbUA9UBlhXdPu5PDOKo1ziEiuiTI4JgNbktbrwrKUmNlZQAGwIan4a2EX1jfMrPAwx11vZsvNbHlDQ0OqXzts5UXh1CO6s0pEcsyIHhw3s2rgJ8CH3L23VXIzUAOcCYwDPjfYse5+p7vPc/d5lZWZaazUVuvOKhHJPVEGx1ZgatL6lLBsSMysDPg18Hl3f7633N23e6Ad+CFBl9iIVFtdpqlHRCTnRBkcy4BZZjbTzAqAK4HFQzkw3P9h4McDB8HDVghmZsAlwMq01jqNaqo09YiI5J7IgsPdu4AbgCXAauABd19lZreY2UUAZnammdUBlwN3mNmq8PArgPnABwe57fYeM3sZeBmYAHw1qmsYrtpqDZCLSO6J7HZcAHd/FHh0QNmXkpaXEXRhDTzup8BPD3POc9NczchMHz+GRDyP1RogF5EcMqQWh5l9yszKLPCD8KG986OuXLaL5RknV5WxZodaHCKSO4baVfX37r4fOB8YC1wD3BpZrXJIbThnlaYeEZFcMdTgsPDnBcBP3H1VUpkcQW11MPVIfZOmHhGR3DDU4HjBzB4nCI4lZlbKgKe9ZXB6N4eI5JqhBsd1wCLgTHdvAeLAhyKrVQ6pCees0hPkIpIrhhocbwFedfdGM7sa+AKwL7pq5Y7eqUd0S66I5IqhBsd3gRYzexPwGYJ5o34cWa1yTG34bg4RkVww1ODo8uC2oIuB77j77UBpdNXKLTVVZWxo0NQjIpIbhhocTWZ2M8FtuL82szyCcQ4ZgtpqTT0iIrljqMGxEGgneJ5jB8HT3l+PrFY5pkZTj4hIDhlScIRhcQ9QbmbvAdrcXWMcQzQjnHpkzQ7dWSUi2W+oU45cAfyJYDLCK4A/Jr8jXI4slmecPEnv5hCR3DDUSQ4/T/AMRz2AmVUCTwAj+r3fI0ltdRlLVu3A3QlmhBcRyU5DHePI6w2N0O4UjhWCJ8g19YiI5IKhtjj+28yWAPeG6wsZMF26HFlt+AT56u37mVSWyHBtRESO3VAHx28C7gTeGH7udPdB3/Utg6up6g0ODZCLSHYb8ouc3P0h4KEI65LTyouDqUf0BLmIZLsjBoeZNQGDvUjCAHf3skhqlaNqqko12aGIZL0jdlW5e6m7lw3yKR1KaJjZAjN71czWm9miQbbPD98m2DXw9l4zu9bM1oWfa5PKzzCzl8Nz3mZZdItSbXUZGxqaae/S1CMikr0iuzPKzGLA7cC7gTnA+8xszoDdNgMfBH424NhxwJeBNwNnAV82s7Hh5u8CHwFmhZ8FEV1C2tVUl9KlqUdEJMtFeUvtWcB6d9/o7h3AfQSTJPZx903u/hKHvhTqXcBv3H2Pu+8FfgMsMLNqoMzdnw8nXfwxcEmE15BWB++sUneViGSvKINjMrAlab0uLBvOsZPD5aOe08yuN7PlZra8oaFhyJWOUt/UI3qCXESyWM4+xOfud7r7PHefV1lZmenqAElTj+jOKhHJYlEGx1ZgatL6lLBsOMduDZeP5ZwjQk1VGau3NxH0tImIZJ8og2MZMMvMZppZAXAlsHiIxy4BzjezseGg+PnAEnffDuw3s7PDu6k+ADwSReWjUlNdyp4DHTRo6hERyVKRBYe7dwE3EITAauABd19lZreY2UUAZnammdURzLp7h5mtCo/dA/wzQfgsA24JywD+Afg+sJ7gFbaPRXUNUegbINcU6yKSpYb85PixcPdHGTCnlbt/KWl5Gf27npL3uwu4a5Dy5cCp6a3p8VNbdXDOqr+ePTLGXkREUpGzg+MjVXlxnBPKE7qzSkSyloIjA2qqy/Qsh4hkLQVHBtRWl2rqERHJWgqODKipKtPUIyKStRQcGdB7Z5VmyhWRbKTgyIAZ44spzM9jtQbIRSQLKTgyID+Wx8lVpazRsxwikoUUHBlSU1XK6u37NfWIiGQdBceRtOyBXesiOXVtdRm7D3TQ0KypR0Qkuyg4Dscd7ns/3HMZtO5N++lrqvRuDhHJTgqOwzGD826BfVvhF9dDz8B3TQ1PbXUpgJ4gF5Gso+A4kqlnwYJ/gXWPw9P/mtZTVxQXUF2e0J1VIpJ1FBxHc+aH4U3vh6dvhVf/O62nrq0u051VIpJ1FBxHYwbv+XeoemPQZbV7Q9pOXVNVyvp6TT0iItlFwTEU8SJY+FPIy4P7r4aOA2k5bW11MPXIhvr0nE9E5HhQcAzV2Olw2V3QsAYWfyK462qYegfINc4hItlEwZGKE8+Fc78IKx+C5/9j2KebMX4Mhfl5rNmh4BCR7BFpcJjZAjN71czWm9miQbYXmtn94fY/mtmMsPwqM1uR9Okxs7nhtqfCc/ZumxjlNRzirZ+GmvfA41+E154Z1qnyY3nMnlSqZzlEJKtEFhxmFgNuB94NzAHeZ2ZzBux2HbDX3U8CvgH8K4C73+Puc919LnAN8Jq7r0g67qre7e5eH9U1DMoMLvkujHsD/PyDwXMew1BbXaoWh4hklShbHGcB6919o7t3APcBFw/Y52Lg7nD5QeAdZmYD9nlfeOzIkSiDK++BrjZ44BroOvZpQ2qqytjV3EF9U1saKygiEp0og2MysCVpvS4sG3Qfd+8C9gHjB+yzELh3QNkPw26qLw4SNACY2fVmttzMljc0NBzrNRxe5clBy2PrC/DY5475NHo3h4hkmxE9OG5mbwZa3H1lUvFV7n4a8Lbwc81gx7r7ne4+z93nVVZWRlPBORcFYx4v/BBe/PExnaJv6hF1V4lIlogyOLYCU5PWp4Rlg+5jZvlAObA7afuVDGhtuPvW8GcT8DOCLrHMOfeL8Ia3w68/G7Q+UnRw6hG1OEQkO0QZHMuAWWY208wKCEJg8YB9FgPXhsuXAb/z8AUVZpYHXEHS+IaZ5ZvZhHA5DrwHWEkm5cXg0rugZCLc/wE4sCvlU/S+m0NEJBtEFhzhmMUNwBJgNfCAu68ys1vM7KJwtx8A481sPfCPQPItu/OBLe6+MamsEFhiZi8BKwhaLN+L6hqGbMx4WPgTONAAD34IurtSOry2uowNDc10dKV3Bl4RkSjkR3lyd38UeHRA2ZeSltuAyw9z7FPA2QPKDgBnpL2i6XDC6fCeb8Aj/wC//Sc4/5+HfGhNdRmd3c6Ghua+wXIRkZFqRA+OZ53Tr4J518Gzt8Gqh4d82BxNPSIiWUTBkW4LboUpZ8IvPw71a4Z0yIzxYyjIz9MU6yKSFRQc6ZZfAFf8GArGwP1XQdu+ox8Sy+PkSRogF5HsoOCIQtkJcPmPYM9r8PD/HNJrZ4M7q9TiEJGRT8ERlRnnwLu+Bq/+Gn7//466e211Gbua22loOvbpS0REjgcFR5Te/DE47XL43ddg/RNH3LVGT5CLSJZQcETJDN77LZh0Cjx4HezddNhd51SXkWfwrSfWsbWx9fjVUUQkRQqOqBWMCR4OxMPXzrYMultFcQH/74o3sXr7fhZ8cym//PNWPA1vGRQRSTcFx/Ew7g1w6Q9gx0r41acP+9rZvz19Co99aj4nTyrlxvtXcMO9f6axpeM4V1ZE5MgUHMfLrPPg7TfDS/fBnw4/S8q08cXc/9G3cNO7TmbJyh2865tLeWZdBNPCi4gcIwXH8TT/Jpi9AJbcDJufP+xusTzj439zEr/8+DmUJuJc84M/8ZXFq2jr7D6OlRURGZyC43jKy4O/vQMqpsEDH4CmHUfc/dTJ5fzqE2/lQ+fM4EfPbuLC255h5dajP1AoIhIlBcfxVlQBC++B9iZ44FroOvIYRiIe48vvPYWfXHcWze1dXHL7H7j9yfV092jgXEQyQ8GRCZPmwMXfgS3Pw+OfH9Ihb5tVyZIb57Pg1Cq+vuRVrrjjOTbvHvwOLRGRKCk4MuXUS+EtN8Cf7oQVA1+pPriK4gK+/b7T+daVc1m7s4l3f2spDyzbott2ReS4UnBk0jv/CWa8DX51I2z/y5AOMTMunjuZ/75xPm+cUsH/euglrv/JC+xq1lQlInJ8KDgyKZYPl/0QisYFDwe27BnyoZMrirjnw2/mCxfW8vSrDSz45lJ+u3pnhJUVEQkoODKtpDJ4srxpBzz0YegZ+i23eXnGh9/2BhZ/4hwmlBRy3d3LufkXL3OgPbVX14qIpCLS4DCzBWb2qpmtN7NFg2wvNLP7w+1/NLMZYfkMM2s1sxXh5z+TjjnDzF4Oj7nNzCzKazgupsyDC74OG34L33wjPPwxWPEzaNwypMNrqsp45IZz+Oj8N3Dfss1ceNszvLh5b8SVFpHRyqIaWDWzGLAWOA+oA5YB73P3V5L2+Qfgje7+MTO7Evhbd18YBsiv3P3UQc77J+CTwB8J3md+m7s/dqS6zJs3z5cvX56eC4vSSz+HNf8Fm34PLbuDsrEzYeb84DPjbVA66YineH7jbj7zwF/Ysb+Nj//NSXzi3JOIx9SwFJHUmdkL7j7vkPIIg+MtwFfc/V3h+s0A7v4vSfssCfd5zszygR1AJTCdQYLDzKqBJ929Jlx/H/B2d//okeqSNcHRq6cH6l+BTc/Aa0th0x+gPXzwr7ImCJCZ82HGW6F43CGH72/r5CuLV/GLF7fypinl/PvCuZxYWXKcL0JEst3hgiM/wu+cDCT3tdQBbz7cPu7eZWb7gPHhtplm9mdgP/AFd38m3L9uwDknD/blZnY9cD3AtGnThnclx1teHlSdGnzO/p/BuMf2v4Qh8kzQjbXse4AF+8wIWyTT/woSZZQl4vz7FXN5Z+0k/vfDL3Phbc/w+Qtqufrs6eRCz56IZFaUwTEc24Fp7r7bzM4Afmlmp6RyAne/E7gTghZHBHU8fvJiMPl/BJ+33gjdnbD1xTBIlsKy78Pzt4PF4IS5fd1aF5x8NmfcOJ+bHnyJLz6yiidW1/P1y97IxLJEpq9IRLJYlMGxFZiatD4lLBtsn7qwq6oc2O1B/1k7gLu/YGYbgNnh/lOOcs7cF4vDtDcHn7++CTrboO5P8FrYtfXst+H334C8OJOmnMndM97K45NO5jPP7eBd32zkX/7uNBacWp3pqxCRLBVlcCwDZpnZTIJf7lcC7x+wz30caqIAABE2SURBVGLgWuA54DLgd+7uZlYJ7HH3bjN7AzAL2Ojue8xsv5mdTTA4/gHg2xFeQ3aIJw4OoPN5aG8OpjMJg8Se+Tfe5T2cV5jgpbyTefzek1k76+186PK/pXRMcaZrLyJZJrLBcQAzuwD4JhAD7nL3r5nZLcByd19sZgngJ8DpwB7gSnffaGaXArcAnUAP8GV3/6/wnPOAHwFFwGPAJ/woF5F1g+Pp1rYPXn8WXluKv7YU27kSgC7ycItjsRixvDwsLwaWF36SlvNiwWtwBy3PO8q2pE9haTAzcMU0KJ8aLk+FRHmG/4BEZDDH/a6qkWTUB8dAB3azYfkSVr2wlJ2NzXhPD3FzqsoKmFJRyOTyQsYW5WPeDd4TvLGwd7mnt6wnLPPDlIfHJZe37oV9W6CrrX99EuVQPu1gqFRM7R8uRWODcBLJBl0dwe30LbvgwK5g+cCuYL1ldzBGSfh7t+/Xrye9GbR3mw9tW9/6YbYtuBXKTjimS8nEXVUyUo0Zz4l//X5O/Ov309bZzbJNe1i6toH71u7i1Q1NAEwoKWT+rAm8bfYE3npSJZWlhen5bnc40BA83Nj4ehAkjZuD9T0bYeNT0Hmg/zEFJYO3VCqmBYEzZkLuBos7dHcE3Y8dTeHPA0ELrmhscDt2ojxo4Uk0OlsP/uI/kBwIgwTDgd0Hb50/hAX/zfLDm1P6/s5a0rr1Kxp822DHheuDbetK/zx2anFIPzv3t7F0bQPPrNvF79fvYs+B4H0hc6rLmD+7kvmzJnDGjLEU5kf0i8o9aJk0vh6Gy+b+4dL4OrTv739MftHBIEkOl4IxkJcf/FLNy0/6DFwf4j5DCSf3oEU18Bd9R3PwDpa+5ebg52GXD4T7N0PP0aaQsSA8eoOkaGww/1nyctFYKB7bfz1RPvICt6cnCErvDlur3UFZ8npyC7dvn+SfPYfZ1jP4vh0HDh8CLbug8zCvL8jLh+IJwT9ciseHP4+wXjQ26wJeXVUKjpT19Dgrt+3jmXW7eHptAy++vpeuHqcoHuMtJ47nbbMmMH92JW+YMOb4Ph/S2jggTDbDvs3h+uYgeKJgeYcPF+xgKPgQ5xuLFQStqYISKAx/FowJl0uTlgfuU3Kw6691L7TuCX627Bmw3HiEf/0SjEkVjT00cPpCZtzB7RD8y7W7PeiK6W4PArJvOfx0d4TLbQeXuzsG7Ju83NF/357OYf9nOmb5ifAX/fikX/iDrYdBMBKDN80UHAqOYWtu7+K5DbvDFkkDm8IXSU2uKGL+7AnMn1XJX504gfLieGYr2t4E++qCLoae7uBf7H2f3n9tDig74voQ9vGeg7/UC8YENwIcLQjyC6L/s+jugrbGQULlcIEThtHA7sKhyMuHWCHkh59YQdJyclkiuPZYYfAzP3FwOVYY3G7eG8wWO3ijRe968vLAbf32yetf1vczr/954kVBIBSMyfkgSJWCQ8GRdpt3t7B0XQNL1zbw7IbdNLd3kWcwd2oFb5tVyfzZlbxpSjn5misr+3S2JQXO3uCXbN8v++QwSCrLsm4YOToFh4IjUp3dPazY0sjStQ0sXbeLl+oacYeyRD7nnBR0aZ01cxzTxxUrSESyhIJDwXFc7T3QwR827AqCZO0uduwPbsEtyM/jDRPGMHtSKbMnlTBrUimzJ5UybVwxsTx1E4iMJAoOBUfGuDvr65tZsaWRdfXNrN3ZxLqdzWxtbO3bpzA/jxMrS/qFyexJJUwdW0yeAkUkI/Qch2SMmTFrUimzJpX2K29u72JdGCJrdzaxtr6ZP762h1+u2Na3TyKex0kTS5g9sTQMlBJmTyplckWRAkUkQxQckjElhfmcPm0sp08b2698f1sn63Y2s25nE2t3NrOuvolnN+zmF38+OJ9lcUGMkyaWMGviwTCZNamEyRVFmjpeJGIKDhlxyhJxzpg+ljOm9w+Ufa2drK8PwqS3u+uZdQ089OLBV7SMKYhx0qRSZk8sYfr4Yk6oKAo+5UVUlScoyNfAvMhwKTgka5QXxTlj+jjOmN7/rYeNLR39xk7W7mziqbUNNDT1n2rBDCpLCjmhoojJFUWcUJHoC5bJ4c+xxXG1WESOQsEhWa+iuIAzZ4zjzBn9A6Wts5vt+9rY1tjK1sZWtvV92li9fT9PrN5Je1dPv2MS8byDQVIetlYqEn3BUlWeIBHX8woyuik4JGcl4jFmThjDzAljBt3u7uw50MG2xrb+wbKvla2NbazZUX9IqwWCCSAnJ7VWqsuD5UllhVSWJJhYVqhwkZym4JBRy8wYX1LI+JJCTpsy+DtB2ru62bGvjW2NbYcEy7r6Zp56tYHWzkPnpiotzKeyrJDKkkImliXCn4VMLC2ksrSQiaUJKksL1TUmWUnBIXIEhfkxpo8fw/Txh2+17GvtZFtjG/VNbdQ3tdOQ9KlvauPlukbqm9pp6Tg0YOIxY0LJwUCpLE0khUv4syzBhJKC6GYkFkmRgkNkGMyMiuICKooLmEPZEfdtbu8KwmR/Gw3N7dTvb+/3s25vK3/e3Mielg4Gey63ojje13IZP6aQiuI4FUVxyosLqCiKU1Ecp7zvZwHlRXHdRSaRiDQ4zGwB8C2CV8d+391vHbC9EPgxcAawG1jo7pvM7DzgVqAA6ABucvffhcc8BVQDvY8dn+/u9VFeh0g6lBTmU1KYf9gxl16d3T3sbu7oa7EEPw+2YOqb2qnb28i+1k72tXYOGjK9xhTEqCgOQqQ3VHqDpTd4KorjlBXFqegtK45TFI+pC00OK7LgMLMYcDtwHlAHLDOzxe7+StJu1wF73f0kM7sS+FdgIbALeK+7bzOzU4ElwOSk465yd80hIjkpHsujqjxBVXkCOPL72Ht6nKa2LhpbO2hs6aSxtZPGlg72tXbS2NKZ9DPYvq6+uW+9s/vwiVMQy6O8t0UTfkoT+ZQVxSlLxCkryqc0cXC5LHFwe2kiX91qOS7KFsdZwHp33whgZvcBFwPJwXEx8JVw+UHgO2Zm7v7npH1WAUVmVuju6X8HokgWy8szyovjlBfHmT5+6Me5Oy0d3X3B0tjawb6+4Dm4vq+1k70tHWzf18ba+k72t3bR1NZJz1GmuCvMzwtDpjdMkpfzw8AJy5LCp3e7WjwjW5TBMRnYkrReB7z5cPu4e5eZ7QPGE7Q4el0KvDggNH5oZt3AQ8BXfZCZGs3seuB6gGnTpg3zUkRyi5kxpjCfMYX5nFBRlNKx7s6Bjm72t3ayv62TprauQZaDgNnf2sX+tiCA6va0sD8s6+juOeJ35OdZvxZMaWH/Vk7pwBAasF6ayNf0/REa0YPjZnYKQffV+UnFV7n7VjMrJQiOawjGSfpx9zuBOyGYHfc4VFdkVDCzvvGaE0gtdHq1dXb3hUhTWxA0+1uD4NnX2klTGEK925raOtm0q6Vvvbn9aO9hD+YzSw6S0qQWTb/wCQN0TEGsL0xLCvMpLowxpiBf0/0PIsrg2ApMTVqfEpYNtk+dmeUTdOjuBjCzKcDDwAfcfUPvAe6+NfzZZGY/I+gSOyQ4RGTkSsRjJOIxJpYefd/BdPc4zW1Baya5pdMUljUltXia2oOfjS0dbN7T0ld+tFZPr6J4jDGFYagU5B9cTgqbkkHCp3ffIITyKQnXc6ElFGVwLANmmdlMgoC4Enj/gH0WA9cCzwGXAb9zdzezCuDXwCJ3/0PvzmG4VLj7LjOLA+8BnojwGkRkBIolje0cq95WT3NbFwfauznQ0cWB9qA109LR3W+5uT3YdqA9KN9zoIMte1r61g90dB113KdXQSyPooIYxQUxiuKxg8sF+RSH60UFMYrjB8uL4nkUF+Qn7RscW1yQ37deXBAjkR87Lq8biCw4wjGLGwjuiIoBd7n7KjO7BVju7ouBHwA/MbP1wB6CcAG4ATgJ+JKZfSksOx84ACwJQyNGEBrfi+oaRCR3DbfVk8zdaevsORgwHV39QiUIoWC9paObts5uWjqC5daO7r4bFXbua6Ols6uvrLWz+4i3Ww9+XWHIhMHzvQ/MY8ZRbgFPld4AKCIyQvUGUmsYNL2BEoRKF60dPUF5Z1J50nprRzf/dNEpTCxLHNP36w2AIiJZxsz6uq7GjSnIdHX6ZP8ojYiIHFcKDhERSYmCQ0REUqLgEBGRlCg4REQkJQoOERFJiYJDRERSouAQEZGUjIonx82sAXg90/UYogn0n1Y+l+jaslcuX5+u7fCmu3vlwMJRERzZxMyWD/aIfy7QtWWvXL4+XVvq1FUlIiIpUXCIiEhKFBwjz52ZrkCEdG3ZK5evT9eWIo1xiIhIStTiEBGRlCg4REQkJQqOEcDMpprZk2b2ipmtMrNPZbpO6WZmMTP7s5n9KtN1STczqzCzB81sjZmtNrO3ZLpO6WJmnw7/Tq40s3vN7NheJTdCmNldZlZvZiuTysaZ2W/MbF34c2wm63isDnNtXw//Xr5kZg+bWUU6vkvBMTJ0AZ9x9znA2cDHzWxOhuuUbp8CVme6EhH5FvDf7l4DvIkcuU4zmwx8Epjn7qcCMeDKzNZq2H4ELBhQtgj4rbvPAn4brmejH3Hotf0GONXd3wisBW5OxxcpOEYAd9/u7i+Gy00Ev3gmZ7ZW6WNmU4ALge9nui7pZmblwHzgBwDu3uHujZmtVVrlA0Vmlg8UA9syXJ9hcfelwJ4BxRcDd4fLdwOXHNdKpclg1+buj7t7V7j6PDAlHd+l4BhhzGwGcDrwx8zWJK2+CfwvoCfTFYnATKAB+GHYFfd9MxuT6Uqlg7tvBf4N2AxsB/a5++OZrVUkJrn79nB5BzApk5WJ0N8Dj6XjRAqOEcTMSoCHgBvdfX+m65MOZvYeoN7dX8h0XSKSD/wP4LvufjpwgOzt6ugn7Ou/mCAcTwDGmNnVma1VtDx4PiHnnlEws88TdInfk47zKThGCDOLE4TGPe7+i0zXJ43OAS4ys03AfcC5ZvbTzFYpreqAOnfvbSE+SBAkueCdwGvu3uDuncAvgL/KcJ2isNPMqgHCn/UZrk9amdkHgfcAV3maHtxTcIwAZmYEfeSr3f3fM12fdHL3m919irvPIBhY/Z2758y/Wt19B7DFzE4Oi94BvJLBKqXTZuBsMysO/46+gxwZ+B9gMXBtuHwt8EgG65JWZraAoJv4IndvSdd5FRwjwznANQT/Gl8Rfi7IdKVkyD4B3GNmLwFzgf+T4fqkRdiKehB4EXiZ4PdFVk/PYWb3As8BJ5tZnZldB9wKnGdm6whaWbdmso7H6jDX9h2gFPhN+HvlP9PyXZpyREREUqEWh4iIpETBISIiKVFwiIhIShQcIiKSEgWHiIikRMEhMsKZ2dtzcVZhyV4KDhERSYmCQyRNzOxqM/tT+KDVHeE7SJrN7BvhOy1+a2aV4b5zzez5pPckjA3LTzKzJ8zsL2b2opmdGJ6+JOmdH/eET3KLZISCQyQNzKwWWAic4+5zgW7gKmAMsNzdTwGeBr4cHvJj4HPhexJeTiq/B7jd3d9EMC9U76ytpwM3AnOANxDMNiCSEfmZroBIjngHcAawLGwMFBFMltcD3B/u81PgF+E7PCrc/emw/G7g52ZWCkx294cB3L0NIDzfn9y9LlxfAcwAfh/9ZYkcSsEhkh4G3O3u/d6wZmZfHLDfsc7x05603I3+35UMUleVSHr8FrjMzCZC33uspxP8P3ZZuM/7gd+7+z5gr5m9LSy/Bng6fPtjnZldEp6j0MyKj+tViAyB/tUikgbu/oqZfQF43MzygE7g4wQvdjor3FZPMA4CwfTd/xkGw0bgQ2H5NcAdZnZLeI7Lj+NliAyJZscViZCZNbt7SabrIZJO6qoSEZGUqMUhIiIpUYtDRERSouAQEZGUKDhERCQlCg4REUmJgkNERFLy/wF5QuN5AXfg9AAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wFKpmE9cQIdW"
      },
      "source": [
        "def sequences_to_tag(sequences): # 예측값을 index_to_ner를 사용하여 태깅 정보로 변경하는 함수.\n",
        "    result = []\n",
        "    for sequence in sequences: # 전체 시퀀스로부터 시퀀스를 하나씩 꺼낸다.\n",
        "        temp = []\n",
        "        for pred in sequence: # 시퀀스로부터 예측값을 하나씩 꺼낸다.\n",
        "            pred_index = np.argmax(pred) # 예를 들어 [0, 0, 1, 0 ,0]라면 1의 인덱스인 2를 리턴한다.\n",
        "            temp.append(index_to_ner[pred_index].replace(\"PAD\", \"O\")) # 'PAD'는 'O'로 변경\n",
        "        result.append(temp)\n",
        "    return result"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bzEfiWsQQJ0n"
      },
      "source": [
        "y_predicted = model.predict([X_test, X_char_test])\n",
        "pred_tags = sequences_to_tag(y_predicted)\n",
        "test_tags = sequences_to_tag(y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YxHz40WFP3T7",
        "outputId": "1341cdae-38d4-4603-d77d-3c4f7b5ed5af"
      },
      "source": [
        "print(classification_report(test_tags, pred_tags))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         art       0.00      0.00      0.00        63\n",
            "         eve       0.36      0.27      0.31        52\n",
            "         geo       0.82      0.86      0.84      7620\n",
            "         gpe       0.96      0.94      0.95      3145\n",
            "         nat       0.62      0.14      0.22        37\n",
            "         org       0.61      0.56      0.59      4033\n",
            "         per       0.73      0.72      0.73      3545\n",
            "         tim       0.87      0.84      0.86      4067\n",
            "\n",
            "   micro avg       0.80      0.79      0.79     22562\n",
            "   macro avg       0.62      0.54      0.56     22562\n",
            "weighted avg       0.79      0.79      0.79     22562\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jIXoMhZsOgmI"
      },
      "source": [
        "## 성능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qC1_wjJSQcdr",
        "outputId": "902727e9-3884-4a67-9241-d1ba6d61ba2c"
      },
      "source": [
        "print(\"F1-score: {:.1%}\".format(f1_score(test_tags, pred_tags)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "F1-score: 79.3%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o3UN2XvynYrD"
      },
      "source": [
        "#BiLSTM-CNN-CRF를 이용한 개체명인식"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pIIC5YkUOle8"
      },
      "source": [
        "## 모델링"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0n534tvi9h4S"
      },
      "source": [
        "import tensorflow as tf\n",
        "from keras_crf import CRFModel"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XhNyGVb2nf4J",
        "outputId": "2c2f38e6-a80c-4984-90dd-feb9305db788"
      },
      "source": [
        "embedding_dim = 128\n",
        "char_embedding_dim = 64\n",
        "dropout_ratio = 0.5\n",
        "hidden_units = 256\n",
        "num_filters = 30\n",
        "kernel_size = 3\n",
        "\n",
        "# 단어 임베딩\n",
        "words_input = Input(shape=(None,),dtype='int32',name='words_input')\n",
        "words = Embedding(input_dim=vocab_size, output_dim=embedding_dim)(words_input)\n",
        "\n",
        "# char 임베딩\n",
        "character_input = Input(shape=(None, max_len_char,),name='char_input')\n",
        "embed_char_out = TimeDistributed(Embedding(len(char_to_index), char_embedding_dim, embeddings_initializer=RandomUniform(minval=-0.5, maxval=0.5)), name='char_embedding')(character_input)\n",
        "dropout = Dropout(dropout_ratio)(embed_char_out)\n",
        "\n",
        "# char 임베딩에 대해서는 Conv1D 수행\n",
        "conv1d_out= TimeDistributed(Conv1D(kernel_size=kernel_size, filters=num_filters, padding='same',activation='tanh', strides=1))(dropout)\n",
        "maxpool_out=TimeDistributed(MaxPooling1D(max_len_char))(conv1d_out)\n",
        "char = TimeDistributed(Flatten())(maxpool_out)\n",
        "char = Dropout(dropout_ratio)(char)\n",
        "\n",
        "# char 임베딩을 Conv1D 수행한 뒤에 단어 임베딩과 연결\n",
        "output = concatenate([words, char])\n",
        "\n",
        "# 연결한 벡터를 가지고 문장의 길이만큼 LSTM을 수행\n",
        "output = Bidirectional(LSTM(hidden_units, return_sequences=True, dropout=dropout_ratio))(output)\n",
        "\n",
        "# 출력층\n",
        "output = TimeDistributed(Dense(tag_size, activation='relu'))(output)\n",
        "\n",
        "base = Model(inputs=[words_input, character_input], outputs=[output])\n",
        "model = CRFModel(base, tag_size)\n",
        "model.compile(optimizer=tf.keras.optimizers.Adam(0.001), metrics='accuracy')\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"crf_model_1\"\n",
            "__________________________________________________________________________________________________\n",
            " Layer (type)                   Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            " char_input (InputLayer)        [(None, None, 15)]   0           []                               \n",
            "                                                                                                  \n",
            " char_embedding (TimeDistribute  (None, None, 15, 64  4736       ['char_input[0][0]']             \n",
            " d)                             )                                                                 \n",
            "                                                                                                  \n",
            " dropout_4 (Dropout)            (None, None, 15, 64  0           ['char_embedding[0][0]']         \n",
            "                                )                                                                 \n",
            "                                                                                                  \n",
            " time_distributed_10 (TimeDistr  (None, None, 15, 30  5790       ['dropout_4[0][0]']              \n",
            " ibuted)                        )                                                                 \n",
            "                                                                                                  \n",
            " time_distributed_11 (TimeDistr  (None, None, 1, 30)  0          ['time_distributed_10[0][0]']    \n",
            " ibuted)                                                                                          \n",
            "                                                                                                  \n",
            " words_input (InputLayer)       [(None, None)]       0           []                               \n",
            "                                                                                                  \n",
            " time_distributed_12 (TimeDistr  (None, None, 30)    0           ['time_distributed_11[0][0]']    \n",
            " ibuted)                                                                                          \n",
            "                                                                                                  \n",
            " embedding_6 (Embedding)        (None, None, 128)    4072832     ['words_input[0][0]']            \n",
            "                                                                                                  \n",
            " dropout_5 (Dropout)            (None, None, 30)     0           ['time_distributed_12[0][0]']    \n",
            "                                                                                                  \n",
            " concatenate_1 (Concatenate)    (None, None, 158)    0           ['embedding_6[0][0]',            \n",
            "                                                                  'dropout_5[0][0]']              \n",
            "                                                                                                  \n",
            " bidirectional_5 (Bidirectional  (None, None, 512)   849920      ['concatenate_1[0][0]']          \n",
            " )                                                                                                \n",
            "                                                                                                  \n",
            " time_distributed_13 (TimeDistr  (None, None, 18)    9234        ['bidirectional_5[0][0]']        \n",
            " ibuted)                                                                                          \n",
            "                                                                                                  \n",
            " crf_1 (CRF)                    [(None, None),       702         ['time_distributed_13[0][0]']    \n",
            "                                 (None, None, 18),                                                \n",
            "                                 (None,),                                                         \n",
            "                                 (18, 18)]                                                        \n",
            "                                                                                                  \n",
            " decode_sequence (Lambda)       (None, None)         0           ['crf_1[0][0]']                  \n",
            "                                                                                                  \n",
            " potentials (Lambda)            (None, None, 18)     0           ['crf_1[0][1]']                  \n",
            "                                                                                                  \n",
            " sequence_length (Lambda)       (None,)              0           ['crf_1[0][2]']                  \n",
            "                                                                                                  \n",
            " kernel (Lambda)                (18, 18)             0           ['crf_1[0][3]']                  \n",
            "                                                                                                  \n",
            "==================================================================================================\n",
            "Total params: 4,943,214\n",
            "Trainable params: 4,943,214\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sCEFX2WR-Far"
      },
      "source": [
        "es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=4)\n",
        "mc = ModelCheckpoint('bilstm_cnn_crf/cp.ckpt', monitor='val_decode_sequence_accuracy', mode='max', verbose=1, save_best_only=True, save_weights_only=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bMIECQqU-MOR",
        "outputId": "ad94afa2-0218-481d-edd3-0858bc28efc7"
      },
      "source": [
        "history = model.fit([X_train, X_char_train], y_train_int, batch_size=128, epochs=15, validation_split=0.1, callbacks=[mc, es])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9270 - loss: 20.0089\n",
            "Epoch 00001: val_decode_sequence_accuracy improved from -inf to 0.96776, saving model to bilstm_cnn_crf/cp.ckpt\n",
            "270/270 [==============================] - 59s 193ms/step - decode_sequence_accuracy: 0.9270 - loss: 19.9610 - val_decode_sequence_accuracy: 0.9678 - val_loss: 7.4661\n",
            "Epoch 2/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9787 - loss: 5.1367\n",
            "Epoch 00002: val_decode_sequence_accuracy improved from 0.96776 to 0.98258, saving model to bilstm_cnn_crf/cp.ckpt\n",
            "270/270 [==============================] - 45s 167ms/step - decode_sequence_accuracy: 0.9787 - loss: 5.1319 - val_decode_sequence_accuracy: 0.9826 - val_loss: 4.0034\n",
            "Epoch 3/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9856 - loss: 3.1769\n",
            "Epoch 00003: val_decode_sequence_accuracy improved from 0.98258 to 0.98532, saving model to bilstm_cnn_crf/cp.ckpt\n",
            "270/270 [==============================] - 45s 165ms/step - decode_sequence_accuracy: 0.9856 - loss: 3.1734 - val_decode_sequence_accuracy: 0.9853 - val_loss: 3.1092\n",
            "Epoch 4/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9878 - loss: 2.4798\n",
            "Epoch 00004: val_decode_sequence_accuracy improved from 0.98532 to 0.98620, saving model to bilstm_cnn_crf/cp.ckpt\n",
            "270/270 [==============================] - 45s 166ms/step - decode_sequence_accuracy: 0.9878 - loss: 2.4816 - val_decode_sequence_accuracy: 0.9862 - val_loss: 2.7894\n",
            "Epoch 5/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9891 - loss: 2.1013\n",
            "Epoch 00005: val_decode_sequence_accuracy did not improve from 0.98620\n",
            "270/270 [==============================] - 45s 167ms/step - decode_sequence_accuracy: 0.9891 - loss: 2.1018 - val_decode_sequence_accuracy: 0.9860 - val_loss: 2.8119\n",
            "Epoch 6/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9898 - loss: 1.8567\n",
            "Epoch 00006: val_decode_sequence_accuracy improved from 0.98620 to 0.98646, saving model to bilstm_cnn_crf/cp.ckpt\n",
            "270/270 [==============================] - 45s 165ms/step - decode_sequence_accuracy: 0.9898 - loss: 1.8559 - val_decode_sequence_accuracy: 0.9865 - val_loss: 2.4847\n",
            "Epoch 7/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9904 - loss: 1.6801\n",
            "Epoch 00007: val_decode_sequence_accuracy improved from 0.98646 to 0.98668, saving model to bilstm_cnn_crf/cp.ckpt\n",
            "270/270 [==============================] - 47s 175ms/step - decode_sequence_accuracy: 0.9904 - loss: 1.6790 - val_decode_sequence_accuracy: 0.9867 - val_loss: 2.4975\n",
            "Epoch 8/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9908 - loss: 1.5379\n",
            "Epoch 00008: val_decode_sequence_accuracy did not improve from 0.98668\n",
            "270/270 [==============================] - 45s 165ms/step - decode_sequence_accuracy: 0.9908 - loss: 1.5376 - val_decode_sequence_accuracy: 0.9866 - val_loss: 2.4164\n",
            "Epoch 9/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9913 - loss: 1.4170\n",
            "Epoch 00009: val_decode_sequence_accuracy did not improve from 0.98668\n",
            "270/270 [==============================] - 45s 165ms/step - decode_sequence_accuracy: 0.9913 - loss: 1.4185 - val_decode_sequence_accuracy: 0.9865 - val_loss: 2.5479\n",
            "Epoch 10/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9915 - loss: 1.3234\n",
            "Epoch 00010: val_decode_sequence_accuracy did not improve from 0.98668\n",
            "270/270 [==============================] - 45s 166ms/step - decode_sequence_accuracy: 0.9915 - loss: 1.3241 - val_decode_sequence_accuracy: 0.9864 - val_loss: 2.5477\n",
            "Epoch 11/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9918 - loss: 1.2319\n",
            "Epoch 00011: val_decode_sequence_accuracy did not improve from 0.98668\n",
            "270/270 [==============================] - 45s 166ms/step - decode_sequence_accuracy: 0.9918 - loss: 1.2317 - val_decode_sequence_accuracy: 0.9864 - val_loss: 2.5617\n",
            "Epoch 12/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9921 - loss: 1.1525\n",
            "Epoch 00012: val_decode_sequence_accuracy did not improve from 0.98668\n",
            "270/270 [==============================] - 45s 165ms/step - decode_sequence_accuracy: 0.9921 - loss: 1.1521 - val_decode_sequence_accuracy: 0.9863 - val_loss: 2.5442\n",
            "Epoch 00012: early stopping\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2UGaqpkrHozN",
        "outputId": "929a1aa5-5d46-4a3d-d4dd-bb64c7663a29"
      },
      "source": [
        "model.load_weights('bilstm_cnn_crf/cp.ckpt')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7fc744572150>"
            ]
          },
          "metadata": {},
          "execution_count": 105
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rXYnKtBgHq6g",
        "outputId": "06ef2cb2-8b7b-4c97-c22e-3be2e024d251"
      },
      "source": [
        "i = 13 # 확인하고 싶은 테스트용 샘플의 인덱스.\n",
        "\n",
        "# 입력한 테스트용 샘플에 대해서 예측 y를 리턴\n",
        "y_predicted = model.predict([np.array([X_test[i]]), np.array([X_char_test[i]])])[0] \n",
        "true = np.argmax(y_test[i], -1) # 원-핫 벡터를 정수 인코딩으로 변경.\n",
        "\n",
        "print(\"{:15}|{:5}|{}\".format(\"단어\", \"실제값\", \"예측값\"))\n",
        "print(35 * \"-\")\n",
        "\n",
        "for word, tag, pred in zip(X_test[i], true, y_predicted[0]):\n",
        "    if word != 0: # PAD값은 제외함.\n",
        "        print(\"{:17}: {:7} {}\".format(index_to_word[word], index_to_ner[tag], index_to_ner[pred]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "단어             |실제값  |예측값\n",
            "-----------------------------------\n",
            "the              : O       O\n",
            "statement        : O       O\n",
            "came             : O       O\n",
            "as               : O       O\n",
            "u.n.             : B-org   B-org\n",
            "secretary-general: I-org   I-org\n",
            "kofi             : B-per   B-per\n",
            "annan            : I-per   I-per\n",
            "met              : O       O\n",
            "with             : O       O\n",
            "officials        : O       O\n",
            "in               : O       O\n",
            "amman            : B-geo   B-geo\n",
            "to               : O       O\n",
            "discuss          : O       O\n",
            "wednesday        : B-tim   B-tim\n",
            "'s               : O       O\n",
            "attacks          : O       O\n",
            ".                : O       O\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "W3WPtK5HIZsz",
        "outputId": "4f290d21-1f66-4a34-b10e-702b568990cb"
      },
      "source": [
        "epochs = range(1, len(history.history['val_loss']) + 1)\n",
        "plt.plot(epochs, history.history['loss'])\n",
        "plt.plot(epochs, history.history['val_loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'val'], loc='upper left')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEWCAYAAABsY4yMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9bnH8c8zk32HMCEQwk7CGraIIEK4dUUBUUFs1W63xd5al7ba2tpeva1tudWrVqtVqlbbUjfEuiCoKAGVRcO+J2yBECALhCxkn9/94wwYIEAS5mQyM8/79corM2fO8gzLN78858zviDEGpZRSgcfh6wKUUkrZQwNeKaUClAa8UkoFKA14pZQKUBrwSikVoDTglVIqQGnAKwWIyEsi8nAL190rIpdf6H6UspsGvFJKBSgNeKWUClAa8MpveFoj94nIRhGpEpEXRKSriCwSkQoRWSIinZqsP01EtohImYhki8igJq+NFJG1nu1eAyJOO9YUEVnv2XaFiGS0sebvi8hOETkiIu+ISHfPchGRx0WkSETKRWSTiAz1vHaNiGz11HZARO5t0x+YCnoa8Mrf3AhcAaQBU4FFwC8BF9a/57sARCQNeAW4x/Pa+8C7IhImImHAv4F/AJ2BNzz7xbPtSOBF4HYgEXgOeEdEwltTqIh8DfgDcBPQDcgHXvW8fCUw0fM+4j3rlHpeewG43RgTCwwFPmnNcZU6QQNe+ZunjDGHjTEHgE+B1caYdcaYGuAtYKRnvVnAQmPMR8aYeuBRIBK4BBgLhAJPGGPqjTHzgS+bHGM28JwxZrUxptEY8zJQ69muNW4BXjTGrDXG1AK/AMaJSG+gHogFBgJijNlmjDno2a4eGCwiccaYo8aYta08rlKABrzyP4ebPK5u5nmM53F3rBEzAMYYN7AfSPG8dsCcOtNefpPHvYCfetozZSJSBqR6tmuN02uoxBqlpxhjPgH+DDwNFInIXBGJ86x6I3ANkC8iy0RkXCuPqxSgAa8CVyFWUANWzxsrpA8AB4EUz7ITejZ5vB/4nTEmoclXlDHmlQusIRqr5XMAwBjzpDFmNDAYq1Vzn2f5l8aY64AkrFbS6608rlKABrwKXK8D14rIZSISCvwUq82yAlgJNAB3iUioiNwAjGmy7V+BH4jIxZ6TodEicq2IxLayhleA74jICE///vdYLaW9InKRZ/+hQBVQA7g95whuEZF4T2upHHBfwJ+DCmIa8CogGWN2ALcCTwElWCdkpxpj6owxdcANwLeBI1j9+gVNts0Bvo/VQjkK7PSs29oalgC/Bt7E+q2hH3Cz5+U4rB8kR7HaOKXAI57XbgP2ikg58AOsXr5SrSZ6ww+llApMOoJXSqkApQGvlFIBSgNeKaUClAa8UkoFqBBfF9BUly5dTO/evX1dhlJK+Y01a9aUGGNczb3WoQK+d+/e5OTk+LoMpZTyGyKSf7bXtEWjlFIBSgNeKaUClAa8UkoFqA7Vg29OfX09BQUF1NTU+LoUW0VERNCjRw9CQ0N9XYpSKkB0+IAvKCggNjaW3r17c+rkf4HDGENpaSkFBQX06dPH1+UopQJEh2/R1NTUkJiYGLDhDiAiJCYmBvxvKUqp9tXhAx4I6HA/IRjeo1KqfflFwJ+TcUPlYagp93UlSinVofh/wCNQWQTVR2zZe1lZGc8880yrt7vmmmsoKyuzoSKllGoZ/w94EQiLgdpKsGFu+7MFfENDwzm3e//990lISPB6PUop1VId/iqaFgmPhZoyaKiF0Aiv7vr+++9n165djBgxgtDQUCIiIujUqRPbt28nNzeX6dOns3//fmpqarj77ruZPXs28NW0C5WVlUyePJlLL72UFStWkJKSwttvv01kZKRX61RKqdP5VcD/z7tb2FrYTK/duKH+OIR8AY7WXUc+uHscD04dctbX58yZw+bNm1m/fj3Z2dlce+21bN68+eTljC+++CKdO3emurqaiy66iBtvvJHExMRT9pGXl8crr7zCX//6V2666SbefPNNbr311lbVqZRSreVXAX9W4rBaNe7GVgd8a40ZM+aUa9WffPJJ3nrrLQD2799PXl7eGQHfp08fRowYAcDo0aPZu3evrTUqpRT4WcCfa6TN0XyoOQbJw6ywt0l0dPTJx9nZ2SxZsoSVK1cSFRXFpEmTmr2WPTw8/ORjp9NJdXW1bfUppdQJ/n+S9YTwWDCNUO/d8IyNjaWioqLZ144dO0anTp2Iiopi+/btrFq1yqvHVkqpC+FXI/hzCo+xvtdVQFiU13abmJjI+PHjGTp0KJGRkXTt2vXka1dffTXPPvssgwYNIj09nbFjx3rtuEopdaHE2HBpYVtlZmaa02/4sW3bNgYNGtSyHRRtA2coJPa3oTr7teq9KqUUICJrjDGZzb0WOC0asK6Hr6uyrqpRSqkgF1gBHx5rhXvdcV9XopRSPmdbwItIuoisb/JVLiL32HU84NQ+vFJKBTnbTrIaY3YAIwBExAkcAN6y8XiIIwRCI6G2AmK72XUopZTyC+3VorkM2GWMOevdv9vK7TbkHa6gpLLWWhAea7Vo3I3ePpRSSvmV9gr4m4FX7NixwyEYoKLGM/lXWCxgrJOtSikVxGwPeBEJA6YBb5zl9dkikiMiOcXFxW06RmxECFV1jTS6DYRFA2K1aXwgJibGJ8dVSqnTtccIfjKw1hhzuLkXjTFzjTGZxphMl8vVpgPEhodgjKGytgEcTuuDTnqiVSkV5Nrjk6xfx6b2zAlR4SE4RKioqSc+MtTqw1ccgsYGcF7YW7z//vtJTU3ljjvuAOChhx4iJCSEpUuXcvToUerr63n44Ye57rrrvPFWlFLKa2wNeBGJBq4AbvfKDhfdD4c2nbHYAfSvb8RtDCbMiZycPjgSHOd5i8nDYPKcs748a9Ys7rnnnpMB//rrr/PBBx9w1113ERcXR0lJCWPHjmXatGl6X1WlVIdia8AbY6qAxPOu6AVOh9DQYDAGRByAgLvh/AF/HiNHjqSoqIjCwkKKi4vp1KkTycnJ/PjHP2b58uU4HA4OHDjA4cOHSU5O9s6bUUopL/CvycbOMdI2DY3sPlRBt/hIXLHhULoTGusgafAFH3bmzJnMnz+fQ4cOMWvWLObNm0dxcTFr1qwhNDSU3r17NztNsFJK+VLATFUQFuIkPMRJRU29tSA81rqFX2PdBe971qxZvPrqq8yfP5+ZM2dy7NgxkpKSCA0NZenSpeTne/3yfqWUumD+NYI/j9iIEEqr6nC7DY6wWGthbSVEdb6g/Q4ZMoSKigpSUlLo1q0bt9xyC1OnTmXYsGFkZmYycOBAL1SvlFLeFXABX1JZS2VtA3ERkSBO63r4Cwx4gE2bvjq526VLF1auXNnsepWVlRd8LKWU8oaAadEARIdZl0tW1jZYt+0Lj4G6SuhAc94rpVR7CaiAdziE6PCQU/vwjXVe6cMrpZS/8YuAb81dp2IjQqhtcFNb3+iZlwafTVvQGh3pzlpKqcDQ4QM+IiKC0tLSFgdgbLh1WqGitgFCwsER2uED3hhDaWkpERERvi5FKRVAOvxJ1h49elBQUEBrJiIrPVZD+UEhMSYcjpdB/SGIq7b68h1UREQEPXr08HUZSqkA0uEDPjQ0lD59+rRqm1ff3szrOQWs++8riNj8Kiz+Ifzgc0gealOVSinV8XT4Fk1bZKW7qK5v5Mu9R6DPRGvhnuW+LUoppdpZQAb82L6JhDkdLNtRDAmp0Lkv7Fnm67KUUqpdBWTAR4WFcHHfzmTnevr2fbJg7+fW9MFKKRUkAjLgAbLSXOwsqqTg6HHom2XdAKRwna/LUkqpdhOwAT8p3bo71PLcEug9wVq4J9t3BSmlVDsL2IDv54ohJSGS7B1FEN0Fug7TE61KqaASsAEvIkxMc7FiVyl1DW7rapp9q6G+2telKaVUuwjYgAerTVNZ28Ca/KNWH76xFvZ/4euylFKqXQR0wF/SL5EQh7Astxh6jrOmD9bLJZVSQSKgAz42IpTM3p2sgI+Ig5TRsFsDXikVHAI64AGy0pLYdrCcw+U1VpumcC3UHPN1WUopZbuAD/gTl0suyy22TrQaN+Sv8HFVSillP1sDXkQSRGS+iGwXkW0iMs7O4zVnYHIsSbHh1rQFPcZASIReLqmUCgp2zyb5J2CxMWaGiIQBUTYf7wwiQlaaiw+2HKLBEUZI6sXah1dKBQXbRvAiEg9MBF4AMMbUGWPK7DreuUxKT6K8poENBWVWH75oC1S2fH55pZTyR3a2aPoAxcDfRGSdiDwvItGnryQis0UkR0RyWnNTj9a4tH8XHALZO4qhzyRr4V5t0yilApudAR8CjAL+YowZCVQB95++kjFmrjEm0xiT6XK5bCkkPiqUUT09l0t2Gw7hcdqmUUoFPDsDvgAoMMas9jyfjxX4PpGV5mJjwTFKqhuh96V6olUpFfBsC3hjzCFgv4ikexZdBmy163jnk+W5XPLTPM/lkkf3QNk+X5WjlFK2s/s6+DuBeSKyERgB/N7m453V0O7xJEaHWZdL9smyFuooXikVwGwNeGPMek9/PcMYM90Yc9TO452Lw2HNLrk8rwR3l4EQ7dI+vFIqoAX8J1mbmpTu4khVHZsKy602zZ5lYIyvy1JKKVsEVcBf2r8LcvJyySyoPAwlub4uSymlbBFUAZ8YE05GSjzLcousETxom0YpFbCCKuABstKTWL+/jLKIFEjoqfPDK6UCVvAFfJoLt4FP80qsNs3eT8Hd6OuylFLK64Iu4EekJhAfGeqZPjjLmhv+4AZfl6WUUl4XdAHvdAgTBnRhWW4x7t4TrIV6PbxSKgAFXcCD1aYprqhlW2UkuAZqH14pFZCCNuCBr9o0+Suhoc7HVSmllHcFZcAnxUUwuFuc53r4idBQDQVf+rospZTyqqAMeLA+1bo2/yjl3caCOLQPr5QKOEEb8FlpLhrchhUF9dBthPbhlVIBJ2gDflSvTsSGh3j68BOtFk1tpa/LUkoprwnagA91OhjfvwvZO4oxfbLA3QD7Vvm6LKWU8pqgDXiwbgJy8FgNOyOGgjMM9mT7uiSllPKa4A54z+WS2bsroccYPdGqlAooQR3w3RMiSesaQ3ZuEfTNgoMb4fgRX5ellFJeEdQBDzApPYkv9xylusd4wFiTjymlVAAI+oDPSnNR1+hmRXUvCI3WNo1SKmAEfcBn9u5EVJiT7J1l0OsSvQGIUipgBH3Ah4c4uaRfItm5RZg+E6E0D8oLfV2WUkpdsKAPeLDaNPuPVHOg0xhrgbZplFIBwNaAF5G9IrJJRNaLSI6dx7oQWWlJAHx0xAWRnTXglVIBoT1G8P9hjBlhjMlsh2O1Sc/EKPp2iSY7txT6TLD68Mb4uiyllLog2qLxmJjmYtXuUup7ToDyAjiy29clKaXUBbE74A3woYisEZHZza0gIrNFJEdEcoqLi20u5+wmpbuobXCzLmS4tUBnl1RK+Tm7A/5SY8woYDJwh4hMPH0FY8xcY0ymMSbT5XLZXM7Zje2bSHiIg0WFURCXopdLKqX8nq0Bb4w54PleBLwFjLHzeBciItTJ2L6JLMsrsaYP3vspuN2+LksppdrMtoAXkWgRiT3xGLgS2GzX8bwhK83F7uIqSpPGwvFSKNri65KUUqrN7BzBdwU+E5ENwBfAQmPMYhuPd8Gy0j2zS9YPshZom0Yp5cdsC3hjzG5jzHDP1xBjzO/sOpa39O0STWrnSBblOyGxv14Pr5Tya3qZZBMiQlaaixW7SmjsNQHyP4fGel+XpZRSbaIBf5pJaUkcr2tkZ/RoqKuEwnW+LkkppdpEA/404/olEuZ08H7lAGuB9uGVUn5KA/400eEhXNSnE4t310HyMP3Ak1LKb2nANyMrzcWOwxVUpoyH/V9AfbWvS1JKqVbTgG/Gidkl18gwaKyFfat8XJFSSrWeBnwz0rrG0C0+greO9AZHiF4uqZTySxrwzThxueTHu6pwp4zWPrxSyi9pwJ/FpHQXFbUNHOw0xrpUsuaYr0tSSqlW0YA/i0v6d8HpEJY3DAbjhr2f+7okpZRqFQ34s4iLCGV0z068ejAZQiK1TaOU8jstCngRuVtE4sTygoisFZEr7S7O17LSXWw4WE1dyhg90aqU8jstHcF/1xhTjjXlbyfgNmCObVV1EFlp1uySO6JGQdFWqCzycUVKKdVyLQ148Xy/BviHMWZLk2UBa0j3OFyx4SyuSrcW6CheKeVHWhrwa0TkQ6yA/8BzI4+Av92RiDBxgItX9idgwuO0D6+U8istDfj/BO4HLjLGHAdCge/YVlUHkpXu4ki1m2NdL9YRvFLKr7Q04McBO4wxZSJyK/ArICguDJ/QvwsOgXXO4XB0LxzN93VJSinVIi0N+L8Ax0VkOPBTYBfwd9uq6kA6RYcxPDWBN8v6Wwu0TaOU8hMtDfgGY4wBrgP+bIx5Goi1r6yOZVJaEgsPxeKOTtI2jVLKb7Q04CtE5BdYl0cuFBEHVh8+KGSluzBGONDpIivgjfF1SUopdV4tDfhZQC3W9fCHgB7AI7ZV1cEMS4mnU1QoKxqHQOVhKN7h65KUUuq8WhTwnlCfB8SLyBSgxhgTFD14AKdDmJjm4p9Fva0F2odXSvmBlk5VcBPwBTATuAlYLSIzWritU0TWich7bS/T97LSXGyqSqAuNlXv06qU8gshLVzvAaxr4IsARMQFLAHmt2Dbu4FtQFybKuwgJnqmLciLHs2QvUvB3QgOp4+rUkqps2tpD95xItw9SluyrYj0AK4Fnm9DbR1Kl5hwhqXE82H1QKg9Bl/81dclKaXUObU04BeLyAci8m0R+TawEHi/Bds9AfyMc0xrICKzRSRHRHKKi4tbWI5vTEp38WzxEOr7XwWLfw6r5/q6JKWUOquWnmS9D5gLZHi+5hpjfn6ubTwnY4uMMWvOs++5xphMY0ymy+VqYdm+kZXmotbtZMmwRyD9Wlh0H6x61tdlKaVUs1rag8cY8ybwZiv2PR6YJiLXABFAnIj80xhzaytr7DBGpCYQFxHC0rwyJs98CeZ/xxrJGzeM+6Gvy1NKqVOccwQvIhUiUt7MV4WIlJ9rW2PML4wxPYwxvYGbgU/8OdwBQpwOJgxwsSy3GOMMhZkvwaCp8MEvYOXTvi5PKaVOcc6AN8bEGmPimvmKNcb49VUxbZWV5uJweS1r95WBMxRm/A0GTYMPfgkr/uzr8pRS6qR2uSerMSbbGDOlPY5lt8nDkkmMDuN/F2/HGOMJ+Rdh8HT48AH4/Elfl6iUUoDedLvVYiNCueeKNL7Yc4QPtx62FjpD4cYXYMj18NGv4bMnfFukUkqhAd8mX78olf5JMcxZtJ26Bs8VoM4QuOF5GHojLHkQPn3Mt0UqpYKeBnwbhDgd/PKagewpqWLe6iY3AHGGwPVzYegM+Ph/YPmjvitSKRX0NODb6D/SkxjfP5E/fZzHser6r15whsD1z8Gwm+CT38KyoJl0UynVwWjAt5GI8MA1gzlWXc/TS3ee+qIzBK5/FjJuhqUPQ/b/+qZIpVRQ04C/AIO7xzFjVA9e+nwv+48cP/VFhxOmPwPDvw7Zv4elf/BNkUqpoKUBf4F+emU6TocwZ/H2M190OOG6p2HELbBsDiz9vd4NSinVbjTgL1ByfATfn9iXhRsPsib/6JkrOJww7c8w8lZY9r8a8kqpdqMB7wW3T+yLKzac3y3can346XQOB0x9CkZ9E5b/ET55WENeKWU7DXgviA4P4d4r01i7r4z3Nx1qfiWHA6b8CUZ9Cz59FD7+jYa8UspWGvBeMmN0KgOTY5mzeBu1DY3Nr+RwwJQnYPR34LPHYMlDGvJKKdtowHuJ0yH88ppB7D9Szd9X5J99RYcDrn0MMr8Lnz8BH/23hrxSyhYa8F40Mc1FVpqLpz7J42hV3dlXPBHyF30PVjwJH/5KQ14p5XUa8F72wLWDqKxt4MlP8s69oghc8yiMmQ0r/wwfPKAhr5TyKg14L0vrGsusi3ryj5X57CmpOvfKIjD5j3DxD2DV09ac8hrySikv0YC3wY+vGEB4iIM5i7adf2URuHoOXPxfsOoZWHy/hrxSyis04G2QFBvBD7L68cGWw6zeXXr+DUTg6j/A2Dtg9bOw6Gca8kqpC6YBb5PvTehLclwEv3t/G253C8JaBK76HYz7EXwxF96/V0NeKXVBNOBtEhnm5L6r0tlYcIx3Nxa2bCMRuPJhuOQu+PJ5WPhTcLvtLVQpFbA04G10/cgUhqbE8cfFO6ipP8uHn04nAlf8BsbfAzkvwMKfaMgrpdpEA95GDs+Hnw6UVfPi53tavqEIXP4QXPpjWPM3mP9t2LdaWzZKqVYJsWvHIhIBLAfCPceZb4x50K7jdVSX9OvC5YOSeGbpLm7KTKVLTHjLNhSByx6E0Cj49P9g69uQ0AuGzYSMm8CVbm/hSim/Z+cIvhb4mjFmODACuFpExtp4vA7r/smDqK5v5Iklua3bUASyfgb35sH0v0DnvtYcNk+PgWcnwIqnoLyF/X2lVNCxLeCNpdLzNNTzFZQ9hv5JMdxycU9e+WI/O4sqWr+DiDgY8Q345r/hJ9vgqj9Y88x/+Ct4bDC8PBXW/h2qy7xfvFLKb0mz85d7a+ciTmAN0B942hjz82bWmQ3MBujZs+fo/PxzTNTlx0ora5n0SDZj+nTmhW9f5J2dluyETW/AptfhyG5whkPaldYNvwdcCaERrd7l1sJyuidEkBAV5p0alVK2EpE1xpjMZl+zM+CbFJAAvAXcaYzZfLb1MjMzTU5Oju31+Mqzy3YxZ9F2/vW9i7mkfxfv7dgYOLDWCvrNb0JVMYTHw+BpVr++16XWBGfn3IXhmexdPPrhDrrFRTD3m5kMTYn3Xo1KKVv4POA9Rfw3cNwY8+jZ1gn0gK+pb+Sy/1tGfGQo7955KU6HeP8gjQ2wJxs2vgHb34O6SojtDsNutEb2ycOs3v5pdd3/5kb+vb6Qq4Z0ZVPBMUqr6vjjjAyuG5Hi/RqVUl5zroC3rQcvIi7PyB0RiQSuAJq5M3XwiAh18rOr09l6sJwFawvsOYgzBPpfDjc8Z52cvfEF6JYBq/4Cz02AZ8bC8kfhqNUKKyqvYdbcVfx7fSH3XZXOs7eO5p07L2V4jwTufnU9f1i0jcaWfBJXKdXh2DaCF5EM4GXAifWD5HVjzG/OtU2gj+DBaoVMf2YFh45Vk33vfxAZ5myfAx8/Alvesnr2+1YCUJU0mmdKR/FW/RgenDWRq4Ykn1y9rsHNb97bwj9X7SMrzcWTXx9JfGRo+9SqlGqxDtGiaYlgCHiAnL1HmPHsSn5yRRp3XTag/Qs4ms+2JX/DufkN0qQA4whB+l1m9evTJ0NY9MlV/7V6Hw++s5kenaL4620j6d85HBprobEeGmqhse7Ur4bTn3vWbaw7bbt667k4rRPD3Ued0TpSSp2fBnwH9F//XMOy3GKy751EUlzrr3ZpK7fb8MSSXJ78ZCcX9Upg7lURdNr5b+vkbPkBCI2G2K5NgrqWxoY6TEMtIdgxZYIABhIHwPBZkDELEnracBylApMGfAe0t6SKKx5fxo2jejDnxox2OebxugZ+8toGFm85xE2ZPfjt9KGEh3haRG437FthtXFqjoEzrMlXKBUNTt7ZXExBhZvxad0Yn94dCQk7db2QcHCGWpdrerazljVdp8ljR4h1rK1vw8bXIP9zq5Ze462gHzIdIvRKHqXORQO+g/rNu1t5acUe3r97AgOT42w91oGyar7/cg7bD5XzwLWD+e743kgrWyLVdY3cv2Ajb68v5NqMbjwyI4OoMC/OdnE037rUc8NrUJpn/aBInwzDb7ZOHDv1HIBSp9OA76DKjteR9Ug2GT3i+cd/XmzbcdbkH+H2f6yhtt7NU98YyaT0pDbvyxjD3OW7mbN4OwOT45h722hSO0d5sVqs6/oL11pBv3k+HC+FqEQYeiNk3Awp2q9X6gQN+A7s+U938/DCbbz83TFkpbm8vv/5awr45YJNdE+I4PlvXUT/pBiv7Dd7RxF3vrKOUKeDp78xinH9Er2y3zM01sPOj2Hjq7D9fevEbGJ/K+gzboJOvew5rlJ+QgO+A6trcHPF48uICHHy/t0TvPbhp0a34X8Xb2fu8t2M75/I098Y5fXpB3YXV/L9v+ewt/Q4D04dzG1je7W67dMq1WWw7R1rZJ//mbWs5yXWydnB0yEywb5jK9VBacB3cIs2HeS/5q3lDzcM4+tjLvwKkoqaeu56ZR1LdxTzzXG9+PWUwYQ67flMW0VNPT9+bT1LthUxKzOV30wf8tWJWzuV7YONr1snZ0tyPf36q62Rff/LrZO5SgUBDfgOzhjDzGdXsrf0ONn3TSImvO0nLvNLq/jeyznsKanioWlDuHWs/S0Mt9vw+JJcnvpkJ6N6JvDsbaNJim2nSz+NgcJ1VtBvmg/HSyCys9WvH34zpIzWfr031VZaE9uV7rTOjbQnY8Dd0OSr8bTnzS1r6fP60543Wld5OUPAEWqd4HeEnvY85OzLz/raWfYRGg0DLm/TH4sGvB9Yt+8o1z+zgju/1p+fXtm2m3ms2FXCD+etBeCZW0ZxST8vTmjWAgs3HuTeNzYQHxnKc7eNZnhqO7dMTvbrX4Md70NDjadfP8vTr+/dvvX4q/oaOLoHSnfBkV1WmJd6Qr3ykK+rO5M4vwpVR4g1lXZrnzcNZocTxGEFfaMn/Bs9PwAa6z3Pmy4//XnDV+uaFt6qMzoJ7str29vXgPcPd76yjo+2HmLpvZPoFh/Zqm3nrc7nwbe30KdLNM9/K5NeidHn38gGWwvLmf2PHIoqaplzwzBuGNXDJ3WcvL7+lH79OOiTZT12N1j/+U6O2pqM7ExjM8vczYz6GpvfR9Nl4bEQ7YKYJOsr+sR311fPoxKtUV17aqy32lylngBvGuTH9nPKrRuiukBiP+uHZee+1vfEfhCT3LbfjtqUOcYT5M2Ec0f+Dc194t9Ncz8kmjwHSB7apkNowPuJ/UeOc9ljy5ia0Z3/u2l4i7apb3Tz2/e28veV+XxtYBJ/unkEsRG+vV78SFUdd8xby8rdpXx/Qh9+fvVAQmw6B9AiZfusOXg2vOsyM4YAABGoSURBVAYlO6xl4rAC4uTo77TwOBkmTZc5mhkFOk8bQTpOXb+mHKqKoLLY+t5Q00yBYoX86cEf4zrtB0JXiO7S8s8DuN1QXuAJ7l1ftVZKd0FZvhUuJ4THe0L8RJD3g8S+1nc9ed2hacD7kT8s2sbc5bt590eXnnc+9rLjddzxr7V8vrOU2yf25WdXD7RnCuI2qG9087uF23hpxV4mDOjCU18f2TFuItLY4LtRnzFQW2HN119Z5An+oibPmy4vhvqq5vcT2bn5HwbhcdYPsxNBfmSPdVnpCaFRnuD2fHXu99VoPCqxY4+E1VlpwPuRY9X1THpkKQOT4/jX9y8+62WHO4sq+d7LX1JYVsPvbxjGjNE+aoWcx2tf7uNX/95M94RI/vrNTNK6xvq6JP9RV2UFfrM/DJr8VlBZDHWeW0E6w6w2SnNBHtvGlorq0DTg/czLK/by4DtbeP6bmVw+uOsZr2fvKOLOf60jPNTBc7eNZnSvzj6osuXW5B/lB/9cw/HaBh6fNYIrm0xLrLykvto67xDtsn5DUUHDJzf8UG33jYt70tcVze8XbaO+8asZHI0xPP/pbr770pekdo7i7R9d2uHDHWB0r068+6NL6Z8Uw+x/rOFPS/Jw601EvCs00hqha7irJjTgO6BQp4NfTB7E7uIqXv1iHwC1DY38/M2NPLxwG1cNSWb+f40jJaF1V9r4UnJ8BK/dPo4bRqXw+JJcfjhvLVW1DeffUCnVZhrwHdTlg5IY27czjy/JY29JFbc+v5rXcwq467IBPP2NUd6dxbGdRIQ6+b+Zw/n1lMF8uPUQN/5lBftKj/u6LKUClvbgO7DNB44x5anPCHUKTofw6MzhTMno7uuyvOLTvGJ+9K91iMDvpg/j2oxuvi5JKb+kPXg/NTQlnlsu7klSbARv3H5JwIQ7wIQBLt750XhSO0Vxx7/W8sN5ayiprD3/hkqpFtMRfAdnjMEYcHSQ69u9raHRzdxPd/PER3lEhzt5aNoQpg3vbu+slEoFEB3B+zERCdhwBwhxOvjhpP4svOtSeiZGc/er67n9H2soqmjuE59KqdbQgFcdwoCusbz5g3H8YvJAsnOLueKx5SxYW0BH+g1TKX9jW8CLSKqILBWRrSKyRUTututYKjCEOB3cntWP9++aQD9XND95fQPfezmHw+U6mleqLewcwTcAPzXGDAbGAneIyGAbj6cCRP+kGN74wSX86tpBfL6rhCseW8YbOft1NK9UK9kW8MaYg8aYtZ7HFcA2IMWu46nA4nQI35vQl0V3T2Rgchz3zd/Id176ksKyal+XppTfaJcevIj0BkYCq5t5bbaI5IhITnFxcXuUo/xIny7RvDp7LA9NHczq3Ue46vHlvPrFPh3NK9UCtl8mKSIxwDLgd8aYBedaVy+TVOeSX1rFz9/cyKrdR5gwoAtzbszwq+kalLKDzy6TFJFQ4E1g3vnCXanz6ZUYzb++N5bfXjeENflHufKxZcxbna+jeaXOws6raAR4AdhmjHnMruOo4OJwCLeN680H90xkRM8EHnhrM7c8v5r9R3ROG6VOZ+cIfjxwG/A1EVnv+brGxuOpIJLaOYp//ufF/P76YWwsOMZVTyzn7yv36jTESjWhUxUov3egrJr739zIp3klXNynM3+ckeGzm44Huoqaet7beJA3cvZTXFnLlYOTmTq8O8N7xOv0Ej6id3RSAc8Ywxs5Bfz2va3Uu9387KqBfPuS3gE9zUN7cbsNq/cc4Y01+3l/00Fq6t0MSIohtXMUn+YVU99oSO0cyZSM7kzJ6MbgbnEa9u1IA14FjYPHqvnlgk0s3VHMRb078ccZw+nTRUfzbVFYVs2bawp4Y00B+44cJzY8hKkjunNTZurJEfux6no+3HKIdzce5POdJTS6DX1d0UzJ6M604d3on6T34LWbBrwKKsYYFqw9wP+8u4XaBjf3XpnOdy/tg1NH8+dVU9/IR1sP83rOfj7bWYIxcEm/RG7KTOWqIclEhp39loBHqupYtPkg7204yKo9pRgDA5NjmTrcGtlr28weGvAqKB0ur+GBtzaxZFsRI3sm8MiM4fRPivF1WR2OMYYtheW8nrOft9cXcqy6npSESGaM7sGM0T1I7RzV6n0Wldfw/qaDvLvxIGvyjwKQ0SOeKRnduDaju35+wYs04FXQMsbw9vpCHnp3C8frGvn+hD5MHmr1iYO9P3+kqo5/rzvA6zn72X6ogrAQB5OHJjNzdCqX9Ev02p/PgbJqFm4s5L2NB9lYcAywbsQ+NaMb1wzrRlJchFeOE6w04FXQK6qo4cG3t7Bo8yEAusSEMzGtC5PSk5g4oAsJUWE+rrB9NDS6+TSvhNdz9rNk22HqGw3De8QzIzOVaRndiY8KtfX4e0uqWLjpIO9uKGT7oQpEYGyfRKYM78bkod3oHB0cfw/epAGvlEdxRS3Lc4tZllvM8rxiyo7X4xAYnprApLQkJqW7GJYSH3Cj+93FlbyxpoAFaws4XF5LYnQY00emMDOzBwOT43xS086iCt7dcJB3Nxayu7gKp0MY378LUzO6ceWQZOIj7f1hEyg04JVqRqPbsKGgjGU7isnOLWZjQRnGQOfoMCYOsEb3EwZ0ITEm3NeltklVbQMLN1nXrH+59yhOhzApzcXMzFS+NjCJsJCOcb8fYwzbDlbw7sZC3ttYyP4j1YQ5HUxMczF1eDcuH9SV6PAQX5fZYWnAK9UCpZW1fJpXYo3uc4sprapDBDJS4slKt0b3w3skdOircYwx5OQf5fUv97Nw00GO1zXS1xXNTZmp3DAypcP3u40xbCg4xnsbrJ79ofIawkMcXDYoiWnDU/iPgS7CQ85+JU8w0oBXqpXcbsOmA8dYlltM9o4i1u8vw20gISqUCQNcTEpzMTHNhSvWd6P72oZGisprKaqopbiihrzDlSxYd4A9JVVEhzmZOrw7MzNTGdUzwS8/eOR2G9bsO8q7Gwp5f9NBSirriI8MZUpGN24Y1cNv35e3acArdYHKjtfxaV4J2Tus/n1JZS0AQ1PimJSWRFa6i5GpCYQ4L7ztcbyu4WRwHy6voaiilqKKGoo9y4oqrGVlx+vP2PbiPp25KTOVycOSiQoLnLZGQ6Obz3aW8Na6A3yw5RA19W56J0YxfWQK149MCepr7DXglfIit9uw9WD5ydH92n1lNLoNcREhTBjgIivNRVa6i65N2iHGGMprGiiuqDkZ3kUVNRw+8bi8huIK63FlbcMZxwxzOnDFhpMUF05SbDhJsRHW9zjP47hwusVHBsVVKBU19SzefIi31h1g5W7rA1WZvTpx/agUpgyz/0qgjkYDXikbHauu5/OdJWTvKGJZbjGHy63R/cDkWKLDQ6wRd3kttQ3uM7aNDHWeGtpxzYR3bDgJUaHajmhGYVk1/15/gLfWHiCvqJIwp9Wvv35kCpPSO86JZDtpwCvVTowxbD9UQfaOYj7bWYzbzRkB7ooNp2ucFdwx4SEa3F5w4tO4C9Ye4J0NByiprKNTVChTMrpzw6gURqQGbr9eA14pFTROfJhrwboDfLjlELUNbvp2iT7Zr2/L1AsdmQa8UiooldfUs3jTIRasK2DV7iMAjOndmetHpXDNsG4B8WEqDXilVNArOHqct9cXsmBtAbuKqwgLcXDFoK5cPzKFrHQXoV64AsoXNOCVUsrDGOszDla/vpAjVXV0jg5j2vDuXD8yhQw/uzuVBrxSSjWjvtHN8txiFqw7wEdbD1PX4KavK5rpI1IYnppAWtcYkuMiOnTgnyvgA+eTEEop1UqhTgeXDerKZYO6cqy6nkWbDrJg3QEe+yj35Dqx4SH07xrDgKQY0rrGMqBrLAOSYugW37GDH3QEr5RSZzhaVUfu4QpyiyrZebiC3MOV5BVVUFJZd3KdmPAQ+ifFkNY1hgFJsQzoGsOArrF0b+fg1xaNUkp5wZGqOvIOV5BXVHnye+7hypNTV8BXwX9ixN+/q/XdruD3ScCLyIvAFKDIGDO0JdtowCul/NHRqjpP2Few0/M9r6iS4oqvgj86zEl/T3snzTPaH5AUQ0pC5AUFv68CfiJQCfxdA14pFYzKjn8V/HmeNk/u4TODf3D3OF6/fVybgt4nJ1mNMctFpLdd+1dKqY4uISqMi3p35qLenU9ZfiL48w5b4V/b0GhL+8bnV9GIyGxgNkDPnj19XI1SStnvbMHvbT7/6JYxZq4xJtMYk+lyuXxdjlJKBQyfB7xSSil7aMArpVSAsi3gReQVYCWQLiIFIvKfdh1LKaXUmey8iubrdu1bKaXU+WmLRimlApQGvFJKBSgNeKWUClAdarIxESkG8n1dRwt0AUp8XYSNAvn96XvzX4H8/i7kvfUyxjT7IaIOFfD+QkRyzjb3QyAI5Pen781/BfL7s+u9aYtGKaUClAa8UkoFKA34tpnr6wJsFsjvT9+b/wrk92fLe9MevFJKBSgdwSulVIDSgFdKqQClAd8KIpIqIktFZKuIbBGRu31dk7eJiFNE1onIe76uxdtEJEFE5ovIdhHZJiLjfF2Tt4jIjz3/JjeLyCsiEuHrmi6EiLwoIkUisrnJss4i8pGI5Hm+d/JljW11lvf2iOff5UYReUtEErxxLA341mkAfmqMGQyMBe4QkcE+rsnb7ga2+boIm/wJWGyMGQgMJ0Dep4ikAHcBmZ77HzuBm31b1QV7Cbj6tGX3Ax8bYwYAH3ue+6OXOPO9fQQMNcZkALnAL7xxIA34VjDGHDTGrPU8rsAKiBTfVuU9ItIDuBZ43te1eJuIxAMTgRcAjDF1xpgy31blVSFApIiEAFFAoY/ruSDGmOXAkdMWXwe87Hn8MjC9XYvykubemzHmQ2NMg+fpKqCHN46lAd9GnhuKjwRW+7YSr3oC+Bng9nUhNugDFAN/87SgnheRaF8X5Q3GmAPAo8A+4CBwzBjzoW+rskVXY8xBz+NDQFdfFmOj7wKLvLEjDfg2EJEY4E3gHmNMua/r8QYRmQIUGWPW+LoWm4QAo4C/GGNGAlX476/4p/D0oq/D+iHWHYgWkVt9W5W9jHV9d8Bd4y0iD2C1gud5Y38a8K0kIqFY4T7PGLPA1/V40XhgmojsBV4FviYi//RtSV5VABQYY078xjUfK/ADweXAHmNMsTGmHlgAXOLjmuxwWES6AXi+F/m4Hq8SkW8DU4BbjJc+oKQB3woiIlg93G3GmMd8XY83GWN+YYzpYYzpjXWC7hNjTMCMAo0xh4D9IpLuWXQZsNWHJXnTPmCsiER5/o1eRoCcQD7NO8C3PI+/Bbztw1q8SkSuxmqPTjPGHPfWfjXgW2c8cBvW6Ha95+saXxelWuxOYJ6IbARGAL/3cT1e4fmtZD6wFtiE9f/arz/Wf5Z7Os8BrhCRPKzfWub4ssa2Ost7+zMQC3zkyZVnvXIsnapAKaUCk47glVIqQGnAK6VUgNKAV0qpAKUBr5RSAUoDXimlApQGvFJeICKTAnEGTuXfNOCVUipAacCroCIit4rIF54Pkzznmf++UkQe98yn/rGIuDzrjhCRVU3m6O7kWd5fRJaIyAYRWSsi/Ty7j2ky3/w8z6dKlfIZDXgVNERkEDALGG+MGQE0ArcA0UCOMWYIsAx40LPJ34Gfe+bo3tRk+TzgaWPMcKw5X07McDgSuAcYDPTF+uSzUj4T4usClGpHlwGjgS89g+tIrAmr3MBrnnX+CSzwzB+fYIxZ5ln+MvCGiMQCKcaYtwCMMTUAnv19YYwp8DxfD/QGPrP/bSnVPA14FUwEeNkYc8rdckTk16et19b5O2qbPG5E/38pH9MWjQomHwMzRCQJTt7jsxfW/4MZnnW+AXxmjDkGHBWRCZ7ltwHLPHfyKhCR6Z59hItIVLu+C6VaSEcYKmgYY7aKyK+AD0XEAdQDd2Dd/GOM57UirD49WFPSPusJ8N3AdzzLbwOeE5HfePYxsx3fhlItprNJqqAnIpXGmBhf16GUt2mLRimlApSO4JVSKkDpCF4ppQKUBrxSSgUoDXillApQGvBKKRWgNOCVUipA/T+CYA6Kp45EvgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YNAv4VRvIcuh"
      },
      "source": [
        "y_predicted = model.predict([X_test, X_char_test])[0]\n",
        "pred_tags = sequences_to_tag_for_crf(y_predicted)\n",
        "test_tags = sequences_to_tag(y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x0IAko4RtUQJ",
        "outputId": "ae6b29e1-fa9c-4210-f124-bcbc98a011af"
      },
      "source": [
        "print(classification_report(test_tags, pred_tags))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         art       0.00      0.00      0.00        63\n",
            "         eve       0.00      0.00      0.00        52\n",
            "         geo       0.80      0.89      0.84      7620\n",
            "         gpe       0.94      0.94      0.94      3145\n",
            "         nat       1.00      0.16      0.28        37\n",
            "         org       0.68      0.55      0.61      4033\n",
            "         per       0.79      0.73      0.76      3545\n",
            "         tim       0.88      0.84      0.86      4067\n",
            "\n",
            "   micro avg       0.82      0.80      0.81     22562\n",
            "   macro avg       0.64      0.51      0.54     22562\n",
            "weighted avg       0.81      0.80      0.80     22562\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3g-LplBROnDt"
      },
      "source": [
        "## 성능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fToBbGDztaGI",
        "outputId": "6fde94cc-3c4a-4fef-dba1-7bda0f039dfe"
      },
      "source": [
        "print(\"F1-score: {:.1%}\".format(f1_score(test_tags, pred_tags)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "F1-score: 80.6%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UN97GyqEw8oG"
      },
      "source": [
        "# BiLSTM-BiLSTM-CRF을 이용한 개체명 인식"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "utbrWJnSOp5s"
      },
      "source": [
        "## 모델링"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mif6bSyT5OXS",
        "outputId": "676a62f6-3502-4bbb-b76a-c670f3daef36"
      },
      "source": [
        "embedding_dim = 128\n",
        "char_embedding_dim = 64\n",
        "dropout_ratio = 0.3\n",
        "hidden_units = 64\n",
        "\n",
        "word_ids = Input(batch_shape=(None, None), dtype='int32', name='word_input')\n",
        "word_embeddings = Embedding(input_dim=vocab_size,\n",
        "                                        output_dim=embedding_dim,\n",
        "                                        name='word_embedding')(word_ids)\n",
        "\n",
        "char_ids = Input(batch_shape=(None, None, None), dtype='int32', name='char_input')\n",
        "char_embeddings = Embedding(input_dim=(len(char_to_index)),\n",
        "                                        output_dim=char_embedding_dim,\n",
        "                                        embeddings_initializer=RandomUniform(minval=-0.5, maxval=0.5),\n",
        "                                        name='char_embedding')(char_ids)\n",
        "\n",
        "char_embeddings = TimeDistributed(Bidirectional(LSTM(hidden_units)))(char_embeddings)\n",
        "word_embeddings = concatenate([word_embeddings, char_embeddings])\n",
        "\n",
        "word_embeddings = Dropout(dropout_ratio)(word_embeddings)\n",
        "output = Bidirectional(LSTM(units=hidden_units, return_sequences=True))(word_embeddings)\n",
        "output = TimeDistributed(Dense(tag_size, activation='relu'))(output)\n",
        "\n",
        "base = Model(inputs=[word_ids, char_ids], outputs=[output])\n",
        "model = CRFModel(base, tag_size)\n",
        "model.compile(optimizer=tf.keras.optimizers.Adam(0.001), metrics='accuracy')\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"crf_model_2\"\n",
            "__________________________________________________________________________________________________\n",
            " Layer (type)                   Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            " char_input (InputLayer)        [(None, None, None)  0           []                               \n",
            "                                ]                                                                 \n",
            "                                                                                                  \n",
            " word_input (InputLayer)        [(None, None)]       0           []                               \n",
            "                                                                                                  \n",
            " char_embedding (Embedding)     (None, None, None,   4736        ['char_input[0][0]']             \n",
            "                                64)                                                               \n",
            "                                                                                                  \n",
            " word_embedding (Embedding)     (None, None, 128)    4072832     ['word_input[0][0]']             \n",
            "                                                                                                  \n",
            " time_distributed_14 (TimeDistr  (None, None, 128)   66048       ['char_embedding[0][0]']         \n",
            " ibuted)                                                                                          \n",
            "                                                                                                  \n",
            " concatenate_2 (Concatenate)    (None, None, 256)    0           ['word_embedding[0][0]',         \n",
            "                                                                  'time_distributed_14[0][0]']    \n",
            "                                                                                                  \n",
            " dropout_6 (Dropout)            (None, None, 256)    0           ['concatenate_2[0][0]']          \n",
            "                                                                                                  \n",
            " bidirectional_7 (Bidirectional  (None, None, 128)   164352      ['dropout_6[0][0]']              \n",
            " )                                                                                                \n",
            "                                                                                                  \n",
            " time_distributed_15 (TimeDistr  (None, None, 18)    2322        ['bidirectional_7[0][0]']        \n",
            " ibuted)                                                                                          \n",
            "                                                                                                  \n",
            " crf_2 (CRF)                    [(None, None),       702         ['time_distributed_15[0][0]']    \n",
            "                                 (None, None, 18),                                                \n",
            "                                 (None,),                                                         \n",
            "                                 (18, 18)]                                                        \n",
            "                                                                                                  \n",
            " decode_sequence (Lambda)       (None, None)         0           ['crf_2[0][0]']                  \n",
            "                                                                                                  \n",
            " potentials (Lambda)            (None, None, 18)     0           ['crf_2[0][1]']                  \n",
            "                                                                                                  \n",
            " sequence_length (Lambda)       (None,)              0           ['crf_2[0][2]']                  \n",
            "                                                                                                  \n",
            " kernel (Lambda)                (18, 18)             0           ['crf_2[0][3]']                  \n",
            "                                                                                                  \n",
            "==================================================================================================\n",
            "Total params: 4,310,992\n",
            "Trainable params: 4,310,992\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D1RNtqNNK6Th"
      },
      "source": [
        "es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=4)\n",
        "mc = ModelCheckpoint('bilstm_bilstm_crf/cp.ckpt', monitor='val_decode_sequence_accuracy', mode='max', verbose=1, save_best_only=True, save_weights_only=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Wl9dIPsIzW-m",
        "outputId": "0c25dd64-984d-49cd-873a-ed6b6db68d44"
      },
      "source": [
        "history = model.fit([X_train, X_char_train], y_train_int, batch_size=128, epochs=15, validation_split=0.1, callbacks=[mc, es])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9263 - loss: 22.6385\n",
            "Epoch 00001: val_decode_sequence_accuracy improved from -inf to 0.97024, saving model to bilstm_bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 62s 195ms/step - decode_sequence_accuracy: 0.9263 - loss: 22.5786 - val_decode_sequence_accuracy: 0.9702 - val_loss: 7.3148\n",
            "Epoch 2/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9787 - loss: 4.9090\n",
            "Epoch 00002: val_decode_sequence_accuracy improved from 0.97024 to 0.98270, saving model to bilstm_bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 47s 173ms/step - decode_sequence_accuracy: 0.9787 - loss: 4.9062 - val_decode_sequence_accuracy: 0.9827 - val_loss: 3.7907\n",
            "Epoch 3/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9852 - loss: 3.0332\n",
            "Epoch 00003: val_decode_sequence_accuracy improved from 0.98270 to 0.98497, saving model to bilstm_bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 47s 174ms/step - decode_sequence_accuracy: 0.9852 - loss: 3.0313 - val_decode_sequence_accuracy: 0.9850 - val_loss: 2.9859\n",
            "Epoch 4/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9875 - loss: 2.3553\n",
            "Epoch 00004: val_decode_sequence_accuracy improved from 0.98497 to 0.98560, saving model to bilstm_bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 47s 173ms/step - decode_sequence_accuracy: 0.9875 - loss: 2.3519 - val_decode_sequence_accuracy: 0.9856 - val_loss: 2.7244\n",
            "Epoch 5/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9889 - loss: 1.9602\n",
            "Epoch 00005: val_decode_sequence_accuracy improved from 0.98560 to 0.98624, saving model to bilstm_bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 46s 172ms/step - decode_sequence_accuracy: 0.9889 - loss: 1.9582 - val_decode_sequence_accuracy: 0.9862 - val_loss: 2.5498\n",
            "Epoch 6/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9899 - loss: 1.6986\n",
            "Epoch 00006: val_decode_sequence_accuracy improved from 0.98624 to 0.98633, saving model to bilstm_bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 46s 172ms/step - decode_sequence_accuracy: 0.9899 - loss: 1.6976 - val_decode_sequence_accuracy: 0.9863 - val_loss: 2.5161\n",
            "Epoch 7/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9906 - loss: 1.5089\n",
            "Epoch 00007: val_decode_sequence_accuracy improved from 0.98633 to 0.98659, saving model to bilstm_bilstm_crf/cp.ckpt\n",
            "270/270 [==============================] - 46s 170ms/step - decode_sequence_accuracy: 0.9906 - loss: 1.5104 - val_decode_sequence_accuracy: 0.9866 - val_loss: 2.5450\n",
            "Epoch 8/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9912 - loss: 1.3638\n",
            "Epoch 00008: val_decode_sequence_accuracy did not improve from 0.98659\n",
            "270/270 [==============================] - 46s 172ms/step - decode_sequence_accuracy: 0.9912 - loss: 1.3640 - val_decode_sequence_accuracy: 0.9865 - val_loss: 2.4838\n",
            "Epoch 9/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9917 - loss: 1.2333\n",
            "Epoch 00009: val_decode_sequence_accuracy did not improve from 0.98659\n",
            "270/270 [==============================] - 46s 171ms/step - decode_sequence_accuracy: 0.9917 - loss: 1.2339 - val_decode_sequence_accuracy: 0.9863 - val_loss: 2.5875\n",
            "Epoch 10/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9920 - loss: 1.1323\n",
            "Epoch 00010: val_decode_sequence_accuracy did not improve from 0.98659\n",
            "270/270 [==============================] - 46s 172ms/step - decode_sequence_accuracy: 0.9920 - loss: 1.1335 - val_decode_sequence_accuracy: 0.9865 - val_loss: 2.6285\n",
            "Epoch 11/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9925 - loss: 1.0415\n",
            "Epoch 00011: val_decode_sequence_accuracy did not improve from 0.98659\n",
            "270/270 [==============================] - 46s 170ms/step - decode_sequence_accuracy: 0.9925 - loss: 1.0404 - val_decode_sequence_accuracy: 0.9860 - val_loss: 2.7645\n",
            "Epoch 12/15\n",
            "270/270 [==============================] - ETA: 0s - decode_sequence_accuracy: 0.9928 - loss: 0.9524\n",
            "Epoch 00012: val_decode_sequence_accuracy did not improve from 0.98659\n",
            "270/270 [==============================] - 46s 169ms/step - decode_sequence_accuracy: 0.9928 - loss: 0.9524 - val_decode_sequence_accuracy: 0.9861 - val_loss: 2.8831\n",
            "Epoch 00012: early stopping\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DZcu-pPrNqpz",
        "outputId": "b14ea385-0598-47de-8ef4-6d9e09258b24"
      },
      "source": [
        "model.load_weights('bilstm_bilstm_crf/cp.ckpt')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7fc730fe7210>"
            ]
          },
          "metadata": {},
          "execution_count": 114
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T_ECUaJ2Nuoj",
        "outputId": "e33f23f1-52fd-45a5-b547-52d321aa9b85"
      },
      "source": [
        "i = 13 # 확인하고 싶은 테스트용 샘플의 인덱스.\n",
        "\n",
        "# 입력한 테스트용 샘플에 대해서 예측 y를 리턴\n",
        "y_predicted = model.predict([np.array([X_test[i]]), np.array([X_char_test[i]])])[0]\n",
        "true = np.argmax(y_test[i], -1) # 원-핫 벡터를 정수 인코딩으로 변경.\n",
        "\n",
        "print(\"{:15}|{:5}|{}\".format(\"단어\", \"실제값\", \"예측값\"))\n",
        "print(35 * \"-\")\n",
        "\n",
        "for word, tag, pred in zip(X_test[i], true, y_predicted[0]):\n",
        "    if word != 0: # PAD값은 제외함.\n",
        "        print(\"{:17}: {:7} {}\".format(index_to_word[word], index_to_ner[tag], index_to_ner[pred]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "단어             |실제값  |예측값\n",
            "-----------------------------------\n",
            "the              : O       O\n",
            "statement        : O       O\n",
            "came             : O       O\n",
            "as               : O       O\n",
            "u.n.             : B-org   B-org\n",
            "secretary-general: I-org   I-org\n",
            "kofi             : B-per   B-per\n",
            "annan            : I-per   I-per\n",
            "met              : O       O\n",
            "with             : O       O\n",
            "officials        : O       O\n",
            "in               : O       O\n",
            "amman            : B-geo   B-geo\n",
            "to               : O       O\n",
            "discuss          : O       O\n",
            "wednesday        : B-tim   B-tim\n",
            "'s               : O       O\n",
            "attacks          : O       O\n",
            ".                : O       O\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "3JeErjaCNyp6",
        "outputId": "d3dd1345-0925-4b55-c43b-cf2a6ebafbb0"
      },
      "source": [
        "epochs = range(1, len(history.history['val_loss']) + 1)\n",
        "plt.plot(epochs, history.history['loss'])\n",
        "plt.plot(epochs, history.history['val_loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'val'], loc='upper left')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEWCAYAAABsY4yMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3yV5f3/8dfnnJzsBSEQSAIJspEliCAIKmBxorUKFay2WtpvtY7a9mtr16/1a21rh6turVXqKI5aF4IKOBgGZO9pAiSEQBbZOdfvj/sOJCGBhJyT+4zP8/E4j3Pnnp/DeJ8r133f1y3GGJRSSoUel9MFKKWU8g8NeKWUClEa8EopFaI04JVSKkRpwCulVIjSgFdKqRClAa8UICL/EJF727juHhGZ2tH9KOVvGvBKKRWiNOCVUipEacCroGF3jfxERNaJyFEReUZEeojIeyJSJiKLRKRLo/WvEJGNIlIsIotFZHCjZaNEZLW93StAdLNjXSYia+xtPxeR4adZ83dFZIeIHBaRt0Sklz1fROSvInJQREpFZL2InGkvu0RENtm17RORH5/WH5gKexrwKthcDUwDBgCXA+8BPwdSsf493wYgIgOAl4A77GXvAv8VkUgRiQTeBF4AugL/tveLve0o4Fnge0AK8ATwlohEtadQEbkQ+D1wLdAT2Au8bC++CJhkf44ke50ie9kzwPeMMQnAmcBH7TmuUg004FWwedgYU2CM2Qd8AqwwxnxpjKkC3gBG2evNBN4xxiw0xtQCDwAxwLnAOMAD/M0YU2uMmQ980egYc4EnjDErjDH1xpjngWp7u/aYDTxrjFltjKkGfgaMF5EsoBZIAAYBYozZbIw5YG9XCwwRkURjzBFjzOp2HlcpQANeBZ+CRtOVLfwcb0/3wmoxA2CM8QK5QLq9bJ9pOtLe3kbTfYC77O6ZYhEpBjLt7dqjeQ3lWK30dGPMR8AjwKPAQRF5UkQS7VWvBi4B9orIEhEZ387jKgVowKvQtR8rqAGrzxsrpPcBB4B0e16D3o2mc4H/M8YkN3rFGmNe6mANcVhdPvsAjDEPGWNGA0Owump+Ys//whgzA+iO1ZX0ajuPqxSgAa9C16vApSIyRUQ8wF1Y3SyfA8uAOuA2EfGIyNeBsY22fQr4voicY58MjRORS0UkoZ01vAR8W0RG2v3392F1Ke0RkbPt/XuAo0AV4LXPEcwWkSS7a6kU8Hbgz0GFMQ14FZKMMVuBOcDDwCGsE7KXG2NqjDE1wNeBG4HDWP31rzfaNgf4LlYXyhFgh71ue2tYBPwSeA3rt4YzgFn24kSsL5IjWN04RcCf7GXXA3tEpBT4PlZfvlLtJvrAD6WUCk3agldKqRClAa+UUiFKA14ppUKUBrxSSoWoCKcLaKxbt24mKyvL6TKUUiporFq16pAxJrWlZQEV8FlZWeTk5DhdhlJKBQ0R2dvaMu2iUUqpEKUBr5RSIUoDXimlQlRA9cG3pLa2lry8PKqqqpwuxa+io6PJyMjA4/E4XYpSKkQEfMDn5eWRkJBAVlYWTQf/Cx3GGIqKisjLyyM7O9vpcpRSISLgu2iqqqpISUkJ2XAHEBFSUlJC/rcUpVTnCviAB0I63BuEw2dUSnWuoAj4kzJeKCuAqlKnK1FKqYAS/AGPwNGDUHnEL3svLi7m73//e7u3u+SSSyguLvZDRUop1TbBH/AiEBkP1WXgh7HtWwv4urq6k2737rvvkpyc7PN6lFKqrQL+Kpo2iUqAqmKoqwZPtE93fffdd7Nz505GjhyJx+MhOjqaLl26sGXLFrZt28aVV15Jbm4uVVVV3H777cydOxc4PuxCeXk5F198MRMnTuTzzz8nPT2d//znP8TExPi0TqWUai6oAv7//Xcjm/a30NduvFBbARErwdW+68iH9Erk15cPbXX5/fffz4YNG1izZg2LFy/m0ksvZcOGDccuZ3z22Wfp2rUrlZWVnH322Vx99dWkpKQ02cf27dt56aWXeOqpp7j22mt57bXXmDNnTrvqVEqp9gqqgG+VuKyuGm99uwO+vcaOHdvkWvWHHnqIN954A4Dc3Fy2b99+QsBnZ2czcuRIAEaPHs2ePXv8WqNSSkGQBfzJWtoc2QtVJZA2zAp7P4mLizs2vXjxYhYtWsSyZcuIjY3l/PPPb/Fa9qioqGPTbrebyspKv9WnlFINgv8ka4OoBDD1UOvb8ExISKCsrKzFZSUlJXTp0oXY2Fi2bNnC8uXLfXpspZTqCL+14EVkIPBKo1l9gV8ZY/7mlwNGJVjvNWUQGeuz3aakpDBhwgTOPPNMYmJi6NGjx7Fl06dP5/HHH2fw4MEMHDiQcePG+ey4SinVUWL8cGnhCQcRcQP7gHOMMa0OTj9mzBjT/IEfmzdvZvDgwW070MHN4PZASr8OVOucdn1WpZQCRGSVMWZMS8s6q4tmCrDzZOHuE1HxUHPUuqpGKaXCXGcF/CzgpZYWiMhcEckRkZzCwsKOHSUywQr3moqO7UcppUKA3wNeRCKBK4B/t7TcGPOkMWaMMWZMamqLz41tu6h467265ZOiSikVTjqjBX8xsNoYU+D3I7kiwBNrnWhVSqkw1xkB/01a6Z7xi6h4q4vGW99ph1RKqUDk14AXkThgGvC6P4/TRGQCYKyTrUopFcb8GvDGmKPGmBRjTIk/j9NEZBwgjvXDx8fHO3JcpZRqLnTuZG3gclshr/3wSqkwF1Rj0bRZVAKUHYD6OnB37CPefffdZGZmcssttwDwm9/8hoiICD7++GOOHDlCbW0t9957LzNmzPBF5Uop5TPBFfDv3Q3560+9nqm3hw+Osa6sOZm0YXDx/a0unjlzJnfcccexgH/11VdZsGABt912G4mJiRw6dIhx48ZxxRVX6HNVlVIBJbgCvq3EBQh4604d8KcwatQoDh48yP79+yksLKRLly6kpaVx5513snTpUlwuF/v27aOgoIC0tDTf1K+UUj4QXAF/kpb2CYp2Wk946jGkw4e95pprmD9/Pvn5+cycOZN58+ZRWFjIqlWr8Hg8ZGVltThMsFJKOSn0TrI2iIqH+mqor+nwrmbOnMnLL7/M/PnzueaaaygpKaF79+54PB4+/vhj9u717xA7Sil1OoKrBd8ekfbwwdXlENu1Q7saOnQoZWVlpKen07NnT2bPns3ll1/OsGHDGDNmDIMGDfJBwUop5VuhG/CeGBC3dT18BwMeYP364yd3u3XrxrJly1pcr7y8vMPHUkopXwjdLhoR63LJ6jLohDHvlVIq0IRuwIPVD++ttfrilVIqzARFwJ/2U6caHuMXBMMHd8aTtZRS4SXgAz46OpqioqJWA7Dea9hbdJTDR1u4WsYdBS6PdaI1gBljKCoqIjo62ulSlFIhJOBPsmZkZJCXl8fJnvZUUFqFAN0TWwjIimKozYfESqtfPkBFR0eTkZHhdBlKqRAS8AHv8XjIzs4+6TpfLNvDr/6zkbduncDwjOSmC9e+DP/9HnzvE+g53H+FKqVUgAn4Lpq2uGpUOrGRbl5c3sINR9mTrPfdSzq3KKWUclhIBHxCtIcZI3vx1tr9lFTUNl2Y2AtS+sPupc4Up5RSDgmJgAeYfU4fqmq9vLY678SFfSfD3s+hvvbEZUopFaJCJuDPTE9iZGYy81bsPfGKm+zJUFMO+1Y5U5xSSjkgZAIeYM64PuwsPMryXYebLsiaCIh20yilwkpIBfxlw3uSFOPhxRXNTrbGdrWuoNmlJ1qVUuEjpAI+2uPmG6MzWLAhn4NlzcZnz54MeSuhpsKZ4pRSqpP5NeBFJFlE5ovIFhHZLCLj/Xk8gNnn9KbOa3j1i9ymC7InW2PD5y73dwlKKRUQ/N2CfxB43xgzCBgBbPbz8eibGs+Efim8tDKXem+jk619xlvDFmg3jVIqTPgt4EUkCZgEPANgjKkxxhT763iNzTmnD/uKK1m89eDxmZFxkHG23vCklAob/mzBZwOFwHMi8qWIPC0icc1XEpG5IpIjIjknG2+mPaYO6UH3hKgT72zNngQH1kLlEZ8cRymlApk/Az4COAt4zBgzCjgK3N18JWPMk8aYMcaYMampqT45sMftYtbZmSzeVkju4UYnVftOBuOFPZ/55DhKKRXI/BnweUCeMWaF/fN8rMDvFLPG9kaAl1Z+dXxm+hjwxGo3jVIqLPgt4I0x+UCuiAy0Z00BNvnreM31So5hyuAevJqTS02d15oZEQl9ztUbnpRSYcHfV9H8EJgnIuuAkcB9fj5eE7PP6c2h8hre35h/fGb2JCjcAmX5rW+olFIhwK8Bb4xZY/evDzfGXGmM6dSzm5P6p9K7a2zTk63Zk613bcUrpUJcSN3J2pzLJVx3Tm9W7j7MtgL7uaxpwyE6WfvhlVIhL6QDHuCa0RlEul38a4V9stXlguzzYNdS0AddK6VCWMgHfEp8FBcPS+O1VXlU1NRZM7MnQ8lXcGS3s8UppZQfhXzAgzWMcFl1HW+t2W/N6Hu+9a7DFiilQlhYBPyYPl0Y2COBFxseBpLSDxJ66olWpVRIC4uAFxHmjOvNhn2lrMsrARGrm2b3UvB6nS5PKaX8IiwCHuDKUenERrqPXzLZdzJUHIKDnXbvlVJKdaqwCfiEaA8zRqbz33X7KamotW54Au2mUUqFrLAJeIA543pTVetl/uo8SMqArmfo9fBKqZAVVgE/tFcSo3onM6/hZGvfydbIkvV1TpemlFI+F1YBDzD7nD7sKjzKsl1FVjdNTRns/9LpspRSyufCLuAvG96TpBgP85Z/BVkN/fCLHa1JKaX8IewCPtrj5prRGSzYmM/B+jhIG6Y3PCmlQlLYBTzA7HF9qPMaXs3Jta6Hz10JtZVOl6WUUj4VlgGf3S2Oif268dLKXOqzJkF9NeSuOPWGSikVRMIy4MF6GMi+4kqWVvUHV4R20yilQk7YBvzUIT3onhDF86sPWc9q1RuelFIhJmwD3uN2MWtsb5ZsK6Sk53jYvxqqSpwuSymlfCZsAx7gm2MzEeCd8gFgvNZNT0opFSLCOuB7JsUwZXAPHtqSjImI0W4apVRICeuAB+thIPkVhsIuo3RcGqVUSPFrwIvIHhFZLyJrRCTHn8c6Xef160aflFgWVg6yhg4uP+h0SUop5ROd0YK/wBgz0hgzphOO1W4ul3Dd2N68XNTXmqHdNEqpEBH2XTQA3xidwXZXXyrdCdpNo5QKGf4OeAN8ICKrRGRuSyuIyFwRyRGRnMLCQj+X07KU+CimD+vF53WD8OoNT0qpEOHvgJ9ojDkLuBi4RUQmNV/BGPOkMWaMMWZMamqqn8tp3ZxxfVhSNwRX8V44ssexOpRSylf8GvDGmH32+0HgDWCsP4/XEaP7dCG/q1We0Va8UioE+C3gRSRORBIapoGLgA3+Ol5HiQjnjZ9AgUmmeOMip8tRSqkO82cLvgfwqYisBVYC7xhj3vfj8TrsyrMyWMmZuL/6FIxxuhyllOqQCH/t2BizCxjhr/37Q0K0h5rMiSTmfUpp7noSew93uiSllDpteplkM8PPmwHAxk/fcrgSpZTqGA34ZvoPHMIBd0+8O5dgtJtGKRXENOBbUJk+kWF161m2vcDpUpRS6rRpwLcgY/TXSJRKPvlEr6ZRSgUvDfgWRPa7AAD37qUcLK1yuBqllDo9GvAtietGdcoQzpGNvPJFrtPVKKXUadGAb0VU//M5x72N11bsoN6rJ1uVUsFHA7412ZOJpIae5ev5aIuOEa+UCj4a8K3pcy5G3EyL3sqLy/c6XY1SSrWbBnxrohOR9LOYHreVpdsL+aqowumKlFKqXTTgTyZ7Mj3LN5Eolfxr5VdOV6OUUu2iAX8yfScjpp6bMw/wak4u1XX1TleklFJtpgF/MhljISKaq5J3cvhoDe9vyHe6IqWUajMN+JPxREPmOaQXr6RPSqyebFVKBRUN+FPpOxkp2Mh3RsbzxZ4jbM0vc7oipZRqEw34U8meDMDVXXcTGeFi3gptxSulgoMG/Kn0HAlRicTv/4xLh/Xk9dX7OFpd53RVSil1Sm0KeBG5XUQSxfKMiKwWkYv8XVxAcEdA1kTYtYQ543pTXl3Hf9bsd7oqpZQ6pba24L9jjCnFenB2F+B64H6/VRVosifDkd2clVjGoLQE5q3Yqw8DUUoFvLYGvNjvlwAvGGM2NpoX+rInASB7PmH2uD5s3F/Ksl1FDhellFIn19aAXyUiH2AF/AIRSQC8/isrwHQfDHHdYdcSrhmdQa+kaO57dzNeHWVSKRXA2hrwNwF3A2cbYyoAD/DttmwoIm4R+VJE3j7NGp0nYrXidy8lOsLFT6cPYsO+Ut74cp/TlSmlVKvaGvDjga3GmGIRmQP8Aihp47a3A5tPp7iAkj0JyvPh0DauGNGLERlJ/GnBVipq9IoapVRgamvAPwZUiMgI4C5gJ/DPU20kIhnApcDTp11hoOhrXQ/PriW4XMIvLhtCfmkVTy3d7WxdSinVirYGfJ2xLhuZATxijHkUSGjDdn8DfspJ+utFZK6I5IhITmFhYRvLcUCXLEjuA7uXAHB2VlcuGZbG40t2UqDPbVVKBaC2BnyZiPwM6/LId0TEhdUP3yoRuQw4aIxZdbL1jDFPGmPGGGPGpKamtrEch2RPgj2fgNcaVfJ/pw+i3mt4YMFWhwtTSqkTtTXgZwLVWNfD5wMZwJ9Osc0E4AoR2QO8DFwoIi+ebqEBoe/5UFUCB9YC0CcljhsnZDF/dR4b9rX1lIRSSnWONgW8HerzgCS7ZV5ljDlpH7wx5mfGmAxjTBYwC/jIGDOnowU7yr4ent1Lj8265YJ+JMd4uPedTXrzk1IqoLR1qIJrgZXANcC1wAoR+YY/CwtI8d0hdfCxfniApBgPd04bwPJdh1m4qcDB4pRSqqm2dtHcg3UN/A3GmG8BY4FftvUgxpjFxpjLTqfAgNN3MuxdBnXVx2ZdN7Y3/brH8/v3tlBTFz73fymlAltbA95ljDnY6OeidmwbWvpNhbpKePtOqK8FIMLt4p5LBrP70FF9KIhSKmC0NaTfF5EFInKjiNwIvAO867+yAli/qTD5blgzD+ZdA1WlAJw/MJXz+nfjwQ+3U1xR43CRSinV9pOsPwGeBIbbryeNMf/rz8IClghc8DOY8ah1yeRzF0PpfkSEey4dTFlVLQ99uMPpKpVSqu3dLMaY14wxP7Jfb/izqKAwag5c9yoc2QNPT4WCjQxKS2Tm2Zn8c9kedhWWO12hUirMnTTgRaRMREpbeJWJSGlnFRmw+k2Bb78HxgvPToddi7lz2gCiIlzc/94Wp6tTSoW5kwa8MSbBGJPYwivBGJPYWUUGtJ7D4eZFkJQBL15N951v8IML+vHBpgKW7dQx45VSzgnPK2F8LSnDasn3ORfe/D5zeY1eiVHc+84mHTNeKeUYDXhfiUmG2a/B8Fl4ltzHv3q+xNb9h3ldx4xXSjlEA96XIiLhqsdh0k/I2jufVxIe5NH3v9Qx45VSjtCA9zURuPAXcPmDnFW3hkeq72HewhVOV6WUCkMa8P4y+kbkulfo5y7g0pXf4tDONU5XpJQKMxrw/tR/Goev/Q8R1BE379Imo1AqpZS/acD7Wc/B4/j3qOfIq0vC+8LXYd2/nS5JKRUmNOA7wfXTz+PmiP9jc8QgeP1m+OTPoGPHK6X8TAO+EyRGe7j5orO4qvTHHMi8DD78rT0apV5do5TyHw34TvLNszPp3b0L1xV9h/pz74BVz8HL34RqHbNGKeUfGvCdJMLt4p5LB7P7cBX/iL0RLv0L7FgE/7gUyvRJUEop39OA70TnD7DGjH/ow+0UD70eZr0Eh7ZZo1EWbnW6PKVUiNGA70SNx4x/8MPtMHA63PiO9YSoZy6CPZ85XaJSKoRowHcya8z43rywbK81Znz6WdZolHGp8MKVsH6+0yUqpUKEBrwDfmSPGf/7hjHju2TBTR9A+mh47Sb47EG9jFIp1WF+C3gRiRaRlSKyVkQ2isj/89exgk1qQhQ/uKAfCzcV8PnOQ9bM2K5w/Zsw9CpY+Ct498fgrXe2UKVUUPNnC74auNAYMwIYCUwXkXF+PF5QuWliNunJMdz79mbqG8aM90TD1c/CubfBF0/Dy7Oh5qizhSqlgpbfAt5YGi7y9tgv7XewRXvc/HT6QDYdKOX11XnHF7hccNHv4JIHYPsC+MdlUH7QuUKVUkHLr33wIuIWkTXAQWChMeaEcXNFZK6I5IhITmFhoT/LCThXjOjFqN7J/GnBVo5WN7urdex3YeaLcHAzPDwaXv0WrH4BSg84U6xSKuiI6YSTeSKSDLwB/NAYs6G19caMGWNycnL8Xk8gWbX3CFc/9jm3TenPj6YNOHGFA2th5VPWTVFldrj3GAb9p0K/aZA5Ftyezi1aKRUwRGSVMWZMi8s6I+DtIn4FVBhjHmhtnXAMeIBb/7WaRZsL+PjH59MzKabllYyBgo2wYyFsXwS5y8FbB1GJ0HeyFfb9p0Fir84tXinVccZYDws6DY4EvIikArXGmGIRiQE+AP5gjHm7tW3CNeBzD1cw5S9LuGx4T/5y7ci2bVRVCrsWHw/8sv3W/O5Dj7fue4/T1r1SgcAYqDgMxXvgyF4o3tv03eWGW784rV2fLOAjOlLzKfQEnhcRN1Zf/6snC/dwltk1lu9MyObxJTu58dwshmckn3qj6EQYcoX1Msbqq9+xELYvhGV/t66lj0ywWvf9p1mBn5Tu/w+jVLiqLofir+zQ3tMsyL+CmrKm68d0hS59IG0YpJzhl5I6rYumLcK1BQ9QWlXLBX9azBnd43ll7jjkNH9dA6C6DHYtOd66L7Wv0uk+BPpNtV69x1sPCVdKtU1dDZTkntj6bnivONR0fU8sJPexQryl9+hEn5QVEH3wbRHOAQ/w4vK9/OLNDTw+ZzTTz0zzzU6NgcItVst+x0LYuwy8tRAZD9mTj3fnJGf65nhKBbOKw1C0Ew7vPLEVXrYfjPf4uq4ISMpsIcCzrPe4bqfdr94eGvBBoq7ey8UPfkJNvZeFd04mMsIPV7FWl1nPht2xyGrdl3xlzU8dZLXs+0+DjLEQGev7YysVCGoq4PAuKNphv3Yen6483HTdhJ4ntr4bAjyxl9V37jAN+CCyeOtBbnzuC35x6WBuPq+vfw9mjDVc8bHW/edQXwPigpT+Vt9gz+GQZr/iUvxbj1K+Ul9r9XsfC/GGIN95vMuyQUIvqw885QxI6We9up4Byb2tu8sDnAZ8kPnWsytZ89URlvzkArrEdWI/eXU57PkU9q+G/PVwYF3T/wyJ6XbYNwr+5N6d8muoUicwxro3pKWW+JE91mXEDaKTrEZLQ4A3hHnXvhAV79hH8AUN+CCzNb+Mix9cyrfGZ/GbK4Y6W8zRIshfZwV+/jor9Iu2H++LjE463sLvaYd/twF6eaY6ffW11hhMNUehtgJqyq3pkrxmLfJdUNtorKaIGCu4u/ZtFOT2K7ZryDZEnLpMUp2mgWkJzBrbmxeX7+X68X04I9XBFkZcCpxxgfVqUFMBBzdZd9k2BH/OM1BXZS13R0GPIVbYpw2HniOgx1CIjHPmMyjfM8b6+24I4iZhXNF0uuaoFcQ1R+1ljaZryu11G+2jvqb144rb6gdP6QdZ5zXtVknoZY3lpI7RFnyAKiyr5oIHFjOubwpP39Dil3Ngqa+zWlX565oGf+URewWx/hM2tPIbgj+um6Nlq1ZUl0HJPii1XyX7rO66kkY/17ZjpFOXx/qCb3h5YptNx1sn9iPjwBPXbNr+OTHdOrmpl/c2oS34IJSaEMUtF/TjD+9v4fMdhzi3X4AHoTsCug+yXsOvteYZY/1a3bh7J/cL2PDa8e3iulvXA0fEgCfGOqnliYWIaPvnmKbLjk03XnaSbSKiT96q83qtlmhdFdRVt/G9qu3bRERDdDLEJB9/j+ly4rzo5M47oVdbCaX7rb+b1sK7uqTZRgLxPayb5RquuIpLbSGYG8I53g7uWGu+hrIjtAUfwKpq65ny5yXERrp585YJxEWFyPdxxWE79Ndb1+jXHLXCsLbSetXZ77VVjaYrml6D3B7uqOPBL26or7b3XWXdE9AhYn+RRFlh3vjdHWXVX1kCVcVQXXryXZ3wZdDlxC+B1t4bvhzqaqzrtY+FdeMQt18VRSceO7abFd6JGfZ7r0bT6dblghrSAUlPsgaxpdsKufG5lVw4qAdPXD8atys0TxSdkjHWybc6O/hrK+wvhYpmXwSNpxt/YVQd/5KIiLJa+C2FcovvLczz2PNdEW0/eVdfZ4V85REr8CuPQGWxPd3ae1u/HOwvsMojnPDYhehkSMqwQzu9WZDbryC4HFC1TLtogtikAan8+vKh/Pqtjfzh/S38/JLBTpfkDBGrBRkRaV25E4zcEdbVHLFd279tfR1UlTQK/xa+HGoqIL570xBP7BX0lwGq06cBHwRuODeLnYXlPLl0F327xTFrbG+nS1KdzR1hXdGkN5updtCADxK/umwIe4oq+MWbG+idEsu5ZwT4SVellOP0otEgEeF28ch1o8juFsf/vLiaXYXlp95IKRXWNOCDSGK0h2dvPBu3S7jp+RyKK05yQ4hSKuxpwAeZzK6xPHn9aPYdqeT7L66ipu40Lx1USoU8DfggNCarK3/4xjCW7zrML95cTyBd6qqUChx6kjVIXTUqg12FR3n4ox306x7P3En+eeSXUip4acAHsTunDmBX4VF+/94WslLiuGioj54CpZQKCdpFE8RcLuGBa0YwPD2J219ew4Z9zccPUUqFMw34IBcT6eapG8bQJdbDzc/nUFBa5XRJSqkA4beAF5FMEflYRDaJyEYRud1fxwp33ROiefqGsymtquW7/8yhsqbe6ZKUUgHAny34OuAuY8wQYBxwi4gM8ePxwtqQXok8NGsU6/eV8KNX1+D16pU1SoU7vwW8MeaAMWa1PV0GbAbS/XU8BVOH9OCeSwbz3oZ8/rxwq9PlKKUc1ilX0YhIFjAKWNHCsrnAXIDevXUQrY66aWI2OwvLefTjnfTtFs/VozOcLkkp5RC/n2QVkXjgNeAOY8wJg1obY540xowxxoxJTU31dzkhT0T47YwzOfeMFO5+fR0rdx92uj/6YxcAAA+FSURBVCSllEP8GvAi4sEK93nGmNf9eSx1nMft4rHZo8nsEsv3Xshhb1E7np2pfCL3cAX3vbuZeSv2Uluvw0koZ/jtiU4iIsDzwGFjzB1t2Uaf6ORbuw8d5aq/f0ZKXCSv/2ACSTEep0sKeQWlVTz80XZe+SKXeq/Ba6Bvahx3Tx/EtCE9kLY+/UmpNjrZE5382YKfAFwPXCgia+zXJX48nmomu1scj88ZzVeHK7j1X6u1JelHReXV3Pv2Jib98WNeXpnLzLMz+fzuKTz1Lev/3dwXVjHzyeWszS12uFIVTvSZrGHg1Zxcfjp/HbPP6c29V56prUgfKqms5amlu3j2s91U1dbz9bMyuH1KfzK7xh5bp7bey8tf5PLgom0cKq/h8hG9+OnXBjZZR6nTpc9kDXPXjslkZ2E5TyzZRb/u8Xx7QrbTJQW9o9V1PPfZbp5cuovSqjouHd6TO6cOoF/3E59/6nG7uH5cH64alc4TS3by1Ce7WLAhnxvO7cOtF/QnKVa7zpR/aAs+THi9hu+/uIpFmwt45oazuWBQd6dLCkpVtfW8uHwvjy3eSdHRGqYO7s6Ppg1kSK/ENu/jQEklf/lgG/NX55EY7eGHF/bj+vF9iIpw+7FyFapO1oLXgA8jFTV1XPP4MvYWVTD/f8YzKK3toRTuauq8vJKTyyMfbaegtJqJ/bpx10UDGNW7y2nvc9P+Un7/3mY+2X6IzK4x/PRrg7hseE/tQlPtogGvjskvqWLGo58S4XLx5i0TSE2IcrqkgFZX7+WNL/fx4IfbyTtSyZg+XbjrooGMPyPFZ8dYuq2Q+97dzJb8MkZkJnPPJYMZm93VZ/tXoU0DXjWxPq+Ea574nME9E3npu+OI9mjXQHNer+Gd9Qf466Jt7Co8yrD0JO66aACTB6T6pYVd7zW8vjqPBz7YSkFpNdOG9ODuiwdxRuqJffpKNaYBr07w/oYDfP/F1Vw+ohcPzRqp3QI2YwyLNh/kzx9sZUt+GQN6xPOjaQP52tDOuYa9sqaeZz7dxWOLd1JV5+W6sb25fWp/usXrb1qqZRrwqkV/X7yDP76/lTum9ueOqQOcLsdRxhg+21HEAx9sZU1uMX1SYrlz6gAuH9ELt6vzv/wOlVfz4KLt/GvlV8R43Hx/cl9umtiXmEj9bUs1pQGvWmSM4cf/Xsdrq/N4cNZIZowMz8E+v9hzmAcWbGXF7sP0Sormtin9uXp0Bh6388/D2XGwnD+8v4WFmwpIS4zmrosG8PWzMhz50lGBSQNetaq6rp7rn17JmrxiXp47jrM6cFVIsFmXV8yfP9jGkm2FdIuP4tYLzuCb5/QOyMsVV+wq4r53N7M2r4RBaQn8/JLBTBqgg/MpDXh1CoeP1nDV3z/jaHUdb94ygYwuoX2H5db8Mv6ycCsLNhaQHOvh+5PP4IbxWQHf/eH1Gt5ef4A/vr+FvCOVnNe/Gz+/ZDCDe4bW5a5lVbUUlFaRX1JNfmkVBaVVFJZVM2Vwd87rr19qzWnAq1PacbCcq/7+Gb2SYvjrzJFEeVxEuAS3S4hwuex3we0W3CLHf3ZJp52gNcZQ5zXU1Rtq6r3U1XuprTfU1nvt16mnP9pykLfW7icuMoKbz8vmponZJEQH152k1XX1vLBsLw9/tIPSqlq+cVYGd100kLSkaKdLO6l6r6Go3Art/BIruA+UVB0LcWteNeXVdSdsGxnhorbeyx1TBvDDC/vh0i6qYzTgVZt8uv0QNzy3kvp2Pu7P7Woa+NZ7oy8IdyvzXYJLhDpva4F8fF5DqHdUtMfFjedm871JfekSF9nh/TmpuKKGRz7awT+X7cXlghvGZ5HdLY7ICBcet4vICOsV1Wg6MsJFpLvZuz0d0YFzDlW19eTbYd34vaDUDvCSKgrKqk/4txXhEronRNEjKZq0xGh6JEbTMymatCRrumEewD1vrOf1L/cxZVB3/jJzpI6OatOAV222+UApuwqPUuf1Uu+1WszH3uu9TX8+9m7N956wftPlLW1X7zV43FYgRbgET0PYtDDtcYm1boQ1ryHImk9b67rwuBtNR1jbpiZEkRhkLfZTyT1cwR8XbOW/a/d3aD8uoVHwu4mKsP4MT/hiiHAT6XZRU++lwA7zksraE/YXHxVBj8Qo0pKiSUuMIS0p6lhgp9khnhIX1eYTxsYY/rlsL797exMZXWJ4/PrRejc2GvBKhYWSyloqauqoqfNSU+elus5LTb332M+1DdP19rK64z+fMN14XrOfq+1pj1uOtbIbWtw9k44HeHyUf8YyzNlzmB/MW01ZVR33Xz0sbK/+aqCjSSoVBpJiPGHRbTEmqytv/3Ait/xrNbe/vIZ1eSXcffGggLisNdDon4hSKuh0T4zmX98dx43nZvHMp7uZ/fQKCsuqnS4r4GjAK6WCksft4jdXDOVvM0eyLq+Yyx7+hNVfHXG6rICiAa+UCmpXjkrn9f+ZQFSEm5lPLOOF5XsJpHOLTtKAV0oFvSG9EvnvrROZ2K8bv3xzAz+Zv46q2nqny3KcBrxSKiQkxXp45oazuX1Kf+avyuMbj39O7uEKp8tylAa8UipkuFzCndMG8MwNY9hbVMHlj3zK0m2FTpflGL8FvIg8KyIHRWSDv46hlFItmTK4B/+9dSI9EqK54bmVPPrxjrDsl/dnC/4fwHQ/7l8ppVqV1S2ON245l8uH9+JPC7byvRdWUVZ14h23ocxvAW+MWQoc9tf+lVLqVGIjI3hw1kh+edkQPtxykBmPfMb2gjKny+o02gevlAppIsJNE7OZd/M5lFbVMuPRz3h3/QGny+oUjge8iMwVkRwRySksDN+TIUop/xrXN4W3f3geA9MS+MG81fz+3c3U+WCE0tNhjOGrogreWruf3729iZ/OX+uX4zg+Fo0x5kngSbAGG3O4HKVUCEtLiuaVueP53dubeGLpLtbvK+Hhb44ixc8PNS8qr2ZdXglrcotZm1fM2txijlRY5wOiIlyMzEzG6zU+H+fe8YBXSqnOFBnh4ndXnsnwjCTueXMDlz/8KY/NGc2IzGSf7L+ypp4N+0tYm1t8LNBzD1cCIAIDuicwbUgPRmQmMyIjmYFpCX4bKM1vwwWLyEvA+UA3oAD4tTHmmZNto8MFK6U604Z9JXzvhVUUllXz2xlDmTW2d7u2r/cath8sOxbma3JL2FZQduzBJunJMYzITGJERjIjMpM5Mz3J58Mo63jwSinViiNHa7jt5S/5ZPshZp2dyW+uGEq058Tn8xpj2FdcydrcEtbmWYG+YV8JFTXWkAiJ0RGMyExmpN0yH56ZRPcE/z9GUceDV0qpVnSJi+Qf3x7LXxZu5dGPd7L5QCmPzRlNbKSbtXlWV8tau6vlUHkNYHXzDO2VyLVjMq1Az0wmKyW2055P3FbagldKKduCjfnc9eraY0+wAqvfvF9qvNVnnpnMSLvfPDLC8YsQAW3BK6VUm3xtaBr9bo3n2U93k9EllhGZSQxLTyIhSJ/jqwGvlFKNnJEaz/9dNczpMnwiMH7HUEop5XMa8EopFaI04JVSKkRpwCulVIjSgFdKqRClAa+UUiFKA14ppUKUBrxSSoWogBqqQEQKgb1O19EG3YBDThfhR6H8+fSzBa9Q/nwd+Wx9jDGpLS0IqIAPFiKS09rYD6EglD+ffrbgFcqfz1+fTbtolFIqRGnAK6VUiNKAPz1POl2An4Xy59PPFrxC+fP55bNpH7xSSoUobcErpVSI0oBXSqkQpQHfDiKSKSIfi8gmEdkoIrc7XZOviYhbRL4UkbedrsXXRCRZROaLyBYR2Swi452uyVdE5E773+QGEXlJRPz/tGc/EpFnReSgiGxoNK+riCwUke32excnazxdrXy2P9n/LteJyBsikuyLY2nAt08dcJcxZggwDrhFRIY4XJOv3Q5sdroIP3kQeN8YMwgYQYh8ThFJB24DxhhjzgTcwCxnq+qwfwDTm827G/jQGNMf+ND+ORj9gxM/20LgTGPMcGAb8DNfHEgDvh2MMQeMMavt6TKsgEh3tirfEZEM4FLgaadr8TURSQImAc8AGGNqjDHFzlblUxFAjIhEALHAfofr6RBjzFLgcLPZM4Dn7enngSs7tSgfaemzGWM+MMbU2T8uBzJ8cSwN+NMkIlnAKGCFs5X41N+AnwJepwvxg2ygEHjO7oJ6WkTinC7KF4wx+4AHgK+AA0CJMeYDZ6vyix7GmAP2dD7Qw8li/Og7wHu+2JEG/GkQkXjgNeAOY0yp0/X4gohcBhw0xqxyuhY/iQDOAh4zxowCjhK8v+I3YfdFz8D6EusFxInIHGer8i9jXd8dctd4i8g9WF3B83yxPw34dhIRD1a4zzPGvO50PT40AbhCRPYALwMXisiLzpbkU3lAnjGm4Teu+ViBHwqmAruNMYXGmFrgdeBch2vyhwIR6Qlgvx90uB6fEpEbgcuA2cZHNyhpwLeDiAhWH+5mY8xfnK7Hl4wxPzPGZBhjsrBO0H1kjAmZVqAxJh/IFZGB9qwpwCYHS/Klr4BxIhJr/xudQoicQG7mLeAGe/oG4D8O1uJTIjIdq3v0CmNMha/2qwHfPhOA67Fat2vs1yVOF6Xa7IfAPBFZB4wE7nO4Hp+wfyuZD6wG1mP9vw7q2/pF5CVgGTBQRPJE5CbgfmCaiGzH+q3lfidrPF2tfLZHgARgoZ0rj/vkWDpUgVJKhSZtwSulVIjSgFdKqRClAa+UUiFKA14ppUKUBrxSSoUoDXilfEBEzg/FEThVcNOAV0qpEKUBr8KKiMwRkZX2zSRP2OPfl4vIX+3x1D8UkVR73ZEisrzRGN1d7Pn9RGSRiKwVkdUicoa9+/hG483Ps+8qVcoxGvAqbIjIYGAmMMEYMxKoB2YDcUCOMWYosAT4tb3JP4H/tcfoXt9o/jzgUWPMCKwxXxpGOBwF3AEMAfpi3fmslGMinC5AqU40BRgNfGE3rmOwBqzyAq/Y67wIvG6PH59sjFliz38e+LeIJADpxpg3AIwxVQD2/lYaY/Lsn9cAWcCn/v9YSrVMA16FEwGeN8Y0eVqOiPyy2XqnO35HdaPpevT/l3KYdtGocPIh8A0R6Q7HnvHZB+v/wTfsda4DPjXGlABHROQ8e/71wBL7SV55InKlvY8oEYnt1E+hVBtpC0OFDWPMJhH5BfCBiLiAWuAWrId/jLWXHcTqpwdrSNrH7QDfBXzbnn898ISI/NbexzWd+DGUajMdTVKFPREpN8bEO12HUr6mXTRKKRWitAWvlFIhSlvwSikVojTglVIqRGnAK6VUiNKAV0qpEKUBr5RSIer/A9Yk6aWuBWv1AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hX3VzmGeN2Ay"
      },
      "source": [
        "y_predicted = model.predict([X_test, X_char_test])[0]\n",
        "pred_tags = sequences_to_tag_for_crf(y_predicted)\n",
        "test_tags = sequences_to_tag(y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nk1jkfQUN5G6",
        "outputId": "baa6ffd4-c2a8-4f68-f578-b1cae00252fb"
      },
      "source": [
        "print(classification_report(test_tags, pred_tags))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         art       0.44      0.06      0.11        63\n",
            "         eve       0.70      0.27      0.39        52\n",
            "         geo       0.83      0.87      0.85      7620\n",
            "         gpe       0.95      0.94      0.94      3145\n",
            "         nat       0.39      0.19      0.25        37\n",
            "         org       0.71      0.55      0.62      4033\n",
            "         per       0.79      0.74      0.76      3545\n",
            "         tim       0.88      0.84      0.86      4067\n",
            "\n",
            "   micro avg       0.83      0.79      0.81     22562\n",
            "   macro avg       0.71      0.56      0.60     22562\n",
            "weighted avg       0.82      0.79      0.81     22562\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gadtb_rmOrcf"
      },
      "source": [
        "## 성능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aP1jPe10AIqb",
        "outputId": "b799effa-f85f-4d5d-96b2-4dabf8219c19"
      },
      "source": [
        "print(\"F1-score: {:.1%}\".format(f1_score(test_tags, pred_tags)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "F1-score: 81.1%\n"
          ]
        }
      ]
    }
  ]
}