distilbert_tf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "htO7JShhI4sa"
   },
   "source": [
    "## Classifying text with DistilBERT and Tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hsdBQBpDAX5X"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "!python3 -m venv venv\n",
    "!source venv/bin/activate\n",
    "!pip install tensorflow transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wew63ncSAmvw"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras import activations, optimizers, losses\n",
    "from transformers import DistilBertTokenizer, TFDistilBertForSequenceClassification\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fZB5Wl0SArdN"
   },
   "source": [
    "This notebook explores binary text classification using `TFDistilBertForSequenceClassification` from the 🤗Transformers library.\n",
    "\n",
    "This notebook is broken up into 5 sections:\n",
    "1. Preprocessing the data\n",
    "2. Fine-tuning the model\n",
    "3. Testing the model\n",
    "4. Using the fine-tuned model to predict new samples\n",
    "5. Saving and loading the model for future use\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C1uUKILjBqq1"
   },
   "source": [
    "Let's start off by taking a look at our dataset. In this example we consider a small corpus of 10 Yelp reviews: 5 positive (class 1) and 5 negative (class 0). BERT (and its variants like DistilBERT) can be a [great tool to use when you have a shortage of training data](https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html) ... that said, don't expect great results with just 10 reviews! Interchanging `x` and `y` with your own dataset is recommended 🙂"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ItIYoHxEB00B"
   },
   "outputs": [],
   "source": [
    "x = [\n",
    "     'Great customer service! The food was delicious! Definitely a come again.',\n",
    "     'The VEGAN options are super fire!!! And the plates come in big portions. Very pleased with this spot, I\\'ll definitely be ordering again',\n",
    "     'Come on, this place is family owned and operated, they are super friendly, the tacos are bomb.',\n",
    "     'This is such a great restaurant. Multiple times during days that we don\\'t want to cook, we\\'ve done takeout here and it\\'s been amazing. It\\'s fast and delicious.',\n",
    "     'Staff is really nice. Food is way better than average. Good cost benefit.',\n",
    "     'pricing for this, while relatively inexpensive for a Las Vegas attraction, is completely over the top.',\n",
    "     'At such a *fine* institution, I find the lack of knowledge and respect for the art appalling',\n",
    "     'If I could give one star I would...I walked out before my food arrived the customer service was horrible!',\n",
    "     'Wow the slowest drive thru I\\'ve ever been at WOWWWW. Horrible I won\\'t be coming back here ever again',\n",
    "     'Service: 1 out of 5 stars. They will mess up your order, not have it ready after 30 mins calling them before. Worst ran family business Ive ever seen.'\n",
    "]\n",
    "\n",
    "y = [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "v06X47TXFFup"
   },
   "source": [
    "# 1. Preprocessing the data\n",
    "\n",
    "Since we are using the [DisitlBERT](https://huggingface.co/transformers/model_doc/distilbert.html) model, the first preprocessing step is to convert each review string in our dataset into a tuple containing the review's [input ids](https://huggingface.co/transformers/glossary.html#input-ids) and [attention masks](https://huggingface.co/transformers/glossary.html#attention-mask) (click the links to learn what these mean!). Luckily, these can easily be computed using the [DistilBertTokenizer](https://huggingface.co/transformers/model_doc/distilbert.html#distilberttokenizer) from 🤗Transformers. \n",
    "\n",
    "Below is an example usage of this tokenizer on the first review. Here `MAX_LEN` specifies the length of each tokenized review. If a tokenized review is shorter than `MAX_LEN`, then it is padded with zeros. If a tokenized review is greater than `MAX_LEN`, then it is truncated (this is why we have `truncation=True` and `padding=True`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136,
     "referenced_widgets": [
      "7cec612b24d1425a8e47242853d81bea",
      "3965736ba40344858c100f4676aded72",
      "c7461045457c4c0e8d4840494c497f16",
      "9a68b44f08ee41268d7a59db4485e464",
      "f0d5f55e4e7544759a08f4e0368dcf5a",
      "7f10b2fc2c8e44b3a6d338aeb83bb329",
      "4e0176d7096f4ffe9813ff7024bdcf26",
      "fe4d1f2a5e5340789f19a640bb83e997"
     ]
    },
    "colab_type": "code",
    "id": "V3NrIFRoBR0z",
    "outputId": "58a3413e-c74f-4760-e555-7f49a44a64f1"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "06a4b1b22630456aba4169cfcc61cc2c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "review: 'Great customer service! The food was delicious! Definitely a come again.'\n",
      "input ids: [101, 2307, 8013, 2326, 999, 1996, 2833, 2001, 12090, 999, 5791, 1037, 2272, 2153, 1012, 102]\n",
      "attention mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n"
     ]
    }
   ],
   "source": [
    "MODEL_NAME = 'distilbert-base-uncased'\n",
    "MAX_LEN = 20\n",
    "\n",
    "review = x[0]\n",
    "\n",
    "tkzr = DistilBertTokenizer.from_pretrained(MODEL_NAME)\n",
    "\n",
    "inputs = tkzr(review, max_length=MAX_LEN, truncation=True, padding=True)\n",
    "\n",
    "print(f'review: \\'{review}\\'')\n",
    "print(f'input ids: {inputs[\"input_ids\"]}')\n",
    "print(f'attention mask: {inputs[\"attention_mask\"]}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HMgiD2cgATjr"
   },
   "source": [
    "Now we must apply this transformation to each review in our corpus. To do this we define a function `construct_encodings`, which maps the tokenizer to each review and aggregates them in `encodings`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def construct_encodings(x, tkzr, max_len, trucation=True, padding=True):\n",
    "    return tkzr(x, max_length=max_len, truncation=trucation, padding=padding)\n",
    "    \n",
    "encodings = construct_encodings(x, tkzr, max_len=MAX_LEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qbSYjFOlOut5"
   },
   "source": [
    "The first stage of preprocessing is done! The second stage is converting our `encodings` and `y` (which holds the classes of the reviews) into a [Tensorflow Dataset object](https://www.tensorflow.org/api_docs/python/tf/data/Dataset). Below is a function to do this:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def construct_tfdataset(encodings, y=None):\n",
    "    if y:\n",
    "        return tf.data.Dataset.from_tensor_slices((dict(encodings),y))\n",
    "    else:\n",
    "        # this case is used when making predictions on unseen samples after training\n",
    "        return tf.data.Dataset.from_tensor_slices(dict(encodings))\n",
    "    \n",
    "tfdataset = construct_tfdataset(encodings, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xUo15LYpPqI1"
   },
   "source": [
    "The third and final preprocessing step is to create [training](https://developers.google.com/machine-learning/glossary#training-set) and [test sets](https://developers.google.com/machine-learning/glossary#test-set): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fKTJqUF5R-o4"
   },
   "outputs": [],
   "source": [
    "TEST_SPLIT = 0.2\n",
    "BATCH_SIZE = 2\n",
    "\n",
    "train_size = int(len(x) * (1-TEST_SPLIT))\n",
    "\n",
    "tfdataset = tfdataset.shuffle(len(x))\n",
    "tfdataset_train = tfdataset.take(train_size)\n",
    "tfdataset_test = tfdataset.skip(train_size)\n",
    "\n",
    "tfdataset_train = tfdataset_train.batch(BATCH_SIZE)\n",
    "tfdataset_test = tfdataset_test.batch(BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6Nqs83LVFUQE"
   },
   "source": [
    "Our data is finally ready. Now we can do the fun part: model fitting! 🙌 🙌 🙌"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "40Hk-8fDR_Pg"
   },
   "source": [
    "# 2. Fine-tuning the model\n",
    "\n",
    "Fine-tuning the model is as easy as instantiating a model instance, [optimizer](https://developers.google.com/machine-learning/glossary#optimizer), and [loss](https://developers.google.com/machine-learning/glossary#loss), and then compiling/fitting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "be5e6479200b4c0087b052840b37805b",
      "b8b41ef764a64885852fe4ed9bb76456",
      "b7317ba5c30a49a7b67a1e0f81c20570",
      "312bc56f99a9450086cc184fd278d0ab",
      "76cbb19604134dc0930d0c814147b6bc",
      "25f9aad52ba04636b68bd64c5a7ee1b9",
      "5d3de7f796a64eb795aafe698ccfe13d",
      "ccc4320df0d44c75abfdaa9df5a93d1c",
      "b237e28ff60e4f4bb3eacb292b132793",
      "5a5732442f3947b3b04ba8a56c0a579f",
      "89654d465ddf45719d866b9a3af04327",
      "0999bf1c6ec444109ffa5ab6a42d7141",
      "085471c3f164441d98b16c3c943be58a",
      "e43fdfb219af4d52977023ed116f394a",
      "8c04721107ab4bffa1b5e2ddc0351b19",
      "518fa0289a684a66a8e998e4467a6290"
     ]
    },
    "colab_type": "code",
    "id": "rVU_LoASQMcp",
    "outputId": "b6332c75-de09-4166-bd41-0419db13386a"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cae9053ed0ff4204b794649e830f2526",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/442 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8b80156506a46338c5bbf0813c8eea1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/363M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertForSequenceClassification: ['vocab_layer_norm', 'activation_13', 'vocab_transform', 'vocab_projector']\n",
      "- This IS expected if you are initializing TFDistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some layers of TFDistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier', 'dropout_19', 'pre_classifier']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f8222b75160>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f8222b75160>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
      "The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
      "The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
      "The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4/4 [==============================] - 21s 1s/step - loss: 0.7196 - accuracy: 0.2667\n",
      "Epoch 2/2\n",
      "4/4 [==============================] - 3s 795ms/step - loss: 0.6553 - accuracy: 0.4500\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f822867f610>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N_EPOCHS = 2\n",
    "\n",
    "model = TFDistilBertForSequenceClassification.from_pretrained(MODEL_NAME)\n",
    "optimizer = optimizers.Adam(learning_rate=3e-5)\n",
    "loss = losses.SparseCategoricalCrossentropy(from_logits=True)\n",
    "model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])\n",
    "\n",
    "model.fit(tfdataset_train, batch_size=BATCH_SIZE, epochs=N_EPOCHS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Sgz5Jfi9cmDv"
   },
   "source": [
    "## 3. Testing the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KBYlX43STjhl"
   },
   "source": [
    "Now we can use our test set to evaluate the performance of the model. Don't expect great results from such a small training set! 😅"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QhmogfJhco0T"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
      "The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 2s 2s/step - loss: 0.5999 - accuracy: 1.0000\n",
      "{'loss': 0.5999139547348022, 'accuracy': 1.0}\n"
     ]
    }
   ],
   "source": [
    "benchmarks = model.evaluate(tfdataset_test, return_dict=True, batch_size=BATCH_SIZE)\n",
    "print(benchmarks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f6u7KjM9G6AP"
   },
   "source": [
    "# 4. Using the fine-tuned model to predict new samples\n",
    "\n",
    "Now that we've trained the model, we can use it to predict new samples. Below is a function whose output is a classifier we can use for predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TCmtvJQUK1DO"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
      "The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.41241995\n"
     ]
    }
   ],
   "source": [
    "def create_predictor(model, model_name, max_len):\n",
    "  tkzr = DistilBertTokenizer.from_pretrained(model_name)\n",
    "  def predict_proba(text):\n",
    "      x = [text]\n",
    "\n",
    "      encodings = construct_encodings(x, tkzr, max_len=max_len)\n",
    "      tfdataset = construct_tfdataset(encodings)\n",
    "      tfdataset = tfdataset.batch(1)\n",
    "\n",
    "      preds = model.predict(tfdataset).logits\n",
    "      preds = activations.softmax(tf.convert_to_tensor(preds)).numpy()\n",
    "      return preds[0][0]\n",
    "    \n",
    "  return predict_proba\n",
    "\n",
    "clf = create_predictor(model, MODEL_NAME, MAX_LEN)\n",
    "print(clf('this restaurant has horrible food'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tyrzzC0VK2VA"
   },
   "source": [
    "## 5. Saving and loading the model for future use\n",
    "\n",
    "We use the built-in [save](https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel.save_pretrained) and [load](https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel.from_pretrained) methods from 🤗Transformers to serialize the model and restore it for future use:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "O4klsMxbLGFa"
   },
   "outputs": [],
   "source": [
    "model.save_pretrained('./model/clf')\n",
    "with open('./model/info.pkl', 'wb') as f:\n",
    "    pickle.dump((MODEL_NAME, MAX_LEN), f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0K0wuj7BPd0n"
   },
   "source": [
    "We save `MODEL_NAME` and `MAX_LEN` for tokenizing/formatting down the line. Below we see how to reload the model for future use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ivGMlcfZOiBv"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some layers from the model checkpoint at ./model/clf were not used when initializing TFDistilBertForSequenceClassification: ['dropout_19']\n",
      "- This IS expected if you are initializing TFDistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some layers of TFDistilBertForSequenceClassification were not initialized from the model checkpoint at ./model/clf and are newly initialized: ['dropout_39']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
      "The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.41241995\n"
     ]
    }
   ],
   "source": [
    "new_model = TFDistilBertForSequenceClassification.from_pretrained('./model/clf')\n",
    "model_name, max_len = pickle.load(open('./model/info.pkl', 'rb'))\n",
    "\n",
    "clf = create_predictor(new_model, model_name, max_len)\n",
    "print(clf('this restaurant has horrible food'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zrOXpMpRQHe2"
   },
   "source": [
    "Unsuprisingly, we get the same prediction as before!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6Ywdze_RQKkD"
   },
   "source": [
    "## Congrats -- you just made a binary text classifier with 🤗Transformers! 🤜💥🤛"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "distilbert_tf",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "085471c3f164441d98b16c3c943be58a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "0999bf1c6ec444109ffa5ab6a42d7141": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_518fa0289a684a66a8e998e4467a6290",
      "placeholder": "​",
      "style": "IPY_MODEL_8c04721107ab4bffa1b5e2ddc0351b19",
      "value": " 363M/363M [00:05&lt;00:00, 62.9MB/s]"
     }
    },
    "25f9aad52ba04636b68bd64c5a7ee1b9": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "312bc56f99a9450086cc184fd278d0ab": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ccc4320df0d44c75abfdaa9df5a93d1c",
      "placeholder": "​",
      "style": "IPY_MODEL_5d3de7f796a64eb795aafe698ccfe13d",
      "value": " 442/442 [00:06&lt;00:00, 69.7B/s]"
     }
    },
    "3965736ba40344858c100f4676aded72": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4e0176d7096f4ffe9813ff7024bdcf26": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "518fa0289a684a66a8e998e4467a6290": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5a5732442f3947b3b04ba8a56c0a579f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5d3de7f796a64eb795aafe698ccfe13d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "76cbb19604134dc0930d0c814147b6bc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "7cec612b24d1425a8e47242853d81bea": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c7461045457c4c0e8d4840494c497f16",
       "IPY_MODEL_9a68b44f08ee41268d7a59db4485e464"
      ],
      "layout": "IPY_MODEL_3965736ba40344858c100f4676aded72"
     }
    },
    "7f10b2fc2c8e44b3a6d338aeb83bb329": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "89654d465ddf45719d866b9a3af04327": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e43fdfb219af4d52977023ed116f394a",
      "max": 363423424,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_085471c3f164441d98b16c3c943be58a",
      "value": 363423424
     }
    },
    "8c04721107ab4bffa1b5e2ddc0351b19": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9a68b44f08ee41268d7a59db4485e464": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fe4d1f2a5e5340789f19a640bb83e997",
      "placeholder": "​",
      "style": "IPY_MODEL_4e0176d7096f4ffe9813ff7024bdcf26",
      "value": " 232k/232k [00:00&lt;00:00, 298kB/s]"
     }
    },
    "b237e28ff60e4f4bb3eacb292b132793": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_89654d465ddf45719d866b9a3af04327",
       "IPY_MODEL_0999bf1c6ec444109ffa5ab6a42d7141"
      ],
      "layout": "IPY_MODEL_5a5732442f3947b3b04ba8a56c0a579f"
     }
    },
    "b7317ba5c30a49a7b67a1e0f81c20570": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_25f9aad52ba04636b68bd64c5a7ee1b9",
      "max": 442,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_76cbb19604134dc0930d0c814147b6bc",
      "value": 442
     }
    },
    "b8b41ef764a64885852fe4ed9bb76456": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "be5e6479200b4c0087b052840b37805b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b7317ba5c30a49a7b67a1e0f81c20570",
       "IPY_MODEL_312bc56f99a9450086cc184fd278d0ab"
      ],
      "layout": "IPY_MODEL_b8b41ef764a64885852fe4ed9bb76456"
     }
    },
    "c7461045457c4c0e8d4840494c497f16": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7f10b2fc2c8e44b3a6d338aeb83bb329",
      "max": 231508,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f0d5f55e4e7544759a08f4e0368dcf5a",
      "value": 231508
     }
    },
    "ccc4320df0d44c75abfdaa9df5a93d1c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e43fdfb219af4d52977023ed116f394a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f0d5f55e4e7544759a08f4e0368dcf5a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "fe4d1f2a5e5340789f19a640bb83e997": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}