2_Training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Computer Vision Nanodegree\n",
    "\n",
    "## Project: Image Captioning\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will train your CNN-RNN model.  \n",
    "\n",
    "You are welcome and encouraged to try out many different architectures and hyperparameters when searching for a good model.\n",
    "\n",
    "This does have the potential to make the project quite messy!  Before submitting your project, make sure that you clean up:\n",
    "- the code you write in this notebook.  The notebook should describe how to train a single CNN-RNN architecture, corresponding to your final choice of hyperparameters.  You should structure the notebook so that the reviewer can replicate your results by running the code in this notebook.  \n",
    "- the output of the code cell in **Step 2**.  The output should show the output obtained when training the model from scratch.\n",
    "\n",
    "This notebook **will be graded**.  \n",
    "\n",
    "Feel free to use the links below to navigate the notebook:\n",
    "- [Step 1](#step1): Training Setup\n",
    "- [Step 2](#step2): Train your Model\n",
    "- [Step 3](#step3): (Optional) Validate your Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='step1'></a>\n",
    "## Step 1: Training Setup\n",
    "\n",
    "In this step of the notebook, you will customize the training of your CNN-RNN model by specifying hyperparameters and setting other options that are important to the training procedure.  The values you set now will be used when training your model in **Step 2** below.\n",
    "\n",
    "You should only amend blocks of code that are preceded by a `TODO` statement.  **Any code blocks that are not preceded by a `TODO` statement should not be modified**.\n",
    "\n",
    "### Task #1\n",
    "\n",
    "Begin by setting the following variables:\n",
    "- `batch_size` - the batch size of each training batch.  It is the number of image-caption pairs used to amend the model weights in each training step. \n",
    "- `vocab_threshold` - the minimum word count threshold.  Note that a larger threshold will result in a smaller vocabulary, whereas a smaller threshold will include rarer words and result in a larger vocabulary.  \n",
    "- `vocab_from_file` - a Boolean that decides whether to load the vocabulary from file. \n",
    "- `embed_size` - the dimensionality of the image and word embeddings.  \n",
    "- `hidden_size` - the number of features in the hidden state of the RNN decoder.  \n",
    "- `num_epochs` - the number of epochs to train the model.  We recommend that you set `num_epochs=3`, but feel free to increase or decrease this number as you wish.  [This paper](https://arxiv.org/pdf/1502.03044.pdf) trained a captioning model on a single state-of-the-art GPU for 3 days, but you'll soon see that you can get reasonable results in a matter of a few hours!  (_But of course, if you want your model to compete with current research, you will have to train for much longer._)\n",
    "- `save_every` - determines how often to save the model weights.  We recommend that you set `save_every=1`, to save the model weights after each epoch.  This way, after the `i`th epoch, the encoder and decoder weights will be saved in the `models/` folder as `encoder-i.pkl` and `decoder-i.pkl`, respectively.\n",
    "- `print_every` - determines how often to print the batch loss to the Jupyter notebook while training.  Note that you **will not** observe a monotonic decrease in the loss function while training - this is perfectly fine and completely expected!  You are encouraged to keep this at its default value of `100` to avoid clogging the notebook, but feel free to change it.\n",
    "- `log_file` - the name of the text file containing - for every step - how the loss and perplexity evolved during training.\n",
    "\n",
    "If you're not sure where to begin to set some of the values above, you can peruse [this paper](https://arxiv.org/pdf/1502.03044.pdf) and [this paper](https://arxiv.org/pdf/1411.4555.pdf) for useful guidance!  **To avoid spending too long on this notebook**, you are encouraged to consult these suggested research papers to obtain a strong initial guess for which hyperparameters are likely to work best.  Then, train a single model, and proceed to the next notebook (**3_Inference.ipynb**).  If you are unhappy with your performance, you can return to this notebook to tweak the hyperparameters (and/or the architecture in **model.py**) and re-train your model.\n",
    "\n",
    "### Question 1\n",
    "\n",
    "**Question:** Describe your CNN-RNN architecture in detail.  With this architecture in mind, how did you select the values of the variables in Task 1?  If you consulted a research paper detailing a successful implementation of an image captioning model, please provide the reference.\n",
    "\n",
    "**Answer:** \n",
    "> 1. My Encoder CNN network is a ResNet-50 architecture. It is a pretrained network readily available in pytorch. Resnet model is one of the state of the art models available. The best thing about Resnet architecture is it removes vanishing/exploding gradienst problem. But as it was trained for a classification problem, we should remove the last layer with a new linear layer to get high level features. \n",
    "> 2. My Decoder RNN network consists of an Embedding layer, LSTM layer and a Linear layer respectively.\n",
    "> 3. For this architecture, I have started with embed_size = 512, hiddedn_size=512, which are taken from the [paper](https://arxiv.org/pdf/1502.03044.pdf). For the batch size, I started with 16 and changed to 32, for which results are better.\n",
    "\n",
    "\n",
    "### (Optional) Task #2\n",
    "\n",
    "Note that we have provided a recommended image transform `transform_train` for pre-processing the training images, but you are welcome (and encouraged!) to modify it as you wish.  When modifying this transform, keep in mind that:\n",
    "- the images in the dataset have varying heights and widths, and \n",
    "- if using a pre-trained model, you must perform the corresponding appropriate normalization.\n",
    "\n",
    "### Question 2\n",
    "\n",
    "**Question:** How did you select the transform in `transform_train`?  If you left the transform at its provided value, why do you think that it is a good choice for your CNN architecture?\n",
    "\n",
    "**Answer:**\n",
    "I have left the 'transform_train' unchanged and i feel that it is a good choice for my architecture. The resnet-50 model requires an input shape of 224x224x3 and this transform provides it. The transforms also provides augmentation by doing random cropping and horizantal flipping which helps the model to generalize well on new data.\n",
    "\n",
    "### Task #3\n",
    "\n",
    "Next, you will specify a Python list containing the learnable parameters of the model.  For instance, if you decide to make all weights in the decoder trainable, but only want to train the weights in the embedding layer of the encoder, then you should set `params` to something like:\n",
    "```\n",
    "params = list(decoder.parameters()) + list(encoder.embed.parameters()) \n",
    "```\n",
    "\n",
    "### Question 3\n",
    "\n",
    "**Question:** How did you select the trainable parameters of your architecture?  Why do you think this is a good choice?\n",
    "\n",
    "**Answer:**\n",
    "As our encoder network, Resnet-50 model is already pretained on imagenet dataset, we don't need to retrain it. But as it was trained for a classification problem, we should remove the last layer with a new linear layer. So, this layer's parameters have to be trained to get the high level features. For decoder network, as we are writing our own architecture, we need to train the network so that it learns from the high level features of images and the captions. So the parameters of all layers are chosen as trainable parameters.\n",
    "\n",
    "### Task #4\n",
    "\n",
    "Finally, you will select an [optimizer](http://pytorch.org/docs/master/optim.html#torch.optim.Optimizer).\n",
    "\n",
    "### Question 4\n",
    "\n",
    "**Question:** How did you select the optimizer used to train your model?\n",
    "\n",
    "**Answer:** \n",
    "I chose Adam optimizer beacuase of it's adaptive nature. The same optimizer was used in the [paper](https://arxiv.org/pdf/1502.03044.pdf) from which I referred."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary successfully loaded from vocab.pkl file!\n",
      "loading annotations into memory...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/414113 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done (t=0.82s)\n",
      "creating index...\n",
      "index created!\n",
      "Obtaining caption lengths...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 414113/414113 [01:03<00:00, 6559.13it/s]\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torchvision import transforms\n",
    "import sys\n",
    "sys.path.append('/opt/cocoapi/PythonAPI')\n",
    "from pycocotools.coco import COCO\n",
    "from data_loader import get_loader\n",
    "from model import EncoderCNN, DecoderRNN\n",
    "import math\n",
    "\n",
    "\n",
    "## TODO #1: Select appropriate values for the Python variables below.\n",
    "batch_size = 32        # batch size\n",
    "vocab_threshold = 5        # minimum word count threshold\n",
    "vocab_from_file = True    # if True, load existing vocab file\n",
    "embed_size = 512           # dimensionality of image and word embeddings\n",
    "hidden_size = 512          # number of features in hidden state of the RNN decoder\n",
    "num_epochs = 5             # number of training epochs\n",
    "save_every = 1             # determines frequency of saving model weights\n",
    "print_every = 100          # determines window for printing average loss\n",
    "log_file = 'training_log.txt'       # name of file with saved training loss and perplexity\n",
    "\n",
    "# (Optional) TODO #2: Amend the image transform below.\n",
    "transform_train = transforms.Compose([ \n",
    "    transforms.Resize(256),                          # smaller edge of image resized to 256\n",
    "    transforms.RandomCrop(224),                      # get 224x224 crop from random location\n",
    "    transforms.RandomHorizontalFlip(),               # horizontally flip image with probability=0.5\n",
    "    transforms.ToTensor(),                           # convert the PIL Image to a tensor\n",
    "    transforms.Normalize((0.485, 0.456, 0.406),      # normalize image for pre-trained model\n",
    "                         (0.229, 0.224, 0.225))])\n",
    "\n",
    "# Build data loader.\n",
    "data_loader = get_loader(transform=transform_train,\n",
    "                         mode='train',\n",
    "                         batch_size=batch_size,\n",
    "                         vocab_threshold=vocab_threshold,\n",
    "                         vocab_from_file=vocab_from_file)\n",
    "\n",
    "# The size of the vocabulary.\n",
    "vocab_size = len(data_loader.dataset.vocab)\n",
    "\n",
    "# Initialize the encoder and decoder. \n",
    "encoder = EncoderCNN(embed_size)\n",
    "decoder = DecoderRNN(embed_size, hidden_size, vocab_size)\n",
    "\n",
    "# Move models to GPU if CUDA is available. \n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "encoder.to(device)\n",
    "decoder.to(device)\n",
    "\n",
    "# Define the loss function. \n",
    "criterion = nn.CrossEntropyLoss().cuda() if torch.cuda.is_available() else nn.CrossEntropyLoss()\n",
    "\n",
    "# TODO #3: Specify the learnable parameters of the model.\n",
    "params = list(encoder.embed.parameters()) + list(decoder.parameters())\n",
    "\n",
    "# TODO #4: Define the optimizer.\n",
    "optimizer = torch.optim.Adam(params, lr=0.001)\n",
    "\n",
    "\n",
    "# Set the total number of training steps per epoch.\n",
    "total_step = math.ceil(len(data_loader.dataset.caption_lengths) / data_loader.batch_sampler.batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='step2'></a>\n",
    "## Step 2: Train your Model\n",
    "\n",
    "Once you have executed the code cell in **Step 1**, the training procedure below should run without issue.  \n",
    "\n",
    "It is completely fine to leave the code cell below as-is without modifications to train your model.  However, if you would like to modify the code used to train the model below, you must ensure that your changes are easily parsed by your reviewer.  In other words, make sure to provide appropriate comments to describe how your code works!  \n",
    "\n",
    "You may find it useful to load saved weights to resume training.  In that case, note the names of the files containing the encoder and decoder weights that you'd like to load (`encoder_file` and `decoder_file`).  Then you can load the weights by using the lines below:\n",
    "\n",
    "```python\n",
    "# Load pre-trained weights before resuming training.\n",
    "encoder.load_state_dict(torch.load(os.path.join('./models', encoder_file)))\n",
    "decoder.load_state_dict(torch.load(os.path.join('./models', decoder_file)))\n",
    "```\n",
    "\n",
    "While trying out parameters, make sure to take extensive notes and record the settings that you used in your various training runs.  In particular, you don't want to encounter a situation where you've trained a model for several hours but can't remember what settings you used :).\n",
    "\n",
    "### A Note on Tuning Hyperparameters\n",
    "\n",
    "To figure out how well your model is doing, you can look at how the training loss and perplexity evolve during training - and for the purposes of this project, you are encouraged to amend the hyperparameters based on this information.  \n",
    "\n",
    "However, this will not tell you if your model is overfitting to the training data, and, unfortunately, overfitting is a problem that is commonly encountered when training image captioning models.  \n",
    "\n",
    "For this project, you need not worry about overfitting. **This project does not have strict requirements regarding the performance of your model**, and you just need to demonstrate that your model has learned **_something_** when you generate captions on the test data.  For now, we strongly encourage you to train your model for the suggested 3 epochs without worrying about performance; then, you should immediately transition to the next notebook in the sequence (**3_Inference.ipynb**) to see how your model performs on the test data.  If your model needs to be changed, you can come back to this notebook, amend hyperparameters (if necessary), and re-train the model.\n",
    "\n",
    "That said, if you would like to go above and beyond in this project, you can read about some approaches to minimizing overfitting in section 4.3.1 of [this paper](http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7505636).  In the next (optional) step of this notebook, we provide some guidance for assessing the performance on the validation dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/5], Step [100/12942], Loss: 3.6668, Perplexity: 39.1276\n",
      "Epoch [1/5], Step [200/12942], Loss: 3.1979, Perplexity: 24.4799\n",
      "Epoch [1/5], Step [300/12942], Loss: 3.3180, Perplexity: 27.6055\n",
      "Epoch [1/5], Step [400/12942], Loss: 3.7326, Perplexity: 41.7871\n",
      "Epoch [1/5], Step [500/12942], Loss: 3.4245, Perplexity: 30.7084\n",
      "Epoch [1/5], Step [600/12942], Loss: 2.9967, Perplexity: 20.0203\n",
      "Epoch [1/5], Step [700/12942], Loss: 3.4199, Perplexity: 30.5673\n",
      "Epoch [1/5], Step [800/12942], Loss: 2.9443, Perplexity: 18.9980\n",
      "Epoch [1/5], Step [900/12942], Loss: 2.6137, Perplexity: 13.6499\n",
      "Epoch [1/5], Step [1000/12942], Loss: 3.1270, Perplexity: 22.8050\n",
      "Epoch [1/5], Step [1100/12942], Loss: 2.8251, Perplexity: 16.8618\n",
      "Epoch [1/5], Step [1200/12942], Loss: 2.9811, Perplexity: 19.7103\n",
      "Epoch [1/5], Step [1300/12942], Loss: 2.7159, Perplexity: 15.1185\n",
      "Epoch [1/5], Step [1400/12942], Loss: 2.8238, Perplexity: 16.8406\n",
      "Epoch [1/5], Step [1500/12942], Loss: 2.7257, Perplexity: 15.2678\n",
      "Epoch [1/5], Step [1600/12942], Loss: 3.2525, Perplexity: 25.8548\n",
      "Epoch [1/5], Step [1700/12942], Loss: 2.7059, Perplexity: 14.9672\n",
      "Epoch [1/5], Step [1800/12942], Loss: 2.4597, Perplexity: 11.7010\n",
      "Epoch [1/5], Step [1900/12942], Loss: 2.8845, Perplexity: 17.8941\n",
      "Epoch [1/5], Step [2000/12942], Loss: 2.6290, Perplexity: 13.8602\n",
      "Epoch [1/5], Step [2100/12942], Loss: 2.8975, Perplexity: 18.1284\n",
      "Epoch [1/5], Step [2200/12942], Loss: 2.5700, Perplexity: 13.0659\n",
      "Epoch [1/5], Step [2300/12942], Loss: 2.6740, Perplexity: 14.4980\n",
      "Epoch [1/5], Step [2400/12942], Loss: 3.1556, Perplexity: 23.4676\n",
      "Epoch [1/5], Step [2500/12942], Loss: 2.8635, Perplexity: 17.5219\n",
      "Epoch [1/5], Step [2600/12942], Loss: 2.5154, Perplexity: 12.3721\n",
      "Epoch [1/5], Step [2700/12942], Loss: 2.3537, Perplexity: 10.5243\n",
      "Epoch [1/5], Step [2800/12942], Loss: 2.6003, Perplexity: 13.4679\n",
      "Epoch [1/5], Step [2900/12942], Loss: 2.9041, Perplexity: 18.2484\n",
      "Epoch [1/5], Step [3000/12942], Loss: 2.2102, Perplexity: 9.1174\n",
      "Epoch [1/5], Step [3100/12942], Loss: 2.9499, Perplexity: 19.1033\n",
      "Epoch [1/5], Step [3200/12942], Loss: 2.4385, Perplexity: 11.4560\n",
      "Epoch [1/5], Step [3300/12942], Loss: 2.9415, Perplexity: 18.9451\n",
      "Epoch [1/5], Step [3400/12942], Loss: 2.5266, Perplexity: 12.5110\n",
      "Epoch [1/5], Step [3500/12942], Loss: 2.3540, Perplexity: 10.5277\n",
      "Epoch [1/5], Step [3600/12942], Loss: 2.5096, Perplexity: 12.3001\n",
      "Epoch [1/5], Step [3700/12942], Loss: 2.7002, Perplexity: 14.8829\n",
      "Epoch [1/5], Step [3800/12942], Loss: 2.2796, Perplexity: 9.7724\n",
      "Epoch [1/5], Step [3900/12942], Loss: 2.2986, Perplexity: 9.9602\n",
      "Epoch [1/5], Step [4000/12942], Loss: 2.3341, Perplexity: 10.3202\n",
      "Epoch [1/5], Step [4100/12942], Loss: 2.8432, Perplexity: 17.1712\n",
      "Epoch [1/5], Step [4200/12942], Loss: 2.4714, Perplexity: 11.8391\n",
      "Epoch [1/5], Step [4300/12942], Loss: 2.2691, Perplexity: 9.6709\n",
      "Epoch [1/5], Step [4400/12942], Loss: 2.4033, Perplexity: 11.0593\n",
      "Epoch [1/5], Step [4500/12942], Loss: 2.3130, Perplexity: 10.1051\n",
      "Epoch [1/5], Step [4600/12942], Loss: 2.4965, Perplexity: 12.1395\n",
      "Epoch [1/5], Step [4700/12942], Loss: 2.5311, Perplexity: 12.5675\n",
      "Epoch [1/5], Step [4800/12942], Loss: 3.0247, Perplexity: 20.5876\n",
      "Epoch [1/5], Step [4900/12942], Loss: 2.4366, Perplexity: 11.4343\n",
      "Epoch [1/5], Step [5000/12942], Loss: 2.2898, Perplexity: 9.8734\n",
      "Epoch [1/5], Step [5100/12942], Loss: 2.3356, Perplexity: 10.3357\n",
      "Epoch [1/5], Step [5200/12942], Loss: 2.1929, Perplexity: 8.9616\n",
      "Epoch [1/5], Step [5300/12942], Loss: 2.6861, Perplexity: 14.6742\n",
      "Epoch [1/5], Step [5400/12942], Loss: 2.2573, Perplexity: 9.5574\n",
      "Epoch [1/5], Step [5500/12942], Loss: 2.8516, Perplexity: 17.3156\n",
      "Epoch [1/5], Step [5600/12942], Loss: 3.0272, Perplexity: 20.6386\n",
      "Epoch [1/5], Step [5700/12942], Loss: 2.4913, Perplexity: 12.0764\n",
      "Epoch [1/5], Step [5800/12942], Loss: 2.5715, Perplexity: 13.0854\n",
      "Epoch [1/5], Step [5900/12942], Loss: 2.1846, Perplexity: 8.8871\n",
      "Epoch [1/5], Step [6000/12942], Loss: 2.1585, Perplexity: 8.6584\n",
      "Epoch [1/5], Step [6100/12942], Loss: 2.2209, Perplexity: 9.2158\n",
      "Epoch [1/5], Step [6200/12942], Loss: 2.2981, Perplexity: 9.9556\n",
      "Epoch [1/5], Step [6300/12942], Loss: 2.1211, Perplexity: 8.3407\n",
      "Epoch [1/5], Step [6400/12942], Loss: 2.5383, Perplexity: 12.6585\n",
      "Epoch [1/5], Step [6500/12942], Loss: 2.3350, Perplexity: 10.3298\n",
      "Epoch [1/5], Step [6600/12942], Loss: 2.3659, Perplexity: 10.6541\n",
      "Epoch [1/5], Step [6700/12942], Loss: 2.3760, Perplexity: 10.7618\n",
      "Epoch [1/5], Step [6800/12942], Loss: 2.5285, Perplexity: 12.5346\n",
      "Epoch [1/5], Step [6900/12942], Loss: 2.2945, Perplexity: 9.9199\n",
      "Epoch [1/5], Step [7000/12942], Loss: 2.2051, Perplexity: 9.0710\n",
      "Epoch [1/5], Step [7100/12942], Loss: 2.4120, Perplexity: 11.1568\n",
      "Epoch [1/5], Step [7200/12942], Loss: 2.1565, Perplexity: 8.6406\n",
      "Epoch [1/5], Step [7300/12942], Loss: 2.2937, Perplexity: 9.9120\n",
      "Epoch [1/5], Step [7400/12942], Loss: 2.2313, Perplexity: 9.3124\n",
      "Epoch [1/5], Step [7500/12942], Loss: 2.7430, Perplexity: 15.5337\n",
      "Epoch [1/5], Step [7600/12942], Loss: 2.3907, Perplexity: 10.9215\n",
      "Epoch [1/5], Step [7700/12942], Loss: 2.3567, Perplexity: 10.5562\n",
      "Epoch [1/5], Step [7800/12942], Loss: 2.1674, Perplexity: 8.7358\n",
      "Epoch [1/5], Step [7900/12942], Loss: 2.1932, Perplexity: 8.9641\n",
      "Epoch [1/5], Step [8000/12942], Loss: 2.5720, Perplexity: 13.0917\n",
      "Epoch [1/5], Step [8100/12942], Loss: 2.4761, Perplexity: 11.8952\n",
      "Epoch [1/5], Step [8200/12942], Loss: 2.2624, Perplexity: 9.6065\n",
      "Epoch [1/5], Step [8300/12942], Loss: 2.6718, Perplexity: 14.4665\n",
      "Epoch [1/5], Step [8400/12942], Loss: 2.3748, Perplexity: 10.7484\n",
      "Epoch [1/5], Step [8500/12942], Loss: 2.4189, Perplexity: 11.2332\n",
      "Epoch [1/5], Step [8600/12942], Loss: 2.1205, Perplexity: 8.3351\n",
      "Epoch [1/5], Step [8700/12942], Loss: 2.1188, Perplexity: 8.3211\n",
      "Epoch [1/5], Step [8800/12942], Loss: 2.3084, Perplexity: 10.0587\n",
      "Epoch [1/5], Step [8900/12942], Loss: 2.2546, Perplexity: 9.5318\n",
      "Epoch [1/5], Step [9000/12942], Loss: 2.5744, Perplexity: 13.1240\n",
      "Epoch [1/5], Step [9100/12942], Loss: 2.2525, Perplexity: 9.5113\n",
      "Epoch [1/5], Step [9200/12942], Loss: 2.2204, Perplexity: 9.2112\n",
      "Epoch [1/5], Step [9300/12942], Loss: 2.5116, Perplexity: 12.3243\n",
      "Epoch [1/5], Step [9400/12942], Loss: 2.2418, Perplexity: 9.4100\n",
      "Epoch [1/5], Step [9500/12942], Loss: 2.0962, Perplexity: 8.1354\n",
      "Epoch [1/5], Step [9600/12942], Loss: 2.1964, Perplexity: 8.9924\n",
      "Epoch [1/5], Step [9700/12942], Loss: 2.3494, Perplexity: 10.4791\n",
      "Epoch [1/5], Step [9800/12942], Loss: 2.3191, Perplexity: 10.1668\n",
      "Epoch [1/5], Step [9900/12942], Loss: 2.3970, Perplexity: 10.9905\n",
      "Epoch [1/5], Step [10000/12942], Loss: 2.1164, Perplexity: 8.3011\n",
      "Epoch [1/5], Step [10100/12942], Loss: 2.3589, Perplexity: 10.5789\n",
      "Epoch [1/5], Step [10200/12942], Loss: 2.1061, Perplexity: 8.2160\n",
      "Epoch [1/5], Step [10300/12942], Loss: 2.1439, Perplexity: 8.5327\n",
      "Epoch [1/5], Step [10400/12942], Loss: 2.0978, Perplexity: 8.1485\n",
      "Epoch [1/5], Step [10500/12942], Loss: 2.8593, Perplexity: 17.4486\n",
      "Epoch [1/5], Step [10600/12942], Loss: 2.2486, Perplexity: 9.4748\n",
      "Epoch [1/5], Step [10700/12942], Loss: 2.1184, Perplexity: 8.3177\n",
      "Epoch [1/5], Step [10800/12942], Loss: 2.4337, Perplexity: 11.4011\n",
      "Epoch [1/5], Step [10900/12942], Loss: 2.3104, Perplexity: 10.0782\n",
      "Epoch [1/5], Step [11000/12942], Loss: 2.5084, Perplexity: 12.2857\n",
      "Epoch [1/5], Step [11100/12942], Loss: 1.9710, Perplexity: 7.1775\n",
      "Epoch [1/5], Step [11200/12942], Loss: 2.2680, Perplexity: 9.6601\n",
      "Epoch [1/5], Step [11300/12942], Loss: 2.3532, Perplexity: 10.5188\n",
      "Epoch [1/5], Step [11400/12942], Loss: 2.3571, Perplexity: 10.5603\n",
      "Epoch [1/5], Step [11500/12942], Loss: 2.3673, Perplexity: 10.6690\n",
      "Epoch [1/5], Step [11600/12942], Loss: 2.1472, Perplexity: 8.5608\n",
      "Epoch [1/5], Step [11700/12942], Loss: 2.0168, Perplexity: 7.5143\n",
      "Epoch [1/5], Step [11800/12942], Loss: 2.2133, Perplexity: 9.1458\n",
      "Epoch [1/5], Step [11900/12942], Loss: 2.0540, Perplexity: 7.7991\n",
      "Epoch [1/5], Step [12000/12942], Loss: 2.8702, Perplexity: 17.6410\n",
      "Epoch [1/5], Step [12100/12942], Loss: 2.0376, Perplexity: 7.6725\n",
      "Epoch [1/5], Step [12200/12942], Loss: 2.0705, Perplexity: 7.9291\n",
      "Epoch [1/5], Step [12300/12942], Loss: 2.1645, Perplexity: 8.7106\n",
      "Epoch [1/5], Step [12400/12942], Loss: 3.3535, Perplexity: 28.6041\n",
      "Epoch [1/5], Step [12500/12942], Loss: 1.9852, Perplexity: 7.2805\n",
      "Epoch [1/5], Step [12600/12942], Loss: 2.3111, Perplexity: 10.0856\n",
      "Epoch [1/5], Step [12700/12942], Loss: 2.2418, Perplexity: 9.4107\n",
      "Epoch [1/5], Step [12800/12942], Loss: 2.2036, Perplexity: 9.0577\n",
      "Epoch [1/5], Step [12900/12942], Loss: 2.0399, Perplexity: 7.6902\n",
      "Epoch [2/5], Step [100/12942], Loss: 2.3052, Perplexity: 10.0261\n",
      "Epoch [2/5], Step [200/12942], Loss: 2.1641, Perplexity: 8.7072\n",
      "Epoch [2/5], Step [300/12942], Loss: 2.1094, Perplexity: 8.2430\n",
      "Epoch [2/5], Step [400/12942], Loss: 2.2873, Perplexity: 9.8481\n",
      "Epoch [2/5], Step [500/12942], Loss: 1.9244, Perplexity: 6.8511\n",
      "Epoch [2/5], Step [600/12942], Loss: 2.1902, Perplexity: 8.9369\n",
      "Epoch [2/5], Step [700/12942], Loss: 2.1697, Perplexity: 8.7557\n",
      "Epoch [2/5], Step [800/12942], Loss: 1.9514, Perplexity: 7.0387\n",
      "Epoch [2/5], Step [900/12942], Loss: 2.0513, Perplexity: 7.7783\n",
      "Epoch [2/5], Step [1000/12942], Loss: 2.1662, Perplexity: 8.7249\n",
      "Epoch [2/5], Step [1100/12942], Loss: 2.5628, Perplexity: 12.9721\n",
      "Epoch [2/5], Step [1200/12942], Loss: 2.0717, Perplexity: 7.9381\n",
      "Epoch [2/5], Step [1300/12942], Loss: 2.1588, Perplexity: 8.6605\n",
      "Epoch [2/5], Step [1400/12942], Loss: 2.0457, Perplexity: 7.7349\n",
      "Epoch [2/5], Step [1500/12942], Loss: 2.0769, Perplexity: 7.9793\n",
      "Epoch [2/5], Step [1600/12942], Loss: 2.3326, Perplexity: 10.3044\n",
      "Epoch [2/5], Step [1700/12942], Loss: 2.1722, Perplexity: 8.7779\n",
      "Epoch [2/5], Step [1800/12942], Loss: 2.1653, Perplexity: 8.7175\n",
      "Epoch [2/5], Step [1900/12942], Loss: 2.7510, Perplexity: 15.6578\n",
      "Epoch [2/5], Step [2000/12942], Loss: 2.2797, Perplexity: 9.7740\n",
      "Epoch [2/5], Step [2100/12942], Loss: 2.1235, Perplexity: 8.3600\n",
      "Epoch [2/5], Step [2200/12942], Loss: 2.1973, Perplexity: 9.0009\n",
      "Epoch [2/5], Step [2300/12942], Loss: 1.9913, Perplexity: 7.3253\n",
      "Epoch [2/5], Step [2400/12942], Loss: 2.5509, Perplexity: 12.8192\n",
      "Epoch [2/5], Step [2500/12942], Loss: 1.8446, Perplexity: 6.3258\n",
      "Epoch [2/5], Step [2600/12942], Loss: 2.1824, Perplexity: 8.8676\n",
      "Epoch [2/5], Step [2700/12942], Loss: 2.5971, Perplexity: 13.4246\n",
      "Epoch [2/5], Step [2800/12942], Loss: 2.1219, Perplexity: 8.3471\n",
      "Epoch [2/5], Step [2900/12942], Loss: 2.0336, Perplexity: 7.6415\n",
      "Epoch [2/5], Step [3000/12942], Loss: 2.0701, Perplexity: 7.9256\n",
      "Epoch [2/5], Step [3100/12942], Loss: 2.1576, Perplexity: 8.6503\n",
      "Epoch [2/5], Step [3200/12942], Loss: 2.0360, Perplexity: 7.6600\n",
      "Epoch [2/5], Step [3300/12942], Loss: 2.0995, Perplexity: 8.1622\n",
      "Epoch [2/5], Step [3400/12942], Loss: 2.2019, Perplexity: 9.0419\n",
      "Epoch [2/5], Step [3500/12942], Loss: 2.7921, Perplexity: 16.3146\n",
      "Epoch [2/5], Step [3600/12942], Loss: 2.1954, Perplexity: 8.9834\n",
      "Epoch [2/5], Step [3700/12942], Loss: 2.2284, Perplexity: 9.2853\n",
      "Epoch [2/5], Step [3800/12942], Loss: 2.0756, Perplexity: 7.9693\n",
      "Epoch [2/5], Step [3900/12942], Loss: 2.0634, Perplexity: 7.8729\n",
      "Epoch [2/5], Step [4000/12942], Loss: 2.1613, Perplexity: 8.6823\n",
      "Epoch [2/5], Step [4100/12942], Loss: 1.9595, Perplexity: 7.0958\n",
      "Epoch [2/5], Step [4200/12942], Loss: 1.9303, Perplexity: 6.8918\n",
      "Epoch [2/5], Step [4300/12942], Loss: 2.0186, Perplexity: 7.5276\n",
      "Epoch [2/5], Step [4400/12942], Loss: 2.2637, Perplexity: 9.6182\n",
      "Epoch [2/5], Step [4500/12942], Loss: 2.5209, Perplexity: 12.4400\n",
      "Epoch [2/5], Step [4600/12942], Loss: 2.0932, Perplexity: 8.1111\n",
      "Epoch [2/5], Step [4700/12942], Loss: 2.3247, Perplexity: 10.2235\n",
      "Epoch [2/5], Step [4800/12942], Loss: 2.0513, Perplexity: 7.7783\n",
      "Epoch [2/5], Step [4900/12942], Loss: 2.1044, Perplexity: 8.2026\n",
      "Epoch [2/5], Step [5000/12942], Loss: 2.1190, Perplexity: 8.3231\n",
      "Epoch [2/5], Step [5100/12942], Loss: 2.4480, Perplexity: 11.5658\n",
      "Epoch [2/5], Step [5200/12942], Loss: 2.4322, Perplexity: 11.3836\n",
      "Epoch [2/5], Step [5300/12942], Loss: 1.8838, Perplexity: 6.5783\n",
      "Epoch [2/5], Step [5400/12942], Loss: 1.9581, Perplexity: 7.0855\n",
      "Epoch [2/5], Step [5500/12942], Loss: 2.0501, Perplexity: 7.7688\n",
      "Epoch [2/5], Step [5600/12942], Loss: 1.9819, Perplexity: 7.2562\n",
      "Epoch [2/5], Step [5700/12942], Loss: 2.2359, Perplexity: 9.3549\n",
      "Epoch [2/5], Step [5800/12942], Loss: 2.2097, Perplexity: 9.1132\n",
      "Epoch [2/5], Step [5900/12942], Loss: 2.4300, Perplexity: 11.3584\n",
      "Epoch [2/5], Step [6000/12942], Loss: 2.0793, Perplexity: 7.9990\n",
      "Epoch [2/5], Step [6100/12942], Loss: 2.1657, Perplexity: 8.7209\n",
      "Epoch [2/5], Step [6200/12942], Loss: 2.1662, Perplexity: 8.7254\n",
      "Epoch [2/5], Step [6300/12942], Loss: 2.1026, Perplexity: 8.1871\n",
      "Epoch [2/5], Step [6400/12942], Loss: 1.8908, Perplexity: 6.6248\n",
      "Epoch [2/5], Step [6500/12942], Loss: 4.2670, Perplexity: 71.3076\n",
      "Epoch [2/5], Step [6600/12942], Loss: 2.1605, Perplexity: 8.6755\n",
      "Epoch [2/5], Step [6700/12942], Loss: 2.0321, Perplexity: 7.6300\n",
      "Epoch [2/5], Step [6800/12942], Loss: 2.2730, Perplexity: 9.7086\n",
      "Epoch [2/5], Step [6900/12942], Loss: 2.4435, Perplexity: 11.5132\n",
      "Epoch [2/5], Step [7000/12942], Loss: 2.3103, Perplexity: 10.0772\n",
      "Epoch [2/5], Step [7100/12942], Loss: 2.1544, Perplexity: 8.6230\n",
      "Epoch [2/5], Step [7200/12942], Loss: 2.1016, Perplexity: 8.1791\n",
      "Epoch [2/5], Step [7300/12942], Loss: 2.0661, Perplexity: 7.8939\n",
      "Epoch [2/5], Step [7400/12942], Loss: 1.9731, Perplexity: 7.1928\n",
      "Epoch [2/5], Step [7500/12942], Loss: 2.0339, Perplexity: 7.6436\n",
      "Epoch [2/5], Step [7600/12942], Loss: 2.0370, Perplexity: 7.6679\n",
      "Epoch [2/5], Step [7700/12942], Loss: 2.0589, Perplexity: 7.8370\n",
      "Epoch [2/5], Step [7800/12942], Loss: 2.0742, Perplexity: 7.9580\n",
      "Epoch [2/5], Step [7900/12942], Loss: 1.8464, Perplexity: 6.3369\n",
      "Epoch [2/5], Step [8000/12942], Loss: 2.0643, Perplexity: 7.8800\n",
      "Epoch [2/5], Step [8100/12942], Loss: 2.0667, Perplexity: 7.8988\n",
      "Epoch [2/5], Step [8200/12942], Loss: 2.1300, Perplexity: 8.4153\n",
      "Epoch [2/5], Step [8300/12942], Loss: 2.0435, Perplexity: 7.7172\n",
      "Epoch [2/5], Step [8400/12942], Loss: 1.9347, Perplexity: 6.9220\n",
      "Epoch [2/5], Step [8500/12942], Loss: 1.9789, Perplexity: 7.2348\n",
      "Epoch [2/5], Step [8600/12942], Loss: 2.1458, Perplexity: 8.5491\n",
      "Epoch [2/5], Step [8700/12942], Loss: 1.9306, Perplexity: 6.8937\n",
      "Epoch [2/5], Step [8800/12942], Loss: 2.1212, Perplexity: 8.3411\n",
      "Epoch [2/5], Step [8900/12942], Loss: 2.1571, Perplexity: 8.6463\n",
      "Epoch [2/5], Step [9000/12942], Loss: 1.9849, Perplexity: 7.2781\n",
      "Epoch [2/5], Step [9100/12942], Loss: 2.3810, Perplexity: 10.8153\n",
      "Epoch [2/5], Step [9200/12942], Loss: 1.9553, Perplexity: 7.0661\n",
      "Epoch [2/5], Step [9300/12942], Loss: 2.3280, Perplexity: 10.2575\n",
      "Epoch [2/5], Step [9400/12942], Loss: 1.8562, Perplexity: 6.3995\n",
      "Epoch [2/5], Step [9500/12942], Loss: 2.3494, Perplexity: 10.4796\n",
      "Epoch [2/5], Step [9600/12942], Loss: 2.0622, Perplexity: 7.8629\n",
      "Epoch [2/5], Step [9700/12942], Loss: 2.0842, Perplexity: 8.0385\n",
      "Epoch [2/5], Step [9800/12942], Loss: 1.9232, Perplexity: 6.8428\n",
      "Epoch [2/5], Step [9900/12942], Loss: 2.0958, Perplexity: 8.1317\n",
      "Epoch [2/5], Step [10000/12942], Loss: 2.0819, Perplexity: 8.0194\n",
      "Epoch [2/5], Step [10100/12942], Loss: 2.4577, Perplexity: 11.6784\n",
      "Epoch [2/5], Step [10200/12942], Loss: 1.8576, Perplexity: 6.4081\n",
      "Epoch [2/5], Step [10300/12942], Loss: 2.0645, Perplexity: 7.8811\n",
      "Epoch [2/5], Step [10400/12942], Loss: 2.1396, Perplexity: 8.4961\n",
      "Epoch [2/5], Step [10500/12942], Loss: 2.1664, Perplexity: 8.7268\n",
      "Epoch [2/5], Step [10600/12942], Loss: 2.1889, Perplexity: 8.9257\n",
      "Epoch [2/5], Step [10700/12942], Loss: 2.2701, Perplexity: 9.6800\n",
      "Epoch [2/5], Step [10800/12942], Loss: 2.1544, Perplexity: 8.6227\n",
      "Epoch [2/5], Step [10900/12942], Loss: 2.6198, Perplexity: 13.7336\n",
      "Epoch [2/5], Step [11000/12942], Loss: 2.1135, Perplexity: 8.2772\n",
      "Epoch [2/5], Step [11100/12942], Loss: 1.9457, Perplexity: 6.9984\n",
      "Epoch [2/5], Step [11200/12942], Loss: 2.1676, Perplexity: 8.7373\n",
      "Epoch [2/5], Step [11300/12942], Loss: 2.0345, Perplexity: 7.6486\n",
      "Epoch [2/5], Step [11400/12942], Loss: 2.1747, Perplexity: 8.8000\n",
      "Epoch [2/5], Step [11500/12942], Loss: 2.1169, Perplexity: 8.3050\n",
      "Epoch [2/5], Step [11600/12942], Loss: 2.2273, Perplexity: 9.2748\n",
      "Epoch [2/5], Step [11700/12942], Loss: 2.2336, Perplexity: 9.3331\n",
      "Epoch [2/5], Step [11800/12942], Loss: 2.2396, Perplexity: 9.3891\n",
      "Epoch [2/5], Step [11900/12942], Loss: 2.0703, Perplexity: 7.9273\n",
      "Epoch [2/5], Step [12000/12942], Loss: 2.1433, Perplexity: 8.5275\n",
      "Epoch [2/5], Step [12100/12942], Loss: 2.2388, Perplexity: 9.3823\n",
      "Epoch [2/5], Step [12200/12942], Loss: 2.2018, Perplexity: 9.0411\n",
      "Epoch [2/5], Step [12300/12942], Loss: 2.0240, Perplexity: 7.5686\n",
      "Epoch [2/5], Step [12400/12942], Loss: 2.0533, Perplexity: 7.7938\n",
      "Epoch [2/5], Step [12500/12942], Loss: 2.4179, Perplexity: 11.2223\n",
      "Epoch [2/5], Step [12600/12942], Loss: 1.9648, Perplexity: 7.1338\n",
      "Epoch [2/5], Step [12700/12942], Loss: 1.9517, Perplexity: 7.0407\n",
      "Epoch [2/5], Step [12800/12942], Loss: 2.0189, Perplexity: 7.5303\n",
      "Epoch [2/5], Step [12900/12942], Loss: 1.9772, Perplexity: 7.2228\n",
      "Epoch [3/5], Step [100/12942], Loss: 2.3266, Perplexity: 10.2433\n",
      "Epoch [3/5], Step [200/12942], Loss: 1.9899, Perplexity: 7.3150\n",
      "Epoch [3/5], Step [300/12942], Loss: 2.1275, Perplexity: 8.3935\n",
      "Epoch [3/5], Step [400/12942], Loss: 1.9681, Perplexity: 7.1574\n",
      "Epoch [3/5], Step [500/12942], Loss: 2.3014, Perplexity: 9.9877\n",
      "Epoch [3/5], Step [600/12942], Loss: 2.1847, Perplexity: 8.8876\n",
      "Epoch [3/5], Step [700/12942], Loss: 1.8827, Perplexity: 6.5711\n",
      "Epoch [3/5], Step [800/12942], Loss: 2.0767, Perplexity: 7.9783\n",
      "Epoch [3/5], Step [900/12942], Loss: 2.1654, Perplexity: 8.7177\n",
      "Epoch [3/5], Step [1000/12942], Loss: 2.5262, Perplexity: 12.5057\n",
      "Epoch [3/5], Step [1100/12942], Loss: 1.9041, Perplexity: 6.7134\n",
      "Epoch [3/5], Step [1200/12942], Loss: 2.1781, Perplexity: 8.8293\n",
      "Epoch [3/5], Step [1300/12942], Loss: 2.6529, Perplexity: 14.1948\n",
      "Epoch [3/5], Step [1400/12942], Loss: 2.1681, Perplexity: 8.7418\n",
      "Epoch [3/5], Step [1500/12942], Loss: 2.0997, Perplexity: 8.1639\n",
      "Epoch [3/5], Step [1600/12942], Loss: 2.0507, Perplexity: 7.7731\n",
      "Epoch [3/5], Step [1700/12942], Loss: 2.1648, Perplexity: 8.7133\n",
      "Epoch [3/5], Step [1800/12942], Loss: 2.1314, Perplexity: 8.4266\n",
      "Epoch [3/5], Step [1900/12942], Loss: 2.0757, Perplexity: 7.9698\n",
      "Epoch [3/5], Step [2000/12942], Loss: 2.2471, Perplexity: 9.4605\n",
      "Epoch [3/5], Step [2100/12942], Loss: 1.8697, Perplexity: 6.4863\n",
      "Epoch [3/5], Step [2200/12942], Loss: 2.0244, Perplexity: 7.5714\n",
      "Epoch [3/5], Step [2300/12942], Loss: 2.2256, Perplexity: 9.2593\n",
      "Epoch [3/5], Step [2400/12942], Loss: 1.7826, Perplexity: 5.9455\n",
      "Epoch [3/5], Step [2500/12942], Loss: 2.8232, Perplexity: 16.8307\n",
      "Epoch [3/5], Step [2600/12942], Loss: 2.0120, Perplexity: 7.4783\n",
      "Epoch [3/5], Step [2700/12942], Loss: 2.1679, Perplexity: 8.7398\n",
      "Epoch [3/5], Step [2800/12942], Loss: 2.1932, Perplexity: 8.9636\n",
      "Epoch [3/5], Step [2900/12942], Loss: 2.2583, Perplexity: 9.5664\n",
      "Epoch [3/5], Step [3000/12942], Loss: 2.2805, Perplexity: 9.7812\n",
      "Epoch [3/5], Step [3100/12942], Loss: 1.9509, Perplexity: 7.0347\n",
      "Epoch [3/5], Step [3200/12942], Loss: 2.4395, Perplexity: 11.4671\n",
      "Epoch [3/5], Step [3300/12942], Loss: 2.0074, Perplexity: 7.4438\n",
      "Epoch [3/5], Step [3400/12942], Loss: 1.9919, Perplexity: 7.3293\n",
      "Epoch [3/5], Step [3500/12942], Loss: 2.1898, Perplexity: 8.9331\n",
      "Epoch [3/5], Step [3600/12942], Loss: 2.1839, Perplexity: 8.8807\n",
      "Epoch [3/5], Step [3700/12942], Loss: 1.9970, Perplexity: 7.3667\n",
      "Epoch [3/5], Step [3800/12942], Loss: 1.9443, Perplexity: 6.9884\n",
      "Epoch [3/5], Step [3900/12942], Loss: 1.9274, Perplexity: 6.8719\n",
      "Epoch [3/5], Step [4000/12942], Loss: 1.8740, Perplexity: 6.5145\n",
      "Epoch [3/5], Step [4100/12942], Loss: 2.5197, Perplexity: 12.4254\n",
      "Epoch [3/5], Step [4200/12942], Loss: 2.1075, Perplexity: 8.2277\n",
      "Epoch [3/5], Step [4300/12942], Loss: 1.8385, Perplexity: 6.2868\n",
      "Epoch [3/5], Step [4400/12942], Loss: 2.1023, Perplexity: 8.1849\n",
      "Epoch [3/5], Step [4500/12942], Loss: 1.8874, Perplexity: 6.6021\n",
      "Epoch [3/5], Step [4600/12942], Loss: 2.1593, Perplexity: 8.6650\n",
      "Epoch [3/5], Step [4700/12942], Loss: 2.2446, Perplexity: 9.4362\n",
      "Epoch [3/5], Step [4800/12942], Loss: 2.0533, Perplexity: 7.7933\n",
      "Epoch [3/5], Step [4900/12942], Loss: 2.1775, Perplexity: 8.8238\n",
      "Epoch [3/5], Step [5000/12942], Loss: 1.9236, Perplexity: 6.8457\n",
      "Epoch [3/5], Step [5100/12942], Loss: 2.1664, Perplexity: 8.7269\n",
      "Epoch [3/5], Step [5200/12942], Loss: 2.5426, Perplexity: 12.7129\n",
      "Epoch [3/5], Step [5300/12942], Loss: 2.1535, Perplexity: 8.6151\n",
      "Epoch [3/5], Step [5400/12942], Loss: 2.2323, Perplexity: 9.3213\n",
      "Epoch [3/5], Step [5500/12942], Loss: 2.0669, Perplexity: 7.9005\n",
      "Epoch [3/5], Step [5600/12942], Loss: 2.0019, Perplexity: 7.4033\n",
      "Epoch [3/5], Step [5700/12942], Loss: 2.0601, Perplexity: 7.8469\n",
      "Epoch [3/5], Step [5800/12942], Loss: 1.9884, Perplexity: 7.3038\n",
      "Epoch [3/5], Step [5900/12942], Loss: 2.0348, Perplexity: 7.6506\n",
      "Epoch [3/5], Step [6000/12942], Loss: 2.3829, Perplexity: 10.8367\n",
      "Epoch [3/5], Step [6100/12942], Loss: 2.1417, Perplexity: 8.5138\n",
      "Epoch [3/5], Step [6200/12942], Loss: 1.9606, Perplexity: 7.1036\n",
      "Epoch [3/5], Step [6300/12942], Loss: 2.1925, Perplexity: 8.9575\n",
      "Epoch [3/5], Step [6400/12942], Loss: 1.9386, Perplexity: 6.9489\n",
      "Epoch [3/5], Step [6500/12942], Loss: 2.0648, Perplexity: 7.8835\n",
      "Epoch [3/5], Step [6600/12942], Loss: 2.3021, Perplexity: 9.9953\n",
      "Epoch [3/5], Step [6700/12942], Loss: 1.9739, Perplexity: 7.1984\n",
      "Epoch [3/5], Step [6800/12942], Loss: 2.0451, Perplexity: 7.7297\n",
      "Epoch [3/5], Step [6900/12942], Loss: 1.9604, Perplexity: 7.1020\n",
      "Epoch [3/5], Step [7000/12942], Loss: 2.0882, Perplexity: 8.0704\n",
      "Epoch [3/5], Step [7100/12942], Loss: 2.0053, Perplexity: 7.4282\n",
      "Epoch [3/5], Step [7200/12942], Loss: 1.8617, Perplexity: 6.4347\n",
      "Epoch [3/5], Step [7300/12942], Loss: 2.2711, Perplexity: 9.6896\n",
      "Epoch [3/5], Step [7400/12942], Loss: 2.0431, Perplexity: 7.7148\n",
      "Epoch [3/5], Step [7500/12942], Loss: 2.2325, Perplexity: 9.3231\n",
      "Epoch [3/5], Step [7600/12942], Loss: 1.8562, Perplexity: 6.3995\n",
      "Epoch [3/5], Step [7700/12942], Loss: 1.9644, Perplexity: 7.1309\n",
      "Epoch [3/5], Step [7800/12942], Loss: 2.0857, Perplexity: 8.0503\n",
      "Epoch [3/5], Step [7900/12942], Loss: 2.0285, Perplexity: 7.6024\n",
      "Epoch [3/5], Step [8000/12942], Loss: 2.3397, Perplexity: 10.3780\n",
      "Epoch [3/5], Step [8100/12942], Loss: 2.1022, Perplexity: 8.1844\n",
      "Epoch [3/5], Step [8200/12942], Loss: 1.8662, Perplexity: 6.4637\n",
      "Epoch [3/5], Step [8300/12942], Loss: 2.0037, Perplexity: 7.4163\n",
      "Epoch [3/5], Step [8400/12942], Loss: 1.7366, Perplexity: 5.6778\n",
      "Epoch [3/5], Step [8500/12942], Loss: 1.8057, Perplexity: 6.0842\n",
      "Epoch [3/5], Step [8600/12942], Loss: 2.0232, Perplexity: 7.5622\n",
      "Epoch [3/5], Step [8700/12942], Loss: 1.6569, Perplexity: 5.2430\n",
      "Epoch [3/5], Step [8800/12942], Loss: 2.0282, Perplexity: 7.6002\n",
      "Epoch [3/5], Step [8900/12942], Loss: 1.9444, Perplexity: 6.9897\n",
      "Epoch [3/5], Step [9000/12942], Loss: 2.1011, Perplexity: 8.1752\n",
      "Epoch [3/5], Step [9100/12942], Loss: 2.5519, Perplexity: 12.8308\n",
      "Epoch [3/5], Step [9200/12942], Loss: 2.7842, Perplexity: 16.1866\n",
      "Epoch [3/5], Step [9300/12942], Loss: 2.3054, Perplexity: 10.0279\n",
      "Epoch [3/5], Step [9400/12942], Loss: 2.6014, Perplexity: 13.4826\n",
      "Epoch [3/5], Step [9500/12942], Loss: 1.9844, Perplexity: 7.2749\n",
      "Epoch [3/5], Step [9600/12942], Loss: 1.9699, Perplexity: 7.1697\n",
      "Epoch [3/5], Step [9700/12942], Loss: 1.8480, Perplexity: 6.3474\n",
      "Epoch [3/5], Step [9800/12942], Loss: 2.2156, Perplexity: 9.1669\n",
      "Epoch [3/5], Step [9900/12942], Loss: 1.9378, Perplexity: 6.9438\n",
      "Epoch [3/5], Step [10000/12942], Loss: 2.5407, Perplexity: 12.6886\n",
      "Epoch [3/5], Step [10100/12942], Loss: 2.1959, Perplexity: 8.9880\n",
      "Epoch [3/5], Step [10200/12942], Loss: 2.2484, Perplexity: 9.4723\n",
      "Epoch [3/5], Step [10300/12942], Loss: 2.0633, Perplexity: 7.8717\n",
      "Epoch [3/5], Step [10400/12942], Loss: 1.9803, Perplexity: 7.2449\n",
      "Epoch [3/5], Step [10500/12942], Loss: 1.9368, Perplexity: 6.9363\n",
      "Epoch [3/5], Step [10600/12942], Loss: 2.2000, Perplexity: 9.0249\n",
      "Epoch [3/5], Step [10700/12942], Loss: 2.0603, Perplexity: 7.8486\n",
      "Epoch [3/5], Step [10800/12942], Loss: 1.9631, Perplexity: 7.1213\n",
      "Epoch [3/5], Step [10900/12942], Loss: 2.0892, Perplexity: 8.0783\n",
      "Epoch [3/5], Step [11000/12942], Loss: 2.1271, Perplexity: 8.3904\n",
      "Epoch [3/5], Step [11100/12942], Loss: 2.4777, Perplexity: 11.9140\n",
      "Epoch [3/5], Step [11200/12942], Loss: 1.8622, Perplexity: 6.4379\n",
      "Epoch [3/5], Step [11300/12942], Loss: 2.2589, Perplexity: 9.5722\n",
      "Epoch [3/5], Step [11400/12942], Loss: 2.1327, Perplexity: 8.4373\n",
      "Epoch [3/5], Step [11500/12942], Loss: 2.1719, Perplexity: 8.7748\n",
      "Epoch [3/5], Step [11600/12942], Loss: 2.0570, Perplexity: 7.8224\n",
      "Epoch [3/5], Step [11700/12942], Loss: 1.8558, Perplexity: 6.3968\n",
      "Epoch [3/5], Step [11800/12942], Loss: 2.1032, Perplexity: 8.1926\n",
      "Epoch [3/5], Step [11900/12942], Loss: 1.8552, Perplexity: 6.3931\n",
      "Epoch [3/5], Step [12000/12942], Loss: 2.1401, Perplexity: 8.5006\n",
      "Epoch [3/5], Step [12100/12942], Loss: 1.8462, Perplexity: 6.3356\n",
      "Epoch [3/5], Step [12200/12942], Loss: 2.1048, Perplexity: 8.2058\n",
      "Epoch [3/5], Step [12300/12942], Loss: 1.8904, Perplexity: 6.6223\n",
      "Epoch [3/5], Step [12400/12942], Loss: 1.9305, Perplexity: 6.8927\n",
      "Epoch [3/5], Step [12500/12942], Loss: 1.6974, Perplexity: 5.4597\n",
      "Epoch [3/5], Step [12600/12942], Loss: 1.7414, Perplexity: 5.7054\n",
      "Epoch [3/5], Step [12700/12942], Loss: 2.0561, Perplexity: 7.8152\n",
      "Epoch [3/5], Step [12800/12942], Loss: 2.0783, Perplexity: 7.9909\n",
      "Epoch [3/5], Step [12900/12942], Loss: 2.0514, Perplexity: 7.7792\n",
      "Epoch [4/5], Step [100/12942], Loss: 1.7943, Perplexity: 6.0152\n",
      "Epoch [4/5], Step [200/12942], Loss: 2.0116, Perplexity: 7.4755\n",
      "Epoch [4/5], Step [300/12942], Loss: 1.9472, Perplexity: 7.0094\n",
      "Epoch [4/5], Step [400/12942], Loss: 2.2933, Perplexity: 9.9079\n",
      "Epoch [4/5], Step [500/12942], Loss: 1.9544, Perplexity: 7.0596\n",
      "Epoch [4/5], Step [600/12942], Loss: 1.8981, Perplexity: 6.6731\n",
      "Epoch [4/5], Step [700/12942], Loss: 2.3023, Perplexity: 9.9969\n",
      "Epoch [4/5], Step [800/12942], Loss: 1.7053, Perplexity: 5.5030\n",
      "Epoch [4/5], Step [900/12942], Loss: 1.9583, Perplexity: 7.0873\n",
      "Epoch [4/5], Step [1000/12942], Loss: 2.0057, Perplexity: 7.4311\n",
      "Epoch [4/5], Step [1100/12942], Loss: 2.0032, Perplexity: 7.4128\n",
      "Epoch [4/5], Step [1200/12942], Loss: 2.2001, Perplexity: 9.0256\n",
      "Epoch [4/5], Step [1300/12942], Loss: 1.7225, Perplexity: 5.5984\n",
      "Epoch [4/5], Step [1400/12942], Loss: 2.1112, Perplexity: 8.2581\n",
      "Epoch [4/5], Step [1500/12942], Loss: 2.3781, Perplexity: 10.7841\n",
      "Epoch [4/5], Step [1600/12942], Loss: 2.3737, Perplexity: 10.7373\n",
      "Epoch [4/5], Step [1700/12942], Loss: 2.1654, Perplexity: 8.7177\n",
      "Epoch [4/5], Step [1800/12942], Loss: 2.3876, Perplexity: 10.8868\n",
      "Epoch [4/5], Step [1900/12942], Loss: 1.8003, Perplexity: 6.0517\n",
      "Epoch [4/5], Step [2000/12942], Loss: 1.9900, Perplexity: 7.3153\n",
      "Epoch [4/5], Step [2100/12942], Loss: 2.1826, Perplexity: 8.8691\n",
      "Epoch [4/5], Step [2200/12942], Loss: 1.9365, Perplexity: 6.9344\n",
      "Epoch [4/5], Step [2300/12942], Loss: 1.9071, Perplexity: 6.7336\n",
      "Epoch [4/5], Step [2400/12942], Loss: 1.9618, Perplexity: 7.1119\n",
      "Epoch [4/5], Step [2500/12942], Loss: 2.1670, Perplexity: 8.7323\n",
      "Epoch [4/5], Step [2600/12942], Loss: 1.9679, Perplexity: 7.1554\n",
      "Epoch [4/5], Step [2700/12942], Loss: 2.1701, Perplexity: 8.7595\n",
      "Epoch [4/5], Step [2800/12942], Loss: 2.0427, Perplexity: 7.7114\n",
      "Epoch [4/5], Step [2900/12942], Loss: 2.0112, Perplexity: 7.4722\n",
      "Epoch [4/5], Step [3000/12942], Loss: 2.0396, Perplexity: 7.6879\n",
      "Epoch [4/5], Step [3100/12942], Loss: 1.8974, Perplexity: 6.6684\n",
      "Epoch [4/5], Step [3200/12942], Loss: 2.4488, Perplexity: 11.5745\n",
      "Epoch [4/5], Step [3300/12942], Loss: 1.8320, Perplexity: 6.2462\n",
      "Epoch [4/5], Step [3400/12942], Loss: 2.2299, Perplexity: 9.2985\n",
      "Epoch [4/5], Step [3500/12942], Loss: 2.2712, Perplexity: 9.6913\n",
      "Epoch [4/5], Step [3600/12942], Loss: 1.9051, Perplexity: 6.7200\n",
      "Epoch [4/5], Step [3700/12942], Loss: 1.8478, Perplexity: 6.3460\n",
      "Epoch [4/5], Step [3800/12942], Loss: 1.9767, Perplexity: 7.2189\n",
      "Epoch [4/5], Step [3900/12942], Loss: 1.8137, Perplexity: 6.1333\n",
      "Epoch [4/5], Step [4000/12942], Loss: 1.9404, Perplexity: 6.9615\n",
      "Epoch [4/5], Step [4100/12942], Loss: 2.4213, Perplexity: 11.2610\n",
      "Epoch [4/5], Step [4200/12942], Loss: 2.1292, Perplexity: 8.4078\n",
      "Epoch [4/5], Step [4300/12942], Loss: 1.9255, Perplexity: 6.8586\n",
      "Epoch [4/5], Step [4400/12942], Loss: 2.1172, Perplexity: 8.3082\n",
      "Epoch [4/5], Step [4500/12942], Loss: 2.3922, Perplexity: 10.9378\n",
      "Epoch [4/5], Step [4600/12942], Loss: 2.0271, Perplexity: 7.5919\n",
      "Epoch [4/5], Step [4700/12942], Loss: 1.9777, Perplexity: 7.2262\n",
      "Epoch [4/5], Step [4800/12942], Loss: 1.7424, Perplexity: 5.7111\n",
      "Epoch [4/5], Step [4900/12942], Loss: 2.1689, Perplexity: 8.7485\n",
      "Epoch [4/5], Step [5000/12942], Loss: 1.9531, Perplexity: 7.0506\n",
      "Epoch [4/5], Step [5100/12942], Loss: 2.1469, Perplexity: 8.5586\n",
      "Epoch [4/5], Step [5200/12942], Loss: 1.7923, Perplexity: 6.0035\n",
      "Epoch [4/5], Step [5300/12942], Loss: 1.9648, Perplexity: 7.1336\n",
      "Epoch [4/5], Step [5400/12942], Loss: 1.9865, Perplexity: 7.2899\n",
      "Epoch [4/5], Step [5500/12942], Loss: 2.4637, Perplexity: 11.7482\n",
      "Epoch [4/5], Step [5600/12942], Loss: 2.7333, Perplexity: 15.3833\n",
      "Epoch [4/5], Step [5700/12942], Loss: 2.0384, Perplexity: 7.6781\n",
      "Epoch [4/5], Step [5800/12942], Loss: 2.0592, Perplexity: 7.8397\n",
      "Epoch [4/5], Step [5900/12942], Loss: 1.9350, Perplexity: 6.9241\n",
      "Epoch [4/5], Step [6000/12942], Loss: 2.1091, Perplexity: 8.2410\n",
      "Epoch [4/5], Step [6100/12942], Loss: 2.0231, Perplexity: 7.5618\n",
      "Epoch [4/5], Step [6200/12942], Loss: 2.1142, Perplexity: 8.2825\n",
      "Epoch [4/5], Step [6300/12942], Loss: 1.8470, Perplexity: 6.3405\n",
      "Epoch [4/5], Step [6400/12942], Loss: 1.9857, Perplexity: 7.2841\n",
      "Epoch [4/5], Step [6500/12942], Loss: 2.4494, Perplexity: 11.5814\n",
      "Epoch [4/5], Step [6600/12942], Loss: 1.7896, Perplexity: 5.9873\n",
      "Epoch [4/5], Step [6700/12942], Loss: 1.9464, Perplexity: 7.0034\n",
      "Epoch [4/5], Step [6800/12942], Loss: 1.9651, Perplexity: 7.1353\n",
      "Epoch [4/5], Step [6900/12942], Loss: 2.6075, Perplexity: 13.5653\n",
      "Epoch [4/5], Step [7000/12942], Loss: 1.7034, Perplexity: 5.4924\n",
      "Epoch [4/5], Step [7100/12942], Loss: 1.9922, Perplexity: 7.3315\n",
      "Epoch [4/5], Step [7200/12942], Loss: 2.1782, Perplexity: 8.8300\n",
      "Epoch [4/5], Step [7300/12942], Loss: 2.0542, Perplexity: 7.8002\n",
      "Epoch [4/5], Step [7400/12942], Loss: 1.9932, Perplexity: 7.3387\n",
      "Epoch [4/5], Step [7500/12942], Loss: 2.0778, Perplexity: 7.9869\n",
      "Epoch [4/5], Step [7600/12942], Loss: 1.7336, Perplexity: 5.6610\n",
      "Epoch [4/5], Step [7700/12942], Loss: 2.0562, Perplexity: 7.8166\n",
      "Epoch [4/5], Step [7800/12942], Loss: 2.2991, Perplexity: 9.9654\n",
      "Epoch [4/5], Step [7900/12942], Loss: 2.0182, Perplexity: 7.5247\n",
      "Epoch [4/5], Step [8000/12942], Loss: 1.9512, Perplexity: 7.0373\n",
      "Epoch [4/5], Step [8100/12942], Loss: 2.1140, Perplexity: 8.2816\n",
      "Epoch [4/5], Step [8200/12942], Loss: 2.2186, Perplexity: 9.1943\n",
      "Epoch [4/5], Step [8300/12942], Loss: 2.1987, Perplexity: 9.0132\n",
      "Epoch [4/5], Step [8400/12942], Loss: 1.9997, Perplexity: 7.3866\n",
      "Epoch [4/5], Step [8500/12942], Loss: 2.0591, Perplexity: 7.8389\n",
      "Epoch [4/5], Step [8600/12942], Loss: 2.9570, Perplexity: 19.2408\n",
      "Epoch [4/5], Step [8700/12942], Loss: 2.0171, Perplexity: 7.5168\n",
      "Epoch [4/5], Step [8800/12942], Loss: 1.9620, Perplexity: 7.1134\n",
      "Epoch [4/5], Step [8900/12942], Loss: 1.9777, Perplexity: 7.2258\n",
      "Epoch [4/5], Step [9000/12942], Loss: 1.9946, Perplexity: 7.3495\n",
      "Epoch [4/5], Step [9100/12942], Loss: 1.9618, Perplexity: 7.1122\n",
      "Epoch [4/5], Step [9200/12942], Loss: 1.9053, Perplexity: 6.7212\n",
      "Epoch [4/5], Step [9300/12942], Loss: 2.1963, Perplexity: 8.9921\n",
      "Epoch [4/5], Step [9400/12942], Loss: 1.8146, Perplexity: 6.1388\n",
      "Epoch [4/5], Step [9500/12942], Loss: 2.2594, Perplexity: 9.5769\n",
      "Epoch [4/5], Step [9600/12942], Loss: 2.0924, Perplexity: 8.1041\n",
      "Epoch [4/5], Step [9700/12942], Loss: 2.1538, Perplexity: 8.6175\n",
      "Epoch [4/5], Step [9800/12942], Loss: 1.7221, Perplexity: 5.5962\n",
      "Epoch [4/5], Step [9900/12942], Loss: 1.9265, Perplexity: 6.8654\n",
      "Epoch [4/5], Step [10000/12942], Loss: 1.9715, Perplexity: 7.1817\n",
      "Epoch [4/5], Step [10100/12942], Loss: 2.1144, Perplexity: 8.2848\n",
      "Epoch [4/5], Step [10200/12942], Loss: 1.9126, Perplexity: 6.7706\n",
      "Epoch [4/5], Step [10300/12942], Loss: 2.2185, Perplexity: 9.1932\n",
      "Epoch [4/5], Step [10400/12942], Loss: 2.0882, Perplexity: 8.0705\n",
      "Epoch [4/5], Step [10500/12942], Loss: 1.8620, Perplexity: 6.4369\n",
      "Epoch [4/5], Step [10600/12942], Loss: 2.0115, Perplexity: 7.4743\n",
      "Epoch [4/5], Step [10700/12942], Loss: 2.0213, Perplexity: 7.5478\n",
      "Epoch [4/5], Step [10800/12942], Loss: 1.8654, Perplexity: 6.4586\n",
      "Epoch [4/5], Step [10900/12942], Loss: 2.2791, Perplexity: 9.7677\n",
      "Epoch [4/5], Step [11000/12942], Loss: 1.7333, Perplexity: 5.6595\n",
      "Epoch [4/5], Step [11100/12942], Loss: 1.8158, Perplexity: 6.1459\n",
      "Epoch [4/5], Step [11200/12942], Loss: 2.0045, Perplexity: 7.4223\n",
      "Epoch [4/5], Step [11300/12942], Loss: 1.8795, Perplexity: 6.5505\n",
      "Epoch [4/5], Step [11400/12942], Loss: 1.9429, Perplexity: 6.9790\n",
      "Epoch [4/5], Step [11500/12942], Loss: 1.9708, Perplexity: 7.1764\n",
      "Epoch [4/5], Step [11600/12942], Loss: 1.9034, Perplexity: 6.7088\n",
      "Epoch [4/5], Step [11700/12942], Loss: 2.1110, Perplexity: 8.2562\n",
      "Epoch [4/5], Step [11800/12942], Loss: 1.9726, Perplexity: 7.1892\n",
      "Epoch [4/5], Step [11900/12942], Loss: 1.8107, Perplexity: 6.1146\n",
      "Epoch [4/5], Step [12000/12942], Loss: 2.2996, Perplexity: 9.9702\n",
      "Epoch [4/5], Step [12100/12942], Loss: 2.2641, Perplexity: 9.6222\n",
      "Epoch [4/5], Step [12200/12942], Loss: 2.1983, Perplexity: 9.0099\n",
      "Epoch [4/5], Step [12300/12942], Loss: 2.1431, Perplexity: 8.5259\n",
      "Epoch [4/5], Step [12400/12942], Loss: 2.1093, Perplexity: 8.2421\n",
      "Epoch [4/5], Step [12500/12942], Loss: 1.9925, Perplexity: 7.3339\n",
      "Epoch [4/5], Step [12600/12942], Loss: 1.9852, Perplexity: 7.2808\n",
      "Epoch [4/5], Step [12700/12942], Loss: 1.9869, Perplexity: 7.2930\n",
      "Epoch [4/5], Step [12800/12942], Loss: 2.1852, Perplexity: 8.8928\n",
      "Epoch [4/5], Step [12900/12942], Loss: 1.8409, Perplexity: 6.3020\n",
      "Epoch [5/5], Step [100/12942], Loss: 2.0647, Perplexity: 7.8827\n",
      "Epoch [5/5], Step [200/12942], Loss: 2.1113, Perplexity: 8.2587\n",
      "Epoch [5/5], Step [300/12942], Loss: 1.9027, Perplexity: 6.7037\n",
      "Epoch [5/5], Step [400/12942], Loss: 1.8791, Perplexity: 6.5479\n",
      "Epoch [5/5], Step [500/12942], Loss: 2.4347, Perplexity: 11.4127\n",
      "Epoch [5/5], Step [600/12942], Loss: 2.1540, Perplexity: 8.6194\n",
      "Epoch [5/5], Step [700/12942], Loss: 2.2527, Perplexity: 9.5132\n",
      "Epoch [5/5], Step [800/12942], Loss: 2.1441, Perplexity: 8.5344\n",
      "Epoch [5/5], Step [900/12942], Loss: 1.9511, Perplexity: 7.0363\n",
      "Epoch [5/5], Step [1000/12942], Loss: 2.0314, Perplexity: 7.6246\n",
      "Epoch [5/5], Step [1100/12942], Loss: 1.9647, Perplexity: 7.1327\n",
      "Epoch [5/5], Step [1200/12942], Loss: 1.7505, Perplexity: 5.7573\n",
      "Epoch [5/5], Step [1300/12942], Loss: 2.0165, Perplexity: 7.5123\n",
      "Epoch [5/5], Step [1400/12942], Loss: 2.0310, Perplexity: 7.6219\n",
      "Epoch [5/5], Step [1500/12942], Loss: 2.0651, Perplexity: 7.8861\n",
      "Epoch [5/5], Step [1600/12942], Loss: 1.8031, Perplexity: 6.0686\n",
      "Epoch [5/5], Step [1700/12942], Loss: 2.0645, Perplexity: 7.8817\n",
      "Epoch [5/5], Step [1800/12942], Loss: 1.8688, Perplexity: 6.4802\n",
      "Epoch [5/5], Step [1900/12942], Loss: 1.8819, Perplexity: 6.5662\n",
      "Epoch [5/5], Step [2000/12942], Loss: 2.5166, Perplexity: 12.3861\n",
      "Epoch [5/5], Step [2100/12942], Loss: 2.0891, Perplexity: 8.0774\n",
      "Epoch [5/5], Step [2200/12942], Loss: 2.3522, Perplexity: 10.5085\n",
      "Epoch [5/5], Step [2300/12942], Loss: 1.8485, Perplexity: 6.3501\n",
      "Epoch [5/5], Step [2400/12942], Loss: 2.0116, Perplexity: 7.4752\n",
      "Epoch [5/5], Step [2500/12942], Loss: 2.2395, Perplexity: 9.3891\n",
      "Epoch [5/5], Step [2600/12942], Loss: 1.6725, Perplexity: 5.3256\n",
      "Epoch [5/5], Step [2700/12942], Loss: 2.2667, Perplexity: 9.6474\n",
      "Epoch [5/5], Step [2800/12942], Loss: 1.9491, Perplexity: 7.0225\n",
      "Epoch [5/5], Step [2900/12942], Loss: 1.9532, Perplexity: 7.0513\n",
      "Epoch [5/5], Step [3000/12942], Loss: 1.9770, Perplexity: 7.2211\n",
      "Epoch [5/5], Step [3100/12942], Loss: 2.3905, Perplexity: 10.9189\n",
      "Epoch [5/5], Step [3200/12942], Loss: 1.9768, Perplexity: 7.2195\n",
      "Epoch [5/5], Step [3300/12942], Loss: 2.2907, Perplexity: 9.8820\n",
      "Epoch [5/5], Step [3400/12942], Loss: 1.8704, Perplexity: 6.4909\n",
      "Epoch [5/5], Step [3500/12942], Loss: 2.0119, Perplexity: 7.4773\n",
      "Epoch [5/5], Step [3600/12942], Loss: 1.9092, Perplexity: 6.7474\n",
      "Epoch [5/5], Step [3700/12942], Loss: 1.8652, Perplexity: 6.4573\n",
      "Epoch [5/5], Step [3800/12942], Loss: 1.6056, Perplexity: 4.9810\n",
      "Epoch [5/5], Step [3900/12942], Loss: 2.6561, Perplexity: 14.2408\n",
      "Epoch [5/5], Step [4000/12942], Loss: 1.8796, Perplexity: 6.5511\n",
      "Epoch [5/5], Step [4100/12942], Loss: 2.2328, Perplexity: 9.3263\n",
      "Epoch [5/5], Step [4200/12942], Loss: 1.8764, Perplexity: 6.5297\n",
      "Epoch [5/5], Step [4300/12942], Loss: 2.1941, Perplexity: 8.9717\n",
      "Epoch [5/5], Step [4400/12942], Loss: 2.1868, Perplexity: 8.9065\n",
      "Epoch [5/5], Step [4500/12942], Loss: 1.6953, Perplexity: 5.4484\n",
      "Epoch [5/5], Step [4600/12942], Loss: 2.0087, Perplexity: 7.4534\n",
      "Epoch [5/5], Step [4700/12942], Loss: 1.6808, Perplexity: 5.3697\n",
      "Epoch [5/5], Step [4800/12942], Loss: 1.8802, Perplexity: 6.5547\n",
      "Epoch [5/5], Step [4900/12942], Loss: 1.9867, Perplexity: 7.2918\n",
      "Epoch [5/5], Step [5000/12942], Loss: 1.9023, Perplexity: 6.7016\n",
      "Epoch [5/5], Step [5100/12942], Loss: 2.0077, Perplexity: 7.4465\n",
      "Epoch [5/5], Step [5200/12942], Loss: 1.9130, Perplexity: 6.7737\n",
      "Epoch [5/5], Step [5300/12942], Loss: 2.0108, Perplexity: 7.4695\n",
      "Epoch [5/5], Step [5400/12942], Loss: 1.9968, Perplexity: 7.3653\n",
      "Epoch [5/5], Step [5500/12942], Loss: 1.9501, Perplexity: 7.0293\n",
      "Epoch [5/5], Step [5600/12942], Loss: 1.9008, Perplexity: 6.6915\n",
      "Epoch [5/5], Step [5700/12942], Loss: 2.4109, Perplexity: 11.1445\n",
      "Epoch [5/5], Step [5800/12942], Loss: 1.6640, Perplexity: 5.2806\n",
      "Epoch [5/5], Step [5900/12942], Loss: 1.9555, Perplexity: 7.0673\n",
      "Epoch [5/5], Step [6000/12942], Loss: 2.1181, Perplexity: 8.3156\n",
      "Epoch [5/5], Step [6100/12942], Loss: 1.9406, Perplexity: 6.9626\n",
      "Epoch [5/5], Step [6200/12942], Loss: 1.7585, Perplexity: 5.8034\n",
      "Epoch [5/5], Step [6300/12942], Loss: 2.2687, Perplexity: 9.6666\n",
      "Epoch [5/5], Step [6400/12942], Loss: 1.9805, Perplexity: 7.2463\n",
      "Epoch [5/5], Step [6500/12942], Loss: 1.7702, Perplexity: 5.8719\n",
      "Epoch [5/5], Step [6600/12942], Loss: 2.2224, Perplexity: 9.2299\n",
      "Epoch [5/5], Step [6700/12942], Loss: 1.7422, Perplexity: 5.7102\n",
      "Epoch [5/5], Step [6800/12942], Loss: 2.3219, Perplexity: 10.1954\n",
      "Epoch [5/5], Step [6900/12942], Loss: 1.8388, Perplexity: 6.2889\n",
      "Epoch [5/5], Step [7000/12942], Loss: 2.1455, Perplexity: 8.5464\n",
      "Epoch [5/5], Step [7100/12942], Loss: 2.0841, Perplexity: 8.0370\n",
      "Epoch [5/5], Step [7200/12942], Loss: 1.7708, Perplexity: 5.8756\n",
      "Epoch [5/5], Step [7300/12942], Loss: 2.0319, Perplexity: 7.6289\n",
      "Epoch [5/5], Step [7400/12942], Loss: 1.8650, Perplexity: 6.4560\n",
      "Epoch [5/5], Step [7500/12942], Loss: 2.0225, Perplexity: 7.5573\n",
      "Epoch [5/5], Step [7600/12942], Loss: 1.9913, Perplexity: 7.3249\n",
      "Epoch [5/5], Step [7700/12942], Loss: 1.7730, Perplexity: 5.8882\n",
      "Epoch [5/5], Step [7800/12942], Loss: 1.8725, Perplexity: 6.5043\n",
      "Epoch [5/5], Step [7900/12942], Loss: 1.8854, Perplexity: 6.5889\n",
      "Epoch [5/5], Step [8000/12942], Loss: 2.0938, Perplexity: 8.1156\n",
      "Epoch [5/5], Step [8100/12942], Loss: 1.8560, Perplexity: 6.3980\n",
      "Epoch [5/5], Step [8200/12942], Loss: 2.0101, Perplexity: 7.4638\n",
      "Epoch [5/5], Step [8300/12942], Loss: 1.9990, Perplexity: 7.3818\n",
      "Epoch [5/5], Step [8400/12942], Loss: 2.0014, Perplexity: 7.3993\n",
      "Epoch [5/5], Step [8500/12942], Loss: 1.7314, Perplexity: 5.6487\n",
      "Epoch [5/5], Step [8600/12942], Loss: 2.1377, Perplexity: 8.4801\n",
      "Epoch [5/5], Step [8700/12942], Loss: 1.9304, Perplexity: 6.8922\n",
      "Epoch [5/5], Step [8800/12942], Loss: 1.9821, Perplexity: 7.2582\n",
      "Epoch [5/5], Step [8900/12942], Loss: 1.8751, Perplexity: 6.5213\n",
      "Epoch [5/5], Step [9000/12942], Loss: 1.9398, Perplexity: 6.9572\n",
      "Epoch [5/5], Step [9100/12942], Loss: 1.9429, Perplexity: 6.9790\n",
      "Epoch [5/5], Step [9200/12942], Loss: 1.8322, Perplexity: 6.2475\n",
      "Epoch [5/5], Step [9300/12942], Loss: 1.8409, Perplexity: 6.3024\n",
      "Epoch [5/5], Step [9400/12942], Loss: 1.8945, Perplexity: 6.6489\n",
      "Epoch [5/5], Step [9500/12942], Loss: 1.9250, Perplexity: 6.8554\n",
      "Epoch [5/5], Step [9600/12942], Loss: 2.0906, Perplexity: 8.0894\n",
      "Epoch [5/5], Step [9700/12942], Loss: 2.3606, Perplexity: 10.5976\n",
      "Epoch [5/5], Step [9800/12942], Loss: 1.9589, Perplexity: 7.0914\n",
      "Epoch [5/5], Step [9900/12942], Loss: 2.0239, Perplexity: 7.5675\n",
      "Epoch [5/5], Step [10000/12942], Loss: 2.0764, Perplexity: 7.9760\n",
      "Epoch [5/5], Step [10100/12942], Loss: 1.7773, Perplexity: 5.9141\n",
      "Epoch [5/5], Step [10200/12942], Loss: 1.8500, Perplexity: 6.3600\n",
      "Epoch [5/5], Step [10300/12942], Loss: 1.9852, Perplexity: 7.2806\n",
      "Epoch [5/5], Step [10400/12942], Loss: 2.1400, Perplexity: 8.4998\n",
      "Epoch [5/5], Step [10500/12942], Loss: 2.0621, Perplexity: 7.8627\n",
      "Epoch [5/5], Step [10600/12942], Loss: 1.8316, Perplexity: 6.2437\n",
      "Epoch [5/5], Step [10700/12942], Loss: 1.9284, Perplexity: 6.8786\n",
      "Epoch [5/5], Step [10800/12942], Loss: 1.8474, Perplexity: 6.3431\n",
      "Epoch [5/5], Step [10900/12942], Loss: 1.8433, Perplexity: 6.3172\n",
      "Epoch [5/5], Step [11000/12942], Loss: 1.8402, Perplexity: 6.2981\n",
      "Epoch [5/5], Step [11100/12942], Loss: 2.1048, Perplexity: 8.2054\n",
      "Epoch [5/5], Step [11200/12942], Loss: 1.8099, Perplexity: 6.1098\n",
      "Epoch [5/5], Step [11300/12942], Loss: 2.1777, Perplexity: 8.8261\n",
      "Epoch [5/5], Step [11400/12942], Loss: 1.9080, Perplexity: 6.7398\n",
      "Epoch [5/5], Step [11500/12942], Loss: 1.7154, Perplexity: 5.5589\n",
      "Epoch [5/5], Step [11600/12942], Loss: 1.8654, Perplexity: 6.4583\n",
      "Epoch [5/5], Step [11700/12942], Loss: 2.0694, Perplexity: 7.9201\n",
      "Epoch [5/5], Step [11800/12942], Loss: 1.8094, Perplexity: 6.1065\n",
      "Epoch [5/5], Step [11900/12942], Loss: 1.8215, Perplexity: 6.1809\n",
      "Epoch [5/5], Step [12000/12942], Loss: 2.0451, Perplexity: 7.7296\n",
      "Epoch [5/5], Step [12100/12942], Loss: 2.0476, Perplexity: 7.7494\n",
      "Epoch [5/5], Step [12200/12942], Loss: 2.0004, Perplexity: 7.3921\n",
      "Epoch [5/5], Step [12300/12942], Loss: 1.7640, Perplexity: 5.8356\n",
      "Epoch [5/5], Step [12400/12942], Loss: 1.9298, Perplexity: 6.8879\n",
      "Epoch [5/5], Step [12500/12942], Loss: 1.9873, Perplexity: 7.2958\n",
      "Epoch [5/5], Step [12600/12942], Loss: 1.7229, Perplexity: 5.6010\n",
      "Epoch [5/5], Step [12700/12942], Loss: 1.7613, Perplexity: 5.8199\n",
      "Epoch [5/5], Step [12800/12942], Loss: 2.1204, Perplexity: 8.3346\n",
      "Epoch [5/5], Step [12900/12942], Loss: 1.9659, Perplexity: 7.1412\n",
      "Epoch [5/5], Step [12942/12942], Loss: 2.3848, Perplexity: 10.8570"
     ]
    }
   ],
   "source": [
    "import torch.utils.data as data\n",
    "import numpy as np\n",
    "import os\n",
    "import requests\n",
    "import time\n",
    "\n",
    "# Open the training log file.\n",
    "f = open(log_file, 'w')\n",
    "\n",
    "old_time = time.time()\n",
    "# response = requests.request(\"GET\", \n",
    "#                             \"http://metadata.google.internal/computeMetadata/v1/instance/attributes/keep_alive_token\", \n",
    "#                             headers={\"Metadata-Flavor\":\"Google\"})\n",
    "\n",
    "for epoch in range(1, num_epochs+1):\n",
    "    \n",
    "    for i_step in range(1, total_step+1):\n",
    "        \n",
    "#         if time.time() - old_time > 60:\n",
    "#             old_time = time.time()\n",
    "#             requests.request(\"POST\", \n",
    "#                              \"https://nebula.udacity.com/api/v1/remote/keep-alive\", \n",
    "#                              headers={'Authorization': \"STAR \" + response.text})\n",
    "        \n",
    "        # Randomly sample a caption length, and sample indices with that length.\n",
    "        indices = data_loader.dataset.get_train_indices()\n",
    "        # Create and assign a batch sampler to retrieve a batch with the sampled indices.\n",
    "        new_sampler = data.sampler.SubsetRandomSampler(indices=indices)\n",
    "        data_loader.batch_sampler.sampler = new_sampler\n",
    "        \n",
    "        # Obtain the batch.\n",
    "        images, captions = next(iter(data_loader))\n",
    "\n",
    "        # Move batch of images and captions to GPU if CUDA is available.\n",
    "        images = images.to(device)\n",
    "        captions = captions.to(device)\n",
    "        \n",
    "        # Zero the gradients.\n",
    "        decoder.zero_grad()\n",
    "        encoder.zero_grad()\n",
    "        \n",
    "        # Pass the inputs through the CNN-RNN model.\n",
    "        features = encoder(images)\n",
    "        outputs = decoder(features, captions)\n",
    "        \n",
    "        # Calculate the batch loss.\n",
    "        loss = criterion(outputs.view(-1, vocab_size), captions.view(-1))\n",
    "        \n",
    "        # Backward pass.\n",
    "        loss.backward()\n",
    "        \n",
    "        # Update the parameters in the optimizer.\n",
    "        optimizer.step()\n",
    "            \n",
    "        # Get training statistics.\n",
    "        stats = 'Epoch [%d/%d], Step [%d/%d], Loss: %.4f, Perplexity: %5.4f' % (epoch, num_epochs, i_step, total_step, loss.item(), np.exp(loss.item()))\n",
    "        \n",
    "        # Print training statistics (on same line).\n",
    "        print('\\r' + stats, end=\"\")\n",
    "        sys.stdout.flush()\n",
    "        \n",
    "        # Print training statistics to file.\n",
    "        f.write(stats + '\\n')\n",
    "        f.flush()\n",
    "        \n",
    "        # Print training statistics (on different line).\n",
    "        if i_step % print_every == 0:\n",
    "            print('\\r' + stats)\n",
    "            \n",
    "    # Save the weights.\n",
    "    if epoch % save_every == 0:\n",
    "        torch.save(decoder.state_dict(), os.path.join('./models', 'decoder-%d.pkl' % epoch))\n",
    "        torch.save(encoder.state_dict(), os.path.join('./models', 'encoder-%d.pkl' % epoch))\n",
    "\n",
    "# Close the training log file.\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='step3'></a>\n",
    "## Step 3: (Optional) Validate your Model\n",
    "\n",
    "To assess potential overfitting, one approach is to assess performance on a validation set.  If you decide to do this **optional** task, you are required to first complete all of the steps in the next notebook in the sequence (**3_Inference.ipynb**); as part of that notebook, you will write and test code (specifically, the `sample` method in the `DecoderRNN` class) that uses your RNN decoder to generate captions.  That code will prove incredibly useful here. \n",
    "\n",
    "If you decide to validate your model, please do not edit the data loader in **data_loader.py**.  Instead, create a new file named **data_loader_val.py** containing the code for obtaining the data loader for the validation data.  You can access:\n",
    "- the validation images at filepath `'/opt/cocoapi/images/train2014/'`, and\n",
    "- the validation image caption annotation file at filepath `'/opt/cocoapi/annotations/captions_val2014.json'`.\n",
    "\n",
    "The suggested approach to validating your model involves creating a json file such as [this one](https://github.com/cocodataset/cocoapi/blob/master/results/captions_val2014_fakecap_results.json) containing your model's predicted captions for the validation images.  Then, you can write your own script or use one that you [find online](https://github.com/tylin/coco-caption) to calculate the BLEU score of your model.  You can read more about the BLEU score, along with other evaluation metrics (such as TEOR and Cider) in section 4.1 of [this paper](https://arxiv.org/pdf/1411.4555.pdf).  For more information about how to use the annotation file, check out the [website](http://cocodataset.org/#download) for the COCO dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# (Optional) TODO: Validate your model."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}