Complete.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "# Sentiment Analysis with Deep Learning using BERT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "true"
   },
   "source": [
    "### Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "- Intermediate-level knowledge of Python 3 (NumPy and Pandas preferably, but not required)\n",
    "- Exposure to PyTorch usage\n",
    "- Basic understanding of Deep Learning and Language Models (BERT specifically)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### Project Outline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "true"
   },
   "source": [
    "**Task 1**: Introduction\n",
    "\n",
    "**Task 2**: Exploratory Data Analysis and Preprocessing\n",
    "\n",
    "**Task 3**: Training/Validation Split\n",
    "\n",
    "**Task 4**: Loading Tokenizer and Encoding our Data\n",
    "\n",
    "**Task 5**: Setting up BERT Pretrained Model\n",
    "\n",
    "**Task 6**: Creating Data Loaders\n",
    "\n",
    "**Task 7**: Setting Up Optimizer and Scheduler\n",
    "\n",
    "**Task 8**: Defining our Performance Metrics\n",
    "\n",
    "**Task 9**: Creating our Training Loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 1. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "true"
   },
   "source": [
    "### What is BERT\n",
    "\n",
    "BERT is a large-scale transformer-based Language Model that can be finetuned for a variety of tasks.\n",
    "\n",
    "For more information, check the following articles. \n",
    "\n",
    "[HuggingFace documentation](https://huggingface.co/transformers/model_doc/bert.html)\n",
    "\n",
    "[Bert documentation](https://characters.fandom.com/wiki/Bert_(Sesame_Street))"
   ]
  },
  
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 2. Exploratory Data Analysis and Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "We will use the SMILE Twitter dataset.\n",
    "\n",
    "_Wang, Bo; Tsakalidis, Adam; Liakata, Maria; Zubiaga, Arkaitz; Procter, Rob; Jensen, Eric (2016): SMILE Twitter Emotion dataset. figshare. Dataset. https://doi.org/10.6084/m9.figshare.3187909.v2_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import pandas as pd\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('Data/smile-annotations-final.csv', names=['id', 'text', 'category'])\n",
    "df.set_index('id', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>611857364396965889</th>\n",
       "      <td>@aandraous @britishmuseum @AndrewsAntonio Merc...</td>\n",
       "      <td>nocode</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>614484565059596288</th>\n",
       "      <td>Dorian Gray with Rainbow Scarf #LoveWins (from...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>614746522043973632</th>\n",
       "      <td>@SelectShowcase @Tate_StIves ... Replace with ...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>614877582664835073</th>\n",
       "      <td>@Sofabsports thank you for following me back. ...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>611932373039644672</th>\n",
       "      <td>@britishmuseum @TudorHistory What a beautiful ...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                 text category\n",
       "id                                                                            \n",
       "611857364396965889  @aandraous @britishmuseum @AndrewsAntonio Merc...   nocode\n",
       "614484565059596288  Dorian Gray with Rainbow Scarf #LoveWins (from...    happy\n",
       "614746522043973632  @SelectShowcase @Tate_StIves ... Replace with ...    happy\n",
       "614877582664835073  @Sofabsports thank you for following me back. ...    happy\n",
       "611932373039644672  @britishmuseum @TudorHistory What a beautiful ...    happy"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nocode               1572\n",
       "happy                1137\n",
       "not-relevant          214\n",
       "angry                  57\n",
       "surprise               35\n",
       "sad                    32\n",
       "happy|surprise         11\n",
       "happy|sad               9\n",
       "disgust|angry           7\n",
       "disgust                 6\n",
       "sad|disgust             2\n",
       "sad|angry               2\n",
       "sad|disgust|angry       1\n",
       "Name: category, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.category.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "df = df[~df.category.str.contains('\\|')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "df = df[df.category != 'nocode']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "happy           1137\n",
       "not-relevant     214\n",
       "angry             57\n",
       "surprise          35\n",
       "sad               32\n",
       "disgust            6\n",
       "Name: category, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.category.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "possible_labels = df.category.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "label_dict = {}\n",
    "for index, possible_label in enumerate(possible_labels):\n",
    "    label_dict[possible_label] = index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "df['label'] = df.category.replace(label_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>614484565059596288</th>\n",
       "      <td>Dorian Gray with Rainbow Scarf #LoveWins (from...</td>\n",
       "      <td>happy</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>614746522043973632</th>\n",
       "      <td>@SelectShowcase @Tate_StIves ... Replace with ...</td>\n",
       "      <td>happy</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>614877582664835073</th>\n",
       "      <td>@Sofabsports thank you for following me back. ...</td>\n",
       "      <td>happy</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>611932373039644672</th>\n",
       "      <td>@britishmuseum @TudorHistory What a beautiful ...</td>\n",
       "      <td>happy</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>611570404268883969</th>\n",
       "      <td>@NationalGallery @ThePoldarkian I have always ...</td>\n",
       "      <td>happy</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                 text  \\\n",
       "id                                                                      \n",
       "614484565059596288  Dorian Gray with Rainbow Scarf #LoveWins (from...   \n",
       "614746522043973632  @SelectShowcase @Tate_StIves ... Replace with ...   \n",
       "614877582664835073  @Sofabsports thank you for following me back. ...   \n",
       "611932373039644672  @britishmuseum @TudorHistory What a beautiful ...   \n",
       "611570404268883969  @NationalGallery @ThePoldarkian I have always ...   \n",
       "\n",
       "                   category  label  \n",
       "id                                  \n",
       "614484565059596288    happy      0  \n",
       "614746522043973632    happy      0  \n",
       "614877582664835073    happy      0  \n",
       "611932373039644672    happy      0  \n",
       "611570404268883969    happy      0  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## Training/Validation Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "X_train, X_val, y_train, y_val = train_test_split(df.index.values, \n",
    "                                                  df.label.values, \n",
    "                                                  test_size=0.15, \n",
    "                                                  random_state=17, \n",
    "                                                  stratify=df.label.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "df['data_type'] = ['not_set']*df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "df.loc[X_train, 'data_type'] = 'train'\n",
    "df.loc[X_val, 'data_type'] = 'val'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category</th>\n",
       "      <th>label</th>\n",
       "      <th>data_type</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">angry</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>train</th>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>val</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">disgust</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>train</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>val</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">happy</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">0</th>\n",
       "      <th>train</th>\n",
       "      <td>966</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>val</th>\n",
       "      <td>171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">not-relevant</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
       "      <th>train</th>\n",
       "      <td>182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>val</th>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">sad</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">4</th>\n",
       "      <th>train</th>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>val</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">surprise</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">5</th>\n",
       "      <th>train</th>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>val</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              text\n",
       "category     label data_type      \n",
       "angry        2     train        48\n",
       "                   val           9\n",
       "disgust      3     train         5\n",
       "                   val           1\n",
       "happy        0     train       966\n",
       "                   val         171\n",
       "not-relevant 1     train       182\n",
       "                   val          32\n",
       "sad          4     train        27\n",
       "                   val           5\n",
       "surprise     5     train        30\n",
       "                   val           5"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(['category', 'label', 'data_type']).count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## Loading Tokenizer and Encoding our Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer\n",
    "from torch.utils.data import TensorDataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', \n",
    "                                          do_lower_case=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "encoded_data_train = tokenizer.batch_encode_plus(\n",
    "    df[df.data_type=='train'].text.values, \n",
    "    add_special_tokens=True, \n",
    "    return_attention_mask=True, \n",
    "    pad_to_max_length=True, \n",
    "    max_length=256, \n",
    "    return_tensors='pt'\n",
    ")\n",
    "\n",
    "encoded_data_val = tokenizer.batch_encode_plus(\n",
    "    df[df.data_type=='val'].text.values, \n",
    "    add_special_tokens=True, \n",
    "    return_attention_mask=True, \n",
    "    pad_to_max_length=True, \n",
    "    max_length=256, \n",
    "    return_tensors='pt'\n",
    ")\n",
    "\n",
    "\n",
    "input_ids_train = encoded_data_train['input_ids']\n",
    "attention_masks_train = encoded_data_train['attention_mask']\n",
    "labels_train = torch.tensor(df[df.data_type=='train'].label.values)\n",
    "\n",
    "input_ids_val = encoded_data_val['input_ids']\n",
    "attention_masks_val = encoded_data_val['attention_mask']\n",
    "labels_val = torch.tensor(df[df.data_type=='val'].label.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "dataset_train = TensorDataset(input_ids_train, attention_masks_train, labels_train)\n",
    "dataset_val = TensorDataset(input_ids_val, attention_masks_val, labels_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1258"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dataset_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "223"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dataset_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## Setting up BERT Pretrained Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "from transformers import BertForSequenceClassification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "model = BertForSequenceClassification.from_pretrained(\"bert-base-uncased\",\n",
    "                                                      num_labels=len(label_dict),\n",
    "                                                      output_attentions=False,\n",
    "                                                      output_hidden_states=False)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## Creating Data Loaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader, RandomSampler, SequentialSampler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "\n",
    "dataloader_train = DataLoader(dataset_train, \n",
    "                              sampler=RandomSampler(dataset_train), \n",
    "                              batch_size=batch_size)\n",
    "\n",
    "dataloader_validation = DataLoader(dataset_val, \n",
    "                                   sampler=SequentialSampler(dataset_val), \n",
    "                                   batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## Setting Up Optimiser and Scheduler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "from transformers import AdamW, get_linear_schedule_with_warmup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "optimizer = AdamW(model.parameters(),\n",
    "                  lr=1e-5, \n",
    "                  eps=1e-8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "epochs = 3\n",
    "\n",
    "scheduler = get_linear_schedule_with_warmup(optimizer, \n",
    "                                            num_warmup_steps=0,\n",
    "                                            num_training_steps=len(dataloader_train)*epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## Defining our Performance Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "Accuracy metric approach originally used in accuracy function in [this tutorial](https://mccormickml.com/2019/07/22/BERT-fine-tuning/#41-bertforsequenceclassification)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "def f1_score_func(preds, labels):\n",
    "    preds_flat = np.argmax(preds, axis=1).flatten()\n",
    "    labels_flat = labels.flatten()\n",
    "    return f1_score(labels_flat, preds_flat, average='weighted')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "def accuracy_per_class(preds, labels):\n",
    "    label_dict_inverse = {v: k for k, v in label_dict.items()}\n",
    "    \n",
    "    preds_flat = np.argmax(preds, axis=1).flatten()\n",
    "    labels_flat = labels.flatten()\n",
    "\n",
    "    for label in np.unique(labels_flat):\n",
    "        y_preds = preds_flat[labels_flat==label]\n",
    "        y_true = labels_flat[labels_flat==label]\n",
    "        print(f'Class: {label_dict_inverse[label]}')\n",
    "        print(f'Accuracy: {len(y_preds[y_preds==label])}/{len(y_true)}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## Creating our Training Loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "Approach adapted from an older version of HuggingFace's `run_glue.py` script. Accessible [here](https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L128)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "seed_val = 17\n",
    "random.seed(seed_val)\n",
    "np.random.seed(seed_val)\n",
    "torch.manual_seed(seed_val)\n",
    "torch.cuda.manual_seed_all(seed_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpu\n"
     ]
    }
   ],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "model.to(device)\n",
    "\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "def evaluate(dataloader_val):\n",
    "\n",
    "    model.eval()\n",
    "    \n",
    "    loss_val_total = 0\n",
    "    predictions, true_vals = [], []\n",
    "    \n",
    "    for batch in dataloader_val:\n",
    "        \n",
    "        batch = tuple(b.to(device) for b in batch)\n",
    "        \n",
    "        inputs = {'input_ids':      batch[0],\n",
    "                  'attention_mask': batch[1],\n",
    "                  'labels':         batch[2],\n",
    "                 }\n",
    "\n",
    "        with torch.no_grad():        \n",
    "            outputs = model(**inputs)\n",
    "            \n",
    "        loss = outputs[0]\n",
    "        logits = outputs[1]\n",
    "        loss_val_total += loss.item()\n",
    "\n",
    "        logits = logits.detach().cpu().numpy()\n",
    "        label_ids = inputs['labels'].cpu().numpy()\n",
    "        predictions.append(logits)\n",
    "        true_vals.append(label_ids)\n",
    "    \n",
    "    loss_val_avg = loss_val_total/len(dataloader_val) \n",
    "    \n",
    "    predictions = np.concatenate(predictions, axis=0)\n",
    "    true_vals = np.concatenate(true_vals, axis=0)\n",
    "            \n",
    "    return loss_val_avg, predictions, true_vals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e84d25a6fefe4df18d98d3870f42a2a9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=3.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Epoch 1', max=40.0, style=ProgressStyle(description_width…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch 1\n",
      "Training loss: 0.7238405920565129\n",
      "Validation loss: 0.6993163057735988\n",
      "F1 Score (Weighted): 0.7407933426764294\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c62986114bfa462e9bb7b455f11753f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Epoch 2', max=40.0, style=ProgressStyle(description_width…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-146-9306225bb55a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     21\u001b[0m         \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m         \u001b[0mloss_train_total\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 23\u001b[0;31m         \u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     24\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     25\u001b[0m         \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclip_grad_norm_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1.0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/basic/lib/python3.7/site-packages/torch/tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[1;32m    193\u001b[0m                 \u001b[0mproducts\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    194\u001b[0m         \"\"\"\n\u001b[0;32m--> 195\u001b[0;31m         \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    196\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    197\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/basic/lib/python3.7/site-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[1;32m     97\u001b[0m     Variable._execution_engine.run_backward(\n\u001b[1;32m     98\u001b[0m         \u001b[0mtensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 99\u001b[0;31m         allow_unreachable=True)  # allow_unreachable flag\n\u001b[0m\u001b[1;32m    100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "for epoch in tqdm(range(1, epochs+1)):\n",
    "    \n",
    "    model.train()\n",
    "    \n",
    "    loss_train_total = 0\n",
    "\n",
    "    progress_bar = tqdm(dataloader_train, desc='Epoch {:1d}'.format(epoch), leave=False, disable=False)\n",
    "    for batch in progress_bar:\n",
    "\n",
    "        model.zero_grad()\n",
    "        \n",
    "        batch = tuple(b.to(device) for b in batch)\n",
    "        \n",
    "        inputs = {'input_ids':      batch[0],\n",
    "                  'attention_mask': batch[1],\n",
    "                  'labels':         batch[2],\n",
    "                 }       \n",
    "\n",
    "        outputs = model(**inputs)\n",
    "        \n",
    "        loss = outputs[0]\n",
    "        loss_train_total += loss.item()\n",
    "        loss.backward()\n",
    "\n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
    "\n",
    "        optimizer.step()\n",
    "        scheduler.step()\n",
    "        \n",
    "        progress_bar.set_postfix({'training_loss': '{:.3f}'.format(loss.item()/len(batch))})\n",
    "         \n",
    "        \n",
    "    torch.save(model.state_dict(), f'finetuned_BERT_epoch_{epoch}.model')\n",
    "        \n",
    "    tqdm.write(f'\\nEpoch {epoch}')\n",
    "    \n",
    "    loss_train_avg = loss_train_total/len(dataloader_train)            \n",
    "    tqdm.write(f'Training loss: {loss_train_avg}')\n",
    "    \n",
    "    val_loss, predictions, true_vals = evaluate(dataloader_validation)\n",
    "    val_f1 = f1_score_func(predictions, true_vals)\n",
    "    tqdm.write(f'Validation loss: {val_loss}')\n",
    "    tqdm.write(f'F1 Score (Weighted): {val_f1}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "model = BertForSequenceClassification.from_pretrained(\"bert-base-uncased\",\n",
    "                                                      num_labels=len(label_dict),\n",
    "                                                      output_attentions=False,\n",
    "                                                      output_hidden_states=False)\n",
    "\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.load_state_dict(torch.load('Models/<<INSERT MODEL NAME HERE>>.model', map_location=torch.device('cpu')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "_, predictions, true_vals = evaluate(dataloader_validation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class: happy\n",
      "Accuracy: 163/171\n",
      "\n",
      "Class: not-relevant\n",
      "Accuracy: 20/32\n",
      "\n",
      "Class: angry\n",
      "Accuracy: 7/9\n",
      "\n",
      "Class: disgust\n",
      "Accuracy: 0/1\n",
      "\n",
      "Class: sad\n",
      "Accuracy: 4/5\n",
      "\n",
      "Class: surprise\n",
      "Accuracy: 2/5\n",
      "\n"
     ]
    }
   ],
   "source": [
    "accuracy_per_class(predictions, true_vals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}