3-Sentiment-Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, all of the analysis we've done has been pretty generic - looking at counts, creating scatter plots, etc. These techniques could be applied to numeric data as well.\n",
    "\n",
    "When it comes to text data, there are a few popular techniques that we'll be going through in the next few notebooks, starting with sentiment analysis. A few key points to remember with sentiment analysis.\n",
    "\n",
    "1. **TextBlob Module:** Linguistic researchers have labeled the sentiment of words based on their domain expertise. Sentiment of words can vary based on where it is in a sentence. The TextBlob module allows us to take advantage of these labels.\n",
    "2. **Sentiment Labels:** Each word in a corpus is labeled in terms of polarity and subjectivity (there are more labels as well, but we're going to ignore them for now). A corpus' sentiment is the average of these.\n",
    "   * **Polarity**: How positive or negative a word is. -1 is very negative. +1 is very positive.\n",
    "   * **Subjectivity**: How subjective, or opinionated a word is. 0 is fact. +1 is very much an opinion.\n",
    "\n",
    "For more info on how TextBlob coded up its [sentiment function](https://planspace.org/20150607-textblob_sentiment/).\n",
    "\n",
    "Let's take a look at the sentiment of the various transcripts, both overall and throughout the comedy routine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment of Routine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll start by reading in the corpus, which preserves word order\n",
    "import pandas as pd\n",
    "\n",
    "data = pd.read_pickle('corpus.pkl')\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create quick lambda functions to find the polarity and subjectivity of each routine\n",
    "# Terminal / Anaconda Navigator: conda install -c conda-forge textblob\n",
    "from textblob import TextBlob\n",
    "\n",
    "pol = lambda x: TextBlob(x).sentiment.polarity\n",
    "sub = lambda x: TextBlob(x).sentiment.subjectivity\n",
    "\n",
    "data['polarity'] = data['transcript'].apply(pol)\n",
    "data['subjectivity'] = data['transcript'].apply(sub)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's plot the results\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.rcParams['figure.figsize'] = [10, 8]\n",
    "\n",
    "for index, comedian in enumerate(data.index):\n",
    "    x = data.polarity.loc[comedian]\n",
    "    y = data.subjectivity.loc[comedian]\n",
    "    plt.scatter(x, y, color='blue')\n",
    "    plt.text(x+.001, y+.001, data['full_name'][index], fontsize=10)\n",
    "    plt.xlim(-.01, .12) \n",
    "    \n",
    "plt.title('Sentiment Analysis', fontsize=20)\n",
    "plt.xlabel('<-- Negative -------- Positive -->', fontsize=15)\n",
    "plt.ylabel('<-- Facts -------- Opinions -->', fontsize=15)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment of Routine Over Time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of looking at the overall sentiment, let's see if there's anything interesting about the sentiment over time throughout each routine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Split each routine into 10 parts\n",
    "import numpy as np\n",
    "import math\n",
    "\n",
    "def split_text(text, n=10):\n",
    "    '''Takes in a string of text and splits into n equal parts, with a default of 10 equal parts.'''\n",
    "\n",
    "    # Calculate length of text, the size of each chunk of text and the starting points of each chunk of text\n",
    "    length = len(text)\n",
    "    size = math.floor(length / n)\n",
    "    start = np.arange(0, length, size)\n",
    "    \n",
    "    # Pull out equally sized pieces of text and put it into a list\n",
    "    split_list = []\n",
    "    for piece in range(n):\n",
    "        split_list.append(text[start[piece]:start[piece]+size])\n",
    "    return split_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's take a look at our data again\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create a list to hold all of the pieces of text\n",
    "list_pieces = []\n",
    "for t in data.transcript:\n",
    "    split = split_text(t)\n",
    "    list_pieces.append(split)\n",
    "    \n",
    "list_pieces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The list has 10 elements, one for each transcript\n",
    "len(list_pieces)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Each transcript has been split into 10 pieces of text\n",
    "len(list_pieces[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the polarity for each piece of text\n",
    "\n",
    "polarity_transcript = []\n",
    "for lp in list_pieces:\n",
    "    polarity_piece = []\n",
    "    for p in lp:\n",
    "        polarity_piece.append(TextBlob(p).sentiment.polarity)\n",
    "    polarity_transcript.append(polarity_piece)\n",
    "    \n",
    "polarity_transcript"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Show the plot for one comedian\n",
    "plt.plot(polarity_transcript[0])\n",
    "plt.title(data['full_name'].index[0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the plot for all comedians\n",
    "plt.rcParams['figure.figsize'] = [16, 12]\n",
    "\n",
    "for index, comedian in enumerate(data.index):    \n",
    "    plt.subplot(3, 4, index+1)\n",
    "    plt.plot(polarity_transcript[index])\n",
    "    plt.plot(np.arange(0,10), np.zeros(10))\n",
    "    plt.title(data['full_name'][index])\n",
    "    plt.ylim(ymin=-.2, ymax=.3)\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Ali Wong stays generally positive throughout her routine. Similar comedians are Louis C.K. and Mike Birbiglia.\n",
    "\n",
    "On the other hand, you have some pretty different patterns here like Bo Burnham who gets happier as time passes and Dave Chappelle who has some pretty down moments in his routine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Additional Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Modify the number of sections the comedy routine is split into and see how the charts over time change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}