t81_558_class_02_5_pandas_features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_02_5_pandas_features.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 2: Python for Machine Learning**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 2 Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* Part 2.1: Introduction to Pandas [[Video]](https://www.youtube.com/watch?v=bN4UuCBdpZc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_1_python_pandas.ipynb)\n",
    "* Part 2.2: Categorical Values [[Video]](https://www.youtube.com/watch?v=4a1odDpG0Ho&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_2_pandas_cat.ipynb)\n",
    "* Part 2.3: Grouping, Sorting, and Shuffling in Python Pandas [[Video]](https://www.youtube.com/watch?v=YS4wm5gD8DM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_3_pandas_grouping.ipynb)\n",
    "* Part 2.4: Using Apply and Map in Pandas for Keras [[Video]](https://www.youtube.com/watch?v=XNCEZ4WaPBY&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_4_pandas_functional.ipynb)\n",
    "* **Part 2.5: Feature Engineering in Pandas for Deep Learning in Keras** [[Video]](https://www.youtube.com/watch?v=BWPTj4_Mi9E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_5_pandas_features.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2.5: Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature engineering is a very important part of machine learning.  Later in this course we will see some techniques for automatic feature engineering.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculated Fields\n",
    "\n",
    "It is possible to add new fields to the dataframe that are calculated from the other fields.  We can create a new column that gives the weight in kilograms.  The equation to calculate a metric weight, given a weight in pounds is:\n",
    "\n",
    "$ m_{(kg)} = m_{(lb)} \\times 0.45359237 $\n",
    "\n",
    "This can be used with the following Python code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>weight_kg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18.0</td>\n",
       "      <td>1589</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>3504</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1675</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165.0</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18.0</td>\n",
       "      <td>1558</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth satellite</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>1557</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc rebel sst</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17.0</td>\n",
       "      <td>1564</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>3449</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford torino</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>27.0</td>\n",
       "      <td>1265</td>\n",
       "      <td>4</td>\n",
       "      <td>140.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>2790</td>\n",
       "      <td>15.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford mustang gl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>394</th>\n",
       "      <td>44.0</td>\n",
       "      <td>966</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>24.6</td>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "      <td>vw pickup</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>395</th>\n",
       "      <td>32.0</td>\n",
       "      <td>1040</td>\n",
       "      <td>4</td>\n",
       "      <td>135.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>2295</td>\n",
       "      <td>11.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge rampage</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>396</th>\n",
       "      <td>28.0</td>\n",
       "      <td>1190</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>2625</td>\n",
       "      <td>18.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford ranger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>397</th>\n",
       "      <td>31.0</td>\n",
       "      <td>1233</td>\n",
       "      <td>4</td>\n",
       "      <td>119.0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>2720</td>\n",
       "      <td>19.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevy s-10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>398 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      mpg  weight_kg  cylinders  displacement  horsepower  weight  \\\n",
       "0    18.0       1589          8         307.0       130.0    3504   \n",
       "1    15.0       1675          8         350.0       165.0    3693   \n",
       "2    18.0       1558          8         318.0       150.0    3436   \n",
       "3    16.0       1557          8         304.0       150.0    3433   \n",
       "4    17.0       1564          8         302.0       140.0    3449   \n",
       "..    ...        ...        ...           ...         ...     ...   \n",
       "393  27.0       1265          4         140.0        86.0    2790   \n",
       "394  44.0        966          4          97.0        52.0    2130   \n",
       "395  32.0       1040          4         135.0        84.0    2295   \n",
       "396  28.0       1190          4         120.0        79.0    2625   \n",
       "397  31.0       1233          4         119.0        82.0    2720   \n",
       "\n",
       "     acceleration  year  origin                       name  \n",
       "0            12.0    70       1  chevrolet chevelle malibu  \n",
       "1            11.5    70       1          buick skylark 320  \n",
       "2            11.0    70       1         plymouth satellite  \n",
       "3            12.0    70       1              amc rebel sst  \n",
       "4            10.5    70       1                ford torino  \n",
       "..            ...   ...     ...                        ...  \n",
       "393          15.6    82       1            ford mustang gl  \n",
       "394          24.6    82       2                  vw pickup  \n",
       "395          11.6    82       1              dodge rampage  \n",
       "396          18.6    82       1                ford ranger  \n",
       "397          19.4    82       1                 chevy s-10  \n",
       "\n",
       "[398 rows x 10 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "df.insert(1, 'weight_kg', (df['weight'] * 0.45359237).astype(int))\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Google API Keys\n",
    "\n",
    "Sometimes you will use external API's to obtain data.  The following examples show how to use the Google API keys to encode addresses for use with neural networks.  To use these, you will need your own Google API key.  The key I have below is not a real key, you need to put your own in there.  Google will ask for a credit card, but unless you use a very large number of lookups, there will be no actual cost.  YOU ARE NOT required to get an Google API key for this class, this only shows you how.  If you would like to get a Google API key, visit this site and obtain one for **geocode**.\n",
    "\n",
    "[Google API Keys](https://developers.google.com/maps/documentation/embed/get-api-key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#GOOGLE_KEY = 'Your Google API Key'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other Examples: Dealing with Addresses\n",
    "\n",
    "Addresses can be difficult to encode into a neural network.  There are many different approaches, and you must consider how you can transform the address into something more meaningful.  Map coordinates can be a good approach.  [Latitude and longitude](https://en.wikipedia.org/wiki/Geographic_coordinate_system) can be a useful encoding.  Thanks to the power of the Internet, it is relatively easy to transform an address into its latitude and longitude values.  The following code determines the coordinates of [Washington University](https://wustl.edu/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'lat': 38.648238, 'lng': -90.30487459999999}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "address = \"1 Brookings Dr, St. Louis, MO 63130\"\n",
    "\n",
    "response = requests.get('https://maps.googleapis.com/maps/api/geocode/json?key={}&address={}'.format(GOOGLE_KEY,address))\n",
    "\n",
    "resp_json_payload = response.json()\n",
    "\n",
    "if 'error_message' in resp_json_payload:\n",
    "    print(resp_json_payload['error_message'])\n",
    "else:\n",
    "    print(resp_json_payload['results'][0]['geometry']['location'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If latitude and longitude are simply fed into the neural network as two features, they might not be overly helpful.  These two values would allow your neural network to cluster locations on a map.  Sometimes cluster locations on a map can be useful.  Consider the percentage of the population that smokes in the USA by state:\n",
    "\n",
    "![Smokers by State](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_6_smokers.png \"Smokers by State\")\n",
    "\n",
    "The above map shows that certain behaviors, like smoking, can be clustered by global region. \n",
    "\n",
    "However, often you will want to transform the coordinates into distances.  It is reasonably easy to estimate the distance between any two points on Earth by using the [great circle distance](https://en.wikipedia.org/wiki/Great-circle_distance) between any two points on a sphere:\n",
    "\n",
    "The following code implements this formula:\n",
    "\n",
    "$\\Delta\\sigma=\\arccos\\bigl(\\sin\\phi_1\\cdot\\sin\\phi_2+\\cos\\phi_1\\cdot\\cos\\phi_2\\cdot\\cos(\\Delta\\lambda)\\bigr)$\n",
    "\n",
    "$d = r \\, \\Delta\\sigma$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distance, St. Louis, MO to Ft. Lauderdale, FL: 1685.304481666787 km\n"
     ]
    }
   ],
   "source": [
    "from math import sin, cos, sqrt, atan2, radians\n",
    "\n",
    "# Distance function\n",
    "def distance_lat_lng(lat1,lng1,lat2,lng2):\n",
    "    # approximate radius of earth in km\n",
    "    R = 6373.0\n",
    "\n",
    "    # degrees to radians (lat/lon are in degrees)\n",
    "    lat1 = radians(lat1)\n",
    "    lng1 = radians(lng1)\n",
    "    lat2 = radians(lat2)\n",
    "    lng2 = radians(lng2)\n",
    "\n",
    "    dlng = lng2 - lng1\n",
    "    dlat = lat2 - lat1\n",
    "\n",
    "    a = sin(dlat / 2)**2 + cos(lat1) * cos(lat2) * sin(dlng / 2)**2\n",
    "    c = 2 * atan2(sqrt(a), sqrt(1 - a))\n",
    "\n",
    "    return R * c\n",
    "\n",
    "# Find lat lon for address\n",
    "def lookup_lat_lng(address):\n",
    "    response = requests.get('https://maps.googleapis.com/maps/api/geocode/json?key={}&address={}'.format(GOOGLE_KEY,address))\n",
    "    json = response.json()\n",
    "    if len(json['results']) == 0:\n",
    "        print(\"Can't find: {}\".format(address))\n",
    "        return 0,0\n",
    "    map = json['results'][0]['geometry']['location']\n",
    "    return map['lat'],map['lng']\n",
    "\n",
    "\n",
    "# Distance between two locations\n",
    "\n",
    "import requests\n",
    "\n",
    "address1 = \"1 Brookings Dr, St. Louis, MO 63130\" \n",
    "address2 = \"3301 College Ave, Fort Lauderdale, FL 33314\"\n",
    "\n",
    "lat1, lng1 = lookup_lat_lng(address1)\n",
    "lat2, lng2 = lookup_lat_lng(address2)\n",
    "\n",
    "print(\"Distance, St. Louis, MO to Ft. Lauderdale, FL: {} km\".format(\n",
    "        distance_lat_lng(lat1,lng1,lat2,lng2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distances can be useful to encode addresses as.  You must consider what distance might be useful for your dataset.  Consider:\n",
    "\n",
    "* Distance to major metropolitan area\n",
    "* Distance to competitor\n",
    "* Distance to distribution center\n",
    "* Distance to retail outlet\n",
    "\n",
    "The following code calculates the distance between 10 universities and washu:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "School 'Princeton', distance to wustl is: 1354.4748428037537\n",
      "School 'Harvard', distance to wustl is: 1670.6348910867227\n",
      "School 'University of Chicago', distance to wustl is: 418.07123096093096\n",
      "School 'Yale', distance to wustl is: 1508.209168740192\n",
      "School 'Columbia University', distance to wustl is: 1418.2512902029155\n",
      "School 'Stanford', distance to wustl is: 2780.7830466634337\n",
      "School 'MIT', distance to wustl is: 1672.4354422735219\n",
      "School 'Duke University', distance to wustl is: 1046.7924543575177\n",
      "School 'University of Pennsylvania', distance to wustl is: 1307.1873732319766\n",
      "School 'Johns Hopkins', distance to wustl is: 1184.3754484499111\n"
     ]
    }
   ],
   "source": [
    "# Encoding other universities by their distance to Washington University\n",
    "\n",
    "schools = [\n",
    "    [\"Princeton University, Princeton, NJ 08544\", 'Princeton'],\n",
    "    [\"Massachusetts Hall, Cambridge, MA 02138\", 'Harvard'],\n",
    "    [\"5801 S Ellis Ave, Chicago, IL 60637\", 'University of Chicago'],\n",
    "    [\"Yale, New Haven, CT 06520\", 'Yale'],\n",
    "    [\"116th St & Broadway, New York, NY 10027\", 'Columbia University'],\n",
    "    [\"450 Serra Mall, Stanford, CA 94305\", 'Stanford'],\n",
    "    [\"77 Massachusetts Ave, Cambridge, MA 02139\", 'MIT'],\n",
    "    [\"Duke University, Durham, NC 27708\", 'Duke University'],\n",
    "    [\"University of Pennsylvania, Philadelphia, PA 19104\", 'University of Pennsylvania'],\n",
    "    [\"Johns Hopkins University, Baltimore, MD 21218\", 'Johns Hopkins']\n",
    "]\n",
    "\n",
    "lat1, lng1 = lookup_lat_lng(\"1 Brookings Dr, St. Louis, MO 63130\")\n",
    "\n",
    "for address, name in schools:\n",
    "    lat2,lng2 = lookup_lat_lng(address)\n",
    "    dist = distance_lat_lng(lat1,lng1,lat2,lng2)\n",
    "    print(\"School '{}', distance to wustl is: {}\".format(name,dist))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}