09.2_Engineering_rare_values_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Engineering Rare Labels / Rare Categories\n",
    "\n",
    "Rare values are labels/ categories within a categorical variable that are only present for a small percentage of the observations.\n",
    "\n",
    "There is no rule of thumb to determine how small is a small percentage, but typically, any value below 5% may cause over-fitting in trees.\n",
    "\n",
    "Rare labels may exist in variables that have intrinsically a huge number of labels, or they can be present in variables with few labels (e.g., 2-10). There is no rule of thumb to determine how many different labels is big (and therefore represent high cardinality) and it will depend as well on how many observations there are in the dataset. In a dataset with 1,000 observations, 100 labels may seem a lot, whereas in a dataset with 100,000 observations it may not be so high.\n",
    "\n",
    "In situations where rare labels are present in variables with only a few categories, the rare label may be adding some information. On the other hand, in variables with a high number of categories, likely there will be very many labels with a low frequency, which will quite likely add noise instead of information. \n",
    "\n",
    "Whether rare labels should be processed before training a machine learning algorithm will depend on the dataset and problem at hand. Ideally, if there are not too many variables, you could try and explore the variables and their categories one at a time, and determine whether the rare labels add noise or information.\n",
    "\n",
    "If, on the other hand, the dataset has very many categorical variables, and exploring one at a time is not an option you may choose to sacrifice the ideal / optimal performance for a higher delivery speed.\n",
    "\n",
    "### Engineering rare labels\n",
    "\n",
    "There are multiple ways of accounting for rare labels. Some of them handle rare labels at the same time of converting labels into numbers. I will explain those in the section \"Engineer labels of categorical variables\".\n",
    "\n",
    "In this section of the course, I will expand on how to handle rare labels by re-categorising the observation that show rare labels for a certain variable. These observations can be re-categorised by:\n",
    "\n",
    "- Replacing the rare label by most frequent label\n",
    "- Grouping the observations that show rare labels into a unique category (with a new label like 'Rare', or 'Other')\n",
    "\n",
    "In this and the comming lectures I will explain when it is convenient to use one or the other way of replacing rare values, and evaluate the consequences of replacing rare labels in variables with:\n",
    "\n",
    "- One predominant category\n",
    "- A small number of different categories\n",
    "- High cardinality\n",
    "\n",
    "**Note that grouping infrequent labels or categories under a new category called 'Rare' or 'Other' is the most common practice in machine learning for businesses.**\n",
    "\n",
    "Specifically, in this lecture I will demonstrate how to work with rare labels in variables with one predominant category using the House Sale dataset from Kaggle.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "===============================================================================\n",
    "\n",
    "## Real Life example: \n",
    "\n",
    "### Predicting Sale Price of Houses\n",
    "\n",
    "The problem at hand aims to predict the final sale price of homes based on different explanatory variables describing aspects of residential homes. Predicting house prices is useful to identify fruitful investments, or to determine whether the price advertised for a house is over or underestimated, before making a buying judgment.\n",
    "\n",
    "============================================================================="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "% matplotlib inline\n",
    "\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "#from sklearn.ensemble import GradientBoostingRegressor\n",
    "\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "pd.set_option('display.max_columns', None) # to display the total number columns present in the dataset\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## House Sale Price dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>LandSlope</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>Condition1</th>\n",
       "      <th>Condition2</th>\n",
       "      <th>BldgType</th>\n",
       "      <th>HouseStyle</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>RoofStyle</th>\n",
       "      <th>RoofMatl</th>\n",
       "      <th>Exterior1st</th>\n",
       "      <th>Exterior2nd</th>\n",
       "      <th>MasVnrType</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>ExterQual</th>\n",
       "      <th>ExterCond</th>\n",
       "      <th>Foundation</th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>BsmtCond</th>\n",
       "      <th>BsmtExposure</th>\n",
       "      <th>BsmtFinType1</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinType2</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>Heating</th>\n",
       "      <th>HeatingQC</th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>1stFlrSF</th>\n",
       "      <th>2ndFlrSF</th>\n",
       "      <th>LowQualFinSF</th>\n",
       "      <th>GrLivArea</th>\n",
       "      <th>BsmtFullBath</th>\n",
       "      <th>BsmtHalfBath</th>\n",
       "      <th>FullBath</th>\n",
       "      <th>HalfBath</th>\n",
       "      <th>BedroomAbvGr</th>\n",
       "      <th>KitchenAbvGr</th>\n",
       "      <th>KitchenQual</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>Functional</th>\n",
       "      <th>Fireplaces</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>GarageType</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <th>GarageFinish</th>\n",
       "      <th>GarageCars</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>GarageQual</th>\n",
       "      <th>GarageCond</th>\n",
       "      <th>PavedDrive</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2003</td>\n",
       "      <td>2003</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>196.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>No</td>\n",
       "      <td>GLQ</td>\n",
       "      <td>706</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>856</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>856</td>\n",
       "      <td>854</td>\n",
       "      <td>0</td>\n",
       "      <td>1710</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>8</td>\n",
       "      <td>Typ</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>RFn</td>\n",
       "      <td>2</td>\n",
       "      <td>548</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Veenker</td>\n",
       "      <td>Feedr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>1Story</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>1976</td>\n",
       "      <td>1976</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>MetalSd</td>\n",
       "      <td>MetalSd</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>CBlock</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>ALQ</td>\n",
       "      <td>978</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0</td>\n",
       "      <td>284</td>\n",
       "      <td>1262</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>1262</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1262</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>6</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>RFn</td>\n",
       "      <td>2</td>\n",
       "      <td>460</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>298</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2001</td>\n",
       "      <td>2002</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>162.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>Mn</td>\n",
       "      <td>GLQ</td>\n",
       "      <td>486</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0</td>\n",
       "      <td>434</td>\n",
       "      <td>920</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>920</td>\n",
       "      <td>866</td>\n",
       "      <td>0</td>\n",
       "      <td>1786</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>6</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>RFn</td>\n",
       "      <td>2</td>\n",
       "      <td>608</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Crawfor</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1915</td>\n",
       "      <td>1970</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Wd Sdng</td>\n",
       "      <td>Wd Shng</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>BrkTil</td>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>No</td>\n",
       "      <td>ALQ</td>\n",
       "      <td>216</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0</td>\n",
       "      <td>540</td>\n",
       "      <td>756</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>961</td>\n",
       "      <td>756</td>\n",
       "      <td>0</td>\n",
       "      <td>1717</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>7</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Detchd</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>Unf</td>\n",
       "      <td>3</td>\n",
       "      <td>642</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>272</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>NoRidge</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>2000</td>\n",
       "      <td>2000</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>350.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>Av</td>\n",
       "      <td>GLQ</td>\n",
       "      <td>655</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0</td>\n",
       "      <td>490</td>\n",
       "      <td>1145</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>1145</td>\n",
       "      <td>1053</td>\n",
       "      <td>0</td>\n",
       "      <td>2198</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>9</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>RFn</td>\n",
       "      <td>3</td>\n",
       "      <td>836</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>192</td>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0   1          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "1   2          20       RL         80.0     9600   Pave   NaN      Reg   \n",
       "2   3          60       RL         68.0    11250   Pave   NaN      IR1   \n",
       "3   4          70       RL         60.0     9550   Pave   NaN      IR1   \n",
       "4   5          60       RL         84.0    14260   Pave   NaN      IR1   \n",
       "\n",
       "  LandContour Utilities LotConfig LandSlope Neighborhood Condition1  \\\n",
       "0         Lvl    AllPub    Inside       Gtl      CollgCr       Norm   \n",
       "1         Lvl    AllPub       FR2       Gtl      Veenker      Feedr   \n",
       "2         Lvl    AllPub    Inside       Gtl      CollgCr       Norm   \n",
       "3         Lvl    AllPub    Corner       Gtl      Crawfor       Norm   \n",
       "4         Lvl    AllPub       FR2       Gtl      NoRidge       Norm   \n",
       "\n",
       "  Condition2 BldgType HouseStyle  OverallQual  OverallCond  YearBuilt  \\\n",
       "0       Norm     1Fam     2Story            7            5       2003   \n",
       "1       Norm     1Fam     1Story            6            8       1976   \n",
       "2       Norm     1Fam     2Story            7            5       2001   \n",
       "3       Norm     1Fam     2Story            7            5       1915   \n",
       "4       Norm     1Fam     2Story            8            5       2000   \n",
       "\n",
       "   YearRemodAdd RoofStyle RoofMatl Exterior1st Exterior2nd MasVnrType  \\\n",
       "0          2003     Gable  CompShg     VinylSd     VinylSd    BrkFace   \n",
       "1          1976     Gable  CompShg     MetalSd     MetalSd       None   \n",
       "2          2002     Gable  CompShg     VinylSd     VinylSd    BrkFace   \n",
       "3          1970     Gable  CompShg     Wd Sdng     Wd Shng       None   \n",
       "4          2000     Gable  CompShg     VinylSd     VinylSd    BrkFace   \n",
       "\n",
       "   MasVnrArea ExterQual ExterCond Foundation BsmtQual BsmtCond BsmtExposure  \\\n",
       "0       196.0        Gd        TA      PConc       Gd       TA           No   \n",
       "1         0.0        TA        TA     CBlock       Gd       TA           Gd   \n",
       "2       162.0        Gd        TA      PConc       Gd       TA           Mn   \n",
       "3         0.0        TA        TA     BrkTil       TA       Gd           No   \n",
       "4       350.0        Gd        TA      PConc       Gd       TA           Av   \n",
       "\n",
       "  BsmtFinType1  BsmtFinSF1 BsmtFinType2  BsmtFinSF2  BsmtUnfSF  TotalBsmtSF  \\\n",
       "0          GLQ         706          Unf           0        150          856   \n",
       "1          ALQ         978          Unf           0        284         1262   \n",
       "2          GLQ         486          Unf           0        434          920   \n",
       "3          ALQ         216          Unf           0        540          756   \n",
       "4          GLQ         655          Unf           0        490         1145   \n",
       "\n",
       "  Heating HeatingQC CentralAir Electrical  1stFlrSF  2ndFlrSF  LowQualFinSF  \\\n",
       "0    GasA        Ex          Y      SBrkr       856       854             0   \n",
       "1    GasA        Ex          Y      SBrkr      1262         0             0   \n",
       "2    GasA        Ex          Y      SBrkr       920       866             0   \n",
       "3    GasA        Gd          Y      SBrkr       961       756             0   \n",
       "4    GasA        Ex          Y      SBrkr      1145      1053             0   \n",
       "\n",
       "   GrLivArea  BsmtFullBath  BsmtHalfBath  FullBath  HalfBath  BedroomAbvGr  \\\n",
       "0       1710             1             0         2         1             3   \n",
       "1       1262             0             1         2         0             3   \n",
       "2       1786             1             0         2         1             3   \n",
       "3       1717             1             0         1         0             3   \n",
       "4       2198             1             0         2         1             4   \n",
       "\n",
       "   KitchenAbvGr KitchenQual  TotRmsAbvGrd Functional  Fireplaces FireplaceQu  \\\n",
       "0             1          Gd             8        Typ           0         NaN   \n",
       "1             1          TA             6        Typ           1          TA   \n",
       "2             1          Gd             6        Typ           1          TA   \n",
       "3             1          Gd             7        Typ           1          Gd   \n",
       "4             1          Gd             9        Typ           1          TA   \n",
       "\n",
       "  GarageType  GarageYrBlt GarageFinish  GarageCars  GarageArea GarageQual  \\\n",
       "0     Attchd       2003.0          RFn           2         548         TA   \n",
       "1     Attchd       1976.0          RFn           2         460         TA   \n",
       "2     Attchd       2001.0          RFn           2         608         TA   \n",
       "3     Detchd       1998.0          Unf           3         642         TA   \n",
       "4     Attchd       2000.0          RFn           3         836         TA   \n",
       "\n",
       "  GarageCond PavedDrive  WoodDeckSF  OpenPorchSF  EnclosedPorch  3SsnPorch  \\\n",
       "0         TA          Y           0           61              0          0   \n",
       "1         TA          Y         298            0              0          0   \n",
       "2         TA          Y           0           42              0          0   \n",
       "3         TA          Y           0           35            272          0   \n",
       "4         TA          Y         192           84              0          0   \n",
       "\n",
       "   ScreenPorch  PoolArea PoolQC Fence MiscFeature  MiscVal  MoSold  YrSold  \\\n",
       "0            0         0    NaN   NaN         NaN        0       2    2008   \n",
       "1            0         0    NaN   NaN         NaN        0       5    2007   \n",
       "2            0         0    NaN   NaN         NaN        0       9    2008   \n",
       "3            0         0    NaN   NaN         NaN        0       2    2006   \n",
       "4            0         0    NaN   NaN         NaN        0      12    2008   \n",
       "\n",
       "  SaleType SaleCondition  SalePrice  \n",
       "0       WD        Normal     208500  \n",
       "1       WD        Normal     181500  \n",
       "2       WD        Normal     223500  \n",
       "3       WD       Abnorml     140000  \n",
       "4       WD        Normal     250000  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the house price dataset from Kaggle\n",
    "\n",
    "data = pd.read_csv('houseprice.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rare value imputation - important\n",
    "\n",
    "The identification of rare labels should be done considering only the presence of rare labels in the training set, and then propagated to the test set. This means, rare labels should be identified in the training set. And then, when those are present in the test set as well, they should be replaced, regardless of whether in the test set they are rare or not (i.e., regardless of whether in the test set they are also present in a tiny percentage of the observations or in a high percentage of observations)\n",
    "\n",
    "In addition, there may be in the test set labels that were not present in the train set. They should be considered rare and preprocessed using the method that was selected to replace rare labels in the training set.\n",
    "\n",
    "For example, let's imagine that we have in the training set the variable 'city' with the labels 'London', 'Manchester' and 'Yorkshire'. 'Yorkshire' is present in less than 5% of the observations so we decide to replace it by 'London', the most frequent city in the training dataset. In the test set, we should also replace 'Yorkshire' by 'London', regardless of the percentage of observations for 'Yorkshire' or whether 'London' is still the most represented city in the test set.\n",
    "\n",
    "In addition, if in the test set we find the category 'Milton Keynes', that was not present in the training set, we should also replace that category by London. This is, all categories present in test set, not present in training set, should be treated as rare values and imputed accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1022, 81), (438, 81))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's go ahead and divide dataset into train and test set\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data, data.SalePrice, test_size=0.3,\n",
    "                                                    random_state=0)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions\n",
    "\n",
    "Below I will write a few functions to convert categories in categorical variables into numbers so we can use them in sklearn and then to quickly test these variables in a random forest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_rf(X_train, y_train, X_test, y_test, columns):\n",
    "    # function to train the random forest\n",
    "    # and test it on train and test sets\n",
    "    \n",
    "    rf = RandomForestRegressor(n_estimators=800, random_state=39)\n",
    "    \n",
    "    if type(columns)==str: # if we train using only 1 variable (pass a string instead of list in the \"columns\" argument of the function)\n",
    "        rf.fit(X_train[columns].to_frame(), y_train.values)\n",
    "        pred_train = rf.predict(X_train[columns].to_frame())\n",
    "        pred_test = rf.predict(X_test[columns].to_frame())\n",
    "        \n",
    "    else: # if we train using multiple variables (pass a list in the argument \"columns\")\n",
    "        rf.fit(X_train[columns], y_train.values)\n",
    "        pred_train = rf.predict(X_train[columns])\n",
    "        pred_test = rf.predict(X_test[columns])\n",
    "        \n",
    "    print('Train set')\n",
    "    print('Random Forests mse: {}'.format(mean_squared_error(y_train, pred_train)))\n",
    "    print('Test set')\n",
    "    print('Random Forests mse: {}'.format(mean_squared_error(y_test, pred_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def labels_to_numbers(X_train, X_test, columns):\n",
    "    # function to encode labels into numbers\n",
    "    # each label will be assigned an ordinal number from 0 onwards\n",
    "    \n",
    "    for col in columns:\n",
    "        labels_dict = {k:i for i, k in enumerate(X_train[col].unique(), 0)}\n",
    "        X_train.loc[:, col] = X_train.loc[:, col].map(labels_dict )\n",
    "        X_test.loc[:, col] = X_test.loc[:, col].map(labels_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables with few categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MasVnrType\n",
      "BrkCmn     0.010274\n",
      "BrkFace    0.304795\n",
      "None       0.591781\n",
      "Stone      0.087671\n",
      "Name: MasVnrType, dtype: float64\n",
      "\n",
      "ExterQual\n",
      "Ex    0.035616\n",
      "Fa    0.009589\n",
      "Gd    0.334247\n",
      "TA    0.620548\n",
      "Name: ExterQual, dtype: float64\n",
      "\n",
      "BsmtCond\n",
      "Fa    0.030822\n",
      "Gd    0.044521\n",
      "Po    0.001370\n",
      "TA    0.897945\n",
      "Name: BsmtCond, dtype: float64\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# the columns in the below list have only 4 different labels\n",
    "# let's inspect them\n",
    "\n",
    "cols = ['MasVnrType', 'ExterQual', 'BsmtCond']\n",
    "for col in cols:\n",
    "    print(data.groupby(col)[col].count()/np.float(len(data)))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variables above have only 4 categories. And in all three cases, there is at least one category that is infrequent, this is, that is present in less than 5% of the observations.\n",
    "\n",
    "When the variable has only a few categories, then perhaps it makes no sense to re-categorise the rare labels into something else. Let's look for example at the first variable MasVnrType. This variable shows only 1 rare label, BrkCmn. Thus, re-categorising it into a new label is not an option, because it will leave the variable in the same situation. Replacing of that label by the most frequent category may be done, but ideally, we should first evaluate the distribution of values (for example house prices), within the rare and frequent label. If they are similar, then it makes sense to merge the categories. If the distributions are different however, I would choose to leave the rare label as such and use the original variable without modifications.\n",
    "\n",
    "Below I will demonstrate the effects of engineering rare labels in variables with few categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MasVnrType     5\n",
       "ExterQual      0\n",
       "BsmtCond      24\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check if there are missing data\n",
    "\n",
    "X_train[cols].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two of the variables have missing data, so let's replace by the most frequent category as we saw on previous lectures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# let's create a function to replace NA by the most frequent category\n",
    "# we have seen this function in previous lectures\n",
    "\n",
    "def impute_na(df_train, df_test, variable):\n",
    "    # find most frequent category\n",
    "    most_frequent_category = df_train.groupby([variable])[variable].count().sort_values(ascending=False).index[0]\n",
    "    \n",
    "    # replace NA\n",
    "    df_train[variable].fillna(most_frequent_category, inplace=True)\n",
    "    df_test[variable].fillna(most_frequent_category, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MasVnrType    0\n",
       "ExterQual     0\n",
       "BsmtCond      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# and now we impute the NA with the function we just created\n",
    "\n",
    "for col in ['MasVnrType', 'BsmtCond']:\n",
    "    impute_na(X_train, X_test, col)\n",
    "    \n",
    "X_train[cols].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's look at those rare labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MasVnrType\n",
      "BrkCmn     0.009785\n",
      "BrkFace    0.294521\n",
      "None       0.600783\n",
      "Stone      0.094912\n",
      "Name: MasVnrType, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(X_train.groupby('MasVnrType')['MasVnrType'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The label BrkCmn is present in less than 1% of the observations. Since it is the only category under-represented, creating a new category called 'Rare' to group this label does not make much sense, as the new label Rare will be in essence the same as BrkCmn, and still under-represented. \n",
    "\n",
    "Thus, we may choose to replace the rare label by the most frequent category, in this case, 'None'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'None'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find the most frequent category, I will use this line in the below function\n",
    "frequent_cat = X_train.groupby('MasVnrType')['MasVnrType'].count().sort_values().tail(1).index.values[0]\n",
    "frequent_cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['BrkCmn']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find the rare label, I will use this line in the below function\n",
    "temp = X_train.groupby('MasVnrType')['MasVnrType'].count()/np.float(len(X_train))\n",
    "[x for x in temp.loc[temp<0.05].index.values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# for comparison, I will replace rare values by both the most frequent category\n",
    "# or by re-categorising them under a new label \"Rare\"\n",
    "# I will create a function to make the 2 rare value imputations at once\n",
    "\n",
    "def rare_imputation(X_train, X_test, variable):\n",
    "    \n",
    "    # find the most frequent category\n",
    "    frequent_cat = X_train.groupby(variable)[variable].count().sort_values().tail(1).index.values[0]\n",
    "    \n",
    "    # find rare labels\n",
    "    temp = X_train.groupby([variable])[variable].count()/np.float(len(X_train))\n",
    "    rare_cat = [x for x in temp.loc[temp<0.05].index.values]\n",
    "    \n",
    "    # create new variables, with Rare labels imputed\n",
    "    \n",
    "    # by the most frequent category\n",
    "    X_train[variable+'_freq_imp'] = np.where(X_train[variable].isin(rare_cat), frequent_cat, X_train[variable])\n",
    "    X_test[variable+'_freq_imp'] = np.where(X_test[variable].isin(rare_cat), frequent_cat, X_test[variable])\n",
    "    \n",
    "    # by adding a new label 'Rare'\n",
    "    X_train[variable+'_rare_imp'] = np.where(X_train[variable].isin(rare_cat), 'Rare', X_train[variable])\n",
    "    X_test[variable+'_rare_imp'] = np.where(X_test[variable].isin(rare_cat), 'Rare', X_test[variable])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Variable MasVnrType"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MasVnrType</th>\n",
       "      <th>MasVnrType_rare_imp</th>\n",
       "      <th>MasVnrType_freq_imp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>BrkFace</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>BrkFace</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>682</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>960</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1384</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1100</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>416</th>\n",
       "      <td>BrkFace</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>BrkFace</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1034</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>853</th>\n",
       "      <td>BrkFace</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>BrkFace</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>472</th>\n",
       "      <td>BrkFace</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>BrkFace</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     MasVnrType MasVnrType_rare_imp MasVnrType_freq_imp\n",
       "64      BrkFace             BrkFace             BrkFace\n",
       "682        None                None                None\n",
       "960        None                None                None\n",
       "1384       None                None                None\n",
       "1100       None                None                None\n",
       "416     BrkFace             BrkFace             BrkFace\n",
       "1034       None                None                None\n",
       "853     BrkFace             BrkFace             BrkFace\n",
       "472     BrkFace             BrkFace             BrkFace\n",
       "1011       None                None                None"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# impute rare labels\n",
    "rare_imputation(X_train, X_test, 'MasVnrType')\n",
    "\n",
    "# visualise the transformed dataset\n",
    "X_train[['MasVnrType', 'MasVnrType_rare_imp', 'MasVnrType_freq_imp']].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MasVnrType\n",
      "BrkCmn     0.009785\n",
      "BrkFace    0.294521\n",
      "None       0.600783\n",
      "Stone      0.094912\n",
      "Name: MasVnrType, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# let's inspect the original variable distribution\n",
    "\n",
    "print(X_train.groupby('MasVnrType')['MasVnrType'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MasVnrType_rare_imp\n",
      "BrkFace    0.294521\n",
      "None       0.600783\n",
      "Rare       0.009785\n",
      "Stone      0.094912\n",
      "Name: MasVnrType_rare_imp, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# and now the distribution after rare label to \"rare\" imputation\n",
    "\n",
    "print(X_train.groupby('MasVnrType_rare_imp')['MasVnrType_rare_imp'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see  that in essence, we did nothing  other that replacing the category name 'BrkCmn' by 'Rare'. This is not useful. **Re-grouping categories under a new label rare, only makes sense for those variables that containing more than 1 rare category**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MasVnrType_freq_imp\n",
      "BrkFace    0.294521\n",
      "None       0.610568\n",
      "Stone      0.094912\n",
      "Name: MasVnrType_freq_imp, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# and now let's inspect the rare label to 'frequent label' imputation\n",
    "print(X_train.groupby('MasVnrType_freq_imp')['MasVnrType_freq_imp'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The observations that originally displayed the rare label BrkCmn are grouped together with those that showed the most frequent label.\n",
    "\n",
    "Let's examine the performance of the three variables in Random Forests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests mse: 4879648935.038812\n",
      "Test set\n",
      "Random Forests mse: 5785517057.405808\n"
     ]
    }
   ],
   "source": [
    "# first we encode the labels into numbers with the function that we wrote a few cells ago\n",
    "\n",
    "labels_to_numbers(X_train, X_test, ['MasVnrType', 'MasVnrType_rare_imp', 'MasVnrType_freq_imp'])\n",
    "\n",
    "# and then we build a random forest using the original distribution\n",
    "train_rf(X_train, y_train, X_test, y_test, 'MasVnrType')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests mse: 4880437822.86853\n",
      "Test set\n",
      "Random Forests mse: 5787398643.51672\n"
     ]
    }
   ],
   "source": [
    "# or the distribution in which we grouped the rare values with those of the most frequent label\n",
    "\n",
    "train_rf(X_train, y_train, X_test, y_test, 'MasVnrType_freq_imp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70662320"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate the difference in mse\n",
    "\n",
    "5785517057-5714854737"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The random forest built over the training set in which we grouped the rare label with the most frequent label has a better performance. Compare the mse of 5.785e9 obtained using the original variable vs 5.787e9 obtained using the variable with the engineered rare label. Thus, as expected, engineering rare labels did boost the performance of the tree based method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Variable ExterQual"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ExterQual\n",
      "Ex    0.029354\n",
      "Fa    0.011742\n",
      "Gd    0.332681\n",
      "TA    0.626223\n",
      "Name: ExterQual, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# let's now explore another variable\n",
    "print(X_train.groupby('ExterQual')['ExterQual'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This variable has 2 categories that are rare, 'Ex' and 'Fa'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ExterQual</th>\n",
       "      <th>ExterQual_rare_imp</th>\n",
       "      <th>ExterQual_freq_imp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>682</th>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>960</th>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1384</th>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1100</th>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>416</th>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1034</th>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>853</th>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>472</th>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     ExterQual ExterQual_rare_imp ExterQual_freq_imp\n",
       "64          TA                 TA                 TA\n",
       "682         TA                 TA                 TA\n",
       "960         TA                 TA                 TA\n",
       "1384        TA                 TA                 TA\n",
       "1100        TA                 TA                 TA\n",
       "416         TA                 TA                 TA\n",
       "1034        TA                 TA                 TA\n",
       "853         TA                 TA                 TA\n",
       "472         TA                 TA                 TA\n",
       "1011        TA                 TA                 TA"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's engineer the rare labels into the 'Rare' or 'most frequent label' \n",
    "# using the function we defined above\n",
    "\n",
    "rare_imputation(X_train, X_test, 'ExterQual')\n",
    "X_train[['ExterQual', 'ExterQual_rare_imp', 'ExterQual_freq_imp']].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ExterQual\n",
      "Ex    0.029354\n",
      "Fa    0.011742\n",
      "Gd    0.332681\n",
      "TA    0.626223\n",
      "Name: ExterQual, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# let's examine the original distribution again\n",
    "print(X_train.groupby('ExterQual')['ExterQual'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ExterQual_freq_imp\n",
      "Gd    0.332681\n",
      "TA    0.667319\n",
      "Name: ExterQual_freq_imp, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# and now the imputation into most frequent label\n",
    "print(X_train.groupby('ExterQual_freq_imp')['ExterQual_freq_imp'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ExterQual_rare_imp\n",
      "Gd      0.332681\n",
      "Rare    0.041096\n",
      "TA      0.626223\n",
      "Name: ExterQual_rare_imp, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# and the imputation into the rare label\n",
    "print(X_train.groupby('ExterQual_rare_imp')['ExterQual_rare_imp'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The imputation into rare label has generated an additional category, called \"Rare\" under which the observations with the labels Ex and Fa are now grouped. On the other had, the frequent label imputation has merged the observations of the labels Ex and Fa with those of the label TA, leaving only 2 categories in that variable.\n",
    "\n",
    "Let's examine the performance of the different methods in random forests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests mse: 3285699437.6173377\n",
      "Test set\n",
      "Random Forests mse: 3327665435.796429\n"
     ]
    }
   ],
   "source": [
    "# first we transform the label strings into numbers so we can use sklearn\n",
    "labels_to_numbers(X_train, X_test, ['ExterQual', 'ExterQual_rare_imp', 'ExterQual_freq_imp'])\n",
    "\n",
    "# and now we build Random Forests using the original variable\n",
    "train_rf(X_train, y_train, X_test, y_test, 'ExterQual')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests mse: 3970209768.435656\n",
      "Test set\n",
      "Random Forests mse: 3747390792.6268826\n"
     ]
    }
   ],
   "source": [
    "# and comparatively, we build random forests using the rare imputation into the 'Rare' category\n",
    "train_rf(X_train, y_train, X_test, y_test, 'ExterQual_rare_imp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests mse: 4777247329.540922\n",
      "Test set\n",
      "Random Forests mse: 5572669187.136114\n"
     ]
    }
   ],
   "source": [
    "# and finally, we build random forests using the rare into frequent label imputation methods\n",
    "train_rf(X_train, y_train, X_test, y_test, 'ExterQual_freq_imp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On this occasion, the random forests built using all the labels in the original variable performs best (mse test: 33e8). Grouping rare variables under 'Rare' stills shows some reasonable performance (mse test: 37e8). However, replacing the infrequent labels by the most frequent ones makes the random forests under-perform quite dramatically (mse test: 55e8). We can try to understand why grouping the rare values together brought a drop in performance by examining the mean house price within each label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ExterQual\n",
       "Ex    364606.5\n",
       "Fa     82250.0\n",
       "Gd    220000.0\n",
       "TA    139450.0\n",
       "Name: SalePrice, dtype: float64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.groupby('ExterQual')['SalePrice'].median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the median house price among the 2 rare categories is extremely different. Thus, merging them into one masks the value added by this label. And this is why, in this case, keeping the labels separate renders better performance of the algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Variable BsmtCond"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BsmtCond\n",
      "Fa    0.032290\n",
      "Gd    0.045010\n",
      "Po    0.001957\n",
      "TA    0.920744\n",
      "Name: BsmtCond, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# let's do the exercise one more time for an additional variable\n",
    "# let's examine the distribution of observations among the different categories within this variable\n",
    "\n",
    "print(X_train.groupby('BsmtCond')['BsmtCond'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can anticipate that replacing all rare labels (Fa, Gd and Po) by the most frequent category (TA) will in essence remove all of the information, as we will end up with only one category. If, alternatively, we group the rare categories into 1, we will end up with 2 categories, presumably removing substantial information as well. See below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rare_imputation(X_train, X_test, 'BsmtCond')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BsmtCond\n",
      "Fa    0.032290\n",
      "Gd    0.045010\n",
      "Po    0.001957\n",
      "TA    0.920744\n",
      "Name: BsmtCond, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# original distribution\n",
    "print(X_train.groupby('BsmtCond')['BsmtCond'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BsmtCond_freq_imp\n",
      "TA    1.0\n",
      "Name: BsmtCond_freq_imp, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# replacing by most frequent\n",
    "\n",
    "print(X_train.groupby('BsmtCond_freq_imp')['BsmtCond_freq_imp'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BsmtCond_rare_imp\n",
      "Rare    0.079256\n",
      "TA      0.920744\n",
      "Name: BsmtCond_rare_imp, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# grouping under rare\n",
    "print(X_train.groupby('BsmtCond_rare_imp')['BsmtCond_rare_imp'].count()/np.float(len(X_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will leave the exercise of comparing how this impacts the performance of random forests to you. You got the idea already."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "In my opinion, engineering rare labels in variables with very few categories like the ones we worked in this notebook, seems unlikely to bring forward a boost in the performance of the algorithm. And this is because few categories are unlikely to bring so much noise. \n",
    "\n",
    "However, if the number of categories increases, then it becomes more important to handle the rare labels.\n",
    "\n",
    "**That is all for this demonstration. I hope you enjoyed the notebook, and see you in the next one.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}