Part 1-Preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Peer-graded Assignment: Segmenting and Clustering Neighborhoods in Toronto"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For this assignment, you will be required to explore and cluster the neighborhoods in Toronto.\n",
    "\n",
    "#### 1- Start by creating a new Notebook for this assignment.\n",
    "#### 2- Use the Notebook to build the code to scrape the following Wikipedia page, https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M, in order to obtain the data that is in the table of postal codes and to transform the data into a pandas dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing necessary libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from bs4 import BeautifulSoup\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# getting data from internet\n",
    "wikipedia_link='https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M'\n",
    "raw_wikipedia_page= requests.get(wikipedia_link).text\n",
    "\n",
    "# using beautiful soup to parse the HTML/XML codes.\n",
    "soup = BeautifulSoup(raw_wikipedia_page,'xml')\n",
    "#print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing-part-1: extracting raw table (from webpage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extracting the raw table inside that webpage\n",
    "table = soup.find('table')\n",
    "\n",
    "Postcode      = []\n",
    "Borough       = []\n",
    "Neighbourhood = []\n",
    "\n",
    "# print(table)\n",
    "\n",
    "# extracting a clean form of the table\n",
    "for tr_cell in table.find_all('tr'):\n",
    "    \n",
    "    counter = 1\n",
    "    Postcode_var      = -1\n",
    "    Borough_var       = -1\n",
    "    Neighbourhood_var = -1\n",
    "    \n",
    "    for td_cell in tr_cell.find_all('td'):\n",
    "        if counter == 1: \n",
    "            Postcode_var = td_cell.text\n",
    "        if counter == 2: \n",
    "            Borough_var = td_cell.text\n",
    "            tag_a_Borough = td_cell.find('a')\n",
    "            \n",
    "        if counter == 3: \n",
    "            Neighbourhood_var = str(td_cell.text).strip()\n",
    "            tag_a_Neighbourhood = td_cell.find('a')\n",
    "            \n",
    "        counter +=1\n",
    "        \n",
    "    if (Postcode_var == 'Not assigned' or Borough_var == 'Not assigned' or Neighbourhood_var == 'Not assigned'): \n",
    "        continue\n",
    "    try:\n",
    "        if ((tag_a_Borough is None) or (tag_a_Neighbourhood is None)):\n",
    "            continue\n",
    "    except:\n",
    "        pass\n",
    "    if(Postcode_var == -1 or Borough_var == -1 or Neighbourhood_var == -1):\n",
    "        continue\n",
    "        \n",
    "    Postcode.append(Postcode_var)\n",
    "    Borough.append(Borough_var)\n",
    "    Neighbourhood.append(Neighbourhood_var)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing-part-2: integrating Postal codes with more than 1 neighbour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num of unique Postal codes: 84\n"
     ]
    }
   ],
   "source": [
    "\n",
    "unique_p = set(Postcode)\n",
    "print('num of unique Postal codes:', len(unique_p))\n",
    "Postcode_u      = []\n",
    "Borough_u       = []\n",
    "Neighbourhood_u = []\n",
    "\n",
    "\n",
    "for postcode_unique_element in unique_p:\n",
    "    p_var = ''; b_var = ''; n_var = ''; \n",
    "    for postcode_idx, postcode_element in enumerate(Postcode):\n",
    "        if postcode_unique_element == postcode_element:\n",
    "            p_var = postcode_element;\n",
    "            b_var = Borough[postcode_idx]\n",
    "            if n_var == '': \n",
    "                n_var = Neighbourhood[postcode_idx]\n",
    "            else:\n",
    "                n_var = n_var + ', ' + Neighbourhood[postcode_idx]\n",
    "    Postcode_u.append(p_var)\n",
    "    Borough_u.append(b_var)\n",
    "    Neighbourhood_u.append(n_var)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Post-processing: creating an appropriate Pandas Dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Postcode</th>\n",
       "      <th>Borough</th>\n",
       "      <th>Neighbourhood</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M5V</td>\n",
       "      <td>Downtown Toronto</td>\n",
       "      <td>CN Tower, King and Spadina, Railway Lands, Sou...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M4L</td>\n",
       "      <td>East Toronto</td>\n",
       "      <td>India Bazaar</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>M6H</td>\n",
       "      <td>West Toronto</td>\n",
       "      <td>Dovercourt Village</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M2K</td>\n",
       "      <td>North York</td>\n",
       "      <td>Bayview Village</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M6S</td>\n",
       "      <td>West Toronto</td>\n",
       "      <td>Runnymede, Swansea</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>M8Z</td>\n",
       "      <td>Etobicoke</td>\n",
       "      <td>Mimico NW, The Queensway West, South of Bloor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>M1J</td>\n",
       "      <td>Scarborough</td>\n",
       "      <td>Scarborough Village</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>M1R</td>\n",
       "      <td>Scarborough</td>\n",
       "      <td>Maryvale, Wexford</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>M4T</td>\n",
       "      <td>Central Toronto</td>\n",
       "      <td>Moore Park</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>M4W</td>\n",
       "      <td>Downtown Toronto</td>\n",
       "      <td>Rosedale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>M9L</td>\n",
       "      <td>North York</td>\n",
       "      <td>Humber Summit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>M1N</td>\n",
       "      <td>Scarborough</td>\n",
       "      <td>Birch Cliff</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>M5R</td>\n",
       "      <td>Central Toronto</td>\n",
       "      <td>The Annex, Yorkville</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>M1K</td>\n",
       "      <td>Scarborough</td>\n",
       "      <td>Ionview, Kennedy Park</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Postcode           Borough  \\\n",
       "0       M5V  Downtown Toronto   \n",
       "1       M4L      East Toronto   \n",
       "2       M6H      West Toronto   \n",
       "3       M2K        North York   \n",
       "4       M6S      West Toronto   \n",
       "5       M8Z         Etobicoke   \n",
       "6       M1J       Scarborough   \n",
       "7       M1R       Scarborough   \n",
       "8       M4T   Central Toronto   \n",
       "9       M4W  Downtown Toronto   \n",
       "10      M9L        North York   \n",
       "11      M1N       Scarborough   \n",
       "12      M5R   Central Toronto   \n",
       "13      M1K       Scarborough   \n",
       "\n",
       "                                        Neighbourhood  \n",
       "0   CN Tower, King and Spadina, Railway Lands, Sou...  \n",
       "1                                        India Bazaar  \n",
       "2                                  Dovercourt Village  \n",
       "3                                     Bayview Village  \n",
       "4                                  Runnymede, Swansea  \n",
       "5       Mimico NW, The Queensway West, South of Bloor  \n",
       "6                                 Scarborough Village  \n",
       "7                                   Maryvale, Wexford  \n",
       "8                                          Moore Park  \n",
       "9                                            Rosedale  \n",
       "10                                      Humber Summit  \n",
       "11                                        Birch Cliff  \n",
       "12                               The Annex, Yorkville  \n",
       "13                              Ionview, Kennedy Park  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toronto_dict = {'Postcode':Postcode_u, 'Borough':Borough_u, 'Neighbourhood':Neighbourhood_u}\n",
    "df_toronto = pd.DataFrame.from_dict(toronto_dict)\n",
    "df_toronto.to_csv('toronto_part1.csv')\n",
    "df_toronto.head(14)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(84, 3)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_toronto.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Please Note: The number of unique postal codes is 84. If you think any part of my code is incorrect, feel free to teach or inform me. \n",
    "#### Thanks and Best Reagrds, \n",
    "#### Ali Dastgheib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}