APIs and Internet Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is an API?\n",
    "\n",
    "https://www.upwork.com/hiring/development/intro-to-apis-what-is-an-api/\n",
    "\n",
    "Application Programming Interface: messenger between client and server. \n",
    "\n",
    "**interface that software uses to access whatever currency it needs: data, server software, or other applications.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In Practice, an API is used to access data or serve from another company's software (like using google as account to sign in) for your own code**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieving Data from APIs:\n",
    "### Beginner Example: International Space Station\n",
    "https://www.dataquest.io/blog/python-api-tutorial/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to receive data: need request and authentication\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "different types of HTTP request: most common are **GET** and **POST**\n",
    "HTTP refers to Hypertext Transfer Protocol, is basically internet's format for client-server communication\n",
    "\n",
    "**GET**  *receives* data from a source\n",
    "\n",
    "**POST** *sends* data to a server to update resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "bad_response = requests.get(\"http://api.open-notify.org/this-api-doesnt-exist\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Responses are returned with 2 things: response code and response data\n",
    "### Most important Request Codes:\n",
    "### 200 - everything good. This is ideal\n",
    "### 404, 500 error\n",
    "for more: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "404"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#fake api, 404 error\n",
    "#bad_response.json() #error\n",
    "bad_response.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"Open Notify API is simple and requires no authentication\n",
    "\n",
    "Servers/ sites often have multiple APIs to access. Each called 'Endpoint'\n",
    "\n",
    "Endpoint we're using: http://open-notify.org/Open-Notify-API/People-In-Space/\n",
    "\"\"\"\n",
    "response = requests.get(\"http://api.open-notify.org/astros.json\")\n",
    "response.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# JSON\n",
    "JSON internet shared language for data: VERY common. Basically python dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'people': [{'craft': 'ISS', 'name': 'Andrew Morgan'},\n",
       "  {'craft': 'ISS', 'name': 'Oleg Skripochka'},\n",
       "  {'craft': 'ISS', 'name': 'Jessica Meir'}],\n",
       " 'message': 'success',\n",
       " 'number': 3}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_obj = response.json()\n",
    "json_obj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'success'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_obj['message']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_obj['number'] # num people on space station"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'craft': 'ISS', 'name': 'Andrew Morgan'},\n",
       " {'craft': 'ISS', 'name': 'Oleg Skripochka'},\n",
       " {'craft': 'ISS', 'name': 'Jessica Meir'}]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_obj['people']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Andrew Morgan'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_obj['people'][0]['name']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Json library in python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### json.dumps() : python to string\n",
    "#### json.loads() string to python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'dict'>\n",
      "{\"people\": [{\"craft\": \"ISS\", \"name\": \"Andrew Morgan\"}, {\"craft\": \"ISS\", \"name\": \"Oleg Skripochka\"}, {\"craft\": \"ISS\", \"name\": \"Jessica Meir\"}], \"message\": \"success\", \"number\": 3}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(type(json_obj))\n",
    "print(json.dumps(json_obj))\n",
    "type(json.dumps(json_obj)) # python string now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"message\": \"success\",\n",
      "    \"request\": {\n",
      "        \"altitude\": 100,\n",
      "        \"datetime\": 1584466549,\n",
      "        \"latitude\": 40.71,\n",
      "        \"longitude\": -74.0,\n",
      "        \"passes\": 5\n",
      "    },\n",
      "    \"response\": [\n",
      "        {\n",
      "            \"duration\": 587,\n",
      "            \"risetime\": 1584489992\n",
      "        },\n",
      "        {\n",
      "            \"duration\": 646,\n",
      "            \"risetime\": 1584495755\n",
      "        },\n",
      "        {\n",
      "            \"duration\": 579,\n",
      "            \"risetime\": 1584501632\n",
      "        },\n",
      "        {\n",
      "            \"duration\": 567,\n",
      "            \"risetime\": 1584507509\n",
      "        },\n",
      "        {\n",
      "            \"duration\": 637,\n",
      "            \"risetime\": 1584513328\n",
      "        }\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "def jprint(obj):\n",
    "    text = json.dumps(obj, sort_keys=True, indent=4)\n",
    "    print(text)\n",
    "jprint(response.json())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"researcher\": {\n",
      "        \"name\": \"Ford Prefect\",\n",
      "        \"relatives\": [\n",
      "            {\n",
      "                \"name\": \"Zaphod Beeblebrox\",\n",
      "                \"species\": \"Betelgeusian\"\n",
      "            }\n",
      "        ],\n",
      "        \"species\": \"Betelgeusian\"\n",
      "    }\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "dict"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_string = \"\"\"\n",
    "{\n",
    "    \"researcher\": {\n",
    "        \"name\": \"Ford Prefect\",\n",
    "        \"species\": \"Betelgeusian\",\n",
    "        \"relatives\": [\n",
    "            {\n",
    "                \"name\": \"Zaphod Beeblebrox\",\n",
    "                \"species\": \"Betelgeusian\"\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\"\"\"\n",
    "data = json.loads(json_string)\n",
    "jprint(data)\n",
    "type(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the process of converting an object into a stream of bytes to store the object or transmit it to memory, database, or a file\n",
    "\n",
    "##### (basically, save an object) or writing\n",
    "\n",
    "## Thus, Deserialization is from memory (could be a file) to object (reading)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    \"president\": {\n",
    "        \"name\": \"Zaphod Beeblebrox\",\n",
    "        \"species\": \"Betelgeusian\"\n",
    "    }\n",
    "}\n",
    "\n",
    "with open(\"data_file.json\", \"w\") as write_file:\n",
    "    # dump takes in data, then place to put it\n",
    "    json.dump(data, write_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dumps vs Dump: \n",
    "dump writes python to file\n",
    "\n",
    "Dumps turns python into string. Pretty similar\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"president\": {\n",
      "        \"name\": \"Zaphod Beeblebrox\",\n",
      "        \"species\": \"Betelgeusian\"\n",
      "    }\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "dict"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open(\"data_file.json\", \"r\") as read_file:\n",
    "    read_data = json.load(read_file)\n",
    "jprint(read_data)\n",
    "type(read_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p> This is that <span style=\"color:red\"> good </span> shit: HTML markdown </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Request with parameters\n",
    "## many endpoints require param"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"message\": \"success\",\n",
      "    \"request\": {\n",
      "        \"altitude\": 100,\n",
      "        \"datetime\": 1584466549,\n",
      "        \"latitude\": 40.71,\n",
      "        \"longitude\": -74.0,\n",
      "        \"passes\": 5\n",
      "    },\n",
      "    \"response\": [\n",
      "        {\n",
      "            \"duration\": 587,\n",
      "            \"risetime\": 1584489992\n",
      "        },\n",
      "        {\n",
      "            \"duration\": 646,\n",
      "            \"risetime\": 1584495755\n",
      "        },\n",
      "        {\n",
      "            \"duration\": 579,\n",
      "            \"risetime\": 1584501632\n",
      "        },\n",
      "        {\n",
      "            \"duration\": 567,\n",
      "            \"risetime\": 1584507509\n",
      "        },\n",
      "        {\n",
      "            \"duration\": 637,\n",
      "            \"risetime\": 1584513328\n",
      "        }\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "parameters = {\n",
    "    \"lat\": 40.71,\n",
    "    \"lon\": -74\n",
    "}\n",
    "\n",
    "response = requests.get(\"http://api.open-notify.org/iss-pass.json\", params=parameters)\n",
    "jprint(response.json())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'duration': 587, 'risetime': 1584489992},\n",
       " {'duration': 646, 'risetime': 1584495755},\n",
       " {'duration': 579, 'risetime': 1584501632},\n",
       " {'duration': 567, 'risetime': 1584507509},\n",
       " {'duration': 637, 'risetime': 1584513328}]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json = response.json()\n",
    "json[\"response\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "risetimes = []\n",
    "for passes in json[\"response\"]:\n",
    "    risetimes.append(passes[\"risetime\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1584489992, 1584495755, 1584501632, 1584507509, 1584513328]\n"
     ]
    }
   ],
   "source": [
    "print(risetimes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[datetime.datetime(2020, 3, 17, 20, 6, 32),\n",
       " datetime.datetime(2020, 3, 17, 21, 42, 35),\n",
       " datetime.datetime(2020, 3, 17, 23, 20, 32),\n",
       " datetime.datetime(2020, 3, 18, 0, 58, 29),\n",
       " datetime.datetime(2020, 3, 18, 2, 35, 28)]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time_convert = lambda time: datetime.fromtimestamp(time)\n",
    "risetimes = list(map(time_convert,risetimes)) # applies function to iterable\n",
    "risetimes # ISS passes over NYC often!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NEXT Steps/ things to check out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.dataquest.io/blog/last-fm-api-python/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}