penguins.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "00b7ba7b-433c-463c-8e5e-8b975a5be463",
   "metadata": {},
   "source": [
    "# Penguins in Production\n",
    "\n",
    "This notebook aims to create a [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) to build an end-to-end Machine Learning system to solve the problem of classifying penguin species.\n",
    "\n",
    "This example uses the [Penguins dataset](https://www.kaggle.com/parulpandey/palmer-archipelago-antarctica-penguin-data).\n",
    "\n",
    "<img src='https://imgur.com/orZWHly.png' alt='Penguins dataset' width=\"900\">\n",
    "\n",
    "Amazon SageMaker is free to try. Your free tier starts from the first month you create your first SageMaker resource and lasts two months. Check out the  [Amazon SageMaker Pricing](https://aws.amazon.com/sagemaker/pricing/) for more information. Also, we'll be working extensively with [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html) and the [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/). Keep their documentation handy.\n",
    "\n",
    "This notebook was created by [Santiago L. Valdarrama](https://twitter.com/svpino) as part of the [Machine Learning School](https://www.ml.school) program."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db6203b0-5a40-4a14-b9bb-6e092f1bb2e7",
   "metadata": {},
   "source": [
    "Let's ensure we are running the latest version of the SakeMaker SDK. **Restart the Kernel** after you run the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "72eb4984-3c21-4033-be45-2de2cfc1257c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mName: sagemaker\n",
      "Version: 2.148.0\n",
      "Summary: Open source library for training and deploying models on Amazon SageMaker.\n",
      "Home-page: https://github.com/aws/sagemaker-python-sdk/\n",
      "Author: Amazon Web Services\n",
      "Author-email: \n",
      "License: Apache License 2.0\n",
      "Location: /usr/local/lib/python3.8/site-packages\n",
      "Requires: attrs, boto3, cloudpickle, google-pasta, importlib-metadata, jsonschema, numpy, packaging, pandas, pathos, platformdirs, protobuf, protobuf3-to-dict, PyYAML, schema, smdebug-rulesconfig, tblib\n",
      "Required-by: \n"
     ]
    }
   ],
   "source": [
    "!pip install -q --upgrade pip\n",
    "!pip install -q --upgrade awscli boto3\n",
    "!pip install -q --upgrade sagemaker\n",
    "!pip show sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b80dfa35-b345-4f38-8426-f0223de0c401",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9992fc69-4868-4e50-a50e-76da237f70ac",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Session 1 - Getting Started\n",
    "\n",
    "This session aims to build a simple [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) with one step to preprocess the [Penguins dataset](https://www.kaggle.com/parulpandey/palmer-archipelago-antarctica-penguin-data). We'll use a [Processing Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing) with a [SKLearnProcessor](https://sagemaker.readthedocs.io/en/stable/frameworks/sklearn/sagemaker.sklearn.html#scikit-learn-processor) to execute a preprocessing script. Check the [SageMaker Pipelines Overview](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) for an introduction to the fundamental components of a SageMaker Pipeline.\n",
    "\n",
    "Here is what the Pipeline will look like at the end of this session:\n",
    "\n",
    "<img src='penguins/images/session1-pipeline.png' alt='Penguins dataset' width=\"600\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "915b1d0b-d9da-4529-aca5-fd1c08a36f48",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sagemaker\n",
    "import numpy as np\n",
    "import boto3\n",
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import urllib.request\n",
    "import argparse\n",
    "import tempfile\n",
    "from pathlib import Path\n",
    "\n",
    "from botocore.exceptions import ClientError\n",
    "from sagemaker.inputs import FileSystemInput\n",
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "from sagemaker.processing import ScriptProcessor\n",
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "from sagemaker.workflow.steps import ProcessingStep\n",
    "from sagemaker.workflow.model_step import ModelStep\n",
    "from sagemaker.workflow.pipeline_context import PipelineSession\n",
    "from sagemaker.workflow.parameters import ParameterInteger, ParameterString, ParameterFloat\n",
    "from sagemaker.workflow.pipeline import Pipeline\n",
    "from sagemaker.workflow.steps import CacheConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef66efe2-6a01-41b5-ba8a-9c82c480994c",
   "metadata": {},
   "source": [
    "Let's start by defining a few variables we'll use throughout this notebook:\n",
    "\n",
    "* `sagemaker_client`: We'll use a [boto3 SageMaker Client](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html) instance to access SageMaker.\n",
    "* `iam_client`: We'll use a [boto3 IAM Client](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam.html) instance to access IAM.\n",
    "* `role`: This is the execution role attached to this notebook. We can use this role with any of the SageMaker services that need it to ensure they run with the appropriate permissions.\n",
    "* `region`: The current region attached to our session. \n",
    "* `sagemaker_session`: The current SageMaker session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "7bb1e9bf-25d8-401b-a7d9-0455b1a7434d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "iam_client = boto3.client(\"iam\")\n",
    "sagemaker_client = boto3.client(\"sagemaker\")\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "sagemaker_session = sagemaker.session.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abb24b35-450a-4b96-a544-38f084433b62",
   "metadata": {},
   "source": [
    "## Step 1 - Creating an S3 Bucket\n",
    "\n",
    "We must create an S3 bucket to upload everything we need during the program. Make sure you set `BUCKET` to the bucket name you want to use. This name has to be unique. The [command line interface](https://docs.aws.amazon.com/cli/latest/index.html) is a simple way to interact with the AWS services. You can combine Python code with bash commands in the same notebook cell, which makes notebooks a very flexible tool.\n",
    "\n",
    "If you want to create a bucket in a region other than `us-east-1`, use this command instead:\n",
    "\n",
    "```\n",
    "!aws s3api create-bucket --bucket $BUCKET --create-bucket-configuration LocationConstraint=$region\n",
    "```\n",
    "\n",
    "The `LocationConstraint` argument should specify the region where you want to create the bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3ea8d2fd-7dd4-43c8-b9d9-5372448d72a6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"Location\": \"/mlschool\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "BUCKET = \"mlschool\"\n",
    "\n",
    "!aws s3api create-bucket --bucket $BUCKET"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accefa0d-31d6-4df4-814a-d74c703689b5",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Step 2 - Downloading the Dataset\n",
    "\n",
    "Let's download the [Penguins dataset](https://www.kaggle.com/parulpandey/palmer-archipelago-antarctica-penguin-data) and store it in S3.\n",
    "\n",
    "The SageMaker Pipeline will use the data stored in this S3 location. A production application can add more data to the exact location, and the Pipeline will still work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "882a8be9-dc31-4df1-9009-3b2d0284cb70",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset S3 location: s3://mlschool/penguins/data.csv\n"
     ]
    }
   ],
   "source": [
    "PENGUINS_FOLDER = Path(\"penguins\")\n",
    "\n",
    "S3_FILEPATH = f\"s3://{BUCKET}/{PENGUINS_FOLDER}\"\n",
    "LOCAL_FILEPATH = Path(PENGUINS_FOLDER)/ \"data.csv\"\n",
    "\n",
    "# Create the local folder if it doesn't exist.\n",
    "PENGUINS_FOLDER.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "# Download the official Penguins dataset and store it locally.\n",
    "urllib.request.urlretrieve(\n",
    "    \"https://storage.googleapis.com/download.tensorflow.org/data/palmer_penguins/penguins_size.csv\", \n",
    "    LOCAL_FILEPATH\n",
    ")\n",
    "\n",
    "# Upload the dataset to S3. We need to do this to make it available to \n",
    "# the preprocessing step.\n",
    "INPUT_DATA_URI = sagemaker.s3.S3Uploader.upload(\n",
    "    local_path=str(LOCAL_FILEPATH), \n",
    "    desired_s3_uri=S3_FILEPATH,\n",
    ")\n",
    "\n",
    "print(f\"Dataset S3 location: {INPUT_DATA_URI}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59e1efbc-5f66-4b44-9555-68a4731e8e7b",
   "metadata": {},
   "source": [
    "We can now load and display the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a19b546a-6884-483b-8cda-0521e2656a99",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species</th>\n",
       "      <th>island</th>\n",
       "      <th>culmen_length_mm</th>\n",
       "      <th>culmen_depth_mm</th>\n",
       "      <th>flipper_length_mm</th>\n",
       "      <th>body_mass_g</th>\n",
       "      <th>sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>39.1</td>\n",
       "      <td>18.7</td>\n",
       "      <td>181.0</td>\n",
       "      <td>3750.0</td>\n",
       "      <td>MALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>39.5</td>\n",
       "      <td>17.4</td>\n",
       "      <td>186.0</td>\n",
       "      <td>3800.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>40.3</td>\n",
       "      <td>18.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>3250.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>36.7</td>\n",
       "      <td>19.3</td>\n",
       "      <td>193.0</td>\n",
       "      <td>3450.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>339</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>340</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>46.8</td>\n",
       "      <td>14.3</td>\n",
       "      <td>215.0</td>\n",
       "      <td>4850.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>341</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>50.4</td>\n",
       "      <td>15.7</td>\n",
       "      <td>222.0</td>\n",
       "      <td>5750.0</td>\n",
       "      <td>MALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>45.2</td>\n",
       "      <td>14.8</td>\n",
       "      <td>212.0</td>\n",
       "      <td>5200.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>343</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>49.9</td>\n",
       "      <td>16.1</td>\n",
       "      <td>213.0</td>\n",
       "      <td>5400.0</td>\n",
       "      <td>MALE</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>344 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    species     island  culmen_length_mm  culmen_depth_mm  flipper_length_mm  \\\n",
       "0    Adelie  Torgersen              39.1             18.7              181.0   \n",
       "1    Adelie  Torgersen              39.5             17.4              186.0   \n",
       "2    Adelie  Torgersen              40.3             18.0              195.0   \n",
       "3    Adelie  Torgersen               NaN              NaN                NaN   \n",
       "4    Adelie  Torgersen              36.7             19.3              193.0   \n",
       "..      ...        ...               ...              ...                ...   \n",
       "339  Gentoo     Biscoe               NaN              NaN                NaN   \n",
       "340  Gentoo     Biscoe              46.8             14.3              215.0   \n",
       "341  Gentoo     Biscoe              50.4             15.7              222.0   \n",
       "342  Gentoo     Biscoe              45.2             14.8              212.0   \n",
       "343  Gentoo     Biscoe              49.9             16.1              213.0   \n",
       "\n",
       "     body_mass_g     sex  \n",
       "0         3750.0    MALE  \n",
       "1         3800.0  FEMALE  \n",
       "2         3250.0  FEMALE  \n",
       "3            NaN     NaN  \n",
       "4         3450.0  FEMALE  \n",
       "..           ...     ...  \n",
       "339          NaN     NaN  \n",
       "340       4850.0  FEMALE  \n",
       "341       5750.0    MALE  \n",
       "342       5200.0  FEMALE  \n",
       "343       5400.0    MALE  \n",
       "\n",
       "[344 rows x 7 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(LOCAL_FILEPATH)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "587d06e0-b711-4e3d-b424-6fa611a51f94",
   "metadata": {},
   "source": [
    "## Step 3 - Preprocessing the Dataset\n",
    "\n",
    "Let's create a script to do feature engineering on the original dataset. We will run this script using a SageMaker Processing Job later in this session.\n",
    "\n",
    "The script should split the data into train, validation, and test sets so we can later train and evaluate a model. We will save the Scikit-Learn pipeline that we use to preprocess the data to use it during inference time.\n",
    "\n",
    "The script uses the [np.split()](https://numpy.org/doc/stable/reference/generated/numpy.split.html) function to split the dataset into three sets in the following way:\n",
    "\n",
    "1. The train set will use the top 70% of the data.\n",
    "2. The validation set will use 15% of the data, starting with the sample after the 70% used for the train set.\n",
    "3. Finally, the test set will use the remaining 15% of the data.\n",
    "\n",
    "Pay special attention to the way the Scikit-Learn pipeline `preprocessor` is used to process the three sets:\n",
    "\n",
    "* First, we use the `fit_transform()` to fit the pipeline on the train set.\n",
    "* Then, we consecutively transform the validation and test sets using `transform()`.\n",
    "\n",
    "Always use `fit_transform()` on the training data to fit the scaling parameters we need to transform the data. For example, `fit_transform()` will learn the mean and variance of the features of the training set. It can then use these same parameters to scale the validation and test sets.\n",
    "\n",
    "That's why we want to save this Scikit-Learn pipeline to use later to scale production data using the same parameters we learned on the train set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fb6ba7c0-1bd6-4fe5-8b7f-f6cbdfd3846c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/preprocessor.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {PENGUINS_FOLDER}/preprocessor.py\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tempfile\n",
    "\n",
    "from pathlib import Path\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, LabelEncoder, StandardScaler\n",
    "from pickle import dump\n",
    "\n",
    "\n",
    "# This is the location where the SageMaker Processing job\n",
    "# will save the input dataset.\n",
    "BASE_DIR = \"/opt/ml/processing\"\n",
    "DATA_FILEPATH = Path(BASE_DIR) / \"input\" / \"data.csv\"\n",
    "\n",
    "\n",
    "def save_splits(base_dir, train, validation, test):\n",
    "    \"\"\"\n",
    "    One of the goals of this script is to output the three\n",
    "    dataset splits. This function will save each of these\n",
    "    splits to disk.\n",
    "    \"\"\"\n",
    "    \n",
    "    train_path = Path(base_dir) / \"train\" \n",
    "    validation_path = Path(base_dir) / \"validation\" \n",
    "    test_path = Path(base_dir) / \"test\"\n",
    "    \n",
    "    train_path.mkdir(parents=True, exist_ok=True)\n",
    "    validation_path.mkdir(parents=True, exist_ok=True)\n",
    "    test_path.mkdir(parents=True, exist_ok=True)\n",
    "    \n",
    "    pd.DataFrame(train).to_csv(train_path / \"train.csv\", header=False, index=False)\n",
    "    pd.DataFrame(validation).to_csv(validation_path / \"validation.csv\", header=False, index=False)\n",
    "    pd.DataFrame(test).to_csv(test_path / \"test.csv\", header=False, index=False)\n",
    "    \n",
    "\n",
    "def save_pipeline(base_dir, pipeline):\n",
    "    \"\"\"\n",
    "    Saves the Scikit-Learn pipeline that we used to\n",
    "    preprocess the data.\n",
    "    \"\"\"\n",
    "    pipeline_path = Path(base_dir) / \"pipeline\"\n",
    "    pipeline_path.mkdir(parents=True, exist_ok=True)\n",
    "    dump(pipeline, open(pipeline_path / \"pipeline.pkl\", 'wb'))\n",
    "    \n",
    "\n",
    "def generate_baseline(base_dir, X_train, y_train):\n",
    "    \"\"\"\n",
    "    Generates a baseline for our model using the train set. \n",
    "    It saves the baseline in a JSON file where every line is \n",
    "    a JSON object.\n",
    "    \"\"\"\n",
    "    baseline_path = Path(base_dir) / \"baseline\" \n",
    "    baseline_path.mkdir(parents=True, exist_ok=True)\n",
    "    \n",
    "    df = X_train.copy()\n",
    "    df[\"groundtruth\"] = y_train\n",
    "\n",
    "    df.to_json(baseline_path / \"baseline.json\", orient='records', lines=True)\n",
    "    \n",
    "    \n",
    "def preprocess(base_dir, data_filepath):\n",
    "    \"\"\"\n",
    "    Preprocesses the supplied raw dataset and splits it into a train, validation,\n",
    "    and a test set.\n",
    "    \"\"\"\n",
    "    \n",
    "    df = pd.read_csv(data_filepath)\n",
    "    \n",
    "    numerical_columns = [column for column in df.columns if df[column].dtype in [\"int64\", \"float64\"]]\n",
    "    \n",
    "    numerical_preprocessor = Pipeline(steps=[\n",
    "        (\"imputer\", SimpleImputer(strategy=\"mean\")),\n",
    "        (\"scaler\", StandardScaler())\n",
    "    ])\n",
    "\n",
    "    categorical_preprocessor = Pipeline(steps=[\n",
    "        (\"imputer\", SimpleImputer(strategy=\"most_frequent\")),\n",
    "        (\"onehot\", OneHotEncoder(handle_unknown=\"ignore\"))\n",
    "    ])\n",
    "\n",
    "    preprocessor = ColumnTransformer(\n",
    "        transformers=[\n",
    "            (\"numerical\", numerical_preprocessor, numerical_columns),\n",
    "            (\"categorical\", categorical_preprocessor, [\"island\"]),\n",
    "        ]\n",
    "    )\n",
    "    \n",
    "\n",
    "    X = df.drop([\"sex\"], axis=1)\n",
    "    columns = list(X.columns)\n",
    "    \n",
    "    X = X.to_numpy()\n",
    "    \n",
    "    np.random.shuffle(X)\n",
    "    train, validation, test = np.split(X, [int(.7 * len(X)), int(.85 * len(X))])\n",
    "    \n",
    "    X_train = pd.DataFrame(train, columns=columns)\n",
    "    X_validation = pd.DataFrame(validation, columns=columns)\n",
    "    X_test = pd.DataFrame(test, columns=columns)\n",
    "    \n",
    "    y_train = X_train.species\n",
    "    y_validation = X_validation.species\n",
    "    y_test = X_test.species\n",
    "    \n",
    "    label_encoder = LabelEncoder()\n",
    "    \n",
    "    y_train = label_encoder.fit_transform(y_train)\n",
    "    y_validation = label_encoder.transform(y_validation)\n",
    "    y_test = label_encoder.transform(y_test)\n",
    "    \n",
    "    X_train.drop([\"species\"], axis=1, inplace=True)\n",
    "    X_validation.drop([\"species\"], axis=1, inplace=True)\n",
    "    X_test.drop([\"species\"], axis=1, inplace=True)\n",
    "\n",
    "    # Let's use the train set to generate a baseline that we can \n",
    "    # later use to measure the quality of our model. This baseline\n",
    "    # will use the original data.\n",
    "    generate_baseline(base_dir, X_train, y_train)\n",
    "    \n",
    "    # Transform the data using the Scikit-Learn pipeline.\n",
    "    X_train = preprocessor.fit_transform(X_train)\n",
    "    X_validation = preprocessor.transform(X_validation)\n",
    "    X_test = preprocessor.transform(X_test)\n",
    "    \n",
    "    train = np.concatenate((X_train, np.expand_dims(y_train, axis=1)), axis=1)\n",
    "    validation = np.concatenate((X_validation, np.expand_dims(y_validation, axis=1)), axis=1)\n",
    "    test = np.concatenate((X_test, np.expand_dims(y_test, axis=1)), axis=1)\n",
    "    \n",
    "    save_splits(base_dir, train, validation, test)\n",
    "    save_pipeline(base_dir, pipeline=preprocessor)\n",
    "        \n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    preprocess(BASE_DIR, DATA_FILEPATH)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "214fbdcb-8ee8-4104-bcd3-d7329693299e",
   "metadata": {},
   "source": [
    "## Step 4 - Testing the Preprocessing Script\n",
    "\n",
    "We can now load the script we just created and run it locally to ensure it outputs every file we need.\n",
    "\n",
    "We will set up a SageMaker Processing Job to run this script, but we always want to test the code locally. In this case, we can call the `preprocess()` function with the local directory and the local copy of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d1f122a4-acff-4687-91b9-bfef13567d88",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Folders: ['baseline', 'train', 'validation', 'test', 'pipeline']\n",
      "\n",
      "Baseline:\n",
      "{\"island\":\"Dream\",\"culmen_length_mm\":53.5,\"culmen_depth_mm\":19.9,\"flipper_length_mm\":205.0,\"body_mass_g\":4500.0,\"groundtruth\":1}\n",
      "{\"island\":\"Torgersen\",\"culmen_length_mm\":38.8,\"culmen_depth_mm\":17.6,\"flipper_length_mm\":191.0,\"body_mass_g\":3275.0,\"groundtruth\":0}\n",
      "{\"island\":\"Torgersen\",\"culmen_length_mm\":38.9,\"culmen_depth_mm\":17.8,\"flipper_length_mm\":181.0,\"body_mass_g\":3625.0,\"groundtruth\":0}\n",
      "{\"island\":\"Biscoe\",\"culmen_length_mm\":50.4,\"culmen_depth_mm\":15.7,\"flipper_length_mm\":222.0,\"body_mass_g\":5750.0,\"groundtruth\":2}\n",
      "{\"island\":\"Dream\",\"culmen_length_mm\":36.3,\"culmen_depth_mm\":19.5,\"flipper_length_mm\":190.0,\"body_mass_g\":3800.0,\"groundtruth\":0}\n"
     ]
    }
   ],
   "source": [
    "from penguins.preprocessor import preprocess\n",
    "\n",
    "with tempfile.TemporaryDirectory() as directory:\n",
    "    preprocess(\n",
    "        base_dir=directory, \n",
    "        data_filepath=LOCAL_FILEPATH\n",
    "    )\n",
    "    \n",
    "    print(f\"Folders: {os.listdir(directory)}\")\n",
    "    \n",
    "    print()\n",
    "    print(\"Baseline:\")\n",
    "    \n",
    "    with open(Path(directory) / 'baseline' / 'baseline.json') as baseline:\n",
    "        lines = [next(baseline) for _ in range(5)]\n",
    "        \n",
    "    for l in lines:\n",
    "        print(l[:-1])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3938a1d-d5bd-4967-ad65-a945d5df1f61",
   "metadata": {},
   "source": [
    "## Step 5 - Pipeline Configuration\n",
    "\n",
    "When creating a SageMaker Pipeline, we can specify a list of parameters we can use on individual pipeline steps. To read more about these parameters, check [Pipeline Parameters](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-parameters.html).\n",
    "\n",
    "These are the parameters that we need right now:\n",
    "\n",
    "* `dataset_location`: This parameter represents the dataset's location in S3. We will use this parameter to indicate the SageMaker Processing Job where to find the dataset. The Processing Job will download the dataset from S3 and make it available on the instance running the script.\n",
    "* `preprocessor_destination`: We need to define the location where the SageMaker Processing Job will store the output. When it finishes, the Processing Job will copy the script's output to the S3 location specified by this parameter. By default, SageMaker uploads the output of a job to a custom location in S3, but unfortunately, if we rely on that functionality, we can't cache the Processing Step in the Pipeline.\n",
    "* `baseline_destination`: This parameter represents the location where we will store the baseline data. We will use this baseline data in Session 6 to compute general statistics about the model. This will help monitor the quality of the model results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "86de7edd-18b0-40d1-ac8e-0f3ef4469be6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset_location = ParameterString(\n",
    "    name=\"dataset_location\",\n",
    "    default_value=INPUT_DATA_URI,\n",
    ")\n",
    "\n",
    "preprocessor_destination = ParameterString(\n",
    "    name=\"preprocessor_destination\",\n",
    "    default_value=f\"{S3_FILEPATH}/preprocessing\",\n",
    ")\n",
    "\n",
    "baseline_destination = ParameterString(\n",
    "    name=\"baseline_destination\",\n",
    "    default_value=f\"{S3_FILEPATH}/baseline\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0b42440-b3c0-4ded-b578-52e21f5fbfdd",
   "metadata": {},
   "source": [
    "## Step 6 - Caching Pipeline Steps\n",
    "\n",
    "While building a pipeline, you only want to rerun every step if you expect a different result. To accomplish this, you can instruct SageMaker to reuse the result of a previous successful run of a pipeline step. You can find more information about this topic in [Caching Pipeline Steps](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-caching.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c7e0a332-e54e-4e57-97c7-9b54f4058738",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cache_config = CacheConfig(\n",
    "    enable_caching=True, \n",
    "    expire_after=\"15d\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9771e08-5560-492a-845e-f06988b6ae1b",
   "metadata": {},
   "source": [
    "## Step 7 - Setting up a Processing Step\n",
    "\n",
    "The first step we need in the pipeline is a [Processing Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing) to run the preprocessing script. This Processing Step will create a SageMaker Processing Job in the background, run the script, and upload the output to S3. You can use Processing Jobs to perform data preprocessing, post-processing, feature engineering, data validation, and model evaluation. Check the [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.ProcessingStep) SageMaker's SDK documentation for more information.\n",
    "\n",
    "A processor gives the Processing Step information about the hardware and software that SageMaker should use to launch the Processing Job. To run the script, we need access to Scikit-Learn, so we can use the [SKLearnProcessor](https://sagemaker.readthedocs.io/en/stable/frameworks/sklearn/sagemaker.sklearn.html#scikit-learn-processor) processor that comes out-of-the-box with the SageMaker's Python SDK. The [Data Processing with Framework Processors](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job-frameworks.html) page discusses other built-in processors you can use. The [Docker Registry Paths and Example Code](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html) page contains information about the available framework versions for each region.\n",
    "\n",
    "We'll see how to use a custom processor in a later session. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4b982bc3-28f0-473f-b8d2-fc35dd251e37",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sklearn_processor = SKLearnProcessor(\n",
    "    base_job_name=\"penguins-preprocessing\",\n",
    "    framework_version=\"0.23-1\",\n",
    "    instance_type=\"ml.t3.medium\",\n",
    "    instance_count=1,\n",
    "    role=role,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57fd8052-dcab-4af1-8eca-5fdf9473f9a0",
   "metadata": {},
   "source": [
    "Here is the [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.ProcessingStep). Notice how it uses the processor that we defined above.\n",
    "\n",
    "We also need to define the list of inputs that we need on the preprocessing script. In this case, the input is the dataset we stored in S3. \n",
    "\n",
    "We have a few outputs that we want SageMaker to capture when the Processing Job finishes. SageMaker will upload every one of these outputs to the location specified by the `preprocessor_destination` parameter except the baseline data, which we will upload to the location specified by the `baseline_destination` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2187b65e-e504-40a5-ad05-82f54885805c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "preprocess_step = ProcessingStep(\n",
    "    name=\"preprocessing\",\n",
    "    processor=sklearn_processor,\n",
    "    inputs=[\n",
    "        ProcessingInput(source=dataset_location, destination=\"/opt/ml/processing/input\"),  \n",
    "    ],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name=\"train\", source=\"/opt/ml/processing/train\", destination=preprocessor_destination),\n",
    "        ProcessingOutput(output_name=\"validation\", source=\"/opt/ml/processing/validation\", destination=preprocessor_destination),\n",
    "        ProcessingOutput(output_name=\"test\", source=\"/opt/ml/processing/test\", destination=preprocessor_destination),\n",
    "        ProcessingOutput(output_name=\"pipeline\", source=\"/opt/ml/processing/pipeline\", destination=preprocessor_destination),\n",
    "        ProcessingOutput(output_name=\"baseline\", source=\"/opt/ml/processing/baseline\", destination=baseline_destination),\n",
    "    ],\n",
    "    code=f\"{PENGUINS_FOLDER}/preprocessor.py\",\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1465690a-67b5-4d68-a026-dd874ee569e8",
   "metadata": {},
   "source": [
    "## Step 8 - Running the Pipeline\n",
    "\n",
    "Let's define and run the SageMaker Pipeline. Check [Pipeline Structure and Execution](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-pipeline.html) for more information about how to define a pipeline and [Run a Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/run-pipeline.html) for information about how to run it.\n",
    "\n",
    "The pipeline uses the parameters we defined before and the Preprocess Step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "564fe6fb-00aa-480f-a58f-3144b2e33a9d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session1_pipeline = Pipeline(\n",
    "    name=\"penguins-session1-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        baseline_destination,\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_step, \n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e16d0ac8-1439-4329-9a1c-c43eafacd0da",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f4410f2f-a17b-4272-abd2-e092e134174c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session1_pipeline.upsert(role_arn=role)\n",
    "execution = session1_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63c190c5-52b5-4ccc-8d42-847a694b8e66",
   "metadata": {},
   "source": [
    "# Session 2 - Model Training and Tuning\n",
    "\n",
    "This session extends the [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) we built in the previous session with a step to train a model. We'll explore the [Training](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-training) and the [Tuning](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-tuning) steps. \n",
    "\n",
    "Here is what the Pipeline will look like at the end of this session:\n",
    "\n",
    "<img src='penguins/images/session2-pipeline.png' alt='Penguins dataset' width=\"600\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4756a3d8-d47b-4a88-a7f2-65ed7d4c1175",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.tuner import HyperparameterTuner\n",
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.workflow.steps import TuningStep\n",
    "from sagemaker.parameter import IntegerParameter\n",
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.tensorflow import TensorFlow\n",
    "from sagemaker.workflow.steps import TrainingStep\n",
    "from sagemaker.workflow.pipeline_context import PipelineSession"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8608092-7aab-4fd2-aa99-47c2db27bdb7",
   "metadata": {},
   "source": [
    "## Step 1 - Training the Model\n",
    "\n",
    "This script is responsible for training a simple neural network on the train data, validating the model, and saving it so we can later use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d92b121d-dcb9-43e8-9ee3-3ececb583e7e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/train.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {PENGUINS_FOLDER}/train.py\n",
    "\n",
    "import os\n",
    "import argparse\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "\n",
    "from pathlib import Path\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.optimizers import SGD\n",
    "\n",
    "\n",
    "def train(base_directory, train_path, validation_path, epochs=50, batch_size=32):\n",
    "    X_train = pd.read_csv(Path(train_path) / \"train.csv\")\n",
    "    y_train = X_train[X_train.columns[-1]]\n",
    "    X_train.drop(X_train.columns[-1], axis=1, inplace=True)\n",
    "    \n",
    "    X_validation = pd.read_csv(Path(validation_path) / \"validation.csv\")\n",
    "    y_validation = X_validation[X_validation.columns[-1]]\n",
    "    X_validation.drop(X_validation.columns[-1], axis=1, inplace=True)\n",
    "    \n",
    "    model = Sequential([\n",
    "        Dense(10, input_shape=(X_train.shape[1],), activation=\"relu\"),\n",
    "        Dense(8, activation=\"relu\"),\n",
    "        Dense(3, activation=\"softmax\"),\n",
    "    ])\n",
    "\n",
    "    model.compile(\n",
    "        optimizer=SGD(learning_rate=0.01),\n",
    "        loss=\"sparse_categorical_crossentropy\",\n",
    "        metrics=[\"accuracy\"]\n",
    "    )\n",
    "\n",
    "    model.fit(\n",
    "        X_train, \n",
    "        y_train, \n",
    "        validation_data=(X_validation, y_validation),\n",
    "        epochs=epochs, \n",
    "        batch_size=batch_size,\n",
    "        verbose=2,\n",
    "    )\n",
    "\n",
    "    predictions = np.argmax(model.predict(X_validation), axis=-1)\n",
    "    print(f\"Validation accuracy: {accuracy_score(y_validation, predictions)}\")\n",
    "    \n",
    "    model_filepath = Path(base_directory) / \"model\" / \"001\"\n",
    "    model.save(model_filepath)\n",
    "    \n",
    "if __name__ == \"__main__\":\n",
    "    # Any hyperparameters provided by the training job are passed to the entry point\n",
    "    # as script arguments. SageMaker will also provide a list of special parameters\n",
    "    # that you can capture here. Here is the full list: \n",
    "    # https://github.com/aws/sagemaker-training-toolkit/blob/master/src/sagemaker_training/params.py\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\"--base_directory\", type=str, default=\"/opt/ml/\")\n",
    "    parser.add_argument(\"--train_path\", type=str, default=os.environ.get(\"SM_CHANNEL_TRAIN\", None))\n",
    "    parser.add_argument(\"--validation_path\", type=str, default=os.environ.get(\"SM_CHANNEL_VALIDATION\", None))\n",
    "    parser.add_argument(\"--epochs\", type=int, default=50)\n",
    "    parser.add_argument(\"--batch_size\", type=int, default=32)\n",
    "    args, _ = parser.parse_known_args()\n",
    "    \n",
    "    train(\n",
    "        base_directory=args.base_directory,\n",
    "        train_path=args.train_path,\n",
    "        validation_path=args.validation_path,\n",
    "        epochs=args.epochs,\n",
    "        batch_size=args.batch_size\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50f0a4fa-ce70-4882-b9f5-8253df03d890",
   "metadata": {},
   "source": [
    "## Step 2 - Testing the Training Script\n",
    "\n",
    "Let's test the script we just created by running it locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "14ea27ce-c453-4cb0-b309-dbecd732957e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "Extension horovod.torch has not been built: /usr/local/lib/python3.8/site-packages/horovod/torch/mpi_lib/_mpi_lib.cpython-38-x86_64-linux-gnu.so not found\n",
      "If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error.\n",
      "Warning! MPI libs are missing, but python applications are still avaiable.\n",
      "[2023-04-24 22:28:05.967 tensorflow-2-6-cpu-py-ml-t3-medium-9169b2e75617c45c79c40579f6a8:69 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\n",
      "[2023-04-24 22:28:06.087 tensorflow-2-6-cpu-py-ml-t3-medium-9169b2e75617c45c79c40579f6a8:69 INFO profiler_config_parser.py:111] Unable to find config at /opt/ml/input/config/profilerconfig.json. Profiler is disabled.\n",
      "8/8 - 1s - loss: 1.3226 - accuracy: 0.3766 - val_loss: 1.2095 - val_accuracy: 0.3922\n",
      "Epoch 2/10\n",
      "8/8 - 0s - loss: 1.2035 - accuracy: 0.3808 - val_loss: 1.1249 - val_accuracy: 0.4314\n",
      "Epoch 3/10\n",
      "8/8 - 0s - loss: 1.1032 - accuracy: 0.4603 - val_loss: 1.0556 - val_accuracy: 0.5686\n",
      "Epoch 4/10\n",
      "8/8 - 0s - loss: 1.0222 - accuracy: 0.5649 - val_loss: 0.9954 - val_accuracy: 0.6471\n",
      "Epoch 5/10\n",
      "8/8 - 0s - loss: 0.9527 - accuracy: 0.6820 - val_loss: 0.9426 - val_accuracy: 0.6667\n",
      "Epoch 6/10\n",
      "8/8 - 0s - loss: 0.8941 - accuracy: 0.7657 - val_loss: 0.8973 - val_accuracy: 0.6863\n",
      "Epoch 7/10\n",
      "8/8 - 0s - loss: 0.8437 - accuracy: 0.7782 - val_loss: 0.8569 - val_accuracy: 0.7059\n",
      "Epoch 8/10\n",
      "8/8 - 0s - loss: 0.8000 - accuracy: 0.7992 - val_loss: 0.8206 - val_accuracy: 0.7647\n",
      "Epoch 9/10\n",
      "8/8 - 0s - loss: 0.7618 - accuracy: 0.8033 - val_loss: 0.7876 - val_accuracy: 0.7843\n",
      "Epoch 10/10\n",
      "8/8 - 0s - loss: 0.7275 - accuracy: 0.8117 - val_loss: 0.7575 - val_accuracy: 0.8039\n",
      "Validation accuracy: 0.803921568627451\n",
      "INFO:tensorflow:Assets written to: /tmp/tmp_6k2rdnb/model/001/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /tmp/tmp_6k2rdnb/model/001/assets\n"
     ]
    }
   ],
   "source": [
    "from penguins.preprocessor import preprocess\n",
    "from penguins.train import train\n",
    "\n",
    "\n",
    "with tempfile.TemporaryDirectory() as directory:\n",
    "    # First, we preprocess the data and create the \n",
    "    # dataset splits.\n",
    "    preprocess(\n",
    "        base_dir=directory, \n",
    "        data_filepath=LOCAL_FILEPATH\n",
    "    )\n",
    "\n",
    "    # Then, we train a model using the train and \n",
    "    # validation splits.\n",
    "    train(\n",
    "        base_directory=directory, \n",
    "        train_path=Path(directory) / \"train\", \n",
    "        validation_path=Path(directory) / \"validation\",\n",
    "        epochs=10\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf9c9362-ef17-4d68-b8c8-cefe21326ba2",
   "metadata": {},
   "source": [
    "## Step 3 - Switching Between Training and Tuning\n",
    "\n",
    "There are two ways we can create a model: Using a [Training Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-training) or using a [Tuning Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-tuning).\n",
    "\n",
    "In this notebook, we will alternate between both methods and use the `USE_TUNING_STEP` flag to indicate which approach we want to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c3df0f7a-e43a-4a7b-9809-0a639fa178d9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "USE_TUNING_STEP = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27cff4c1-6510-4d99-8ae1-cb14927b87c7",
   "metadata": {},
   "source": [
    "## Step 4 - Setting up a Training Step\n",
    "\n",
    "We can now create a [Training Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-training) that we can add to the pipeline. This Training Step will create a SageMaker Training Job in the background, run the training script, and upload the output to S3. Check the [TrainingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TrainingStep) SageMaker's SDK documentation for more information. \n",
    "\n",
    "SageMaker uses the concept of an [Estimator](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html) to handle end-to-end training and deployment tasks. For this example, we will use the built-in [TensorFlow Estimator](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-estimator) to run the training script we wrote before. The [Docker Registry Paths and Example Code](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html) page contains information about the available framework versions for each region. Here, you can also check the available SageMaker [Deep Learning Container images](https://github.com/aws/deep-learning-containers/blob/master/available_images.md).\n",
    "\n",
    "Notice the list of hyperparameters defined below. SageMaker will pass these hyperparameters as arguments to the entry point of the training script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "90fe82ae-6a2c-4461-bc83-bb52d8871e3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hyperparameters = {\n",
    "    \"epochs\": 50,\n",
    "    \"batch_size\": 32,\n",
    "}\n",
    "\n",
    "estimator = TensorFlow(\n",
    "    entry_point=f\"{PENGUINS_FOLDER}/train.py\",\n",
    "    hyperparameters=hyperparameters,\n",
    "    framework_version=\"2.6\",\n",
    "    py_version=\"py38\",\n",
    "    instance_type=\"ml.m5.large\",\n",
    "    instance_count=1,\n",
    "    script_mode=True,\n",
    "    disable_profiler=True,\n",
    "    role=role,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "545d2b43-3bb5-4fe9-b3e4-cb8eb55c8a21",
   "metadata": {},
   "source": [
    "We can now create the [TrainingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TrainingStep) using the estimator we defined before.\n",
    "\n",
    "This step will receive the train and validation split from the preprocessing step as inputs. Notice how we reference both splits using the `preprocess_step` variable. This creates a dependency between the Training and Processing Step we defined in Session 1. When we build a new Pipeline, we'll see that the Training Step will run once the Processing Step finishes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "99e4850c-83d6-4f4e-a813-d5a3f4bb7486",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "training_step = TrainingStep(\n",
    "    name=\"training\",\n",
    "    estimator=estimator,\n",
    "    inputs={\n",
    "        \"train\": TrainingInput(\n",
    "            s3_data=preprocess_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"train\"\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\"\n",
    "        ),\n",
    "        \"validation\": TrainingInput(\n",
    "            s3_data=preprocess_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"validation\"\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\"\n",
    "        )\n",
    "    },\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5814e258-c633-4e9a-85c5-6ed0f168b503",
   "metadata": {},
   "source": [
    "## Step 5 - Setting up a Tuning Step\n",
    "\n",
    "Let's now create a [Tuning Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-tuning) to add it to our pipeline. This Tuning Step will create a SageMaker Hyperparameter Tuning Job in the background and use the training script to train different model variants and choose the best one. Check the [TuningStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TuningStep) SageMaker's SDK documentation for more information.\n",
    "\n",
    "The Tuning Step requires a [HyperparameterTuner](https://sagemaker.readthedocs.io/en/stable/api/training/tuner.html) reference to configure the Hyperparameter Tuning Job. In this example, the tuner will use the same `Estimator` we defined to train the model.\n",
    "\n",
    "Here is the configuration that we'll use to find the best model:\n",
    "\n",
    "1. `objective_metric_name`: This is the name of the metric the tuner will use to determine the best model.\n",
    "2. `objective_type`: This is the objective of the tuner. Should it \"Minimize\" the metric or \"Maximize\" it? In this example, since we are using the validation accuracy of the model, we want the objective to be \"Maximize.\" If we were using the loss of the model, we would set the objective to \"Minimize.\"\n",
    "3. `metric_definitions`: Defines how the tuner will determine the metric's value by looking at the output logs of the training process.\n",
    "\n",
    "The tuner expects the list of the hyperparameters you want to explore. You can use subclasses of the [Parameter](https://sagemaker.readthedocs.io/en/stable/api/training/parameter.html#sagemaker.parameter.ParameterRange) class to specify different types of hyperparameters. This example explores different values for the `epochs` hyperparameter.\n",
    "\n",
    "Finally, you can control the number of jobs and how many of them will run in parallel using the following two arguments:\n",
    "\n",
    "* `max_jobs`: Defines the maximum total number of training jobs to start for the hyperparameter tuning job.\n",
    "* `max_parallel_jobs`: Defines the maximum number of parallel training jobs to start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "038ff2e5-ed28-445b-bc03-4e996ec2286f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hyperparameter_ranges = {\n",
    "    \"epochs\": IntegerParameter(10, 50),\n",
    "}\n",
    "\n",
    "objective_metric_name = \"val_accuracy\"\n",
    "objective_type = \"Maximize\"\n",
    "metric_definitions = [{\"Name\": objective_metric_name, \"Regex\": \"val_accuracy: ([0-9\\\\.]+)\"}]\n",
    "    \n",
    "tuner = HyperparameterTuner(\n",
    "    estimator,\n",
    "    objective_metric_name,\n",
    "    hyperparameter_ranges,\n",
    "    metric_definitions,\n",
    "    objective_type=objective_type,\n",
    "    max_jobs=3,\n",
    "    max_parallel_jobs=3,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59de0743-6c9e-4895-b060-f58a6d60c50d",
   "metadata": {},
   "source": [
    "We can now create the [TuningStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TuningStep). \n",
    "\n",
    "This step will use the tuner we configured before and will receive the train and validation split from the preprocessing step as inputs. Notice how we reference both splits using the `preprocess_step` variable. This creates a dependency between the Tuning and Processing Steps we defined in Session 1. When we build a new Pipeline, we'll see that the Tuning Step will run once the Processing Step finishes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "cf5dd9a7-8643-4fbb-8eb4-40f39011e27b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tuning_step = TuningStep(\n",
    "    name = \"tuning\",\n",
    "    tuner=tuner,\n",
    "    inputs={\n",
    "        \"train\": TrainingInput(\n",
    "            s3_data=preprocess_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"train\"\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\"\n",
    "        ),\n",
    "        \"validation\": TrainingInput(\n",
    "            s3_data=preprocess_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"validation\"\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\"\n",
    "        )\n",
    "    },\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4babe38c-1682-42d2-8442-101d17aa89b5",
   "metadata": {},
   "source": [
    "## Step 6 - Running the Pipeline\n",
    "\n",
    "We can now define and run the SageMaker Pipeline using the Training or Tuning Step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "9799ab39-fcae-41f4-a68b-85ab71b3ba9a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session2_pipeline = Pipeline(\n",
    "    name=\"penguins-session2-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        baseline_destination,\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_step, \n",
    "        tuning_step if USE_TUNING_STEP else training_step\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0e0708c-3692-4d09-a269-bbd14f7b0226",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "id": "77c6d523-6810-4bc7-8fc4-64ae978cf245",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    }
   ],
   "source": [
    "session2_pipeline.upsert(role_arn=role)\n",
    "execution = session2_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21d40fe8-ba74-4c12-9555-d8ea33d1c8b4",
   "metadata": {},
   "source": [
    "# Session 3 - Model Evaluation\n",
    "\n",
    "This session extends the [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) with a step to evaluate the model. We'll use a [Processing Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing) with a [ScriptProcessor](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.processing.ScriptProcessor) running TensorFlow to execute an evaluation script. \n",
    "\n",
    "Here is what the Pipeline will look like at the end of this session:\n",
    "\n",
    "<img src='penguins/images/session3-pipeline.png' alt='Penguins dataset' width=\"600\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "34fc262f-a1cf-4f94-9c60-e7c8e83cfdfb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tarfile\n",
    "\n",
    "from sagemaker.tensorflow import TensorFlowProcessor\n",
    "from sagemaker.workflow.properties import PropertyFile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eaa9691-f49f-48af-b272-3d4d17563b01",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Step 1 - Evaluating the Model\n",
    "\n",
    "This script is responsible for loading the model we created and evaluating it on the test set. Before finishing, this script will generate an evaluation report of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3ee3ab26-afa5-4ceb-9f7a-005d5fdea646",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/evaluation.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {PENGUINS_FOLDER}/evaluation.py\n",
    "\n",
    "import os\n",
    "import json\n",
    "import tarfile\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from pathlib import Path\n",
    "from tensorflow import keras\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "\n",
    "MODEL_PATH = \"/opt/ml/processing/model/\"\n",
    "TEST_PATH = \"/opt/ml/processing/test/\"\n",
    "OUTPUT_PATH = \"/opt/ml/processing/evaluation/\"\n",
    "\n",
    "\n",
    "def evaluate(model_path, test_path, output_path):\n",
    "    # The first step is to extract the model package provided\n",
    "    # by SageMaker.\n",
    "    with tarfile.open(Path(model_path) / \"model.tar.gz\") as tar:\n",
    "        tar.extractall(path=Path(model_path))\n",
    "        \n",
    "    # We can now load the model from disk.\n",
    "    model = keras.models.load_model(Path(model_path) / \"001\")\n",
    "    \n",
    "    X_test = pd.read_csv(Path(test_path) / \"test.csv\")\n",
    "    y_test = X_test[X_test.columns[-1]]\n",
    "    X_test.drop(X_test.columns[-1], axis=1, inplace=True)\n",
    "    \n",
    "    predictions = np.argmax(model.predict(X_test), axis=-1)\n",
    "    accuracy = accuracy_score(y_test, predictions)\n",
    "    print(f\"Test accuracy: {accuracy}\")\n",
    "\n",
    "    # Let's add the accuracy of the model to our evaluation report.\n",
    "    evaluation_report = {\n",
    "        \"metrics\": {\n",
    "            \"accuracy\": {\n",
    "                \"value\": accuracy\n",
    "            },\n",
    "        },\n",
    "    }\n",
    "    \n",
    "    # We need to save the evaluation report to the output path.\n",
    "    Path(output_path).mkdir(parents=True, exist_ok=True)\n",
    "    with open(Path(output_path) / \"evaluation.json\", \"w\") as f:\n",
    "        f.write(json.dumps(evaluation_report))\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    evaluate(\n",
    "        model_path=MODEL_PATH, \n",
    "        test_path=TEST_PATH,\n",
    "        output_path=OUTPUT_PATH\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dcc79a0-adfd-4ce9-8580-5cd228c3c2d9",
   "metadata": {},
   "source": [
    "## Step 2 - Testing the Evaluation Script\n",
    "\n",
    "Let's test the script we just created by running it locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9a2540d8-278a-4953-bc54-0469d154427d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "8/8 - 1s - loss: 1.1982 - accuracy: 0.2678 - val_loss: 1.1541 - val_accuracy: 0.3333\n",
      "Epoch 2/10\n",
      "8/8 - 0s - loss: 1.1323 - accuracy: 0.2720 - val_loss: 1.1049 - val_accuracy: 0.3333\n",
      "Epoch 3/10\n",
      "8/8 - 0s - loss: 1.0832 - accuracy: 0.3138 - val_loss: 1.0676 - val_accuracy: 0.3529\n",
      "Epoch 4/10\n",
      "8/8 - 0s - loss: 1.0483 - accuracy: 0.4017 - val_loss: 1.0385 - val_accuracy: 0.4314\n",
      "Epoch 5/10\n",
      "8/8 - 0s - loss: 1.0218 - accuracy: 0.4979 - val_loss: 1.0145 - val_accuracy: 0.5098\n",
      "Epoch 6/10\n",
      "8/8 - 0s - loss: 1.0007 - accuracy: 0.5690 - val_loss: 0.9953 - val_accuracy: 0.5294\n",
      "Epoch 7/10\n",
      "8/8 - 0s - loss: 0.9824 - accuracy: 0.5983 - val_loss: 0.9771 - val_accuracy: 0.5294\n",
      "Epoch 8/10\n",
      "8/8 - 0s - loss: 0.9646 - accuracy: 0.6109 - val_loss: 0.9598 - val_accuracy: 0.5882\n",
      "Epoch 9/10\n",
      "8/8 - 0s - loss: 0.9473 - accuracy: 0.6234 - val_loss: 0.9414 - val_accuracy: 0.6078\n",
      "Epoch 10/10\n",
      "8/8 - 0s - loss: 0.9301 - accuracy: 0.6360 - val_loss: 0.9220 - val_accuracy: 0.6078\n",
      "Validation accuracy: 0.6078431372549019\n",
      "INFO:tensorflow:Assets written to: /tmp/tmp5ikpry6o/model/001/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /tmp/tmp5ikpry6o/model/001/assets\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.6470588235294118\n"
     ]
    }
   ],
   "source": [
    "from penguins.preprocessor import preprocess\n",
    "from penguins.train import train\n",
    "from penguins.evaluation import evaluate\n",
    "\n",
    "\n",
    "with tempfile.TemporaryDirectory() as directory:\n",
    "    # First, we preprocess the data and create the \n",
    "    # dataset splits.\n",
    "    preprocess(\n",
    "        base_dir=directory, \n",
    "        data_filepath=LOCAL_FILEPATH\n",
    "    )\n",
    "\n",
    "    # Then, we train a model using the train and \n",
    "    # validation splits.\n",
    "    train(\n",
    "        base_directory=directory, \n",
    "        train_path=Path(directory) / \"train\", \n",
    "        validation_path=Path(directory) / \"validation\",\n",
    "        epochs=10\n",
    "    )\n",
    "    \n",
    "    # After training a model, we need to prepare a package just like\n",
    "    # SageMaker would. This package is what the evaluation script is\n",
    "    # expecting as an input.\n",
    "    with tarfile.open(Path(directory) / \"model.tar.gz\", \"w:gz\") as tar:\n",
    "        tar.add(Path(directory) / \"model\" / \"001\", arcname=\"001\")\n",
    "        \n",
    "    \n",
    "    # We can now call the evaluation script.\n",
    "    evaluate(\n",
    "        model_path=directory, \n",
    "        test_path=Path(directory) / \"test\",\n",
    "        output_path=Path(directory) / \"evaluation\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0e0c3a1-70be-4348-8f22-6a3c443663f5",
   "metadata": {},
   "source": [
    "## Step 3 - Pipeline Configuration\n",
    "\n",
    "We need to define a new Pipeline parameter with the location where the Processing Step running the evaluation script will store the report. As we did with the Processing Step running the preprocessing script, we want to prevent SageMaker from appending a timestamp to their auto-generated location. We can't cache that step if we let SageMaker use a timestamp. That's the goal of the `evaluation_destination` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "3125f743-45c3-461b-be0c-bed8b3a4cb77",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "evaluation_destination = ParameterString(\n",
    "    name=\"evaluation_destination\",\n",
    "    default_value=f'{S3_FILEPATH}/evaluation',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "971d0b30-edf1-4b05-a570-e4f1d5a312e1",
   "metadata": {},
   "source": [
    "## Step 4 - Setting up a Processor\n",
    "\n",
    "To run the evaluation script, we can use a [Processing Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing). Check the [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.ProcessingStep) SageMaker's SDK documentation for more information.\n",
    "\n",
    "Whenever you want to run a Processing Job using a machine learning framework, you can use an instance of the [FrameworkProcessor](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.processing.FrameworkProcessor) class. For example, the [TensorFlowProcessor](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job-frameworks-tensorflow.html) subclass will give you access to TensorFlow. You can also configure a Processing Job from scratch using a [ScriptProcessor](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.processing.ScriptProcessor) instance combined with the [sagemaker.image_uris.retrieve()](https://sagemaker.readthedocs.io/en/stable/api/utility/image_uris.html) function for generating the URI of one of the  SageMaker pre-built docker images. \n",
    "\n",
    "This time, we will use a [TensorFlowProcessor](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job-frameworks-tensorflow.html) because we need our script to have access to TensorFlow and Scikit-Learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "14efc051-fcdd-4cfc-8751-26fae45fc6c5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    }
   ],
   "source": [
    "tensorflow_processor = TensorFlowProcessor(\n",
    "    framework_version=\"2.6\",\n",
    "    py_version=\"py38\",\n",
    "    base_job_name=\"penguins-evaluation-processor\",\n",
    "    instance_type=\"ml.m5.large\",\n",
    "    instance_count=1,\n",
    "    role=role\n",
    ")\n",
    "\n",
    "# By default, the TensorFlowProcessor runs the script using\n",
    "# /bin/bash as its entrypoint. We want to ensure we run it \n",
    "# using python3.\n",
    "tensorflow_processor.framework_entrypoint_command = [\"python3\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "188ba1cc-1bed-4b78-ba3c-842945bb26ea",
   "metadata": {},
   "source": [
    "## Step 5 - Configuring the Model Input\n",
    "\n",
    "The model we created is one of the inputs we need to provide to the Processing Step that runs the evaluation script. Currently, we create a model using either a Training Step or a Tuning Step, so we can use the `USE_TUNING_STEP` flag to configure the input to the Processing Step.\n",
    "\n",
    "In case we are using the Tuning Step, we can use the [TuningStep.get_top_model_s3_uri()](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TuningStep.get_top_model_s3_uri) function to get the model artifacts from the top performing training job of the Hyperparameter Tuning Job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ffcebf04-53af-45e1-8f40-cc205cce8d72",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This is the input in case we want to use the best model generated\n",
    "# by the Tuning Step.\n",
    "tuning_model_input = ProcessingInput(\n",
    "    source=tuning_step.get_top_model_s3_uri(\n",
    "        top_k=0, \n",
    "        s3_bucket=sagemaker_session.default_bucket()\n",
    "    ),\n",
    "    destination=\"/opt/ml/processing/model\",\n",
    ")\n",
    "\n",
    "# This is the input in case we want to use the trained model\n",
    "# from the Training Step.\n",
    "training_model_input = ProcessingInput(\n",
    "    source=training_step.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "    destination=\"/opt/ml/processing/model\"\n",
    ")\n",
    "\n",
    "# We can now select the appropriate input depending on which step\n",
    "# we are using.\n",
    "model_input = tuning_model_input if USE_TUNING_STEP else training_model_input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bec1109a-6c26-4464-8338-94960729d212",
   "metadata": {},
   "source": [
    "## Step 6 - Setting up a Processing Step\n",
    "\n",
    "We can now create a [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.ProcessingStep) to run the evaluation script. \n",
    "\n",
    "The inputs of this step will be the model we created and the test set we generated during the preprocessing phase. The output will be the evaluation report file.\n",
    "\n",
    "The [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.ProcessingStep) lets us specify a list of [PropertyFile](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.properties.PropertyFile) instances from the output of the job. We can use this to map the evaluation report generated in the evaluation script. Check [How to Build and Manage Property Files](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-propertyfile.html) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "48139a07-5c8e-4bc6-b666-bf9531f7f520",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We want to map the evaluation report that we generate inside\n",
    "# the evaluation script so we can later reference it.\n",
    "evaluation_report = PropertyFile(\n",
    "    name=\"evaluation-report\",\n",
    "    output_name=\"evaluation\",\n",
    "    path=\"evaluation.json\"\n",
    ")\n",
    "\n",
    "\n",
    "# Notice how this step uses the model generated by the tuning or training\n",
    "# step, and the test set generated by the preprocessing step.\n",
    "evaluation_step = ProcessingStep(\n",
    "    name=\"evaluation\",\n",
    "    processor=tensorflow_processor,\n",
    "    inputs=[\n",
    "        model_input,\n",
    "        ProcessingInput(\n",
    "            source=preprocess_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"test\"\n",
    "            ].S3Output.S3Uri,\n",
    "            destination=\"/opt/ml/processing/test\"\n",
    "        )\n",
    "    ],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name=\"evaluation\", source=\"/opt/ml/processing/evaluation\", destination=evaluation_destination),\n",
    "    ],\n",
    "    code=f\"{PENGUINS_FOLDER}/evaluation.py\",\n",
    "    property_files=[evaluation_report],\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "196f8cb8-d284-425b-8c4a-f644b4ac7ebe",
   "metadata": {},
   "source": [
    "## Step 7 - Running the Pipeline\n",
    "\n",
    "We can now add the model evaluation step to the pipeline.\n",
    "\n",
    "We will configure the pipeline to run the Tuning Step or the Training Step, depending on the value of the `USE_TUNING_STEP` flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "32638959-642d-4068-8fbc-3355c618115b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session3_pipeline = Pipeline(\n",
    "    name=\"penguins-session3-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        evaluation_destination,\n",
    "        baseline_destination,\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_step, \n",
    "        tuning_step if USE_TUNING_STEP else training_step,\n",
    "        evaluation_step\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04e98fb1-fdff-4ba4-aea3-0d9f271fb24c",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "4864ab08-7acc-4719-970e-fcce66c5fef5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    }
   ],
   "source": [
    "session3_pipeline.upsert(role_arn=role)\n",
    "execution = session3_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2917ee33-f895-45e6-83d3-e34255ba550b",
   "metadata": {},
   "source": [
    "# Session 4 - Model Registration\n",
    "\n",
    "This session extends the [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) with a step to register a new model if it reaches a predefined accuracy threshold. We'll use a [Condition Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-condition) to determine whether the model's accuracy is above a threshold and a [Model Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-model) to register the model. After we register the model, we'll deploy it manually. To learn more about the Model Registry, check [Register and Deploy Models with Model Registry](https://docs.aws.amazon.com/sagemaker/latest/dg/model-registry.html).\n",
    "\n",
    "Here is what the Pipeline will look like at the end of this session:\n",
    "\n",
    "<img src='penguins/images/session4-pipeline.png' alt='Pipeline' width=\"600\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "6545d1a3-c1ec-4eae-92fb-5b11d8f1dbf4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "from sagemaker import ModelPackage\n",
    "from sagemaker.model import Model\n",
    "from sagemaker.tensorflow.model import TensorFlowModel\n",
    "from sagemaker.model_metrics import MetricsSource, ModelMetrics \n",
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo\n",
    "from sagemaker.workflow.condition_step import ConditionStep\n",
    "from sagemaker.workflow.fail_step import FailStep\n",
    "from sagemaker.workflow.functions import JsonGet\n",
    "from sagemaker.workflow.functions import Join"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8696f63c-c036-46be-b352-4e23b5db0553",
   "metadata": {},
   "source": [
    "## Step 1 - Approval and Threshold Configuration\n",
    "\n",
    "We are going to use two new [Pipeline Parameters](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-parameters.html) in our pipeline:\n",
    "\n",
    "* `model_approval_status`: This parameter represents the default approval status we will use when registering a new model. Check [Update the Approval Status of a Model](https://docs.aws.amazon.com/sagemaker/latest/dg/model-registry-approve.html) for more information about the different approval status of a model and how you can update them.\n",
    "* `accuracy_threshold`: This parameter represents the minimum accuracy that the model should reach for it to be registered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "3f28a689-c5ca-4373-b615-2adba4f2664a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_approval_status = ParameterString(\n",
    "    name=\"model_approval_status\", \n",
    "    default_value=\"Approved\"\n",
    ")\n",
    "\n",
    "accuracy_threshold = ParameterFloat(\n",
    "    name=\"accuracy_threshold\", \n",
    "    default_value=0.70\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f96fbc5-0647-49f1-817f-85c5477e91ab",
   "metadata": {},
   "source": [
    "## Step 2 - Configuring the Model Assets\n",
    "\n",
    "We need to specify the location of the model assets to register a model. Currently, we create a model using either a Training Step or a Tuning Step, so we can use the `USE_TUNING_STEP` flag to configure the model.\n",
    "\n",
    "In case we are using the Tuning Step, we can use the [TuningStep.get_top_model_s3_uri()](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TuningStep.get_top_model_s3_uri) function to get the model artifacts from the top performing training job of the Hyperparameter Tuning Job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "2d6e6241-ebf7-4a02-b9eb-f30639b70f8f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This is the model data in case we want to use the best model generated\n",
    "# by the Tuning Step.\n",
    "tuning_model_data = tuning_step.get_top_model_s3_uri(\n",
    "    top_k=0, \n",
    "    s3_bucket=sagemaker_session.default_bucket()\n",
    ")\n",
    "\n",
    "# This is the model data in case we want to use the trained model\n",
    "# from the Training Step.\n",
    "training_model_data = training_step.properties.ModelArtifacts.S3ModelArtifacts\n",
    "\n",
    "# We can now select the appropriate model data depending on which step\n",
    "# we are using.\n",
    "model_data = tuning_model_data if USE_TUNING_STEP else training_model_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e705514-5931-4f36-9633-776c2305bf9a",
   "metadata": {},
   "source": [
    "## Step 3 - Configuring the Model\n",
    "\n",
    "The model we trained uses TensorFlow, so we can use the built-in [TensorFlowModel](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-serving-model) class to create an instance of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "0687fa7f-24d8-4c67-9378-1a4bdc9a9ccc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model = TensorFlowModel(\n",
    "    model_data=model_data,\n",
    "    framework_version=\"2.6\",\n",
    "    sagemaker_session=PipelineSession(),\n",
    "    role=role,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "789a48e6-507d-42fb-8abc-ebde997e3d43",
   "metadata": {},
   "source": [
    "## Step 4 - Setting up the Model Metrics\n",
    "\n",
    "When we register a model, we can specify a set of [ModelMetrics](https://sagemaker.readthedocs.io/en/stable/api/inference/model_monitor.html#sagemaker.model_metrics.ModelMetrics). We can use the evaluation report we generated during the \n",
    "Evaluation step to populate these statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "4982146f-0c0f-4938-b5d0-06db45a58531",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_metrics = ModelMetrics(\n",
    "    model_statistics=MetricsSource(\n",
    "        s3_uri=Join(on=\"/\", values=[\n",
    "            evaluation_step.arguments['ProcessingOutputConfig']['Outputs'][0]['S3Output']['S3Uri'],\n",
    "            \"evaluation.json\"]\n",
    "        ),\n",
    "        content_type=\"application/json\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "441fac3a-5ecc-441f-84c3-717c4c7ba290",
   "metadata": {},
   "source": [
    "## Step 5 - Setting up a Model Step\n",
    "\n",
    "We can now create a [Model Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-model) to register the model. Check the [ModelStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.model_step.ModelStep) SageMaker's SDK documentation for more information. We aim to create a new version of the model and register it in the Model Registry. Check [Register a Model Version](https://docs.aws.amazon.com/sagemaker/latest/dg/model-registry-version.html) for more information about model registration.\n",
    "\n",
    "This step will use the `Model` instance we configured before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "52a48cef-fb78-412b-a5c6-977eafe98e27",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.tensorflow.model:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    }
   ],
   "source": [
    "model_package_group_name = \"penguins-model-package-group\"\n",
    "\n",
    "register_model_step = ModelStep(\n",
    "    name=\"register-model\",\n",
    "    step_args=model.register(\n",
    "        model_package_group_name=model_package_group_name,\n",
    "        model_metrics=model_metrics,\n",
    "        approval_status=model_approval_status,\n",
    "        content_types=[\"text/csv\"],\n",
    "        response_types=[\"text/csv\"],\n",
    "        inference_instances=[\"ml.m5.large\"],\n",
    "        transform_instances=[\"ml.m5.large\"],\n",
    "        domain=\"MACHINE_LEARNING\",\n",
    "        task=\"CLASSIFICATION\",\n",
    "        framework=\"TENSORFLOW\",\n",
    "        framework_version=\"2.6\",\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c110f7-fe72-4db8-9d06-cfb9a0f2bfbd",
   "metadata": {},
   "source": [
    "## Step 6 - Setting up a Condition Step\n",
    "\n",
    "We only want to register a new model if its accuracy exceeds a predefined threshold. We can use a [Condition Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-condition) together with the evaluation report we generated in the Evaluation step to accomplish this. Check the [ConditionStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#conditionstep) SageMaker's SDK documentation for more information.\n",
    "\n",
    "In this example, we will use a [ConditionGreaterThanOrEqualTo](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.conditions.ConditionGreaterThanOrEqualTo) condition to compare the model's accuracy with the threshold. Look at the [Conditions](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_building_pipeline.html#conditions) section in the documentation for more information about the types of supported conditions.\n",
    "\n",
    "If the model's accuracy is not greater than or equal our threshold, we will send the pipeline to a [Fail Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-fail) with the appropriate error message. Check the [FailStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.fail_step.FailStep) SageMaker's SDK documentation for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "36e2a2b1-6711-4266-95d8-d2aebd52e199",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "condition_gte = ConditionGreaterThanOrEqualTo(\n",
    "    left=JsonGet(\n",
    "        step_name=evaluation_step.name,\n",
    "        property_file=evaluation_report,\n",
    "        json_path=\"metrics.accuracy.value\"\n",
    "    ),\n",
    "    right=accuracy_threshold\n",
    ")\n",
    "\n",
    "fail_step = FailStep(\n",
    "    name=\"fail\",\n",
    "    error_message=Join(\n",
    "        on=\" \", \n",
    "        values=[\n",
    "            \"Execution failed because the model's accuracy was lower than\", \n",
    "            accuracy_threshold\n",
    "        ]\n",
    "    ),\n",
    ")\n",
    "\n",
    "condition_step = ConditionStep(\n",
    "    name=\"check-model-accuracy\",\n",
    "    conditions=[condition_gte],\n",
    "    if_steps=[register_model_step],\n",
    "    else_steps=[fail_step], \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2309b8fa-f03e-4959-853f-dc2416f82bdd",
   "metadata": {},
   "source": [
    "## Step 7 - Running the Pipeline\n",
    "\n",
    "We can now add the registration of the model to the pipeline. Notice how we add the Condition Step, which will call the Model Step if the condition passes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "f70bcd33-b499-4e2b-953e-94d1ed96c10a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session4_pipeline = Pipeline(\n",
    "    name=\"penguins-session4-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        baseline_destination,\n",
    "        evaluation_destination,\n",
    "        model_approval_status,\n",
    "        accuracy_threshold,\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_step, \n",
    "        tuning_step if USE_TUNING_STEP else training_step, \n",
    "        evaluation_step,\n",
    "        condition_step\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7a938a4-7f3a-401b-a654-12e0ec1726e7",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "45b01065-5ba4-4c5c-9735-859b5234614a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker.workflow._utils:Popping out 'CertifyForMarketplace' from the pipeline definition since it will be overridden in pipeline execution time.\n",
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    }
   ],
   "source": [
    "session4_pipeline.upsert(role_arn=role)\n",
    "execution = session4_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1502a7e9-e6fa-4706-8294-7abd51f85d07",
   "metadata": {},
   "source": [
    "## Step 8 - Loading the Latest Approved Model\n",
    "\n",
    "Now that we registered the model, we can load the latest approved model from the Model Registry to deploy it to an endpoint.\n",
    "\n",
    "We can use `boto3` to query the list of approved models and get the latest one. Check the [boto3 SageMaker Client API](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html) for a list of every available method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "ff299800-7509-48b4-a343-e18ac4a30a3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_latest_approved_model_package(model_package_group_name):\n",
    "    \"\"\"\n",
    "    Returns the latest approved model package registered under the \n",
    "    specified model package group.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        # We can use the boto3 SageMaker's API to list the existing\n",
    "        # model packages with the specified name. We only care about\n",
    "        # approved models.\n",
    "        response = sagemaker_client.list_model_packages(\n",
    "            ModelPackageGroupName=model_package_group_name,\n",
    "            ModelApprovalStatus=\"Approved\",\n",
    "            SortBy=\"CreationTime\",\n",
    "            MaxResults=100,\n",
    "        )\n",
    "        approved_packages = response[\"ModelPackageSummaryList\"]\n",
    "\n",
    "        # If we get a NextToken back, we need to deal with pagination.\n",
    "        while len(approved_packages) == 0 and \"NextToken\" in response:\n",
    "            response = sagemaker_client.list_model_packages(\n",
    "                ModelPackageGroupName=model_package_group_name,\n",
    "                ModelApprovalStatus=\"Approved\",\n",
    "                SortBy=\"CreationTime\",\n",
    "                MaxResults=100,\n",
    "                NextToken=response[\"NextToken\"],\n",
    "            )\n",
    "            approved_packages.extend(response[\"ModelPackageSummaryList\"])\n",
    "\n",
    "        if len(approved_packages) == 0:\n",
    "            print(f\"No approved model pacakages for \\\"{model_package_group_name}\\\"\")\n",
    "            return None\n",
    "\n",
    "        # At this point we identified the latest approved model,\n",
    "        # so we can return it.\n",
    "        print(f\"Latest approved model package: {approved_packages[0]['ModelPackageArn']}\")\n",
    "        return approved_packages[0]\n",
    "\n",
    "    except ClientError as e:\n",
    "        print(e.response[\"Error\"][\"Message\"])\n",
    "        raise Exception(e.response[\"Error\"][\"Message\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3380addd-bb8c-4812-8120-fd97d13a7a4b",
   "metadata": {},
   "source": [
    "We can now use the `get_latest_approved_model_package()` function to get the latest approved model from the Model Registry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "cc6b8c80-aba9-40ce-99c8-ea7abe5f6f99",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Latest approved model package: arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins-model-package-group/10\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'ModelPackageGroupName': 'penguins-model-package-group',\n",
       " 'ModelPackageVersion': 10,\n",
       " 'ModelPackageArn': 'arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins-model-package-group/10',\n",
       " 'CreationTime': datetime.datetime(2023, 4, 24, 17, 30, 27, 135000, tzinfo=tzlocal()),\n",
       " 'InferenceSpecification': {'Containers': [{'Image': '325223348818.dkr.ecr.us-east-1.amazonaws.com/penguins:latest',\n",
       "    'ImageDigest': 'sha256:5cdafa465bf301b4b083eec1696ced447d5adc126a7bbc1afc0bf37011e07b1f',\n",
       "    'ModelDataUrl': 's3://sagemaker-us-east-1-325223348818/penguins-2023-04-24-17-02-15-960/pipelines-ud0y90sg2m0e-register-custom-mode-4mqfqfOQdh/output/model.tar.gz',\n",
       "    'Environment': {'PIPELINE_S3_LOCATION': 's3://mlschool/penguins/preprocessing',\n",
       "     'SAGEMAKER_CONTAINER_LOG_LEVEL': '20',\n",
       "     'SAGEMAKER_PROGRAM': 'inference.py',\n",
       "     'SAGEMAKER_REGION': 'us-east-1',\n",
       "     'SAGEMAKER_SUBMIT_DIRECTORY': '/opt/ml/model/code'},\n",
       "    'Framework': 'TENSORFLOW',\n",
       "    'FrameworkVersion': '2.6'}],\n",
       "  'SupportedRealtimeInferenceInstanceTypes': ['ml.m5.large'],\n",
       "  'SupportedContentTypes': ['application/json'],\n",
       "  'SupportedResponseMIMETypes': ['application/json']},\n",
       " 'ModelPackageStatus': 'Completed',\n",
       " 'ModelPackageStatusDetails': {'ValidationStatuses': [],\n",
       "  'ImageScanStatuses': []},\n",
       " 'CertifyForMarketplace': False,\n",
       " 'ModelApprovalStatus': 'Approved',\n",
       " 'CreatedBy': {'IamIdentity': {'Arn': 'arn:aws:sts::325223348818:assumed-role/AmazonSageMaker-ExecutionRole-20230312T160501/sagemaker-pipeline-ud0y90sg2m0e-register-custom-mode',\n",
       "   'PrincipalId': 'AROAUXOGSPJJGTQ2RIUND:sagemaker-pipeline-ud0y90sg2m0e-register-custom-mode'}},\n",
       " 'MetadataProperties': {'GeneratedBy': 'arn:aws:sagemaker:us-east-1:325223348818:pipeline/penguins-session5-pipeline/execution/ud0y90sg2m0e'},\n",
       " 'ModelMetrics': {'ModelQuality': {'Statistics': {'ContentType': 'application/json',\n",
       "    'S3Uri': 's3://mlschool/penguins/evaluation/evaluation.json'}},\n",
       "  'Bias': {},\n",
       "  'Explainability': {}},\n",
       " 'Domain': 'MACHINE_LEARNING',\n",
       " 'Task': 'CLASSIFICATION',\n",
       " 'ResponseMetadata': {'RequestId': 'dec9863d-d6f4-411c-a5f9-3851621eff01',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'dec9863d-d6f4-411c-a5f9-3851621eff01',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '1823',\n",
       "   'date': 'Mon, 24 Apr 2023 22:28:31 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "approved_model_package = get_latest_approved_model_package(model_package_group_name)\n",
    "model_description = None\n",
    "\n",
    "if approved_model_package:\n",
    "    approved_model_package_arn = approved_model_package[\"ModelPackageArn\"]\n",
    "\n",
    "    model_description = sagemaker_client.describe_model_package(\n",
    "        ModelPackageName=approved_model_package_arn\n",
    "    )\n",
    "\n",
    "model_description"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46fb161e-ee0a-468d-9af9-136ab91ea1db",
   "metadata": {},
   "source": [
    "## Step 9 - Deploying the Model\n",
    "\n",
    "We can now deploy the latest approved model to an endpoint.\n",
    "\n",
    "Using the ARN of the model package from the Model Registry, we can deploy the model by creating a [ModelPackage](https://sagemaker.readthedocs.io/en/stable/api/inference/model.html#sagemaker.model.ModelPackage) instance and calling its `deploy()` method. The model information lives in the Model Registry, so we don't need to specify anything else."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "4304fa43-3d56-4be6-ac47-cca95399c91e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: 8-2023-04-24-14-58-45-037\n",
      "INFO:sagemaker:Creating endpoint-config with name penguins-endpoint-0424145845\n",
      "INFO:sagemaker:Creating endpoint with name penguins-endpoint-0424145845\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----!"
     ]
    }
   ],
   "source": [
    "model_package = ModelPackage(\n",
    "    model_package_arn=approved_model_package_arn, \n",
    "    sagemaker_session=sagemaker_session,\n",
    "    role=role, \n",
    ")\n",
    "\n",
    "# We can use the current time to generate a unique signature\n",
    "# to generate new assets every time we deploy the model.\n",
    "signature = time.strftime(\"%m%d%H%M%S\", time.localtime())\n",
    "endpoint_name = \"penguins-endpoint-\" + signature\n",
    "\n",
    "model_package.deploy(\n",
    "    endpoint_name=endpoint_name,\n",
    "    initial_instance_count=1, \n",
    "    instance_type=\"ml.m5.large\", \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27605ac3-3127-437d-ae8c-faf7de305e2c",
   "metadata": {},
   "source": [
    "## Step 10 - Testing the Endpoint\n",
    "\n",
    "Using a [Predictor](https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html#sagemaker.predictor.Predictor) from the endpoint name, we can test our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "ad4a6861-86e1-4e97-8f69-db2bb6469eba",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction: 2\n"
     ]
    }
   ],
   "source": [
    "predictor = Predictor(endpoint_name=endpoint_name)\n",
    "\n",
    "# The payload we need to provide the model is in CSV format. Notice how the model expects data that's\n",
    "# already transformed. We can't provide the original data from our dataset because the model will not\n",
    "# work with it.\n",
    "payload = \"0.6569590202313976, -1.0813829646495108, 1.2097102831892812, 0.9226343641317372, 1.0, 0.0, 0.0\"\n",
    "response = predictor.predict(payload, initial_args={\"ContentType\": \"text/csv\"})\n",
    "\n",
    "# We can decode the output of the endpoint and print the \"predictions\" key.\n",
    "predictions = json.loads(response.decode(\"utf-8\"))[\"predictions\"]\n",
    "print(f\"Prediction: {np.argmax(predictions, axis=1)[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77e4c712-4502-408a-a6d2-d5d9b4debedb",
   "metadata": {},
   "source": [
    "## Step 11 - Cleaning up\n",
    "\n",
    "Before you finish, don't forget to clean up after yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "690a5e01-face-4e06-abfc-5573a2d2b902",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Deleting endpoint configuration with name: penguins-endpoint-0424145845\n",
      "INFO:sagemaker:Deleting endpoint with name: penguins-endpoint-0424145845\n"
     ]
    }
   ],
   "source": [
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "565cf77e-7fc7-406e-a2e2-40c553f459f7",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Session 5 - Model Deployment\n",
    "\n",
    "This session extends the [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) with a step to deploy the model to an endpoint automatically. We'll use a [Lambda Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda) to create an endpoint and deploy the model. To control the endpoint's inputs and outputs, we'll modify the model's assets to include code that customizes the processing of a request. \n",
    "\n",
    "In the previous session, we deployed the model to an endpoint that expects the input to be in CSV format. Instead, we want our endpoint to handle unprocessed data in JSON format. Here is an example of the payload we want the endpoint to support:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"island\": \"Biscoe\",\n",
    "    \"culmen_length_mm\": 48.6,\n",
    "    \"culmen_depth_mm\": 16.0,\n",
    "    \"flipper_length_mm\": 230.0,\n",
    "    \"body_mass_g\": 5800.0,\n",
    "}\n",
    "```\n",
    "\n",
    "At the end of this session, our Pipeline will look like this:\n",
    "\n",
    "<img src='penguins/images/session5-pipeline.png' alt='Penguins dataset' width=\"600\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "8d585140-7940-4543-9954-b74352e8ff3e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow.model import TensorFlowModel\n",
    "from sagemaker.tensorflow.model import TensorFlowPredictor\n",
    "from sagemaker.workflow.lambda_step import LambdaStep, LambdaOutput, LambdaOutputTypeEnum\n",
    "from sagemaker.workflow.parameters import ParameterBoolean\n",
    "from sagemaker.lambda_helper import Lambda\n",
    "from sagemaker.serializers import JSONSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07fc27c6-31d4-454d-ae5b-1aeba25f0ac0",
   "metadata": {},
   "source": [
    "## Step 1 - Preparing the Inference Code\n",
    "\n",
    "Deploying the model we trained directly to an endpoint doesn't lets us control the data that goes in and comes out of the endpoint. Fortunately, SageMaker allows us to include an `inference.py` file with the model assets from where we can control how the endpoint works. You can see more information about how this works by checking the [SageMaker TensorFlow Serving Container](https://github.com/aws/sagemaker-tensorflow-serving-container) documentation.\n",
    "\n",
    "Let's start by setting up a local folder where we will create the `inference.py` script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "ac9cec59-c812-4ca9-9f71-d6725de03c3a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "CODE_FOLDER = PENGUINS_FOLDER / \"code\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a34c1c1-66af-4cf2-b103-8612bd60ce0e",
   "metadata": {},
   "source": [
    "We will include the inference code as part of the model assets to control the inference process on the SageMaker endpoint. SageMaker will automatically call the `handler()` function for every request to the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "id": "57e7ad9e-598b-4e28-9a20-a93ad2bb5f7d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/code/inference.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $CODE_FOLDER/inference.py\n",
    "\n",
    "import os\n",
    "import json\n",
    "import boto3\n",
    "import requests\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from pickle import load\n",
    "from pathlib import Path\n",
    "\n",
    "\n",
    "PIPELINE_FILE = Path(\"/tmp\") / \"pipeline.pkl\"\n",
    "\n",
    "# By default, we will read the S3 location of the Scikit-Learn\n",
    "# pipeline from an environment variable that we'll configure\n",
    "# when registering the model.\n",
    "PIPELINE_S3_LOCATION = os.environ.get(\"PIPELINE_S3_LOCATION\", None)\n",
    "\n",
    "\n",
    "s3 = boto3.resource(\"s3\")\n",
    "\n",
    "\n",
    "def handler(data, context, pipeline_s3_location=PIPELINE_S3_LOCATION):\n",
    "    \"\"\"\n",
    "    This is the entrypoint that will be called by SageMaker when the endpoint\n",
    "    receives a request. You can see more information at \n",
    "    https://github.com/aws/sagemaker-tensorflow-serving-container.\n",
    "    \"\"\"\n",
    "    print(\"Handling endpoint request\")\n",
    "    \n",
    "    download_pipeline(pipeline_s3_location)\n",
    "    \n",
    "    instance = _process_input(data, context)\n",
    "    output = _predict(instance, context)\n",
    "    return _process_output(output, context)\n",
    "\n",
    "\n",
    "def download_pipeline(pipeline_s3_location):\n",
    "    \"\"\"\n",
    "    This function will download the Scikit-Learn pipeline from S3\n",
    "    if it doesn't already exist. We need the pipeline to pre-process\n",
    "    the data before we run it through the model.\n",
    "    \"\"\"\n",
    "    \n",
    "    s3_parts = pipeline_s3_location.split('/', 3)\n",
    "    bucket = s3_parts[2]\n",
    "    key = s3_parts[3]\n",
    "    \n",
    "    if not PIPELINE_FILE.exists():\n",
    "        s3.Bucket(bucket).download_file(f\"{key}/pipeline.pkl\", str(PIPELINE_FILE))\n",
    "\n",
    "\n",
    "def transform(payload):\n",
    "    \"\"\"\n",
    "    This function transforms the payload in the request using the\n",
    "    Scikit-Learn pipeline that we created during the preprocessing step.\n",
    "    \"\"\"\n",
    "    \n",
    "    print(\"Transforming input data...\")\n",
    "\n",
    "    pipeline = load(open(PIPELINE_FILE, 'rb'))\n",
    "    \n",
    "    island = payload.get(\"island\", \"Biscoe\")\n",
    "    culmen_length_mm = payload.get(\"culmen_length_mm\", 0)\n",
    "    culmen_depth_mm = payload.get(\"culmen_depth_mm\", 0)\n",
    "    flipper_length_mm = payload.get(\"flipper_length_mm\", 0)\n",
    "    body_mass_g = payload.get(\"body_mass_g\", 0)\n",
    "    \n",
    "    data = pd.DataFrame(\n",
    "        columns=[\"island\", \"culmen_length_mm\", \"culmen_depth_mm\", \"flipper_length_mm\", \"body_mass_g\"], \n",
    "        data=[[\n",
    "            island, \n",
    "            culmen_length_mm, \n",
    "            culmen_depth_mm, \n",
    "            flipper_length_mm, \n",
    "            body_mass_g\n",
    "        ]]\n",
    "    )\n",
    "    \n",
    "    result = pipeline.transform(data)\n",
    "    return result[0].tolist()\n",
    "    \n",
    "\n",
    "def _process_input(data, context):\n",
    "    print(\"Processing input data...\")\n",
    "    \n",
    "    if context is None:\n",
    "        # The context will be None when we are testing the code\n",
    "        # directly from a notebook. In that case, we can use the\n",
    "        # data directly.\n",
    "        endpoint_input = data\n",
    "    elif context.request_content_type in (\"application/json\", \"application/octet-stream\"):\n",
    "        # When the endpoint is running, we will receive a context\n",
    "        # object. We need to parse the input and turn it into \n",
    "        # JSON in that case.\n",
    "        endpoint_input = json.loads(data.read().decode(\"utf-8\"))\n",
    "\n",
    "        if endpoint_input is None:\n",
    "            raise ValueError(\"There was an error parsing the input request.\")\n",
    "    else:\n",
    "        raise ValueError(f\"Unsupported content type: {context.request_content_type or 'unknown'}\")\n",
    "        \n",
    "    return transform(endpoint_input)\n",
    "\n",
    "\n",
    "def _predict(instance, context):\n",
    "    print(\"Sending input data to model to make a prediction...\")\n",
    "    \n",
    "    model_input = json.dumps({\"instances\": [instance]})\n",
    "    \n",
    "    if context is None:\n",
    "        # The context will be None when we are testing the code\n",
    "        # directly from a notebook. In that case, we want to return\n",
    "        # a fake prediction back.\n",
    "        result = {\n",
    "            \"predictions\": [\n",
    "                [0.2, 0.5, 0.3]\n",
    "            ]\n",
    "        }\n",
    "    else:\n",
    "        # When the endpoint is running, we will receive a context\n",
    "        # object. In that case we need to send the instance to the\n",
    "        # model to get a prediction back.\n",
    "        response = requests.post(context.rest_uri, data=model_input)\n",
    "        \n",
    "        if response.status_code != 200:\n",
    "            raise ValueError(response.content.decode('utf-8'))\n",
    "            \n",
    "        result = json.loads(response.content)\n",
    "    \n",
    "    print(f\"Response: {result}\")\n",
    "    return result\n",
    "\n",
    "\n",
    "def _process_output(output, context):\n",
    "    print(\"Processing prediction received from the model...\")\n",
    "    \n",
    "    response_content_type = \"application/json\" if context is None else context.accept_header\n",
    "    \n",
    "    prediction = np.argmax(output[\"predictions\"][0])\n",
    "    confidence = output[\"predictions\"][0][prediction]\n",
    "    \n",
    "    print(f\"Prediction: {prediction}. Confidence: {confidence}\")\n",
    "    \n",
    "    result = json.dumps({\n",
    "        \"prediction\": int(prediction),\n",
    "        \"confidence\": confidence\n",
    "    }), response_content_type\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "656e7e63-39a8-4859-af1e-3e7b1573f9a2",
   "metadata": {},
   "source": [
    "## Step 2 - Testing the Inference Code\n",
    "\n",
    "Let's test the inference code locally to ensure it works before deploying it. The `handler()` function is the entry point that will be called by SageMaker whenever the endpoint receives a request.\n",
    "\n",
    "When testing the inference code, we want to set the `context` to `None` so the function recognizes we call it locally. We also want to set the `pipeline_s3_location` to the S3 location of the Scikit-Learn pipeline. We generated this pipeline in the preprocessing step from Session 1 and uploaded it to the location specified by the `proprocessor_destination` pipeline configuration variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "a7aca49b-2080-4068-80e6-8b3f10e54177",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Handling endpoint request\n",
      "Processing input data...\n",
      "Transforming input data...\n",
      "Sending input data to model to make a prediction...\n",
      "Response: {'predictions': [[0.2, 0.5, 0.3]]}\n",
      "Processing prediction received from the model...\n",
      "Prediction: 1. Confidence: 0.5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/site-packages/sklearn/base.py:310: UserWarning: Trying to unpickle estimator SimpleImputer from version 0.23.2 when using version 0.24.2. This might lead to breaking code or invalid results. Use at your own risk.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.8/site-packages/sklearn/base.py:310: UserWarning: Trying to unpickle estimator StandardScaler from version 0.23.2 when using version 0.24.2. This might lead to breaking code or invalid results. Use at your own risk.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.8/site-packages/sklearn/base.py:310: UserWarning: Trying to unpickle estimator Pipeline from version 0.23.2 when using version 0.24.2. This might lead to breaking code or invalid results. Use at your own risk.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.8/site-packages/sklearn/base.py:310: UserWarning: Trying to unpickle estimator OneHotEncoder from version 0.23.2 when using version 0.24.2. This might lead to breaking code or invalid results. Use at your own risk.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.8/site-packages/sklearn/base.py:310: UserWarning: Trying to unpickle estimator ColumnTransformer from version 0.23.2 when using version 0.24.2. This might lead to breaking code or invalid results. Use at your own risk.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('{\"prediction\": 1, \"confidence\": 0.5}', 'application/json')"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from container.code.inference import handler\n",
    "\n",
    "handler(\n",
    "    data={\n",
    "        \"island\": \"Biscoe\",\n",
    "        \"culmen_length_mm\": 48.6,\n",
    "        \"culmen_depth_mm\": 16.0,\n",
    "        \"flipper_length_mm\": 230.0,\n",
    "        \"body_mass_g\": 5800.0,\n",
    "    }, \n",
    "    context=None, \n",
    "    pipeline_s3_location=preprocessor_destination.default_value\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed63f158-06f3-438b-b489-bb0496f2eddd",
   "metadata": {},
   "source": [
    "## Step 3 - Registering the Model\n",
    "\n",
    "We can now register a new [TensorFlowModel](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-serving-model). We must also ensure SageMaker repackages the model assets to include the `inference.py` file.\n",
    "\n",
    "SageMaker triggers a repack whenever we specify the `source_dir` attribute. We want that attribute to point to the local folder containing the `inference.py` file. SageMaker will automatically modify the original `model.tar.gz` package to include a `/code` folder containing the file. Since we need access to Scikit-Learn in our script, we can include a `requirements.txt` file in the same `/code` folder, and SageMaker will install everything in it. To repack the model assets, SageMaker will automatically include a new step in the pipeline right before registering the model.\n",
    "\n",
    "Here is what the new `model.tar.gz` package will look like:\n",
    "\n",
    "```\n",
    "model/\n",
    "    |--[model_version_number]\n",
    "        |--assets/\n",
    "        |--variables/\n",
    "        |--saved_model.pb\n",
    "code/\n",
    "    |--inference.py\n",
    "    |--requirements.txt\n",
    "```\n",
    "\n",
    "Let's use a [ModelStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.model_step.ModelStep) to register the model. Notice the following:\n",
    "\n",
    "* `model_data`: We use the model assets we generated during the Training or Tuning Step. We determined which assets to use back in Session 4 and stored them in the `model_data` variable.\n",
    "* `source_dir`: This points to the local folder containing the `inference.py` file. SageMaker will trigger a repack to include the `/code` folder in the model assets.\n",
    "* `env`: Our custom inference code expects an environment variable `PIPELINE_S3_LOCATION` to point to the location of the Scikit-Learn pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "07513884-c7bd-4710-9730-f8ca7fe904a5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.tensorflow.model:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n",
      "/usr/local/lib/python3.8/site-packages/sagemaker/workflow/pipeline_context.py:270: UserWarning: Running within a PipelineSession, there will be No Wait, No Logs, and No Job being started.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "model = TensorFlowModel(\n",
    "    model_data=model_data,\n",
    "    entry_point=\"inference.py\",\n",
    "    source_dir=str(CODE_FOLDER),\n",
    "    env={\n",
    "        \"PIPELINE_S3_LOCATION\": preprocessor_destination,\n",
    "    },\n",
    "    framework_version=\"2.6\",\n",
    "    sagemaker_session=PipelineSession(),\n",
    "    role=role,\n",
    ")\n",
    "\n",
    "\n",
    "register_model_step = ModelStep(\n",
    "    name=\"register-model\",\n",
    "    step_args=model.register(\n",
    "        content_types=[\"application/json\"],\n",
    "        response_types=[\"application/json\"],\n",
    "        inference_instances=[\"ml.m5.large\"],\n",
    "        domain=\"MACHINE_LEARNING\",\n",
    "        task=\"CLASSIFICATION\",\n",
    "        framework=\"TENSORFLOW\",\n",
    "        framework_version=\"2.6\",\n",
    "        model_package_group_name=model_package_group_name,\n",
    "        model_metrics=model_metrics,\n",
    "        approval_status=model_approval_status,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b40574a9-4936-4194-ad52-40079a25a8b4",
   "metadata": {},
   "source": [
    "## Step 4 - Data Capture Configuration\n",
    "\n",
    "We can use [Data Capture](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-data-capture.html) to record the inputs and outputs of the endpoint to use them later for monitoring the model. Amazon SageMaker Model Monitor automatically parses this captured data and compares metrics from this data with a baseline that you create for the model. \n",
    "\n",
    "We'll enable Data Capture and use the following settings:\n",
    "\n",
    "* `data_capture_enabled`: Whether we want to enable Data Capture.\n",
    "* `data_capture_percentage`: Represents the percentage of information that flows through the endpoint that we want to capture. For this example, we'll set that to 100%.\n",
    "* `data_capture_destination`: Specifies the S3 location where we want to store the captured data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "497007c4-1dea-4fab-b4ef-6882cd6716c2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_capture_enabled = ParameterBoolean(\n",
    "    name=\"data_capture_enabled\",\n",
    "    default_value=True,\n",
    ")\n",
    "\n",
    "data_capture_percentage = ParameterInteger(\n",
    "    name=\"data_capture_percentage\",\n",
    "    default_value=100,\n",
    ")\n",
    "\n",
    "data_capture_destination = ParameterString(\n",
    "    name=\"data_capture_destination\",\n",
    "    default_value=f\"{S3_FILEPATH}/capture\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "436b60d5-6be3-428f-9979-4fe9eebfc5eb",
   "metadata": {},
   "source": [
    "## Step 5 - Deploying the Model\n",
    "\n",
    "In Session 4, we registered the model as part of the pipeline but deployed it manually. Instead, we can use a [Lambda Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda) to deploy the model automatically.\n",
    "\n",
    "Let's start by writing the Lambda function to take the model information and create a new hosting endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "28c75afd-d0d0-47f4-b7b6-9d590c5b600e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/lambda.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $PENGUINS_FOLDER/lambda.py\n",
    "\n",
    "import os\n",
    "import json\n",
    "import boto3\n",
    "\n",
    "sagemaker = boto3.client(\"sagemaker\")\n",
    "\n",
    "def lambda_handler(event, context):\n",
    "    model_name = event[\"model_name\"]\n",
    "    endpoint_config_name = event[\"endpoint_config_name\"]\n",
    "    endpoint_name = event[\"endpoint_name\"]\n",
    "    data_capture_enabled = event[\"data_capture_enabled\"]\n",
    "    data_capture_percentage = event[\"data_capture_percentage\"]\n",
    "    data_capture_destination = event[\"data_capture_destination\"]\n",
    "    role = event[\"role\"]\n",
    "    \n",
    "    \n",
    "    # The first step is to create a new model. We will use the\n",
    "    # ARN of the model we registered.\n",
    "    sagemaker.create_model(\n",
    "        ModelName=model_name, \n",
    "        ExecutionRoleArn=role, \n",
    "        Containers=[{\n",
    "            \"ModelPackageName\": event[\"model_package_arn\"]\n",
    "        }] \n",
    "    )\n",
    "\n",
    "    # Then, we need to create an Endpoint Configuration.\n",
    "    # Here is where we specify the hardware we need for the\n",
    "    # endpoint and the Data Capture configuration. \n",
    "    sagemaker.create_endpoint_config(\n",
    "        EndpointConfigName=endpoint_config_name,\n",
    "        ProductionVariants=[\n",
    "            {\n",
    "                \"ModelName\": model_name,\n",
    "                \"InstanceType\": \"ml.m5.large\",\n",
    "                \"InitialVariantWeight\": 1,\n",
    "                \"InitialInstanceCount\": 1,\n",
    "                \"VariantName\": \"AllTraffic\",\n",
    "            }\n",
    "        ],\n",
    "        DataCaptureConfig={\n",
    "            \"EnableCapture\": data_capture_enabled,\n",
    "            \"InitialSamplingPercentage\": data_capture_percentage,\n",
    "            \"DestinationS3Uri\": data_capture_destination,\n",
    "            \"CaptureOptions\": [\n",
    "                {\n",
    "                    'CaptureMode': \"Input\"\n",
    "                },\n",
    "                {\n",
    "                    'CaptureMode': \"Output\"\n",
    "                },\n",
    "            ],\n",
    "            \"CaptureContentTypeHeader\": {\n",
    "                \"JsonContentTypes\": [\n",
    "                    \"application/json\",\n",
    "                    \"application/octect-stream\"\n",
    "                ]\n",
    "            }\n",
    "        },\n",
    "    )\n",
    "\n",
    "    # Finally, we can create the new Endpoint using the\n",
    "    # Endpoint Configuration we created above.\n",
    "    sagemaker.create_endpoint(\n",
    "        EndpointName=endpoint_name, \n",
    "        EndpointConfigName=endpoint_config_name,\n",
    "    )\n",
    "    \n",
    "    return {\n",
    "        \"statusCode\": 200,\n",
    "        \"body\": json.dumps(\"Endpoint deployed successfully\"),\n",
    "        \"model_name\": model_name,\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c66a4446-6db4-4515-bb1d-88ee21013bb2",
   "metadata": {},
   "source": [
    "We need to ensure our Lambda function has permission to interact with SageMaker, so let's create a new role to run the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "id": "f9941979-c085-4ef9-8c66-908378d89a91",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def create_lambda_role(role_name):\n",
    "    try:\n",
    "        response = iam_client.create_role(\n",
    "            RoleName = role_name,\n",
    "            AssumeRolePolicyDocument = json.dumps({\n",
    "                \"Version\": \"2012-10-17\",\n",
    "                \"Statement\": [\n",
    "                    {\n",
    "                        \"Effect\": \"Allow\",\n",
    "                        \"Principal\": {\n",
    "                            \"Service\": \"lambda.amazonaws.com\"\n",
    "                        },\n",
    "                        \"Action\": \"sts:AssumeRole\"\n",
    "                    }\n",
    "                ]\n",
    "            }),\n",
    "            Description=\"Lambda Pipeline Role\"\n",
    "        )\n",
    "\n",
    "        role_arn = response['Role']['Arn']\n",
    "\n",
    "        iam_client.attach_role_policy(\n",
    "            RoleName=role_name,\n",
    "            PolicyArn='arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'\n",
    "        )\n",
    "\n",
    "        iam_client.attach_role_policy(\n",
    "            PolicyArn='arn:aws:iam::aws:policy/AmazonSageMakerFullAccess',\n",
    "            RoleName=role_name\n",
    "        )\n",
    "\n",
    "        return role_arn\n",
    "\n",
    "    except iam_client.exceptions.EntityAlreadyExistsException:\n",
    "        response = iam_client.get_role(RoleName=role_name)\n",
    "        return response['Role']['Arn']\n",
    "    \n",
    "\n",
    "lambda_role = create_lambda_role(\"lambda-pipeline-role\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad4dcc96-2890-43ff-a6cd-f2a9969adc52",
   "metadata": {},
   "source": [
    "## Step 6 - Setting up the Lambda Step\n",
    "\n",
    "Let's define the [LambdaStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.lambda_step.LambdaStep) that will run the function to deploy the model. In this example, this step will create a new Lambda function every time it runs, so we must remove any old function after the pipeline finishes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "id": "7c6eda89-70f3-49b4-8983-90f4db72fd02",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We can use the current time to generate a unique signature\n",
    "# to generate new assets every time we deploy the model.\n",
    "signature = time.strftime(\"%m%d%H%M%S\", time.localtime())\n",
    "model_name = \"penguins-model-\" + signature\n",
    "endpoint_config_name = \"penguins-endpoint-config-\" + signature\n",
    "endpoint_name = \"penguins-endpoint-\" + signature\n",
    "lambda_function_name = \"deploy-endpoint-fn-\" + signature \n",
    "\n",
    "\n",
    "deploy_step = LambdaStep(\n",
    "    name=\"deploy\",\n",
    "    lambda_func=Lambda(\n",
    "        function_name=lambda_function_name,\n",
    "        execution_role_arn=lambda_role,\n",
    "        script=str(PENGUINS_FOLDER / \"lambda.py\"),\n",
    "        handler=\"lambda.lambda_handler\",\n",
    "        timeout=600,\n",
    "        memory_size=10240,\n",
    "    ),\n",
    "    inputs={\n",
    "        \"model_name\": model_name,\n",
    "        \"endpoint_config_name\": endpoint_config_name,\n",
    "        \"endpoint_name\": endpoint_name,\n",
    "        \n",
    "        # We use the ARN of the model we registered to\n",
    "        # deploy it to the endpoint.\n",
    "        \"model_package_arn\": register_model_step.properties.ModelPackageArn,\n",
    "        \n",
    "        \"data_capture_enabled\": data_capture_enabled,\n",
    "        \"data_capture_percentage\": data_capture_percentage,\n",
    "        \"data_capture_destination\": data_capture_destination,\n",
    "        \"role\": role,\n",
    "    },\n",
    "    outputs=[\n",
    "        LambdaOutput(output_name=\"model_name\", output_type=LambdaOutputTypeEnum.String)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cfe2299-be2a-46a7-809e-6174d44abddb",
   "metadata": {},
   "source": [
    "## Step 7 - Setting up the Condition Step\n",
    "\n",
    "As we saw in Session 4, we only want to register a new model if its accuracy exceeds a predefined threshold. We can use a [Condition Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-condition).\n",
    "\n",
    "If the condition succeeds, we will register and deploy the custom model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "caacdde0-a65f-4225-8303-2b0106da1ed8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "condition_step = ConditionStep(\n",
    "    name=\"check-model-accuracy\",\n",
    "    conditions=[condition_gte],\n",
    "    if_steps=[\n",
    "        register_model_step, deploy_step\n",
    "    ],\n",
    "    else_steps=[fail_step], \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ade6d67c-6906-434d-9c41-17a4efeb38d2",
   "metadata": {},
   "source": [
    "## Step 8 - Running the Pipeline\n",
    "\n",
    "We can now run the pipeline. If the pipeline succeeds, there will be a new running endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "id": "d1969a5e-2ebf-474f-a275-ae0946c2fab8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session5_pipeline = Pipeline(\n",
    "    name=\"penguins-session5-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        baseline_destination,\n",
    "        evaluation_destination,\n",
    "        model_approval_status,\n",
    "        accuracy_threshold,\n",
    "        data_capture_enabled,\n",
    "        data_capture_percentage,\n",
    "        data_capture_destination,\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_step, \n",
    "        tuning_step if USE_TUNING_STEP else training_step, \n",
    "        evaluation_step,\n",
    "        condition_step\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de309b9-6011-40fe-b03d-ff991700c517",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "id": "9e01390f-31a5-44b0-a9ea-2c4697a57aa1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker.workflow._utils:Popping out 'CertifyForMarketplace' from the pipeline definition since it will be overridden in pipeline execution time.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n",
      "Using provided s3_resource\n"
     ]
    }
   ],
   "source": [
    "session5_pipeline.upsert(role_arn=role)\n",
    "execution = session5_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "536ca1b1-fdc6-4260-b6f7-ee3c6aab4e76",
   "metadata": {},
   "source": [
    "## Step 9 - Testing the Endpoint\n",
    "\n",
    "We can now create a [Predictor](https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html) to test the endpoint with a few examples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e32865-705d-4a2e-a7df-7df198aa8c96",
   "metadata": {},
   "source": [
    "First, let's wait for the endpoint to be ready to service traffic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "id": "17098e7e-fc61-4027-b98f-7f192ffdf9fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "waiter = sagemaker_client.get_waiter(\"endpoint_in_service\")\n",
    "waiter.wait(\n",
    "    EndpointName=endpoint_name,\n",
    "    WaiterConfig={\n",
    "        \"Delay\": 10,\n",
    "        \"MaxAttempts\": 30\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "016bcfc1-32ba-4ec8-93f6-0c04b00cc4f8",
   "metadata": {},
   "source": [
    "Now that the endpoint is in service, we can create a [Predictor](https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html) using a JSON serializer and a deserializer to have it automatically serialize and deserialize the information to and from the endpoint. Check [Serializers](https://sagemaker.readthedocs.io/en/stable/api/inference/serializers.html) and [Deserializers](https://sagemaker.readthedocs.io/en/stable/api/inference/deserializers.html) for a list of supported serializers and deserializers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "862118a0-c00b-4f5a-9374-68a94dbfbc0c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictor = Predictor(\n",
    "    endpoint_name=endpoint_name,\n",
    "    serializer=JSONSerializer(),\n",
    "    deserializer=JSONDeserializer()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "529b98a7-2f36-4e4e-a277-c5172e5fbe51",
   "metadata": {
    "tags": []
   },
   "source": [
    "Running one example through the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "id": "8148ef38-92ed-4f9a-8024-69e2d9d6b5d4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'prediction': 0, 'confidence': 0.569238842}"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.predict({\n",
    "    \"island\": \"Dream\",\n",
    "    \"culmen_length_mm\": 46.4,\n",
    "    \"culmen_depth_mm\": 18.6,\n",
    "    \"flipper_length_mm\": 190.0,\n",
    "    \"body_mass_g\": 3450.0,\n",
    "    \n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9075ef4c-36a8-4c5d-88ef-2a3e49abdd51",
   "metadata": {},
   "source": [
    "Running another example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "id": "2476337f-a39b-45d5-ab57-63567c3f0438",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'prediction': 2, 'confidence': 0.975528777}"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.predict({\n",
    "    \"island\": \"Biscoe\",\n",
    "    \"culmen_length_mm\": 48.6,\n",
    "    \"culmen_depth_mm\": 16.0,\n",
    "    \"flipper_length_mm\": 230.0,\n",
    "    \"body_mass_g\": 5800.0,\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d34fcdcd-cc9e-4cf5-8da1-3dc73cb6b59a",
   "metadata": {},
   "source": [
    "## Step 10 - Cleaning up\n",
    "\n",
    "Before you finish, don't forget to clean up after yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "id": "0cf877e9-e1bc-40f0-acaa-64b6a5e6650f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Deleting endpoint configuration with name: penguins-endpoint-config-0424233344\n",
      "INFO:sagemaker:Deleting endpoint with name: penguins-endpoint-0424233344\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'ResponseMetadata': {'RequestId': '8d384fa8-adb0-4228-b67e-fae76c9dca9d',\n",
       "  'HTTPStatusCode': 204,\n",
       "  'HTTPHeaders': {'date': 'Mon, 24 Apr 2023 23:39:37 GMT',\n",
       "   'content-type': 'application/json',\n",
       "   'connection': 'keep-alive',\n",
       "   'x-amzn-requestid': '8d384fa8-adb0-4228-b67e-fae76c9dca9d'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.delete_endpoint()\n",
    "sagemaker_client.delete_model(ModelName=model_name)\n",
    "\n",
    "lambda_client = boto3.client('lambda')\n",
    "lambda_client.delete_function(FunctionName=lambda_function_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "917fe920-172d-4b07-bd7d-b2ecea536319",
   "metadata": {},
   "source": [
    "# Session 6 - Model Monitoring\n",
    "\n",
    "Here is what the Pipeline will look like at the end of this session:\n",
    "\n",
    "<img src='penguins/images/session6-pipeline.png' alt='Penguins dataset' width=\"600\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "748c47c5-71e6-4f76-9a8b-9a5f4716e806",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.check_job_config import CheckJobConfig\n",
    "from sagemaker.workflow.quality_check_step import DataQualityCheckConfig, QualityCheckStep, ModelQualityCheckConfig\n",
    "from sagemaker.workflow.execution_variables import ExecutionVariables\n",
    "from sagemaker.model_monitor import DatasetFormat, model_monitoring\n",
    "from sagemaker.drift_check_baselines import DriftCheckBaselines\n",
    "from sagemaker.workflow.parameters import ParameterBoolean\n",
    "from sagemaker.inputs import CreateModelInput, TransformInput\n",
    "from sagemaker.model import Model\n",
    "from sagemaker.transformer import Transformer\n",
    "from sagemaker.workflow.steps import CreateModelStep, TransformStep"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2938b5a9-0e8d-448a-bd30-ee61bcd90907",
   "metadata": {},
   "source": [
    "## Step 1 - Data Quality Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "eb805c2d-2aeb-494e-bd4f-dbb60753d990",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_quality_skip_check = ParameterBoolean(\n",
    "    name=\"data_quality_skip_check\", \n",
    "    default_value=True\n",
    ")\n",
    "\n",
    "data_quality_register_new_baseline = ParameterBoolean(\n",
    "    name=\"data_quality_register_new_baseline\", \n",
    "    default_value=True\n",
    ")\n",
    "\n",
    "data_quality_supplied_baseline_statistics = ParameterString(\n",
    "    name=\"data_quality_supplied_baseline_statistics\", \n",
    "    default_value=\"\"\n",
    ")\n",
    "\n",
    "data_quality_supplied_baseline_constraints = ParameterString(\n",
    "    name=\"data_quality_supplied_baseline_constraints\", \n",
    "    default_value=\"\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24c26ac4-5d30-41e9-8952-e4deb39de819",
   "metadata": {},
   "source": [
    "## Step 2 - Setting up Data Quality Check Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "0b80bcab-d2c5-437c-a1c8-8eea208c0e29",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: .\n",
      "INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.\n"
     ]
    }
   ],
   "source": [
    "check_job_config = CheckJobConfig(\n",
    "    instance_type=\"ml.t3.xlarge\",\n",
    "    instance_count=1,\n",
    "    sagemaker_session=sagemaker_session,\n",
    "    volume_size_in_gb=20,\n",
    "    role=role,\n",
    ")\n",
    "\n",
    "data_quality_check_config = DataQualityCheckConfig(\n",
    "    baseline_dataset=preprocess_step.properties.ProcessingOutputConfig.Outputs[\"train\"].S3Output.S3Uri,\n",
    "    dataset_format=DatasetFormat.csv(header=False, output_columns_position=\"START\"),\n",
    "    output_s3_uri=Join(on='/', values=[S3_FILEPATH, \"monitor\", \"data-quality\"]),\n",
    ")\n",
    "\n",
    "data_quality_check_step = QualityCheckStep(\n",
    "    name=\"data-quality-check\",\n",
    "    skip_check=data_quality_skip_check,\n",
    "    register_new_baseline=data_quality_register_new_baseline,\n",
    "    quality_check_config=data_quality_check_config,\n",
    "    check_job_config=check_job_config,\n",
    "    supplied_baseline_statistics=data_quality_supplied_baseline_statistics,\n",
    "    supplied_baseline_constraints=data_quality_supplied_baseline_constraints,\n",
    "    model_package_group_name=model_package_group_name,\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81430dfd-2524-43e4-bfe9-c6545316005d",
   "metadata": {},
   "source": [
    "## Step 3 - Transforming the Baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "322858a6-029e-4c20-a243-0cfc9e38c2fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "custom_model = Model(\n",
    "    image_uri=CUSTOM_CONTAINER_IMAGE_URI,\n",
    "    model_data=model_data,\n",
    "    entry_point=\"inference.py\",\n",
    "    source_dir=\"container/code\",\n",
    "    sagemaker_session=PipelineSession(),\n",
    "    role=role,\n",
    ")\n",
    "\n",
    "create_model_step = ModelStep(\n",
    "    name=\"create-model\",\n",
    "    step_args=custom_model.create(instance_type=\"ml.m5.large\"),\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "1987a788-de7a-4f60-ac8d-819d9ffcdf8e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "transformer = Transformer(\n",
    "    model_name=create_model_step.properties.ModelName,\n",
    "    instance_type=\"ml.c5.xlarge\",\n",
    "    instance_count=1,\n",
    "    accept=\"application/json\",\n",
    "    strategy=\"SingleRecord\",\n",
    "    assemble_with=\"Line\",\n",
    "    output_path=f\"{S3_FILEPATH}/transform\",\n",
    ")\n",
    "\n",
    "transform_step = TransformStep(\n",
    "    name=\"transform\",\n",
    "    transformer=transformer,\n",
    "    inputs=TransformInput(\n",
    "        data=preprocess_step.properties.ProcessingOutputConfig.Outputs[\"baseline\"].S3Output.S3Uri,\n",
    "        # data=\"s3://mlschool/penguins/preprocessing/baseline/\",\n",
    "        join_source=\"Input\",\n",
    "        content_type=\"application/json\",\n",
    "        split_type=\"Line\",\n",
    "    ),\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59a24666-906e-4eba-8c7b-ab4db82f233a",
   "metadata": {},
   "source": [
    "## Step 4 - Model Quality Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "2eb811ae-599f-4379-bdcf-47d2e1c67e91",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_quality_skip_check = ParameterBoolean(\n",
    "    name=\"model_quality_skip_check\", \n",
    "    default_value=True\n",
    ")\n",
    "\n",
    "model_quality_register_new_baseline = ParameterBoolean(\n",
    "    name=\"model_quality_register_new_baseline\", \n",
    "    default_value=True\n",
    ")\n",
    "\n",
    "model_quality_supplied_baseline_statistics = ParameterString(\n",
    "    name=\"model_quality_supplied_baseline_statistics\", \n",
    "    default_value=\"\"\n",
    ")\n",
    "\n",
    "model_quality_supplied_baseline_constraints = ParameterString(\n",
    "    name=\"model_quality_supplied_baseline_constraints\", \n",
    "    default_value=\"\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fafc7c4-6fef-4832-8b99-8c45d078fdd2",
   "metadata": {},
   "source": [
    "## Step 5 - Setting up Model Quality Check Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "9aa3a284-8763-4000-a263-70314b530652",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: .\n",
      "INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.\n"
     ]
    }
   ],
   "source": [
    "model_quality_check_config = ModelQualityCheckConfig(\n",
    "    baseline_dataset=transform_step.properties.TransformOutput.S3OutputPath,\n",
    "    dataset_format=DatasetFormat.json(lines=True),\n",
    "    output_s3_uri=Join(on='/', values=[S3_FILEPATH, \"monitor\", \"model-quality\"]),\n",
    "    problem_type=\"MulticlassClassification\",\n",
    "    inference_attribute=\"$.SageMakerOutput.prediction\",\n",
    "    ground_truth_attribute=\"groundtruth\",\n",
    ")\n",
    "\n",
    "model_quality_check_step = QualityCheckStep(\n",
    "    name=\"model-quality-check\",\n",
    "    skip_check=model_quality_skip_check,\n",
    "    register_new_baseline=model_quality_register_new_baseline,\n",
    "    quality_check_config=model_quality_check_config,\n",
    "    check_job_config=check_job_config,\n",
    "    supplied_baseline_statistics=model_quality_supplied_baseline_statistics,\n",
    "    supplied_baseline_constraints=model_quality_supplied_baseline_constraints,\n",
    "    model_package_group_name=model_package_group_name,\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "693535ba-fca7-4e89-a4cb-b4f333fa2d03",
   "metadata": {},
   "source": [
    "## Step 6 - Setting up the Model Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "a773f134-ac2f-4dba-976e-9b7f0b384b6e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_metrics = ModelMetrics(\n",
    "    model_data_statistics=MetricsSource(\n",
    "        s3_uri=data_quality_check_step.properties.CalculatedBaselineStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_data_constraints=MetricsSource(\n",
    "        s3_uri=data_quality_check_step.properties.CalculatedBaselineConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    # model_statistics=MetricsSource(\n",
    "    #     s3_uri=Join(on=\"\", values=[\n",
    "    #         evaluation_step.arguments['ProcessingOutputConfig']['Outputs'][0]['S3Output']['S3Uri'],\n",
    "    #         \"/evaluation.json\"]\n",
    "    #     ),\n",
    "    #     content_type=\"application/json\",\n",
    "    # ),\n",
    "    \n",
    "    model_statistics=MetricsSource(\n",
    "        s3_uri=model_quality_check_step.properties.CalculatedBaselineStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    \n",
    "    model_constraints=MetricsSource(\n",
    "        s3_uri=model_quality_check_step.properties.CalculatedBaselineConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d3c7f19-6d15-4fd2-8616-50459ee46415",
   "metadata": {},
   "source": [
    "## Step 7 - Setting up the Drift Check Baselines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "8302f356-97f3-4325-9a9a-6a0712de9133",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "drift_check_baselines = DriftCheckBaselines(\n",
    "    model_data_statistics=MetricsSource(\n",
    "        s3_uri=data_quality_check_step.properties.BaselineUsedForDriftCheckStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_data_constraints=MetricsSource(\n",
    "        s3_uri=data_quality_check_step.properties.BaselineUsedForDriftCheckConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_statistics=MetricsSource(\n",
    "        s3_uri=model_quality_check_step.properties.BaselineUsedForDriftCheckStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_constraints=MetricsSource(\n",
    "        s3_uri=model_quality_check_step.properties.BaselineUsedForDriftCheckConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba3487a0-05ad-4f3a-8f50-9884dc2aef64",
   "metadata": {},
   "source": [
    "## Step 8 - Setting up a Model Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "7056a009-91c0-4955-90dd-b90ef8cab149",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "register_custom_model_step = ModelStep(\n",
    "    name=\"register-custom-model\",\n",
    "    step_args=custom_model.register(\n",
    "        content_types=[\"application/json\"],\n",
    "        response_types=[\"application/json\"],\n",
    "        inference_instances=[\"ml.m5.large\"],\n",
    "        domain=\"MACHINE_LEARNING\",\n",
    "        task=\"CLASSIFICATION\",\n",
    "        framework=\"TENSORFLOW\",\n",
    "        framework_version=\"2.6\",\n",
    "        model_package_group_name=model_package_group_name,\n",
    "        model_metrics=model_metrics,\n",
    "        drift_check_baselines=drift_check_baselines,\n",
    "        approval_status=model_approval_status,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d00b5e6-9858-4acc-bbfe-a2ce24ec20e0",
   "metadata": {},
   "source": [
    "## Step 9 - Setting up the Condition Step\n",
    "\n",
    "We only want to register a new model if its accuracy is above a predefined threshold. We can use a [Condition Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-condition) to accomplish this. Check the [ConditionStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#conditionstep) SageMaker's SDK documentation for more information.\n",
    "\n",
    "In this example we are going to use a [ConditionGreaterThanOrEqualTo](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.conditions.ConditionGreaterThanOrEqualTo) condition to compare the model's accuracy with the threshold. Take a look at the [Conditions](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_building_pipeline.html#conditions) section in the documentation for more information about the types of supported conditions.\n",
    "\n",
    "If the condition succeeds, we will register the model and deploy it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "bacaa9c6-22b0-48df-b138-95b6422fe834",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "condition_step = ConditionStep(\n",
    "    name=\"check-model-accuracy\",\n",
    "    conditions=[condition_gte],\n",
    "    if_steps=[\n",
    "        register_custom_model_step, deploy_step\n",
    "    ],\n",
    "    else_steps=[fail_step], \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ef087fd-d75a-4c85-9739-31b1a8f4e04f",
   "metadata": {},
   "source": [
    "## Step X - Updating dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "f0944d2e-33d9-472a-b1e8-3214c413c2b7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "step = tuning_step if USE_TUNING_STEP else training_step\n",
    "step.add_depends_on([data_quality_check_step])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c95a7905-2550-4979-b885-f2daabb5d45e",
   "metadata": {},
   "source": [
    "## Step 10 - Running the Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "4da5e453-acd8-47a0-a39f-264d05dd93d0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session6_pipeline = Pipeline(\n",
    "    name=\"penguins-session6-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        baseline_destination,\n",
    "        data_capture_enabled,\n",
    "        data_capture_percentage,\n",
    "        data_capture_destination,\n",
    "        data_quality_skip_check,\n",
    "        data_quality_register_new_baseline,\n",
    "        data_quality_supplied_baseline_statistics,\n",
    "        data_quality_supplied_baseline_constraints,\n",
    "        model_quality_skip_check,\n",
    "        model_quality_register_new_baseline,\n",
    "        model_quality_supplied_baseline_statistics,\n",
    "        model_quality_supplied_baseline_constraints,\n",
    "        evaluation_destination,\n",
    "        model_approval_status,\n",
    "        accuracy_threshold,\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_step, \n",
    "        data_quality_check_step,\n",
    "        tuning_step if USE_TUNING_STEP else training_step, \n",
    "        create_model_step,\n",
    "        transform_step,\n",
    "        model_quality_check_step,\n",
    "        evaluation_step,\n",
    "        condition_step\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24713751-b109-47be-8d6f-5f4c7511f6d4",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "b54dbf6e-9b52-41a6-bec3-91c56536b3a2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    }
   ],
   "source": [
    "session6_pipeline.upsert(role_arn=role)\n",
    "execution = session6_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41ad4f27-8ddb-44c8-b472-08fc69eab34e",
   "metadata": {},
   "source": [
    "## Step 13 - Cleaning up\n",
    "\n",
    "Before you finish, don't forget to clean up after yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "ab96ad3e-2160-40ad-a924-748baab86128",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'PipelineArn': 'arn:aws:sagemaker:us-east-1:325223348818:pipeline/penguins-session6-pipeline',\n",
       " 'ResponseMetadata': {'RequestId': 'c3b2aa9a-9440-4ea4-bad7-20c87734a68c',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'c3b2aa9a-9440-4ea4-bad7-20c87734a68c',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '94',\n",
       "   'date': 'Tue, 18 Apr 2023 18:07:31 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session6_pipeline.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d007635-494f-498d-8856-4553878905d4",
   "metadata": {},
   "source": [
    "## Resources\n",
    "\n",
    "* [SageMaker Inference Toolkit](https://github.com/aws/sagemaker-inference-toolkit)\n",
    "* [Amazon SageMaker Model Monitor](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_monitoring.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a0335ed-643d-4755-a1b2-ce129a8d7730",
   "metadata": {},
   "source": [
    "## Cleaning Up\n",
    "\n",
    "Here we can do a more deep clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cda93bb-29f8-47ad-a25f-f7b21df90325",
   "metadata": {},
   "outputs": [],
   "source": [
    "def delete_pipeline(pipeline):\n",
    "    if pipeline:\n",
    "        pipeline.delete()\n",
    "\n",
    "delete_pipeline(session1_pipeline)\n",
    "delete_pipeline(session2_pipeline)\n",
    "delete_pipeline(session3_pipeline)\n",
    "delete_pipeline(session4_pipeline)\n",
    "delete_pipeline(session5_pipeline)\n",
    "delete_pipeline(session6_pipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8704c94-7a42-45f9-b006-74832f07cbc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's delete every model we registered under our model package group\n",
    "for mp in sagemaker_client.list_model_packages(ModelPackageGroupName=model_package_group_name)[\"ModelPackageSummaryList\"]:\n",
    "    print(f\"Deleting {mp['ModelPackageArn']}\")\n",
    "    sagemaker_client.delete_model_package(ModelPackageName=mp[\"ModelPackageArn\"])\n",
    "\n",
    "# We can now delete the model package group.    \n",
    "sagemaker_client.delete_model_package_group(ModelPackageGroupName=model_package_group_name)"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (TensorFlow 2.6 Python 3.8 CPU Optimized)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/tensorflow-2.6-cpu-py38-ubuntu20.04-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}