docs/on_the_fly_selector_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Purposes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial:\n",
    "* provides a theoretical description of adaptive selection;\n",
    "* demonstrates how to use `OnTheFlySelector` for a problem of forecasting large amount of time series;\n",
    "* suggests a set of metrics such that properties of on-the-fly selection are revealed by them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Background"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to avoid copying and pasting sentences from docstrings, let us extract and format all necessary information. This is done with a class that hides too specialized code under easy-to-read names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils_for_demo_of_on_the_fly_selector import Helper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Time series forecasting has a property that all observations are\n",
      "    ordered. Depending on position, behavior of series can vary and so\n",
      "    one method can yield better results at some moments while\n",
      "    another method can outperform it at some other moments. This is\n",
      "    the reason why adaptive selection is useful for a significant\n",
      "    number of series.\n"
     ]
    }
   ],
   "source": [
    "Helper().why_adaptive_selection()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To continue, read what `OnTheFlySelector` class is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    This class provides functionality for adaptive short-term\n",
      "    forecasting based on selection from a pool of models.\n",
      "\n",
      "    Fitting goes like this: an instance of the class applies all its\n",
      "    candidate models to each of time series from the learning sample,\n",
      "    evaluates models performances on specified number of last folds,\n",
      "    and, finally, selects winning model for each of the time series.\n",
      "    To predict future values of a time series, corresponding to it\n",
      "    winning model is used.\n",
      "\n",
      "    The class is designed for a case of plenty of time series and\n",
      "    plenty of simple forecasters - if so, it is too expensive to store\n",
      "    all forecasts in any place other than operating memory and it is\n",
      "    better to compute them on-the-fly and then store only selected\n",
      "    values.\n",
      "\n",
      "    As for terminology, simple forecaster means a forecaster that\n",
      "    has no fitting. By default, the class uses moving average,\n",
      "    moving median, and exponential moving average, but you can pass\n",
      "    your own simple forecaster to initialization of a new instance.\n",
      "\n",
      "    Selection is preferred over stacking, because base forecasters are\n",
      "    quite similar to each other and so they have many common mistakes.\n",
      "\n",
      "    Advantages of adaptive on-the-fly selection are as follows:\n",
      "    * It always produces sane results, abnormal over-forecasts or\n",
      "      under-forecasts are impossible;\n",
      "    * Each time series is tailored individually, this is not a model\n",
      "      that predicts for several time series without taking into\n",
      "      consideration their identities;\n",
      "    * Uses no external features and so can be used for modelling of\n",
      "      residuals of a more complicated model;\n",
      "    * Can be easily paralleled or distributed, can deal with thousands\n",
      "      of time series in one call.\n",
      "\n",
      "    Limitations of adaptive on-the-fly selection are as follows:\n",
      "    * Not suitable for time series that are strongly influenced by\n",
      "      external factors;\n",
      "    * Not suitable for non-stationary time series (e.g., time series\n",
      "      with trend or seasonality) until they are not made stationary;\n",
      "    * Long-term forecasts made with it converge to constants.\n",
      "\n",
      "    \n"
     ]
    }
   ],
   "source": [
    "Helper().what_is_on_the_fly_selector()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let us print the list of parameters that can be passed when a new instance of `OnTheFlySelector` class is being created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ":param candidates: Optional[Dict[Any, List[Dict[str, Any]]]] = None\n",
      "        forecasters to select from, mapping from instances of\n",
      "        regressors to kwargs of their initializations, default value\n",
      "        results in moving averages of ten distinct windows,\n",
      "        moving medians of eight distinct windows and exponential\n",
      "        moving averages of ten distinct half-lives\n",
      ":param evaluation_fn: Optional[Callable] = None\n",
      "        function that is used for selection of the best forecasters,\n",
      "        the bigger its value, the better is a forecaster, default is\n",
      "        negative mean squared error\n",
      ":param horizon: int = 1\n",
      "        number of steps ahead to be forecasted at each iteration,\n",
      "        default is 1\n",
      ":param n_evaluational_rounds: int = 1\n",
      "        number of iterations at each of which forecasters make\n",
      "        predictions; structure of rounds is determined as follows:\n",
      "        the next round is obtained from the preceding round by going\n",
      "        one step forward and the last round ends at ends of time\n",
      "        series; default value is 1, i.e., forecasters are evaluated\n",
      "        at the last `horizon` observations from each series.\n",
      ":param verbose: int = 0\n",
      "        if it is greater than 0, a progress bar with tried candidates\n",
      "        is shown, default is 0\n"
     ]
    }
   ],
   "source": [
    "Helper().which_parameters_does_on_the_fly_selector_have()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Statements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import datetime\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "from forecastonishing.selection.on_the_fly_selector import OnTheFlySelector\n",
    "from forecastonishing.selection.paralleling import (\n",
    "    fit_selector_in_parallel,\n",
    "    predict_with_selector_in_parallel\n",
    ")\n",
    "from forecastonishing.miscellaneous.metrics import (\n",
    "    overall_r_squared,\n",
    "    averaged_r_squared,\n",
    "    averaged_censored_mape\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Extraction and Brief Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset that is used here is a set of synthetic time series that are drawn from a generative model trained on lots of real-world time series, so the problem under consideration is quite realistic.\n",
    "\n",
    "First of all, download the dataset if it has not been downloaded before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unit</th>\n",
       "      <th>item</th>\n",
       "      <th>date</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-11-01</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-11-02</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-11-03</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-11-04</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-11-05</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   unit  item       date  value\n",
       "0     1     1 2017-11-01   14.0\n",
       "1     1     1 2017-11-02   11.0\n",
       "2     1     1 2017-11-03   15.0\n",
       "3     1     1 2017-11-04    8.0\n",
       "4     1     1 2017-11-05   10.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_to_dataset = 'time_series_dataset.csv'\n",
    "if os.path.isfile(path_to_dataset):\n",
    "    df = pd.read_csv(path_to_dataset, parse_dates=[2])\n",
    "else:\n",
    "    df = pd.read_csv(\n",
    "        \"https://docs.google.com/spreadsheets/\" +\n",
    "        \"d/1TF0bAf9wOpIXIvIsazMCLEoHQ1y6dTkYYdYRRleC5lM/export?format=csv\",\n",
    "        parse_dates=[2]\n",
    "    )\n",
    "    df.to_csv(path_to_dataset, index=False)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many time series are there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7949"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_time_series = len(df.groupby(['unit', 'item']))\n",
    "n_time_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "61.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df.index) / n_time_series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each time series includes two months of observations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us define some metrics, but before that a quick remark is here — so much attention is paid to this section, because it explains which properties one can expect from on-the-fly selection and which properties, conversely, one can not expect from it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An interesting combination is to use both $R^2$ coefficient computed in a batch for all time series and $R^2$ coefficient computed for each time series separately and then averaged over all of them. The former metric reports how well levels of different time series are grasped, whereas the latter one reports how well individual dynamics and deviations from a corresponding mean are predicted.\n",
    "\n",
    "In addition, MAPE (mean absolute percentage error) computed for each time series separately, censored from above at 100% level, and averaged over all time series, can be displayed too, because it shows how far predictions are from actual values in relative terms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see why these three metrics differ, look at the below example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "example_df = pd.DataFrame(\n",
    "    [[1, 2, 3],\n",
    "     [1, 4, 5],\n",
    "     [2, 10, 9],\n",
    "     [2, 9, 10]],\n",
    "    columns=['key', 'actual_value', 'prediction']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91061452513966479"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "overall_r_squared(example_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above metric is high, because two series from `example_df` have different levels and predictions are near the corresponding levels which means that variation across levels is reflected in predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.5"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "averaged_r_squared(example_df, ['key'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alas, this metric is negative, because variation around individual means is not reflected at all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24.027777777777779"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "averaged_censored_mape(example_df, ['key'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, value of the third metric is neither decent nor poor. Predictions are not too far from actual values — relative difference is about 24%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Launch Itself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "horizon = 3\n",
    "n_evaluational_rounds = 10\n",
    "\n",
    "# This is not a parameter of `OnTheFlySelector` or any of its methods.\n",
    "# It is introduced, because, by default, `OnTheFlySelector` does not\n",
    "# use older lags, so it allows filtering redundant observations.\n",
    "max_lag_to_use = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_test_frontier = df['date'].max() - datetime.timedelta(days=horizon-1)\n",
    "n_training_days = horizon + n_evaluational_rounds + max_lag_to_use + 1\n",
    "train_df = df[\n",
    "    (df['date'] < train_test_frontier) &\n",
    "    (df['date'] >= train_test_frontier - datetime.timedelta(days=n_training_days))\n",
    "]\n",
    "test_df = df[df['date'] >= train_test_frontier]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "selector = OnTheFlySelector(\n",
    "    horizon=horizon,\n",
    "    n_evaluational_rounds=n_evaluational_rounds,\n",
    "    verbose=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 28/28 [22:18<00:00, 51.35s/it]\n",
      "100%|██████████| 28/28 [22:19<00:00, 51.05s/it]\n",
      "100%|██████████| 28/28 [22:20<00:00, 51.60s/it]\n",
      "100%|██████████| 28/28 [22:20<00:00, 50.79s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 752 ms, sys: 164 ms, total: 916 ms\n",
      "Wall time: 22min 21s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "selector = fit_selector_in_parallel(\n",
    "    selector,\n",
    "    train_df,\n",
    "    name_of_target='value',\n",
    "    series_keys=['unit', 'item'],\n",
    "    n_processes=4\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 432 ms, sys: 64 ms, total: 496 ms\n",
      "Wall time: 1min 50s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "predictions_df = predict_with_selector_in_parallel(\n",
    "    selector,\n",
    "    train_df,\n",
    "    n_processes=4\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Sorting is necessary after parallel execution.\n",
    "evaluation_df = predictions_df.reset_index().sort_values(by=['unit', 'item', 'index'])\n",
    "evaluation_df['actual_value'] = test_df.sort_values(by=['unit', 'item', 'date'])['value'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.90524827491432336"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "overall_r_squared(evaluation_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since 1 is the maximal possible value of $R^2$, the above score is very good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-7.0785745955648114"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "averaged_r_squared(evaluation_df, ['unit', 'item'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Negative value indicates that true mean is better than the forecast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "37.5152389675739"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "averaged_censored_mape(evaluation_df, ['unit', 'item'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Censored MAPE about 38% is not a bad score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it can be seen, simple forecasters can not predict future dynamic of the time series under consideration, especially multiple steps ahead. However, cross-sectional variance is grasped almost perfectly. Also note that the most frequent value in the dataset is 0 which makes MAPE too pessimistic, because even very close to 0 positive forecast is evaluated as maximal error if actual value is 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you need more examples of how to use `OnTheFlySelector` class, please look at `tests/on_the_fly_selector_tests.py` file."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "forecastonishing_env",
   "language": "python",
   "name": "forecastonishing_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}