TicTacToe_Agent+(1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rznSDgbvGggG"
   },
   "source": [
    "## Tic-Tac-Toe Agent\n",
    "​\n",
    "In this notebook, you will learn to build an RL agent (using Q-learning) that learns to play Numerical Tic-Tac-Toe with odd numbers. The environment is playing randomly with the agent, i.e. its strategy is to put an even number randomly in an empty cell. The following is the layout of the notebook:\n",
    "        - Defining epsilon-greedy strategy\n",
    "        - Tracking state-action pairs for convergence\n",
    "        - Define hyperparameters for the Q-learning algorithm\n",
    "        - Generating episode and applying Q-update equation\n",
    "        - Checking convergence in Q-values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8eDb8PxBGggH"
   },
   "source": [
    "#### Importing libraries\n",
    "Write the code to import Tic-Tac-Toe class from the environment file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6SFNYceFGggJ"
   },
   "outputs": [],
   "source": [
    "from TCGame_Env1 import TicTacToe #- import your class from environment file\n",
    "import collections\n",
    "import numpy as np\n",
    "import random\n",
    "import pickle\n",
    "import time\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wYLQyopEG8nz"
   },
   "outputs": [],
   "source": [
    "# Function to convert state array into a string to store it as keys in the dictionary\n",
    "# states in Q-dictionary will be of form: x-4-5-3-8-x-x-x-x\n",
    "#   x | 4 | 5\n",
    "#   ----------\n",
    "#   3 | 8 | x\n",
    "#   ----------\n",
    "#   x | x | x\n",
    "\n",
    "def Q_state(state):\n",
    "\n",
    "    return ('-'.join(str(e) for e in state)).replace('nan','x')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZebMOoiVHBBr"
   },
   "outputs": [],
   "source": [
    "# Defining a function which will return valid (all possible actions) actions corresponding to a state\n",
    "# Important to avoid errors during deployment.\n",
    "\n",
    "def valid_actions(state):\n",
    "\n",
    "    valid_Actions = []\n",
    "    \n",
    "    valid_Actions = [i for i in env.action_space(state)[0]] ###### -------please call your environment as env\n",
    "    return valid_Actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IRciPUkYHDWf"
   },
   "outputs": [],
   "source": [
    "# Defining a function which will add new Q-values to the Q-dictionary. \n",
    "def add_to_dict(state):\n",
    "    state1 = Q_state(state)\n",
    "    \n",
    "    valid_act = valid_actions(state)\n",
    "    if state1 not in Q_dict.keys():\n",
    "        for action in valid_act:\n",
    "            Q_dict[state1][action]=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fNNi_EfHGggM"
   },
   "source": [
    "#### Epsilon-greedy strategy - Write your code here\n",
    "\n",
    "(you can build your epsilon-decay function similar to the one given at the end of the notebook)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m0lMfqiJGggN"
   },
   "outputs": [],
   "source": [
    "# Defining epsilon-greedy policy. You can choose any function epsilon-decay strategy\n",
    "\n",
    "def epsilon_greedy(state, time):\n",
    "    epsilon = min_epsilon + (max_epsilon - min_epsilon) * np.exp(-0.000001*time)\n",
    "    z = np.random.random()\n",
    "    if z > epsilon:\n",
    "        state1 = Q_state(state)\n",
    "        action = max(Q_dict[state1],key=Q_dict[state1].get)\n",
    "    else:\n",
    "        agent_actions, env_actions = env.action_space(state)\n",
    "        action = random.choice(list(agent_actions))\n",
    "    return action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "H2kyQHOMGggR"
   },
   "source": [
    "#### Tracking the state-action pairs for checking convergence - write your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qcxZ29vdGggS"
   },
   "outputs": [],
   "source": [
    "# Initialise Q_dictionary as 'Q_dict' and States_tracked as 'States_track' (for convergence)\n",
    "Q_dict = collections.defaultdict(dict)\n",
    "States_track = collections.defaultdict(dict)\n",
    "rewards_tracked =  {(2,1):0,(5,5):0, (8,3): 0, (9,7):0}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vs73iv8fHOxV"
   },
   "outputs": [],
   "source": [
    "# Initialise states to be tracked\n",
    "def initialise_tracking_states():\n",
    "    sample_q_values = [('x-x-x-x-x-x-x-x-x',(6,5)),('x-x-x-x-x-x-x-x-x',(1,9)),\n",
    "                       ('x-3-x-x-1-x-x-x-x',(7,5)),('x-5-x-x-x-x-5-7-x',(8,2))]\n",
    "    for q_value in sample_q_values:\n",
    "        state = q_value[0]\n",
    "        action = q_value[1]\n",
    "        state1 = Q_state(state)\n",
    "        States_track[state1][action] = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dAbwJDMVHpwl"
   },
   "outputs": [],
   "source": [
    "#Defining a function to save the Q-dictionary as a pickle file\n",
    "\n",
    "def save_obj(obj, name ):\n",
    "    with open(name + '.pkl', 'wb') as f:\n",
    "        pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6Pyj7nMVHsBi"
   },
   "outputs": [],
   "source": [
    "def save_tracking_states():\n",
    "    for state in States_track.keys():\n",
    "        for action in States_track[state].keys():\n",
    "            if state in Q_dict and action in Q_dict[state]:\n",
    "                States_track[state][action].append(Q_dict[state][action])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B_8xSluUHvew"
   },
   "outputs": [],
   "source": [
    "initialise_tracking_states()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-iPt--E9GggV"
   },
   "source": [
    "#### Define hyperparameters  ---write your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "G0_f5czFGggW"
   },
   "outputs": [],
   "source": [
    "EPISODES = 20000\n",
    "LR = 0.01                   #learning rate\n",
    "GAMMA = 0.91\n",
    "\n",
    "max_epsilon = 1.0\n",
    "min_epsilon = 0.001\n",
    "\n",
    "threshold = 2000      \n",
    "policy_threshold = 30000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Md6twJ7wGggh"
   },
   "source": [
    "### Q-update loop ---write your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ldCgQuDNGggj"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34.35876989364624\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "for episode in range(EPISODES):\n",
    "    ##### Start writing your code from the next line\n",
    "    env = TicTacToe()\n",
    "    curr_state = env.state\n",
    "    isTerminated = False\n",
    "    add_to_dict(curr_state)\n",
    "    total_reward = 0\n",
    "\n",
    "    while not isTerminated:\n",
    "        current_state_ele = Q_state(curr_state)\n",
    "        curr_action = epsilon_greedy(curr_state, episode)\n",
    "        next_state, reward, isTerminated = env.step(curr_state, curr_action)\n",
    "\n",
    "        next_state_ele = Q_state(next_state)\n",
    "        add_to_dict(next_state)\n",
    "\n",
    "        if isTerminated:\n",
    "            Q_dict[current_state_ele][curr_action] += LR * (\n",
    "                (reward - Q_dict[current_state_ele][curr_action]))\n",
    "        else:\n",
    "            max_next = max(Q_dict[next_state_ele],\n",
    "                           key=Q_dict[next_state_ele].get)\n",
    "            Q_dict[current_state_ele][curr_action] += LR * (\n",
    "                (reward + (GAMMA * (Q_dict[next_state_ele][max_next]))) -\n",
    "                Q_dict[current_state_ele][curr_action])\n",
    "\n",
    "        curr_state = next_state\n",
    "        total_reward += reward\n",
    "\n",
    "    #TRACKING Q-VALUES\n",
    "    if (episode == threshold-1):        #at the 1999th episode\n",
    "        initialise_tracking_states()\n",
    "\n",
    "    if ((episode+1) % threshold) == 0:   #every 2000th episode\n",
    "        save_tracking_states()\n",
    "        save_obj(States_track,'States_tracked')\n",
    "\n",
    "    #SAVING POLICY\n",
    "    if ((episode+1)% policy_threshold ) == 0:  #every 30000th episodes, the Q-dict will be saved\n",
    "        save_obj(Q_dict,'Policy')\n",
    "\n",
    "elapsed_time = time.time() - start_time\n",
    "save_obj(States_track, 'States_tracked')\n",
    "save_obj(Q_dict, 'Policy')\n",
    "print(elapsed_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "t6eMFbb8Ggg2"
   },
   "source": [
    "#### Check the Q-dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fr9d2fcVGgg4"
   },
   "outputs": [],
   "source": [
    "#Q_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "F1tnDJWkGgg9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "47289"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(Q_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cFgUqfcQGghB"
   },
   "outputs": [],
   "source": [
    "# try checking for one of the states - that which action your agent thinks is the best  -----This will not be evaluated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KGPZEQDFGghG"
   },
   "source": [
    "#### Check the states tracked for Q-values convergence\n",
    "(non-evaluative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9s1Tvz8HGghH"
   },
   "outputs": [],
   "source": [
    "# Write the code for plotting the graphs for state-action pairs tracked"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pVQInsg7GghL"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "b2Opp8_NITkC"
   },
   "source": [
    "### Epsilon - decay check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gQ_D_JsuGghR"
   },
   "outputs": [],
   "source": [
    "max_epsilon = 1.0\n",
    "min_epsilon = 0.001\n",
    "time = np.arange(0,5000000)\n",
    "epsilon = []\n",
    "for i in range(0,5000000):\n",
    "    epsilon.append(min_epsilon + (max_epsilon - min_epsilon) * np.exp(-0.000001*i))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 347
    },
    "colab_type": "code",
    "id": "J7c2xADQGghV",
    "outputId": "cb60fce3-570b-45fb-bd83-abde3d13b273"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEFCAYAAADzHRw3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgo0lEQVR4nO3deXxU9b3/8dcnk5UsBLJAyAJhVXYkgGJRu6igrWivti7V6tVS+qtdf78u997+2tvb5XeX3nu9bV2qVWtvrdRWr+JS0dZdyhJkkbBogEDClpAQsu/f3x8zYhoCGWAmJzPzfj4e55GZc05m3qPy5vidc77HnHOIiEjki/M6gIiIhIYKXUQkSqjQRUSihApdRCRKqNBFRKJEvFdvnJ2d7caNG+fV24uIRKQNGzYccc7l9LfNs0IfN24cpaWlXr29iEhEMrO9J9umIRcRkSihQhcRiRIqdBGRKKFCFxGJEip0EZEoMWChm9lDZlZtZltPst3M7KdmVm5mW8zsvNDHFBGRgQRzhP4rYPEpti8BJgWWZcC9Zx9LRERO14CF7px7Hag7xS5LgV87vzVAppnlhSpgX+8dbuSfntlGe1d3uN5CRCQihWIMPR+o7PW8KrDuBGa2zMxKzay0pqbmjN6s8mgLD721h9W7as/o90VEolUoCt36WdfvXTOcc/c750qccyU5Of1euTqghROySU308WLZ4TP6fRGRaBWKQq8CCns9LwAOhOB1+5Wc4OOSc3J5adthunt0tyURkfeFotBXArcEznY5HzjmnDsYgtc9qcunjeZIUzsb9x0N59uIiESUASfnMrPHgEuAbDOrAr4HJAA45+4DngeuAMqBFuC2cIV93yVTckjwGavKDlEybmS4305EJCIMWOjOuRsG2O6AL4YsURAykhNYOCGbVWWH+fsrzsWsv2F8EZHYErFXil4+bTT76lrYebjR6ygiIkNCxBb6x6bmYgartupsFxERiOBCz01P5ryiEawqO+R1FBGRISFiCx3g8mmj2Hawgcq6Fq+jiIh4LsILfTQAL27TsIuISEQX+tisVM4Zna5hFxERIrzQAS6bNpr1FXVUN7Z5HUVExFMRX+gfn5mHc7Bqq47SRSS2RXyhTx6VzqTcNJ7dEtbZBkREhryIL3SAK2fmsa6ijuoGDbuISOyKjkKf4R92+aOGXUQkhkVFoU8alc6UUek8p2EXEYlhUVHo4B92Wb+3jkPHNOwiIrEpagr9iuPDLjpKF5HYFDWFPjE3jXNGa9hFRGJX1BQ6+M9JL917lIPHWr2OIiIy6KKq0K+YkQfA8+/obBcRiT1RVejjc9KYmpfBM5vDdo9qEZEhK6oKHeCq2WPYVFnP3tpmr6OIiAyq6Cv0WWMwg6c26ihdRGJL1BX6mMwUFhSP5OlN+/Hfv1pEJDZEXaEDXDMnn91HmtlSdczrKCIigyYqC33x9DwSfXE8tWm/11FERAZNVBb68JQEPnpuLs9sPkBXd4/XcUREBkVUFjrA0tn5HGnq4K1dtV5HEREZFFFb6B8+J4eM5Hie2qhhFxGJDVFb6EnxPq6cmceqskO0dHR5HUdEJOyittABrp6dT0tHNy9tO+x1FBGRsIvqQp83biT5mSn8YUOV11FERMIuqgs9Ls74m/PyebP8CAfqNQOjiES3qC50gGvnFuIcPPm2jtJFJLoFVehmttjMdppZuZl9u5/tw83sGTPbbGZlZnZb6KOemaKsYZw/fiS/31ClqQBEJKoNWOhm5gPuBpYAU4EbzGxqn92+CGxzzs0CLgH+3cwSQ5z1jH2qpJC9tS2s21PndRQRkbAJ5gh9PlDunNvtnOsAVgBL++zjgHQzMyANqAOGzLmCS6bnkZYUz+/15aiIRLFgCj0fqOz1vCqwrrefA+cCB4B3gK8450645t7MlplZqZmV1tTUnGHk05eS6OMTs/J4bstBmtqHzN8zIiIhFUyhWz/r+g5GXw5sAsYAs4Gfm1nGCb/k3P3OuRLnXElOTs5pRj0715UU0trZzfO6ibSIRKlgCr0KKOz1vAD/kXhvtwFPOr9yYA9wTmgihsacwkwm5KTyeGnlwDuLiESgYAp9PTDJzIoDX3ReD6zss88+4KMAZjYKmALsDmXQs2VmfKqkkNK9R9ld0+R1HBGRkBuw0J1zXcCdwCpgO/C4c67MzJab2fLAbj8AFprZO8CfgW85546EK/SZuua8fHxxxu/W6yhdRKJPfDA7OeeeB57vs+6+Xo8PAJeFNlro5aYnc+m5o/j9hiq+ftlkkuJ9XkcSEQmZqL9StK+bzi+irrmDF7Ye8jqKiEhIxVyhXzghm7FZw3h07T6vo4iIhFTMFXpcnHHj/CLW7anjvcONXscREQmZmCt0gGvnFpDoi9NRuohElZgs9Ky0JBZPH82Tb1fR2tHtdRwRkZCIyUIHuGlBEQ1tXTy7pe81UiIikSlmC31+8Ugm5qZp2EVEokbMFrqZcdOCIjZV1rN1/zGv44iInLWYLXSAT55XQEqCj0dWV3gdRUTkrMV0oQ9PSeBv5ubz9OYD1Da1ex1HROSsxHShA9y6cBwdXT38VmPpIhLhYr7QJ+ams2hSNv+9Zi+d3Sfck0NEJGLEfKED/O2FxVQ3tvP8O7r5hYhELhU6cPHkHIqzU3n4rQqvo4iInDEVOv75XT57wVg2Vdazcd9Rr+OIiJwRFXrAtSWFpCXF8yudwigiEUqFHpCWFM91JQU8t+Ugh461eR1HROS0qdB7uW1hMT3O8fDqPV5HERE5bSr0XoqyhnHFjDx+u2YfDW2dXscRETktKvQ+Pn/RBBrbu3hMFxqJSIRRofcxo2A4F07M4qG39tDepbnSRSRyqND78fmLJnC4oZ2nN2mudBGJHCr0fiyalM25eRnc//puenqc13FERIKiQu+HmbH84vGUVzfx8o5qr+OIiARFhX4SV8zIIz8zhfte2+V1FBGRoKjQTyLBF8cdi4op3XuUtbtrvY4jIjIgFfopXD+viOy0RH72crnXUUREBqRCP4WURB/LLhrPm+VH2LBXk3aJyNCmQh/ATQvGMjI1kZ+9/J7XUURETkmFPoDUpHjuWFTMqztr2FxZ73UcEZGTUqEH4ZYLxjE8JUFj6SIypAVV6Ga22Mx2mlm5mX37JPtcYmabzKzMzF4LbUxvpSXFc/uHivnT9sNs3X/M6zgiIv0asNDNzAfcDSwBpgI3mNnUPvtkAvcAVznnpgHXhT6qtz67cBzpyfH8XEfpIjJEBXOEPh8od87tds51ACuApX32uRF40jm3D8A5F3WXVw5PSeC2C4t5oeyQjtJFZEgKptDzgcpez6sC63qbDIwws1fNbIOZ3dLfC5nZMjMrNbPSmpqaM0vsoTsWFTM8JYF/f3Gn11FERE4QTKFbP+v6zlgVD8wFrgQuB/6vmU0+4Zecu985V+KcK8nJyTntsF7LSE7gC5dM4JWdNayvqPM6jojIXwmm0KuAwl7PC4C+88pWAS8455qdc0eA14FZoYk4tHz2gnHkpCfxry/swDnNxCgiQ0cwhb4emGRmxWaWCFwPrOyzz9PAIjOLN7NhwAJge2ijDg0piT6+/JGJrK84ymvvRt6wkYhErwEL3TnXBdwJrMJf0o8758rMbLmZLQ/ssx14AdgCrAN+6ZzbGr7Y3vr0vCIKRqTwb6t2ar50ERkyzKthg5KSEldaWurJe4fCExuq+N+/38w9N53HFTPyvI4jIjHCzDY450r626YrRc/Q1XPymZSbxk9e3Elnd4/XcUREVOhnyhdnfHPxOeyuaeaxdfu8jiMiokI/Gx87N5fzx4/krj+9R0Nbp9dxRCTGqdDPgpnxnSuncrSlg7tf0ZQAIuItFfpZmp4/nGvm5PPwmxVU1rV4HUdEYpgKPQS+cfkU4uLgX1dpSgAR8Y4KPQTyhqfwuUXjeWbzATbu063qRMQbKvQQ+fzFE8hOS+IHz27TlAAi4gkVeoikJcXzzcun8Pa+ep58e7/XcUQkBqnQQ+jauQXMLszk//1xh05jFJFBp0IPobg445+WTqO2uZ27XnrP6zgiEmNU6CE2syCTG+YX8chfKthxqMHrOCISQ1ToYfCNy6aQnhzPd58u0xekIjJoVOhhMCI1kW9cPoV1e+pYubnvvUBERMJDhR4m188rYmbBcH743HaOteoLUhEJPxV6mPjijB9dPYPapnb++Y87vI4jIjFAhR5GMwqGc/uHinls3T7W7K71Oo6IRDkVeph97dLJFIxI4e+ffIe2zm6v44hIFFOhh9mwxHh+fM0Mdh9p1hS7IhJWKvRBcNHkHK6Zk8+9r+5i56FGr+OISJRSoQ+S71x5LunJ8XzriS109+jcdBEJPRX6IMlKS+Ifr5rGpsp6fvH6Lq/jiEgUUqEPoqtmjWHJ9NH850vvaloAEQk5FfogMjN+ePV0hqck8PXfbaajq8frSCISRVTogywrLYkfXzODbQcb+PnLmpFRREJHhe6By6aN5m/OK+DuV3exubLe6zgiEiVU6B757iemkpuexNce30RLR5fXcUQkCqjQPTI8JYGfXDeLPUea+adntnkdR0SigArdQxdOzGb5xRNYsb6SZ7doml0ROTsqdI99/dLJzCnK5O+efIfKuhav44hIBFOheyzBF8dPr58DDr68YiOd3TqVUUTOTFCFbmaLzWynmZWb2bdPsd88M+s2s2tDFzH6FY4cxo8/OYON++q560/veh1HRCLUgIVuZj7gbmAJMBW4wcymnmS/fwFWhTpkLPjErDF8uqSQe17dxas7q72OIyIRKJgj9PlAuXNut3OuA1gBLO1nvy8BTwBqozP0j1dNY8qodL6yYpPG00XktAVT6PlAZa/nVYF1x5lZPnANcN+pXsjMlplZqZmV1tTUnG7WqJeS6OMXN8+lxzm+8OgG3RBDRE5LMIVu/azrO//rXcC3nHOnbCDn3P3OuRLnXElOTk6QEWPL2KxU/vNTs9m6v4HvPV3mdRwRiSDBFHoVUNjreQHQ96TpEmCFmVUA1wL3mNnVoQgYiz42dRRf+shEfldayYp1+7yOIyIRIj6IfdYDk8ysGNgPXA/c2HsH51zx+4/N7FfAs865p0IXM/Z89WOT2VRZz3dXljFldDpzikZ4HUlEhrgBj9Cdc13AnfjPXtkOPO6cKzOz5Wa2PNwBY5Uvzviv6+cwOiOZZf+9gQP1rV5HEpEhzpzz5nZoJSUlrrS01JP3jiTvHm7kk/espmjkMP7whQsYlhjM/1SJSLQysw3OuZL+tulK0SFu8qh0fnbjHHYcauBrv9tEj+5HKiInoUKPAB+ekss/XDmVVWWH+feXdnodR0SGKP3/e4T42wvHUV7dyN2v7KI4O41r5xZ4HUlEhhgVeoQwM75/1XT21rbw7Se2kJ2WyCVTcr2OJSJDiIZcIkhifBy/uHkuk0al878efZstVfVeRxKRIUSFHmHSkxN45LZ5jBiWyG0Pr6fiSLPXkURkiFChR6DcjGR+fft8epzjsw+vo6ax3etIIjIEqNAj1IScNB68dR6HG9q49eF1HGvt9DqSiHhMhR7Bzisawb2fmcu7hxu59eF1NLV3eR1JRDykQo9wH56Sy89uOI8tVce4/Vfrae3QlLsisUqFHgUWTx/Nf3xqFusq6lj236W0d6nURWKRCj1KLJ2dz798ciZvvHeELz66kY4u3WxaJNao0KPIp+YV8oOl0/jT9sMs/43ueCQSa1ToUebmC8bxo2um8/KOaj7361KNqYvEEBV6FLppwVh+ct0s3io/orNfRGKICj1KXTu3gLuun0Pp3qPc8uBaGtp0nrpItFOhR7GrZo3h7hvP4539x/j0L9ZQ3dDmdSQRCSMVepRbPH00D906j321zXzy3tXsqmnyOpKIhIkKPQYsmpTDimUX0NrRzbX3rmbjvqNeRxKRMFChx4gZBcN54gsLyUhJ4MYH1vLyjsNeRxKREFOhx5Bx2an8YflCJuam8blfb+CR1RV4dZNwEQk9FXqMyUlPYsWy8/nwlFy+t7KM7zy1lc5uXVUqEg1U6DEoNSmeX9w8l89fPJ5H1+7jsw+to76lw+tYInKWVOgxyhdn/N2Sc/nJdbMorTjK1Xe/RXm1zoARiWQq9Bh37dwCfvu5BTS2dXH13W/xx3cOeh1JRM6QCl0oGTeSlV/6EBNz0/jCo2/zo+e2aVxdJAKp0AWA/MwUHv/8BXz2grE88MYebnxgDYd1ZalIRFGhy3GJ8XF8f+l0/uv62Wzd38CVP32TN9874nUsEQmSCl1OsHR2Pk/feSGZwxL4zINr+dFz23QXJJEIoEKXfk0elc4zd36Iz5xfxANv7OGT96zWWTAiQ1xQhW5mi81sp5mVm9m3+9l+k5ltCSyrzWxW6KPKYEtJ9PHDq2fwwC0lHDzWxsd/9gaPrt2rq0tFhqgBC93MfMDdwBJgKnCDmU3ts9se4GLn3EzgB8D9oQ4q3rl06ihe+Moi5o0byT/8z1ZueWgd++tbvY4lIn0Ec4Q+Hyh3zu12znUAK4ClvXdwzq12zr0/hd8aoCC0McVruRnJPHLbfH6wdBob9h7lsv94jd+s2UtPj47WRYaKYAo9H6js9bwqsO5kbgf+eDahZGiKizNuvmAcq756EbOLMvnOU1u56Zdr2Vfb4nU0ESG4Qrd+1vV7WGZmH8Zf6N86yfZlZlZqZqU1NTXBp5QhpXDkMH5z+wJ+fM0M3tl/jMvvep17X91FR5cuRhLxUjCFXgUU9npeABzou5OZzQR+CSx1ztX290LOufudcyXOuZKcnJwzyStDhJlx44IiXvzaRSyalM2/vLCDK376Bqt36bx1Ea8EU+jrgUlmVmxmicD1wMreO5hZEfAkcLNz7t3Qx5ShakxmCvffUsJDt5bQ3tXNjQ+s5asrNlLdqKtMRQZb/EA7OOe6zOxOYBXgAx5yzpWZ2fLA9vuA7wJZwD1mBtDlnCsJX2wZaj5yzigWTsjmnlfKue+13fx5ezVf/MhEbl04juQEn9fxRGKCeXVOcUlJiSstLfXkvSW89hxp5ofPbuPPO6rJz0zhW0vO4RMz8wj8ZS8iZ8HMNpzsgFlXikrIFWen8uCt8/jtHQsYnpLAlx/byNX3rKa0os7raCJRTYUuYbNwYjbPfOlD/Nu1Mzl0rJVr7/sLdzyynrIDx7yOJhKVNOQig6Klo4sH39jDA2/spqGtiyXTR/O1SyczeVS619FEIsqphlxU6DKojrV28uCbe3jozT00d3TxiZlj+PJHJzExN83raCIRQYUuQ87R5g4eeGM3v1pdQWtnN4unjWb5xROYVZjpdTSRIU2FLkPWkaZ2fvVWBb/+SwUNbV0snJDF8osnsGhSts6KEemHCl2GvKb2Lh5bu49fvrmbww3tTBuTwR2LirliRh5J8TqPXeR9KnSJGO1d3Ty98QC/eH0Xu2qayU5L4sb5hdx0/lhGZSR7HU/Ecyp0iTg9PY43y4/wyOoKXt5Zjc+My6eP5taF4ygZO0LDMRKzTlXoA176L+KFuDjjosk5XDQ5h721zfxmzV5+t76S57YcZFJuGp+eV8jVc/LJTkvyOqrIkKEjdIkYLR1drNx0gN+VVrJxXz3xccZHz83l0/MKuWhSDvE+XScn0U9DLhJ13jvcyOOllTz59n5qmzvITU9i6ewxXDUrn+n5GRqSkailQpeo1dHVw8s7qvnDhkpe3VlDV49jfHYqH581hqtmjdEFSxJ1VOgSE+pbOvjj1kOs3HSANXtqcQ6m5mVw5cw8Lp82WuUuUUGFLjGnuqGNZ7ccZOXmA2yqrAdgfE4ql00dzWXTRjG7IJO4OA3LSORRoUtMO1Dfyp+2H+bFssOs2V1LV48jJz2JS6eO4qPn5HL++CxSk3TCl0QGFbpIwLGWTl7ZWc2L2w7x6s4aWjq6SfAZJWNHctHkHC6enMO5een6UlWGLBW6SD/au7oprTjKa+/W8Pq7New41AhATnoSiyZl86GJ2SwYn0V+ZorHSUU+oEIXCcKhY228/p6/3N8sP0J9SycAhSNTWFCcxYLikZw/PovCkcM8TiqxTIUucpq6exw7DjWwdncda3bXsq6i7njB52emML94JOcVZTKnaARTRqeToIuaZJCo0EXOUk+P493qRtburmPtnlrW7TnKkaZ2AJIT4piZn8mcoveXEZpITMJGhS4SYs45qo62sqmyno376tlYeZSy/Q10dPcAMDojmWljMpg2JoOpY4YzbUwGBSNS9GWrnDVNziUSYmZG4chhFI4cxidmjQH8X7JuO9DAxn31bKmqp+xAA6/srKYncMyUkRzP1DEZTAsU/ORR6UzMTSM5QfO9S2io0EVCJCnex5yiEcwpGnF8XWtHNzsONbDtYANlB/zLb9bspb3LfyRvBkUjhzEpN41Jo9L9P3PTmZCbyrBE/fGU06P/YkTCKCXxxJLv6u5hz5Fm3j3cxHvVjbwX+PnauzV0dvsP5838X74WZ6dSnJ3K2KxUirOHMTYrlcIRw0iM15ewciIVusggi/fF+Y/GR6UDecfXd3b3sLe2OVDw/qXiSDP/8/Z+Gtu7ju8XZ5A/IoVxWamMzRrGuKxU8jNTyB+RwpjMFLJSEzVWH6NU6CJDRIIvjom56UzMTWdJr/XOOeqaO6iobaHiSDN7a5v9j2ubeXrTARrbuv7qdZIT4hiTmeIv+cAyJlD4ecOTyU1PJiVR4/bRSIUuMsSZGVlpSWSlJTF37Ii/2uac41hrJ1VHWzlQ38r++lb2H23lwDH/z+0HGzjS1HHCa2Ykx5ObkcyojCRGpSeTm5FMbnoSo95fl5FMTnqSvrCNMCp0kQhmZmQOSyRzWCLT84f3u09bZ/fxsj90rI3qxnaqG9o43NDO4cY21u6po7qx7fj4fW+piT5GpiUyMjWJrNRERqYmHv85MjWRrF7bstIS9UWux/RPXyTKJSf4GJ+Txvick88H75zjaEsn1Y2Bom9oo6axndqmDuqa26lt7uBwQxvbDzZQ29xBR+Asnb4S4+MYnpJARnI8w1MSTlgy+vx8f0lPjic1MV5TGp8lFbqIYGbHj7rPGX3qfZ1zNHd0U9fUwZHmduqaOqhr7qC2uYP61g4aWjs5Flhqmtopr2niWEsnje1dDHQdY2qij9SkeNKS4kkLlHxqUry/8JMC2xID2wL7pSbFk5roIznBR0qij5SEwJLoIyk+Lqa+IA6q0M1sMfBfgA/4pXPun/tst8D2K4AW4Fbn3NshzioiQ4CZ+Qs3KZ6irOAnKuvpcTS2dR0v+95LU3snTW1dNLV309zeRVNHF01tXTS3d1F1tIXm48+7j1+NG1xWSI7/oOiTE+J6PfYxLPGD8k9O8JEU7yMxPo6kwJIYH0eiL46khDgSfR9sSwwsx/fz+QL7fLDei5uWD1joZuYD7gYuBaqA9Wa20jm3rdduS4BJgWUBcG/gp4gIAHFxxvBhCQwflnBWr9PR1eMv/cDS3N5FS0c3rZ3dtHV20xp43NrZTVuvx60dPf7tgX0a27qoaWw//ry1o5v2rp7T+gvjlJ/XOP4XQmJ8HAk+/xLvM26cX8Qdi8aH5H16C+YIfT5Q7pzbDWBmK4ClQO9CXwr82vknhlljZplmluecOxjyxCIS0/xHx4mMSE0My+v39Dg6uv3F3tHV4y/5Xkt7V7f/Z3cP7Z299+vus98H2zq731/8r52dlhSW7MEUej5Q2et5FSceffe3Tz7wV4VuZsuAZQBFRUWnm1VEJOzi4ozkOF9EnrIZzCBPf98o9P1qI5h9cM7d75wrcc6V5OTkBJNPRESCFEyhVwGFvZ4XAAfOYB8REQmjYAp9PTDJzIrNLBG4HljZZ5+VwC3mdz5wTOPnIiKDa8AxdOdcl5ndCazCf9riQ865MjNbHth+H/A8/lMWy/Gftnhb+CKLiEh/gjoP3Tn3PP7S7r3uvl6PHfDF0EYTEZHToUmVRUSihApdRCRKqNBFRKKEuYFmywnXG5vVAHvP8NezgSMhjBMJ9Jljgz5zbDibzzzWOdfvhTyeFfrZMLNS51yJ1zkGkz5zbNBnjg3h+swachERiRIqdBGRKBGphX6/1wE8oM8cG/SZY0NYPnNEjqGLiMiJIvUIXURE+lChi4hEiYgrdDNbbGY7zazczL7tdZ5wM7OHzKzazLZ6nWWwmFmhmb1iZtvNrMzMvuJ1pnAzs2QzW2dmmwOf+fteZxoMZuYzs41m9qzXWQaDmVWY2TtmtsnMSkP++pE0hh64v+m79Lq/KXBDn/ubRhUzuwhown+Lv+le5xkMZpYH5Dnn3jazdGADcHWU/3s2INU512RmCcCbwFecc2s8jhZWZvZ1oATIcM593Os84WZmFUCJcy4sF1JF2hH68fubOuc6gPfvbxq1nHOvA3Ve5xhMzrmDzrm3A48bge34b2kYtZxfU+BpQmCJnKOtM2BmBcCVwC+9zhItIq3QT3bvUolSZjYOmAOs9ThK2AWGHzYB1cBLzrlo/8x3Ad8EejzOMZgc8KKZbQjcYzmkIq3Qg7p3qUQHM0sDngC+6pxr8DpPuDnnup1zs/HfwnG+mUXtEJuZfRyods5t8DrLILvQOXcesAT4YmBINWQirdB179IYERhHfgJ41Dn3pNd5BpNzrh54FVjsbZKwuhC4KjCmvAL4iJn9xttI4eecOxD4WQ38D/5h5JCJtEIP5v6mEuECXxA+CGx3zv2H13kGg5nlmFlm4HEK8DFgh6ehwsg593fOuQLn3Dj8f45fds59xuNYYWVmqYEv+TGzVOAyIKRnr0VUoTvnuoD372+6HXjcOVfmbarwMrPHgL8AU8ysysxu9zrTILgQuBn/UdumwHKF16HCLA94xcy24D9weck5FxOn8sWQUcCbZrYZWAc855x7IZRvEFGnLYqIyMlF1BG6iIicnApdRCRKqNBFRKKECl1EJEqo0EVEBsHpTrRnZp8ys22Bydp+G9Tv6CwXEZHwO52J9sxsEvA48BHn3FEzyw1cjHRKOkIXERkE/U20Z2YTzOyFwNwub5jZOYFNnwPuds4dDfzugGUOKnQRES/dD3zJOTcX+D/APYH1k4HJZvaWma0xs6CmgYgPU0gRETmFwORzC4Hf+2e7ACAp8DMemARcgn/OqjfMbHpgnp+TUqGLiHgjDqgPzLDZVxWwxjnXCewxs534C379QC8oIiKDLDAl9B4zuw78k9KZ2azA5qeADwfWZ+Mfgtk90Guq0EVEBsFJJtq7Cbg9MGFXGR/cgW0VUGtm24BXgG8452oHfA+dtigiEh10hC4iEiVU6CIiUUKFLiISJVToIiJRQoUuIhIlVOgiIlFChS4iEiX+P1tiUre4Y5glAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(time, epsilon)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "59BRf43IJiQ1"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "TicTacToe_Agent.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}