A-4,optuna可视化调参魔法指南.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a6086209-af84-4acc-9882-96896080051b",
   "metadata": {},
   "source": [
    "# A-4,optuna可视化调参魔法指南"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d69f27c",
   "metadata": {},
   "source": [
    "Optuna是一款开源的调参工具，github star数量超过7k, 是目前最受欢迎的调参框架之一。\n",
    "\n",
    "其主要优点如下：\n",
    "\n",
    "1，Optuna拥有许多非常先进的调参算法(如贝叶斯优化，遗传算法采样等)，这些算法往往可以在几十上百次的尝试过程中找到一个不可微问题的较优解。\n",
    "\n",
    "2，通过配合optuna-dashboard，可以可视化整个调参过程，从各个方面加深对问题的理解，这是一个令人心动的特性😋！\n",
    "\n",
    "\n",
    "另外，Optuna还有如下一些非常实用的特性：\n",
    "\n",
    "1，通过将搜索结果存储到sqlite或mysql、postgresql，Optuna支持断点续搜。\n",
    "\n",
    "2，Optuna支持剪枝策略，提前结束一些中间返回结果较差的采样点从而加快搜索进程。\n",
    "\n",
    "3，Optuna支持手动指定一些超参采样点，也可以添加已经计算过的采样点及其结果作为初始化样本点。\n",
    "\n",
    "4，Optuna提供ask and tell 接口模式，无需显式定义目标函数，直接在循环中调优超参。\n",
    "\n",
    "5，Optuna封装了非常丰富的基于plotly的可视化函数，便于分析调参结果。\n",
    "\n",
    "6，通过将搜索结果存储到mysql或postgresql，并设置分布式模式，Optuna支持多机分布式搜索，通过并行方式加快搜索进程。\n",
    "\n",
    "\n",
    "我们将首先展示一些非常实用的综合应用范例演示optuna在算法调优实践中的魔力。\n",
    "\n",
    "然后展示一些optuna的基础特性范例详细讲解optuna的主要特性和API应用方法。\n",
    "\n",
    "\n",
    "综合应用范例：\n",
    "\n",
    "一，optuna对pytorch模型调参范例\n",
    "\n",
    "二，optuna对tensorflow模型调参范例\n",
    "\n",
    "三，optuna对多模型加权融合范例\n",
    "\n",
    "\n",
    "基础特性范例：\n",
    "\n",
    "四，单参数空间搜索范例\n",
    "\n",
    "五，网格参数空间搜索范例\n",
    "\n",
    "六，断点续搜范例\n",
    "\n",
    "七，剪枝策略范例\n",
    "\n",
    "八，各种调参可视化函数范例\n",
    "\n",
    "九，手动添加超参数采样点范例\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef690b43",
   "metadata": {},
   "source": [
    "参考文档\n",
    "\n",
    "optuna官方文档： https://optuna.readthedocs.io/en/stable/tutorial/index.html\n",
    "\n",
    "optuna更多范例库：https://github.com/optuna/optuna-examples 【价值非常大，强烈建议参考】\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "832f1556",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "820f05b2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2464efce",
   "metadata": {},
   "source": [
    "### 〇，环境准备"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18914cee",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install optuna -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "!pip install optuna-dashboard -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "!pip install plotly -i https://pypi.tuna.tsinghua.edu.cn/simple\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3965f23",
   "metadata": {},
   "source": [
    "在后台异步执行打开 optuna-dashboard 可视化 监控页面，\n",
    "\n",
    "然后浏览器中输入：http://localhost:8083/dashboard/ 查看监控页面，类似tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "16475b96",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Listening on http://0.0.0.0:8083/\n",
      "Hit Ctrl-C to quit.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Process is interrupted.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "127.0.0.1 - - [02/Jan/2023 20:59:29] \"GET / HTTP/1.1\" 302 0\n",
      "127.0.0.1 - - [02/Jan/2023 20:59:29] \"GET /api/studies HTTP/1.1\" 200 137\n",
      "127.0.0.1 - - [02/Jan/2023 20:59:29] \"GET /favicon.ico HTTP/1.1\" 304 0\n",
      "127.0.0.1 - - [02/Jan/2023 20:59:39] \"DELETE /api/studies/1 HTTP/1.1\" 204 0\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "nohup optuna-dashboard --host 0.0.0.0  --port 8083 sqlite:///optuna.db & "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88892447",
   "metadata": {},
   "outputs": [],
   "source": [
    "#杀死 optuna-dashboard\n",
    "#!ps aux|grep optuna-dashboard \n",
    "#!kill -9 1161"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "06f60344",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "liangyun2         3650   0.0  0.0 408628368   1664 s004  S+    8:55下午   0:00.00 grep optuna-dashboard\n",
      "liangyun2         3648   0.0  0.0 408630256   2560 s004  Ss+   8:55下午   0:00.01 /bin/zsh -c ps aux|grep optuna-dashboard\n",
      "liangyun2         1737   0.0  0.9 410255136 145312   ??  S     8:43下午   0:02.87 /Users/liangyun2/miniforge3/envs/tf26/bin/python3.9 /Users/liangyun2/miniforge3/envs/tf26/bin/optuna-dashboard --host 0.0.0.0 --port 8083 sqlite:///optuna.db\n"
     ]
    }
   ],
   "source": [
    "!ps aux|grep optuna-dashboard "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "78d22240",
   "metadata": {},
   "outputs": [],
   "source": [
    "!kill -9 1737"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "050bcccf-f945-4678-800f-2d07643d5398",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "117e5912-38fd-4347-807d-54f99757da81",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b4804599",
   "metadata": {},
   "source": [
    "### 一，optuna对pytorch模型调参范例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f63ccf05",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import optuna\n",
    "from optuna.trial import TrialState\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import torch.utils.data\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "\n",
    "\n",
    "DEVICE = torch.device(\"cpu\")\n",
    "BATCHSIZE = 128\n",
    "CLASSES = 10\n",
    "DIR = os.getcwd()\n",
    "EPOCHS = 10\n",
    "N_TRAIN_EXAMPLES = BATCHSIZE * 30\n",
    "N_VALID_EXAMPLES = BATCHSIZE * 10\n",
    "\n",
    "\n",
    "def define_model(trial):\n",
    "    # We optimize the number of layers, hidden units and dropout ratio in each layer.\n",
    "    n_layers = trial.suggest_int(\"n_layers\", 1, 3)\n",
    "    layers = []\n",
    "\n",
    "    in_features = 28 * 28\n",
    "    for i in range(n_layers):\n",
    "        out_features = trial.suggest_int(\"n_units_{}\".format(i), 16, 64)\n",
    "        layers.append(nn.Linear(in_features, out_features))\n",
    "        layers.append(nn.ReLU())\n",
    "        p = trial.suggest_float(\"dropout_{}\".format(i), 0.1, 0.5)\n",
    "        layers.append(nn.Dropout(p))\n",
    "\n",
    "        in_features = out_features\n",
    "    layers.append(nn.Linear(in_features, CLASSES))\n",
    "    layers.append(nn.LogSoftmax(dim=1))\n",
    "\n",
    "    return nn.Sequential(*layers)\n",
    "\n",
    "\n",
    "def get_mnist():\n",
    "    # Load FashionMNIST dataset.\n",
    "    train_loader = torch.utils.data.DataLoader(\n",
    "        datasets.FashionMNIST(DIR, train=True, download=True, transform=transforms.ToTensor()),\n",
    "        batch_size=BATCHSIZE,\n",
    "        shuffle=True,\n",
    "    )\n",
    "    valid_loader = torch.utils.data.DataLoader(\n",
    "        datasets.FashionMNIST(DIR, train=False, transform=transforms.ToTensor()),\n",
    "        batch_size=BATCHSIZE,\n",
    "        shuffle=True,\n",
    "    )\n",
    "\n",
    "    return train_loader, valid_loader\n",
    "\n",
    "\n",
    "def objective(trial):\n",
    "\n",
    "    # Generate the model.\n",
    "    model = define_model(trial).to(DEVICE)\n",
    "\n",
    "    # Generate the optimizers.\n",
    "    optimizer_name = trial.suggest_categorical(\"optimizer\", [\"Adam\", \"RMSprop\", \"SGD\"])\n",
    "    lr = trial.suggest_float(\"lr\", 1e-5, 1e-1, log=True)\n",
    "    optimizer = getattr(optim, optimizer_name)(model.parameters(), lr=lr)\n",
    "\n",
    "    # Get the FashionMNIST dataset.\n",
    "    train_loader, valid_loader = get_mnist()\n",
    "\n",
    "    # Training of the model.\n",
    "    for epoch in range(EPOCHS):\n",
    "        model.train()\n",
    "        for batch_idx, (data, target) in enumerate(train_loader):\n",
    "            # Limiting training data for faster epochs.\n",
    "            if batch_idx * BATCHSIZE >= N_TRAIN_EXAMPLES:\n",
    "                break\n",
    "\n",
    "            data, target = data.view(data.size(0), -1).to(DEVICE), target.to(DEVICE)\n",
    "\n",
    "            optimizer.zero_grad()\n",
    "            output = model(data)\n",
    "            loss = F.nll_loss(output, target)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "        # Validation of the model.\n",
    "        model.eval()\n",
    "        correct = 0\n",
    "        with torch.no_grad():\n",
    "            for batch_idx, (data, target) in enumerate(valid_loader):\n",
    "                # Limiting validation data.\n",
    "                if batch_idx * BATCHSIZE >= N_VALID_EXAMPLES:\n",
    "                    break\n",
    "                data, target = data.view(data.size(0), -1).to(DEVICE), target.to(DEVICE)\n",
    "                output = model(data)\n",
    "                # Get the index of the max log-probability.\n",
    "                pred = output.argmax(dim=1, keepdim=True)\n",
    "                correct += pred.eq(target.view_as(pred)).sum().item()\n",
    "\n",
    "        accuracy = correct / min(len(valid_loader.dataset), N_VALID_EXAMPLES)\n",
    "\n",
    "        \n",
    "        #attention here \n",
    "        trial.report(accuracy, epoch)\n",
    "        # Handle pruning based on the intermediate value.\n",
    "        if trial.should_prune():\n",
    "            raise optuna.exceptions.TrialPruned()\n",
    "\n",
    "    return accuracy\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    storage_name = \"sqlite:///optuna.db\"\n",
    "    study = optuna.create_study(\n",
    "        pruner=optuna.pruners.MedianPruner(n_warmup_steps=3), direction=\"maximize\",\n",
    "        study_name=\"fashion_mnist_torch\", storage=storage_name,load_if_exists=True\n",
    "    )\n",
    "    \n",
    "    study.optimize(objective, n_trials=20, timeout=1200)\n",
    "\n",
    "    best_params = study.best_params\n",
    "    best_value = study.best_value\n",
    "    print(\"\\n\\nbest_value = \"+str(best_value))\n",
    "    print(\"best_params:\")\n",
    "    print(best_params)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e74b92d7",
   "metadata": {},
   "source": [
    "![](https://p.ipic.vip/4fjrpv.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea6987bd",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b48f79d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ed7a1147",
   "metadata": {},
   "source": [
    "### 二，optuna对tensorflow模型调参范例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf3031fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import optuna\n",
    "from optuna.integration import TFKerasPruningCallback\n",
    "from optuna.trial import TrialState\n",
    "\n",
    "import tensorflow as tf \n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "from tensorflow.keras.optimizers import RMSprop\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "\n",
    "N_TRAIN_EXAMPLES = 3000\n",
    "N_VALID_EXAMPLES = 1000\n",
    "BATCHSIZE = 128\n",
    "CLASSES = 10\n",
    "EPOCHS = 20\n",
    "\n",
    "\n",
    "def create_model(trial):\n",
    "    # We optimize the number of layers, hidden units and dropout in each layer and\n",
    "    # the learning rate of RMSProp optimizer.\n",
    "\n",
    "    # We define our MLP.\n",
    "    n_layers = trial.suggest_int(\"n_layers\", 1, 3)\n",
    "    model = keras.models.Sequential()\n",
    "    for i in range(n_layers):\n",
    "        num_hidden = trial.suggest_int(\"n_units_{}\".format(i), 4, 64, log=True)\n",
    "        model.add(layers.Dense(num_hidden, activation=\"relu\"))\n",
    "        dropout = trial.suggest_float(\"dropout_{}\".format(i), 0.1, 0.5)\n",
    "        model.add(layers.Dropout(rate=dropout))\n",
    "    model.add(layers.Dense(CLASSES, activation=\"softmax\"))\n",
    "\n",
    "    # We compile our model with a sampled learning rate.\n",
    "    learning_rate = trial.suggest_float(\"learning_rate\", 1e-5, 1e-1, log=True)\n",
    "    model.compile(\n",
    "        loss=\"categorical_crossentropy\",\n",
    "        optimizer=RMSprop(learning_rate=learning_rate),\n",
    "        metrics=[\"accuracy\"],\n",
    "    )\n",
    "\n",
    "    return model\n",
    "\n",
    "\n",
    "def objective(trial):\n",
    "    \n",
    "    tf.keras.backend.clear_session()\n",
    "\n",
    "    # The data is split between train and validation sets.\n",
    "    (x_train, y_train), (x_valid, y_valid) = keras.datasets.mnist.load_data()\n",
    "    x_train = x_train.reshape(60000, 784)[:N_TRAIN_EXAMPLES].astype(\"float32\") / 255\n",
    "    x_valid = x_valid.reshape(10000, 784)[:N_VALID_EXAMPLES].astype(\"float32\") / 255\n",
    "\n",
    "    # Convert class vectors to binary class matrices.\n",
    "    y_train = to_categorical(y_train[:N_TRAIN_EXAMPLES], CLASSES)\n",
    "    y_valid = to_categorical(y_valid[:N_VALID_EXAMPLES], CLASSES)\n",
    "\n",
    "    # Generate our trial model.\n",
    "    model = create_model(trial)\n",
    "\n",
    "    # Fit the model on the training data.\n",
    "    # The KerasPruningCallback checks for pruning condition every epoch.\n",
    "    model.fit(\n",
    "        x_train,\n",
    "        y_train,\n",
    "        batch_size=BATCHSIZE,\n",
    "        callbacks=[TFKerasPruningCallback(trial, \"val_accuracy\")],\n",
    "        epochs=EPOCHS,\n",
    "        validation_data=(x_valid, y_valid),\n",
    "        verbose=0,\n",
    "    )\n",
    "\n",
    "    # Evaluate the model accuracy on the validation set.\n",
    "    score = model.evaluate(x_valid, y_valid, verbose=0)\n",
    "    return score[1]\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    storage_name = \"sqlite:///optuna.db\"\n",
    "    study = optuna.create_study(\n",
    "        pruner=optuna.pruners.MedianPruner(n_warmup_steps=10), direction=\"maximize\",\n",
    "        study_name=\"mnist-tf\", storage=storage_name,load_if_exists=True\n",
    "    )\n",
    "    study.optimize(objective, n_trials=20)\n",
    "    \n",
    "    best_params = study.best_params\n",
    "    best_value = study.best_value\n",
    "    print(\"\\n\\nbest_value = \"+str(best_value))\n",
    "    print(\"best_params:\")\n",
    "    print(best_params)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c62aa6f2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f2642555",
   "metadata": {},
   "source": [
    "### 三，optuna对多模型加权融合范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1497e262",
   "metadata": {},
   "source": [
    "多模型加权融合是一个常见的提升机器学习效果的方案。\n",
    "\n",
    "但是各个模型的权重如何确定呢？\n",
    "\n",
    "有些方案是使用线性回归或者逻辑回归模型进行学习，这种方案一般叫做stacking ensemble，但是这种方案一般是对可微的Loss进行优化的，无法直接对auc,acc等不可微的评价指标进行优化。\n",
    "\n",
    "由于optuna是一个强大的不可微问题调优工具，我们可以使用它来寻找模型融合的权重，直接对auc,acc等不可微的评价指标进行优化，当给予足够的搜索次数时，其结果相比stacking ensemble通常更加有竞争力。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "90d5ec89",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "from copy import deepcopy\n",
    "\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.svm import SVC \n",
    "\n",
    "from sklearn.metrics import roc_auc_score \n",
    "\n",
    "# 一，准备数据\n",
    "data,target = make_classification(n_samples=3000,n_features=20,\n",
    "        n_informative=12,n_redundant=4,n_repeated=0,n_classes=2,\n",
    "        n_clusters_per_class=4)\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(data, target)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c21c2a53",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# 二，训练3个基础模型\n",
    "\n",
    "tree = DecisionTreeClassifier()\n",
    "mlp = MLPClassifier()\n",
    "svc = SVC(probability=True) \n",
    "\n",
    "mlp.fit(x_train,y_train)\n",
    "tree.fit(x_train,y_train)\n",
    "svc.fit(x_train,y_train);\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e38cb13-82ee-4997-9333-cbc9b0cb3c7b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "737d582f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mlp_score: 0.9671402373324008\n",
      "tree_score: 0.7599489085991965\n",
      "svc_score: 0.9673329004359895\n"
     ]
    }
   ],
   "source": [
    "# 三，评估单模型效果\n",
    "def get_test_auc(model):\n",
    "    probs = model.predict_proba(x_test)[:,1]\n",
    "    val_auc = roc_auc_score(y_test,probs)\n",
    "    return val_auc\n",
    "\n",
    "print(\"mlp_score:\",get_test_auc(mlp))\n",
    "print(\"tree_score:\",get_test_auc(tree))\n",
    "print(\"svc_score:\",get_test_auc(svc))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f52f16f5-7fdd-4f74-aa61-a2ab2f4302be",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n",
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n",
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n",
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n",
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n",
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stacking_score: 0.9720139002861405\n"
     ]
    }
   ],
   "source": [
    "# 四， stacking方案效果\n",
    "from sklearn.ensemble import StackingClassifier \n",
    "from sklearn.linear_model import LogisticRegression \n",
    "\n",
    "stacking = StackingClassifier(\n",
    "    estimators=[('mlp',mlp),('tree',tree),('svc',svc)],\n",
    "    final_estimator=LogisticRegression())\n",
    "\n",
    "stacking.fit(x_train,y_train)\n",
    "\n",
    "print(\"stacking_score:\",get_test_auc(stacking))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c21f7cb-7a4d-4151-adb3-7db3788cd9aa",
   "metadata": {},
   "source": [
    "![](./data/stacking融合.jpeg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cda6f57-9b49-4057-a97a-c72980f5ad61",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "77b0fdaf-28be-45d1-9336-a9149145219f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n",
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n",
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n",
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n",
      "/Users/liangyun2/miniforge3/envs/tf26/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:679: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# 五，获取CV预测结果\n",
    "\n",
    "# 为了充分利用训练数据集，采用类似stacking的方式，用5折CV的方式获取各个模型在训练集的预测结果\n",
    "\n",
    "def get_cv_preds(model,x_train,y_train):\n",
    "    \n",
    "    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)\n",
    "    cv_preds = np.zeros(len(y_train))\n",
    "    \n",
    "    for idx, (train_idx, valid_idx) in enumerate(cv.split(x_train, y_train)):\n",
    "        xtrain_i, xvalid_i = x_train[train_idx], x_train[valid_idx]\n",
    "        ytrain_i, yvalid_i = y_train[train_idx], y_train[valid_idx]\n",
    "        model_idx = deepcopy(model)\n",
    "        model_idx.fit(xtrain_i,ytrain_i)\n",
    "        probs_valid_idx = model_idx.predict_proba(xvalid_i)[:,1]\n",
    "        cv_preds[valid_idx] = probs_valid_idx\n",
    "    return cv_preds\n",
    "\n",
    "preds_cv = {name: get_cv_preds(eval(name),x_train,y_train)\n",
    "             for name in ['mlp','tree','svc']}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ba4b770d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "127.0.0.1 - - [02/Jan/2023 21:13:14] \"GET /dashboard HTTP/1.1\" 304 0\n",
      "127.0.0.1 - - [02/Jan/2023 21:13:14] \"GET /api/studies HTTP/1.1\" 200 137\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "best_value = 0.969764007134815\n",
      "best_params:\n",
      "{'mlp': 68, 'svc': 55, 'tree': 3}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "127.0.0.1 - - [02/Jan/2023 21:13:18] \"GET /api/studies/1?after=0 HTTP/1.1\" 200 207662\n",
      "127.0.0.1 - - [02/Jan/2023 21:13:23] \"GET /api/studies/1/param_importances?objective_id=0 HTTP/1.1\" 200 319\n",
      "127.0.0.1 - - [02/Jan/2023 21:13:29] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:13:39] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:13:50] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:14:01] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:14:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:14:22] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:14:33] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:14:43] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:14:55] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:15:08] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:15:19] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:15:31] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n"
     ]
    }
   ],
   "source": [
    "# 六， optuna搜索融合权重\n",
    "\n",
    "import optuna \n",
    "optuna.logging.set_verbosity(optuna.logging.WARNING)\n",
    "\n",
    "def objective(trial):\n",
    "    weights = {name:trial.suggest_int(name, 1, 100) for name in ['mlp','tree','svc']}\n",
    "    probs = sum([weights[name]*preds_cv[name] for name in ['mlp','tree','svc']])/sum(\n",
    "        [weights[name] for name in ['mlp','tree','svc']])\n",
    "    \n",
    "    cv_auc = roc_auc_score(y_train,probs)\n",
    "    trial.report(cv_auc, 0)\n",
    "    return cv_auc\n",
    "\n",
    "storage_name = \"sqlite:///optuna.db\"\n",
    "study = optuna.create_study(\n",
    "    direction=\"maximize\",\n",
    "    study_name=\"optuna_ensemble\", storage=storage_name,load_if_exists=False\n",
    ")\n",
    "study.optimize(objective, n_trials=500, timeout=600)\n",
    "\n",
    "best_params = study.best_params\n",
    "best_value = study.best_value\n",
    "print(\"\\n\\nbest_value = \"+str(best_value))\n",
    "print(\"best_params:\")\n",
    "print(best_params)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "aaa37176",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "optuna_ensemble_score: 0.9726846533134486\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "127.0.0.1 - - [02/Jan/2023 21:15:44] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:15:56] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:16:08] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:16:20] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:16:30] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:16:41] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:16:56] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:17:10] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:17:24] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:17:38] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:17:52] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:18:06] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:18:20] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:18:32] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:18:46] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:19:00] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:19:12] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:19:23] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:19:35] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:19:47] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:19:59] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:20:10] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:20:24] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:20:36] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:20:48] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:20:59] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:21:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:21:23] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:21:34] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:21:45] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:21:59] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:22:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:22:23] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:22:34] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:22:46] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:22:57] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:23:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:23:26] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:23:37] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:23:48] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:24:02] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:24:16] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:24:30] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:24:42] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:24:54] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:25:05] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:25:16] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:25:30] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:25:44] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:25:56] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:26:07] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:26:20] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:26:31] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:26:43] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:26:56] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:27:10] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:27:22] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:27:33] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:27:45] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:27:57] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:28:10] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:28:21] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:28:34] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:28:45] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:28:56] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:29:08] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:29:19] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:29:33] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:29:45] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:29:57] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:30:08] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:30:22] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:30:33] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:30:45] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:30:56] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:31:07] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:31:19] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:31:33] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:31:48] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:31:59] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:32:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:32:23] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:32:37] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:32:51] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:33:03] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:33:15] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:33:27] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:33:39] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:33:50] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:34:01] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:34:15] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:34:26] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:34:38] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:34:51] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:35:02] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:35:12] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:35:24] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:35:35] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:35:50] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:36:03] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:36:15] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:36:27] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:36:39] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:36:50] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:37:02] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:37:13] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:37:25] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:37:36] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:37:48] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:37:59] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:38:10] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:38:23] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:38:34] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:38:47] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:38:58] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:39:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:39:22] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:39:35] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:39:48] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:39:59] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:40:10] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:40:24] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:40:38] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:40:52] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:41:06] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:41:18] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:41:29] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:41:41] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:41:53] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:42:06] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:42:20] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:42:32] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:42:44] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:42:58] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:43:10] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:43:23] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:43:35] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:43:47] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:44:01] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:44:15] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:44:29] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:44:43] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:44:55] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:45:08] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:45:22] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:45:36] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:45:48] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:46:02] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:46:16] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:46:30] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:46:42] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:46:56] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:47:10] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:47:24] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:47:36] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:47:47] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:48:00] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:48:13] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:48:24] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:48:37] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:48:48] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:49:02] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:49:16] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:49:30] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:49:42] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:49:56] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:50:07] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:50:21] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:50:35] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:50:49] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:51:03] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:51:15] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:51:28] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:51:39] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:51:51] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:52:05] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:52:19] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:52:31] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:52:44] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:52:56] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:53:10] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:53:22] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:53:35] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:53:49] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:54:03] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:54:17] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:54:29] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:54:42] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:54:54] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:55:08] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:55:22] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:55:36] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:55:50] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:56:05] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:56:16] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:56:29] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:56:42] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:56:55] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:57:08] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:57:19] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:57:30] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:57:44] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:57:57] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:58:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:58:25] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:58:39] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:58:53] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:59:07] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:59:21] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:59:34] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:59:47] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 21:59:57] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:00:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:00:25] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:00:39] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:00:53] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:01:08] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:01:19] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:01:30] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:01:44] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:01:58] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:02:12] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:02:26] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:02:40] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:02:54] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:03:08] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:03:20] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:03:31] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:03:45] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:03:59] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:04:13] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:04:27] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:04:39] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:04:53] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:05:07] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:05:21] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:05:35] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:05:49] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:06:00] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:06:14] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:06:27] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:06:40] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:06:53] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:07:05] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:07:19] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:07:31] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:07:45] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:07:59] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:08:13] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:08:27] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:08:41] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:08:55] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:09:09] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:09:23] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:09:37] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:09:49] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:10:02] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:10:15] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:10:28] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:10:41] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:10:54] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:11:05] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:11:19] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:11:33] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:11:47] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:11:59] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:12:12] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:12:26] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:12:40] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:12:54] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:13:06] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:13:20] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:13:34] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:13:46] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:13:59] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:14:12] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:14:25] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:14:38] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:14:51] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:15:01] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:15:13] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:15:27] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:15:41] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:15:55] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:16:07] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:16:21] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:16:35] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:16:49] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:17:03] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:17:15] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:17:28] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:17:42] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:17:56] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:18:10] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:18:24] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:18:38] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:18:52] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:19:06] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:19:21] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:19:31] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:19:43] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:19:57] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:20:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:20:25] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:20:39] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:20:53] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:21:07] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:21:21] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:21:35] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:21:49] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:22:03] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:22:15] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:22:29] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:22:41] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:22:55] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:23:09] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:23:23] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:23:37] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:23:50] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:24:03] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:24:15] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:24:29] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:24:43] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:24:57] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:25:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:25:26] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:25:39] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:25:52] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:26:03] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:26:16] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:26:29] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:26:42] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:26:55] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:27:08] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:27:19] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:27:33] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:27:47] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:28:01] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:28:15] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:28:29] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:28:43] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:28:57] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:29:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:29:23] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:29:37] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:29:51] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:30:05] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:30:19] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:30:33] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:30:47] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:31:01] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:31:15] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:31:29] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:31:41] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:31:54] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:32:07] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:32:21] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:32:35] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:32:49] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:33:03] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:33:17] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:33:32] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:33:45] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:33:58] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:34:11] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:34:22] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:34:33] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:34:43] \"GET /api/studies/1?after=500 HTTP/1.1\" 200 915\n",
      "127.0.0.1 - - [02/Jan/2023 22:34:46] \"GET /api/studies HTTP/1.1\" 200 137\n"
     ]
    }
   ],
   "source": [
    "# 七， optuna权重融合效果\n",
    "preds_test = {name:(eval(name)).predict_proba(x_test)[:,1] for name in ['mlp','tree','svc']}\n",
    "def test_score(weights):\n",
    "    probs = sum([weights[name]*preds_test[name] for name in ['mlp','tree','svc']])/sum(\n",
    "        [weights[name] for name in ['mlp','tree','svc']])\n",
    "    test_auc = roc_auc_score(y_test,probs)\n",
    "    return test_auc\n",
    "print('optuna_ensemble_score:',test_score(best_params))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2a57d09-7571-4bc9-829b-542b7a77741c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "feaa8b4f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2443798c",
   "metadata": {},
   "source": [
    "以下范例为基础特性范例讲解"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fb4e9eb",
   "metadata": {},
   "source": [
    "### 四，单参数空间搜索范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f6b19f3",
   "metadata": {},
   "source": [
    "Optuna支持的调参算法主要包括以下这些：\n",
    "\n",
    "```\n",
    "optuna.samplers.GridSampler(网格搜索采样)\n",
    "optuna.samplers.RandomSampler(随机搜索采样)\n",
    "optuna.samplers.TPESampler(贝叶斯优化采样)\n",
    "optuna.samplers.NSGAIISampler(遗传算法采样)\n",
    "optuna.samplers.CmaEsSampler(协方差矩阵自适应演化策略采样，非常先进的优化算法)\n",
    "```\n",
    "\n",
    "此外，还可以用以下方法将部分超参固定，仅对其它一些参数进行超参优化。\n",
    "```\n",
    "optuna.samplers.PartialFixedSampler(部分参数固定采样算法)\n",
    "```\n",
    "\n",
    "可以在optuna.create_study时候用sampler参数指定。\n",
    "\n",
    "如果不指定的话，一般在单目标优化算法中，使用的是optuna.samplers.TPESampler调参算法。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "088930c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display \n",
    "import optuna\n",
    "import numpy as np \n",
    "import pandas as pd \n",
    "import plotly.graph_objs as go \n",
    "import plotly.express as px \n",
    "\n",
    "\n",
    "optuna.logging.set_verbosity(optuna.logging.WARNING)\n",
    "\n",
    "# 1, 定义目标函数\n",
    "def objective(trial:optuna.trial.Trial):\n",
    "    x = trial.suggest_float(\"x\", -10, 10)\n",
    "    return (x - 2) ** 2\n",
    "\n",
    "# 2, 执行搜索过程\n",
    "study = optuna.create_study(\n",
    "    sampler=optuna.samplers.CmaEsSampler(),\n",
    "    direction = \"minimize\",\n",
    "    study_name = \"simple_task\")\n",
    "\n",
    "study.optimize(objective, n_trials=1000,show_progress_bar = True)\n",
    "\n",
    "# 3, 获取最优超参\n",
    "best_params = study.best_params\n",
    "best_value = study.best_value\n",
    "print(\"\\n\\nbest_value = \"+str(best_value))\n",
    "print(\"best_params:\")\n",
    "print(best_params)\n",
    "\n",
    "# 4, 绘制搜索过程\n",
    "\n",
    "dftrials = study.trials_dataframe()[[\"number\",\"value\"]]\n",
    "losses = dftrials[\"value\"].tolist()\n",
    "\n",
    "minlosses = [np.min(losses[0:i+1]) for i in range(len(losses))] \n",
    "steps = range(len(losses))\n",
    "\n",
    "dflog = pd.DataFrame({\"steps\":steps,\"losses\":losses,\"minlosses\":minlosses})\n",
    "fig = px.scatter(data_frame=dflog,x = \"steps\",y = \"losses\", size_max = 500,log_y=True)\n",
    "fig.update_traces(marker = dict(size = 20, color = \"rgba(0,0,255,0.4)\"),name = \"all\",showlegend = True)\n",
    "fig.add_traces(go.Scatter(x=list(steps), y=minlosses, name = \"best\", showlegend=True)) \n",
    "fig.update_layout(legend=dict(orientation=\"v\"))\n",
    "fig.show() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfe93047",
   "metadata": {},
   "source": [
    "```\n",
    "best_value = 0.0013535850035239266\n",
    "best_params:\n",
    "{'x': 2.0367910995150176}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3befc61",
   "metadata": {},
   "source": [
    "![](https://p.ipic.vip/cq006d.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7730afa7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6986e39a",
   "metadata": {},
   "source": [
    "### 五，网格参数空间搜索范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efef7eac",
   "metadata": {},
   "source": [
    "多个参数可以用字典表述成网格参数空间形式。\n",
    "\n",
    "optuna支持各种各样的输入参数类型。常见的有以下一些\n",
    "\n",
    "``` python\n",
    "trail = optuna.trial.Trial\n",
    "trail.suggest_categorical\n",
    "trail.suggest_discrete_uniform\n",
    "trail.suggest_float\n",
    "trail.suggest_int\n",
    "trail.suggest_loguniform\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2f497ec",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "644420f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt \n",
    "import optuna \n",
    "\n",
    "import plotly.graph_objs as go \n",
    "import plotly.express as px \n",
    "\n",
    "\n",
    "optuna.logging.set_verbosity(optuna.logging.WARNING)\n",
    "\n",
    "# 1, 定义目标函数\n",
    "def objective(trial:optuna.trial.Trial):\n",
    "    x = trial.suggest_float(\"x\", -1, 1)\n",
    "    f = trial.suggest_categorical(\"f\",['sin','sinh','cos','cosh'])\n",
    "    dic = {'sin':np.sin,'cos':np.cos,'sinh':np.sinh,'cosh':np.cosh}\n",
    "    fn = dic[f]\n",
    "    return fn(x)\n",
    "\n",
    "\n",
    "# 2, 执行搜索过程\n",
    "study = optuna.create_study(\n",
    "    sampler=optuna.samplers.TPESampler(seed=123),\n",
    "    direction = \"minimize\",\n",
    "    study_name = \"grid_task\")\n",
    "\n",
    "study.optimize(objective, n_trials=100,show_progress_bar = True)\n",
    "\n",
    "# 3, 获取最优超参\n",
    "best_params = study.best_params\n",
    "best_value = study.best_value\n",
    "print(\"\\n\\nbest_value = \"+str(best_value))\n",
    "print(\"best_params:\")\n",
    "print(best_params)\n",
    "\n",
    "# 4, 绘制搜索过程\n",
    "dftrials = study.trials_dataframe()[[\"number\",\"value\"]]\n",
    "display(dftrials)\n",
    "losses = dftrials[\"value\"].tolist()\n",
    "\n",
    "minlosses = [np.min(losses[0:i+1]) for i in range(len(losses))] \n",
    "steps = range(len(losses))\n",
    "\n",
    "dflog = pd.DataFrame({\"steps\":steps,\"losses\":losses,\"minlosses\":minlosses})\n",
    "fig = px.scatter(data_frame=dflog,x = \"steps\",y = \"losses\", size_max = 500)\n",
    "fig.update_traces(marker = dict(size = 20, color = \"rgba(0,0,255,0.4)\"),name = \"all\",showlegend = True)\n",
    "fig.add_traces(go.Scatter(x=list(steps), y=minlosses, name = \"best\", showlegend=True)) \n",
    "fig.update_layout(legend=dict(orientation=\"v\"))\n",
    "fig.show() \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51f6a2e5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "46e593c3",
   "metadata": {},
   "source": [
    "### 六，断点续搜范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55235ae4",
   "metadata": {},
   "source": [
    "有时候超参搜索过程中有可能中间机器会死掉，可以使用sqlite/mysql等存储方式存储搜索结果到数据库文件。\n",
    "\n",
    "这样便可以读取历史搜索结果，继续搜索。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "312c8875",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import optuna\n",
    "optuna.logging.set_verbosity(optuna.logging.ERROR)\n",
    "\n",
    "# 1, 定义目标函数\n",
    "def objective(trial):\n",
    "    x = trial.suggest_float(\"x\", -10, 10)\n",
    "    return (x - 2) ** 2\n",
    "\n",
    "\n",
    "# 2, 执行搜索过程\n",
    "study_name = \"example-study\"  \n",
    "storage_name = \"sqlite:///{}.db\".format(study_name)\n",
    "\n",
    "#period0\n",
    "study0 = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True)\n",
    "study0.optimize(objective, n_trials=10)\n",
    "\n",
    "#period1\n",
    "study1 = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True)\n",
    "study1.optimize(objective, n_trials=10)\n",
    "\n",
    "#period2\n",
    "study2 = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True)\n",
    "dftrials = study2.trials_dataframe(attrs=(\"number\", \"value\", \"params\", \"state\"))\n",
    "\n",
    "# 3, 获取最优超参\n",
    "best_params = study2.best_params\n",
    "best_value = study2.best_value\n",
    "print(\"\\n\\nbest_value = \"+str(best_value))\n",
    "print(\"best_params:\")\n",
    "print(best_params)\n",
    "\n",
    "display(dftrials)\n",
    "\n",
    "\n",
    "# 4, 绘制搜索过程\n",
    "fig = optuna.visualization.plot_optimization_history(study2)\n",
    "#fig.layout.yaxis.type = 'log'\n",
    "fig.update_layout({\"yaxis.type\":\"log\"})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be2c0d76",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "fd117eda",
   "metadata": {},
   "source": [
    "### 七，剪枝策略范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89458262",
   "metadata": {},
   "source": [
    "optuna支持多种剪枝策略，如果某个超参采样点返回的中间结果和之前采样点相比表现得没有希望，optuna可以提前结束这个采样点的训练，从而节约时间。\n",
    "\n",
    "* optuna.pruners.HyperbandPruner: pruner refers to http://www.jmlr.org/papers/volume18/16-558/16-558.pdf\n",
    "\n",
    "* optuna.pruners.MedianPruner: Prune if the trial's best intermediate result is worse than median of intermediate results of previous trials at the same step.\n",
    "\n",
    "* optuna.pruners.ThresholdPruner: Pruner to detect outlying metrics of the trials\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "646efdb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import optuna\n",
    "from optuna.visualization import plot_optimization_history\n",
    "\n",
    "from sklearn.datasets import load_iris \n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "optuna.logging.set_verbosity(optuna.logging.ERROR)\n",
    "\n",
    "def objective(trial):\n",
    "\n",
    "    \n",
    "    iris = load_iris() \n",
    "    data,target = iris[\"data\"],iris[\"target\"]\n",
    "\n",
    "    x_train, x_valid, y_train, y_valid = train_test_split(data, target)\n",
    "    classes = list(set(target))\n",
    "\n",
    "    clf = MLPClassifier(\n",
    "        hidden_layer_sizes=tuple(\n",
    "            [trial.suggest_int(\"n_units_l{}\".format(i), 32, 64) for i in range(3)]\n",
    "        ),\n",
    "        learning_rate_init=trial.suggest_float(\"lr_init\", 1e-5, 1e-1, log=True),\n",
    "    )\n",
    "\n",
    "    for step in range(100):\n",
    "        clf.partial_fit(x_train, y_train, classes=classes)\n",
    "        value = clf.score(x_valid, y_valid)\n",
    "\n",
    "        # Report intermediate objective value.\n",
    "        trial.report(value, step)\n",
    "\n",
    "        # Handle pruning based on the intermediate value.\n",
    "        if trial.should_prune():\n",
    "            raise optuna.TrialPruned()\n",
    "\n",
    "    return value\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "\n",
    "    study = optuna.create_study(direction=\"maximize\", pruner=optuna.pruners.MedianPruner())\n",
    "    study.optimize(objective, n_trials=100, timeout=600)\n",
    "    plot_optimization_history(study).show()\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdab7171",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0c095c9f",
   "metadata": {},
   "source": [
    "### 八，各种调参可视化函数范例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "344b671c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import optuna\n",
    "from optuna.visualization import plot_contour\n",
    "from optuna.visualization import plot_intermediate_values\n",
    "from optuna.visualization import plot_optimization_history\n",
    "from optuna.visualization import plot_parallel_coordinate\n",
    "from optuna.visualization import plot_param_importances\n",
    "from optuna.visualization import plot_slice\n",
    "\n",
    "from sklearn.datasets import load_iris \n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "optuna.logging.set_verbosity(optuna.logging.ERROR)\n",
    "\n",
    "def objective(trial):\n",
    "\n",
    "    \n",
    "    iris = load_iris() \n",
    "    data,target = iris[\"data\"],iris[\"target\"]\n",
    "\n",
    "    x_train, x_valid, y_train, y_valid = train_test_split(data, target)\n",
    "    classes = list(set(target))\n",
    "\n",
    "    clf = MLPClassifier(\n",
    "        hidden_layer_sizes=tuple(\n",
    "            [trial.suggest_int(\"n_units_l{}\".format(i), 32, 64) for i in range(3)]\n",
    "        ),\n",
    "        learning_rate_init=trial.suggest_float(\"lr_init\", 1e-5, 1e-1, log=True),\n",
    "    )\n",
    "\n",
    "    for step in range(100):\n",
    "        clf.partial_fit(x_train, y_train, classes=classes)\n",
    "        value = clf.score(x_valid, y_valid)\n",
    "\n",
    "        # Report intermediate objective value.\n",
    "        trial.report(value, step)\n",
    "\n",
    "        # Handle pruning based on the intermediate value.\n",
    "        if trial.should_prune():\n",
    "            raise optuna.TrialPruned()\n",
    "\n",
    "    return value\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "\n",
    "    study = optuna.create_study(direction=\"maximize\", pruner=optuna.pruners.MedianPruner())\n",
    "    study.optimize(objective, n_trials=100, timeout=600)\n",
    "\n",
    "    # Visualize the optimization history.\n",
    "    plot_optimization_history(study).show()\n",
    "\n",
    "    # Visualize the learning curves of the trials.\n",
    "    plot_intermediate_values(study).show()\n",
    "\n",
    "    # Visualize high-dimensional parameter relationships.\n",
    "    plot_parallel_coordinate(study).show()\n",
    "\n",
    "    # Select parameters to visualize.\n",
    "    plot_parallel_coordinate(study, params=[\"lr_init\", \"n_units_l0\"]).show()\n",
    "\n",
    "    # Visualize hyperparameter relationships.\n",
    "    plot_contour(study).show()\n",
    "\n",
    "    # Select parameters to visualize.\n",
    "    plot_contour(study, params=[\"n_units_l0\", \"n_units_l1\"]).show()\n",
    "\n",
    "    # Visualize individual hyperparameters.\n",
    "    plot_slice(study).show()\n",
    "\n",
    "    # Select parameter\n",
    "    # Visualize parameter importances.\n",
    "    plot_param_importances(study).show()\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6317441",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9b4c1362",
   "metadata": {},
   "source": [
    "### 九，手动添加超参数采样点范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff7d9658",
   "metadata": {},
   "source": [
    "有时候，我们在使用特定的采样算法之前，想尝试一些人工指定的超参采样点，在Optuna中我们可以用\n",
    "\n",
    "study.enqueue_trial 将这些人工指定超参采样点推入测试队列。\n",
    "\n",
    "另外，如果我们已经手动执行了一些超参采样点，并获得了结果，我们可以用 study.add_trail的方法\n",
    "\n",
    "将这些结果添加到已评估的超参采样点列表中。后续的超参数采样将会考虑这些采样点。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18a790e2",
   "metadata": {},
   "source": [
    "**1，指定手工采样点**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53e60c5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import lightgbm as lgb\n",
    "import numpy as np\n",
    "import sklearn.datasets\n",
    "import sklearn.metrics\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import optuna\n",
    "\n",
    "def objective(trial):\n",
    "    data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)\n",
    "    train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)\n",
    "    dtrain = lgb.Dataset(train_x, label=train_y)\n",
    "    dvalid = lgb.Dataset(valid_x, label=valid_y)\n",
    "\n",
    "    param = {\n",
    "        \"objective\": \"binary\",\n",
    "        \"metric\": \"auc\",\n",
    "        \"verbosity\": -1,\n",
    "        \"boosting_type\": \"gbdt\",\n",
    "        \"bagging_fraction\": min(trial.suggest_float(\"bagging_fraction\", 0.4, 1.0 + 1e-12), 1),\n",
    "        \"bagging_freq\": trial.suggest_int(\"bagging_freq\", 0, 7),\n",
    "        \"min_child_samples\": trial.suggest_int(\"min_child_samples\", 5, 100),\n",
    "    }\n",
    "\n",
    "    # Add a callback for pruning.\n",
    "    pruning_callback = optuna.integration.LightGBMPruningCallback(trial, \"auc\")\n",
    "    gbm = lgb.train(\n",
    "        param, dtrain, valid_sets=[dvalid], verbose_eval=False, callbacks=[pruning_callback]\n",
    "    )\n",
    "\n",
    "    preds = gbm.predict(valid_x)\n",
    "    pred_labels = np.rint(preds)\n",
    "    accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)\n",
    "    return accuracy\n",
    "\n",
    "study = optuna.create_study(direction=\"maximize\", pruner=optuna.pruners.MedianPruner())\n",
    "\n",
    "study.enqueue_trial(\n",
    "    {\n",
    "        \"bagging_fraction\": 1.0,\n",
    "        \"bagging_freq\": 0,\n",
    "        \"min_child_samples\": 20,\n",
    "    }\n",
    ")\n",
    "\n",
    "study.enqueue_trial(\n",
    "    {\n",
    "        \"bagging_fraction\": 0.75,\n",
    "        \"bagging_freq\": 5,\n",
    "        \"min_child_samples\": 20,\n",
    "    }\n",
    ")\n",
    "\n",
    "import logging\n",
    "import sys\n",
    "\n",
    "# Add stream handler of stdout to show the messages to see Optuna works expectedly.\n",
    "optuna.logging.get_logger(\"optuna\").addHandler(logging.StreamHandler(sys.stdout))\n",
    "study.optimize(objective, n_trials=100, timeout=600)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd45bb84",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4cd9102e",
   "metadata": {},
   "source": [
    "**2，添加已评估采样点**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb7da213",
   "metadata": {},
   "outputs": [],
   "source": [
    "import lightgbm as lgb\n",
    "import numpy as np\n",
    "import sklearn.datasets\n",
    "import sklearn.metrics\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import optuna\n",
    "\n",
    "def objective(trial):\n",
    "    data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)\n",
    "    train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)\n",
    "    dtrain = lgb.Dataset(train_x, label=train_y)\n",
    "    dvalid = lgb.Dataset(valid_x, label=valid_y)\n",
    "\n",
    "    param = {\n",
    "        \"objective\": \"binary\",\n",
    "        \"metric\": \"auc\",\n",
    "        \"verbosity\": -1,\n",
    "        \"boosting_type\": \"gbdt\",\n",
    "        \"bagging_fraction\": min(trial.suggest_float(\"bagging_fraction\", 0.4, 1.0 + 1e-12), 1),\n",
    "        \"bagging_freq\": trial.suggest_int(\"bagging_freq\", 0, 7),\n",
    "        \"min_child_samples\": trial.suggest_int(\"min_child_samples\", 5, 100),\n",
    "    }\n",
    "\n",
    "    # Add a callback for pruning.\n",
    "    pruning_callback = optuna.integration.LightGBMPruningCallback(trial, \"auc\")\n",
    "    gbm = lgb.train(\n",
    "        param, dtrain, valid_sets=[dvalid], verbose_eval=False, callbacks=[pruning_callback]\n",
    "    )\n",
    "\n",
    "    preds = gbm.predict(valid_x)\n",
    "    pred_labels = np.rint(preds)\n",
    "    accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)\n",
    "    return accuracy\n",
    "\n",
    "study = optuna.create_study(direction=\"maximize\", pruner=optuna.pruners.MedianPruner())\n",
    "study.add_trial(\n",
    "    optuna.trial.create_trial(\n",
    "        params={\n",
    "            \"bagging_fraction\": 1.0,\n",
    "            \"bagging_freq\": 0,\n",
    "            \"min_child_samples\": 20,\n",
    "        },\n",
    "        distributions={\n",
    "            \"bagging_fraction\": optuna.distributions.UniformDistribution(0.4, 1.0 + 1e-12),\n",
    "            \"bagging_freq\": optuna.distributions.IntUniformDistribution(0, 7),\n",
    "            \"min_child_samples\": optuna.distributions.IntUniformDistribution(5, 100),\n",
    "        },\n",
    "        value=0.94,\n",
    "    )\n",
    ")\n",
    "study.add_trial(\n",
    "    optuna.trial.create_trial(\n",
    "        params={\n",
    "            \"bagging_fraction\": 0.75,\n",
    "            \"bagging_freq\": 5,\n",
    "            \"min_child_samples\": 20,\n",
    "        },\n",
    "        distributions={\n",
    "            \"bagging_fraction\": optuna.distributions.UniformDistribution(0.4, 1.0 + 1e-12),\n",
    "            \"bagging_freq\": optuna.distributions.IntUniformDistribution(0, 7),\n",
    "            \"min_child_samples\": optuna.distributions.IntUniformDistribution(5, 100),\n",
    "        },\n",
    "        value=0.95,\n",
    "    )\n",
    ")\n",
    "study.optimize(objective, n_trials=100, timeout=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d56c56d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}