7-6,DeepCross模型.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "71bcfb14",
   "metadata": {},
   "source": [
    "\n",
    "# 7-6，DeepCross模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c91553e",
   "metadata": {},
   "source": [
    "谷歌在CTR预估和推荐排序模型方面有3篇相对重要的文章。\n",
    "\n",
    "第1篇是2016年的Deep&Wide,第2篇是2017年的Deep&Cross(DCN),第3篇是2020年的DCN-V2.\n",
    "\n",
    "这3篇文章的核心思想分别用1句话概括如下：\n",
    "\n",
    "第1篇Deep&Wide说，线性回归LR具有好的记忆能力(Wide部分),多层感知机DNN具有好的泛化能力(Deep部分),把它们并行的结合起来食用口感真香.\n",
    "\n",
    "第2篇Deep&Cross说，将Deep&Wide的Wide部分换成一个CrossNet，可以自动捕获多层级特征交叉，减少人工特征工程。新的模型叫做DCN口感更香！\n",
    "\n",
    "第3篇DCNV2说，DCN的CrossNet的仅仅用一个向量来建模交叉能力, 拟合能力太弱, 配方升级换成矩阵吧。升级后的DCNV2量大管饱，又香又饱！什么，矩阵太大跑的慢，不太好消化? 别急，我们还有秘制配方，把矩阵换成两个低秩矩阵的乘积，这个用了秘制配方的模型叫做DCN-Mix，嘎嘣脆，又香又饱还好消化，你值得拥有！\n",
    "\n",
    "今天我们就来尝尝DCN、DCNV2以及嘎嘣脆的DCN-Mix！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "672ebc71",
   "metadata": {},
   "source": [
    "参考文档：\n",
    "\n",
    "* Deep&Wide论文： https://arxiv.org/pdf/1606.07792v1.pdf\n",
    "\n",
    "* DCN论文：https://arxiv.org/pdf/1708.05123.pdf \n",
    "\n",
    "* DCNV2论文： https://arxiv.org/pdf/2008.13535.pdf\n",
    "\n",
    "* 解密Deep&Cross: https://zhuanlan.zhihu.com/p/55234968 \n",
    "\n",
    "* 大规模排序系统中的特征交叉DCN-V2: https://zhuanlan.zhihu.com/p/353223660\n",
    "\n",
    "* 代码实现参考：https://github.com/shenweichen/DeepCTR-Torch/blob/bc881dcd417fec64f840b0cacce124bc86b3687c/deepctr_torch/layers/interaction.py#L406-L537\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39112727",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "<font color=\"red\">\n",
    " \n",
    "公众号 **算法美食屋** 回复关键词：**pytorch**， 获取本项目源码和所用数据集百度云盘下载链接。\n",
    "    \n",
    "</font> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5eb6a1a",
   "metadata": {},
   "source": [
    "## 一，DeepCross原理解析"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14735953",
   "metadata": {},
   "source": [
    "谷歌的DeepCross系列模型(DCN-vector, DCN-matrix, DCN-mix) 围绕解决的核心问题三高问题。\n",
    "\n",
    "如何实现 高效率，高表达能力 的 高阶显式特征交叉？\n",
    "\n",
    "这三高问题分别是什么含义呢？\n",
    "\n",
    "1，高阶：至少支持三阶特征交叉。实践证明，在CTR和推荐领域，二阶特征交叉和三阶特征交叉是有用的，四阶及以上的特征交叉没啥用。FM只是二阶显式特征交叉。 \n",
    "\n",
    "2，高效率：特征交叉模块的预测效率最好是O(n),其中n为特征数量。\n",
    "\n",
    "3，高表达能力：特征交叉模块有足够大的参数容量来捕获各种可能的特征交叉。\n",
    "\n",
    "下面我们分别看看 DCN-vector， DCN-matrix, DCN-mix在三高问题的解决思路和解决程度。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b5aa021",
   "metadata": {},
   "source": [
    "### 1， DCN-vector "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e5959a9",
   "metadata": {},
   "source": [
    "DCN-vector 也就是 2017年发布的最初版本的DCN， 很好地解决了三高问题中的两高。\n",
    "\n",
    "1，高阶 (DCN-vector可以很方便地支持三阶及以上的显式特征交叉)\n",
    "\n",
    "2，高效率 (DCN-vector特征交叉模块的推理复杂度是O(n)，非常高效，和FM不分伯仲)\n",
    "\n",
    "我们来看看DCN-vector怎么做的显式特征交叉。\n",
    "\n",
    "先看图和公式。\n",
    "\n",
    "有两个要点，一个是用和x0等长的参数向量来捕获交叉关系，第二个是参照了残差模块的设计将低阶量加回到输出中。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17a7563e",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wduan1hmj20a30570sv.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b78c0cf",
   "metadata": {},
   "source": [
    "代码和图一样好懂。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1e109057",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "class CrossNetVector(nn.Module):\n",
    "    def __init__(self, d_in, n_cross=2):\n",
    "        super().__init__()\n",
    "        self.n_cross = n_cross\n",
    "        self.linears = nn.ModuleList([nn.Linear(d_in,1,bias=False) for i in range(self.n_cross)])\n",
    "        self.biases = nn.ParameterList(\n",
    "            [nn.Parameter(torch.zeros(d_in)) for i in range(self.n_cross)])\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            xi = x0*self.linears[i](xi)+self.biases[i]+xi\n",
    "        return xi"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7fcd0b6",
   "metadata": {},
   "source": [
    "CrossNetVector的结构非常简洁。每升高一阶特征交叉，主要运算是将上一阶特征向量xi和交叉模块参数向量w做点积，并数乘原始特征向量x0。\n",
    "\n",
    "该计算的计算前后向量的维度不变，计算量和特征向量的长度n成正比，所以复杂度是O(n).\n",
    "\n",
    "值得注意的是，虽然CrossNetVector的复杂度很低仅仅是O(n),  但是 $x_{0}*x^{'}*\\omega$ 这一项展开后是包括任意两个特征的交叉乘积贡献的。\n",
    "\n",
    "我们知道任意两个特征的交叉共有 n(n-1)/2 种组合方式，这个复杂度是O(n^2)的，那么为什么 CrossNetVector 能够以 O(n)的复杂度实现包括O(n^2)量级的特征交叉组合贡献的呢？\n",
    "\n",
    "原因是CrossNetVector这种结构实际上具有权重共享的特性。\n",
    "\n",
    "假设有5个特征 x1,x2,x3,x4,x5, 如果要独立地描述它们之间的两两交叉要10个参数，但是CrossNetVector中仅仅有5个表述特征交叉系数的参数 w1,w2,w3,w4,w5\n",
    "\n",
    "其中 x1x2的交叉项的系数是 (w1+w2), x1x3的交叉项系数时 (w1+w3),...,x1x5的交叉项系数是 (w1+w5), 可见，w1这个参数被用在了 x1和任何参数的交叉项中作为组成部分。\n",
    "\n",
    "这就是一种权重共享的特性，这种特性有助于DCN变得轻便高效，并像FM那样学习到稀疏特征的交叉。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4965699",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4daccd07",
   "metadata": {},
   "source": [
    "### 2, DCN-matrix "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8114c672",
   "metadata": {},
   "source": [
    "DCN-vector 虽然解决了三高中的两高: 高阶、高效率。\n",
    "\n",
    "但是它在另一高也就是高表达能力方面有些缺陷，它仅仅用一个和输入特征等长的向量来刻画特征之间的相互作用，不足以表述丰富的特征交叉模式。\n",
    "\n",
    "CrossNetVector的参数数量相比Deep部分的MLP的参数数量来看实在微不足道，可能存在记忆容量瓶颈。\n",
    "\n",
    "换言之，它的权值共享可能做的有些过头了。\n",
    "\n",
    "正是基于这样的洞察，DCN-matrix 在DCN-vector基础上做了一些轻微的改动，用矩阵代替了向量，并适配性地修改了一些运算符，强化了显式特征交叉项的表达能力。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76da09f6",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wgsq1zcyj208u0473yj.jpg) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bce4a3d7",
   "metadata": {},
   "source": [
    "代码比图更加好懂。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "46e93a0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "class CrossNetMatrix(nn.Module):\n",
    "    def __init__(self, d_in, n_cross=2):\n",
    "        super().__init__()\n",
    "        self.n_cross = n_cross\n",
    "        self.linears = nn.ModuleList([nn.Linear(d_in,d_in) for i in range(self.n_cross)])\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            xi = x0*self.linears[i](xi)+xi\n",
    "        return xi"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "337a9a52",
   "metadata": {},
   "source": [
    "可以看到 CrossNetMatrix 用一个 $n\\times n$ 的交互矩阵$W$来刻画 特征之间的交互，实际上$W$的第i行第j列的元素 $W_{ij}$\n",
    "\n",
    "表达的就是第i个特征和第j个特征之间的交互项系数。 因此 CrossNetMatrix 的复杂度是O(n^2)的，并且具有很好的解释型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b88a0f8",
   "metadata": {},
   "source": [
    "于是 CrossNetMatrix整体来说是牺牲了 高效性，换来了高表达能力。不过，在许多特征数量不算特别多的情况下，例如n在1000以下的时候，CrossNetMatrix依旧是非常高效的。\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e28fe65",
   "metadata": {},
   "source": [
    "### 3, DCN-mix "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4a18321",
   "metadata": {},
   "source": [
    "DCN-mix可以说是 DCN-matrix基础上提升计算效率，同时保持高表达能力的一种方案。 \n",
    "\n",
    "其主要思想洞察如下。\n",
    "\n",
    "1，矩阵分解：根据SVD主成分分析理论，庞大的交互矩阵W可以由为两个低秩矩阵U和V的乘积来近似。这样可以将参数量和计算复杂度降低到O(n)。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wid7h72rj209s01ga9v.jpg)\n",
    "\n",
    "2，专家融合：使用两个低秩矩阵U和V的乘积来近似W提升了效率但是降低了表达能力，可以使用MOE思想(Mixture of Experts)融合多个低秩空间的专家网络来提升表达能力。\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wibupquej20eb0ayjrt.jpg)\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wiet82zxj209302ma9y.jpg)\n",
    "\n",
    "3，低秩变换：可以在U和V之间在低秩空间上做一些变换适当再拉一拉表达能力。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wifm6877j20ac01d744.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2fbbc1e",
   "metadata": {},
   "source": [
    "整体来说，DCN-mix用到的这3个技巧还是有些难度的，可以结合代码理解一下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c0db9756",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "class CrossNetMix(nn.Module):\n",
    "    def __init__(self, d_in, n_cross =2, low_rank=32, n_experts=4):\n",
    "        super().__init__()\n",
    "        self.d_in = d_in\n",
    "        self.n_cross = n_cross\n",
    "        self.low_rank = low_rank\n",
    "        self.n_experts = n_experts\n",
    "\n",
    "        # U: (d_in, low_rank)\n",
    "        self.U_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, d_in, low_rank))) for i in range(self.n_cross)])\n",
    "\n",
    "        # V: (d_in, low_rank)\n",
    "        self.V_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, d_in, low_rank))) for i in range(self.n_cross)])\n",
    "\n",
    "        # C: (low_rank, low_rank)\n",
    "        self.C_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, low_rank, low_rank))) for i in range(self.n_cross)])\n",
    "\n",
    "        # G: (d_in, 1)\n",
    "        self.gating = nn.ModuleList([nn.Linear(d_in, 1, bias=False) for i in range(self.n_experts)])\n",
    "\n",
    "        # Bias \n",
    "        self.biases = nn.ParameterList([nn.Parameter(torch.zeros(d_in)) for i in range(self.n_cross)])\n",
    "\n",
    "\n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            output_of_experts = []\n",
    "            gating_score_of_experts = []\n",
    "            for expert_id in range(self.n_experts):\n",
    "\n",
    "                # (1) G(xi)\n",
    "                # compute the gating score by xi\n",
    "                gating_score_of_experts.append(self.gating[expert_id](xi))\n",
    "\n",
    "                # (2) E(xi)\n",
    "                # project the input xi to low_rank space\n",
    "                v_x = xi@(self.V_list[i][expert_id])   # (batch_size, low_rank)\n",
    "\n",
    "                # nonlinear activation in low rank space\n",
    "                v_x = torch.tanh(v_x)\n",
    "                v_x = v_x@self.C_list[i][expert_id]     # (batch_size, low_rank)\n",
    "                v_x = torch.tanh(v_x)\n",
    "\n",
    "                # project back to d_in space\n",
    "                uv_x = v_x@(self.U_list[i][expert_id].T)  # (batch_size, d_in)\n",
    "                expert_out = x0*(uv_x + self.biases[i])\n",
    "                output_of_experts.append(expert_out)\n",
    "\n",
    "            # (3) mixture of low-rank experts\n",
    "            output_of_experts = torch.stack(output_of_experts, 2)  # (batch_size, d_in, n_experts)\n",
    "            gating_score_of_experts = torch.stack(gating_score_of_experts, 1)  # (batch_size, n_experts, 1)\n",
    "            moe_out = torch.bmm(output_of_experts, gating_score_of_experts.softmax(1))\n",
    "            xi = torch.squeeze(moe_out) + xi  # (batch_size, d_in)\n",
    "\n",
    "        return xi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ac338ea",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "bc2fcc0e",
   "metadata": {},
   "source": [
    "## 二，DeepCross的pytorch实现"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbff274e",
   "metadata": {},
   "source": [
    "下面是DeepCross的一个pytorch完整实现。\n",
    "\n",
    "可以通过cross_type 来指定是哪个版本的DCN模型。\n",
    "\n",
    "* cross_type = \"vector\" 选择 DCNV1, 也可以称之为 DCN-vector \n",
    "\n",
    "* cross_type = \"matrix\" 选择 DCNV2, 也可以称之为 DCN-matrix \n",
    "\n",
    "* cross_type = \"mix\" 选择 DCN-mix \n",
    "\n",
    "除了三种CrossNet结构的实现外，还有两点细节值得注意：\n",
    "\n",
    "1，DeepCross对类别特征的处理与DeepFM等模型不太一样，是全部转换成embedding后再和数值类特征拼在一起，这种特性可以让各个类别特征根据其类别数量选择适当的embedding长度。\n",
    "\n",
    "2，DeepCross中Deep部分和Cross部分的关系可以是串行的(stacked=True)，也可以是并行的(stacked=False)。具体哪种效果更好，取决于数据集。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c607ceba",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn,Tensor \n",
    "import torch.nn.functional as F \n",
    "\n",
    "#离散特征编码\n",
    "class CatEmbeddingSqrt(nn.Module):\n",
    "    \"\"\"\n",
    "    离散特征使用Embedding层编码, d_embed等于sqrt(category)\n",
    "    输入shape: [batch_size,d_in], \n",
    "    输出shape: [batch_size,d_out]\n",
    "    \"\"\"\n",
    "    def __init__(self, categories, d_embed_max = 100):\n",
    "        super().__init__()\n",
    "        self.categories = categories\n",
    "        self.d_embed_list = [min(max(int(x**0.5), 2),d_embed_max) for x in categories]\n",
    "        self.embedding_list = nn.ModuleList([nn.Embedding(self.categories[i],self.d_embed_list[i])\n",
    "                            for i in range(len(categories))])\n",
    "        self.d_cat_sum = sum(self.d_embed_list)\n",
    "        \n",
    "    def forward(self, x_cat):\n",
    "        \"\"\"\n",
    "        param x_cat: Long tensor of size ``(batch_size, d_in)``\n",
    "        \"\"\"\n",
    "        x_out = torch.cat([self.embedding_list[i](x_cat[:,i]) \n",
    "                           for i in range(len(self.categories)) ],dim=1)\n",
    "        return x_out\n",
    "    \n",
    "#deep部分\n",
    "class MLP(nn.Module):\n",
    "    def __init__(self, d_in, d_layers, dropout):\n",
    "        super().__init__()\n",
    "        layers = []\n",
    "        for d in d_layers:\n",
    "            layers.append(nn.Linear(d_in, d))\n",
    "            layers.append(nn.BatchNorm1d(d))\n",
    "            layers.append(nn.ReLU())\n",
    "            layers.append(nn.Dropout(p=dropout))\n",
    "            d_in = d\n",
    "        self.mlp = nn.Sequential(*layers)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.mlp(x)\n",
    "    \n",
    "\n",
    "#3种CrossNet的实现    \n",
    "class CrossNetVector(nn.Module):\n",
    "    def __init__(self, d_in, n_cross=2):\n",
    "        super().__init__()\n",
    "        self.n_cross = n_cross\n",
    "        self.linears = nn.ModuleList([nn.Linear(d_in,1,bias=False) for i in range(self.n_cross)])\n",
    "        self.biases = nn.ParameterList(\n",
    "            [nn.Parameter(torch.zeros(d_in)) for i in range(self.n_cross)])\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            xi = x0*self.linears[i](xi)+self.biases[i]+xi\n",
    "        return xi\n",
    "    \n",
    "    \n",
    "class CrossNetMatrix(nn.Module):\n",
    "    def __init__(self, d_in, n_cross=2):\n",
    "        super().__init__()\n",
    "        self.n_cross = n_cross\n",
    "        self.linears = nn.ModuleList([nn.Linear(d_in,d_in) for i in range(self.n_cross)])\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            xi = x0*self.linears[i](xi)+xi\n",
    "        return xi\n",
    "    \n",
    "\n",
    "class CrossNetMix(nn.Module):\n",
    "    def __init__(self, d_in, n_cross =2, low_rank=32, n_experts=4):\n",
    "        super().__init__()\n",
    "        self.d_in = d_in\n",
    "        self.n_cross = n_cross\n",
    "        self.low_rank = low_rank\n",
    "        self.n_experts = n_experts\n",
    "\n",
    "        # U: (d_in, low_rank)\n",
    "        self.U_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, d_in, low_rank))) for i in range(self.n_cross)])\n",
    "        \n",
    "        # V: (d_in, low_rank)\n",
    "        self.V_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, d_in, low_rank))) for i in range(self.n_cross)])\n",
    "        \n",
    "        # C: (low_rank, low_rank)\n",
    "        self.C_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, low_rank, low_rank))) for i in range(self.n_cross)])\n",
    "        \n",
    "        # G: (d_in, 1)\n",
    "        self.gating = nn.ModuleList([nn.Linear(d_in, 1, bias=False) for i in range(self.n_experts)])\n",
    "\n",
    "        # Bias \n",
    "        self.biases = nn.ParameterList([nn.Parameter(torch.zeros(d_in)) for i in range(self.n_cross)])\n",
    "        \n",
    "\n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            output_of_experts = []\n",
    "            gating_score_of_experts = []\n",
    "            for expert_id in range(self.n_experts):\n",
    "                \n",
    "                # (1) G(xi)\n",
    "                # compute the gating score by xi\n",
    "                gating_score_of_experts.append(self.gating[expert_id](xi))\n",
    "\n",
    "                # (2) E(xi)\n",
    "                # project the input xi to low_rank space\n",
    "                v_x = xi@(self.V_list[i][expert_id])   # (batch_size, low_rank)\n",
    "\n",
    "                # nonlinear activation in low rank space\n",
    "                v_x = torch.tanh(v_x)\n",
    "                v_x = v_x@self.C_list[i][expert_id]     # (batch_size, low_rank)\n",
    "                v_x = torch.tanh(v_x)\n",
    "\n",
    "                # project back to d_in space\n",
    "                uv_x = v_x@(self.U_list[i][expert_id].T)  # (batch_size, d_in)\n",
    "                expert_out = x0*(uv_x + self.biases[i])\n",
    "                output_of_experts.append(expert_out)\n",
    "\n",
    "            # (3) mixture of low-rank experts\n",
    "            output_of_experts = torch.stack(output_of_experts, 2)  # (batch_size, d_in, n_experts)\n",
    "            gating_score_of_experts = torch.stack(gating_score_of_experts, 1)  # (batch_size, n_experts, 1)\n",
    "            moe_out = torch.bmm(output_of_experts, gating_score_of_experts.softmax(1))\n",
    "            xi = torch.squeeze(moe_out) + xi  # (batch_size, d_in)\n",
    "            \n",
    "        return xi\n",
    "    \n",
    "    \n",
    "class DeepCross(nn.Module):\n",
    "    \"\"\"\n",
    "    DeepCross三种模型(DCN-vec,DCN-matrix,DCN-mix)的统一实现。\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, d_numerical, categories, d_embed_max = 8,\n",
    "                 n_cross=2, cross_type = \"matrix\", low_rank=32, n_experts=4, \n",
    "                 mlp_layers = [128,64,32] ,mlp_dropout = 0.25, \n",
    "                 stacked = True, \n",
    "                 n_classes = 1):\n",
    "        \n",
    "        super().__init__()\n",
    "        \n",
    "        if cross_type=='mix':\n",
    "            assert low_rank is not None and n_experts is not None\n",
    "        \n",
    "        if d_numerical is None:\n",
    "            d_numerical = 0\n",
    "        if categories is None:\n",
    "            categories = []\n",
    "            \n",
    "        self.categories = categories\n",
    "        self.n_classes = n_classes\n",
    "        self.stacked = stacked\n",
    "        \n",
    "        \n",
    "        self.cat_embedding = CatEmbeddingSqrt(categories, d_embed_max) if categories else None\n",
    "        \n",
    "        self.d_in = d_numerical \n",
    "        if self.cat_embedding:\n",
    "            self.d_in+=self.cat_embedding.d_cat_sum\n",
    "            \n",
    "        if cross_type==\"vector\":\n",
    "            self.cross_layer = CrossNetVector(self.d_in,n_cross)\n",
    "        elif cross_type==\"matrix\":\n",
    "            self.cross_layer = CrossNetMatrix(self.d_in,n_cross)\n",
    "        elif cross_type==\"mix\":\n",
    "            self.cross_layer = CrossNetMix(self.d_in,n_cross,low_rank,n_experts)\n",
    "        else:\n",
    "            raise NotImplementedError(\"cross_type should  be one of ('vector','matrix','mix') !\")\n",
    "        \n",
    "        self.mlp = MLP(\n",
    "            d_in= self.d_in,\n",
    "            d_layers = mlp_layers,\n",
    "            dropout = mlp_dropout\n",
    "        )\n",
    "        \n",
    "        if self.stacked:\n",
    "            self.last_linear = nn.Linear(mlp_layers[-1],n_classes)\n",
    "        else:\n",
    "            self.last_linear = nn.Linear(self.d_in+mlp_layers[-1],n_classes)\n",
    "        \n",
    "\n",
    "    def forward(self, x):\n",
    "        \n",
    "        \"\"\"\n",
    "        x_num: numerical features\n",
    "        x_cat: category features\n",
    "        \"\"\"\n",
    "        x_num,x_cat = x\n",
    "        \n",
    "        #embedding \n",
    "        x_total = []\n",
    "        if x_num is not None:\n",
    "            x_total.append(x_num)\n",
    "        if self.cat_embedding is not None:\n",
    "            x_total.append(self.cat_embedding(x_cat))\n",
    "        x_total = torch.cat(x_total, dim=-1)\n",
    "        \n",
    "        \n",
    "        #cross部分\n",
    "        x_cross = self.cross_layer(x_total)\n",
    "        \n",
    "        \n",
    "        #deep部分\n",
    "        if self.stacked:\n",
    "            x_deep = self.mlp(x_cross)\n",
    "            x_out = self.last_linear(x_deep)\n",
    "        else:\n",
    "            x_deep = self.mlp(x_total)\n",
    "            x_deep_cross = torch.cat([x_deep,x_cross],axis = 1)\n",
    "            x_out = self.last_linear(x_deep_cross)\n",
    "            \n",
    "        if self.n_classes==1:\n",
    "            x_out = x_out.squeeze(-1)\n",
    "        \n",
    "        return x_out \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4fb91e6f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 0.1160, -0.1248], grad_fn=<SqueezeBackward1>)\n",
      "tensor([0.4309, 0.2105], grad_fn=<SqueezeBackward1>)\n",
      "tensor([0.0336, 0.4770], grad_fn=<SqueezeBackward1>)\n"
     ]
    }
   ],
   "source": [
    "##测试 DeepCross\n",
    "\n",
    "x_num = torch.randn(2,3)\n",
    "x_cat = torch.randint(0,2,(2,3))\n",
    "\n",
    "dcn_vec = DeepCross(d_numerical = 3, categories = [4,3,2], d_embed_max = 4,\n",
    "        n_cross=2, cross_type = \"vector\", \n",
    "        mlp_layers = [20,20], mlp_dropout=0.25,\n",
    "        stacked = False,\n",
    "        n_classes = 1)\n",
    "\n",
    "dcn_matrix = DeepCross(d_numerical = 3, categories = [4,3,2], d_embed_max = 4,\n",
    "        n_cross=2, cross_type = \"matrix\", \n",
    "        mlp_layers = [20,20], mlp_dropout=0.25,\n",
    "        stacked = True,\n",
    "        n_classes = 1)\n",
    "\n",
    "dcn_mix = DeepCross(d_numerical = 3, categories = [4,3,2], d_embed_max = 4,\n",
    "        n_cross=2, cross_type = \"mix\", low_rank=32, n_experts=4, \n",
    "        mlp_layers = [20,20], mlp_dropout=0.25,\n",
    "        stacked = False,\n",
    "        n_classes = 1)\n",
    "\n",
    "\n",
    "print(dcn_vec((x_num,x_cat)))\n",
    "print(dcn_matrix((x_num,x_cat)))\n",
    "print(dcn_mix((x_num,x_cat)))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "516b5f6c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "371196d6",
   "metadata": {},
   "source": [
    "## 三，Criteo数据集完整范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "888bd54a",
   "metadata": {},
   "source": [
    "Criteo数据集是一个经典的广告点击率CTR预测数据集。\n",
    "\n",
    "这个数据集的目标是通过用户特征和广告特征来预测某条广告是否会为用户点击。\n",
    "\n",
    "数据集有13维数值特征(I1-I13)和26维类别特征(C14-C39), 共39维特征, 特征中包含着许多缺失值。\n",
    "\n",
    "训练集4000万个样本，测试集600万个样本。数据集大小超过100G.\n",
    "\n",
    "此处使用的是采样100万个样本后的cretio_small数据集。 \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1c74075",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4d6b5ddc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd \n",
    "import datetime \n",
    "\n",
    "from sklearn.model_selection import train_test_split \n",
    "\n",
    "import torch \n",
    "from torch import nn \n",
    "from torch.utils.data import Dataset,DataLoader  \n",
    "import torch.nn.functional as F \n",
    "import torchkeras \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c79eaa4",
   "metadata": {},
   "source": [
    "### 1，准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2c810d87",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder,QuantileTransformer\n",
    "from sklearn.pipeline import Pipeline \n",
    "from sklearn.impute import SimpleImputer \n",
    "\n",
    "dfdata = pd.read_csv(\"./eat_pytorch_datasets/criteo_small.zip\",sep=\"\\t\",header=None)\n",
    "dfdata.columns = [\"label\"] + [\"I\"+str(x) for x in range(1,14)] + [\n",
    "    \"C\"+str(x) for x in range(14,40)]\n",
    "\n",
    "cat_cols = [x for x in dfdata.columns if x.startswith('C')]\n",
    "num_cols = [x for x in dfdata.columns if x.startswith('I')]\n",
    "num_pipe = Pipeline(steps = [('impute',SimpleImputer()),('quantile',QuantileTransformer())])\n",
    "\n",
    "for col in cat_cols:\n",
    "    dfdata[col]  = LabelEncoder().fit_transform(dfdata[col])\n",
    "\n",
    "dfdata[num_cols] = num_pipe.fit_transform(dfdata[num_cols])\n",
    "\n",
    "categories = [dfdata[col].max()+1 for col in cat_cols]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "21d7ef14",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch.utils.data import Dataset,DataLoader \n",
    "\n",
    "#DataFrame转换成torch数据集Dataset, 特征分割成X_num,X_cat方式\n",
    "class DfDataset(Dataset):\n",
    "    def __init__(self,df,\n",
    "                 label_col,\n",
    "                 num_features,\n",
    "                 cat_features,\n",
    "                 categories,\n",
    "                 is_training=True):\n",
    "        \n",
    "        self.X_num = torch.tensor(df[num_features].values).float() if num_features else None\n",
    "        self.X_cat = torch.tensor(df[cat_features].values).long() if cat_features else None\n",
    "        self.Y = torch.tensor(df[label_col].values).float() \n",
    "        self.categories = categories\n",
    "        self.is_training = is_training\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.Y)\n",
    "    \n",
    "    def __getitem__(self,index):\n",
    "        if self.is_training:\n",
    "            return ((self.X_num[index],self.X_cat[index]),self.Y[index])\n",
    "        else:\n",
    "            return (self.X_num[index],self.X_cat[index])\n",
    "    \n",
    "    def get_categories(self):\n",
    "        return self.categories\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "dc41f88f",
   "metadata": {},
   "outputs": [],
   "source": [
    "dftrain_val,dftest = train_test_split(dfdata,test_size=0.2)\n",
    "dftrain,dfval = train_test_split(dftrain_val,test_size=0.2)\n",
    "\n",
    "ds_train = DfDataset(dftrain,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n",
    "\n",
    "ds_val = DfDataset(dfval,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n",
    "\n",
    "ds_test = DfDataset(dftest,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "51a784c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "dl_train = DataLoader(ds_train,batch_size = 2048,shuffle=True)\n",
    "dl_val = DataLoader(ds_val,batch_size = 2048,shuffle=False)\n",
    "dl_test = DataLoader(ds_test,batch_size = 2048,shuffle=False)\n",
    "\n",
    "for features,labels in dl_train:\n",
    "    break \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "937bec29",
   "metadata": {},
   "source": [
    "### 2，定义模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "63435a35",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 此处我们选择的交叉模块是CrossNetMatrix, 也就是构建的DCNV2模型 \n",
    "# 读者也可以尝试CrossNetVector和CrossNetMix \n",
    "def create_net():\n",
    "    net = DeepCross(\n",
    "        d_numerical= ds_train.X_num.shape[1],\n",
    "        categories= ds_train.get_categories(),\n",
    "        d_embed_max = 8, \n",
    "        n_cross = 2, cross_type = \"matrix\",\n",
    "        mlp_layers = [128,64,32], mlp_dropout=0.25,\n",
    "        stacked = True,\n",
    "        n_classes = 1\n",
    "        \n",
    "    )\n",
    "    return net \n",
    "\n",
    "from torchkeras import summary\n",
    "\n",
    "net = create_net()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2869f709",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "33b34700",
   "metadata": {},
   "source": [
    "### 3，训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b16c0558-af36-4a41-a96f-661a2f11a811",
   "metadata": {},
   "source": [
    "我们使用梦中情炉torchkeras来实现最优雅的训练循环。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "32761275",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras.metrics import AUC\n",
    "from torchkeras import KerasModel \n",
    "\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "\n",
    "metrics_dict = {\"auc\":AUC()}\n",
    "\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=0.002, weight_decay=0.001) \n",
    "\n",
    "model = KerasModel(net,\n",
    "                   loss_fn = loss_fn,\n",
    "                   metrics_dict= metrics_dict,\n",
    "                   optimizer = optimizer\n",
    "                  )         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "306a3dc9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< 🐌 cpu is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiEAAAGJCAYAAABcsOOZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB8Z0lEQVR4nO3dd1hT1xsH8G8SIGyQIVtAUVwILvi5qra0ruJA6644WmsdVVHraN2ttLVa3GiLbW21TtRWrFapWLUoFlwooigKIluGbEjO749rIoEAYV7G+3me+5CcnHvzXkLIm3PPEDDGGAghhBBC6pmQ7wAIIYQQ0jxREkIIIYQQXlASQgghhBBeUBJCCCGEEF5QEkIIIYQQXlASQgghhBBeUBJCCCGEEF5QEkIIIYQQXlASQgghhBBeUBJCeLVmzRoIBAKkpqbyHUq9efLkCQQCAX766Se+QyF16JtvvkH79u0hlUr5DqXeNMT3s5+fH1q1aoWCggK+QyFKUBJCmqUNGzbgxIkTfIfRpGVkZGDmzJkwNTWFjo4OBg4ciPDwcJX2FQgE5W5vv/22Qt2EhATMnDkT9vb20NLSQps2beDt7Y20tDSFet9//z369+8PMzMziMVi2NvbY9q0aXjy5EmZ58/MzMSnn36Ktm3bQktLC7a2tpgxYwZiY2NVij8rKwtff/01li5dCqGw8n+z+fn5KCoqUunYTVlUVBQWLlyI3r17Q1NTEwKBQOnrAwALFy5Et27dYGRkBG1tbXTo0AFr1qxBdna2Qr2pU6eisLAQu3fvroczIFWlxncAhPBhw4YNGDNmDEaOHMl3KE2SVCrFsGHDcOvWLSxZsgQmJibYuXMnBgwYgLCwMLRt27bC/X/55ZcyZf/99x+2bNmCd955R16WnZ2NXr16IScnB7Nnz4aNjQ1u3bqF7du348KFCwgLC5MnATdu3IC9vT2GDx+OFi1aICYmBt9//z1OnTqFW7duwdLSUh7722+/jXv37mH27Nlo164doqOjsXPnTpw9exaRkZHQ09OrMP69e/eiuLgYEyZMKLfO9evXsXXrVvz1119ITk6GQCCAlZUVRo0ahU8++QQODg4VPkdTFBISgq1bt6Jjx47o0KEDbt68WW7d69evo1+/fpg2bRo0NTVx48YNfPXVVzh//jz++ecf+euuqakJLy8vbN68GfPmzYNAIKinsyEqYYTwaPXq1QwAS0lJqdfn1dHRYV5eXvX6nDIxMTEMAPvxxx95ef76cOjQIQaAHTlyRF6WnJzMDA0N2YQJE6p1zBkzZjCBQMDi4uLkZfv372cA2KlTpxTqrlq1igFg4eHhFR7zv//+YwCYj4+PvOzKlSsMANu+fbtC3b179zIALCAgoNJYu3TpwiZPnqz0saKiIjZnzhwmEAhYv3792Lfffsv++OMPduzYMbZhwwbm4uLCNDU1yzx/Y1DT93NaWhrLyspijDG2ceNGBoDFxMSovP+3337LALCQkBCFctnrHBQUVK24SN2hyzGkQUhNTcXYsWOhr68PY2NjzJ8/H/n5+WXq/frrr+jevTu0tLRgZGSE8ePHIy4uTqHOw4cPMXr0aJibm0NTUxPW1tYYP348MjMzAXBN/Tk5Ofj555/lTfxTp05VGldSUhLU1NSwdu3aMo9FRUVBIBBg+/btAIAXL15g8eLFcHJygq6uLvT19TFkyBDcunWrhr8doLCwEKtWrUL37t1hYGAAHR0d9OvXDxcuXFCoFxwcDIFAgODgYIXy8vqh3L9/H2PHjoWpqSm0tLTg6OiIzz77rMbxHj16FGZmZvD09JSXmZqaYuzYsTh58mSVr88XFBTg2LFj6N+/P6ytreXlWVlZAAAzMzOF+hYWFgAALS2tCo9rZ2cHgLt0VFvHjImJwe3bt+Hu7q708WnTpuHAgQM4ffo0/vnnHyxatAjvvvsuPD09sXz5cty4cQN+fn5YvHgx/Pz8yuwfHx+P6dOnyy8rderUCXv37lWoI/s7OHToEFasWAFzc3Po6Ohg+PDhZd4vAHDkyBH5+8rExASTJ09GfHx8mXqq/r1kZGRg6tSpMDQ0hIGBAaZNm4bc3NwKf28AYGRkVGkrU0WUvZ4A0L17dxgZGeHkyZPVPjapI3xnQaR5k31zcnJyYh4eHmz79u1s8uTJDAB7//33Fep+8cUXTCAQsHHjxrGdO3eytWvXMhMTE2ZnZ8fS09MZY4wVFBQwe3t7Zmlpyb744gv2ww8/sLVr17KePXuyJ0+eMMYY++WXX5hYLGb9+vVjv/zyC/vll1/Yv//+W26Mb775JuvYsWOZ8rVr1zKRSMQSExMZY4xdv36dtWnThi1btozt3r2brVu3jllZWTEDAwMWHx8v3686LSEpKSnMwsKCeXt7s127drFvvvmGOTo6MnV1dXbjxg15vQsXLjAA7MKFCwr7K3vOW7duMX19fWZsbMyWL1/Odu/ezT799FPm5OQkr1NYWMhSUlJU2iQSiXw/BwcHNmTIkDLn8cMPPzAA7Pbt2yqfO2OMBQQEMADs+++/Vyi/e/cuEwqFrHfv3iwkJITFxcWxwMBAZm1tzUaOHKn0WKmpqSwpKYldv36deXh4MADsr7/+kj+ekpLCdHR0WPv27VlQUBB79uwZCw4OZk5OTqxnz56sqKiowlh//fXXcs9x3759TEdHh0VERMjLpFIpe/nypcLzSyQSdurUKaalpSX/u2WMscTERGZtbc1sbGzYunXr2K5du9jw4cMZAPbdd9/J68n+DpycnFiXLl3Y5s2b2bJly5impiZr164dy83Nldf98ccfGQDWs2dP9t1337Fly5YxLS0thfcVY6r9vcjez127dmWenp5s586d7IMPPmAA2Kefflrh7600VVpCioqKWEpKCouPj2dnz55l7du3Z3p6eiwtLa1MXXd3d9a9e/cqxUDqHiUhhFeyf1rDhw9XKJ89ezYDwG7dusUYY+zJkydMJBKxL7/8UqHenTt3mJqamrz8xo0bZS4DKFOVyzG7d+9mANidO3cUyjt27MjefPNN+f38/HyFD2LGuA9/sVjM1q1bp1BW1SSkuLiYFRQUKJSlp6czMzMzNn36dHlZVZKQN954g+np6bGnT58q1JVKpWWOp8pW8sNCR0dHIS6ZwMBABoCdOXNG5XNnjLHRo0czsVis8KEo88MPPzBDQ0OFWLy8vMpNFsRisbyesbEx27p1a5k6p06dYhYWFgrHHDRokEKyUJ7PP/+cAShTVyqVMnt7e+br6ysvO3nyJLO0tGQAWKtWrdjZs2cVfpejRo1iK1askNefMWMGs7CwYKmpqQrHHj9+PDMwMJAnF7LXzcrKSn55gzHGDh8+zACwLVu2MMa4JLNly5asc+fOLC8vT+H8AbBVq1bJy1T5e5G9n0u/9qNGjWLGxsaV/u5KUiUJCQkJUXiNHB0dy/zty8ycOZNpaWlVKQZS9+hyDGkQ5syZo3B/3rx5AIDTp08DAAICAiCVSjF27FikpqbKN3Nzc7Rt21Z+WcLAwAAAcPbsWZWaf1Xh6ekJNTU1HDp0SF4WERGBe/fuYdy4cfIysVgs7wwnkUiQlpYGXV1dODo6qjwqpDwikQgaGhoAuI6TL168QHFxMXr06FGtY6ekpOCff/7B9OnT0apVK4XHSnbcc3Z2xrlz51TazM3N5fvl5eVBLBaXeV5NTU3546rKyspCYGAghg4dCkNDwzKPW1lZwdXVFb6+vjh+/Di8vb2xf/9+LFu2TOnx/vzzT5w+fRqbNm1Cq1atkJOTU6aOqakpunbtii+//BInTpzAmjVrcOnSJUybNq3SeNPS0qCmpgZdXV2F8rCwMCQnJ2PGjBkAuMsqEyZMgKurK44dO4aFCxdi+vTpCvuMHDlSfmmNMYZjx47Bw8MDjDGF98GgQYOQmZlZ5m9hypQpCpc3xowZAwsLC/n76r///kNycjJmz54tf20AYNiwYWjfvj0CAwMBqP73IjNr1iyF+/369UNaWpr8Uldt6dixI86dO4cTJ07g008/hY6OTpnRMTItWrRAXl5erf1fILWDRseQBqH0aIk2bdpAKBTKh+c9fPgQjLFyR1Woq6sDAOzt7eHt7Y3Nmzdj//796NevH4YPH47JkyfLE5SqMjExwVtvvYXDhw9j/fr1AIBDhw5BTU1Noc+DVCrFli1bsHPnTsTExEAikcgfMzY2rtZzl/Tzzz9j06ZNuH//vsJwTnt7+yof6/HjxwCAzp07V1ivRYsW5fZtqIiWlpbSfh+yfj6V9aso6dixY8jPz8ekSZPKPHblyhW8++67uHr1Knr06AGA++DW19fH2rVrMX36dHTs2FFhn4EDBwIAhgwZghEjRqBz587Q1dXF3LlzAXC/m4EDB2Lfvn0YPXo0AGDEiBGws7PD1KlT8eeff2LIkCEqxy8TFhaGHj16yJOT/fv3w8rKCkePHoVIJAIAGBoaKiQ6ZmZmSElJAcAlAhkZGdizZw/27Nmj9DmSk5MV7pd+vwgEAjg4OMjfV0+fPgUAODo6ljlW+/btcfnyZQCq/73IlE5UWrRoAQBIT0+Hvr6+SsdQhb6+vvzvc8SIEThw4ABGjBiB8PBwODs7K9RljAFQnjQR/lASQhqk0v8opFIpBAIB/vzzT/k/7JJKfuvctGkTpk6dipMnT+Kvv/7CJ598Ah8fH1y9elWhU2NVjB8/HtOmTcPNmzfh4uKCw4cP46233oKJiYm8zoYNG7By5UpMnz4d69evh5GREYRCIRYsWFDjCat+/fVXTJ06FSNHjsSSJUvQsmVLiEQi+Pj44NGjR/J65f2DLZkQVUVhYSFevHihUl1TU1P5a2NhYYGEhIQydWRlsuGwqti/fz8MDAzw7rvvlnls9+7dMDMzkycgMsOHD8eaNWvw77//lklCSmrTpg26du2K/fv3y5OQn376Cfn5+WWeb/jw4QC4xKeiJMTY2BjFxcV4+fKlQitEWlqawnk/efIEXbt2Vfh7dnV1VThWXFycPIGV/Q1NnjwZXl5eSp+7S5cu5cZVn5S9R4HXiUBd8fT0xPvvv4+DBw+WSULS09Ohra1dpQSY1D1KQkiD8PDhQ4Vv9NHR0ZBKpfLe7m3atAFjDPb29mjXrl2lx3NycoKTkxM+//xz/Pvvv+jTpw/8/PzwxRdfAKj6t6GRI0fio48+kl+SefDgAZYvX65Q5+jRoxg4cCD8/f0VyjMyMhSSleo4evQoWrdujYCAAIXYV69erVBP9o2z9OgA2TdemdatWwPgLitV5N9//5W3HFQmJiZG/nq5uLjg0qVLkEqlCpN1Xbt2Ddra2iq9hgCXtFy4cAFTp05VenknKSlJaYIlaykqLi6u9Dny8vIUWm2SkpLAGCtzXFWP2b59ewDc76NkUqCvry8foQUA5ubmCA0NVdhX1uIAcB/Y/v7+8m/6pqam0NPTg0QiUbl16uHDhwr3GWOIjo6Wx2VrawuAG+n15ptvKtSNioqSP67q3wvfCgoKIJVKFX7PMjExMejQoQMPUZGKUJ8Q0iDs2LFD4f62bdsAQP6N09PTEyKRCGvXri3zbYoxJp8dMysrq8yHhJOTE4RCocIHjY6OTpkP6ooYGhpi0KBBOHz4MA4ePAgNDY0yE52JRKIysR05ckTpUMeqkn2zLHn8a9euISQkRKGera0tRCIR/vnnH4XynTt3Ktw3NTXFG2+8gb1795aZBbTkc1S3T8iYMWOQlJSEgIAAeVlqaiqOHDkCDw8PhYTi0aNHCq05JR08eBBSqVTppRgAaNeuHZKSksoMSf7tt98AAF27dgXAJQ7p6ell9g8NDcWdO3cUWlLatWsHxhgOHz5c4THL06tXLwBcf4uSOnTogOvXr8tbNEaMGIEbN25g1apVePz4MS5duoQlS5YA4CZWGz16NJ49e4b58+cD4P4GRo8ejWPHjilNBmSXbUrat28fXr58Kb9/9OhRJCQkyN9XPXr0QMuWLeHn56fw/vjzzz8RGRmJYcOGAVD976W+ZGRkKJ1h9ocffgCAMi1jABAeHo7evXvXeWykiuq/Lywhr5Ueortjxw75EN2JEycq1PXx8WEAWO/evdk333zDdu3axT799FPWtm1btnHjRsYYY8ePH2dWVlZswYIFbOfOnWzr1q2sZ8+eTF1dXWECo6FDhzIdHR22adMm9ttvv7GrV69WGqts6KWenh7z8PAo87hsgqypU6eyPXv2sHnz5jEjIyPWunVr1r9/f3m96oyOkU2UNXz4cLZ79262bNkyZmhoyDp16sRsbW0V6o4fP56pqakxb29vtmPHDjZkyBDWvXv3Ms958+ZNpqurKx9yuWfPHrZixQrm7OysclzlKS4uZv/73/+Yrq4uW7t2LduxYwfr1KkT09PTY/fv31eoa2trW+YcZLp3784sLS3LjDqSuX//PtPR0WG6urps+fLlzM/Pj02YMIEBYG+//ba8Xnp6unzEzqZNm5ifnx+bM2cO09bWZkZGRuzBgwfyuqmpqczc3JxpaGiwTz75hO3evZt99NFHTCQSsU6dOpUZpaRM586dy0zKlp+fzwwMDNjx48flZRs2bGBCoZABYGpqamzLli3ykR7vvPMOe/z4scIxEhMTma2tLdPW1mbz589nu3fvZj4+Puy9995jLVq0kNcrPURXNvRWU1OTOTg4sJycHHld2RBdNzc35uvry5YvX860tbXLDNFV5e+lvMnKZM9R2cRjGRkZbP369Wz9+vVs8ODBDABbtGgRW79+Pdu2bZu83vHjx5mNjQ1buHAh27lzJ/P19WWjR49mAoGA9ejRo8xrJJus7Pz58xU+P6l/lIQQXsn+ad27d4+NGTOG6enpsRYtWrC5c+cqDBmUOXbsGOvbty/T0dGRz+UwZ84cFhUVxRhj7PHjx2z69OmsTZs2TFNTkxkZGbGBAweW+edz//599sYbbzAtLS35kM7KZGVlyev/+uuvZR7Pz89nixYtYhYWFkxLS4v16dOHhYSEsP79+9c4CZFKpWzDhg3M1taWicVi1rVrV3bq1Cnm5eVV5gM8JSWFjR49mmlra7MWLVqwjz76iEVERCh9zoiICDZq1ChmaGjINDU1maOjI1u5cqXKcVXkxYsXbMaMGczY2Jhpa2uz/v37s+vXr5epV14Scv/+fQaAeXt7V/g89+/fZ2PGjGE2NjZMXV2d2drassWLFyt80BYUFLD58+ezLl26MH19fXm9GTNmKP1gfPbsGZs+fTqzt7dnGhoazMLCgn344YcqzwS6efNmpqurqzAfB2Pc33vr1q3Zixcv5GXx8fHsn3/+kc83c/nyZZacnFzusZOSkticOXPk52tubs7eeusttmfPHnkdWRLy22+/seXLl7OWLVsyLS0tNmzYsDJDbBnjZrjt2rUrE4vFzMjIiE2aNIk9e/asTL3K/l5qmoTI3hvKtpJ/I9HR0WzKlCmsdevWTEtLi2lqarJOnTqx1atXs+zs7DLHXbp0KWvVqpXCcGLSMAgY46EtjRBCmrDMzEy0bt0a33zzjXxILsCNDurTpw9EIhFOnjwpn4W1tKNHj2LUqFHldvCsTHBwMAYOHIgjR45gzJgx1TpGU1FQUAA7OzssW7ZMfmmLNBzUJ4QQQmqZgYEBPv30U2zcuFFhZJSmpiZOnz4NgUAAR0dHLF26FP/88w+ePn2K+/fvY9++fejVqxe8vLxqPLcM4fz4449QV1cvM3cJaRioJYQQHqkyBNbAwICGFTYxhYWF2L59O7Zv346YmBh5uaamJkaNGoW1a9dWutJwRaglhDQWNESXEB6pMgT2xx9/LHeBPdI4aWhowNvbG97e3njy5Ani4+OhqamJDh06QFtbm+/wCKk31BJCCI/S09MRFhZWYZ1OnTqV23eAEEIaM0pCCCGEEMIL6phKCCGEEF5QnxAlpFIpnj9/Dj09PVrsiBBCCKkCxhhevnwJS0tLhWUblKEkRInnz5/DxsaG7zAIIYSQRisuLq7SRUMpCVFCtvJlXFxcrS47TQghhDR1WVlZsLGxUVhFujy8JyE7duzAxo0bkZiYCGdnZ2zbtq3MctYl+fr6YteuXYiNjYWJiQnGjBkDHx8faGpqyuvEx8dj6dKl+PPPP5GbmwsHBwf8+OOPShc1UkZ2CUZfX5+SEEIIIaQaVOnOwGsScujQIXh7e8PPzw9ubm7w9fXFoEGDEBUVhZYtW5apf+DAASxbtgx79+5F79698eDBA0ydOhUCgQCbN28GwA157NOnDwYOHIg///wTpqamePjwoXyJc0IIIYQ0DLwO0XVzc0PPnj2xfft2AFyHUBsbG8ybNw/Lli0rU3/u3LmIjIxEUFCQvGzRokW4du0aLl++DABYtmwZrly5gkuXLlU7rqysLBgYGCAzM5NaQgghhJAqqMpnKG9DdAsLCxEWFgZ3d/fXwQiFcHd3R0hIiNJ9evfujbCwMISGhgIAHj9+jNOnT2Po0KHyOr///jt69OiB9957Dy1btkTXrl3x/fffVxhLQUEBsrKyFDZCCCGE1C3eLsekpqZCIpHAzMxModzMzAz3799Xus/EiRORmpqKvn37gjGG4uJizJo1CytWrJDXefz4MXbt2gVvb2+sWLEC169fxyeffAINDQ14eXkpPa6Pjw/Wrl1bpfhlzy+RSKq0H2kYRCIR1NTUaAg2IYTwiPeOqVURHByMDRs2YOfOnXBzc0N0dDTmz5+P9evXY+XKlQC4Szo9evTAhg0bAABdu3ZFREQE/Pz8yk1Cli9fDm9vb/l9Wc/e8hQWFiIhIQG5ubm1eHakvmlra8PCwgIaGhp8h0IIIc0Sb0mIiYkJRCIRkpKSFMqTkpJgbm6udJ+VK1fi/fffxwcffAAAcHJyQk5ODmbOnInPPvsMQqEQFhYW6Nixo8J+HTp0wLFjx8qNRSwWQywWqxS3VCpFTEwMRCIRLC0toaGhQd+mGxnGGAoLC5GSkoKYmBi0bdu20gl1CCGE1D7ekhANDQ10794dQUFBGDlyJADuAz4oKAhz585Vuk9ubm6ZDwuRSASA+2ABgD59+iAqKkqhzoMHD2Bra1srcRcWFso70NJql42XlpYW1NXV8fTpUxQWFioM8SaEEL5JJMClS0BCAmBhAfTrB7z6uGtSeL0c4+3tDS8vL/To0QOurq7w9fVFTk4Opk2bBgCYMmUKrKys4OPjAwDw8PDA5s2b0bVrV/nlmJUrV8LDw0OejCxcuBC9e/fGhg0bMHbsWISGhmLPnj3Ys2dPrcZO35wbP3oNCSENUUAAMH8+8OzZ6zJra2DLFsDTs/aep0EkOoxn27ZtY61atWIaGhrM1dWVXb16Vf5Y//79mZeXl/x+UVERW7NmDWvTpg3T1NRkNjY2bPbs2Sw9PV3hmH/88Qfr3LkzE4vFrH379mzPnj1ViikzM5MBYJmZmWUey8vLY/fu3WN5eXlVOiZpeOi1JIQ0NMeOMSYQMAYobgIBtx07VnvPY22t+BzW1rVz/Io+Q0vjdZ6QhqqiMc75+fmIiYmBvb09NeE3cvRaEkIaEokEsLNTbAEpSSAAzM2Bq1cBfX1AWxuoTr/6gABgzBgu9Sh9fAA4erRmLS5VmSekUY2OaWoaRFNYNdnZ2WHBggVYsGAB36EQQkijl5MD7N5dfgICcElDQgJQsoujmhqXjGhrA++8A/z88+vHxo3jfurocI/r6ACamtxlHWXND4xxiciCBcCIEfXzeURJCE/q65pfSQMGDICLiwt8fX1rfKzr169DR0en5kERQkgzVlQEvPEG8N9/QHGxavsIhYBUyt0uLgaysrgtM1OxXkCA6seUYQyIi+O+IA8YULV9q4OSEB6U1xQWH8+V17QprLoYY5BIJFBTq/zPwtTUtB4iIoSQhqeqrdiMAY8eAZcvc1tR0esWC3V1rhWkuBgwMQFSUyt//qAgoHdvIDeX21f2s+SATcYAP7+ydW7d4vavTEJC5XVqRc27oDQ91e2Ymp1d/iarXlxctjNQ6c5H1tZcvYqOW1VeXl4MgML2448/MgDs9OnTrFu3bkxdXZ1duHCBRUdHs+HDh7OWLVsyHR0d1qNHD3bu3DmF49na2rLvvvtOfh8A+/7779nIkSOZlpYWc3BwYCdPnlQptuLiYjZ9+nRmZ2fHNDU1Wbt27Zivr69Cnf79+7P58+crlI0YMUKh43J+fj779NNPmbW1NdPQ0GBt2rRhP/zwQ7nPSx1TCSFVpWqHzlu3GPP1ZWzMGMbMzRXri8WM5ee/rnvtGmNPnrz+fFDWMVX2+WBjo/j5UFUXLpT/+VNyu3Ch+s9RlY6pNEaxFunqlr+NHs3VuXSp8mt+z55x9WTs7Moer6q2bNmCXr164cMPP0RCQgISEhLks8IuW7YMX331FSIjI9GlSxdkZ2dj6NChCAoKwo0bNzB48GB4eHggNja2wudYu3Ytxo4di9u3b2Po0KGYNGkSXrx4UWlsUqkU1tbWOHLkCO7du4dVq1ZhxYoVOHz4cJXOccqUKfjtt9+wdetWREZGYvfu3dCtzi+LEEKUkLVil/4fHh/P/Y8vOSfmypVc34qjR4HERK4DaZ8+wNKlXL2Sc1y6unL9PEQi7pI8oPh4yfu+vjXrq9GvH3fpv7w5NgUCwMaGq1cf6HJMPVO1iau2m8IMDAygoaEBbW1t+Yy0sjV61q1bh7ffflte18jICM7OzvL769evx/Hjx/H777+XO5EcAEydOhUTJkwAAGzYsAFbt25FaGgoBg8eXGFs6urqCmv32NvbIyQkBIcPH8bYsWNVOr8HDx7g8OHDOHfunHxRxNatW6u0LyGEVEYi4frxldehEwDmzQNGjuSShMGDuUssfftyH+g9enCdQivj6cklLsr6DPr61vxSvSzRGTOGSzhKnk9tJTpVQUlILcrOLv8x2QtqYaHasUrWe/Kk2iGppEePHgr3s7OzsWbNGgQGBiIhIQHFxcXIy8urtCWkS5cu8ts6OjrQ19dHcnKySjHs2LEDe/fuRWxsLPLy8lBYWAgXFxeVz+HmzZsQiUTo37+/yvsQQoiqKmvFBrgvj7IOnR9/zG3V4enJjU6pq9GTdZ3oVAUlIbVIlcEisqaw+HjlGbVAwD1esimsrgehlB7lsnjxYpw7dw7ffvstHBwcoKWlhTFjxqCwsLDC46irqyvcFwgEkMq6cFfg4MGDWLx4MTZt2oRevXpBT08PGzduxLVr1+R1hEKhfGp+maKiIvltLS2tSp+HEEKqq75bsUWiuh2dUteJjqooCalnfDaFaWhoQCKRVFrvypUrmDp1KkaNGgWAaxl5UofNMVeuXEHv3r0xe/ZsedmjR48U6piamiKhxLtbIpEgIiICAwcOBMAtZiiVSnHx4kX55RhCCKkt1WnFbujqOtFRBXVM5YGsKczKSrHc2rpuh+fa2dnh2rVrePLkCVJTU8ttpWjbti0CAgJw8+ZN3Lp1CxMnTlSpRaO62rZti//++w9nz57FgwcPsHLlSly/fl2hzptvvonAwEAEBgbi/v37+Pjjj5GRkaFwbl5eXpg+fTpOnDiBmJgYBAcHV7lzKyGEKNPQOnQ2FZSE8MTTk+vrceECcOAA9zMmpm6vxS1evBgikQgdO3aEqalpuX08Nm/ejBYtWqB3797w8PDAoEGD0K1btzqL66OPPoKnpyfGjRsHNzc3pKWlKbSKAMD06dPh5eWFKVOmoH///mjdurW8FURm165dGDNmDGbPno327dvjww8/RE5OTp3FTQhpPupj5EpzRGvHKEFrxzQP9FoSQqpK2WzXNjb136GzIaO1YwghhJBa9tdfwNChDaNDZ1NBl2NInZs1axZ0dXWVbrNmzeI7PEIIqdSjR8CQIYC9PbdOy4ABwIQJ3E9KQKqPWkJInVu3bh0WL16s9LHKmuoIIaQh2LiRWzSua1egRQu+o2k6KAkhda5ly5Zo2bIl32EQQki1JCQAP/7I3V62jN9Ymhq6HEMIIYRUYMsWoLAQ6NWLhuDWNkpCCCGEkHJkZgK7dnG3ly8vf54QUj2UhBBCCCHl2LWL64jaqRMwbBjf0TQ9lIQQQggh5XjwgPu5dCkgpE/MWke/UkIIIaQce/cCt28D48fzHUnTREkIjySMITg9Hb8lJSE4PR2SBj55rZ2dHXx9ffkOgxBC6pWTE1BqkXBSS2iILk8CUlIwPzoazwoK5GXWYjG2ODjA09SUx8gIIYSEhQFmZtyidaTuUEsIDwJSUjDm7l2FBAQA4gsKMObuXQSkpPAUGSGEEMaAadOA1q2BP/7gO5qmjZKQWsAYQ45EotKWVVyMTx4+hLILL7Ky+dHRyCourvRYVVl7cM+ePbC0tIRUKlUoHzFiBKZPn45Hjx5hxIgRMDMzg66uLnr27Inz589X+3eyefNmODk5QUdHBzY2Npg9ezays7Plj69ZswYuLi4K+/j6+sLOzk6hbO/evejUqRPEYjEsLCwwd+7casdECCGq+PNP4M4dQCwG+vblO5qmrUEkITt27ICdnR00NTXh5uaG0NDQCuv7+vrC0dERWlpasLGxwcKFC5Gfn6+07ldffQWBQIAFCxbUQeScXKkUupcuqbQZXL6M+MLCco/FADwrKIDB5cuVHiu3VEJRkffeew9paWm4cOGCvOzFixc4c+YMJk2ahOzsbAwdOhRBQUG4ceMGBg8eDA8PD8TGxlbrdyIUCrF161bcvXsXP//8M/7++298+umnVTrGrl27MGfOHMycORN37tzB77//DgcHh2rFQwghqvLx4X7OmkVTtNc13vuEHDp0CN7e3vDz84Obmxt8fX0xaNAgREVFKZ3q+8CBA1i2bBn27t2L3r1748GDB5g6dSoEAgE2b96sUPf69evYvXs3unTpUl+n02C1aNECQ4YMwYEDB/DWW28BAI4ePQoTExMMHDgQQqEQzs7O8vrr16/H8ePH8fvvv1er9aFk0mdnZ4cvvvgCs2bNws6dO1U+xhdffIFFixZh/vz58rKePXtWORZCCFHV5cvcpqEBLFzIdzRNH+9JyObNm/Hhhx9i2rRpAAA/Pz8EBgZi7969WKZkkv5///0Xffr0wcSJEwFwH3ATJkzAtWvXFOplZ2dj0qRJ+P777/HFF1/U6TloC4XIVnEu338yMjD0zp1K6512csIbhoaVPm9VTJo0CR9++CF27twJsViM/fv3Y/z48RAKhcjOzsaaNWsQGBiIhIQEFBcXIy8vr9otIefPn4ePjw/u37+PrKwsFBcXIz8/H7m5udDW1q50/+TkZDx//lyeMBFCSH34+mvu55QpgKUlv7E0B7xejiksLERYWBjc3d3lZUKhEO7u7ggJCVG6T+/evREWFia/ZPP48WOcPn0aQ4cOVag3Z84cDBs2TOHY5SkoKEBWVpbCVhUCgQA6IpFK2ztGRrAWi1HezL8CADZiMd4xMqr0WIIqzh/s4eEBxhgCAwMRFxeHS5cuYdKkSQCAxYsX4/jx49iwYQMuXbqEmzdvwsnJCYUVXDoqz5MnT/Duu++iS5cuOHbsGMLCwrBjxw4AkB9PKBSW6dNSVFQkv62lpVXl5yWEkJq4cwc4dYqbmn3JEr6jaR54bQlJTU2FRCKBmZmZQrmZmRnu37+vdJ+JEyciNTUVffv2BWMMxcXFmDVrFlasWCGvc/DgQYSHh+P69esqxeHj44O1a9dW/0SqQCQQYIuDA8bcvQsBoNBBVZZS+Do4QFQHCxRoamrC09MT+/fvR3R0NBwdHdGtWzcAwJUrVzB16lSMGjUKANeS9OTJk2o9T1hYGKRSKTZt2gThq9aaw4cPK9QxNTVFYmIiGGPyZOrmzZvyx/X09GBnZ4egoCAMHDiwWnEQQkhV3LkD6OgAQ4cC7drxHU3z0CA6plZFcHAwNmzYgJ07dyI8PBwBAQEIDAzE+vXrAQBxcXGYP38+9u/fD01NTZWOuXz5cmRmZsq3uLi4ujwFeJqa4minTrASixXKrcViHO3UqU7nCZk0aZL8cpesFQQA2rZti4CAANy8eRO3bt3CxIkTy4ykUZWDgwOKioqwbds2PH78GL/88gv8/PwU6gwYMAApKSn45ptv8OjRI+zYsQN//vmnQp01a9Zg06ZN2Lp1Kx4+fIjw8HBs27atWjERQkhlJk4Enj4Fvv2W70iaEcajgoICJhKJ2PHjxxXKp0yZwoYPH650n759+7LFixcrlP3yyy9MS0uLSSQSdvz4cQaAiUQi+QaACQQCJhKJWHFxcaVxZWZmMgAsMzOzzGN5eXns3r17LC8vT/UTLUexVMouvHjBDiQmsgsvXrBiqbTGx6yMRCJhFhYWDAB79OiRvDwmJoYNHDiQaWlpMRsbG7Z9+3bWv39/Nn/+fHkdW1tb9t1336n0PJs3b2YWFhZMS0uLDRo0iO3bt48BYOnp6fI6u3btYjY2NkxHR4dNmTKFffnll8zW1lbhOH5+fszR0ZGpq6szCwsLNm/evBqcvaLafC0JIYRwKvoMLU3AGL9zhbu5ucHV1VX+DVcqlaJVq1aYO3eu0o6p3bt3h7u7O76W9R4C8Ntvv2HGjBl4+fIlcnNz8fTpU4V9pk2bhvbt22Pp0qXo3LlzpTFlZWXBwMAAmZmZ0NfXV3gsPz8fMTExsLe3V7mlhTRM9FoSQgAgOZlbqI7mBKkdFX2Glsb76Bhvb294eXmhR48ecHV1ha+vL3JycuSjZaZMmQIrKyv4vBq47eHhgc2bN6Nr165wc3NDdHQ0Vq5cCQ8PD4hEIujp6ZVJNHR0dGBsbKxSAkIIIaR52bIF2LAB+PhjoAqzCJBawHsSMm7cOKSkpGDVqlVITEyEi4sLzpw5I++sGhsbK+/cCACff/45BAIBPv/8c8THx8PU1BQeHh748ssv+TqFZmf//v346KOPlD5ma2uLu3fv1nNEhBBSPVlZwKvBe3j7bX5jaY54vxzTENHlmIq9fPkSSUlJSh9TV1eHra1tPUdUPfRaEkI2bgQ+/RRo3x64exeo4vRLRIlGdTmGND56enrQ09PjOwxCCKmR/HxANtH20qWUgPCBfuXVRA1IjR+9hoQ0b/v2AYmJgLU1NzyX1D9KQqpIXV0dAJCbm8tzJKSmZK+h7DUlhDQfEgnwzTfc7UWLuLViSP2jyzFVJBKJYGhoiOTkZACAtrZ2ladPJ/xijCE3NxfJyckwNDSESCTiOyRCSD17+hRgDDAyAj78kO9omi9KQqrB3NwcAOSJCGmcDA0N5a8lIaR5ad0aiIriNh0dvqNpvigJqQaBQAALCwu0bNlSYdE10nioq6tTCwghzZyaGtCpE99RNG+UhNSASCSiDzJCCGlk/vgDGDSI+oE0BNQxlRBCSLNx7RowfDg3L0hBAd/REEpCCCGENBtffcX9HDAAKLWQOeEBJSGEEEKahchI4MQJQCAAlizhOxoCUBJCCCGkmZDNCzJyJNChA6+hkFcoCSGEENLkxcUBv/7K3V62jN9YyGuUhBBCCGnyNm0CiouBN98EXF35jobIUBJCCCGkSWOMmyEVoFaQhobmCSGEENKkCQTA8ePAnTtA5858R0NKoiSEEEJIs+DkxHcEpDS6HEMIIaTJunQJSEjgOwpSHkpCCCGENEmFhcCECYC9PXD5Mt/REGUoCSGEENIk7d8PxMcDRkZAz558R0OUoT4hhBBCmgyJhLsEEx8PrF7NlXl70xTtDRUlIYQQQpqEgABg/nzg2bPXZQIBYGHBX0ykYpSEEEIIafQCAoAxY7g5QUpiDHj/fUBLC/D05Cc2Uj7qE0IIIaRRk0i4FpDSCUhJCxZw9UjDQkkIIYSQRu3SJcVLMKUxxq0dc+lS/cVEVENJCCGEkEZN1XlAaL6QhqdBJCE7duyAnZ0dNDU14ebmhtDQ0Arr+/r6wtHREVpaWrCxscHChQuRn58vf9zHxwc9e/aEnp4eWrZsiZEjRyIqKqquT4MQQggPVO14Sh1UGx7ek5BDhw7B29sbq1evRnh4OJydnTFo0CAkJycrrX/gwAEsW7YMq1evRmRkJPz9/XHo0CGsWLFCXufixYuYM2cOrl69inPnzqGoqAjvvPMOcnJy6uu0CCGE1JOsrIofFwgAGxugX7/6iYeoTsBYRV156p6bmxt69uyJ7du3AwCkUilsbGwwb948LFOy3OHcuXMRGRmJoKAgedmiRYtw7do1XC5nSryUlBS0bNkSFy9exBtvvFFpTFlZWTAwMEBmZib09fWreWaEEELq2pEjwMSJQHExd18gUOygKhBwP48epdEx9aUqn6G8toQUFhYiLCwM7u7u8jKhUAh3d3eEhIQo3ad3794ICwuTX7J5/PgxTp8+jaFDh5b7PJmZmQAAIyMjpY8XFBQgKytLYSOEENKw/fgjMH48l4CMHw8cOgRYWSnWsbamBKQh43WekNTUVEgkEpiZmSmUm5mZ4f79+0r3mThxIlJTU9G3b18wxlBcXIxZs2YpXI4pSSqVYsGCBejTpw86l7OGs4+PD9auXVuzkyGEEFJvtm0DPvmEu/3BB4CfHyASAaNHv160zsKCuwQjEvEbKykf731Cqio4OBgbNmzAzp07ER4ejoCAAAQGBmL9+vVK68+ZMwcRERE4ePBgucdcvnw5MjMz5VtcXFxdhU8IIaSGfv31dQKycCGwZ8/rREMkAgYM4BauGzCAEpCGjteWEBMTE4hEIiQlJSmUJyUlwdzcXOk+K1euxPvvv48PPvgAAODk5IScnBzMnDkTn332GYTC13nV3LlzcerUKfzzzz+wtrYuNw6xWAwxLSxACCGNwvDh3IJ0Q4dy68PI+n2QxofXlhANDQ10795doZOpVCpFUFAQevXqpXSf3NxchUQDAESvUl1ZH1vGGObOnYvjx4/j77//hr29fR2dASGEkPpQsrOpvj7wzz/AmjWUgDR2vK8d4+3tDS8vL/To0QOurq7w9fVFTk4Opk2bBgCYMmUKrKys4OPjAwDw8PDA5s2b0bVrV7i5uSE6OhorV66Eh4eHPBmZM2cODhw4gJMnT0JPTw+JiYkAAAMDA2hpafFzooQQQqqluBiYMQPo2BFYupQr09TkNyZSO3hPQsaNG4eUlBSsWrUKiYmJcHFxwZkzZ+SdVWNjYxVaPj7//HMIBAJ8/vnniI+Ph6mpKTw8PPDll1/K6+zatQsAMGDAAIXn+vHHHzF16tQ6PydCCCG1o6CAG4IbEACoqXEdTx0c+I6K1Bbe5wlpiGieEEII4V9uLje09uxZQEMDOHwYGDGC76hIZaryGcp7SwghhBBSWlYW8O673HBbbW3gxAng7bf5jorUNkpCCCGENChpacDgwcB//3GdUE+fBvr04TsqUhcoCSGEENKg/Pknl4CYmHCXYrp14zsiUlcoCSGEENKgTJ4MpKcDb73FjYghTRclIYQQQnj38CFgbAzIlviaN4/feEj9aHTTthNCCGlabt8G+vYFhgwBXr7kOxpSn6glhBBCSL2RSBQXmNPQAIYNAzIyAEtLID8f0NPjO0pSXygJIYQQUi8CAoD584Fnz16XCQTclOy9enGjYAwNeQuP8ICSEEIIIXUuIAAYM0ZxDRjg9f05cygBaY6oTwghhJA6JZFwLSDlzc8tEADLl3P1SPNCSQghhJA6demS4iWY0hgD4uK4eqR5oSSEEEJInUpIqN16pOmgJIQQQkidsrCo3Xqk6aAkhBBCSJ3q1w8wNy//cYEAsLHh6pHmhZIQQgghdUoo5OYAUUYg4H76+gIiUb2FRBoISkIIIYTUqVOngPBwLsko3SJibQ0cPQp4evITG+EXzRNCCCGkzuTlccNzAWDxYuDLLxVnTO3Xj1pAmjNKQgghhNSZjRuBmBiuxePzz7mEY8AAvqMiDQVdjiGEEFInYmIAHx/u9qZNgK4uv/GQhoeSEEIIIXUiJISbBfXNN4H33uM7GtIQ0eUYQgghdWLiRKBHD24EjGwUDCElURJCCCGkzrRrx3cEpCGjyzGEEEJq1a+/Av/9x3cUpDGgJIQQQkitiY0FPvoIcHUFrl3jOxrS0DWIJGTHjh2ws7ODpqYm3NzcEBoaWmF9X19fODo6QktLCzY2Nli4cCHy8/NrdExCCCE1t3gxkJsL9O3LJSKEVIT3JOTQoUPw9vbG6tWrER4eDmdnZwwaNAjJyclK6x84cADLli3D6tWrERkZCX9/fxw6dAgrVqyo9jEJIYTU3PnzwJEj3DTt27dTZ1RSOQFjjPEZgJubG3r27Int27cDAKRSKWxsbDBv3jwsW7asTP25c+ciMjISQUFB8rJFixbh2rVruHz5crWOWVpWVhYMDAyQmZkJfX392jhNQghp0goLAWdn4P59YN48YOtWviMifKnKZyivLSGFhYUICwuDu7u7vEwoFMLd3R0hISFK9+nduzfCwsLkl1ceP36M06dPY+jQodU+ZkFBAbKyshQ2Qgghqtu6lUtATE2Bdev4joY0FrwO0U1NTYVEIoGZmZlCuZmZGe7fv690n4kTJyI1NRV9+/YFYwzFxcWYNWuW/HJMdY7p4+ODtWvX1sIZEUJI85OYCMj+hX79NWBoyGs4pBHhvU9IVQUHB2PDhg3YuXMnwsPDERAQgMDAQKxfv77ax1y+fDkyMzPlW1xcXC1GTAghTZupKbB5M/Duu4CXF9/RkMaE15YQExMTiEQiJCUlKZQnJSXBvPR6z6+sXLkS77//Pj744AMAgJOTE3JycjBz5kx89tln1TqmWCyGWCyuhTMihJDmRyQCPvyQ2wipCl5bQjQ0NNC9e3eFTqZSqRRBQUHo1auX0n1yc3MhFCqGLXq1DjRjrFrHJIQQUnXFxdxwXEKqi/fLMd7e3vj+++/x888/IzIyEh9//DFycnIwbdo0AMCUKVOwfPlyeX0PDw/s2rULBw8eRExMDM6dO4eVK1fCw8NDnoxUdkxCCCE1t2MH0KEDcOoU35GQxor3tWPGjRuHlJQUrFq1ComJiXBxccGZM2fkHUtjY2MVWj4+//xzCAQCfP7554iPj4epqSk8PDzw5ZdfqnxMQgghNZOUBKxaBWRlAc+f8x0Naax4nyekIaJ5QgghpGJTpwI//8ytknv1KtcvhBCgEc0TQgghpPH5918uAQG4mVEpASHVRUkIIYQQlUkkwNy53O3p0wE3N37jIY0bJSGEEEJUtmcPcOMGNyHZV1/xHQ1p7CgJIYQQorKwMO7n+vXcJGWE1ATvo2MIIYQ0Hj/8ALz/PtCnD9+RkKaAkhBCCCFV0r8/3xGQpoIuxxBCCKmQVAqsWQMkJPAdCWlqKAkhhBBSob17uVVyXV2BwkK+oyFNCSUhhBBCyvXiBbBsGXfb2xvQ0OA3HtK0UBJCCCGkXCtXAmlpQKdOr+cHIaS2UBJCCCFEqRs3AD8/7vb27YC6Or/xkKaHkhBCCCFlSKXAnDncz/HjgQED+I6INEWUhBBCCHnt5k1gyBCc+eomQkIAHR1g40a+gyJNFc0TQgghBAC3Lkzc5mOwO3MGbQ17YsECF9jYANbWfEdGmipqCSGEEIKAAMDODsj45Q8AQPbBP3D0KFdGSF2hJIQQQpq5gABgzBig8FkSXHALANAVN1H4LBljxnCPE1IXKAkhhJBmTCIB5s8HGAMG4azCY++8ur9gAVePkNpGSQghhDRDUikQFQV8/jnw7BlXNhSBKIIIAFAEEYYiEIwBcXHApUs8BkuaLOqYSgghjYBEwiUCCQmAhQXQrx8gEqm+f0oKII2LhxmSAAA3woAPZ3KPdQUgcHmBIVGnoJ7HNXmoQ4KhWqfQzfE82E0j5P8LQL/UQc3MACurGp8bab4EjDHGdxANTVZWFgwMDJCZmQl9/dLvOkIIqV8BAdwlE1mLBcCNWNmyBfD0LFs/P5+baOzatddbTAwQZfMW2sX9Xe7zSAUCCEt8JJS+X8ZbbwHnz1fnlEgTVpXPUGoJIYQ0aRLGcCkjAwmFhbDQ0EA/Q0OIBAK+w1KZrNNo6VwgPp4rP3r0dSKSlcXlBbduAUVFZY912mYW2r0MBzIylD5X6YSjwgTE0BD46CPVT4QQJaglRAlqCSGkaQhIScH86Gg8KyiQl1mLxdji4ABPU9Nae57CYoadlzPwKL0QbVpoYHZfQ2io1TzRkUi4IbIlW0BK09bmkg+RiEtULC2BxESgZUvAzY3bXF2Bnj25vAHJycCsWcDx42ACAQRV+QgQCLgnGTWKm8+9ZcsaniFpiqryGUpJiBKUhBDS+AWkpGDM3bso/Q9Olhoc7dSpVhKRT/9IwebCaEiMXyc6ojQxvDUc8I1H1Y+fl8f132jVCggOBgYOfPWAkAFOGYBxIZCmAdwxBKTc2fz99+t6ly9zl2psbbmcQSZHIsHt7GzczM7GjexsmBw/jiVffgm93FyoSaWVxiUViSDU0wN27wbGjq3yeZHmgy7HEEKaNQljmB8dXSYBAQAGLhFZEB2NESYmNbo08+kfKdioe7fs87cowEbBXeCPTuUmIufPA/fuAU+fKm4pKVwSERfHdUIFAPRLAeZGAy1fJzpIFgPbHYBLpq/rAejbF0gpLMT5dC7ZuPEq8XiQmwuFVON//8MP+/bhl6+/xjvXrqGi3wIDkD5wIIz376fWD1KrqpWEZGZmQiKRwMjISKH8xYsXUFNTo9YDQohKarO/hpQxxObnIzI3F3+kpSlcgimNAYgrKIDOP/9AT00NWkIhtIVCaItE5f7UKnVfgwnwLR5zBywdshCAFPi2IBoZs43xPFaIggLg3LnXVdatK3/Ya3Y2UFzMjYJBvxRgbdlEByYFXPl3bRFtqYGVMa9aOV6+RHxhodLjmmtooKuuLrrq6sLl1U/7sDBI/vsPahVMBCIRCqHt6koJCKl11bocM2TIEHh4eGD27NkK5X5+fvj9999x+vTpKh1vx44d2LhxIxITE+Hs7Ixt27bB1dVVad0BAwbg4sWLZcqHDh2KwMBAAEB2djaWLVuGEydOIC0tDfb29vjkk08wa9YsleKhyzGE1L3q9tcolEoRnZeHyNxcRObkcD9zcxGVm4tcFS4r1DsJgAIRkC+EvYUI2iIuiUmOEyI/QwR9DSEMNUUw1hHBRF8IMwMRjHSE0HmV6My5FQOmU1w20QFeN+so0VZLSyHZcNHVhblYXLaiiwvYrVsVtoRIAUS2bYvs//6DG/1PJJWo8z4hRkZGuHLlCjp06KBQfv/+ffTp0wdpaWkqH+vQoUOYMmUK/Pz84ObmBl9fXxw5cgRRUVFoqSTrfvHiBQpLZPlpaWlwdnbGDz/8gKlTpwIAZs6cib///hs//PAD7Ozs8Ndff2H27NkICAjA8OHDK42JkhBC6pYq/TUGGRnh/qtE496rRCMyJweP8vNRXM6/LXWBAO20tGCsro5/MjMrjeNAhw5w1tVFrkSCXKlU6c/sYiniU6WITZbgeZoUyZkSpOVIUWCcC7TNqdkvopY4aGriDUNDebLhrKsLPTUVGroTE181t7wmG5arbHiuVUAA5nfvjsU2NhA2ohFGpH7VeZ+QgoICFBcXlykvKipCXl5elY61efNmfPjhh5g2bRoArjUlMDAQe/fuxbJly8rUL30J6ODBg9DW1sZ7770nL/v333/h5eWFAQMGAOCSkt27dyM0NFSlJIQQUncq668BAGPv3kVFs4TrikTooK39etPRQQdtbbTW1ISaUAgJY7C7ehXP8gvKbUGw0RRjbMuWCpd/cnKAO3e40STCV/NJT54M7N+v5BjO6YDvrUrPdw06YpKrbpnkJqecpCenxO2HeXkIz86u9DnW2dtjgplZpfXKOKs4TTsTiSDR1cXdGTPQ3t8fguxsCEpcpnkrNBRLW7RAUHo69nXoADMNjao/JyElVCsJcXV1xZ49e7Bt2zaFcj8/P3Tv3l3l4xQWFiIsLAzLly+XlwmFQri7uyMkJESlY/j7+2P8+PHQ0dGRl/Xu3Ru///47pk+fDktLSwQHB+PBgwf47rvvlB6joKAABSWahLOyslQ+B0KIahhjeFZQgH2JiRX21wAgT0BaqqsrJBmyzUoshqCCb+IigQATUh24TqNSKC5QIQUgAN596oC/MgS4dQu4eZPbHjzgRqA+fAg4OHDVO3cGtLSALl0AF5fXm2MHQ5ieF0PSokD5AhhSQJQuxvIRptUerhucno6BtypPdCyqmwycPs1lW4wBjEEwfDjU/fzg1LIlsHSpfCgvBAIwgQCrIyNxdMgQ/JWeDufr1/FLhw54u9QXQ0KqolpJyBdffAF3d3fcunULb731FgAgKCgI169fx19//aXycVJTUyGRSGBWKoM3MzPD/fv3K90/NDQUERER8Pf3Vyjftm0bZs6cCWtra6ipqUEoFOL777/HG2+8ofQ4Pj4+WLt2rcpxE9Ic1KTTaLFUiqi8PHlHyZuvRmikKWlBLc/udu0w09KyerFLgN9mmwL2ncqOKkkRAzscsOuSKXYp2dfCghuVIktC5s8HlixRNkW6AN4aDtwomHISHW8NhxrNF9LP0BDWYjHiCwqUthwJwPWj6WdoWPWDFxcDZ85wi8gYGpYdetuyJTdT2uHDwEcfQZCRgTYXLuC/ffswLioKETk5GHT7Npa2aoV1dnZQF9JSZKTqqpWE9OnTByEhIdi4cSMOHz4MLS0tdOnSBf7+/mjbtm1tx1guf39/ODk5lenEum3bNly9ehW///47bG1t8c8//2DOnDmwtLSEu7t7meMsX74c3t7e8vtZWVmwsbGp8/gJaaiq0mm09PwTN7OzcScnB/lKOomqCQRoJRbjcX5+pTG009KqctzFxcD9+8Cvv76a4OuZKXDFpNz5NVq1Avr0ed264ezMLYdSUkVhfONhCvzRqew8IenVnyekJJFAgC0ODhhz9y4EgEIiIkttfB0cqjeiKC8PaN0asLeveOKxsWOBAQO4VpEnT9BRIEBot27wfvQIfs+f46vYWARnZOC3Dh1gV43XjDRvvE5WVlhYCG1tbRw9ehQjR46Ul3t5eSEjIwMnT54sd9+cnBxYWlpi3bp1mD9/vrw8Ly8PBgYGOH78OIYNGyYv/+CDD/Ds2TOcOXOm0rioYyppzirrNPq5rS10RCJ5K8eDvDyl39J1RSI46+igq56efIRGR21tqAuFsLt6tdJv9zH/+1+FH66FhVzSoa3N3f/9d2DcOG7dFFUdOABMmKB6/XJjqaMZU2WUJYU2YjF8azrzq0RStVXwStU/lpKCGffvI1MigYFIhB8cHTGGhvE2e3XeMTU2NrbCx1u1aqXScTQ0NNC9e3cEBQXJkxCpVIqgoCDMnTu3wn2PHDmCgoICTJ48WaG8qKgIRUVFEJZqGhSJRJA2xOF7hDQgqnQaXf/0aZnHZPNPlBwO2kZLq9wRFFscHDA64i530FKXMZig7Lf7ggKuw2h4OBAWxv28fRvYvBmYM4erY2vLJSC6utyX+zt3Kj/fUgNDqk1DTYAFA1rUzsGU8DQ1xQgTk9pfA6cqCYiS+qNNTdFdVxcTIyMRkpWF9+7dw0fp6fjOwQFaVT02aZaqlYTY2dlV2ClMUsGkN6V5e3vDy8sLPXr0gKurK3x9fZGTkyMfLTNlyhRYWVnBx8dHYT9/f3+MHDkSxsbGCuX6+vro378/lixZAi0tLdja2uLixYvYt28fNm/eXIWzJKThqs1JviSM4WFuLsKzs3EyJaXSTqMAMMDAAIOMjOBS0fwTFblkCmzpBMwp1V8jleuvgfmmgCfXQXTsWCAigmv1KC0i4vXtTp24SzFt23L9LO3suEXelLX1CgTcrKT9+lUtbD6JBAIMaFF3iU512Wlp4aKLC1Y/eYKvYmOxOyEBV7KycLBjR3QqMWCAEGWqlYTcuHFD4X5RURFu3LiBzZs348svv6zSscaNG4eUlBSsWrUKiYmJcHFxwZkzZ+SdVWNjY8u0akRFReHy5cvldoI9ePAgli9fjkmTJuHFixewtbXFl19+qfJkZYQ0ZDVZlK1IKsW93FyEv3yJ8BKdRnOq2Eo409KyekNCwbXoz58Prr/GZeX9NRbEACNGcP0zbt7k9jMyArp3B7p1e/2zdevXx1VTAxwdX9/fsoVbZVa25pqMLFfz9a16QwBRTl0oxIbWrfGmoSEmR0YiIicHPcPCsNXBATMsLCr80kqat1rtExIYGIiNGzciODi4tg7JC+oTQhqqqizKlieR4E5ODsJfvsSN7GyEZ2fjdnY2CpW85bWEQrjo6sJMXR0nVJhs8IKzc7W/lZ8/D7z9duX1Llzg+kOeOQN06MB1Iq3qZ1lAAJfwlFyF1saGS0A8Pat2LKKapMJCeEVG4mx6OgBgrKkp9jg6wuDV5Gm12YpHGibeFrBzdHTE9evXa/OQhJBXVOmvMSMqCidSUnAzJwf3cnKUTvilLxKhm54euunqopueHrrq6sJRWxsigUA+yVdtDgktLARiYl63UiQlqbafbFG2wYNVfqoyPD25FpVLl7jjWVhwl2CoBaTumGlo4HSXLtgUF4cVMTE4nJKC6y9f4reOHRFfUFDtVjzSNFUrCSk9mRdjDAkJCVizZk29DtElpDm5lJFRaX+NjOJi/JKcLL9voq6O7iWSjW56erDX1Cy3w2htDQmNjuYm4zx7lmvR0NPj+mcIBICVlQoni9rrNCoScS0qpP4IBQIsadUKbxgaYvy9e4jJz0fv8HAou+gXX1CAMXfvKrTikeajWkmIoaFhmWt8jDHY2Njg4MGDtRIYIURReSujluZpYgIvc3N009WtdGZRpfubmuJop06Y/zAazwpfJz1WlXxjDQ7m5rU6exZ4/FjxMW1tLgmRdQa1tm5anUaJcm76+rjZowc+uH8fR1NTldaRrcG3IDoaI0xM6NJMM1OtJOTChQsK94VCIUxNTeHg4AA1VRZNIoSohDGGW9nZ+DUpCT8mJqq0zzwrq5qPorhkCrbABDDKkHcaZS8MAV8B4MlNsnnzJtCxI6Cpye1y6hSw69UUpGpq3CRggwcDgwZxk4DJ+peLRNRptDkxUFPDbCurcpMQgEtE4goKcCkjo0GOACJ1p1oZQ//+/QEA9+7dQ2xsLAoLC5Geno4HDx4AAC0SR0gNxebn40BSEn5NSsLd3Fx5eelLJCXVaArvEgICuASBMQEQ9/oD4bkAGD2aa6G4fx9ISeE6jQ4axD3u6clNwjloEDBwIHcJpjyensDRo2U7jVpbU6fRpihRxVa8BBXrkaajWknI48eP4enpidu3b0MgEEA2wEbW7FuVeUIIIZyMoiIcTUnBr0lJuFhiGXqxQAAPExNMNjNDvlSKCffuAajlKbxfkQ2fVXaZRFZ26RL3U1dXMYHo3ZvbVEWdRpsPVRfYq/ZCfKTRqlYSMn/+fNjZ2eH8+fOwt7fHtWvX8OLFCyxatAjffvttbcdISKOj6jDEAqkUp9PS8GtSEk6lpcmHzwoA9Dc0xGQzM4w2MYGhurp8H3WBQOkIgxpP4Q2uX0fJxKI8330HzJ4N1PQzgzqNNg+VLcQHAC3U1NDXwKBe4yL8q1YSEhISgr///hsmJiYQCoUQiUTo27cvfHx88Mknn5SZzIyQ5qSyycSkjOFKZiZ+TUrC4ZQUZJSYCrSzjg7eNzPDhJYtYSPrbFFKbU/hXVzMJR/HjnFrqajCzKzmCQhpPioadSWTXlyM9+7dww+OjjAukXSTpq1aSYhEIoHeqwu+JiYmeP78ORwdHWFra4uoqKhaDZCQxqS8ycRkwxBHmZgg7OVLPC2RoFhqaGCSmRkmm5mhi66uSs9T0ym8CwuBoCCuX8bJk4AK85MpqK3hs6T5kI+6UrIQ35uGhjiQnIwTqakIzcrCLx064E3qoNosVCsJ6dy5M27dugV7e3u4ubnhm2++gYaGBvbs2YPWJedRJqQZUWUysYBXIwT0RCKMMTXFZDMz9Odhxsi33gIuX35938QEGDUKGDkSmDkTeP6chs+S2ldRK958a2tMuHcPUXl5cL91C0tsbLDe3h4apZbtIE1LtZKQzz//HDk5OQCAdevW4d1330W/fv1gbGyMQ4cO1WqAhDQWqkwmBgCrbG2xrFWrGq0yKpGo1qEzJwf480/gxAlg925Atp7Y229zE4qNHv16xItsdP3WrTR8ltSd8lrxuurpIaxHD3hHR2NPQgK+iYtDUHo6DnTsiHba2jxESupDra0d8+LFC7Ro0aJJLFREa8eQ6jiQlIRJkZGV1+vQodqLvwHK10Oxtubm3vD0BLKyuDk7jh3jEpC8PK7OkSNccgEAubnc/B7lfcmkNVcIn46npOCDqCi8KC6GtlCIbW3bYpq5eZP4fGkOqvIZWqsL2DUVlISQqmCM4fSLF1gcHY37sk/8CtRk8bfXc3golsv+N3fvDty+zfX5kLG35/aZPh1o317151K1tYWQuhBfUID3IyNxISMDADDG1BR72rVDC+q02uBRElJDlIQQVTDG8EdaGtY9eYKw7OxK68smE4v53/+q1QdEIgHs7MofQlvy8omjI5d4jB4NuLhUffVZQhoCCWP4Ni4On8fEoJgxWIvF+LVDB/Sv4YR8pG7xtoouIc2BlDGcTE3FuqdPcfNV8qEtFGKOlRU6aGtjxqsRYrU9mdilSxXP4SFLQPbuBaZOpcSDNH4igQBLW7XCW4aGmBgZiYd5eRh48yaWt2qFNXZ2UKdOq40eJSGEqEjKGAJSUrD+6VPcftUxW1ckwlwrK3hbW8P01cQZBmpqdTKZ2L//qlZPU5MSENK09NDXR3j37pgfHY29iYnYEBuL8686rbbR0uI7PFIDdDlGCbocQ0qSMIajKSlY/+SJfB0XPZEIn1hZYaGNjdKJlVSdMVUVFy4A69ZxE4qpWp9mISVN1dHkZHz44AEyiouhKxJhe9u2mGJmRp1WGxC6HENILZAwhkPJyfji6VNEvko+DEQizLe2xgJr6wo7yNV0MrGSIiK4BEQkAsRibmSLMjSHB2kOxrRsCTd9fbwfGYmLmZmYev8+/kxLg1+7djBUV6/VLwCk7lESQkgpxVIpfnuVfDx4NdrFUE0NC62t8YmVlcI6LrUtORnYsQNwcno9nHbaNCAxEZg1C7h+/XU5zeFBmisbTU0Eubjg69hYrIqJwaGUFIRkZWGmhQX8EhLKXTKBNDx0OUYJuhzTtJX3TalIKsX+pCR8GRuL6FfJh5GaGrxtbDDPygr6anWXs0dGAps3A7/8AhQUcEnIrVvK+3bQHB6EvBaalYWJ9+7hUX6+0sdlb6GjnTpRIlJPaIhuDVES0nQpW1zOSkMDw01McPbFCzx+9Y/MWE0Ni21sMMfKCnp1lHwwxvXf2LQJOH36dXnPnsCiRcB775U/mRjN4UHIaxlFRbAKCUGuVKr08ZoOjydVQ31CCFGi3MXlCgux6/lzAICpujqW2NjgY0tL6NYg+VAlSZgzB9i1i7stEAAjRnDJR58+lY9uEYmo8ykhMjezs8tNQABuuHxcQQEuZWTUWl8tUjsoCSHNQkWLy8kYikSIdnOr8WWX8qZV37AB8PAAZPMsDR8O/PQT1+djwQKgbdsaPS0hzVZCySmCa6EeqT+UhJBmQZXF5TIkEoS/fFmjb0rlTav+7BkwZQo3g+nRo1zZoEFcuZFRtZ+OEALA4tUcPbVVj9Qfmm6ONGmMMQSlp2PRo0cq1a/JNyWJhGsBqaiX1alTQHExd1sgoASEkNrQz9AQ1mIxKrqKqS4QoJWmZr3FRFTTIJKQHTt2wM7ODpqamnBzc0NoaGi5dQcMGACBQFBmGzZsmEK9yMhIDB8+HAYGBtDR0UHPnj0RGxtb16dCGojs4mLsio9H5+vX4X7rFsJVWNsFqNk3pcqmVQe4kS+XL1f7KQghSogEAmxxcACAchORIsbgFh6Of14tiEcaBt6TkEOHDsHb2xurV69GeHg4nJ2dMWjQICQnJyutHxAQgISEBPkWEREBkUiE9957T17n0aNH6Nu3L9q3b4/g4GDcvn0bK1euhCZlwU3eo7w8eEdHwzokBLMfPsS93FzoikT42MIC5urq5f6DEgCwEYvRrwYLYyUk1G49QojqPE1NcbRTJ1iJxQrlNmIxdrdrh+66ukgtKsJbt25h96uO6IR/vA/RdXNzQ8+ePbF9+3YAgFQqhY2NDebNm4dly5ZVur+vry9WrVqFhIQE6OjoAADGjx8PdXV1/PLLL9WKiYboNi5SxnA+PR1bnz3D6Rcv5J1PHbS0MM/KCl7m5jBQU5OPjgGULy5X3XkEwsMBPT0gPh4YOLDy+jStOiF1p7x5gHIlEsyIisLBV19wP7a0xBYHB1oErw5U5TOU199+YWEhwsLC4O7uLi8TCoVwd3dHSEiISsfw9/fH+PHj5QmIVCpFYGAg2rVrh0GDBqFly5Zwc3PDiRMnyj1GQUEBsrKyFDbS8L0sLsb2Z8/QMTQUg27fRuCrBGSIkRFOOzkhytUVn1hbw+DVaJfyvilZi8XVSkBu3gRGjgS6dwdWrOCG4Vpblz+8ViDgJhWjadUJqTuyJRMmmJlhQIsW8nlBtEUiHOjQAT729hAA2PX8Od65fRupNGKGV7wmIampqZBIJDAzM1MoNzMzQ2JiYqX7h4aGIiIiAh988IG8LDk5GdnZ2fjqq68wePBg/PXXXxg1ahQ8PT1x8eJFpcfx8fGBgYGBfLOxsanZiZFqkzCG4PR0/JaUhOD0dEiUNNQ9yM3FJw8fwiokBPOioxGVlydfUC7K1RWnu3TBEGNjCJVkA56mpnjyv//hgrMzDnTogAvOzoj53/+qlIBERHAjYLp2BU6e5JIL2UKeW7ZwP0s/NU2rTgj/BAIBltna4vfOnaEnEiE4IwM9w8NxW8U+Y6T2Neohuv7+/nBycoKrq6u8TPpqwpoRI0Zg4cKFAAAXFxf8+++/8PPzQ//+/cscZ/ny5fD29pbfz8rKokSEB8pmM5Wt+zDSxARnXrzAtvh4nHnxQv54e21tzLWywhQzM5VnNq3u4nKRkcDatcDhw9wIGIEAGDcOWLUK6NCBq+PpyQ3BVTZPCE2rTkjD8K6JCa5264YRERGIzstD7/Bw7OvQgaZ15wGvSYiJiQlEIhGSkpIUypOSkmBubl7hvjk5OTh48CDWrVtX5phqamro2LGjQnmHDh1wuZxhCWKxGOJSTfSkfpU7m2lBAUbfvQtzDQ0kvmo2FQAYZmyMT6ys4N6iRb0t4R0YCBw6xN1+7z1g9WqgU6ey9Tw9udlPaVp1Qhqujjo6uNatG8bdu4fz6ekYffcu1tjZYaWtrdJWVFI3eL0co6Ghge7duyMoKEheJpVKERQUhF69elW475EjR1BQUIDJkyeXOWbPnj0RFRWlUP7gwQPY2trWXvCk1lQ0m6msLLGwEPpCIbytrfHQzQ1/ODnhbSOjOk1AoqOBkqPFZ88GvLy4heUOH1aegMjIplWfMIH7SQkIIQ2Pkbo6/nRywgJrawDAmidP8N7du8iWTeZD6hzvl2O8vb3h5eWFHj16wNXVFb6+vsjJycG0adMAAFOmTIGVlRV8fHwU9vP398fIkSNhbGxc5phLlizBuHHj8MYbb2DgwIE4c+YM/vjjDwQHB9fHKZEqClZhNlMAONixI4aYmNT4+Spb1+XxY+CLL4B9+4COHbkOqEIhoK3NTbNOCGk61IRCfOfggC46Opj14AECUlMRfeMGTnbuDDtZZy9SZ3hPQsaNG4eUlBSsWrUKiYmJcHFxwZkzZ+SdVWNjYyEsNYQqKioKly9fxl9//aX0mKNGjYKfnx98fHzwySefwNHREceOHUPfvn3r/HyasvKGvlVFoVSKezk5uJGdjfDsbIS/fIn/Xr5Uad8MiaQ6YSsob12XLVu4US5ffMElGrIvQq1aARkZNLMpIU3dNAsLtNfWxqiICNzOyUHP8HAc7dQJ/WswdxCpHO/zhDRENE9IWRV1Gi2vM1eeRII7OTkIf/lSnnDcyclBYTX/5C44O9fJui4CAVcmEnGtJAC3rsvatYCbW7WfjhDSCD3Lz8fIiAiEZWdDTSDAVgcHfGxlxXdYjUpVPkMpCVGCkhBF5XUaLTnJ19stWuBmdjbXwvEq6biXkwNlbRcGIhG66emhm64uuunpwVlHB4Nv30Z8YaHSfiECcAlPzP/+V+WWFxmJBLCzq3xa9bfeAtatA3r3rtbTEEKagLxXE5v99mpis1mvJjbTqMeJzWqj5ZkvlITUECUhr0kYg93VqxX22VADUF43LlN1dXTX00NXXV150mGvqVmmQ2ldzWYqExxMs5kSQlTHGMM3cXFY/vgxGIA3DAxwtFMnmGpo1HmCUJ2W54akKp+hvPcJIQ2XlDEcTEqqtNOoLAGxFovliYbsp6WGhkojWGSzmSp74/nWwhvvzh3V6tG6LoQQgJvYbGmrVuikrY2JkZH4JzMTPcPC8Im1Nb579qzOEoSKpisYc/dujb+QNTTUEqJEY2wJqWlmniORICInBzezs3Hr1XY7JwfZKnYG3dW2LWbVwnXT2vyGkZ7OTRz266/AP/+otg+1hBBCSovMycHwVxObKVNbLbaVtTzXxqXp+kAtIc1MVZruGGOILyjArZwc3MrOlicdD/PylPbHUAdQpEIM7bW1a3QOMtWdzVSZ774D1q9/fV8sBspr1BEIuFEytK4LIaS0Djo6COnaFdYhIShQ8r2dgUsQFkRHY4SJidIEQcIYciUS5EqlyJVIkFPy9qufYS9fVtjyzADEFRTgUkZGrf2f5BslIY1cZU13G1u3homGhkLC8aKciXjM1NXhoqsL5xKbg6YmHEJDEV9QUGGn0X48DmOTSrl5P379FRg9Ghg8mCufNAk4cYL7OWEC8N9/3OgYQHGEDK3rQgipTEROjtIEREaWIDhfvw51obBMolHRvlWV0IQW3aMkpBFTZabRxY8fl3lMBK7lwllXVyHpMNPQUPo8WxwcMObuXQigvNOor4NDrTUNVjaRWEkREVziceAAEBfHlWVkvE5CHB2B27df12/VitZ1IYRUj6of/Hdzcyt8XABAWyiEtkgEbaEQOiKR/Ha+VIpQFeZNsijnf3VjRElII1QsleJubi72JSaqNNOos44O+hsaypOOjtra0KzCV/667jQqU9FEYrIEQSIBNm/mko+SCYaBAdfKMWVKJedC67oQQqpB1Q/+dXZ2cNXXf51gvEo4ZLc1hcJyO+vL+oQ05Jbn2kYdU5Wo7Y6pNelsyRhDXEEBrmVlIfTlS1zLykLYy5fIfbVasCoOdOiACa9moK2JuhyWVtFEYgDXgiFLRLp1A27cANTVgXff5S63DBsGaGrWSiiEEFKGqglCTTuNljddgew5GsPoGOqY2oBUdbx3ZnExrpdIOK5lZSGpqGzXUH2RCA6amgjPyak0hlprupMKgFstgAQAFgD6gbu2U0MSCdcCoiwdlpV98gnXgiESAcuXcyNfxoyh6dQJIfVDJBDUy6Xp8lqeAcDTxKTBJyBVRS0hStRWS0hlM40e7NgRbbS0EPoq2bj28iXuK7meqCYQoIuODtz09eGqpwc3fX04amuDAfWSmQOqXSqpLppIjBDSWCj7YmlTy5emAcWW5+i8PKx68gRaQiEeubnBQiyuteepC9QS0gCo0ml03L17Sve119RUSDi66upCq5xOC/WRmZd3qSQ+nisveamkpLw8bkXauLjX27Nnr29/9RUwcqTqE4TRRGKEEL55mppihIlJnU+pXnK6AsYY/nzxAiFZWVjz5Al2OzrW6nPxiZKQOnJJxeXpdYVC9DEw4JKOV4mHaRUun9R1p1FVLpV8+CHXSTQ+Hnj/feCNN7jys2eBUaPKP7Zs4I6FhWqxqFqPEELqUm3OZ6QKgUCAb1q3Rr+bN+GfkICF1tZor6NTb89flygJqSOqDufa7eiIiTXsNFqXmfmlS5Uv+vbiBbfiLAC0b/86CbGxAVq04C7b2NiU3Tp14ur168fViY9XnuzQRGKEkOaur6Ehhhsb4/e0NCyPicHxzp35DqlWUBJSR1TtDGpZS51GazMzz80F/v2X66tx5Ihq+wwcyPXX6Nv3dVm3blyCUhmRiOtbMmYMl3DQRGKEEFKWT+vWOJWWhhOpqbiSmYk+BgZ8h1Rj1DFVidromFpfw7nkz1eFSb5KKyzk1lYJDua20FBAyYCcCtVGp1FlnV9tbGgiMUIIkfkwKgo/JCSgt74+LnftqtICofWtKp+hlIQoUdujY4C6WZ5e/jxVHLmSmwskJgKtW3P3MzO5oa4lpx6xtuZaN954A1i5EkhKqvhSSUxM7bRU1CSZIoSQpu55QQEcrl1DnlSK4506YWQDHLJLSUgN1eZkZXU9nEuVSb6GDHl9eSU4GLh2DejRgyuTGTUK0NPjWjMGDADs7V8fQ/YcgPJLJeWNjiGEEFL7Pnv8GBtiY9FeWxt3evSAmlDId0gKKAmpoYY0Y2qFx5UAdnYVdxwVi7kWjtKXV+ztgQcPADUVewXRpRJCCGkYMouL0ebqVaQVF2NPu3b40NKS75AUUBJSQ7WdhNQVVSf5AgArq9edRwcM4C7FVDUPokslhBDSMPjGxWHho0ew0NDAQzc36DSgf8Y0WVkzoerkXd99x7Vi1LTxRSSiGUsJIaQh+NjKClvi4/EkPx++z57hM1tbvkOqloZ1IYlUiaqTd7m41DwBIYQQ0nCIhUJ8aW8PAPg6NhYpKs5N1dBQEtKIySb5Ki/BEAi4fhs0yRchhDQ941u2RDddXbyUSPDF06d8h1MtlIQ0YrJJvpShSb4IIaRpEwoE+PrVXAu7nj/H47w8niOqugaRhOzYsQN2dnbQ1NSEm5sbQkNDy607YMAACASCMtuwYcOU1p81axYEAgF8fX3rKHp+eXoCGzeWLbe2pqGzhBDS1LkbGeGdFi1QxBg+i4nhO5wq4z0JOXToELy9vbF69WqEh4fD2dkZgwYNQnJystL6AQEBSEhIkG8REREQiUR47733ytQ9fvw4rl69CssGNnyptmVncz/79AEOHOBmL42JoQSEEEKag69bt4YAwMHkZPyXlcV3OFXCexKyefNmfPjhh5g2bRo6duwIPz8/aGtrY+/evUrrGxkZwdzcXL6dO3cO2traZZKQ+Ph4zJs3D/v374e6unp9nApvfv+d+zljBjBhAjeChS7BEEJI8+Cip4dJrxZCXfr4MRrTzBu8JiGFhYUICwuDu7u7vEwoFMLd3R0hISEqHcPf3x/jx4+HTolljaVSKd5//30sWbIEnWRLtVagoKAAWVlZCltjUVjIdT7V1QXKuSJFCCGkiVtvZwcNgQB/Z2Tgr/R0vsNRGa9JSGpqKiQSCcxKLWVvZmaGxMTESvcPDQ1FREQEPvjgA4Xyr7/+Gmpqavjkk09UisPHxwcGBgbyzcbGRvWT4JmGBnDiBJCWBrRsyXc0hBBC+GCnpYU5VlYAgKWPHkHaSFpDeL8cUxP+/v5wcnKCq6urvCwsLAxbtmzBTz/9pPLqgsuXL0dmZqZ8i4uLq6uQ64yGBt8REEII4dNntrYwEIlwKycH+5OS+A5HJbwmISYmJhCJREgq9ctKSkqCubl5hfvm5OTg4MGDmDFjhkL5pUuXkJycjFatWkFNTQ1qamp4+vQpFi1aBDs7O6XHEovF0NfXV9gag9xc4PFjvqMghBDSEBirq2NZq1YAgM9jYpAvkfAcUeV4TUI0NDTQvXt3BAUFycukUimCgoLQq1evCvc9cuQICgoKMHnyZIXy999/H7dv38bNmzflm6WlJZYsWYKzZ8/WyXnw5fRpoE0bYPhwviMhhBDSEMy3toaVhgZiCwqw4/lzvsOpFO9rx3h7e8PLyws9evSAq6srfH19kZOTg2nTpgEApkyZAisrK/j4+Cjs5+/vj5EjR8LY2Fih3NjYuEyZuro6zM3N4ejoWLcnU89ko2LateM3DkIIIQ2DlkiEdfb2mBEVhS+fPsV0c3O0aMAjRHlPQsaNG4eUlBSsWrUKiYmJcHFxwZkzZ+SdVWNjYyEUKjbYREVF4fLly/jrr7/4CLlBKC4GAgO52yNG8BsLIYSQhsPL3Byb4+JwNzcXX8XG4us2bfgOqVwC1pgGFNeTqixDzJeLF7n5QIyNgcREQI33dJIQQkhDcSo1FR4RERALBHjo5gYbTc16e+6qfIY26tExzZnsUsywYZSAEEIIUTTM2BhvGBiggDGsfvKE73DKRUlII8QYcPIkd5s6pRJCCClNUGJxu58TExEhW9+jgaEkpBGKjAQePeLmBhk0iO9oCCGENET/MzDAaBMTSAEsa6DzOVAS0gjZ2QEBAYCPDzddOyGEEKLMhtatIQIQ+OIFLmZk8B1OGZSENELa2sCoUYC3N9+REEIIacjaaWtj5quV5D999KjBLW5HSQghhBDShK22s4OOUIjQly9xNCWF73AUUBLSyPz+O7B6NXD3Lt+REEIIaQzMNDSw+NXCrCtiYlAklfIc0WuUhDQy/v7AunXA8eN8R0IIIaSxWGRjg5bq6ojOy8P3CQl8hyNHSUgjkpsLnDvH3aahuYQQQlSlp6aG1a8WcV375AleFhfzG9ArlIQ0IufPA3l5gK0t4OTEdzSEEEIakw8tLOCgpYXkoiJsiovjOxwAlIQ0KrJZUocPBwQCfmMhhBDSuKgLhdhgbw8A+DYuDokFBTxHRElIoyGRAH/8wd2mBesIIYRUxxhTU7jq6SFHKsWaJ08QnJ6O35KSEJyeDgkPw3dpATslGuICdiEhQO/egIEBkJICNOCVmQkhhDRgFzMyMODmzTLl1mIxtjg4wNPUtEbHpwXsmqDYWMDQEBgyhBIQQggh1ZdWVKS0PL6gAGPu3kVAPc4lQklIIzFuHJCcDGzdynckhBBCGisJY5gfHa30MdllkQXR0fV2aYaSkEZEXR2oYSsZIYSQZuxSRgaeVdAhlQGIKyjApXpaZ4aSkEYgPR2gnjuEEEJqKqGwsFbr1RQlIY3AyJGAvT1w8SLfkRBCCGnMLDQ0arVeTanVy7OQaktLAy5fBqRSbpIyQgghpLr6GRrCWixGfEEBlDWwC8CNkulnaFgv8VBLSAN3+jSXgHTpAryacZcQQgipFpFAgC0ODgC4hKMk2X1fBweI6mlGTEpCGriTJ7mftFYMIYSQ2uBpaoqjnTrBSixWKLcWi3G0U6cazxNSFXQ5pgHLzwfOnOFuUxJCCCGktniammKEiQkuZWQgobAQFhoa6GdoWG8tIDKUhDRgwcFATg5gaQl07853NIQQQpoSkUCAAS1a8BoDXY5pwGQL1nl4AEJ6pQghhDQx1BLSgE2cCIhEwJgxfEdCCCGE1L4G8f16x44dsLOzg6amJtzc3BAaGlpu3QEDBkAgEJTZhg0bBgAoKirC0qVL4eTkBB0dHVhaWmLKlCl4/vx5fZ1OrenbF9i2Dejfn+9ICCGEkNrHexJy6NAheHt7Y/Xq1QgPD4ezszMGDRqE5ORkpfUDAgKQkJAg3yIiIiASifDee+8BAHJzcxEeHo6VK1ciPDwcAQEBiIqKwnDq2UkIIYQ0KALG+J0Q3M3NDT179sT27dsBAFKpFDY2Npg3bx6WLVtW6f6+vr5YtWoVEhISoKOjo7TO9evX4erqiqdPn6JVq1aVHrMqyxDXlY0bAVdXoE8fQI0umhFCCGkkqvIZyuvHW2FhIcLCwrB8+XJ5mVAohLu7O0JCQlQ6hr+/P8aPH19uAgIAmZmZEAgEMCxnBriCggIUlFjQJysrS7UTqCOxscCnn3KdUZOSABMTXsMhhBBC6gSvl2NSU1MhkUhgZmamUG5mZobExMRK9w8NDUVERAQ++OCDcuvk5+dj6dKlmDBhQrkZmY+PDwwMDOSbjY1N1U6klv3xB/ezTx9KQAghhDRdvPcJqQl/f384OTnB1dVV6eNFRUUYO3YsGGPYtWtXucdZvnw5MjMz5VtcXFxdhawSmiWVEEJIc8Dr5RgTExOIRCIkJSUplCclJcHc3LzCfXNycnDw4EGsW7dO6eOyBOTp06f4+++/K7wuJRaLIS41fS1fMjO5ScoASkIIIYQ0bby2hGhoaKB79+4ICgqSl0mlUgQFBaFXr14V7nvkyBEUFBRg8uTJZR6TJSAPHz7E+fPnYWxsXOux15WzZ4GiIsDREWjXju9oCCGEkLrD+7gLb29veHl5oUePHnB1dYWvry9ycnIwbdo0AMCUKVNgZWUFHx8fhf38/f0xcuTIMglGUVERxowZg/DwcJw6dQoSiUTev8TIyAgaGhr1c2LVJJsldcQIfuMghBBC6hrvSci4ceOQkpKCVatWITExES4uLjhz5oy8s2psbCyEpeYsj4qKwuXLl/HXX3+VOV58fDx+f/VJ7uLiovDYhQsXMGDAgDo5j9rAGHDzJnebLsUQQghp6nifJ6Qh4nOeEKkUCAsDunXjpmwnhBBCGpNGM08IKUsoBHr25DsKQgghpO416iG6TQljgETCdxSEEEJI/aEkpIG4exewtATmzuU7EkIIIaR+UBLSQJw8CSQnA0+f8h0JIYQQUj8oCWkgZENzaVQMIYSQ5oKSkAYgIQEIDeVuv/suv7EQQggh9YWSkAbg1Cnup6srYGHBbyyEEEJIfaEkpAGQLVhHs6QSQghpTigJ4VlODnD+PHeb+oMQQghpTmiyMp7l5wMLFwI3bgCdOvEdDSGEEFJ/KAnhmbExUGptPkIIIaRZoMsxhBBCCOEFJSE8evAACAwE8vL4joQQQgipf5SE8Mjfn5sXZNYsviMhhBBC6h8lITySzZI6dCi/cRBCCCF8oCSEJw8eAPfvA+rqwODBfEdDCCGE1D9KQngiawUZMAAwMOA1FEIIIYQXlITwhBasI4QQ0txREsKD1FTgyhXutocHv7EQQgghfKEkhAcXLgBSKeDiAtja8h0NIYQQwg+aMZUHY8YAkZFciwghhBDSXFESwgOBAGjfnu8oCCGEEH7R5RhCCCGE8IKSkHq2ZAkwdixw7RrfkRBCCCH8oiSkHkmlwIEDwJEjQHo639EQQggh/GoQSciOHTtgZ2cHTU1NuLm5ITQ0tNy6AwYMgEAgKLMNGzZMXocxhlWrVsHCwgJaWlpwd3fHw4cP6+NUKhQWBjx/DujqAgMH8h0NIYQQwi/ek5BDhw7B29sbq1evRnh4OJydnTFo0CAkJycrrR8QEICEhAT5FhERAZFIhPfee09e55tvvsHWrVvh5+eHa9euQUdHB4MGDUJ+fn59nZYCiQQIDgY2bODuDxoEiMW8hEIIIYQ0HIxnrq6ubM6cOfL7EomEWVpaMh8fH5X2/+6775ienh7Lzs5mjDEmlUqZubk527hxo7xORkYGE4vF7LffflPpmJmZmQwAy8zMrMKZKHfsGGPW1owBrzcjI66cEEIIaWqq8hnKa0tIYWEhwsLC4O7uLi8TCoVwd3dHSEiISsfw9/fH+PHjoaOjAwCIiYlBYmKiwjENDAzg5uZW7jELCgqQlZWlsNWGgABuTpBnzxTL09O58oCAWnkaQgghpFHiNQlJTU2FRCKBmZmZQrmZmRkSExMr3T80NBQRERH44IMP5GWy/apyTB8fHxgYGMg3Gxubqp5KGRIJMH8+1/ZRmqxswQKuHiGEENIc8d4npCb8/f3h5OQEV1fXGh1n+fLlyMzMlG9xcXE1ju3SpbItICUxBsTFcfUIIYSQ5ojXJMTExAQikQhJSUkK5UlJSTA3N69w35ycHBw8eBAzZsxQKJftV5VjisVi6OvrK2w1lZBQu/UIIYSQpobXJERDQwPdu3dHUFCQvEwqlSIoKAi9evWqcN8jR46goKAAkydPVii3t7eHubm5wjGzsrJw7dq1So9ZmywsarceIYQQ0tTwvnaMt7c3vLy80KNHD7i6usLX1xc5OTmYNm0aAGDKlCmwsrKCj4+Pwn7+/v4YOXIkjI2NFcoFAgEWLFiAL774Am3btoW9vT1WrlwJS0tLjBw5sr5OC/36AdbWQHy88n4hAgH3eL9+9RYSIYQQ0qDwnoSMGzcOKSkpWLVqFRITE+Hi4oIzZ87IO5bGxsZCKFRssImKisLly5fx119/KT3mp59+ipycHMycORMZGRno27cvzpw5A01NzTo/HxmRCNiyhRsFIxAoJiICAffT15erRwghhDRHAsaUfU9v3rKysmBgYIDMzMwa9w8JCOBGyZTspGpjwyUgnp41i5MQQghpaKryGcp7S0hT5+kJjBjBjYJJSOD6gPTrRy0ghBBCCCUh9UAkAgYM4DsKQgghpGFp1POEEEIIIaTxoiSEEEIIIbygJIQQQgghvKAkhBBCCCG8oCSEEEIIIbygJIQQQgghvKAhukrI5m/LysriORJCCCGkcZF9dqoyFyolIUq8fPkSAGBjY8NzJIQQQkjj9PLlSxgYGFRYh6ZtV0IqleL58+fQ09ODQLbQSxOVlZUFGxsbxMXF1XiK+saEzpvOuzmg825e5w00jHNnjOHly5ewtLQss/ZbadQSooRQKIS1tTXfYdQrfX39ZvdmBei8mxs67+aluZ43wP+5V9YCIkMdUwkhhBDCC0pCCCGEEMILSkKaObFYjNWrV0MsFvMdSr2i86bzbg7ovJvXeQON79ypYyohhBBCeEEtIYQQQgjhBSUhhBBCCOEFJSGEEEII4QUlIYQQQgjhBSUhTZiPjw969uwJPT09tGzZEiNHjkRUVFSF+/z0008QCAQKm6amZj1FXDvWrFlT5hzat29f4T5HjhxB+/btoampCScnJ5w+fbqeoq09dnZ2Zc5bIBBgzpw5Sus31tf6n3/+gYeHBywtLSEQCHDixAmFxxljWLVqFSwsLKClpQV3d3c8fPiw0uPu2LEDdnZ20NTUhJubG0JDQ+voDKqnovMuKirC0qVL4eTkBB0dHVhaWmLKlCl4/vx5hcesznuFD5W95lOnTi1zHoMHD670uI35NQeg9P0uEAiwcePGco/Z0F5zSkKasIsXL2LOnDm4evUqzp07h6KiIrzzzjvIycmpcD99fX0kJCTIt6dPn9ZTxLWnU6dOCudw+fLlcuv++++/mDBhAmbMmIEbN25g5MiRGDlyJCIiIuox4pq7fv26wjmfO3cOAPDee++Vu09jfK1zcnLg7OyMHTt2KH38m2++wdatW+Hn54dr165BR0cHgwYNQn5+frnHPHToELy9vbF69WqEh4fD2dkZgwYNQnJycl2dRpVVdN65ubkIDw/HypUrER4ejoCAAERFRWH48OGVHrcq7xW+VPaaA8DgwYMVzuO3336r8JiN/TUHoHC+CQkJ2Lt3LwQCAUaPHl3hcRvUa85Is5GcnMwAsIsXL5Zb58cff2QGBgb1F1QdWL16NXN2dla5/tixY9mwYcMUytzc3NhHH31Uy5HVr/nz57M2bdowqVSq9PGm8FoDYMePH5ffl0qlzNzcnG3cuFFelpGRwcRiMfvtt9/KPY6rqyubM2eO/L5EImGWlpbMx8enTuKuqdLnrUxoaCgDwJ4+fVpunaq+VxoCZefu5eXFRowYUaXjNMXXfMSIEezNN9+ssE5De82pJaQZyczMBAAYGRlVWC87Oxu2trawsbHBiBEjcPfu3foIr1Y9fPgQlpaWaN26NSZNmoTY2Nhy64aEhMDd3V2hbNCgQQgJCanrMOtMYWEhfv31V0yfPr3CRRibwmtdUkxMDBITExVeTwMDA7i5uZX7ehYWFiIsLExhH6FQCHd390b9N5CZmQmBQABDQ8MK61XlvdKQBQcHo2XLlnB0dMTHH3+MtLS0cus2xdc8KSkJgYGBmDFjRqV1G9JrTklIMyGVSrFgwQL06dMHnTt3Lreeo6Mj9u7di5MnT+LXX3+FVCpF79698ezZs3qMtmbc3Nzw008/4cyZM9i1axdiYmLQr18/vHz5Umn9xMREmJmZKZSZmZkhMTGxPsKtEydOnEBGRgamTp1abp2m8FqXJnvNqvJ6pqamQiKRNKm/gfz8fCxduhQTJkyocBGzqr5XGqrBgwdj3759CAoKwtdff42LFy9iyJAhkEgkSus3xdf8559/hp6eHjw9PSus19Bec1pFt5mYM2cOIiIiKr3216tXL/Tq1Ut+v3fv3ujQoQN2796N9evX13WYtWLIkCHy2126dIGbmxtsbW1x+PBhlb4lNAX+/v4YMmQILC0ty63TFF5rUlZRURHGjh0Lxhh27dpVYd2m8l4ZP368/LaTkxO6dOmCNm3aIDg4GG+99RaPkdWfvXv3YtKkSZV2Lm9orzm1hDQDc+fOxalTp3DhwgVYW1tXaV91dXV07doV0dHRdRRd3TM0NES7du3KPQdzc3MkJSUplCUlJcHc3Lw+wqt1T58+xfnz5/HBBx9Uab+m8FrLXrOqvJ4mJiYQiURN4m9AloA8ffoU586dq/JS7pW9VxqL1q1bw8TEpNzzaEqvOQBcunQJUVFRVX7PA/y/5pSENGGMMcydOxfHjx/H33//DXt7+yofQyKR4M6dO7CwsKiDCOtHdnY2Hj16VO459OrVC0FBQQpl586dU2glaEx+/PFHtGzZEsOGDavSfk3htba3t4e5ubnC65mVlYVr166V+3pqaGige/fuCvtIpVIEBQU1qr8BWQLy8OFDnD9/HsbGxlU+RmXvlcbi2bNnSEtLK/c8msprLuPv74/u3bvD2dm5yvvy/prz3TOW1J2PP/6YGRgYsODgYJaQkCDfcnNz5XXef/99tmzZMvn9tWvXsrNnz7JHjx6xsLAwNn78eKapqcnu3r3LxylUy6JFi1hwcDCLiYlhV65cYe7u7szExIQlJyczxsqe85UrV5iamhr79ttvWWRkJFu9ejVTV1dnd+7c4esUqk0ikbBWrVqxpUuXlnmsqbzWL1++ZDdu3GA3btxgANjmzZvZjRs35KNAvvrqK2ZoaMhOnjzJbt++zUaMGMHs7e1ZXl6e/Bhvvvkm27Ztm/z+wYMHmVgsZj/99BO7d+8emzlzJjM0NGSJiYn1fn7lqei8CwsL2fDhw5m1tTW7efOmwvu9oKBAfozS513Ze6WhqOjcX758yRYvXsxCQkJYTEwMO3/+POvWrRtr27Yty8/Plx+jqb3mMpmZmUxbW5vt2rVL6TEa+mtOSUgTBkDp9uOPP8rr9O/fn3l5ecnvL1iwgLVq1YppaGgwMzMzNnToUBYeHl7/wdfAuHHjmIWFBdPQ0GBWVlZs3LhxLDo6Wv546XNmjLHDhw+zdu3aMQ0NDdapUycWGBhYz1HXjrNnzzIALCoqqsxjTeW1vnDhgtK/a9m5SaVStnLlSmZmZsbEYjF76623yvw+bG1t2erVqxXKtm3bJv99uLq6sqtXr9bTGammovOOiYkp9/1+4cIF+TFKn3dl75WGoqJzz83NZe+88w4zNTVl6urqzNbWln344Ydlkomm9prL7N69m2lpabGMjAylx2jor7mAMcbqtKmFEEIIIUQJ6hNCCCGEEF5QEkIIIYQQXlASQgghhBBeUBJCCCGEEF5QEkIIIYQQXlASQgghhBBeUBJCCCGEEF5QEkIIIYQQXlASQghpFoKDgyEQCJCRkcF3KISQVygJIYQQQggvKAkhhBBCCC8oCSGE1AupVAofHx/Y29tDS0sLzs7OOHr0KIDXl0oCAwPRpUsXaGpq4n//+x8iIiIUjnHs2DF06tQJYrEYdnZ22LRpk8LjBQUFWLp0KWxsbCAWi+Hg4AB/f3+FOmFhYejRowe0tbXRu3dvREVF1e2JE0LKRUkIIaRe+Pj4YN++ffDz88Pdu3excOFCTJ48GRcvXpTXWbJkCTZt2oTr16/D1NQUHh4eKCoqAsAlD2PHjsX48eNx584drFmzBitXrsRPP/0k33/KlCn47bffsHXrVkRGRmL37t3Q1dVViOOzzz7Dpk2b8N9//0FNTQ3Tp0+vl/MnhCjB2/q9hJBmIz8/n2lra7N///1XoXzGjBlswoQJ8iXLDx48KH8sLS2NaWlpsUOHDjHGGJs4cSJ7++23FfZfsmQJ69ixI2OMsaioKAaAnTt3TmkMsuc4f/68vCwwMJABYHl5ebVynoSQqqGWEEJInYuOjkZubi7efvtt6Orqyrd9+/bh0aNH8nq9evWS3zYyMoKjoyMiIyMBAJGRkejTp4/Ccfv06YOHDx9CIpHg5s2bEIlE6N+/f4WxdOnSRX7bwsICAJCcnFzjcySEVJ0a3wEQQpq+7OxsAEBgYCCsrKwUHhOLxQqJSHVpaWmpVE9dXV1+WyAQAOD6qxBC6h+1hBBC6lzHjh0hFosRGxsLBwcHhc3GxkZe7+rVq/Lb6enpePDgATp06AAA6NChA65cuaJw3CtXrqBdu3YQiURwcnKCVCpV6GNCCGnYqCWEEFLn9PT0sHjxYixcuBBSqRR9+/ZFZmYmrly5An19fdja2gIA1q1bB2NjY5iZmeGzzz6DiYkJRo4cCQBYtGgRevbsifXr12PcuHEICQnB9u3bsXPnTgCAnZ0dvLy8MH36dGzduhXOzs54+vQpkpOTMXbsWL5OnRBSAUpCCCH1Yv369TA1NYWPjw8eP34MQ0NDdOvWDStWrJBfDvnqq68wf/58PHz4EC4uLvjjjz+goaEBAOjWrRsOHz6MVatWYf369bCwsMC6deswdepU+XPs2rULK1aswOzZs5GWloZWrVphxYoVfJwuIUQFAsYY4zsIQkjzFhwcjIEDByI9PR2GhoZ8h0MIqSfUJ4QQQgghvKAkhBBCCCG8oMsxhBBCCOEFtYQQQgghhBeUhBBCCCGEF5SEEEIIIYQXlIQQQgghhBeUhBBCCCGEF5SEEEIIIYQXlIQQQgghhBeUhBBCCCGEF/8HcNlatwU5vtAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='18' class='progress-bar-interrupted' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      18.00% [18/100] [10:01<45:41]\n",
       "      <br>\n",
       "      ████████████████████100.00% [79/79] [val_loss=0.5354, val_auc=0.7271]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< val_auc without improvement in 5 epoch,early stopping >>>>>> \n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "dfhistory = model.fit(train_data = dl_train,\n",
    "    val_data = dl_val,\n",
    "    epochs=100,\n",
    "    ckpt_path='checkpoint',\n",
    "    patience=5,\n",
    "    monitor='val_auc',\n",
    "    mode='max',\n",
    "    plot=True,\n",
    "    cpu=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd77d8cf-e69c-430e-8229-f3f6b213388f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a97bac7d",
   "metadata": {},
   "source": [
    "### 4，评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "91871493-8ce5-41e6-8966-6ed1f852925d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████| 98/98 [00:04<00:00, 22.22it/s, val_auc=0.781, val_loss=0.466]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'val_loss': 0.46555993021750935, 'val_auc': 0.7811386585235596}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(dl_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81bde359-249f-44b2-a810-2ec7f6eb05cb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7a8cb4cd",
   "metadata": {},
   "source": [
    "### 5，使用模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2f99aaf7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7811383534406475\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "model.eval()\n",
    "dl_test = model.accelerator.prepare(dl_test)\n",
    "with torch.no_grad():\n",
    "    result = torch.cat([model.forward(t[0]) for t in dl_test])\n",
    "\n",
    "preds = F.sigmoid(result)\n",
    "labels = torch.cat([x[-1] for x in dl_test])\n",
    "\n",
    "val_auc = roc_auc_score(labels.numpy(),preds.numpy())\n",
    "print(val_auc)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dba04d27-8887-4cde-a6a5-04664fc1ef43",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1d139d55",
   "metadata": {},
   "source": [
    "### 6，保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c717899-cd41-4770-8f5e-f1c507fba044",
   "metadata": {},
   "source": [
    "模型最佳权重已经保存在 model.fit(ckpt_path) 传入的参数中了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "bf7f187f-ec7d-4ea5-b95f-9e65cc781ffe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net_clone = create_net()\n",
    "net_clone.load_state_dict(torch.load(model.ckpt_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ac90f13",
   "metadata": {},
   "source": [
    "**如果本书对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果对本书内容理解上有需要进一步和作者交流的地方，欢迎在公众号\"算法美食屋\"下留言。作者时间和精力有限，会酌情予以回复。\n",
    "\n",
    "也可以在公众号后台回复关键字：**加群**，加入读者交流群和大家讨论。\n",
    "\n",
    "![算法美食屋logo.png](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,md",
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}