7-5,FiBiNET模型.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "827f51d5",
   "metadata": {},
   "source": [
    "\n",
    "# 7-5，FiBiNET模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fccfc16",
   "metadata": {},
   "source": [
    "神经网络的结构设计有3个主流的高级技巧：\n",
    "\n",
    "* 1，高低融合 (将高层次特征与低层次特征融合，提升特征维度的丰富性和多样性，像人一样同时考虑整体和细节)\n",
    "* 2，权值共享 (一个权值矩阵参与多个不同的计算，降低参数规模并同时缓解样本稀疏性，像人一样一条知识多处运用)\n",
    "* 3，动态适应 (不同的输入样本使用不同的权值矩阵，动态地进行特征选择并赋予特征重要度解释性，像人一样聚焦重要信息排除干扰信息)\n",
    "\n",
    "技巧应用范例：\n",
    "\n",
    "* 1，高低融合 (DeepWide,UNet,特征金字塔FPN...)\n",
    "* 2，权值共享 (CNN,RNN,FM,DeepFM,BlinearFFM...)\n",
    "* 3，动态适应 (各种Attention机制...)\n",
    "\n",
    "新浪微博广告推荐技术团队2019年发布的CTR预估模型FiBiNET同时巧妙地运用了以上3种技巧，是神经网络结构设计的教科书级的范例。\n",
    "\n",
    "在此介绍给大家。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "582eca22",
   "metadata": {},
   "source": [
    "参考资料：\n",
    "* FiBiNET论文：https://arxiv.org/pdf/1905.09433.pdf\n",
    "* FiBiNET-结合特征重要性和双线性特征交互进行CTR预估：https://zhuanlan.zhihu.com/p/72931811\n",
    "* 代码实现：https://github.com/xue-pai/FuxiCTR/blob/main/fuxictr/pytorch/models/FiBiNET.py \n",
    "* SENet原理：https://zhuanlan.zhihu.com/p/65459972"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7e24de6",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "<font color=\"red\">\n",
    " \n",
    "公众号 **算法美食屋** 回复关键词：**pytorch**， 获取本项目源码和所用数据集百度云盘下载链接。\n",
    "    \n",
    "</font> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8828a0e5",
   "metadata": {},
   "source": [
    "## 一，FiBiNET原理解析"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca95583d",
   "metadata": {},
   "source": [
    "FiBiNET全称为Feature Importance and Bilinear Interaction Network.\n",
    "\n",
    "顾名思义，其主要的创意有2个。\n",
    "\n",
    "第一个是Feature Importance，通过借鉴SENet（Squeeze-and-Excitation）Attention机制实现特征选择和重要度解释。\n",
    "\n",
    "第二个是Bilinear Interaction Network，这是应用权值共享技巧对 FFM(Field-Aware FM)结构进行改进的一种结构。\n",
    "\n",
    "同时，FiBiNET保留了DeepWide的高低融合的网络架构。\n",
    "\n",
    "所以它综合使用了 高低融合、权值共享、动态适应 这3种神经网络结构设计的高级技巧。一个不落，Triple kill!\n",
    "\n",
    "我们重点介绍一下 SENet Attention 和 Bilinear Interaction."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f400a738",
   "metadata": {},
   "source": [
    "### 1, SENet Attention "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bf342b7",
   "metadata": {},
   "source": [
    "SENet 全称为 Squeeze-and-Excitation Network，是一种通过注意力机制计算特征重要度的网络模块。\n",
    "\n",
    "最早是在CV领域引入，通过在ResNet结构上添加SENet Attention模块，赢得了ImageNet 2017竞赛分类任务的冠军。\n",
    "\n",
    "如何计算各个Feature Map(通道)的特征重要度(注意力权重)呢？\n",
    "\n",
    "SENet的思想非常简洁。\n",
    "\n",
    "step1: 通过全局池化将各个Feature Map由一个一个的矩阵汇总成一个一个的标量。此即Squeeze操作。\n",
    "\n",
    "step2：通过一个2层MLP将汇总成得到的一个一个的标量所构成的向量进行变换，得到注意力权重。此即Excitation操作。 \n",
    "细节一点地说，这个2层的MLP的第1层将通道数量缩减成原来的1/3, 第2层再将通道数恢复。并且每层后面都接入了激活函数。\n",
    "\n",
    "step3：用注意力权重乘以原始的Feature Map。这个是Re-Weight操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f634cfe",
   "metadata": {},
   "source": [
    "图片示意如下。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2sfkkfcy3j208w06fgll.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a79f99ba",
   "metadata": {},
   "source": [
    "pytorch代码实现如下，可能比图片更加好懂。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a704b92e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "class SENetAttention(nn.Module):\n",
    "    \"\"\"\n",
    "    Squeeze-and-Excitation Attention\n",
    "    输入shape: [batch_size, num_fields, d_embed]   #num_fields即num_features\n",
    "    输出shape: [batch_size, num_fields, d_embed]\n",
    "    \"\"\"\n",
    "    def __init__(self, num_fields, reduction_ratio=3):\n",
    "        super().__init__()\n",
    "        reduced_size = max(1, int(num_fields / reduction_ratio))\n",
    "        self.excitation = nn.Sequential(nn.Linear(num_fields, reduced_size, bias=False),\n",
    "                                        nn.ReLU(),\n",
    "                                        nn.Linear(reduced_size, num_fields, bias=False),\n",
    "                                        nn.ReLU())\n",
    "\n",
    "    def forward(self, x):\n",
    "        Z = torch.mean(x, dim=-1, out=None) #1,Sequeeze\n",
    "        A = self.excitation(Z) #2,Excitation\n",
    "        V = x * A.unsqueeze(-1) #3,Re-Weight\n",
    "        return V"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "097ab325",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "57638416",
   "metadata": {},
   "source": [
    "### 2, Bilinear Interaction "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f049c68c",
   "metadata": {},
   "source": [
    "Bilinear Interaction实际上是FFM在权值共享思想下的一种改进，也可以称之为Bilinear FFM。\n",
    "\n",
    "我们先说说FFM(Field-Aware FM)，再看看这个Bilinear FFM 怎么改进的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a745eac",
   "metadata": {},
   "source": [
    "FM用隐向量之间的点积来计算特征之间的交叉，并且一个特征用一个隐向量来表示。\n",
    "\n",
    "FFM认为一个特征用一个隐向量来表达太粗糙了，如果这个特征和不同分组(Field)的特征来做交叉，应该用不同的隐向量。\n",
    "\n",
    "举例来说，考虑一个广告点击预测的场景，广告类别 和 用户所在城市、用户职业之间的交叉。\n",
    "\n",
    "在FM中 一个确定的广告类别 比如游戏广告 不论是和用户所在城市，还是用户职业交叉，都用同一个隐向量。\n",
    "\n",
    "但是FFM认为，用户所在城市和用户职业是两类完全不同的特征(不同Field)，描述它们的向量空间应该是完全不相关的，FM用一个相同的隐向量来和它们做点积不合理。\n",
    "\n",
    "所以，FFM引入了Field(域)的概念，和不同Field的特征做交叉，要使用不同的隐向量。\n",
    "\n",
    "实践表明，FFM这个思路是有效的, FFM的作者阮毓钦正是凭借这个方案赢得了2015年kaggle举办的Criteo比赛的冠军。\n",
    "\n",
    "但是FFM有个很大的缺点，就是参数量太多了。\n",
    "\n",
    "对于FM来说，每个特征只有一个隐向量，假设有n个特征，每个隐向量维度为k，全部隐向量参数矩阵的大小 size = n k.\n",
    "\n",
    "但是对于FFM，有过有f个不同的field，每个特征都将有f-1个隐向量，全部隐向量的参数矩阵的大小增大为 size = (f-1) n k. \n",
    "\n",
    "通常的应用场景中，Field的数量有几十几百维，而Feature的数量有数万数百万维。\n",
    "\n",
    "很显然，FFM将隐向量的参数规模扩大了几十几百倍。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b57c8810",
   "metadata": {},
   "source": [
    "FFM的本质思想是在做特征交叉的时候要区分不同的Field，其实现方式是和不同的Field做交叉时用不同的隐向量。\n",
    "\n",
    "有没有办法保留FFM中区分不同Field的特性，并降低参数规模呢？\n",
    "\n",
    "BilinearFFM说，我有办法，权重共享走起来！\n",
    "\n",
    "BilinearFFM不直接针对不同Field设计不同的隐向量，而是引入了Field变换矩阵来区分不同的Field。\n",
    "\n",
    "每个特征还是一个隐向量，但是和不同的Field的特征做交叉时，先乘上这个特征所在Field的变换矩阵，然后再做后面的点积。\n",
    "\n",
    "因此，同属一个Field的特征共享一个Field变换矩阵。这种bilinear_type叫做 field_each.\n",
    "\n",
    "Field变换矩阵的大小是k^2, 这种方式下，全部隐向量的参数大小加上共享变换矩阵的参数大小一共是 size = n k + f k^2 \n",
    "\n",
    "由于k和f远小于n，这种Bilinear方式相比FM增加的参数量可以忽略不计。\n",
    "\n",
    "\n",
    "除了 同属一个Field的特征共享一个Field变换矩阵外，我们还可以更加简单粗暴一点，所有特征共享一个变换矩阵.\n",
    "\n",
    "这种bilinear_type叫做 field_all.这种方式下，size = n k + k^2 \n",
    "\n",
    "\n",
    "我们也可以更加精细一点，相同的Field组合之间的交互共享一个变换矩阵，这种bilinear_type叫做field_interaction. \n",
    "\n",
    "总共有f(f-1)/2种组合，这种方式下， size = n k + k^2 f(f-1)/2\n",
    "\n",
    "以上就是BilinearFFM的基本思想。\n",
    "\n",
    "\n",
    "FiBiNET中用到的Bilinear Interaction相比BilinearFFM, 还有一处小改动，将点积改成了哈达玛积，如下图所示。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65661157",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2sfj6wh0rj209v08daad.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32bac64f",
   "metadata": {},
   "source": [
    "pytorch代码实现如下，整体不难理解。作2点说明。\n",
    "\n",
    "1，Field概念说明\n",
    "\n",
    "在FFM相关的文章中，引入了Field的概念，以和Feature区分，一个Field中可以包括多个Feature. \n",
    "\n",
    "实际上Field就是我们通常理解的特征，包括数值特征和类别特征，但是Feature是数值特征或者类别特征onehot后的特征。一个类别特征对应一个Field，但是对应多个Feature。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2sh8t8j6pj20gc058mx9.jpg)\n",
    "\n",
    "2，combinations函数说明\n",
    "\n",
    "组合函数combinations从num_fields中任取2种作为组合，共有 num_fields*(num_fields-1)中组合方式。\n",
    "\n",
    "所以输出的Field数量变成了 num_fields*(num_fields-1)/2。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "09d50922",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "from itertools import combinations\n",
    "class BilinearInteraction(nn.Module):\n",
    "    \"\"\"\n",
    "    双线性FFM\n",
    "    输入shape: [batch_size, num_fields, d_embed] #num_fields即num_features\n",
    "    输出shape: [batch_size, num_fields*(num_fields-1)/2, d_embed]\n",
    "    \"\"\"\n",
    "    def __init__(self, num_fields, d_embed, bilinear_type=\"field_interaction\"):\n",
    "        super().__init__()\n",
    "        self.bilinear_type = bilinear_type\n",
    "        if self.bilinear_type == \"field_all\":\n",
    "            self.bilinear_layer = nn.Linear(d_embed, d_embed, bias=False)\n",
    "        elif self.bilinear_type == \"field_each\":\n",
    "            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)\n",
    "                                                 for i in range(num_fields)])\n",
    "        elif self.bilinear_type == \"field_interaction\":\n",
    "            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)\n",
    "                                                 for i, j in combinations(range(num_fields), 2)])\n",
    "        else:\n",
    "            raise NotImplementedError()\n",
    "\n",
    "    def forward(self, feature_emb):\n",
    "        feature_emb_list = torch.split(feature_emb, 1, dim=1)\n",
    "        if self.bilinear_type == \"field_all\":\n",
    "            bilinear_list = [self.bilinear_layer(v_i) * v_j\n",
    "                             for v_i, v_j in combinations(feature_emb_list, 2)]\n",
    "        elif self.bilinear_type == \"field_each\":\n",
    "            bilinear_list = [self.bilinear_layer[i](feature_emb_list[i]) * feature_emb_list[j]\n",
    "                             for i, j in combinations(range(len(feature_emb_list)), 2)]\n",
    "        elif self.bilinear_type == \"field_interaction\":\n",
    "            bilinear_list = [self.bilinear_layer[i](v[0]) * v[1]\n",
    "                             for i, v in enumerate(combinations(feature_emb_list, 2))]\n",
    "        return torch.cat(bilinear_list, dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efe26681",
   "metadata": {},
   "source": [
    "### 二，FiBiNET的pytorch实现"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1f340fd",
   "metadata": {},
   "source": [
    "下面是FiBiNET的一个pytorch实现。\n",
    "\n",
    "核心代码是SENetAttention模块和BilinearInteraction模块的实现。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2sffi4huoj20g70a4gm4.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "93153b47",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "from itertools import combinations\n",
    "\n",
    "class NumEmbedding(nn.Module):\n",
    "    \"\"\"\n",
    "    连续特征用linear层编码\n",
    "    输入shape: [batch_size,num_features, d_in], # d_in 通常是1\n",
    "    输出shape: [batch_size,num_features, d_out]\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, n: int, d_in: int, d_out: int, bias: bool = False) -> None:\n",
    "        super().__init__()\n",
    "        self.weight = nn.Parameter(torch.Tensor(n, d_in, d_out))\n",
    "        self.bias = nn.Parameter(torch.Tensor(n, d_out)) if bias else None\n",
    "        with torch.no_grad():\n",
    "            for i in range(n):\n",
    "                layer = nn.Linear(d_in, d_out)\n",
    "                self.weight[i] = layer.weight.T\n",
    "                if self.bias is not None:\n",
    "                    self.bias[i] = layer.bias\n",
    "\n",
    "    def forward(self, x_num):\n",
    "        assert x_num.ndim == 3\n",
    "        #x = x_num[..., None] * self.weight[None]\n",
    "        #x = x.sum(-2)\n",
    "        x = torch.einsum(\"bfi,fij->bfj\",x_num,self.weight)\n",
    "        if self.bias is not None:\n",
    "            x = x + self.bias[None]\n",
    "        return x\n",
    "    \n",
    "class CatEmbedding(nn.Module):\n",
    "    \"\"\"\n",
    "    离散特征用Embedding层编码\n",
    "    输入shape: [batch_size, num_features], \n",
    "    输出shape: [batch_size, num_features, d_embed]\n",
    "    \"\"\"\n",
    "    def __init__(self, categories, d_embed):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(sum(categories), d_embed)\n",
    "        self.offsets = nn.Parameter(\n",
    "                torch.tensor([0] + categories[:-1]).cumsum(0),requires_grad=False)\n",
    "        \n",
    "        nn.init.xavier_uniform_(self.embedding.weight.data)\n",
    "\n",
    "    def forward(self, x_cat):\n",
    "        \"\"\"\n",
    "        x_cat: Long tensor of size ``(batch_size, features_num)``\n",
    "        \"\"\"\n",
    "        x = x_cat + self.offsets[None]\n",
    "        return self.embedding(x) \n",
    "    \n",
    "class CatLinear(nn.Module):\n",
    "    \"\"\"\n",
    "    离散特征用Embedding实现线性层（等价于先F.onehot再nn.Linear()）\n",
    "    输入shape: [batch_size, num_features ], \n",
    "    输出shape: [batch_size, d_out]\n",
    "    \"\"\"\n",
    "    def __init__(self, categories, d_out=1):\n",
    "        super().__init__()\n",
    "        self.fc = nn.Embedding(sum(categories), d_out)\n",
    "        self.bias = nn.Parameter(torch.zeros((d_out,)))\n",
    "        self.offsets = nn.Parameter(\n",
    "                torch.tensor([0] + categories[:-1]).cumsum(0),requires_grad=False)\n",
    "        nn.init.xavier_uniform_(self.fc.weight.data)\n",
    "\n",
    "    def forward(self, x_cat):\n",
    "        \"\"\"\n",
    "        Long tensor of size ``(batch_size, num_features)``\n",
    "        \"\"\"\n",
    "        x = x_cat + self.offsets[None]\n",
    "        return torch.sum(self.fc(x), dim=1) + self.bias \n",
    "    \n",
    "class SENetAttention(nn.Module):\n",
    "    \"\"\"\n",
    "    Squeeze-and-Excitation Attention\n",
    "    输入shape: [batch_size, num_fields, d_embed]   #num_fields即num_features\n",
    "    输出shape: [batch_size, num_fields, d_embed]\n",
    "    \"\"\"\n",
    "    def __init__(self, num_fields, reduction_ratio=3):\n",
    "        super().__init__()\n",
    "        reduced_size = max(1, int(num_fields / reduction_ratio))\n",
    "        self.excitation = nn.Sequential(nn.Linear(num_fields, reduced_size, bias=False),\n",
    "                                        nn.ReLU(),\n",
    "                                        nn.Linear(reduced_size, num_fields, bias=False),\n",
    "                                        nn.ReLU())\n",
    "\n",
    "    def forward(self, x):\n",
    "        Z = torch.mean(x, dim=-1, out=None) #1,Sequeeze\n",
    "        A = self.excitation(Z) #2,Excitation\n",
    "        V = x * A.unsqueeze(-1) #3,Re-Weight\n",
    "        return V\n",
    "    \n",
    "class BilinearInteraction(nn.Module):\n",
    "    \"\"\"\n",
    "    双线性FFM\n",
    "    输入shape: [batch_size, num_fields, d_embed] #num_fields即num_features\n",
    "    输出shape: [batch_size, num_fields*(num_fields-1)/2, d_embed]\n",
    "    \"\"\"\n",
    "    def __init__(self, num_fields, d_embed, bilinear_type=\"field_interaction\"):\n",
    "        super().__init__()\n",
    "        self.bilinear_type = bilinear_type\n",
    "        if self.bilinear_type == \"field_all\":\n",
    "            self.bilinear_layer = nn.Linear(d_embed, d_embed, bias=False)\n",
    "        elif self.bilinear_type == \"field_each\":\n",
    "            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)\n",
    "                                                 for i in range(num_fields)])\n",
    "        elif self.bilinear_type == \"field_interaction\":\n",
    "            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)\n",
    "                                                 for i, j in combinations(range(num_fields), 2)])\n",
    "        else:\n",
    "            raise NotImplementedError()\n",
    "\n",
    "    def forward(self, feature_emb):\n",
    "        feature_emb_list = torch.split(feature_emb, 1, dim=1)\n",
    "        if self.bilinear_type == \"field_all\":\n",
    "            bilinear_list = [self.bilinear_layer(v_i) * v_j\n",
    "                             for v_i, v_j in combinations(feature_emb_list, 2)]\n",
    "        elif self.bilinear_type == \"field_each\":\n",
    "            bilinear_list = [self.bilinear_layer[i](feature_emb_list[i]) * feature_emb_list[j]\n",
    "                             for i, j in combinations(range(len(feature_emb_list)), 2)]\n",
    "        elif self.bilinear_type == \"field_interaction\":\n",
    "            bilinear_list = [self.bilinear_layer[i](v[0]) * v[1]\n",
    "                             for i, v in enumerate(combinations(feature_emb_list, 2))]\n",
    "        return torch.cat(bilinear_list, dim=1)\n",
    "    \n",
    "\n",
    "#mlp\n",
    "class MultiLayerPerceptron(nn.Module):\n",
    "    def __init__(self, d_in, d_layers, dropout, \n",
    "                 d_out = 1):\n",
    "        super().__init__()\n",
    "        layers = []\n",
    "        for d in d_layers:\n",
    "            layers.append(nn.Linear(d_in, d))\n",
    "            layers.append(nn.BatchNorm1d(d))\n",
    "            layers.append(nn.ReLU())\n",
    "            layers.append(nn.Dropout(p=dropout))\n",
    "            d_in = d\n",
    "        layers.append(nn.Linear(d_layers[-1], d_out))\n",
    "        self.mlp = nn.Sequential(*layers)\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        float tensor of size ``(batch_size, d_in)``\n",
    "        \"\"\"\n",
    "        return self.mlp(x)\n",
    "    \n",
    "\n",
    "#fibinet \n",
    "class FiBiNET(nn.Module):\n",
    "    \n",
    "    def __init__(self,\n",
    "                 d_numerical, \n",
    "                 categories, \n",
    "                 d_embed,\n",
    "                 mlp_layers, \n",
    "                 mlp_dropout,\n",
    "                 reduction_ratio = 3,\n",
    "                 bilinear_type = \"field_interaction\",\n",
    "                 n_classes = 1):\n",
    "        \n",
    "        super().__init__()\n",
    "        \n",
    "        if d_numerical is None:\n",
    "            d_numerical = 0\n",
    "        if categories is None:\n",
    "            categories = []\n",
    "            \n",
    "        self.categories = categories\n",
    "        self.n_classes = n_classes\n",
    "        \n",
    "        self.num_linear = nn.Linear(d_numerical,n_classes) if d_numerical else None\n",
    "        self.cat_linear = CatLinear(categories,n_classes) if categories else None\n",
    "        \n",
    "        self.num_embedding = NumEmbedding(d_numerical,1,d_embed) if d_numerical else None\n",
    "        self.cat_embedding = CatEmbedding(categories, d_embed) if categories else None\n",
    "        \n",
    "        num_fields = d_numerical+len(categories)\n",
    "        \n",
    "        self.se_attention = SENetAttention(num_fields, reduction_ratio)\n",
    "        self.bilinear = BilinearInteraction(num_fields, d_embed, bilinear_type)\n",
    "        \n",
    "        mlp_in = num_fields * (num_fields - 1) * d_embed\n",
    "        self.mlp = MultiLayerPerceptron(\n",
    "            d_in= mlp_in,\n",
    "            d_layers = mlp_layers,\n",
    "            dropout = mlp_dropout,\n",
    "            d_out = n_classes\n",
    "        )\n",
    "        \n",
    "        \n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        x_num: numerical features\n",
    "        x_cat: category features\n",
    "        \"\"\"\n",
    "        x_num,x_cat = x\n",
    "        \n",
    "        #一，wide部分\n",
    "        x_linear = 0.0\n",
    "        if self.num_linear:\n",
    "            x_linear = x_linear + self.num_linear(x_num) \n",
    "        if self.cat_linear:\n",
    "            x_linear = x_linear + self.cat_linear(x_cat)\n",
    "            \n",
    "        #二，deep部分 \n",
    "        \n",
    "        #1，embedding\n",
    "        x_embedding = []\n",
    "        if self.num_embedding:\n",
    "            x_embedding.append(self.num_embedding(x_num[...,None]))\n",
    "        if self.cat_embedding:\n",
    "            x_embedding.append(self.cat_embedding(x_cat))\n",
    "        x_embedding = torch.cat(x_embedding,dim=1)\n",
    "        \n",
    "        #2，interaction\n",
    "        se_embedding = self.se_attention(x_embedding)\n",
    "        ffm_out = self.bilinear(x_embedding)\n",
    "        se_ffm_out = self.bilinear(se_embedding)\n",
    "        x_interaction = torch.flatten(torch.cat([ffm_out, se_ffm_out], dim=1), start_dim=1)\n",
    "        \n",
    "        #3，mlp\n",
    "        x_deep = self.mlp(x_interaction)\n",
    "        \n",
    "        #三，高低融合\n",
    "        x_out = x_linear+x_deep\n",
    "        if self.n_classes==1:\n",
    "            x_out = x_out.squeeze(-1)\n",
    "        return x_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6b39cb4a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 1.4711, -0.9382], grad_fn=<SqueezeBackward1>)\n"
     ]
    }
   ],
   "source": [
    "##测试 FiBiNET\n",
    "\n",
    "model = FiBiNET(d_numerical = 3, categories = [4,3,2],\n",
    "        d_embed = 4, mlp_layers = [20,20], mlp_dropout=0.25,\n",
    "        reduction_ratio = 3,\n",
    "        bilinear_type = \"field_interaction\",\n",
    "        n_classes = 1)\n",
    "\n",
    "x_num = torch.randn(2,3)\n",
    "x_cat = torch.randint(0,2,(2,3))\n",
    "print(model((x_num,x_cat)))  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d2a960d",
   "metadata": {},
   "source": [
    "```\n",
    "tensor([-0.8621,  0.6743], grad_fn=<SqueezeBackward1>)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec37a4df",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "72a05548",
   "metadata": {},
   "source": [
    "## 三，Criteo数据集完整范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79dd5aef",
   "metadata": {},
   "source": [
    "Criteo数据集是一个经典的广告点击率CTR预测数据集。\n",
    "\n",
    "这个数据集的目标是通过用户特征和广告特征来预测某条广告是否会为用户点击。\n",
    "\n",
    "数据集有13维数值特征(I1-I13)和26维类别特征(C14-C39), 共39维特征, 特征中包含着许多缺失值。\n",
    "\n",
    "训练集4000万个样本，测试集600万个样本。数据集大小超过100G.\n",
    "\n",
    "此处使用的是采样100万个样本后的cretio_small数据集。 \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16474d85",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "70d99f21",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd \n",
    "\n",
    "from sklearn.model_selection import train_test_split \n",
    "\n",
    "import torch \n",
    "from torch import nn \n",
    "from torch.utils.data import Dataset,DataLoader  \n",
    "import torch.nn.functional as F \n",
    "import torchkeras \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e3f720c",
   "metadata": {},
   "source": [
    "### 1，准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "edf49cd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder,QuantileTransformer\n",
    "from sklearn.pipeline import Pipeline \n",
    "from sklearn.impute import SimpleImputer \n",
    "\n",
    "dfdata = pd.read_csv(\"./eat_pytorch_datasets/criteo_small.zip\",sep=\"\\t\",header=None)\n",
    "dfdata.columns = [\"label\"] + [\"I\"+str(x) for x in range(1,14)] + [\n",
    "    \"C\"+str(x) for x in range(14,40)]\n",
    "\n",
    "cat_cols = [x for x in dfdata.columns if x.startswith('C')]\n",
    "num_cols = [x for x in dfdata.columns if x.startswith('I')]\n",
    "num_pipe = Pipeline(steps = [('impute',SimpleImputer()),('quantile',QuantileTransformer())])\n",
    "\n",
    "for col in cat_cols:\n",
    "    dfdata[col]  = LabelEncoder().fit_transform(dfdata[col])\n",
    "\n",
    "dfdata[num_cols] = num_pipe.fit_transform(dfdata[num_cols])\n",
    "\n",
    "categories = [dfdata[col].max()+1 for col in cat_cols]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "86e22fca",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch.utils.data import Dataset,DataLoader \n",
    "\n",
    "#DataFrame转换成torch数据集Dataset, 特征分割成X_num,X_cat方式\n",
    "class DfDataset(Dataset):\n",
    "    def __init__(self,df,\n",
    "                 label_col,\n",
    "                 num_features,\n",
    "                 cat_features,\n",
    "                 categories,\n",
    "                 is_training=True):\n",
    "        \n",
    "        self.X_num = torch.tensor(df[num_features].values).float() if num_features else None\n",
    "        self.X_cat = torch.tensor(df[cat_features].values).long() if cat_features else None\n",
    "        self.Y = torch.tensor(df[label_col].values).float() \n",
    "        self.categories = categories\n",
    "        self.is_training = is_training\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.Y)\n",
    "    \n",
    "    def __getitem__(self,index):\n",
    "        if self.is_training:\n",
    "            return ((self.X_num[index],self.X_cat[index]),self.Y[index])\n",
    "        else:\n",
    "            return (self.X_num[index],self.X_cat[index])\n",
    "    \n",
    "    def get_categories(self):\n",
    "        return self.categories\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a77fcdf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "dftrain_val,dftest = train_test_split(dfdata,test_size=0.2)\n",
    "dftrain,dfval = train_test_split(dftrain_val,test_size=0.2)\n",
    "\n",
    "ds_train = DfDataset(dftrain,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n",
    "\n",
    "ds_val = DfDataset(dfval,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n",
    "\n",
    "ds_test = DfDataset(dftest,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "77e4b9cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "dl_train = DataLoader(ds_train,batch_size = 2048,shuffle=True)\n",
    "dl_val = DataLoader(ds_val,batch_size = 2048,shuffle=False)\n",
    "dl_test = DataLoader(ds_test,batch_size = 2048,shuffle=False)\n",
    "\n",
    "for features,labels in dl_train:\n",
    "    break \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e83da026",
   "metadata": {},
   "source": [
    "### 2，定义模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6107212d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_net():\n",
    "    net = FiBiNET(\n",
    "        d_numerical= ds_train.X_num.shape[1],\n",
    "        categories= ds_train.get_categories(),\n",
    "        d_embed = 8, mlp_layers = [128,64,32], mlp_dropout=0.25,\n",
    "        reduction_ratio = 3,\n",
    "        bilinear_type = \"field_all\",\n",
    "        n_classes = 1\n",
    "        \n",
    "    )\n",
    "    return net \n",
    "\n",
    "from torchkeras import summary\n",
    "\n",
    "net = create_net()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dd54b90",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e51aff2e",
   "metadata": {},
   "source": [
    "### 3，训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3353895-6d7d-4504-84cd-7817badca0cc",
   "metadata": {},
   "source": [
    "我们使用梦中情炉torchkeras来实现最优雅的训练循环。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "81598c6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras.metrics import AUC\n",
    "from torchkeras import KerasModel \n",
    "\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "\n",
    "metrics_dict = {\"auc\":AUC()}\n",
    "\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=0.002, weight_decay=0.001) \n",
    "\n",
    "model = KerasModel(net,\n",
    "                   loss_fn = loss_fn,\n",
    "                   metrics_dict= metrics_dict,\n",
    "                   optimizer = optimizer\n",
    "                  )         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cd242b51",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< 🐌 cpu is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiEAAAGJCAYAAABcsOOZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABp30lEQVR4nO3dd1zU9R8H8NfdAccGEWQICooTETe5UtMyMxVJUdOcDXOkkZVWampKS0PTMgt/WZmKipWa5side09cuBEQBQRl3X1+f3y9k2Pv7wGv5+NxD+4+3899733H+L75TIUQQoCIiIionCnlDoCIiIiqJiYhREREJAsmIURERCQLJiFEREQkCyYhREREJAsmIURERCQLJiFEREQkCyYhREREJAsmIURERCQLJiFkdD799FMoFArcu3dP7lDKzbVr16BQKPDzzz/LHQqVoS+//BINGzaEVquVO5RyU16/z+fOnYOJiQnOnDlTpq9DpYtJCNETc+bMwR9//CF3GJVaQkIC3nzzTTg5OcHKygpdunTBsWPHCvVchUKR5+355583qBsdHY0333wTXl5esLCwQN26dREcHIz4+Hh9Ha1Wi59//hm9e/eGh4cHrKys0KRJE3z22WdITU3N8foxMTEYMWIEatSoAQsLC7Ro0QKrV68u9HtPSkrCF198gQ8//BBKZcF/elNTU5GRkVHo81dm27ZtQ5cuXeDo6Ah7e3u0adMGv/76q0Gdxo0bo2fPnpg2bZpMUVJxmMgdAJGxmDNnDvr164eAgAC5Q6mUtFotevbsiZMnT+L999+Ho6MjvvvuO3Tu3BlHjx5FvXr18n1+9osOABw5cgTz58/HCy+8oC9LTk5G27ZtkZKSgjFjxsDDwwMnT57EwoULsWPHDhw9ehRKpRKPHj3CiBEj8Mwzz2D06NGoUaMG9u/fj+nTp2P79u34999/oVAoAEgJRIcOHRATE4MJEybAxcUF4eHhCAoKwvLly/Hqq68W+P6XLl2KzMxMDBo0KM86hw8fxoIFC7BlyxbExsZCoVCgZs2a6Nu3L9555x14e3sX+DqVzV9//YWAgAC0bdtW36oSHh6OoUOH4t69e3j33Xf1dUePHo2XXnoJV65cQd26dWWMmgpNEBmZ6dOnCwAiLi6uXF/XyspKDBs2rFxfUycqKkoAEP/73/9kef3ysGrVKgFArF69Wl8WGxsr7O3txaBBg4p1zlGjRgmFQiFu3rypL1u+fLkAIDZs2GBQd9q0aQKAOHbsmBBCiLS0NLFv374c55wxY4YAILZu3aov+/LLLwUAsX37dn2ZRqMRrVu3Fi4uLiItLa3AWJs2bSqGDBmS67GMjAwxduxYoVAoRMeOHcXXX38t1q9fL9auXSvmzJkjmjVrJszNzcXChQsLfB1jU9Lf5+eff164ubmJ1NRUfVlGRoaoW7euaNq0qUHd9PR0Ua1aNTF16tQSxUzlh90xZLTu3buHoKAg2Nraonr16pgwYUKuzeS//fYbWrZsCQsLCzg4OGDgwIG4efOmQZ1Lly7hlVdegYuLC8zNzeHu7o6BAwciMTERgNTUn5KSgmXLlumb+IcPH55rXDExMTAxMcGMGTNyHIuMjIRCocDChQsBAPfv38ekSZPg6+sLa2tr2NraokePHjh58mQJPx0gPT0d06ZNQ8uWLWFnZwcrKyt07NgRO3bsMKi3c+dOKBQK7Ny506A8r3EoFy5cQFBQEJycnGBhYYEGDRrg448/LnG8a9asgbOzMwIDA/VlTk5OCAoKwp9//om0tLQinS8tLQ1r165Fp06d4O7uri9PSkoCADg7OxvUd3V1BQBYWFgAAMzMzNCuXbsc5+3bty8A4Pz58/qyPXv2wMnJCc8995y+TKlUIigoCHfv3sWuXbvyjTUqKgqnTp1Ct27dcj0+YsQI/P777/j777+xe/duvPfee3j55ZcRGBiIKVOm4Pjx41i8eDEmTZqExYsX53j+7du3MXLkSDg7O0OtVsPHxwdLly41qKP7OVi1ahU++ugjuLi4wMrKCr17987x+wIAq1ev1v9eOTo6YsiQIbh9+3aOeoX9eUlISMDw4cNhb28POzs7jBgxAo8ePcr3cwOk72e1atWgVqv1ZSYmJnB0dNR/L3VMTU3RuXNn/PnnnwWel4wDu2PIaAUFBcHT0xMhISE4cOAAFixYgAcPHuCXX37R15k9ezamTp2KoKAgvP7664iLi8O3336LZ599FsePH4e9vT3S09PRvXt3pKWlYfz48XBxccHt27exYcMGJCQkwM7ODr/++itef/11tGnTBm+++SYA5Nmc6+zsjE6dOiE8PBzTp083OLZq1SqoVCr0798fAHD16lX88ccf6N+/P7y8vBATE4MffvgBnTp1wrlz5+Dm5lbszycpKQk//fQTBg0ahDfeeAMPHz5EWFgYunfvjkOHDqFZs2ZFPuepU6fQsWNHmJqa4s0334SnpyeuXLmC9evXY/bs2QCAjIwMffJWEAcHB/34h+PHj6NFixY5xkO0adMGS5YswcWLF+Hr61voWP/++28kJCRg8ODBBuXPPvsslEolJkyYgLlz58Ld3R2nTp3C7NmzERAQgIYNG+Z73rt37wIAHB0d9WVpaWk5LngAYGlpCQA4evRojnEpWf33338AgBYtWuQ49uuvv2LdunU4ePAgfHx8AABCCKSkpMDa2hqAlJC/9tprcHR0RP/+/dGjRw/Url0bgJQUP/PMM1AoFBg3bhycnJywadMmjBo1CklJSZg4caLB682ePRsKhQIffvghYmNjERoaim7duuHEiRP69/jzzz9jxIgRaN26NUJCQhATE4P58+dj3759+t8roHA/LzpBQUHw8vJCSEgIjh07hp9++gk1atTAF198kefnBgCdO3fGF198galTp2LYsGFQKBT4/fffceTIEYSHh+eo37JlS/z5559ISkqCra1tvucmIyB3UwxRdrrm2969exuUjxkzRgAQJ0+eFEIIce3aNaFSqcTs2bMN6p0+fVqYmJjoy48fP56jGyA3RemO+eGHHwQAcfr0aYPyxo0bi+eee07/ODU1VWg0GoM6UVFRQq1Wi5kzZxqUoYjdMZmZmTm6AR48eCCcnZ3FyJEj9WU7duwQAMSOHTtyxJH9NZ999llhY2Mjrl+/blBXq9XmOF9hblFRUfrnWVlZGcSls3HjRgFAbN68udDvXQghXnnlFaFWq8WDBw9yHPvpp5+Evb29QSzDhg0TGRkZBZ63W7duwtbW1uC848ePF0qlUly7ds2g7sCBAwUAMW7cuHzP+cknnwgA4uHDhwblWq1WeHl5idDQUH3Zn3/+Kdzc3AQAUatWLfHPP/8YfJZ9+/YVH330kb7+qFGjhKurq7h3716O2Ozs7MSjR4+EEE+/bzVr1hRJSUn6euHh4QKAmD9/vhBC6tKoUaOGaNKkiXj8+LG+3oYNGwQAMW3aNH1ZYX5edL/P2b/3ffv2FdWrV8/3cxNCiOTkZBEUFCQUCoX+e2lpaSn++OOPXOv//vvvAoA4ePBggecm+bE7hozW2LFjDR6PHz8egPQfMABERERAq9UiKCgI9+7d099cXFxQr149fbeEnZ0dAOCff/4pVPNvYQQGBsLExASrVq3Sl505cwbnzp3DgAED9GVqtVr/n79Go0F8fDysra3RoEGDQs8KyYtKpYKZmRkAadDn/fv3kZmZiVatWhXr3HFxcdi9ezdGjhyJWrVqGRzTDdAEAD8/P2zdurVQNxcXF/3zHj9+bNCkrmNubq4/XlhJSUnYuHEjXnrpJf1/5VnVrFkTbdq0QWhoKNatW4fg4GAsX74ckydPzve8c+bMwbZt2/D5558bnPf111+HSqVCUFAQ/vvvP1y5cgUhISFYt25doWKPj4+HiYmJvmVD5+jRo4iNjcWoUaMASN0qgwYNQps2bbB27Vq8++67GDlypMFzAgIC9F1rQgisXbsWvXr1ghDC4Pege/fuSExMzPGzMHToUNjY2Ogf9+vXD66urvrfqyNHjiA2NhZjxozRf28AoGfPnmjYsCE2btwIoPA/LzqjR482eNyxY0fEx8fru8/yolarUb9+ffTr1w8rVqzAb7/9hlatWmHIkCE4cOBAjvrVqlUDgCo1xb8iY3cMGa3ssyXq1q0LpVKJa9euAZDGeQgh8pxVYWpqCgDw8vJCcHAw5s2bh+XLl6Njx47o3bs3hgwZok9QisrR0RFdu3ZFeHg4Zs2aBUDqijExMTEY86DVajF//nx89913iIqKgkaj0R+rXr16sV47q2XLlmHu3Lm4cOGCwXROLy+vIp/r6tWrAIAmTZrkW69atWp5jm3Ij4WFRa7jPnTjfHLr7sjL2rVrkZqamqMrBgD27duHl19+GQcOHECrVq0ASBduW1tbzJgxAyNHjkTjxo1zPG/VqlX45JNPMGrUKLz99tsGx5o2bYrff/8do0ePRvv27QEALi4uCA0Nxdtvv50juSiso0ePolWrVvrnL1++HDVr1sSaNWugUqkAAPb29hgxYoT+Oc7OzoiLiwMgJQIJCQlYsmQJlixZkutrxMbGGjzO/vuiUCjg7e2t/726fv06AKBBgwY5ztWwYUPs3bsXQOF/XnSyJyq6ZOHBgwf5dpuMGzcOBw4cwLFjx/QJfVBQEHx8fDBhwgQcPHjQoL4QQv++yPgxCaEKI/sfFa1WC4VCgU2bNun/YGeV9cIwd+5cDB8+HH/++Se2bNmCd955Rz/WJOugxqIYOHAgRowYgRMnTqBZs2YIDw9H165dDcYSzJkzB1OnTsXIkSMxa9Ys/RiJiRMnlnjBqt9++w3Dhw9HQEAA3n//fdSoUQMqlQohISG4cuWKvl5ef4yzJkRFkZ6ejvv37xeqrpOTk/574+rqiujo6Bx1dGVFGR+zfPly2NnZ4eWXX85x7IcffoCzs7M+AdHp3bs3Pv30U/z33385kpCtW7di6NCh6NmzZ64DPwGpxaB37944efIkNBoNWrRooW+RqF+/fr7xVq9eHZmZmXj48KFBK0R8fLzB+7527RqaN29u8PPcpk0bg3PdvHlTn8DqfoaGDBmCYcOG5fraTZs2zTe28pLb7yjwNGnITXp6OsLCwvDBBx8YjCUyNTVFjx49sHDhQqSnp+tbBAEpqQEMx/SQ8WISQkbr0qVLBv/RX758GVqtFp6engCklhEhBLy8vAq8CACAr68vfH198cknn+C///5D+/btsXjxYnz22WcAiv6fU0BAAN566y19l8zFixcxZcoUgzpr1qxBly5dEBYWZlCekJBQ4j+Sa9asQZ06dRAREWEQe/bBsrr/OBMSEgzKdf/x6tSpUwcAClxx8r///kOXLl0KFWNUVJT++9WsWTPs2bMHWq3W4IJy8OBBWFpaFup7CEhJy44dOzB8+PBcu3diYmJyTbB0LUWZmZkG5QcPHkTfvn3RqlUrhIeHw8Qk7z+LZmZmaN26tf7xtm3bAKDAliHdYNioqCiDpMDW1tZgkK+LiwsOHTpk8FxdiwMgXbDDwsL0r+fk5AQbGxtoNJpCt05dunTJ4LEQApcvX9bHpRvwGhkZaTAbSFemO17Yn5eSiI+PR2ZmZp7fT61Wm+NYVFQUlEploX+eSF4cE0JGa9GiRQaPv/32WwBAjx49AEjjMlQqFWbMmJHjvykhhH51zKSkpBwXHl9fXyiVSoPuASsrqxwX6vzY29uje/fuCA8Px8qVK2FmZpZjoTOVSpUjttWrV+c61bGodP9ZZj3/wYMHsX//foN6tWvXhkqlwu7duw3Kv/vuO4PHTk5OePbZZ7F06VLcuHHD4FjW1yjumJB+/fohJiYGERER+rJ79+5h9erV6NWrl0FCceXKFYPWnKxWrlwJrVaba1cMILVKxMTE5JiSvGLFCgBA8+bN9WXnz59Hz5494enpiQ0bNhSpS+jSpUtYvHgxXn755QIveG3btgUgjbfIqlGjRjh8+LC+RaNPnz44fvw4pk2bhqtXr2LPnj14//33AUizi1555RXcunULEyZMACD9DLzyyitYu3ZtrsmArtsmq19++QUPHz7UP16zZg2io6P1v1etWrVCjRo1sHjxYoPfj02bNuk/L6DwPy8lUaNGDdjb22PdunVIT0/XlycnJ2P9+vVo2LBhju/Z0aNH4ePjU+yuVipncoyGJcqPbjS9r6+v6NWrl1i0aJEYMmSIACBeffVVg7ohISECgGjXrp348ssvxffffy8++OADUa9ePfHVV18JIYRYt26dqFmzppg4caL47rvvxIIFC0Tr1q2Fqamp2L9/v/5cL730krCyshJz584VK1asEAcOHCgw1t9++00AEDY2NqJXr145jusWyBo+fLhYsmSJGD9+vHBwcBB16tQRnTp10tcrzuyYpUuX6mcR/fDDD2Ly5MnC3t5e+Pj4iNq1axvUHThwoDAxMRHBwcFi0aJFokePHqJly5Y5XvPEiRPC2tpaVK9eXUyZMkUsWbJEfPTRR8LPz6/QceUlMzNTPPPMM8La2lrMmDFDLFq0SPj4+AgbGxtx4cIFg7q1a9fO8R50WrZsKdzc3HLMOtK5cOGCsLKyEtbW1mLKlCli8eLFYtCgQQKAeP755/X1kpKShIeHh1AqleLzzz8Xv/76q8Htv//+Mzhvo0aNxLRp08RPP/0kPv74Y+Hg4CBq164tbt26Vaj336RJkxyLsqWmpgo7Ozuxbt06fdmcOXOEUqkUAISJiYmYP3++flbICy+8IK5evWpwjrt374ratWsLS0tLMWHCBPHDDz+IkJAQ0b9/f1GtWjV9Pd3sGF9fX9G0aVPxzTffiMmTJwtzc3Ph7e0tUlJS9HX/97//CQDC399fhIaGiilTpghLS0vh6elpMGuoMD8veS1WpnuNrDOocvPZZ58JAKJ58+bim2++EV9//bVo1KiRACB+++03g7rp6enCwcFBfPLJJ/mek4wHkxAyOro/WufOnRP9+vUTNjY2olq1amLcuHEGUwZ11q5dKzp06CCsrKyElZWVaNiwoRg7dqyIjIwUQghx9epVMXLkSFG3bl1hbm4uHBwcRJcuXcS2bdsMznPhwgXx7LPPCgsLC/2UzoIkJSXp62f/gyiEdJF57733hKurq7CwsBDt27cX+/fvF506dSpxEqLVasWcOXNE7dq1hVqtFs2bNxcbNmwQw4YNy3EBj4uLE6+88oqwtLQU1apVE2+99ZY4c+ZMrq955swZ0bdvX2Fvby/Mzc1FgwYNSm0Fyvv374tRo0aJ6tWrC0tLS9GpUydx+PDhHPXySkIuXLggAIjg4OB8X+fChQuiX79+wsPDQ5iamoratWuLSZMmGVxodZ95Xrfs3/+BAwcKDw8PYWZmJtzc3MTo0aNFTExMod/7vHnzhLW1tX7KrM706dNFnTp1xP379/Vlt2/fFrt37xZ3794VQgixd+9eERsbm+e5Y2JixNixY/Xv18XFRXTt2lUsWbJEX0eXhKxYsUJMmTJF1KhRQ1hYWIiePXvmmGIrhLTCbfPmzYVarRYODg5i8ODBuSZcBf28lDQJEUJaBbdNmzbC3t5eWFhYCH9/f7FmzZoc9TZt2iQAiEuXLhV4TjIOCiFKqd2MiIjylJiYiDp16uDLL7/UT8kFpNlB7du3h0qlwp9//qlf2TW7NWvWoG/fvnkO8CzIzp070aVLF6xevRr9+vUr1jmMXUBAABQKhX7qNBk/jgkhIioHdnZ2+OCDD/DVV18ZzIwyNzfH33//DYVCgQYNGuDDDz/E7t27cf36dVy4cAG//PIL2rZti2HDhpV4bZnK7Pz589iwYYN+yjxVDGwJITIyhZkCa2dnV6RBlGT80tPTsXDhQixcuBBRUVH6cnNzc/Tt2xczZswocKfh/FSFlhCqeDhFl8jIFGYK7P/+9788N9ijisnMzAzBwcEIDg7GtWvXcPv2bZibm6NRo0b6PWqIKhu2hBAZmQcPHuDo0aP51vHx8clz7AARUUXBJISIiIhkwYGpREREJAuOCcmFVqvFnTt3YGNjw02QiIiIikAIgYcPH8LNzc1gi4bcMAnJxZ07d+Dh4SF3GERERBXWzZs3C9wglElILnS7XN68eTPfLaaJiIjIUFJSEjw8PAx2jM4Lk5Bc6LpgbG1tmYQQEREVQ2GGM3BgKhEREcmCSQgRERHJQvYkZNGiRfD09IS5uTn8/f1x6NChfOuHhoaiQYMGsLCwgIeHB959912kpqaW6JxERERU/mQdE7Jq1SoEBwdj8eLF8Pf3R2hoKLp3747IyEjUqFEjR/3ff/8dkydPxtKlS9GuXTtcvHgRw4cPh0KhwLx584p1zuISQiAzMxMajabUzknlx9TUtNi7kRIRUemQdcVUf39/tG7dGgsXLgQgrc/h4eGB8ePHY/LkyTnqjxs3DufPn8f27dv1Ze+99x4OHjyIvXv3FuucuUlKSoKdnR0SExNzHZianp6O6OhoPHr0qMjvmYyDQqGAu7s7rK2t5Q6FiKhSKegampVsLSHp6ek4evQopkyZoi9TKpXo1q0b9u/fn+tz2rVrh99++w2HDh1CmzZtcPXqVfz999947bXXin1OAEhLS0NaWpr+cVJSUp51tVotoqKioFKp4ObmBjMzMy5oVsEIIRAXF4dbt26hXr16bBEhIpKJbEnIvXv3oNFo4OzsbFDu7OyMCxcu5PqcV199Fffu3UOHDh303SGjR4/GRx99VOxzAkBISAhmzJhRqLjT09P1rSvc2bLicnJywrVr15CRkcEkhIiqJI0G2LMHiI4GXF2Bjh2B8v5zKPvA1KLYuXMn5syZg++++w7Hjh1DREQENm7ciFmzZpXovFOmTEFiYqL+dvPmzQKfU9BStGTc2HpFRFVZRATg6Ql06QK8+qr01dNTKi9PsrWEODo6QqVSISYmxqA8JiYGLi4uuT5n6tSpeO211/D6668DAHx9fZGSkoI333wTH3/8cbHOCQBqtRpqtbqE74iIiMj4RUQA/foB2UeE3r4tla9ZAwQGlk8ssv07b2ZmhpYtWxoMMtVqtdi+fTvatm2b63MePXqUowVC15QuhCjWOYmIiKoKjQaYMCFnAgI8LZs4UapXHmTtUwgODsaPP/6IZcuW4fz583j77beRkpKCESNGAACGDh1qMMi0V69e+P7777Fy5UpERUVh69atmDp1Knr16qVPRgo6p7HQaICdO4EVK6SvFW2mr6enJ0JDQ+UOg4iIimDPHuDWrbyPCwHcvCnVKw+yrhMyYMAAxMXFYdq0abh79y6aNWuGzZs36weW3rhxw6Dl45NPPoFCocAnn3yC27dvw8nJCb169cLs2bMLfU5jEBEhZaJZfxDc3YH588u2Caxz585o1qxZqSQPhw8fhpWVVcmDIiKichMdXbr1SkrWdUKMVX5znFNTUxEVFQUvLy+Ym5sX+dx59cXpxkmWZV9cQUmIEAIajQYmJpV/X8OSfh+JiCqinTulQagF2bED6Ny5eK9RlHVCOMWjFKWk5H1LTS1cX9yECUBycsHnLarhw4dj165dmD9/PhQKBRQKBX7++WcoFAps2rQJLVu2hFqtxt69e3HlyhX06dMHzs7OsLa2RuvWrbFt2zaD82XvjlEoFPjpp5/Qt29fWFpaol69evjrr78KFZtGo8GoUaPg5eUFCwsLNGjQAPPnzzeo07lzZ0ycONGgLCAgAMOHD9c/TktLw4cffggPDw+o1Wp4e3sjLCysSJ8TEVFVplAAHh7SdN3ywCSkFFlb53175ZXC9cXdugV06GBY7umZ83xFNX/+fLRt2xZvvPEGoqOjER0dDQ8PDwDA5MmT8fnnn+P8+fNo2rQpkpOT8dJLL2H79u04fvw4XnzxRfTq1Qs3btzI9zVmzJiBoKAgnDp1Ci+99BIGDx6M+/fvFxibVquFu7s7Vq9ejXPnzmHatGn46KOPEB4eXqT3OHToUKxYsQILFizA+fPn8cMPP3BFVCKiJ44cAXr3fvo4+0oFusehoeW3Xkjlb3c3IoXtY0tPL/3XtrOzg5mZGSwtLfXTlXULuM2cORPPP/+8vq6DgwP8/Pz0j2fNmoV169bhr7/+wrhx4/J8jeHDh2PQoEEAgDlz5mDBggU4dOgQXnzxxXxjMzU1NVgszsvLC/v370d4eDiCgoIK9f4uXryI8PBwbN26Fd26dQMA1KlTp1DPJSKq7M6cAbp3Bx4+BDp1At56C/jgg5xjE0NDy296LsAkpFRl70bJSqUCDhwo3Hm++cbw8bVrxQ6pUFq1amXwODk5GZ9++ik2btyI6OhoZGZm4vHjxwW2hDRt2lR/38rKCra2toiNjS1UDIsWLcLSpUtx48YNPH78GOnp6WjWrFmh38OJEyegUqnQqVOnQj+HiKgquHYN6NYNuH8f8PcH1q8HbGyAoCD5V0xlElKKCpos0rGjlGnevp37uBCFQjr+5B/5Qp+3pLLPcpk0aRK2bt2Kr7/+Gt7e3rCwsEC/fv2QXkATjampqcFjhUIBrVZb4OuvXLkSkyZNwty5c9G2bVvY2Njgq6++wsGDB/V1lEolso+hzsjI0N+3sLAo8HWIiKoiFxegTRvg+nVg0yYpAQGkhKO4g09LC8eElCOVSpqGC8jTF2dmZgZNIRYk2bdvH4YPH46+ffvC19cXLi4uuFaGzTH79u1Du3btMGbMGDRv3hze3t64cuWKQR0nJydEZ+nP0mg0OHPmjP6xr68vtFotdu3aVWZxEhFVRObmwNq1wL//AtWqyR2NISYh5SwwUJqGW7OmYbm7e9kvlevp6YmDBw/i2rVruHfvXp6tFPXq1UNERAROnDiBkydP4tVXXy1Ui0Zx1atXD0eOHME///yDixcvYurUqTh8+LBBneeeew4bN27Exo0bceHCBbz99ttISEgweG/Dhg3DyJEj8ccffyAqKgo7d+4s8uBWIqLKID4emDv3aau7qSlQvbq8MeWGSYgMAgOlProdO4Dff5e+RkWV/WCgSZMmQaVSoXHjxnBycspzjMe8efNQrVo1tGvXDr169UL37t3RokWLMovrrbfeQmBgIAYMGAB/f3/Ex8djzJgxBnVGjhyJYcOGYejQoejUqRPq1KmDLtkmu3///ffo168fxowZg4YNG+KNN95ASnHmMxMRVWCJidIg1EmTgKlT5Y4mf1ysLBdluVgZGQd+H4moMkpJkRKQffsAJydg926gYcPyjYGLlREREVUxqalA375SAmJvD2zZUv4JSFExCaEyN3r0aFhbW+d6Gz16tNzhERFVeBkZwIABwNat0ozKTZuAIqxyIBtO0aUyN3PmTEyaNCnXYwU11RERUcFGjgT++kuaCbNhA/DMM3JHVDhMQqjM1ahRAzVq1JA7DCKiSqtHD2ka7po18q/9URRMQoiIiCq4V18FunYFnJ3ljqRoOCaEiIioAvr2W2kFbp2KloAATEKIiIgqnC++AN55B3j22fz3LTN2TEKIiIgqkEWLgMmTpftvvQVYW8sbT0kwCSEiIqogfv4ZGDdOuv/JJ8AHH8gaTokxCZGJRgjsfPAAK2JisPPBA2gqwMK1np6eCA0NlTsMIqIqafVqYNQo6f7EicDMmbKGUyo4O0YGEXFxmHD5Mm6lpenL3NVqzPf2RqCTk4yRERGRMdqyRZoBo9UCr78OzJuXczf2iogtIeUsIi4O/c6eNUhAAOB2Whr6nT2LiLg4mSIjIiJj5eMDeHtLicjixZUjAQGYhJQKIQRSNJoCb0mZmXjn0iXk1vGiK5tw+TKSMjMLdb6i7D24ZMkSuLm5QavVGpT36dMHI0eOxJUrV9CnTx84OzvD2toarVu3xrZt24r9mcybNw++vr6wsrKCh4cHxowZg+QsQ7g//fRTNMu2pnBoaCg8PT0NypYuXQofHx+o1Wq4urpinK4zlIioCqlZE9i7VxoTolLJHU3pYXdMKXik1cJ6z54Sn0cAuJWWBru9ewtVP7ljR1gV8qexf//+GD9+PHbs2IGuXbsCAO7fv4/Nmzfj77//RnJyMl566SXMnj0barUav/zyC3r16oXIyEjUqlWryO9FqVRiwYIF8PLywtWrVzFmzBh88MEH+O677wp9ju+//x7BwcH4/PPP0aNHDyQmJmLfvn1FjoWIqCI6fhy4eFHaEwYAqleXN56ywCSkiqhWrRp69OiB33//XZ+ErFmzBo6OjujSpQuUSiX8/Pz09WfNmoV169bhr7/+Klbrw8SJE/X3PT098dlnn2H06NFFSkI+++wzvPfee5gwYYK+rHXr1kWOhYioojl3DnjhBSA+XtoPpk8fuSMqG0xCSoGlUonkjh0LrLc7IQEvnT5dYL2/fX3xrL19oV63KAYPHow33ngD3333HdRqNZYvX46BAwdCqVQiOTkZn376KTZu3Ijo6GhkZmbi8ePHuHHjRpFeQ2fbtm0ICQnBhQsXkJSUhMzMTKSmpuLRo0ewtLQs8PmxsbG4c+eOPmEiIqoqrlwBunUD7t0DWrWqWHvBFBXHhJQChUIBK5WqwNsLDg5wV6uR13giBQAPtRovODgU6nyKIo5M6tWrF4QQ2LhxI27evIk9e/Zg8ODBAIBJkyZh3bp1mDNnDvbs2YMTJ07A19cX6enpRf48rl27hpdffhlNmzbF2rVrcfToUSxatAgA9OdTKpU5xrRkZGTo71tYWBT5dYmIKrpbt6Q9YKKjgSZNgM2bATs7uaMqO2wJKUcqhQLzvb3R7+xZKACDAaq6dCLU2xuqMhr2bG5ujsDAQCxfvhyXL19GgwYN0KJFCwDAvn37MHz4cPTt2xcAkJycjGvXrhXrdY4ePQqtVou5c+dC+aS1Jjw83KCOk5MT7t69CyGEPpk6ceKE/riNjQ08PT2xfft2dOnSpVhxEBFVJDExUgJy/bo0E2br1so5DiQrtoSUs0AnJ6zx8UFNtdqg3F2txhofnzJfJ2Tw4MHYuHEjli5dqm8FAYB69eohIiICJ06cwMmTJ/Hqq6/mmElTWN7e3sjIyMC3336Lq1ev4tdff8XixYsN6nTu3BlxcXH48ssvceXKFSxatAibNm0yqPPpp59i7ty5WLBgAS5duoRjx47h22+/LVZMRETG7OFDaQzIxYtArVrA9u2Ai4vcUZU9JiEyCHRywrVnnsEOPz/83qgRdvj5IeqZZ8plobLnnnsODg4OiIyMxKuvvqovnzdvHqpVq4Z27dqhV69e6N69u76VpKj8/Pwwb948fPHFF2jSpAmWL1+OkJAQgzqNGjXCd999h0WLFsHPzw+HDh3CpEmTDOoMGzYMoaGh+O677+Dj44OXX34Zly5dKlZMRETGzNpaGgfi4iIlIMWYlFghKURRFpuoIpKSkmBnZ4fExETY2toaHEtNTUVUVBS8vLxgbm4uU4RUUvw+EpGxEQKIjQWcneWOpGTyu4ZmxzEhRERE5USjAfbskQaeVq8OHDoETJokTcNVKCp+AlJUTEKoyJYvX4633nor12O1a9fG2bNnyzkiIiLjFxEBTJggzYDJav164OBBeWKSG5MQKrLevXvD398/12OmpqblHA0RkfGLiAD69ZO6XLI7dEg6HhhY/nHJjUkIFZmNjQ1sbGzkDoOIqELQaKQWkLxGYCoUwMSJ0qqolWlfmMLg7Jhi4njeio3fPyIqL3v25OyCyUoI4OZNqV5VwySkiHTdDY8ePZI5EioJ3cqtqqr2bwcRlbvo6NKtV5mwO6aIVCoV7O3tERsbCwCwtLQs8vLpJC+tVou4uDhYWlrCxIS/AkRUtlxdS7deZcK/wMXg8mQZO10iQhWPUqlErVq1mEASUZnZvx+4dAkYPBhwdwdu3859XIhCIR0vxD6olQ6TkGJQKBRwdXVFjRo1DDZdo4rDzMxMv68NEVFpysgAZs4E5swBTE2BFi2A+fOl2TEKhWEiovs/KDS06g1KBZiElIhKpeKYAiIi0ouMBIYMAY4ckR4HBQEeHtKOuGvW5FwnxN1dSkCq4vRcgEkIERFRiQkBLF4MvPce8PgxUK0a8MMPQP/+T+sEBkrTcHUrprq6Sl0wVfl/WSYhREREJSAE0Lcv8Oef0uPnnwf+9z+gZs2cdVUqoHPncg3PqLFTnIiIqAQUCqBNG0CtlsZ+bN6cewJCObElhIiIqIgePgTi4oA6daTHH34odb3UqydvXBUNW0KIiIiKYP9+oFkzoHdvIDVVKlOpmIAUB5MQIiKiQsjIAKZNAzp0AK5elVpDrl+XO6qKjd0xREREBYiMBF57DTh8WHo8ZAiwcCFgZydvXBUdW0KIiIjyIATw/fdA8+ZSAmJvD6xcCfz6KxOQ0sCWECIiojwIAYSHS2t/dO0K/PyztMAYlQ4mIURERNlotYBSKd2WLQP++AMYN056TKWHHycREdETycnAG28A77zztKxWLekxE5DSx5YQIiIiSFNvX3sNuHJFSjjeeQeoX1/uqCo35nVERFSlZWQA06dLU2+vXJFaPv79lwlIeZA9CVm0aBE8PT1hbm4Of39/HDp0KM+6nTt3hkKhyHHr2bOnvk5MTAyGDx8ONzc3WFpa4sUXX8SlS5fK460QEVEFc/Ei0L49MHOmNA5kyBDg1CmgUye5I6saZE1CVq1aheDgYEyfPh3Hjh2Dn58funfvjtjY2FzrR0REIDo6Wn87c+YMVCoV+j/ZplAIgYCAAFy9ehV//vknjh8/jtq1a6Nbt25ISUkpz7dGRERGQKMBdu4EVqyQvmo0T4+lpwPdunHqrayEjNq0aSPGjh2rf6zRaISbm5sICQkp1PO/+eYbYWNjI5KTk4UQQkRGRgoA4syZMwbndHJyEj/++GOh40pMTBQARGJiYqGfQ0RExmXtWiHc3YWQJtpKN3d3qVwnPFyIrl2FuHlTvjgrm6JcQ2VrCUlPT8fRo0fRrVs3fZlSqUS3bt2wf//+Qp0jLCwMAwcOhJWVFQAgLS0NAGBubm5wTrVajb179+Z5nrS0NCQlJRnciIio4oqIAPr1A27dMiy/dQt45RXpOCBtOrd1K9f+kItsSci9e/eg0Wjg7OxsUO7s7Iy7d+8W+PxDhw7hzJkzeP311/VlDRs2RK1atTBlyhQ8ePAA6enp+OKLL3Dr1i1ER0fnea6QkBDY2dnpbx4eHsV/Y0REJCuNBpgwQWr7yMv48U+7ZhSK8omLcpJ9YGpxhYWFwdfXF23atNGXmZqaIiIiAhcvXoSDgwMsLS2xY8cO9OjRA8p8JnhPmTIFiYmJ+tvNmzfL4y0QEVEZ2LMnZwtIdnfuSPVIXrKtE+Lo6AiVSoWYmBiD8piYGLi4uOT73JSUFKxcuRIzZ87Mcaxly5Y4ceIEEhMTkZ6eDicnJ/j7+6NVq1Z5nk+tVkOtVhfvjRARkVHJp+G7WPWo7MjWEmJmZoaWLVti+/bt+jKtVovt27ejbdu2+T539erVSEtLw5AhQ/KsY2dnBycnJ1y6dAlHjhxBnz59Si12IiIyHvfvS7Nf/vlHeuzqWrjnFbYelR1ZV0wNDg7GsGHD0KpVK7Rp0wahoaFISUnBiBEjAABDhw5FzZo1ERISYvC8sLAwBAQEoHr16jnOuXr1ajg5OaFWrVo4ffo0JkyYgICAALzwwgvl8p6IiKhsCQGcPg1s3Cjd9u+X1vh4/nmge3egY0dpoGleXTIKhXS8Y8fyjZtykjUJGTBgAOLi4jBt2jTcvXsXzZo1w+bNm/WDVW/cuJFjLEdkZCT27t2LLVu25HrO6OhoBAcHIyYmBq6urhg6dCimTp1a5u+FiIjK3rvvAmvXAtmH7jVpAjzzjHRfpQLmz5dmxwCGA1R1g1BDQ6V6JC+FEPmNH66akpKSYGdnh8TERNja2sodDhFRlRQVBfz3HzB48NOyF1+Uul3MzYGuXYGePYGXXgJq1875/IgIaZZM1hYRDw8pAQkMLPPwq6yiXEOZhOSCSQgRUfnLyAD27XvazXL+vFR+8+bTdTx27gQePQK6dAEsLAo+p0YjzYKJjpbGgHTsyBaQslaUayh30SUiolJV1Av/nj3AggXAli1A1rUiVSppX5f4+KdJSOfORYtFpSr6c6j8MAkhIqJSk1sXiLu7NEYjMFAaQHrsmJSc1KwpHb91C1izRrrv5AT06CF1sbzwAlCtWvm/Byo/TEKIiKhU6JZKz97Jf/u2tFR6ly5SF8vdu8Ds2cBHH0nHu3cHpk6Vxne0asXukqqESQgREZVYfkul68p27JC+WltL4zp0HByAXNaepCqASQgRERVbUhJw4gQQHl7wUukA8PXXwLhxABepJoBJCBERFcG2bcDhw8Dx49Lt8uWiPd/NjQkIPcUkhIioEijNqahCSGt0HD8unW/cuKfH3n9favnIysNDuv33X8Hn5lLplBWTECKiCq6gGSkFuXgROHjwaevGiRNAQoJ0zMwMePNN6SsA9OoFNGwItGgBNG8ONGsGODpKSZCnpzQINbdxIVwqnXLDJISIqALLb0ZKv37S1FddIvL4sbTnysmTwOuvP13C/OOPn06R1TE1lZZCb94cSE6WBo8CeQ8gzbpUukLBpdKpcJiEEBFVUIWZkTJqlJSonDgBXLggPQeQljyvU0e636GDNG22efOnt8aNn7Z+FFZgoJTM5NYqw6XSKTdctj0XXLadiCqCnTultTeKwslJSjLmzpVaOsoCl0qv2rhsOxFRJaTVAleuPB278fffhXtev37A0KHSOA43t6fdI2WFS6VTYTEJISIyQunpwLlzUkuCs7NUtmwZMHJk0c81diyTAjJOTEKIiIqptLodkpOlwaK6Fo7jx4GzZ6VEZNEiYMwYqZ6fn7TGhq+v1KXi5ycNFI2L44wUqpiYhBARFUNxp8XGxQGZmU/Xyzh4EGjbNvckws4OSEl5+rhZM+DhQ2nmio6rK2ekUMXFgam54MBUIspPXtNidRf+NWuAvn2B69cNWzeOH5emzk6cCHzzjVQ3MRGwt5fGamSdndK8ubTuRmHGb+SWEHl4cEYKyaMo11AmIblgEkJEedEtypXXPikKhZRQpKQ8XfAru8GDgd9+e/r43j1pwa+SxsUZKWQMODuGiKgMaLXA6tX5b9QmhNTaYW8vdZv4+Bi2bvj5ATY2hs8paQICcEYKVUxMQoiIstFqgRs3gPh4oGVLqUwIqYvjzp3CnWP6dGlAaVEX/CKqSpiEEFGFUtrdDtevA2fOSLNRzp6VpsWePy91pzRoIK0yCkjdLLVqATExT1cdzU+zZkxAiArCJISIKozizkjRaqVk4+xZKYkYNerpsb59pQGj2ZmZAZaW0nOVSqnsjz+kbhZvb27URlQamIQQUYVQlI3adu8G9u+XWjXOnpVaNh49ko6ZmQHDhgEmT/76NW8uTZn18ZH2S/HxkW516z6to6NbNIwbtRGVDs6OyQVnxxAZl4JmpADSeI2oKOni379/zl1hzcykLegbNwa++w6oVq1kMXFaLFHuODuGiCqVPXvyT0AA4OZNqV7nzkC3blIrhq5Vo3Hj3Fs2SiIwEOjTh9NiiUqCSQgRGaXUVClpMDGRLvKFoav31lvSraxxWixRySjlDoCISOfKFWDhQqBnT8DBAdi1SyrXLXFekMLWIyLjwJYQIpJNaqqUaGzaJN0uXjQ8vmsX0LWr1M3h7s4ZKUSVDVtCiKhcpaY+vX/hAvDii9Jsk4sXpa6XTp2Azz+XdpWdMUOqp1JJdYCce6nIPiPlxAmgRw/pKxEVCVtCiKhMPX5s2Nrh7w/8+qt0zM8PaN0aaNpUuo536ybtHJubwEBpxktu64TIOiNl7Vpg82bpjTRrJlMQRBUTp+jmglN0iZ4qzgqlV648TTp27JASER03NymJKMzusKUVT5lq1kxqtmnWLPdVz4iqGE7RJaJSUdgVSjMzDae/vvKKdF3WqVlTaunQtXYUNwEBjGxGSkzM0zd64gQQGwvUqCFrSEQVCZMQIspVQSuUfvuttKT5pk3S6qS3bgFWVlKdPn2k5c179ABeeglo0qRkiYfR+uefnI9fe02eWIgqIHbH5ILdMVTVFWaF0uz+/ltKOqoS7YABwNq1UGo00KpUQL9+UK5cKXdYRLJidwwRlYjBCqVKAfgmANXTgXgz4LQ9oJWaNfz8gFdflZKPJk3kirYM3b4tdbnk4t/799F6wwbYPNlSV6nRIGnDBhzZtg3POTjkfj5nZ6lviogAMAkhoieEkKbMbt8OLFv2pLBjHDDuMlAj7WnFWDWw0BvY44QPPwQGDZIl3PIxdCjw77+5HnoOgDZbH5P1o0d47vnn8z5f167Atm2lGCAVRCME9iQkIDo9Ha5mZuhobw9VpewbrJiYhBBVYcnJ0rTX7dula+2dO1kOdowDZpzN+STHNKl8ug9cXZ3KLVZZjB4NHDsGJCTkeliZrTc7+2MD9vbls5Y86UXExWHC5cu4lfY0iXZXqzHf2xuBTpX8Z7eC4JiQXHBMCFVWcXHSBA4fH+lxQgJQvbo0wBQAzM2B9u2BTl0EPm1wANrqaUBu/zRqAdUDNR71eQZmJpX8v8rYWCkZWbcOQqGAoih/MhUKqYmpb19g8WLOnClHEXFx6Hf2LLJ/t3Q/rWt8fJiIlBGOCSEiAEBSkjS+Y/t26XbqlJRk7N0rHbe3B4YPB1xcpJ6Cdu2kRGTngwRoT6blfWIloKmeho+uXcHzDg6opVbDQ62GdWluU5uHsmxef6TR4PLjx7j8+DEuPfl6+fFjXJo0Ce18fbF43jzYPnoEE13Wlo9MpRKPrazw12efAQMGoKmlJRpotTBTVv6FquXuAtEIgQmXL+dIQABAQEpEJl6+jD6OjuyakRlbQnLBlhCSW0kX5PrqK2DdOuDQIelcWbVoIZVnP9+N1FTsSkjA7sREbLh3D3czMoocdzUTE9RSq1HL3BweT77qEpRa5uZwMzODSQkuwqXRvJ6cmYkrqalScvHokUHCcSc9Pd/nej98iLCQEHTcvz/XBiIdAWBTmzYYPnky4qpV05ebKhRoZGmJptbWaGplpf/qYmYGRQkvhnJf+HXKuwtEIwSSMjORmJmJhCe3vYmJmHrtWoHP3eHnh85Zvj9UOtgSQlSBFXaBMEBKMI4eBQ4cAMaPf7oWx3//SWt3AEDdusBzz0ktHV26SD0CQghcfvQYuxITsTshAbsSEnA9LZ+Wjzz429ggRavFjdRUJGk0eJCZiQeZmTiZkpJrfSWAmlmSEoOE5cn9aiYmuV6Q82pev52Whn5nzxo0rydnZuZo0dB9jS4g0XAwMYG3hQXqWVjA+8mtnqUlvC0sUN3UFNqDB6E5dAgm2bO7LDRKJZp06oRZ/v44lZyMUykpOJWcjCSNRrqf7fNxNDU1SEqaWlujsaUlLAqZeRrL2IeifI8A6ecwVatFQpYkIlGjMXyc/Wu24w/z+T4UpKCfBSp7bAnJBVtCSC55LRCmuyavXg00bPi0e2XXLiAxUTp2+bKUcADA1q3AzZtS4lG7tvTH/tyjR9j9pKVj15P/mLNSAWhpY4Nn7e3R3tYWYy9dQnR6eq5N2gpIF7moZ57R/7edmJmJm6mpuJGWhhupqbiZlmZw/2ZaGjIL8efGUqnMkaC4m5lhclQU4vJpnbFSKtHM2hpXUlNxt4CLS3VdovEkuciadDiYmuYfYLNmECdPFtgSosi2jLsQAjfS0nAqORmnnyQlp1JSEPnoEXLr3FECqG9pmSM5qaVWGyRpco190AiBZI0GSU8SgQcZGQg4exb38vkeqRUKNLGyQlKWRCK9lC5BFkol7E1MYGdiAgWA848eFfgctoSUjaJcQ5mE5IJJCMmhMAuEKZVPB5Hq2NlJy5jPnv10wKlGCJxKTtYnHHsSE3NcHMwUCrSxtcWzdnboZG+Ptra2sMkypkN3cQNgcIEr7sVNIwRi0tOl5CR7svLkcX5JRlE5mprmbNF48rVaQYlGXu7elfrHstAqFFAKof+ao76zc76nfKzR4PyjRwYtJidTUvK8mNuqVPqkpImVFT69dg2xedTNniwKIfBYq0VSZiaSsiQQBvcLceyhRoPkErRAZKcEYPckgbA3MYGdSqVPKLJ+tc+jjp2JicFYG40Q8DxwALfT0nJNonWGOTtjrrc3qhf354FyxSSkhJiEkBx27pS6SwDku0CYmZm03X3XrlI3S4sWgFahxbHkZH3Xyt7ERCRmu0hYKJVoa2uLTvb2eNbODv62tgU29+fWzO+hViO0jJr5H2s0uJWtBeVGaioOPnyIM3l08WQ1zs0Nw1xc4G1hAfuyuLAsWyaN5H1CqFTItLbGhVGj0DAsDCbJyVBk/dyXLZPWGiki8SRhO5WlxeRUcjLOPXqEjGL8ya5haor0J2MnCh5SWzSmCgVsVSooFYpCJZHve3igd/XqBgmFtUpV4jEx2eWXRGd97Ghqirl16+I1Z+dSj6GqYhJSQkxCqiY5B/YJAYSEAB9/jAIXCFu2DAgarMHhhw/1A0n/S0xESrYmEhuVCu3t7NDJzg7P2tujlY1NsWZmGMOAx50PHqBL1h3x8lDmzesDBkgLqwiRc+ptlqm8UCikW//+QCku456h1SLy0SN9UrL5/v08x9/kRwGpRcXGxAS2KhVsn3y1yXq/kMfUT36mjOZ7lEV+SbSrmRnevHhRn9x2tbfH9/Xro56lZbnEVpkxCSkhJiFVj1wD+5KTga+/lq5TkZEwXCAs63Ve++TxLif4dU7HBUUS0rL96lYzMcGzTxKOTvb28LOyKtFMFGNSUPN6bmNUSl1mprSoSlKSNLf5hx+AoKCc9cLDpUXJEhIAW1vg/v2iTW0qgsJe+L+vVw9dqlXTJxNWZdDyYBTfozziyiuJztBqMffmTcy4fh2pWi3UCgU+qV0bH9SqVSWmUpcVJiElxCSkainLgX2ZWi3ShUD6k69pWi0SUgRMzLVI02rxKEOgR28tkh4JqKw00Ey6ANhm5r5AWDbOpqb6hONZOzv4WFlBWYmbk0t7jEqRPXwIPPss4OVV8MJjulaRa9ek0cM2NmUSkrFd+GX/HhXT1ceP8fbFi9jy4AEAoJGlJZbUr48O9vbyBlZBMQkpISYhVYfuj/itfKanWiqV6OHgkCOZyPE4l2Ol3f8OAO+5u+MNNzfUt7Cocn3Y5T1GJQeNpmitGkWtXwzGduGX/XtUTEIIrIiNxbuXL+sH+r7h6oov6tQp/kDmKopJSAkxCak6djx4gOcK0ZxdatIVQKYSdlYKWJgqoVYoYKZUwkyhQLJGU6i1On5v1AiDCphxUZkZwxgVY2NsF/6K/D26n5GBD69exU/R0QCkQb2h3t4YWKNGlUv6i4tJSAkxCan87mdkYNndu/jqxg1EF2JE/wgXF7SztdUnDGqlUn/fTGmYTKiVSqxZqcSnnyiQnqLUJx4tmynw6iAFgoKkxceyM8aBfVRxVOQLvzHak5CAty5e1K830r1aNXxfvz68LCxkjsz4MQkpISYhlZMQAgeSkrD4zh2sio3NMbAzP/ld+DMypMXBPD2Bxo2lsr17paXWGzWStrofOBCoVy//1zC2/n2iqi5Nq8WXN25g9vXrSBMCFkolpnt6ItjdHaYcuJqnolxD+SlSpZeUmYnvbt+G35EjaHf8OH6JiUGaEGhmbY3v69WDm5lZnuNAFZCatTtmG6Cm0Ujrerz1lrT5W8+ewMKFT4+3awecPAmcPQtMnVpwAgIAKoUC87299a+bPQ4ACPX2ZgJCVE7USiWmenriVOvW6GJvj8daLSZfvYqWR4/igG6pYioR7h1Dldaxhw+x+M4d/B4To19Dw0KpxMAaNTDazQ2tbWygUChQw8wM/c6ezbGIUfYLvxDAkSPAihXAqlXAnTtP6zo7A1m73pVKoGnToscc6OSENT4+uU4XNvaBfUSVVX1LS2z388MvMTF47/JlnE5JQbvjxzHazQ0hderArhx2j66sZG8JWbRoETw9PWFubg5/f38cOnQoz7qdO3eGQqHIcevZs6e+TnJyMsaNGwd3d3dYWFigcePGWLx4cXm8FTICKRoNlkZHo83Ro2h59Ch+jI5GilaLRpaWmO/tjdtt22Jpw4ZoY2urH2QW6OSESQ99oIxXG5xLGa/GpIeGMwv69we++UZKQOzsgJEjpa6YW7eAGTNK5z0EOjnh2jPPYIefH35v1Ag7/PwQ9cwzTECIZKRQKDDMxQUX2rTBcBcXCADf37mDRocOYU1sLDiyoXhkTd9WrVqF4OBgLF68GP7+/ggNDUX37t0RGRmJGrnMwY+IiEB6lo2p4uPj4efnh/79++vLgoOD8e+//+K3336Dp6cntmzZgjFjxsDNzQ29e/cul/dF5e9sSgp+uHMHv9y9q1+u3FShQD8nJ4x2c0NHO7s8R7ZHRABf93OCUDgaLJWuOW2Pr7QKtFolrUmlUEhJx/nz0hiPF18E1OpcT1liKoWCg0+JjJCjmRn+17Ahhjo7462LF3Hp8WP0P3cOPR0csKh+fdQ2N5c7xApF1oGp/v7+aN26NRY+6UzXarXw8PDA+PHjMXny5AKfHxoaimnTpiE6OhpWVlYAgCZNmmDAgAGYOnWqvl7Lli3Ro0cPfPbZZ4WKiwNTK4Y0rRZr4+Kw+M4d7MnSP1vX3BxvurlhuIsLapiZ5XuOwmwa5+go7UNWxss9EFEFk6rRIOTGDYTcuIEMIWCpVGKWlxfeqVmz0qxWXBwVYmBqeno6jh49im7duj0NRqlEt27dsH///kKdIywsDAMHDtQnIADQrl07/PXXX7h9+zaEENixYwcuXryIF154Ic/zpKWlISkpyeBGxuvSo0d4/8oVuO/fj8Hnz2NPYiJUAAIdHbGlaVNc9PfHB7VqFZiAAMCePfknIABw755Uj4goK3OVCjO8vHCyVSt0tLPDI60W7125gjbHjuEIryOFIlt3zL1796DRaOCcbdElZ2dnXLhwocDnHzp0CGfOnEFYWJhB+bfffos333wT7u7uMDExgVKpxI8//ohnn302z3OFhIRgRml16FOZyNBq8Vd8PBbfuYNtT5ZWBqQBm2+6umKUqyvcitA3otVKg0efrEdUoMLWI6Kqp5GVFXY2a4b/3b2L969cwfHkZPgfO4bxNWtilpcXbDhwNU8V9pMJCwuDr68v2rRpY1D+7bff4sCBA/jrr79Qu3Zt7N69G2PHjoWbm5tBq0tWU6ZMQXBwsP5xUlISPDw8yjR+uRjbgkYFxXMjNRU/Rkfjp+ho3H0yHkgBoIeDA0a7uaGHg0Ohmz3T04H164GffgJcXYGlS6WvhVHYekRUNSkVCoxydUWv6tURfPkylsfGYv7t21h77x4W1quHPo6OAIzvb7DcZBsTkp6eDktLS6xZswYBAQH68mHDhiEhIQF//vlnns9NSUmBm5sbZs6ciQkTJujLHz9+DDs7O6xbt85gxszrr7+OW7duYfPmzYWKrbKOCZFrp9iixvNN3bqwUKmw+M4d/B0fr99/xdnUFKNcXfGGqys8i7Bq4fnzQFgY8MsvQFycVGZpKe0xZm4ujQm5fVvamT07hUJa3TQqimNCiKjwtty/j7cvXsTV1FQAQICjI15ycMDM69eN5m9wWakQY0LMzMzQsmVLbN++XV+m1Wqxfft2tG3bNt/nrl69GmlpaRgyZIhBeUZGBjIyMqDM9p+xSqWCVlsWW4lVHLpNrrJv1HY7LQ39zp5FhO7qLHM8t9LS0P/cObx8+jQ2PElAutrbY3XjxrjRti1m16lT6AQkIkJaNKxxY2DuXCkBcXUFpkyRFhKzspISi/nzpfrZ/xnRPQ4NZQJCREXzgoMDzrRujSm1asFEocAf9+7hzYsXjeZvsLGQtTsmODgYw4YNQ6tWrdCmTRuEhoYiJSUFI0aMAAAMHToUNWvWREhIiMHzwsLCEBAQgOrVqxuU29raolOnTnj//fdhYWGB2rVrY9euXfjll18wb968cntfxkYjBCZcvpzrUuACUvfG+EuX0MrGBsonZfqbEHk+1uZzLLfHuvqZQuDtixdzjUdHAWBizZoYXbMm6ltaFup9CiHddDno+fPA/v1SAvHyy8CoUUCPHkD27tnAQGDNGmDCBMNBqu7uUgISGFiolyciMmChUmFOnToIcnKC/7FjSM+luVX3N3ji5cvo4+hY5bpmZE1CBgwYgLi4OEybNg13795Fs2bNsHnzZv1g1Rs3buRo1YiMjMTevXuxZcuWXM+5cuVKTJkyBYMHD8b9+/dRu3ZtzJ49G6NHjy7z92Os9iQk5LtVvQBwJz0dtQ8cKL+gCiAA9HZ0LFQCcu8e8Ouv0liP6dOlNT0AYPhwKeEYNkxaWj0/gYFAnz7SLJjoaKnFpGNHtoAQUcklZGbmmoDoCAA309Lw2bVrGO7qilpqdZXZsZcb2OWiso0JWRETg1fPny+wngLSIlmKJ/eVWe4rIK0YmNfjotR9pNXiXiF2rs1vy3qNBti2TUo8/vxT2kQOkPZw2bChwFMTEZWbwv4N1nEyNUUrGxu01t1sbeFciCUHjEVRrqHFaglJTEyERqOBg4ODQfn9+/dhYmJSKS7clYlrIX94/y2nLeILu2V9bnFrtcDMmdLMlps3n5a3aiV1twwaVJqREhGVXGH/BtczN0dUWhriMjKw6f59bLp/X3/MXa1+mpTY2KCVjQ3sTU3LKuRyU6wkZODAgejVqxfGjBljUB4eHo6//voLf//9d6kER6Wjg50drJRK/SZu2em2iM++U2xZ6WhvD3e1usAt63XxaDRPu0WUSmDLFikBqVYNGDJESj78/MoldCKiIivs37zz/v7I0GpxMiUFh5OScPjhQxx5+BDnHz3CrbQ03EpLw7p79/TP87awMEhMmtvYwKoIfcjGMF24WN0xDg4O2LdvHxo1amRQfuHCBbRv3x7x8fGlFqAcKlN3jBAC71y+jIW3b+d6XPfjtsbHp1yniEXExeGVM2elztCsw360UlBrm/ig3l0nhIUB4eHA6dOAbhzyP/8A9+8DfftKU2yJiIydbkYgkPtu3fn9DX6YmYljyckGiYlu6m9WSgA+Vlb6lpLWNjZoam0Ns1zWUirLJRvKvDsmLS0NmZmZOcozMjLw+PHj4pySyoAQAu9mSUDGuLnhr/h449gifo8TMN8HGHsZqJFl0Ow9NbDQGx/cccKVK0+Lw8OBt9+W7nfvXr6hEhGVVKCTE9b4+OR64S/ob7CNiQk62dujU5bW6viMDBx5+FCfmBx++BDR6ek4nZKC0ykpWHr3LgDATKGAn7W1QWJy4dEjBJ07l6NVRjdduDz/KS1WS0iXLl3QpEkTfPvttwblY8eOxalTp7Cngm+0URlaQoQQeP/KFcx9Muf0x/r18bqbm1E0vxlsGqcUBjvX4rQ9oJXiUamkGSujRkmJB2eqEFFFV5Z/g++kpUkJSZYWk/u5NBgogDyXSNB1DUU980yx4yrKNbRYSci+ffvQrVs3tG7dGl27dgUAbN++HYcPH8aWLVvQsWPHYgVuLCp6EiKEwOSrV/Hlk5Gbi+vXx1tubjJH9dTOnUCXLgXXW7uWa3QQERWXEAJRqakGicmhpCSkFuKyv6MEExXKfMXU9u3bY//+/fDw8EB4eDjWr18Pb29vnDp1qsInIBWdEAIfR0XpE5BF9eoZVQICFH4zuHyWNiEiogIoFArUsbDAgBo18LW3N3Y1b46fGjYs1HOjn+zVVdaKvVhZs2bNsHz58tKMhUpICIFp164h5MYNAMC33t4YU7OmzFHlxE3jiIjkUbOQ04ULO624pIqVhNx4cpHLS61atYoVDJXMjGvX8Nn16wCAUG9vjHN3lzmi3BW0RYJu0zg2qhERla6iLpFQ1oqVhHh6eua7pKxGoyl2QFQ8s65dw4wnCcjcunUxwUgTkF9/BZ5sDQRASjiydk9y0zgiorKjUigw39sb/c6ezTFAVXdVD/X2LrcJC8UaE3L8+HEcO3ZMfzt48CAWL16M+vXrY/Xq1aUdIxVgzvXrmHbtGgDgizp1EOzhIW9Aefj+e2DoUGl2zPDhwOrVQPbeInd3aTM5DkglIiobuunCNdVqg3J3tbrc14wq1b1jNm7ciK+++go7d+4srVPKoiLNjvnixg1MvnoVADDHywtTateWOaLcffUV8MEH0v1x44D586XVTzUabhpHRCSHspouXOaLleWlQYMGOHz4cGmekvIx9+ZNfQIyy9PTaBMQIYAnYWLKFGD27KfdLioV0LmzbKEREVVZKoWiXPYLy0+xkpCkpCSDx0IIREdH49NPP0W9evVKJTDK3zc3b2LSkyVFP/X0xCeenvIGlA+FAli0COjRA+jdW+5oiIjIWBQrCbG3t88xMFUIAQ8PD6xcubJUAqO8Lbh1C8FPEpCptWtjuhEmIBoN8MMPwBtvAKamUtcLExAiIsqqWEnIjh07DB4rlUo4OTnB29sbJial2sND2Sy6fRsTLl8GAHxUqxZmGGECkpEhDUBduRLYtw/gcjJERJSbYmUMnTp1AgCcO3cON27cQHp6Oh48eICLFy8CAHrzX94ysfj2bYy7dAkA8KGHBz7z8sp3qrQcUlOBAQOAv/4CTEyAgAC5IyIiImNVrCTk6tWrCAwMxKlTp6BQKKCbYKO7IHKdkNK35M4dvP0kAZnk4YGQOnWMLgFJTpaSju3bAXNzae+Xl16SOyoiIjJWxVonZMKECfD09ERsbCwsLS1x5swZ7N69G61atarw03ON0dLoaLz1pJXpXXd3fGmECUhCgrTT7fbtgLU1sGkTExAiIspfsVpC9u/fj3///ReOjo5QKpVQqVTo0KEDQkJC8M477+D48eOlHWeVtezuXbweGQkAeKdmTcytW9foEhAhgD59gP/+A+ztgc2bAX9/uaMiIiJjV6yWEI1GAxsbGwCAo6Mj7ty5AwCoXbs2Ip9cMKnkfr17FyMuXIAAMNbNDaHe3kaXgADSFNzp04FatYCdO5mAEBFR4RSrJaRJkyY4efIkvLy84O/vjy+//BJmZmZYsmQJ6tSpU9oxVkm/x8Rg+JMEZLSbG76tV8/oEhAhni469txzwMWLQLZVgImIiPJUrJaQTz75BFqtFgAwc+ZMREVFoWPHjvj777+xYMGCUg2wKloZE4PXzp+HFsAbrq5YZIQJyIULQPPmwLlzT8uYgBARUVGU2t4x9+/fR7Vq1YzuYlkccu4dszo2FoPOnYMGwCgXFyxp0ABKI/tMT5wAXngBiIsDunUDtm6VOyIiIjIWRbmGFqslJDcODg6VIgGR09q4OH0CMtxIE5ADB4AuXaQEpEULYMUKuSMiIqKKqtSSECqZdXFxGPgkAXnN2Rk/GWEC8u+/UstHQgLQvr302NFR7qiIiKiiYhJiBP66dw9B584hUwgMrlED/2vYsFS2Uy5NGzdK636kpEiJyD//AHZ2ckdFREQVGZMQmW2Mj0e/s2eRKQQG1qiBn40wARECmDcPSEuTNqFbvx6wspI7KiIiquiYhMhoU3w8As+cQYYQCHJywq8NG8JEaXzfEoUCiIgApk0D1qyRlmQnIiIqKeO74lUR/9y/j75nziBdCLzi6IjfGjUyugTk4MGn9+3sgBkzAFNT+eIhIqLKxbiuepWURgjsfPAAK2JisPPBA/wTH4+AM2eQJgQCHB2xonFjmBpZAjJnDvDMM8DXX8sdCRERVVbFWjGVCi8iLg4TLl/GrbS0HMd6V6+OVUaWgAgBfPQR8Pnn0uPkZHnjISKiyotJSBmKiItDv7NnkddqcK86O8PMiBIQrRZ45x1g0SLp8ddfA++9J29MRERUeRnPFbCS0QiBCZcv55mAKAC8f+UKNKWzYG2JZWYCI0dKCYhCAfzwAxMQIiIqW0xCysiehIRcu2B0BICbaWnYk5BQbjHlRQjg1VeBZcsAlQr49VfgzTfljoqIiCo7JiFlJDo9vVTrlRaNBti5U1pufedO6bFCIa2AamYmTcEdPLhcQyIioiqKY0LKiKuZWanWKw0REcCECcCtW0/L3N2B+fOl8t69AS+vcguHiIiqOLaElJGO9vZwV6uR19qnCgAeajU62tuXSzwREUC/foYJCADcvi2VR0QwASEiovLFJKSMqBQKzPf2BoAciYjucai3d7ks0a7RSC0duY2B1ZVNnCjVIyIiKi9MQspQoJMT1vj4oKZabVDurlZjjY8PAp2cyiWOPXtytoBkJQRw86ZUj4iIqLxwTEgZC3RyQh9HR+xJSEB0ejpczczQ0d6+XDepi44u3XpERESlgUlIOVApFOhcrZpsr+/qWrr1iIiISgO7Y6qAjh2lWTB5Nb4oFICHh1SPiIiovDAJqQJUKmkabm50iUloqFSPiIiovDAJqSICA4HlywE7O8Nyd3dpgbLAQHniIiKiqotjQqqQQYOAoCBpFkx0tDQGpGNHtoAQEZE8mIRUMSoV0Lmz3FEQERGxO6bKOH9e2pjuwQO5IyEiIpIwCakifv4ZGDoUGDtW7kiIiIgkTEKqACGkvWEAoG9feWMhIiLSYRJSBZw5A1y+DKjVQI8eckdDREQkYRJSBehaQbp3B6yt5Y2FiIhIh0lIFcCuGCIiMkZGkYQsWrQInp6eMDc3h7+/Pw4dOpRn3c6dO0OhUOS49ezZU18nt+MKhQJfffVVebwdo3LlCnDqlDQ1t1cvuaMhIiJ6SvYkZNWqVQgODsb06dNx7Ngx+Pn5oXv37oiNjc21fkREBKKjo/W3M2fOQKVSoX///vo6WY9HR0dj6dKlUCgUeOWVV8rrbRmN//6TvnbuDFSvLmsoREREBhRCCCFnAP7+/mjdujUWLlwIANBqtfDw8MD48eMxefLkAp8fGhqKadOmITo6GlZWVrnWCQgIwMOHD7F9+/ZCxZSUlAQ7OzskJibC1ta28G/GSEVHA/HxQJMmckdCRESVXVGuobK2hKSnp+Po0aPo1q2bvkypVKJbt27Yv39/oc4RFhaGgQMH5pmAxMTEYOPGjRg1alSe50hLS0NSUpLBrTJxdWUCQkRExkfWJOTevXvQaDRwdnY2KHd2dsbdu3cLfP6hQ4dw5swZvP7663nWWbZsGWxsbBCYzw5tISEhsLOz0988PDwK/yaMmFYrdwRERER5k31MSEmEhYXB19cXbdq0ybPO0qVLMXjwYJibm+dZZ8qUKUhMTNTfbt68WRbhlruXXgKefx44cULuSIiIiHKSdQM7R0dHqFQqxMTEGJTHxMTAxcUl3+empKRg5cqVmDlzZp519uzZg8jISKxatSrfc6nVaqjV6sIHXgHExwPbtgEaDWBjI3c0REREOcnaEmJmZoaWLVsaDBjVarXYvn072rZtm+9zV69ejbS0NAwZMiTPOmFhYWjZsiX8/PxKLeaKYv16KQHx8wPq1pU7GiIiopxk744JDg7Gjz/+iGXLluH8+fN4++23kZKSghEjRgAAhg4diilTpuR4XlhYGAICAlA9j3mnSUlJWL16db7jRSoz3QJl+QyFISIikpWs3TEAMGDAAMTFxWHatGm4e/cumjVrhs2bN+sHq964cQNKpWGuFBkZib1792LLli15nnflypUQQmDQoEFlGr8xevgQ0H00TEKIiMhYyb5OiDGq6OuEhIcDAwYA9eoBkZGAQiF3REREVFVUmHVCqGxk7YphAkJERMZK9u4YKn0vvQQ8eABUwVXqiYioAmF3TC4qencMERGRXNgdQ0REREaPSUglkpkJLF4M3L4tdyREREQFYxJSiezeDbz9NtC8OfeNISIi48ckpBJZt0762qsXoOR3loiIjBwvVZWEVvs0CeECZUREVBEwCakkDh+WxoLY2ABdu8odDRERUcGYhFQSugXKevYEzM3ljYWIiKgwmIRUAkJwwzoiIqp4mIRUAtevSze1GujRQ+5oiIiICofLtlcCnp5AbCxw/DhgbS13NERERIXDlpBKwt4e6NJF7iiIiIgKj0lIBcedf4iIqKJiElLBzZsH+PsD4eFyR0JERFQ0TEIquLVrgUOHgPh4uSMhIiIqGiYhFdidO8D+/YBCAQQEyB0NERFR0TAJqcD++EP62rYt4OoqayhERERFxiSkAuMCZUREVJExCamg4uOBnTul+337yhoKERFRsTAJqaDWrwc0GsDPD6hTR+5oiIiIio4rplZQXl5Av37S9FwiIqKKiElIBdWpk3QjIiKqqNgdQ0RERLJgElIB/fEHEBkpdxREREQlwySkgklNBV57DWjYUNo1l4iIqKJiElLBbNsGJCcD7u7SzBgiIqKKiklIBaNboKxvX0DJ7x4REVVgvIxVIJmZwJ9/Sve5SioREVV0TEIqkN27gfv3AUdHoEMHuaMhIiIqGSYhFYiuK6ZPH8CEK7wQEVEFxySkAtm2TfrKrhgiIqoM+P90BXL8OLB1K9C1q9yREBERlRyTkArEwgLo3VvuKIiIiEoHu2OIiIhIFkxCKoDTp4FGjYBZs+SOhIiIqPQwCakAIiKACxeAI0fkjoSIiKj0MAmpAHRTczkrhoiIKhMmIUbu8mXg1ClApQJ69ZI7GiIiotLDJMTIrVsnfe3SBXBwkDcWIiKi0sQkxMixK4aIiCorJiFG7PZt4MABQKEAAgLkjoaIiKh0cbEyI5aeDowcCcTFAa6uckdDRERUupiEGDEvLyAsTO4oiIiIyga7Y4iIiEgWTEKM1NGjwOHDgBByR0JERFQ2mIQYqRkzgDZtgLlz5Y6EiIiobDAJMUIPHwJbtkj3e/SQNxYiIqKywiTECG3aBKSlAfXrA40byx0NERFR2WASYoSyLlCmUMgbCxERUVlhEmJkUlOBjRul+1wllYiIKjMmIUZm2zYgORlwdwdatZI7GiIiorIjexKyaNEieHp6wtzcHP7+/jh06FCedTt37gyFQpHj1rNnT4N658+fR+/evWFnZwcrKyu0bt0aN27cKOu3Uio2bZK+9u3LrhgiIqrcZE1CVq1aheDgYEyfPh3Hjh2Dn58funfvjtjY2FzrR0REIDo6Wn87c+YMVCoV+vfvr69z5coVdOjQAQ0bNsTOnTtx6tQpTJ06Febm5uX1tkpk/nxgxw5gzBi5IyEiIipbCiHkWw7L398frVu3xsKFCwEAWq0WHh4eGD9+PCZPnlzg80NDQzFt2jRER0fDysoKADBw4ECYmpri119/LXZcSUlJsLOzQ2JiImxtbYt9HiIioqqmKNdQ2VpC0tPTcfToUXTr1u1pMEolunXrhv379xfqHGFhYRg4cKA+AdFqtdi4cSPq16+P7t27o0aNGvD398cff/yR73nS0tKQlJRkcCMiIqKyJVsScu/ePWg0Gjg7OxuUOzs74+7duwU+/9ChQzhz5gxef/11fVlsbCySk5Px+eef48UXX8SWLVvQt29fBAYGYteuXXmeKyQkBHZ2dvqbh4dH8d9YMWm1gL8/MH48EB9f7i9PRERU7mQfmFpcYWFh8PX1RZs2bfRlWq0WANCnTx+8++67aNasGSZPnoyXX34ZixcvzvNcU6ZMQWJiov528+bNMo8/u0OHpNuyZYC1dbm/PBERUbmTLQlxdHSESqVCTEyMQXlMTAxcXFzyfW5KSgpWrlyJUaNG5TiniYkJGmdbZrRRo0b5zo5Rq9WwtbU1uJU33QJlL78MqNXl/vJERETlTrYkxMzMDC1btsT27dv1ZVqtFtu3b0fbtm3zfe7q1auRlpaGIUOG5Dhn69atERkZaVB+8eJF1K5du/SCL2VCGK6SSkREVBWYyPniwcHBGDZsGFq1aoU2bdogNDQUKSkpGDFiBABg6NChqFmzJkJCQgyeFxYWhoCAAFSvXj3HOd9//30MGDAAzz77LLp06YLNmzdj/fr12LlzZ3m8pWI5fRq4cgUwNwdefFHuaIiIiMqHrEnIgAEDEBcXh2nTpuHu3bto1qwZNm/erB+seuPGDSiVho01kZGR2Lt3L7botpnNpm/fvli8eDFCQkLwzjvvoEGDBli7di06dOhQ5u+nuHStIN27czwIERFVHbKuE2KsynudkKZNpdaQZcuAoUPL/OWIiIjKTFGuobK2hBCQkQE89xyQkiINSiUiIqoqKuwU3crC1BQIDQUuXwYcHOSOhoiIqPwwCTES3KyOiIiqGiYhMoqNBf79F8jMlDsSIiKi8sckREbh4UDXrkCvXnJHQkREVP6YhMho3Trpa5Y9/IiIiKoMJiEyuXcP0O2p17evvLEQERHJgUmITNavBzQaoFkzoE4duaMhIiIqf0xCZMK9YoiIqKpjEiKDhw8B3arzTEKIiKiqYhIig23bgPR0oH59oHFjuaMhIiKSB5dtl0FAAHD0qDQ4lYuUERFRVcUkRAYKBdCihdxREBERyYvdMURERCQLJiHlLDgYGD4cOHlS7kiIiIjkxSSkHGVmAsuWSbcHD+SOhoiISF5MQsrR7t3A/fuAoyPQoYPc0RAREcmLA1PLgUYD7NkDfP659LhXL8CEnzwREVVxbAkpYxERgKcn0KULsHWrVLZ+/dMVU4mIiKoqJiFlKCIC6NcPuHXLsDw+XipnIkJERFUZk5AyotEAEyYAQuQ8piubOFGqR0REVBUxCSkje/bkbAHJSgjg5k2pHhERUVXEJKSMREeXbj0iIqLKhklIGXF1Ld16RERElQ2TkDLSsSPg7p73BnUKBeDhIdUjIiKqipiElBGVCpg/X7qfPRHRPQ4NleoRERFVRUxCylBgILBmDVCzpmG5u7tUHhgoT1xERETGgOt2lrHAQKBPH2kWTHS0NAakY0e2gBARETEJKQcqFdC5s9xREBERGRd2xxAREZEsmIQQERGRLJiEEBERkSyYhBAREZEsmIQQERGRLJiEEBERkSw4RTcXQggAQFJSksyREBERVSy6a6fuWpofJiG5ePjwIQDAw8ND5kiIiIgqpocPH8LOzi7fOgpRmFSlitFqtbhz5w5sbGygyGsHugosKSkJHh4euHnzJmxtbeUOx6jws8kdP5e88bPJHT+XvFX2z0YIgYcPH8LNzQ1KZf6jPtgSkgulUgl3d3e5wyhztra2lfIXoDTws8kdP5e88bPJHT+XvFXmz6agFhAdDkwlIiIiWTAJISIiIlkwCamC1Go1pk+fDrVaLXcoRoefTe74ueSNn03u+LnkjZ/NUxyYSkRERLJgSwgRERHJgkkIERERyYJJCBEREcmCSQgRERHJgklIFRISEoLWrVvDxsYGNWrUQEBAACIjI+UOy+h8/vnnUCgUmDhxotyhGIXbt29jyJAhqF69OiwsLODr64sjR47IHZasNBoNpk6dCi8vL1hYWKBu3bqYNWtWofbKqGx2796NXr16wc3NDQqFAn/88YfBcSEEpk2bBldXV1hYWKBbt264dOmSPMGWo/w+l4yMDHz44Yfw9fWFlZUV3NzcMHToUNy5c0e+gGXCJKQK2bVrF8aOHYsDBw5g69atyMjIwAsvvICUlBS5QzMahw8fxg8//ICmTZvKHYpRePDgAdq3bw9TU1Ns2rQJ586dw9y5c1GtWjW5Q5PVF198ge+//x4LFy7E+fPn8cUXX+DLL7/Et99+K3do5S4lJQV+fn5YtGhRrse//PJLLFiwAIsXL8bBgwdhZWWF7t27IzU1tZwjLV/5fS6PHj3CsWPHMHXqVBw7dgwRERGIjIxE7969ZYhUZoKqrNjYWAFA7Nq1S+5QjMLDhw9FvXr1xNatW0WnTp3EhAkT5A5Jdh9++KHo0KGD3GEYnZ49e4qRI0calAUGBorBgwfLFJFxACDWrVunf6zVaoWLi4v46quv9GUJCQlCrVaLFStWyBChPLJ/Lrk5dOiQACCuX79ePkEZCbaEVGGJiYkAAAcHB5kjMQ5jx45Fz5490a1bN7lDMRp//fUXWrVqhf79+6NGjRpo3rw5fvzxR7nDkl27du2wfft2XLx4EQBw8uRJ7N27Fz169JA5MuMSFRWFu3fvGvxO2dnZwd/fH/v375cxMuOTmJgIhUIBe3t7uUMpV9zArorSarWYOHEi2rdvjyZNmsgdjuxWrlyJY8eO4fDhw3KHYlSuXr2K77//HsHBwfjoo49w+PBhvPPOOzAzM8OwYcPkDk82kydPRlJSEho2bAiVSgWNRoPZs2dj8ODBcodmVO7evQsAcHZ2Nih3dnbWHyMgNTUVH374IQYNGlRpN7TLC5OQKmrs2LE4c+YM9u7dK3cosrt58yYmTJiArVu3wtzcXO5wjIpWq0WrVq0wZ84cAEDz5s1x5swZLF68uEonIeHh4Vi+fDl+//13+Pj44MSJE5g4cSLc3Nyq9OdCRZeRkYGgoCAIIfD999/LHU65Y3dMFTRu3Dhs2LABO3bsgLu7u9zhyO7o0aOIjY1FixYtYGJiAhMTE+zatQsLFiyAiYkJNBqN3CHKxtXVFY0bNzYoa9SoEW7cuCFTRMbh/fffx+TJkzFw4ED4+vritddew7vvvouQkBC5QzMqLi4uAICYmBiD8piYGP2xqkyXgFy/fh1bt26tcq0gAJOQKkUIgXHjxmHdunX4999/4eXlJXdIRqFr1644ffo0Tpw4ob+1atUKgwcPxokTJ6BSqeQOUTbt27fPMY374sWLqF27tkwRGYdHjx5BqTT886lSqaDVamWKyDh5eXnBxcUF27dv15clJSXh4MGDaNu2rYyRyU+XgFy6dAnbtm1D9erV5Q5JFuyOqULGjh2L33//HX/++SdsbGz0fbJ2dnawsLCQOTr52NjY5BgXY2VlherVq1f58TLvvvsu2rVrhzlz5iAoKAiHDh3CkiVLsGTJErlDk1WvXr0we/Zs1KpVCz4+Pjh+/DjmzZuHkSNHyh1auUtOTsbly5f1j6OionDixAk4ODigVq1amDhxIj777DPUq1cPXl5emDp1Ktzc3BAQECBf0OUgv8/F1dUV/fr1w7Fjx7BhwwZoNBr932MHBweYmZnJFXb5k3t6DpUfALne/ve//8kdmtHhFN2n1q9fL5o0aSLUarVo2LChWLJkidwhyS4pKUlMmDBB1KpVS5ibm4s6deqIjz/+WKSlpckdWrnbsWNHrn9Xhg0bJoSQpulOnTpVODs7C7VaLbp27SoiIyPlDboc5Pe5REVF5fn3eMeOHXKHXq4UQlTBJf6IiIhIdhwTQkRERLJgEkJERESyYBJCREREsmASQkRERLJgEkJERESyYBJCREREsmASQkRERLJgEkJERESyYBJCRFXCzp07oVAokJCQIHcoRPQEkxAiIiKSBZMQIiIikgWTECIqF1qtFiEhIfDy8oKFhQX8/PywZs0aAE+7SjZu3IimTZvC3NwczzzzDM6cOWNwjrVr18LHxwdqtRqenp6YO3euwfG0tDR8+OGH8PDwgFqthre3N8LCwgzqHD16FK1atYKlpSXatWuHyMjIsn3jRJQnJiFEVC5CQkLwyy+/YPHixTh79izeffddDBkyBLt27dLXef/99zF37lwcPnwYTk5O6NWrFzIyMgBIyUNQUBAGDhyI06dP49NPP8XUqVPx888/658/dOhQrFixAgsWLMD58+fxww8/wNra2iCOjz/+GHPnzsWRI0dgYmKCkSNHlsv7J6JcyL2NLxFVfqmpqcLS0lL8999/BuWjRo0SgwYN0m97vnLlSv2x+Ph4YWFhIVatWiWEEOLVV18Vzz//vMHz33//fdG4cWMhhBCRkZECgNi6dWuuMeheY9u2bfqyjRs3CgDi8ePHpfI+iaho2BJCRGXu8uXLePToEZ5//nlYW1vrb7/88guuXLmir9e2bVv9fQcHBzRo0ADnz58HAJw/fx7t27c3OG/79u1x6dIlaDQanDhxAiqVCp06dco3lqZNm+rvu7q6AgBiY2NL/B6JqOhM5A6AiCq/5ORkAMDGjRtRs2ZNg2NqtdogESkuCwuLQtUzNTXV31coFACk8SpEVP7YEkJEZa5x48ZQq9W4ceMGvL29DW4eHh76egcOHNDff/DgAS5evIhGjRoBABo1aoR9+/YZnHffvn2oX78+VCoVfH19odVqDcaYEJFxY0sIEZU5GxsbTJo0Ce+++y60Wi06dOiAxMRE7Nu3D7a2tqhduzYAYObMmahevTqcnZ3x8ccfw9HREQEBAQCA9957D61bt8asWbMwYMAA7N+/HwsXLsR3330HAPD09MSwYcMwcuRILFiwAH5+frh+/TpiY2MRFBQk11snonwwCSGicjFr1iw4OTkhJCQEV69ehb29PVq0aIGPPvpI3x3y+eefY8KECbh06RKaNWuG9evXw8zMDADQokULhIeHY9q0aZg1axZcXV0xc+ZMDB8+XP8a33//PT766COMGTMG8fHxqFWrFj766CM53i4RFYJCCCHkDoKIqradO3eiS5cuePDgAezt7eUOh4jKCceEEBERkSyYhBAREZEs2B1DREREsmBLCBEREcmCSQgRERHJgkkIERERyYJJCBEREcmCSQgRERHJgkkIERERyYJJCBEREcmCSQgRERHJ4v+hdxQqrxVJ9QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='13' class='progress-bar-interrupted' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      13.00% [13/100] [35:57<4:00:39]\n",
       "      <br>\n",
       "      ████████████████████100.00% [79/79] [val_loss=0.4703, val_auc=0.7779]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< val_auc without improvement in 5 epoch,early stopping >>>>>> \n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "dfhistory = model.fit(train_data = dl_train,\n",
    "    val_data = dl_val,\n",
    "    epochs=100,\n",
    "    ckpt_path='checkpoint',\n",
    "    patience=5,\n",
    "    monitor='val_auc',\n",
    "    mode='max',\n",
    "    plot=True,\n",
    "    cpu=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62e2b26d-2350-48b5-a6ec-d05bf1620b57",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8994bf3e",
   "metadata": {},
   "source": [
    "### 4，评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "21a1a856-fe49-412e-bcfd-2e1c10424f1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████| 98/98 [00:16<00:00,  5.99it/s, val_auc=0.781, val_loss=0.466]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'val_loss': 0.46646365553748853, 'val_auc': 0.781028687953949}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(dl_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bbe2633-638b-4eb7-a024-9954b5d91e91",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1b04a53e",
   "metadata": {},
   "source": [
    "### 5，使用模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8b356206",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7810287729660677\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "model.eval()\n",
    "dl_test = model.accelerator.prepare(dl_test)\n",
    "with torch.no_grad():\n",
    "    result = torch.cat([model.forward(t[0]) for t in dl_test])\n",
    "\n",
    "preds = F.sigmoid(result)\n",
    "labels = torch.cat([x[-1] for x in dl_test])\n",
    "\n",
    "val_auc = roc_auc_score(labels.numpy(),preds.numpy())\n",
    "print(val_auc)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f021a0bd-1ee5-43d6-85e3-432b7a9e1f8f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "48965f11",
   "metadata": {},
   "source": [
    "### 6，保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9ce7957-6042-44ca-bee9-dee0b3b42d55",
   "metadata": {},
   "source": [
    "模型最佳权重已经保存在 model.fit(ckpt_path) 传入的参数中了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7c343ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "net_clone = create_net()\n",
    "net_clone.load_state_dict(torch.load(model.ckpt_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b54f14a3",
   "metadata": {},
   "source": [
    "**如果本书对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果对本书内容理解上有需要进一步和作者交流的地方，欢迎在公众号\"算法美食屋\"下留言。作者时间和精力有限，会酌情予以回复。\n",
    "\n",
    "也可以在公众号后台回复关键字：**加群**，加入读者交流群和大家讨论。\n",
    "\n",
    "![算法美食屋logo.png](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,md",
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}