fastai_NLP/TF-IDF.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "# 多行输出\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF_IDF\n",
    "\n",
    "[TF-IDF与余弦相似度 - 知乎](https://zhuanlan.zhihu.com/p/32826433)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> TF-IDF(term frequency=inverse document frequency)是⼀种⽤于资讯检索与文本挖掘的常⽤加权技术。TF-IDF是⼀种统计方法，⽤以评估⼀字词对于⼀个文件集或⼀个语料库中的其中⼀份⽂件的重要程度。字词的重要性随着它在⽂件中出现的次数成正比增加，但同时会随着它在语料库中出现的频率成反比下降。TF-IDF加权的各种形式常备搜索引擎应⽤，作为文件与用户查询之间相关程度的度量或评级\n",
    "\n",
    "TF: 词频，IDF就是在词频的基础上，要对每个词分配一个“重要性”权重。最常见的词 （\"的\"、\"是\"、\"在\"）给予最小的权重，较常见的词（\"中国\"）给予较小的权重，较少见的词 （\"蜜蜂\"、\"养殖\"）给予较大的权重。这个权重叫做\"逆⽂档频率\"（Inverse Document Frequency，缩写为IDF），它的大小与一个词的常见程度成反比。 知道了\"词频\"（TF）和\"逆文档频率\"（IDF）以后，将这两个值相乘，就得到了一个词的TF-IDF 值。某个词对文章的重要性越高，它的TF-IDF值就越大。所以，排在最前面的几个词，就是这篇文章的关键词。\n",
    "$$\\text{TF}_i=\\frac{\\text{某个词在文档中出现的次数}}{\\text{该文档中词的总数}}$$\n",
    "\n",
    "$$\\text{IDF}_i=log{\\frac{\\text{语料库的文档总数}}{\\text{包含改词的文档总数} + 1}}$$\n",
    "\n",
    "$$\\text{TF-IDF}=\\text{TF}_i * \\text{IDF}_i$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer  \n",
    "from sklearn.feature_extraction.text import CountVectorizer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus=[\"I come to China to travel\", \n",
    "        \"This is a car polupar in China\",          \n",
    "        \"I love tea and Apple \",   \n",
    "        \"The work is to write some papers in science\"] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizer=CountVectorizer()\n",
    "transformer = TfidfTransformer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = vectorizer.fit_transform(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 0],\n",
       "        [0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0],\n",
       "        [1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0],\n",
       "        [0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1]],\n",
       "       dtype=int64)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.todense()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf = transformer.fit_transform(v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0.        , 0.        , 0.        , 0.34884223, 0.44246214,\n",
       "         0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "         0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "         0.69768446, 0.44246214, 0.        , 0.        ],\n",
       "        [0.        , 0.        , 0.4533864 , 0.35745504, 0.        ,\n",
       "         0.35745504, 0.35745504, 0.        , 0.        , 0.4533864 ,\n",
       "         0.        , 0.        , 0.        , 0.        , 0.4533864 ,\n",
       "         0.        , 0.        , 0.        , 0.        ],\n",
       "        [0.5       , 0.5       , 0.        , 0.        , 0.        ,\n",
       "         0.        , 0.        , 0.5       , 0.        , 0.        ,\n",
       "         0.        , 0.        , 0.5       , 0.        , 0.        ,\n",
       "         0.        , 0.        , 0.        , 0.        ],\n",
       "        [0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "         0.28113163, 0.28113163, 0.        , 0.35657982, 0.        ,\n",
       "         0.35657982, 0.35657982, 0.        , 0.35657982, 0.        ,\n",
       "         0.28113163, 0.        , 0.35657982, 0.35657982]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf.todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 一步到位=向量化+TF-IDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf2 = TfidfVectorizer()\n",
    "ret = tfidf2.fit_transform(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0.        , 0.        , 0.        , 0.34884223, 0.44246214,\n",
       "         0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "         0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "         0.69768446, 0.44246214, 0.        , 0.        ],\n",
       "        [0.        , 0.        , 0.4533864 , 0.35745504, 0.        ,\n",
       "         0.35745504, 0.35745504, 0.        , 0.        , 0.4533864 ,\n",
       "         0.        , 0.        , 0.        , 0.        , 0.4533864 ,\n",
       "         0.        , 0.        , 0.        , 0.        ],\n",
       "        [0.5       , 0.5       , 0.        , 0.        , 0.        ,\n",
       "         0.        , 0.        , 0.5       , 0.        , 0.        ,\n",
       "         0.        , 0.        , 0.5       , 0.        , 0.        ,\n",
       "         0.        , 0.        , 0.        , 0.        ],\n",
       "        [0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "         0.28113163, 0.28113163, 0.        , 0.35657982, 0.        ,\n",
       "         0.35657982, 0.35657982, 0.        , 0.35657982, 0.        ,\n",
       "         0.28113163, 0.        , 0.35657982, 0.35657982]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ret.todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 余弦相似性"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "余弦相似性通过测量两个向量的夹角的余弦值来度量它们之间的相似性。0度角的余弦值是1，而其他任何角度的余弦值都不大于1；并且其最小值是-1。从而两个向量之间的角度的余弦值确定两个向量是否大致指向相同的方向。两个向量有相同的指向时，余弦相似度的值为1；两个向量夹角为90°时，余弦相似度的值为0；两个向量指向完全相反的方向时，余弦相似度的值为-1。这 结果是与向量的长度无关的，仅与向量的指向方向相关。余弦相似度通常用于正空间，因此给出的值为0到1之间。\n",
    "- 余弦相似度因此可以给出两篇文档在其主题方面的相似度。\n",
    "- 余弦相似性就是 $cos(\\theta)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}