pyserini_covid19_default.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "pyserini_covid19_default.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/castorini/anserini-notebooks/blob/master/pyserini_covid19_default.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vUpmSFflwzR1",
        "colab_type": "text"
      },
      "source": [
        "# Pyserini Demo on COVID-19 Dataset (Title + Abstract Index)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0RtnmBN6A-Oq",
        "colab_type": "text"
      },
      "source": [
        "This notebook provides a demo on how to get started in searching the [COVID-19 Open Research Dataset](https://pages.semanticscholar.org/coronavirus-research) (release of 2020/04/10) from AI2.\n",
        "Here, we'll be working with the title + abstract index.\n",
        "We have a [separate notebook](https://github.com/castorini/anserini-notebooks/blob/master/pyserini_covid19_paragraph.ipynb) that demonstrates working with the paragraph index, which will likely yield better search results.\n",
        "However, this index is smaller and easier to manipulate, so it's a good starting point."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f2vNd7bpJlDZ",
        "colab_type": "text"
      },
      "source": [
        "First, install Python dependencies:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E_lt0-pXJia0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "!pip install pyserini==0.9.0.0\n",
        "\n",
        "import json\n",
        "import os\n",
        "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-11-openjdk-amd64\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xbXmstbGypef",
        "colab_type": "text"
      },
      "source": [
        "Let's grab the pre-built index:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JT_OJKftdqGP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "!wget https://www.dropbox.com/s/j55t617yhvmegy8/lucene-index-covid-2020-04-10.tar.gz\n",
        "!tar xvfz lucene-index-covid-2020-04-10.tar.gz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EwVSvHBz9RMM",
        "colab_type": "text"
      },
      "source": [
        "Sanity check of index size (should be 1.5G):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KVXWA6WS0aqJ",
        "colab_type": "code",
        "outputId": "e7a1e504-794d-421d-937a-a2b4a566e4d1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!du -h lucene-index-covid-2020-04-10"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1.6G\tlucene-index-covid-2020-04-10\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NkD0kKxW9mHP",
        "colab_type": "text"
      },
      "source": [
        "You can use `pysearch` to search over an index. Here's the basic usage:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yFZlcqEX0t1f",
        "colab_type": "code",
        "outputId": "f0e6edee-4c1e-4cf4-f45e-5e79d6d3b105",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 207
        }
      },
      "source": [
        "from pyserini.search import pysearch\n",
        "\n",
        "searcher = pysearch.SimpleSearcher('lucene-index-covid-2020-04-10/')\n",
        "hits = searcher.search('incubation period of COVID-19', 10)\n",
        "\n",
        "# Prints the first 10 hits\n",
        "for i in range(0, 10):\n",
        "    print(f'{i+1:2} {hits[i].docid} {hits[i].score:.5f} {hits[i].lucene_document.get(\"title\")} {hits[i].lucene_document.get(\"doi\")}')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " 1 vkgnwxzc 11.50990 Estimate the incubation period of coronavirus 2019 (COVID-19) 10.1101/2020.02.24.20027474\n",
            " 2 slapc5xt 11.13590 A Chinese Case of COVID-19 Did Not Show Infectivity During the Incubation Period: Based on an Epidemiological Survey 10.3961/jpmph.20.048\n",
            " 3 djq0lvr2 11.07600 Is a 14-day quarantine period optimal for effectively controlling coronavirus disease 2019 (COVID-19)? 10.1101/2020.03.15.20036533\n",
            " 4 8anqfkmo 10.93740 The Incubation Period of Coronavirus Disease 2019 (COVID-19) From Publicly Reported Confirmed Cases: Estimation and Application 10.7326/m20-0504\n",
            " 5 it4ka7v0 10.91810 Estimation of incubation period distribution of COVID-19 using disease onset forward time: a novel cross-sectional and forward follow-up study 10.1101/2020.03.06.20032417\n",
            " 6 ibx89gqw 10.78020 Estimating the distribution of the incubation period of 2019 novel coronavirus (COVID-19) infection between travelers to Hubei, China and non-travelers 10.1101/2020.02.13.20022822\n",
            " 7 tne83uu0 10.54640 Epidemiologic Characteristics of COVID-19 in Guizhou, China 10.1101/2020.03.01.20028944\n",
            " 8 aul6ahww 10.52700 Transmission of COVID-19 in the terminal stage of incubation period: a familial cluster 10.1016/j.ijid.2020.03.027\n",
            " 9 ls2z145c 10.49810 Study on assessing early epidemiological parameters of coronavirus disease epidemic in China 10.3760/cma.j.cn112338-20200205-00069\n",
            "10 9jatvium 10.45870 COVID-19 in a patient with long-term use of glucocorticoids: A study of a familial cluster 10.1016/j.clim.2020.108413\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iiXaE-TDH7uc",
        "colab_type": "text"
      },
      "source": [
        "From the hits array, use `.lucene_document` to access the underlying indexed Lucene `Document`, and from there, call `.get(field)` to fetch specific fields, like \"title\", \"doc\", etc.\n",
        "The complete list of available fields is [here](https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/index/generator/CovidGenerator.java#L46)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EBct_zb8Lyvp",
        "colab_type": "text"
      },
      "source": [
        "For hit #2, \"A Chinese Case of COVID-19 Did Not Show Infectivity During the Incubation Period\", we don't have the full text, but we can access available information via `.raw`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WAYzpshzL7St",
        "colab_type": "code",
        "outputId": "aa83f54b-c7ed-41ca-f9f1-cfb978270670",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 377
        }
      },
      "source": [
        "hit2_json = json.loads(hits[1].raw)\n",
        "print(json.dumps(hit2_json, indent=4))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"abstract\": \"Controversy remains over whether the novel coronavirus 2019 (COVID-19) virus may have infectivity during the incubation period before the onset of symptoms. The author had the opportunity to examine the infectivity of COVID-19 during the incubation period by conducting an epidemiological survey on a confirmed patient who had visited Jeju Island during the incubation period. The epidemiological findings support the claim that the COVID-19 virus does not have infectivity during the incubation period.\",\n",
            "    \"authors\": \"Bae, Jong-Myon\",\n",
            "    \"cord_uid\": \"slapc5xt\",\n",
            "    \"doi\": \"10.3961/jpmph.20.048\",\n",
            "    \"full_text_file\": \"\",\n",
            "    \"has_pdf_parse\": \"False\",\n",
            "    \"has_pmc_xml_parse\": \"False\",\n",
            "    \"journal\": \"Journal of Preventive Medicine and Public Health\",\n",
            "    \"license\": \"unk\",\n",
            "    \"Microsoft Academic Paper ID\": \"2029228287\",\n",
            "    \"pmcid\": \"\",\n",
            "    \"publish_time\": \"2020-03-02\",\n",
            "    \"pubmed_id\": \"32114755\",\n",
            "    \"sha\": \"\",\n",
            "    \"source_x\": \"WHO\",\n",
            "    \"title\": \"A Chinese Case of COVID-19 Did Not Show Infectivity During the Incubation Period: Based on an Epidemiological Survey\",\n",
            "    \"url\": \"https://doi.org/10.3961/jpmph.20.048\",\n",
            "    \"WHO #Covidence\": \"#3073\"\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xw6EjwPgPB_b",
        "colab_type": "text"
      },
      "source": [
        "For hit #4, \"The Incubation Period of Coronavirus Disease 2019 (COVID-19) From Publicly Reported Confirmed Cases\", we have the full text, which we can also fetch via `.raw`:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lGLZhQgANy3I",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "hit4_json = json.loads(hits[3].raw)\n",
        "\n",
        "# Uncomment to print... warning, it's long! :)\n",
        "# print(json.dumps(hit4_json, indent=4))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MrNEuT6uXuxD",
        "colab_type": "text"
      },
      "source": [
        "Finally, if you want to create a DataFrame comprising all the results, here's a snippet of code to do so:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PUQZt6iIWQSV",
        "colab_type": "code",
        "outputId": "e076bdd9-45cf-4678-9cbf-43d40f079d5f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "ranks = list(range(1, len(hits)+1))\n",
        "docids = [ hit.docid for hit in hits]\n",
        "scores = [ hit.score for hit in hits]\n",
        "titles = [ hit.lucene_document.get('title') for hit in hits]\n",
        "dois = [ hit.lucene_document.get('doi') for hit in hits]\n",
        "raw = [ hit.raw for hit in hits ]\n",
        "data = {'rank': ranks, 'docid': docids, 'score': scores, 'title': titles, 'doi': dois, 'raw': raw} \n",
        "\n",
        "df = pd.DataFrame(data)\n",
        "df"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>rank</th>\n",
              "      <th>docid</th>\n",
              "      <th>score</th>\n",
              "      <th>title</th>\n",
              "      <th>doi</th>\n",
              "      <th>raw</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>vkgnwxzc</td>\n",
              "      <td>11.5099</td>\n",
              "      <td>Estimate the incubation period of coronavirus ...</td>\n",
              "      <td>10.1101/2020.02.24.20027474</td>\n",
              "      <td>{\\n    \"paper_id\": \"c1ae608c7ffb926a0f50a6a34c...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>slapc5xt</td>\n",
              "      <td>11.1359</td>\n",
              "      <td>A Chinese Case of COVID-19 Did Not Show Infect...</td>\n",
              "      <td>10.3961/jpmph.20.048</td>\n",
              "      <td>{\"abstract\":\"Controversy remains over whether ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>djq0lvr2</td>\n",
              "      <td>11.0760</td>\n",
              "      <td>Is a 14-day quarantine period optimal for effe...</td>\n",
              "      <td>10.1101/2020.03.15.20036533</td>\n",
              "      <td>{\\n    \"paper_id\": \"d09c0f71b1a404a592d0dcad2c...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>8anqfkmo</td>\n",
              "      <td>10.9374</td>\n",
              "      <td>The Incubation Period of Coronavirus Disease 2...</td>\n",
              "      <td>10.7326/m20-0504</td>\n",
              "      <td>{\\n    \"paper_id\": \"ce8609a60724d457d5b5916d57...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>it4ka7v0</td>\n",
              "      <td>10.9181</td>\n",
              "      <td>Estimation of incubation period distribution o...</td>\n",
              "      <td>10.1101/2020.03.06.20032417</td>\n",
              "      <td>{\\n    \"paper_id\": \"6d3b3f4ab80a61c45f82c61c6c...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>ibx89gqw</td>\n",
              "      <td>10.7802</td>\n",
              "      <td>Estimating the distribution of the incubation ...</td>\n",
              "      <td>10.1101/2020.02.13.20022822</td>\n",
              "      <td>{\"abstract\":\"Objectives: Amid the continuing s...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>tne83uu0</td>\n",
              "      <td>10.5464</td>\n",
              "      <td>Epidemiologic Characteristics of COVID-19 in G...</td>\n",
              "      <td>10.1101/2020.03.01.20028944</td>\n",
              "      <td>{\\n    \"paper_id\": \"7e9cd4bbf0fba1cc0bcded4004...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8</td>\n",
              "      <td>aul6ahww</td>\n",
              "      <td>10.5270</td>\n",
              "      <td>Transmission of COVID-19 in the terminal stage...</td>\n",
              "      <td>10.1016/j.ijid.2020.03.027</td>\n",
              "      <td>{\\n    \"paper_id\": \"b9ffacdcfda1d28c4e66e502c2...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>ls2z145c</td>\n",
              "      <td>10.4981</td>\n",
              "      <td>Study on assessing early epidemiological param...</td>\n",
              "      <td>10.3760/cma.j.cn112338-20200205-00069</td>\n",
              "      <td>{\"abstract\":\"Objective: To study the early dyn...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>9jatvium</td>\n",
              "      <td>10.4587</td>\n",
              "      <td>COVID-19 in a patient with long-term use of gl...</td>\n",
              "      <td>10.1016/j.clim.2020.108413</td>\n",
              "      <td>{\\n    \"paper_id\": \"aecea2084e32367c875a9266e8...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   rank  ...                                                raw\n",
              "0     1  ...  {\\n    \"paper_id\": \"c1ae608c7ffb926a0f50a6a34c...\n",
              "1     2  ...  {\"abstract\":\"Controversy remains over whether ...\n",
              "2     3  ...  {\\n    \"paper_id\": \"d09c0f71b1a404a592d0dcad2c...\n",
              "3     4  ...  {\\n    \"paper_id\": \"ce8609a60724d457d5b5916d57...\n",
              "4     5  ...  {\\n    \"paper_id\": \"6d3b3f4ab80a61c45f82c61c6c...\n",
              "5     6  ...  {\"abstract\":\"Objectives: Amid the continuing s...\n",
              "6     7  ...  {\\n    \"paper_id\": \"7e9cd4bbf0fba1cc0bcded4004...\n",
              "7     8  ...  {\\n    \"paper_id\": \"b9ffacdcfda1d28c4e66e502c2...\n",
              "8     9  ...  {\"abstract\":\"Objective: To study the early dyn...\n",
              "9    10  ...  {\\n    \"paper_id\": \"aecea2084e32367c875a9266e8...\n",
              "\n",
              "[10 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    }
  ]
}