Skip to content

allenai/scirepeval

Repository files navigation

SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

This repo contains the code to train, evaluate and reproduce the representation learning models and results on the benchmark introduced in SciRepEval.

Quick Setup

Clone the repo and setup the environment as follows:

git clone git@github.com:allenai/scirepeval.git
cd scirepeval
conda create -n scirepeval python=3.8
conda activate scirepeval
pip install -r requirements.txt

Usage

Please refer to the following for further usage:

Training - Train multi-task/multi-format transformer models or adapter modules

Inference - Using the trained SciRepEval models to generate embeddings.

Evaluation - Evaluate trained models on custom tasks OR customize existing evaluation config for SciRepEval benchmark tasks

Benchmarking - Simply evaluate models(pretrained from HuggingFace/local checkpoints) on SciRepEval and generate a report

Benchmark Details

SciRepEval consists of 24 scientific document tasks to train and evaluate scientific document representation models. The tasks are divided across 4 task formats- classification CLF, regression RGN, proximity (nearest neighbors) retrieval PRX and ad-hoc search SRCH. The table below gives a brief overview of the tasks with their HuggingFace datasets config names, if applicable. The benchmark dataset can be downloaded from AWS S3 or HuggingFace as follows:

AWS S3 via CLI

mkdir scirepeval_data && mkdir scirepeval_data/train && mkdir scirepeval_data/test && cd scirepeval_data
aws s3 --no-sign-request sync s3://ai2-s2-research-public/scirepeval/train train
aws s3 --no-sign-request sync s3://ai2-s2-research-public/scirepeval/test test

The AWS CLI commands can be run with the --dryrun flag to list the files being copied. The entire dataset is ~24 GB in size.

HuggingFace Datasets

The training, validation and raw evaluation data is available at allenai/scirepeval, while the labelled test examples are available at allenai/scirepeval_test.

import datasets
#training/validation/eval metadata
dataset = datasets.load_dataset(allenai/scirepeval, <hf config name>)

#labelled test examples
dataset = datasets.load_dataset(allenai/scirepeval_test, <hf config name>)

Since we want to evaluate document representations, every dataset consists of two parts: test metadata (text for representation generation available under allenai/scirepeval) and labelled examples (available under allenai/scirepeval_test)

Format Name Train Metric HF Config HF Test Config
CLF MeSH Descriptors Y F1 Macro mesh_descriptors mesh_descriptors
CLF Fields of study Y F1 Macro fos fos
CLF Biomimicry N F1 Binary biomimicry biomimicry
CLF DRSM N F1 Macro drsm drsm
CLF SciDocs-MAG N F1 Macro scidocs_mag_mesh scidocs_mag
CLF SciDocs-Mesh Diseases N F1 Macro scidocs_mag_mesh scidocs_mesh
RGN Citation Count Y Kendall's Tau cite_count cite_count
RGN Year of Publication Y Kendall's Tau pub_year pub_year
RGN Peer Review Score N Kendall's Tau peer_review_score_hIndex peer_review_score
RGN Max Author hIndex N Kendall's Tau peer_review_score_hIndex hIndex
RGN Tweet Mentions N Kendall's Tau tweet_mentions tweet_mentions
PRX Same Author Detection Y MAP same_author same_author
PRX Highly Influential Citations Y MAP high_influence_cite high_influence_cite
PRX Citation Prediction Y - cite_prediction -
PRX S2AND* N B^3 F1 - -
PRX Paper-Reviewer Matching** N Precision@5,10 paper_reviewer_matching paper_reviewer_matching, reviewers
PRX RELISH N NDCG relish relish
PRX SciDocs-Cite N MAP, NDCG scidocs_view_cite_read scidocs_cite
PRX SciDocs-CoCite N MAP, NDCG scidocs_view_cite_read scidocs_cocite
PRX SciDocs-CoView N MAP, NDCG scidocs_view_cite_read scidocs_view
PRX SciDocs-CoRead N MAP, NDCG scidocs_view_cite_read scidocs_read
SRCH Search Y NDCG search search
SRCH NFCorpus N NDCG nfcorpus nfcorpus
SRCH TREC-CoVID N NDCG trec_covid trec_covid

*S2AND requires the evaluation dataset in a specific format so to evaluate your model on the task please follow these instructions.

**Combinations of multiple datasets - 1, 2, 3, also dataset of papers authored by potential reviewers is required for evaluation; hence the multiple dataset configs.

License

The aggregate benchmark is released under ODC-BY license. By downloading this data you acknowledge that you have read and agreed to all the terms in this license. For constituent datasets, also go through the individual licensing requirements, as applicable.

Citation

Please cite the SciRepEval work as:

@article{Singh2022SciRepEvalAM,
  title={SciRepEval: A Multi-Format Benchmark for Scientific Document Representations},
  author={Amanpreet Singh and Mike D'Arcy and Arman Cohan and Doug Downey and Sergey Feldman},
  journal={ArXiv},
  year={2022},
  volume={abs/2211.13308}
}

About

SciRepEval benchmark training and evaluation scripts

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •  

Languages