ExpMRC

ExpMRC is a benchmark for the Explainability evaluation of Machine Reading Comprehension. ExpMRC contains four subsets of popular MRC datasets with additionally annotated evidences, including SQuAD, CMRC 2018, RACE⁺ (similar to RACE), and C³, covering span-extraction and multiple-choice questions MRC tasks in both English and Chinese.

Paper [Cui et al., 2022] BibTeX [Cui et al., 2022]

Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):

• Download ExpMRC Development Set

To evaluate your models, we have also made available the evaluation script for official evaluation, with sample predictions on each subset. To run the evaluation, use python eval_expmrc.py <path_to_dev> <path_to_predictions>

• ExpMRC Evaluation Script

• Sample Prediction Files on Dev Set

You may also be interested in a quick baseline system based on pre-trained language model, such as BERT. The codes are distributed under the Apache-2.0 license.

• Get Baseline Code

To preserve the integrity of test results, we do not release the test sets to the public. Instead, we require you to upload your model onto CodaLab, so that we can run it on the test sets for you. You can follow the instructions on CodaLab (which is similar to SQuAD, CMRC 2018 submission). Submission Tutorial

Please read the following before submission. Also, we strongly suggest the participants follow our baseline settings to ensure a fair comparison for academic purposes.

You are free to use any open-source machine reading comprehension data or automatically generated data for training systems (both labeled and unlabeled).

You are NOT allowed to use any newly human-annotated data (which is not publicly available) for training, which violates our submission policy.

We do not encourage using the development set of ExpMRC for training (though it is not prohibited). Such submissions will be marked with an asterisk (*).

The leaderboard is updated once new submissions are included. The ranking is based on the "Overall F1", which considers both answer and evidence F1 (but not solely the product of them). The details of the evaluation process can be seen here.

Ask us questions at our GitHub repository or at expmrc [at] 126 [dot] com .