pedants_qa_evaluation

This dataset evaluates candidate answers for various question-answering (QA) tasks across multiple datasets such as Jeopardy!, hotpotQA, nq-open, narrativeQA, and BIOMRC, etc. See details in paper. It contains questions, reference answers (ground truth), model-generated candidate answers, and human judgments indicating whether the candidate answers are correct.

---

Dataset Details

---

Download the Dataset

The dataset can be loaded using the Hugging Face datasets library. Use the following code to download it:

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("zli12321/pedants_qa_evaluation_bench")

# View the first few rows of the training set
print(dataset["train"].to_pandas().head())

Evaluation Metrics

Refer to Evaluation Metrics and github to test.

📄 Citation

@misc{li2024pedantscheapeffectiveinterpretable,
      title={PEDANTS: Cheap but Effective and Interpretable Answer Equivalence}, 
      author={Zongxia Li and Ishani Mondal and Yijun Liang and Huy Nghiem and Jordan Lee Boyd-Graber},
      year={2024},
      eprint={2402.11161},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2402.11161}, 
}