QAEdit: Benchmarking Editing on QA Tasks

QAEdit is a tailored dataset derived from three widely used QA benchmarks—Natural Questions, TriviaQA, and SimpleQA—to enable editing of LLMs with answers grounded in real-world tasks.

Dataset Sources

• Repository: Revisit-Editing-Evaluation
• Paper: The Mirage of Model Editing: Revisiting Evaluation in the Wild

Uses

You can download QAEdit and place it in the data folder of Revisit-Editing-Evaluation, then perform editing following the corresponding guidelines.

Dataset Structure

Each entry in QAEdit contains the following fields:

• Source: the originating QA benchmark (i.e., NQ, TQA, or SQA).
• prompt: the question to be edited.
• target: the correct answer to the question being edited.
• subject: the subject of the question.
• rephrase: a paraphrased version of the question, used to evaluate the generalization capability of the edited knowledge.
• locality: unrelated QA pairs, included to assess the impact of editing on irrelevant knowledge.

Dataset Creation

Curation Rationale

While existing work reports remarkable success of model editing techniques, their effectiveness in real-world applications remains unclear. To rigorously examine their practical utility, we introduce QAEdit, to enable model editing for correcting LLM errors in QA tasks and assess its improvement for real-world applications.

Source Data

• Natural Questions
• TriviaQA
• SimpleQA

Data Collection and Processing

While these QA benchmarks provide questions and answers as edit prompts and targets respectively, they lack essential fields that mainstream editing methods require for editing and evaluation. To obtain required subjects for editing, we employ GPT-4 (gpt-4-1106-preview) to extract them directly from the questions. To align with the previous editing evaluation protocol, we evaluate: i) reliability using original edit prompts; ii) generalization through GPT-4 paraphrased prompts; and iii) locality using unrelated QA pairs from ZsRE locality set.

Citation

If you find our work helpful, please cite our paper~

BibTeX:

@misc{yang2025miragemodeleditingrevisiting,
      title={The Mirage of Model Editing: Revisiting Evaluation in the Wild}, 
      author={Wanli Yang and Fei Sun and Jiajun Tan and Xinyu Ma and Qi Cao and Dawei Yin and Huawei Shen and Xueqi Cheng},
      year={2025},
      eprint={2502.11177},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.11177}, 
}