Dataset Description

This collection includes 6 widely-used datasets for open-domain question answering and retrieval evaluation: 2WikiMultihopQA, HotpotQA,Musique,PopQA,TrivialQA,PubMedQA

Our evaluation code is at https://github.com/AQ-MedAI/RagQALeaderboard.

Leaderboard

Overall Performance of Different Models on Various Tasks:

Performance of Different Models on Specific Datasets

Currently, in the Medical Domain, we have selected relatively few datasets, so the evaluation might contain some randomness. In the future, we plan to include more related datasets. Additionally, we will continue to evaluate more models.

Inference

Installation

git clone https://github.com/AQ-MedAI/RagQALeaderboard
cd RagQALeaderboard/
pip install -r requirements.txt
# Make sure hf CLI is installed: pip install -U "huggingface_hub[cli]"
hf download AQ-MedAI/RAG-OmniQA --repo-type=dataset

Run Evaluation

python eval.py --model-name "Qwen3" --model-path "/path/to/model" --eval-dataset hotpotqa popqa

Customize Configuration: You can modify the configuration files in the config/ directory (e.g., api_prompt_config_en.json) to customize evaluation parameters. Generate Report: After evaluation, HTML reports and JSON results will be saved in the reports/ directory.

For more details, pls see our github repo https://github.com/AQ-MedAI/RagQALeaderboard.

Dataset

Each dataset contains the following fields:

• query: The input question or query.
• groundtruth: The correct answer(s) to the query.
• golden_docs: Documents that contain the evidence or support for the correct answer.
• noise_docs: Distractor documents that are related to the query but do not contain the correct answer.

This structure enables evaluation of both retrieval accuracy and answer generation performance in multi-hop and single-hop reasoning scenarios.

Document Pool

We also provide a unified documents_pool derived from Wikipedia, serving as a retrieval corpus. This pool has been pre-processed using Contriever for initial retrieval, making it efficient and convenient for training and evaluating retrieval models.

The document pool supports plug-and-play integration with standard retrieval and QA pipelines, allowing researchers to perform end-to-end experiments with minimal setup.

Dataset Structure

The dataset files are located inside the final_data folder.

.
├── final_data/
│   ├── 2wiki.jsonl
│   ├── documents_pool.json
│   ├── hotpot_distractor.jsonl
│   ├── musique.jsonl
│   ├── popqa.jsonl
│   ├── pubmed.jsonl
│   └── triviaqa.jsonl
└── README.md

How to Use

You can use the code as below. https://github.com/AQ-MedAI/RagQALeaderboard