RAG-QA Leaderboard
See the codeA standardized and fair evaluation framework and leaderboard for Retrieval-Augmented Generation (RAG) systems.
RAGQA-Leaderboard aims to provide researchers and developers with a unified and reproducible benchmark for evaluating the performance of RAG models. We have integrated a suite of popular and high-frequency question-answering datasets and offer a streamlined, one-click evaluation pipeline that generates detailed reports, making model comparison and analysis easier than ever.
📊 Standardized Evaluation Framework: Provides a unified and fair evaluation pipeline, ensuring that different models are compared under the same conditions for reproducible results.
📚 Comprehensive Dataset Integration:
HotpotQA, PopQA, MusiqueQA, TriviaQA, and more.📈 Multi-Dimensional Metrics:
📄 One-Click Reporting:
Generate comprehensive evaluation reports with a single command.
Outputs reports in both HTML for easy visualization and analysis, and JSON for programmatic use. This makes it effortless to analyze and compare performance across different models.
🧩 Modular RAG Evaluation: Go beyond end-to-end testing. This framework allows for the isolated evaluation of individual RAG components—such as the Retriever and the Generator—enabling targeted analysis and debugging.
🚀 Flexible Model Inference:
Our comprehensive evaluation dataset, RAG-QA, is publicly available on the Hugging Face Hub.
![RAG-QA-Leaderboard]
Clone the repository:
git clone https://github.com/AQ-MedAI/RagQALeaderboard
cd RagQALeaderboard/
Install dependencies:
pip install -r requirements.txt
Download Data
# Make sure hf CLI is installed: pip install -U "huggingface_hub[cli]"
hf download AQ-MedAI/RAG-OmniQA --repo-type=dataset
transformers or vllm):pip install transformers vllm
pip install openai
To evaluate a model, use the following make command:
export EVAL_MODEL=<model_path>
make eval-all
This will evaluate the model for all datasets specified in the Makefile.
If you want to evaluate a specific dataset, use:
export EVAL_MODEL=<model_path>
make eval-single DATASETS="hotpotqa popqa"
Alternatively, you can use the Python script directly:
python eval.py --model-name "Qwen3" --model-path "/path/to/model" --eval-dataset hotpotqa popqa
if you want to run with api:
python eval.py --model-name <model_name> --model-path <api_url> --api-key <api_key>
You can modify the configuration files in the config/ directory (e.g., api_prompt_config_en.json) to customize evaluation parameters.
After evaluation, HTML reports and JSON results will be saved in the reports/ directory.
You can also run the following command to get
python get_report.py --result-dir <your_result_dir>
RAGQA-Leaderboard/
├── src/ # Core source code
│ ├── data.py # Data processing module
│ ├── report/ # Report generation module
│ ├── models/ # Model interface module
│ ├── eval_main.py # Evaluation entry point
│ └── logger.py # Logging module
├── data/ # Datasets
├── config/ # Configuration files
├── tests/ # Unit tests
├── reports/ # Output directory for evaluation reports
├── eval.py # Main evaluation script
├── README.md # Project documentation
└── Makefile # Automation scripts
We have curated a comprehensive dataset by collecting and processing popular question-answering (QA) datasets. Our processing pipeline ensures that each question is paired with its corresponding golden document(s) and a set of noise documents (approximately 50). Additionally, we provide a large-scale retrieval pool consisting of approximately 1.09 million documents from Wikipedia.
The detailed construction process is as follows:
We collected a total of 30,135 queries from three main categories of QA datasets:
We employed Contriever-MS as our retriever. For each question, we retrieved the top 50 documents from our Wikipedia corpus. These retrieved documents, after removing any exact matches with the corresponding golden document(s), constitute the set of noise documents.
The final retrieval pool was constructed by merging all golden and noise documents from the entire collection and then performing a final deduplication to ensure a unique set of documents.
config/api_prompt_config_en.json: Default configuration for English evaluation.config/api_prompt_config_ch.json: Default configuration for Chinese evaluation.config/default_prompt_config.json: General configuration file.@misc{RagQALeaderboard2026,
author = {AQ-MedAI},
title = {RagQALeaderboard: A Comprehensive Leaderboard for RAG-based Medical Question Answering},
year = {2026},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/AQ-MedAI/RagQALeaderboard}}
}
Run the following command to execute the test cases:
pytest tests/
We gratefully acknowledge the creators and maintainers of the publicly available datasets integrated into RAGQA-Leaderboard. Specifically:
These datasets are copyright of their respective authors and we use them solely for research and non-commercial evaluation purposes. Please cite their works appropriately if you use the leaderboard or these datasets in your own publication.
This project is licensed under the Apache License 2.0. See the LICENSE file for details.
We welcome contributions! Feel free to submit issues and pull requests. For major changes, please open an issue first to discuss what you would like to change.
RAG-QA Leaderboard
See the codeA standardized and fair evaluation framework and leaderboard for Retrieval-Augmented Generation (RAG) systems.
RAGQA-Leaderboard aims to provide researchers and developers with a unified and reproducible benchmark for evaluating the performance of RAG models. We have integrated a suite of popular and high-frequency question-answering datasets and offer a streamlined, one-click evaluation pipeline that generates detailed reports, making model comparison and analysis easier than ever.
📊 Standardized Evaluation Framework: Provides a unified and fair evaluation pipeline, ensuring that different models are compared under the same conditions for reproducible results.
📚 Comprehensive Dataset Integration:
HotpotQA, PopQA, MusiqueQA, TriviaQA, and more.📈 Multi-Dimensional Metrics:
📄 One-Click Reporting:
Generate comprehensive evaluation reports with a single command.
Outputs reports in both HTML for easy visualization and analysis, and JSON for programmatic use. This makes it effortless to analyze and compare performance across different models.
🧩 Modular RAG Evaluation: Go beyond end-to-end testing. This framework allows for the isolated evaluation of individual RAG components—such as the Retriever and the Generator—enabling targeted analysis and debugging.
🚀 Flexible Model Inference:
Our comprehensive evaluation dataset, RAG-QA, is publicly available on the Hugging Face Hub.
![RAG-QA-Leaderboard]
Clone the repository:
git clone https://github.com/AQ-MedAI/RagQALeaderboard
cd RagQALeaderboard/
Install dependencies:
pip install -r requirements.txt
Download Data
# Make sure hf CLI is installed: pip install -U "huggingface_hub[cli]"
hf download AQ-MedAI/RAG-OmniQA --repo-type=dataset
transformers or vllm):pip install transformers vllm
pip install openai
To evaluate a model, use the following make command:
export EVAL_MODEL=<model_path>
make eval-all
This will evaluate the model for all datasets specified in the Makefile.
If you want to evaluate a specific dataset, use:
export EVAL_MODEL=<model_path>
make eval-single DATASETS="hotpotqa popqa"
Alternatively, you can use the Python script directly:
python eval.py --model-name "Qwen3" --model-path "/path/to/model" --eval-dataset hotpotqa popqa
if you want to run with api:
python eval.py --model-name <model_name> --model-path <api_url> --api-key <api_key>
You can modify the configuration files in the config/ directory (e.g., api_prompt_config_en.json) to customize evaluation parameters.
After evaluation, HTML reports and JSON results will be saved in the reports/ directory.
You can also run the following command to get
python get_report.py --result-dir <your_result_dir>
RAGQA-Leaderboard/
├── src/ # Core source code
│ ├── data.py # Data processing module
│ ├── report/ # Report generation module
│ ├── models/ # Model interface module
│ ├── eval_main.py # Evaluation entry point
│ └── logger.py # Logging module
├── data/ # Datasets
├── config/ # Configuration files
├── tests/ # Unit tests
├── reports/ # Output directory for evaluation reports
├── eval.py # Main evaluation script
├── README.md # Project documentation
└── Makefile # Automation scripts
We have curated a comprehensive dataset by collecting and processing popular question-answering (QA) datasets. Our processing pipeline ensures that each question is paired with its corresponding golden document(s) and a set of noise documents (approximately 50). Additionally, we provide a large-scale retrieval pool consisting of approximately 1.09 million documents from Wikipedia.
The detailed construction process is as follows:
We collected a total of 30,135 queries from three main categories of QA datasets:
We employed Contriever-MS as our retriever. For each question, we retrieved the top 50 documents from our Wikipedia corpus. These retrieved documents, after removing any exact matches with the corresponding golden document(s), constitute the set of noise documents.
The final retrieval pool was constructed by merging all golden and noise documents from the entire collection and then performing a final deduplication to ensure a unique set of documents.
config/api_prompt_config_en.json: Default configuration for English evaluation.config/api_prompt_config_ch.json: Default configuration for Chinese evaluation.config/default_prompt_config.json: General configuration file.@misc{RagQALeaderboard2026,
author = {AQ-MedAI},
title = {RagQALeaderboard: A Comprehensive Leaderboard for RAG-based Medical Question Answering},
year = {2026},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/AQ-MedAI/RagQALeaderboard}}
}
Run the following command to execute the test cases:
pytest tests/
We gratefully acknowledge the creators and maintainers of the publicly available datasets integrated into RAGQA-Leaderboard. Specifically:
These datasets are copyright of their respective authors and we use them solely for research and non-commercial evaluation purposes. Please cite their works appropriately if you use the leaderboard or these datasets in your own publication.
This project is licensed under the Apache License 2.0. See the LICENSE file for details.
We welcome contributions! Feel free to submit issues and pull requests. For major changes, please open an issue first to discuss what you would like to change.