🔥 Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation 🔥

ReRanking Datasets : A comprehensive collection of retrieval and reranking datasets with full passage contexts, including titles, text, and metadata for in-depth research.

A curated collection of ready-to-use datasets for retrieval and reranking research. Created by Abdelrahman Abdallah.

About This Repository

Welcome to ReRanking Datasets! This repository simplifies retrieval research by providing preprocessed datasets for commonly used retrievers. Instead of running multiple retrievers on your own, download the precomputed results directly and focus on your research!

What's Inside?

This dataset collection includes popular question-answering datasets:

The following table provides an overview of the availability of different retrieval methods (BM25, DPR, ColBERT, ANCE, BGE, Contriever) for each dataset.

✅ Completed ⏳ Part Completed, Pending other Parts 🕒 Pending

Each dataset has N different files, one for each retriever. These files are organized and ready for direct use, saving you time and computation. All passages retrieved from psgs_w100 you can download them from psgs_w100.tsv

Need Full Context?

If you require the complete dataset, including passage titles and full text, you can find it in the full collection:

👉 ReRanking Contexts

Format

The datasets are provided in JSON format, structured as follows:

[
    {
        "question": "...",
        "answers": ["...", "...", ...],
        "ctxs": [
            {
                "id": "...",         // Passage ID from database TSV file
                "score": "...",      // Retriever score
                "has_answer": true|false  // Whether the passage contains the answer
            }
        ]
    }
]

Ongoing Work

Our goal is to make this repository the go-to resource for retrieval and reranking datasets. Here's what we are currently working on:

• Expanding the dataset collection to include more benchmarks, such as MS MARCO, TREC, and others.
• Adding retrieval outputs from more retrievers, including cutting-edge methods like ColBERT, ANCE, and more.
• Continuously improving dataset organization and documentation to ensure researchers' ease of use.

If you have suggestions for additional datasets or retrievers you'd like to see included, feel free to reach out or contribute!

Why Use This Repository?

• 🚀 Time-Saving: Avoid the hassle of running retrievers yourself.
• 📂 Precomputed Results: Access retrieval outputs for multiple retrievers in one place.
• 🎯 Research-Ready: Use datasets directly for reranking tasks or as baselines in your studies.
• 🛠️ Versatility: Compatible with diverse retrieval and reranking pipelines.

How to Use

Simply visit the dataset page on HuggingFace and download the files for your desired dataset and retriever. Each file is structured to integrate seamlessly into your retrieval or reranking workflows.

Example

from datasets import load_dataset

# Use streaming mode
dataset = load_dataset("abdoelsayed/reranking-datasets-light", data_files={"test": "bm25/nq-test.json"}, streaming=True)

# Iterate through the dataset
for entry in dataset["test"].take(10):  # Process 10 entries as an example
    print(entry)

Contribution & Feedback

We are actively expanding this repository and welcome contributions from the research community. Have a suggestion for a new dataset or retriever? Found an issue? Feel free to open an issue or reach out via Twitter.

🌟 Citation

Please kindly cite our paper if helps your research:

@article{abdallah2025rankify,
  title={Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation},
  author={Abdallah, Abdelrahman and Mozafari, Jamshid and Piryani, Bhawna and Ali, Mohammed and Jatowt, Adam},
  journal={arXiv preprint arXiv:2502.02464},
  year={2025}
}

Star History

🎉 Happy Researching!

Paper: Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation