rank1-training-data: Training Dataset for rank1 Reasoning Rerankers

📄 Paper | 🚀 GitHub Repository

This dataset contains the training data used to develop the rank1 family of reasoning rerankers with LLaMA Factory. It includes query-document pairs with relevance judgments and reasoning chains that guided the models to make binary relevance decisions.

Dataset Description

The rank1-training-data dataset is a comprehensive collection of training examples used to teach rank1 models how to reason about document relevance. Each example contains:

1. A query
2. A document/passage
3. A reasoning chain that analyzes the relevance in a step-by-step manner
4. A binary relevance judgment (true/false)

The dataset is designed to train models to generate explicit reasoning chains before making relevance judgments, which helps break down complex relevance decisions into logical steps.

Dataset Structure

The dataset follows this structure:

{
  "instruction": "The general task instruction",
  "input": "The search query text and passage text",
  "output": "The reasoning chain to generate and have loss on",
}

Associated Models and Resources

Usage Examples

See the Github repo for the training configurations.

Citation

If you use this dataset in your research, please cite:

@misc{weller2025rank1testtimecomputereranking,
      title={Rank1: Test-Time Compute for Reranking in Information Retrieval}, 
      author={Orion Weller and Kathryn Ricci and Eugene Yang and Andrew Yates and Dawn Lawrie and Benjamin Van Durme},
      year={2025},
      eprint={2502.18418},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2502.18418}, 
}

License

MIT License