rank1-R1-MSMARCO: Reasoning Outputs from MS MARCO Dataset

📄 Paper | 🚀 GitHub Repository

This dataset contains outputs from Deepseek's R1 model on the MS MARCO passage dataset, used to train rank1. It showcases the reasoning chains and relevance judgments generated when determining document relevance for information retrieval queries.

Dataset Description

The rank1-R1-MSMARCO dataset consists of reasoning chains and relevance judgments produced on the MS MARCO passage dataset. The dataset includes:

1. Queries from MS MARCO
2. Passages/documents from MS MARCO
3. Reasoning outputs in two formats:
   • Raw reasoning outputs (r1_output)
   • Cleaned reasoning outputs (clean_r1_output)
4. Binary relevance judgments (true or false)

Dataset Structure

Based on the screenshot, the dataset includes the following columns:

The dataset contains 635k rows in the training split.

Associated Models and Resources

Usage Examples

Loading and Exploring the Dataset

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("jhu-clsp/rank1-r1-msmarco")

# Access the train split
train_data = dataset["train"]

# Examine sample entries
for i in range(3):
    sample = train_data[i]
    print(f"Query: {sample['query']}")
    print(f"Document text: {sample['text'][:100]}...")
    print(f"Reasoning: {sample['r1_output'][:100]}...")
    print(f"Relevance judgment: {sample['answer']}")
    print("---")

# Filter for specific query types
health_queries = [item for item in train_data if "symptoms" in item["query"].lower()]

Citation

If you use this dataset in your research, please cite:

@misc{weller2025rank1testtimecomputereranking,
      title={Rank1: Test-Time Compute for Reranking in Information Retrieval}, 
      author={Orion Weller and Kathryn Ricci and Eugene Yang and Andrew Yates and Dawn Lawrie and Benjamin Van Durme},
      year={2025},
      eprint={2502.18418},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2502.18418}, 
}

License

MIT License