RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark

A benchmark for evaluating reasoning-intensive conversational information retrieval systems.

Statistics

Domains

Per-Domain Statistics

Dataset Structure

Subsets and Splits

Each subset has 11 domain splits:

Data Fields

Benchmark (conversations):

Metadata fields:

Each turn contains:

subquestion_reasoning_metadata:

Corpus (documents):

Usage

from datasets import load_dataset

# Load a specific domain
biology_benchmark = load_dataset("RECOR-Benchmark/RECOR", "benchmark", split="biology")
biology_corpus = load_dataset("RECOR-Benchmark/RECOR", "corpus", split="biology")

# Load all domains for a subset
all_benchmarks = load_dataset("RECOR-Benchmark/RECOR", "benchmark")  # Returns all splits
all_corpus = load_dataset("RECOR-Benchmark/RECOR", "corpus")  # Returns all splits
# Access specific domain: all_benchmarks["biology"], all_corpus["economics"], etc.

# Get relevant documents for a conversation turn
conv = biology_benchmark[0]
turn = conv["turns"][0]
gold_ids = turn["gold_doc_ids"]
relevant_docs = [doc for doc in biology_corpus if doc["doc_id"] in gold_ids]

# Available splits (domains):
# biology, earth_science, economics, psychology, robotics, sustainable_living
# Drones, hardware, law, medicalsciences, politics

Iterate Through Conversations

for conv in biology_benchmark:
    print(f"ID: {conv['id']}")
    for turn in conv["turns"]:
        print(f"  Turn {turn['turn_id']}: {turn['query']}")

File Structure

data/
├── benchmark/           # Conversations (11 files)
│   └── {domain}_benchmark.jsonl
└── corpus/              # Documents (11 files)
    └── {domain}_documents.jsonl

Evaluation

For evaluation code and metrics, see the GitHub Repository.

License

MIT License