❗Important❗

Due to legal reasons, we cannot rehost the original positive documents for the hard-query (HQ) data, so we provide a data processing script to download and merge them below.

ReasonIR Dataset

This dataset contains synthetic examples used to train ReasonIR-8B.

• Paper: https://arxiv.org/abs/2504.20595
• Code: https://github.com/facebookresearch/ReasonIR
• Model: https://huggingface.co/reasonir/ReasonIR-8B

Varied-Length (VL) Data

For varied-length (VL) data, we provide the full dataset with (query, positive doc, negative doc). To load the VL dataset, simply:

from datasets import load_dataset
vl_dataset = load_dataset("reasonir/reasonir-data", "vl")

Hard-query (HQ) Data

For hard-query (HQ) data, we provide the dataset with (query, positive doc identifier, negative doc). Therefore, to load the HQ dataset, both our repository and the original datastore from BRIGHT need to be loaded. We provide a simple script below to combine the two:

from datasets import load_dataset

def get_doc_and_ids(doc_pairs):
    doc_ids = []
    documents = []
    for dp in doc_pairs:
        doc_ids.append(str(dp['id']))
        documents.append(dp['content'])
    return documents, doc_ids
    
def process_pos_id2doc(entry, id2doc):
    pos_docs = entry["pos"]
    res = []
    for pos in pos_docs:
        instruction, doc_id = pos[0], pos[1]
        doc = id2doc[doc_id]
        res.append([instruction, doc])
    entry["pos"] = res
    return entry

hq_dataset = load_dataset("reasonir/reasonir-data", "hq")
bright_docs = load_dataset("xlangai/BRIGHT", "documents")
all_docs = []   
all_ids = []
for task in bright_docs.keys():
    docs, ids = get_doc_and_ids(bright_docs[task])
    all_docs.extend(docs)
    all_ids.extend(ids)

id2doc = {}
for i in range(len(all_docs)):
    id2doc[all_ids[i]] = all_docs[i]

hq_dataset = hq_dataset.map(lambda x: process_pos_id2doc(x, id2doc))