Dataset Card for RLHN-680K

Dataset Description

Repository | Paper | ArXiv

RLHN is a cascading LLM framework designed to accurately relabel hard negatives in existing IR/RAG training datasets, such as MS MARCO and HotpotQA.

This Tevatron dataset (680K training pairs) contains the queries, positives + relabeled hard negatives, remaining hard negatives for 7 datasets in the BGE training collection.

This repository contains the training pairs that can be used to fine-tune embedding, ColBERT or multi-vector, and reranker models.

The original dataset (bad quality; containing false negatives) can be found at rlhn/default-680K.

Note: RLHN datasets are not new training datasets, but rather existing BGE collection training datasets with hard negatives cleaned!

Dataset Structure

To access the data using HuggingFace datasets:

rlhn = datasets.load_dataset('rlhn/rlhn-680K')

# training set:
for data in freshstack['train']:
    query_id = data["query_id"]                            # md5 hash of the query_id
    query = data["query"]                                  # query text
    subset = data["subset"]                                # training dataset, e.g., fiqa or msmarco_passage

    # positive passages
    for positive_passage in data["positive_passages"]:
        doc_id = positive_passage["docid"]
        title = positive_passage["title"]                  # title is usually empty, added in text
        text = positive_passage["text"]                    # contains both the title & text

    # hard negative passages
    for negative_passage in data["negative_passages"]:
        doc_id = negative_passage["docid"]
        title = negative_passage["title"]                  # title is usually empty, added in text
        text = negative_passage["text"]                    # contains both the title & text

Original Dataset Statistics

The following table contains the number of training pairs for each training dataset included in RLHN. These numbers are for the default setting.

License

The RLHN dataset is made available with the CC-BY-SA 4.0 license.

Hashing & IDs

We generate the md5 hash as the unique identifier (ID) for both the query & documents, using the code below:

import hashlib

def get_md5_hash(text):
  """Calculates the MD5 hash of a given string.
  Args:
    text: The string to hash.
  Returns:
    The MD5 hash of the string as a hexadecimal string.
  """
  text_bytes = text.encode('utf-8')  # Encode the string to bytes
  md5_hash = hashlib.md5(text_bytes).hexdigest()
  return md5_hash

Citation

@misc{thakur2025relabel,
      title={Fixing Data That Hurts Performance: Cascading LLMs to Relabel Hard Negatives for Robust Information Retrieval}, 
      author={Nandan Thakur and Crystina Zhang and Xueguang Ma and Jimmy Lin},
      year={2025},
      eprint={2505.16967},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2505.16967}, 
}