BRIGHT benchmark

BRIGHT is the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. The queries are collected from diverse domains (StackExchange, LeetCode, and math competitions), all sourced from realistic human data. Experiments show that existing retrieval models perform poorly on BRIGHT, where the highest score is only 22.1 measured by nDCG@10. BRIGHT provides a good testbed for future retrieval research in more realistic and challenging settings. More details are in the paper.

Dataset Structure

We unify all the datasets with consistent formats. They are organized into three subsets, examples, documents, and long_documents:

• examples:
  • query: the query for retrieval
  • reasoning: the gold reasoning steps annotated by humans (they help people understand the relevance between queries and documents, but are not used in any experiment in the paper)
  • id: the index of the instance
  • excluded_ids: a list of the ids (string) to exclude during evaluation (only for theoremqa/aops/leetcode)
  • gold_ids_long: a list of the ids (string) of the ground truth documents, corresponding to the ids of the long_documents subset
  • gold_ids: a list of the ids (string) of the ground truth documents, corresponding to the indices of the documents subset
• documents:
  • id: the index of the document
  • content: document content (short version split from the complete web page, blogs, etc., or a problem and solution pair)
• long_documents (not applicable to theoremqa/aops/leetcode):
  • id: the index of the document
  • content: document content (long version corresponding to the complete web page, blogs, etc.)

Dataset Statistics

Data Loading

Each dataset can be easily loaded. For example, to load biology examples:

from datasets import load_dataset
data = load_dataset('xlangai/BRIGHT', 'examples')['biology']

Citation

If you find our work helpful, please cite us:

@misc{BRIGHT,
  title={BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval},
  author={Su, Hongjin and Yen, Howard and Xia, Mengzhou and Shi, Weijia and Muennighoff, Niklas and Wang, Han-yu and Liu, Haisu and Shi, Quan and Siegel, Zachary S and Tang, Michael and Sun, Ruoxi and Yoon, Jinsung and Arik, Sercan O and Chen, Danqi and Yu, Tao},
  url={https://arxiv.org/abs/2407.12883},
  year={2024},
}