Bright-Pro

Bright-Pro is an expert-annotated extension of the BRIGHT benchmark for reasoning-intensive retrieval. Each query is paired with a multi-aspect reasoning decomposition, weighted importance scores per aspect, and a curated set of gold passages organized by aspect — enabling fine-grained analysis of whether a retriever covers the complementary reasoning aspects required to answer a query, rather than just surfacing one relevant passage.

Bright-Pro builds on the seven StackExchange subsets of BRIGHT. Annotators (1) decomposed each query's information need into reasoning aspects, (2) assigned an importance weight (Likert 1–3) to each aspect, (3) re-audited and consolidated BRIGHT's original positives, (4) collected new aspect-grounded positives, and (5) had a second annotator from the same field re-examine the result.

Subsets and statistics

Bright-Pro covers 739 queries across 7 StackExchange domains with 2,763 reasoning aspects and 5,272 gold passages drawn from a unified corpus of 526,319 documents.

Configurations

Bright-Pro has three configurations. Each configuration has 7 splits, one per StackExchange domain.

examples — query-level annotations

aspects — reasoning-aspect annotations

documents — corpus

Quick start

from datasets import load_dataset

# Per-task loading (any of the 7 SE domains)
examples = load_dataset("yale-nlp/Bright-Pro", "examples", split="biology")
aspects  = load_dataset("yale-nlp/Bright-Pro", "aspects",  split="biology")
docs     = load_dataset("yale-nlp/Bright-Pro", "documents", split="biology")

print(examples[0]["query"])
print(aspects[0]["content"], aspects[0]["weight"])
print(docs[0]["content"][:200])

To recover the per-query aspect weights as probabilities (Σ = 1):

from collections import defaultdict

raw_sum = defaultdict(float)
for a in aspects:
    qstem = a["id"].rsplit("-a", 1)[0]      # e.g. "biology-0"
    raw_sum[qstem] += a["weight"]

aspect_weight = {a["id"]: a["weight"] / raw_sum[a["id"].rsplit("-a", 1)[0]] for a in aspects}

To build the inverse doc_id → aspect_id map for a task:

doc_to_aspect = {}
for a in aspects:
    for d in a["supporting_docs"]:
        doc_to_aspect[d] = a["id"]

Evaluation

Bright-Pro supports two complementary evaluation regimes:

1. Static retrieval. A retriever ranks the per-task corpus once. Primary metric: α-nDCG@k (with α = 0.5) over the aspect-weighted gold set, complemented by Aspect-Recall@k, NDCG@k, and Recall@k. The metric rewards covering complementary aspects rather than over-retrieving from a single one.

2. Agentic retrieval. A retriever is plugged into an LLM agent that iteratively issues search queries and synthesizes a final answer. The agent loop is evaluated under two protocols:

   • Fixed-round (R ∈ {1, 2, 3} rounds, top-5 per round) — measures retriever quality under matched interaction budgets via cumulative α-nDCG@5R, reasoning completeness, and overall answer quality.
   • Adaptive-round — the agent decides when to stop. Measures both answer quality and interaction efficiency via the Efficiency-Quality Reward (AER) = OQ × exp(−γ (R−1)), with γ = 0.05.

Reasoning completeness and overall quality are rated by an LLM-as-Judge against a reference answer constructed from the annotated aspects and their supporting passages.

Differences from BRIGHT

Bright-Pro keeps BRIGHT's queries and corpus URLs but extends the gold-side annotation:

• Aspects. Each query is decomposed into 2–6 reasoning aspects with Likert importance weights (BRIGHT has no aspect annotation).
• Aspect-grounded gold. Original BRIGHT positives are re-audited (some discarded as topic-only), and new aspect-relevant passages are collected from the live web. Each gold passage is tied to exactly one aspect.
• Reference answers. Each query has a reference long-form answer with citations to gold docs, used to drive the LLM-as-Judge evaluation.
• Scope. Bright-Pro covers only the 7 StackExchange domains of BRIGHT — the Coding (LeetCode, Pony) and Theorem (AOPS, TheoremQA) subsets are excluded because they rely on syntactic or formal-logic matching rather than open-domain natural-language reasoning.

License

Released under the MIT License. The underlying StackExchange queries and the BRIGHT corpus retain their original licenses; consult the BRIGHT dataset card for upstream attribution.