BRIGHT-Pro × MTEB integration: cross-validation

Companion repository for embeddings-benchmark/mteb#4651 — the PR adding BRIGHT-Pro as a retrieval benchmark in MTEB.

This repo cross-validates that running 4 retrievers through the new MTEB task classes (BrightPro{Domain}Retrieval) reproduces the numbers from BRIGHT-Pro's own evaluation harness on all 7 StackExchange domains.

Setup

Each (model × domain) cell is evaluated end-to-end via:

import mteb
mteb.evaluate(
    mteb.get_model_meta(model_name),
    mteb.get_task(task_name),
    encode_kwargs={"batch_size": ...},
)

with the per-model max_seq_length aligned to BRIGHT-Pro's original settings:

The MTEB task prompt is set to "Given a {domain} post, retrieve relevant passages that help answer the post" — the same instruction body BRIGHT-Pro's paper uses, so the Qwen3 / ReasonIR wrappers format it into the identical Instruct:…\nQuery: / <|user|>…\n<|embed|>\n prefix as the BRIGHT-Pro harness.

Results

nDCG@10 per (domain, model). Δ = MTEB − BRIGHT-Pro.

Per-model summary (7 domains each)

The three dense-embedding models reproduce BRIGHT-Pro's numbers within 3% on every cell, with two of them sub-1%. This confirms the MTEB task classes (BrightPro{Domain}Retrieval) pass identical corpora, queries, and qrels to the evaluation pipeline as BRIGHT-Pro's own harness — any remaining differences are attributable to model-side details (precision, EOS handling, tokenizer behaviour), not to the task integration.

Re-validation on the mteb 2.18.1 rebase (2026-07-14)

After rebasing the PR branch onto mteb main (2.18.1, which now includes the #4684 fix), all 4 retrievers were re-run end-to-end on the 7 BrightPro tasks. Dense models reproduce BRIGHT-Pro harness scores with mean |dnDCG@10| = 0.0029 (max 0.0152); ReasonIR-8B is within +/-0.005 on every domain. Full table: comparison_2p18.md, per-cell outputs in results_2p18/.

Root cause analysis of the ReasonIR-8B gap

The ~1–3 nDCG-point gap for ReasonIR-8B above is not noise. We traced it to a separate MTEB bug, independent of this BrightPro task integration, in mteb/models/abs_encoder.py's get_task_instruction:

if self.instruction_template and len(instruction) > 0:
    return self.format_instruction(instruction, prompt_type)
return instruction

When a task defines prompt={"query": "..."} without a "document" key (BRIGHT-Pro, BRIGHT, BRIGHT v1.1, and many other retrieval tasks), the document-side instruction is "". The len(instruction) > 0 gate then causes get_task_instruction to return "" directly — the model's instruction_template is never invoked for documents.

For models whose instruction_template is a callable that emits a non-empty prefix on empty input — most prominently ReasonIR-8B and GritLM-7B (<|embed|>\n), also Octen, Sarashina v2 (text: ), BMRetriever — the document-side prefix the model was trained with is silently dropped. Documents get encoded without the prefix.

Diagnostics

The scripts/verify_reasonir_*.py + results/diagnostics/verify_reasonir_*.json files trace the gap step by step. The decisive observations on BrightProBiologyRetrieval:

Empirical impact of the fix

ReasonIR-8B × BrightProBiologyRetrieval, full 60K corpus:

The fix narrows the MTEB-vs-paper gap from +0.020 nDCG to +0.009 nDCG. The residual ~0.009 is implementation-level noise (sentence-transformers' tokenizer/padding vs HuggingFace's custom .encode, bf16 numerical precision).

A separate MTEB issue + PR for this bug will reference this analysis.

Layout

scripts/
  run_mteb_eval.py     # single (model, task) evaluation
  submit_eval.sh       # SLURM script (GPU)
  submit_eval_cpu.sh   # SLURM script (BM25, CPU)
  dispatch_all.sh      # fire one job per (model, task)
  build_comparison.py  # aggregate JSON results -> markdown table
results/
  BrightPro{Domain}Retrieval__{model_slug}.json  # one per cell
  diagnostics/
    verify_reasonir_*.json  # gap-investigation artifacts

Reproducing

# one (model, task) cell
MODEL_NAME=Qwen/Qwen3-Embedding-8B \
TASK_NAME=BrightProBiologyRetrieval \
MAX_SEQ_LENGTH=4096 BATCH_SIZE=16 \
python scripts/run_mteb_eval.py