REPAIR: Resolving Long-Tail Confusion in Scientific Retrievers via Fact-Verified Iterative Refinement

EMNLP 2026  |  Paper  |  Models

---

Overview

REPAIR (REtriever via Epistemic API-Guided Iterative Refinement) is a self-evolving data augmentation framework for scientific dense retrievers. It iteratively synthesizes training data to address long-tail concept confusion and high fact-sensitivity , two structural properties that fundamentally distinguish scientific literature from general-domain text.

❓ Quick quiz: BCL1 or BCL2? If you hesitated, congrats..
you're performing at the level of a state-of-the-art scientific retriever 😅

✨ Findings:

1. Scientific retrievers don't fail because they're too small.
2. The retriever knows where it's weak.
3. In science, what matters most is fact-checking the long tail and actually training on it.

Key Ideas

REPAIR iterates three stages: (1) Diagnosis identifies long-tail distractor concepts that the retriever confuses; (2) Expansion grounds concepts in externally verified evidence from scientific APIs; (3) Differentiation resolves fine-grained factual distinctions via hard negative mining.

---

🤗 Pretrained Models

REPAIR retrievers are released as LoRA adapters on Qwen2.5 (r=16, α=32). Load them with peft; the base model is downloaded automatically.

nDCG@10 for retrieval benchmarks, Spearman's ρ for BIOSSES. Full comparison against 19 baselines is in the paper.

import torch, torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
from peft import PeftModel

ADAPTER = "yerim0210/REPAIR_Scientific_Retrievers_1.5B"
BASE    = "Qwen/Qwen2.5-1.5B"

tok   = AutoTokenizer.from_pretrained(ADAPTER)
model = PeftModel.from_pretrained(AutoModel.from_pretrained(BASE, dtype=torch.bfloat16), ADAPTER).eval().cuda()
if tok.pad_token is None:
    tok.pad_token, tok.padding_side = tok.eos_token, "left"

def last_token_pool(h, mask):
    if mask[:, -1].sum() == mask.shape[0]:
        return h[:, -1]
    return h[torch.arange(h.shape[0], device=h.device), mask.sum(1) - 1]

@torch.no_grad()
def encode(texts, max_length=512):
    b = tok(texts, max_length=max_length - 1, truncation=True, padding=False,
            return_attention_mask=False, return_token_type_ids=False)
    b["input_ids"] = [ids + [tok.eos_token_id] for ids in b["input_ids"]]   # EOS pooling token
    b = tok.pad(b, padding=True, return_attention_mask=True, return_tensors="pt").to("cuda")
    return F.normalize(last_token_pool(model(**b, use_cache=False).last_hidden_state, b["attention_mask"]), dim=-1)

task = "Given a query, retrieve passages that are relevant to the query."
q = encode([f"{task}\nQuery: Which protein does venetoclax inhibit?"])
d = encode([f"Represent this passage.\nPassage: {p}" for p in [
    "Venetoclax is a selective BCL-2 inhibitor approved for chronic lymphocytic leukemia.",
    "BCL1 (cyclin D1) rearrangement t(11;14) is the hallmark of mantle cell lymphoma.",
]])
print(q @ d.T)   # BCL-2 passage scores higher

Query / passage formats follow eval/: queries are prefixed with a one-line task instruction and Query:, passages with Represent this passage.\nPassage:. Each model card lists the per-benchmark instructions.

---

Installation

git clone https://github.com/yerimoh/REPAIR.git
cd REPAIR
pip install -r requirements.txt

Requirements

• Python ≥ 3.9
• PyTorch ≥ 2.0 with CUDA ≥ 11.8 (Single GPU required — DataParallel is not supported due to DynamicCache)
• Hugging Face transformers, peft
• faiss-gpu for efficient nearest-neighbor search

pip install torch>=2.0 transformers peft faiss-gpu tqdm requests

---

Repository Structure

REPAIR/
├── diagnosis/                          # Stage I — Diagnosis
│   ├── 01_diagnosis_margin.py          #   Cosine-margin computation → Qconf
│   ├── 02_pathA_ccs_regex.py           #   Intra-query CCS (regex entity extraction) → Cintra
│   ├── 02_pathA_ccs_matdetector.py     #   Intra-query CCS (MatDetector extraction) → Cintra
│   ├── 03_pathB_finch.py               #   Inter-query FINCH clustering → Cinter
│   ├── 04_merge_cconf.py               #   Cconf = Cintra ∪ Cinter
│   └── utils_matdetector.py            #   MatDetector entity extraction utilities
│
├── expansion/                          # Stage II — Expansion
│   ├── 05_query_generation.py          #   LLM-based confusing query generation
│   ├── 06_semantic_scholar.py          #   Semantic Scholar API retrieval & verification
│   └── 06_elsevier.py                  #   Elsevier API retrieval & verification
│
├── differentiation/                    # Stage III — Differentiation
│   ├── 07_embed_candidates.py          #   Embed expansion candidate documents
│   ├── 08_hard_negative_selection.py   #   Model-aware hard negative selection (Eq. 7)
│   └── 09_consistency_filter.py        #   Consistency filter (Eq. 6) + single d⁻
│
├── eval/                               # Evaluation scripts
│   ├── eval_mesh.py
│   └── test_data/
│
├── run_repair.sh                       # End-to-end pipeline runner
└── requirements.txt

---

Methodology

Retriever Formulation

REPAIR uses a shared decoder-only LM backbone $M_\theta$ (Qwen2.5 at 500M / 1.5B / 7B) with EOS pooling:

$$e_{q/d} = \text{Pool}{\text{EOS}}!\left(M\theta!\left(q/d \oplus [\text{EOS}]\right)\right)$$

Relevance is scored by dot product $s_\theta(q, d) = e_q^\top e_d$, and the retriever returns $\text{TopK}\theta(q) \subset \mathcal{D}$ under $s\theta$.

---

Stage I — Diagnosis

Goal: Identify the long-tail scientific concepts responsible for retrieval confusion.

Step 1 — Low-Margin Query Selection

For each training query $q$ with ground-truth positive $d^+$, compute the confusion margin:

$$\Delta_\theta(q) = s_\theta(q, d^+) - \max_{d \neq d^+} s_\theta(q, d)$$

Queries in the bottom percentile of $\Delta_\theta$ form the confusion set $\mathcal{Q}_\text{conf}$.

python diagnosis/01_diagnosis_margin.py
# → confused_train_queries.bottom_30pct.cos.jsonl

Step 2 — Intra-Query Distractor Mining (PathA → Cintra)

Aggregate over the confusion set: $$D^+ = {d^+(q)}{q \in \mathcal{Q}\text{conf}}, \qquad D^- = \bigcup_{q \in \mathcal{Q}_\text{conf}} D^-(q)$$

Score each extracted concept $e \in \mathcal{E}$ with the Confusing Concept Score (CCS) [Eq. 5]:

$$\text{CCS}(e) = \frac{df(e;, D^-)}{df(e;, D^+) + \varepsilon}$$

where $df(e; \cdot)$ is the document frequency of $e$ in the set, and $\varepsilon > 0$ prevents division by zero. A high CCS identifies concepts that frequently distract the retriever (appear in $D^-$) but are rare in ground-truth positives ($D^+$).

Cintra is constructed by selecting the highest-CCS concept per query from its $D^-(q)$.

python diagnosis/02_pathA_ccs_regex.py \
    --pct 30 \
    --dia_out_dir <diagnosis_output_dir> \
    --train_pt    <train_data.pt> \
    --out_dir     <patha_output_dir> \
    --top_neg_n   30
# → cintra.bottom_30pct.jsonl
# → per_query_neg_concepts.bottom_30pct.json  (used by PathB)

Step 3 — Inter-Query Distractor Mining (PathB → Cinter)

Cluster $\mathcal{Q}_\text{conf}$ using the FINCH algorithm (Sarfraz et al., 2019):

1. For each query $i$, find its first nearest neighbor $\text{nn}(i) = \arg\max_{j \neq i} \cos(e_i, e_j)$
2. Build an undirected graph with edges $i \leftrightarrow \text{nn}(i)$
3. Connected components of the graph = FINCH clusters

This approach is parameter-free — no cluster count $k$ is needed. From each resulting cluster, aggregate the candidate concepts extracted from member queries and select the most frequent one → Cinter.

python diagnosis/03_pathB_finch.py \
    --pct               30 \
    --dia_out_dir       <diagnosis_output_dir> \
    --out_dir           <pathb_output_dir> \
    --train_query_cache <query_cache.normfp16.pt> \
    --patha_out_dir     <patha_output_dir> \
    --device            cuda
# → pathB_finch_train.components.bottom_30pct.json  (Cinter)

Step 4 — Distractor Concept Set

$$\mathcal{C}\text{conf} = \mathcal{C}\text{intra} \cup \mathcal{C}_\text{inter}$$

python diagnosis/04_merge_cconf.py --pct 30
# → final_terms.bottom_30pct.txt   (Cconf)
# → final_terms.bottom_30pct.json

---

Stage II — Expansion

Goal: Ground each distractor concept $c \in \mathcal{C}\text{conf}$ in externally fact-verified evidence, yielding validated tuples $(q\text{new}, d^+, D^-_\text{cand})$.

1. Concept Grounding: Retrieve physical/chemical attributes (synonyms, molecular weight, crystal structure, etc.) from PubChem and Materials Project for each $c$.
2. Confusing Query Generation: Sample 1–3 grounded concepts and prompt $M_\theta$ to generate a candidate query $q_\text{model}$ that it finds inherently ambiguous.
3. Verification via External APIs: Query Semantic Scholar / Elsevier with $q_\text{model}$. Discard if no results are returned. For valid searches:
   • Top-matching document title → verified query $q_\text{new}$
   • Top-matching document content → positive $d^+$
   • Remaining highly similar documents → negative candidates $D^-_\text{cand}$

Generation continues until the number of valid tuples reaches $2 \times |\mathcal{T}_0|$.

# LLM-based query generation from Cconf
python expansion/05_query_generation.py

# Semantic Scholar API retrieval & verification
python expansion/06_semantic_scholar.py \
    --concept_txt <final_terms.bottom_30pct.txt> \
    --key_file    <semantic_api_keys.txt> \
    --out_jsonl   <expansion.jsonl> \
    --target_k    20 \
    --resume

---

Stage III — Differentiation

Goal: Construct training triplets $(q_\text{new}, d^+, d^-)$ using a single model-aware hard negative.

Consistency Filtering [Eq. 6]

Define a local candidate pool $\mathcal{D}\text{pool} = {d^+} \cup D^-\text{cand}$. Retain a tuple only if the current retriever ranks $d^+$ in the top $\kappa$ positions of $\mathcal{D}_\text{pool}$:

$$\mathbf{1}\text{keep}(q\text{new}) = \mathbf{1}!\left[\text{rank}!\left(d^+ \mid q_\text{new};, \mathcal{D}_\text{pool}\right) \leq \kappa\right]$$

This ensures queries are answerable before hard negative mining proceeds.

Single Hard Negative Selection [Eq. 7]

Select the hardest negative from the API-verified candidates:

$$d^- = \arg\max_{d \in D^-\text{cand}} s\theta(q_\text{new}, d)$$

Using a single hard negative yields a more semantically meaningful decision boundary than multiple easy negatives.

# Embed expansion candidates
python differentiation/07_embed_candidates.py \
    --input_jsonl <expansion.jsonl> \
    --output_dir  <embed_cache/> \
    --num_gpus    1

# Model-aware hard negative selection
python differentiation/08_hard_negative_selection.py \
    --input_jsonl   <expansion.jsonl> \
    --embedding_dir <embed_cache/> \
    --output_jsonl  <hard_negatives.jsonl> \
    --top_k         15

# Consistency filter + single d⁻ selection
python differentiation/09_consistency_filter.py \
    --input_jsonl   <hard_negatives.jsonl> \
    --embedding_dir <embed_cache/> \
    --output_jsonl  <final_train_data.jsonl> \
    --top_k         15 \
    --consistency_k 5

---

Iterative Contrastive Optimization

The retriever is updated via InfoNCE contrastive loss:

$$\mathcal{L}(q_\text{new}) = -\log \frac{\exp!\left(s_\theta(q_\text{new}, d^+)/\tau\right)}{\displaystyle\sum_{d \in {d^+} \cup \mathcal{N}} \exp!\left(s_\theta(q_\text{new}, d)/\tau\right)}$$

After each update, the margin landscape ${\Delta_\theta(q)}$ is recomputed and Stages I–III are repeated. REPAIR uses 2 iterations by default; performance continues to improve beyond that, but with diminishing returns.

---

End-to-End Pipeline

Run the full Diagnosis → Expansion → Differentiation pipeline:

# Default (PCT=30, single GPU)
sr 1 48 --exclude=hockney --qos=q-high-yerim.oh bash run_repair.sh

# Adjust bottom-percentile threshold
PCT=20 bash run_repair.sh

Pipeline Steps

Key Hyperparameters

---

Evaluated Benchmarks

---

Citation

If you find REPAIR useful in your research, please cite:

@misc{oh2026repairresolvinglongtailconfusion,
      title={REPAIR: Resolving Long-Tail Confusion in Scientific Retrievers via Fact-Verified Iterative Refinement},
      author={Yerim Oh and Gunhee Kim},
      year={2026},
      booktitle={Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
      eprint={2609.18262},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2609.18262},
}

---

License

This project is licensed under the MIT License. See LICENSE for details.

---

Acknowledgements

• FINCH Clustering (Sarfraz et al., 2019) — parameter-free clustering used in inter-query distractor mining
• Semantic Scholar Open Research Corpus — external evidence retrieval for fact verification
• PubChem & Materials Project — concept grounding metadata
• Qwen2.5 — LLM backbone for the REPAIR retriever
• BMRetriever (Xu et al., 2024) — baseline model and evaluation framework