CoQuIR workspace

Everything needed to reproduce the CoQuIR leaderboard and then attack it with a method paper. Start with notes/00_benchmark_card.md, then notes/02_method_plan.md.

Code_Retrieval/
├── papers/          arXiv 2506.11066v2 (PDF + extracted text) and the ACL 2026 version
├── CoQuIR/          upstream repo (TRUMANCFY/CoQuIR)
├── mteb-coquir/     the authors' mteb fork, where the seven tasks are defined
├── data/            the 7 evaluation sets from https://huggingface.co/CoQuIR (~330 MB)
├── baselines/       the paper's Tables 2 and 3 as CSV — the numbers to beat
├── scripts/         self-contained evaluation harness
├── kaggle/          sharded sweep runner for the Kaggle P100 fleet — see kaggle/README.md
├── runs/            evaluation outputs (results.json + table.md per run)
└── notes/           benchmark card, leaderboard, method plan, paper prompts

Published to Kaggle so any account in the fleet can attach them: trungkiet/coquir-data (the seven evaluation sets) and trungkiet/coquir-harness (this harness plus the sweep runner). Both are public — that is what lets 17 separate accounts attach them without per-account sharing.

Setup

pip install torch transformers sentence-transformers numpy      # core
pip install bm25s PyStemmer                                     # lexical baseline
pip install pdfplumber                                          # only for extract_paper_tables.py
python scripts/download_data.py                                 # ~330 MB, idempotent
python scripts/coquir_data.py                                   # sanity: prints the subset inventory

data/ is not in git — re-fetch it with download_data.py.

Running an evaluation

# lexical baseline, all seven datasets
python scripts/run_eval.py --model bm25s --device cpu

# dense retriever with the model's own prefixes
python scripts/run_eval.py --model st:intfloat/e5-large-v2 \
    --query-prefix "query: " --doc-prefix "passage: "

# HF AutoModel with explicit pooling (CodeBERT / CodeSage / RepLLaMA style)
python scripts/run_eval.py --model hf:microsoft/graphcodebert-base --pooling cls

# instruction-following, and the instruction-sensitivity probe
python scripts/run_eval.py --model st:hkunlp/instructor-large --instruction pos
python scripts/run_eval.py --model st:hkunlp/instructor-large --instruction neg

# smoke test in under a minute
python scripts/run_eval.py --model bm25s --device cpu \
    --tasks Defects4JPreferenceRetrieval --max-queries 50

Each run writes runs/<name>/results.json (per-subset detail) and runs/<name>/table.md (the four-metric summary in the paper's column order).

Why a custom harness instead of the authors' fork

The fork pins an old mteb API, re-downloads from the Hub on every run, and keeps PPA/MRS in a notebook rather than in the evaluator. scripts/ reads the local JSONL directly and computes all four metrics in one pass, reproducing the fork's per-language subset splitting and the notebook's exact rank-based PPA/MRS definition. The details that change the numbers — top_k=200 being part of the metric, and out-of-top-k pairs scoring 0 rather than 0.5 — are documented in scripts/coquir_metrics.py and the benchmark card.

Hardware note

This workstation has no NVIDIA GPU (CPU-only torch). Smoke tests run here; real sweeps go on the RTX 6000 Pro box. A full seven-dataset pass is ~135k documents plus ~43k queries — minutes for a 335M encoder on GPU, well under an hour for an 8B one.