SAPER-CLIP Benchmarks

Shared benchmarking toolkit for SAPER, ProteinCLIP, SAPER-CLIP, and related protein-function-description methods. Provides canonical metric implementations, a GPT-4o LLM-as-Judge harness, and inference wrappers for the baselines reported in our ICML 2026 workshop submission (under review).

What's in this repo vs what's not

In this repo:

• Metric implementations (BLEU, Meta-BLEU, METEOR, ROUGE, Entity-BLEU, Entity-P/R/F1) used in the SAPER-CLIP paper, with parameters documented in METRICS_ALIGNMENT.md.
• GPT-4o LLM-as-Judge scorer and per-model / per-match-type aggregation.
• Model wrappers (Qwen-3, Llama-3.1, Prot2Text-V2, RAPM, MMseqs2, Esm2Text) for running baseline inference on the four SAPER OOD datasets.
• Cached predictions + metrics for those baselines under results/.
• Design and audit docs under docs/.

Not in this repo:

• SAPER-CLIP adapter weights (released separately with the paper).
• Experimental outputs from our SAPER / Phase A / SAPER-CLIP runs (live in an internal location referenced from HANDOFF_NOTES.md).
• Raw SAPER datasets — use the upstream SAPER release, point SAPER_DATA_ROOT at your local copy.
• Model checkpoints (Llama, Qwen, Prot2Text-V2, ESM2) — fetch from HuggingFace.

How collaborators should use it

Two typical flows:

(1) Score an existing predictions file against Entity-BLEU / Entity-F1:

import os, sys
sys.path.insert(0, "/path/to/this/clone")
from metrics import entity

res = entity.compute_metrics(
    predictions_path="my_model_catalytic_activity_OOD_predictions.json",
    dataset_path=f"{os.environ['SAPER_DATA_ROOT']}/catalytic_activity_OOD.json",
)
print(res)   # entity_bleu2, entity_bleu4, entity_precision, entity_recall, entity_f1

(2) Run a baseline end-to-end:

export SAPER_DATA_ROOT=/path/to/saper/dataset
python run_qwen3_benchmark.py \
    --model_dir /path/to/Qwen3-8B \
    --results_dir ./results/Qwen3-8B-3shot \
    --datasets catalytic_activity_OOD

See HANDOFF_NOTES.md for the full onboarding walk-through.

Scope

This toolkit supports inference and/or evaluation for the following methods:

• SAPER (weighted FAISS + MMseqs2 similarity retrieval). The SAPER model itself is not in this directory — it lives at /data/rbg/users/linrui/SAPER/SAPER/. Run SAPER from that directory; pipe its predictions into this toolkit's evaluation for scoring.
• Phase A ProteinCLIP (sequence-only contrastive joint embedding). Experiments: /data/rbg/users/linrui/SAPER/proteinclip_experiments/phase_a_comparison/.
• SAPER-CLIP (ESM2 + ProstT5 structural-augmented joint embedding).
• Hybrid (RRF or C2 weighted fusion of SAPER + SAPER-CLIP). Experiments: /data/rbg/users/linrui/SAPER/proteinclip_experiments/saper_clip_hybrid/.
• Prot2Text-V2 (direct generation baseline).
• RAPM (retrieval-augmented protein model baseline).
• Qwen-3 8B / Llama-3.1 8B (3-shot LLM baselines).
• MMseqs2 (nearest-neighbor retrieval baseline).
• ESM2Text-Base (habdine/Esm2Text-Base, direct decoder baseline).

Directory layout

benchmarks/
├── README.md                 # this file
├── METRICS_ALIGNMENT.md      # canonical metric definitions + discrepancies
├── HANDOFF_NOTES.md          # collaborator onboarding
├── metrics/                  # thin re-export module: benchmarks.metrics.{entity,rapm,judge}
├── fix_entity_bleu.py        # canonical Entity-BLEU / Entity-P/R/F1
├── rapm_rescore.py           # RAPM-official Meta-BLEU / METEOR / ROUGE
├── llm_judge_score.py        # GPT-4o LLM-as-Judge (4-axis rubric)
├── aggregate_judge_scores.py # per-model / per-match-type aggregation
├── fix_llama_predictions.py  # post-hoc cleanup of Llama outputs
├── download_model.py         # HuggingFace snapshot helper
├── run_qwen3_benchmark.py    # Qwen-3 8B 3-shot wrapper
├── run_llama_benchmark.py    # Llama-3.1 8B 3-shot wrapper
├── run_prot2text_benchmark.py
├── run_esm2text_benchmark.py
├── run_mmseqs2_baseline.py
├── run_*_benchmark.sh        # launch scripts for the above
├── Prot2Text-V2/             # upstream clone — not tracked (see note below)
├── RAPM_official/            # upstream clone — not tracked (see note below)
├── results/                  # cached predictions + metrics per model
│   ├── Qwen3-8B-3shot/
│   ├── Llama-3.1-8B-3shot/
│   ├── Prot2Text-V2/
│   ├── MMseqs2-retrieval/
│   ├── BioReason-Pro-RL/     # reference baseline (wrapper removed)
│   └── BioT5plus/            # reference baseline (wrapper removed)
└── docs/                     # design docs + historical audits
    ├── METRICS_DOCUMENTATION.md
    ├── METRIC_AUDIT.md
    ├── METRIC_FIXES_BEFORE_AFTER.md
    ├── JUDGE_COMPARISON.md
    ├── JUDGE_PROMPT.md
    ├── ALL_BASELINE_METRICS.csv
    └── ALL_BASELINE_METRICS.md

model_cache/, venv/, venv2/, .hf_cache/, and .pip_cache/ are excluded from the repository (see .gitignore).

Metrics

All metrics are computed on the 0–100 scale. Canonical parameters are documented in METRICS_ALIGNMENT.md; this section is a quick summary.

• Corpus BLEU-2/4: n-gram overlap aggregated across all predictions with NLTK corpus_bleu and SmoothingFunction.method1. Matches the SAPER paper's Table 3 reporting convention.
• Per-sample BLEU-2/4: the same metric computed per prediction with sentence_bleu + method1 smoothing, then averaged. Suitable for paired t-tests between methods.
• Meta-BLEU-2/4: BLEU restricted to metadata tokens (entity names, enzyme classes, chemical reaction components). Follows SAPER 2024 convention.
• METEOR: NLTK meteor_score on tokenized predictions and references (meta-filtered tokens, matching SAPER).
• ROUGE-1/2/L: rouge_score.RougeScorer F1 values.
• Entity-BLEU-2/4, Entity-P/R/F1: biological-entity matching with an Aho-Corasick automaton built from the dataset's extracted entity vocabulary. See fix_entity_bleu.py for the matcher construction.
• Exact Match: full-text string equality (reported by the per-wrapper evaluate() helpers).
• LLM-as-Judge (GPT-4o): four discrete axes (recall, precision, specificity, plausibility) on a {0,3,5,7,10} scale. See docs/JUDGE_PROMPT.md for the full rubric.

Usage

Evaluation (lexical metrics) on an existing predictions file

import sys
sys.path.insert(0, "/path/to/benchmarks")
from metrics import entity

res = entity.compute_metrics(
    predictions_path="results/Qwen3-8B-3shot/catalytic_activity_OOD_predictions.json",
    dataset_path="/data/rbg/users/linrui/SAPER/dataset/catalytic_activity_OOD.json",
)

For the full SAPER-CLIP paper evaluation pipeline (BLEU + Meta-BLEU + METEOR + ROUGE + Entity-F1 + paired t-tests) see proteinclip_experiments/phase_a_comparison/rag_pilot/scripts/v2_phase4_eval.py.

LLM-as-Judge

export OPENAI_API_KEY=...
python llm_judge_score.py \
    --predictions_dir results/Qwen3-8B-3shot \
    --judge_model gpt-4o \
    --dataset_dir /data/rbg/users/linrui/SAPER/dataset

Running a baseline

Each run_*_benchmark.py script is self-contained; see its header docstring for CLI flags. All scripts consume the four OOD datasets at /data/rbg/users/linrui/SAPER/dataset/.

Predictions JSONL format

For interoperability with v2_phase4_eval.py and friends, predictions should be written as one JSON object per line with these fields:

• test_array_index (int) — index into the source dataset JSON
• parsed_description (str) — the model's predicted description
• error (str or null) — non-empty means the sample was skipped

Configuration

Paths default to the author's CSAIL workspace. Override via environment variables:

All argparse defaults can also be overridden at invocation time with the usual --flag value CLI syntax.

Dependencies

• Python ≥ 3.10
• transformers, torch
• nltk (with punkt and wordnet downloaded)
• rouge_score
• scipy, numpy
• pyahocorasick (for the entity matcher)
• openai (for the LLM-as-Judge)
• tqdm
• Method-specific: mmseqs2 binary on PATH for run_mmseqs2_baseline.py

See setup_env.sh for the reference environment bootstrap.

Citation

Citation placeholder — add the ICML 2026 workshop entry once accepted.

Upstream repositories (not tracked in this repo)

Prot2Text-V2/ and RAPM_official/ are local clones of their respective upstream repositories. They are gitignored here (each carries its own .git/ directory) — collaborators should clone them into this directory:

git clone https://github.com/pierre-pessi/prot2text-v2 Prot2Text-V2
git clone https://github.com/ZhuoluHou/RAPM RAPM_official   # adjust to actual URL

Acknowledgments

Builds on SAPER (Liu et al. 2024), ProteinCLIP, Prot2Text-V2, and the RAPM benchmark protocol.