NVAR: A Natural Variant Dataset for Rethinking How We Evaluate Functional Effects

This repository contains the code for the NVAR benchmark, which evaluates protein language models and language models on natural variant effect prediction.

Dataset

The dataset is publicly available on HuggingFace: sujigwak/NVAR

All scripts automatically download nvar.jsonl from HuggingFace if not found locally.

---

Directory Structure

_NVAR/
├── data/                          # Auto-downloaded dataset
│   └── nvar.jsonl
├── output/
│   ├── llm/                       # LLM inference results
│   └── mllm/                      # Multimodal LLM inference results
├── cache/
│   ├── pdb_files/                 # ESMFold PDB cache
│   └── offload/                   # Model offload cache
├── crawl/
│   └── variants.py                # UniProt variant crawler
├── model_generation/
│   ├── llm/
│   │   ├── qwen3.py               # Qwen3 inference (w/ function)
│   │   └── qwen3_abl.py           # Qwen3 inference (w/o function)
│   └── mllm/
│       ├── esm2text.py            # ESM2Text inference
│       ├── prot2text.py           # Prot2Text inference
│       └── prot2textv2.py         # Prot2Text-V2 inference
└── Prot2Text/                     # Customized Prot2Text repo

---

Installation

pip install -r requirements.txt

Requirements

torch
transformers>=4.43.0
accelerate>=0.30.0
pandas
numpy
matplotlib
scikit-learn
tqdm
huggingface_hub
rouge-score
nltk
requests
pyarrow

---

Usage

1. LLM (Qwen3)

# With reference function
python model_generation/llm/qwen3.py

# Without reference function (ablation)
python model_generation/llm/qwen3_abl.py

2. ESM2Text

python model_generation/mllm/esm2text.py

# Options
python model_generation/mllm/esm2text.py \
    --input_file data/nvar.jsonl \
    --output_dir output/mllm/esm2text_answer.jsonl \
    --device mps   # cuda / cpu / mps

3. Prot2Text

⚠️ Requires model weights. Download from HuggingFace or specify local path.

python model_generation/mllm/prot2text.py \
    --model_path habdine/Prot2Text-Large-v1-1 \
    --sequence_index 1 \
    --device cuda

4. Prot2Text-V2

⚠️ Requires CUDA GPU. Not supported on Mac.

python model_generation/mllm/prot2textv2.py

5. UniProt Variant Crawler

# accessions.txt: one UniProt accession ID per line
python crawl/variants.py \
    --accessions accessions.txt \
    --shard-id 0 \
    --num-shards 4 \
    --write-empty

# Parallel execution
for i in 0 1 2 3; do
    python crawl/variants.py \
        --accessions accessions.txt \
        --shard-id $i \
        --num-shards 4 &
done
wait

---

Notes

• data/, output/, cache/ are auto-created at runtime.
• All output paths are relative to _NVAR/ root.
• Prot2Text/ is a customized version of the original Prot2Text repository.