GFMBench-API

GFMBench-API is an extensible benchmarking suite for assessing genomic foundation models (GFMs) across a diverse set of downstream tasks, including classification, variant effect prediction, and zero-shot evaluation.

This repository accompanies the paper “GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models” (bioRxiv, 2026). If you use GFMBench-API in your research, please cite the paper:

@article{larey2026gfmbench,
  title   = {GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models},
  author  = {Larey, Ariel and Dahan, Elay and Bleiweiss, Amit and Kellerman, Raizy and Leib, Guy and Nayshool, Omri and Ofer, Dan and Zinger, Tal and Dominissini, Dan and Rechavi, Gideon and Bussola, Nicole and Lee, Simon and O'Connell, Shane and Hoang, Dung and Wirth, Marissa and Charney, Alexander W. and Shavit, Yoli and Daniel, Nati},
  journal = {bioRxiv},
  year    = {2026},
  doi     = {10.64898/2026.02.19.706811}
}

Quick Start

Installation

GFMBench-API separates package dependencies from model dependencies. Model runtimes (Evo2, Nucleotide Transformer, DNABERT2, etc.) are maintained by their own projects and are not bundled into the core package.

1. Create a virtual environment (choose one option):

   Option A: Using pip (venv)

   python -m venv gfmbench_env
   source gfmbench_env/bin/activate  # On Windows: gfmbench_env\Scripts\activate
   

   Option B: Using conda

   conda create -n gfmbench_env python=3.11
   conda activate gfmbench_env
   

2. Install dependencies for your model first, following that model's own environment setup. Examples:

   Model	Dependency source
   Evo2	evo2 pyproject.toml
   Nucleotide Transformer (NTv3)	nucleotide-transformer setup.py
   DNABERT2	DNABERT_2 requirements.txt

3. Install GFMBench-API core dependencies on top of your model environment:

   pip install -r basic_requirements.txt
   

   basic_requirements.txt contains only what the gfmbench_api package needs (tasks, metrics, data I/O, etc.) — no model-specific libraries.

   Note: If your model’s dependencies conflict with basic_requirements.txt (or with other models), you can still keep the benchmark stack in one environment and run model inference in a separate Python environment and process. See usage_examples/sanity_models/isolated_mock_model.py and the Isolated model environments section below.

---

What’s Included?

GFMBench-API provides:

• A core API package for unified GFM evaluation (gfmbench_api/).
• A suite of standard benchmark tasks covering supervised and zero-shot scenarios.
• Consistent interfaces for models and tasks, enabling both out-of-the-box use and customized evaluation pipelines.
• Example scripts and templates (usage_examples/) to get started quickly or for rapid prototyping.

---

Repository Organization

gfmbench_api_rep/
├── gfmbench_api/              # Main API package
│   ├── benchmark_report/      # CSV report utilities
│   ├── metrics/               # Built-in metrics (AUROC, AUPRC, etc.)
│   ├── tasks/                 # Task definitions
│   │   ├── base/              # Base task/model classes
│   │   └── concrete/          # 20+ ready-to-use tasks
│   └── utils/                 # Misc utilities (data I/O, download helpers, inference cache)
├── usage_examples/            # Getting started scripts and toy models
│   ├── run_benchmark.py
│   ├── trainers/
│   └── sanity_models/
├── logs/                      # Logs (autocreated)
└── basic_requirements.txt     # GFMBench-API core only (model-agnostic)

---

Supported Benchmarks & Tasks

GFMBench-API supports evaluation on 22 unique tasks, grouped as:

Supervised Classification & Variant Prediction

Zero-Shot Variant Effect Prediction

Several zero-shot variant effect prediction tasks repeat the same reference sequence across variants. For tasks where that pattern is common, reference sequences are cached during evaluation to avoid redundant forward passes and improve efficiency. Caching is applied only where it offers a meaningful memory and latency tradeoff. To disable caching, set "cache_size": 0 in task_config. To cap RAM usage, set "cache_size" to a positive value in GB.

---

How Model & Task Interfaces Work

Task API

All task classes expose a consistent interface (see gfmbench_api/tasks/base/):

• get_task_name()
• get_task_attributes() — metadata (e.g. number of labels, dataset splits)
• get_finetune_dataset()
• eval_test_set(model)
• eval_validation_set(model)
• eval_cross_validation_fold(model, train_indices)

Model Integration

Simply implement the methods below in your model class; inheritance is not required. The API uses duck typing and will only call methods needed by the specific tasks you run.

Classification probabilities use shape [batch, classes] for single-label tasks, [batch, labels] for binary multi-label tasks, and [batch, labels, classes] for multi-class multi-label tasks.

• Any not-implemented methods can simply return None and metrics depending on them will be skipped.
• See gfmbench_api/tasks/base/base_gfm_model.py for detailed docstrings.
• Step-by-step tutorial (mock model → NTv3 8M, BEND + GUE): usage_examples/add_new_model_tutorial.ipynb.

Isolated model environments

Tasks, metrics, and data I/O stay in the host env (basic_requirements.txt). If your model’s packages conflict with that stack, run inference in a separate Python env and process: a thin host adapter implements the infer_* methods above and forwards each call to a worker subprocess.

See usage_examples/sanity_models/isolated_mock_model.py for a complete example (mock outputs via the worker-only package randomgen). The module docstring covers setup, what to change (DEFAULT_ISOLATED_PYTHON / worker_python), and how to run:

python usage_examples/run_benchmark.py \
  --model IsolatedMock \
  --linear_prob \
  --sanity_check_mode \
  --root_data_dir_path /path/to/data \
  --csv_path /tmp/isolated_mock.csv

The first inference call prints a host→worker environment switch so you can confirm isolation.

Note: For full fine-tuning, training must run in the same environment and process as the model (weights and autograd cannot cross the host↔worker boundary). Linear probing can keep the backbone isolated and train only the host-side projection layer.

The same adapter pattern works for compiled runtimes (e.g. ONNX Runtime, TensorRT): implement the infer_* methods, run the session/engine inside the adapter, and return NumPy arrays. GFMBench-API does not require the backbone to be a live PyTorch module. Full fine-tuning of a compiled graph is usually out of band; linear probing on frozen compiled embeddings still works.

---

Example: Running Benchmarks

The usage_examples/run_benchmark.py script

This reference script demonstrates a standard workflow:

• Loading a model (built-in or your own)
• Running a configurable set of tasks
• Supervised fine-tuning or zero-shot evaluation
• Auto-generating benchmark CSVs

Common CLI Arguments

Key Variables to Adjust in run_benchmark.py

• Root data directory
  ⚠️ Update around line 164:

  root_data_dir_path = "/path/to/your/data/"
  

• Sanity check mode (100-sample subsets)

  sanity_check_mode = True  # Set to False for full eval
  

• Task configuration
  Edit parameters like max_sequence_length, batch_size, or cache_size.
• Training parameters
  Change num_epochs, learning_rate, etc. in the training_params dict.
• Reproducibility: Using num_workers > 0 may cause non-deterministic results on several tasks; for full reproducibility in usage_examples/run_benchmark.py, keep the default num_workers = 0.
• Task list
  Edit/add the tasks in the tasks = [...] list.
• Model registry
  Add your models to the MODEL_REGISTRY dict.

Example usage

Default (no checkpoint):

python usage_examples/run_benchmark.py \
    --model DNABERT2 \
    --report_algo_name dna_bert2_baseline \
    --csv_path results/baseline_results.csv

With a custom model checkpoint:

python usage_examples/run_benchmark.py \
    --model DNABERT2 \
    --checkpoint_path /path/to/model.pt \
    --report_algo_name my_custom_model \
    --csv_path results/my_results.csv \
    --epochs 5

Linear probe (freeze backbone, train only projection):

python usage_examples/run_benchmark.py \
    --model DNABERT2 \
    --linear_prob \
    --report_algo_name linear_probe \
    --csv_path results/linear_probe_results.csv

Disable inference cache:

python usage_examples/run_benchmark.py \
    --model DNABERT2 \
    --cache_size 0 \
    --report_algo_name no_cache \
    --csv_path results/no_cache_results.csv

Cap inference cache RAM (200 GB):

python usage_examples/run_benchmark.py \
    --model DNABERT2 \
    --cache_size 200 \
    --report_algo_name capped_cache \
    --csv_path results/capped_cache_results.csv

Inference caching

Selected zero-shot variant effect prediction tasks cache repeated reference sequences during evaluation (see above). Set "cache_size": 0 in task_config to turn caching off, or a positive GB value to cap cache RAM.

On top of that, usage_examples/run_benchmark.py uses the same caching utility (gfmbench_api/utils/caching_utils.py) in two other places:

• Linear probing (--linear_prob): caches frozen backbone forwards when only the projection layer is trained.
• Supervised variant effect prediction: caches reference sequences during evaluation.

When omitted, --cache_size is unlimited (a warning is logged at startup). When a cap is reached, existing entries are still served but new entries are not stored until the cache is cleared. Pass --cache_size 0 to disable all caching.

---

Data Preparation

Root Data Directory

All task data is stored under a single root directory (customize the path!):

root_data_dir_path = "/path/to/your/data/"

Task Data Structure

Each task creates a subdirectory, e.g.:

<root_data_dir_path>/
├── gue_promoter_all/
│   ├── train.csv
│   ├── dev.csv
│   └── test.csv
├── clinvar_vepeval/
│   └── ...
└── ...

• Task subdirectory names correspond to class names.
• If files are missing, they will be automatically downloaded on first use.

Dataset Format

Sequence Symbol Conventions

GFMBench-API uses the following standard symbols for DNA sequences in datasets:

Notes:

• All sequences are case-insensitive (automatically converted to uppercase)
• The P padding character should be added when handling boundary cases
• Unknown nucleotides (N) may appear in reference genome data or when sequence information is incomplete
• When creating custom tasks, ensure your sequences use only these accepted symbols

---

Benchmark Output

Results are auto-saved in a single CSV with this schema:

• Each (task, metric) is a row.
• Each model name (as passed to --report_algo_name) is a column.
• If a result is missing, its cell is NO_RESULTS.
• Incremental saving ensures safe restarts/interruption recovery.

Logs are written to logs/ for all major steps (run progress, model errors, fine-tune loss curves, auto-downloading messages, etc).

---

Customizing

Example: Minimal Custom Pipeline

For a tutorial walkthrough (mock model → NTv3 8M on BEND and GUE), see usage_examples/add_new_model_tutorial.ipynb.

from gfmbench_api.tasks.concrete.gue_promoter_all_task import GuePromoterAllTask
from gfmbench_api.benchmark_report import BenchmarkReport

task = GuePromoterAllTask(
    root_data_dir_path="/your/data/path",
    task_config={"max_sequence_length": 2500, "batch_size": 16},
)

my_model = MyCustomModel(device="cuda")  # Implements the required inference methods

# Optionally, supervised fine-tuning:
if task.get_task_attributes()["has_finetuning_data"]:
    dataset = task.get_finetune_dataset()
    # ... Your custom training code ...
    # After fine-tuning, wrap model with projection layer for classification (my_model = wrapped_model)

# Evaluate
my_model.eval()
results = task.eval_test_set(my_model)
print(results)

# Save results
report = BenchmarkReport(csv_path="results.csv")
report.add_scores("gue_promoter_all", "my_model", results)
report.save_csv()

---

Adding Tasks

To add a new concrete task, inherit from the appropriate base class based on your task type and implement the required methods.

Note: If your task has unique functionality that doesn't fit the standard task types, you can inherit directly from BaseGFMTask and implement all required methods from scratch.

Task Type Hierarchy

Choose the appropriate base class based on your task:

• Single-sequence classification: Inherit from BaseGFMSupervisedSingleSeqTask

• Variant-effect classification: Inherit from BaseGFMSupervisedVariantEffectTask

• Classification mode is derived from _get_num_labels(): one label is single-label classification; multiple labels are independent classification targets.

• Supervised Regression: Inherit from BaseGFMSupervisedRegressionTask

  • Implement is_spatial_binned() to return True for binned outputs or False for sequence-level outputs

• Zero-Shot SNV Variant Effect: Inherit from BaseGFMZeroShotSNVTask

  • For zero-shot evaluation of single-nucleotide variants (SNVs) with equal-length reference and variant sequences

• Zero-Shot General Indel: Inherit from BaseGFMZeroShotGeneralIndelTask

  • For zero-shot evaluation of insertions/deletions with variable-length sequences

Required Methods to Implement

All concrete tasks must implement:

1. get_task_name() -> str: Return the task name (must match the data directory name)
2. _get_default_max_seq_len() -> int: Return the default maximum sequence length
3. _create_datasets() or _create_test_dataset(): Create and return datasets
   • Supervised tasks: Return Tuple[Optional[Dataset], Optional[Dataset], Dataset] (train, validation, test)
   • Zero-shot tasks: Return Dataset (test only)
   • Important: When creating datasets, you must account for:
     • self.max_sequence_length: Truncate sequences if they exceed this length (use truncate_sequence_from_ends() from gfmbench_api.utils.preprocutils)
     • self.max_num_samples: Limit the number of samples per split if specified (for sanity testing with smaller datasets)
4. get_conditional_input_meta_data_frame() -> Optional[pd.DataFrame]: Return metadata DataFrame if task uses conditional inputs, otherwise return None

Additional methods for supervised classification tasks:

5. _get_num_labels() -> int: Return the number of predicted labels
6. _get_num_classes() -> int: Return the number of classes per label

Additional methods for supervised regression tasks:

5. _get_num_labels() -> int: Return the number of continuous labels per sequence or bin
6. is_spatial_binned() -> bool: Return whether outputs include a spatial bin dimension

Additional methods for zero-shot tasks:

5. use_reference_cache() -> bool: Return True if repeated reference sequences make eval-time caching worthwhile; otherwise False.

Example: Supervised Single-Sequence Task

from gfmbench_api.tasks.base import (
    BaseGFMSupervisedSingleSeqTask,
)
import os
import pandas as pd
import torch
import numpy as np

class MyCustomTask(BaseGFMSupervisedSingleSeqTask):
    def get_task_name(self) -> str:
        return "my_custom_task"
    
    def _get_default_max_seq_len(self) -> int:
        return 512
    
    def _get_num_classes(self) -> int:
        return 2  # Binary classification

    def _get_num_labels(self) -> int:
        return 1
    
    def _create_datasets(self):
        data_dir = os.path.join(self.root_data_dir_path, self.get_task_name())
        train_df = pd.read_csv(os.path.join(data_dir, "train.csv"))
        val_df = pd.read_csv(os.path.join(data_dir, "dev.csv")) if os.path.exists(os.path.join(data_dir, "dev.csv")) else None
        test_df = pd.read_csv(os.path.join(data_dir, "test.csv"))
        
        # Data process: Account for self.max_num_samples and self.max_sequence_length
        train_dataset = [(seq, label, np.array([])) for seq, label in zip(train_df['sequence'], train_df['label'])]
        validation_dataset = [(seq, label, np.array([])) for seq, label in zip(val_df['sequence'], val_df['label'])] if val_df is not None else None
        test_dataset = [(seq, label, np.array([])) for seq, label in zip(test_df['sequence'], test_df['label'])]
        
        return train_dataset, validation_dataset, test_dataset
    
    def get_conditional_input_meta_data_frame(self):
        return None

Example: Zero-Shot SNV Task

from gfmbench_api.tasks.base.base_gfm_zeroshot_snv_task import BaseGFMZeroShotSNVTask
import os
import pandas as pd
import numpy as np

class MyZeroShotTask(BaseGFMZeroShotSNVTask):
    def get_task_name(self) -> str:
        return "my_zero_shot_task"
    
    def _get_default_max_seq_len(self) -> int:
        return 1024
    
    def _create_test_dataset(self):
        data_dir = os.path.join(self.root_data_dir_path, self.get_task_name())
        test_df = pd.read_csv(os.path.join(data_dir, "test.csv"))
        
        # Data process: Account for self.max_num_samples and self.max_sequence_length
        test_dataset = [
            (var_seq, ref_seq, label, np.array([]))
            for var_seq, ref_seq, label in zip(test_df['variant_sequence'], test_df['reference_sequence'], test_df['label'])
        ]
        
        return test_dataset
    
    def _get_variant_position_in_sequence(self) -> int:
        return self.max_sequence_length // 2
    
    def get_conditional_input_meta_data_frame(self):
        return None

    def use_reference_cache(self) -> bool:
        return True  # Set False if references are mostly unique per variant

Contributing

Contributions to GFMBench-API are welcome. Please open a pull request with your suggested changes.

NOTICE

This project will download and install additional third-party open source software projects, including datasets licensed with non-commercial terms. Review the license terms of these open source projects before use.

Third-party components and attributions: see THIRD_PARTY_NOTICES.md (Python dependencies, Hugging Face assets, reference genomes, and other data URLs). Example: LicenseRef-UCSC-Genome-Browser — https://genome.ucsc.edu/license/

License

NVIDIA-authored code in this repository is licensed under the Apache License, Version 2.0. The full text is in LICENSE.

Each contributed .py file includes NVIDIA SPDX-FileCopyrightText, SPDX-License-Identifier: Apache-2.0, and the short Apache-2.0 notice (through “limitations under the License”), then third-party URL notices scoped to that file (or the line stating that the module does not embed third-party data download URLs). Python package attributions remain in THIRD_PARTY_NOTICES.md. The full Apache-2.0 text is in LICENSE.

Citation

@article{larey2026gfmbench, title={GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models}, author={Ariel Larey, Elay Dahan, Amit Bleiweiss, Raizy Kellerman, Guy Leib, Omri Nayshool, Dan Ofer, Tal Zinger, Dan Dominissini, Gideon Rechavi, Nicole Bussola, Simon Lee, Shane O’Connell, Dung Hoang, Marissa Wirth, Alexander W. Charney, Yoli Shavit, Nati Daniel}, journal={bioRxiv}, pages={2026--02}, year={2026}, publisher={Cold Spring Harbor Laboratory} }