NucEL

Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable Representations

Ke Ding, Brian Parker, Jiayu Wen. Australian National University · Preprint, August 2025.

NucEL is the first ELECTRA-style genomic foundation model. A small generator proposes single-nucleotide substitutions to a masked DNA sequence, and a larger ModernBERT discriminator is trained to detect which positions were replaced — giving dense supervision across all tokens instead of the ~15 % covered by masked language modeling. At 93 M parameters, NucEL outperforms MLM-based models of similar size and matches or exceeds models up to 25× larger (e.g. NT-Multi-2.5B) on GUE, GB, and NT.

---

Highlights

• ELECTRA RTD for genomics. Dense supervision across every base; ~6.7× more signal per forward pass than 15 %-masking MLM.
• Single-nucleotide tokenization. Vocab of 27 tokens (4 nt + 7 special + 16 reserved). No k-mer collapse, no BPE drift; preserves base-level detail for tasks like motif discovery.
• ModernBERT backbone. Hybrid local (128 bp window) and global (every 3 layers) attention, FlashAttention-2, RoPE, GeGLU.
• Compute-efficient. Pre-trained on the human genome only with 8× A100 GPUs (50 epochs, global batch 192).
• State-of-the-art on standard benchmarks despite using ~25× fewer parameters than the strongest baselines (see Results).

Model card

Results

GUE (Table 1, MCC; F1 for CVC). Bold = best; underlined = second-best.

Genomic Benchmarks (Table 2, accuracy; mean ± std over 5 seeds)

NT benchmark (Table 3, 18 tasks, MCC; mean over 10 seeds)

NucEL's average 0.664 matches NT2-Multi-500M (0.660 in the paper's Table 3 / 0.661 NT-Multi-2.5B) while using 27× fewer parameters. See docs/figure2_tradeoff.png for the full efficiency-performance tradeoff.

---

Installation

git clone https://github.com/FreakingPotato/NucEL.git
cd NucEL

# Editable install (recommended)
pip install -e .

# Or install the listed pinned dependencies
pip install -r requirements.txt

Tested with Python 3.10, PyTorch 2.1+. FlashAttention-2 is optional but strongly recommended on A100 / H100.

About the transformers version

NucEL's tokenizer is a custom k-mer class registered via auto_map. To let AutoTokenizer.from_pretrained honor this auto_map, install transformers>=4.48,<5.4:

pip install "transformers>=4.48,<5.4" torch

Starting with transformers==5.4.0 (released 2026-03-27), modernbert-based checkpoints were added to a MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS blocklist that disables auto_map for tokenizer auto-discovery. NucEL is collateral damage of that change. The model side (AutoModel.from_pretrained) is unaffected, but the tokenizer must be loaded explicitly — see Quickstart on transformers 5.4+.

Quickstart

import torch
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("FreakingPotato/NucEL", trust_remote_code=True)
model     = AutoModel.from_pretrained("FreakingPotato/NucEL").eval()

inputs = tokenizer("ATCGATCGATGCATGCATGC", return_tensors="pt")
with torch.no_grad():
    h = model(**inputs).last_hidden_state          # (1, L, 512)
cls = h[:, 0, :]                                   # (1, 512)

Runnable: examples/quickstart.py.

Quickstart on transformers 5.4+

If you cannot pin transformers<5.4, bypass AutoTokenizer and import NucEL_Tokenizer directly. Once you have nucel installed via pip install -e ., that's a one-liner:

import torch
from transformers import AutoModel
from nucel.tokenizer import NucEL_Tokenizer

tokenizer = NucEL_Tokenizer.from_pretrained("FreakingPotato/NucEL")
model     = AutoModel.from_pretrained("FreakingPotato/NucEL").eval()

inputs = tokenizer("ATCGATCGATGCATGCATGC", return_tensors="pt")
with torch.no_grad():
    h = model(**inputs).last_hidden_state

Without installing this repo, you can also fetch tokenizer.py from the Hub at runtime:

import sys
from huggingface_hub import snapshot_download
local = snapshot_download("FreakingPotato/NucEL",
                          allow_patterns=["tokenizer*.py", "tokenizer_config.json",
                                          "vocab.json", "special_tokens_map.json"])
sys.path.insert(0, local)
from tokenizer import NucEL_Tokenizer
tokenizer = NucEL_Tokenizer.from_pretrained(local)

Reproducing pre-training

scripts/pretrain.py has the paper-exact defaults baked in. To reproduce the released checkpoint on 8× A100:

deepspeed scripts/pretrain.py \
    --batch_size 24 \                   # 24 * 8 GPUs = 192 global
    --epochs 50 \
    --learning_rate 1e-4 \
    --dataset_script_path FreakingPotato/nt2_homo_sapiens_genome_1024

All other architecture and training hyperparameters default to the values reported in Sections 3.2 and 3.4 of the paper.

Reproducing GUE results

# All 12 human-subset tasks, 3 seeds, default to FreakingPotato/NucEL
bash scripts/run_gue.sh

# A single task
python scripts/finetune_gue.py --task prom_300_all --random_seed 42

# List all tasks
python scripts/finetune_gue.py --list_tasks

Per-seed CSVs land in benchmark_GUE_results/FreakingPotato_NucEL/<task>/... and an aggregate summary_seed*_k*.csv at the root.

Embedding visualization

Reproduces Figure 3 / Table 4 (gene-biotype embeddings) and the GUE-style t-SNEs:

# Biotype embeddings (Fig 3 / Table 4)
python scripts/embedding_visualization.py \
    --mode biotype \
    --target_model NucEL \
    --pooling cls \
    --input_fasta path/to/ensembl_biotypes.fasta \
    --output_dir embeddings/

# GUE regulatory / TF marker embeddings (Section 4.1)
python scripts/embedding_visualization.py \
    --mode gue \
    --target_model NucEL \
    --pooling mean \
    --input_fasta path/to/regulatory_elements_3000.fasta

The same script supports --target_model DNABERT2 | NT2_100m | HyenaDNA for head-to-head comparisons against the baselines reported in the paper.

Repository layout

NucEL/
├── nucel/                   # importable package
│   ├── tokenizer.py        # NucEL_Tokenizer (k-mer, k = 1)
│   ├── modeling.py         # ElectraForPreTraining + ModernBert discriminator
│   └── data.py             # GenomeDataset + ElectraDataCollator
├── scripts/
│   ├── pretrain.py         # ELECTRA pre-training (paper-exact defaults)
│   ├── finetune_gue.py     # GUE benchmark fine-tuning
│   ├── run_gue.sh          # 3-seed driver script
│   └── embedding_visualization.py  # Section 4.1 / 4.6
├── examples/
│   └── quickstart.py
├── hf_patch/               # AutoTokenizer fix for Hugging Face Hub
└── docs/
    ├── architecture.png    # Figure 1
    └── figure2_tradeoff.png  # Figure 2 (tokenization + efficiency)

Citation

@article{ding2025nucel,
  title   = {NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training
             for Efficient and Interpretable Representations},
  author  = {Ding, Ke and Parker, Brian and Wen, Jiayu},
  journal = {bioRxiv},
  year    = {2025},
  doi     = {10.1101/2025.08.17.670700},
  url     = {https://www.biorxiv.org/content/10.1101/2025.08.17.670700v1}
}

License

Apache-2.0. See LICENSE.