PeptideTransformer

Model comparison for antimicrobial peptide (AMP) property prediction, built around one question: is prediction accuracy limited more by the model or by the data? Fine-tunes two protein language models, ProtBERT-BFD and ESM-2, and benchmarks both against classical ML baselines.

---

Overview

Codebase for a Master's thesis on data quality in antimicrobial peptide modeling. Two prediction tasks:

• Hemolysis classification - hemolytic vs non-hemolytic against human erythrocytes.
• MIC regression - minimum inhibitory concentration (log₁₀ µM), one dataset per organism.

Every result is read along three axes:

Headline finding: model architecture is not the main lever. A measurement-noise floor caps achievable performance (MIC replicates vary by about 0.4 log₁₀, and 30 to 36 % of repeatedly-measured hemolysis sequences carry contradictory labels). The similarity-aware cluster split drops mean MIC test R² from about 0.30 to 0.18 by removing near-duplicate peptides that the random split shared across train and test.

---

Repository structure

data/
  regression_data/      # MIC base data, one CSV per organism
  hemo_train/           # hemolysis base data (WhiteLab + Threshold labeling)
  gram/                 # Gram-staining data (deferred exploratory analysis)
  data_from_database/   # raw public-database sources for preprocessing
notebooks/              # data-quality analysis + shared nb_utils.py
src/
  cli.py                # subcommand dispatch (python -m src / peptide-transformer)
  modeling/             # transformer fine-tuning (ProtBERT + ESM-2)
    fine_tune.py        # fine_tune() pipeline entry
    fine_tune_utils.py  # data prep, tokenizer, backbone overrides, model factory
    config_templates.py # YAML scaffolding templates
    configs/            # one YAML per organism: random_split/ and cluster_split/
    PeptideBERTClasses/ # model heads (BERT + ESM) + Trainer + Dataset
  machine_learning/     # classical ML training + grid search
  data_preprocessing/   # split + dataset prep (random + MMseqs2 cluster)
  data_analysis/        # embedding clustering, data inspection
  evaluation/           # shared eval, metrics.json artifacts, paired test
tests/                  # pytest suite (hermetic, CPU) - see Tests below

Splits (_train/_val/_test.csv) are generated by data_init / cluster_init, not shipped. data_preprocessing/ needs internal raw data for the MIC set and is included for traceability, not full reproducibility.

---

Setup

Conda (recommended):

conda env create -f environment.yml
conda activate peptide-transformer

Adjust pytorch-cuda in environment.yml to your driver version, or drop it for CPU-only.

Or pip:

pip install torch==2.4.0 --extra-index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

Optionally pip install -e . to expose the peptide-transformer command instead of python -m src.

---

Usage

All commands run from the repo root as python -m src <subcommand> (or peptide-transformer <subcommand> after pip install -e .).

1. Create splits (run once; splits are not shipped). Splitting is at the sequence level, so no peptide leaks across train/val/test.

python -m src data_init    --task regression   # or: cls, gram
python -m src cluster_init --task regression    # similarity-aware split, writes to <dir>_cluster/

Splits land next to each base file. cluster_init needs the MMseqs2 binary on PATH (conda install -c bioconda mmseqs2); defaults --min_seq_id 0.5 --coverage 0.8 --kmer 5.

The gram task is a deferred exploratory analysis (Gram-positive vs negative), not part of the main results. It is just another dataset that runs through the same split and training commands.

2. Train classical ML baselines. Grid search over Dummy, Ridge/LogReg, ExtraTrees, XGBoost and SVR/SVC on k-mer TF-IDF features.

python -m src ml_regress  --data_dir data/regression_data/ --output_dir ml_plots/
python -m src ml_classify --data_dir data/hemo_train/      --output_dir ml_plots/

--features adds biochemical descriptors, --seed sets the CV seed, --organism <substring> restricts the run. Point --data_dir at a *_cluster/ folder to train on the cluster split.

3. Fine-tune a transformer. Configs live in src/modeling/configs/{random_split,cluster_split}/, one per organism; a bare --config_path name is resolved across those folders. The bert_model subcommand fine-tunes either backbone (the command name is kept for interface stability).

python -m src bert_model --config_path <name>.yaml   # ProtBERT (default)
python -m src bert_model --pipe_configs default       # every config in a directory

ESM-2 reuses the same ProtBERT config and only swaps the encoder at launch. Output paths are suffixed with --model_name, so ESM runs never overwrite the ProtBERT outputs:

python -m src bert_model --config_path <name>.yaml \
    --backbone esm --model_path facebook/esm2_t33_650M_UR50D --model_name esm650m

Scaffold a fresh config with generate_bert_model_config:

python -m src generate_bert_model_config --config_name x.yaml --file_path configs/ --model_class regression

---

Tests

A hermetic pytest suite (CPU, no pretrained weights downloaded) covers the transformer heads for both backbones, tokenization and the leakage guard, the CLI, the cluster-split integrity invariant, the evaluation metrics, and the classical-ML pipeline end to end. It runs in CI on every push.

pip install -r requirements-dev.txt
python -m pytest

One opt-in integration test fine-tunes the small esm2_t6_8M end to end (downloads about 30 MB):

RUN_INTEGRATION=1 python -m pytest tests/test_esm_integration.py

---

Config reference

Essential fields in a training YAML:

Configs default to model_path: Rostlab/prot_bert_bfd; ESM is selected at launch via the CLI overrides above, not by editing the YAML. Training-control fields (epochs, early stopping, use_cpu, fast_debug_mode) are documented in a generated template and in the existing configs under src/modeling/configs/.

---

Reproducibility

Splitting, ML baselines and clustering default to seed 42. Transformers use the HuggingFace TrainingArguments default seed 42. Regenerate the exact splits with data_init / cluster_init, then run the training commands above on the matching CSVs.

For variance estimates, run_variance_sweep.py retrains each baseline across five seeds (42, 1, 2, 3, 4) on freshly regenerated random and cluster splits, and src/evaluation/paired_test.py runs a paired Wilcoxon test across organisms. Both consume the per-run metrics.json / predictions.csv written into each output dir.

---

Results

Model results (classical ML and BERT) are in RESULTS.md. The measurement and label noise that limits achievable performance is analyzed in notebooks/data_quality_mic.ipynb and notebooks/data_quality_hemo.ipynb.

---

Data sources & acknowledgements

• WhiteLab hemolysis data - the "WhiteLab" hemolysis dataset originates from the peptide-dashboard repository (UR WhiteLab). See that repository for licensing terms.

  Ansari, M. & White, A. D. (2023). Serverless Prediction of Peptide Properties with Recurrent Neural Networks. Journal of Chemical Information and Modeling, 63(8), 2546-2553. https://doi.org/10.1021/acs.jcim.2c01317

• "Threshold" hemolysis labeling - the activity-based labeling scheme used for the "Threshold" dataset (see label_by_threshold() in src/data_preprocessing/binary/happenn_preprocess.py) follows the scheme introduced by:

  Timmons, P. B. & Hewage, C. M. (2020). HAPPENN is a novel tool for hemolytic activity prediction for therapeutic peptides which employs neural networks. Scientific Reports, 10, 10869. https://doi.org/10.1038/s41598-020-67701-3

• MIC dataset - the underlying peptide/MIC data was provided by the IBVA working group at Technische Hochschule Mittelhessen (Gießen) as an aggregation of several public AMP databases, including DBAASP, APD3, CAMP and others. See the thesis for the full list of source databases.

  Pirtskhalava, M. et al. (2021). DBAASP v3: database of antimicrobial/cytotoxic activity and structure of peptides as a resource for development of new therapeutics. Nucleic Acids Research, 49(D1), D288-D297. https://doi.org/10.1093/nar/gkaa991

  Wang, G., Li, X. & Wang, Z. (2016). APD3: the antimicrobial peptide database as a tool for research and education. Nucleic Acids Research, 44(D1), D1087-D1093. https://doi.org/10.1093/nar/gkv1278

  Thomas, S., Karnik, S., Barai, R. S. et al. (2010). CAMP: a useful resource for research on antimicrobial peptides. Nucleic Acids Research, 38(suppl_1), D774-D780. https://doi.org/10.1093/nar/gkp1021

• ProtBERT-BFD - fine-tuning is based on the pretrained Rostlab/prot_bert_bfd model.

  Elnaggar, A. et al. (2020). ProtTrans: towards cracking the language of life's code through self-supervised deep learning and high performance computing. https://doi.org/10.1101/2020.07.12.199554

• ESM-2 - the second transformer backbone uses Meta AI's pretrained ESM-2 checkpoints.

  Lin, Z. et al. (2023). Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637), 1123-1130. https://doi.org/10.1126/science.ade2574

• Data download/decoding approach for the WhiteLab .npz files was adapted from ChakradharG/PeptideBERT.

---

License

This project is licensed under the PolyForm Noncommercial License 1.0.0 - free to use, modify, and share for any noncommercial purpose.