ProtGPT3 — Code Release

This repository contains the code accompanying the NeurIPS paper submission called "ProtGPT3: an Open-source family of Promptable and Aligned Protein Language Models". It covers the full pipeline from pretraining the two types of protein language models (single-sequence and MSA-based), through alignment and conditional generation, to inference-time compute strategies.

---

Repository Structure

single_sequence_pretraining

Code for pretraining our single-sequence protein language models from scratch. This includes data processing, model architecture, and training loops for the base single-sequence models that underpin the downstream alignment and conditional generation experiments.

msa_model_pretraining

Code for pretraining our multiple sequence alignment (MSA)-based protein language models from scratch. Contains the MSA-specific architecture and training infrastructure used to produce the base MSA models evaluated throughout the paper.

dpo_alignment

Code for aligning pretrained single-sequence models using Direct Preference Optimization (DPO). This pipeline addresses a bias in the base models toward generating low-complexity sequences, using preference data to steer generation toward higher-complexity, more biologically realistic outputs.

sft_vs_prompting

Code for the comparison between supervised fine-tuning (SFT) of single-sequence models and few-shot prompting of MSA models for conditional protein generation. Experiments here evaluate how well each approach can target specific protein families without full retraining.

msa_test_time_compute

Code for our inference-time compute experiments on MSA models. This implements a homolog-based Feynman–Kac inference procedure that uses sequential Monte Carlo-style weighting to steer protein sequence generation toward desired targets at test time, without any additional training.

Ethics

Modeling the distribution of natural protein sequences and predicting sequence fitness are capabilities with significant potential for computational protein design. Should the methods introduced here — or extensions of them — see widespread adoption, it is important that the research community and practitioners carefully consider the intended applications of any designed proteins and their broader societal implications. As with all generative technologies in the life sciences, there is a responsibility to ensure that these tools are applied in ways that are safe, beneficial, and free from misuse. We encourage those building on this work to implement appropriate oversight mechanisms at every stage of a project — from initial conception through experimental validation and eventual deployment — in order to minimize unintended consequences and promote responsible use.