GLProtein: Global-and-Local Structure Aware Protein Representation Learning

This is the implement of the EMNLP'2025 Findings paper "GLProtein: Global-and-Local Structure Aware Protein Representation Learning". GLProtein combines protein-masked modelling with triplet structure similarity scoring, protein 3D distance encoding and substructure-based amino acid molecule encoding.

---

Environments for Pre-training

Main dependencies

• python 3.7
• pytorch 1.9
• transformer 4.5.1+
• deepspeed 0.6.5
• lmdb

We make small changes to the deepspeed.py file under transformers library (❗required for pre-training). The changes can be applied by running:

cp replace_code/deepspeed.py path_to/python3.7/dist-packages/transformers/deepspeed.py

---

For Global Structure Information Modeling

We employ TM-Vec to construct a robust set of positive and negative samples for protein structure similarity analysis.

---

For Substructure-based Molecular Encoding

We use mol2vec to process and derive representations for amino acid molecules to obtain fine-grained molecular structure information.

---

Environments for Downstream Tasks

Main dependencies

• python 3.7
• pytorch 1.9
• transformer 4.5.1+
• lmdb
• tape_proteins
• scikit-multilearn
• PyYAML
• PyTorch Geometric

Note PyTorch Geometric is required for the PPI (protein-protein interaction) task. Check your PyTorch and cuda versions, and follow the installation instructions to install correctly.

Since the tape_proteins library only implemented the P@L metric for the contact prediction task, we add the P@L/5 and P@L/2 metrics by running the following script:

cp replace_code/tape/modeling_utils.py pathto/python3.7/dist-packages/tape/models/modeling_utils.py

---

Data Preparation

The pre-training protein 3D coordinates datasets can be dowloaded from here.

The data for TAPE tasks and the PPI task can be downloaded from here. The data for the PROBE tasks can be acquired via link.

---

Models Preparation

You also need to download the following pre-trained models:

• ProtBERT for initializing the protein encoder.

Then, configure paths in script/run_pretrain.sh (PRETRAIN_DATA_DIR, ENCODER_MODEL_PATH, TEXT_MODEL_PATH) accordingly.

Run the following script for pre-training:

sh ./script/run_pretrain.sh

The detailed arguments are listed in src/training_args.py.

---

Downstream Tasks (Fine-tuning)

In this part, we fine-tune the pre-trained model (i.e., a checkpoint of GLProtein) on various downstream tasks.

❗NOTE: You will need to change some paths for downstream data and extracted embeddings (PPI and PROBE tasks) before running the code.

TAPE Tasks

Secondary structure prediction, contact prediction, remote homology detection, stability prediction, and fluorescence prediction are tasks from TAPE.

For these tasks, we provide scripts for fine-tuning under script/ (❗Preferred). You may need to modify the DATA_DIR and OUTPUT_DIR paths in run_main.sh before running the scripts.

For example, you can fine-tune GLProtein for contact prediction by running the following script:

sh ./script/run_contact.sh

You can also use the running codes in run_downstream.py to write shell files with custom configurations.

• run_downstream.py: support {ss3, ss8, contact, remote_homology, fluorescence, stability} tasks;
• run_stability.py: support stability task;

An example of fine-tuning GLProtein for contact prediction (script/run_contact.sh) is as follows:

bash run_main.sh \
      --model output/pretrained/GLProtein/encoder \
      --output_file contact-GLProtein \
      --task_name contact \
      --do_train True \
      --epoch 5 \
      --optimizer AdamW \
      --per_device_batch_size 1 \
      --gradient_accumulation_steps 8 \
      --eval_step 50 \
      --eval_batchsize 1 \
      --warmup_ratio 0.08 \
      --learning_rate 3e-5 \
      --seed 3 \
      --frozen_bert False

Arguments for the training and evalution script are as follows,

• --task_name: Specify downstream task. The script supports {ss3, ss8, contact, remote_homology, fluorescence, stability} tasks;
• --model: The name or path of a pre-trained protein language model checkpoint.
• --output_file: The path to save fine-tuned checkpoints and logs.
• --do_train: Specify if you want to fine-tune the pretrained model on downstream tasks. Set this to False if you want to evaluate a fine-tuned checkpoint on the test set.
• --epoch: Number of epochs for training.
• --optimizer: The optimizer to use, e.g., AdamW.
• --per_device_batch_size: Batch size per GPU.
• --gradient_accumulation_steps: The number of gradient accumulation steps.
• --eval_step: Number of steps to run evaluation on validation set.
• --eval_batchsize: Evaluation batch size.
• --warmup_ratio: Ratio of total training steps used for a linear warmup from 0 to learning_rate.
• --learning_rate: Learning rate for fine-tuning
• --seed: Set seed for reproducibility
• --frozen_bert: Specify if you want to freeze the encoder in the pretrained model.

More detailed parameters can be found in run_main.sh. Note that the best checkpoint is saved in OUTPUT_DIR/.

PROBE Tasks

Semantic similarity inference and binding affinity estimation are tasks from PROBE. The code for PROBE can be found in src/benchmark/PROBE.

To validate GLProtein on these two tasks, you need to:

• Configure paths in src/benchmark/PROBE/extract_embeddings.py to your pre-trained model and PROBE data accordingly.
• Extract embeddings using pre-trained GLProtein by running src/benchmark/PROBE/extract_embeddings.py.
• Change paths listed in src/benchmark/PROBE/bin/probe_config.yaml accordingly.
• Run src/benchmark/PROBE/bin/PROBE.py.

Detailed instructions and explanations of outputs can be found in PROBE.

PPI Task

The code for PPI can be found in src/benchmark/GNN_PPI, which was modified based on GNN-PPI.

To validate GLProtein for PPI prediction:

• Configure paths in src/benchmark/GNN_PPI/extract_protein_embeddings.py to your pre-trained model and PPI data accordingly.
• Extract embeddings using pre-trained GLProtein by running src/benchmark/GNN_PPI/extract_protein_embeddings.py.
• Change paths listed in src/benchmark/GNN_PPI/run.py accordingly.
• Run src/benchmark/GNN_PPI/run.py.

---

Acknowledgements

The implementation cannot proceed without referencing OntoProtein and KeAP.

---

Citation

@article{liu2025glprotein,
  title={GLProtein: Global-and-Local Structure Aware Protein Representation Learning},
  author={Liu, Yunqing and Fan, Wenqi and Wei, Xiaoyong and Li, Qing},
  journal={arXiv preprint arXiv:2506.06294},
  year={2025}
}