DistilProtBert

Official DistilProtBert implementation, a distilled version of ProtBert-UniRef100 model.

Check out our paper DistilProtBert: A distilled protein language model used to distinguish between real proteins and their randomly shuffled counterparts for more details.

Model is available at Hugging Face model page.

Pretraining dataset: UniRef50

This repository is based on ProtBert-UniRef100 implementation from ProtTrans repository.

Model details

Evaluation tasks

Secondary structure prediction (Q3)

Please follow SS3 fine-tuning notebook and change model_init first argument to 'yarongef/DistilProtBert'.

def model_init():
    return AutoModelForTokenClassification.from_pretrained('yarongef/DistilProtBert',
                                                           num_labels=len(unique_tags),
                                                           id2label=id2tag,
                                                           label2id=tag2id,
                                                           gradient_checkpointing=False)

Datasets can be found at ProtTrans repository.

Results

---

Membrane vs water soluble (Q2)

Please follow MS fine-tuning notebook and change model_init first argument to 'yarongef/DistilProtBert'.

def model_init():
    return AutoModelForSequenceClassification.from_pretrained('yarongef/DistilProtBert')

Datasets can be found at ProtTrans repository.

Results

---

Distinguish between proteins and their k-let shuffled versions

Notebook for creating the datasets.

Datasets used for training and test: singlets, doublets and triplets.

Training:

• Extract the relevant k-let training set features via feature extraction notebook
• Run the following training script

Inference:

• Download the relevant k-let classification model from here
• Extract the relevant k-let test set features via feature extraction notebook
• Run inference notebook

Singlet

Doublet

Triplet

Contact

Create an issue to report bugs, propose new functions or ask for help.

License

MIT License

Citation

If you use this code or one of our pretrained models for your publication, please cite our paper:

@article {
	author = {Geffen, Yaron and Ofran, Yanay and Unger, Ron},
	title = {DistilProtBert: A distilled protein language model used to distinguish between real proteins and their randomly shuffled counterparts},
	year = {2022},
	doi = {10.1093/bioinformatics/btac474},
	URL = {https://doi.org/10.1093/bioinformatics/btac474},
	journal = {Bioinformatics}
}