Protein Language Modeling Course

---

1. Topics
2. Course Structure
3. References
4. Resources

Course Structure

Working Locally

1. Install Docker
2. From the root of the repository in the terminal run:

   docker compose up --build -d
   

3. Open JupyterLab in any browser via 0.0.0.0:8888

To stop run: docker compose stop To stop and remove containers: docker compose down

Working VM

1. Once you have a VM, download docker in it
   • (e.g., apt install via ssh)
   • e.g. in VM shell run below

     # Add Docker's official GPG key:
     sudo apt-get update
     sudo apt-get install ca-certificates curl
     sudo install -m 0755 -d /etc/apt/keyrings
     sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
     sudo chmod a+r /etc/apt/keyrings/docker.asc
     
     # Add the repository to Apt sources:
     echo \
     "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
     $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
     sudo apt-get update
     
     sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
     

2. Clone this repo into the VM or to keep things slim you can move only the below needed files onto VM (e.g., via scp)
   • notebooks and data folders
   • Dockerfile, docker-compose.yml
   • requirements.txt
3. Then run docker compose up --build -d
   • if you encounter permission denied you may need to run the above command with elevated permissions e.g. sudo docker compose up --build -d
   • Note: for troubleshooting you can use docker logs <container-name> e.g., docker logs jupyter
4. The URL to the JupyterLab will be: <VM's IP address>:8888 e.g. 12.26.38.408:8888
   • Note: may have to expose port 8888 via the VM's network security group
5. On your local device, open a browser of your choosing. In the address bar input the URL to JupyterLab. And you should be good to go :)
6. Remember to stop the VM container once you are done

References

---

1. Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio

2. Attention Is All You Need Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

3. MSA Transformer Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John F. Canny, Pieter Abbeel, Tom Sercu, Alexander Rives

4. Transformer-based deep learning for predicting protein properties in the life sciences Abel Chandra, Laura Tünnermann, Tommy Löfstedt, Regina Gratz

5. BERTology Meets Biology: Interpreting Attention in Protein Language Models Jesse Vig, Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, Nazneen Fatema Rajani

6. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel, Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Baek, David Baker

7. Large language models generate functional protein sequences across diverse families Ali Madani, Ben Krause, Eric R. Greene, Subu Subramanian, Benjamin P. Mohr, James M. Holton, Jose Luis Olmos Jr., Caiming Xiong, Zachary Z. Sun, Richard Socher, James S. Fraser & Nikhil Naik

8. Learning functional properties of proteins with language modelsSerbulent Unsal, Heval Atas, Muammer Albayrak, Kemal Turhan, Aybar C. Acar & Tunca Doğan

9. Learning the protein language: Evolution, structure, and function Tristan Bepler, Bonnie Berger

10. The language of proteins: NLP, machine learning & protein sequences Dan Ofer, Nadav Brandes, Michal Linial

11. Evolutionary-scale prediction of atomic-level protein structure with a language model ZEMING LIN, HALIL AKIN, ROSHAN RAO, BRIAN HIE, ZHONGKAI ZHU, WENTING LU, NIKITA SMETANIN, ROBERT VERKUIL, ORI KABELI, ..., ALEXANDER RIVES

12. Generative power of a protein language model trained on multiple sequence alignments Damiano Sgarbossa, Umberto Lupo, Anne-Florence Bitbol

13. How Huge Protein Language Models Could Disrupt Structural Biology

14. Embeddings from protein language models predict conservation and variant effects Céline Marquet, Michael Heinzinger, Tobias Olenyi, Christian Dallago, Kyra Erckert, Michael Bernhofer, Dmitrii Nechaev & Burkhard Rost

15. Collectively encoding protein properties enriches protein language models Jingmin An & Xiaogang Weng

16. ProGen: Language Modeling for Protein Generation Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R. Eguchi, Po-Ssu Huang, Richard Socher

17. Transformer protein language models are unsupervised structure learners Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, Alexander Rives

18. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences Alexander Rives, Joshua Meier, Tom Sercu and Rob Fergus

19. NetSurfP-3.0: accurate and fast prediction of protein structural features by protein language models and deep learning Magnus Haraldson Høie, Erik Nicolas Kiehl, Bent Petersen, Morten Nielsen, Ole Winther, Henrik Nielsen, Jeppe Hallgren, Paolo Marcatili

20. Modeling Protein Using Large-scale Pretrain Language Model Yijia Xiao, Jiezhong Qiu, Ziang Li, Chang-Yu Hsieh, Jie Tang github

21. Deciphering antibody affinity maturation with language models and weakly supervised learning Jeffrey A. Ruffolo, Jeffrey J. Gray, Jeremias Sulam github

22. Protein embeddings improve phage-host interaction prediction Mark Edward M. Gonzales, Jennifer C. Ureta, View ORCID ProfileAnish M.S. Shrestha github

23. ProteinBERT: a universal deep-learning model of protein sequence and function Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, Michal Linial github

24. ProtGPT2 is a deep unsupervised language model for protein design Noelia Ferruz, Steffen Schmidt & Birte Höcker Hugging Face

25. Protein-Protein Interaction Prediction is Achievable with Large Language Models Logan Hallee, Jason P. Gleghorn

26. Accurate prediction of virus-host protein-protein interactions via a Siamese neural network using deep protein sequence embeddings Sumit Madan, Victoria Demina, Marcus Stapf, Oliver Ernst, Holger Fröhlich

27. Structure-informed Language Models Are Protein Designers Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei YE, Quanquan Gu

28. Graph-BERT and language model-based framework for protein–protein interaction identification Kanchan Jha, Sourav Karmakar & Sriparna Saha

29. Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Charlotte Rochereau, Burkhard Rost

30. Contrastive learning in protein language space predicts interactions between drugs and protein targets Rohit Singh, Samuel Sledzieski, Bryan Bryson, Bonnie Berger

31. De novo design of protein structure and function with RFdiffusion Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel, Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Sergey Ovchinnikov, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Baek & David Baker

32. Single-sequence protein structure prediction using a language model and deep learning Ratul Chowdhury, Nazim Bouatta, Surojit Biswas, Christina Floristean, Anant Kharkar, Koushik Roy, Charlotte Rochereau, Gustaf Ahdritz, Joanna Zhang, George M. Church, Peter K. Sorger & Mohammed AlQuraishi

33. Before and after AlphaFold2: An overview of protein structure prediction Letícia M. F. Bertoline, Angélica N. Lima, Jose E. Krieger, and Samantha K. Teixeira

34. Evaluating Protein Transfer Learning with TAPE Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, Yun S. Song

35. FLIP: Benchmark tasks in fitness landscape inference for proteins Christian Dallago, Jody Mou, Kadina E. Johnston, Bruce J. Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, Kevin K. Yang

36. Standards, tooling and benchmarks to probe representation learning on proteins Joaquin Gomez Sanchez, Sebastian Franz, Michael Heinzinger, Burkhard Rost, Christian Dallago

37. Learning meaningful representations of protein sequences Nicki Skafte Detlefsen, Søren Hauberg & Wouter Boomsma

38. Language modelling for biological sequences – curated datasets and baselines Jose Juan Almagro Armenteros, Alexander Rosenberg Johansen, Ole Winther, Henrik Nielsen

39. Language models generalize beyond natural proteins Robert Verkuil, Ori Kabeli, Yilun Du, Basile I. M. Wicky, Lukas F. Milles, Justas Dauparas, David Baker, Sergey Ovchinnikov, Tom Sercu, Alexander Rives

40. ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction Seyone Chithrananda, Gabriel Grand, Bharath Ramsundar

41. SETH predicts nuances of residue disorder from protein embeddings Dagmar Ilzhöfer, Michael Heinzinger, Burkhard Rost

42. Exploiting pretrained biochemical language models for targeted drug design Gökçe Uludoğan, Elif Ozkirimli, Kutlu O Ulgen, Nilgün Karalı, Arzucan Özgür

43. xTrimoPGLM: unified 100-billion-parameter pretrained transformer for deciphering the language of proteins Bo Chen, Xingyi Cheng, Pan Li, Yangli-ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, Chiming Liu, Aohan Zeng, Yuxiao Dong, Jie Tang, Le Song

44. PTM-Mamba: a PTM-aware protein language model with bidirectional gated Mamba blocks Fred Zhangzhi Peng, Chentong Wang, Tong Chen, Benjamin Schussheim, Sophia Vincoff, Pranam Chatterjee

45. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, Chris Olah

46. InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders Elana Simon, James Zou

Other Resources

---

• Protein Language Workshop

• Awesome protein representation learning

• Evolutionary Scale Modeling - ESM

• Huggingface ESM

• Ankh

• Protein Sequence Embeddings (ProSE)

• ProTrans

• ConPlex

• Deep Learning with Proteins

• Awesome AI-based for Protein Design

• Amazon SageMaker Protein Classification

• Huggingface Protein Language Modeling - pytorch

• Huggingface Protein Language Modeling - tensorflow

• Building a Text Generation Model from Scratch

• Training the Transformer Model

• GearNet: Geometry-Aware Relational Graph Neural Network

• UniLanguage

• SETH